A small guide to distributed crawling

[Yee]

Nice read I don’t see threads like this a lot, I do have one question: why favor RabbitMQ over Redis as message queue ?

I always thought holding state in a distributed architecture is an anti-pattern not to mention RabbitMQ high cpu/memory consumption.
It's not ideally fit for modern ETL pipeline like it use to simply by showing database characteristic behavior and therefore holding state which again is not suitable for scaling.

 

[iamcholo]

Nice read I don’t see threads like this a lot, I do have one question: why favor RabbitMQ over Redis as message queue ?

I always thought holding state in a distributed architecture is an anti-pattern not to mention RabbitMQ high cpu/memory consumption.
It's not ideally fit for modern ETL pipeline like it use to simply by showing database characteristic behavior and therefore holding state which again is not suitable for scaling.

I like this thread too.

Things like Apache Kafka or maybe Redis as Queue system (AMPQ specs) could be a replace for RabbitMQ.

 

[iamcholo]

BTW... I found this project in Github.
Hope helps someone.

crawlab
Distributed web crawler admin platform for spiders management regardless of languages and frameworks
https://github.com/crawlab-team/crawlab

 

[pasdoy]

I like this thread too.

Things like Apache Kafka or maybe Redis as Queue system (AMPQ specs) could be a replace for RabbitMQ.

Indeed any of these will do.

 

[jhaniraj333]

Looks Something Good. Let me try.
Thanks for Sharing Buddy.