A small guide to distributed crawling

Yee said:

Nice read I don’t see threads like this a lot, I do have one question: why favor RabbitMQ over Redis as message queue ?

I always thought holding state in a distributed architecture is an anti-pattern not to mention RabbitMQ high cpu/memory consumption.
It's not ideally fit for modern ETL pipeline like it use to simply by showing database characteristic behavior and therefore holding state which again is not suitable for scaling.

Click to expand...

iamcholo said:

I like this thread too.

Things like Apache Kafka or maybe Redis as Queue system (AMPQ specs) could be a replace for RabbitMQ.

Click to expand...