Any ideas to prevent getting blacklisted while scraping?

tounsi7orr

BANNED
Joined
Apr 21, 2014
Messages
180
Reaction score
13
Hello,
I'm a web scraper (Not expert), I'm currently trying to make a bot to scrape amazon products.
I face a terrible problem: I'm getting banned when I scrape about 300-400 pages.
I used some fake browser agents and they worked a little better but still getting banned.
I read that I must use proxies but it's expensive for me to buy dozens of proxies just for testing.
Note: my scraping speed is 2500 pages/hour (It's kinda slow)
My goal is to scrap 1 million pages in a week, I will use multithreading technique, but first how to avoid getting blacklisted?
 
On that kind of bulk pages. It's good to implement nearly human behavioral bots to not get banned.My personal statement is 1 million pages too much for 1 week.
 
well,you are getting blacklisted obviously,because you are requesting too much pages in a small amount of time. So you have only two options
1. do it slowly (well,yeah,its not much useful)
2. use proxies. Why don't you try to look for a free list and try it just for a purpose of testing and then think about buying some good ones?
 
I get that this is an old thread, but FWIW, I have been able to scrape to my hearts content, capturing millions of images with one script (that is what it is supposed to do), with not so much as a wrinkle. Actually, my NAS has filled a few times, which was annoying and caused a lot of errors, but other than that and some slow system performance from my boxes, I am thinking I am almost ready to up my ante and get more refined. Then, perhaps see about doing some paid scraping.
 
Back
Top