tounsi7orr
BANNED
- Apr 21, 2014
- 180
- 13
Hello,
I'm a web scraper (Not expert), I'm currently trying to make a bot to scrape amazon products.
I face a terrible problem: I'm getting banned when I scrape about 300-400 pages.
I used some fake browser agents and they worked a little better but still getting banned.
I read that I must use proxies but it's expensive for me to buy dozens of proxies just for testing.
Note: my scraping speed is 2500 pages/hour (It's kinda slow)
My goal is to scrap 1 million pages in a week, I will use multithreading technique, but first how to avoid getting blacklisted?
I'm a web scraper (Not expert), I'm currently trying to make a bot to scrape amazon products.
I face a terrible problem: I'm getting banned when I scrape about 300-400 pages.
I used some fake browser agents and they worked a little better but still getting banned.
I read that I must use proxies but it's expensive for me to buy dozens of proxies just for testing.
Note: my scraping speed is 2500 pages/hour (It's kinda slow)
My goal is to scrap 1 million pages in a week, I will use multithreading technique, but first how to avoid getting blacklisted?