TrailBlazer
Junior Member
- Aug 11, 2012
- 176
- 52
I want to hire a Python/Scrapy expert to code me and teach me how to use a Scrapy bot that does the following.
I want to be able to have Scrapy read a text file with a seed list of around 100k urls, have Scrapy visit each URL, and extract all external URLs (URLs of Other Sites) found on each of those Seed URLs and export the results to a separate text file.
Scrapy should only visit the URLs in the text file, not spider out and follow any other URL.
I want to be able to have Scrapy work as fast as possible, I have a very powerful server with a 1GBS line. Each URL in my list is from a unique domain, so I won't be hitting any 1 site hard at all and thus won't be encountering IP blocks.
PM me your skype if you think you can make this
I want to be able to have Scrapy read a text file with a seed list of around 100k urls, have Scrapy visit each URL, and extract all external URLs (URLs of Other Sites) found on each of those Seed URLs and export the results to a separate text file.
Scrapy should only visit the URLs in the text file, not spider out and follow any other URL.
I want to be able to have Scrapy work as fast as possible, I have a very powerful server with a 1GBS line. Each URL in my list is from a unique domain, so I won't be hitting any 1 site hard at all and thus won't be encountering IP blocks.
PM me your skype if you think you can make this