1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Link Extractor with Scrapy

Discussion in 'Hire a Freelancer' started by TrailBlazer, Aug 28, 2016.

  1. TrailBlazer

    TrailBlazer Junior Member

    Joined:
    Aug 11, 2012
    Messages:
    170
    Likes Received:
    51
    I want to hire a Python/Scrapy expert to code me and teach me how to use a Scrapy bot that does the following.

    I want to be able to have Scrapy read a text file with a seed list of around 100k urls, have Scrapy visit each URL, and extract all external URLs (URLs of Other Sites) found on each of those Seed URLs and export the results to a separate text file.

    Scrapy should only visit the URLs in the text file, not spider out and follow any other URL.

    I want to be able to have Scrapy work as fast as possible, I have a very powerful server with a 1GBS line. Each URL in my list is from a unique domain, so I won't be hitting any 1 site hard at all and thus won't be encountering IP blocks.

    PM me your skype if you think you can make this