1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Scrap many domains

Discussion in 'Black Hat SEO' started by ExpiredDomain, Aug 24, 2014.

  1. ExpiredDomain

    ExpiredDomain Newbie

    Joined:
    Nov 15, 2013
    Messages:
    7
    Likes Received:
    0
    Hi,

    Hello there,

    I need a tool to crawl hundreds of thousands of URLs currently I use Xenu but it takes too much time.

    Is there a tool or a tool to mass-crawler URL?

    Thank you for your help
     
  2. SEO Power

    SEO Power Elite Member

    Joined:
    Jul 14, 2014
    Messages:
    2,637
    Likes Received:
    680
    Occupation:
    Self employed
    Location:
    Houston, TX
    Xenu is the best I know of.
     
  3. Bot Solutions

    Bot Solutions Regular Member

    Joined:
    Aug 19, 2014
    Messages:
    215
    Likes Received:
    361
    Occupation:
    Bot maker
    Location:
    Hungary
    Home Page:
    Scrape what?
    Email addresses?
     
  4. ExpiredDomain

    ExpiredDomain Newbie

    Joined:
    Nov 15, 2013
    Messages:
    7
    Likes Received:
    0
    No, I just want to crawl URLs and retrieve some information.

    Xenu is fine but I need a lot more power to crawl, I now launching on multiple computers simultaneously, but this solution is not good.

    If someone has a solution even a paid tool or software to create, I'm interested.
     
  5. King SEO

    King SEO Jr. VIP Jr. VIP

    Joined:
    Mar 19, 2014
    Messages:
    1,784
    Likes Received:
    139
    Occupation:
    Full Time Marketer
  6. ExpiredDomain

    ExpiredDomain Newbie

    Joined:
    Nov 15, 2013
    Messages:
    7
    Likes Received:
    0
    Thank you King SEO

    To be more precise in my application, I want a tool that would give me only broken links of a site (no such host in Xenu).

    Is there a fee tools to do mass?
    Does it seem complicated to develop such a tool?

    Thank you for your help
     
  7. theauthorityblog

    theauthorityblog Junior Member

    Joined:
    Jul 14, 2014
    Messages:
    190
    Likes Received:
    57
    Several people here have claimed to have custom crawlers. You can PM them to see if they want to sell it to you. Also there is an opensource project called scrappy which I believe is a decent scrapper and you can customize if you have some programming experience.
     
  8. PriyankaS

    PriyankaS Newbie

    Joined:
    Oct 30, 2014
    Messages:
    41
    Likes Received:
    2
    Share link please.
     
  9. dzuster

    dzuster Regular Member

    Joined:
    Nov 15, 2008
    Messages:
    401
    Likes Received:
    17
    I also need these tools? I'll try also xenu