URL Scraper

Discussion in 'Black Hat SEO Tools' started by Bloox, Jan 7, 2010.

  1. Bloox

    Bloox Regular Member

    Joined:
    Oct 20, 2009
    Messages:
    213
    Likes Received:
    46
    Location:
    Hamburg
    Hey guys,

    there are so many tools around but I was not able to find a (web) software which allows me to collect all URLs out of a text (or a txt file). Do you know any programme that I could use for that?

    Best regards
    Bloox
     
  2. gogetta

    gogetta Jr. VIP Jr. VIP

    Joined:
    Oct 4, 2008
    Messages:
    724
    Likes Received:
    1,327
    Occupation:
    Internet Hustla
    Location:
    Trapped Online
    I know of a tool that can help, but its a copy & paste. Go here: http://www.surf7.net/product/tools/extractor.php

    ..and paste the data in the input box, then click on type of extract and choose web. That should work, but if you have a really large amount of data you may get a script error.
     
    • Thanks Thanks x 1
  3. Bloox

    Bloox Regular Member

    Joined:
    Oct 20, 2009
    Messages:
    213
    Likes Received:
    46
    Location:
    Hamburg
    Thanks for your fast answer! Somehow it does not work for me. Email extracting works, but when I paste some webadresses it always shows "none" in the ouput window (yes, I chose the right extractin type...). Any ideas?
     
  4. gogetta

    gogetta Jr. VIP Jr. VIP

    Joined:
    Oct 4, 2008
    Messages:
    724
    Likes Received:
    1,327
    Occupation:
    Internet Hustla
    Location:
    Trapped Online

    Do you have the http:// or just the www. at he beginning of your urls? Cause they need to have http://

    But you can use excel to add them. Just do a search www. replace http://
     
    • Thanks Thanks x 1
  5. catman08

    catman08 Junior Member

    Joined:
    Jan 11, 2008
    Messages:
    170
    Likes Received:
    109
    Occupation:
    IM
    Location:
    Europe
    I could write such a scraper for you. PM me if a scraper based on PHP works for ya.

    cheers
    catman
     
    • Thanks Thanks x 1
  6. Bloox

    Bloox Regular Member

    Joined:
    Oct 20, 2009
    Messages:
    213
    Likes Received:
    46
    Location:
    Hamburg
    Yes that works! Thank you so much!
     
  7. makingfastcash22

    makingfastcash22 Senior Member

    Joined:
    Feb 15, 2009
    Messages:
    1,152
    Likes Received:
    178
    look for the google scraper on this forum, it works pretty good.

    I just wish there were a way to scrape high ranking .edu only.

    Damn it does scrape .edu

    But I am not sure how to get WP edu's for Comment poster.
     
    Last edited: Jan 7, 2010
  8. planetsucces

    planetsucces Newbie

    Joined:
    Mar 27, 2013
    Messages:
    1
    Likes Received:
    0