1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Extracting URL's

Discussion in 'Black Hat SEO' started by BuzzKill, Nov 15, 2007.

  1. BuzzKill

    BuzzKill Executive VIP

    Joined:
    Aug 30, 2007
    Messages:
    508
    Likes Received:
    173
    Occupation:
    Entrepreneur
    Location:
    Milwaukee WI
    What would be the best program for extracting URL's? Specifically from SE's. It would be great if you could control how deep it went so that it didnt spider off into oblivion. Give it a search url and have it grab all the URLs that are displayed. Any ideas?
     
  2. theprovider777

    theprovider777 Registered Member

    Joined:
    Sep 3, 2007
    Messages:
    93
    Likes Received:
    7
    Occupation:
    To Provide and Serve.
    Location:
    Land of many provisions.
    Home Page:
    Well, a simple PHP/CURL script would do the job, just work out the regular expression or google it. My brain is incapable of memorizing regex; I have to go through the entire regex tutorial everytime I need it.

    But yeah, something like that should do it. It'll just take a little more research and work on your part. Hey, it's good for you! ;)
     
  3. Alchemist

    Alchemist Newbie

    Joined:
    Nov 7, 2007
    Messages:
    13
    Likes Received:
    1
    Location:
    England
    here yah go just made a simple proggy in VB that will extract all the urls from a webpage ive included the source code just incase you want to add anything..
    hope this is what you was looking for

    gethere.org/ExUrls.rar

    would post an image but the crazy rules only allow me 2 post image after 15 posts.. oh well
     
  4. Alchemist

    Alchemist Newbie

    Joined:
    Nov 7, 2007
    Messages:
    13
    Likes Received:
    1
    Location:
    England
    [​IMG]

    http://gethere.org/ExUrls.rar
     
  5. BuzzKill

    BuzzKill Executive VIP

    Joined:
    Aug 30, 2007
    Messages:
    508
    Likes Received:
    173
    Occupation:
    Entrepreneur
    Location:
    Milwaukee WI
    You are a gentleman and a scholar.
     
  6. BuzzKill

    BuzzKill Executive VIP

    Joined:
    Aug 30, 2007
    Messages:
    508
    Likes Received:
    173
    Occupation:
    Entrepreneur
    Location:
    Milwaukee WI
    This is true. I must say, the people in this forum are a find bunch of people. Lots of patience and lots of brains, I have only found one other forum with the same type of people, Im fairly sure its bad etiquette to post other boards, but the name has an 8 in it. Smart, helpful people as well.
     
  7. Alchemist

    Alchemist Newbie

    Joined:
    Nov 7, 2007
    Messages:
    13
    Likes Received:
    1
    Location:
    England
    No worrys i can make the php script theprovider777 mentioned also if you need it just uses Curl & Split()
     
  8. eklavya

    eklavya Newbie

    Joined:
    Oct 23, 2007
    Messages:
    24
    Likes Received:
    4
    My two cents :

    This is a free utility which extract URLs from SEs very fast.

    www (dot) focalmedia (dot)net/urlextractor

    Regards
     
  9. botosaurus

    botosaurus Newbie

    Joined:
    Nov 15, 2007
    Messages:
    5
    Likes Received:
    0
    wow! This is great stuff! Thanks a ton, Alchemist!
     
  10. bathgate

    bathgate Newbie

    Joined:
    Nov 17, 2007
    Messages:
    11
    Likes Received:
    0
    thanks Alchemist just what i was looking 4 :)
     
  11. Wombat

    Wombat Registered Member

    Joined:
    Apr 26, 2007
    Messages:
    84
    Likes Received:
    66
    Xenu link sleuth is very good and free too.
    http://home.snafu.de/tilman/xenulink.html