Extracting URL's

[BuzzKill]

What would be the best program for extracting URL's? Specifically from SE's. It would be great if you could control how deep it went so that it didnt spider off into oblivion. Give it a search url and have it grab all the URLs that are displayed. Any ideas?

 

[theprovider777]

Well, a simple PHP/CURL script would do the job, just work out the regular expression or google it. My brain is incapable of memorizing regex; I have to go through the entire regex tutorial everytime I need it.

But yeah, something like that should do it. It'll just take a little more research and work on your part. Hey, it's good for you!

 

[Alchemist]

here yah go just made a simple proggy in VB that will extract all the urls from a webpage ive included the source code just incase you want to add anything..
hope this is what you was looking for

gethere.org/ExUrls.rar

would post an image but the crazy rules only allow me 2 post image after 15 posts.. oh well

 

[Alchemist]

http://gethere.org/ExUrls.rar

 

[BuzzKill]

You are a gentleman and a scholar.

 

[BuzzKill]

Well, a simple PHP/CURL script would do the job, just work out the regular expression or google it. My brain is incapable of memorizing regex; I have to go through the entire regex tutorial everytime I need it.

But yeah, something like that should do it. It'll just take a little more research and work on your part. Hey, it's good for you!

This is true. I must say, the people in this forum are a find bunch of people. Lots of patience and lots of brains, I have only found one other forum with the same type of people, Im fairly sure its bad etiquette to post other boards, but the name has an 8 in it. Smart, helpful people as well.

 

[Alchemist]

No worrys i can make the php script theprovider777 mentioned also if you need it just uses Curl & Split()

 

[eklavya]

My two cents :

This is a free utility which extract URLs from SEs very fast.

www (dot) focalmedia (dot)net/urlextractor

Regards

 

[botosaurus]

wow! This is great stuff! Thanks a ton, Alchemist!

 

[bathgate]

thanks Alchemist just what i was looking 4

 

[Wombat]

Xenu link sleuth is very good and free too.
http://home.snafu.de/tilman/xenulink.html