How do I extract ALL urls in an entire website?

[ChrisChris]

I'd like to extract all the urls in a website to find expired domains. Id like to just enter funnyDOTcom and let it extract all urls from ALL the pages on the site. Simple but sweet.

Any good programs for this?

Xenu dies on me after 2mil.
Scrapebox ink extractor is retarded when it comes to this.
Other options?

 

[zagard]

:1: screaming frog?

 

[ChrisChris]

:1: screaming frog?

ooo, I had forgotten about that.

Is it capable of scanning even 10m pages on 1 site, then scanning them all for broken links without freezing, or dying. Im just very sick of scanning for broken links and ending up with like 3% scanned then "run out of memory".

 

[maddawgmackem]

haven't used screaming frog but won't scrape box and the broken link checker work ?

 

[ziplack]

i guest scrapebox can do the job

 

[zagard]

Tested Screaming Frog with dmoz, i gt 25 url/s at 20 threads, at that speed 10m is like 5 days and when i run it on max threads (200) the url/s is at 50-ish but the program turn irresponsive after 10 mins bcs proc usage was like 94%.

too slow, myb shld stick to scrapebox as the two ppl above said

ooo, I had forgotten about that.

Is it capable of scanning even 10m pages on 1 site, then scanning them all for broken links without freezing, or dying. Im just very sick of scanning for broken links and ending up with like 3% scanned then "run out of memory".

 

[misteryou.]

i use this stuff http://www.brokenlinkcheck.com to scan sites and find broken links because i didn't know any other way lol

 

[supacatt]

i use this stuff http://www.brokenlinkcheck.com to scan sites and find broken links because i didn't know any other way lol

Never heard of this site before. Thanks for the link.

 

[ChrisChris]

Ya, Im still using xenu for the time being, but will grab that scraming frog bulk checker license to see if its at least a little more consistent.

Wish I knew of other alternatives tho, without building one myself.

Any other bulk broken link checkers out there? To find expired domains for pbn?

 

[ChrisChris]

lol, screaming frog unresponsive, lovely.

Ya, I just want some expired domains with links to high authority places like wikipedia, so I thought scrapebox could handle it, but it freezes after a very short while, much worse than xenu. Screaming frogs free version sure doesnt do much, 500 link limit is good for a test, but no more.

How do you like screaming frogs unlimited $99 version? 5 days is pretty bad, but better than nothing, which is what Ive got now lol.

 

[Repulsor]

I can make a scrapper which would go to a link and fetch all links for further fetching, and scrapes domains from each and every page and saves into a database. You can then export the domains as a file of whatever size you need, say 10k domains and you can run it through scrapebox or any other tools for broken links.

Having it coded well, it would never really load the server at all. A shared hosting would be more than sufficient.

 

[jo199]

Scrapebox is good dude.