Tool to get all URLs from root domain (large list)

J

jb2008

Senior Member
Joined
Jul 15, 2010
Messages
1,158
Reaction score
976
I have tried things like Gwebsitecrawler, all SB addons, but nothing does the task of getting all the URLs of a site (i've got a list of 120,000 root domains whose internal URLs I need to get).

This is basically an alternative to the G0ogle site: command. The reason I need to do this is because G indexes only a small fraction of an entire site, and I am missing about 90% of viable targets.

Does anyone have any ideas?
 
Advertise on BHW
Maybe you should buy a SBOX coaching program to learn all of its functionalities. If you don't know this, then you don't know a lot of the secrets. :P
 
Hey OP,

I can code a very simple python script for you (no worries, I'm not gonna charge) that would do what you are asking if you are willing to wait for about 24 hours (as I have a ton of shit to do...).

Drop me a PM.

Cheers
 
Ok,

I'll do you guys a favour and will code it up as soon as I get some time to. You'd basically just need a crawler and thats all. Give me 24-48 hours. Would you like the source code or should I make the tool publicly available through my gapps account?

Cheers
 
You could use Xenu's link sleuth, and automate it in some way...
 
@macdonjo, wtf are you talking about, SB can harvest so you could use the site operator (which as I explained does not return anywhere near the amount of URLs there actually are) or if you get an XML sitemap you can use the sitemap scraper, but many, many sites don't have xml sitemaps, so please enlighten me...

@scriptomania, well, if you are able to do it successfully I would be willing to pay. I'm a man of modest means but I don't expect custom made scripts for free. If I pay we would keep it private, but alternatively you could choose to release it to the public. It depends on what you want to do.

Yes, it's basically a crawler. I basically need it to go through a list of root domains and return all URLs in the site for each domain. So it's basically like a lot of sitemap generators, but automated for multiple domains, rather than just generating all URLs for just 1 root domain (site).

It's very useful because search engines don't index a massive amount of URLs for whatever reasons, many of which are viable posting targets.
 
link can be harvest by SB but google allow only 1000 url.
@scriptomania, waiting to see what you will make for us ;)
 
I use webdataextractor too.... It scans the site faster and gets all URL's...it's easy.
 
I looked at WDE and it can only do 1 site at a time.

I have a list of 100,000+ root domains that I need to have all URLs for.
 
Last edited:
My 2 cents here, just use scrapebox's link checker. Spend a little time to do an experiment on it and I'm sure you will get the way. :)
 
jb2008 said:
That looks very interesting... can it do large lists of URLs (root domains) and not just one at a time?
Click to expand...

It can do it to list of URL's. Url's will be picked up from a text file, it will scan each one on them one by one. It can harvest urls, emails, phone & fax ( I never tried this option) with lots of advance settings. Moreover its the fastest harvester I have ever used.
 
Hmm never knew about the 1000 limitation per site. What about yahoo?
 
100,000+ root domains, and need it to spider all of those? so if they only had 10 pages, you'd get a result of 1M URLs?
 
@luccha, but it won't go through the whole site and list all the URLs for all the sites. If you load a list of URLs, it just gets data for one page.

@fred, And SB link checker? It does have its uses but I suppose you are suggesting checking for links (internal only) on the page, but the problem with that is that it won't spider the whole site. Unless EVERY page is linked from the home page, it won't return much. I tried this and I wasn't getting many URLs at all. It's a nice idea, but in practise, it doesn't work.

@claymc, yes, I actually have 2 lists of 100,000 URLs. So 200k root domains total. I need to spider all these, extract all the URLs, and then manually filter in notepad++ to hopefully filter out a lot of unpostable pages. I would probably end up with around 200,000,000 URLs in one text file, so I would have no choice but to split it into about 50 parts and edit each one individually. (ouch, but can be done in a couple days)

Since there seems to be no solution for what at first to me appeared to be a simple problem, I have just paid someone on vworker to build me a crawler. More than I was hoping to pay but I need this.
 
Last edited:
Who not just create a sitemap, then export the urls?

Or sleuth?
 
Well, it doesn't matter but I don't want something where you have to insert every damn link...
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top