Tool to get all URLs from root domain (large list)

Exactly banel, I have searched but I haven't found a "sitemap generator" that can work automated for a list of root domains.
 
Advertise on BHW
jb2008 said:
@luccha, but it won't go through the whole site and list all the URLs for all the sites. If you load a list of URLs, it just gets data for one page.

@fred, And SB link checker? It does have its uses but I suppose you are suggesting checking for links (internal only) on the page, but the problem with that is that it won't spider the whole site. Unless EVERY page is linked from the home page, it won't return much. I tried this and I wasn't getting many URLs at all. It's a nice idea, but in practise, it doesn't work.

@claymc, yes, I actually have 2 lists of 100,000 URLs. So 200k root domains total. I need to spider all these, extract all the URLs, and then manually filter in notepad++ to hopefully filter out a lot of unpostable pages. I would probably end up with around 200,000,000 URLs in one text file, so I would have no choice but to split it into about 50 parts and edit each one individually. (ouch, but can be done in a couple days)

Since there seems to be no solution for what at first to me appeared to be a simple problem, I have just paid someone on vworker to build me a crawler. More than I was hoping to pay but I need this.
Click to expand...


Negative ! It will pick each url one by one. Then scan the whole site for that url and move on to next.
 
jb2008 said:
Exactly banel, I have searched but I haven't found a "sitemap generator" that can work automated for a list of root domains.
Click to expand...

Well I he had a list of about 100 then I might just be able to help him out

But you guys carry on working it out
 
scriptomania said:
Ok,

I'll do you guys a favour and will code it up as soon as I get some time to. You'd basically just need a crawler and thats all. Give me 24-48 hours. Would you like the source code or should I make the tool publicly available through my gapps account?

Cheers
Click to expand...

PM me your PayPal, ill send you a Beer, if you'll give me this smart tool :)

Thank You
 
jb2008 said:
@fred, And SB link checker? It does have its uses but I suppose you are suggesting checking for links (internal only) on the page, but the problem with that is that it won't spider the whole site. Unless EVERY page is linked from the home page, it won't return much. I tried this and I wasn't getting many URLs at all. It's a nice idea, but in practise, it doesn't work.
Click to expand...


Sorry, I mean Scrapebox's link extractor :)
 
OK, so I tried Xenu's but if I want to "crawl", let's say, 100 websites, I get over 1million links to analyze because Xenu's count links to images as actual links and this "bug" is f***ing wrong.
Can anyone suggest a site crawler similar to Xenu's, that can deal with LISTS and that get only ACTUAL URLS, and not images, css files and other shit ???
 
No one can help me with a software that can do what I said upper???
 
I've done it !
But this gets only links from homepage. :(
 
wannabie said:
Your not doing it right then
Click to expand...
I'm using this:
Code: 
=importXml("[URL="http://www.google.com/"]www.domain.com[/URL]"; "//a/@href")
And I get only links from first page. What should I use to display all the links?
 
Hello I think you just upload your domains list into scrapebox's link extractor, check for internal then click start. It would scrape their internal backlinks which is the pages of their domains..
 
If I understand you correct, I think you can do what you need with A1 Sitemap Generator if you are willing to automate using the command line. (If you disable "Simplified easy mode" you can configure the options to pick the kinds of URLs you want. Then load that project using command line and just override the website URL.)
 
Screaming Frog SEO Spider is what I use for this stuff. :)
 
Hello,

I need some help:

Homepage: mathematicalforex.c!o!m
Hidden website: mathematicalforex.c!o!m/cohesys27. h t m l

How can I find these hidden websites? The are no password restrictions. But such sites are not listed on google. I could not find a robots.txt. Is there any useful tool? Will Screaming frog find it?
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top