[Question] Finding Noindex URLS

[ghackseo]

Hi guys, I need a pro to figure this out or advise if it's even possible.

Basically I'm trying to find all the url's on a domain that's not indexed within Search Engines. (I don't own the domain.)

I have the root domain obviously and one other inner-page url. Obviously a "site:domain.com" search wont work as nothing is indexed.

 

[mindmaster]

The allmighty scrapebox will save your ass.
First try and scrape using the site/inurl operators. Use all search engines.
Then use the Internal addon scraper. If those not indexed urls have links on the website, scrapebox will find them.
Also sitemap scraper addon might work.

Let us know how it goes.

 

[ghackseo]

Thanks for the response, however, there are zero links to each of these pages from the other pages and neither is there a sitemap, so scraping them isn't really possible.

I'll need some sort of solution to see exactly what's on the server. Anyone?

 

[rogerke]

So, how do you know the domain has noindex tag pages?

 

[ghackseo]

So, how do you know the domain has noindex tag pages?

That's besides the question, but anyway, I know because I've followed a few links in e-mails (it's a sort of "underground" newsletter), but I haven't been part of it for very long and I'm 99% sure there has to be a lot of other pages that I've missed.

 

[rogerke]

That's besides the question, but anyway, I know because I've followed a few links in e-mails (it's a sort of "underground" newsletter), but I haven't been part of it for very long and I'm 99% sure there has to be a lot of other pages that I've missed.

It is a relevant question, because the same method you used to find a few links might be scaled up. Since it seem to be newsletters you're going after you might try to find some common footprints in the URL structure (like newsletter-07-07.html, newsletter07-08.html) and find them all. Alternatively if the newsletters have backlinks you might use a backlink checker like Majestic, do a brand query in search engines or use a service like Similarweb to find the top reffering pages by traffic. It probably won't find them all, but it seems the only way to go.