1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

[Question] Finding Noindex URLS

Discussion in 'Black Hat SEO' started by ghackseo, Sep 30, 2015.

  1. ghackseo

    ghackseo Regular Member

    Joined:
    Mar 18, 2015
    Messages:
    283
    Likes Received:
    72
    Hi guys, I need a pro to figure this out or advise if it's even possible.

    Basically I'm trying to find all the url's on a domain that's not indexed within Search Engines. (I don't own the domain.)

    I have the root domain obviously and one other inner-page url. Obviously a "site:domain.com" search wont work as nothing is indexed.
     
  2. mindmaster

    mindmaster Jr. VIP Jr. VIP

    Joined:
    Sep 16, 2010
    Messages:
    2,876
    Likes Received:
    1,306
    Home Page:
    The allmighty scrapebox will save your ass.
    First try and scrape using the site/inurl operators. Use all search engines.
    Then use the Internal addon scraper. If those not indexed urls have links on the website, scrapebox will find them.
    Also sitemap scraper addon might work.

    Let us know how it goes.
     
  3. ghackseo

    ghackseo Regular Member

    Joined:
    Mar 18, 2015
    Messages:
    283
    Likes Received:
    72
    Thanks for the response, however, there are zero links to each of these pages from the other pages and neither is there a sitemap, so scraping them isn't really possible.

    I'll need some sort of solution to see exactly what's on the server. Anyone?
     
  4. rogerke

    rogerke Regular Member

    Joined:
    Oct 5, 2014
    Messages:
    264
    Likes Received:
    145
    So, how do you know the domain has noindex tag pages?
     
  5. ghackseo

    ghackseo Regular Member

    Joined:
    Mar 18, 2015
    Messages:
    283
    Likes Received:
    72
    That's besides the question, but anyway, I know because I've followed a few links in e-mails (it's a sort of "underground" newsletter), but I haven't been part of it for very long and I'm 99% sure there has to be a lot of other pages that I've missed.
     
  6. rogerke

    rogerke Regular Member

    Joined:
    Oct 5, 2014
    Messages:
    264
    Likes Received:
    145
    It is a relevant question, because the same method you used to find a few links might be scaled up. Since it seem to be newsletters you're going after you might try to find some common footprints in the URL structure (like newsletter-07-07.html, newsletter07-08.html) and find them all. Alternatively if the newsletters have backlinks you might use a backlink checker like Majestic, do a brand query in search engines or use a service like Similarweb to find the top reffering pages by traffic. It probably won't find them all, but it seems the only way to go.