1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Noindex tag but over 800 000 indexed

Discussion in 'Black Hat SEO' started by aidenhera, Jul 14, 2017.

  1. aidenhera

    aidenhera Elite Member

    Joined:
    Nov 30, 2016
    Messages:
    1,817
    Likes Received:
    388
    Gender:
    Male
    Im automating some profiles and whats shocking me is that some of them are noindex.. and whats even better there are a lot of indexed pages...

    url is: https://www.goodreads.com/user/show/
    and query is site:https://www.goodreads.com/user/show/

    [​IMG]

    [​IMG]

    Other example is site:https://www.change.org/p/

    What does it mean? Lets say owners implemented the noindex tag later, shouldnt the pages dissapear? im automating some profiles and Im not sure if I will be able to force index of these or not lol


    edit

    even better thing is that some of sites with noindex tag were indexed in last month ! from these domains. Its very unlikely the metatag was added in last month. Probably google is ignoring that meta tag for some domains.
     
    • Thanks Thanks x 1
  2. gman777

    gman777 Jr. VIP Jr. VIP

    Joined:
    Apr 7, 2016
    Messages:
    670
    Likes Received:
    545
    Yeah, good question man. That's what I'm wondering too. Somebody from here is selling profile links from sites with no index. Perhaps META no index doesn't enforce the no index, but only "discourage" the search engines.
     
  3. aidenhera

    aidenhera Elite Member

    Joined:
    Nov 30, 2016
    Messages:
    1,817
    Likes Received:
    388
    Gender:
    Male
    actually thats my fail - these profiles dont have no index meta tag, but when I make one theres noindex tag. In both cases goodreads and change sites. They must block profiles and unblock certain ones, but I dont know whats the requirement yet.
     
  4. LatteGrande

    LatteGrande Jr. VIP Jr. VIP Premium Member

    Joined:
    Jan 19, 2011
    Messages:
    2,191
    Likes Received:
    609
    Location:
    404 Not Found
    Maybe because the robots.txt file is blocking specific URLs from Google web crawlers, so they can't see the tag.
     
    • Thanks Thanks x 1
  5. gman777

    gman777 Jr. VIP Jr. VIP

    Joined:
    Apr 7, 2016
    Messages:
    670
    Likes Received:
    545
    There are other websites with profile link indexed even with noindex tag. For example this one:
    https://www.edutopia.org/users/andyblack52

    noindex.png

    Yes man. That's probably the reason.
     
  6. aidenhera

    aidenhera Elite Member

    Joined:
    Nov 30, 2016
    Messages:
    1,817
    Likes Received:
    388
    Gender:
    Male
    agree. but if you search for site:https://www.edutopia.org/users/
    then it will reveal profiles without noindex tag. (however same folder, url etc) it must be something in account settings.



    actually i made my profile on goodreads without noindex tag - it was private because age wasnt filled so they wanted to defend me from google crawlers...
     
  7. gman777

    gman777 Jr. VIP Jr. VIP

    Joined:
    Apr 7, 2016
    Messages:
    670
    Likes Received:
    545
    That's a generic 404 error. Has nothing to do with /users/ folder.
     
  8. aidenhera

    aidenhera Elite Member

    Joined:
    Nov 30, 2016
    Messages:
    1,817
    Likes Received:
    388
    Gender:
    Male
    I mean site you linked has the same properties indexed that do not have noindex tag.

    [​IMG]
     
  9. askary

    askary Regular Member

    Joined:
    Jan 6, 2015
    Messages:
    364
    Likes Received:
    75
    how did you check? there are no robots meta tags
    the same with change.org/p/
     
  10. askary

    askary Regular Member

    Joined:
    Jan 6, 2015
    Messages:
    364
    Likes Received:
    75
    )))))so how they indexed if they are blocked with robots.txt?
     
  11. LatteGrande

    LatteGrande Jr. VIP Jr. VIP Premium Member

    Joined:
    Jan 19, 2011
    Messages:
    2,191
    Likes Received:
    609
    Location:
    404 Not Found
    If Google can't see the noindex tag then they indexed the URLs.