1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Google's No-Robot List

Discussion in 'BlackHat Lounge' started by The Scarlet Pimp, Jan 24, 2009.

  1. The Scarlet Pimp

    The Scarlet Pimp Jr. VIP Jr. VIP Premium Member

    Joined:
    Apr 2, 2008
    Messages:
    788
    Likes Received:
    3,129
    Occupation:
    Chair moistener.
    Location:
    Cyberspace
    This is kinda interesting. It's the folders that G doesn't want spidered by
    other search engines...

    http://snurl.com/anb9v
     
    • Thanks Thanks x 1
  2. aмillionaírе

    aмillionaírе Jr. VIP Jr. VIP Premium Member

    Joined:
    Apr 20, 2008
    Messages:
    532
    Likes Received:
    358
    I was looking for such a thing for months, that synchronizes well :)
    This was used to help resolve the Proxy Caching issue it looks like, but why wouldn't Google want other search engines being messy? O_O

    * These are things that Google indexes but they're really not legit content, like keyword in domain that leads to a 404 but the page has a high PR rank, or a redirect that has a TITLE, example:

    Code:
    http://www.google.bs/search?client=firefox-a&rls=org.mozilla%3Aen-US%3Aofficial&channel=s&hl=en&q=allen+payne+nude&btnG=Google+Search
    This result is from a case study that outlays a Google exploit, I don't search for Allen Payne Nude :)
     
    Last edited: Jan 24, 2009