1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

googlebot crawl/index/cache

Discussion in 'White Hat SEO' started by captchadreams, May 4, 2012.

  1. captchadreams

    captchadreams Power Member

    Joined:
    Sep 19, 2008
    Messages:
    504
    Likes Received:
    131
    I have a question. I got into an argument about the googlebot and what it can actually index and can't.

    another person said googlebot could still crawl and index private pages on websites and indexes them.

    to prove my point I posted a google cache which is basically just a screenshot of when the googlebot crawled that specific page and the cache said "only members can view this page". since googlebot isn't a member of that site.. it can't view all the threads in otherwords.

    he said that googlebot could still crawl and index all the private pages it just didn't put them in their cache.. he knew this because he's had private sites where googlebot crawled.

    I don't know as much about SEO, as I get my traffic from blackhatting social networks.. so I could be wrong.. if I am please explain how this works to me.

    my understanding is basically if you can't view a page, thread, profile, whatever when you're logged out.. google isn't going to be able to see it either.

    thanks.
     
    Last edited: May 4, 2012
  2. milian

    milian Power Member

    Joined:
    Jan 9, 2010
    Messages:
    515
    Likes Received:
    69
    Occupation:
    SEO stuff
    Location:
    Around
    • Thanks Thanks x 1
  3. jessica236

    jessica236 Newbie

    Joined:
    Apr 30, 2012
    Messages:
    24
    Likes Received:
    1
    Occupation:
    SEO[search engine optimization]
    Obviously google can't crawl a login require page,only public pages get crawl and indexed by googlebot .

    "Google Site Search can't provide search results for pages that require passwords."
     
    • Thanks Thanks x 1
  4. monk13

    monk13 Power Member

    Joined:
    Jul 10, 2009
    Messages:
    547
    Likes Received:
    268
    oh ffs. seriously, this is what happens when people that haven't figured out how to use google webmaster tools try to do seo

    http://support.google.com/webmasters/bin/answer.py?hl=en&answer=158587

    use the fetch as googlebot function and see what googlebot sees in realtime
    protip: use fetch as googlebot to force google to index a page right then and there
     
    • Thanks Thanks x 1
  5. captchadreams

    captchadreams Power Member

    Joined:
    Sep 19, 2008
    Messages:
    504
    Likes Received:
    131
    I thought it was pretty self explanatory as well. but wanted to make 100% sure.. so thanks for responses.