1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Can Google crawl a page behind login wall

Discussion in 'White Hat SEO' started by kontentguy, Aug 4, 2017.

  1. kontentguy

    kontentguy Regular Member UnGagged Attendee

    Joined:
    Jun 10, 2014
    Messages:
    397
    Likes Received:
    130
    Gender:
    Male
    Occupation:
    White hat agency
    I was trained that Google can't crawl a page that has a log in.

    However i have a site that has 2 homepages:
    1) standard page that you dont need to be logged in to view
    2) customized homepage that you only see when logged in

    I noticed that the homepage is ranking for keywords that only appear on the homepage when a user is logged in.

    How is this possible?

    (To be fair, the keywords its ranking for are in the same niche as the non-logged in homepage, obviously).

    But is google now crawling logged in pages? How is that possible.
     
  2. frostbolt

    frostbolt Newbie

    Joined:
    Jun 3, 2011
    Messages:
    21
    Likes Received:
    5
    Gender:
    Male
    Location:
    Michigan
    Google chrome can do what it wants read the TOS along with android platform no information is safe from the google!
     
  3. largesiteguy

    largesiteguy Newbie

    Joined:
    Nov 6, 2016
    Messages:
    32
    Likes Received:
    2
    do you see content from your protected area indexed in google?
     
  4. kontentguy

    kontentguy Regular Member UnGagged Attendee

    Joined:
    Jun 10, 2014
    Messages:
    397
    Likes Received:
    130
    Gender:
    Male
    Occupation:
    White hat agency
    good q -

    the only protected content is the headers
    and they appear in google search results but for other pages on the site...
    hmm
     
  5. KHer0

    KHer0 Supreme Member

    Joined:
    Mar 22, 2011
    Messages:
    1,230
    Likes Received:
    1,160
    Occupation:
    Architect
    Pretty easy to solve, here is what you gonna do :-

    1 - Search in google for "site:yoursitename.com" without quotes of course.
    2 - It will show all indexed pages in your website
    3 - If you are hiding pages, they shouldn't appear in the results
    4 - If you are hiding paragraphs and parts of content while indexing the url, then click the small arrow at the end of the url (in google results page) to choose " see google cache".
    5 - Now you can see how google see your content and if the hidden parts are there.
    6 - In both situation if you content is there, then you have a coding problem or folder permissions problem
    7 - If your hidden content isn't there, then stop worrying about how google ranks your website :D
     
    • Thanks Thanks x 1
  6. davids355

    davids355 Jr. VIP Jr. VIP

    Joined:
    Apr 25, 2011
    Messages:
    9,833
    Likes Received:
    7,443
    Home Page:
    Just because the keyword doesn't appear on a page, it can still rank for it.
     
  7. kickthat

    kickthat Jr. VIP Jr. VIP

    Joined:
    Sep 18, 2014
    Messages:
    354
    Likes Received:
    404
    Gender:
    Male
    Location:
    UK
    How is the logged in content displayed?
     
  8. andy1

    andy1 Junior Member

    Joined:
    Jan 15, 2009
    Messages:
    104
    Likes Received:
    18
    yeah search in google as site:domain.com to see if its their.
     
  9. kontentguy

    kontentguy Regular Member UnGagged Attendee

    Joined:
    Jun 10, 2014
    Messages:
    397
    Likes Received:
    130
    Gender:
    Male
    Occupation:
    White hat agency
    YES

    ty
     
  10. I know SEO

    I know SEO Marketplace Mod Moderator

    Joined:
    Nov 29, 2012
    Messages:
    15,724
    Likes Received:
    6,113
    What version of the page can you see in the Google cache for the URL?
     
  11. kickthat

    kickthat Jr. VIP Jr. VIP

    Joined:
    Sep 18, 2014
    Messages:
    354
    Likes Received:
    404
    Gender:
    Male
    Location:
    UK
    @kontentguy

    I asked this and it didn't get a response
    It's probably not this simple, but just to rule out the obvious...

    I presume you are not using css or JavaScript to just hide content before a user logs in, right? Just because if the content is there in the source code, but hidden to a user, then of course a crawler can pick it up.
     
  12. redarrow

    redarrow Elite Member

    Joined:
    Apr 1, 2013
    Messages:
    4,302
    Likes Received:
    983
    yes there no hiding they own the server
    and a lot of illegal websites get caught and prison this way ...

    the same as hiding your domain reg if it needed can be still gotten

    private policy get chuck in the bin if big boys afther u.
     
  13. redarrow

    redarrow Elite Member

    Joined:
    Apr 1, 2013
    Messages:
    4,302
    Likes Received:
    983
    you can tell google to not index a page with metatags or htaccess but google can still see it but not serve it public.
     
  14. kontentguy

    kontentguy Regular Member UnGagged Attendee

    Joined:
    Jun 10, 2014
    Messages:
    397
    Likes Received:
    130
    Gender:
    Male
    Occupation:
    White hat agency
    the non logged in version

    it was picking up <h2>'s that were only displayed behind login wall
    but i realize in retrospect that these headers exist elsewhere in the site

    i was panicking bc i was pitching a new client and some clown on their team was asking me why didnt i address non-logged in version of their site, and im like, bc google cant see it so its not relevant to us at this point etc. then he showed me the loggedin h2's were indeed showing in serps but its simply bc it was from other, non-loggedin pages, which took me a bit to realize, in my panic-induced stressed state of mind.

    anyways, all good here now
    i got the client :)
     
  15. Michelle91

    Michelle91 Newbie

    Joined:
    Aug 7, 2017
    Messages:
    18
    Likes Received:
    1
    Gender:
    Female
    Home Page:
    As far as I know it can't. Otherwise we should be seeing indexed wp-admin and stuff like that in Google. It all depends on what you have added in your robots.txt file.
     
  16. kontentguy

    kontentguy Regular Member UnGagged Attendee

    Joined:
    Jun 10, 2014
    Messages:
    397
    Likes Received:
    130
    Gender:
    Male
    Occupation:
    White hat agency
    it also depends if i even had a chance to check robots haha
     
  17. aidenhera

    aidenhera Elite Member

    Joined:
    Nov 30, 2016
    Messages:
    1,605
    Likes Received:
    279
    Gender:
    Male
    its NOT possible. if that pages rank for that kw thats because of milion other factors. Lets not make 10 pages thread about a private forum section ranking probably because of how the CMS was constructed. (that could produce some snippets or code with link (on other inner pages), that could get indexed by G).

    However, forums (with active users) have infinite internal power... I wonder how much these 9999999 threads with a link to "homepage" in the top bar influence ranking of homepage in G. probably very very very very significantly


    (assuming you wanna rank for certain PC amazon afilate, you create board about computers, let these geeks create threads, and rank homepage for hard kw due to internal power)
     
  18. roadhamster

    roadhamster Regular Member

    Joined:
    Mar 12, 2012
    Messages:
    335
    Likes Received:
    242
    Depends on the platform you use, for instance, there are a lot of forums out there using software that allows the googlebot to crawl their forums, but non-registered or non-logged-in users can't.
    How to check: if you have Firefox or Chrome, install a user-agent switcher add-on, go to your website, clear the cookies and your cache, change the user-agent of your browser to Googlebot'useragent and check if you can access your website. If you can then also Googlebot can index your pages. If you can't, big chance that Google isn't able to crawl pages behind your login.
     
    • Thanks Thanks x 1
  19. timothyc3r

    timothyc3r Jr. VIP Jr. VIP

    Joined:
    Apr 24, 2017
    Messages:
    222
    Likes Received:
    17
    Gender:
    Male
    Occupation:
    SEO Specialist
    Location:
    Philippines
    Google doesn't log in nor will any other search engine. If you put content behind a wall, form, etc, then you're blocking access. However, Google offers a service called First Click Free for Web. This may well solve your problem but only applies to Google.