1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Robots.txt file contents showing as meta description...?

Discussion in 'Black Hat SEO' started by Thesiege84, Jul 29, 2013.

  1. Thesiege84

    Thesiege84 Regular Member

    Joined:
    May 5, 2009
    Messages:
    360
    Likes Received:
    97
    Occupation:
    Money Churner
    Location:
    Churning Money
    Hey everyone,

    Hope your online ventures are going well.

    Ive had something very strange happen to one of our sites and i really don't know why and was wondering if anyone else has had this happen and if so did you find out what was causing it?

    An inner page of one of my authority sites is showing the robots.txt file contents as the meta description so its showing this in Google:

    User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ Sitemap: http://www.*****************.***/sitemap.xml.gz

    I have no idea why its doing this, there isnt even a robots.txt file on the server but if you go to domain.com/robots.txt one loads. I think this is because wordpress dynamically creates it if you dont have one but when i uploaded my own it replaces it.

    We have all in one seo plugin installed and manually create meta titles and descriptions that show 90% of the time for this page.

    So this description has been coming and going randomly for a couple of months and when it does come back our rankings drop a page or two.

    Has anyone seen this before??
     
  2. glew

    glew Junior Member

    Joined:
    Feb 10, 2008
    Messages:
    141
    Likes Received:
    93
  3. Thesiege84

    Thesiege84 Regular Member

    Joined:
    May 5, 2009
    Messages:
    360
    Likes Received:
    97
    Occupation:
    Money Churner
    Location:
    Churning Money
    Thanks for the quick response but this isnt really whats happening.

    The page that it is showing on isnt actually the robots.txt file. Its an inner page with content but its just using the robots.txt file contents as the meta description.
     
  4. GreyKnight

    GreyKnight Regular Member

    Joined:
    Mar 19, 2013
    Messages:
    399
    Likes Received:
    200
    Home Page:
    Strange. Never seen like this before.
    Try looking up in your Google Webmaster Tools, to see if there is error there in your robots.txt.
    Or remove your robots.txt for a while, and see what happens.
     
  5. Thesiege84

    Thesiege84 Regular Member

    Joined:
    May 5, 2009
    Messages:
    360
    Likes Received:
    97
    Occupation:
    Money Churner
    Location:
    Churning Money
    Thanks for the reply.

    I checked in WMT and it says there is no problem with it, i also tested it against the page it was displaying on.

    If i remove the robots.txt file wordpress dynamically creates one so you can see it on domain.com/robots.txt but there is no robots.txt file in server root.

    If i put a robots.txt file in root it shows that one instead, so i guess i could create a blank one but it still wouldnt answer why this is happening.

    I also noticed in WMT that our category page has the same meta description as the article so it was pulling a duplicate meta description warning in WMT.

    So ive just removed it from the category page.... maybe this was causing it?..... probably not cos this duplicate was also happening with the other categories (if no description present it seems to grab first post in cat description) but they are fine in serps.