1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Latest Posts not indexed! Check my Robot.txt Plz

Discussion in 'White Hat SEO' started by kaizersuzi, Nov 4, 2009.

  1. kaizersuzi

    kaizersuzi Junior Member

    Joined:
    Dec 31, 2008
    Messages:
    107
    Likes Received:
    132
    Location:
    Next Door2U
    Hi all,

    I'm using wp for my PE site. I did some changes to my robot.txt below and noticed that my latest articles were never indexed after a few days (normally all my posts used to take an hour, few hours at most).

    I'm not sure if there is something in my robot that caused this issue.

    Please have a look at my robot and let me know if you found something.
    BTW, my metatags read like this:

    Tite: post title
    Description: post description
    Keywords: post keywords
    Robot: noarchive

    Here is my robot.txt:
    Code:
    User-agent: *
    Disallow:  /cgi-bin
    Disallow: /wp-admin
    Disallow: /wp-includes
    Disallow: /wp-content/plugins
    Disallow: /wp-content/cache
    Disallow: /wp-content/themes
    Disallow: /trackback
    Disallow: /feed
    Disallow: */trackback
    Disallow: */feed
    Disallow: /author
    Disallow: /search/*/feed
    Disallow: /search/*/*
    Disallow: /privacy-policy/
    Disallow: /about/
    Disallow: /disclaimer/
    Disallow: /contact-us/
    Disallow: /images/
    Disallow:/uncategorized/
    Disallow:/submit-your-site/
    Disallow:/tag/
    Disallow:/out/
    Disallow:/category/
    Disallow:/penisenlargement/penis-enlargement/extender/sizegenetics/
    Sitemap:http://www.example.com/sitemap.xml
    
    User-agent: Googlebot-Image
    Allow: /
    
    User-agent: Googlebot-Mobile
    Allow: /
    
    User-agent: ia_archiver-web.archive.org
    Disallow: /
    
    Thanks in advance for your help.