1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Robots.txt and Indexing problem?!

Discussion in 'White Hat SEO' started by devbrah, Jan 12, 2012.

  1. devbrah

    devbrah Junior Member

    Joined:
    Oct 10, 2011
    Messages:
    110
    Likes Received:
    14
    Hi guys,

    So i've been trying to rank a MNS for the last month and a bit, and my webmaster tools has been set up the entire time.

    This is what my robots.txt looks like currently:

    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/

    As you can see, it's a wordpress site.

    My rankings have been dropped, and stayed down, and my impressions have dropped to <10 from an average of around 20/day.

    This I would attribute to not doing much with the site lately as i've been really busy, but when I look at my keywords, this is all it's giving me:


    1. disallow
    2. admin
    3. agent
    4. user

    What's going on here? It's like it's not even indexing any of my content at all.

    Can anyone provide any insight?

    Thanks!!
     
  2. orlandolongwood

    orlandolongwood Junior Member

    Joined:
    Aug 16, 2009
    Messages:
    137
    Likes Received:
    85
    Occupation:
    failed novelist
    Location:
    Austin, TX
    What is your URL?
     
  3. hounghang

    hounghang Newbie

    Joined:
    Dec 14, 2011
    Messages:
    15
    Likes Received:
    0
    User-agent: * pointing to the web browser web crawler robot

    Disallow: /wp-admin/
    Disallow: /wp-includes/


    in Disallow means web browser crawler robot will not crawl on that folder because that folder is important for wordpress if browser crawler robot will index it your website have the possibility to have


    Hope that makes sense
     
  4. istanbul

    istanbul Registered Member

    Joined:
    Aug 10, 2008
    Messages:
    73
    Likes Received:
    4
    interesting. look like robot.txt files cause problem for lots people on last days.
     
  5. devbrah

    devbrah Junior Member

    Joined:
    Oct 10, 2011
    Messages:
    110
    Likes Received:
    14
    I don't want to give out my URL as i'm new to ranking niche sites, and don't want someone swooping in and outranking me for my target keyphrases haha.

    Anyone have any ideas as to why this is happening?
     
  6. orlandolongwood

    orlandolongwood Junior Member

    Joined:
    Aug 16, 2009
    Messages:
    137
    Likes Received:
    85
    Occupation:
    failed novelist
    Location:
    Austin, TX
    Is your robots.txt file being delivered as a text file?
     
    Last edited: Jan 14, 2012
  7. Knoxgates

    Knoxgates Supreme Member

    Joined:
    Aug 9, 2008
    Messages:
    1,266
    Likes Received:
    918
    USe this Robots.txt example
    Code:
    Sitemap: http://www.yourwebsite.com/sitemap.xml
    User-agent: *
    Disallow: /wp-content/
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Disallow: /wp-
    Disallow: /feed/
    Disallow: /trackback/
    Disallow: /cgi-bin/
    User-agent: Googlebot
    Disallow: /*.php$
    Disallow: /*.js$
    Disallow: /*.cgi$
    Disallow: /*.xhtml$
    Disallow: /*.php*
    Disallow: */trackback*
    Disallow: /*?*
    Disallow: /z/
    Disallow: /wp-*
    Disallow: /*.inc$
    Disallow: /*.css$
    Disallow: /*.txt$
     
  8. LightSaviour

    LightSaviour Newbie

    Joined:
    Jan 14, 2012
    Messages:
    22
    Likes Received:
    1
    Location:
    Isreal
    What's the purpose of them?
     
  9. devbrah

    devbrah Junior Member

    Joined:
    Oct 10, 2011
    Messages:
    110
    Likes Received:
    14
    ... i'm not an expert on the subject, but this looks like it's disallowing pretty much every possible file extension I would want to use.

    Am I wrong?
     
  10. devbrah

    devbrah Junior Member

    Joined:
    Oct 10, 2011
    Messages:
    110
    Likes Received:
    14
    The purpose of a robots.txt file is to let search engines know what files they can and can't index.
     
  11. devbrah

    devbrah Junior Member

    Joined:
    Oct 10, 2011
    Messages:
    110
    Likes Received:
    14
    nobody can help me with this?
     
  12. devbrah

    devbrah Junior Member

    Joined:
    Oct 10, 2011
    Messages:
    110
    Likes Received:
    14
    Google webmaster tools tells me it's good.

    This is what's currently in there:

    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/

    Any ideas?

    I think this is hurting my ranks :(

    My first niche site isn't going as expected haha.