1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Robot txt template for wordpress

Discussion in 'White Hat SEO' started by JayKay123, Sep 7, 2016.

  1. JayKay123

    JayKay123 Newbie

    Joined:
    Mar 8, 2016
    Messages:
    13
    Likes Received:
    1
    Robot txt template for wordpressHi all,
    I have been advise that my previous robot text was rubbish, yet most of the information was taken from the websites regarding the best seo practises for wordpress. I would like to know on your thoughts of the best template robot txt disallows / allows / no index for wordpress websites?
    Thank you
     
  2. ekapek

    ekapek Jr. VIP Jr. VIP Premium Member

    Joined:
    Aug 2, 2010
    Messages:
    277
    Likes Received:
    48
    Home Page:
    It depends really on yours site. For most wordpress sites all site is set for indexing (without wp-admin). Sometimes it is good to block tags/pagination/category but it can be done with plugins like All in Seo etc
     
  3. JayKay123

    JayKay123 Newbie

    Joined:
    Mar 8, 2016
    Messages:
    13
    Likes Received:
    1
    Thanks, I really looking for the best setup for seo, everyone says its important to have robot txt.
     
  4. Vik Pathania

    Vik Pathania Regular Member

    Joined:
    Oct 13, 2014
    Messages:
    273
    Likes Received:
    54
    Gender:
    Male
    Occupation:
    Digital Marketing Specialist
    Location:
    New Delhi
    Here is the syntex:

    User-Agent: *
    2 Allow: /wp-content/uploads/
    3 Disallow: /wp-content/plugins/
    4 Disallow: /readme.html


    But, I would like to tell you that now there is no need to block access to your wp-includes directory and your plugins directory or wp-admin. if you block any URL using robots.txt then crawler will not crawl that website and also will now distribute link value pointing at those pages. If you don’t do anything, WordPress has (by my doing) a robots meta x-http header on the admin pages that prevents search engines from showing these pages in the search results..

    For more information you can check https://yoast.com/wordpress-robots-txt-example/.