1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Is My Robotstxt for WP correct? "Disalllow" once or twice

Discussion in 'White Hat SEO' started by freckletone, May 18, 2015.

  1. freckletone

    freckletone Jr. VIP Jr. VIP

    Joined:
    Jul 22, 2009
    Messages:
    500
    Likes Received:
    82
    Does this look right? I wanted to block some of the backlink checker sites out there?
    Just want to make sure I'm not blocking Googlebot etc... Does this look right?


    User-agent: *
    Disallow: /wp-admin/

    User-agent: SemrushBot
    User-agent: linkdexbot/2.0
    User-agent: MJ12bot
    User-agent: SemrushBot-SA
    Disallow: /
     
  2. seeplusplus

    seeplusplus Power Member

    Joined:
    Aug 18, 2008
    Messages:
    517
    Likes Received:
    164
    Yes your blocking ALL crawlers (if they respect the robots directive) from the admin, and just those listed from the entire site.
     
    • Thanks Thanks x 1
  3. freckletone

    freckletone Jr. VIP Jr. VIP

    Joined:
    Jul 22, 2009
    Messages:
    500
    Likes Received:
    82
    Thanks, just wanted to make sure!
     
  4. SirKonstantine

    SirKonstantine Junior Member

    Joined:
    Dec 4, 2011
    Messages:
    150
    Likes Received:
    75
    Yup, looks good. GWM and Bing WMT as well as Yandex WMT all have robots.txt checkers so you can verify if their robots can access a part of the site.