Is My Robotstxt for WP correct? "Disalllow" once or twice

Discussion in 'White Hat SEO' started by freckletone, May 18, 2015.

  1. freckletone

    freckletone Jr. VIP Jr. VIP

    Joined:
    Jul 22, 2009
    Messages:
    565
    Likes Received:
    86
    Does this look right? I wanted to block some of the backlink checker sites out there?
    Just want to make sure I'm not blocking Googlebot etc... Does this look right?


    User-agent: *
    Disallow: /wp-admin/

    User-agent: SemrushBot
    User-agent: linkdexbot/2.0
    User-agent: MJ12bot
    User-agent: SemrushBot-SA
    Disallow: /
     
  2. seeplusplus

    seeplusplus Power Member

    Joined:
    Aug 18, 2008
    Messages:
    520
    Likes Received:
    167
    Yes your blocking ALL crawlers (if they respect the robots directive) from the admin, and just those listed from the entire site.
     
    • Thanks Thanks x 1
  3. freckletone

    freckletone Jr. VIP Jr. VIP

    Joined:
    Jul 22, 2009
    Messages:
    565
    Likes Received:
    86
    Thanks, just wanted to make sure!
     
  4. SirKonstantine

    SirKonstantine Junior Member

    Joined:
    Dec 4, 2011
    Messages:
    150
    Likes Received:
    76
    Yup, looks good. GWM and Bing WMT as well as Yandex WMT all have robots.txt checkers so you can verify if their robots can access a part of the site.