1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Robots.txt file - what should I disallow on HTML/CSS website?

Discussion in 'White Hat SEO' started by RoyCooper, Jul 19, 2017.

  1. RoyCooper

    RoyCooper Registered Member

    Joined:
    Mar 2, 2017
    Messages:
    76
    Likes Received:
    4
    I'm not using WordPress, just HTML/CSS on my website.

    What should I be disallowing for security? Right now I'm just allowing everything and anything...
     
  2. RoyCooper

    RoyCooper Registered Member

    Joined:
    Mar 2, 2017
    Messages:
    76
    Likes Received:
    4
    I meant security as in getting rid of any association with websites/bots that might link to your website and give it negative SEO.
    So basically how to get rid of spam bots with robots.txt? Which are the bad ones?
     
  3. CookServers

    CookServers Junior Member

    Joined:
    Mar 15, 2017
    Messages:
    147
    Likes Received:
    14
    Well you should disallow those login pages, like login.php, etc. Pages that you do not think should appear on the search engine should be disallowed. Most likely administrator pages - those shouldn't appear. And sometimes those php backend pages (coding) also get indexed - you can add those to the disallow list.