Robots.txt file - what should I disallow on HTML/CSS website?

RoyCooper

Junior Member
Joined
Mar 2, 2017
Messages
108
Reaction score
7
I'm not using WordPress, just HTML/CSS on my website.

What should I be disallowing for security? Right now I'm just allowing everything and anything...
 
I meant security as in getting rid of any association with websites/bots that might link to your website and give it negative SEO.
So basically how to get rid of spam bots with robots.txt? Which are the bad ones?
 
Well you should disallow those login pages, like login.php, etc. Pages that you do not think should appear on the search engine should be disallowed. Most likely administrator pages - those shouldn't appear. And sometimes those php backend pages (coding) also get indexed - you can add those to the disallow list.
 
Back
Top