I am building a site that i do not want indexed but besides a robot.txt, I want to block spiders and being indexed in anyway from anyone as best as possible.
Is there a way to display a blank page to these spiders or some other type of method besides your standard disallow robots.txt?
Also can password protected pages get indexed as well? The whole site is going to be password protected not sure if that makes a difference.
Is there a way to display a blank page to these spiders or some other type of method besides your standard disallow robots.txt?
Also can password protected pages get indexed as well? The whole site is going to be password protected not sure if that makes a difference.