How do i stop bots going to my site
- Thread starter Julien1
- Start date
Skygrinder
Newbie
- Dec 10, 2009
- 11
- 1
Google "robots.txt"
cashking009
Junior Member
- Jul 31, 2009
- 188
- 41
add this meta tag to ur page
HTML:
<html>
<head>
<title>...</title>
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
</head>Are you talking abiut the legitinate SEO bots like GoogleBots, Yahoo Slurp etc, or are you refering to scraper bots that are just scraping all your content.
If the first, then the suggestions made above are great and will work.
If the second lot, then robots.txt instructions are just gonna get ignored.
The best way wouyld be to put some code on your site that checked for UserAgent (yes, i know its not perfect), and if there is no user agent specified then redirect them somewhere else.
I'm sure you can get rind if a lot of rubbish bots this way.
Hope this helps
If the first, then the suggestions made above are great and will work.
If the second lot, then robots.txt instructions are just gonna get ignored.
The best way wouyld be to put some code on your site that checked for UserAgent (yes, i know its not perfect), and if there is no user agent specified then redirect them somewhere else.
I'm sure you can get rind if a lot of rubbish bots this way.
Hope this helps
MrBlue
Senior Member
- Dec 18, 2009
- 975
- 695
robots.txt can be easily circumvented.
You would need to add something similar to your .htaccess file:
You would need to add something similar to your .htaccess file:
Code:
RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^.*(AdsBot-Google|ia_archiver|Scooter|Ask.Jeeves|Baiduspider|Exabot|FAST.Enterprise.Crawler|FAST-WebCrawler|www\.neomo\.de|Gigabot|Mediapartners-Google|Google.Desktop|Feedfetcher-Google|Googlebot|heise-IT-Markt-Crawler|heritrix|ibm.com\cs/crawler|ICCrawler|ichiro|MJ12bot|MetagerBot|msnbot-NewsBlogs|msnbot|msnbot-media|NG-Search|lucene.apache.org|NutchCVS|OmniExplorer_Bot|online.link.validator|psbot0|Seekbot|Sensis.Web.Crawler|SEO.search.Crawler|Seoma.\[SEO.Crawler\]|SEOsearch|Snappy|www.urltrends.com|www.tkl.iis.u-tokyo.ac.jp/~crawler|SynooBot|[email protected]|TurnitinBot|voyager|W3.SiteSearch.Crawler|W3C-checklink|W3C_Validator|www.WISEnutbot.com|yacybot|Yahoo-MMCrawler|Yahoo\!.DE.Slurp|Yahoo\!.Slurp|YahooSeeker).* [NC]
RewriteRule .* - [F]