Block all bots and crawlers?

kippen · Jan 18, 2024

I want to use a domain as a redirect to my money site. I don’t want any bots/crawlers following this redirect (google, facebook, private bots that don’t adhere to robots.txt) I don’t care about SEO and indexing, I’m only interested in human traffic that I have already targeted getting through this redirect.

What can I do to achieve this? Is Cloudflare Turnstile any help?

Peter Ngo · Jan 18, 2024

is there any content on the domain? Or you just doing a 301 redirect? Is it being hosted?

You generally can do that via .htacess if you are using a LAMP stack, you will need to haev mod_rewrite enabled on your server.

tazarbm · Jan 19, 2024

add this into your robots.txt file (although I'm not sure that it blocks google, google can still choose to ignore the rules in robots.txt):

Code:

User-Agent: *
Disallow: /cgi-bin
Disallow: /wp-
Disallow: /wp-includes
Disallow: /wp-admin/
Disallow: /wp-content/plugins
Disallow: /?s=
Disallow: *&s=
Disallow: /search
Disallow: /author/
Disallow: *?attachment_id=
Disallow: */feed
Disallow: */rss
Disallow: */embed
Allow: /wp-content/uploads/
Allow: /wp-content/themes/
Allow: /*/*.js
Allow: /*/*.css
Allow: /wp-*.png
Allow: /wp-*.jpg
Allow: /wp-*.jpeg
Allow: /wp-*.gif
Allow: /wp-*.svg
Allow: /wp-*.pdf

Block all bots and crawlers?

kippen

Registered Member

Peter Ngo

Elite Member

tazarbm

Elite Member

Main Menu

Marketplace

Making Money

BlackHat World