How do I make my domain "uncrawlable"?

Ming the Merciless

Ming the Merciless

Power Member
Joined
Sep 30, 2012
Messages
706
Reaction score
847
I need some help. How can I make my domain "uncrawlable"? I am doing something that is "non-SEO" with it, so I want to get rid of those extra clicks caused by Google crawler bots. Basically it is just a domain that is being forwarded straight to an offer, so there is no website involved. Is there a way to get this accomplished?
 
Advertise on BHW
Code: 
RewriteEngine On

RewriteCond %{HTTP_USER_AGENT} (googlebot|bingbot|Baiduspider) [NC]
RewriteRule .* - [R=403,L]

Put that into .htaccess of the specific folder you don't want crawled. Robots.txt can be useless, .htaccess is what will keep Google from crawling your site. Robots.txt is basically just a way of saying what you want, with .htaccess you order & get what you want.
 
In short, robots.txt can be ignored but not the same thing happens with .htaccess.
 
You could always tell webmaster tools to remove the site altogether if it's only Google you are bothered about ?
 
Thank you for all the answers. Basically, I am only forwarding the domain (there is no hosting involved), so I really do not see anywhere to make folders, etc. The domain is with godaddy if that helps.
 
If it's not hosted search engines don't know it exists so it won't be crawled anyway.
Just re-direct through your domain registrar cPanel and forget about it ;-)
 
Kaistar said:
If it's not hosted search engines don't know it exists so it won't be crawled anyway.
Just re-direct through your domain registrar cPanel and forget about it ;-)
Click to expand...

Well, thats strange. Because this is a brand new domain and it is only being forwarded and it has been crawled multiple times already from the big G. So, no offense, but I think that you are wrong.

The only reason I care is because I am trying to get an accurate gauge on clicks with this particular campaign I am running and the G bots are screwing up the count.
 
Ming the Merciless said:
Well, thats strange. Because this is a brand new domain and it is only being forwarded and it has been crawled multiple times already from the big G. So, no offense, but I think that you are wrong.
Click to expand...

If your domain is sat at domain registry level with no nameservers modified and no website for it to display there is nothing to crawl.

If you are getting hits on this brand new domain that suggests it was recently dropped by someone else and is still indexed so still getting crawled.

Have a look into the domain using ahrefs and site:domain.com and see if there are any results.
 
Kaistar said:
If your domain is sat at domain registry level with no nameservers modified and no website for it to display there is nothing to crawl.

If you are getting hits on this brand new domain that suggests it was recently dropped by someone else and is still indexed so still getting crawled.

Have a look into the domain using ahrefs and site:domain.com and see if there are any results.
Click to expand...

I checked ahref like you said. No results came up.

This is a brand new domain that has never been registered before.

The only thing I am doing is forwarding it to an offer landing page (not hosted by me).

I guess as a last resort I can just make a landing page of my own on my host and add "robots.txt" to a folder. I was just hoping for something simpler.

Thanks for the response.
 
Ming the Merciless said:
Thank you for all the answers. Basically, I am only forwarding the domain (there is no hosting involved), so I really do not see anywhere to make folders, etc. The domain is with godaddy if that helps.
Click to expand...

Well, there is no way to use .htaccess without hosting.
You can do the following:

domain without hosting -> domain with hosting(use .htaccess)->your offer.
Not sure if the middle domain will affect you in any way.
 
I don't use GoDaddy myself (too many horror stories) so unfamiliar with how they work.
Is there a part of their site that maybe automatically posts something like "Newly Registered" just as a cheap gimmick that Google may have slurped up ?

Seriously though, if your domain has never been used before and never been hosted by a server there is nothing to crawl. Something, somewhere must have displayed your url publicly.

Strange one that...
 
It happens often to me - brand new domains, NEVER registered before and no backlinks, get indexed at google. When i am working on a website i just put a temporary "under construction" page.
My last domain got crawled by http://mawords.com/ and google crawled mawords. I have no idea what mawords is.
 
theMagicNumber said:
It happens often to me - brand new domains, NEVER registered before and no backlinks, get indexed at google. When i am working on a website i just put a temporary "under construction" page.
Click to expand...

This gives Google something to crawl.
By putting up a page you have created a website, even if it's just the 1 single construction page and search engines will automatically find it, eventually.
Unless you add a noindex meta tag to your header.

If you buy a domain and never do anything but redirect it straight from your domain registrar then search engines simply cannot see it.

Until of course, you start to post the domain all over the internet. Then they see the link from another site and follow it.

The only way I can see of stopping this is to add a 'noindex' tag each time you post the url but this is impracticable if you are say posting on Facebook and forums etc.

You're going to have to create a landing page and stop that from being crawled. Noindex meta tags will be needed and use the above .htaccess and robots.txt as well.

If you are already getting hits then it's already on the radar so I would recommend adding your domain to GWT and removing the url from it's index.
 
Kaistar said:
This gives Google something to crawl.
By putting up a page you have created a website, even if it's just the 1 single construction page and search engines will automatically find it, eventually.
Unless you add a noindex meta tag to your header.

If you buy a domain and never do anything but redirect it straight from your domain registrar then search engines simply cannot see it.

Until of course, you start to post the domain all over the internet. Then they see the link from another site and follow it.

The only way I can see of stopping this is to add a 'noindex' tag each time you post the url but this is impracticable if you are say posting on Facebook and forums etc.

You're going to have to create a landing page and stop that from being crawled. Noindex meta tags will be needed and use the above .htaccess and robots.txt as well.

If you are already getting hits then it's already on the radar so I would recommend adding your domain to GWT and removing the url from it's index.
Click to expand...

Thanks again for all the help Kaistar.
 
Kaistar said:
This gives Google something to crawl.
By putting up a page you have created a website, even if it's just the 1 single construction page and search engines will automatically find it, eventually.
Click to expand...

You are missing the point. No one knows the existence of my domain, but the registrars - putting a simple page on it doesn't change that fact.
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top