1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

How do I make my domain "uncrawlable"?

Discussion in 'Black Hat SEO' started by Ming the Merciless, Jun 28, 2014.

  1. Ming the Merciless

    Ming the Merciless Jr. VIP Jr. VIP Premium Member

    Joined:
    Sep 30, 2012
    Messages:
    680
    Likes Received:
    820
    Location:
    United States
    I need some help. How can I make my domain "uncrawlable"? I am doing something that is "non-SEO" with it, so I want to get rid of those extra clicks caused by Google crawler bots. Basically it is just a domain that is being forwarded straight to an offer, so there is no website involved. Is there a way to get this accomplished?
     
  2. SEOShine

    SEOShine Registered Member

    Joined:
    Jul 12, 2013
    Messages:
    62
    Likes Received:
    5
    Create a robots.txt in the directory where your index.xx file and write this in:
     
  3. Zso

    Zso BANNED BANNED

    Joined:
    Nov 18, 2007
    Messages:
    497
    Likes Received:
    880
    Code:
    RewriteEngine On
    
    RewriteCond %{HTTP_USER_AGENT} (googlebot|bingbot|Baiduspider) [NC]
    RewriteRule .* - [R=403,L]
    Put that into .htaccess of the specific folder you don't want crawled. Robots.txt can be useless, .htaccess is what will keep Google from crawling your site. Robots.txt is basically just a way of saying what you want, with .htaccess you order & get what you want.
     
  4. Zso

    Zso BANNED BANNED

    Joined:
    Nov 18, 2007
    Messages:
    497
    Likes Received:
    880
    In short, robots.txt can be ignored but not the same thing happens with .htaccess.
     
  5. Kaistar

    Kaistar Power Member

    Joined:
    Jan 9, 2010
    Messages:
    562
    Likes Received:
    333
    You could always tell webmaster tools to remove the site altogether if it's only Google you are bothered about ?
     
  6. Ming the Merciless

    Ming the Merciless Jr. VIP Jr. VIP Premium Member

    Joined:
    Sep 30, 2012
    Messages:
    680
    Likes Received:
    820
    Location:
    United States
    Thank you for all the answers. Basically, I am only forwarding the domain (there is no hosting involved), so I really do not see anywhere to make folders, etc. The domain is with godaddy if that helps.
     
  7. Kaistar

    Kaistar Power Member

    Joined:
    Jan 9, 2010
    Messages:
    562
    Likes Received:
    333
    If it's not hosted search engines don't know it exists so it won't be crawled anyway.
    Just re-direct through your domain registrar cPanel and forget about it ;-)
     
  8. ImBored

    ImBored Newbie

    Joined:
    Jun 28, 2014
    Messages:
    9
    Likes Received:
    1
    Create a robots.txt in the directory and done ;P
     
  9. Ming the Merciless

    Ming the Merciless Jr. VIP Jr. VIP Premium Member

    Joined:
    Sep 30, 2012
    Messages:
    680
    Likes Received:
    820
    Location:
    United States
    Well, thats strange. Because this is a brand new domain and it is only being forwarded and it has been crawled multiple times already from the big G. So, no offense, but I think that you are wrong.

    The only reason I care is because I am trying to get an accurate gauge on clicks with this particular campaign I am running and the G bots are screwing up the count.
     
  10. Numbuh362

    Numbuh362 Elite Member

    Joined:
    Aug 22, 2012
    Messages:
    1,569
    Likes Received:
    462
    robots.txt and block all bots #problemsolved
     
  11. Kaistar

    Kaistar Power Member

    Joined:
    Jan 9, 2010
    Messages:
    562
    Likes Received:
    333
    If your domain is sat at domain registry level with no nameservers modified and no website for it to display there is nothing to crawl.

    If you are getting hits on this brand new domain that suggests it was recently dropped by someone else and is still indexed so still getting crawled.

    Have a look into the domain using ahrefs and site:domain.com and see if there are any results.
     
  12. Ming the Merciless

    Ming the Merciless Jr. VIP Jr. VIP Premium Member

    Joined:
    Sep 30, 2012
    Messages:
    680
    Likes Received:
    820
    Location:
    United States
    I checked ahref like you said. No results came up.

    This is a brand new domain that has never been registered before.

    The only thing I am doing is forwarding it to an offer landing page (not hosted by me).

    I guess as a last resort I can just make a landing page of my own on my host and add "robots.txt" to a folder. I was just hoping for something simpler.

    Thanks for the response.
     
  13. theMagicNumber

    theMagicNumber Regular Member

    Joined:
    May 13, 2010
    Messages:
    345
    Likes Received:
    195
    Well, there is no way to use .htaccess without hosting.
    You can do the following:

    domain without hosting -> domain with hosting(use .htaccess)->your offer.
    Not sure if the middle domain will affect you in any way.
     
  14. Kaistar

    Kaistar Power Member

    Joined:
    Jan 9, 2010
    Messages:
    562
    Likes Received:
    333
    I don't use GoDaddy myself (too many horror stories) so unfamiliar with how they work.
    Is there a part of their site that maybe automatically posts something like "Newly Registered" just as a cheap gimmick that Google may have slurped up ?

    Seriously though, if your domain has never been used before and never been hosted by a server there is nothing to crawl. Something, somewhere must have displayed your url publicly.

    Strange one that...
     
  15. theMagicNumber

    theMagicNumber Regular Member

    Joined:
    May 13, 2010
    Messages:
    345
    Likes Received:
    195
    It happens often to me - brand new domains, NEVER registered before and no backlinks, get indexed at google. When i am working on a website i just put a temporary "under construction" page.
    My last domain got crawled by http://mawords.com/ and google crawled mawords. I have no idea what mawords is.
     
  16. Kaistar

    Kaistar Power Member

    Joined:
    Jan 9, 2010
    Messages:
    562
    Likes Received:
    333
    This gives Google something to crawl.
    By putting up a page you have created a website, even if it's just the 1 single construction page and search engines will automatically find it, eventually.
    Unless you add a noindex meta tag to your header.

    If you buy a domain and never do anything but redirect it straight from your domain registrar then search engines simply cannot see it.

    Until of course, you start to post the domain all over the internet. Then they see the link from another site and follow it.

    The only way I can see of stopping this is to add a 'noindex' tag each time you post the url but this is impracticable if you are say posting on Facebook and forums etc.

    You're going to have to create a landing page and stop that from being crawled. Noindex meta tags will be needed and use the above .htaccess and robots.txt as well.

    If you are already getting hits then it's already on the radar so I would recommend adding your domain to GWT and removing the url from it's index.
     
    • Thanks Thanks x 1
  17. Ming the Merciless

    Ming the Merciless Jr. VIP Jr. VIP Premium Member

    Joined:
    Sep 30, 2012
    Messages:
    680
    Likes Received:
    820
    Location:
    United States
    Thanks again for all the help Kaistar.
     
  18. lucasbraydon

    lucasbraydon Supreme Member

    Joined:
    Nov 9, 2013
    Messages:
    1,231
    Likes Received:
    1,561
    Location:
    Here and there
    OP have you been following my journey or something :p
     
  19. Ming the Merciless

    Ming the Merciless Jr. VIP Jr. VIP Premium Member

    Joined:
    Sep 30, 2012
    Messages:
    680
    Likes Received:
    820
    Location:
    United States
    lol. I have, actually. Very inspirational... I must say! Looking forward to 07/07......

    those damn googlebots!
     
    • Thanks Thanks x 1
  20. theMagicNumber

    theMagicNumber Regular Member

    Joined:
    May 13, 2010
    Messages:
    345
    Likes Received:
    195
    You are missing the point. No one knows the existence of my domain, but the registrars - putting a simple page on it doesn't change that fact.