I blocked Google and it still indexed me?

ShadeDream · Mar 20, 2010

I've bought a domain (as an example, hello.com) on which I have set up a wordpress blog. By default I have decided to block Google and all other search engines until I complete my site. While working on something else I have decided to make around 5 backlinks for hello.com while it was still blocking search engines. Today, I googled the domain without the extension and to my surprise it's indexed as "www.hello.com/". Is it because of the backlinks? I mean, it must be because how else would it be indexed, but how is this possible?

ronmojohny · Mar 20, 2010

Google ignores the robots.txt file sometimes. I think you can verify ownership in Google Webmaster tools, and then ask them to remove it from the index. (from within webmaster tools)

ShadeDream · Mar 20, 2010

I actually don't want to remove it since I will need it indexed anyway. I was just wondering why this happens.

nufaman · Mar 20, 2010

How exactly were you blocking it?

195471 · Mar 21, 2010

ShadeDream said:
I actually don't want to remove it since I will need it indexed anyway. I was just wondering why this happens.

There's a difference between spidering and indexing. If you disallow a bot in your robots.txt, that doesn't mean that it will prevent your pages from getting indexed. If you really don't want a page to be indexed, the best thing to do, imho, is to add a meta tag that blocks indexing:

Code:

<meta name="robots" content="noindex" />

You need to add it to every page that you don't want indexed.

Edit: some reading material...

Code:

http://sebastians-pamphlets.com/crawling-vs-indexing/
http://sebastians-pamphlets.com/smart-robots-txt/

ShadeDream · Mar 21, 2010

nufaman said:
How exactly were you blocking it?

robots.txt

But as 195471 said, I have just found this on Google:

Removing your entire website using a robots.txt file
Print

You can use a robots.txt file to request that search engines remove your site and prevent robots from crawling it in the future. (It's important to note that if a robot discovers your site by other means - for example, by following a link to your URL from another site - your content may still appear in our index and our search results. To entirely prevent a page from being added to the Google index even if other sites link to it, http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=93710.)

I blocked Google and it still indexed me?

ShadeDream

BANNED

ronmojohny

Regular Member

ShadeDream

BANNED

nufaman

Elite Member

195471

Regular Member

ShadeDream

BANNED

Main Menu

Marketplace

Making Money

BlackHat World