Google Wont Stop Crawling My Site !!!!

dragonlube

Elite Member

Jr. VIP

Jan 22, 2011

3,691

781

• Mar 16, 2011

• #1

Here is my problem, the site is not done, wordpress privacy is set to not allow spiders in, the source says <meta name='robots' content='noindex,nofollow' />
so why in the hell did google bot crawl my site twice today and every day before that since I started working on it.

thanks...I know, weired problem isnt, I just wish it does that when the sites ready.

 

shadowedsniper

Junior Member

Aug 15, 2008

167

102

• Mar 16, 2011

• #2

AHHH MAKE IT STOP!!!!!

Lol just password protect the entire root directory if you're worried but I don't think it's a bad thing. At least you know there wont be any hesitation of being crawled when you actually want it.

 

zalpriest

Regular Member

Mar 10, 2011

254

53

• Mar 16, 2011

• #3

This ^ imo you should be happy.

 

zalpriest

Regular Member

Mar 10, 2011

254

53

• Mar 16, 2011

• #4

This ^ imo you should be happy.

 

A

AdamMeds

Newbie

Nov 24, 2010

16

3

• Mar 16, 2011

• #5

I wish I had your problems .

 

KHalnay4k

Junior Member

Mar 11, 2011

167

38

• Mar 16, 2011

• #6

lOl Dude , its not any Kinda problem ;

its A SEO Bliss for u

enjoy it ...

I'll only recommend u to improve the content of your website because the all is Being done is Crawling ..
So if your content would be good den after crawling google will show it on Searchengine

its a best thing -- you'll get googlers to u

Improve the Content Quality so every search may have ur result ..
Add every necessary Keyword ^_^

ENJOY THIS BLISS MY MATE

 

S

sagarpatil

Power Member

Mar 2, 2009

547

332

• Mar 16, 2011

• #7

you need to make the changes in robots.txt file to make it stop crawling.

 

dragonlube

Elite Member

Jr. VIP

Jan 22, 2011

3,691

781

• Mar 16, 2011
• Thread Starter Thread Starter

• #8

sure, but there is nothing good about google seeing double and triple dup content on a bunch of pages. This is a 3 year old domain I had parked, it hasn't updated the titles for the indexed pages in the SERPs so maybe it just comes to chill and smoke a joint every now and then

 

neo

Power Member

May 5, 2007

792

672

• Mar 16, 2011

• #9

use the robots.txt file to disallow googlebot.

Googlebot will still crawl your site to check robots.txt, but won't go further (at least officially). But I would just have some place holder page to keep the bot happy so that when my site is ready getting the new pages indexed will be much faster.

If you block google bot and get your site removed from index, it could take months before it can reappear.

-Neo

 

Theodore

Power Member

Oct 13, 2009

683

268

• Mar 16, 2011

• #10

Wish Google would crawl my site that oftern, im still waiting on them to update my meta tags... lol

 

NapsteR

Elite Member

Mar 2, 2011

3,027

2,428

• Mar 17, 2011

• #11

Modify your robots.txt file. I hide some of folders by this file only...