1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Google Wont Stop Crawling My Site !!!!

Discussion in 'White Hat SEO' started by dragonlube, Mar 16, 2011.

  1. dragonlube

    dragonlube Regular Member

    Joined:
    Jan 22, 2011
    Messages:
    420
    Likes Received:
    113
    Location:
    Behind the bushes
    Here is my problem, the site is not done, wordpress privacy is set to not allow spiders in, the source says <meta name='robots' content='noindex,nofollow' />
    so why in the hell did google bot crawl my site twice today and every day before that since I started working on it.

    thanks...I know, weired problem isnt, I just wish it does that when the sites ready.
     
  2. shadowedsniper

    shadowedsniper Junior Member

    Joined:
    Aug 15, 2008
    Messages:
    168
    Likes Received:
    100
    AHHH MAKE IT STOP!!!!!

    Lol just password protect the entire root directory if you're worried but I don't think it's a bad thing. At least you know there wont be any hesitation of being crawled when you actually want it.
     
  3. zalpriest

    zalpriest Regular Member

    Joined:
    Mar 10, 2011
    Messages:
    254
    Likes Received:
    51
    Occupation:
    SEO, Restaurant Owner
    Location:
    Beach,Florida
    Home Page:
    This ^ imo you should be happy.
     
  4. zalpriest

    zalpriest Regular Member

    Joined:
    Mar 10, 2011
    Messages:
    254
    Likes Received:
    51
    Occupation:
    SEO, Restaurant Owner
    Location:
    Beach,Florida
    Home Page:
    This ^ imo you should be happy.
     
  5. AdamMeds

    AdamMeds Newbie

    Joined:
    Nov 24, 2010
    Messages:
    16
    Likes Received:
    3
    Occupation:
    Affiliate Manager for MedsPartners
    Location:
    Cyprus
    Home Page:
    I wish I had your problems :D.
     
  6. KHalnay4k

    KHalnay4k Junior Member

    Joined:
    Mar 11, 2011
    Messages:
    167
    Likes Received:
    38
    Location:
    . .. -_- Owl City -_- .. .
    lOl Dude , its not any Kinda problem ;

    its A SEO Bliss for u :p

    enjoy it ;) ...

    I'll only recommend u to improve the content of your website because the all is Being done is Crawling ..
    So if your content would be good den after crawling google will show it on Searchengine

    its a best thing -- you'll get googlers to u ;)

    Improve the Content Quality so every search may have ur result ..
    Add every necessary Keyword ^_^

    ENJOY THIS BLISS MY MATE :p
     
  7. sagarpatil

    sagarpatil Regular Member

    Joined:
    Mar 2, 2009
    Messages:
    458
    Likes Received:
    278
    you need to make the changes in robots.txt file to make it stop crawling.
     
  8. dragonlube

    dragonlube Regular Member

    Joined:
    Jan 22, 2011
    Messages:
    420
    Likes Received:
    113
    Location:
    Behind the bushes
    sure, but there is nothing good about google seeing double and triple dup content on a bunch of pages. This is a 3 year old domain I had parked, it hasn't updated the titles for the indexed pages in the SERPs so maybe it just comes to chill and smoke a joint every now and then :)
     
  9. neo

    neo Power Member

    Joined:
    May 5, 2007
    Messages:
    500
    Likes Received:
    365
    use the robots.txt file to disallow googlebot.

    Googlebot will still crawl your site to check robots.txt, but won't go further (at least officially). But I would just have some place holder page to keep the bot happy so that when my site is ready getting the new pages indexed will be much faster.

    If you block google bot and get your site removed from index, it could take months before it can reappear.

    -Neo
     
  10. Theodore

    Theodore Power Member

    Joined:
    Oct 13, 2009
    Messages:
    679
    Likes Received:
    266
    Wish Google would crawl my site that oftern, im still waiting on them to update my meta tags... lol
     
  11. NapsteR

    NapsteR Jr. VIP Jr. VIP

    Joined:
    Mar 2, 2011
    Messages:
    2,779
    Likes Received:
    2,374
    Occupation:
    Full Time IMer
    Location:
    http://www.seophd.com