1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Google won't index my sites because of crawl errors?

Discussion in 'Black Hat SEO' started by kuryakin, Mar 7, 2013.

  1. kuryakin

    kuryakin Regular Member

    Joined:
    Jun 26, 2012
    Messages:
    216
    Likes Received:
    47
    Hi people, hope someone can advise me, I have been doing my first bit of SEO for 2 wordpress sites and it was going ok but i am not ranked at all and i see on webmaster tools i have the following errors:DNS
    Server connectivity
    Robots.txt fetch

    The First 2 sound like issues with my hosting maybe? But the robots.txt file IS accessible- It looks like this: User-agent: *
    Disallow: /wp-admin/ Disallow: /wp-includesAnd if i use 'fetch as google' the site is accessible.. Does anyone know what might be going on? This is really annoying as i've been working hard on these sites and now i cant get them ranked! Thanks....
     
  2. horselditty

    horselditty Newbie

    Joined:
    Feb 23, 2012
    Messages:
    12
    Likes Received:
    11
    Occupation:
    this
    Location:
    Straya
    did you use a plugin for the robots.txt or sitemap creation?
     
  3. kuryakin

    kuryakin Regular Member

    Joined:
    Jun 26, 2012
    Messages:
    216
    Likes Received:
    47
    No, the robots.txt is the default one with wordpress. I didn't actually create the sitemap, i bought the sites and they were already there.. Why would that be a problem? Cheers
     
  4. gsy159

    gsy159 Power Member

    Joined:
    Apr 29, 2011
    Messages:
    655
    Likes Received:
    158
    go to google webmaster tools add your site there and put as sitemap site.com/rss if you dont want to create a xml file.
    You can track there if its indexed or not.
     
  5. SpookSEO

    SpookSEO Senior Member

    Joined:
    Dec 15, 2012
    Messages:
    848
    Likes Received:
    180
    Occupation:
    Linkbuilder
    Location:
    London, UK
    Home Page:
    When did Google last try to access robots.txt? Are you sure you've given it enough time to retry since setting up the site?
     
  6. kuryakin

    kuryakin Regular Member

    Joined:
    Jun 26, 2012
    Messages:
    216
    Likes Received:
    47
    Thanks folks, i submitted a sitemap- hopefully that will help. And Google last tried to access robots.txt the day before yesterday after everything was set up.. still didn't work.

    Does anyone know what the DNS and ServerConnectivity errors could mean? Could they all be linked to the robots.txt issue?
     
  7. gsy159

    gsy159 Power Member

    Joined:
    Apr 29, 2011
    Messages:
    655
    Likes Received:
    158
    shoot me your site url per pm I will tell you what the dns problem means
     
  8. innozemec

    innozemec Jr. VIP Jr. VIP

    Joined:
    Aug 19, 2011
    Messages:
    5,288
    Likes Received:
    1,799
    Location:
    www.Indexification.com
    Home Page:
    ServerConnectivity refers to problems in the connection with your server, has nothing to do with the robots.txt

    you sure your hosting is up all the time?