1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Changing robots.txt

Discussion in 'Black Hat SEO' started by Iceman1978, Aug 15, 2014.

  1. Iceman1978

    Iceman1978 Registered Member

    Joined:
    Nov 15, 2013
    Messages:
    68
    Likes Received:
    4
    Location:
    Kent,England
    My adsense account says one of my sites is having a crawler error with a robot denied and a page not found error. I have(I think) fixed the page not found error but I am bamboozled by the robots.txt one. Being Google,there explanation and "remedy" was to talk in circles and not actually explain very much!
    Can anyone tell me how to fix the robots.txt?
     
  2. nethious

    nethious Newbie

    Joined:
    Oct 25, 2011
    Messages:
    31
    Likes Received:
    7
    does it have the correct permissions?
     
  3. Iceman1978

    Iceman1978 Registered Member

    Joined:
    Nov 15, 2013
    Messages:
    68
    Likes Received:
    4
    Location:
    Kent,England
    What do you mean by correct permission?
     
  4. Iceman1978

    Iceman1978 Registered Member

    Joined:
    Nov 15, 2013
    Messages:
    68
    Likes Received:
    4
    Location:
    Kent,England
    bumpety bump bump
     
  5. campo

    campo Newbie

    Joined:
    Aug 1, 2014
    Messages:
    49
    Likes Received:
    4
    Location:
    74 68 65 20 77 65 62
    depends on what page you have.
    When you look at the robots.txt, have a look for any disallow statements and remove them unless you know they should be there.
    cant post a link or even what to google :s
    Disallow should only be for directories you dont want google to index such as admin, shopping carts, my account and so on....
     
    • Thanks Thanks x 1