1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Robots.txt

Discussion in 'Web Design' started by Seljo, Sep 23, 2014.

  1. Seljo

    Seljo Regular Member

    Joined:
    Jul 3, 2013
    Messages:
    384
    Likes Received:
    137
    My problem is the following:

    I had a domain setup with robots.txt to not index .

    Now I have made a new WP install on this domain and deleted the robots.txt, the funny thing is that when I type the url/robots.txt it still finds it ?

    I noticed this when I registered my site with gwt and it won't index my site no matter what I do.

    I would be more than happy, if someone has the solution :)
     
  2. Asif WILSON Khan

    Asif WILSON Khan Executive VIP Premium Member

    Joined:
    Nov 10, 2012
    Messages:
    10,125
    Likes Received:
    28,564
    Gender:
    Male
    Occupation:
    Fun Lovin' Criminal
    Location:
    London
    Home Page:
  3. Seljo

    Seljo Regular Member

    Joined:
    Jul 3, 2013
    Messages:
    384
    Likes Received:
    137
    I use SEO plugin YOAST I deleted it there too, but it still shows up.
    I have CDN disabled it and retryed same sotry.

    Looking at the privacy settings now.
     
  4. Seljo

    Seljo Regular Member

    Joined:
    Jul 3, 2013
    Messages:
    384
    Likes Received:
    137
    Privacy settings are disabled.
     
  5. xrfanatic

    xrfanatic Jr. VIP Jr. VIP Premium Member

    Joined:
    Aug 28, 2010
    Messages:
    371
    Likes Received:
    166
    Location:
    http://bit.ly/slb64
    Home Page:
    I haven't experienced anything like that regarding deleting files , but it happens on some cloud based hostings. For example when you change the contents of the file it usually takes some time to see changes. In my case it takes usually around 4-8 hours. I hope the post was helpful. :)
     
  6. Seljo

    Seljo Regular Member

    Joined:
    Jul 3, 2013
    Messages:
    384
    Likes Received:
    137
    Well It isn't. I tried and uploaded a new robots.txt with modified values and it show the right one. But when I delete it again, the old one shows up even if nothing is there. Its giving me headaches :(
     
  7. Asif WILSON Khan

    Asif WILSON Khan Executive VIP Premium Member

    Joined:
    Nov 10, 2012
    Messages:
    10,125
    Likes Received:
    28,564
    Gender:
    Male
    Occupation:
    Fun Lovin' Criminal
    Location:
    London
    Home Page:
    Use Cpanel or FTP into the server and manually delete the file.
     
  8. Seljo

    Seljo Regular Member

    Joined:
    Jul 3, 2013
    Messages:
    384
    Likes Received:
    137
    Done that many times, It does not help :(
     
  9. Asif WILSON Khan

    Asif WILSON Khan Executive VIP Premium Member

    Joined:
    Nov 10, 2012
    Messages:
    10,125
    Likes Received:
    28,564
    Gender:
    Male
    Occupation:
    Fun Lovin' Criminal
    Location:
    London
    Home Page:
    Which CDN are you using?
     
  10. Seljo

    Seljo Regular Member

    Joined:
    Jul 3, 2013
    Messages:
    384
    Likes Received:
    137
    Cloudflare the Free version from hostgator.

    Currently I disabled it and I also disabled wpsupercache. Probably I need to wait a day or so it gets clean ?
     
  11. Asif WILSON Khan

    Asif WILSON Khan Executive VIP Premium Member

    Joined:
    Nov 10, 2012
    Messages:
    10,125
    Likes Received:
    28,564
    Gender:
    Male
    Occupation:
    Fun Lovin' Criminal
    Location:
    London
    Home Page:
  12. Seljo

    Seljo Regular Member

    Joined:
    Jul 3, 2013
    Messages:
    384
    Likes Received:
    137
    Ok Enabled Cloudflare again and purged the god damn thing. Still nothing happened but I will wait a few hours and check again.