Robots.txt

My problem is the following:

I had a domain setup with robots.txt to not index .

Now I have made a new WP install on this domain and deleted the robots.txt, the funny thing is that when I type the url/robots.txt it still finds it ?

I noticed this when I registered my site with gwt and it won't index my site no matter what I do.

I would be more than happy, if someone has the solution

 

Have you checked the directory to make sure it is deleted?
Are you using any caching plugin or CloudFlare?
Have you checked your WP privacy settings?
http://codex.wordpress.org/Settings_Reading_Screen
Are you using an SEO Plugin?

 

I use SEO plugin YOAST I deleted it there too, but it still shows up.
I have CDN disabled it and retryed same sotry.

Looking at the privacy settings now.

 

Privacy settings are disabled.

 

I haven't experienced anything like that regarding deleting files , but it happens on some cloud based hostings. For example when you change the contents of the file it usually takes some time to see changes. In my case it takes usually around 4-8 hours. I hope the post was helpful.

 

Well It isn't. I tried and uploaded a new robots.txt with modified values and it show the right one. But when I delete it again, the old one shows up even if nothing is there. Its giving me headaches

 

Use Cpanel or FTP into the server and manually delete the file.

 

Done that many times, It does not help

 

Which CDN are you using?

 

Cloudflare the Free version from hostgator.

Currently I disabled it and I also disabled wpsupercache. Probably I need to wait a day or so it gets clean ?

 

Sign into Cloudflare and Purge the cache.

https://www.cloudflare.com/



https://www.cloudflare.com/cloudflare-settings?


Then use CCleaner or clear your browser cache too.

 

Ok Enabled Cloudflare again and purged the god damn thing. Still nothing happened but I will wait a few hours and check again.