Robots.txt

S

Seljo

Regular Member

Jul 3, 2013

384

137

• Sep 23, 2014

• #1

My problem is the following:

I had a domain setup with robots.txt to not index .

Now I have made a new WP install on this domain and deleted the robots.txt, the funny thing is that when I type the url/robots.txt it still finds it ?

I noticed this when I registered my site with gwt and it won't index my site no matter what I do.

I would be more than happy, if someone has the solution

 

Wilson Grant Fisk

Elite Member

Nov 10, 2012

15,677

46,840

• Sep 23, 2014

• #2

Have you checked the directory to make sure it is deleted?
Are you using any caching plugin or CloudFlare?
Have you checked your WP privacy settings?
http://codex.wordpress.org/Settings_Reading_Screen
Are you using an SEO Plugin?

 

S

Seljo

Regular Member

Jul 3, 2013

384

137

• Sep 23, 2014
• Thread Starter Thread Starter

• #3

I use SEO plugin YOAST I deleted it there too, but it still shows up.
I have CDN disabled it and retryed same sotry.

Looking at the privacy settings now.

 

S

Seljo

Regular Member

Jul 3, 2013

384

137

• Sep 23, 2014
• Thread Starter Thread Starter

• #4

Privacy settings are disabled.

 

X

xrfanatic

BANNED

Aug 28, 2010

466

201

• Sep 23, 2014

• #5

I haven't experienced anything like that regarding deleting files , but it happens on some cloud based hostings. For example when you change the contents of the file it usually takes some time to see changes. In my case it takes usually around 4-8 hours. I hope the post was helpful.

 

S

Seljo

Regular Member

Jul 3, 2013

384

137

• Sep 23, 2014
• Thread Starter Thread Starter

• #6

Well It isn't. I tried and uploaded a new robots.txt with modified values and it show the right one. But when I delete it again, the old one shows up even if nothing is there. Its giving me headaches

 

Wilson Grant Fisk

Elite Member

Nov 10, 2012

15,677

46,840

• Sep 23, 2014

• #7

Use Cpanel or FTP into the server and manually delete the file.

 

S

Seljo

Regular Member

Jul 3, 2013

384

137

• Sep 23, 2014
• Thread Starter Thread Starter

• #8

Done that many times, It does not help

 

Wilson Grant Fisk

Elite Member

Nov 10, 2012

15,677

46,840

• Sep 23, 2014

• #9

Seljo said:

Done that many times, It does not help

Click to expand...

Which CDN are you using?

 

S

Seljo

Regular Member

Jul 3, 2013

384

137

• Sep 23, 2014
• Thread Starter Thread Starter

• #10

Cloudflare the Free version from hostgator.

Currently I disabled it and I also disabled wpsupercache. Probably I need to wait a day or so it gets clean ?

 

Wilson Grant Fisk

Elite Member

Nov 10, 2012

15,677

46,840

• Sep 23, 2014

• #11

Seljo said:

Cloudflare the Free version from hostgator.

Currently I disabled it and I also disabled wpsupercache. Probably I need to wait a day or so it gets clean ?

Click to expand...

Sign into Cloudflare and Purge the cache.

https://www.cloudflare.com/

https://www.cloudflare.com/cloudflare-settings?


Then use CCleaner or clear your browser cache too.

 

S

Seljo

Regular Member

Jul 3, 2013

384

137

• Sep 23, 2014
• Thread Starter Thread Starter

• #12

Ok Enabled Cloudflare again and purged the god damn thing. Still nothing happened but I will wait a few hours and check again.