weird Robots.txt issue

Bigdadddy said:

don't leave disallow blank instead add relative path of location of webpage you want not to be indexed.

for example in wordpress

User-agent: *
Disallow: /wp-admin/

hope you get the idea

Click to expand...

BrandonAntony said:

even with this in my robots file it still doesnt work:

User-agent: *
Allow: /wp-admin/admin-ajax.php
Allow: /*/*.css
Allow: /*/*.js
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /readme.html
Disallow: /license.txt
Disallow: /xmlrpc.php
Disallow: /wp-login.php
Disallow: /wp-register.php
Disallow: */disclaimer/*
Disallow: *?attachment_id=
Disallow: /privacy-policy

User-agent: Googlebot
Allow: /

User-agent: Googlebot-Image
Allow: /wp-content/uploads/

User-agent: Mediapartners-Google
Allow: /

User-agent: AdsBot-Google
Allow: /

User-agent: AdsBot-Google-Mobile
Allow: /

User-agent: Bingbot
Allow: /

User-agent: Msnbot
Allow: /

User-agent: msnbot-media
Allow: /wp-content/uploads/

User-agent: Applebot
Allow: /

User-agent: Yandex
Allow: /

User-agent: YandexImages
Allow: /wp-content/uploads/

User-agent: Slurp
Allow: /

User-agent: DuckDuckBot
Allow: /

User-agent: Qwantify
Allow: /

Click to expand...

Bigdadddy said:

Are you clearing the cache ?

Click to expand...

• .htaccess
• noindex tag presence on the page
• server cache - clear it so googlebot can visit updated version

tiiberius said:

There can be multiple robots.txt locations.

Some sites have one in root and the other in the /wp-content, for example (if using WordPress).

Are you sure the robots.txt you're working with is located in the root folder?

Also... maybe the error message is just generic and it's actually blocked by something else.

Check:

• .htaccess
• noindex tag presence on the page
• server cache - clear it so googlebot can visit updated version

Click to expand...

BrandonAntony said:

yes, even did a fresh install of wordpress and deleted all the databases and I still get the same thing

Click to expand...

BrandonAntony said:

thanks for the reply, the robots file was in the root folder. I did a full wipe of the site with a fresh install of wordpress and its still saying robots are being blocked. htaccess is blank basically, its a fresh install so no no index tage. Contacted bluehost and they are no help.

Click to expand...

BrandonAntony said:

thanks for the reply, the robots file was in the root folder. I did a full wipe of the site with a fresh install of wordpress and its still saying robots are being blocked. htaccess is blank basically, its a fresh install so no no index tage. Contacted bluehost and they are no help.

Click to expand...

tiiberius said:

There are multiple .htaccess files as well. One is root and one is WP folder.

Looks like that all that remains is outdated cache.

If not, it seems like some king of unique bug.

Click to expand...

BrandonAntony said:

Thanks for the help, it seems to have sorted itself out. I think google might not have crawled the robots file and I just had to wait

Click to expand...

tiiberius said:

Glad you fixed it.

Probably cache then.

But they messed up Analytics recently so maybe this one too

Click to expand...

ThisIsMyLife said:

Googlebot treats Disallow here in your case as wild card. So, Just give the right directives where they can obey. And in WordPress case just type this
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Create one in your directory to overwrite this one, clear cache and request index again.

Click to expand...

BrandonAntony said:

I think I fixed it, for anyone else wondering I just changed my site url from https:// to http:// and then back to https:// and now it looks like I can get google to crawl the page.

Click to expand...