Crawl bots getting blocked

Billywedlock · Nov 19, 2020

I am far from expert, but having some error signs coming up with certain crawl bots. Am I right in thinking it is a cloudflare security issue ? (see below . )

When using the Rich test results checker, I also get an error message, though nothing shows up in search console. It says googlebot blocket by robots.txt , so strange it shows there but not in GSC.

Any ideas or comment welcomed. Thank you

Billywedlock · Nov 19, 2020

Billywedlock said:
I am far from expert, but having some error signs coming up with certain crawl bots. Am I right in thinking it is a cloudflare security issue ? (see below . )

When using the Rich test results checker, I also get an error message, though nothing shows up in search console. It says googlebot blocket by robots.txt , so strange it shows there but not in GSC.

Any ideas or comment welcomed. Thank you

Social God · Nov 19, 2020

Google Search Console has been having problems lately. I’d wait several weeks at least!

I think they’re still having an update. You can follow them on Twitter for information.

Most of my websites generally index within several days at least.

John-S · Nov 19, 2020

Google Search Console (GSC) has been having issues recently. I'd stand by a little while in any event!

I believe they're actually having an update. You can follow them on Twitter for data.

bbmaid · Nov 30, 2020

Have you checked your robots.txt file just to make sure you have not blocked off anything? If you are using WP, perhaps by mistake, have you checked the box that says disallow/discourage search engines under Genera Settings > Reading?

Crawl bots getting blocked

Billywedlock

Newbie

Billywedlock

Newbie

Attachments

Social God

Supreme Member

John-S

Registered Member

bbmaid

Newbie

Main Menu

Marketplace

Making Money

BlackHat World