New Cloudflare tool checks for 'AI Agent' support

Cloudflare have released a new tool for those interested in all the AI buzzwords like 'MCP' and 'Markdown negotiation': https://isitagentready.com/

It checks multiple categories, such as discoverability and bot access, for those wanting to make their sites as easy as possible for AI Agents to digest and work with, and gives pointers on how to improve things. Personally, as someone who doesn't even bother with sitemaps 99% of the time, this just all looks like busy work to me. And looking at this graph, most sites out there aren't bothered either:



Get ready to see loads of 'AI Agent Discoverability' services cropping up.

The interesting part for scrapers is the bot access section — it essentially audits whether your robots.txt and Cloudflare firewall rules are blocking recognized AI crawlers (GPTBot, ClaudeBot, etc.) vs. unrecognized ones. The irony is that sites trying to be "agent-ready" are loosening exactly the bot access controls that stop generic scrapers. The llms.txt spec is useful though: if a target site publishes one, it's a clean structured map of their content hierarchy — saves you from having to reverse-engineer site structure manually.

vierasen said:

The interesting part for scrapers is the bot access section — it essentially audits whether your robots.txt and Cloudflare firewall rules are blocking recognized AI crawlers (GPTBot, ClaudeBot, etc.) vs. unrecognized ones. The irony is that sites trying to be "agent-ready" are loosening exactly the bot access controls that stop generic scrapers. The llms.txt spec is useful though: if a target site publishes one, it's a clean structured map of their content hierarchy — saves you from having to reverse-engineer site structure manually.

Click to expand...

Yeah, I'm not sure if I should be recommending people use llms.txt just for the fact that more widespread use might be beneficial for my own scraping efforts in the future

That's the honest dilemma and you're right to see it. llms.txt adoption is a long-tail bet though — most sites publishing it are publishers actively trying to be scraped by AI platforms for brand visibility, so they're already the low-friction targets. The high-value scraping targets (marketplaces, aggregators, competitive intel) will never publish one. So recommending it publicly costs you almost nothing on the sites that matter — the real moat is still in ASN-aware proxies and TLS fingerprinting, not the map file.