Blocking of ai crawlers, is really blocking our own AEO visibility?

[Anos_King]

With more users relying on ai regenerated answers, I've noticed most of the sites using cloudflare and similar tools to control ai crawlers.

But.. ai search system rely on web content to discover and retrieve information for this answer, could blocking these crawlers actually limit our own AEO visibility?

I need to know whether selectively allowing search related crawler blocking unwanted ai scraping becoming the better approach for AEO

 

[BlackShield]

With more users relying on ai regenerated answers, I've noticed most of the sites using cloudflare and similar tools to control ai crawlers.

But.. ai search system rely on web content to discover and retrieve information for this answer, could blocking these crawlers actually limit our own AEO visibility?

I need to know whether selectively allowing search related crawler blocking unwanted ai scraping becoming the better approach for AEO

Yes, potentially. If you block AI crawlers that an AI search engine uses to discover and retrieve your content, you're reducing its ability to cite or surface your pages in AI answers.

But I wouldn't blindly allow every AI bot. I'd separate them by purpose:

AI search/retrieval crawlers: generally allow if AEO visibility is a priority.
AI training crawlers: decide separately based on your content/licensing strategy.
Unknown/suspicious bots: block or rate-limit where appropriate.

 

[devenconway]

Yeah, definitely. I would allow the crawling bots for those search engines that you actually need while blocking other bots who are clearly there just to scrape your website.

 

[medialounge]

I am sure you know what you want to achieve and as consequence, make use of those Ai bots that are relevant to your purpose. Those can even help you in rankin and authority. Anything else which doesn't suit your purpose you block

 

[alarichaye]

With more users relying on ai regenerated answers, I've noticed most of the sites using cloudflare and similar tools to control ai crawlers.

But.. ai search system rely on web content to discover and retrieve information for this answer, could blocking these crawlers actually limit our own AEO visibility?

I need to know whether selectively allowing search related crawler blocking unwanted ai scraping becoming the better approach for AEO

i am relatively new to AEO but this question was bugging me as well. blocking all of the crawlers might not be good because if ur conttent is being used for searching from such crawlers then it might become difficult. are there any selective approaches in blocking crawlers?

 

[MajestySEO]

yes, selectively blocking them is the right approach. I do allow AI search bots, it covers all main AI bots like ChatGPT, Claude, Gemini, and Perplexity, and block training bots.

In cloudflare, you can set this up under Security - WAF - Custom Rules. create rules that look at the bot’s user agent and then either allow or block it.

 

[Nedd]

Yes, blocking AI crawlers may result in a lower volume of content being discovered by AI search engines, so it's important to consider both privacy and exposure.

 

[Thenos SEO]

I’d avoid blocking all AI crawlers without checking what each one does. A selective approach could reduce unwanted scraping while keeping useful discovery channels open for AI search.

 

[Monkey D. Luffy]

Blocking all AI crawlers can hurt AEO, better to allow the useful ones and block the rest.