Directory & Pages for AI Bots ONLY - Will this Hurt SEO / Google Ranking?

john_doe_

Junior Member
Joined
Feb 22, 2022
Messages
161
Reaction score
35
As Google stated, if webpage display different versions for user & crawler, might be penalized, "cloaking", but how about if we serve a folder and all subpages to certain AI bots (include Google AI Overview) crawlers only and deny all content to actual user?
For example:
site: sample.com
existing folders: /product/a, /product/b, /blog/content1, /blog/content2, and etc.,
and we've earned certain result of higher position and DA of the site, product, content pages,

Now, if we create a folder names /product-specs/, and create a lot of content on the basis of fact, maybe product a is usually for group A, but not we separate them into different dimensions for "Fan-out":
/product-specs/a-for-user-in-german,
/product-specs/a-for-user-in-poland,
/product-specs/a-for-user-in-german-kids

will this be a problem and impact our ranking in Google SERP and get penalty by Google?

Please advise.
 
Yes, that could definitely become risky if search engine crawlers see content that normal users cannot access. Google may treat that as cloaking, especially if the pages exist mainly for AI crawlers or ranking manipulation. Creating many thin fan out pages with very similar content can also trigger duplicate or low-quality content issues. It’s usually safer to create useful location or audience specific pages that real users can actually visit and benefit from.
 
Yes, that could definitely become risky if search engine crawlers see content that normal users cannot access. Google may treat that as cloaking, especially if the pages exist mainly for AI crawlers or ranking manipulation. Creating many thin fan out pages with very similar content can also trigger duplicate or low-quality content issues. It’s usually safer to create useful location or audience specific pages that real users can actually visit and benefit from.

Yeah, that kind of setup is pretty risky and generally not aligned with how search engines expect content to be served.

From what I’ve seen, anything that creates a mismatch between what users see and what crawlers see can eventually lead to issues. Same goes for large-scale “template” or doorway-style pages that don’t really offer much unique value.

It usually works better to focus on pages that are actually useful for real users instead of trying to scale purely through structure. When the intent and content are genuinely helpful, things tend to hold up better long term.

Thanks for the replies, I do respect and appreciate, I have same knowledge with you guys, but why I wanted to ask here is:
  1. It's said that if a web page displays "Video" to user while gives "text" to Google, is considered as cloaking as well - but in the actual case, I've never seen any penalty, on the contrary, they called "optimize for crawler's efficient" and the websites get a lot indexed and ranking improvements after this "optimization";
  2. If the directory is set to allow certain 4 AI crawlers' crawling, how come Google search bots know the existence of the directory and sub-pages since we've disallow all other type visits?
    1. if Google bots (except AI) has no idea of the existence, how the website get penalty?
Kindly discuss.
Thanks.
 
will this be a problem and impact our ranking in Google SERP and get penalty by Google?
a limited use for 20-30 pages is okay. considering your site has 1K to 2K pages.
overdoing it might trigger spam/cloaking.
 
Back
Top