Filter Page Indexing Issue

Hello Everyone,

I’m encountering a difficult issue with some filtered URLs on my site being indexed in Google.

URL's Pattern -

• example.com/category-name/colour/Dark%20Grey
• example.com/category-name/trousers/size/9/10
• example.com/category-name/gender/size/6/9
• example.com/category-name/gender/size/XL/SF
• example.com/category-name/gender/trousers/size/XXS/LL
• example.com/category-name/gender/trousers/size/XXXL/SF
• example.com/gender/category-name/size/13/14
• example.com/gender/category-name/size/XXS/SL
• example.com/gender/category-name/size/XXS/XS
• example.com/gender/size/9/12
• example.com/gender/size/M/L
• example.com/gender/category-name/colour/Light%20Purple

I’ve taken the following steps to block them:

Steps Implemented:

1. Blocked in robots.txt:

Example rule:

• Disallow: */size/
• Disallow: */colour/

These rules should prevent crawling of these filtered URLs.

2. Meta Robots Tag:

Pages are served with <meta name="robots" content="noindex, nofollow"/> to ensure they are excluded from the index.

Canonical Tags:

1. All filtered pages are canonicalised to the main page avoid indexing and duplication.

Current Problem:

Despite the above steps, all filtered URLs are still being indexed by Google.

Google Search Console shows the URLs as indexed but blocked by robots.txt, which indicates Google cannot crawl them but somehow indexed them.

I understand that if the canonical tag is properly configured, we do not need to set the noindex robots tag, but our developer did so just to be safe. These are also blocked in the robots.txt file. So I'm not sure if all of this multiple blocking is causing an issue or not.

Possible Causes (My Thoughts):

• Conflicting Signals: Could robots.txt be preventing Google from processing the meta robots noindex?

What I’ve Checked So Far:

• The URLs are not linked internally on the website or included in the sitemap.
• They appear in the source code as part of the layered navigation but are not visibly linked for users.

Questions for the Forum:

• How do I ensure Google stops indexing these URLs? Should I temporarily remove the robots.txt block to let Google process the noindex tag?
• Could having multiple directives (robots.txt + noindex + canonical) cause confusion?
• Any other suggestions for ensuring these URLs are permanently excluded from Google’s index?

Any help or insight would be greatly appreciated!

It It seems like the issue arises from search filters or parameters (like 'size 12') generating dynamic URLs, which Google indexes quickly. This happens when a user searches for a specific product or filter (e.g., size or color), and the website automatically generates URLs with the filter criteria. These dynamic URLs are created when users apply filters, and Google ends up indexing them.

I have heard about using AJAX to prevent this, but I haven’t personally implemented it. Have you considered using plugins like FacetWP, WooCommerce Product Filter, or Search & Filter Pro? These plugins implement AJAX based filtering, which helps stop the generation of new URLs for each filter combination, effectively preventing Google from indexing them.

Your process and guesses are totally in point,
So, you either allow these URLs in robots.txt to let them get recrawled, then no-indexed based on the meta/canonical tags,
Or, you leave them blocked in robots.txt, G bots will fail several crawl/recrawl attempts and will get them out of the index eventually,

I'll personally go with the 1st option,
It's better for SEO technical-wise rather than just blocking the bots (too late for that),
They're part of the website pages from the point of view of Google algo currently and you're just blocking them,
You're letting them know that these pages should not be indexed and they're duplicates of other main pages that should be used instead,

Hope this makes sense & helps,
Best of luck!