Is Robots.txt enough to hide pages from google search?

From my experience, not really. Robots.txt only tells search engines not to crawl certain pages, but it doesn't stop google to from indexing them if they are linked from other websites. For stronger control, I have found that using a noindex meta tag or password protecting the page is a better option.
What do you think - have you used other methods to hide disallowed pages from google?

ZenWolf said:

From my experience, not really. Robots.txt only tells search engines not to crawl certain pages, but it doesn't stop google to from indexing them if they are linked from other websites. For stronger control, I have found that using a noindex meta tag or password protecting the page is a better option.
What do you think - have you used other methods to hide disallowed pages from google?

Click to expand...

Robots.txt is a mere instruction, there's nothing preventing the bots from disobeying the instructions if it was programmed to. If you really don't want Google to crawl and/or index the pages, you may need a stronger method such as password protecting the pages.

Robots.txt is a good first step, but you're right, it doesn't guarantee pages will not be indexed if they are linked elsewhere. Adding a noindex meta tag is best for preventing indexing.

For extra security, blocking access via server-side authentication works best. I usually combine methods depending on the sensitivity of the page.

Robots.txt only stops Google from reading the page, but it can still show up in search if other sites link to it. I use the noindex tag instead.. it tells Google not to show the page at all. Removing it from the sitemap or using Search Console also helps.

Yeah, totally agree. Robots.txt is more like a polite request—Google can still index the page if it finds a link elsewhere. I’ve had better luck combining noindex with .htaccess rules or using canonical tags strategically. Just blocking it isn’t enough these days.

That's a great point about robots.txt Its often misunderstood. I totally agree its more of a polite request than a firm directive. I've definitely seen pages sneak into Google's index despite being disallowed in robots.txt especially when external links are involved. Besides noindex meta tags which I use frequently I've also found success with using the X-Robots-Tag HTTP header. Its similar to the noindex meta tag but implemented on the server side. Its particularly useful for non-HTML files like PDFs or images. Also sometimes if I am dealing with sensitive data I would just remove the page or content altogether. That's probably the surest way to keep things completely out of search results.

robots.txt instructs bots how to crawl and how not to. If you don’t want a page indexed you use noindex meta tag.

You can tag a page as nofollow on RankMath.

Block the googlebot in .htaccess or use a cdn that have option to block bots for certain pages.

Robots.txt is more like a polite request, not a lock. I usually go with the noindex meta tag too—it’s way more reliable if you really want to keep pages out of search results.

No, robots.txt only tells search engines not to crawl pages, but it doesn’t stop them from indexing if other sites link to them. To fully hide pages, use a "noindex" meta tag or restrict access with authentication.

Robots.txt tells search engines not to crawl some pages, and they complain. Google will give you "Blocked by Robots.txt" error in GSC. It's better to do it with a noindex directive in the page. If you have too many pages, you can do it with htaccess. For example, if you don't want to index your pdf pages, then add a directive in htaccess to make them nofollow and noindex. Crawlers will check the page, find that they are noindex, and return. If you block crawlers through robots file, then they'll never know that the page is noindex as they don't (in theory) crawl the page.

If you have sensitive or premium content in pages (like personal details, membership content, copyright-protected content, etc.), then it's better to lock them or isolate them from search engine crawlers. I have found some of my pages in Bing search even after adding noindex and nofollow in both page and robots.txt, so search engines may not strictly follow these directives. So, it's better not to keep private content in their reach.