robots.txt question


Elite Member
Aug 16, 2016
Do you guys know anything about changing the robots.txt file?

I know that articles from medium regular members (those with @ before user ID) used to rank.

For example a regular user's URL https:// @username/ the-story-title-here

But by checking their robots file it shows that it stops crawlers and GoogleBot from crawling these pages. Here's their robots.txt content

User-Agent: *
Disallow: /m/
Disallow: /me/
Disallow: /@me$
Disallow: /@me/
Disallow: /*/*/edit
Allow: /_/
Allow: /_/api/users/*/meta
Allow: /_/api/users/*/profile/stream
Allow: /_/api/posts/*/responses
Allow: /_/api/posts/*/responsesStream
Allow: /_/api/posts/*/related

My question is: how come there are links indexed by Google under these conditions?
Check page source of the original page and you will see

<meta name="robots" content="index, follow">
AdBlock Detected

We get it, advertisements are annoying!

Sure, ad-blocking software does a great job at blocking ads, but it also blocks useful features and essential functions on BlackHatWorld and other forums. These functions are unrelated to ads, such as internal links and images. For the best site experience please disable your AdBlocker.

I've Disabled AdBlock