Medium.com robots.txt question

Chris.Roark

Elite Member
Aug 16, 2016
1,942
686
Do you guys know anything about medium.com changing the robots.txt file?

I know that articles from medium regular members (those with @ before user ID) used to rank.

For example a regular user's URL https:// medium.com/ @username/ the-story-title-here

But by checking their robots file https://medium.com/robots.txt it shows that it stops crawlers and GoogleBot from crawling these pages. Here's their robots.txt content

User-Agent: *
Disallow: /m/
Disallow: /me/
Disallow: /@me$
Disallow: /@me/
Disallow: /*/*/edit
Allow: /_/
Allow: /_/api/users/*/meta
Allow: /_/api/users/*/profile/stream
Allow: /_/api/posts/*/responses
Allow: /_/api/posts/*/responsesStream
Allow: /_/api/posts/*/related
Sitemap: https://medium.com/sitemap/sitemap.xml

My question is: how come there are links indexed by Google under these conditions?
 
Check page source of the original page and you will see

<meta name="robots" content="index, follow">
 
Back
Top
AdBlock Detected

We get it, advertisements are annoying!

Sure, ad-blocking software does a great job at blocking ads, but it also blocks useful features and essential functions on BlackHatWorld and other forums. These functions are unrelated to ads, such as internal links and images. For the best site experience please disable your AdBlocker.

I've Disabled AdBlock