Medium.com robots.txt question

Chris.Roark

Elite Member
Joined
Aug 16, 2016
Messages
1,942
Reaction score
687
Do you guys know anything about medium.com changing the robots.txt file?

I know that articles from medium regular members (those with @ before user ID) used to rank.

For example a regular user's URL https:// medium.com/ @username/ the-story-title-here

But by checking their robots file https://medium.com/robots.txt it shows that it stops crawlers and GoogleBot from crawling these pages. Here's their robots.txt content

User-Agent: *
Disallow: /m/
Disallow: /me/
Disallow: /@me$
Disallow: /@me/
Disallow: /*/*/edit
Allow: /_/
Allow: /_/api/users/*/meta
Allow: /_/api/users/*/profile/stream
Allow: /_/api/posts/*/responses
Allow: /_/api/posts/*/responsesStream
Allow: /_/api/posts/*/related
Sitemap: https://medium.com/sitemap/sitemap.xml

My question is: how come there are links indexed by Google under these conditions?
 
Check page source of the original page and you will see

<meta name="robots" content="index, follow">
 
Back
Top