Chris.Roark
Elite Member
- Aug 16, 2016
- 1,942
- 687
Do you guys know anything about medium.com changing the robots.txt file?
I know that articles from medium regular members (those with @ before user ID) used to rank.
For example a regular user's URL https:// medium.com/ @username/ the-story-title-here
But by checking their robots file https://medium.com/robots.txt it shows that it stops crawlers and GoogleBot from crawling these pages. Here's their robots.txt content
User-Agent: *
Disallow: /m/
Disallow: /me/
Disallow: /@me$
Disallow: /@me/
Disallow: /*/*/edit
Allow: /_/
Allow: /_/api/users/*/meta
Allow: /_/api/users/*/profile/stream
Allow: /_/api/posts/*/responses
Allow: /_/api/posts/*/responsesStream
Allow: /_/api/posts/*/related
Sitemap: https://medium.com/sitemap/sitemap.xml
My question is: how come there are links indexed by Google under these conditions?
I know that articles from medium regular members (those with @ before user ID) used to rank.
For example a regular user's URL https:// medium.com/ @username/ the-story-title-here
But by checking their robots file https://medium.com/robots.txt it shows that it stops crawlers and GoogleBot from crawling these pages. Here's their robots.txt content
User-Agent: *
Disallow: /m/
Disallow: /me/
Disallow: /@me$
Disallow: /@me/
Disallow: /*/*/edit
Allow: /_/
Allow: /_/api/users/*/meta
Allow: /_/api/users/*/profile/stream
Allow: /_/api/posts/*/responses
Allow: /_/api/posts/*/responsesStream
Allow: /_/api/posts/*/related
Sitemap: https://medium.com/sitemap/sitemap.xml
My question is: how come there are links indexed by Google under these conditions?