fb3003
Elite Member
- Aug 14, 2015
- 5,035
- 3,500
guys - i was reading this https://yoast.com/wordpress-robots-txt-example/ and have looked at their robots file: https://yoast.com/robots.txt, which has 1 line:
User-Agent: *
so basically, suggestion is not to put anything in robots.txt. what works for you guys, specially if you have wordpress. my robots txt is this:
User-agent: *
Disallow: */author/*
Disallow: */tag/*
Disallow: */category/*
Sitemap: https://www.MYWEBSITE.com/sitemap_index.xml
I am thinking to remove the 3 disallow lines, so that I remove all coverage issues on google search console.
thoughts
User-Agent: *
so basically, suggestion is not to put anything in robots.txt. what works for you guys, specially if you have wordpress. my robots txt is this:
User-agent: *
Disallow: */author/*
Disallow: */tag/*
Disallow: */category/*
Sitemap: https://www.MYWEBSITE.com/sitemap_index.xml
I am thinking to remove the 3 disallow lines, so that I remove all coverage issues on google search console.
thoughts