robots.txt and wordpress

fb3003

Elite Member
Joined
Aug 14, 2015
Messages
5,035
Reaction score
3,500
guys - i was reading this https://yoast.com/wordpress-robots-txt-example/ and have looked at their robots file: https://yoast.com/robots.txt, which has 1 line:
User-Agent: *

so basically, suggestion is not to put anything in robots.txt. what works for you guys, specially if you have wordpress. my robots txt is this:
User-agent: *
Disallow: */author/*
Disallow: */tag/*
Disallow: */category/*

Sitemap: https://www.MYWEBSITE.com/sitemap_index.xml

I am thinking to remove the 3 disallow lines, so that I remove all coverage issues on google search console.

thoughts
 
Just don't use Yoast go for a custom solution. You're giving them so much with that plugin that it's ridiculous. And yes, for your question, leave almost nothing there unless you specifically want to block stuff. It should just be the:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap:
https://whatever.sitemap.xml

Ideally.
 
Last edited:
Back
Top