Robots.txt

[Eternalman]

Hey, all, I was wondering if anyone could suggest the best format for robots txt because I have used the one that I once found somewhere online which was somewhat like this:

all the usual like disallowing admin plus with some disallowing of; /?s=, /search/, /*/feed/,

So, how necessary is it to specify to not index the feed or the search?

Has anyone felt a noticeable difference between when you specify all the small details, the version simply disallowing admin?

 

[sageshark]

Good question.

I always keep it simple with 3 lines only as follows -

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://mysitename.com/sitemap_index.xml

If you have any security or optimization plugin, they will add a line or two in your robots.txt file

I haven't faced any issues till now.

Let's see what others say.

 

[Deleted member 1268689]

The format you use now that is working good. But if you have any especial requirement, you can fo some changes like no index page and google api for instant index.

 

[Eternalman]

Good question.

I always keep it simple with 3 lines only as follows -

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://mysitename.com/sitemap_index.xml

If you have any security or optimization plugin, they will add a line or two in your robots.txt file

I haven't faced any issues till now.

Let's see what others say.

That is what I have, simple and straightforward, but my competitor has a long list of disallows and allows, something like this:


User-agent: Googlebot
Allow: *.js
Allow: *.css

Allow: /wp-admin/admin-ajax.php


Disallow: /?s=
Disallow: /search/
Disallow: /wp-admin
Disallow: /*/feed/
Disallow: /wp-login.php
Disallow: /wp-register.php
Disallow: /trackback/


User-agent: *

Allow: /wp-admin/admin-ajax.php

Disallow: /wp-admin
Disallow: /wp-login.php
Disallow: /trackback/
Disallow: /wp-register.php

Sitemaps, (which I didn't include for obvious reasons)

I know what it all means, I just never thought it, like, or as if you are handholding the bot and showing him around your website, and telling him what is important and what is not. you know,

So, Now I am wondering if this is a better format, because it saves on crawl budget, but I just don't have a concrete evidence which has yielded better results.