Meaning of X robot tags

There are multiple ways of managing robots

Via robots.txt, via html tags as @dekhajam commented, and via http headers (X-Robots-Tag) as you are asking
All are exactly the same. You can choose whichever you want. Be aware that not all crawlers and not all tags will be accepted via robots.txt in the last months as Google updated recently.

There are some that are better to be put in the robots.txt and others that are better in the http or html tags

Check this
https://yoast.com/x-robots-tag-play/

And this (unsupported via robots.txt)
https://webmasters.googleblog.com/2019/07/a-note-on-unsupported-rules-in-robotstxt.html

I would not recommend using http headers but meta html tags
 
Robots meta tags are pieces of code that provide crawlers instructions for how to crawl or index web page content.
There are 2 types, those that are part of the HTML page (like the meta robotstag) and those that the web server sends as HTTP headers (such as x-robots-tag)

With x-robots-tags you can do the same as with the normal meta-robots-tags.

Keep in mind that you only want to index pages which provides value to the user.

Checkout moz.com they have pretty good fundamental learning resources
/learn/seo/robots-meta-directives
 
Back
Top