Via robots.txt, via html tags as @dekhajam commented, and via http headers (X-Robots-Tag) as you are asking
All are exactly the same. You can choose whichever you want. Be aware that not all crawlers and not all tags will be accepted via robots.txt in the last months as Google updated recently.
There are some that are better to be put in the robots.txt and others that are better in the http or html tags
Robots meta tags are pieces of code that provide crawlers instructions for how to crawl or index web page content.
There are 2 types, those that are part of the HTML page (like the meta robotstag) and those that the web server sends as HTTP headers (such as x-robots-tag)
With x-robots-tags you can do the same as with the normal meta-robots-tags.
Keep in mind that you only want to index pages which provides value to the user.
Checkout moz.com they have pretty good fundamental learning resources
/learn/seo/robots-meta-directives
This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
By continuing to use this site, you are consenting to our use of cookies.