robots.txt question

OldHead

Newbie
Joined
Jul 18, 2021
Messages
10
Reaction score
6
I'm setting up RankMath and notice the robot.txt file says something that confused me but now it looks like it might just be example and not actually code. It says this so far:

#****************************************************************************
# robots.txt
# : Robots, spiders, and search engines use this file to detmine which
# content they should *not* crawl while indexing your website.
# : This system is called "The Robots Exclusion Standard."
# : It is strongly encouraged to use a robots.txt validator to check
# for valid syntax before any robots read it!
#
# Examples:
#
# Instruct all robots to stay out of the admin area.
# : User-agent: *
# : Disallow: /admin/
#
# Restrict Google and MSN from indexing your images.
# : User-agent: Googlebot
# : Disallow: /images/
# : User-agent: MSNBot
# : Disallow: /images/
#****************************************************************************

User-agent: *
Disallow:

It looks like the # is just text and not actually writing anything to the file. All I saw was it saying restricting google from indexing images but it looks like it's just showing an example if you wanted to do that. Am I correct that it's not actually writing that code and just giving a tutorial on how to if you wanted to?
 
Everything in "#" is actually a comment. You should enter anything if you have in the part:

User-agent: *
Disallow:
 
Gotcha, makes sense. Just wanted to make sure, thanks for the reply guys.
 
Back
Top