Should I edit robots.txt?

Rollfic

Elite Member

Dec 13, 2019

1,580

724

• Mar 10, 2021

• #1

So there is no robots.txt in my file manager. Searched it more than one time lol.

This is what my robots.txt look like from the URL -

Should I remove the Disallow and Allow and add Crawl-Delay of 5 or something?

 

hideath

Elite Member

Jun 16, 2019

6,620

7,370

• Mar 10, 2021

• #2

you don't want crawlers to actually check this folder,
you can add rules to disallow tags and categories if you want,
apart from that, nothing to worry about

 

Rollfic

Elite Member

Dec 13, 2019

1,580

724

• Mar 10, 2021
• Thread Starter Thread Starter

• #3

hideath said:

you don't want crawlers to actually check this folder,
you can add rules to disallow tags and categories if you want,
apart from that, nothing to worry about

Click to expand...

Still wondering why there is no robots.txt file in my file manager? Do web hosters hide that nowadays?

 

fb3003

Elite Member

Aug 14, 2015

5,035

3,500

• Mar 10, 2021

• #4

Rollfic said:

Still wondering why there is no robots.txt file in my file manager? Do web hosters hide that nowadays?

Click to expand...

not that I know of.

please see this
https://yoa.st/robots-txt

yoast point of view on best practice on how your robots.txt should look like

 

Mymusic

BANNED

Nov 18, 2018

287

95

• Mar 10, 2021

• #5

If you want to edit your robot.txt file you can install the rank math plugin and there you will find the option to edit it within the plugin options