Is it okay for robots.txt ?

0

01rahat01

Newbie

Jan 30, 2023

6

2

• Oct 7, 2023

• #1

for PHP and laravel based websites these two lines are okay ? Or have to add something else for crawling search engines?

User-agent: *
Disallow:

I added this line to my robots.txt file and put it in my public_html directory

 

tazarbm

Elite Member

Executive VIP

Jr. VIP

Oct 28, 2020

11,663

14,636

• Oct 7, 2023

• #2

the User-agent: * part tells ALL bots to do something, but I'm not sure that the Disallow: part is accurate. I usually see it accompanied by a / or a folder's name or something, I'm not sure that a naked Disallow: will do anything (I'm not techy enough, sorry)

And yeah, robots.txt should go into public_html for sure...

 

U

ultiox

Newbie

Jul 15, 2014

15

8

• Oct 7, 2023

• #3

Your command will not take any action because it is empty.

 

K

kainatayoub

Registered Member

Aug 4, 2023

60

8

• Oct 9, 2023

• #4

its is empty your command is not working.

 

Vitaminized

Registered Member

Aug 31, 2022

65

36

• Oct 9, 2023

• #5

No-no, always specify "User-agent"!

 

NeedMyLink

Elite Member

Jr. VIP

Nov 23, 2020

5,930

1,706

• Oct 9, 2023

• #6

With that setup, you're allowing all bots to crawl everything on your site. Generally okay, but might want to reconsider if you've got any private folders or data.For a PHP or Laravel site, this basic setup is fine. Just remember, if you've got any admin paths, staging areas, or anything sensitive, it's wise to disallow those.