Huge Robots.txt DB. Interesting to check out

[Panther28]

https://openrobotstxt.org/

Just someone explain why there is a Neil Patel bot? lol

 

[Yoram]

No ahrefs crawler, damn they said it was the 2nd most active crawler on the web lol

 

[seoargentina]

Now Google it's in real danger.

People will say "Neil Patel it".

 

[tiiberius]

No ahrefs crawler, damn they said it was the 2nd most active crawler on the web lol

Well these are mentions, not activity.

It is first in the list of directed ones actually:
https://openrobotstxt.org/stats/most-directed

-

 

[Yoram]

Well these are mentions, not activity.

It is first in the list of directed ones actually:
https://openrobotstxt.org/stats/most-directed

-

Wow that's impressive! Doesn't Google have to catch up? Feels like they should be #1

 

[Steptoe]

https://openrobotstxt.org/

Just someone explain why there is a Neil Patel bot? lol

View attachment 447762

Olde Neil is resurrecting the referer spam tactic perhaps as a 'new technique'

 

[hideath]

No ahrefs crawler, damn they said it was the 2nd most active crawler on the web lol

Now Google it's in real danger.

People will say "Neil Patel it".

Wow that's impressive! Doesn't Google have to catch up? Feels like they should be #1

You surely missed that's a "placeholder"