Huge Robots.txt DB. Interesting to check out

Panther28 · May 29, 2025

https://openrobotstxt.org/

Just someone explain why there is a Neil Patel bot? lol

Yoram · May 29, 2025

No ahrefs crawler, damn they said it was the 2nd most active crawler on the web lol

seoargentina · May 29, 2025

Now Google it's in real danger.

People will say "Neil Patel it".

tiiberius · May 29, 2025

ReaL_GainZ said:
No ahrefs crawler, damn they said it was the 2nd most active crawler on the web lol

Well these are mentions, not activity.

It is first in the list of directed ones actually:
https://openrobotstxt.org/stats/most-directed

-

Yoram · May 29, 2025

tiiberius said:
Well these are mentions, not activity.

It is first in the list of directed ones actually:
https://openrobotstxt.org/stats/most-directed

-

Wow that's impressive! Doesn't Google have to catch up? Feels like they should be #1

Steptoe · May 29, 2025

Panther28 said:
https://openrobotstxt.org/

Just someone explain why there is a Neil Patel bot? lol

View attachment 447762

Olde Neil is resurrecting the referer spam tactic perhaps as a 'new technique'

hideath · May 29, 2025

ReaL_GainZ said:
No ahrefs crawler, damn they said it was the 2nd most active crawler on the web lol

seoargentina said:
Now Google it's in real danger.

People will say "Neil Patel it".

ReaL_GainZ said:
Wow that's impressive! Doesn't Google have to catch up? Feels like they should be #1

You surely missed that's a "placeholder"

Huge Robots.txt DB. Interesting to check out

Panther28

Elite Member

Yoram

Elite Member

seoargentina

Registered Member

tiiberius

Elite Member

Yoram

Elite Member

Steptoe

Elite Member

hideath

Elite Member

Main Menu

Marketplace

Making Money

BlackHat World