This things must be settled asap: Robots Exclusion Protocol

Since the revolution of AI generated content, do you believe that Robots Exclusion Protocol will be widely followed by the search engine or AI bots like chatgpt, perplexity or grok?

I think there should happen some lawsuit against this.

Though they are trying to convince us by putting links in the summarize but recent research of cloudflare shows that people are more reluctant to click on those citation links.

The result? less and less click to the real source and more concern to the publishers.

Large publisher like bbc, businessinsider, fastcompany are already showing deep concern regarding their content protection.

You bring up a valid point. While some AI bots respect the Robots Exclusion Protocol, many don’t, and it’s becoming a bigger issue for publishers losing traffic. The citation links are helpful in theory, but if users aren’t clicking through, the value to content creators drops. A legal push might be inevitable if big publishers keep raising the alarm.

The Robots Exclusion Protocol is a standard, but not all bots strictly follow it especially newer AI bots.
Lawsuits might push for better enforcement, but the real challenge is balancing content protection and fair use.

Publishers can limit bot access by using robots.txt with stricter server rules or exploring tools like Cloudflare for bot management.

For content monetization, focus on licensing or paid API partnerships with trusted platforms.