Interesting X article on Brave Search (and it's integration with Claude)

We all know each of the big AI companies uses search results as a basis for RAG queries, and as none of them have their own sizable proprietary index (yet!), they're using third parties for this purpose. Gemini is obviously powered by Google's index, OpenAI is heavily dependent on Bing (though I'm sure you'll have seen their crawler in the wild by now), and Claude uses the lesser-know Brave Search. The thing about Brave Search is that a great deal of the tech behind it is open source: https://github.com/brave/web-discovery-project

This X article has dived deep into the guts of Brave Search, and it makes for some interesting reading:
A few of the things mentioned certainly sparked my interest, such as lower HTML length ratio pages being discarded by the crawler - I tend to make my bulk sites as lean as possible on the html, so I will be testing with more added markup. Engagement signals also look automatable to some extent.

Remember, if you want to be cited by Claude, you most likely need to be in Brave Search's index!

as a long-time Brave user, and as a non-tech literate dude, does this mean that the fake-ass AI is going to steal my content (probably search history, but what else??) for its useless training? Or did I not understand this correctly?

richer semantic markup is worth testing if you’re running bulk sites. getting indexed in Brave is pretty straightforward compared to Google.

tazarbm said:

as a long-time Brave user, and as a non-tech literate dude, does this mean that the fake-ass AI is going to steal my content (probably search history, but what else??) for its useless training? Or did I not understand this correctly?

Click to expand...

You can toggle Web Discovery Project on/off here: brave://settings/search
I think it's off by default.

Steptoe said:

You can toggle Web Discovery Project on/off here: brave://settings/search
I think it's off by default.

Click to expand...

It was already off. Thanks!

I like brave. sadly they are still behind Bing. updating pages to newer content. For example I had change OpenGraph image for my entire site 6 months ago. Google updated it within a month, Bing took 3-4 months, Brave is still shown old images, old favicon, etc. But it certainly indexed newer pages and subdomains of my site.

good reminder that ranking in google isn't the whole picture anymore.
getting indexed where ai systems pull data from is becoming just as important.