Leaked Google Document: “We Have No Moat, And Neither Does OpenAI”

[RealDaddy]

(https://simonwillison.net/2023/May/4/no-moat/)

The premise of the paper is that while OpenAI and Google continue to race to build the most powerful language models, their efforts are rapidly being eclipsed by the work happening in the open-source community.

While our models still hold a slight edge in terms of quality, the gap is closing astonishingly quickly. Open-source models are faster, more customizable, more private, and pound-for-pound more capable. They are doing things with $100 and 13B params that we struggle with at $10M and 540B. And they are doing so in weeks, not months.

They go on to explain quite how much innovation happened in the open-source community following the release of Meta’s LLaMA model in March:

A tremendous outpouring of innovation followed, with just days between major developments (see The Timeline for the full breakdown). Here we are, barely a month later, and there are variants with instruction tuning, quantization, quality improvements, human evals, multimodality, RLHF, etc. etc. many of which build on each other.

Most importantly, they have solved the scaling problem to the extent that anyone can tinker. Many of the new ideas are from ordinary people. The barrier to entry for training and experimentation has dropped from the total output of a major research organization to one person, an evening, and a beefy laptop.

Why We Could Have Seen It Coming?
In many ways, this shouldn’t be a surprise to anyone. The current renaissance in open-source LLMs comes hot on the heels of a renaissance in image generation. The similarities are not lost on the community, with many calling this the "Stable Diffusion moment" for LLMs.

The paper concludes with some fascinating thoughts on strategy. Google has already found it difficult to keep its advantages protected from competitors such as OpenAI, and now that the wider research community is collaborating in the open they are going to find it even harder:

Keeping our technology secret was always a tenuous proposition. Google researchers are leaving for other companies on a regular cadence, so we can assume they know everything we know, and will continue to for as long as that pipeline is open.

But holding on to a competitive advantage in technology becomes even harder now that cutting edge research in LLMs is affordable. Research institutions all over the world are building on each other’s work, exploring the solution space in a breadth-first way that far outstrips our own capacity. We can try to hold tightly to our secrets while outside innovation dilutes their value, or we can try to learn from each other.

As for OpenAI itself?

And in the end, OpenAI doesn’t matter. They are making the same mistakes we are in their posture relative to open source, and their ability to maintain an edge is necessarily in question. Open-source alternatives can and will eventually eclipse them unless they change their stance. In this respect, at least, we can make the first move.

 

[kumbhak11]

Can someone explain all these in easy words and what significance it holds for us?

 

[RealDaddy]

Can someone explain all these in easy words and what significance it holds for us?

Google has no clue about the future of Google and OpenAI.

 

[WIZDAHL]

Can someone explain all these in easy words and what significance it holds for us?

It's clickbait bs is all.

Marketers like us aren't going to have a say in what OpenAI and Google do. We will have to take whatever they dish out.

 

[WPRipper]

Google has no clue about the future of Google and OpenAI.

And yet, the question remains: why should we care? It's a battle between big tech dudes. We hold no control.

 

[itz_styx]

ah thats been out for a while, but it was interesting to see their opinion on all this.
also a good summary of the open source process on LLMs

 

[itz_styx]

And yet, the question remains: why should we care? It's a battle between big tech dudes. We hold no control.

not anymore thats what they say - "The barrier to entry for training and experimentation has dropped from the total output of a major research organization to one person, an evening, and a beefy laptop."

 

[RealDaddy]

And yet, the question remains: why should we care? It's a battle between big tech dudes. We hold no control.

One of the big tech is important for SEO.

 

[tiiberius]

Very nicely put and summarized.

And yet, the question remains: why should we care? It's a battle between big tech dudes. We hold no control.

I personally make a living working “with” one. And appearantly we can make an impact indirectly.

 

[MoonStarz]

well they know Bard won't change anything at all, I believe they will have to reesctructure many things if they want to survivie now because that is a really strong statement and seems they are getting worry they are not making any progres to keep on top of chatgpt.

 

[lotsofmoney]

Google has no clue about the future of Google and OpenAI.

No open has the clue. That is why biggest companies in the world disappear in 10-20 years.

 

[chdseo]

Google need something different to get into AI race as Google AI program was very late as compare to competitors

 

[Ashk881]

It's true. Smaller models like Vicuna-13b can already perform decently close to chatGPT. Its only a matter of time until some open source org gets enough money to make a gpt-4 alternative.

Can someone explain all these in easy words and what significance it holds for us?

Neither Google nor OpenAI has any "exclusive sauce" that makes their LLM better. The technology is out in the open and well understood. The only thing both have is fuckton of money to compute and access to heap tons of data.

The significance this holds for normal people like you and me is the fact that more innovation in the llm-space is happening out in the open, in GitHub projects and uni PhDs, than it is happening inside the labs of OpenAI or Google.

Today you can't run ChatGpt or Bard on your computer, but you can run some open source projects like Home Assistant or Vicuna on a beefy computer. They are not as good as chatGPT, but they are getting better. Only a matter of time until they are optimised enough so we can run them on homeservers.

 

[RealDaddy]

Google need something different to get into AI race as Google AI program was very late as compare to competitors

Generative AI is their answer.

 

[RealDaddy]

well they know Bard won't change anything at all, I believe they will have to reesctructure many things if they want to survivie now because that is a really strong statement and seems they are getting worry they are not making any progres to keep on top of chatgpt.

I think Generative AI is a very smart move on their part.

 

[Curious Ed]

Can someone explain all these in easy words and what significance it holds for us?

Google and open AI are both fucked.

What do multinationals do to give themselves a competitive advantage?

They lobby and lobby hard to raise the bar to entry so high the little guy won't have a chance.

Expect draconian AI legislation in coming and the rise of 'black hat LLMs'

 

[vin42]

I think hardware matters and the one who has access to quantum tech will have an edge. Quantum machine learning is the future. Quantum hardware will make training time in seconds what takes months on a gpu server farm....and the results will be exponential leading to superintelligence....singularity is not far away

 

[Jaordas]

The things I can do with AI even with a shitty laptop already sorta worries me.

On top of the ease of creating self learning models that you can hook up to the internet and give full interation ability

someone in their basement could have already created an ai who on runaway code will become the singularity.

Not to be all doomsday about it

I dont know how to feel about google being that worried, many of us depend on their business model

to make our own money, so pretending their thoughts on the situation is irrelevant isnt the brightest thing to me.

For people who already made their money with google and live off investments now, I can imagine

they dont care and would be happy for google to fall, but some of us want to keep milking

the world as it is for longer.

This isnt to say that we dont need to adapt, but their are winners and losers in changes, and

when you are currently a winner, you dont want your current bread and butter to become a "loser"

 

[chdseo]

Generative AI is their answer.

what is this?

 

[jorgerg89]

what is this?

What I understand so far is that Generative AI produces content. They are LLM (Large Language models) that generate chat responses (chatGPT), designs (Dall-E, Stable Diffusion), deepfakes, etc.

Traditional AI was focused on detecting patterns and making "decisions" based on them. You could argue that, for example, Google Algorithm for detecting Black Hat SEO or GMB fake reviews used Traditional AI.

Both are AI, one focused on "responding" to created things (Traditional AI), the other focused on creating them (Generative AI).

This is what I've come to understand since Nov 2022, but my understanding might not be accurate. If someone else can correct me I would appreciate it too.