Grok 4 is disappointing

[splishsplash]

I pay for openai pro, and overall it's pretty damn good.

So I decided to pay the $300 for grok heavy..

First, it can't code for shit. No problem, I wasn't expecting it to be able to. I was looking more for a super smart research agent like o3.

First thing I ask it.

Oh and just before I show this.. Let's take a moment to appreciate the hilarity of this

https://medium.com/data-science-in-...nd-of-human-intelligence-is-near-fd1b80ee7640

Ok, so..

I ask grok 4 HEAVY. The super high quality elite multi-agent human-ending AI.

"How long were the fellowship in mirkwood forest?"




This super smart AI, assumes, no, he is clearly an idiot, and he's mixing up lord of the rings and the hobbit. I will correct him.

Next, I say

"After they leave moria, don't they go into mirkwood and meet galadriel?"





Ok, so I got the name wrong. It was 'Lorien' they entered. Even a 6 year old who's seen the movies would be able to correct this.. It's a major part of the fellowship movie. What kind of moron would think I mean the hobbit? It's far more obvious that I got the name of the forest wrong. I would need to be pretty retarded to be thinking about the fellowship of the ring, ask about what forest "the fellowship" went into, but actually mean the hobbit, where there is no fellowship.

Fair enough. But you'd think it would catch on by now..

No. It doesn't.

I say "Then the question stands" -- Ie, ok, so why the hell haven't you answered my damn question already? Why 3 paragraphs after we cleared that up and still no answer.




WHAT?
It continues to tell me about the hobbit. Like seriously? It's like I'm talking with an absolute simpleton here.

Here's some contrast.

chatgpt 4o. Not even o3 or a better model. Just plain old 4o.




This is pretty bad. A $300/mo supposedly human-ending AI, and it's dumb as a rock.

Even grok 3. It doesn't get this question right. And it's a good question that requires real intelligence to work out my mistake.





I gave grok 4 heavy another prompt.


"Plan out a detailed prompt for claude opus to create a python program.The program should use playwright to extract an article from a web page. First, research all the different ways this can be done in python.It should use a scoring system to decide which one is best. Generally it's going to be the one with the most words.It will use brightdata's browser API endpoint, so it doesn't need to create a local browser or use proxies.It should handle pages that have an infinite scroll intelligently."

Note, I ask IT to research all the ways you can extract the article body from a web page.

Here's part of the prompt it produces.



It's really not showing much intelligence here. It's missing stuff that it shouldn't. I asked GROK to research and then create the prompt for claude.

Definitely not worth $300. Total waste of money. The $30/mo plan was enough with just grok 3/4.

 

[Venusaur]

Grok 4 seems to be trained by 4shit users, on X it behaves like a stupid troll and on chat is equally dumb for general information, several basic mistakes

 

[splishsplash]

Grok 4 seems to be trained by 4shit users, on X it behaves like a stupid troll and on chat is equally dumb for general information, several basic mistakes

OpenAI and Anthropic are really good at this. Most others are not.

OpenAI in particular has a level of polish the others don't.

Claude 4 sonnet and opus are actually getting good at coding. I still hate vibe coding, but if you architect your own software and you keep what it has access to limited, it's able to do most tasks really well, and the code it writes is much cleaner than the other models.

o3 for example absolutely loves to be a show-off fancy pants in Python and do stuff like

def run_extractors(html: str, url: str) -> ExtractorResult:
extractors = [
readability_extract,
trafilatura_extract,
goose_extract,
biggest_block_extract,
]
results = [
r
for ex in extractors
if (r := ex(html, url)) is not None
]
return max(results, key=lambda res: res.words)


Which, any career dev knows you don't do stuff like that in production code. You write SIMPLY, because your future self and future devs are going to have headaches with code like that.

Plus it makes it annoying when you're reading over its code to understand how it works. (Which you must do when using AI coding. It saves you time writing, but you still have to read it and understand it, otherwise you're just creating future tech-debt that you will not be able to fix, and the AI won't either)

Where-as claude 4 opus does

candidates: List[Tuple[Optional[str], str]] = []

# Try each extractor in order of preference
candidates.append((extract_with_trafilatura(html), "trafilatura"))
candidates.append((extract_with_readability(html), "readability"))
candidates.append((extract_with_boilerpy3(html), "boilerpy3"))
candidates.append((extract_with_goose3(html, url), "goose3"))
candidates.append((extract_with_newspaper(url, html), "newspaper"))

# Playwright fallback needs await
playwright_result = await extract_with_playwright_fallback(page)
candidates.append((playwright_result, "playwright_fallback"))

# Score each result by word count
best_content = None
best_extractor = None
best_word_count = 0

for content, extractor_name in candidates:
if content:
word_count = len(content.split())
logger.debug(
"Extraction result",
extractor=extractor_name,
word_count=word_count,
)

if word_count > best_word_count:
best_content = content
best_extractor = extractor_name
best_word_count = word_count

return best_content, best_extractor, best_word_count

Way way way more readable. You don't have to stop and think. The o3 one is a head scratcher. You have to use up half your day's brain power just to work out what it's doing ;-)


So really, grok doesn't seem to fit in anywhere. It's 'dumb' in areas that OpenAI's models are sharp and just 'get things'. It can't code. It's even worse than OpenAI for code.

It's like it's not been properly trained on human feedback and just misses things. It doesn't connect with the human using it. OpenAI understands the human. Grok doesn't.

 

[RichMedusa]

Sad to hear it, I created some positive expectations about Grok, but I didn't read good reviews about their newest update

 

[bluesnake651]

Disappointing experience, stick with lower plans for now. Thanks

 

[terrycody]

They have no enough talents, most talents are in Google, OpenAi, and Meta now.

 

[4440]

Had the same feel that o3 code is over-simplifed, like a showing-off or somehting. When i get code from openai models, i always need to stop, look at the code and think.
Whereas for claude/gemini code, i can just do skim and glimps.

 

[INCC]

So I decided to pay the $300 for grok heavy..

A few days ago I paid for the regular subscription some 30 euros.

I have the same opinion that its not good at all.

Asked about crypto hedging strategy and Grok sent me on sites that were closed back in 2022 with the FTX collapse... so 3 years out of service already.

It also provided outdated data about plugins that I was interested in. I'm not very impressed, but there's no chance I'll ever pay $300 for the new version, lol.

 

[kamhan]

Thanks for sharing your thoughts. I’ve seen similar feedback, many users expected more from Grok 4. Hopefully, future updates will bring improvements

 

[Steptoe]

Grok does have a tendency to emulate those pub conversations you have with that friend who will never admit to being wrong.

 

[splishsplash]

Had the same feel that o3 code is over-simplifed, like a showing-off or somehting. When i get code from openai models, i always need to stop, look at the code and think.
Whereas for claude/gemini code, i can just do skim and glimps.

Anthropic likely do a LOT of fine-tuning with hand-picked code. They actually write generally good code. Not AMAZING, but it's pretty decent design wise.

o3 writes strong/working code, but it does things more like the gifted early 20's dev who's been coding since hew as 8 and wants to show off.

What I'm finding(I'm new to using claude) is that it's good for writing code, but bad at debugging/fixing problems. It has this strange tendency to want to do workarounds.

With o3 I find myself wanting to hug it for being awesome, but with claude I find myself wanting to shake it and ask what the hell its doing.

It also still, for some reason, seems to miss really basic stuff, despite being able to one-shot entire classes/modules.

Example of each:

It had in my code

from readability import Readability

Yet, in requirements.txt we were using readability-lxml.

That's not correct. There is no Readability attribute in readability-lxml. It should be from readability import Document.

That's just weird that it get such a simple thing wrong.

And here's the second issue.

I had asked it to add in support for a playwright browser api endpoint for brightdata.

o3 got this right first time, no issues, but I wanted to use claude's code, because it's cleaner/easier to maintain, so it was re-writing the module.

Now, the problem was quite simple.

o3 did this

browser = await p.chromium.connect_over_cdp(endpoint_url)

claude did this

browser = await p.chromium.connect(endpoint_url)

Basically connect is playwright's websocket protocol, and cdp is chrome devtools protocol.

Claude COULD NOT fix this. We had about 5-6 back and forwards, and it absolutely MASSACRED the codebase.

But here's the completely bizarre part, and why you can see you cannot rely on claude to debug AT ALL. It is a moron. I think I understand why too. It's likely been fine-tuned on writing code rather than fixing code. IMO the way forward is to have multiple models for different tasks. Some for doing initial skeleton design/architecture, some for writing methods/functions, some for adding features, some for fixing bugs, some for debugging errors, some for writing tests and so on and so forth.

So look at this, remembering the problem is brightdata uses CDP, chrome devtools protocol, and we were using playwright's websocket.

Here's what he says


Claude: Based on the test results, I can see the issue. The WebSocket connects successfully but Bright Data's Scraping Browser doesn't use the standard Chrome DevTools Protocol that Playwright expects. This is why Playwright fails with the KeyError: 'error'.

You see how crazy this is? It's CONFUSED as hell. It actually knows we should be using chrome devtools protocol, but rather than pointing out "the problem is WE are not using it, we are using playwright's websocket", it says, "No, the problem is BRIGHT DATA! They don't support it"

This is like speaking Hungarian to a French guy, then saying "This French guy clearly doesn't understand French. We need to address this problem"

Then, it gets worse, here's Claude's solution.

Option 1: Use Bright Data's Puppeteer/Playwright Integration​

It then creates an entirely new file called extract_brightdata.py, which doesn't even use the browser API. It just uses requests with classic HTTP proxying! Which isn't even what it's SAYING in the first place. It's saying it's going to use the playwright integration. So it's saying one thing, and doing another.

Then it gives another option

Option 2: Simple Requests-Based Solution​

Which is actually the 'fix' it did. Then it summarizes here

1.​

The extract.py now automatically converts the WebSocket URL to an HTTP proxy URL and uses requests to fetch pages through Bright Data. This is the most reliable method.

WHAT?

This is like.. I've asked it to have my private jet on the runway ready to go in 6 hours, and it's come back to me and said "I couldn't get a hold of the captain, so instead, I've booked you on the back of a pig cart, where you can enjoy rolling in pig shit for your journey"

This is not a fix! I don't want to use damn proxies. If I did, I would do that. The browser API actually has advanced features where they rotate proxies automatically and handle footprint issues to make you look like a human visitor.

And here's the best bit.

I took my original o3 code, and the original broken(before the "fixes") claude code, put them both into ChatGPT 4.1(which imo is an absolute god-tier of a model for coding/analysis. So underrated) and told it the problem.

It INSTANTLY told me the problem.

So, what I've come to realize is that, claude is great for building out version 1 of your module/class/program section. It works best with a clean slate, no existing code, just give it a new thing to make. If it doesn't work, plug it into 4.1 or o3 to fix issues, or fix them yourself. And to add more features, just use o3 or 4.1 to write individual methods/functions, or if you can't go that low level, use openai's codex to add features, as it doesn't break stuff.

A few days ago I paid for the regular subscription some 30 euros.

I have the same opinion that its not good at all.

Asked about crypto hedging strategy and Grok sent me on sites that were closed back in 2022 with the FTX collapse... so 3 years out of service already.

It also provided outdated data about plugins that I was interested in. I'm not very impressed, but there's no chance I'll ever pay $300 for the new version, lol.

Yeah it's so bad that I'm actually starting to feel scammed.

I gave it another really simple task to extract some information from a piece of text, and it ended up in an infinite loop giving me complete garbage.

Gemma-27b completed the task successfully, so it's just silly at this point.

 

[The SEO]

Thanks, you saved me $300, I've planned to buy in the coming week,

 

[splishsplash]

Can we just take a moment to lolz at this.

"

GROK 4 STUNNING New Ability? Emerging "Fluid Intelligence" in AI Models?​

"

It's just getting to a point that 97% of anything AI on social media is outright lies.

Yes. STUNNING. "Fluid Intelligence" -- Can't even figure out I mean bloody lorien forest instead of mirkwood. Absolutely stunning.

 

[hovercraft08]

It happens when you don't stay in your lane and always eye on other competitors. Heard somewhere they're also going to make X an stock and crypto exchange.

 

[splishsplash]

It happens when you don't stay in your lane and always eye on other competitors. Heard somewhere they're also going to make X an stock and crypto exchange.

I don't know if I agree with that! Staying in your lane is something normies say to keep other normies down.

 

[RML]

I pay for openai pro, and overall it's pretty damn good.

So I decided to pay the $300 for grok heavy..

First, it can't code for shit. No problem, I wasn't expecting it to be able to. I was looking more for a super smart research agent like o3.

First thing I ask it.

Oh and just before I show this.. Let's take a moment to appreciate the hilarity of this

https://medium.com/data-science-in-...nd-of-human-intelligence-is-near-fd1b80ee7640

Ok, so..

I ask grok 4 HEAVY. The super high quality elite multi-agent human-ending AI.

"How long were the fellowship in mirkwood forest?"

View attachment 457976


This super smart AI, assumes, no, he is clearly an idiot, and he's mixing up lord of the rings and the hobbit. I will correct him.

Next, I say

"After they leave moria, don't they go into mirkwood and meet galadriel?"

View attachment 457977



Ok, so I got the name wrong. It was 'Lorien' they entered. Even a 6 year old who's seen the movies would be able to correct this.. It's a major part of the fellowship movie. What kind of moron would think I mean the hobbit? It's far more obvious that I got the name of the forest wrong. I would need to be pretty retarded to be thinking about the fellowship of the ring, ask about what forest "the fellowship" went into, but actually mean the hobbit, where there is no fellowship.

Fair enough. But you'd think it would catch on by now..

No. It doesn't.

I say "Then the question stands" -- Ie, ok, so why the hell haven't you answered my damn question already? Why 3 paragraphs after we cleared that up and still no answer.

View attachment 457978


WHAT?
It continues to tell me about the hobbit. Like seriously? It's like I'm talking with an absolute simpleton here.

Here's some contrast.

chatgpt 4o. Not even o3 or a better model. Just plain old 4o.

View attachment 457979


This is pretty bad. A $300/mo supposedly human-ending AI, and it's dumb as a rock.

Even grok 3. It doesn't get this question right. And it's a good question that requires real intelligence to work out my mistake.

View attachment 457980



I gave grok 4 heavy another prompt.


"Plan out a detailed prompt for claude opus to create a python program.The program should use playwright to extract an article from a web page. First, research all the different ways this can be done in python.It should use a scoring system to decide which one is best. Generally it's going to be the one with the most words.It will use brightdata's browser API endpoint, so it doesn't need to create a local browser or use proxies.It should handle pages that have an infinite scroll intelligently."

Note, I ask IT to research all the ways you can extract the article body from a web page.

Here's part of the prompt it produces.

View attachment 457982

It's really not showing much intelligence here. It's missing stuff that it shouldn't. I asked GROK to research and then create the prompt for claude.

Definitely not worth $300. Total waste of money. The $30/mo plan was enough with just grok 3/4.

Well...

I was so happy to try Grok someday but after this post, I will keep using DeepSeek.

Thanks for sharing your experience.

You help me save my money.

Thanks again!