I fixed this problem by re-coding my website from scratch. Took me 2 weeks but I now have a 100% pass rate on my website.
Okay some advice:
1. Google is not testing how fast your page loads. Get that incorrect idea out of your head. Your page could take several minutes to load and still pass the test.
2. Google is testing how fast the first screen of your page loads on a simulated Google Nexus device. A small tablet device.
What's important here is the top of your page that will fit in the screen of a Google Nexus.
If the top of your page is plain text and then the slow to load images are further down, that's completely okay.
3. The biggest issue is likely to be having a large framework that needs to load before you can render the top of the page.
The tricks I used were:
Merged multiple CSS files into one. Less different resources that need to be fetched before rendering starts the better the score.
Converting images to low quality .webp (if they are at the top of the page) then embedding them into the web page as base64 encodes.
You need to use a picture tag for this otherwise it will break on old iOS devices.
<picture>
<source srcset="data:image/webp;base64,JKasdasddasasda full encoding string here" type="image/webp">
<source srcset="/static/img/imagename.png">
<img src="/static/img/imageName.png" alt="Image Alt Text">
</picture>
Dropped client side rendering for server side rendering.
Caching static portions of the page.
For pages that weren't passing, I used a cut down customized CSS file, that works just for that page, which I embedded directly into the page in a <style> tag.
That's how the problem is solved. How in earth you apply this to wordpress I have no idea through.