Replying to my own comment in case it helps someone else. I've spent the last 90 minutes or so reading up on what I can to help speed up my site.
I mainly use
https://www.webpagetest.org/ for speed test and also
https://gtmetrix.com/ which is free to register for but you don't need to register
I read this about slow plugins:
https://premium.wpmudev.org/blog/the-slowest-wordpress-plugins-and-alternatives-to-try-instead/
And of course I had number 1 and 2 on the list, Broken Link Checker and Contact Form 7.
So I deleted Broken Link Checker and then I discovered that Thrive have added their own contact form which is cool and fully customisable, so I used it instead of Contact Form 7.
I then just did a lot of trial and error of disabling a plugin I thought wasn't essential and then redoing the test on webpagetest.org.
After a lot of fluting about I disabled Async Javascipt, which was a plugin I added in order to try and speed up the site so was probably not needed.
Far Future Expiry, which apparently should speed up your site overall and is recommended on GTMetrix, but it seems to slow down the time to first byte, TTFB ,and page load time on my site.
Redirection, which helps you do 301 redirects etc, I'm guessing I don't need it active all the time.
UpdraftPlus for backups, I probably only need to do backups once a week or so. I can just activate it, do the backup every so often myself and then deactivate it.
And as suggested above, I disabled Yoast, as I pretty much know what I should do for on-page SEO myself at this stage. I can always activate after every few posts to review everything and make sure it's setup right and then deactivate again.
But it has made a massive saving on time. My site has tolerable page load speed now.
I also installed, used, then deleted WP Optimize to tidy up my database which is good housekeeping to do every couple of months.