Core Web Vitals won't validate. Any other experts here?

[herps]

Here's the scenario:
We have a Wordpress site (which will go unnamed) that has thousands of pages of content, gets about 4mil views per month and we've optimized the hell out of it. On local lighthouse tests we're in the green for everything and under 1.5s total load time. Our ad network (AdThrive) doesn't seem to be impeding on performance very much. All tests are done in incognito with no browser plugins installed. Google Page Speed Insights reports similar results. Lab data looks good. Field data is totally off though. CLS is 0 on lab data, but CLS goes above the threshold for field data. We've been trying to get this to validate since March.

Optimization Approach:

• CloudFlare with HTML Edge Caching (Business plan) and all the other bells and whistles
• WP Rocket caching + Automated Critical Path CSS (with a custom fallback CPCSS) to almost eliminate CLS
• Perfmatters to reduce number of active scripts on specific page types
• Painstaking optimizations on the theme HTML itself. This means reduction of HTML/CSS complexity to the point where we can't reduce it much more than this. Also merged many mini plugins with our theme file to reduce number of file transfers from CSS/JS.
• All animations were disabled

Everything in the Google Search Console is red when some should at least be yellow. 0 pages are validating. Obviously, lab data is pretty terrible.

Testing Approach:

• Google Page Speed Insights is our gold standard and our primary metric
• Lighthouse is used more frequently for testing small changes
• Test results are sometimes compared against GTMetrix
• We also use semrush to crawl the entire website for issues
• We're about to crawl the entire website using Lighthouse CLI

My background: Programmer for over 26 years and I've optimized other websites of this size that are validating. So please feel free to throw technical jargon at me.

I feel like we've left no stone unturned here, but maybe we're overlooking something? Does anyone have ideas on what we could be missing? Is it maybe possible that our users are using such shitty phones that they're killing our averages? We do have international visitors. Please help!

Fun fact: Did you know Lighthouse plays fireworks for you if your site gets 100%? See attached image for screenshot taken from potato phone.

 

[smootie79]

on Wprocket on scripts to delay under File Optimization put your domain name yourdomain.com in that area and try it, I did that for a few sites and it worked, it broke a few sites too but if it does just remove it, if it don't, it should work

 

[hideath]

if you have a problem with LCS, it's usually because of content shifting and not necessarily have anything to do with the website speed,
this can be caused by lazy loading one of the top parts of the website like logo, featured image in top, placing ad in the top of the page or anything else

 

[herps]

More info! Attached is a screenshot of a Lighthouse test on one of our more popular articles (identifying information removed for privacy reasons). This is tested on a pretty fast PC and forgot to run in incognito so the results might be skewed. Also attached is a Google Page Speed Insights test result. You can see how far field data is from our lab test.

 

[fb3003]

Here's the scenario:
We have a Wordpress site (which will go unnamed) that has thousands of pages of content, gets about 4mil views per month and we've optimized the hell out of it. On local lighthouse tests we're in the green for everything and under 1.5s total load time. Our ad network (AdThrive) doesn't seem to be impeding on performance very much. All tests are done in incognito with no browser plugins installed. Google Page Speed Insights reports similar results. Lab data looks good. Field data is totally off though. CLS is 0 on lab data, but CLS goes above the threshold for field data. We've been trying to get this to validate since March.

Optimization Approach:

• CloudFlare with HTML Edge Caching (Business plan) and all the other bells and whistles
• WP Rocket caching + Automated Critical Path CSS (with a custom fallback CPCSS) to almost eliminate CLS
• Perfmatters to reduce number of active scripts on specific page types
• Painstaking optimizations on the theme HTML itself. This means reduction of HTML/CSS complexity to the point where we can't reduce it much more than this. Also merged many mini plugins with our theme file to reduce number of file transfers from CSS/JS.
• All animations were disabled

Everything in the Google Search Console is red when some should at least be yellow. 0 pages are validating. Obviously, lab data is pretty terrible.

Testing Approach:

• Google Page Speed Insights is our gold standard and our primary metric
• Lighthouse is used more frequently for testing small changes
• Test results are sometimes compared against GTMetrix
• We also use semrush to crawl the entire website for issues
• We're about to crawl the entire website using Lighthouse CLI

My background: Programmer for over 26 years and I've optimized other websites of this size that are validating. So please feel free to throw technical jargon at me.

I feel like we've left no stone unturned here, but maybe we're overlooking something? Does anyone have ideas on what we could be missing? Is it maybe possible that our users are using such shitty phones that they're killing our averages? We do have international visitors. Please help!

Fun fact: Did you know Lighthouse plays fireworks for you if your site gets 100%? See attached image for screenshot taken from potato phone.

1. what happens when you turn off cirical css in wp-rocket?
2. what is your theme?
3. can you switch to autoptimzie + wpfastest cache premium and test again? i have had good results from using these 2 together.
4. can you backup your whole site, and switch theme to generatepress to see if you are still having the same issue for web vitals?

 

[M4DM4X]

I don't know if your theme allows it but generally to speed up my sites I put all the scripts in the footer, including gg analytics one. For this specific one I use minimalanalytics script instead of gg's which is faster.
All along with gzip compression (via htaccess) and images compression (or conversion to webp format).
You can also get rid of google fonts, and lighten your css.
I'm sure you already know all of this since you are experimented and these are basic stuff, but maybe it will help some other fellows optimizing their site.

 

[Shaybib]

Did you run validate fix on gsc? Field data takes time to get data so maybe it will get fixed if you only recentlly optimize. Incognito is not importent as i think speed insight run from google servers.

 

[herps]

Hey all thanks for the feedback! A really long post with responses to everything....

on Wprocket on scripts to delay under File Optimization put your domain name yourdomain.com in that area and try it, I did that for a few sites and it worked, it broke a few sites too but if it does just remove it, if it don't, it should work

I haven't tried this yet. Does this defer all the scripts til the end?

if you have a problem with LCS, it's usually because of content shifting and not necessarily have anything to do with the website speed,
this can be caused by lazy loading one of the top parts of the website like logo, featured image in top, placing ad in the top of the page or anything else

Yeah we're only lazy loading stuff below the fold. A lot of our images have been moved below the fold as well.

1. what happens when you turn off cirical css in wp-rocket?
2. what is your theme?
3. can you switch to autoptimzie + wpfastest cache premium and test again? i have had good results from using these 2 together.
4. can you backup your whole site, and switch theme to generatepress to see if you are still having the same issue for web vitals?

1. With our theme, critical CSS is necessary to keep things from shifting. When it turns off, we'll get a flash of unstyled content. The horror! lol
2. At this point, we've diverged from the original theme so much it might as well be considered custom. I've gone through several thousand lines of CSS and PHP to reduce the complexity of the selectors, eliminate unnecessary class names and ids, and reduce the overall complexity of the code for maximum performance gains. The theme also output buffers and removes comments, newlines and unnecessary spaces to compensate for HTML to text ratio.
3. I've used autoptimize + wpfastest cache for smaller websites. It takes much more effort to get the same performance gains that I do out of WP Rocket. Not saying one tool is better than the other...just saying WP Rocket works better for my workflow.
4. GeneratePress is a great starter tool, but at this stage we need something really fine tuned for our blog.

I don't know if your theme allows it but generally to speed up my sites I put all the scripts in the footer, including gg analytics one. For this specific one I use minimalanalytics script instead of gg's which is faster.
All along with gzip compression (via htaccess) and images compression (or conversion to webp format).
You can also get rid of google fonts, and lighten your css.
I'm sure you already know all of this since you are experimented and these are basic stuff, but maybe it will help some other fellows optimizing their site.

That's a good tip. We've moved a lot of our scripts to the lowest part that we can move them. We do have ad scripts and some of those require us to keep things at the top.

Did you run validate fix on gsc? Field data takes time to get data so maybe it will get fixed if you only recentlly optimize. Incognito is not importent as i think speed insight run from google servers.

Yes! We've been adjusting, validating, waiting. The picture of the 100% score and fireworks in my first post was actually dated March 13, 2021. That's how long we've been trying to get this to validate! Something I've noticed is that one of the smaller sites I work on gets about 40k hits per month gets validated much faster. Takes them days rather than weeks to see a result. This client with less traffic also only gets traffic in the US whereas this 4mil viewer website (in the original post) has an international audience and takes weeks to validate. There's another site I've optimized that takes weeks also and they have about 4m views per month globally as well.

I found this today and wonder how many users with sub-par internet/tech abroad could be screwing up our field data...
https://www.searchenginejournal.com/countries-with-slow-internet-can-affect-core-web-vitals-scores/402684/
Also, currently I'm scanning about 4,000 links from this site using npm lighthouse-batch. I think I'll have more info to share if anyone is interested after I compile this data tomorrow.

 

[justalazyseo]

Hello!

I work as a dev and technical expert, so this topic has been very much on my table recently. As with all things SEO, this depends. On paper, your optimization seems good. And as already mentioned in the thread, Field Data is an aggregate of last 28 days, so if improvements are recent, you just need to wait. Similar mindset as with ranking - the changes are not instant.

As you are developer, I would give you a solid advice - forget Lighthouse and PSI. Use it only to report to higher-ups, maybe take some high-level pointers, but that's that. For your purposes, you should use DevTools only. Run a performance trace of your website and you will see in much more detail what is going on there. It's absolutely the best tool to understand what is going on code level. Sometimes these issues are much simpler that you think and tools do not pick it up (eg. I just was with a product where GTM had errored out somehow and did not deploy removal of super heavy tracking scripts). For whatever reason PSI did not report this, but one Performance Trace showed clearly that this script was executing. Very simplistic example, but true.

In addition, there is phenomena that I have seen which is overoptimizing for PSI. It's one test, run always with some parameters. Sometimes in efforts just to make PSI score higher, the implementations are actually negative to users as it's optimized for this particular device only.

Honestly, planning to crawl whole page with Lighthouse CLI would be a total waste of time and resources. Take one performance trace from each post type. This will tell you about the high-level issues that implementations of each might have. If one individual post is having something wrong with it (let's say 1MB image), you can pick it up from GSC.

I understand that this is my opinion, but I have been on this rodeo for dozens of products again and again, especially after the announcement of the update. Seriously DevTools will make your life so much easier.

Could you clarify these couple things:
- Are you using vanilla wordpress?
- Is there headless implementation? If so, what framework/bundler? Also how are you doing the code splitting?
- Do you have any dynamic content on page? (Live feed etc.)
- When did the performance sprints end and start?

Let me know.

PS: If you're willing, I can take a quick look at your flamechart. You can send the url via DM, but for obvious reasons I understand if you refuse.

 

[fb3003]

Hello!

I work as a dev and technical expert, so this topic has been very much on my table recently. As with all things SEO, this depends. On paper, your optimization seems good. And as already mentioned in the thread, Field Data is an aggregate of last 28 days, so if improvements are recent, you just need to wait. Similar mindset as with ranking - the changes are not instant.

As you are developer, I would give you a solid advice - forget Lighthouse and PSI. Use it only to report to higher-ups, maybe take some high-level pointers, but that's that. For your purposes, you should use DevTools only. Run a performance trace of your website and you will see in much more detail what is going on there. It's absolutely the best tool to understand what is going on code level. Sometimes these issues are much simpler that you think and tools do not pick it up (eg. I just was with a product where GTM had errored out somehow and did not deploy removal of super heavy tracking scripts). For whatever reason PSI did not report this, but one Performance Trace showed clearly that this script was executing. Very simplistic example, but true.

In addition, there is phenomena that I have seen which is overoptimizing for PSI. It's one test, run always with some parameters. Sometimes in efforts just to make PSI score higher, the implementations are actually negative to users as it's optimized for this particular device only.

Honestly, planning to crawl whole page with Lighthouse CLI would be a total waste of time and resources. Take one performance trace from each post type. This will tell you about the high-level issues that implementations of each might have. If one individual post is having something wrong with it (let's say 1MB image), you can pick it up from GSC.

I understand that this is my opinion, but I have been on this rodeo for dozens of products again and again, especially after the announcement of the update. Seriously DevTools will make your life so much easier.

Could you clarify these couple things:
- Are you using vanilla wordpress?
- Is there headless implementation? If so, what framework/bundler? Also how are you doing the code splitting?
- Do you have any dynamic content on page? (Live feed etc.)
- When did the performance sprints end and start?

Let me know.

PS: If you're willing, I can take a quick look at your flamechart. You can send the url via DM, but for obvious reasons I understand if you refuse.

many thanks for solid advice, even though the post did not come from me! may I know what optimization plugins you use on your side? have you done anything special to get ready for the upcoming web vitals?

 

[herps]

@justalazyseo - Thank you for the offer! If you don't mind, I'm going to attempt to try the performance trace per your suggestion first. That's great advice. Strange how I've used this feature before and yet overlooked it.

In terms of 28 day field data...yeah we've been 'in the green' in terms of GPSI since March. So GSC should at least give us 'needs impmrovement' for most pages, but it isn't. Not even one page. I've had several other sites validate much faster significantly with less work. Everything we have done since March has just been really tedious adjustments to shave down latency. For instance, I refactored the menu code to eliminate extra class names and div depth that was unnecessary. That said, I'll see what the performance trace yields!

 

[WorkingNow]

CLS issues can be resolved by adding dimensions to your images and disabling lazyloading on stuff above the fold.

 

[herps]

CLS issues can be resolved by adding dimensions to your images and disabling lazyloading on stuff above the fold.

All of our images have height/width info attached to them for proper aspect ratios. Lazyloading is done below the fold only. Even the logo above the fold is hardcoded and as a SVG. We are getting a score of 0 on lab tests. As justalazyseo mentioned, there could be something else that isn't being picked up by Lighthouse.

 

[justalazyseo]

many thanks for solid advice, even though the post did not come from me! may I know what optimization plugins you use on your side? have you done anything special to get ready for the upcoming web vitals?

For wordpress WP-rocket can be quite a good one when implemented correctly - there are some things that I would take in consideration for QA however:
- Exclude above the fold images from lazyloading if that is included in your setup. Lazyloading above the fold images can easily lead to high LCP score. This is because they are often quite large (hero images and such) and it already takes time to load: request, response, download, layout, paint. If they are being lazyloaded, you can add the script exection into that mix - the main problem there is that when the script starts executing (at FCP pretty much) there is a lot going on on the main thread, let's say building a news feed, which can then queue the script pretty far, inflating LCP. This depends a little bit on exact implementation, but in general rule of thumb I recommend not lazyloading above the fold, at least vanilla way. Workaround can be having a placeholder <div> with fixed dimensions where then jquery injects the whole <img> tag.

- Especially if you're using non-standard WP implementation (jquery building main content, react, webpack), double check that lazyload is actually working. With certain implementations, especially if the page is not cached properly or lazyload is shipped within a large bundle. WP-rocket can actually miss the lazyload script being imporant and totally block it from doing anything so images will load with src tags with first HTML. For vanilla WP, where data-src attribute is most likely hardcoded to tags this is not a problem (except when images are above fold for above mentioned reason). There is a variety of reasons that cause this, but especially I've seen module bundlers like WebPack cause this.

- Where the data-src is hardcoded, WP-rocket can also block Lazyload execution at times, causing LCP go even higher.

In terms of how I am preparing to Web Vitals is not much out of the ordinary, except I am much more zealous about keeping within the performance budget. As a techie, I always pushed for high performance, because especially for slow sites it's a low hanging fruit to improve conversion and engagement (latter being a ranking metric). So lucky me

 

[justalazyseo]

All of our images have height/width info attached to them for proper aspect ratios. Lazyloading is done below the fold only. Even the logo above the fold is hardcoded and as a SVG. We are getting a score of 0 on lab tests. As justalazyseo mentioned, there could be something else that isn't being picked up by Lighthouse.

After shitting on PSI for a little bit in my first message, I have to mention that the lab metrics for CLS are actually fairly accurate - this is because CLS is a very quantifiable metric and doesn't really depend on ISP, connection or client config and hardware excluding screen size ofc.

When you say all images have dimension info attached to them, do you mean the image tags or that sized are intrinsic?

 

[herps]

In terms of how I am preparing to Web Vitals is not much out of the ordinary, except I am much more zealous about keeping within the performance budget. As a techie, I always pushed for high performance, because especially for slow sites it's a low hanging fruit to improve conversion and engagement (latter being a ranking metric). So lucky me

Same! I was a game developer in a past life and optimizing triangles for framerate is a lot like optimizing divs/css for core web vitals. In this case, I was brought onto this project after several jr level devs built it before me.

After shitting on PSI for a little bit in my first message, I have to mention that the lab metrics for CLS are actually fairly accurate - this is because CLS is a very quantifiable metric and doesn't really depend on ISP, connection or client config and hardware excluding screen size ofc.

When you say all images have dimension info attached to them, do you mean the image tags or that sized are intrinsic?

I agree with these statements 100%. However, for some reason we're going beyond the threshold for CLS even though lab tests are showing CLS=0 or near 0 and has been like this since March. This is really where I'm confused about our score vs field data.