Does website speed really matter after a certain point?

Extra 100ms don't help much I think. But there are many things to consider. For example, you site may load fast but does it load fast for the first time visitor.

If your website loads in 1-3 seconds, it's pretty fast, and will load quick enough for everyone to not leave your website immediately. 100ms won't do very much.

One thing to consider, if your website is internationally, is to use a CDN like Cloudflare. This way the website loads equally fast for everyone, no matter from which location they are accessing it.

If we are speaking about ranking for example, then yes, pagespeed really does affect your rankings on search engines. One of the biggest reasons that your rankings can be affected by page speeds is that it’s widely accepted that Google’s algorithm considers loading time as an important ranking factor. This was highlighted started back in 2010 when Google made public their regard of page speed as an important issue that webmasters should be focusing on.
There are other indirect ways that pagespeed is affecting your SEO, and this is something that experts have desperately been trying to educate businesses on for a number of years.

Following.