How to reduce initial server response time?

I'm using WP rocket and Perfmatters plugin. Now what should I do to reduce this problem?

Sometimes you can not do more, it's all about your hosting. Go with a better hosting.

80% of the time it's hosting. If you use a plan where they share the ressources (e.g. CPU cores, RAM etc.) across users, then you will sometimes get less allocation which will cause lag in server response time. There is also a possibility that the code of the plugin is bad, but I think that's rarely the case with wordpress scritps as they are field tested usually.

It's already enough to pay 5-10 bucks per month for hosting to have good initial response time

Before diving into how to improve your server response time (or TTFB), we need to define what causes that issue in the first place...

1. Slow application logic
2. Slow database queries
3. Slow routing
4. Frameworks
5. Libraries
6. CPU resource starvation
7. Memory starvation
8. Slow disk speed
9. High server traffic

...these signs are usually due to a lousy server or a server not capable of supporting the level of traffic your site is getting.

So, based on that, this is what you need to do in order to improve your server TTFB...

• Use a Fast Web Host - Duh, right?
  • Rocket.net, WPX Hosting, and Cloudways Vultr High-Frequency plan are at the top of my list
• Use Caching - This will only help with recurrent visitors though.
  • WP Rocket would be my top choice here
  • Also, make sure to enable caching at a database level
  • As a rule of thumb, don´t clear the server cache that often.
  • Also, set an expired header policy of 1 year for all the static resources
• Use a CDN - This is close to mandatory unless you´re targeting a local market
  • Cloudflare will do the trick - CDNPerf
• If you´re on WordPress, make sure to control the heartbeat settings
  • WP Rocket can do this for you as well!
• Optimize your Database - Caching it will also do wonders!
  • WP Rocket can help with this as well
  • Reducing your database queries will also come a long way for you
  • When it comes to caching database queries, look into Redis Object Cache (free plugin)
• Use a Premium DNS Service
  • According to DNSPerf, Cloudflares offers the fastest DNS on the market
• Compress Your Site Files -This is why minification is mandatory as well
  • WP Rocket can help you with this as well
  • Make sure your hosting offers HTTP/2 protocol
• Keep Your System Updated
  • Themes
  • Plugins
  • PHP
• Pick your themes and plugins with ease!
  • Stay away from page builders - Tools like Oxygen, Breakdance, Bricks, and even Gutenberg should be your top priority
  • When it comes to themes, GeneratePress, Astra, OceanWP, Neve, and Kadence are at the top of the list
• Delay the execution of non-critical third-party scripts - THIS IS IMPORTANT!
  • Stuff like instant messaging apps, heat maps, analytics, etc., should NOT interfere with your site´s loading speed.

I hope this helps, cheers!!