Hi,
I am working on new website and need to know what are the important technical error should be optimized.
So, i want to listen from you that what kind of technical errors you check on your website? Waiting for your replies.
The first thing I check on a new site that "should be ranking but isn't" isn't robots.txt, it's the HTTP headers. curl -I https://site.com takes five seconds.
I've found three sites this year where a dev set X-Robots-Tag: noindex on staging, the site went live, nobody removed that one line in .htaccess or nginx, and Google silently obeyed it for weeks — sometimes months. The site loaded fine for humans. Sitemaps submitted. Pages looked perfect in a browser. Zero traffic. The problem lived in a header you can't see unless you know to look for it.
Here's the part that makes me twitchy: sometimes curl -I lies to you. If you're behind Cloudflare or any CDN that injects headers, the noindex might only appear for Googlebot's user-agent. I watched someone chase this for three days — GSC reported noindex, every curl check came back clean, Screaming Frog said everything was fine. The CDN had a rule that only fired X-Robots-Tag when the visitor looked like Google. You need curl -I -H "User-Agent: Googlebot" to catch it.
And if you're on Next.js or React: check your shared layout components. I've seen a single <meta name="robots" content="noindex"> in a layout file meant for staging propagate to every page in production. Nobody noticed because they were spot-checking individual route files, not the shared wrapper. One component. Entire site invisible.
Quick bonus: don't trust site:yourdomain.com as a health check. It's notoriously incomplete and hides orphans. And a sitemap submission isn't the same as internal links — Google treats sitemap-only pages as second-class citizens if nothing internal points to them.
If you do nothing else: curl -I with the Googlebot UA, then check your layout components. That's thirty seconds that's saved me more time than any audit tool.