Crawl data website

mrtonysmith

Newbie
Joined
Jan 7, 2025
Messages
16
Reaction score
10
in my robot txt file is disallowing files containing css, js, font, style because it will waste resources if let bot crawl these data.

when I check on semrush, I am notified that there is a lack of crawl data from css, js files

So should I delete it in the robot file
 
Lol Bro, you basically told Google to put on a blindfold and guess what your site looks like. Blocking CSS and JS in robots.txt isn’t “saving resources,” it’s like turning the lights off and expecting the bot to describe your house.
 
Yes, remove the ban — Google must see CSS and JS, otherwise it will render the site as broken. This is not saving resources, but harming SEO.
For now it is like OF without photo
 
This is a very common issue, but it's important to keep those files crawlable. While it might seem like a waste of resources, search engines like Google need to crawl CSS and JS files to fully render and understand your website's content and layout. Blocking them can prevent Google from seeing your site as a user would, which could negatively impact your rankings. Therefore, you should remove the disallow rules for these file types from your robots.txt file.
 
Back
Top