Hi there,
I am using Ezoic and as you may know, the system generates a lot of files. To optimize my crawl budget, I did this in my robots.txt :
Disallow: /beardeddragon/*
Disallow: /porpoiseant/*
Disallow: /detroitchicago/*
Disallow: /tardisrocinante/*
I assume that folders will change from a website to another, but you can look at those into your crawl logs or using developers tools on your browser.
Those are the folders that Ezoic creates and uses to serve his files. I guess those are not useful for Google neither rendering. Most of them are JS or images files.
What do you think ? Is it a good thing to block Google to crawl those files that seems to be dynamic + not really used by Google ?
Thanks a lot
I am using Ezoic and as you may know, the system generates a lot of files. To optimize my crawl budget, I did this in my robots.txt :
Disallow: /beardeddragon/*
Disallow: /porpoiseant/*
Disallow: /detroitchicago/*
Disallow: /tardisrocinante/*
I assume that folders will change from a website to another, but you can look at those into your crawl logs or using developers tools on your browser.
Those are the folders that Ezoic creates and uses to serve his files. I guess those are not useful for Google neither rendering. Most of them are JS or images files.
What do you think ? Is it a good thing to block Google to crawl those files that seems to be dynamic + not really used by Google ?
Thanks a lot