Can Screaming Frog crawl a website by its sitemap.xml ? Cache warming task.

[Impaler]

Hello, can anyone answer if and HOW can Screaming Frog be used to crawl a website via its sitemap.xml and visit each and every link in it once. I imagine a page loading every few seconds with a cronjob or something like that.

The deal is that I need quite a huge website. I mean tens of thousands of pages to be viewed and opened once so their images would get cached by the hosting server.

It's more of a cache warming situation. I don't analysis or anything.

If this program is not suitable can you recommend me another one? It could run on Windows. Thanks.

 

[hideath]

This can be easily done by python,
You can use requests and visit x set of URLs each few seconds to avoid getting the server overloaded

 

[Impaler]

Sadly no ide ahow to use python :/

 

[Roger Marquez]

Sadly no ide ahow to use python :/

No need to know or learn Python (although you should) when you can use chatGPT

 

[blackbeans]

No need to know or learn Python (although you should) when you can use chatGPT

Can you share a Youtube video on people using CGPT to get Python capabilities in crawling online content?

That would be awesome!

 

[joecal]

you can do it with Screaming frog without any issues. Choose "List mode" and then in Settings provide the XML sitemaps you want to crawl. You can also do it with a command line in terminal, but you need to prepare and save the config file prior to that. If you want to crawl images or CSS / JS assets, you can tick the corresponding boxes in Crawl Settings. Also, you can in Limits tab you can choose "Limit: 0" to make sure that Screaming Frog won't jump around various URLs, and take only the URLs in the sitemap that you provide.