Screaming Frog and Cloudflare under attack mode

SirLouen

SirLouen

Elite Member
Executive VIP
Jr. VIP
Joined
Jan 17, 2015
Messages
5,263
Reaction score
4,956
In the past few days I've found issues crawling a site with Cloudflare under attack mode as it used to be possible through the Authentication forms, just by passing manually the captcha.

But now it doesn't work anymore.

Here is a video of what I'm trying to explain


I've set up a simple page on https://webdirectoryforum.org to showcase this example, in case anyone would like to check

Has anyone found a way to bypass this crawling issue with Screaming Frog?
 
Advertise on BHW
If the website you want to crawl with Screaming Frog is your own or under your control, you could just disable under attack mode for the duration of the crawl and then re-enable it. But of course if it isn't your site that isn't an option. I've got a few issues with CloudFlare myself on a little scraping project, annoying!
 
Steptoe said:
If the website you want to crawl with Screaming Frog is your own or under your control, you could just disable under attack mode for the duration of the crawl and then re-enable it. But of course if it isn't your site that isn't an option. I've got a few issues with CloudFlare myself on a little scraping project, annoying!
Click to expand...
There are many sites that like to live "under attack" for some reason. It's difficult to keep on changes for such sites without asking to keep activating and removing everytime one wants to use SF.

Anyway I'm sure there should be a method, although maybe Screaming Frog is not the solution because I'm not sure they can manage this task. Maybe I should run old Xenu?
 
Ahh good olde Xenu... but I doubt that would work as CloudFlare's anti-bot is javascript heavy. I'm using scrapeops API at the moment for bypassing CloudFlare anti-bot when I encounter it, though I'm not sure it would be suitable for a full site crawl.
 
Steptoe said:
Ahh good olde Xenu... but I doubt that would work as CloudFlare's anti-bot is javascript heavy. I'm using scrapeops API at the moment for bypassing CloudFlare anti-bot when I encounter it, though I'm not sure it would be suitable for a full site crawl.
Click to expand...
I found this article, from ScrapeOps, very interesting approaches
https://scrapeops.io/web-scraping-p...5-smart-proxy-with-cloudflare-built-in-bypass
I'm not a super expert with Screaming Frog, never specialized enough to be a real pro, but I wondered if I could use a proxy and send some parameters in the scraping mode to use a scraper like FlareSolverr in combination with Screaming Frog.
 
If you are using Screaming Frog to crawl your website and Cloudflare under attack mode has been enabled, you may experience some limitations in terms of the number of requests that can be made to your website. Cloudflare's under attack mode is designed to protect your website against attacks such as DDoS attacks, but it may also restrict the number of requests that can be made by a crawler like Screaming Frog.

To ensure that your crawling activities are not affected by Cloudflare under attack mode, you can try the following steps:

1. Whitelist your IP address in Cloudflare. This will allow you to bypass the security checks applied by Cloudflare and access your website without restrictions.

2. Reduce the number of requests made by Screaming Frog. You can adjust the crawl settings in Screaming Frog to limit the number of requests made, which should help you stay within the limits set by Cloudflare.

3. Disable Cloudflare under attack mode temporarily. If you need to carry out a larger crawl that exceeds the limits set by Cloudflare, you can consider disabling under attack mode temporarily. However, this should only be done if you are confident that your website can handle the increased traffic without compromising on security.
 
Eliteseos said:
If you are using Screaming Frog to crawl your website and Cloudflare under attack mode has been enabled, you may experience some limitations in terms of the number of requests that can be made to your website. Cloudflare's under attack mode is designed to protect your website against attacks such as DDoS attacks, but it may also restrict the number of requests that can be made by a crawler like Screaming Frog.

To ensure that your crawling activities are not affected by Cloudflare under attack mode, you can try the following steps:

1. Whitelist your IP address in Cloudflare. This will allow you to bypass the security checks applied by Cloudflare and access your website without restrictions.

2. Reduce the number of requests made by Screaming Frog. You can adjust the crawl settings in Screaming Frog to limit the number of requests made, which should help you stay within the limits set by Cloudflare.

3. Disable Cloudflare under attack mode temporarily. If you need to carry out a larger crawl that exceeds the limits set by Cloudflare, you can consider disabling under attack mode temporarily. However, this should only be done if you are confident that your website can handle the increased traffic without compromising on security.
Click to expand...
Be sincere. Have you used GPT for this answer?
 
SirLouen said:
In the past few days I've found issues crawling a site with Cloudflare under attack mode as it used to be possible through the Authentication forms, just by passing manually the captcha.

But now it doesn't work anymore.

Here is a video of what I'm trying to explain


I've set up a simple page on https://webdirectoryforum.org to showcase this example, in case anyone would like to check

Has anyone found a way to bypass this crawling issue with Screaming Frog?
Click to expand...
Cloudflare recently added the new turnstile feature to their UAM challenge, screaming frog doesn't have any features to bypass it, you could contact the developers and ask them to implement it.
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top