Ecommerce site is ranking but site is block google crawler

I have been doing competitor analysis and noticed that one of our competitors is ranking very well despite having a low Domain Rating (DR).

I ran their site through Screaming Frog, but it didn't work, showing response codes: Internal Client Error (4xx). I used an extension to simulate different crawlers, including Googlebot, and found that the site is not accessible by these crawlers.

Does anyone know about this type of strategy? If you search "shrooms online (Canada)," the second-ranking e-commerce site is the one I'm talking about.

Thank you.

I think your competitor is using a strategy of "cloaking". i.e., website can detect the user agent of the crawler and serve a different version of the page or block it entirely. This would explain why you're seeing a "4xx" error when using Screaming Frog or other crawler simulations.

jaerehan said:

I think your competitor is using a strategy of "cloaking". i.e., website can detect the user agent of the crawler and serve a different version of the page or block it entirely. This would explain why you're seeing a "4xx" error when using Screaming Frog or other crawler simulations.

Click to expand...

aside form cloaking , do you think they do other server-side measures ?

do you have a recommended cloaking software?

potluck_hoe said:

do you have a recommended cloaking software?

Click to expand...

NordVPN, TorGuard, Windscribe, Surfshark