Help with 403 Forbidden Error When Crawling E-shop for SEO

[katunar]

Hello everyone!

I recently started a position as an SEO Specialist at a company, and I'm hoping for some advice on an issue I'm facing.

When I try to crawl the company's e-shop using Screaming Frog, I keep getting a 403 Forbidden Client Error. I understand this restriction is likely in place for security reasons, but I'm wondering if Screaming Frog is actually problematic for use? My manager's explanation was that the tool might appear as an "attack," potentially increasing bandwidth usage.

Does anyone have suggestions on how I might still gather the necessary data for SEO analysis without causing issues? Thanks in advance!

 

[Steptoe]

You can try throttling your requests in Screaming Frog so that your crawl looks less like an attack. Or you could ask the company's tech team to whitelist a certain IP or useragent you can use for such tasks.

 

[TaMNoNa]

Good luck in your new position!

Have you checked to ensure there are no geo restrictions on your server?

 

[katunar]

Good luck in your new position!

Have you checked to ensure there are no geo restrictions on your server?

there are not. i checked our other website that goes under same server and it was running. They have blocked it on purpose need to talk to them make them trust that screaming frog is legit. Thanks for the info!!

 

[Shazim804]

Good luck in your new position!

Have you checked to ensure there are no geo restrictions on your server?

 

[katunar]

Good luck in your new position!

Have you checked to ensure there are no geo restrictions on your server?

there are not. i checked our other website that goes under same server and it was running. They have blocked it on purpose need to talk to them make them trust that screaming frog is legit. Thanks for the info!!