• On Wednesday, 19th February between 10:00 and 11:00 UTC, the forum will go down for maintenance. Read More

Can't crawl the entire site in Screaming Frog, why?

cockus

Regular Member
Joined
May 6, 2017
Messages
486
Reaction score
31
I want to crawl a site that has over 100 pages, but screaming frog only displays 8 pages in the crawling report. It is a gambling site with a geolocation block. I have tried to use a VPN but it still does not allow crawling. The site is indexed on Google.

Screaming frog says
http//domain.com - status code 301 - moved permanently redirected - non-indexable
https://domain.com - status code 301 - moved permanently redirected - non-indexable
https://www.domain.com - status code 200 - indexable
+ 5 ther random JS pages

Any idea of how to allow Screaming Frog to bypass this block and allow it to crawl the site?
 
They have probably blocked screaming frog crawler
 
Back
Top