cockus
Regular Member
- May 6, 2017
- 486
- 31
I want to crawl a site that has over 100 pages, but screaming frog only displays 8 pages in the crawling report. It is a gambling site with a geolocation block. I have tried to use a VPN but it still does not allow crawling. The site is indexed on Google.
Screaming frog says
http//domain.com - status code 301 - moved permanently redirected - non-indexable
https://domain.com - status code 301 - moved permanently redirected - non-indexable
https://www.domain.com - status code 200 - indexable
+ 5 ther random JS pages
Any idea of how to allow Screaming Frog to bypass this block and allow it to crawl the site?
Screaming frog says
http//domain.com - status code 301 - moved permanently redirected - non-indexable
https://domain.com - status code 301 - moved permanently redirected - non-indexable
https://www.domain.com - status code 200 - indexable
+ 5 ther random JS pages
Any idea of how to allow Screaming Frog to bypass this block and allow it to crawl the site?