Hello,
Do you guys know a webtool or a software (free or paid) that can scrape 10 000 webpages (different domains) and generate an Excel file with :
Robots.txt is present or not. if yes, does the webpage is allow for googlebot
x-robots-tag HTTP header is present ? if yes does is it nofollow, follow, noindex, index ?
Meta robots is follow, noindex, nofollow or index ?
canonical tag is present ? if yes does the canonical tag = the URL of the webpage ?
canonical HTTP header is present ? If yes does the canonical = the URL of the webpage ?
Thank you very much for your help.
Do you guys know a webtool or a software (free or paid) that can scrape 10 000 webpages (different domains) and generate an Excel file with :
Robots.txt is present or not. if yes, does the webpage is allow for googlebot
x-robots-tag HTTP header is present ? if yes does is it nofollow, follow, noindex, index ?
Meta robots is follow, noindex, nofollow or index ?
canonical tag is present ? if yes does the canonical tag = the URL of the webpage ?
canonical HTTP header is present ? If yes does the canonical = the URL of the webpage ?
Thank you very much for your help.