Finding results in google index (potentially scraping)

Hi Guys,

I'm trying to find a way to scrape google index to search for sites that are blocked by robot.txt or aren't allowing google to index the sites.

i.e. generally in google search results the home page will actually have a "description" in the index that says "No information is available for this page."

IS there any way to search for sites that are .net sites that are blocked.

You might want to read this: https://www.scrapehero.com/how-to-prevent-getting-blacklisted-while-scraping/ and the reddit article linked below the article.

Thanks not quite what I was looking for.