Any Bulk Meta Robot Scraper

[singhavn]

Is there any tool that can tell me which url/link contains "NoIndex Meta tag" out of thousands of links so I can know that those links cannot be indexed.

I know there are couple of chrome extensions but they can't help in checking Bulk.

I need to analyse thousands of links to find out which one are eligible to index in google.

 

[TomTheCat]

If you have PHP I can make a code and share it here.

 

[iceapple]

A few lines in python should do it.

 

[xfiles999]

A few lines in python should do it.

Would you mind sharing the lines. I am too willing to know that.

 

[iceapple]

import requests
import lxml.html
urls = [
    "https://www.google.com",
    "https://www.shopify.com/blogsearch",
]
for url in urls:
    resp = requests.get(url)
    xml = lxml.html.fromstring(resp.content)
    if xml.xpath('//meta[@name="robots" and contains(@content, "noindex")]'):
        print(f"{url} contains noindex tag!")

Install the following libraries using pip - pip install requests lxml

 

[Steptoe]

Scrapebox's Page Scanner addon will do that, you just need to set up a custom footprint for it eg 'name="robots" content="noindex"'

 

[itz_styx]

import requests
import lxml.html
urls = [
    "https://www.google.com",
    "https://www.shopify.com/blogsearch",
]
for url in urls:
    resp = requests.get(url)
    xml = lxml.html.fromstring(resp.content)
    if xml.xpath('//meta[@name="robots" and contains(@content, "noindex")]'):
        print(f"{url} contains noindex tag!")

Install the following libraries using pip - pip install requests lxml

that would be very slow for thousands or imagine hundreds of thousands of urls.
also whats the default timeout? might hang for a while if a site doesnt respond, which slows down everything even further.

Scrapebox's Page Scanner addon will do that, you just need to set up a custom footprint for it eg 'name="robots" content="noindex"'

thats a better solution as its multithreaded, but needs much more patterns for example if a site contains: content="noindex, noarchive" etc
in the python example at least there is xpath.

 

[ali.tab]

If you are working on one website, you can use an auditor software such as Website Auditor from SEOpowersuite.
but if you are working on many URLs from different domains, Scrapebox is one of the best options.

 

[iceapple]

that would be very slow for thousands or imagine hundreds of thousands of urls.
also whats the default timeout? might hang for a while if a site doesnt respond, which slows down everything even further.



thats a better solution as its multithreaded, but needs much more patterns for example if a site contains: content="noindex, noarchive" etc
in the python example at least there is xpath.

import requests
import lxml.html
import threading

THREADS = 100
TIMEOUT = 10

urls = [
    "https://www.google.com",
    "https://www.shopify.com/blogsearch",
]

def worker():
    while len(urls):
        url = urls.pop()
        resp = requests.get(url, timeout=TIMEOUT)
        xml = lxml.html.fromstring(resp.content)
        if xml.xpath('//meta[@name="robots" and contains(@content, "noindex")]'):
            print(f"{url} contains noindex tag!")
    return None

ths = []
for _ in range(THREADS):
    th = threading.Thread(target=worker)
    th.start()
    ths.append(th)
    
[th.join() for th in ths]

Updated code with threads [100] and timeout [10s], it's better to read URLs from a file and write output to a file.

 

[Kurt angle]

Python code .. you can use with that.

 

[bhuva47]

You can Use Google Sheet

 

[themoon9286]

python

 

[singhavn]

Thanks everyone for your responses. I will definetily try scrapebox addon and python.