How to bulk check if links are indexable (using scrapebox)

[Krise656]

Hi there,

as indexing services that work are a bit expensive for each generated link, I would like to check which links are indexable and which not.
I have seen a thread here explaining to do this with scrapebox - but I did not really understand how.
Anyone may please help.

 

[Ramazan]

Just check robots.txt and robots meta tag for understand indexable or not. Python should do this easily for bulk.

 

[netizenwizard]

Hi there,

as indexing services that work are a bit expensive for each generated link, I would like to check which links are indexable and which not.
I have seen a thread here explaining to do this with scrapebox - but I did not really understand how.
Anyone may please help.

First things first, make sure you've got the latest version of Scrapebox. Then, load up your list of links and go to the 'Page Scanner' section. From there, select 'Indexability Test' and let it rip!
Scrapebox will then go through your list and check each link to see if it's indexable or not.

 

[Krise656]

First things first, make sure you've got the latest version of Scrapebox. Then, load up your list of links and go to the 'Page Scanner' section. From there, select 'Indexability Test' and let it rip!
Scrapebox will then go through your list and check each link to see if it's indexable or not.

Thanks a lot for your answer. When I open Page Scanner (64bit) the only possible check I can perform is platforms.msk. Do I need a special footprint thatfore. Genuine SB Version.

 

[loopline]

indexability is not a platform built into the page scanner, unless I have missed it somewhere. You would have to build that test.

What data are you looking for specifically? like what on the html page or in the robots file, for you, determines if its indexable for you or not?

 

[Steptoe]

I would think just 'noindex' as a mask would cover the majority of issues.

 

[Krise656]

indexability is not a platform built into the page scanner, unless I have missed it somewhere. You would have to build that test.

What data are you looking for specifically? like what on the html page or in the robots file, for you, determines if its indexable for you or not?

I would think just 'noindex' as a mask would cover the majority of issues.

Yes, just to check if "noindex"

 

[kayz9x]

Check file Robots.txt and Meta tag noindex.

 

[Krise656]

Check file Robots.txt and Meta tag noindex.

I am lokking for a method to bulk check. Checking 2.000 links manually is a bit time consuming

 

[loopline]

Sure, then you could use the page scanner addon to check the domain.com for the no index tag and then also check the robots.txt file for the noindex option. You need to build that but its pretty easy, here is a video

 

[Krise656]

Sure, then you could use the page scanner addon to check the domain.com for the no index tag and then also check the robots.txt file for the noindex option. You need to build that but its pretty easy, here is a video

Sorry for my late answer, just saw today you wrote.
I think I managed it. Just made a footprint "noindex" and another one "nofollow"
Thanks a lot @loopline

 

[Craigmazin]

I usually use robots txt for checking the link index

 

[Krise656]

I usually use robots txt for checking the link index

Can you do this in bulk, like 25.000 links?

 

[Craigmazin]

Can you do this in bulk, like 25.000 links?

Unfortunately, no. But it works well for checking specific links.

 

[Array Experts]

Have you tried screaming frog for the same task?

 

[Krise656]

Unfortunately, no. But it works well for checking specific links.

Have you tried screaming frog for the same task?

I wanted, and works now, Test thousands of Link from GSA SER.

Have Not tried screaming frog, but will Test If it can handle this amount.

 

[loopline]

Can you do this in bulk, like 25.000 links?

Yes, you could do this with scrapebox. Compared to what scrapebox can handle, 25,000 links is not really bulk.

The scrapebox page scanner 64bit could handle functionally, over 100 million urls at a time. Now you would need a monster machine for that, and I would recommend working in chunks of 500K urls to 1 million urls and test what your machine can handle first.

 

[Krise656]

Yes, you could do this with scrapebox. Compared to what scrapebox can handle, 25,000 links is not really bulk.

The scrapebox page scanner 64bit could handle functionally, over 100 million urls at a time. Now you would need a monster machine for that, and I would recommend working in chunks of 500K urls to 1 million urls and test what your machine can handle first.

I use scrapebox only for some tasks, but I know that it is one of or the mightiest tool in our business.
Never tried 1 Million links, but 25.000 to 100.000 links is what I am testing after getting them from SER.
I would love to learn more about the tool and I will do, but this takes time
@loopline: I am always using your great YT channel if I have questions - thanks thatfore!!!