Bot for scraping URLs indexed

Hi there!

I was looking for a bot or software that scrapes all the urls of a website that Google has indexed. I know with screaming I can extract all the URLs of a website, but I need to know which of them are indexed. Does someone know any tool? Thanks

Just google this: site:www.yourwebsite.com

All the urls you see are indexed by google. If the url is not indexed then you will not see it

-Sasha

Doesn't ScrapeBox do this?

scrapebox? Just writting site:domain.com on the left side? yourwebsite.com doesnt work dude, have you check it? Thanks guys!

Johan bucco said:

scrapebox? Just writting site:domain.com on the left side? yourwebsite.com doesnt work dude, have you check it? Thanks guys!

Click to expand...

I have tried with scrapebox adding the command site:mydomain.com and scrapebox gives me 676 results and if I writte it on google search bar it gives me 878 results...

Johan bucco said:

I have tried with scrapebox adding the command site:mydomain.com and scrapebox gives me 676 results and if I writte it on google search bar it gives me 878 results...

Click to expand...

If you write in google.com try clicking thru, you will likely see as you get closer to the last page it changes the number. Google does this all the time, they want to post big numbers, so they show you a lot on the first page. But often when you say "ok google give me the results" they change it, sometimes from millions down to hundreds.

At any rate google sees Scrapebox as a browser just like firefox or chrome for example, so you get the same thing in Scrapebox.

So you could do site:domain.com

or crawl the whole site with scrapebox and then check indexed in google. That way you would also know which ones are not indexed so you can try and index them. Just be aware that checking indexed will require a LOT of proxies or a big delay.

You can also use the sitemap scrapper from scrapebox and just check all those pages against the G index.