Scrape ALL urls from google - Not only 60 pages!

[Kebabman]

hello,

how can i scrape ALL urls from google serps?
I am able to scrape 60 pages from google serps with zennoposter.
After that no more results show up.

So i am stuck at 60 x 10 = 600 urls.

What can i do?

Thank you ☺

 

[satyr85]

Zennoposter is not tool for scraping G. There are many other tools for scraping like gscraper, scrapebox, hrefer. I use zenno for 2 years and belive me its not way to go.

Google dont show all pages when you search for keyword. Google show only part of them so lets say you want to scrape wordpress blogs and you search for "powered by wordpress" you will get 600 results.

When you search for "powered by wordpress" + X where X is random keyword from list of 1000 most popular english words you will get 1000x600 results. Some will be duplicates but there will be alot of unique domains. Hope this will help you.

 

[seoways]

try with scrapbox & SEnuke.. may get you more result page URLs..

 

[Kebabman]

Your workarround works fine! :cheer2:

 

[seohq]

You can do that by using additional workd, for example you want to scrape all links for keyword "blackhatworld", scrape like this:
blackhatworld aab
blackhatworld jxna
blackhatworld ogox
blackhatworld uxuxu
blackhatworld aabb

Create 1-2K like this and you'll scrape more for same keyword.

 

[donaldbeck]

You can do that by using additional workd, for example you want to scrape all links for keyword "blackhatworld", scrape like this:
blackhatworld aab
blackhatworld jxna
blackhatworld ogox
blackhatworld uxuxu
blackhatworld aabb

Create 1-2K like this and you'll scrape more for same keyword.

This is the right idea, but you would probably be better served by using normal words instead of random characters.

 

[DunDidIt2X]

What are you trying to accomplish? Are you looking for sites to post too for scripting on zenno poster or what exactly is it that you want to achieve.

 

[cindylove]

try with scrapbox

 

[andishm]

hello,

how can i scrape ALL urls from google serps?
I am able to scrape 60 pages from google serps with zennoposter.
After that no more results show up.

So i am stuck at 60 x 10 = 600 urls.

What can i do?

Thank you ☺

For your information for any search results in Google there can be 2 situations:
1. Either Google show 100 pages 10 with each until you have custom settings set. However it will show you 2 millions results but never show you all 2 millions 1000 results max and this is how i see Google since years and years. So until Google show them you cannot crawl those 2 millions.

2. Many times you search a keyword but google dont even show 1000 results why so because of too many similar results with duplicate content etc this is what called "search results omitted" language used by google.

While with scrapebox for eg you can scrape 1000 and you can use variety of keywords with good footprint to scrape more sites but one thing remember no software can give you list of 2 million results shown in any keyword search as google donot show more than 1000 at a time

 

[cooltoad]

Man Zenno is mnay things, but using it for hardcore SE scraping, BIG No No!
With repeated updates, your bot will break all the time. For just $57, get yourself Scrapebox and scrape the shit out of Google.

cheers
CT

 

[Orbit143]

scrape some more keywords from wonder wheel or something, then scrape urls and remove duplicates.

 

[gullsinn]

Scrapebox, Scrapeboard and XRM can do this job mate.