1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Scrape ALL urls from google - Not only 60 pages!

Discussion in 'Black Hat SEO' started by Kebabman, Jul 29, 2013.

  1. Kebabman

    Kebabman Newbie

    Joined:
    Mar 21, 2013
    Messages:
    31
    Likes Received:
    4
    hello,

    how can i scrape ALL urls from google serps?
    I am able to scrape 60 pages from google serps with zennoposter.
    After that no more results show up.

    So i am stuck at 60 x 10 = 600 urls.

    What can i do?

    Thank you ☺
     
  2. satyr85

    satyr85 Power Member

    Joined:
    Aug 7, 2011
    Messages:
    580
    Likes Received:
    444
    Location:
    Poland
    Zennoposter is not tool for scraping G. There are many other tools for scraping like gscraper, scrapebox, hrefer. I use zenno for 2 years and belive me its not way to go.

    Google dont show all pages when you search for keyword. Google show only part of them so lets say you want to scrape wordpress blogs and you search for "powered by wordpress" you will get 600 results.

    When you search for "powered by wordpress" + X where X is random keyword from list of 1000 most popular english words you will get 1000x600 results. Some will be duplicates but there will be alot of unique domains. Hope this will help you.
     
    • Thanks Thanks x 1
  3. seoways

    seoways Jr. VIP Jr. VIP Premium Member UnGagged Attendee

    Joined:
    Dec 19, 2009
    Messages:
    4,650
    Likes Received:
    724
    Location:
    Behind you!
    try with scrapbox & SEnuke.. may get you more result page URLs..
     
    • Thanks Thanks x 1
  4. Kebabman

    Kebabman Newbie

    Joined:
    Mar 21, 2013
    Messages:
    31
    Likes Received:
    4
    Your workarround works fine! :cheer2:
     
  5. seohq

    seohq Newbie

    Joined:
    Jul 30, 2013
    Messages:
    6
    Likes Received:
    1
    Location:
    New Orlean (LA)
    You can do that by using additional workd, for example you want to scrape all links for keyword "blackhatworld", scrape like this:
    blackhatworld aab
    blackhatworld jxna
    blackhatworld ogox
    blackhatworld uxuxu
    blackhatworld aabb

    Create 1-2K like this and you'll scrape more for same keyword.
     
  6. donaldbeck

    donaldbeck Power Member

    Joined:
    Dec 28, 2006
    Messages:
    585
    Likes Received:
    212
    This is the right idea, but you would probably be better served by using normal words instead of random characters.
     
  7. Skyebug77

    Skyebug77 Jr. VIP Jr. VIP Premium Member

    Joined:
    Mar 22, 2012
    Messages:
    1,469
    Likes Received:
    1,022
    Occupation:
    Marketing
    Location:
    Portland,Or
    What are you trying to accomplish? Are you looking for sites to post too for scripting on zenno poster or what exactly is it that you want to achieve.
     
  8. cindylove

    cindylove Newbie

    Joined:
    Nov 23, 2012
    Messages:
    29
    Likes Received:
    1
    try with scrapbox
     
  9. andishm

    andishm Regular Member Premium Member

    Joined:
    Jul 21, 2011
    Messages:
    362
    Likes Received:
    52
    Location:
    SEO World
    For your information for any search results in Google there can be 2 situations:
    1. Either Google show 100 pages 10 with each until you have custom settings set. However it will show you 2 millions results but never show you all 2 millions 1000 results max and this is how i see Google since years and years. So until Google show them you cannot crawl those 2 millions.

    2. Many times you search a keyword but google dont even show 1000 results why so because of too many similar results with duplicate content etc this is what called "search results omitted" language used by google.

    While with scrapebox for eg you can scrape 1000 and you can use variety of keywords with good footprint to scrape more sites but one thing remember no software can give you list of 2 million results shown in any keyword search as google donot show more than 1000 at a time :)
     
  10. cooltoad

    cooltoad Senior Member

    Joined:
    Sep 10, 2010
    Messages:
    934
    Likes Received:
    549
    Occupation:
    None of your business
    Location:
    On Vacation
    Man Zenno is mnay things, but using it for hardcore SE scraping, BIG No No!
    With repeated updates, your bot will break all the time. For just $57, get yourself Scrapebox and scrape the shit out of Google.

    cheers
    CT
     
  11. Orbit143

    Orbit143 Senior Member

    Joined:
    Aug 8, 2010
    Messages:
    893
    Likes Received:
    588
    Location:
    /home
    scrape some more keywords from wonder wheel or something, then scrape urls and remove duplicates.
     
  12. gullsinn

    gullsinn Jr. VIP Jr. VIP Premium Member

    Joined:
    Dec 24, 2009
    Messages:
    2,429
    Likes Received:
    2,210
    Gender:
    Male
    Occupation:
    Jobless :D
    Location:
    Graveyard
    Home Page:
    Scrapebox, Scrapeboard and XRM can do this job mate.