1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

view more than 1000 search results.

Discussion in 'BlackHat Lounge' started by outsidethesquare, Jul 10, 2010.

  1. outsidethesquare

    outsidethesquare Regular Member

    Joined:
    Sep 9, 2009
    Messages:
    269
    Likes Received:
    159
    When searching with Google or any popular search engine, you may notice that a search may say something like "About 100,000,000 results" or something of the like.

    However when you set it to display 10 results per page, the last page you can view is page 100. When you set it to display 100 results per page, the last page you can view is 10.

    If you change the URL to go past this, Google will display the message: "Sorry, Google does not serve more than 1000 results for any query. (You asked for results starting from [number here])."

    Does anybody know of a search engine that allows you to view more than 1000 results for a query?
     
  2. risefromdeath

    risefromdeath Power Member

    Joined:
    Jul 1, 2009
    Messages:
    650
    Likes Received:
    107
    You mean to say you can never see 1001th result in google?
     
  3. outsidethesquare

    outsidethesquare Regular Member

    Joined:
    Sep 9, 2009
    Messages:
    269
    Likes Received:
    159
    Nobody can ever see the 1001th result in google. or yahoo, or bing.
     
  4. TheMatrix

    TheMatrix BANNED BANNED

    Joined:
    Dec 20, 2008
    Messages:
    3,444
    Likes Received:
    7,279
    This is going to be really quick & dirty, just a one pager. The problem solved here is when you search for something and you get twenty thousand results back. You know there are valuable results in the results beyond the first thousand, but you can't get to them. Or can you?

    I'm here to show you how to get the best results out of all the twenty thousand, which may very well be different (and more) than the first one thousand Google shows you. Here's how we're going to do it.

    Step 1: Search as usual. We're going to search for ?dog house? (w/o the double quotes). At the time I'm writing this I'm getting 62.5 million results. So, how are we going to get out of the dog house..? :)

    Step 2: On the blue line before the sponsored results, you'll see a button saying ?show options?. Click that. A left column appears, with several selections.

    Step 3: The second group of selections deals with time spans. These time spans refer to the time(s) that Google discovered the page(s) shown in the result page. Click ?Specific Date Range?. A drop down box appears letting you specify the start and end date of the period you're interested in. Today is 7/10/2010, so let's pick the last ten days, 7/1/2010 to 7/10/2010. Click ?search? once you filled out the dates. By the way, the default setting for this selection is ?anytime?, which means Google shows you the best pages discovered since it started searching the web.

    Step 4: Now you'll see the best results discovered by Google during the time span you specified. The top one hundred URLs are very different from the top one hundred shown in the default time period search.

    That's the whole method. Now, why is this useful?
    First of all, you can use this with many of Google's operators, e.g. 'link:', to discover back link creation velocity for a certain page. When narrowed to a specific time period, these operators will show you many more results than they do with the default time period.
     
  5. outsidethesquare

    outsidethesquare Regular Member

    Joined:
    Sep 9, 2009
    Messages:
    269
    Likes Received:
    159
    Interesting idea, however not entirely practical for what I want. I want to harvest the URL of every single result (over 1 million) and I know that more than 1000 were created in single day ranges...

    Hoping there are other search engines out there that display over 1000 results?
     
  6. TheMatrix

    TheMatrix BANNED BANNED

    Joined:
    Dec 20, 2008
    Messages:
    3,444
    Likes Received:
    7,279
    Dude, scrape a thousand KW's then!
     
  7. bbrez1

    bbrez1 Power Member

    Joined:
    Feb 21, 2009
    Messages:
    675
    Likes Received:
    2,360
    1. Scrape all the google suggestions of your keyword
    2. Search all of these suggestions and scrape the results
    3. Win!
     
  8. outsidethesquare

    outsidethesquare Regular Member

    Joined:
    Sep 9, 2009
    Messages:
    269
    Likes Received:
    159
    It's not a keyword lol. I'm doing a site: search and need to scrape the million plus results lol :(
     
  9. bbrez1

    bbrez1 Power Member

    Joined:
    Feb 21, 2009
    Messages:
    675
    Likes Received:
    2,360
    Well you did not state that anywhere so I did not know :)
     
  10. mehdvirus

    mehdvirus Junior Member

    Joined:
    Dec 19, 2008
    Messages:
    117
    Likes Received:
    107
    I want the same thing for example :
    "powered by vbulletin" this bring more than 10 billion results but it only goes as far as page 76 !!
     
  11. pavan

    pavan Elite Member

    Joined:
    Mar 30, 2008
    Messages:
    1,603
    Likes Received:
    432
    in that case
    create a crawler or else you can use some website downloader to crawl the website which you want to scrape :)
     
  12. outsidethesquare

    outsidethesquare Regular Member

    Joined:
    Sep 9, 2009
    Messages:
    269
    Likes Received:
    159
    Yes, I know lol. Thankyou for offering advice though, I still appreciate the effort.
     
  13. flaktrak

    flaktrak Power Member

    Joined:
    Mar 1, 2010
    Messages:
    522
    Likes Received:
    139
    Location:
    South!
    Hope you have alot of ip's to burn!
     
  14. mehdvirus

    mehdvirus Junior Member

    Joined:
    Dec 19, 2008
    Messages:
    117
    Likes Received:
    107
    that's not what he wants !!
    he wants to scrape google results not in a website!!