1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Tool to extract ALL the indexed URL

Discussion in 'Black Hat SEO Tools' started by sunny155, May 1, 2013.

  1. sunny155

    sunny155 Newbie

    Joined:
    Dec 3, 2012
    Messages:
    4
    Likes Received:
    0
    Hi Guys,

    What tool should I use to extract all the Indexed URL's from Google. Running a query site:domain giving me a lengthy result (1000+)
    So is there any free/paid tool for that?

    Thanks!
     
  2. alaltaierii

    alaltaierii Supreme Member

    Joined:
    Jun 11, 2010
    Messages:
    1,408
    Likes Received:
    349
    Scrapebox. It's $57 from scrapebox.com/bhw
     
  3. monkid

    monkid Power Member

    Joined:
    Jan 2, 2010
    Messages:
    680
    Likes Received:
    310
  4. sunny155

    sunny155 Newbie

    Joined:
    Dec 3, 2012
    Messages:
    4
    Likes Received:
    0
    Thanks all!
     
  5. crazyflx

    crazyflx Elite Member

    Joined:
    Nov 9, 2009
    Messages:
    1,674
    Likes Received:
    4,825
    Location:
    http://CRAZYFLX.COM
    Home Page:
    Actually, while the above answers are similar to what I would say, none of them are actually correct.

    There is no tool that will allow you to see/scrape more than the first 1k search results for any single query.

    For instance, if I were to input the following footprint into scrapebox:

    site:blackhatworld.com (which, when searched, has over 1 million results), I'll never be able to scrape more than the first 1k results for just that one query.

    Google simply will not display more than the first 1k search results.

    However, using scrapebox and the same footprint combined with different keywords, will allow you to get more than 1k results for that same query.

    For instance, each of the following three footprints would scrape 1k individual results, totaling in 3k total results (there are bound to be duplicates of course):

    apples site:blackhatworld.com
    dogs site:blackhatworld.com
    cats site:blackhatworld.com

    With a very large list of keywords, you'll be able to get more than just 1k results, but not with any single query.

    If the site you're trying to scrape the URLs from google with has a sitemap (most wordpress sites do), scrapebox also has a sitemap scraper which will grab every URL on the sitemap in a single scrape.

    So, scrapebox is definitely the way to go, but I thought you should know what you're actually going to be getting should you buy it (which I highly recommend).
     
  6. sunny155

    sunny155 Newbie

    Joined:
    Dec 3, 2012
    Messages:
    4
    Likes Received:
    0
    Thanks for solution. Will refer this post for future project.