1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Google search url's SCRAPING HOW TO??

Discussion in 'Black Hat SEO' started by 5thday, Jun 21, 2008.

  1. 5thday

    5thday Newbie

    Joined:
    Jun 20, 2008
    Messages:
    11
    Likes Received:
    0
    hi

    so i need to scrape google search results but i only need links/urls from these results.I am using some search operators like inurl: and i get like 1500 results.Now, what i wanna do is to get urls from these 1500results,no need for anything else, just urls.There must be some simple script or app that can do that...Any ideas?

    thanks for any suggestion:cool:
     
  2. snowwhite

    snowwhite Power Member

    Joined:
    Feb 12, 2008
    Messages:
    730
    Likes Received:
    1,415
    Occupation:
    Psychiatrist
    Location:
    Las Islas Filipinas
    use the expired domain software. It scapes urls, just copy and paste
     
  3. oldenstylehats

    oldenstylehats Elite Member Premium Member

    Joined:
    Apr 10, 2008
    Messages:
    1,893
    Likes Received:
    1,196
    You could use PHP with CURL.

    Check here:
    Code:
    http://www.devnewz.com/devnewz-3-20041221UsingPHPCURLLibrarytoScrapetheInternet.html
     
  4. 5thday

    5thday Newbie

    Joined:
    Jun 20, 2008
    Messages:
    11
    Likes Received:
    0
    hi guys

    thanks for ultrafast response and suggestions .i am pretty noobish when it comes to php and curl so cannot handle this one myself yet:)

    snowwhite, this expired domain software sound like smth i am looking for now but google gives me few "expired domains software" so i don't know which one you have in mind?Please could you clarify?
    this one:
    hxxp://www.expireddomainsleuth.com/
    or this one
    hxxp://www.domainsoftware.org/
    or this one?
    http://hxxp://www.dnware.com/

    thanks again for your help
     
    Last edited: Jun 21, 2008
  5. snowwhite

    snowwhite Power Member

    Joined:
    Feb 12, 2008
    Messages:
    730
    Likes Received:
    1,415
    Occupation:
    Psychiatrist
    Location:
    Las Islas Filipinas
    Code:
    http://rapidshare.com/files/124077482/ExpiredDomainV2.msi.html
     
    • Thanks Thanks x 2
  6. stealthisblog

    stealthisblog Regular Member

    Joined:
    May 26, 2008
    Messages:
    289
    Likes Received:
    238
    Location:
    New York City
    if you're coding it in PHP, just use regex to grab the URL's in the links, its easy. If you need help PM me and ill make it for you if i have the time.
     
  7. 5thday

    5thday Newbie

    Joined:
    Jun 20, 2008
    Messages:
    11
    Likes Received:
    0
    hi again

    thanks a lot snowwhite for the link, i ve just tested it and unfortunately this tool doesn't do exactly what i need.It can scrape google but the output are just domain names like outputdomain.com and not full URL's like outputdomain.com/happyscarping.php which i need.anyway thanks for your time;)

    If anyone have any suggestion i am still looking for some ideas.I can't code myself so i am looking for smth already made.There are some tools i have found but they scrape just max 100 results from google and i need like few thousands.For better understanding just have a look at the link below.Basically i wanna get smth very very similar but with all the results available
    :)

    hxxp://goohackle.com/scripts/google_parser.php

    thank you
     
  8. interpro

    interpro Registered Member Premium Member

    Joined:
    Feb 27, 2008
    Messages:
    92
    Likes Received:
    210
    Home Page:
    Try hxxp://3w.planetsourcecode.c0m

    They have a lot of scripts under the PHP section that can do this sort of thing and they're free

    I recall one script that I think was called 'spider' and it would scrape links or images or whatever you need. Give it a try.
     
  9. 80degreez

    80degreez Newbie

    Joined:
    May 23, 2008
    Messages:
    14
    Likes Received:
    1
    I can code you a URL scraper in Perl, shoot me a PM.
     
  10. Djingel

    Djingel Junior Member

    Joined:
    Apr 9, 2007
    Messages:
    150
    Likes Received:
    84
    I am using Web Scraper Plus+ for this kind of scraping, no code skills needed.
     
  11. doczoidberg

    doczoidberg Junior Member

    Joined:
    Feb 10, 2008
    Messages:
    173
    Likes Received:
    55
    Occupation:
    Student
    Location:
    Germany
    search for serp harvester in download section, there is a free tool
     
  12. timestimes

    timestimes BANNED BANNED

    Joined:
    May 12, 2009
    Messages:
    11
    Likes Received:
    4
    Web Scraper Plus+

    HTML:
    h**p://filestube.com/3db0f942551befea03ea/details.html

    Pass: AdaMskIvelWalenty
     
    Last edited: May 17, 2009
  13. nycdude

    nycdude Jr. VIP Jr. VIP Premium Member

    Joined:
    Oct 1, 2009
    Messages:
    485
    Likes Received:
    560
    Location:
    Mazatlán
    Needs a password to unzip file....

    edit: my bad, the password is right in the post.
     
    Last edited: Apr 26, 2011
  14. zenmoog

    zenmoog Newbie

    Joined:
    Dec 26, 2009
    Messages:
    22
    Likes Received:
    6
    that download is impossible to install, missing volume (001)