1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Scraping with cURL? Use Multi cURL to improve performance

Discussion in 'PHP & Perl' started by rjaus, Jan 24, 2011.

Tags:
  1. rjaus

    rjaus Newbie

    Joined:
    Apr 23, 2010
    Messages:
    35
    Likes Received:
    4
    Hey Everyone,

    I get alot of info from these forums and I finally have something to contribute.

    I've done alot of cURL scraping in my time and have never understood how to use or properlly implement cURL Multi Handle.

    And now that I do, I wont ever go back.

    So if you use cURL regularly to do scraping of any kind I suggest you check out this article. It does a great job of explaining how to use cURL Multi, which I havent found else where, and it provides an optimized approach as well.

    I cant post links.... so Google to find it.
    Great info.

    If it helps, please give me some thanks.

    P.S I didnt write the blog post, just sharing this great info.

    P.P.S I am writing up a php proxy checking script using cURL Multi, which Ill share when I'm finished.
     
    • Thanks Thanks x 1
  2. masterwaldo

    masterwaldo Registered Member

    Joined:
    Jul 3, 2008
    Messages:
    95
    Likes Received:
    49
    Is this the link?:
    Code:
    http://www.onlineaspect.com/2009/01/26/how-to-use-curl_multi-without-blocking/
     
    • Thanks Thanks x 1
  3. rjaus

    rjaus Newbie

    Joined:
    Apr 23, 2010
    Messages:
    35
    Likes Received:
    4
    yea thanks.

    Via that link you can also see that this has now been put up on Google code. There you can get a object oriented version called "Rolling Curl".