Scraping with cURL? Use Multi cURL to improve performance

Discussion in 'PHP & Perl' started by rjaus, Jan 24, 2011.

Tags:
  1. rjaus

    rjaus Newbie

    Joined:
    Apr 23, 2010
    Messages:
    36
    Likes Received:
    5
    Hey Everyone,

    I get alot of info from these forums and I finally have something to contribute.

    I've done alot of cURL scraping in my time and have never understood how to use or properlly implement cURL Multi Handle.

    And now that I do, I wont ever go back.

    So if you use cURL regularly to do scraping of any kind I suggest you check out this article. It does a great job of explaining how to use cURL Multi, which I havent found else where, and it provides an optimized approach as well.

    I cant post links.... so Google to find it.
    Great info.

    If it helps, please give me some thanks.

    P.S I didnt write the blog post, just sharing this great info.

    P.P.S I am writing up a php proxy checking script using cURL Multi, which Ill share when I'm finished.
     
    • Thanks Thanks x 1
  2. masterwaldo

    masterwaldo Registered Member

    Joined:
    Jul 3, 2008
    Messages:
    95
    Likes Received:
    49
    Is this the link?:
    Code:
    http://www.onlineaspect.com/2009/01/26/how-to-use-curl_multi-without-blocking/
     
    • Thanks Thanks x 1
  3. rjaus

    rjaus Newbie

    Joined:
    Apr 23, 2010
    Messages:
    36
    Likes Received:
    5
    yea thanks.

    Via that link you can also see that this has now been put up on Google code. There you can get a object oriented version called "Rolling Curl".