Scraping with cURL? Use Multi cURL to improve performance

Hey Everyone,

I get alot of info from these forums and I finally have something to contribute.

I've done alot of cURL scraping in my time and have never understood how to use or properlly implement cURL Multi Handle.

And now that I do, I wont ever go back.

So if you use cURL regularly to do scraping of any kind I suggest you check out this article. It does a great job of explaining how to use cURL Multi, which I havent found else where, and it provides an optimized approach as well.

I cant post links.... so Google to find it.

how-to-use-curl_multi-without-blocking

Click to expand...

Great info.

If it helps, please give me some thanks.

P.S I didnt write the blog post, just sharing this great info.

P.P.S I am writing up a php proxy checking script using cURL Multi, which Ill share when I'm finished.

Is this the link?:

yea thanks.

Via that link you can also see that this has now been put up on Google code. There you can get a object oriented version called "Rolling Curl".