TUTORIAL: cUrl for newbs

bruce6667

Junior Member
Joined
Apr 6, 2008
Messages
136
Reaction score
106
What's cUrl?

cUrl is a set of functions for php that emulate a browser. You can use it to do things like scraping and spamming. Basically, cUrl lets you automate and repeat any web action.

Basic example of screen scraping

Code:
<?php

$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, 'http://www.google.com');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

echo curl_exec($ch);

curl_close($ch);

?>

So there are four stages:
// 1. start it up
$ch = curl_init();
// 2. set some options
curl_setopt($ch, CURLOPT_URL, 'http://www.google.com');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
// 3. execute it... you don't have to echo
echo curl_exec($ch);
// 4. close it
curl_close($ch);

Most Common Options

CURLOPT_URL -> the url you want to GET/POST to
CURLOPT_RETURNTRANSFER -> return it instead of printing it
CURLOPT_REFERER -> you can fake the referer if you want
CURLOPT_USERAGENT -> you can fake this as well
CURLOPT_FOLLOWLOCATION -> follow any redirects
CURLOPT_POST -> you can do POSTs with cUrl, example: submitting a form
CURLOPT_POSTFIELDS -> you have to tell it what to post

Examples of Most Common Options

Code:
curl_setopt($ch, CURLOPT_URL, 'http://blackhatworld.com')
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true)
curl_setopt($ch, CURLOPT_REFERER, 'http://gmail.com')
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla')
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true)
curl_setopt($ch, CURLOPT_POST, true)
curl_setopt($ch, CURLOPT_POSTFIELDS, 'boo=radley&atticus=finch')

Sometimes its easier to set POSTFIELDS in an array.

Code:
$postfields = array(
	'boo' => 'radley',
	'do' => 'why not',
	'name' => 'bruce6667',
);

curl_setopt($ch, CURLOPT_POSTFIELDS, $posfields)

Limitations... cUrl doesn't do javascript or flash. That's about it.
 
Well .. I guess I havea suggestion that really made my CURL apps a lot more easier to implement.

I downloaded HTTP Analyzer and you can submit,search,login whatever you need your program to do and HA will record for you post data & cookies that you can copy paste into your curl apps.

It's a great great great help to view the exact data your browser is sending and emulate it accordingly in php.

I even played around with flash games that send data back to server .. And you know .. there are some games that give prizes for the best player ?? lol :) I don't ever pay for electronic gadgets .. I just win them :))
 
very interesting gabixx I will have to do a little searching, you might have some competition in the near future >=)
 
hehehe .... there's enough for everybody .. not afraid really .. bring it on! :D
 
Thanks for that bruce! This is a list of some good firefox extensions that I like to use to get up close and personal with the HTTP protocol and automate browser tasks. Playing with these helped me learn a lot about http and once you know how everything works it makes it a ton easier to implement it in code along the lines of what Gabixxx was saying.

chickenfoot
Code:
http://groups.csail.mit.edu/uid/chickenfoot/

web developer plugin
Code:
https://addons.mozilla.org/en-US/firefox/addon/60

live http headers
Code:
http://livehttpheaders.mozdev.org/

firebug
Code:
https://addons.mozilla.org/en-US/firefox/addon/1843
 
Well .. I guess I havea suggestion that really made my CURL apps a lot more easier to implement.

I downloaded HTTP Analyzer and you can submit,search,login whatever you need your program to do and HA will record for you post data & cookies that you can copy paste into your curl apps.

It's a great great great help to view the exact data your browser is sending and emulate it accordingly in php.

I even played around with flash games that send data back to server .. And you know .. there are some games that give prizes for the best player ?? lol :) I don't ever pay for electronic gadgets .. I just win them :))

Wow that is a great idea. So basically you are just recording the post made back to the server, then modifying the point value so that you are the high scoring individual?

Do most games like this not hash the score or something similar to prevent this?

Man I have too many projects :-p
 
Wow that is a great idea. So basically you are just recording the post made back to the server, then modifying the point value so that you are the high scoring individual?

Do most games like this not hash the score or something similar to prevent this?

Man I have too many projects :-p

Haha the first time I ever did that, I did it with that "Punch The Monkey" game, which used GET String variables. I hadn't had a computer for more than six months, and I had a bunch of those silly stuffed monkeys with boxing gloves :D
 
if you are having problems with javascript you can also use the rhino engine from command line with something like shell_exec to figure out what is going on on the sites end even if the js is encoded.

over and out, t.
 
also something else to note is that in non multipart forms you are able to use a string packet

for instance a valid packet to login to joe shmoes shitty non multipart form website would be

$packet='username=john&password=owiejgoiw&auth_token=923840234&login=Yes';

but if the form was multipart you would have to do something along these lines

$packet=array('username'=>john,'password'=>'oiwegjoiweg','auth_token'=>4223023,'login'=>'Yes');

then of course

curl_setopt($ch,CURLOPT_POSTFIELDS,$packet);
$s=curl_exec($ch); echo($s);
 
Curl is great, I toyed with it a bit when I was using it to automate filling out online contest forms (kind of like the guy from Real Genius).
 
Hi! I have some problem with cUrl.


PHP:
$c = curl_init();

curl_setopt($c, CURLOPT_URL, 'http://www.site.com/img.php?id=25582512');
curl_setopt($c, CURLOPT_HEADER, 1);
curl_setopt($c, CURLOPT_VERBOSE, 1);
curl_setopt($c, CURLOPT_RETURNTRANSFER,1);
curl_setopt($c, CURLOPT_USERAGENT, 'Mozilla/5.0 (X11; U; Linux i686;pl; rv:1.8.0.3) Gecko/20060426 Firefox/1.5.0.3');
curl_setopt($c, CURLOPT_ENCODING, 'gzip');
curl_setopt($c, CURLOPT_ENCODING, 'deflate');
curl_setopt($c, CURLOPT_ENCODING, '');
curl_setopt($c, CURLOPT_COOKIEJAR, 'cookies.txt');
curl_setopt($c, CURLOPT_COOKIEFILE, 'cookies.txt');
curl_setopt($c, CURLOPT_FOLLOWLOCATION, 1);
$page = curl_exec($c);
Everything in this code is OK, but there is a problem with site which i want to connect.
When i put this
Code:
http://www.site.com/img.php?id=25582512
into a browser it redirect me to http://www.site.com/img.php?id=25582512&q=&abc=3140088334&si=aHR0cDovL2ltZzEyNy5ob3RsaW5raW1hZ2UuY29tL2ltZy5waHA%2FaWQ9ODQ4NjMxNzI%3D
The pink part's in this url are always different. So i can't connect with site corectly. Do you have any idea how to pass it?
 
Back
Top