1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

TUTORIAL: cUrl for newbs

Discussion in 'PHP & Perl' started by bruce6667, Aug 28, 2008.

  1. bruce6667

    bruce6667 Junior Member Premium Member

    Joined:
    Apr 6, 2008
    Messages:
    136
    Likes Received:
    104
    What's cUrl?

    cUrl is a set of functions for php that emulate a browser. You can use it to do things like scraping and spamming. Basically, cUrl lets you automate and repeat any web action.

    Basic example of screen scraping

    Code:
    <?php
    
    $ch = curl_init();
    
    curl_setopt($ch, CURLOPT_URL, 'http://www.google.com');
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    
    echo curl_exec($ch);
    
    curl_close($ch);
    
    ?>
    So there are four stages:
    // 1. start it up
    $ch = curl_init();
    // 2. set some options
    curl_setopt($ch, CURLOPT_URL, 'http://www.google.com');
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    // 3. execute it... you don't have to echo
    echo curl_exec($ch);
    // 4. close it
    curl_close($ch);

    Most Common Options

    CURLOPT_URL -> the url you want to GET/POST to
    CURLOPT_RETURNTRANSFER -> return it instead of printing it
    CURLOPT_REFERER -> you can fake the referer if you want
    CURLOPT_USERAGENT -> you can fake this as well
    CURLOPT_FOLLOWLOCATION -> follow any redirects
    CURLOPT_POST -> you can do POSTs with cUrl, example: submitting a form
    CURLOPT_POSTFIELDS -> you have to tell it what to post

    Examples of Most Common Options

    Code:
    curl_setopt($ch, CURLOPT_URL, 'http://blackhatworld.com')
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true)
    curl_setopt($ch, CURLOPT_REFERER, 'http://gmail.com')
    curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla')
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true)
    curl_setopt($ch, CURLOPT_POST, true)
    curl_setopt($ch, CURLOPT_POSTFIELDS, 'boo=radley&atticus=finch')
    Sometimes its easier to set POSTFIELDS in an array.

    Code:
    $postfields = array(
    	'boo' => 'radley',
    	'do' => 'why not',
    	'name' => 'bruce6667',
    );
    
    curl_setopt($ch, CURLOPT_POSTFIELDS, $posfields)
    Limitations... cUrl doesn't do javascript or flash. That's about it.
     
    • Thanks Thanks x 9
  2. gabixxx

    gabixxx Junior Member

    Joined:
    Dec 23, 2007
    Messages:
    135
    Likes Received:
    70
    Well .. I guess I havea suggestion that really made my CURL apps a lot more easier to implement.

    I downloaded HTTP Analyzer and you can submit,search,login whatever you need your program to do and HA will record for you post data & cookies that you can copy paste into your curl apps.

    It's a great great great help to view the exact data your browser is sending and emulate it accordingly in php.

    I even played around with flash games that send data back to server .. And you know .. there are some games that give prizes for the best player ?? lol :) I don't ever pay for electronic gadgets .. I just win them :))
     
  3. syns0r

    syns0r Junior Member

    Joined:
    Mar 31, 2007
    Messages:
    141
    Likes Received:
    31
    very interesting gabixx I will have to do a little searching, you might have some competition in the near future >=)
     
  4. gabixxx

    gabixxx Junior Member

    Joined:
    Dec 23, 2007
    Messages:
    135
    Likes Received:
    70
    hehehe .... there's enough for everybody .. not afraid really .. bring it on! :D
     
  5. blazed

    blazed Junior Member

    Joined:
    Aug 15, 2008
    Messages:
    178
    Likes Received:
    119
    Thanks for that bruce! This is a list of some good firefox extensions that I like to use to get up close and personal with the HTTP protocol and automate browser tasks. Playing with these helped me learn a lot about http and once you know how everything works it makes it a ton easier to implement it in code along the lines of what Gabixxx was saying.

    chickenfoot
    Code:
    http://groups.csail.mit.edu/uid/chickenfoot/
    
    web developer plugin
    Code:
    https://addons.mozilla.org/en-US/firefox/addon/60
    
    live http headers
    Code:
    http://livehttpheaders.mozdev.org/
    
    firebug
    Code:
    https://addons.mozilla.org/en-US/firefox/addon/1843
    
     
  6. Dev-Gank

    Dev-Gank Regular Member

    Joined:
    Jul 17, 2008
    Messages:
    216
    Likes Received:
    48
    Wow that is a great idea. So basically you are just recording the post made back to the server, then modifying the point value so that you are the high scoring individual?

    Do most games like this not hash the score or something similar to prevent this?

    Man I have too many projects :p
     
  7. goawayplease

    goawayplease Regular Member

    Joined:
    Apr 10, 2008
    Messages:
    299
    Likes Received:
    67
    Haha the first time I ever did that, I did it with that "Punch The Monkey" game, which used GET String variables. I hadn't had a computer for more than six months, and I had a bunch of those silly stuffed monkeys with boxing gloves :D
     
  8. taky

    taky BANNED BANNED

    Joined:
    Jul 15, 2007
    Messages:
    92
    Likes Received:
    12
    if you are having problems with javascript you can also use the rhino engine from command line with something like shell_exec to figure out what is going on on the sites end even if the js is encoded.

    over and out, t.
     
  9. taky

    taky BANNED BANNED

    Joined:
    Jul 15, 2007
    Messages:
    92
    Likes Received:
    12
    also something else to note is that in non multipart forms you are able to use a string packet

    for instance a valid packet to login to joe shmoes shitty non multipart form website would be

    $packet='username=john&password=owiejgoiw&auth_token=923840234&login=Yes';

    but if the form was multipart you would have to do something along these lines

    $packet=array('username'=>john,'password'=>'oiwegjoiweg','auth_token'=>4223023,'login'=>'Yes');

    then of course

    curl_setopt($ch,CURLOPT_POSTFIELDS,$packet);
    $s=curl_exec($ch); echo($s);
     
  10. dvdcowboy

    dvdcowboy Junior Member

    Joined:
    May 10, 2008
    Messages:
    188
    Likes Received:
    39
    Curl is great, I toyed with it a bit when I was using it to automate filling out online contest forms (kind of like the guy from Real Genius).
     
  11. cyklotrial

    cyklotrial Regular Member

    Joined:
    Oct 13, 2008
    Messages:
    248
    Likes Received:
    82
    Location:
    Wonderland
    Hi! I have some problem with cUrl.


    PHP:
    $c curl_init();

    curl_setopt($cCURLOPT_URL'http://www.site.com/img.php?id=25582512');
    curl_setopt($cCURLOPT_HEADER1);
    curl_setopt($cCURLOPT_VERBOSE1);
    curl_setopt($cCURLOPT_RETURNTRANSFER,1);
    curl_setopt($cCURLOPT_USERAGENT'Mozilla/5.0 (X11; U; Linux i686;pl; rv:1.8.0.3) Gecko/20060426 Firefox/1.5.0.3');
    curl_setopt($cCURLOPT_ENCODING'gzip');
    curl_setopt($cCURLOPT_ENCODING'deflate');
    curl_setopt($cCURLOPT_ENCODING'');
    curl_setopt($cCURLOPT_COOKIEJAR'cookies.txt');
    curl_setopt($cCURLOPT_COOKIEFILE'cookies.txt');
    curl_setopt($cCURLOPT_FOLLOWLOCATION1);
    $page curl_exec($c);
    Everything in this code is OK, but there is a problem with site which i want to connect.
    When i put this
    Code:
    http://www.site.com/img.php?id=25582512
    into a browser it redirect me to http://www.site.com/img.php?id=25582512&q=&abc=3140088334&si=aHR0cDovL2ltZzEyNy5ob3RsaW5raW1hZ2UuY29tL2ltZy5waHA%2FaWQ9ODQ4NjMxNzI%3D
    The pink part's in this url are always different. So i can't connect with site corectly. Do you have any idea how to pass it?