1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Posting with cURL - hidden fields

Discussion in 'General Programming Chat' started by Packers, Apr 7, 2011.

  1. Packers

    Packers Registered Member

    Joined:
    Jan 31, 2011
    Messages:
    77
    Likes Received:
    7
    This isn't specific to php.

    Looking at the yahoo.com page for signing into an email, I checked the source and noticed that a lot of other fields are posted along side the username and password. If I was to make a bot that used cURL (or anything else for that matter, that doesnt explicitly load the webpage up) would I have to post all those hidden fields too? The obvious way would be to code it so that I can pick up all the fields that ARE posted, and then repost them when I post a username and password. Wondering whether this is needed though!

    Cheers!
     
  2. kaidoristm

    kaidoristm Power Member

    Joined:
    Feb 13, 2009
    Messages:
    561
    Likes Received:
    726
    Occupation:
    Freelancer
    Location:
    Estonia
    Home Page:
    Ok at some point i made an as universal form scraper as i could.

    $from is your scraped page just pass entire page to that function
    $name is the form name where to grab all names and values
    $remove you can pass an array to remove specific values

    this function will return you array where elements are as name=value or null if nothing has found.

    PHP:
    function get_vars($from,$name,$remove=array())
    {
        
    $pair = array();
        
    $replacement = array('/\s\s+/','/\v/');
        
    $from preg_replace($replacement,' '$from);

        
    preg_match('|<form[^<]*'.$name.'[^<]*>(.*?)</form>|'$from$form);
        if(isset(
    $form[1]))
        {
            
    preg_match_all('|<input(.*?)\>|'$form[1], $inputs);    
            if(isset(
    $inputs[1]))
            {    
                foreach(
    $inputs[1] as $key => $value)
                {
                    
    preg_match('|name=[\'"](.*?)[\'"]|'$value$inpn);
                    
    preg_match('|value=[\'"](.*?)[\'"]|'$value$inpv);
                    
                    if(isset(
    $inpn[1]))
                    {

                        if(!
    in_array($inpn[1], $remove))
                        {
                            if(!
    $inpv[1])
                            {
                                
    $pair[]= $inpn[1]."=";
                            }
                            else
                            {
                                
    $pair[]= $inpn[1]."=".$inpv[1];
                            }
                        }
                    }
                }
                return 
    $pair;
            }
        }
        
        else
        {
            return 
    null;
        }
    }
    Example usage as well (save it an do an test run it will gather all names and values from yahoo login page)

    PHP:
    function curl_it($url)
    {
        
    $timeout 10;
        
    $useragent "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.16) Gecko/20110319 Firefox/3.6.16";
        
    $cookiefile tempnam('tmp/','COO');

        
    $ch curl_init();
        
    curl_setopt($chCURLOPT_URL$url);
        
    curl_setopt($chCURLOPT_USERAGENT$useragent);
        
    curl_setopt($chCURLOPT_TIMEOUT$timeout);
        
    curl_setopt($chCURLOPT_AUTOREFERERtrue); 
        
    curl_setopt($chCURLOPT_COOKIEJAR$cookiefile);
        
    curl_setopt($chCURLOPT_COOKIEFILE$cookiefile);
        
    curl_setopt($chCURLOPT_SSL_VERIFYHOSTfalse);
        
    curl_setopt($chCURLOPT_SSL_VERIFYPEERfalse);
        
    curl_setopt($chCURLOPT_HEADERfalse);
        
    curl_setopt($chCURLOPT_FOLLOWLOCATIONtrue);
        
    curl_setopt($chCURLOPT_RETURNTRANSFERtrue);

        
    $output curl_exec($ch);
        
    $status curl_getinfo($chCURLINFO_HTTP_CODE);

        
    $dir 'tmp/';
        foreach (
    glob($dir.''.'COO*') as $filename)
        {
           
    "$filename size ".filesize($filename)."\n";
           
    unlink($filename);
        }
        
        return 
    $output;
    }
    $data curl_it("https://login.yahoo.com/");

    if(
    $data)
    {
        
    print_r(get_vars($data,"login_form",$remove=array()));
    }
     
    • Thanks Thanks x 1
    Last edited: Apr 7, 2011
  3. Stalli0n

    Stalli0n Junior Member

    Joined:
    Nov 17, 2010
    Messages:
    115
    Likes Received:
    83
    Location:
    Europe
    Use LiveHttpHeaders for Firefox or Wireshark for example and look at the POST Request.

    You have to send the same fields with Curl wheter their hidden or not.

    I don't know what kaidoristm posted but you can extract them easily with preg_match
     
  4. artizhay

    artizhay BANNED BANNED

    Joined:
    Nov 21, 2010
    Messages:
    1,867
    Likes Received:
    1,335
    Yes, you will have to POST the hidden fields too. I haven't looked at the Yahoo sign-in HTML source, but see if the hidden fields change values when you refresh the page. Often, md5 hashes and other non-static information are in hidden fields, so Yahoo may not like the data you POST if it doesn't match what it's expecting. However, give it a try and see how it goes.
     
  5. Stalli0n

    Stalli0n Junior Member

    Joined:
    Nov 17, 2010
    Messages:
    115
    Likes Received:
    83
    Location:
    Europe
    Code:
    https://login.yahoo.com/config/login?
    
    POST /config/login? HTTP/1.1
    Host: login.yahoo.com
    User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; de; rv:1.9.2.16) Gecko/20110319 Firefox/3.6.16
    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
    Accept-Language: de-de,de;q=0.8,en-us;q=0.5,en;q=0.3
    Accept-Encoding: gzip,deflate
    Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
    Keep-Alive: 115
    Connection: keep-alive
    Referer: https://login.yahoo.com/config/login?.src=fpctx&.intl=us&.done=http%3A%2F%2Fwww.yahoo.com%2F
    Cookie: B=61eka916prg2m&b=3&s=15; CH=AgBNnggQACT5EAAs8BAAFXUQAAPCEAAGfhAAKNgQACAkEAAlQBAAL3sQAAOU; HP=1
    Content-Type: application/x-www-form-urlencoded
    Content-Length: 357
    .tries=1&.src=fpctx&.md5=&.hash=&.js=&.last=&promo=&.intl=us&.bypass=&.partner=&.u=625pdvh6ps22o&.v=0&.challenge=ZMVzKQn9lB0nhJOA9DF7YRWFwoy7&.yplus=&.emailCode=&pkg=&stepid=&.ev=&hasMsgr=0&.chkP=Y&.done=http%3A%2F%2Fwww.yahoo.com%2F&.pd=fpctx_ver%3D0%26c%3D%26ivt%3D%26sg%3D&pad=5&aad=5&login=stalli0n%40blackhatworld.com&passwd=blackhat&.save=&passwd_raw=
    HTTP/1.1 200 OK
    Date: Thu, 07 Apr 2011 18:54:40 GMT
    P3P: policyref="http://info.yahoo.com/w3c/p3p.xml", CP="CAO DSP COR CUR ADM DEV TAI PSA PSD IVAi IVDi CONi TELo OTPi OUR DELi SAMi OTRi UNRi PUBi IND PHY ONL UNI PUR FIN COM NAV INT DEM CNT STA POL HEA PRE LOC GOV"
    X-Frame-Options: DENY
    Cache-Control: private
    Connection: close
    Transfer-Encoding: chunked
    Content-Type: text/html
    Content-Encoding: gzip
    
    Looks like most of them are empty so shouldn't be much work ;)
     
  6. Packers

    Packers Registered Member

    Joined:
    Jan 31, 2011
    Messages:
    77
    Likes Received:
    7
    thanks guys!

    wish I could just get my head around regex. Then I can start fully writing my own scripts :) Getting there gradually, such messy syntax!
     
  7. xpwizard

    xpwizard Junior Member

    Joined:
    Nov 6, 2010
    Messages:
    198
    Likes Received:
    122
    If your new to regex, then try the following sites (I can't post urls):

    txt2re (*) c*m ---> Useful for basic regex generation. Note that it's good for static values, but not so good if your trying to find a variable.

    gskinner (*) c*m (forwardslash) RegExr ---> Useful to test your regex statements in an online environment.
     
  8. gimme4free

    gimme4free Executive VIP Jr. VIP Premium Member

    Joined:
    Oct 22, 2008
    Messages:
    1,884
    Likes Received:
    1,932
    Specific to hidden fields:
    PHP:
    function getHiddenFields($html){
        
    $ret = array();
        if (
    preg_match_all('#(<input[^>]*type="hidden"[^>]*>)#is'$html$matches)){
            
    $hfields $matches[1];
            foreach (
    $hfields as $hf) {
                if (
    preg_match('#name="([^"]+)"#is'$hf$matches)){
                    
    $hf_name $matches[1];
                    
    $hf_val '';
                    if (
    preg_match('#value="([^"]+)"#is'$hf$matches)){
                        
    $hf_val $matches[1];
                        }
                    
    $ret[] = urlencode(htmlspecialchars_decode($hf_name)).'='.urlencode(htmlspecialchars_decode($hf_val));
                    }
                }
            }
        return 
    $ret;
        }
    Yahoo login script I made for another thread (working):
    PHP:
    <?php
    $curl_defaults 
    = array(
        
    CURLOPT_RETURNTRANSFER => true
        
    CURLOPT_HEADER => 0,
        
    CURLOPT_USERAGENT => 'Mozilla/5.0 (Windows NT 5.1; rv:2.0b11) Gecko/20100101 Firefox/4.0b11',
        
    CURLOPT_FOLLOWLOCATION => 1,
        
    CURLOPT_AUTOREFERER => 1,
        
    CURLOPT_RETURNTRANSFER => 1,
        
    CURLOPT_CONNECTTIMEOUT => 5,
        
    CURLOPT_TIMEOUT => 20,
        
    CURLOPT_VERBOSE => 0,
        
    CURLOPT_SSL_VERIFYHOST => 0,
        
    CURLOPT_SSL_VERIFYPEER => 0
        
    );
    function 
    Return_Content_From_URL($url,$cookies){
        global 
    $curl_defaults;
        
    $ch curl_init();
        
    curl_setopt_array($ch$curl_defaults);
        
    curl_setopt($chCURLOPT_URL,$url);
        
    curl_setopt($chCURLOPT_COOKIEJAR$cookies.'.txt');
        
    curl_setopt($chCURLOPT_COOKIEFILE$cookies.'.txt');
        
    $htmlcurl_exec($ch);
        
    curl_close($ch);
        return 
    $html;
        }
    function 
    Post_And_Return($url,$cookies,$data,$referrer){
        global 
    $curl_defaults;
        
    $ch curl_init();
        
    curl_setopt_array($ch$curl_defaults);
        
    curl_setopt($chCURLOPT_URL,$url);
        
    curl_setopt($chCURLOPT_REFERER$referrer);
        
    curl_setopt($chCURLOPT_POSTFIELDS,$data);
        
    curl_setopt($chCURLOPT_COOKIEJAR$cookies.'.txt');
        
    curl_setopt($chCURLOPT_COOKIEFILE$cookies.'.txt');
        
    $html curl_exec($ch);
        
    curl_close($ch);
        return 
    $html;
        }
    function 
    getHiddenFields($html){
        
    $ret = array();
        if (
    preg_match_all('#(<input[^>]*type="hidden"[^>]*>)#is'$html$matches)){
            
    $hfields $matches[1];
            foreach (
    $hfields as $hf) {
                if (
    preg_match('#name="([^"]+)"#is'$hf$matches)){
                    
    $hf_name $matches[1];
                    
    $hf_val '';
                    if (
    preg_match('#value="([^"]+)"#is'$hf$matches)){
                        
    $hf_val $matches[1];
                        }
                    
    $ret[] = urlencode(htmlspecialchars_decode($hf_name)).'='.urlencode(htmlspecialchars_decode($hf_val));
                    }
                }
            }
        return 
    $ret;
        }
    // Set Cookies File
    $cookies "yahoo";
    // Set Account Details
    $email "email@email.com";
    $password "password";
    // Return Content From Login Page
    $url "https://login.yahoo.com/config/login?";
    $content Return_Content_From_URL($url,$cookies);
    // Scrape Form Values
    $fields getHiddenFields($content);
    // Join Hidden Fields
    $data join('&'$fields);
    // Form Post Data
    $data.="&login=".$email;
    $data.="&passwd=".$password;
    // Post Login Info
    $url "https://login.yahoo.com/config/login?";
    $referrer "http://my.yahoo.com/";
    $content Post_And_Return($url,$cookies,$data,$referrer);
    // Go To My Yahoo
    $url "http://mail.yahoo.com/";
    $content Return_Content_From_URL($url,$cookies);
    echo 
    "<textarea>".htmlentities($content)."</textarea>";
    ?> 
     
    • Thanks Thanks x 1
  9. maximviper

    maximviper BANNED BANNED

    Joined:
    Oct 25, 2010
    Messages:
    338
    Likes Received:
    86

    Always helpful. i wish i cud REP+ you :)
    There you go exact yahoo login code for OP .
     
  10. The Doctor

    The Doctor Jr. VIP Jr. VIP

    Joined:
    Dec 18, 2010
    Messages:
    612
    Likes Received:
    188
    Occupation:
    Computer Scientist, Engineer, Programmer.
    Location:
    ☆☆☆☆☆☆
    Do you have any written for sending an email?