Posting with cURL - hidden fields

Packers · Apr 7, 2011

This isn't specific to php.

Looking at the yahoo.com page for signing into an email, I checked the source and noticed that a lot of other fields are posted along side the username and password. If I was to make a bot that used cURL (or anything else for that matter, that doesnt explicitly load the webpage up) would I have to post all those hidden fields too? The obvious way would be to code it so that I can pick up all the fields that ARE posted, and then repost them when I post a username and password. Wondering whether this is needed though!

Cheers!

kaidoristm · Apr 7, 2011

Ok at some point i made an as universal form scraper as i could.

$from is your scraped page just pass entire page to that function
$name is the form name where to grab all names and values
$remove you can pass an array to remove specific values

this function will return you array where elements are as name=value or null if nothing has found.

PHP:

function get_vars($from,$name,$remove=array())
{
    $pair = array();
    $replacement = array('/\s\s+/','/\v/');
    $from = preg_replace($replacement,' ', $from);

    preg_match('|<form[^<]*'.$name.'[^<]*>(.*?)</form>|', $from, $form);
    if(isset($form[1]))
    {
        preg_match_all('|<input(.*?)\>|', $form[1], $inputs);    
        if(isset($inputs[1]))
        {    
            foreach($inputs[1] as $key => $value)
            {
                preg_match('|name=[\'"](.*?)[\'"]|', $value, $inpn);
                preg_match('|value=[\'"](.*?)[\'"]|', $value, $inpv);
                
                if(isset($inpn[1]))
                {

                    if(!in_array($inpn[1], $remove))
                    {
                        if(!$inpv[1])
                        {
                            $pair[]= $inpn[1]."=";
                        }
                        else
                        {
                            $pair[]= $inpn[1]."=".$inpv[1];
                        }
                    }
                }
            }
            return $pair;
        }
    }
    
    else
    {
        return null;
    }
}

Example usage as well (save it an do an test run it will gather all names and values from yahoo login page)

PHP:

function curl_it($url)
{
    $timeout = 10;
    $useragent = "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.16) Gecko/20110319 Firefox/3.6.16";
    $cookiefile = tempnam('tmp/','COO');

    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, $url);
    curl_setopt($ch, CURLOPT_USERAGENT, $useragent);
    curl_setopt($ch, CURLOPT_TIMEOUT, $timeout);
    curl_setopt($ch, CURLOPT_AUTOREFERER, true); 
    curl_setopt($ch, CURLOPT_COOKIEJAR, $cookiefile);
    curl_setopt($ch, CURLOPT_COOKIEFILE, $cookiefile);
    curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
    curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
    curl_setopt($ch, CURLOPT_HEADER, false);
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

    $output = curl_exec($ch);
    $status = curl_getinfo($ch, CURLINFO_HTTP_CODE);

    $dir = 'tmp/';
    foreach (glob($dir.''.'COO*') as $filename)
    {
       "$filename size ".filesize($filename)."\n";
       unlink($filename);
    }
    
    return $output;
}
$data = curl_it("https://login.yahoo.com/");

if($data)
{
    print_r(get_vars($data,"login_form",$remove=array()));
}

Stalli0n · Apr 7, 2011

Use LiveHttpHeaders for Firefox or Wireshark for example and look at the POST Request.

You have to send the same fields with Curl wheter their hidden or not.

I don't know what kaidoristm posted but you can extract them easily with preg_match

artizhay · Apr 7, 2011

Yes, you will have to POST the hidden fields too. I haven't looked at the Yahoo sign-in HTML source, but see if the hidden fields change values when you refresh the page. Often, md5 hashes and other non-static information are in hidden fields, so Yahoo may not like the data you POST if it doesn't match what it's expecting. However, give it a try and see how it goes.

Stalli0n · Apr 7, 2011

Code:

https://login.yahoo.com/config/login?

POST /config/login? HTTP/1.1
Host: login.yahoo.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; de; rv:1.9.2.16) Gecko/20110319 Firefox/3.6.16
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: de-de,de;q=0.8,en-us;q=0.5,en;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Connection: keep-alive
Referer: https://login.yahoo.com/config/login?.src=fpctx&.intl=us&.done=http%3A%2F%2Fwww.yahoo.com%2F
Cookie: B=61eka916prg2m&b=3&s=15; CH=AgBNnggQACT5EAAs8BAAFXUQAAPCEAAGfhAAKNgQACAkEAAlQBAAL3sQAAOU; HP=1
Content-Type: application/x-www-form-urlencoded
Content-Length: 357
.tries=1&.src=fpctx&.md5=&.hash=&.js=&.last=&promo=&.intl=us&.bypass=&.partner=&.u=625pdvh6ps22o&.v=0&.challenge=ZMVzKQn9lB0nhJOA9DF7YRWFwoy7&.yplus=&.emailCode=&pkg=&stepid=&.ev=&hasMsgr=0&.chkP=Y&.done=http%3A%2F%2Fwww.yahoo.com%2F&.pd=fpctx_ver%3D0%26c%3D%26ivt%3D%26sg%3D&pad=5&aad=5&login=stalli0n%40blackhatworld.com&passwd=blackhat&.save=&passwd_raw=
HTTP/1.1 200 OK
Date: Thu, 07 Apr 2011 18:54:40 GMT
P3P: policyref="http://info.yahoo.com/w3c/p3p.xml", CP="CAO DSP COR CUR ADM DEV TAI PSA PSD IVAi IVDi CONi TELo OTPi OUR DELi SAMi OTRi UNRi PUBi IND PHY ONL UNI PUR FIN COM NAV INT DEM CNT STA POL HEA PRE LOC GOV"
X-Frame-Options: DENY
Cache-Control: private
Connection: close
Transfer-Encoding: chunked
Content-Type: text/html
Content-Encoding: gzip

Looks like most of them are empty so shouldn't be much work

Packers · Apr 7, 2011

thanks guys!

wish I could just get my head around regex. Then I can start fully writing my own scripts

Getting there gradually, such messy syntax!

xpwizard · Apr 8, 2011

If your new to regex, then try the following sites (I can't post urls):

txt2re (*) c*m ---> Useful for basic regex generation. Note that it's good for static values, but not so good if your trying to find a variable.

gskinner (*) c*m (forwardslash) RegExr ---> Useful to test your regex statements in an online environment.

gimme4free · Apr 8, 2011

Specific to hidden fields:

PHP:

function getHiddenFields($html){
	$ret = array();
	if (preg_match_all('#(<input[^>]*type="hidden"[^>]*>)#is', $html, $matches)){
		$hfields = $matches[1];
		foreach ($hfields as $hf) {
			if (preg_match('#name="([^"]+)"#is', $hf, $matches)){
				$hf_name = $matches[1];
				$hf_val = '';
				if (preg_match('#value="([^"]+)"#is', $hf, $matches)){
					$hf_val = $matches[1];
					}
				$ret[] = urlencode(htmlspecialchars_decode($hf_name)).'='.urlencode(htmlspecialchars_decode($hf_val));
				}
			}
		}
	return $ret;
	}

Yahoo login script I made for another thread (working):

PHP:

<?php
$curl_defaults = array(
    CURLOPT_RETURNTRANSFER => true, 
    CURLOPT_HEADER => 0,
    CURLOPT_USERAGENT => 'Mozilla/5.0 (Windows NT 5.1; rv:2.0b11) Gecko/20100101 Firefox/4.0b11',
    CURLOPT_FOLLOWLOCATION => 1,
    CURLOPT_AUTOREFERER => 1,
    CURLOPT_RETURNTRANSFER => 1,
    CURLOPT_CONNECTTIMEOUT => 5,
    CURLOPT_TIMEOUT => 20,
    CURLOPT_VERBOSE => 0,
    CURLOPT_SSL_VERIFYHOST => 0,
    CURLOPT_SSL_VERIFYPEER => 0
    );
function Return_Content_From_URL($url,$cookies){
    global $curl_defaults;
    $ch = curl_init();
    curl_setopt_array($ch, $curl_defaults);
    curl_setopt($ch, CURLOPT_URL,$url);
    curl_setopt($ch, CURLOPT_COOKIEJAR, $cookies.'.txt');
    curl_setopt($ch, CURLOPT_COOKIEFILE, $cookies.'.txt');
    $html= curl_exec($ch);
    curl_close($ch);
    return $html;
    }
function Post_And_Return($url,$cookies,$data,$referrer){
    global $curl_defaults;
    $ch = curl_init();
    curl_setopt_array($ch, $curl_defaults);
    curl_setopt($ch, CURLOPT_URL,$url);
    curl_setopt($ch, CURLOPT_REFERER, $referrer);
    curl_setopt($ch, CURLOPT_POSTFIELDS,$data);
    curl_setopt($ch, CURLOPT_COOKIEJAR, $cookies.'.txt');
    curl_setopt($ch, CURLOPT_COOKIEFILE, $cookies.'.txt');
    $html = curl_exec($ch);
    curl_close($ch);
    return $html;
    }
function getHiddenFields($html){
    $ret = array();
    if (preg_match_all('#(<input[^>]*type="hidden"[^>]*>)#is', $html, $matches)){
        $hfields = $matches[1];
        foreach ($hfields as $hf) {
            if (preg_match('#name="([^"]+)"#is', $hf, $matches)){
                $hf_name = $matches[1];
                $hf_val = '';
                if (preg_match('#value="([^"]+)"#is', $hf, $matches)){
                    $hf_val = $matches[1];
                    }
                $ret[] = urlencode(htmlspecialchars_decode($hf_name)).'='.urlencode(htmlspecialchars_decode($hf_val));
                }
            }
        }
    return $ret;
    }
// Set Cookies File
$cookies = "yahoo";
// Set Account Details
$email = "[email protected]";
$password = "password";
// Return Content From Login Page
$url = "https://login.yahoo.com/config/login?";
$content = Return_Content_From_URL($url,$cookies);
// Scrape Form Values
$fields = getHiddenFields($content);
// Join Hidden Fields
$data = join('&', $fields);
// Form Post Data
$data.="&login=".$email;
$data.="&passwd=".$password;
// Post Login Info
$url = "https://login.yahoo.com/config/login?";
$referrer = "http://my.yahoo.com/";
$content = Post_And_Return($url,$cookies,$data,$referrer);
// Go To My Yahoo
$url = "http://mail.yahoo.com/";
$content = Return_Content_From_URL($url,$cookies);
echo "<textarea>".htmlentities($content)."</textarea>";
?>

maximviper · Apr 8, 2011

gimme4free said:

Specific to hidden fields:

PHP:

function getHiddenFields($html){
    $ret = array();
    if (preg_match_all('#(<input[^>]*type="hidden"[^>]*>)#is', $html, $matches)){
        $hfields = $matches[1];
        foreach ($hfields as $hf) {
            if (preg_match('#name="([^"]+)"#is', $hf, $matches)){
                $hf_name = $matches[1];
                $hf_val = '';
                if (preg_match('#value="([^"]+)"#is', $hf, $matches)){
                    $hf_val = $matches[1];
                    }
                $ret[] = urlencode(htmlspecialchars_decode($hf_name)).'='.urlencode(htmlspecialchars_decode($hf_val));
                }
            }
        }
    return $ret;
    }

Yahoo login script I made for another thread (working):

PHP:

<?php
$curl_defaults = array(
    CURLOPT_RETURNTRANSFER => true, 
    CURLOPT_HEADER => 0,
    CURLOPT_USERAGENT => 'Mozilla/5.0 (Windows NT 5.1; rv:2.0b11) Gecko/20100101 Firefox/4.0b11',
    CURLOPT_FOLLOWLOCATION => 1,
    CURLOPT_AUTOREFERER => 1,
    CURLOPT_RETURNTRANSFER => 1,
    CURLOPT_CONNECTTIMEOUT => 5,
    CURLOPT_TIMEOUT => 20,
    CURLOPT_VERBOSE => 0,
    CURLOPT_SSL_VERIFYHOST => 0,
    CURLOPT_SSL_VERIFYPEER => 0
    );
function Return_Content_From_URL($url,$cookies){
    global $curl_defaults;
    $ch = curl_init();
    curl_setopt_array($ch, $curl_defaults);
    curl_setopt($ch, CURLOPT_URL,$url);
    curl_setopt($ch, CURLOPT_COOKIEJAR, $cookies.'.txt');
    curl_setopt($ch, CURLOPT_COOKIEFILE, $cookies.'.txt');
    $html= curl_exec($ch);
    curl_close($ch);
    return $html;
    }
function Post_And_Return($url,$cookies,$data,$referrer){
    global $curl_defaults;
    $ch = curl_init();
    curl_setopt_array($ch, $curl_defaults);
    curl_setopt($ch, CURLOPT_URL,$url);
    curl_setopt($ch, CURLOPT_REFERER, $referrer);
    curl_setopt($ch, CURLOPT_POSTFIELDS,$data);
    curl_setopt($ch, CURLOPT_COOKIEJAR, $cookies.'.txt');
    curl_setopt($ch, CURLOPT_COOKIEFILE, $cookies.'.txt');
    $html = curl_exec($ch);
    curl_close($ch);
    return $html;
    }
function getHiddenFields($html){
    $ret = array();
    if (preg_match_all('#(<input[^>]*type="hidden"[^>]*>)#is', $html, $matches)){
        $hfields = $matches[1];
        foreach ($hfields as $hf) {
            if (preg_match('#name="([^"]+)"#is', $hf, $matches)){
                $hf_name = $matches[1];
                $hf_val = '';
                if (preg_match('#value="([^"]+)"#is', $hf, $matches)){
                    $hf_val = $matches[1];
                    }
                $ret[] = urlencode(htmlspecialchars_decode($hf_name)).'='.urlencode(htmlspecialchars_decode($hf_val));
                }
            }
        }
    return $ret;
    }
// Set Cookies File
$cookies = "yahoo";
// Set Account Details
$email = "[email protected]";
$password = "password";
// Return Content From Login Page
$url = "https://login.yahoo.com/config/login?";
$content = Return_Content_From_URL($url,$cookies);
// Scrape Form Values
$fields = getHiddenFields($content);
// Join Hidden Fields
$data = join('&', $fields);
// Form Post Data
$data.="&login=".$email;
$data.="&passwd=".$password;
// Post Login Info
$url = "https://login.yahoo.com/config/login?";
$referrer = "http://my.yahoo.com/";
$content = Post_And_Return($url,$cookies,$data,$referrer);
// Go To My Yahoo
$url = "http://mail.yahoo.com/";
$content = Return_Content_From_URL($url,$cookies);
echo "<textarea>".htmlentities($content)."</textarea>";
?>

Always helpful. i wish i cud REP+ you

There you go exact yahoo login code for OP .

The Doctor · Apr 8, 2011

Do you have any written for sending an email?

Posting with cURL - hidden fields

Packers

Registered Member

kaidoristm

BANNED

Stalli0n

Junior Member

artizhay

BANNED

Stalli0n

Junior Member

Packers

Registered Member

xpwizard

Junior Member

gimme4free

Elite Member

maximviper

BANNED

The Doctor

Senior Member

Main Menu

Marketplace

Making Money

BlackHat World