Wizard's Ultimate Twitter Journey

dgusic said:

What about that amazon product, how did that go ?

Click to expand...

wizard04 said:

I should start looking for a scraper, I have one that one member shared but its only scraping number ID's which is not compatible with FL.

Click to expand...

<?php
$hostname="localhost";
$username="";
$password="";
$connection = mysql_connect ($hostname,$username,$password);
mysql_select_db("twitter");


// a_follow table contains dozens of accounts I want to follow
$query = "SELECT * FROM a_follow WHERE myinc2 = 0 ORDER BY id LIMIT 0,1";
$result = mysql_query($query);
$myid = -1;
while($r=mysql_fetch_array($result)){
    $myid = $r["id"];
    $myaccount = $r["handle"];
}
// myinc2 is a counter column, all start at 0. when all 0's have incremented to 1 ...
// switch back to all 0s and loop through the list again ...
if($myid == -1){
    $query = "UPDATE a_follow SET myinc2 = 0";
    $result = mysql_query($query);
    $query = "SELECT * FROM a_follow WHERE myinc2 = 0 ORDER BY id LIMIT 0,1";
    $result = mysql_query($query);
    while($r=mysql_fetch_array($result)){
        $myid = $r["id"];
        $myaccount = $r["handle"];
    }    
}
// a token is the cursor that twitter API gives you, so you can scroll through accounts ...
// -1 cursor means it's starting from beginning of list. you need this so you can go through
// a million followers, 200 at a time, the token is your bookmarker
$query = "SELECT * FROM a_tokens WHERE account = '$myaccount'";
$result = mysql_query($query);
$cursor = -2;
while($r=mysql_fetch_array($result)){
    $cursor = $r["token"];
}
if($cursor == '-2'){
    $query = "INSERT INTO a_tokens(account,token) VALUES('$myaccount','-1')";
    $result = mysql_query($query);
    $cursor = '-1';
}




// magic hocus pocus connect to the twitter API


    function buildBaseString($baseURI, $method, $params) {
        $r = array();
        ksort($params);
        foreach($params as $key=>$value){
            $r[] = "$key=" . rawurlencode($value);
        }
        return $method."&" . rawurlencode($baseURI) . '&' . rawurlencode(implode('&', $r));
    }


    function buildAuthorizationHeader($oauth) {
        $r = 'Authorization: OAuth ';
        $values = array();
        foreach($oauth as $key=>$value)
            $values[] = "$key=\"" . rawurlencode($value) . "\"";
        $r .= implode(', ', $values);
        return $r;
    }


    $url = "https://api.twitter.com/1.1/followers/list.json";


    $oauth_access_token = "get your own from twitter";
    $oauth_access_token_secret = "get your own from twitter";
    $consumer_key = "get your own from twitter";
    $consumer_secret = "get your own from twitter";


    $oauth = array('screen_name' => $myaccount,
                   'cursor' => $cursor,
                   'count' => 200,
                   'oauth_consumer_key' => $consumer_key,
                   'oauth_nonce' => time(),
                   'oauth_signature_method' => 'HMAC-SHA1',
                   'oauth_token' => $oauth_access_token,
                   'oauth_timestamp' => time(),
                   'oauth_version' => '1.0'
                 );                    
    
    $base_info = buildBaseString($url, 'GET', $oauth);
    $composite_key = rawurlencode($consumer_secret) . '&' . rawurlencode($oauth_access_token_secret);
    $oauth_signature = base64_encode(hash_hmac('sha1', $base_info, $composite_key, true));
    $oauth['oauth_signature'] = $oauth_signature;


    // Make requests
    $header = array(buildAuthorizationHeader($oauth), 'Expect:');
    $options = array( CURLOPT_HTTPHEADER => $header,
        CURLOPT_PROXY => "insert your proxy:80",
                      CURLOPT_HEADER => false,
                      CURLOPT_URL => $url.'?screen_name='.$myaccount.'&cursor='.$cursor.'&count=200',
                      CURLOPT_RETURNTRANSFER => true,
                      CURLOPT_SSL_VERIFYPEER => false);


    $feed = curl_init();
    curl_setopt_array($feed, $options);
    $json = curl_exec($feed);
    curl_close($feed);


    $twitter_data = json_decode($json,true);
    
    foreach($twitter_data AS $key => $value){
        if($key == 'next_cursor_str'){
            $next_cursor = $value;
        }
        if($key == 'users'){
            foreach ($value AS $key2 => $value2){
                if($key == 'users'){
                    $id = $value2["id"];
                    $name = $value2["name"];
                    $screen_name = $value2["screen_name"];
                    $description = $value2["description"];
                    $followers_count = $value2["followers_count"];
                    $friends_count = $value2["friends_count"];
                    $statuses_count = $value2["statuses_count"];
                    $statustime = '00-00-00 00:00:00';
                    $statustime = @$value2["status"]["created_at"];
                    $statime = date('Y-m-d H:i:s', strtotime($statustime));
                    $allfollow[$id]["name"] = mysql_real_escape_string(@$name);
                    $allfollow[$id]["screen_name"] = mysql_real_escape_string(@$screen_name);
                    $allfollow[$id]["statuses_time"] = mysql_real_escape_string(@$statime);
                    $allfollow[$id]["description"] = mysql_real_escape_string(@$description);
                    $allfollow[$id]["followers_count"] = @$followers_count;
                    $allfollow[$id]["friends_count"] = @$friends_count;
                    $allfollow[$id]["statuses_count"] = @$statuses_count;
                }    
            }
        }
    }
    $q = array();
    foreach($allfollow AS $key => $value){
        $q[] = "INSERT INTO a_master(mainid,twitid,belongsto,name,screen_name,description,followers_count,friends_count,statuses_count,statuses_time) VALUES('$key-$myaccount',$key,'$myaccount','$value[name]','$value[screen_name]','$value[description]',$value[followers_count],$value[friends_count],$value[statuses_count],'$value[statuses_time]') ON DUPLICATE KEY UPDATE followers_count = $value[followers_count],friends_count = $value[friends_count],statuses_count = $value[statuses_count],statuses_time = '$value[statuses_time]',mypurge = 0";
    }
    //print_r($allfollow);
    
    foreach($q AS $value){
        $result = mysql_query($value);
        //echo "$value<br />\n";
    }
    if($next_cursor != ''){    
        if($next_cursor == '0'){
            $next_cursor = '-1';
            $query = "UPDATE a_follow SET myinc2 = 1 WHERE handle = '$myaccount'";
            $result = mysql_query($query);
        }
        $query = "UPDATE a_tokens SET token = '$next_cursor' WHERE account = '$myaccount'";
        $result = mysql_query($query);
    }

CREATE TABLE IF NOT EXISTS `a_master` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `mainid` varchar(70) NOT NULL,
  `twitid` varchar(50) NOT NULL,
  `belongsto` varchar(50) NOT NULL,
  `name` varchar(50) NOT NULL,
  `screen_name` varchar(50) NOT NULL,
  `description` varchar(256) NOT NULL,
  `followers_count` int(11) NOT NULL,
  `friends_count` int(11) NOT NULL,
  `statuses_count` int(11) NOT NULL,
  `statuses_time` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
  `created` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  `mypurge` int(11) NOT NULL,
  PRIMARY KEY (`id`),
  UNIQUE KEY `mainid` (`mainid`),
  KEY `statuses_count` (`statuses_count`),
  KEY `friends_count` (`friends_count`),
  KEY `belongsto` (`belongsto`),
  KEY `statuses_time` (`statuses_time`),
  KEY `created` (`created`),
  KEY `screen_name` (`screen_name`),
  KEY `followers_count` (`followers_count`),
  KEY `twitid` (`twitid`)
) ENGINE=InnoDB  DEFAULT CHARSET=latin1 AUTO_INCREMENT=0;

SELECT * FROM `a_master` WHERE (DATEDIFF(NOW(), statuses_time) <= 30) AND ((friends_count < 1000) OR (friends_count >= (followers_count * .7))) GROUP BY twitid

jekleni said:

Hey @phpbuilt,

I think its actually better to scrape without any API.. I made my own scraper that uses only 1 account, that doesnt need to be pva or anything, and can scrape 3k per minute on my connection, roughly 2k per minute on average internet connection. Thats 180k per hour, 4,5 mil per day
Im not realy gonna give it away, but I think directly scraping from twitter.com/user/followers is far better and FASTER!

Click to expand...

phpbuilt said:

I made my own in PHP. It needs a phone verified account, after phone is verified you have to create API keys.

I target an account and scrape their followers, but I get all the details about each one (their last status so you can see how active they are, how many followers they have, how many friends, how many tweets, etc).

The specific API call is to https://api.twitter.com/1.1/followers/list.json, it returns 200 accounts each time, and the limit is 1 call per minute (actually 15 calls in a 15 minute block). That makes the maximum info returned per account 288k per day ... I have 4 accounts running and it pulls down a million users per day (and when my millions list is finished, it loops back to the beginning and rechecks accounts because someone new might have went active, etc).

The bonus of doing it this way: you're not going to hit maximum CPU limits -- it barely uses any CPU. It's not going to "crash" like FL seems to when its cache hits around 200k? (not sure I don't use it) ... and you can easily juggle 100 million accounts with pinpoint accuracy. There's no guessing when a list is going to run out.

Also, it sounds like with FL, if you have a million people to follow and 80 accounts, you have to make 80 lists dividing the million into smaller lists ... with my setup, my accounts grab from the same list (whenever an account grabs a name, no other account grabs it).

The bad part of doing it this way -- got to learn some new tricks lol.

But from what I've heard the bad parts of FL are, people who use FL could really do well to generate their own lists. That way, you can determine ahead of time things like how active an account is, whether they have a follow/follower ratio that meets your filters, BEFORE you feed the list to FL. That way, there's no guessing whether FL will use most of the list or skip a large portion because the list are fakes, etc.

I'm not going to be able to troubleshoot the code below for anyone ... you'll have to learn some PHP, however if you have some php skills you should be able to work through it ...

Code:

<?php
$hostname="localhost";
$username="";
$password="";
$connection = mysql_connect ($hostname,$username,$password);
mysql_select_db("twitter");


// a_follow table contains dozens of accounts I want to follow
$query = "SELECT * FROM a_follow WHERE myinc2 = 0 ORDER BY id LIMIT 0,1";
$result = mysql_query($query);
$myid = -1;
while($r=mysql_fetch_array($result)){
    $myid = $r["id"];
    $myaccount = $r["handle"];
}
// myinc2 is a counter column, all start at 0. when all 0's have incremented to 1 ...
// switch back to all 0s and loop through the list again ...
if($myid == -1){
    $query = "UPDATE a_follow SET myinc2 = 0";
    $result = mysql_query($query);
    $query = "SELECT * FROM a_follow WHERE myinc2 = 0 ORDER BY id LIMIT 0,1";
    $result = mysql_query($query);
    while($r=mysql_fetch_array($result)){
        $myid = $r["id"];
        $myaccount = $r["handle"];
    }    
}
// a token is the cursor that twitter API gives you, so you can scroll through accounts ...
// -1 cursor means it's starting from beginning of list. you need this so you can go through
// a million followers, 200 at a time, the token is your bookmarker
$query = "SELECT * FROM a_tokens WHERE account = '$myaccount'";
$result = mysql_query($query);
$cursor = -2;
while($r=mysql_fetch_array($result)){
    $cursor = $r["token"];
}
if($cursor == '-2'){
    $query = "INSERT INTO a_tokens(account,token) VALUES('$myaccount','-1')";
    $result = mysql_query($query);
    $cursor = '-1';
}




// magic hocus pocus connect to the twitter API


    function buildBaseString($baseURI, $method, $params) {
        $r = array();
        ksort($params);
        foreach($params as $key=>$value){
            $r[] = "$key=" . rawurlencode($value);
        }
        return $method."&" . rawurlencode($baseURI) . '&' . rawurlencode(implode('&', $r));
    }


    function buildAuthorizationHeader($oauth) {
        $r = 'Authorization: OAuth ';
        $values = array();
        foreach($oauth as $key=>$value)
            $values[] = "$key=\"" . rawurlencode($value) . "\"";
        $r .= implode(', ', $values);
        return $r;
    }


    $url = "https://api.twitter.com/1.1/followers/list.json";


    $oauth_access_token = "get your own from twitter";
    $oauth_access_token_secret = "get your own from twitter";
    $consumer_key = "get your own from twitter";
    $consumer_secret = "get your own from twitter";


    $oauth = array('screen_name' => $myaccount,
                   'cursor' => $cursor,
                   'count' => 200,
                   'oauth_consumer_key' => $consumer_key,
                   'oauth_nonce' => time(),
                   'oauth_signature_method' => 'HMAC-SHA1',
                   'oauth_token' => $oauth_access_token,
                   'oauth_timestamp' => time(),
                   'oauth_version' => '1.0'
                 );                    
    
    $base_info = buildBaseString($url, 'GET', $oauth);
    $composite_key = rawurlencode($consumer_secret) . '&' . rawurlencode($oauth_access_token_secret);
    $oauth_signature = base64_encode(hash_hmac('sha1', $base_info, $composite_key, true));
    $oauth['oauth_signature'] = $oauth_signature;


    // Make requests
    $header = array(buildAuthorizationHeader($oauth), 'Expect:');
    $options = array( CURLOPT_HTTPHEADER => $header,
        CURLOPT_PROXY => "insert your proxy:80",
                      CURLOPT_HEADER => false,
                      CURLOPT_URL => $url.'?screen_name='.$myaccount.'&cursor='.$cursor.'&count=200',
                      CURLOPT_RETURNTRANSFER => true,
                      CURLOPT_SSL_VERIFYPEER => false);


    $feed = curl_init();
    curl_setopt_array($feed, $options);
    $json = curl_exec($feed);
    curl_close($feed);


    $twitter_data = json_decode($json,true);
    
    foreach($twitter_data AS $key => $value){
        if($key == 'next_cursor_str'){
            $next_cursor = $value;
        }
        if($key == 'users'){
            foreach ($value AS $key2 => $value2){
                if($key == 'users'){
                    $id = $value2["id"];
                    $name = $value2["name"];
                    $screen_name = $value2["screen_name"];
                    $description = $value2["description"];
                    $followers_count = $value2["followers_count"];
                    $friends_count = $value2["friends_count"];
                    $statuses_count = $value2["statuses_count"];
                    $statustime = '00-00-00 00:00:00';
                    $statustime = @$value2["status"]["created_at"];
                    $statime = date('Y-m-d H:i:s', strtotime($statustime));
                    $allfollow[$id]["name"] = mysql_real_escape_string(@$name);
                    $allfollow[$id]["screen_name"] = mysql_real_escape_string(@$screen_name);
                    $allfollow[$id]["statuses_time"] = mysql_real_escape_string(@$statime);
                    $allfollow[$id]["description"] = mysql_real_escape_string(@$description);
                    $allfollow[$id]["followers_count"] = @$followers_count;
                    $allfollow[$id]["friends_count"] = @$friends_count;
                    $allfollow[$id]["statuses_count"] = @$statuses_count;
                }    
            }
        }
    }
    $q = array();
    foreach($allfollow AS $key => $value){
        $q[] = "INSERT INTO a_master(mainid,twitid,belongsto,name,screen_name,description,followers_count,friends_count,statuses_count,statuses_time) VALUES('$key-$myaccount',$key,'$myaccount','$value[name]','$value[screen_name]','$value[description]',$value[followers_count],$value[friends_count],$value[statuses_count],'$value[statuses_time]') ON DUPLICATE KEY UPDATE followers_count = $value[followers_count],friends_count = $value[friends_count],statuses_count = $value[statuses_count],statuses_time = '$value[statuses_time]',mypurge = 0";
    }
    //print_r($allfollow);
    
    foreach($q AS $value){
        $result = mysql_query($value);
        //echo "$value<br />\n";
    }
    if($next_cursor != ''){    
        if($next_cursor == '0'){
            $next_cursor = '-1';
            $query = "UPDATE a_follow SET myinc2 = 1 WHERE handle = '$myaccount'";
            $result = mysql_query($query);
        }
        $query = "UPDATE a_tokens SET token = '$next_cursor' WHERE account = '$myaccount'";
        $result = mysql_query($query);
    }

and a dump of the a_master db columns ...

Code:

CREATE TABLE IF NOT EXISTS `a_master` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `mainid` varchar(70) NOT NULL,
  `twitid` varchar(50) NOT NULL,
  `belongsto` varchar(50) NOT NULL,
  `name` varchar(50) NOT NULL,
  `screen_name` varchar(50) NOT NULL,
  `description` varchar(256) NOT NULL,
  `followers_count` int(11) NOT NULL,
  `friends_count` int(11) NOT NULL,
  `statuses_count` int(11) NOT NULL,
  `statuses_time` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
  `created` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  `mypurge` int(11) NOT NULL,
  PRIMARY KEY (`id`),
  UNIQUE KEY `mainid` (`mainid`),
  KEY `statuses_count` (`statuses_count`),
  KEY `friends_count` (`friends_count`),
  KEY `belongsto` (`belongsto`),
  KEY `statuses_time` (`statuses_time`),
  KEY `created` (`created`),
  KEY `screen_name` (`screen_name`),
  KEY `followers_count` (`followers_count`),
  KEY `twitid` (`twitid`)
) ENGINE=InnoDB  DEFAULT CHARSET=latin1 AUTO_INCREMENT=0;

Again, please take note ... nobody is going to easily understand this code. It's not a tutorial on how to do it yourself. There's a slight chance someone might find it useful, but for most people you've got a lot of studying to do to make it work for you.

So to show how cool it can be if you scrape yourself into your own database ... I run the following query to return accounts to follow ...

Code:

SELECT * FROM `a_master` WHERE (DATEDIFF(NOW(), statuses_time) <= 30) AND ((friends_count < 1000) OR (friends_count >= (followers_count * .7))) GROUP BY twitid

That query results in a list of everyone who has made a tweet in the last month, who has less than 1000 people they follow, or (if they have more than 1000 they follow), they're followed by at least 70% of those they follow (so if they have 2000 followers, they need at least 1400 friends). Those are my current rules for following people, but I can easily adjust it to whatever I want.

Click to expand...

mediabuzz said:

Not sure if you posted this already its a long thread and I couldnt find it but what is your bounce rate and how are you shortening your links or are you using twitter cards?

Click to expand...

wizard04 said:

Update

Total gained followers for today: N/A

Total followers: N/A
AdSense earnings for yesterday: 22.30

Website Unique Visitors for yesterday: 2995


I was out last night till 6am, I didn't have the time to make an update.
I am really pleased with the earnings and overall improvement of my earnings, I am hoping to start pulling 600 euros a month in the next few months (I am wishing).

Click to expand...

phpbuilt said:

All this blizzard of a scraping you're doing. Are you getting last activity of user, their followers, their friends, etc (everything about that user so you can make a decision whether to follow them)? Because there's this API ...

https://dev.twitter.com/rest/reference/get/friends/ids

5000 users at a time, one request per minute, where my 4 accounts would pull 20k per minute, 1.2 million per hour. But I don't do that, because twitter names without their last activity, their followers and friends is kind of worthless to me.

Click to expand...

jekleni said:

Damn, relax mate.. haha, someone sure got up on a wrong foot today. It was just a suggestion.. Im new at coding and like my program.. It checks for active users ofcourse, and also if they got avatar etc.

Some people here just cant take advice without all the rage lol.

Click to expand...

phpbuilt said:

I'm not wanting to hijack Wizard's thread by making a bunch of twitter scraping comments, but there's no rage, honest. You used @phpbuilt, so I took that as a hint that you wanted me to respond.

twitter.com/user/followers, afaik doesn't contain follower count, following count, date of last tweet for each user, so its apples/oranges compared to the API source code I posted.

Everyone that wants to simulate Wizard's journey has to come up with a list of accounts to follow somehow. If they scrape twitter.com/user/followers, and get a long list of only account names, what do they do with it? Maybe FL sorts it all out, I don't know, but from listening to Wizard comment about FL ... if you give FL a bad unfiltered list (with old accounts not being used, or their follower/folowee ratio is out of whack), then FL doesn't respond very well.

If you have a secondary process that scrapes twitter yet again, getting the followers/folowees/last tweet date for each individual user, that's got to be really slow. Might as well use the API and get all the data you need during the first scrape.

Click to expand...

jekleni said:

Wizard, how many threads do you have set in FL, for following? I had 20 so far, and it could barely follow 300 a day, on 50 accounts.. now i set it to 50, so i hope its gonna work better.

Also, does anyone know if its safe to have 2 or maybe even 3 accounts per proxy? I have private proxies, currently only running 1 acc/proxy

EDIT: How do you have blog urls set to? I mean blog post urls.. domain.com/post-name?

Click to expand...

wizard04 said:

For 20 acc, 20 threads.

You can safely have 4 acc on one proxy (I am telling this from my own testing).

Does it really matter? Anyway I am using the simple domain,com/article/ structure.

Click to expand...