1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Scraping twitter followers?

Discussion in 'PHP & Perl' started by blackbadger, Mar 10, 2016.

Tags:
  1. blackbadger

    blackbadger Registered Member

    Joined:
    Sep 11, 2013
    Messages:
    58
    Likes Received:
    13
    Location:
    UK
    Inspired by another member (phpbuilt) on here I'm about to embark on building my own twitter follower scraper using the twitter API.

    I've read the api limits call to something like 15 calls in 15 minutes. Each call return 200 followers. So, in an hour, using only one account I can scrape 12,000 followers. In a day 288k followers.

    Let' say I want to run 4 or 5 instances of this php script at the same time ... whats the best way to do that? cron?

    Thanks in advance.
     
  2. Frozen27

    Frozen27 Newbie

    Joined:
    Aug 8, 2013
    Messages:
    20
    Likes Received:
    4
    Cron is a good way to go. 4 or 5 instances means the same amount of accounts and proxies, since using the same IP for all the accounts will get you banned. I've set my scrapper for 15 calls every 20 mins for obvious reasons.
     
  3. AdvancedDevelopment

    AdvancedDevelopment BANNED BANNED

    Joined:
    Mar 23, 2016
    Messages:
    91
    Likes Received:
    28
    I'd say running cron jobs would be the way to go with this, also why not use proxy's to run more than 1 account at a time? Here's an example I wrote for you on how to use proxies with PHP CURL, rather simple.

    Code:
    $Proxy = 'IP:PORT';
    
    
    curl_setopt($ch, CURLOPT_PROXY, $Proxy);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    
    If you plan on using private proxies that require a Username & Password, add these two lines.

    Code:
    $ProxyLogin = 'Username:Password';
    
    
    curl_setopt($ch, CURLOPT_PROXYUSERPWD, $ProxyLogin);
    
     
  4. WhereIsMyMoney

    WhereIsMyMoney Newbie

    Joined:
    Apr 7, 2016
    Messages:
    28
    Likes Received:
    11
    You are right about 15 request per 15 minutes, but each request returns not 200 but up to (if the user has fewer followers you will obviously get less) 5000 follower IDs.

    So if you do the math in an hour you can get up to 300k followers and in one day up to 7200k so I don't see the point in running multiple instances scrappers.


    You may try to run multiple scrappers if you will do user/lookup for those scrapped IDs which I assume you will. In this case, you can get information for 24k to 72k users in an hour.