1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Please help me with this twitter scraper script

Discussion in 'Twitter' started by kittyfranklin, Oct 28, 2015.

  1. kittyfranklin

    kittyfranklin Registered Member

    Joined:
    Jan 29, 2013
    Messages:
    77
    Likes Received:
    12
    Hey everyone,

    So I found this script from like 2011 or something, which uses a Google doc spreadsheet with script to scrape the followers or following of a Twitter account.

    So far I have been able to get it to scrap 5,000 users, but it will not go past that, and when I run it again, it just scrapes the same 5,000 again.

    I have a client who is following 40,000 people and he would like his whitelist (people he never wants to unfollow) to be made up of only verified users, and this scraper has a column that shows if the user is verified.

    Does anyone know of any code I can add to the script so that when it runs the second time, it starts up where it left off?

    Here are the original links about the script:

    https://mashe.hawksey.info/2011/03/export-twitter-followers/

    https://mashe.hawksey.info/2011/08/export-more-twitter-followers/#comment-12920

    Here is the link with the added script to try to get the scraping to continue without stopping (I'm not even sure if this is accomplishing anything, since Twitter is just doing the 5000 limit)

    https://gist.github.com/mhawksey/1121340

    Also, when I use the Google doc spreadsheet, when I am on the script editing page, and I click "Run", it comes up with this error:

    ScriptProperties API is deprecated.
    File: Code Line: 501
    The API has been marked as deprecated which means that the feature should be avoided and may be removed in the future. Consider using an alternative solution.

    This is line 501: if (!ScriptProperties.getProperty("cursor")){

    But when I am in the spreadsheet and I use the spreadsheet's function to scrape the "friends" of the account, it will give me the first 5,000 people each time.

    Thanks to anyone who can help! :)
     
  2. fidodido

    fidodido Junior Member

    Joined:
    Aug 12, 2015
    Messages:
    113
    Likes Received:
    27
  3. kittyfranklin

    kittyfranklin Registered Member

    Joined:
    Jan 29, 2013
    Messages:
    77
    Likes Received:
    12
    Thanks fidodido. I'm not a developer, but I did have to create the twitter app to make this work. I don't understand programming/scripts, etc, so I can't fix this to make it work right. I believe it worked properly in 2011 when the guy made it.

    I do understand twitter's API limits, but I was just wondering if anyone knew how I could edit the script so that when I ran it again, it would start back up where I left off once I hit the limit and waited, instead of it just doing the first 5,000 each time. I was hoping I could get the first 5,000, then wait and run it again to get 5,001-10,000, and so on...
     
  4. kittyfranklin

    kittyfranklin Registered Member

    Joined:
    Jan 29, 2013
    Messages:
    77
    Likes Received:
    12
    So anyone on here know how to program scripts for Twitter scraping?