1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Would you like to have a fast twitter followers scraper?

Discussion in 'Twitter' started by mnunes532, Aug 14, 2014.

  1. mnunes532

    mnunes532 Power Member

    Joined:
    Jan 21, 2014
    Messages:
    708
    Likes Received:
    178
    Hey there,

    a twitter bot I'm currently using is facing some problems on followers scrapping module. Since that module is something I really need, I decided to make a simple webpage that scrapes followers using twitter api. It can scrape as fast as 5000 users in less than 5 seconds and save them into a text file. I'm still building it, yet I would like to know if you want me to share it with you.

    You would need to install xampp, create a twitter app, and download some files. It would only work on your computer yet it would be possible to put in your hosting too.

    Of course I would explain all steps in detail.

    Let me know if you want this or not. I would share it till the end of the week.

    Best regards.
     
  2. erichon

    erichon Regular Member

    Joined:
    Dec 12, 2011
    Messages:
    388
    Likes Received:
    61
    YES PLEASE my twtdom is not srcraping well
     
  3. mnunes532

    mnunes532 Power Member

    Joined:
    Jan 21, 2014
    Messages:
    708
    Likes Received:
    178
    Only one person interested?

    I manage to create 9 twitter apps using 3 different accounts. From my calculations it is possible to scrape up to 2 million users per hour using those 9 apps. The more apps I have, more followers I can scrape. This is because each app/user has limits in order to use the twitter api.

    I can make it possible to scrape followers from a list of users. Like for example, you want to scrape followers from 5 singers. You would put those 5 singers in a input box, and it would scrape all the followers from all those accounts.
     
  4. SmartIphone

    SmartIphone Newbie

    Joined:
    Jan 22, 2014
    Messages:
    12
    Likes Received:
    0
    I am interested !!!
     
  5. mij33

    mij33 Junior Member

    Joined:
    Mar 18, 2009
    Messages:
    116
    Likes Received:
    8
    yes im interested , i used to use scraperwiki but they got shut down and ive been looking for an alternative recently and the bots ive used dont scrape well .
     
  6. Repulsor

    Repulsor Power Member

    Joined:
    Jun 11, 2013
    Messages:
    707
    Likes Received:
    267
    Location:
    PHP Scripting ;)
    I have created something like this before for a client of mine. PHP script, Yeah. Well, there is rate limiting and probably content limiting too. I dont think you can get the full of a popular entity. May be most of it, but I dont think you can fetch the full.

    Good luck though.
     
  7. Jeremy96x

    Jeremy96x Junior Member

    Joined:
    Aug 9, 2014
    Messages:
    149
    Likes Received:
    24
    Seem like a pretty good scrapper, but (sorry for the noobishness) what do you mean with ""create a twitter app"
     
  8. leetchart

    leetchart Regular Member

    Joined:
    Aug 15, 2012
    Messages:
    453
    Likes Received:
    108
    I would like to take a look at this tool.
    Also features like scrape users that have reply on tweet will pretty good
     
  9. mnunes532

    mnunes532 Power Member

    Joined:
    Jan 21, 2014
    Messages:
    708
    Likes Received:
    178
    I can fetch all followers from any user. ;)

    You need to create a twitter app, on your developer account. I will explain how you do it. ;)

    It is not a "tool", it is more like a webpage. ;)

    I don't know if I will add more features since I don't need them.
     
  10. Fortythieve

    Fortythieve BANNED BANNED

    Joined:
    Jun 8, 2013
    Messages:
    419
    Likes Received:
    70
    Show us something and dont be all talk op
     
  11. agentblue

    agentblue Newbie Premium Member

    Joined:
    Aug 15, 2014
    Messages:
    15
    Likes Received:
    3
    Occupation:
    programmer
    Home Page:
    Im interested in the service, not the hosting of the tools, can you provide it as a service for a fee?
     
  12. mnunes532

    mnunes532 Power Member

    Joined:
    Jan 21, 2014
    Messages:
    708
    Likes Received:
    178
    I'm still working on it, or else I would have shared it already. ;)

    I'll create a new thread about this later today or tomorrow.
     
  13. JerryShaw

    JerryShaw Newbie

    Joined:
    Jul 25, 2014
    Messages:
    14
    Likes Received:
    0
    Yes please! With good instructions. I tried creating a few bots but couldn't implement them! <3
     
  14. SoldMySoulToSeo

    SoldMySoulToSeo Newbie

    Joined:
    Jun 29, 2014
    Messages:
    24
    Likes Received:
    1
    This sounds very handy. I'll probably need a clear step by step of putting it together. Would you say it is simple once you get all the necessaries set up?
    By simple. I mean the apps and what not.
    On another note: Xampp is still alive? what-the-hell. HAHA. I'll do it. Why not.
    Anyways. Thanks op, and keep us updated.
     
  15. csibbs

    csibbs Newbie

    Joined:
    Sep 27, 2012
    Messages:
    14
    Likes Received:
    3
    i would be interested.
     
  16. tek007

    tek007 Registered Member

    Joined:
    Mar 24, 2013
    Messages:
    82
    Likes Received:
    13
    Please I'm interested in this
     
  17. ItsBlinkHere

    ItsBlinkHere Regular Member

    Joined:
    Apr 27, 2014
    Messages:
    409
    Likes Received:
    150
    Location:
    At Large
    Its not possible to get usernames that fast. I scrape user_ids at that speed but not the usernames. You would have to do a user/lookup to then get the names. You would have to have a shitload of accounts to do anything fast though. Also you can't alternate cursor values between multiple accounts. This means that if I start scraping @justinbeiber with @my_account then I can't use other accounts to speed up the process. Also you can only make 15 api calls every 15mins. That means that you can only make 60 calls per hour or 300k user_ids scraped per hour with each account.
     
  18. mnunes532

    mnunes532 Power Member

    Joined:
    Jan 21, 2014
    Messages:
    708
    Likes Received:
    178
    You are totally right and that's why I have over than 10 apps :)

    I'm able to scrape 1 million of usernames per 15 minutes. :)
     
  19. ItsBlinkHere

    ItsBlinkHere Regular Member

    Joined:
    Apr 27, 2014
    Messages:
    409
    Likes Received:
    150
    Location:
    At Large
    Yeah, I was just wanting to point out that you can't scrape 1 million from one user in 15mins. It would be from multiple users. Cursor value's are not cross compatible between accounts. At least to my knowledge.
     
  20. mnunes532

    mnunes532 Power Member

    Joined:
    Jan 21, 2014
    Messages:
    708
    Likes Received:
    178
    That's not a problem since you can create 3 apps per day per account. Do it for a whole week and you will have 21 apps and no need to worry about much about rate limits.