1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Free Twitter scraper

Discussion in 'Twitter' started by aus10, Dec 1, 2014.

  1. aus10

    aus10 Junior Member

    Joined:
    Jul 6, 2013
    Messages:
    194
    Likes Received:
    146
    Hey all,

    I've been pretty fed up with scrapers out there, so I'm just building my own. Right now it scrapes usernames or tweets from a recent keyword search, as many as you'd like at a time; I'll be adding more to this same link as I go. I plan on it being the best scraper out there, and I'll always keep it free.

    Here it is: https://github.com/austenallred/twitter-scraper
     
    • Thanks Thanks x 4
  2. leetchart

    leetchart Regular Member

    Joined:
    Aug 15, 2012
    Messages:
    453
    Likes Received:
    108
    Keep this bot updated, in which programming language was writen?
     
  3. aus10

    aus10 Junior Member

    Joined:
    Jul 6, 2013
    Messages:
    194
    Likes Received:
    146
    UPDATE: I just added a tweet scraper for unlimited usernames and unlimited number of tweets (so long as you stay within Twitter's rate limits).

    And it is written in ruby.
     
  4. jamie3000

    jamie3000 Senior Member Premium Member

    Joined:
    Jun 30, 2014
    Messages:
    1,073
    Likes Received:
    470
    Occupation:
    Finance coder looking for semi-retirement
    Location:
    uk
    Do you know the limits for scraping users through the api? Nice share mate
     
  5. ItsBlinkHere

    ItsBlinkHere Regular Member

    Joined:
    Apr 27, 2014
    Messages:
    409
    Likes Received:
    150
    Location:
    At Large
    Just google it, it should be one of the top results. I can't remember right now but I think you can make 200 calls every 15mins. I'm not sure though.
     
  6. lord1027

    lord1027 Elite Member

    Joined:
    Sep 20, 2013
    Messages:
    3,174
    Likes Received:
    2,222
    Fantastic work, hope you keep this tool updated :)
     
  7. aus10

    aus10 Junior Member

    Joined:
    Jul 6, 2013
    Messages:
    194
    Likes Received:
    146
    I've set up a cron job to run it every 30 minutes. I automaticlaly scrape 22,500 usernames every half hour, and haven't had a problem
     
  8. aus10

    aus10 Junior Member

    Joined:
    Jul 6, 2013
    Messages:
    194
    Likes Received:
    146
    Any other features you guys would like?
     
  9. Numbuh362

    Numbuh362 Elite Member

    Joined:
    Aug 22, 2012
    Messages:
    1,569
    Likes Received:
    462
    thanks man, great share.
     
  10. kez1000

    kez1000 Supreme Member

    Joined:
    Jul 24, 2009
    Messages:
    1,402
    Likes Received:
    1,340
    Location:
    UK
    How do I use this??? not too sure what I should be doing
     
  11. aus10

    aus10 Junior Member

    Joined:
    Jul 6, 2013
    Messages:
    194
    Likes Received:
    146
    Download the files, make sure you have Ruby installed, and follow the instructions in the readme. Add me on skype if you get lost.
     
  12. CyberAlien

    CyberAlien Regular Member

    Joined:
    Apr 14, 2010
    Messages:
    484
    Likes Received:
    231
    Is this script scraping the data or using Twitter's API?
     
  13. macdonjo3

    macdonjo3 Jr. VIP Jr. VIP Premium Member

    Joined:
    Nov 8, 2009
    Messages:
    5,563
    Likes Received:
    4,317
    Location:
    Toronto
    Home Page:
    lol OP. the newbies may need an .exe or this is going to build into a massive thread of confused noobs.

    CyberAlien, if you bothered to open the source or even spend 20 seconds looking at it, you can see it uses the API.

    config.consumer_key = "YOUR_CONSUMER_KEY"
    config.consumer_secret = "YOUR_CONSUMER_SECRET"
    config.access_token = "YOUR_ACCESS_TOKEN"
    config.access_token_secret = "YOUR_ACCESS_SECRET"
     
  14. oOMaGiiKzZOo

    oOMaGiiKzZOo Registered Member

    Joined:
    Jul 27, 2014
    Messages:
    80
    Likes Received:
    23
    right on haha i have no idea how to run a ruby file :p
     
  15. aus10

    aus10 Junior Member

    Joined:
    Jul 6, 2013
    Messages:
    194
    Likes Received:
    146
    :) Follow the instructions in the readme or add me on skype and I can explain it
     
  16. oliva999

    oliva999 Power Member

    Joined:
    Feb 6, 2010
    Messages:
    571
    Likes Received:
    68
    Does it scrape pictures or vines?
     
  17. badkarma

    badkarma Newbie

    Joined:
    Nov 17, 2014
    Messages:
    9
    Likes Received:
    1
    Location:
    Chicago
    Thanks for the share! I also took a look at the book you're writing, can't wait to see it :cool:
     
  18. lostgringos

    lostgringos Senior Member

    Joined:
    Dec 5, 2008
    Messages:
    837
    Likes Received:
    266
    Occupation:
    Online Reputation Manager
    Location:
    Dumaguete City, Philippines
    Hi, what's your Skype id?

    Thanks

    Frank
     
  19. aus10

    aus10 Junior Member

    Joined:
    Jul 6, 2013
    Messages:
    194
    Likes Received:
    146
    skype id - austenallred (as you can see by clicking the skype button under every comment i post)
     
  20. aus10

    aus10 Junior Member

    Joined:
    Jul 6, 2013
    Messages:
    194
    Likes Received:
    146
    Also, I just added a simple follower file and function to the bot; soon it should be the only twitter bot you need.