Free Twitter scraper

Discussion in 'Twitter' started by aus10, Dec 1, 2014.

  1. aus10

    aus10 Jr. VIP Jr. VIP

    Joined:
    Jul 6, 2013
    Messages:
    228
    Likes Received:
    150
    Hey all,

    I've been pretty fed up with scrapers out there, so I'm just building my own. Right now it scrapes usernames or tweets from a recent keyword search, as many as you'd like at a time; I'll be adding more to this same link as I go. I plan on it being the best scraper out there, and I'll always keep it free.

    Here it is: https://github.com/austenallred/twitter-scraper
     
    • Thanks Thanks x 4
  2. leetchart

    leetchart Regular Member

    Joined:
    Aug 15, 2012
    Messages:
    453
    Likes Received:
    108
    Keep this bot updated, in which programming language was writen?
     
  3. aus10

    aus10 Jr. VIP Jr. VIP

    Joined:
    Jul 6, 2013
    Messages:
    228
    Likes Received:
    150
    UPDATE: I just added a tweet scraper for unlimited usernames and unlimited number of tweets (so long as you stay within Twitter's rate limits).

    And it is written in ruby.
     
  4. jamie3000

    jamie3000 Elite Member Premium Member

    Joined:
    Jun 30, 2014
    Messages:
    2,029
    Likes Received:
    931
    Occupation:
    Owner of BigGuestPosting.com
    Location:
    uk
    Home Page:
    Do you know the limits for scraping users through the api? Nice share mate
     
  5. ItsBlinkHere

    ItsBlinkHere Regular Member

    Joined:
    Apr 27, 2014
    Messages:
    408
    Likes Received:
    150
    Location:
    At Large
    Just google it, it should be one of the top results. I can't remember right now but I think you can make 200 calls every 15mins. I'm not sure though.
     
  6. lord1027

    lord1027 Elite Member

    Joined:
    Sep 20, 2013
    Messages:
    3,181
    Likes Received:
    2,241
    Fantastic work, hope you keep this tool updated :)
     
  7. aus10

    aus10 Jr. VIP Jr. VIP

    Joined:
    Jul 6, 2013
    Messages:
    228
    Likes Received:
    150
    I've set up a cron job to run it every 30 minutes. I automaticlaly scrape 22,500 usernames every half hour, and haven't had a problem
     
  8. aus10

    aus10 Jr. VIP Jr. VIP

    Joined:
    Jul 6, 2013
    Messages:
    228
    Likes Received:
    150
    Any other features you guys would like?
     
  9. Numbuh362

    Numbuh362 Elite Member

    Joined:
    Aug 22, 2012
    Messages:
    1,567
    Likes Received:
    462
    thanks man, great share.
     
  10. kez1000

    kez1000 Supreme Member

    Joined:
    Jul 24, 2009
    Messages:
    1,421
    Likes Received:
    1,350
    Location:
    UK
    How do I use this??? not too sure what I should be doing
     
  11. aus10

    aus10 Jr. VIP Jr. VIP

    Joined:
    Jul 6, 2013
    Messages:
    228
    Likes Received:
    150
    Download the files, make sure you have Ruby installed, and follow the instructions in the readme. Add me on skype if you get lost.
     
  12. CyberAlien

    CyberAlien Power Member

    Joined:
    Apr 14, 2010
    Messages:
    505
    Likes Received:
    248
    Is this script scraping the data or using Twitter's API?
     
  13. macdonjo3

    macdonjo3 Jr. VIP Jr. VIP

    Joined:
    Nov 8, 2009
    Messages:
    5,786
    Likes Received:
    4,467
    Location:
    Toronto
    lol OP. the newbies may need an .exe or this is going to build into a massive thread of confused noobs.

    CyberAlien, if you bothered to open the source or even spend 20 seconds looking at it, you can see it uses the API.

    config.consumer_key = "YOUR_CONSUMER_KEY"
    config.consumer_secret = "YOUR_CONSUMER_SECRET"
    config.access_token = "YOUR_ACCESS_TOKEN"
    config.access_token_secret = "YOUR_ACCESS_SECRET"
     
  14. oOMaGiiKzZOo

    oOMaGiiKzZOo Junior Member

    Joined:
    Jul 27, 2014
    Messages:
    112
    Likes Received:
    36
    right on haha i have no idea how to run a ruby file :p
     
  15. aus10

    aus10 Jr. VIP Jr. VIP

    Joined:
    Jul 6, 2013
    Messages:
    228
    Likes Received:
    150
    :) Follow the instructions in the readme or add me on skype and I can explain it
     
  16. oliva999

    oliva999 Power Member

    Joined:
    Feb 6, 2010
    Messages:
    657
    Likes Received:
    79
    Does it scrape pictures or vines?
     
  17. badkarma

    badkarma Newbie

    Joined:
    Nov 17, 2014
    Messages:
    15
    Likes Received:
    2
    Location:
    Chicago
    Thanks for the share! I also took a look at the book you're writing, can't wait to see it :cool:
     
  18. lostgringos

    lostgringos Senior Member

    Joined:
    Dec 5, 2008
    Messages:
    839
    Likes Received:
    268
    Gender:
    Male
    Occupation:
    Personal Escort Assistrant
    Location:
    Canada
    Home Page:
    Hi, what's your Skype id?

    Thanks

    Frank
     
  19. aus10

    aus10 Jr. VIP Jr. VIP

    Joined:
    Jul 6, 2013
    Messages:
    228
    Likes Received:
    150
    skype id - austenallred (as you can see by clicking the skype button under every comment i post)
     
  20. aus10

    aus10 Jr. VIP Jr. VIP

    Joined:
    Jul 6, 2013
    Messages:
    228
    Likes Received:
    150
    Also, I just added a simple follower file and function to the bot; soon it should be the only twitter bot you need.