Free Twitter scraper

[vinclancy]

Hey all,

I've been pretty fed up with scrapers out there, so I'm just building my own. Right now it scrapes usernames or tweets from a recent keyword search, as many as you'd like at a time; I'll be adding more to this same link as I go. I plan on it being the best scraper out there, and I'll always keep it free.

Here it is: https://github.com/austenallred/twitter-scraper

 

[leetchart]

Keep this bot updated, in which programming language was writen?

 

[vinclancy]

UPDATE: I just added a tweet scraper for unlimited usernames and unlimited number of tweets (so long as you stay within Twitter's rate limits).

And it is written in ruby.

 

[jamie3000]

Do you know the limits for scraping users through the api? Nice share mate

 

[ItsBlinkHere]

Do you know the limits for scraping users through the api? Nice share mate

Just google it, it should be one of the top results. I can't remember right now but I think you can make 200 calls every 15mins. I'm not sure though.

 

[lord1027]

Fantastic work, hope you keep this tool updated

 

[vinclancy]

I've set up a cron job to run it every 30 minutes. I automaticlaly scrape 22,500 usernames every half hour, and haven't had a problem

 

[vinclancy]

Any other features you guys would like?

 

[Numbuh362]

thanks man, great share.

 

[kez1000]

How do I use this??? not too sure what I should be doing

 

[vinclancy]

Download the files, make sure you have Ruby installed, and follow the instructions in the readme. Add me on skype if you get lost.

 

[CyberAlien]

Is this script scraping the data or using Twitter's API?

 

[macdonjo3]

lol OP. the newbies may need an .exe or this is going to build into a massive thread of confused noobs.

Is this script scraping the data or using Twitter's API?

CyberAlien, if you bothered to open the source or even spend 20 seconds looking at it, you can see it uses the API.

config.consumer_key = "YOUR_CONSUMER_KEY"
config.consumer_secret = "YOUR_CONSUMER_SECRET"
config.access_token = "YOUR_ACCESS_TOKEN"
config.access_token_secret = "YOUR_ACCESS_SECRET"

 

[oOMaGiiKzZOo]

lol OP. the newbies may need an .exe or this is going to build into a massive thread of confused noobs.

right on haha i have no idea how to run a ruby file

 

[vinclancy]

Follow the instructions in the readme or add me on skype and I can explain it

 

[oliva999]

Does it scrape pictures or vines?

 

[badkarma]

Thanks for the share! I also took a look at the book you're writing, can't wait to see it

 

[lostgringos]

Hi, what's your Skype id?

Thanks

Frank

 

[vinclancy]

skype id - austenallred (as you can see by clicking the skype button under every comment i post)

 

[vinclancy]

Also, I just added a simple follower file and function to the bot; soon it should be the only twitter bot you need.