Best Language For Scrapping Data From Twitter

[FlyingCow]

Hey everyone,

I'm looking to scrape data from Twitter. What would you say is the best programming language or tool for this task? I'm open to suggestions and experiences!

Thanks!

 

[spectrejoe]

Python is one of the most used languages for scraping. If you gonna use their API there's a ton of options. If you just wanna get tweets from specific users or from search queries you can try nitter wrappers

 

[FUTP]

Hey everyone,

I'm looking to scrape data from Twitter. What would you say is the best programming language or tool for this task? I'm open to suggestions and experiences!

Thanks!

In my opinion there is no better language, best language is which you are familiar with and also it depends on what kind of app are you going to create. Sure, if it is desktop application js won`t be the best choice, and if you are creating web tool c++ won`t be too.

 

[ReverseEngineering]

Considering the required speed, safety (closed source, not to be decompiled), stability, platform independence and scalability I would use C++.

There are also many C/C++ libraries available for regular tasks like HTTP requests, cookies, ...

If I were you I would also use the private (!) Twitter API behind the Twitter iOS app, since the security limitations are much lower there.

 

[BertIsFat]

I am not a coder and tried something similar recently, had some pretty great results with free chatGPT.

 

[ReverseEngineering]

I am not a coder and tried something similar recently, had some pretty great results with free chatGPT.

Yes, nice. But I'm 100 % sure that ChatGPT won't be able to reverse engineer protected apps, rebuild HTTP requests and code bots for it during the next years. ;-)

 

[thedorf]

What would you say is the best programming language or tool for this task?

I would say Python - especially if you are a beginning programmer or new to scraping.

 

[Sickfly]

I‘m using python. Best to start with, and yeah chatGPT give some great inspirations for beginners

 

[plugyy]

BingAI gives amazing details. Python is what I use for scraping on Twitter.

 

[BertIsFat]

Yes, nice. But I'm 100 % sure that ChatGPT won't be able to reverse engineer protected apps, rebuild HTTP requests and code bots for it during the next years. ;-)

100% correct but since OP seems to be a beginner like me and primarily wants to scrape some data now instead of going through the learning path of a programming language.

But if someone would want to go the C++ road, where would you recommend to start to learn for advanced botting purposes and using "private" api's?

 

[ReverseEngineering]

where would you recommend to start to learn for advanced botting purposes and using "private" api's?

Honestly, I don't know any single video course or large tutorial about it. The problem is that the app owners do prefer to take everything down. Like it happened to many GitHub repositories.

Some outdated medi*** and blog posts exist, but you have to read like 100 to 200 of them to combine all the knowledge.

But you should start with understanding how SSL pinning works, how to bypass it at iOS/Android. Unfortunately it's a permanent cat and mouse fight between app owners and bot coders.

 

[atlas_http]

JavaScript and Node.js have a lot of libraries for doing this on npm. There are libraries for dealing with TLS, making secure asynchronous HTTP requests, parsing the HTML, etc. All of these would fit your needs.

 

[voidtreader]

Most language would do but python would be a good choice. I made some scrapping bots not too long ago using the scrappy library and it was working fine

 

[dollysheep]

Python + lxml lib

 

[Duda Soft]

I tried scraping with Nodejs and then converted it to the Golang which is incredibly faster than other languages. I didn't try Python yet.

 

[robertcodelife]

Python

 

[Finally Blue]

Python I guess