Who's scraping twitter?

[bartosimpsonio]

Are any of you scraping twitter in the Elon era? What tools do you use? Since the API is now also out of reach for most, I assume there's a new method to scrape them?

 

[gigateub]

I don't know who it was today, but he broke X

 

[Lukmat]

Are any of you scraping twitter in the Elon era? What tools do you use? Since the API is now also out of reach for most, I assume there's a new method to scrape them?

Dolphin + Selenium custom driver + python worked 3 months ago

 

[MR.Spuf]

Are any of you scraping twitter in the Elon era? What tools do you use? Since the API is now also out of reach for most, I assume there's a new method to scrape them?

I saw lots of tools (very cheap) on Apify. Lots of scrapers with good reviews. I haven't used it, I've used it only the Instagram scrapers and they're good so I suspect the other ones are good too.

 

[rocktheboat]

I think brightdata also has a lot of data on twitter

 

[Proxy Cake]

Python with residential proxies is all you need pretty much

 

[bartosimpsonio]

I don't know who it was today, but he broke X

LOL wasn't me
Looks like their intern submitted some broken code

Dolphin + Selenium custom driver + python worked 3 months ago

Sounds like THE setup. Does dolphin avoid selenium detection?

I saw lots of tools (very cheap) on Apify. Lots of scrapers with good reviews. I haven't used it, I've used it only the Instagram scrapers and they're good so I suspect the other ones are good too.

Will take a look

SociAIpro.com

Do they sell data sets or scraping tools?

I think brightdata also has a lot of data on twitter

How much would a terabyte data set cost?

Python with residential proxies is all you need pretty much

Straight python, no selenium? How do you scrape their API?

 

[Lukmat]

Sounds like THE setup. Does dolphin avoid selenium detection?

Dolphin guys made custom chrome drivers to avoid selenium reveal

 

[imccafey]

As with most things, custom scrapers. Explore your options when it comes to tools but consider having one custom made too, especially if you can or know the right people who could.

 

[bartosimpsonio]

Dolphin guys made custom chrome drivers to avoid selenium reveal

Definitely gonna test this out on some projects

 

[NodeMaven]

Are any of you scraping twitter in the Elon era? What tools do you use? Since the API is now also out of reach for most, I assume there's a new method to scrape them?

Yup, still scraping Twitter — not with the API. Puppeteer + stealth + good resi proxies with sticky sessions is the move now. Fast rotations get you flagged quick. Think slow, human-like, headless browser vibes.

 

[bartosimpsonio]

Yup, still scraping Twitter — not with the API. Puppeteer + stealth + good resi proxies with sticky sessions is the move now. Fast rotations get you flagged quick. Think slow, human-like, headless browser vibes.

Great tips! Definitely gonna play around with Puppeteer

 

[crankocial]

Are any of you scraping twitter in the Elon era? What tools do you use? Since the API is now also out of reach for most, I assume there's a new method to scrape them?

I do tg: @sosocialx

 

[samsymian]

I used tweetscrape, it offers free trial aswell

 

[V]

Dolphin guys made custom chrome drivers to avoid selenium reveal

I did not know about this. This is awesome. I use Selenium for a number of tasks, so do you think it won't be too difficult to use this solution?

 

[Deleted member 15724]

I did not know about this. This is awesome. I use Selenium for a number of tasks, so do you think it won't be too difficult to use this solution?

The documentation isn't the best, but once you figure it out, it's pretty simple. All you need is to connect their custom chromedriver with your browser ID and you can use selenium like normal

If I had to start over I'd do it in playwright though. The chromedriver setup is a bit messy. I ran into an issue where they updated the browser version and never updated the chromedriver version (it has to be their custom version). I had to add a bunch of try/excepts just to get my bot running again and wait over a week for them to update. Never again

 

[Lukmat]

I did not know about this. This is awesome. I use Selenium for a number of tasks, so do you think it won't be too difficult to use this solution?

No, dolphin have modified chrome driver to hide automation, they have link in docs to download and exchange in your script

 

[llmafk]

I think x’s graphql endpoints still serve the timeline and search payloads if you supply a valid bearer token. You will need to verify but if so, you could spin up a headless Chromium, grab the bearer + guest tokens, then use requests or axios to do all subsequent graphQL calls.