Best methods for twitter scraping in 2025

[jamie3000]

Hi All,

There are around 1,000 twitter accounts I want to monitor the tweets of in as close to real-time as possible.

I've not done any Twitter/X automation for a long time and I now see the official API is heavily restricted. I also see the unofficial API that the browser uses also needs a logged in session for most things now.

I'm also told that Twitter/X is quite strict on browser automation, selenium etc.

What are my options here? I just want to load a Twitter/X list (the list feature) of 1,000 users and that will show me their tweets, but I'm being told that doing that every 30 seconds will get my account banned.

Is that true?

 

[Steptoe]

It's heavily gated now.
You could take a look at how Nitter is handling it: https://github.com/zedeus/nitter
Or piggyback some public Nitter instances: https://twiiit.com/

 

[jamie3000]

It's heavily gated now.
You could take a look at how Nitter is handling it: https://github.com/zedeus/nitter
Or piggyback some public Nitter instances: https://twiiit.com/

Thanks mate. What I wanted was very simple so my solution I made just now is very simple. I'm just using an extension to refresh the list page and download the rendered HTML haha. Embarrassingly simple but its all I need at the moment. I'll move it into a little local VM if it holds over the next few days.

 

[Steptoe]

Thanks mate. What I wanted was very simple so my solution I made just now is very simple. I'm just using an extension to refresh the list page and download the rendered HTML haha. Embarrassingly simple but its all I need at the moment. I'll move it into a little local VM if it holds over the next few days.

The easy solutions are always the best

 

[onlyzorro]

Hi All,

There are around 1,000 twitter accounts I want to monitor the tweets of in as close to real-time as possible.

I've not done any Twitter/X automation for a long time and I now see the official API is heavily restricted. I also see the unofficial API that the browser uses also needs a logged in session for most things now.

I'm also told that Twitter/X is quite strict on browser automation, selenium etc.

What are my options here? I just want to load a Twitter/X list (the list feature) of 1,000 users and that will show me their tweets, but I'm being told that doing that every 30 seconds will get my account banned.

Is that true?

You can track large lists by spacing out checks or using multiple accounts.

 

[Blue Money]

At that scale the safest approach is to slow the refresh and let the list update naturally. Batching checks and treating it more like a feed than a scraper tends to work fine long term.