Looking for FRESH ideas. High-Speed Truth Social Scraping

• Using Playwright to simulate real browsers (truthsocial JS required - site blocks non-JS requests)
• Truth Social has aggressive rate limiting: ~3-5 requests then 30-second timeout and kind of random
• Need to catch new posts the INSTANT they're published
• My current approach feels massively inefficient

• Bought residential rotating proxy pool (10k+ IPs)
• Use Playwright with device simulation
• Make requests every 1-2 seconds, rotating through IPs
• Hope to stay under rate limits by distributing across thousands of IPs

• Make requests every 1-2 seconds, rotating through IPs

lovingsoul said:

I'm trying to scrape Truth Social posts with near real-time accuracy (milliseconds delay) - basically simulating a live stream/socket connection. There's NO official API, so I'm stuck with web scraping.

Current Situation:

• Using Playwright to simulate real browsers (truthsocial JS required - site blocks non-JS requests)
• Truth Social has aggressive rate limiting: ~3-5 requests then 30-second timeout and kind of random
• Need to catch new posts the INSTANT they're published
• My current approach feels massively inefficient

My Current "Brute Force" Idea (that i'm developing):

• Bought residential rotating proxy pool (10k+ IPs)
• Use Playwright with device simulation
• Make requests every 1-2 seconds, rotating through IPs
• Hope to stay under rate limits by distributing across thousands of IPs

The Problem:
This seems incredibly wasteful and expensive. There must be a more efficient way (or that's what I hope)
Is my IP rotation approach completely wrong for this use-case?

I have found some tools that already do this (getting live-data from Truth Social, Apify....), some of them free to use with limits and some paying options. I would like to develop my own system that I will then wrap up with more things.

Any idea is really appreciated. Thank you

Click to expand...

Why the brute-force rotation is suboptimal​

• Costly: residential proxies at 10k IPs × many Playwright instances = huge monthly spend.
• Inefficient: repeating full page loads every 1–2s wastes CPU, network, and triggers rate limits easily.
• Fragile: site defenses (bot heuristics, fingerprinting, account throttles) will eventually detect and throttle.
• Latency ceiling: “milliseconds” is unrealistic via repeated HTTP polling. The only realistic low-latency approach is maintaining an open connection (browser/WebSocket/SSE) so updates push to you.

Better approach (big idea)​

• One persistent session ≈ instant detection of new DOM nodes/messages.
• Minimizes requests and proxy usage (you reuse the same session/IP/connection).
• Harder for server to rate-limit you (no repeated logins/loads).
• You can intercept underlying WebSocket frames (if present) for robust parsing.