Twitter Scrapping Risk Usinng Selenium
- Thread starter FlyingCow
- Start date
- Jul 2, 2020
- 806
- 658
Since you plan to scrape only the risk of getting banned is lower, because you don't "spam" X with massive posts.
Selenium needs a lot of resources, why don't you simply do HTTPS requests with a script? You don't need any browser emulation for scraping.
Selenium needs a lot of resources, why don't you simply do HTTPS requests with a script? You don't need any browser emulation for scraping.
nirose
Elite Member
- Oct 24, 2008
- 1,820
- 801
Because their website is javascript heavy where content is loaded by javascript calls and you'd get nothing out of a "request" other than placeholder html text. You'd also have to manage cookies and authentication for that request and Twitter API is expensive af.
BiggusD1ckus
Banned - Abuse of Staff
- Nov 15, 2023
- 375
- 194
any vcc provider you'd recommend for going blue?
Ashk881
Elite Member
- Aug 24, 2021
- 1,695
- 2,072
Because Twitter throws captchas randomly in between and it's rendered with Javascript. You can't scrape the content directly with plain http requests unless you are using the client-side api somehow.
- Jul 2, 2020
- 806
- 658
The trick is to do HTTPS requests based on the private X app API, not the website. Similar to TikTok, Instagram, Snapchat, ...
Sickfly
Registered Member
- Nov 3, 2023
- 64
- 30
Pst dot net, or flexcards dot cards.. but pst is easier
D
Deleted member 1197157
Guest
Hi, can you DM me on - @floweroflights on Telegram?
Veil123
Elite Member
- Jun 13, 2012
- 1,839
- 3,357
Try to use a modified library (e.g. https://github.com/ultrafunkamsterdam/undetected-chromedriver) as even though Twitter does not have any complex detection algorithms, they can detect the standard (unmodified) web driver.
- Jul 2, 2020
- 806
- 658
Sure.