Scraping Reddit and proxies/VPNs

Scraping Reddit with datacenter proxy IPs seems to be picked up and blocked by Reddit unless you are logged into an account. Residential proxies may work for this, but is there a better way (or cheaper)?

I would be fine creating an account on there to do it, but I don't want to create it on my own ip. There are accounts you can buy but I hate to spend too much just to scrape 300-400 pages on there.

Are there any other options?

Are you using Python? For the scrapper?

Arpito said:

Are you using Python? For the scrapper?

Click to expand...

yep, using Selenium. Using things like wget I get a 403 blocked error even setting a user agent if I do it from a datacenter. I'm very familiar with quite a few programming languages if you had ideas that use a different language.

Zooker said:

yep, using Selenium. Using things like wget I get a 403 blocked error even setting a user agent if I do it from a datacenter. I'm very familiar with quite a few programming languages if you had ideas that use a different language.

Click to expand...

How many requests you get by scrap section?

you run the script, how many requests you get before the script stop?

And do you have limited “scrap until X results”?

Maybe that logic can help you

On the first request I get the reddit page "can we see some id please" where it says to log in. Even launching a new chromium browser set to use a proxy it shows up immediately. I tried quite a few of my proxies which are datacenter ones admittedly so I don't get anywhere that way. Looking on the developer console I see reddit serves a 403. So using wget and such and setting a proxy shows a 403 as well.

I did just see they have an api, but didn't they fairly recently go from some amount of free usage to being overpriced due to ai scraping?

You'll definitely need to use residential proxies to scrape Reddit as well as a stealth version of puppeteer

Cant you use PRAW for this?