How are you guys scraping reddit?

[kaun_dom]

Anyone here scraping reddit post without api as reddit are not giving thier api

 

[sochiba]

You can still scrape data directly through framework, it’s just a bit slower (or more work) to set up. Are you looking for a specific dataset?

 

[kaun_dom]

You can still scrape data directly through framework, it’s just a bit slower (or more work) to set up. Are you looking for a specific dataset?

Beautiful soap ?

I vibecoded reddit research skills for my openclaw long time back and now I am planning to work on it

 

[GeeLark AI]

Anyone here scraping reddit post without api as reddit are not giving thier api

You can use Python with requests and BeautifulSoup to scrape publicly accessible Reddit pages, but expect rate limits and layout changes.

Also for larger scale scraping, check Reddit’s current terms first, some third party tools also offer Reddit data without requiring you to manage the API yourself.

 

[shijin]

Still hitting the old .json endpoints with randomized residential proxies and 3-5 second delays, Reddit hasn’t fully killed that path yet and it pulls full post + comment trees without touching the official API

 

[noisor]

Anyone here scraping reddit post without api as reddit are not giving thier api

Custom python script with rotating residential proxies and cloudflare Solver. It is costly but there is no other way anymore.

 

[OnlineProxy]

basic requests get blocked almost instantly now, reddit detects that stuff way too easily. right now you pretty much have to emulate a full browser to pass their checks. it works but you will need powerful servers for the ram usage and really good proxies so you dont get banned right away

 

[Steptoe]

Set up some proxy scripts on Google Compute Engine instances - these get through pretty well.

 

[Utopic]

You can try using Python with BeautifulSoup, but Reddit has gotten pretty strict with scraping lately.

 

[Proxellor]

for your research tool i'd test a small batch of public posts first and log the actual responses. if Reddit starts returning 403s, throwing more proxies at it won't necessarily help. also BeautifulSoup only parses what you actually receive, it won't magically fetch missing comments.

 

[RichIndian]

Using floppydata's web unlocker with SeleniumBase, works well but hard to set up from the first try

 

[Mivox]

Depends on how much data you need. For a small project, I'd start with an existing scraping service instead of maintaining my own setup. Much easier to manage.

 

[Tiktiki]

You can pull the post data by appending .json to the Reddit URL and fetching it with a tool like curl.
Reddit serves a public JSON view of any page, so no API key is needed.

 

[Steptoe]

You can pull the post data by appending .json to the Reddit URL and fetching it with a tool like curl.
Reddit serves a public JSON view of any page, so no API key is needed.

Considering they're removing RSS feeds on Nov 13th I can see the .json trick being on the chopping board soon.

 

[V]

Set up some proxy scripts on Google Compute Engine instances - these get through pretty well.

Oooh, that's a nice idea, mate. You being sneaky with them, lol. Using their own poison to kill them.

 

[lukem121]

The .json endpoints still work but they’re the first thing to get rate limited, so the proxy quality matters more than the code.

 

[akaseo]

Anyone here scraping reddit post without api as reddit are not giving thier api

there are 3rd party apis present on apify plus if there is something specific you want to scrape and it's worth a few 100 dollars then i would say get an rdp and make a script that scrapes live in browser sends you the second you need something or your script or whatever data you wanted to scrape is found.

 

[Paradocs]

I scrape Reddit by requesting the .json version of each post URL.
It sidesteps the official API and returns the full post data in a simple JSON payload.

 

[bacman]

randomized residential proxies with few seconds delays

 

[xavierfok]

old.reddit.com html. this is my current setup. its stable and easy to parse. the catch is that some comments do not load fully.

my friend told me to add .json to any URL and u get clean data with the full comments. let me know if that works