How to scrape long Reddit threads?

[jolamo]

Is there an easy way to scrape all of the comments & nested comments for really long Reddit threads with over 1000 comments?

 

[pancholipratik]

For this you'll need a custom bot. And a lot of proxies.

 

[RedditHelper]

Yes you can create a python script for that that will scrape all the reddit comment and gives you a pdf of that

 

[buyupvotes]

Use firecrawl.

 

[imccafey]

Probably trickier for long threads, doubt there's a a one-button solution for it. You'll probably want a custom setup using the API or something built on top of a scraper to do those comments under "load more" for full threads. Most commercial tools likely wouldn't handle that level of depth right.

 

[muhbootloader]

If you want the easiest route, I’d probably look at the Reddit API first, but for really deep threads it can get annoying fast with collapsed replies. Pushshift used to help a lot for this stuff too, though I’m not sure how reliable it is now.

 

[Cloud Node]

Use Apify if you want it done within 5 mins without writing any code.You can also use PRAW if you wnt a free permanent solution integrate into your supabase workflow.

 

[CyHead]

You could easily do it with puppeteer or just a userscript depending on the volume of scraping you need to do.

 

[hossen10]

You can use Reddit API, THIS is the best Way.

 

[Houcinovic]

you can scrape all comments, but the nested replies are tricky to scrape cause they consume a lot of api requests (if you use official reddit API), depending on how deep they are.


you can search on rapidapi marketplace for reddit scrapers

 

[Rikachu]

To download long reddit threads without going crazy, the most direct way is to use the reddit page itself with a simple trick, if you take any URL from the thread and add something to the end, the page will return the entire post structure with the main comments, the problem is that in very long threads, Reddit doesn't show you all the replies at once; it loads them in batches.

For huge threads, you can use a regular reddit account and a python tool called PRAW, with it, you can ask reddit to replace the load more comments buttons and retrieve the entire tree of replies and sub replies, however, if the thread has five thousand comments, this can take quite a while because each request to reddit takes time.

For very old or massive threads, there are external services that archive historical comments and allow you to consult them without the strict limitations of the official API, but for everyday use, having an established account and using reddit's tools from a well configured browser is sufficient, the important thing is not to submit queries too quickly in a row, as reddit may temporarily block you, patience and taking breaks are key.

 

[JinkHost]

Look on Apify if there are any finished scrapers for your purpose, or get someone to create you one if you don't know how to code or vibe code.

 

[phoscus]

You can use the official API or trusted scraping tools that support paginated comment extraction, since long threads require handling nested replies and rate limits properly.

 

[Danielcross]

Is there an easy way to scrape all of the comments & nested comments for really long Reddit threads with over 1000 comments?

Not really a simple way to do it manually especially with how comments are loaded, most structured methods handle this better than basic scraping

 

[joepistriad]

is there a way to avoid hitting reddit's rate limit when scraping over 1000 comments like that?

 

[XxTinkerbelxX]

Easiest way I’ve found is using the Reddit API (via PRAW). It can pull full threads including nested comments, but you need to handle pagination properly. Not exactly one click, but once set up it’s solid.

 

[HydraProxy]

For deep threads with tons of nested comments, standard web scraping via Selenium or BeautifulSoup will break when hitting "load more comments" buttons.

The easiest route is using PRAW (Python Reddit API Wrapper) and calling submission.comments.replace_more(limit=None). It automatically expands and fetches all nested, deep-level comments for you without losing the thread hierarchy.

 

[Steptoe]

Have a look at the 'Reddit .json trick' and enjoy it while it lasts!