Scraping every subreddit on Reddit

webguy2000 · Apr 12, 2021

Currently the only way I know to get a list of subreddits is from the Community directory page. This contains about 40,000 subreddits, the problem is, theres millions more that are NOT listed in here because of their age and subscriber count. Is there any other way to get a list of all subreddits on the entire site?

Oblock · Apr 12, 2021

Use a good scraper like scrapy

GoGuerilla · Apr 12, 2021

Pushshift Dataset will have them

GoGuerilla · Apr 12, 2021

Can't edit, but here:

https://files.pushshift.io/reddit/subreddits/
Not exactly up to date but close enough

toysonline · Apr 12, 2021

GoGuerilla said:
Can't edit, but here:

https://files.pushshift.io/reddit/subreddits/
Not exactly up to date but close enough

thanks for this...

Bots-Scrapers · Apr 12, 2021

How about reddit list?

spectrejoe · Apr 12, 2021

webguy2000 said:
Currently the only way I know to get a list of subreddits is from the Community directory page. This contains about 40,000 subreddits, the problem is, theres millions more that are NOT listed in here because of their age and subscriber count. Is there any other way to get a list of all subreddits on the entire site?

Just scrape /r/all new posts and extract the subreddit links. All new posts get shown there so the longer u scrape the more unique subdomains you will get.

Thats what i do. I made a bot that will search for queries being posted there and will reply to them with my url if they contain certain keywords

DecentLad · Apr 12, 2021

GoGuerilla said:
Can't edit, but here:

https://files.pushshift.io/reddit/subreddits/
Not exactly up to date but close enough

How do u read the fuckin file, cant it be stored in plaintext? or am i just stupid:/

MountProxies · Apr 13, 2021

webguy2000 said:
Currently the only way I know to get a list of subreddits is from the Community directory page. This contains about 40,000 subreddits, the problem is, theres millions more that are NOT listed in here because of their age and subscriber count. Is there any other way to get a list of all subreddits on the entire site?

Have you given a thought on getting a custom scraper built? You might need to hire a developer for this - the idea seems pretty neat though, being able to scrape subreddits.

Scraping every subreddit on Reddit

webguy2000

Power Member

Oblock

Banned - Promoting other groups or forums

GoGuerilla

Power Member

GoGuerilla

Power Member

toysonline

Power Member

Bots-Scrapers

Elite Member

spectrejoe

Elite Member

DecentLad

Junior Member

MountProxies

Junior Member

Main Menu

Marketplace

Making Money

BlackHat World