Scraping every subreddit on Reddit

[webguy2000]

Currently the only way I know to get a list of subreddits is from the Community directory page. This contains about 40,000 subreddits, the problem is, theres millions more that are NOT listed in here because of their age and subscriber count. Is there any other way to get a list of all subreddits on the entire site?

 

[Oblock]

Use a good scraper like scrapy

 

[GoGuerilla]

Pushshift Dataset will have them

 

[GoGuerilla]

Can't edit, but here:

https://files.pushshift.io/reddit/subreddits/
Not exactly up to date but close enough

 

[toysonline]

Can't edit, but here:

https://files.pushshift.io/reddit/subreddits/
Not exactly up to date but close enough

thanks for this...

 

[Bots-Scrapers]

How about reddit list?

 

[spectrejoe]

Currently the only way I know to get a list of subreddits is from the Community directory page. This contains about 40,000 subreddits, the problem is, theres millions more that are NOT listed in here because of their age and subscriber count. Is there any other way to get a list of all subreddits on the entire site?

Just scrape /r/all new posts and extract the subreddit links. All new posts get shown there so the longer u scrape the more unique subdomains you will get.


Thats what i do. I made a bot that will search for queries being posted there and will reply to them with my url if they contain certain keywords

 

[DecentLad]

Can't edit, but here:

https://files.pushshift.io/reddit/subreddits/
Not exactly up to date but close enough

How do u read the fuckin file, cant it be stored in plaintext? or am i just stupid:/

 

[MountProxies]

Currently the only way I know to get a list of subreddits is from the Community directory page. This contains about 40,000 subreddits, the problem is, theres millions more that are NOT listed in here because of their age and subscriber count. Is there any other way to get a list of all subreddits on the entire site?

Have you given a thought on getting a custom scraper built? You might need to hire a developer for this - the idea seems pretty neat though, being able to scrape subreddits.