Looking for 9gag , reddit scraper

Hey guys,

Do you know about some tools which will scrape content from 9gag and reddit?

But not all posts only posts with lets say 10 000 points.


Thank for every hint

your best bet is getting a bot coded. I dont think there are tools out there already that do that.

Based on the specifications of your project, it looks very interesting indeed.

I really like the specs that you laid out in terms of quality filtration.

You might even want to use CROSS REFERENCING features.

What I mean by that is you can have the scraper.

Look at how many points a particular piece of content has and then cross reference it with Facebook shares and other indicators of social media viralizations.

With that said, I don't think any official tool exists that has the features you're looking for.

You may want to go to places like Upwork, and hire a custom coder for this job.

If done right, you can make quite a bit of money off this content by simply republishing it on Facebook.

blackbeans said:

Based on the specifications of your project, it looks very interesting indeed.

I really like the specs that you laid out in terms of quality filtration.

You might even want to use CROSS REFERENCING features.

What I mean by that is you can have the scraper.

Look at how many points a particular piece of content has and then cross reference it with Facebook shares and other indicators of social media viralizations.

With that said, I don't think any official tool exists that has the features you're looking for.

You may want to go to places like Upwork, and hire a custom coder for this job.

If done right, you can make quite a bit of money off this content by simply republishing it on Facebook.

Click to expand...

It's essentially automating what those viral sites pay people to do lol

I love going on reddit and seeing most of the top r/all posts on Facebook later that night.

I cant post links yet, but in the Download section of the forum there is a nice image scrapper for reddit

This is really simple to do, at least with Reddit. They provide a RSS version of all subreddits like this:

h-t-t-p || rss.reddit.[TLD]/r/funny (can't post links)

This is enough that you can fetch it and parse it in a Perl or Python script.
Furthermore, you can refine your query like this:

h-t-t-p || rss.reddit.[TLD]/r/funny/top/?sort=top&t=all&count=100&limit=100 (can't post links)

Let's see what's going on here:

rss.reddit.[TLD] - Fetch the RSS version of Reddit, which comes formatted as an XML document.
/r/funny - The subreddit
/top/ - The filter: Can also be "Hot", "New", or "Rising"
?sort=top&t=all - Getting the "top" results of "all time"
&count=100 - Get the posts after this number in the listing. In the above query, post listings start at number 101.
limit=100 - The number of posts to return. Default is 25. Maximum is 100.

Source: h-t-t-p-s || reddit.[TLD]/dev/api (can't post links)

I know it's the API documentation and I'm not sure I would use the API when building the scraper, but a lot of the commands work as GET variables in the URL as well.

I'm sorry for the messed up links. I'm new here and can't post them... if a mod sees this, maybe consider cleaning it up a bit for readability.