1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Looking for 9gag , reddit scraper

Discussion in 'Black Hat SEO' started by wriper, Feb 23, 2016.

  1. wriper

    wriper Senior Member

    Joined:
    Jul 31, 2011
    Messages:
    1,193
    Likes Received:
    312
    Hey guys,

    Do you know about some tools which will scrape content from 9gag and reddit?

    But not all posts only posts with lets say 10 000 points.


    Thank for every hint
     
  2. accelerator_dd

    accelerator_dd Jr. VIP Jr. VIP

    Joined:
    May 14, 2010
    Messages:
    2,448
    Likes Received:
    1,009
    Occupation:
    SEO
    Location:
    IM Wonderland
    your best bet is getting a bot coded. I dont think there are tools out there already that do that.
     
    • Thanks Thanks x 1
  3. blackbeans

    blackbeans Jr. VIP Jr. VIP

    Joined:
    Nov 29, 2008
    Messages:
    1,363
    Likes Received:
    239
    Occupation:
    Your Secret Weapon
    Home Page:
    Based on the specifications of your project, it looks very interesting indeed.

    I really like the specs that you laid out in terms of quality filtration.

    You might even want to use CROSS REFERENCING features.

    What I mean by that is you can have the scraper.

    Look at how many points a particular piece of content has and then cross reference it with Facebook shares and other indicators of social media viralizations.

    With that said, I don't think any official tool exists that has the features you're looking for.

    You may want to go to places like Upwork, and hire a custom coder for this job.

    If done right, you can make quite a bit of money off this content by simply republishing it on Facebook.
     
  4. NSMITH15

    NSMITH15 Newbie

    Joined:
    Sep 11, 2013
    Messages:
    20
    Likes Received:
    3
    It's essentially automating what those viral sites pay people to do lol

    I love going on reddit and seeing most of the top r/all posts on Facebook later that night.
     
  5. patadeperro

    patadeperro Registered Member

    Joined:
    Jul 5, 2011
    Messages:
    63
    Likes Received:
    18
    I cant post links yet, but in the Download section of the forum there is a nice image scrapper for reddit
     
  6. Sordalumni

    Sordalumni Newbie

    Joined:
    Feb 20, 2016
    Messages:
    14
    Likes Received:
    2
    This is really simple to do, at least with Reddit. They provide a RSS version of all subreddits like this:

    h-t-t-p || rss.reddit.[TLD]/r/funny (can't post links)

    This is enough that you can fetch it and parse it in a Perl or Python script.
    Furthermore, you can refine your query like this:

    h-t-t-p || rss.reddit.[TLD]/r/funny/top/?sort=top&t=all&count=100&limit=100 (can't post links)

    Let's see what's going on here:

    rss.reddit.[TLD]
    - Fetch the RSS version of Reddit, which comes formatted as an XML document.
    /r/funny
    - The subreddit
    /top/ - The filter: Can also be "Hot", "New", or "Rising"
    ?sort=top&t=all - Getting the "top" results of "all time"
    &count=100 - Get the posts after this number in the listing. In the above query, post listings start at number 101.
    limit=100 - The number of posts to return. Default is 25. Maximum is 100.

    Source: h-t-t-p-s || reddit.[TLD]/dev/api (can't post links)

    I know it's the API documentation and I'm not sure I would use the API when building the scraper, but a lot of the commands work as GET variables in the URL as well.

    I'm sorry for the messed up links. I'm new here and can't post them... if a mod sees this, maybe consider cleaning it up a bit for readability.
     
    Last edited: Mar 1, 2016