1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Is this worthy? Scrapper for videos and files

Discussion in 'BlackHat Lounge' started by bae, Sep 9, 2016.

  1. bae

    bae Newbie

    Joined:
    Sep 9, 2016
    Messages:
    28
    Likes Received:
    26
    Gender:
    Male
    Location:
    Portugal
    Hi guys,

    I'm unemployed for half year, i'm 25 but worked 5 years. So i started to build my own PBN. I already write a lot about programming and sec but never earned money from it. Now i'm exploring the movies streaming and started this project.

    It's a tool to scrap videos, albums, softwares from websites written in PHP. It's configured with an API (for now CLI only but i can also write WP plugins and Windows/Linux softwares) to manage websites (to scrap from and publish to). It has workers running in background (queue system has a limit for each queue) executing the queues to fetch new posts from sites, download the files and upload. It's fuly automated, fetches new content when there's none to download/upload and keep a limit on running workers. Only need a cron each 10 minutes to see if there's any queue to create a new worker and execute it.

    To scrap the content, each website has his own filters to each scrapper to scrap title, excerpt, images, etc. I'm even creating custom fields in wordpress getting info like filesize, imdb, etc allowing it to be implemented with the theme.

    I'm trying to extend it as much as possible: download providers, upload providers, fetch providers and queue providers. I'll also invest on documentation and i'm writting unit tests. For now the queue is only working with filesystem but i'll implement redis to scale this tool.

    I've a cache system, non sql database and queue all in filesystem. With the providers it can be extented but i think people will prefer it if it's easy to use: download and execute.


    I've searched a lot and found out many stuff to do this but hey, i'm broke living at my parents house without the possibility to contribute in the house bills so i decided to give it a try. If i can't get money from this on the upcoming weeks i'll have to emigrate. If in 2 months i'm able to earn at least a minimum average salary with it i'll pay a debt i've and publish this code in MIT license.

    Is this worthy? I'm doing this on a specific country with low competitors on streamming, so i'm reuploading all their translated stuff.

    PS: my favourite stuff is Transmission Service. For now it only interacts with transmission to add downloads but i think it would be cool to scrap trackers, get trending files and uploading them to several 2.0.
    I would love to be able to follow the if this than thet approach. If X site publish, then reupload to N and Y.

    [​IMG]
     
    Last edited: Sep 9, 2016