1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Need advice in setting up a file sharing website.

Discussion in 'BlackHat Lounge' started by NX_NULL, Nov 3, 2014.

  1. NX_NULL

    NX_NULL Regular Member Premium Member

    Joined:
    Dec 31, 2008
    Messages:
    309
    Likes Received:
    425
    I am going to run a file sharing website.

    need advice on setting server configuration to avoid bottle neck and be able to handle the traffic when the website grows.

    below is the server I am going to get

    CPU: DUAL INTEL Xeon E5-2620 v2 2100MHz 15MB Cache 6Core
    RAM: 64 GB
    PORT: 1 Gbps public port& 1 Gbps private port
    HDD: 2 x 4 TB

    I can add up to 32HDD to that server.

    What raid to you suggest me to use to be able to expand it with more hard drive when the websites grow?

    Is 4TB hard drives a good choice or its too big when it become filled with thousands of files it become too slow?

    How many hard drives can this server handle and the performance stay at acceptable rate?
     
  2. Repulsor

    Repulsor Power Member

    Joined:
    Jun 11, 2013
    Messages:
    712
    Likes Received:
    267
    Location:
    PHP Scripting ;)
    Sorry for not being of help, but my main concern is how are you planning to monetize? Or you just want it to grow big now, and monetize later once its famous? Just curious.
     
  3. ShadeDream

    ShadeDream Elite Member

    Joined:
    Nov 27, 2008
    Messages:
    2,209
    Likes Received:
    5,230
    Location:
    He who laughs last, laughs longest.
    Doesn't sound like you know what you're doing, I would suggest much more research before you start anything, unless of course you've got money to burn.

    I would also advice to look around forums dedicated to hosting as BHW will probably give you limited responses.
     
  4. NX_NULL

    NX_NULL Regular Member Premium Member

    Joined:
    Dec 31, 2008
    Messages:
    309
    Likes Received:
    425
    Monetizing will be done by showing advertisement and selling premium accounts. this niche is too competitive and I am not expecting to cover my expenses in first year, but its a kind of investment.
     
  5. Repulsor

    Repulsor Power Member

    Joined:
    Jun 11, 2013
    Messages:
    712
    Likes Received:
    267
    Location:
    PHP Scripting ;)
    Just a friendly advice, I have started forums before only to die after few months. This seems to play with x times the money a startup forum needs. So think twice before you are spending the first dollar on it.

    And once you have thought about it twice, think about it once more before you do.
     
    • Thanks Thanks x 2
  6. jazzc

    jazzc Moderator Staff Member Moderator Jr. VIP

    Joined:
    Jan 27, 2009
    Messages:
    2,468
    Likes Received:
    10,148
    You left out the most important thing - Traffic / month ;)

    Also, I don't see the point of the 64GB RAM. What are you going to do with all there doing nothing? I'd go for multiple "normal" servers, with 4x2TB drives (for RAID, so 2TB per server effectively).
     
    • Thanks Thanks x 1
  7. ShadeDream

    ShadeDream Elite Member

    Joined:
    Nov 27, 2008
    Messages:
    2,209
    Likes Received:
    5,230
    Location:
    He who laughs last, laughs longest.
    I would imagine he has no clue about this.
     
  8. NX_NULL

    NX_NULL Regular Member Premium Member

    Joined:
    Dec 31, 2008
    Messages:
    309
    Likes Received:
    425
    64GB RAM comes with server by default.
    Mutli servers outperform a single strong server but their price will become higher in this datacenter.
    I was wondering if there wouldnt be any bottleneck If I use large HDDs with Raid


    no i am not clueless, I am doing my researches and I am fully aware of the risk.
    by the way thanks for your caution and you are free to go to other threads and let other members benefit from your wisdom too!
     
  9. AmateRasu

    AmateRasu Power Member

    Joined:
    Nov 12, 2013
    Messages:
    755
    Likes Received:
    423
    Location:
    Manila,Philippines
    if you are planning to host files 1 GBPS is not enough and the download speed on your network will be terrible too :) it would be nice if will be using 2 x 10 GBPS running on LACP (load balancer) :) if you are curious with the price here it is :

    [​IMG]
     
    • Thanks Thanks x 1
  10. ShadeDream

    ShadeDream Elite Member

    Joined:
    Nov 27, 2008
    Messages:
    2,209
    Likes Received:
    5,230
    Location:
    He who laughs last, laughs longest.
    I always do, you're welcome. Jokes aside, I'm just critical didn't mean to be rude. Sorry about that.
     
  11. FBGuru

    FBGuru Senior Member

    Joined:
    Sep 22, 2013
    Messages:
    928
    Likes Received:
    1,171
    Location:
    Personality Type : ESTP
    There are a couple of things you've to consider when choosing your server.

    1. Do you allow illegal content? For ex, adult videos/cracked softwares/pirated movies/copyrighted music albums.
    2. If your answer to any of the above options is Yes, your site/server must be hosted offshore. I.e, on a country that doesn't give a shit about US legislations.
    3. If your answer to all of the above options is NO, you should have an active team to monitor and take down the reported links but it's going to be one hell of a task when your site grows bigger. Although you can deal it with automation.
    4. The server is simply going to respond to the GET requests, so you don't need that much processing power. Same thing applies to memory. 64GB memory is overkill for a startup.
    5. You can better spend that money on bigger ports and bandwidth.
    6. Host the Database on an SSD server running on same node/network/array of your web server so there's low to zero latency.
    7. 4TB HDD's have much higher failure rates than the 3TB's and 3TB's have much higher failure rates than the 2TB's so you may have to stick with the 2TB's in a raid 10 setup to have solid redundancy. You don't want your users to complain about the lost data on the interweb.
    8. Get a dedicated RAID card to add the HDDs when your site grows.

    Well, that's a little more than a couple of things, but you need to seriously think about doing this, especially when you're the only investor to run this whole project for a year. Megaupload rocked the world of file sharing sites in the past, but it's nowhere as near after they re-launched MEGA with lots of cool security features and 50GB FREE storage. Look at all these filesharing sites that you see on multiuploads/uploadseeds. What's your selling point that makes your site special and differentiate between file hosting sites that are up for the last 5 years?

    As a food for thought, you can consider about building a cloud hosting site similar to box/Dropbox/sugarsync/Google drive, but it's going to set you back with a 5 figure budget even if you want to make it as cheap as possible. But if you somehow get it done and grow some audience, you're gonna get tons of investors and with the help of BH Knowledge, you can easily generate the Buzz around the WWW to increase your profit.

    By saying all this, I suggest you start small with a normal server maybe with 4TB RAID10 storage and 16Gigs of RAM and get the script done and set the site go live. You will get lots of insights once the site starts kicking. You can come back then to get more proper guidance to scale-up your setup.
     
    • Thanks Thanks x 1
  12. stharthjw6j

    stharthjw6j BANNED BANNED

    Joined:
    Dec 23, 2013
    Messages:
    588
    Likes Received:
    250
    ^^^Funnily, the Napster story opened the gateways to show how people CAN do this, just from knowing what went wrong with them, and now it's all over the place.
     
  13. Mr.Whitehat

    Mr.Whitehat Senior Member

    Joined:
    Apr 23, 2009
    Messages:
    855
    Likes Received:
    220
    Occupation:
    Wandering Around !
    Location:
    Dating Moolah Babe^
    Starting a file host isnt so simple op. There are plenty things to consider regarding script, bug fixes, hack attempts, ddos and more. Moreover it will ran out of space if you run an affiliate program and if u aint gonna run an aff program, none will upload and share a single file.
     
  14. imdarealboss

    imdarealboss Newbie

    Joined:
    Nov 8, 2013
    Messages:
    5
    Likes Received:
    0
    hi friend i can help you im it expert i can help you config your server and raid 5 i would recomend and as u grow raid 10, skype williamupper lets chats some more
     
  15. Repulsor

    Repulsor Power Member

    Joined:
    Jun 11, 2013
    Messages:
    712
    Likes Received:
    267
    Location:
    PHP Scripting ;)
    HIRE THIS GUY :p
     
  16. NX_NULL

    NX_NULL Regular Member Premium Member

    Joined:
    Dec 31, 2008
    Messages:
    309
    Likes Received:
    425
    thanks for your great in-depth response
    what server spec do you suggest for database?
     
  17. FBGuru

    FBGuru Senior Member

    Joined:
    Sep 22, 2013
    Messages:
    928
    Likes Received:
    1,171
    Location:
    Personality Type : ESTP
    For the normal server setup with 4TB RAID10 and 16GB RAM, a 64GB SSD server with 4/8 Gigs of RAM will suffice for the MySQL server.

    All my web servers are running NGINX and VARNISH with dedicated MySQL server plus another server for caching via Memcached and my sites hardly takes more than 1.2 secs under heavy load. Usually its less than 600ms. My server can handle 7 million+ hits/hour with just 4GB RAM. I suggest you too to go with the NGINX server.
     
    • Thanks Thanks x 1