Need advice in setting up a file sharing website.

[NX_NULL]

I am going to run a file sharing website.

need advice on setting server configuration to avoid bottle neck and be able to handle the traffic when the website grows.

below is the server I am going to get

CPU: DUAL INTEL Xeon E5-2620 v2 2100MHz 15MB Cache 6Core
RAM: 64 GB
PORT: 1 Gbps public port& 1 Gbps private port
HDD: 2 x 4 TB

I can add up to 32HDD to that server.

What raid to you suggest me to use to be able to expand it with more hard drive when the websites grow?

Is 4TB hard drives a good choice or its too big when it become filled with thousands of files it become too slow?

How many hard drives can this server handle and the performance stay at acceptable rate?

 

[Repulsor]

Sorry for not being of help, but my main concern is how are you planning to monetize? Or you just want it to grow big now, and monetize later once its famous? Just curious.

 

[ShadeDream]

Doesn't sound like you know what you're doing, I would suggest much more research before you start anything, unless of course you've got money to burn.

I would also advice to look around forums dedicated to hosting as BHW will probably give you limited responses.

 

[NX_NULL]

Sorry for not being of help, but my main concern is how are you planning to monetize? Or you just want it to grow big now, and monetize later once its famous? Just curious.

Monetizing will be done by showing advertisement and selling premium accounts. this niche is too competitive and I am not expecting to cover my expenses in first year, but its a kind of investment.

 

[Repulsor]

Monetizing will be done by showing advertisement and selling premium accounts. this niche is too competitive and I am not expecting to cover my expenses in first year, but its a kind of investment.

Just a friendly advice, I have started forums before only to die after few months. This seems to play with x times the money a startup forum needs. So think twice before you are spending the first dollar on it.

And once you have thought about it twice, think about it once more before you do.

 

[jazzc]

You left out the most important thing - Traffic / month

Also, I don't see the point of the 64GB RAM. What are you going to do with all there doing nothing? I'd go for multiple "normal" servers, with 4x2TB drives (for RAID, so 2TB per server effectively).

 

[ShadeDream]

You left out the most important thing - Traffic / month

I would imagine he has no clue about this.

 

[NX_NULL]

You left out the most important thing - Traffic / month

Also, I don't see the point of the 64GB RAM. What are you going to do with all there doing nothing? I'd go for multiple "normal" servers, with 4x2TB drives (for RAID, so 2TB per server effectively).

64GB RAM comes with server by default.
Mutli servers outperform a single strong server but their price will become higher in this datacenter.
I was wondering if there wouldnt be any bottleneck If I use large HDDs with Raid

I would imagine he has no clue about this.

no i am not clueless, I am doing my researches and I am fully aware of the risk.
by the way thanks for your caution and you are free to go to other threads and let other members benefit from your wisdom too!

 

[AmateRasu]

if you are planning to host files 1 GBPS is not enough and the download speed on your network will be terrible too it would be nice if will be using 2 x 10 GBPS running on LACP (load balancer) if you are curious with the price here it is :

 

[ShadeDream]

no i am not clueless, I am doing my researches and I am fully aware of the risk.
by the way thanks for your caution and you are free to go to other threads and let other members benefit from your wisdom too!

I always do, you're welcome. Jokes aside, I'm just critical didn't mean to be rude. Sorry about that.

 

[FBGuru]

There are a couple of things you've to consider when choosing your server.

1. Do you allow illegal content? For ex, adult videos/cracked softwares/pirated movies/copyrighted music albums.
2. If your answer to any of the above options is Yes, your site/server must be hosted offshore. I.e, on a country that doesn't give a shit about US legislations.
3. If your answer to all of the above options is NO, you should have an active team to monitor and take down the reported links but it's going to be one hell of a task when your site grows bigger. Although you can deal it with automation.
4. The server is simply going to respond to the GET requests, so you don't need that much processing power. Same thing applies to memory. 64GB memory is overkill for a startup.
5. You can better spend that money on bigger ports and bandwidth.
6. Host the Database on an SSD server running on same node/network/array of your web server so there's low to zero latency.
7. 4TB HDD's have much higher failure rates than the 3TB's and 3TB's have much higher failure rates than the 2TB's so you may have to stick with the 2TB's in a raid 10 setup to have solid redundancy. You don't want your users to complain about the lost data on the interweb.
8. Get a dedicated RAID card to add the HDDs when your site grows.

Well, that's a little more than a couple of things, but you need to seriously think about doing this, especially when you're the only investor to run this whole project for a year. Megaupload rocked the world of file sharing sites in the past, but it's nowhere as near after they re-launched MEGA with lots of cool security features and 50GB FREE storage. Look at all these filesharing sites that you see on multiuploads/uploadseeds. What's your selling point that makes your site special and differentiate between file hosting sites that are up for the last 5 years?

As a food for thought, you can consider about building a cloud hosting site similar to box/Dropbox/sugarsync/Google drive, but it's going to set you back with a 5 figure budget even if you want to make it as cheap as possible. But if you somehow get it done and grow some audience, you're gonna get tons of investors and with the help of BH Knowledge, you can easily generate the Buzz around the WWW to increase your profit.

By saying all this, I suggest you start small with a normal server maybe with 4TB RAID10 storage and 16Gigs of RAM and get the script done and set the site go live. You will get lots of insights once the site starts kicking. You can come back then to get more proper guidance to scale-up your setup.

 

[stharthjw6j]

^^^Funnily, the Napster story opened the gateways to show how people CAN do this, just from knowing what went wrong with them, and now it's all over the place.

 

[Mr.Whitehat]

Starting a file host isnt so simple op. There are plenty things to consider regarding script, bug fixes, hack attempts, ddos and more. Moreover it will ran out of space if you run an affiliate program and if u aint gonna run an aff program, none will upload and share a single file.

 

[imdarealboss]

hi friend i can help you im it expert i can help you config your server and raid 5 i would recomend and as u grow raid 10, skype williamupper lets chats some more

 

[Repulsor]

hi friend i can help you im it expert i can help you config your server and raid 5 i would recomend and as u grow raid 10, skype williamupper lets chats some more

HIRE THIS GUY

 

[NX_NULL]

There are a couple of things you've to consider when choosing your server.

1. Do you allow illegal content? For ex, adult videos/cracked softwares/pirated movies/copyrighted music albums.
2. If your answer to any of the above options is Yes, your site/server must be hosted offshore. I.e, on a country that doesn't give a shit about US legislations.
3. If your answer to all of the above options is NO, you should have an active team to monitor and take down the reported links but it's going to be one hell of a task when your site grows bigger. Although you can deal it with automation.
4. The server is simply going to respond to the GET requests, so you don't need that much processing power. Same thing applies to memory. 64GB memory is overkill for a startup.
5. You can better spend that money on bigger ports and bandwidth.
6. Host the Database on an SSD server running on same node/network/array of your web server so there's low to zero latency.
7. 4TB HDD's have much higher failure rates than the 3TB's and 3TB's have much higher failure rates than the 2TB's so you may have to stick with the 2TB's in a raid 10 setup to have solid redundancy. You don't want your users to complain about the lost data on the interweb.
8. Get a dedicated RAID card to add the HDDs when your site grows.

Well, that's a little more than a couple of things, but you need to seriously think about doing this, especially when you're the only investor to run this whole project for a year. Megaupload rocked the world of file sharing sites in the past, but it's nowhere as near after they re-launched MEGA with lots of cool security features and 50GB FREE storage. Look at all these filesharing sites that you see on multiuploads/uploadseeds. What's your selling point that makes your site special and differentiate between file hosting sites that are up for the last 5 years?

As a food for thought, you can consider about building a cloud hosting site similar to box/Dropbox/sugarsync/Google drive, but it's going to set you back with a 5 figure budget even if you want to make it as cheap as possible. But if you somehow get it done and grow some audience, you're gonna get tons of investors and with the help of BH Knowledge, you can easily generate the Buzz around the WWW to increase your profit.

By saying all this, I suggest you start small with a normal server maybe with 4TB RAID10 storage and 16Gigs of RAM and get the script done and set the site go live. You will get lots of insights once the site starts kicking. You can come back then to get more proper guidance to scale-up your setup.

thanks for your great in-depth response
what server spec do you suggest for database?

 

[FBGuru]

thanks for your great in-depth response
what server spec do you suggest for database?

For the normal server setup with 4TB RAID10 and 16GB RAM, a 64GB SSD server with 4/8 Gigs of RAM will suffice for the MySQL server.

All my web servers are running NGINX and VARNISH with dedicated MySQL server plus another server for caching via Memcached and my sites hardly takes more than 1.2 secs under heavy load. Usually its less than 600ms. My server can handle 7 million+ hits/hour with just 4GB RAM. I suggest you too to go with the NGINX server.