Windows Server for high CPU tasks - is this even possible??

[examo1]

Hi, I am using a custom script which requires 40 browsers to be open/running at roughly the same time (multi-threaded) plus some background processes. The data on the page is continuously updated and each browser consumes around 0.45 GB in memory whilst running.

I am currently able to open around 6-8 browsers comfortably on my current machine, after which things start slowing down. My current specs are:

Intel Core i7 2600 CPU @ 3.40GHz
#Cores: 4
#Threads: 8
10 Mbps Internet Speed

From reading similar threads, it looks like I need a dedicated server - maybe from wholesaleinternet.net/dedicated/ (thanks @djelica), but I am not sure what specs would work and if I should be running this over multiple servers rather than a single one at once?

Any advice on server specs, or even other suppliers would be much help. Thanks.

 

[Viniyois]

Yea, if you gonna use VPS for that, you gonna spend lots of money. Dedicated is what you need. you need high CPU + RAM and that's when it becomes problematic.

I am dealing with the similar situation and in the long term having high speed internet + PC at home is cheaper option in my opinion (electricity bill + internet + PC price was much cheaper long term when I calculated). But you have to run it 24/7 at home and make sure connection is not down.

You can get dedicated, but powerful stats cost quite a lot.

 

[HudsonWebServices]

Yea, if you gonna use VPS for that, you gonna spend lots of money. Dedicated is what you need. you need high CPU + RAM and that's when it becomes problematic.

I am dealing with the similar situation and in the long term having high speed internet + PC at home is cheaper option in my opinion (electricity bill + internet + PC price was much cheaper long term when I calculated). But you have to run it 24/7 at home and make sure connection is not down.

You can get dedicated, but powerful stats cost quite a lot.

Nailed it right on the head!

 

[djelica]

Thank you for the credit, I appreciate it.

While I'm here ill try to quickly give my two cents since i faced some similar tasks in the past.

First, scalability is not always easy task. You may get better performance with better specs but you may not get improvement what you were expecting.

Second, are you sure that you need browsers at all? Removing browsers from the setup and doing same thing browsers do programatically you can achieve massive improvement (10x to 100x or more) even on the same home machine. This of course is true if your task allows you to bypass browsers completely and that you are technically capable to do it. (For nerds: yes browsers can be fully excluded from anything, just some things are pain in the ass to do programatically that you want to use browser and not fck with exclusion)

Your home pc specs are not bad (you should have mentioned ram too), but i suspect that you have HDD and not SSD. Trust me, you want SSD if you dont have one.

And of course, if profits are big enough to pay server rent, then why not give it a go.

 

[HudsonWebServices]

I'd honestly go with wholesale as well, I'd grab whatever you can afford. They have very limited availability. This one is available , Dual Xeon E5-2670v2 96GB 2x480GB SSD Preconfig.

 

[examo1]

Yea, if you gonna use VPS for that, you gonna spend lots of money. Dedicated is what you need. you need high CPU + RAM and that's when it becomes problematic.

I am dealing with the similar situation and in the long term having high speed internet + PC at home is cheaper option in my opinion (electricity bill + internet + PC price was much cheaper long term when I calculated). But you have to run it 24/7 at home and make sure connection is not down.

You can get dedicated, but powerful stats cost quite a lot.

Nailed it right on the head!

Thank you for the credit, I appreciate it.

While I'm here ill try to quickly give my two cents since i faced some similar tasks in the past.

First, scalability is not always easy task. You may get better performance with better specs but you may not get improvement what you were expecting.

Second, are you sure that you need browsers at all? Removing browsers from the setup and doing same thing browsers do programatically you can achieve massive improvement (10x to 100x or more) even on the same home machine. This of course is true if your task allows you to bypass browsers completely and that you are technically capable to do it. (For nerds: yes browsers can be fully excluded from anything, just some things are pain in the ass to do programatically that you want to use browser and not fck with exclusion)

Your home pc specs are not bad (you should have mentioned ram too), but i suspect that you have HDD and not SSD. Trust me, you want SSD if you dont have one.

And of course, if profits are big enough to pay server rent, then why not give it a go.

Thanks guys, all this info is very helpful and much appreciated! The plan is to eventually have a better PC/setup at home, but as the initial investment for a server is low compared to all the upgrades/replacements - I would like to try a 'proof of concept' using this first, ideally.

Unfortunately the browser setup is better suited for this as the site sends live data and updates it's structure regularly (encrypting packets, SSL, etc). The cost in time to manually review and adjust every time it's updated would end up costing more than the hardware itself.

I'd honestly go with wholesale as well, I'd grab whatever you can afford. They have very limited availability. This one is available , Dual Xeon E5-2670v2 96GB 2x480GB SSD Preconfig.

Thanks, that's the one I was looking at too. My current RAM is 8GB with 4 Cores/8 Thread (HDD) and as mentioned, handles around 6 browsers ok. Do you think this one should handle 40 browsers @ ~0.45 GB/Browser?

My only worry with these servers is the OS, as I am developing using Python 3 and Windows 10 based tools, which won't be supported in Windows 2008 Enterprise. Is it allowed/fairly straight forward to change to Win10 on a dedicated server such as this?

Edit: That ones out of stock now, but would still be interested in seeing your thoughts on the above please, as i'll pick up a similar one when it's back in stock. Thanks.

 

[MuayThai]

...My current RAM is 8GB with 4 Cores/8 Thread (HDD) and as mentioned, handles around 6 browsers ok. Do you think this one should handle 40 browsers @ ~0.45 GB/Browser?

IMHO, you need a machine with a huge amount of RAM; the CPU(s) is not that important for you.

 

[djelica]

Thanks guys, all this info is very helpful and much appreciated! The plan is to eventually have a better PC/setup at home, but as the initial investment for a server is low compared to all the upgrades/replacements - I would like to try a 'proof of concept' using this first, ideally.

Unfortunately the browser setup is better suited for this as the site sends live data and updates it's structure regularly (encrypting packets, SSL, etc). The cost in time to manually review and adjust every time it's updated would end up costing more than the hardware itself.



Thanks, that's the one I was looking at too. My current RAM is 8GB with 4 Cores/8 Thread (HDD) and as mentioned, handles around 6 browsers ok. Do you think this one should handle 40 browsers @ ~0.45 GB/Browser?

My only worry with these servers is the OS, as I am developing using Python 3 and Windows 10 based tools, which won't be supported in Windows 2008 Enterprise. Is it allowed/fairly straight forward to change to Win10 on a dedicated server such as this?

Edit: That ones out of stock now, but would still be interested in seeing your thoughts on the above please, as i'll pick up a similar one when it's back in stock. Thanks.

Server with that specs should be able to handle over 100 browsers running simultaneously. If you do the math 100 browsers x 0.45 gb ram each = 45gb, so you should have room for more if needed. Those two CPUs are beasts. So give it a go.

For OS doubt. You can message them and simply ask if they can install windows 10 for you (they will replay in short time, they want you as a customer). In the past I used VPS with windows 7 installed (from other company tho), I don't see reason not to be able to have windows 10.

 

[NxtGenLedz]

OP, what is your budget? Also, have you thought about buying an old server from ebay for your home? Would save you a ton. I'd imagine you'd want a very high cpu core count (im talking 24 cores) and lots of ram.

 

[djelica]

One more thing: if you are starving for performance, and you are getting two SSD's included, you may request RAID 0 ssd setup. This setup allow discs to write to both at the same time (almost the same speed as writing to one) and reading from both in the same time (almost doubling reading speed). Just keep in mind that you need to do regular backups on this setup since if one disc fails, data on both is destroyed.

Good luck and best wishes

 

[satyr85]

Go with dual Xeon 2670 or similar, ssd etc. For heavy cpu tasks dual Xeon 2670 is comparable to 3-5 i7 2600.
Wholesale is one of cheapest dedi sellers on market, you wont find better deal.

You may request RAID 0 ssd setup.

There is better option than RAID0 - get server with alot of ram and put bots on ramdrive. Nothing is comparable to ramdrive.

Edit:

IMHO, you need a machine with a huge amount of RAM; the CPU(s) is not that important for you.

Not true, im running similar setup across many servers since few years, CPU is always bottleneck, not ram.

For example:
Xeon 1230v3 cpu, 32 gigs of ram. Usually CPU is maxed when ram usage is around 14gb
Dual Xeon 2660 - CPU is killed when 50 GB of ram is consumed.

All depends what you are actually doing. If you just open website, sit there for 10 minutes and do nothing - you need ram.
If you open website and you perform some tasks, clicks or page reloads - CPU usage will be high, ram will stay low.

 

[NxtGenLedz]

https://www.ebay.com/itm/Supermicro...506814?hash=item44162828fe:g:TYsAAOSwS~FaXnlb

for $400 one time u get 96GB ram for ramdrive and 24 cores. good luck finding something better.

 

[satyr85]

https://www.ebay.com/itm/Supermicro...506814?hash=item44162828fe:g:TYsAAOSwS~FaXnlb

for $400 one time u get 96GB ram for ramdrive and 24 cores. good luck finding something better.

Nice one, here is cinebench result:
https://hwbot.org/submission/3326967_vanden_cinebench___r15_4x_xeon_mp_x7460_1749_cb
So this quad cpu is comparable to dual Xeon 2660

 

[WhcTech]

this can be interresting for someone who want to do proxy, remote desktop etc..

 

[djelica]

Go with dual Xeon 2670 or similar, ssd etc. For heavy cpu tasks dual Xeon 2670 is comparable to 3-5 i7 2600.
Wholesale is one of cheapest dedi sellers on market, you wont find better deal.


There is better option than RAID0 - get server with alot of ram and put bots on ramdrive. Nothing is comparable to ramdrive.

Edit:

Not true, im running similar setup across many servers since few years, CPU is always bottleneck, not ram.

For example:
Xeon 1230v3 cpu, 32 gigs of ram. Usually CPU is maxed when ram usage is around 14gb
Dual Xeon 2660 - CPU is killed when 50 GB of ram is consumed.

All depends what you are actually doing. If you just open website, sit there for 10 minutes and do nothing - you need ram.
If you open website and you perform some tasks, clicks or page reloads - CPU usage will be high, ram will stay low.

Good suggestion for ramdrive, i completely forgot about it. RAID0 will increase computer performance overall and that includes boot time, programs startup time, anything that reads data from disc where ramdrive is not available (funny, even ramdrive startup time), and best of all no action is needed after initial setup.

So to sum it up: Shitload of RAM + RAID0 + ramdrive.

some serious performance porn going on here

 

[Lukmat]

Depends what you do. It's possible to make...headless browser for your task. If it's bot, do that.

 

[examo1]

OP, what is your budget? Also, have you thought about buying an old server from ebay for your home? Would save you a ton. I'd imagine you'd want a very high cpu core count (im talking 24 cores) and lots of ram.

I'm looking at around $100-120/month in server cost, obviously the cheaper the better! As this is more for a 'test case' type project - I would prefer renting a server for at least a month and if goes well and I see it being long term, going with a home based server as you recommended. Honestly though, i'm surprised to see that it would only take ~4 months to pay off for the hardware itself...

Server with that specs should be able to handle over 100 browsers running simultaneously. If you do the math 100 browsers x 0.45 gb ram each = 45gb, so you should have room for more if needed. Those two CPUs are beasts. So give it a go.

For OS doubt. You can message them and simply ask if they can install windows 10 for you (they will replay in short time, they want you as a customer). In the past I used VPS with windows 7 installed (from other company tho), I don't see reason not to be able to have windows 10.

Thanks for that, I understand the ram is within limits but as @satyr85 mentions, I feel that the CPU might cap out before reaching those limits. As when I ran some tests on my own desktop, I noticed the program running very slow even when RAM was available and found that with 6 browsers (2.7GB) + ~2.5GB for OS/background processes - total 5.2GB was pretty much my limit for this application, although I had 8 GB total available.

Also thanks for the suggestion regarding OS, i've sent them a message and will keep you guys updated on their response - for future reference.

As this is really for testing purposes, what are your guys thoughts on multiple smaller servers? Eg, something like this:
Dual Xeon 5520 3TB Preconfig
2.26GHz - 2 processors, 8 Cores / 16 threads, 24GB DDR3, 3TB SATA

How many browsers do you think this setup could handle? As at $40 it won't be too bad for testing and I can pass data via websockets between 2-3 servers to get the same end result.

 

[djelica]

Depends what you do. It's possible to make...headless browser for your task. If it's bot, do that.

Exactly what I wrote, but programming would be time consuming process as OP stated, and in most cases you dont want to fck for days with it, just want damn thing running asap.

 

[satyr85]

Dual Xeon 5520 3TB Preconfig
2.26GHz - 2 processors, 8 Cores / 16 threads, 24GB DDR3, 3TB SATA

I would say nope. Its comparable to Xeon 1230v3 or little slower, so comparable to your current i7 CPU.
+ HDD, you wont benefit from hdd.

 

[djelica]

I would say nope. Its comparable to Xeon 1230v3 or little slower, so comparable to your current i7 CPU.
+ HDD, you wont benefit from hdd.

Same what i thought. For your setup go with more cores as they will be able to dedicate to more processes so you will get better performance in the end.