Need over 1500 cores server

[Emaxxgold]

Finding it extremely difficult i need around 1500 cores total for a browser automation script need to run around 3k open browsers at one time via bas i don't mind buying x15 100 core servers but this is also a big struggle anyone have any ideas or know any providers thanks

 

[madacot233]

Finding it extremely difficult i need around 1500 cores total for a browser automation script need to run around 3k open browsers at one time via bas i don't mind buying x15 100 core servers but this is also a big struggle anyone have any ideas or know any providers thanks

Scale horizontally, so look for a hosting provider that provides 0.5GB ram 1 core servers that has API access (Azure, GCP, AWS, Linode, etc)

OR

Optimize your code, one browser tab can be isolated cookies / session storage wise - so run 10 per browser to save RAM / CPU cycles.

OR

Unless your trying to harvest something like akamai cookies, you likely don't need that amount of active browsers.

 

[wankawonk]

you probably don't need 1500 cores

you probably need 1500 threads and a bunch of proxies

 

[Sandman_]

Cheapest way is get a bunch of dual cpu servers and build yourself your own personal cloud.

 

[Jefe12]

I am using 1 cpu, i cant help you

 

[bartosimpsonio]

This sounds like a fun project. OP is definitely up to something interesting.

 

[M0n3y.]

For 1500 cores, it's worth to build your own server racks tbh. Threadrippers have up to 128 cores/threads per processor so 5 dual CPU motherboards should suffice.

 

[Redaline]

Sorry to be rude, but do you actually know what you are doing or you are just trying to get some attention?

There is like zero need to have one server with 1500 cores for browser automation.

 

[Alexion]

Let's say you really do need to run 3,000 active browsers in the same time. PhantomJS uses roughly 5 MB of RAM per session. Your only bottleneck is RAM. You probably need around 15-20 GB of RAM, which is rather cheap. CPU wise, I don't want to exaggerate, but I honestly think even LESS than 16 cores will do the job. Especially if the code you are running is well written. SSD I/O would not be a problem, so it's irrelevant.

Give us more details. It's impossible you require 1 core for each 2 browsers.

I can run over 200 Seleniums on 1 core. Over 400 if it's PhantomJS or anything less fancy.

 

[theeyeinside]

For 1500 cores, it's worth to build your own server racks tbh. Threadrippers have up to 128 cores/threads per processor so 5 dual CPU motherboards should suffice.

Threadrippers are single processor systems. You meant EPYC systems. They scale to multiple processors. But they are VERY expensive

Using Threadripper 2990WX chips on single processor boards- @ $1800 each plus $1000 for the MB, Memory, Video and PS gives $2700 for a 32/64 system - $82 per core

3000 cores cost around $246,000 in hardware however it seems overkill.

You were thinking of assigning one browser per core, each one with a different IP and fingerprint?

I suggest building a private cloud with 3 or 4 single servers - 8/16 or 12/24 and totally make sure it runs before scaling up.

Get a 100 or 200 browsers working before scaling.

On Webhostingtalk they have hungry vendors that that might give you a deal on cold iron servers.

Then hire a management company that will configure the cloud for you and manage it. They usually charge $30 per server per month

Sincerely,

The Eye

 

[Gofesss]

Strongly believe you will find the best server fit for your needs if you look through NetShop-isp.com.cy specials.
Based on the number of positive reviews from users who have been with them for a long time, can say that you can rely on them.

 

[mangeshthakur]

There are numerous options for purchasing a new server, but the first step is to carefully determine your exact server configuration. The second step is to determine whether you want a managed or an unmanaged server. There is a significant difference between managed and unmanaged servers, and the most significant difference is that managed servers are managed by the hosting company, and you will not require external support because the hosting company includes the cost of technical support in the billing. If you choose an unmanaged server, you will be required to hire someone to manage your technical support tasks for your server and to resolve any errors that may be prohibitively expensive for a small business. However, there is an alternative that you can use to outsource your server management tasks to a technical support company, which is much more affordable for a single user as well, so the decision is now yours to make.

 

[kodmanyagha]

You can open lots of chromes with different user data directory. For this you can use '--user-data-dir' parameter. Then you can open lots of chrome windows with different data directories. There is same thing in firefox too.