Namecheap + Namesilo Down - RadiusDC Datacentre is having issues - Will Post Updates here

[BlogPro]

Alright, so a lot of you are panicking because earlier today Namecheap went down, followed by Namesilo (both prominent Domain/DNS providers)

The outage is majorly because RadiusDC - a datacenter in Phoenix, Arizona has been having issues that has caused customer machines to go offline.

RadiusDC is a prominent datacenter and a ton of Namecheap and Namesilo hardware is in the centre.

The issue is entirely because of temperature/air-conditioning failure inside the datacenter which caused unusually high ambient temperatures that began affecting client machines.

PhoenixNap/RadiusDC has posted an update here - https://status.phoenixnap.com/incidents/dnjp5pfv2mmh

Our datacenter provider, RadiusDC is continuing to work with on site vendors to restore temperatures to normal in Phoenix node. We continue to work with them to bring systems back online when possible. For those who missed the previous update, it is duplicated here for visibility:

Our datacenter provider has provided an update on the situation:

Dear Customers,

As we noted in our communication at approximately 7:30 am local time, PHX-01 is experiencing elevated white space temperatures caused by multiple utility bump events attributable to overnight storms in the area. RadiusDC has on-site vendors working rapidly to resolve the situation and restore adequate cooling. RadiusDC technicians have activated mitigation efforts to reduce rising space temperatures. Thus far we have restored Chiller D into operation as of 7:45 am local time. Additional Radius DC remediation measures include the following action items (anticipated commencement local time; anticipated completion local time):

Chiller C restoration (targeted 12:30 pm)
Temporary on floor cooling and heat removal equipment added (approximately 10:00 am)
Additional temporary chiller and temporary backup generator added (targeted approximately 4:00 pm)

Following the restoration of Chiller D, we have seen significant temperature reductions and expect that reduction to continue over the next 60 minutes.

We will provide future communications in 20-minute intervals to advise status of the remaining remediation efforts.

Sincerely,

RadiusDC

We colocate a server in the DC and so are keeping an eye on this. And will post updates as and when we get them.

P.S - I don't own or work for either Namecheap or Namesilo (I am just a customer for the same datacenter) - so don't ask me why they didn't have geographic redundancy and one datacenter failure took down their site and entire support infrastructure.

Namecheap lists hosting locations in Phoenix, the UK, Amsterdam and Singapore.

I also cannot answer the question as to why did a failure of the facility's cooling infrastructure result in customers' services having to be shut down instead of automatically continuing from another geographic site?

Again, just a messenger/customer.

Hope ya'll get your sites backup rapidly.

Abhi

 

[Venusaur]

Too many people creating AI generated garbage to spam social media for a few cents, in the end there was not enough water to quickly cool down the most important datacenters xD

 

[BlogPro]

Too many people creating AI generated garbage to spam social media for a few cents, in the end there was not enough water to quickly cool down the most important datacenters xD

Don't you dare diss my monkey vlogs.

 

[harpage]

why they didn't have geographic redundancy

Nobody foresaw this and redundancy is expensive.

 

[proadsempire]

It's absurd how they manage their infrastructures by causing downtime for their customers... I don't understand this service and honestly I wouldn't even entrust them with a domain/space.

 

[Beklink]

How they can't fix a cooling issue for hours is crazy

 

[BlogPro]

Nobody foresaw this and redundancy is expensive.

Nobody foresaw the fact that a single outage could bring down an international bigwig... As a matter of fact two big wigs.

Even if you can temporarily ignore customers, their servers and hosting plans.

How can you, who manages millions of domains and their DNS data, go down so horribly that your main site is down, your support desk is completely down and even your customer's DNS is down...

 

[proadsempire]

They should have had backups in case of problems...but they weren't smart enough.

 

[BlogPro]

Update

Namecheap main website is back online. Namesilo continues to stay down.

Namecheap hosting seems to be seeing partial restoration.

No update from the data centre. Looks like Namecheap has kicked in some redundancy.

No restoration update from the DC yet.

 

[CPANathan]

Sounds like something the owner of Namecheap would say...

Appreciate the posts Namecheap CEO

 

[CarlosTheGaucho]

I would be curious to see what exactly happened and the whole evolution and cause of that.

I'm a NameSilo client since 2008 or so, but may look for an alternative as they hiked their prices for .com recently.