Lesson learned, I lost my all sites today

[Fardin2012]

Around 12 hours ago suddenly my dedicated server went offline. I thought it was just a simple web server issue, so I tried to connect using SSH but failed. Then after trying rebooting I opened a support ticket, support guy quickly activate KVM console and confirmed me. On KVM console I found my NVMe disk is not working, so again I contacted support then they give me the bad news, That disk is gone and I dont have any way other than replace the faulty disk. There no way to get back my data from faulty disk.

I am an idiot and learned a lesson by losing everything today, I never taken any offsite backup, my backup files also in that server (just used a separate directory). When I thinking I have to start all over again, I'm becoming crazy.... what an idiot I am.

So guys always take offsite backup, don't store your backup files in the same server.

 

[zebestof230]

It's not your fault. Maybe Google has your website saved somewhere to get back content.

Sorry to head this, that's something everyone can fall into.

 

[Sofiamartinez]

I had this issue before, didnt take backup and lost everything. I rebuilt the site using google indexed pages and also web archive. Hope you have some data there.

 

[Ndiqi]

It’s very common, even with such warranty you can’t always get 100% the latest copy. Always have backup copy on your own hard drive~

 

[paraztos]

You can check your sites on https://smartial.net/. Maybe some articles and content is there.

 

[Ormosis]

Around 12 hours ago suddenly my dedicated server went offline. I thought it was just a simple web server issue, so I tried to connect using SSH but failed. Then after trying rebooting I opened a support ticket, support guy quickly activate KVM console and confirmed me. On KVM console I found my NVMe disk is not working, so again I contacted support then they give me the bad news, That disk is gone and I dont have any way other than replace the faulty disk. There no way to get back my data from faulty disk.

I am an idiot and learned a lesson by losing everything today, I never taken any offsite backup, my backup files also in that server (just used a separate directory). When I thinking I have to start all over again, I'm becoming crazy.... what an idiot I am.

So guys always take offsite backup, don't store your backup files in the same server.

I am so sorry to hear about that, it must be quite devastating. I'm not here to tell you what you could have done better, but automatic backups are fairly cheap and easy to setup. Technologies such as incremental backups allow you to save space and money even when your websites are huge, however they take away the convenience of a one-click-to-restore backup.

If you are lucky, one or more of your sites have been archived (https://web.archive.org). Another option would be to perform some forensic data recovery, which I highly doubt a hosting provider would be willing to do. If you can convince them to ship you the hard disk instead of disposing of it, you can most likely recover your data with a bit of work (and professional tools).

 

[Diablo9975]

Yes, for me also same issue happens, i need to change my main website address as well due to it. ( some one trying to hack all the time)
now i always take backup each 2 to 3 days on my local pc as well.

its pain when such things happens.

 

[atooxi]

Did u do some hardening to the server ? Were u connecting using the default ssh port 22 ?

 

[Fardin2012]

I am so sorry to hear about that, it must be quite devastating. I'm not here to tell you what you could have done better, but automatic backups are fairly cheap and easy to setup. Technologies such as incremental backups allow you to save space and money even when your websites are huge, however they take away the convenience of a one-click-to-restore backup.

If you are lucky, one or more of your sites have been archived (https://web.archive.org). Another option would be to perform some forensic data recovery, which I highly doubt a hosting provider would be willing to do. If you can convince them to ship you the hard disk instead of disposing of it, you can most likely recover your data with a bit of work (and professional tools).

Thanks, that is a NVMe m2 disk, server provider wiling to send me but I am confused, they said power section is not working.

 

[Ormosis]

Did u do some hardening to the server ? Were u connecting using the default ssh port 22 ?

From the way it sounds, the hard disk is dead. I doubt it's a matter of hardening.

 

[HenryObi]

I am an idiot and learned a lesson by losing everything today, I never taken any offsite backup, my backup files also in that server (just used a separate directory). When I thinking I have to start all over again, I'm becoming crazy.... what an idiot I am.

Sorry for the loss of data.

I don't know the kind of site you have but make use of search console to take note of every single URL.

You could recreate your site by getting any of your content you could lay your hands on. Then ensure URL match exactly. The URL part would be the determinant of how your site survives(SEO-wise).

If any or majority of your pages lead to error 404 instead of the right pages, then consider your site dead and done.

Act as fast as you can.

By the way, I thought hosts are supposed to keep a copy of your files elsewhere for you?

 

[Ormosis]

Thanks, that is a NVMe m2 disk, server provider wiling to send me but I am confused, they said power section is not working.

That's a good news. If it's a matter of power rather than corrupted data, you could probably find a way (or a person) to temporarily power the disk, extract and transfer the data elsewhere. I don't want to give you false hopes, but you should try everything you can before giving up on hopes.

 

[SteveWhite]

If its power issue, you dont have to worry about it. your data should be safe.

For future for production site that make money.
1. Always go for Raid1 Array, when u order server, take 2 same conf disks & ask them to make it as raid1 before installing os. This way data will be safe from ssd / hdd failures.
2. setup a cron to make sql db backup everyday or every 1week and auto send it to a cloud instance where u keep ur safe data. ( better set it to around midnight time or when ur site load is low if u have a high traffic site ).

 

[Fardin2012]

Sorry for the loss of data.

I don't know the kind of site you have but make use of search console to take note of every single URL.

You could recreate your site by getting any of your content you could lay your hands on. Then ensure URL match exactly. The URL part would be the determinant of how your site survives(SEO-wise).

If any or majority of your pages lead to error 404 instead of the right pages, then consider your site dead and done.

Act as fast as you can.

By the way, I thought hosts are supposed to keep a copy of your files elsewhere for you?

Boss, for dedicated server they dont give any backup option, so I stored all my backup in a separate directory but in same disk.

If its power issue, you dont have to worry about it. your data should be safe.

For future for production site that make money.
1. Always go for Raid1 Array, when u order server, take 2 same conf disks & ask them to make it as raid1 before installing os. This way data will be safe from ssd / hdd failures.
2. setup a cron to make sql db backup everyday or every 1week and auto send it to a cloud instance where u keep ur safe data. ( better set it to around midnight time or when ur site load is low if u have a high traffic site ).
If you are not in dedicated or vps, u can actually ask the webhost for such info to get auto backup of ur sql db.

Yes Boss, my big mistake was RAID0, I thought use both 512GB NVMe space but now I lost everything.

 

[Gogol]

Sad to hear that. Guys and girls, take your backups seriously. You need some kind of automatic backup like let’s say, a cron script that will push data to dropbox/g drive every x hrs. Don’t do it manually, because you will feel lazy some day and it will ruin your business.

Even if you can’t do your own auto backup, I am sure your hosting provider has such service already (even for dedicated server it should have a snapshot service I think). Opt for it and be safe.

 

[thedorf]

Sorry to hear your troubles. I know you know this but for everyone else always, always, always take external backups. I worked as a DBA for 8 years and got burned too many times, even haven taken backups.

 

[nakamura]

Happened to me as well in September 2019. Since then I make backups every 2-3 months and save them on my external hard drive + cloud.

 

[Fardin2012]

Sad to hear that. Guys and girls, take your backups seriously. You need some kind of automatic backup like let’s say, a cron script that will push data to dropbox/g drive every x hrs. Don’t do it manually, because you will feel lazy some day and it will ruin your business.

Even if you can’t do your own auto backup, I am sure your hosting provider has such service already (even for dedicated server it should have a snapshot service I think). Opt for it and be safe.

My server provider (hetzner) give autobackup solution for cloud but no backup or management support for dedicated server.

 

[Gogol]

My server provider (hetzner) give autobackup solution for cloud but no backup or management support for dedicated server.

Then cronjobs it is. Should not be too difficult. You don’t need to backup the script files. Github can take care of that (git clone when setting a new server up. git pull when there are changes). You just need to backup your uploads folder (if any) and the db dump, which should be simple for any web developer to code.

 

[4440]

same happened to me, I've learned my lesson.