Ransomware the end of the 3-2-1 backup?
There have been a few recent cases of Hive users being hit with ransomware and losing access to their account. So I felt it would be a good time to talk about ways to prevent this. As someone who has been in IT since High School, I have recommend the 3-2-1 approach to backups forever. This simply means you will always have three copies of your data, stored on two different types of media, and one copy off site.
Three copies of your data
This typically is satisfied with your current copy of your data, your backup, and your offsite backup.
Two different types of media
This used to mean hard drive and tape but most people are not using tape and an external USB drive is a suitable option.
One copy off-site
This to me is the most important step, this will protect against corruption, theft, and fire. This is your last resort hail mary pass option, but without it you are at extreme risk of data loss.
While I still recommend this to everyone, I have added an additional requirement for the modern age of false security. I highly recommend at least one immutable backup.
Immutable backup
An immutable backup is a backup that cannot be erased. Typically when you backup, you backup to a tape or an external hard drive. If your machine is comprimised, a hacker will wipe or encrypt data on your machine. They will also attempt to destory/encrypt any backups or remote storage they can get their hands on. This leaves you at their mercy even with proper 3-2-1 backups.
The truth about ransomware
A lot of time you will not receive a decryption key and the hacker may not even have it, even if you pay the Bitcoin they demand. Some hacker organizations are more organized than a typical corporation and even have dedicated sales reps and support staff to help larger companies decrypt their data and collect ransom. Some are no better than a script kiddie with little to no understand of what they are doing. Either way, paying ransom should never be an option, doing so harms all users by increasing the power of these organizations.
How to create an immutable backup
This step can be tricky and is not as turn key as the other steps. Most backups solutions don't offer this and franky don't even discuss it. It is as important as the other steps though, and without it you are 100% at risk of full data loss.
The easiest way to create an immutable backup is to rotate external hard drives on a daily/weekly basis. Even better if you can store these offsite for a period of time. There are companies like Iron Mountain that do this for companies typically storing tapes for years on their behalf.
With the increasing speed of home Internet connections, digital off-site backups are becoming more and more attractive. These are just as vulnerable to being wiped as local copies of your data. There are options though, services that use the ZFS file system and have immutable snapshots that cannot be wiped for X number of days. This means if your machine and all your backups get wiped, you can still restore from a snapshot.
If you are highly technical, you can use a server or NAS to host a ZFS/BTRFS file system with snapshots. If your server is completely wiped, you are of course out of luck, but if your workstation is compromised, it will not be able to wipe out the snapshots without also having access to the file server.
Depending on how critical your data is, I personally have zero tolerance for data loss, the steps you take to protect it will scale with that demand.
Do you have three copies of your data?
Is one of them off-site?
If you are into crypto, this should be a big concern to you. If you haven't already, consider your current risk acceptance and act accordingly. ZFS & BTRFS are the popular options for file systems that allow snapshots. Unfortunately, most workstations won't be able to run these file systems. Although BTRFS is becoming more common for Linux machines.
While immutable snapshots will require more space than your original data, the difference is generally a lot less than you would expect. The other benefit of snapshots is the ability to restore terabytes of data within a second or even mounting your old data as another drive to recover data manually.
No matter what you do, it can all be pointless if your backups don't work when you need them.
Always test your backups! In fact, make a schedule to check them regularlly. I won't even get started on bit rot.
I used to run backups at a place I worked where I would put a DAT tape in at the end of the day and move it to a safe place when it finished the next morning. Now the quantities of data many of us have are so massive that it gets harder to be safe. I have used Google Drive as my offsite storage, but if my own drive gets wiped and syncs then I can still lose it all. I ought to get a big USB drive to take a copy every so often. There are more critical files that might fit on a flash drive.
I assume there must be network storage backup systems that would be resistant to mass wipes or scrambling, but that may not be viable for personal use. I need to look into this more.
Thanks for reminding me.
gulp I would never trust critical data on a flash drive for backups. They are one step up from SD cards, but not a lot better.
Also keep in mind, SSD are not designed for long term offline storage. Many SSD are rated for only 2-5 years of integrity without power.
There are lots of options, I named a few, but they can get pretty complicated. Although there are some simplier options as well, just getting a USB HDD offsite on a regular basis can help a lot.
How complicated it gets is directly proportional to how much it will cost you if you lost your data.
I have over 8 terabytes of data I consider important. Mostly digital photos and video going back decades at this point. Even a couple of old home movies from the 1950s when my Mom was a toddler. I make a backup to an external hard drive periodically and even keep another "offsite" though it is just where I work which isn't very far away. The offsite one doesn't get updated as often as it should. But I figure that's about the best I can do for now.
This is a great reminder post.🎶
Although I do love immutable, and use this for my customers as well, I do wonder why at home 2 external USB drives aren't enough in your opinion. I mean, connect them only when creating the backup. And rotate both for each next backup. The rotation isn't doing a lot, just that whenever a new backup is created and something happens right at that time, the other USB drive is there that has the last backup.
To set up an immutable backup at home, seems a bit overdone, though immutable is indeed great. Well, when I can buy this for not too much money, I will go for something like that.
Currently, I don't do this rotation, and perhaps I back up with a little too low frequency. I shall up my game here 🙃
They are not enough for me, as this does not solve the one off-site backup requirement. It does help with immutability (at least in some degree). As long as at least one backup is offline when your machine gets compromised you are safe. But are you? I worked in IT my entire life, I had my first computer business while I was in High School, and I can't tell you how many times backups are not working when you need to use them. As a consultant, backups were always the last thing on people's mind when buying a server, they were also frequently screwed up in execution.
It depends on your tolerance to loss, and there are cheap ways to do this with as little as two external USB drives. I used to rotate three external USB drives for years, with one going to my parents house each week. With internet speeds the way they are, it is becoming more and more viable to off-site backup everything.
Offsite backup through an online connection is possible indeed (soon and I can get 2GBit/s to upload, ridiculously fast). Still, it is an online more or less permanent connection, which I won't like too much for a proper backup. I learned in my Telco days to unplug whatever is sensitive. That is what I do with my backup USB drive. And soon drives. The one copy (perhaps indeed a 3rd drive of some sort, or the online variant will do I suppose) at another location is indeed something that I need to account for.
For a few years am working in IT myself. My employer has quite an expensive setup for backup with storage rings across the country, two of them. Immutability is possible on both rings. Not too many prospects/customers (including the big boys) don't want to pay for the immutability though—too much storage cost. Well, most companies didn't experience ransomware attacks, is what I can conclude, and CFOs not giving enough money to their CIOs and CTOs.
The old 321 includes a tape as an offline backup. Although a bit old-fashioned, I fully understand this method since this is offline after the tape is taken out of the carousel. My employer decided a few years ago, that this isn't modern enough anymore. Well, I suppose they decided this is too hands-on with a car driving to various data centres every single day to take the tapes and bring them to a large vault. Still today I'm pulled in quite lengthy discussions with prospects who aren't too pleased the tape option isn't there anymore. Fortunately, most of the time I can convince them our current setup is secure enough. Funnily enough, I like my backups to be offline 😆 I guess I can lie well enough about the best solutions and my preferences to my customers-to-be 😆 yes, my 'sales' stories I make quite personal, therefore I feel that I am lying to them 😱
I have been doing this for years...this is awesome advice @themarkymark! Thank you.
piece of paper seems like a good solution
This, 100%.
I recently helped someone dealing with data loss precipitated by a power outage, and their practice of leaving their USB backup drive in the computer caused their backup to fail. However, because they persisted in using paper to maintain double entry accounting they were able to rebuild by re-entering the data.
Where possible, keeping paper copies of keys, addresses, and accounting data is another backup mechanism that is immune to hackers.
Thanks!
I have used pen drives to back things up but they seem to lose data very easily. I do wonder if the Trezor and Ledger devices are equally unreliable! lol SSD are supposed to be not good for long term storage, so we have just the typical slower external drives we could use. What software would you recommend for taking snapshots of your drives on a typical windows or linux system?
I highly recommend TrueNAS as a network file server which uses ZFS. This supports any operating system as your desktop. Rsync.net also has a remote ZFS for off-site where you can send backups to the cloud with immutable snapshots.
The easiest, is just using a couple of USB HDD and never have them all connected to your machine at the same time. I wouldn't recommend running any one drive for longer than 7 days, although daily switching would be ideal it is time consuming.
I think of this whenever I read articles like this, and back up my data. I should add the offsite option and I will try this 3-2-1 strategy. I really like the idea of using an online company to automatically back up my data offsite.
Thanks for sharing.
Late to the party again
I got an email saying "they'd: recorded me on cam and had all my whatever and to send Bitcoin to therewhereeveritis
I don't think they did their research very well 😆
Fuck broke and the most they'd catch me doing is eating cheap biscuits in bed. Oh crumbs
Hi @themarkymark, hope to find you well...
Do you know if, based on the reports of those affected, there is any explanation or how they are operating?
I ask this because I worry that there is some form of Brute Force to identify the Hive keys. Would this be possible?
I arrived at your post after reading reports from people affected by sepa666 and I noticed that many drained accounts appear to be from steem/old users, that is, who are not using Hive and had their accounts compromised, thus believing that this attack These accounts with the highest frequency in 2024 must have some factor in common, but it was not directed within the Hive dapps.
Anyway, curiosities to always be alert.
And yes, I have about 5 devices with my encrypted keys and 1 notebook saved and 1 notebook for everyday life with easier access.
Malware infection. Can read about one of them here:
https://peakd.com/hive-110011/@abaddon15/do-not-download-suspicious-files
Tks, yes, i read it... what i still find strange is lot of inactive keys being hacked by same guy
aiming hive users?
Yeah as well as Steem.