opsmere:~$pl

Backups in a small business: what to check

Almost every company has some kind of backup. What it’s worth only becomes clear when you need to restore something from it: after a disk failure, a virus attack or a folder deleted by mistake.

The short answer: you need one copy in the office, another outside it, a warning when a backup fails, and a test that checks the files really come back. Below I describe how to check this in your own company and how I set it up in my own infrastructure.

Four questions worth asking yourself

Before you change anything, answer four questions.

  1. Where exactly is the backup? If it’s in the same room as the server, a fire, a flood or a theft will take both.
  2. Is there a second copy outside the office? A backup on a USB drive hanging next to the computer is still the same location.
  3. How will you find out that the backup has stopped running? If the answer is “I’ll notice when I need it”, you won’t find out at all.
  4. When did someone last restore a file from it? A backup nobody has ever restored anything from is only a hope.

If you can’t answer one of these questions, you already have a list of things to fix.

One copy in the office, another outside it

An old rule says: three copies of the data, on two different kinds of media, one of them outside the company. The working data is the first copy. The second is a fast backup in the office, from which you can restore a single file in a few minutes. The third sits somewhere else and has to survive whatever destroys the whole office.

The office backup usually goes to a separate network device with several disks. It’s good if the disks run as a mirror, meaning every piece of data sits on two of them. Then the failure of one disk doesn’t wipe out the backup. A device like this also checks itself: it regularly reads all the data and compares it with checksums, and when one of the two copies of a block is damaged, it repairs it from the other.

In my setup, a backup server makes the backups of the virtual machines, and its data sits on a mirrored set of disks like this. The jobs run at night, in snapshot mode, so they don’t stop the systems from working. I keep the last seven daily backups, four weekly and three monthly ones. That way you can go back to yesterday, to last week or to last month, not just to last night.

The second copy, outside the company, is usually the cloud or a device in another place. Where possible, it’s worth setting it up so that it can’t be changed or deleted for a set period. Then a virus that encrypts the computers in the office can’t encrypt the backup too.

Not everything has to go outside the company. For this copy I pick only the most important parts and send them to S3-compatible cloud storage with Object Lock. For the set period nobody can change or delete those files, not even someone who has taken over the administrator password.

A warning about every error

The most common backup failure looks like this: the job stops working, nobody sees it, and six months later it turns out the last good backup is six months old. That’s why every backup job for my machines and workstation sends a message after every run, successful or not. No message is a signal too, because it means the job didn’t start at all.

The message has to reach someone who will read it. In a small business this can be the hardest part: the “it@” mailbox belongs to nobody, so nobody opens it. Set the address of a specific person and check that the message doesn’t land in spam.

The second warning is about the disks. Disk health can be read and alerted on as it happens: the number of read and write errors, checksum errors, the state of the disk’s self-test. If any counter stops being zero, you look for the cause straight away: the disk, the cable or the memory. A disk that reports bad sectors by itself gets replaced before a second one fails. It’s also worth setting up notifications about the state of the disk array itself.

The restore test

A restore test means restoring selected files from the backup and checking that they open. Seeing on a list that the backup exists is not enough. Those are two different things.

Do it like this:

  1. Pick a few files: a document from accounting, a spreadsheet, a photo. Ideally ones that someone really uses.
  2. Restore them to a separate folder, don’t overwrite the originals.
  3. Open each file and check that it isn’t cut off and that its content is correct.
  4. Write down how long it took and what was unclear along the way. That’s the draft of the instructions for the day you need to restore more than three files.

The test has to be repeated. A sensible minimum is once a quarter. I do it once a month, always on the 29th: I restore one randomly chosen file and a whole virtual machine in a separate environment. A full restore of a machine shows problems you don’t see with single files, and a fixed date keeps the test from fading away.

What usually goes wrong

  • The backup sits next to the server. A fire or a flood takes both.
  • A virus encrypts the backup along with the data. This happens when the backup is attached like an ordinary disk with full write access.
  • The job has been failing for months. Nobody reads the messages, or there aren’t any.
  • The disk holding the backup fails, and there is no second one. One disk is one failure away from losing the backup.
  • Changing one option while the system is running. When rebuilding a set of disks it’s easy to use the wrong command. One adds a disk as a mirror, the other as a separate, non-redundant part. The first keeps the data, the second can end in losing it. That’s why before every change like this I have a fresh backup, and I check the command twice.
  • Data damage with no error at all in the file system. Only regular checksum checks find it. On a set of disks without a mirror, the system can only notice the damage, not repair it. Then what saves you is a backup the file can come back from.

Replacing one disk with two to make a mirror is not a fifteen-minute job. With an 8 TB disk the sync takes 12 to 24 hours, and you mustn’t move on to the next step until it finishes.

When to do it yourself and when to ask for help

You can take the first step yourself: write down what you have, where it is and who gets the warnings. You can also restore a few files and see what happens. It costs an hour and tells you more than any questionnaire.

Help is needed when the whole thing has to be put together: the local backup, the copy outside the company, retention, notifications and restore instructions. Especially when you have servers, virtual machines or data in several places at once. It’s also worth asking for help when nobody in the company looks after the backups and there’s nobody to read the warnings.

I set up backups for small businesses and describe it on the backups page. If you don’t know what state your backups are in today, start with a free audit. I’ll check whether the backups really exist and restore selected files in a test.

All posts