4. Creating the Stage 1 Back Up

Having made your production backups, you need to preserve your partition information so that you can rebuild your partitions.

The script make.fdisk scans a hard drive for partition information, and saves it in several files. The first is an executable script, one per hard drive, called make.dev.x (where "x" is the name of the device file, e.g. hda or sda). Second is mount.dev.x, which creates mount points and mounts the newly created partitions on them. The next, dev.x, is the commands necessary for fdisk to build the partitions. Last is an input file, e.g. dev.sda.sfd, for sfdisk to create partions. (sfdisk is preferable and the scripts will used it if they find it.) You specify which hard drive you want to build scripts for (and thus the file names) by naming the associated device file as the argument to make.fdisk. For example, on a typical IDE system,

bash# make.fdisk /dev/hda

spits out the scripts make.dev.hda, mount.dev.hda and the input files for fdisk and sfdisk, dev.hda and dev.hda.sfd, respectively.

In addition, if make.fdisk encounters a FAT partition, it preserves the partition's boot sector in a file named dev.xy, where x is the drive's device name (e.g. sdc, hda) and y is the partition number. The boot sector is the first sector, 512 bytes, of the partition. This sector is restored at the same time the partitions are rebuilt, in the script make.dev.hda.

Fortunately, the price of hard drives is plummeting almost as fast as the public's trust in politicians after an election. So it is good that the output files are text, and allow hand editing. That's the most difficult but most flexible way to rebuild on a larger replacement drive. (See the To Do list.)

The script save.metadata preserves other metadata as needed. The script saves the partition information in the file fdisk.hda in the root of the ZIP disk. It is a good idea to print this file and your /etc/fstab so that you have hard copy should you ever have to restore the partition data manually. You can save a tree by toggling between two virtual consoles, running fdisk in one and catting /etc/fstab or /fdisk.hda as needed. However, doing so is error prone.

You will also want to preserve files relevant to your restoration method. For example, if you use NFS to save your data, you will need to preserve hosts.allow, hosts.deny, exports, etc. Also, if you are using any network-backed restoration process, such as Amanda or Quick Restore, you will need to preserve networking files like HOSTNAME, hosts, etc. and the relevant software tree. The simplest way to handle these and similar questions is to preserve the entire /etc directory.

There is no way a 100 MB ZIP drive is going to hold a server installation of a modern distribution of Linux. We have to be much more selective than simply preserving the whole kazoo. What files do we need?

To determine the directories needed at boot, we look at the boot initialization file /etc/rc.sysinit (or equivalent). It sets its own path like so:

PATH=/bin:/sbin:/usr/bin:/usr/sbin
export PATH

Trial and error indicated that we needed some other directories as well.

In reading the script save.metadata, note that we aren't necessarily saving files that are called with absolute paths.

We may require several iterations of back up, test the bare metal restore, re-install from CD and try again, before we have a working backup script. While I worked on this HOWTO, I made five such iterations before I had a successful restoration. That is one reason why it is essential to use scripts whenever possible. Test thoroughly!

One thing you can do on an RPM based system is use the rpm program to determine which files are where. For example, to get a complete list of the files used by the openssh package, run:

bash# rpm -ql openssh

Similarly of debian boxes, using apt-file.

bash# apt-file search openssh-server

There are some things you don't need, like the man pages. You can inspect each one and decide whether to back it up or not.

NoteWARNING
 

The second stage of restoration is run without overwriting previously restored files. This means that the files restored in the first stage are the ones that will be used after full restoration. So update your bare metal backups whenever you update files in these directories!

NoteWARNING
 

Recent kernels have incorporated a new ATA (IDE) hard drive driver, libata. Because of this, parallel ATA drives (PATA) now show up as SCSI drives, as serial ATA (SATA) have always done. However, not all rescue distributions use this new driver. There is a line toward the bottom of save.metadata which very carefully replaces "/dev/sda" with "/dev/hda". Use this as a template if you have multiple IDE hard drives. Comment it out or delete it if this is not an issue for you.

Note that there is no guaranteed mapping! Systems with multiple hard drives may have confusing mappings. Be sure to edit this line carefully. Check it if you add or remove a hard drive of any interface type to or from your system!

N.B: if you have libata IDE drive issues, the grub-install line at the end of restore.metadata won't work. If it doesn't, use your rescue disk to do the same. Or burn and boot to the boot image that is made as part of the first stage backup. Boot to it and do the second stage restore as usual. The second stage restore should re-run grub-install or you can run it manually.

4.1. The Archive

All of this gets stored into an archive under /var/lib/bare.metal.recovery. Each day a first stage backup is made a new directory is prepared, with the date encoded as YYYYMMDD, and the day's archive deposited therein. It is up to you to prune obsolete archives. It is a good idea to keep at least one old archive around in case the computer crashes while you are making an archive. If a second archive is made in a day, the earlier one for that day is replaced.

The files in the archive directory include a README.txt, which has information about the backup and the computer the backup was made on. Other files are there in case hand intervention is required.

Below the daily archive directory are several text files and three directories. The scripts reside in bin, the tarballs in data, and information about the system such as partitions and LVM volume backups are in metadata.

To create a CD, simply use a script or graphical tool to create a CD starting at the daily archive directory. It is up to you to be sure your archive will fit onto your medium, or to make other arrangements.

Also in the archive are checksum files for each daily archive. Use those to check the integrity of the daily archives. This is useful for checking archives when you copy them to other computers (such as a backup server) or to the target prior to resoration. If you launch the bare metal backup from the backup server using get, that script will copy the checksum files to the server, and check the archive copy on the server.

4.2. Theme And Variations

4.2.1. No ZIP drive

This backup process used to require you to have the ZIP disk drive present at each backup. It now creates the archive in a directory, which you can back up over the net. Then you only need to build a ZIP disk (with cp -rp) on the backup server when you need to restore.

The backup process will be faster than directly writing to the ZIP drive, but you should check that the resulting directory will fit on your ZIP disk (with the output of du -hs $target.zip in the script save.metadata)! See the definition of the variable zip in that script.

One of my laptops has problems running both a network card and a ZIP drive, so this is the process I use to back it up. I keep a backup image as well as the current one, so that I have a fallback in case the computer crashes during a backup.

4.2.2. CD-ROM

This is similar to the no ZIP drive option above. Save your backups to a directory on your hard drive, as noted. Then use mkisofs to create an ISO 9660 image from that directory, and burn it. This does not work with some CD-ROM based restoration Linuxes, like Knoppix, because the Linux has to have the CD-ROM drive. Unless you have two CD-ROM drives, say one in a USB clamshell. I have a DVD burner set up this way with exactly this in mind. Better, have Finnix load itself into memory on boot and then use the CD-ROM drive from which you booted.

These remarks should also apply to DVDs.

Test your CDs on the drive you will use at restoration time. If you find you need to hack the scripts, you can copy them to /tmp, usually a RAM drive, and edit them there. The scripts will run there. As a RAM disk is volatile, be sure to save your changes before you reboot!

Much of rest of this section is old information. I recommend you use Finnix.

You can remastering Knoppix or Finnix with your first and second stage backups on the CD-ROM. You should also be able to remaster Finnix.

These days many computers come with a CD-ROM drive but no floppy diskette. And floppy drives do fail. So it's a good idea to burn your CD-ROM with a bootable image on it. The bad news is that the "El Torito" format supports 1.2 MB, 1.44 MB and 2.88 MB floppy images, and tomsrtbt uses a 1.7 MB floppy. The good news is that you can get a 2.88 MB version, tomsrtbt-2.0.103.ElTorito.288.img, from the same mirrors where you get the floppy image. Place a copy [1] in the root directory of the backup files. Then use the mkisofs command line option -b to specify tomsrtbt-2.0.103.ElTorito.288.img as the boot image file.

The only down side of this process is that many older BIOSes do not support 2.88 MB floppy images on CD-ROMs. Most of those will boot to a tomsrtbt floppy.

An alternative is to use Syslinux. It is not dependent on a floppy diskette image, and you can build your own CD with a number of tools, such as tomsrtbt, on it.

You may have to adjust the BIOS options to allow the computer to boot to CD-ROM drive. If you can't do that, either because the BIOS won't support booting to CD-ROM, or because you can't get into the BIOS, see Smart Boot Manager (SBM) as described in the Resources.

One variant is to dispense with the tarballs in the first stage, and create a tarball of the entire system. When you build your restoration CD, put the monster tarball in the data directory of the CD. The scripts will pick that up and restore for you, combining the first and second stages. This eliminates a lot of the cruft related to permissions and ownership in restore.metadata and save.metadata

4.2.3. NFS

If you back up across your network to a backup server, you will have all the files on it you need. Set up the directory where you keep all your backups as an NFS export.

Then, on Finnix, do the following (tab completion is very nice here):

# mkdir /mnt/nfs
# /etc/init.d/portmap start
# mount server:/path/of/exportedfs /mnt/nfs
# cd /mnt/nfs/.../bin

Now restore as usual.

There are several advantages to NFS for this job: You don't have to worry about space on a CD-ROM or ZIP drive. You can edit scripts on the server and they are preserved when you reboot the target.

4.2.4. Multiple ZIP disks

By splitting up the two first stage scripts, restore.metadata and save.metadata, you could spread the first stage metadata across multiple ZIP disks.

4.2.5. Excluding From First Stage Saving

There are time when you need to squeeze a few megabytes from the first stage data, especially when you are pushing the limit of your ZIP disk. The function crunch in the script save.metadata takes multiple parameters to feed to tar. It can also take the --exclude parameter. So, for example, you can exclude the samba and X11 directories under /etc like so:

crunch etc --exclude etc/samba --exclude etc/X11 etc

Why those two? Because they're hard drive space hogs and we don't need them when booting after the first stage.

If you keep multiple kernels around, you can eliminate the modules for all of the kernels you won't boot to. Check your lilo.conf or /boot/grub/menu.lst to see which kernel you will use, and then check /lib/modules for module directories you can exclude.

How to find more good candidates for exclusion? List the target directories with ls -alSr for individual files, and du | sort -n for directories.

Another, neater, way to exclude directories is to put a complete list of directories into a file, then refer to it via the tar option --exclude-from=FILENAME.

4.2.6. Initrd

If your system uses an initial RAM disk, or initrd, to boot, make sure that restore.metadata creates the directory /initrd. The easiest way to do this is to ensure that it is included in the list of directories used in the directory creating loop toward the end.

Your system will probably use an initrd if it boots from a SCSI drive or has root on an ext3fs partition. Check /etc/lilo.conf or /boot/grub/menu.lst to see if it calls for one.

When in doubt, check the root directory to see if /initrd is there.

Notes

[1]

I emphasize copy because mkisofs will mung the file in the directory from which it makes the ISO image.