By Mitch Stuart
Copyright © 2003-2005 FullSpan Software
-
Usage subject to license
Software Version: 1.6
-
Document Version: $Revision: 1.5 $, $Date: 2005/04/16 07:59:16 $
5. Alternate Notification Configuration
6. Programming and Testing Information
tar archiving utility. It is designed for use by technically knowledgable users (programmers, system administrators, or other power users), for backing up a small network or individual computers.
Here is an example of how FSBackup can be used to backup a small network of Windows and Linux computers:
cron job on Linux and a Scheduled Task on Windows). A full backup is created once a week, and an incremental backup is created on the other days.
Scheduled Task copies the collected backup files to removable storage (e.g., a DVD+RW disk).
Getting FSBackup
tar and gzip to create the backup file. These tools provide standard file formats and well-tested implementations. tar.gz files are portable across many platforms and tools (for example, WinZip can open and manipulate these files).
tar. Any file that you cannot manually put into a tar file will not be able to be backed up. For example: "special files", such as the Windows registry, or files that are "in use" by an application that does not allow concurrent read access. There are more sophisticated backup and disk image utilities that can handle these situations.
fsbackup.sh type bkname defdir destdir [options]
For example:
fsbackup.sh full jdoe-backup /home/jdoe/etc/backup-defs /home/jdoe/backup
FSBackup creates a tar.gz file containing the files you specify. It uses
GNU tar and gzip.
type can be: full, diff, or incr. full is a backup of all files. diff is a differential backup: a backup of new and modified files since the most recent full backup. incr is an incremental backup: a backup of new and modified files since the most recent backup of any type.
bkname is a descriptive name for the backup. Choose a name that is suitable for use in a filename, since this name is used as the base of the filename for the backup definition file, data file, log file, and timestamp file.
defdir is the directory where the backup definition file (see below) is stored.
destdir is the directory where the files created by FSBackup will be stored; it must already exist when you invoke FSBackup.
destdir. These subdirectories are created if they do not exist yet:
destdir/current holds the most recent full backup for each bkname, plus any incremental or differential backups that were done since that full backup.
destdir/archive holds previous full backups and their corresponding incremental and/or differential backups.
destdir/tstamp contains timestamp files that are used to support the incremental and differential features. By tracking the most recent backup time, FSBackup is able to instruct tar to only capture files created or updated after that time.
bkname-timestamp-type.tar.gz and the log file bkname-timestamp-type.log in the directory destdir/current. For example, if the backup is named backup-websrv, then the filenames for a full backup would look like:
backup-websrv-2003-04-09-011500-full.tar.gz
backup-websrv-2003-04-09-011500-full.log
The current directory contains a maximum of one full backup for each bkname, plus any incremental and/or differential backups that have been created since that full backup. When a new full backup is created, FSBackup examines the destdir/current directory to see if there are any existing backups for the same bkname. If so, those files are moved to the destdir/archive directory before beginning the new full backup.
-maxa[rchive] number
Default is 4. This is the number of prior full backup sets that FSBackup will keep in the archive directory (in addition to the current bacukp set that is kept in the current directory). After this threshold is reached, when a new full backup is created, the oldest prior backup for bkname in archive is deleted.-c[ollect] collectdir
If the collect option is specified, the backup and log file will be copied to the directory collectdir. This is useful for doing local backups on several computers, and having the files pushed to a central location, usually in preparation for copying to removable media.-maxc[ollect] number
Default is 2. This is the number of prior full backup sets that FSBackup will keep in the collectdir directory. After this threshold is reached, when a new full backup is created, the oldest prior backup for bkname in collectdir is deleted.-n[otify] address
If the notify option is specified, a brief summary of the backup (the number of files and size of the backup file) will be sent to the address given. Specify an email address to send the summary via email, or specify an address of '-' to print the summary to stdout.
bkname-list.sh in the defdir directory. For example, if bkname is jdoe-backup then the backup definition file would be defdir/jdoe-backup-list.sh.
This file should contain a set of commands to generate the list of files and/or directories to back up. The definition file will be source'd (not invoked) to generate the list of files. It can be as simple or complex as needed. It can simply echo a list of files, or it can run one or more programs to generate the list. The list must be in the format of one file or directory per line, as consumed by the tar --files-from option.
Here is a very simple definition file:
echo 'somefile'
echo 'somedir'
Here is a more complex file:
#!/bin/bash
cd /home/user1/data
echo -e '-C\n/home/someuser/data'
echo -e "file1\ndir2"
cd /home/user2/data
echo '-C'
echo '/home/user2/data'
find . -maxdepth 1 -not -name 'db' -not -name '.' -printf '%P\n'
ls -t db/checkpoint.* | head -1
Here are some tips for creating a backup definition file:
echo 'somefile'). You can also echo a list of names, as long as you insert newlines so that each name appears on its own line when the script is run (e.g., echo -e "file1\ndir2").
#!/bin/bash line is optional. Because the bkname-list.sh file will be sourced, not invoked, it is not required. However, you may want to include this line to simplify running the file as a standalone script for testing, so you do not have to prefix the script name with 'source ' or '. '.
cd commands are not needed for FSBackup, but they allow the bkname-list.sh script to be run for testing, so you can see the file list that will be generated when FSBackup runs the script.
-C commands are a tar feature that allow you to instruct tar to change directories. For example, let's say you need to backup two files that are in separate directory trees (i.e., their only common parent directory is /):
/home/jane/a/b/c/d/hello.txt
/var/db/x/y/foo.log
Without the -C flag, you would have to run FSBackup from the / directory and use a backup definition file like this:
echo 'home/jane/a/b/c/d/hello.txt'
echo 'var/db/x/y/foo.log'
The tar file will store the files with all the path information shown, that is, 'home/jane/a/b/c/d/hello.txt' and 'var/db/x/y/foo.log'.
But let's say you don't want to store all the parent directories. With the -C option, you could use a definition file like this:
cd /home/jane/a/b/c
echo '-C'
echo '/home/jane/a/b/c'
echo 'd/hello.txt'
cd /var/db/x
echo '-C\n/var/db/x'
echo 'y/foo.log'
With this file, tar will store the files as 'd/hello.txt' and 'y/foo.log'. Obviously this example is simplified, but you can see that the -C option is handy (in fact, essential) when you want to 'factor out' intervening directories.
In the output from your -list.sh script, the -C needs to appear on one line, and then the directory on the next line.
tar does not support wildcards in file or directory names that you specify for inclusion in the tar file. Therefore, to use wildcards you should use ls, find, or some other command in your backup definition file, to expand the wildcards to specific names that tar will process. tar does support wildcards for files that you want to exclude - this is discussed below.
tar when you run FSBackup. Then, do a test backup and examine the resulting log file and tar.gz file, to ensure that the files you intended are included.
bkname-exclude.txt in the defdir directory. This file should contain a list of filenames or filename patterns to exclude from the backup. The format of the file is one pattern per line, as consumed by the tar --exclude-from option. For example:
*.bak
fsbackup.sh file to a directory of your choice.
FSBackup may work with other flavors of *nix, but it has only been tested on Linux.
bash, tar, ls, and so on). Once you have installed Cygwin, you will be able to use FSBackup on Windows just as you do on Linux.
There are other *nix command packages like Cygwin that run on Windows, and FSBackup may work with those, but it has only been tested with Cygwin.
Email notifications in Windows
If you want to use the -notify option to have FSBackup send an email summary of each backup, you will need to find a Windows version of the mail utility. Assuming you have Cygwin installed, here is one way to make the mail utility available on Windows:
sSMTP package if you do not have it installed already (it is not included in the default Cygwin base package).
ssmtp-config command to configure sSMTP for your mail server (here is some information on sSMTP configuration).
mail to invoke the ssmtp utility, and put it in a directory on your path. An example mail.sh script is included in the download package for FSBackup, or you can view the script here.
mail script from the command line. Once you get this working, you now have a bare-bones mail utility that has sufficient functionality to be used for the FSBackup -notify option.
mail command to send email notification. If you prefer to use a different command to send email or other notification, you can set the FSBACKUP_MAIL environment variable. If this variable is set, when you request notification FSBackup will issue the command:
$FSBACKUP_MAIL <summary> <destination>
summary is a one-line summary of the backup statistics, and destination is the notification address specified on the FSBackup command line.
For example, I like to use my JMailSend program to send email notifications, because it is written in Java and therefore cross-platform, and it is easy to configure. I set FSBACKUP_MAIL like this:
export FSBACKUP_MAIL="$HOME/bin/jmailsend-backup-summary.sh"
jmailsend-backup-summary.sh is something like this:
java -jar "$JMAILSEND_JAR" "$JMAILSEND_PROPERTIES" "$1" "$2" <<EOT
EOT
bash script that uses several standard *nix commands (tar, find, awk, and so on).
It has been tested on Linux and on the Cygwin environment under Windows.
You can use the fsbackup-test.sh script included in the FSBackup download package to test FSBackup in your environment. The script takes several minutes to run, because it contains some sleep commands to allow time to elapse for some of the tests.
tar Limitations
There are a couple of limitations in the way that the tar utility works that had an effect on the way FSBackup was written:
--null option to separate file/directory names with null characters instead of with line breaks. Because file/directory names themselves can contain special characters such as line breaks, using --null is documented to be a safer way of dealing with arbitrary names. However, when the tar --null option is used, it disables the -C option. Since I feel that the -C option is essential for FSBackup, I was not able to use --null.
--files-from instead of -T). However, the long form of the -C option, --directory=directory, cannot be used when the --files-from option is being used. Therefore, I had to use the -C short form of the option.