OnyxBackupVM

Citrix Hypervisor/XCP-ng VM Backup

OnyxBackupVM
Copyright (c) 2017-2020 OnyxFire, Inc.
	
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program.  If not, see <http://www.gnu.org/licenses/>.

Overview

• The OnyxBackupVM tool is run from a Citrix Hypervisor or XCP-ng host and utilizes the native xe vm-export and xe vdi-export commands to backup both Linux and Windows VMs.
• The backup is run after a respective vm-snapshot or vdi-snapshot occurs, which allows for the backup to execute while the VM is up and running.
• During the backup of specified VMs, this tool collects additional VM metadata using XAPI. This additional information can be useful during VM restore situations and is stored in ".meta" files.
• Typically, OnyxBackupVM is implemented through scheduled crontab entries or can be run manually on an ssh session. It is important to keep in mind that the backup process does use critical dom0 resources, so running a backup during heavy workloads should be avoided (especially if used with compress option).
• The SRs where one or more VDIs are located require sufficient free space to hold a complete snapshot of a VM. The temporary snapshots that are created during the backup process are deleted after the backup has completed.

Quick Start Checklist

1. OnyxBackupVM will require lots of file storage; setup a storage server with an exported VM backup share.
   • Frequently NFS is used for the storage server, with many installation and configuration sources available on the web.
   • An optional SMB/CIFS mode can be enabled via the share_type option in the config file.
2. For all of the servers in a given pool, mount the new share at your desired filesystem location.
   • NOTE: Make sure to add the new mount information to the /etc/fstab file to ensure it is remounted on a reboot of the host.
3. Download and extract the latest release to your desired execution location, such as /mnt/onyxbackup
   • Generally you would extract to and run OnyxBackupVM from the mounted backup share so that the same version, backups, logs, and configuration are visible to all hosts, though this is not a requirement
   • <OnyxBackupVM path>/etc
     • Contains onyxbackup.example (example onyxbackup.cfg file that is heavily commented)
     • Contains logging.example (example of logging.json with default logging configuration)
     • Location for optional configuration file onyxbackup.cfg for overriding default configuration
     • Location for optional logging.json for overriding default logging settings
   • <OnyxBackupVM path>/logs
     • Contains OnyxBackupVM log files OnyxBackupVM.log and debug.log based upon default logging configuration
   • <OnyxBackupVM path>/exports
     • Contains all the VM/VDI backups
     • Can be independently configured to be located wherever you desire using the backup_dir option
4. Inspect and customize certain options in the <OnyxBackupVM path>/etc/onyxbackup.cfg, /etc/onyxbackup.cfg, and/or ~/onyxbackup.cfg as desired.
   • The configuration files are read in the following order with a "last match wins" convention
     • <OnyxBackupVM path>/etc/onyxbackup.cfg
     • /etc/onyxbackup.cfg
     • ~/onyxbackup.cfg
     • Optional config file using -c <file> or --config <file> command-line option
5. Initially use the --preview command-line option to confirm the resulting configuration from files and command-line flags before running an actual backup.
6. Set up a crontab entry or method for executing backups on a schedule
   • You can run OnyxBackupVM from any host in the pool with all available options, not just the master; running it on a host with little to no VMs on it may be optimal since dom0 resources are used for backups.

OnyxBackupVM Command Usage

Basic usage:

usage:

onyxbackup-vm.py [-h] [-v] [-l LEVEL] [-c FILE] [-o] [-ov] [-oe] [-d PATH] [-p]
	[-H] [-C] [-F FORMAT] [--preview] [-e STRING] [-E STRING] [-x STRING]

optional arguments:

-h, --help
	show this help message and exit
-v, --version
	show program's version number and exit
-l LEVEL, --log-level LEVEL
	Log Level (Default: info)
-c FILE, --config FILE
	Config file for runtime overrides
-o, --override-exports
	Override vm_exports list with command-line values
	NOTE: Must be specified before any -e options
-ov, --override-vdi-exports
	Override vdi_exports list with command-line values
	NOTE: Must be specified before any -E options
-oe, --override-excludes
	Override excludes list with command-line values
	NOTE: Must be specified before any -x options
-d PATH, --backup-dir PATH
	Backups directory (Default: \<OnyxBackupVMPath\>/exports)
-p, --pool-backup
	Backup Pool DB
-H, --host-backup
	Backup Hosts in Pool (dom0)
-C, --compress
	Compress on export (vm-exports only)
-F FORMAT, --format FORMAT
	VDI export format (vdi-exports only, Default: raw)
--preview
	Preview resulting config and exit
-e STRING, --vm-export STRING
	Appends VM name or Regex for vm-export to existing list (unless specified after -o option) (Default: ".*")
	NOTE: Specify multiple times for multiple values
-E STRING, --vdi-export STRING
	Appends VM name or Regex for vdi-export  to existing list (unless specified after -ov option) (Default: None)
	NOTE: Specify multiple times for multiple values
-x STRING, --exclude STRING
	Appends VM name or Regex for exclusion to existing list (unless specified after -oe option) (Default: None)
	NOTE: Specify multiple times for multiple values

Some usage examples:

# Backup all VMs in the pool
./onyxbackup-vm.py

# Backup a single VM by name (case sensitive)
./onyxbackup-vm.py  -e 'DEV-MYSQL'

# Backup a single VM by name with spaces in name
./onyxbackup-vm.py -e 'DEV MYSQL'

# Backup VMs by regex which matches more than one VM
./onyxbackup-vm.py -e 'DEV-MY.*'

# Backup VM by name and keep last 2 backups (overrides max_backups)
./onyxbackup-vm.py  -e 'DEV-MYSQL:2'

# Export just root disk (xvda) for a single VM by name
./onyxbackup-vm.py -E 'DEV-MYSQL'

# Export just xvdb disk from a single VM by name without overriding number of backups to keep
./onyxbackup-vm.py -E 'DEV-MYSQL:-1:xvdb'

# Export 2 disks of a VM by name and keep last 7 backups
./onyxbackup-vm.py -E 'DEV-MYSQL:7:xvda;xvdb'

# A mix of the options to show some typical selections if not using config file to specify VMs
./onyxbackup-vm.py -e 'PRD-.*' -e 'MYSQL123' -E 'APPSERVER01:8:xvda;xvdc' -e 'DEV-.*' -x 'DEV-SHORTtest' -p -H

A few words about Python REGEX syntax

For the handling of wildcard VMs, OnyxBackupVM incorporates the native python regex library of regular expressions. WARNING: The syntax is slightly different from what is processed with the Linux family of grep commands. In python, a string followed by * causes the resulting regular expression to match zero or more repetitions of the preceding regular expression, as many repetitions as are possible. For example, ab* will match "a", "ab", or "a" followed by any number of "b" characters. Therefore, if you had PRD-a*, this will match PRD-a, PRD-aa, PRD-aaaSomething, as well as PRD- but also PRD-Test, PRD-123 and anything else starting with PRD- since zero occurences of the "a" after the "-" are matched. To avoid this, PRD-a.* should be used, instead, indicating in this case PRD-a followed by any single character (".") zero or more("*") times.

Note that the current implementation uses the re.match function which by design is expected to be front-anchored. You can explicitly preface a search string with the "^" operator, if desired, but it is already implied and the results will be the same.

A VM name is considered to be a simple (non-wildcard) name, provided that it contains only combinations of any of the following: letters (upper as well as lower case), numerals 0 through 9, the space character, hyphens (dashes), and underscore characters. These will not be handled using regex operations!

Configuration

Configuration of backups follows the below order of precendence and lower precedence matches will not match the same VM already matched by a higher precedence match. If a VM is matched for VDI export and for VM export both will be performed.

1. Excludes
2. VDI Exports (just disk) and/or VM Exports (full backup)

Examples of configuration file syntax for VMs to backup

Exclude simple host names (non-wildcard instances)
excludes = DEV-phishing-test,PRD-PRINTER1

Exclude using end-of-string - handled as a regex expression
excludes = PRD-PRINT$

Exclude PRD-a as well as PRD-a followed by anything (note the use of .* instead of *)
excludes = PRD-a.*

Exclude DEV-V followed by anything and ending in TEMP (end-of-string)
excludes = DEV-V.*TEMP$

Backup just the root VDI of DEV-web3
vdi_exports = DEV-web3

Same as above, except retain seven copies
vdi_exports = DEV-web3:7

Same again except use default max backups and backup first two disks (note semi-colon separator)
vdi_exports = DEV-web3:-1:xvda;xvdb

Backup just the root VDI for anything starting with DEV followed by anything, followed
by either the string "TEST" or the string "test" followed by anything else and also backup a
specific VM
vdi_exports = DEV-.*(TEST|test).*,DEV-EUWWW01

Same as above, exce