virtnbdbackup

Backup utility for libvirt, using the latest changed block tracking features. Create online, thin provisioned full and incremental or differential backups of your kvm/qemu virtual machines.

<!-- START doctoc generated TOC please keep comment here to allow auto update --> <!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->

• About
• Prerequisites
  • Libvirt versions <= 7.6.0 (Debian Bullseye, Ubuntu 20.x)
  • RHEL/Centos Stream, Alma, Rocky Linux
    • Version <= 8.5
    • Version >= 8.6
  • Environment dependencies
• Installation
  • Python package
  • RPM package
  • Debian package
  • Virtualenv
  • Docker images
• Backup modes and concept
• Incremental backups of RAW disks and direct attached volumes (LVM, ZFS, ..)
  • QCOW Format versions
  • Libvirt versions < v10.10.0
  • Libvirt versions >= v10.10.0
• Backup Examples
  • Local full/incremental backup
  • Backing up offline virtual domains
  • Application consistent backups
  • Rotating backups
  • Excluding disks
  • Estimating differential/incremental backup size
  • Backup threshold
  • Backup concurrency
  • Compression
  • Remote Backup
    • QEMU Sessions
    • NBD with TLS (NBDSSL)
    • Using a separate network for data transfer
    • Piping data to other hosts
  • Kernel/initrd and additional files
  • Windows Bitlocker recovery keys
• Restore examples
  • Dumping backup information
  • Verifying created backups
  • Complete restore
  • Process only specific disks during restore
  • Point in time recovery
  • Restoring with modified virtual machine config
  • Remote Restore
  • Restore with enabled compression
• Post restore steps and considerations
• Single file restore and instant recovery
• Transient virtual machines: checkpoint persistency on clusters
• Supported Hypervisors
  • Ovirt, RHEV or OLVM
  • OpenNebula
• Authentication
• Internals
  • Backup Format
  • Extents
  • Backup I/O and performance: scratch files
  • Debugging
• FAQ
  • The thin provisioned backups are bigger than the original qcow images
  • Backup fails with "Cannot store dirty bitmaps in qcow2 v2 files"
  • Backup fails with "unable to execute QEMU command 'transaction': Bitmap already exists"
  • Backup fails with "Bitmap inconsistency detected: please cleanup checkpoints using virsh and execute new full backup"
  • Backup fails with "Error during checkpoint removal: [internal error: bitmap 'XX' not found in backing chain of 'XX']"
  • Backup fails with "Virtual machine does not support required backup features, please adjust virtual machine configuration."
  • Backup fails with "Timed out during operation: cannot acquire state change lock"
  • Backup fails with "Failed to bind socket to /var/tmp/virtnbdbackup.XX: Permission denied"
  • High memory usage during backup
  • fstrim and (incremental) backup sizes
  • Test your backups!
  • Links

<!-- END doctoc generated TOC please keep comment here to allow auto update -->

About

Existing backup solutions or scripts for libvirt/kvm usually depend on the external snapshot feature to create backups, sometimes even require to shutdown or pause the virtual machine.

Recent additions to both the libvirt and qemu projects have introduced new capabilities that allow to create online (full and incremental) backups, by using so called dirty bitmaps (or changed block tracking).

virtnbdbackup uses these features to create online full and incremental or differential backups.

virtnbdrestore can be used to re-construct the complete image from the thin provisioned backups.

virtnbdmap can be used to map an thin provisioned backup image into a block device on-the-fly, for easy single file restore or even instant boot from an backup image.

For backing up standalone qemu virtual machines not managed by libvirt, see this project: qmpbackup

Prerequisites

Obviously you require a libvirt/qemu version that supports the incremental backup features. Since libvirt v7.6.0 and qemu-6.1 the required features are enabled by default and are considered production ready: everything will work out of the box.

Following, you will find a short overview which older libvirt versions may require further adjustments to the virtual machine config.

Libvirt versions <= 7.6.0 (Debian Bullseye, Ubuntu 20.x)

If you are using Debian Bullseye or Ubuntu 20.x, the included libvirt version already has an almost complete support for incremental backup, although it doesn't work properly with migration or some block jobs.

If you don't want to use migration or other blockjobs you can enable the incremental backup feature on these libvirt versions. Change the virtual machine config using virsh edit <vm> like so: (the first line must be changed, too!):

 <domain type='kvm' id='1' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
 [..]
 <qemu:capabilities>
   <qemu:add capability='incremental-backup'/>
 </qemu:capabilities>
 [..]
 </domain>

Note:

You must power cycle the virtual machine after enabling the feature! Upstream libvirt strongly discourages enabling the feature on production systems for these libvirt versions.

RHEL/Centos Stream, Alma, Rocky Linux

Version <= 8.5

Up to RHEL/Centos8/Almalinux 8.5, libvirt packages from the advanced virtualization stream support all required features. To install libvirt from the stream use:

yum install centos-release-advanced-virtualization
yum makecache
yum module install virt

and enable the feature by adjusting the virtual machine config.

Version >= 8.6

As of RHEL 8.6, the advanced virtualization stream has been deprecated, and all components supporting the new feature are included in the virt:rhel module, the feature is enabled by default. (Details)

Environment dependencies

• python libvirt module version >= 6.0.0 (yum install python3-libvirt)
• python libnbd bindings (https://github.com/libguestfs/libnbd) version >= 1.5.5 (yum install python3-libnbd)
• The virtual machine should use qcow version 3 images to support the full feature set.

Installation

There are several ways to install the utility, below you will find an short description for each of them. For Debian and RHEL/SuSE based derivates see releases for pre-built packages.

Note:

Please consider to check past issues related to installation if you face any troubles before opening a new issue.

Python package

 git clone https://github.com/abbbi/virtnbdbackup && cd virtnbdbackup
 pip install .

Note:

Do not install the "nbd" package available on PyPI, it does not provide the required nbd bindings (unfortunately has the same name). You have to additionally install the provided python3-libnbd packages by your distribution, or compile the libnbd bindings by yourself.

RPM package

Packages for RHEL/Fedora and OpenSUSE are available via releases.

To create an