Intel® QuickAssist Technology (QAT) QATzip Library

QATzip is a user-space library built on top of the Intel® QuickAssist Technology (QAT) user-space library. It provides extended compression and decompression capabilities by offloading these operations to Intel® QAT Accelerators. QATzip generates data in the standard gzip* format (RFC1952) with extended headers or lz4* blocks and using the lz4* frame format. The resulting data can be decompressed using any compliant gzip* or lz4* implementation. QATzip is optimized to fully leverage the performance benefits of Intel® QuickAssist Technology.

Supported Formats
Features
QATzip Compression Level Mapping
Hardware Requirements
Software Requirements
Install Instructions
Test QATzip
Performance Test With QATzip
QATzip API Manual
Limitations
Licensing
Legal

Supported Formats

| Data Format | Algorithm | QAT Device | Description | |-------------------------|-----------|-------------------|-----------------------------------------------------------------------------------------------| | QZ_DEFLATE_4B | deflate* | QAT 1.x & QAT 2.0 | Data is in DEFLATE* with a 4-byte header. | | QZ_DEFLATE_GZIP | deflate* | QAT 1.x & QAT 2.0 | Data is in DEFLATE* wrapped by a Gzip* header and footer. | | QZ_DEFLATE_GZIP_EXT | deflate* | QAT 1.x & QAT 2.0 | Data is in DEFLATE* wrapped by an Intel® QAT Gzip* extension header and footer. | | QZ_DEFLATE_RAW | deflate* | QAT 1.x & QAT 2.0 | Data is in raw DEFLATE* without any additional header. (Compression only; decompression falls back to software.) | | QZ_LZ4 | lz4* | QAT 2.0 | Data is in LZ4* wrapped by an lz4* frame. | | QZ_LZ4S | lz4s* | QAT 2.0 | Data is in LZ4S* blocks. |

Features

Accelerated compression and decompression using Intel® QuickAssist Technology, including utilities for file compression and decompression.
Dynamic memory allocation for zero-copy operations via qzMalloc() and qzFree(), enabling pinned, contiguous buffers for DMA operations.
Instance over-subscription, allowing multiple threads in the same process to share fewer hardware instances seamlessly.
Memory allocation backed by huge pages and kernel memory for pinned, contiguous memory access, with fallback to huge pages during kernel memory contention.
Configurable accelerator device sharing across processes.
Optional software failover for compression and decompression, ensuring functionality even when system resources are insufficient.
Streaming interfaces for compression and decompression to improve compression ratios and throughput for piecemeal data submissions.
Asynchronous interfaces for compression and decompression to achieve lower latency and higher throughput (not supported by the qzip utility). This is particularly beneficial for fewer instances or smaller data packets (below 64KB).
Latency Sensitive Mode: Designed for high-stress scenarios, this mode offloads part of the workload to the CPU, leveraging Intel's multi-core processors to enhance throughput and reduce latency.

Note: It is not recommended to enable this mode in low-stress scenarios or for smaller data workloads (below 8KB), as it may result in reduced throughput.
Adaptive polling mechanisms to reduce CPU usage under stress.
Support for compressing files and directories into the 7z format using the qzip utility.

Compatibility with QATzip Gzip* format, which includes a 10-byte header and an 8-byte footer:

| ID1 (1B) | ID2(0x8B) (1B) | Compression Method (8 = DEFLATE*) (1B) | Flags (1B) | Modification Time (4B) | Extra Flags (1B) | OS (1B) | Deflate Block | CRC32 (4B) | ISIZE (4B) |

Support for QATzip Gzip* extended format, which extends the standard 10-byte Gzip* header by an additional 14 bytes:

| Length of ext. header (2B) | SI1('Q') (1B) | SI2('Z') (1B) | Length of subheader (2B) | Intel(R) defined field 'Chunksize' (4B) | Intel(R) defined field 'Blocksize' (4B) |

Support for Intel® QATzip 4-byte headers, indicating the length of the compressed block:
```
| Intel(R) defined Header (4B) | deflate\* block |
```

Support for QATzip lz4* format, structured as follows:

| MagicNb (4B) | FLG (1B) | BD (1B) | CS (8B) | HC (1B) | lz4\* Block | EndMark (4B) |

QATzip Compression Level Mapping

The following table shows how standard software zlib* compression levels map to QATzip levels for QAT 1.x and QAT 2.0:

| Software zlib* Levels | QAT 1.x Equivalent | QAT 2.0 Equivalent | |-------------------------|--------------------|---------------------| | 1 - 4 | QATzip Level 1 | QATzip Level 1 | | 5 | QATzip Level 5 | QATzip Level 1 | | 6 - 8 | QATzip Level 5 | QATzip Level 6 | | 9 | QATzip Level 9 | QATzip Level 9 | | 10 - 12 | Unsupported | QATzip Level 9 |

Refer to QAT Compression levels that summarizes how QATzip Level translates to hardware-accelerated levels for each QAT generation.

Hardware Requirements

This QATzip library supports compression and decompression offload on the platforms with the following QAT devices acceleration devices:

Software Requirements

Intel® QuickAssist Technology Driver for Linux* HW v2.0 or v1.7 (Out of Tree), latest from Intel® QuickAssist Technology
Intel® QATlib for Linux (in-tree) - v25.08 or later
Zlib: v1.2.7 or later
LZ4: v1.8.3 or later
Zstandard (zstd): v1.5.0 or later

Distributions like Fedora 34+, RHEL 8.4+ & 9.0+, CentOS 9 Stream, SUSE SLES15 SP3+, Ubuntu 24.04+ and Debian13+ include the qatzip RPM package in their repositories. This package is built with the QAT_HW qatlib in-tree driver, specifically for 4xxx devices.

Install Instructions

Install with the in-tree QAT package

See the QATlib installation guide for detailed instructions.

From RPM (Fedora 34+, RHEL 8.4+, CentOS 9+, Ubuntu 24.04+, Debian 13+):

# RHEL-based
sudo dnf install -y qatzip qatzip-devel

# Debian-based
sudo apt -y install qatzip libqatzip3 libqatzip-dev

From Source Code:

cd QATzip/
export QZ_ROOT=$(pwd)
./autogen.sh
./configure
make clean && make && sudo make install

Install with the out-of-tree QAT package

Refer to the QAT Installation Guide for detailed setup instructions.

Note: For non-root users, see the non-root user guide. When SVM is disabled, QAT hardware requires DMA-accessible memory. Use QAT USDM component to allocate and free DMA-able memory (see the USDM settings guide).

Install dependencies:

# RHEL-based
sudo dnf install -y autoconf autoconf-archive automake libtool zlib-devel lz4-devel numactl-devel

# Debian-based
sudo apt -y install autoconf autoconf-archive automake libtool zlib1g-dev liblz4-dev libnuma-dev

Configure:

cd QATzip/
export QZ_ROOT=$(pwd)
export ICP_ROOT=/QAT/PACKAGE/PATH
./autogen.sh
./configure  # Run ./configure -h for options

Build and install:

make clean && make && sudo make install

Install from Docker Image

Pre-built image: intel/intel-qatzip
Build image: Use the Dockerfile Note: This is built with QATlib in-tree driver

Configuration

Note: This section applies only to out-of-tree QAT packages. For in-tree QATlib, see the QATlib configuration guide.

The QATzip library requires a [SHIM] sec

QATzip

Install / Use

README