HDiffPatch

english | 中文版

HDiffPatch is a C\C++ library and command-line tools for diff & patch between binary files or directories(folder); cross-platform; fast running; create small delta/differential; support large files and limit memory requires when diff & patch.

HDiffPatch defines its own patch file format, this lib is also compatible with the bsdiff4 & endsley/bsdiff patch file format and partially compatible with the open-vcdiff & xdelta3 patch file format VCDIFF(RFC 3284).

if need patch (OTA) on embedded systems,MCU,NB-IoT..., see demo HPatchLite, +tinyuz decompressor can run on 1KB RAM devices! HPatchLite also supports a simple inplace-patch implementation to support storage-constrained devices.

update your own Android Apk? Jar or Zip file diff & patch? try ApkDiffPatch, to create smaller delta/differential! NOTE: ApkDiffPath can't be used by Android app store, because it requires re-signing apks before diff.

sfpatcher not require re-signing apks (like archive-patcher), is designed for Android app store, create smaller patch file for apk, patch speed up by a factor of xx than archive-patcher & run with O(1) memory.

if you not have the old versions(too many or not obtain or have been modified), thus cannot create the delta in advance. you can see sync demo hsynz (like zsync), the new version is only need released once, and the owners of the old version get the information about the new version and do the diff&patch themselves. hsynz support zstd compressor & run much faster than zsync; also compatible with the zsync's file format.
additional, if you have the new file locally & not the old file, but can get a hash certificate file(.hsyni) of the old file, you can also create a hpatchz format patch file(usage scenario like rsync); see the demo cmdline hsign_diff in hsynz.

NOTE: This library does not deal with file metadata, such as file last write time, permissions, link file, etc... To this library, a file is just as a stream of bytes; You can extend this library or use other tools.

---

Releases/Binaries

Download from latest release : Command line app for Windows, Linux, MacOS; and .so patch lib for Android.
use cmdline to create a delta:
$hdiffz -m-6 -SD -c-zstd-21-24 -d oldPath newPath outDiffFile
if file is very large, try changing -m-6 to -s-64
apply the delta:
$hpatchz oldPath diffFile outNewPath

Build it yourself

$ cd <dir>/HDiffPatch

Linux or MacOS X

Try:
$ make LDEF=0 LZMA=0 ZSTD=0 MD5=0 XXH=0
bzip2 : if the build fails with fatal error: bzlib.h: No such file or directory, use your system's package manager to install the libbz2 package and try again; or download & make with libbz2 source code:

$ git clone https://github.com/sisong/bzip2.git ../bzip2
$ make LDEF=0 LZMA=0 ZSTD=0 MD5=0 XXH=0 BZIP2=1

if need lzma zstd & md5 xxh... default support, Try:

$ git clone https://github.com/sisong/libmd5.git ../libmd5
$ git clone https://github.com/sisong/xxHash.git ../xxHash
$ git clone https://github.com/sisong/lzma.git ../lzma
$ git clone https://github.com/sisong/zstd.git ../zstd
$ git clone https://github.com/sisong/zlib.git ../zlib
$ git clone https://github.com/sisong/libdeflate.git ../libdeflate
$ make

Tip: You can use $ make -j to compile in parallel.

Windows

Before you build builds/vc/HDiffPatch.sln by Visual Studio, first get the libraries into sibling folders, like so:

$ git clone https://github.com/sisong/libmd5.git ../libmd5
$ git clone https://github.com/sisong/xxHash.git ../xxHash
$ git clone https://github.com/sisong/lzma.git ../lzma
$ git clone https://github.com/sisong/zstd.git ../zstd
$ git clone https://github.com/sisong/zlib.git   ../zlib
$ git clone https://github.com/sisong/libdeflate.git ../libdeflate
$ git clone https://github.com/sisong/bzip2.git  ../bzip2

libhpatchz.so for Android

• install Android NDK
• $ cd <dir>/HDiffPatch/builds/android_ndk_jni_mk
• $ build_libs.sh (or $ build_libs.bat on windows, then got *.so files)
• import file com/github/sisong/HPatch.java (from HDiffPatch/builds/android_ndk_jni_mk/java/) & .so files, java code can call the patch function in libhpatchz.so

---

diff command line usage:

diff usage: hdiffz [options] oldPath newPath outDiffFile
compress usage: hdiffz [-c-...] "" newPath outDiffFile
test usage: hdiffz -t oldPath newPath testDiffFile
resave usage: hdiffz [-c-...] diffFile outDiffFile
print info: hdiffz -info diffFile
get manifest: hdiffz [-g#...] [-C-checksumType] inputPath -M#outManifestTxtFile
manifest diff: hdiffz [options] -M-old#oldManifestFile -M-new#newManifestFile oldPath newPath outDiffFile

  oldPath newPath inputPath can be file or directory(folder),
  oldPath can empty, and input parameter ""
options:
  -m[-matchScore]
      DEFAULT; all file load into Memory; best diffFileSize;
      requires (newFileSize+ oldFileSize*5(or *9 when oldFileSize>=2GB))+O(1)
        bytes of memory;
      matchScore>=0, DEFAULT -m-6, recommended bin: 0--4 text: 4--9 etc...
  -s[-matchBlockSize]
      all file load as Stream; fast;
      requires O(oldFileSize*16/matchBlockSize+matchBlockSize*5*parallelThreadNumber)bytes of memory;
      matchBlockSize>=4, DEFAULT -s-64, recommended 16,32,48,1k,64k,1m etc...
  -block-fastMatchBlockSize
      must run with -m;
      set block match befor slow byte-by-byte match, DEFAULT -block-1k;
      if set -block-0, means don't use block match;
      fastMatchBlockSize>=4, recommended 128,4k,64k, etc...
      if newData similar to oldData then diff speed++ & diff memory--,
      but small possibility outDiffFile's size+
  -cache
      must run with -m;
      set is use a big cache for slow match, DEFAULT false;
      if newData not similar to oldData then diff speed++,
      big cache max used O(oldFileSize) memory, and build slow(diff speed--)
  -SD[-stepSize]
      create single compressed diffData, only need one decompress buffer
      when patch, and support step by step patching when step by step downloading!
        and supports multi-thread patching!
      stepSize>=(1024*4), DEFAULT -SD-256k, recommended 64k,2m etc...
  -BSD
      create diffFile compatible with bsdiff4, unsupport input directory(folder).
      also support run with -SD (not used stepSize), then create single compressed
      diffFile compatible with endsley/bsdiff (https://github.com/mendsley/bsdiff).
  -VCD[-compressLevel[-dictSize]]
      create diffFile compatible with VCDIFF, unsupport input directory(folder).
      DEFAULT no compress, out format same as $open-vcdiff ... or $xdelta3 -S -e -n ...
      if set compressLevel, out format same as $xdelta3 -S lzma -e -n ...
      compress by 7zXZ(xz), compressLevel in {0..9}, DEFAULT level 7;
      dictSize can like 4096 or 4k or 4m or 16m etc..., DEFAULT 8m
      support compress by multi-thread parallel.
      NOTE: out diffFile used large source window size!
  -p-parallelThreadNumber
      if parallelThreadNumber>1 then open multi-thread Parallel mode;
      DEFAULT -p-4; requires more memory!
  -p-search-searchThreadNumber
      DEFAULT searchThreadNumber same as parallelThreadNumber;
      old file on HDD hard drives WARNING: multi-thread search need frequent random
        disk reads when -s-matchBlockSize or -block-fastMatchBlockSize(run with -m),
        causes slowdown; at this time, need to close(searchThreadNumber<=1) multi-thread
        search mode or reduce the number of searchThreadNumber!
  -c-compressType[-compressLevel]
      set outDiffFile Compress type, DEFAULT uncompress;
      for resave diffFile,recompress diffFile to outDiffFile by new set;
      support compress type & level & dict:
        -c-zlib[-{1..9}[-dictBits]]     DEFAULT level 9
            dictBits can 9--15, DEFAULT 15.
            support run by multi-thread parallel, fast!
        -c-lde