Par2cron

<div align="center"> <img alt="Logo" src="assets/par2cron.png" width="150"> <br> <h1>par2cron</h1> <p>PAR2 Integrity & Self-Repair Engine<br>Selective automated protection for directory trees</p> </div> <div align="center"> <a href="https://github.com/desertwitch/par2cron/releases"><img alt="Release" src="https://img.shields.io/github/release/desertwitch/par2cron.svg"></a> <a href="https://go.dev/"><img alt="Go Version" src="https://img.shields.io/badge/Go-%3E%3D%201.25.5-%23007d9c"></a> <a href="https://pkg.go.dev/github.com/desertwitch/par2cron"><img alt="Go Reference" src="https://pkg.go.dev/badge/github.com/desertwitch/par2cron.svg"></a> <a href="https://goreportcard.com/report/github.com/desertwitch/par2cron"><img alt="Go Report" src="https://goreportcard.com/badge/github.com/desertwitch/par2cron"></a> <a href="./LICENSE"><img alt="License" src="https://img.shields.io/github/license/desertwitch/par2cron"></a> <br> <a href="https://app.codecov.io/gh/desertwitch/par2cron"><img alt="Codecov" src="https://codecov.io/github/desertwitch/par2cron/graph/badge.svg?token=SLUM5DRVHR"></a> <a href="https://github.com/desertwitch/par2cron/actions/workflows/golangci-lint.yml"><img alt="Lint" src="https://github.com/desertwitch/par2cron/actions/workflows/golangci-lint.yml/badge.svg"></a> <a href="https://github.com/desertwitch/par2cron/actions/workflows/golang-tests.yml"><img alt="Tests" src="https://github.com/desertwitch/par2cron/actions/workflows/golang-tests.yml/badge.svg"></a> <a href="https://github.com/desertwitch/par2cron/actions/workflows/golang-build.yml"><img alt="Build" src="https://github.com/desertwitch/par2cron/actions/workflows/golang-build.yml/badge.svg"></a> </div> <div align="center"> ^{This software is in development, expect more frequent releases until a stable release.} </div>

Table of Contents

• Overview
• Quick Start
• Installation
  • Dependencies
  • Building from source
  • Running a built executable
• Usage
  • par2cron create
  • par2cron verify
  • par2cron repair
  • par2cron info
  • par2cron check-config
• Exit Codes
• Output Streams
• State Management
• Creation Arguments
• Creation Modes
• Marker Files
  • Marker Filename
  • Marker Configuration
• Ignore Files
• Configuration
• Limitations
• License

Overview

par2cron is a tool that wraps par2cmdline (a parity-based file recovery tool) to achieve automated periodic integrity creation, verification and repair within any given directory tree. It is designed for use with non-changing WORM-type of files, perfect for adding a degree of protection to media libraries or backups.

The driving idea is that you do not need to invest in a filesystem (like ZFS) that protects all your data, at the disadvantage of additional complexities, when you really only care that important subsets of your data remain protected.

A given directory tree on any filesystem is scanned for marker files, and a PAR2 set created for every directory containing such a _par2cron file. For verification, the program loads the PAR2 sets and verifies that the data which they are protecting is healthy, otherwise flagging the PAR2 set for repair. Once repair runs, corrupted or missing files are recovered. Many command-line tunables, as well as configuration directives, are offered for more granular adjustment of how to create, when to verify and in what situation to repair.

A set-and-forget setup is as easy as adding three commands to crontab:

• par2cron create
• par2cron verify
• par2cron repair

That being set up, you can simply protect any valuable folder by just placing a _par2cron file in it; the tool will create a PAR2 set and pick it up into the periodic verification and repair cycle - now protected from corruption/bitrot.

Quick Start

A default setup involves adding three simple crontab entries:

0 1 * * * par2cron create /mnt/storage
0 3 * * * par2cron verify /mnt/storage
0 5 * * * par2cron repair /mnt/storage

Once configured, protecting a new folder is as simple as:

• Navigating to any directory within /mnt/storage

• Creating an empty "marker" file named _par2cron

• Done - your files are protected after the next scheduled run!

PAR2 sets are then verified and repaired with by the set up periodic tasks.

A condensed quick guide and cheatsheet can be found in the QUICKGUIDE file.

One PAR2 per folder: To keep your mental model simple, marker-based PAR2 creation does not recurse into subfolders by default. The flat protection scope ensures that you know exactly which files a PAR2 covers.

Installation

To build from source, a Makefile is included with the project's source code. Running make all will compile the application and pull in any necessary dependencies. make check runs the test suite and static analysis tools.

For convenience, precompiled static binaries for common architectures are released through GitHub. These can be installed into /usr/bin/ or respective system locations; ensure they are executable by running chmod +x before use.

All builds from source are designed to generate reproducible builds, meaning that they should compile as byte-identical to the respective released binaries and also have the exact same checksums upon integrity verification.

Dependencies

• par2 (the binary of the par2cmdline tool):
  • Debian/Ubuntu: apt install par2
  • macOS: brew install par2
  • Fedora: dnf install par2cmdline

Building from source

git clone https://github.com/desertwitch/par2cron.git
cd par2cron
make all

Running a built executable

./par2cron --help

Usage

The program is divided into separate commands to achieve its tasks:

| Command | Purpose | | :---------------------- | :----------------------------------------------------- | | par2cron create | Creates PAR2 sets for directories with marker files | | par2cron verify | Verifies existing PAR2 sets in a directory tree | | par2cron repair | Repairs corrupted files using PAR2 recovery data | | par2cron info | Shows verification cycle and configuration statistics | | par2cron check-config | Validates a par2cron YAML configuration file |

par2cron create

Scans a directory tree for "_par2cron" marker files
Creates PAR2 sets for directories containing a marker file

Usage:
  par2cron create [flags] <dir> [-- par2-args...]

Examples:

Use configuration file instead of CLI arguments:
  par2cron create -c /tmp/par2cron.yaml /mnt/storage

Pass "-r15 -n1" (15% redundancy, 1 recovery file) to par2:
  par2cron create /mnt/storage -- -r15 -n1

Run for around 1 hour (as soft limit), hide created files:
  par2cron create -d 1h --hidden /mnt/storage

Flags:
  -c, --config string       path to a par2cron YAML configuration file
  -d, --duration duration   time budget per run (best effort/soft limit)
  -g, --glob string         PAR2 set default glob (files to include) (default "*")
  -h, --help                help for create
      --hidden              create PAR2 sets and related files as hidden (dotfiles)
      --json                output structured logs in JSON format
  -l, --log-level level     minimum level of emitted logs (debug|info|warn|error) (default info)
  -m, --mode mode           PAR2 set default mode; creates a set per (file|folder|recursive) (default folder)
  -v, --verify              PAR2 sets must pass verification as part of creation

par2cron verify

Verifies all protected data using the existing PAR2 sets
Corrupted/missing files are flagged for the repair operation

Usage:
  par2cron verify [flags] <dir> [-- par2-args...]

Examples:

Use configuration file instead of CLI arguments:
  par2cron verify -c /tmp/par2cron.yaml /mnt/storage

Verify all sets, argument "-q" (quiet mode) for par2:
  par2cron verify /mnt/storage -- -q

Verify sets not verified < 7 days, run around 2 hours:
  par2cron verify -a 7d -d 2h /mnt/storage

Flags:
  -a, --age duration                 minimum time between re-verifications (skip if verified within this period)
  -i, --calc-run-interval duration   how often you run par2cron verify (for backlog calculations) (default 24h)
  -c, --config string                path to a par2cron YAML configuration file
  -d, --duration duration            time budget per run (best effort/soft limit)
  -h, --help                         help for verify
  -e, --include-external             include PAR2 sets without a par2cron manifest (and create one)
      --json                         output structured logs in JSON format
  -l, --log-level level              minimum level of emitted logs (debug|info|warn|error) (default info)
      --skip-not-created             skip PAR2 sets without a par2cron manifest containing a creation record

External PAR2: par2cron can verify existing sets created by other tools. Use the --include-external flag to pull these into the verification cycle (creating par2cron manifests for them in the process).

par2cron repair

Repair all data flagged as repairable during verification
Uses existing PAR2 sets to recover corrupted/missing files

Usage:
  par2cron repair [flags] <dir> [-- par2-args...]

Examples:

Use configuration file instead of CLI arguments:
  par2cron rep