About the OPNsense tools

In conjunction with src.git, ports.git, core.git and plugins.git they create sets, packages and images for the OPNsense project.

Setting up a build system

Install FreeBSD 14.3-RELEASE for amd64 on a machine with at least 40GB of hard disk and at least 8GB of RAM to successfully build all standard images. All tasks require a root user. Do the following to grab the repositories (overwriting standard ports and src):

# pkg install git
# cd /usr
# git clone https://github.com/opnsense/tools
# cd tools
# make update

Note that the OPNsense repositories can also be setup in a non-/usr directory by setting ROOTDIR. For example:

# mkdir -p /tmp/opnsense
# cd /tmp/opnsense
# git clone https://github.com/opnsense/tools
# cd tools
# env ROOTDIR=/tmp/opnsense make update

TL;DR

# make dvd

If successful, a dvd image can be found under:

# make print-IMAGESDIR

Detailed build steps and options

How to specify build options on the command line

The build is broken down into individual stages: base, kernel, ports, plugins and core can be built separately and repeatedly without affecting the other stages. All stages can be reinvoked and continue building without cleaning the previous progress. A final stage assembles all five stages into a target image.

All build steps are invoked via make(1):

# make step OPTION="value"

Available early build options are:

SETTINGS: the name of the requested local configuration
CONFIGDIR: read configuration from other directory and override SETTINGS (make sure to use an absolute path when specifying)

Available build options are:

ABI: a custom ABI (defaults to SETTINGS)
ADDITIONS: a list of packages/plugins to add to images
ARCH: the target architecture if not native
COMSPEED: serial speed, e.g. "115200" (default)
DEBUG: build a debug kernel with additional object information
DEVICE: loads device-specific modifications, e.g. "A10" (default)
KERNEL: the kernel config to use, e.g. SMP (default)
MIRRORS: a list of mirrors to prefetch sets from
NAME: "OPNsense" (default)
PRIVKEY: the private key for signing sets
PUBKEY: the public key for signing sets
SUFFIX: the suffix of top package name (default is empty)
TYPE: the base name of the top package to be installed
UEFI: use amd64 hybrid images for said images, e.g. "vga vm"
VERSION: a version tag (if applicable)
ZFS: build VM images with ZFS (pool name "zroot") when set

How to specify build options via configuration file

The configuration file is required at "CONFIGDIR/build.conf". Its contents can be modified to adapt a non-standard build environment and to avoid excessive Makefile arguments.

A local override exists as "CONFIGDIR/build.conf.local" and is parsed first to allow more flexible overrides. Use with care.

How to run individual or composite build steps

Kernel, base, packages and release sets are stored under:

# make print-SETSDIR

All final images are stored under:

# make print-IMAGESDIR

Build the userland binaries, bootloader and administrative files:

# make base

Build the kernel and loadable kernel modules:

# make kernel

Build all the third-party ports:

# make ports

Build additional plugins if needed:

# make plugins

Wrap up our core as a package:

# make core

A dvd live image is created using:

# make dvd

A serial memstick live image is created using:

# make serial

A vga memstick live image is created using:

# make vga

A flash card full disk image is created using:

# make nano

A virtual machine full disk image is created using:

# make vm

A special embedded device image based on vm variety:

# make factory

Release sets can be built as follows although the result is an unpredictable set of images depending on the previous build states:

# make release

However, the release target is necessary for the following target which includes sanity checks, proper clearing of the images directory and core package version alignment:

# make distribution

Cross-building for other architecures

This feature is currently experimental and requires installation of packages for cross building / user mode emulation and additional boot files to be installed as prompted by the build system.

A cross-build on the operating system sources is executed by specifying the target architecture and custom kernel:

# make base kernel DEVICE=BANANAPI

In order to speed up building of using an emulated packages build, the xtools set can be created like so:

# make xtools DEVICE=BANANAPI

The xtools set is then used during the packages build similar to the distfiles set.

# make packages DEVICE=BANANAPI

The final image is built using:

# make arm-<size> DEVICE=BANANAPI

Currently available device are: BANANAPI and RPI2.

About other scripts and tweaks

Device-specific settings

Device-specific settings can be found and added in the device/ directory. Of special interest are hooks into the build process for required non-default settings for image builds. The .conf files are shell scripts that can define hooks in the form of e.g.:

serial_hook()
{
    # ${1} is the target file system root
    touch ${1}/my_custom_file
}

These hooks are available for all image types, namely dvd, nano, serial, vga and vm. Device-specific hooks are loaded after config-specific hooks and both of them can coexist in a given build.

Updating the code repositories

Updating all or individual repositories can be done as follows:

# make update[-<repo1>[,...]] [VERSION=git.tag]

Available update options are: core, plugins, ports, portsref, src, tools

VERSION can be used to update to the matching git tag instead of HEAD.

Regression tests and ports audit

Before building images, you can run the regression tests to check the integrity of your core.git modifications plus generate output for the style checker:

# make test

To check the binary packages from ports against the upstream vulnerability database run the following:

# make audit

Advanced package builds

Package sets ready for web server deployment are automatically generated and modified by ports, plugins and core steps. The build automatically caches temporary build dependencies to avoid spurious rebuilds. These packages are later discarded to provide a slim runtime set only.

If signing keys are available, the packages set will be signed twice, first embedded into repository metadata (inside) and then again as a flat file (outside) to ensure integrity.

For faster ports building it may be of use to cache all distribution files before running the actual build:

# make distfiles

For targeted rebuilding of already built packages the following works:

# make ports-<packagename>[,...]
# make plugins-<packagename>[,...]
# make core-<packagename>[,...]

Please note that reissuing ports builds will clear plugins and core progress. However, following option apply to PORTSENV:

BATCH=no Developer mode with shell after each build failure
DEPEND=no Do not tamper with plugins or core packages
MISMATCH=no Rebuild packages that have a version mismatch
PRUNE=no Do not check ports integrity prior to rebuild

The defaults for these ports options are set to "yes". A sample invoke is as follows:

# make ports-curl PORTSENV="DEPEND=no PRUNE=no"

Both ports and plugins builds allow to override the current list derived from their respective configuration files, i.e.:

# make ports PORTSLIST="security/openssl"
# make plugins PLUGINSLIST="devel/debug"

Acquiring precompiled sets from the mirrors or another local directory

Compiled sets can be prefetched from a mirror if they exist, while removing any previously available set:

# make prefetch-<option>[,...] [VERSION=<full_version>]

If another build configuration is used locally that is compatible, the sets can be cloned from there as well:

# make clone-<option>[,...] TO=<major_version>

Available prefetch or clone options are:

base: select matching base set
distfiles: select matching distfiles set (clone only)
kernel: select matching kernel set
packages: select matching packages set

Using signatures to verify integrity

Signing for all sets can be redone or applied to a previous run that did not sign by invoking:

# make sign-base,kernel,packages

A verification of all available set signatures is done via:

# make verify

Nano image size adjustment

Nano images can be adjusted in size using an argument as follows:

# make nano-<size>

Virtual machine images

Virtual machine images come in varying disk formats and sizes. For this reason they are not included in our binary releases. The default format is vmdk with 20G and 1G swap. If you want to change that you can manually alter the invoke using:

# make vm-<format>[,<size>[,<swap>[,<extras>]]]

Available virtual machine disk formats are:

qcow: Qemu, KVM (legacy format)
qcow2: Qemu, KVM (not backwards-compatible)
raw: Unformatted (sector by sector)
vhd: VirtualPC, Hyper-V, Xen (dynamic size)
vhdf: Azure, VirtualPC, Hyper-V, Xen (fixed size)
vmdk: VMWare, VirtualBox (dynamic size)

The swap argument is either its size or set to "off" to disable.

The extras argument can be any extras.conf hook in case the

Tools

Install / Use

README