---

name: hdf5 description: Understanding HDF5 storage features used by NEXTCDF-4, including Superblock v3, float16, compound types, reference types, bitfield types, and HDF5 2.1.1 small floating-point types. metadata: author: netcdf-analysis version: "1.0" date: "2026-08-22"

HDF5 Skill

This skill covers the HDF5 features that NEXTCDF-4 depends on: Superblock v3, the half-precision float16 datatype, compound types, HDF5 reference types, bitfield types, and the small floating-point types added in HDF5 2.1.1. It is intended to guide implementation of the NEXTCDF-4 rewrite of netcdf-c's libhdf5/ backend.

Overview

HDF5 is the storage format behind NetCDF-4. NEXTCDF-4 is a rewrite of the NetCDF-C HDF5 backend (libhdf5/) that targets HDF5 1.14.x and HDF5 2.1.1+. The rewrite exploits newer HDF5 capabilities while preserving read access to existing NetCDF-4 files.

Key areas covered here:

1. Superblock v3 — the new file superblock introduced in HDF5 1.14.x.
2. Float16 — the IEEE 754 half-precision floating point datatype.
3. Compound types — the HDF5 mechanism used for user-defined compound types and complex numbers.
4. Reference types — HDF5 object and region references.

Superblock v3

What It Is

The HDF5 superblock is the first object in an HDF5 file. It stores the file format version, free-space information, root group address, and driver information. Versions:

| Version | Introduced In | Notes | |---------|---------------|-------| | 0 | HDF5 1.0 | Original superblock; 8-byte signature at offset 0 (89 HDF \r \n 1a \n). | | 1 | HDF5 1.6 | Minor changes to size fields. | | 2 | HDF5 1.8 | Added checksum and extension mechanisms; used by classic netcdf-c. | | 3 | HDF5 1.14.0 | Adds post-heap access, larger fields, and newer root group object header. |

Why NEXTCDF-4 Uses It

Superblock v3 provides:

• Access to the post-heap for future format extensions.
• Cleaner evolution toward HDF5 2.x.
• Larger internal length fields that avoid some legacy v0/v2 limitations.

HDF5 API

To create a Superblock v3 file, set the "latest" library version bounds on the file access property list:

hid_t fapl_id = H5Pcreate(H5P_FILE_ACCESS);
H5Pset_libver_bounds(fapl_id, H5F_LIBVER_LATEST, H5F_LIBVER_LATEST);

hid_t file_id = H5Fcreate("file.h5", H5F_ACC_TRUNC, H5P_DEFAULT, fapl_id);

To query the superblock version of an open file, use H5Fget_info2:

H5F_info2_t info;
H5Fget_info2(file_id, &info);
/* info.super.version contains the superblock version */

Compatibility Implications

• Files created with H5F_LIBVER_LATEST (Superblock v3) require HDF5 1.14.0 or later to read.
• Older HDF5 1.12.x and 1.10.x libraries cannot open Superblock v3 files.
• NEXTCDF-4 therefore requires HDF5 1.14.x or newer when creating any NEXTCDF-4 files, including compatibility-mode files.

Compatibility Mode in NEXTCDF-4

Two compatibility flags are supported:

• NC_CLASSIC_MODEL — restricts the file to the classic NetCDF-3 data model with the same restrictions as upstream netcdf-c. Only the root group is allowed, no user-defined types, only one unlimited dimension (first and slowest-varying), and only classic atomic types. Like all NEXTCDF-4 files, the file is written with Superblock v3 and requires HDF5 1.14.x or later to read. NC_CLASSIC_MODEL and NC_NETCDF4_MODEL are mutually exclusive.

• NC_NETCDF4_MODEL — allows the enhanced NetCDF-4 data model but forbids the new NEXTCDF-4-specific types. Like native-mode files, these files are always written with Superblock v3 and require HDF5 1.14.x or later to read. They remain readable by upstream netcdf-c when it is linked against HDF5 1.14.x or later.

Implementation Guidance

• Set the file access property list bounds to H5F_LIBVER_LATEST for both lower and upper bound in all create modes (native, NC_NETCDF4_MODEL, and NC_CLASSIC_MODEL).
• Do not create any of the new NEXTCDF-4 types when NC_NETCDF4_MODEL is set.
• At NEP configure time, disable NEXTCDF-4 entirely (NEP_HAS_NEXTCDF4=0) if the detected HDF5 version is older than 1.14.x.
• Add a runtime check that refuses to create files when the linked HDF5 library is too old to write Superblock v3.

Float16

What It Is

Float16 is the IEEE 754 binary16 half-precision floating point format:

• 1 sign bit
• 5 exponent bits
• 10 mantissa bits
• Total: 16 bits (2 bytes)

It provides reduced precision and range compared to float32, but uses half the storage and often improves I/O throughput.

HDF5 Datatype

HDF5 1.14.x introduces native support for float16:

hid_t f16le = H5Tcopy(H5T_IEEE_F16LE);  /* little-endian half precision */
hid_t f16be = H5Tcopy(H5T_IEEE_F16BE);  /* big-endian half precision */

You can also compose the type explicitly:

hid_t f16 = H5Tcreate(H5T_FLOAT, 2);
H5Tset_fields(f16, 15, 10, 5, 0, 10);
H5Tset_size(f16, 2);
H5Tset_ebias(f16, 15);
H5Tset_precision(f16, 16);
H5Tset_order(f16, H5T_ORDER_LE);

NetCDF Mapping

NEXTCDF-4 proposes a new NetCDF type:

#define NC_FLOAT16 17  /* first free nc_type slot after NC_COMPOUND (16) */

Memory layout is exactly the IEEE 754 binary16 representation. The C API uses arrays of uint16_t (raw bits) for I/O. Platform half-precision convenience wrappers may be added later, but the canonical memory type is uint16_t.

API Additions

Convenience functions (optional):

int nc_def_var_float16(int ncid, const char *name, int ndims,
                       const int *dimids, int *varidp);
int nc_put_var_float16(int ncid, int varid, const uint16_t *data);
int nc_get_var_float16(int ncid, int varid, uint16_t *data);

The standard path is to use nc_def_var with NC_FLOAT16 and the generic nc_put_var/nc_get_var family.

Implementation Notes

• Float16 variables cannot be coordinate variables (no meaningful ordering).
• Fill values for NC_FLOAT16 should be representable as binary16 NaN or a chosen half value.
• When converting between memory types, avoid silent promotion to float32 unless the user requests it.
• Float16 is only allowed when NC_NETCDF4_MODEL is not set.

Compound Types

What They Are

An HDF5 compound type is a collection of named members, each with its own datatype and offset. It is the HDF5 equivalent of a C struct.

typedef struct {
    int x;
    double y;
} point_t;

Creating a Compound Type

hid_t compound_type_id = H5Tcreate(H5T_COMPOUND, sizeof(point_t));
H5Tinsert(compound_type_id, "x", HOFFSET(point_t, x), H5T_STD_I32LE);
H5Tinsert(compound_type_id, "y", HOFFSET(point_t, y), H5T_IEEE_F64LE);

Reading Compound Metadata

int nmembers = H5Tget_nmembers(compound_type_id);
for (int i = 0; i < nmembers; i++) {
    char *name = H5Tget_member_name(compound_type_id, i);
    hid_t member_type = H5Tget_member_type(compound_type_id, i);
    size_t offset = H5Tget_member_offset(compound_type_id, i);
    /* ... */
    free(name);
    H5Tclose(member_type);
}

NetCDF Mapping

NetCDF-4 user-defined compound types map directly to HDF5 compound types. NEXTCDF-4 preserves this behavior unchanged:

• Member names are preserved exactly and are case-sensitive.
• Nested compound types are supported.
• Fixed-size arrays inside a compound member are supported.
• Padding and alignment follow the HDF5 type definition.

Complex Numbers as Compound Types

NEXTCDF-4 uses HDF5 compound types to represent complex numbers:

/* Single-precision complex */
hid_t cplx = H5Tcreate(H5T_COMPOUND, 2 * sizeof(float));
H5Tinsert(cplx, "r", 0, H5T_IEEE_F32LE);
H5Tinsert(cplx, "i", sizeof(float), H5T_IEEE_F32LE);

/* Double-precision complex */
hid_t dcplx = H5Tcreate(H5T_COMPOUND, 2 * sizeof(double));
H5Tinsert(dcplx, "r", 0, H5T_IEEE_F64LE);
H5Tinsert(dcplx, "i", sizeof(double), H5T_IEEE_F64LE);

Proposed NetCDF type constants:

#define NC_COMPLEX       18  /* float complex  */
#define NC_DOUBLECOMPLEX 19  /* double complex */

Memory layout is the portable compound { float r; float i; } (or { double r; double i; }). This matches the C _Complex layout on most platforms and is the canonical NetCDF ABI. On platforms without native _Complex support, users interact with the same {r, i} compound layout.

Implementation Notes

• Use H5Tpack only when the user requests it; otherwise preserve the declared layout.
• When writing, verify that the memory type layout matches the file type layout.
• Compound types are allowed in NC_NETCDF4_MODEL mode, but complex-number compound types are not.

Reference Types

What They Are

HDF5 references are opaque values that point to objects or data regions inside an HDF5 file. They allow one dataset or attribute to refer to another without embedding paths.

There are two reference types:

1. Object Reference (H5T_STD_REF_OBJ): an 8-byte opaque token identifying an HDF5 object (group, dataset, committed type, etc.).
2. Region Reference (H5T_STD_REF_DSETREG): an opaque token identifying a hyperslab selection within a dataset.

Object References

hdf_ref_t ref;  /* opaque 8-byte type, H5R_OBJECT */
H5Rcreate(&ref, file_id, "/g1/dset", H5R_OBJECT, H5P_DEFAULT);

Region References

hdf_reg_ref_t ref;  /* opaque region reference type, H5R_DATASET_REGION */
hid_t space_id = H5Dget_space(dset_id);
hsize_t start[2] = {0, 0};
hsize_t count[2] = {10, 20};
H5Sselect_hyperslab(space_id, H5S_SELECT_SET, start, NULL, count, NULL);
H5Rcreate(&ref, file_id, "/g1/dset", H5R_DATASET_REGION, space_id);

NetCDF Mapping

NEXTCDF-4 proposes two new opaque NetCDF types:

#define NC_REF_OBJECT 20
#define NC_REF_REGION 21

• Variables of these types store HDF5 references as opaque byte arrays.
• The NetCDF API treats them as opaque types; it does not dereference them.
• Future API extensions may add dereferencing functions, but that is out of scope for the initial rewrite.

Reading and Writing

Because references are file-local tokens, they cannot be copied meaningfully between files. This means:

• Reading references from one file and writing them to another is generally undefined.
• Reference-typed variables must be read and written as whole arrays; per-element hyperslab writes are allowed only within the same file.

Restrictions

• Reference types cannot be coordinate variables.
• Reference types cannot appear inside compound types in the initial implementation.
• Reference types are not allowed in NC_NETCDF4_MODEL mode.
• Region references require the referenced dataset to remain open; keep the file open while reading region references.

Implementation Guidance

• Detect H5T_REFERENCE with H5Tget_class(type_id) == H5T_REFERENCE.
• Use H5Tget_ref_type(type_id) to distinguish H5R_OBJECT from H5R_DATASET_REGION.
• Store references as opaque blobs in memory. Do not interpret the bytes.
• When writing, create the dataset with the exact HDF5 reference datatype.

Bitfield Types

HDF5 provides atomic bitfield types (H5T_BITFIELD) for storing raw bit patterns without numeric interpretation. Predefined sizes are 8, 16, 32, and 64 bits: H5T_STD_B8LE/BE, H5T_STD_B16LE/BE, H5T_STD_B32LE/BE, H5T_STD_B64LE/BE.

Classic netcdf-c does not support bitfields. NEXTCDF-4 maps them to new base NetCDF types because they appear in existing HDF5 files (for example, packed quality flags and bitmasks in HDF-EOS and remote-sensing products):

#define NC_BITFIELD8  22  /* maps to H5T_STD_B8LE/BE  */
#define NC_BITFIELD16 23  /* maps to H5T_STD_B16LE/BE */
#define NC_BITFIELD32 24  /* maps to H5T_STD_B32LE/BE */
#define NC_BITFIELD64 25  /* maps to H5T_STD_B64LE/BE */

Memory representation is the matching unsigned integer size (uint8_t, uint16_t, uint32_t, uint64_t). The NetCDF API treats the values as unsigned integers; decoding individual bits is the caller's responsibility.

Implementation Guidance

• Detect bitfie