ConfigDB

Sming library providing strongly typed JSON configuration database support.

Applications requiring complex non-volatile configuration data typically store this in JSON files. Although libraries such as ArduinoJson provide flexible and easy access to these, there are a few drawbacks:

• Once configuration exceeds a certain size it becomes unmanageable as a single JSON file
• Configuration data is typically managed on an ad-hoc basis using structures, and code is required to read/write these structures
• Code must deal with various common conditions such as default values, range checking, formatting, etc.
• Clients such as web applications or other external processors require further code to interpret configuration data

Design goals:

• Describe data once using a standard schema
• Generate API code from the schema which ensures consistent access, applies default values, checks value ranges, etc.
• Make use of available compile-time checks where possible (definitely for value types)
• Allow use of existing tools to enable easy access to data in other languages and on other platforms
• Allow partitioning of database into multiple stores to control use of RAM and enable more efficient updates for related data groups
• Reduce RAM usage as database size increases
• Maintain good read/write performance
• Pluggable system to support backing stores other than JSON, such as binary formats
• Keep application interface independent of implementation detail

Usage:

• Database is described in a JSON schema.
• A python script (tools/dbgen.py) parses the schema and generates class structure for application use.
• Content of database can be streamed to/from web clients

JsonSchema

The database structure is defined using a standard JSON schema <https://json-schema.org>. A good introduction is to take the Tour <https://tour.json-schema.org/>.

An initial schema can be created from an existing sample JSON configuration file using a generation tool such as https://github.com/saasquatch/json-schema-inferrer. Go to the online demo and paste in your JSON configuration. The resulting schema can then be edited and further customised.

Standardised web editors can also be generated using tools such as https://github.com/json-editor/json-editor. Go to the online demo and scroll down to Schema.

.. note::

Sming uses JSON schema for:

    - Hardware configuration :source:`Sming/Components/Storage/schema.json`
    - IFS build scripts :source:`Sming/Components/IFS/tools/fsbuild/schema.json`
    - USB config :source:`Sming/Libraries/USB/schema.json`

It's probably fair to consider it a standard part of the framework.

Configuration JSON is validated against the .cfgdb schema files as a pre-build step. This can also be done separately using a tool such as check-jsonschema::

pip install check-jsonschema check-jsonschema --schemafile basic-config.cfgdb sample-config.json

Separate documentation can be generated from JSON Schema using various tools such as JSON Schema for Humans <https://coveooss.github.io/json-schema-for-humans/>__. For example::

python -m pip install json-schema-for-humans generate-schema-doc basic-config.cfgdb basic-config.md --config template_name=md

Schema rules

See the :sample:Basic_Config sample schema. The test application contains further examples.

• Root object is always a :cpp:class:ConfigDB::Database
• A database is always rooted in a directory
• An optional include array annotation can be added to specify additional header files required for custom types used in the schema.
• Contains one or more stores. The root (un-named) object is the primary store, with the filename _root.json.
• Immediate children of the root may have a store value attached to place them into a new store. This can have any value, typically true.
• A custom type can be defined for Property accessors using the ctype annotation. This means that with an IP address property, for example, you can use :cpp:class:IpAddress instead of :cpp:class:String because it can be constructed from a String. The database still stores the value internally as a regular String.
• Enumerated Properties_ can be defined for any type; ctype is used here to optionally define a custom enum class type for these values.
• Re-useable definitions can be created using the $ref <https://json-schema.org/understanding-json-schema/structuring#dollarref>__ schema keyword. Such definitions should be contained within the $defs section of the schema.
• Arrays_ of simple types (including Strings) or objects are supported.
• Unions_ can contain any one of a number of user-defined object types, and can be used in object arrays.
• Null values are not supported. If encountered in existing datasets then ConfigDB uses the default value for the property.
• Multiple property types, such as "type": ["boolean", "null"] are not supported. A type must be defined for all properties and must be a string value. This also applies to array items.

.. highlight: json

Aliases

Properties may have alternative names to support reading legacy datasets. For example::

{ "$schema": "http://json-schema.org/draft-07/schema#", "type": "object", "properties": { "trans_fin_interval": { "type":"object", "alias": "transfin_interval", "properties":{ "type":"integer" } } } }

Existing JSON data using the old transfin_interval name will be accepted during loading. When changes are made the new (canonical) name of trans_fin_interval will be used.

If multiple aliases are required for a property, provide them as a list.

Integers

Items with **integer** type default to a regular signed 32-bit :cpp:type:`int32_t`. This can be changed by setting a *minimum* and or *maximum* attribute. The database will then use the smallest C++ type appropriate for that range. If the range excludes negative values then an unsigned value will be used.

The maximum supported integer range is for a signed 64-bit integer.

Note that when setting values via *setXXX* methods, streaming updates or inside arrays, integers will be **clipped** to the defined range, never truncated. For example::

  "properties": {
    "Pin": {
      "type": "integer",
      "minimum": 0,
      "maximum": 63
    }
  }

This creates a *pin* property of type :cpp:type:`uint8_t`. The updater will have a *setPin(int64_t)* method, which if called with values outside of the range 0-63 will be clamped.


Floating-point numbers

Items with number type are considered floating-point values. They are not stored internally as float or double but instead use a base-10 representation.

This provides more flexibility in how these values are used and allows applications to work with very large or small numbers without requiring any floating-point arithmetic.

See :cpp:class:ConfigDB::number_t and :cpp:class:ConfigDB::Number for details. There is also :cpp:class:ConfigDB::const_number_t to ease support for format conversion at compile time.

Enumerated properties

JsonSchema offers the `enum <https://json-schema.org/understanding-json-schema/reference/enum>`__ keyword to restrict values to a set of known values. For example::

  {
    "$schema": "http://json-schema.org/draft-07/schema#",
    "type": "object",
    "properties": {
      "color": {
        "type": "string",
        "enum": [
          "red",
          "green",
          "blue"
        ]
      }
    }
  }

ConfigDB treats these as an *indexed map*, so *red* has the index 0, *green* is 1 and *blue* 2. Indices are of type *uint8_t*. The example has an intrinsic *minimum* of 0 and *maximum* of 2. As with other numeric properties, attempting to set values outside this range are clipped.

The default is 0 (the first string in the list). If a default value is given in the schema, it must match an item in the *enum* array.

The corresponding `setColor`, `getColor` methods set or retrieve the value as a number. Adding *"ctype": "Color"* to the property will generate an *enum class* definition instead. This is the preferred approach.

The *color* value itself will be stored as a *string* with one of the given values. The *integer* and *number* types are also supported, which can be useful for generating constant lookup tables.


Ranges
~~~~~~

All *integer* and *number* properties have a :cpp:class:`TRange` definition available should applications require it.
Typically this is defined inside the containing object, such as 
**BasicConfig::Color::Brightness::redRange** for a property named **red**.
For simple arrays, it is for example **BasicConfig::General::Numbers::itemRange**.
For enumerated types, it is defined within the associated type, for example **TestConfigEnum::ColorType::range**. Within arrays of such types, this can also be found in the array itself **TestConfigEnum::Root::Colors::itemType.range**.


Arrays
~~~~~~

ConfigDB uses the **array** schema keyword to implement both *simple* arrays (containing integers, numbers or Strings) and *object* arrays.

Simple arrays are accessed via the :cpp:class:`ConfigDB::Array` class. All elements must be of the same type. A **default** value may be specified which is applied automatically for uninitialised stores. The :cpp:func:`ConfigDB::Object::loadArrayDefaults` method may also be used during updates to load these default definitions.

The :cpp:class:`ConfigDB::ObjectArray` type can be used for arrays of objects or unions. Default values are not currently supported for these.

.. important::

  Be very careful when removing items from an array (via :cpp:func:`Array::removeItem`).
  Don't hold Object or Property references to items in the array during the operation as they may become invalid if the array ordering changes.
  Do not rely on the item index for removin