FBase

Hybrid time-series column storage database engine written in Java

Contents

• Program Description
  • General Information
  • Application Areas
  • Minimum Technical Requirements
  • Getting Started with the Project
    • Building the Project
    • Using the Project
    • Prepare the environment to run unit tests
    • Prepare the environment to run integration tests
      • PostgreSQL
      • Oracle
      • MS SQL
      • Clickhouse
      • Loading test data into Clickhouse
  • Description of the logical structure
  • DB architecture
    • Data storage format
  • Algorithms
    • DB initialization algorithm
  • Program API
    • API sequence diagram
    • Used methods
      • FStore interface
      • FBase class
      • Parameters
        • Table type - enum TType
        • Table indexing type - enum IType
        • Table backend type - enum BType
        • Data grouping function type - enum GroupFunction
        • Table column storage parameters - CSType class
          • Data storage type - enum SType
          • Java data storage type - enum CType
          • JDBC/CSV data storage type - enum DataType
      • Input parameters
        • DB settings - FBaseConfig class
        • SProfile table settings
      • Output parameters
        • Table settings - TProfile class
        • Table column settings - CProfile class
        • Object to get data aggregates from stacked API - StackedColumn class
        • Object to get data aggregates from gantt API - class GanttColumn
    • Code use cases
      • Load table metadata
      • Load table metadata directly
      • Load metadata via JDBC
      • Load metadata from CSV file
      • Load data into table directly
      • Load data into table from JDBC ResultSet
      • Load data into table from JDBC ResultSet in batch mode
      • Get a list of grouped data by column
        • Example for COUNT grouping function using Stacked API and SQL
        • Example for SUM and AVG grouping function using Stacked API and SQL
      • Get a list of grouped data by column with filter
        • Example with filter for COUNT grouping function using Stacked API and SQL
      • Get a list of data with two-level grouping
        • Example of two-level grouping using Gantt API and SQL
      • Get raw data by column
      • Get raw data by table
      • Get raw data for table with filter
        • Example to get raw data for all table columns with filter using Raw API and SQL
      • Get raw data for regular tables using BatchResultSet
      • Get raw data for time-series tables in time range using BatchResultSet
      • Get latest timestamp from table
      • Synchronize data with Berkley DB persistent storage
        • Example of implementation of syncBackendDb FBase API call for Berkley DB backend
      • Close Berkley DB persistent storage
        • Example of implementation of the closeBackendDb FBase API call for the Berkley DB backend
• Download
• Development
• Documentation
• License
• Support
• Contact

Program Description

General Information

FBase is a hybrid columnar database for storing time series data. Written in the Java programming language.

Key Features:

• Columnar data storage format;
• Storage of time series data;
• Storage format with built-in support for data deduplication and compression;
• Setting the storage format at the table and block level;
• Automatic selection of the deduplication algorithm based on the collected data;
• Use of the Berkley DB database as a backend for local storage;
• Support for basic JDBC data types for PostgreSQL, Oracle, MS SQL and ClickHouse databases;
• In read-only mode, automatic generation of SQL queries to tables with time series data;

Return to Contents

Application areas

The library can be used as a lightweight analytical database for backend data processing systems on the Java platform to solve the following application problems;

• storing collected data and analytical processing of small and medium amounts of data for various embedded Java applications;
• storing and processing Internet of Things (IoT) data on the devices themselves as temporary storage, cache, and other tasks related to local data processing;
• metadata storage for querying external data sources using automatic SQL generation based on the data model for analytical applications.

Return to Contents

Minimum technical requirements

FBase is compatible with Java 21+ and comes with a small set of dependencies.

Hardware requirements

Table 1. Hardware requirements

| Parameter | Description | |:-----------------|:-----------------------------------------------------------------------------------------------------------------------------------| | CPU and RAM | Processor with a frequency of at least 500 MHz and a memory capacity of at least 250 MB, depending on the volume of processed data | | CPU architecture | Intel 64-bit (x86_64), AMD 64-bit (x86_64), Arm 64-bit (aarch64) | | Disk | Disk size depending on the volume of processed data |

Software requirements

Table 2. Software requirements

| Software | Requirements | |:---------|:-------------------------| | Java | Java version 21+ | | Maven | Not lower than version 3 | | Git | Latest current version |

Operating system requirements

Table 3. Operating system requirements

| Software | Requirements | |:-----------------|:----------------------| | Operating system | Windows, Linux, MacOS |

Return to Contents

Getting started with the project

Building the project

Make sure you have JDK 21+, Maven and the latest Git

  java -version
  mvn -version
  git --version

Download the FBase repository sources:

  git clone <<url>>
  cd fbase

To compile the project sources, run:

  mvn clean compile

To run unit tests, run:

  mvn clean test

To build the project into a FBase jar file, run:

  mvn clean package

To build and install the FBase jar file into your local Maven repository, run:

  mvn clean install

Notes:

• Running project build includes execution of unit tests, which take about 7-8 minutes.
• This is mainly the execution time of tests from the FBaseRunnerTest class, in which various configurations for 45 tests are launched.
• Information on tests from FBaseRunnerTest is presented in the table:

Table 4. Parameters and statistics for FBaseRunnerTest tests

| Test name | PermutationState | Number of configurations | Total tests | Execution time | |------------------------|------------------|--------------------------|-------------|----------------| | testPermutationNone | NONE | 36 | 1 620 | ~ 3 seconds | | testPermutationPartial | PARTIAL | 648 | 29 160 | ~ 30 seconds | | testPermutatio