(f)db

NOTE: At the moment, I am exploring and integrating all potential high-performance transport options, including TCP, to establish a solid foundation for benchmarking. In the future, I may discard any transport methods that prove inefficient based on the benchmarking results. Concurrently, I will begin writing convenient wrappers around the underlying packages to streamline usage and deployments.

CURRENTLY UNDER ACTIVE DEVELOPMENT

Meaning everything about this repository can be changed. Best to keep track, not use the code yet besides to play with it...

(f)db is born from frustrations experienced while working with key-value (KV) databases. The goal is to build extremely fast transport layers on top of KV databases.

This project is going to be either f**k databases or fast database... There is no third solution...

Why is this necessary, and is it just another over-engineered solution in the web world?

It might not be essential for every use case, but it becomes particularly interesting if you require a distributed key-value database with replicated data across multiple nodes. Additionally, it offers the ability to access and manage data efficiently across multiple servers, each sharing a single connection endpoint, to offload data to different databases and/or nodes.

For instance, imagine using the same client to seamlessly push data to either an MDBX or DuckDB instance, regardless of whether they are hosted on the same or different servers. This abstraction provides flexibility in data management and replication across distributed systems.

Why I need this?

I require a solution that enables different services to write to and read from different databases while using a unified transport layer. Achieving this with existing tools is complex, if not impossible, without a dedicated transport layer on top of the database. This transport layer allows for cross-service communication and interaction with the underlying database.

This project addresses that gap by building a high-performance, flexible transport layer that enables seamless interaction with various databases in a distributed environment.

Why Multiple Transports?

One core feature of (f)db is its support for multiple transport protocols, including UDP, UDS, TCP, and QUIC. The reason for this is simple: no single transport protocol is perfect. Each excels in different areas, such as throughput, latency, reliability, or efficiency under specific network conditions.

By offering multiple transport options, (f)db provides the flexibility to fine-tune and optimize performance depending on your specific needs and environment. This allows users to select the most appropriate transport based on their use case, whether it's high-throughput, low-latency, or optimized for specific infrastructure.

Networking

This project uses gnet, an event-driven, high-performance networking library built for Go. gnet is much faster than Go’s standard net package for several reasons:

• Event-Driven Model: gnet uses an event-driven approach, meaning it reacts to specific network events (like data being available to read or connections being closed) instead of continually polling or blocking threads like the standard net package. This reduces CPU overhead and leads to faster processing.
• Efficient Use of Goroutines: Unlike Go’s net package, gnet minimizes the use of goroutines, which helps reduce the context-switching overhead that can slow down highly concurrent applications. Instead, gnet directly handles network I/O in an optimized way.
• Zero-Copy Data Handling: gnet offers zero-copy mechanisms for data processing, meaning that memory is not repeatedly copied between kernel and user space, significantly improving throughput for high-performance networking applications.
• Multi-Event Processing: gnet allows the processing of multiple events within a single loop, which can lead to higher efficiency when handling a large number of simultaneous connections. This feature is particularly valuable for real-time, high-frequency applications that need to handle many clients at once.

By leveraging gnet’s capabilities, (f)db can react quickly to network events, optimize throughput, and minimize latencies, making it ideal for distributed systems where performance is critical.

eBPF Integration

As part of the ongoing development of high-performance transport layers for (f)db, eBPF (Extended Berkeley Packet Filter) has been integrated to enhance packet processing and monitoring capabilities within the network stack.

By leveraging eBPF, we can build efficient, scalable, and secure transport layers that directly interact with network traffic, providing real-time insights and optimizations for packet flow, without adding latency or reducing throughput.

• Real-Time Packet Filtering: With eBPF, (f)db can selectively filter out unnecessary traffic at the kernel level, significantly reducing the amount of data that needs to be processed in user space.
• Network Performance Insights: eBPF programs can monitor performance metrics, such as packet drops, latencies, and bandwidth usage, allowing for dynamic tuning of transport protocols based on real-time traffic conditions.
• Low-Latency Processing: eBPF’s ability to operate within the kernel ensures that packet processing happens as close to the hardware as possible, minimizing delays and improving overall system responsiveness.
• Ring Buffer for Data Handling: eBPF uses a ring buffer to store and transfer network data efficiently to user-space applications, ensuring fast and reliable communication between the kernel and (f)db.

Right now some basic program exists that can be loaded but it does not do anything more then writing into ring buffer

This is something to be dealt with later on...

For example, can be used for mass writes without ACK, different types of discoveries or for example DDoS detection or non-whitelisted server ip access, idk...

Future Considerations

While achieving all these goals without using advanced techniques like DPDK (Data Plane Development Kit) will be challenging, the initial prototype will focus on building a solid foundation. Advanced optimizations can be layered on top later, depending on the project's needs and performance demands.

This approach ensures that (f)db remains both adaptable and scalable, with the potential to handle a variety of use cases while maintaining high performance.

Diagram

graph TD;
    %% Main Entry Point and CLI Commands
    A[Main Entry Point - main.go] --> B[CLI Manager - urfave/cli]
    B --> C1[Certs Command]
    B --> C2[Benchmark Command]
    B --> C3[Serve Command]

    %% FDB Initialization
    C2 -->|Initializes| D[FDB Instance]
    C3 -->|Initializes| D

    %% FDB Components
    D -->|Creates| E[Transport Manager]
    D -->|Creates| F[Database Manager]
    D -->|Sets up| G[Logger]
    D -->|Configures| H[Pprof Profiler]

    %% Transport Manager and Transports
    E -->|Registers| I1[QUIC Transport]
    E -->|Registers| I2[UDS Transport]
    E -->|Registers| I3[TCP Transport]
    E -->|Registers| I4[UDP Transport]
    E -->|Registers| I5[Dummy Transport]

    %% Individual Transports and Servers
    I1 -->|Implements| J1[QUIC Server]
    I2 -->|Implements| J2[UDS Server]
    I3 -->|Implements| J3[TCP Server]
    I4 -->|Implements| J4[UDP Server]
    I5 -->|Implements| J5[Dummy Server]

    %% Handlers and Operations
    subgraph Transport Handlers
        J1 --> K[Read/Write Handlers]
        J2 --> K
        J3 --> K
        J4 --> K
        J5 --> K
    end

    K -->|Performs| L1[Handle Read]
    K -->|Performs| L2[Handle Write]

    %% Database Interaction
    L1 --> M[Database Manager]
    L2 --> M

    M -->|Uses| N[MDBX Database]

    %% Benchmarking Suite
    subgraph Benchmarking
        C2 --> O[Suite Manager]
        O -->|Manages| P1[QUIC Suite]
        O -->|Manages| P2[Dummy Suite]
        O -->|Manages| P3[UDS Suite]
        O -->|Manages| P4[TCP Suite]
        O -->|Manages| P5[UDP Suite]

        P1 -->|Runs| Q[Write/Read Benchmarks]
        P2 -->|Runs| Q
        P3 -->|Runs| Q
        P4 -->|Runs| Q
        P5 -->|Runs| Q
    end

    %% Configuration and Logger
    D --> R[Configuration]
    D --> G
    G --> S[Zap Logger]

    %% Pprof Profiler
    D --> H
    H --> T[Pprof Server]

    %% Database Manager Details
    M --> U[DB Manager]
    U --> V[Provides DB Access]

    %% Legend
    classDef component fill:#fcfcfc,stroke:#333,stroke-width:2px;
    class A,B,C1,C2,C3,D,E,F,G,H,M,N,O,P1,P2,P3,P4,P5,Q,R,S,T,U,V component;

Explanation of the Diagram:

1. Main Entry Point: This is the main starting point of the application, located in main.go. It utilizes the urfave/cli package to manage multiple CLI commands, such as certs, benchmark, and serve. Each command has specific function