ByteFacets Spinel

High-Performance Column-Oriented Data Streaming Library

Spinel is an efficient, column-oriented data streaming framework designed for real-time tabular data processing. Built for both embedded and distributed architectures, Spinel enables efficient data transformations with minimal memory overhead and maximum throughput.

Key Benefits

Exceptional Performance

Column-Oriented Storage: Optimized memory layout with superior cache locality and reduced fragmentation
Reference-Passing Design: Zero-copy data access through pointer-like references to source rows
Single-Pass Processing: Data flows directly through transformations without intermediate copies
Minimal Memory Footprint: Array-based storage eliminates object overhead

Real-Time Capabilities

Event-Driven Architecture: Push-based updates with minimal latencies
Live Data Streaming: WebSocket and gRPC integration for real-time client updates
Incremental Processing: Only changed data is processed and transmitted

Flexible Integration

Embeddable Library: Seamlessly integrate into existing applications (Kafka, Spring Boot, microservices)
Multi-Process Communication: Built-in IPC operators for distributed systems
Protocol Agnostic: Support for WebSockets, gRPC, and custom protocols

Use Cases

Real-Time Dashboards

Perfect for financial trading platforms, IoT monitoring, and business intelligence dashboards requiring:

Live market data feeds with sub-millisecond updates
Real-time KPI monitoring and alerting
Interactive data exploration with instant filtering and aggregation
Multi-user concurrent access with efficient resource utilization

Process-to-Process Data Pipelines

Ideal for microservices architectures and data processing workflows:

High-throughput ETL pipelines with complex joins and transformations
Real-time analytics engines processing streaming data
Event sourcing systems with live projections
Inter-service communication with structured data contracts

Embeddable

Seamlessly embed into existing applications. All transforms and data processing are just plain Java, and require no other infrastructure. Your topology can be totally contained in your process, or you can connect topologies across processes.

You can also author your own operators by conforming to the TransformInput and TransformOutput interfaces.

Event flow into an Operator begins with a Schema update, which describes the data to the operator, but also is how the operator will access the data. Data in Spinel is typically column-oriented and accessible through the Field interfaces.

Example Architecture Overview

         +-----------+     +-----------+
         |   order   |     |  product  |
         |   table   |     |   table   |
         +---------+-+     +-+---------+
                   |         |
               +---v---------v---+
               |      join       |
               | (on product_id) |
               +--------+--------+
               |                 |
        +------v------+   +------v------+
        |    filter   |   |   group-by  |
        |    (user)   |   |  (category) |
        +-------------+   +-------------+
               |                 |
       +-------v-------+ +-------v-------+
       |   grpc sink   | |   web-socket  |
       |               | |     client    |
       +---------------+ +---------------+

Core Operators

Data Storage & Access

KeyedTables: Indexed tables by primitive types (int, long, string)
Column-Oriented Fields: Each field stored in optimized arrays

Transformations

Filter: High-performance row filtering with custom predicates
Join: Inner/outer joins with configurable key handling strategies
GroupBy: Real-time aggregations with incremental updates
Projection: Field selection, aliasing, reordering, and calculated fields
Union: Merge multiple data streams into unified output
Conflation: Reduce update frequency to conserve CPU and network bandwidth
Projection: Projections for calculated fields, field selection, and aliasing

Integration & Communication

Subscription Management: Multi-client subscription handling
Protocol Adapters: WebSocket, gRPC, NATS.io, and custom protocol support

Performance Characteristics

Memory Efficiency

Column-oriented storage reduces memory fragmentation
Reference-passing eliminates unnecessary data copying
Compact, array-based field indexing

Processing Speed

Single-pass transformations minimize CPU cycles
Cache-friendly data access patterns

Network Efficiency

Protocol Buffer encoding for minimal bandwidth usage
Incremental updates reduce network traffic by 90%
Built-in conflation prevents message flooding

Integration Examples

NATS.io Integration (NATS Examples)

Tables can be emitted to and sourced from NATS KV Buckets. There are two varieties of sourcing from a KV bucket:

Custom KV (a non-spinel KV Bucket publisher) exposed as a Spinel Table

You control the deserialization of the bucket's entries.

KeyValue mdKeyValue = connection.keyValue(mdBucketName);
mdAdapter = NatsKvAdapterBuilder.natsKvAdapter()
            // custom handler for decoding and writing data to schema fields
            .updateHandler(new MdKeyValueHandler())
            // adapter must know the schema ahead of time
            .addFields(marketDataBucketFields())
            .eventLoop(eventLoop)
            .build();
mdKeyValue.watchAll(mdAdapter.keyValueWatcher());

Spinel KV (a spinel KV Bucket Sink) exposed as a Spinel Table

The framework encodes the schema into the bucket allowing some flexibility with schema changes between publisher and consumer

// Server/Publisher side
Connection connection = Nats.connect(options);
KeyValue ordersKeyValue = connection.keyValue(bucketName);
NatsKvSink sink =
       NatsKvSinkBuilder.natsKvSink()
               .keyValueBucket(ordersKeyValue)
               .subjectBuilder(
                       FieldSequenceNatsSubjectBuilder.fieldSequenceNatsSubjectBuilder(
                               List.of("InstrumentId", "OrderId")))
               .build();
Connector.connectInputToOutput(sink, orders);

// Client/Consumer side
KeyValue ordersKeyValue = connection.keyValue(bucketName);
NatsKvSource orderSource = NatsKvSourceBuilder.natsKvSource().eventLoop(eventLoop).build();
ordersKeyValue.watchAll(orderSource.keyValueWatcher());

Vaadin Integration (in development)

(Vaadin Integration README)

Spring Boot Integration (in development)

@Configuration
public class TopologyBuilder {
   private final RegisteredOutputsTable outputs = RegisteredOutputsTable.registeredOutputsTable();
   private final EventLoop eventLoop;
   private final DefaultSubscriptionProvider subscriptionProvider;

   public TopologyBuilder() {
      this.eventLoop = new DefaultEventLoop(r -> { return new Thread(r, "server-thread"); });
      // .... setup transforms and register them in the RegisteredOutputsTable 
      outputs.register("orders", orders);
      outputs.register("instruments", instruments);
      outputs.register("order-view", join.output());
      subscriptionProvider = DefaultSubscriptionProvider.defaultSubscriptionProvider(outputs);
   }

   @Bean
   HandlerMapping handlerMapping() {
      final var mapping =
              new SimpleUrlHandlerMapping(
                      Map.of("/ws/spinel", new SpinelWebSocketHandler(subscriptionProvider, eventLoop)));
      mapping.setOrder(-1);
      return mapping;
   }

   @Bean
   public OutputRegistry registry() {
      return outputs;
   }
   ...

Kafka Streams Integration (coming soon)

Use Spinel transformations to operate over state stores.

Quick Start

Basic Table Operations

// Create a keyed table
IntIndexedStructTable<Order> orders = intIndexedStructTable(Order.class).build();
Order facade = orders.createFacade(); // reusable facade

// Add data
orders.beginAdd(1, facade)
    .setInstrumentId(100)
    .setQuantity(500)
    .setPrice(25.50);
orders.endAdd();

// Query data
int row = orders.lookupKeyRow(1);
orders.moveToRow(facade, row);
double price = facade.getPrice(); // 25.50

Real-Time Transformations

// Create a join between orders and instruments
Join orderView = JoinBuilder.lookupJoin("order-view")
    .inner()
    .joinOn(List.of("InstrumentId"), List.of("InstrumentId"), 10)
    .build();

// Connect data sources
Connector.connectInputToOutput(orderView.leftInput(), orders);
Connector.connectInputToOutput(orderView.rightInput(), instruments);

// Register for real-time subscriptions
OutputRegistry registry = RegisteredOutputsTable.registeredOutputsTable();
registry.register("enriched-orders", orderView.output());

Topology Composition

Spinel offers two powerful approaches for building data processing topologies:

1. TransformBuilder API - Declarative Approach

The TransformBuilder provides a fluent, declarative API for complex topologies:

TransformBuilder transform = TransformBuilder.transform();
transform.intIndexedStructTable(Order.class)
    .then()
         .filter("open-orders").where("open == true")
    .then()
         .groupBy("open-by-instrument")
           .groupByFields("InstrumentId")
           .addAggregation(sumToInt("Quantity", "TotalQuantity"))
           .addAggregation(sumToInt("Notional", "TotalNotional"));
transf

Spinel

Install / Use

README