FlowG

</div>

FlowG is a log management platform that lets you ingest, transform, and query logs using a visual pipeline builder. It handles structured logs without requiring predefined schemas and relies on BadgerDB as its storage backend.

For more information, consult the documentation.

You wish to contribute? Please read our guidelines first.

:sparkles: Key Features

Pipelines: Build and manage log processing pipelines visually, with support for transformations with VRL scripts and conditional routing to dedicated streams
Dynamic Logs: Manage logs with varying structures without needing predefined schemas
Interoperability: Configuration and log ingestion can entirely be done via the REST API
Observability: Query and visualize logs in real-time in the integrated Web interface

pipeline screenshot

stream screenshot

:question: Why should I use this?

FlowG's primary goal is to make log refinement and routing as easy as possible. As such, it relies on React Flow to help you build such pipelines with as little code as possible. For the actual code part, we use the Vector Remap Language, which gives a solid base for log refinement.

It aims to replace tools like Logstash by integrating the feature right in the solution.

It also leverages BadgerDB which is a battle tested Key/Value database, with the right feature set to easily support indexing dynamically structured logs, as well as log compression.

:zap: Benchmark

Run with (see Build section bellow):

export BENCHMARK_ITERATIONS="100000"
cargo install oha
task docker:build test:bench

System:

CPU: 13th Gen Intel(R) Core(TM) i7-13700K
RAM: 64GB

Result:

Summary:
  Success rate: 100.00%
  Total:        5702.4524 ms
  Slowest:      150.2867 ms
  Fastest:      0.1578 ms
  Average:      2.8478 ms
  Requests/sec: 17536.3148

  Total data:   3.53 MiB
  Size/request: 37 B
  Size/sec:     633.64 KiB

:building_construction: Build

Requirements:

Go 1.22+
C Toolchain
Rust and Cargo (edition 2021)
NodeJS

go install github.com/go-task/task/v3/cmd/task@latest
task build:all

Then, start the server with:

./bin/flowg-server \
  --auth-dir ./data/auth \
  --log-dir ./data/logs \
  --config-dir ./data/config \
  --http-bind 127.0.0.1:5080 \
  --mgmt-bind 127.0.0.1:9113 \
  --syslog-bind 127.0.0.1:5514

Now, you can access:

the WebUI at http://localhost:5080
the API documentation at http://localhost:5080/api/docs
the Syslog Server at udp://localhost:5514
the Management API at http://localhost:9113:
- /health: health check, always return 200 OK
- /metrics: Prometheus Exporter

A default user root (password: root) and a default pipeline are bootsrapped if no configuration exists during startup.

To build the Docker image linksociety/flowg:latest locally:

task docker:build

:rocket: Deploy

Using Docker:

docker run \
  -p 5080:5080/tcp \
  -p 9113:9113/tcp \
  -p 5514:5514/udp \
  -v flowg-data:/data \
  linksociety/flowg:latest

:warning: EXPERIMENTAL :warning: Using Kubernetes (and Helm):

helm install flowg ./k8s/charts/flowg -n flowg-system --create-namespace

:memo: License

This software is released under the terms of the MIT License

Flowg

Install / Use

README