中文

Table of Contents

• Introduce
• 10X Throughput
• Zero Dependencies
• Try it
  • Build
  • Run server
  • Run benchmark
  • Client usage
  • Client examples
  • Server configuration
• Cluster management
  • Configure a multi-node cluster
  • Run admin tools
  • Cluster changes
• Advanced
  • Import project to IDE
  • Build raft server through code
  • Trim JRE with JLink
• About me

Introduce

The Dongting project is a high-performance engine that integrates RAFT, configuration server, messaging queues. Features are as follows:

• Multi RAFT group support: Running multiple RAFT groups within a same process. Dynamic addition, removal, and updating of RAFT groups (sharding), allowing your cluster to scale dynamically. The state machine runs in the raft framework can be customized.
• Distributed configuration server DtKV: Tree-based structure, supports linear consistency general K/V operations, watch, TTL expiration, and distributed lock, similar to etcd.
  • DtKV is an in-memory database, so the total data size cannot be too large, but it uses raft log as redo log, creates snapshots periodically, and will not lose a single record even in power failure.
  • Natively supports tree-like directories, the complexity of many operations is O(1), such as watching directories, very efficient.
  • Supports temporary directories, which will be automatically deleted as a whole after TTL expires. The deletion operation is atomic.
  • Does not support transactions, but provides CAS and very easy-to-use distributed locks.
• (Planned) MQ (message queues): Use RAFT log as message queue log.

10X Throughput

Dongting is developed using performance-oriented programming.

In simple tests where both server and client run on the same machine, modern high-performance PCs using server default settings can easily achieve over 1 million TPS with the benchmark program, and RT won't be too large either.

Zero Dependencies

Are you still troubled by dependency management issues that spread like wildfire? Have you ever been ridiculed for a 1GB image size? You have been saved now!

Dongting has no dependencies. It does not rely on any third-party jar files (only SLF4J is optional, if it is not in the classpath, the project will use JDK logging). Dongting core has only two JAR packages, client and server together are less than 1MB, you can easily embed it into your program, whether it is client or server.

Dongting does not place high demands on your JDK: client only requires Java 8, server only requires Java 11.

Dongting does not require using high-performance hardware, such as RDMA or Optane, and it does not rely on any third-party services such as storage services provided by Amazon or other cloud service providers. It can even run well on HDD disks and Raspberry Pi.

Dongting does not require you to adjust Linux kernel parameters to achieve optimal performance (in production environments you may not even have permission to do so).

Dongting is completely modular (while still being compatible with java8), and the server can be deployed with a minimized JRE using jlink.

Try it

Build

First build, the artifacts are under target/dongting-dist:

mvn clean package -DskipUTs

If you want to install the artifact:

mvn clean install -Dmaven.test.skip=true

The directory structure after build is as follows:

dongting-dist/
├── bin/                      # Scripts directory
│   ├── benchmark.sh          # Benchmark script
│   ├── benchmark.bat         # Benchmark script (Windows)
│   ├── start-dongting.sh     # Start server script (Linux/Mac)
│   ├── start-dongting.bat    # Start server script (Windows)
│   ├── stop-dongting.sh      # Stop server script (Linux/Mac)
│   ├── stop-dongting.bat     # Stop server script (Windows)
│   ├── dongting-admin.sh     # Admin tool script (Linux/Mac)
│   └── dongting-admin.bat    # Admin tool script (Windows)
├── lib/                      # JAR packages directory
│   ├── dongting-client-x.y.z-SNAPSHOT.jar
│   ├── dongting-server-x.y.z-SNAPSHOT.jar
│   ├── dongting-dist-x.y.z-SNAPSHOT.jar # Bootstrap jar, not required if building raft server through code
│   ├── slf4j-api-x.y.z.jar
│   ├── logback-x.y.z.jar
│   └── logback-x.y.z.jar
├── conf/                     # Configuration files directory
│   ├── config.properties
│   ├── servers.properties
│   ├── client.properties
│   ├── logback-server.xml
│   ├── logback-admin.xml
│   └── logback-benchmark.xml
├── docs/                     # Documentation directory
└── logs/                     # Logs directory
    ├── dongting-server.log   # Server log (generated at runtime)
    └── dongting-stats.log    # Stats log (generated at runtime)

Run server

In the bin directory, run the following command to start the server:

./start-dongting.sh

The server will start and listen on port 9331(servers internal communication, e.g., raft replication) and 9332(service port). By default, a DtKV instance with groupId 0 will be started.

Run the following command to stop the server:

./stop-dongting.sh

Run benchmark

Run the following command to start the benchmark client:

./benchmark.sh -g 0

You may need to adjust parameters to achieve maximum throughput, for example:

./benchmark.sh -g 0 --max-pending 10000 --client-count 2

Try Java 21 virtual threads (need Java 21)

./benchmark.sh -g 0 --sync --thread-count 4000

See my results (AMD 9700X 6C12T, ZhiTai TiPro 9000 running in PCI-E 4.0 mode):

PS D:\dongting-dist\bin> .\benchmark.bat -g 0
Configuration:
  Config file: D:\dongting-dist\conf\client.properties
  Servers: 1,127.0.0.1:9332
  Group ID: 0

Benchmark config:
  Java 21, async put, 10000 keys, 256 bytes value, 2000 total maxPending
  1 clients, one thread per client

Warming up for 3 seconds...

[Warmup] TPS: 1,133,471, Success: 1,133,471, Fail: 0, Avg: 1,754 us, Max: 30,000 us
[Warmup] TPS: 1,355,100, Success: 1,355,100, Fail: 0, Avg: 1,483 us, Max: 13,000 us
[Warmup] TPS: 1,349,302, Success: 1,349,302, Fail: 0, Avg: 1,456 us, Max: 14,000 us
Warmup complete, starting benchmark...

[Now] TPS: 1,389,229, Success: 1,389,229, Fail: 0, Avg: 1,440 us, Max: 13,000 us
[Now] TPS: 1,331,667, Success: 1,331,667, Fail: 0, Avg: 1,494 us, Max: 13,000 us
[Now] TPS: 1,324,262, Success: 1,324,262, Fail: 0, Avg: 1,504 us, Max: 12,000 us
[Now] TPS: 1,369,651, Success: 1,369,651, Fail: 0, Avg: 1,453 us, Max: 13,000 us
[Now] TPS: 1,380,549, Success: 1,380,549, Fail: 0, Avg: 1,461 us, Max: 13,000 us
[Now] TPS: 1,344,967, Success: 1,344,967, Fail: 0, Avg: 1,479 us, Max: 19,000 us
[Now] TPS: 1,327,548, Success: 1,327,548, Fail: 0, Avg: 1,499 us, Max: 24,000 us
[Now] TPS: 1,363,815, Success: 1,363,815, Fail: 0, Avg: 1,458 us, Max: 12,000 us
[Now] TPS: 1,359,154, Success: 1,359,154, Fail: 0, Avg: 1,472 us, Max: 13,000 us
[Now] TPS: 1,345,889, Success: 1,345,889, Fail: 0, Avg: 1,476 us, Max: 13,000 us

Benchmark config:
  Java 21, async put, 10000 keys, 256 bytes value, 2000 total maxPending
  1 clients, one thread per client

[Final] TPS: 1,353,905, Success: 13,539,049, Fail: 0, Avg: 1,473 us, Max: 24,000 us

PS D:\dongting-dist\bin>

Client usage

The built-in DtKV in Dongting supports the following features:

• get/batchGet
• put/batchPut
• remove/batchRemove
• list
• mkdir
• compareAndSet
• putTemp/makeTempDir (node with ttl)
• createLock/createAutoRenewalLock

DtKV uses . as a separator for keys. For example, you can access key1 under the dir1 directory using "dir1.key1". The value cannot be null or empty string.

To use the DtKV client, you need to include the dongting-client.jar (300+KB), with no other dependencies.

The following is a simple example of initializing the DtKV client:

// dongting supports multi-raft, so we need to specify the group id, default group id is 0
int groupId = 0;
KvClient kvClient = new KvClient();
kvClient.start();
// add node definition at runtime, each node has a unique positive integer id and a host:servicePort address
kvClient.getRaftClient().clientAddNode("1,127.0.0.1:9332");
// kvClient.getRaftClient().clientAddNode("1,192.168.0.1:9332;2,192.168.0.2:9332;3,192.168.0.3:9332");
// add group definition at runtime, here we add a group with groupId 0 and 3 nodes with ids 1, 2, and 3
kvClient.getRaftClient().clientAddOrUpdateGroup(groupId, new int[]{1,2,3});

Make sure to specify the correct port. Each raft node exposes two ports: One is the replicate port, default 9331, which is used for internal communication such as replication between raft nodes. The AdminRaftClient also connects to this port. The other is the service port, default 9332, which is used for connections from clients like KvClient.

KvClient provides synchronous and asynchronous interfaces. For a single KvClient, asynchronous operations achieve maximum throughput, while synchronous operations require many threads (or virtual threads) to reach higher throughput. It is important to note that callbacks for asynchronous operations may be executed on raft thread or IO threads. Therefore, you should never perform any blocking or CPU-intensive operations within these callbacks. If you are unsure or lack advanced skills, it is strongly recommended