Downloads

| Stargazers Over Time | Contributors Over Time | |:-----------------------------------------------------------------------------------------------------------------:|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:| | | |

Introduction中文

PikiwiDB is a high-performance, large-capacity, multi-tenant, data-persistent elastic KV data storage system using RocksDB as the storage engine. It is fully compatible with the Redis protocol and supports its commonly used data structures, such as string/hash/list/zset/set/geo/hyperloglog/pubsub/bitmap/stream, etc. Redis Interface.

When Redis's in-memory usage exceeds 16GiB, it faces problems such as limited memory capacity, single-threaded blocking, long startup recovery time, high memory hardware costs, easily filled buffers, and high switching costs when one master and multiple replicas fail. The emergence of PikiwiDB is not to replace Redis but to complement it. PikiwiDB strives to completely comply with the Redis protocol, inherit Redis's convenient operation and maintenance design, and solve the bottleneck problem of Redis running out of memory capacity once the data volume becomes huge by using persistent storage. Additionally, PikiwiDB can support master-slave mode using the slaveof command, and it also supports full and incremental data synchronization.

PikiwiDB can be deployed in a single-machine master-slave mode (slaveof) or in a Codis cluster mode, allowing for simple scaling and shrinking. Migration from Redis to PikiwiDB can be smoothly executed by tools.

PikiwiDB Features

Protocol Compatibility: Fully compatible with the Redis protocol, emphasizing high performance, large capacity, low cost, and scalability.
Data Structures: Supports Redis's common data structures, including String, Hash, List, Zset, Set, Geo, Hyperloglog, Pubsub, Bitmap, Stream, ACL, etc.
Cold and Hot Data: Caches hot data and persistently stores the full data in RocksDB, implementing a hierarchical storage of cold and hot data.
High Capacity: Compared to Redis's in-memory storage, PikiwiDB supports data volumes in the hundreds of gigabytes, significantly reducing server resource consumption and enhancing data reliability.
Deployment Modes: Supports single-machine master-slave mode (slaveof) and Codis cluster mode, making scaling and shrinking simple.
Easy Migration: Smooth migration from Redis to PikiwiDB without modifying code.
Convenient Operation and Maintenance: Comprehensive operation and maintenance command documentation.

PikiwiDB Storage Engine Architecture

Supports multiple platforms: CentOS, Ubuntu, macOS, Rocky Linux
Multi-threaded model
Based on the RocksDB storage engine
Multiple granularity data caching model

Deployment Modes

1. Master-Slave Mode

Architecture similar to Redis
Good compatibility with Redis protocol and data structures
Each data structure uses a separate RocksDB instance
Master-slave adopts binlog asynchronous replication

PikiwiDB-Master-Slave

2. Distributed Cluster Mode

Adopts Codis architecture, supports multiple groups
Each group forms a master-slave set
Elastic scaling based on groups

PikiwiDB-Cluster

PikiwiDB User Showcase

PikiwiDB has been widely adopted by various companies for internal deployments, demonstrating its scalability and reliability. Some notable usage instances include:

360 Company: Internal deployment with a scale of 10,000+ instances, each having a data volume of 1.8TB.
Weibo: Internal deployment with 10,000+ instances.
Ximalaya(Xcache): 6,000+ instances with a massive data volume exceeding 120TB.
Getui (个推) Company: Internal deployment involving 300+ instances, with a cumulative data volume surpassing 30TB.

Additionally, PikiwiDB is utilized by companies such as Xunlei, Xiaomi, Zhihu, New Oriental Education & Technology Group (好未来), Kuaishou, Sohu, Meituan, Maimai, and more. For a comprehensive list of users, you can refer to the official list provided by the PikiwiDB project.

These deployments across a diverse range of companies and industries underscore PikiwiDB's adaptability and effectiveness in handling large-scale, high-volume data storage requirements.

1. Binary Package Installation

Users can directly download the latest binary version package from releases.

2. Compilation from Source

2.1 Supported Platforms
- Linux - CentOS
- Linux - Ubuntu
- macOS(Darwin)
2.2 Required Library Software
- gcc g++ supporting C++17 (version >= 9)
- make
- cmake (version >= 3.18)
- autoconf
- tar
2.3 Compilation Process
- 2.3.1. Get the source code
```
  git clone https://github.com/OpenAtomFoundation/pikiwidb.git
```
- 2.3.2. Switch to the latest release version
```
  git tag          # Check the latest release tag (e.g., v3.4.1)
  git checkout TAG # Switch to the latest version (e.g., git checkout v3.4.1)
```
- 2.3.3. Execute compilation
  If the machine's gcc version is less than 9, especially on CentOS6 or CentOS7, you need to upgrade the gcc version first. Execute the following commands:
```
  sudo yum -y install centos-release-scl
  sudo yum -y install devtoolset-9-gcc devtoolset-9-gcc-c++
  scl enable devtoolset-9 bash
```
  For the initial compilation, it is recommended to use the build script build.sh, which checks if the required software is available on the local machine.
```
  ./build.sh
```
  Note: The compiled files will be saved in the output directory.
  
  PikiwiDB is compiled by default in release mode, which does not support debugging. If debugging is needed, compile in debug mode.
```
  rm -rf output/
  cmake -B output -DCMAKE_BUILD_TYPE=Debug
  cd output && make
```
  Other components, such as codis, can also be compiled using build.sh.
```
  # Compile codis, default target, build-all
  ./build.sh codis

  # Compile codis, but only build codis-proxy
  ./build.sh codis codis-proxy
```
- 2.3.4. (Supplementary) Manual compilation based on Docker images
  - Centos7
    Reference link
```
    #1.Start a Centos container locally
```

Pikiwidb

Install / Use

README

Introduction中文

PikiwiDB Features

PikiwiDB Storage Engine Architecture

Deployment Modes

1. Master-Slave Mode

2. Distributed Cluster Mode

PikiwiDB User Showcase

Getting Started with PikiwiDB

1. Binary Package Installation

2. Compilation from Source

2.1 Supported Platforms

2.2 Required Library Software

2.3 Compilation Process