JunoDB - A secure, consistent and highly available key-value store

What is JunoDB

JunoDB is PayPal's home-grown secure, consistent and highly available key-value store providing low, single digit millisecond, latency at any scale.

• High Level Architecture
• Wire protocol
• Contributing

Getting Started with the JunoDB Server

Clone the repository from github

git clone https://github.com/paypal/junodb.git

Set BUILDTOP variable

export BUILDTOP=<path_to_junodb_folder>/junodb
cd $BUILDTOP

Continue building JunoDB server with

1. Docker build or
2. Manual build <br>

Docker Build

• Docker
  • Install Dependencies
  • Build JunoDB
  • Run JunoDB
  • Generate Secrets for Dev
  • Validate JunoDB

<!-- toc -->

Note : Docker build supported platforms

• Linux (Ubuntu)
• OS X (macOS)

<h3 id="docker_install_dependencies">Install Dependencies</h3>

Install Docker Engine version 20.10.0+ Check for existing docker version

docker version

Install Docker if not installed or version is older than 20.10.0

docker/setup.sh
#If you are not added to the docker group, you will have to logout and login in the machine after running docker/setup.sh

If the user is not added to docker group, you may add manually. Logout and re-login after this step.

sudo usermod -a -G docker $USER

Verify that your username is added to docker group

groups

#or
cat /etc/group | grep docker

<h3 id="docker_build_junodb">Build JunoDB</h3>

#Login to docker hub account 
docker login

# Build junodb docker images
#etcd
#clustercfg
#storageserv
#proxy 
#junoclient


docker/build.sh 

<h3 id="docker_run_junodb">Run JunoDB</h3>

# Setup junodb network and start junodb services
#etcd
#clustercfg
#storageserv
#proxy 
#junoclient


# JunoDB proxy service listens on port 
# :5080 TLS and :8080 TCP
docker/start.sh 

Shutdown JunoDB services

# This will shutdown junodb services
#etcd
#clustercfg
#storageserv
#proxy 
#junoclient

docker/shutdown.sh 

Manually Run JunoDB services

#This can be done instead of ./start.sh to start up the docker services

cd $BUILDTOP/docker/manifest

# To run junodb services in --detach mode (recommended)
docker compose up -d

# Juno proxy service listens on port 
# :5080 TLS and :8080 TCP

#To view the running containers 
docker ps

# To stop junodb services
docker compose down

<h3 id="docker_secrets">Generate Secrets for Dev</h3>

<br>

NOTE: secrets for TLS and Encryption can be generated for dev/testing.

sh $BUILDTOP/docker/manifest/config/secrets/gensecrets.sh

## generated secrets
# server.crt/server.pem - certificate/key for junodb proxy for TLS 
# ca.crt - CA cert
# keystore.toml - sample keystore file

<h3 id="docker_validate_junodb">Validate JunoDB</h3>

Login to docker client and check connection with proxy

docker exec -it junoclient bash -c 'nc -vz proxy 5080'

<br>

You can also test the junodb server by running junocli and junoload

<br>

JunoCLI<br>

The following commands log in to the docker client and run the ./junocli command directly. The proxy ip is aliased as "proxy"

1. CREATE

docker exec -it junoclient bash -c '/opt/juno/junocli create -s proxy:8080 -c config.toml -ns test_ns test_key test_value'

2. GET

docker exec -it junoclient bash -c '/opt/juno/junocli get -s proxy:8080 -c config.toml -ns test_ns test_key'

3. UPDATE

docker exec -it junoclient bash -c '/opt/juno/junocli update -s proxy:8080 -c config.toml -ns test_ns test_key test_value_updated'

4. DESTROY

docker exec -it junoclient bash -c '/opt/juno/junocli destroy -s proxy:8080 -c config.toml -ns test_ns test_key'

More about junocli here <br>

<br>

Junoload<br>

The following command logs in to the docker client and runs the ./junoload command directly. Junoload can be used for benchmarking juno server. The proxy ip is aliased as "proxy"

docker exec -it junoclient bash -c '/opt/juno/junoload -s proxy:5080 -ssl -c config.toml -o 1'

More about junoload here

<br> <br>

Manual Build

Note : Manual build supported platforms

• Linux (Ubuntu 20.04)

The following sections explain the process for manually building the JunoDB server without Docker. These instructions are based on an Ubuntu 20.04.5 system

• Manual
  • Install Dependencies
  • Build JunoDB
  • Run JunoDB
  • Validate
  • Shutdown JunoDB

<h3 id="manual_install_dependencies">Install Dependencies</h3>

Install OpenSSL 1.0.2g+

sudo apt install openssl

Install multilog

sudo apt install daemontools

Install dependencies for rocksdb

sudo apt-get install build-essential libgflags-dev libsnappy-dev zlib1g-dev libbz2-dev liblz4-dev libzstd-dev -y

<br> Install Python

#install python
sudo apt-get install python3.8
#set soft link
cd /usr/bin
sudo ln -s python3.8 python

<h3 id="manual_build_junodb">Build JunoDB</h3>

binary_build/build.sh

<br>

<h3 id="manual_run_junodb">Run JunoDB</h3>

export JUNO_BUILD_DIR=$BUILDTOP/release-binary/code-build
script/deploy.sh

<br>

<h3 id="manual_validate_junodb">Validate JunoDB</h3>

#Validate if deploy was successful by checking if the proxy (junoserv), storage (junostorageserv), and etcd (junoclusterserv) processes are running
ps -eaf | grep juno
#There should be 41 processes running
#5 for junoclusterserv (3 logs, 1 etcdsvr.py, 1 etcdsvr_exe)
#20 for junostorageserv (6 logs, 1 manager, 12 workers, 1 monitor)
#16 for junoserv (6 logs, 1 manager, 8 workers, 1 monitor)

<br>

Test out the server using junocli and junoload command

See instructions for junocli here <br> See instructions for junoload here

Run functional tests

#Assuming user is in $BUILDTOP folder
test/functest/configsetup.sh
cd test/functest
$BUILDTOP/release-binary/tool/go/bin/go test -v -config=config.toml

<br>

Run unit tests

#Assuming user is in $BUILDTOP folder
cd test/unittest
$BUILDTOP/release-binary/tool/go/bin/go test -v

<h3 id="manual_shutdown_junodb">Shutdown JunoDB Services</h3>

#Assuming user is in $BUILDTOP folder
script/deploy.sh stop

<br>