Instaclustr Esop

image:https://img.shields.io/maven-central/v/com.instaclustr/esop-core.svg?label=Maven%20Central[link=https://central.sonatype.com/search?q=esop-core] image:https://img.shields.io/circleci/build/github/instaclustr/esop/master.svg["Instaclustr",link="https://app.circleci.com/pipelines/github/instaclustr/esop?branch=master"]

Swiss knife for Apache Cassandra backup and restore

image::Esop.png[Esop,width=50%]

• Website: https://www.instaclustr.com/

• Documentation: https://www.instaclustr.com/support/documentation/

This repository is home of backup and restoration tools from Instaclustr for Cassandra called https://en.wikipedia.org/wiki/Aesop[Esop]

Esop of version 2.0.0 is not compatible with any Esop of version 1.x.x. Esop 2.0.0 has changed the manifest format which is uploaded to a remote location hence, as of now, Esop 2.0.0 can not read manifests for versions 1.x.x.

It is recommended all your Esop instances to 4.0.3 version before upgrading them to 4.1.0 version. Esop v4.1.0 depends on the AWS S3 Client Encryption v4.0.0 version which is not compatible with versions older than v3.6.0.

Esop is able to perform these operations and has these features:

• Backup and restore of SSTables
• Backup and restore of commit logs
• Restoration of data into a Cassandra schema or diffrent table schema
• Backing-up to and restoring from S3 (Oracle and Ceph via Object Gateway too), Azure, or GCP, or into any local destination or other storage providing they are easily implementable
• listing of backups and their removal (from remote location, s3, azure, gcp), global removal of backups across all nodes in cluster
• periodic removal of backups (e.g. after 10 days)
• Effective upload and download—it will upload only SSTables which are not present remotely so any subsequent backups will upload and restores will download only the difference
• When used in connection with https://github.com/instaclustr/icarus[Instaclustr Icarus] it is possible to backup simultaneously so there might be more concurrent backups which may overlap what they backup
• Possible to restore whole node / cluster from scratch
• In connection with Icarus, it is possible to restore on a running cluster so no downtime is necessary
• It takes care of details such as initial tokens, auto bootstrapping, and so on...
• Ability to throttle the bandwidth used for backup
• Point-in-time restoration of commit logs
• verification of downloaded data - computes hases upon upload and download and it has to match otherwise restoration fails
• it is possible to restore tables under different names so they do not clash with your current tables ideal when you want to investigate / check data before you restore the original tables, to see what data you will have once you restore it
• retry of failed operations against s3 when uploading / downloading failure happens
• support of multiple data directories for Cassandra node

This tool is used as a command line utility and it is meant to be executed from a shell or from scripts. However, this tooling is also embedded seamlessly into Instaclustr Icarus. The advantage of using Icarus is that you may backup and restore your node (or whole cluster) remotely by calling a respective REST endpoint so Icarus can execute respective backup or restore operation. Icarus is designed to be run alongside a node and it talks to Cassandra via JMX (no need to expose JMX publicly).

In addition, this tool has to be run in the very same context/environment as a Cassandra node—it needs to see the whole directory structure of a node (data dir etc.) as it will upload these files during a backup and download them on a restore. If you want to be able to restore and backup remotely, use Icarus which embeds this project.

Supporter Cassandra Versions

Since we are talking to Cassandra via JMX, almost any Cassandra version is supported. We are testing this tool with Cassandra 5.x and 4.x.

Usage

Released artifact is on https://search.maven.org/artifact/com.instaclustr/esop[Maven Central]. You may want to build it on your own by standard Maven targets. After this project is built by mvn clean install (refer to <<build and tests>> for more details), the binary is in target and it is called instaclustr-esop.jar. This binary is all you need to backup/restore. It is the command line application, invoke it without any arguments to see help. You can invoke help backup for backup command, for example.

---

$ java -jar target/esop.jar Missing required subcommand. Usage: <main class> [-V] COMMAND -V, --version print version information and exit Commands: backup Take a snapshot of this nodes Cassandra data and upload it to remote storage. Defaults to a snapshot of all keyspaces and their column families, but may be restricted to specific keyspaces or a single column-family. restore Restore the Cassandra data on this node to a specified point-in-time. commitlog-backup Upload archived commit logs to remote storage. commitlog-restore Restores archived commit logs to node.

You get detailed help by invoking help subcommand like this:

---

$ java -jar target/esop.jar backup help

Connecting to Cassandra Node

As already mentioned, this tool expects to be invoked alongside a node - it needs to be able to read/write into Cassandra data directories. For other operations such as knowing tokens etc., it connects to respective node via JMX. By default, it will try to connect to service:jmx:rmi:///jndi/rmi://127.0.0.1:7199/jmxrmi. It is possible to override this and other related settings via the command line arguments. It is also possible to connect to such nodes securely if it is necessary, and this tool also supports specifying keystore, truststore, user name and password etc. For brevity, please consult the command line help.

If you do not want to specify credentials on the command line, you can put them into a file and reference it by --jmx-credentials options. The content of this file is treated as a standard Java property file, expecting this content:

---

username=jmxusername password=jmxpassword keystorePassword=keystorepassword truststorePassword=truststorepassword

Not all sub-commands require the connection to Cassandra to exist. As of now, a JMX connection is necessary for:

. backup of tables/keyspaces . restore of tables/keyspaces (hard linking and importing strategies)

The next release of this tool might relax these requirements so it would be possible to backup and restore a node which is offline.

For backup and restore of commit logs, it is not necessary to have a node up as well in case you need to restore a node from scratch or if you use <<In-place restoration strategy>>.

Storage Location

Data to backup and restore from, are located in a remote storage. This setting is controlled by flag --storage-location. The storage location flag has very specific structure which also indicates where data will be uploaded. Locations consist of a storage protocol and path. Please keep in mind that the protocol we are using is not a real protocol. It is merely a mnemonic. Use either s3, gcp, azure or file.

The format is:

protocol://bucket/cluster/datacenter/node

• protocol is either s3,azure,'gcp, or file.
• bucket is name of the bucket data will be uploaded to/downloaded from, for example my-bucket
• cluster is name of the cluster, for example, test-cluster
• datacenter is name of the datacenter a node belongs to, for example datacenter1
• node is identified of a node. It might be e.g. 1, or it might be equal to node id (uuid)

The structure of a storage location is validated upon every request.

If we want to backup to S3, it would look like:

s3://cassandra-backups/test-cluster/datacenter1/1

In S3, data for that node will be stored under key test-cluster/datacenter1/1. The same mechanism works for other clouds.

For file protocol, use file:///data/backups/test-cluster/dc1/node1. In every case, file has to start with full path (file:///, three slashes). File location does not have a notion of a bucket, but we are using it here regardless—in the following examples, the bucket will be a.

It does not matter you put slash at the end of whole location, it will be removed.

.file path resolution |=== |storage location |path

|file:///tmp/some/path/a/b/c/d |/tmp/some/path/a

|file:///tmp/a/b/c/d |/tmp/a |===

Authentication Against a Cloud

In order to be able to download from and upload to a remote bucket, this tool needs to pick up security credentials to do so. This varies across clouds. file protocol does not need any authentication.

S3

The resolution of credentials for S3 uses the same resolution mechanism as the official AWS S3 client uses. The most notable fact is that if no credentials are set explicitly, it will try to resolve them from environment properties of the node it runs on. If that node runs in AWS EC2, it will resolve them by help of that particular instance.

S3 connectors will expect to find environment properties AWS_ACCESS_KEY_ID and AWS_SECRET_KEY. They will also accept AWS_REGION.

It is possible to connect to S3 via proxy; please consult "--use-proxy" flag and "--proxy-*" family of settings on command line.

Azure

Azure module expects AZURE_STORAGE_CONNECTION_STRING or AZURE_STORAGE_ACCOUNT and AZURE_STORAGE_KEY environment variable to be set. Only one of the options are necessary. If both are set, it will fail.

GCP

GCP module expects GOOGLE_APPLICATION_CREDENTIALS environment property or google.application.credentials to be set with the path to service account credentials.

Directory Structure of a Remote Destination

Cassandra data files as well as some meta-data needed for successful restorati