Trapheus

<p align="center"> <img width="300" height="280" src="screenshots/Trapheus.png"> </p> <p align="center"> <b>Restore RDS instances in AWS without worrying about client downtime or configuration retention.</b><br/> _{Trapheus can restore individual RDS instance or a RDS cluster. Modelled as a state machine, with the help of AWS step functions, Trapheus restores the RDS instance in a much faster way than the usual SQL dump preserving the same instance endpoint and confgurations as before.} </p> <p align="center"><a href="https://circleci.com/gh/intuit/Trapheus"><img src="https://circleci.com/gh/intuit/Trapheus.svg?style=svg" alt="TravisCI Build Status"/></a> <a href = "https://coveralls.io/github/intuit/Trapheus?branch=master"><img src= "https://coveralls.io/repos/github/intuit/Trapheus/badge.svg?branch=master" alt = "Coverage"/></a> <a href="https://www.serverless.com"><img src="https://github.com/intuit/Trapheus/assets/112308320/c7764b65-fb72-46ed-824b-c82c6374ec8b" alt="serverless badge"/></a> <a href="https://github.com/intuit/Trapheus/releases"><img src="https://img.shields.io/github/v/release/intuit/trapheus.svg" alt="release badge"/></a> </p>

<img src="https://ch-resources.oss-cn-shanghai.aliyuncs.com/images/lang-icons/icon128px.png" width="22px" /> English | 简体中文 | français

• Important: this application uses various AWS services and there are costs associated with these services after the Free Tier usage - please see the AWS pricing page for details.

Table of Contents

• ➤ Pre-Requisites
• ➤ Parameters
• ➤ Tagging
• ➤ Instructions
• ➤ Execution
• ➤ How it Works
• ➤ Contributing to Trapheus
• ➤ Blogs
• ➤ Contributors

Pre-Requisites

The app requires the following AWS resources to exist before installation:

1. python3.11 installed on local machine following this.

2. Configure AWS SES

   • Configure the SES sender and receiver email (SES Console->Email Addresses).
     • An SES email alert is configured to notify the user about any failures in the state machine. The sender email parameter is needed to configure the email ID through which the alert is sent out. The receiver email parameter is needed to set the email ID to which the alert is sent.

3. Create the S3 bucket where the system is going to store the cloud formation templates:

   • Proposed Name: trapheus-cfn-s3-[account-id]-[region]. It is recommended that the name contains your:
     • account-id, as the bucket names need to be global (prevents someone else having the same name)
     • region, to easily keep track when you have trapheus-s3 buckets in multiple regions

4. A VPC (region specific). The same VPC/region should be used for both the RDS instance(s), to be used in Trapheus, and Trapheus' lambdas.

   • Region selection consideration. Regions that support:
     • Email receiving . Check Parameters -> 'RecipientEmail' for more.
   • Example minimal VPC setup:
     • VPC console:
       • name: Trapheus-VPC-[region] (specify the [region] where you VPC is created - to easily keep track when you have Trapheus-VPCs in multiple regions)
       • IPv4 CIDR block: 10.0.0.0/16
     • VPC console->Subnets page and create two private subnets:
       • Subnet1:
         • VPC: Trapheus-VPC-[region]
         • Availability Zone: choose one
         • IPv4 CIDR block: 10.0.0.0/19
       • Subnet2:
         • VPC: Trapheus-VPC-[region]
         • Availability Zone: choose a different one than the Subnet1 AZ.
         • IPv4 CIDR block: 10.0.32.0/19
     • You have created a VPC with only two private subnets. If you are creating non-private subnets, check the ratio between private, public subnets, private subnet with dedicated custom network ACL and spare capacity.

5. One or more instances of an RDS database that you wish to restore.

   • Example minimal free RDS setup:
     • Engine options: MySQL
     • Templates: Free tier
     • Settings: enter password
     • Connectivity: VPC: Trapheus-VPC-[region]

Parameters

The following are the parameters for creating the cloudformation template:

1. --s3-bucket : [Optional] The name of the CloudFormation template S3 bucket from the Pre-Requisites.
2. vpcID : [Required] The id of the VPC from the Pre-Requisites. The lambdas from the Trapheus state machine will be created in this VPC.
3. Subnets : [Required] A comma separated list of private subnet ids (region specific) from the Pre-Requisites VPC.
4. SenderEmail : [Required] The SES sending email configured in the Pre-Requisites
5. RecipientEmail : [Required] Comma separated list of recipient email addresses configured in Pre-Requisites.
6. UseVPCAndSubnets : [Optional] Whether to use the vpc and subnets to create a security group and link the security group and vpc to the lambdas. When UseVPCAndSubnets left out (default) or set to 'true', lambdas are connected to a VPC in your account, and by default the function can't access the RDS (or other services) if VPC doesn't provide access (either by routing outbound traffic to a NAT gateway in a public subnet, or having a VPC endpoint, both of which incur cost or require more setup). If set to 'false', the lambdas will run in a default Lambda owned VPC that has access to RDS (and other AWS services).
7. SlackWebhookUrls : [Optional] Comma separated list of Slack webhooks for failure alerts.

Tagging

This section describes the currently supported set of tags and how to take advantage of them in support of common use cases around billing insight generation.

Tags provide additional means of grouping and subdividing costs, whether for the purposes of analysis or cost distribution. Explanations for how tags can be applied to resources inside AWS is provided here. To facilitate a consistent approach to handling known and foreseen use cases, the following tags has been added at stack level and as well as at resource levels.

<b>AppName</b> - Name of the application, default Trapheus <br> <b>AppComponent</b> - Name of the component, since this application targeted for DB restore, default component database <br> <b>AppFunction</b> - Application function name, default RestoreDB <br>

If you would like to change above defaults, change it in samconfig.toml

Every resource has tags as well, which override the default tag.

For example:

  Tags:
    AppComponent: "Lambda"
    AppFunction: "RenameDBInstance"

Instructions

Setup

To setup the Trapheus in your AWS account, follow the steps below:

1. Clone the Trapheus Git repository
2. AWS Credentials configuration. Trapheus uses boto3 as client library to talk with Amazon Web Services. Feel free to use any environment variables that boto3 supports to supply authentication credentials.
3. Run pip install -r requirements.txt to install the dependency graph
4. Run python install.py

<p align="center"><img src="screenshots/Trapheus.gif?raw=true"/></p>

Still facing an issue? Check the Issues section or open a new issue

The above will setup a CFT in your AWS account with the name provided during installation.

TO BE NOTED: The CFT creates the following resources:

1. DBRestoreStateMachine Step function state machine
2. Multiple lambdas to execute various steps in the state machine
3. LambdaExecutionRole: used across all lambdas to perform multiple tasks across RDS
4. StatesExecutionRole: IAM role with permissions for executing the state machine and invoking lambdas
5. S3 bucket: rds-snapshots-<your_account_id> where snapshots will be exported to
6. KMS key: is required to start export task of snapshot to s3
7. DBRestoreStateMachineEventRule: A Cloudwatch rule in disabled state, that can be used following above instructions based on user requirement
8. CWEventStatesExecutionRole: IAM role used by DBRestoreStateMachineEventRule CloudWatch rule, to allow execution of the state machine from CloudW