Sultan

An Open edX Remote Devstack by Appsembler

This toolkit helps your run devstack on cloud (currently GCP). It will also provide you with the necessary tools to help you using your local machine for development, and your cloud instance for showing and testing.

The ultimate goal of this toolkit is to help you running projects under development on the cloud while having the luxury to run the engineering side on your machine. No dependencies headaches anymore.

Some available and ready-to-use devstack images are going to help you running the project in a couple of minutes. You can write, delete, and start from scratch anytime with no regrets.

Documentation

An extensive documentation on the architecture and the toolkit commands can be found in the repo's wiki page here.

Appsembler Internal Doc here.

1. Before you start

• Make sure you have your SSH key added into our GCP Appsembler Devstack (appsembler-devstack-30) project. You can check that at GCP Compute Metadata.
• Make sure you have GCloud command-line tools installed.
• Make sure you have a working python 3.7 environment.

2. Quick Start

2.1. Clone and configure

If you're on Linux, follow the next steps to set up your Sultan devstack

$ git clone git@github.com:appsembler/sultan.git
$ cd sultan
$ sultan config init  # Make sure you're in the correct python environment, this will install the required package immediatly one you run it.
## configs/.configs.username is created

If you're on macOS, follow the steps to set up your Sultan devstack

$ git clone git@github.com:appsembler/sultan.git
$ cd sultan
$ ./sultan config init

NOTE

All the generated files will exist in $SULTAN_HOME, you can change the value of this from you configurations file.

2.2. Required configurations

The following configurations must be overridden in your recently-created config file:

• SSH_KEY: The private key that has access to the GCP project where we will create your instance
• PROJECT_ID: The GCloud project ID.
• SERVICE_ACCOUNT_EMAIL: A string that we will use to create a service account.
• SERVICE_KEY_PATH: Path to the service account key created in the steps above.

More about this later in Working with configurations section.

2.3. Create your first devstack

Create an instance from a pre-packaged image

$ sultan instance setup --image devstack-juniper

3. Working with configurations

3.1. Local environment set up

To setup the configurations on your local machine, we need to make sure that your local environment is correctly set up.

1. You'll need to create a service account in the GCloud project that will host your devstack instance (official docs here).
2. Grant the service account permission to create an instance (official docs here).
3. Grant the service account roles/compute.instanceAdmin role (See also the list of roles).
4. Download the service account on your machine.

3.2. Getting sultan configurations ready

1. After downloading the service account, edit the value of SERVICE_KEY_PATH to match its location. (Follow the official docs (here) to generate a key for your service account).
2. Edit (uncomment, set) the following values in your configs file

## File configs/.configs.$USER
SSH_KEY="$(HOME)/.ssh/id_rsa"
PROJECT_ID=<gcp-project-id>
SERVICE_ACCOUNT_EMAIL=email@<gcp-project-id>.iam.gserviceaccount.com
SERVICE_KEY_PATH=/path/to/service/key.json

NOTE

• This example has my values
• You should change the value SERVICE_KEY_PATH if you decided to copy and paste the configurations above.

3.3. Optional configurations

Preemptible machine

Preemptible VMs are highly affordable, short-lived compute instances suitable for fault-tolerant workloads. They offer the same machine types and options as regular compute instances and last for up to 24 hours, and can reduce your Compute Engine costs by up to 80%!

Sultan allows you to setup a preemptible machine, you can do that by setting the configuration variable PREEMPTIBLE to true. Just something to note here, preemptible machines are not suitable for long provisioning work, we only recommend using them with sultan instance setup --image command. If you noticed a freeze in your machine's shell, it means that your machine got interrupted, and you might have to restart the session again.

Machine lifespan

When you create a sultan instance, the instance will be configured to run for a specific amount of time configured in ALIVE_TIME in the configurations file. We sat the default lifespan to 6 hours, when your machine powers off you can start it again using

$ sultan instance start

To stop the machine manually before the timeout use

$ sultan instance stop

Exposed ports

For security reasons, Sultan firewall will restrict access to the ports in EXPOSED_PORTS in your .configs file. Here's the full list of the ports you might want to enable.

NOTE

You need to run sultan instance restrict everytime you change the port.

| Service | Port | Role | |----------------------------------|-------|---------| | ssh | 22 | Machine | | lms | 18000 | Default | | studio | 18010 | Default | | forum | 44567 | Default | | amc | 22 | Extra | | discovery | 18381 | Default | | ecommerce | 18130 | Default | | credentials | 18150 | Default | | edx_notes_api | 18120 | Default | | frontend-app-publisher | 18400 | Default | | gradebook | 1994 | Default | | registrar | 18734 | Extra | | program-console | 1976 | Extra | | frontend-app-learning | 2000 | Extra | | frontend-app-library-authoring | 3001 | Extra | | course-authoring | 2001 | Extra | | xqueue | 18040 | Extra |

Accessing sultan from any directory on your machine

If you want to access sultan command line from any directory, add this repo dir to your PATH

$ export PATH=$PATH:$(pwd)  # pwd should translate to /path/to/projects/sultan

Enabling auto completion

To enable auto completion

$ source extras/sultan-completion.bash  # For bash shell
$ source extras/sultan-completion.zsh   # For zsh shell

NOTE

You can add the command above to your ~/.bashrc or ~/.zshrc

Check the best zone for your machine

This is in order to have as small latency to your machine and mounted dirs as possible. You can check it here: http://www.gcping.com/ Following example will be how I’ve set it up (the ZONE value) in Europe.

For minimum latency. You can set ZONE value to match the nearest GCP zone to you.

## File configs/.configs.$USER
ZONE=europe-west3-c

Debugging

If this is your first time creating your instance, we recommend enabling DEBUG mode to be able to to see a verbose output in case something fails:

## File configs/.configs.$USER
DEBUG=false

Devstack run command

We created a custom env variable for the devstack run command. EdX uses dev.up to get the devstack services running. However, in Appsembler we use some other services that requires extra environment variables such as HOST. So simply what we can do here is

## File configs/.configs.$USER
DEVSTACK_RUN_COMMAND="HOST=tahoe dev.up"

NOTE

You need to configure GCloud on your machine before setting the variables above.

4. Creating your first edX devstack instance

4.1. Setting up the devstack

Simple as running

$ IMAGE_NAME=devstack-juniper  # Or any other devstack image name
$ sultan instance setup --image $IMAGE_NAME

The previous command will spin an instance for you from an already created image. However, if you prefer to run the full setup from scratch just don't supply an image argument.

$ sultan instance setup

Make yourself some coffee. Go for a jog. Recreate the Sistine Chapel painting. Actually, go and built the chapel first. Then do the painting. You’ll have time. A full setup takes quite long. But, once it’s finished…

4.2. Bring up devstack up

To run the devstack

$ sultan devstack up

It works. Noice!!

Verify everything is reachable

To verify that your instance has been created:

$ sultan instance status  ## Prints your GCP instance status.
$ sultan instance ip  ## Shows you the instance IP.
$ sultan instance ping  ## Verifies that Ansible is able to reach your instance.

To verify that your devstack is running:

$ curl -I devstack.tahoe:18010  ## Curls your Studio site.

NOTE

LMS and Studio might take up to 3 minutes to fully spin. You can check their logs by running sultan devstack make lms-logs.

Accessing the default created LMS site, Studio, edx admin

Our Devstack automat