Aerostat Beam Engine

<img align="right" src="images/beamengine_third.jpg" width="50%"/>Redis-backed highly-scale-able and cloud-fit distributed media processing engine. A Node.js web application and library, Aerostat Beam Engine provides the following:

• A resilient, media-aware cache of data structured ready for processing by FFmpeg libraries, which can be backed by file or object stores;
• Connections to stateless clients that provide access to the media-aware cache through the Content Beam API, an HTTP/S API for transporting media data, pushed and pulled, compressed or uncompressed, and streamed as if live;
• Job queues allowing multiple worker clients to carry out media transformations just-in-time or just-in-case, executing on the local system, distributed across many systems, GPU accelerated or via serverless compute services like AWS Lambda.

The engine is a web server application that provides access to read and write data stored in the underlying redis cache, which may be a single redis instance or a cluster of master/slave redis instances. Also included a library to build customised workers, triggered by rules, maintaining and building content relationships. A small library of example [pre-built workers](#pre-built workers) is provided.

Beam engine is an implementation of the core of the Agile Media Blueprint.

Work in progress. This README describes the to be state of Beam Engine and the current implementation falls short of what is described. See the TODO notes for details of implementation still to be done. For Node.js FFmpeg native bindings, please see Aerostat Beam Coder.

Installation

Before installation can take place, it is necessary to identify some connected networked resources, to include:

• system(s)/container(s)/VM(s) to run the Node.js Beam Engine web apps, possibly behind a load balancer such as NGINX or AWS Elastic Load Balancing.
• system(s)/container(s)/VM(s) to run redis, either a single instance, replicated or as a cluster. Alternatively, use a redis-backed cache service such as AWS Elasticache or Azure Cache for Redis.
• system(s)/container(s)/VM(s) to run workers written in Node.js, some of which may need access to a shared file system to persist cached data.
• access to serverless processing capability such as AWS Lambda or Azure Functions (optional).
• systems with GPUs for workers that can do accelerated GPU processing (optional).

It is recommended that for the distributed processing of uncompressed HD video data, all network interconnects run at speeds at or over 10Gbs.

For development purposes, it is possible to run redis, beam engine and workers all on the same system.

Node.js prerequisites

On x86_64 Windows, Mac or Linux platforms, Beam Engine uses the latest Long Term Support version of Node.js which is currently the most recent v10. Download or install Node.js from here or use a system package manager.

The Aerostat Beam Engine web app and workers depend on Aerostat Beam Coder, a module that as a native addon requires compilation as part of its installation process. The Node.js native addon build tool is called node-gyp. Follow the node-gyp installation instructions to ensure that each Node.js system is ready to build native extensions.

Note: For MacOSX Mojave, install the following package after xcode-select --install:

/Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg

The Beam Coder installer downloads suitable FFmpeg .dlls on Windows and uses homebrew to install FFmpeg .dylib libraries on Mac. For Linux, install FFmpeg development packages suitable for your Linux version, ensuring that the packages include the shared library .so files.

Redis

For local working, install redis on a system or systems with a substantial amount of memory. Follow the instructions from redis themselves. For testing and low criticality systems, a single Redis instance or replicating instance may be sufficient.

For mission critical applications, consider establishing a cluster. Beam engine uses ioredis, a module with support for single connections and connections to members of a redis cluster.

For cloud environments, consider using Redis backed cache service, creating a connection to a virtual private network where the beam engines and workers are located. Such services include:

• AWS ElastiCache for Redis;
• Microsoft's Azure Cache for Redis
• Google Cloud Platform Cloud Memorystore

Many pre-built redis packages exist for Linux distributions, such as the redis-server package for Ubuntu. Redis is also available as a container, such as the redis docker container. For development and testing on Windows, consider installing redis using the Windows Subsystem for Linux. Note that for some operations, this approach is one or two orders of magnitude slower than installing Redis on a Linux build directly and, as such, should only be used for feature testing and not performance testing.

For large media operations where redis is being used only as a temporary cache, performance will be improved by disabling redis persistence features. This can be achieved in one of the following two ways:

1. Using the redis-cli:

    config set appendonly no
    config set save ""
    config rewrite

2. Edit the redis.conf file, setting:

    appendonly no
    save ""

Development

To run a Beam Engine in development mode, clone the module from github. In the modules root folder, edit the config.json file to match local settings. Then run npm install followed by npm run dev. This will install dependencies and start the development web server that runs using the automated restart module nodemon. To restart, type rs.

To see debug information from Redis connections (ioredis), the web framework (Koa 2) and from the job queues (Bull), set the debug flag as follows.

From bash:

$ DEBUG="*,-not_this"; NODE_DEBUG='bull'; npm run dev

From Windows Powershell:

PS > $env:DEBUG="*,-not_this"; $env:NODE_DEBUG="bull"; npm run dev

To create your own workers, install beam engine in your project.

npm install beamengine

Production

From a cloned copy of the repository, run (my_config.json is optional):

npm start my_config.json

Alternatively, install beam engine as a global application (prepend sudo for Unix platforms):

npm install -g beamengine

With the NODE_ENV variable set to production, run beam engine as a command (my_config.json is optional):

beamengine my_config.json

Configuration

Configuration of an Aerostat Beam Engine is achieved based on Javascript files in the config folder, with one for development (default), one for test and one for production. Which base configuration is selected depends on the current value of the NODE_ENV environment variable. Each configuration file has the following properties:

| property name | type | description | | ------------------------- | ------- | ---------------------------------------------- | | redis.cluster | Boolean | True if connecting to a redis cluster | | redis.host | string | IP address of primary redis host | | redis.port | int | Port number of primary redis host connection | | redis.password | string | Password to authenticate with redis (optional) | | redis.db | int | Redis database to be used for main cache | | redis.pool | int | Maximum pool size for Redis connection pool | | redis.prepend | string | Value to prepend to beamengine database keys | | redis.packetTTL | int | Time (ms) before expiry of packet data | | redis.frameTTL | int | Time (ms) before expiry of frame data | | redis.ephemeralTTL | int | Time (ms) before expiry of ephemeral data blob | | redis.closeTimeout | int | Time (ms) before error when closing Redis pool | | app.port | int | Local port on which to run the app server | | jobs... | Object | See workers section |

The properties can be overridden with these environment variables, depending on NODE_ENV:

| property name | development | test | production | | ---------------------- | ---------------------------- | ----------------------------- | ----------------------------- | | redis.cluster | BEAM_DEV_REDIS_CLUSTER | BEAM_TEST_REDIS_CLUSTER | BEAM_PROD_REDIS_CLUSTER