Emissary Dark Knight - some code just wants to watch the core burn

Introduction
Minimum Requirements
Getting Started
Contact Us

Introduction

Emissary is a P2P based data-driven workflow engine that runs in a heterogeneous possibly widely dispersed, multi-tiered P2P network of compute resources. Workflow itineraries are not pre-planned as in conventional workflow engines, but are discovered as more information is discovered about the data. There is typically no user interaction in an Emissary workflow, rather the data is processed in a goal oriented fashion until it reaches a completion state.

Emissary is highly configurable, but in this base implementation does almost nothing. Users of this framework are expected to provide classes that extend emissary.place.ServiceProviderPlace to perform work on emissary.core.IBaseDataObject payloads.

A variety of things can be done and the workflow is managed in stages, e.g. STUDY, ID, COORDINATE, TRANSFORM, ANALYZE, IO, REVIEW.

The classes responsible for directing the workflow are the emissary.core.MobileAgent and classes derived from it, which manage the path of a set of related payload objects through the workflow and the emissary.directory.DirectoryPlace which manages the available services, their cost and quality and keep the P2P network connected.

Maven Site and Javadoc hosted on GitHub Pages: https://code.nsa.gov/emissary/

Minimum Requirements

Linux or MacOSX operating system
JDK 11
Apache Maven 3.6.3+

Getting Started

Read through the DEVELOPING.md guide for information on installing required components, pulling the source code, building and running Emissary.

Building

Run mvn clean package to compile, test, and package Emissary

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  9.132 s
[INFO] Finished at: 2022-01-10T22:31:05Z
[INFO] ------------------------------------------------------------------------

Running

There is one bash script in Emissary that runs everything. It is in the top level Emissary directory. The script runs the emissary.Emissary class which has several Picocli commands available to handle different functions.

No arguments

If the emissary script is run without any arguments, you will get a listing of all the configuration subcommands and a brief description.

./emissary

Help

Running ./emissary help will give you the same output as running with no arguments. If you want to see more detailed information on a command, add the command name after help. For example, see all the arguments with descriptions for the server command, run:

./emissary help server

Common parameters

The rest of commands all have (-b or --projectBase) arguments that can be set, but it must match PROJECT_BASE.

The config directory is defaulted to <projectBase>/config but can also be passed in with (-c or --config). When running from the git checkout, you should use target as the projectBase. Feel free to modify config files in target/config before you start.

Logging is handled by logback. You can point to a custom file with the --logbackConfig argument.

See the help -c <commandName> for each command to get more info.

Server (Standalone)

This command will start up an Emissary server and initialize all the places, a pickup place, and drop off filters that are configured. It will start in standalone mode if -m or --mode is not specified. By default, the number of MobileAgents is calculated based on the specs of the machine. On modern computers, this can be high. You can control the number of agents with -a or --agents. Here is an example run.

./emissary server -a 2

Without further configuration, it will start on http://localhost:8001. If you browse to that url, you will need to enter the username and password defined in target/config/jetty-users.properties.

[!CAUTION] Security Notice: Default Credentials

Change Default Credentials Immediately The usernames and passwords listed below are for demonstration purposes only. Using default credentials in a production environment poses a significant security risk.

Action Required: Update your jetty-users.properties file with strong, unique passwords before deploying.

| Default Username | Default Password | |------------------|------------------| | emissary | emissary123 | | console | console123 |

The default PickUpPlace is configured to read files from target/data/InputData. If you copy files into that directory, you will see Emissary process them. Keep in mind, only toUpper and toLower are configured, so the output will not be too interesting.

Pause

Stop the service from taking work

./emissary server --pause

Unpause

Allow a paused service to take work

./emissary server --unpause

Invalidate

Invalidate services that are refreshable. This is a "light" refresh that is a no-downtime approach that invalidates a ServiceProviderRefreshablePlace. When the place is then pulled from DirectoryPlace, the place is recreated using the same keys for DirectoryPlace and Namespace, but the configurator is reloaded and the place is able to reload a subset of its configs.

./emissary server --invalidate

Refresh

Force refresh of services. This is a "hard" refresh, the server is paused and there is a wait for the MoblieAgents to drain. Once the server is fully idle, all ServiceProviderRefreshablePlace existing keys are removed from DirectoyPlace and Namespace, and the places are fully recreated. This allows for changes to service names, proxies, deny lists, etc. The server is then unpaused to resume processing. Any failure in refresh would put the server in a bad state, so the server is shutdown.

./emissary server --refresh

Stop

Shutdown the service

./emissary server --stop

Kill

Force the shutdown of the service

./emissary server --kill

Agents (Standalone)

The agents command shows the number of MobileAgents for the configured host and what those agents are doing. By default, the port is 9001, but you can use -p or --port to change that.
Assuming you are running on 8001 from the server command above, try:

./emissary agents -p 8001

Pool (Standalone)

Pool is a collapsed view of agents for a node. It, too, defaults to port 9001. To run for the standalone server started above run

./emissary pool -p 8001

This command is more useful for a cluster as it a more digestible view of every node.

Env

The Env Command requires a server to be running. It will ask the server for some configuration values, like PROJECT_BASE and BIN_DIR. With no arguments, it will dump an unformatted json response.

./emissary env

But you can also dump a response suitable for sourcing in bash.

./emissary env --bashable

Starting the Emissary server actually calls this endpoint and dumps out $PROJECT_BASE}/env.sh with the configured variables. This is done so that shell scripts can source $PROJECT_BASE}/env.sh and then have those variable available without having to worry with configuring them elsewhere.

Config

The config command allows you to see the effective configuration for a specified place/service/class. Since Emissary uses flavors, this command will show the resulting configuration of a class after all flavors have been applied. This command can be used to connect to a running Emissary node by specifying the -h for host (default is localhost) and -p for the port (default is 8001). To connect to a locally running Emissary on port 8001, any of the following commands will work:

./emissary config --place emissary.place.sample.ToLowerPlace
./emissary config --place emissary.place.sample.ToLowerPlace -h localhost -p 8001

Optionally, you can specify offline mode using --offline to use the configuration files specified in your local CONFIG_DIR:

./emissary config --place emissary.place.sample.ToLowerPlace --offline

In offline mode, you can provide flavors to see the differences in configurations:

./emissary config --place emissa

Emissary

Install / Use

README

Table of Contents