A Saga execution coordinator implementing a micro-process manager.

Build

This project currently depends on org.gautelis:vopn as a sibling project during local development. If vopn is not already installed in your local Maven repository, build and install it first and then build this project:

mvn -f ../java-vopn/pom.xml install
mvn test

Historically both libraries were published to Maven Central. The published coordinates are still the same, but the examples below describe the current 2.0-SNAPSHOT development branch rather than the older Central release.

Tests

The default Maven test run is intended to be a deterministic regression suite:

mvn test

The old long-running demo/stress harness is still available, but it is no longer part of the default test phase. Run it explicitly when you want to exercise the library interactively:

mvn -Dtest=test.AppDemo test

The asynchronous recovery manager now uses the local ExecutorWorkQueue implementation by default. Its shutdown semantics differ from the older vopn multi-worker queue: queued recovery work is dropped on stop(), and then rebuilt from persisted database state on the next start()/recover() cycle. This matches the intended process-manager lifecycle, where start() and stop() follow application startup and shutdown.

Release Notes

• 2.0-SNAPSHOT: the asynchronous recovery manager now defaults to ExecutorWorkQueue. Shutdown no longer preserves queued in-memory recovery backlog across stop(). Pending recovery, abandonment, and retirement work is instead reconstructed from persisted process state after the next startup and recovery cycle.

Description of content

This library implements a Saga execution coordinator (SEC), suitable as a micro-process manager. Out of the box, a local embedded Derby database is used to persist processes and process activities; if the database does not exist, the database and tables are created automatically — trivializing development use.

In a non-development scenario, process data may be stored in the application database or in a separate database. DDL and SQL statements for some database managers are available under contrib/.

A key concern has been to provide a relatively simple abstraction over the Saga pattern. This library hides details from the utilizing application.

A key insight from developing this library is that the hard problem is reasoning around the fail states. The Saga pattern only offer so much and in the end you need to understand how the fail states affect your business application. Of course you need to understand that in a globally (distributed) transaction environment as well, but that environment makes it easier to reason about.

Published releases have previously been available from Maven Central as:

<dependency>
    <groupId>org.gautelis</groupId>
    <artifactId>muprocessmanager</artifactId>
    <version>1.10</version>
</dependency>

Background

Hector Garcia-Molina and Kenneth Salem presented an article in 1987 describing a means to tackle long lived transactions, a situation where it is not feasible to model the transaction using the mechanisms provided by a backing database or the typical transaction manager.

Lately this pattern, aptly dubbed Saga, has become popular when implementing micro-services. Let us denote such micro-services that implement a process consisting of a series of individual activities, thus needing transactions in one form or another, a micro-process.

To be clear, the situation we are considering here, is a series of activities in a process flow where the process as a whole should either succeed or fail in the ACID sense. Additionally, we will consider invoking distributed services, in an environment where we do not want to use a transaction manager and where a local database transaction cannot guarantee ACID characteristics for the process.

Having a transaction manager, the individual activities could participate in a global (distributed) transaction, having local transactions that individually participates in, say, the two-phase-commit (2PC) consensus-protocol on top of the individual transactions.

Without a transaction manager and without participating in a global transaction, is it even possible to guarantee ACID characteristics? The answer is, in short, no. On the other hand, if it is possible to waive some of the constraints of the ACID characteristics, it could be possible.

If we could, for instance, trade consistency with eventual consistency or allow intermediate changes to the database to be visible to the outside, the Saga article lays out a pattern for implementing micro-services that does not demand global transactions. We are making a trade here in order to drop some demands on a runtime environment.

What the Saga pattern effectively does, is making compensational behaviour explicit in the software model, by pairing any local transaction with the corresponding compensation and stipulating how they should behave. Compensational code is not new, but the Saga pattern makes it easier to reason around the failure states of the micro-process.

A practical Saga

This software offering implements and hides the details of executing forward actions (along the "happy path") and reverting to backward actions in the form of compensations if the forward actions fail.

The individual compensations are pushed on a persistent "stack" (logged to persistent store) ahead of attempting to execute the local transaction. If any such local transaction fails, the "forward motion" in the process stops and the μprocessmanager reverts to pop'ing compensations from the stack — executing them in a "backwards motion".

The compensation is normally done synchronously in the process and an exception is thrown that both describes the problem as well as interrupts the micro-process. The machinery around the μprocessmanager is kind of boring (as it should be :) while the emphasis has to be on the internal failure states.

If the process fails, i.e. the process thread dies before the compensations are executed completely, the μprocessmanager runs recover activities in the background picking up aborted processes from the persistent store and executing the compensations. Eventually individual processes may remain in a partly recovered state, which will need some kind of external action.

Processes are identified through a correlation ID.

Index

Recover functionality are found in MuProcessManager::recover.

State transitions related to forward activities are found in MuProcess::execute and backwards activities in MuProcess::compensate.

All reading and writing from database (the persistent compensation log) is implemented in MuPersistentLog.

Example activities are found in the test-package.

Example

How to prepare the μprocessmanager:

// Prepare process manager
MuProcessManager mngr;
try {
    mngr = MuProcessManagerFactory.getManager(dataSource);
    mngr.start();
}
catch (Exception e) {
    String info = "Failed to prepare process manager: " + e.getMessage();
    throw new Exception(info, e);
}

How to instantiate and run a process:

// A correlation ID identifying this process
String correlationId = UUID.randomUUID().toString();

// This implements a micro-process, consisting of a series
// of individual activities.
MuProcess process = null;
try {
    MuNativeActivityParameters parameters = new MuNativeActivityParameters();

    process = mngr.newProcess(correlationId);
    
    parameters.put("arg1", "param1");
    process.execute(new FirstActivity(), parameters);
    
    parameters.put("arg2", 42);
    process.execute(new SecondActivity(), parameters);
    
    parameters.put("arg3", true);
    process.execute(new ThirdActivity(), parameters);
    
    parameters.put("arg4", 22 / 7.0);
    process.execute(new FourthActivity(), parameters);
    
    process.finished();   
    
} catch (MuProcessForwardBehaviourException mpfae) {
    // Forward activity failed, but compensations were successful
    String info = "No success, but managed to compensate: " + mpfae.getMessage();
    log.info(info, mpfae);
    
} catch (MuProcessBackwardBehaviourException mpbae) {
    // Forward activity failed and so did some compensation activities
    String info = "Process and compensation failure: " + mpbae.getMessage();
    log.warn(info, mpbae);
    
}
catch (Throwable t) {
    // Other reasons for failure not necessarily related to the activity
    if (null != process) {
        process.failed();
    }

    String info = "Process failure: " + t.getMessage();
    log.warn(info);
}

An individual activity from the example above (test.FirstActivity), in this case a simple dummy load with probabilistic failure behaviour:

package test;

import org.gautelis.muprocessmanager.MuActivity;
import org.gautelis.muprocessmanager.MuForwardActivityContext;
import org.gautelis.muprocessmanager.MuBackwardActivityContext;

public class FirstActivity implements MuActi