autoflow.js

autoflow (formerly named react) is a javascript module to make it easier to work with asynchronous code, by reducing boilerplate code and improving error and exception handling while allowing variable and task dependencies when defining flow. This project is applying the concepts of Reactive programming or Dataflow to controlling application flow.

This async flow control module is initially designed to work with Node.js but is planned to be extended to browser and other environments.

Autoflow (formerly named react) got its original name from similarities with how "chain reactions" work in the physical world. You start the reaction and then it cascades and continues until complete.

Also "Reactive Programming" or "Dataflow" describe defining flow which reacts to the data similar to how a spreadsheet updates cells. These are good examples of how autoflow controls flow based on when data is available

• Reactive programming - http://en.wikipedia.org/wiki/Reactive_programming
• Dataflow programming - http://en.wikipedia.org/wiki/Dataflow
• Dataflow Programming: Handling Huge Data Loads Without Adding Complexity (Dr. Dobb's Sept 19, 2011) - http://drdobbs.com/database/231400148

It takes inspiration from several projects including:

• Tim Caswell and Elijah Insua's conductor - Article
• Caolan McMahon's async

Project status - INACTIVE

I am not currently working on autoflow or autoflow family projects. I am happy to accept contributors if further development is needed.

Goals

• Minimize boilerplate code needed for working with asynchronous functions
• Minimize the need to customize your code simply to use async flow control. The use of a flow control module ideally should not affect the way you write your code, it should only help take over some of the burden.
• Improved error and exception handling
• Provide useful stack traces and context information for easier debugging
• Make code more readable and easier to understand which should translate to less defects
• Provide the right level of abstraction to make it easier to refactor code, without being too magical
• Allow the mixing of pure functions, method calls, and callback style functions in the flow

Supports

• Async node-style callback(err, arg...) functions
• Sync functions which directly return value
• Object instance method calls
• Class method calls
• selectFirst flow where the first task that returns defined, non-null value is used
• Promise style functions - also automatic resolution of promise inputs (optionally loaded with autoflow.resolvePromises();)
• Use of resulting flow function as callback style or promise style (if no callback provided). This is provided via plugin corresponding to the promise library used. See https://github.com/jeffbski/autoflow-deferred and https://github.com/jeffbski/autoflow-q
• Supports ES5 browsers (can work with others by using polyfills)
• (Planned) iteration on arrays, streams, sockets
• (Planned) event emitter integration
• Tested on node 0.8, 0.10, 0.11

The tasks can be mixed, meaning you can use async, sync, object method calls, class method calls, etc in the same flow.

Concept

Borrowing heavily from Tim and Elijah's ideas for conductor, this async flow control module provides a way to construct a flow from a collection of rules based on functions or methods (referred to as tasks in this module). It allows dependencies to be defined between the tasks so they can run in parallel as their dependencies are satisfied. autoflow can use both variable dependencies and task dependencies.

As tasks complete, autoflow watches the dependencies and kicks off additional tasks that have all their dependencies met and are ready to execute. This allows the flow to run at maximum speed without needing to arbitrarily block tasks into groups of parallel and serial flow.

To reduce the boilerplate code needed and improve error handling, autoflow automatically provides callback functions for your asynchronous code. These autoflow-provided callback functions perform these steps:

1. Check for error and handle it by calling outer callback function with this error after augmenting it with additional context information for easier debugging
2. Save the callback variables into a context for future reference
3. Call back into autoflow (and it will kick off additional tasks that are now ready to go)
4. Using the dependencies specified for each

Design

• Parse and validate DSL rules at module load time creating AST
• Validate the flow AST at module load time - determine if dependencies can all be met as defined
• Execute the flow AST by calling the function with arguments

Installing

npm install autoflow

OR

Pull from github - http://github.com/jeffbski/autoflow

Examples

<a name="defaultDSL"/> ### Example using default DSL interface

• Simple example showing flow definition of two async functions feeding a synchronous function.

• First two async functions inputs are satisfied by the flow inputs, so they will both run immediately in parallel.

• The last function waits for the outputs of the previous ones, then executes synchronously.

• Finally the flow calls the callback with the output values once all the tasks have completed.

// in your foobar module
var autoflow = require('autoflow');

// some normal async and sync functions
function loadFoo(fooPath, cb) {
  setTimeout(function () {
    cb(null, [fooPath, 'data'].join(':'));
  }, 10);
}

function loadBar(barPath, barP2, cb) {
  setTimeout(function () {
    cb(null, [barPath, barP2, 'data'].join(':'));
  }, 10);
}

function render(foo, bar) {
  return ['<html>', foo, '/', bar, '</html>'].join('');
}

// define fn, glue together with autoflow, it will parallelize
// starts with name and in/out params, then the tasks
var loadRender = autoflow('loadRender', 'fooPath, barPath, barP2, cb -> err, renderedOut',
  loadFoo, 'fooPath, cb -> err, foo',    // async cb function
  loadBar, 'barPath, barP2, cb -> err, bar',  // async cb function
  render, 'foo, bar -> renderedOut'  // sync function using outputs from first two
);

exports.loadRender = loadRender;  // is a normal fn created by autoflow

// in a different module far far away, use this as any other node function
var foobar = require('foobar');
foobar.loadRender('foo.txt', 'bar.txt', 'BBB', function (err, renderedOut) {
  // tasks in loadRender were parallelized based on their input dependencies
  console.error('results:', renderedOut);
});

Below is a graph of how the dependencies are mapped by autoflow which Also indicates how the tasks will be executed. This was generated by the autoflow plugin autoflow-graphviz which you can use to also graph your flows.

User API

The main function returned from require('autoflow') can be used to define the AST used for the processing of the rules or flow.

It takes the following arguments to define a flow function:

var fn = autoflow('loadRender', 'fooPath, barPath, barP2, cb -> err, renderedOut',
  loadFoo, 'fooPath, cb -> err, foo',
  loadBar, 'barPath, barP2, cb -> err, bar',
  render, 'foo, bar -> renderedOut'
);

Note: graphic uses the old name react, new name is autoflow

1. flow/function name - string - represents the name of the flow or function that will be created. autoflow will use the name when generating events so you can monitor progress and performance and also when errors occur.
2. In/out flow parameter definition - string - the inputs and outputs for the flow function. The parameters are specified in one single string for easy typing, separated by commas. The output follows the input after being separated by a ->. Use the parameter name cb or callback to specify the Node style callback and err to represent the error parameter as the first output parameter of the callback. Literal values can also be specified directly (true, false, numbers, this, null). Literal strings can simply be quoted using single or double quotes.
3. optional flow options - object - if an object is provided immediately after the in/out flow def, then these options will be provided to autoflow to customize the flow. The locals property can contain an object map of any local variables you want to reference in the flow (other than what is passed in as parameters). For example: { locals: { foo: foo, bar: bar }} would make local vars available in the flow. Note that global variables are already available in the flow.
4. function reference or method string - specify the function to be called for this task, or if calling a method off of an object being passed in or returned by a task, use a string to specify like 'obj.method'. These can be asynchronous Node-style callback cb(err, ...) functions or synchronous functions which simply return values directly.
5. in/out task parameter definition - string - similar to the in/out flow parameter definition above, these are the inputs and outputs that are passed to a task function and returned from a task function. The inputs will need to match either those from the flow inputs or outputs from other tasks that will run before this task. autoflow will use the inputs as dependencies, so it will invoke and wait for response from the tasks that provide the dependent inputs. So simply by specifying inputs and outputs for the tasks, autoflow will prioritize and parallelize tasks to run as fast as possible. Use cb or callback along with err to speci