tamejs

This package is a source-to-source translator that outputs JavaScript. The input dialect looks a lot like JavaScript, but introduces the await primitive, which allows asynchronous callback style code to work more like straight-line threaded code. tamejs is written in JavaScript.

One of the core powers of the tamejs rewriting idea is that it's fully compatible with existing vanilla-JS code (like node.js's libraries). That is, existing node.js can call code that's been output by the tamejs rewriter, and conversely, code output by the tamejs rewriter can call existing node.js code. Thus, tamejs is incrementally deployable --- you can keep all of your old code and just write the new bits in tamejs! So try it out and let us know what you think.

NEWS

Now available in NEWS.md. Version v0.4 just released, with initial support for what everyone has been asking for --- Tame-aware stack traces! See the section "Debugging and Stack Traces..." below for more details. Also, we've added autocbs, that fire whenever your tamed function returns.

Code Examples

Here is a simple example that prints "hello" 10 times, with 100ms delay slots in between:

for (var i = 0; i < 10; i++) {
    await { setTimeout (defer (), 100); }
    console.log ("hello");
}

There is one new language addition here, the await { ... } block, and also one new primitive function, defer. The two of them work in concert. A function must "wait" at the close of a await block until all deferrals made in that await block are fulfilled. The function defer returns a callback, and a callee in an await block can fulfill a deferral by simply calling the callback it was given. In the code above, there is only one deferral produced in each iteration of the loop, so after it's fulfilled by setTimer in 100ms, control continues past the await block, onto the log line, and back to the next iteration of the loop. The code looks and feels like threaded code, but is still in the asynchronous idiom (if you look at the rewritten code output by the tamejs compiler).

This next example does the same, while showcasing power of the await{..} language addition. In the example below, the two timers are fired in parallel, and only when both have fulfilled their deferrals (after 100ms), does progress continue...

for (var i = 0; i < 10; i++) {
    await { 
        setTimeout (defer (), 100); 
        setTimeout (defer (), 10); 
    }
    console.log ("hello");
}

Now for something more useful. Here is a parallel DNS resolver that will exit as soon as the last of your resolutions completes:

var dns = require("dns");

function do_one (cb, host) {
    var err, ip;
    await { dns.resolve (host, "A", defer (err, ip));}
    if (err) { console.log ("ERROR! " + err); } 
    else { console.log (host + " -> " + ip); }
    cb();
}

function do_all (lst) {
    await {
        for (var i = 0; i < lst.length; i++) {
            do_one (defer (), lst[i]);
        }
    }
}

do_all (process.argv.slice (2));

You can run this on the command line like so:

node src/13out.js yahoo.com google.com nytimes.com okcupid.com tinyurl.com

And you will get a response:

yahoo.com -> 72.30.2.43,98.137.149.56,209.191.122.70,67.195.160.76,69.147.125.65
google.com -> 74.125.93.105,74.125.93.99,74.125.93.104,74.125.93.147,74.125.93.106,74.125.93.103
nytimes.com -> 199.239.136.200
okcupid.com -> 66.59.66.6
tinyurl.com -> 195.66.135.140,195.66.135.139

If you want to run these DNS resolutions in serial (rather than parallel), then the change from above is trivial: just switch the order of the await and for statements above:

function do_all (lst) {
    for (var i = 0; i < lst.length; i++) {
        await {
            do_one (defer (), lst[i]);
        }
    }
}

Slightly More Advanced Example

We've shown parallel and serial work flows, what about something in between? For instance, we might want to make progress in parallel on our DNS lookups, but not smash the server all at once. A compromise is windowing, which can be achieved in tamejs conveniently in a number of different ways. The 2007 academic paper on tame suggests a technique called a rendezvous. A rendezvous is implemented in tamejs as a pure JS construct (no rewriting involved), which allows a program to continue as soon as the first deferral is fulfilled (rather than the last):

function do_all (lst, windowsz) {
    var rv = new tame.Rendezvous ();
    var nsent = 0;
    var nrecv = 0;

    while (nrecv < lst.length) {
        if (nsent - nrecv < windowsz && nsent < n) {
            do_one (rv.id (nsent).defer (), lst[nsent]);
            nsent++;
        } else {
            var evid;
            await { rv.wait (defer (evid)); }
            console.log ("got back lookup nsent=" + evid);
            nrecv++;
        }
    }
}

This code maintains two counters: the number of requests sent, and the number received. It keeps looping until the last lookup is received. Inside the loop, if there is room in the window and there are more to send, then send; otherwise, wait and harvest. Rendezvous.defer makes a deferral much like the defer primitive, but it can be labeled with an idenfitier. This way, the waiter can know which deferral has fulfileld. In this case we use the variable nsent as the defer ID --- it's the ID of this deferral in launch order. When we harvest the deferral, rv.wait fires its callback with the ID of the deferral that's harvested.

Note that with windowing, the arrival order might not be the same as the issue order. In this example, a slower DNS lookup might arrive after faster ones, even if issued before them.

Composing Serial And Parallel Patterns

In Tame, arbitrary composition of serial and parallel control flows is possible with just normal functional decomposition. Therefore, we don't allow direct await nesting. With inline anonymous JavaScript functions, you can consicely achieve interesting patterns. The code below launches 10 parallel computations, each of which must complete two serial actions before finishing:

function f(cb) {
    await {
        for (var i = 0; i < n; i++) {
            (function (cb) {
                await { setTimeout (defer (), 5*Math.random ()); }
                await { setTimeout (defer (), 4*Math.random ()); }
                cb();
             })(defer ());
        }
    }
    cb();
}

autocb

Most of the times, a tamed function will call its callback and return at the same time. To get this behavior "for free", you can simply name this callback autocb and it will fire whenver your tamed function returns. For instance, the above example could be equivalently written as:

function f(autocb) {
    await {
        for (var i = 0; i < n; i++) {
            (function (autocb) {
                await { setTimeout (defer (), 5*Math.random ()); }
                await { setTimeout (defer (), 4*Math.random ()); }
             })(defer ());
        }
    }
}

In the first example, recall, you call cb() explicitly. In this example, because the callback is named autocb, it's fired automatically when the tamed function returns.

If your callback needs to fulfill with a value, then you can pass that value via return. Consider the following function, that waits for a random number of seconds between 0 and 4. After waiting, it then fulfills its callback cb with the amount of time it waited:

function rand_wait(cb) {
    var time = Math.floor (Math.random()*5);
    if (time == 0) {
         cb(0); return;
    }
    await setTimeout (defer (), time);
    cb(time); // return here, implicitly.....
}

This function can written equivalently with autocb as:

function rand_wait(autocb) {
    var time = Math.floor (Math.random()*5);
    if (time == 0) {
        return 0;
    }
    await setTimeout (defer (), time);
    return time;
}

Implicitly, return 0; is mapped by the tamejs compiler to autocb(0); return.

Installing and Using

Install via npm:

npm install -g tamejs

You can their either use the tamejs compiler on the command line:

tamejs -o <outfile> <infile>
node <outfile> # or whatever you want

Or as an extension to node's module import system:

require ('tamejs').register (); // register the *.tjs suffix
require ("mylib.tjs");          // then use node.js's import as normal

If you want a different extension, this will work:

require ('tamejs').register ('tamejs'); // register the *.tamejs suffix
require ("mylib.tamejs");               // then use node.js's import as normal

Or, finally, you can call register to do a few things at once, including multiple suffix registrations:

// Will register suffixes 'tamejs' and 'yojs'; will
// also enable tame stack tracing, and disable caching of
// .tjs files included at runtime
require ('tamejs').register ({ extension       : [ 'tamejs', 'yojs'], 
                               catchExceptions : true,
			       disableCache    : true })
require ("mylib.tamejs");
require ("yourlib.yojs");

API and Documentation

defer

defer can be called in one of two ways.

Inline Variable Declaration

The first allows for inline declaration of the callback slot variables:


await { dns.resolve ("okcupid.com", defer (var err, ip)); }

In the tamed output code, the variables err and ip will be declared right before the start of the await block that contains them.

Tamejs

Install / Use

README