mage 🧙‍♂️

A simple cross-platform[^1] interprocess communication (IPC) library written in C++. Mage is heavily inspired by Chromium's [Mojo IPC library], and written by [Dominic Farolino] (a Chromium engineer) in his free time.

Chromium's Mojo is feature-rich and battle-tested, but it has a lot of Chromium-specific dependencies that prevent it from being portable and used by other applications. The motivation for Mage was to create a watered-down version of Mojo, with no dependencies on the Chromium tree, for learning purposes and for use in arbitrary C++ applications.

Right now, Mage's only dependency is [//base], a simple threading & scheduling library developed alongside Mage, although design work for separating the two entirely is being considered, to make Mage even more standalone. Mage is also built with [Bazel] but can be integrated with other toolchains.

<details><summary>History!</summary>

The user-facing IDL portion of Mojo was based on Darin Fisher's [ipc_idl] prototype, which describes a very similar IDL that python generates C++ bindings from, with the Jinja2 templating engine.

In the mid-2010s, development on the "guts" of Mojo began, with a simple messaging library called [ports]. If you're a Google employee, you can read [Mojo Core Ports Overview]. Ports now lives inside Mojo in the Chromium tree: https://source.chromium.org/chromium/chromium/src/+/main:mojo/core/ports/.

The Mojo language-specific bindings (like mojo::Remote, etc.) are built on top of mojo/core which is in turn built on top of ports.

Mojo is built in an impressively layered fashion, allowing for its internals (much of the stuff underneath the bindings layer) to be swapped out for an entirely different backing implementation with no (Chromium) developer consequences. Ken Rockot built the [ipcz] library to experiment replacing much of the Mojo internals with his IPC system that implements message passing with shared memory pools instead of explicit message passing with sockets and file descriptors, etc.

The ultimate goal with Mage is to make it, too, as modular as possible, allowing it to be used with any external threading & scheduling library, not just [//base]. It would be cool someday if the internals were also decoupled from the public bindings enough such that we could experiment with other internal messaging providers such as [ipcz], for example.

</details>

---

Table of contents

• Supported platforms
• Overview
  • mage::Remote<T>
  • mage::Receiver<T>
• Magen Interface Definition Language (IDL)
• Using Mage in your application
  • 1. Write your interface in a .magen file
  • 2. Build your .magen file
  • 3. Implement your interface in C++
  • 4. Use a Remote to send cross-process IPCs
  • Sending MessagePipes cross-process
• Mage invitations
• Threading & task scheduling
• Building and running the tests
  • Debugging
• Design limitations
• Security considerations

---

Supported platforms

In progress:

Overview

Mage IPC allows you to seamlessly send asynchronous messages to an object that lives in another process, thread, or even the same thread, without the sender having to know anything about where the target object actually is. Messages are described by user-provided interface files.

To get started, you need to be familiar with three concepts from the [public API]:

• mage::MessagePipe
• mage::Remote<T>
• mage::Receiver<T>

Messages are sent over bidirectional message pipes, each end of which is represented by a mage::MessagePipe, which can be passed over interfaces, even to other processes. Given a pair of entangled MessagePipes, you'll ultimately bind one end to a Remote and the other to a Receiver. It is through these objects that arbitrary user messages get passed as IPCs.

mage::Remote<T>

Once bound, a Remote<magen::Foo> represents a local "proxy" for a magen::Foo interface, whose concrete implementation may live in another process. You can synchronously invoke any of the magen::Foo interface methods on a Remote<magen::Foo>, and the proxy will forward the message to the right place, wherever the corresponding Receiver<magen::Foo> lives, even if it's moving around.

mage::MessagePipe remote_pipe = /* get pipe from somewhere */;
mage::Remote<magen::Foo> remote(remote_pipe);

// Start sending IPCs!
remote->ArbitraryMessage("some payload");

mage::Receiver<T>

Messages sent over a bound Remote<magen::Foo> get queued on the other end's MessagePipe until it is bound to a corresponding Receiver<magen::Foo>, which represents the concrete implementation of the Mage interface magen::Foo. The receiver itself does not handle messages sent by the remote, but rather it has a reference to a user-provided C++ object that implements the interface, and it forwards messages to it. Receivers are typically owned by the concrete implementation of the relevant interface.

Here's an example:

// Instances of this class can receive asynchronous IPCs from other processes.
class FooImpl final : public magen::Foo {
 public:
  Bind(mage::MessagePipe foo_receiver) {
    // Tell `receiver_` that `this` is the concrete implementation of
    // `magen::Foo` and its interface methods.
    receiver_.Bind(foo_receiver, this);
  }

  // Implementation of `magen::Foo`. These methods get invoked by `receiver_`
  // when IPCs come in from the remote.
  void ArbitraryMessage(string) { /* ... */ }
  void AnotherIPC(MessagePipe) { /* ... */ }

 private:
  // The corresponding remote may live in another process.
  mage::Receiver<magen::Foo> receiver_;
};

Magen Interface Definition Language (IDL)

Magen is the [IDL] that describes Mage interfaces. Interfaces are written in .magen files by consumers of Mage, and are understood by the magen_idl(...) Bazel rule which generates C++ bindings for the interfaces.

The Magen IDL is quite simple (and much less feature-rich than Mojo's IDL). Each .magen file describes a single interface with the interface keyword, which can have any number of methods described by their names and parameters.

Single line C-style comments are supported. Here are a list of supported parameter types:

• bool
• int
• long
• double
• char
• string
• MessagePipe

The types are self-explanatory, with the exception of MessagePipe. A MessagePipe that is not bound to a Remote or Receiver can be passed from one process, over an existing IPC interface, to be bound to a Remote/Receiver in another process. This is the basic primitive with which it's possible to expand the number of connections spanning two processes.

Here's an example of an interface:

// This interface is implemented by the parent process. It is used by its child
// processes to communicate commands to the parent.
interface ParentProcess {
  // Child tells parent process to navigate to `url`, with an arbitrary delay of
  // `delay` seconds.
  NavigateToURL(string url, int delay);

  // ...
  OpenFile(string name, bool truncate);

  // The parent binds this to a local `mage::Remote<magen::ChildProcess>` so it
  // can send messages *back* to its child.
  BindChildProcessRemote(MessagePipe child_remote);
}

Using Mage in your application

Using Mage to provide IPC support in an application is pretty simple; there are only a few steps:

1. Write your interface in a .magen file
2. Build your .magen file
3. Implement your interface in C++
4. Use a Remote to send IPCs to your cross-process interface (or any thread)

Let's assume you have a networking application (main.cc) that takes URLs from user input and fetches them, but you want to do the fetching in separate process (network_process.cc). Specifically, main.cc will spin up the network process and tell it what URLs to fetch and when. Consider the project structure:

my_project/
├─ src/
│  ├─ BUILD
│  ├─ main.cc
├─ network_process/
│  ├─ BUILD
│  ├─ socket.h
│  ├─ network_process.cc
├─ WORKSPACE

1. Write your interface in a .magen file

The first thing you need to do is write the Magen interface that main.cc will use to send messages to the network process. This includes a FetchURL IPC that contains a URL. Magen interfaces are typically defined in a magen/ directory:

// network_process/magen/network_process.magen

interface NetworkProcess {
  FetchURL(string url);
}

2. Build your .magen file

Next, you need to tell your BUILD file about the interface in your .magen file, so it can "build" it (generate the requisite C++ code). magen/ directories get their own BUILD files that invoke the Mage build process.

# network_process/magen/BUILD

load("@mage//mage/public/parser:magen_idl.bzl", "magen_idl")

# Generates `network_process.magen.h`, which can be included by depending on the
# ":include" target below.
magen_idl(
  name = "include",
  srcs = [
    "network_process.magen",
  ],
)

This tells Mage to generate a C++ header called network_process.magen.h based on the supplied interface. Both main.cc and network_process.cc can #include this header by listing the :include rule as a dependency. For example, you'd modify src/BUILD like so:

cc_binary(
  name = "main",
  srcs = [
    "main.cc",