Libretto

Declarative concurrency and stream processing library for Scala.

Motivation

... or why another concurrency and stream processing library.

Libretto grew out of frustration with existing libraries. Here is an attempt to summarize the pain points.

• Dissatisfaction with effects being a prerequisite for concurrency.

  Existing libraries tend to build concurrency on top of effects: completing a Promise is an effect, spawning an actor or a fiber is an effect, enqueueing a message in a queue is an effect.

  Libretto rejects the assumption that effects are a prerequisite for concurrency.

  Although stream processing libraries do provide some high-level operations for declarative concurrency that abstract away from the underlying effects, we haven't come across a library with a set of high-level operations that is expressive enough for a moderately complex application.

• Underdelivery on the promise of writing mere descriptions or blueprints of programs.

  Substantial parts of such "descriptions" are opaque Scala functions.

  Moreover, even in an IO monad-style program, we still manipulate live objects with identities, such as fibers, mutable references, queues, "pre-materialized" blueprints, ...

  That does not fit our idea of writing mere descriptions of programs.

• The cognitive load of making sure all the wires are connected.

  In ensuring that

  • each Promise is completed and completed only once,
  • every spawned computation (Future, actor, fiber) is awaited,
  • all streams are consumed, all sinks are fed,
  • all messages from a queue are processed (or intentionally discarded),

  programmers are typically left on their own, with no help from the type system.

• Callbacks everywhere.

  It is disappointing how callback-ridden our programs, including our functional programs, are when it comes to concurrency and accessing resources.

  By a callback we mean a function object that performs effects (on resources it has captured) and is passed to someone else to be executed.

  One common example of a callback-style interface is an HTTP server taking, during construction, a request handler callback. The request handler has captured resources, such as a reference to mutable state or a database connector, and performs side-effects on them.

  The problem with callbacks is that they create non-local interaction, which makes it hard to reason about programs.

• Lack of expressivity of the high-level stream operations.

  Building custom dataflow topologies with existing libraries, if at all possible, requires escaping to a different paradigm, one which is more low-level and imperative.

  We are talking about that case when you needed a slight variation on a stream operator and ended up "manually" shoveling messages between queues, mutable references and/or promises.

• Inability to express protocols of interaction in types.

  In practice this leads to having to handle a lot of illegal state.

  To give an example, consider a protocol between two components, Alice and Bob, according to which Alice sends A to Bob, Bob replies with B or C, and then Alice replies with D if Bob sent B and with E if Bob sent C.

  In a common (but unsatisfactory) implementation, Bob formally accepts any of A, D, E at all times and raises or replies with an error if the protocol is violated, i.e. if it is not in a state in which it can accept the message.

Features

• Concurrent by design

  Libretto programs are naturally concurrent, without special control-flow operators.

• Session types

• Statically checked linearity

  That every producer is connected to a consumer, etc., is guaranteed statically, i.e. before execution^(*).

  ^(*) For programs written in point-free style, this is at compile time. For programs written using λ-notation, it is at assembly time, i.e. when converting to the point-free representation.

• Direct-style programming

  All interaction of a component with its environment is stated by its type signature (much like inputs and outputs of a pure function).

  This is in contrast to callback-style which leads to spooky action at a distance. See also What's the problem with callbacks?.

• Expressiveness

  Libretto provides a self-contained DSL that is powerful enough to express many different concepts without using escape hatches to the underlying layer.

  Streams themselves and dynamic stream topologies are expressible without needing built-in support.

• Graphical notation

  We are often able to visualize the architecture of a system as a diagram, but there is usually a big gap between the diagram and the code.

  Libretto primitives have graphical notation that composes into larger diagrams as we compose the program, preserving close correspondence between code and graphical representation.

• Programs as data

  Libretto programs are just data structures. This opens new possibilities: they can be inspected, manipulated, optimized, given different interpretations, used for code generation, sent over the network, ... It also means that many different Libretto implementations are possible.

  See also Are Libretto programs any more amenable to inspection than IO monad programs?

Libretto in a Nutshell

Libretto takes inspiration from linear logic, with a twist of interpreting the multiplicative conjunction, ⊗, in a concurrent fashion.

The correspondence of linear logic to certain monoidal categories led to a point-free notation that enforces linearity statically, despite the lack of linear types in the host language.

There is also lambda syntax, whose linearity checking is deferred until assembly time (i.e. when constructing the Libretto blueprint, before execution).

Primitives for racing and recursion were added.

Documentation

Scaladoc (Found something undocumented or not documented clearly? We should do better. Do not hesitate to submit a documentation request.)

Tutorial

Presentations

Typed Interaction with Session Types (using Scala and Libretto). Functional Scala 2022 [video] [slides]
Shows how Libretto expresses session types.

Custom Stream Operators Made Safe And Simple with Libretto. Scalar 2023 [video] [slides]
Shows the expressiveness of Libretto in implementing stream operators. Presents examples that are hard to express in other stream libraries.

Concurrent All The Way Down (Functional Concurrency with Libretto). LambdaDays 2023 [video] [slides]
Concurrency without threads or side-effects.

When Your DSL Needs to Support User-Defined Domain Functions. ScalaDays 2023 [video] [slides]
Presents the DSL embedding technique behind Libretto: delambdification followed by "compiling to categories" (i.e. to point-free representation).

Monads Are Not About Sequencing. Functional Scala 2023 [video] [slides]
One of the presented examples, the concurrent, non-deterministic Writer, is in Libretto.

Caveats

It is all too common for software projects to highlight only the good parts or to overstate their features. We value your time and want to be absolutely honest about the limitations. However, it is hard to know what we are missing. That's why we need your help.

Do you find that libretto does not hold up to some of its promises? Do you find that the project description omits some important limitation? Do you feel that you have spent too much time to find out something that could have been presented more readily? Please, let us know.

Batteries not included. The streams library is under-developed. I/O library (file, network) is non-existent at the moment. There are currently no connectors or wrappers for things like HTTP servers, DB connectors, message brokers, ...

Not yet a good story for supervision/error recovery.

Flawed proof-of-concept implementation. The current implementation is leaky. It is not a design flaw, just an implementation flaw.

Best practices not established. Programming in Libretto is quite different from conventional concurrency libraries. It is not clear what the best practices should be.

Not used to this level of concurrency. In conventional programming languages and libraries, concurrency is a scarce special thing. In Libretto, concurrency is the default and sequentiality is a special thing. The consequences might be surprising.

Q&A

Did not find an answer to your question? Do not hesitate to ask us.

What is Libretto for?