Fractalide

image::https://raw.githubusercontent.com/fractalide/fractalide/master/pkgs/hyperflow/imgs/fractalide.png[Fractalide,align="center"]

Reusable Reproducible Composable Software

image:https://img.shields.io/badge/license-MPLv2-blue.svg[LICENSE,link=https://github.com/fractalide/fractalide/blob/master/LICENSE] image:https://travis-ci.org/fractalide/fractalide.svg?branch=master["Build Status", link="https://travis-ci.org/fractalide/fractalide"]

== Welcome

// tag::doc[]

== What is this?

Fractalide is a free and open source service programming platform using dataflow graphs. Graph nodes represent computations, while graph edges represent typed data (may also describe tensors) communicated between them. This flexible architecture can be applied to many different computation problems, initially the focus will be Microservices to be expanded out into the Internet of Things.

Fractalide is in the same vein as the https://en.wikipedia.org/wiki/Apache_NiFi[NSA's Niagrafiles] (now known as https://nifi.apache.org/[Apache-NiFi]) or https://en.wikipedia.org/wiki/TensorFlow[Google's TensorFlow] but stripped of all Java, Python and GUI bloat. Fractalide faces big corporate players like http://abinitio.com/[Ab Initio], a company that charges a lot of money for dataflow solutions.

Truly reusable and reproducible efficient nodes is what differentiates Fractalide from the others. It's this feature that allows open communities to mix and match nodes quickly and easily.

== Features

Fractalide stands on the shoulders of giants by combining the strengths of each language into one programming model.

[cols="9*^"] |=== |Op |Technology |Safe|Zero-cost Abstractions|Reuse|Reproducible|Distributed Type System| Concurrent| Service Config Man.

| |NixOS | | | | | | |✔ |+ |Nix Expr | | | |✔ | | | |+ |Rust |✔ |✔ | | | |✔ | |+ |Flow-based Programming | | |✔ | | |✔ | |+ |Cap'n Proto| | | | |✔ | | |= |Fractalide Model |✔ |✔ |✔ |✔ |✔ |✔ |✔ |===

== What's new from different perspectives

=== Nix Programmers

Fractalide brings safe, fast, reusable, black-box dataflow functions and a means to compose them.

Tagline: "Nixpkgs is not enough! Here, have 'Nixfuncs' too!"

=== Rust Programmers

Fractalide brings reproducible, reusable, black-box dataflow functions, a means to compose them and a https://www.usenix.org/legacy/event/lisa02/tech/full_papers/traugott/traugott_html/[congruent] model of configuration management.

Tagline: Safety extended beyond the application boundary into infrastructure.

=== Flow-based Programmers

Fractalide brings safe fast reproducible classical Flow-based programming components, and a https://www.usenix.org/legacy/event/lisa02/tech/full_papers/traugott/traugott_html/[congruent] model of configuration management.

Tagline: Reproducible components!

=== Programmers

Fractalide brings safe, fast, reusable, reproducible, black-box dataflow functions, a means to compose them and a https://www.usenix.org/legacy/event/lisa02/tech/full_papers/traugott/traugott_html/[congruent] model of configuration management.

Tagline: Here, have a beer!

== Solved problems

=== Modules-code coupling

Language level modules become tightly coupled with the rest of the code, moving around these modules also poses a problem.

==== Solution

An unanticipated outcome occurred when combining FBP and Nix. It's become our peanut butter and jam combination, so to say, but requires a bit of explaining, so hang tight.

===== Reproducibility

Nix is a content addressable store, so is git, so is docker, except that docker's SHA resolution is at container level and git's SHA resolution is at changeset level. Nix on the other hand has a SHA resolution at package level, and it's known as a derivation. If you're trying to create reproducible systems this is the correct resolution. Too big and you're copying around large container sized images with multiple versions occupying gigabytes of space, too small and you run into problems of git not being able to scale to support thousands of binaries that build an operating system. Therefore Nix subsumes Docker.

Indeed it's these simple derivations that allow python 2.7 and 3.0 to exist side-by-side without conflicts. It's what allows the Nix community to compose an entire operating system, NixOS. These derivations are what makes NixOS a congruent configuration management system, and congruent systems are reproducible systems. They have to be.

===== Reusability

Flow-based programming in our books has delivered on its promise. In our system FBP components are known as nodes and they are reusable, clean and composable. It's a very nice way to program computers. Though, we've found, the larger the network of nodes, the more overhead required to build, manage, version, package, connect, test and distribute all these moving pieces. This really doesn't weigh well against FBP's advantages. Still, there is this beautiful reusable side that is highly advantageous! If only we could take the good parts?

===== Reproducibility + Reusability

When nix is assigned the responsibility of declaratively building fbp nodes, a magic thing happens. All that manual overhead of having to build, manage and package etc gets done once and only once by the node author, and completely disappears for everyone thereafter. We're left with the reusable good parts that FBP has to offer. Indeed the greatest overhead a node user has, is typing the node's name. We've gone further and distilled the overhead to a few lines, no more intimidating than a typical config file such as Cargo.toml:

[source, nix, subs="none"]

{ agent, edges, mods, pkgs }:

agent { src = ./.; edges = with edges; [ PrimText FsPath ]; mods = with mods.rs; [ rustfbp rusqlite ]; osdeps = with pkgs; [ sqlite pkgconfig ]; }

Now just to be absolutely clear of the implications; it's possible to call an extremely complex community developed hierarchy of potentially 1000+ nodes, where each node might have different https://crates.io dependencies, they might have OS level dependencies such as openssl etc and nix will ensure the entire hierarchy is correctly built and made available. All this is done by just typing the node name and issuing a build command.

It's this feature that sets us apart from Google TensorFlow and Apache-NiFi. It contains the DNA to build a massive sprawling community of open source programmers, this and the C4, that is. It's our hope anyway!

=== Complex configuration management model

The vast majority of system configuration management solutions use either the divergent or convergent model.

We're going to quote Steve Traugott's excellent work verbatim.

==== Divergent

image::https://raw.githubusercontent.com/fractalide/fractalide/master/doc/images/divergent.png[]

[quote, Steve Traugott]

---

"One quick way to tell if a shop is divergent is to ask how changes are made on production hosts, how those same changes are incorporated into the baseline build for new or replacement hosts, and how they are made on hosts that were down at the time the change was first deployed. If you get different answers, then the shop is likely divergent.

The symptoms of divergence include unpredictable host behavior, unscheduled downtime, unexpected package and patch installation failure, unclosed security vulnerabilities, significant time spent "firefighting", and high troubleshooting and maintenance costs."

---

==== Convergent

image::https://raw.githubusercontent.com/fractalide/fractalide/master/doc/images/convergent.png[]

[quote, Steve Traugott]

---

"The baseline description in a converging infrastructure is characteristically an incomplete description of machine state. You can quickly detect convergence in a shop by asking how many files are currently under management control. If an approximate answer is readily available and is on the order of a few hundred files or less, then the shop is likely converging legacy machines on a file-by-file basis.

A convergence tool is an excellent means of bringing some semblance of order to a chaotic infrastructure. Convergent tools typically work by sampling a small subset of the disk - via a checksum of one or more files, for example - and taking some action in response to what they find. The samples and actions are often defined in a declarative or descriptive language that is optimized for this use. This emulates and preempts the firefighting behavior of a reactive human systems administrator - "see a problem, fix it." Automating this process provides great economies of scale and speed over doing the same thing manually.

Because convergence typically includes an intentional process of managing a specific subset of files, there will always be unmanaged files on each host. Whether current differences between unmanaged files will have an impact on future changes is undecidable, because at any point in time we do not know the entire set of future changes, or what files they will depend on.

It appears that a central problem with convergent administration of an initially divergent infrastructure is that there is no documentation or knowledge as to when convergence is complete. One must treat the whole infrastructure as if the convergence is incomplete, whether it is or not. So without more information, an attempt to converge formerly divergent hosts to an ideal configuration is a never-ending process. By contrast, an infrastructure based upon first loading a known baseline configuration on all hosts, and limited to purely orthog