CoWasm: Collaborative WebAssembly for Servers and Browsers

URL: https://github.com/sagemathinc/cowasm

Welcome

CoWasm, or "Collaborative WebAssembly," goes far beyond just Python. The collaboration aspects will be part of CoCalc at some point in the future. CoWasm will support various technologies (such as libgit2 and realtime sync) that are important foundations for collaboration.

The underlying software components that CoWasm is built on (i.e., those we didn't write) are mostly extremely stable and mature. Zig is less stable, but we mostly use Zig for its amazing cross-compilation support and packaging of clang/llvm and musl-libc, which are themselves both very mature. Many other components, such as Python, Dash, and Numpy, are extremely mature, multi-decade-old projects. Moreover, other components of CoWasm, such as memfs, are libraries with 10M+ downloads per week that are heavily used in production.

The goal of CoWasm is overall somewhat similar to emscripten, WebAssembly.sh, wapm.io, and Pyodide combined, in various ways:

Unlike WebAssembly.sh and wapm.io (but similar to Pyodide), we make very heavy use of shared dynamic libraries (e.g., -fPIC code), which is only possible because of a plugin contributed from emscripten to LLVM. The "Co" in CoWasm suggests "collaboration" or "sharing," which also reflects how the binaries in this project are structured.
We use actual editline (similar to readline) instead of a JavaScript terminal. Moreover, unlike other WebAssembly shells, we use a real command-line shell (dash = Debian Almquist Shell). We also have a user space including ports of many coreutils, e.g., ls, head, tail, etc.
Unlike emscripten, we use modern TypeScript, our code is more modular, and we make use of existing components when possible (e.g., the Node.js memfs project), instead of writing our own.
A core design constraint is to efficiently run on a wide range of platforms-not just in the browser like emscripten, and not just on servers like wasmer. CoWasm should run on servers, desktops (e.g., as an Electron app), iPad/iOS apps, and in web browsers.
There is no business-unfriendly GPL code in CoWasm. CoWasm itself is extremely liberally licensed and business-friendly. The license of all new code and most components is 3-clause BSD. CoWasm will serve as a foundation for other projects with more restrictive licenses:
- CoCalc will build on top of CoWasm to provide a graphical interface and real-time collaboration, and that will be a commercial product.
- Products like GP/PARI SageMath will build on CoWasm to provide GPL-licensed mathematics software.

Packages

There are several subdirectories that each contain packages:

core – Core functionality, including dynamic linking, a WASI implementation, OpenSSL, and nontrivial POSIX extensions to Node.js written in Zig. (This is analogous to emscripten.org.)
python – A WebAssembly build of Python, along with some nontrivial scientific libraries. (This is analogous to pyodide.org.)
web – Examples of how to use the other packages in web applications; this includes building https://cowasm.org.
desktop – A native Electron app that provides a sandboxed WebAssembly Python terminal running on your native filesystem.
sagemath – The beginning of a port of https://sagemath.org to WebAssembly. Vast amounts of work remain.

Python

An exciting package in CoWasm is python-wasm, which is a build of Python for WebAssembly that supports both servers and browsers. It also supports extension modules such as numpy. See python/README.md for more details.

Build from Source

We support and regularly test building CoWasm from source on the following platforms:

x86 macOS and aarch64 macOS (Apple Silicon M1)
x86 and aarch64 Linux

Prerequisites

You need Node.js version at least 16.x, pnpm and several standard dev tools listed below. The dependency you need for every possible package are as follows:

On MacOS, install the XCode command line tools.
On Linux apt-based system, e.g., on Ubuntu 22.04:

sudo apt-get update && sudo apt-get install git make cmake curl dpkg-dev m4 yasm texinfo python-is-python3 libtool tcl zip libncurses-dev

If you also want to install node v18 and pnpm on Ubuntu, you could do:

     sudo curl -sL https://deb.nodesource.com/setup_18.x | bash - \
  && sudo apt-get install -y nodejs \
  && curl -fsSL https://get.pnpm.io/install.sh | sh -

On Linux RPM based system, e.g., Fedora 37:

dnf install git make cmake curl dpkg-dev m4 yasm texinfo libtool tcl zip ncurses-devel perl

On ArchLinux:

pacman -Sy binutils git nodejs npm cmake curl m4 yasm texinfo python libtool tcl zip unzip patch binutils diffutils

NOTE: pnpm is not in the Arch Linux official package sets. You may want to install it from Arch User Repository (AUR) AUR (en) - pnpm.

Currently, the only way to build CoWasm from source on MS Windows is to use a Docker container running Linux. Using WSL2 (maybe) works but is too slow.

In addition you need to install Node.js version at least 16.x and install the pnpm package manager.

Notes about Zig

You do NOT need to install Zig, and it doesn't matter if you have a random version of Zig on your system already. A zig binary is download automatically. We use zig (instead of any system-wide clang, etc. compilers) for building all compiled code and write some code in the zig language. Since zig is fairly unstable it is critical to use the exact version that we provide.

Build

To build any package in the src/packages directory, cd into that directory, then:

make

That's it, usually. You do not have to run build at the top level and you can start with building any specific package -- it will automatically cause any required dependencies to get installed or built.

You can also force building of every single package and running its test suite if you want:

~/cowasm$ make test
...
##########################################################
#                                                        #
#   CONGRATULATIONS -- FULL COWASM TEST SUITE PASSED!    #
#
#   Fri Oct 28 12:32:19 AM UTC 2022
#   Linux aarch64
#   Git Branch: main
#                                                        #
##########################################################

Depending on your computer, the build should take less than 30 minutes, and about 6GB's of disk space.

This installs a specific tested version of Zig and Nodejs, then builds native and WebAssembly versions of CPython and many dependencies, and also builds all the Typescript code. It also builds many other interesting programs with ports to WebAssembly, though many are not completely finished (e.g., there is the dash shell and ports of tar and BSD coreutils). As mentioned, building from source is regularly tested on Linux and MacOS with both x86_64 and ARM (M1) processors:

Linux: tested on both x86_64 and aarch64 Ubuntu
MacOS: tested on both x86_64 and M1 mac with standard XCode command live dev tools installed.

CoWasm does not use the compilers on the system, and instead uses clang/llvm as shipped with Zig. If you're using Windows, you'll have to use Linux via a virtual machine or Docker.

Pull latest code, build and test

At the top level run ./bin/rebuild-all :

~/cowasm$ ./bin/rebuild-all

This does make clean, pulls the latest code, then runs the full build and test suite. Fortunately, zig caches a lot of build artifacts, so subsequent builds are faster.

NOTE/WARNING: Zig's cache is in ~/.cache/zig and it can get HUGE. As far as I can tell, I think it just grows and grows without bound (it's not an LRU cache), and I think there are no tools to "manage" it besides just rm -rf it periodically.

What is tested?

Note that running make test at the top level does NOT run the full test suite of every package, since it takes quite a while and there are still some failing tests, since CoWasm doesn't support enough of what Python expects. It does run a large supported subset of the cpython test suite (it's the part that I got to pass so far, which is over 80%). As an example, the sympy test suite is massive, takes a very long time to run, and doesn't even work for me natively; instead, we just run a handful of small tests to ensure sympy is working at all. Similarly, for Cython, we run all their demos, but not the full test suite. A longer term goal of CoWasm is to support a second more thorough testing regime that runs the full test suite of each package. There will likely always be issues due to WASM not being a multiuser POSIX system, but it's good to know what those issues are!

You can also use the WebAssembly Python REPL directly on the command line.

~/cowasm$ ./bin/python-wasm 
Python 3.11.0 (main, Oct 27 2022, 10:03:11) [Clang 15.0.3 (git@github.com:ziglang/zig-bootstrap.git 0ce789d0f7a4d89fdc4d9571 on wasi
Type "help", "copyright", "credits" or "license" for more information.
>>> 2 + 3
5
>>> import sys
>>> sys.platform
'wasi'
>>> sys.executable
'/Users/wstein/build/cocalc/src/data/projects/2c9318d1-4f8b-4910-8da7-68a965514c95/cow

Cowasm

Install / Use

README