(pre-alpha) JamRTC -- Jam sessions with Janus!

This is an attempt to create a simple prototype for doing jam sessions using WebRTC as a technology, and in particular taking advantage of Janus as an open source WebRTC server for the purpose. The main reason why this effort uses Janus and not P2P or another server is simply that I'm the main author of Janus itself, and so that's what I'm most familiar with: you're of course free to try and come up with different attempts that use different backends, or no backend at all (the more we use WebRTC to help music, the better!).

It's very early stages, and mostly meant as a playground for interested developers, musicians and more in general tinkerers, to experiment whether it's a viable road or not.

Motivation

In 2021, I made a presentation at FOSDEM called Can WebRTC help musicians?, which tried to go through some common music-based scenarios, and how WebRTC could possibly help implement them on the Internet when real-time communication is required, especially in light of the impact the COVID pandemic had on the music world. The presentation was mostly an overview, and probably asked more questions than it answered, but it did contain a section on jam sessions as a key scenario WebRTC might help with, considering the real-time nature of the technology.

It started from a few assumptions, they most important one being that browsers (usually the main target for WebRTC applications) are not a good device for participating in a jam session: in fact, the audio pipeline they implement presents some obstacles, like too much tinkering with the audio they capture (AGC, AEC), too much latency (integrated buffers and synchronization mechanisms), no way to easily capture generic devices (made for generic microphones, mostly), and no way to interact with lower-latency audio systems on the operating system (e.g., the JACK Audio Connection Kit). As such, the idea was to try and come up with a native application instead, in order to attempt to overcome the obstacles above. This is what this repo is trying to provide a foundation for.

Why not Jamulus, Jacktrip, <insert your favourite tool here>?

This effort is not trying to replace anything, let alone well known applications: for instance, while I never used it, I'm well aware that Jamulus is very widespread among musicians, who use it successfully every day. Heck, I even thought of calling this project "JAMus" (as in Jam over Janus) and then changed my mind because it sounded too much like Jamulus itself! :grin: The only purpose of this project is to experiment with whether or not WebRTC (which the other tools are not built upon) can also help build something comparable in terms of requirements, especially considering there's advantages in having WebRTC as part of the stack in a scenario like this: if it grows into something more than that, then great, but that's not the point.

If you're looking for something you can use RIGHT AWAY and is known to work great, then Jamulus or a similar application is indeed what you should use! If you're curious about WebRTC and music, then this is a good place to experiment with that, knowing it's not much more than a toy right now.

Why WebRTC?

Why not? :grin: Jokes apart, the main reason is that, as technology, WebRTC should be suited for the job, since it's main purpose is exchanging live audio/video/data streams in real-time. While it's mostly used for traditional use cases like conferencing, streaming, and stuff like that, the tools are there to do something in the "creative" department as well.

Apart from that, there are some other advantages in using WebRTC for the job. If the music streams you're exchanging are WebRTC-based, e.g., via an SFU like the Janus VideoRoom or others, then it's very easy to distribute the same streams to "traditional" endpoints as well, like browsers, mobile endpoints, or others. This means you could build a simple web app to serve a small or larger audience, for instance, whose requirements for super-low latency would not be the same as those of the players involved. At the same time, WebRTC streams are easy to integrate in other frameworks for further processing, e.g., for mixing, processing, transcoding, or broadcasting purposes, which make them a useful tool for additional requirements and use cases: you can find some examples in a Janus Workshop I did at ClueCon TGI2021 (slides).

Architecture

As anticipated, the main idea behind this effort was to try and write a native WebRTC application to optimize the process as much as possible, especially where latency is involved. I decided to base this on @GStreamer 's webrtcbin, for a few different reasons:

• I already had a bit of experience with it, having used it briefly in a couple of other projects;
• GStreamer comes with a great set of plugins for decoding, encoding and rendering media;
• GStreamer has dedicated plugins to interact with JACK as well;
• it's relatively easy to build a GUI on top of GStreamer's building blocks (I used GTK+ 3.0, here).

At the same time, I wanted to use Janus as a reference server for all endpoints engaged in a session. This led to the diagram you can see below:

In a nutshell, the idea was to have this GStreamer-based native application join a Janus VideoRoom, and then:

1. publish media to the server (e.g., your guitar and your webcam/mic);
2. subscribe to media from other participants (e.g., drum and bass player) via the same server.

I conceived the application to not only allow you to share musical instruments, but also to have a parallel channel to interact with the other participants. As such, by default when launching JamRTC it will create two WebRTC PeerConnections: one to send your webcam + microphone, and another to send your instrument as an audio-only stream; of course you can choose whether you want to publish everything, just something, or nothing at all. When other participants join the session, the tool is conceived to automatically subscribe to whatever they're sharing: all the feeds are then rendered in a very simple (and super-ugly) UI, where audio streams are visualized via a wavescope.

It's important to point out that, when using JACK as the audio backend, JamRTC is conceived not to automatically pick the devices to capture: this is up to you to do, by creating the connections you want manually. I'll get back to this later on, when explaining how to test what's currently there.

Building JamRTC

The main dependencies of JamRTC are:

• GLib
• pkg-config
• GStreamer (>= 1.16)
• GTK+ 3
• Glade
• libwebsockets

Make sure the related development versions of the libraries are installed, before attempting to build JamRTC, as to keep things simple the Makefile is actually very raw and naive: it makes use of pkg-config to detect where the libraries are installed, but if some are not available it will still try to proceed (and will fail with possibly misleading error messages). All of the libraries should be available in most repos (they definitely are on Fedora, which is what I use everyday, and to my knowledge Ubuntu as well).

Once the dependencies are installed, all you need to do to build JamRTC is to type:

make

This will create a JamRTC executable. Trying to launch that without arguments should display a help section:

[lminiero@lminiero JamRTC]$ ./JamRTC
Usage:
  JamRTC [OPTION?] -- Jam sessions with Janus!

Help Options:
  -h, --help              Show help options

Application Options:
  -w, --ws                Address of the Janus WebSockets backend (e.g., ws://localhost:8188; required)
  -r, --room              Room to join (e.g., 1234; required)
  -d, --display           Display name to use in the room (e.g., Lorenzo; required)
  -M, --no-mic            Don't add an audio source for the local microphone (default: enable audio chat)
  -W, --no-webcam         Don't add a video source for the local webcam (default: enable video chat)
  -v, --video-device      Video device to use for the video chat (default: /dev/video0)
  -i, --instrument        Description of the instrument (e.g., Guitar; default: unknown)
  -s, --stereo            Whether the instrument will be stereo or mono (default: mono)
  -I, --no-instrument     Don't add a source for the local instrument (default: enable instrument)
  -b, --jitter-buffer     Jitter buffer to use in RTP, in milliseconds (default: 0, no buffering)
  -c, --src-opts          Custom properties to add to jackaudiosrc (local instrument only)
  -S, --stun-server       STUN server to use, if any (hostname:port)
  -T, --turn-server       TURN server to use, if any (username:password@host:port)
  -l, --log-level         Logging level (0=disable logging, 7=maximum log level; default: 4)
  -J, --no-jack           For testing purposes, use autoaudiosrc/autoaudiosink instead (default: use JACK)

Running JamRTC

At the moment, you can only customize the behaviour of JamRTC at startup, via command-line arguments, as there are no visual settings in the GUI or parameters you can tweak dynamically. This means that there are some required arguments to start the tool, namely:

• The address of the WebSocket backend for the Janus instance to use;
• The unique ID of