avplumber - make your own libav processing graph

avplumber is a graph-based real-time processing framework. Graph can be reconfigured on the fly using a text API. Most nodes are based on FFmpeg's libavcodec, libavformat & libavfilter. You can create entire transcoding & filtering chain in it, replacing FFmpeg in many use cases.

avplumber was created because we were experienced with FFmpeg and wanted to have its features, plus more flexibility. For example, it is possible to:

• encode once and send encoded packets to multiple outputs.
• filter video (using FFmpeg's filter graph syntax) in multiple threads. It is possible since FFmpeg 6.0, but we needed this feature long before its release.
• maintain output timestamps continuity and audio-video synchronization even when input timestamps jump.
• insert fallback slate ("we'll be back shortly") when input stream breaks.
• monitor input stream health, analyzing speed, actual FPS & sample rate, audio levels.
• reconfigure processing graph on the fly.

Furthermore, it was designed to allow easy prototyping of new video & audio processing blocks (nodes in graph) without writing so much boilerplate code that is needed in case of libavfilter or GStreamer.

However, it does not replace FFmpeg in all use cases. For example, subtitles aren't supported due to limitations of the underlying library - avcpp.

Curious about history and applications of this project? Read Story of avplumber — open source multimedia streaming engine from Amagi at Amagi Engineering blog.

Quick start

Note: be sure to check other branches (tree view) if you want to test latest features.

Make sure to clone this repo with --recursive option.

git clone --recursive https://github.com/amagimedia/avplumber
docker build -t avplumber .
docker run -p 20200:20200 avplumber -p 20200

or if you don't want to use Docker but have Ubuntu:

apt install git gcc pkg-config make cmake libavcodec-dev libavdevice-dev libavfilter-dev libavformat-dev libavutil-dev libswresample-dev libcurl4-openssl-dev libboost-thread-dev libboost-system-dev libssl-dev
make -j`nproc`
./avplumber

and in a different terminal:

nc localhost 20200

and you can type some commands (see Control protocol) or paste a script (e.g. from examples/ directory)

Development on Windows

Development on Windows can be done using Docker and VSCode Dev Containers.

1. Enable symbolic links by following these steps.
2. Clone this repo git clone --recursive https://github.com/amagimedia/avplumber
3. Open it in VSCode
4. Open Command Palette and run Dev Containers: Reopen in Container command

Development container comes with all required dependencies and clangd installed.

Demo

To quickly run demo with FFmpeg test source, use the provided Docker Compose file:

script=remux_analyze_audio.avplumber docker compose -f examples/compose/rtmp_test_source.yml up

After Docker pulls and builds everything, you should see stream statistics JSON lines, once per second.

Output stream will be available at rtmp://localhost/live/output

Change script to complicated_transcoder.avplumber to test transcoding.

This demo uses MediaMTX as streaming server.

Running Docker on recent Mac OSX versions

brew install docker docker-compose colima
colima start

Build process details

The build is driven by Makefile variables. Set them on the make command line, e.g.:

make -j`nproc` HAVE_CUDA=1 HAVE_DRM=1 HAVE_NVCC=1

• BUILD_TYPE: Debug (default) or Release
  • Debug enables debug-only nodes (jittergen, delaygen).
  • Release sets compiler flags to more optimization.
• HAVE_CUDA=1: enable CUDA support and CUDA-based nodes. Uses dynlink loader, so does not require anything during compilation and lack of CUDA libraries in runtime is non-fatal (nodes not using CUDA will work normally)
• HAVE_GL=1: enable OpenGL & EGL dependency, required by drm_prime_to_cuda, cuda_to_egl_image
• HAVE_VAAPI=1: enable VAAPI paths (and implicitly OpenGL/EGL). Links -lva -lGL -lEGL -lGLESv2. Requires libva-dev and GL/EGL development packages.
• HAVE_DRM=1: enable DMA-BUF IPC source and DRM-dependent paths. Requires libdrm-dev.
• HAVE_JACK=1: enable jack_sink. Links -ljack. Requires libjack-dev.
• HAVE_NVCC=1: build CUDA PTX for GPU color conversion used by cuda_to_egl_image. Requires nvcc and OpenGL/EGL at build time.
• HAVE_SCTE35=1: build SCTE35 libraries and scte35_parse node (used for inserting ads and switching to regional programs in TV distribution systems)
• EMBED_IN=obs: builds nodes and adds fields specific to OBS source plugin

Feature gates:

• cuda_to_egl_image builds only when HAVE_CUDA=1 HAVE_GL=1 HAVE_NVCC=1.
• drm_prime_to_cuda builds only when HAVE_CUDA=1 HAVE_GL=1 HAVE_DRM=1.
• HAVE_GL is auto-enabled when HAVE_VAAPI=1
• scte35_parse builds only when HAVE_SCTE35=1

Using as a library

avplumber can be built as a static library: make static_library will make libavplumber.a which your app or library can link to. library_examples/obs-avplumber-source/CMakeLists.txt is an example of CMake integration.

Public API is contained in src/avplumber.hpp.

Example: library_examples/obs-avplumber-source - source plugin for OBS supporting video decoder to texture direct VRAM copy.

Developing custom nodes

See doc/developing_nodes.md

Graph

An avplumber instance consists of a directed acyclic graph of interconnected nodes.

Edges = queues

Nodes in the graph are connected by edges. Edge is implemented as a queue. queue.plan_capacity can be used to change its size. Type of data inside queue is determinated automatically when the queue is created.

Data types:

• av::Packet - encoded media packet
• av::VideoFrame - raw video frame
• av::AudioSamples - raw audio frame (usually 1024 samples of all channels)
• EglImageFrame - GPU RGBA image passed by EGLImageKHR handle with PTS/timebase

Some nodes support multiple input/output types - they work like templates/generics in programming languages (and are implemented this way). If the data type can be deduced from source or sink edges, there is no need to provide it explicitly. But if it can't be, use template syntax in type field of the node JSON object:

node_type<data_type>

for example:

split<av::VideoFrame>

Topology

Some nodes require that other node implementing specific features (an interface) is placed before (up) or after (down) it:

• input/input_rec before demux
• mux before output
• video format metadata source before enc_video. It can be dec_video, assume_video_format, rescale_video or filter_video
• FPS metadata source before enc_video, extract_timestamps and filter_video. It can be dec_video, force_fps, filter_video or sentinel_video
• audio metadata source before enc_audio and sentinel_audio. It can be dec_audio, assume_audio_format or filter_audio
• time base source before bsf, enc_video, enc_audio, extract_timestamps, filter_video, filter_audio, sentinel_video, sentinel_audio. It can be assume_video_format, assume_audio_format, dec_video, dec_audio, filter_video, filter_audio, force_fps, packet_relay or resample_audio
• encoder (enc_video/enc_audio), bsf or packet_relay before mux

Control methods

avplumber is controlled using text commands on TCP socket, so it can be controlled manually using netcat or telnet. --port argument specifies the port to listen on.

--script argument specifies commands to execute on startup.

Control protocol

Control protocol loosely follows MVCP.

On new connection, server (avplumber) sends a line: 100 VTR READY

Client issues a command followed by arguments separated by spaces and ending with the new line character. The last argument may contain spaces.

Server (avplumber) responds with line with status code and information:

• 200 OK - command accepted, empty response
• 201 OK - command accepted, response will follow and an empty line marks the end of response
• 400 Unknown command: ...
• 500 ERROR: ...
• BYE and connection close - special response for bye command

Commands:

hello

Replies with HELLO

version

Replies with app version and build date

bye

Closes the connection

Nodes management & control

node.add { ...json object... }

Add node

node.add_create { ...json object... }

Add and create node (without starting it right now)

node.add_start { ...json object... }

Add, create and start node

node.delete name

Delete node

node.start name

Start node

node.stop name

Stop node (auto_restart action is inhibited)

node.stop_wait name

Stop node, return reply when node really stopped

node.auto_restart name

Stop node and trigger its auto_restart action. This command is probably most useful for restarting input after changing its URL, with confidence that it will be handled the same way as if the previous input stream finishe