Rffmpeg

<p align="center"> <img alt="License: GPLv3+" src="https://img.shields.io/github/license/joshuaboniface/rffmpeg"/> <img alt="Code Style: Black" src="https://img.shields.io/badge/code%20style-black-000000.svg"/> <a href="https://matrix.to/#/#rffmpeg:matrix.org"> <img alt="Chat on Matrix" src="https://img.shields.io/matrix/rffmpeg:matrix.org.svg?logo=matrix"/> </a> <a href="https://www.patreon.com/joshuaboniface"> <img alt="Support me on Patreon" src="https://img.shields.io/endpoint.svg?url=https%3A%2F%2Fshieldsio-patreon.vercel.app%2Fapi%3Fusername%3Djoshuaboniface%26type%3Dpatrons&style=flat"/> </a> <a href="https://github.com/sponsors/joshuaboniface"> <img alt="Support me on GitHub" src="https://img.shields.io/github/sponsors/joshuaboniface?label=GitHub%20Sponsors"> </a> </p>

rffmpeg is a remote FFmpeg wrapper used to execute FFmpeg commands on a remote server via SSH. It is most useful in situations involving media servers such as Jellyfin (our reference user), where one might want to perform transcoding actions with FFmpeg on a remote machine or set of machines which can better handle transcoding, take advantage of hardware acceleration, or distribute transcodes across multiple servers for load balancing.

Quick usage

1. Install the required Python 3 dependencies: click, yaml and subprocess (sudo apt install python3-click python3-yaml python3-subprocess in Debian) and optionally install psycopg2 with sudo apt install python3-psycopg2 for Postgresql support.

2. Create the directory /etc/rffmpeg.

3. Optionally, copy the rffmpeg.yml.sample file to /etc/rffmpeg/rffmpeg.yml and edit it to suit your needs.

4. Install rffmpeg somewhere useful, for instance at /usr/local/bin/rffmpeg.

5. Create symlinks for the command names ffmpeg and ffprobe to rffmpeg, for example sudo ln -s /usr/local/bin/rffmpeg /usr/local/bin/ffmpeg and sudo ln -s /usr/local/bin/rffmpeg /usr/local/bin/ffprobe.

6. Initialize the database and add a target host, for example sudo rffmpeg init && rffmpeg add myhost.domain.tld.

7. Set your media program to use rffmpeg via the ffmpeg symlink name created above, instead of any other ffmpeg binary.

8. Profit!

rffmpeg does require a little bit more configuration to work properly however. For a comprehensive installation tutorial based on a reference setup, please see the SETUP guide.

NOTE Jellyfin 10.10.x and newer require an additional TMPDIR environment variable set to somewhere exported to the remote machine, or these paths will not work properly. Edit your Jellyfin startup/service configuration to set that. See the setup guide for more details.

Setup and Usage

The rffmpeg Configuration file

rffmpeg will look at /etc/rffmpeg/rffmpeg.yml (or a path specified by the RFFMPEG_CONFIG environment variable) for a configuration file. If it doesn't find one, defaults will be used instead. You can use this file to override many configurable default values to better fit your environment. The defaults should be sensible for anyone using Jellyfin and following the SETUP guide.

The example configuration file at rffmpeg.yml.sample shows all available options; this file can be copied as-is to the above location and edited to suit your needs; simply uncomment any lines you want to change. Note that if you do specify a file, you must ensure that all top-level categories are present or it will error out.

NOTE: If you are running into problems with rffmpeg, you must use the config file to adjust logging -> debug to true to obtain more detailed logs before requesting help.

Each option has an explanatory comment above it detailing its purpose.

Since the configuration file is YAML, ensure that you do not use "Tab" characters inside of it, only spaces.

CLI interface to rffmpeg

rffmpeg is a Click-based application; thus, all commands have a -h or --help flag to show usage and additional options that may be specified.

Initializing rffmpeg

After first installing rffmpeg, you must initialize the database with the rffmpeg init command.

Note that by default, sudo/root privilege is required for this command to create the required data paths, but afterwards, rffmpeg can be run by anyone in the configured group (by default the sudo group). You can bypass the sudo requirement with the --no-root command, for example when running in a rootless container; this will require the running user to have write permissions to the state and database parent directories, and will not perform any permissions modifications on the resulting files.

Viewing Status

Once installed and initialized, you can see the status of the rffmpeg system with the rffmpeg status command. This will show all configured target hosts, their states, and any active commands being run.

Adding or Removing Target Hosts

To add a target host, use the rffmpeg add command. You must add at least one target host for rffmpeg to be useful. This command takes the optional -w/--weight flag to adjust the weight of the target host (see below). A host can also be added more than once for a pseudo-weight, but this is an advanced usage.

To remove a target host, use the rffmpeg remove command. This command takes either a target host name/IP, which affects all instances of that name, or a specific host ID. Removing an in-use target host will not terminate any running processes, though it may result in undefined behaviour within rffmpeg. Before removing a host it is best to ensure there is nothing using it.

Viewing the Logfile

The rffmpeg CLI offers a convenient way to view the log file. Use rffmpeg log to view the entire logfile in the default pager (usually less), or use rffmpeg log -f to follow any new log entries after that point (like tail -0 -f).

Important Considerations

Localhost and Fallback

If one of the configured target hosts is called localhost or 127.0.0.1, rffmpeg will run the ffmpeg/ffprobe commands locally without SSH. This can be useful if the local machine is also a powerful transcoding device, but you still want to offload some transcoding jobs to other machines.

In addition, rffmpeg will fall back to localhost automatically, even if it is not explicitly configured, should it be unable to find any working remote hosts. This helps prevent situations where rffmpeg cannot be run due to none of the remote host(s) being available.

The exact path to the local ffmpeg and ffprobe binaries can be overridden in the configuration, should their paths not match those of the remote system(s).

Hardware Acceleration

Note that if hardware acceleration is configured in the calling application, the exact same hardware acceleration modes must be available on all configured hosts, and, for fallback to work, the local host as well, or the ffmpeg commands will fail.

This is an explicit requirement, and there is no easy way around this without rewriting the passed arguments, which is explicitly out-of-scope for rffmpeg (see the FAQ entry below about mangling arguments).

You should always use a lowest-common-denominator approach when deciding what hardware acceleration option(s) to enable, such that any configured host can run any process, or accept that fallback will not work if all remote hosts are unavailable.

Target Host Selection

When more than one target host is present, rffmpeg uses the following rules to select a target host. These rules are evaluated each time a new rffmpeg alias process is spawned based on the current state (actively running processes, etc.).

1. Any hosts marked bad are ignored.

2. All remaining hosts are iterated through in an indeterminate order (Python dictionary with root key as the host ID). For each host:

   a. If the host is not localhost/127.0.0.1, it is tested to ensure it is reachable (responds to ffmpeg -version over SSH). If it is not reachable, it is marked bad for the duration of this processes' runtime and skipped.

   b. If the host is idle (has no running processes), it is immediately chosen and the iteration stops.

   c. If the host is active (has at least one running process), it is checked against the host with the current fewest number of processes, adjusted for host weight. If it has the fewest, it takes over this role.

3. Once all hosts have been iterated through, at least one host should have been chosen: either the first idle host, or the host with the fewest number of active processes. rffmpeg will then begin running against this host. If no valid target host was found, localhost is used (see section Localhost and Fallback above).

Target Host Weights and Duplicated Target Hosts

When adding a host to rffmpeg, a weight can be specified. Weights are used during the calculation of the fewest number of processes among hosts. The actual number of processes running on the host is floor divided (rounded down to the nearest divisible integer) by the weight to give a "weighted count", which is then used in the determination. This option allows one host to take on more processes than other nodes, as it will be chosen as the "least busy" host more often.

For example, consider two hosts: host1 with weight 1, and host2 with weight 5. host2 would have its actual number of processes floor divided by 5, and thus any number of processes under 5 would count as 0, any number of processes between 5 and 10 would count as 1, and so on, resulting in host2 being chosen over host1 even if it had several processes. Thus, host2 would on average handle 5x more ffmpeg processes than host1 would.

Host weighting is a fairly blunt instrument, and only becomes important when many simultaneous ffmpeg processes/transcodes are occurring at once across at least 2 rem