Transcoder

pytranscoder

Python wrapper for ffmpeg for batch, concurrent, or clustered transcoding using defined profiles and optional rules engine for automation.

Read The Docs

News

Sadly this project has reached end of life. It will remain here but no further development will happen unless there is some compelling need. Instead, a new project called wandarr has been created from the foundation of pytranscoder. It is far simpler and doesn't try to be "all things to all people". Gone are the rules, profiles, and mixins. Instead there are simple templates. Clustering is still supported, but simplified and now supporting multiple type of hardware acceleration on the same machine (something pytranscoder could not do)

What it is

This script is intended to help automate transcoding for people encoding lots of video. It is more than a wrapper - it is a workflow and job manager.

There are 2 modes: local and clustered. Local mode is the most common usage and is for running this script on the same machine where it is installed. Cluster mode turns pytranscoder into a remote encoding manager. In this mode it delegates and manages encode jobs running on multiple hosts. This requires more advanced configuration and is documented separately in Cluster.md

The remainder of this document focuses on using pytranscoder in local mode.

Features:

• Sequential or concurrent transcoding.
• Concurrent mode allows you to make maximum use of your nVidia CUDA-enabled graphics card or Intel accelerated video (QSV)
• Preserves all streams but allows for filtering by audio and subtitle language.
• Configurable transcoding profiles
• Configurable rules and criteria to auto-match a video file to a transcoding profile
• Mixins are profile fragments you can provide at runtime to dynmically change your selected profiles.
• Transcode from a list of files (queue) or all on the command line
• Cluster mode allows use of other machines See Cluster.md for details.
• On-the-fly compression monitoring and optional early job termination if not compressing as expected.
• Experimental support for HandBrakeCLI

Requirements

• Linux or MacOS, Windows 10. For Windows, WSL (Ubuntu) recommended.
• latest ffmpeg (3.4.3-2 or higher, lower versions may still work)
• nVidia graphics card with latest nVidia CUDA drivers (optional)
• Intel CPU with QSV enabled (optional)
• Python 3 (3.7 or higher)

Support

Please log issues or questions via the github home page for now.

Installation

There are a few possible ways to install a python app - one of these should work for you.

Linux The confusion is due to the fact that not all distributions or OS's install pip3 by default. Either way, pytranscoder is available in the pypi repo.

pip3 install --user pytranscoder-ffmpeg
# or...
python3 -m pip install --user pytranscoder-ffmpeg 

Windows (WSL - Ubuntu) Windows Subsystem for Linux is the best option, but requires a couple of maintenance steps first if you don't have pip3:

sudo apt update
sudo apt upgrade
sudo install python3-pip

# now we can install
pip3 install --user pytranscoder-ffmpeg

At this point you have a choice - either install ffmpeg for Windows ffmpeg.exe or install in bash as an Ubuntu package. Either will work but there are caveats, or you could install both and not worry.

• ffmpeg.exe can be run in Windows command shell or from bash but requires special attention when configuring pytranscoder paths.
• ffmpeg apt package can only be run from bash but is a more natural Linux path mapping.

After installing you will find this document and others in $HOME/.local/shared/doc/pytranscoder (on Linux/MacOS) and in $HOME/AppData/Python/pythonversion/shared/doc/pytranscoder** (on Windows). Also available online

Upgrading

Whatever method above for installing works for you, just use the --upgrade option to update, ie: pip3 install --upgrade pytranscoder-ffmpeg

Operation - Profiles and Rules

A profile is a named group of ffmpeg commandline options to encode a specific way. You can define all the combinations you use regularly in transcode.yml for easy selection later. At least 1 profile definition is required.

A rule is a YAML syntax of predefined predicates to allow simple matching on source media details and relate to a specific profile. For example, if you transcode 720p differently than 1080p, and still different than 4k you can set up rules to match those 3 resolutions to a specific transcode profile. Easy - let the script do the work of selecting the right ffmpeg options.

But you aren't required to use rules. You can specify the profile name on the commandline each run using the -p option. Or you can define 1 rule that acts as a default (see examples). It's up to you. But using rules is a great way to automate a tedious manual workflow.

When changing or adding profiles and rules it is useful to test them out by running in --dry-run mode first, which will show you everything that would happen if running for real.

Configuration

There is a supplied sample transcode.yml config file, or you can download it here. This can be customized all you like, however be sure to preserve the YAML formatting. Either specify this file on the commandline with the -y option or copy it to your home directory as .transcode.yml (default)

There are 3 sections in the file:

config - Global configuration information

Sample

config:
  default_queue_file:   '/path/to/default/list/of/files/if/none/given'
  ffmpeg:               '/usr/bin/ffmpeg'       # path to ffmpeg for this config
  ssh:                '/usr/bin/ssh'    # used only in cluster mode (optional)
  queues:
    qsv:                1                   # sequential encodes
    cuda:               2                   # maximum of 2 encodes at a time
  colorize:             yes

| setting | purpose | | ----------- | ----------- | | default_queue_file | A queue file is just a text file listing out all the media you want to encode. It is not required, but useful when automating a workflow. You can always indicate a queue file on the command line. This just sets the default, if any. | | ffmpeg | Full path to ffmpeg on this host | | ssh | Full path to ssh on this host | | queues | If using concurrency, define your queues here. The queue name is whatever you want. Each name specifies a maximum number of concurrent encoding jobs. If none defined, a default sequential queue is used. | | colorize | optional, defaults to "no". If "yes" terminal output will have some color added |

profiles - Transcoding profiles (ffmpeg options)

Profiles are used to provide ffmpeg with various options for encoding. One profile definition is required, but mostly likely you will define multiples for different encoding scenarios. The name of the profile can be provided on the command line to select the appropriate one for your needs. Alternatively, you can define rules (see below) to auto-match media with profiles for a less manual encoding workflow.

Sample:

profiles:

  # some common, reusable settings to keep things tidy
  common:
    output_options_subtitles:
      - "-c:s copy"
       - "-f matroska"
    output_options_audio:
      - "-c:a copy"
    output_options_video:
      - "-crf 20"
    extension: '.mkv'
    threshold: 20
    threshold_check: 30

  #
  # Sample Intel QSV transcode setup (note to customize options for your environment)
  #
  hevc_qsv:
    include: common
    output_options_video:         # mixin-enabled section - overrides common
      - "-c:v hevc_qsv"
      - "-preset medium"
      - "-qp 21"
      - "-b:v 7M"
    
  #
  # Sample nVidia transcode setup
  #

  hevc_cuda:                  # profile name
      include: common
      input_options:          # ffmpeg input options
        - "-hwaccel cuvid"    # REQUIRED for CUDA
        - "-c:v h264_cuvid"   # hardware decoding too
      output_options_video:   # mixen-enabled - overrides common
        - "-c:v hevc_nvenc"   # REQUIRED for CUDA
        - "-profile:v main"
        - "-preset medium"
      output_options_audio:
        - "-c:a copy"
      
      queue: cuda		# manage this encode in the 'cuda' queue defined globally
      
      # optionally you can filter out audio/subtitle tracks you don't need
      audio:
         exclude_languages:
             - "chi"
             - "spa"
             - "fre"
             - "ger"
         default_language: eng
 
      subtitle:
         exclude_languages:
             - "chi"
             - "spa"
             - "fre"
             - "por"
             - "ger"
             - "jpn"
         default_language: eng
  #
  # This is a mixin, a profile fragment that can be provided on the command line
  # to alter a profile (ie. pytranscoder -p hevc_cuda -m mp3_hq file.mp4)

  mp3_hq:
    output_options_audio:
      - "-c:a mp3lame"
      - "-b:a 384k"

  x264:                            # profile name
      include: common
      input_options: 
      output_options:
        - "-c:v x264"
        
h264_cuda_anime:
    include: common
    input_options:
    output_options:
      - "-c:v h264_nvenc"
      - "-tune animation"
 

| setting | purpose | | ----------- | ----------- | | input_options | ffmpeg options related to the input (see ffmpeg docs) | | output_options | ffmpeg options related to the output (see ffmpeg docs). Use for generic options, not mixin-enabled | | output_options_video |