MetalPetal

<br/> <br/> <br/>

An image processing framework based on Metal.

<!-- TOC depthFrom:2 -->

• Design Overview
  • Goals
  • Core Components
    • MTIContext
    • MTIImage
    • MTIFilter
    • MTIKernel
  • Optimizations
  • Concurrency Considerations
  • Advantages over Core Image
• Builtin Filters
• Example Code
  • Create a MTIImage
  • Apply a Filter
  • Render a MTIImage
  • Display a MTIImage
  • Connect Filters (Swift)
  • Process Video Files
  • Process Live Video (with VideoIO)
• Best Practices
• Build Custom Filter
  • Shader Function Arguments Encoding
  • Simple Single Input / Output Filters
  • Fully Custom Filters
  • Multiple Draw Calls in One Render Pass
  • Custom Vertex Data
  • Custom Processing Module
• Alpha Types
  • Alpha Handling of Built-in Filters
• Color Spaces
  • Color Spaces for Inputs
  • Color Spaces for Outputs
  • Color Spaces for CVPixelBuffer
  • Color Space Conversions
• Extensions
  • Working with SceneKit
  • Working with SpriteKit
  • Working with Core Image
  • Working with JavaScript
  • Texture Loader
• Install
  • CocoaPods
    • Sub-pod Swift
    • Sub-pod AppleSilicon
  • Swift Package Manager
• iOS Simulator Support
• Quick Look Debug Support
• Trivia
• Contribute
• License

<!-- /TOC -->

Design Overview

MetalPetal is an image processing framework based on Metal designed to provide real-time processing for still image and video with easy to use programming interfaces.

This chapter covers the key concepts of MetalPetal, and will help you to get a better understanding of its design, implementation, performance implications and best practices.

Goals

MetalPetal is designed with the following goals in mind.

• Easy to use API

  Provides convenience APIs and avoids common pitfalls.

• Performance

  Use CPU, GPU and memory efficiently.

• Extensibility

  Easy to create custom filters as well as plugin your custom image processing unit.

• Swifty

  Provides a fluid experience for Swift programmers.

Core Components

Some of the core concepts of MetalPetal are very similar to those in Apple's Core Image framework.

MTIContext

Provides an evaluation context for rendering MTIImages. It also stores a lot of caches and state information, so it's more efficient to reuse a context whenever possible.

MTIImage

A MTIImage object is a representation of an image to be processed or produced. It does directly represent image bitmap data instead it has all the information necessary to produce an image or more precisely a MTLTexture. It consists of two parts, a recipe of how to produce the texture (MTIImagePromise) and other information such as how a context caches the image (cachePolicy), and how the texture should be sampled (samplerDescriptor).

MTIFilter

A MTIFilter represents an image processing effect and any parameters that control that effect. It produces a MTIImage object as output. To use a filter, you create a filter object, set its input images and parameters, and then access its output image. Typically, a filter class owns a static kernel (MTIKernel), when you access its outputImage property, it asks the kernel with the input images and parameters to produce an output MTIImage.

MTIKernel

A MTIKernel represents an image processing routine. MTIKernel is responsible for creating the corresponding render or compute pipeline state for the filter, as well as building the MTIImagePromise for a MTIImage.

Optimizations

MetalPetal does a lot of optimizations for you under the hood.

It automatically caches functions, kernel states, sampler states, etc.

It utilizes Metal features like programmable blending, memoryless render targets, resource heaps and metal performance shaders to make the render fast and efficient. On macOS, MetalPetal can also take advantage of the TBDR architecture of Apple silicon.

Before rendering, MetalPetal can look into your image render graph and figure out the minimal number of intermediate textures needed to do the rendering, saving memory, energy and time.

It can also re-organize the image render graph if multiple “recipes” can be concatenated to eliminate redundant render passes. (MTIContext.isRenderGraphOptimizationEnabled)

Concurrency Considerations

MTIImage objects are immutable, which means they can be shared safely among threads.

However, MTIFilter objects are mutable and thus cannot be shared safely among threads.

A MTIContext contains a lot of states and caches. There's a thread-safe mechanism for MTIContext objects, making it safe to share a MTIContext object among threads.

Advantages over Core Image

• Fully customizable vertex and fragment functions.

• MRT (Multiple Render Targets) support.

• Generally better performance. (Detailed benchmark data needed)

Builtin Filters

• Color Matrix

• Color Lookup

  Uses an color lookup table to remap the colors in an image.

• Opacity

• Exposure

• Saturation

• Brightness

• Contrast

• Color Invert

• Vibrance

  Adjusts the saturation of an image while keeping pleasing skin tones.

• RGB Tone Curve

• Blend Modes

  • Normal
  • Multiply
  • Overlay
  • Screen
  • Hard Light
  • Soft Light
  • Darken
  • Lighten
  • Color Dodge
  • Add (Linear Dodge)
  • Color Burn
  • Linear Burn
  • Lighter Color
  • Darker Color
  • Vivid Light
  • Linear Light
  • Pin Light
  • Hard Mix
  • Difference
  • Exclusion
  • Subtract
  • Divide
  • Hue
  • Saturation
  • Color
  • Luminosity
  • ColorLookup512x512
  • Custom Blend Mode

• Blend with Mask

• Transform

• Crop

• Pixellate

• Multilayer Composite

• MPS Convolution

• MPS Gaussian Blur

• MPS Definition

• MPS Sobel

• MPS Unsharp Mask

• MPS Box Blur

• High Pass Skin Smoothing

• CLAHE (Contrast-Limited Adaptive Histogram Equalization)

• Lens Blur (Hexagonal Bokeh Blur)

• Surface Blur

• Bulge Distortion

• Chroma Key Blend

• Color Halftone

• Dot Screen

• Round Corner (Circular/Continuous Curve)

• All Core Image Filters

Example Code

Create a MTIImage

You can create a MTIImage object from nearly any source of image data, including:

• URLs referencing image files to be loaded
• Metal textures
• CoreVideo image or pixel buffers (CVImageBufferRef or CVPixelBufferRef)
• Image bitmap data in memory
• Texture data from a given texture or image asset name
• Core Image CIImage objects
• MDLTexture objects
• SceneKit and SpriteKit scenes

let imageFromCGImage = MTIImage(cgImage: cgImage, isOpaque: true)

let imageFromCIImage = MTIImage(ciImage: ciImage)

let imageFromCoreVideoPixelBuffer = MTIImage(cvPixelBuffer: pixelBuffer, alphaType: .alphaIsOne)

let imageFromContentsOfURL = MTIImage(contentsOf: url)

// unpremultiply alpha if needed
let unpremultipliedAlphaImage = image.unpremultiplyingAlpha()

Apply a Filter

let inputImage = ...

let filter = MTISaturationFilter()
filter.saturation = 0
filter.inputImage = inputImage

let outputImage = filter.outputImage

Render a MTIImage

let options = MTIContextOptions()

guard let device = MTLCreateSystemDefaultDevice(), let context = try? MTIContext(device: device, options: options) else {
    return
}

let image: MTIImage = ...

do {
    try context.render(image, to: pixelBuffer) 
    
    //context.makeCIImage(from: