VideoContext
An experimental HTML5 & WebGL video composition and rendering API.
Install / Use
/learn @bbc/VideoContextREADME
VideoContext
The VideoContext is an experimental HTML5/WebGL media processing and sequencing library for creating interactive and responsive videos on the web.
It consists of two main components. A graph based, shader accelerated processing pipeline, and a media playback sequencing timeline.
The design is heavily inspired by the Web Audio API, so it should feel familiar for people with experience in the Web Audio world.
Live examples can be found here
Table of Contents
- Demo
- Debugging
- Documentation
- Node Types
- Writing Custom Effect Definitions
- Advanced Examples
- Development
Demo
View on CodeSandbox.
<!DOCTYPE html>
<html>
<head>
<title></title>
<script type="text/javascript" src="../dist/videocontext.js"></script>
</head>
<body>
<!--
A canvas needs to define its width and height to know how many pixels you can draw onto it.
Its CSS width and height will define the space it takes on screen
If omitted, the canvas dimensions will be 300x150 and your videos will not rendered at their
optimal definition
https://webglfundamentals.org/webgl/lessons/webgl-resizing-the-canvas.html
-->
<canvas id="canvas" width="1280" height="720" style="width: 852px; height: 480px"></canvas>
<script type="text/javascript">
window.onload = function(){
var canvas = document.getElementById("canvas");
var videoCtx = new VideoContext(canvas);
var videoNode1 = videoCtx.video("./video1.mp4");
videoNode1.start(0);
videoNode1.stop(4);
var videoNode2 = videoCtx.video("./video2.mp4");
videoNode2.start(2);
videoNode2.stop(6);
var crossFade = videoCtx.transition(VideoContext.DEFINITIONS.CROSSFADE);
crossFade.transition(2,4,0.0,1.0, "mix");
videoNode1.connect(crossFade);
videoNode2.connect(crossFade);
crossFade.connect(videoCtx.destination);
videoCtx.play();
};
</script>
</body>
</html>
Debugging
If you need to debug video context graphs or get a better insight into what is happening under the hood, there's a new browser extension for chrome, videocontext-devtools
Documentation
API Documentation can be built using ESDoc by running the following commands:
yarn install
yarn doc
The documentation will be generated in the "./doc" folder of the repository.
Node Types
There are a number of different types of nodes which can be used in the VideoContext's processing graph. Here's a quick list of each one. Following that is a more in-depth discussion of each type.
- VideoNode - Plays video.
- AudioNode - Plays audio.
- ImageNode - Displays images for specified time.
- CanvasNode - Displays output of canvas for specified time.
- EffectNode - Applies shader to limited number of inputs.
- TransitionNode - Applies shader to limited number of inputs. Modifies properties at specific times.
- CompositingNode - Applies same shader to unlimited inputs, rendering to same output.
- DestinationNode - Node representing output canvas. Can only be one.
VideoNode
A video source node.
View on CodeSandbox.
var videoNode = videoCtx.video("./video1.mp4");
videoNode.connect(videoCtx.destination);
videoNode.start(0);
videoNode.stop(4);
For best results the video played by a VideoNode should be encoded with a fast decode profile. The following avconv line shows how this can be achieved.
avconv -i input.mp4 -tune fastdecode -strict experimental output.mp4
AudioNode
An audio source node.
View on CodeSandbox.
var audioNode = videoCtx.audio("./audio.mp3");
audioNode.connect(videoCtx.destination);
audioNode.start(0);
audioNode.stop(4);
ImageNode
An image source node.
View on CodeSandbox.
var imageNode = videoCtx.image("cats.png");
imageNode.connect(videoCtx.destination);
imageNode.start(0);
imageNode.stop(4);
CanvasNode
A canvas source node.
View on CodeSandbox.
var canvas = document.getElementById("input-canvas");
var canvasNode = videoCtx.canvas(canvas);
canvasNode.connect(videoCtx.destination);
canvasNode.start(0);
canvasNode.stop(4);
CustomSourceNode
Sometimes using the pre-built node is just not enough. You can create your own source nodes that host more logic and let you hook into the VideoContext Node API easily.
The following example shows how to create a custom source node that can play a HLS VOD:
View on CodeSandbox
import Hls from "hls.js";
class HLSNode extends VideoContext.NODES.VideoNode {
constructor(src, gl, renderGraph, currentTime, playbackRate, sourceOffset, preloadTime, hlsOptions = {}) {
//Create a video element.
const video = document.createElement("video");
super(video, gl, renderGraph, currentTime, playbackRate, sourceOffset, preloadTime);
//Create a HLS object.
this.hls = new Hls(hlsOptions);
//Bind the video element.
this.hls.attachMedia(video);
//Set the source path.
this._src = src;
this._displayName = "HLSNode";
this._elementType = "hls";
}
_load() {
//Load the video source on first load.
if (!this._loadTriggered) {
this.hls.loadSource(this._src);
}
super._load();
}
destroy() {
if (this.hls) {
this.hls.destroy();
}
super.destroy();
}
}
//Setup the video context.
const canvas = document.getElementById("canvas");
const ctx = new VideoContext(canvas);
//Create a custom HLS source node and play it for 60 seconds.
const hlsNode = ctx.customSourceNode(HLSNode, "https://video-dev.github.io/streams/x36xhzz/x36xhzz.m3u8");
hlsNode.start(0);
hlsNode.stop(60);
//Set-up the processing chain.
hlsNode.connect(ctx.destination);
//start playback.
ctx.play();
Another use case for custom node types would be to play GIFs. The custom node would be in charge of decode the GIF frames and paint them on a canvas depending on the _update calls from VideoContext.
EffectNode
An EffectNode is the simplest form of processing node. It's built from a definition object, which is a combination of fragment shader code, vertex shader code, input descriptions, and property descriptions. There are a number of common operations available as node descriptions accessible as static properties on the VideoContext at VideoContext.DEFINITIONS.
The vertex and shader code is GLSL code which gets compiled to produce the shader program. The input description tells the VideoContext how many ports there are to connect to and the name of the image associated with the port within the shader code. Inputs are always render-able textures (i.e images, videos, canvases). The property descriptions tell the VideoContext what controls to attached to the EffectNode and the name, type, and default value of the control within the shader code.
The following is a an example of a simple shader description used to describe a monochrome effect. It has one input (the image to be processed) and two modifiable properties to control the color RGB mix for the processing result.
View on CodeSandbox.
var monochromeDescription = {
title:"Monochrome",
description: "Change images to a single chroma (e.g can be used to make a black & white filter). Input color mix and output color mix can be adjusted.",
vertexShader : `
attribute vec2 a_position;
attribute vec2 a_texCoord;
varying vec2 v_texCoord;
void main() {
gl_Position = vec4(vec2(2.0,2.0)*a_position-vec2(1.0, 1.0), 0.0, 1.0);
v_texCoord = a_texCoord;
}`,
fragmentShader : `
precision mediump float;
uniform sampler2D u_image;
uniform vec3 inputMix;
uniform vec3 outputMix;
varying vec2 v_texCoord;
varying float v_mix;
void main(){
vec4 color = texture2D(u_image, v_texCoord);
float mono = color[0]*inputMix[0] + color[1]*inputMix[1] + color[2]*inputMix[2];
color[0] = mono * outputMix[0];
color[1] = mono * outputMix[1];
color[2] = mono * outputMix[2];
gl_FragColor = color;
}`,
properties:{
"inputMix":{type:"uniform", value:[0.4,0.6,0.2]},
"outputMix":{type:"uniform", value:[1.0,1.0,1.0]}
},
inputs:["u_image"]
};
Here's an example of how the above node description might be used to apply sepia like effect to a video.
//Setup the video context.
var canvas = document.getElementById("canvas");
var ctx = new
