What is VRS?

VRS is a file format optimized to record & playback streams of sensor data, such as images, audio samples, and any other discrete sensors (IMU, temperature, etc), stored in per-device streams of time-stamped records.

VRS was first created to record images and sensor data from early prototypes of the Quest device, to develop the device’s positional tracking system now known as Insight, and Quest's hand tracking software. It is also the file format used by the Aria glasses.

<div align="center"> <table align="center" border="0" cellspacing="0" cellpadding="0"> <tr> <td align="right"><b>C++</b>&nbsp;&nbsp;</td> <td> <a href="https://github.com/facebookresearch/vrs/releases"><img alt="Latest Release" src="https://img.shields.io/github/v/release/facebookresearch/vrs.svg" /></a> <a href="https://github.com/facebookresearch/vrs/blob/main/LICENSE"><img alt="License" src="https://img.shields.io/badge/License-Apache--2.0-blue.svg"/></a> </td> </tr> <tr> <td align="right"><b>Conda</b>&nbsp;&nbsp;</td> <td> <a href="https://anaconda.org/conda-forge/vrs"><img src="https://anaconda.org/conda-forge/vrs/badges/version.svg" alt="Conda Version" /></a> <a href="https://anaconda.org/conda-forge/vrs"><img src="https://anaconda.org/conda-forge/vrs/badges/downloads.svg" alt="Conda Downloads" /></a> </td> </tr> <tr> <td align="right"><b>Python</b>&nbsp;&nbsp;</td> <td> <a href="https://pepy.tech/project/vrs"><img alt="Python Downloads" src="https://pepy.tech/badge/vrs"></a> </td> </tr> </table> </div>

Main features

• VRS files contain multiple streams of time-sorted records generated by a set of sensors(camera, IMU, thermometer, GPS, etc), typically one set of sensors per stream.
• The file and each stream contain an independent set of tags, which are string name/value pairs that describe them.
• Streams may contain Configuration, State and Data records, each with a timestamp in a common time domain for the whole file.
  Typically, streams contain with one Configuration and one State record, followed one to millions of Data records.
• Records are structured as a succession of typed content blocks.
  Typical content blocks are metadata, image, audio and custom content blocks.
• Metadata content blocks contain raw sensor data described once per stream, making the file format very efficient. The marginal cost of adding 1 byte of data to each metadata content block of a stream is 1 byte per record (or less, when lossless compression happens).
• Records can be losslessly compressed using lz4 or zstd, which can be fast enough to compress while recording on device.
• Multiple threads can create records concurrently for the same file, without CPU contention.
• VRS supports huge file size (tested with multi terabytes use cases).
• VRS supports chunked files: auto-chunking on creation, automated chunk detection for playback.
• Playback is optimized for timestamp order, which is key for network streaming.
• Random-access playback is supported.
• Custom FileHandler implementations can add support for cloud storage streaming.

Documentation

The VRS documentation explains how VRS works. It is complemented by the API documentation, while the sample code and the sample apps below demonstrate in code how to use the API.

We plan on having a VRS Users group dedicated on discussing VRS usage. Stay tuned for details.

Getting Started

To work with VRS files, the vrs open source project provides a C++ library with external open source dependencies such as boost, fmt, lz4, zstd, xxhash, and googletest for unit tests. To build & run VRS, you’ll need a C++17 compiler, such as a recent enough version of clang or Visual Studio.

The simplest way to build VRS is to install the libraries on your system using some package system, such as Brew on macOS, or apt on Ubuntu, and then use cmake to build & test. VRS supports many other platforms such as Windows, Android, iOS and other flavors of Linux, but we currently only provide instructions for macOS, Ubuntu and Windows. You can also build VRS in a container or isolated environment and avoid installing any library on your system.

Instructions (macOS and Ubuntu and container)

Install build tools & libraries (macOS)

• install Brew, following the instruction on Brew's web site.
• install tools & libraries:

  brew install cmake git ninja ccache boost fmt libpng jpeg jpeg-turbo
  brew install lz4 zstd xxhash glog googletest eigen
  brew install qt5 portaudio pybind11 opus wget ninja
  brew install node doxygen
  

Install build tools & libraries (Ubuntu)

These instructions are validated using Ubuntu 20.04, whereas Ubuntu 18.04 doesn't install recent enough versions of cmake, fmt, lz4, and zstd, and is therefore not supported.

• install tools & libraries:

  sudo apt-get install cmake git ninja-build ccache libgtest-dev libfmt-dev libjpeg-dev libturbojpeg-dev libpng-dev
  sudo apt-get install liblz4-dev libzstd-dev libxxhash-dev nasm yasm libeigen3-dev
  sudo apt-get install libboost-dev
  sudo apt-get install qtbase5-dev portaudio19-dev libopus-dev
  sudo apt-get install npm doxygen
  

Optional: GPU-accelerated H.265 decoding (Linux x86_64, NVIDIA)

XPRS can decode H.265 on NVIDIA GPUs via NVDEC. This requires the header-only nv-codec-headers at build time (no CUDA toolkit needed; the driver's libcuda/libnvcuvid are loaded at runtime):

git clone --branch n12.1.14.0 https://github.com/FFmpeg/nv-codec-headers.git
sudo make -C nv-codec-headers install PREFIX=/usr

Then configure with -DBUILD_WITH_XPRS=ON -DENABLE_NVCODEC=ON. The resulting library is CUDA-agnostic: it auto-accelerates color (YUV420) H.265 streams when an NVIDIA driver is present and falls back to CPU decoding otherwise (monochrome streams always use CPU, as NVDEC does not support them).

Build & run (macOS & Linux)

• Run cmake:

cmake -S <path_to_vrs_folder> -B <path_to_build_folder> -G Ninja

If you want to build vrsplayer, you need to specify where your installation of Qt is. Where Qt is depends on how you've installed it, using a package manager such as Brew or APT, or downloading it directly from Qt's official website.

To tell cmake where to find Qt, you can either add -DCMAKE_PREFIX_PATH=<path_to_qt> to the cmake command above, or set the environment variable QT_DIR=<path_to_qt> to point to your Qt installation (same path). As a sanity check, you should be able to find the qmake tool at <path_to_qt>/bin/qmake.

Note: We ran into strange build issues when Qt5 and Qt6 were both installed at the same time via Brew on macOS, but uninstalling either fixed the problem.

Qt 5 or Qt 6

At this time, vrsplayer is mostly tested using Qt 5.15.3 LTS, but the code has been updated to build and run with Qt 6.3.0. However, testing with Qt 6 was pretty superficial.

• Build everything & run tests:

cd <path_to_build_folder>
ninja all
ctest -j8

• To include VRS in your cmake project:

cd <path_to_build_folder>
ninja install # install VRS on your system as a library cmake can find

In your cmake project (probably one your project's CMakeLists.txt files):

find_package(vrslib REQUIRED) # find the vrs package, break if not found

add_executable(your_app your_app.cpp) # that's your app
target_link_libraries(your_app vrs::vrslib) # so your app can use the vrs includes and libraries

You can then use VRS in your your_app.cpp code:

#include <vrs/RecordFileReader.h>

int main() {
  vrs::RecordFileReader reader;
  if (reader.openFile("myfile.vrs") == 0) {
    do something...
  }
  return 0;
}

Build & run with H.265 (HEVC) decoding support (macOS & Linux)

VRS can now decode H.265 (HEVC) streams in the reader when built with the XPRS feature. To enable it:

1. Install FFmpeg (decoding-only build) From the repo root, run:

   ./build_third_party_libs/build_ffmpeg_linuxunix.sh
   

   This will fetch and compile FFmpeg into ~/vrs_third_party_libs/ffmpeg/. Remove that folder and rerun the script if you ever need to reinstall or update.

2. Configure & build VRS with XPRS

   cmake -S . -B build -G Ninja -DBUILD_WITH_XPRS=ON
   cd build
   ninja all
   

   With -DBUILD_WITH_XPRS=ON, the VRS reader links against your custom FFmpeg build and gains full H.265 decoding support.

3. Use the new feature Any VRS file containing HEVC streams will now be readable by both the vrs CLI tool and vrsplayer visualizer if you have QT installed.

Note:

• XPRS-based decoding is currently not supported on Pixi or Windows—if you hit any issues, please check that your FFmpeg build completed without errors and open an issue on the repo.
• By enabling H.265 (HEVC) decoding feature, the project depends on FFmpeg, which is licensed under the Lesser General Public License (LGPL) version 2.1 or later. You can obtain the source code of FFmpeg, matching the version used in this library, by visiting the official FFmpeg website: https://github.com/FFmpeg/FFmpeg/archive/refs/tags/n7.1.tar.gz.

Container build & Usage

• Build VRS in a container and use it on your local data:

cd <path_to_vrs_folder>
podman/docker build . -t vrs
podman/docker run -it --volume <your_local_da