HoloLens 2 Sensor Streaming

HoloLens 2 server software and Python client library for streaming sensor data via TCP. Created to stream HoloLens data in real time over WiFi to a Linux machine for research purposes but also works on Windows and macOS. The server is offered as a standalone application (appxbundle) or as a plugin (dll) compatible with Unity, Unreal, and native UWP applications.

Supported interfaces

Research Mode Visible Light Cameras (640x480 @ 30 FPS, Grayscale, H264 or HEVC encoded)
- Left Front
- Left Left
- Right Front
- Right Right
Research Mode Depth
- AHAT (512x512 @ 45 FPS, 16-bit Depth + 16-bit AB, H264 or HEVC encoded or Lossless* Zdepth for Depth)
- Long Throw (320x288 @ 5 FPS, 16-bit Depth + 16-bit AB, PNG encoded)
Research Mode IMU
- Accelerometer (m/s^2)
- Gyroscope (deg/s)
- Magnetometer
Front Camera (1920x1080 @ 30 FPS, RGB, H264 or HEVC encoded)
Microphone (2 channels @ 48000 Hz, 16-bit PCM, AAC encoded or 5 channels @ 48000 Hz, 32-bit Float, RAW)
Spatial Input (30 Hz)
- Head Tracking
- Eye Tracking
- Hand Tracking
Spatial Mapping (3D Meshes)
Scene Understanding (3D Meshes + Semantic labels for planar surfaces)
Voice Input
Extended Eye Tracking (30, 60, or 90 FPS)
Extended Audio (Microphone + Application audio, RAW or AAC encoded)
- Internal Microphone mirror
- External USB-C Microphone
Extended Video (H264 or HEVC encoded)
- Internal Front Camera mirror
- External USB-C Camera
Extended Depth (Lossless* Zdepth encoded)
- External USB-C Depth Camera
Device Portal Mixed Reality Capture (H264/AAC encoded)
- Front Camera (+ Holograms)
- Microphone (+ Application audio)

Additional features

Download calibration data (e.g., camera intrinsics, extrinsics, undistort maps) for the Front Camera and Research Mode sensors (except RM IMU Magnetometer).
Optional per-frame pose for the Front Camera and Research Mode sensors.
Support for Mixed Reality Capture (Holograms in Front Camera video).
Support for Shared capture for Front Camera and Extended Video.
Client can configure the bitrate and properties of the H264, HEVC, and AAC encoded streams.
Client can configure the resolution and framerate of the Front Camera. See here for a list of supported configurations.
Client can configure the focus, white balance, and exposure of the Front Camera [example].
Frame timestamps can be converted to Windows FILETIME (UTC) for external synchronization [example].
Client can exchange messages with a Unity, Unreal, or native UWP application using the plugin [send][receive].
Server application can run in background (alongside other applications) when running in flat mode [example].

Integrations

Subprojects

hl2da plugin: Access sensor data from Unity, Unreal, and native UWP applications running on the HoloLens 2.
hl2av plugin: Simple H264 encoder and decoder for HoloLens 2.
mrtk_remote_ui: Unity/MRTK framework for creating and controlling simple User Interfaces on HoloLens 2 remotely from Python code.
MARCuS: Interactive Rubik's Cube Solver for HoloLens 2 using Unity and MRTK.
hl2flo: Create optical flow datasets using HoloLens 2.

Technical Report

Our paper provides an overview of the code, features, and examples for the first released version of the application (1.0.11.0). For newer versions, please refer to the examples in the viewer directory. If hl2ss is useful for your research, please cite our report:

@article{dibene2022hololens,
  title={HoloLens 2 Sensor Streaming},
  author={Dibene, Juan C and Dunn, Enrique},
  journal={arXiv preprint arXiv:2211.02648},
  year={2022}
}

Preparation

Before using the server software configure your HoloLens as follows:

Update your HoloLens: Settings -> Update & Security -> Windows Update.
Enable developer mode: Settings -> Update & Security -> For developers -> Use developer features.
Enable device portal: Settings -> Update & Security -> For developers -> Device Portal.
Enable research mode: Refer to the Enabling Research Mode section in HoloLens Research Mode.

Please note that enabling Research Mode on the HoloLens increases battery usage.

Installation (sideloading)

The server application is distributed as a single appxbundle file and can be installed using one of the two following methods.

Method 1 (local)

On your HoloLens, open Microsoft Edge and navigate to this repository.
Download the latest appxbundle.
Open the appxbundle and tap Install.

Method 2 (remote)

Download the latest appxbundle.
Go to the Device Portal and navigate to Views -> Apps. Under Deploy apps, select Local Storage, click Browse, and select the appxbundle.
Click Install, wait for the installation to complete, then click Done.

You can find the server application (hl2ss) in the All apps list.

Permissions

The first time the server runs it will ask for the necessary permissions to access sensor data. If there are any issues please verify that the server application (hl2ss.exe) has access to:

Camera (Settings -> Privacy -> Camera).
Eye tracker (Settings -> Privacy -> Eye tracker).
Microphone (Settings -> Privacy -> Microphone).
User movements (Settings -> Privacy -> User movements).

Python client

The Python scripts in the viewer directory demonstrate how to connect to the server, receive the data, unpack it, and decode it in real time. Additional samples show how to associate data from multiple streams. Run the server on your HoloLens and set the host variable of the Python scripts to your HoloLens IP address.

Interfaces

RM VLC: viewer/client_stream_rm_vlc.py
RM Depth AHAT: viewer/client_stream_rm_depth_ahat.py
RM Depth Long Throw: viewer/client_stream_rm_depth_longthrow.py
RM IMU: viewer/client_stream_rm_imu.py
Front Camera: viewer/client_stream_pv.py
Microphone (2 channels): viewer/client_stream_microphone.py
Microphone Array (5 channels): viewer/client_stream_microphone_array.py
Spatial Input: viewer/client_stream_si.py
Remote Configuration: viewer/client_ipc_rc.py
Spatial Mapping: viewer/client_ipc_sm.py
Scene Understanding: viewer/client_ipc_su.py
Voice Input: viewer/client_ipc_vi.py
Unity Message Queue: viewer/client_ipc_umq.py (plugin only)
Extended Eye Tracking: viewer/client_stream_eet.py
Extended Audio: viewer/client_stream_extended_audio.py
Extended Video: viewer/client_stream_extended_video.py
Guest Message Queue: viewer/client_ipc_gmq.py (plugin only)
Extended Depth: viewer/client_stream_extended_depth.py
Device Portal MRC: viewer/client_stream_dp_mrc.py (hl2ss server optional)

Required packages

OpenCV pip install opencv-python
PyAV pip install av
NumPy pip install numpy
Requests pip install requests

Optional packages

pynput pip install pynput
Open3D pip install open3d
PyAudio pip install PyAudio

TouchDesigner client

Provides set of TouchDesigner components for receiving and working with hl2ss data in real time. Main scene with all components and samples can be found in the hl2ss_td directory. Core component (hl2ss_core) is basically a wrapper around existing Python client - enabling its usage within TouchDesigner. User can easily build customized setup by using multiple hl2ss_core components, where each of them provides single data stream.

Before using the component make sure to create venv in hl2ss_td directory and install packages from requirements.txt using the same Python version as used by your TouchDesigner installation.

Requires TouchDesigner 2023.11880+ (tested on Windows).

Unity plugin

For streaming sensor data from a Unity application. A sample Unity project (2020.3.42f1) can be found in the

Hl2ss

Install / Use

README