We Moved!

This project has been renamed and moved to <a href=https://github.com/AsterTrack/AsterTrack>AsterTrack</a> after a long period of development. <br> This repository will remain only as an archive, as nearly every part of the project changed.

<p align="center"> <img alt="Hardware Prototype of MakersVR" src="Documentation/Media/Banner_AT_Baked.png" width="100%"/> </p>

MakersVR

Welcome to MakersVR, an open source VR hardware project focused around a cheap optical tracking system. <br> The goal of this project is to create a cheap but capable tracking system that competes with existing commercial tracking systems like SteamVR Lighthouse, has the flexibility of a professional optical tracking system found in motion capture studios, but is cheap and easy to build. </br> It uses multi-cameras to triangulate individual small LEDs in space, so it is very easy to develop custom tracked DIY hardware, e.g. gun stocks, by adding LEDs on them in an arbitrary pattern, and it's also possible to develop fingertip tracking. </br> Since the concept of such a system is decades old and still used in professional MoCap installations, although at a much higher price point, it could also integrate into existing MoCap focused solutions such as marker-based facial tracking and full body solvers. </br> Standard 3-Point (or more) Full Body Tracking for VR is of course possible by default, and will likely be the most accurate next to SteamVR trackers, but by far not the cheapest solution for FBT.

How does it work?

The basis is an optical tracking systems with two or more cameras observing multiple active markers made with LEDs. <br> The <a href="https://ar-tracking.com/technology/technical-details/">ART system</a> is quite similar and provides a great overview of the general idea. <br> Each tracking camera, called <i>Marker Detector</i>, is a Raspberry Pi Zero plus camera. It is executing a Blob Detection algorithm to get the LEDs position in camera space. These are send over a serial connection to a central microcontroller, the <i>Marker Tracker</i>. This serial connection is a CAT6 cable, but is currently used for power, camera sync and UART (might be switched to SPI later). <br> The centralized <i>Marker Tracker</i> is currently a STM32F103 microcontroller, the cheap Blue Pill board. It is connected to up to three <i>Marker Detectors</i> at once as well as the Host PC over a USB connection. It collects the Blob2D data from each <i>Marker Detector</i> and currently immediately sends it to the Host PC using a isochronous Full-Speed USB 2.0 connection. It also times the frame of each <i>Marker Detector</i> down to 0.1ms and sends synchronization feedback back over the serial connection. <br> On the Host PC, the <i>Configurator</i> program (and later the driver) connects to the USB device and 2D Blobs are streamed in. These 2D Blobs are converted to 3D rays using the position of the cameras in the room and the poses of the markers are inferred. This is still in debate on how exactly this will work, there are several approaches that I will be trying out. <br> In the Documentation you can find <a href="https://github.com/Seneral/MakersVR/tree/master/Documentation/Design">more detailed design documents</a>, alongside a <a href="Documentation/PreliminaryPartsList.ods">preliminary parts list</a>. This places the currently expected costs at below 200€ for a recommended system (3 cameras, 6 tracking targets) and 62€ for the absolute cheapest (2 cheap cameras, 3 tracking targets).

Current State

I currently have a two-camera setup working as a prototype, and it works with my calibration marker prototype and tracking marker prototype. However I have not had luck with accurate intrinsic calibration yet, so the accuracy is not that great yet. Also since the cameras are not synced yet, quick movements disturb the marker greatly (and there's not filtering yet, either). <br> Here's one of the <i>Marker Detector</i> with the <i>Marker Tracker</i>:

<p align="center"> <img alt="Hardware Prototype of MakersVR" src="Documentation/Media/HW_Prototype_Annotated.jpg" width="50%"/> <br> The <i>Marker Detector</i> and <i>Marker Tracker</i> prototypes </p> The actual tracking system on the host PC is functionally done and has been developed and successfully tested in testing environments. However it might need some adjustments to work practically. <br> The <i>Configurator</i> software currently contains a full testing mode and a device mode which connects to the actual hardware. In both modes you can do intrinsic calibration and extrinsic&room calibration phases using the calibration marker, as well as marker calibration and tracking phases using a tracking marker. <br> <p align="center"> <img alt="Software Configurator of MakersVR" src="Documentation/Media/SW_Configurator_Annotated.png" width="60%"/> <br> The <i>Configurator</i> streaming blobs from the Prototype Hardware </p> The marker detection in the multi-camera marker tracking works similar to other professional tracking systems, with each 3D point identified based on the relations to its neighbours within the marker. This means markers can be easily read in using a calibration phase. <br> <p align="center"> <img alt="Multi-Camera Tracking" src="Documentation/Media/SW_Multi-Camera-Tracking.gif" width="40%"/> <br> Simulated marker tracking of a WMR-controller-like tracker. </p> Any marker that has LEDs with (relatively) unique distances works fine. But a cheap and easy to produce marker is a huge challenge. The WMR-like ring simulated above is way too expensive for home production in low quantities, so a more traditional design is used, a base supporting several sticks with LEDs at the tip (see the <a href="https://ar-tracking.com/products/markers-targets/targets/">ART targets</a> to get an idea). The base is going to be 3D-printed and houses power distribution (1S LiPo), and can be adapted to support many different marker layouts - however the current prototype is made out of polymorph plastic, which allows for even quicker prototyping, <a href="Documentation/Media/HW_Marker_PolymorphPlastic.jpg">as seen here</a>. The arms are made out of polypropylene straws, as they can survive quite a beating (for when you inevitably punch the walls) and can be easily replaced. The LED tips need to be spherical and well diffused, the current design is using a 3mm flat-top LED surrounded by sanded hot glue: <br> <p align="center"> <img alt="Marker Prototype" src="Documentation/Media/HW_Marker_Prototype.jpg" width="71.875%"/> <br> Marker Prototype powered by a 1S LiPo. Proportions and positioning can vary. </p> One hardware problem I'm still facing is the camera selection. The ecosystem of cameras for the Raspberry Pi is very limited, with the official modules having terribly low field of view, resulting in a smaller tracking volume, and most inofficial modules basing on the older camera module with much worse performance or a completely different chip alltogether with questionable support. Currently, I'm seeing if the Official Camera Module V2 with limited Field of View is viable: <br> <p align="center"> <img alt="Software Configurator of MakersVR" src="Documentation/Media/HW_CameraTrackingVolume.png" width="60%"/> <br> Tracking volume of three Marker Detectors using the Camera Module V2 in a small room </p>

Going Forward

Although having a functional second prototype, it is not ready for use yet, and there's a lot to improve still: <br>

1. Develop alternate continuous calibration algorithm in hopes of achieving better accuracy while being easier for the user
2. Implement camera sync feedback (else fast movements are inaccurate)
3. Add additional CPU blob detection pass to increase accuracy
4. Add support for multiple properly tracked markers (currently simple frame-by-frame detection)
5. Adapt SteamVR driver (SerialHMD) to general tracker use for FBT
6. Improve QPU backend on the <i>Marker Detector</i> to improve latency (can be reduced by up to 65% still)
7. Develop power-management electronics to keep LiPo safe and develop PCB
8. Develop final marker design(s), create 3D-print base
9. Create a (3D-printed?) case for the electronics
10. Add support for a more powerful microcontroller as the MarkerTracker: </br> Support more cameras at once, support chaining with more controllers, do onboard tracking, send of the final tracking data over bluetooth to standalone devices or over SPI port for embedded applications, all without requiring a host PC.

Along the way, depending on interest, I will expand the documentation on this project. <br>

Rebuilding

Although the software is not finalized yet, the first prototype can be build in its current state using just a soldering iron and it will stay the same for the forseeable future. All of the accuracy and performance improvements are purely software work. The required hardware components (with some parameters to adjust to budget) can be found <a href="Documentation/PreliminaryPartsList.ods">here</a>. If there's interest I'll add detailed instructions how to wire everything and tips for creating the calibration marker and tracking marker. <br> The software already has all required instructions for compiling in the respective subfolders, especially the testing mode of the <i>Configurator</i> software can be used without hardware.

Hardware

The Controller (MarkerTracker) is just a simple MicroController that doesn't do much right now, the code is only running on an STM32F103 right now though. </br> For each camera (MarkerDetector), only a Raspberry Pi Zero (or another model) is required, along with a MIPI camera. Currently, all standard camera modules are supported (V1 and V2), as well as all their derivatives, each with different drawbacks. Technically any monochromatic camera can be used, assuming a I2C driver implementation, as the RAW 8bit data from them is directly usable without an ISP at all. The <a href="