MediaPipeUnityPlugin

Unity plugin to run MediaPipe

Generate Convert Improve

Install / Use

/learn @homuler/MediaPipeUnityPlugin

About this skill

Quality Score

0/100

README

MediaPipe Unity Plugin

This is a Unity (>= 2022.3) Native Plugin to use MediaPipe (0.10.22).

The goal of this project is to port the MediaPipe API (C++) one by one to C# so that it can be called from Unity.
This approach may sacrifice performance when you need to call multiple APIs in a loop, but it gives you the flexibility to use MediaPipe instead.

With this plugin, you can

Write MediaPipe code in C#.
Run MediaPipe's official solution on Unity.
Run your custom Calculator and CalculatorGraph on Unity.
- :warning: Depending on the type of input/output, you may need to write C++ code.

:smile_cat: Hello World!

Here is a Hello World! example.
Compare it with the official code!

using Mediapipe;
using UnityEngine;

public sealed class HelloWorld : MonoBehaviour
{
    private const string _ConfigText = @"
input_stream: ""in""
output_stream: ""out""
node {
  calculator: ""PassThroughCalculator""
  input_stream: ""in""
  output_stream: ""out1""
}
node {
  calculator: ""PassThroughCalculator""
  input_stream: ""out1""
  output_stream: ""out""
}
";

    private void Start()
    {
        using var graph = new CalculatorGraph(_ConfigText);
        using var poller = graph.AddOutputStreamPoller<string>("out");
        graph.StartRun();

        for (var i = 0; i < 10; i++)
        {
            graph.AddPacketToInputStream("in", Packet.CreateStringAt("Hello World!", i));
        }

        graph.CloseInputStream("in");
        var packet = new Packet<string>();

        while (poller.Next(packet))
        {
            Debug.Log(packet.Get());
        }
        graph.WaitUntilDone();
    }
}

For more detailed usage, see the API Overview page or the tutorials.

:hammer_and_wrench: Installation

Please first download the pre-built package from the releases page.

| file | contents | | ------------------------------------- | ------------------------------------------------------------------------ | | MediaPipeUnityPlugin-all.zip | All the source code with required libraries. | | MediaPipeUnityPlugin-all-stripped.zip | Same as MediaPipeUnityPlugin-all.zip but the symbols are stripped. | | com.github.homuler.mediapipe-*.tgz | A tarball package | | MediaPipeUnityPlugin.*.unitypackage | A .unitypackage file |

If you need to run sample scenes on your mobile devices, prefer MediaPipeUnityPlugin-all.zip or MediaPipeUnityPlugin-all-stripped.zip.
To run sample scenes on your mobile devices, you need to place required models properly, but most required setup is already done in MediaPipeUnityPlugin-all.zip.

Build the plugin by yourself

:warning: In most cases, you don't need to build the plugin by yourself. Only if the pre-built package doesn't work for you, please build the plugin by yourself.

This repository doesn't include required libraries or models, so if you clone this repository, you need to build the plugin by yourself.
See the build guide for more details.

Build a package by yourself

If you want, you can also build the plugin by yourself using MediaPipeUnityPlugin-all(-stripped).zip.

Build a unity package

Open this project
Click Tools > Export Unitypackage

MediaPipeUnity.[version].unitypackage file will be created at the project root.

Build a local tarball file

Install npm command
Build a tarball file

cd Packages/com.github.homuler.mediapipe
npm pack
# com.github.homuler.mediapipe-[version].tgz will be created

mv com.github.homuler.mediapipe-[version].tgz your/favorite/path

Supported Platforms

:warning: GPU mode is not supported on macOS and Windows.

[^1]: Tested on Arch Linux. [^2]: Running MediaPipe on Windows is experimental.

Supported Solutions

This plugin implements the following MediaPipe Tasks C# APIs.

cf. The official available solutions

Legacy Solutions

You can also use MediaPipe Framework, which allows you to run Legacy Solutions. However, please note that support for these solutions has ended.

:book: Usage

Once you've downloaded the pre-built package, please import the plugin into your project.

For Android

:skull_and_crossbones: If you need to build your app for Android, please ensure you include libstdc++_shared.so in your APK[^3], otherwise DllNotFoundException will be thrown at runtime.

[^3]: mediapipe_android.aar contains libopencv_java4.so and it depends on libstdc++_shared.so. However, some project or plugins may already include libstdc++_shared.so, so we don't include libstdc++_shared.so in mediapipe_android.aar.

The easiest way to include libstdc++_shared.so in your APK is to place it in the Assets/Plugins/Android directory of your project.

You can also include libstdc++_shared.so at build time by adding the following code to your mainTemplate.gradle file, and the sample project is using this method.

// Include libc++_shared.so
task copyLibcppShared {
    doLast {
        def ndkDir = android.ndkDirectory
        def abiFilters = an

Related Skills

node-connect

335.2k

Diagnose OpenClaw node connection and pairing failures for Android, iOS, and macOS companion apps

frontend-design

82.5k

Create distinctive, production-grade frontend interfaces with high design quality. Use this skill when the user asks to build web components, pages, or applications. Generates creative, polished code that avoids generic AI aesthetics.

openai-whisper-api

335.2k

Transcribe audio via OpenAI Audio Transcriptions API (Whisper).

commit-push-pr

82.5k

Commit, push, and open a PR