c2ffi

Convert a C header .h to a FFI (foreign function interface) .json data structure for the purposes of generating bindings to other languages.

For differences between the other GitHub project under the same name https://github.com/rpav/c2ffi please read this section.

Background: Why?

Problem

When creating applications (especially games) with higher level languages (such as C#, Java, Python), it's sometimes necessary to dip down into C for access to a native library with better raw performance and overall better portability of different low-level APIs accross various platforms. This works great, however, the problem is that maintaining the higher level language bindings by hand becomes time consuming, error-prone, and in some cases quite tricky, especially when the C library changes frequently.

Note that C++ or other low-level languages are not considered as part of the problem scope because they do not align to specific goals. Though, perhaps Zig or some other language may emerge in the future as superior to C for such goals. The goals are the following:

• Portability. For better or worse, C can be used as the industry's standard portable assembler even if technically speaking it is not. Writing a native library in the C language (with some constraints) and building it for multiple targets such as Windows, macOS, Linux, iOS, Android, etc, is the path of least resistance. This is especially true for more non-traditional targets such as RaspberryPi, WebAssembly or even consoles.

• Interopability. The C language, specfically the usage of data structures and functions in limited scope, is a common demonitor between C and higher level languages. This makes interaction between C and and other languages not only correct but as fast and efficient as possible.

• Maintainability. Writing and maintaining a C code project is arguably simpler due to C being a relatively small language in comparison to C++/ObjectiveC. This makes the C language arguably easier to learn and work with, especially if limited in scope such as avoiding the use of function-like macros. This is important for open-source projects (in contrast to proprietary-enterprise-like projects) where one of the barriers to development is knowledge sharing at scale in a decentralized fashion.

Solution

Automate the first step of generating bindings for a higher level language by parsing a cross-platform C .h file using libclang and extracting out the minimal FFI (foreign function interface) data as .json.

Refer to the following 2 graphs as examples for a FFI between C and target language X.

Example diagram: platform specific.

graph LR

    subgraph C library: Linux

    C_HEADER(C header file <br> .h)
    C_SOURCE(C/C++/ObjC source code <br> .c/.cpp/.m)

    C_HEADER --- C_SOURCE

    end

    subgraph c2ffi: extract

    EXTRACT_FFI_LINUX[Extract <br> FFI]

    C_HEADER -.-> EXTRACT_FFI_LINUX

    end

    subgraph Artifacts: native library

    C_COMPILED_LINUX(compiled C code <br> .so)

    end

    subgraph Artifacts: target-platform

    C_HEADER -.-> C_COMPILED_LINUX
    C_SOURCE -.-> C_COMPILED_LINUX

    PLATFORM_FFI_LINUX(platform FFI <br> .json)

    EXTRACT_FFI_LINUX -.-> PLATFORM_FFI_LINUX

    end

    subgraph Your bindgen tool

    PLATFORM_FFI_LINUX --> X_CODE_GENERATOR[X language code <br> generator]

    end

    subgraph Your app

    C_COMPILED_LINUX === X_SOURCE
    X_CODE_GENERATOR -.-> X_SOURCE(X language source code)

    end

Example diagram: cross-platform.

graph LR

    subgraph C library

    C_HEADER(C header file <br> .h)
    C_SOURCE(C/C++/ObjC source code <br> .c/.cpp/.m)

    C_HEADER --- C_SOURCE

    end

    subgraph c2ffi: extract

    EXTRACT_FFI_WINDOWS[Extract <br> FFI]
    EXTRACT_FFI_MACOS[Extract <br> FFI]
    EXTRACT_FFI_LINUX[Extract <br> FFI]

    C_HEADER -.-> |Windows| EXTRACT_FFI_WINDOWS
    C_HEADER -.-> |macOS| EXTRACT_FFI_MACOS
    C_HEADER -.-> |Linux| EXTRACT_FFI_LINUX

    end

    subgraph Native library

    C_COMPILED_WINDOWS(compiled C code <br> .dll)
    C_COMPILED_MACOS(compiled C code <br> .dylib)
    C_COMPILED_LINUX(compiled C code <br> .so)

    end

    subgraph Artifacts: target-platform
    
    C_HEADER -.-> |Windows| C_COMPILED_WINDOWS
    C_SOURCE -.-> |Windows| C_COMPILED_WINDOWS

    C_HEADER -.-> |macOS| C_COMPILED_MACOS
    C_SOURCE -.-> |macOS| C_COMPILED_MACOS

    C_HEADER -.-> |Linux| C_COMPILED_LINUX
    C_SOURCE -.-> |Linux| C_COMPILED_LINUX

    PLATFORM_FFI_WINDOWS(platform FFI <br> .json)
    PLATFORM_FFI_MACOS(platform FFI <br> .json)
    PLATFORM_FFI_LINUX(platform FFI <br> .json)

    EXTRACT_FFI_WINDOWS -.-> |Windows| PLATFORM_FFI_WINDOWS
    EXTRACT_FFI_MACOS -.-> |macOS| PLATFORM_FFI_MACOS
    EXTRACT_FFI_LINUX -.-> |Linux| PLATFORM_FFI_LINUX

    end

    subgraph c2ffi: merge

    MERGE_FFI["Merge platform FFIs to a cross-platform FFI"]

    PLATFORM_FFI_WINDOWS -.-> |Any OS| MERGE_FFI
    PLATFORM_FFI_MACOS -.-> |Any OS| MERGE_FFI
    PLATFORM_FFI_LINUX -.-> |Any OS| MERGE_FFI

    end

    subgraph Artifacts: cross-platform

    CROSS_FFI(Cross-platform FFI <br> .json)

    MERGE_FFI -.-> CROSS_FFI

    end

    subgraph Your bindgen tool

    CROSS_FFI --> X_CODE_GENERATOR[X language code <br> generator]

    end

    subgraph Your app

    C_COMPILED_WINDOWS === |Windows| X_SOURCE
    C_COMPILED_MACOS === |macoS| X_SOURCE
    C_COMPILED_LINUX === |Linux| X_SOURCE
    X_CODE_GENERATOR -.-> X_SOURCE(X language source code)

    end

Differences between https://github.com/rpav/c2ffi

I originally had this project named as something different but then re-wrote it with tests under the name c2ffi as that accurately describes the project. Unfortunately it has the same name as another project (https://github.com/rpav/c2ffi) with similar goals. If someone has a better name I am open to suggestions. Perhaps c2ffix where the x is for cross-platform?

This project is different in the following ways:

• This project is licensed under MIT. The other project is licensed under GPL2.
• This project is written in C# and interacts with libclang over C interopability, the other project is written in C++. Additionally, this project has a C# library via a NuGet package that contains the code for serializing and deserializing the model to/from .json.
• This project only supports C. The other one apparently supports C++, ObjC, etc.
• This project fully supports macros objects by parsing via C++ using auto.
• This project is intended to be used for generating a cross-platform FFI. Specific things which break portability such as variadic functions and bit-fields are not supported in this project by design. This project (extract step) outputs a .json file for each target platform (clang target triple), then this program (merge step) merges these platform specific .json files into a cross-platform .json and checks if it is indeed cross-platform. If it failed at the merge step then there is likely something wrong with the C code which makes it not portable between one or more target platforms. The other project does not check for cross-platform and is rather left upon the developer.
• This project supports various options for configuring extract and merge steps including skipping C declarations by using regular expression matching.
• This project also includes a brain dump of things which one should do and do not in C for interoperability (see next section).
• This project is used directly by another project c2cs to generate C# bindings.

Limitations: Is my C library FFI ready?

c2ffi does not work for every C library. This is due to some technical limitations where some usages of C for cross-platform foreign function interface (FFI) are not appropriate. Everything in the external linkage of the C API is subject to the following list for being "FFI Ready". Think of it as the check list to creating a cross-platform C library for usage by other languages.

Note that the internals of the C library is irrelevant and to which this list does not apply. It is then possible to use C++/ObjectiveC behind a implementation file (.cpp or .m respectively) or reference C++/ObjectiveC from a C implementation file (.c); all that c2ffi needs is the C header file (.h).

|Supported|Description| |:-:|-| |✅|Variable externs ^{1, 3, 7}| |✅|Function externs ^{1, 3, 7}| |✅|Function prototypes (a.k.a., function pointers.) ^{3, 7}| |✅|Enums ³| |✅|Structs ^{2, 4, 7}| |✅|Unions ^{2, 4, 7}| |✅|Opaque types. ^{2, 7}| |✅|Typedefs (a.k.a, type aliases) ^{2, 7}| |❌|Function-like macros ⁵| |✅|Object-like macros ^{2, 6, 7}|

¹: When declaring your external functions or variables, do set the default visibility explictly. This is necessary because c2ffi is configured to have the visibiity set to hidden when using libclang via the flag -fvisibility=hidden so that only the strict subset of functions and variables intended for FFI are extracted. Most C libraries will have an API_DECL macro object defined which can be redefined to also set the visibility. See ffi_helper.h for an example in C. You can also use the config .json file to define your API_DECL macro object.

Bad

#if defined(WIN32) || defined(_WIN32) || defined(__WIN32__)
    #define MY_API_DECL __declspec(dllexport) // no visibility explictly set when using Clang, thus visibility is 'hidden'
#else
    #define MY_API_DECL extern // no visibility explictly set, thus visibility is 'hidden'
#endif
...
MY_API_DECL c