Source2 Schema Dumper

This metamod-source plugin provides capabilities to dump Source2 schema definitions from CS2, Deadlock and Dota2 in a json/kv3 format for later processing in various ways.

Usage

To perform schema dump run dump_schema command when server has started and initialized (calling this too early on startup might be unavailable). By default running this command without any extra args would dump base level schema definitions.

This command supports the following arguments:

• help: Prints help message with up to date flags.
• verbose: Provides verbose output of the dump process, mostly useful for debugging.
• as_json: Dumps to a json file (Default).
• as_kv3: Dumps to a kv3 file.
• metatags: Dump metatags.
• atomics: Dump atomics.
• pulse_bindings: Dump pulse bindings.
• module_metadata: Dump module metadata. Only windows is currently supported!
• split_atomics: Splits templated atomic names and leaves only base name leaving templated stuff. (Makes CUtlVector<int> to be named as CUtlVector for example).
• ignore_parents: Ignores parent scope decls and removes inlined structs/classes converting them from A::B to A__B.
• apply_netvar_overrides: Applies netvar overrides to types (MNetworkVarTypeOverride metatags).
• all: Shorthand version of providing metatags, atomics, pulse_bindings and module_metadata flags.
• for_cpp: Use optimal flags for cpp generation later, similar to providing split_atomics, ignore_parents and apply_netvar_overrides flags.

[!NOTE] Pulse bindings are heavily under development by valve, so these are expected to break with each engine update in the supported game list, and would require manual update to the code most likely!

Example usage:

• dump_schema metatags pulse_bindings: Would dump pulse_bindings and general schema information with metatags.
• dump_schema all for_cpp: Would provide best result for later cpp generation as well as dumps everything it can.

Generator scripts

This plugin comes with a set of particular python generator scripts located in the root of a plugin folder (addons/schemadump/) that accept JSON schema dumps:

• generate_cpp.py: Generates fully cpp compatible definitions out of dumped schema. This script is mainly useful for later usage for idaclang or similar stuff, it also supports supplying hl2sdk definitions of common utl structs.

  It supports the following args:

  • -i --input: The path to the JSON schema file or a folder containing schema files, in which case the newest file would be processed. Default is ./dumps/ dir.
  • -o --output: The path to the output C++ file or a directory. Default is ./generated/ dir.
  • -s --silent: Disables stdout output.
  • -c --comments: Generate help comments for resulting class/enum definitions.
  • -g --generate-classes: A list of class/enum definitions to generate. Use "all" to generate all classes (Default).
  • -a --static-assert: Generate static assertions of resulting class/enum definitions to ensure their validity.
  • -d --supply-hl2sdk: Supplies hl2sdk class/enum definitions if applicable to the generated file.
  • -p --preferred-project: Prefer server or client project for generation. (Could only be server or client). Default is server.

[!NOTE] Schema dumps with parent scope, might not generate correct code without supplying hl2sdk definitions!

• generate_cpp_defs.py: Example script to generate s2ze compatible class definitions out of dumped schema.

  It supports the following args:

  • -i --input: The path to the JSON schema file or a folder containing schema files, in which case the newest file would be processed. Default is ./dumps/ dir.
  • -o --output: The path to the output C++ file or a directory. Default is ./generated/ dir.
  • -s --silent: Disables stdout output.
  • -c --comments: Generate help comments for resulting class/enum definitions.
  • -g --generate-classes: A list of class/enum definitions to generate. Doesn't default to "all" and expects you to provide classes to generate.
  • -p --preferred-project: Prefer server or client project for generation. (Could only be server or client). Default is server.

[!NOTE] This script is mostly an example of how you can automate dumping schema to macro headers for plugins that use s2ze alike schema systems.

• generate_pulse_bindings.py: Generates source2 pulse bindings. Mostly for reference use. Requires pulse bindings to be dumped first in resulting dump.

  It supports the following args:

  • -i --input: Path to JSON schema file or a folder containing schema files, in which case the newest file would be processed. Default is ./dumps/ dir.
  • -o --output: Path to output C++ file or a directory. Default is ./generated/ dir.
  • -s --silent: Disables stdout output.
  • -n --no-comments: Don't generate help comments for resulting domain definitions.
  • -g --generate-domains: A list of domain definitions to generate. Use "all" to generate all domains (Default).

IDAClang usage example

One of the main advantages of this plugin and its scripts is that it allows for a full idaclang compilation support! But it requires a small setup beforehand if you are new to idaclang and only starting to setting up, some key notes to know:

• This guide was written with windows in mind, it should be possible to do the same for linux binaries, but be aware that schema differs per platform, so you'd need a linux dump first, as well as some corrections to compile arguments!
• Only versions of ida 7.7 and above are supported, as that was the first version where idaclang was introduced;
• Make sure compiler used by ida is clang, Options->Compiler locate Source parser, should be set to clang. To parse c++ file, use File->Load file->Parse C header file;
• Compile arguments:
  • Target platform, make sure you are targetting correct platform (example: -target x86_64-pc-windows-msvc19.16.27045);
  • C++ version, 17 and above should be supported, but was tested on c++17 (example: -std=c++17);
  • Some hl2sdk code expects certain defines set, notably: X64BITS, PLATFORM_64BITS (example: -DX64BITS -DPLATFORM_64BITS);
  • There are few other handy defines and arguments I'd suggest using (especially for hl2sdk support), so here's the complete argument string for reference that was used during test (ida 7.7 but should work for further versions as well):

    -x c++ -std=c++17 -Wno-c++11-narrowing -target x86_64-pc-windows-msvc19.16.27045 -D_CRT_SECURE_NO_DEPRECATE -D_CRT_SECURE_NO_WARNINGS -D_CRT_NONSTDC_NO_DEPRECATE -D_GLIBCXX_DEBUG=0 -D_ITERATOR_DEBUG_LEVEL=0 -DDEBUG -D_DEBUG -DWIN32 -D_WINDOWS -DCOMPILER_MSVC -DCOMPILER_MSVC64 -DX64BITS -DPLATFORM_64BITS -D_WIN32=1 -D_MSC_VER=1900 -D__clang__ -D__clang_major__=17 -D_CRT_USE_BUILTIN_OFFSETOF=1
    

• Include paths:
  • Generally for non-hl2sdk build you should be fine with whatever idaclang has already, but if you want to have hl2sdk to build correctly, here's the paths list you need to have for it to locate everything correctly:
    • {HL2SDKPATH}\public
    • {HL2SDKPATH}\game\shared
    • {HL2SDKPATH}\game\server
    • {HL2SDKPATH}\public\tier0
    • {HL2SDKPATH}\public\tier1
    • {HL2SDKPATH}\public\mathlib
    • {HL2SDKPATH}\public\entity2
    • {HL2SDKPATH}\public\engine
    • {HL2SDKPATH}\public\game\server
    • {HL2SDKPATH}\thirdparty\protobuf-3.21.8\src
    • {HL2SDKPATH}\common
    • Replace {HL2SDKPATH} with an absolute path to hl2sdk for a game you are targetting (like hl2sdk-cs2);
• If you want to preserve arguments and include paths between binaries, you can edit {IDAPATH}/cfg/idaclang.cfg in there edit CLANG_ARGV, where include directories could be supplied with -I argument (example -ID:\\hl2sdk-cs2\\public);

Some gotchas & ABI differences

Since idaclang uses clang under the hood to compile c++, it would mismatch on some stuff with MSVC, you can see what is supported here. But here's the general gist of what you'd most likely encounter:

• Member function pointers are of a different size (16 on clang, 8 on MSVC) which would produce different sized structs. That's not used in schema, but is used in hl2sdk if you plan on not only limiting with schema (Notable cases in hl2sdk: CUtlDelegate, CUtlBuffer);
• Inlined structs/classes of templated classes might not be picked up correctly by idaclang and might be missing, example:

  template <typename T>
  class TemplateTest
  {
      struct SubStruct
      {
          T m_Val;
      };
      
      SubStruct *m_Dummy;
  };
  
  TemplateTest<int> testvar;
  

  in this example, TemplateTest<int>::SubStruct would be only be declared but not defined in ida's type database, producing incomplete results. There's no global fix for this (if you know any lmk!!!), you can only fix it locally per template instance by providing explicit template specialization of said type which would compile everything there's for it, example fix for the above case would be adding this before any usages of said template specialization:

  template class TemplateTest<int>;