greenpack news, February 2025

Partial generics support

2025 February 26: In v0.530.0 I've added more support for generics. Previously we had simply skipped any generic struct. Now we process generic structs, and serialize any non-generic fields. Generic (type parameterized) fields will require reflection to serialize them, and so they are skipped.

• work-around in the absence of full generics support

I may add automatic reflection support for generic fields in the future, but for now there is a pretty easy, manual, work-around.

The pre and post serialization hooks can be used to manually use reflection and serialize your generic fields into a reserved []byte field that you add to your structs for this purpose.

Update 2025-March-24, v0.535.0: Ouch! Finding the generic instantiations requires a whole lot more (slow!) code analysis, and it significantly slows down greenpack on codebases that don't need it. I've turned this part of generics support off for now. Full(er) generics support can be re-activated with the -generics=true flag, but the default is now false. So, while the non-generic fields of a generic struct are still serialized, and pre/post load hooks still called, without -generics=true we (a) don't attempt to locate instantiations; we (b) don't automagically write test code for the first instantiation; and (c) we skip serializing generic fields (those with type parameters) in generic structs. Via the hooks, you can of course still implement the workaround described just above.

Automatic pointer Dedup off by default for now.

2025 January 12: In v0.529.0 I have turned off pointer dedup by default because the dedup Encode/Decode was producing some errors that I have not had time to track down. For now you'll want to manually dedup pointers. Marshal/Unmarshal has never supported pointer dedup so it was never affected. The env var GREENPACK_DEDUP can be set (to true) to reverse the default (have dedup on again) under go generate. It also accepts false but that is the default now.

December 2024: SQL support.

As of v0.520.0, adding -sql mariadb to the usual go generate annoation, e.g. //go:generate greenpack -sql mariadb will cause greenpack to generate methods for writing to, and reading from, MariaDB. See https://github.com/glycerine/greenpack/blob/master/testdata/_sqldemo_gen.go for an example of the generated code.

November 2024: two new things:

1. I compare greenpack to CBOR here: https://github.com/glycerine/demo_cbor Long story short: CBOR is 3-4x slower than greenpack, and comes in too many flavors.

2. I wrote an RPC system with modern cryptography and greenpack for serialization. Its pretty nice. https://github.com/glycerine/rpc25519 performs better than the other systems measured. It has better latency and throughput -- even when encrypted -- than gRPC and rpcx.

October 2024: Go-module compatibility.

Since the versioning tags here were created long before Go modules were invented, it turns out the schemes were not compatible. Greenpack was at v5 but the import path is never going to have /v5 at the end, so... we'll try renaming the tags to v0.xxx.0 style tags, and see what happens.

There is also https://github.com/glycerine/greenpack2 which I know works (it is simply a clone with renamed tags and a new module path) because it has never been published with a higher version before.

But I'm going to try to make the orignial greenpack (no 2) also work, because I import it in a ton of places.

Cool: it seems to work. The only thing is that the idea of the "@latest" go module isn't going to work, because Google's proxy remembers the old v5.1.2 version that never supported modules. Since the v5.1.2 will always look like the "latest" even though its been renamed to v0.512.0, you have to manually use the v0.xxx.0 versions. Ironically, they are actually the latest.

For reference purposes in case I need to go back to it, list.of.old.tags.txt list.of.rename.operations.txt document the old tags and how they were renamed (or many were deleted if not renamed).

The true latest as of this README is v0.515.0.

April 2019: time.Duration

Version 5.0.9 includes native support for time.Duration serialization.

December 2017: interfaces

Version 5.0.4 includes two new features:

• a) interfaces are supported, and automatically detected

(Since the parser didn't used to distinguish interfaces from structs, manual annotation of interface types was required, using the msgp:",iface" tag. This has now been fixed. However the manual tagging of interfaces is still available for cases when you are using an inteface from another package whose source we haven't parsed.)

Required if using interfaces: your container must implement the ConcreteFactory interface; the method NewValueAsInterface. See the example/test https://github.com/glycerine/greenpack/blob/master/_generated/def.go#L341 and https://github.com/glycerine/greenpack/blob/master/_generated/def.go#L345 .

• b) de-duplication of pointers and interfaces allows serialization of repeated/shared pointers efficiently.

NB: de-duplication is only available for EncodeMsg / DecodeMsg based on a Writer/Reader stream. The MarshalMsg/UnmarshalMsg API doesn't have a place to store the deduplication state, so at the moment de-dup from a []byte isn't supported. This isn't a huge limitation as it is trivial to turn a []byte into a stream if need be. For example, use w := msgp.NewWriter(bytes.NewBuffer(by)) to get a writer that wraps the by []byte.

greenpack: a serialization convention for msgpack2; adds field versioning and type annotation.

greenpack is a simple convention for naming fields in msgpack data: we take the original field name and append a version number and basic type indicator.

• See also the fork/cousin project https://github.com/glycerine/truepack if you encode/decode naked ints (not inside a struct) and want type info preserved.

the main idea

//given this definition, defined in Go:
type A struct {
  Name     string      `zid:"0"`
  Bday     time.Time   `zid:"1"`
  Phone    string      `zid:"2"`
  Sibs     int         `zid:"3"`
  GPA      float64     `zid:"4"`
  Friend   bool        `zid:"5"`
}

then when greenpack serializes, the it looks like msgpack2 on the wire with extended field names:
         
greenpack
--------              
a := A{               
  "Name_zid00_str"  :  "Atlanta",
  "Bday_zid01_tim"  :  tm("1990-12-20"),
  "Phone_zid02_str" :  "650-555-1212",  
  "Sibs_zid03_i64"  :  3,               
  "GPA_zid04_f64"   :  3.95,            
  "Friend_zid05_boo":  true,            
}

Notice the only thing that changed with respect to the msgpack2 encoding is that the the fieldnames have been extended to contain a version and a type clue.

msgpack2 [https://github.com/msgpack/msgpack/blob/master/spec.md] [http://msgpack.org] enjoys wide cross-language support, and provides efficient and self-contained data serialization. We find only two problems with msgpack2: weak support for data evolution, and insufficiently strong typing of integers.

The greenpack format addresses these problems while keeping serialized data fully self-describing. Greenpack is independent of any external schema, but as an optimization uses the Go source file itself as a schema to maintain current versioning and type information. Dynamic languages still have an easy time reading greenpack--it is just msgpack2. There's no need to worry about locating the schema under which data was written, as data stays self-contained.

The central idea of greenpack: start with msgpack2, and append version numbers and type clues to the end of the field names when stored on the wire. We say type "clues" because the type information clarifies the original size and signed-ness of the type, which adds the missing detail to integers needed to fully reconstruct the original data from the serialization. This address the problem that commonly msgpack2 implementations ignore the spec and encode numbers using the smallest unsigned type possible, which corrupts the original type information and can induce decoding errors for large and negative numbers.

If you've ever had your msgpack crash your server because you tried to change the type of a field but keep the same name, then you know how fragile msgpack can be. The type clue fixes that.

The version zid number gives us the ability to evolve our data without crashes. The moniker zid reveals greenpacks evolution from zebrapack, where it stood for "zebrapack version id". Rather than rework all the tooling to expect gid, which might be confused with a GUID, we simply keep the convention. zid indicates the field version.

An additional advantage of the zid numbering is that it makes the serialization consistent and reproducible, since greenpack writes fields in zid order.

One last easy idea: use the Go language struct definition syntax as our serialization schema. There is no need to invent a completely different format. Serialization for Go developers should be almost trivially easy. While we are focused on a serialization format for Go, because other language can read msgpack2, they can also readily read the data. While the schema is optional, greenpack (this repo) provides code generation tools based on the schema (Go file) that generates extremely fast serialization code.

the need for stronger integer typing

Starting point: msgpack2 is great. It is has an easy to read spec, it defines a compact serialization format, and it has wide language support from both dynamic and compiled languages.

Nonetheless, data update conflicts still happen and can be hard to resolve. Encoders could use the guidance from type clues to avoid signed versus unsigned integer encodings.

For instance, sadly the widely emulated C-encoder for msgpack chooses to encode signed positive integers as unsigned integ