Goldflake

Generator to generate Bigint ID or unique key for 174 years (Approx 200k unique Id/sec/machine on 8k distributed machines at once) inspired by Twitter's Snowflake and Sonyflake

Goldflake is a distributed unique ID generator inspired by Twitter's Snowflake & Sonyflake

Goal:

To become application & database oriented unique bigint id generator for lifetime without compromising performance.

Why Goldflake id instead of Sonyflake?

1. Sonyflake generator is good, if you are generating all Ids at application end.
2. Sonyflake has a limit of 25K id/sec on single node, which seems low in today's scenarios when we are talking about cloud & databases can be created with high system configurations easily.
3. I have tried & tested Sonyflake bits configuration on Postgres-12 on my machine (Dell XPS, Intel® Core™ i7-9750H Processor 6 Core 12 Threads, 16GB RAM, SSD) & observed duplicate key conflict, which is a very basic configuration for Databases now a days.
4. Goldflake is usnig 11 bits for sequence instead of 8 (as used by Sonyflake) which gives more unique ID generation on databases as well.

So it has a different bit assignment from Snowflake & Sonyflake. A Goldflake ID is composed of

39 bits for time in units of 10 msec                      // Similar to Sonyflake
13 bits for a machine id or app instance id or shard id   // Balance between Snowflake and Sonyflake (If you can live with 8192 machines)
11 bits for a sequence number                             // To generate 2^11=2048 unique ids per 10 millisecond or 204800 keys/sec on one node

For PostgreSQL -https://github.com/AmreeshTyagi/goldflake-pg

Comparison chart

| | Twitter Snowflake | Sonyflake | Goldflake | |-------------------------------------------------- |------------------------- |------------------------------------------------------------- |------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Lifetime to generate unique id on single machine | 69 | 174 | 174 | | No. of distributed machines | 1024 | 65536 | 8192 | | No. of unique IDs per 10 milliseconds | 40960 | 256 | 2048 | | Max No. of unique IDs per second on single node | 4096000 = 4096k | 25600 = 25.6k | 204800 = 204.8k | | Max No. of unique IDs per second on all nodes | 4194304000 | 1677721600 | 1677721600 | | Database oriented | Yes | No | Yes | | Application oriented | Yes | Yes | Yes | | Comment | Less number of machines | DB will return duplicate soon in case of heavy concurrency | 50% application generated IDs & 50% database generated IDs makes it a best fit to use it on app server and db server. I believe 4096 database shards are enough to handle potential large load. Though it depends on solution design as well. |