Log-structured data store

Design Doc

Requirements

• Rotatable
• Revertable
• Commit pipeline
• High concurrency
• Crash-safe
• Friendly interface, easy to integrate

RFC

TODO

Example

Used as a WAL driver

The full outline of wal commit pipeline operation is as follows:

                           |--------------- Guarded ------------|   |------------- Concurrent ------------|
  Insert                   |<------------- Preparing ---------->|   |<------------ Committing ------------|
 ---------> OnRequest ---> Alloc LSN ---> Prepare insert buffer ---> Fill insert buffer --> Wait sync done ----->  Return response
                                      |                         |                                            |
                                      |                         |                                            |
                                       -> Prepare log to WAL ==>                                             |(Sync done)
                                                                |                                            |
                                                                |                                            |
WalCommitter: --- Commit ---> Commit ===> Commit ---> Commit ===> Commit logging to WAL ---> Commit ---> ... |
                                        |                       |                               |            |
                                        |                       |                               |            |
WalSyncer   :                    ----- Sync -----------------> Sync -------------------------> Sync ------------------> ...

Used as a metadata-like storage engine

TODO

Used as a columnar storage database storage engine

TODO