Kerf1

Source

kerf1: https://github.com/kevinlawler/kerf1/tree/master/src

kerf2 (kerf): https://github.com/kevinlawler/kerf/

What is Kerf?

Kerf is a columnar tick database and time-series language for Linux/OSX/BSD/iOS/Android. It is written in C and natively speaks JSON and SQL. Kerf can be used for trading platforms, feedhandlers, low-latency networking, high-volume analysis of realtime and historical data, logfile processing, and more.

Manual

The manual, currently in progress, is here:

https://github.com/kevinlawler/kerf1/tree/master/manual

The manual covers much more than the whirlwind guide below.

Guides

Kerf Screencast - Stock Basics https://youtu.be/CcJP8TX7CVc
Kerf Screencast - Stock Basics II: Stats https://youtu.be/Pi6FXIYvTkk
Kerf Screencast - CSV & Text Tricks: https://youtu.be/aDETohEScJM
Kerf Screencast - Bloomberg B-PIPE Datafeed: https://youtu.be/eVu50oSBZAE

Time Bars - https://getkerf.wordpress.com/2016/06/21/time-bars/

Whirlwind Language Guide:

TYPES

<pre><code> CHAR "abc" or 'abc' INT 1 FLOAT 2.0 or 1e6 or 1.2E+01 STAMP 2015.03.31 or 01:23:45.877 or 2015.03.31T01:23:45.123 NULL null ARRAY [5, 6, 7, 8] MAP {b:2, c:3, d:4} TABLE {{b:2, c:3, d:4}} </code></pre>

There are a few other types which we'll skip discussing for now.

OPERATIONS

Let's look at some good ways to make arrays. 'Range' comes directly from Python and accepts 1, 2, or 3 arguments:

range(4)
  [0, 1, 2, 3]

range(2, 6)
  [2, 3, 4, 5]

range(0, 20, 3)
  [0, 3, 6, 9, 12, 15, 18]

'Rand' accepts 0, 1, or 2 arguments:

rand()       //FLOAT from [0,1)
  0.164771

rand(5)      //INT
  2

rand(9.0)    //FLOAT
  8.86153

rand(4, 3.0) //4x FLOAT
  [2.44598, 2.87178, 1.14531, 0.676305]

rand(4, [11, 22]) //from a list
  [22, 11, 11, 22]

Rand may return different values for you.

Now let's look ahead:

The exponentiation operator ** comes from Ruby and Python.

range(10**6) //a big list

timing true
sum(range(10**6))  //sum first million numbers
  499999500000

Now back to basic arithmetic:

+ - * / ** plus minus times divide pow
        //wait, which is the operation and which is the name?
2 + 2
plus(2, 2)
        //I guess they both work
2 plus 2
        //and that works too

This is nice because parenthesized prefix notation disambiguates dyadic/binary infix operations.

        //oh, one of those guys

Then the ambiguous-appearing

0.5 * x**2 

becomes

times(1/2, x**2) 
        //well that's not so bad
        //
        //actually, I like that better... I wonder why?

        

Or, emphasizing the center operation:

(divide(1, 2) * pow(x,2))

Neither of these are ambiguous. Of course, you can always fall back to parentheses:

((1 / 2) * (x**2))

Format code as if order-of-operations does not exist.

I find the "functional" notation for arithmetic also helps when the arguments are arrays or maps instead of scalars. This can cue the reader that something heavier-duty is happening.

Other common operators are present. The exclamation point '!' is not, the percent sign '%' is modulo, and so on.

!0
 1
not 0
 1

-33 % 4
  3  //mathematical definition
-33 mod 4
  3

.Math.TAU / 2
  3.14159

JSON

Kerf speaks JSON:

eval('1+1')
  2

a: [[1, 2, 3], {a:"alpha", b:"bravo", c:"3pO"}, null]
match(a, eval(json_from_kerf(a)))
  1

ASSIGNMENT

Assignment is ':', the colon character. It's colon and not '=' because:

1. JSON uses : for assignment, as in {a:1}
2. SQL uses = for comparison, as in WHERE user_id=456
3. Kerf is a superset of both JSON and SQL.

Which looks like

a: [11, 22, 33, 44]
a[2]
    33

We could force assignment to be '=' but I don't think it improves the language.

a[0]:5
a
  [5, 22, 33, 44]

Indexing into maps:

a: {b:2, c:3}
a['c']
  3
{b:2, c:3}['c']
  3

VECTOR OPERATIONS

Arrays vectorize automatically. This means CHAR, INT, FLOAT, and STAMP types are fast and efficient in lists of the same kind. Arrays also mostly keep track of when they're sorted. This means Kerf will invisibly use binary search or interpolation search if it appears advantageous.

The following notion of conformability comes from K:

Adding a single value to a longer list applies it like so: 100 + [0, 10, 20] gives [100, 110, 120]

Kerf extends this notion to work with lists of length 1 as well:

[100] + [0, 10, 20]

gives

  [100, 110, 120]

This also works piecewise:

  [100, 110, 120] + [40, 50, 60]

gives

    [140, 160, 180]

Conformability extends all the way down. This

[[1], [1,1,1]] + [[2], [2, 2, 2]]

gives

  [[3], [3, 3, 3]]

Flatten? Sure

flatten [[3], [3, 3, 3]]
  [3, 3, 3, 3]

Works with maps and tables, too.

{a:2, b:20} + {a:3, b:30, c:100}
  {a:5, b:50, c:100}

Some operations yield array-wise results.

![0, 1, 0]
  [1, 0, 1]

[2, 3, 4, 4, 4] <= [3, 3, 3, 3, 3]
  [1, 1, 0, 0, 0]

Operations are optimized for vectors

timing 1
a: range(10**6)
sum(a)
a+a
min(a)

TABLES

Kerf extends JSON to include the concept of tables. Tables are created just like maps except you use double curly-braces. The names of the keys in that case are instead the names of the columns. So {{a:1, b:2}} is a table and {a:1, b:2} is a map. The convenience constructor {{a,b,c}} also creates a table. This table will have empty arrays for columns. Table columns are always arrays. If you pass something that isn't an array it will be coerced into an array.

The following are all equivalent ways to make a table:

  {{id:1, time:now(), brightness:48.6}}
is the same as 
  {{id:[1], time:[now()], brightness:[48.6]}}
is the same as 
  INSERT INTO {{id, time, brightness}} VALUES [[1], [now()], [48.6]] //bulk insert/append
is the same as
  INSERT INTO {{id, time, brightness}} VALUES {id:1, time:now(), brightness:48.6}   //insert map
is the same as
  INSERT INTO {{id, time, brightness}} VALUES {{id:1, time:now(), brightness:48.6}} //append table
is the same as
  INSERT INTO {{}} VALUES {id:1, time:now(), brightness:48.6} //empty tables are special
is the same as
  a:{{}}
  INSERT INTO a VALUES {id:1, time:now(), brightness:48.6}
is the same as
  id:[1]
  time:[now()]
  brightness:[48.6]
  {{id:id, time:time, brightness:brightness}}

They are all printed as

┌──┬───────────────────────┬──────────┐
│id│time                   │brightness│
├──┼───────────────────────┼──────────┤
│ 1│2015.07.06T16:23:50.509│      48.6│
└──┴───────────────────────┴──────────┘

SQL inserts and updates are forms of assignment: they are always "saved". Bulk inserts are much faster than single inserts. The columns id, time, and brightness are vectorized as INT, STAMP, and FLOAT vectors respectively. The preceding tables all exist in-memory only.

READS/WRITES

You can read and write arbitrary objects, including in-memory tables, using the following functions. These are not really designed for transactional reads and writes, more like per-session reads and writes.

  read_from_path('path.to.file')
  write_to_path('path.to.file', object);
  

For an on-disk table that handles transactional writes, you'll want a mapped object. On Linux, OSX, BSD and other systems, the virtual memory limit for mapped objects is slightly less than 47-bits, or 128T, which for most people is effectively unlimited.

Apple's iOS operating system restricts virtual memory allocations to something less than 2G in size, even on devices with 64-bit pointers. So mapping very large tables will not get far around the memory limitations of the mobile device. Apple really should look into raising it: it may be a legacy restriction from some now-outdated concerns. On OS X the virtual memory limit is effectively unrestricted.

You can open tables on disk via the open_table(filepath) call. Here it is via the Objective-C API:

NSString *path = [[kerf suggestedTableDirectoryPath] stringByAppendingPathComponent:@"my.table"];
[kerf jsonObjectFromCall:@"a: open_table($1)" withArgumentArray:@[path]]
[kerf jsonObjectFromCall:@"insert into a values {{id: 4}}"]);
[kerf jsonObjectFromCall:@"a"]

Modifications to the variable cause the inserts to persist to the disk. They will be there the next time you open the table. Most variables in Kerf use reference counting or copy-on-write to ensure uniqueness. Mapped values like opened tables are different: all reference the same open item. Changes to one affect the other.

CSV/TSV/ETC LOADING

Loading CSVs into tables should be easy. The function for reading a CSV into an in-memory table is read_table_from_csv, and it is used in this way:

read_table_from_csv(csv_file, fields, header_rows)

so that

csv_file: 'my_logs01.csv'
fields: 'SFI'
header_rows: 1
table: read_table_from_csv(csv_file, fields, header_rows)

will load a file that looks like:

Racer, Max Speed, Wins
Mario, 30.01, 10
Luigi, 28.02, 12
Toad,  25.00,  7

as so:

KeRF> t: read_table_from_csv('my_logs01.csv', 'SFI', 1)
┌─────┬─────────┬────┐
│Racer│Max Speed│Wins│
├─────┼─────────┼────┤
│Mario│    30.01│  10│
│Luigi│    28.02│  12│
│ Toad│     25.0│   7│
└─────┴─────────┴────┘

The currently supported list of field identifiers is "IFSEGNZ