gorqlite - a Go client for rqlite

gorqlite is a Go client for rqlite that provides easy-to-use abstractions for working with the rqlite API.

It provides an idiomatic API specialized for rqlite and a database/sql driver (read below for more information on it). The main API provides similar semantics to database/sql, such as Open(), Query() and QueryOne(), Next()/Scan()/Map(), Write() and WriteOne(), etc.

If you're looking for a simpler Go client library, which provides a thin, easy-to-use and more robust abstraction for accessing rqlite, check out https://github.com/rqlite/rqlite-go-http.

Status

This client library is used in production by various groups, including Replicated. Check out their blog post on their use of rqlite.

Features

Abstracts the rqlite HTTP API interaction - the POSTs, JSON handling, etc. You submit your SQL and get back an iterator with familiar database/sql semantics (Next(), Scan(), etc.) or a map[column name as string]interface{}.
Timings and other metadata (e.g., num rows affected, last insert ID, etc.) are conveniently available and parsed into appropriate types.
A connection abstraction allows gorqlite to discover and remember the rqlite leader. gorqlite will automatically try other peers if the leader is lost, enabling fault-tolerant API operations.
Timeout can be set on a per-Connection basis to accommodate those with far-flung empires.
Use familiar database URL connection strings to connection, optionally including rqlite authentication and/or specific rqlite consistency levels.
Only a single node needs to be specified in the connection. By default gorqlite will talk to that node and figure out the rest of the cluster from its redirects and status API. This is known as Cluster Discovery.
Depending on your deployment, you may wish to disable Cluster Discovery. If you do disable it only the provided URL will be used to communicate with the API instead of discovering the leader and peers and retrying failed requests with different peers. To disable Cluster Discovery add disableClusterDiscovery=true as a URL Query Parameter when connecting to rqlite e.g. http://localhost:14001?disableClusterDiscovery=true.
- This is helpful, for example, when using a Kubernetes service to handle the load balancing of the requests across healthy nodes. In such a setup Kubernetes takes care of finding a node to communicate with.
Support for several rqlite-specific operations:
- Leader() and Peers() to examine the cluster.
- SetConsistencyLevel() can be called at any time on a connection to change the consistency level for future operations.
- Timing can be referenced on a per-result basis to retrieve the timings information for executed operations as float64, per the rqlite API.
Trace(io.Writer)/Trace(nil) can be used to turn on/off debugging information on everything gorqlite does to a io.Writer of your choice.
No external dependencies. Uses only standard library functions.

Install

go get github.com/rqlite/gorqlite

Examples

// These URLs are just generic database URLs, not rqlite API URLs,
// so you don't need to worry about the various rqlite paths ("/db/query"), etc.
// just supply the base URL and not "db" or anything after it.

// Yes, you need the http or https.

// No, you cannot specify a database name in the URL (this is sqlite, after all).

conn, err := gorqlite.Open("http://") // connects to localhost on 4001 without auth
conn, err := gorqlite.Open("https://") // same but with https
conn, err := gorqlite.Open("https://localhost:4001/") // same only explicitly

// With authentication:
conn, err := gorqlite.Open("https://mary:secret2@localhost:4001/")
// different server, setting the rqlite consistency level
conn, err := gorqlite.Open("https://mary:secret2@server1.example.com:4001/?level=none")
// same without auth, setting the rqlite consistency level
conn, err := gorqlite.Open("https://server2.example.com:4001/?level=weak")
// different port, setting the rqlite consistency level and timeout
conn, err := gorqlite.Open("https://localhost:2265/?level=strong&timeout=30")
// different port, disabling cluster discovery in the client
conn, err := gorqlite.Open("https://localhost:2265/?disableClusterDiscovery=true")

// With your own HTTP client, allowing full control over the HTTP configuration.
// This is useful for skipping verification of certificates, enabling mutual TLS,
// or set specific timeouts.
client := &http.Client{}
conn, err := gorqlite.OpenWithClient("https://mary:secret2@localhost:4001/", client)

// Change our minds
conn.SetConsistencyLevel("none")
conn.SetConsistencyLevel("weak")
conn.SetConsistencyLevel("strong")

// Simulate database/sql Prepare()
statements := make ([]string,0)
pattern := "INSERT INTO secret_agents(id, hero_name, abbrev) VALUES (%d, '%s', '%3s')"
statements = append(statements,fmt.Sprintf(pattern,125718,"Speed Gibson","Speed"))
statements = append(statements,fmt.Sprintf(pattern,209166,"Clint Barlow","Clint"))
statements = append(statements,fmt.Sprintf(pattern,44107,"Barney Dunlap","Barney"))
results, err := conn.Write(statements)

// now we have an array of []WriteResult 

for n, v := range WriteResult {
	fmt.Printf("for result %d, %d rows were affected\n",n,v.RowsAffected)
	if v.Err != nil {
		fmt.Printf("   we have this error: %s\n",v.Err.Error())
	}
}

// or if we have an auto_increment column
res, err := conn.WriteOne("INSERT INTO foo (name) values ('bar')")
fmt.Printf("last insert id was %d\n",res.LastInsertID)

// just like database/sql, you're required to Next() before any Scan() or Map()

// note that rqlite is only going to send JSON types - see the encoding/json docs
// which means all numbers are float64s.  gorqlite will convert to int64s for you
// because it is convenient but other formats you will have to handle yourself

var id int64
var name string
rows, err := conn.QueryOne("select id, name from secret_agents where id > 500")
fmt.Printf("query returned %d rows\n",rows.NumRows)
for rows.Next() {
	err := rows.Scan(&id, &name)
	fmt.Printf("this is row number %d\n",rows.RowNumber)
	fmt.Printf("there are %d rows overall%d\n",rows.NumRows)
}

// just like WriteOne()/Write(), QueryOne() takes a single statement,
// while Query() takes a []string.  You'd only use Query() if you wanted
// to transactionally group a bunch of queries, and then you'd get back
// a []QueryResult

// alternatively, use Next()/Map()

for rows.Next() {
	m, err := rows.Map()
	// m is now a map[column name as string]interface{}
	id := m["name"].(float64) // the only json number type
	name := m["name"].(string)
}

// get rqlite cluster information
leader, err := conn.Leader()
// err could be set if the cluster wasn't answering, etc.
fmt.Println("current leader is"leader)
peers, err := conn.Peers()
for n, p := range peers {
	fmt.Printf("cluster peer %d: %s\n",n,p)
}

// turn on debug tracing to the io.Writer of your choice.
// gorqlite will verbosely write very granular debug information.
// this is similar to perl's DBI->Trace() facility.
// note that this is done at the package level, not the connection
// level, so you can debug Open() etc. if need be.

f, err := os.OpenFile("/tmp/deep_insights.log",OS_RDWR|os.O_CREATE|os.O_APPEND,0644)
gorqlite.TraceOn(f)

// change my mind and watch the trace
gorqlite.TraceOn(os.Stderr)

// turn off
gorqlite.TraceOff()

// using parameterized statements
wr, err := conn.WriteParameterized(
	[]gorqlite.ParameterizedStatement{
		{
			Query:     "INSERT INTO secret_agents(id, name, secret) VALUES(?, ?, ?)",
			Arguments: []interface{}{7, "James Bond", "not-a-secret"},
		},
	},
)
seq, err := conn.QueueParameterized(
	[]gorqlite.ParameterizedStatement{
		{
			Query:     "INSERT INTO secret_agents(id, name, secret) VALUES(?, ?, ?)",
			Arguments: []interface{}{7, "James Bond", "not-a-secret"},
		},
	},
)
qr, err := conn.QueryParameterized(
	[]gorqlite.ParameterizedStatement{
		{
			Query:     "SELECT id, name from secret_agents where id > ?",
			Arguments: []interface{}{3},
		},
	},
)

// alternatively
wr, err := conn.WriteOneParameterized(
	gorqlite.ParameterizedStatement{
		Query:     "INSERT INTO secret_agents(id, name, secret) VALUES(?, ?, ?)",
		Arguments: []interface{}{7, "James Bond", "not-a-secret"},
	},
)
seq, err := conn.QueueOneParameterized(
	gorqlite.ParameterizedStatement{
		Query:     "INSERT INTO secret_agents(id, name, secret) VALUES(?, ?, ?)",
		Arguments: []interface{}{7, "James Bond", "not-a-secret"},
	},
)
qr, err := conn.QueryOneParameterized(
	gorqlite.ParameterizedStatement{
		Query:     "SELECT id, name from secret_agents where id > ?",
		Arguments: []interface{}{3},
	},
)

// using nullable types
var name gorqlite.NullString
rows, err := conn.QueryOne("select name from secret_agents where id = 7")
for rows.Next() {
	err := rows.Scan(&name)
}
if name.Valid {
	// use name.String
} else {
	// NULL value
}

Queued Writes

The client does support Queued Writes. Instead of calling the Write() functions, call the queueing versions instead.

var seq int64
var err error

seq, err = conn.QueueOne("CREATE TABLE " + testTableName() + " (id integer, name text)")
seq, err = conn.Queue(...)

Controlling HTTP communications

If you need full control over the HTTP connection to rqlite, you can pass in a custom HTTP client object. This can be useful if you wish to control certification verification, configure Certificate Authorities, or enable mutual TLS.

For example, say you wish this library to skip verification of any certificates presented by rqlite:

// Create a TLS transport which skips verification of certificates.
tn := &http.Transpor

Gorqlite

Install / Use

README