Tobari

Tobari is a scoped coverage measurement tool for Go.

"Tobari" (帷) is a Japanese word meaning "curtain" or "veil," similar to the English word "cover".

Tobari provides coverage measurement capabilities that introduce a new concept called Scoped Coverage in addition to the coverage features provided by runtime/coverage. This feature enables clear mapping between test code and its impact area, allowing for high-precision test code generation using AI and other tools.

Background

To understand what Tobari enables, we first need to understand the current coverage mechanisms provided by Go.

The most common coverage measurement method we use is specifying coverage options with go test, such as go test -cover. We can control coverage target packages with -coverpkg and output coverage results with -coverprofile. However, using coverage through go test means we can only measure coverage when writing test code in Go. This means coverage can only be measured for tests starting from functions like func TestFoo(t *testing.T). For example, it was not possible to measure coverage for a binary created with go build after the fact.

To improve this, Go version 1.20 and later allows the -cover option with go build and supports the runtime/coverage package. With go build -cover, applications can be built with coverage instrumentation points. The runtime/coverage API allows coverage counter initialization and result output at any timing.

This made it possible to measure coverage of servers implemented in Go using E2E testing tools not written in Go when implementing HTTP or gRPC servers. Coverage functionality that was only available during go test execution became available at any timing during application runtime.

However, when actually trying to use runtime/coverage, you'll notice that some operational considerations are needed. Coverage measurement with runtime/coverage increases coverage counters when processing passes through embedded coverage measurement points, but it doesn't care what caused the passage. This is similar to traffic surveys for automobiles, where the number of cars passing a certain location is measured, but the type of cars is not considered. This mechanism becomes problematic in situations like measuring coverage for server applications:

• To measure E2E test effectiveness, you want to measure coverage only when E2E testing tools access the server
  • Accesses other than from E2E tests should not be measured
• For test acceleration, you want to access the server concurrently from E2E testing tools, but manage access contexts separately
  • For example, when E2E test scenarios A and B exist, you want to measure accesses from A and B separately even when accessing the server concurrently
• Asynchronous processing by Goroutines created through methods not originating from E2E test requests should be excluded from coverage measurement

To meet these requirements, the server application must serialize and handle requests, ensuring that accesses from A and B are not processed simultaneously. Additionally, you need to implement mechanisms to determine E2E test accesses by referencing headers and reject other requests. Furthermore, what if you want to use this running server application for purposes other than E2E testing? For example, when performing manual verification in parallel with E2E test verification. In this case, other requests may reach the server while E2E tests are running, and these requests should not be rejected. Also, serializing processing defeats the purpose of concurrent access for test acceleration. Moreover, there's no way to ignore asynchronous processing not originating from E2E tests.

Therefore, I conceived the Scoped Coverage approach and decided to develop Tobari.

What is Scoped Coverage ?

What Scoped Coverage provides over runtime/coverage coverage measurement is measuring "what passed through". Using the E2E testing example, when measuring coverage from scenario A and B accesses, it distinguishes between "access from A" and "access from B". Additionally, it measures only asynchronous processing originating from E2E tests. This enables coverage measurement limited to the scope you want to measure. Concurrent access is also possible.

Coverage also needs to record "places that should be passed" in addition to recording "places that were passed". Coverage is calculated using a formula like this:

Coverage (%) = (Places passed / Places that should be passed) * 100

In normal coverage measurement, all files in packages specified by coverpkg could be considered "places that should be passed", requiring no special processing. However, Scoped Coverage is different. How should we define "places that should be passed" ? For example, when calculating coverage for scenario A, if places that scenario A will never pass are included in "places that should be passed", coverage will never reach 100% no matter how hard you try.

Tobari determines "places that should be passed" by reverse calculation from the results of what was passed. It extracts inter-function dependencies through static analysis in advance and calculates functions that could potentially be passed based on the dependency relationships of actually passed functions. Considering this, the formula becomes:

Coverage (%) = (Places passed / All places in functions that could potentially be called from passed functions) * 100

How to Use

Using tobari is very simple with 3 steps:

1. Installation

First, install the tobari tool with the following command:

go install github.com/goccy/tobari/cmd/tobari@latest

Important: The version of the tobari CLI tool and the tobari library used in your application must match. For example, if you install tobari@v0.2.1, your go.mod should also require github.com/goccy/tobari v0.2.1. Version mismatch will cause fingerprint errors during linking.

2. Use the API (like using the runtime/coverage package)

Using a gRPC server as an example:

tobari.CoverWithName serves as the entry point for coverage measurement. Specify a name in the first argument to distinguish coverage units. The behavior with runtime/coverage is the same as specifying an empty name. For gRPC, the name is obtained from metadata. If there's no metadata, it's treated as a normal request and the function exits without measuring coverage.

tobariInterceptor := func(ctx context.Context, req any, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (any, error) {
  md, ok := metadata.FromIncomingContext(ctx)
  if !ok { return handler(ctx, req) }
  scenarioNames, exists := md["E2E-Tool-Scenario-Name"]
  if !exists { return handler(ctx, req) }
  if len(scenarioNames) == 0 { return handler(ctx, req) }

  scenarioName := scenarioNames[0]

  var (
    res any
    err error
  )
  tobari.CoverWithName(scenarioName, func() {
    res, err = handler(ctx, req)
  })
  return res, err
}

grpcServer := grpc.NewServer(grpc.UnaryInterceptor(tobariInterceptor))

When outputting coverage data, you can use WriteCoverprofileByName or CoverprofileMap:

• WriteCoverprofileByName: Get coverprofile results for a specified name
• CoverprofileMap: Return the relationship between names and coverprofiles in map format

These APIs can be executed by creating a separate gRPC server and calling specific endpoints when E2E tests finish.

3. Build the Application

Then, when building the application you want to measure coverage for, simply specify GOFLAGS as follows:

GOFLAGS="$(tobari flags)" go build .

# With build tags
GOFLAGS="$(tobari flags -tags=timetzdata)" go build .

This example shows distinguishing coverage results by name, but you can also use it the same way as runtime/coverage. For specific APIs, please refer here.

Embedding Source Code

Tobari supports embedding the original source code into instrumented binaries with the --embed-code (-E) option. This is useful for archiving the exact source that was compiled, enabling offline coverage analysis without access to the original source tree.

# Using tobari flags
GOFLAGS="$(tobari flags -E)" go build .

# Using tobari directly as toolexec
go build -cover -toolexec='tobari --embed-code' ./...

To retrieve the embedded sources at runtime, use ReadCoverArchivedFile():

re