go modules

This is an overview of using go modules, the differences between it and $GOPATH-based development, and some of the tools and techniques it makes available to go developers.

This content is also available as a video recording.

$GOPATH: the old way
- Building
- Distribution
go modules: the new way

$GOPATH: the old way

Building

Source was located under $GOPATH/src, whether we liked it or not.

go get github.com/liggitt/a
go get github.com/liggitt/b

cat $GOPATH/src/github.com/liggitt/a/make-cake.go

package main

import (
	"github.com/liggitt/a/helpers"
	"github.com/liggitt/b"
)

func main() {
	b.Cake()
	helpers.PrintSuccess()
}

cat $GOPATH/src/github.com/liggitt/a/helpers/helpers.go

package helpers

import "fmt"

func PrintSuccess() {
	fmt.Println("cake!")
}

cat $GOPATH/src/github.com/liggitt/b/cake.go

package b

func Cake() {
	// TODO: implement. the cake is a lie.
}

cd $GOPATH/src/github.com/liggitt/a
go run .

cake!

What happens

go gathers dependencies of my main package (e.g. go list -deps .)
my main package imports github.com/liggitt/a/helpers and github.com/liggitt/b
go searches for those import paths in the following places, in order:
1. github.com/liggitt/a/vendor/<import-path>
2. $GOPATH/src/<import-path>
3. $GOROOT/src/<import-path>
in this case it finds them in $GOPATH, builds, and runs

Benefits

Hermetic (if you fully control all the content in $GOPATH)
No network access
Allows local development

Potential problems

What if I need multiple things in my $GOPATH at different code levels?
- workaround: have a separate $GOPATH per project
- workaround: place a copy of all code inside a vendor dir in each project
What if I don't want to structure my directories like $GOPATH requires?
- you don't always get what you want
What if source in $GOPATH accidentally drifts from the authoritative source?
- whatever is in $GOPATH gets used, for better or worse:
  - can be bad if you weren't expecting it
  - can be good if you're intentionally developing multiple components at once

Questions/Answers

Q: What is the full import path of $GOPATH/src/github.com/liggitt/b?

A: The relative path under $GOPATH/src, so github.com/liggitt/b

Q: What version of github.com/liggitt/b does github.com/liggitt/a use?

A: Whatever is sitting in $GOPATH/src/github.com/liggitt/b

Q: What version of github.com/liggitt/b does github.com/liggitt/a prefer to use?

A: What's a version?

Distribution

go get github.com/liggitt/a

What happens

go resolves the version-control location for the specified import path
go downloads the source to $GOPATH/src/<import-path>
go also resolves and downloads transitive dependencies to $GOPATH/src/<import-path>

Benefits

Simple distribution for simple things

Potential problems

No versioning, you always get master
No versioning of dependencies, you always get master of those as well
Import path is coupled to location
Random things get dumped into $GOPATH

go modules: the new way

Note: if you want to work through this demo yourself, all the commands should work as described. If you want something prepared in advance, the results of walking through this exercise are at https://github.com/liggitt/a/tree/demo

Defining a module

Instead of being defined by a path relative to $GOPATH, modules are just a tree of Go source files with a go.mod file in the tree's root directory. The tree can be located anywhere.

Let's remove our projects from $GOPATH:

rm -fr $GOPATH/src/github.com/liggitt/{a,b}

And try to build them outside our $GOPATH:

mkdir -p $HOME/tmp/modules/can/be/anywhere
cd $HOME/tmp/modules/can/be/anywhere
git clone https://github.com/liggitt/a.git
cd a
go run .

We see the problems we expect:

make-cake.go:4:2: cannot find package "github.com/liggitt/a/helpers" in any of:
	/Users/liggitt/.gvm/gos/go1.12.1/src/github.com/liggitt/a/helpers (from $GOROOT)
	/Users/liggitt/go/src/github.com/liggitt/a/helpers (from $GOPATH)
make-cake.go:5:2: cannot find package "github.com/liggitt/b" in any of:
	/Users/liggitt/.gvm/gos/go1.12.1/src/github.com/liggitt/b (from $GOROOT)
	/Users/liggitt/go/src/github.com/liggitt/b (from $GOPATH)

Without a relative path to $GOPATH, go has no way of knowing our helpers subpackage is github.com/liggitt/a/helpers. It also doesn't have a way to find github.com/liggitt/b.

Let's turn our package into a go module:

go mod init github.com/liggitt/a

go: creating new go.mod: module github.com/liggitt/a

cat go.mod

module github.com/liggitt/a

go 1.12

Commit the initial version of our go.mod file:

git add . && git commit -m "initial go.mod file"

What happens

From go help go.mod:

The module verb defines the module path
The go verb sets the expected language version

Now when we run, go can figure out our helpers package is github.com/liggitt/a/helpers by finding the closest parent dir containing a go.mod file, looking at the name of the module it defines, then appending the relative path to the helpers directory to get the full import path.

Benefits

Allows developing outside of $GOPATH

Resolving dependencies automatically

Since we can no longer assume all go source is located under $GOPATH, how does go find source for dependencies? go run (and go build, go list, etc) will now fetch dependencies automatically from their canonical locations (the same place go get would fetch them) at run time, if needed:

go run .

go: finding github.com/liggitt/b v1.0.0
go: downloading github.com/liggitt/b v1.0.0
go: extracting github.com/liggitt/b v1.0.0
cake!

What happens

That did a few things:

It noticed a dependency that wasn't represented in our go.mod file, so it resolved and added a require directive for it (v1.0.0 happened to be the latest version):
```
git diff go.mod
```
```
diff --git a/go.mod b/go.mod
index 34c6e02..9869e5e 100644
--- a/go.mod
+++ b/go.mod
@@ -1,3 +1,5 @@
 module github.com/liggitt/a
 
 go 1.12
+
+require github.com/liggitt/b v1.0.0
```
From go help go.mod:
- The require verb requires a particular module at a given version or later (editors note: the "or later" will be important)
The syntax of a require directive is require <module> <version>.

You can also group multiple require directives into a block, just like imports:
```
require (
	example.com/thing1 v2.3.4
	example.com/thing2 v1.2.3
)
```
You can specify any resolveable tag, branch name, or SHA as a require version, and it will be canonicalized the next time the module graph is computed, and the go.mod file automatically updated.

You can also run go get <module>@<version>, and go will resolve and add a require directive for the specified module and version to your go.mod file:
```
go get github.com/liggitt/b@v1.0.0
```
Commit the updated go.mod file:
```
git add . && git commit -m "require github.com/liggitt/b@v1.0.0"
```

go run also downloaded the new dependency to a local module cache:

find $GOPATH/pkg/mod/cache/download

/Users/liggitt/go/pkg/mod/cache/download
/Users/liggitt/go/pkg/mod/cache/download/github.com
/Users/liggitt/go/pkg/mod/cache/download/github.com/liggitt
/Users/liggitt/go/pkg/mod/cache/download/github.com/liggitt/b
/Users/liggitt/go/pkg/mod/cache/download/github.com/liggitt/b/@v
/Users/liggitt/go/pkg/mod/cache/download/github.com/liggitt/b/@v/v1.0.0.mod
/Users/liggitt/go/pkg/mod/cache/download/github.com/liggitt/b/@v/list.lock
/Users/liggitt/go/pkg/mod/cache/download/github.com/liggitt/b/@v/v1.0.0.zip
/Users/liggitt/go/pkg/mod/cache/download/github.com/liggitt/b/@v/v1.0.0.lock
/Users/liggitt/go/pkg/mod/cache/download/github.com/liggitt/b/@v/v1.0.0.ziphash
/Users/liggitt/go/pkg/mod/cache/download/github.com/liggitt/b/@v/v1.0.0.info
/Users/liggitt/go/pkg/mod/cache/download/gith

Gomodules

Install / Use

README

go modules

$GOPATH: the old way

Building

What happens

Benefits

Potential problems

Questions/Answers

Distribution

What happens

Benefits

Potential problems

go modules: the new way

Defining a module

What happens

Benefits

Resolving dependencies automatically

What happens

Related Skills