cmd/go: downgrading a module with go get retains indirect dependencies

[mwf]

What version of Go are you using (go version)?

Latest go devel: go version devel +d278f09333 Thu Jul 19 05:40:37 2018 +0000 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/ikorolev/Library/Caches/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/ikorolev/.gvm/pkgsets/go1.11beta1/global:/Users/ikorolev/dev/go"
GOPROXY=""
GORACE=""
GOROOT="/Users/ikorolev/.gvm/gos/go1.11beta1"
GOTMPDIR=""
GOTOOLDIR="/Users/ikorolev/.gvm/gos/go1.11beta1/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/_b/d1934m9s587_8t_6ngv3hnc00000gp/T/go-build301560759=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

It's maybe connected with #26474 but is much more severe.

Sorry, can't give you a standalone reproduction again - bug is produced only with downloading updated dependencies.
Consider the following flow:

C(v0.1.0) uses A(v0.1.0):

module github.com/mwf/vgo-modules/c

require github.com/mwf/vgo-modules/a v0.1.0

Then A(v0.2.0) is released and C(v0.2.0) depending on new A(v0.2.0).

We got a project main which uses C(v0.1.0) and we want to upgrade C to v0.2.0

cd `mktemp -d`
git clone https://github.com/mwf/vgo-modules .
cd c_user
echo "\n$(cat go.mod)"
echo "\n$(cat go.sum)"
go get github.com/mwf/vgo-modules/[email protected]
echo "\n$(cat go.mod)"

Output:

module github.com/mwf/vgo-modules/c_user

require github.com/mwf/vgo-modules/c v0.1.0

github.com/mwf/vgo-modules/a v0.1.0 h1:QD+ijrXwAYk4CMTOqEAzcogJoF4zLfit2alZVmT80EM=
github.com/mwf/vgo-modules/a v0.1.0/go.mod h1:XGJvSKC62ygHgRNmDf4RqXyp0zngXIJMLc6BaSfyTfI=
github.com/mwf/vgo-modules/c v0.1.0 h1:db5PK5vzxlLviLcUsd1YCaLjG3+35hw6OMKSdVzqPMo=
github.com/mwf/vgo-modules/c v0.1.0/go.mod h1:qfiW4sL7v2Bnu6kBwvwxeVylzuPa6cxOldlsx2NCIRI=

module github.com/mwf/vgo-modules/c_user

require github.com/mwf/vgo-modules/c v0.2.0

All seems OK, than we run tests and understand that C(v0.2.0) is broken - go test .

--- FAIL: TestC (0.00s)
    main_test.go:12: c.CA: got CAaaaa, expected: CA
FAIL
FAIL    github.com/mwf/vgo-modules/c_user   0.014s

We decide to downgrade C back to v0.1.0 used before:

go get github.com/mwf/vgo-modules/[email protected]
echo "\n$(cat go.mod)"
go test .

What did you expect to see?

go.mod should contain downgraded c dependency only

module github.com/mwf/vgo-modules/c_user

require github.com/mwf/vgo-modules/c v0.1.0

What did you see instead?

After the downgrade we get unexpected github.com/mwf/vgo-modules/a v0.2.0 // indirect and tests still fail.

module github.com/mwf/vgo-modules/c_user

require (
    github.com/mwf/vgo-modules/a v0.2.0 // indirect
    github.com/mwf/vgo-modules/c v0.1.0
)

--- FAIL: TestC (0.00s)
    main_test.go:12: c.CA: got CAaaaa, expected: CA
FAIL
FAIL    github.com/mwf/vgo-modules/c_user   0.018s

Some thoughts

We should definitely downgrade the transitive dependencies in this case.

But I understand it's a nontrivial task in general - because we should somehow distinguish cases when we intentionally add a v0.1.1 in main, then upgrade && downgrade c.

Ideally, we should go back to a v0.1.1 and c v.1.0, but to achieve this we should track the history somehow - and we don't have such feature using go.sum now.

cc @myitcv @rsc

[bcmills]

Ideally, we should go back to a v0.1.1 and c v.1.0, but to achieve this we should track the history somehow

In the typical case you can just check out the previous go.mod from version control, right?

It gets to be more of a problem when you're upgrading and downgrading a bunch of unrelated dependencies in the same change, but that seems easy to avoid: how big is this problem in practice?
(Concrete examples of workflows that can lead to this kind of problem would be helpful.)

$ cd `mktemp -d`
$ git clone https://github.com/mwf/vgo-modules .
$ cd c_user

c_user$ go list -m all
go: finding github.com/mwf/vgo-modules/c v0.1.0
github.com/mwf/vgo-modules/c_user
github.com/mwf/vgo-modules/a v0.1.0
github.com/mwf/vgo-modules/c v0.1.0

c_user$ go get github.com/mwf/vgo-modules/[email protected]
go: finding github.com/mwf/vgo-modules/c v0.2.0
go: finding github.com/mwf/vgo-modules/a v0.2.0
go: downloading github.com/mwf/vgo-modules/c v0.2.0
go: downloading github.com/mwf/vgo-modules/a v0.2.0

c_user$ go list -m all
github.com/mwf/vgo-modules/c_user
github.com/mwf/vgo-modules/a v0.2.0
github.com/mwf/vgo-modules/c v0.2.0

c_user$ git checkout HEAD go.mod

c_user$ go list -m all
github.com/mwf/vgo-modules/c_user
github.com/mwf/vgo-modules/a v0.1.0
github.com/mwf/vgo-modules/c v0.1.0

[mwf]

VCS checkout is great, but it helps only in trivial case when you update, run tests and determine the failure. Then you rollback with VCS and you're done.

But imagine more complex situation, when the bug is hidden - either it's not covered with tests properly or it's just too complex and happens in some particular circumstances in runtime. With a panic for example or even some slow goroutine or file descriptor leak.

So the gap between updating the go.mod and getting the bug could be hours-days. And even several application releases could happen in this time, with other dependency upgrades possible.

Imagine this bug is in production and all you want in this minute is a quick hotfix. You may try to use the VCS, checkout the dependency version from that commit and just take it.
But I bet it could be a hard task if this dependency has a big transitive dependency tree like https://github.com/coreos/etcd for example.

So in case of VCS checkout you will have to doublecheck the transitive dependencies in go.mod after go mod -sync.
I would prefer instead if go get <rollbacked version> does this job for me and takes precisely the transitive dependencies from the previous version, which were stably used.

And the current behaviour doesn't match with doc from go help module-get saying

Similarly, downgrading one dependency may
require downgrading other dependenceis, and 'go get' does
this automatically as well.

It doesn't downgrade the other dependencies now actually.

PS There is a typo in the docs :) dependenceis -> dependencies

[bcmills]

I would prefer instead if go get <rollbacked version> does this job for me and takes precisely the transitive dependencies from the previous version, which were stably used.

Rolling back to a previous state is conceptually very different from go get. go get means “make the minimal change necessary to get to the requested version”, not “return me to a point in history in which I depended on that version”.

The go tool is not a version control system. It interprets the current contents of your go.mod file, not its complete history.

[rsc]

This is working as designed, as intended, and as documented: go get means make the exact, unique minimum number changes to effect that command, nothing more, nothing less. You are asking for go get to magically infer the "more" you want. That's just not possible.

You are asking for version control on go.mod. The go.mod file has been engineered to be version control-friendly, precisely so that job can be reasonably done with your version control system of choice, instead of having the go command reimplement a new version control system of its own.

Sorry, I know this is not the answer you want.