Skip to content
This repository has been archived by the owner on Feb 1, 2023. It is now read-only.

Data race in bitswap on start #464

Closed
Stebalien opened this issue Feb 26, 2021 · 0 comments · Fixed by #465
Closed

Data race in bitswap on start #464

Stebalien opened this issue Feb 26, 2021 · 0 comments · Fixed by #465

Comments

@Stebalien
Copy link
Member

There's actually another data race I can reproduce with the exact same test code, if I comment out the code that causes the first race. It concerns go-bitswap's internal/decision/blockstoreManager. I haven't investigated further just yet; the goroutine call stack dump is fairly complex, unfortunately.

WARNING: DATA RACE
Read at 0x00c001713dc0 by goroutine 390:
  github.com/ipfs/go-bitswap/internal/decision.(*blockstoreManager).addJob()
      /home/mvdan/go/pkg/mod/github.com/ipfs/[email protected]/internal/decision/blockstoremanager.go:58 +0x84
  github.com/ipfs/go-bitswap/internal/decision.(*blockstoreManager).jobPerKey()
      /home/mvdan/go/pkg/mod/github.com/ipfs/[email protected]/internal/decision/blockstoremanager.go:115 +0x1f4
  github.com/ipfs/go-bitswap/internal/decision.(*blockstoreManager).getBlockSizes()
      /home/mvdan/go/pkg/mod/github.com/ipfs/[email protected]/internal/decision/blockstoremanager.go:72 +0x1bc
  github.com/ipfs/go-bitswap/internal/decision.(*Engine).MessageReceived()
      /home/mvdan/go/pkg/mod/github.com/ipfs/[email protected]/internal/decision/engine.go:453 +0x36d
  github.com/ipfs/go-bitswap.(*Bitswap).ReceiveMessage()
      /home/mvdan/go/pkg/mod/github.com/ipfs/[email protected]/bitswap.go:444 +0x169
  github.com/ipfs/go-bitswap/network.(*impl).handleNewStream()
      /home/mvdan/go/pkg/mod/github.com/ipfs/[email protected]/network/ipfs_impl.go:402 +0x585
  github.com/ipfs/go-bitswap/network.(*impl).handleNewStream-fm()
      /home/mvdan/go/pkg/mod/github.com/ipfs/[email protected]/network/ipfs_impl.go:378 +0x5e
  github.com/libp2p/go-libp2p/p2p/host/basic.(*BasicHost).SetStreamHandler.func1()
      /home/mvdan/go/pkg/mod/github.com/libp2p/[email protected]/p2p/host/basic/basic_host.go:566 +0xb9

Previous write at 0x00c001713dc0 by goroutine 19:
  github.com/ipfs/go-bitswap/internal/decision.(*blockstoreManager).start()
      /home/mvdan/go/pkg/mod/github.com/ipfs/[email protected]/internal/decision/blockstoremanager.go:33 +0x44
  github.com/ipfs/go-bitswap/internal/decision.(*Engine).StartWorkers()
      /home/mvdan/go/pkg/mod/github.com/ipfs/[email protected]/internal/decision/engine.go:235 +0x6a
  github.com/ipfs/go-bitswap.New()
      /home/mvdan/go/pkg/mod/github.com/ipfs/[email protected]/bitswap.go:219 +0x18b4
  github.com/ipfs/go-ipfs/core/node.OnlineExchange.func1()
      /home/mvdan/go/pkg/mod/github.com/ipfs/[email protected]/core/node/core.go:94 +0x1e4
  runtime.call128()
      /usr/lib/go/src/runtime/asm_amd64.s:553 +0x58
  reflect.Value.Call()
      /usr/lib/go/src/reflect/value.go:337 +0xd8
  go.uber.org/dig.defaultInvoker()
      /home/mvdan/go/pkg/mod/go.uber.org/[email protected]/dig.go:284 +0x7a
  go.uber.org/dig.(*node).Call()
      /home/mvdan/go/pkg/mod/go.uber.org/[email protected]/dig.go:710 +0x2d3
  go.uber.org/dig.paramSingle.Build()
      /home/mvdan/go/pkg/mod/go.uber.org/[email protected]/param.go:245 +0x38e
  go.uber.org/dig.(*paramSingle).Build()
      <autogenerated>:1 +0xe7
  go.uber.org/dig.paramList.BuildList()
      /home/mvdan/go/pkg/mod/go.uber.org/[email protected]/param.go:201 +0xf7
  go.uber.org/dig.(*node).Call()
      /home/mvdan/go/pkg/mod/go.uber.org/[email protected]/dig.go:701 +0x156
  go.uber.org/dig.paramSingle.Build()
      /home/mvdan/go/pkg/mod/go.uber.org/[email protected]/param.go:245 +0x38e
  go.uber.org/dig.(*paramSingle).Build()
      <autogenerated>:1 +0xe7
  go.uber.org/dig.paramList.BuildList()
      /home/mvdan/go/pkg/mod/go.uber.org/[email protected]/param.go:201 +0xf7
  go.uber.org/dig.(*node).Call()
      /home/mvdan/go/pkg/mod/go.uber.org/[email protected]/dig.go:701 +0x156
  go.uber.org/dig.paramSingle.Build()
      /home/mvdan/go/pkg/mod/go.uber.org/[email protected]/param.go:245 +0x38e
  go.uber.org/dig.(*paramSingle).Build()
      <autogenerated>:1 +0xe7
  go.uber.org/dig.paramList.BuildList()
      /home/mvdan/go/pkg/mod/go.uber.org/[email protected]/param.go:201 +0xf7
  go.uber.org/dig.(*node).Call()
      /home/mvdan/go/pkg/mod/go.uber.org/[email protected]/dig.go:701 +0x156
  go.uber.org/dig.paramSingle.Build()
      /home/mvdan/go/pkg/mod/go.uber.org/[email protected]/param.go:245 +0x38e
  go.uber.org/dig.(*paramSingle).Build()
      <autogenerated>:1 +0xe7
  go.uber.org/dig.paramObjectField.Build()
      /home/mvdan/go/pkg/mod/go.uber.org/[email protected]/param.go:384 +0x193
  go.uber.org/dig.paramObject.Build()
      /home/mvdan/go/pkg/mod/go.uber.org/[email protected]/param.go:311 +0x130
  go.uber.org/dig.(*paramObject).Build()
      <autogenerated>:1 +0xe7
  go.uber.org/dig.paramList.BuildList()
      /home/mvdan/go/pkg/mod/go.uber.org/[email protected]/param.go:201 +0xf7
  go.uber.org/dig.(*Container).Invoke()
      /home/mvdan/go/pkg/mod/go.uber.org/[email protected]/dig.go:432 +0x384
  go.uber.org/fx.(*App).executeInvokes()
      /home/mvdan/go/pkg/mod/go.uber.org/[email protected]/app.go:692 +0x5c4
  go.uber.org/fx.New()
      /home/mvdan/go/pkg/mod/go.uber.org/[email protected]/app.go:471 +0xb72
  github.com/ipfs/go-ipfs/core.NewNode()
      /home/mvdan/go/pkg/mod/github.com/ipfs/[email protected]/core/builder.go:41 +0x2ef
  test.TestRace()
      /home/mvdan/src/test/ipfs_test.go:49 +0x5e4
  github.com/syndtr/goleveldb/leveldb.(*DB).checkAndCleanFiles()
      /home/mvdan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_util.go:52 +0x301
  github.com/syndtr/goleveldb/leveldb.openDB()
      /home/mvdan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:130 +0x9bb
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/mvdan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/storage/file_storage.go:458 +0x309
  fmt.Fscanf()
      /usr/lib/go/src/fmt/scan.go:143 +0xee
  fmt.Sscanf()
      /usr/lib/go/src/fmt/scan.go:114 +0x191
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/mvdan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/storage/file_storage.go:643 +0xa6
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/mvdan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/storage/file_storage.go:458 +0x309
  fmt.Fscanf()
      /usr/lib/go/src/fmt/scan.go:143 +0xee
  fmt.Sscanf()
      /usr/lib/go/src/fmt/scan.go:114 +0x284
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/mvdan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/storage/file_storage.go:657 +0x19e
  fmt.(*ss).doScanf()
      /usr/lib/go/src/fmt/scan.go:1230 +0x411
  fmt.Fscanf()
      /usr/lib/go/src/fmt/scan.go:143 +0xee
  fmt.Sscanf()
      /usr/lib/go/src/fmt/scan.go:114 +0x191
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/mvdan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/storage/file_storage.go:643 +0xa6
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/mvdan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/storage/file_storage.go:458 +0x309
  fmt.Fscanf()
      /usr/lib/go/src/fmt/scan.go:143 +0xee
  fmt.Sscanf()
      /usr/lib/go/src/fmt/scan.go:114 +0x191
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/mvdan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/storage/file_storage.go:643 +0xa6
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/mvdan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/storage/file_storage.go:458 +0x309
  github.com/syndtr/goleveldb/leveldb.(*DB).recoverJournal()
      /home/mvdan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:476 +0xc8
  github.com/syndtr/goleveldb/leveldb.openDB()
      /home/mvdan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:125 +0x997
  github.com/syndtr/goleveldb/leveldb.Open()
      /home/mvdan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:197 +0x25a
  fmt.Fscanf()
      /usr/lib/go/src/fmt/scan.go:143 +0xee
  fmt.Sscanf()
      /usr/lib/go/src/fmt/scan.go:114 +0x191
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/mvdan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/storage/file_storage.go:643 +0xa6
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/mvdan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/storage/file_storage.go:458 +0x309
  github.com/syndtr/goleveldb/leveldb.(*session).recover.func1()
      /home/mvdan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/session.go:113 +0xfc
  github.com/syndtr/goleveldb/leveldb.(*session).recover()
      /home/mvdan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/session.go:121 +0x1a7f
  github.com/syndtr/goleveldb/leveldb.Open()
      /home/mvdan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:183 +0x109
  github.com/syndtr/goleveldb/leveldb.OpenFile()
      /home/mvdan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:219 +0xbb
  github.com/ipfs/go-ds-leveldb.NewDatastore()
      /home/mvdan/go/pkg/mod/github.com/ipfs/[email protected]/datastore.go:46 +0x40c
  github.com/ipfs/go-ipfs/plugin/plugins/levelds.(*datastoreConfig).Create()
      /home/mvdan/go/pkg/mod/github.com/ipfs/[email protected]/plugin/plugins/levelds/levelds.go:85 +0xfa
  github.com/ipfs/go-ipfs/repo/fsrepo.measureDatastoreConfig.Create()
      /home/mvdan/go/pkg/mod/github.com/ipfs/[email protected]/repo/fsrepo/datastores.go:243 +0x5c
  github.com/ipfs/go-ipfs/repo/fsrepo.(*measureDatastoreConfig).Create()
      <autogenerated>:1 +0xac
  github.com/ipfs/go-ipfs/repo/fsrepo.(*mountDatastoreConfig).Create()
      /home/mvdan/go/pkg/mod/github.com/ipfs/[email protected]/repo/fsrepo/datastores.go:154 +0x13e
  github.com/ipfs/go-ipfs/repo/fsrepo.(*FSRepo).openDatastore()
      /home/mvdan/go/pkg/mod/github.com/ipfs/[email protected]/repo/fsrepo/fsrepo.go:440 +0x4aa
  github.com/ipfs/go-ipfs/repo/fsrepo.open()
      /home/mvdan/go/pkg/mod/github.com/ipfs/[email protected]/repo/fsrepo/fsrepo.go:169 +0x476
  github.com/ipfs/go-ipfs/repo/fsrepo.Open.func1()
      /home/mvdan/go/pkg/mod/github.com/ipfs/[email protected]/repo/fsrepo/fsrepo.go:113 +0x4e
  github.com/ipfs/go-ipfs/repo.(*OnlyOne).Open()
      /home/mvdan/go/pkg/mod/github.com/ipfs/[email protected]/repo/onlyone.go:35 +0x207
  github.com/ipfs/go-ipfs/repo/fsrepo.Open()
      /home/mvdan/go/pkg/mod/github.com/ipfs/[email protected]/repo/fsrepo/fsrepo.go:115 +0x9c
  test.TestRace()
      /home/mvdan/src/test/ipfs_test.go:38 +0x43c
  testing.tRunner()
      /usr/lib/go/src/testing/testing.go:1194 +0x202

Goroutine 390 (running) created at:
  github.com/libp2p/go-libp2p/p2p/host/basic.(*BasicHost).newStreamHandler()
      /home/mvdan/go/pkg/mod/github.com/libp2p/[email protected]/p2p/host/basic/basic_host.go:416 +0x7a9
  github.com/libp2p/go-libp2p/p2p/host/basic.(*BasicHost).newStreamHandler-fm()
      /home/mvdan/go/pkg/mod/github.com/libp2p/[email protected]/p2p/host/basic/basic_host.go:373 +0x5e
  github.com/libp2p/go-libp2p-swarm.(*Conn).start.func1.1()
      /home/mvdan/go/pkg/mod/github.com/libp2p/[email protected]/swarm_conn.go:125 +0x11a

Goroutine 19 (running) created at:
  testing.(*T).Run()
      /usr/lib/go/src/testing/testing.go:1239 +0x5d7
  testing.runTests.func1()
      /usr/lib/go/src/testing/testing.go:1512 +0xa6
  testing.tRunner()
      /usr/lib/go/src/testing/testing.go:1194 +0x202
  testing.runTests()
      /usr/lib/go/src/testing/testing.go:1510 +0x612
  testing.(*M).Run()
      /usr/lib/go/src/testing/testing.go:1418 +0x3b3
  main.main()
      _testmain.go:43 +0x236

Originally posted by @mvdan in ipfs/kubo#7947 (comment)
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix a startup race by creating the blockstoremanager process on init ipfs/go-bitswap
1 participant
@Stebalien