Update to latest version of ouroboros-network and other deps #569

dcoutts · 2020-02-13T16:28:23Z

Most code changes due to ouroboros-network and cardano-ledger

Closes #566

cardano-node/src/Cardano/Tracing/ToObjectOrphans.hs

intricate

Apparently, there's a problem that arises as a result of the changes in this PR.

I'm running the 3 node cluster that's used for benchmarking (benchmarking/cluster3nodes/) and, for some reason, the nodes aren't exchanging transactions with one another.

The local transaction submission protocol seems to be working as node-0 is indeed receiving the initial transactions submitted by the transaction generator and adding them to the mempool, but there seems to be an issue with the node-to-node transaction submission protocol since none of the other nodes ever receive any transactions.

I can confirm that everything works as expected prior to updating the dependencies. I'm currently doing some debugging to see if I can determine what's going on.

erikd · 2020-02-14T03:52:04Z

@intricate That is a really interesting side effect. Ideally we would figure out what went wrong here and add a test to the dependency that would catch this error in the dependency's test suite before it bubbles up into the node.

intricate · 2020-02-14T04:32:19Z

On this branch, it appears that I'm only seeing 2 mux-layer trace events related to MiniProtocolNum 4 (the node-to-node transaction submission protocol) and they're both MuxTraceChannelRecvStart events related to each of the other nodes in the cluster:

...
{"at":"2020-02-14T03:58:49.57Z","env":"1.5.0:cd9ef","ns":["cardano","node","Mux"],"data":{"event":"Channel Receive Start on MiniProtocolNum 4","kind":"MuxTrace","bearer":"ConnectionId {localAddress = 127.0.0.1:XXXXX, remoteAddress = 127.0.0.1:3002}"},...}
...
{"at":"2020-02-14T03:59:09.33Z","env":"1.5.0:cd9ef","ns":["cardano","node","Mux"],"data":{"event":"Channel Receive Start on MiniProtocolNum 4","kind":"MuxTrace","bearer":"ConnectionId {localAddress = 127.0.0.1:XXXXX, remoteAddress = 127.0.0.1:3001}"},...}
...

In Network.Mux:

    recv :: m (Maybe BL.ByteString)
    recv = do
        -- We receive CBOR encoded messages as ByteStrings (possibly partial) from the
        -- matching ingress queueu. This is the same queue the 'demux' thread writes to.
        traceWith tracer $ MuxTraceChannelRecvStart mc
        blob <- atomically $ do
            blob <- readTVar q
            if blob == BL.empty
                then retry
                else writeTVar q BL.empty >> return blob
        -- say $ printf "recv mid %s mode %s blob len %d" (show mid) (show md) (BL.length blob)
        traceWith tracer $ MuxTraceChannelRecvEnd mc blob
        return $ Just blob

After a successful recv, we should also trace a MuxTraceChannelRecvEnd. However, that never ends up happening for the node-to-node tx submission protocol. Could we be indefinitely blocking in that STM transaction?

karknu · 2020-02-14T05:26:34Z

Could we be indefinitely blocking in that STM transaction?

Yes, that thread will block forever if the peer never sends any data on the corresponding miniprotocol.

intricate · 2020-02-14T05:50:23Z

In an effort to track down the problem, I've been testing out the 3-node cluster benchmarking scripts with builds of cardano-node based on master, but depending on different revisions of ouroboros-network.

IntersectMBO/ouroboros-network@`a43961c`

Node-to-node transaction submission works fine with the 3-node cluster for my build based on cardano-node master (f1a5b69d205599ad99c07f19ed9d6cf50575564a) with the following dependencies:

ouroboros-network@a43961c1133b9e1b0826f3d4d1bc5b93819037d6
cardano-base@965b8b143632faea16680a195e6de57091382700

To reproduce, see branch intricate/upd-deps-a43961c.

IntersectMBO/ouroboros-network@`c1b9ba3`

We are able to reproduce the same node-to-node transaction submission problem given a build based on cardano-node master with the following dependencies:

ouroboros-network@c1b9ba34ee2eac870a10e5194a9e2c85ddf595ec
cardano-base@965b8b143632faea16680a195e6de57091382700

To reproduce, see branch intricate/upd-deps-c1b9ba3.

Thoughts

As a result, I'm thinking that the problem has been introduced in ouroboros-network somewhere within the range a43961c1133b9e1b0826f3d4d1bc5b93819037d6..c1b9ba34ee2eac870a10e5194a9e2c85ddf595ec.

At a glance, I'm suspecting that the changes introduced by IntersectMBO/ouroboros-network#1536 could be the culprit, but I haven't looked at that PR very closely yet.

cardano-node/src/Cardano/Node/Run.hs

cardano-node/src/Cardano/Tracing/ToObjectOrphans.hs

karknu · 2020-02-14T08:08:40Z

I'm looking into this.

Jimbo4350 · 2020-02-14T13:25:02Z

Side note: I've tested ./scripts/shelley-testnet-live.sh and transaction submission successfully
on f4e5cc6afcaead00f15a4f6560becc07c6f6dee2

[Mempool:Info:155] {"kind":"TraceMempoolAddedTx","txAdded":"tx: Tx 6875a2f9e27d625b with inputs [TxInUtxo a8386f9e #0], outputs: [TxOut 863000000000000 lovelace -> 2cWKMJemoBahs2gd4sQX1XgSLVLyAMrtNy2qgp4t2BYjBFygD3dLo4ejXC4p3C3pnc6qd]\nwitnesses: [\n    VKWitness: key = pub:3ec9c54e, key hash = f0f85d06, sig = <signature>\n]","mempoolSize":"MempoolSize {msNumTxs = 1, msNumBytes = 249}"}
[LocalTxSubmissionProtocol:Debug:155] TraceLabelPeer (ConnectionId {localAddress = /home/jordan/Repos/Work/userepotool/cardano-node/socket/0, remoteAddress = }) Send MsgAcceptTx
[LocalTxSubmissionProtocol:Debug:155] TraceLabelPeer (ConnectionId {localAddress = /home/jordan/Repos/Work/userepotool/cardano-node/socket/0, remoteAddress = }) Recv MsgDone

intricate · 2020-02-14T14:15:14Z

FYI: I have tested ./scripts/shelley-testnet-live.sh and transaction submission successfully
on f4e5cc6afcaead00f15a4f6560becc07c6f6dee2

[Mempool:Info:155] {"kind":"TraceMempoolAddedTx","txAdded":"tx: Tx 6875a2f9e27d625b with inputs [TxInUtxo a8386f9e #0], outputs: [TxOut 863000000000000 lovelace -> 2cWKMJemoBahs2gd4sQX1XgSLVLyAMrtNy2qgp4t2BYjBFygD3dLo4ejXC4p3C3pnc6qd]\nwitnesses: [\n    VKWitness: key = pub:3ec9c54e, key hash = f0f85d06, sig = <signature>\n]","mempoolSize":"MempoolSize {msNumTxs = 1, msNumBytes = 249}"}
[LocalTxSubmissionProtocol:Debug:155] TraceLabelPeer (ConnectionId {localAddress = /home/jordan/Repos/Work/userepotool/cardano-node/socket/0, remoteAddress = }) Send MsgAcceptTx
[LocalTxSubmissionProtocol:Debug:155] TraceLabelPeer (ConnectionId {localAddress = /home/jordan/Repos/Work/userepotool/cardano-node/socket/0, remoteAddress = }) Recv MsgDone

Right, but those logs are related to the LocalTxSubmissionProtocol which I've observed does work fine, but I'm currently experiencing issues with the node-to-node tx submission protocol.

The fix is in, it now needs verifying.

cardano-node/ChangeLog.md

Most code changes due to ouboros-network and cardano-ledger

`Rational` as required Update `./scripts/genesis.sh` to reflect the above change.

It disables the on-demand start for now, uses eager starts.

intricate

LGTM 💯

The issue looks like it's been resolved. Thanks @karknu @dcoutts

intricate

Oops. Forgot to address @mrBliss review.

Chairman --socket-path argument can be given multiple times. It indicates to which unix sockets to connect to (and check for consensus). Chairman now identifies nodes by `SocketPath`, `CoreNodeId` is not used now, maybe it can be removed from `ourobors-consensus` as well. Nix scripts are not updated, and need to accomodate for chairman CLI changes.

intricate

Okay, LGTM for real this time.

CodiePP · 2020-02-17T16:52:13Z

ran cluster3nodes bm with tx-gen on top of this branch: looks fine

CodiePP

fine for me

intricate · 2020-02-17T16:54:10Z

Just awaiting a CI job fix from @deepfire (#577) and then we can merge.

Jimbo4350

LGTM, just awaiting the chairman fix.

569: Update to latest version of ouroboros-network and other deps r=deepfire a=dcoutts Most code changes due to ouroboros-network and cardano-ledger Closes #566 Co-authored-by: Duncan Coutts <[email protected]> Co-authored-by: Luke Nadur <[email protected]> Co-authored-by: Samuel Leathers <[email protected]> Co-authored-by: Jordan Millar <[email protected]> Co-authored-by: Marcin Szamotulski <[email protected]> Co-authored-by: Kosyrev Serge <[email protected]>

deepfire · 2020-02-17T17:51:54Z

bors r+
..as agreed with Duncan.

iohk-bors · 2020-02-17T17:51:55Z

Already running a review

iohk-bors · 2020-02-17T18:12:48Z

Build succeeded

dcoutts requested a review from Jimbo4350 February 13, 2020 16:28

dcoutts force-pushed the dcoutts/update-dependencies branch from c3d7be5 to 65622a2 Compare February 13, 2020 17:01

intricate self-assigned this Feb 13, 2020

intricate reviewed Feb 13, 2020

View reviewed changes

cardano-node/src/Cardano/Tracing/ToObjectOrphans.hs Show resolved Hide resolved

intricate mentioned this pull request Feb 13, 2020

Update dependencies #568

Closed

intricate force-pushed the dcoutts/update-dependencies branch 2 times, most recently from 2317874 to cd9ef4d Compare February 13, 2020 21:05

intricate changed the title ~~Update to latest version of ouboros-network and other deps~~ Update to latest version of ouroboros-network and other deps Feb 13, 2020

intricate requested a review from mrBliss February 13, 2020 21:09

intricate previously requested changes Feb 14, 2020

View reviewed changes

mrBliss reviewed Feb 14, 2020

View reviewed changes

cardano-node/src/Cardano/Node/Run.hs Outdated Show resolved Hide resolved

cardano-node/src/Cardano/Tracing/ToObjectOrphans.hs Show resolved Hide resolved

cardano-node/src/Cardano/Tracing/ToObjectOrphans.hs Show resolved Hide resolved

karknu mentioned this pull request Feb 14, 2020

Start responder protocols with agency. IntersectMBO/ouroboros-network#1646

Closed

dcoutts force-pushed the dcoutts/update-dependencies branch from 3080e30 to 934e0e6 Compare February 16, 2020 22:25

dcoutts requested a review from intricate February 16, 2020 22:53

karknu reviewed Feb 17, 2020

View reviewed changes

cardano-node/ChangeLog.md Outdated Show resolved Hide resolved

intricate force-pushed the dcoutts/update-dependencies branch from 805df21 to 2caface Compare February 17, 2020 15:00

dcoutts and others added 6 commits February 17, 2020 09:01

Update to latest version of ouroboros-network and other deps

75025b9

Most code changes due to ouboros-network and cardano-ledger

Implement TraceSlotIsImmutable cases

619dc56

Implement TraceVolDBEvent cases

ca61107

fix cabal.project sha256 hashes and improve CI jobs

6624e30

Fix parseFraction so that it will parse a Double but convert it to a

077fe45

`Rational` as required Update `./scripts/genesis.sh` to reflect the above change.

Update delegate_share in cluster3nodes genesis.sh script

37fc205

dcoutts added 2 commits February 17, 2020 09:01

Bump ouroboros-network to version with fix for on-demand starting

123476b

It disables the on-demand start for now, uses eager starts.

Bump version to 1.6.0 and update changelog

e88e3e1

intricate force-pushed the dcoutts/update-dependencies branch from 2caface to e88e3e1 Compare February 17, 2020 15:01

intricate approved these changes Feb 17, 2020

View reviewed changes

intricate suggested changes Feb 17, 2020

View reviewed changes

intricate mentioned this pull request Feb 17, 2020

More detailed TraceImmDBEvent and TraceVolDBEvent logging #576

Closed

intricate mentioned this pull request Feb 17, 2020

tx submission tracing details at default log level #527

Closed

Set cdbVolValidation field in runValid case of customiseChainDbArgs

3405af7

intricate approved these changes Feb 17, 2020

View reviewed changes

deepfire added 2 commits February 17, 2020 19:26

Makefile: add 'test-chairmans-cluster target

77a2111

chairmans-cluster: update the test to node CLI changes

a376106

CodiePP approved these changes Feb 17, 2020

View reviewed changes

Jimbo4350 approved these changes Feb 17, 2020

View reviewed changes

iohk-bors bot merged commit a376106 into master Feb 17, 2020

iohk-bors bot deleted the dcoutts/update-dependencies branch February 17, 2020 18:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update to latest version of ouroboros-network and other deps #569

Update to latest version of ouroboros-network and other deps #569

dcoutts commented Feb 13, 2020 •

edited by intricate

Loading

intricate left a comment

erikd commented Feb 14, 2020

intricate commented Feb 14, 2020 •

edited

Loading

karknu commented Feb 14, 2020 •

edited

Loading

intricate commented Feb 14, 2020

karknu commented Feb 14, 2020

Jimbo4350 commented Feb 14, 2020 •

edited

Loading

intricate commented Feb 14, 2020 •

edited

Loading

intricate left a comment •

edited

Loading

intricate left a comment

intricate left a comment

CodiePP commented Feb 17, 2020

CodiePP left a comment

intricate commented Feb 17, 2020

Jimbo4350 left a comment

deepfire commented Feb 17, 2020

iohk-bors bot commented Feb 17, 2020

iohk-bors bot commented Feb 17, 2020

Update to latest version of ouroboros-network and other deps #569

Update to latest version of ouroboros-network and other deps #569

Conversation

dcoutts commented Feb 13, 2020 • edited by intricate Loading

intricate left a comment

Choose a reason for hiding this comment

erikd commented Feb 14, 2020

intricate commented Feb 14, 2020 • edited Loading

karknu commented Feb 14, 2020 • edited Loading

intricate commented Feb 14, 2020

IntersectMBO/ouroboros-network@a43961c

IntersectMBO/ouroboros-network@c1b9ba3

Thoughts

karknu commented Feb 14, 2020

Jimbo4350 commented Feb 14, 2020 • edited Loading

intricate commented Feb 14, 2020 • edited Loading

intricate left a comment • edited Loading

Choose a reason for hiding this comment

intricate left a comment

Choose a reason for hiding this comment

intricate left a comment

Choose a reason for hiding this comment

CodiePP commented Feb 17, 2020

CodiePP left a comment

Choose a reason for hiding this comment

intricate commented Feb 17, 2020

Jimbo4350 left a comment

Choose a reason for hiding this comment

deepfire commented Feb 17, 2020

iohk-bors bot commented Feb 17, 2020

iohk-bors bot commented Feb 17, 2020

Build succeeded

dcoutts commented Feb 13, 2020 •

edited by intricate

Loading

intricate commented Feb 14, 2020 •

edited

Loading

karknu commented Feb 14, 2020 •

edited

Loading

IntersectMBO/ouroboros-network@`a43961c`

IntersectMBO/ouroboros-network@`c1b9ba3`

Jimbo4350 commented Feb 14, 2020 •

edited

Loading

intricate commented Feb 14, 2020 •

edited

Loading

intricate left a comment •

edited

Loading