feat(gossipsub): introduce backpressure #4914

jxs · 2023-11-22T23:29:50Z

Follows @mxinden implementation suggestion on #4667 (comment)

Notes & open questions

Publish, GRAFT, PRUNE and Subscription messages are prioritized, Publish messages are cap'ed, using sender.clone().try_send doesn't work as the queue is shared by the Receiver. I used an AtomicUsize for that.
Used async-channel to be able to check if is_empty() in the ConnectionHandler.
Reused the InsufficientPeers peers Error as to not introduce another variant and make the changes breaking, but this is probably a breaking change in the sense that the internal behaviour is changing no?
if this design makes sense I can then add a test to test queues fill.

Change checklist

I have performed a self-review of my own code
I have made corresponding changes to the documentation
I have added tests that prove my fix is effective or that my feature works
A changelog entry has been made in the appropriate crates

thomaseizinger

Great start! Left some comments! :)

protocols/gossipsub/src/behaviour.rs

protocols/gossipsub/src/types.rs

protocols/gossipsub/src/config.rs

thomaseizinger · 2023-11-22T23:37:26Z

protocols/gossipsub/src/types.rs

+    let (priority_sender, priority_receiver) = async_channel::unbounded();
+    let (non_priority_sender, non_priority_receiver) = async_channel::bounded(cap / 2);


async-channel is an MPMC channel but we don't ever .clone() the Receiver, right? So why not use the mpsc channel from futures?

The futures::channel::mpsc implementation gives you one slot per Sender + whatever capacity you initialize the channel with. So by doing sender.clone().try_send, I think we are guaranteed to be able to send a message, even if the Sender is immediately dropped after. Technically, that makes the channel unbounded but if we only do it for Control messages, that shouldn't matter much because they are so small.

I think that is what @mxinden meant in #4667 (comment).

async-channel is an MPMC channel but we don't ever .clone() the Receiver, right? So why not use the mpsc channel from futures?

because we need to check if the channel is empty on the connection handler when determining if we need to create the outbound stream:

if !self.send_queue.is_empty() && self.outbound_substream.is_none() && !self.outbound_substream_establishing {

handler.rs:231

You can wrap the Receiver in a Peekable and check if it has an item: https://docs.rs/futures/latest/futures/stream/trait.StreamExt.html#method.peekable

thanks Thomas I wasn't aware of this! Nor of the property Max mentioned regarding the Sender clone. See my question regarding MPMC on #4914 (comment)

thomaseizinger · 2023-11-22T23:51:31Z

protocols/gossipsub/src/behaviour.rs

            let event = RpcOut::Subscribe(topic_hash.clone());
-            self.send_message(peer, event);
+            self.send_message(peer, event)
+                .expect("Subscribe messages should be always sent");


I think it would be useful if we'd split RpcOut to avoid this. Something like:

ControlRpcOut

Publish

Forward

And then we can have:

fn send_control_message(); // Never fails fn send_publish() -> Result<(), ...>; // Can fail fn send_forward(); // Never fails but pushes into a different channel

yeah thought about it, but ControlAction has different types of priorities, IHAVE/IWANT are low and GRAFT/PRUNE are high priority, all of them are under the ControlAction enum as struct variants. We could split them into isolated structures to then use both on PriorityRpcOut and and RpcOut so that we could do:

fn prune(&mut self, graft: Graft); fn graft(&mut self, graft: Graft);

but ControlAction is pub so that would be a breaking change

but ControlAction has different types of priorities, IHAVE/IWANT are low and GRAFT/PRUNE are high priority

Why is this distinction relevant?

The goal of this pull request is to prevent unbounded growth of send_queue. Neither IHAVE/IWANT nor GRAFT/PRUNE are large. Correct me if I am wrong. Thus differentiating among the two is not relevant for this pull request.

I was following @AgeManning on #4667 (comment) but you are probably right Max, updated the code to implement Thomas suggested methods into RpcSender which allow us to not split RpcOut and meanwhile also remove send_message and the double iterations on Behaviour, ptal Thomas

I think the distinction is important.

GRAFT/PRUNE are very important in maintaining the overlay mesh network. If we drop these messages we could break the whole network, i.e nodes could start randomly having broken links and will not know to find new mesh peers and messages can stop being forwarded through the network.

IHAVE/IWANT are very low priority. In fact if we are struggling with send/receiving messages we probably want to drop these along with forward messages. Although the messages themselves are fairly small (but we can send a large number of message-ids, like a few thousand (configurable)), they both induce a greater message load. IHAVE can induce IWANT requests from other peers, which ends up making us publish more messages (which we don't want if we are struggling). IWANT will request more messages to be sent to us and further consume bandwidth. An IWANT can request a fair amount more messages (up to 500 in lighthhouse).

For these reasons, I think if we are in the unhappy case where we are struggling, we want to drop IHAVE/IWANT and forward messages, but we definitely do not want to drop GRAFT/PRUNE if we can help it.

As I write this, we probably also want to make sure that messages that we respond to an IWANT with, gets sent as a "forward" message and not a "publish" message.
If it is sent too late, we will have already broken our promise, as they are time-bound anyway. We want to remove these from sending if we can't send them in time.

Distinguishing in GRAFT/PRUNE and IHAVE/IWANT makes sense to me. Thank you for expanding on it.

That said, I suggest not enlarging the scope of this pull request. I suggest to fix the unbounded send_queue problem in this pull request (#4667). Once merged I suggest following up with one or more pull requests introducing various optimizations, e.g. time based prioritization of forward messages, differentiation of GRAFT/PRUNE and IHAVE/IWANT, ...

I won't block this pull request in case you want to do all of this in one. Though I expect separate pull requests to be significantly either to implement, review and debug.

thomaseizinger · 2023-11-22T23:53:32Z

protocols/gossipsub/src/behaviour.rs

+        if sender.try_send(rpc.clone()).is_err() {
+            tracing::debug!(peer=%peer_id, "Dropping message as peer is full");
+            return Err(());
+        }


We should differentiate between Closed and Full here and perhaps return the message back to the caller in Err.

but Closed is unreachable no?

Not necessarily. A connection runs in a different task and on a multi-threaded executor, this connection could get dropped and thus the receiver freed before we receive the event that the connection is closed (where we'd clean up the state).

It might not matter though for this case so probably fine :)

thomaseizinger · 2023-11-22T23:54:28Z

protocols/gossipsub/src/behaviour.rs

+            tracing::debug!(peer=%peer_id, "Dropping message as peer is full");
+            return Err(());
+        }
+
        if let Some(m) = self.metrics.as_mut() {
            if let RpcOut::Publish(ref message) | RpcOut::Forward(ref message) = rpc {
                // register bytes sent on the internal metrics.
                m.msg_sent(&message.topic, message.raw_protobuf_len());


If you compute this before we use try_send, you can avoid the .clone() on the message.

but if the send fails when we are adding wrong metrics data as the message was not actually sent no?

protocols/gossipsub/src/behaviour.rs

thomaseizinger · 2023-11-22T23:56:45Z

protocols/gossipsub/src/behaviour/tests.rs

Ugh, this is so painful ...

We should really rewrite these tests to go against the Swarm API using libp2p-swarm-test. Changing tests AND implementation in one PR makes me a lot less confident in a green CI.

There are a lot of tests to change. Can we perhaps make this a group effort? (cc @mxinden @AgeManning)

It probably makes sense to first rewrite a few in order to create some utility functions for setting up networks. But once those are in place, perhaps we can chip away at them together? There are 85 tests to change ..

I also don't want to blow the scope of this work too much. It is just that reviewing all the changes to these tests also takes a lot of time but it is an effort that we aren't going to benefit from in the future.

Yeah agree Thomas, let me look into that

Yeah. I'm happy to help out also. My opinion is to try and get these changes in first (we'll be doing fairly extensive tests on live networks as well as small scale simulations) to give us confidence in the changes.
Then in a future PR we re-write all the tests.

thomaseizinger · 2023-11-23T00:19:52Z

protocols/gossipsub/src/handler.rs

-                        self.send_queue.shrink_to_fit();
-                        self.outbound_substream =
-                            Some(OutboundSubstreamState::PendingSend(substream, message));
+                    if let Poll::Ready(Some(message)) = self.send_queue.poll_next_unpin(cx) {


Can we now get rid of OutboundSubstreamState and just use an async move with a loop inside? I guess that would require us to be able to .clone() the Receiver so we can keep one around in case the stream fails and we want to re-establish it.

Probably not worth doing as part of this to keep the scope small but it would be good to debate, why we are retrying to establish the stream if it failed. Any form of loop that retries something is always a bit sus in my eyes. We could as well rely on an upper layer to re-establish the connection. I don't see why trying 5 times to re-establish the stream is fundamentally different to just disabling ourselves after the first one.

Yeah agree we should try it on a subsequent PR

and clone it per ConnectionHandler, this will allow us to always have an open Receiver.

jxs

did not forget about Thomas remark on https://github.com/libp2p/rust-libp2p/pull/4914/files#r1402807803. I am just trying to close the api so that we know what we need from the updated tests

protocols/gossipsub/src/behaviour.rs

jxs · 2023-11-25T22:46:03Z

protocols/gossipsub/src/behaviour.rs

            let event = RpcOut::Subscribe(topic_hash.clone());
-            self.send_message(peer, event);
+            self.send_message(peer, event)
+                .expect("Subscribe messages should be always sent");


I was following @AgeManning on #4667 (comment) but you are probably right Max, updated the code to implement Thomas suggested methods into RpcSender which allow us to not split RpcOut and meanwhile also remove send_message and the double iterations on Behaviour, ptal Thomas

to allow for better handling of each message send.

…-backpressure-cont

protocols/gossipsub/CHANGELOG.md

mxinden · 2023-12-11T12:02:10Z

protocols/gossipsub/Cargo.toml

@@ -37,12 +37,13 @@ sha2 = "0.10.8"
 smallvec = "1.11.2"
 tracing = "0.1.37"
 void = "1.0.2"
+async-channel = "1.9.0"


Still needed?

no no, thanks Max, updated

protocols/gossipsub/src/behaviour.rs

mxinden · 2023-12-11T12:15:40Z

protocols/gossipsub/src/behaviour.rs

+            .connected_peers
+            .get_mut(&peer_id)
+            .expect("Peerid should exist")
+            .clone();


Why is this clone needed?

it's not, thanks Max!

mxinden · 2023-12-11T12:17:49Z

protocols/gossipsub/src/types.rs

+    /// Send a `RpcOut::Subscribe` message to the `RpcReceiver`
+    /// this is high priority. By cloning `futures::channel::mpsc::Sender`
+    /// we get one extra slot in the channel's capacity.
+    pub(crate) fn subscribe(&mut self, topic: TopicHash) {


Should subscribe return a Result for the case where all channels to the remote peer are closed?

Shouldn't we assume that situation as unreachable? i.e. there will always be at least one connection id per peer? as we add them and remove them on new and closed connections.

If not all PeerConnections send methods should account for that right?

See also #4914 (comment).

mxinden · 2023-12-11T12:34:11Z

protocols/gossipsub/src/types.rs

+                }
+            }
+        }
+        unreachable!("At least one peer should be available");


Why is this unreachable? Say that all channels to the given peer are closed, the for loop above will not return and thus this unreachable is hit, no?

yeah, it's as I asked above, isn't the situation of having all channels to a given peer closed unreachable?
On handle_established_{inbound,outbound}_connection we add the PeerId and the ConnectionId to connected_peers, which is then removed on on_connection_closed removing the ConnectionId, and the PeerId if `remaining_established_ is zero.

Say that each connection has an error, e.g. here:

rust-libp2p/swarm/src/connection/pool/task.rs

Line 237 in c3500bb

Err(error) => {

Then each connection is going to close their channel.

Say that "the same" time (concurrently) the user calls Behaviour::publish.

In such case, the Behaviour would still track the connections, even though each channel would be dropped.

Does the above make sense?

mxinden · 2023-12-11T12:38:35Z

protocols/gossipsub/src/types.rs

+                Err(err) if err.is_full() => {
+                    tracing::trace!("Queue is full, dropped Forward message");
+                    return;
+                }
+                // Channel is closed, try another sender.
+                Err(err) => {
+                    rpc = err.into_inner();
+                }


Still have to give this more thought, though of the top of my head, trying a different connection in both cases is fine, no?

mxinden · 2023-12-11T12:38:54Z

protocols/gossipsub/src/types.rs

+    pub(crate) non_priority: Sender<RpcOut>,
+}
+
+/// `RpcOut` sender that is priority aware.


Suggested change

/// `RpcOut` sender that is priority aware.

/// `RpcOut` receiver that is priority aware.

thanks Max, addressed!

mergify · 2024-01-16T04:12:11Z

This pull request has merge conflicts. Could you please resolve them @jxs? 🙏

…-backpressure-cont

superseeds #4914 with some changes and improvements, namely: - introduce a `Delay` for `Forward` and `Publish` messages, messages that take more than the configured delay to be sent are discarded - introduce scoring and penalize slow peers - remove control pool - report slow peers with the number of failed messages Pull-Request: #5595.

drHuangMHT · 2024-11-29T06:20:16Z

@jxs Are we closing this? #5595 has been merged.

jxs · 2024-11-29T10:34:49Z

@jxs Are we closing this? #5595 has been merged.

yup thanks @drHuangMHT!
Superseeded by #5595

superseeds libp2p#4914 with some changes and improvements, namely: - introduce a `Delay` for `Forward` and `Publish` messages, messages that take more than the configured delay to be sent are discarded - introduce scoring and penalize slow peers - remove control pool - report slow peers with the number of failed messages Pull-Request: libp2p#5595.

@folex

* Use newer version for rcgen enable RISC-V builds. * Fix deprecation warnings for errors due to move to newer rcgen * Fix rcgen version in webrtc * chore: parameterise s3 build cache setup (libp2p#5586) As we're setting up a new cache bucket, we'd like to be able to control its' configuration via GitHub vars/secrets fully. FYI, the secrets are not set up yet. --------- Co-authored-by: João Oliveira <[email protected]> Co-authored-by: Guillaume Michel <[email protected]> * fix(autonat): reject inbound dial request from peer if its not connected (libp2p#5597) ## Description As discovered and described in the issue below, there are situations where an incoming AutoNAT dial can come from a non-connected peer. However `resolve_inbound_request` expects that this situation cannot occur. This PR adds a check upfront and refuses the incoming dial when no connected peer is found. Fixes libp2p#5570. ## Change checklist - [x] I have performed a self-review of my own code - [x] I have made corresponding changes to the documentation - [ ] I have added tests that prove my fix is effective or that my feature works - [x] A changelog entry has been made in the appropriate crates Co-authored-by: João Oliveira <[email protected]> * chore(ci): only run interop tests on commits to master (libp2p#5604) ## Description This is done as temporary measure to unblock PR merging as the CI is currently broken Co-authored-by: Guillaume Michel <[email protected]> * fix(ci): address cargo-deny advisories (libp2p#5596) ## Description by updating: - `bytes` to 1.7.1, `1.6.0` was [yanked](https://crates.io/crates/bytes/1.6.0) - `quinn-proto` to 0.11.8 to address [RUSTSEC-2024-0373](https://rustsec.org/advisories/RUSTSEC-2024-0373.html) - thirtyfour-macros to 0.1.11 to remove `proc-macro-error` dependency and address [RUSTSEC-2024-0370](https://rustsec.org/advisories/RUSTSEC-2024-0370.html) * chore(ci): address beta clippy lints (libp2p#5606) Co-authored-by: Darius Clark <[email protected]> * fix(swarm): don't report `NewExternalAddrCandidate` if already confirmed (libp2p#5582) ## Description Currently, `NewExternalAddrCandidate` events are emitted for every connections. However, we continue to get this event even when `autonat` has already confirmed that this address is external. So we should not continue to advertise the "candidate" event. ## Notes & open questions We have made the changes in the `swarm` instead of `identify` because it does not make it necessary to duplicate the `ConfirmedExternalAddr` vector in the `identify` Behaviour. Moreover, if any future Behaviour emit `NewExternalAddrCandidate`, the same rule will also be applied. I had to edit the `autonat_v2` tests which were always expecting a `NewExternalAddrCandidate` but the address was already confirmed. ## Change checklist  - [x] I have performed a self-review of my own code - [ ] I have made corresponding changes to the documentation - [ ] I have added tests that prove my fix is effective or that my feature works - [x] A changelog entry has been made in the appropriate crates --------- Co-authored-by: Darius Clark <[email protected]> Co-authored-by: Guillaume Michel <[email protected]> * chore(autonat-v2): fix dial_back_to_non_libp2p test (libp2p#5621) ## Description By avoiding dialing an external address Co-authored-by: Guillaume Michel <[email protected]> * chore: update interop test run condition (libp2p#5611) ## Description Follow up to libp2p#5604. Interop tests only work on the main `rust-libp2p` repo, and not on forks, because of the S3 cache (introduced in libp2p#5586). The interop tests currently don't run in the PRs, but they run after the PRs are merged to `master`. This PR is trying to run interop tests in PR that are branches of the main repo (not forks). * deps: update metrics example dependencies (libp2p#5617) ## Description and address cargo-deny [RUSTSEC-2024-0376](https://rustsec.org/advisories/RUSTSEC-2024-0376.html) * refactor(examples): use tokio instead of async-std in relay-server (libp2p#5600) ## Description  Following on issue libp2p#4449 refactor: use tokio instead of async-std in the relay-servert example and remove unnecessary dependencies ## Notes & open questions  Fails on testing with the [whole punch tutorial](https://docs.rs/libp2p/0.54.1/libp2p/tutorials/hole_punching/index.html) possibly because of my networking topology. connection established event registered. I will publish the logs and testing information as a comment ## Change checklist * Removed unnecessary dependencies on examples/relay-server/Cargo.toml * Updated tokio version to "1.37.0"  - [ ] I have performed a self-review of my own code - [ ] I have made corresponding changes to the documentation - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] A changelog entry has been made in the appropriate crates --------- Co-authored-by: David E. Perez Negron R. <[email protected]> Co-authored-by: Guillaume Michel <[email protected]> Co-authored-by: João Oliveira <[email protected]> * fix(server): removing dependency on libp2p-lookup (libp2p#5610) ## Description Remove dependency on [`libp2p-lookup`](https://github.com/mxinden/libp2p-lookup) which is no longer maintained, and [makes checks fail](https://github.com/libp2p/rust-libp2p/actions/runs/11016492372/job/30628121728). ## Change checklist  - [x] I have performed a self-review of my own code - [x] I have made corresponding changes to the documentation - [ ] I have added tests that prove my fix is effective or that my feature works - [x] A changelog entry has been made in the appropriate crates --------- Co-authored-by: João Oliveira <[email protected]> * chore: update igd-next to 0.15.1 (libp2p#5625) ## Description  Resolves libp2p#5506. ## Notes & open questions  ## Change checklist  - [ ] I have performed a self-review of my own code - [ ] I have made corresponding changes to the documentation - [ ] I have added tests that prove my fix is effective or that my feature works - [x] A changelog entry has been made in the appropriate crates * deps: bump Swatinem/rust-cache from 2.7.3 to 2.7.5 (libp2p#5633) Bumps [Swatinem/rust-cache](https://github.com/swatinem/rust-cache) from 2.7.3 to 2.7.5. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/swatinem/rust-cache/releases">Swatinem/rust-cache's releases</a>.</em></p> <blockquote> <h2>v2.7.5</h2> <h2>What's Changed</h2> <ul> <li>Upgrade checkout action from version 3 to 4 by <a href="https://github.com/carsten-wenderdel"><code>@carsten-wenderdel</code></a> in <a href="https://redirect.github.com/Swatinem/rust-cache/pull/190">Swatinem/rust-cache#190</a></li> <li>fix: usage of <code>deprecated</code> version of <code>node</code> by <a href="https://github.com/hamirmahal"><code>@hamirmahal</code></a> in <a href="https://redirect.github.com/Swatinem/rust-cache/pull/197">Swatinem/rust-cache#197</a></li> <li>Only run macOsWorkaround() on macOS by <a href="https://github.com/heksesang"><code>@heksesang</code></a> in <a href="https://redirect.github.com/Swatinem/rust-cache/pull/206">Swatinem/rust-cache#206</a></li> <li>Support Cargo.lock format cargo-lock v4 by <a href="https://github.com/NobodyXu"><code>@NobodyXu</code></a> in <a href="https://redirect.github.com/Swatinem/rust-cache/pull/211">Swatinem/rust-cache#211</a></li> </ul> <h2>New Contributors</h2> <ul> <li><a href="https://github.com/carsten-wenderdel"><code>@carsten-wenderdel</code></a> made their first contribution in <a href="https://redirect.github.com/Swatinem/rust-cache/pull/190">Swatinem/rust-cache#190</a></li> <li><a href="https://github.com/hamirmahal"><code>@hamirmahal</code></a> made their first contribution in <a href="https://redirect.github.com/Swatinem/rust-cache/pull/197">Swatinem/rust-cache#197</a></li> <li><a href="https://github.com/heksesang"><code>@heksesang</code></a> made their first contribution in <a href="https://redirect.github.com/Swatinem/rust-cache/pull/206">Swatinem/rust-cache#206</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/Swatinem/rust-cache/compare/v2.7.3...v2.7.5">https://github.com/Swatinem/rust-cache/compare/v2.7.3...v2.7.5</a></p> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/Swatinem/rust-cache/commit/82a92a6e8fbeee089604da2575dc567ae9ddeaab"><code>82a92a6</code></a> 2.7.5</li> <li><a href="https://github.com/Swatinem/rust-cache/commit/598fe25fa107b2fd526fc6471f6e48de7cd12083"><code>598fe25</code></a> update dependencies, rebuild</li> <li><a href="https://github.com/Swatinem/rust-cache/commit/8f842c2d455cfe3d0d5a4b28f53f5389b51b71bf"><code>8f842c2</code></a> Support Cargo.lock format cargo-lock v4 (<a href="https://redirect.github.com/swatinem/rust-cache/issues/211">#211</a>)</li> <li><a href="https://github.com/Swatinem/rust-cache/commit/96a8d65dbafbc7d145a9b2b6c3b12ee335738cd2"><code>96a8d65</code></a> Only run macOsWorkaround() on macOS (<a href="https://redirect.github.com/swatinem/rust-cache/issues/206">#206</a>)</li> <li><a href="https://github.com/Swatinem/rust-cache/commit/9bdad043e88c75890e36ad3bbc8d27f0090dd609"><code>9bdad04</code></a> fix: usage of <code>deprecated</code> version of <code>node</code> (<a href="https://redirect.github.com/swatinem/rust-cache/issues/197">#197</a>)</li> <li><a href="https://github.com/Swatinem/rust-cache/commit/f7a52f691454d93c6ce0dff6666a5cb399b8d06e"><code>f7a52f6</code></a> "add jsonpath test"</li> <li><a href="https://github.com/Swatinem/rust-cache/commit/2bceda39122b2cc71e6e26ad729b92b44d101f4b"><code>2bceda3</code></a> "update dependencies"</li> <li><a href="https://github.com/Swatinem/rust-cache/commit/640a22190e7a783d4c409684cea558f081f92012"><code>640a221</code></a> Upgrade checkout action from version 3 to 4 (<a href="https://redirect.github.com/swatinem/rust-cache/issues/190">#190</a>)</li> <li><a href="https://github.com/Swatinem/rust-cache/commit/158274163087d4d4d49dfcc6a39806493e413240"><code>1582741</code></a> update dependencies</li> <li>See full diff in <a href="https://github.com/swatinem/rust-cache/compare/23bce251a8cd2ffc3c1075eaa2367cf899916d84...82a92a6e8fbeee089604da2575dc567ae9ddeaab">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=Swatinem/rust-cache&package-manager=github_actions&previous-version=2.7.3&new-version=2.7.5)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) You can trigger a rebase of this PR by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feat: make runtime features optional in swarm-test (libp2p#5551) ## Description  Sometimes a test uses custom swarm building logic and doesn't need `fn new_ephemeral`, and sometimes a test uses `tokio` runtime other than `async-std`. This PR adds the `tokio` runtime support and makes both `async-std` and `tokio` runtimes optional behind features to make it more flexible. ## Notes & open questions  ## Change checklist  - [x] I have performed a self-review of my own code - [x] I have made corresponding changes to the documentation - [ ] I have added tests that prove my fix is effective or that my feature works - [x] A changelog entry has been made in the appropriate crates --------- Co-authored-by: João Oliveira <[email protected]> * chore: fix typo in comment (libp2p#5643) ## Description  ## Notes & open questions  ## Change checklist  - [x] I have performed a self-review of my own code - [x] I have made corresponding changes to the documentation - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] A changelog entry has been made in the appropriate crates * deps(ci): update cargo-semver-checks (libp2p#5647) * fix(swarm-test): set proper version (libp2p#5648) ## Description To unblock CI Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * feat(kad): add `Behavior::find_closest_local_peers()` (libp2p#5645) ## Description Fixes libp2p#5626 ## Notes & open questions This is the nicest way I came up with, I decided to leave `get_closest_local_peers` as is since it does return all peers, not just `replication_factor` peers. Looking at libp2p#2436 it is not clear if @folex really needed all peers returned or it just happened that way. I'm also happy to change proposed API to return all peers if that is preferred by others. It is very unfortunate that `&mut self` is needed for this, I created libp2p#5644 that if resolved will allow to have `&self` instead. ## Change checklist - [x] I have performed a self-review of my own code - [x] I have made corresponding changes to the documentation - [ ] I have added tests that prove my fix is effective or that my feature works - [x] A changelog entry has been made in the appropriate crates Co-authored-by: João Oliveira <[email protected]> * feat(gossipsub): apply `max_transmit_size` to the published message (libp2p#5642) ## Description When trying to publish a message using gossipsub's `publish` method, it should be possible to predict whether it will fit in the limit defined by the `max_transmit_size` config option. If this limit applies to the final protobuf payload, it's not possible to know that in advance because the size of the added fields is not fixed. This change makes the limit apply to the passed message size instead of the final wire size. ## Notes & open questions This is a minor version change because it changes the meaning of the existing config option. However, for the existing clients the limit will only become more permissive, so it shouldn't break anything. ## Change checklist  - [x] I have performed a self-review of my own code - [x] I have made corresponding changes to the documentation - [ ] I have added tests that prove my fix is effective or that my feature works - [x] A changelog entry has been made in the appropriate crates --------- Co-authored-by: Darius Clark <[email protected]> Co-authored-by: João Oliveira <[email protected]> * chore(ci): address clippy beta lints (libp2p#5649) Co-authored-by: Darius Clark <[email protected]> * feat: refactor distributed-key-value-store example (libp2p#5652) ## Description ref libp2p#4449 Refactored distributed-key-value-store example to use `tokio` instead of `async-std` ## Change checklist  - [x] I have performed a self-review of my own code - [x] I have made corresponding changes to the documentation - [x] I have added tests that prove my fix is effective or that my feature works - [x] A changelog entry has been made in the appropriate crates * chore: refactor ping tests (libp2p#5655) ## Description ref libp2p#4449 Refactored ping tests to use `tokio` instead of `async-std`. ## Change checklist - [x] I have performed a self-review of my own code - [x] I have made corresponding changes to the documentation - [x] I have added tests that prove my fix is effective or that my feature works - [x] A changelog entry has been made in the appropriate crates * fix(websocket): don't dial `/dnsaddr` addresses (libp2p#5613) ## Description Returns `Error::InvalidMultiaddr` when `parse_ws_dial_addr` is called with `/dnsaddr`. As per its specification, `/dnsaddr` domains are not meant to be directly dialed, instead it should be appended with `_dnsaddr.` and used for DNS lookups afterwards Related: libp2p#5529 Fixes: libp2p#5601 ## Notes & open questions * Is it okay to return an error, or should I perform a DNS lookup and resolve that DNS afterwards if address has `/dnsaddr`? * If so, how should I handle that case where DNS lookup returns multiple multiaddrs? ## Change checklist - [x] I have performed a self-review of my own code - [ ] I have made corresponding changes to the documentation - [x] I have added tests that prove my fix is effective or that my feature works - [x] A changelog entry has been made in the appropriate crates --------- Co-authored-by: Darius Clark <[email protected]> * chore: fix some comments (libp2p#5661) ## Description  ## Notes & open questions  ## Change checklist  - [x] I have performed a self-review of my own code - [ ] I have made corresponding changes to the documentation - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] A changelog entry has been made in the appropriate crates --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * chore: identify::Config fields private (libp2p#5663) ## Description Closes libp2p#5660 ## Change checklist - [x] I have performed a self-review of my own code - [x] I have made corresponding changes to the documentation - [x] I have added tests that prove my fix is effective or that my feature works - [x] A changelog entry has been made in the appropriate crates * chore(protocols): fix some typos in comment (libp2p#5665) ## Description  fix some typos in comment ## Notes & open questions  ## Change checklist  - [ ] I have performed a self-review of my own code - [ ] I have made corresponding changes to the documentation - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] A changelog entry has been made in the appropriate crates Signed-off-by: wangjingcun <[email protected]> * chore(ci): fix interop tests region, and run them again on each PR (libp2p#5666) * chore: deprecate `void` crate (libp2p#5676) ## Description  The `void` crate provides a `Void` type that is conceptually equivalent to the [`never` type(!)](https://doc.rust-lang.org/std/primitive.never.html). This PR tries to remove `void` crate from the dependency tree by replacing `void::Void` with [`std::convert::Infallible`](https://doc.rust-lang.org/std/convert/enum.Infallible.html) that will eventually become an alias of the `never` type(!) > This enum has the same role as [the ! “never” type](https://doc.rust-lang.org/std/primitive.never.html), which is unstable in this version of Rust. When ! is stabilized, we plan to make Infallible a type alias to it: ## Notes & open questions  ## Change checklist  - [x] I have performed a self-review of my own code - [ ] I have made corresponding changes to the documentation - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] A changelog entry has been made in the appropriate crates * chore: replace async-std with tokio in autonat tests (libp2p#5671) ## Description ref libp2p#4449 Refactored `autonat` tests to use `tokio` instead of `async-std`. * chore: bump crate versions and update changelogs for libp2p#5676 (libp2p#5678) ## Description  This PR bumps crate versions and add changelog entries for crates that are changed in libp2p#5676 Question: When should a crate version bump in the current release process? Should it be right before or right after publishing? I see most of current crate versions are published while some are not (e.g. [email protected] [email protected] and [email protected] etc.) ## Notes & open questions  ## Change checklist  - [x] I have performed a self-review of my own code - [ ] I have made corresponding changes to the documentation - [ ] I have added tests that prove my fix is effective or that my feature works - [x] A changelog entry has been made in the appropriate crates --------- Co-authored-by: João Oliveira <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * chore(ci): add a mergify batch queue for external PRs (libp2p#5668) and remove the if conditions no longer required as we are running them again on every PR --------- Co-authored-by: Guillaume Michel <[email protected]> Co-authored-by: João Oliveira <jxs> * chore: refactor dcutr and gossipsub tests to use tokio instead ref libp2p#4449 Refactored dcutr and gossipsub tests to use `tokio` instead of `async-std`. Pull-Request: libp2p#5662. * chore(gossispsub): deprecate futures-ticker to address [RUSTSEC-2024-0384 ](https://rustsec.org/advisories/RUSTSEC-2024-0384.html). Use `futures-timer` and `Delay` instead Pull-Request: libp2p#5674. * chore: update FUNDING.json I've added addresses to receive Drips donations. This is required for FIL-RetroPGF-2 in the coming days, we should get this into `master` ASAP. This is a dedicated address controlled by Sigma Prime (including @AgeManning and myself). Pull-Request: libp2p#5685. * feat(gossipsub): introduce backpressure superseeds libp2p#4914 with some changes and improvements, namely: - introduce a `Delay` for `Forward` and `Publish` messages, messages that take more than the configured delay to be sent are discarded - introduce scoring and penalize slow peers - remove control pool - report slow peers with the number of failed messages Pull-Request: libp2p#5595. * chore(deps): upgrade `thiserror` to 2.0 Changes: - upgrade `thiserror` crate from `1` to `2` - move `thiserror` to `workspace.dependencies` - sort `workspace.dependencies` - ~run `cargo update` to update `Cargo.lock`~ (Skipping changelog as `thiserror` does not present in any public APIs) Pull-Request: libp2p#5689. * fix: typos in documentation files Pull-Request: libp2p#5693. * fix(gossipsub): fix mesh/fanout inconsistencies When a peer unsubscribes also remove it from fanout. Pull-Request: libp2p#5690. * fix(ci): Clippy Beta Fixes CI failure in Clippy (Beta) https://github.com/libp2p/rust-libp2p/actions/runs/12055058396/job/33614543029 Pull-Request: libp2p#5700. * chore: introduce rustfmt.toml Pull-Request: libp2p#5695. * feat(kad): make Distance private field public make Distance private field (U256) public So that some `network density` in Distance can be calculated as below: ```rust let density = U256::MAX / U256::from(estimated_network_size); let density_distance = Distance(estimated_distance); ``` Pull-Request: libp2p#5705. * chore: fix some typos in comment fix some typos in comment Pull-Request: libp2p#5721. * fix(libp2p): expose builder phase error May close libp2p#4829 and libp2p#4824. Export three error types `BehaviourError`, `TransportError`, `WebsocketError` with rename. Feature gated `WebsocketError`. Exported at crate root as [suggested](libp2p#4824 (comment)). Pull-Request: libp2p#5726. * feat(request-response): Add connection id to behaviour events Closes libp2p#5716. Added connection id to the events emitted by a request-response Behaviour and adapted the code accordingly. Pull-Request: libp2p#5719. * chore(ci): update Rust stable version Update Rust stable version in our CI to the latest stable version 1.83.0. Pull-Request: libp2p#5730. * chore(roadmap): fix typo Pull-Request: libp2p#5732. * chore(deps): upgrade uint to 0.10 This PR upgrade `uint` to `0.10` https://github.com/paritytech/parity-common/blob/master/uint/CHANGELOG.md#0100---2024-09-11 (Skipping changelog as there's no changes in public APIs) Pull-Request: libp2p#5699. * fix: RUSTSEC-2024-0421 by upgrading idna https://rustsec.org/advisories/RUSTSEC-2024-0421.html Pull-Request: libp2p#5727. * deps(metrics-example): update opentelemetry to 0.27 this will help fix the `cargo deny` situation as `opentelemetry-otlp` `0.25` has `tokio` [locked to `~1.38.0`](https://crates.io/crates/opentelemetry-otlp/0.25.0/dependencies) 🤷‍♂️ which then impedes us tfrom updating `netlink-sys` Pull-Request: libp2p#5735. * fix: update Cargo.lock To finally address [RUSTSEC-2024-0384](https://rustsec.org/advisories/RUSTSEC-2024-0384). Thanks @hanabi1224 for submitting the [upstream update](rust-netlink/netlink-sys#25) ❤️ Pull-Request: libp2p#5737. * chore: add Unicode V3 license to deny.toml Add `Unicode` `v3` License to our `cargo.deny` file. This is required because of the [`icu_collections`](https://crates.io/crates/icu_collections) dependency which is authored by The Unicode Consortium. in my ignorance, seems to be safe as the Open Source Initiative [approves it](https://opensource.org/license/unicode-license-v3), and the main `deny.toml` has also [added it](https://github.com/EmbarkStudios/cargo-deny/pull/713/files#diff-1040309c64844eb1b6b63d8fd67938adbf9461f1b3c61f12cf738f064a02d3deR50). Pull-Request: libp2p#5738. * chore(core): avoid unused props matching on connection.rs Pull-Request: libp2p#5734. * feat(swarm): set default for idle-connection-timeout to 10s (libp2p#4967) ## Description With the move to a global idle-connection-timeout, connections are being closed much more aggressively. This causes problems in situations where, e.g. an application wants to use a connection shortly after an event has been emitted from the `Swarm`. With a default of 0 seconds, such a connection is instantly considered idle and therefore closed, despite the application wanting to use it again just moments later. Whilst it is possible to structure application code to mitigate this, it is unnecessarily complicated. Additionally, connections being closed instantly if not in use is a foot-gun for newcomers to the library. From a technical point-of-view, instantly closing idle connections is nice. In reality, it is an impractical default. Hence, we change this default to 10s. 10 seconds is considered to be an acceptable default as it strikes a balance between allowing some pause between network activity, yet frees up resources that are (supposedly) no longer needed. Resolves: libp2p#4912. * chore(deps): bump golang.org/x/crypto from 0.21.0 to 0.31.0 (libp2p#5736) Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.21.0 to 0.31.0. for the wasm webtransport tests * fix(identify): validate public key from remote peer Related libp2p#5706 Discard `Info` messages received from a remote peer that contain a public key that doesn't match their peer ID, and log a warning. Don't emit `identify::Received` events to the swarm containing whatever public key they sent. Pull-Request: libp2p#5707. * deps(memory-connection-limits): update sysinfo to 0.33 Pull-Request: libp2p#5739. * feat(SwarmBuilder): add with_connection_timeout method Small PR to be able to change the `connection_timeout` value given to the `TransportTimeout` when building the `Swarm` with a `SwarmBuilder`. Pull-Request: libp2p#5575. * chore(core): use `matches!` in connection.rs Pull-Request: libp2p#5740. * chore: fix format with nightly Fix formatting with nightly. Without nightly, some of our `rustfmt.toml` rules aren't applied because they are not stable yet (e.g. our `group_imports` rule). Our CI didn't catch it yet because it doesn't check fmt on nightly. Will fix that in a follow-up PR. Pull-Request: libp2p#5742. * chore(docs): fix typos in documentation Pull-Request: libp2p#5744. * chore(ci): check rustfmt with nightly toolchain Some of our rustfmt rules aren't stable yet. Thus our rustfmt CI check should run on nightly. Pull-Request: libp2p#5743. * fix(kad): improve memory allocation when iterating over kbuckets Proposal to fix libp2p#5712. I have changed to `ClosestIter` structure to only allocate when `kbucket_size` is higher than `K_VALUE` and only once along the life of `ClosestIter`. I think I did not break anything but I would really like some experienced people with Kademlia to take a look (@guillaumemichel 😉). Pull-Request: libp2p#5715. * chore(spellchecker): fix typos in comment - use spellchecker, and check false positive manually ``` codespell -L crate,COMIT,comit -S CHANGELOG.md ``` - maybe we can add typo checker in github action ref: https://github.com/codespell-project/actions-codespell Pull-Request: libp2p#5750. * chore: add SQD Network to notable users list SQD Network uses Libp2p and gossipsub protocol in particular to organize a decentralized network of p2p workers that provide fast access to stored data. Pull-Request: libp2p#5749. * feat(mdns): emit `ToSwarm::NewExternalAddrOfPeer` on discovery fixes libp2p#5104 and superseeds libp2p#5179 Pull-Request: libp2p#5753. * fix: multiple typos of different importance This pull request addresses several typos and minor grammatical inconsistencies found across multiple files in the repository. The fixes aim to improve clarity, readability, and maintain consistency in the project's documentation and workflows. ### Key Changes: - **`.github/ISSUE_TEMPLATE/bug_report.yml`**: - Fixed grammatical issues in descriptions and labels. - **`.github/ISSUE_TEMPLATE/enhancement.yml`**: - Corrected label text and standardized phrasing. - **`.github/ISSUE_TEMPLATE/feature_request.yml`**: - Improved label consistency and readability. - **`.github/pull_request_template.md`**: - Enhanced phrasing and corrected minor grammatical errors. - **`.github/workflows/ci.yml`**: - Adjusted text to ensure consistent phrasing and clarity in workflow steps. These updates ensure a more professional and coherent presentation across the project documentation and workflows. Pull-Request: libp2p#5756. * chore(deps): remove unused deps Remove unused dependencies (Not sure how to bump `libp2p-identity` and `libp2p-swarm-derive` versions in a PR) Pull-Request: libp2p#5747. * chore(mdns): revert version bump Revert version bump of libp2p#5753, because libp2p-mdns-0.46.1 isn't released yet. Pull-Request: libp2p#5762. * deps(quic): update quinn to 0.11.6 Among other changes, it includes a fix for this issue which often reproduces with libp2p: quinn-rs/quinn#1889 Pull-Request: libp2p#5757. * chore: introduce `libp2p-test-utils` Fixes libp2p#4992 Based on the conversation in libp2p#4992. Pull-Request: libp2p#5725. * feat(gossipsub): implement gossipsub 1.2 beta This PR implements gossipsub 1.2 beta bringing changes over from lighthouse ref PR: sigp/lighthouse#5422 Please include any relevant issues in here, for example: libp2p/specs#548 Pull-Request: libp2p#5697. * chore(deps): update Cargo.lock - run `cargo update` - lock `webrtc-ice = "=0.10.0"` to not break webrtc smoke tests - fix `cargo clippy` warnings - update `deny.toml` accordingly Pull-Request: libp2p#5755. * feat(gossipsub): Allow setting a size threshold for IDONTWANT messages This PR adds configurable parameter that sets minimum message size for which `IDONTWANT` messages would be send. This is an optimisation trick, discussion regarding the same can be found [here](sigp/lighthouse#6437) Pull-Request: libp2p#5770. * chore(kad): revert version bump Revert version bump, because `libp2p-kad-0.47.0` isn't released yet. Pull-Request: libp2p#5776. * chore(autonat): revert version bump Revert version bump, `libp2p-autonat-v0.13.1` isn't released yet. Pull-Request: libp2p#5777. * chore(server): revert version bump Revert version bump, `libp2p-server-v0.12.6` isn't released yet. Pull-Request: libp2p#5780. * chore(identify): revert version bump Revert version bump, `libp2p-identify-v0.46.0` isn't released yet. Pull-Request: libp2p#5778. * chore(ci): add Zlib to deny.toml Similar to libp2p#5738, [`foldhash`](https://crates.io/crates/foldhash) is the dependency that requires it. It was introduced by `hashbrown` which is dependency of a lot of crates [here](rust-lang/hashbrown#563). `hashbrown` is MIT, and Zlib is also an Open Source Initiative [approved license](https://opensource.org/license/zlib). If you prefer we can also do as the upstream `cargo-deny` do and just add `foldhash` [to the exceptions](https://github.com/EmbarkStudios/cargo-deny/pull/618/files#diff-1040309c64844eb1b6b63d8fd67938adbf9461f1b3c61f12cf738f064a02d3deR56) but I can't see any advantage to it. Cc @hanabi1224 Pull-Request: libp2p#5769. * chore(allow-block-list): revert version bump Revert version bump, `libp2p-allow-block-list-v0.4.1` isn't released yet. Pull-Request: libp2p#5779. * chore(kad): remove default constructor for ProtocolConfig Remove items that were deprecated in libp2p#5122. Pull-Request: libp2p#5774. * feat: broadcasting `idontwant` for published messages This PR introduces an optimisation to preemptively broadcast `IDONTWANT` for published messages, preventing redundant downloads of the same message received over gossip. This feature is opt-in and respects the existing `idontwant_message_size_threshold`. By default, `IDONTWANT` messages will only be sent for published messages larger than 1000 bytes. reference PRs: [spec](libp2p/specs#642) Pull-Request: libp2p#5773. * fix(gossipsub): make sure we have fanout peers when publish This PR is to submit a fix in Lighthouse. sigp/lighthouse#6738 An `InsufficientPeers` error can occur under a particular condition, even if we have peers subscribed to a topic. Pull-Request: libp2p#5793. * feat(request-response): allow custom sizes for `json` and `cbor` codec resolves libp2p#5791. Pull-Request: libp2p#5792. * chore: update MSRV Pull-Request: libp2p#5650. * chore: address clippy lints for Rust 1.85.0-beta Pull-Request: libp2p#5802. * deps: bump Swatinem/rust-cache from 2.7.5 to 2.7.7 Pull-Request: libp2p#5775. * refactor: mark {In, Out}boundOpenInfo as deprecated Mark `{In, Out}boundOpenInfo` as deprecated. May close libp2p#3268. Pull-Request: libp2p#5242. * chore(identify): fix changelog entry Amendment to libp2p#5778. Merged CHANGELOG entries of `0.45.1` and `0.46.0`. Version `0.45.1` has been superseded by `0.46.0` before it was released. Pull-Request: libp2p#5803. * update webrtc crate * update CHANGELOG.md * fix websocket doc * chore(ci): fix wasm tests see here rust-lang/compiler-team#607 for more info Pull-Request: libp2p#5805. * chore(swarm): append missing changelog Add missing changelog entry for libp2p#5242. Pull-Request: libp2p#5806. * ci(mergify): upgrade configuration to current format Pull-Request: libp2p#5683. * fix(tcp): make `TCP_NODELAY` actually the default Nagle's algorithm is actually disabled by setting `TCP_NODELAY` to _true_. In the current master, it is set to _false_, i.e. Nagle's algorithm is enabled. This PR sets `TCP_NODELAY` per default to `true` and fixes the description of `tcp::Config::new`. Fixes libp2p#4890. Pull-Request: libp2p#5764. * chore(ci): update mergify script With libp2p#5683 the automatically merge of `send-it` labeled PR's was lost. This PR brings it back Pull-Request: libp2p#5808. * chore(interop-tests): remove static linking flags for dependency builds When building the dependencies, we are not linking anything so we don't need to specify the static linking flags. Pull-Request: libp2p#5121. * deps(if-watch): update if-watch to v3.2.1 Bumps the `if-watch` version to 3.2.1, aiming to fix libp2p#5628. Since this [`if-watch` PR](libp2p/if-watch#37), the `system-configuration` version has been updated, which enables compiling for iOS targets. Pull-Request: libp2p#5758. * chore(ci): update mergify script missed this, with this dependabot is not able to merge PR's by itself as it needs the `send-it` label. also add a clause for the dismiss of stale approvals to not do so when the last commit was done by one of the `rust-libp2p` maintainers, so that we don't dismiss approvals if the last commit was a merge of master into the PR that is necessary before merging the PR. Latest example of this happening [here](libp2p#5758 (comment)) Pull-Request: libp2p#5809. * chore(test-utils): revert libp2p#5725 We cannot publish the crates using `libp2p-test-utils`, as cargo seems to required `dev-dependencies` to also be published Pull-Request: libp2p#5810. * fix: minor version bump packages affected by libp2p#5676 By switching from `void::Void` to `std::convert::Infallible`, libp2p#5676 changed the output types of some `NetworkBehavior` implementations in different protocols. This can cause a type mismatch for the user and therefore is a breaking change. libp2p#5678 (follow-up PR that version-bumped the crates affected by libp2p#5676) only bumped the patch version of the affected crates. The current PR now changes it to a minor version bump for all crates where types in a `NetworkBehavior` implementation were affected. It also reverts the version bump and CHANGELOG entry in `libp2p-quic` that was added with libp2p#5678, because that crate was never touched by the original PR. Pull-Request: libp2p#5811. * chore: prepare libp2p-webrtc-websys 0.4.0 Pull-Request: libp2p#5807. * chore(*): prepare release Bump minor version of all crates that depend on `libp2p-core`, to avoid conflicting versions or `libp2p-core` being pulled in multiple times. Pull-Request: libp2p#5813. * chore(ci): fix workspace version enforcing job Cue due to an [update](https://crates.io/crates/tomlq/0.2.0) to the `tomlq` tool, there's now a mismatch between the `$CRATE_VERSION` and the `$SPECIFIED_VERSION` variables: ```bash Package version: 0.43.0 Specified version: "0.43.0" ``` This PR attempts to fix the "Enforce version in `workspace.dependencies` matches the latest version" job by locking to the previous version until (and if) the situation is solved. Also see cryptaliagy/tomlq#22 Pull-Request: libp2p#5817. * refactor(gossipsub): remove duplicated call to `inbound_transform` May close libp2p#4369. Pull-Request: libp2p#5767. * chore(ci): update tomlq and address the version lock introduced on libp2p#5817 Pull-Request: libp2p#5821. * WIP allow test runs manually --------- Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: wangjingcun <[email protected]> Co-authored-by: haurog <[email protected]> Co-authored-by: Piotr Galar <[email protected]> Co-authored-by: João Oliveira <[email protected]> Co-authored-by: Guillaume Michel <[email protected]> Co-authored-by: Stefan <[email protected]> Co-authored-by: Darius Clark <[email protected]> Co-authored-by: stormshield-frb <[email protected]> Co-authored-by: P1R0 <[email protected]> Co-authored-by: David E. Perez Negron R. <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: hanabi1224 <[email protected]> Co-authored-by: yanziseeker <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Nazar Mokrynskyi <[email protected]> Co-authored-by: Dzmitry Kalabuk <[email protected]> Co-authored-by: Krishang Shah <[email protected]> Co-authored-by: Hamza <[email protected]> Co-authored-by: wangjingcun <[email protected]> Co-authored-by: Paul Hauner <[email protected]> Co-authored-by: leopardracer <[email protected]> Co-authored-by: maqi <[email protected]> Co-authored-by: needsure <[email protected]> Co-authored-by: DrHuangMHT <[email protected]> Co-authored-by: Bastien Faivre <[email protected]> Co-authored-by: Elena Frank <[email protected]> Co-authored-by: lfg2 <[email protected]> Co-authored-by: Sergey Melnychuk <[email protected]> Co-authored-by: Thomas Eizinger <[email protected]> Co-authored-by: James Hiew <[email protected]> Co-authored-by: X6 <[email protected]> Co-authored-by: Tristav <[email protected]> Co-authored-by: argentpapa <[email protected]> Co-authored-by: Dzmitry Kalabuk <[email protected]> Co-authored-by: hopinheimer <[email protected]> Co-authored-by: crStiv <[email protected]> Co-authored-by: Akihito Nakano <[email protected]> Co-authored-by: Breno Teodoro <[email protected]>

jxs requested a review from thomaseizinger November 22, 2023 23:30

jxs force-pushed the gossipsub-backpressure-cont branch from bbac280 to a85f1f8 Compare November 22, 2023 23:39

feat(gossipsub): introduce backpressure

a15e72b

jxs force-pushed the gossipsub-backpressure-cont branch from a85f1f8 to a15e72b Compare November 22, 2023 23:39

thomaseizinger reviewed Nov 23, 2023

View reviewed changes

jxs added 2 commits November 23, 2023 15:19

remove duplicated poll from RpcReceiver

76fb737

make RpcReceiver part of RpcSender,

dd13fcd

and clone it per ConnectionHandler, this will allow us to always have an open Receiver.

jxs commented Nov 25, 2023

View reviewed changes

split send_message into multiple RpcSender methods,

92a171c

to allow for better handling of each message send.

jxs force-pushed the gossipsub-backpressure-cont branch from 40e651c to 92a171c Compare November 27, 2023 15:54

jxs added 2 commits November 27, 2023 16:38

cargo clippy

547db09

add CHANGELOG.md entry

b08a4eb

AgeManning mentioned this pull request Nov 30, 2023

Large Memory Consumption Tracking Issue (OOM) sigp/lighthouse#4918

Closed

update to use futures::channel and connected_peers

da9008b

jxs force-pushed the gossipsub-backpressure-cont branch from bf76c6f to 4422990 Compare December 9, 2023 16:44

jxs requested a review from mxinden December 9, 2023 16:44

jxs force-pushed the gossipsub-backpressure-cont branch from 4422990 to 09143b5 Compare December 9, 2023 16:46

Merge branch 'master' of github.com:libp2p/rust-libp2p into gossipsub…

e421174

…-backpressure-cont

jxs force-pushed the gossipsub-backpressure-cont branch from 09143b5 to e421174 Compare December 9, 2023 16:47

mxinden reviewed Dec 11, 2023

View reviewed changes

jxs mentioned this pull request Dec 12, 2023

fix(gossipsub): send iwant messages immediately #4875

Closed

4 tasks

address review

8afbc4f

jxs added 3 commits August 22, 2024 11:32

Merge branch 'master' of github.com:libp2p/rust-libp2p into gossipsub…

cb1b29e

…-backpressure-cont

improve wording

946e1d5

clippy

863ba9f

jxs force-pushed the gossipsub-backpressure-cont branch from ffe9e74 to a9d74c2 Compare August 22, 2024 17:09

update cargo.toml

1072cc7

jxs force-pushed the gossipsub-backpressure-cont branch from a9d74c2 to 1072cc7 Compare August 22, 2024 17:14

jxs mentioned this pull request Sep 10, 2024

feat(gossipsub): introduce backpressure #5595

Merged

jxs closed this Nov 29, 2024

		let (priority_sender, priority_receiver) = async_channel::unbounded();
		let (non_priority_sender, non_priority_receiver) = async_channel::bounded(cap / 2);

	/// `RpcOut` sender that is priority aware.
	/// `RpcOut` receiver that is priority aware.

feat(gossipsub): introduce backpressure #4914

feat(gossipsub): introduce backpressure #4914

Conversation

jxs commented Nov 22, 2023 • edited Loading

Notes & open questions

Change checklist

thomaseizinger left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AgeManning Nov 27, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jxs Nov 23, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jxs left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mergify bot commented Jan 16, 2024

drHuangMHT commented Nov 29, 2024

jxs commented Nov 29, 2024

jxs commented Nov 22, 2023 •

edited

Loading

AgeManning Nov 27, 2023 •

edited

Loading

jxs Nov 23, 2023 •

edited

Loading