Program account notifications are not sent for every confirmed block #18587
Comments
|
This is probably not limited to program account subscriptions, but there are occasionally confirmed slots with program account changes which do not emit any notifications. I used a modified version of @Arrowana's demo which produced the below behavior on testnet and clearly shows that the confirmed slot 85270792 should have emitted notifications but didn't. Received notifications: Confirmed transactions: |
|
@CriesofCarrots do you have bandwidth to pick this up next week? |
jstarry
changed the title
|
Thanks for the report and the helpful example @Arrowana! |
|
Chiming in to share that we're seeing the same, incredibly-unreliable events coming into our |
ghost
mentioned this issue
|
@SkynetCapital if you can confirm whether the log events are unreliable in the same way as I described above, that would be really helpful! You could confirm by checking if all logs are received for some slots, but no logs are received for other slots. Thanks! |
Sure thing, I'll need a little work to return the slot from the response in my code. In the mean time, I ran this test which listened to my recipient pubkey with Also feel free to disregard the class/method names in the Logger, just re-used this test. |
|
cc @lijunwangs |
|
Looking into this. @Arrowana for the missing notifications for your subscriptions, do you know if the validator was running during when you were expecting the notifications? |
|
@lijunwangs I confirmed this behaviour and my validator was definitely running the whole time 😝 |
|
Thanks for confirming that. Runtime issue then. |
|
We have experienced similar issues when using the |
|
Running against my validator with testnet -- I see the following: |
…ribe (#19092) #18587: programSubscribe is missing notifications randomly. The issue is because of two reasons Not all rooted slots get OptimisticallyConfirmed notifications The OptimisticallyConfirmed notifications can be out of order for slots: slot A and B with A < B can see notification for B first before A. Summary of Changes Changed OptimisticallyConfirmedBankTracker to send notifications for parent banks if they have not been notified yet. We use a new variable last_notified_slot to track that. Tests: With my validator running against testnet, before the fix, it was failing 75% of time, with the fix, it is passing consistently. Using the program mentioned in #18587.
…ribe (#19092) #18587: programSubscribe is missing notifications randomly. The issue is because of two reasons Not all rooted slots get OptimisticallyConfirmed notifications The OptimisticallyConfirmed notifications can be out of order for slots: slot A and B with A < B can see notification for B first before A. Summary of Changes Changed OptimisticallyConfirmedBankTracker to send notifications for parent banks if they have not been notified yet. We use a new variable last_notified_slot to track that. Tests: With my validator running against testnet, before the fix, it was failing 75% of time, with the fix, it is passing consistently. Using the program mentioned in #18587. (cherry picked from commit 1a372a7)
…ribe (backport #19092) (#19207) * Fixing missing pubsub notification for programSubscribe and logsSubscribe (#19092) #18587: programSubscribe is missing notifications randomly. The issue is because of two reasons Not all rooted slots get OptimisticallyConfirmed notifications The OptimisticallyConfirmed notifications can be out of order for slots: slot A and B with A < B can see notification for B first before A. Summary of Changes Changed OptimisticallyConfirmedBankTracker to send notifications for parent banks if they have not been notified yet. We use a new variable last_notified_slot to track that. Tests: With my validator running against testnet, before the fix, it was failing 75% of time, with the fix, it is passing consistently. Using the program mentioned in #18587. (cherry picked from commit 1a372a7) * Use v1.7 api Co-authored-by: Lijun Wang <[email protected]> Co-authored-by: Tyera Eulberg <[email protected]>
|
Fix merged into master. |
…ribe (solana-labs#19092) Not all rooted slots get OptimisticallyConfirmed notifications The OptimisticallyConfirmed notifications can be out of order for slots: slot A and B with A < B can see notification for B first before A. Summary of Changes Changed OptimisticallyConfirmedBankTracker to send notifications for parent banks if they have not been notified yet. We use a new variable last_notified_slot to track that. Tests: With my validator running against testnet, before the fix, it was failing 75% of time, with the fix, it is passing consistently. Using the program mentioned in solana-labs#18587.
…ribe (solana-labs#19092) Not all rooted slots get OptimisticallyConfirmed notifications The OptimisticallyConfirmed notifications can be out of order for slots: slot A and B with A < B can see notification for B first before A. Summary of Changes Changed OptimisticallyConfirmedBankTracker to send notifications for parent banks if they have not been notified yet. We use a new variable last_notified_slot to track that. Tests: With my validator running against testnet, before the fix, it was failing 75% of time, with the fix, it is passing consistently. Using the program mentioned in solana-labs#18587.
…ribe (#19092) #18587: programSubscribe is missing notifications randomly. The issue is because of two reasons Not all rooted slots get OptimisticallyConfirmed notifications The OptimisticallyConfirmed notifications can be out of order for slots: slot A and B with A < B can see notification for B first before A. Summary of Changes Changed OptimisticallyConfirmedBankTracker to send notifications for parent banks if they have not been notified yet. We use a new variable last_notified_slot to track that. Tests: With my validator running against testnet, before the fix, it was failing 75% of time, with the fix, it is passing consistently. Using the program mentioned in #18587. (cherry picked from commit 1a372a7) # Conflicts: # core/src/optimistically_confirmed_bank_tracker.rs # core/src/rpc_subscriptions.rs
…ribe (backport #19092) (#19455) * Fixing missing pubsub notification for programSubscribe and logsSubscribe (#19092) #18587: programSubscribe is missing notifications randomly. The issue is because of two reasons Not all rooted slots get OptimisticallyConfirmed notifications The OptimisticallyConfirmed notifications can be out of order for slots: slot A and B with A < B can see notification for B first before A. Summary of Changes Changed OptimisticallyConfirmedBankTracker to send notifications for parent banks if they have not been notified yet. We use a new variable last_notified_slot to track that. Tests: With my validator running against testnet, before the fix, it was failing 75% of time, with the fix, it is passing consistently. Using the program mentioned in #18587. (cherry picked from commit 1a372a7) # Conflicts: # core/src/optimistically_confirmed_bank_tracker.rs # core/src/rpc_subscriptions.rs * Fix conflicts Co-authored-by: Lijun Wang <[email protected]> Co-authored-by: Tyera Eulberg <[email protected]>
|
This issue has been automatically locked since there has not been any activity in past 7 days after it was closed. Please open a new issue for related bugs. |
Problem
It appears that programSubscribe often drops notifications.
Code snippet to reproduce: https://github.com/Arrowana/sol-rpc-race-example
It listens to the stake program for Authorized staker being the Keypair pubKey,
in a loop it sends a transaction to create a new stake account being the Authorized staker and withdrawer on a new account.
Example on mainnet-beta, it was notified "received" for a few created stake accounts but then dropped a few other:
devnet: seems to notify of every single new program account without any problem
testnet: even more unreliable
Note: I tried removing the filter and it does not help, it still drops the expected transactions
Proposed Solution
TBD
The text was updated successfully, but these errors were encountered: