[improve][broker] PIP-192 made split handler idempotent

[heesung-sn]

Master Issue: #16691

Motivation

Raising a PR to implement: #16691

Modifications

This PR

• Makes split handler idempotent .
• Makes Leader's orphan monitor keep trying to send split msg until finished.
• Select bundle boundaries at the SplitScheduler to have the same split boundaries for each Split handler retry.
• Adds a split condition to check if the parent's Splitting state has moved.
• Made Admin Unload command forceful to unload any bundles in invalid states.

Verifying this change

• Make sure that the change passes the CI checks.

This change added tests and can be verified as follows:

• *Updated unit tests.

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• Anything that affects deployment

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

We will have separate PRs to update the Doc later.

Matching PR in forked repository

PR in forked repository: heesung-sn#41

[Demogorgon314]

If namespace bundles do not contain the parent bundle, we shouldn't continue to verify the child bundles, right?

[heesung-sn]

If namespace bundles do not contain the parent bundle, this means the namespace bundles must have child bundles(the namespace policy must have been updated). Here, we redundantly verify if the child bundles are there. If there is no parent bundle, and if there are no child bundles, the current logic will retry forever. I can't think of a case that the current logics lead to this state, but in the worst case, the error count of the split metrics should constantly increase, expecting the admins to unload this parent bundle.

[heesung-sn]

@Demogorgon314 @gaoran10 PLAL by any chance.

[heesung-sn]

@Demogorgon314 @Technoboy- PTAL by any chance.

[heesung-sn]

@Demogorgon314 fyi, I updated the code

• to continue splitting after the state override by the monitor.
• to fix the dst and src broker order in the splitting state data copy.

[heesung-sn]

/pulsarbot rerun-failure-checks

[heesung-sn]

@Demogorgon314 @Technoboy- plz merge if looking good