[BUG] org.opensearch.index.engine.InternalEngineTests.testLastRefreshCheckpoint is flaky

[reta]

Describe the bug

The org.opensearch.index.engine.InternalEngineTests.testLastRefreshCheckpoint is flaky:

com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=649, name=Thread-197, state=RUNNABLE, group=TGRP-InternalEngineTests]
	at __randomizedtesting.SeedInfo.seed([B448FCE4648CA4EB:7A3CB282E14976E1]:0)
Caused by: java.lang.AssertionError: 
Expected: a value equal to or greater than <0L>
     but: <-1L> was less than <0L>
	at __randomizedtesting.SeedInfo.seed([B448FCE4648CA4EB]:0)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
	at org.junit.Assert.assertThat(Assert.java:964)
	at org.junit.Assert.assertThat(Assert.java:930)
	at org.opensearch.index.engine.InternalEngineTests.lambda$testLastRefreshCheckpoint$84(InternalEngineTests.java:6643)
	at java.base/java.lang.Thread.run(Thread.java:1623)

See please: https://build.ci.opensearch.org/job/gradle-check/21580/testReport/junit/org.opensearch.index.engine/InternalEngineTests/testLastRefreshCheckpoint/

To Reproduce

./gradlew ':server:test' --tests "org.opensearch.index.engine.InternalEngineTests.testLastRefreshCheckpoint" -Dtests.seed=B448FCE4648CA4EB

Expected behavior
Test must always pass

Plugins
Standard

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• CI

Additional context
The flaky assertion was introduced in #7653

CC @ankitkala

[ankitkala]

com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=649, name=Thread-197, state=RUNNABLE, group=TGRP-InternalEngineTests]
	at __randomizedtesting.SeedInfo.seed([B448FCE4648CA4EB:7A3CB282E14976E1]:0)
Caused by: java.lang.AssertionError: 
Expected: a value equal to or greater than <0L>
     but: <-1L> was less than <0L>
	at __randomizedtesting.SeedInfo.seed([B448FCE4648CA4EB]:0)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
	at org.junit.Assert.assertThat(Assert.java:964)
	at org.junit.Assert.assertThat(Assert.java:930)
	at org.opensearch.index.engine.InternalEngineTests.lambda$testLastRefreshCheckpoint$84(InternalEngineTests.java:6643)
	at java.base/java.lang.Thread.run(Thread.java:1623)

[ankitkala]

The initialising value for currentOngoingRefreshCheckpoint might be initializing -1 in some cases. I'm trying to repro this locally but haven't got any failure yet even after 8K test iterations.

currentOngoingRefreshCheckpoint/LastRefreshedCheckpointListener.pendingCheckpoint gets initialised from localCheckpointTracker.getProcessedCheckpoint which should never be negative (code link).

[ankitkala]

I was not able to reproduce this even after 50K tests iterations.

pendingCheckpoint and refreshedCheckpoint are initialized with same value. pendingCheckpoint keeps track of the the processedCheckpoint for the ongoing refresh and updates the refreshedCheckpoint with the value after refresh. Only reason this error above (pendingCheckpoint=-1 && refreshedCheckpoint=0) can happen is if the pendingCheckpoint was set to -1 during refresh whereas refreshedCheckpoint was initialized with 0