Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] org.opensearch.test.test.InternalTestClusterTests.testDifferentRolesMaintainPathOnRestart is flaky #11026

Open
reta opened this issue Oct 31, 2023 · 1 comment
Open

[BUG] org.opensearch.test.test.InternalTestClusterTests.testDifferentRolesMaintainPathOnRestart is flaky #11026

reta opened this issue Oct 31, 2023 · 1 comment
Assignees
@gauravruhela
Labels
bug Something isn't working flaky-test Random test failure that succeeds on second run Other

Comments

@reta
Copy link
Collaborator

reta commented Oct 31, 2023

Describe the bug
The test case org.opensearch.test.test.InternalTestClusterTests.testDifferentRolesMaintainPathOnRestart is flaky:

org.opensearch.test.test.InternalTestClusterTests.testDifferentRolesMaintainPathOnRestart


org.apache.lucene.store.AlreadyClosedException: FileLock invalidated by an external force: NativeFSLock(path=/var/jenkins/workspace/gradle-check/search/test/framework/build/testrun/test/temp/org.opensearch.test.test.InternalTestClusterTests_FB1571D6730DA36F-001/tempDir-005/0/nodes/0/_state/write.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive invalid],creationTime=2023-10-31T12:59:49.118955892Z)
	at __randomizedtesting.SeedInfo.seed([FB1571D6730DA36F:D05A41B9E036509A]:0)
	at app//org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:165)
	at app//org.apache.lucene.store.LockValidatingDirectoryWrapper.deleteFile(LockValidatingDirectoryWrapper.java:36)
	at app//org.apache.lucene.util.FileDeleter.delete(FileDeleter.java:234)
	at app//org.apache.lucene.util.FileDeleter.delete(FileDeleter.java:227)
	at app//org.apache.lucene.util.FileDeleter.decRef(FileDeleter.java:109)
	at app//org.apache.lucene.index.IndexFileDeleter.decRef(IndexFileDeleter.java:631)
	at app//org.apache.lucene.index.IndexFileDeleter.checkpoint(IndexFileDeleter.java:589)
	at app//org.apache.lucene.index.IndexWriter.rollbackInternalNoCommit(IndexWriter.java:2519)
	at app//org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2449)
	at app//org.apache.lucene.index.IndexWriter.rollback(IndexWriter.java:2441)
	at app//org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1364)
	at app//org.opensearch.common.util.io.IOUtils.close(IOUtils.java:89)
	at app//org.opensearch.common.util.io.IOUtils.close(IOUtils.java:131)
	at app//org.opensearch.common.util.io.IOUtils.close(IOUtils.java:81)
	at app//org.opensearch.gateway.PersistedClusterStateService$MetadataIndexWriter.close(PersistedClusterStateService.java:588)
	at app//org.opensearch.common.util.io.IOUtils.close(IOUtils.java:89)
	at app//org.opensearch.common.util.io.IOUtils.close(IOUtils.java:131)
	at app//org.opensearch.common.util.io.IOUtils.close(IOUtils.java:114)
	at app//org.opensearch.gateway.PersistedClusterStateService$Writer.close(PersistedClusterStateService.java:890)
	at app//org.opensearch.common.util.io.IOUtils.close(IOUtils.java:89)
	at app//org.opensearch.gateway.GatewayMetaState$LucenePersistedState.close(GatewayMetaState.java:656)
	at app//org.opensearch.gateway.GatewayMetaState$AsyncLucenePersistedState.close(GatewayMetaState.java:515)
	at app//org.opensearch.common.util.io.IOUtils.close(IOUtils.java:89)
	at app//org.opensearch.common.util.io.IOUtils.close(IOUtils.java:131)
	at app//org.opensearch.common.util.io.IOUtils.close(IOUtils.java:114)
	at app//org.opensearch.cluster.coordination.PersistedStateRegistry.close(PersistedStateRegistry.java:49)
	at app//org.opensearch.common.util.io.IOUtils.close(IOUtils.java:89)
	at app//org.opensearch.cluster.coordination.CoordinationState.close(CoordinationState.java:607)
	at app//org.opensearch.cluster.coordination.Coordinator.doClose(Coordinator.java:903)
	at app//org.opensearch.common.lifecycle.AbstractLifecycleComponent.close(AbstractLifecycleComponent.java:120)
	at app//org.opensearch.common.util.io.IOUtils.close(IOUtils.java:89)
	at app//org.opensearch.common.util.io.IOUtils.close(IOUtils.java:131)
	at app//org.opensearch.common.util.io.IOUtils.close(IOUtils.java:114)
	at app//org.opensearch.node.Node.close(Node.java:1631)
	at app//org.opensearch.test.InternalTestCluster$NodeAndClient.close(InternalTestCluster.java:1128)
	at app//org.opensearch.test.InternalTestCluster$NodeAndClient.closeForRestart(InternalTestCluster.java:1070)
	at app//org.opensearch.test.InternalTestCluster.fullRestart(InternalTestCluster.java:2120)
	at app//org.opensearch.test.InternalTestCluster.fullRestart(InternalTestCluster.java:2008)
	at app//org.opensearch.test.test.InternalTestClusterTests.testDifferentRolesMaintainPathOnRestart(InternalTestClusterTests.java:436)
	at [email protected]/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at [email protected]/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at [email protected]/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at [email protected]/java.lang.reflect.Method.invoke(Method.java:568)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at app//org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at app//org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at [email protected]/java.lang.Thread.run(Thread.java:833)

To Reproduce
Steps to reproduce the behavior:

./gradlew ':test:framework:test' --tests "org.opensearch.test.test.InternalTestClusterTests.testDifferentRolesMaintainPathOnRestart" -Dtests.seed=FB1571D6730DA36F

Expected behavior
The test must always pass

Plugins
Standard

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

  • CI

Additional context
@reta reta added bug Something isn't working untriaged flaky-test Random test failure that succeeds on second run and removed untriaged labels Oct 31, 2023
@reta reta mentioned this issue Oct 31, 2023
Merged
@andrross andrross added the Other label Feb 21, 2024
@shiv0408
Copy link
Member

shiv0408 commented Mar 7, 2024

I ran this test with mentioned seed and this test passed, also I ran the test 1000 times to try reproducing it and test passed in all the runs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@gauravruhela gauravruhela

Labels
bug Something isn't working flaky-test Random test failure that succeeds on second run Other
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@reta @peternied @andrross @gauravruhela @shiv0408