[BUG] Test org.opensearch.action.admin.cluster.stats.ClusterStatsIT.testNodeRolesWithDataNodeLegacySetting is flaky

[kotwanikunal]

Describe the bug
Test org.opensearch.action.admin.cluster.stats.ClusterStatsIT.testNodeRolesWithDataNodeLegacySetting is flaky

To Reproduce

./gradlew ':server:internalClusterTest' --tests "org.opensearch.action.admin.cluster.stats.ClusterStatsIT.testNodeRolesWithDataNodeLegacySettings" -Dtests.seed=5568C86430BA738A -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ar-LY -Dtests.timezone=Atlantic/Bermuda

Screenshots

rg.opensearch.action.admin.cluster.stats.ClusterStatsIT > testNodeRolesWithDataNodeLegacySettings FAILED
    java.lang.IllegalArgumentException: duplicate element: [cluster_manager]
        at __randomizedtesting.SeedInfo.seed([5568C86430BA738A:7C15068B9D7AB34C]:0)
        at java.base/java.util.ImmutableCollections$Set12.<init>(ImmutableCollections.java:791)
        at java.base/java.util.Set.of(Set.java:487)
        at org.opensearch.action.admin.cluster.stats.ClusterStatsIT.testNodeRolesWithDataNodeLegacySettings(ClusterStatsIT.java:415)

https://build.ci.opensearch.org/job/gradle-check/14643/console

Additional context
#7323

[dblock]

Another one in #7333, https://build.ci.opensearch.org/job/gradle-check/14650/. Got a consistent repro with ./gradlew ':server:internalClusterTest' --tests "org.opensearch.action.admin.cluster.stats.ClusterStatsIT.testNodeRolesWithDataNodeLegacySettings" -Dtests.seed=464D695B7CAC6C07 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ca -Dtests.timezone=Africa/Bamako -Druntime.java=19

[kotwanikunal]

I think the flakiness was introduced as a part of #6331

[reta]

Still happening (on 2.x at least): https://build.ci.opensearch.org/job/gradle-check/15182/

Error Message

java.lang.IllegalArgumentException: duplicate element: [data, remote_cluster_client]

Stacktrace

java.lang.IllegalArgumentException: duplicate element: [data, remote_cluster_client]
	at __randomizedtesting.SeedInfo.seed([58F1ABE3FCC33CFA:718C650C5103FC3C]:0)
	at java.base/java.util.ImmutableCollections$Set12.<init>(ImmutableCollections.java:791)
	at java.base/java.util.Set.of(Set.java:487)
	at org.opensearch.action.admin.cluster.stats.ClusterStatsIT.testNodeRolesWithDataNodeLegacySettings(ClusterStatsIT.java:415)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)

[kotwanikunal]

The fix wasn't backported @reta. Doing it now.