[BUG] repository-s3 plugin is broken in mainline

[rishabh6788]

Describe the bug

Getting below error while trying to register an s3 repository using latest build for 3.0.0-alpha1

[ec2-user@ip-172-31-23-58 default]$ curl -X PUT "http://opense-clust-***********-692e187bb21cd8d5.elb.us-east-1.amazonaws.com/_snapshot/benchmark-workloads-repo-3x" -H "Content-Type: application/json" -d '
> {
>   "type": "s3",
>   "settings": {
>     "bucket": "benchmark-***********",
>     "base_path": "workload-*******",
>     "region": "*****"
>   }
> }'

{"error":{"root_cause":[{"type":"repository_exception","reason":"[benchmark-workloads-repo-3x] failed to create repository"}],"type":"repository_exception","reason":"[benchmark-workloads-repo-3x] failed to create repository","caused_by":{"type":"null_pointer_exception","reason":"Cannot invoke \"org.opensearch.repositories.s3.async.AsyncExecutorContainer.getStreamReader()\" because \"this.normalExecutorBuilder\" is null"}},"status":500}[

Related component

Plugins

To Reproduce

1. Download the latest tar for 3.0.0-alpha1 version.
2. Install respository-s3 plugin and make sure the required credentials and permissions are configured.
3. register a snapshot repository
4. See error

Expected behavior

The cluster should return 200 response.

Additional Details

Use tarball https://artifacts.opensearch.org/snapshots/core/opensearch/3.0.0-alpha1-SNAPSHOT/opensearch-min-3.0.0-alpha1-SNAPSHOT-linux-x64-latest.tar.gz

[prudhvigodithi]

Hey @rishabh6788 in past I have implemented snapshot capabilities for the OpenSearch K8s Operator opensearch-project/opensearch-k8s-operator#278, this issue has some details related to IAM role and commands, so AFAIK the snapshot repo registration and creating of snapshot is for sure working in past.

[reta]

@rishabhmaurya I suspect it is caused by #16962, could you please double check? thank you wrong pointers, sorry about that

[rishabh6788]

@reta Above PR was merged yesterday, I am experiencing the same error on a tarball created on Feb-10th as well.

[rishabh6788]

Hey @rishabh6788 in past I have implemented snapshot capabilities for the OpenSearch K8s Operator opensearch-project/opensearch-k8s-operator#278, this issue has some details related to IAM role and commands, so AFAIK the snapshot repo registration and creating of snapshot is for sure working in past.

it was working fine up until 10-days back. Last successful run was with commit-id b823d1f794c27f92c257128b62b955ae10cf7f6c. The next run which started failing had commit -d d0a65d3b457e11d514171ecc0a77450fdd7d25bb.
Something broke in between these two commits.

[rishabh6788]

For more context, I encountered this in our nightly benchmark workflow, that runs on the latest build of mainline daily.
The cluster ec2 has instance role attached that has appropriate policy required for respository-s3 plugin to work.

[rishabh6788]

@peterzhuamazon The repository-s3 plugin on all alpha builds is broken.

[peterzhuamazon]

Hi @reta @andrross @expani any lead on fixing this issue?

cc: @getsaurabh02

Thanks!

[andrross]

@peterzhuamazon Are we properly pulling the alpha1 version of the repository plugin when doing opensearch-plugin install respository-s3? I'm not able to reproduce this issue when I install the plugin from a zip file directly build from source, but I can reproduce when just installing by name where it downloads the zip.

[rishabh6788]

I think I got the issue, the code is pulling plugin from old 3.0.0 version https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/3.0.0/latest/linux/x64/tar/builds/opensearch/core-plugins/repository-s3-3.0.0.zip.
This should now be pointing to alpha.

I wil verify and then close the issue.

[rishabh6788]

@peterzhuamazon If I do bin/opensearch-plugin install repository-s3 on an alpha build, where does the plugin pulled from?

[peterzhuamazon]

@peterzhuamazon If I do bin/opensearch-plugin install repository-s3 on an alpha build, where does the plugin pulled from?

https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/3.0.0-alpha1/latest/linux/x64/tar/builds/opensearch/core-plugins/repository-s3-3.0.0-alpha1.zip

[andrross]

@peterzhuamazon If I do bin/opensearch-plugin install repository-s3 on an alpha build, where does the plugin pulled from?

https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/3.0.0-alpha1/latest/linux/x64/tar/builds/opensearch/core-plugins/repository-s3-3.0.0-alpha1.zip

@peterzhuamazon This doesn't seem to be correct. The zip downloaded by bin/opensearch-plugin install repository-s3 has netty 4.1.115 whereas the latest alpha snapshot at the link above has netty 4.1.118.

[peterzhuamazon]

@peterzhuamazon If I do bin/opensearch-plugin install repository-s3 on an alpha build, where does the plugin pulled from?

https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/3.0.0-alpha1/latest/linux/x64/tar/builds/opensearch/core-plugins/repository-s3-3.0.0-alpha1.zip

@peterzhuamazon This doesn't seem to be correct. The zip downloaded by bin/opensearch-plugin install repository-s3 has netty 4.1.115 whereas the latest alpha snapshot at the link above has netty 4.1.118.

What is the link for bin/opensearch-plugin install repository-s3 by default in 3.0.0-alpha1 (if not giving the full url)? I thought this should only run if we have already released official version and it will pull from official public bucket.

The link in previous comments is for staging only.

Unless you are pulling the non-qualifier version from staging bucket, which is extremely old build for now.

[andrross]

What is the link for bin/opensearch-plugin install repository-s3 by default in 3.0.0-alpha1 (if not giving the full url)?

@peterzhuamazon I don't know, I was asking you this question :)

Edit: this is the code that constructs the URL to download:

OpenSearch/distribution/tools/plugin-cli/src/main/java/org/opensearch/tools/cli/plugin/InstallPluginCommand.java

Lines 350 to 372 in 679a08f

	if (isSnapshot == true) {
	baseUrl = String.format(
	Locale.ROOT,
	"https://artifacts.opensearch.org/snapshots/plugins/%s/%s",
	pluginId,
	Build.CURRENT.getQualifiedVersion()
	);
	} else {
	baseUrl = String.format(
	Locale.ROOT,
	"https://artifacts.opensearch.org/releases/plugins/%s/%s",
	pluginId,
	Build.CURRENT.getQualifiedVersion()
	);
	}
	final String platformUrl = String.format(
	Locale.ROOT,
	"%s/%s-%s-%s.zip",
	baseUrl,
	pluginId,
	platform,
	Build.CURRENT.getQualifiedVersion()
	);

It does not currently use the alpha qualifier and is downloading this version of the plugin: https://artifacts.opensearch.org/snapshots/plugins/repository-s3/3.0.0-SNAPSHOT/repository-s3-3.0.0-SNAPSHOT.zip

[peterzhuamazon]

It does not currently use the alpha qualifier and is downloading this version of the plugin: https://artifacts.opensearch.org/snapshots/plugins/repository-s3/3.0.0-SNAPSHOT/repository-s3-3.0.0-SNAPSHOT.zip

Yeah that explains it, seems like snapshots is still pulling from the non-qualifier version 😄

[peterzhuamazon]

Have an offline talk with @andrross and seems like the getQualifiedVersion() simply returns version instead of version+qualifier.

[andrross]

@rishabh6788 If you need to unblock yourself, you can pass the URL directly, i.e.:

./bin/opensearch-plugin install https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/3.0.0-alpha1/latest/linux/x64/tar/builds/opensearch/core-plugins/repository-s3-3.0.0-alpha1.zip

[peterzhuamazon]

Hi @andrross any update on fixing the link?

Thanks.

[andrross]

@peterzhuamazon I can reproduce this issue with a locally-built min distribution. However, I believe the install plugin code is actually correct. The issue is that the core repository itself doesn't know anything about the "alpha1" version, it's only supplied as a parameter to the maven publish command.

However, if I download the latest published snapshot of the min distribution (https://artifacts.opensearch.org/snapshots/core/opensearch/3.0.0-alpha1-SNAPSHOT/opensearch-min-3.0.0-alpha1-SNAPSHOT-linux-arm64-latest.tar.gz), then extract the manifest from one of the jar files (e.g. jar -xvf opensearch-core-3.0.0-alpha1-SNAPSHOT.jar META-INF/MANIFEST.MF), then it contains this:

% cat META-INF/MANIFEST.MF
Manifest-Version: 1.0
Module-Origin: https://github.com/opensearch-project/OpenSearch.git
Change: 176a4f01dea48a9a5f149ba50c3d51fe6f11e9d4
X-Compile-OpenSearch-Version: 3.0.0-alpha1-SNAPSHOT
X-Compile-Lucene-Version: 10.1.0
X-Compile-OpenSearch-Snapshot: true
Implementation-Title: org.opensearch#opensearch-core;3.0.0-alpha1-SNAPSH
 OT
Implementation-Version: 3.0.0-alpha1-SNAPSHOT
Built-Status: integration
Built-By: ci-runner
Built-OS: Linux
Build-Timezone: UTC
Build-Date-UTC: 2025-02-26T18:24:16.369085763Z
Build-Date: 2025-02-26T18:24:01.434323242Z
Gradle-Version: 8.12.1
Created-By: 23+37 (Eclipse Adoptium)
X-Compile-Target-JDK: 21
X-Compile-Source-JDK: 21
Build-Java-Version: 23

The X-Compile-OpenSearch-Version is what the code is looking for here, so for that min distribution it will then install the correct plugin.

So the upshot is that I can't actually reproduce this issue with the linked tarball (https://artifacts.opensearch.org/snapshots/core/opensearch/3.0.0-alpha1-SNAPSHOT/opensearch-min-3.0.0-alpha1-SNAPSHOT-linux-x64-latest.tar.gz), only with a min distribution built directly from source.

[peterzhuamazon]

Hi @andrross we build the artifact specifically with -Dbuild.version_qualifier=alpha1, could you try that from your local as well?

Thanks.

[andrross]

Thanks @peterzhuamazon! That actually fixes it for me. Here's a test of the "alpha1" build locally, with verbose output from the install plugin command:

./build/distribution/local/opensearch-3.0.0-alpha1-SNAPSHOT/bin/opensearch-plugin install repository-s3 --verbose
-> Installing repository-s3
Checking if url exists: https://artifacts.opensearch.org/snapshots/plugins/repository-s3/3.0.0-alpha1-SNAPSHOT/repository-s3-darwin-aarch64-3.0.0-alpha1-SNAPSHOT.zip
-> Downloading repository-s3 from opensearch
Retrieving zip from https://artifacts.opensearch.org/snapshots/plugins/repository-s3/3.0.0-alpha1-SNAPSHOT/repository-s3-3.0.0-alpha1-SNAPSHOT.zip
[=================================================] 100%
- Plugin information:
Name: repository-s3
Description: The S3 repository plugin adds S3 repositories
Version: 3.0.0-alpha1-SNAPSHOT
OpenSearch Version: 3.0.0
Java Version: 21
Native Controller: false
Extended Plugins: []
 * Classname: org.opensearch.repositories.s3.S3RepositoryPlugin

I believe it is downloading the correct artifact and seems to work properly in my testing.

[andrross]

I'm going to close this as I don't think there's a problem here. My initial confusion was from not providing -Dbuild.version_qualifier=alpha1 on my local builds. @rishabh6788 please reopen if you think there is still a bug.