Parallel loop failing with java.lang.NoSuchMethodError: java.nio.ByteBuffer.limit(I)Ljava/nio/ByteBuffer; #1613

FredericoCoelhoNunes · 2021-07-14T08:59:58Z

Hi, trying to run a parallel loop using Digdag latest version, and running into a cryptical Java error.
Here's the error trace:

io.digdag.core.agent.OperatorManager: Task failed with unexpected error: java.nio.ByteBuffer.limit(I)Ljava/nio/ByteBuffer;
java.lang.NoSuchMethodError: java.nio.ByteBuffer.limit(I)Ljava/nio/ByteBuffer;
	at io.digdag.core.database.AESGCMSecretCrypto.decryptSecret(AESGCMSecretCrypto.java:173)
	at io.digdag.core.database.DatabaseSecretStore.getSecret(DatabaseSecretStore.java:49)
	at io.digdag.core.agent.GrantedPrivilegedVariables.lambda$privilegedSecretProvider$0(GrantedPrivilegedVariables.java:25)
	at io.digdag.spi.SecretProvider.getSecret(SecretProvider.java:20)
	at io.digdag.util.UserSecretTemplate.lambda$format$0(UserSecretTemplate.java:55)
	at io.digdag.util.UserSecretTemplate.replaceAll(UserSecretTemplate.java:65)
	at io.digdag.util.UserSecretTemplate.format(UserSecretTemplate.java:55)
	at io.digdag.core.agent.GrantedPrivilegedVariables.lambda$buildAccessor$1(GrantedPrivilegedVariables.java:62)
	at io.digdag.core.agent.GrantedPrivilegedVariables.get(GrantedPrivilegedVariables.java:81)
	at io.digdag.util.CommandOperators.collectEnvironmentVariables(CommandOperators.java:18)
	at io.digdag.standards.operator.PyOperatorFactory$PyOperator.runCommand(PyOperatorFactory.java:296)
	at io.digdag.standards.operator.PyOperatorFactory$PyOperator.runCode(PyOperatorFactory.java:159)
	at io.digdag.standards.operator.PyOperatorFactory$PyOperator.runTask(PyOperatorFactory.java:115)
	at io.digdag.util.BaseOperator.run(BaseOperator.java:35)
	at io.digdag.core.agent.OperatorManager.callExecutor(OperatorManager.java:365)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invokeMain(DigdagTimedMethodInterceptor.java:58)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invoke(DigdagTimedMethodInterceptor.java:31)
	at io.digdag.core.agent.OperatorManager.runWithWorkspace(OperatorManager.java:298)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invokeMain(DigdagTimedMethodInterceptor.java:58)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invoke(DigdagTimedMethodInterceptor.java:31)
	at io.digdag.core.agent.OperatorManager.lambda$runWithHeartbeat$2(OperatorManager.java:151)
	at io.digdag.core.agent.ExtractArchiveWorkspaceManager.withExtractedArchive(ExtractArchiveWorkspaceManager.java:77)
	at io.digdag.core.agent.OperatorManager.runWithHeartbeat(OperatorManager.java:149)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invokeMain(DigdagTimedMethodInterceptor.java:58)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invoke(DigdagTimedMethodInterceptor.java:31)
	at io.digdag.core.agent.OperatorManager.run(OperatorManager.java:132)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invokeMain(DigdagTimedMethodInterceptor.java:58)
	at io.digdag.server.metrics.DigdagTimedMethodInterceptor.invoke(DigdagTimedMethodInterceptor.java:31)
	at io.digdag.core.agent.MultiThreadAgent.lambda$run$0(MultiThreadAgent.java:132)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Any help would be very appreciated.
Thank you!

The text was updated successfully, but these errors were encountered:

hiroyuki-sato · 2021-07-14T09:58:36Z

Hello, @FredericoCoelhoNunes

Could you provide us about your environment?

Digdag version: latest means 10.0.1?
Reproducable dig file
Java version
OS Version

FredericoCoelhoNunes · 2021-07-16T11:11:14Z

Hi @hiroyuki-sato , sorry for the delay.

Here is the info:

Digdag version: obtained from https://dl.digdag.io/digdag-latest, the output of diagdag --version is 0.10.1
Java version: output of java -version is:

openjdk version "1.8.0_292"
OpenJDK Runtime Environment (build 1.8.0_292-8u292-b10-0+deb9u1-b10)
OpenJDK 64-Bit Server VM (build 25.292-b10, mixed mode)

OS version: Output of lsb_release -da:

No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 9.13 (stretch)
Release:        9.13
Codename:       stretch

As for a reproducible example:

dig file:

+my_test:
  _parallel: true
  for_each>:
    schema: [
      {
        file:     "MY-FILE"
      }
    ]
  _do:
    py>: test_module.test_print
    _env:
      MY_SECRET:         ${secret:MY_SECRET}

test_module.py file:

import digdag

def test_print():
    print(f'{schema["file"]}')

Setting up:

digdag push --project . test_project -e localhost:9090
digdag secrets --project test_project --set MY_SECRET=12345 -e localhost:9090

A few final important notes:

the problem seems to be in the secrets
the problem was solved by downgrading to Digdag version 0.9.42

Thanks!!

hiroyuki-sato · 2021-07-19T01:16:49Z

Hello, @FredericoCoelhoNunes

Did you setup secret-encryption-key ?

FredericoCoelhoNunes · 2021-07-19T08:26:31Z

Hi @hiroyuki-sato , forgot to specify that in my minimum reproducible example, but yes I did.
Technically I also have a Postgres database running with docker-compose and I set up Digdag in the following way:

echo "
server.bind=0.0.0.0
server.port=9090
server.admin.bind=0.0.0.0
server.admin.port=9091
database.maximumPoolSize=10
database.type=postgresql
database.user=${DIGDAG_DB_USER}
database.password=${DIGDAG_DB_PASSWORD}
database.host=${DIGDAG_DB_HOST}
database.port=${DIGDAG_DB_PORT}
database.database=${DIGDAG_DB_NAME}
digdag.secret-encryption-key=<MY KEY>
" > digdag_server.properties

digdag server \
  --task-log ./digdag/sessions/ \
  --max-task-threads 5 \
  --config ./digdag_server.properties

hiroyuki-sato · 2021-07-20T10:27:50Z

Hello, @FredericoCoelhoNunes
Thank you for your report. I could reproduce it.
https://github.com/hiroyuki-sato/digdag-support/tree/master/20210720_core_1613

I'll investigate it. Please use 0.9.x until fix this issue.

FredericoCoelhoNunes · 2021-07-20T10:49:13Z

No worries @hiroyuki-sato , thanks for verifying. Keep up the great work!

komamitsu · 2021-07-21T08:15:03Z

@fernandomrtnz Thanks for reporting this issue! And @hiroyuki-sato Thanks for your support as always 😉

I also reproduced this issue with 0.10.1 while didn't reproduce with 0.10.0. Let me look into it.

komamitsu · 2021-07-21T08:19:57Z

This exception doesn't happen with 0.10.1 built from source code. Looks like a packaging issue?

komamitsu · 2021-07-21T09:07:10Z

Hmm. I checked the diff (especially around io/digdag/core/database/AESGCMSecretCrypto.class) between https://github.com/treasure-data/digdag/releases/download/v0.10.1/digdag-0.10.1.jar and https://github.com/treasure-data/digdag/releases/download/v0.10.0/digdag-0.10.0.jar, but didn't find suspicious diff...

@szyn Do you still have digdag-0.10.1.jar which you built and used for the release on you laptop or somewhere? If so, could you try to reproduce this issue with it when you have time?

yoyama · 2021-07-21T09:45:39Z

I guess 0.10.1 may be built with Java11.

yoyama · 2021-07-21T09:51:13Z

May be helpful information.
jetty/jetty.project#3244

komamitsu · 2021-07-21T12:53:04Z

Hmm. I compiled v0.10.1 source code using Java 11 and tested it, but I didn't reproduce this issue. This issue happened only with the released jar on my side.

~~We've specified targetCompatibility = 1.8 https://github.com/treasure-data/digdag/blob/v0.10.1/build.gradle#L56, so generated class files should be compatible with Java 8.~~ (edit: I was confusing javac --target and javac --release...)

I also checked io/digdag/core/database/AESGCMSecretCrypto.class as mentioned earlier and the target major version is Java 8.

Classfile xxxxx/0.10.1/io/digdag/core/database/AESGCMSecretCrypto.class
  Last modified Jun 4, 2021; size 6116 bytes
  MD5 checksum ac3b4f562b60b087bdfdb5c5c953ec5b
  Compiled from "AESGCMSecretCrypto.java"
public class io.digdag.core.database.AESGCMSecretCrypto implements io.digdag.core.crypto.SecretCrypto
  minor version: 0
  major version: 52

komamitsu · 2021-07-21T13:17:27Z

Hmm. I compiled v0.10.1 source code using Java 11 and tested it, but I didn't reproduce this issue. This issue happened only with the released jar on my side.

Oops. My test was wrong. I thought I had run the executable using Java 8, but I actually ran it with Java 11 that was the same version with the build. I reproduced this issue when using Java 8. Sorry for the confusion.

komamitsu · 2021-07-21T13:38:16Z

Maybe we'd better explicitly add --release 8 option to javac like this

diff --git a/build.gradle b/build.gradle
index 240042c7a8..6167dc0c04 100644
--- a/build.gradle
+++ b/build.gradle
@@ -55,7 +55,10 @@ subprojects {
     sourceCompatibility = 1.8
     targetCompatibility = 1.8

-    compileJava.options.encoding = 'UTF-8'
+    compileJava {
+        options.encoding = 'UTF-8'
+        options.compilerArgs.addAll(['--release', '8'])
+    }
     compileTestJava.options.encoding = 'UTF-8'

     dependencies {

With this option, I confirmed this issue isn't reproduced even with the combination of Java 11 for build and Java 8 for execution.

But, of course, the easiest fix is to build with Java 8 and release as 0.10.2 😉

FYI: latest Gradle has release option https://docs.gradle.org/current/dsl/org.gradle.api.tasks.compile.CompileOptions.html#org.gradle.api.tasks.compile.CompileOptions:release

szyn · 2021-07-21T14:03:50Z

Thank you for reporting this issue, I also could reproduce this by using Java8.
As @yoyama pointed out, this is the reason why I build digdag with Java 11.

szyn · 2021-07-21T14:08:33Z

Maybe we'd better explicitly add --release 8 option to javac like this

Thanks, that's good to know, and I agree with you.

But, of course, the easiest fix is to build with Java 8 and release as 0.10.2 😉

I think so too. Let me build with Java 8 and release it as 0.10.2 later.

hiroyuki-sato · 2021-07-21T16:05:04Z

@komamitsu @yoyama @szyn Thank you for your reply.

JFYI. @dmikurube 's comment on Twitter
Rough translate in English: It is necessary to build with Java 8 if Digdag supports Java 8.

In my opinion, dropping Java 8 supports is another idea if many contributors use Java 11 mainly.

hiroyuki-sato · 2021-08-12T11:02:03Z

Hello, @FredericoCoelhoNunes

Digdag 0.10.2 has been released.
Could you try it?

You can update using digdag selfupdate 0.10.2.

FredericoCoelhoNunes · 2021-08-12T11:12:46Z

Hi @hiroyuki-sato , unfortunately I won't be able to test it in the near future as I have moved to a different project. Thanks!

hiroyuki-sato · 2021-08-12T11:23:29Z

Hi, @FredericoCoelhoNunes Thank you for your comment.
@szyn I confirmed this issue has been fixed in Digdag 0.10.2. Thanks!

szyn · 2021-08-17T10:03:47Z

@hiroyuki-sato I'd appreciate your confirmation!
Let me close this issue.

szyn added this to the v0.10.2 milestone Jul 22, 2021

szyn closed this as completed Aug 17, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Parallel loop failing with java.lang.NoSuchMethodError: java.nio.ByteBuffer.limit(I)Ljava/nio/ByteBuffer; #1613

Parallel loop failing with java.lang.NoSuchMethodError: java.nio.ByteBuffer.limit(I)Ljava/nio/ByteBuffer; #1613

FredericoCoelhoNunes commented Jul 14, 2021

hiroyuki-sato commented Jul 14, 2021

FredericoCoelhoNunes commented Jul 16, 2021 •

edited

Loading

hiroyuki-sato commented Jul 19, 2021

FredericoCoelhoNunes commented Jul 19, 2021

hiroyuki-sato commented Jul 20, 2021

FredericoCoelhoNunes commented Jul 20, 2021

komamitsu commented Jul 21, 2021

komamitsu commented Jul 21, 2021

komamitsu commented Jul 21, 2021

yoyama commented Jul 21, 2021

yoyama commented Jul 21, 2021

komamitsu commented Jul 21, 2021 •

edited

Loading

komamitsu commented Jul 21, 2021

komamitsu commented Jul 21, 2021 •

edited

Loading

szyn commented Jul 21, 2021

szyn commented Jul 21, 2021 •

edited

Loading

hiroyuki-sato commented Jul 21, 2021

hiroyuki-sato commented Aug 12, 2021

FredericoCoelhoNunes commented Aug 12, 2021

hiroyuki-sato commented Aug 12, 2021

szyn commented Aug 17, 2021

Parallel loop failing with java.lang.NoSuchMethodError: java.nio.ByteBuffer.limit(I)Ljava/nio/ByteBuffer; #1613

Parallel loop failing with java.lang.NoSuchMethodError: java.nio.ByteBuffer.limit(I)Ljava/nio/ByteBuffer; #1613

Comments

FredericoCoelhoNunes commented Jul 14, 2021

hiroyuki-sato commented Jul 14, 2021

FredericoCoelhoNunes commented Jul 16, 2021 • edited Loading

hiroyuki-sato commented Jul 19, 2021

FredericoCoelhoNunes commented Jul 19, 2021

hiroyuki-sato commented Jul 20, 2021

FredericoCoelhoNunes commented Jul 20, 2021

komamitsu commented Jul 21, 2021

komamitsu commented Jul 21, 2021

komamitsu commented Jul 21, 2021

yoyama commented Jul 21, 2021

yoyama commented Jul 21, 2021

komamitsu commented Jul 21, 2021 • edited Loading

komamitsu commented Jul 21, 2021

komamitsu commented Jul 21, 2021 • edited Loading

szyn commented Jul 21, 2021

szyn commented Jul 21, 2021 • edited Loading

hiroyuki-sato commented Jul 21, 2021

hiroyuki-sato commented Aug 12, 2021

FredericoCoelhoNunes commented Aug 12, 2021

hiroyuki-sato commented Aug 12, 2021

szyn commented Aug 17, 2021

FredericoCoelhoNunes commented Jul 16, 2021 •

edited

Loading

komamitsu commented Jul 21, 2021 •

edited

Loading

komamitsu commented Jul 21, 2021 •

edited

Loading

szyn commented Jul 21, 2021 •

edited

Loading