centos-infra

#1607 Unable to successfully launch agent pods with Jenkins v2.479.3
Closed: Fixed with Explanation by arrfab. Opened by anoopcs.

Closed: Fixed with Explanation
Reopen Issue

Logs indicate some mismatch in the JDK runtime version:

2025/03/05 08:10:44 [go-init] No pre-start command defined, skip
2025/03/05 08:10:44 [go-init] Main command launched : /usr/local/bin/run-jnlp-client
OPENSHIFT_JENKINS_JVM_ARCH='', CONTAINER_MEMORY_IN_MB='8796093022207', using /usr/lib/jvm/jre-11-openjdk/bin/java
Downloading http://172.30.104.61:80/jnlpJars/remoting.jar ...
+ cd
+ exec java -Duser.home=/home/jenkins -Dcom.redhat.fips=false -XX:+UseParallelGC -XX:MinHeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -cp /home/jenkins/remoting.jar hudson.remoting.jnlp.Main -headless -url http://172.30.104.61:80/ -tunnel 172.30.121.229:50000 cfca78ef1a26b37f295205608e696ea8f6d9e405101ae9d3a872258d79e1b400 cico-workspace-t93jg
WARNING: Using deprecated entrypoint "java -cp agent.jar hudson.remoting.jnlp.Main". Use "java -jar agent.jar" instead.
WARNING: The "-headless" argument is deprecated and will be removed without replacement in a future release.
WARNING: Providing the secret and agent name as positional arguments is deprecated; use "-secret" and "-name" instead.
Mar 05, 2025 8:10:44 AM hudson.remoting.Launcher createEngine
INFO: Setting up agent: cico-workspace-t93jg
Mar 05, 2025 8:10:44 AM hudson.remoting.Engine startEngine
INFO: Using Remoting version: 3261.v9c670a_4748a_9
Mar 05, 2025 8:10:44 AM hudson.remoting.Engine startEngine
WARNING: No Working Directory. Using the legacy JAR Cache location: /home/jenkins/.jenkins/cache/jars
Mar 05, 2025 8:10:44 AM hudson.remoting.Launcher$CuiListener status
INFO: Locating server among [http://172.30.104.61:80/]
Mar 05, 2025 8:10:44 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
Mar 05, 2025 8:10:44 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting TCP connection tunneling is enabled. Skipping the TCP Agent Listener Port availability check
Mar 05, 2025 8:10:44 AM hudson.remoting.Launcher$CuiListener status
INFO: Agent discovery successful
  Agent address: 172.30.121.229
  Agent port:    50000
  Identity:      c8:70:32:9c:f8:5a:23:91:f8:f8:a0:29:c8:4a:17:7b
Mar 05, 2025 8:10:44 AM hudson.remoting.Launcher$CuiListener status
INFO: Handshaking
Mar 05, 2025 8:10:44 AM hudson.remoting.Launcher$CuiListener status
INFO: Connecting to 172.30.121.229:50000
Mar 05, 2025 8:10:44 AM hudson.remoting.Launcher$CuiListener status
INFO: Server reports protocol JNLP4-connect-proxy not supported, skipping
Mar 05, 2025 8:10:44 AM hudson.remoting.Launcher$CuiListener status
INFO: Trying protocol: JNLP4-connect
Mar 05, 2025 8:10:44 AM org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader run
INFO: Waiting for ProtocolStack to start.
Mar 05, 2025 8:10:44 AM hudson.remoting.Launcher$CuiListener status
INFO: Remote identity confirmed: c8:70:32:9c:f8:5a:23:91:f8:f8:a0:29:c8:4a:17:7b
Mar 05, 2025 8:10:44 AM hudson.remoting.Launcher$CuiListener status
INFO: Connected
Mar 05, 2025 8:10:44 AM hudson.remoting.UserRequest perform
WARNING: LinkageError while performing UserRequest:hudson.slaves.SlaveComputer$SlaveVersion@328c4ac3
java.lang.UnsupportedClassVersionError: Failed to load hudson.slaves.SlaveComputer$SlaveVersion
    at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:472)
    at hudson.remoting.RemoteClassLoader.loadRemoteClass(RemoteClassLoader.java:301)
    at hudson.remoting.RemoteClassLoader.loadWithMultiClassLoader(RemoteClassLoader.java:277)
    at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:236)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
    at java.base/java.lang.Class.forName0(Native Method)
    at java.base/java.lang.Class.forName(Class.java:398)
    at hudson.remoting.MultiClassLoaderSerializer$Input.resolveClass(MultiClassLoaderSerializer.java:133)
    at java.base/java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:2003)
    at java.base/java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1870)
    at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2201)
    at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1687)
    at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:489)
    at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:447)
    at hudson.remoting.UserRequest.deserialize(UserRequest.java:312)
    at hudson.remoting.UserRequest.perform(UserRequest.java:196)
    at hudson.remoting.UserRequest.perform(UserRequest.java:50)
    at hudson.remoting.Request$2.run(Request.java:391)
    at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:81)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:140)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.UnsupportedClassVersionError: hudson/slaves/SlaveComputer$SlaveVersion has been compiled by a more recent version of the Java Runtime (class file version 61.0), this version of the Java Runtime only recognizes class file versions up to 55.0
    at java.base/java.lang.ClassLoader.defineClass1(Native Method)
    at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1017)
    at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:878)
    at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:470)
    ... 24 more

Mar 05, 2025 8:10:44 AM hudson.remoting.Launcher$CuiListener status
INFO: Terminated
Mar 05, 2025 8:10:55 AM hudson.remoting.Launcher$CuiListener status
INFO: Performing onReconnect operation.
Mar 05, 2025 8:10:55 AM hudson.remoting.Launcher$CuiListener status
INFO: onReconnect operation completed.
Mar 05, 2025 8:10:55 AM hudson.remoting.Launcher$CuiListener status
INFO: Locating server among [http://172.30.104.61:80/]
Mar 05, 2025 8:10:55 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
Mar 05, 2025 8:10:55 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting TCP connection tunneling is enabled. Skipping the TCP Agent Listener Port availability check
Mar 05, 2025 8:10:55 AM hudson.remoting.Launcher$CuiListener status
INFO: Agent discovery successful
  Agent address: 172.30.121.229
  Agent port:    50000
  Identity:      c8:70:32:9c:f8:5a:23:91:f8:f8:a0:29:c8:4a:17:7b
Mar 05, 2025 8:10:55 AM hudson.remoting.Launcher$CuiListener status
INFO: Handshaking
Mar 05, 2025 8:10:55 AM hudson.remoting.Launcher$CuiListener status
INFO: Connecting to 172.30.121.229:50000
Mar 05, 2025 8:10:55 AM hudson.remoting.Launcher$CuiListener status
INFO: Server reports protocol JNLP4-connect-proxy not supported, skipping
Mar 05, 2025 8:10:55 AM hudson.remoting.Launcher$CuiListener status
INFO: Trying protocol: JNLP4-connect
Mar 05, 2025 8:10:55 AM org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader run
INFO: Waiting for ProtocolStack to start.
Mar 05, 2025 8:10:55 AM hudson.remoting.Launcher$CuiListener status
INFO: Remote identity confirmed: c8:70:32:9c:f8:5a:23:91:f8:f8:a0:29:c8:4a:17:7b
Mar 05, 2025 8:10:55 AM hudson.remoting.Launcher$CuiListener status
INFO: Connected
Mar 05, 2025 8:10:55 AM hudson.remoting.UserRequest perform
WARNING: LinkageError while performing UserRequest:hudson.slaves.SlaveComputer$SlaveVersion@4950db43
java.lang.UnsupportedClassVersionError: Failed to load hudson.slaves.SlaveComputer$SlaveVersion
    at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:472)
    at hudson.remoting.RemoteClassLoader.loadRemoteClass(RemoteClassLoader.java:301)
    at hudson.remoting.RemoteClassLoader.loadWithMultiClassLoader(RemoteClassLoader.java:277)
    at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:236)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
    at java.base/java.lang.Class.forName0(Native Method)
    at java.base/java.lang.Class.forName(Class.java:398)
    at hudson.remoting.MultiClassLoaderSerializer$Input.resolveClass(MultiClassLoaderSerializer.java:133)
    at java.base/java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:2003)
    at java.base/java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1870)
    at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2201)
    at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1687)
    at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:489)
    at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:447)
    at hudson.remoting.UserRequest.deserialize(UserRequest.java:312)
    at hudson.remoting.UserRequest.perform(UserRequest.java:196)
    at hudson.remoting.UserRequest.perform(UserRequest.java:50)
    at hudson.remoting.Request$2.run(Request.java:391)
    at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:81)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:140)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.UnsupportedClassVersionError: hudson/slaves/SlaveComputer$SlaveVersion has been compiled by a more recent version of the Java Runtime (class file version 61.0), this version of the Java Runtime only recognizes class file versions up to 55.0
    at java.base/java.lang.ClassLoader.defineClass1(Native Method)
    at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1017)
    at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:878)
    at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:470)
    ... 24 more

Mar 05, 2025 8:10:55 AM hudson.remoting.Launcher$CuiListener status
INFO: Terminated

instance: https://jenkins-samba.apps.ocp.cloud.ci.centos.org/

Jenkins recently (starting with 2.479) started mandating Java 17+ and controller and agent Java versions need to match for things to work.

Looking at quay.io/centosci/cico-workspace:latest it still uses Java 11, and I see no relevant changes in the "upstream" quay.io/openshift/origin-jenkins-agent-base:latest container either.

Can confirm I see the same issue in our Jenkins instance after the upgrade. In my case I don't see any configured clouds, since the Jenkins can't enable the OpenShift sync plugin due to dependency issues:

Some plugins could not be loaded due to unsatisfied dependencies. Fix these issues and restart Jenkins to re-enable these plugins.

Dependency errors:

Kubernetes plugin (4306.vc91e951ea_eb_d)
    Update required: Credentials Binding Plugin (credentials-binding 642.v737c34dea_6c2) to be updated to 657.v2b_19db_7d6e6d or higher

Some of the above failures also result in additional indirectly dependent plugins not being able to load.

Indirectly dependent plugins:

OpenShift Sync (1.1.0.802.v45585f8cdc07)
    Failed to load: Kubernetes plugin (kubernetes 4306.vc91e951ea_eb_d)

But I can't upgrade the Credentials Binding Plugin, because the Jenkins version is too old:

Credentials BindingVersion 687.v619cb_15e923f
Build Wrapperscredentials
Allows credentials to be bound to environment variables for use from miscellaneous build steps.
Warning: This plugin requires newer versions of dependencies and at least one of those plugins is not compatible with the installed version. This is usually the case because its behavior changed, or it uses a different settings format than the installed version. Jobs using that plugin may need to be reconfigured, and/or you may not be able to cleanly revert to the prior version without manually restoring old settings. Consult the plugin release notes for details.
The following plugins are incompatible:

    Plain Credentials (182.v468b_97b_9dcb_8)
    Credentials (1337.v60b_d7b_c7b_c9f)
    SSH Credentials (308.ve4497b_ccd8f4)

All plugins in the instance are currently on their latest compatible versions, at least according to the plugin manager.

In the Ceph-CSI project we hit the same problem as what Samba sees.

This seems to be the main issue:

Caused by: java.lang.UnsupportedClassVersionError: hudson/slaves/SlaveComputer$SlaveVersion has been compiled by a more recent version of the Java Runtime (class file version 61.0), this version of the Java Runtime only recognizes class file versions up to 55.0

Because of that, the cico-workspace pods do not come up, and jobs do not start to run.

Metadata Update from @arrfab:
- Issue assigned to arrfab

Metadata Update from @arrfab:
- Issue tagged with: centos-ci-infra, high-gain, high-trouble, investigation

I managed to resolve the borked plugin stuff I mentioned before by manually downgrading the kubernetes plugin to 4174.v4230d0ccd951 (no idea how it got upgraded to the incompatible version in the first place...), so I can now finally join the can't run cico-workspace pods club:

Caused by: java.lang.UnsupportedClassVersionError: hudson/slaves/SlaveComputer$SlaveVersion has been compiled by a more recent version of the Java Runtime (class file version 61.0), this version of the Java Runtime only recognizes class file versions up to 55.0

@mrc0mmand : currently working on a staging version of updated cico-workspace container to give it a try .. easier/better to sync on irc for now but we'll post updates here as we progress

status update : while I think that we should have a staging container soon (see https://github.com/centosci/images/commit/04f69a45905e3165e5a115cb30fb8f9bfe2ed10d), we can't test it right now as despite multiple attemps, quay.io doesn't seem to be able to build anything.
The github => quay.ci trigger works, but the only output we have from quay.io is : 

An internal system error occurred while building; the build will be retried in the next few minutes.

And then nothing :/
I pinged people from quay.io to have a look but no answer so far

Metadata Update from @arrfab:
- Issue priority set to: Waiting on External (was: Needs Review)
- Issue tagged with: blocked

quay.io/centosci/cico-workspace:staging seems to be working for Ceph-CSI.

I've update the image location (was :latest) under Cloud/openshift/templates/cico-workspace in the Jenkins admin UI and started a job with succes.

Metadata Update from @arrfab:
- Issue untagged with: blocked

@devos : thanks for the feedback !
I just promoted to :latest (see https://quay.io/repository/centosci/cico-workspace?tab=tags) and so that should fix the issue for all tenants using by default that image for pod definition.

For the record, quay.io integration with github is broken so wasn't triggering any proper build. I had to move to gitlab.com (https://gitlab.com/CentOS/infra/containers/cico-workspace) for a new repo and setup webhook to trigger build at quay (how I was able to have :staging and :main/:latest working again)

I'd like to see if we still have other issues as it seems some people complained about jenkins having lost it's sync/configmap to define the cloud env itself from within jenkins ...

Jenkins recently (starting with 2.479) started mandating Java 17+ and controller and agent Java versions need to match for things to work.

Looking at quay.io/centosci/cico-workspace:latest it still uses Java 11, and I see no relevant changes in the "upstream" quay.io/openshift/origin-jenkins-agent-base:latest container either.

that upstream container isn't maintained anymore by openshift .. they were supposed to switch to a new one (based on ubi9) but nowhere to be seen (see https://github.com/openshift/jenkins/issues/1766)

As confirmed by other tenants on irc (#centos-ci on irc.libera.chat) it seems resolved now.
Closing for now but feel free to reopen in case something else is needed

PS : the fact that for some tenants they aren't on correct imagestream is "interesting" but probably because they update themselvers their deploymentconfig some time ago and so not using the imagestream jenkins:2 that is tracking upstream one

Metadata Update from @arrfab:
- Issue close_status updated to: Fixed with Explanation
- Issue status updated to: Closed (was: Open)

Log in to comment on this ticket.

Metadata

centos-ci-infra high-gain high-trouble investigation

None
None
Powered by Pagure 1.0.0
Documentation • File an Issue • About • SSH Hostkey/Fingerprint
© Red Hat, Inc. and others.