it looks like we are somehow broken in scheduling jobs since Wednesday (12/13). We have 21 jobs in our queue but nothing is scheduled with ‘Jenkins’ doesn’t have label ‘cico-workspace’ error https://jenkins-networkmanager.apps.ocp.cloud.ci.centos.org/job/NetworkManager-1.44-c9s/227/console
restarting jenkins didn't help either
I can see executors in launching and suspending state, no idea what that means
Metadata Update from @arrfab: - Issue assigned to arrfab
Metadata Update from @arrfab: - Issue tagged with: centos-ci-infra, high-gain, high-trouble
after quick investigation , we decided (also due to lack of time to investigate the real root cause) to just reprovision the jenkins pod from ansible, and rsync job/content to the newly provisioned PV (on AWS EFS) That's currently in progress and we'll restart jenkins when done. no ETA (aws efs is slow from the control node)
Status update (discussion happened in "sync" on irc) : new jenkins was redeployed again and dufyf api key updated in secret. All jobs were moved to storage accessible by @vbenes and so moving these to correct folder and reloading config from disk (see for example https://sigs.centos.org/guide/ci/#migration-to-new-ci-instance) should be enough.
Closing so ticket as Infra work is now done
Metadata Update from @arrfab: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.