(I know it's the holiday season, just filing this so I don't forget)
Looks like since this morning Duffy fails to provision new C8S (and possibly C9S) nodes. The C8S metal pool is completely empty, and the C8S virt and C9S metal pools have several nodes permanently stuck in "provisioning". No idea if this is something Duffy specific or a more widespread issue, since various Fedora CIs (Testing Farm, Packit) are borked as well.
@nphilipp @arrfab hey, could someone please take a look at this, or at least drain the virt-ec2-t2-centos-8s-x86_64 pool, since all the machines there are stuck in "provisioning"?
virt-ec2-t2-centos-8s-x86_64
I was just looking when you updated the ticket. I'd like to see in duffy itself a good logging to detect what's wrong and so have actions but ..
In the meantime : update nodes set state='failed' where state='provisioning' and pool like '%ec2%'; and so it unblocked the provisioning so ec2 machines were provisioned (and I see automatically moved to "deployed" state)
update nodes set state='failed' where state='provisioning' and pool like '%ec2%';
Metadata Update from @arrfab: - Issue assigned to arrfab
Metadata Update from @arrfab: - Issue tagged with: centos-ci-infra, high-trouble, medium-gain
Metadata Update from @arrfab: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.