CentOS Automotive noticed that the CI and other components were having problems with unsigned packages coming from some requests to buildlogs.centos.org. With some detective work by the toolchain team it was narrowed down to the fact that requests answered by .nyc.cdn77.com servers would give unsigned packages but .fra.cdn77.com
Steps used to duplicate. 1. determine ip addresses for end nodes
ssmoogen@ssmoogen-rh:~$ host 195.181.170.19 19.170.181.195.in-addr.arpa domain name pointer 610407756.fra.cdn77.com. ssmoogen@ssmoogen-rh:~$ host 156.146.36.23 23.36.146.156.in-addr.arpa domain name pointer 137173278.nyc.cdn77.com.
$ curl -v --resolve 'buildlogs.cdn.centos.org:80:195.181.170.19' --location --output x0.rpm 'http://buildlogs.centos.org/9-stream/automotive/x86_64/packages-main/Packages/o/osbuildtest-ostree-compliance-mode-0.1-1.el9iv.noarch.rpm' $ curl -v --resolve 'buildlogs.cdn.centos.org:80:156.146.36.23' --location --output x1.rpm 'http://buildlogs.centos.org/9-stream/automotive/x86_64/packages-main/Packages/o/osbuildtest-ostree-compliance-mode-0.1-1.el9iv.noarch.rpm'
$ rpm -qip x0.rpm Name : osbuildtest-ostree-compliance-mode Version : 0.1 Release : 1.el9iv Architecture: noarch Install Date: (not installed) Group : Unspecified Size : 2331 License : GPLv2 Signature : RSA/SHA256, Mon 29 May 2023 16:10:28 EDT, Key ID 4b411a9068e964ca Source RPM : osbuildtest-ostree-compliance-mode-0.1-1.el9iv.src.rpm Build Date : Fri 31 Mar 2023 04:54:41 EDT Build Host : x86-04.bsys.centos.org Packager : CBS <cbs@centos.org> Vendor : CentOS Community Build Service Summary : Populates the ostree-compliance-mode.conf file with the checksum for the current booted deployment. Description : Populates the ostree-compliance-mode.conf file with the checksum for the current booted deployment. This is required by the ostree-compliance-mode rpm which allows the system to move to a modifiable state, in compliance with GPLv3 $ rpm -qip x1.rpm Name : osbuildtest-ostree-compliance-mode Version : 0.1 Release : 1.el9iv Architecture: noarch Install Date: (not installed) Group : Unspecified Size : 2331 License : GPLv2 Signature : (none) Source RPM : osbuildtest-ostree-compliance-mode-0.1-1.el9iv.src.rpm Build Date : Fri 31 Mar 2023 04:54:41 EDT Build Host : x86-04.bsys.centos.org Packager : CBS <cbs@centos.org> Vendor : CentOS Community Build Service Summary : Populates the ostree-compliance-mode.conf file with the checksum for the current booted deployment. Description : Populates the ostree-compliance-mode.conf file with the checksum for the current booted deployment. This is required by the ostree-compliance-mode rpm which allows the system to move to a modifiable state, in compliance with GPLv3
Fabian said to open a ticket to track this
Metadata Update from @arrfab: - Issue assigned to arrfab
Metadata Update from @arrfab: - Issue tagged with: cbs, centos-build-pipeline, high-gain, medium-trouble
When we enabled signing for all -testing tags, we processed again all these tags, to ensure that signed packages (and so referenced in repodata) would be going out. As buildlogs.centos.org is also backed by CDN77, we used their API to purge all cached files at all edge locations but it seems that it wasn't done everywhere :/
-testing
We don't have a view on all their infra but we just reissued a purge-all api call to see if that would help (it seems that just path/filename is checked at their side and because it's same NEVR from an rpm PoV it doesn't see that as a changed/updated file)
Michael Ho (in a different communication channel) said that it seems to have fixed it . If that's the case, can you confirm so that we can close this ticket ?
We reused your test case and confirm that returned .rpm packages have identical checksum :
cf997b74fe24f9a7ce2639a8a97aa07b9996bd1936d1cc9572dc2e9b3a4d11d4 x0.rpm cf997b74fe24f9a7ce2639a8a97aa07b9996bd1936d1cc9572dc2e9b3a4d11d4 x1.rpm
Metadata Update from @arrfab: - Issue priority set to: Waiting on Reporter (was: Needs Review)
I can confirm it is working as expected now.
thanks for the feedback
Metadata Update from @arrfab: - Issue close_status updated to: Fixed with Explanation - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.