Of course, if all the arches we intend to publish for an image are configured as non-failable in the relevant composes (so the compose dies if they fail), that avoids the issue, but I don't think that's the case for all images we publish for all composes.

@dwalsh @nalin @tomsweeneyredhat @umohnan do you have any thoughts on this? I think in theory what I'd like to do is something like, for each image/repo:

• Get the list of arches for which that image built successfully in this compose
• Retrieve the existing manifest for that repo/version from the first registry in the list
• Update the manifest, replacing the existing image for each successfully built image with the new one, but retaining the existing image for any arch that failed this time but is in the existing manifest
• Publish the updated manifest

However, I can't find any indication that "retrieving the existing manifest" is a thing you can do, at least with buildah...

alternatively we can have the tool keep a local cache of past manifests, but that's rather sensitive to deployment issues and I don't really like it.

"manifest add --all" can add all of the entries from an already-pushed image index in a registry to one that's being built locally. Subsequently removing an entry that you intend to replace with a new image would still require some scripting, though.

That said, I really don't recommend getting into a situation where the contents of the images for various architectures listed in an image index differ by things other than the architecture of the binaries they contain.

Thanks. But then...how do you suggest handling the problem for tags like '40' and 'latest'? We need to publish updated images, but we don't want to lose arches that fail a compose.

Do you suggest simply not publishing unless all arches pass, or what? That can be difficult if one arch is considered much less 'supported' than others (e.g. s390x or ppc64le vs. x86_64 or aarch64). If we have a bug affecting s390x image build we don't really want that to hold up updating the x86_64 images...

I may be missing the purpose of a manifest here, too, I guess. I was kinda assuming it's VERY IMPORTANT - like if I've got e.g. FROM quay.io/fedora/fedora:40 in a Containerfile, or I'm doing podman run -i fedora-minimal:40, or I'm doing toolbox enter, the manifest with the matching tag is being used to decide what image to serve me.

If that's not what the manifest is for - if, rather, when I do podman run -i fedora-minimal:40, or whatever, the registry figures out the appropriate arch and gives me the image most recently tagged fedora-minimal:40-(arch), or whatever - it seems less of a concern. Maybe it's fine for the manifest to really record only the images from the specific compose it's associated with?

A tag can resolve to only one image or index at a given time. When pulling an image, the client gets a document back when it requests the manifest using the tag. It parses it, notices that its mediaType field marks it as an index rather than an image manifest, selects one of the entries from the index, and pulls the image using the digest from that entry.

so...basically, yes, if we're tagging the images as fedora-minimal:40-x86_64 and the manifest as fedora-minimal:40 (and so on), it is the manifest that is used when something tries to pull fedora-minimal:40?

Yes, whether or not any of the images listed in the image index tagged 40 have other tags (such as 40-x86_64) that also point to them directly is not a factor when a client attempts to pull 40.

so, we do kinda still have the problem to resolve, then. we don't want images for failed arches to just "disappear", so we either have to consider all arches we intend to build to be 'critical' and not sync if a single one fails, or we have to be okay with the 'hybrid' manifest when an arch fails. unless anyone sees another option I'm missing.

If builds failing for one architecture or another are happening frequently, but are fixable when they happen in a quick enough time frame, my suggestion would be to push an image index to 40 when all of the builds succeed, so that the images it lists are always comparable with each other, and to push each successful per-architecture build to 40-$arch when they succeed, even if that means that 40 and, say, 40-x86_64 may differ for a short time.

Well, everything is fixable, but the question is whether we always have the resources/urgency to actually fix them :)

But thanks, I think we'll go with that approach for now, and if it turns out to have problems - if we get failing arches, don't manage to fix them fast enough, and people complain about the manifests going stale - we can maybe switch to the "generate hybrid manifests" approach. I've written the code for "refuse to publish a manifest if any arch from the existing manifest failed in the new compose" approach, and I think it should be fairly easy to tweak that to the "hybrid" approach if necessary. PR coming soon (I still need to write the tests).

closing this as we did go with the "only publish if all existing arches are in the current compose" approach.