AMQP is designed to have dynamic topics, consumers can use the # character to match a topic prefix, or the * to accept any value until the next . character. Consumers interested in all AWS images can do org.fedoraproject.prod.fedora_image_uploader.published.v1.aws.#.

I agree including the release number and sub-variant in the message body is a good idea. I also think having the release number in the topic is a mistake; a consumer is more likely going to want to filter on whether it's Rawhide, Branched, or Stable. However, I do think the topic should include the sub-variant, milestone (rawhide, branched, stable), and architecture.

It shouldn't break the schema stuff as the schema isn't looked up by topic, but by a key in the AMQP message header ("fedora_messaging_schema"). That key maps to the value to look up in the Python entry point, and while it's best for it to map to the message's topic prefix as that solves the problem of ensuring it's unique, it could be any string that is valid for the Python entry point API. For the AWS messages it's fedora_image_uploader.published.v1.aws.

Having a message per architecture instead of one that covers all the architectures is a bit unfortunate and I guess we could refactor things to make that not be the case, but as handlers currently get an image at a time instead of all of them it would require a bit of shuffling. I'm open to it if it's causing a headache to use the messages, though (and if we do that, obviously we would drop the architecture from the topic).

All that to say, I'm interested in taking part of this, but I would like to leave the topic dynamic (replacing relnum with milestone).

Oh, also to be clear, if that proposal works for you and you would rather not write the code I can take care of that, just let me know :smile:

It's not having a message per arch that's the issue, that's fine. It's just that I don't really see what having the arch in the topic helps with. I'm used to 'just take the message, then if you need to know what arch it's for, that's in the message body'.

I guess if you want to filter messages through topic subscriptions? We don't really do that with any other messages in fedora that I'm aware of, though. I guess that's why it surprised me. And it feels kinda awkward to me - like I doubt we'd ever manage to put everything anyone might want to filter on into the message topic, so they're still going to have to examine the properties of the message anyway, in which case why not do all the filtering that way so it's not split across two mechanisms?

We don't have different pungi.compose messages for f41 composes, f42 composes, branched composes, rawhide composes, cloud composes, container composes...we just have org.fedoraproject.prod.pungi.compose.phase.# and you get the details from the message. We don't have different topics for bodhi based on the properties of the update, or for koji based on the properties of the build. If you look through https://fedora-messaging.readthedocs.io/en/stable/user-guide/schemas.html I can't find anything else that does it this way.

Yeah, the topics of Fedora messages are unfortunate. I didn't push hard on fixing them when migrating to AMQP and I regret not documenting "good" topics better. It seems I buried it in the docs, but yeah.

I guess I thought we'd move and then I'd have the "let's fix topics" discussion, but I was young(er) and naive(r).

It's not a huge deal for every consumer to start with a "let's apply additional filtering after the AMQP topic filtering" step, but having rich filtering available at the topic level was something I wanted for notifications, among other things.

Okay, well, I can work with however you want to do it.

The schema stuff really doesn't seem to be working, though, because if you look at https://apps.fedoraproject.org/datagrepper/raw?category=fedora_image_uploader&delta=172800 , it doesn't show the human-readable formats we define in the schema. Or...do they just need publishing somewhere? Or is datagrepper (sigh) still using the old fedmsg way of doing it, with that repo that defines all the human-readable formats?

It might be that datagrepper needs to install the fedora-image-uploader-messages package, I can chase that down after we merge the schema updates and publish a new version on PyPI

I'm going to pick this commit, tweak it a bit, and include it in the ELN PR I'm working on.

Thanks!

Okay, well, I can work with however you want to do it.

The schema stuff really doesn't seem to be working, though, because if you look at https://apps.fedoraproject.org/datagrepper/raw?category=fedora_image_uploader&delta=172800 , it doesn't show the human-readable formats we define in the schema. Or...do they just need publishing somewhere? Or is datagrepper (sigh) still using the old fedmsg way of doing it, with that repo that defines all the human-readable formats?

Opened https://github.com/fedora-infra/datagrepper/pull/708 for this