our monitoring started to confirm that just ~6min before you created the ticket :)
It should be all back online afaics

Do we know if we've lost email in the archives?
I'm seeing one locally that I can't find in hyperkitty :(

(going to ask if I'm the only one to have received it)

If that helps, the headers say it should be accessible at: https://lists.centos.org/hyperkitty/list/automotive-sig@lists.centos.org/message/2LBML5IF5JJWBKRCK27N3DJTAMHVIFGY/

But even from https://lists.centos.org/hyperkitty/list/automotive-sig@lists.centos.org/ I don't see the thread at all (subject is: [Centos-automotive-sig] Re: Proposal for SIG docs update).

I confirm that they were processed by mailman itself (so mail were delivered) but problem seems to be at the hyperkitty level, unable to parse some mails and so we have 3 mails in the hyperkitty spool/queue, not archived :

pwd ; ll
/var/lib/mailman3/archives/hyperkitty/spool
total 32
-rw-rw----. 1 mailman mailman  7899 Sep  3 21:36 1725399390.4403923+ebe5797c3f1e12a3e672e554580bb9d898e1084c.pck
-rw-rw----. 1 mailman mailman 11182 Sep  3 21:36 1725399390.4953074+0eb160ef72d1c0b044d50934bcc0901d008b2207.pck
-rw-rw----. 1 mailman mailman 10557 Sep  3 21:36 1725399390.5693307+a7d2e444d6037fbed8654914f7d2d4b810898a3a.pck

mailman3 is supposed to try in loop mails it wasn't able to archive to hyperkitty but there is probably another issue maybe in the way the mails are formatted . BTW, I confirm (by looking at the pickle files) that these are just for automotive-sig

just curious : does it archive mails sent after these ones ? I don't see anything obvious in mailman log but it's also not in verbose/debug mode (that we don't want to turn on by default for obvious reasons)

I replied to the original email and I don't see my reply in there to :(

I had a quick look at the mails stuck in the spool (base64 encoded) and I realised that they contain unicode (emoji).
So in parallel, I found https://gitlab.com/mailman/hyperkitty/-/issues/248 which explains the mysql charset.

I looked at our mysql db/table for hyperkitty/mailmanweb and they were already declared as utf8mb4 :

mysql> SELECT CCSA.character_set_name FROM information_schema.`TABLES` T,
    ->        information_schema.`COLLATION_CHARACTER_SET_APPLICABILITY` CCSA
    -> WHERE CCSA.collation_name = T.table_collation
    ->   AND T.table_schema = "mailmanweb"
    ->   AND T.table_name = "hyperkitty_email";  
+--------------------+
| CHARACTER_SET_NAME |
+--------------------+
| utf8mb4            |
+--------------------+
1 row in set (0.00 sec)

But following the links in the hyperkitty bug report, I saw that they added an option (commented out) for mysql to also enfoce utb8mb4 for the databases settings and so this was backported to our ansible role and applied :

https://github.com/CentOS/ansible-role-mailman3/commit/304f9950ec858ba63f793eb79d1608b7dbd99dca

All mails in the hyperkitty spool were then processed and started to appear (for example : https://lists.centos.org/hyperkitty/list/automotive-sig@lists.centos.org/thread/AZC7VQLYJGKCEZZQ4SFXKTGZAHLFLQL7/ which, if you expand history would have the emoji/unicode )

So closing this ticket with explanations and that should ensure compatibility with unicode from now on (was working for mailman core, so smtp mails going in/out)

Thanks for tracking and fixing this! :)