It sounds like NSS isn't seeing the token. If NSS doesn't report it to certmonger then there isn't anything I can do.

Try generating a key on it:

certutil -G -g 2048 -k rsa -d /etc/pki/nssdb -h "my first token"

If that success, try listing the key

certutil -K -d /etc/pki/nssdb -h "my first token"

That should tell us whether NSS can use the token.

Thank you for your fast reply, the commands worked fine:

certutil -G -g 2048 -k rsa -d /etc/pki/nssdb -h "my first token"

Enter Password or Pin for "my first token":

A random seed must be generated that will be used in the
creation of your key. One of the easiest ways to create a
random seed is to use the timing of keystrokes on a keyboard.

To begin, type keys on the keyboard until this progress meter
is full. DO NOT USE THE AUTOREPEAT FUNCTION ON YOUR KEYBOARD!

Continue typing until the progress meter is full:

|************|

Finished. Press enter to continue:

Generating key. This may take a few moments...

[root@localhost user]# certutil -K -d /etc/pki/nssdb -h "my first token"
certutil: Checking token "my first token" in slot "my first token"
Enter Password or Pin for "my first token":
< 0> rsa df16639f7ad35bc08c38605bd229e809334a9138 (orphan)

[root@localhost user]# pkcs11-tool -v --module /usr/lib64/pkcs11/libtpm2_pkcs11.so --slot-index=0 --list-objects
Using slot with index 0 (0x1)
Public Key Object; RSA 2048 bits
label:
ID: df16639f7ad35bc08c38605bd229e809334a9138
Usage: encrypt, verify
Access: local

...and also modutil (part of nss-tools) sees the token. Strange that NSS does not report it to certmonger though, while it actually reports the empty/uninitialized one.

If there is something else you need me to test, please let me know.

What release of certmonger are you using?

I've made quite a few improvements to token handling in certmonger over the last year or so.

I'm on the latest one (from RHEL 9)

[root@localhost user]# certmonger -v
certmonger 0.79.17
[root@localhost user]# yum list certmonger
Last metadata expiration check: 2:55:41 ago on Mon 26 Feb 2024 11:13:16 AM CET.
Installed Packages
certmonger.x86_64 0.79.17-2.el9 @appstream
Available Packages
certmonger.src 0.79.17-2.el9 appstream-source
[root@localhost user]# cat /etc/redhat-release
CentOS Stream release 9

Should I try an older version?

A newer version might work better but you'd have to build it yourself. 0.79.19 has some fixes in token handling.

Thank you, I've built it (from your fc41 src rpm with "rpmbuild --rebuild certmonger-0.79.19-5.fc41.src.rpm") , certmonger now reports:
[root@localhost user]# certmonger -v
certmonger 0.79.19

...however, the token is still not found, I see the same debug messages:
Feb 27 19:32:29 localhost.localdomain certmonger[2683]: 2024-02-27 19:32:29 [2683] Read value "0" from "/proc/sys/crypto/fips_enabled".
Feb 27 19:32:29 localhost.localdomain certmonger[2683]: 2024-02-27 19:32:29 [2683] Not attempting to set NSS FIPS mode.
Feb 27 19:32:29 localhost.localdomain certmonger[2683]: 2024-02-27 19:32:29 [2683] Found token 'NSS Certificate DB'.
Feb 27 19:32:29 localhost.localdomain certmonger[2683]: 2024-02-27 19:32:29 [2683] Token is named "NSS Certificate DB", not "my first token", skipping.
Feb 27 19:32:29 localhost.localdomain certmonger[2683]: 2024-02-27 19:32:29 [2683] Skipping NSS internal slot (NSS Generic Crypto Services).
Feb 27 19:32:29 localhost.localdomain certmonger[2683]: 2024-02-27 19:32:29 [2683] Found token ''.
Feb 27 19:32:29 localhost.localdomain certmonger[2683]: 2024-02-27 19:32:29 [2683] Token is named "", not "my first token", skipping.
Feb 27 19:32:29 localhost.localdomain certmonger[2683]: 2024-02-27 19:32:29 [2683] Error locating a key.

Something else I could try?...this issue should be reproducible, I was at work today and was able to reproduce it on a RHEL 8.9 fully updated.

Unfortunately I don't have a machine with a TPM to try to duplicate it myself.

This is a stab in the dark, but maybe this change will help:

diff --git a/src/keyiread-n.c b/src/keyiread-n.c
index c2f3928b..c68b0a71 100644
--- a/src/keyiread-n.c
+++ b/src/keyiread-n.c
@@ -145,7 +145,7 @@ cm_keyiread_n_get_keys(struct cm_store_entry *entry,
 int readwrite)
        }
        /* Find the tokens that we might use for key storage. */
-       mech = 0;
+       mech = CKM_INVALID_MECHANISM;
        slotlist = PK11_GetAllTokens(mech, PR_FALSE, PR_FALSE, NULL);
        if (slotlist == NULL) {
                cm_log(1, "Error locating token to be used for key storage.\n");

Thank you, but it did not change the outcome.
I see the same code in another file but I'm not sure if I should try the change in that one as well:
root@localhost tmp]# grep -rC3 "CKM_INVALID_MECHANISM" ./usr/
./usr/src/debug/certmonger-0.79.19-5.el9.x86_64/src/certsave-n.c- _exit(CM_CERTSAVE_STATUS_INTERNAL_ERROR);
./usr/src/debug/certmonger-0.79.19-5.el9.x86_64/src/certsave-n.c- }
./usr/src/debug/certmonger-0.79.19-5.el9.x86_64/src/certsave-n.c- / Find the tokens that we might use for cert storage. /
./usr/src/debug/certmonger-0.79.19-5.el9.x86_64/src/certsave-n.c: mech = CKM_INVALID_MECHANISM;
./usr/src/debug/certmonger-0.79.19-5.el9.x86_64/src/certsave-n.c- slotlist = PK11_GetAllTokens(mech, PR_FALSE, PR_FALSE, NULL);
./usr/src/debug/certmonger-0.79.19-5.el9.x86_64/src/certsave-n.c- if (slotlist == NULL) {
./usr/src/debug/certmonger-0.79.19-5.el9.x86_64/src/certsave-n.c- cm_log(1, "Error getting list of tokens.\n");
--
./usr/src/debug/certmonger-0.79.19-5.el9.x86_64/src/keyiread-n.c- }
./usr/src/debug/certmonger-0.79.19-5.el9.x86_64/src/keyiread-n.c-
./usr/src/debug/certmonger-0.79.19-5.el9.x86_64/src/keyiread-n.c- / Find the tokens that we might use for key storage. /
./usr/src/debug/certmonger-0.79.19-5.el9.x86_64/src/keyiread-n.c: mech = CKM_INVALID_MECHANISM;
./usr/src/debug/certmonger-0.79.19-5.el9.x86_64/src/keyiread-n.c- slotlist = PK11_GetAllTokens(mech, PR_FALSE, PR_FALSE, NULL);
./usr/src/debug/certmonger-0.79.19-5.el9.x86_64/src/keyiread-n.c- if (slotlist == NULL) {
./usr/src/debug/certmonger-0.79.19-5.el9.x86_64/src/keyiread-n.c- cm_log(1, "Error locating token to be used for key storage.\n");
[root@localhost tmp]# grep -rC3 "mech = 0" ./usr/
./usr/src/debug/certmonger-0.79.19-5.el9.x86_64/src/submit-n.c- }
./usr/src/debug/certmonger-0.79.19-5.el9.x86_64/src/submit-n.c-
./usr/src/debug/certmonger-0.79.19-5.el9.x86_64/src/submit-n.c- / Find the tokens that we might use for key storage. /
./usr/src/debug/certmonger-0.79.19-5.el9.x86_64/src/submit-n.c: mech = 0;
./usr/src/debug/certmonger-0.79.19-5.el9.x86_64/src/submit-n.c- slotlist = PK11_GetAllTokens(mech, PR_FALSE, PR_FALSE, NULL);
./usr/src/debug/certmonger-0.79.19-5.el9.x86_64/src/submit-n.c- if (slotlist == NULL) {
./usr/src/debug/certmonger-0.79.19-5.el9.x86_64/src/submit-n.c- cm_log(1, "Error locating token to be used for key storage.\n");

Concerning the TPM, I actually tried on a test VM with a virtualTPM (from VMware player, however I guess RHEL virtualization has a vTPM as well). At work I have RHEL licenses, so I can open a support ticket, if you prefer...in which case the support should provide a VM with vTPM for testing purposes, I presume.

I also see there is also a TPM emulator (swtpm), I'll try to see if I can reproduce an that one and report back.

It is reporting the failure in keyiread-n.c. A similar "token not seen" error was seen with hardware tokens related to certs (certsave-n.c). I have not seen an issue when submitting requests against a token.
I avoided swtpm on the chance that it doesn't fully emulate a TPM.
I also have no experience with tpm2_pkcs11 whose setup seems rather complicated and convoluted. Do you have a pointer to simple instructions on initializing a token? The docs in the project README are not sufficient for me: FAPI errors out the wazoo, tokens not showing using pk11tool, etc, modutil showing unknown slots and other missing tokens.

Sure, I can explain my setup, however it's in a VM running on vm-player. Not sure if you testing in the same conditions.
So I tested by creating a new VM (on latest vmware player) with vTPM (added from the before starting the OS install) and firmware type UEFI with the following steps:
1) fresh standard (no sec profile chosen) install of Workstation from latest CentOS Stream 9 ISO or RHEL 9.3 (worked the same for both)

2) once system booted:
[root@localhost user]# yum install nss-tools certmonger tpm2-pkcs11 opensc

3) As per the link in my first post, create an alias [root@localhost user]# alias tpm2pkcs11-tool="pkcs11-tool --module /usr/lib64/pkcs11/libtpm2_pkcs11.so"

4) list token slots: [root@localhost user]# tpm2pkcs11-tool --list-token-slots
WARNING:fapi:src/tss2-fapi/ifapi_io.c:339:ifapi_io_check_create_dir() Directory /root/.local/share/tpm2-tss/user/keystore does not exist, creating
WARNING:fapi:src/tss2-fapi/ifapi_io.c:339:ifapi_io_check_create_dir() Directory /var/lib/tpm2-tss/system/keystore/policy does not exist, creating
WARNING:fapi:src/tss2-fapi/api/Fapi_List.c:226:Fapi_List_Finish() Profile of path not provisioned: /HS/SRK
ERROR:fapi:src/tss2-fapi/api/Fapi_List.c:81:Fapi_List() ErrorCode (0x00060034) Entities_List
ERROR: Listing FAPI token objects failed.
Available slots:
Slot 0 (0x1):
token state: uninitialized

5) ...the FAPI errors from previous step are harmless (according to https://github.com/tpm2-software/tpm2-pkcs11/issues/655 ) and can be suppressed with:
root@localhost user]# export TSS2_LOG=fapi+NONE
try again to list slots :
[root@localhost user]# tpm2pkcs11-tool --list-token-slots
ERROR: Listing FAPI token objects failed.
Available slots:
Slot 0 (0x1):
token state: uninitialized

....one FAPI message still persists, considered harmless as per above link

6) Initialize a token at Slot Index 0:
[root@localhost user]# tpm2pkcs11-tool --slot-index=0 --init-token --label="my first token" --so-pin="mysopin"
ERROR: Listing FAPI token objects failed.
Using slot with index 0 (0x1)
Token successfully initialized

7) list the slots again:
[root@localhost user]# tpm2pkcs11-tool --list-token-slots
ERROR: Listing FAPI token objects failed.
Available slots:
Slot 0 (0x1): my first token
token label : my first token
token manufacturer : VMW
token model : VMware TPM2
token flags : login required, rng, token initialized, PIN initialized
hardware version : 1.16
firmware version : 2.101
serial num : 0000000000000000
pin min/max : 0/128
Slot 1 (0x2):
token state: uninitialized

8) set the user PIN:
[root@localhost user]# tpm2pkcs11-tool --slot-index=0 --init-pin --so-pin="mysopin" --login --pin="myuserpin"
ERROR: Listing FAPI token objects failed.
Using slot with index 0 (0x1)
User PIN successfully initialized

9) list the modules in default nssdb:
[root@localhost user]# modutil -dbdir sql:/etc/pki/nssdb/ -list
ERROR: Listing FAPI token objects failed.

Listing of PKCS #11 Modules

1. NSS Internal Crypto Services
   uri: pkcs11:library-manufacturer=Mozilla%20Foundation;library-description=NSS%20Internal%20Crypto%20Services;library-version=3.90
   slots: 2 slots attached
   status: loaded

   slot: NSS Internal Cryptographic Services
   token: NSS Generic Crypto Services
   uri: pkcs11:token=NSS%20Generic%20Crypto%20Services;manufacturer=Mozilla%20Foundation;serial=0000000000000000;model=NSS%203

   slot: NSS User Private Key and Certificate Services
   token: NSS Certificate DB
   uri: pkcs11:token=NSS%20Certificate%20DB;manufacturer=Mozilla%20Foundation;serial=0000000000000000;model=NSS%203

2. p11-kit-proxy
   library name: p11-kit-proxy.so
   uri: pkcs11:library-manufacturer=PKCS%2311%20Kit;library-description=PKCS%2311%20Kit%20Proxy%20Module;library-version=1.1
   slots: 2 slots attached
   status: loaded

   slot: my first token
   token: my first token
   uri: pkcs11:token=my%20first%20token;manufacturer=VMW%20%20;serial=0000000000000000;model=VMware%20TPM2

   slot:
   token:
   uri: pkcs11:manufacturer=VMW%20%20;serial=0000000000000000;model=VMware%20TPM2

   ---

...our token appears (via p11-kit-proxy) without any extra config

10) optionally get more details about the tpm2-pkcs11 tokens:
[root@localhost user]# modutil -dbdir sql:/etc/pki/nssdb/ -list p11-kit-proxy
ERROR: Listing FAPI token objects failed.

Name: p11-kit-proxy
Library file: p11-kit-proxy.so
Manufacturer: PKCS#11 Kit
Description: PKCS#11 Kit Proxy Module
PKCS #11 Version 3.0
Library Version: 1.1
Cipher Enable Flags: None
Default Mechanism Flags: None

Slot: my first token
Slot Mechanism Flags: None
Manufacturer: VMW
Type: Hardware
Version Number: 1.16
Firmware Version: 2.101
Status: Enabled
Token Name: my first token
Token Manufacturer: VMW
Token Model: VMware TPM2
Token Serial Number: 0000000000000000
Token Version: 1.16
Token Firmware Version: 2.101
Access: NOT Write Protected
Login Type: Login required
User Pin: Initialized

Slot:
Slot Mechanism Flags: None
Manufacturer: VMW
Type: Hardware
Version Number: 1.16
Firmware Version: 2.101
Status: Enabled
Token Name:
Token Manufacturer: VMW
Token Model: VMware TPM2
Token Serial Number: 0000000000000000
Token Version: 1.16
Token Firmware Version: 2.101
Access: NOT Write Protected
Login Type: Login required
User Pin: NOT Initialized

11) try rsa key generation:
[root@localhost user]# certutil -G -g 2048 -k rsa -d /etc/pki/nssdb -h "my first token"
ERROR: Listing FAPI token objects failed.
Enter Password or Pin for "my first token":

A random seed must be generated that will be used in the
creation of your key. One of the easiest ways to create a
random seed is to use the timing of keystrokes on a keyboard.

To begin, type keys on the keyboard until this progress meter
is full. DO NOT USE THE AUTOREPEAT FUNCTION ON YOUR KEYBOARD!

Continue typing until the progress meter is full:

|************|

Finished. Press enter to continue:

Generating key. This may take a few moments...

12) list the key:
[root@localhost user]# certutil -K -d /etc/pki/nssdb -h "my first token"
ERROR: Listing FAPI token objects failed.
certutil: Checking token "my first token" in slot "my first token"
Enter Password or Pin for "my first token":
< 0> rsa 861ac3960948de671db2c6604148cd3d33647293 (orphan)

or:
[root@localhost user]# tpm2pkcs11-tool --slot-index=0 --list-objects
ERROR: Listing FAPI token objects failed.
Using slot with index 0 (0x1)
Public Key Object; RSA 2048 bits
label:
ID: 861ac3960948de671db2c6604148cd3d33647293
Usage: encrypt, verify
Access: local

I should also mention that for certmonger testing I've also disabled selinux, as it was interfering.

It's a state problem with the PKCS#11 module. Running commands as root the state is stored in /root/.tpm2_pkcs11 if /etc/tpm2_pkcs11/tpm2_pkcs11.sqlite3 doesn't exist.

When running in certmonger the module is using /run/certmonger/.tpm2_pkcs11

The directory is created empty on first use. So it truly has no idea about the token you created as root.

I don't know why /run/certmonger is used. It may be some environment variable that certmonger or systemd sets but I didn't find one after a brief look.

So if you move /root/.tpm2_pkcs11/tpm2_pkcs11.sqlite3 to /etc/tpm2_pkcs11/tpm2_pkcs11.sqlite3 then certmonger should work.

I had to create the /etc/tpm2_pkcs11 directory myself.

Thank you! yes, certmonger finds the token if "tpm2_pkcs11.sqlite3" is moved into /etc/tpm2_pkcs11/ .

I also tested with a normal (non-root) user and the same problem exists. So, I guess it means that the module, when running in certmonger, does not check the user's "default" tpm2_pkcs11 store (for ex, for root "/root/.tpm2_pkcs11) , instead it looks in the "central" one which does not exist by default on RHEL (/etc/tpm2_pkcs11) and maybe in the (empty) one in certmonger's working directory ( /run/certmonger/.tpm2_pkcs11)?

Tomorrow at work I'll also have the SCEP server so I'll try a full enrollment and report back.

Based on strace output it looks in /etc/tpm2_pkcs11 first then falls back to, well, something. In this case depending on context either /root/.tpm2_pkcs11 or within certmonger as /run/certmonger/.tpm2_pkcs11. I didn't spend a lot of time investigating the fallback. It would probably require digging through the tpm2_pkcs11 code. I'm guessing it's an environment variable or something set by systemd influencing it. I don't think it's worth figuring out since the central store is most reliable and would require fewer SELinux permissions to allow.

A full request to SCEP CA (with keypair generated by certmonger in the token) worked just fine.

For the fallback, you are right, it's explained in their code https://github.com/tpm2-software/tpm2-pkcs11/blob/master/tools/tpm2_pkcs11/command.py ( def get_default_store_path() ) and line 90:
3 The location of the store directory. If specified, the directory MUST exist.
If not specified performs a search by looking at environment variable
TPM2_PKCS11_STORE and if not set then /etc/tpm2_pkcs11 and if not found or
no write access, then $HOME/.tpm2_pkcs11 and if not found or cannot be created,
then defaults to using the current working directory."
which would explain the fallback to certmonger's working dir, /run/certmonger/.tpm2_pkcs11, as all other options did not exist.

With the central store it works fine, not sure why it's not there by default...maybe it could be an issue as it's shared by root and normal users alike, as the users will also need write access to that sqlite db, which would mean accessing root's tokens?

Thank you again for your help!

I'm glad it's working for you.

I'd suggest filing a Fedora BZ against the tpm2_pkcs11 package asking to at least create the /etc/tpm2_pkcs11 directory by default. IIRC there is limited storage in a TPM so perhaps allowing users to store data there is not desired by default.

Marking ticket as done.