#284 Staging deployment: all syncing fails
Closed: Fixed by frantisekz. Opened by frantisekz.

For some time (since 2024-03-26 07:00:03 UTC), the stg instance of our app is unable to do any syncing with:

2024-04-05 07:35:28 DEBUG [blockerbugs ] Using production config [__init__:41]
2024-04-05 07:35:28 DEBUG [blockerbugs ] Using openshift config [__init__:44]
2024-04-05 07:35:28 INFO [blockerbugs ] No extra config found, using defaults [__init__:59]
2024-04-05 07:35:29 INFO [bugzilla.base] Using RHBugzilla for URL containing .redhat.com
Traceback (most recent call last):
File "/opt/app-root/src/blockerbugs/cli.py", line 367, in <module>
exit = main()
^^^^^^
File "/opt/app-root/src/blockerbugs/cli.py", line 363, in main
args.func(args)
File "/opt/app-root/src/blockerbugs/cli.py", line 210, in sync
sync_bugs(args)
File "/opt/app-root/src/blockerbugs/cli.py", line 189, in sync_bugs
sync = BugSync(db)
^^^^^^^^^^^
File "/opt/app-root/lib64/python3.11/site-packages/blockerbugs/util/bug_sync.py", line 41, in __init__
bz_interface.BlockerBugs())
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/app-root/lib64/python3.11/site-packages/blockerbugs/util/bz_interface.py", line 90, in __init__
self.bz: bugzilla.Bugzilla = bz or create_bugzilla()
^^^^^^^^^^^^^^^^^
File "/opt/app-root/lib64/python3.11/site-packages/blockerbugs/util/bz_interface.py", line 57, in create_bugzilla
assert ok is True
^^^^^^^^^^
AssertionError

Issue tagged with: next

Lovely, thanks. This is where it fails:

    if bz.api_key:
        # Verify successful authentication. This immediately raises an exception if auth fails.
        ok = bz.logged_in
        assert ok is True

Our API key expired, I just checked. I'll create a new one.

Metadata Update from @kparal:
- Issue untagged with: next

Issue tagged with: next

I created a new bugzilla.stage API key. It's stored in our password manager. Frantisek, can you update it in proper places and deploy it? Thanks.

I also checked bugzilla production instance, and our API key should have expired on 2024-02-19. But it didn't. IIRC, they disabled API expiration on production bugzilla after complaints. But it seems they never did on staging. So at this moment, it seems we don't need to be concerned about production, just staging.

Metadata Update from @kparal:
- Issue untagged with: next

Issue tagged with: next

OMG Pagure, stop being a pita!

Metadata Update from @kparal:
- Issue assigned to frantisekz

API key in the secrets storage updated: https://pagure.io/fedora-infrastructure/issue/11877

However... the playbook isn't too happy with "fatal: [os-control01.stg.iad2.fedoraproject.org]: FAILED! =>" Sigh... will leave this open till I fix it.

Deployment issues fixed (via https://pagure.io/fedora-infra/ansible/c/a6495f1eae968c758b729564e86d86dc9258be0e?branch=main ; https://pagure.io/fedora-infra/ansible/c/8cd673e3184b3963897fc429017f4e866bc8323d?branch=main ; https://pagure.io/fedora-infra/ansible/c/3310ca5783235cbd04fe3eee763355d7d03ae22a?branch=main) and redeployed with the new keys, sync is working fine now.

Metadata Update from @frantisekz:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

This issue has been migrated to Fedora Forge:
https://forge.fedoraproject.org/quality/blockerbugs/issues/284

Please continue any further discussion there.

Log in to comment on this ticket.

Metadata