After upgrading Red Hat Satellite server from the 6.15 to the 6.16 release, thousands of "cookie store unavailable: :hash, error: key not found: :hash" errors detected in Satellite's production.log

Issue

• Getting lots of errors like the following one after Satellite upgrade from the 6.15 to the 6.16 release:

  cookie store unavailable: :hash, error: key not found: :hash
  

• Operations success rate on the Satellite server was critically low due to these errors.

• When proceeding to upgrade the external Capsules, the following error was encountered:

  Failure: ERF50-5345 [Foreman::WrappedException]: Unable to connect ([ProxyAPI::ProxyException]: ERF12-7885 [ProxyAPI::ProxyException]: Unable to fetch logs ([ArgumentError]: cookie store unavailable: :hash, error: key not found: :hash) for Capsule https://loadbalancer.example.com:9090/logs)
  

• Hosts registered to the Satellite/Capsule servers report the following error in /var/log/rhsm/rhsm.log:

  2025-04-16 23:15:39,832 [WARNING] dnf.py:3689895:MainThread @cache.py:745 - Unable to query for content access updates: HTTP error (500 - Internal Server Error): cookie store unavailable: :hash, error: key not found: :hash
  2025-04-16 23:15:52,004 [INFO] dnf.py:3690631:MainThread @entcertlib.py:108 - certs updated:
  Total updates: 0
  Found (local) serial# [3458679433474532973]
  Expected (UEP) serial# [3458679433474532973]
  Added (new)
  <NONE>
  Deleted (rogue):
  <NONE>
  2025-04-16 23:16:47,408 [INFO] dnf.py:3694003:MainThread @entcertlib.py:108 - certs updated:
  Total updates: 0
  Found (local) serial# [3458679433474532973]
  Expected (UEP) serial# [3458679433474532973]
  Added (new)
  <NONE>
  Deleted (rogue):
  <NONE>
  2025-04-16 23:16:47,562 [WARNING] dnf.py:3694003:MainThread @cache.py:745 - Unable to query for content access updates: HTTP error (500 - Internal Server Error): cookie store unavailable: :hash, error: key not found: :hash
  2025-04-16 23:16:51,103 [INFO] dnf.py:3694075:MainThread @entcertlib.py:108 - certs updated:
  Total updates: 0
  

• The count of the cookie store unavailable: :hash, error: key not found: :hash errors in /var/log/foreman/production.log on the Satellite server kept growing drastically:

  # cat /var/log/foreman/production.log| grep "cookie store unavailable" | wc -l
  18003
  

  and after one hour:

  # cat /var/log/foreman/production.log| grep "cookie store unavailable" | wc -l
  32071