The system is experiencing a problem centered around OFD lock contention on a single file: /tmp/krb5cc_<UID>

Issue

• The system running RHAAP workloads is experiencing an elevated CPU %sys usage.
• Perf data shows the primary contributors to the elevated CPU usage are uwsgi and awx-manage processes, which are consuming a significant number of CPU cycles due to repeated spinlock busy-waiting.

Samples: 14M of event 'cpu-clock:pppH', Event count (approx.
  Overhead  Command          Shared Object
+   58.26%  uwsgi            [kernel.kallsyms]             ◆
           - 40.72% native_queued_spin_lock_slowpath               ▒
              - 40.72% _raw_spin_lock                              ▒
                 - 40.72% posix_lock_inode                         ▒
                    - 40.72% fcntl_setlk                           ▒
                         do_fcntl                                  ▒
                         __x64_sys_fcntl                           ▒
                         do_syscall_64                             ▒
                         entry_SYSCALL_64_after_hwframe            ▒
                         0x7f658ca4ea3c                            ▒
           + 3.08% _raw_spin_lock                                  ▒
           + 2.99% posix_lock_inode                                ▒
           + 1.76% do_syscall_64                                   ▒
           + 1.47% posix_locks_conflict                            ▒
           + 0.83% avc_has_perm                                    ▒
             0.79% copy_user_generic_unrolled                      ▒
           + 0.71% kmem_cache_free                                 ▒
           + 0.53% kmem_cache_alloc                                ▒
           + 0.51% syscall_slow_exit_work                          ▒
+   15.27%  awx-manage       [kernel.kallsyms]             ▒
           - 10.30% native_queued_spin_lock_slowpath               ▒
                _raw_spin_lock                                     ▒
              - posix_lock_inode                                   ▒
                 - 10.30% fcntl_setlk                              ▒
                      do_fcntl                                     ▒
                      __x64_sys_fcntl                              ▒
                      do_syscall_64                                ▒
                    + entry_SYSCALL_64_after_hwframe               ▒
           + 0.82% _raw_spin_lock                                  ▒
           + 0.77% posix_lock_inode                                ▒
           + 0.51% do_syscall_64                                   ▒
        ...