Dead lock occurrs in kmem_cache_destroy_memcg_children

Solution Unverified - Updated -

Issue

  • A process was blocked in D state at mutex_lock() after kmem_cache_destory_memcg_children(), and the system crashed with Kernel panic - not syncing: hung_task: blocked tasks message.
crash> bt 6982
PID: 6982   TASK: ffff8800d2f64e70  CPU: 7   COMMAND: "dockerd"
 #0 [ffff8800a9fcfb40] __schedule at ffffffff8168af25
 #1 [ffff8800a9fcfba8] schedule_preempt_disabled at ffffffff8168c669
 #2 [ffff8800a9fcfbb8] __mutex_lock_slowpath at ffffffff8168a2c5
 #3 [ffff8800a9fcfc10] mutex_lock at ffffffff8168972f
 #4 [ffff8800a9fcfc28] kmem_cache_destroy_memcg_children at ffffffff811f3cfe
 #5 [ffff8800a9fcfc50] kmem_cache_destroy at ffffffff811a56d9
 #6 [ffff8800a9fcfc70] bioset_free at ffffffff8123741e
 #7 [ffff8800a9fcfc90] __dm_destroy at ffffffffa00029de [dm_mod]
 #8 [ffff8800a9fcfcd0] dm_destroy at ffffffffa0003833 [dm_mod]
 #9 [ffff8800a9fcfce0] dev_remove at ffffffffa00093de [dm_mod]
#10 [ffff8800a9fcfd10] ctl_ioctl at ffffffffa0009a35 [dm_mod]
#11 [ffff8800a9fcfea8] dm_ctl_ioctl at ffffffffa0009d63 [dm_mod]
#12 [ffff8800a9fcfeb8] do_vfs_ioctl at ffffffff81211ed5
#13 [ffff8800a9fcff30] sys_ioctl at ffffffff81212151
#14 [ffff8800a9fcff80] system_call_fastpath at ffffffff816964c9
    RIP: 00007f5a74036567  RSP: 00007f5a1cfd0b40  RFLAGS: 00000293
    RAX: 0000000000000010  RBX: ffffffff816964c9  RCX: ffffffffffffffff
    RDX: 00007f5a0c01d4a0  RSI: 00000000c138fd04  RDI: 000000000000000a
    RBP: 00007f5a74354ef3   R8: 0000000000000001   R9: 0000000000010b60
    R10: 00007f5a0c000078  R11: 0000000000000246  R12: 00007f5a1cfd0bd0
    R13: 00007f5a0c01d3c0  R14: 00007f5a74354ef3  R15: 00007f5a7435503a
    ORIG_RAX: 0000000000000010  CS: 0033  SS: 002b
  • A process was blocked in D state at mutex_lock() after kmem_cache_destroy_memcg_children(). The kmem_cache_destroy_memcg_childre() was called once again from kmem_cache_destroy().
crash> bt 0xffff88078aed2f10
PID: 45393  TASK: ffff88078aed2f10  CPU: 1   COMMAND: "kworker/u128:4"
 #0 [ffff8800cfb8bba0] __schedule at ffffffff8168af25
 #1 [ffff8800cfb8bc08] schedule_preempt_disabled at ffffffff8168c669
 #2 [ffff8800cfb8bc18] __mutex_lock_slowpath at ffffffff8168a2c5
 #3 [ffff8800cfb8bc70] mutex_lock at ffffffff8168972f
 #4 [ffff8800cfb8bc88] kmem_cache_destroy_memcg_children at ffffffff811f3cfe
 #5 [ffff8800cfb8bcb0] kmem_cache_destroy at ffffffff811a56d9
 #6 [ffff8800cfb8bcd0] kmem_cache_destroy_memcg_children at ffffffff811f3d49
 #7 [ffff8800cfb8bcf8] kmem_cache_destroy at ffffffff811a56d9
 #8 [ffff8800cfb8bd18] nf_conntrack_cleanup_net_list at ffffffffa034a5db [nf_conntrack]
 #9 [ffff8800cfb8bd60] nf_conntrack_pernet_exit at ffffffffa034b45d [nf_conntrack]
#10 [ffff8800cfb8bd88] ops_exit_list at ffffffff815667f3
#11 [ffff8800cfb8bdb8] cleanup_net at ffffffff815678d0
#12 [ffff8800cfb8be20] process_one_work at ffffffff810a7f3b
#13 [ffff8800cfb8be68] worker_thread at ffffffff810a8d76
#14 [ffff8800cfb8bec8] kthread at ffffffff810b052f
#15 [ffff8800cfb8bf50] ret_from_fork at ffffffff81696418

Environment

  • Red Hat Enterprise Linux 7

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content