Dead lock occurrs in kmem_cache_destroy_memcg_children
Issue
- A process was blocked in
D
state atmutex_lock()
afterkmem_cache_destory_memcg_children()
, and the system crashed withKernel panic - not syncing: hung_task: blocked tasks
message.
crash> bt 6982
PID: 6982 TASK: ffff8800d2f64e70 CPU: 7 COMMAND: "dockerd"
#0 [ffff8800a9fcfb40] __schedule at ffffffff8168af25
#1 [ffff8800a9fcfba8] schedule_preempt_disabled at ffffffff8168c669
#2 [ffff8800a9fcfbb8] __mutex_lock_slowpath at ffffffff8168a2c5
#3 [ffff8800a9fcfc10] mutex_lock at ffffffff8168972f
#4 [ffff8800a9fcfc28] kmem_cache_destroy_memcg_children at ffffffff811f3cfe
#5 [ffff8800a9fcfc50] kmem_cache_destroy at ffffffff811a56d9
#6 [ffff8800a9fcfc70] bioset_free at ffffffff8123741e
#7 [ffff8800a9fcfc90] __dm_destroy at ffffffffa00029de [dm_mod]
#8 [ffff8800a9fcfcd0] dm_destroy at ffffffffa0003833 [dm_mod]
#9 [ffff8800a9fcfce0] dev_remove at ffffffffa00093de [dm_mod]
#10 [ffff8800a9fcfd10] ctl_ioctl at ffffffffa0009a35 [dm_mod]
#11 [ffff8800a9fcfea8] dm_ctl_ioctl at ffffffffa0009d63 [dm_mod]
#12 [ffff8800a9fcfeb8] do_vfs_ioctl at ffffffff81211ed5
#13 [ffff8800a9fcff30] sys_ioctl at ffffffff81212151
#14 [ffff8800a9fcff80] system_call_fastpath at ffffffff816964c9
RIP: 00007f5a74036567 RSP: 00007f5a1cfd0b40 RFLAGS: 00000293
RAX: 0000000000000010 RBX: ffffffff816964c9 RCX: ffffffffffffffff
RDX: 00007f5a0c01d4a0 RSI: 00000000c138fd04 RDI: 000000000000000a
RBP: 00007f5a74354ef3 R8: 0000000000000001 R9: 0000000000010b60
R10: 00007f5a0c000078 R11: 0000000000000246 R12: 00007f5a1cfd0bd0
R13: 00007f5a0c01d3c0 R14: 00007f5a74354ef3 R15: 00007f5a7435503a
ORIG_RAX: 0000000000000010 CS: 0033 SS: 002b
- A process was blocked in
D
state atmutex_lock()
afterkmem_cache_destroy_memcg_children()
. Thekmem_cache_destroy_memcg_childre()
was called once again fromkmem_cache_destroy()
.
crash> bt 0xffff88078aed2f10
PID: 45393 TASK: ffff88078aed2f10 CPU: 1 COMMAND: "kworker/u128:4"
#0 [ffff8800cfb8bba0] __schedule at ffffffff8168af25
#1 [ffff8800cfb8bc08] schedule_preempt_disabled at ffffffff8168c669
#2 [ffff8800cfb8bc18] __mutex_lock_slowpath at ffffffff8168a2c5
#3 [ffff8800cfb8bc70] mutex_lock at ffffffff8168972f
#4 [ffff8800cfb8bc88] kmem_cache_destroy_memcg_children at ffffffff811f3cfe
#5 [ffff8800cfb8bcb0] kmem_cache_destroy at ffffffff811a56d9
#6 [ffff8800cfb8bcd0] kmem_cache_destroy_memcg_children at ffffffff811f3d49
#7 [ffff8800cfb8bcf8] kmem_cache_destroy at ffffffff811a56d9
#8 [ffff8800cfb8bd18] nf_conntrack_cleanup_net_list at ffffffffa034a5db [nf_conntrack]
#9 [ffff8800cfb8bd60] nf_conntrack_pernet_exit at ffffffffa034b45d [nf_conntrack]
#10 [ffff8800cfb8bd88] ops_exit_list at ffffffff815667f3
#11 [ffff8800cfb8bdb8] cleanup_net at ffffffff815678d0
#12 [ffff8800cfb8be20] process_one_work at ffffffff810a7f3b
#13 [ffff8800cfb8be68] worker_thread at ffffffff810a8d76
#14 [ffff8800cfb8bec8] kthread at ffffffff810b052f
#15 [ffff8800cfb8bf50] ret_from_fork at ffffffff81696418
Environment
- Red Hat Enterprise Linux 7
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.