bhyve bhyve debian with docker unstable

For some reason debian is realy unstable CPU lock up

Less:
[ 1709.431166] Hardware name: FreeBSD BHYVE/BHYVE, BIOS 13.0 11/10/2020
[ 1709.431166] Call Trace:
[ 1709.431166]  <IRQ>
[ 1709.435041]  dump_stack+0x6b/0x83
[ 1709.435041]  panic+0x101/0x2d7
[ 1709.435041]  watchdog_timer_fn.cold+0xc/0x16
[ 1709.435041]  ? lockup_detector_update_enable+0x50/0x50
[ 1709.435041]  __hrtimer_run_queues+0x12a/0x280
[ 1709.439176]  hrtimer_interrupt+0x110/0x2c0
[ 1709.439176]  __sysvec_apic_timer_interrupt+0x5f/0xe0
[ 1709.439176]  asm_call_irq_on_stack+0x12/0x20
[ 1709.439176]  </IRQ>
[ 1709.439176]  sysvec_apic_timer_interrupt+0x72/0x80
[ 1709.439176]  asm_sysvec_apic_timer_interrupt+0x12/0x20
[ 1709.439176] RIP: 0010:smp_call_function_many_cond+0x28d/0x2e0
[ 1709.439176] Code: e8 b8 72 3a 00 3b 05 56 f2 72 01 89 c7 0f 83 07 fe ff ff 48 63 c7 49 8b 16 48 03 14 c5 00 b9 f7 a8 8b 42 08 a8 01 74 09 f3 90 <8b> 42 08 a8 01 75 f7 eb c9 48 c7 c2 e0 e5 66 a9 4c 89 fe 44 89 f7
[ 1709.447177] RSP: 0018:ffffa7a44049bc48 EFLAGS: 00000202
[ 1709.450978] RAX: 0000000000000011 RBX: 000000000003a560 RCX: 0000000000000004
[ 1709.450978] RDX: ffff9a7aefd3a560 RSI: 0000000000000000 RDI: 0000000000000004
[ 1709.450978] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000004
[ 1709.455193] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[ 1709.455193] R13: 0000000000000006 R14: ffff9a7aefdf4b80 R15: 0000000000000008
[ 1709.455193]  ? smp_call_function_many_cond+0x268/0x2e0
[ 1709.455193]  ? load_new_mm_cr3+0xd0/0xd0
[ 1709.455193]  ? load_new_mm_cr3+0xd0/0xd0
[ 1709.455193]  on_each_cpu+0x2b/0x60
[ 1709.455193]  cpa_flush+0x59/0x1b0
[ 1709.455193]  change_page_attr_set_clr+0x158/0x1c0
[ 1709.463168]  set_memory_ro+0x26/0x30
[ 1709.463168]  bpf_int_jit_compile+0x468/0x4a0
[ 1709.463168]  bpf_prog_select_runtime+0xe0/0x190
[ 1709.463168]  bpf_migrate_filter+0x124/0x170
[ 1709.466962]  bpf_prog_create_from_user+0x180/0x200
[ 1709.466962]  do_seccomp+0x2bc/0xa40
[ 1709.466962]  ? __do_sys_prctl+0x3a/0x670
[ 1709.466962]  do_syscall_64+0x33/0x80
[ 1709.466962]  entry_SYSCALL_64_after_hwframe+0x61/0xc6
[ 1709.471191] RIP: 0033:0x4c2bca
[ 1709.471191] Code: e8 3b 87 fa ff 48 8b 7c 24 10 48 8b 74 24 18 48 8b 54 24 20 4c 8b 54 24 28 4c 8b 44 24 30 4c 8b 4c 24 38 48 8b 44 24 08 0f 05 <48> 3d 01 f0 ff ff 76 20 48 c7 44 24 40 ff ff ff ff 48 c7 44 24 48
[ 1709.474950] RSP: 002b:000000c0001caf90 EFLAGS: 00000206 ORIG_RAX: 000000000000009d
[ 1709.474950] RAX: ffffffffffffffda RBX: 000000c000028000 RCX: 00000000004c2bca
[ 1709.479171] RDX: 000000c0001cb038 RSI: 0000000000000002 RDI: 0000000000000016
[ 1709.479171] RBP: 000000c0001cafe8 R08: 0000000000000000 R09: 0000000000000000
[ 1709.479171] R10: 0000000000000000 R11: 0000000000000206 R12: 00000000000003bb
[ 1709.482942] R13: 000000c000169dd0 R14: 0000000000000001 R15: 0000000000000001
[ 1709.482942] Kernel Offset: 0x26e00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 1709.482942] Rebooting in 60 seconds..
 
So what it boils down to sysctl
Code:
kernel.nmi_watchdog = 0
kernel.softlockup_panic=0

EDIT: on the guest
 
That didn't solve anything unfortunately.

Code:
[  810.452326] rcu: INFO: rcu_sched self-detected stall on CPU
[  810.453509] rcu:     6-....: (5249 ticks this GP) idle=d06/1/0x4000000000000000 softirq=33004/33004 fq
s=2299
[  810.455377]  (t=5250 jiffies g=127981 q=915)
[  810.456339] NMI backtrace for cpu 6
[  810.457173] CPU: 6 PID: 2952 Comm: containerd-shim Not tainted 5.10.0-21-amd64 #1 Debian 5.10.162-1
[  810.458985] Hardware name: FreeBSD BHYVE/BHYVE, BIOS 13.0 11/10/2020
[  810.460317] Call Trace:
[  810.460972]  <IRQ>
[  810.461508]  dump_stack+0x6b/0x83
[  810.462259]  nmi_cpu_backtrace.cold+0x32/0x69
[  810.463196]  ? lapic_can_unplug_cpu+0x80/0x80
[  810.464134]  nmi_trigger_cpumask_backtrace+0xdf/0xf0
[  810.465172]  rcu_dump_cpu_stacks+0xa2/0xd4
[  810.466067]  rcu_sched_clock_irq.cold+0x1ff/0x3d6
[  810.467069]  update_process_times+0x8c/0xc0
[  810.467991]  tick_sched_handle+0x22/0x60
[  810.468898]  tick_sched_timer+0x80/0xc0
[  810.469758]  ? tick_do_update_jiffies64.part.0+0xc0/0xc0
[  810.470901]  __hrtimer_run_queues+0x12a/0x280
[  810.471949]  hrtimer_interrupt+0x110/0x2c0
[  810.472921]  __sysvec_apic_timer_interrupt+0x5f/0xe0
[  810.474042]  asm_call_irq_on_stack+0x12/0x20
[  810.475035]  </IRQ>
[  810.475616]  sysvec_apic_timer_interrupt+0x72/0x80
[  810.476744]  asm_sysvec_apic_timer_interrupt+0x12/0x20
[  810.477895] RIP: 0010:smp_call_function_many_cond+0x28d/0x2e0
[  810.479157] Code: e8 b8 72 3a 00 3b 05 56 f2 72 01 89 c7 0f 83 07 fe ff ff 48 63 c7 49 8b 16 48 03 14
c5 00 b9 d7 94 8b 42 08 a8 01 74 09 f3 90 <8b> 42 08 a8 01 75 f7 eb c9 48 c7 c2 e0 e5 46 95 4c 89 fe 44 8
9 f7
[  810.482862] RSP: 0018:ffffb479c15dbcd0 EFLAGS: 00000202
[  810.484034] RAX: 0000000000000011 RBX: 000000000003a540 RCX: 0000000000000003
[  810.485535] RDX: ffff8fa42fcfa540 RSI: 0000000000000000 RDI: 0000000000000003
[  810.487038] RBP: ffffffff93c7a230 R08: 0000000000000000 R09: 0000000000000003
[  810.488654] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8fa42fdb3580
[  810.490190] R13: 0000000000000005 R14: ffff8fa42fdb4b80 R15: 0000000000000008
[  810.491696]  ? x86_configure_nx+0x50/0x50
[  810.492626]  ? flush_tlb_func_common.constprop.0+0x1e0/0x1e0
[  810.493873]  ? flush_tlb_func_common.constprop.0+0x1e0/0x1e0
[  810.495119]  ? x86_configure_nx+0x50/0x50
[  810.496058]  on_each_cpu_cond_mask+0x2f/0x90
[  810.497050]  flush_tlb_mm_range+0xc1/0xf0
[  810.497975]  tlb_finish_mmu+0x117/0x1a0
[  810.498872]  madvise_free_single_vma+0x13a/0x1a0
[  810.499914]  ? pipe_write+0x3fb/0x680
[  810.500777]  ? find_vma_prev+0x1b/0xb0
[  810.501650]  do_madvise.part.0+0x428/0xb60
[  810.502588]  ? switch_fpu_return+0x44/0xc0
[  810.503538]  __x64_sys_madvise+0x58/0x70
[  810.504482]  do_syscall_64+0x33/0x80
[  810.505341]  entry_SYSCALL_64_after_hwframe+0x61/0xc6
[  810.506499] RIP: 0033:0x46ef57
[  810.507255] Code: 8b 24 24 48 8b 6c 24 10 48 83 c4 18 c3 cc cc cc cc cc cc 48 8b 7c 24 08 48 8b 74 24
10 8b 54 24 18 48 c7 c0 1c 00 00 00 0f 05 <89> 44 24 20 c3 cc cc cc cc 48 8b 7c 24 08 8b 74 24 10 8b 54 2
4 14
[  810.510974] RSP: 002b:00007f1ca57f9cb0 EFLAGS: 00000246 ORIG_RAX: 000000000000001c
[  810.512580] RAX: ffffffffffffffda RBX: 000000c000400000 RCX: 000000000046ef57
[  810.514116] RDX: 0000000000000008 RSI: 0000000000002000 RDI: 000000c0002f4000
[  810.515612] RBP: 00007f1ca57f9cf0 R08: 000000c0002f6000 R09: 01e2ffff80ffe7e0
[  810.517109] R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000025
[  810.518627] R13: 0000000000001018 R14: 000080c000000000 R15: 0000000000000004
[  816.811812] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [khugepaged:67]
[  816.813255] Modules linked in: binfmt_misc nfsv3 nfs_acl nfs lockd grace nfs_ssc fscache xt_nat xt_tcp
udp veth xt_conntrack nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack_netlink nf_conntrack nf_defrag_ipv6
 nf_defrag_ipv4 xfrm_user xfrm_algo nft_counter xt_addrtype nft_compat nf_tables libcrc32c nfnetlink br_n
etfilter bridge stp llc intel_rapl_msr overlay intel_rapl_common nfit libnvdimm rapl evdev serio_raw pcsp
kr button drm fuse configfs sunrpc ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 crc32c_generic dm_c
rypt dm_mod crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel ghash_clmulni_intel virtio_net ne
t_failover failover aesni_intel virtio_blk libaes crypto_simd psmouse virtio_pci cryptd virtio_ring glue_
helper virtio
[  816.825015] CPU: 1 PID: 67 Comm: khugepaged Not tainted 5.10.0-21-amd64 #1 Debian 5.10.162-1
[  816.826672] Hardware name: FreeBSD BHYVE/BHYVE, BIOS 13.0 11/10/2020
[  816.828036] RIP: 0010:smp_call_function_many_cond+0x28d/0x2e0
[  816.829221] Code: e8 b8 72 3a 00 3b 05 56 f2 72 01 89 c7 0f 83 07 fe ff ff 48 63 c7 49 8b 16 48 03 14
c5 00 b9 d7 94 8b 42 08 a8 01 74 09 f3 90 <8b> 42 08 a8 01 75 f7 eb c9 48 c7 c2 e0 e5 46 95 4c 89 fe 44 8
9 f7
[  816.832724] RSP: 0018:ffffb479c0277d38 EFLAGS: 00000202
[  816.833815] RAX: 0000000000000011 RBX: 000000000003a3a0 RCX: 0000000000000003
[  816.835229] RDX: ffff8fa42fcfa3a0 RSI: 0000000000000000 RDI: 0000000000000003
[  816.836687] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000003
[  816.838215] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[  816.839638] R13: 0000000000000007 R14: ffff8fa42fc74b80 R15: 0000000000000008
[  816.841057] FS:  0000000000000000(0000) GS:ffff8fa42fc40000(0000) knlGS:0000000000000000
[  816.842654] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  816.843829] CR2: 00007f8dd2750000 CR3: 0000000102fac005 CR4: 00000000003706e0
[  816.845243] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  816.846708] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  816.848124] Call Trace:
[  816.848739]  ? mm_take_all_locks+0x1c0/0x1c0
[  816.849668]  smp_call_function+0x21/0x30
[  816.850539]  khugepaged+0x1613/0x2560
[  816.851365]  ? collapse_pte_mapped_thp+0x4a0/0x4a0
[  816.852401]  kthread+0x11b/0x140
[  816.853160]  ? __kthread_bind_mask+0x60/0x60
[  816.854098]  ret_from_fork+0x22/0x30

Message from syslogd@moby at Feb  5 15:42:39 ...
 kernel:[  816.811812] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [khugepaged:67]
[  837.246824] watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [containerd-shim:2952]
[  837.248484] Modules linked in: binfmt_misc nfsv3 nfs_acl nfs lockd grace nfs_ssc fscache xt_nat xt_tcp
udp veth xt_conntrack nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack_netlink nf_conntrack nf_defrag_ipv6
 nf_defrag_ipv4 xfrm_user xfrm_algo nft_counter xt_addrtype nft_compat nf_tables libcrc32c nfnetlink br_n
etfilter bridge stp llc intel_rapl_msr overlay intel_rapl_common nfit libnvdimm rapl evdev serio_raw pcsp
kr button drm fuse configfs sunrpc ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 crc32c_generic dm_c
rypt dm_mod crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel ghash_clmulni_intel virtio_net ne
t_failover failover aesni_intel virtio_blk libaes crypto_simd psmouse virtio_pci cryptd virtio_ring glue_
helper virtio
[  837.260903] CPU: 6 PID: 2952 Comm: containerd-shim Tainted: G             L    5.10.0-21-amd64 #1 Debi
an 5.10.162-1
[  837.263079] Hardware name: FreeBSD BHYVE/BHYVE, BIOS 13.0 11/10/2020
[  837.264405] RIP: 0010:smp_call_function_many_cond+0x28d/0x2e0
[  837.265659] Code: e8 b8 72 3a 00 3b 05 56 f2 72 01 89 c7 0f 83 07 fe ff ff 48 63 c7 49 8b 16 48 03 14
c5 00 b9 d7 94 8b 42 08 a8 01 74 09 f3 90 <8b> 42 08 a8 01 75 f7 eb c9 48 c7 c2 e0 e5 46 95 4c 89 fe 44 8
9 f7
[  837.269238] RSP: 0018:ffffb479c15dbcd0 EFLAGS: 00000202
[  837.270384] RAX: 0000000000000011 RBX: 000000000003a540 RCX: 0000000000000003
[  837.271867] RDX: ffff8fa42fcfa540 RSI: 0000000000000000 RDI: 0000000000000003
[  837.273457] RBP: ffffffff93c7a230 R08: 0000000000000000 R09: 0000000000000003
[  837.274940] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8fa42fdb3580
[  837.276400] R13: 0000000000000005 R14: ffff8fa42fdb4b80 R15: 0000000000000008
[  837.277905] FS:  00007f1ca57fa700(0000) GS:ffff8fa42fd80000(0000) knlGS:0000000000000000
[  837.279616] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  837.280826] CR2: 00007fcf8c682530 CR3: 0000000102572006 CR4: 00000000003706e0
[  837.282243] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  837.283654] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  837.285079] Call Trace:
[  837.285684]  ? flush_tlb_func_common.constprop.0+0x1e0/0x1e0
[  837.286868]  ? flush_tlb_func_common.constprop.0+0x1e0/0x1e0
[  837.288114]  ? x86_configure_nx+0x50/0x50
[  837.289117]  on_each_cpu_cond_mask+0x2f/0x90
[  837.290081]  flush_tlb_mm_range+0xc1/0xf0
[  837.291015]  tlb_finish_mmu+0x117/0x1a0
[  837.291892]  madvise_free_single_vma+0x13a/0x1a0
[  837.292912]  ? pipe_write+0x3fb/0x680
[  837.293784]  ? find_vma_prev+0x1b/0xb0
[  837.294664]  do_madvise.part.0+0x428/0xb60
[  837.295578]  ? switch_fpu_return+0x44/0xc0
[  837.296485]  __x64_sys_madvise+0x58/0x70
[  837.297406]  do_syscall_64+0x33/0x80
[  837.298236]  entry_SYSCALL_64_after_hwframe+0x61/0xc6
[  837.299353] RIP: 0033:0x46ef57
[  837.300084] Code: 8b 24 24 48 8b 6c 24 10 48 83 c4 18 c3 cc cc cc cc cc cc 48 8b 7c 24 08 48 8b 74 24
10 8b 54 24 18 48 c7 c0 1c 00 00 00 0f 05 <89> 44 24 20 c3 cc cc cc cc 48 8b 7c 24 08 8b 74 24 10 8b 54 2
4 14
[  837.303702] RSP: 002b:00007f1ca57f9cb0 EFLAGS: 00000246 ORIG_RAX: 000000000000001c
[  837.305306] RAX: ffffffffffffffda RBX: 000000c000400000 RCX: 000000000046ef57
[  837.306795] RDX: 0000000000000008 RSI: 0000000000002000 RDI: 000000c0002f4000
[  837.308345] RBP: 00007f1ca57f9cf0 R08: 000000c0002f6000 R09: 01e2ffff80ffe7e0
[  837.309893] R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000025
[  837.311348] R13: 0000000000001018 R14: 000080c000000000 R15: 0000000000000004

Message from syslogd@moby at Feb  5 15:42:59 ...
 kernel:[  837.246824] watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [containerd-shim:2952]
[  844.826848] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [khugepaged:67]
[  844.828411] Modules linked in: binfmt_misc nfsv3 nfs_acl nfs lockd grace nfs_ssc fscache xt_nat xt_tcp
udp veth xt_conntrack nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack_netlink nf_conntrack nf_defrag_ipv6
 nf_defrag_ipv4 xfrm_user xfrm_algo nft_counter xt_addrtype nft_compat nf_tables libcrc32c nfnetlink br_n
etfilter bridge stp llc intel_rapl_msr overlay intel_rapl_common nfit libnvdimm rapl evdev serio_raw pcsp
kr button drm fuse configfs sunrpc ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 crc32c_generic dm_c
rypt dm_mod crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel ghash_clmulni_intel virtio_net ne
t_failover failover aesni_intel virtio_blk libaes crypto_simd psmouse virtio_pci cryptd virtio_ring glue_
helper virtio
[  844.841277] CPU: 1 PID: 67 Comm: khugepaged Tainted: G             L    5.10.0-21-amd64 #1 Debian 5.10
.162-1
[  844.843251] Hardware name: FreeBSD BHYVE/BHYVE, BIOS 13.0 11/10/2020
[  844.844617] RIP: 0010:smp_call_function_many_cond+0x28d/0x2e0
[  844.845858] Code: e8 b8 72 3a 00 3b 05 56 f2 72 01 89 c7 0f 83 07 fe ff ff 48 63 c7 49 8b 16 48 03 14
c5 00 b9 d7 94 8b 42 08 a8 01 74 09 f3 90 <8b> 42 08 a8 01 75 f7 eb c9 48 c7 c2 e0 e5 46 95 4c 89 fe 44 8
9 f7
[  844.849419] RSP: 0018:ffffb479c0277d38 EFLAGS: 00000202
[  844.850539] RAX: 0000000000000011 RBX: 000000000003a3a0 RCX: 0000000000000003
[  844.851996] RDX: ffff8fa42fcfa3a0 RSI: 0000000000000000 RDI: 0000000000000003
[  844.853431] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000003
[  844.854903] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[  844.856346] R13: 0000000000000007 R14: ffff8fa42fc74b80 R15: 0000000000000008
[  844.857902] FS:  0000000000000000(0000) GS:ffff8fa42fc40000(0000) knlGS:0000000000000000
[  844.859548] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  844.860745] CR2: 00007f8dd2750000 CR3: 0000000102fac005 CR4: 00000000003706e0
[  844.862174] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  844.863649] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  844.865087] Call Trace:
[  844.865706]  ? mm_take_all_locks+0x1c0/0x1c0
[  844.866640]  smp_call_function+0x21/0x30
[  844.867514]  khugepaged+0x1613/0x2560
[  844.868341]  ? collapse_pte_mapped_thp+0x4a0/0x4a0
[  844.869405]  kthread+0x11b/0x140
[  844.870199]  ? __kthread_bind_mask+0x60/0x60
[  844.871237]  ret_from_fork+0x22/0x30

Message from syslogd@moby at Feb  5 15:43:07 ...
 kernel:[  844.826848] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [khugepaged:67]
[  865.261583] watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [containerd-shim:2952]
[  865.263192] Modules linked in: binfmt_misc nfsv3 nfs_acl nfs lockd grace nfs_ssc fscache xt_nat xt_tcp
udp veth xt_conntrack nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack_netlink nf_conntrack nf_defrag_ipv6
 nf_defrag_ipv4 xfrm_user xfrm_algo nft_counter xt_addrtype nft_compat nf_tables libcrc32c nfnetlink br_n
etfilter bridge stp llc intel_rapl_msr overlay intel_rapl_common nfit libnvdimm rapl evdev serio_raw pcsp
kr button drm fuse configfs sunrpc ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 crc32c_generic dm_c
rypt dm_mod crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel ghash_clmulni_intel virtio_net ne
t_failover failover aesni_intel virtio_blk libaes crypto_simd psmouse virtio_pci cryptd virtio_ring glue_
helper virtio
[  865.275494] CPU: 6 PID: 2952 Comm: containerd-shim Tainted: G             L    5.10.0-21-amd64 #1 Debi
an 5.10.162-1
[  865.277704] Hardware name: FreeBSD BHYVE/BHYVE, BIOS 13.0 11/10/2020
[  865.279074] RIP: 0010:smp_call_function_many_cond+0x28d/0x2e0
[  865.280344] Code: e8 b8 72 3a 00 3b 05 56 f2 72 01 89 c7 0f 83 07 fe ff ff 48 63 c7 49 8b 16 48 03 14
c5 00 b9 d7 94 8b 42 08 a8 01 74 09 f3 90 <8b> 42 08 a8 01 75 f7 eb c9 48 c7 c2 e0 e5 46 95 4c 89 fe 44 8
9 f7
[  865.283935] RSP: 0018:ffffb479c15dbcd0 EFLAGS: 00000202
[  865.285068] RAX: 0000000000000011 RBX: 000000000003a540 RCX: 0000000000000003
[  865.286558] RDX: ffff8fa42fcfa540 RSI: 0000000000000000 RDI: 0000000000000003
[  865.288019] RBP: ffffffff93c7a230 R08: 0000000000000000 R09: 0000000000000003
[  865.289475] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8fa42fdb3580
[  865.290933] R13: 0000000000000005 R14: ffff8fa42fdb4b80 R15: 0000000000000008
[  865.292387] FS:  00007f1ca57fa700(0000) GS:ffff8fa42fd80000(0000) knlGS:0000000000000000
[  865.294091] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  865.295333] CR2: 00007fcf8c682530 CR3: 0000000102572006 CR4: 00000000003706e0
[  865.296821] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  865.298273] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  865.299784] Call Trace:
[  865.300433]  ? flush_tlb_func_common.constprop.0+0x1e0/0x1e0
[  865.301688]  ? flush_tlb_func_common.constprop.0+0x1e0/0x1e0
[  865.302912]  ? x86_configure_nx+0x50/0x50
[  865.303837]  on_each_cpu_cond_mask+0x2f/0x90
[  865.304806]  flush_tlb_mm_range+0xc1/0xf0
[  865.305748]  tlb_finish_mmu+0x117/0x1a0
[  865.306687]  madvise_free_single_vma+0x13a/0x1a0
[  865.307732]  ? pipe_write+0x3fb/0x680
[  865.308566]  ? find_vma_prev+0x1b/0xb0
[  865.309415]  do_madvise.part.0+0x428/0xb60
[  865.310377]  ? switch_fpu_return+0x44/0xc0
[  865.311335]  __x64_sys_madvise+0x58/0x70
[  865.312242]  do_syscall_64+0x33/0x80
[  865.313069]  entry_SYSCALL_64_after_hwframe+0x61/0xc6
[  865.314157] RIP: 0033:0x46ef57
[  865.314883] Code: 8b 24 24 48 8b 6c 24 10 48 83 c4 18 c3 cc cc cc cc cc cc 48 8b 7c 24 08 48 8b 74 24
10 8b 54 24 18 48 c7 c0 1c 00 00 00 0f 05 <89> 44 24 20 c3 cc cc cc cc 48 8b 7c 24 08 8b 74 24 10 8b 54 2
4 14
[  865.318763] RSP: 002b:00007f1ca57f9cb0 EFLAGS: 00000246 ORIG_RAX: 000000000000001c
[  865.320291] RAX: ffffffffffffffda RBX: 000000c000400000 RCX: 000000000046ef57
[  865.321725] RDX: 0000000000000008 RSI: 0000000000002000 RDI: 000000c0002f4000
[  865.323151] RBP: 00007f1ca57f9cf0 R08: 000000c0002f6000 R09: 01e2ffff80ffe7e0
[  865.324875] R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000025
[  865.326430] R13: 0000000000001018 R14: 000080c000000000 R15: 0000000000000004

Message from syslogd@moby at Feb  5 15:43:27 ...
 kernel:[  865.261583] watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [containerd-shim:2952]
[  872.841516] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [khugepaged:67]
[  872.843012] Modules linked in: binfmt_misc nfsv3 nfs_acl nfs lockd grace nfs_ssc fscache xt_nat xt_tcp
udp veth xt_conntrack nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack_netlink nf_conntrack nf_defrag_ipv6
 nf_defrag_ipv4 xfrm_user xfrm_algo nft_counter xt_addrtype nft_compat nf_tables libcrc32c nfnetlink br_n
etfilter bridge stp llc intel_rapl_msr overlay intel_rapl_common nfit libnvdimm rapl evdev serio_raw pcsp
kr button drm fuse configfs sunrpc ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 crc32c_generic dm_c
rypt dm_mod crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel ghash_clmulni_intel virtio_net ne
t_failover failover aesni_intel virtio_blk libaes crypto_simd psmouse virtio_pci cryptd virtio_ring glue_
helper virtio
[  872.855972] CPU: 1 PID: 67 Comm: khugepaged Tainted: G             L    5.10.0-21-amd64 #1 Debian 5.10
.162-1
[  872.858103] Hardware name: FreeBSD BHYVE/BHYVE, BIOS 13.0 11/10/2020
[  872.859477] RIP: 0010:smp_call_function_many_cond+0x28d/0x2e0
[  872.860684] Code: e8 b8 72 3a 00 3b 05 56 f2 72 01 89 c7 0f 83 07 fe ff ff 48 63 c7 49 8b 16 48 03 14
c5 00 b9 d7 94 8b 42 08 a8 01 74 09 f3 90 <8b> 42 08 a8 01 75 f7 eb c9 48 c7 c2 e0 e5 46 95 4c 89 fe 44 8
9 f7
[  872.864238] RSP: 0018:ffffb479c0277d38 EFLAGS: 00000202
[  872.865394] RAX: 0000000000000011 RBX: 000000000003a3a0 RCX: 0000000000000003
[  872.866835] RDX: ffff8fa42fcfa3a0 RSI: 0000000000000000 RDI: 0000000000000003
[  872.868302] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000003
[  872.869765] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[  872.871212] R13: 0000000000000007 R14: ffff8fa42fc74b80 R15: 0000000000000008
[  872.872767] FS:  0000000000000000(0000) GS:ffff8fa42fc40000(0000) knlGS:0000000000000000
[  872.874593] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  872.875880] CR2: 00007f8dd2750000 CR3: 0000000102fac005 CR4: 00000000003706e0
[  872.877436] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  872.879003] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  872.880776] Call Trace:
[  872.881425]  ? mm_take_all_locks+0x1c0/0x1c0
[  872.882365]  smp_call_function+0x21/0x30
[  872.883242]  khugepaged+0x1613/0x2560
[  872.884130]  ? collapse_pte_mapped_thp+0x4a0/0x4a0
[  872.885255]  kthread+0x11b/0x140
[  872.886029]  ? __kthread_bind_mask+0x60/0x60
[  872.887017]  ret_from_fork+0x22/0x30

Message from syslogd@moby at Feb  5 15:43:35 ...
 kernel:[  872.841516] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [khugepaged:67]
[  873.497858] rcu: INFO: rcu_sched self-detected stall on CPU
[  873.499125] rcu:     6-....: (20954 ticks this GP) idle=d06/1/0x4000000000000000 softirq=33004/33004 f
qs=9026
[  873.501117]  (t=21003 jiffies g=127981 q=6356)
[  873.502218] NMI backtrace for cpu 6
[  873.503101] CPU: 6 PID: 2952 Comm: containerd-shim Tainted: G             L    5.10.0-21-amd64 #1 Debi
an 5.10.162-1
[  873.505283] Hardware name: FreeBSD BHYVE/BHYVE, BIOS 13.0 11/10/2020
[  873.506583] Call Trace:
[  873.507203]  <IRQ>
[  873.507783]  dump_stack+0x6b/0x83
[  873.508635]  nmi_cpu_backtrace.cold+0x32/0x69
[  873.509604]  ? lapic_can_unplug_cpu+0x80/0x80
[  873.510588]  nmi_trigger_cpumask_backtrace+0xdf/0xf0
[  873.511713]  rcu_dump_cpu_stacks+0xa2/0xd4
[  873.512673]  rcu_sched_clock_irq.cold+0x1ff/0x3d6
[  873.513717]  update_process_times+0x8c/0xc0
[  873.514743]  tick_sched_handle+0x22/0x60
[  873.515830]  tick_sched_timer+0x80/0xc0
[  873.516932]  ? tick_do_update_jiffies64.part.0+0xc0/0xc0
[  873.518284]  __hrtimer_run_queues+0x12a/0x280
[  873.519403]  hrtimer_interrupt+0x110/0x2c0
[  873.520481]  __sysvec_apic_timer_interrupt+0x5f/0xe0
[  873.521665]  asm_call_irq_on_stack+0x12/0x20
[  873.522700]  </IRQ>
[  873.523278]  sysvec_apic_timer_interrupt+0x72/0x80
[  873.524361]  asm_sysvec_apic_timer_interrupt+0x12/0x20
[  873.525561] RIP: 0010:smp_call_function_many_cond+0x28d/0x2e0
[  873.526833] Code: e8 b8 72 3a 00 3b 05 56 f2 72 01 89 c7 0f 83 07 fe ff ff 48 63 c7 49 8b 16 48 03 14
c5 00 b9 d7 94 8b 42 08 a8 01 74 09 f3 90 <8b> 42 08 a8 01 75 f7 eb c9 48 c7 c2 e0 e5 46 95 4c 89 fe 44 8
9 f7
[  873.530697] RSP: 0018:ffffb479c15dbcd0 EFLAGS: 00000202
[  873.531928] RAX: 0000000000000011 RBX: 000000000003a540 RCX: 0000000000000003
[  873.533599] RDX: ffff8fa42fcfa540 RSI: 0000000000000000 RDI: 0000000000000003
[  873.535270] RBP: ffffffff93c7a230 R08: 0000000000000000 R09: 0000000000000003
[  873.536811] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8fa42fdb3580
[  873.538389] R13: 0000000000000005 R14: ffff8fa42fdb4b80 R15: 0000000000000008
[  873.540114]  ? x86_configure_nx+0x50/0x50
[  873.541122]  ? flush_tlb_func_common.constprop.0+0x1e0/0x1e0
[  873.542468]  ? flush_tlb_func_common.constprop.0+0x1e0/0x1e0
[  873.543737]  ? x86_configure_nx+0x50/0x50
[  873.544688]  on_each_cpu_cond_mask+0x2f/0x90
[  873.545744]  flush_tlb_mm_range+0xc1/0xf0
[  873.546731]  tlb_finish_mmu+0x117/0x1a0
[  873.547753]  madvise_free_single_vma+0x13a/0x1a0
[  873.548987]  ? pipe_write+0x3fb/0x680
[  873.550076]  ? find_vma_prev+0x1b/0xb0
[  873.551028]  do_madvise.part.0+0x428/0xb60
[  873.552009]  ? switch_fpu_return+0x44/0xc0
[  873.552991]  __x64_sys_madvise+0x58/0x70
[  873.553923]  do_syscall_64+0x33/0x80
[  873.554780]  entry_SYSCALL_64_after_hwframe+0x61/0xc6
[  873.555925] RIP: 0033:0x46ef57
[  873.556690] Code: 8b 24 24 48 8b 6c 24 10 48 83 c4 18 c3 cc cc cc cc cc cc 48 8b 7c 24 08 48 8b 74 24
10 8b 54 24 18 48 c7 c0 1c 00 00 00 0f 05 <89> 44 24 20 c3 cc cc cc cc 48 8b 7c 24 08 8b 74 24 10 8b 54 2
4 14
[  873.560539] RSP: 002b:00007f1ca57f9cb0 EFLAGS: 00000246 ORIG_RAX: 000000000000001c
[  873.562203] RAX: ffffffffffffffda RBX: 000000c000400000 RCX: 000000000046ef57
[  873.563777] RDX: 0000000000000008 RSI: 0000000000002000 RDI: 000000c0002f4000
[  873.565423] RBP: 00007f1ca57f9cf0 R08: 000000c0002f6000 R09: 01e2ffff80ffe7e0
[  873.567019] R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000025
[  873.568528] R13: 0000000000001018 R14: 000080c000000000 R15: 0000000000000004
 
Just reinstalled os with lower version of debian instead of bullseye I used buster done no more crashes. Not used to this from debian, but seems that the os that's called stable isn't just there yet !?
 
Just reinstalled os with lower version of debian instead of bullseye I used buster done no more crashes. Not used to this from debian, but seems that the os that's called stable isn't just there yet !?
I'm experiencing the same problem both with a Debian 11 VM and an Alpine Linux VM - they're both running "mailu" on docker. All the other bhyve VMs in the server are stable, even the docker ones, even with the same combination of Debian (and Alpine) and docker versions. I'm trying to nail out what could be causing this issue, but I haven't many clues...
 
Hi, found this thread by searching for bhyve and soft lockup and it is finally time for registering over here.
Already found the issue in this thread on ttuenas forums with pretty similar issues: https://www.truenas.com/community/threads/bhyve-with-ubuntu-19-04-keeps-locking-up.79191/page-3

Apparently there is some change in Linux kernel >5.4, as some users state that downgrading fixes the issue.

Tried Ubuntu 22.10 with docker on bhyve in truenas and cannot pin it down right now.

Overprovisioning shouldn't be a problem, as the machine has a c3758 with 8 cores, no hyper threading and 128gb ram, while the VM uses 8 cores, 16gb ram.

How may I help with some logs?

Running docker with librenms and paperless-ngx 24/7 only, other container onlz temporary for testing.

Issue started before using librenms, so probably it boils down to paperless-ngx.
 
So did it resolve the issue?

edit: Tested install of latest debian yesterday seems like it is still crashing. Can I apply this patch to 13.2 ? I'm willing to setup a test environment and see if it is the same? Still got some intel i7 i5 and i3. I'm willing to set it up and test. First confirm if the issue happens to be there as well and then see if this patch resolves it?

If it is at all useful?
 
Seems like my post above got mangled somehow. Anyways I've installed docker-ce rather then the docker.io included in the pkg tree and it still happens but docker.ce has considerable less issues then docker.io

Although my system has 20 cores, having lowering the vm from 8 cores to 6 cores made it not crash up until now. Somehow lowering the cores for the vm made it not crash, using docker.ce makes it crash less frequent. Combining those 2 things made it run for a week without issues.

EDIT: tested this with FreeBSD13.2p3
 
Back
Top