Bugzilla – Attachment 870 Details for
Bug 6151
soft lockup when read at kubelet
Home
|
New
|
Browse
|
Search
|
[?]
|
Reports
|
Requests
|
Help
|
New Account
|
Log In
[x]
dmesg when soft lockup comes out
dmesg.log (text/plain), 1.01 MB, created by
chuanguo
on 2023-08-14 10:46:34 UTC
(
hide
)
Description:
dmesg when soft lockup comes out
Filename:
MIME Type:
Creator:
chuanguo
Created:
2023-08-14 10:46:34 UTC
Size:
1.01 MB
patch
obsolete
>[6125571.569503] x23: 0000000000000000 x22: 0000000000000001 >[6125571.570009] x23: 0000000000000020 x22: ffff59fd7fe49800 >[6125571.570517] x21: ffff000036f7b9d4 x20: ffff000036f7b9b0 >[6125571.571042] x21: ffff8b9bbb73fc00 x20: ffff59fd7fe4b8a8 >[6125571.571558] x19: 0000000000000001 x18: 0000000000000000 >[6125571.572059] x19: 0000000000000004 x18: 0000000000000000 >[6125571.572565] x17: 0000000000000001 x16: 0000000000000000 >[6125571.573078] x17: 0000000000000658 x16: 0000000000000000 >[6125571.573587] x15: 0000000000000000 x14: 0000000000000000 >[6125571.574085] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6125571.574582] x13: 0000000000000000 x12: 0000000000000000 >[6125571.575086] x13: ffff8b8d711a0000 x12: ffff8b8d7119f350 >[6125571.575582] x11: 0000000000000000 x10: 0000000000000d60 >[6125571.576211] x11: 0000000000000000 x10: ffff00001d773c80 >[6125571.576792] x9 : ffff59fd7ede5a0c x8 : ffff8b7d9d9c3040 >[6125571.577336] x9 : ffff59fd7ef3dad0 x8 : ffff8b8d7119f000 >[6125571.577897] x7 : 0018080ba9727534 x6 : 000000000000003e >[6125571.578405] x7 : 0000000000000000 x6 : 0000000000000001 >[6125571.578963] x5 : 00000000fa83b2da x4 : ffff8b7d9d9c2880 >[6125571.579478] x5 : 00000000fffffffb x4 : ffffffffffffffff >[6125571.579978] x3 : dead000000000200 x2 : ffff000036f7b960 >[6125571.580489] x3 : 000000000000002b x2 : 000033d020060000 >[6125571.580997] x1 : ffff000036f7b9d4 x0 : 0000000000000001 >[6125571.581504] x1 : ffff4a2dc835f7e0 x0 : 000000000000006b >[6125571.582031] Call trace: >[6125571.582612] Call trace: >[6125571.582894] multi_cpu_stop+0xf4/0x180 >[6125571.583157] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6125571.583582] cpu_stopper_thread+0xec/0x178 >[6125571.584065] memcg_numa_stat_show+0x1d0/0x260 >[6125571.584539] smpboot_thread_fn+0x158/0x1d8 >[6125571.584958] cgroup_seqfile_show+0x50/0xd0 >[6125571.585392] kthread+0x130/0x138 >[6125571.585876] kernfs_seq_show+0x30/0x38 >[6125571.586630] seq_read+0x148/0x478 >[6125571.587052] kernfs_fop_read+0x2c/0x1e0 >[6125571.587457] __vfs_read+0x20/0x48 >[6125571.587857] vfs_read+0x98/0x168 >[6125571.588212] ksys_read+0x6c/0xe8 >[6125571.588577] __arm64_sys_read+0x20/0x28 >[6125571.588976] el0_svc_common.constprop.0+0xa8/0x200 >[6125571.589526] el0_svc_handler+0x30/0x80 >[6125571.589916] el0_svc+0x10/0x14 >[6125611.531012] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kubelet:18850] >[6125611.531887] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6125611.538595] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6125611.544433] CPU: 0 PID: 18850 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6125611.545527] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6125611.546479] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6125611.546940] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6125611.547428] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6125611.547930] sp : ffff00001fe33c20 >[6125611.548266] x29: ffff00001fe33c20 x28: ffff8b9c01dc4000 >[6125611.548769] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6125611.549281] x25: 0000000000000003 x24: 0000000000000000 >[6125611.549805] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6125611.550319] x21: ffff8dac7dbe3400 x20: ffff59fd7fe4b8a8 >[6125611.550821] x19: 0000000000000000 x18: 0000000000000000 >[6125611.551338] x17: 0000000000000002 x16: 0000000000000000 >[6125611.551853] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6125611.552435] x13: ffff8b8d71fcc000 x12: ffff8b8d71fcb307 >[6125611.552970] x11: 0000000000000000 x10: ffff00001fe33c80 >[6125611.553489] x9 : ffff59fd7ef3dad0 x8 : 000000000000bfdd >[6125611.553989] x7 : 0000000000000004 x6 : 0000000000000005 >[6125611.554506] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6125611.555028] x3 : 000000000000000f x2 : 00003390404c0000 >[6125611.555636] x1 : ffff4a108064ba80 x0 : 000000000000004f >[6125611.556187] Call trace: >[6125611.556445] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6125611.556983] memcg_numa_stat_show+0x1d0/0x260 >[6125611.557415] cgroup_seqfile_show+0x50/0xd0 >[6125611.557871] kernfs_seq_show+0x30/0x38 >[6125611.558253] seq_read+0x148/0x478 >[6125611.558625] kernfs_fop_read+0x2c/0x1e0 >[6125611.559001] __vfs_read+0x20/0x48 >[6125611.559324] vfs_read+0x98/0x168 >[6125611.559714] ksys_read+0x6c/0xe8 >[6125611.560040] __arm64_sys_read+0x20/0x28 >[6125611.560437] el0_svc_common.constprop.0+0xa8/0x200 >[6125611.560955] el0_svc_handler+0x30/0x80 >[6125611.561335] el0_svc+0x10/0x14 >[6125655.531710] watchdog: BUG: soft lockup - CPU#44 stuck for 23s! [kubelet:18923] >[6125655.532538] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6125655.539875] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6125655.545383] CPU: 44 PID: 18923 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6125655.546344] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6125655.547289] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6125655.547739] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6125655.548274] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6125655.548771] sp : ffff0000301bbc20 >[6125655.549136] x29: ffff0000301bbc20 x28: ffff8bac11dac000 >[6125655.549616] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6125655.550109] x25: 0000000000000003 x24: 0000000000000000 >[6125655.550634] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6125655.551234] x21: ffff8dccf751e400 x20: ffff59fd7fe4b8a8 >[6125655.551831] x19: 0000000000000000 x18: 0000000000000000 >[6125655.552374] x17: 0000000000000002 x16: 0000000000000000 >[6125655.552946] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6125655.553509] x13: ffff8bcc4195e000 x12: ffff8bcc4195d31c >[6125655.554134] x11: 0000000000000000 x10: ffff0000301bbc80 >[6125655.554722] x9 : ffff59fd7ef3dad0 x8 : 0000000000018341 >[6125655.555376] x7 : 0000000000000004 x6 : 0000000000000005 >[6125655.555855] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6125655.556343] x3 : 0000000000000007 x2 : 00003180404be000 >[6125655.556824] x1 : ffff4a2dd74cf5b0 x0 : 0000000000000007 >[6125655.557297] Call trace: >[6125655.557539] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6125655.557994] memcg_numa_stat_show+0x1d0/0x260 >[6125655.558397] cgroup_seqfile_show+0x50/0xd0 >[6125655.558795] kernfs_seq_show+0x30/0x38 >[6125655.559165] seq_read+0x148/0x478 >[6125655.559476] kernfs_fop_read+0x2c/0x1e0 >[6125655.559833] __vfs_read+0x20/0x48 >[6125655.560138] vfs_read+0x98/0x168 >[6125655.560456] ksys_read+0x6c/0xe8 >[6125655.560777] __arm64_sys_read+0x20/0x28 >[6125655.561145] el0_svc_common.constprop.0+0xa8/0x200 >[6125655.561624] el0_svc_handler+0x30/0x80 >[6125655.561971] el0_svc+0x10/0x14 >[6125659.531224] watchdog: BUG: soft lockup - CPU#13 stuck for 23s! [migration/13:77] >[6125659.531998] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6125659.539759] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6125659.546301] CPU: 13 PID: 77 Comm: migration/13 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6125659.547414] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6125659.548378] pstate: 60000005 (nZCv daif -PAN -UAO) >[6125659.548898] pc : multi_cpu_stop+0xf4/0x180 >[6125659.549343] lr : multi_cpu_stop+0xb4/0x180 >[6125659.549786] sp : ffff000008f43d50 >[6125659.550140] x29: ffff000008f43d50 x28: 0000000000000000 >[6125659.550746] x27: ffff000030123938 x26: 0000000000000060 >[6125659.551313] x25: 0000000000000000 x24: 0000000000000000 >[6125659.551862] x23: 0000000000000000 x22: 0000000000000001 >[6125659.552456] x21: ffff0000301239d4 x20: ffff0000301239b0 >[6125659.553012] x19: 0000000000000001 x18: 0000000000000000 >[6125659.553515] x17: 0000000000000001 x16: 0000000000000000 >[6125659.554050] x15: 0000000000000000 x14: 0000000000000000 >[6125659.554540] x13: 0000000000000000 x12: 0000000000000000 >[6125659.555029] x11: 0000000000000063 x10: 0000000000000d60 >[6125659.555640] x9 : ffff59fd7ede5a0c x8 : ffff8b8d9e998dc0 >[6125659.556192] x7 : 0000e6377b223a34 x6 : 00000000000000af >[6125659.556749] x5 : 00000000fa83b2da x4 : ffff8b8d9e998600 >[6125659.557227] x3 : dead000000000200 x2 : ffff000030123960 >[6125659.557732] x1 : ffff0000301239d4 x0 : 0000000000000001 >[6125659.558201] Call trace: >[6125659.558428] multi_cpu_stop+0xf4/0x180 >[6125659.558796] cpu_stopper_thread+0xec/0x178 >[6125659.559191] smpboot_thread_fn+0x158/0x1d8 >[6125659.559603] kthread+0x130/0x138 >[6125699.531184] watchdog: BUG: soft lockup - CPU#10 stuck for 23s! [kubelet:3381] >[6125699.531972] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6125699.538432] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6125699.544396] CPU: 10 PID: 3381 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6125699.545503] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6125699.546341] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6125699.546797] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6125699.547298] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6125699.547769] sp : ffff00001d773c20 >[6125699.548075] x29: ffff00001d773c20 x28: ffff8b9c1cb8e000 >[6125699.548555] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6125699.549029] x25: 0000000000000010 x24: 0000000000000000 >[6125699.549495] x23: 0000000000000020 x22: ffff59fd7fe49800 >[6125699.549963] x21: ffff8b9c16417c00 x20: ffff59fd7fe4b8a8 >[6125699.550435] x19: 0000000000000004 x18: 0000000000000000 >[6125699.550942] x17: 0000000000000655 x16: 0000000000000000 >[6125699.551428] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6125699.551944] x13: ffff8b8d72d56000 x12: ffff8b8d72d55350 >[6125699.552454] x11: 0000000000000000 x10: ffff00001d773c80 >[6125699.552941] x9 : ffff59fd7ef3dad0 x8 : ffff8b8d72d55000 >[6125699.553412] x7 : 0000000000000000 x6 : 0000000000000001 >[6125699.553887] x5 : 00000000fffffffb x4 : ffffffffffffffff >[6125699.554358] x3 : 0000000000000038 x2 : 000033d01fe85000 >[6125699.554962] x1 : ffff4a1c04346ff0 x0 : 0000000000000078 >[6125699.555542] Call trace: >[6125699.555798] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6125699.556290] memcg_numa_stat_show+0x1d0/0x260 >[6125699.556734] cgroup_seqfile_show+0x50/0xd0 >[6125699.557145] kernfs_seq_show+0x30/0x38 >[6125699.557525] seq_read+0x148/0x478 >[6125699.557858] kernfs_fop_read+0x2c/0x1e0 >[6125699.558224] __vfs_read+0x20/0x48 >[6125699.558567] vfs_read+0x98/0x168 >[6125699.558894] ksys_read+0x6c/0xe8 >[6125699.559216] __arm64_sys_read+0x20/0x28 >[6125699.559591] el0_svc_common.constprop.0+0xa8/0x200 >[6125699.560084] el0_svc_handler+0x30/0x80 >[6125699.560472] el0_svc+0x10/0x14 >[6125739.531259] watchdog: BUG: soft lockup - CPU#15 stuck for 22s! [kubelet:60635] >[6125739.532064] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6125739.538822] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6125739.544667] CPU: 15 PID: 60635 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6125739.545710] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6125739.546571] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6125739.547016] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6125739.547511] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6125739.548001] sp : ffff00001df63c20 >[6125739.548316] x29: ffff00001df63c20 x28: ffff8b7b610b6000 >[6125739.548798] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6125739.549258] x25: 0000000000000003 x24: 0000000000000000 >[6125739.549759] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6125739.550233] x21: ffff8dbb9b377c00 x20: ffff59fd7fe4b8a8 >[6125739.550793] x19: 0000000000000000 x18: 0000000000000000 >[6125739.551336] x17: 0000000000000659 x16: 0000000000000000 >[6125739.551930] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6125739.552495] x13: ffff8b8bdf96f000 x12: ffff8b8bdf96e311 >[6125739.552994] x11: 0000000000000000 x10: ffff00001df63c80 >[6125739.553485] x9 : ffff59fd7ef3dad0 x8 : 000000000004ef90 >[6125739.554022] x7 : 0000000000000005 x6 : 0000000000000006 >[6125739.554665] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6125739.555235] x3 : 0000000000000009 x2 : 000033904042a000 >[6125739.555807] x1 : ffff4a0cea9238b0 x0 : 0000000000000049 >[6125739.556372] Call trace: >[6125739.556625] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6125739.557106] memcg_numa_stat_show+0x1d0/0x260 >[6125739.557520] cgroup_seqfile_show+0x50/0xd0 >[6125739.557914] kernfs_seq_show+0x30/0x38 >[6125739.558261] seq_read+0x148/0x478 >[6125739.558623] kernfs_fop_read+0x2c/0x1e0 >[6125739.558990] __vfs_read+0x20/0x48 >[6125739.559337] vfs_read+0x98/0x168 >[6125739.559649] ksys_read+0x6c/0xe8 >[6125739.560016] __arm64_sys_read+0x20/0x28 >[6125739.560430] el0_svc_common.constprop.0+0xa8/0x200 >[6125739.560935] el0_svc_handler+0x30/0x80 >[6125739.561283] el0_svc+0x10/0x14 >[6125783.531341] watchdog: BUG: soft lockup - CPU#20 stuck for 22s! [kubelet:18288] >[6125783.532145] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6125783.538640] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6125783.544277] CPU: 20 PID: 18288 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6125783.545229] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6125783.546039] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6125783.546547] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6125783.547053] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6125783.547644] sp : ffff00001f91bc20 >[6125783.547977] x29: ffff00001f91bc20 x28: ffff8dac36520000 >[6125783.548556] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6125783.549120] x25: 0000000000000003 x24: 0000000000000000 >[6125783.549648] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6125783.550202] x21: ffff8dccf805a400 x20: ffff59fd7fe4b8a8 >[6125783.550730] x19: 0000000000000000 x18: 0000000000000000 >[6125783.551305] x17: 0000000000000002 x16: 0000000000000000 >[6125783.551864] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6125783.552389] x13: ffff8b9b31180000 x12: ffff8b9b3117f31c >[6125783.552948] x11: 0000000000000000 x10: ffff00001f91bc80 >[6125783.553464] x9 : ffff59fd7ef3dad0 x8 : 0000000000018148 >[6125783.554026] x7 : 0000000000000004 x6 : 0000000000000005 >[6125783.554667] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6125783.555169] x3 : 0000000000000034 x2 : 000033d01ffb1000 >[6125783.555703] x1 : ffff4a0f07af0920 x0 : 0000000000000074 >[6125783.556220] Call trace: >[6125783.556473] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6125783.557045] memcg_numa_stat_show+0x1d0/0x260 >[6125783.557503] cgroup_seqfile_show+0x50/0xd0 >[6125783.557964] kernfs_seq_show+0x30/0x38 >[6125783.558374] seq_read+0x148/0x478 >[6125783.558717] kernfs_fop_read+0x2c/0x1e0 >[6125783.559137] __vfs_read+0x20/0x48 >[6125783.559494] vfs_read+0x98/0x168 >[6125783.559818] ksys_read+0x6c/0xe8 >[6125783.560182] __arm64_sys_read+0x20/0x28 >[6125783.560587] el0_svc_common.constprop.0+0xa8/0x200 >[6125783.561089] el0_svc_handler+0x30/0x80 >[6125783.561438] el0_svc+0x10/0x14 >[6125827.531099] watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [migration/5:36] >[6125827.531663] watchdog: BUG: soft lockup - CPU#41 stuck for 22s! [kubelet:18304] >[6125827.531987] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6125827.532698] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6125827.539382] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6125827.546073] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6125827.552240] CPU: 5 PID: 36 Comm: migration/5 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6125827.558130] CPU: 41 PID: 18304 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6125827.559169] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6125827.560152] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6125827.561034] pstate: 60000005 (nZCv daif -PAN -UAO) >[6125827.561841] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6125827.562297] pc : multi_cpu_stop+0xf4/0x180 >[6125827.562738] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6125827.563133] lr : multi_cpu_stop+0xb4/0x180 >[6125827.563645] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6125827.563647] sp : ffff00001f943c20 >[6125827.564140] sp : ffff000008dfbd50 >[6125827.564628] x29: ffff00001f943c20 x28: ffff8d8cedd04000 >[6125827.564937] x29: ffff000008dfbd50 x28: 0000000000000000 >[6125827.565249] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6125827.565821] x27: ffff00003dc03938 x26: 0000000000000060 >[6125827.566322] x25: 0000000000000010 x24: 0000000000000000 >[6125827.566884] x25: 0000000000000000 x24: 0000000000000000 >[6125827.567368] x23: 0000000000000020 x22: ffff59fd7fe49800 >[6125827.567884] x23: 0000000000000000 x22: 0000000000000001 >[6125827.568378] x21: ffff8b7c1ae7d800 x20: ffff59fd7fe4b8a8 >[6125827.568845] x21: ffff00003dc039d4 x20: ffff00003dc039b0 >[6125827.569309] x19: 0000000000000004 x18: 0000000000000000 >[6125827.569811] x19: 0000000000000001 x18: 0000000000000000 >[6125827.570347] x17: 0000000000000002 x16: 0000000000000000 >[6125827.570858] x17: 0000000000000001 x16: 0000000000000000 >[6125827.571345] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6125827.571856] x15: 0000000000000000 x14: 0000000000000000 >[6125827.572375] x13: ffff8bcd57f14000 x12: ffff8bcd57f13345 >[6125827.572901] x13: 0000000000000000 x12: 0000000000000000 >[6125827.573411] x11: 0000000000000000 x10: ffff00001f943c80 >[6125827.574018] x11: 0000000000000023 x10: 0000000000000d60 >[6125827.574599] x9 : ffff59fd7ef3dad0 x8 : 0000000000000021 >[6125827.575106] x9 : ffff59fd7ede5a0c x8 : ffff8b7d9da24180 >[6125827.575630] x7 : 0000000000000001 x6 : 0000000000000002 >[6125827.576177] x7 : ffff8b7d6a380000 x6 : 0000000000000161 >[6125827.576754] x5 : 00000000fffffffa x4 : ffffffffffffffff >[6125827.577268] x5 : 000031804048c000 x4 : ffff8b7d9da239c0 >[6125827.577759] x3 : 0000000000000033 x2 : 000033d01ff98000 >[6125827.578253] x3 : dead000000000200 x2 : ffff00003dc03960 >[6125827.578771] x1 : ffff4a2dd170e9e0 x0 : 0000000000000073 >[6125827.579265] x1 : ffff00003dc039d4 x0 : 0000000000000001 >[6125827.579736] Call trace: >[6125827.580228] Call trace: >[6125827.580703] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6125827.580948] multi_cpu_stop+0xf4/0x180 >[6125827.581200] memcg_numa_stat_show+0x1d0/0x260 >[6125827.581715] cpu_stopper_thread+0xec/0x178 >[6125827.582088] cgroup_seqfile_show+0x50/0xd0 >[6125827.582521] smpboot_thread_fn+0x158/0x1d8 >[6125827.582925] kernfs_seq_show+0x30/0x38 >[6125827.583367] kthread+0x130/0x138 >[6125827.583769] seq_read+0x148/0x478 >[6125827.584953] kernfs_fop_read+0x2c/0x1e0 >[6125827.585334] __vfs_read+0x20/0x48 >[6125827.585709] vfs_read+0x98/0x168 >[6125827.586071] ksys_read+0x6c/0xe8 >[6125827.586367] __arm64_sys_read+0x20/0x28 >[6125827.586742] el0_svc_common.constprop.0+0xa8/0x200 >[6125827.587198] el0_svc_handler+0x30/0x80 >[6125827.587560] el0_svc+0x10/0x14 >[6125867.531171] watchdog: BUG: soft lockup - CPU#9 stuck for 22s! [kubelet:18889] >[6125867.531958] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6125867.538797] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6125867.544819] CPU: 9 PID: 18889 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6125867.545822] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6125867.546765] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6125867.547240] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6125867.547746] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6125867.548255] sp : ffff0000300f3c20 >[6125867.548588] x29: ffff0000300f3c20 x28: ffff8b7d51248000 >[6125867.549142] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6125867.549671] x25: 0000000000000003 x24: 0000000000000000 >[6125867.550177] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6125867.550673] x21: ffff8dad98a7e000 x20: ffff59fd7fe4b8a8 >[6125867.551174] x19: 0000000000000000 x18: 0000000000000000 >[6125867.551653] x17: 0000000000000002 x16: 0000000000000000 >[6125867.552133] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6125867.552654] x13: ffff8b9d205b9000 x12: ffff8b9d205b8307 >[6125867.553151] x11: 0000000000000000 x10: ffff0000300f3c80 >[6125867.553740] x9 : ffff59fd7ef3dad0 x8 : 000000000000c16f >[6125867.554292] x7 : 0000000000000004 x6 : 0000000000000005 >[6125867.554787] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6125867.555296] x3 : 0000000000000005 x2 : 000033804048c000 >[6125867.555799] x1 : ffff4a27a3185930 x0 : 0000000000000045 >[6125867.556299] Call trace: >[6125867.556552] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6125867.557030] memcg_numa_stat_show+0x1d0/0x260 >[6125867.557455] cgroup_seqfile_show+0x50/0xd0 >[6125867.557854] kernfs_seq_show+0x30/0x38 >[6125867.558257] seq_read+0x148/0x478 >[6125867.558579] kernfs_fop_read+0x2c/0x1e0 >[6125867.558958] __vfs_read+0x20/0x48 >[6125867.559385] vfs_read+0x98/0x168 >[6125867.559731] ksys_read+0x6c/0xe8 >[6125867.560094] __arm64_sys_read+0x20/0x28 >[6125867.560447] el0_svc_common.constprop.0+0xa8/0x200 >[6125867.560948] el0_svc_handler+0x30/0x80 >[6125867.561304] el0_svc+0x10/0x14 >[6125911.532074] watchdog: BUG: soft lockup - CPU#63 stuck for 22s! [kubelet:61154] >[6125911.532939] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6125911.539421] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6125911.545230] CPU: 63 PID: 61154 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6125911.546341] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6125911.547328] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6125911.547849] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6125911.548418] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6125911.549002] sp : ffff000059c5bc20 >[6125911.549366] x29: ffff000059c5bc20 x28: ffff8babf2798000 >[6125911.549936] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6125911.550498] x25: 0000000000000003 x24: 0000000000000000 >[6125911.551058] x23: 0000000000000008 x22: ffff59fd7fe49800 >[6125911.551633] x21: ffff8dbb81bb7c00 x20: ffff59fd7fe4b8a8 >[6125911.552195] x19: 0000000000000001 x18: 0000000000000000 >[6125911.552761] x17: 0000000000000002 x16: 0000000000000000 >[6125911.553428] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6125911.554038] x13: ffff8bbd40772000 x12: ffff8bbd40771311 >[6125911.554627] x11: 0000000000000000 x10: ffff000059c5bc80 >[6125911.555205] x9 : ffff59fd7ef3dad0 x8 : 000000000004ee9d >[6125911.555776] x7 : 0000000000000005 x6 : 0000000000000006 >[6125911.556350] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6125911.556935] x3 : 000000000000000b x2 : 000031904045c000 >[6125911.557511] x1 : ffff4a2b55b136e8 x0 : 000000000000000b >[6125911.558099] Call trace: >[6125911.558392] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6125911.558954] memcg_numa_stat_show+0x1d0/0x260 >[6125911.559448] cgroup_seqfile_show+0x50/0xd0 >[6125911.559898] kernfs_seq_show+0x30/0x38 >[6125911.560311] seq_read+0x148/0x478 >[6125911.560687] kernfs_fop_read+0x2c/0x1e0 >[6125911.561102] __vfs_read+0x20/0x48 >[6125911.561467] vfs_read+0x98/0x168 >[6125911.561882] ksys_read+0x6c/0xe8 >[6125911.562240] __arm64_sys_read+0x20/0x28 >[6125911.562669] el0_svc_common.constprop.0+0xa8/0x200 >[6125911.563234] el0_svc_handler+0x30/0x80 >[6125911.563728] el0_svc+0x10/0x14 >[6125955.531135] watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [kubelet:61154] >[6125955.531176] watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [migration/10:62] >[6125955.531929] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6125955.532695] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6125955.539348] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6125955.546091] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6125955.552168] CPU: 7 PID: 61154 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6125955.557941] CPU: 10 PID: 62 Comm: migration/10 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6125955.558938] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6125955.559960] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6125955.560859] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6125955.561757] pstate: 60000005 (nZCv daif -PAN -UAO) >[6125955.562233] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6125955.562709] pc : multi_cpu_stop+0x78/0x180 >[6125955.563351] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6125955.563354] sp : ffff000059c5bc20 >[6125955.563756] lr : multi_cpu_stop+0xb4/0x180 >[6125955.563759] sp : ffff000008ecbd50 >[6125955.564348] x29: ffff000059c5bc20 x28: ffff8b7d6040c000 >[6125955.564685] x29: ffff000008ecbd50 x28: 0000000000000000 >[6125955.565121] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6125955.565424] x27: ffff00005875b938 x26: 0000000000000060 >[6125955.565915] x25: 0000000000000010 x24: 0000000000000000 >[6125955.566471] x25: 0000000000000000 x24: 0000000000000000 >[6125955.567021] x23: 0000000000000020 x22: ffff59fd7fe49800 >[6125955.567479] x23: 0000000000000000 x22: 0000000000000001 >[6125955.568030] x21: ffff8b7d1b4f8400 x20: ffff59fd7fe4b8a8 >[6125955.568538] x21: ffff00005875b9d4 x20: ffff00005875b9b0 >[6125955.569034] x19: 0000000000000004 x18: 0000000000000000 >[6125955.569552] x19: 0000000000000001 x18: 0000000000000000 >[6125955.570116] x17: 0000000000000002 x16: 0000000000000000 >[6125955.570637] x17: 0000000000000001 x16: 0000000000000000 >[6125955.571149] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6125955.571669] x15: 0000000000000000 x14: 0000000000000000 >[6125955.572180] x13: ffff8b7d1ce6f000 x12: ffff8b7d1ce6e345 >[6125955.572712] x13: 0000000000000000 x12: 0000000000000000 >[6125955.573330] x11: 0000000000000000 x10: ffff000059c5bc80 >[6125955.573929] x11: 00000000000002cc x10: 0000000000000d60 >[6125955.574423] x9 : ffff59fd7ef3dad0 x8 : 0000000000000021 >[6125955.574920] x9 : ffff59fd7ede5a0c x8 : ffff8b8d9e907540 >[6125955.575433] x7 : 0000000000000001 x6 : 0000000000000002 >[6125955.575925] x7 : ffff8b8d8059ec00 x6 : 000000000000002d >[6125955.576425] x5 : 00000000fffffffa x4 : ffffffffffffffff >[6125955.576943] x5 : 0000319040443000 x4 : ffff8b8d9e906d80 >[6125955.577456] x3 : 000000000000003a x2 : 000033d0401d5000 >[6125955.577957] x3 : dead000000000200 x2 : ffff00005875b988 >[6125955.578438] x1 : ffff4a2f626d85f0 x0 : 000000000000003a >[6125955.578955] x1 : ffff00005875b9d4 x0 : 0000000000000001 >[6125955.579472] Call trace: >[6125955.579968] Call trace: >[6125955.580500] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6125955.581010] multi_cpu_stop+0x78/0x180 >[6125955.581257] memcg_numa_stat_show+0x1d0/0x260 >[6125955.581553] cpu_stopper_thread+0xec/0x178 >[6125955.582068] cgroup_seqfile_show+0x50/0xd0 >[6125955.582424] smpboot_thread_fn+0x158/0x1d8 >[6125955.582842] kernfs_seq_show+0x30/0x38 >[6125955.583314] kthread+0x130/0x138 >[6125955.583747] seq_read+0x148/0x478 >[6125955.585191] kernfs_fop_read+0x2c/0x1e0 >[6125955.585583] __vfs_read+0x20/0x48 >[6125955.585947] vfs_read+0x98/0x168 >[6125955.586262] ksys_read+0x6c/0xe8 >[6125955.586599] __arm64_sys_read+0x20/0x28 >[6125955.586971] el0_svc_common.constprop.0+0xa8/0x200 >[6125955.587425] el0_svc_handler+0x30/0x80 >[6125955.587789] el0_svc+0x10/0x14 >[6125995.531182] watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [kubelet:3380] >[6125995.532021] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6125995.538683] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6125995.544566] CPU: 10 PID: 3380 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6125995.545537] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6125995.546416] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6125995.546890] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6125995.547464] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6125995.547983] sp : ffff000031ba3c20 >[6125995.548387] x29: ffff000031ba3c20 x28: ffff8b7cddcbe000 >[6125995.548928] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6125995.549479] x25: 0000000000000003 x24: 0000000000000000 >[6125995.549995] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6125995.550510] x21: ffff8d9d08817800 x20: ffff59fd7fe4b8a8 >[6125995.551001] x19: 0000000000000000 x18: 0000000000000000 >[6125995.551485] x17: 0000000000000002 x16: 0000000000000000 >[6125995.551969] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6125995.552469] x13: ffff8b8d72d57000 x12: ffff8b8d72d562fd >[6125995.553070] x11: 0000000000000000 x10: ffff000031ba3c80 >[6125995.553594] x9 : ffff59fd7ef3dad0 x8 : 0000000000014ea8 >[6125995.554139] x7 : 0000000000000004 x6 : 0000000000000005 >[6125995.554640] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6125995.555149] x3 : 000000000000003f x2 : 000033d01ff34000 >[6125995.555700] x1 : ffff4a21807ae6f0 x0 : 000000000000007f >[6125995.556208] Call trace: >[6125995.556510] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6125995.556991] memcg_numa_stat_show+0x1d0/0x260 >[6125995.557458] cgroup_seqfile_show+0x50/0xd0 >[6125995.557866] kernfs_seq_show+0x30/0x38 >[6125995.558242] seq_read+0x148/0x478 >[6125995.558561] kernfs_fop_read+0x2c/0x1e0 >[6125995.558933] __vfs_read+0x20/0x48 >[6125995.559256] vfs_read+0x98/0x168 >[6125995.559561] ksys_read+0x6c/0xe8 >[6125995.559876] __arm64_sys_read+0x20/0x28 >[6125995.560227] el0_svc_common.constprop.0+0xa8/0x200 >[6125995.560684] el0_svc_handler+0x30/0x80 >[6125995.561032] el0_svc+0x10/0x14 >[6126039.531680] watchdog: BUG: soft lockup - CPU#42 stuck for 22s! [kubelet:3380] >[6126039.532605] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6126039.539254] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6126039.545189] CPU: 42 PID: 3380 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6126039.546209] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6126039.547022] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6126039.547441] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6126039.547932] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6126039.548431] sp : ffff000031ba3c20 >[6126039.548758] x29: ffff000031ba3c20 x28: ffff8dccf8440000 >[6126039.549255] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6126039.549742] x25: 0000000000000003 x24: 0000000000000000 >[6126039.550246] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6126039.550734] x21: ffff8dbbf7be8800 x20: ffff59fd7fe4b8a8 >[6126039.551225] x19: 0000000000000000 x18: 0000000000000000 >[6126039.551731] x17: 0000000000000002 x16: 0000000000000000 >[6126039.552212] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6126039.552827] x13: ffff8bcd52fe4000 x12: ffff8bcd52fe3311 >[6126039.553344] x11: 0000000000000000 x10: ffff000031ba3c80 >[6126039.553835] x9 : ffff59fd7ef3dad0 x8 : 000000000004ec05 >[6126039.554375] x7 : 0000000000000005 x6 : 0000000000000006 >[6126039.554868] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6126039.555354] x3 : 000000000000002f x2 : 000031d0404c0000 >[6126039.555832] x1 : ffff4a0f0b0c2000 x0 : 000000000000002f >[6126039.556325] Call trace: >[6126039.556597] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6126039.557108] memcg_numa_stat_show+0x1d0/0x260 >[6126039.557497] cgroup_seqfile_show+0x50/0xd0 >[6126039.557921] kernfs_seq_show+0x30/0x38 >[6126039.558274] seq_read+0x148/0x478 >[6126039.558588] kernfs_fop_read+0x2c/0x1e0 >[6126039.558922] __vfs_read+0x20/0x48 >[6126039.559241] vfs_read+0x98/0x168 >[6126039.559583] ksys_read+0x6c/0xe8 >[6126039.559897] __arm64_sys_read+0x20/0x28 >[6126039.560314] el0_svc_common.constprop.0+0xa8/0x200 >[6126039.560813] el0_svc_handler+0x30/0x80 >[6126039.561202] el0_svc+0x10/0x14 >[6126083.531260] watchdog: BUG: soft lockup - CPU#15 stuck for 22s! [kubelet:60635] >[6126083.532070] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6126083.538340] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6126083.544372] CPU: 15 PID: 60635 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6126083.545359] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6126083.546263] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6126083.546734] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6126083.547254] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6126083.547768] sp : ffff00001df63c20 >[6126083.548101] x29: ffff00001df63c20 x28: ffff8bad4c29c000 >[6126083.548606] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6126083.549084] x25: 0000000000000010 x24: 0000000000000000 >[6126083.549608] x23: 0000000000000020 x22: ffff59fd7fe49800 >[6126083.550124] x21: ffff8b7ca2bb0400 x20: ffff59fd7fe4b8a8 >[6126083.550635] x19: 0000000000000004 x18: 0000000000000000 >[6126083.551146] x17: 0000000000000008 x16: 0000000000000000 >[6126083.551653] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6126083.552259] x13: ffff8b8bdf969000 x12: ffff8b8bdf968345 >[6126083.552782] x11: 0000000000000000 x10: ffff00001df63c80 >[6126083.553292] x9 : ffff59fd7ef3dad0 x8 : 0000000000000021 >[6126083.553787] x7 : 0000000000000001 x6 : 0000000000000002 >[6126083.554292] x5 : 00000000fffffffa x4 : ffffffffffffffff >[6126083.554806] x3 : 0000000000000001 x2 : 0000318040428000 >[6126083.555311] x1 : ffff4a2d23ce69c0 x0 : 0000000000000001 >[6126083.555818] Call trace: >[6126083.556112] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6126083.556598] memcg_numa_stat_show+0x1d0/0x260 >[6126083.557025] cgroup_seqfile_show+0x50/0xd0 >[6126083.557451] kernfs_seq_show+0x30/0x38 >[6126083.557870] seq_read+0x148/0x478 >[6126083.558188] kernfs_fop_read+0x2c/0x1e0 >[6126083.558579] __vfs_read+0x20/0x48 >[6126083.558948] vfs_read+0x98/0x168 >[6126083.559277] ksys_read+0x6c/0xe8 >[6126083.559685] __arm64_sys_read+0x20/0x28 >[6126083.560042] el0_svc_common.constprop.0+0xa8/0x200 >[6126083.560539] el0_svc_handler+0x30/0x80 >[6126083.560904] el0_svc+0x10/0x14 >[6126123.531539] watchdog: BUG: soft lockup - CPU#33 stuck for 23s! [kubelet:19532] >[6126123.532466] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6126123.538915] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6126123.545076] CPU: 33 PID: 19532 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6126123.546079] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6126123.546950] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6126123.547454] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6126123.547976] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6126123.548559] sp : ffff00001e943c20 >[6126123.548892] x29: ffff00001e943c20 x28: ffff8b7bbd4dc000 >[6126123.549464] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6126123.549958] x25: 0000000000000003 x24: 0000000000000000 >[6126123.550470] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6126123.551024] x21: ffff8d9cafc87c00 x20: ffff59fd7fe4b8a8 >[6126123.551546] x19: 0000000000000000 x18: 0000000000000000 >[6126123.552214] x17: 0000000000000009 x16: 0000000000000000 >[6126123.552782] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6126123.553277] x13: ffff8bbd33bb7000 x12: ffff8bbd33bb62fd >[6126123.553809] x11: 0000000000000000 x10: ffff00001e943c80 >[6126123.554346] x9 : ffff59fd7ef3dad0 x8 : 0000000000014deb >[6126123.554890] x7 : 0000000000000004 x6 : 0000000000000005 >[6126123.555414] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6126123.555967] x3 : 000000000000000c x2 : 0000339040475000 >[6126123.556486] x1 : ffff4a0f1b2a67a0 x0 : 000000000000004c >[6126123.557014] Call trace: >[6126123.557279] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6126123.557801] memcg_numa_stat_show+0x1d0/0x260 >[6126123.558228] cgroup_seqfile_show+0x50/0xd0 >[6126123.558662] kernfs_seq_show+0x30/0x38 >[6126123.559053] seq_read+0x148/0x478 >[6126123.559373] kernfs_fop_read+0x2c/0x1e0 >[6126123.559747] __vfs_read+0x20/0x48 >[6126123.560111] vfs_read+0x98/0x168 >[6126123.560462] ksys_read+0x6c/0xe8 >[6126123.560794] __arm64_sys_read+0x20/0x28 >[6126123.561167] el0_svc_common.constprop.0+0xa8/0x200 >[6126123.561671] el0_svc_handler+0x30/0x80 >[6126123.562129] el0_svc+0x10/0x14 >[6126127.531178] watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [migration/10:62] >[6126127.532186] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6126127.539653] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6126127.545725] CPU: 10 PID: 62 Comm: migration/10 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6126127.546706] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6126127.547557] pstate: 60000005 (nZCv daif -PAN -UAO) >[6126127.548083] pc : multi_cpu_stop+0x78/0x180 >[6126127.548473] lr : multi_cpu_stop+0xb4/0x180 >[6126127.548927] sp : ffff000008ecbd50 >[6126127.549259] x29: ffff000008ecbd50 x28: 0000000000000000 >[6126127.549761] x27: ffff000053573938 x26: 0000000000000060 >[6126127.550256] x25: 0000000000000000 x24: 0000000000000000 >[6126127.550795] x23: 0000000000000000 x22: 0000000000000001 >[6126127.551315] x21: ffff0000535739d4 x20: ffff0000535739b0 >[6126127.551903] x19: 0000000000000001 x18: 0000000000000000 >[6126127.552440] x17: 0000000000000001 x16: 0000000000000000 >[6126127.552953] x15: 0000000000000000 x14: 0000000000000000 >[6126127.553483] x13: 0000000000000000 x12: 0000000000000000 >[6126127.554030] x11: 000000000000193d x10: 0000000000000d60 >[6126127.554539] x9 : ffff59fd7ede5a0c x8 : ffff8b8d9e907540 >[6126127.555054] x7 : ffff8b8d8059ec00 x6 : 00000000000000e3 >[6126127.555541] x5 : 0000319040443000 x4 : ffff8b8d9e906d80 >[6126127.556038] x3 : dead000000000200 x2 : ffff000053573960 >[6126127.556542] x1 : ffff0000535739d4 x0 : 0000000000000001 >[6126127.557059] Call trace: >[6126127.557311] multi_cpu_stop+0x78/0x180 >[6126127.557683] cpu_stopper_thread+0xec/0x178 >[6126127.558091] smpboot_thread_fn+0x158/0x1d8 >[6126127.558488] kthread+0x130/0x138 >[6126167.531650] watchdog: BUG: soft lockup - CPU#40 stuck for 23s! [kubelet:6818] >[6126167.532529] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6126167.539219] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6126167.545259] CPU: 40 PID: 6818 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6126167.546298] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6126167.547152] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6126167.547620] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6126167.548107] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6126167.548652] sp : ffff000030643c20 >[6126167.548978] x29: ffff000030643c20 x28: ffff8bac0cab6000 >[6126167.549476] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6126167.549978] x25: 0000000000000003 x24: 0000000000000000 >[6126167.550497] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6126167.551015] x21: ffff8dabe9b3dc00 x20: ffff59fd7fe4b8a8 >[6126167.551535] x19: 0000000000000000 x18: 0000000000000000 >[6126167.552166] x17: 0000000000000002 x16: 0000000000000000 >[6126167.552716] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6126167.553242] x13: ffff8bcbee284000 x12: ffff8bcbee283307 >[6126167.553732] x11: 0000000000000000 x10: ffff000030643c80 >[6126167.554238] x9 : ffff59fd7ef3dad0 x8 : 000000000000c7be >[6126167.554735] x7 : 0000000000000004 x6 : 0000000000000005 >[6126167.555221] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6126167.555704] x3 : 0000000000000007 x2 : 00003380404be000 >[6126167.556176] x1 : ffff4a0cf6945930 x0 : 0000000000000047 >[6126167.556686] Call trace: >[6126167.556971] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6126167.557416] memcg_numa_stat_show+0x1d0/0x260 >[6126167.557801] cgroup_seqfile_show+0x50/0xd0 >[6126167.558217] kernfs_seq_show+0x30/0x38 >[6126167.558584] seq_read+0x148/0x478 >[6126167.558911] kernfs_fop_read+0x2c/0x1e0 >[6126167.559268] __vfs_read+0x20/0x48 >[6126167.559588] vfs_read+0x98/0x168 >[6126167.559881] ksys_read+0x6c/0xe8 >[6126167.560174] __arm64_sys_read+0x20/0x28 >[6126167.560552] el0_svc_common.constprop.0+0xa8/0x200 >[6126167.561008] el0_svc_handler+0x30/0x80 >[6126167.561338] el0_svc+0x10/0x14 >[6126171.531475] watchdog: BUG: soft lockup - CPU#29 stuck for 22s! [migration/29:159] >[6126171.532329] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6126171.538804] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6126171.544614] CPU: 29 PID: 159 Comm: migration/29 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6126171.545733] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6126171.546580] pstate: 60000005 (nZCv daif -PAN -UAO) >[6126171.547026] pc : multi_cpu_stop+0x78/0x180 >[6126171.547423] lr : multi_cpu_stop+0xb4/0x180 >[6126171.547793] sp : ffff00000a1dbd50 >[6126171.548123] x29: ffff00000a1dbd50 x28: 0000000000000000 >[6126171.548614] x27: ffff000062cb3938 x26: 0000000000000060 >[6126171.549094] x25: 0000000000000000 x24: 0000000000000000 >[6126171.549640] x23: 0000000000000000 x22: 0000000000000001 >[6126171.550146] x21: ffff000062cb39d4 x20: ffff000062cb39b0 >[6126171.550653] x19: 0000000000000001 x18: 0000000000000000 >[6126171.551147] x17: 0000000000000001 x16: 0000000000000000 >[6126171.551761] x15: 0000000000000000 x14: 0000000000000000 >[6126171.552303] x13: 0000000000000000 x12: 0000000000000000 >[6126171.552814] x11: 0000000000000000 x10: 0000000000000d60 >[6126171.553374] x9 : ffff59fd7ede5a0c x8 : ffff8bad9e998dc0 >[6126171.553871] x7 : 001482f11cc8e7cc x6 : 00000000000003d3 >[6126171.554424] x5 : 00000000efe4b99a x4 : ffff8bad9e998600 >[6126171.554971] x3 : dead000000000200 x2 : ffff000062cb3960 >[6126171.555461] x1 : ffff000062cb39d4 x0 : 0000000000000001 >[6126171.555954] Call trace: >[6126171.556205] multi_cpu_stop+0x78/0x180 >[6126171.556614] cpu_stopper_thread+0xec/0x178 >[6126171.557021] smpboot_thread_fn+0x158/0x1d8 >[6126171.557425] kthread+0x130/0x138 >[6126211.531664] watchdog: BUG: soft lockup - CPU#41 stuck for 23s! [kubelet:18144] >[6126211.532545] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6126211.538839] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6126211.544788] CPU: 41 PID: 18144 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6126211.545804] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6126211.546636] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6126211.547078] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6126211.547610] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6126211.548104] sp : ffff0000351c3c20 >[6126211.548422] x29: ffff0000351c3c20 x28: ffff8d9cfb372000 >[6126211.548912] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6126211.549412] x25: 0000000000000003 x24: 0000000000000000 >[6126211.549936] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6126211.550461] x21: ffff8dcd3cd60000 x20: ffff59fd7fe4b8a8 >[6126211.550977] x19: 0000000000000000 x18: 0000000000000000 >[6126211.551614] x17: 0000000000000002 x16: 0000000000000000 >[6126211.552141] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6126211.552723] x13: ffff8bcd57f11000 x12: ffff8bcd57f1031c >[6126211.553250] x11: 0000000000000000 x10: ffff0000351c3c80 >[6126211.553816] x9 : ffff59fd7ef3dad0 x8 : 0000000000018c91 >[6126211.554312] x7 : 0000000000000005 x6 : 0000000000000006 >[6126211.554855] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6126211.555346] x3 : 0000000000000001 x2 : 0000338040428000 >[6126211.555868] x1 : ffff4a1f1aab53f0 x0 : 0000000000000041 >[6126211.556349] Call trace: >[6126211.556593] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6126211.557068] memcg_numa_stat_show+0x1d0/0x260 >[6126211.557493] cgroup_seqfile_show+0x50/0xd0 >[6126211.557924] kernfs_seq_show+0x30/0x38 >[6126211.558357] seq_read+0x148/0x478 >[6126211.558691] kernfs_fop_read+0x2c/0x1e0 >[6126211.559069] __vfs_read+0x20/0x48 >[6126211.559448] vfs_read+0x98/0x168 >[6126211.559784] ksys_read+0x6c/0xe8 >[6126211.560112] __arm64_sys_read+0x20/0x28 >[6126211.560585] el0_svc_common.constprop.0+0xa8/0x200 >[6126211.561060] el0_svc_handler+0x30/0x80 >[6126211.561514] el0_svc+0x10/0x14 >[6126215.531003] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [migration/0:13] >[6126215.531865] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6126215.539437] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6126215.545579] CPU: 0 PID: 13 Comm: migration/0 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6126215.546629] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6126215.547528] pstate: 60000005 (nZCv daif -PAN -UAO) >[6126215.548020] pc : multi_cpu_stop+0xf4/0x180 >[6126215.548423] lr : multi_cpu_stop+0xb4/0x180 >[6126215.548824] sp : ffff00000894bd50 >[6126215.549152] x29: ffff00000894bd50 x28: 0000000000000000 >[6126215.549664] x27: ffff00003dc03388 x26: 0000000000000060 >[6126215.550175] x25: 0000000000000000 x24: 0000000000000000 >[6126215.550666] x23: 0000000000000000 x22: 0000000000000001 >[6126215.551272] x21: ffff00003dc03424 x20: ffff00003dc03400 >[6126215.551779] x19: 0000000000000001 x18: 0000000000000000 >[6126215.552288] x17: 0000000000000001 x16: 0000000000000000 >[6126215.552845] x15: 0000000000000000 x14: 0000000000000000 >[6126215.553363] x13: 0000000000000000 x12: ffff59fd7fe531a8 >[6126215.553871] x11: 0000000000000000 x10: 0000000000000d60 >[6126215.554384] x9 : ffff59fd7ede5a0c x8 : ffff8b7d9d959f00 >[6126215.554897] x7 : ffff8b7d6b1b1800 x6 : 000000000000016e >[6126215.555477] x5 : 000031804040f000 x4 : ffff8b7d9d959740 >[6126215.555981] x3 : dead000000000200 x2 : ffff00003dc033b0 >[6126215.556545] x1 : ffff00003dc03424 x0 : 0000000000000001 >[6126215.557089] Call trace: >[6126215.557376] multi_cpu_stop+0xf4/0x180 >[6126215.557756] cpu_stopper_thread+0xec/0x178 >[6126215.558148] smpboot_thread_fn+0x158/0x1d8 >[6126215.558593] kthread+0x130/0x138 >[6126255.531088] watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [kubelet:18334] >[6126255.531897] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6126255.538627] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6126255.544443] CPU: 4 PID: 18334 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6126255.545539] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6126255.546407] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6126255.546904] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6126255.547420] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6126255.547950] sp : ffff00001f9a3c20 >[6126255.548366] x29: ffff00001f9a3c20 x28: ffff8b7b65038000 >[6126255.548966] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6126255.549506] x25: 0000000000000010 x24: 0000000000000000 >[6126255.550065] x23: 0000000000000020 x22: ffff59fd7fe49800 >[6126255.550641] x21: ffff8b8bc46c1400 x20: ffff59fd7fe4b8a8 >[6126255.551222] x19: 0000000000000004 x18: 0000000000000000 >[6126255.551761] x17: 0000000000000002 x16: 0000000000000000 >[6126255.552323] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6126255.552888] x13: ffff8b7d1b322000 x12: ffff8b7d1b32134c >[6126255.553387] x11: 0000000000000000 x10: ffff00001f9a3c80 >[6126255.553938] x9 : ffff59fd7ef3dad0 x8 : ffff8b7d1b321000 >[6126255.554493] x7 : 0000000000000000 x6 : 0000000000000001 >[6126255.555013] x5 : 00000000fffffffb x4 : ffffffffffffffff >[6126255.555524] x3 : 0000000000000009 x2 : 000031904042a000 >[6126255.556063] x1 : ffff4a0cf43cfff0 x0 : 0000000000000009 >[6126255.556596] Call trace: >[6126255.556840] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6126255.557368] memcg_numa_stat_show+0x1d0/0x260 >[6126255.557804] cgroup_seqfile_show+0x50/0xd0 >[6126255.558231] kernfs_seq_show+0x30/0x38 >[6126255.558591] seq_read+0x148/0x478 >[6126255.558954] kernfs_fop_read+0x2c/0x1e0 >[6126255.559332] __vfs_read+0x20/0x48 >[6126255.559675] vfs_read+0x98/0x168 >[6126255.559988] ksys_read+0x6c/0xe8 >[6126255.560331] __arm64_sys_read+0x20/0x28 >[6126255.560701] el0_svc_common.constprop.0+0xa8/0x200 >[6126255.561285] el0_svc_handler+0x30/0x80 >[6126255.561662] el0_svc+0x10/0x14 >[6126295.531009] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kubelet:11266] >[6126295.531849] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6126295.538514] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6126295.544437] CPU: 0 PID: 11266 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6126295.545471] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6126295.546347] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6126295.546841] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6126295.547341] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6126295.547905] sp : ffff0000545dbc20 >[6126295.548212] x29: ffff0000545dbc20 x28: ffff8bac31520000 >[6126295.548730] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6126295.549223] x25: 0000000000000003 x24: 0000000000000000 >[6126295.549723] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6126295.550229] x21: ffff8d9caea3c800 x20: ffff59fd7fe4b8a8 >[6126295.550846] x19: 0000000000000000 x18: 0000000000000000 >[6126295.551368] x17: 0000000000000002 x16: 0000000000000000 >[6126295.551906] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6126295.552392] x13: ffff8b8d71fcd000 x12: ffff8b8d71fcc2fd >[6126295.552911] x11: 0000000000000000 x10: ffff0000545dbc80 >[6126295.553406] x9 : ffff59fd7ef3dad0 x8 : 0000000000014bb2 >[6126295.553921] x7 : 0000000000000004 x6 : 0000000000000005 >[6126295.554489] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6126295.555020] x3 : 0000000000000004 x2 : 0000318040473000 >[6126295.555546] x1 : ffff4a0f23ce7960 x0 : 0000000000000004 >[6126295.556040] Call trace: >[6126295.556284] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6126295.556791] memcg_numa_stat_show+0x1d0/0x260 >[6126295.557224] cgroup_seqfile_show+0x50/0xd0 >[6126295.557641] kernfs_seq_show+0x30/0x38 >[6126295.558012] seq_read+0x148/0x478 >[6126295.558417] kernfs_fop_read+0x2c/0x1e0 >[6126295.558802] __vfs_read+0x20/0x48 >[6126295.559116] vfs_read+0x98/0x168 >[6126295.559447] ksys_read+0x6c/0xe8 >[6126295.559768] __arm64_sys_read+0x20/0x28 >[6126295.560127] el0_svc_common.constprop.0+0xa8/0x200 >[6126295.560703] el0_svc_handler+0x30/0x80 >[6126295.561083] el0_svc+0x10/0x14 >[6126299.531193] watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [migration/11:67] >[6126299.532074] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6126299.538650] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6126299.544512] CPU: 11 PID: 67 Comm: migration/11 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6126299.545528] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6126299.546397] pstate: 60000005 (nZCv daif -PAN -UAO) >[6126299.546850] pc : multi_cpu_stop+0xf4/0x180 >[6126299.547259] lr : multi_cpu_stop+0xb4/0x180 >[6126299.547633] sp : ffff000008ef3d50 >[6126299.547952] x29: ffff000008ef3d50 x28: 0000000000000000 >[6126299.548482] x27: ffff00005b6eb938 x26: 0000000000000060 >[6126299.549014] x25: 0000000000000000 x24: 0000000000000000 >[6126299.549495] x23: 0000000000000000 x22: 0000000000000001 >[6126299.550016] x21: ffff00005b6eb9d4 x20: ffff00005b6eb9b0 >[6126299.550509] x19: 0000000000000001 x18: 0000000000000000 >[6126299.551102] x17: 0000000000000001 x16: 0000000000000000 >[6126299.551618] x15: 0000000000000000 x14: 0000000000000000 >[6126299.552136] x13: 0000000000000000 x12: 0000000000000000 >[6126299.552697] x11: 0000000000000022 x10: 0000000000000d60 >[6126299.553241] x9 : ffff59fd7ede5a0c x8 : ffff8b8d9e94d2c0 >[6126299.553884] x7 : 0002f67b34470796 x6 : 0000000000000101 >[6126299.554447] x5 : 00000000fa83b2da x4 : ffff8b8d9e94cb00 >[6126299.554974] x3 : dead000000000200 x2 : ffff00005b6eb988 >[6126299.555471] x1 : ffff00005b6eb9d4 x0 : 0000000000000001 >[6126299.556017] Call trace: >[6126299.556258] multi_cpu_stop+0xf4/0x180 >[6126299.556680] cpu_stopper_thread+0xec/0x178 >[6126299.557073] smpboot_thread_fn+0x158/0x1d8 >[6126299.557502] kthread+0x130/0x138 >[6126339.531918] watchdog: BUG: soft lockup - CPU#55 stuck for 23s! [kubelet:4440] >[6126339.532768] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6126339.539086] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6126339.544738] CPU: 55 PID: 4440 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6126339.545706] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6126339.546649] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6126339.547126] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6126339.547628] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6126339.548131] sp : ffff0000329abc20 >[6126339.548433] x29: ffff0000329abc20 x28: ffff8b8c3be10000 >[6126339.548941] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6126339.549438] x25: 0000000000000003 x24: 0000000000000000 >[6126339.549914] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6126339.550482] x21: ffff8dac7f89b800 x20: ffff59fd7fe4b8a8 >[6126339.550965] x19: 0000000000000000 x18: 0000000000000000 >[6126339.551445] x17: 0000000000000002 x16: 0000000000000000 >[6126339.551927] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6126339.552433] x13: ffff8b9d1eb4b000 x12: ffff8b9d1eb4a307 >[6126339.552924] x11: 0000000000000000 x10: ffff0000329abc80 >[6126339.553439] x9 : ffff59fd7ef3dad0 x8 : 000000000000c3e8 >[6126339.553921] x7 : 0000000000000004 x6 : 0000000000000005 >[6126339.554466] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6126339.554947] x3 : 0000000000000012 x2 : 000033a040443000 >[6126339.555478] x1 : ffff4a19036f6510 x0 : 0000000000000052 >[6126339.555963] Call trace: >[6126339.556203] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6126339.556690] memcg_numa_stat_show+0x1d0/0x260 >[6126339.557139] cgroup_seqfile_show+0x50/0xd0 >[6126339.557522] kernfs_seq_show+0x30/0x38 >[6126339.557874] seq_read+0x148/0x478 >[6126339.558177] kernfs_fop_read+0x2c/0x1e0 >[6126339.558534] __vfs_read+0x20/0x48 >[6126339.558842] vfs_read+0x98/0x168 >[6126339.559137] ksys_read+0x6c/0xe8 >[6126339.559442] __arm64_sys_read+0x20/0x28 >[6126339.559798] el0_svc_common.constprop.0+0xa8/0x200 >[6126339.560226] el0_svc_handler+0x30/0x80 >[6126339.560688] el0_svc+0x10/0x14 >[6126383.531276] watchdog: BUG: soft lockup - CPU#16 stuck for 23s! [kubelet:18321] >[6126383.532001] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6126383.538630] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6126383.544250] CPU: 16 PID: 18321 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6126383.545184] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6126383.546043] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6126383.546492] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6126383.547009] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6126383.547482] sp : ffff00001f953c20 >[6126383.547801] x29: ffff00001f953c20 x28: ffff8b8ba16be000 >[6126383.548334] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6126383.548857] x25: 0000000000000003 x24: 0000000000000000 >[6126383.549346] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6126383.549837] x21: ffff8dac29e1ec00 x20: ffff59fd7fe4b8a8 >[6126383.550480] x19: 0000000000000000 x18: 0000000000000000 >[6126383.551053] x17: 0000000000000009 x16: 0000000000000000 >[6126383.551602] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6126383.552102] x13: ffff8b9ba2457000 x12: ffff8b9ba2456307 >[6126383.552653] x11: 0000000000000000 x10: ffff00001f953c80 >[6126383.553191] x9 : ffff59fd7ef3dad0 x8 : 000000000000c38d >[6126383.553719] x7 : 0000000000000004 x6 : 0000000000000005 >[6126383.554230] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6126383.554760] x3 : 0000000000000034 x2 : 000033d01ffb1000 >[6126383.555283] x1 : ffff4a2b4fd56ff0 x0 : 0000000000000074 >[6126383.555802] Call trace: >[6126383.556062] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6126383.556566] memcg_numa_stat_show+0x1d0/0x260 >[6126383.556972] cgroup_seqfile_show+0x50/0xd0 >[6126383.557346] kernfs_seq_show+0x30/0x38 >[6126383.557770] seq_read+0x148/0x478 >[6126383.558079] kernfs_fop_read+0x2c/0x1e0 >[6126383.558438] __vfs_read+0x20/0x48 >[6126383.558748] vfs_read+0x98/0x168 >[6126383.559055] ksys_read+0x6c/0xe8 >[6126383.559368] __arm64_sys_read+0x20/0x28 >[6126383.559733] el0_svc_common.constprop.0+0xa8/0x200 >[6126383.560287] el0_svc_handler+0x30/0x80 >[6126383.560672] el0_svc+0x10/0x14 >[6126427.531260] watchdog: BUG: soft lockup - CPU#15 stuck for 23s! [kubelet:61153] >[6126427.532086] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6126427.538489] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6126427.544283] CPU: 15 PID: 61153 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6126427.545286] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6126427.546167] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6126427.546631] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6126427.547153] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6126427.547704] sp : ffff000058b4bc20 >[6126427.548028] x29: ffff000058b4bc20 x28: ffff8dabf9ab2000 >[6126427.548537] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6126427.549059] x25: 0000000000000003 x24: 0000000000000000 >[6126427.549537] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6126427.550134] x21: ffff8dcd200f7400 x20: ffff59fd7fe4b8a8 >[6126427.550630] x19: 0000000000000000 x18: 0000000000000000 >[6126427.551183] x17: 0000000000000002 x16: 0000000000000000 >[6126427.551747] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6126427.552258] x13: ffff8b8bdf96b000 x12: ffff8b8bdf96a31c >[6126427.552834] x11: 0000000000000000 x10: ffff000058b4bc80 >[6126427.553359] x9 : ffff59fd7ef3dad0 x8 : 0000000000018326 >[6126427.553874] x7 : 0000000000000004 x6 : 0000000000000005 >[6126427.554445] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6126427.554963] x3 : 0000000000000011 x2 : 000033a04042a000 >[6126427.555506] x1 : ffff4a1dc99673d0 x0 : 0000000000000051 >[6126427.556019] Call trace: >[6126427.556263] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6126427.556759] memcg_numa_stat_show+0x1d0/0x260 >[6126427.557186] cgroup_seqfile_show+0x50/0xd0 >[6126427.557586] kernfs_seq_show+0x30/0x38 >[6126427.558038] seq_read+0x148/0x478 >[6126427.558362] kernfs_fop_read+0x2c/0x1e0 >[6126427.558757] __vfs_read+0x20/0x48 >[6126427.559124] vfs_read+0x98/0x168 >[6126427.559494] ksys_read+0x6c/0xe8 >[6126427.559901] __arm64_sys_read+0x20/0x28 >[6126427.560283] el0_svc_common.constprop.0+0xa8/0x200 >[6126427.560809] el0_svc_handler+0x30/0x80 >[6126427.561178] el0_svc+0x10/0x14 >[6126467.531258] watchdog: BUG: soft lockup - CPU#15 stuck for 22s! [kubelet:4832] >[6126467.532112] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6126467.538655] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6126467.544552] CPU: 15 PID: 4832 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6126467.545529] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6126467.546384] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6126467.546829] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6126467.547384] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6126467.547893] sp : ffff00005e9c3c20 >[6126467.548204] x29: ffff00005e9c3c20 x28: ffff8b7d5be10000 >[6126467.548751] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6126467.549253] x25: 0000000000000003 x24: 0000000000000000 >[6126467.549865] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6126467.550373] x21: ffff8d9d8698fc00 x20: ffff59fd7fe4b8a8 >[6126467.550922] x19: 0000000000000000 x18: 0000000000000000 >[6126467.551493] x17: 0000000000000002 x16: 0000000000000000 >[6126467.552001] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6126467.552536] x13: ffff8b8bdf970000 x12: ffff8b8bdf96f2fe >[6126467.553057] x11: 0000000000000000 x10: ffff00005e9c3c80 >[6126467.553604] x9 : ffff59fd7ef3dad0 x8 : 0000000000014d3a >[6126467.554101] x7 : 0000000000000004 x6 : 0000000000000005 >[6126467.554647] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6126467.555177] x3 : 000000000000002b x2 : 000031d04045c000 >[6126467.555739] x1 : ffff4a28c3873470 x0 : 000000000000002b >[6126467.556226] Call trace: >[6126467.556536] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6126467.557014] memcg_numa_stat_show+0x1d0/0x260 >[6126467.557470] cgroup_seqfile_show+0x50/0xd0 >[6126467.557859] kernfs_seq_show+0x30/0x38 >[6126467.558272] seq_read+0x148/0x478 >[6126467.558631] kernfs_fop_read+0x2c/0x1e0 >[6126467.559008] __vfs_read+0x20/0x48 >[6126467.559415] vfs_read+0x98/0x168 >[6126467.559740] ksys_read+0x6c/0xe8 >[6126467.560089] __arm64_sys_read+0x20/0x28 >[6126467.560478] el0_svc_common.constprop.0+0xa8/0x200 >[6126467.560946] el0_svc_handler+0x30/0x80 >[6126467.561342] el0_svc+0x10/0x14 >[6126511.531088] watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [kubelet:18354] >[6126511.531895] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6126511.538604] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6126511.544460] CPU: 4 PID: 18354 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6126511.545425] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6126511.546286] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6126511.546754] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6126511.547273] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6126511.547784] sp : ffff00001f9dbc20 >[6126511.548107] x29: ffff00001f9dbc20 x28: ffff8bbba5b34000 >[6126511.548632] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6126511.549226] x25: 0000000000000003 x24: 0000000000000000 >[6126511.549735] x23: 0000000000000008 x22: ffff59fd7fe49800 >[6126511.550248] x21: ffff8dbbb499fc00 x20: ffff59fd7fe4b8a8 >[6126511.550759] x19: 0000000000000001 x18: 0000000000000000 >[6126511.551337] x17: 000000000000064e x16: 0000000000000000 >[6126511.551897] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6126511.552392] x13: ffff8b7d1b326000 x12: ffff8b7d1b325312 >[6126511.552937] x11: 0000000000000000 x10: ffff00001f9dbc80 >[6126511.553437] x9 : ffff59fd7ef3dad0 x8 : 000000000004edc2 >[6126511.553976] x7 : 0000000000000005 x6 : 0000000000000006 >[6126511.554523] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6126511.555021] x3 : 0000000000000008 x2 : 0000319040411000 >[6126511.555574] x1 : ffff4a1dceb6b628 x0 : 0000000000000008 >[6126511.556097] Call trace: >[6126511.556343] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6126511.556855] memcg_numa_stat_show+0x1d0/0x260 >[6126511.557284] cgroup_seqfile_show+0x50/0xd0 >[6126511.557672] kernfs_seq_show+0x30/0x38 >[6126511.558090] seq_read+0x148/0x478 >[6126511.558418] kernfs_fop_read+0x2c/0x1e0 >[6126511.558788] __vfs_read+0x20/0x48 >[6126511.559214] vfs_read+0x98/0x168 >[6126511.559551] ksys_read+0x6c/0xe8 >[6126511.559910] __arm64_sys_read+0x20/0x28 >[6126511.560283] el0_svc_common.constprop.0+0xa8/0x200 >[6126511.560806] el0_svc_handler+0x30/0x80 >[6126511.561206] el0_svc+0x10/0x14 >[6126555.531617] watchdog: BUG: soft lockup - CPU#38 stuck for 22s! [kubelet:18289] >[6126555.532424] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6126555.539324] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6126555.544946] CPU: 38 PID: 18289 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6126555.545927] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6126555.546774] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6126555.547240] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6126555.547744] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6126555.548286] sp : ffff00001f81bc20 >[6126555.548663] x29: ffff00001f81bc20 x28: ffff8d9d9bd6c000 >[6126555.549261] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6126555.549816] x25: 0000000000000003 x24: 0000000000000000 >[6126555.550340] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6126555.550859] x21: ffff8dcd332f5000 x20: ffff59fd7fe4b8a8 >[6126555.551435] x19: 0000000000000000 x18: 0000000000000000 >[6126555.551885] x17: 0000000000000002 x16: 0000000000000000 >[6126555.552439] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6126555.552975] x13: ffff8bbd2ccc4000 x12: ffff8bbd2ccc331d >[6126555.553531] x11: 0000000000000000 x10: ffff00001f81bc80 >[6126555.554089] x9 : ffff59fd7ef3dad0 x8 : 0000000000017fac >[6126555.554606] x7 : 0000000000000004 x6 : 0000000000000005 >[6126555.555144] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6126555.555680] x3 : 0000000000000031 x2 : 000033d01ff66000 >[6126555.556252] x1 : ffff4a1f1de4c060 x0 : 0000000000000071 >[6126555.556800] Call trace: >[6126555.557044] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6126555.557510] memcg_numa_stat_show+0x1d0/0x260 >[6126555.558024] cgroup_seqfile_show+0x50/0xd0 >[6126555.558455] kernfs_seq_show+0x30/0x38 >[6126555.559009] seq_read+0x148/0x478 >[6126555.559374] kernfs_fop_read+0x2c/0x1e0 >[6126555.559784] __vfs_read+0x20/0x48 >[6126555.560141] vfs_read+0x98/0x168 >[6126555.560532] ksys_read+0x6c/0xe8 >[6126555.560875] __arm64_sys_read+0x20/0x28 >[6126555.561268] el0_svc_common.constprop.0+0xa8/0x200 >[6126555.561733] el0_svc_handler+0x30/0x80 >[6126555.562099] el0_svc+0x10/0x14 >[6126599.531258] watchdog: BUG: soft lockup - CPU#15 stuck for 22s! [kubelet:18888] >[6126599.532163] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6126599.538779] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6126599.544530] CPU: 15 PID: 18888 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6126599.545585] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6126599.546477] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6126599.546977] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6126599.547546] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6126599.548086] sp : ffff00001fec3c20 >[6126599.548508] x29: ffff00001fec3c20 x28: ffff8dac24d34000 >[6126599.549016] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6126599.549578] x25: 0000000000000010 x24: 0000000000000000 >[6126599.550188] x23: 0000000000000020 x22: ffff59fd7fe49800 >[6126599.550691] x21: ffff8b9bb08d7c00 x20: ffff59fd7fe4b8a8 >[6126599.551248] x19: 0000000000000004 x18: 0000000000000000 >[6126599.551741] x17: 0000000000000002 x16: 0000000000000000 >[6126599.552286] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6126599.552853] x13: ffff8b8bdf970000 x12: ffff8b8bdf96f350 >[6126599.553396] x11: 0000000000000000 x10: ffff00001fec3c80 >[6126599.553905] x9 : ffff59fd7ef3dad0 x8 : ffff8b8bdf96f000 >[6126599.554371] x7 : 0000000000000000 x6 : 0000000000000001 >[6126599.554829] x5 : 00000000fffffffb x4 : ffffffffffffffff >[6126599.555300] x3 : 0000000000000003 x2 : 000033804045a000 >[6126599.555790] x1 : ffff4a2dc62a02a0 x0 : 0000000000000043 >[6126599.556262] Call trace: >[6126599.556558] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6126599.557065] memcg_numa_stat_show+0x1d0/0x260 >[6126599.557556] cgroup_seqfile_show+0x50/0xd0 >[6126599.557946] kernfs_seq_show+0x30/0x38 >[6126599.558335] seq_read+0x148/0x478 >[6126599.558740] kernfs_fop_read+0x2c/0x1e0 >[6126599.559104] __vfs_read+0x20/0x48 >[6126599.559481] vfs_read+0x98/0x168 >[6126599.559800] ksys_read+0x6c/0xe8 >[6126599.560177] __arm64_sys_read+0x20/0x28 >[6126599.560583] el0_svc_common.constprop.0+0xa8/0x200 >[6126599.561143] el0_svc_handler+0x30/0x80 >[6126599.561512] el0_svc+0x10/0x14 >[6126639.531633] watchdog: BUG: soft lockup - CPU#39 stuck for 22s! [kubelet:11266] >[6126639.532452] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6126639.538882] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6126639.544586] CPU: 39 PID: 11266 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6126639.545657] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6126639.546606] pstate: 20000005 (nzCv daif -PAN -UAO) >[6126639.547058] pc : css_next_descendant_pre+0x24/0x88 >[6126639.547562] lr : mem_cgroup_iter+0xc4/0x378 >[6126639.548011] sp : ffff0000545dbc10 >[6126639.548426] x29: ffff0000545dbc10 x28: ffff8b7b9f9ca000 >[6126639.549067] x27: ffff59fd7f671bf8 x26: ffff59fd7f8b5a88 >[6126639.549565] x25: 0000000000000010 x24: 0000000000000001 >[6126639.550114] x23: ffff8b7b9f9ca000 x22: 0000000000000000 >[6126639.550618] x21: ffff8b7b9f9ca000 x20: ffff8dbd9e88a000 >[6126639.551150] x19: ffff8b7b9f9ca000 x18: 0000000000000000 >[6126639.551667] x17: 0000000000000649 x16: 0000000000000000 >[6126639.552215] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6126639.552748] x13: ffff8bbd2cd04000 x12: ffff8bbd2cd03307 >[6126639.553265] x11: 0000000000000000 x10: ffff0000545dbc80 >[6126639.553795] x9 : ffff59fd7ef3cdc4 x8 : 000000000000c41e >[6126639.554296] x7 : 0000000000000004 x6 : 0000000000000005 >[6126639.554812] x5 : 00000000fffffff7 x4 : 0000000000000000 >[6126639.555318] x3 : 0000000000000000 x2 : 0000000000000000 >[6126639.555814] x1 : ffff8b7b9f9ca000 x0 : ffff8b7b9f9ca000 >[6126639.556328] Call trace: >[6126639.556592] css_next_descendant_pre+0x24/0x88 >[6126639.557049] mem_cgroup_iter+0xc4/0x378 >[6126639.557440] memcg_numa_stat_show+0x1e4/0x260 >[6126639.557861] cgroup_seqfile_show+0x50/0xd0 >[6126639.558365] kernfs_seq_show+0x30/0x38 >[6126639.558740] seq_read+0x148/0x478 >[6126639.559054] kernfs_fop_read+0x2c/0x1e0 >[6126639.559508] __vfs_read+0x20/0x48 >[6126639.559919] vfs_read+0x98/0x168 >[6126639.560251] ksys_read+0x6c/0xe8 >[6126639.560571] __arm64_sys_read+0x20/0x28 >[6126639.561015] el0_svc_common.constprop.0+0xa8/0x200 >[6126639.561557] el0_svc_handler+0x30/0x80 >[6126639.561918] el0_svc+0x10/0x14 >[6126683.531260] watchdog: BUG: soft lockup - CPU#15 stuck for 22s! [kubelet:18930] >[6126683.532117] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6126683.538870] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6126683.544594] CPU: 15 PID: 18930 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6126683.545570] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6126683.546434] pstate: 20000005 (nzCv daif -PAN -UAO) >[6126683.546921] pc : css_next_descendant_pre+0x24/0x88 >[6126683.547389] lr : mem_cgroup_iter+0xc4/0x378 >[6126683.547900] sp : ffff0000303fbc10 >[6126683.548230] x29: ffff0000303fbc10 x28: ffff8dbc70fdc000 >[6126683.548802] x27: ffff59fd7f671bf8 x26: ffff59fd7f8b5a88 >[6126683.549285] x25: 0000000000000010 x24: 0000000000000001 >[6126683.549780] x23: ffff8dbc70fdc000 x22: 0000000000000000 >[6126683.550280] x21: ffff8dbc70fdc000 x20: ffff8dbd9e88a000 >[6126683.550787] x19: ffff8dbc70fdc000 x18: 0000000000000000 >[6126683.551335] x17: 0000000000000002 x16: 0000000000000000 >[6126683.551846] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6126683.552348] x13: ffff8b8bdf96b000 x12: ffff8b8bdf96a31c >[6126683.552906] x11: 0000000000000000 x10: ffff0000303fbc80 >[6126683.553448] x9 : ffff59fd7ef3cdc4 x8 : 00000000000181ec >[6126683.553958] x7 : 0000000000000004 x6 : 0000000000000005 >[6126683.554512] x5 : 00000000fffffff7 x4 : 0000000000000000 >[6126683.555047] x3 : 0000000000000000 x2 : 0000000000000000 >[6126683.555657] x1 : ffff8dbc70fdc000 x0 : ffff8dbc70fdc000 >[6126683.556200] Call trace: >[6126683.556450] css_next_descendant_pre+0x24/0x88 >[6126683.556900] mem_cgroup_iter+0xc4/0x378 >[6126683.557371] memcg_numa_stat_show+0x1e4/0x260 >[6126683.557772] cgroup_seqfile_show+0x50/0xd0 >[6126683.558315] kernfs_seq_show+0x30/0x38 >[6126683.558699] seq_read+0x148/0x478 >[6126683.559094] kernfs_fop_read+0x2c/0x1e0 >[6126683.559472] __vfs_read+0x20/0x48 >[6126683.559875] vfs_read+0x98/0x168 >[6126683.560202] ksys_read+0x6c/0xe8 >[6126683.560599] __arm64_sys_read+0x20/0x28 >[6126683.560971] el0_svc_common.constprop.0+0xa8/0x200 >[6126683.561491] el0_svc_handler+0x30/0x80 >[6126683.561843] el0_svc+0x10/0x14 >[6126687.531114] watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [migration/6:41] >[6126687.531934] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6126687.538716] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6126687.544425] CPU: 6 PID: 41 Comm: migration/6 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6126687.545445] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6126687.546310] pstate: 60000005 (nZCv daif -PAN -UAO) >[6126687.546846] pc : multi_cpu_stop+0x78/0x180 >[6126687.547247] lr : multi_cpu_stop+0xb4/0x180 >[6126687.547646] sp : ffff000008e23d50 >[6126687.548132] x29: ffff000008e23d50 x28: 0000000000000000 >[6126687.548695] x27: ffff000052ac3938 x26: 0000000000000060 >[6126687.549251] x25: 0000000000000000 x24: 0000000000000000 >[6126687.549811] x23: 0000000000000000 x22: 0000000000000001 >[6126687.550364] x21: ffff000052ac39d4 x20: ffff000052ac39b0 >[6126687.550923] x19: 0000000000000001 x18: 0000000000000000 >[6126687.551479] x17: 0000000000000001 x16: 0000000000000000 >[6126687.552032] x15: 0000000000000000 x14: 0000000000000000 >[6126687.552603] x13: 0000000000000000 x12: 0000000000000000 >[6126687.553163] x11: 0000000000000000 x10: 0000000000000d60 >[6126687.553724] x9 : ffff59fd7ede5a0c x8 : ffff8b7d9da79f00 >[6126687.554288] x7 : ffff8b7d6d737800 x6 : 0000000000000106 >[6126687.554852] x5 : 00003180404a5000 x4 : ffff8b7d9da79740 >[6126687.555412] x3 : dead000000000200 x2 : ffff000052ac3960 >[6126687.555964] x1 : ffff000052ac39d4 x0 : 0000000000000001 >[6126687.556519] Call trace: >[6126687.556796] multi_cpu_stop+0x78/0x180 >[6126687.557204] cpu_stopper_thread+0xec/0x178 >[6126687.557634] smpboot_thread_fn+0x158/0x1d8 >[6126687.558162] kthread+0x130/0x138 >[6126723.531098] watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [migration/5:36] >[6126723.531180] watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [kubelet:19592] >[6126723.531970] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6126723.532645] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6126723.539209] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6126723.545730] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6126723.551707] CPU: 5 PID: 36 Comm: migration/5 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6126723.557707] CPU: 10 PID: 19592 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6126723.558666] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6126723.559740] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6126723.560666] pstate: 60000005 (nZCv daif -PAN -UAO) >[6126723.561550] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6126723.562039] pc : multi_cpu_stop+0x78/0x180 >[6126723.562579] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6126723.562971] lr : multi_cpu_stop+0xb4/0x180 >[6126723.563507] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6126723.563909] sp : ffff000008dfbd50 >[6126723.564409] sp : ffff000030fcbc20 >[6126723.564733] x29: ffff000008dfbd50 x28: 0000000000000000 >[6126723.565061] x29: ffff000030fcbc20 x28: ffff8b9b7db2a000 >[6126723.565581] x27: ffff000051f83938 x26: 0000000000000060 >[6126723.566144] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6126723.566666] x25: 0000000000000000 x24: 0000000000000000 >[6126723.567152] x25: 0000000000000003 x24: 0000000000000000 >[6126723.567806] x23: 0000000000000000 x22: 0000000000000001 >[6126723.568307] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6126723.568855] x21: ffff000051f839d4 x20: ffff000051f839b0 >[6126723.569325] x21: ffff8d8ce059c000 x20: ffff59fd7fe4b8a8 >[6126723.569823] x19: 0000000000000001 x18: 0000000000000000 >[6126723.570332] x19: 0000000000000000 x18: 0000000000000000 >[6126723.570853] x17: 0000000000000001 x16: 0000000000000000 >[6126723.571340] x17: 0000000000000002 x16: 0000000000000000 >[6126723.571831] x15: 0000000000000000 x14: 0000000000000000 >[6126723.572336] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6126723.572851] x13: 0000000000000000 x12: 0000000000000000 >[6126723.573396] x13: ffff8b8d72d55000 x12: ffff8b8d72d542f3 >[6126723.573895] x11: 0000000000000000 x10: 0000000000000d60 >[6126723.574404] x11: 0000000000000000 x10: ffff000030fcbc80 >[6126723.574914] x9 : ffff59fd7ede5a0c x8 : ffff8b7d9da24180 >[6126723.575452] x9 : ffff59fd7ef3dad0 x8 : 0000000000037456 >[6126723.575966] x7 : ffff8b7d6d734400 x6 : 0000000000000155 >[6126723.576478] x7 : 0000000000000005 x6 : 0000000000000006 >[6126723.576997] x5 : 000031804048c000 x4 : ffff8b7d9da239c0 >[6126723.577606] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6126723.578167] x3 : dead000000000200 x2 : ffff000051f83960 >[6126723.578712] x3 : 000000000000000e x2 : 00003390404a7000 >[6126723.579218] x1 : ffff000051f839d4 x0 : 0000000000000001 >[6126723.579728] x1 : ffff4a0cfce23500 x0 : 000000000000004e >[6126723.580240] Call trace: >[6126723.580814] Call trace: >[6126723.581079] multi_cpu_stop+0x78/0x180 >[6126723.581340] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6126723.581684] cpu_stopper_thread+0xec/0x178 >[6126723.582130] memcg_numa_stat_show+0x1d0/0x260 >[6126723.582520] smpboot_thread_fn+0x158/0x1d8 >[6126723.582952] cgroup_seqfile_show+0x50/0xd0 >[6126723.583335] kthread+0x130/0x138 >[6126723.583725] kernfs_seq_show+0x30/0x38 >[6126723.584397] seq_read+0x148/0x478 >[6126723.584734] kernfs_fop_read+0x2c/0x1e0 >[6126723.585097] __vfs_read+0x20/0x48 >[6126723.585416] vfs_read+0x98/0x168 >[6126723.585739] ksys_read+0x6c/0xe8 >[6126723.586057] __arm64_sys_read+0x20/0x28 >[6126723.586433] el0_svc_common.constprop.0+0xa8/0x200 >[6126723.586891] el0_svc_handler+0x30/0x80 >[6126723.587259] el0_svc+0x10/0x14 >[6126767.531003] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [migration/0:13] >[6126767.531196] watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [kubelet:18889] >[6126767.531802] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6126767.532464] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6126767.539167] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6126767.545965] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6126767.552060] CPU: 0 PID: 13 Comm: migration/0 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6126767.557913] CPU: 11 PID: 18889 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6126767.558939] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6126767.559969] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6126767.560864] pstate: 60000005 (nZCv daif -PAN -UAO) >[6126767.561716] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6126767.562258] pc : multi_cpu_stop+0xf4/0x180 >[6126767.562261] lr : multi_cpu_stop+0xb4/0x180 >[6126767.562716] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6126767.562719] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6126767.563158] sp : ffff00000894bd50 >[6126767.563606] sp : ffff0000300f3c20 >[6126767.564158] x29: ffff00000894bd50 x28: 0000000000000000 >[6126767.564730] x29: ffff0000300f3c20 x28: ffff8bcc743fe000 >[6126767.565074] x27: ffff00003e1d3938 x26: 0000000000000060 >[6126767.565448] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6126767.565955] x25: 0000000000000000 x24: 0000000000000000 >[6126767.566498] x25: 0000000000000003 x24: 0000000000000000 >[6126767.566998] x23: 0000000000000000 x22: 0000000000000001 >[6126767.567637] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6126767.568207] x21: ffff00003e1d39d4 x20: ffff00003e1d39b0 >[6126767.568720] x21: ffff8dbc10eccc00 x20: ffff59fd7fe4b8a8 >[6126767.569276] x19: 0000000000000001 x18: 0000000000000000 >[6126767.569817] x19: 0000000000000000 x18: 0000000000000000 >[6126767.570415] x17: 0000000000000001 x16: 0000000000000000 >[6126767.570910] x17: 0000000000000002 x16: 0000000000000000 >[6126767.571455] x15: 0000000000000000 x14: 0000000000000000 >[6126767.571958] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6126767.572463] x13: 0000000000000000 x12: 0000000000000000 >[6126767.572966] x13: ffff8b8d8045f000 x12: ffff8b8d8045e311 >[6126767.573462] x11: 0000000000000000 x10: 0000000000000d60 >[6126767.573963] x11: 0000000000000000 x10: ffff0000300f3c80 >[6126767.574471] x9 : ffff59fd7ede5a0c x8 : ffff8b7d9d959f00 >[6126767.574962] x9 : ffff59fd7ef3dad0 x8 : 000000000004ed74 >[6126767.575470] x7 : ffff8b7d818e0400 x6 : 0000000000000110 >[6126767.576024] x7 : 0000000000000005 x6 : 0000000000000006 >[6126767.576555] x5 : 000031804040f000 x4 : ffff8b7d9d959740 >[6126767.577054] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6126767.577678] x3 : dead000000000200 x2 : ffff00003e1d3960 >[6126767.578179] x3 : 0000000000000013 x2 : 000033a04045c000 >[6126767.578670] x1 : ffff00003e1d39d4 x0 : 0000000000000001 >[6126767.579179] x1 : ffff4a15e19762c8 x0 : 0000000000000053 >[6126767.579691] Call trace: >[6126767.580199] Call trace: >[6126767.580706] multi_cpu_stop+0xf4/0x180 >[6126767.581231] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6126767.581480] cpu_stopper_thread+0xec/0x178 >[6126767.581739] memcg_numa_stat_show+0x1d0/0x260 >[6126767.582124] smpboot_thread_fn+0x158/0x1d8 >[6126767.582597] cgroup_seqfile_show+0x50/0xd0 >[6126767.582987] kthread+0x130/0x138 >[6126767.583393] kernfs_seq_show+0x30/0x38 >[6126767.584846] seq_read+0x148/0x478 >[6126767.585174] kernfs_fop_read+0x2c/0x1e0 >[6126767.585553] __vfs_read+0x20/0x48 >[6126767.585883] vfs_read+0x98/0x168 >[6126767.586208] ksys_read+0x6c/0xe8 >[6126767.586535] __arm64_sys_read+0x20/0x28 >[6126767.586913] el0_svc_common.constprop.0+0xa8/0x200 >[6126767.587475] el0_svc_handler+0x30/0x80 >[6126767.587890] el0_svc+0x10/0x14 >[6126811.531228] watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [kubelet:18288] >[6126811.531301] watchdog: BUG: soft lockup - CPU#18 stuck for 22s! [migration/18:103] >[6126811.532069] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6126811.532844] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6126811.539438] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6126811.545943] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6126811.551777] CPU: 13 PID: 18288 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6126811.557739] CPU: 18 PID: 103 Comm: migration/18 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6126811.558851] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6126811.559895] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6126811.560815] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6126811.561682] pstate: 60000005 (nZCv daif -PAN -UAO) >[6126811.562152] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6126811.562630] pc : multi_cpu_stop+0xf4/0x180 >[6126811.562633] lr : multi_cpu_stop+0xb4/0x180 >[6126811.563153] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6126811.563523] sp : ffff00000a01bd50 >[6126811.563910] sp : ffff00001f91bc20 >[6126811.564409] x29: ffff00000a01bd50 x28: 0000000000000000 >[6126811.564748] x29: ffff00001f91bc20 x28: ffff8b7d5d736000 >[6126811.565069] x27: ffff00005612b938 x26: 0000000000000060 >[6126811.565581] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6126811.566097] x25: 0000000000000000 x24: 0000000000000000 >[6126811.566611] x25: 0000000000000010 x24: 0000000000000000 >[6126811.567209] x23: 0000000000000000 x22: 0000000000000001 >[6126811.567753] x23: 0000000000000020 x22: ffff59fd7fe49800 >[6126811.568348] x21: ffff00005612b9d4 x20: ffff00005612b9b0 >[6126811.568907] x21: ffff8b8d6a3b7000 x20: ffff59fd7fe4b8a8 >[6126811.569457] x19: 0000000000000001 x18: 0000000000000000 >[6126811.570026] x19: 0000000000000004 x18: 0000000000000000 >[6126811.570571] x17: 0000000000000001 x16: 0000000000000000 >[6126811.571120] x17: 0000000000000002 x16: 0000000000000000 >[6126811.571613] x15: 0000000000000000 x14: 0000000000000000 >[6126811.572080] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6126811.572561] x13: 0000000000000000 x12: 0000000000000000 >[6126811.573053] x13: ffff8b8d73419000 x12: ffff8b8d7341834c >[6126811.573596] x11: 000000000000005f x10: 0000000000000d60 >[6126811.574145] x11: 0000000000000000 x10: ffff00001f91bc80 >[6126811.574640] x9 : ffff59fd7ede5a0c x8 : ffff8b9d9e92f540 >[6126811.575222] x9 : ffff59fd7ef3dad0 x8 : ffff8b8d73418000 >[6126811.575767] x7 : ffff8b9d5fd26800 x6 : 000000000000007a >[6126811.576354] x7 : 0000000000000000 x6 : 0000000000000001 >[6126811.576990] x5 : 000031a040443000 x4 : ffff8b9d9e92ed80 >[6126811.577490] x5 : 00000000fffffffb x4 : ffffffffffffffff >[6126811.577991] x3 : dead000000000200 x2 : ffff00005612b988 >[6126811.578502] x3 : 0000000000000035 x2 : 000033d0402e8000 >[6126811.579000] x1 : ffff00005612b9d4 x0 : 0000000000000001 >[6126811.579492] x1 : ffff4a2a64275010 x0 : 0000000000000035 >[6126811.579969] Call trace: >[6126811.580497] Call trace: >[6126811.581013] multi_cpu_stop+0xf4/0x180 >[6126811.581249] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6126811.581549] cpu_stopper_thread+0xec/0x178 >[6126811.581885] memcg_numa_stat_show+0x1d0/0x260 >[6126811.582328] smpboot_thread_fn+0x158/0x1d8 >[6126811.582755] cgroup_seqfile_show+0x50/0xd0 >[6126811.583145] kthread+0x130/0x138 >[6126811.583520] kernfs_seq_show+0x30/0x38 >[6126811.584676] seq_read+0x148/0x478 >[6126811.585013] kernfs_fop_read+0x2c/0x1e0 >[6126811.585385] __vfs_read+0x20/0x48 >[6126811.585701] vfs_read+0x98/0x168 >[6126811.586020] ksys_read+0x6c/0xe8 >[6126811.586349] __arm64_sys_read+0x20/0x28 >[6126811.586729] el0_svc_common.constprop.0+0xa8/0x200 >[6126811.587306] el0_svc_handler+0x30/0x80 >[6126811.587681] el0_svc+0x10/0x14 >[6126851.531800] watchdog: BUG: soft lockup - CPU#49 stuck for 23s! [kubelet:19532] >[6126851.532648] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6126851.539676] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6126851.545218] CPU: 49 PID: 19532 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6126851.546159] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6126851.547102] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6126851.547596] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6126851.548075] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6126851.548577] sp : ffff00001e943c20 >[6126851.548880] x29: ffff00001e943c20 x28: ffff8bcca0d2e000 >[6126851.549353] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6126851.549829] x25: 0000000000000003 x24: 0000000000000000 >[6126851.550307] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6126851.550790] x21: ffff8d8d4ae2f800 x20: ffff59fd7fe4b8a8 >[6126851.551337] x19: 0000000000000000 x18: 0000000000000000 >[6126851.551861] x17: 0000000000000008 x16: 0000000000000000 >[6126851.552350] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6126851.552860] x13: ffff8b9d269b0000 x12: ffff8b9d269af2f3 >[6126851.553359] x11: 0000000000000000 x10: ffff00001e943c80 >[6126851.553846] x9 : ffff59fd7ef3dad0 x8 : 00000000000378b1 >[6126851.554349] x7 : 0000000000000005 x6 : 0000000000000006 >[6126851.554852] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6126851.555367] x3 : 000000000000000b x2 : 000031904045c000 >[6126851.555954] x1 : ffff4a2341035b10 x0 : 000000000000000b >[6126851.556427] Call trace: >[6126851.556763] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6126851.557243] memcg_numa_stat_show+0x1d0/0x260 >[6126851.557668] cgroup_seqfile_show+0x50/0xd0 >[6126851.558045] kernfs_seq_show+0x30/0x38 >[6126851.558387] seq_read+0x148/0x478 >[6126851.558693] kernfs_fop_read+0x2c/0x1e0 >[6126851.559050] __vfs_read+0x20/0x48 >[6126851.559360] vfs_read+0x98/0x168 >[6126851.559664] ksys_read+0x6c/0xe8 >[6126851.559955] __arm64_sys_read+0x20/0x28 >[6126851.560304] el0_svc_common.constprop.0+0xa8/0x200 >[6126851.560757] el0_svc_handler+0x30/0x80 >[6126851.561115] el0_svc+0x10/0x14 >[6126895.531089] watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [kubelet:18847] >[6126895.531949] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6126895.538606] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6126895.544601] CPU: 4 PID: 18847 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6126895.545566] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6126895.546540] pstate: 20000005 (nzCv daif -PAN -UAO) >[6126895.547009] pc : find_next_bit+0x58/0x68 >[6126895.547399] lr : cpumask_next+0x1c/0x28 >[6126895.547780] sp : ffff00001fe3bc10 >[6126895.548134] x29: ffff00001fe3bc10 x28: ffff8bcca13b2000 >[6126895.548650] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6126895.549188] x25: 0000000000000003 x24: 0000000000000000 >[6126895.549677] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6126895.550168] x21: ffff8d9c1b9afc00 x20: ffff59fd7fe4b8a8 >[6126895.550661] x19: 0000000000000000 x18: 0000000000000000 >[6126895.551176] x17: 0000000000000009 x16: 0000000000000000 >[6126895.551674] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6126895.552233] x13: ffff8b7d1b322000 x12: ffff8b7d1b3212fd >[6126895.552771] x11: 0000000000000000 x10: ffff00001fe3bc80 >[6126895.553308] x9 : ffff59fd7ef3dad0 x8 : 0000000000015cd1 >[6126895.553810] x7 : 0000000000000004 x6 : 0000000000000005 >[6126895.554376] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6126895.554885] x3 : 0000000000000028 x2 : 0000000000000068 >[6126895.555391] x1 : 0000000000000400 x0 : ffff59fd7fe4b8a8 >[6126895.555898] Call trace: >[6126895.556146] find_next_bit+0x58/0x68 >[6126895.556595] mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6126895.557074] memcg_numa_stat_show+0x1d0/0x260 >[6126895.557509] cgroup_seqfile_show+0x50/0xd0 >[6126895.557930] kernfs_seq_show+0x30/0x38 >[6126895.558385] seq_read+0x148/0x478 >[6126895.558708] kernfs_fop_read+0x2c/0x1e0 >[6126895.559141] __vfs_read+0x20/0x48 >[6126895.559500] vfs_read+0x98/0x168 >[6126895.559819] ksys_read+0x6c/0xe8 >[6126895.560167] __arm64_sys_read+0x20/0x28 >[6126895.560593] el0_svc_common.constprop.0+0xa8/0x200 >[6126895.561055] el0_svc_handler+0x30/0x80 >[6126895.561417] el0_svc+0x10/0x14 >[6126939.531556] watchdog: BUG: soft lockup - CPU#34 stuck for 23s! [kubelet:57297] >[6126939.532393] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6126939.539468] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6126939.545011] CPU: 34 PID: 57297 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6126939.546070] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6126939.546978] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6126939.547467] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6126939.548022] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6126939.548590] sp : ffff00003f4f3c20 >[6126939.548982] x29: ffff00003f4f3c20 x28: ffff8dac2270a000 >[6126939.549515] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6126939.550060] x25: 0000000000000003 x24: 0000000000000000 >[6126939.550599] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6126939.551248] x21: ffff8dac3047c800 x20: ffff59fd7fe4b8a8 >[6126939.551791] x19: 0000000000000000 x18: 0000000000000000 >[6126939.552344] x17: 0000000000000002 x16: 0000000000000000 >[6126939.552861] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6126939.553364] x13: ffff8bbd32626000 x12: ffff8bbd32625307 >[6126939.553864] x11: 0000000000000000 x10: ffff00003f4f3c80 >[6126939.554396] x9 : ffff59fd7ef3dad0 x8 : 000000000000ca7e >[6126939.554896] x7 : 0000000000000004 x6 : 0000000000000005 >[6126939.555439] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6126939.555951] x3 : 000000000000003e x2 : 000033d040239000 >[6126939.556556] x1 : ffff4a2dcded6070 x0 : 000000000000003e >[6126939.557054] Call trace: >[6126939.557294] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6126939.557775] memcg_numa_stat_show+0x1d0/0x260 >[6126939.558181] cgroup_seqfile_show+0x50/0xd0 >[6126939.558561] kernfs_seq_show+0x30/0x38 >[6126939.558930] seq_read+0x148/0x478 >[6126939.559247] kernfs_fop_read+0x2c/0x1e0 >[6126939.559634] __vfs_read+0x20/0x48 >[6126939.559962] vfs_read+0x98/0x168 >[6126939.560327] ksys_read+0x6c/0xe8 >[6126939.560681] __arm64_sys_read+0x20/0x28 >[6126939.561043] el0_svc_common.constprop.0+0xa8/0x200 >[6126939.561467] el0_svc_handler+0x30/0x80 >[6126939.561897] el0_svc+0x10/0x14 >[6126943.531177] watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [migration/10:62] >[6126943.532066] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6126943.538688] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6126943.544327] CPU: 10 PID: 62 Comm: migration/10 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6126943.545388] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6126943.546432] pstate: 60000005 (nZCv daif -PAN -UAO) >[6126943.546930] pc : multi_cpu_stop+0x78/0x180 >[6126943.547316] lr : multi_cpu_stop+0xb4/0x180 >[6126943.547739] sp : ffff000008ecbd50 >[6126943.548069] x29: ffff000008ecbd50 x28: 0000000000000000 >[6126943.548571] x27: ffff000051c6b938 x26: 0000000000000060 >[6126943.549082] x25: 0000000000000000 x24: 0000000000000000 >[6126943.549589] x23: 0000000000000000 x22: 0000000000000001 >[6126943.550151] x21: ffff000051c6b9d4 x20: ffff000051c6b9b0 >[6126943.550687] x19: 0000000000000001 x18: 0000000000000000 >[6126943.551219] x17: 0000000000000001 x16: 0000000000000000 >[6126943.551716] x15: 0000000000000000 x14: 0000000000000000 >[6126943.552268] x13: 0000000000000000 x12: 0000000000000000 >[6126943.552754] x11: 0000000000000000 x10: 0000000000000d60 >[6126943.553272] x9 : ffff59fd7ede5a0c x8 : ffff8b8d9e907540 >[6126943.553821] x7 : ffff8b8d8059ec00 x6 : 0000000000000304 >[6126943.554329] x5 : 0000319040443000 x4 : ffff8b8d9e906d80 >[6126943.554881] x3 : dead000000000200 x2 : ffff000051c6b960 >[6126943.555421] x1 : ffff000051c6b9d4 x0 : 0000000000000001 >[6126943.556048] Call trace: >[6126943.556298] multi_cpu_stop+0x78/0x180 >[6126943.556725] cpu_stopper_thread+0xec/0x178 >[6126943.557157] smpboot_thread_fn+0x158/0x1d8 >[6126943.557532] kthread+0x130/0x138 >[6126983.531184] watchdog: BUG: soft lockup - CPU#10 stuck for 23s! [kubelet:19592] >[6126983.531987] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6126983.539477] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6126983.545527] CPU: 10 PID: 19592 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6126983.546686] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6126983.547543] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6126983.548003] pc : mem_cgroup_nr_lru_pages.isra.7+0x9c/0xc8 >[6126983.548584] lr : mem_cgroup_nr_lru_pages.isra.7+0xac/0xc8 >[6126983.549102] sp : ffff000030fcbc20 >[6126983.549454] x29: ffff000030fcbc20 x28: 0000000000000000 >[6126983.549970] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6126983.550466] x25: 0000000000000010 x24: 0000000000000000 >[6126983.550948] x23: ffff8b9c402fc5d0 x22: 0000000000000020 >[6126983.551454] x21: ffff59fd7fe49800 x20: ffff59fd7fe4b8a8 >[6126983.551958] x19: 0000000000000004 x18: 0000000000000000 >[6126983.552421] x17: 0000000000000002 x16: 0000000000000000 >[6126983.552928] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6126983.553507] x13: ffff8b8d72d58000 x12: ffff8b8d72d57326 >[6126983.554024] x11: 0000000000000000 x10: ffff000030fcbc80 >[6126983.554540] x9 : ffff59fd7ef3da4c x8 : 000000000000d941 >[6126983.555048] x7 : 0000000000000004 x6 : 0000000000000005 >[6126983.555591] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6126983.556177] x3 : 0000000000000039 x2 : 000033d01fe9e000 >[6126983.556684] x1 : ffff4a1f1196cd90 x0 : 0000000000000079 >[6126983.557214] Call trace: >[6126983.557466] mem_cgroup_nr_lru_pages.isra.7+0x9c/0xc8 >[6126983.557963] memcg_numa_stat_show+0x14c/0x260 >[6126983.558415] cgroup_seqfile_show+0x50/0xd0 >[6126983.558830] kernfs_seq_show+0x30/0x38 >[6126983.559265] seq_read+0x148/0x478 >[6126983.559565] kernfs_fop_read+0x2c/0x1e0 >[6126983.559961] __vfs_read+0x20/0x48 >[6126983.560265] vfs_read+0x98/0x168 >[6126983.560618] ksys_read+0x6c/0xe8 >[6126983.560950] __arm64_sys_read+0x20/0x28 >[6126983.561385] el0_svc_common.constprop.0+0xa8/0x200 >[6126983.561861] el0_svc_handler+0x30/0x80 >[6126983.562227] el0_svc+0x10/0x14 >[6127023.531226] watchdog: BUG: soft lockup - CPU#13 stuck for 23s! [kubelet:35996] >[6127023.532117] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6127023.539141] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6127023.544758] CPU: 13 PID: 35996 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6127023.545842] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6127023.546818] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6127023.547268] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6127023.547780] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6127023.548322] sp : ffff00003290bc20 >[6127023.548683] x29: ffff00003290bc20 x28: ffff8dac076ee000 >[6127023.549189] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6127023.549742] x25: 0000000000000003 x24: 0000000000000000 >[6127023.550220] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6127023.550742] x21: ffff8d9c4583fc00 x20: ffff59fd7fe4b8a8 >[6127023.551277] x19: 0000000000000000 x18: 0000000000000000 >[6127023.551767] x17: 000000000000000a x16: 0000000000000000 >[6127023.552315] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6127023.552817] x13: ffff8b8d73419000 x12: ffff8b8d734182fd >[6127023.553344] x11: 0000000000000000 x10: ffff00003290bc80 >[6127023.553885] x9 : ffff59fd7ef3dad0 x8 : 0000000000014d5c >[6127023.554372] x7 : 0000000000000004 x6 : 0000000000000005 >[6127023.554887] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6127023.555483] x3 : 000000000000001c x2 : 000033b040475000 >[6127023.556067] x1 : ffff4a0cf7e637a0 x0 : 000000000000005c >[6127023.556603] Call trace: >[6127023.556842] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6127023.557312] memcg_numa_stat_show+0x1d0/0x260 >[6127023.557733] cgroup_seqfile_show+0x50/0xd0 >[6127023.558173] kernfs_seq_show+0x30/0x38 >[6127023.558569] seq_read+0x148/0x478 >[6127023.558928] kernfs_fop_read+0x2c/0x1e0 >[6127023.559264] __vfs_read+0x20/0x48 >[6127023.559584] vfs_read+0x98/0x168 >[6127023.559957] ksys_read+0x6c/0xe8 >[6127023.560288] __arm64_sys_read+0x20/0x28 >[6127023.560676] el0_svc_common.constprop.0+0xa8/0x200 >[6127023.561149] el0_svc_handler+0x30/0x80 >[6127023.561526] el0_svc+0x10/0x14 >[6127027.531736] watchdog: BUG: soft lockup - CPU#46 stuck for 23s! [migration/46:246] >[6127027.532706] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6127027.539947] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6127027.545694] CPU: 46 PID: 246 Comm: migration/46 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6127027.546697] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6127027.547509] pstate: 60000005 (nZCv daif -PAN -UAO) >[6127027.547945] pc : multi_cpu_stop+0x78/0x180 >[6127027.548319] lr : multi_cpu_stop+0xb4/0x180 >[6127027.548709] sp : ffff00000a493d50 >[6127027.549038] x29: ffff00000a493d50 x28: 0000000000000000 >[6127027.549524] x27: ffff00001f7eb938 x26: 0000000000000060 >[6127027.550002] x25: 0000000000000000 x24: 0000000000000000 >[6127027.550497] x23: 0000000000000000 x22: 0000000000000001 >[6127027.550978] x21: ffff00001f7eb9d4 x20: ffff00001f7eb9b0 >[6127027.551521] x19: 0000000000000001 x18: 0000000000000000 >[6127027.551977] x17: 0000000000000001 x16: 0000000000000000 >[6127027.552545] x15: 0000000000000000 x14: 0000000000000000 >[6127027.553122] x13: 0000000000000000 x12: 0000000000000000 >[6127027.553696] x11: 0000000000000245 x10: 0000000000000d60 >[6127027.554194] x9 : ffff59fd7ede5a0c x8 : ffff8bcd9e996400 >[6127027.554716] x7 : 00181b6d2c98274e x6 : 00000000000002e8 >[6127027.555238] x5 : 00000000eac0c6e6 x4 : ffff8bcd9e995c40 >[6127027.555806] x3 : dead000000000200 x2 : ffff00001f7eb988 >[6127027.556330] x1 : ffff00001f7eb9d4 x0 : 0000000000000001 >[6127027.556852] Call trace: >[6127027.557080] multi_cpu_stop+0x78/0x180 >[6127027.557437] cpu_stopper_thread+0xec/0x178 >[6127027.557838] smpboot_thread_fn+0x158/0x1d8 >[6127027.558280] kthread+0x130/0x138 >[6127067.531011] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [migration/1:16] >[6127067.531151] watchdog: BUG: soft lockup - CPU#8 stuck for 23s! [kubelet:18923] >[6127067.531881] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6127067.532548] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6127067.539244] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6127067.546014] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6127067.551688] CPU: 1 PID: 16 Comm: migration/1 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6127067.557560] CPU: 8 PID: 18923 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6127067.558579] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6127067.559674] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6127067.560697] pstate: 60000005 (nZCv daif -PAN -UAO) >[6127067.561616] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6127067.562148] pc : multi_cpu_stop+0x78/0x180 >[6127067.562640] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6127067.563037] lr : multi_cpu_stop+0xb4/0x180 >[6127067.563538] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6127067.563540] sp : ffff0000301bbc20 >[6127067.563956] sp : ffff000008d5bd50 >[6127067.564459] x29: ffff0000301bbc20 x28: ffff8b9c15944000 >[6127067.564853] x29: ffff000008d5bd50 x28: 0000000000000000 >[6127067.565279] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6127067.565782] x27: ffff00003f193938 x26: 0000000000000060 >[6127067.566268] x25: 0000000000000003 x24: 0000000000000000 >[6127067.566784] x25: 0000000000000000 x24: 0000000000000000 >[6127067.567286] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6127067.567842] x23: 0000000000000000 x22: 0000000000000001 >[6127067.568364] x21: ffff8dccfd2e6000 x20: ffff59fd7fe4b8a8 >[6127067.568859] x21: ffff00003f1939d4 x20: ffff00003f1939b0 >[6127067.569335] x19: 0000000000000000 x18: 0000000000000000 >[6127067.569827] x19: 0000000000000001 x18: 0000000000000000 >[6127067.570332] x17: 0000000000000002 x16: 0000000000000000 >[6127067.570833] x17: 0000000000000001 x16: 0000000000000000 >[6127067.571350] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6127067.571831] x15: 0000000000000000 x14: 0000000000000000 >[6127067.572377] x13: ffff8b8d72fac000 x12: ffff8b8d72fab31c >[6127067.572880] x13: 0000000000000000 x12: 0000000000000000 >[6127067.573396] x11: 0000000000000000 x10: ffff0000301bbc80 >[6127067.573892] x11: 0000000000000000 x10: 0000000000000d60 >[6127067.574396] x9 : ffff59fd7ef3dad0 x8 : 0000000000018453 >[6127067.574903] x9 : ffff59fd7ede5a0c x8 : ffff8b7d9d9752c0 >[6127067.575526] x7 : 0000000000000004 x6 : 0000000000000005 >[6127067.576034] x7 : 000700f06024b16e x6 : 00000000000002f4 >[6127067.576601] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6127067.577159] x5 : 00000000fa83b2da x4 : ffff8b7d9d974b00 >[6127067.577713] x3 : 0000000000000017 x2 : 000031a0404c0000 >[6127067.578307] x3 : dead000000000200 x2 : ffff00003f193960 >[6127067.578873] x1 : ffff4a0f1365db90 x0 : 0000000000000017 >[6127067.579435] x1 : ffff00003f1939d4 x0 : 0000000000000001 >[6127067.580031] Call trace: >[6127067.580588] Call trace: >[6127067.581187] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6127067.581456] multi_cpu_stop+0x78/0x180 >[6127067.581752] memcg_numa_stat_show+0x1d0/0x260 >[6127067.582231] cpu_stopper_thread+0xec/0x178 >[6127067.582597] cgroup_seqfile_show+0x50/0xd0 >[6127067.583070] smpboot_thread_fn+0x158/0x1d8 >[6127067.583465] kernfs_seq_show+0x30/0x38 >[6127067.583875] kthread+0x130/0x138 >[6127067.584277] seq_read+0x148/0x478 >[6127067.585366] kernfs_fop_read+0x2c/0x1e0 >[6127067.585771] __vfs_read+0x20/0x48 >[6127067.586098] vfs_read+0x98/0x168 >[6127067.586472] ksys_read+0x6c/0xe8 >[6127067.586793] __arm64_sys_read+0x20/0x28 >[6127067.587257] el0_svc_common.constprop.0+0xa8/0x200 >[6127067.587727] el0_svc_handler+0x30/0x80 >[6127067.588099] el0_svc+0x10/0x14 >[6127111.531121] watchdog: BUG: soft lockup - CPU#6 stuck for 23s! [kubelet:18923] >[6127111.531207] watchdog: BUG: soft lockup - CPU#12 stuck for 23s! [migration/12:72] >[6127111.531928] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6127111.532672] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6127111.539371] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6127111.545986] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6127111.551869] CPU: 6 PID: 18923 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6127111.557722] CPU: 12 PID: 72 Comm: migration/12 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6127111.558696] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6127111.559696] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6127111.560589] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6127111.561464] pstate: 60000005 (nZCv daif -PAN -UAO) >[6127111.561931] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6127111.562388] pc : multi_cpu_stop+0xf4/0x180 >[6127111.562884] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6127111.563292] lr : multi_cpu_stop+0xb4/0x180 >[6127111.563843] sp : ffff0000301bbc20 >[6127111.564243] sp : ffff000008f1bd50 >[6127111.564245] x29: ffff000008f1bd50 x28: 0000000000000000 >[6127111.564608] x29: ffff0000301bbc20 x28: ffff8bcc6aa34000 >[6127111.565036] x27: ffff000060753938 x26: 0000000000000060 >[6127111.565569] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6127111.566151] x25: 0000000000000000 x24: 0000000000000000 >[6127111.566754] x25: 0000000000000010 x24: 0000000000000000 >[6127111.567248] x23: 0000000000000000 x22: 0000000000000001 >[6127111.567792] x23: 0000000000000020 x22: ffff59fd7fe49800 >[6127111.568311] x21: ffff0000607539d4 x20: ffff0000607539b0 >[6127111.568869] x21: ffff8b7b99e1e000 x20: ffff59fd7fe4b8a8 >[6127111.569433] x19: 0000000000000001 x18: 0000000000000000 >[6127111.570020] x19: 0000000000000004 x18: 0000000000000000 >[6127111.570554] x17: 0000000000000001 x16: 0000000000000000 >[6127111.571104] x17: 0000000000000002 x16: 0000000000000000 >[6127111.571622] x15: 0000000000000000 x14: 0000000000000000 >[6127111.572168] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6127111.572727] x13: 0000000000000000 x12: 0000000000000000 >[6127111.573233] x13: ffff8b7d1c80d000 x12: ffff8b7d1c80c345 >[6127111.573728] x11: 00000000000000b6 x10: 0000000000000d60 >[6127111.574309] x11: 0000000000000000 x10: ffff0000301bbc80 >[6127111.574973] x9 : ffff59fd7ede5a0c x8 : ffff8b8d9e953040 >[6127111.575535] x9 : ffff59fd7ef3dad0 x8 : 0000000000000021 >[6127111.576142] x7 : ffff8b8d7c5a2c00 x6 : 0000000000000026 >[6127111.576713] x7 : 0000000000000001 x6 : 0000000000000002 >[6127111.577288] x5 : 0000319040475000 x4 : ffff8b8d9e952880 >[6127111.577902] x5 : 00000000fffffffa x4 : ffffffffffffffff >[6127111.578450] x3 : dead000000000200 x2 : ffff000060753988 >[6127111.579002] x3 : 0000000000000031 x2 : 000033d01ff66000 >[6127111.579620] x1 : ffff0000607539d4 x0 : 0000000000000001 >[6127111.580143] x1 : ffff4a2ddd291b10 x0 : 0000000000000071 >[6127111.580780] Call trace: >[6127111.581323] Call trace: >[6127111.581843] multi_cpu_stop+0xf4/0x180 >[6127111.582098] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6127111.582338] cpu_stopper_thread+0xec/0x178 >[6127111.582771] memcg_numa_stat_show+0x1d0/0x260 >[6127111.583247] smpboot_thread_fn+0x158/0x1d8 >[6127111.583684] cgroup_seqfile_show+0x50/0xd0 >[6127111.584130] kthread+0x130/0x138 >[6127111.584568] kernfs_seq_show+0x30/0x38 >[6127111.585768] seq_read+0x148/0x478 >[6127111.586103] kernfs_fop_read+0x2c/0x1e0 >[6127111.586474] __vfs_read+0x20/0x48 >[6127111.586853] vfs_read+0x98/0x168 >[6127111.587182] ksys_read+0x6c/0xe8 >[6127111.587521] __arm64_sys_read+0x20/0x28 >[6127111.587898] el0_svc_common.constprop.0+0xa8/0x200 >[6127111.588387] el0_svc_handler+0x30/0x80 >[6127111.588783] el0_svc+0x10/0x14 >[6127151.531183] watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [kubelet:55652] >[6127151.532005] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6127151.538915] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6127151.544853] CPU: 10 PID: 55652 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6127151.545897] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6127151.546831] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6127151.547329] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6127151.547842] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6127151.548378] sp : ffff000056e73c20 >[6127151.548795] x29: ffff000056e73c20 x28: ffff8b7bbd4dc000 >[6127151.549331] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6127151.549839] x25: 0000000000000003 x24: 0000000000000000 >[6127151.550328] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6127151.550835] x21: ffff8dac54518800 x20: ffff59fd7fe4b8a8 >[6127151.551358] x19: 0000000000000000 x18: 0000000000000000 >[6127151.551876] x17: 0000000000000009 x16: 0000000000000000 >[6127151.552375] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6127151.552892] x13: ffff8b8d72d58000 x12: ffff8b8d72d57307 >[6127151.553399] x11: 0000000000000000 x10: ffff000056e73c80 >[6127151.553895] x9 : ffff59fd7ef3dad0 x8 : 000000000000c18e >[6127151.554370] x7 : 0000000000000004 x6 : 0000000000000005 >[6127151.554990] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6127151.555519] x3 : 0000000000000037 x2 : 000033d01fffc000 >[6127151.556021] x1 : ffff4a0f1b2a69c0 x0 : 0000000000000077 >[6127151.556547] Call trace: >[6127151.556820] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6127151.557295] memcg_numa_stat_show+0x1d0/0x260 >[6127151.557764] cgroup_seqfile_show+0x50/0xd0 >[6127151.558225] kernfs_seq_show+0x30/0x38 >[6127151.558587] seq_read+0x148/0x478 >[6127151.558971] kernfs_fop_read+0x2c/0x1e0 >[6127151.559387] __vfs_read+0x20/0x48 >[6127151.559761] vfs_read+0x98/0x168 >[6127151.560139] ksys_read+0x6c/0xe8 >[6127151.560472] __arm64_sys_read+0x20/0x28 >[6127151.560916] el0_svc_common.constprop.0+0xa8/0x200 >[6127151.561396] el0_svc_handler+0x30/0x80 >[6127151.561802] el0_svc+0x10/0x14 >[6127195.531633] watchdog: BUG: soft lockup - CPU#39 stuck for 22s! [kubelet:11266] >[6127195.532547] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6127195.539256] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6127195.544928] CPU: 39 PID: 11266 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6127195.545969] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6127195.546867] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6127195.547391] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6127195.547873] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6127195.548328] sp : ffff0000545dbc20 >[6127195.548705] x29: ffff0000545dbc20 x28: ffff8b7b97f94000 >[6127195.549189] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6127195.549769] x25: 0000000000000003 x24: 0000000000000000 >[6127195.550254] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6127195.550826] x21: ffff8dbb850c8800 x20: ffff59fd7fe4b8a8 >[6127195.551339] x19: 0000000000000000 x18: 0000000000000000 >[6127195.551893] x17: 0000000000000002 x16: 0000000000000000 >[6127195.552435] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6127195.552941] x13: ffff8bbd2cd01000 x12: ffff8bbd2cd00311 >[6127195.553495] x11: 0000000000000000 x10: ffff0000545dbc80 >[6127195.553993] x9 : ffff59fd7ef3dad0 x8 : 000000000004ed52 >[6127195.554612] x7 : 0000000000000005 x6 : 0000000000000006 >[6127195.555122] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6127195.555622] x3 : 0000000000000019 x2 : 000031b04042a000 >[6127195.556130] x1 : ffff4a0c63485bb0 x0 : 0000000000000019 >[6127195.556644] Call trace: >[6127195.556947] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6127195.557420] memcg_numa_stat_show+0x1d0/0x260 >[6127195.557835] cgroup_seqfile_show+0x50/0xd0 >[6127195.558257] kernfs_seq_show+0x30/0x38 >[6127195.558631] seq_read+0x148/0x478 >[6127195.558962] kernfs_fop_read+0x2c/0x1e0 >[6127195.559351] __vfs_read+0x20/0x48 >[6127195.559675] vfs_read+0x98/0x168 >[6127195.559979] ksys_read+0x6c/0xe8 >[6127195.560297] __arm64_sys_read+0x20/0x28 >[6127195.560687] el0_svc_common.constprop.0+0xa8/0x200 >[6127195.561147] el0_svc_handler+0x30/0x80 >[6127195.561509] el0_svc+0x10/0x14 >[6127239.531310] watchdog: BUG: soft lockup - CPU#18 stuck for 22s! [kubelet:18847] >[6127239.532099] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6127239.539254] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6127239.544918] CPU: 18 PID: 18847 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6127239.545841] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6127239.546661] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6127239.547109] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6127239.547590] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6127239.548080] sp : ffff00001fe3bc20 >[6127239.548415] x29: ffff00001fe3bc20 x28: ffff8d8cda198000 >[6127239.548918] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6127239.549419] x25: 0000000000000003 x24: 0000000000000000 >[6127239.549917] x23: 0000000000000008 x22: ffff59fd7fe49800 >[6127239.550394] x21: ffff8dcce1265c00 x20: ffff59fd7fe4b8a8 >[6127239.550919] x19: 0000000000000001 x18: 0000000000000000 >[6127239.551426] x17: 0000000000000002 x16: 0000000000000000 >[6127239.551917] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6127239.552431] x13: ffff8b9d205f7000 x12: ffff8b9d205f631c >[6127239.552935] x11: 0000000000000000 x10: ffff00001fe3bc80 >[6127239.553422] x9 : ffff59fd7ef3dad0 x8 : 000000000001871d >[6127239.554022] x7 : 0000000000000005 x6 : 0000000000000006 >[6127239.554536] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6127239.555016] x3 : 0000000000000030 x2 : 000033d04026b000 >[6127239.555511] x1 : ffff4a1dd73555f8 x0 : 0000000000000030 >[6127239.555996] Call trace: >[6127239.556234] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6127239.556695] memcg_numa_stat_show+0x1d0/0x260 >[6127239.557119] cgroup_seqfile_show+0x50/0xd0 >[6127239.557518] kernfs_seq_show+0x30/0x38 >[6127239.557879] seq_read+0x148/0x478 >[6127239.558276] kernfs_fop_read+0x2c/0x1e0 >[6127239.558640] __vfs_read+0x20/0x48 >[6127239.559049] vfs_read+0x98/0x168 >[6127239.559370] ksys_read+0x6c/0xe8 >[6127239.559747] __arm64_sys_read+0x20/0x28 >[6127239.560141] el0_svc_common.constprop.0+0xa8/0x200 >[6127239.560626] el0_svc_handler+0x30/0x80 >[6127239.560978] el0_svc+0x10/0x14 >[6127283.531167] watchdog: BUG: soft lockup - CPU#9 stuck for 22s! [kubelet:18334] >[6127283.532032] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6127283.538717] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6127283.544575] CPU: 9 PID: 18334 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6127283.545555] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6127283.546418] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6127283.546880] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6127283.547384] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6127283.547880] sp : ffff00001f9a3c20 >[6127283.548210] x29: ffff00001f9a3c20 x28: ffff8bbc553f8000 >[6127283.548743] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6127283.549275] x25: 0000000000000010 x24: 0000000000000000 >[6127283.549781] x23: 0000000000000020 x22: ffff59fd7fe49800 >[6127283.550249] x21: ffff8b8c733db400 x20: ffff59fd7fe4b8a8 >[6127283.550727] x19: 0000000000000004 x18: 0000000000000000 >[6127283.551228] x17: 0000000000000002 x16: 0000000000000000 >[6127283.551709] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6127283.552205] x13: ffff8b9d205bc000 x12: ffff8b9d205bb34b >[6127283.552722] x11: 0000000000000000 x10: ffff00001f9a3c80 >[6127283.553218] x9 : ffff59fd7ef3dad0 x8 : ffff8b9d205bb000 >[6127283.553824] x7 : 0000000000000000 x6 : 0000000000000001 >[6127283.554361] x5 : 00000000fffffffb x4 : ffffffffffffffff >[6127283.554908] x3 : 0000000000000020 x2 : 000033c040410000 >[6127283.555467] x1 : ffff4a1f182d7180 x0 : 0000000000000060 >[6127283.555976] Call trace: >[6127283.556273] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6127283.556811] memcg_numa_stat_show+0x1d0/0x260 >[6127283.557283] cgroup_seqfile_show+0x50/0xd0 >[6127283.557747] kernfs_seq_show+0x30/0x38 >[6127283.558137] seq_read+0x148/0x478 >[6127283.558492] kernfs_fop_read+0x2c/0x1e0 >[6127283.558880] __vfs_read+0x20/0x48 >[6127283.559244] vfs_read+0x98/0x168 >[6127283.559560] ksys_read+0x6c/0xe8 >[6127283.559864] __arm64_sys_read+0x20/0x28 >[6127283.560241] el0_svc_common.constprop.0+0xa8/0x200 >[6127283.560694] el0_svc_handler+0x30/0x80 >[6127283.561073] el0_svc+0x10/0x14 >[6127323.531496] watchdog: BUG: soft lockup - CPU#30 stuck for 22s! [kubelet:18322] >[6127323.532312] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6127323.539360] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6127323.545031] CPU: 30 PID: 18322 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6127323.545983] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6127323.546813] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6127323.547284] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6127323.547818] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6127323.548285] sp : ffff00001f8dbc20 >[6127323.548614] x29: ffff00001f8dbc20 x28: ffff8babf1a9c000 >[6127323.549121] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6127323.549620] x25: 0000000000000003 x24: 0000000000000000 >[6127323.550109] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6127323.550631] x21: ffff8dabc22e7000 x20: ffff59fd7fe4b8a8 >[6127323.551182] x19: 0000000000000000 x18: 0000000000000000 >[6127323.551670] x17: 000000000000064a x16: 0000000000000000 >[6127323.552223] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6127323.552744] x13: ffff8bac181e2000 x12: ffff8bac181e1307 >[6127323.553235] x11: 0000000000000000 x10: ffff00001f8dbc80 >[6127323.553797] x9 : ffff59fd7ef3dad0 x8 : 000000000000bf49 >[6127323.554276] x7 : 0000000000000004 x6 : 0000000000000005 >[6127323.554816] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6127323.555321] x3 : 000000000000002c x2 : 000033d020079000 >[6127323.555850] x1 : ffff4a1dccce57c0 x0 : 000000000000006c >[6127323.556361] Call trace: >[6127323.556618] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6127323.557096] memcg_numa_stat_show+0x1d0/0x260 >[6127323.557520] cgroup_seqfile_show+0x50/0xd0 >[6127323.557923] kernfs_seq_show+0x30/0x38 >[6127323.558285] seq_read+0x148/0x478 >[6127323.558600] kernfs_fop_read+0x2c/0x1e0 >[6127323.558994] __vfs_read+0x20/0x48 >[6127323.559306] vfs_read+0x98/0x168 >[6127323.559615] ksys_read+0x6c/0xe8 >[6127323.559926] __arm64_sys_read+0x20/0x28 >[6127323.560300] el0_svc_common.constprop.0+0xa8/0x200 >[6127323.560815] el0_svc_handler+0x30/0x80 >[6127323.561208] el0_svc+0x10/0x14 >[6127367.531106] watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [kubelet:57297] >[6127367.531965] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6127367.539128] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6127367.545083] CPU: 5 PID: 57297 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6127367.546064] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6127367.546925] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6127367.547389] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6127367.547900] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6127367.548397] sp : ffff00003f4f3c20 >[6127367.548775] x29: ffff00003f4f3c20 x28: ffff8dac12964000 >[6127367.549287] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6127367.549853] x25: 0000000000000003 x24: 0000000000000000 >[6127367.550390] x23: 0000000000000008 x22: ffff59fd7fe49800 >[6127367.550976] x21: ffff8dcce88a0400 x20: ffff59fd7fe4b8a8 >[6127367.551485] x19: 0000000000000001 x18: 0000000000000000 >[6127367.551993] x17: 0000000000000002 x16: 0000000000000000 >[6127367.552513] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6127367.553170] x13: ffff8b7b9f098000 x12: ffff8b7b9f09731c >[6127367.553718] x11: 0000000000000000 x10: ffff00003f4f3c80 >[6127367.554218] x9 : ffff59fd7ef3dad0 x8 : 00000000000186b0 >[6127367.554757] x7 : 0000000000000005 x6 : 0000000000000006 >[6127367.555246] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6127367.555736] x3 : 0000000000000009 x2 : 000033904042a000 >[6127367.556253] x1 : ffff4a2dc5dec6a8 x0 : 0000000000000049 >[6127367.556764] Call trace: >[6127367.557014] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6127367.557526] memcg_numa_stat_show+0x1d0/0x260 >[6127367.557994] cgroup_seqfile_show+0x50/0xd0 >[6127367.558422] kernfs_seq_show+0x30/0x38 >[6127367.558821] seq_read+0x148/0x478 >[6127367.559137] kernfs_fop_read+0x2c/0x1e0 >[6127367.559486] __vfs_read+0x20/0x48 >[6127367.559801] vfs_read+0x98/0x168 >[6127367.560116] ksys_read+0x6c/0xe8 >[6127367.560423] __arm64_sys_read+0x20/0x28 >[6127367.560816] el0_svc_common.constprop.0+0xa8/0x200 >[6127367.561270] el0_svc_handler+0x30/0x80 >[6127367.561629] el0_svc+0x10/0x14 >[6127411.531089] watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [kubelet:7308] >[6127411.531893] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6127411.539119] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6127411.545038] CPU: 4 PID: 7308 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6127411.546007] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6127411.546866] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6127411.547321] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6127411.547826] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6127411.548343] sp : ffff00001b39bc20 >[6127411.548698] x29: ffff00001b39bc20 x28: ffff8dac2282e000 >[6127411.549204] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6127411.549773] x25: 0000000000000010 x24: 0000000000000000 >[6127411.550301] x23: 0000000000000020 x22: ffff59fd7fe49800 >[6127411.550937] x21: ffff8b8be11c6800 x20: ffff59fd7fe4b8a8 >[6127411.551503] x19: 0000000000000004 x18: 0000000000000000 >[6127411.552042] x17: 000000000000057f x16: 0000000000000000 >[6127411.552597] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6127411.553194] x13: ffff8b7d1b321000 x12: ffff8b7d1b32034b >[6127411.553725] x11: 0000000000000000 x10: ffff00001b39bc80 >[6127411.554274] x9 : ffff59fd7ef3dad0 x8 : ffff8b7d1b320000 >[6127411.554778] x7 : 0000000000000000 x6 : 0000000000000001 >[6127411.555322] x5 : 00000000fffffffb x4 : ffffffffffffffff >[6127411.555874] x3 : 0000000000000007 x2 : 00003180404be000 >[6127411.556394] x1 : ffff4a2dc674bb90 x0 : 0000000000000007 >[6127411.556925] Call trace: >[6127411.557175] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6127411.557658] memcg_numa_stat_show+0x1d0/0x260 >[6127411.558087] cgroup_seqfile_show+0x50/0xd0 >[6127411.558486] kernfs_seq_show+0x30/0x38 >[6127411.558860] seq_read+0x148/0x478 >[6127411.559180] kernfs_fop_read+0x2c/0x1e0 >[6127411.559570] __vfs_read+0x20/0x48 >[6127411.559904] vfs_read+0x98/0x168 >[6127411.560217] ksys_read+0x6c/0xe8 >[6127411.560553] __arm64_sys_read+0x20/0x28 >[6127411.560934] el0_svc_common.constprop.0+0xa8/0x200 >[6127411.561399] el0_svc_handler+0x30/0x80 >[6127411.561774] el0_svc+0x10/0x14 >[6127451.531088] watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [kubelet:18294] >[6127451.531935] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6127451.538937] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6127451.545032] CPU: 4 PID: 18294 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6127451.546115] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6127451.547056] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6127451.547526] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6127451.548046] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6127451.548541] sp : ffff00001f803c20 >[6127451.548859] x29: ffff00001f803c20 x28: ffff8b7be583c000 >[6127451.549409] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6127451.549937] x25: 0000000000000003 x24: 0000000000000000 >[6127451.550482] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6127451.551013] x21: ffff8d9cde73f800 x20: ffff59fd7fe4b8a8 >[6127451.551533] x19: 0000000000000000 x18: 0000000000000000 >[6127451.552039] x17: 0000000000000002 x16: 0000000000000000 >[6127451.552686] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6127451.553238] x13: ffff8b7d1b321000 x12: ffff8b7d1b3202fd >[6127451.553713] x11: 0000000000000000 x10: ffff00001f803c80 >[6127451.554264] x9 : ffff59fd7ef3dad0 x8 : 0000000000014e1f >[6127451.554781] x7 : 0000000000000004 x6 : 0000000000000005 >[6127451.555296] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6127451.555789] x3 : 0000000000000023 x2 : 000033c04045b000 >[6127451.556295] x1 : ffff4a1d4497b0a0 x0 : 0000000000000063 >[6127451.556808] Call trace: >[6127451.557047] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6127451.557526] memcg_numa_stat_show+0x1d0/0x260 >[6127451.557946] cgroup_seqfile_show+0x50/0xd0 >[6127451.558341] kernfs_seq_show+0x30/0x38 >[6127451.558696] seq_read+0x148/0x478 >[6127451.559023] kernfs_fop_read+0x2c/0x1e0 >[6127451.559401] __vfs_read+0x20/0x48 >[6127451.559800] vfs_read+0x98/0x168 >[6127451.560135] ksys_read+0x6c/0xe8 >[6127451.560470] __arm64_sys_read+0x20/0x28 >[6127451.560907] el0_svc_common.constprop.0+0xa8/0x200 >[6127451.561401] el0_svc_handler+0x30/0x80 >[6127451.561811] el0_svc+0x10/0x14 >[6127495.531755] watchdog: BUG: soft lockup - CPU#47 stuck for 22s! [kubelet:18888] >[6127495.532736] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6127495.539865] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6127495.545535] CPU: 47 PID: 18888 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6127495.546560] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6127495.547487] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6127495.547965] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6127495.548474] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6127495.548967] sp : ffff00001fec3c20 >[6127495.549294] x29: ffff00001fec3c20 x28: ffff8bcc3adb6000 >[6127495.549789] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6127495.550303] x25: 0000000000000003 x24: 0000000000000000 >[6127495.550830] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6127495.551338] x21: ffff8dbbc8cc0800 x20: ffff59fd7fe4b8a8 >[6127495.551801] x19: 0000000000000000 x18: 0000000000000000 >[6127495.552380] x17: 0000000000000002 x16: 0000000000000000 >[6127495.552894] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6127495.553362] x13: ffff8bcd52fd9000 x12: ffff8bcd52fd8311 >[6127495.553836] x11: 0000000000000000 x10: ffff00001fec3c80 >[6127495.554342] x9 : ffff59fd7ef3dad0 x8 : 000000000004edab >[6127495.554919] x7 : 0000000000000005 x6 : 0000000000000006 >[6127495.555425] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6127495.555925] x3 : 000000000000001c x2 : 000031b040475000 >[6127495.556411] x1 : ffff4a1dd9683210 x0 : 000000000000001c >[6127495.556929] Call trace: >[6127495.557198] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6127495.557666] memcg_numa_stat_show+0x1d0/0x260 >[6127495.558086] cgroup_seqfile_show+0x50/0xd0 >[6127495.558469] kernfs_seq_show+0x30/0x38 >[6127495.558852] seq_read+0x148/0x478 >[6127495.559254] kernfs_fop_read+0x2c/0x1e0 >[6127495.559607] __vfs_read+0x20/0x48 >[6127495.559916] vfs_read+0x98/0x168 >[6127495.560216] ksys_read+0x6c/0xe8 >[6127495.560524] __arm64_sys_read+0x20/0x28 >[6127495.560911] el0_svc_common.constprop.0+0xa8/0x200 >[6127495.561336] el0_svc_handler+0x30/0x80 >[6127495.561666] el0_svc+0x10/0x14 >[6127539.531372] watchdog: BUG: soft lockup - CPU#22 stuck for 22s! [kubelet:18356] >[6127539.532246] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6127539.539968] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6127539.546494] CPU: 22 PID: 18356 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6127539.547609] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6127539.548515] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6127539.549066] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6127539.549595] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6127539.550137] sp : ffff00001edd3c20 >[6127539.550500] x29: ffff00001edd3c20 x28: ffff8bac93db0000 >[6127539.550992] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6127539.551599] x25: 0000000000000003 x24: 0000000000000000 >[6127539.552255] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6127539.552848] x21: ffff8dcd2a01ac00 x20: ffff59fd7fe4b8a8 >[6127539.553433] x19: 0000000000000000 x18: 0000000000000000 >[6127539.553981] x17: 0000000000000002 x16: 0000000000000000 >[6127539.554511] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6127539.555043] x13: ffff8b9d2055b000 x12: ffff8b9d2055a31c >[6127539.555524] x11: 0000000000000000 x10: ffff00001edd3c80 >[6127539.556002] x9 : ffff59fd7ef3dad0 x8 : 00000000000181ef >[6127539.556490] x7 : 0000000000000004 x6 : 0000000000000005 >[6127539.557023] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6127539.557502] x3 : 0000000000000013 x2 : 000033a04045c000 >[6127539.558020] x1 : ffff4a2dcea87eb0 x0 : 0000000000000053 >[6127539.558518] Call trace: >[6127539.558831] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6127539.559296] memcg_numa_stat_show+0x1d0/0x260 >[6127539.559700] cgroup_seqfile_show+0x50/0xd0 >[6127539.560088] kernfs_seq_show+0x30/0x38 >[6127539.560462] seq_read+0x148/0x478 >[6127539.560764] kernfs_fop_read+0x2c/0x1e0 >[6127539.561127] __vfs_read+0x20/0x48 >[6127539.561450] vfs_read+0x98/0x168 >[6127539.561863] ksys_read+0x6c/0xe8 >[6127539.562222] __arm64_sys_read+0x20/0x28 >[6127539.562635] el0_svc_common.constprop.0+0xa8/0x200 >[6127539.563098] el0_svc_handler+0x30/0x80 >[6127539.563450] el0_svc+0x10/0x14 >[6127583.531182] watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [kubelet:18891] >[6127583.532194] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6127583.538724] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6127583.544520] CPU: 10 PID: 18891 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6127583.545583] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6127583.546484] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6127583.546941] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6127583.547465] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6127583.548004] sp : ffff0000302fbc20 >[6127583.548321] x29: ffff0000302fbc20 x28: ffff8b9b7a22a000 >[6127583.548845] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6127583.549401] x25: 0000000000000010 x24: 0000000000000000 >[6127583.549925] x23: 0000000000000020 x22: ffff59fd7fe49800 >[6127583.550511] x21: ffff8b8bae806800 x20: ffff59fd7fe4b8a8 >[6127583.551000] x19: 0000000000000004 x18: 0000000000000000 >[6127583.551672] x17: 0000000000000002 x16: 0000000000000000 >[6127583.552162] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6127583.552764] x13: ffff8b8d72d57000 x12: ffff8b8d72d5634b >[6127583.553265] x11: 0000000000000000 x10: ffff0000302fbc80 >[6127583.553893] x9 : ffff59fd7ef3dad0 x8 : ffff8b8d72d56000 >[6127583.554503] x7 : 0000000000000000 x6 : 0000000000000001 >[6127583.554978] x5 : 00000000fffffffb x4 : ffffffffffffffff >[6127583.555593] x3 : 000000000000001d x2 : 000033b04048e000 >[6127583.556062] x1 : ffff4a0cfd9c7fa0 x0 : 000000000000005d >[6127583.556571] Call trace: >[6127583.556868] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6127583.557386] memcg_numa_stat_show+0x1d0/0x260 >[6127583.557854] cgroup_seqfile_show+0x50/0xd0 >[6127583.558281] kernfs_seq_show+0x30/0x38 >[6127583.558638] seq_read+0x148/0x478 >[6127583.559001] kernfs_fop_read+0x2c/0x1e0 >[6127583.559358] __vfs_read+0x20/0x48 >[6127583.559669] vfs_read+0x98/0x168 >[6127583.559976] ksys_read+0x6c/0xe8 >[6127583.560282] __arm64_sys_read+0x20/0x28 >[6127583.560657] el0_svc_common.constprop.0+0xa8/0x200 >[6127583.561094] el0_svc_handler+0x30/0x80 >[6127583.561553] el0_svc+0x10/0x14 >[6127623.531340] watchdog: BUG: soft lockup - CPU#20 stuck for 23s! [kubelet:18288] >[6127623.532084] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6127623.538503] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6127623.544575] CPU: 20 PID: 18288 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6127623.545529] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6127623.546316] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6127623.546746] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6127623.547231] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6127623.547706] sp : ffff00001f91bc20 >[6127623.548016] x29: ffff00001f91bc20 x28: ffff8b7b67174000 >[6127623.548480] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6127623.548953] x25: 0000000000000003 x24: 0000000000000000 >[6127623.549435] x23: 0000000000000008 x22: ffff59fd7fe49800 >[6127623.549928] x21: ffff8d9c43bb7c00 x20: ffff59fd7fe4b8a8 >[6127623.550401] x19: 0000000000000001 x18: 0000000000000000 >[6127623.550887] x17: 0000000000000652 x16: 0000000000000000 >[6127623.551460] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6127623.551979] x13: ffff8b9b3117c000 x12: ffff8b9b3117b2fd >[6127623.552470] x11: 0000000000000000 x10: ffff00001f91bc80 >[6127623.553042] x9 : ffff59fd7ef3dad0 x8 : 000000000001506e >[6127623.553542] x7 : 0000000000000004 x6 : 0000000000000005 >[6127623.554038] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6127623.554508] x3 : 000000000000003e x2 : 000033d01ff1b000 >[6127623.554979] x1 : ffff4a0ceef72ce8 x0 : 000000000000007e >[6127623.555473] Call trace: >[6127623.555717] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6127623.556165] memcg_numa_stat_show+0x1d0/0x260 >[6127623.556587] cgroup_seqfile_show+0x50/0xd0 >[6127623.556965] kernfs_seq_show+0x30/0x38 >[6127623.557319] seq_read+0x148/0x478 >[6127623.557630] kernfs_fop_read+0x2c/0x1e0 >[6127623.557993] __vfs_read+0x20/0x48 >[6127623.558307] vfs_read+0x98/0x168 >[6127623.558632] ksys_read+0x6c/0xe8 >[6127623.558933] __arm64_sys_read+0x20/0x28 >[6127623.559309] el0_svc_common.constprop.0+0xa8/0x200 >[6127623.559800] el0_svc_handler+0x30/0x80 >[6127623.560149] el0_svc+0x10/0x14 >[6127667.531779] watchdog: BUG: soft lockup - CPU#48 stuck for 23s! [kubelet:18288] >[6127667.532662] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6127667.539237] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6127667.544797] CPU: 48 PID: 18288 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6127667.545802] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6127667.546617] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6127667.547079] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6127667.547558] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6127667.548051] sp : ffff00001f91bc20 >[6127667.548368] x29: ffff00001f91bc20 x28: ffff8bacdeefa000 >[6127667.548845] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6127667.549321] x25: 0000000000000003 x24: 0000000000000000 >[6127667.549796] x23: 0000000000000008 x22: ffff59fd7fe49800 >[6127667.550265] x21: ffff8d9c6911fc00 x20: ffff59fd7fe4b8a8 >[6127667.550813] x19: 0000000000000001 x18: 0000000000000000 >[6127667.551367] x17: 0000000000000658 x16: 0000000000000000 >[6127667.551852] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6127667.552346] x13: ffff8b9d2060c000 x12: ffff8b9d2060b2fd >[6127667.552861] x11: 0000000000000000 x10: ffff00001f91bc80 >[6127667.553350] x9 : ffff59fd7ef3dad0 x8 : 0000000000014e40 >[6127667.553833] x7 : 0000000000000004 x6 : 0000000000000005 >[6127667.554318] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6127667.554828] x3 : 0000000000000008 x2 : 0000319040411000 >[6127667.555317] x1 : ffff4a2dc93db918 x0 : 0000000000000008 >[6127667.555796] Call trace: >[6127667.556037] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6127667.556518] memcg_numa_stat_show+0x1d0/0x260 >[6127667.556929] cgroup_seqfile_show+0x50/0xd0 >[6127667.557325] kernfs_seq_show+0x30/0x38 >[6127667.557683] seq_read+0x148/0x478 >[6127667.558007] kernfs_fop_read+0x2c/0x1e0 >[6127667.558367] __vfs_read+0x20/0x48 >[6127667.558677] vfs_read+0x98/0x168 >[6127667.559006] ksys_read+0x6c/0xe8 >[6127667.559324] __arm64_sys_read+0x20/0x28 >[6127667.559698] el0_svc_common.constprop.0+0xa8/0x200 >[6127667.560147] el0_svc_handler+0x30/0x80 >[6127667.560497] el0_svc+0x10/0x14 >[6127711.531260] watchdog: BUG: soft lockup - CPU#15 stuck for 23s! [kubelet:7308] >[6127711.532129] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6127711.539114] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6127711.544958] CPU: 15 PID: 7308 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6127711.545977] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6127711.546869] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6127711.547390] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6127711.547936] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6127711.548453] sp : ffff00001b39bc20 >[6127711.548824] x29: ffff00001b39bc20 x28: ffff8bcc3a8c2000 >[6127711.549367] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6127711.549864] x25: 0000000000000003 x24: 0000000000000000 >[6127711.550399] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6127711.550981] x21: ffff8dbba7747c00 x20: ffff59fd7fe4b8a8 >[6127711.551476] x19: 0000000000000000 x18: 0000000000000000 >[6127711.552025] x17: 0000000000000009 x16: 0000000000000000 >[6127711.552533] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6127711.553040] x13: ffff8b8bdf96c000 x12: ffff8b8bdf96b311 >[6127711.553604] x11: 0000000000000000 x10: ffff00001b39bc80 >[6127711.554135] x9 : ffff59fd7ef3dad0 x8 : 000000000004ed91 >[6127711.554664] x7 : 0000000000000005 x6 : 0000000000000006 >[6127711.555172] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6127711.555692] x3 : 0000000000000008 x2 : 0000339040411000 >[6127711.556212] x1 : ffff4a0e44a38280 x0 : 0000000000000048 >[6127711.556748] Call trace: >[6127711.556999] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6127711.557490] memcg_numa_stat_show+0x1d0/0x260 >[6127711.557901] cgroup_seqfile_show+0x50/0xd0 >[6127711.558288] kernfs_seq_show+0x30/0x38 >[6127711.558656] seq_read+0x148/0x478 >[6127711.558991] kernfs_fop_read+0x2c/0x1e0 >[6127711.559380] __vfs_read+0x20/0x48 >[6127711.559718] vfs_read+0x98/0x168 >[6127711.560052] ksys_read+0x6c/0xe8 >[6127711.560383] __arm64_sys_read+0x20/0x28 >[6127711.560854] el0_svc_common.constprop.0+0xa8/0x200 >[6127711.561430] el0_svc_handler+0x30/0x80 >[6127711.561788] el0_svc+0x10/0x14 >[6127755.531016] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kubelet:18288] >[6127755.531826] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6127755.538765] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6127755.544591] CPU: 1 PID: 18288 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6127755.545627] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6127755.546517] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6127755.547004] pc : mem_cgroup_nr_lru_pages.isra.7+0x9c/0xc8 >[6127755.547520] lr : mem_cgroup_nr_lru_pages.isra.7+0xac/0xc8 >[6127755.548032] sp : ffff00001f91bc20 >[6127755.548370] x29: ffff00001f91bc20 x28: 0000000000000000 >[6127755.548888] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6127755.549421] x25: 0000000000000010 x24: 0000000000000000 >[6127755.549943] x23: ffff8dac3090a5d0 x22: 0000000000000020 >[6127755.550618] x21: ffff59fd7fe49800 x20: ffff59fd7fe4b8a8 >[6127755.551159] x19: 0000000000000004 x18: 0000000000000000 >[6127755.551719] x17: 0000000000000002 x16: 0000000000000000 >[6127755.552232] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6127755.552828] x13: ffff8b7d1ada8000 x12: ffff8b7d1ada7326 >[6127755.553367] x11: 0000000000000000 x10: ffff00001f91bc80 >[6127755.553902] x9 : ffff59fd7ef3da4c x8 : 000000000000d927 >[6127755.554444] x7 : 0000000000000004 x6 : 0000000000000005 >[6127755.554948] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6127755.555452] x3 : 0000000000000014 x2 : 000033a040475000 >[6127755.555949] x1 : ffff4a2dcffa5c70 x0 : 0000000000000054 >[6127755.556450] Call trace: >[6127755.556701] mem_cgroup_nr_lru_pages.isra.7+0x9c/0xc8 >[6127755.557207] memcg_numa_stat_show+0x14c/0x260 >[6127755.557673] cgroup_seqfile_show+0x50/0xd0 >[6127755.558120] kernfs_seq_show+0x30/0x38 >[6127755.558489] seq_read+0x148/0x478 >[6127755.558823] kernfs_fop_read+0x2c/0x1e0 >[6127755.559250] __vfs_read+0x20/0x48 >[6127755.559596] vfs_read+0x98/0x168 >[6127755.559908] ksys_read+0x6c/0xe8 >[6127755.560347] __arm64_sys_read+0x20/0x28 >[6127755.560742] el0_svc_common.constprop.0+0xa8/0x200 >[6127755.561255] el0_svc_handler+0x30/0x80 >[6127755.561661] el0_svc+0x10/0x14 >[6127795.531003] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [migration/0:13] >[6127795.531464] watchdog: BUG: soft lockup - CPU#28 stuck for 23s! [kubelet:18924] >[6127795.531902] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6127795.532659] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6127795.539098] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6127795.545339] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6127795.551444] CPU: 0 PID: 13 Comm: migration/0 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6127795.557286] CPU: 28 PID: 18924 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6127795.558337] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6127795.559305] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6127795.560303] pstate: 60000005 (nZCv daif -PAN -UAO) >[6127795.561213] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6127795.561720] pc : multi_cpu_stop+0x78/0x180 >[6127795.562149] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6127795.562153] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6127795.562552] lr : multi_cpu_stop+0xb4/0x180 >[6127795.562555] sp : ffff00000894bd50 >[6127795.563082] sp : ffff00003e19bc20 >[6127795.563623] x29: ffff00000894bd50 x28: 0000000000000000 >[6127795.564016] x29: ffff00003e19bc20 x28: ffff8b8c71848000 >[6127795.564340] x27: ffff00005b39b938 x26: 0000000000000060 >[6127795.564720] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6127795.565217] x25: 0000000000000000 x24: 0000000000000000 >[6127795.565717] x25: 0000000000000003 x24: 0000000000000000 >[6127795.566228] x23: 0000000000000000 x22: 0000000000000001 >[6127795.566728] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6127795.567238] x21: ffff00005b39b9d4 x20: ffff00005b39b9b0 >[6127795.567732] x21: ffff8d8d2c0c8400 x20: ffff59fd7fe4b8a8 >[6127795.568235] x19: 0000000000000001 x18: 0000000000000000 >[6127795.568744] x19: 0000000000000000 x18: 0000000000000000 >[6127795.569234] x17: 0000000000000001 x16: 0000000000000000 >[6127795.569721] x17: 0000000000000004 x16: 0000000000000000 >[6127795.570366] x15: 0000000000000000 x14: 0000000000000000 >[6127795.570934] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6127795.571443] x13: 0000000000000000 x12: 0000000000000000 >[6127795.571930] x13: ffff8bad67874000 x12: ffff8bad678732f3 >[6127795.572434] x11: 0000000000000000 x10: 0000000000000d60 >[6127795.572943] x11: 0000000000000000 x10: ffff00003e19bc80 >[6127795.573473] x9 : ffff59fd7ede5a0c x8 : ffff8b7d9d959f00 >[6127795.573955] x9 : ffff59fd7ef3dad0 x8 : 00000000000376a3 >[6127795.574496] x7 : ffff8b7d818e0400 x6 : 0000000000000014 >[6127795.575007] x7 : 0000000000000005 x6 : 0000000000000006 >[6127795.575520] x5 : 000031804040f000 x4 : ffff8b7d9d959740 >[6127795.576040] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6127795.576553] x3 : dead000000000200 x2 : ffff00005b39b960 >[6127795.577089] x3 : 000000000000001b x2 : 000031b04045c000 >[6127795.577603] x1 : ffff00005b39b9d4 x0 : 0000000000000001 >[6127795.578122] x1 : ffff4a1f1d044010 x0 : 000000000000001b >[6127795.578636] Call trace: >[6127795.579165] Call trace: >[6127795.579692] multi_cpu_stop+0x78/0x180 >[6127795.580285] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6127795.580537] cpu_stopper_thread+0xec/0x178 >[6127795.580776] memcg_numa_stat_show+0x1d0/0x260 >[6127795.581178] smpboot_thread_fn+0x158/0x1d8 >[6127795.581688] cgroup_seqfile_show+0x50/0xd0 >[6127795.582060] kthread+0x130/0x138 >[6127795.582455] kernfs_seq_show+0x30/0x38 >[6127795.583885] seq_read+0x148/0x478 >[6127795.584204] kernfs_fop_read+0x2c/0x1e0 >[6127795.584579] __vfs_read+0x20/0x48 >[6127795.584962] vfs_read+0x98/0x168 >[6127795.585262] ksys_read+0x6c/0xe8 >[6127795.585555] __arm64_sys_read+0x20/0x28 >[6127795.586027] el0_svc_common.constprop.0+0xa8/0x200 >[6127795.586501] el0_svc_handler+0x30/0x80 >[6127795.586874] el0_svc+0x10/0x14 >[6127839.531182] watchdog: BUG: soft lockup - CPU#10 stuck for 23s! [kubelet:13350] >[6127839.531928] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6127839.538456] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6127839.544235] CPU: 10 PID: 13350 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6127839.545255] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6127839.546134] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6127839.546643] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6127839.547208] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6127839.547734] sp : ffff0000575d3c20 >[6127839.548122] x29: ffff0000575d3c20 x28: ffff8b7c1adde000 >[6127839.548669] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6127839.549202] x25: 0000000000000003 x24: 0000000000000000 >[6127839.549837] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6127839.550379] x21: ffff8dbc4d3c4000 x20: ffff59fd7fe4b8a8 >[6127839.550925] x19: 0000000000000000 x18: 0000000000000000 >[6127839.551481] x17: 0000000000000004 x16: 0000000000000000 >[6127839.551983] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6127839.552624] x13: ffff8b8d72d55000 x12: ffff8b8d72d54311 >[6127839.553155] x11: 0000000000000000 x10: ffff0000575d3c80 >[6127839.553739] x9 : ffff59fd7ef3dad0 x8 : 000000000004eff3 >[6127839.554277] x7 : 0000000000000005 x6 : 0000000000000006 >[6127839.554786] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6127839.555364] x3 : 000000000000001d x2 : 000031b04048e000 >[6127839.555941] x1 : ffff4a1cfee0f080 x0 : 000000000000001d >[6127839.556479] Call trace: >[6127839.556753] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6127839.557255] memcg_numa_stat_show+0x1d0/0x260 >[6127839.557696] cgroup_seqfile_show+0x50/0xd0 >[6127839.558130] kernfs_seq_show+0x30/0x38 >[6127839.558565] seq_read+0x148/0x478 >[6127839.558918] kernfs_fop_read+0x2c/0x1e0 >[6127839.559323] __vfs_read+0x20/0x48 >[6127839.559740] vfs_read+0x98/0x168 >[6127839.560075] ksys_read+0x6c/0xe8 >[6127839.560426] __arm64_sys_read+0x20/0x28 >[6127839.560824] el0_svc_common.constprop.0+0xa8/0x200 >[6127839.561326] el0_svc_handler+0x30/0x80 >[6127839.561731] el0_svc+0x10/0x14 >[6127883.531663] watchdog: BUG: soft lockup - CPU#41 stuck for 23s! [kubelet:18612] >[6127883.532440] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6127883.539111] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6127883.544620] CPU: 41 PID: 18612 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6127883.545525] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6127883.546339] pstate: 20000005 (nzCv daif -PAN -UAO) >[6127883.546776] pc : find_next_bit+0x2c/0x68 >[6127883.547135] lr : cpumask_next+0x1c/0x28 >[6127883.547574] sp : ffff000050243c10 >[6127883.547923] x29: ffff000050243c10 x28: 0000000000000000 >[6127883.548444] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6127883.548923] x25: 0000000000000010 x24: 0000000000000000 >[6127883.549516] x23: ffff8bace6f365d0 x22: 0000000000000020 >[6127883.550014] x21: ffff59fd7fe49800 x20: ffff59fd7fe4b8a8 >[6127883.550547] x19: 0000000000000004 x18: 0000000000000000 >[6127883.551136] x17: 0000000000000002 x16: 0000000000000000 >[6127883.551676] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6127883.552214] x13: ffff8bcd57f12000 x12: ffff8bcd57f11326 >[6127883.552779] x11: 0000000000000000 x10: ffff000050243c80 >[6127883.553290] x9 : ffff59fd7ef3da4c x8 : 000000000000dbbf >[6127883.553765] x7 : 0000000000000004 x6 : 0000000000000005 >[6127883.554237] x5 : 00000000fffffff7 x4 : 0000000000000000 >[6127883.554704] x3 : 0000000000000007 x2 : 00000000000001c0 >[6127883.555186] x1 : 0000000000000400 x0 : ffff59fd7fe4b8a8 >[6127883.555747] Call trace: >[6127883.555983] find_next_bit+0x2c/0x68 >[6127883.556331] mem_cgroup_nr_lru_pages.isra.7+0xac/0xc8 >[6127883.556795] memcg_numa_stat_show+0x14c/0x260 >[6127883.557208] cgroup_seqfile_show+0x50/0xd0 >[6127883.557582] kernfs_seq_show+0x30/0x38 >[6127883.557924] seq_read+0x148/0x478 >[6127883.558231] kernfs_fop_read+0x2c/0x1e0 >[6127883.558606] __vfs_read+0x20/0x48 >[6127883.558931] vfs_read+0x98/0x168 >[6127883.559245] ksys_read+0x6c/0xe8 >[6127883.559645] __arm64_sys_read+0x20/0x28 >[6127883.560011] el0_svc_common.constprop.0+0xa8/0x200 >[6127883.560477] el0_svc_handler+0x30/0x80 >[6127883.560907] el0_svc+0x10/0x14 >[6127923.531259] watchdog: BUG: soft lockup - CPU#15 stuck for 22s! [kubelet:55880] >[6127923.532115] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6127923.539175] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6127923.545065] CPU: 15 PID: 55880 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6127923.546056] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6127923.546927] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6127923.547415] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6127923.547931] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6127923.548425] sp : ffff00003e28bc20 >[6127923.548744] x29: ffff00003e28bc20 x28: ffff8bbb621e4000 >[6127923.549395] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6127923.549878] x25: 0000000000000003 x24: 0000000000000000 >[6127923.550360] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6127923.550883] x21: ffff8d9c3115fc00 x20: ffff59fd7fe4b8a8 >[6127923.551390] x19: 0000000000000000 x18: 0000000000000000 >[6127923.551898] x17: 0000000000000002 x16: 0000000000000000 >[6127923.552414] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6127923.552938] x13: ffff8b8bdf96c000 x12: ffff8b8bdf96b2fd >[6127923.553446] x11: 0000000000000000 x10: ffff00003e28bc80 >[6127923.553964] x9 : ffff59fd7ef3dad0 x8 : 0000000000014dfa >[6127923.554477] x7 : 0000000000000004 x6 : 0000000000000005 >[6127923.554990] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6127923.555508] x3 : 000000000000002e x2 : 000031d0404a7000 >[6127923.556107] x1 : ffff4a1b63dcfe70 x0 : 000000000000002e >[6127923.556667] Call trace: >[6127923.556906] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6127923.557417] memcg_numa_stat_show+0x1d0/0x260 >[6127923.557888] cgroup_seqfile_show+0x50/0xd0 >[6127923.558273] kernfs_seq_show+0x30/0x38 >[6127923.558659] seq_read+0x148/0x478 >[6127923.559059] kernfs_fop_read+0x2c/0x1e0 >[6127923.559440] __vfs_read+0x20/0x48 >[6127923.559802] vfs_read+0x98/0x168 >[6127923.560125] ksys_read+0x6c/0xe8 >[6127923.560437] __arm64_sys_read+0x20/0x28 >[6127923.560842] el0_svc_common.constprop.0+0xa8/0x200 >[6127923.561347] el0_svc_handler+0x30/0x80 >[6127923.561702] el0_svc+0x10/0x14 >[6127967.531183] watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [kubelet:18144] >[6127967.532095] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6127967.538560] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6127967.544243] CPU: 10 PID: 18144 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6127967.545258] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6127967.546130] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6127967.546669] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6127967.547209] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6127967.547771] sp : ffff0000351c3c20 >[6127967.548125] x29: ffff0000351c3c20 x28: ffff8babefa5a000 >[6127967.548789] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6127967.549344] x25: 0000000000000003 x24: 0000000000000000 >[6127967.549877] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6127967.550411] x21: ffff8dccc8b37c00 x20: ffff59fd7fe4b8a8 >[6127967.550968] x19: 0000000000000000 x18: 0000000000000000 >[6127967.551504] x17: 0000000000000650 x16: 0000000000000000 >[6127967.552086] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6127967.552602] x13: ffff8b8d72d57000 x12: ffff8b8d72d5631c >[6127967.553194] x11: 0000000000000000 x10: ffff0000351c3c80 >[6127967.553769] x9 : ffff59fd7ef3dad0 x8 : 0000000000017f10 >[6127967.554312] x7 : 0000000000000004 x6 : 0000000000000005 >[6127967.554818] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6127967.555375] x3 : 0000000000000025 x2 : 000033c04048d000 >[6127967.555887] x1 : ffff4a2b4ffb5e60 x0 : 0000000000000065 >[6127967.556426] Call trace: >[6127967.556690] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6127967.557187] memcg_numa_stat_show+0x1d0/0x260 >[6127967.557696] cgroup_seqfile_show+0x50/0xd0 >[6127967.558106] kernfs_seq_show+0x30/0x38 >[6127967.558571] seq_read+0x148/0x478 >[6127967.558995] kernfs_fop_read+0x2c/0x1e0 >[6127967.559371] __vfs_read+0x20/0x48 >[6127967.559700] vfs_read+0x98/0x168 >[6127967.560020] ksys_read+0x6c/0xe8 >[6127967.560336] __arm64_sys_read+0x20/0x28 >[6127967.560719] el0_svc_common.constprop.0+0xa8/0x200 >[6127967.561177] el0_svc_handler+0x30/0x80 >[6127967.561530] el0_svc+0x10/0x14 >[6128011.531025] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kubelet:7308] >[6128011.531754] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6128011.538756] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6128011.544523] CPU: 2 PID: 7308 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6128011.545483] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6128011.546348] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6128011.546799] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6128011.547299] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6128011.547808] sp : ffff00001b39bc20 >[6128011.548139] x29: ffff00001b39bc20 x28: ffff8bbce97fe000 >[6128011.548746] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6128011.549291] x25: 0000000000000010 x24: 0000000000000000 >[6128011.549866] x23: 0000000000000020 x22: ffff59fd7fe49800 >[6128011.550364] x21: ffff8b8cc833c800 x20: ffff59fd7fe4b8a8 >[6128011.550917] x19: 0000000000000004 x18: 0000000000000000 >[6128011.551530] x17: 0000000000000002 x16: 0000000000000000 >[6128011.552056] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6128011.552586] x13: ffff8b7ba68fd000 x12: ffff8b7ba68fc34b >[6128011.553110] x11: 0000000000000000 x10: ffff00001b39bc80 >[6128011.553636] x9 : ffff59fd7ef3dad0 x8 : ffff8b7ba68fc000 >[6128011.554147] x7 : 0000000000000000 x6 : 0000000000000001 >[6128011.554666] x5 : 00000000fffffffb x4 : ffffffffffffffff >[6128011.555177] x3 : 0000000000000019 x2 : 000033b04042a000 >[6128011.555715] x1 : ffff4a2c447d4eb0 x0 : 0000000000000059 >[6128011.556312] Call trace: >[6128011.556614] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6128011.557106] memcg_numa_stat_show+0x1d0/0x260 >[6128011.557588] cgroup_seqfile_show+0x50/0xd0 >[6128011.558009] kernfs_seq_show+0x30/0x38 >[6128011.558524] seq_read+0x148/0x478 >[6128011.558859] kernfs_fop_read+0x2c/0x1e0 >[6128011.559283] __vfs_read+0x20/0x48 >[6128011.559633] vfs_read+0x98/0x168 >[6128011.559953] ksys_read+0x6c/0xe8 >[6128011.560307] __arm64_sys_read+0x20/0x28 >[6128011.560736] el0_svc_common.constprop.0+0xa8/0x200 >[6128011.561205] el0_svc_handler+0x30/0x80 >[6128011.561693] el0_svc+0x10/0x14 >[6128051.531089] watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [kubelet:18924] >[6128051.531899] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6128051.538764] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6128051.544781] CPU: 4 PID: 18924 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6128051.545797] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6128051.546645] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6128051.547123] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6128051.547657] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6128051.548296] sp : ffff00003e19bc20 >[6128051.548672] x29: ffff00003e19bc20 x28: ffff8d8cf09d2000 >[6128051.549255] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6128051.549757] x25: 0000000000000003 x24: 0000000000000000 >[6128051.550288] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6128051.550857] x21: ffff8d9c79e0c400 x20: ffff59fd7fe4b8a8 >[6128051.551472] x19: 0000000000000000 x18: 0000000000000000 >[6128051.551976] x17: 0000000000000002 x16: 0000000000000000 >[6128051.552561] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6128051.553107] x13: ffff8b7d1b324000 x12: ffff8b7d1b3232fc >[6128051.553614] x11: 0000000000000000 x10: ffff00003e19bc80 >[6128051.554118] x9 : ffff59fd7ef3dad0 x8 : 000000000001522b >[6128051.554638] x7 : 0000000000000004 x6 : 0000000000000005 >[6128051.555134] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6128051.555629] x3 : 000000000000002f x2 : 000031d0404c0000 >[6128051.556129] x1 : ffff4a2dcce62f20 x0 : 000000000000002f >[6128051.556640] Call trace: >[6128051.556894] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6128051.557406] memcg_numa_stat_show+0x1d0/0x260 >[6128051.557860] cgroup_seqfile_show+0x50/0xd0 >[6128051.558357] kernfs_seq_show+0x30/0x38 >[6128051.558780] seq_read+0x148/0x478 >[6128051.559097] kernfs_fop_read+0x2c/0x1e0 >[6128051.559455] __vfs_read+0x20/0x48 >[6128051.559805] vfs_read+0x98/0x168 >[6128051.560159] ksys_read+0x6c/0xe8 >[6128051.560467] __arm64_sys_read+0x20/0x28 >[6128051.560844] el0_svc_common.constprop.0+0xa8/0x200 >[6128051.561339] el0_svc_handler+0x30/0x80 >[6128051.561706] el0_svc+0x10/0x14 >[6128095.531146] watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [migration/8:51] >[6128095.531308] watchdog: BUG: soft lockup - CPU#18 stuck for 22s! [kubelet:4832] >[6128095.532028] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6128095.532716] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6128095.539364] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6128095.545591] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6128095.551281] CPU: 8 PID: 51 Comm: migration/8 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6128095.557818] CPU: 18 PID: 4832 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6128095.558912] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6128095.559831] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6128095.560752] pstate: 60000005 (nZCv daif -PAN -UAO) >[6128095.561688] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6128095.562231] pc : multi_cpu_stop+0xf4/0x180 >[6128095.562234] lr : multi_cpu_stop+0xb4/0x180 >[6128095.562672] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6128095.562676] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6128095.563097] sp : ffff000008e73d50 >[6128095.563459] sp : ffff00005e9c3c20 >[6128095.564002] x29: ffff000008e73d50 x28: 0000000000000000 >[6128095.564475] x29: ffff00005e9c3c20 x28: ffff8bac39eae000 >[6128095.564842] x27: ffff00005b39b938 x26: 0000000000000060 >[6128095.565147] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6128095.565677] x25: 0000000000000000 x24: 0000000000000000 >[6128095.566211] x25: 0000000000000003 x24: 0000000000000000 >[6128095.566750] x23: 0000000000000000 x22: 0000000000000001 >[6128095.567231] x23: 0000000000000008 x22: ffff59fd7fe49800 >[6128095.567929] x21: ffff00005b39b9d4 x20: ffff00005b39b9b0 >[6128095.568405] x21: ffff8dac6e226000 x20: ffff59fd7fe4b8a8 >[6128095.568949] x19: 0000000000000001 x18: 0000000000000000 >[6128095.569485] x19: 0000000000000001 x18: 0000000000000000 >[6128095.570008] x17: 0000000000000001 x16: 0000000000000000 >[6128095.570540] x17: 0000000000000002 x16: 0000000000000000 >[6128095.571050] x15: 0000000000000000 x14: 0000000000000000 >[6128095.571525] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6128095.572025] x13: 0000000000000000 x12: 0000000000000000 >[6128095.572530] x13: ffff8b9d205f1000 x12: ffff8b9d205f0306 >[6128095.573061] x11: 0000000000000000 x10: 0000000000000d60 >[6128095.573563] x11: 0000000000000000 x10: ffff00005e9c3c80 >[6128095.574088] x9 : ffff59fd7ede5a0c x8 : ffff8b8d9e8d6400 >[6128095.574587] x9 : ffff59fd7ef3dad0 x8 : 000000000000bf31 >[6128095.575081] x7 : ffff8b8d8059c000 x6 : 0000000000000388 >[6128095.575581] x7 : 0000000000000004 x6 : 0000000000000005 >[6128095.576094] x5 : 0000319040411000 x4 : ffff8b8d9e8d5c40 >[6128095.576586] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6128095.577166] x3 : dead000000000200 x2 : ffff00005b39b960 >[6128095.577668] x3 : 0000000000000014 x2 : 000031a040475000 >[6128095.578324] x1 : ffff00005b39b9d4 x0 : 0000000000000001 >[6128095.578831] x1 : ffff4a13015b6298 x0 : 0000000000000014 >[6128095.579420] Call trace: >[6128095.579885] Call trace: >[6128095.580510] multi_cpu_stop+0xf4/0x180 >[6128095.581003] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6128095.581245] cpu_stopper_thread+0xec/0x178 >[6128095.581490] memcg_numa_stat_show+0x1d0/0x260 >[6128095.581926] smpboot_thread_fn+0x158/0x1d8 >[6128095.582401] cgroup_seqfile_show+0x50/0xd0 >[6128095.582867] kthread+0x130/0x138 >[6128095.583283] kernfs_seq_show+0x30/0x38 >[6128095.584802] seq_read+0x148/0x478 >[6128095.585170] kernfs_fop_read+0x2c/0x1e0 >[6128095.585534] __vfs_read+0x20/0x48 >[6128095.585834] vfs_read+0x98/0x168 >[6128095.586132] ksys_read+0x6c/0xe8 >[6128095.586444] __arm64_sys_read+0x20/0x28 >[6128095.586816] el0_svc_common.constprop.0+0xa8/0x200 >[6128095.587238] el0_svc_handler+0x30/0x80 >[6128095.587581] el0_svc+0x10/0x14 >[6128139.531256] watchdog: BUG: soft lockup - CPU#15 stuck for 22s! [kubelet:21523] >[6128139.532075] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6128139.538892] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6128139.544977] CPU: 15 PID: 21523 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6128139.545974] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6128139.546839] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6128139.547340] pc : mem_cgroup_nr_lru_pages.isra.7+0x9c/0xc8 >[6128139.547987] lr : mem_cgroup_nr_lru_pages.isra.7+0xac/0xc8 >[6128139.548464] sp : ffff00001b87bc20 >[6128139.548791] x29: ffff00001b87bc20 x28: 0000000000000000 >[6128139.549310] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6128139.549788] x25: 0000000000000010 x24: 0000000000000000 >[6128139.550278] x23: ffff8b7d42a0c5d0 x22: 0000000000000020 >[6128139.550834] x21: ffff59fd7fe49800 x20: ffff59fd7fe4b8a8 >[6128139.551341] x19: 0000000000000004 x18: 0000000000000000 >[6128139.551851] x17: 0000000000000002 x16: 0000000000000000 >[6128139.552378] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6128139.552935] x13: ffff8b8bdf96e000 x12: ffff8b8bdf96d326 >[6128139.553426] x11: 0000000000000000 x10: ffff00001b87bc80 >[6128139.553960] x9 : ffff59fd7ef3da4c x8 : 000000000000d952 >[6128139.554445] x7 : 0000000000000004 x6 : 0000000000000005 >[6128139.554966] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6128139.555581] x3 : 0000000000000015 x2 : 000031a04048e000 >[6128139.556129] x1 : ffff4a2f649f6700 x0 : 0000000000000015 >[6128139.556629] Call trace: >[6128139.556879] mem_cgroup_nr_lru_pages.isra.7+0x9c/0xc8 >[6128139.557498] memcg_numa_stat_show+0x14c/0x260 >[6128139.557948] cgroup_seqfile_show+0x50/0xd0 >[6128139.558419] kernfs_seq_show+0x30/0x38 >[6128139.558810] seq_read+0x148/0x478 >[6128139.559137] kernfs_fop_read+0x2c/0x1e0 >[6128139.559536] __vfs_read+0x20/0x48 >[6128139.559871] vfs_read+0x98/0x168 >[6128139.560185] ksys_read+0x6c/0xe8 >[6128139.560514] __arm64_sys_read+0x20/0x28 >[6128139.560877] el0_svc_common.constprop.0+0xa8/0x200 >[6128139.561347] el0_svc_handler+0x30/0x80 >[6128139.561694] el0_svc+0x10/0x14 >[6128179.531356] watchdog: BUG: soft lockup - CPU#21 stuck for 22s! [kubelet:61154] >[6128179.532145] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6128179.539701] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6128179.546227] CPU: 21 PID: 61154 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6128179.547417] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6128179.548398] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6128179.548933] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6128179.549493] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6128179.550127] sp : ffff000059c5bc20 >[6128179.550489] x29: ffff000059c5bc20 x28: ffff8bbb3ff00000 >[6128179.551127] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6128179.551719] x25: 0000000000000003 x24: 0000000000000000 >[6128179.552287] x23: 0000000000000008 x22: ffff59fd7fe49800 >[6128179.552861] x21: ffff8d7c0bc18000 x20: ffff59fd7fe4b8a8 >[6128179.553432] x19: 0000000000000001 x18: 0000000000000000 >[6128179.553993] x17: 0000000000000002 x16: 0000000000000000 >[6128179.554559] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6128179.555126] x13: ffff8b9d20208000 x12: ffff8b9d202072e8 >[6128179.555728] x11: 0000000000000000 x10: ffff000059c5bc80 >[6128179.556294] x9 : ffff59fd7ef3dad0 x8 : 00000000000200bd >[6128179.556828] x7 : 0000000000000005 x6 : 0000000000000006 >[6128179.557435] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6128179.558006] x3 : 000000000000000d x2 : 000031904048e000 >[6128179.558538] x1 : ffff4a2b55fd4358 x0 : 000000000000000d >[6128179.559025] Call trace: >[6128179.559260] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6128179.559726] memcg_numa_stat_show+0x1d0/0x260 >[6128179.560158] cgroup_seqfile_show+0x50/0xd0 >[6128179.560647] kernfs_seq_show+0x30/0x38 >[6128179.561003] seq_read+0x148/0x478 >[6128179.561321] kernfs_fop_read+0x2c/0x1e0 >[6128179.561751] __vfs_read+0x20/0x48 >[6128179.562080] vfs_read+0x98/0x168 >[6128179.562407] ksys_read+0x6c/0xe8 >[6128179.562793] __arm64_sys_read+0x20/0x28 >[6128179.563252] el0_svc_common.constprop.0+0xa8/0x200 >[6128179.563772] el0_svc_handler+0x30/0x80 >[6128179.564178] el0_svc+0x10/0x14 >[6128183.531460] watchdog: BUG: soft lockup - CPU#28 stuck for 22s! [migration/28:154] >[6128183.532366] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6128183.539438] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6128183.545176] CPU: 28 PID: 154 Comm: migration/28 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6128183.546146] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6128183.547020] pstate: 60000005 (nZCv daif -PAN -UAO) >[6128183.547550] pc : multi_cpu_stop+0x78/0x180 >[6128183.548046] lr : multi_cpu_stop+0xb4/0x180 >[6128183.548429] sp : ffff00000a1b3d50 >[6128183.548848] x29: ffff00000a1b3d50 x28: 0000000000000000 >[6128183.549398] x27: ffff000033b3b938 x26: 0000000000000060 >[6128183.550003] x25: 0000000000000000 x24: 0000000000000000 >[6128183.550646] x23: 0000000000000000 x22: 0000000000000001 >[6128183.551226] x21: ffff000033b3b9d4 x20: ffff000033b3b9b0 >[6128183.551777] x19: 0000000000000001 x18: 0000000000000000 >[6128183.552319] x17: 0000000000000001 x16: 0000000000000000 >[6128183.552791] x15: 0000000000000000 x14: 0000000000000000 >[6128183.553412] x13: 0000000000000000 x12: 0000000000000000 >[6128183.553999] x11: 0000000000000000 x10: 0000000000000d60 >[6128183.554525] x9 : ffff59fd7ede5a0c x8 : ffff8bad9e953040 >[6128183.555017] x7 : ffff8bad78975c00 x6 : 00000000000000b4 >[6128183.555625] x5 : 000031b040475000 x4 : ffff8bad9e952880 >[6128183.556266] x3 : dead000000000200 x2 : ffff000033b3b988 >[6128183.556789] x1 : ffff000033b3b9d4 x0 : 0000000000000001 >[6128183.557363] Call trace: >[6128183.557629] multi_cpu_stop+0x78/0x180 >[6128183.558053] cpu_stopper_thread+0xec/0x178 >[6128183.558453] smpboot_thread_fn+0x158/0x1d8 >[6128183.558836] kthread+0x130/0x138 >[6128223.531076] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kubelet:18849] >[6128223.531886] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6128223.539310] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6128223.545639] CPU: 3 PID: 18849 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6128223.546773] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6128223.547739] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6128223.548234] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6128223.548754] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6128223.549297] sp : ffff000030123c20 >[6128223.549633] x29: ffff000030123c20 x28: ffff8b8bc4b5c000 >[6128223.550202] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6128223.550711] x25: 0000000000000003 x24: 0000000000000000 >[6128223.551270] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6128223.551768] x21: ffff8d8cc924cc00 x20: ffff59fd7fe4b8a8 >[6128223.552279] x19: 0000000000000000 x18: 0000000000000000 >[6128223.552796] x17: 0000000000000002 x16: 0000000000000000 >[6128223.553295] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6128223.553791] x13: ffff8b7d1cf91000 x12: ffff8b7d1cf902f3 >[6128223.554295] x11: 0000000000000000 x10: ffff000030123c80 >[6128223.554810] x9 : ffff59fd7ef3dad0 x8 : 0000000000037773 >[6128223.555314] x7 : 0000000000000005 x6 : 0000000000000006 >[6128223.555842] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6128223.556370] x3 : 0000000000000036 x2 : 000033d040301000 >[6128223.556986] x1 : ffff4a0cef42c670 x0 : 0000000000000036 >[6128223.557552] Call trace: >[6128223.557832] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6128223.558355] memcg_numa_stat_show+0x1d0/0x260 >[6128223.558874] cgroup_seqfile_show+0x50/0xd0 >[6128223.559325] kernfs_seq_show+0x30/0x38 >[6128223.559756] seq_read+0x148/0x478 >[6128223.560117] kernfs_fop_read+0x2c/0x1e0 >[6128223.560533] __vfs_read+0x20/0x48 >[6128223.560917] vfs_read+0x98/0x168 >[6128223.561267] ksys_read+0x6c/0xe8 >[6128223.561659] __arm64_sys_read+0x20/0x28 >[6128223.562075] el0_svc_common.constprop.0+0xa8/0x200 >[6128223.562577] el0_svc_handler+0x30/0x80 >[6128223.563030] el0_svc+0x10/0x14 >[6128267.531371] watchdog: BUG: soft lockup - CPU#22 stuck for 22s! [kubelet:18924] >[6128267.532287] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6128267.539069] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6128267.544403] CPU: 22 PID: 18924 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6128267.545347] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6128267.546145] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6128267.546669] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6128267.547161] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6128267.547654] sp : ffff00003e19bc20 >[6128267.548034] x29: ffff00003e19bc20 x28: ffff8b7d6005c000 >[6128267.548584] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6128267.549103] x25: 0000000000000003 x24: 0000000000000000 >[6128267.549649] x23: 0000000000000008 x22: ffff59fd7fe49800 >[6128267.550172] x21: ffff8dad98a78000 x20: ffff59fd7fe4b8a8 >[6128267.550684] x19: 0000000000000001 x18: 0000000000000000 >[6128267.551213] x17: 0000000000000002 x16: 0000000000000000 >[6128267.551736] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6128267.552266] x13: ffff8b9d2055d000 x12: ffff8b9d2055c307 >[6128267.552788] x11: 0000000000000000 x10: ffff00003e19bc80 >[6128267.553270] x9 : ffff59fd7ef3dad0 x8 : 000000000000c42c >[6128267.553756] x7 : 0000000000000004 x6 : 0000000000000005 >[6128267.554258] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6128267.554747] x3 : 000000000000001e x2 : 000033b0404a7000 >[6128267.555270] x1 : ffff4a2c44e0fd98 x0 : 000000000000005e >[6128267.555755] Call trace: >[6128267.556005] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6128267.556496] memcg_numa_stat_show+0x1d0/0x260 >[6128267.557009] cgroup_seqfile_show+0x50/0xd0 >[6128267.557394] kernfs_seq_show+0x30/0x38 >[6128267.557744] seq_read+0x148/0x478 >[6128267.558052] kernfs_fop_read+0x2c/0x1e0 >[6128267.558423] __vfs_read+0x20/0x48 >[6128267.558734] vfs_read+0x98/0x168 >[6128267.559104] ksys_read+0x6c/0xe8 >[6128267.559429] __arm64_sys_read+0x20/0x28 >[6128267.559801] el0_svc_common.constprop.0+0xa8/0x200 >[6128267.560243] el0_svc_handler+0x30/0x80 >[6128267.560619] el0_svc+0x10/0x14 >[6128311.531228] watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [kubelet:55880] >[6128311.532107] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6128311.538847] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6128311.544604] CPU: 13 PID: 55880 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6128311.545615] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6128311.546641] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6128311.547115] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6128311.547624] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6128311.548137] sp : ffff00003e28bc20 >[6128311.548532] x29: ffff00003e28bc20 x28: ffff8b9ce7fac000 >[6128311.549137] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6128311.549667] x25: 0000000000000003 x24: 0000000000000000 >[6128311.550171] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6128311.550683] x21: ffff8dcd491e0400 x20: ffff59fd7fe4b8a8 >[6128311.551210] x19: 0000000000000000 x18: 0000000000000000 >[6128311.551744] x17: 0000000000000002 x16: 0000000000000000 >[6128311.552280] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6128311.552774] x13: ffff8b8d73420000 x12: ffff8b8d7341f31c >[6128311.553269] x11: 0000000000000000 x10: ffff00003e28bc80 >[6128311.553782] x9 : ffff59fd7ef3dad0 x8 : 000000000001823f >[6128311.554306] x7 : 0000000000000004 x6 : 0000000000000005 >[6128311.554814] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6128311.555325] x3 : 000000000000002d x2 : 000031d04048e000 >[6128311.555844] x1 : ffff4a1f232a66d0 x0 : 000000000000002d >[6128311.556443] Call trace: >[6128311.556756] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6128311.557287] memcg_numa_stat_show+0x1d0/0x260 >[6128311.557694] cgroup_seqfile_show+0x50/0xd0 >[6128311.558066] kernfs_seq_show+0x30/0x38 >[6128311.558412] seq_read+0x148/0x478 >[6128311.558739] kernfs_fop_read+0x2c/0x1e0 >[6128311.559093] __vfs_read+0x20/0x48 >[6128311.559428] vfs_read+0x98/0x168 >[6128311.559757] ksys_read+0x6c/0xe8 >[6128311.560101] __arm64_sys_read+0x20/0x28 >[6128311.560496] el0_svc_common.constprop.0+0xa8/0x200 >[6128311.561007] el0_svc_handler+0x30/0x80 >[6128311.561413] el0_svc+0x10/0x14 >[6128355.531021] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [migration/2:21] >[6128355.531710] watchdog: BUG: soft lockup - CPU#44 stuck for 22s! [kubelet:13350] >[6128355.531853] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6128355.532526] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6128355.539064] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6128355.545344] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6128355.551354] CPU: 2 PID: 21 Comm: migration/2 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6128355.557101] CPU: 44 PID: 13350 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6128355.558107] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6128355.559112] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6128355.559990] pstate: 60000005 (nZCv daif -PAN -UAO) >[6128355.560826] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6128355.561289] pc : multi_cpu_stop+0x78/0x180 >[6128355.561760] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6128355.562152] lr : multi_cpu_stop+0xb4/0x180 >[6128355.562688] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6128355.562691] sp : ffff0000575d3c20 >[6128355.563097] sp : ffff000008d83d50 >[6128355.563637] x29: ffff0000575d3c20 x28: ffff8b7d43e04000 >[6128355.563962] x29: ffff000008d83d50 x28: 0000000000000000 >[6128355.564308] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6128355.564892] x27: ffff000030a53938 x26: 0000000000000060 >[6128355.565370] x25: 0000000000000010 x24: 0000000000000000 >[6128355.565873] x25: 0000000000000000 x24: 0000000000000000 >[6128355.566465] x23: 0000000000000020 x22: ffff59fd7fe49800 >[6128355.566967] x23: 0000000000000000 x22: 0000000000000001 >[6128355.567509] x21: ffff8b8d6ecc6800 x20: ffff59fd7fe4b8a8 >[6128355.568015] x21: ffff000030a539d4 x20: ffff000030a539b0 >[6128355.568524] x19: 0000000000000004 x18: 0000000000000000 >[6128355.569035] x19: 0000000000000001 x18: 0000000000000000 >[6128355.569553] x17: 0000000000000002 x16: 0000000000000000 >[6128355.570076] x17: 0000000000000001 x16: 0000000000000000 >[6128355.570578] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6128355.571098] x15: 0000000000000000 x14: 0000000000000000 >[6128355.571647] x13: ffff8bcc4195a000 x12: ffff8bcc4195934b >[6128355.572176] x13: 0000000000000000 x12: 0000000000000000 >[6128355.572721] x11: 0000000000000000 x10: ffff0000575d3c80 >[6128355.573215] x11: 0000000000000000 x10: 0000000000000d60 >[6128355.573688] x9 : ffff59fd7ef3dad0 x8 : ffff8bcc41959000 >[6128355.574213] x9 : ffff59fd7ede5a0c x8 : ffff8b7d9d9c3040 >[6128355.574696] x7 : 0000000000000000 x6 : 0000000000000001 >[6128355.575214] x7 : ffff8b7d6e665800 x6 : 00000000000000f3 >[6128355.575705] x5 : 00000000fffffffb x4 : ffffffffffffffff >[6128355.576336] x5 : 0000318040441000 x4 : ffff8b7d9d9c2880 >[6128355.576817] x3 : 0000000000000009 x2 : 000033904042a000 >[6128355.577366] x3 : dead000000000200 x2 : ffff000030a53960 >[6128355.577840] x1 : ffff4a2f655c1b60 x0 : 0000000000000049 >[6128355.578378] x1 : ffff000030a539d4 x0 : 0000000000000001 >[6128355.578909] Call trace: >[6128355.579416] Call trace: >[6128355.579942] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6128355.580203] multi_cpu_stop+0x78/0x180 >[6128355.580434] memcg_numa_stat_show+0x1d0/0x260 >[6128355.580900] cpu_stopper_thread+0xec/0x178 >[6128355.581275] cgroup_seqfile_show+0x50/0xd0 >[6128355.581754] smpboot_thread_fn+0x158/0x1d8 >[6128355.582132] kernfs_seq_show+0x30/0x38 >[6128355.582538] kthread+0x130/0x138 >[6128355.582958] seq_read+0x148/0x478 >[6128355.583929] kernfs_fop_read+0x2c/0x1e0 >[6128355.584277] __vfs_read+0x20/0x48 >[6128355.584616] vfs_read+0x98/0x168 >[6128355.584916] ksys_read+0x6c/0xe8 >[6128355.585239] __arm64_sys_read+0x20/0x28 >[6128355.585599] el0_svc_common.constprop.0+0xa8/0x200 >[6128355.586173] el0_svc_handler+0x30/0x80 >[6128355.586527] el0_svc+0x10/0x14 >[6128395.531279] watchdog: BUG: soft lockup - CPU#16 stuck for 23s! [kubelet:60635] >[6128395.532028] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6128395.538560] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6128395.543945] CPU: 16 PID: 60635 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6128395.544850] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6128395.545768] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6128395.546212] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6128395.546685] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6128395.547167] sp : ffff00001df63c20 >[6128395.547611] x29: ffff00001df63c20 x28: ffff8b7bf1fb0000 >[6128395.548089] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6128395.548629] x25: 0000000000000003 x24: 0000000000000000 >[6128395.549109] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6128395.549588] x21: ffff8d8d1584e000 x20: ffff59fd7fe4b8a8 >[6128395.550052] x19: 0000000000000000 x18: 0000000000000000 >[6128395.550527] x17: 0000000000000002 x16: 0000000000000000 >[6128395.551012] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6128395.551489] x13: ffff8b9ba2458000 x12: ffff8b9ba24572f3 >[6128395.551978] x11: 0000000000000000 x10: ffff00001df63c80 >[6128395.552453] x9 : ffff59fd7ef3dad0 x8 : 0000000000037573 >[6128395.552922] x7 : 0000000000000005 x6 : 0000000000000006 >[6128395.553406] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6128395.553900] x3 : 0000000000000031 x2 : 000033d040284000 >[6128395.554384] x1 : ffff4a1d015ecf60 x0 : 0000000000000031 >[6128395.554963] Call trace: >[6128395.555218] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6128395.555786] memcg_numa_stat_show+0x1d0/0x260 >[6128395.556198] cgroup_seqfile_show+0x50/0xd0 >[6128395.556592] kernfs_seq_show+0x30/0x38 >[6128395.556927] seq_read+0x148/0x478 >[6128395.557245] kernfs_fop_read+0x2c/0x1e0 >[6128395.557607] __vfs_read+0x20/0x48 >[6128395.557909] vfs_read+0x98/0x168 >[6128395.558211] ksys_read+0x6c/0xe8 >[6128395.558511] __arm64_sys_read+0x20/0x28 >[6128395.558885] el0_svc_common.constprop.0+0xa8/0x200 >[6128395.559350] el0_svc_handler+0x30/0x80 >[6128395.559713] el0_svc+0x10/0x14 >[6128439.531151] watchdog: BUG: soft lockup - CPU#8 stuck for 23s! [kubelet:55880] >[6128439.531976] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6128439.538716] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6128439.544477] CPU: 8 PID: 55880 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6128439.545600] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6128439.546490] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6128439.547013] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6128439.547522] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6128439.548149] sp : ffff00003e28bc20 >[6128439.548525] x29: ffff00003e28bc20 x28: ffff8bcca2d3e000 >[6128439.549023] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6128439.549593] x25: 0000000000000003 x24: 0000000000000000 >[6128439.550064] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6128439.550646] x21: ffff8dacad72b000 x20: ffff59fd7fe4b8a8 >[6128439.551193] x19: 0000000000000000 x18: 0000000000000000 >[6128439.551670] x17: 0000000000000002 x16: 0000000000000000 >[6128439.552172] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6128439.552670] x13: ffff8b8d72fae000 x12: ffff8b8d72fad307 >[6128439.553196] x11: 0000000000000000 x10: ffff00003e28bc80 >[6128439.553676] x9 : ffff59fd7ef3dad0 x8 : 000000000000c13b >[6128439.554164] x7 : 0000000000000004 x6 : 0000000000000005 >[6128439.554661] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6128439.555143] x3 : 0000000000000024 x2 : 000033c040474000 >[6128439.555761] x1 : ffff4a1f23e604d0 x0 : 0000000000000064 >[6128439.556266] Call trace: >[6128439.556515] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6128439.557040] memcg_numa_stat_show+0x1d0/0x260 >[6128439.557485] cgroup_seqfile_show+0x50/0xd0 >[6128439.557918] kernfs_seq_show+0x30/0x38 >[6128439.558279] seq_read+0x148/0x478 >[6128439.558656] kernfs_fop_read+0x2c/0x1e0 >[6128439.559018] __vfs_read+0x20/0x48 >[6128439.559373] vfs_read+0x98/0x168 >[6128439.559740] ksys_read+0x6c/0xe8 >[6128439.560047] __arm64_sys_read+0x20/0x28 >[6128439.560421] el0_svc_common.constprop.0+0xa8/0x200 >[6128439.560938] el0_svc_handler+0x30/0x80 >[6128439.561306] el0_svc+0x10/0x14 >[6128483.531258] watchdog: BUG: soft lockup - CPU#15 stuck for 23s! [kubelet:18289] >[6128483.531704] watchdog: BUG: soft lockup - CPU#44 stuck for 23s! [migration/44:236] >[6128483.532119] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6128483.532782] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6128483.539242] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6128483.545797] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6128483.551588] CPU: 15 PID: 18289 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6128483.557449] CPU: 44 PID: 236 Comm: migration/44 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6128483.558507] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6128483.559543] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6128483.560479] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6128483.561314] pstate: 60000005 (nZCv daif -PAN -UAO) >[6128483.561825] pc : mem_cgroup_nr_lru_pages.isra.7+0x9c/0xc8 >[6128483.562387] pc : multi_cpu_stop+0x78/0x180 >[6128483.562935] lr : mem_cgroup_nr_lru_pages.isra.7+0xac/0xc8 >[6128483.562938] sp : ffff00001f81bc20 >[6128483.563327] lr : multi_cpu_stop+0xb4/0x180 >[6128483.563841] x29: ffff00001f81bc20 x28: 0000000000000000 >[6128483.564137] sp : ffff00000a443d50 >[6128483.564529] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6128483.565123] x29: ffff00000a443d50 x28: 0000000000000000 >[6128483.565454] x25: 0000000000000010 x24: 0000000000000000 >[6128483.565939] x27: ffff000035c93938 x26: 0000000000000060 >[6128483.566449] x23: ffff8b7b518565d0 x22: 0000000000000020 >[6128483.566959] x25: 0000000000000000 x24: 0000000000000000 >[6128483.567510] x21: ffff59fd7fe49800 x20: ffff59fd7fe4b8a8 >[6128483.568031] x23: 0000000000000000 x22: 0000000000000001 >[6128483.568543] x19: 0000000000000004 x18: 0000000000000000 >[6128483.569103] x21: ffff000035c939d4 x20: ffff000035c939b0 >[6128483.569588] x17: 0000000000000002 x16: 0000000000000000 >[6128483.570064] x19: 0000000000000001 x18: 0000000000000000 >[6128483.570544] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6128483.571100] x17: 0000000000000001 x16: 0000000000000000 >[6128483.571572] x13: ffff8b8bdf96f000 x12: ffff8b8bdf96e325 >[6128483.572083] x15: 0000000000000000 x14: 0000000000000000 >[6128483.572642] x11: 0000000000000000 x10: ffff00001f81bc80 >[6128483.573114] x13: 0000000000000000 x12: 0000000000000000 >[6128483.573713] x9 : ffff59fd7ef3da4c x8 : 000000000000d967 >[6128483.574265] x11: 0000000000000000 x10: 0000000000000d60 >[6128483.574748] x7 : 0000000000000004 x6 : 0000000000000005 >[6128483.575313] x9 : ffff59fd7ede5a0c x8 : ffff8bcd9e98b040 >[6128483.575834] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6128483.576354] x7 : 00c91c92788f3a60 x6 : 0000000000000317 >[6128483.576896] x3 : 0000000000000034 x2 : 000033d0402cf000 >[6128483.577386] x5 : 00000000fa83b2da x4 : ffff8bcd9e98a880 >[6128483.577928] x1 : ffff4a2b545fc740 x0 : 0000000000000034 >[6128483.578430] x3 : dead000000000200 x2 : ffff000035c93988 >[6128483.578997] Call trace: >[6128483.579486] x1 : ffff000035c939d4 x0 : 0000000000000001 >[6128483.580052] mem_cgroup_nr_lru_pages.isra.7+0x9c/0xc8 >[6128483.580340] Call trace: >[6128483.580910] memcg_numa_stat_show+0x14c/0x260 >[6128483.581378] multi_cpu_stop+0x78/0x180 >[6128483.581625] cgroup_seqfile_show+0x50/0xd0 >[6128483.582062] cpu_stopper_thread+0xec/0x178 >[6128483.582418] kernfs_seq_show+0x30/0x38 >[6128483.582785] smpboot_thread_fn+0x158/0x1d8 >[6128483.583160] seq_read+0x148/0x478 >[6128483.583510] kthread+0x130/0x138 >[6128483.583902] kernfs_fop_read+0x2c/0x1e0 >[6128483.583907] __vfs_read+0x20/0x48 >[6128483.585360] vfs_read+0x98/0x168 >[6128483.585678] ksys_read+0x6c/0xe8 >[6128483.586049] __arm64_sys_read+0x20/0x28 >[6128483.586421] el0_svc_common.constprop.0+0xa8/0x200 >[6128483.586886] el0_svc_handler+0x30/0x80 >[6128483.587259] el0_svc+0x10/0x14 >[6128523.531228] watchdog: BUG: soft lockup - CPU#13 stuck for 23s! [kubelet:4831] >[6128523.532025] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6128523.538804] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6128523.544560] CPU: 13 PID: 4831 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6128523.545616] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6128523.546486] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6128523.546994] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6128523.547586] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6128523.548096] sp : ffff00005ec83c20 >[6128523.548473] x29: ffff00005ec83c20 x28: ffff8bcc4c454000 >[6128523.549015] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6128523.549571] x25: 0000000000000003 x24: 0000000000000000 >[6128523.550060] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6128523.550637] x21: ffff8d9c8c540400 x20: ffff59fd7fe4b8a8 >[6128523.551226] x19: 0000000000000000 x18: 0000000000000000 >[6128523.551714] x17: 0000000000000002 x16: 0000000000000000 >[6128523.552241] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6128523.552815] x13: ffff8b8d73420000 x12: ffff8b8d7341f2fc >[6128523.553360] x11: 0000000000000000 x10: ffff00005ec83c80 >[6128523.553857] x9 : ffff59fd7ef3dad0 x8 : 000000000001503c >[6128523.554429] x7 : 0000000000000004 x6 : 0000000000000005 >[6128523.555087] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6128523.555582] x3 : 0000000000000024 x2 : 000031c040474000 >[6128523.556118] x1 : ffff4a2de304de60 x0 : 0000000000000024 >[6128523.556667] Call trace: >[6128523.556955] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6128523.557434] memcg_numa_stat_show+0x1d0/0x260 >[6128523.557866] cgroup_seqfile_show+0x50/0xd0 >[6128523.558330] kernfs_seq_show+0x30/0x38 >[6128523.558769] seq_read+0x148/0x478 >[6128523.559121] kernfs_fop_read+0x2c/0x1e0 >[6128523.559545] __vfs_read+0x20/0x48 >[6128523.559859] vfs_read+0x98/0x168 >[6128523.560253] ksys_read+0x6c/0xe8 >[6128523.560633] __arm64_sys_read+0x20/0x28 >[6128523.560987] el0_svc_common.constprop.0+0xa8/0x200 >[6128523.561513] el0_svc_handler+0x30/0x80 >[6128523.561885] el0_svc+0x10/0x14 >[6128567.531215] watchdog: BUG: soft lockup - CPU#12 stuck for 23s! [kubelet:18288] >[6128567.532085] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6128567.538704] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6128567.544169] CPU: 12 PID: 18288 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6128567.545306] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6128567.546160] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6128567.546610] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6128567.547118] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6128567.547609] sp : ffff00001f91bc20 >[6128567.547926] x29: ffff00001f91bc20 x28: ffff8bbc0f978000 >[6128567.548432] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6128567.548941] x25: 0000000000000003 x24: 0000000000000000 >[6128567.549454] x23: 0000000000000008 x22: ffff59fd7fe49800 >[6128567.549970] x21: ffff8dcd1fd9f400 x20: ffff59fd7fe4b8a8 >[6128567.550479] x19: 0000000000000001 x18: 0000000000000000 >[6128567.550983] x17: 0000000000000004 x16: 0000000000000000 >[6128567.551500] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6128567.552012] x13: ffff8b8d711a0000 x12: ffff8b8d7119f31b >[6128567.552548] x11: 0000000000000000 x10: ffff00001f91bc80 >[6128567.553064] x9 : ffff59fd7ef3dad0 x8 : 00000000000181c9 >[6128567.553576] x7 : 0000000000000004 x6 : 0000000000000005 >[6128567.554097] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6128567.554714] x3 : 0000000000000013 x2 : 000033a04045c000 >[6128567.555256] x1 : ffff4a1ceb647ff8 x0 : 0000000000000053 >[6128567.555812] Call trace: >[6128567.556047] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6128567.556577] memcg_numa_stat_show+0x1d0/0x260 >[6128567.557021] cgroup_seqfile_show+0x50/0xd0 >[6128567.557468] kernfs_seq_show+0x30/0x38 >[6128567.557826] seq_read+0x148/0x478 >[6128567.558181] kernfs_fop_read+0x2c/0x1e0 >[6128567.558581] __vfs_read+0x20/0x48 >[6128567.558941] vfs_read+0x98/0x168 >[6128567.559264] ksys_read+0x6c/0xe8 >[6128567.559600] __arm64_sys_read+0x20/0x28 >[6128567.559990] el0_svc_common.constprop.0+0xa8/0x200 >[6128567.560456] el0_svc_handler+0x30/0x80 >[6128567.560827] el0_svc+0x10/0x14 >[6128611.531003] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [migration/0:13] >[6128611.531257] watchdog: BUG: soft lockup - CPU#15 stuck for 23s! [kubelet:19532] >[6128611.531844] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6128611.532587] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6128611.539259] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6128611.546326] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6128611.552213] CPU: 0 PID: 13 Comm: migration/0 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6128611.558186] CPU: 15 PID: 19532 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6128611.559132] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6128611.560053] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6128611.560928] pstate: 60000005 (nZCv daif -PAN -UAO) >[6128611.561756] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6128611.562216] pc : multi_cpu_stop+0xf4/0x180 >[6128611.562664] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6128611.562668] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6128611.563083] lr : multi_cpu_stop+0xb4/0x180 >[6128611.563581] sp : ffff00001e943c20 >[6128611.564193] sp : ffff00000894bd50 >[6128611.564605] x29: ffff00001e943c20 x28: ffff8d8cf296c000 >[6128611.564930] x29: ffff00000894bd50 x28: 0000000000000000 >[6128611.565253] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6128611.565791] x27: ffff00001fec3938 x26: 0000000000000060 >[6128611.566392] x25: 0000000000000010 x24: 0000000000000000 >[6128611.567016] x25: 0000000000000000 x24: 0000000000000000 >[6128611.567521] x23: 0000000000000020 x22: ffff59fd7fe49800 >[6128611.568026] x23: 0000000000000000 x22: 0000000000000001 >[6128611.568630] x21: ffff8b9b7cb57c00 x20: ffff59fd7fe4b8a8 >[6128611.569135] x21: ffff00001fec39d4 x20: ffff00001fec39b0 >[6128611.569672] x19: 0000000000000004 x18: 0000000000000000 >[6128611.570225] x19: 0000000000000001 x18: 0000000000000000 >[6128611.570763] x17: 0000000000000002 x16: 0000000000000000 >[6128611.571322] x17: 0000000000000001 x16: 0000000000000000 >[6128611.571827] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6128611.572329] x15: 0000000000000000 x14: 0000000000000000 >[6128611.572851] x13: ffff8b8bdf96b000 x12: ffff8b8bdf96a34f >[6128611.573356] x13: 0000000000000000 x12: 0000000000000000 >[6128611.573859] x11: 0000000000000000 x10: ffff00001e943c80 >[6128611.574469] x11: 0000000000000017 x10: 0000000000000d60 >[6128611.574974] x9 : ffff59fd7ef3dad0 x8 : ffff8b8bdf96a000 >[6128611.575509] x9 : ffff59fd7ede5a0c x8 : ffff8b7d9d959f00 >[6128611.576058] x7 : 0000000000000000 x6 : 0000000000000001 >[6128611.576555] x7 : ffff8b7d1ccd1000 x6 : 00000000000002b9 >[6128611.577095] x5 : 00000000fffffffb x4 : ffffffffffffffff >[6128611.577619] x5 : 000031804040f000 x4 : ffff8b7d9d959740 >[6128611.578221] x3 : 0000000000000027 x2 : 000031c0404bf000 >[6128611.578763] x3 : dead000000000200 x2 : ffff00001fec3960 >[6128611.579268] x1 : ffff4a0cfdc22e30 x0 : 0000000000000027 >[6128611.579772] x1 : ffff00001fec39d4 x0 : 0000000000000001 >[6128611.580344] Call trace: >[6128611.580896] Call trace: >[6128611.581412] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6128611.581664] multi_cpu_stop+0xf4/0x180 >[6128611.581913] memcg_numa_stat_show+0x1d0/0x260 >[6128611.582395] cpu_stopper_thread+0xec/0x178 >[6128611.582755] cgroup_seqfile_show+0x50/0xd0 >[6128611.583177] smpboot_thread_fn+0x158/0x1d8 >[6128611.583587] kernfs_seq_show+0x30/0x38 >[6128611.583987] kthread+0x130/0x138 >[6128611.584477] seq_read+0x148/0x478 >[6128611.585582] kernfs_fop_read+0x2c/0x1e0 >[6128611.585972] __vfs_read+0x20/0x48 >[6128611.586399] vfs_read+0x98/0x168 >[6128611.586717] ksys_read+0x6c/0xe8 >[6128611.587058] __arm64_sys_read+0x20/0x28 >[6128611.587480] el0_svc_common.constprop.0+0xa8/0x200 >[6128611.587987] el0_svc_handler+0x30/0x80 >[6128611.588391] el0_svc+0x10/0x14 >[6128651.531412] watchdog: BUG: soft lockup - CPU#25 stuck for 22s! [migration/25:139] >[6128651.531661] watchdog: BUG: soft lockup - CPU#41 stuck for 22s! [kubelet:6818] >[6128651.532264] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6128651.532936] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6128651.539347] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6128651.546090] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6128651.551784] CPU: 25 PID: 139 Comm: migration/25 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6128651.557861] CPU: 41 PID: 6818 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6128651.558926] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6128651.559891] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6128651.560848] pstate: 60000005 (nZCv daif -PAN -UAO) >[6128651.561700] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6128651.562216] pc : multi_cpu_stop+0xf4/0x180 >[6128651.562219] lr : multi_cpu_stop+0xb4/0x180 >[6128651.562677] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6128651.563060] sp : ffff00000a13bd50 >[6128651.563465] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6128651.564077] x29: ffff00000a13bd50 x28: 0000000000000000 >[6128651.564387] sp : ffff000030643c20 >[6128651.564894] x27: ffff000061483938 x26: 0000000000000060 >[6128651.565392] x29: ffff000030643c20 x28: ffff8b8c6fc78000 >[6128651.565728] x25: 0000000000000000 x24: 0000000000000000 >[6128651.566242] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6128651.566747] x23: 0000000000000000 x22: 0000000000000001 >[6128651.567246] x25: 0000000000000003 x24: 0000000000000000 >[6128651.567771] x21: ffff0000614839d4 x20: ffff0000614839b0 >[6128651.568273] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6128651.568779] x19: 0000000000000001 x18: 0000000000000000 >[6128651.569347] x21: ffff8dacad184800 x20: ffff59fd7fe4b8a8 >[6128651.569843] x17: 0000000000000001 x16: 0000000000000000 >[6128651.570369] x19: 0000000000000000 x18: 0000000000000000 >[6128651.570862] x15: 0000000000000000 x14: 0000000000000000 >[6128651.571395] x17: 0000000000000002 x16: 0000000000000000 >[6128651.571951] x13: 0000000000000000 x12: 0000000000000000 >[6128651.572459] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6128651.572956] x11: 0000000000000000 x10: 0000000000000d60 >[6128651.573418] x13: ffff8bcd57f12000 x12: ffff8bcd57f11306 >[6128651.574042] x9 : ffff59fd7ede5a0c x8 : ffff8bad9e901f00 >[6128651.574520] x11: 0000000000000000 x10: ffff000030643c80 >[6128651.575011] x7 : 00113229815ff540 x6 : 0000000000000216 >[6128651.575508] x9 : ffff59fd7ef3dad0 x8 : 000000000000bc9b >[6128651.576014] x5 : 00000000f5257d14 x4 : ffff8bad9e901740 >[6128651.576525] x7 : 0000000000000004 x6 : 0000000000000005 >[6128651.577043] x3 : dead000000000200 x2 : ffff000061483960 >[6128651.577532] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6128651.578041] x1 : ffff0000614839d4 x0 : 0000000000000001 >[6128651.578526] x3 : 0000000000000014 x2 : 000031a040475000 >[6128651.579017] Call trace: >[6128651.579503] x1 : ffff4a1f2305d4b0 x0 : 0000000000000014 >[6128651.580009] multi_cpu_stop+0xf4/0x180 >[6128651.580256] Call trace: >[6128651.580752] cpu_stopper_thread+0xec/0x178 >[6128651.581102] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6128651.581348] smpboot_thread_fn+0x158/0x1d8 >[6128651.581706] memcg_numa_stat_show+0x1d0/0x260 >[6128651.582170] kthread+0x130/0x138 >[6128651.582548] cgroup_seqfile_show+0x50/0xd0 >[6128651.583631] kernfs_seq_show+0x30/0x38 >[6128651.584095] seq_read+0x148/0x478 >[6128651.584419] kernfs_fop_read+0x2c/0x1e0 >[6128651.584785] __vfs_read+0x20/0x48 >[6128651.585157] vfs_read+0x98/0x168 >[6128651.585464] ksys_read+0x6c/0xe8 >[6128651.585759] __arm64_sys_read+0x20/0x28 >[6128651.586143] el0_svc_common.constprop.0+0xa8/0x200 >[6128651.586585] el0_svc_handler+0x30/0x80 >[6128651.586960] el0_svc+0x10/0x14 >[6128695.531228] watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [kubelet:18930] >[6128695.532111] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6128695.538898] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6128695.544741] CPU: 13 PID: 18930 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6128695.545799] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6128695.546710] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6128695.547159] pc : mem_cgroup_nr_lru_pages.isra.7+0x9c/0xc8 >[6128695.547668] lr : mem_cgroup_nr_lru_pages.isra.7+0xac/0xc8 >[6128695.548168] sp : ffff0000303fbc20 >[6128695.548491] x29: ffff0000303fbc20 x28: 0000000000000000 >[6128695.549002] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6128695.549522] x25: 0000000000000010 x24: 0000000000000000 >[6128695.550033] x23: ffff8bac417fa5d0 x22: 0000000000000020 >[6128695.550519] x21: ffff59fd7fe49800 x20: ffff59fd7fe4b8a8 >[6128695.551029] x19: 0000000000000004 x18: 0000000000000000 >[6128695.551518] x17: 0000000000000002 x16: 0000000000000000 >[6128695.552016] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6128695.552556] x13: ffff8b8d7341e000 x12: ffff8b8d7341d326 >[6128695.553067] x11: 0000000000000000 x10: ffff0000303fbc80 >[6128695.553683] x9 : ffff59fd7ef3da4c x8 : 000000000000d97f >[6128695.554220] x7 : 0000000000000004 x6 : 0000000000000005 >[6128695.554740] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6128695.555242] x3 : 000000000000002b x2 : 000033d020060000 >[6128695.555717] x1 : ffff4a14c15a5ed0 x0 : 000000000000006b >[6128695.556202] Call trace: >[6128695.556453] mem_cgroup_nr_lru_pages.isra.7+0x9c/0xc8 >[6128695.556957] memcg_numa_stat_show+0x14c/0x260 >[6128695.557372] cgroup_seqfile_show+0x50/0xd0 >[6128695.557777] kernfs_seq_show+0x30/0x38 >[6128695.558143] seq_read+0x148/0x478 >[6128695.558466] kernfs_fop_read+0x2c/0x1e0 >[6128695.558820] __vfs_read+0x20/0x48 >[6128695.559131] vfs_read+0x98/0x168 >[6128695.559426] ksys_read+0x6c/0xe8 >[6128695.559749] __arm64_sys_read+0x20/0x28 >[6128695.560128] el0_svc_common.constprop.0+0xa8/0x200 >[6128695.560593] el0_svc_handler+0x30/0x80 >[6128695.560947] el0_svc+0x10/0x14 >[6128735.531088] watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [kubelet:18294] >[6128735.531893] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6128735.538724] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6128735.544781] CPU: 4 PID: 18294 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6128735.545787] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6128735.546663] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6128735.547114] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6128735.547665] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6128735.548171] sp : ffff00001f803c20 >[6128735.548506] x29: ffff00001f803c20 x28: ffff8bbb77a90000 >[6128735.549067] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6128735.549569] x25: 0000000000000003 x24: 0000000000000000 >[6128735.550086] x23: 0000000000000008 x22: ffff59fd7fe49800 >[6128735.550590] x21: ffff8d8ccc35e800 x20: ffff59fd7fe4b8a8 >[6128735.551123] x19: 0000000000000001 x18: 0000000000000000 >[6128735.551610] x17: 0000000000000002 x16: 0000000000000000 >[6128735.552129] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6128735.552636] x13: ffff8b7d1b324000 x12: ffff8b7d1b3232f3 >[6128735.553273] x11: 0000000000000000 x10: ffff00001f803c80 >[6128735.553784] x9 : ffff59fd7ef3dad0 x8 : 0000000000037a6e >[6128735.554279] x7 : 0000000000000005 x6 : 0000000000000006 >[6128735.554796] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6128735.555292] x3 : 0000000000000035 x2 : 000033d0402e8000 >[6128735.555858] x1 : ffff4a0ce9d67aa8 x0 : 0000000000000035 >[6128735.556398] Call trace: >[6128735.556651] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6128735.557221] memcg_numa_stat_show+0x1d0/0x260 >[6128735.557685] cgroup_seqfile_show+0x50/0xd0 >[6128735.558170] kernfs_seq_show+0x30/0x38 >[6128735.558534] seq_read+0x148/0x478 >[6128735.558859] kernfs_fop_read+0x2c/0x1e0 >[6128735.559224] __vfs_read+0x20/0x48 >[6128735.559551] vfs_read+0x98/0x168 >[6128735.559868] ksys_read+0x6c/0xe8 >[6128735.560198] __arm64_sys_read+0x20/0x28 >[6128735.560589] el0_svc_common.constprop.0+0xa8/0x200 >[6128735.561060] el0_svc_handler+0x30/0x80 >[6128735.561440] el0_svc+0x10/0x14 >[6128779.531939] watchdog: BUG: soft lockup - CPU#56 stuck for 22s! [kubelet:19532] >[6128779.532671] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6128779.540288] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6128779.546972] CPU: 56 PID: 19532 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6128779.548085] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6128779.549082] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6128779.549599] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6128779.550179] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6128779.550743] sp : ffff00001e943c20 >[6128779.551104] x29: ffff00001e943c20 x28: ffff8bac34024000 >[6128779.551710] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6128779.552286] x25: 0000000000000003 x24: 0000000000000000 >[6128779.552855] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6128779.553490] x21: ffff8d9cad790400 x20: ffff59fd7fe4b8a8 >[6128779.554047] x19: 0000000000000000 x18: 0000000000000000 >[6128779.554545] x17: 0000000000000002 x16: 0000000000000000 >[6128779.555032] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6128779.555523] x13: ffff8bbd2cf96000 x12: ffff8bbd2cf952fd >[6128779.556092] x11: 0000000000000000 x10: ffff00001e943c80 >[6128779.556610] x9 : ffff59fd7ef3dad0 x8 : 0000000000014ae8 >[6128779.557199] x7 : 0000000000000004 x6 : 0000000000000005 >[6128779.557677] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6128779.558204] x3 : 000000000000002c x2 : 000031d040475000 >[6128779.558739] x1 : ffff4a0fdf6a7230 x0 : 000000000000002c >[6128779.559274] Call trace: >[6128779.559548] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6128779.560034] memcg_numa_stat_show+0x1d0/0x260 >[6128779.560498] cgroup_seqfile_show+0x50/0xd0 >[6128779.560963] kernfs_seq_show+0x30/0x38 >[6128779.561319] seq_read+0x148/0x478 >[6128779.561684] kernfs_fop_read+0x2c/0x1e0 >[6128779.562074] __vfs_read+0x20/0x48 >[6128779.562464] vfs_read+0x98/0x168 >[6128779.562818] ksys_read+0x6c/0xe8 >[6128779.563221] __arm64_sys_read+0x20/0x28 >[6128779.563601] el0_svc_common.constprop.0+0xa8/0x200 >[6128779.564115] el0_svc_handler+0x30/0x80 >[6128779.564488] el0_svc+0x10/0x14 >[6128783.531254] watchdog: BUG: soft lockup - CPU#15 stuck for 22s! [migration/15:87] >[6128783.532083] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6128783.539382] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6128783.545185] CPU: 15 PID: 87 Comm: migration/15 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6128783.546179] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6128783.547036] pstate: 60000005 (nZCv daif -PAN -UAO) >[6128783.547493] pc : multi_cpu_stop+0xf4/0x180 >[6128783.547889] lr : multi_cpu_stop+0xb4/0x180 >[6128783.548258] sp : ffff000008f93d50 >[6128783.548586] x29: ffff000008f93d50 x28: 0000000000000000 >[6128783.549063] x27: ffff000054323938 x26: 0000000000000060 >[6128783.549556] x25: 0000000000000000 x24: 0000000000000000 >[6128783.550114] x23: 0000000000000000 x22: 0000000000000001 >[6128783.550596] x21: ffff0000543239d4 x20: ffff0000543239b0 >[6128783.551100] x19: 0000000000000001 x18: 0000000000000000 >[6128783.551589] x17: 0000000000000001 x16: 0000000000000000 >[6128783.552100] x15: 0000000000000000 x14: 0000000000000000 >[6128783.552610] x13: 0000000000000000 x12: 0000000000000000 >[6128783.553213] x11: 00000000000001db x10: 0000000000000d60 >[6128783.553710] x9 : ffff59fd7ede5a0c x8 : ffff8b8d9e9e4180 >[6128783.554245] x7 : ffff8b8d7c5a1000 x6 : 00000000000000d0 >[6128783.554754] x5 : 00003190404c0000 x4 : ffff8b8d9e9e39c0 >[6128783.555262] x3 : dead000000000200 x2 : ffff000054323960 >[6128783.555756] x1 : ffff0000543239d4 x0 : 0000000000000001 >[6128783.556255] Call trace: >[6128783.556495] multi_cpu_stop+0xf4/0x180 >[6128783.556857] cpu_stopper_thread+0xec/0x178 >[6128783.557241] smpboot_thread_fn+0x158/0x1d8 >[6128783.557618] kthread+0x130/0x138 >[6128823.531228] watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [kubelet:19592] >[6128823.532104] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6128823.538877] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6128823.544537] CPU: 13 PID: 19592 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6128823.545614] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6128823.546536] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6128823.546988] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6128823.547508] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6128823.548048] sp : ffff000030fcbc20 >[6128823.548376] x29: ffff000030fcbc20 x28: ffff8b7b67a9e000 >[6128823.548973] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6128823.549543] x25: 0000000000000003 x24: 0000000000000000 >[6128823.550131] x23: 0000000000000008 x22: ffff59fd7fe49800 >[6128823.550656] x21: ffff8dccc7343800 x20: ffff59fd7fe4b8a8 >[6128823.551213] x19: 0000000000000001 x18: 0000000000000000 >[6128823.551712] x17: 0000000000000002 x16: 0000000000000000 >[6128823.552232] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6128823.552891] x13: ffff8b8d7341b000 x12: ffff8b8d7341a31c >[6128823.553483] x11: 0000000000000000 x10: ffff000030fcbc80 >[6128823.554079] x9 : ffff59fd7ef3dad0 x8 : 00000000000182dc >[6128823.554629] x7 : 0000000000000004 x6 : 0000000000000005 >[6128823.555140] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6128823.555641] x3 : 0000000000000037 x2 : 000033d04031a000 >[6128823.556131] x1 : ffff4a0ce5e25088 x0 : 0000000000000037 >[6128823.556663] Call trace: >[6128823.556916] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6128823.557408] memcg_numa_stat_show+0x1d0/0x260 >[6128823.557847] cgroup_seqfile_show+0x50/0xd0 >[6128823.558241] kernfs_seq_show+0x30/0x38 >[6128823.558623] seq_read+0x148/0x478 >[6128823.558959] kernfs_fop_read+0x2c/0x1e0 >[6128823.559332] __vfs_read+0x20/0x48 >[6128823.559657] vfs_read+0x98/0x168 >[6128823.559979] ksys_read+0x6c/0xe8 >[6128823.560306] __arm64_sys_read+0x20/0x28 >[6128823.560676] el0_svc_common.constprop.0+0xa8/0x200 >[6128823.561130] el0_svc_handler+0x30/0x80 >[6128823.561486] el0_svc+0x10/0x14 >[6128867.531482] watchdog: BUG: soft lockup - CPU#29 stuck for 22s! [kubelet:18356] >[6128867.531672] watchdog: BUG: soft lockup - CPU#42 stuck for 22s! [migration/42:226] >[6128867.532336] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6128867.533124] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6128867.539506] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6128867.546010] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6128867.551757] CPU: 29 PID: 18356 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6128867.557519] CPU: 42 PID: 226 Comm: migration/42 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6128867.558508] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6128867.559485] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6128867.560289] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6128867.561115] pstate: 60000005 (nZCv daif -PAN -UAO) >[6128867.561615] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6128867.561619] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6128867.562112] pc : multi_cpu_stop+0x78/0x180 >[6128867.562688] sp : ffff00001edd3c20 >[6128867.563237] lr : multi_cpu_stop+0xb4/0x180 >[6128867.563624] x29: ffff00001edd3c20 x28: ffff8b7b75d6e000 >[6128867.563980] sp : ffff00000a3f3d50 >[6128867.564360] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6128867.564864] x29: ffff00000a3f3d50 x28: 0000000000000000 >[6128867.565196] x25: 0000000000000010 x24: 0000000000000000 >[6128867.565695] x27: ffff000034cab938 x26: 0000000000000060 >[6128867.566195] x23: 0000000000000020 x22: ffff59fd7fe49800 >[6128867.566678] x25: 0000000000000000 x24: 0000000000000000 >[6128867.567162] x21: ffff8b8ba9e7d000 x20: ffff59fd7fe4b8a8 >[6128867.567706] x23: 0000000000000000 x22: 0000000000000001 >[6128867.568246] x19: 0000000000000004 x18: 0000000000000000 >[6128867.568736] x21: ffff000034cab9d4 x20: ffff000034cab9b0 >[6128867.569218] x17: 0000000000000002 x16: 0000000000000000 >[6128867.569723] x19: 0000000000000001 x18: 0000000000000000 >[6128867.570241] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6128867.570734] x17: 0000000000000001 x16: 0000000000000000 >[6128867.571216] x13: ffff8bad55e5a000 x12: ffff8bad55e5934a >[6128867.571698] x15: 0000000000000001 x14: ffff8dbd9cf59c00 >[6128867.572207] x11: 0000000000000000 x10: ffff00001edd3c80 >[6128867.572789] x13: 0000000000000268 x12: ffff8dbbbfad5180 >[6128867.573291] x9 : ffff59fd7ef3dad0 x8 : ffff8bad55e59000 >[6128867.573823] x11: 000000000000001e x10: 0000000000000d60 >[6128867.574327] x7 : 0000000000000000 x6 : 0000000000000001 >[6128867.574823] x9 : ffff59fd7ede5a0c x8 : ffff8bcd9e8ff540 >[6128867.575350] x5 : 00000000fffffffb x4 : ffffffffffffffff >[6128867.575838] x7 : 00054500ee3b905e x6 : 00000000000003df >[6128867.576385] x3 : 0000000000000010 x2 : 000031a040411000 >[6128867.576898] x5 : 00000000fa83b2da x4 : ffff8bcd9e8fed80 >[6128867.577475] x1 : ffff4a2b5e304060 x0 : 0000000000000010 >[6128867.578054] x3 : dead000000000200 x2 : ffff000034cab988 >[6128867.578610] Call trace: >[6128867.579131] x1 : ffff000034cab9d4 x0 : 0000000000000001 >[6128867.579700] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6128867.579966] Call trace: >[6128867.580559] memcg_numa_stat_show+0x1d0/0x260 >[6128867.581044] multi_cpu_stop+0x78/0x180 >[6128867.581343] cgroup_seqfile_show+0x50/0xd0 >[6128867.581752] cpu_stopper_thread+0xec/0x178 >[6128867.582163] kernfs_seq_show+0x30/0x38 >[6128867.582655] smpboot_thread_fn+0x158/0x1d8 >[6128867.583061] seq_read+0x148/0x478 >[6128867.583433] kthread+0x130/0x138 >[6128867.583831] kernfs_fop_read+0x2c/0x1e0 >[6128867.584876] __vfs_read+0x20/0x48 >[6128867.585263] vfs_read+0x98/0x168 >[6128867.585559] ksys_read+0x6c/0xe8 >[6128867.585879] __arm64_sys_read+0x20/0x28 >[6128867.586326] el0_svc_common.constprop.0+0xa8/0x200 >[6128867.586866] el0_svc_handler+0x30/0x80 >[6128867.587259] el0_svc+0x10/0x14 >[6128907.531016] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kubelet:37248] >[6128907.531804] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6128907.539257] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6128907.545607] CPU: 1 PID: 37248 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6128907.546721] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6128907.547678] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6128907.548188] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6128907.548780] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6128907.549363] sp : ffff00001b93bc20 >[6128907.549715] x29: ffff00001b93bc20 x28: ffff8b7d512f4000 >[6128907.550207] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6128907.550676] x25: 0000000000000003 x24: 0000000000000000 >[6128907.551160] x23: 0000000000000008 x22: ffff59fd7fe49800 >[6128907.551644] x21: ffff8dad9889a400 x20: ffff59fd7fe4b8a8 >[6128907.552254] x19: 0000000000000001 x18: 0000000000000000 >[6128907.552787] x17: 0000000000000002 x16: 0000000000000000 >[6128907.553293] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6128907.553828] x13: ffff8b7d1ada2000 x12: ffff8b7d1ada1306 >[6128907.554330] x11: 0000000000000000 x10: ffff00001b93bc80 >[6128907.554833] x9 : ffff59fd7ef3dad0 x8 : 000000000000eaf5 >[6128907.555341] x7 : 0000000000000004 x6 : 0000000000000005 >[6128907.555845] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6128907.556351] x3 : 000000000000001c x2 : 000033b040475000 >[6128907.556874] x1 : ffff4a26428f74e8 x0 : 000000000000005c >[6128907.557383] Call trace: >[6128907.557628] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6128907.558095] memcg_numa_stat_show+0x1d0/0x260 >[6128907.558519] cgroup_seqfile_show+0x50/0xd0 >[6128907.558916] kernfs_seq_show+0x30/0x38 >[6128907.559279] seq_read+0x148/0x478 >[6128907.559616] kernfs_fop_read+0x2c/0x1e0 >[6128907.560000] __vfs_read+0x20/0x48 >[6128907.560339] vfs_read+0x98/0x168 >[6128907.560667] ksys_read+0x6c/0xe8 >[6128907.561001] __arm64_sys_read+0x20/0x28 >[6128907.561371] el0_svc_common.constprop.0+0xa8/0x200 >[6128907.561855] el0_svc_handler+0x30/0x80 >[6128907.562370] el0_svc+0x10/0x14 >[6128951.531369] watchdog: BUG: soft lockup - CPU#22 stuck for 22s! [kubelet:4440] >[6128951.532258] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6128951.538412] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6128951.544068] CPU: 22 PID: 4440 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6128951.545015] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6128951.545818] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6128951.546294] pc : find_next_bit+0x24/0x68 >[6128951.546656] lr : cpumask_next+0x1c/0x28 >[6128951.547063] sp : ffff0000329abc10 >[6128951.547385] x29: ffff0000329abc10 x28: ffff8b8d53f1a000 >[6128951.547924] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6128951.548436] x25: 0000000000000003 x24: 0000000000000000 >[6128951.548970] x23: 0000000000000008 x22: ffff59fd7fe49800 >[6128951.549448] x21: ffff8dbd16d7fc00 x20: ffff59fd7fe4b8a8 >[6128951.549923] x19: 0000000000000001 x18: 0000000000000000 >[6128951.550489] x17: 0000000000000002 x16: 0000000000000000 >[6128951.551003] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6128951.551601] x13: ffff8b9d2055c000 x12: ffff8b9d2055b310 >[6128951.552220] x11: 0000000000000000 x10: ffff0000329abc80 >[6128951.552789] x9 : ffff59fd7ef3dad0 x8 : 000000000004f0bc >[6128951.553275] x7 : 0000000000000005 x6 : 0000000000000006 >[6128951.553746] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6128951.554208] x3 : fffffffffffff000 x2 : 0000000000000000 >[6128951.554694] x1 : 0000000000000400 x0 : ffff59fd7fe4b8a8 >[6128951.555185] Call trace: >[6128951.555432] find_next_bit+0x24/0x68 >[6128951.555762] mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6128951.556217] memcg_numa_stat_show+0x1d0/0x260 >[6128951.556640] cgroup_seqfile_show+0x50/0xd0 >[6128951.557060] kernfs_seq_show+0x30/0x38 >[6128951.557479] seq_read+0x148/0x478 >[6128951.557792] kernfs_fop_read+0x2c/0x1e0 >[6128951.558177] __vfs_read+0x20/0x48 >[6128951.558492] vfs_read+0x98/0x168 >[6128951.558789] ksys_read+0x6c/0xe8 >[6128951.559093] __arm64_sys_read+0x20/0x28 >[6128951.559474] el0_svc_common.constprop.0+0xa8/0x200 >[6128951.559900] el0_svc_handler+0x30/0x80 >[6128951.560245] el0_svc+0x10/0x14 >[6128955.531225] watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [migration/13:77] >[6128955.532153] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6128955.538672] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6128955.544511] CPU: 13 PID: 77 Comm: migration/13 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6128955.545622] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6128955.546491] pstate: 60000005 (nZCv daif -PAN -UAO) >[6128955.546993] pc : multi_cpu_stop+0xf4/0x180 >[6128955.547385] lr : multi_cpu_stop+0xb4/0x180 >[6128955.547787] sp : ffff000008f43d50 >[6128955.548111] x29: ffff000008f43d50 x28: 0000000000000000 >[6128955.548619] x27: ffff0000551c3938 x26: 0000000000000060 >[6128955.549118] x25: 0000000000000000 x24: 0000000000000000 >[6128955.549642] x23: 0000000000000000 x22: 0000000000000001 >[6128955.550155] x21: ffff0000551c39d4 x20: ffff0000551c39b0 >[6128955.550656] x19: 0000000000000001 x18: 0000000000000000 >[6128955.551143] x17: 0000000000000001 x16: 0000000000000000 >[6128955.551731] x15: 0000000000000000 x14: 0000000000000000 >[6128955.552212] x13: 0000000000000000 x12: 0000000000000000 >[6128955.552736] x11: 0000000000000000 x10: 0000000000000d60 >[6128955.553233] x9 : ffff59fd7ede5a0c x8 : ffff8b8d9e998dc0 >[6128955.553728] x7 : ffff8b8d72c94000 x6 : 00000000000000ab >[6128955.554227] x5 : 000031904048e000 x4 : ffff8b8d9e998600 >[6128955.554736] x3 : dead000000000200 x2 : ffff0000551c3960 >[6128955.555293] x1 : ffff0000551c39d4 x0 : 0000000000000001 >[6128955.555856] Call trace: >[6128955.556100] multi_cpu_stop+0xf4/0x180 >[6128955.556517] cpu_stopper_thread+0xec/0x178 >[6128955.556898] smpboot_thread_fn+0x158/0x1d8 >[6128955.557297] kthread+0x130/0x138 >[6128995.531325] watchdog: BUG: soft lockup - CPU#19 stuck for 22s! [kubelet:18891] >[6128995.532133] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6128995.538281] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6128995.543768] CPU: 19 PID: 18891 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6128995.544842] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6128995.545660] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6128995.546140] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6128995.546610] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6128995.547089] sp : ffff0000302fbc20 >[6128995.547406] x29: ffff0000302fbc20 x28: ffff8b7bba57c000 >[6128995.547916] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6128995.548409] x25: 0000000000000003 x24: 0000000000000000 >[6128995.548907] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6128995.549387] x21: ffff8dbbedb17800 x20: ffff59fd7fe4b8a8 >[6128995.549892] x19: 0000000000000000 x18: 0000000000000000 >[6128995.550378] x17: 0000000000000002 x16: 0000000000000000 >[6128995.550866] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6128995.551453] x13: ffff8b9d20230000 x12: ffff8b9d2022f313 >[6128995.551960] x11: 0000000000000000 x10: ffff0000302fbc80 >[6128995.552435] x9 : ffff59fd7ef3dad0 x8 : 000000000004edfa >[6128995.552928] x7 : 0000000000000005 x6 : 0000000000000006 >[6128995.553414] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6128995.553907] x3 : 0000000000000000 x2 : 000031804040f000 >[6128995.554451] x1 : ffff4a0f06f34a00 x0 : 0000000000000000 >[6128995.554926] Call trace: >[6128995.555168] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6128995.555657] memcg_numa_stat_show+0x1d0/0x260 >[6128995.556068] cgroup_seqfile_show+0x50/0xd0 >[6128995.556487] kernfs_seq_show+0x30/0x38 >[6128995.556856] seq_read+0x148/0x478 >[6128995.557173] kernfs_fop_read+0x2c/0x1e0 >[6128995.557519] __vfs_read+0x20/0x48 >[6128995.557857] vfs_read+0x98/0x168 >[6128995.558168] ksys_read+0x6c/0xe8 >[6128995.558488] __arm64_sys_read+0x20/0x28 >[6128995.558835] el0_svc_common.constprop.0+0xa8/0x200 >[6128995.559282] el0_svc_handler+0x30/0x80 >[6128995.559649] el0_svc+0x10/0x14 >[6129039.531493] watchdog: BUG: soft lockup - CPU#30 stuck for 22s! [kubelet:18890] >[6129039.532291] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6129039.538609] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6129039.544486] CPU: 30 PID: 18890 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6129039.545513] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6129039.546307] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6129039.546752] pc : mem_cgroup_nr_lru_pages.isra.7+0x9c/0xc8 >[6129039.547244] lr : mem_cgroup_nr_lru_pages.isra.7+0xac/0xc8 >[6129039.547754] sp : ffff00001fd9bc20 >[6129039.548066] x29: ffff00001fd9bc20 x28: 0000000000000000 >[6129039.548581] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6129039.549098] x25: 0000000000000010 x24: 0000000000000000 >[6129039.549583] x23: ffff8dbbe31b25d0 x22: 0000000000000020 >[6129039.550071] x21: ffff59fd7fe49800 x20: ffff59fd7fe4b8a8 >[6129039.550585] x19: 0000000000000004 x18: 0000000000000000 >[6129039.551187] x17: 0000000000000002 x16: 0000000000000000 >[6129039.551699] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6129039.552202] x13: ffff8bac181e5000 x12: ffff8bac181e432a >[6129039.552720] x11: 0000000000000000 x10: ffff00001fd9bc80 >[6129039.553239] x9 : ffff59fd7ef3da4c x8 : 000000000000deeb >[6129039.553767] x7 : 0000000000000004 x6 : 0000000000000005 >[6129039.554270] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6129039.554775] x3 : 0000000000000039 x2 : 000033d01fe9e000 >[6129039.555260] x1 : ffff4a1dcdd4d0e0 x0 : 0000000000000079 >[6129039.555751] Call trace: >[6129039.555999] mem_cgroup_nr_lru_pages.isra.7+0x9c/0xc8 >[6129039.556497] memcg_numa_stat_show+0x14c/0x260 >[6129039.556900] cgroup_seqfile_show+0x50/0xd0 >[6129039.557315] kernfs_seq_show+0x30/0x38 >[6129039.557686] seq_read+0x148/0x478 >[6129039.557996] kernfs_fop_read+0x2c/0x1e0 >[6129039.558343] __vfs_read+0x20/0x48 >[6129039.558665] vfs_read+0x98/0x168 >[6129039.558966] ksys_read+0x6c/0xe8 >[6129039.559283] __arm64_sys_read+0x20/0x28 >[6129039.559644] el0_svc_common.constprop.0+0xa8/0x200 >[6129039.560111] el0_svc_handler+0x30/0x80 >[6129039.560484] el0_svc+0x10/0x14 >[6129079.531214] watchdog: BUG: soft lockup - CPU#12 stuck for 23s! [kubelet:18303] >[6129079.532011] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6129079.538349] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6129079.544259] CPU: 12 PID: 18303 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6129079.545248] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6129079.546125] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6129079.546580] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6129079.547120] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6129079.547629] sp : ffff00001f93bc20 >[6129079.547961] x29: ffff00001f93bc20 x28: ffff8d8d943e6000 >[6129079.548472] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6129079.548990] x25: 0000000000000003 x24: 0000000000000000 >[6129079.549506] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6129079.550024] x21: ffff8d9c47fdfc00 x20: ffff59fd7fe4b8a8 >[6129079.550558] x19: 0000000000000000 x18: 0000000000000000 >[6129079.551205] x17: 0000000000000631 x16: 0000000000000000 >[6129079.551708] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6129079.552200] x13: ffff8b8d7119f000 x12: ffff8b8d7119e300 >[6129079.552764] x11: 0000000000000000 x10: ffff00001f93bc80 >[6129079.553293] x9 : ffff59fd7ef3dad0 x8 : 000000000001582c >[6129079.553830] x7 : 0000000000000004 x6 : 0000000000000005 >[6129079.554315] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6129079.554846] x3 : 0000000000000022 x2 : 000033c040442000 >[6129079.555374] x1 : ffff4a2b6314a2b0 x0 : 0000000000000062 >[6129079.555895] Call trace: >[6129079.556202] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6129079.556678] memcg_numa_stat_show+0x1d0/0x260 >[6129079.557115] cgroup_seqfile_show+0x50/0xd0 >[6129079.557479] kernfs_seq_show+0x30/0x38 >[6129079.557825] seq_read+0x148/0x478 >[6129079.558135] kernfs_fop_read+0x2c/0x1e0 >[6129079.558533] __vfs_read+0x20/0x48 >[6129079.558911] vfs_read+0x98/0x168 >[6129079.559200] ksys_read+0x6c/0xe8 >[6129079.559527] __arm64_sys_read+0x20/0x28 >[6129079.559944] el0_svc_common.constprop.0+0xa8/0x200 >[6129079.560403] el0_svc_handler+0x30/0x80 >[6129079.560871] el0_svc+0x10/0x14 >[6129123.531208] watchdog: BUG: soft lockup - CPU#12 stuck for 23s! [migration/12:72] >[6129123.531371] watchdog: BUG: soft lockup - CPU#22 stuck for 23s! [kubelet:18356] >[6129123.532088] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6129123.532789] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6129123.539170] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6129123.545565] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6129123.551281] CPU: 12 PID: 72 Comm: migration/12 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6129123.556776] CPU: 22 PID: 18356 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6129123.557768] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6129123.558732] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6129123.559589] pstate: 60000005 (nZCv daif -PAN -UAO) >[6129123.560388] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6129123.560982] pc : multi_cpu_stop+0x78/0x180 >[6129123.560985] lr : multi_cpu_stop+0xb4/0x180 >[6129123.561468] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6129123.561920] sp : ffff000008f1bd50 >[6129123.562324] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6129123.562326] sp : ffff00001edd3c20 >[6129123.562867] x29: ffff000008f1bd50 x28: 0000000000000000 >[6129123.563221] x29: ffff00001edd3c20 x28: ffff8bbca9bd0000 >[6129123.563768] x27: ffff00005ae9b938 x26: 0000000000000060 >[6129123.564067] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6129123.564714] x25: 0000000000000000 x24: 0000000000000000 >[6129123.565234] x25: 0000000000000003 x24: 0000000000000000 >[6129123.565764] x23: 0000000000000000 x22: 0000000000000001 >[6129123.566268] x23: 0000000000000008 x22: ffff59fd7fe49800 >[6129123.566767] x21: ffff00005ae9b9d4 x20: ffff00005ae9b9b0 >[6129123.567230] x21: ffff8dacad183c00 x20: ffff59fd7fe4b8a8 >[6129123.567734] x19: 0000000000000001 x18: 0000000000000000 >[6129123.568225] x19: 0000000000000001 x18: 0000000000000000 >[6129123.568723] x17: 0000000000000001 x16: 0000000000000000 >[6129123.569191] x17: 0000000000000002 x16: 0000000000000000 >[6129123.569712] x15: 0000000000000000 x14: 0000000000000000 >[6129123.570189] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6129123.570797] x13: 0000000000000000 x12: 0000000000000000 >[6129123.571287] x13: ffff8b9d2055c000 x12: ffff8b9d2055b30a >[6129123.571801] x11: 0000000000000000 x10: 0000000000000d60 >[6129123.572289] x11: 0000000000000000 x10: ffff00001edd3c80 >[6129123.572805] x9 : ffff59fd7ede5a0c x8 : ffff8b8d9e953040 >[6129123.573291] x9 : ffff59fd7ef3dad0 x8 : 000000000000e102 >[6129123.573809] x7 : ffff8b8d7c5a2c00 x6 : 0000000000000374 >[6129123.574345] x7 : 0000000000000004 x6 : 0000000000000005 >[6129123.574871] x5 : 0000319040475000 x4 : ffff8b8d9e952880 >[6129123.575350] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6129123.575969] x3 : dead000000000200 x2 : ffff00005ae9b960 >[6129123.576480] x3 : 0000000000000023 x2 : 000033c04045b000 >[6129123.576969] x1 : ffff00005ae9b9d4 x0 : 0000000000000001 >[6129123.577444] x1 : ffff4a2dcc5067d8 x0 : 0000000000000063 >[6129123.577926] Call trace: >[6129123.578467] Call trace: >[6129123.579005] multi_cpu_stop+0x78/0x180 >[6129123.579537] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6129123.579785] cpu_stopper_thread+0xec/0x178 >[6129123.580042] memcg_numa_stat_show+0x1d0/0x260 >[6129123.580555] smpboot_thread_fn+0x158/0x1d8 >[6129123.581021] cgroup_seqfile_show+0x50/0xd0 >[6129123.581401] kthread+0x130/0x138 >[6129123.581800] kernfs_seq_show+0x30/0x38 >[6129123.581804] seq_read+0x148/0x478 >[6129123.583613] kernfs_fop_read+0x2c/0x1e0 >[6129123.584039] __vfs_read+0x20/0x48 >[6129123.584411] vfs_read+0x98/0x168 >[6129123.584743] ksys_read+0x6c/0xe8 >[6129123.585089] __arm64_sys_read+0x20/0x28 >[6129123.585481] el0_svc_common.constprop.0+0xa8/0x200 >[6129123.585976] el0_svc_handler+0x30/0x80 >[6129123.586325] el0_svc+0x10/0x14 >[6129167.531402] watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [kubelet:19532] >[6129167.532186] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6129167.538746] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6129167.544451] CPU: 24 PID: 19532 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6129167.545422] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6129167.546291] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6129167.546730] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6129167.547207] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6129167.547681] sp : ffff00001e943c20 >[6129167.547981] x29: ffff00001e943c20 x28: ffff8bbc54858000 >[6129167.548506] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6129167.548999] x25: 0000000000000003 x24: 0000000000000000 >[6129167.549496] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6129167.549997] x21: ffff8dcd2d947400 x20: ffff59fd7fe4b8a8 >[6129167.550595] x19: 0000000000000000 x18: 0000000000000000 >[6129167.551094] x17: 0000000000000004 x16: 0000000000000000 >[6129167.551603] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6129167.552179] x13: ffff8bad569cf000 x12: ffff8bad569ce31c >[6129167.552693] x11: 0000000000000000 x10: ffff00001e943c80 >[6129167.553216] x9 : ffff59fd7ef3dad0 x8 : 0000000000018483 >[6129167.553733] x7 : 0000000000000004 x6 : 0000000000000005 >[6129167.554195] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6129167.554703] x3 : 0000000000000033 x2 : 000033d01ff98000 >[6129167.555281] x1 : ffff4a1f1d0450c0 x0 : 0000000000000073 >[6129167.555821] Call trace: >[6129167.556092] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6129167.556610] memcg_numa_stat_show+0x1d0/0x260 >[6129167.557083] cgroup_seqfile_show+0x50/0xd0 >[6129167.557493] kernfs_seq_show+0x30/0x38 >[6129167.557940] seq_read+0x148/0x478 >[6129167.558264] kernfs_fop_read+0x2c/0x1e0 >[6129167.558644] __vfs_read+0x20/0x48 >[6129167.558957] vfs_read+0x98/0x168 >[6129167.559277] ksys_read+0x6c/0xe8 >[6129167.559582] __arm64_sys_read+0x20/0x28 >[6129167.559948] el0_svc_common.constprop.0+0xa8/0x200 >[6129167.560523] el0_svc_handler+0x30/0x80 >[6129167.560918] el0_svc+0x10/0x14 >[6129171.531350] watchdog: BUG: soft lockup - CPU#21 stuck for 22s! [migration/21:118] >[6129171.532147] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6129171.538245] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6129171.543783] CPU: 21 PID: 118 Comm: migration/21 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6129171.544771] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6129171.545592] pstate: 60000005 (nZCv daif -PAN -UAO) >[6129171.546022] pc : multi_cpu_stop+0xf4/0x180 >[6129171.546391] lr : multi_cpu_stop+0xb4/0x180 >[6129171.546767] sp : ffff00000a093d50 >[6129171.547089] x29: ffff00000a093d50 x28: 0000000000000000 >[6129171.547578] x27: ffff00001f8db938 x26: 0000000000000060 >[6129171.548058] x25: 0000000000000000 x24: 0000000000000000 >[6129171.548587] x23: 0000000000000000 x22: 0000000000000001 >[6129171.549066] x21: ffff00001f8db9d4 x20: ffff00001f8db9b0 >[6129171.549557] x19: 0000000000000001 x18: 0000000000000000 >[6129171.550171] x17: 0000000000000001 x16: 0000000000000000 >[6129171.550669] x15: 0000000000000000 x14: 0000000000000000 >[6129171.551146] x13: 0000000000000000 x12: 0000000000000000 >[6129171.551653] x11: 0000000000000025 x10: 0000000000000d60 >[6129171.552181] x9 : ffff59fd7ede5a0c x8 : ffff8b9d9e914180 >[6129171.552679] x7 : ffff8b9d24ffcc00 x6 : 0000000000000119 >[6129171.553177] x5 : 000031a04048e000 x4 : ffff8b9d9e9139c0 >[6129171.553654] x3 : dead000000000200 x2 : ffff00001f8db960 >[6129171.554118] x1 : ffff00001f8db9d4 x0 : 0000000000000001 >[6129171.554593] Call trace: >[6129171.554848] multi_cpu_stop+0xf4/0x180 >[6129171.555211] cpu_stopper_thread+0xec/0x178 >[6129171.555607] smpboot_thread_fn+0x158/0x1d8 >[6129171.556008] kthread+0x130/0x138 >[6129211.531197] watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [kubelet:18294] >[6129211.532052] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6129211.538606] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6129211.544374] CPU: 11 PID: 18294 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6129211.545370] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6129211.546269] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6129211.546743] pc : find_next_bit+0x24/0x68 >[6129211.547123] lr : cpumask_next+0x1c/0x28 >[6129211.547493] sp : ffff00001f803c10 >[6129211.547821] x29: ffff00001f803c10 x28: ffff8bcc38ed4000 >[6129211.548344] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6129211.548862] x25: 0000000000000010 x24: 0000000000000000 >[6129211.549362] x23: 0000000000000020 x22: ffff59fd7fe49800 >[6129211.549952] x21: ffff8b8bab71ec00 x20: ffff59fd7fe4b8a8 >[6129211.550459] x19: 0000000000000004 x18: 0000000000000000 >[6129211.551034] x17: 0000000000000002 x16: 0000000000000000 >[6129211.551548] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6129211.552071] x13: ffff8b8d8045a000 x12: ffff8b8d8045934c >[6129211.552596] x11: 0000000000000000 x10: ffff00001f803c80 >[6129211.553181] x9 : ffff59fd7ef3dad0 x8 : ffff8b8d80459000 >[6129211.553672] x7 : 0000000000000000 x6 : 0000000000000001 >[6129211.554231] x5 : 00000000fffffffb x4 : ffffffffffffffff >[6129211.554769] x3 : ffffffc000000000 x2 : 0000000000000040 >[6129211.555437] x1 : 0000000000000400 x0 : ffff59fd7fe4b8a8 >[6129211.555994] Call trace: >[6129211.556242] find_next_bit+0x24/0x68 >[6129211.556635] mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6129211.557163] memcg_numa_stat_show+0x1d0/0x260 >[6129211.557577] cgroup_seqfile_show+0x50/0xd0 >[6129211.558015] kernfs_seq_show+0x30/0x38 >[6129211.558366] seq_read+0x148/0x478 >[6129211.558729] kernfs_fop_read+0x2c/0x1e0 >[6129211.559114] __vfs_read+0x20/0x48 >[6129211.559494] vfs_read+0x98/0x168 >[6129211.559913] ksys_read+0x6c/0xe8 >[6129211.560293] __arm64_sys_read+0x20/0x28 >[6129211.560675] el0_svc_common.constprop.0+0xa8/0x200 >[6129211.561187] el0_svc_handler+0x30/0x80 >[6129211.561548] el0_svc+0x10/0x14 >[6129251.531167] watchdog: BUG: soft lockup - CPU#9 stuck for 23s! [kubelet:19531] >[6129251.532003] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6129251.538711] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6129251.544557] CPU: 9 PID: 19531 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6129251.545584] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6129251.546421] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6129251.546887] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6129251.547442] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6129251.547931] sp : ffff00001fd4bc20 >[6129251.548241] x29: ffff00001fd4bc20 x28: ffff8b8c5f756000 >[6129251.548785] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6129251.549253] x25: 0000000000000003 x24: 0000000000000000 >[6129251.549844] x23: 0000000000000008 x22: ffff59fd7fe49800 >[6129251.550361] x21: ffff8dbc48257c00 x20: ffff59fd7fe4b8a8 >[6129251.550863] x19: 0000000000000001 x18: 0000000000000000 >[6129251.551382] x17: 0000000000000002 x16: 0000000000000000 >[6129251.551883] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6129251.552371] x13: ffff8b9d205b9000 x12: ffff8b9d205b8311 >[6129251.552883] x11: 0000000000000000 x10: ffff00001fd4bc80 >[6129251.553396] x9 : ffff59fd7ef3dad0 x8 : 000000000004f021 >[6129251.553896] x7 : 0000000000000005 x6 : 0000000000000006 >[6129251.554391] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6129251.554892] x3 : 000000000000003d x2 : 000033d01ff02000 >[6129251.555389] x1 : ffff4a1cfcf96368 x0 : 000000000000007d >[6129251.555898] Call trace: >[6129251.556133] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6129251.556631] memcg_numa_stat_show+0x1d0/0x260 >[6129251.557049] cgroup_seqfile_show+0x50/0xd0 >[6129251.557456] kernfs_seq_show+0x30/0x38 >[6129251.557824] seq_read+0x148/0x478 >[6129251.558123] kernfs_fop_read+0x2c/0x1e0 >[6129251.558490] __vfs_read+0x20/0x48 >[6129251.558825] vfs_read+0x98/0x168 >[6129251.559141] ksys_read+0x6c/0xe8 >[6129251.559453] __arm64_sys_read+0x20/0x28 >[6129251.559943] el0_svc_common.constprop.0+0xa8/0x200 >[6129251.560400] el0_svc_handler+0x30/0x80 >[6129251.560815] el0_svc+0x10/0x14 >[6129255.531021] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [migration/2:21] >[6129255.531885] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6129255.538782] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6129255.544575] CPU: 2 PID: 21 Comm: migration/2 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6129255.545536] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6129255.546388] pstate: 60000005 (nZCv daif -PAN -UAO) >[6129255.546895] pc : multi_cpu_stop+0x78/0x180 >[6129255.547288] lr : multi_cpu_stop+0xb4/0x180 >[6129255.547715] sp : ffff000008d83d50 >[6129255.548050] x29: ffff000008d83d50 x28: 0000000000000000 >[6129255.548609] x27: ffff00003f4f3938 x26: 0000000000000060 >[6129255.549098] x25: 0000000000000000 x24: 0000000000000000 >[6129255.549674] x23: 0000000000000000 x22: 0000000000000001 >[6129255.550171] x21: ffff00003f4f39d4 x20: ffff00003f4f39b0 >[6129255.550678] x19: 0000000000000001 x18: 0000000000000000 >[6129255.551227] x17: 0000000000000001 x16: 0000000000000000 >[6129255.551773] x15: 0000000000000000 x14: 0000000000000000 >[6129255.552279] x13: 0000000000000000 x12: 0000000000000000 >[6129255.552833] x11: 0000000000000072 x10: 0000000000000d60 >[6129255.553328] x9 : ffff59fd7ede5a0c x8 : ffff8b7d9d9c3040 >[6129255.553826] x7 : 0002ad2f15bda0e0 x6 : 0000000000000391 >[6129255.554328] x5 : 00000000fa83b2da x4 : ffff8b7d9d9c2880 >[6129255.554846] x3 : dead000000000200 x2 : ffff00003f4f3960 >[6129255.555352] x1 : ffff00003f4f39d4 x0 : 0000000000000001 >[6129255.555850] Call trace: >[6129255.556103] multi_cpu_stop+0x78/0x180 >[6129255.556467] cpu_stopper_thread+0xec/0x178 >[6129255.556876] smpboot_thread_fn+0x158/0x1d8 >[6129255.557281] kthread+0x130/0x138 >[6129295.531153] watchdog: BUG: soft lockup - CPU#8 stuck for 23s! [kubelet:18288] >[6129295.531966] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6129295.538505] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6129295.544510] CPU: 8 PID: 18288 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6129295.545506] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6129295.546376] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6129295.546851] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6129295.547430] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6129295.547939] sp : ffff00001f91bc20 >[6129295.548241] x29: ffff00001f91bc20 x28: ffff8b9b7ec46000 >[6129295.548761] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6129295.549358] x25: 0000000000000010 x24: 0000000000000000 >[6129295.549850] x23: 0000000000000020 x22: ffff59fd7fe49800 >[6129295.550360] x21: ffff8b7b631d0000 x20: ffff59fd7fe4b8a8 >[6129295.550862] x19: 0000000000000004 x18: 0000000000000000 >[6129295.551366] x17: 0000000000000002 x16: 0000000000000000 >[6129295.551896] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6129295.552387] x13: ffff8b8d72fa9000 x12: ffff8b8d72fa8346 >[6129295.552940] x11: 0000000000000000 x10: ffff00001f91bc80 >[6129295.553437] x9 : ffff59fd7ef3dad0 x8 : 0000000000000021 >[6129295.553964] x7 : 0000000000000001 x6 : 0000000000000002 >[6129295.554501] x5 : 00000000fffffffa x4 : ffffffffffffffff >[6129295.555026] x3 : 000000000000002f x2 : 000031d0404c0000 >[6129295.555560] x1 : ffff4a2dcd31c698 x0 : 000000000000002f >[6129295.556067] Call trace: >[6129295.556314] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6129295.556784] memcg_numa_stat_show+0x1d0/0x260 >[6129295.557202] cgroup_seqfile_show+0x50/0xd0 >[6129295.557623] kernfs_seq_show+0x30/0x38 >[6129295.557982] seq_read+0x148/0x478 >[6129295.558297] kernfs_fop_read+0x2c/0x1e0 >[6129295.558655] __vfs_read+0x20/0x48 >[6129295.558975] vfs_read+0x98/0x168 >[6129295.559409] ksys_read+0x6c/0xe8 >[6129295.559740] __arm64_sys_read+0x20/0x28 >[6129295.560116] el0_svc_common.constprop.0+0xa8/0x200 >[6129295.560590] el0_svc_handler+0x30/0x80 >[6129295.560937] el0_svc+0x10/0x14 >[6129335.531070] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [migration/3:26] >[6129335.531212] watchdog: BUG: soft lockup - CPU#12 stuck for 22s! [kubelet:4439] >[6129335.531938] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6129335.532589] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6129335.539030] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6129335.545497] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6129335.551497] CPU: 3 PID: 26 Comm: migration/3 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6129335.557250] CPU: 12 PID: 4439 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6129335.558319] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6129335.559447] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6129335.560365] pstate: 60000005 (nZCv daif -PAN -UAO) >[6129335.561284] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6129335.561787] pc : multi_cpu_stop+0xf4/0x180 >[6129335.562241] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6129335.562716] lr : multi_cpu_stop+0xb4/0x180 >[6129335.563313] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6129335.563722] sp : ffff000008dabd50 >[6129335.564247] sp : ffff00001e7ebc20 >[6129335.564566] x29: ffff000008dabd50 x28: 0000000000000000 >[6129335.564946] x29: ffff00001e7ebc20 x28: ffff8b8ccc964000 >[6129335.565488] x27: ffff000062ffb938 x26: 0000000000000060 >[6129335.565990] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6129335.566539] x25: 0000000000000000 x24: 0000000000000000 >[6129335.567019] x25: 0000000000000003 x24: 0000000000000000 >[6129335.567525] x23: 0000000000000000 x22: 0000000000000001 >[6129335.568045] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6129335.568586] x21: ffff000062ffb9d4 x20: ffff000062ffb9b0 >[6129335.569219] x21: ffff8dad30ff0400 x20: ffff59fd7fe4b8a8 >[6129335.569736] x19: 0000000000000001 x18: 0000000000000000 >[6129335.570311] x19: 0000000000000000 x18: 0000000000000000 >[6129335.570853] x17: 0000000000000001 x16: 0000000000000000 >[6129335.571351] x17: 0000000000000002 x16: 0000000000000000 >[6129335.571917] x15: 0000000000000001 x14: ffff8dbc5d0e7400 >[6129335.572437] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6129335.572946] x13: 00000000000004f8 x12: ffff8dbc5d588580 >[6129335.573498] x13: ffff8b8d7119c000 x12: ffff8b8d7119b307 >[6129335.574010] x11: 0000000000000025 x10: 0000000000000d60 >[6129335.574533] x11: 0000000000000000 x10: ffff00001e7ebc80 >[6129335.575060] x9 : ffff59fd7ede5a0c x8 : ffff8b7d9da18dc0 >[6129335.575624] x9 : ffff59fd7ef3dad0 x8 : 000000000000d494 >[6129335.576178] x7 : ffff8b7d6b1b3000 x6 : 0000000000000022 >[6129335.576684] x7 : 0000000000000004 x6 : 0000000000000005 >[6129335.577216] x5 : 000031804045a000 x4 : ffff8b7d9da18600 >[6129335.577723] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6129335.578259] x3 : dead000000000200 x2 : ffff000062ffb960 >[6129335.578772] x3 : 0000000000000036 x2 : 000033d040301000 >[6129335.579452] x1 : ffff000062ffb9d4 x0 : 0000000000000001 >[6129335.579934] x1 : ffff4a29c3aba7a0 x0 : 0000000000000036 >[6129335.580437] Call trace: >[6129335.580988] Call trace: >[6129335.581304] multi_cpu_stop+0xf4/0x180 >[6129335.581543] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6129335.581909] cpu_stopper_thread+0xec/0x178 >[6129335.582495] memcg_numa_stat_show+0x1d0/0x260 >[6129335.582922] smpboot_thread_fn+0x158/0x1d8 >[6129335.583377] cgroup_seqfile_show+0x50/0xd0 >[6129335.583824] kthread+0x130/0x138 >[6129335.584222] kernfs_seq_show+0x30/0x38 >[6129335.584918] seq_read+0x148/0x478 >[6129335.585248] kernfs_fop_read+0x2c/0x1e0 >[6129335.585591] __vfs_read+0x20/0x48 >[6129335.585938] vfs_read+0x98/0x168 >[6129335.586280] ksys_read+0x6c/0xe8 >[6129335.586621] __arm64_sys_read+0x20/0x28 >[6129335.587009] el0_svc_common.constprop.0+0xa8/0x200 >[6129335.587484] el0_svc_handler+0x30/0x80 >[6129335.587848] el0_svc+0x10/0x14 >[6129379.531010] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kubelet:7308] >[6129379.531748] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6129379.538374] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6129379.544312] CPU: 0 PID: 7308 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6129379.545316] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6129379.546219] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6129379.546681] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6129379.547195] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6129379.547700] sp : ffff00001b39bc20 >[6129379.548039] x29: ffff00001b39bc20 x28: ffff8b9c31702000 >[6129379.548661] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6129379.549295] x25: 0000000000000003 x24: 0000000000000000 >[6129379.549792] x23: 0000000000000008 x22: ffff59fd7fe49800 >[6129379.550305] x21: ffff8dcd0db15400 x20: ffff59fd7fe4b8a8 >[6129379.550867] x19: 0000000000000001 x18: 0000000000000000 >[6129379.551376] x17: 0000000000000002 x16: 0000000000000000 >[6129379.551904] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6129379.552420] x13: ffff8b8d71fcc000 x12: ffff8b8d71fcb31c >[6129379.552931] x11: 0000000000000000 x10: ffff00001b39bc80 >[6129379.553447] x9 : ffff59fd7ef3dad0 x8 : 0000000000018671 >[6129379.553984] x7 : 0000000000000004 x6 : 0000000000000005 >[6129379.554504] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6129379.555041] x3 : 0000000000000036 x2 : 000033d01ffe3000 >[6129379.555534] x1 : ffff4a1b6392f398 x0 : 0000000000000076 >[6129379.556116] Call trace: >[6129379.556370] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6129379.556928] memcg_numa_stat_show+0x1d0/0x260 >[6129379.557352] cgroup_seqfile_show+0x50/0xd0 >[6129379.557782] kernfs_seq_show+0x30/0x38 >[6129379.558137] seq_read+0x148/0x478 >[6129379.558465] kernfs_fop_read+0x2c/0x1e0 >[6129379.558945] __vfs_read+0x20/0x48 >[6129379.559301] vfs_read+0x98/0x168 >[6129379.559622] ksys_read+0x6c/0xe8 >[6129379.559964] __arm64_sys_read+0x20/0x28 >[6129379.560342] el0_svc_common.constprop.0+0xa8/0x200 >[6129379.560821] el0_svc_handler+0x30/0x80 >[6129379.561193] el0_svc+0x10/0x14 >[6129423.531293] watchdog: BUG: soft lockup - CPU#17 stuck for 22s! [kubelet:13350] >[6129423.532067] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6129423.538376] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6129423.543811] CPU: 17 PID: 13350 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6129423.544803] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6129423.545639] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6129423.546102] pc : mem_cgroup_nr_lru_pages.isra.7+0x9c/0xc8 >[6129423.546582] lr : mem_cgroup_nr_lru_pages.isra.7+0xac/0xc8 >[6129423.547046] sp : ffff0000575d3c20 >[6129423.547353] x29: ffff0000575d3c20 x28: 0000000000000000 >[6129423.547909] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6129423.548514] x25: 0000000000000010 x24: 0000000000000000 >[6129423.549090] x23: ffff8bbcc564a5d0 x22: 0000000000000020 >[6129423.549695] x21: ffff59fd7fe49800 x20: ffff59fd7fe4b8a8 >[6129423.550281] x19: 0000000000000004 x18: 0000000000000000 >[6129423.550859] x17: 0000000000000002 x16: 0000000000000000 >[6129423.551451] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6129423.552038] x13: ffff8b9b753c5000 x12: ffff8b9b753c4326 >[6129423.552660] x11: 0000000000000000 x10: ffff0000575d3c80 >[6129423.553146] x9 : ffff59fd7ef3da4c x8 : 000000000000dce7 >[6129423.553751] x7 : 0000000000000004 x6 : 0000000000000005 >[6129423.554354] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6129423.554943] x3 : 000000000000001b x2 : 000033b04045c000 >[6129423.555512] x1 : ffff4a2b63d02460 x0 : 000000000000005b >[6129423.556117] Call trace: >[6129423.556399] mem_cgroup_nr_lru_pages.isra.7+0x9c/0xc8 >[6129423.557006] memcg_numa_stat_show+0x14c/0x260 >[6129423.557493] cgroup_seqfile_show+0x50/0xd0 >[6129423.557936] kernfs_seq_show+0x30/0x38 >[6129423.558445] seq_read+0x148/0x478 >[6129423.558794] kernfs_fop_read+0x2c/0x1e0 >[6129423.559177] __vfs_read+0x20/0x48 >[6129423.559511] vfs_read+0x98/0x168 >[6129423.559899] ksys_read+0x6c/0xe8 >[6129423.560197] __arm64_sys_read+0x20/0x28 >[6129423.560551] el0_svc_common.constprop.0+0xa8/0x200 >[6129423.561037] el0_svc_handler+0x30/0x80 >[6129423.561391] el0_svc+0x10/0x14 >[6129467.531027] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kubelet:6818] >[6129467.531815] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6129467.538513] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6129467.544243] CPU: 2 PID: 6818 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6129467.545301] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6129467.546261] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6129467.546707] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6129467.547257] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6129467.547762] sp : ffff000030643c20 >[6129467.548169] x29: ffff000030643c20 x28: ffff8d9cbd220000 >[6129467.548709] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6129467.549313] x25: 0000000000000010 x24: 0000000000000000 >[6129467.549847] x23: 0000000000000020 x22: ffff59fd7fe49800 >[6129467.550380] x21: ffff8b8c37922000 x20: ffff59fd7fe4b8a8 >[6129467.550917] x19: 0000000000000004 x18: 0000000000000000 >[6129467.551417] x17: 0000000000000002 x16: 0000000000000000 >[6129467.551976] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6129467.552528] x13: ffff8b7ba68fa000 x12: ffff8b7ba68f934c >[6129467.553061] x11: 0000000000000000 x10: ffff000030643c80 >[6129467.553594] x9 : ffff59fd7ef3dad0 x8 : ffff8b7ba68f9000 >[6129467.554172] x7 : 0000000000000000 x6 : 0000000000000001 >[6129467.554711] x5 : 00000000fffffffb x4 : ffffffffffffffff >[6129467.555277] x3 : 000000000000000e x2 : 00003390404a7000 >[6129467.555834] x1 : ffff4a2802c074b0 x0 : 000000000000004e >[6129467.556424] Call trace: >[6129467.556680] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6129467.557158] memcg_numa_stat_show+0x1d0/0x260 >[6129467.557590] cgroup_seqfile_show+0x50/0xd0 >[6129467.558080] kernfs_seq_show+0x30/0x38 >[6129467.558448] seq_read+0x148/0x478 >[6129467.558784] kernfs_fop_read+0x2c/0x1e0 >[6129467.559165] __vfs_read+0x20/0x48 >[6129467.559495] vfs_read+0x98/0x168 >[6129467.559803] ksys_read+0x6c/0xe8 >[6129467.560141] __arm64_sys_read+0x20/0x28 >[6129467.560517] el0_svc_common.constprop.0+0xa8/0x200 >[6129467.560977] el0_svc_handler+0x30/0x80 >[6129467.561335] el0_svc+0x10/0x14 >[6129507.531259] watchdog: BUG: soft lockup - CPU#15 stuck for 22s! [kubelet:18303] >[6129507.532185] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6129507.538686] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6129507.544290] CPU: 15 PID: 18303 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6129507.545302] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6129507.546199] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6129507.546729] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6129507.547310] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6129507.547893] sp : ffff00001f93bc20 >[6129507.548237] x29: ffff00001f93bc20 x28: ffff8bad4c1d8000 >[6129507.548747] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6129507.549329] x25: 0000000000000003 x24: 0000000000000000 >[6129507.549911] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6129507.550412] x21: ffff8dad945c4000 x20: ffff59fd7fe4b8a8 >[6129507.550949] x19: 0000000000000000 x18: 0000000000000000 >[6129507.551492] x17: 0000000000000002 x16: 0000000000000000 >[6129507.551997] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6129507.552549] x13: ffff8b8bdf96c000 x12: ffff8b8bdf96b308 >[6129507.553043] x11: 0000000000000000 x10: ffff00001f93bc80 >[6129507.553631] x9 : ffff59fd7ef3dad0 x8 : 000000000000d637 >[6129507.554222] x7 : 0000000000000004 x6 : 0000000000000005 >[6129507.554728] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6129507.555303] x3 : 000000000000003b x2 : 000033d0401ee000 >[6129507.555811] x1 : ffff4a27029b1be0 x0 : 000000000000003b >[6129507.556310] Call trace: >[6129507.556566] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6129507.557048] memcg_numa_stat_show+0x1d0/0x260 >[6129507.557466] cgroup_seqfile_show+0x50/0xd0 >[6129507.557955] kernfs_seq_show+0x30/0x38 >[6129507.558361] seq_read+0x148/0x478 >[6129507.558723] kernfs_fop_read+0x2c/0x1e0 >[6129507.559090] __vfs_read+0x20/0x48 >[6129507.559404] vfs_read+0x98/0x168 >[6129507.559795] ksys_read+0x6c/0xe8 >[6129507.560128] __arm64_sys_read+0x20/0x28 >[6129507.560562] el0_svc_common.constprop.0+0xa8/0x200 >[6129507.561023] el0_svc_handler+0x30/0x80 >[6129507.561425] el0_svc+0x10/0x14 >[6129551.531152] watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [kubelet:18888] >[6129551.531978] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6129551.538512] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6129551.544148] CPU: 8 PID: 18888 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6129551.545194] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6129551.546155] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6129551.546635] pc : mem_cgroup_nr_lru_pages.isra.7+0x9c/0xc8 >[6129551.547130] lr : mem_cgroup_nr_lru_pages.isra.7+0xac/0xc8 >[6129551.547786] sp : ffff00001fec3c20 >[6129551.548110] x29: ffff00001fec3c20 x28: 0000000000000000 >[6129551.548722] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6129551.549267] x25: 0000000000000010 x24: 0000000000000000 >[6129551.549771] x23: ffff8b9ccbb665d0 x22: 0000000000000020 >[6129551.550298] x21: ffff59fd7fe49800 x20: ffff59fd7fe4b8a8 >[6129551.550865] x19: 0000000000000004 x18: 0000000000000000 >[6129551.551369] x17: 0000000000000008 x16: 0000000000000000 >[6129551.551918] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6129551.552415] x13: ffff8b8d72fb0000 x12: ffff8b8d72faf328 >[6129551.552928] x11: 0000000000000000 x10: ffff00001fec3c80 >[6129551.553525] x9 : ffff59fd7ef3da4c x8 : 000000000000df97 >[6129551.554015] x7 : 0000000000000004 x6 : 0000000000000005 >[6129551.554625] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6129551.555204] x3 : 0000000000000022 x2 : 000033c040442000 >[6129551.555701] x1 : ffff4a27426867d0 x0 : 0000000000000062 >[6129551.556323] Call trace: >[6129551.556574] mem_cgroup_nr_lru_pages.isra.7+0x9c/0xc8 >[6129551.557088] memcg_numa_stat_show+0x14c/0x260 >[6129551.557605] cgroup_seqfile_show+0x50/0xd0 >[6129551.558008] kernfs_seq_show+0x30/0x38 >[6129551.558367] seq_read+0x148/0x478 >[6129551.558685] kernfs_fop_read+0x2c/0x1e0 >[6129551.559051] __vfs_read+0x20/0x48 >[6129551.559364] vfs_read+0x98/0x168 >[6129551.559686] ksys_read+0x6c/0xe8 >[6129551.560009] __arm64_sys_read+0x20/0x28 >[6129551.560374] el0_svc_common.constprop.0+0xa8/0x200 >[6129551.560832] el0_svc_handler+0x30/0x80 >[6129551.561196] el0_svc+0x10/0x14 >[6129591.531650] watchdog: BUG: soft lockup - CPU#40 stuck for 22s! [kubelet:18804] >[6129591.532459] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6129591.538778] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6129591.544281] CPU: 40 PID: 18804 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6129591.545200] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6129591.546045] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6129591.546486] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6129591.546984] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6129591.547669] sp : ffff00001fe63c20 >[6129591.548046] x29: ffff00001fe63c20 x28: ffff8b7b699b8000 >[6129591.548631] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6129591.549142] x25: 0000000000000003 x24: 0000000000000000 >[6129591.549687] x23: 0000000000000008 x22: ffff59fd7fe49800 >[6129591.550269] x21: ffff8d8ccce9bc00 x20: ffff59fd7fe4b8a8 >[6129591.550730] x19: 0000000000000001 x18: 0000000000000000 >[6129591.551241] x17: 0000000000000009 x16: 0000000000000000 >[6129591.551749] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6129591.552272] x13: ffff8bcbee281000 x12: ffff8bcbee2802f4 >[6129591.552789] x11: 0000000000000000 x10: ffff00001fe63c80 >[6129591.553322] x9 : ffff59fd7ef3dad0 x8 : 0000000000037f97 >[6129591.553854] x7 : 0000000000000005 x6 : 0000000000000006 >[6129591.554423] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6129591.554945] x3 : 0000000000000003 x2 : 000033804045a000 >[6129591.555477] x1 : ffff4a0cfce1faf8 x0 : 0000000000000043 >[6129591.555981] Call trace: >[6129591.556224] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6129591.556755] memcg_numa_stat_show+0x1d0/0x260 >[6129591.557282] cgroup_seqfile_show+0x50/0xd0 >[6129591.557717] kernfs_seq_show+0x30/0x38 >[6129591.558121] seq_read+0x148/0x478 >[6129591.558443] kernfs_fop_read+0x2c/0x1e0 >[6129591.558788] __vfs_read+0x20/0x48 >[6129591.559111] vfs_read+0x98/0x168 >[6129591.559450] ksys_read+0x6c/0xe8 >[6129591.559802] __arm64_sys_read+0x20/0x28 >[6129591.560168] el0_svc_common.constprop.0+0xa8/0x200 >[6129591.560663] el0_svc_handler+0x30/0x80 >[6129591.561016] el0_svc+0x10/0x14 >[6129635.531178] watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [migration/10:62] >[6129635.531275] watchdog: BUG: soft lockup - CPU#16 stuck for 22s! [kubelet:18890] >[6129635.532123] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6129635.532911] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6129635.540295] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6129635.547007] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6129635.552656] CPU: 10 PID: 62 Comm: migration/10 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6129635.558406] CPU: 16 PID: 18890 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6129635.559509] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6129635.560589] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6129635.561545] pstate: 60000005 (nZCv daif -PAN -UAO) >[6129635.562506] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6129635.562923] pc : multi_cpu_stop+0xf4/0x180 >[6129635.563363] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6129635.563806] lr : multi_cpu_stop+0xb4/0x180 >[6129635.563809] sp : ffff000008ecbd50 >[6129635.564282] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6129635.564727] x29: ffff000008ecbd50 x28: 0000000000000000 >[6129635.565032] sp : ffff00001fd9bc20 >[6129635.565623] x27: ffff0000302fb938 x26: 0000000000000060 >[6129635.566094] x29: ffff00001fd9bc20 x28: ffff8b8c33094000 >[6129635.566421] x25: 0000000000000000 x24: 0000000000000000 >[6129635.566977] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6129635.567485] x23: 0000000000000000 x22: 0000000000000001 >[6129635.567967] x25: 0000000000000003 x24: 0000000000000000 >[6129635.568458] x21: ffff0000302fb9d4 x20: ffff0000302fb9b0 >[6129635.568941] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6129635.569448] x19: 0000000000000001 x18: 0000000000000000 >[6129635.569954] x21: ffff8d9c81a77c00 x20: ffff59fd7fe4b8a8 >[6129635.570483] x17: 0000000000000001 x16: 0000000000000000 >[6129635.570996] x19: 0000000000000000 x18: 0000000000000000 >[6129635.571503] x15: 0000000000000000 x14: 0000000000000000 >[6129635.572028] x17: 0000000000000002 x16: 0000000000000000 >[6129635.572541] x13: 0000000000000000 x12: 0000000000000000 >[6129635.573062] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6129635.573569] x11: 000000000000000c x10: 0000000000000d60 >[6129635.574082] x13: ffff8b9ba2454000 x12: ffff8b9ba24532fe >[6129635.574582] x9 : ffff59fd7ede5a0c x8 : ffff8b8d9e907540 >[6129635.575052] x11: 0000000000000000 x10: ffff00001fd9bc80 >[6129635.575538] x7 : ffff8b8d721ba400 x6 : 0000000000000238 >[6129635.576029] x9 : ffff59fd7ef3dad0 x8 : 0000000000015719 >[6129635.576543] x5 : 0000319040443000 x4 : ffff8b8d9e906d80 >[6129635.577117] x7 : 0000000000000004 x6 : 0000000000000005 >[6129635.577688] x3 : dead000000000200 x2 : ffff0000302fb960 >[6129635.578158] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6129635.578762] x1 : ffff0000302fb9d4 x0 : 0000000000000001 >[6129635.579300] x3 : 0000000000000017 x2 : 000031a0404c0000 >[6129635.579808] Call trace: >[6129635.580340] x1 : ffff4a2dda1616d0 x0 : 0000000000000017 >[6129635.580853] multi_cpu_stop+0xf4/0x180 >[6129635.581114] Call trace: >[6129635.581636] cpu_stopper_thread+0xec/0x178 >[6129635.582000] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6129635.582247] smpboot_thread_fn+0x158/0x1d8 >[6129635.582627] memcg_numa_stat_show+0x1d0/0x260 >[6129635.582633] cgroup_seqfile_show+0x50/0xd0 >[6129635.583116] kthread+0x130/0x138 >[6129635.583502] kernfs_seq_show+0x30/0x38 >[6129635.585007] seq_read+0x148/0x478 >[6129635.585339] kernfs_fop_read+0x2c/0x1e0 >[6129635.585721] __vfs_read+0x20/0x48 >[6129635.586061] vfs_read+0x98/0x168 >[6129635.586376] ksys_read+0x6c/0xe8 >[6129635.586694] __arm64_sys_read+0x20/0x28 >[6129635.587180] el0_svc_common.constprop.0+0xa8/0x200 >[6129635.587686] el0_svc_handler+0x30/0x80 >[6129635.588074] el0_svc+0x10/0x14 >[6129679.531279] watchdog: BUG: soft lockup - CPU#16 stuck for 22s! [kubelet:18280] >[6129679.532182] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6129679.539515] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6129679.544986] CPU: 16 PID: 18280 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6129679.545917] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6129679.546889] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6129679.547328] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6129679.547904] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6129679.548466] sp : ffff00001f893c20 >[6129679.548772] x29: ffff00001f893c20 x28: ffff8babe3b7e000 >[6129679.549285] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6129679.549776] x25: 0000000000000003 x24: 0000000000000000 >[6129679.550283] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6129679.550794] x21: ffff8d9c22e28800 x20: ffff59fd7fe4b8a8 >[6129679.551272] x19: 0000000000000000 x18: 0000000000000000 >[6129679.551763] x17: 0000000000000004 x16: 0000000000000000 >[6129679.552242] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6129679.552718] x13: ffff8b9ba2455000 x12: ffff8b9ba24542fe >[6129679.553214] x11: 0000000000000000 x10: ffff00001f893c80 >[6129679.553709] x9 : ffff59fd7ef3dad0 x8 : 000000000001599b >[6129679.554181] x7 : 0000000000000004 x6 : 0000000000000005 >[6129679.554670] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6129679.555186] x3 : 000000000000002d x2 : 000033d020092000 >[6129679.555695] x1 : ffff4a2b460583f0 x0 : 000000000000006d >[6129679.556212] Call trace: >[6129679.556572] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6129679.557077] memcg_numa_stat_show+0x1d0/0x260 >[6129679.557546] cgroup_seqfile_show+0x50/0xd0 >[6129679.557959] kernfs_seq_show+0x30/0x38 >[6129679.558338] seq_read+0x148/0x478 >[6129679.558673] kernfs_fop_read+0x2c/0x1e0 >[6129679.559147] __vfs_read+0x20/0x48 >[6129679.559471] vfs_read+0x98/0x168 >[6129679.559789] ksys_read+0x6c/0xe8 >[6129679.560098] __arm64_sys_read+0x20/0x28 >[6129679.560454] el0_svc_common.constprop.0+0xa8/0x200 >[6129679.560935] el0_svc_handler+0x30/0x80 >[6129679.561282] el0_svc+0x10/0x14 >[6129723.531018] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kubelet:4439] >[6129723.531852] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6129723.538598] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6129723.544524] CPU: 1 PID: 4439 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6129723.545531] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6129723.546556] pstate: a0000005 (NzCv daif -PAN -UAO) >[6129723.547022] pc : mem_cgroup_node_nr_lru_pages+0x50/0x128 >[6129723.547526] lr : memcg_numa_stat_show+0x1d0/0x260 >[6129723.548000] sp : ffff00001e7ebc20 >[6129723.548330] x29: ffff00001e7ebc20 x28: ffff8bbbb1d52000 >[6129723.548853] x27: ffff59fd7f671bf8 x26: ffff59fd7f8b5a88 >[6129723.549388] x25: 0000000000000010 x24: ffff59fd7fe4c9a0 >[6129723.549977] x23: ffff8b7d1cb3f080 x22: ffff59fd7f671e38 >[6129723.550481] x21: ffff8dbbb25c7c00 x20: 000000000000000c >[6129723.551036] x19: 0000000000004e49 x18: 0000000000000000 >[6129723.551545] x17: 0000000000000009 x16: 0000000000000000 >[6129723.552056] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6129723.552574] x13: ffff8b7d1ada6000 x12: ffff8b7d1ada5312 >[6129723.553109] x11: 0000000000000000 x10: ffff00001e7ebc80 >[6129723.553629] x9 : ffff59fd7ef3dad0 x8 : 000000000004edf2 >[6129723.554137] x7 : 0000000000000005 x6 : 0000000000000006 >[6129723.554727] x5 : 00000000fffffff6 x4 : ffff8dbd9e88a000 >[6129723.555289] x3 : 00000000000002a4 x2 : 0000000000000003 >[6129723.555933] x1 : ffff8dbdbfffe380 x0 : ffff8bbbb1d52000 >[6129723.556547] Call trace: >[6129723.556847] mem_cgroup_node_nr_lru_pages+0x50/0x128 >[6129723.557320] memcg_numa_stat_show+0x1d0/0x260 >[6129723.557741] cgroup_seqfile_show+0x50/0xd0 >[6129723.558241] kernfs_seq_show+0x30/0x38 >[6129723.558628] seq_read+0x148/0x478 >[6129723.558945] kernfs_fop_read+0x2c/0x1e0 >[6129723.559330] __vfs_read+0x20/0x48 >[6129723.559643] vfs_read+0x98/0x168 >[6129723.559931] ksys_read+0x6c/0xe8 >[6129723.560255] __arm64_sys_read+0x20/0x28 >[6129723.560634] el0_svc_common.constprop.0+0xa8/0x200 >[6129723.561079] el0_svc_handler+0x30/0x80 >[6129723.561445] el0_svc+0x10/0x14 >[6129767.531540] watchdog: BUG: soft lockup - CPU#33 stuck for 22s! [kubelet:18929] >[6129767.532381] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6129767.538922] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6129767.544617] CPU: 33 PID: 18929 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6129767.545610] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6129767.546644] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6129767.547151] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6129767.547673] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6129767.548219] sp : ffff00003001bc20 >[6129767.548592] x29: ffff00003001bc20 x28: ffff8b9c1cb8a000 >[6129767.549140] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6129767.549657] x25: 0000000000000003 x24: 0000000000000000 >[6129767.550138] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6129767.550716] x21: ffff8dcd62800800 x20: ffff59fd7fe4b8a8 >[6129767.551319] x19: 0000000000000000 x18: 0000000000000000 >[6129767.551821] x17: 0000000000000004 x16: 0000000000000000 >[6129767.552398] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6129767.552945] x13: ffff8bbd33bb7000 x12: ffff8bbd33bb631d >[6129767.553445] x11: 0000000000000000 x10: ffff00003001bc80 >[6129767.553992] x9 : ffff59fd7ef3dad0 x8 : 0000000000018bae >[6129767.554501] x7 : 0000000000000005 x6 : 0000000000000006 >[6129767.555016] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6129767.555555] x3 : 0000000000000028 x2 : 000033d020015000 >[6129767.556175] x1 : ffff4a1ce5d3f070 x0 : 0000000000000068 >[6129767.556765] Call trace: >[6129767.557016] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6129767.557527] memcg_numa_stat_show+0x1d0/0x260 >[6129767.557959] cgroup_seqfile_show+0x50/0xd0 >[6129767.558383] kernfs_seq_show+0x30/0x38 >[6129767.558767] seq_read+0x148/0x478 >[6129767.559085] kernfs_fop_read+0x2c/0x1e0 >[6129767.559475] __vfs_read+0x20/0x48 >[6129767.559775] vfs_read+0x98/0x168 >[6129767.560066] ksys_read+0x6c/0xe8 >[6129767.560411] __arm64_sys_read+0x20/0x28 >[6129767.560791] el0_svc_common.constprop.0+0xa8/0x200 >[6129767.561246] el0_svc_handler+0x30/0x80 >[6129767.561620] el0_svc+0x10/0x14 >[6129811.532056] watchdog: BUG: soft lockup - CPU#62 stuck for 22s! [kubelet:18304] >[6129811.532796] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6129811.539592] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6129811.545716] CPU: 62 PID: 18304 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6129811.546710] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6129811.547554] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6129811.548032] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6129811.548537] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6129811.549031] sp : ffff00001f943c20 >[6129811.549345] x29: ffff00001f943c20 x28: ffff8bcca9602000 >[6129811.549829] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6129811.550344] x25: 0000000000000010 x24: 0000000000000000 >[6129811.550830] x23: 0000000000000020 x22: ffff59fd7fe49800 >[6129811.551335] x21: ffff8b7c43a09400 x20: ffff59fd7fe4b8a8 >[6129811.551815] x19: 0000000000000004 x18: 0000000000000000 >[6129811.552313] x17: 0000000000000002 x16: 0000000000000000 >[6129811.552833] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6129811.553364] x13: ffff8bbd3393b000 x12: ffff8bbd3393a347 >[6129811.553864] x11: 0000000000000000 x10: ffff00001f943c80 >[6129811.554353] x9 : ffff59fd7ef3dad0 x8 : 0000000000000021 >[6129811.554832] x7 : 0000000000000001 x6 : 0000000000000002 >[6129811.555319] x5 : 00000000fffffffa x4 : ffffffffffffffff >[6129811.555976] x3 : 000000000000002b x2 : 000033d020060000 >[6129811.556463] x1 : ffff4a2120567130 x0 : 000000000000006b >[6129811.556951] Call trace: >[6129811.557194] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6129811.557670] memcg_numa_stat_show+0x1d0/0x260 >[6129811.558123] cgroup_seqfile_show+0x50/0xd0 >[6129811.558507] kernfs_seq_show+0x30/0x38 >[6129811.558885] seq_read+0x148/0x478 >[6129811.559200] kernfs_fop_read+0x2c/0x1e0 >[6129811.559577] __vfs_read+0x20/0x48 >[6129811.559900] vfs_read+0x98/0x168 >[6129811.560241] ksys_read+0x6c/0xe8 >[6129811.560561] __arm64_sys_read+0x20/0x28 >[6129811.560979] el0_svc_common.constprop.0+0xa8/0x200 >[6129811.561495] el0_svc_handler+0x30/0x80 >[6129811.561866] el0_svc+0x10/0x14 >[6129851.531277] watchdog: BUG: soft lockup - CPU#16 stuck for 23s! [kubelet:19531] >[6129851.532015] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6129851.538344] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6129851.543936] CPU: 16 PID: 19531 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6129851.544904] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6129851.545932] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6129851.546394] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6129851.546986] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6129851.547495] sp : ffff00001fd4bc20 >[6129851.547922] x29: ffff00001fd4bc20 x28: ffff8d8cc762c000 >[6129851.548409] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6129851.548974] x25: 0000000000000003 x24: 0000000000000000 >[6129851.549465] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6129851.550016] x21: ffff8d7d47c40800 x20: ffff59fd7fe4b8a8 >[6129851.550625] x19: 0000000000000000 x18: 0000000000000000 >[6129851.551094] x17: 0000000000000002 x16: 0000000000000000 >[6129851.551673] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6129851.552167] x13: ffff8b9ba2453000 x12: ffff8b9ba24522ea >[6129851.552735] x11: 0000000000000000 x10: ffff00001fd4bc80 >[6129851.553215] x9 : ffff59fd7ef3dad0 x8 : 000000000002260d >[6129851.553758] x7 : 0000000000000005 x6 : 0000000000000006 >[6129851.554226] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6129851.554770] x3 : 0000000000000037 x2 : 000033d01fffc000 >[6129851.555336] x1 : ffff4a2b4a445770 x0 : 0000000000000077 >[6129851.555803] Call trace: >[6129851.556081] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6129851.556576] memcg_numa_stat_show+0x1d0/0x260 >[6129851.557025] cgroup_seqfile_show+0x50/0xd0 >[6129851.557415] kernfs_seq_show+0x30/0x38 >[6129851.557760] seq_read+0x148/0x478 >[6129851.558078] kernfs_fop_read+0x2c/0x1e0 >[6129851.558430] __vfs_read+0x20/0x48 >[6129851.558737] vfs_read+0x98/0x168 >[6129851.559038] ksys_read+0x6c/0xe8 >[6129851.559324] __arm64_sys_read+0x20/0x28 >[6129851.559666] el0_svc_common.constprop.0+0xa8/0x200 >[6129851.560088] el0_svc_handler+0x30/0x80 >[6129851.560422] el0_svc+0x10/0x14 >[6129859.531070] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [migration/3:26] >[6129859.531949] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6129859.538994] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6129859.544894] CPU: 3 PID: 26 Comm: migration/3 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6129859.546017] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6129859.546907] pstate: 60000005 (nZCv daif -PAN -UAO) >[6129859.547384] pc : multi_cpu_stop+0x78/0x180 >[6129859.547805] lr : multi_cpu_stop+0xb4/0x180 >[6129859.548213] sp : ffff000008dabd50 >[6129859.548556] x29: ffff000008dabd50 x28: 0000000000000000 >[6129859.549086] x27: ffff0000300f3938 x26: 0000000000000060 >[6129859.549588] x25: 0000000000000000 x24: 0000000000000000 >[6129859.550094] x23: 0000000000000000 x22: 0000000000000001 >[6129859.550608] x21: ffff0000300f39d4 x20: ffff0000300f39b0 >[6129859.551113] x19: 0000000000000001 x18: 0000000000000000 >[6129859.551624] x17: 0000000000000001 x16: 0000000000000000 >[6129859.552126] x15: 0000000000000000 x14: 0000000000000000 >[6129859.552636] x13: 0000000000000000 x12: 0000000000000000 >[6129859.553151] x11: 0000000000000009 x10: 0000000000000d60 >[6129859.553662] x9 : ffff59fd7ede5a0c x8 : ffff8b7d9da18dc0 >[6129859.554179] x7 : ffff8b7d1ccd2800 x6 : 00000000000003da >[6129859.554696] x5 : 000031804045a000 x4 : ffff8b7d9da18600 >[6129859.555307] x3 : dead000000000200 x2 : ffff0000300f3960 >[6129859.555849] x1 : ffff0000300f39d4 x0 : 0000000000000001 >[6129859.556358] Call trace: >[6129859.556614] multi_cpu_stop+0x78/0x180 >[6129859.556986] cpu_stopper_thread+0xec/0x178 >[6129859.557389] smpboot_thread_fn+0x158/0x1d8 >[6129859.557789] kthread+0x130/0x138 >[6129895.531436] watchdog: BUG: soft lockup - CPU#26 stuck for 23s! [kubelet:18303] >[6129895.532290] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6129895.538945] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6129895.544786] CPU: 26 PID: 18303 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6129895.545875] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6129895.546729] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6129895.547196] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6129895.547686] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6129895.548193] sp : ffff00001f93bc20 >[6129895.548526] x29: ffff00001f93bc20 x28: ffff8bacc51c4000 >[6129895.549035] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6129895.549557] x25: 0000000000000003 x24: 0000000000000000 >[6129895.550070] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6129895.550602] x21: ffff8d9d32ddc000 x20: ffff59fd7fe4b8a8 >[6129895.551080] x19: 0000000000000000 x18: 0000000000000000 >[6129895.551584] x17: 0000000000007d6f x16: 0000000000000000 >[6129895.552064] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6129895.552565] x13: ffff8bad55d1d000 x12: ffff8bad55d1c2fe >[6129895.553073] x11: 0000000000000000 x10: ffff00001f93bc80 >[6129895.553606] x9 : ffff59fd7ef3dad0 x8 : 0000000000014ac3 >[6129895.554093] x7 : 0000000000000004 x6 : 0000000000000005 >[6129895.554644] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6129895.555241] x3 : 0000000000000004 x2 : 0000338040473000 >[6129895.555726] x1 : ffff4a2541a30900 x0 : 0000000000000044 >[6129895.556224] Call trace: >[6129895.556464] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6129895.556927] memcg_numa_stat_show+0x1d0/0x260 >[6129895.557400] cgroup_seqfile_show+0x50/0xd0 >[6129895.557775] kernfs_seq_show+0x30/0x38 >[6129895.558173] seq_read+0x148/0x478 >[6129895.558596] kernfs_fop_read+0x2c/0x1e0 >[6129895.559012] __vfs_read+0x20/0x48 >[6129895.559372] vfs_read+0x98/0x168 >[6129895.559727] ksys_read+0x6c/0xe8 >[6129895.560034] __arm64_sys_read+0x20/0x28 >[6129895.560386] el0_svc_common.constprop.0+0xa8/0x200 >[6129895.560845] el0_svc_handler+0x30/0x80 >[6129895.561250] el0_svc+0x10/0x14 >[6129939.531100] watchdog: BUG: soft lockup - CPU#5 stuck for 23s! [migration/5:36] >[6129939.531708] watchdog: BUG: soft lockup - CPU#44 stuck for 23s! [kubelet:18850] >[6129939.531946] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6129939.532612] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6129939.539130] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6129939.545772] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6129939.551451] CPU: 5 PID: 36 Comm: migration/5 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6129939.557265] CPU: 44 PID: 18850 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6129939.558344] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6129939.559399] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6129939.560303] pstate: 60000005 (nZCv daif -PAN -UAO) >[6129939.561224] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6129939.561716] pc : multi_cpu_stop+0x78/0x180 >[6129939.562227] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6129939.562615] lr : multi_cpu_stop+0xb4/0x180 >[6129939.562617] sp : ffff000008dfbd50 >[6129939.563101] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6129939.563489] x29: ffff000008dfbd50 x28: 0000000000000000 >[6129939.563804] sp : ffff00001fe33c20 >[6129939.564335] x27: ffff000061483938 x26: 0000000000000060 >[6129939.565038] x29: ffff00001fe33c20 x28: ffff8dabe9d60000 >[6129939.565378] x25: 0000000000000000 x24: 0000000000000000 >[6129939.565954] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6129939.566470] x23: 0000000000000000 x22: 0000000000000001 >[6129939.567002] x25: 0000000000000003 x24: 0000000000000000 >[6129939.567504] x21: ffff0000614839d4 x20: ffff0000614839b0 >[6129939.568025] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6129939.568550] x19: 0000000000000001 x18: 0000000000000000 >[6129939.569075] x21: ffff8dbb9e6c7c00 x20: ffff59fd7fe4b8a8 >[6129939.569586] x17: 0000000000000001 x16: 0000000000000000 >[6129939.570108] x19: 0000000000000000 x18: 0000000000000000 >[6129939.570618] x15: 0000000000000001 x14: ffff8dbc68ff2800 >[6129939.571097] x17: 0000000000000002 x16: 0000000000000000 >[6129939.571606] x13: 0000000000000521 x12: ffff8dbc5d588580 >[6129939.572074] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6129939.572595] x11: 0000000000000072 x10: 0000000000000d60 >[6129939.573111] x13: ffff8bcc41960000 x12: ffff8bcc4195f312 >[6129939.573628] x9 : ffff59fd7ede5a0c x8 : ffff8b7d9da24180 >[6129939.574139] x11: 0000000000000000 x10: ffff00001fe33c80 >[6129939.574812] x7 : ffff8b7d6a380000 x6 : 0000000000000042 >[6129939.575303] x9 : ffff59fd7ef3dad0 x8 : 000000000004f02c >[6129939.575825] x5 : 000031804048c000 x4 : ffff8b7d9da239c0 >[6129939.576393] x7 : 0000000000000005 x6 : 0000000000000006 >[6129939.576918] x3 : dead000000000200 x2 : ffff000061483960 >[6129939.577416] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6129939.577912] x1 : ffff0000614839d4 x0 : 0000000000000001 >[6129939.578392] x3 : 0000000000000020 x2 : 000033c040410000 >[6129939.578904] Call trace: >[6129939.579390] x1 : ffff4a0cf51e7740 x0 : 0000000000000060 >[6129939.579909] multi_cpu_stop+0x78/0x180 >[6129939.580145] Call trace: >[6129939.580656] cpu_stopper_thread+0xec/0x178 >[6129939.581021] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6129939.581278] smpboot_thread_fn+0x158/0x1d8 >[6129939.581689] memcg_numa_stat_show+0x1d0/0x260 >[6129939.582156] kthread+0x130/0x138 >[6129939.582536] cgroup_seqfile_show+0x50/0xd0 >[6129939.583655] kernfs_seq_show+0x30/0x38 >[6129939.584023] seq_read+0x148/0x478 >[6129939.584332] kernfs_fop_read+0x2c/0x1e0 >[6129939.584814] __vfs_read+0x20/0x48 >[6129939.585166] vfs_read+0x98/0x168 >[6129939.585480] ksys_read+0x6c/0xe8 >[6129939.585865] __arm64_sys_read+0x20/0x28 >[6129939.586236] el0_svc_common.constprop.0+0xa8/0x200 >[6129939.586744] el0_svc_handler+0x30/0x80 >[6129939.587100] el0_svc+0x10/0x14 >[6129983.531756] watchdog: BUG: soft lockup - CPU#47 stuck for 23s! [kubelet:57297] >[6129983.532567] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6129983.538952] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6129983.544537] CPU: 47 PID: 57297 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6129983.545465] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6129983.546365] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6129983.546873] pc : mem_cgroup_nr_lru_pages.isra.7+0x9c/0xc8 >[6129983.547410] lr : mem_cgroup_nr_lru_pages.isra.7+0xac/0xc8 >[6129983.547951] sp : ffff00003f4f3c20 >[6129983.548277] x29: ffff00003f4f3c20 x28: 0000000000000000 >[6129983.548829] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6129983.549354] x25: 0000000000000010 x24: 0000000000000000 >[6129983.549822] x23: ffff8dac062065d0 x22: 0000000000000020 >[6129983.550349] x21: ffff59fd7fe49800 x20: ffff59fd7fe4b8a8 >[6129983.550837] x19: 0000000000000004 x18: 0000000000000000 >[6129983.551405] x17: 0000000000000002 x16: 0000000000000000 >[6129983.551897] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6129983.552418] x13: ffff8bcd52fd9000 x12: ffff8bcd52fd8327 >[6129983.552960] x11: 0000000000000000 x10: ffff00003f4f3c80 >[6129983.553424] x9 : ffff59fd7ef3da4c x8 : 000000000000dc34 >[6129983.553923] x7 : 0000000000000004 x6 : 0000000000000005 >[6129983.554539] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6129983.555077] x3 : 0000000000000009 x2 : 000031904042a000 >[6129983.555654] x1 : ffff4a1dce4665d0 x0 : 0000000000000009 >[6129983.556149] Call trace: >[6129983.556426] mem_cgroup_nr_lru_pages.isra.7+0x9c/0xc8 >[6129983.556966] memcg_numa_stat_show+0x14c/0x260 >[6129983.557364] cgroup_seqfile_show+0x50/0xd0 >[6129983.557782] kernfs_seq_show+0x30/0x38 >[6129983.558126] seq_read+0x148/0x478 >[6129983.558476] kernfs_fop_read+0x2c/0x1e0 >[6129983.558849] __vfs_read+0x20/0x48 >[6129983.559174] vfs_read+0x98/0x168 >[6129983.559551] ksys_read+0x6c/0xe8 >[6129983.559851] __arm64_sys_read+0x20/0x28 >[6129983.560221] el0_svc_common.constprop.0+0xa8/0x200 >[6129983.560774] el0_svc_handler+0x30/0x80 >[6129983.561120] el0_svc+0x10/0x14 >[6129987.531239] watchdog: BUG: soft lockup - CPU#14 stuck for 22s! [migration/14:82] >[6129987.532066] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6129987.538621] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6129987.544461] CPU: 14 PID: 82 Comm: migration/14 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6129987.545524] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6129987.546396] pstate: 60000005 (nZCv daif -PAN -UAO) >[6129987.546893] pc : multi_cpu_stop+0xf4/0x180 >[6129987.547291] lr : multi_cpu_stop+0xb4/0x180 >[6129987.547690] sp : ffff000008f6bd50 >[6129987.548008] x29: ffff000008f6bd50 x28: 0000000000000000 >[6129987.548565] x27: ffff0000627a3938 x26: 0000000000000060 >[6129987.549063] x25: 0000000000000000 x24: 0000000000000000 >[6129987.549609] x23: 0000000000000000 x22: 0000000000000001 >[6129987.550148] x21: ffff0000627a39d4 x20: ffff0000627a39b0 >[6129987.550691] x19: 0000000000000001 x18: 0000000000000000 >[6129987.551245] x17: 0000000000000001 x16: 0000000000000000 >[6129987.551826] x15: 0000000000000000 x14: 0000000000000000 >[6129987.552326] x13: 0000000000000000 x12: 0000000000000000 >[6129987.552830] x11: 0000000000000000 x10: 0000000000000d60 >[6129987.553363] x9 : ffff59fd7ede5a0c x8 : ffff8b8d9e99e400 >[6129987.553908] x7 : ffff8b8d7bc1b000 x6 : 000000000000034b >[6129987.554566] x5 : 00003190404a7000 x4 : ffff8b8d9e99dc40 >[6129987.555083] x3 : dead000000000200 x2 : ffff0000627a3960 >[6129987.555630] x1 : ffff0000627a39d4 x0 : 0000000000000001 >[6129987.556250] Call trace: >[6129987.556509] multi_cpu_stop+0xf4/0x180 >[6129987.556920] cpu_stopper_thread+0xec/0x178 >[6129987.557330] smpboot_thread_fn+0x158/0x1d8 >[6129987.557726] kthread+0x130/0x138 >[6130023.531104] watchdog: BUG: soft lockup - CPU#5 stuck for 23s! [kubelet:61154] >[6130023.531912] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6130023.538658] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6130023.544624] CPU: 5 PID: 61154 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6130023.545626] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6130023.546496] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6130023.546956] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6130023.547517] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6130023.548069] sp : ffff000059c5bc20 >[6130023.548383] x29: ffff000059c5bc20 x28: ffff8b7b65b44000 >[6130023.548893] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6130023.549402] x25: 0000000000000003 x24: 0000000000000000 >[6130023.549911] x23: 0000000000000008 x22: ffff59fd7fe49800 >[6130023.550409] x21: ffff8d8cd9848800 x20: ffff59fd7fe4b8a8 >[6130023.550896] x19: 0000000000000001 x18: 0000000000000000 >[6130023.551386] x17: 0000000000000009 x16: 0000000000000000 >[6130023.551888] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6130023.552386] x13: ffff8b7b9f096000 x12: ffff8b7b9f0952f5 >[6130023.552991] x11: 0000000000000000 x10: ffff000059c5bc80 >[6130023.553554] x9 : ffff59fd7ef3dad0 x8 : 00000000000375e7 >[6130023.554153] x7 : 0000000000000005 x6 : 0000000000000006 >[6130023.554706] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6130023.555233] x3 : 0000000000000024 x2 : 000031c040474000 >[6130023.555770] x1 : ffff4a0cf624fe68 x0 : 0000000000000024 >[6130023.556321] Call trace: >[6130023.556578] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6130023.557097] memcg_numa_stat_show+0x1d0/0x260 >[6130023.557568] cgroup_seqfile_show+0x50/0xd0 >[6130023.557965] kernfs_seq_show+0x30/0x38 >[6130023.558374] seq_read+0x148/0x478 >[6130023.558702] kernfs_fop_read+0x2c/0x1e0 >[6130023.559075] __vfs_read+0x20/0x48 >[6130023.559396] vfs_read+0x98/0x168 >[6130023.559700] ksys_read+0x6c/0xe8 >[6130023.560014] __arm64_sys_read+0x20/0x28 >[6130023.560468] el0_svc_common.constprop.0+0xa8/0x200 >[6130023.560934] el0_svc_handler+0x30/0x80 >[6130023.561307] el0_svc+0x10/0x14 >[6130067.531084] watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [migration/4:31] >[6130067.531181] watchdog: BUG: soft lockup - CPU#10 stuck for 23s! [kubelet:18612] >[6130067.531915] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6130067.532602] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6130067.539158] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6130067.545744] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6130067.551682] CPU: 4 PID: 31 Comm: migration/4 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6130067.557855] CPU: 10 PID: 18612 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6130067.558850] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6130067.559845] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6130067.560799] pstate: 60000005 (nZCv daif -PAN -UAO) >[6130067.561748] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6130067.562227] pc : multi_cpu_stop+0x78/0x180 >[6130067.562727] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6130067.563148] lr : multi_cpu_stop+0xb4/0x180 >[6130067.563646] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6130067.563649] sp : ffff000050243c20 >[6130067.564158] sp : ffff000008dd3d50 >[6130067.564729] x29: ffff000050243c20 x28: ffff8b7b6cd24000 >[6130067.565069] x29: ffff000008dd3d50 x28: 0000000000000000 >[6130067.565395] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6130067.565949] x27: ffff0000545db938 x26: 0000000000000060 >[6130067.566494] x25: 0000000000000003 x24: 0000000000000000 >[6130067.566972] x25: 0000000000000000 x24: 0000000000000000 >[6130067.567504] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6130067.568044] x23: 0000000000000000 x22: 0000000000000001 >[6130067.568519] x21: ffff8dabffc67000 x20: ffff59fd7fe4b8a8 >[6130067.569052] x21: ffff0000545db9d4 x20: ffff0000545db9b0 >[6130067.569552] x19: 0000000000000000 x18: 0000000000000000 >[6130067.570060] x19: 0000000000000001 x18: 0000000000000000 >[6130067.570561] x17: 0000000000000002 x16: 0000000000000000 >[6130067.571057] x17: 0000000000000001 x16: 0000000000000000 >[6130067.571543] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6130067.572043] x15: 0000000000000000 x14: 0000000000000000 >[6130067.572564] x13: ffff8b8d72d57000 x12: ffff8b8d72d5630a >[6130067.573074] x13: 0000000000000000 x12: 0000000000000000 >[6130067.573595] x11: 0000000000000000 x10: ffff000050243c80 >[6130067.574207] x11: 00000000000003e8 x10: 0000000000000d60 >[6130067.574714] x9 : ffff59fd7ef3dad0 x8 : 000000000000bded >[6130067.575218] x9 : ffff59fd7ede5a0c x8 : ffff8b7d9da1e400 >[6130067.575754] x7 : 0000000000000004 x6 : 0000000000000005 >[6130067.576346] x7 : ffff8b7d1ccd3000 x6 : 00000000000000b2 >[6130067.576856] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6130067.577388] x5 : 0000318040473000 x4 : ffff8b7d9da1dc40 >[6130067.577891] x3 : 000000000000001f x2 : 000031b0404c0000 >[6130067.578460] x3 : dead000000000200 x2 : ffff0000545db960 >[6130067.579007] x1 : ffff4a0cfec91560 x0 : 000000000000001f >[6130067.579568] x1 : ffff0000545db9d4 x0 : 0000000000000001 >[6130067.580126] Call trace: >[6130067.580692] Call trace: >[6130067.581209] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6130067.581454] multi_cpu_stop+0x78/0x180 >[6130067.581702] memcg_numa_stat_show+0x1d0/0x260 >[6130067.582206] cpu_stopper_thread+0xec/0x178 >[6130067.582614] cgroup_seqfile_show+0x50/0xd0 >[6130067.583044] smpboot_thread_fn+0x158/0x1d8 >[6130067.583436] kernfs_seq_show+0x30/0x38 >[6130067.583982] kthread+0x130/0x138 >[6130067.584387] seq_read+0x148/0x478 >[6130067.585384] kernfs_fop_read+0x2c/0x1e0 >[6130067.585731] __vfs_read+0x20/0x48 >[6130067.586052] vfs_read+0x98/0x168 >[6130067.586361] ksys_read+0x6c/0xe8 >[6130067.586670] __arm64_sys_read+0x20/0x28 >[6130067.587046] el0_svc_common.constprop.0+0xa8/0x200 >[6130067.587526] el0_svc_handler+0x30/0x80 >[6130067.587894] el0_svc+0x10/0x14 >[6130111.531418] watchdog: BUG: soft lockup - CPU#25 stuck for 23s! [kubelet:18924] >[6130111.532310] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6130111.538821] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6130111.544427] CPU: 25 PID: 18924 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6130111.545403] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6130111.546278] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6130111.546712] pc : mem_cgroup_nr_lru_pages.isra.7+0x9c/0xc8 >[6130111.547281] lr : mem_cgroup_nr_lru_pages.isra.7+0xac/0xc8 >[6130111.547818] sp : ffff00003e19bc20 >[6130111.548181] x29: ffff00003e19bc20 x28: 0000000000000000 >[6130111.548702] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6130111.549213] x25: 0000000000000010 x24: 0000000000000000 >[6130111.549732] x23: ffff8dac3f7105d0 x22: 0000000000000020 >[6130111.550298] x21: ffff59fd7fe49800 x20: ffff59fd7fe4b8a8 >[6130111.550812] x19: 0000000000000004 x18: 0000000000000000 >[6130111.551340] x17: 0000000000000009 x16: 0000000000000000 >[6130111.551920] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6130111.552414] x13: ffff8bad55995000 x12: ffff8bad5599432a >[6130111.552966] x11: 0000000000000000 x10: ffff00003e19bc80 >[6130111.553568] x9 : ffff59fd7ef3da4c x8 : 000000000000de00 >[6130111.554065] x7 : 0000000000000004 x6 : 0000000000000005 >[6130111.554708] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6130111.555206] x3 : 0000000000000000 x2 : 000033804040f000 >[6130111.555766] x1 : ffff4a0f0eb5fdc0 x0 : 0000000000000040 >[6130111.556245] Call trace: >[6130111.556485] mem_cgroup_nr_lru_pages.isra.7+0x9c/0xc8 >[6130111.556997] memcg_numa_stat_show+0x14c/0x260 >[6130111.557455] cgroup_seqfile_show+0x50/0xd0 >[6130111.557865] kernfs_seq_show+0x30/0x38 >[6130111.558269] seq_read+0x148/0x478 >[6130111.558626] kernfs_fop_read+0x2c/0x1e0 >[6130111.559009] __vfs_read+0x20/0x48 >[6130111.559347] vfs_read+0x98/0x168 >[6130111.559673] ksys_read+0x6c/0xe8 >[6130111.559999] __arm64_sys_read+0x20/0x28 >[6130111.560388] el0_svc_common.constprop.0+0xa8/0x200 >[6130111.560893] el0_svc_handler+0x30/0x80 >[6130111.561224] el0_svc+0x10/0x14 >[6130151.531277] watchdog: BUG: soft lockup - CPU#16 stuck for 22s! [kubelet:21523] >[6130151.532083] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6130151.538735] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6130151.544589] CPU: 16 PID: 21523 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6130151.545665] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6130151.546633] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6130151.547146] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6130151.547690] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6130151.548220] sp : ffff00001b87bc20 >[6130151.548557] x29: ffff00001b87bc20 x28: ffff8bbc005d6000 >[6130151.549083] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6130151.549584] x25: 0000000000000003 x24: 0000000000000000 >[6130151.550073] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6130151.550628] x21: ffff8d7c2155c800 x20: ffff59fd7fe4b8a8 >[6130151.551124] x19: 0000000000000000 x18: 0000000000000000 >[6130151.551605] x17: 0000000000000002 x16: 0000000000000000 >[6130151.552096] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6130151.552620] x13: ffff8b9ba2455000 x12: ffff8b9ba24542ec >[6130151.553220] x11: 0000000000000000 x10: ffff00001b87bc80 >[6130151.553743] x9 : ffff59fd7ef3dad0 x8 : 000000000002633d >[6130151.554284] x7 : 0000000000000005 x6 : 0000000000000006 >[6130151.554875] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6130151.555380] x3 : 0000000000000027 x2 : 000033c0404bf000 >[6130151.555908] x1 : ffff4a0cfdc29c90 x0 : 0000000000000067 >[6130151.556452] Call trace: >[6130151.556730] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6130151.557252] memcg_numa_stat_show+0x1d0/0x260 >[6130151.557689] cgroup_seqfile_show+0x50/0xd0 >[6130151.558071] kernfs_seq_show+0x30/0x38 >[6130151.558416] seq_read+0x148/0x478 >[6130151.558736] kernfs_fop_read+0x2c/0x1e0 >[6130151.559141] __vfs_read+0x20/0x48 >[6130151.559450] vfs_read+0x98/0x168 >[6130151.559789] ksys_read+0x6c/0xe8 >[6130151.560091] __arm64_sys_read+0x20/0x28 >[6130151.560444] el0_svc_common.constprop.0+0xa8/0x200 >[6130151.560990] el0_svc_handler+0x30/0x80 >[6130151.561353] el0_svc+0x10/0x14 >[6130195.531119] watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [kubelet:18322] >[6130195.531993] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6130195.538951] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6130195.544836] CPU: 6 PID: 18322 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6130195.545856] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6130195.546725] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6130195.547214] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6130195.547779] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6130195.548260] sp : ffff00001f8dbc20 >[6130195.548640] x29: ffff00001f8dbc20 x28: ffff8b7d59c74000 >[6130195.549158] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6130195.549714] x25: 0000000000000003 x24: 0000000000000000 >[6130195.550220] x23: 0000000000000008 x22: ffff59fd7fe49800 >[6130195.550765] x21: ffff8d9d87c8fc00 x20: ffff59fd7fe4b8a8 >[6130195.551315] x19: 0000000000000001 x18: 0000000000000000 >[6130195.551850] x17: 0000000000000002 x16: 0000000000000000 >[6130195.552350] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6130195.552969] x13: ffff8b7d1c80c000 x12: ffff8b7d1c80b300 >[6130195.553504] x11: 0000000000000000 x10: ffff00001f8dbc80 >[6130195.554031] x9 : ffff59fd7ef3dad0 x8 : 0000000000014b33 >[6130195.554559] x7 : 0000000000000004 x6 : 0000000000000005 >[6130195.555079] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6130195.555583] x3 : 000000000000000e x2 : 00003190404a7000 >[6130195.556107] x1 : ffff4a27830bb1c8 x0 : 000000000000000e >[6130195.556636] Call trace: >[6130195.556882] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6130195.557376] memcg_numa_stat_show+0x1d0/0x260 >[6130195.557803] cgroup_seqfile_show+0x50/0xd0 >[6130195.558221] kernfs_seq_show+0x30/0x38 >[6130195.558603] seq_read+0x148/0x478 >[6130195.558935] kernfs_fop_read+0x2c/0x1e0 >[6130195.559311] __vfs_read+0x20/0x48 >[6130195.559628] vfs_read+0x98/0x168 >[6130195.559945] ksys_read+0x6c/0xe8 >[6130195.560261] __arm64_sys_read+0x20/0x28 >[6130195.560653] el0_svc_common.constprop.0+0xa8/0x200 >[6130195.561131] el0_svc_handler+0x30/0x80 >[6130195.561486] el0_svc+0x10/0x14 >[6130199.531660] watchdog: BUG: soft lockup - CPU#41 stuck for 23s! [migration/41:221] >[6130199.532507] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6130199.539072] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6130199.545001] CPU: 41 PID: 221 Comm: migration/41 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6130199.546110] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6130199.547038] pstate: 60000005 (nZCv daif -PAN -UAO) >[6130199.547584] pc : multi_cpu_stop+0xf4/0x180 >[6130199.547972] lr : multi_cpu_stop+0xb4/0x180 >[6130199.548384] sp : ffff00000a3cbd50 >[6130199.548714] x29: ffff00000a3cbd50 x28: 0000000000000000 >[6130199.549269] x27: ffff00001b87b938 x26: 0000000000000060 >[6130199.549787] x25: 0000000000000000 x24: 0000000000000000 >[6130199.550284] x23: 0000000000000000 x22: 0000000000000001 >[6130199.550751] x21: ffff00001b87b9d4 x20: ffff00001b87b9b0 >[6130199.551264] x19: 0000000000000001 x18: 0000000000000000 >[6130199.551727] x17: 0000000000000001 x16: 0000000000000000 >[6130199.552213] x15: 0000000000000000 x14: 0000000000000000 >[6130199.552725] x13: 0000000000000000 x12: 0000000000000000 >[6130199.553294] x11: 0000000000000023 x10: 0000000000000d60 >[6130199.553823] x9 : ffff59fd7ede5a0c x8 : ffff8bcd9e8f9f00 >[6130199.554306] x7 : 000968365b25ea14 x6 : 00000000000003fa >[6130199.554854] x5 : 00000000fa83b2da x4 : ffff8bcd9e8f9740 >[6130199.555395] x3 : dead000000000200 x2 : ffff00001b87b988 >[6130199.555941] x1 : ffff00001b87b9d4 x0 : 0000000000000001 >[6130199.556562] Call trace: >[6130199.556847] multi_cpu_stop+0xf4/0x180 >[6130199.557238] cpu_stopper_thread+0xec/0x178 >[6130199.557691] smpboot_thread_fn+0x158/0x1d8 >[6130199.558097] kthread+0x130/0x138 >[6130239.531259] watchdog: BUG: soft lockup - CPU#15 stuck for 22s! [kubelet:18889] >[6130239.532179] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6130239.538812] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6130239.544616] CPU: 15 PID: 18889 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6130239.545679] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6130239.546593] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6130239.547127] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6130239.547704] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6130239.548201] sp : ffff0000300f3c20 >[6130239.548625] x29: ffff0000300f3c20 x28: ffff8dac1490c000 >[6130239.549192] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6130239.549679] x25: 0000000000000003 x24: 0000000000000000 >[6130239.550271] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6130239.550841] x21: ffff8dbbcdc80800 x20: ffff59fd7fe4b8a8 >[6130239.551337] x19: 0000000000000000 x18: 0000000000000000 >[6130239.551901] x17: 0000000000000002 x16: 0000000000000000 >[6130239.552509] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6130239.553090] x13: ffff8b8bdf96e000 x12: ffff8b8bdf96d314 >[6130239.553560] x11: 0000000000000000 x10: ffff0000300f3c80 >[6130239.554091] x9 : ffff59fd7ef3dad0 x8 : 000000000004ed1c >[6130239.554573] x7 : 0000000000000005 x6 : 0000000000000006 >[6130239.555058] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6130239.555546] x3 : 0000000000000015 x2 : 000033a04048e000 >[6130239.556041] x1 : ffff4a2dc70adf90 x0 : 0000000000000055 >[6130239.556555] Call trace: >[6130239.556795] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6130239.557266] memcg_numa_stat_show+0x1d0/0x260 >[6130239.557686] cgroup_seqfile_show+0x50/0xd0 >[6130239.558059] kernfs_seq_show+0x30/0x38 >[6130239.558492] seq_read+0x148/0x478 >[6130239.558815] kernfs_fop_read+0x2c/0x1e0 >[6130239.559262] __vfs_read+0x20/0x48 >[6130239.559585] vfs_read+0x98/0x168 >[6130239.559894] ksys_read+0x6c/0xe8 >[6130239.560242] __arm64_sys_read+0x20/0x28 >[6130239.560676] el0_svc_common.constprop.0+0xa8/0x200 >[6130239.561208] el0_svc_handler+0x30/0x80 >[6130239.561569] el0_svc+0x10/0x14 >[6130283.531259] watchdog: BUG: soft lockup - CPU#15 stuck for 22s! [kubelet:18888] >[6130283.532079] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6130283.539557] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6130283.546051] CPU: 15 PID: 18888 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6130283.547156] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6130283.548115] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6130283.548676] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6130283.549219] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6130283.549798] sp : ffff00001fec3c20 >[6130283.550161] x29: ffff00001fec3c20 x28: ffff8dbc72060000 >[6130283.550738] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6130283.551333] x25: 0000000000000010 x24: 0000000000000000 >[6130283.551915] x23: 0000000000000020 x22: ffff59fd7fe49800 >[6130283.552591] x21: ffff8b8c350c4800 x20: ffff59fd7fe4b8a8 >[6130283.553138] x19: 0000000000000004 x18: 0000000000000000 >[6130283.553630] x17: 0000000000000002 x16: 0000000000000000 >[6130283.554229] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6130283.554766] x13: ffff8b8bdf969000 x12: ffff8b8bdf96834e >[6130283.555317] x11: 0000000000000000 x10: ffff00001fec3c80 >[6130283.555806] x9 : ffff59fd7ef3dad0 x8 : ffff8b8bdf968000 >[6130283.556291] x7 : 0000000000000000 x6 : 0000000000000001 >[6130283.556831] x5 : 00000000fffffffb x4 : ffffffffffffffff >[6130283.557388] x3 : 000000000000001a x2 : 000033b040443000 >[6130283.557908] x1 : ffff4a0f1cebb230 x0 : 000000000000005a >[6130283.558415] Call trace: >[6130283.558670] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6130283.559107] memcg_numa_stat_show+0x1d0/0x260 >[6130283.559493] cgroup_seqfile_show+0x50/0xd0 >[6130283.559885] kernfs_seq_show+0x30/0x38 >[6130283.560226] seq_read+0x148/0x478 >[6130283.560547] kernfs_fop_read+0x2c/0x1e0 >[6130283.560895] __vfs_read+0x20/0x48 >[6130283.561226] vfs_read+0x98/0x168 >[6130283.561566] ksys_read+0x6c/0xe8 >[6130283.561886] __arm64_sys_read+0x20/0x28 >[6130283.562418] el0_svc_common.constprop.0+0xa8/0x200 >[6130283.562886] el0_svc_handler+0x30/0x80 >[6130283.563312] el0_svc+0x10/0x14 >[6130323.531136] watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [kubelet:18849] >[6130323.532042] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6130323.538626] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6130323.544561] CPU: 7 PID: 18849 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6130323.545551] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6130323.546439] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6130323.546892] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6130323.547395] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6130323.547896] sp : ffff000030123c20 >[6130323.548227] x29: ffff000030123c20 x28: ffff8b7d5f820000 >[6130323.548771] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6130323.549335] x25: 0000000000000003 x24: 0000000000000000 >[6130323.549933] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6130323.550446] x21: ffff8dad94c97c00 x20: ffff59fd7fe4b8a8 >[6130323.551010] x19: 0000000000000000 x18: 0000000000000000 >[6130323.551567] x17: 0000000000000002 x16: 0000000000000000 >[6130323.552164] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6130323.552779] x13: ffff8b7d1ce69000 x12: ffff8b7d1ce6830a >[6130323.553319] x11: 0000000000000000 x10: ffff000030123c80 >[6130323.553817] x9 : ffff59fd7ef3dad0 x8 : 000000000000bd92 >[6130323.554374] x7 : 0000000000000004 x6 : 0000000000000005 >[6130323.554924] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6130323.555420] x3 : 0000000000000039 x2 : 000033d0401bc000 >[6130323.555970] x1 : ffff4a2dc57600a0 x0 : 0000000000000039 >[6130323.556466] Call trace: >[6130323.556728] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6130323.557202] memcg_numa_stat_show+0x1d0/0x260 >[6130323.557618] cgroup_seqfile_show+0x50/0xd0 >[6130323.558005] kernfs_seq_show+0x30/0x38 >[6130323.558383] seq_read+0x148/0x478 >[6130323.558708] kernfs_fop_read+0x2c/0x1e0 >[6130323.559141] __vfs_read+0x20/0x48 >[6130323.559505] vfs_read+0x98/0x168 >[6130323.559905] ksys_read+0x6c/0xe8 >[6130323.560233] __arm64_sys_read+0x20/0x28 >[6130323.560611] el0_svc_common.constprop.0+0xa8/0x200 >[6130323.561178] el0_svc_handler+0x30/0x80 >[6130323.561606] el0_svc+0x10/0x14 >[6130327.531178] watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [migration/10:62] >[6130327.532088] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6130327.538764] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6130327.544670] CPU: 10 PID: 62 Comm: migration/10 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6130327.545736] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6130327.546644] pstate: 60000005 (nZCv daif -PAN -UAO) >[6130327.547101] pc : multi_cpu_stop+0x78/0x180 >[6130327.547553] lr : multi_cpu_stop+0xb4/0x180 >[6130327.547995] sp : ffff000008ecbd50 >[6130327.548361] x29: ffff000008ecbd50 x28: 0000000000000000 >[6130327.548897] x27: ffff000062fc3938 x26: 0000000000000060 >[6130327.549395] x25: 0000000000000000 x24: 0000000000000000 >[6130327.549975] x23: 0000000000000000 x22: 0000000000000001 >[6130327.550474] x21: ffff000062fc39d4 x20: ffff000062fc39b0 >[6130327.551006] x19: 0000000000000001 x18: 0000000000000000 >[6130327.551522] x17: 0000000000000001 x16: 0000000000000000 >[6130327.552165] x15: 0000000000000000 x14: 0000000000000000 >[6130327.552722] x13: 0000000000000000 x12: 0000000000000000 >[6130327.553262] x11: 0000000000000168 x10: 0000000000000d60 >[6130327.553864] x9 : ffff59fd7ede5a0c x8 : ffff8b8d9e907540 >[6130327.554399] x7 : ffff8b8d8059ec00 x6 : 00000000000002f0 >[6130327.554939] x5 : 0000319040443000 x4 : ffff8b8d9e906d80 >[6130327.555458] x3 : dead000000000200 x2 : ffff000062fc3988 >[6130327.556054] x1 : ffff000062fc39d4 x0 : 0000000000000001 >[6130327.556564] Call trace: >[6130327.556803] multi_cpu_stop+0x78/0x180 >[6130327.557175] cpu_stopper_thread+0xec/0x178 >[6130327.557556] smpboot_thread_fn+0x158/0x1d8 >[6130327.557952] kthread+0x130/0x138 >[6130367.531801] watchdog: BUG: soft lockup - CPU#49 stuck for 22s! [kubelet:18804] >[6130367.532661] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6130367.538917] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6130367.544492] CPU: 49 PID: 18804 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6130367.545474] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6130367.546286] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6130367.546727] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6130367.547208] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6130367.547727] sp : ffff00001fe63c20 >[6130367.548069] x29: ffff00001fe63c20 x28: ffff8b8c38774000 >[6130367.548625] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6130367.549098] x25: 0000000000000003 x24: 0000000000000000 >[6130367.549639] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6130367.550110] x21: ffff8dac6b209400 x20: ffff59fd7fe4b8a8 >[6130367.550600] x19: 0000000000000000 x18: 0000000000000000 >[6130367.551105] x17: 0000000000000002 x16: 0000000000000000 >[6130367.551678] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6130367.552227] x13: ffff8b9d269af000 x12: ffff8b9d269ae307 >[6130367.552725] x11: 0000000000000000 x10: ffff00001fe63c80 >[6130367.553245] x9 : ffff59fd7ef3dad0 x8 : 000000000000c214 >[6130367.553746] x7 : 0000000000000004 x6 : 0000000000000005 >[6130367.554213] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6130367.554691] x3 : 000000000000000c x2 : 0000339040475000 >[6130367.555176] x1 : ffff4a15c1e253c0 x0 : 000000000000004c >[6130367.555680] Call trace: >[6130367.555996] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6130367.556449] memcg_numa_stat_show+0x1d0/0x260 >[6130367.556863] cgroup_seqfile_show+0x50/0xd0 >[6130367.557236] kernfs_seq_show+0x30/0x38 >[6130367.557629] seq_read+0x148/0x478 >[6130367.557995] kernfs_fop_read+0x2c/0x1e0 >[6130367.558396] __vfs_read+0x20/0x48 >[6130367.558709] vfs_read+0x98/0x168 >[6130367.559021] ksys_read+0x6c/0xe8 >[6130367.559320] __arm64_sys_read+0x20/0x28 >[6130367.559680] el0_svc_common.constprop.0+0xa8/0x200 >[6130367.560128] el0_svc_handler+0x30/0x80 >[6130367.560544] el0_svc+0x10/0x14 >[6130411.531694] watchdog: BUG: soft lockup - CPU#43 stuck for 22s! [kubelet:18288] >[6130411.532558] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6130411.539112] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6130411.544821] CPU: 43 PID: 18288 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6130411.545823] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6130411.546660] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6130411.547140] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6130411.547620] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6130411.548101] sp : ffff00001f91bc20 >[6130411.548416] x29: ffff00001f91bc20 x28: ffff8b8bc6202000 >[6130411.548953] x27: ffff59fd7fe4bbf4 x26: fffffffffffffc8e >[6130411.549474] x25: 0000000000000003 x24: 0000000000000000 >[6130411.550028] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6130411.550535] x21: ffff8dbb941dfc00 x20: ffff59fd7fe4b8a8 >[6130411.551021] x19: 0000000000000000 x18: 0000000000000000 >[6130411.551686] x17: 0000000000000002 x16: 0000000000000000 >[6130411.552199] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6130411.552801] x13: ffff8bcbfbc10000 x12: ffff8bcbfbc0f311 >[6130411.553332] x11: 0000000000000000 x10: ffff00001f91bc80 >[6130411.553836] x9 : ffff59fd7ef3dad0 x8 : 000000000004ed94 >[6130411.554356] x7 : 0000000000000005 x6 : 0000000000000006 >[6130411.554853] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6130411.555357] x3 : 000000000000002b x2 : 000033d020060000 >[6130411.555851] x1 : ffff4a0cecc47ca0 x0 : 000000000000006b >[6130411.556347] Call trace: >[6130411.556600] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6130411.557076] memcg_numa_stat_show+0x1d0/0x260 >[6130411.557506] cgroup_seqfile_show+0x50/0xd0 >[6130411.557909] kernfs_seq_show+0x30/0x38 >[6130411.558267] seq_read+0x148/0x478 >[6130411.558600] kernfs_fop_read+0x2c/0x1e0 >[6130411.558968] __vfs_read+0x20/0x48 >[6130411.559292] vfs_read+0x98/0x168 >[6130411.559612] ksys_read+0x6c/0xe8 >[6130411.559909] __arm64_sys_read+0x20/0x28 >[6130411.560285] el0_svc_common.constprop.0+0xa8/0x200 >[6130411.560758] el0_svc_handler+0x30/0x80 >[6130411.561114] el0_svc+0x10/0x14 >[6130415.531099] watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [migration/5:36] >[6130415.531964] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6130415.538497] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6130415.544533] CPU: 5 PID: 36 Comm: migration/5 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6130415.545512] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6130415.546396] pstate: 60000005 (nZCv daif -PAN -UAO) >[6130415.546859] pc : multi_cpu_stop+0xf4/0x180 >[6130415.547276] lr : multi_cpu_stop+0xb4/0x180 >[6130415.547690] sp : ffff000008dfbd50 >[6130415.548015] x29: ffff000008dfbd50 x28: 0000000000000000 >[6130415.548531] x27: ffff00003dc03938 x26: 0000000000000060 >[6130415.549060] x25: 0000000000000000 x24: 0000000000000000 >[6130415.549578] x23: 0000000000000000 x22: 0000000000000001 >[6130415.550098] x21: ffff00003dc039d4 x20: ffff00003dc039b0 >[6130415.550614] x19: 0000000000000001 x18: 0000000000000000 >[6130415.551142] x17: 0000000000000001 x16: 0000000000000000 >[6130415.551799] x15: 0000000000000000 x14: 0000000000000000 >[6130415.552368] x13: 0000000000000000 x12: 0000000000000000 >[6130415.552879] x11: 0000000000000000 x10: 0000000000000d60 >[6130415.553445] x9 : ffff59fd7ede5a0c x8 : ffff8b7d9da24180 >[6130415.553954] x7 : ffff8b7d6a380000 x6 : 0000000000000031 >[6130415.554485] x5 : 000031804048c000 x4 : ffff8b7d9da239c0 >[6130415.555042] x3 : dead000000000200 x2 : ffff00003dc03960 >[6130415.555533] x1 : ffff00003dc039d4 x0 : 0000000000000001 >[6130415.556084] Call trace: >[6130415.556328] multi_cpu_stop+0xf4/0x180 >[6130415.556726] cpu_stopper_thread+0xec/0x178 >[6130415.557139] smpboot_thread_fn+0x158/0x1d8 >[6130415.557551] kthread+0x130/0x138 >[6130455.531294] watchdog: BUG: soft lockup - CPU#17 stuck for 22s! [kubelet:4832] >[6130455.532145] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6130455.538270] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6130455.543847] CPU: 17 PID: 4832 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6130455.544820] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6130455.545727] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6130455.546200] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6130455.546692] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6130455.547181] sp : ffff00005e9c3c20 >[6130455.547501] x29: ffff00005e9c3c20 x28: ffff8b7b9893c000 >[6130455.547977] x27: ffff59fd7fe4bbf4 x26: ffffffffffffffff >[6130455.548446] x25: 0000000000000003 x24: 0000000000000000 >[6130455.548935] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6130455.549414] x21: ffff8dccf0d53000 x20: ffff59fd7fe4b8a8 >[6130455.549901] x19: 0000000000000000 x18: 0000000000000000 >[6130455.550376] x17: 0000000000000002 x16: 0000000000000000 >[6130455.550863] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6130455.551447] x13: ffff8b9b753c4000 x12: ffff8b9b753c331c >[6130455.551920] x11: 0000000000000000 x10: ffff00005e9c3c80 >[6130455.552408] x9 : ffff59fd7ef3dad0 x8 : 0000000000018228 >[6130455.552903] x7 : 0000000000000004 x6 : 0000000000000005 >[6130455.553456] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6130455.553940] x3 : 000000000000000b x2 : 000033904045c000 >[6130455.554422] x1 : ffff4a0f0ae713e0 x0 : 000000000000004b >[6130455.554907] Call trace: >[6130455.555135] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6130455.555581] memcg_numa_stat_show+0x1d0/0x260 >[6130455.555989] cgroup_seqfile_show+0x50/0xd0 >[6130455.556361] kernfs_seq_show+0x30/0x38 >[6130455.556760] seq_read+0x148/0x478 >[6130455.557123] kernfs_fop_read+0x2c/0x1e0 >[6130455.557497] __vfs_read+0x20/0x48 >[6130455.557819] vfs_read+0x98/0x168 >[6130455.558210] ksys_read+0x6c/0xe8 >[6130455.558541] __arm64_sys_read+0x20/0x28 >[6130455.559023] el0_svc_common.constprop.0+0xa8/0x200 >[6130455.559497] el0_svc_handler+0x30/0x80 >[6130455.559894] el0_svc+0x10/0x14 >[6130499.531555] watchdog: BUG: soft lockup - CPU#34 stuck for 22s! [kubelet:18888] >[6130499.531672] watchdog: BUG: soft lockup - CPU#42 stuck for 22s! [migration/42:226] >[6130499.532364] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6130499.533040] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6130499.539311] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6130499.545631] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6130499.551329] CPU: 34 PID: 18888 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6130499.556984] CPU: 42 PID: 226 Comm: migration/42 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6130499.558030] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6130499.559025] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6130499.559898] pstate: 20000005 (nzCv daif -PAN -UAO) >[6130499.560891] pstate: 60000005 (nZCv daif -PAN -UAO) >[6130499.561368] pc : css_next_descendant_pre+0x24/0x88 >[6130499.561875] pc : multi_cpu_stop+0x78/0x180 >[6130499.562349] lr : mem_cgroup_iter+0xc4/0x378 >[6130499.562351] sp : ffff00001fec3c10 >[6130499.562829] lr : multi_cpu_stop+0xb4/0x180 >[6130499.563212] x29: ffff00001fec3c10 x28: 0000000000000000 >[6130499.563540] sp : ffff00000a3f3d50 >[6130499.563900] x27: ffff59fd7f671bf8 x26: ffff59fd7f8b5a88 >[6130499.564372] x29: ffff00000a3f3d50 x28: 0000000000000000 >[6130499.564709] x25: 0000000000000010 x24: 0000000000000001 >[6130499.565234] x27: ffff00005c26b938 x26: 0000000000000060 >[6130499.565756] x23: ffff8b8bf1740000 x22: 0000000000000000 >[6130499.566241] x25: 0000000000000000 x24: 0000000000000000 >[6130499.566744] x21: ffff8b8bf1740000 x20: ffff8dbd9e88a000 >[6130499.567227] x23: 0000000000000000 x22: 0000000000000001 >[6130499.567734] x19: ffff8b8bf1740000 x18: 0000000000000000 >[6130499.568255] x21: ffff00005c26b9d4 x20: ffff00005c26b9b0 >[6130499.568778] x17: 0000000000000002 x16: 0000000000000000 >[6130499.569269] x19: 0000000000000001 x18: 0000000000000000 >[6130499.569778] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6130499.570263] x17: 0000000000000001 x16: 0000000000000000 >[6130499.570915] x13: ffff8bbd32623000 x12: ffff8bbd32622326 >[6130499.571399] x15: 0000000000000000 x14: ffff8dbd25470800 >[6130499.572003] x11: 0000000000000000 x10: ffff00001fec3c80 >[6130499.572469] x13: 000000000000018b x12: ffff8dbc25fafd80 >[6130499.572975] x9 : ffff59fd7ef3cdc4 x8 : 000000000000dc17 >[6130499.573443] x11: 000000000000001e x10: 0000000000000d60 >[6130499.573917] x7 : 0000000000000004 x6 : 0000000000000005 >[6130499.574389] x9 : ffff59fd7ede5a0c x8 : ffff8bcd9e8ff540 >[6130499.574933] x5 : 00000000fffffff7 x4 : 0000000000000000 >[6130499.575411] x7 : ffff8bcd74d41c00 x6 : 00000000000000f5 >[6130499.575991] x3 : 0000000000000000 x2 : 0000000000000000 >[6130499.576553] x5 : 000031d040443000 x4 : ffff8bcd9e8fed80 >[6130499.577082] x1 : ffff8b8bf1740000 x0 : ffff8b8bf1740000 >[6130499.577637] x3 : dead000000000200 x2 : ffff00005c26b988 >[6130499.578152] Call trace: >[6130499.578697] x1 : ffff00005c26b9d4 x0 : 0000000000000001 >[6130499.579212] css_next_descendant_pre+0x24/0x88 >[6130499.579480] Call trace: >[6130499.580037] mem_cgroup_iter+0xc4/0x378 >[6130499.580543] multi_cpu_stop+0x78/0x180 >[6130499.580885] memcg_numa_stat_show+0x160/0x260 >[6130499.581284] cpu_stopper_thread+0xec/0x178 >[6130499.581701] cgroup_seqfile_show+0x50/0xd0 >[6130499.582169] smpboot_thread_fn+0x158/0x1d8 >[6130499.582593] kernfs_seq_show+0x30/0x38 >[6130499.583008] kthread+0x130/0x138 >[6130499.583438] seq_read+0x148/0x478 >[6130499.584519] kernfs_fop_read+0x2c/0x1e0 >[6130499.584950] __vfs_read+0x20/0x48 >[6130499.585253] vfs_read+0x98/0x168 >[6130499.585585] ksys_read+0x6c/0xe8 >[6130499.585915] __arm64_sys_read+0x20/0x28 >[6130499.586354] el0_svc_common.constprop.0+0xa8/0x200 >[6130499.586878] el0_svc_handler+0x30/0x80 >[6130499.587292] el0_svc+0x10/0x14 >[6130539.531135] watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [kubelet:18334] >[6130539.532054] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6130539.538819] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6130539.544772] CPU: 7 PID: 18334 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6130539.545822] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6130539.546690] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6130539.547173] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6130539.547669] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6130539.548199] sp : ffff00001f9a3c20 >[6130539.548547] x29: ffff00001f9a3c20 x28: ffff8bbbb229e000 >[6130539.549085] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6130539.549594] x25: 0000000000000003 x24: 0000000000000000 >[6130539.550162] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6130539.550801] x21: ffff8d8cfa139400 x20: ffff59fd7fe4b8a8 >[6130539.551303] x19: 0000000000000000 x18: 0000000000000000 >[6130539.551803] x17: 0000000000000002 x16: 0000000000000000 >[6130539.552307] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6130539.552839] x13: ffff8b7d1ce6d000 x12: ffff8b7d1ce6c2f3 >[6130539.553394] x11: 0000000000000000 x10: ffff00001f9a3c80 >[6130539.553892] x9 : ffff59fd7ef3dad0 x8 : 0000000000037681 >[6130539.554423] x7 : 0000000000000005 x6 : 0000000000000006 >[6130539.554949] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6130539.555460] x3 : 0000000000000037 x2 : 000033d04031a000 >[6130539.555955] x1 : ffff4a11c03054e0 x0 : 0000000000000037 >[6130539.556457] Call trace: >[6130539.556712] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6130539.557180] memcg_numa_stat_show+0x1d0/0x260 >[6130539.557604] cgroup_seqfile_show+0x50/0xd0 >[6130539.557978] kernfs_seq_show+0x30/0x38 >[6130539.558337] seq_read+0x148/0x478 >[6130539.558663] kernfs_fop_read+0x2c/0x1e0 >[6130539.559039] __vfs_read+0x20/0x48 >[6130539.559356] vfs_read+0x98/0x168 >[6130539.559690] ksys_read+0x6c/0xe8 >[6130539.559994] __arm64_sys_read+0x20/0x28 >[6130539.560476] el0_svc_common.constprop.0+0xa8/0x200 >[6130539.560973] el0_svc_handler+0x30/0x80 >[6130539.561346] el0_svc+0x10/0x14 >[6130543.531735] watchdog: BUG: soft lockup - CPU#46 stuck for 22s! [migration/46:246] >[6130543.532647] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6130543.540195] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6130543.546790] CPU: 46 PID: 246 Comm: migration/46 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6130543.547756] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6130543.548552] pstate: 60000005 (nZCv daif -PAN -UAO) >[6130543.548971] pc : multi_cpu_stop+0xf4/0x180 >[6130543.549382] lr : multi_cpu_stop+0xb4/0x180 >[6130543.549769] sp : ffff00000a493d50 >[6130543.550089] x29: ffff00000a493d50 x28: 0000000000000000 >[6130543.550732] x27: ffff0000575d3938 x26: 0000000000000060 >[6130543.551187] x25: 0000000000000000 x24: 0000000000000000 >[6130543.551652] x23: 0000000000000000 x22: 0000000000000001 >[6130543.552138] x21: ffff0000575d39d4 x20: ffff0000575d39b0 >[6130543.552638] x19: 0000000000000001 x18: 0000000000000000 >[6130543.553135] x17: 0000000000000001 x16: 0000000000000000 >[6130543.553698] x15: 0000000000000000 x14: 0000000000000000 >[6130543.554195] x13: 0000000000000000 x12: 0000000000000000 >[6130543.554731] x11: 0000000000000027 x10: 0000000000000d60 >[6130543.555267] x9 : ffff59fd7ede5a0c x8 : ffff8bcd9e996400 >[6130543.555809] x7 : ffff8bcd59273c00 x6 : 00000000000003ad >[6130543.556339] x5 : 000031d0404a7000 x4 : ffff8bcd9e995c40 >[6130543.556845] x3 : dead000000000200 x2 : ffff0000575d3988 >[6130543.557337] x1 : ffff0000575d39d4 x0 : 0000000000000001 >[6130543.557826] Call trace: >[6130543.558122] multi_cpu_stop+0xf4/0x180 >[6130543.558484] cpu_stopper_thread+0xec/0x178 >[6130543.558882] smpboot_thread_fn+0x158/0x1d8 >[6130543.559312] kthread+0x130/0x138 >[6130583.531259] watchdog: BUG: soft lockup - CPU#15 stuck for 22s! [kubelet:61154] >[6130583.532073] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6130583.538751] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6130583.544649] CPU: 15 PID: 61154 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6130583.545647] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6130583.546500] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6130583.547004] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6130583.547553] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6130583.548043] sp : ffff000059c5bc20 >[6130583.548410] x29: ffff000059c5bc20 x28: ffff8bbcd1e4c000 >[6130583.548984] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6130583.549494] x25: 0000000000000003 x24: 0000000000000000 >[6130583.550171] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6130583.550677] x21: ffff8dbd276cfc00 x20: ffff59fd7fe4b8a8 >[6130583.551197] x19: 0000000000000000 x18: 0000000000000000 >[6130583.551745] x17: 0000000000000002 x16: 0000000000000000 >[6130583.552316] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6130583.552857] x13: ffff8b8bdf96b000 x12: ffff8b8bdf96a311 >[6130583.553344] x11: 0000000000000000 x10: ffff000059c5bc80 >[6130583.553860] x9 : ffff59fd7ef3dad0 x8 : 000000000004ed06 >[6130583.554366] x7 : 0000000000000005 x6 : 0000000000000006 >[6130583.554876] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6130583.555394] x3 : 000000000000003e x2 : 000033d01ff1b000 >[6130583.555953] x1 : ffff4a2ce464c650 x0 : 000000000000007e >[6130583.556524] Call trace: >[6130583.556803] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6130583.557332] memcg_numa_stat_show+0x1d0/0x260 >[6130583.557807] cgroup_seqfile_show+0x50/0xd0 >[6130583.558250] kernfs_seq_show+0x30/0x38 >[6130583.558653] seq_read+0x148/0x478 >[6130583.559094] kernfs_fop_read+0x2c/0x1e0 >[6130583.559519] __vfs_read+0x20/0x48 >[6130583.559876] vfs_read+0x98/0x168 >[6130583.560344] ksys_read+0x6c/0xe8 >[6130583.560677] __arm64_sys_read+0x20/0x28 >[6130583.561080] el0_svc_common.constprop.0+0xa8/0x200 >[6130583.561535] el0_svc_handler+0x30/0x80 >[6130583.561903] el0_svc+0x10/0x14 >[6130627.531497] watchdog: BUG: soft lockup - CPU#30 stuck for 22s! [kubelet:18929] >[6130627.532324] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6130627.538711] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6130627.544514] CPU: 30 PID: 18929 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6130627.545438] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6130627.546382] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6130627.546819] pc : mem_cgroup_nr_lru_pages.isra.7+0x9c/0xc8 >[6130627.547322] lr : mem_cgroup_nr_lru_pages.isra.7+0xac/0xc8 >[6130627.547839] sp : ffff00003001bc20 >[6130627.548159] x29: ffff00003001bc20 x28: 0000000000000000 >[6130627.548671] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6130627.549172] x25: 0000000000000010 x24: 0000000000000000 >[6130627.549678] x23: ffff8b9b9edb65d0 x22: 0000000000000020 >[6130627.550343] x21: ffff59fd7fe49800 x20: ffff59fd7fe4b8a8 >[6130627.550835] x19: 0000000000000004 x18: 0000000000000000 >[6130627.551296] x17: 0000000000000002 x16: 0000000000000000 >[6130627.551761] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6130627.552277] x13: ffff8bac181e5000 x12: ffff8bac181e4326 >[6130627.552823] x11: 0000000000000000 x10: ffff00003001bc80 >[6130627.553358] x9 : ffff59fd7ef3da4c x8 : 000000000000dc5d >[6130627.553872] x7 : 0000000000000004 x6 : 0000000000000005 >[6130627.554380] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6130627.554863] x3 : 0000000000000020 x2 : 000033c040410000 >[6130627.555372] x1 : ffff4a0cfff4d820 x0 : 0000000000000060 >[6130627.555856] Call trace: >[6130627.556089] mem_cgroup_nr_lru_pages.isra.7+0x9c/0xc8 >[6130627.556568] memcg_numa_stat_show+0x14c/0x260 >[6130627.557010] cgroup_seqfile_show+0x50/0xd0 >[6130627.557393] kernfs_seq_show+0x30/0x38 >[6130627.557757] seq_read+0x148/0x478 >[6130627.558089] kernfs_fop_read+0x2c/0x1e0 >[6130627.558465] __vfs_read+0x20/0x48 >[6130627.558798] vfs_read+0x98/0x168 >[6130627.559100] ksys_read+0x6c/0xe8 >[6130627.559414] __arm64_sys_read+0x20/0x28 >[6130627.559910] el0_svc_common.constprop.0+0xa8/0x200 >[6130627.560370] el0_svc_handler+0x30/0x80 >[6130627.560779] el0_svc+0x10/0x14 >[6130667.531121] watchdog: BUG: soft lockup - CPU#6 stuck for 23s! [kubelet:61154] >[6130667.531983] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6130667.538682] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6130667.544586] CPU: 6 PID: 61154 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6130667.545575] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6130667.546431] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6130667.546895] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6130667.547396] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6130667.547904] sp : ffff000059c5bc20 >[6130667.548238] x29: ffff000059c5bc20 x28: ffff8bacbe190000 >[6130667.548750] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6130667.549262] x25: 0000000000000003 x24: 0000000000000000 >[6130667.549963] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6130667.550505] x21: ffff8d8d1fc44000 x20: ffff59fd7fe4b8a8 >[6130667.550985] x19: 0000000000000000 x18: 0000000000000000 >[6130667.551472] x17: 0000000000000002 x16: 0000000000000000 >[6130667.552011] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6130667.552563] x13: ffff8b7d1c80f000 x12: ffff8b7d1c80e2f3 >[6130667.553090] x11: 0000000000000000 x10: ffff000059c5bc80 >[6130667.553616] x9 : ffff59fd7ef3dad0 x8 : 000000000003782b >[6130667.554118] x7 : 0000000000000005 x6 : 0000000000000006 >[6130667.554611] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6130667.555113] x3 : 000000000000000b x2 : 000033904045c000 >[6130667.555591] x1 : ffff4a1f1d0389d0 x0 : 000000000000004b >[6130667.556089] Call trace: >[6130667.556351] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6130667.556837] memcg_numa_stat_show+0x1d0/0x260 >[6130667.557308] cgroup_seqfile_show+0x50/0xd0 >[6130667.557751] kernfs_seq_show+0x30/0x38 >[6130667.558115] seq_read+0x148/0x478 >[6130667.558458] kernfs_fop_read+0x2c/0x1e0 >[6130667.558829] __vfs_read+0x20/0x48 >[6130667.559188] vfs_read+0x98/0x168 >[6130667.559582] ksys_read+0x6c/0xe8 >[6130667.559894] __arm64_sys_read+0x20/0x28 >[6130667.560266] el0_svc_common.constprop.0+0xa8/0x200 >[6130667.560761] el0_svc_handler+0x30/0x80 >[6130667.561178] el0_svc+0x10/0x14 >[6130711.531742] watchdog: BUG: soft lockup - CPU#46 stuck for 23s! [kubelet:18930] >[6130711.532612] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6130711.539277] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6130711.544651] CPU: 46 PID: 18930 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6130711.545620] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6130711.546411] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6130711.546878] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6130711.547360] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6130711.547866] sp : ffff0000303fbc20 >[6130711.548176] x29: ffff0000303fbc20 x28: ffff8d7c2e12e000 >[6130711.548692] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6130711.549295] x25: 0000000000000003 x24: 0000000000000000 >[6130711.549758] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6130711.550223] x21: ffff8dabea5ab000 x20: ffff59fd7fe4b8a8 >[6130711.550707] x19: 0000000000000000 x18: 0000000000000000 >[6130711.551191] x17: 0000000000000002 x16: 0000000000000000 >[6130711.551742] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6130711.552208] x13: ffff8bcbecece000 x12: ffff8bcbececd307 >[6130711.552685] x11: 0000000000000000 x10: ffff0000303fbc80 >[6130711.553171] x9 : ffff59fd7ef3dad0 x8 : 000000000000bd05 >[6130711.553636] x7 : 0000000000000004 x6 : 0000000000000005 >[6130711.554108] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6130711.554634] x3 : 0000000000000022 x2 : 000031c040442000 >[6130711.555102] x1 : ffff4a0cfa1982c0 x0 : 0000000000000022 >[6130711.555645] Call trace: >[6130711.555874] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6130711.556307] memcg_numa_stat_show+0x1d0/0x260 >[6130711.556707] cgroup_seqfile_show+0x50/0xd0 >[6130711.557087] kernfs_seq_show+0x30/0x38 >[6130711.557452] seq_read+0x148/0x478 >[6130711.557763] kernfs_fop_read+0x2c/0x1e0 >[6130711.558127] __vfs_read+0x20/0x48 >[6130711.558447] vfs_read+0x98/0x168 >[6130711.558774] ksys_read+0x6c/0xe8 >[6130711.559065] __arm64_sys_read+0x20/0x28 >[6130711.559573] el0_svc_common.constprop.0+0xa8/0x200 >[6130711.560040] el0_svc_handler+0x30/0x80 >[6130711.560462] el0_svc+0x10/0x14 >[6130755.531323] watchdog: BUG: soft lockup - CPU#19 stuck for 23s! [kubelet:3202] >[6130755.532134] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6130755.538342] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6130755.544151] CPU: 19 PID: 3202 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6130755.545125] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6130755.545970] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6130755.546492] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6130755.547001] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6130755.547504] sp : ffff00003187bc20 >[6130755.547844] x29: ffff00003187bc20 x28: ffff8bcc71d70000 >[6130755.548420] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6130755.549048] x25: 0000000000000003 x24: 0000000000000000 >[6130755.549575] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6130755.550069] x21: ffff8dbc0d9b0400 x20: ffff59fd7fe4b8a8 >[6130755.550553] x19: 0000000000000000 x18: 0000000000000000 >[6130755.551039] x17: 0000000000000002 x16: 0000000000000000 >[6130755.551537] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6130755.552046] x13: ffff8b9d2022a000 x12: ffff8b9d20229311 >[6130755.552522] x11: 0000000000000000 x10: ffff00003187bc80 >[6130755.552988] x9 : ffff59fd7ef3dad0 x8 : 000000000004ecfe >[6130755.553477] x7 : 0000000000000005 x6 : 0000000000000006 >[6130755.553952] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6130755.554427] x3 : 0000000000000015 x2 : 000031a04048e000 >[6130755.554883] x1 : ffff4a12c0935020 x0 : 0000000000000015 >[6130755.555360] Call trace: >[6130755.555595] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6130755.556061] memcg_numa_stat_show+0x1d0/0x260 >[6130755.556480] cgroup_seqfile_show+0x50/0xd0 >[6130755.556862] kernfs_seq_show+0x30/0x38 >[6130755.557216] seq_read+0x148/0x478 >[6130755.557527] kernfs_fop_read+0x2c/0x1e0 >[6130755.557872] __vfs_read+0x20/0x48 >[6130755.558179] vfs_read+0x98/0x168 >[6130755.558513] ksys_read+0x6c/0xe8 >[6130755.558912] __arm64_sys_read+0x20/0x28 >[6130755.559298] el0_svc_common.constprop.0+0xa8/0x200 >[6130755.559784] el0_svc_handler+0x30/0x80 >[6130755.560142] el0_svc+0x10/0x14 >[6130799.531020] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [migration/2:21] >[6130799.531212] watchdog: BUG: soft lockup - CPU#12 stuck for 23s! [kubelet:7308] >[6130799.531873] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6130799.532603] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6130799.539164] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6130799.545839] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6130799.551870] CPU: 2 PID: 21 Comm: migration/2 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6130799.557481] CPU: 12 PID: 7308 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6130799.558566] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6130799.559550] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6130799.560436] pstate: 60000005 (nZCv daif -PAN -UAO) >[6130799.561321] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6130799.561788] pc : multi_cpu_stop+0x78/0x180 >[6130799.562257] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6130799.562648] lr : multi_cpu_stop+0xb4/0x180 >[6130799.562650] sp : ffff000008d83d50 >[6130799.563172] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6130799.563174] sp : ffff00001b39bc20 >[6130799.563576] x29: ffff000008d83d50 x28: 0000000000000000 >[6130799.563895] x29: ffff00001b39bc20 x28: ffff8bbc281fa000 >[6130799.564436] x27: ffff000058b4b938 x26: 0000000000000060 >[6130799.564771] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6130799.565291] x25: 0000000000000000 x24: 0000000000000000 >[6130799.565865] x25: 0000000000000010 x24: 0000000000000000 >[6130799.566402] x23: 0000000000000000 x22: 0000000000000001 >[6130799.566954] x23: 0000000000000020 x22: ffff59fd7fe49800 >[6130799.567466] x21: ffff000058b4b9d4 x20: ffff000058b4b9b0 >[6130799.567976] x21: ffff8b7bd1817800 x20: ffff59fd7fe4b8a8 >[6130799.568618] x19: 0000000000000001 x18: 0000000000000000 >[6130799.569109] x19: 0000000000000004 x18: 0000000000000000 >[6130799.569601] x17: 0000000000000001 x16: 0000000000000000 >[6130799.570087] x17: 0000000000000002 x16: 0000000000000000 >[6130799.570601] x15: 0000000000000000 x14: 0000000000000000 >[6130799.571086] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6130799.571593] x13: 0000000000000000 x12: 0000000000000000 >[6130799.572104] x13: ffff8b8d7119b000 x12: ffff8b8d7119a345 >[6130799.572649] x11: 0000000000000035 x10: 0000000000000d60 >[6130799.573193] x11: 0000000000000000 x10: ffff00001b39bc80 >[6130799.573695] x9 : ffff59fd7ede5a0c x8 : ffff8b7d9d9c3040 >[6130799.574228] x9 : ffff59fd7ef3dad0 x8 : 0000000000000021 >[6130799.574704] x7 : ffff8b7d1ccd2000 x6 : 00000000000000e6 >[6130799.575193] x7 : 0000000000000001 x6 : 0000000000000002 >[6130799.575737] x5 : 0000318040441000 x4 : ffff8b7d9d9c2880 >[6130799.576299] x5 : 00000000fffffffa x4 : ffffffffffffffff >[6130799.576873] x3 : dead000000000200 x2 : ffff000058b4b960 >[6130799.577371] x3 : 0000000000000015 x2 : 000033a04048e000 >[6130799.577914] x1 : ffff000058b4b9d4 x0 : 0000000000000001 >[6130799.578524] x1 : ffff4a1c6339e490 x0 : 0000000000000055 >[6130799.579129] Call trace: >[6130799.579634] Call trace: >[6130799.580193] multi_cpu_stop+0x78/0x180 >[6130799.580747] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6130799.581010] cpu_stopper_thread+0xec/0x178 >[6130799.581262] memcg_numa_stat_show+0x1d0/0x260 >[6130799.581675] smpboot_thread_fn+0x158/0x1d8 >[6130799.582167] cgroup_seqfile_show+0x50/0xd0 >[6130799.582585] kthread+0x130/0x138 >[6130799.583016] kernfs_seq_show+0x30/0x38 >[6130799.584465] seq_read+0x148/0x478 >[6130799.584810] kernfs_fop_read+0x2c/0x1e0 >[6130799.585191] __vfs_read+0x20/0x48 >[6130799.585517] vfs_read+0x98/0x168 >[6130799.585857] ksys_read+0x6c/0xe8 >[6130799.586155] __arm64_sys_read+0x20/0x28 >[6130799.586503] el0_svc_common.constprop.0+0xa8/0x200 >[6130799.586972] el0_svc_handler+0x30/0x80 >[6130799.587331] el0_svc+0x10/0x14 >[6130839.531119] watchdog: BUG: soft lockup - CPU#6 stuck for 23s! [kubelet:19531] >[6130839.531947] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6130839.538776] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6130839.544499] CPU: 6 PID: 19531 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6130839.545483] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6130839.546373] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6130839.546831] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6130839.547338] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6130839.547835] sp : ffff00001fd4bc20 >[6130839.548256] x29: ffff00001fd4bc20 x28: ffff8bbb5bbbc000 >[6130839.548771] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6130839.549357] x25: 0000000000000003 x24: 0000000000000000 >[6130839.549870] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6130839.550368] x21: ffff8d9c32787c00 x20: ffff59fd7fe4b8a8 >[6130839.550914] x19: 0000000000000000 x18: 0000000000000000 >[6130839.551461] x17: 0000000000000002 x16: 0000000000000000 >[6130839.551958] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6130839.552449] x13: ffff8b7d1c80e000 x12: ffff8b7d1c80d2fd >[6130839.552995] x11: 0000000000000000 x10: ffff00001fd4bc80 >[6130839.553541] x9 : ffff59fd7ef3dad0 x8 : 0000000000014c63 >[6130839.554057] x7 : 0000000000000004 x6 : 0000000000000005 >[6130839.554561] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6130839.555058] x3 : 000000000000000a x2 : 0000339040443000 >[6130839.555563] x1 : ffff4a0ce8150dd0 x0 : 000000000000004a >[6130839.556063] Call trace: >[6130839.556316] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6130839.556802] memcg_numa_stat_show+0x1d0/0x260 >[6130839.557262] cgroup_seqfile_show+0x50/0xd0 >[6130839.557662] kernfs_seq_show+0x30/0x38 >[6130839.558091] seq_read+0x148/0x478 >[6130839.558509] kernfs_fop_read+0x2c/0x1e0 >[6130839.558934] __vfs_read+0x20/0x48 >[6130839.559248] vfs_read+0x98/0x168 >[6130839.559574] ksys_read+0x6c/0xe8 >[6130839.559921] __arm64_sys_read+0x20/0x28 >[6130839.560301] el0_svc_common.constprop.0+0xa8/0x200 >[6130839.560812] el0_svc_handler+0x30/0x80 >[6130839.561202] el0_svc+0x10/0x14 >[6130883.531227] watchdog: BUG: soft lockup - CPU#13 stuck for 23s! [kubelet:11266] >[6130883.531657] watchdog: BUG: soft lockup - CPU#41 stuck for 23s! [migration/41:221] >[6130883.532056] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6130883.532760] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6130883.539196] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6130883.545493] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6130883.551333] CPU: 13 PID: 11266 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6130883.556924] CPU: 41 PID: 221 Comm: migration/41 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6130883.557963] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6130883.558916] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6130883.559782] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6130883.560648] pstate: 60000005 (nZCv daif -PAN -UAO) >[6130883.561135] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6130883.561581] pc : multi_cpu_stop+0x78/0x180 >[6130883.562101] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6130883.562104] sp : ffff0000545dbc20 >[6130883.562555] lr : multi_cpu_stop+0xb4/0x180 >[6130883.563050] x29: ffff0000545dbc20 x28: ffff8dac43168000 >[6130883.563431] sp : ffff00000a3cbd50 >[6130883.563877] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6130883.564390] x29: ffff00000a3cbd50 x28: 0000000000000000 >[6130883.564715] x25: 0000000000000003 x24: 0000000000000000 >[6130883.565227] x27: ffff000035c93938 x26: 0000000000000060 >[6130883.565684] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6130883.566141] x25: 0000000000000000 x24: 0000000000000000 >[6130883.566697] x21: ffff8dccf885b400 x20: ffff59fd7fe4b8a8 >[6130883.567153] x23: 0000000000000000 x22: 0000000000000001 >[6130883.567693] x19: 0000000000000000 x18: 0000000000000000 >[6130883.568339] x21: ffff000035c939d4 x20: ffff000035c939b0 >[6130883.568887] x17: 0000000000000009 x16: 0000000000000000 >[6130883.569383] x19: 0000000000000001 x18: 0000000000000000 >[6130883.569911] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6130883.570363] x17: 0000000000000001 x16: 0000000000000000 >[6130883.570916] x13: ffff8b8d7341c000 x12: ffff8b8d7341b31c >[6130883.571387] x15: 0000000000000000 x14: ffff8dbd956a3800 >[6130883.571872] x11: 0000000000000000 x10: ffff0000545dbc80 >[6130883.572413] x13: 000000000000017b x12: ffff8dbd59919180 >[6130883.572937] x9 : ffff59fd7ef3dad0 x8 : 0000000000018031 >[6130883.573544] x11: 000000000000001e x10: 0000000000000d60 >[6130883.574109] x7 : 0000000000000004 x6 : 0000000000000005 >[6130883.574605] x9 : ffff59fd7ede5a0c x8 : ffff8bcd9e8f9f00 >[6130883.575169] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6130883.575668] x7 : 00107efb5b4d88c8 x6 : 0000000000000291 >[6130883.576196] x3 : 000000000000001f x2 : 000033b0404c0000 >[6130883.576740] x5 : 00000000fa83b2da x4 : ffff8bcd9e8f9740 >[6130883.577236] x1 : ffff4a0f065d53f0 x0 : 000000000000005f >[6130883.577775] x3 : dead000000000200 x2 : ffff000035c93988 >[6130883.578390] Call trace: >[6130883.578874] x1 : ffff000035c939d4 x0 : 0000000000000001 >[6130883.579443] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6130883.579668] Call trace: >[6130883.580184] memcg_numa_stat_show+0x1d0/0x260 >[6130883.580654] multi_cpu_stop+0x78/0x180 >[6130883.580896] cgroup_seqfile_show+0x50/0xd0 >[6130883.581297] cpu_stopper_thread+0xec/0x178 >[6130883.581656] kernfs_seq_show+0x30/0x38 >[6130883.582050] smpboot_thread_fn+0x158/0x1d8 >[6130883.582438] seq_read+0x148/0x478 >[6130883.582799] kthread+0x130/0x138 >[6130883.583201] kernfs_fop_read+0x2c/0x1e0 >[6130883.584177] __vfs_read+0x20/0x48 >[6130883.584484] vfs_read+0x98/0x168 >[6130883.584803] ksys_read+0x6c/0xe8 >[6130883.585152] __arm64_sys_read+0x20/0x28 >[6130883.585522] el0_svc_common.constprop.0+0xa8/0x200 >[6130883.586020] el0_svc_handler+0x30/0x80 >[6130883.586380] el0_svc+0x10/0x14 >[6130927.531199] watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [kubelet:18847] >[6130927.531969] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6130927.538733] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6130927.544498] CPU: 11 PID: 18847 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6130927.545591] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6130927.546510] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6130927.546970] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6130927.547486] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6130927.548171] sp : ffff00001fe3bc20 >[6130927.548511] x29: ffff00001fe3bc20 x28: ffff8bbc22174000 >[6130927.549032] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6130927.549604] x25: 0000000000000010 x24: 0000000000000000 >[6130927.550190] x23: 0000000000000020 x22: ffff59fd7fe49800 >[6130927.550770] x21: ffff8b8c45babc00 x20: ffff59fd7fe4b8a8 >[6130927.551313] x19: 0000000000000004 x18: 0000000000000000 >[6130927.551866] x17: 0000000000000002 x16: 0000000000000000 >[6130927.552441] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6130927.553000] x13: ffff8b8d8045f000 x12: ffff8b8d8045e34b >[6130927.553536] x11: 0000000000000000 x10: ffff00001fe3bc80 >[6130927.554037] x9 : ffff59fd7ef3dad0 x8 : ffff8b8d8045e000 >[6130927.554617] x7 : 0000000000000000 x6 : 0000000000000001 >[6130927.555154] x5 : 00000000fffffffb x4 : ffffffffffffffff >[6130927.555700] x3 : 0000000000000000 x2 : 000031804040f000 >[6130927.556247] x1 : ffff4a1ce6db09a0 x0 : 0000000000000000 >[6130927.556804] Call trace: >[6130927.557037] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6130927.557521] memcg_numa_stat_show+0x1d0/0x260 >[6130927.558076] cgroup_seqfile_show+0x50/0xd0 >[6130927.558456] kernfs_seq_show+0x30/0x38 >[6130927.558805] seq_read+0x148/0x478 >[6130927.559167] kernfs_fop_read+0x2c/0x1e0 >[6130927.559563] __vfs_read+0x20/0x48 >[6130927.559953] vfs_read+0x98/0x168 >[6130927.560273] ksys_read+0x6c/0xe8 >[6130927.560584] __arm64_sys_read+0x20/0x28 >[6130927.561038] el0_svc_common.constprop.0+0xa8/0x200 >[6130927.561523] el0_svc_handler+0x30/0x80 >[6130927.561915] el0_svc+0x10/0x14 >[6130967.531016] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kubelet:18924] >[6130967.531862] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6130967.538611] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6130967.544351] CPU: 1 PID: 18924 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6130967.545346] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6130967.546234] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6130967.546691] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6130967.547368] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6130967.547874] sp : ffff00003e19bc20 >[6130967.548272] x29: ffff00003e19bc20 x28: ffff8b9b82202000 >[6130967.548780] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6130967.549313] x25: 0000000000000003 x24: 0000000000000000 >[6130967.549802] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6130967.550357] x21: ffff8dabe9fc1000 x20: ffff59fd7fe4b8a8 >[6130967.550972] x19: 0000000000000000 x18: 0000000000000000 >[6130967.551478] x17: 0000000000000002 x16: 0000000000000000 >[6130967.552089] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6130967.552681] x13: ffff8b7d1ada7000 x12: ffff8b7d1ada6307 >[6130967.553214] x11: 0000000000000000 x10: ffff00003e19bc80 >[6130967.553714] x9 : ffff59fd7ef3dad0 x8 : 000000000000c092 >[6130967.554270] x7 : 0000000000000004 x6 : 0000000000000005 >[6130967.554803] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6130967.555312] x3 : 0000000000000026 x2 : 000033c0404a6000 >[6130967.555853] x1 : ffff4a0cf8ecaa30 x0 : 0000000000000066 >[6130967.556374] Call trace: >[6130967.556627] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6130967.557173] memcg_numa_stat_show+0x1d0/0x260 >[6130967.557690] cgroup_seqfile_show+0x50/0xd0 >[6130967.558120] kernfs_seq_show+0x30/0x38 >[6130967.558617] seq_read+0x148/0x478 >[6130967.558938] kernfs_fop_read+0x2c/0x1e0 >[6130967.559301] __vfs_read+0x20/0x48 >[6130967.559638] vfs_read+0x98/0x168 >[6130967.559964] ksys_read+0x6c/0xe8 >[6130967.560290] __arm64_sys_read+0x20/0x28 >[6130967.560721] el0_svc_common.constprop.0+0xa8/0x200 >[6130967.561174] el0_svc_handler+0x30/0x80 >[6130967.561565] el0_svc+0x10/0x14 >[6131011.531227] watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [kubelet:24254] >[6131011.531458] watchdog: BUG: soft lockup - CPU#28 stuck for 22s! [migration/28:154] >[6131011.532016] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6131011.532758] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6131011.539224] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6131011.545719] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6131011.551748] CPU: 13 PID: 24254 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6131011.557558] CPU: 28 PID: 154 Comm: migration/28 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6131011.558557] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6131011.559648] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6131011.560603] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6131011.561526] pstate: 60000005 (nZCv daif -PAN -UAO) >[6131011.562033] pc : mem_cgroup_nr_lru_pages.isra.7+0x9c/0xc8 >[6131011.562036] lr : mem_cgroup_nr_lru_pages.isra.7+0xac/0xc8 >[6131011.562530] pc : multi_cpu_stop+0xf4/0x180 >[6131011.563113] sp : ffff00001f7ebc20 >[6131011.563684] lr : multi_cpu_stop+0xb4/0x180 >[6131011.563686] sp : ffff00000a1b3d50 >[6131011.564078] x29: ffff00001f7ebc20 x28: 0000000000000000 >[6131011.564431] x29: ffff00000a1b3d50 x28: 0000000000000000 >[6131011.564849] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6131011.565170] x27: ffff00001f8eb938 x26: 0000000000000060 >[6131011.565657] x25: 0000000000000010 x24: 0000000000000000 >[6131011.566143] x25: 0000000000000000 x24: 0000000000000000 >[6131011.566678] x23: ffff8b7bb619e5d0 x22: 0000000000000020 >[6131011.567311] x23: 0000000000000000 x22: 0000000000000001 >[6131011.567880] x21: ffff59fd7fe49800 x20: ffff59fd7fe4b8a8 >[6131011.568371] x21: ffff00001f8eb9d4 x20: ffff00001f8eb9b0 >[6131011.568901] x19: 0000000000000004 x18: 0000000000000000 >[6131011.569380] x19: 0000000000000001 x18: 0000000000000000 >[6131011.569901] x17: 000000000000000a x16: 0000000000000000 >[6131011.570397] x17: 0000000000000001 x16: 0000000000000000 >[6131011.570913] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6131011.571436] x15: 0000000000000000 x14: 0000000000000000 >[6131011.571952] x13: ffff8b8d7341c000 x12: ffff8b8d7341b326 >[6131011.572445] x13: 0000000000000000 x12: 0000000000000000 >[6131011.572959] x11: 0000000000000000 x10: ffff00001f7ebc80 >[6131011.573459] x11: 0000000000000071 x10: 0000000000000d60 >[6131011.574099] x9 : ffff59fd7ef3da4c x8 : 000000000000de20 >[6131011.574624] x9 : ffff59fd7ede5a0c x8 : ffff8bad9e953040 >[6131011.575136] x7 : 0000000000000004 x6 : 0000000000000005 >[6131011.575650] x7 : ffff8bad55fa3c00 x6 : 000000000000010a >[6131011.576141] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6131011.576651] x5 : 000031b040475000 x4 : ffff8bad9e952880 >[6131011.577271] x3 : 000000000000003f x2 : 000033d040252000 >[6131011.577796] x3 : dead000000000200 x2 : ffff00001f8eb988 >[6131011.578332] x1 : ffff4a0f123b1a50 x0 : 000000000000003f >[6131011.578876] x1 : ffff00001f8eb9d4 x0 : 0000000000000001 >[6131011.579364] Call trace: >[6131011.579890] Call trace: >[6131011.580388] mem_cgroup_nr_lru_pages.isra.7+0x9c/0xc8 >[6131011.580870] multi_cpu_stop+0xf4/0x180 >[6131011.581151] memcg_numa_stat_show+0x14c/0x260 >[6131011.581409] cpu_stopper_thread+0xec/0x178 >[6131011.581880] cgroup_seqfile_show+0x50/0xd0 >[6131011.582212] smpboot_thread_fn+0x158/0x1d8 >[6131011.582610] kernfs_seq_show+0x30/0x38 >[6131011.582995] kthread+0x130/0x138 >[6131011.583392] seq_read+0x148/0x478 >[6131011.584939] kernfs_fop_read+0x2c/0x1e0 >[6131011.585338] __vfs_read+0x20/0x48 >[6131011.585687] vfs_read+0x98/0x168 >[6131011.586007] ksys_read+0x6c/0xe8 >[6131011.586347] __arm64_sys_read+0x20/0x28 >[6131011.586741] el0_svc_common.constprop.0+0xa8/0x200 >[6131011.587387] el0_svc_handler+0x30/0x80 >[6131011.587772] el0_svc+0x10/0x14 >[6131051.531295] watchdog: BUG: soft lockup - CPU#17 stuck for 22s! [kubelet:19531] >[6131051.532197] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6131051.538599] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6131051.544310] CPU: 17 PID: 19531 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6131051.545262] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6131051.546106] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6131051.546541] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6131051.547138] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6131051.547667] sp : ffff00001fd4bc20 >[6131051.548007] x29: ffff00001fd4bc20 x28: ffff8d8cd3fa8000 >[6131051.548580] x27: ffff59fd7fe4bbf4 x26: 000000000000082b >[6131051.549083] x25: 0000000000000003 x24: 0000000000000000 >[6131051.549561] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6131051.550045] x21: ffff8d7c3125dc00 x20: ffff59fd7fe4b8a8 >[6131051.550530] x19: 0000000000000000 x18: 0000000000000000 >[6131051.551020] x17: 0000000000000002 x16: 0000000000000000 >[6131051.551511] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6131051.552041] x13: ffff8b9b753c7000 x12: ffff8b9b753c62e9 >[6131051.552535] x11: 0000000000000000 x10: ffff00001fd4bc80 >[6131051.553029] x9 : ffff59fd7ef3dad0 x8 : 00000000000206c9 >[6131051.553502] x7 : 0000000000000005 x6 : 0000000000000006 >[6131051.553984] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6131051.554477] x3 : 0000000000000010 x2 : 000033a040411000 >[6131051.554945] x1 : ffff4a1dcf26db80 x0 : 0000000000000050 >[6131051.555505] Call trace: >[6131051.555740] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6131051.556191] memcg_numa_stat_show+0x1d0/0x260 >[6131051.556633] cgroup_seqfile_show+0x50/0xd0 >[6131051.557139] kernfs_seq_show+0x30/0x38 >[6131051.557500] seq_read+0x148/0x478 >[6131051.557806] kernfs_fop_read+0x2c/0x1e0 >[6131051.558167] __vfs_read+0x20/0x48 >[6131051.558549] vfs_read+0x98/0x168 >[6131051.558860] ksys_read+0x6c/0xe8 >[6131051.559203] __arm64_sys_read+0x20/0x28 >[6131051.559579] el0_svc_common.constprop.0+0xa8/0x200 >[6131051.560022] el0_svc_handler+0x30/0x80 >[6131051.560370] el0_svc+0x10/0x14 >[6131095.531130] watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [migration/7:46] >[6131095.531752] watchdog: BUG: soft lockup - CPU#47 stuck for 22s! [kubelet:61155] >[6131095.531955] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6131095.532584] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6131095.539189] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6131095.545404] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6131095.551404] CPU: 7 PID: 46 Comm: migration/7 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6131095.556961] CPU: 47 PID: 61155 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6131095.557945] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6131095.558917] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6131095.559788] pstate: 60000005 (nZCv daif -PAN -UAO) >[6131095.560634] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6131095.561100] pc : multi_cpu_stop+0xf4/0x180 >[6131095.561527] pc : mem_cgroup_iter+0xd4/0x378 >[6131095.561925] lr : multi_cpu_stop+0xb4/0x180 >[6131095.561928] sp : ffff000008e4bd50 >[6131095.562313] lr : mem_cgroup_iter+0xc4/0x378 >[6131095.562717] x29: ffff000008e4bd50 x28: 0000000000000000 >[6131095.563036] sp : ffff000059dbbc30 >[6131095.563436] x27: ffff000062ffb668 x26: 0000000000000060 >[6131095.563924] x29: ffff000059dbbc30 x28: ffff8b9b3f7a8000 >[6131095.564257] x25: 0000000000000000 x24: 0000000000000000 >[6131095.564747] x27: ffff59fd7f671bf8 x26: ffff59fd7f8b5a88 >[6131095.565242] x23: 0000000000000000 x22: 0000000000000001 >[6131095.565729] x25: 0000000000000010 x24: 0000000000000001 >[6131095.566253] x21: ffff000062ffb704 x20: ffff000062ffb6e0 >[6131095.566856] x23: ffff8b9b3f7a8000 x22: 0000000000000000 >[6131095.567365] x19: 0000000000000001 x18: 0000000000000000 >[6131095.567863] x21: ffff8b9b3f7a8000 x20: ffff8dbd9e88a000 >[6131095.568426] x17: 0000000000000001 x16: 0000000000000000 >[6131095.568948] x19: ffff8babf4970000 x18: 0000000000000000 >[6131095.569453] x15: 0000000000000001 x14: ffff8dbc5d0e7400 >[6131095.570070] x17: 0000000000000002 x16: 0000000000000000 >[6131095.570631] x13: 00000000000008e7 x12: ffff8dbc5d588580 >[6131095.571200] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6131095.571754] x11: 0000000000000029 x10: 0000000000000d60 >[6131095.572323] x13: ffff8bcd52fd9000 x12: ffff8bcd52fd82f3 >[6131095.572873] x9 : ffff59fd7ede5a0c x8 : ffff8b7d9da7f540 >[6131095.573440] x11: 0000000000000000 x10: ffff000059dbbc80 >[6131095.573995] x7 : ffff8b7d6a381000 x6 : 000000000000005d >[6131095.574548] x9 : ffff59fd7edd46f8 x8 : 00000000000379ca >[6131095.575108] x5 : 00003180404be000 x4 : ffff8b7d9da7ed80 >[6131095.575672] x7 : 0000000000000005 x6 : 0000000000000006 >[6131095.576223] x3 : dead000000000200 x2 : ffff000062ffb690 >[6131095.576904] x5 : 00000000fffffff6 x4 : 0000000000000000 >[6131095.577456] x1 : ffff000062ffb704 x0 : 0000000000000001 >[6131095.577927] x3 : ffff8bcca40be058 x2 : ffff8b9b3f7a8000 >[6131095.578450] Call trace: >[6131095.578907] x1 : ffff8babf4970048 x0 : ffff8babf4970000 >[6131095.579417] multi_cpu_stop+0xf4/0x180 >[6131095.579704] Call trace: >[6131095.580265] cpu_stopper_thread+0xec/0x178 >[6131095.580605] mem_cgroup_iter+0xd4/0x378 >[6131095.580861] smpboot_thread_fn+0x158/0x1d8 >[6131095.581228] memcg_numa_stat_show+0x1e4/0x260 >[6131095.581608] kthread+0x130/0x138 >[6131095.581986] cgroup_seqfile_show+0x50/0xd0 >[6131095.583159] kernfs_seq_show+0x30/0x38 >[6131095.583577] seq_read+0x148/0x478 >[6131095.583933] kernfs_fop_read+0x2c/0x1e0 >[6131095.584345] __vfs_read+0x20/0x48 >[6131095.584720] vfs_read+0x98/0x168 >[6131095.585082] ksys_read+0x6c/0xe8 >[6131095.585437] __arm64_sys_read+0x20/0x28 >[6131095.585853] el0_svc_common.constprop.0+0xa8/0x200 >[6131095.586457] el0_svc_handler+0x30/0x80 >[6131095.586911] el0_svc+0x10/0x14 >[6131139.531016] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kubelet:18804] >[6131139.531839] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6131139.538514] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6131139.544249] CPU: 1 PID: 18804 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6131139.545277] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6131139.546267] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6131139.546767] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6131139.547326] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6131139.547831] sp : ffff00001fe63c20 >[6131139.548163] x29: ffff00001fe63c20 x28: ffff8d8cef924000 >[6131139.548683] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6131139.549238] x25: 0000000000000003 x24: 0000000000000000 >[6131139.549780] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6131139.550300] x21: ffff8dac23555c00 x20: ffff59fd7fe4b8a8 >[6131139.550802] x19: 0000000000000000 x18: 0000000000000000 >[6131139.551317] x17: 0000000000000002 x16: 0000000000000000 >[6131139.551835] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6131139.552388] x13: ffff8b7d1ada2000 x12: ffff8b7d1ada1307 >[6131139.552948] x11: 0000000000000000 x10: ffff00001fe63c80 >[6131139.553500] x9 : ffff59fd7ef3dad0 x8 : 000000000000bfa8 >[6131139.554044] x7 : 0000000000000004 x6 : 0000000000000005 >[6131139.554582] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6131139.555106] x3 : 0000000000000039 x2 : 000033d0401bc000 >[6131139.555626] x1 : ffff4a2dd20719f0 x0 : 0000000000000039 >[6131139.556224] Call trace: >[6131139.556474] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6131139.556977] memcg_numa_stat_show+0x1d0/0x260 >[6131139.557486] cgroup_seqfile_show+0x50/0xd0 >[6131139.557892] kernfs_seq_show+0x30/0x38 >[6131139.558267] seq_read+0x148/0x478 >[6131139.558603] kernfs_fop_read+0x2c/0x1e0 >[6131139.558988] __vfs_read+0x20/0x48 >[6131139.559376] vfs_read+0x98/0x168 >[6131139.559706] ksys_read+0x6c/0xe8 >[6131139.560094] __arm64_sys_read+0x20/0x28 >[6131139.560509] el0_svc_common.constprop.0+0xa8/0x200 >[6131139.560992] el0_svc_handler+0x30/0x80 >[6131139.561416] el0_svc+0x10/0x14 >[6131183.531182] watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [kubelet:18612] >[6131183.532015] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6131183.538790] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6131183.544594] CPU: 10 PID: 18612 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6131183.545642] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6131183.546688] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6131183.547118] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6131183.547601] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6131183.548094] sp : ffff000050243c20 >[6131183.548413] x29: ffff000050243c20 x28: ffff8b9c693e0000 >[6131183.548948] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6131183.549590] x25: 0000000000000003 x24: 0000000000000000 >[6131183.550149] x23: 0000000000000008 x22: ffff59fd7fe49800 >[6131183.550702] x21: ffff8dccdd6c1800 x20: ffff59fd7fe4b8a8 >[6131183.551262] x19: 0000000000000001 x18: 0000000000000000 >[6131183.551829] x17: 0000000000000002 x16: 0000000000000000 >[6131183.552378] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6131183.552929] x13: ffff8b8d72d56000 x12: ffff8b8d72d5531c >[6131183.553476] x11: 0000000000000000 x10: ffff000050243c80 >[6131183.554040] x9 : ffff59fd7ef3dad0 x8 : 0000000000018ac4 >[6131183.554600] x7 : 0000000000000005 x6 : 0000000000000006 >[6131183.555162] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6131183.555809] x3 : 000000000000001d x2 : 000031b04048e000 >[6131183.556383] x1 : ffff4a0cfa641c18 x0 : 000000000000001d >[6131183.556953] Call trace: >[6131183.557224] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6131183.557752] memcg_numa_stat_show+0x1d0/0x260 >[6131183.558203] cgroup_seqfile_show+0x50/0xd0 >[6131183.558632] kernfs_seq_show+0x30/0x38 >[6131183.558987] seq_read+0x148/0x478 >[6131183.559307] kernfs_fop_read+0x2c/0x1e0 >[6131183.559736] __vfs_read+0x20/0x48 >[6131183.560075] vfs_read+0x98/0x168 >[6131183.560455] ksys_read+0x6c/0xe8 >[6131183.560876] __arm64_sys_read+0x20/0x28 >[6131183.561332] el0_svc_common.constprop.0+0xa8/0x200 >[6131183.561851] el0_svc_handler+0x30/0x80 >[6131183.562246] el0_svc+0x10/0x14 >[6131223.531725] watchdog: BUG: soft lockup - CPU#45 stuck for 22s! [kubelet:18322] >[6131223.532516] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6131223.539053] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6131223.544604] CPU: 45 PID: 18322 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6131223.545669] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6131223.546513] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6131223.546959] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6131223.547516] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6131223.547999] sp : ffff00001f8dbc20 >[6131223.548353] x29: ffff00001f8dbc20 x28: ffff8bbd6148a000 >[6131223.548900] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6131223.549474] x25: 0000000000000003 x24: 0000000000000000 >[6131223.549951] x23: 0000000000000008 x22: ffff59fd7fe49800 >[6131223.550478] x21: ffff8d8d51c6dc00 x20: ffff59fd7fe4b8a8 >[6131223.551029] x19: 0000000000000001 x18: 0000000000000000 >[6131223.551504] x17: 0000000000000002 x16: 0000000000000000 >[6131223.552046] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6131223.552565] x13: ffff8bcd524a7000 x12: ffff8bcd524a62f3 >[6131223.553103] x11: 0000000000000000 x10: ffff00001f8dbc80 >[6131223.553600] x9 : ffff59fd7ef3dad0 x8 : 000000000003815c >[6131223.554105] x7 : 0000000000000005 x6 : 0000000000000006 >[6131223.554593] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6131223.555147] x3 : 0000000000000026 x2 : 000031c0404a6000 >[6131223.555758] x1 : ffff4a2ac40e5998 x0 : 0000000000000026 >[6131223.556245] Call trace: >[6131223.556569] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6131223.557040] memcg_numa_stat_show+0x1d0/0x260 >[6131223.557481] cgroup_seqfile_show+0x50/0xd0 >[6131223.557873] kernfs_seq_show+0x30/0x38 >[6131223.558284] seq_read+0x148/0x478 >[6131223.558636] kernfs_fop_read+0x2c/0x1e0 >[6131223.558983] __vfs_read+0x20/0x48 >[6131223.559320] vfs_read+0x98/0x168 >[6131223.559618] ksys_read+0x6c/0xe8 >[6131223.559908] __arm64_sys_read+0x20/0x28 >[6131223.560256] el0_svc_common.constprop.0+0xa8/0x200 >[6131223.560713] el0_svc_handler+0x30/0x80 >[6131223.561067] el0_svc+0x10/0x14 >[6131267.531151] watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [kubelet:3381] >[6131267.531907] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6131267.538383] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6131267.543953] CPU: 8 PID: 3381 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6131267.544901] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6131267.545956] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6131267.546497] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6131267.547004] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6131267.547593] sp : ffff00001d773c20 >[6131267.547961] x29: ffff00001d773c20 x28: ffff8bcc4c626000 >[6131267.548523] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6131267.549146] x25: 0000000000000003 x24: 0000000000000000 >[6131267.549715] x23: 0000000000000008 x22: ffff59fd7fe49800 >[6131267.550214] x21: ffff8daca74d0c00 x20: ffff59fd7fe4b8a8 >[6131267.550796] x19: 0000000000000001 x18: 0000000000000000 >[6131267.551294] x17: 0000000000000002 x16: 0000000000000000 >[6131267.551817] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6131267.552370] x13: ffff8b8d72fae000 x12: ffff8b8d72fad307 >[6131267.552880] x11: 0000000000000000 x10: ffff00001d773c80 >[6131267.553389] x9 : ffff59fd7ef3dad0 x8 : 000000000000bf27 >[6131267.553925] x7 : 0000000000000004 x6 : 0000000000000005 >[6131267.554438] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6131267.554966] x3 : 0000000000000007 x2 : 00003180404be000 >[6131267.555556] x1 : ffff4a1dd6091718 x0 : 0000000000000007 >[6131267.556062] Call trace: >[6131267.556362] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6131267.556863] memcg_numa_stat_show+0x1d0/0x260 >[6131267.557310] cgroup_seqfile_show+0x50/0xd0 >[6131267.557714] kernfs_seq_show+0x30/0x38 >[6131267.558128] seq_read+0x148/0x478 >[6131267.558452] kernfs_fop_read+0x2c/0x1e0 >[6131267.558835] __vfs_read+0x20/0x48 >[6131267.559216] vfs_read+0x98/0x168 >[6131267.559529] ksys_read+0x6c/0xe8 >[6131267.559849] __arm64_sys_read+0x20/0x28 >[6131267.560259] el0_svc_common.constprop.0+0xa8/0x200 >[6131267.560728] el0_svc_handler+0x30/0x80 >[6131267.561095] el0_svc+0x10/0x14 >[6131311.531150] watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [kubelet:11266] >[6131311.532009] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6131311.538647] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6131311.544325] CPU: 8 PID: 11266 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6131311.545382] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6131311.546276] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6131311.546783] pc : mem_cgroup_nr_lru_pages.isra.7+0x94/0xc8 >[6131311.547365] lr : mem_cgroup_nr_lru_pages.isra.7+0xac/0xc8 >[6131311.547909] sp : ffff0000545dbc20 >[6131311.548230] x29: ffff0000545dbc20 x28: 0000000000000000 >[6131311.548801] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6131311.549407] x25: 0000000000000010 x24: 0000000000000000 >[6131311.549906] x23: ffff8dac274825d0 x22: 0000000000000020 >[6131311.550504] x21: ffff59fd7fe49800 x20: ffff59fd7fe4b8a8 >[6131311.551061] x19: 0000000000000004 x18: 0000000000000000 >[6131311.551554] x17: 0000000000000004 x16: 0000000000000000 >[6131311.552142] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6131311.552737] x13: ffff8b8d72fae000 x12: ffff8b8d72fad327 >[6131311.553235] x11: 0000000000000000 x10: ffff0000545dbc80 >[6131311.553805] x9 : ffff59fd7ef3da4c x8 : 000000000000d977 >[6131311.554270] x7 : 0000000000000004 x6 : 0000000000000005 >[6131311.554920] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6131311.555456] x3 : 0000000000000000 x2 : 000031804040f000 >[6131311.555945] x1 : 0000000000000080 x0 : 0000000000000000 >[6131311.556451] Call trace: >[6131311.556702] mem_cgroup_nr_lru_pages.isra.7+0x94/0xc8 >[6131311.557223] memcg_numa_stat_show+0x14c/0x260 >[6131311.557639] cgroup_seqfile_show+0x50/0xd0 >[6131311.558040] kernfs_seq_show+0x30/0x38 >[6131311.558478] seq_read+0x148/0x478 >[6131311.558804] kernfs_fop_read+0x2c/0x1e0 >[6131311.559157] __vfs_read+0x20/0x48 >[6131311.559512] vfs_read+0x98/0x168 >[6131311.559848] ksys_read+0x6c/0xe8 >[6131311.560190] __arm64_sys_read+0x20/0x28 >[6131311.560594] el0_svc_common.constprop.0+0xa8/0x200 >[6131311.561165] el0_svc_handler+0x30/0x80 >[6131311.561519] el0_svc+0x10/0x14 >[6131351.531229] watchdog: BUG: soft lockup - CPU#13 stuck for 23s! [kubelet:4439] >[6131351.532008] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6131351.538716] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6131351.544233] CPU: 13 PID: 4439 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6131351.545359] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6131351.546275] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6131351.546725] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6131351.547310] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6131351.547902] sp : ffff00001e7ebc20 >[6131351.548225] x29: ffff00001e7ebc20 x28: ffff8d8cf2258000 >[6131351.548771] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6131351.549273] x25: 0000000000000003 x24: 0000000000000000 >[6131351.549827] x23: 0000000000000008 x22: ffff59fd7fe49800 >[6131351.550391] x21: ffff8d9c7af17800 x20: ffff59fd7fe4b8a8 >[6131351.550906] x19: 0000000000000001 x18: 0000000000000000 >[6131351.551472] x17: 0000000000000002 x16: 0000000000000000 >[6131351.552061] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6131351.552562] x13: ffff8b8d7341b000 x12: ffff8b8d7341a2fd >[6131351.553157] x11: 0000000000000000 x10: ffff00001e7ebc80 >[6131351.553720] x9 : ffff59fd7ef3dad0 x8 : 0000000000014b92 >[6131351.554210] x7 : 0000000000000004 x6 : 0000000000000005 >[6131351.554908] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6131351.555423] x3 : 0000000000000030 x2 : 000033d01ff4d000 >[6131351.555985] x1 : ffff4a2dd22bf858 x0 : 0000000000000070 >[6131351.556517] Call trace: >[6131351.556772] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6131351.557297] memcg_numa_stat_show+0x1d0/0x260 >[6131351.557754] cgroup_seqfile_show+0x50/0xd0 >[6131351.558144] kernfs_seq_show+0x30/0x38 >[6131351.558512] seq_read+0x148/0x478 >[6131351.558875] kernfs_fop_read+0x2c/0x1e0 >[6131351.559273] __vfs_read+0x20/0x48 >[6131351.559627] vfs_read+0x98/0x168 >[6131351.559947] ksys_read+0x6c/0xe8 >[6131351.560316] __arm64_sys_read+0x20/0x28 >[6131351.560706] el0_svc_common.constprop.0+0xa8/0x200 >[6131351.561270] el0_svc_handler+0x30/0x80 >[6131351.561638] el0_svc+0x10/0x14 >[6131355.531365] watchdog: BUG: soft lockup - CPU#22 stuck for 22s! [migration/22:123] >[6131355.532210] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6131355.538668] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6131355.544171] CPU: 22 PID: 123 Comm: migration/22 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6131355.545283] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6131355.546136] pstate: 60000005 (nZCv daif -PAN -UAO) >[6131355.546672] pc : multi_cpu_stop+0xf4/0x180 >[6131355.547156] lr : multi_cpu_stop+0xb4/0x180 >[6131355.547545] sp : ffff00000a0bbd50 >[6131355.547878] x29: ffff00000a0bbd50 x28: 0000000000000000 >[6131355.548352] x27: ffff000030123938 x26: 0000000000000060 >[6131355.548918] x25: 0000000000000000 x24: 0000000000000000 >[6131355.549422] x23: 0000000000000000 x22: 0000000000000001 >[6131355.549970] x21: ffff0000301239d4 x20: ffff0000301239b0 >[6131355.550534] x19: 0000000000000001 x18: 0000000000000000 >[6131355.551014] x17: 0000000000000001 x16: 0000000000000000 >[6131355.551601] x15: 0000000000000000 x14: 0000000000000000 >[6131355.552087] x13: 0000000000000000 x12: 0000000000000000 >[6131355.552654] x11: 00000000000000d4 x10: 0000000000000d60 >[6131355.553126] x9 : ffff59fd7ede5a0c x8 : ffff8b9d9e90d2c0 >[6131355.553693] x7 : ffff8b9d24ffc400 x6 : 00000000000002b1 >[6131355.554184] x5 : 000031a0404a7000 x4 : ffff8b9d9e90cb00 >[6131355.554825] x3 : dead000000000200 x2 : ffff000030123988 >[6131355.555350] x1 : ffff0000301239d4 x0 : 0000000000000001 >[6131355.555837] Call trace: >[6131355.556067] multi_cpu_stop+0xf4/0x180 >[6131355.556484] cpu_stopper_thread+0xec/0x178 >[6131355.556980] smpboot_thread_fn+0x158/0x1d8 >[6131355.557350] kthread+0x130/0x138 >[6131395.531244] watchdog: BUG: soft lockup - CPU#14 stuck for 23s! [kubelet:18930] >[6131395.531719] watchdog: BUG: soft lockup - CPU#45 stuck for 23s! [migration/45:241] >[6131395.532121] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6131395.532966] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6131395.540331] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6131395.547439] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6131395.553195] CPU: 14 PID: 18930 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6131395.559184] CPU: 45 PID: 241 Comm: migration/45 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6131395.560243] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6131395.561286] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6131395.562233] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6131395.563102] pstate: 60000005 (nZCv daif -PAN -UAO) >[6131395.563594] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6131395.564052] pc : multi_cpu_stop+0x78/0x180 >[6131395.564705] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6131395.565085] lr : multi_cpu_stop+0xb4/0x180 >[6131395.565087] sp : ffff00000a46bd50 >[6131395.565636] sp : ffff0000303fbc20 >[6131395.566038] x29: ffff00000a46bd50 x28: 0000000000000000 >[6131395.566421] x29: ffff0000303fbc20 x28: ffff8bcc591f6000 >[6131395.566764] x27: ffff00001f91b938 x26: 0000000000000060 >[6131395.567272] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6131395.567777] x25: 0000000000000000 x24: 0000000000000000 >[6131395.568267] x25: 0000000000000003 x24: 0000000000000000 >[6131395.568731] x23: 0000000000000000 x22: 0000000000000001 >[6131395.569192] x23: 0000000000000008 x22: ffff59fd7fe49800 >[6131395.569666] x21: ffff00001f91b9d4 x20: ffff00001f91b9b0 >[6131395.570151] x21: ffff8dccfd369000 x20: ffff59fd7fe4b8a8 >[6131395.570643] x19: 0000000000000001 x18: 0000000000000000 >[6131395.571129] x19: 0000000000000001 x18: 0000000000000000 >[6131395.571595] x17: 0000000000000001 x16: 0000000000000000 >[6131395.572091] x17: 0000000000000002 x16: 0000000000000000 >[6131395.572589] x15: 0000000000000000 x14: 0000000000000000 >[6131395.573073] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6131395.573587] x13: 0000000000000000 x12: 0000000000000000 >[6131395.574085] x13: ffff8b8d722d4000 x12: ffff8b8d722d331c >[6131395.574671] x11: 000000000000000c x10: 0000000000000d60 >[6131395.575165] x11: 0000000000000000 x10: ffff0000303fbc80 >[6131395.575665] x9 : ffff59fd7ede5a0c x8 : ffff8bcd9e990dc0 >[6131395.576224] x9 : ffff59fd7ef3dad0 x8 : 00000000000190d4 >[6131395.576778] x7 : 0021d0622b6a0b86 x6 : 00000000000003b7 >[6131395.577276] x7 : 0000000000000005 x6 : 0000000000000006 >[6131395.577858] x5 : 00000000fa83b2da x4 : ffff8bcd9e990600 >[6131395.578428] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6131395.578932] x3 : dead000000000200 x2 : ffff00001f91b988 >[6131395.579493] x3 : 0000000000000018 x2 : 000033b040411000 >[6131395.579998] x1 : ffff00001f91b9d4 x0 : 0000000000000001 >[6131395.580497] x1 : ffff4a0f0c12ca98 x0 : 0000000000000058 >[6131395.581009] Call trace: >[6131395.581492] Call trace: >[6131395.581971] multi_cpu_stop+0x78/0x180 >[6131395.582209] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6131395.582453] cpu_stopper_thread+0xec/0x178 >[6131395.582795] memcg_numa_stat_show+0x1d0/0x260 >[6131395.583318] smpboot_thread_fn+0x158/0x1d8 >[6131395.583724] cgroup_seqfile_show+0x50/0xd0 >[6131395.584131] kthread+0x130/0x138 >[6131395.584667] kernfs_seq_show+0x30/0x38 >[6131395.585781] seq_read+0x148/0x478 >[6131395.586115] kernfs_fop_read+0x2c/0x1e0 >[6131395.586506] __vfs_read+0x20/0x48 >[6131395.586861] vfs_read+0x98/0x168 >[6131395.587149] ksys_read+0x6c/0xe8 >[6131395.587507] __arm64_sys_read+0x20/0x28 >[6131395.587883] el0_svc_common.constprop.0+0xa8/0x200 >[6131395.588359] el0_svc_handler+0x30/0x80 >[6131395.588746] el0_svc+0x10/0x14 >[6131439.531197] watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [kubelet:18288] >[6131439.532061] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6131439.539480] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6131439.545946] CPU: 11 PID: 18288 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6131439.546956] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6131439.547851] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6131439.548322] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6131439.548838] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6131439.549445] sp : ffff00001f91bc20 >[6131439.549811] x29: ffff00001f91bc20 x28: ffff8bcc346b2000 >[6131439.550308] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6131439.550818] x25: 0000000000000010 x24: 0000000000000000 >[6131439.551321] x23: 0000000000000020 x22: ffff59fd7fe49800 >[6131439.551818] x21: ffff8b9bb8247c00 x20: ffff59fd7fe4b8a8 >[6131439.552339] x19: 0000000000000004 x18: 0000000000000000 >[6131439.552935] x17: 0000000000000009 x16: 0000000000000000 >[6131439.553431] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6131439.554049] x13: ffff8b8d8045f000 x12: ffff8b8d8045e350 >[6131439.554558] x11: 0000000000000000 x10: ffff00001f91bc80 >[6131439.555054] x9 : ffff59fd7ef3dad0 x8 : ffff8b8d8045e000 >[6131439.555597] x7 : 0000000000000000 x6 : 0000000000000001 >[6131439.556062] x5 : 00000000fffffffb x4 : ffffffffffffffff >[6131439.556590] x3 : 0000000000000003 x2 : 000031804045a000 >[6131439.557137] x1 : ffff4a2dc8110300 x0 : 0000000000000003 >[6131439.557646] Call trace: >[6131439.557937] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6131439.558453] memcg_numa_stat_show+0x1d0/0x260 >[6131439.558939] cgroup_seqfile_show+0x50/0xd0 >[6131439.559327] kernfs_seq_show+0x30/0x38 >[6131439.559768] seq_read+0x148/0x478 >[6131439.560082] kernfs_fop_read+0x2c/0x1e0 >[6131439.560443] __vfs_read+0x20/0x48 >[6131439.560818] vfs_read+0x98/0x168 >[6131439.561138] ksys_read+0x6c/0xe8 >[6131439.561495] __arm64_sys_read+0x20/0x28 >[6131439.561874] el0_svc_common.constprop.0+0xa8/0x200 >[6131439.562352] el0_svc_handler+0x30/0x80 >[6131439.562716] el0_svc+0x10/0x14 >[6131479.531680] watchdog: BUG: soft lockup - CPU#42 stuck for 23s! [kubelet:4440] >[6131479.532525] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6131479.539709] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6131479.545336] CPU: 42 PID: 4440 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6131479.546352] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6131479.547194] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6131479.547657] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6131479.548155] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6131479.548653] sp : ffff0000329abc20 >[6131479.548974] x29: ffff0000329abc20 x28: ffff8bbd713a8000 >[6131479.549461] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6131479.549970] x25: 0000000000000003 x24: 0000000000000000 >[6131479.550447] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6131479.550933] x21: ffff8dad9a00e400 x20: ffff59fd7fe4b8a8 >[6131479.551439] x19: 0000000000000000 x18: 0000000000000000 >[6131479.551948] x17: 000000000000000a x16: 0000000000000000 >[6131479.552443] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6131479.552961] x13: ffff8bcd52fe2000 x12: ffff8bcd52fe1307 >[6131479.553448] x11: 0000000000000000 x10: ffff0000329abc80 >[6131479.554047] x9 : ffff59fd7ef3dad0 x8 : 000000000000c0e5 >[6131479.554535] x7 : 0000000000000004 x6 : 0000000000000005 >[6131479.555125] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6131479.555606] x3 : 0000000000000029 x2 : 000033d02002e000 >[6131479.556077] x1 : ffff4a2441c968a0 x0 : 0000000000000069 >[6131479.556610] Call trace: >[6131479.556857] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6131479.557344] memcg_numa_stat_show+0x1d0/0x260 >[6131479.557813] cgroup_seqfile_show+0x50/0xd0 >[6131479.558206] kernfs_seq_show+0x30/0x38 >[6131479.558551] seq_read+0x148/0x478 >[6131479.558876] kernfs_fop_read+0x2c/0x1e0 >[6131479.559240] __vfs_read+0x20/0x48 >[6131479.559553] vfs_read+0x98/0x168 >[6131479.559856] ksys_read+0x6c/0xe8 >[6131479.560163] __arm64_sys_read+0x20/0x28 >[6131479.560559] el0_svc_common.constprop.0+0xa8/0x200 >[6131479.560983] el0_svc_handler+0x30/0x80 >[6131479.561335] el0_svc+0x10/0x14 >[6131523.531554] watchdog: BUG: soft lockup - CPU#34 stuck for 23s! [kubelet:18334] >[6131523.532436] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6131523.539127] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6131523.545089] CPU: 34 PID: 18334 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6131523.546206] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6131523.547176] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6131523.547671] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6131523.548206] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6131523.548817] sp : ffff00001f9a3c20 >[6131523.549169] x29: ffff00001f9a3c20 x28: ffff8bad31ada000 >[6131523.549721] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6131523.550235] x25: 0000000000000003 x24: 0000000000000000 >[6131523.550799] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6131523.551350] x21: ffff8dbd59ff0400 x20: ffff59fd7fe4b8a8 >[6131523.551949] x19: 0000000000000000 x18: 0000000000000000 >[6131523.552457] x17: 0000000000000002 x16: 0000000000000000 >[6131523.553055] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6131523.553729] x13: ffff8bbd32628000 x12: ffff8bbd32627311 >[6131523.554268] x11: 0000000000000000 x10: ffff00001f9a3c80 >[6131523.554781] x9 : ffff59fd7ef3dad0 x8 : 000000000004ee38 >[6131523.555280] x7 : 0000000000000005 x6 : 0000000000000006 >[6131523.555828] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6131523.556366] x3 : 0000000000000013 x2 : 000033a04045c000 >[6131523.556867] x1 : ffff4a2b48115990 x0 : 0000000000000053 >[6131523.557351] Call trace: >[6131523.557617] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6131523.558103] memcg_numa_stat_show+0x1d0/0x260 >[6131523.558520] cgroup_seqfile_show+0x50/0xd0 >[6131523.558928] kernfs_seq_show+0x30/0x38 >[6131523.559292] seq_read+0x148/0x478 >[6131523.559607] kernfs_fop_read+0x2c/0x1e0 >[6131523.559984] __vfs_read+0x20/0x48 >[6131523.560318] vfs_read+0x98/0x168 >[6131523.560641] ksys_read+0x6c/0xe8 >[6131523.560957] __arm64_sys_read+0x20/0x28 >[6131523.561331] el0_svc_common.constprop.0+0xa8/0x200 >[6131523.561801] el0_svc_handler+0x30/0x80 >[6131523.562170] el0_svc+0x10/0x14 >[6131567.531294] watchdog: BUG: soft lockup - CPU#17 stuck for 23s! [kubelet:60635] >[6131567.532235] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6131567.539588] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6131567.545179] CPU: 17 PID: 60635 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6131567.546147] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6131567.547062] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6131567.547501] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6131567.547970] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6131567.548431] sp : ffff00001df63c20 >[6131567.548742] x29: ffff00001df63c20 x28: ffff8d8cbad16000 >[6131567.549214] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6131567.549705] x25: 0000000000000003 x24: 0000000000000000 >[6131567.550199] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6131567.550690] x21: ffff8dccd53a4c00 x20: ffff59fd7fe4b8a8 >[6131567.551170] x19: 0000000000000000 x18: 0000000000000000 >[6131567.551656] x17: 0000000000000002 x16: 0000000000000000 >[6131567.552127] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6131567.552608] x13: ffff8b9b753c1000 x12: ffff8b9b753c031c >[6131567.553220] x11: 0000000000000000 x10: ffff00001df63c80 >[6131567.553702] x9 : ffff59fd7ef3dad0 x8 : 0000000000018b14 >[6131567.554190] x7 : 0000000000000005 x6 : 0000000000000006 >[6131567.554697] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6131567.555202] x3 : 0000000000000019 x2 : 000033b04042a000 >[6131567.555695] x1 : ffff4a2b4ada2400 x0 : 0000000000000059 >[6131567.556173] Call trace: >[6131567.556406] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6131567.556874] memcg_numa_stat_show+0x1d0/0x260 >[6131567.557277] cgroup_seqfile_show+0x50/0xd0 >[6131567.557719] kernfs_seq_show+0x30/0x38 >[6131567.558073] seq_read+0x148/0x478 >[6131567.558386] kernfs_fop_read+0x2c/0x1e0 >[6131567.558766] __vfs_read+0x20/0x48 >[6131567.559118] vfs_read+0x98/0x168 >[6131567.559429] ksys_read+0x6c/0xe8 >[6131567.559804] __arm64_sys_read+0x20/0x28 >[6131567.560165] el0_svc_common.constprop.0+0xa8/0x200 >[6131567.560670] el0_svc_handler+0x30/0x80 >[6131567.561024] el0_svc+0x10/0x14 >[6131611.531151] watchdog: BUG: soft lockup - CPU#8 stuck for 23s! [kubelet:61154] >[6131611.532057] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6131611.538566] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6131611.544398] CPU: 8 PID: 61154 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6131611.545507] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6131611.546477] pstate: 00000005 (nzcv daif -PAN -UAO) >[6131611.546932] pc : __ll_sc___cmpxchg_case_mb_8+0x18/0x28 >[6131611.547459] lr : mem_cgroup_iter+0x32c/0x378 >[6131611.547935] sp : ffff000059c5bc30 >[6131611.548249] x29: ffff000059c5bc30 x28: ffff8b9c45d9c000 >[6131611.548827] x27: ffff59fd7f671bf8 x26: ffff59fd7f8b5a88 >[6131611.549362] x25: 0000000000000010 x24: 0000000000000001 >[6131611.549877] x23: ffff8b9c45d9c000 x22: 0000000000000000 >[6131611.550439] x21: ffff8b9c45d9c000 x20: ffff8dbd9e88a000 >[6131611.550933] x19: ffff8b8c8e97e000 x18: 0000000000000000 >[6131611.551485] x17: 0000000000000000 x16: 0000000000000006 >[6131611.551965] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6131611.552482] x13: ffff8b8d72fac000 x12: ffff8b8d72fab34b >[6131611.553220] x11: 0000000000000000 x10: ffff000059c5bc80 >[6131611.553739] x9 : ffff59fd7edd46f8 x8 : ffff8b8d72fab000 >[6131611.554238] x7 : 0000000000000000 x6 : 0000000000000001 >[6131611.554801] x5 : 00000000fffffffb x4 : 0000000000000000 >[6131611.555369] x3 : ffff8b8c8e97e010 x2 : 0000000000000007 >[6131611.555899] x1 : 0000000000000006 x0 : ffff8b8c8e97e010 >[6131611.556407] Call trace: >[6131611.556689] __ll_sc___cmpxchg_case_mb_8+0x18/0x28 >[6131611.557193] memcg_numa_stat_show+0x1e4/0x260 >[6131611.557643] cgroup_seqfile_show+0x50/0xd0 >[6131611.558072] kernfs_seq_show+0x30/0x38 >[6131611.558561] seq_read+0x148/0x478 >[6131611.558870] kernfs_fop_read+0x2c/0x1e0 >[6131611.559337] __vfs_read+0x20/0x48 >[6131611.559665] vfs_read+0x98/0x168 >[6131611.559957] ksys_read+0x6c/0xe8 >[6131611.560274] __arm64_sys_read+0x20/0x28 >[6131611.560655] el0_svc_common.constprop.0+0xa8/0x200 >[6131611.561099] el0_svc_handler+0x30/0x80 >[6131611.561476] el0_svc+0x10/0x14 >[6131651.531372] watchdog: BUG: soft lockup - CPU#22 stuck for 22s! [kubelet:18289] >[6131651.532182] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6131651.539592] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6131651.545372] CPU: 22 PID: 18289 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6131651.546319] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6131651.547159] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6131651.547605] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6131651.548081] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6131651.548597] sp : ffff00001f81bc20 >[6131651.548905] x29: ffff00001f81bc20 x28: ffff8b8c34c66000 >[6131651.549406] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6131651.549943] x25: 0000000000000003 x24: 0000000000000000 >[6131651.550443] x23: 0000000000000008 x22: ffff59fd7fe49800 >[6131651.550964] x21: ffff8d9ca7cdfc00 x20: ffff59fd7fe4b8a8 >[6131651.551448] x19: 0000000000000001 x18: 0000000000000000 >[6131651.551932] x17: 0000000000000002 x16: 0000000000000000 >[6131651.552531] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6131651.553106] x13: ffff8b9d20559000 x12: ffff8b9d205582fd >[6131651.553625] x11: 0000000000000000 x10: ffff00001f81bc80 >[6131651.554111] x9 : ffff59fd7ef3dad0 x8 : 0000000000014d6b >[6131651.554586] x7 : 0000000000000004 x6 : 0000000000000005 >[6131651.555075] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6131651.555568] x3 : 000000000000000c x2 : 0000339040475000 >[6131651.556052] x1 : ffff4a0f1b5018f8 x0 : 000000000000004c >[6131651.556558] Call trace: >[6131651.556793] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6131651.557267] memcg_numa_stat_show+0x1d0/0x260 >[6131651.557682] cgroup_seqfile_show+0x50/0xd0 >[6131651.558174] kernfs_seq_show+0x30/0x38 >[6131651.558519] seq_read+0x148/0x478 >[6131651.558886] kernfs_fop_read+0x2c/0x1e0 >[6131651.559223] __vfs_read+0x20/0x48 >[6131651.559532] vfs_read+0x98/0x168 >[6131651.559879] ksys_read+0x6c/0xe8 >[6131651.560197] __arm64_sys_read+0x20/0x28 >[6131651.560602] el0_svc_common.constprop.0+0xa8/0x200 >[6131651.561143] el0_svc_handler+0x30/0x80 >[6131651.561540] el0_svc+0x10/0x14 >[6131695.531181] watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [kubelet:18848] >[6131695.532086] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6131695.539152] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6131695.544997] CPU: 10 PID: 18848 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6131695.545927] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6131695.546768] pstate: 20000005 (nzCv daif -PAN -UAO) >[6131695.547235] pc : find_next_bit+0x58/0x68 >[6131695.547618] lr : cpumask_next+0x1c/0x28 >[6131695.547987] sp : ffff000030213c10 >[6131695.548307] x29: ffff000030213c10 x28: ffff8b9c4321e000 >[6131695.548895] x27: ffff59fd7fe4bbf4 x26: ffffffffffffffed >[6131695.549413] x25: 0000000000000003 x24: 0000000000000000 >[6131695.549935] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6131695.550470] x21: ffff8dbc6d2e9c00 x20: ffff59fd7fe4b8a8 >[6131695.551055] x19: 0000000000000000 x18: 0000000000000000 >[6131695.551596] x17: 0000000000000002 x16: 0000000000000000 >[6131695.552191] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6131695.552740] x13: ffff8b8d72d55000 x12: ffff8b8d72d54311 >[6131695.553241] x11: 0000000000000000 x10: ffff000030213c80 >[6131695.553756] x9 : ffff59fd7ef3dad0 x8 : 000000000004f016 >[6131695.554260] x7 : 0000000000000005 x6 : 0000000000000006 >[6131695.554735] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6131695.555224] x3 : 0000000000000004 x2 : 0000000000000044 >[6131695.555728] x1 : 0000000000000400 x0 : ffff59fd7fe4b8a8 >[6131695.556260] Call trace: >[6131695.556516] find_next_bit+0x58/0x68 >[6131695.556889] mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6131695.557386] memcg_numa_stat_show+0x1d0/0x260 >[6131695.557819] cgroup_seqfile_show+0x50/0xd0 >[6131695.558226] kernfs_seq_show+0x30/0x38 >[6131695.558601] seq_read+0x148/0x478 >[6131695.558938] kernfs_fop_read+0x2c/0x1e0 >[6131695.559332] __vfs_read+0x20/0x48 >[6131695.559652] vfs_read+0x98/0x168 >[6131695.559966] ksys_read+0x6c/0xe8 >[6131695.560287] __arm64_sys_read+0x20/0x28 >[6131695.560689] el0_svc_common.constprop.0+0xa8/0x200 >[6131695.561145] el0_svc_handler+0x30/0x80 >[6131695.561500] el0_svc+0x10/0x14 >[6131739.531183] watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [kubelet:18888] >[6131739.532149] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6131739.538598] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6131739.544644] CPU: 10 PID: 18888 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6131739.545731] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6131739.546656] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6131739.547128] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6131739.547636] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6131739.548157] sp : ffff00001fec3c20 >[6131739.548492] x29: ffff00001fec3c20 x28: ffff8bbd71c8c000 >[6131739.548989] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6131739.549498] x25: 0000000000000010 x24: 0000000000000000 >[6131739.550017] x23: 0000000000000020 x22: ffff59fd7fe49800 >[6131739.550504] x21: ffff8b8d854b8c00 x20: ffff59fd7fe4b8a8 >[6131739.551012] x19: 0000000000000004 x18: 0000000000000000 >[6131739.551523] x17: 0000000000000001 x16: 0000000000000000 >[6131739.552121] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6131739.552693] x13: ffff8b8d72d55000 x12: ffff8b8d72d5434c >[6131739.553246] x11: 0000000000000000 x10: ffff00001fec3c80 >[6131739.553792] x9 : ffff59fd7ef3dad0 x8 : ffff8b8d72d54000 >[6131739.554372] x7 : 0000000000000000 x6 : 0000000000000001 >[6131739.554923] x5 : 00000000fffffffb x4 : ffffffffffffffff >[6131739.555468] x3 : 000000000000001e x2 : 000033b0404a7000 >[6131739.556015] x1 : ffff4a2341664e70 x0 : 000000000000005e >[6131739.556555] Call trace: >[6131739.556846] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6131739.557327] memcg_numa_stat_show+0x1d0/0x260 >[6131739.557729] cgroup_seqfile_show+0x50/0xd0 >[6131739.558163] kernfs_seq_show+0x30/0x38 >[6131739.558534] seq_read+0x148/0x478 >[6131739.558909] kernfs_fop_read+0x2c/0x1e0 >[6131739.559311] __vfs_read+0x20/0x48 >[6131739.559620] vfs_read+0x98/0x168 >[6131739.559975] ksys_read+0x6c/0xe8 >[6131739.560286] __arm64_sys_read+0x20/0x28 >[6131739.560684] el0_svc_common.constprop.0+0xa8/0x200 >[6131739.561255] el0_svc_handler+0x30/0x80 >[6131739.561599] el0_svc+0x10/0x14 >[6131779.531244] watchdog: BUG: soft lockup - CPU#14 stuck for 22s! [kubelet:18303] >[6131779.532128] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6131779.538974] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6131779.545085] CPU: 14 PID: 18303 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6131779.546125] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6131779.546989] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6131779.547483] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6131779.547988] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6131779.548478] sp : ffff00001f93bc20 >[6131779.548798] x29: ffff00001f93bc20 x28: ffff8bbb5ba6e000 >[6131779.549270] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6131779.549745] x25: 0000000000000003 x24: 0000000000000000 >[6131779.550270] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6131779.550752] x21: ffff8dabe0369c00 x20: ffff59fd7fe4b8a8 >[6131779.551306] x19: 0000000000000000 x18: 0000000000000000 >[6131779.551889] x17: 0000000000000002 x16: 0000000000000000 >[6131779.552442] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6131779.552939] x13: ffff8b8d722d5000 x12: ffff8b8d722d4307 >[6131779.553437] x11: 0000000000000000 x10: ffff00001f93bc80 >[6131779.553923] x9 : ffff59fd7ef3dad0 x8 : 000000000000bd1e >[6131779.554462] x7 : 0000000000000004 x6 : 0000000000000005 >[6131779.555070] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6131779.555594] x3 : 000000000000001a x2 : 000033b040443000 >[6131779.556109] x1 : ffff4a0ceab6de30 x0 : 000000000000005a >[6131779.556635] Call trace: >[6131779.556877] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6131779.557345] memcg_numa_stat_show+0x1d0/0x260 >[6131779.557730] cgroup_seqfile_show+0x50/0xd0 >[6131779.558108] kernfs_seq_show+0x30/0x38 >[6131779.558455] seq_read+0x148/0x478 >[6131779.558773] kernfs_fop_read+0x2c/0x1e0 >[6131779.559149] __vfs_read+0x20/0x48 >[6131779.559464] vfs_read+0x98/0x168 >[6131779.559780] ksys_read+0x6c/0xe8 >[6131779.560105] __arm64_sys_read+0x20/0x28 >[6131779.560486] el0_svc_common.constprop.0+0xa8/0x200 >[6131779.560946] el0_svc_handler+0x30/0x80 >[6131779.561315] el0_svc+0x10/0x14 >[6131823.531257] watchdog: BUG: soft lockup - CPU#15 stuck for 22s! [kubelet:18280] >[6131823.531733] watchdog: BUG: soft lockup - CPU#46 stuck for 22s! [migration/46:246] >[6131823.532092] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6131823.532812] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6131823.539264] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6131823.545563] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6131823.551528] CPU: 15 PID: 18280 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6131823.557144] CPU: 46 PID: 246 Comm: migration/46 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6131823.558221] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6131823.559388] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6131823.560306] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6131823.561305] pstate: 60000005 (nZCv daif -PAN -UAO) >[6131823.561816] pc : mem_cgroup_nr_lru_pages.isra.7+0x9c/0xc8 >[6131823.562265] pc : multi_cpu_stop+0xf4/0x180 >[6131823.562819] lr : mem_cgroup_nr_lru_pages.isra.7+0xac/0xc8 >[6131823.562821] sp : ffff00001f893c20 >[6131823.563253] lr : multi_cpu_stop+0xb4/0x180 >[6131823.563256] sp : ffff00000a493d50 >[6131823.563831] x29: ffff00001f893c20 x28: 0000000000000000 >[6131823.564191] x29: ffff00000a493d50 x28: 0000000000000000 >[6131823.564583] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6131823.564936] x27: ffff00001f7eb938 x26: 0000000000000060 >[6131823.565429] x25: 0000000000000010 x24: 0000000000000000 >[6131823.565914] x25: 0000000000000000 x24: 0000000000000000 >[6131823.566462] x23: ffff8bacdbc8a5d0 x22: 0000000000000020 >[6131823.567006] x23: 0000000000000000 x22: 0000000000000001 >[6131823.567573] x21: ffff59fd7fe49800 x20: ffff59fd7fe4b8a8 >[6131823.568128] x21: ffff00001f7eb9d4 x20: ffff00001f7eb9b0 >[6131823.568648] x19: 0000000000000004 x18: 0000000000000000 >[6131823.569127] x19: 0000000000000001 x18: 0000000000000000 >[6131823.569656] x17: 0000000000000004 x16: 0000000000000000 >[6131823.570168] x17: 0000000000000001 x16: 0000000000000000 >[6131823.570664] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6131823.571153] x15: 0000000000000000 x14: 0000000000000000 >[6131823.571750] x13: ffff8b8bdf96b000 x12: ffff8b8bdf96a327 >[6131823.572298] x13: 0000000000000000 x12: 0000000000000000 >[6131823.572885] x11: 0000000000000000 x10: ffff00001f893c80 >[6131823.573398] x11: 0000000000000013 x10: 0000000000000d60 >[6131823.574000] x9 : ffff59fd7ef3da4c x8 : 000000000000d9d9 >[6131823.574570] x9 : ffff59fd7ede5a0c x8 : ffff8bcd9e996400 >[6131823.575080] x7 : 0000000000000004 x6 : 0000000000000005 >[6131823.575704] x7 : 000a00b582876a50 x6 : 00000000000003dd >[6131823.576243] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6131823.576745] x5 : 00000000fa83b2da x4 : ffff8bcd9e995c40 >[6131823.577273] x3 : 000000000000002c x2 : 000033d020079000 >[6131823.577736] x3 : dead000000000200 x2 : ffff00001f7eb988 >[6131823.578247] x1 : ffff4a28e2d82610 x0 : 000000000000006c >[6131823.578835] x1 : ffff00001f7eb9d4 x0 : 0000000000000001 >[6131823.579327] Call trace: >[6131823.579857] Call trace: >[6131823.580416] mem_cgroup_nr_lru_pages.isra.7+0x9c/0xc8 >[6131823.580997] multi_cpu_stop+0xf4/0x180 >[6131823.581336] memcg_numa_stat_show+0x14c/0x260 >[6131823.581604] cpu_stopper_thread+0xec/0x178 >[6131823.582089] cgroup_seqfile_show+0x50/0xd0 >[6131823.582474] smpboot_thread_fn+0x158/0x1d8 >[6131823.582883] kernfs_seq_show+0x30/0x38 >[6131823.583261] kthread+0x130/0x138 >[6131823.583688] seq_read+0x148/0x478 >[6131823.585140] kernfs_fop_read+0x2c/0x1e0 >[6131823.585531] __vfs_read+0x20/0x48 >[6131823.585847] vfs_read+0x98/0x168 >[6131823.586205] ksys_read+0x6c/0xe8 >[6131823.586506] __arm64_sys_read+0x20/0x28 >[6131823.586891] el0_svc_common.constprop.0+0xa8/0x200 >[6131823.587412] el0_svc_handler+0x30/0x80 >[6131823.587779] el0_svc+0x10/0x14 >[6131863.531010] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kubelet:12233] >[6131863.531873] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6131863.538490] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6131863.544335] CPU: 0 PID: 12233 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6131863.545406] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6131863.546338] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6131863.546829] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6131863.547418] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6131863.547970] sp : ffff000056233c20 >[6131863.548295] x29: ffff000056233c20 x28: ffff8b8c3bb6a000 >[6131863.548833] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6131863.549333] x25: 0000000000000003 x24: 0000000000000000 >[6131863.549837] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6131863.550346] x21: ffff8d8cfdec3000 x20: ffff59fd7fe4b8a8 >[6131863.550843] x19: 0000000000000000 x18: 0000000000000000 >[6131863.551465] x17: 0000000000000002 x16: 0000000000000000 >[6131863.551962] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6131863.552525] x13: ffff8b8d71fca000 x12: ffff8b8d71fc92f3 >[6131863.553082] x11: 0000000000000000 x10: ffff000056233c80 >[6131863.553618] x9 : ffff59fd7ef3dad0 x8 : 0000000000037696 >[6131863.554182] x7 : 0000000000000005 x6 : 0000000000000006 >[6131863.554741] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6131863.555256] x3 : 0000000000000000 x2 : 000031804040f000 >[6131863.555794] x1 : ffff4a0f212c1b80 x0 : 0000000000000000 >[6131863.556294] Call trace: >[6131863.556585] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6131863.557066] memcg_numa_stat_show+0x1d0/0x260 >[6131863.557497] cgroup_seqfile_show+0x50/0xd0 >[6131863.557902] kernfs_seq_show+0x30/0x38 >[6131863.558264] seq_read+0x148/0x478 >[6131863.558594] kernfs_fop_read+0x2c/0x1e0 >[6131863.558965] __vfs_read+0x20/0x48 >[6131863.559282] vfs_read+0x98/0x168 >[6131863.559608] ksys_read+0x6c/0xe8 >[6131863.559949] __arm64_sys_read+0x20/0x28 >[6131863.560411] el0_svc_common.constprop.0+0xa8/0x200 >[6131863.561009] el0_svc_handler+0x30/0x80 >[6131863.561417] el0_svc+0x10/0x14 >[6131867.531177] watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [migration/10:62] >[6131867.532042] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6131867.538548] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6131867.544394] CPU: 10 PID: 62 Comm: migration/10 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6131867.545357] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6131867.546215] pstate: 60000005 (nZCv daif -PAN -UAO) >[6131867.546675] pc : multi_cpu_stop+0x78/0x180 >[6131867.547067] lr : multi_cpu_stop+0xb4/0x180 >[6131867.547466] sp : ffff000008ecbd50 >[6131867.547811] x29: ffff000008ecbd50 x28: 0000000000000000 >[6131867.548338] x27: ffff00005b663938 x26: 0000000000000060 >[6131867.548840] x25: 0000000000000000 x24: 0000000000000000 >[6131867.549353] x23: 0000000000000000 x22: 0000000000000001 >[6131867.549866] x21: ffff00005b6639d4 x20: ffff00005b6639b0 >[6131867.550382] x19: 0000000000000001 x18: 0000000000000000 >[6131867.550979] x17: 0000000000000001 x16: 0000000000000000 >[6131867.551498] x15: 0000000000000000 x14: 0000000000000000 >[6131867.552072] x13: 0000000000000000 x12: 0000000000000000 >[6131867.552684] x11: 0000000000000000 x10: 0000000000000d60 >[6131867.553227] x9 : ffff59fd7ede5a0c x8 : ffff8b8d9e907540 >[6131867.553764] x7 : 00208d76e41e5864 x6 : 00000000000001c1 >[6131867.554300] x5 : 00000000efe4b99a x4 : ffff8b8d9e906d80 >[6131867.554851] x3 : dead000000000200 x2 : ffff00005b663988 >[6131867.555445] x1 : ffff00005b6639d4 x0 : 0000000000000001 >[6131867.555967] Call trace: >[6131867.556269] multi_cpu_stop+0x78/0x180 >[6131867.556652] cpu_stopper_thread+0xec/0x178 >[6131867.557134] smpboot_thread_fn+0x158/0x1d8 >[6131867.557530] kthread+0x130/0x138 >[6131907.531279] watchdog: BUG: soft lockup - CPU#16 stuck for 22s! [kubelet:18888] >[6131907.532110] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6131907.538463] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6131907.544131] CPU: 16 PID: 18888 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6131907.545198] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6131907.546054] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6131907.546555] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6131907.547054] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6131907.547626] sp : ffff00001fec3c20 >[6131907.547975] x29: ffff00001fec3c20 x28: ffff8b7be0b84000 >[6131907.548512] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6131907.549049] x25: 0000000000000003 x24: 0000000000000000 >[6131907.549602] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6131907.550111] x21: ffff8d9cc7f3fc00 x20: ffff59fd7fe4b8a8 >[6131907.550712] x19: 0000000000000000 x18: 0000000000000000 >[6131907.551257] x17: 0000000000000009 x16: 0000000000000000 >[6131907.551737] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6131907.552246] x13: ffff8b9ba2452000 x12: ffff8b9ba24512fd >[6131907.552746] x11: 0000000000000000 x10: ffff00001fec3c80 >[6131907.553217] x9 : ffff59fd7ef3dad0 x8 : 0000000000014e30 >[6131907.553694] x7 : 0000000000000004 x6 : 0000000000000005 >[6131907.554178] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6131907.554671] x3 : 0000000000000035 x2 : 000033d0402e8000 >[6131907.555194] x1 : ffff4a1ce851de00 x0 : 0000000000000035 >[6131907.555674] Call trace: >[6131907.555903] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6131907.556344] memcg_numa_stat_show+0x1d0/0x260 >[6131907.556739] cgroup_seqfile_show+0x50/0xd0 >[6131907.557177] kernfs_seq_show+0x30/0x38 >[6131907.557520] seq_read+0x148/0x478 >[6131907.557872] kernfs_fop_read+0x2c/0x1e0 >[6131907.558272] __vfs_read+0x20/0x48 >[6131907.558603] vfs_read+0x98/0x168 >[6131907.558905] ksys_read+0x6c/0xe8 >[6131907.559205] __arm64_sys_read+0x20/0x28 >[6131907.559576] el0_svc_common.constprop.0+0xa8/0x200 >[6131907.560019] el0_svc_handler+0x30/0x80 >[6131907.560370] el0_svc+0x10/0x14 >[6131951.531387] watchdog: BUG: soft lockup - CPU#23 stuck for 22s! [kubelet:12233] >[6131951.532224] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6131951.538578] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6131951.544296] CPU: 23 PID: 12233 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6131951.545243] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6131951.546088] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6131951.546534] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6131951.547014] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6131951.547526] sp : ffff000056233c20 >[6131951.547829] x29: ffff000056233c20 x28: ffff8bcc56052000 >[6131951.548341] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6131951.548829] x25: 0000000000000003 x24: 0000000000000000 >[6131951.549332] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6131951.549834] x21: ffff8d9c9c2e0800 x20: ffff59fd7fe4b8a8 >[6131951.550423] x19: 0000000000000000 x18: 0000000000000000 >[6131951.550905] x17: 0000000000000002 x16: 0000000000000000 >[6131951.551394] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6131951.551880] x13: ffff8b9d220b3000 x12: ffff8b9d220b22fd >[6131951.552367] x11: 0000000000000000 x10: ffff000056233c80 >[6131951.552870] x9 : ffff59fd7ef3dad0 x8 : 0000000000014bdb >[6131951.553390] x7 : 0000000000000004 x6 : 0000000000000005 >[6131951.553927] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6131951.554406] x3 : 0000000000000025 x2 : 000031c04048d000 >[6131951.554888] x1 : ffff4a0f0edc03a0 x0 : 0000000000000025 >[6131951.555359] Call trace: >[6131951.555603] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6131951.556043] memcg_numa_stat_show+0x1d0/0x260 >[6131951.556499] cgroup_seqfile_show+0x50/0xd0 >[6131951.556896] kernfs_seq_show+0x30/0x38 >[6131951.557262] seq_read+0x148/0x478 >[6131951.557584] kernfs_fop_read+0x2c/0x1e0 >[6131951.557948] __vfs_read+0x20/0x48 >[6131951.558280] vfs_read+0x98/0x168 >[6131951.558579] ksys_read+0x6c/0xe8 >[6131951.558879] __arm64_sys_read+0x20/0x28 >[6131951.559223] el0_svc_common.constprop.0+0xa8/0x200 >[6131951.559703] el0_svc_handler+0x30/0x80 >[6131951.560043] el0_svc+0x10/0x14 >[6131995.531199] watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [kubelet:18303] >[6131995.532052] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6131995.538635] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6131995.544548] CPU: 11 PID: 18303 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6131995.545529] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6131995.546385] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6131995.546891] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6131995.547445] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6131995.547951] sp : ffff00001f93bc20 >[6131995.548255] x29: ffff00001f93bc20 x28: ffff8bcc4d928000 >[6131995.548767] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6131995.549300] x25: 0000000000000003 x24: 0000000000000000 >[6131995.549797] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6131995.550447] x21: ffff8dccf5862800 x20: ffff59fd7fe4b8a8 >[6131995.550933] x19: 0000000000000000 x18: 0000000000000000 >[6131995.551429] x17: 0000000000000002 x16: 0000000000000000 >[6131995.551999] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6131995.552548] x13: ffff8b8d8045a000 x12: ffff8b8d8045931c >[6131995.553076] x11: 0000000000000000 x10: ffff00001f93bc80 >[6131995.553616] x9 : ffff59fd7ef3dad0 x8 : 0000000000019411 >[6131995.554142] x7 : 0000000000000005 x6 : 0000000000000006 >[6131995.554677] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6131995.555193] x3 : 000000000000003b x2 : 000033d01fed0000 >[6131995.555752] x1 : ffff4a2dde5540c0 x0 : 000000000000007b >[6131995.556243] Call trace: >[6131995.556517] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6131995.557001] memcg_numa_stat_show+0x1d0/0x260 >[6131995.557459] cgroup_seqfile_show+0x50/0xd0 >[6131995.557915] kernfs_seq_show+0x30/0x38 >[6131995.558285] seq_read+0x148/0x478 >[6131995.558618] kernfs_fop_read+0x2c/0x1e0 >[6131995.559070] __vfs_read+0x20/0x48 >[6131995.559437] vfs_read+0x98/0x168 >[6131995.559749] ksys_read+0x6c/0xe8 >[6131995.560190] __arm64_sys_read+0x20/0x28 >[6131995.560589] el0_svc_common.constprop.0+0xa8/0x200 >[6131995.561085] el0_svc_handler+0x30/0x80 >[6131995.561449] el0_svc+0x10/0x14 >[6132039.531258] watchdog: BUG: soft lockup - CPU#15 stuck for 22s! [kubelet:18889] >[6132039.532056] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6132039.539480] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6132039.545893] CPU: 15 PID: 18889 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6132039.546838] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6132039.547675] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6132039.548131] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6132039.548649] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6132039.549158] sp : ffff0000300f3c20 >[6132039.549494] x29: ffff0000300f3c20 x28: ffff8bad73f72000 >[6132039.550104] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6132039.550669] x25: 0000000000000010 x24: 0000000000000000 >[6132039.551178] x23: 0000000000000020 x22: ffff59fd7fe49800 >[6132039.551660] x21: ffff8b8ba2a31400 x20: ffff59fd7fe4b8a8 >[6132039.552160] x19: 0000000000000004 x18: 0000000000000000 >[6132039.552667] x17: 0000000000000002 x16: 0000000000000000 >[6132039.553184] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6132039.553699] x13: ffff8b8bdf970000 x12: ffff8b8bdf96f34c >[6132039.554194] x11: 0000000000000000 x10: ffff0000300f3c80 >[6132039.554712] x9 : ffff59fd7ef3dad0 x8 : ffff8b8bdf96f000 >[6132039.555231] x7 : 0000000000000000 x6 : 0000000000000001 >[6132039.555756] x5 : 00000000fffffffb x4 : ffffffffffffffff >[6132039.556267] x3 : 000000000000000f x2 : 00003190404c0000 >[6132039.556757] x1 : ffff4a2b45b957d0 x0 : 000000000000000f >[6132039.557256] Call trace: >[6132039.557511] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6132039.557996] memcg_numa_stat_show+0x1d0/0x260 >[6132039.558423] cgroup_seqfile_show+0x50/0xd0 >[6132039.558827] kernfs_seq_show+0x30/0x38 >[6132039.559193] seq_read+0x148/0x478 >[6132039.559527] kernfs_fop_read+0x2c/0x1e0 >[6132039.560024] __vfs_read+0x20/0x48 >[6132039.560375] vfs_read+0x98/0x168 >[6132039.560725] ksys_read+0x6c/0xe8 >[6132039.561074] __arm64_sys_read+0x20/0x28 >[6132039.561450] el0_svc_common.constprop.0+0xa8/0x200 >[6132039.561931] el0_svc_handler+0x30/0x80 >[6132039.562315] el0_svc+0x10/0x14 >[6132079.531104] watchdog: BUG: soft lockup - CPU#5 stuck for 23s! [kubelet:37248] >[6132079.531872] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6132079.538374] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6132079.544441] CPU: 5 PID: 37248 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6132079.545433] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6132079.546312] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6132079.546785] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6132079.547303] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6132079.547804] sp : ffff00001b93bc20 >[6132079.548129] x29: ffff00001b93bc20 x28: ffff8b7c1add8000 >[6132079.548674] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6132079.549192] x25: 0000000000000003 x24: 0000000000000000 >[6132079.549851] x23: 0000000000000008 x22: ffff59fd7fe49800 >[6132079.550418] x21: ffff8daca59bb400 x20: ffff59fd7fe4b8a8 >[6132079.550929] x19: 0000000000000001 x18: 0000000000000000 >[6132079.551468] x17: 0000000000000002 x16: 0000000000000000 >[6132079.552035] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6132079.552634] x13: ffff8b7b9f093000 x12: ffff8b7b9f092307 >[6132079.553149] x11: 0000000000000000 x10: ffff00001b93bc80 >[6132079.553693] x9 : ffff59fd7ef3dad0 x8 : 000000000000bf76 >[6132079.554198] x7 : 0000000000000004 x6 : 0000000000000005 >[6132079.554691] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6132079.555193] x3 : 000000000000003d x2 : 000033d040220000 >[6132079.555738] x1 : ffff4a1f174cfcd8 x0 : 000000000000003d >[6132079.556282] Call trace: >[6132079.556533] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6132079.557012] memcg_numa_stat_show+0x1d0/0x260 >[6132079.557446] cgroup_seqfile_show+0x50/0xd0 >[6132079.557862] kernfs_seq_show+0x30/0x38 >[6132079.558257] seq_read+0x148/0x478 >[6132079.558594] kernfs_fop_read+0x2c/0x1e0 >[6132079.558961] __vfs_read+0x20/0x48 >[6132079.559284] vfs_read+0x98/0x168 >[6132079.559748] ksys_read+0x6c/0xe8 >[6132079.560072] __arm64_sys_read+0x20/0x28 >[6132079.560479] el0_svc_common.constprop.0+0xa8/0x200 >[6132079.560955] el0_svc_handler+0x30/0x80 >[6132079.561364] el0_svc+0x10/0x14 >[6132083.531209] watchdog: BUG: soft lockup - CPU#12 stuck for 22s! [migration/12:72] >[6132083.532055] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6132083.538375] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6132083.544169] CPU: 12 PID: 72 Comm: migration/12 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6132083.545179] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6132083.546078] pstate: 60000005 (nZCv daif -PAN -UAO) >[6132083.546531] pc : multi_cpu_stop+0xf4/0x180 >[6132083.546970] lr : multi_cpu_stop+0xb4/0x180 >[6132083.547412] sp : ffff000008f1bd50 >[6132083.547738] x29: ffff000008f1bd50 x28: 0000000000000000 >[6132083.548253] x27: ffff000050993938 x26: 0000000000000060 >[6132083.548730] x25: 0000000000000000 x24: 0000000000000000 >[6132083.549239] x23: 0000000000000000 x22: 0000000000000001 >[6132083.549863] x21: ffff0000509939d4 x20: ffff0000509939b0 >[6132083.550359] x19: 0000000000000001 x18: 0000000000000000 >[6132083.550939] x17: 0000000000000001 x16: 0000000000000000 >[6132083.551436] x15: 0000000000000001 x14: ffff8dbd4af33400 >[6132083.551961] x13: 000000000000652d x12: ffff8dbd6f405d80 >[6132083.552471] x11: 0000000000000035 x10: 0000000000000d60 >[6132083.553086] x9 : ffff59fd7ede5a0c x8 : ffff8b8d9e953040 >[6132083.553625] x7 : ffff8b8d7c5a2c00 x6 : 000000000000029c >[6132083.554212] x5 : 0000319040475000 x4 : ffff8b8d9e952880 >[6132083.554752] x3 : dead000000000200 x2 : ffff000050993988 >[6132083.555277] x1 : ffff0000509939d4 x0 : 0000000000000001 >[6132083.555788] Call trace: >[6132083.556033] multi_cpu_stop+0xf4/0x180 >[6132083.556407] cpu_stopper_thread+0xec/0x178 >[6132083.556802] smpboot_thread_fn+0x158/0x1d8 >[6132083.557217] kthread+0x130/0x138 >[6132123.531007] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kubelet:3381] >[6132123.531810] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6132123.538321] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6132123.544139] CPU: 0 PID: 3381 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6132123.545133] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6132123.546034] pstate: a0000005 (NzCv daif -PAN -UAO) >[6132123.546493] pc : mem_cgroup_node_nr_lru_pages+0x4c/0x128 >[6132123.547009] lr : memcg_numa_stat_show+0x1d0/0x260 >[6132123.547463] sp : ffff00001d773c20 >[6132123.547788] x29: ffff00001d773c20 x28: ffff8b9c44f32000 >[6132123.548289] x27: ffff59fd7f671bf8 x26: ffff59fd7f8b5a88 >[6132123.548794] x25: 0000000000000010 x24: ffff59fd7fe4c9a0 >[6132123.549396] x23: ffff8b7d1ac0ce80 x22: ffff59fd7f671e38 >[6132123.549954] x21: ffff8dbd9e88a000 x20: 000000000000000d >[6132123.550451] x19: 0000000000004822 x18: 0000000000000000 >[6132123.550940] x17: 0000000000000002 x16: 0000000000000000 >[6132123.551428] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6132123.551948] x13: ffff8b8d71fc9000 x12: ffff8b8d71fc831c >[6132123.552490] x11: 0000000000000000 x10: ffff00001d773c80 >[6132123.553070] x9 : ffff59fd7ef3dad0 x8 : 0000000000018c2f >[6132123.553572] x7 : 0000000000000005 x6 : 0000000000000006 >[6132123.554080] x5 : 00000000fffffff6 x4 : ffff8dbd9e88a000 >[6132123.554578] x3 : 00000000000002a5 x2 : 0000000000000003 >[6132123.555080] x1 : ffff8dcdbfe60a80 x0 : ffff8b9c44f32000 >[6132123.555583] Call trace: >[6132123.555827] mem_cgroup_node_nr_lru_pages+0x4c/0x128 >[6132123.556276] memcg_numa_stat_show+0x1d0/0x260 >[6132123.556709] cgroup_seqfile_show+0x50/0xd0 >[6132123.557104] kernfs_seq_show+0x30/0x38 >[6132123.557468] seq_read+0x148/0x478 >[6132123.557787] kernfs_fop_read+0x2c/0x1e0 >[6132123.558223] __vfs_read+0x20/0x48 >[6132123.558573] vfs_read+0x98/0x168 >[6132123.558892] ksys_read+0x6c/0xe8 >[6132123.559389] __arm64_sys_read+0x20/0x28 >[6132123.559781] el0_svc_common.constprop.0+0xa8/0x200 >[6132123.560232] el0_svc_handler+0x30/0x80 >[6132123.560647] el0_svc+0x10/0x14 >[6132167.531082] watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [migration/4:31] >[6132167.531242] watchdog: BUG: soft lockup - CPU#14 stuck for 23s! [kubelet:18923] >[6132167.531903] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6132167.532635] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6132167.539245] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6132167.545966] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6132167.551836] CPU: 4 PID: 31 Comm: migration/4 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6132167.557440] CPU: 14 PID: 18923 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6132167.558441] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6132167.559519] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6132167.560452] pstate: 60000005 (nZCv daif -PAN -UAO) >[6132167.561419] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6132167.561936] pc : multi_cpu_stop+0xf4/0x180 >[6132167.562448] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6132167.562878] lr : multi_cpu_stop+0xb4/0x180 >[6132167.563448] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6132167.563450] sp : ffff0000301bbc20 >[6132167.563884] sp : ffff000008dd3d50 >[6132167.564380] x29: ffff0000301bbc20 x28: ffff8bac4102c000 >[6132167.564740] x29: ffff000008dd3d50 x28: 0000000000000000 >[6132167.565091] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6132167.565670] x27: ffff0000545db938 x26: 0000000000000060 >[6132167.566212] x25: 0000000000000010 x24: 0000000000000000 >[6132167.566797] x25: 0000000000000000 x24: 0000000000000000 >[6132167.567387] x23: 0000000000000020 x22: ffff59fd7fe49800 >[6132167.567918] x23: 0000000000000000 x22: 0000000000000001 >[6132167.568436] x21: ffff8b8c3622d400 x20: ffff59fd7fe4b8a8 >[6132167.569141] x21: ffff0000545db9d4 x20: ffff0000545db9b0 >[6132167.569671] x19: 0000000000000004 x18: 0000000000000000 >[6132167.570251] x19: 0000000000000001 x18: 0000000000000000 >[6132167.570759] x17: 0000000000000002 x16: 0000000000000000 >[6132167.571321] x17: 0000000000000001 x16: 0000000000000000 >[6132167.571871] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6132167.572405] x15: 0000000000000000 x14: ffff8dbd58a41800 >[6132167.572982] x13: ffff8b8d722d3000 x12: ffff8b8d722d234c >[6132167.573539] x13: 00000000000115a6 x12: ffff8dbd61e28980 >[6132167.574023] x11: 0000000000000000 x10: ffff0000301bbc80 >[6132167.574539] x11: 0000000000000124 x10: 0000000000000d60 >[6132167.575038] x9 : ffff59fd7ef3dad0 x8 : ffff8b8d722d2000 >[6132167.575548] x9 : ffff59fd7ede5a0c x8 : ffff8b7d9da1e400 >[6132167.576151] x7 : 0000000000000000 x6 : 0000000000000001 >[6132167.576690] x7 : ffff8b7d1ccd3000 x6 : 00000000000001e9 >[6132167.577193] x5 : 00000000fffffffb x4 : ffffffffffffffff >[6132167.577714] x5 : 0000318040473000 x4 : ffff8b7d9da1dc40 >[6132167.578219] x3 : 000000000000000c x2 : 0000319040475000 >[6132167.578811] x3 : dead000000000200 x2 : ffff0000545db960 >[6132167.579395] x1 : ffff4a15618ac880 x0 : 000000000000000c >[6132167.580007] x1 : ffff0000545db9d4 x0 : 0000000000000001 >[6132167.580508] Call trace: >[6132167.581002] Call trace: >[6132167.581467] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6132167.581708] multi_cpu_stop+0xf4/0x180 >[6132167.581932] memcg_numa_stat_show+0x1d0/0x260 >[6132167.582415] cpu_stopper_thread+0xec/0x178 >[6132167.582775] cgroup_seqfile_show+0x50/0xd0 >[6132167.583203] smpboot_thread_fn+0x158/0x1d8 >[6132167.583594] kernfs_seq_show+0x30/0x38 >[6132167.583977] kthread+0x130/0x138 >[6132167.584374] seq_read+0x148/0x478 >[6132167.585435] kernfs_fop_read+0x2c/0x1e0 >[6132167.585822] __vfs_read+0x20/0x48 >[6132167.586157] vfs_read+0x98/0x168 >[6132167.586449] ksys_read+0x6c/0xe8 >[6132167.586770] __arm64_sys_read+0x20/0x28 >[6132167.587137] el0_svc_common.constprop.0+0xa8/0x200 >[6132167.587617] el0_svc_handler+0x30/0x80 >[6132167.587977] el0_svc+0x10/0x14 >[6132207.531879] watchdog: BUG: soft lockup - CPU#53 stuck for 23s! [kubelet:18849] >[6132207.532675] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6132207.539223] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6132207.544818] CPU: 53 PID: 18849 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6132207.545738] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6132207.546551] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6132207.547000] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6132207.547497] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6132207.547977] sp : ffff000030123c20 >[6132207.548291] x29: ffff000030123c20 x28: ffff8dac08b90000 >[6132207.548932] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6132207.549413] x25: 0000000000000003 x24: 0000000000000000 >[6132207.549882] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6132207.550362] x21: ffff8d8cf28bd800 x20: ffff59fd7fe4b8a8 >[6132207.550852] x19: 0000000000000000 x18: 0000000000000000 >[6132207.551332] x17: 000000000000064e x16: 0000000000000000 >[6132207.551831] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6132207.552311] x13: ffff8b9d1d259000 x12: ffff8b9d1d2582f3 >[6132207.552801] x11: 0000000000000000 x10: ffff000030123c80 >[6132207.553293] x9 : ffff59fd7ef3dad0 x8 : 0000000000037710 >[6132207.553762] x7 : 0000000000000005 x6 : 0000000000000006 >[6132207.554242] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6132207.554732] x3 : 0000000000000001 x2 : 0000338040428000 >[6132207.555208] x1 : ffff4a1c03b6c850 x0 : 0000000000000041 >[6132207.555738] Call trace: >[6132207.556000] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6132207.556468] memcg_numa_stat_show+0x1d0/0x260 >[6132207.556879] cgroup_seqfile_show+0x50/0xd0 >[6132207.557339] kernfs_seq_show+0x30/0x38 >[6132207.557689] seq_read+0x148/0x478 >[6132207.558052] kernfs_fop_read+0x2c/0x1e0 >[6132207.558490] __vfs_read+0x20/0x48 >[6132207.558807] vfs_read+0x98/0x168 >[6132207.559116] ksys_read+0x6c/0xe8 >[6132207.559410] __arm64_sys_read+0x20/0x28 >[6132207.559789] el0_svc_common.constprop.0+0xa8/0x200 >[6132207.560241] el0_svc_handler+0x30/0x80 >[6132207.560603] el0_svc+0x10/0x14 >[6132251.531078] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [kubelet:61154] >[6132251.531942] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6132251.538634] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6132251.544339] CPU: 3 PID: 61154 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6132251.545360] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6132251.546210] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6132251.546749] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6132251.547275] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6132251.547790] sp : ffff000059c5bc20 >[6132251.548210] x29: ffff000059c5bc20 x28: ffff8b9c0a2ee000 >[6132251.548748] x27: ffff59fd7fe4bbf4 x26: fffffffffffffc8e >[6132251.549259] x25: 0000000000000003 x24: 0000000000000000 >[6132251.549774] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6132251.550283] x21: ffff8d9c376ffc00 x20: ffff59fd7fe4b8a8 >[6132251.550792] x19: 0000000000000000 x18: 0000000000000000 >[6132251.551287] x17: 000000000000063f x16: 0000000000000000 >[6132251.551793] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6132251.552317] x13: ffff8b7d1cf97000 x12: ffff8b7d1cf962fd >[6132251.552835] x11: 0000000000000000 x10: ffff000059c5bc80 >[6132251.553349] x9 : ffff59fd7ef3dad0 x8 : 0000000000014bd5 >[6132251.553853] x7 : 0000000000000004 x6 : 0000000000000005 >[6132251.554354] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6132251.554867] x3 : 000000000000000b x2 : 000033904045c000 >[6132251.555383] x1 : ffff4a0ce8601a60 x0 : 000000000000004b >[6132251.555905] Call trace: >[6132251.556152] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6132251.556648] memcg_numa_stat_show+0x1d0/0x260 >[6132251.557088] cgroup_seqfile_show+0x50/0xd0 >[6132251.557489] kernfs_seq_show+0x30/0x38 >[6132251.557866] seq_read+0x148/0x478 >[6132251.558285] kernfs_fop_read+0x2c/0x1e0 >[6132251.558674] __vfs_read+0x20/0x48 >[6132251.559075] vfs_read+0x98/0x168 >[6132251.559428] ksys_read+0x6c/0xe8 >[6132251.559753] __arm64_sys_read+0x20/0x28 >[6132251.560123] el0_svc_common.constprop.0+0xa8/0x200 >[6132251.560580] el0_svc_handler+0x30/0x80 >[6132251.560941] el0_svc+0x10/0x14 >[6132295.531115] watchdog: BUG: soft lockup - CPU#6 stuck for 23s! [migration/6:41] >[6132295.531150] watchdog: BUG: soft lockup - CPU#8 stuck for 23s! [kubelet:19592] >[6132295.531913] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6132295.532599] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6132295.539124] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6132295.545656] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6132295.551615] CPU: 6 PID: 41 Comm: migration/6 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6132295.557426] CPU: 8 PID: 19592 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6132295.558506] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6132295.559483] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6132295.560361] pstate: 60000005 (nZCv daif -PAN -UAO) >[6132295.561228] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6132295.561685] pc : multi_cpu_stop+0xf4/0x180 >[6132295.562141] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6132295.562145] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6132295.562578] lr : multi_cpu_stop+0xb4/0x180 >[6132295.563084] sp : ffff000030fcbc20 >[6132295.563698] sp : ffff000008e23d50 >[6132295.564081] x29: ffff000030fcbc20 x28: ffff8bace458e000 >[6132295.564452] x29: ffff000008e23d50 x28: 0000000000000000 >[6132295.564775] x27: ffff59fd7fe4bbf4 x26: fffffffffffffdc2 >[6132295.565272] x27: ffff00001fe63938 x26: 0000000000000060 >[6132295.565766] x25: 0000000000000003 x24: 0000000000000000 >[6132295.566264] x25: 0000000000000000 x24: 0000000000000000 >[6132295.566769] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6132295.567267] x23: 0000000000000000 x22: 0000000000000001 >[6132295.567762] x21: ffff8dcd4bdf9c00 x20: ffff59fd7fe4b8a8 >[6132295.568399] x21: ffff00001fe639d4 x20: ffff00001fe639b0 >[6132295.569003] x19: 0000000000000000 x18: 0000000000000000 >[6132295.569514] x19: 0000000000000001 x18: 0000000000000000 >[6132295.570030] x17: 0000000000000008 x16: 0000000000000000 >[6132295.570507] x17: 0000000000000001 x16: 0000000000000000 >[6132295.571058] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6132295.571644] x15: 0000000000000000 x14: 0000000000000000 >[6132295.572142] x13: ffff8b8d72fae000 x12: ffff8b8d72fad31c >[6132295.572708] x13: 0000000000000000 x12: 0000000000000000 >[6132295.573309] x11: 0000000000000000 x10: ffff000030fcbc80 >[6132295.573814] x11: 00000000000000c8 x10: 0000000000000d60 >[6132295.574427] x9 : ffff59fd7ef3dad0 x8 : 0000000000018ac6 >[6132295.574923] x9 : ffff59fd7ede5a0c x8 : ffff8b7d9da79f00 >[6132295.575444] x7 : 0000000000000005 x6 : 0000000000000006 >[6132295.576003] x7 : 000db28e66faee16 x6 : 00000000000003ca >[6132295.576589] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6132295.577105] x5 : 00000000fa83b2da x4 : ffff8b7d9da79740 >[6132295.577633] x3 : 0000000000000036 x2 : 000033d040301000 >[6132295.578274] x3 : dead000000000200 x2 : ffff00001fe63960 >[6132295.578809] x1 : ffff4a1f1cddac20 x0 : 0000000000000036 >[6132295.579310] x1 : ffff00001fe639d4 x0 : 0000000000000001 >[6132295.579832] Call trace: >[6132295.580377] Call trace: >[6132295.580893] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6132295.581169] multi_cpu_stop+0xf4/0x180 >[6132295.581418] memcg_numa_stat_show+0x1d0/0x260 >[6132295.581878] cpu_stopper_thread+0xec/0x178 >[6132295.582228] cgroup_seqfile_show+0x50/0xd0 >[6132295.582641] smpboot_thread_fn+0x158/0x1d8 >[6132295.583032] kernfs_seq_show+0x30/0x38 >[6132295.583409] kthread+0x130/0x138 >[6132295.583799] seq_read+0x148/0x478 >[6132295.584764] kernfs_fop_read+0x2c/0x1e0 >[6132295.585129] __vfs_read+0x20/0x48 >[6132295.585451] vfs_read+0x98/0x168 >[6132295.585767] ksys_read+0x6c/0xe8 >[6132295.586076] __arm64_sys_read+0x20/0x28 >[6132295.586445] el0_svc_common.constprop.0+0xa8/0x200 >[6132295.586898] el0_svc_handler+0x30/0x80 >[6132295.587313] el0_svc+0x10/0x14 >[6132339.531017] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kubelet:24254] >[6132339.531927] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6132339.538950] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6132339.544875] CPU: 1 PID: 24254 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6132339.545858] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6132339.546744] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6132339.547288] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6132339.547893] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6132339.548470] sp : ffff00001f7ebc20 >[6132339.548802] x29: ffff00001f7ebc20 x28: ffff8dccec760000 >[6132339.549352] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6132339.549852] x25: 0000000000000010 x24: 0000000000000000 >[6132339.550427] x23: 0000000000000020 x22: ffff59fd7fe49800 >[6132339.550966] x21: ffff8b7b6c4e0000 x20: ffff59fd7fe4b8a8 >[6132339.551473] x19: 0000000000000004 x18: 0000000000000000 >[6132339.551957] x17: 0000000000000002 x16: 0000000000000000 >[6132339.552483] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6132339.553061] x13: ffff8b7d1ada1000 x12: ffff8b7d1ada0345 >[6132339.553598] x11: 0000000000000000 x10: ffff00001f7ebc80 >[6132339.554111] x9 : ffff59fd7ef3dad0 x8 : 0000000000000021 >[6132339.554655] x7 : 0000000000000001 x6 : 0000000000000002 >[6132339.555161] x5 : 00000000fffffffa x4 : ffffffffffffffff >[6132339.555676] x3 : 000000000000002c x2 : 000031d040475000 >[6132339.556229] x1 : ffff4a1dc8da7d00 x0 : 000000000000002c >[6132339.556797] Call trace: >[6132339.557044] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6132339.557679] memcg_numa_stat_show+0x1d0/0x260 >[6132339.558147] cgroup_seqfile_show+0x50/0xd0 >[6132339.558542] kernfs_seq_show+0x30/0x38 >[6132339.558966] seq_read+0x148/0x478 >[6132339.559291] kernfs_fop_read+0x2c/0x1e0 >[6132339.559662] __vfs_read+0x20/0x48 >[6132339.560061] vfs_read+0x98/0x168 >[6132339.560378] ksys_read+0x6c/0xe8 >[6132339.560730] __arm64_sys_read+0x20/0x28 >[6132339.561091] el0_svc_common.constprop.0+0xa8/0x200 >[6132339.561577] el0_svc_handler+0x30/0x80 >[6132339.561992] el0_svc+0x10/0x14 >[6132379.531183] watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [kubelet:18850] >[6132379.532113] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6132379.539436] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6132379.545133] CPU: 10 PID: 18850 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6132379.546091] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6132379.546991] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6132379.547598] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6132379.548102] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6132379.548655] sp : ffff00001fe33c20 >[6132379.548998] x29: ffff00001fe33c20 x28: ffff8b9b8b8ba000 >[6132379.549510] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6132379.550086] x25: 0000000000000003 x24: 0000000000000000 >[6132379.550625] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6132379.551154] x21: ffff8d9c46fb0400 x20: ffff59fd7fe4b8a8 >[6132379.551660] x19: 0000000000000000 x18: 0000000000000000 >[6132379.552163] x17: 0000000000000002 x16: 0000000000000000 >[6132379.552677] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6132379.553246] x13: ffff8b8d72d57000 x12: ffff8b8d72d562fd >[6132379.553754] x11: 0000000000000000 x10: ffff00001fe33c80 >[6132379.554296] x9 : ffff59fd7ef3dad0 x8 : 0000000000014b87 >[6132379.554807] x7 : 0000000000000004 x6 : 0000000000000005 >[6132379.555305] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6132379.555799] x3 : 0000000000000020 x2 : 000031c040410000 >[6132379.556355] x1 : ffff4a0cfc970be0 x0 : 0000000000000020 >[6132379.556861] Call trace: >[6132379.557105] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6132379.557674] memcg_numa_stat_show+0x1d0/0x260 >[6132379.558088] cgroup_seqfile_show+0x50/0xd0 >[6132379.558509] kernfs_seq_show+0x30/0x38 >[6132379.558874] seq_read+0x148/0x478 >[6132379.559202] kernfs_fop_read+0x2c/0x1e0 >[6132379.559593] __vfs_read+0x20/0x48 >[6132379.559903] vfs_read+0x98/0x168 >[6132379.560203] ksys_read+0x6c/0xe8 >[6132379.560571] __arm64_sys_read+0x20/0x28 >[6132379.560936] el0_svc_common.constprop.0+0xa8/0x200 >[6132379.561462] el0_svc_handler+0x30/0x80 >[6132379.561808] el0_svc+0x10/0x14 >[6132423.531230] watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [kubelet:61154] >[6132423.532043] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6132423.538529] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6132423.544075] CPU: 13 PID: 61154 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6132423.545025] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6132423.545946] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6132423.546403] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6132423.547073] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6132423.547629] sp : ffff000059c5bc20 >[6132423.547954] x29: ffff000059c5bc20 x28: ffff8d8ced870000 >[6132423.548438] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6132423.548985] x25: 0000000000000003 x24: 0000000000000000 >[6132423.549484] x23: 0000000000000008 x22: ffff59fd7fe49800 >[6132423.550087] x21: ffff8dccefdb8000 x20: ffff59fd7fe4b8a8 >[6132423.550634] x19: 0000000000000001 x18: 0000000000000000 >[6132423.551142] x17: 0000000000000002 x16: 0000000000000000 >[6132423.551697] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6132423.552229] x13: ffff8b8d73420000 x12: ffff8b8d7341f31c >[6132423.552755] x11: 0000000000000000 x10: ffff000059c5bc80 >[6132423.553284] x9 : ffff59fd7ef3dad0 x8 : 0000000000018c60 >[6132423.553786] x7 : 0000000000000005 x6 : 0000000000000006 >[6132423.554310] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6132423.554873] x3 : 0000000000000033 x2 : 000033d0402b6000 >[6132423.555373] x1 : ffff4a2dca43bdb8 x0 : 0000000000000033 >[6132423.555943] Call trace: >[6132423.556268] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6132423.556746] memcg_numa_stat_show+0x1d0/0x260 >[6132423.557305] cgroup_seqfile_show+0x50/0xd0 >[6132423.557720] kernfs_seq_show+0x30/0x38 >[6132423.558194] seq_read+0x148/0x478 >[6132423.558508] kernfs_fop_read+0x2c/0x1e0 >[6132423.558941] __vfs_read+0x20/0x48 >[6132423.559268] vfs_read+0x98/0x168 >[6132423.559574] ksys_read+0x6c/0xe8 >[6132423.559946] __arm64_sys_read+0x20/0x28 >[6132423.560314] el0_svc_common.constprop.0+0xa8/0x200 >[6132423.560824] el0_svc_handler+0x30/0x80 >[6132423.561184] el0_svc+0x10/0x14 >[6132467.531020] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [migration/2:21] >[6132467.531708] watchdog: BUG: soft lockup - CPU#44 stuck for 22s! [kubelet:6818] >[6132467.531837] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6132467.532479] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6132467.539086] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6132467.545532] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6132467.551339] CPU: 2 PID: 21 Comm: migration/2 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6132467.557106] CPU: 44 PID: 6818 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6132467.558076] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6132467.559126] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6132467.560035] pstate: 60000005 (nZCv daif -PAN -UAO) >[6132467.560902] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6132467.561370] pc : multi_cpu_stop+0x78/0x180 >[6132467.561794] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6132467.562188] lr : multi_cpu_stop+0xb4/0x180 >[6132467.562694] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6132467.563079] sp : ffff000008d83d50 >[6132467.563610] sp : ffff000030643c20 >[6132467.563926] x29: ffff000008d83d50 x28: 0000000000000000 >[6132467.564236] x29: ffff000030643c20 x28: ffff8bcc4d2c8000 >[6132467.564745] x27: ffff00003dc03938 x26: 0000000000000060 >[6132467.565311] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6132467.565862] x25: 0000000000000000 x24: 0000000000000000 >[6132467.566402] x25: 0000000000000010 x24: 0000000000000000 >[6132467.567081] x23: 0000000000000000 x22: 0000000000000001 >[6132467.567629] x23: 0000000000000020 x22: ffff59fd7fe49800 >[6132467.568209] x21: ffff00003dc039d4 x20: ffff00003dc039b0 >[6132467.568737] x21: ffff8b8bf2a70800 x20: ffff59fd7fe4b8a8 >[6132467.569293] x19: 0000000000000001 x18: 0000000000000000 >[6132467.569771] x19: 0000000000000004 x18: 0000000000000000 >[6132467.570274] x17: 0000000000000001 x16: 0000000000000000 >[6132467.570758] x17: 0000000000000002 x16: 0000000000000000 >[6132467.571287] x15: 0000000000000000 x14: 0000000000000000 >[6132467.571854] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6132467.572363] x13: 0000000000000000 x12: 0000000000000000 >[6132467.572846] x13: ffff8bcc41960000 x12: ffff8bcc4195f34c >[6132467.573380] x11: 0000000000000000 x10: 0000000000000d60 >[6132467.573858] x11: 0000000000000000 x10: ffff000030643c80 >[6132467.574359] x9 : ffff59fd7ede5a0c x8 : ffff8b7d9d9c3040 >[6132467.574840] x9 : ffff59fd7ef3dad0 x8 : ffff8bcc4195f000 >[6132467.575345] x7 : ffff8b7d6b1b2800 x6 : 000000000000016c >[6132467.575861] x7 : 0000000000000000 x6 : 0000000000000001 >[6132467.576362] x5 : 0000318040441000 x4 : ffff8b7d9d9c2880 >[6132467.576962] x5 : 00000000fffffffb x4 : ffffffffffffffff >[6132467.577480] x3 : dead000000000200 x2 : ffff00003dc03960 >[6132467.577971] x3 : 0000000000000014 x2 : 000031a040475000 >[6132467.578488] x1 : ffff00003dc039d4 x0 : 0000000000000001 >[6132467.578971] x1 : ffff4a0f1d8197e0 x0 : 0000000000000014 >[6132467.579472] Call trace: >[6132467.579973] Call trace: >[6132467.580224] multi_cpu_stop+0x78/0x180 >[6132467.580467] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6132467.580880] cpu_stopper_thread+0xec/0x178 >[6132467.581339] memcg_numa_stat_show+0x1d0/0x260 >[6132467.581728] smpboot_thread_fn+0x158/0x1d8 >[6132467.582139] cgroup_seqfile_show+0x50/0xd0 >[6132467.582556] kthread+0x130/0x138 >[6132467.582991] kernfs_seq_show+0x30/0x38 >[6132467.583703] seq_read+0x148/0x478 >[6132467.584117] kernfs_fop_read+0x2c/0x1e0 >[6132467.584518] __vfs_read+0x20/0x48 >[6132467.584838] vfs_read+0x98/0x168 >[6132467.585191] ksys_read+0x6c/0xe8 >[6132467.585523] __arm64_sys_read+0x20/0x28 >[6132467.585899] el0_svc_common.constprop.0+0xa8/0x200 >[6132467.586371] el0_svc_handler+0x30/0x80 >[6132467.586845] el0_svc+0x10/0x14 >[6132507.531228] watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [kubelet:18334] >[6132507.532076] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6132507.538725] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6132507.544297] CPU: 13 PID: 18334 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6132507.545240] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6132507.546112] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6132507.546707] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6132507.547223] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6132507.547719] sp : ffff00001f9a3c20 >[6132507.548060] x29: ffff00001f9a3c20 x28: ffff8bace7a30000 >[6132507.548576] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6132507.549082] x25: 0000000000000003 x24: 0000000000000000 >[6132507.549589] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6132507.550096] x21: ffff8dacd6d7c800 x20: ffff59fd7fe4b8a8 >[6132507.550586] x19: 0000000000000000 x18: 0000000000000000 >[6132507.551091] x17: 0000000000000002 x16: 0000000000000000 >[6132507.551598] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6132507.552108] x13: ffff8b8d73419000 x12: ffff8b8d73418307 >[6132507.552632] x11: 0000000000000000 x10: ffff00001f9a3c80 >[6132507.553131] x9 : ffff59fd7ef3dad0 x8 : 000000000000bf9b >[6132507.553689] x7 : 0000000000000004 x6 : 0000000000000005 >[6132507.554199] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6132507.554725] x3 : 0000000000000018 x2 : 000031b040411000 >[6132507.555248] x1 : ffff4a1f1a15cb30 x0 : 0000000000000018 >[6132507.555752] Call trace: >[6132507.555991] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6132507.556586] memcg_numa_stat_show+0x1d0/0x260 >[6132507.557015] cgroup_seqfile_show+0x50/0xd0 >[6132507.557456] kernfs_seq_show+0x30/0x38 >[6132507.557804] seq_read+0x148/0x478 >[6132507.558132] kernfs_fop_read+0x2c/0x1e0 >[6132507.558536] __vfs_read+0x20/0x48 >[6132507.558852] vfs_read+0x98/0x168 >[6132507.559182] ksys_read+0x6c/0xe8 >[6132507.559497] __arm64_sys_read+0x20/0x28 >[6132507.559909] el0_svc_common.constprop.0+0xa8/0x200 >[6132507.560411] el0_svc_handler+0x30/0x80 >[6132507.560777] el0_svc+0x10/0x14 >[6132551.532055] watchdog: BUG: soft lockup - CPU#62 stuck for 22s! [kubelet:7308] >[6132551.532847] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6132551.539120] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6132551.544634] CPU: 62 PID: 7308 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6132551.545585] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6132551.546504] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6132551.547049] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6132551.547523] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6132551.548041] sp : ffff00001b39bc20 >[6132551.548366] x29: ffff00001b39bc20 x28: ffff8bcc5e37a000 >[6132551.548914] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6132551.549413] x25: 0000000000000003 x24: 0000000000000000 >[6132551.549918] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6132551.550398] x21: ffff8dccfc2cd000 x20: ffff59fd7fe4b8a8 >[6132551.550878] x19: 0000000000000000 x18: 0000000000000000 >[6132551.551360] x17: 0000000000000002 x16: 0000000000000000 >[6132551.551844] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6132551.552324] x13: ffff8bbd33939000 x12: ffff8bbd3393831c >[6132551.552841] x11: 0000000000000000 x10: ffff00001b39bc80 >[6132551.553324] x9 : ffff59fd7ef3dad0 x8 : 0000000000018847 >[6132551.553792] x7 : 0000000000000005 x6 : 0000000000000006 >[6132551.554295] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6132551.554761] x3 : 000000000000001e x2 : 000031b0404a7000 >[6132551.555257] x1 : ffff4a0f1ab8a3e0 x0 : 000000000000001e >[6132551.555737] Call trace: >[6132551.556060] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6132551.556585] memcg_numa_stat_show+0x1d0/0x260 >[6132551.557054] cgroup_seqfile_show+0x50/0xd0 >[6132551.557464] kernfs_seq_show+0x30/0x38 >[6132551.557880] seq_read+0x148/0x478 >[6132551.558188] kernfs_fop_read+0x2c/0x1e0 >[6132551.558617] __vfs_read+0x20/0x48 >[6132551.558968] vfs_read+0x98/0x168 >[6132551.559340] ksys_read+0x6c/0xe8 >[6132551.559669] __arm64_sys_read+0x20/0x28 >[6132551.560045] el0_svc_common.constprop.0+0xa8/0x200 >[6132551.560518] el0_svc_handler+0x30/0x80 >[6132551.560905] el0_svc+0x10/0x14 >[6132595.531212] watchdog: BUG: soft lockup - CPU#12 stuck for 22s! [kubelet:18891] >[6132595.531998] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6132595.538717] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6132595.545120] CPU: 12 PID: 18891 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6132595.546345] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6132595.547219] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6132595.547657] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6132595.548168] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6132595.548671] sp : ffff0000302fbc20 >[6132595.549000] x29: ffff0000302fbc20 x28: ffff8b9bfbd26000 >[6132595.549505] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6132595.550020] x25: 0000000000000010 x24: 0000000000000000 >[6132595.550527] x23: 0000000000000020 x22: ffff59fd7fe49800 >[6132595.551047] x21: ffff8b8bc8176c00 x20: ffff59fd7fe4b8a8 >[6132595.551561] x19: 0000000000000004 x18: 0000000000000000 >[6132595.552098] x17: 0000000000000001 x16: 0000000000000000 >[6132595.552607] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6132595.553126] x13: ffff8b8d7119a000 x12: ffff8b8d7119934c >[6132595.553686] x11: 0000000000000000 x10: ffff0000302fbc80 >[6132595.554218] x9 : ffff59fd7ef3dad0 x8 : ffff8b8d71199000 >[6132595.554716] x7 : 0000000000000000 x6 : 0000000000000001 >[6132595.555180] x5 : 00000000fffffffb x4 : ffffffffffffffff >[6132595.555804] x3 : 000000000000003f x2 : 000033d040252000 >[6132595.556360] x1 : ffff4a0cf58e4a30 x0 : 000000000000003f >[6132595.556858] Call trace: >[6132595.557162] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6132595.557614] memcg_numa_stat_show+0x1d0/0x260 >[6132595.558036] cgroup_seqfile_show+0x50/0xd0 >[6132595.558430] kernfs_seq_show+0x30/0x38 >[6132595.558788] seq_read+0x148/0x478 >[6132595.559112] kernfs_fop_read+0x2c/0x1e0 >[6132595.559470] __vfs_read+0x20/0x48 >[6132595.559790] vfs_read+0x98/0x168 >[6132595.560116] ksys_read+0x6c/0xe8 >[6132595.560442] __arm64_sys_read+0x20/0x28 >[6132595.560831] el0_svc_common.constprop.0+0xa8/0x200 >[6132595.561301] el0_svc_handler+0x30/0x80 >[6132595.561659] el0_svc+0x10/0x14 >[6132635.531259] watchdog: BUG: soft lockup - CPU#15 stuck for 22s! [kubelet:18288] >[6132635.532015] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6132635.539544] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6132635.545994] CPU: 15 PID: 18288 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6132635.547089] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6132635.548053] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6132635.548551] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6132635.549123] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6132635.549637] sp : ffff00001f91bc20 >[6132635.550027] x29: ffff00001f91bc20 x28: ffff8bbb95522000 >[6132635.550536] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6132635.551031] x25: 0000000000000003 x24: 0000000000000000 >[6132635.551513] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6132635.552079] x21: ffff8dabf63f9400 x20: ffff59fd7fe4b8a8 >[6132635.552565] x19: 0000000000000000 x18: 0000000000000000 >[6132635.553134] x17: 0000000000000002 x16: 0000000000000000 >[6132635.553706] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6132635.554262] x13: ffff8b8bdf96d000 x12: ffff8b8bdf96c307 >[6132635.554792] x11: 0000000000000000 x10: ffff00001f91bc80 >[6132635.555283] x9 : ffff59fd7ef3dad0 x8 : 000000000000bd55 >[6132635.555918] x7 : 0000000000000004 x6 : 0000000000000005 >[6132635.556466] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6132635.556987] x3 : 000000000000003e x2 : 000033d01ff1b000 >[6132635.557550] x1 : ffff4a0ced8065f0 x0 : 000000000000007e >[6132635.558080] Call trace: >[6132635.558400] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6132635.558878] memcg_numa_stat_show+0x1d0/0x260 >[6132635.559351] cgroup_seqfile_show+0x50/0xd0 >[6132635.559808] kernfs_seq_show+0x30/0x38 >[6132635.560220] seq_read+0x148/0x478 >[6132635.560551] kernfs_fop_read+0x2c/0x1e0 >[6132635.560941] __vfs_read+0x20/0x48 >[6132635.561354] vfs_read+0x98/0x168 >[6132635.561656] ksys_read+0x6c/0xe8 >[6132635.562012] __arm64_sys_read+0x20/0x28 >[6132635.562491] el0_svc_common.constprop.0+0xa8/0x200 >[6132635.562988] el0_svc_handler+0x30/0x80 >[6132635.563414] el0_svc+0x10/0x14 >[6132679.531256] watchdog: BUG: soft lockup - CPU#15 stuck for 22s! [kubelet:18321] >[6132679.532106] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6132679.538594] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6132679.544165] CPU: 15 PID: 18321 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6132679.545320] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6132679.546236] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6132679.546687] pc : mem_cgroup_nr_lru_pages.isra.7+0x9c/0xc8 >[6132679.547193] lr : mem_cgroup_nr_lru_pages.isra.7+0xac/0xc8 >[6132679.547694] sp : ffff00001f953c20 >[6132679.548016] x29: ffff00001f953c20 x28: 0000000000000000 >[6132679.548521] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6132679.549013] x25: 0000000000000010 x24: 0000000000000000 >[6132679.549526] x23: ffff8dabeaf925d0 x22: 0000000000000020 >[6132679.550037] x21: ffff59fd7fe49800 x20: ffff59fd7fe4b8a8 >[6132679.550542] x19: 0000000000000004 x18: 0000000000000000 >[6132679.551039] x17: 0000000000000002 x16: 0000000000000000 >[6132679.551606] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6132679.552144] x13: ffff8b8bdf96b000 x12: ffff8b8bdf96a327 >[6132679.552665] x11: 0000000000000000 x10: ffff00001f953c80 >[6132679.553174] x9 : ffff59fd7ef3da4c x8 : 000000000000dd7d >[6132679.553714] x7 : 0000000000000004 x6 : 0000000000000005 >[6132679.554302] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6132679.554865] x3 : 000000000000002d x2 : 000031d04048e000 >[6132679.555475] x1 : ffff4a0cf830cbd0 x0 : 000000000000002d >[6132679.555992] Call trace: >[6132679.556244] mem_cgroup_nr_lru_pages.isra.7+0x9c/0xc8 >[6132679.556715] memcg_numa_stat_show+0x14c/0x260 >[6132679.557169] cgroup_seqfile_show+0x50/0xd0 >[6132679.557597] kernfs_seq_show+0x30/0x38 >[6132679.558027] seq_read+0x148/0x478 >[6132679.558336] kernfs_fop_read+0x2c/0x1e0 >[6132679.558711] __vfs_read+0x20/0x48 >[6132679.559115] vfs_read+0x98/0x168 >[6132679.559437] ksys_read+0x6c/0xe8 >[6132679.559797] __arm64_sys_read+0x20/0x28 >[6132679.560170] el0_svc_common.constprop.0+0xa8/0x200 >[6132679.560675] el0_svc_handler+0x30/0x80 >[6132679.561044] el0_svc+0x10/0x14 >[6132719.531877] watchdog: BUG: soft lockup - CPU#53 stuck for 23s! [kubelet:4440] >[6132719.532672] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6132719.539402] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6132719.545434] CPU: 53 PID: 4440 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6132719.546413] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6132719.547324] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6132719.547754] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6132719.548251] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6132719.548775] sp : ffff0000329abc20 >[6132719.549108] x29: ffff0000329abc20 x28: ffff8d8cf6e06000 >[6132719.549621] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6132719.550213] x25: 0000000000000003 x24: 0000000000000000 >[6132719.550739] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6132719.551294] x21: ffff8d7c5054c800 x20: ffff59fd7fe4b8a8 >[6132719.551813] x19: 0000000000000000 x18: 0000000000000000 >[6132719.552330] x17: 0000000000000648 x16: 0000000000000000 >[6132719.552816] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6132719.553315] x13: ffff8b9d1d25c000 x12: ffff8b9d1d25b2e9 >[6132719.553823] x11: 0000000000000000 x10: ffff0000329abc80 >[6132719.554294] x9 : ffff59fd7ef3dad0 x8 : 000000000001f806 >[6132719.554769] x7 : 0000000000000005 x6 : 0000000000000006 >[6132719.555390] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6132719.555882] x3 : 000000000000002d x2 : 000031d04048e000 >[6132719.556370] x1 : ffff4a2dca68eab0 x0 : 000000000000002d >[6132719.556915] Call trace: >[6132719.557163] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6132719.557660] memcg_numa_stat_show+0x1d0/0x260 >[6132719.558062] cgroup_seqfile_show+0x50/0xd0 >[6132719.558439] kernfs_seq_show+0x30/0x38 >[6132719.558819] seq_read+0x148/0x478 >[6132719.559152] kernfs_fop_read+0x2c/0x1e0 >[6132719.559565] __vfs_read+0x20/0x48 >[6132719.559903] vfs_read+0x98/0x168 >[6132719.560241] ksys_read+0x6c/0xe8 >[6132719.560559] __arm64_sys_read+0x20/0x28 >[6132719.560922] el0_svc_common.constprop.0+0xa8/0x200 >[6132719.561387] el0_svc_handler+0x30/0x80 >[6132719.561726] el0_svc+0x10/0x14 >[6132763.531259] watchdog: BUG: soft lockup - CPU#15 stuck for 23s! [kubelet:18850] >[6132763.532084] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6132763.538615] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6132763.544627] CPU: 15 PID: 18850 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6132763.545603] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6132763.546538] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6132763.547070] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6132763.547568] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6132763.548155] sp : ffff00001fe33c20 >[6132763.548521] x29: ffff00001fe33c20 x28: ffff8b9c45c52000 >[6132763.549023] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6132763.549595] x25: 0000000000000003 x24: 0000000000000000 >[6132763.550098] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6132763.550622] x21: ffff8d9ce03cfc00 x20: ffff59fd7fe4b8a8 >[6132763.551196] x19: 0000000000000000 x18: 0000000000000000 >[6132763.551688] x17: 0000000000000002 x16: 0000000000000000 >[6132763.552211] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6132763.552786] x13: ffff8b8bdf96c000 x12: ffff8b8bdf96b2fd >[6132763.553341] x11: 0000000000000000 x10: ffff00001fe33c80 >[6132763.553821] x9 : ffff59fd7ef3dad0 x8 : 0000000000014bc7 >[6132763.554354] x7 : 0000000000000004 x6 : 0000000000000005 >[6132763.554943] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6132763.555449] x3 : 0000000000000026 x2 : 000031c0404a6000 >[6132763.555947] x1 : ffff4a1d2401c840 x0 : 0000000000000026 >[6132763.556441] Call trace: >[6132763.556696] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6132763.557185] memcg_numa_stat_show+0x1d0/0x260 >[6132763.557615] cgroup_seqfile_show+0x50/0xd0 >[6132763.558007] kernfs_seq_show+0x30/0x38 >[6132763.558386] seq_read+0x148/0x478 >[6132763.558718] kernfs_fop_read+0x2c/0x1e0 >[6132763.559085] __vfs_read+0x20/0x48 >[6132763.559409] vfs_read+0x98/0x168 >[6132763.559727] ksys_read+0x6c/0xe8 >[6132763.560030] __arm64_sys_read+0x20/0x28 >[6132763.560411] el0_svc_common.constprop.0+0xa8/0x200 >[6132763.560860] el0_svc_handler+0x30/0x80 >[6132763.561186] el0_svc+0x10/0x14 >[6132807.531075] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [kubelet:18280] >[6132807.531161] watchdog: BUG: soft lockup - CPU#9 stuck for 23s! [migration/9:57] >[6132807.531935] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6132807.532598] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6132807.539164] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6132807.545842] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6132807.551647] CPU: 3 PID: 18280 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6132807.557698] CPU: 9 PID: 57 Comm: migration/9 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6132807.558694] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6132807.559703] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6132807.560665] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6132807.561588] pstate: 60000005 (nZCv daif -PAN -UAO) >[6132807.562044] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6132807.562557] pc : multi_cpu_stop+0xf4/0x180 >[6132807.563110] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6132807.563112] sp : ffff00001f893c20 >[6132807.563554] lr : multi_cpu_stop+0xb4/0x180 >[6132807.563556] sp : ffff000008ea3d50 >[6132807.564112] x29: ffff00001f893c20 x28: ffff8b8be6bbe000 >[6132807.564531] x29: ffff000008ea3d50 x28: 0000000000000000 >[6132807.564956] x27: ffff59fd7fe4bbf4 x26: ffffffffffffe6ec >[6132807.565270] x27: ffff00005080b938 x26: 0000000000000060 >[6132807.565772] x25: 0000000000000003 x24: 0000000000000000 >[6132807.566289] x25: 0000000000000000 x24: 0000000000000000 >[6132807.566802] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6132807.567316] x23: 0000000000000000 x22: 0000000000000001 >[6132807.567827] x21: ffff8dbbedb6fc00 x20: ffff59fd7fe4b8a8 >[6132807.568338] x21: ffff00005080b9d4 x20: ffff00005080b9b0 >[6132807.568867] x19: 0000000000000000 x18: 0000000000000000 >[6132807.569392] x19: 0000000000000001 x18: 0000000000000000 >[6132807.569924] x17: 0000000000000009 x16: 0000000000000000 >[6132807.570445] x17: 0000000000000001 x16: 0000000000000000 >[6132807.570960] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6132807.571451] x15: 0000000000000000 x14: 0000000000000000 >[6132807.571933] x13: ffff8b7d1cf94000 x12: ffff8b7d1cf93311 >[6132807.572405] x13: 0000000000000000 x12: 0000000000000000 >[6132807.572904] x11: 0000000000000000 x10: ffff00001f893c80 >[6132807.573388] x11: 0000000000000012 x10: 0000000000000d60 >[6132807.573907] x9 : ffff59fd7ef3dad0 x8 : 000000000004eeeb >[6132807.574510] x9 : ffff59fd7ede5a0c x8 : ffff8b8d9e901f00 >[6132807.575022] x7 : 0000000000000005 x6 : 0000000000000006 >[6132807.575527] x7 : ffff8b8d8059e000 x6 : 0000000000000234 >[6132807.576023] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6132807.576523] x5 : 000031904042a000 x4 : ffff8b8d9e901740 >[6132807.577088] x3 : 0000000000000011 x2 : 000031a04042a000 >[6132807.577587] x3 : dead000000000200 x2 : ffff00005080b988 >[6132807.578088] x1 : ffff4a0f06a7eb30 x0 : 0000000000000011 >[6132807.578598] x1 : ffff00005080b9d4 x0 : 0000000000000001 >[6132807.579099] Call trace: >[6132807.579600] Call trace: >[6132807.580120] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6132807.580634] multi_cpu_stop+0xf4/0x180 >[6132807.580865] memcg_numa_stat_show+0x1d0/0x260 >[6132807.581109] cpu_stopper_thread+0xec/0x178 >[6132807.581597] cgroup_seqfile_show+0x50/0xd0 >[6132807.581957] smpboot_thread_fn+0x158/0x1d8 >[6132807.582375] kernfs_seq_show+0x30/0x38 >[6132807.582789] kthread+0x130/0x138 >[6132807.583198] seq_read+0x148/0x478 >[6132807.584755] kernfs_fop_read+0x2c/0x1e0 >[6132807.585142] __vfs_read+0x20/0x48 >[6132807.585546] vfs_read+0x98/0x168 >[6132807.585876] ksys_read+0x6c/0xe8 >[6132807.586197] __arm64_sys_read+0x20/0x28 >[6132807.586566] el0_svc_common.constprop.0+0xa8/0x200 >[6132807.587026] el0_svc_handler+0x30/0x80 >[6132807.587401] el0_svc+0x10/0x14 >[6132851.531324] watchdog: BUG: soft lockup - CPU#19 stuck for 23s! [kubelet:11266] >[6132851.532120] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6132851.538533] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6132851.544006] CPU: 19 PID: 11266 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6132851.544990] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6132851.545871] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6132851.546299] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6132851.546787] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6132851.547371] sp : ffff0000545dbc20 >[6132851.547674] x29: ffff0000545dbc20 x28: ffff8b7d6005e000 >[6132851.548173] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6132851.548726] x25: 0000000000000003 x24: 0000000000000000 >[6132851.549215] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6132851.549745] x21: ffff8dcd52578400 x20: ffff59fd7fe4b8a8 >[6132851.550281] x19: 0000000000000000 x18: 0000000000000000 >[6132851.550816] x17: 0000000000000002 x16: 0000000000000000 >[6132851.551364] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6132851.551925] x13: ffff8b9d20230000 x12: ffff8b9d2022f31c >[6132851.552450] x11: 0000000000000000 x10: ffff0000545dbc80 >[6132851.552974] x9 : ffff59fd7ef3dad0 x8 : 0000000000018ecc >[6132851.553488] x7 : 0000000000000005 x6 : 0000000000000006 >[6132851.554132] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6132851.554636] x3 : 0000000000000002 x2 : 0000318040441000 >[6132851.555143] x1 : ffff4a2ce39c6f30 x0 : 0000000000000002 >[6132851.555630] Call trace: >[6132851.555869] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6132851.556373] memcg_numa_stat_show+0x1d0/0x260 >[6132851.556793] cgroup_seqfile_show+0x50/0xd0 >[6132851.557187] kernfs_seq_show+0x30/0x38 >[6132851.557529] seq_read+0x148/0x478 >[6132851.557847] kernfs_fop_read+0x2c/0x1e0 >[6132851.558227] __vfs_read+0x20/0x48 >[6132851.558528] vfs_read+0x98/0x168 >[6132851.558830] ksys_read+0x6c/0xe8 >[6132851.559155] __arm64_sys_read+0x20/0x28 >[6132851.559529] el0_svc_common.constprop.0+0xa8/0x200 >[6132851.560011] el0_svc_handler+0x30/0x80 >[6132851.560380] el0_svc+0x10/0x14 >[6132895.531016] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kubelet:13350] >[6132895.531517] watchdog: BUG: soft lockup - CPU#32 stuck for 23s! [migration/32:174] >[6132895.531868] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6132895.532541] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6132895.539035] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6132895.545616] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6132895.551404] CPU: 1 PID: 13350 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6132895.557324] CPU: 32 PID: 174 Comm: migration/32 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6132895.558311] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6132895.559465] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6132895.560298] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6132895.561176] pstate: 60000005 (nZCv daif -PAN -UAO) >[6132895.561677] pc : mem_cgroup_nr_lru_pages.isra.7+0x9c/0xc8 >[6132895.562124] pc : multi_cpu_stop+0x78/0x180 >[6132895.562127] lr : multi_cpu_stop+0xb4/0x180 >[6132895.562685] lr : mem_cgroup_nr_lru_pages.isra.7+0xac/0xc8 >[6132895.562687] sp : ffff0000575d3c20 >[6132895.563133] sp : ffff00000a253d50 >[6132895.563611] x29: ffff0000575d3c20 x28: 0000000000000000 >[6132895.564113] x29: ffff00000a253d50 x28: 0000000000000000 >[6132895.564435] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6132895.564760] x27: ffff000031a73938 x26: 0000000000000060 >[6132895.565257] x25: 0000000000000010 x24: 0000000000000000 >[6132895.565733] x25: 0000000000000000 x24: 0000000000000000 >[6132895.566264] x23: ffff8b8c4fe025d0 x22: 0000000000000020 >[6132895.566754] x23: 0000000000000000 x22: 0000000000000001 >[6132895.567261] x21: ffff59fd7fe49800 x20: ffff59fd7fe4b8a8 >[6132895.567764] x21: ffff000031a739d4 x20: ffff000031a739b0 >[6132895.568308] x19: 0000000000000004 x18: 0000000000000000 >[6132895.568828] x19: 0000000000000001 x18: 0000000000000000 >[6132895.569305] x17: 0000000000000002 x16: 0000000000000000 >[6132895.569861] x17: 0000000000000001 x16: 0000000000000000 >[6132895.570419] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6132895.570913] x15: 0000000000000101 x14: ffff8dbb67809800 >[6132895.571465] x13: ffff8b7d1ada5000 x12: ffff8b7d1ada4327 >[6132895.571964] x13: 0000000000001337 x12: ffff8dbd2c500980 >[6132895.572493] x11: 0000000000000000 x10: ffff0000575d3c80 >[6132895.573039] x11: 000000000000001e x10: 0000000000000d60 >[6132895.573641] x9 : ffff59fd7ef3da4c x8 : 000000000000dcdd >[6132895.574136] x9 : ffff59fd7ede5a0c x8 : ffff8bbd9e946400 >[6132895.574710] x7 : 0000000000000004 x6 : 0000000000000005 >[6132895.575212] x7 : ffff8bbd6ef27800 x6 : 0000000000000298 >[6132895.575719] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6132895.576227] x5 : 000031c040410000 x4 : ffff8bbd9e945c40 >[6132895.576738] x3 : 0000000000000005 x2 : 000033804048c000 >[6132895.577229] x3 : dead000000000200 x2 : ffff000031a73988 >[6132895.577736] x1 : ffff4a1cfe4a8730 x0 : 0000000000000045 >[6132895.578245] x1 : ffff000031a739d4 x0 : 0000000000000001 >[6132895.578733] Call trace: >[6132895.579230] Call trace: >[6132895.579746] mem_cgroup_nr_lru_pages.isra.7+0x9c/0xc8 >[6132895.580265] multi_cpu_stop+0x78/0x180 >[6132895.580521] memcg_numa_stat_show+0x14c/0x260 >[6132895.580526] cgroup_seqfile_show+0x50/0xd0 >[6132895.580773] cpu_stopper_thread+0xec/0x178 >[6132895.581255] kernfs_seq_show+0x30/0x38 >[6132895.581626] smpboot_thread_fn+0x158/0x1d8 >[6132895.582052] seq_read+0x148/0x478 >[6132895.582435] kthread+0x130/0x138 >[6132895.582826] kernfs_fop_read+0x2c/0x1e0 >[6132895.584795] __vfs_read+0x20/0x48 >[6132895.585116] vfs_read+0x98/0x168 >[6132895.585476] ksys_read+0x6c/0xe8 >[6132895.585800] __arm64_sys_read+0x20/0x28 >[6132895.586173] el0_svc_common.constprop.0+0xa8/0x200 >[6132895.586626] el0_svc_handler+0x30/0x80 >[6132895.586997] el0_svc+0x10/0x14 >[6132935.531228] watchdog: BUG: soft lockup - CPU#13 stuck for 23s! [kubelet:11759] >[6132935.532065] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6132935.538644] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6132935.544574] CPU: 13 PID: 11759 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6132935.545566] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6132935.546419] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6132935.546931] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6132935.547438] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6132935.547942] sp : ffff00005541bc20 >[6132935.548268] x29: ffff00005541bc20 x28: ffff8b9c440e2000 >[6132935.548788] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6132935.549294] x25: 0000000000000003 x24: 0000000000000000 >[6132935.549821] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6132935.550306] x21: ffff8d9c96eafc00 x20: ffff59fd7fe4b8a8 >[6132935.550788] x19: 0000000000000000 x18: 0000000000000000 >[6132935.551258] x17: 0000000000000640 x16: 0000000000000000 >[6132935.551716] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6132935.552189] x13: ffff8b8d7341a000 x12: ffff8b8d734192fd >[6132935.552716] x11: 0000000000000000 x10: ffff00005541bc80 >[6132935.553210] x9 : ffff59fd7ef3dad0 x8 : 0000000000014df4 >[6132935.553824] x7 : 0000000000000004 x6 : 0000000000000005 >[6132935.554304] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6132935.554806] x3 : 0000000000000032 x2 : 000033d04029d000 >[6132935.555347] x1 : ffff4a0f065d3410 x0 : 0000000000000032 >[6132935.555900] Call trace: >[6132935.556141] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6132935.556676] memcg_numa_stat_show+0x1d0/0x260 >[6132935.557102] cgroup_seqfile_show+0x50/0xd0 >[6132935.557513] kernfs_seq_show+0x30/0x38 >[6132935.557865] seq_read+0x148/0x478 >[6132935.558196] kernfs_fop_read+0x2c/0x1e0 >[6132935.558571] __vfs_read+0x20/0x48 >[6132935.558900] vfs_read+0x98/0x168 >[6132935.559262] ksys_read+0x6c/0xe8 >[6132935.559575] __arm64_sys_read+0x20/0x28 >[6132935.559955] el0_svc_common.constprop.0+0xa8/0x200 >[6132935.560412] el0_svc_handler+0x30/0x80 >[6132935.560779] el0_svc+0x10/0x14 >[6132979.531152] watchdog: BUG: soft lockup - CPU#8 stuck for 23s! [kubelet:18924] >[6132979.531968] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6132979.538495] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6132979.544321] CPU: 8 PID: 18924 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6132979.545328] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6132979.546158] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6132979.546593] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6132979.547111] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6132979.547631] sp : ffff00003e19bc20 >[6132979.547954] x29: ffff00003e19bc20 x28: ffff8b8c345d0000 >[6132979.548470] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6132979.548996] x25: 0000000000000003 x24: 0000000000000000 >[6132979.549506] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6132979.550010] x21: ffff8dccfb7c0800 x20: ffff59fd7fe4b8a8 >[6132979.550519] x19: 0000000000000000 x18: 0000000000000000 >[6132979.551036] x17: 0000000000000002 x16: 0000000000000000 >[6132979.551551] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6132979.552055] x13: ffff8b8d72fb0000 x12: ffff8b8d72faf31c >[6132979.552572] x11: 0000000000000000 x10: ffff00003e19bc80 >[6132979.553180] x9 : ffff59fd7ef3dad0 x8 : 00000000000186c5 >[6132979.553701] x7 : 0000000000000005 x6 : 0000000000000006 >[6132979.554184] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6132979.554655] x3 : 0000000000000033 x2 : 000033d01ff98000 >[6132979.555136] x1 : ffff4a20bfe57080 x0 : 0000000000000073 >[6132979.555619] Call trace: >[6132979.555899] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6132979.556443] memcg_numa_stat_show+0x1d0/0x260 >[6132979.556872] cgroup_seqfile_show+0x50/0xd0 >[6132979.557327] kernfs_seq_show+0x30/0x38 >[6132979.557719] seq_read+0x148/0x478 >[6132979.558025] kernfs_fop_read+0x2c/0x1e0 >[6132979.558404] __vfs_read+0x20/0x48 >[6132979.558723] vfs_read+0x98/0x168 >[6132979.559036] ksys_read+0x6c/0xe8 >[6132979.559416] __arm64_sys_read+0x20/0x28 >[6132979.559795] el0_svc_common.constprop.0+0xa8/0x200 >[6132979.560374] el0_svc_handler+0x30/0x80 >[6132979.560747] el0_svc+0x10/0x14 >[6133023.531077] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [kubelet:18929] >[6133023.531251] watchdog: BUG: soft lockup - CPU#15 stuck for 23s! [migration/15:87] >[6133023.531902] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6133023.532676] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6133023.539204] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6133023.546036] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6133023.551763] CPU: 3 PID: 18929 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6133023.557427] CPU: 15 PID: 87 Comm: migration/15 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6133023.558408] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6133023.559371] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6133023.560267] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6133023.561173] pstate: 60000005 (nZCv daif -PAN -UAO) >[6133023.561644] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6133023.562093] pc : multi_cpu_stop+0x78/0x180 >[6133023.562578] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6133023.563080] lr : multi_cpu_stop+0xb4/0x180 >[6133023.563585] sp : ffff00003001bc20 >[6133023.563985] sp : ffff000008f93d50 >[6133023.564361] x29: ffff00003001bc20 x28: ffff8b7b63288000 >[6133023.564688] x29: ffff000008f93d50 x28: 0000000000000000 >[6133023.565292] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6133023.565819] x27: ffff00000aab3938 x26: 0000000000000060 >[6133023.566376] x25: 0000000000000010 x24: 0000000000000000 >[6133023.566929] x25: 0000000000000000 x24: 0000000000000000 >[6133023.567536] x23: 0000000000000020 x22: ffff59fd7fe49800 >[6133023.568045] x23: 0000000000000000 x22: 0000000000000001 >[6133023.568591] x21: ffff8b7b8d864800 x20: ffff59fd7fe4b8a8 >[6133023.569159] x21: ffff00000aab39d4 x20: ffff00000aab39b0 >[6133023.569732] x19: 0000000000000004 x18: 0000000000000000 >[6133023.570240] x19: 0000000000000001 x18: 0000000000000000 >[6133023.570837] x17: 0000000000000002 x16: 0000000000000000 >[6133023.571431] x17: 0000000000000001 x16: 0000000000000000 >[6133023.571929] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6133023.572471] x15: 0000000000000000 x14: ffff8dbd7035c800 >[6133023.573067] x13: ffff8b7d1cf94000 x12: ffff8b7d1cf93346 >[6133023.573598] x13: 0000000000006e01 x12: ffff8dbd6f405d80 >[6133023.574172] x11: 0000000000000000 x10: ffff00003001bc80 >[6133023.574723] x11: 0000000000000032 x10: 0000000000000d60 >[6133023.575352] x9 : ffff59fd7ef3dad0 x8 : 0000000000000021 >[6133023.575862] x9 : ffff59fd7ede5a0c x8 : ffff8b8d9e9e4180 >[6133023.576397] x7 : 0000000000000001 x6 : 0000000000000002 >[6133023.576971] x7 : ffff8b8d7c5a1000 x6 : 0000000000000390 >[6133023.577521] x5 : 00000000fffffffa x4 : ffffffffffffffff >[6133023.578005] x5 : 00003190404c0000 x4 : ffff8b8d9e9e39c0 >[6133023.578549] x3 : 000000000000003b x2 : 000033d01fed0000 >[6133023.579034] x3 : dead000000000200 x2 : ffff00000aab3988 >[6133023.579522] x1 : ffff4a0cf416e110 x0 : 000000000000007b >[6133023.580035] x1 : ffff00000aab39d4 x0 : 0000000000000001 >[6133023.580628] Call trace: >[6133023.581166] Call trace: >[6133023.581409] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6133023.581668] multi_cpu_stop+0x78/0x180 >[6133023.582180] memcg_numa_stat_show+0x1d0/0x260 >[6133023.582535] cpu_stopper_thread+0xec/0x178 >[6133023.583035] cgroup_seqfile_show+0x50/0xd0 >[6133023.583407] smpboot_thread_fn+0x158/0x1d8 >[6133023.583789] kernfs_seq_show+0x30/0x38 >[6133023.584176] kthread+0x130/0x138 >[6133023.584540] seq_read+0x148/0x478 >[6133023.585172] kernfs_fop_read+0x2c/0x1e0 >[6133023.585589] __vfs_read+0x20/0x48 >[6133023.585933] vfs_read+0x98/0x168 >[6133023.586266] ksys_read+0x6c/0xe8 >[6133023.586603] __arm64_sys_read+0x20/0x28 >[6133023.586982] el0_svc_common.constprop.0+0xa8/0x200 >[6133023.587438] el0_svc_handler+0x30/0x80 >[6133023.587795] el0_svc+0x10/0x14 >[6133063.531151] watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [kubelet:55880] >[6133063.531963] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6133063.538444] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6133063.544050] CPU: 8 PID: 55880 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6133063.545026] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6133063.545902] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6133063.546354] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6133063.546858] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6133063.547338] sp : ffff00003e28bc20 >[6133063.547737] x29: ffff00003e28bc20 x28: ffff8bbbc6888000 >[6133063.548255] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6133063.548765] x25: 0000000000000003 x24: 0000000000000000 >[6133063.549370] x23: 0000000000000008 x22: ffff59fd7fe49800 >[6133063.549917] x21: ffff8d9c6cbef800 x20: ffff59fd7fe4b8a8 >[6133063.550468] x19: 0000000000000001 x18: 0000000000000000 >[6133063.550949] x17: 0000000000000002 x16: 0000000000000000 >[6133063.551413] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6133063.551873] x13: ffff8b8d72fa9000 x12: ffff8b8d72fa82fd >[6133063.552435] x11: 0000000000000000 x10: ffff00003e28bc80 >[6133063.552954] x9 : ffff59fd7ef3dad0 x8 : 0000000000014b80 >[6133063.553498] x7 : 0000000000000004 x6 : 0000000000000005 >[6133063.554053] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6133063.554601] x3 : 0000000000000010 x2 : 000033a040411000 >[6133063.555098] x1 : ffff4a2dc77aa9a8 x0 : 0000000000000050 >[6133063.555641] Call trace: >[6133063.555899] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6133063.556365] memcg_numa_stat_show+0x1d0/0x260 >[6133063.556789] cgroup_seqfile_show+0x50/0xd0 >[6133063.557181] kernfs_seq_show+0x30/0x38 >[6133063.557546] seq_read+0x148/0x478 >[6133063.557871] kernfs_fop_read+0x2c/0x1e0 >[6133063.558238] __vfs_read+0x20/0x48 >[6133063.558576] vfs_read+0x98/0x168 >[6133063.558893] ksys_read+0x6c/0xe8 >[6133063.559206] __arm64_sys_read+0x20/0x28 >[6133063.559576] el0_svc_common.constprop.0+0xa8/0x200 >[6133063.560042] el0_svc_handler+0x30/0x80 >[6133063.560400] el0_svc+0x10/0x14 >[6133067.531689] watchdog: BUG: soft lockup - CPU#43 stuck for 23s! [migration/43:231] >[6133067.532622] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6133067.539082] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6133067.544599] CPU: 43 PID: 231 Comm: migration/43 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6133067.545573] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6133067.546448] pstate: 60000005 (nZCv daif -PAN -UAO) >[6133067.546904] pc : multi_cpu_stop+0x78/0x180 >[6133067.547283] lr : multi_cpu_stop+0xb4/0x180 >[6133067.547679] sp : ffff00000a41bd50 >[6133067.547983] x29: ffff00000a41bd50 x28: 0000000000000000 >[6133067.548475] x27: ffff00001f9db938 x26: 0000000000000060 >[6133067.548972] x25: 0000000000000000 x24: 0000000000000000 >[6133067.549528] x23: 0000000000000000 x22: 0000000000000001 >[6133067.550012] x21: ffff00001f9db9d4 x20: ffff00001f9db9b0 >[6133067.550513] x19: 0000000000000001 x18: 0000000000000000 >[6133067.551020] x17: 0000000000000001 x16: 0000000000000000 >[6133067.551508] x15: 0000000000000000 x14: 0000000000000000 >[6133067.551977] x13: 0000000000000000 x12: 0000000000000000 >[6133067.552617] x11: 000000000000003a x10: 0000000000000d60 >[6133067.553128] x9 : ffff59fd7ede5a0c x8 : ffff8bcd9e9452c0 >[6133067.553634] x7 : 000a2bece15fce38 x6 : 0000000000000385 >[6133067.554186] x5 : 00000000f5257d14 x4 : ffff8bcd9e944b00 >[6133067.554669] x3 : dead000000000200 x2 : ffff00001f9db988 >[6133067.555205] x1 : ffff00001f9db9d4 x0 : 0000000000000001 >[6133067.555671] Call trace: >[6133067.555961] multi_cpu_stop+0x78/0x180 >[6133067.556316] cpu_stopper_thread+0xec/0x178 >[6133067.556718] smpboot_thread_fn+0x158/0x1d8 >[6133067.557092] kthread+0x130/0x138 >[6133107.531089] watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [kubelet:18334] >[6133107.531206] watchdog: BUG: soft lockup - CPU#12 stuck for 22s! [migration/12:72] >[6133107.531949] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6133107.532737] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6133107.539141] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6133107.545710] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6133107.551441] CPU: 4 PID: 18334 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6133107.557179] CPU: 12 PID: 72 Comm: migration/12 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6133107.558169] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6133107.559143] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6133107.560016] pstate: 20000005 (nzCv daif -PAN -UAO) >[6133107.560903] pstate: 60000005 (nZCv daif -PAN -UAO) >[6133107.561365] pc : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6133107.561818] pc : multi_cpu_stop+0xf4/0x180 >[6133107.562430] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6133107.562806] lr : multi_cpu_stop+0xb4/0x180 >[6133107.562809] sp : ffff000008f1bd50 >[6133107.563314] sp : ffff00001f9a3c20 >[6133107.563742] x29: ffff000008f1bd50 x28: 0000000000000000 >[6133107.564062] x29: ffff00001f9a3c20 x28: ffff8bbd6fde6000 >[6133107.564389] x27: ffff000036c8b308 x26: 0000000000000060 >[6133107.565051] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6133107.565603] x25: 0000000000000000 x24: 0000000000000000 >[6133107.566129] x25: 0000000000000003 x24: 0000000000000000 >[6133107.566590] x23: 0000000000000000 x22: 0000000000000001 >[6133107.567108] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6133107.567607] x21: ffff000036c8b3a4 x20: ffff000036c8b380 >[6133107.568160] x21: ffff8dcd9c5f0400 x20: ffff59fd7fe4b8a8 >[6133107.568689] x19: 0000000000000001 x18: 0000000000000000 >[6133107.569216] x19: 0000000000000000 x18: 0000000000000000 >[6133107.569738] x17: 0000000000000001 x16: 0000000000000000 >[6133107.570305] x17: 0000000000000001 x16: 0000000000000000 >[6133107.570847] x15: 0000000000000000 x14: ffff8dbd94421000 >[6133107.571389] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6133107.571875] x13: 00000000000002cf x12: ffff8dbcf9e6bd80 >[6133107.572496] x13: ffff8b7d1b325000 x12: ffff8b7d1b32431c >[6133107.573064] x11: 000000000000001e x10: 0000000000000d60 >[6133107.573647] x11: 0000000000000000 x10: ffff00001f9a3c80 >[6133107.574154] x9 : ffff59fd7ede5a0c x8 : ffff8b8d9e953040 >[6133107.574688] x9 : ffff59fd7ef3dad0 x8 : 0000000000018720 >[6133107.575175] x7 : ffff8b8d7c5a2c00 x6 : 0000000000000014 >[6133107.575699] x7 : 0000000000000005 x6 : 0000000000000006 >[6133107.576196] x5 : 0000319040475000 x4 : ffff8b8d9e952880 >[6133107.576697] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6133107.577210] x3 : dead000000000200 x2 : ffff000036c8b358 >[6133107.577734] x3 : 0000000000000023 x2 : 0000000000000023 >[6133107.578256] x1 : ffff000036c8b3a4 x0 : 0000000000000001 >[6133107.578761] x1 : 0000000000000023 x0 : 0000000000000023 >[6133107.579284] Call trace: >[6133107.579796] Call trace: >[6133107.580323] multi_cpu_stop+0xf4/0x180 >[6133107.580583] mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6133107.580839] cpu_stopper_thread+0xec/0x178 >[6133107.581220] memcg_numa_stat_show+0x1d0/0x260 >[6133107.581713] smpboot_thread_fn+0x158/0x1d8 >[6133107.582226] cgroup_seqfile_show+0x50/0xd0 >[6133107.582676] kthread+0x130/0x138 >[6133107.583129] kernfs_seq_show+0x30/0x38 >[6133107.584303] seq_read+0x148/0x478 >[6133107.584657] kernfs_fop_read+0x2c/0x1e0 >[6133107.585081] __vfs_read+0x20/0x48 >[6133107.585416] vfs_read+0x98/0x168 >[6133107.585793] ksys_read+0x6c/0xe8 >[6133107.586132] __arm64_sys_read+0x20/0x28 >[6133107.586542] el0_svc_common.constprop.0+0xa8/0x200 >[6133107.587013] el0_svc_handler+0x30/0x80 >[6133107.587426] el0_svc+0x10/0x14 >[6133151.531308] watchdog: BUG: soft lockup - CPU#18 stuck for 22s! [kubelet:4439] >[6133151.532197] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6133151.538706] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6133151.544455] CPU: 18 PID: 4439 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6133151.545548] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6133151.546453] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6133151.546927] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6133151.547458] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6133151.548130] sp : ffff00001e7ebc20 >[6133151.548431] x29: ffff00001e7ebc20 x28: ffff8bbc0a78a000 >[6133151.548983] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6133151.549459] x25: 0000000000000003 x24: 0000000000000000 >[6133151.549939] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6133151.550413] x21: ffff8dcd2a0ee000 x20: ffff59fd7fe4b8a8 >[6133151.550899] x19: 0000000000000000 x18: 0000000000000000 >[6133151.551381] x17: 0000000000000002 x16: 0000000000000000 >[6133151.551995] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6133151.552513] x13: ffff8b9d205f1000 x12: ffff8b9d205f031c >[6133151.553027] x11: 0000000000000000 x10: ffff00001e7ebc80 >[6133151.553547] x9 : ffff59fd7ef3dad0 x8 : 0000000000018868 >[6133151.554070] x7 : 0000000000000005 x6 : 0000000000000006 >[6133151.554596] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6133151.555151] x3 : 0000000000000034 x2 : 000033d01ffb1000 >[6133151.555685] x1 : ffff4a1c22a442b0 x0 : 0000000000000074 >[6133151.556163] Call trace: >[6133151.556444] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6133151.556897] memcg_numa_stat_show+0x1d0/0x260 >[6133151.557351] cgroup_seqfile_show+0x50/0xd0 >[6133151.557739] kernfs_seq_show+0x30/0x38 >[6133151.558144] seq_read+0x148/0x478 >[6133151.558459] kernfs_fop_read+0x2c/0x1e0 >[6133151.558812] __vfs_read+0x20/0x48 >[6133151.559148] vfs_read+0x98/0x168 >[6133151.559443] ksys_read+0x6c/0xe8 >[6133151.559794] __arm64_sys_read+0x20/0x28 >[6133151.560153] el0_svc_common.constprop.0+0xa8/0x200 >[6133151.560672] el0_svc_handler+0x30/0x80 >[6133151.561040] el0_svc+0x10/0x14 >[6133195.531257] watchdog: BUG: soft lockup - CPU#15 stuck for 22s! [kubelet:4832] >[6133195.532151] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6133195.539493] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6133195.546025] CPU: 15 PID: 4832 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6133195.547077] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6133195.547946] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6133195.548400] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6133195.548945] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6133195.549480] sp : ffff00005e9c3c20 >[6133195.549823] x29: ffff00005e9c3c20 x28: ffff8dcd88810000 >[6133195.550381] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6133195.550904] x25: 0000000000000010 x24: 0000000000000000 >[6133195.551510] x23: 0000000000000020 x22: ffff59fd7fe49800 >[6133195.552028] x21: ffff8b8c5480d800 x20: ffff59fd7fe4b8a8 >[6133195.552550] x19: 0000000000000004 x18: 0000000000000000 >[6133195.553074] x17: 0000000000000002 x16: 0000000000000000 >[6133195.553656] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6133195.554200] x13: ffff8b8bdf96c000 x12: ffff8b8bdf96b34e >[6133195.554802] x11: 0000000000000000 x10: ffff00005e9c3c80 >[6133195.555338] x9 : ffff59fd7ef3dad0 x8 : ffff8b8bdf96b000 >[6133195.555828] x7 : 0000000000000000 x6 : 0000000000000001 >[6133195.556353] x5 : 00000000fffffffb x4 : ffffffffffffffff >[6133195.556959] x3 : 0000000000000036 x2 : 000033d040301000 >[6133195.557464] x1 : ffff4a1cf6abf520 x0 : 0000000000000036 >[6133195.557926] Call trace: >[6133195.558162] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6133195.558629] memcg_numa_stat_show+0x1d0/0x260 >[6133195.559063] cgroup_seqfile_show+0x50/0xd0 >[6133195.559466] kernfs_seq_show+0x30/0x38 >[6133195.559843] seq_read+0x148/0x478 >[6133195.560163] kernfs_fop_read+0x2c/0x1e0 >[6133195.560531] __vfs_read+0x20/0x48 >[6133195.560848] vfs_read+0x98/0x168 >[6133195.561175] ksys_read+0x6c/0xe8 >[6133195.561600] __arm64_sys_read+0x20/0x28 >[6133195.561981] el0_svc_common.constprop.0+0xa8/0x200 >[6133195.562429] el0_svc_handler+0x30/0x80 >[6133195.562782] el0_svc+0x10/0x14 >[6133199.531476] watchdog: BUG: soft lockup - CPU#29 stuck for 23s! [migration/29:159] >[6133199.532329] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6133199.538834] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6133199.544677] CPU: 29 PID: 159 Comm: migration/29 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6133199.545722] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6133199.546585] pstate: 60000005 (nZCv daif -PAN -UAO) >[6133199.547131] pc : multi_cpu_stop+0x78/0x180 >[6133199.547583] lr : multi_cpu_stop+0xb4/0x180 >[6133199.548007] sp : ffff00000a1dbd50 >[6133199.548375] x29: ffff00000a1dbd50 x28: 0000000000000000 >[6133199.548891] x27: ffff00001d773938 x26: 0000000000000060 >[6133199.549420] x25: 0000000000000000 x24: 0000000000000000 >[6133199.549937] x23: 0000000000000000 x22: 0000000000000001 >[6133199.550494] x21: ffff00001d7739d4 x20: ffff00001d7739b0 >[6133199.551052] x19: 0000000000000001 x18: 0000000000000000 >[6133199.551641] x17: 0000000000000001 x16: 0000000000000000 >[6133199.552122] x15: 0000000000000000 x14: 0000000000000000 >[6133199.552597] x13: 0000000000000000 x12: 0000000000000000 >[6133199.553087] x11: 0000000000000141 x10: 0000000000000d60 >[6133199.553591] x9 : ffff59fd7ede5a0c x8 : ffff8bad9e998dc0 >[6133199.554090] x7 : ffff8bad55fa6000 x6 : 0000000000000373 >[6133199.554564] x5 : 000031b04048e000 x4 : ffff8bad9e998600 >[6133199.555077] x3 : dead000000000200 x2 : ffff00001d773988 >[6133199.555596] x1 : ffff00001d7739d4 x0 : 0000000000000001 >[6133199.556114] Call trace: >[6133199.556371] multi_cpu_stop+0x78/0x180 >[6133199.556752] cpu_stopper_thread+0xec/0x178 >[6133199.557162] smpboot_thread_fn+0x158/0x1d8 >[6133199.557573] kthread+0x130/0x138 >[6133235.531009] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kubelet:18304] >[6133235.531898] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6133235.539244] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6133235.545739] CPU: 0 PID: 18304 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6133235.546824] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6133235.547794] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6133235.548393] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6133235.548895] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6133235.549486] sp : ffff00001f943c20 >[6133235.549795] x29: ffff00001f943c20 x28: ffff8babf031a000 >[6133235.550308] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6133235.550830] x25: 0000000000000003 x24: 0000000000000000 >[6133235.551419] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6133235.551914] x21: ffff8d9c1a4bc800 x20: ffff59fd7fe4b8a8 >[6133235.552420] x19: 0000000000000000 x18: 0000000000000000 >[6133235.552956] x17: 0000000000000002 x16: 0000000000000000 >[6133235.553502] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6133235.554013] x13: ffff8b8d71fd0000 x12: ffff8b8d71fcf2ff >[6133235.554605] x11: 0000000000000000 x10: ffff00001f943c80 >[6133235.555152] x9 : ffff59fd7ef3dad0 x8 : 0000000000014c15 >[6133235.555678] x7 : 0000000000000004 x6 : 0000000000000005 >[6133235.556260] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6133235.556802] x3 : 000000000000000f x2 : 00003190404c0000 >[6133235.557322] x1 : ffff4a2b4ce79b90 x0 : 000000000000000f >[6133235.557902] Call trace: >[6133235.558161] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6133235.558658] memcg_numa_stat_show+0x1d0/0x260 >[6133235.559107] cgroup_seqfile_show+0x50/0xd0 >[6133235.559517] kernfs_seq_show+0x30/0x38 >[6133235.559907] seq_read+0x148/0x478 >[6133235.560264] kernfs_fop_read+0x2c/0x1e0 >[6133235.560670] __vfs_read+0x20/0x48 >[6133235.561024] vfs_read+0x98/0x168 >[6133235.561467] ksys_read+0x6c/0xe8 >[6133235.561853] __arm64_sys_read+0x20/0x28 >[6133235.562265] el0_svc_common.constprop.0+0xa8/0x200 >[6133235.562776] el0_svc_handler+0x30/0x80 >[6133235.563156] el0_svc+0x10/0x14 >[6133279.531709] watchdog: BUG: soft lockup - CPU#44 stuck for 22s! [kubelet:18892] >[6133279.532491] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6133279.539138] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6133279.545084] CPU: 44 PID: 18892 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6133279.546009] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6133279.546810] pstate: 20000005 (nzCv daif -PAN -UAO) >[6133279.547273] pc : css_next_descendant_pre+0x24/0x88 >[6133279.547722] lr : mem_cgroup_iter+0xc4/0x378 >[6133279.548105] sp : ffff000019e1bc10 >[6133279.548443] x29: ffff000019e1bc10 x28: ffff8b9b6d7dc000 >[6133279.548953] x27: ffff59fd7f671bf8 x26: ffff59fd7f8b5a88 >[6133279.549446] x25: 0000000000000010 x24: 0000000000000001 >[6133279.549939] x23: ffff8b9b6d7dc000 x22: 0000000000000000 >[6133279.550472] x21: ffff8b9b6d7dc000 x20: ffff8dbd9e88a000 >[6133279.551079] x19: ffff8b9b6d7dc000 x18: 0000000000000000 >[6133279.551571] x17: 0000000000000002 x16: 0000000000000000 >[6133279.552087] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6133279.552583] x13: ffff8bcc4195a000 x12: ffff8bcc41959313 >[6133279.553091] x11: 0000000000000000 x10: ffff000019e1bc80 >[6133279.553600] x9 : ffff59fd7ef3cdc4 x8 : 000000000004f101 >[6133279.554111] x7 : 0000000000000005 x6 : 0000000000000006 >[6133279.554633] x5 : 00000000fffffff6 x4 : 0000000000000000 >[6133279.555130] x3 : 0000000000000000 x2 : 0000000000000000 >[6133279.555650] x1 : ffff8b9b6d7dc000 x0 : ffff8b9b6d7dc000 >[6133279.556148] Call trace: >[6133279.556411] css_next_descendant_pre+0x24/0x88 >[6133279.556932] mem_cgroup_iter+0xc4/0x378 >[6133279.557306] memcg_numa_stat_show+0x1e4/0x260 >[6133279.557736] cgroup_seqfile_show+0x50/0xd0 >[6133279.558123] kernfs_seq_show+0x30/0x38 >[6133279.558479] seq_read+0x148/0x478 >[6133279.558792] kernfs_fop_read+0x2c/0x1e0 >[6133279.559147] __vfs_read+0x20/0x48 >[6133279.559458] vfs_read+0x98/0x168 >[6133279.559763] ksys_read+0x6c/0xe8 >[6133279.560067] __arm64_sys_read+0x20/0x28 >[6133279.560451] el0_svc_common.constprop.0+0xa8/0x200 >[6133279.561010] el0_svc_handler+0x30/0x80 >[6133279.561370] el0_svc+0x10/0x14 >[6133283.531099] watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [migration/5:36] >[6133283.531961] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6133283.538537] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6133283.544428] CPU: 5 PID: 36 Comm: migration/5 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6133283.545420] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6133283.546359] pstate: 60000005 (nZCv daif -PAN -UAO) >[6133283.546844] pc : multi_cpu_stop+0xf4/0x180 >[6133283.547231] lr : multi_cpu_stop+0xb4/0x180 >[6133283.547631] sp : ffff000008dfbd50 >[6133283.547949] x29: ffff000008dfbd50 x28: 0000000000000000 >[6133283.548464] x27: ffff00003dc03938 x26: 0000000000000060 >[6133283.548960] x25: 0000000000000000 x24: 0000000000000000 >[6133283.549460] x23: 0000000000000000 x22: 0000000000000001 >[6133283.549969] x21: ffff00003dc039d4 x20: ffff00003dc039b0 >[6133283.550471] x19: 0000000000000001 x18: 0000000000000000 >[6133283.551075] x17: 0000000000000001 x16: 0000000000000000 >[6133283.551679] x15: 0000000000000000 x14: 0000000000000000 >[6133283.552174] x13: 0000000000000000 x12: 0000000000000000 >[6133283.552696] x11: 0000000000000000 x10: 0000000000000d60 >[6133283.553204] x9 : ffff59fd7ede5a0c x8 : ffff8b7d9da24180 >[6133283.553737] x7 : ffff8b7d6a380000 x6 : 000000000000029a >[6133283.554237] x5 : 000031804048c000 x4 : ffff8b7d9da239c0 >[6133283.554739] x3 : dead000000000200 x2 : ffff00003dc03960 >[6133283.555248] x1 : ffff00003dc039d4 x0 : 0000000000000001 >[6133283.555799] Call trace: >[6133283.556086] multi_cpu_stop+0xf4/0x180 >[6133283.556468] cpu_stopper_thread+0xec/0x178 >[6133283.556893] smpboot_thread_fn+0x158/0x1d8 >[6133283.557293] kthread+0x130/0x138 >[6133323.531229] watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [kubelet:18929] >[6133323.531997] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6133323.538440] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6133323.544095] CPU: 13 PID: 18929 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6133323.545068] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6133323.545927] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6133323.546356] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6133323.546841] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6133323.547344] sp : ffff00003001bc20 >[6133323.547663] x29: ffff00003001bc20 x28: ffff8bac25a8c000 >[6133323.548180] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6133323.548704] x25: 0000000000000010 x24: 0000000000000000 >[6133323.549223] x23: 0000000000000020 x22: ffff59fd7fe49800 >[6133323.549762] x21: ffff8b7ba4695800 x20: ffff59fd7fe4b8a8 >[6133323.550256] x19: 0000000000000004 x18: 0000000000000000 >[6133323.550865] x17: 0000000000000002 x16: 0000000000000000 >[6133323.551358] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6133323.551882] x13: ffff8b8d73419000 x12: ffff8b8d73418348 >[6133323.552422] x11: 0000000000000000 x10: ffff00003001bc80 >[6133323.553056] x9 : ffff59fd7ef3dad0 x8 : 0000000000000021 >[6133323.553602] x7 : 0000000000000001 x6 : 0000000000000002 >[6133323.554111] x5 : 00000000fffffffa x4 : ffffffffffffffff >[6133323.554605] x3 : 0000000000000017 x2 : 000033a0404c0000 >[6133323.555165] x1 : ffff4a1943173d10 x0 : 0000000000000057 >[6133323.555743] Call trace: >[6133323.555979] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6133323.556572] memcg_numa_stat_show+0x1d0/0x260 >[6133323.556976] cgroup_seqfile_show+0x50/0xd0 >[6133323.557386] kernfs_seq_show+0x30/0x38 >[6133323.557817] seq_read+0x148/0x478 >[6133323.558135] kernfs_fop_read+0x2c/0x1e0 >[6133323.558506] __vfs_read+0x20/0x48 >[6133323.558906] vfs_read+0x98/0x168 >[6133323.559220] ksys_read+0x6c/0xe8 >[6133323.559545] __arm64_sys_read+0x20/0x28 >[6133323.559981] el0_svc_common.constprop.0+0xa8/0x200 >[6133323.560582] el0_svc_handler+0x30/0x80 >[6133323.560948] el0_svc+0x10/0x14 >[6133363.531089] watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [kubelet:19532] >[6133363.531907] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6133363.538497] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6133363.544443] CPU: 4 PID: 19532 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6133363.545418] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6133363.546273] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6133363.546740] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6133363.547263] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6133363.547778] sp : ffff00001e943c20 >[6133363.548109] x29: ffff00001e943c20 x28: ffff8d8cedad6000 >[6133363.548634] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6133363.549141] x25: 0000000000000003 x24: 0000000000000000 >[6133363.549670] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6133363.550276] x21: ffff8d9c7bca4800 x20: ffff59fd7fe4b8a8 >[6133363.550825] x19: 0000000000000000 x18: 0000000000000000 >[6133363.551324] x17: 0000000000000002 x16: 0000000000000000 >[6133363.551824] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6133363.552329] x13: ffff8b7d1b324000 x12: ffff8b7d1b323300 >[6133363.552866] x11: 0000000000000000 x10: ffff00001e943c80 >[6133363.553357] x9 : ffff59fd7ef3dad0 x8 : 0000000000014b8b >[6133363.553850] x7 : 0000000000000004 x6 : 0000000000000005 >[6133363.554332] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6133363.554820] x3 : 000000000000003e x2 : 000033d01ff1b000 >[6133363.555318] x1 : ffff4a2dd1974c00 x0 : 000000000000007e >[6133363.555816] Call trace: >[6133363.556058] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6133363.556536] memcg_numa_stat_show+0x1d0/0x260 >[6133363.556963] cgroup_seqfile_show+0x50/0xd0 >[6133363.557366] kernfs_seq_show+0x30/0x38 >[6133363.557724] seq_read+0x148/0x478 >[6133363.558040] kernfs_fop_read+0x2c/0x1e0 >[6133363.558407] __vfs_read+0x20/0x48 >[6133363.558729] vfs_read+0x98/0x168 >[6133363.559039] ksys_read+0x6c/0xe8 >[6133363.559366] __arm64_sys_read+0x20/0x28 >[6133363.559754] el0_svc_common.constprop.0+0xa8/0x200 >[6133363.560321] el0_svc_handler+0x30/0x80 >[6133363.560748] el0_svc+0x10/0x14 >[6133407.531228] watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [kubelet:61155] >[6133407.532048] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6133407.538368] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6133407.544153] CPU: 13 PID: 61155 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6133407.545180] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6133407.546059] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6133407.546504] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6133407.547064] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6133407.547660] sp : ffff000059dbbc20 >[6133407.547990] x29: ffff000059dbbc20 x28: ffff8b8bd612c000 >[6133407.548514] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6133407.549087] x25: 0000000000000003 x24: 0000000000000000 >[6133407.549577] x23: 0000000000000008 x22: ffff59fd7fe49800 >[6133407.550168] x21: ffff8dccd5919c00 x20: ffff59fd7fe4b8a8 >[6133407.550665] x19: 0000000000000001 x18: 0000000000000000 >[6133407.551153] x17: 0000000000000002 x16: 0000000000000000 >[6133407.551654] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6133407.552199] x13: ffff8b8d7341b000 x12: ffff8b8d7341a31f >[6133407.552773] x11: 0000000000000000 x10: ffff000059dbbc80 >[6133407.553257] x9 : ffff59fd7ef3dad0 x8 : 00000000000185ee >[6133407.553812] x7 : 0000000000000004 x6 : 0000000000000005 >[6133407.554298] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6133407.554787] x3 : 0000000000000006 x2 : 00003380404a5000 >[6133407.555263] x1 : ffff4a0ceab7c3d8 x0 : 0000000000000046 >[6133407.555826] Call trace: >[6133407.556141] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6133407.556648] memcg_numa_stat_show+0x1d0/0x260 >[6133407.557065] cgroup_seqfile_show+0x50/0xd0 >[6133407.557457] kernfs_seq_show+0x30/0x38 >[6133407.557796] seq_read+0x148/0x478 >[6133407.558124] kernfs_fop_read+0x2c/0x1e0 >[6133407.558459] __vfs_read+0x20/0x48 >[6133407.558788] vfs_read+0x98/0x168 >[6133407.559081] ksys_read+0x6c/0xe8 >[6133407.559405] __arm64_sys_read+0x20/0x28 >[6133407.559773] el0_svc_common.constprop.0+0xa8/0x200 >[6133407.560348] el0_svc_handler+0x30/0x80 >[6133407.560720] el0_svc+0x10/0x14 >[6133451.531026] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kubelet:18929] >[6133451.531855] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6133451.538567] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6133451.544613] CPU: 2 PID: 18929 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6133451.545615] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6133451.546508] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6133451.546982] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6133451.547510] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6133451.548025] sp : ffff00003001bc20 >[6133451.548365] x29: ffff00003001bc20 x28: ffff8b7d6043c000 >[6133451.548893] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6133451.549457] x25: 0000000000000010 x24: 0000000000000000 >[6133451.550052] x23: 0000000000000020 x22: ffff59fd7fe49800 >[6133451.550594] x21: ffff8b8d2aa41800 x20: ffff59fd7fe4b8a8 >[6133451.551158] x19: 0000000000000004 x18: 0000000000000000 >[6133451.551700] x17: 0000000000000008 x16: 0000000000000000 >[6133451.552256] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6133451.552789] x13: ffff8b7ba68f9000 x12: ffff8b7ba68f834b >[6133451.553287] x11: 0000000000000000 x10: ffff00003001bc80 >[6133451.553811] x9 : ffff59fd7ef3dad0 x8 : ffff8b7ba68f8000 >[6133451.554342] x7 : 0000000000000000 x6 : 0000000000000001 >[6133451.554824] x5 : 00000000fffffffb x4 : ffffffffffffffff >[6133451.555317] x3 : 000000000000001f x2 : 000031b0404c0000 >[6133451.555832] x1 : ffff4a2f61670c10 x0 : 000000000000001f >[6133451.556344] Call trace: >[6133451.556611] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6133451.557079] memcg_numa_stat_show+0x1d0/0x260 >[6133451.557488] cgroup_seqfile_show+0x50/0xd0 >[6133451.557889] kernfs_seq_show+0x30/0x38 >[6133451.558264] seq_read+0x148/0x478 >[6133451.558596] kernfs_fop_read+0x2c/0x1e0 >[6133451.558977] __vfs_read+0x20/0x48 >[6133451.559309] vfs_read+0x98/0x168 >[6133451.559726] ksys_read+0x6c/0xe8 >[6133451.560067] __arm64_sys_read+0x20/0x28 >[6133451.560451] el0_svc_common.constprop.0+0xa8/0x200 >[6133451.560957] el0_svc_handler+0x30/0x80 >[6133451.561315] el0_svc+0x10/0x14 >[6133455.531147] watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [migration/8:51] >[6133455.531966] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6133455.538503] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6133455.544286] CPU: 8 PID: 51 Comm: migration/8 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6133455.545271] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6133455.546133] pstate: 60000005 (nZCv daif -PAN -UAO) >[6133455.546583] pc : multi_cpu_stop+0x78/0x180 >[6133455.546988] lr : multi_cpu_stop+0xb4/0x180 >[6133455.547389] sp : ffff000008e73d50 >[6133455.547704] x29: ffff000008e73d50 x28: 0000000000000000 >[6133455.548215] x27: ffff00001be93938 x26: 0000000000000060 >[6133455.548734] x25: 0000000000000000 x24: 0000000000000000 >[6133455.549247] x23: 0000000000000000 x22: 0000000000000001 >[6133455.549894] x21: ffff00001be939d4 x20: ffff00001be939b0 >[6133455.550456] x19: 0000000000000001 x18: 0000000000000000 >[6133455.550950] x17: 0000000000000001 x16: 0000000000000000 >[6133455.551537] x15: 0000000000000000 x14: 0000000000000000 >[6133455.552034] x13: 0000000000000000 x12: 0000000000000000 >[6133455.552555] x11: 0000000000000000 x10: 0000000000000d60 >[6133455.553041] x9 : ffff59fd7ede5a0c x8 : ffff8b8d9e8d6400 >[6133455.553530] x7 : ffff8b8d721b9400 x6 : 000000000000033a >[6133455.554019] x5 : 0000319040411000 x4 : ffff8b8d9e8d5c40 >[6133455.554561] x3 : dead000000000200 x2 : ffff00001be93988 >[6133455.555117] x1 : ffff00001be939d4 x0 : 0000000000000001 >[6133455.555655] Call trace: >[6133455.555953] multi_cpu_stop+0x78/0x180 >[6133455.556326] cpu_stopper_thread+0xec/0x178 >[6133455.556731] smpboot_thread_fn+0x158/0x1d8 >[6133455.557110] kthread+0x130/0x138 >[6133491.531136] watchdog: BUG: soft lockup - CPU#7 stuck for 23s! [kubelet:61155] >[6133491.531718] watchdog: BUG: soft lockup - CPU#45 stuck for 23s! [migration/45:241] >[6133491.532017] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6133491.532719] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6133491.539211] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6133491.545528] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6133491.551386] CPU: 7 PID: 61155 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6133491.556773] CPU: 45 PID: 241 Comm: migration/45 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6133491.557753] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6133491.558775] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6133491.559744] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6133491.560591] pstate: 60000005 (nZCv daif -PAN -UAO) >[6133491.561057] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6133491.561522] pc : multi_cpu_stop+0x78/0x180 >[6133491.562027] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6133491.562029] sp : ffff000059dbbc20 >[6133491.562403] lr : multi_cpu_stop+0xb4/0x180 >[6133491.562912] x29: ffff000059dbbc20 x28: ffff8bac1437c000 >[6133491.563256] sp : ffff00000a46bd50 >[6133491.563652] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6133491.564156] x29: ffff00000a46bd50 x28: 0000000000000000 >[6133491.564493] x25: 0000000000000003 x24: 0000000000000000 >[6133491.565027] x27: ffff00001df63938 x26: 0000000000000060 >[6133491.565534] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6133491.566062] x25: 0000000000000000 x24: 0000000000000000 >[6133491.566589] x21: ffff8d9c6043f800 x20: ffff59fd7fe4b8a8 >[6133491.567052] x23: 0000000000000000 x22: 0000000000000001 >[6133491.567560] x19: 0000000000000000 x18: 0000000000000000 >[6133491.568051] x21: ffff00001df639d4 x20: ffff00001df639b0 >[6133491.568590] x17: 0000000000000002 x16: 0000000000000000 >[6133491.569090] x19: 0000000000000001 x18: 0000000000000000 >[6133491.569689] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6133491.570231] x17: 0000000000000001 x16: 0000000000000000 >[6133491.570741] x13: ffff8b7d1ce69000 x12: ffff8b7d1ce682fc >[6133491.571250] x15: 0000000000000000 x14: 0000000000000000 >[6133491.571760] x11: 0000000000000000 x10: ffff000059dbbc80 >[6133491.572248] x13: 0000000000000000 x12: 0000000000000000 >[6133491.572790] x9 : ffff59fd7ef3dad0 x8 : 0000000000015070 >[6133491.573301] x11: 0000000000000085 x10: 0000000000000d60 >[6133491.573811] x7 : 0000000000000004 x6 : 0000000000000005 >[6133491.574329] x9 : ffff59fd7ede5a0c x8 : ffff8bcd9e990dc0 >[6133491.574854] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6133491.575363] x7 : 000125573fcef290 x6 : 000000000000032d >[6133491.575901] x3 : 0000000000000021 x2 : 000031c040429000 >[6133491.576392] x5 : 00000000f5257d14 x4 : ffff8bcd9e990600 >[6133491.576923] x1 : ffff4a1dd861cce0 x0 : 0000000000000021 >[6133491.577409] x3 : dead000000000200 x2 : ffff00001df63988 >[6133491.577912] Call trace: >[6133491.578396] x1 : ffff00001df639d4 x0 : 0000000000000001 >[6133491.578901] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6133491.579135] Call trace: >[6133491.579790] memcg_numa_stat_show+0x1d0/0x260 >[6133491.580330] multi_cpu_stop+0x78/0x180 >[6133491.580590] cgroup_seqfile_show+0x50/0xd0 >[6133491.580990] cpu_stopper_thread+0xec/0x178 >[6133491.581348] kernfs_seq_show+0x30/0x38 >[6133491.581798] smpboot_thread_fn+0x158/0x1d8 >[6133491.582274] seq_read+0x148/0x478 >[6133491.582642] kthread+0x130/0x138 >[6133491.583037] kernfs_fop_read+0x2c/0x1e0 >[6133491.584192] __vfs_read+0x20/0x48 >[6133491.584572] vfs_read+0x98/0x168 >[6133491.584909] ksys_read+0x6c/0xe8 >[6133491.585252] __arm64_sys_read+0x20/0x28 >[6133491.585647] el0_svc_common.constprop.0+0xa8/0x200 >[6133491.586174] el0_svc_handler+0x30/0x80 >[6133491.586547] el0_svc+0x10/0x14 >[6133535.531136] watchdog: BUG: soft lockup - CPU#7 stuck for 23s! [kubelet:18291] >[6133535.531977] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6133535.538715] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6133535.544701] CPU: 7 PID: 18291 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6133535.545712] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6133535.546586] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6133535.547059] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6133535.547566] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6133535.548072] sp : ffff00001f8ebc20 >[6133535.548410] x29: ffff00001f8ebc20 x28: ffff8dac06bdc000 >[6133535.548935] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6133535.549584] x25: 0000000000000003 x24: 0000000000000000 >[6133535.550097] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6133535.550608] x21: ffff8dbbbd0afc00 x20: ffff59fd7fe4b8a8 >[6133535.551116] x19: 0000000000000000 x18: 0000000000000000 >[6133535.551624] x17: 0000000000000002 x16: 0000000000000000 >[6133535.552157] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6133535.552689] x13: ffff8b7d1ce6a000 x12: ffff8b7d1ce69312 >[6133535.553213] x11: 0000000000000000 x10: ffff00001f8ebc80 >[6133535.553780] x9 : ffff59fd7ef3dad0 x8 : 000000000004f17a >[6133535.554272] x7 : 0000000000000005 x6 : 0000000000000006 >[6133535.554779] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6133535.555265] x3 : 0000000000000036 x2 : 000033d040301000 >[6133535.555760] x1 : ffff4a1dccf404d0 x0 : 0000000000000036 >[6133535.556296] Call trace: >[6133535.556556] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6133535.557093] memcg_numa_stat_show+0x1d0/0x260 >[6133535.557531] cgroup_seqfile_show+0x50/0xd0 >[6133535.557974] kernfs_seq_show+0x30/0x38 >[6133535.558347] seq_read+0x148/0x478 >[6133535.558679] kernfs_fop_read+0x2c/0x1e0 >[6133535.559164] __vfs_read+0x20/0x48 >[6133535.559493] vfs_read+0x98/0x168 >[6133535.559815] ksys_read+0x6c/0xe8 >[6133535.560129] __arm64_sys_read+0x20/0x28 >[6133535.560549] el0_svc_common.constprop.0+0xa8/0x200 >[6133535.561014] el0_svc_handler+0x30/0x80 >[6133535.561389] el0_svc+0x10/0x14 >[6133579.531016] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kubelet:55652] >[6133579.531271] watchdog: BUG: soft lockup - CPU#16 stuck for 23s! [migration/16:92] >[6133579.531846] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6133579.532582] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6133579.539209] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6133579.545645] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6133579.551578] CPU: 1 PID: 55652 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6133579.557034] CPU: 16 PID: 92 Comm: migration/16 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6133579.558007] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6133579.559117] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6133579.560028] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6133579.560903] pstate: 60000005 (nZCv daif -PAN -UAO) >[6133579.561423] pc : mem_cgroup_nr_lru_pages.isra.7+0x9c/0xc8 >[6133579.561885] pc : multi_cpu_stop+0x78/0x180 >[6133579.561888] lr : multi_cpu_stop+0xb4/0x180 >[6133579.562463] lr : mem_cgroup_nr_lru_pages.isra.7+0xac/0xc8 >[6133579.562836] sp : ffff000008fbbd50 >[6133579.563332] sp : ffff000056e73c20 >[6133579.563805] x29: ffff000008fbbd50 x28: 0000000000000000 >[6133579.564110] x29: ffff000056e73c20 x28: 0000000000000000 >[6133579.564409] x27: ffff0000545c3938 x26: 0000000000000060 >[6133579.564891] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6133579.565381] x25: 0000000000000000 x24: 0000000000000000 >[6133579.565915] x25: 0000000000000010 x24: 0000000000000000 >[6133579.566446] x23: 0000000000000000 x22: 0000000000000001 >[6133579.567024] x23: ffff8b9ca90705d0 x22: 0000000000000020 >[6133579.567506] x21: ffff0000545c39d4 x20: ffff0000545c39b0 >[6133579.568054] x21: ffff59fd7fe49800 x20: ffff59fd7fe4b8a8 >[6133579.568563] x19: 0000000000000001 x18: 0000000000000000 >[6133579.569179] x19: 0000000000000004 x18: 0000000000000000 >[6133579.569692] x17: 0000000000000001 x16: 0000000000000000 >[6133579.570216] x17: 0000000000000002 x16: 0000000000000000 >[6133579.570738] x15: 34342d333636342d x14: 6165393762616263 >[6133579.571239] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6133579.571743] x13: 242a0022001a0012 x12: 686769682d64616f >[6133579.572303] x13: ffff8b7d1ada4000 x12: ffff8b7d1ada3328 >[6133579.572819] x11: 6c6b726f770d0a04 x10: 0000000000000d60 >[6133579.573380] x11: 0000000000000000 x10: ffff000056e73c80 >[6133579.573854] x9 : ffff59fd7ede5a0c x8 : ffff8b9d9e946400 >[6133579.574338] x9 : ffff59fd7ef3da4c x8 : 000000000000dd41 >[6133579.574854] x7 : 001ee379f6a36968 x6 : 00000000000002f1 >[6133579.575353] x7 : 0000000000000004 x6 : 0000000000000005 >[6133579.575848] x5 : 00000000f5257d14 x4 : ffff8b9d9e945c40 >[6133579.576389] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6133579.576886] x3 : dead000000000200 x2 : ffff0000545c3988 >[6133579.577420] x3 : 0000000000000013 x2 : 000031a04045c000 >[6133579.577885] x1 : ffff0000545c39d4 x0 : 0000000000000001 >[6133579.578426] x1 : ffff4a1f619a8690 x0 : 0000000000000013 >[6133579.579047] Call trace: >[6133579.579638] Call trace: >[6133579.580121] multi_cpu_stop+0x78/0x180 >[6133579.580407] mem_cgroup_nr_lru_pages.isra.7+0x9c/0xc8 >[6133579.580652] cpu_stopper_thread+0xec/0x178 >[6133579.581045] memcg_numa_stat_show+0x14c/0x260 >[6133579.581529] smpboot_thread_fn+0x158/0x1d8 >[6133579.582007] cgroup_seqfile_show+0x50/0xd0 >[6133579.582395] kthread+0x130/0x138 >[6133579.582905] kernfs_seq_show+0x30/0x38 >[6133579.583963] seq_read+0x148/0x478 >[6133579.584304] kernfs_fop_read+0x2c/0x1e0 >[6133579.584684] __vfs_read+0x20/0x48 >[6133579.585024] vfs_read+0x98/0x168 >[6133579.585350] ksys_read+0x6c/0xe8 >[6133579.585665] __arm64_sys_read+0x20/0x28 >[6133579.586043] el0_svc_common.constprop.0+0xa8/0x200 >[6133579.586504] el0_svc_handler+0x30/0x80 >[6133579.586864] el0_svc+0x10/0x14 >[6133619.531859] watchdog: BUG: soft lockup - CPU#52 stuck for 23s! [kubelet:18889] >[6133619.532773] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6133619.539192] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6133619.544639] CPU: 52 PID: 18889 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6133619.545607] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6133619.546451] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6133619.546886] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6133619.547367] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6133619.547847] sp : ffff0000300f3c20 >[6133619.548159] x29: ffff0000300f3c20 x28: ffff8d8cee41e000 >[6133619.548746] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6133619.549229] x25: 0000000000000003 x24: 0000000000000000 >[6133619.549709] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6133619.550227] x21: ffff8bcc4c0f3800 x20: ffff59fd7fe4b8a8 >[6133619.550716] x19: 0000000000000000 x18: 0000000000000000 >[6133619.551282] x17: 0000000000000002 x16: 0000000000000000 >[6133619.551769] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6133619.552293] x13: ffff8b9d1d4b5000 x12: ffff8b9d1d4b42e0 >[6133619.552789] x11: 0000000000000000 x10: ffff0000300f3c80 >[6133619.553260] x9 : ffff59fd7ef3dad0 x8 : 000000000002e423 >[6133619.553733] x7 : 0000000000000005 x6 : 0000000000000006 >[6133619.554211] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6133619.554703] x3 : 0000000000000036 x2 : 000033d01ffe3000 >[6133619.555213] x1 : ffff4a2dd3338350 x0 : 0000000000000076 >[6133619.555729] Call trace: >[6133619.555974] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6133619.556473] memcg_numa_stat_show+0x1d0/0x260 >[6133619.556905] cgroup_seqfile_show+0x50/0xd0 >[6133619.557360] kernfs_seq_show+0x30/0x38 >[6133619.557767] seq_read+0x148/0x478 >[6133619.558103] kernfs_fop_read+0x2c/0x1e0 >[6133619.558628] __vfs_read+0x20/0x48 >[6133619.558992] vfs_read+0x98/0x168 >[6133619.559311] ksys_read+0x6c/0xe8 >[6133619.559619] __arm64_sys_read+0x20/0x28 >[6133619.559970] el0_svc_common.constprop.0+0xa8/0x200 >[6133619.560406] el0_svc_handler+0x30/0x80 >[6133619.560750] el0_svc+0x10/0x14 >[6133663.531069] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [migration/3:26] >[6133663.531754] watchdog: BUG: soft lockup - CPU#47 stuck for 23s! [kubelet:19531] >[6133663.531942] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6133663.532591] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6133663.539224] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6133663.546027] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6133663.551961] CPU: 3 PID: 26 Comm: migration/3 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6133663.557556] CPU: 47 PID: 19531 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6133663.558634] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6133663.559583] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6133663.560474] pstate: 60000005 (nZCv daif -PAN -UAO) >[6133663.561346] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6133663.561793] pc : multi_cpu_stop+0x78/0x180 >[6133663.562246] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6133663.562640] lr : multi_cpu_stop+0xb4/0x180 >[6133663.563119] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6133663.563544] sp : ffff000008dabd50 >[6133663.564026] sp : ffff00001fd4bc20 >[6133663.564423] x29: ffff000008dabd50 x28: 0000000000000000 >[6133663.564759] x29: ffff00001fd4bc20 x28: ffff8dabe99ba000 >[6133663.565299] x27: ffff00003dc03938 x26: 0000000000000060 >[6133663.565798] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6133663.566372] x25: 0000000000000000 x24: 0000000000000000 >[6133663.566906] x25: 0000000000000003 x24: 0000000000000000 >[6133663.567481] x23: 0000000000000000 x22: 0000000000000001 >[6133663.567940] x23: 0000000000000008 x22: ffff59fd7fe49800 >[6133663.568598] x21: ffff00003dc039d4 x20: ffff00003dc039b0 >[6133663.569103] x21: ffff8d7c726b5800 x20: ffff59fd7fe4b8a8 >[6133663.569652] x19: 0000000000000001 x18: 0000000000000000 >[6133663.570188] x19: 0000000000000001 x18: 0000000000000000 >[6133663.570733] x17: 0000000000000001 x16: 0000000000000000 >[6133663.571230] x17: 0000000000000002 x16: 0000000000000000 >[6133663.571786] x15: 0000000000000000 x14: 0000000000000000 >[6133663.572260] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6133663.572863] x13: 0000000000000000 x12: 0000000000000000 >[6133663.573350] x13: ffff8bcd52fe0000 x12: ffff8bcd52fdf2ea >[6133663.573864] x11: 0000000000000000 x10: 0000000000000d60 >[6133663.574408] x11: 0000000000000000 x10: ffff00001fd4bc80 >[6133663.575005] x9 : ffff59fd7ede5a0c x8 : ffff8b7d9da18dc0 >[6133663.575500] x9 : ffff59fd7ef3dad0 x8 : 0000000000020d70 >[6133663.576061] x7 : ffff8b7d6b1b3000 x6 : 000000000000014b >[6133663.576556] x7 : 0000000000000005 x6 : 0000000000000006 >[6133663.577084] x5 : 000031804045a000 x4 : ffff8b7d9da18600 >[6133663.577615] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6133663.578287] x3 : dead000000000200 x2 : ffff00003dc03960 >[6133663.578803] x3 : 000000000000001d x2 : 000031b04048e000 >[6133663.579350] x1 : ffff00003dc039d4 x0 : 0000000000000001 >[6133663.579857] x1 : ffff4a0f1ef8ba58 x0 : 000000000000001d >[6133663.580360] Call trace: >[6133663.580851] Call trace: >[6133663.581095] multi_cpu_stop+0x78/0x180 >[6133663.581325] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6133663.581684] cpu_stopper_thread+0xec/0x178 >[6133663.582149] memcg_numa_stat_show+0x1d0/0x260 >[6133663.582540] smpboot_thread_fn+0x158/0x1d8 >[6133663.582962] cgroup_seqfile_show+0x50/0xd0 >[6133663.583357] kthread+0x130/0x138 >[6133663.583747] kernfs_seq_show+0x30/0x38 >[6133663.583751] seq_read+0x148/0x478 >[6133663.584801] kernfs_fop_read+0x2c/0x1e0 >[6133663.585163] __vfs_read+0x20/0x48 >[6133663.585511] vfs_read+0x98/0x168 >[6133663.585915] ksys_read+0x6c/0xe8 >[6133663.586224] __arm64_sys_read+0x20/0x28 >[6133663.586589] el0_svc_common.constprop.0+0xa8/0x200 >[6133663.587130] el0_svc_handler+0x30/0x80 >[6133663.587522] el0_svc+0x10/0x14 >[6133707.531227] watchdog: BUG: soft lockup - CPU#13 stuck for 23s! [kubelet:18321] >[6133707.532044] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6133707.538370] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6133707.544169] CPU: 13 PID: 18321 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6133707.545148] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6133707.546067] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6133707.546531] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6133707.547063] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6133707.547555] sp : ffff00001f953c20 >[6133707.547996] x29: ffff00001f953c20 x28: ffff8bcc7c3ca000 >[6133707.548563] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6133707.549055] x25: 0000000000000003 x24: 0000000000000000 >[6133707.549530] x23: 0000000000000008 x22: ffff59fd7fe49800 >[6133707.550099] x21: ffff8dac7e78ac00 x20: ffff59fd7fe4b8a8 >[6133707.550609] x19: 0000000000000001 x18: 0000000000000000 >[6133707.551166] x17: 0000000000000002 x16: 0000000000000000 >[6133707.551670] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6133707.552175] x13: ffff8b8d7341a000 x12: ffff8b8d73419308 >[6133707.552748] x11: 0000000000000000 x10: ffff00001f953c80 >[6133707.553264] x9 : ffff59fd7ef3dad0 x8 : 000000000000c36f >[6133707.553748] x7 : 0000000000000004 x6 : 0000000000000005 >[6133707.554230] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6133707.554739] x3 : 0000000000000015 x2 : 000033a04048e000 >[6133707.555265] x1 : ffff4a1cea5f5798 x0 : 0000000000000055 >[6133707.555774] Call trace: >[6133707.556030] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6133707.556523] memcg_numa_stat_show+0x1d0/0x260 >[6133707.556941] cgroup_seqfile_show+0x50/0xd0 >[6133707.557322] kernfs_seq_show+0x30/0x38 >[6133707.557682] seq_read+0x148/0x478 >[6133707.558151] kernfs_fop_read+0x2c/0x1e0 >[6133707.558535] __vfs_read+0x20/0x48 >[6133707.558905] vfs_read+0x98/0x168 >[6133707.559214] ksys_read+0x6c/0xe8 >[6133707.559534] __arm64_sys_read+0x20/0x28 >[6133707.559917] el0_svc_common.constprop.0+0xa8/0x200 >[6133707.560368] el0_svc_handler+0x30/0x80 >[6133707.560735] el0_svc+0x10/0x14 >[6133751.531121] watchdog: BUG: soft lockup - CPU#6 stuck for 23s! [kubelet:11759] >[6133751.531909] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6133751.538565] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6133751.544220] CPU: 6 PID: 11759 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6133751.545223] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6133751.546122] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6133751.546588] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6133751.547109] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6133751.547710] sp : ffff00005541bc20 >[6133751.548072] x29: ffff00005541bc20 x28: ffff8b9c4309a000 >[6133751.548639] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6133751.549172] x25: 0000000000000003 x24: 0000000000000000 >[6133751.549705] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6133751.550196] x21: ffff8dcd30f18000 x20: ffff59fd7fe4b8a8 >[6133751.550763] x19: 0000000000000000 x18: 0000000000000000 >[6133751.551364] x17: 0000000000000002 x16: 0000000000000000 >[6133751.551868] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6133751.552517] x13: ffff8b7d1c80d000 x12: ffff8b7d1c80c31d >[6133751.553074] x11: 0000000000000000 x10: ffff00005541bc80 >[6133751.553627] x9 : ffff59fd7ef3dad0 x8 : 0000000000018e47 >[6133751.554124] x7 : 0000000000000005 x6 : 0000000000000006 >[6133751.554751] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6133751.555291] x3 : 000000000000003b x2 : 000033d0401ee000 >[6133751.555893] x1 : ffff4a1f12e7f130 x0 : 000000000000003b >[6133751.556464] Call trace: >[6133751.556748] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6133751.557281] memcg_numa_stat_show+0x1d0/0x260 >[6133751.557786] cgroup_seqfile_show+0x50/0xd0 >[6133751.558282] kernfs_seq_show+0x30/0x38 >[6133751.558698] seq_read+0x148/0x478 >[6133751.559021] kernfs_fop_read+0x2c/0x1e0 >[6133751.559426] __vfs_read+0x20/0x48 >[6133751.559785] vfs_read+0x98/0x168 >[6133751.560134] ksys_read+0x6c/0xe8 >[6133751.560446] __arm64_sys_read+0x20/0x28 >[6133751.560825] el0_svc_common.constprop.0+0xa8/0x200 >[6133751.561419] el0_svc_handler+0x30/0x80 >[6133751.561849] el0_svc+0x10/0x14 >[6133795.531229] watchdog: BUG: soft lockup - CPU#13 stuck for 23s! [kubelet:18612] >[6133795.532004] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6133795.539446] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6133795.545481] CPU: 13 PID: 18612 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6133795.546439] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6133795.547398] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6133795.547894] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6133795.548392] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6133795.548909] sp : ffff000050243c20 >[6133795.549286] x29: ffff000050243c20 x28: ffff8bbb63f5e000 >[6133795.549780] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6133795.550276] x25: 0000000000000010 x24: 0000000000000000 >[6133795.550798] x23: 0000000000000020 x22: ffff59fd7fe49800 >[6133795.551295] x21: ffff8b9b67b7fc00 x20: ffff59fd7fe4b8a8 >[6133795.551814] x19: 0000000000000004 x18: 0000000000000000 >[6133795.552323] x17: 000000000000063a x16: 0000000000000000 >[6133795.552841] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6133795.553329] x13: ffff8b8d73419000 x12: ffff8b8d73418352 >[6133795.553836] x11: 0000000000000000 x10: ffff000050243c80 >[6133795.554341] x9 : ffff59fd7ef3dad0 x8 : ffff8b8d73418000 >[6133795.554850] x7 : 0000000000000000 x6 : 0000000000000001 >[6133795.555341] x5 : 00000000fffffffb x4 : ffffffffffffffff >[6133795.555859] x3 : 0000000000000016 x2 : 000031a0404a7000 >[6133795.556346] x1 : ffff4a1f24571ac0 x0 : 0000000000000016 >[6133795.556849] Call trace: >[6133795.557105] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6133795.557722] memcg_numa_stat_show+0x1d0/0x260 >[6133795.558145] cgroup_seqfile_show+0x50/0xd0 >[6133795.558572] kernfs_seq_show+0x30/0x38 >[6133795.558927] seq_read+0x148/0x478 >[6133795.559243] kernfs_fop_read+0x2c/0x1e0 >[6133795.559653] __vfs_read+0x20/0x48 >[6133795.559971] vfs_read+0x98/0x168 >[6133795.560314] ksys_read+0x6c/0xe8 >[6133795.560674] __arm64_sys_read+0x20/0x28 >[6133795.561039] el0_svc_common.constprop.0+0xa8/0x200 >[6133795.561514] el0_svc_handler+0x30/0x80 >[6133795.561846] el0_svc+0x10/0x14 >[6133835.531279] watchdog: BUG: soft lockup - CPU#16 stuck for 22s! [kubelet:11266] >[6133835.531996] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6133835.538445] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6133835.543856] CPU: 16 PID: 11266 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6133835.544812] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6133835.545657] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6133835.546094] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6133835.546581] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6133835.547174] sp : ffff0000545dbc20 >[6133835.547475] x29: ffff0000545dbc20 x28: ffff8bacc3472000 >[6133835.547999] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6133835.548515] x25: 0000000000000003 x24: 0000000000000000 >[6133835.548992] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6133835.549481] x21: ffff8d9ce7cf8800 x20: ffff59fd7fe4b8a8 >[6133835.549961] x19: 0000000000000000 x18: 0000000000000000 >[6133835.550438] x17: 0000000000000002 x16: 0000000000000000 >[6133835.550920] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6133835.551398] x13: ffff8b9ba2453000 x12: ffff8b9ba24522ff >[6133835.551957] x11: 0000000000000000 x10: ffff0000545dbc80 >[6133835.552443] x9 : ffff59fd7ef3dad0 x8 : 0000000000014ba2 >[6133835.552947] x7 : 0000000000000004 x6 : 0000000000000005 >[6133835.553440] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6133835.553941] x3 : 000000000000002b x2 : 000033d020060000 >[6133835.554406] x1 : ffff4a1d02656990 x0 : 000000000000006b >[6133835.554877] Call trace: >[6133835.555126] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6133835.555622] memcg_numa_stat_show+0x1d0/0x260 >[6133835.556024] cgroup_seqfile_show+0x50/0xd0 >[6133835.556480] kernfs_seq_show+0x30/0x38 >[6133835.556839] seq_read+0x148/0x478 >[6133835.557223] kernfs_fop_read+0x2c/0x1e0 >[6133835.557596] __vfs_read+0x20/0x48 >[6133835.557939] vfs_read+0x98/0x168 >[6133835.558266] ksys_read+0x6c/0xe8 >[6133835.558572] __arm64_sys_read+0x20/0x28 >[6133835.558937] el0_svc_common.constprop.0+0xa8/0x200 >[6133835.559391] el0_svc_handler+0x30/0x80 >[6133835.559742] el0_svc+0x10/0x14 >[6133879.531018] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [migration/2:21] >[6133879.531182] watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [kubelet:11759] >[6133879.531855] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6133879.532553] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6133879.539110] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6133879.545827] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6133879.551612] CPU: 2 PID: 21 Comm: migration/2 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6133879.557667] CPU: 10 PID: 11759 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6133879.558641] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6133879.559561] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6133879.560419] pstate: 60000005 (nZCv daif -PAN -UAO) >[6133879.561276] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6133879.561739] pc : multi_cpu_stop+0x78/0x180 >[6133879.562200] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6133879.562607] lr : multi_cpu_stop+0xb4/0x180 >[6133879.563098] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6133879.563100] sp : ffff00005541bc20 >[6133879.563494] sp : ffff000008d83d50 >[6133879.564010] x29: ffff00005541bc20 x28: ffff8d9cd43d0000 >[6133879.564330] x29: ffff000008d83d50 x28: 0000000000000000 >[6133879.564664] x27: ffff59fd7fe4bbf4 x26: fffffffffffffc83 >[6133879.565176] x27: ffff00001bff3938 x26: 0000000000000060 >[6133879.565668] x25: 0000000000000003 x24: 0000000000000000 >[6133879.566251] x25: 0000000000000000 x24: 0000000000000000 >[6133879.566825] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6133879.567337] x23: 0000000000000000 x22: 0000000000000001 >[6133879.567852] x21: ffff8dcd23554c00 x20: ffff59fd7fe4b8a8 >[6133879.568361] x21: ffff00001bff39d4 x20: ffff00001bff39b0 >[6133879.568895] x19: 0000000000000000 x18: 0000000000000000 >[6133879.569440] x19: 0000000000000001 x18: 0000000000000000 >[6133879.569969] x17: 0000000000000009 x16: 0000000000000000 >[6133879.570532] x17: 0000000000000001 x16: 0000000000000000 >[6133879.571061] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6133879.571624] x15: 0000000000000000 x14: ffff8dbcb6681400 >[6133879.572135] x13: ffff8b8d72d51000 x12: ffff8b8d72d5031e >[6133879.572648] x13: 00000000000001a6 x12: ffff8dbcdef68d80 >[6133879.573191] x11: 0000000000000000 x10: ffff00005541bc80 >[6133879.573711] x11: 000000000000001e x10: 0000000000000d60 >[6133879.574228] x9 : ffff59fd7ef3dad0 x8 : 0000000000018d92 >[6133879.574737] x9 : ffff59fd7ede5a0c x8 : ffff8b7d9d9c3040 >[6133879.575298] x7 : 0000000000000005 x6 : 0000000000000006 >[6133879.575844] x7 : ffff8b7d6b1b2800 x6 : 0000000000000008 >[6133879.576355] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6133879.577003] x5 : 0000318040441000 x4 : ffff8b7d9d9c2880 >[6133879.577548] x3 : 000000000000001a x2 : 000033b040443000 >[6133879.578071] x3 : dead000000000200 x2 : ffff00001bff3960 >[6133879.578649] x1 : ffff4a1cf6625700 x0 : 000000000000005a >[6133879.579188] x1 : ffff00001bff39d4 x0 : 0000000000000001 >[6133879.579678] Call trace: >[6133879.580180] Call trace: >[6133879.580727] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6133879.581028] multi_cpu_stop+0x78/0x180 >[6133879.581273] memcg_numa_stat_show+0x1d0/0x260 >[6133879.581752] cpu_stopper_thread+0xec/0x178 >[6133879.582130] cgroup_seqfile_show+0x50/0xd0 >[6133879.582572] smpboot_thread_fn+0x158/0x1d8 >[6133879.582959] kernfs_seq_show+0x30/0x38 >[6133879.583361] kthread+0x130/0x138 >[6133879.583777] seq_read+0x148/0x478 >[6133879.584849] kernfs_fop_read+0x2c/0x1e0 >[6133879.585244] __vfs_read+0x20/0x48 >[6133879.585606] vfs_read+0x98/0x168 >[6133879.585900] ksys_read+0x6c/0xe8 >[6133879.586257] __arm64_sys_read+0x20/0x28 >[6133879.586745] el0_svc_common.constprop.0+0xa8/0x200 >[6133879.587256] el0_svc_handler+0x30/0x80 >[6133879.587649] el0_svc+0x10/0x14 >[6133923.531097] watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [migration/5:36] >[6133923.531151] watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [kubelet:18805] >[6133923.531961] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6133923.532632] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6133923.539165] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6133923.545578] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6133923.551461] CPU: 5 PID: 36 Comm: migration/5 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6133923.557312] CPU: 8 PID: 18805 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6133923.558378] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6133923.559349] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6133923.560196] pstate: 60000005 (nZCv daif -PAN -UAO) >[6133923.561025] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6133923.561509] pc : multi_cpu_stop+0xf4/0x180 >[6133923.562013] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6133923.562391] lr : multi_cpu_stop+0xb4/0x180 >[6133923.562393] sp : ffff000008dfbd50 >[6133923.562934] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6133923.562936] sp : ffff00001f30bc20 >[6133923.563350] x29: ffff000008dfbd50 x28: 0000000000000000 >[6133923.563737] x29: ffff00001f30bc20 x28: ffff8bbc5be08000 >[6133923.564284] x27: ffff00001bff3938 x26: 0000000000000060 >[6133923.564648] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6133923.565123] x25: 0000000000000000 x24: 0000000000000000 >[6133923.565715] x25: 0000000000000010 x24: 0000000000000000 >[6133923.566331] x23: 0000000000000000 x22: 0000000000000001 >[6133923.566830] x23: 0000000000000020 x22: ffff59fd7fe49800 >[6133923.567370] x21: ffff00001bff39d4 x20: ffff00001bff39b0 >[6133923.567864] x21: ffff8b8c3622f800 x20: ffff59fd7fe4b8a8 >[6133923.568414] x19: 0000000000000001 x18: 0000000000000000 >[6133923.568978] x19: 0000000000000004 x18: 0000000000000000 >[6133923.569460] x17: 0000000000000001 x16: 0000000000000000 >[6133923.569999] x17: 0000000000000002 x16: 0000000000000000 >[6133923.570502] x15: 642f737070612f3a x14: 5437414855505756 >[6133923.570984] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6133923.571512] x13: 4b583647334d365a x12: 4b51324d54505136 >[6133923.572012] x13: ffff8b8d72fad000 x12: ffff8b8d72fac34f >[6133923.572516] x11: 36492f6c2f327961 x10: 0000000000000d60 >[6133923.573016] x11: 0000000000000000 x10: ffff00001f30bc80 >[6133923.573524] x9 : ffff59fd7ede5a0c x8 : ffff8b7d9da24180 >[6133923.574014] x9 : ffff59fd7ef3dad0 x8 : ffff8b8d72fac000 >[6133923.574499] x7 : 0012582760a9eb68 x6 : 0000000000000148 >[6133923.575009] x7 : 0000000000000000 x6 : 0000000000000001 >[6133923.575530] x5 : 00000000f5257d14 x4 : ffff8b7d9da239c0 >[6133923.576041] x5 : 00000000fffffffb x4 : ffffffffffffffff >[6133923.576645] x3 : dead000000000200 x2 : ffff00001bff3960 >[6133923.577147] x3 : 000000000000001c x2 : 000033b040475000 >[6133923.577650] x1 : ffff00001bff39d4 x0 : 0000000000000001 >[6133923.578163] x1 : ffff4a14414d8950 x0 : 000000000000005c >[6133923.578667] Call trace: >[6133923.579228] Call trace: >[6133923.579738] multi_cpu_stop+0xf4/0x180 >[6133923.580248] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6133923.580530] cpu_stopper_thread+0xec/0x178 >[6133923.580782] memcg_numa_stat_show+0x1d0/0x260 >[6133923.581147] smpboot_thread_fn+0x158/0x1d8 >[6133923.581657] cgroup_seqfile_show+0x50/0xd0 >[6133923.582042] kthread+0x130/0x138 >[6133923.582456] kernfs_seq_show+0x30/0x38 >[6133923.582460] seq_read+0x148/0x478 >[6133923.584392] kernfs_fop_read+0x2c/0x1e0 >[6133923.584829] __vfs_read+0x20/0x48 >[6133923.585203] vfs_read+0x98/0x168 >[6133923.585536] ksys_read+0x6c/0xe8 >[6133923.585859] __arm64_sys_read+0x20/0x28 >[6133923.586348] el0_svc_common.constprop.0+0xa8/0x200 >[6133923.586852] el0_svc_handler+0x30/0x80 >[6133923.587271] el0_svc+0x10/0x14 >[6133963.531151] watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [kubelet:18321] >[6133963.531933] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6133963.538685] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6133963.544470] CPU: 8 PID: 18321 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6133963.545465] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6133963.546424] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6133963.546921] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6133963.547414] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6133963.547927] sp : ffff00001f953c20 >[6133963.548286] x29: ffff00001f953c20 x28: ffff8b7c1adda000 >[6133963.548807] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6133963.549377] x25: 0000000000000003 x24: 0000000000000000 >[6133963.549877] x23: 0000000000000008 x22: ffff59fd7fe49800 >[6133963.550344] x21: ffff8dacadb06800 x20: ffff59fd7fe4b8a8 >[6133963.550873] x19: 0000000000000001 x18: 0000000000000000 >[6133963.551335] x17: 0000000000000002 x16: 0000000000000000 >[6133963.551892] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6133963.552376] x13: ffff8b8d72faf000 x12: ffff8b8d72fae309 >[6133963.552891] x11: 0000000000000000 x10: ffff00001f953c80 >[6133963.553409] x9 : ffff59fd7ef3dad0 x8 : 000000000000c367 >[6133963.553914] x7 : 0000000000000004 x6 : 0000000000000005 >[6133963.554468] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6133963.554939] x3 : 0000000000000014 x2 : 000033a040475000 >[6133963.555477] x1 : ffff4a1f0b2421f8 x0 : 0000000000000054 >[6133963.556058] Call trace: >[6133963.556317] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6133963.556841] memcg_numa_stat_show+0x1d0/0x260 >[6133963.557265] cgroup_seqfile_show+0x50/0xd0 >[6133963.557657] kernfs_seq_show+0x30/0x38 >[6133963.558068] seq_read+0x148/0x478 >[6133963.558433] kernfs_fop_read+0x2c/0x1e0 >[6133963.558838] __vfs_read+0x20/0x48 >[6133963.559165] vfs_read+0x98/0x168 >[6133963.559479] ksys_read+0x6c/0xe8 >[6133963.559832] __arm64_sys_read+0x20/0x28 >[6133963.560224] el0_svc_common.constprop.0+0xa8/0x200 >[6133963.560733] el0_svc_handler+0x30/0x80 >[6133963.561058] el0_svc+0x10/0x14 >[6134007.531020] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [migration/2:21] >[6134007.531308] watchdog: BUG: soft lockup - CPU#18 stuck for 22s! [kubelet:61155] >[6134007.531833] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6134007.532542] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6134007.539041] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6134007.545317] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6134007.551296] CPU: 2 PID: 21 Comm: migration/2 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6134007.557073] CPU: 18 PID: 61155 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6134007.558186] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6134007.559232] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6134007.560155] pstate: 60000005 (nZCv daif -PAN -UAO) >[6134007.561069] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6134007.561568] pc : multi_cpu_stop+0xf4/0x180 >[6134007.562051] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6134007.562501] lr : multi_cpu_stop+0xb4/0x180 >[6134007.563008] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6134007.563010] sp : ffff000059dbbc20 >[6134007.563407] sp : ffff000008d83d50 >[6134007.563940] x29: ffff000059dbbc20 x28: ffff8b9b83d4a000 >[6134007.564257] x29: ffff000008d83d50 x28: 0000000000000000 >[6134007.564570] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6134007.565099] x27: ffff00001b313938 x26: 0000000000000060 >[6134007.565601] x25: 0000000000000003 x24: 0000000000000000 >[6134007.566220] x25: 0000000000000000 x24: 0000000000000000 >[6134007.566769] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6134007.567288] x23: 0000000000000000 x22: 0000000000000001 >[6134007.567849] x21: ffff8dbba9b10800 x20: ffff59fd7fe4b8a8 >[6134007.568396] x21: ffff00001b3139d4 x20: ffff00001b3139b0 >[6134007.568927] x19: 0000000000000000 x18: 0000000000000000 >[6134007.569420] x19: 0000000000000001 x18: 0000000000000000 >[6134007.569963] x17: 0000000000000002 x16: 0000000000000000 >[6134007.570528] x17: 0000000000000001 x16: 0000000000000000 >[6134007.571029] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6134007.571597] x15: 0000000000000000 x14: 0000000000000000 >[6134007.572128] x13: ffff8b9d205f2000 x12: ffff8b9d205f1313 >[6134007.572642] x13: 0000000000000000 x12: 0000000000000000 >[6134007.573141] x11: 0000000000000000 x10: ffff000059dbbc80 >[6134007.573656] x11: 0000000000000000 x10: 0000000000000d60 >[6134007.574148] x9 : ffff59fd7ef3dad0 x8 : 000000000004f12a >[6134007.574652] x9 : ffff59fd7ede5a0c x8 : ffff8b7d9d9c3040 >[6134007.575153] x7 : 0000000000000005 x6 : 0000000000000006 >[6134007.575728] x7 : ffff8b8d82f180c0 x6 : 00000000000001a1 >[6134007.576234] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6134007.576796] x5 : 0000318040441000 x4 : ffff8b7d9d9c2880 >[6134007.577284] x3 : 0000000000000030 x2 : 000033d01ff4d000 >[6134007.577839] x3 : dead000000000200 x2 : ffff00001b313960 >[6134007.578431] x1 : ffff4a0d01b78ca0 x0 : 0000000000000070 >[6134007.579012] x1 : ffff00001b3139d4 x0 : 0000000000000001 >[6134007.579548] Call trace: >[6134007.580081] Call trace: >[6134007.580605] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6134007.580852] multi_cpu_stop+0xf4/0x180 >[6134007.581100] memcg_numa_stat_show+0x1d0/0x260 >[6134007.581646] cpu_stopper_thread+0xec/0x178 >[6134007.582049] cgroup_seqfile_show+0x50/0xd0 >[6134007.582567] smpboot_thread_fn+0x158/0x1d8 >[6134007.582989] kernfs_seq_show+0x30/0x38 >[6134007.583441] kthread+0x130/0x138 >[6134007.583836] seq_read+0x148/0x478 >[6134007.584905] kernfs_fop_read+0x2c/0x1e0 >[6134007.585275] __vfs_read+0x20/0x48 >[6134007.585683] vfs_read+0x98/0x168 >[6134007.586003] ksys_read+0x6c/0xe8 >[6134007.586317] __arm64_sys_read+0x20/0x28 >[6134007.586683] el0_svc_common.constprop.0+0xa8/0x200 >[6134007.587148] el0_svc_handler+0x30/0x80 >[6134007.587498] el0_svc+0x10/0x14 >[6134051.531214] watchdog: BUG: soft lockup - CPU#12 stuck for 22s! [kubelet:61155] >[6134051.532057] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6134051.538840] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6134051.544719] CPU: 12 PID: 61155 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6134051.545888] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6134051.546765] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6134051.547277] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6134051.547833] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6134051.548374] sp : ffff000059dbbc20 >[6134051.548698] x29: ffff000059dbbc20 x28: ffff8bace36be000 >[6134051.549291] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6134051.549806] x25: 0000000000000010 x24: 0000000000000000 >[6134051.550419] x23: 0000000000000020 x22: ffff59fd7fe49800 >[6134051.550978] x21: ffff8b7d73839800 x20: ffff59fd7fe4b8a8 >[6134051.551498] x19: 0000000000000004 x18: 0000000000000000 >[6134051.552005] x17: 0000000000000002 x16: 0000000000000000 >[6134051.552576] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6134051.553097] x13: ffff8b8d7119c000 x12: ffff8b8d7119b348 >[6134051.553630] x11: 0000000000000000 x10: ffff000059dbbc80 >[6134051.554103] x9 : ffff59fd7ef3dad0 x8 : 0000000000000021 >[6134051.554601] x7 : 0000000000000001 x6 : 0000000000000002 >[6134051.555067] x5 : 00000000fffffffa x4 : ffffffffffffffff >[6134051.555684] x3 : 000000000000003f x2 : 000033d01ff34000 >[6134051.556220] x1 : ffff4a1f65b4a650 x0 : 000000000000007f >[6134051.556792] Call trace: >[6134051.557043] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6134051.557611] memcg_numa_stat_show+0x1d0/0x260 >[6134051.558036] cgroup_seqfile_show+0x50/0xd0 >[6134051.558503] kernfs_seq_show+0x30/0x38 >[6134051.558881] seq_read+0x148/0x478 >[6134051.559245] kernfs_fop_read+0x2c/0x1e0 >[6134051.559617] __vfs_read+0x20/0x48 >[6134051.559974] vfs_read+0x98/0x168 >[6134051.560296] ksys_read+0x6c/0xe8 >[6134051.560621] __arm64_sys_read+0x20/0x28 >[6134051.561063] el0_svc_common.constprop.0+0xa8/0x200 >[6134051.561583] el0_svc_handler+0x30/0x80 >[6134051.561979] el0_svc+0x10/0x14 >[6134091.531182] watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [kubelet:61155] >[6134091.531976] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6134091.539635] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6134091.546051] CPU: 10 PID: 61155 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6134091.547144] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6134091.548153] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6134091.548671] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6134091.549271] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6134091.549749] sp : ffff000059dbbc20 >[6134091.550176] x29: ffff000059dbbc20 x28: ffff8bac0c6b8000 >[6134091.550691] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6134091.551190] x25: 0000000000000003 x24: 0000000000000000 >[6134091.551721] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6134091.552221] x21: ffff8dac12ecb000 x20: ffff59fd7fe4b8a8 >[6134091.552754] x19: 0000000000000000 x18: 0000000000000000 >[6134091.553282] x17: 0000000000000002 x16: 0000000000000000 >[6134091.553763] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6134091.554269] x13: ffff8b8d72d52000 x12: ffff8b8d72d51308 >[6134091.554769] x11: 0000000000000000 x10: ffff000059dbbc80 >[6134091.555397] x9 : ffff59fd7ef3dad0 x8 : 000000000000c367 >[6134091.555894] x7 : 0000000000000004 x6 : 0000000000000005 >[6134091.556414] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6134091.556916] x3 : 0000000000000014 x2 : 000031a040475000 >[6134091.557415] x1 : ffff4a1de12bc680 x0 : 0000000000000014 >[6134091.557934] Call trace: >[6134091.558180] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6134091.558674] memcg_numa_stat_show+0x1d0/0x260 >[6134091.559138] cgroup_seqfile_show+0x50/0xd0 >[6134091.559549] kernfs_seq_show+0x30/0x38 >[6134091.559928] seq_read+0x148/0x478 >[6134091.560328] kernfs_fop_read+0x2c/0x1e0 >[6134091.560709] __vfs_read+0x20/0x48 >[6134091.561080] vfs_read+0x98/0x168 >[6134091.561400] ksys_read+0x6c/0xe8 >[6134091.561714] __arm64_sys_read+0x20/0x28 >[6134091.562124] el0_svc_common.constprop.0+0xa8/0x200 >[6134091.562630] el0_svc_handler+0x30/0x80 >[6134091.562991] el0_svc+0x10/0x14 >[6134135.531150] watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [kubelet:18334] >[6134135.531734] watchdog: BUG: soft lockup - CPU#46 stuck for 22s! [migration/46:246] >[6134135.532028] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6134135.532832] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6134135.540195] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6134135.547344] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6134135.552864] CPU: 8 PID: 18334 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6134135.558528] CPU: 46 PID: 246 Comm: migration/46 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6134135.559507] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6134135.560524] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6134135.561404] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6134135.562294] pstate: 60000005 (nZCv daif -PAN -UAO) >[6134135.562767] pc : mem_cgroup_nr_lru_pages.isra.7+0x9c/0xc8 >[6134135.562771] lr : mem_cgroup_nr_lru_pages.isra.7+0xac/0xc8 >[6134135.563241] pc : multi_cpu_stop+0xf4/0x180 >[6134135.563757] sp : ffff00001f9a3c20 >[6134135.564329] lr : multi_cpu_stop+0xb4/0x180 >[6134135.564834] x29: ffff00001f9a3c20 x28: 0000000000000000 >[6134135.565147] sp : ffff00000a493d50 >[6134135.565543] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6134135.566072] x29: ffff00000a493d50 x28: 0000000000000000 >[6134135.566388] x25: 0000000000000010 x24: 0000000000000000 >[6134135.566871] x27: ffff000035c93938 x26: 0000000000000060 >[6134135.567375] x23: ffff8d9c631725d0 x22: 0000000000000020 >[6134135.567877] x25: 0000000000000000 x24: 0000000000000000 >[6134135.568354] x21: ffff59fd7fe49800 x20: ffff59fd7fe4b8a8 >[6134135.568883] x23: 0000000000000000 x22: 0000000000000001 >[6134135.569395] x19: 0000000000000004 x18: 0000000000000000 >[6134135.569932] x21: ffff000035c939d4 x20: ffff000035c939b0 >[6134135.570498] x17: 0000000000000002 x16: 0000000000000000 >[6134135.571017] x19: 0000000000000001 x18: 0000000000000000 >[6134135.571538] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6134135.572065] x17: 0000000000000001 x16: 0000000000000000 >[6134135.572577] x13: ffff8b8d72fb0000 x12: ffff8b8d72faf329 >[6134135.573039] x15: 0000000000000000 x14: ffff8dbd956a3800 >[6134135.573563] x11: 0000000000000000 x10: ffff00001f9a3c80 >[6134135.574031] x13: 0000000000000115 x12: ffff8dbd59919180 >[6134135.574512] x9 : ffff59fd7ef3da4c x8 : 000000000000dcc6 >[6134135.575096] x11: 000000000000001e x10: 0000000000000d60 >[6134135.575602] x7 : 0000000000000004 x6 : 0000000000000005 >[6134135.576096] x9 : ffff59fd7ede5a0c x8 : ffff8bcd9e996400 >[6134135.576617] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6134135.577103] x7 : ffff8bcd74d43c00 x6 : 0000000000000350 >[6134135.577632] x3 : 0000000000000033 x2 : 000033d01ff98000 >[6134135.578176] x5 : 000031d0404a7000 x4 : ffff8bcd9e995c40 >[6134135.578730] x1 : ffff4a1dccf3c730 x0 : 0000000000000073 >[6134135.579216] x3 : dead000000000200 x2 : ffff000035c93988 >[6134135.579745] Call trace: >[6134135.580262] x1 : ffff000035c939d4 x0 : 0000000000000001 >[6134135.580788] mem_cgroup_nr_lru_pages.isra.7+0x9c/0xc8 >[6134135.581013] Call trace: >[6134135.581501] memcg_numa_stat_show+0x14c/0x260 >[6134135.581961] multi_cpu_stop+0xf4/0x180 >[6134135.582221] cgroup_seqfile_show+0x50/0xd0 >[6134135.582622] cpu_stopper_thread+0xec/0x178 >[6134135.582979] kernfs_seq_show+0x30/0x38 >[6134135.583373] smpboot_thread_fn+0x158/0x1d8 >[6134135.583793] seq_read+0x148/0x478 >[6134135.584161] kthread+0x130/0x138 >[6134135.584546] kernfs_fop_read+0x2c/0x1e0 >[6134135.585784] __vfs_read+0x20/0x48 >[6134135.586104] vfs_read+0x98/0x168 >[6134135.586425] ksys_read+0x6c/0xe8 >[6134135.586807] __arm64_sys_read+0x20/0x28 >[6134135.587210] el0_svc_common.constprop.0+0xa8/0x200 >[6134135.587686] el0_svc_handler+0x30/0x80 >[6134135.588147] el0_svc+0x10/0x14 >[6134175.531214] watchdog: BUG: soft lockup - CPU#12 stuck for 23s! [kubelet:18930] >[6134175.532012] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6134175.539005] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6134175.544865] CPU: 12 PID: 18930 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6134175.545823] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6134175.546663] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6134175.547116] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6134175.547657] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6134175.548168] sp : ffff0000303fbc20 >[6134175.548500] x29: ffff0000303fbc20 x28: ffff8b9c135c6000 >[6134175.549019] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6134175.549534] x25: 0000000000000003 x24: 0000000000000000 >[6134175.550027] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6134175.550527] x21: ffff8d9cc00f7c00 x20: ffff59fd7fe4b8a8 >[6134175.551027] x19: 0000000000000000 x18: 0000000000000000 >[6134175.551528] x17: 0000000000000002 x16: 0000000000000000 >[6134175.552050] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6134175.552568] x13: ffff8b8d7119b000 x12: ffff8b8d7119a2fe >[6134175.553069] x11: 0000000000000000 x10: ffff0000303fbc80 >[6134175.553580] x9 : ffff59fd7ef3dad0 x8 : 0000000000014b69 >[6134175.554071] x7 : 0000000000000004 x6 : 0000000000000005 >[6134175.554660] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6134175.555170] x3 : 0000000000000036 x2 : 000033d01ffe3000 >[6134175.555736] x1 : ffff4a1bc3f65300 x0 : 0000000000000076 >[6134175.556232] Call trace: >[6134175.556489] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6134175.556983] memcg_numa_stat_show+0x1d0/0x260 >[6134175.557429] cgroup_seqfile_show+0x50/0xd0 >[6134175.557832] kernfs_seq_show+0x30/0x38 >[6134175.558191] seq_read+0x148/0x478 >[6134175.558531] kernfs_fop_read+0x2c/0x1e0 >[6134175.558923] __vfs_read+0x20/0x48 >[6134175.559263] vfs_read+0x98/0x168 >[6134175.559582] ksys_read+0x6c/0xe8 >[6134175.559904] __arm64_sys_read+0x20/0x28 >[6134175.560275] el0_svc_common.constprop.0+0xa8/0x200 >[6134175.560751] el0_svc_handler+0x30/0x80 >[6134175.561117] el0_svc+0x10/0x14 >[6134179.531130] watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [migration/7:46] >[6134179.531981] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6134179.538673] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6134179.544390] CPU: 7 PID: 46 Comm: migration/7 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6134179.545531] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6134179.546463] pstate: 60000005 (nZCv daif -PAN -UAO) >[6134179.546990] pc : multi_cpu_stop+0xf4/0x180 >[6134179.547438] lr : multi_cpu_stop+0xb4/0x180 >[6134179.547854] sp : ffff000008e4bd50 >[6134179.548174] x29: ffff000008e4bd50 x28: 0000000000000000 >[6134179.548783] x27: ffff00003dc03938 x26: 0000000000000060 >[6134179.549354] x25: 0000000000000000 x24: 0000000000000000 >[6134179.549841] x23: 0000000000000000 x22: 0000000000000001 >[6134179.550429] x21: ffff00003dc039d4 x20: ffff00003dc039b0 >[6134179.551031] x19: 0000000000000001 x18: 0000000000000000 >[6134179.551526] x17: 0000000000000001 x16: 0000000000000000 >[6134179.552112] x15: 0000000000000000 x14: ffff8dbbb59b9c00 >[6134179.552679] x13: 000000000000050e x12: ffff8dbc5d588580 >[6134179.553179] x11: 0000000000000023 x10: 0000000000000d60 >[6134179.553725] x9 : ffff59fd7ede5a0c x8 : ffff8b7d9da7f540 >[6134179.554236] x7 : ffff8b7d6a381000 x6 : 00000000000001f7 >[6134179.554902] x5 : 00003180404be000 x4 : ffff8b7d9da7ed80 >[6134179.555448] x3 : dead000000000200 x2 : ffff00003dc03960 >[6134179.555958] x1 : ffff00003dc039d4 x0 : 0000000000000001 >[6134179.556467] Call trace: >[6134179.556726] multi_cpu_stop+0xf4/0x180 >[6134179.557083] cpu_stopper_thread+0xec/0x178 >[6134179.557489] smpboot_thread_fn+0x158/0x1d8 >[6134179.557893] kthread+0x130/0x138 >[6134219.531168] watchdog: BUG: soft lockup - CPU#9 stuck for 23s! [kubelet:18930] >[6134219.531997] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6134219.538860] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6134219.544875] CPU: 9 PID: 18930 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6134219.545889] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6134219.546801] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6134219.547262] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6134219.547794] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6134219.548301] sp : ffff0000303fbc20 >[6134219.548624] x29: ffff0000303fbc20 x28: ffff8bcc842ae000 >[6134219.549120] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6134219.549631] x25: 0000000000000003 x24: 0000000000000000 >[6134219.550135] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6134219.550648] x21: ffff8dcd2263ac00 x20: ffff59fd7fe4b8a8 >[6134219.551152] x19: 0000000000000000 x18: 0000000000000000 >[6134219.551684] x17: 0000000000000002 x16: 0000000000000000 >[6134219.552178] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6134219.552686] x13: ffff8b9d205bf000 x12: ffff8b9d205be31d >[6134219.553175] x11: 0000000000000000 x10: ffff0000303fbc80 >[6134219.553684] x9 : ffff59fd7ef3dad0 x8 : 0000000000018ac5 >[6134219.554267] x7 : 0000000000000005 x6 : 0000000000000006 >[6134219.554780] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6134219.555324] x3 : 000000000000001c x2 : 000033b040475000 >[6134219.555896] x1 : ffff4a1d007dd3d0 x0 : 000000000000005c >[6134219.556382] Call trace: >[6134219.556709] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6134219.557150] memcg_numa_stat_show+0x1d0/0x260 >[6134219.557550] cgroup_seqfile_show+0x50/0xd0 >[6134219.557975] kernfs_seq_show+0x30/0x38 >[6134219.558369] seq_read+0x148/0x478 >[6134219.558742] kernfs_fop_read+0x2c/0x1e0 >[6134219.559127] __vfs_read+0x20/0x48 >[6134219.559523] vfs_read+0x98/0x168 >[6134219.559832] ksys_read+0x6c/0xe8 >[6134219.560150] __arm64_sys_read+0x20/0x28 >[6134219.560521] el0_svc_common.constprop.0+0xa8/0x200 >[6134219.560943] el0_svc_handler+0x30/0x80 >[6134219.561329] el0_svc+0x10/0x14 >[6134263.531083] watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [migration/4:31] >[6134263.531754] watchdog: BUG: soft lockup - CPU#47 stuck for 23s! [kubelet:11759] >[6134263.531890] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6134263.532574] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6134263.539237] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6134263.545825] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6134263.551679] CPU: 4 PID: 31 Comm: migration/4 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6134263.557368] CPU: 47 PID: 11759 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6134263.558466] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6134263.559440] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6134263.560342] pstate: 60000005 (nZCv daif -PAN -UAO) >[6134263.561231] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6134263.561759] pc : multi_cpu_stop+0xf4/0x180 >[6134263.562205] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6134263.562594] lr : multi_cpu_stop+0xb4/0x180 >[6134263.563103] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6134263.563502] sp : ffff000008dd3d50 >[6134263.564127] sp : ffff00005541bc20 >[6134263.564485] x29: ffff000008dd3d50 x28: 0000000000000000 >[6134263.564829] x29: ffff00005541bc20 x28: ffff8dccf8f88000 >[6134263.565401] x27: ffff00003dc03938 x26: 0000000000000060 >[6134263.565927] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6134263.566492] x25: 0000000000000000 x24: 0000000000000000 >[6134263.567017] x25: 0000000000000010 x24: 0000000000000000 >[6134263.567531] x23: 0000000000000000 x22: 0000000000000001 >[6134263.568008] x23: 0000000000000020 x22: ffff59fd7fe49800 >[6134263.568521] x21: ffff00003dc039d4 x20: ffff00003dc039b0 >[6134263.568994] x21: ffff8b7b8b191400 x20: ffff59fd7fe4b8a8 >[6134263.569505] x19: 0000000000000001 x18: 0000000000000000 >[6134263.569978] x19: 0000000000000004 x18: 0000000000000000 >[6134263.570467] x17: 0000000000000001 x16: 0000000000000000 >[6134263.570958] x17: 0000000000000002 x16: 0000000000000000 >[6134263.571470] x15: 0000000000000000 x14: 0000000000000000 >[6134263.571941] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6134263.572524] x13: 0000000000000000 x12: 0000000000000000 >[6134263.573108] x13: ffff8bcd52fd9000 x12: ffff8bcd52fd8347 >[6134263.573596] x11: 0000000000000000 x10: 0000000000000d60 >[6134263.574251] x11: 0000000000000000 x10: ffff00005541bc80 >[6134263.574745] x9 : ffff59fd7ede5a0c x8 : ffff8b7d9da1e400 >[6134263.575265] x9 : ffff59fd7ef3dad0 x8 : 0000000000000021 >[6134263.575881] x7 : ffff8b7d6b1b3800 x6 : 00000000000002cc >[6134263.576444] x7 : 0000000000000001 x6 : 0000000000000002 >[6134263.576998] x5 : 0000318040473000 x4 : ffff8b7d9da1dc40 >[6134263.577550] x5 : 00000000fffffffa x4 : ffffffffffffffff >[6134263.578097] x3 : dead000000000200 x2 : ffff00003dc03960 >[6134263.578659] x3 : 0000000000000032 x2 : 000033d04029d000 >[6134263.579217] x1 : ffff00003dc039d4 x0 : 0000000000000001 >[6134263.579772] x1 : ffff4a1dd83bff00 x0 : 0000000000000032 >[6134263.580328] Call trace: >[6134263.580894] Call trace: >[6134263.581157] multi_cpu_stop+0xf4/0x180 >[6134263.581431] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6134263.581828] cpu_stopper_thread+0xec/0x178 >[6134263.582366] memcg_numa_stat_show+0x1d0/0x260 >[6134263.582799] smpboot_thread_fn+0x158/0x1d8 >[6134263.583270] cgroup_seqfile_show+0x50/0xd0 >[6134263.583701] kthread+0x130/0x138 >[6134263.584236] kernfs_seq_show+0x30/0x38 >[6134263.585002] seq_read+0x148/0x478 >[6134263.585367] kernfs_fop_read+0x2c/0x1e0 >[6134263.585784] __vfs_read+0x20/0x48 >[6134263.586155] vfs_read+0x98/0x168 >[6134263.586508] ksys_read+0x6c/0xe8 >[6134263.586878] __arm64_sys_read+0x20/0x28 >[6134263.587300] el0_svc_common.constprop.0+0xa8/0x200 >[6134263.587822] el0_svc_handler+0x30/0x80 >[6134263.588234] el0_svc+0x10/0x14 >[6134303.531228] watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [kubelet:18322] >[6134303.531928] watchdog: BUG: soft lockup - CPU#56 stuck for 22s! [migration/56:297] >[6134303.532052] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6134303.532766] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6134303.539057] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6134303.545536] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6134303.551471] CPU: 13 PID: 18322 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6134303.557582] CPU: 56 PID: 297 Comm: migration/56 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6134303.558574] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6134303.559713] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6134303.560638] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6134303.561512] pstate: 60000005 (nZCv daif -PAN -UAO) >[6134303.561956] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6134303.562437] pc : multi_cpu_stop+0xf4/0x180 >[6134303.562962] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6134303.562964] sp : ffff00001f8dbc20 >[6134303.563346] lr : multi_cpu_stop+0xb4/0x180 >[6134303.563349] sp : ffff00000a62bd50 >[6134303.563986] x29: ffff00001f8dbc20 x28: ffff8bbc22100000 >[6134303.564296] x29: ffff00000a62bd50 x28: 0000000000000000 >[6134303.564694] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6134303.564998] x27: ffff00001df63938 x26: 0000000000000060 >[6134303.565503] x25: 0000000000000003 x24: 0000000000000000 >[6134303.565982] x25: 0000000000000000 x24: 0000000000000000 >[6134303.566485] x23: 0000000000000008 x22: ffff59fd7fe49800 >[6134303.567003] x23: 0000000000000000 x22: 0000000000000001 >[6134303.567507] x21: ffff8d9c8125f800 x20: ffff59fd7fe4b8a8 >[6134303.567975] x21: ffff00001df639d4 x20: ffff00001df639b0 >[6134303.568489] x19: 0000000000000001 x18: 0000000000000000 >[6134303.568998] x19: 0000000000000001 x18: 0000000000000000 >[6134303.569497] x17: 0000000000000002 x16: 0000000000000000 >[6134303.569988] x17: 0000000000000001 x16: 0000000000000000 >[6134303.570514] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6134303.571009] x15: 0000000000000001 x14: ffff8dbd58e18800 >[6134303.571488] x13: ffff8b8d7341b000 x12: ffff8b8d7341a2fe >[6134303.571974] x13: 0000000000012abc x12: ffff8dbd61e28980 >[6134303.572496] x11: 0000000000000000 x10: ffff00001f8dbc80 >[6134303.573008] x11: 000000000000050f x10: 0000000000000d60 >[6134303.573505] x9 : ffff59fd7ef3dad0 x8 : 0000000000014d7d >[6134303.574080] x9 : ffff59fd7ede5a0c x8 : ffff8bbd9ea70dc0 >[6134303.574601] x7 : 0000000000000004 x6 : 0000000000000005 >[6134303.575126] x7 : ffff8bbd327e0000 x6 : 00000000000003bb >[6134303.575623] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6134303.576205] x5 : 000033d0401a3000 x4 : ffff8bbd9ea70600 >[6134303.576797] x3 : 0000000000000032 x2 : 000033d04029d000 >[6134303.577291] x3 : dead000000000200 x2 : ffff00001df63988 >[6134303.577782] x1 : ffff4a2dd797a0a8 x0 : 0000000000000032 >[6134303.578319] x1 : ffff00001df639d4 x0 : 0000000000000001 >[6134303.578818] Call trace: >[6134303.579350] Call trace: >[6134303.579919] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6134303.580386] multi_cpu_stop+0xf4/0x180 >[6134303.580664] memcg_numa_stat_show+0x1d0/0x260 >[6134303.580928] cpu_stopper_thread+0xec/0x178 >[6134303.581457] cgroup_seqfile_show+0x50/0xd0 >[6134303.581816] smpboot_thread_fn+0x158/0x1d8 >[6134303.582227] kernfs_seq_show+0x30/0x38 >[6134303.582593] kthread+0x130/0x138 >[6134303.582960] seq_read+0x148/0x478 >[6134303.584448] kernfs_fop_read+0x2c/0x1e0 >[6134303.584858] __vfs_read+0x20/0x48 >[6134303.585196] vfs_read+0x98/0x168 >[6134303.585502] ksys_read+0x6c/0xe8 >[6134303.585851] __arm64_sys_read+0x20/0x28 >[6134303.586254] el0_svc_common.constprop.0+0xa8/0x200 >[6134303.586752] el0_svc_handler+0x30/0x80 >[6134303.587096] el0_svc+0x10/0x14 >[6134347.531114] watchdog: BUG: soft lockup - CPU#6 stuck for 23s! [migration/6:41] >[6134347.531227] watchdog: BUG: soft lockup - CPU#13 stuck for 23s! [kubelet:18805] >[6134347.531941] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6134347.532634] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6134347.539344] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6134347.546060] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6134347.552020] CPU: 6 PID: 41 Comm: migration/6 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6134347.557947] CPU: 13 PID: 18805 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6134347.558946] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6134347.559931] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6134347.560820] pstate: 60000005 (nZCv daif -PAN -UAO) >[6134347.561713] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6134347.562176] pc : multi_cpu_stop+0xf4/0x180 >[6134347.562645] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6134347.563032] lr : multi_cpu_stop+0xb4/0x180 >[6134347.563034] sp : ffff000008e23d50 >[6134347.563672] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6134347.564072] x29: ffff000008e23d50 x28: 0000000000000000 >[6134347.564416] sp : ffff00001f30bc20 >[6134347.564953] x27: ffff00005ec83938 x26: 0000000000000060 >[6134347.565459] x29: ffff00001f30bc20 x28: ffff8b9b69c1a000 >[6134347.565816] x25: 0000000000000000 x24: 0000000000000000 >[6134347.566328] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6134347.566845] x23: 0000000000000000 x22: 0000000000000001 >[6134347.567379] x25: 0000000000000003 x24: 0000000000000000 >[6134347.567924] x21: ffff00005ec839d4 x20: ffff00005ec839b0 >[6134347.568496] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6134347.569038] x19: 0000000000000001 x18: 0000000000000000 >[6134347.569582] x21: ffff8dccda884000 x20: ffff59fd7fe4b8a8 >[6134347.570163] x17: 0000000000000001 x16: 0000000000000000 >[6134347.570698] x19: 0000000000000000 x18: 0000000000000000 >[6134347.571203] x15: 0000000000000000 x14: 0000000000000000 >[6134347.571701] x17: 000000000000063f x16: 0000000000000000 >[6134347.572214] x13: 0000000000000000 x12: 0000000000000000 >[6134347.572750] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6134347.573379] x11: 00000000000000e6 x10: 0000000000000d60 >[6134347.573873] x13: ffff8b8d7341c000 x12: ffff8b8d7341b31d >[6134347.574448] x9 : ffff59fd7ede5a0c x8 : ffff8b7d9da79f00 >[6134347.575022] x11: 0000000000000000 x10: ffff00001f30bc80 >[6134347.575647] x7 : ffff8b7d3cee7400 x6 : 0000000000000158 >[6134347.576193] x9 : ffff59fd7ef3dad0 x8 : 0000000000018aae >[6134347.576798] x5 : 00003180404a5000 x4 : ffff8b7d9da79740 >[6134347.577311] x7 : 0000000000000005 x6 : 0000000000000006 >[6134347.577903] x3 : dead000000000200 x2 : ffff00005ec83960 >[6134347.578443] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6134347.578938] x1 : ffff00005ec839d4 x0 : 0000000000000001 >[6134347.579515] x3 : 0000000000000013 x2 : 000033a04045c000 >[6134347.580100] Call trace: >[6134347.580755] x1 : ffff4a0f1c300a20 x0 : 0000000000000053 >[6134347.581301] multi_cpu_stop+0xf4/0x180 >[6134347.581577] Call trace: >[6134347.582127] cpu_stopper_thread+0xec/0x178 >[6134347.582499] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6134347.582759] smpboot_thread_fn+0x158/0x1d8 >[6134347.583297] memcg_numa_stat_show+0x1d0/0x260 >[6134347.583800] kthread+0x130/0x138 >[6134347.584229] cgroup_seqfile_show+0x50/0xd0 >[6134347.585409] kernfs_seq_show+0x30/0x38 >[6134347.585766] seq_read+0x148/0x478 >[6134347.586149] kernfs_fop_read+0x2c/0x1e0 >[6134347.586542] __vfs_read+0x20/0x48 >[6134347.586878] vfs_read+0x98/0x168 >[6134347.587205] ksys_read+0x6c/0xe8 >[6134347.587562] __arm64_sys_read+0x20/0x28 >[6134347.587951] el0_svc_common.constprop.0+0xa8/0x200 >[6134347.588461] el0_svc_handler+0x30/0x80 >[6134347.588832] el0_svc+0x10/0x14 >[6134391.531291] watchdog: BUG: soft lockup - CPU#17 stuck for 23s! [kubelet:18322] >[6134391.532078] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6134391.538703] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6134391.544382] CPU: 17 PID: 18322 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6134391.545351] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6134391.546187] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6134391.546630] pc : mem_cgroup_nr_lru_pages.isra.7+0x9c/0xc8 >[6134391.547149] lr : mem_cgroup_nr_lru_pages.isra.7+0xac/0xc8 >[6134391.547731] sp : ffff00001f8dbc20 >[6134391.548091] x29: ffff00001f8dbc20 x28: 0000000000000000 >[6134391.548717] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6134391.549286] x25: 0000000000000010 x24: 0000000000000000 >[6134391.549848] x23: ffff8bbb40eda5d0 x22: 0000000000000020 >[6134391.550418] x21: ffff59fd7fe49800 x20: ffff59fd7fe4b8a8 >[6134391.550998] x19: 0000000000000004 x18: 0000000000000000 >[6134391.551573] x17: 000000000000000a x16: 0000000000000000 >[6134391.552160] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6134391.552742] x13: ffff8b9b753c3000 x12: ffff8b9b753c2328 >[6134391.553397] x11: 0000000000000000 x10: ffff00001f8dbc80 >[6134391.553970] x9 : ffff59fd7ef3da4c x8 : 000000000000da10 >[6134391.554532] x7 : 0000000000000004 x6 : 0000000000000005 >[6134391.555071] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6134391.555608] x3 : 000000000000001c x2 : 000031b040475000 >[6134391.556090] x1 : ffff4a2b590feaa0 x0 : 000000000000001c >[6134391.556587] Call trace: >[6134391.556836] mem_cgroup_nr_lru_pages.isra.7+0x9c/0xc8 >[6134391.557291] memcg_numa_stat_show+0x14c/0x260 >[6134391.557718] cgroup_seqfile_show+0x50/0xd0 >[6134391.558089] kernfs_seq_show+0x30/0x38 >[6134391.558446] seq_read+0x148/0x478 >[6134391.558850] kernfs_fop_read+0x2c/0x1e0 >[6134391.559240] __vfs_read+0x20/0x48 >[6134391.559578] vfs_read+0x98/0x168 >[6134391.559937] ksys_read+0x6c/0xe8 >[6134391.560303] __arm64_sys_read+0x20/0x28 >[6134391.560723] el0_svc_common.constprop.0+0xa8/0x200 >[6134391.561261] el0_svc_handler+0x30/0x80 >[6134391.561651] el0_svc+0x10/0x14 >[6134395.531642] watchdog: BUG: soft lockup - CPU#40 stuck for 23s! [migration/40:215] >[6134395.532517] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6134395.539151] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6134395.544882] CPU: 40 PID: 215 Comm: migration/40 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6134395.545827] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6134395.546662] pstate: 60000005 (nZCv daif -PAN -UAO) >[6134395.547096] pc : multi_cpu_stop+0xf4/0x180 >[6134395.547524] lr : multi_cpu_stop+0xb4/0x180 >[6134395.547889] sp : ffff00000a39bd50 >[6134395.548234] x29: ffff00000a39bd50 x28: 0000000000000000 >[6134395.548750] x27: ffff00005541b938 x26: 0000000000000060 >[6134395.549252] x25: 0000000000000000 x24: 0000000000000000 >[6134395.549749] x23: 0000000000000000 x22: 0000000000000001 >[6134395.550266] x21: ffff00005541b9d4 x20: ffff00005541b9b0 >[6134395.550743] x19: 0000000000000001 x18: 0000000000000000 >[6134395.551250] x17: 0000000000000001 x16: 0000000000000000 >[6134395.551751] x15: 0000000000000000 x14: 0000000000000000 >[6134395.552267] x13: 0000000000000000 x12: 0000000000000000 >[6134395.552769] x11: 0000000000000017 x10: 0000000000000d60 >[6134395.553364] x9 : ffff59fd7ede5a0c x8 : ffff8bcd9e8de400 >[6134395.553928] x7 : ffff8bcd56764000 x6 : 0000000000000315 >[6134395.554402] x5 : 000031d040411000 x4 : ffff8bcd9e8ddc40 >[6134395.554970] x3 : dead000000000200 x2 : ffff00005541b988 >[6134395.555442] x1 : ffff00005541b9d4 x0 : 0000000000000001 >[6134395.555978] Call trace: >[6134395.556211] multi_cpu_stop+0xf4/0x180 >[6134395.556585] cpu_stopper_thread+0xec/0x178 >[6134395.556978] smpboot_thread_fn+0x158/0x1d8 >[6134395.557367] kthread+0x130/0x138 >[6134431.531016] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kubelet:18890] >[6134431.531823] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6134431.538499] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6134431.544390] CPU: 1 PID: 18890 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6134431.545393] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6134431.546314] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6134431.546818] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6134431.547314] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6134431.547818] sp : ffff00001fd9bc20 >[6134431.548176] x29: ffff00001fd9bc20 x28: ffff8b7bd397a000 >[6134431.548690] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6134431.549183] x25: 0000000000000003 x24: 0000000000000000 >[6134431.549674] x23: 0000000000000008 x22: ffff59fd7fe49800 >[6134431.550171] x21: ffff8d7c08922400 x20: ffff59fd7fe4b8a8 >[6134431.550669] x19: 0000000000000001 x18: 0000000000000000 >[6134431.551208] x17: 0000000000000334 x16: 0000000000000000 >[6134431.551732] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6134431.552210] x13: ffff8b7d1ada1000 x12: ffff8b7d1ada02ea >[6134431.552849] x11: 0000000000000000 x10: ffff00001fd9bc80 >[6134431.553336] x9 : ffff59fd7ef3dad0 x8 : 0000000000020d14 >[6134431.553845] x7 : 0000000000000005 x6 : 0000000000000006 >[6134431.554400] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6134431.554950] x3 : 000000000000002c x2 : 000033d020079000 >[6134431.555468] x1 : ffff4a2b506b53d8 x0 : 000000000000006c >[6134431.556039] Call trace: >[6134431.556286] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6134431.556820] memcg_numa_stat_show+0x1d0/0x260 >[6134431.557221] cgroup_seqfile_show+0x50/0xd0 >[6134431.557721] kernfs_seq_show+0x30/0x38 >[6134431.558122] seq_read+0x148/0x478 >[6134431.558443] kernfs_fop_read+0x2c/0x1e0 >[6134431.558808] __vfs_read+0x20/0x48 >[6134431.559225] vfs_read+0x98/0x168 >[6134431.559579] ksys_read+0x6c/0xe8 >[6134431.559896] __arm64_sys_read+0x20/0x28 >[6134431.560300] el0_svc_common.constprop.0+0xa8/0x200 >[6134431.560769] el0_svc_handler+0x30/0x80 >[6134431.561142] el0_svc+0x10/0x14 >[6134435.531209] watchdog: BUG: soft lockup - CPU#12 stuck for 23s! [migration/12:72] >[6134435.532080] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6134435.539055] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6134435.545064] CPU: 12 PID: 72 Comm: migration/12 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6134435.546053] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6134435.546990] pstate: 60000005 (nZCv daif -PAN -UAO) >[6134435.547476] pc : multi_cpu_stop+0x78/0x180 >[6134435.547865] lr : multi_cpu_stop+0xb4/0x180 >[6134435.548234] sp : ffff000008f1bd50 >[6134435.548590] x29: ffff000008f1bd50 x28: 0000000000000000 >[6134435.549094] x27: ffff000054d43938 x26: 0000000000000060 >[6134435.549587] x25: 0000000000000000 x24: 0000000000000000 >[6134435.550076] x23: 0000000000000000 x22: 0000000000000001 >[6134435.550586] x21: ffff000054d439d4 x20: ffff000054d439b0 >[6134435.551069] x19: 0000000000000001 x18: 0000000000000000 >[6134435.551569] x17: 0000000000000001 x16: 0000000000000000 >[6134435.552055] x15: 0000000000000000 x14: 0000000000000000 >[6134435.552538] x13: 0000000000000000 x12: 0000000000000000 >[6134435.553178] x11: 0000000000000146 x10: 0000000000000d60 >[6134435.553693] x9 : ffff59fd7ede5a0c x8 : ffff8b8d9e953040 >[6134435.554245] x7 : ffff8b8d7c5a2c00 x6 : 0000000000000167 >[6134435.554812] x5 : 0000319040475000 x4 : ffff8b8d9e952880 >[6134435.555313] x3 : dead000000000200 x2 : ffff000054d43988 >[6134435.555826] x1 : ffff000054d439d4 x0 : 0000000000000001 >[6134435.556335] Call trace: >[6134435.556588] multi_cpu_stop+0x78/0x180 >[6134435.556953] cpu_stopper_thread+0xec/0x178 >[6134435.557392] smpboot_thread_fn+0x158/0x1d8 >[6134435.557844] kthread+0x130/0x138 >[6134475.531307] watchdog: BUG: soft lockup - CPU#18 stuck for 22s! [kubelet:18322] >[6134475.532165] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6134475.539823] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6134475.546381] CPU: 18 PID: 18322 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6134475.547346] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6134475.548271] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6134475.548779] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6134475.549287] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6134475.549786] sp : ffff00001f8dbc20 >[6134475.550111] x29: ffff00001f8dbc20 x28: ffff8bccadef6000 >[6134475.550575] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6134475.551044] x25: 0000000000000003 x24: 0000000000000000 >[6134475.551517] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6134475.552003] x21: ffff8d8d331f3c00 x20: ffff59fd7fe4b8a8 >[6134475.552641] x19: 0000000000000000 x18: 0000000000000000 >[6134475.553118] x17: 0000000000000002 x16: 0000000000000000 >[6134475.553588] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6134475.554070] x13: ffff8b9d205f6000 x12: ffff8b9d205f52f4 >[6134475.554559] x11: 0000000000000000 x10: ffff00001f8dbc80 >[6134475.555048] x9 : ffff59fd7ef3dad0 x8 : 0000000000037ac8 >[6134475.555516] x7 : 0000000000000005 x6 : 0000000000000006 >[6134475.555988] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6134475.556470] x3 : 000000000000001a x2 : 000033b040443000 >[6134475.556989] x1 : ffff4a1f158a9b20 x0 : 000000000000005a >[6134475.557472] Call trace: >[6134475.557702] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6134475.558210] memcg_numa_stat_show+0x1d0/0x260 >[6134475.558609] cgroup_seqfile_show+0x50/0xd0 >[6134475.559033] kernfs_seq_show+0x30/0x38 >[6134475.559378] seq_read+0x148/0x478 >[6134475.559741] kernfs_fop_read+0x2c/0x1e0 >[6134475.560139] __vfs_read+0x20/0x48 >[6134475.560463] vfs_read+0x98/0x168 >[6134475.560776] ksys_read+0x6c/0xe8 >[6134475.561093] __arm64_sys_read+0x20/0x28 >[6134475.561483] el0_svc_common.constprop.0+0xa8/0x200 >[6134475.561910] el0_svc_handler+0x30/0x80 >[6134475.562247] el0_svc+0x10/0x14 >[6134483.531660] watchdog: BUG: soft lockup - CPU#41 stuck for 22s! [migration/41:221] >[6134483.532593] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6134483.539061] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6134483.544769] CPU: 41 PID: 221 Comm: migration/41 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6134483.545884] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6134483.546723] pstate: 60000005 (nZCv daif -PAN -UAO) >[6134483.547173] pc : multi_cpu_stop+0x78/0x180 >[6134483.547574] lr : multi_cpu_stop+0xb4/0x180 >[6134483.548019] sp : ffff00000a3cbd50 >[6134483.548347] x29: ffff00000a3cbd50 x28: 0000000000000000 >[6134483.548854] x27: ffff00003187b938 x26: 0000000000000060 >[6134483.549489] x25: 0000000000000000 x24: 0000000000000000 >[6134483.549958] x23: 0000000000000000 x22: 0000000000000001 >[6134483.550435] x21: ffff00003187b9d4 x20: ffff00003187b9b0 >[6134483.550907] x19: 0000000000000001 x18: 0000000000000000 >[6134483.551390] x17: 0000000000000001 x16: 0000000000000000 >[6134483.551856] x15: 0000000000000000 x14: 0000000000000000 >[6134483.552453] x13: 0000000000000000 x12: 0000000000000000 >[6134483.552972] x11: 0000000000000017 x10: 0000000000000d60 >[6134483.553468] x9 : ffff59fd7ede5a0c x8 : ffff8bcd9e8f9f00 >[6134483.553957] x7 : ffff8bcd56762c00 x6 : 0000000000000389 >[6134483.554431] x5 : 000031d04042a000 x4 : ffff8bcd9e8f9740 >[6134483.554949] x3 : dead000000000200 x2 : ffff00003187b988 >[6134483.555439] x1 : ffff00003187b9d4 x0 : 0000000000000001 >[6134483.556013] Call trace: >[6134483.556250] multi_cpu_stop+0x78/0x180 >[6134483.556637] cpu_stopper_thread+0xec/0x178 >[6134483.557051] smpboot_thread_fn+0x158/0x1d8 >[6134483.557436] kthread+0x130/0x138 >[6134519.531229] watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [kubelet:3380] >[6134519.532126] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6134519.538676] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6134519.544716] CPU: 13 PID: 3380 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6134519.545686] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6134519.546535] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6134519.546984] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6134519.547484] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6134519.547975] sp : ffff000031ba3c20 >[6134519.548341] x29: ffff000031ba3c20 x28: ffff8d8cdd14a000 >[6134519.548883] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6134519.549456] x25: 0000000000000003 x24: 0000000000000000 >[6134519.549945] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6134519.550488] x21: ffff8dac19f55000 x20: ffff59fd7fe4b8a8 >[6134519.551067] x19: 0000000000000000 x18: 0000000000000000 >[6134519.551670] x17: 0000000000000002 x16: 0000000000000000 >[6134519.552297] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6134519.552800] x13: ffff8b8d7341b000 x12: ffff8b8d7341a308 >[6134519.553378] x11: 0000000000000000 x10: ffff000031ba3c80 >[6134519.553895] x9 : ffff59fd7ef3dad0 x8 : 000000000000ca09 >[6134519.554375] x7 : 0000000000000004 x6 : 0000000000000005 >[6134519.554940] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6134519.555448] x3 : 0000000000000013 x2 : 000033a04045c000 >[6134519.555987] x1 : ffff4a1dd0e92f80 x0 : 0000000000000053 >[6134519.556553] Call trace: >[6134519.556786] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6134519.557302] memcg_numa_stat_show+0x1d0/0x260 >[6134519.557691] cgroup_seqfile_show+0x50/0xd0 >[6134519.558096] kernfs_seq_show+0x30/0x38 >[6134519.558518] seq_read+0x148/0x478 >[6134519.558832] kernfs_fop_read+0x2c/0x1e0 >[6134519.559241] __vfs_read+0x20/0x48 >[6134519.559575] vfs_read+0x98/0x168 >[6134519.559905] ksys_read+0x6c/0xe8 >[6134519.560232] __arm64_sys_read+0x20/0x28 >[6134519.560619] el0_svc_common.constprop.0+0xa8/0x200 >[6134519.561096] el0_svc_handler+0x30/0x80 >[6134519.561456] el0_svc+0x10/0x14 >[6134563.531165] watchdog: BUG: soft lockup - CPU#9 stuck for 22s! [kubelet:57297] >[6134563.532044] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6134563.538572] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6134563.544431] CPU: 9 PID: 57297 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6134563.545333] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6134563.546182] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6134563.546684] pc : mem_cgroup_nr_lru_pages.isra.7+0x9c/0xc8 >[6134563.547251] lr : mem_cgroup_nr_lru_pages.isra.7+0xac/0xc8 >[6134563.547790] sp : ffff00003f4f3c20 >[6134563.548130] x29: ffff00003f4f3c20 x28: 0000000000000000 >[6134563.548677] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6134563.549172] x25: 0000000000000010 x24: 0000000000000000 >[6134563.549683] x23: ffff8babed8b25d0 x22: 0000000000000020 >[6134563.550163] x21: ffff59fd7fe49800 x20: ffff59fd7fe4b8a8 >[6134563.550658] x19: 0000000000000004 x18: 0000000000000000 >[6134563.551155] x17: 0000000000000004 x16: 0000000000000000 >[6134563.551642] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6134563.552248] x13: ffff8b9d205bd000 x12: ffff8b9d205bc328 >[6134563.552795] x11: 0000000000000000 x10: ffff00003f4f3c80 >[6134563.553337] x9 : ffff59fd7ef3da4c x8 : 000000000000de91 >[6134563.553854] x7 : 0000000000000004 x6 : 0000000000000005 >[6134563.554397] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6134563.554900] x3 : 0000000000000001 x2 : 0000338040428000 >[6134563.555434] x1 : ffff4a2b4c2be5b0 x0 : 0000000000000041 >[6134563.555985] Call trace: >[6134563.556235] mem_cgroup_nr_lru_pages.isra.7+0x9c/0xc8 >[6134563.556778] memcg_numa_stat_show+0x14c/0x260 >[6134563.557187] cgroup_seqfile_show+0x50/0xd0 >[6134563.557590] kernfs_seq_show+0x30/0x38 >[6134563.557949] seq_read+0x148/0x478 >[6134563.558287] kernfs_fop_read+0x2c/0x1e0 >[6134563.558667] __vfs_read+0x20/0x48 >[6134563.558998] vfs_read+0x98/0x168 >[6134563.559307] ksys_read+0x6c/0xe8 >[6134563.559630] __arm64_sys_read+0x20/0x28 >[6134563.560018] el0_svc_common.constprop.0+0xa8/0x200 >[6134563.560474] el0_svc_handler+0x30/0x80 >[6134563.560852] el0_svc+0x10/0x14 >[6134603.531016] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kubelet:13350] >[6134603.531859] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6134603.538575] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6134603.544688] CPU: 1 PID: 13350 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6134603.545663] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6134603.546526] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6134603.547026] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6134603.547535] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6134603.548045] sp : ffff0000575d3c20 >[6134603.548419] x29: ffff0000575d3c20 x28: ffff8b7b8d408000 >[6134603.548938] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6134603.549448] x25: 0000000000000003 x24: 0000000000000000 >[6134603.549985] x23: 0000000000000008 x22: ffff59fd7fe49800 >[6134603.550501] x21: ffff8d8cdb34b000 x20: ffff59fd7fe4b8a8 >[6134603.551009] x19: 0000000000000001 x18: 0000000000000000 >[6134603.551613] x17: 0000000000000002 x16: 0000000000000000 >[6134603.552150] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6134603.552665] x13: ffff8b7d1ada1000 x12: ffff8b7d1ada02f4 >[6134603.553152] x11: 0000000000000000 x10: ffff0000575d3c80 >[6134603.553654] x9 : ffff59fd7ef3dad0 x8 : 0000000000039d3c >[6134603.554170] x7 : 0000000000000005 x6 : 0000000000000006 >[6134603.554723] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6134603.555252] x3 : 0000000000000017 x2 : 000033a0404c0000 >[6134603.555752] x1 : ffff4a1dcb324e48 x0 : 0000000000000057 >[6134603.556303] Call trace: >[6134603.556561] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6134603.557091] memcg_numa_stat_show+0x1d0/0x260 >[6134603.557568] cgroup_seqfile_show+0x50/0xd0 >[6134603.557972] kernfs_seq_show+0x30/0x38 >[6134603.558398] seq_read+0x148/0x478 >[6134603.558747] kernfs_fop_read+0x2c/0x1e0 >[6134603.559208] __vfs_read+0x20/0x48 >[6134603.559543] vfs_read+0x98/0x168 >[6134603.559865] ksys_read+0x6c/0xe8 >[6134603.560250] __arm64_sys_read+0x20/0x28 >[6134603.560631] el0_svc_common.constprop.0+0xa8/0x200 >[6134603.561102] el0_svc_handler+0x30/0x80 >[6134603.561555] el0_svc+0x10/0x14 >[6134647.531389] watchdog: BUG: soft lockup - CPU#23 stuck for 22s! [kubelet:18354] >[6134647.532098] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6134647.538478] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6134647.544115] CPU: 23 PID: 18354 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6134647.545072] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6134647.545953] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6134647.546390] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6134647.546869] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6134647.547380] sp : ffff00001f9dbc20 >[6134647.547701] x29: ffff00001f9dbc20 x28: ffff8dac224e0000 >[6134647.548195] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6134647.548693] x25: 0000000000000003 x24: 0000000000000000 >[6134647.549191] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6134647.549689] x21: ffff8d9c6e0e0000 x20: ffff59fd7fe4b8a8 >[6134647.550185] x19: 0000000000000000 x18: 0000000000000000 >[6134647.550687] x17: 0000000000000002 x16: 0000000000000000 >[6134647.551289] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6134647.551762] x13: ffff8b9d220b4000 x12: ffff8b9d220b32fe >[6134647.552246] x11: 0000000000000000 x10: ffff00001f9dbc80 >[6134647.552798] x9 : ffff59fd7ef3dad0 x8 : 0000000000014d41 >[6134647.553380] x7 : 0000000000000004 x6 : 0000000000000005 >[6134647.553855] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6134647.554393] x3 : 000000000000001b x2 : 000033b04045c000 >[6134647.554912] x1 : ffff4a2dd01ff450 x0 : 000000000000005b >[6134647.555421] Call trace: >[6134647.555681] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6134647.556169] memcg_numa_stat_show+0x1d0/0x260 >[6134647.556663] cgroup_seqfile_show+0x50/0xd0 >[6134647.557073] kernfs_seq_show+0x30/0x38 >[6134647.557534] seq_read+0x148/0x478 >[6134647.557863] kernfs_fop_read+0x2c/0x1e0 >[6134647.558252] __vfs_read+0x20/0x48 >[6134647.558605] vfs_read+0x98/0x168 >[6134647.558906] ksys_read+0x6c/0xe8 >[6134647.559210] __arm64_sys_read+0x20/0x28 >[6134647.559570] el0_svc_common.constprop.0+0xa8/0x200 >[6134647.560002] el0_svc_handler+0x30/0x80 >[6134647.560358] el0_svc+0x10/0x14 >[6134691.531183] watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [kubelet:18929] >[6134691.532037] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6134691.538684] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6134691.544618] CPU: 10 PID: 18929 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6134691.545659] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6134691.546503] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6134691.546940] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6134691.547443] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6134691.547956] sp : ffff00003001bc20 >[6134691.548273] x29: ffff00003001bc20 x28: ffff8bac34024000 >[6134691.548777] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6134691.549285] x25: 0000000000000003 x24: 0000000000000000 >[6134691.549784] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6134691.550269] x21: ffff8dbc0c748400 x20: ffff59fd7fe4b8a8 >[6134691.550878] x19: 0000000000000000 x18: 0000000000000000 >[6134691.551377] x17: 0000000000000002 x16: 0000000000000000 >[6134691.551920] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6134691.552550] x13: ffff8b8d72d57000 x12: ffff8b8d72d56312 >[6134691.553040] x11: 0000000000000000 x10: ffff00003001bc80 >[6134691.553601] x9 : ffff59fd7ef3dad0 x8 : 000000000004f1d6 >[6134691.554143] x7 : 0000000000000005 x6 : 0000000000000006 >[6134691.554687] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6134691.555173] x3 : 0000000000000010 x2 : 000033a040411000 >[6134691.555725] x1 : ffff4a0f24d4cf40 x0 : 0000000000000050 >[6134691.556261] Call trace: >[6134691.556562] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6134691.557014] memcg_numa_stat_show+0x1d0/0x260 >[6134691.557474] cgroup_seqfile_show+0x50/0xd0 >[6134691.557889] kernfs_seq_show+0x30/0x38 >[6134691.558343] seq_read+0x148/0x478 >[6134691.558655] kernfs_fop_read+0x2c/0x1e0 >[6134691.559077] __vfs_read+0x20/0x48 >[6134691.559394] vfs_read+0x98/0x168 >[6134691.559726] ksys_read+0x6c/0xe8 >[6134691.560140] __arm64_sys_read+0x20/0x28 >[6134691.560497] el0_svc_common.constprop.0+0xa8/0x200 >[6134691.561134] el0_svc_handler+0x30/0x80 >[6134691.561491] el0_svc+0x10/0x14 >[6134735.531026] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kubelet:18291] >[6134735.531910] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6134735.538459] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6134735.544261] CPU: 2 PID: 18291 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6134735.545279] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6134735.546151] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6134735.546602] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6134735.547109] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6134735.547615] sp : ffff00001f8ebc20 >[6134735.547931] x29: ffff00001f8ebc20 x28: ffff8b8c6b9a6000 >[6134735.548432] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6134735.548951] x25: 0000000000000010 x24: 0000000000000000 >[6134735.549466] x23: 0000000000000020 x22: ffff59fd7fe49800 >[6134735.549973] x21: ffff8b7bf015d800 x20: ffff59fd7fe4b8a8 >[6134735.550570] x19: 0000000000000004 x18: 0000000000000000 >[6134735.551090] x17: 000000000000065c x16: 0000000000000000 >[6134735.551596] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6134735.552098] x13: ffff8b7ba68ff000 x12: ffff8b7ba68fe347 >[6134735.552639] x11: 0000000000000000 x10: ffff00001f8ebc80 >[6134735.553161] x9 : ffff59fd7ef3dad0 x8 : 0000000000000021 >[6134735.553721] x7 : 0000000000000001 x6 : 0000000000000002 >[6134735.554205] x5 : 00000000fffffffa x4 : ffffffffffffffff >[6134735.554756] x3 : 0000000000000001 x2 : 0000338040428000 >[6134735.555273] x1 : ffff4a1cc35f0830 x0 : 0000000000000041 >[6134735.555840] Call trace: >[6134735.556086] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6134735.556646] memcg_numa_stat_show+0x1d0/0x260 >[6134735.557084] cgroup_seqfile_show+0x50/0xd0 >[6134735.557514] kernfs_seq_show+0x30/0x38 >[6134735.557878] seq_read+0x148/0x478 >[6134735.558254] kernfs_fop_read+0x2c/0x1e0 >[6134735.558632] __vfs_read+0x20/0x48 >[6134735.558980] vfs_read+0x98/0x168 >[6134735.559301] ksys_read+0x6c/0xe8 >[6134735.559634] __arm64_sys_read+0x20/0x28 >[6134735.560020] el0_svc_common.constprop.0+0xa8/0x200 >[6134735.560565] el0_svc_handler+0x30/0x80 >[6134735.560942] el0_svc+0x10/0x14 >[6134739.531146] watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [migration/8:51] >[6134739.531942] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6134739.538637] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6134739.544683] CPU: 8 PID: 51 Comm: migration/8 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6134739.545686] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6134739.546516] pstate: 60000005 (nZCv daif -PAN -UAO) >[6134739.546991] pc : multi_cpu_stop+0x78/0x180 >[6134739.547391] lr : multi_cpu_stop+0xb4/0x180 >[6134739.547795] sp : ffff000008e73d50 >[6134739.548125] x29: ffff000008e73d50 x28: 0000000000000000 >[6134739.548685] x27: ffff0000545d3938 x26: 0000000000000060 >[6134739.549222] x25: 0000000000000000 x24: 0000000000000000 >[6134739.549726] x23: 0000000000000000 x22: 0000000000000001 >[6134739.550291] x21: ffff0000545d39d4 x20: ffff0000545d39b0 >[6134739.550914] x19: 0000000000000001 x18: 0000000000000000 >[6134739.551431] x17: 0000000000000001 x16: 0000000000000000 >[6134739.551981] x15: 0000000000000000 x14: 0000000000000000 >[6134739.552575] x13: 0000000000000000 x12: 0000000000000000 >[6134739.553069] x11: 000000000000024c x10: 0000000000000d60 >[6134739.553578] x9 : ffff59fd7ede5a0c x8 : ffff8b8d9e8d6400 >[6134739.554117] x7 : ffff8b8d8059c000 x6 : 00000000000002ca >[6134739.554624] x5 : 0000319040411000 x4 : ffff8b8d9e8d5c40 >[6134739.555132] x3 : dead000000000200 x2 : ffff0000545d3988 >[6134739.555638] x1 : ffff0000545d39d4 x0 : 0000000000000001 >[6134739.556140] Call trace: >[6134739.556388] multi_cpu_stop+0x78/0x180 >[6134739.556762] cpu_stopper_thread+0xec/0x178 >[6134739.557163] smpboot_thread_fn+0x158/0x1d8 >[6134739.557571] kthread+0x130/0x138 >[6134775.531258] watchdog: BUG: soft lockup - CPU#15 stuck for 22s! [kubelet:61153] >[6134775.532043] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6134775.538742] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6134775.544615] CPU: 15 PID: 61153 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6134775.545588] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6134775.546482] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6134775.546914] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6134775.547408] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6134775.547950] sp : ffff000058b4bc20 >[6134775.548249] x29: ffff000058b4bc20 x28: ffff8b8c77dd0000 >[6134775.548769] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6134775.549265] x25: 0000000000000003 x24: 0000000000000000 >[6134775.549771] x23: 0000000000000008 x22: ffff59fd7fe49800 >[6134775.550366] x21: ffff8d9cfdc57800 x20: ffff59fd7fe4b8a8 >[6134775.550877] x19: 0000000000000001 x18: 0000000000000000 >[6134775.551425] x17: 0000000000000002 x16: 0000000000000000 >[6134775.551920] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6134775.552408] x13: ffff8b8bdf969000 x12: ffff8b8bdf9682fe >[6134775.552907] x11: 0000000000000000 x10: ffff000058b4bc80 >[6134775.553458] x9 : ffff59fd7ef3dad0 x8 : 000000000001535a >[6134775.554013] x7 : 0000000000000004 x6 : 0000000000000005 >[6134775.554555] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6134775.555117] x3 : 000000000000000f x2 : 00003390404c0000 >[6134775.555616] x1 : ffff4a1f1b41f408 x0 : 000000000000004f >[6134775.556086] Call trace: >[6134775.556323] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6134775.556786] memcg_numa_stat_show+0x1d0/0x260 >[6134775.557203] cgroup_seqfile_show+0x50/0xd0 >[6134775.557607] kernfs_seq_show+0x30/0x38 >[6134775.557969] seq_read+0x148/0x478 >[6134775.558313] kernfs_fop_read+0x2c/0x1e0 >[6134775.558691] __vfs_read+0x20/0x48 >[6134775.559032] vfs_read+0x98/0x168 >[6134775.559352] ksys_read+0x6c/0xe8 >[6134775.559727] __arm64_sys_read+0x20/0x28 >[6134775.560107] el0_svc_common.constprop.0+0xa8/0x200 >[6134775.560675] el0_svc_handler+0x30/0x80 >[6134775.561048] el0_svc+0x10/0x14 >[6134819.531678] watchdog: BUG: soft lockup - CPU#42 stuck for 22s! [kubelet:18848] >[6134819.532498] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6134819.538971] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6134819.544756] CPU: 42 PID: 18848 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6134819.545684] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6134819.546515] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6134819.546938] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6134819.547414] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6134819.547919] sp : ffff000030213c20 >[6134819.548229] x29: ffff000030213c20 x28: ffff8bcd44b00000 >[6134819.548758] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6134819.549285] x25: 0000000000000003 x24: 0000000000000000 >[6134819.549773] x23: 0000000000000008 x22: ffff59fd7fe49800 >[6134819.550357] x21: ffff8dbcbf0b8800 x20: ffff59fd7fe4b8a8 >[6134819.550847] x19: 0000000000000001 x18: 0000000000000000 >[6134819.551315] x17: 0000000000000002 x16: 0000000000000000 >[6134819.551785] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6134819.552254] x13: ffff8bcd52fe5000 x12: ffff8bcd52fe4312 >[6134819.552794] x11: 0000000000000000 x10: ffff000030213c80 >[6134819.553298] x9 : ffff59fd7ef3dad0 x8 : 0000000000050c8b >[6134819.553803] x7 : 0000000000000005 x6 : 0000000000000006 >[6134819.554303] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6134819.554793] x3 : 0000000000000010 x2 : 000033a040411000 >[6134819.555274] x1 : ffff4a23c134ed98 x0 : 0000000000000050 >[6134819.555754] Call trace: >[6134819.555985] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6134819.556487] memcg_numa_stat_show+0x1d0/0x260 >[6134819.556986] cgroup_seqfile_show+0x50/0xd0 >[6134819.557389] kernfs_seq_show+0x30/0x38 >[6134819.557795] seq_read+0x148/0x478 >[6134819.558097] kernfs_fop_read+0x2c/0x1e0 >[6134819.558458] __vfs_read+0x20/0x48 >[6134819.558804] vfs_read+0x98/0x168 >[6134819.559158] ksys_read+0x6c/0xe8 >[6134819.559544] __arm64_sys_read+0x20/0x28 >[6134819.559997] el0_svc_common.constprop.0+0xa8/0x200 >[6134819.560482] el0_svc_handler+0x30/0x80 >[6134819.560848] el0_svc+0x10/0x14 >[6134823.531130] watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [migration/7:46] >[6134823.531984] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6134823.538781] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6134823.544647] CPU: 7 PID: 46 Comm: migration/7 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6134823.545637] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6134823.546508] pstate: 60000005 (nZCv daif -PAN -UAO) >[6134823.547013] pc : multi_cpu_stop+0x78/0x180 >[6134823.547408] lr : multi_cpu_stop+0xb4/0x180 >[6134823.547795] sp : ffff000008e4bd50 >[6134823.548109] x29: ffff000008e4bd50 x28: 0000000000000000 >[6134823.548775] x27: ffff00005b1d3938 x26: 0000000000000060 >[6134823.549282] x25: 0000000000000000 x24: 0000000000000000 >[6134823.549786] x23: 0000000000000000 x22: 0000000000000001 >[6134823.550404] x21: ffff00005b1d39d4 x20: ffff00005b1d39b0 >[6134823.550890] x19: 0000000000000001 x18: 0000000000000000 >[6134823.551443] x17: 0000000000000001 x16: 0000000000000000 >[6134823.551952] x15: 0000000000000000 x14: ffff8dbb7bfa8800 >[6134823.552457] x13: 0000000000003844 x12: ffff8dbb78910d80 >[6134823.553003] x11: 000000000000008b x10: 0000000000000d60 >[6134823.553516] x9 : ffff59fd7ede5a0c x8 : ffff8b7d9da7f540 >[6134823.554023] x7 : ffff8b7d6d735800 x6 : 0000000000000357 >[6134823.554527] x5 : 00003180404be000 x4 : ffff8b7d9da7ed80 >[6134823.555025] x3 : dead000000000200 x2 : ffff00005b1d3960 >[6134823.555522] x1 : ffff00005b1d39d4 x0 : 0000000000000001 >[6134823.556020] Call trace: >[6134823.556265] multi_cpu_stop+0x78/0x180 >[6134823.556627] cpu_stopper_thread+0xec/0x178 >[6134823.557016] smpboot_thread_fn+0x158/0x1d8 >[6134823.557410] kthread+0x130/0x138 >[6134863.531243] watchdog: BUG: soft lockup - CPU#14 stuck for 22s! [kubelet:18890] >[6134863.532023] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6134863.538599] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6134863.544424] CPU: 14 PID: 18890 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6134863.545428] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6134863.546321] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6134863.546834] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6134863.547349] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6134863.547854] sp : ffff00001fd9bc20 >[6134863.548177] x29: ffff00001fd9bc20 x28: ffff8bac40d8c000 >[6134863.548690] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6134863.549210] x25: 0000000000000010 x24: 0000000000000000 >[6134863.549816] x23: 0000000000000020 x22: ffff59fd7fe49800 >[6134863.550414] x21: ffff8b7b6a6eac00 x20: ffff59fd7fe4b8a8 >[6134863.550955] x19: 0000000000000004 x18: 0000000000000000 >[6134863.551468] x17: 0000000000000002 x16: 0000000000000000 >[6134863.551987] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6134863.552504] x13: ffff8b8d722d8000 x12: ffff8b8d722d7347 >[6134863.553029] x11: 0000000000000000 x10: ffff00001fd9bc80 >[6134863.553526] x9 : ffff59fd7ef3dad0 x8 : 0000000000000021 >[6134863.554041] x7 : 0000000000000001 x6 : 0000000000000002 >[6134863.554652] x5 : 00000000fffffffa x4 : ffffffffffffffff >[6134863.555153] x3 : 0000000000000012 x2 : 000033a040443000 >[6134863.555714] x1 : ffff4a0cefb1e100 x0 : 0000000000000052 >[6134863.556238] Call trace: >[6134863.556482] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6134863.557057] memcg_numa_stat_show+0x1d0/0x260 >[6134863.557512] cgroup_seqfile_show+0x50/0xd0 >[6134863.557918] kernfs_seq_show+0x30/0x38 >[6134863.558313] seq_read+0x148/0x478 >[6134863.558639] kernfs_fop_read+0x2c/0x1e0 >[6134863.559035] __vfs_read+0x20/0x48 >[6134863.559333] vfs_read+0x98/0x168 >[6134863.559756] ksys_read+0x6c/0xe8 >[6134863.560115] __arm64_sys_read+0x20/0x28 >[6134863.560487] el0_svc_common.constprop.0+0xa8/0x200 >[6134863.560994] el0_svc_handler+0x30/0x80 >[6134863.561348] el0_svc+0x10/0x14 >[6134903.531152] watchdog: BUG: soft lockup - CPU#8 stuck for 23s! [kubelet:4832] >[6134903.531911] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6134903.538470] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6134903.544301] CPU: 8 PID: 4832 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6134903.545307] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6134903.546212] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6134903.546687] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6134903.547259] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6134903.547796] sp : ffff00005e9c3c20 >[6134903.548125] x29: ffff00005e9c3c20 x28: ffff8bcc6a0dc000 >[6134903.548613] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6134903.549110] x25: 0000000000000003 x24: 0000000000000000 >[6134903.549767] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6134903.550295] x21: ffff8d9c8c9c8800 x20: ffff59fd7fe4b8a8 >[6134903.550797] x19: 0000000000000000 x18: 0000000000000000 >[6134903.551315] x17: 0000000000000002 x16: 0000000000000000 >[6134903.551867] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6134903.552439] x13: ffff8b8d72fab000 x12: ffff8b8d72faa2fe >[6134903.552945] x11: 0000000000000000 x10: ffff00005e9c3c80 >[6134903.553510] x9 : ffff59fd7ef3dad0 x8 : 0000000000014cde >[6134903.553998] x7 : 0000000000000004 x6 : 0000000000000005 >[6134903.554602] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6134903.555118] x3 : 000000000000000a x2 : 0000319040443000 >[6134903.555643] x1 : ffff4a2de4569df0 x0 : 000000000000000a >[6134903.556137] Call trace: >[6134903.556384] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6134903.556852] memcg_numa_stat_show+0x1d0/0x260 >[6134903.557269] cgroup_seqfile_show+0x50/0xd0 >[6134903.557658] kernfs_seq_show+0x30/0x38 >[6134903.558018] seq_read+0x148/0x478 >[6134903.558388] kernfs_fop_read+0x2c/0x1e0 >[6134903.558804] __vfs_read+0x20/0x48 >[6134903.559134] vfs_read+0x98/0x168 >[6134903.559589] ksys_read+0x6c/0xe8 >[6134903.559916] __arm64_sys_read+0x20/0x28 >[6134903.560344] el0_svc_common.constprop.0+0xa8/0x200 >[6134903.560832] el0_svc_handler+0x30/0x80 >[6134903.561239] el0_svc+0x10/0x14 >[6134907.531070] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [migration/3:26] >[6134907.531930] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6134907.538486] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6134907.544359] CPU: 3 PID: 26 Comm: migration/3 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6134907.545366] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6134907.546228] pstate: 60000005 (nZCv daif -PAN -UAO) >[6134907.546689] pc : multi_cpu_stop+0xf4/0x180 >[6134907.547072] lr : multi_cpu_stop+0xb4/0x180 >[6134907.547513] sp : ffff000008dabd50 >[6134907.547831] x29: ffff000008dabd50 x28: 0000000000000000 >[6134907.548314] x27: ffff0000329ab938 x26: 0000000000000060 >[6134907.548857] x25: 0000000000000000 x24: 0000000000000000 >[6134907.549437] x23: 0000000000000000 x22: 0000000000000001 >[6134907.549941] x21: ffff0000329ab9d4 x20: ffff0000329ab9b0 >[6134907.550481] x19: 0000000000000001 x18: 0000000000000000 >[6134907.551058] x17: 0000000000000001 x16: 0000000000000000 >[6134907.551553] x15: 0000000000000000 x14: 0000000000000000 >[6134907.552140] x13: 0000000000000000 x12: 0000000000000000 >[6134907.552637] x11: 00000000000000be x10: 0000000000000d60 >[6134907.553170] x9 : ffff59fd7ede5a0c x8 : ffff8b7d9da18dc0 >[6134907.553666] x7 : 000b888bff07cb6e x6 : 0000000000000027 >[6134907.554252] x5 : 00000000fa83b2da x4 : ffff8b7d9da18600 >[6134907.554812] x3 : dead000000000200 x2 : ffff0000329ab960 >[6134907.555396] x1 : ffff0000329ab9d4 x0 : 0000000000000001 >[6134907.555940] Call trace: >[6134907.556186] multi_cpu_stop+0xf4/0x180 >[6134907.556579] cpu_stopper_thread+0xec/0x178 >[6134907.557064] smpboot_thread_fn+0x158/0x1d8 >[6134907.557506] kthread+0x130/0x138 >[6134947.531168] watchdog: BUG: soft lockup - CPU#9 stuck for 23s! [kubelet:57297] >[6134947.532004] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6134947.538549] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6134947.544463] CPU: 9 PID: 57297 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6134947.545484] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6134947.546371] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6134947.546836] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6134947.547338] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6134947.547846] sp : ffff00003f4f3c20 >[6134947.548181] x29: ffff00003f4f3c20 x28: ffff8b7bbaea4000 >[6134947.548703] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6134947.549311] x25: 0000000000000003 x24: 0000000000000000 >[6134947.549833] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6134947.550350] x21: ffff8dcd32cdec00 x20: ffff59fd7fe4b8a8 >[6134947.550881] x19: 0000000000000000 x18: 0000000000000000 >[6134947.551438] x17: 0000000000000002 x16: 0000000000000000 >[6134947.551956] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6134947.552518] x13: ffff8b9d205bb000 x12: ffff8b9d205ba31d >[6134947.552995] x11: 0000000000000000 x10: ffff00003f4f3c80 >[6134947.553493] x9 : ffff59fd7ef3dad0 x8 : 00000000000188ec >[6134947.554005] x7 : 0000000000000005 x6 : 0000000000000006 >[6134947.554529] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6134947.555018] x3 : 000000000000000a x2 : 0000319040443000 >[6134947.555568] x1 : ffff4a1cf7431ac0 x0 : 000000000000000a >[6134947.556167] Call trace: >[6134947.556412] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6134947.557001] memcg_numa_stat_show+0x1d0/0x260 >[6134947.557424] cgroup_seqfile_show+0x50/0xd0 >[6134947.557870] kernfs_seq_show+0x30/0x38 >[6134947.558282] seq_read+0x148/0x478 >[6134947.558647] kernfs_fop_read+0x2c/0x1e0 >[6134947.559108] __vfs_read+0x20/0x48 >[6134947.559441] vfs_read+0x98/0x168 >[6134947.559761] ksys_read+0x6c/0xe8 >[6134947.560101] __arm64_sys_read+0x20/0x28 >[6134947.560525] el0_svc_common.constprop.0+0xa8/0x200 >[6134947.561012] el0_svc_handler+0x30/0x80 >[6134947.561376] el0_svc+0x10/0x14 >[6134991.531648] watchdog: BUG: soft lockup - CPU#40 stuck for 23s! [kubelet:11266] >[6134991.532613] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6134991.539152] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6134991.544886] CPU: 40 PID: 11266 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6134991.545873] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6134991.546756] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6134991.547280] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6134991.547808] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6134991.548329] sp : ffff0000545dbc20 >[6134991.548768] x29: ffff0000545dbc20 x28: ffff8bad0257a000 >[6134991.549285] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6134991.549802] x25: 0000000000000010 x24: 0000000000000000 >[6134991.550349] x23: 0000000000000020 x22: ffff59fd7fe49800 >[6134991.550855] x21: ffff8b7bdcf86400 x20: ffff59fd7fe4b8a8 >[6134991.551323] x19: 0000000000000004 x18: 0000000000000000 >[6134991.551822] x17: 0000000000000002 x16: 0000000000000000 >[6134991.552394] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6134991.552953] x13: ffff8bcbee288000 x12: ffff8bcbee287347 >[6134991.553480] x11: 0000000000000000 x10: ffff0000545dbc80 >[6134991.554086] x9 : ffff59fd7ef3dad0 x8 : 0000000000000021 >[6134991.554585] x7 : 0000000000000001 x6 : 0000000000000002 >[6134991.555161] x5 : 00000000fffffffa x4 : ffffffffffffffff >[6134991.555675] x3 : 0000000000000000 x2 : 000033804040f000 >[6134991.556163] x1 : ffff4a0f23382c90 x0 : 0000000000000040 >[6134991.556702] Call trace: >[6134991.556941] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6134991.557432] memcg_numa_stat_show+0x1d0/0x260 >[6134991.557854] cgroup_seqfile_show+0x50/0xd0 >[6134991.558298] kernfs_seq_show+0x30/0x38 >[6134991.558754] seq_read+0x148/0x478 >[6134991.559122] kernfs_fop_read+0x2c/0x1e0 >[6134991.559501] __vfs_read+0x20/0x48 >[6134991.559853] vfs_read+0x98/0x168 >[6134991.560163] ksys_read+0x6c/0xe8 >[6134991.560476] __arm64_sys_read+0x20/0x28 >[6134991.560849] el0_svc_common.constprop.0+0xa8/0x200 >[6134991.561299] el0_svc_handler+0x30/0x80 >[6134991.561645] el0_svc+0x10/0x14 >[6134995.531083] watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [migration/4:31] >[6134995.531893] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6134995.539355] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6134995.545195] CPU: 4 PID: 31 Comm: migration/4 Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6134995.546218] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6134995.547071] pstate: 60000005 (nZCv daif -PAN -UAO) >[6134995.547561] pc : multi_cpu_stop+0xf4/0x180 >[6134995.547951] lr : multi_cpu_stop+0xb4/0x180 >[6134995.548358] sp : ffff000008dd3d50 >[6134995.548783] x29: ffff000008dd3d50 x28: 0000000000000000 >[6134995.549286] x27: ffff00003dc03938 x26: 0000000000000060 >[6134995.549834] x25: 0000000000000000 x24: 0000000000000000 >[6134995.550383] x23: 0000000000000000 x22: 0000000000000001 >[6134995.550887] x21: ffff00003dc039d4 x20: ffff00003dc039b0 >[6134995.551431] x19: 0000000000000001 x18: 0000000000000000 >[6134995.552010] x17: 0000000000000001 x16: 0000000000000000 >[6134995.552578] x15: 0000000000000000 x14: 0000000000000000 >[6134995.553115] x13: 0000000000000000 x12: 0000000000000000 >[6134995.553684] x11: 0000000000000000 x10: 0000000000000d60 >[6134995.554190] x9 : ffff59fd7ede5a0c x8 : ffff8b7d9da1e400 >[6134995.554748] x7 : ffff8b7d6b1b3800 x6 : 0000000000000157 >[6134995.555232] x5 : 0000318040473000 x4 : ffff8b7d9da1dc40 >[6134995.555772] x3 : dead000000000200 x2 : ffff00003dc03960 >[6134995.556276] x1 : ffff00003dc039d4 x0 : 0000000000000001 >[6134995.556793] Call trace: >[6134995.557036] multi_cpu_stop+0xf4/0x180 >[6134995.557402] cpu_stopper_thread+0xec/0x178 >[6134995.557805] smpboot_thread_fn+0x158/0x1d8 >[6134995.558217] kthread+0x130/0x138 >[6135031.531153] watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [kubelet:18354] >[6135031.531963] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6135031.539434] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6135031.545815] CPU: 8 PID: 18354 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6135031.546907] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6135031.547873] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6135031.548495] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6135031.549093] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6135031.549636] sp : ffff00001f9dbc20 >[6135031.549994] x29: ffff00001f9dbc20 x28: ffff8b7d42c1a000 >[6135031.550559] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6135031.551130] x25: 0000000000000003 x24: 0000000000000000 >[6135031.551697] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6135031.552248] x21: ffff8dad919d8800 x20: ffff59fd7fe4b8a8 >[6135031.552811] x19: 0000000000000000 x18: 0000000000000000 >[6135031.553368] x17: 0000000000000002 x16: 0000000000000000 >[6135031.553928] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6135031.554492] x13: ffff8b8d72fab000 x12: ffff8b8d72faa308 >[6135031.555055] x11: 0000000000000000 x10: ffff00001f9dbc80 >[6135031.555620] x9 : ffff59fd7ef3dad0 x8 : 000000000000c2f1 >[6135031.556177] x7 : 0000000000000004 x6 : 0000000000000005 >[6135031.556737] x5 : 00000000fffffff7 x4 : ffffffffffffffff >[6135031.557293] x3 : 0000000000000002 x2 : 0000338040441000 >[6135031.557865] x1 : ffff4a2f64eb3c00 x0 : 0000000000000042 >[6135031.558524] Call trace: >[6135031.558828] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6135031.559315] memcg_numa_stat_show+0x1d0/0x260 >[6135031.559774] cgroup_seqfile_show+0x50/0xd0 >[6135031.560186] kernfs_seq_show+0x30/0x38 >[6135031.560609] seq_read+0x148/0x478 >[6135031.560942] kernfs_fop_read+0x2c/0x1e0 >[6135031.561378] __vfs_read+0x20/0x48 >[6135031.561720] vfs_read+0x98/0x168 >[6135031.562036] ksys_read+0x6c/0xe8 >[6135031.562367] __arm64_sys_read+0x20/0x28 >[6135031.562760] el0_svc_common.constprop.0+0xa8/0x200 >[6135031.563225] el0_svc_handler+0x30/0x80 >[6135031.563598] el0_svc+0x10/0x14 >[6135075.531153] watchdog: BUG: soft lockup - CPU#8 stuck for 23s! [kubelet:18291] >[6135075.531945] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6135075.538702] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6135075.544578] CPU: 8 PID: 18291 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6135075.545552] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6135075.546427] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6135075.546882] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6135075.547467] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6135075.547974] sp : ffff00001f8ebc20 >[6135075.548408] x29: ffff00001f8ebc20 x28: ffff8bcc6a73c000 >[6135075.548924] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6135075.549426] x25: 0000000000000003 x24: 0000000000000000 >[6135075.549937] x23: 0000000000000008 x22: ffff59fd7fe49800 >[6135075.550463] x21: ffff8dccf032a400 x20: ffff59fd7fe4b8a8 >[6135075.550998] x19: 0000000000000001 x18: 0000000000000000 >[6135075.551518] x17: 0000000000000002 x16: 0000000000000000 >[6135075.552045] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6135075.552600] x13: ffff8b8d72fa9000 x12: ffff8b8d72fa831d >[6135075.553118] x11: 0000000000000000 x10: ffff00001f8ebc80 >[6135075.553633] x9 : ffff59fd7ef3dad0 x8 : 0000000000018d61 >[6135075.554137] x7 : 0000000000000005 x6 : 0000000000000006 >[6135075.554657] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6135075.555183] x3 : 000000000000002e x2 : 000031d0404a7000 >[6135075.555707] x1 : ffff4a2dc77afcd8 x0 : 000000000000002e >[6135075.556210] Call trace: >[6135075.556447] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6135075.556954] memcg_numa_stat_show+0x1d0/0x260 >[6135075.557396] cgroup_seqfile_show+0x50/0xd0 >[6135075.557803] kernfs_seq_show+0x30/0x38 >[6135075.558269] seq_read+0x148/0x478 >[6135075.558637] kernfs_fop_read+0x2c/0x1e0 >[6135075.559028] __vfs_read+0x20/0x48 >[6135075.559397] vfs_read+0x98/0x168 >[6135075.559717] ksys_read+0x6c/0xe8 >[6135075.560071] __arm64_sys_read+0x20/0x28 >[6135075.560468] el0_svc_common.constprop.0+0xa8/0x200 >[6135075.560998] el0_svc_handler+0x30/0x80 >[6135075.561364] el0_svc+0x10/0x14 >[6135115.531152] watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [kubelet:18888] >[6135115.531920] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6135115.538411] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6135115.544109] CPU: 8 PID: 18888 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6135115.545078] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6135115.545950] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6135115.546414] pc : mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6135115.546924] lr : mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6135115.547417] sp : ffff00001fec3c20 >[6135115.547842] x29: ffff00001fec3c20 x28: ffff8bbb56f9e000 >[6135115.548355] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6135115.548867] x25: 0000000000000003 x24: 0000000000000000 >[6135115.549377] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6135115.549860] x21: ffff8d8cc8837800 x20: ffff59fd7fe4b8a8 >[6135115.550349] x19: 0000000000000000 x18: 0000000000000000 >[6135115.550841] x17: 0000000000000004 x16: 0000000000000000 >[6135115.551323] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6135115.551867] x13: ffff8b8d72fab000 x12: ffff8b8d72faa2f4 >[6135115.552358] x11: 0000000000000000 x10: ffff00001fec3c80 >[6135115.552877] x9 : ffff59fd7ef3dad0 x8 : 000000000003773c >[6135115.553383] x7 : 0000000000000005 x6 : 0000000000000006 >[6135115.553906] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6135115.554410] x3 : 0000000000000037 x2 : 000033d01fffc000 >[6135115.554924] x1 : ffff4a0ca2b1f260 x0 : 0000000000000077 >[6135115.555412] Call trace: >[6135115.555656] mem_cgroup_node_nr_lru_pages+0xac/0x128 >[6135115.556127] memcg_numa_stat_show+0x1d0/0x260 >[6135115.556588] cgroup_seqfile_show+0x50/0xd0 >[6135115.557020] kernfs_seq_show+0x30/0x38 >[6135115.557426] seq_read+0x148/0x478 >[6135115.557755] kernfs_fop_read+0x2c/0x1e0 >[6135115.558252] __vfs_read+0x20/0x48 >[6135115.558581] vfs_read+0x98/0x168 >[6135115.558930] ksys_read+0x6c/0xe8 >[6135115.559293] __arm64_sys_read+0x20/0x28 >[6135115.559667] el0_svc_common.constprop.0+0xa8/0x200 >[6135115.560213] el0_svc_handler+0x30/0x80 >[6135115.560569] el0_svc+0x10/0x14 >[6135159.531259] watchdog: BUG: soft lockup - CPU#15 stuck for 22s! [kubelet:60635] >[6135159.532096] Modules linked in: tcp_diag(E) inet_diag(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) rpcsec_gss_krb5(E) ipt_REJECT(E) nf_reject_ipv4(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) xt_multiport(E) ipt_rpfilter(E) xt_set(E) ip6_tables(E) ip6t_rpfilter(E) ip_set_hash_ip(E) ip_set_hash_net(E) xt_statistic(E) ip6t_MASQUERADE(E) xt_comment(E) xt_mark(E) ip_vs_sh(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs(E) xt_nat(E) veth(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nfsd(E) auth_rpcgss(E) lockd(E) grace(E) sunrpc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nft_chain_route_ipv4(E) >[6135159.538840] ip_set(E) nf_tables(E) nfnetlink(E) vfat(E) fat(E) ipmi_ssif(E) mousedev(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) crc32_ce(E) crct10dif_ce(E) ses(E) ghash_ce(E) enclosure(E) sha2_ce(E) scsi_transport_sas(E) sha256_arm64(E) sha1_ce(E) sbsa_gwdt(E) ipmi_si(E) sch_fq_codel(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ast(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) nvme(E) nvme_core(E) mlx5_core(E) drm(E) igb(E) mlxfw(E) megaraid_sas(E) i2c_algo_bit(E) i2c_designware_platform(E) devlink(E) i2c_designware_core(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E) [last unloaded: nft_reject] >[6135159.544445] CPU: 15 PID: 60635 Comm: kubelet Kdump: loaded Tainted: G EL 4.19.91-26.6.2.kos5.aarch64 #1 >[6135159.545538] Hardware name: GreatWall \xe6\x93\x8e天EF860/GW-748F2A-FTG, BIOS KunLun BIOS V4.0 2021-08-25 >[6135159.546389] pstate: 80000005 (Nzcv daif -PAN -UAO) >[6135159.546828] pc : find_next_bit+0x24/0x68 >[6135159.547219] lr : cpumask_next+0x1c/0x28 >[6135159.547715] sp : ffff00001df63c10 >[6135159.548036] x29: ffff00001df63c10 x28: ffff8b9c1dad8000 >[6135159.548587] x27: ffff59fd7fe4bbf4 x26: 0000000000000000 >[6135159.549151] x25: 0000000000000003 x24: 0000000000000000 >[6135159.549724] x23: 0000000000000000 x22: ffff59fd7fe49800 >[6135159.550204] x21: ffff8dbc20677c00 x20: ffff59fd7fe4b8a8 >[6135159.550725] x19: 0000000000000000 x18: 0000000000000000 >[6135159.551250] x17: 0000000000000002 x16: 0000000000000000 >[6135159.551743] x15: ffffffffffffffff x14: ffff59fd7fe49748 >[6135159.552240] x13: ffff8b8bdf96c000 x12: ffff8b8bdf96b312 >[6135159.552744] x11: 0000000000000000 x10: ffff00001df63c80 >[6135159.553254] x9 : ffff59fd7ef3dad0 x8 : 000000000004eecf >[6135159.553781] x7 : 0000000000000005 x6 : 0000000000000006 >[6135159.554297] x5 : 00000000fffffff6 x4 : ffffffffffffffff >[6135159.554789] x3 : ffffffffffff8000 x2 : 0000000000000000 >[6135159.555295] x1 : 0000000000000400 x0 : ffff59fd7fe4b8a8 >[6135159.555824] Call trace: >[6135159.556070] find_next_bit+0x24/0x68 >[6135159.556425] mem_cgroup_node_nr_lru_pages+0xbc/0x128 >[6135159.556898] memcg_numa_stat_show+0x1d0/0x260 >[6135159.557325] cgroup_seqfile_show+0x50/0xd0 >[6135159.557851] kernfs_seq_show+0x30/0x38 >[6135159.558213] seq_read+0x148/0x478 >[6135159.558595] kernfs_fop_read+0x2c/0x1e0 >[6135159.559039] __vfs_read+0x20/0x48 >[6135159.559354] vfs_read+0x98/0x168 >[6135159.559663] ksys_read+0x6c/0xe8 >[6135159.560025] __arm64_sys_read+0x20/0x28 >[6135159.560415] el0_svc_common.constprop.0+0xa8/0x200 >[6135159.560947] el0_svc_handler+0x30/0x80 >[6135159.561320] el0_svc+0x10/0x14
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Raw
Actions:
View
Attachments on
bug 6151
: 870