Description of problem: 4.19 x86内核上ltp测试套ftrace_stress_test.sh导致crash：Kernel panic - not syncing: softlockup: hung tasks 复现几率：不确定（1/3） 之前5.10 arm内核上同一用例也产生crash，但看起来好像不是一个问题，仅供参考 https://bugzilla.openanolis.cn/show_bug.cgi?id=408 Version-Release number of selected component (if applicable): # uname -a Linux i22e11409.eu95sqa 4.19.91-224.git.e0d4c9266.an8.x86_64 #1 SMP Mon Feb 7 13:32:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux [root@i22e11409 ltp]# cat /etc/os-release NAME="Anolis OS" VERSION="8.2" ID="anolis" ID_LIKE="rhel fedora centos" VERSION_ID="8.2" PLATFORM_ID="platform:an8" PRETTY_NAME="Anolis OS 8.2" ANSI_COLOR="0;31" HOME_URL="https://openanolis.cn/" How reproducible: 1.git clone https://github.com/linux-test-project/ltp.git cd ltp make autotools ./configure make make install 2. ./runltp -f tracing -s ftrace-stress-test Actual results: vmcore解析信息如下： # crash -x /usr/lib/debug/usr/lib/modules/4.19.91-224.git.e0d4c9266.an8.x86_64/vmlinux vmcore crash 7.2.7-3.el8.1 Copyright (C) 2002-2020 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb (GDB) 7.6 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-unknown-linux-gnu"... WARNING: kernel relocated [880MB]: patching 99399 gdb minimal_symbol values KERNEL: /usr/lib/debug/usr/lib/modules/4.19.91-224.git.e0d4c9266.an8.x86_64/vmlinux DUMPFILE: vmcore [PARTIAL DUMP] CPUS: 96 DATE: Tue Feb 8 09:35:22 2022 UPTIME: 11:21:07 LOAD AVERAGE: 6.65, 9.52, 25.52 TASKS: 1034 NODENAME: i22e11409.eu95sqa RELEASE: 4.19.91-224.git.e0d4c9266.an8.x86_64 VERSION: #1 SMP Mon Feb 7 13:32:17 UTC 2022 MACHINE: x86_64 (2500 Mhz) MEMORY: 767.4 GB PANIC: "Kernel panic - not syncing: softlockup: hung tasks" PID: 829701 COMMAND: "sh" TASK: ffff953f730bc180 [THREAD_INFO: ffff953f730bc180] CPU: 14 STATE: TASK_RUNNING (PANIC) crash> bt PID: 829701 TASK: ffff953f730bc180 CPU: 14 COMMAND: "sh" #0 [ffff9540aef83c68] ftrace_location_range at ffffffffb818411a #1 [ffff9540aef83ca0] __crash_kexec at ffffffffb81478ea #2 [ffff9540aef83d68] machine_kexec at ffffffffb806480a #3 [ffff9540aef83db8] __crash_kexec at ffffffffb81478ea #4 [ffff9540aef83e78] panic at ffffffffb80a1195 #5 [ffff9540aef83f20] __hrtimer_run_queues at ffffffffb8129ff0 #6 [ffff9540aef83f78] hrtimer_interrupt at ffffffffb812aae0 #7 [ffff9540aef83fd8] smp_apic_timer_interrupt at ffffffffb8a025aa #8 [ffff9540aef83ff0] apic_timer_interrupt at ffffffffb8a01b0f --- <IRQ stack> --- #9 [ffffbceadb537c18] apic_timer_interrupt at ffffffffb8a01b0f [exception RIP: trace_event_buffer_reserve+77] RIP: ffffffffb81a29cd RSP: ffffbceadb537cc0 RFLAGS: 00000246 RAX: 0000000000000246 RBX: ffffbceadb537cd0 RCX: 0000000000000020 RDX: 0000000000000020 RSI: ffff9540ae876bb0 RDI: ffffbceadb537cd0 RBP: ffffef6b91d6c640 R8: 0000000000000246 R9: 0000000000000000 R10: ffffffffb952cfe0 R11: ffff9543affd5dc0 R12: 0000000000000000 R13: 0000000000000000 R14: 00000000006280c0 R15: ffff9540ae876bb0 ORIG_RAX: ffffffffffffff13 CS: 0010 SS: 0018 #10 [ffffbceadb537cc8] trace_event_raw_event_mm_page_alloc at ffffffffb822f765 #11 [ffffbceadb537d30] __alloc_pages_nodemask at ffffffffb820ba19 #12 [ffffbceadb537e68] __get_free_pages at ffffffffb820652d #13 [ffffbceadb537e70] ftrace_profile_pages_init at ffffffffb8183766 #14 [ffffbceadb537e98] ftrace_profile_write at ffffffffb81884f0 #15 [ffffbceadb537ed0] vfs_write at ffffffffb82b32f9 #16 [ffffbceadb537f00] ksys_write at ffffffffb82b3563 #17 [ffffbceadb537f30] do_syscall_64 at ffffffffb80040ff #18 [ffffbceadb537f50] entry_SYSCALL_64_after_hwframe at ffffffffb8a00085 RIP: 00007fa04aa09b28 RSP: 00007ffeba75ed38 RFLAGS: 00000246 RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007fa04aa09b28 RDX: 0000000000000002 RSI: 000056066f14e5a0 RDI: 0000000000000001 RBP: 000056066f14e5a0 R8: 000000000000000a R9: 00007fa04aa9ac80 R10: 000000000000000a R11: 0000000000000246 R12: 00007fa04acda6c0 R13: 0000000000000002 R14: 00007fa04acd5880 R15: 0000000000000002 ORIG_RAX: 0000000000000001 CS: 0033 SS: 002b 部分dmesg信息如下，具体信息见附件vmcore-dmesg.txt： [40845.732064] LTP: starting ftrace-stress-test (ftrace_stress_test.sh 90) [40870.210853] watchdog: BUG: soft lockup - CPU#14 stuck for 23s! [sh:829701] [40870.223568] Modules linked in: brd(E) nft_fib_inet(E) nft_fib_ipv4(E) nft_fib_ipv6(E) nft_fib(E) nft_reject_inet(E) nf_reject_ipv4(E) nf_reject_ipv6(E) nft_reject(E) nft_ct(E) nf_tables_set(E) ip6_tables(E) ip_tables(E) nft_limit(E) xt_limit(E) xt_multiport(E) xt_LOG(E) nft_compat(E) nft_counter(E) rpcsec_gss_krb5(E) nfsv4(E) dns_resolver(E) nfsv3(E) nfs(E) fscache(E) mpls_gso(E) mpls_iptunnel(E) mpls_router(E) sit(E) geneve(E) ipvlan(E) ah4(E) macvtap(E) tap(E) macvlan(E) dccp_ipv6(E) dccp_ipv4(E) dccp(E) sch_sfq(E) sch_sfb(E) sch_prio(E) sch_pie(E) sch_hhf(E) sch_hfsc(E) sch_fq(E) sch_codel(E) sch_netem(E) tcp_bbr(E) sctp(E) nfsd(E) auth_rpcgss(E) nfs_acl(E) lockd(E) grace(E) sunrpc(E) btrfs(E) xor(E) zstd_decompress(E) zstd_compress(E) xxhash(E) raid6_pq(E) nls_utf8(E) isofs(E) fuse(E) bpfilter(E) [40870.335619] pcrypt(E) crypto_user(E) authenc(E) vmac(E) chacha20_x86_64(E) chacha20_generic(E) poly1305_x86_64(E) poly1305_generic(E) chacha20poly1305(E) salsa20_generic(E) sha3_generic(E) tun(E) msdos(E) vfat(E) fat(E) xfs(E) xfrm4_mode_tunnel(E) ip_vti(E) tunnel4(E) esp6(E) xfrm6_mode_tunnel(E) ip6_vti(E) esp4_offload(E) esp4(E) xfrm4_mode_transport(E) macsec(E) vxlan(E) ip6_udp_tunnel(E) udp_tunnel(E) 8021q(E) garp(E) mrp(E) bridge(E) stp(E) llc(E) ip6_gre(E) ip6_tunnel(E) tunnel6(E) ip_gre(E) gre(E) ip_tunnel(E) cls_u32(E) sch_htb(E) dummy(E) tls(E) raid0(E) dm_mod(E) bcache(E) crc64(E) veth(E) overlay(E) squashfs(E) loop(E) binfmt_misc(E) bonding(E) nft_chain_nat_ipv6(E) nf_nat_ipv6(E) nft_chain_route_ipv6(E) nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) [40870.449276] libcrc32c(E) nft_chain_route_ipv4(E) ip_set(E) nf_tables(E) nfnetlink(E) intel_rapl_msr(E) iTCO_wdt(E) intel_rapl_common(E) iTCO_vendor_support(E) isst_if_common(E) skx_edac(E) nfit(E) ipmi_ssif(E) x86_pkg_temp_thermal(E) intel_powerclamp(E) coretemp(E) kvm_intel(E) kvm(E) irqbypass(E) crct10dif_pclmul(E) crc32_pclmul(E) ghash_clmulni_intel(E) pcbc(E) aesni_intel(E) glue_helper(E) pcspkr(E) mei_me(E) ipmi_si(E) joydev(E) ioatdma(E) mousedev(E) ipmi_devintf(E) i2c_i801(E) lpc_ich(E) mei(E) dca(E) wmi(E) ipmi_msghandler(E) pcc_cpufreq(E) acpi_power_meter(E) acpi_pad(E) sch_fq_codel(E) sd_mod(E) sg(E) ast(E) i2c_algo_bit(E) drm_kms_helper(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) fb_sys_fops(E) ttm(E) crc32c_intel(E) nvme(E) mlx5_core(E) ahci(E) nvme_core(E) mlxfw(E) drm(E) libahci(E) [40870.562519] i2c_core(E) libata(E) [last unloaded: dummy_del_mod] [40870.574736] CPU: 14 PID: 829701 Comm: sh Kdump: loaded Tainted: P W OEL 4.19.91-224.git.e0d4c9266.an8.x86_64 #1 [40870.591969] Hardware name: Foxconn AliServer Thor04-12U/Thunder, BIOS GB1A168F 12/16/2020 [40870.606346] RIP: 0010:trace_event_buffer_reserve+0x4d/0x80 [40870.617998] Code: 47 80 78 78 00 75 4b 48 89 d1 48 89 fb 9c 58 0f 1f 44 00 00 49 89 c0 48 89 43 20 65 44 8b 0d 4a 31 e7 47 41 81 e1 ff ff ff 7f <48> 89 77 10 44 89 4f 28 41 8b 52 40 e8 c2 b3 fe ff 48 89 43 08 48 [40870.649834] RSP: 0018:ffffbceadb537cc0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 [40870.664129] RAX: 0000000000000246 RBX: ffffbceadb537cd0 RCX: 0000000000000020 [40870.678018] RDX: 0000000000000020 RSI: ffff9540ae876bb0 RDI: ffffbceadb537cd0 [40870.691878] RBP: ffffef6b91d6c640 R08: 0000000000000246 R09: 0000000000000000 [40870.705746] R10: ffffffffb952cfe0 R11: ffff9543affd5dc0 R12: 0000000000000000 [40870.719639] R13: 0000000000000000 R14: 00000000006280c0 R15: ffff9540ae876bb0 [40870.733561] FS: 00007fa04b326740(0000) GS:ffff9540aef80000(0000) knlGS:0000000000000000 [40870.748504] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [40870.761075] CR2: 000056066f14f5a8 CR3: 000000bbac01a002 CR4: 00000000007706e0 [40870.775114] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [40870.789233] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [40870.803384] PKRU: 55555554 [40870.813017] Call Trace: [40870.822344] trace_event_raw_event_mm_page_alloc+0x45/0xb0 [40870.834699] ? get_page_from_freelist+0x5/0x1750 [40870.846144] __alloc_pages_nodemask+0x249/0xff0 [40870.857489] ? gfp_pfmemalloc_allowed+0x70/0x70 [40870.868786] ? __get_free_pages+0xd/0x30 [40870.879463] ? do_int3+0xb8/0x190 [40870.889508] ? ftrace_ops_list_func+0xd0/0x130 [40870.900701] ? ftrace_regs_call+0x5/0x72 [40870.911383] ? __alloc_pages_nodemask+0x5/0xff0 [40870.922580] __get_free_pages+0xd/0x30 [40870.932940] ftrace_profile_pages_init+0x76/0xd0 [40870.944178] ftrace_profile_write+0x190/0x1f0 [40870.955161] vfs_write+0xa9/0x1a0 [40870.965049] ksys_write+0x43/0xb0 [40870.974857] do_syscall_64+0x5f/0x1b0 [40870.984958] ? prepare_exit_to_usermode+0x4c/0xb0 [40870.996106] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [40871.007572] RIP: 0033:0x7fa04aa09b28 [40871.017541] Code: 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 35 4b 2d 00 8b 00 85 c0 75 17 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 58 c3 0f 1f 80 00 00 00 00 41 54 49 89 d4 55 [40871.049925] RSP: 002b:00007ffeba75ed38 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [40871.064453] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007fa04aa09b28 [40871.078541] RDX: 0000000000000002 RSI: 000056066f14e5a0 RDI: 0000000000000001 [40871.092669] RBP: 000056066f14e5a0 R08: 000000000000000a R09: 00007fa04aa9ac80 [40871.106825] R10: 000000000000000a R11: 0000000000000246 R12: 00007fa04acda6c0 [40871.120998] R13: 0000000000000002 R14: 00007fa04acd5880 R15: 0000000000000002 [40871.135328] Kernel panic - not syncing: softlockup: hung tasks [40871.148173] CPU: 14 PID: 829701 Comm: sh Kdump: loaded Tainted: P W OEL 4.19.91-224.git.e0d4c9266.an8.x86_64 #1 [40871.166267] Hardware name: Foxconn AliServer Thor04-12U/Thunder, BIOS GB1A168F 12/16/2020 [40871.181486] Call Trace: [40871.190894] <IRQ> [40871.199836] dump_stack+0x66/0x90 [40871.210064] panic+0xf9/0x25c [40871.219744] ? 0xffffffffc03a3001 [40871.229457] ? 0xffffffffc03a3001 [40871.239181] watchdog_timer_fn.cold.2+0x16/0x16 [40871.250095] ? report_softlockup+0x1a0/0x1a0 [40871.260688] __hrtimer_run_queues+0xf0/0x260 [40871.271300] hrtimer_interrupt+0x100/0x220 [40871.281710] smp_apic_timer_interrupt+0x6a/0x140 [40871.292558] apic_timer_interrupt+0xf/0x20 [40871.302849] </IRQ> [40871.311029] RIP: 0010:trace_event_buffer_reserve+0x4d/0x80 [40871.322597] Code: 47 80 78 78 00 75 4b 48 89 d1 48 89 fb 9c 58 0f 1f 44 00 00 49 89 c0 48 89 43 20 65 44 8b 0d 4a 31 e7 47 41 81 e1 ff ff ff 7f <48> 89 77 10 44 89 4f 28 41 8b 52 40 e8 c2 b3 fe ff 48 89 43 08 48 [40871.354036] RSP: 0018:ffffbceadb537cc0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 [40871.368070] RAX: 0000000000000246 RBX: ffffbceadb537cd0 RCX: 0000000000000020 [40871.381699] RDX: 0000000000000020 RSI: ffff9540ae876bb0 RDI: ffffbceadb537cd0 [40871.395360] RBP: ffffef6b91d6c640 R08: 0000000000000246 R09: 0000000000000000 [40871.409049] R10: ffffffffb952cfe0 R11: ffff9543affd5dc0 R12: 0000000000000000 [40871.422774] R13: 0000000000000000 R14: 00000000006280c0 R15: ffff9540ae876bb0 [40871.436685] trace_event_raw_event_mm_page_alloc+0x45/0xb0 [40871.448852] ? get_page_from_freelist+0x5/0x1750 [40871.460172] __alloc_pages_nodemask+0x249/0xff0 [40871.471380] ? gfp_pfmemalloc_allowed+0x70/0x70 [40871.482572] ? __get_free_pages+0xd/0x30 [40871.493151] ? do_int3+0xb8/0x190 [40871.503111] ? ftrace_ops_list_func+0xd0/0x130 [40871.514227] ? ftrace_regs_call+0x5/0x72 [40871.524883] ? __alloc_pages_nodemask+0x5/0xff0 [40871.536101] __get_free_pages+0xd/0x30 [40871.546494] ftrace_profile_pages_init+0x76/0xd0 [40871.557779] ftrace_profile_write+0x190/0x1f0 [40871.568794] vfs_write+0xa9/0x1a0 [40871.578750] ksys_write+0x43/0xb0 [40871.588673] do_syscall_64+0x5f/0x1b0 [40871.598662] ? prepare_exit_to_usermode+0x4c/0xb0 [40871.609411] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [40871.620485] RIP: 0033:0x7fa04aa09b28 [40871.630020] Code: 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 35 4b 2d 00 8b 00 85 c0 75 17 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 58 c3 0f 1f 80 00 00 00 00 41 54 49 89 d4 55 [40871.661438] RSP: 002b:00007ffeba75ed38 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [40871.675443] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007fa04aa09b28 [40871.689006] RDX: 0000000000000002 RSI: 000056066f14e5a0 RDI: 0000000000000001 [40871.702634] RBP: 000056066f14e5a0 R08: 000000000000000a R09: 00007fa04aa9ac80 [40871.716277] R10: 000000000000000a R11: 0000000000000246 R12: 00007fa04acda6c0 [40871.729950] R13: 0000000000000002 R14: 00007fa04acd5880 R15: 0000000000000002 [40871.918530] Kernel Offset: 0x37000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) Expected results: Additional info: # free -mh total used free shared buff/cache available Mem: 755Gi 3.6Gi 749Gi 18Mi 2.5Gi 747Gi Swap: 2.0Gi 0B 2.0Gi [root@i22e11409 ltp]# lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 96 On-line CPU(s) list: 0-95 Thread(s) per core: 2 Core(s) per socket: 24 Socket(s): 2 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Platinum 8163 CPU @ 2.50GHz Stepping: 4 CPU MHz: 1000.542 CPU max MHz: 3100.0000 CPU min MHz: 1000.0000 BogoMIPS: 5000.00 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 1024K L3 cache: 33792K NUMA node0 CPU(s): 0-95 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku ospke md_clear flush_l1d
Created attachment 147 [details] vmcore-dmesg
复现本问题的时候产生一个看起来不太一样的vmcore，具体信息见 https://bugzilla.openanolis.cn/show_bug.cgi?id=433
ftrace-test这个是一个已知问题，关闭问题