Bug 2264 - [Anolis8 4.19][debug-kernel][nightly][x86_64] rcutorture的cpuhotplug测试集下2个case fail
Summary: [Anolis8 4.19][debug-kernel][nightly][x86_64] rcutorture的cpuhotplug测试集下2个case...
Status: NEW
Alias: None
Product: Anolis OS 8
Classification: Anolis OS
Component: Others (show other bugs) Others
Version: 8.6
Hardware: x86_64 Linux
: P3-Medium S3-normal
Target Milestone: ---
Assignee: Jacob
QA Contact: shuming
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2022-09-23 15:25 UTC by zhixin01
Modified: 2022-11-24 16:07 UTC (History)
7 users (show)

See Also:

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description zhixin01 alibaba_cloud_group 2022-09-23 15:25:34 UTC 
[缺陷描述]：
Anolis8 ck-4.19 x86_64环境（4.19.91-467.git.d4d03de3a.an8.x86_64+debug）nightly测试，cpuhotplug测试集下cpuhotplug-busted和cpuhotplug-busted_srcud用例执行失败，报错日志如下:

cpuhotplug-busted用例执行过程dmesg部分失败日志如下：
[ 5894.695613] busted-torture: Stopping rcu_torture_cbflood task
[ 5894.702337] busted-torture: Stopping rcu_torture_cbflood task
[ 5894.709046] busted-torture: Stopping rcu_torture_cbflood task
[ 5894.715752] busted-torture: Stopping rcu_torture_cbflood task
[ 5894.722492] busted-torture: rtc:           (null) ver: 88788 tfle: 0 rta: 88788 rtaf: 0 rtf: 88779 rtmbe: 2065 rtbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 4442795 onoff: 4000/4000:4043/4043 24,889:23,6183 551850:651382 (HZ=1000) barrier: 0/0:0 cbflood: 0
[ 5894.748372] busted-torture: !!! Reader Pipe:  4463412474 62275 8482 385 73 34 12 11 3 4 2157
[ 5894.758138] busted-torture: Reader Batch:  4463485910 0 0 0 0 0 0 0 0 0 0
[ 5894.766249] busted-torture: Free-Block Circulation:  88787 88787 88786 88785 88784 88783 88782 88781 88780 88779 0
[ 5894.778027] busted-torture:--- End of test: FAILURE: nreaders=43 nfakewriters=4 stat_interval=60 verbose=1 test_no_idle_hz=1 shuffle_interval=3 stutter=5 irqreader=1 fqs_duration=0 fqs_holdoff=0 fqs_stutter=3 test_boost=1/0 test_boost_interval=7 test_boost_duration=4 shutdown_secs=0 stall_cpu=0 stall_cpu_holdoff=10 stall_cpu_irqsoff=0 n_barrier_cbs=0 onoff_interval=3 onoff_holdoff=30

cpuhotplug-busted_srcud执行过程dmesg部分报错日志如下：
[ 7251.952692] busted_srcud-torture: Stopping rcu_torture_cbflood task
[ 7251.952717] busted_srcud-torture: Stopping rcu_torture_cbflood task
[ 7251.952748] busted_srcud-torture: Stopping rcu_torture_cbflood task
[ 7251.952770] busted_srcud-torture: Stopping rcu_torture_cbflood task
[ 7251.952802] busted_srcud-torture: Stopping rcu_torture_cbflood task
[ 7251.952917] busted_srcud-torture: rtc:           (null) ver: 37084 tfle: 0 rta: 37084 rtaf: 0 rtf: 37075 rtmbe: 0 rtbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 4290216 onoff: 8565/8565:8615/8615 24,889:23,6183 1029810:1379210 (HZ=1000) barrier: 0/0:0 cbflood: 2643
[ 7251.952929] busted_srcud-torture: !!! Reader Pipe:  2943512888 986315 249 78 68 81 102 107 137 130 323
[ 7251.952980] busted_srcud-torture: Reader Batch:  2940575982 3915253 5011 714 741 1087 578 49 16 15 1032
[ 7251.952989] busted_srcud-torture: Free-Block Circulation:  37083 37083 37082 37081 37080 37079 37078 37077 37076 37075 0
[ 7251.953000] rcu: busted_srcud-torture: Tree SRCU g0 per-CPU(idx=0): 0(0,0 (null)) 1(0,0 (null)) 2(0,0 (null)) 3(0,0 (null)) 4(0,0 (null)) 5(0,0 (null)) 6(0,0 (null)) 7(0,0 (null)) 8(0,0 (null)) 9(0,0 (null)) 10(0,0 (null)) 11(0,0 (null)) 12(0,0 (null)) 13(0,0 (null)) 14(0,0 (null)) 15(0,0 (null)) 16(0,0 (null)) 17(0,0 (null)) 18(0,0 (null)) 19(0,0 (null)) 20(0,0 (null)) 21(0,0 (null)) 22(0,0 (null)) 23(0,0 (null)) 24(0,0 (null)) 25(0,0 (null)) 26(0,0 (null)) 27(0,0 (null)) 28(0,0 (null)) 29(0,0 (null)) 30(0,0 (null)) 31(0,0 (null)) 32(0,0 (null)) 33(0,0 (null)) 34(0,0 (null)) 35(0,0 (null)) 36(0,0 (null)) 37(0,0 (null)) 38(0,0 (null)) 39(0,0 (null)) 40(0,0 (null)) 41(0,0 (null)) 42(0,0 (null)) 43(0,0 (null)) 44(0,0 (null)) 45(0,0 (null)) 46(0,0 (null)) 47(0,0 (null)) 48(0,0 (null)) 49(0,0 (null))
[ 7251.953118] busted_srcud-torture:--- End of test: FAILURE: nreaders=44 nfakewriters=4 stat_interval=60 verbose=1 test_no_idle_hz=1 shuffle_interval=3 stutter=5 irqreader=1 fqs_duration=0 fqs_holdoff=0 fqs_stutter=3 test_boost=1/0 test_boost_interval=7 test_boost_duration=4 shutdown_secs=0 stall_cpu=0 stall_cpu_holdoff=10 stall_cpu_irqsoff=0 n_barrier_cbs=0 onoff_interval=3 onoff_holdoff=30

[重现步骤]：
1）安装kernel-debug等内核：
http://ci-storage.alibaba.net/kernel/anolis8/ck-4.19/x86_64/20220921_467/kernel-debug-core-4.19.91-467.git.d4d03de3a.an8.x86_64.rpm
http://ci-storage.alibaba.net/kernel/anolis8/ck-4.19/x86_64/20220921_467/kernel-debug-4.19.91-467.git.d4d03de3a.an8.x86_64.rpm
http://ci-storage.alibaba.net/kernel/anolis8/ck-4.19/x86_64/20220921_467/kernel-debug-modules-4.19.91-467.git.d4d03de3a.an8.x86_64.rpm
http://ci-storage.alibaba.net/kernel/anolis8/ck-4.19/x86_64/20220921_467/kernel-debug-devel-4.19.91-467.git.d4d03de3a.an8.x86_64.rpm
http://ci-storage.alibaba.net/kernel/anolis8/ck-4.19/x86_64/20220921_467/kernel-headers-4.19.91-467.git.d4d03de3a.an8.x86_64.rpm
2）reboot环境使得新内核生效
3）modprobe rcutorture onoff_interval=3 onoff_holdoff=30 torture_type=busted
4) sleep 450
5）rmmod rcutorture
6）modprobe rcutorture onoff_interval=3 onoff_holdoff=30 torture_type=busted_srcud
7)sleep 450
8）rmmod rcutorture

[期望结果]：
rcutorture测试pass
[实际结果]：
cpuhotplug测试集下cpuhotplug-busted和cpuhotplug-busted_srcud用例执行失败

[测试环境]： 
# uname -r
4.19.91-467.git.d4d03de3a.an8.x86_64+debug

# cat /etc/os-release
NAME="Anolis OS"
VERSION="8.6"
ID="anolis"
ID_LIKE="rhel fedora centos"
VERSION_ID="8.6"
PLATFORM_ID="platform:an8"
PRETTY_NAME="Anolis OS 8.6"
ANSI_COLOR="0;31"
HOME_URL="https://openanolis.cn/"

# lscpu
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              96
On-line CPU(s) list: 0-95
Thread(s) per core:  2
Core(s) per socket:  24
Socket(s):           2
NUMA node(s):        1
Vendor ID:           GenuineIntel
BIOS Vendor ID:      Intel(R) Corporation
CPU family:          6
Model:               85
Model name:          Intel(R) Xeon(R) Platinum 8163 CPU @ 2.50GHz
BIOS Model name:     Intel(R) Xeon(R) Platinum 8163 CPU @ 2.50GHz
Stepping:            4
CPU MHz:             999.954
CPU max MHz:         3100.0000
CPU min MHz:         1000.0000
BogoMIPS:            5000.00
Virtualization:      VT-x
L1d cache:           32K
L1i cache:           32K
L2 cache:            1024K
L3 cache:            33792K
NUMA node0 CPU(s):   0-95
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku ospke md_clear flush_l1d arch_capabilities

# cat /proc/cmdline
BOOT_IMAGE=(hd0,gpt2)/vmlinuz-4.19.91-467.git.d4d03de3a.an8.x86_64+debug root=UUID=fbaa442c-62df-4234-aa61-574c3d45b5d2 ro console=tty0 console=ttyS0,115200 rd.driver.pre=ahci crashkernel=0M-2G:0M,2G-8G:192M,8G-:256M

# dmidecode -t 0
# dmidecode 3.3
Getting SMBIOS data from sysfs.
SMBIOS 3.0.0 present.

Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
        Vendor: Inspur
        Version: 3.20.34
        Release Date: 03/09/2019
        Address: 0xF0000
        Runtime Size: 64 kB
        ROM Size: 16 MB
        Characteristics:
                PCI is supported
                BIOS is upgradeable
                BIOS shadowing is allowed
                Boot from CD is supported
                Selectable boot is supported
                BIOS ROM is socketed
                EDD is supported
                5.25"/1.2 MB floppy services are supported (int 13h)
                3.5"/720 kB floppy services are supported (int 13h)
                3.5"/2.88 MB floppy services are supported (int 13h)
                Print screen service is supported (int 5h)
                Serial services are supported (int 14h)
                Printer services are supported (int 17h)
                ACPI is supported
                USB legacy is supported
                BIOS boot specification is supported
                Targeted content distribution is supported
                UEFI is supported

[出现频率]
必现
Comment 1 shanxifanshi alibaba_cloud_group 2022-09-23 15:29:18 UTC 
非cpuhotplug的场景，这2个type不会fail吗
Comment 2 yunhe123 alibaba_cloud_group 2022-10-10 11:12:05 UTC 
anolis8-5.10-debug-aarch nightly 物理机存在同样的问题，日志如下：
cpuhotplug-busted用例部分日志：
[20304.810213] busted-torture: Stopping rcu_torture_reader task
[20304.816185] busted-torture: Stopping rcu_torture_reader task
[20304.822166] busted-torture: Stopping rcu_torture_fakewriter task
[20304.828491] busted-torture: Stopping rcu_torture_fakewriter task
[20304.834808] busted-torture: Stopping rcu_torture_fakewriter task
[20304.841137] busted-torture: Stopping rcu_torture_fakewriter task
[20304.854211] busted-torture: Stopping rcu_torture_stats task
[20304.860167] busted-torture: rtc: 0000000000000000 VER: 21070 tfle: 0 rta: 21070 rtaf: 0 rtf: 21061 rtmbe: 137 rtbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 953008 onoff: 1085/1085:1089/1089 10,258:10,604 78402:96787 (HZ=250) barrier: 0/0:0 read-exits: 444
[20304.884049] busted-torture: !!! Reader Pipe:  8559351805 76 450 5 10 3 20 2 11 18 170
[20304.892619] busted-torture: Reader Batch:  8559353306 0 0 0 0 0 0 0 0 0 0
[20304.900076] busted-torture: Free-Block Circulation:  21069 21069 21068 21067 21066 21065 21064 21063 21062 21061 0
[20304.911159] busted-torture: Stopping rcu_torture_stats
[20304.917117] busted-torture: rtc: 0000000000000000 VER: 21070 tfle: 0 rta: 21070 rtaf: 0 rtf: 21061 rtmbe: 137 rtbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 953008 onoff: 1085/1085:1089/1089 10,258:10,604 78402:96787 (HZ=250) barrier: 0/0:0 read-exits: 444
[20304.942011] busted-torture: !!! Reader Pipe:  8559351805 76 450 5 10 3 20 2 11 18 170
[20304.950888] busted-torture: Reader Batch:  8559353306 0 0 0 0 0 0 0 0 0 0
[20304.958700] busted-torture: Free-Block Circulation:  21069 21069 21068 21067 21066 21065 21064 21063 21062 21061 0
[20304.979993] busted-torture:--- End of test: FAILURE: nreaders=51 nfakewriters=4 stat_interval=60 verbose=1 test_no_idle_hz=1 shuffle_interval=3 stutter=5 irqreader=1 fqs_duration=0 fqs_holdoff=0 fqs_stutter=3 test_boost=1/0 test_boost_interval=7 test_boost_duration=4 shutdown_secs=0 stall_cpu=0 stall_cpu_holdoff=10 stall_cpu_irqsoff=0 stall_cpu_block=0 n_barrier_cbs=0 onoff_interval=3 onoff_holdoff=30 read_exit_delay=13 read_exit_burst=16
[2022-10-10 02:08:46] modprobe rcutorture onoff_interval=3 onoff_holdoff=30 torture_type=srcu
[2022-10-10 02:08:47] sleep 400
[2022-10-10 02:15:27] rmmod rcutorture


cpuhotplug-busted_srcud用例日志：
[21517.165835] busted_srcud-torture: Stopping rcu_torture_reader task
[21517.172460] busted_srcud-torture: Stopping rcu_torture_reader task
[21517.179108] busted_srcud-torture: Stopping rcu_torture_reader task
[21517.185653] busted_srcud-torture: Stopping rcu_torture_reader task
[21517.192182] busted_srcud-torture: Stopping rcu_torture_fakewriter task
[21517.199083] busted_srcud-torture: Stopping rcu_torture_fakewriter task
[21517.205946] busted_srcud-torture: Stopping rcu_torture_fakewriter task
[21517.212811] busted_srcud-torture: Stopping rcu_torture_fakewriter task
[21517.227879] busted_srcud-torture: Stopping rcu_torture_stats task
[21517.234390] busted_srcud-torture: rtc: 0000000000000000 VER: 10245 tfle: 0 rta: 10245 rtaf: 0 rtf: 10236 rtmbe: 0 rtbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 837241 onoff: 3377/3377:3393/3393 10,258:9,604 196084:235876 (HZ=250) barrier: 0/0:0 read-exits: 458
[21517.258992] busted_srcud-torture: !!! Reader Pipe:  1475450282 259290 111 5 1 3 4 2 2 8 72
[21517.267971] busted_srcud-torture: Reader Batch:  1474939259 768176 2264 49 7 0 1 0 0 1 97
[21517.276839] busted_srcud-torture: Free-Block Circulation:  10244 10244 10243 10242 10241 10240 10239 10238 10237 10236 0
[21517.288456] rcu: busted_srcud-torture: Tree SRCU g118988 per-CPU(idx=1): 0(0,0 .) 1(0,0 .) 2(0,0 .) 3(0,0 .) 4(0,0 .) 5(0,0 .) 6(0,0 .) 7(0,0 .) 8(0,0 .) 9(0,0 .) 10(0,0 .) 11(0,0 .) 12(0,0 .) 13(0,0 .) 14(0,0 .) 15(0,0 .) 16(0,0 .) 17(0,0 .) 18(0,0 .) 19(0,0 .) 20(0,0 .) 21(0,0 .) 22(0,0 .) 23(0,0 .) 24(0,0 .) 25(0,0 .) 26(0,0 .) 27(0,0 .) 28(0,0 .) 29(0,0 .) 30(0,0 .) 31(0,0 .) 32(0,0 .) 33(0,0 .) 34(0,0 .) 35(0,0 .) 36(0,0 .) 37(0,0 .) 38(0,0 .) 39(0,0 .) 40(0,0 .) 41(0,0 .) 42(0,0 .) 43(0,0 .) 44(0,0 .) 45(0,0 .) 46(0,0 .) 47(0,0 .) 48(0,0 .) 49(0,0 .) 50(0,0 .) 51(0,0 .) 52(0,0 .) 53(0,0 .) 54(0,0 .) 55(0,0 .) 56(0,0 .) 57(0,0 .) 58(0,0 .) 59(0,0 .) 60(0,0 .) 61(0,0 .) 62(0,0 .) 63(0,0 .) 64(0,0 .) 65(0,0 .) 66(0,0 .) 67(0,0 .) 68(0,0 .) 69(0,0 .) 70(0,0 .) 71(0,0 .) 72(0,0 .) 73(0,0 .) 74(0,0 .) 75(0,0 .) 76(0,0 .) 77(0,0 .) 78(0,0 .) 79(0,0 .) 80(0,0 .) 81(0,0 .) 82(0,0 .) 83(0,0 .) 84(0,0 .) 85(0,0 .) 86(0,0 .) 87(0,0 .) 88(0,0 .) 89(0,0 .) 90(0,0 .) 91(0,0 .) 92(0,0 .) 93(0,0 .)
[21517.389007] busted_srcud-torture: Stopping rcu_torture_stats
[21517.398732] busted_srcud-torture: rtc: 0000000000000000 VER: 10245 tfle: 0 rta: 10245 rtaf: 0 rtf: 10236 rtmbe: 0 rtbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 837241 onoff: 3377/3377:3393/3393 10,258:9,604 196084:235876 (HZ=250) barrier: 0/0:0 read-exits: 458
[21517.425949] busted_srcud-torture: !!! Reader Pipe:  1475450282 259290 111 5 1 3 4 2 2 8 72
[21517.435835] busted_srcud-torture: Reader Batch:  1474939259 768176 2264 49 7 0 1 0 0 1 97
[21517.445616] busted_srcud-torture: Free-Block Circulation:  10244 10244 10243 10242 10241 10240 10239 10238 10237 10236 0
[21517.458138] rcu: busted_srcud-torture: Tree SRCU g0 per-CPU(idx=0): 0(0,0 .) 1(0,0 .) 2(0,0 .) 3(0,0 .) 4(0,0 .) 5(0,0 .) 6(0,0 .) 7(0,0 .) 8(0,0 .) 9(0,0 .) 10(0,0 .) 11(0,0 .) 12(0,0 .) 13(0,0 .) 14(0,0 .) 15(0,0 .) 16(0,0 .) 17(0,0 .) 18(0,0 .) 19(0,0 .) 20(0,0 .) 21(0,0 .) 22(0,0 .) 23(0,0 .) 24(0,0 .) 25(0,0 .) 26(0,0 .) 27(0,0 .) 28(0,0 .) 29(0,0 .) 30(0,0 .) 31(0,0 .) 32(0,0 .) 33(0,0 .) 34(0,0 .) 35(0,0 .) 36(0,0 .) 37(0,0 .) 38(0,0 .) 39(0,0 .) 40(0,0 .) 41(0,0 .) 42(0,0 .) 43(0,0 .) 44(0,0 .) 45(0,0 .) 46(0,0 .) 47(0,0 .) 48(0,0 .) 49(0,0 .) 50(0,0 .) 51(0,0 .) 52(0,0 .) 53(0,0 .) 54(0,0 .) 55(0,0 .) 56(0,0 .) 57(0,0 .) 58(0,0 .) 59(0,0 .) 60(0,0 .) 61(0,0 .) 62(0,0 .) 63(0,0 .) 64(0,0 .) 65(0,0 .) 66(0,0 .) 67(0,0 .) 68(0,0 .) 69(0,0 .) 70(0,0 .) 71(0,0 .) 72(0,0 .) 73(0,0 .) 74(0,0 .) 75(0,0 .) 76(0,0 .) 77(0,0 .) 78(0,0 .) 79(0,0 .) 80(0,0 .) 81(0,0 .) 82(0,0 .) 83(0,0 .) 84(0,0 .) 85(0,0 .) 86(0,0 .) 87(0,0 .) 88(0,0 .) 89(0,0 .) 90(0,0 .) 91(0,0 .) 92(0,0 .) 93(0,0 .)
[21517.613998] busted_srcud-torture:--- End of test: FAILURE: nreaders=46 nfakewriters=4 stat_interval=60 verbose=1 test_no_idle_hz=1 shuffle_interval=3 stutter=5 irqreader=1 fqs_duration=0 fqs_holdoff=0 fqs_stutter=3 test_boost=1/0 test_boost_interval=7 test_boost_duration=4 shutdown_secs=0 stall_cpu=0 stall_cpu_holdoff=10 stall_cpu_irqsoff=0 stall_cpu_block=0 n_barrier_cbs=0 onoff_interval=3 onoff_holdoff=30 read_exit_delay=13 read_exit_burst=16
[2022-10-10 02:28:59] modprobe rcutorture onoff_interval=3 onoff_holdoff=30 torture_type=tasks
[2022-10-10 02:28:59] sleep 400
[2022-10-10 02:35:39] rmmod rcutorture
Comment 3 zhixin01 alibaba_cloud_group 2022-10-17 13:39:29 UTC 
(In reply to shanxifanshi from comment #1)
> 非cpuhotplug的场景，这2个type不会fail吗
非cpuhotplug场景同样会FAIL，之前已提单：
https://bugzilla.openanolis.cn/show_bug.cgi?id=2228
Comment 4 anolislw alibaba_cloud_group 2022-11-24 16:07:45 UTC 
anolis8.6 ck-4.19 aarch64中 kernel-debug nightly测试，rcutorture测试套下cpuhotplug-busted与cpuhotplug-busted_srcud也存在同样的问题