[缺陷描述]： bcc：test_histogram.py用例失败，报错Failed to attach BPF program b'kprobe__finish_task_switch' to kprobe b'finish_task_switch' 软件版本： # rpm -qa |grep bcc bcc-tools-0.27.0-1.an23.aarch64 bcc-0.27.0-1.an23.aarch64 python3-bcc-0.27.0-1.an23.noarch bcc-devel-0.27.0-1.an23.aarch64 失败日志如下： # python test_histogram.py cannot attach kprobe, probe entry may not exist E k_1 & k_2 = 0 0 size : count distribution 512 -> 1023 : 2 |************* | 1024 -> 2047 : 0 | | 2048 -> 4095 : 0 | | 4096 -> 8191 : 6 |****************************************| k_1 & k_2 = 16 0 size : count distribution 8 -> 15 : 1 |******************** | 16 -> 31 : 2 |****************************************| k_1 & k_2 = 32 0 size : count distribution 256 -> 511 : 1 |****************************************| k_1 & k_2 = 48 0 size : count distribution 256 -> 511 : 2 |*** | 512 -> 1023 : 0 | | 1024 -> 2047 : 0 | | 2048 -> 4095 : 0 | | 4096 -> 8191 : 2 |*** | 8192 -> 16383 : 0 | | 16384 -> 32767 : 21 |****************************************| k_1 & k_2 = 48 1 size : count distribution 128 -> 255 : 1 |****************************************| k_1 & k_2 = 64 0 size : count distribution 4096 -> 8191 : 2 |****************************************| k_1 & k_2 = 112 0 size : count distribution 512 -> 1023 : 4 |******************************** | 1024 -> 2047 : 0 | | 2048 -> 4095 : 5 |****************************************| ... ====================================================================== ERROR: test_chars (__main__.TestHistogram) ---------------------------------------------------------------------- Traceback (most recent call last): File "/root/bcc/bcc/bcc-0.27.0/tests/python/test_histogram.py", line 61, in test_chars b = BPF(text=b""" File "/usr/lib/python3.10/site-packages/bcc/__init__.py", line 487, in __init__ self._trace_autoload() File "/usr/lib/python3.10/site-packages/bcc/__init__.py", line 1456, in _trace_autoload self.attach_kprobe( File "/usr/lib/python3.10/site-packages/bcc/__init__.py", line 845, in attach_kprobe raise Exception("Failed to attach BPF program %s to kprobe %s" Exception: Failed to attach BPF program b'kprobe__finish_task_switch' to kprobe b'finish_task_switch', it's not traceable (either non-existing, inlined, or marked as "notrace") ---------------------------------------------------------------------- Ran 4 tests in 3.021s FAILED (errors=1) [复现概率]： 必现 [复现环境]： 内核： # uname -r 6.6.25-2_rc1.an23.x86_64 # cat /etc/os-release NAME="Anolis OS" VERSION="23" ID="anolis" VERSION_ID="23" PLATFORM_ID="platform:an23" PRETTY_NAME="Anolis OS 23" ANSI_COLOR="0;31" HOME_URL="https://openanolis.cn/" BUG_REPORT_URL="https://bugzilla.openanolis.cn/" CPU信息： # lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 52 bits physical, 57 bits virtual Byte Order: Little Endian CPU(s): 4 On-line CPU(s) list: 0-3 Vendor ID: GenuineIntel BIOS Vendor ID: Alibaba Cloud Model name: Intel(R) Xeon(R) Platinum 8475B BIOS Model name: pc-q35-df-2.1 CPU @ 0.0GHz BIOS CPU family: 1 CPU family: 6 Model: 143 Thread(s) per core: 2 Core(s) per socket: 2 Socket(s): 1 Stepping: 8 CPU(s) scaling MHz: 84% CPU max MHz: 3800.0000 CPU min MHz: 800.0000 BogoMIPS: 5400.00 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtsc p lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault ibrs_enhanced fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512c d sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx_vnni avx512_bf16 wbnoinvd ida arat hwp hwp_notify hwp_act_window h wp_epp hwp_pkg_req avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntd q rdpid bus_lock_detect cldemote movdiri movdir64b enqcmd fsrm md_clear serialize tsxldtrk amx_bf16 avx512_fp16 amx_tile amx_int 8 arch_capabilities Virtualization features: Hypervisor vendor: KVM Virtualization type: full Caches (sum of all): L1d: 96 KiB (2 instances) L1i: 64 KiB (2 instances) L2: 4 MiB (2 instances) L3: 97.5 MiB (1 instance) NUMA: NUMA node(s): 1 NUMA node0 CPU(s): 0-3 Vulnerabilities: Gather data sampling: Not affected Itlb multihit: Not affected L1tf: Not affected Mds: Not affected Meltdown: Not affected Mmio stale data: Unknown: No mitigations Reg file data sampling: Not affected Retbleed: Not affected Spec rstack overflow: Not affected Spec store bypass: Vulnerable Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Spectre v2: Mitigation; Enhanced / Automatic IBRS, RSB filling, PBRSB-eIBRS SW sequence Srbds: Not affected Tsx async abort: Not affected 内存信息： # free -h total used free shared buff/cache available Mem: 15Gi 585Mi 6.9Gi 364Mi 7.6Gi 13Gi Swap: 0B 0B 0B [复现步骤]： yum install bcc-tools yum-utils rpm-build python3-pyroute2.noarch git clone https://gitee.com/src-anolis-os/bcc.git --branch a23 cd bcc yum-builddep -y bcc.spec rpmbuild -D "_topdir $(pwd)" \ -D "_sourcedir $(pwd)" \ -D "_builddir $(pwd)" \ -bp bcc.spec cd bcc-0.27.0/tests/python python test_clang.py [预期结果]： 用例执行成功 [实际结果]： 用例执行失败
cat /sys/kernel/debug/tracing/available_filter_functions | grep finish_task_switch finish_task_switch.isra.0 可用trace的函数做了修改，请确认该修改是否为版本规划要求，如果函数修改对应的kernel selftest的用例需要适配
finish_task_switch 会被GCC10以上版本优化为 finish_task_switch.isra.0，其中 isra 是 inter-procedural scalar replacement of aggregates 的缩写。 如果用户想trace finish_task_switch 函数，有多种方法： 1. 让 finish_task_switch 不被 GCC 优化，方法有删掉 finish_task_switch 前面的 static 或者修改为 asmlinkage __visible 2. 直接trace finish_task_switch.isra.0，这个在bcc另外的example里有实现，比如./bcc/examples/tracing/task_switch.py 里 b = BPF(src_file="task_switch.c") b.attach_kprobe(event_re="^finish_task_switch$|^finish_task_switch\.isra\.\d$", fn_name="count_sched") 实测可以成功 trace 本例 test_histogram.py 里的实现: int kprobe__finish_task_switch 无法修改为 finish_task_switch.isra.0。 本bugzilla所示只是个例子，用户可以自行通过 trace finish_task_switch.isra.0 的方法实现功能，非内核问题，上游bcc本例无修改，置为 wont fix
如上述开发定位所说，非内核问题，本问题单关闭。
在6.6.71-3_rc1.al8.aarch64内核上也有同样的问题，在此做个纪录
【6.6.102-5.100_rc2.al8.aarch64+64k】【alinux3】【倚天710】python test_histogram.py 失败。报错Failed to attach BPF program b'kprobe__finish_task_switch' to kprobe b'finish_task_switch' 环境信息： #python --version Python 3.6.8 #ll /var/tmp/tone/build/bcc/bcc-0.25.0.tar.gz -rw-r--r-- 1 root root 4984989 Jul 5 2024 /var/tmp/tone/build/bcc/bcc-0.25.0.tar.gz #rpm -qa |grep bcc python3-bcc-0.30.0-3.al8.noarch bcc-0.30.0-3.al8.aarch64 bcc-tools-0.30.0-3.al8.aarch64 bcc-devel-0.30.0-3.al8.aarch64 日志如下 #python ./tests/python/test_histogram.py /usr/lib/python3.6/site-packages/bcc/__init__.py:428: DeprecationWarning: not a bytes object: '\n#include <uapi/linux/ptrace.h>\n#include <linux/sched.h>\n#include <linux/version.h>\ntypedef struct { char name[TASK_COMM_LEN]; u64 slot; } Key;\nBPF_HISTOGRAM(hist1, Key, 1024);\nint kprobe__finish_task_switch(struct pt_regs *ctx, struct task_struct *prev) {\n#if LINUX_VERSION_CODE < KERNEL_VERSION(5,5,0)\n Key k = {.slot = bpf_log2l(prev->real_start_time)};\n#else\n Key k = {.slot = bpf_log2l(prev->start_boottime)};\n#endif\n if (!bpf_get_current_comm(&k.name, sizeof(k.name))) {\n hist1.increment(k);\n hist1.atomic_increment(k);\n }\n return 0;\n}\n' text = _assert_is_bytes(text) cannot attach kprobe, probe entry may not exist E/usr/lib/python3.6/site-packages/bcc/__init__.py:428: DeprecationWarning: not a bytes object: '\n#include <uapi/linux/ptrace.h>\n#include <uapi/linux/fs.h>\nstruct hist_s_key {\n u64 key_1;\n u64 key_2;\n};\nstruct hist_key {\n struct hist_s_key s_key;\n u64 slot;\n};\nBPF_HISTOGRAM(mk_hist, struct hist_key, 1024);\nint kprobe__vfs_read(struct pt_regs *ctx, struct file *file,\n char __user *buf, size_t count) {\n struct hist_key key = {.slot = bpf_log2l(count)};\n key.s_key.key_1 = (unsigned long)buf & 0x70;\n key.s_key.key_2 = (unsigned long)buf & 0x7;\n mk_hist.increment(key);\n return 0;\n}\n' text = _assert_is_bytes(text) /usr/lib/python3.6/site-packages/bcc/__init__.py:619: DeprecationWarning: not a bytes object: 'mk_hist' name = _assert_is_bytes(name) k_1 & k_2 = 0 0 size : count distribution 512 -> 1023 : 21 |************************** | 1024 -> 2047 : 2 |** | 2048 -> 4095 : 0 | | 4096 -> 8191 : 32 |****************************************| 8192 -> 16383 : 0 | | 16384 -> 32767 : 0 | | 32768 -> 65535 : 0 | | 65536 -> 131071 : 0 | | 131072 -> 262143 : 2 |** | k_1 & k_2 = 0 4 size : count distribution 1024 -> 2047 : 4 |****************************************| k_1 & k_2 = 16 0 size : count distribution 16 -> 31 : 1 |***** | 32 -> 63 : 0 | | 64 -> 127 : 0 | | 128 -> 255 : 0 | | 256 -> 511 : 0 | | 512 -> 1023 : 7 |****************************************| 1024 -> 2047 : 0 | | 2048 -> 4095 : 0 | | 4096 -> 8191 : 2 |*********** | k_1 & k_2 = 32 0 size : count distribution 16 -> 31 : 1 |*** | 32 -> 63 : 0 | | 64 -> 127 : 0 | | 128 -> 255 : 0 | | 256 -> 511 : 0 | | 512 -> 1023 : 11 |****************************************| 1024 -> 2047 : 0 | | 2048 -> 4095 : 0 | | 4096 -> 8191 : 3 |********** | 8192 -> 16383 : 2 |******* | 16384 -> 32767 : 1 |*** | k_1 & k_2 = 32 1 size : count distribution 128 -> 255 : 10 |****************************************| k_1 & k_2 = 48 0 size : count distribution 512 -> 1023 : 9 |****************************************| 1024 -> 2047 : 0 | | 2048 -> 4095 : 0 | | 4096 -> 8191 : 2 |******** | 8192 -> 16383 : 2 |******** | k_1 & k_2 = 64 0 size : count distribution 16 -> 31 : 1 |*** | 32 -> 63 : 0 | | 64 -> 127 : 0 | | 128 -> 255 : 0 | | 256 -> 511 : 0 | | 512 -> 1023 : 11 |****************************************| 1024 -> 2047 : 0 | | 2048 -> 4095 : 0 | | 4096 -> 8191 : 11 |****************************************| 8192 -> 16383 : 2 |******* | 16384 -> 32767 : 0 | | 32768 -> 65535 : 0 | | 65536 -> 131071 : 2 |******* | k_1 & k_2 = 80 0 size : count distribution 512 -> 1023 : 8 |****************************************| 1024 -> 2047 : 0 | | 2048 -> 4095 : 0 | | 4096 -> 8191 : 6 |****************************** | 8192 -> 16383 : 1 |***** | k_1 & k_2 = 96 0 size : count distribution 512 -> 1023 : 10 |****************************************| 1024 -> 2047 : 0 | | 2048 -> 4095 : 0 | | 4096 -> 8191 : 4 |**************** | 8192 -> 16383 : 0 | | 16384 -> 32767 : 1 |**** | k_1 & k_2 = 112 0 size : count distribution 2 -> 3 : 1 |***** | 4 -> 7 : 0 | | 8 -> 15 : 0 | | 16 -> 31 : 0 | | 32 -> 63 : 0 | | 64 -> 127 : 0 | | 128 -> 255 : 0 | | 256 -> 511 : 0 | | 512 -> 1023 : 7 |****************************************| 1024 -> 2047 : 0 | | 2048 -> 4095 : 0 | | 4096 -> 8191 : 4 |********************** | ./usr/lib/python3.6/site-packages/bcc/__init__.py:428: DeprecationWarning: not a bytes object: '\n#include <uapi/linux/ptrace.h>\nstruct bpf_map;\nBPF_HISTOGRAM(hist1);\nBPF_HASH(stub);\nint kprobe__htab_map_delete_elem(struct pt_regs *ctx, struct bpf_map *map, u64 *k) {\n hist1.increment(bpf_log2l(*k));\n hist1.atomic_increment(bpf_log2l(*k));\n return 0;\n}\n' text = _assert_is_bytes(text) /usr/lib/python3.6/site-packages/bcc/__init__.py:619: DeprecationWarning: not a bytes object: 'stub' name = _assert_is_bytes(name) /usr/lib/python3.6/site-packages/bcc/__init__.py:619: DeprecationWarning: not a bytes object: 'hist1' name = _assert_is_bytes(name) ./usr/lib/python3.6/site-packages/bcc/__init__.py:428: DeprecationWarning: not a bytes object: '\n#include <uapi/linux/ptrace.h>\nstruct bpf_map;\ntypedef struct { void *map; u64 slot; } Key;\nBPF_HISTOGRAM(hist1, Key, 1024);\nBPF_HASH(stub1);\nBPF_HASH(stub2);\nint kprobe__htab_map_delete_elem(struct pt_regs *ctx, struct bpf_map *map, u64 *k) {\n hist1.increment((Key){map, bpf_log2l(*k)});\n hist1.atomic_increment((Key){map, bpf_log2l(*k)});\n return 0;\n}\n' text = _assert_is_bytes(text) /usr/lib/python3.6/site-packages/bcc/__init__.py:619: DeprecationWarning: not a bytes object: 'stub1' name = _assert_is_bytes(name) /usr/lib/python3.6/site-packages/bcc/__init__.py:619: DeprecationWarning: not a bytes object: 'stub2' name = _assert_is_bytes(name) . ====================================================================== ERROR: test_chars (__main__.TestHistogram) ---------------------------------------------------------------------- Traceback (most recent call last): File "./tests/python/test_histogram.py", line 79, in test_chars """) File "/usr/lib/python3.6/site-packages/bcc/__init__.py", line 487, in __init__ self._trace_autoload() File "/usr/lib/python3.6/site-packages/bcc/__init__.py", line 1468, in _trace_autoload fn_name=fn.name) File "/usr/lib/python3.6/site-packages/bcc/__init__.py", line 857, in attach_kprobe (fn_name, event)) Exception: Failed to attach BPF program b'kprobe__finish_task_switch' to kprobe b'finish_task_switch', it's not traceable (either non-existing, inlined, or marked as "notrace") ---------------------------------------------------------------------- Ran 4 tests in 3.453s FAILED (errors=1)
【6.6.102-5.100_rc2.al8.aarch64+64k】【alinux3】【倚天710】bcc-0.30.0 版本 python test_histogram.py 运行pass。 日志如下； #python test_histogram.py /usr/lib/python3.6/site-packages/bcc/__init__.py:830: DeprecationWarning: not a bytes object: '^finish_task_switch$|^finish_task_switch\\.isra\\.\\d$' event_re = _assert_is_bytes(event_re) Bucket ptr = b'swapper/102' value : count distribution 0 -> 1 : 0 | | 2 -> 3 : 0 | | 4 -> 7 : 0 | | 8 -> 15 : 0 | | 16 -> 31 : 0 | | 32 -> 63 : 0 | | 64 -> 127 : 0 | | 128 -> 255 : 0 | | 256 -> 511 : 0 | | 512 -> 1023 : 0 | | 1024 -> 2047 : 0 | | 2048 -> 4095 : 0 | | 4096 -> 8191 : 0 | | 8192 -> 16383 : 0 | | 16384 -> 32767 : 0 | | 32768 -> 65535 : 0 | | 65536 -> 131071 : 0 | | 131072 -> 262143 : 0 | | 262144 -> 524287 : 0 | | 524288 -> 1048575 : 0 | | 1048576 -> 2097151 : 0 | | 2097152 -> 4194303 : 0 | | 4194304 -> 8388607 : 0 | | 8388608 -> 16777215 : 0 | | 16777216 -> 33554431 : 0 | | 33554432 -> 67108863 : 0 | | 67108864 -> 134217727 : 0 | | 134217728 -> 268435455 : 0 | | 268435456 -> 536870911 : 0 | | 536870912 -> 1073741823 : 0 | | 1073741824 -> 2147483647 : 0 | | 2147483648 -> 4294967295 : 0 | | 4294967296 -> 8589934591 : 0 | | 8589934592 -> 17179869183 : 0 | | 17179869184 -> 34359738367 : 0 | | 34359738368 -> 68719476735 : 0 | | 68719476736 -> 137438953471 : 0 | | 137438953472 -> 274877906943 : 0 | | 274877906944 -> 549755813887 : 0 | | 549755813888 -> 1099511627775 : 0 | | 1099511627776 -> 2199023255551 : 0 | | 2199023255552 -> 4398046511103 : 0 | | 4398046511104 -> 8796093022207 : 0 | | 8796093022208 -> 17592186044415 : 0 | | 17592186044416 -> 35184372088831 : 0 | | 35184372088832 -> 70368744177663 : 182 |********************| …………省略特别多…… Bucket ptr = b'swapper/9' value : count distribution 0 -> 1 : 0 | | 2 -> 3 : 0 | | 4 -> 7 : 0 | | 8 -> 15 : 0 | | 16 -> 31 : 0 | | 32 -> 63 : 0 | | 64 -> 127 : 0 | | 128 -> 255 : 0 | | 256 -> 511 : 0 | | 512 -> 1023 : 0 | | 1024 -> 2047 : 0 | | 2048 -> 4095 : 0 | | 4096 -> 8191 : 0 | | 8192 -> 16383 : 0 | | 16384 -> 32767 : 0 | | 32768 -> 65535 : 0 | | 65536 -> 131071 : 0 | | 131072 -> 262143 : 0 | | 262144 -> 524287 : 0 | | 524288 -> 1048575 : 0 | | 1048576 -> 2097151 : 0 | | 2097152 -> 4194303 : 0 | | 4194304 -> 8388607 : 0 | | 8388608 -> 16777215 : 0 | | 16777216 -> 33554431 : 0 | | 33554432 -> 67108863 : 0 | | 67108864 -> 134217727 : 0 | | 134217728 -> 268435455 : 0 | | 268435456 -> 536870911 : 0 | | 536870912 -> 1073741823 : 0 | | 1073741824 -> 2147483647 : 0 | | 2147483648 -> 4294967295 : 0 | | 4294967296 -> 8589934591 : 0 | | 8589934592 -> 17179869183 : 0 | | 17179869184 -> 34359738367 : 0 | | 34359738368 -> 68719476735 : 20 |********************| . k_1 & k_2 = 0 0 size : count distribution 512 -> 1023 : 10 | | 1024 -> 2047 : 2545 |****************************************| 2048 -> 4095 : 0 | | 4096 -> 8191 : 2 | | k_1 & k_2 = 32 1 size : count distribution 128 -> 255 : 10 |****************************************| k_1 & k_2 = 64 0 size : count distribution 16 -> 31 : 1 |******** | 32 -> 63 : 0 | | 64 -> 127 : 0 | | 128 -> 255 : 0 | | 256 -> 511 : 0 | | 512 -> 1023 : 0 | | 1024 -> 2047 : 0 | | 2048 -> 4095 : 0 | | 4096 -> 8191 : 5 |****************************************| ... ---------------------------------------------------------------------- Ran 4 tests in 5.129s OK [root@v43c07453.sqa.na131 /home/wxr02268921/bcc/bcc-0.30.0/tests/python] #echo $? 0