Bug 4885 - [ANCK-5.10-15][Anolis8][x86_64]使用bpftrace src包做bpftrace-test测试，编译失败
Summary: [ANCK-5.10-15][Anolis8][x86_64]使用bpftrace src包做bpftrace-test测试，编译失败
Status: CLOSED FIXED
Alias: None
Product: Anolis OS 8
Classification: Anolis OS
Component: BaseOS Packages (show other bugs) BaseOS Packages
Version: 8.8
Hardware: x86_64 Linux
: P2-High S2-major
Target Milestone: rc1
Assignee: gaochang
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2023-05-08 17:30 UTC by shanxifanshi
Modified: 2023-08-04 14:44 UTC (History)
14 users (show)

See Also:

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description shanxifanshi alibaba_cloud_group 2023-05-08 17:30:39 UTC 
[缺陷描述]：
Anolis8 5.10 x86_64 使用bpftrace src包做bpftrace-test测试，编译失败

# rpm -q llvm
llvm-13.0.1-1.0.3.module+an8.7.0+10996+1588f068.x86_64

部分编译报错日志如下：
[ 30%] Linking CXX executable bpftrace
   Value *seq = CreateLoad(getInt64Ty(), CreateGEP(meta, getInt64(0)), "seq");
                                                                    ^
In file included from /usr/include/llvm/Support/ErrorHandling.h:17,
                 from /usr/include/llvm/ADT/Hashing.h:48,
                 from /usr/include/llvm/ADT/Optional.h:18,
                 from /usr/include/llvm/ADT/STLExtras.h:19,
                 from /usr/include/llvm/ADT/StringRef.h:12,
                 from /root/rpmbuild/BUILD/bpftrace-0.12.1/src/bpforc.h:3,
                 from /root/rpmbuild/BUILD/bpftrace-0.12.1/src/bpftrace.h:15,
                 from /root/rpmbuild/BUILD/bpftrace-0.12.1/src/ast/irbuilderbpf.h:4,
                 from /root/rpmbuild/BUILD/bpftrace-0.12.1/src/ast/async_event_types.h:3,
                 from /root/rpmbuild/BUILD/bpftrace-0.12.1/src/ast/irbuilderbpf.cpp:5:
/usr/include/llvm/IR/IRBuilder.h:1787:3: note: declared here
   LLVM_ATTRIBUTE_DEPRECATED(
   ^~~~~~~~~~~~~~~~~~~~~~~~~
/usr/bin/ld: libbpftrace.a(bpftrace.cpp.o): undefined reference to symbol 'bpf_iter_create@@LIBBPF_0.0.9'
//usr/lib64/libbpf.so.0: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
make[2]: *** [src/CMakeFiles/bpftrace.dir/build.make:108: src/bpftrace] Error 1
make[1]: *** [CMakeFiles/Makefile2:1235: src/CMakeFiles/bpftrace.dir/all] Error 2
make: *** [Makefile:146: all] Error 2

[复现概率]：
必现

[复现环境]：
内核：
# uname -r
5.10.134-656.git.b68340408.an8.x86_64

# cat /etc/os-release
NAME="Anolis OS"
VERSION="8.8"
ID="anolis"
ID_LIKE="rhel fedora centos"
VERSION_ID="8.8"
PLATFORM_ID="platform:an8"
PRETTY_NAME="Anolis OS 8.8"
ANSI_COLOR="0;31"
HOME_URL="https://openanolis.cn/"

CPU信息：
# lscpu
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              24
On-line CPU(s) list: 0-23
Thread(s) per core:  2
Core(s) per socket:  12
Socket(s):           1
NUMA node(s):        1
Vendor ID:           GenuineIntel
BIOS Vendor ID:      Intel
CPU family:          6
Model:               63
Model name:          Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
BIOS Model name:     Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
Stepping:            2
CPU MHz:             2494.991
CPU max MHz:         2500.0000
CPU min MHz:         1200.0000
BogoMIPS:            4988.45
Virtualization:      VT-x
L1d cache:           32K
L1i cache:           32K
L2 cache:            256K
L3 cache:            30720K
NUMA node0 CPU(s):   0-23
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear flush_l1d

内存信息：
# free -h
              total        used        free      shared  buff/cache   available
Mem:           62Gi       1.8Gi        60Gi       118Mi       963Mi        60Gi
Swap:         2.0Gi       121Mi       1.9Gi

[复现步骤]：
1. 下载跟bpftrace版本匹配的源码包
yum install bpftrace
mkdir -p /root/bpftrace
cd /root/bpftrace
wget https://mirrors.openanolis.cn/anolis/8/AppStream/source/Packages/bpftrace-0.12.1-4.0.1.an8.src.rpm

2.解压源码并打上必要的补丁
rpm2cpio bpftrace-0.12.1-4.0.1.an8.src.rpm |cpio -div
yum -y install rpm-build yum-utils cmake llvm patch
mkdir -p  /root/rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS}
echo "%_topdir  /root/rpmbuild" >~/.rpmmacros
cp -r /root/bpftrace/* /root/rpmbuild/SOURCES
cp -r /root/bpftrace/*.spec /root/rpmbuild/SPECS
yum-builddep -y /root/rpmbuild/SPECS/bpftrace.spec
rpmbuild -bp /root/rpmbuild/SPECS/bpftrace.spec

3.编译
cd /root/rpmbuild/BUILD/
wget -q http://242510.oss-cn-hangzhou-zmf.aliyuncs.com/googletest.tar.gz
bpftrace_src=`find . -name bpftrace-\[0-9\].* -type d`
tar xf googletest.tar.gz -C $bpftrace_src
cd $bpftrace_src
cmake . -DCMAKE_BUILD_TYPE=RelWithDebInfo  -DBUILD_TESTING:BOOL=ON  -DBUILD_SHARED_LIBS:BOOL=OFF -DOFFLINE_BUILDS:BOOL=ON
sed -i "s|https://github.com/google|$(readlink -f .)|" tests/gtest-git-prefix/tmp/gtest-git-gitclone.cmake
make -j8


[预期结果]：
bpftrace源码编译成功

[实际结果]：
bpftrace源码编译失败
Comment 1 wangpingping alibaba_cloud_group 2023-05-19 17:58:17 UTC 
5.10.134-14.1.al8.x86_64内核依旧有此问题；
Comment 2 yunmeng365524 2023-06-19 21:31:54 UTC 
编译失败了，请帮忙确认一下。
Comment 3 dtcccc alibaba_cloud_group 2023-06-20 16:13:27 UTC 
看起来静态库去找动态库的符号出问题了

-DBUILD_SHARED_LIBS:BOOL=ON
设成这样就编译ok了

为什么当时这里要设置OFF呢？
Comment 4 dtcccc alibaba_cloud_group 2023-06-20 16:21:26 UTC 
另，编译用户态工具包相关问题属于baseos领域，非内核领域
Comment 5 shanxifanshi alibaba_cloud_group 2023-06-20 16:29:52 UTC 
(In reply to dtcccc from comment #3)
> 看起来静态库去找动态库的符号出问题了
> 
> -DBUILD_SHARED_LIBS:BOOL=ON
> 设成这样就编译ok了
> 
> 为什么当时这里要设置OFF呢？

这个是按照当时xuchunmei给的bpftrace测试用例编译步骤进行测试的，步骤里是设置成OFF的，具体原因不太清楚，不过刚开始时候是能够编译成功的，后来就会失败。
Comment 6 gaochang alibaba_cloud_group 2023-06-21 18:45:06 UTC 
@shanxifans
Comment 7 gaochang alibaba_cloud_group 2023-06-21 18:46:49 UTC 
麻烦修改该 bug 到 8.8 BaseOS 中吧
请问，编译问题按照如上修改是否已解决编译问题？
@shanxifanshi
Comment 8 shanxifanshi alibaba_cloud_group 2023-06-25 09:51:16 UTC 
(In reply to gaochang from comment #7)
> 麻烦修改该 bug 到 8.8 BaseOS 中吧
> 请问，编译问题按照如上修改是否已解决编译问题？
> @shanxifanshi

将-DBUILD_SHARED_LIBS:BOOL设置成ON，是可以成功编译的
Comment 9 shanxifanshi alibaba_cloud_group 2023-07-07 11:36:54 UTC 
在rc3版本测试时，再次出现了编译失败的问题，原因是编译需要的googletest.tar.gz文件好像是服务器上不存在了

# wget http://242510.oss-cn-hangzhou-zmf.aliyuncs.com/googletest.tar.gz
--2023-07-07 11:27:09--  http://242510.oss-cn-hangzhou-zmf.aliyuncs.com/googletest.tar.gz
Connecting to 11.238.151.75:16... failed: Connection refused.

# ping 242510.oss-cn-hangzhou-zmf.aliyuncs.com
PING 242510.oss-cn-hangzhou-zmf.aliyuncs.com (100.67.214.229) 56(84) bytes of data.
64 bytes from 100.67.214.229 (100.67.214.229): icmp_seq=1 ttl=42 time=7.70 ms
64 bytes from 100.67.214.229 (100.67.214.229): icmp_seq=2 ttl=42 time=7.71 ms
^C
--- 242510.oss-cn-hangzhou-zmf.aliyuncs.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 7.703/7.707/7.712/0.087 ms


编译报错日志：
-- Found LibElf: /usr/lib64/libelf.so
tar: Error is not recoverable: exiting now
CMake Error at /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
-- Configuring incomplete, errors occurred!
  Please install the libcereal development package (missing:
  LIBCEREAL_INCLUDE_DIRS)
Call Stack (most recent call first):
  /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
  cmake/FindLibCereal.cmake:14 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
  CMakeLists.txt:109 (find_package)


sed: can't read tests/gtest-git-prefix/tmp/gtest-git-gitclone.cmake: No such file or directory
make: *** No targets specified and no makefile found.  Stop.
Comment 10 shanxifanshi alibaba_cloud_group 2023-07-12 14:53:24 UTC 
(In reply to shanxifanshi from comment #9)
> 在rc3版本测试时，再次出现了编译失败的问题，原因是编译需要的googletest.tar.gz文件好像是服务器上不存在了
> 
> # wget http://242510.oss-cn-hangzhou-zmf.aliyuncs.com/googletest.tar.gz
> --2023-07-07 11:27:09-- 
> http://242510.oss-cn-hangzhou-zmf.aliyuncs.com/googletest.tar.gz
> Connecting to 11.238.151.75:16... failed: Connection refused.
> 
> # ping 242510.oss-cn-hangzhou-zmf.aliyuncs.com
> PING 242510.oss-cn-hangzhou-zmf.aliyuncs.com (100.67.214.229) 56(84) bytes
> of data.
> 64 bytes from 100.67.214.229 (100.67.214.229): icmp_seq=1 ttl=42 time=7.70 ms
> 64 bytes from 100.67.214.229 (100.67.214.229): icmp_seq=2 ttl=42 time=7.71 ms
> ^C
> --- 242510.oss-cn-hangzhou-zmf.aliyuncs.com ping statistics ---
> 2 packets transmitted, 2 received, 0% packet loss, time 1001ms
> rtt min/avg/max/mdev = 7.703/7.707/7.712/0.087 ms
> 
> 
> 编译报错日志：
> -- Found LibElf: /usr/lib64/libelf.so
> tar: Error is not recoverable: exiting now
> CMake Error at
> /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
> -- Configuring incomplete, errors occurred!
>   Please install the libcereal development package (missing:
>   LIBCEREAL_INCLUDE_DIRS)
> Call Stack (most recent call first):
>   /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:594
> (_FPHSA_FAILURE_MESSAGE)
>   cmake/FindLibCereal.cmake:14 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
>   CMakeLists.txt:109 (find_package)
> 
> 
> sed: can't read tests/gtest-git-prefix/tmp/gtest-git-gitclone.cmake: No such
> file or directory
> make: *** No targets specified and no makefile found.  Stop.

--上面的分析不太对，googletest.tar.gz在服务器上其实是存在的，上次是因为代理配置有误，导致没下载下来。

真正fail的原因是因为bpftrace包在7月1号发生了更新，由原来的0.12.1-4.0.1版本升级到了0.16.0-1版本，我们将bpftrace包更新到最新的0.16.0-1之后继续测试，就会发生编译失败的问题（-DBUILD_SHARED_LIBS:BOOL已经设置为ON）

对比了两个版本bpftrace src包安装之后的目录，结合报错情况
# ll
total 2828
-rw-r--r-- 1 root root 1430802 Jul 12 14:11 bpftrace-0.16.0-1.an8.src.rpm
-rw-r--r-- 1 root root   20243 Jul 12 14:11 bpftrace-0.16.0-IR-builder-get-rid-of-getPointerElementType-calls.patch
-rw-r--r-- 1 root root    1831 Jul 12 14:11 bpftrace-0.16.0-RHEL-8-aarch64-fixes-statsnoop-and-opensnoop.patch
-rw-r--r-- 1 root root    2197 Jul 12 14:11 bpftrace-0.16.0-RHEL8-remove-not-existing-attachpoints-from-tools.patch
-rw-r--r-- 1 root root 1023480 Jul 12 14:11 bpftrace-0.16.0.tar.gz
-rw-r--r-- 1 root root    5575 Jul 12 14:11 bpftrace-0.16.0-tcpdrop-Fix-ERROR-Error-attaching-probe-kprobe-tcp_d.patch
-rw-r--r-- 1 root root     718 Jul 12 14:11 bpftrace-0.16.0-tools-old-mdflush.bt-fix-BPFTRACE_HAVE_BTF-macro.patch
-rw-r--r-- 1 root root    6716 Jul 12 14:11 bpftrace.spec
-rw-r--r-- 1 root root  385903 Jul 12 14:11 cereal-1.3.2.tar.gz

# yum repoinfo
Last metadata expiration check: 3:12:40 ago on Wed 12 Jul 2023 11:25:51 AM CST.
Repo-id            : AppStream
Repo-name          : AnolisOS-8 - AppStream
Repo-revision      : 1689130519
Repo-updated       : Wed 12 Jul 2023 10:55:21 AM CST
Repo-pkgs          : 8,600
Repo-available-pkgs: 7,189
Repo-size          : 19 G
Repo-baseurl       : http://mirrors.openanolis.cn/anolis/8/AppStream/x86_64/os
Repo-expire        : 172,800 second(s) (last: Wed 12 Jul 2023 11:25:37 AM CST)
Repo-filename      : /etc/yum.repos.d/AnolisOS-AppStream.repo

Repo-id            : BaseOS
Repo-name          : AnolisOS-8 - BaseOS
Repo-revision      : 1689130387
Repo-updated       : Wed 12 Jul 2023 10:53:07 AM CST
Repo-pkgs          : 2,196
Repo-available-pkgs: 2,194
Repo-size          : 3.7 G
Repo-baseurl       : http://mirrors.openanolis.cn/anolis/8/BaseOS/x86_64/os
Repo-expire        : 172,800 second(s) (last: Wed 12 Jul 2023 11:25:40 AM CST)
Repo-filename      : /etc/yum.repos.d/AnolisOS-BaseOS.repo

Repo-id            : Extras
Repo-name          : AnolisOS-8 - Extras
Repo-revision      : 1676272204
Repo-updated       : Mon 13 Feb 2023 03:10:04 PM CST
Repo-pkgs          : 7
Repo-available-pkgs: 7
Repo-size          : 81 k
Repo-baseurl       : http://mirrors.openanolis.cn/anolis/8/Extras/x86_64/os
Repo-expire        : 172,800 second(s) (last: Wed 12 Jul 2023 11:25:42 AM CST)
Repo-filename      : /etc/yum.repos.d/AnolisOS-Extras.repo

Repo-id            : Plus
Repo-name          : AnolisOS-8 - Plus
Repo-revision      : 1685324178
Repo-updated       : Mon 29 May 2023 09:36:18 AM CST
Repo-pkgs          : 222
Repo-available-pkgs: 222
Repo-size          : 3.7 G
Repo-baseurl       : http://mirrors.openanolis.cn/anolis/8/Plus/x86_64/os
Repo-expire        : 172,800 second(s) (last: Wed 12 Jul 2023 11:25:42 AM CST)
Repo-filename      : /etc/yum.repos.d/AnolisOS-Plus.repo

Repo-id            : PowerTools
Repo-name          : AnolisOS-8 - PowerTools
Repo-revision      : 1689056091
Repo-updated       : Tue 11 Jul 2023 02:14:52 PM CST
Repo-pkgs          : 1,697
Repo-available-pkgs: 1,142
Repo-size          : 5.5 G
Repo-baseurl       : http://mirrors.openanolis.cn/anolis/8/PowerTools/x86_64/os
Repo-expire        : 172,800 second(s) (last: Wed 12 Jul 2023 11:25:42 AM CST)
Repo-filename      : /etc/yum.repos.d/AnolisOS-PowerTools.repo

Repo-id            : epel
Repo-name          : Extra Packages for Enterprise Linux 8 - x86_64
Repo-revision      : 1689125629
Repo-updated       : Wed 12 Jul 2023 09:37:16 AM CST
Repo-pkgs          : 9,769
Repo-available-pkgs: 9,767
Repo-size          : 17 G
Repo-baseurl       : http://mirrors.aliyun.com/epel/8/Everything/x86_64
Repo-expire        : 172,800 second(s) (last: Wed 12 Jul 2023 11:25:51 AM CST)
Repo-filename      : /etc/yum.repos.d/epel.repo

应该是新的bpftrace src包安装后多了一个cereal包，而这个包编译需要libcereal-devel文件，但我们的repo源中又没有该文件，导致编译失败。

以上是测试同学的分析，麻烦开发同学看下能否提供libcereal-devel包
Comment 11 shanxifanshi alibaba_cloud_group 2023-07-12 15:13:43 UTC 
缺少libcereal-devel包问题，已经重新提单跟踪
https://bugzilla.openanolis.cn/show_bug.cgi?id=5847
Comment 12 shuming admin 2023-08-04 14:44:13 UTC 
原始问题已修复，新问题在5847中跟踪。