Bug 8121 - [Anolis23.1][RC2][6.6.7-1] vmcore解析失败
Summary: [Anolis23.1][RC2][6.6.7-1] vmcore解析失败
Status: CLOSED FIXED
Alias: None
Product: Anolis OS 23
Classification: Anolis OS
Component: ---> ToBeTriaged (show other bugs) ---> ToBeTriaged
Version: 23.1
Hardware: All Linux
: P3-Medium S3-normal
Target Milestone: ---
Assignee: gaochang
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2024-01-31 19:57 UTC by anolislw
Modified: 2024-05-22 10:38 UTC (History)
6 users (show)

See Also:

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description anolislw alibaba_cloud_group 2024-01-31 19:57:58 UTC 
Description of problem:
anolis23 内核为6.6.7-1_rc2.an23.x86_64得环境（ECS) ,VMCORE解析失败。

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. 按照kernel-debuginfo相关内核debug包
2. 收供触发vmcore: echo c > /proc/sysrq-trigger #需要确保kdump服务正常
3. vmcore解析：
   cd /var/crash/127.0.0.1-2024-02-01-03\:23\:46/
    crash /usr/lib/debug/lib/modules/6.6.7-1_rc2.an23.x86_64/vmlinux vmcore

Actual results:
[root@iZbp1f1di6pkqcght0wc4nZ 127.0.0.1-2024-02-01-03:23:46]# crash /usr/lib/debug/lib/modules/6.6.7-1_rc2.an23.x86_64/vmlinux vmcore

crash 8.0.2-2.an23
Copyright (C) 2002-2022  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2022  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
Copyright (C) 2015, 2021  VMware, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.

GNU gdb (GDB) 10.2
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...

WARNING: kernel version inconsistency between vmlinux and dumpfile

please wait... (gathering module symbol data)
crash: invalid structure member offset: module_core_size
       FILE: kernel.c  LINE: 3781  FUNCTION: module_init()

[/usr/bin/crash] error trace: 5616659fa73c => 56166567e940 => 5616657088df => 561665793ed1

[root@iZbp1f1di6pkqcght0wc4nZ 127.0.0.1-2024-02-01-03:23:46]# uname -r
6.6.7-1_rc2.an23.x86_64
[root@iZbp1f1di6pkqcght0wc4nZ 127.0.0.1-2024-02-01-03:23:46]# cat /etc/anolis-release
Anolis OS release 23
[root@iZbp1f1di6pkqcght0wc4nZ 127.0.0.1-2024-02-01-03:23:46]#


Expected results:
vmcore解析无异常

Additional info:
[root@iZbp1f1di6pkqcght0wc4nZ ~]# uname -a
Linux iZbp1f1di6pkqcght0wc4nZ 6.6.7-1_rc2.an23.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Jan 26 16:30:26 CST 2024 x86_64 GNU/Linux
[root@iZbp1f1di6pkqcght0wc4nZ ~]#
[root@iZbp1f1di6pkqcght0wc4nZ ~]# cat /etc/anolis-release
Anolis OS release 23
[root@iZbp1f1di6pkqcght0wc4nZ ~]#
[root@iZbp1f1di6pkqcght0wc4nZ ~]# df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        4.0M     0  4.0M   0% /dev
tmpfs            31G     0   31G   0% /dev/shm
tmpfs            13G  552K   13G   1% /run
/dev/vda2        40G   14G   27G  33% /
tmpfs            31G     0   31G   0% /tmp
tmpfs           6.2G  4.0K  6.2G   1% /run/user/0
[root@iZbp1f1di6pkqcght0wc4nZ ~]#
[root@iZbp1f1di6pkqcght0wc4nZ ~]# free -g
               total        used        free      shared  buff/cache   available
Mem:              61           0          60           0           0          60
Swap:              0           0           0
[root@iZbp1f1di6pkqcght0wc4nZ ~]#
[root@iZbp1f1di6pkqcght0wc4nZ ~]# cat /proc/cmdline
BOOT_IMAGE=(hd0,gpt2)/boot/vmlinuz-6.6.7-1_rc2.an23.x86_64 root=UUID=b17c104a-c396-4b6f-88a7-ad0207ec63f8 ro rhgb crashkernel=0M-2G:0M,2G-8G:192M,8G-:256M cryptomgr.notests cgroup.memory=nokmem rcupdate.rcu_cpu_stall_timeout=300 quiet biosdevname=0 net.ifnames=0 console=tty0 console=ttyS0,115200n8 noibrs nvme_core.io_timeout=4294967295 nvme_core.admin_timeout=4294967295
[root@iZbp1f1di6pkqcght0wc4nZ ~]#
[root@iZbp1f1di6pkqcght0wc4nZ ~]# lscpu
Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         48 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  16
  On-line CPU(s) list:   0-15
Vendor ID:               AuthenticAMD
  BIOS Vendor ID:        Alibaba Cloud
  Model name:            AMD EPYC 7T83 64-Core Processor
    BIOS Model name:     pc-i440fx-2.1  CPU @ 0.0GHz
    BIOS CPU family:     1
    CPU family:          25
    Model:               1
    Thread(s) per core:  2
    Core(s) per socket:  8
    Socket(s):           1
    Stepping:            1
    BogoMIPS:            5090.43
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2
                          ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid ext
                         d_apicid aperfmperf tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic
                          movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misali
                         gnsse 3dnowprefetch osvw topoext vmmcall tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx s
                         map clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr rdpru wbnoinvd arat v
                         aes vpclmulqdq rdpid fsrm
Virtualization features:
  Hypervisor vendor:     KVM
  Virtualization type:   full
Caches (sum of all):
  L1d:                   256 KiB (8 instances)
  L1i:                   256 KiB (8 instances)
  L2:                    4 MiB (8 instances)
  L3:                    32 MiB (1 instance)
NUMA:
  NUMA node(s):          1
  NUMA node0 CPU(s):     0-15
Vulnerabilities:
  Gather data sampling:  Not affected
  Itlb multihit:         Not affected
  L1tf:                  Not affected
  Mds:                   Not affected
  Meltdown:              Not affected
  Mmio stale data:       Not affected
  Retbleed:              Not affected
  Spec rstack overflow:  Vulnerable: Safe RET, no microcode
  Spec store bypass:     Vulnerable
  Spectre v1:            Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:            Mitigation; Retpolines, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected
  Srbds:                 Not affected
  Tsx async abort:       Not affected
[root@iZbp1f1di6pkqcght0wc4nZ ~]#
Comment 1 anolislw alibaba_cloud_group 2024-01-31 20:00:47 UTC 
附加下 x86环境下crash与kexec-tools得版本信息：
========================================

[root@iZbp1f1di6pkqcght0wc4nZ ~]# rpm -q crash
crash-8.0.2-2.an23.x86_64
[root@iZbp1f1di6pkqcght0wc4nZ ~]# rpm -q kexec-tools
kexec-tools-2.0.26-4.an23.x86_64
[root@iZbp1f1di6pkqcght0wc4nZ ~]# uname -r
6.6.7-1_rc2.an23.x86_64
[root@iZbp1f1di6pkqcght0wc4nZ ~]#
[root@iZbp1f1di6pkqcght0wc4nZ ~]# cat /etc/anolis-release
Anolis OS release 23
Comment 2 Banana alibaba_cloud_group 2024-02-01 10:22:23 UTC 
An23.1 5.10.134-16.2_rc2.an23.aarch64 ECS环境有相同问题

[root@localhost crash]# crash /usr/lib/debug/lib/modules/6.6.7-1_rc2.an23.aarch64/vmlinux 127.0.0.1-2024-01-31-15\:00\:39/vmcore

crash 8.0.2-2.an23
Copyright (C) 2002-2022  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2022  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
Copyright (C) 2015, 2021  VMware, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.

GNU gdb (GDB) 10.2
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-unknown-linux-gnu".
Type "show configuration" for configuration details.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...

please wait... (gathering module symbol data)
crash: invalid structure member offset: module_core_size
       FILE: kernel.c  LINE: 3781  FUNCTION: module_init()

[/usr/bin/crash] error trace: aaaad0a452cc => aaaad0710b30 => aaaad0790ef8 => aaaad08024f0

[root@localhost ~]# cat /etc/os-release
NAME="Anolis OS"
VERSION="23"
ID="anolis"
VERSION_ID="23"
PLATFORM_ID="platform:an23"
PRETTY_NAME="Anolis OS 23"
ANSI_COLOR="0;31"
HOME_URL="https://openanolis.cn/"
BUG_REPORT_URL="https://bugzilla.openanolis.cn/"

[root@localhost ~]# uname -a
Linux localhost.localdomain 6.6.7-1_rc2.an23.aarch64 #1 SMP Fri Jan 26 16:26:55 CST 2024 aarch64 aarch64 aarch64 GNU/Linux

[root@localhost ~]# cat /proc/cmdline
BOOT_IMAGE=(hd0,gpt2)/boot/vmlinuz-5.10.134-16.2_rc2.an23.aarch64 root=UUID=c0be7eac-d880-43e3-8d87-6b9e67f3ee1f ro rhgb crashkernel=0M-2G:0M,2G-64G:256M,64G-:384M iommu.passthrough=1 iommu.strict=0 cryptomgr.notests cgroup.memory=nokmem rcupdate.rcu_cpu_stall_timeout=300 quiet selinux=1 console=tty0 biosdevname=0 net.ifnames=0 console=ttyAMA0,115200n8 noibrs nvme_core.io_timeout=4294967295 nvme_core.admin_timeout=4294967295
Comment 3 Banana alibaba_cloud_group 2024-02-01 10:23:23 UTC 
(In reply to Banana from comment #2)
> An23.1 5.10.134-16.2_rc2.an23.aarch64 ECS环境有相同问题
> 
> [root@localhost crash]# crash
> /usr/lib/debug/lib/modules/6.6.7-1_rc2.an23.aarch64/vmlinux
> 127.0.0.1-2024-01-31-15\:00\:39/vmcore
> 
> 崩溃8.0.2-2.an23
> 版权所有 (C) 2002-2022 红帽公司
> 版权所有 (C) 2004、2005、2006、2010 IBM 公司
> 版权所有 (C) 1999-2006 惠普公司
> 版权所有 (C) 2005、2006、2011、2012 富士通有限公司
> 版权所有 (C) 2006、2007 VA Linux Systems Japan KK
> 版权所有 (C) 2005、2011、2020-2022 NEC 公司
> 版权所有 (C) 1999、2002、2007 Silicon Graphics, Inc.
> 版权所有 (C) 1999、2000、2001、2002 Mission Critical Linux, Inc.
> 版权所有 (C) 2015、2021 VMware, Inc.
> 该程序是免费软件，受 GNU 通用公共许可证保护，
> 欢迎您对其进行更改和/或分发其副本
> 一定条件下。输入“帮助复印”查看情况。
> 该程序绝对没有任何保证。输入“帮助保修”了解详细信息。
> 
> GNU gdb (GDB) 10.2
> 版权所有 (C) 2021 自由软件基金会, Inc.
> 许可证 GPLv3+：GNU GPL 版本 3 或更高版本 < http://gnu.org/licenses/gpl.html >
> 这是免费软件：您可以自由更改和重新分发它。
> 在法律允许的范围内，不提供任何保证。
> 输入“显示复制”和“显示保修”以了解详细信息。
> 该 GDB 被配置为“aarch64-unknown-linux-gnu”。
> 键入“显示配置”以获取配置详细信息。
> 在线查找 GDB 手册和其他文档资源：
>     < http://www.gnu.org/software/gdb/documentation/ >。
> 
> 如需帮助，请输入“帮助”。
> 输入“apropos word”来搜索与“word”相关的命令...
> 
> 请稍候...（正在收集模块符号数据）
> 崩溃：无效的结构成员偏移量：module_core_size
>        文件：kernel.c 行：3781 函数：module_init()
> 
> [/usr/bin/crash] 错误跟踪：aaaad0a452cc => aaaad0710b30 => aaaad0790ef8 =>
> aaaad08024f0
> 
> [root@localhost ~]# cat /etc/os-release
> 名称=“阿诺利斯操作系统”
> 版本=“23”
> ID=“阿诺利斯”
> 版本_ID =“23”
> PLATFORM_ID="平台：an23"
> PRETTY_NAME="阿诺利斯操作系统 23"
> ANSI_COLOR=“0;31”
> HOME_URL=" https://openanolis.cn/ "
> BUG_REPORT_URL=" https://bugzilla.openanolis.cn/ "
> 
> [root@localhost ~]# uname -a
> Linux localhost.localdomain 6.6.7-1_rc2.an23.aarch64 #1 SMP 1 月 26 日星期五
> 16:26:55 CST 2024 aarch64 aarch64 aarch64 GNU/Linux
> 
> [root@localhost ~]# cat /proc/cmdline
> BOOT_IMAGE=(hd0,gpt2)/boot/vmlinuz-5.10.134-16.2_rc2.an23.aarch64
> root=UUID=c0be7eac-d880-43e3-8d87-6b9e67f3ee1f ro rhgb
> crashkernel=0M-2G:0M,2G-64G:256M ,64G-:384M iommu.passthrough=1
> iommu.strict=0 cryptomgr.notests cgroup.memory=nokmem
> rcupdate.rcu_cpu_stall_timeout=300 安静 selinux=1 console=tty0 biosdevname=0
> net.ifnames=0 console=ttyAMA0,115200n8 noibrs nvme_core
> .io_timeout=4294967295 nvme_core.admin_timeout=4294967295

纠正：
应该是An23.1 6.6.7-1_rc2.an23.aarch64 ECS环境有相同问题
Comment 4 xiangzao alibaba_cloud_group 2024-02-21 12:42:17 UTC 
makedumpfile 版本为 1.7.2
1.7.2 只支持到 v6.0 内核
需要升级到 1.7.4， 支持 v6.6 内核
Comment 5 xiangzao alibaba_cloud_group 2024-02-21 14:49:22 UTC 
(In reply to xiangzao from comment #4)
> makedumpfile 版本为 1.7.2
> 1.7.2 只支持到 v6.0 内核
> 需要升级到 1.7.4， 支持 v6.6 内核

需要至少升级到 kexec-tools 2.0.27-3

* Wed Nov 08 2023 Tao Liu <ltao@redhat.com> - 2.0.27-3
- Rebase makedumpfile to v1.7.4
- powerpc: update kdumpctl to load kernel signing key for fadump
- powerpc: update kdumpctl to remove deletion of kernel signing key once loaded

目前源上最高版本为 kexec-tools-2.0.26-4.an23.x86_64
Comment 6 xiangzao alibaba_cloud_group 2024-02-21 19:37:49 UTC 
(In reply to xiangzao from comment #5)
> (In reply to xiangzao from comment #4)
> > makedumpfile 版本为 1.7.2
> > 1.7.2 只支持到 v6.0 内核
> > 需要升级到 1.7.4， 支持 v6.6 内核
> 
> 需要至少升级到 kexec-tools 2.0.27-3
> 
> * Wed Nov 08 2023 Tao Liu <ltao@redhat.com> - 2.0.27-3
> - Rebase makedumpfile to v1.7.4
> - powerpc: update kdumpctl to load kernel signing key for fadump
> - powerpc: update kdumpctl to remove deletion of kernel signing key once
> loaded
> 
> 目前源上最高版本为 kexec-tools-2.0.26-4.an23.x86_64

https://github.com/crash-utility/crash/commit/7750e61fdb2a083f26156a5338aa2ebe26447f3f

crash 工具也需要升级到 8.0.4
Comment 7 xiangzao alibaba_cloud_group 2024-02-27 11:36:26 UTC 
已验证目前 kexec-tools-2.0.26-5.an23.x86_64 和 crash-8.0.4-1.an23.x86_64 能够正常生成并解析 vmcore
Comment 8 Banana alibaba_cloud_group 2024-04-19 11:54:32 UTC 
在an23 arm环境6.6.25-2_rc1内核仍存在此问题，源内无可正常解析的crash、kexec-tools版本

[root@iZbp143ti4ccpaufkzata7Z kernel]# rpm -q crash
crash-8.0.2-2.an23.aarch64
[root@iZbp143ti4ccpaufkzata7Z kernel]# rpm -q kexec-tools
kexec-tools-2.0.26-3.an23.aarch64

[root@iZbp143ti4ccpaufkzata7Z crash]# crash /usr/lib/debug/lib/modules/6.6.25-2_rc1.an23.aarch64/vmlinux 127.0.0.1-2024-04-19-10\:34\:57/vmcore

crash 8.0.2-2.an23
Copyright (C) 2002-2022  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2022  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
Copyright (C) 2015, 2021  VMware, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.

GNU gdb (GDB) 10.2
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-unknown-linux-gnu".
Type "show configuration" for configuration details.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...

please wait... (gathering module symbol data)
crash: invalid structure member offset: module_core_size
       FILE: kernel.c  LINE: 3781  FUNCTION: module_init()

[/usr/bin/crash] error trace: aaaae10752cc => aaaae0d40b30 => aaaae0dc0ef8 => aaaae0e324f0



[root@iZbp143ti4ccpaufkzata7Z kernel]# uname -r
6.6.25-2_rc1.an23.aarch64

[root@iZbp143ti4ccpaufkzata7Z kernel]# cat /etc/os-release
NAME="Anolis OS"
VERSION="23"
ID="anolis"
VERSION_ID="23"
PLATFORM_ID="platform:an23"
PRETTY_NAME="Anolis OS 23"
ANSI_COLOR="0;31"
HOME_URL="https://openanolis.cn/"
BUG_REPORT_URL="https://bugzilla.openanolis.cn/"
Comment 9 anolislw alibaba_cloud_group 2024-04-19 12:33:32 UTC 
(In reply to Banana from comment #8)
> 在an23 arm环境6.6.25-2_rc1内核仍存在此问题，源内无可正常解析的crash、kexec-tools版本
> 
> [root@iZbp143ti4ccpaufkzata7Z kernel]# rpm -q crash
> crash-8.0.2-2.an23.aarch64
> [root@iZbp143ti4ccpaufkzata7Z kernel]# rpm -q kexec-tools
> kexec-tools-2.0.26-3.an23.aarch64
> 
> [root@iZbp143ti4ccpaufkzata7Z crash]# crash
> /usr/lib/debug/lib/modules/6.6.25-2_rc1.an23.aarch64/vmlinux
> 127.0.0.1-2024-04-19-10\:34\:57/vmcore
> 
> crash 8.0.2-2.an23
> Copyright (C) 2002-2022  Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
> Copyright (C) 1999-2006  Hewlett-Packard Co
> Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
> Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
> Copyright (C) 2005, 2011, 2020-2022  NEC Corporation
> Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
> Copyright (C) 2015, 2021  VMware, Inc.
> This program is free software, covered by the GNU General Public License,
> and you are welcome to change it and/or distribute copies of it under
> certain conditions.  Enter "help copying" to see the conditions.
> This program has absolutely no warranty.  Enter "help warranty" for details.
> 
> GNU gdb (GDB) 10.2
> Copyright (C) 2021 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
> Type "show copying" and "show warranty" for details.
> This GDB was configured as "aarch64-unknown-linux-gnu".
> Type "show configuration" for configuration details.
> Find the GDB manual and other documentation resources online at:
>     <http://www.gnu.org/software/gdb/documentation/>.
> 
> For help, type "help".
> Type "apropos word" to search for commands related to "word"...
> 
> please wait... (gathering module symbol data)
> crash: invalid structure member offset: module_core_size
>        FILE: kernel.c  LINE: 3781  FUNCTION: module_init()
> 
> [/usr/bin/crash] error trace: aaaae10752cc => aaaae0d40b30 => aaaae0dc0ef8
> => aaaae0e324f0
> 
> 
> 
> [root@iZbp143ti4ccpaufkzata7Z kernel]# uname -r
> 6.6.25-2_rc1.an23.aarch64
> 
> [root@iZbp143ti4ccpaufkzata7Z kernel]# cat /etc/os-release
> NAME="Anolis OS"
> VERSION="23"
> ID="anolis"
> VERSION_ID="23"
> PLATFORM_ID="platform:an23"
> PRETTY_NAME="Anolis OS 23"
> ANSI_COLOR="0;31"
> HOME_URL="https://openanolis.cn/"
> BUG_REPORT_URL="https://bugzilla.openanolis.cn/"


 在[6.6.25-2][RC1][Anolis23][x86_64]环境（kernel:6.6.25-2_rc1）下vmcore可以正常解析
------------- 记录 ------------
[root@iZbp1c9jzchxjqive233ugZ ~]# rpm -q kexec-tools
kexec-tools-2.0.26-5.an23.x86_64
[root@iZbp1c9jzchxjqive233ugZ ~]# rpm -q crash
crash-8.0.4-3.an23.x86_64
[root@iZbp1c9jzchxjqive233ugZ ~]# crash /usr/lib/debug/lib/modules/6.6.25-2_rc1.an23.x86_64/vmlinux /var/crash/127.0.0.1-2024-04-19-11\:17\:30/vmcore

crash 8.0.4-3.an23
Copyright (C) 2002-2022  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2022  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
Copyright (C) 2015, 2021  VMware, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.

GNU gdb (GDB) 10.2
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...

      KERNEL: /usr/lib/debug/lib/modules/6.6.25-2_rc1.an23.x86_64/vmlinux  [TAINTED]
    DUMPFILE: /var/crash/127.0.0.1-2024-04-19-11:17:30/vmcore  [PARTIAL DUMP]
        CPUS: 4
        DATE: Fri Apr 19 11:17:27 CST 2024
      UPTIME: 00:06:18
LOAD AVERAGE: 1.28, 0.91, 0.46
       TASKS: 244
    NODENAME: iZbp1c9jzchxjqive233ugZ
     RELEASE: 6.6.25-2_rc1.an23.x86_64
     VERSION: #1 SMP PREEMPT_DYNAMIC Thu Apr 11 15:05:18 CST 2024
     MACHINE: x86_64  (2700 Mhz)
      MEMORY: 15.7 GB
       PANIC: "Kernel panic - not syncing: sysrq triggered crash"
         PID: 4102
     COMMAND: "bash"
        TASK: ffff995386dd0000  [THREAD_INFO: ffff995386dd0000]
         CPU: 1
       STATE: TASK_RUNNING (PANIC)

crash> bt
PID: 4102     TASK: ffff995386dd0000  CPU: 1    COMMAND: "bash"
 #0 [ffffb58500ebbc80] machine_kexec at ffffffff810795cb
 #1 [ffffb58500ebbcd8] __crash_kexec at ffffffff8120a632
 #2 [ffffb58500ebbd90] panic at ffffffff8110ad49
 #3 [ffffb58500ebbe10] sysrq_handle_crash at ffffffff8185838a
 #4 [ffffb58500ebbe18] __handle_sysrq at ffffffff81858a24
 #5 [ffffb58500ebbe48] write_sysrq_trigger at ffffffff81859098
 #6 [ffffb58500ebbe58] proc_reg_write at ffffffff814f8dc9
 #7 [ffffb58500ebbe70] vfs_write at ffffffff81457a9b
 #8 [ffffb58500ebbf00] ksys_write at ffffffff81458107
 #9 [ffffb58500ebbf38] do_syscall_64 at ffffffff81cf344b
#10 [ffffb58500ebbf50] entry_SYSCALL_64_after_hwframe at ffffffff81e0012a
    RIP: 00007fcf4a4fc844  RSP: 00007ffde50955b8  RFLAGS: 00000202
    RAX: ffffffffffffffda  RBX: 0000000000000002  RCX: 00007fcf4a4fc844
    RDX: 0000000000000002  RSI: 0000560ad8e2caa0  RDI: 0000000000000001
    RBP: 0000560ad8e2caa0   R8: 0000000000000073   R9: 0000000000000001
    R10: 0000000000000000  R11: 0000000000000202  R12: 0000000000000002
    R13: 00007fcf4a6305c0  R14: 00007fcf4a5ce100  R15: 0000000000000000
    ORIG_RAX: 0000000000000001  CS: 0033  SS: 002b
crash> ^Z
[2]+  Stopped                 crash /usr/lib/debug/lib/modules/6.6.25-2_rc1.an23.x86_64/vmlinux /var/crash/127.0.0.1-2024-04-19-11\:17\:30/vmcore
[root@iZbp1c9jzchxjqive233ugZ ~]# cat /etc/os-release
NAME="Anolis OS"
VERSION="23"
ID="anolis"
VERSION_ID="23"
PLATFORM_ID="platform:an23"
PRETTY_NAME="Anolis OS 23"
ANSI_COLOR="0;31"
HOME_URL="https://openanolis.cn/"
BUG_REPORT_URL="https://bugzilla.openanolis.cn/"

[root@iZbp1c9jzchxjqive233ugZ ~]# uname -r
6.6.25-2_rc1.an23.x86_64
Comment 10 Banana alibaba_cloud_group 2024-04-19 14:21:06 UTC 
(In reply to Banana from comment #8)
> 在an23 arm环境6.6.25-2_rc1内核仍存在此问题，源内无可正常解析的crash、kexec-tools版本
> 
> [root@iZbp143ti4ccpaufkzata7Z kernel]# rpm -q crash
> crash-8.0.2-2.an23.aarch64
> [root@iZbp143ti4ccpaufkzata7Z kernel]# rpm -q kexec-tools
> kexec-tools-2.0.26-3.an23.aarch64
> 
> [root@iZbp143ti4ccpaufkzata7Z crash]# crash
> /usr/lib/debug/lib/modules/6.6.25-2_rc1.an23.aarch64/vmlinux
> 127.0.0.1-2024-04-19-10\:34\:57/vmcore
> 
> crash 8.0.2-2.an23
> Copyright (C) 2002-2022  Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
> Copyright (C) 1999-2006  Hewlett-Packard Co
> Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
> Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
> Copyright (C) 2005, 2011, 2020-2022  NEC Corporation
> Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
> Copyright (C) 2015, 2021  VMware, Inc.
> This program is free software, covered by the GNU General Public License,
> and you are welcome to change it and/or distribute copies of it under
> certain conditions.  Enter "help copying" to see the conditions.
> This program has absolutely no warranty.  Enter "help warranty" for details.
> 
> GNU gdb (GDB) 10.2
> Copyright (C) 2021 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
> Type "show copying" and "show warranty" for details.
> This GDB was configured as "aarch64-unknown-linux-gnu".
> Type "show configuration" for configuration details.
> Find the GDB manual and other documentation resources online at:
>     <http://www.gnu.org/software/gdb/documentation/>.
> 
> For help, type "help".
> Type "apropos word" to search for commands related to "word"...
> 
> please wait... (gathering module symbol data)
> crash: invalid structure member offset: module_core_size
>        FILE: kernel.c  LINE: 3781  FUNCTION: module_init()
> 
> [/usr/bin/crash] error trace: aaaae10752cc => aaaae0d40b30 => aaaae0dc0ef8
> => aaaae0e324f0
> 
> 
> 
> [root@iZbp143ti4ccpaufkzata7Z kernel]# uname -r
> 6.6.25-2_rc1.an23.aarch64
> 
> [root@iZbp143ti4ccpaufkzata7Z kernel]# cat /etc/os-release
> NAME="Anolis OS"
> VERSION="23"
> ID="anolis"
> VERSION_ID="23"
> PLATFORM_ID="platform:an23"
> PRETTY_NAME="Anolis OS 23"
> ANSI_COLOR="0;31"
> HOME_URL="https://openanolis.cn/"
> BUG_REPORT_URL="https://bugzilla.openanolis.cn/"

新增nightly源安装最新版本的crash、kexec-tools工具，crash触发并解析成功，但是有几个新增依赖，这几个依赖均存在于nightly源中，在an23默认源中不存在，请分析是否需要添加到默认源中
crash                8.0.4-3.an23   
glibc                2.38-1.an23    
glibc-common         2.38-1.an23    
glibc-devel          2.38-1.an23    
glibc-gconv-extra    2.38-1.an23    
glibc-langpack-en    2.38-1.an23    
kexec-tools          2.0.26-5.an23
Comment 11 gaochang alibaba_cloud_group 2024-04-29 14:16:56 UTC 
Nightly 源已可以获取，请再次测试
Comment 12 anolislw alibaba_cloud_group 2024-04-30 15:19:52 UTC 
(In reply to gaochang from comment #11)
> Nightly 源已可以获取，请再次测试

在搭乘 6.6.7-1_rc2.an23.x86_64 内核环境下（内部x86物理机）部署了nightly源，重新安装kexec-tools与crash，vmcore可以正常解析，验证信息如下：
==============================
[root@anolis ~]# crash /usr/lib/debug/lib/modules/6.6.7-1_rc2.an23.x86_64/vmlinux /var/crash/127.0.0.1-2024-04-30-15\:09\:13/vmcore

crash 8.0.4-3.an23
Copyright (C) 2002-2022  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2022  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
Copyright (C) 2015, 2021  VMware, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.

GNU gdb (GDB) 10.2
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...

      KERNEL: /usr/lib/debug/lib/modules/6.6.7-1_rc2.an23.x86_64/vmlinux  [TAINTED]
    DUMPFILE: /var/crash/127.0.0.1-2024-04-30-15:09:13/vmcore  [PARTIAL DUMP]
        CPUS: 96
        DATE: Tue Apr 30 15:09:09 CST 2024
      UPTIME: 00:03:21
LOAD AVERAGE: 1.22, 3.10, 1.54
       TASKS: 1482
    NODENAME: anolis
     RELEASE: 6.6.7-1_rc2.an23.x86_64
     VERSION: #1 SMP PREEMPT_DYNAMIC Fri Jan 26 16:30:26 CST 2024
     MACHINE: x86_64  (2499 Mhz)
      MEMORY: 255.7 GB
       PANIC: "Kernel panic - not syncing: sysrq triggered crash"
         PID: 14398
     COMMAND: "bash"
        TASK: ffff888123b2b180  [THREAD_INFO: ffff888123b2b180]
         CPU: 8
       STATE: TASK_RUNNING (PANIC)

crash> bt
PID: 14398    TASK: ffff888123b2b180  CPU: 8    COMMAND: "bash"
 #0 [ffffc9003b7afca8] machine_kexec at ffffffff97079e0b
 #1 [ffffc9003b7afcf8] __crash_kexec at ffffffff97200a51
 #2 [ffffc9003b7afda8] panic at ffffffff9710d849
 #3 [ffffc9003b7afe20] sysrq_handle_crash at ffffffff9780c79a
 #4 [ffffc9003b7afe28] __handle_sysrq at ffffffff9780cde4
 #5 [ffffc9003b7afe58] write_sysrq_trigger at ffffffff9780d458
 #6 [ffffc9003b7afe68] proc_reg_write at ffffffff974d53a9
 #7 [ffffc9003b7afe80] vfs_write at ffffffff9743a259
 #8 [ffffc9003b7aff08] ksys_write at ffffffff9743a877
 #9 [ffffc9003b7aff38] do_syscall_64 at ffffffff97c7844b
#10 [ffffc9003b7aff50] entry_SYSCALL_64_after_hwframe at ffffffff97e000ea
    RIP: 00007f4423cfc844  RSP: 00007ffc8cff67b8  RFLAGS: 00000202
    RAX: ffffffffffffffda  RBX: 0000000000000002  RCX: 00007f4423cfc844
    RDX: 0000000000000002  RSI: 0000560955d3a600  RDI: 0000000000000001
    RBP: 0000560955d3a600   R8: 0000000000000073   R9: 0000000000000001
    R10: 0000000000000000  R11: 0000000000000202  R12: 0000000000000002
    R13: 00007f4423e305c0  R14: 00007f4423dce100  R15: 0000000000000000
    ORIG_RAX: 0000000000000001  CS: 0033  SS: 002b
crash> ^C
crash> ^Z
[1]+  已停止               crash /usr/lib/debug/lib/modules/6.6.7-1_rc2.an23.x86_64/vmlinux /var/crash/127.0.0.1-2024-04-30-15\:09\:13/vmcore

[root@anolis ~]# uname -r
6.6.7-1_rc2.an23.x86_64

[root@anolis ~]# cat /etc/os-release
NAME="Anolis OS"
VERSION="23"
ID="anolis"
VERSION_ID="23"
PLATFORM_ID="platform:an23"
PRETTY_NAME="Anolis OS 23"
ANSI_COLOR="0;31"
HOME_URL="https://openanolis.cn/"
BUG_REPORT_URL="https://bugzilla.openanolis.cn/"

[root@anolis ~]# rpm -q crash
crash-8.0.4-3.an23.x86_64

[root@anolis ~]# rpm -q kexec-tools
kexec-tools-2.0.26-5.an23.x86_64
Comment 13 gaochang alibaba_cloud_group 2024-05-09 19:01:59 UTC 
已解决
Comment 14 anolislw alibaba_cloud_group 2024-05-22 10:38:44 UTC 
(In reply to anolislw from comment #12)
> (In reply to gaochang from comment #11)
> > Nightly 源已可以获取，请再次测试
> 
> 在搭乘 6.6.7-1_rc2.an23.x86_64
> 内核环境下（内部x86物理机）部署了nightly源，重新安装kexec-tools与crash，vmcore可以正常解析，验证信息如下：
> ==============================
> [root@anolis ~]# crash
> /usr/lib/debug/lib/modules/6.6.7-1_rc2.an23.x86_64/vmlinux
> /var/crash/127.0.0.1-2024-04-30-15\:09\:13/vmcore
> 
> crash 8.0.4-3.an23
> Copyright (C) 2002-2022  Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
> Copyright (C) 1999-2006  Hewlett-Packard Co
> Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
> Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
> Copyright (C) 2005, 2011, 2020-2022  NEC Corporation
> Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
> Copyright (C) 2015, 2021  VMware, Inc.
> This program is free software, covered by the GNU General Public License,
> and you are welcome to change it and/or distribute copies of it under
> certain conditions.  Enter "help copying" to see the conditions.
> This program has absolutely no warranty.  Enter "help warranty" for details.
> 
> GNU gdb (GDB) 10.2
> Copyright (C) 2021 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
> Type "show copying" and "show warranty" for details.
> This GDB was configured as "x86_64-pc-linux-gnu".
> Type "show configuration" for configuration details.
> Find the GDB manual and other documentation resources online at:
>     <http://www.gnu.org/software/gdb/documentation/>.
> 
> For help, type "help".
> Type "apropos word" to search for commands related to "word"...
> 
>       KERNEL: /usr/lib/debug/lib/modules/6.6.7-1_rc2.an23.x86_64/vmlinux 
> [TAINTED]
>     DUMPFILE: /var/crash/127.0.0.1-2024-04-30-15:09:13/vmcore  [PARTIAL DUMP]
>         CPUS: 96
>         DATE: Tue Apr 30 15:09:09 CST 2024
>       UPTIME: 00:03:21
> LOAD AVERAGE: 1.22, 3.10, 1.54
>        TASKS: 1482
>     NODENAME: anolis
>      RELEASE: 6.6.7-1_rc2.an23.x86_64
>      VERSION: #1 SMP PREEMPT_DYNAMIC Fri Jan 26 16:30:26 CST 2024
>      MACHINE: x86_64  (2499 Mhz)
>       MEMORY: 255.7 GB
>        PANIC: "Kernel panic - not syncing: sysrq triggered crash"
>          PID: 14398
>      COMMAND: "bash"
>         TASK: ffff888123b2b180  [THREAD_INFO: ffff888123b2b180]
>          CPU: 8
>        STATE: TASK_RUNNING (PANIC)
> 
> crash> bt
> PID: 14398    TASK: ffff888123b2b180  CPU: 8    COMMAND: "bash"
>  #0 [ffffc9003b7afca8] machine_kexec at ffffffff97079e0b
>  #1 [ffffc9003b7afcf8] __crash_kexec at ffffffff97200a51
>  #2 [ffffc9003b7afda8] panic at ffffffff9710d849
>  #3 [ffffc9003b7afe20] sysrq_handle_crash at ffffffff9780c79a
>  #4 [ffffc9003b7afe28] __handle_sysrq at ffffffff9780cde4
>  #5 [ffffc9003b7afe58] write_sysrq_trigger at ffffffff9780d458
>  #6 [ffffc9003b7afe68] proc_reg_write at ffffffff974d53a9
>  #7 [ffffc9003b7afe80] vfs_write at ffffffff9743a259
>  #8 [ffffc9003b7aff08] ksys_write at ffffffff9743a877
>  #9 [ffffc9003b7aff38] do_syscall_64 at ffffffff97c7844b
> #10 [ffffc9003b7aff50] entry_SYSCALL_64_after_hwframe at ffffffff97e000ea
>     RIP: 00007f4423cfc844  RSP: 00007ffc8cff67b8  RFLAGS: 00000202
>     RAX: ffffffffffffffda  RBX: 0000000000000002  RCX: 00007f4423cfc844
>     RDX: 0000000000000002  RSI: 0000560955d3a600  RDI: 0000000000000001
>     RBP: 0000560955d3a600   R8: 0000000000000073   R9: 0000000000000001
>     R10: 0000000000000000  R11: 0000000000000202  R12: 0000000000000002
>     R13: 00007f4423e305c0  R14: 00007f4423dce100  R15: 0000000000000000
>     ORIG_RAX: 0000000000000001  CS: 0033  SS: 002b
> crash> ^C
> crash> ^Z
> [1]+  已停止               crash
> /usr/lib/debug/lib/modules/6.6.7-1_rc2.an23.x86_64/vmlinux
> /var/crash/127.0.0.1-2024-04-30-15\:09\:13/vmcore
> 
> [root@anolis ~]# uname -r
> 6.6.7-1_rc2.an23.x86_64
> 
> [root@anolis ~]# cat /etc/os-release
> NAME="Anolis OS"
> VERSION="23"
> ID="anolis"
> VERSION_ID="23"
> PLATFORM_ID="platform:an23"
> PRETTY_NAME="Anolis OS 23"
> ANSI_COLOR="0;31"
> HOME_URL="https://openanolis.cn/"
> BUG_REPORT_URL="https://bugzilla.openanolis.cn/"
> 
> [root@anolis ~]# rpm -q crash
> crash-8.0.4-3.an23.x86_64
> 
> [root@anolis ~]# rpm -q kexec-tools
> kexec-tools-2.0.26-5.an23.x86_64

问题已解决，单子可以关闭！