3933 – ixgbe initiating reset due to tx timeout

Bug 3933 - ixgbe initiating reset due to tx timeout

Summary: ixgbe initiating reset due to tx timeout

Status:	NEW

Alias:	None

Product:	ANCK 4.19 Dev
Classification:	ANCK
Component:	drivers (show other bugs)	drivers
Sub Component:
Version:	4.19-024.x
Hardware:	x86_64 Linux

Importance:	P3-Medium S1-blocker
Target Milestone:	---
Assignee:	maqiao
QA Contact:	shuming

URL:
Whiteboard:
Keywords:

Depends on:
Blocks:

Reported:	2023-02-02 18:58 UTC by djhwyh
Modified:	2024-11-05 15:20 UTC (History)
CC List:	3 users (show)

See Also:

Attachments
vmcore-dmesg.txt (13.79 MB, text/plain) 2023-02-13 17:57 UTC, djhwyh	Details
need: unzip vmcore-dmesg.zip (13.79 MB, application/octet-stream) 2023-02-13 18:02 UTC, djhwyh	Details
View All Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.

Description djhwyh 2023-02-02 18:58:16 UTC

Description of problem:
多台服务器，而且是多个接入交换机下的，最近两天频繁出现网卡reset报错，系统内核和网卡驱动都没修改过，网卡都是英特尔 intel X540-T2 双口万兆电网卡。

Version-Release number of selected component (if applicable):
网卡驱动信息
 ethtool -i eth2
driver: igb
version: 5.4.0-k
firmware-version: 1.63, 0x800009fa
expansion-rom-version:
bus-info: 0000:08:00.2
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes


How reproducible:
主动复现步骤暂时无法确定。

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
一台服务器上message报错信息举例如下：
Feb  2 09:41:50 docker68 kernel: oom_reaper: reaped process 1003665 (nginx), now anon-rss:0kB, file-rss:0kB, shmem-rss:7076kB
Feb  2 09:41:52 docker68 kernel: ixgbe 0000:02:00.0 eth4: initiating reset due to tx timeout
Feb  2 09:41:53 docker68 kernel: ixgbe 0000:02:00.0 eth4: Reset adapter
Feb  2 09:41:59 docker68 kernel: ixgbe 0000:02:00.0 eth4: NIC Link is Up 10 Gbps, Flow Control: None
Feb  2 09:42:00 docker68 kernel: php-fpm invoked oom-killer: gfp_mask=0x6000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=958
Feb  2 09:42:00 docker68 kernel: php-fpm cpuset=8055f243eb1c52a0a3ec0880dc0c9c9b0f77061d4ac8fd25e88070e15c7dc772 mems_allowed=0-1
Feb  2 09:42:00 docker68 kernel: CPU: 3 PID: 1004112 Comm: php-fpm Kdump: loaded Tainted: G        W  OE     4.19.91-24.8.an8.x86_64 #1
Feb  2 09:42:00 docker68 kernel: Hardware name: New H3C Technologies Co., Ltd. R4900 G2/RS32M2C9S, BIOS 1.01.24 01/21/2019
Feb  2 09:42:00 docker68 kernel: Call Trace:
Feb  2 09:42:00 docker68 kernel: dump_stack+0x66/0x90
Feb  2 09:42:00 docker68 kernel: dump_memcg_header+0x12/0x39
Feb  2 09:42:00 docker68 kernel: oom_kill_process.cold.7+0x42/0x47
Feb  2 09:42:00 docker68 kernel: out_of_memory+0x19d/0x4a0
Feb  2 09:42:00 docker68 kernel: mem_cgroup_out_of_memory+0xd2/0xe0
Feb  2 09:42:00 docker68 kernel: try_charge+0x7c9/0x820
Feb  2 09:42:00 docker68 kernel: ? release_pages+0x3a1/0x430
Feb  2 09:42:00 docker68 kernel: mem_cgroup_charge+0x81/0x1f0
Feb  2 09:42:00 docker68 kernel: wp_page_copy+0x108/0x700
Feb  2 09:42:00 docker68 kernel: ? reuse_swap_page+0xf5/0x2e0
Feb  2 09:42:00 docker68 kernel: do_wp_page+0x7a/0x640
Feb  2 09:42:00 docker68 kernel: __handle_mm_fault+0x96a/0xa30
Feb  2 09:42:00 docker68 kernel: handle_mm_fault+0xc2/0x1e0
Feb  2 09:42:00 docker68 kernel: __do_page_fault+0x22a/0x490
Feb  2 09:42:00 docker68 kernel: do_page_fault+0x32/0x110
Feb  2 09:42:00 docker68 kernel: ? page_fault+0x8/0x30
Feb  2 09:42:00 docker68 kernel: page_fault+0x1e/0x30
Feb  2 09:42:00 docker68 kernel: RIP: 0033:0x7f2e90
Feb  2 09:42:00 docker68 kernel: Code: 01 00 00 4d 85 c0 7e 0d 48 8b 4b 20 49 39 c8 0f 8c d5 00 00 00 48 8b 05 ce e4 98 00 31 ff be 00 40 00 00 48 8b 90 98 00 00 00 <e8> 9b ff 00 00 48 89 43 38 48 8b 05 50 de 98 00 48 83 78 70 00 75
Feb  2 09:42:00 docker68 kernel: RSP: 002b:00007ffd85f83c50 EFLAGS: 00010246
Feb  2 09:42:00 docker68 kernel: RAX: 00000000011bbc60 RBX: 00000000011bc040 RCX: 00000000000001ec
Feb  2 09:42:00 docker68 kernel: RDX: 0000000000000000 RSI: 0000000000004000 RDI: 0000000000000000
Feb  2 09:42:00 docker68 kernel: RBP: 00007f2680c5600a R08: 0000000000800000 R09: f273143615999a0d
Feb  2 09:42:00 docker68 kernel: R10: 00000000022d60b0 R11: 0000ca63bb9b4120 R12: 0000000000000000
Feb  2 09:42:00 docker68 kernel: R13: 00007f2680c56000 R14: 00000000011bbf00 R15: 0000000000000000
Feb  2 09:42:00 docker68 kernel: Task in /kubepods/burstable/podd028c80a-21b2-4721-83f5-7e68cbc1e42d/8055f243eb1c52a0a3ec0880dc0c9c9b0f77061d4ac8fd25e88070e15c7dc772 killed as a result of limit of /kubepods/burstable/podd028c80a-21b2-4721-83f5-7e68cbc1e42d
Feb  2 09:42:00 docker68 kernel: memory: usage 8388608kB, limit 8388608kB, failcnt 46741853
Feb  2 09:42:00 docker68 kernel: memory+swap: usage 8388608kB, limit 9007199254740988kB, failcnt 0

Comment 1 maqiao alibaba_cloud_group

2023-02-13 10:08:29 UTC

更新进展：
2023.2.3：
根据upstream stable仓库中的bugfix补丁的commit log，怀疑是该补丁能够修复问题：08c2d7c8aa871027c7e56bf61335e70dd51bf017 (ixgbe: Fix packet corruption due to missing DMA sync)。
打上该补丁后，重新编译驱动，替换后进一步观察是否复现。

2023.2.10:
问题继续复现，该补丁无效。
下载官网源码包5.18.6和5.16.5版本，基于4.19.91-24.6.an8版本进行编译出包，安装rpm包以后继续观察。

Comment 2 maqiao alibaba_cloud_group

2023-02-13 10:08:53 UTC

intel ixgbe官网源码包下载地址：
https://www.intel.com/content/www/us/en/download/14302/intel-network-adapter-driver-for-pcie-intel-10-gigabit-ethernet-network-connections-under-linux.html

Comment 3 djhwyh 2023-02-13 17:57:15 UTC

Created attachment 630 [details]
vmcore-dmesg.txt

Comment 4 djhwyh 2023-02-13 18:02:03 UTC

Created attachment 631 [details]
need: unzip vmcore-dmesg.zip

CPU: 5 PID: 1251570 Comm: kworker/u80:2 Kdump: loaded Tainted: G        W  OEL    4.19.91-24.8.an8.x86_64 #1

Comment 5 djhwyh 2023-02-13 18:06:15 UTC

(In reply to maqiao from comment #1)
> 更新进展：
> 2023.2.3：
> 根据upstream stable仓库中的bugfix补丁的commit
> log，怀疑是该补丁能够修复问题：08c2d7c8aa871027c7e56bf61335e70dd51bf017 (ixgbe: Fix packet
> corruption due to missing DMA sync)。
> 打上该补丁后，重新编译驱动，替换后进一步观察是否复现。
> 
> 2023.2.10:
> 问题继续复现，该补丁无效。
> 下载官网源码包5.18.6和5.16.5版本，基于4.19.91-24.6.an8版本进行编译出包，安装rpm包以后继续观察。

今天，我对之前打过上述patch的一台复现报错的服务器，尝试用ethtool 的打开了所有debug消息输出后（执行命令是ethtool -s eth4 msglvl 0xffffff），又发现Reset报错了，另外一个新情况是，后来服务器直接crash了，有对应crash日志，大概如下，详细dmesg见https://bugzilla.openanolis.cn/show_bug.cgi?id=3933#c4 。请帮忙分析下。
      KERNEL: /usr/lib/debug/lib/modules/4.19.91-24.8.an8.x86_64/vmlinux
    DUMPFILE: vmcore  [PARTIAL DUMP]
        CPUS: 40
        DATE: Mon Feb 13 15:55:21 2023
      UPTIME: 102 days, 21:21:03
LOAD AVERAGE: 78.84, 56.58, 29.14
       TASKS: 1848
    NODENAME: docker66.add.zzzc2.qihoo.net
     RELEASE: 4.19.91-24.8.an8.x86_64
     VERSION: #1 SMP Tue Aug 31 11:30:53 CST 2021
     MACHINE: x86_64  (2195 Mhz)
      MEMORY: 191.9 GB
       PANIC: "general protection fault: 0000 [#1] SMP PTI"
         PID: 1251570
     COMMAND: "kworker/u80:2"
        TASK: ffff8cc37b59e100  [THREAD_INFO: ffff8cc37b59e100]
         CPU: 5
       STATE: TASK_RUNNING (PANIC)

crash> bt
PID: 1251570  TASK: ffff8cc37b59e100  CPU: 5   COMMAND: "kworker/u80:2"
 #0 [ffffb7d54ec77a98] machine_kexec at ffffffffb8063a8a
 #1 [ffffb7d54ec77ae8] __crash_kexec at ffffffffb813ff4a
 #2 [ffffb7d54ec77ba8] panic at ffffffffb809cf97
 #3 [ffffb7d54ec77c40] general_protection at ffffffffb8a0113e
    [exception RIP: hex_dump_to_buffer+153]
    RIP: ffffffffb8487c99  RSP: ffffb7d54ec77cf8  RFLAGS: 00010206
    RAX: 0001c683800000c0  RBX: 0000000000000010  RCX: 0000000000000010
    RDX: ffffb7d54ec77d76  RSI: 0000000000000010  RDI: 0000000000000010
    RBP: ffffb7d54ec77d75   R8: 0000000000000021   R9: 0000000000000083
    R10: 0000000000000000  R11: 0000000000000001  R12: 0000000000000010
    R13: 0000000000000083  R14: 0001c683800000c0  R15: 00000000000005f0
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #4 [ffffb7d54ec77d48] print_hex_dump at ffffffffb848812a
 #5 [ffffb7d54ec77e30] ixgbe_service_task.cold.61 at ffffffffc0d69ecd [ixgbe]
 #6 [ffffb7d54ec77e98] process_one_work at ffffffffb80b715f
 #7 [ffffb7d54ec77ed8] worker_thread at ffffffffb80b7380
 #8 [ffffb7d54ec77f10] kthread at ffffffffb80bce62
 #9 [ffffb7d54ec77f50] ret_from_fork at ffffffffb8a00215

Comment 6 djhwyh 2023-02-17 18:12:47 UTC

(In reply to djhwyh from comment #5)
> (In reply to maqiao from comment #1)
> > 更新进展：
> > 2023.2.3：
> > 根据upstream stable仓库中的bugfix补丁的commit
> > log，怀疑是该补丁能够修复问题：08c2d7c8aa871027c7e56bf61335e70dd51bf017 (ixgbe: Fix packet
> > corruption due to missing DMA sync)。
> > 打上该补丁后，重新编译驱动，替换后进一步观察是否复现。
> > 
> > 2023.2.10:
> > 问题继续复现，该补丁无效。
> > 下载官网源码包5.18.6和5.16.5版本，基于4.19.91-24.6.an8版本进行编译出包，安装rpm包以后继续观察。
> 
> 今天，我对之前打过上述patch的一台复现报错的服务器，尝试用ethtool 的打开了所有debug消息输出后（执行命令是ethtool -s eth4
> msglvl
> 0xffffff），又发现Reset报错了，另外一个新情况是，后来服务器直接crash了，有对应crash日志，大概如下，详细dmesg见https://
> bugzilla.openanolis.cn/show_bug.cgi?id=3933#c4 。请帮忙分析下。
>       KERNEL: /usr/lib/debug/lib/modules/4.19.91-24.8.an8.x86_64/vmlinux
>     DUMPFILE: vmcore  [PARTIAL DUMP]
>         CPUS: 40
>         DATE: Mon Feb 13 15:55:21 2023
>       UPTIME: 102 days, 21:21:03
> LOAD AVERAGE: 78.84, 56.58, 29.14
>        TASKS: 1848
>     NODENAME: docker66.add.zzzc2.qihoo.net
>      RELEASE: 4.19.91-24.8.an8.x86_64
>      VERSION: #1 SMP Tue Aug 31 11:30:53 CST 2021
>      MACHINE: x86_64  (2195 Mhz)
>       MEMORY: 191.9 GB
>        PANIC: "general protection fault: 0000 [#1] SMP PTI"
>          PID: 1251570
>      COMMAND: "kworker/u80:2"
>         TASK: ffff8cc37b59e100  [THREAD_INFO: ffff8cc37b59e100]
>          CPU: 5
>        STATE: TASK_RUNNING (PANIC)
> 
> crash> bt
> PID: 1251570  TASK: ffff8cc37b59e100  CPU: 5   COMMAND: "kworker/u80:2"
>  #0 [ffffb7d54ec77a98] machine_kexec at ffffffffb8063a8a
>  #1 [ffffb7d54ec77ae8] __crash_kexec at ffffffffb813ff4a
>  #2 [ffffb7d54ec77ba8] panic at ffffffffb809cf97
>  #3 [ffffb7d54ec77c40] general_protection at ffffffffb8a0113e
>     [exception RIP: hex_dump_to_buffer+153]
>     RIP: ffffffffb8487c99  RSP: ffffb7d54ec77cf8  RFLAGS: 00010206
>     RAX: 0001c683800000c0  RBX: 0000000000000010  RCX: 0000000000000010
>     RDX: ffffb7d54ec77d76  RSI: 0000000000000010  RDI: 0000000000000010
>     RBP: ffffb7d54ec77d75   R8: 0000000000000021   R9: 0000000000000083
>     R10: 0000000000000000  R11: 0000000000000001  R12: 0000000000000010
>     R13: 0000000000000083  R14: 0001c683800000c0  R15: 00000000000005f0
>     ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
>  #4 [ffffb7d54ec77d48] print_hex_dump at ffffffffb848812a
>  #5 [ffffb7d54ec77e30] ixgbe_service_task.cold.61 at ffffffffc0d69ecd [ixgbe]
>  #6 [ffffb7d54ec77e98] process_one_work at ffffffffb80b715f
>  #7 [ffffb7d54ec77ed8] worker_thread at ffffffffb80b7380
>  #8 [ffffb7d54ec77f10] kthread at ffffffffb80bce62
>  #9 [ffffb7d54ec77f50] ret_from_fork at ffffffffb8a00215

我们对业务下线流量的三台服务器进行了iperf3的压力测试，一开始故障能够多次复现，我们升级了其中一台的ixgbe驱动到社区推荐的5.16.5版本。但是目前用于测试的三台服务器都暂时不能复现报错了。我们在另一台业务服务器上发现如下message的log 请帮忙看能否分析出什么异常，操作系统版本和驱动之前都是一样的。
Feb 13 11:16:59 docker002 kernel: ------------[ cut here ]------------
Feb 13 11:16:59 docker002 kernel: NETDEV WATCHDOG: eth4 (ixgbe): transmit queue 23 timed out
Feb 13 11:16:59 docker002 kernel: WARNING: CPU: 27 PID: 148 at net/sched/sch_generic.c:466 dev_watchdog+0x1f3/0x200
Feb 13 11:16:59 docker002 kernel: Modules linked in: ip_set_hash_ip(E) bluetooth(E) ecdh_generic(E) rfkill(E) ceph(E) libceph(E) dns_resolver(E) fscache(E) binfmt_misc(E) udp_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) cls_bpf(E) sch_ingress(E) xt_TPROXY(E) nf_tproxy_ipv6(E) nf_tproxy_ipv4(E) xt_CT(E) nft_chain_route_ipv6(E) veth(E) bpfilter(E) xt_socket(E) nf_socket_ipv4(E) nf_socket_ipv6(E) ip6table_filter(E) ip6table_raw(E) ip6table_mangle(E) ip6_tables(E) iptable_filter(E) iptable_raw(E) iptable_mangle(E) iptable_nat(E) xt_set(E) ip_set_hash_ipportnet(E) ip_set_hash_ipportip(E) ip_set_hash_ipport(E) ip_set_bitmap_port(E) dummy(E) nft_chain_route_ipv4(E) xt_comment(E) xt_mark(E) ip_set(E) fuse(E) xt_conntrack(E) ipt_MASQUERADE(E) nf_conntrack_netlink(E) nft_counter(E) xt_addrtype(E) nft_compat(E)
Feb 13 11:16:59 docker002 kernel: nft_chain_nat_ipv4(E) nf_nat_ipv4(E) nf_tables(E) nfnetlink(E) overlay(E) ip_vs_ftp(E) nf_nat(E) ip_vs_sed(E) ip_vs_nq(E) ip_vs_fo(E) ip_vs_sh(E) ip_vs_dh(E) ip_vs_lblcr(E) ip_vs_lblc(E) ip_vs_wrr(E) ip_vs_rr(E) ip_vs_wlc(E) ip_vs_lc(E) ip_vs(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) toa(OE) br_netfilter(E) bridge(E) stp(E) llc(E) tcp_diag(E) inet_diag(E) intel_rapl_msr(E) intel_rapl_common(E) sb_edac(E) x86_pkg_temp_thermal(E) intel_powerclamp(E) coretemp(E) kvm_intel(E) kvm(E) iTCO_wdt(E) irqbypass(E) iTCO_vendor_support(E) ast(E) ipmi_ssif(E) crct10dif_pclmul(E) mxm_wmi(E) crc32_pclmul(E) ttm(E) ghash_clmulni_intel(E) drm_kms_helper(E) pcbc(E) syscopyarea(E) sysfillrect(E) aesni_intel(E) crypto_simd(E) cryptd(E) glue_helper(E) sysimgblt(E) fb_sys_fops(E) mei_me(E) drm(E)
Feb 13 11:16:59 docker002 kernel: ses(E) enclosure(E) scsi_transport_sas(E) pcspkr(E) mei(E) lpc_ich(E) i2c_i801(E) ipmi_si(E) ipmi_devintf(E) wmi(E) pcc_cpufreq(E) ipmi_msghandler(E) acpi_power_meter(E) sch_fq_codel(E) ip_tables(E) xfs(E) libcrc32c(E) sd_mod(E) sg(E) ahci(E) libahci(E) ixgbe(E) crc32c_intel(E) libata(E) megaraid_sas(E) mdio(E) igb(E) i2c_algo_bit(E) dca(E) i2c_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
Feb 13 11:16:59 docker002 kernel: CPU: 27 PID: 148 Comm: ksoftirqd/27 Kdump: loaded Tainted: G        W  OE     4.19.91-24.8.an8.x86_64 #1
Feb 13 11:16:59 docker002 kernel: Hardware name: New H3C Technologies Co., Ltd. R4900 G2/RS32M2C9S, BIOS 1.01.24 01/21/2019
Feb 13 11:16:59 docker002 kernel: RIP: 0010:dev_watchdog+0x1f3/0x200
Feb 13 11:16:59 docker002 kernel: Code: 00 48 63 4d e0 eb 93 4c 89 e7 c6 05 d5 1e c9 00 01 e8 d1 60 fc ff 89 d9 4c 89 e6 48 c7 c7 10 33 18 a1 48 89 c2 e8 57 dd 8f ff <0f> 0b eb c0 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 c7 47 08
Feb 13 11:16:59 docker002 kernel: RSP: 0018:ffffb94c4caa7da8 EFLAGS: 00010296
Feb 13 11:16:59 docker002 kernel: RAX: 000000000000003a RBX: 0000000000000017 RCX: 0000000000000000
Feb 13 11:16:59 docker002 kernel: RDX: ffff8c76ffa5f4c0 RSI: ffff8c76ffa56ac8 RDI: ffff8c76ffa56ac8
Feb 13 11:16:59 docker002 kernel: RBP: ffff8c76f1880480 R08: 0000000000000000 R09: 0000000000aaaaaa
Feb 13 11:16:59 docker002 kernel: R10: 0000000000000000 R11: 0000000000000001 R12: ffff8c76f1880000
Feb 13 11:16:59 docker002 kernel: R13: 000000000000001b R14: dead000000000200 R15: 0000000000000001
Feb 13 11:16:59 docker002 kernel: FS:  0000000000000000(0000) GS:ffff8c76ffa40000(0000) knlGS:0000000000000000
Feb 13 11:16:59 docker002 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 13 11:16:59 docker002 kernel: CR2: 000000c0166cf000 CR3: 000000145c20a004 CR4: 00000000003706e0
Feb 13 11:16:59 docker002 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Feb 13 11:16:59 docker002 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Feb 13 11:16:59 docker002 kernel: Call Trace:
Feb 13 11:16:59 docker002 kernel: ? qdisc_reset+0xe0/0xe0
Feb 13 11:16:59 docker002 kernel: call_timer_fn+0x2b/0x130
Feb 13 11:16:59 docker002 kernel: expire_timers+0x97/0xf0
Feb 13 11:16:59 docker002 kernel: run_timer_softirq+0x73/0x140
Feb 13 11:16:59 docker002 kernel: ? __switch_to+0x156/0x4f0
Feb 13 11:16:59 docker002 kernel: ? __switch_to_asm+0x35/0x70
Feb 13 11:16:59 docker002 kernel: __do_softirq+0xe7/0x2d3
Feb 13 11:16:59 docker002 kernel: ? sort_range+0x20/0x20
Feb 13 11:16:59 docker002 kernel: run_ksoftirqd+0x26/0x40
Feb 13 11:16:59 docker002 kernel: smpboot_thread_fn+0xc5/0x160
Feb 13 11:16:59 docker002 kernel: kthread+0x112/0x130
Feb 13 11:16:59 docker002 kernel: ? kthread_park+0x80/0x80
Feb 13 11:16:59 docker002 kernel: ret_from_fork+0x35/0x40
Feb 13 11:16:59 docker002 kernel: ---[ end trace dad93c2d8ce45437 ]---
Feb 13 11:16:59 docker002 kernel: ixgbe 0000:02:00.0 eth4: initiating reset due to tx timeout
Feb 13 11:16:59 docker002 kernel: ixgbe 0000:02:00.0 eth4: Reset adapter

Comment 7 djhwyh 2023-02-23 11:10:36 UTC

我们对一台还在报错的服务器网卡添加了如下参数：
ethtool -s ethX msglvl 0x2007
在报错复现时候，messages里有下面的输出，请帮忙分析：
Feb 23 08:36:04 docker058 kernel: ixgbe 0000:02:00.0 eth4: initiating reset due to tx timeout
Feb 23 08:36:04 docker058 kernel: ixgbe 0000:02:00.0: Net device Info
Feb 23 08:36:04 docker058 kernel: ixgbe: Device Name     state            trans_start
Feb 23 08:36:04 docker058 kernel: ixgbe: eth4            0000000000000003 000000046A1BADBE
Feb 23 08:36:04 docker058 kernel: ixgbe 0000:02:00.0: Register Dump
Feb 23 08:36:04 docker058 kernel: ixgbe:  Register Name   Value
Feb 23 08:36:04 docker058 kernel: ixgbe: CTRL            00000000
Feb 23 08:36:04 docker058 kernel: ixgbe: STATUS          00080004
Feb 23 08:36:04 docker058 kernel: ixgbe: CTRL_EXT        10010000
Feb 23 08:36:04 docker058 kernel: ixgbe: EICR            00000208
Feb 23 08:36:04 docker058 kernel: ixgbe: SRRCTL[0-7]     12000402 12000402 12000402 12000402 12000402 12000402 12000402 12000402
Feb 23 08:36:04 docker058 kernel: ixgbe: SRRCTL[8-15]    12000402 12000402 12000402 12000402 12000402 12000402 12000402 12000402
Feb 23 08:36:04 docker058 kernel: ixgbe: SRRCTL[16-23]   12000402 12000402 12000402 12000402 12000402 12000402 12000402 12000402
Feb 23 08:36:04 docker058 kernel: ixgbe: SRRCTL[24-31]   12000402 12000402 12000402 12000402 12000402 12000402 12000402 12000402
Feb 23 08:36:04 docker058 kernel: ixgbe: SRRCTL[32-39]   12000402 12000402 12000402 12000402 12000402 12000402 12000402 12000402
Feb 23 08:36:04 docker058 kernel: ixgbe: SRRCTL[40-47]   00000402 00000402 00000402 00000402 00000402 00000402 00000402 00000402
Feb 23 08:36:04 docker058 kernel: ixgbe: SRRCTL[48-55]   00000402 00000402 00000402 00000402 00000402 00000402 00000402 00000402
Feb 23 08:36:04 docker058 kernel: ixgbe: SRRCTL[56-63]   00000402 00000402 00000402 00000402 00000402 00000402 00000402 00000402
Feb 23 08:36:04 docker058 kernel: ixgbe: DRXCTL[0-7]     000002a0 000002a0 000002a0 000002a0 000002a0 000002a0 000002a0 000002a0
Feb 23 08:36:04 docker058 kernel: ixgbe: DRXCTL[8-15]    000002a0 000002a0 000002a0 000002a0 000002a0 000002a0 000002a0 000002a0
Feb 23 08:36:04 docker058 kernel: ixgbe: DRXCTL[16-23]   000002a0 000002a0 000002a0 000002a0 000002a0 000002a0 000002a0 000002a0
Feb 23 08:36:04 docker058 kernel: ixgbe: DRXCTL[24-31]   000002a0 000002a0 000002a0 000002a0 000002a0 000002a0 000002a0 000002a0
Feb 23 08:36:04 docker058 kernel: ixgbe: DRXCTL[32-39]   000002a0 000002a0 000002a0 000002a0 000002a0 000002a0 000002a0 000002a0
Feb 23 08:36:04 docker058 kernel: ixgbe: DRXCTL[40-47]   0000b200 0000b200 0000b200 0000b200 0000b200 0000b200 0000b200 0000b200
Feb 23 08:36:04 docker058 kernel: ixgbe: DRXCTL[48-55]   0000b200 0000b200 0000b200 0000b200 0000b200 0000b200 0000b200 0000b200
Feb 23 08:36:04 docker058 kernel: ixgbe: DRXCTL[56-63]   0000b200 0000b200 0000b200 0000b200 0000b200 0000b200 0000b200 0000b200
Feb 23 08:36:04 docker058 kernel: ixgbe: RDLEN[0-7]      00002000 00002000 00002000 00002000 00002000 00002000 00002000 00002000
Feb 23 08:36:04 docker058 kernel: ixgbe: RDLEN[8-15]     00002000 00002000 00002000 00002000 00002000 00002000 00002000 00002000
Feb 23 08:36:04 docker058 kernel: ixgbe: RDLEN[16-23]    00002000 00002000 00002000 00002000 00002000 00002000 00002000 00002000
Feb 23 08:36:04 docker058 kernel: ixgbe: RDLEN[24-31]    00002000 00002000 00002000 00002000 00002000 00002000 00002000 00002000
Feb 23 08:36:04 docker058 kernel: ixgbe: RDLEN[32-39]    00002000 00002000 00002000 00002000 00002000 00002000 00002000 00002000
Feb 23 08:36:04 docker058 kernel: ixgbe: RDLEN[40-47]    00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Feb 23 08:36:04 docker058 kernel: ixgbe: RDLEN[48-55]    00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Feb 23 08:36:04 docker058 kernel: ixgbe: RDLEN[56-63]    00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Feb 23 08:36:04 docker058 kernel: ixgbe: RDH[0-7]        000001f4 000000b5 000000af 00000124 00000156 000001b9 00000115 000000dc
Feb 23 08:36:04 docker058 kernel: ixgbe: RDH[8-15]       000001dc 000000a8 000000b2 00000103 0000010e 00000088 000000b5 000000b2
Feb 23 08:36:04 docker058 kernel: ixgbe: RDH[16-23]      00000064 00000017 0000000e 00000024 00000094 0000017f 0000007b 0000001f
Feb 23 08:36:04 docker058 kernel: ixgbe: RDH[24-31]      00000038 000000e9 0000007a 000001bf 00000168 00000100 00000003 0000000d
Feb 23 08:36:04 docker058 kernel: ixgbe: RDH[32-39]      00000007 0000003e 00000040 00000039 0000004f 00000022 0000001b 0000007f
Feb 23 08:36:04 docker058 kernel: ixgbe: RDH[40-47]      00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Feb 23 08:36:04 docker058 kernel: ixgbe: RDH[48-55]      00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Feb 23 08:36:04 docker058 kernel: ixgbe: RDH[56-63]      00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Feb 23 08:36:04 docker058 kernel: ixgbe: RDT[0-7]        000001ef 000000af 0000009f 0000010f 0000014f 000001af 0000010f 000000cf
Feb 23 08:36:04 docker058 kernel: ixgbe: RDT[8-15]       000001cf 0000009f 000000af 000000ff 000000ff 0000007f 000000af 000000af
Feb 23 08:36:04 docker058 kernel: ixgbe: RDT[16-23]      0000005f 0000000f 000001ff 0000001f 0000008f 0000016f 0000006f 0000000f
Feb 23 08:36:04 docker058 kernel: ixgbe: RDT[24-31]      0000002f 000000df 0000006f 000001af 0000015f 000000ff 000001ff 000001ff
Feb 23 08:36:04 docker058 kernel: ixgbe: RDT[32-39]      000001ff 0000002f 0000003f 0000002f 0000003f 0000001f 0000000f 0000006f
Feb 23 08:36:04 docker058 kernel: ixgbe: RDT[40-47]      00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Feb 23 08:36:04 docker058 kernel: ixgbe: RDT[48-55]      00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Feb 23 08:36:04 docker058 kernel: ixgbe: RDT[56-63]      00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Feb 23 08:36:04 docker058 kernel: ixgbe: RXDCTL[0-7]     42008600 42008600 42008600 42008600 42008600 42008600 42008600 42008600
Feb 23 08:36:04 docker058 kernel: ixgbe: RXDCTL[8-15]    42008600 42008600 42008600 42008600 42008600 42008600 42008600 42008600
Feb 23 08:36:04 docker058 kernel: ixgbe: RXDCTL[16-23]   42008600 42008600 42008600 42008600 42008600 42008600 42008600 42008600
Feb 23 08:36:04 docker058 kernel: ixgbe: RXDCTL[24-31]   42008600 42008600 42008600 42008600 42008600 42008600 42008600 42008600
Feb 23 08:36:04 docker058 kernel: ixgbe: RXDCTL[32-39]   42008600 42008600 42008600 42008600 42008600 42008600 42008600 42008600
Feb 23 08:36:04 docker058 kernel: ixgbe: RXDCTL[40-47]   00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Feb 23 08:36:04 docker058 kernel: ixgbe: RXDCTL[48-55]   00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Feb 23 08:36:04 docker058 kernel: ixgbe: RXDCTL[56-63]   00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Feb 23 08:36:04 docker058 kernel: ixgbe: RDBAL[0-7]      a3c10000 65eae000 c0ef2000 ceac2000 d400a000 b981c000 66ec4000 89c42000
Feb 23 08:36:04 docker058 kernel: ixgbe: RDBAL[8-15]     cd6bc000 5186e000 c64ce000 c3e56000 d344a000 d6fd2000 d63b8000 c1a70000
Feb 23 08:36:04 docker058 kernel: ixgbe: RDBAL[16-23]    c1058000 d5988000 904ac000 7bffc000 b68a2000 0eeae000 d472c000 ff7ca000
Feb 23 08:36:04 docker058 kernel: ixgbe: RDBAL[24-31]    d5e62000 3852c000 d1ff0000 2e370000 d0d2c000 f6618000 0505a000 0860c000
Feb 23 08:36:04 docker058 kernel: ixgbe: RDBAL[32-39]    fb306000 18eac000 ed8d2000 62c16000 da3b8000 dfaa2000 69e22000 511aa000
Feb 23 08:36:04 docker058 kernel: ixgbe: RDBAL[40-47]    00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Feb 23 08:36:04 docker058 kernel: ixgbe: RDBAL[48-55]    00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Feb 23 08:36:04 docker058 kernel: ixgbe: RDBAL[56-63]    00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Feb 23 08:36:04 docker058 kernel: ixgbe: RDBAH[0-7]      0000001a 00000002 0000000b 00000027 00000027 00000027 0000000e 00000015
Feb 23 08:36:04 docker058 kernel: ixgbe: RDBAH[8-15]     00000027 0000000a 0000004f 0000004f 0000004f 0000004f 0000004f 0000004f
Feb 23 08:36:04 docker058 kernel: ixgbe: RDBAH[16-23]    0000004f 0000004f 0000002b 0000003e 0000000b 00000011 00000027 0000000f
Feb 23 08:36:04 docker058 kernel: ixgbe: RDBAH[24-31]    00000027 00000005 00000027 0000001a 00000027 00000009 0000003e 0000002a
Feb 23 08:36:04 docker058 kernel: ixgbe: RDBAH[32-39]    00000034 00000041 00000030 0000003b 00000041 0000003e 00000039 00000041
Feb 23 08:36:04 docker058 kernel: ixgbe: RDBAH[40-47]    00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Feb 23 08:36:04 docker058 kernel: ixgbe: RDBAH[48-55]    00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Feb 23 08:36:04 docker058 kernel: ixgbe: RDBAH[56-63]    00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Feb 23 08:36:04 docker058 kernel: ixgbe: TDBAL[0-7]      93558000 d51b0000 3a3f8000 bfd72000 d0be2000 d2b2e000 bd5f0000 ba350000
Feb 23 08:36:04 docker058 kernel: ixgbe: TDBAL[8-15]     7dc22000 d64de000 de3f0000 3a62c000 6d81c000 c2a10000 86d04000 3f08c000
Feb 23 08:36:04 docker058 kernel: ixgbe: TDBAL[16-23]    65806000 fd698000 49c6e000 64918000 b8556000 d69b4000 f332e000 d72aa000
Feb 23 08:36:04 docker058 kernel: ixgbe: TDBAL[24-31]    d720a000 a9494000 420e0000 d0ee6000 000c4000 67848000 5cf4a000 3b79c000
Feb 23 08:36:04 docker058 kernel: ixgbe: TDBAL[32-39]    c8bf6000 d5c88000 bd994000 c727e000 c214e000 c66b2000 c7096000 c3320000
Feb 23 08:36:04 docker058 kernel: ixgbe: TDBAL[40-47]    00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Feb 23 08:36:04 docker058 kernel: ixgbe: TDBAL[48-55]    00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Feb 23 08:36:04 docker058 kernel: ixgbe: TDBAL[56-63]    00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Feb 23 08:36:04 docker058 kernel: ixgbe: TDBAH[0-7]      00000008 00000027 00000010 00000027 00000027 00000027 00000027 00000027
Feb 23 08:36:04 docker058 kernel: ixgbe: TDBAH[8-15]     00000001 00000027 00000031 0000003e 00000037 00000032 0000002b 00000042
Feb 23 08:36:04 docker058 kernel: ixgbe: TDBAH[16-23]    00000032 0000002c 0000003b 0000002d 00000018 00000027 0000001e 00000027
Feb 23 08:36:04 docker058 kernel: ixgbe: TDBAH[24-31]    00000027 0000000e 00000011 00000027 0000001f 0000000e 0000002b 0000002d
Feb 23 08:36:04 docker058 kernel: ixgbe: TDBAH[32-39]    0000004f 0000004f 0000004f 0000004f 0000004f 0000004f 0000004f 0000004f
Feb 23 08:36:04 docker058 kernel: ixgbe: TDBAH[40-47]    00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Feb 23 08:36:04 docker058 kernel: ixgbe: TDBAH[48-55]    00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Feb 23 08:36:04 docker058 kernel: ixgbe: TDBAH[56-63]    00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Feb 23 08:36:04 docker058 kernel: ixgbe: TDLEN[0-7]      00002000 00002000 00002000 00002000 00002000 00002000 00002000 00002000
Feb 23 08:36:04 docker058 kernel: ixgbe: TDLEN[8-15]     00002000 00002000 00002000 00002000 00002000 00002000 00002000 00002000
Feb 23 08:36:04 docker058 kernel: ixgbe: TDLEN[16-23]    00002000 00002000 00002000 00002000 00002000 00002000 00002000 00002000
Feb 23 08:36:04 docker058 kernel: ixgbe: TDLEN[24-31]    00002000 00002000 00002000 00002000 00002000 00002000 00002000 00002000
Feb 23 08:36:04 docker058 kernel: ixgbe: TDLEN[32-39]    00002000 00002000 00002000 00002000 00002000 00002000 00002000 00002000
Feb 23 08:36:04 docker058 kernel: ixgbe: TDLEN[40-47]    00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Feb 23 08:36:04 docker058 kernel: ixgbe: TDLEN[48-55]    00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Feb 23 08:36:04 docker058 kernel: ixgbe: TDLEN[56-63]    00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Feb 23 08:36:04 docker058 kernel: ixgbe: TDH[0-7]        00000125 0000017c 00000198 000001e5 000000b4 000001f8 000000d2 0000006d
Feb 23 08:36:04 docker058 kernel: ixgbe: TDH[8-15]       00000032 000001f0 000001e4 000000ef 00000102 00000065 0000007f 00000075
Feb 23 08:36:04 docker058 kernel: ixgbe: TDH[16-23]      000000ce 00000050 000000cd 00000083 000000f2 000001ff 0000007d 00000182
Feb 23 08:36:04 docker058 kernel: ixgbe: TDH[24-31]      000001a4 000001bc 000001f4 000000e7 00000028 00000009 00000061 00000043
Feb 23 08:36:04 docker058 kernel: ixgbe: TDH[32-39]      0000006e 00000152 0000004c 000000bb 000000ac 000000f9 00000079 0000014c
Feb 23 08:36:04 docker058 kernel: ixgbe: TDH[40-47]      00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Feb 23 08:36:04 docker058 kernel: ixgbe: TDH[48-55]      00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Feb 23 08:36:04 docker058 kernel: ixgbe: TDH[56-63]      00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Feb 23 08:36:04 docker058 kernel: ixgbe: TDT[0-7]        00000125 0000017c 00000198 000001e5 000000b4 000001f8 000000d2 0000006d
Feb 23 08:36:04 docker058 kernel: ixgbe: TDT[8-15]       00000032 000001f0 000001e4 000000ef 00000102 00000065 0000007f 00000075
Feb 23 08:36:04 docker058 kernel: ixgbe: TDT[16-23]      000000ce 00000050 000000cd 00000083 000000f2 000001ff 0000007d 00000182
Feb 23 08:36:04 docker058 kernel: ixgbe: TDT[24-31]      000001a4 000001bc 000001f4 000000e7 00000028 00000009 00000061 00000043
Feb 23 08:36:04 docker058 kernel: ixgbe: TDT[32-39]      0000006e 00000152 0000004c 000000bb 000000ac 000000f9 00000079 0000014c
Feb 23 08:36:04 docker058 kernel: ixgbe: TDT[40-47]      00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Feb 23 08:36:04 docker058 kernel: ixgbe: TDT[48-55]      00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Feb 23 08:36:04 docker058 kernel: ixgbe: TDT[56-63]      00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Feb 23 08:36:04 docker058 kernel: ixgbe: TXDCTL[0-7]     02080120 02080120 02080120 02080120 02080120 02080120 02080120 02080120
Feb 23 08:36:04 docker058 kernel: ixgbe: TXDCTL[8-15]    02080120 02080120 02080120 02080120 02080120 02080120 02080120 02080120
Feb 23 08:36:04 docker058 kernel: ixgbe: TXDCTL[16-23]   02080120 02080120 02080120 02080120 02080120 02080120 02080120 02080120
Feb 23 08:36:04 docker058 kernel: ixgbe: TXDCTL[24-31]   02080120 02080120 02080120 02080120 02080120 02080120 02080120 02080120
Feb 23 08:36:04 docker058 kernel: ixgbe: TXDCTL[32-39]   02080120 02080120 02080120 02080120 02080120 02080120 02080120 02080120
Feb 23 08:36:04 docker058 kernel: ixgbe: TXDCTL[40-47]   00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Feb 23 08:36:04 docker058 kernel: ixgbe: TXDCTL[48-55]   00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Feb 23 08:36:04 docker058 kernel: ixgbe: TXDCTL[56-63]   00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Feb 23 08:36:04 docker058 kernel: ixgbe 0000:02:00.0: TX Rings Summary
Feb 23 08:36:04 docker058 kernel: ixgbe:  Queue [NTU] [NTC] [bi(ntc)->dma  ]     leng              ntw        timestamp
Feb 23 08:36:04 docker058 kernel: ixgbe:      0   125   125 00000030CC4E0000 00000000           (null) 0000000000000000
Feb 23 08:36:04 docker058 kernel: ixgbe:      1   17C   17C 000000393E4744C6 00000000           (null) 000000046A154FE5
Feb 23 08:36:04 docker058 kernel: ixgbe:      2   198   198 0000000000000000 00000000           (null) 0000000000000000
Feb 23 08:36:04 docker058 kernel: ixgbe:      3   1E5   1DF 0000001EE578F0DA 00000036 0000000035425a90 000000046A1B9A28
Feb 23 08:36:04 docker058 kernel: ixgbe:      4    B4    B4 0000000000000000 00000000           (null) 0000000000000000
Feb 23 08:36:04 docker058 kernel: ixgbe:      5   1F8   1F8 0000000000000000 00000000           (null) 0000000000000000
Feb 23 08:36:04 docker058 kernel: ixgbe:      6    D2    D2 00000035A0EAF602 00000000           (null) 000000046A1575B4
Feb 23 08:36:04 docker058 kernel: ixgbe:      7    6D    6D 0000000000000000 00000000           (null) 0000000000000000
Feb 23 08:36:05 docker058 kernel: ixgbe:      8    32    32 0000004E7990D802 00000000           (null) 000000046A17D2F8
Feb 23 08:36:05 docker058 kernel: ixgbe:      9   1F0   1EC 0000000CA4150002 0000003A 000000009575f75a 000000046A1B9AFC
Feb 23 08:36:05 docker058 kernel: ixgbe:     10   1E4   1E4 0000000000000000 00000000           (null) 0000000000000000
Feb 23 08:36:05 docker058 kernel: ixgbe:     11    EF    EF 0000000000000000 00000000           (null) 0000000000000000
Feb 23 08:36:05 docker058 kernel: ixgbe:     12   102   102 0000000000000000 00000000           (null) 0000000000000000
Feb 23 08:36:05 docker058 kernel: ixgbe:     13    65    65 0000000000000000 00000000           (null) 0000000000000000
Feb 23 08:36:05 docker058 kernel: ixgbe:     14    7F    7F 0000000000000000 00000000           (null) 0000000000000000
Feb 23 08:36:05 docker058 kernel: ixgbe:     15    75    75 0000000000000000 00000000           (null) 0000000000000000
Feb 23 08:36:05 docker058 kernel: ixgbe:     16    CE    CE 0000000000000000 00000000           (null) 0000000000000000
Feb 23 08:36:05 docker058 kernel: ixgbe:     17    50    50 0000000000000000 00000000           (null) 0000000000000000
Feb 23 08:36:05 docker058 kernel: ixgbe:     18    CD    CD 0000000000000000 00000000           (null) 0000000000000000
Feb 23 08:36:05 docker058 kernel: ixgbe:     19    83    83 0000000000000000 00000000           (null) 0000000000000000
Feb 23 08:36:05 docker058 kernel: ixgbe:     20    F2    F2 0000001C86C8E4DA 00000000           (null) 000000046A17D341
Feb 23 08:36:05 docker058 kernel: ixgbe:     21   1FF   1FF 0000003A50915402 00000000           (null) 000000046A18243A
Feb 23 08:36:05 docker058 kernel: ixgbe:     22    7D    7D 0000000E840C8CCE 00000000           (null) 000000046A154DD6
Feb 23 08:36:05 docker058 kernel: ixgbe:     23   182   182 0000000000000000 00000000           (null) 0000000000000000
Feb 23 08:36:05 docker058 kernel: ixgbe:     24   1A4   1A4 0000000000000000 00000000           (null) 0000000000000000
Feb 23 08:36:05 docker058 kernel: ixgbe:     25   1BC   1BC 0000001F035A58DA 00000000           (null) 000000046A1902CF
Feb 23 08:36:05 docker058 kernel: ixgbe:     26   1F4   1F4 0000000000000000 00000000           (null) 0000000000000000
Feb 23 08:36:05 docker058 kernel: ixgbe:     27    E7    E7 0000003A51F03C02 00000000           (null) 000000046A161674
Feb 23 08:36:05 docker058 kernel: ixgbe:     28    28    28 0000000AAAA53CDA 00000000           (null) 000000046A18CA83
Feb 23 08:36:05 docker058 kernel: ixgbe:     29     9     9 0000002883705CC6 00000000           (null) 000000046A18BFC0
Feb 23 08:36:05 docker058 kernel: ixgbe:     30    61    61 0000000000000000 00000000           (null) 0000000000000000
Feb 23 08:36:05 docker058 kernel: ixgbe:     31    43    43 0000000000000000 00000000           (null) 0000000000000000
Feb 23 08:36:05 docker058 kernel: ixgbe:     32    6E    6E 0000000000000000 00000000           (null) 0000000000000000
Feb 23 08:36:05 docker058 kernel: ixgbe:     33   152   152 0000000000000000 00000000           (null) 0000000000000000
Feb 23 08:36:05 docker058 kernel: ixgbe:     34    4C    4C 0000000000000000 00000000           (null) 0000000000000000
Feb 23 08:36:05 docker058 kernel: ixgbe:     35    BB    BB 0000000000000000 00000000           (null) 0000000000000000
Feb 23 08:36:05 docker058 kernel: ixgbe:     36    AC    AC 0000000000000000 00000000           (null) 0000000000000000
Feb 23 08:36:05 docker058 kernel: ixgbe:     37    F9    F9 0000000000000000 00000000           (null) 0000000000000000
Feb 23 08:36:05 docker058 kernel: ixgbe:     38    79    79 0000000000000000 00000000           (null) 0000000000000000
Feb 23 08:36:05 docker058 kernel: ixgbe:     39   14C   14C 0000000000000000 00000000           (null) 0000000000000000
Feb 23 08:36:05 docker058 kernel: ixgbe 0000:02:00.0: RX Rings Summary
Feb 23 08:36:05 docker058 kernel: ixgbe: Queue [NTU] [NTC]
Feb 23 08:36:05 docker058 kernel: ixgbe:     0   1EF   1F4
Feb 23 08:36:05 docker058 kernel: ixgbe:     1    AF    B5
Feb 23 08:36:05 docker058 kernel: ixgbe:     2    9F    AF
Feb 23 08:36:05 docker058 kernel: ixgbe:     3   10F   118
Feb 23 08:36:05 docker058 kernel: ixgbe:     4   14F   156
Feb 23 08:36:05 docker058 kernel: ixgbe:     5   1AF   1B9
Feb 23 08:36:05 docker058 kernel: ixgbe:     6   10F   115
Feb 23 08:36:05 docker058 kernel: ixgbe:     7    CF    DC
Feb 23 08:36:05 docker058 kernel: ixgbe:     8   1CF   1DC
Feb 23 08:36:05 docker058 kernel: ixgbe:     9    9F    A0
Feb 23 08:36:05 docker058 kernel: ixgbe:    10    AF    B2
Feb 23 08:36:05 docker058 kernel: ixgbe:    11    FF   103
Feb 23 08:36:05 docker058 kernel: ixgbe:    12    FF   10E
Feb 23 08:36:05 docker058 kernel: ixgbe:    13    7F    88
Feb 23 08:36:05 docker058 kernel: ixgbe:    14    AF    B5
Feb 23 08:36:05 docker058 kernel: ixgbe:    15    AF    B2
Feb 23 08:36:05 docker058 kernel: ixgbe:    16    5F    64
Feb 23 08:36:05 docker058 kernel: ixgbe:    17     F    17
Feb 23 08:36:05 docker058 kernel: ixgbe:    18   1FF     E
Feb 23 08:36:05 docker058 kernel: ixgbe:    19    1F    24
Feb 23 08:36:05 docker058 kernel: ixgbe:    20    8F    94
Feb 23 08:36:05 docker058 kernel: ixgbe:    21   16F   17F
Feb 23 08:36:05 docker058 kernel: ixgbe:    22    6F    7B
Feb 23 08:36:05 docker058 kernel: ixgbe:    23     F    1F
Feb 23 08:36:05 docker058 kernel: ixgbe:    24    2F    38
Feb 23 08:36:05 docker058 kernel: ixgbe:    25    DF    E9
Feb 23 08:36:05 docker058 kernel: ixgbe:    26    6F    7A
Feb 23 08:36:05 docker058 kernel: ixgbe:    27   1AF   1BF
Feb 23 08:36:05 docker058 kernel: ixgbe:    28   15F   168
Feb 23 08:36:05 docker058 kernel: ixgbe:    29    FF   100
Feb 23 08:36:05 docker058 kernel: ixgbe:    30   1FF     3
Feb 23 08:36:05 docker058 kernel: ixgbe:    31   1FF     D
Feb 23 08:36:05 docker058 kernel: ixgbe:    32   1FF     7
Feb 23 08:36:05 docker058 kernel: ixgbe:    33    2F    3E
Feb 23 08:36:05 docker058 kernel: ixgbe:    34    3F    40
Feb 23 08:36:05 docker058 kernel: ixgbe:    35    2F    39
Feb 23 08:36:05 docker058 kernel: ixgbe:    36    3F    4F
Feb 23 08:36:05 docker058 kernel: ixgbe:    37    1F    22
Feb 23 08:36:05 docker058 kernel: ixgbe:    38     F    1B
Feb 23 08:36:05 docker058 kernel: ixgbe:    39    6F    7F
Feb 23 08:36:05 docker058 kernel: ixgbe 0000:02:00.0 eth4: Reset adapter
Feb 23 08:36:13 docker058 kernel: ixgbe 0000:02:00.0 eth4: NIC Link is Up 10 Gbps, Flow Control: None

Comment 8 djhwyh 2023-03-02 15:47:50 UTC

更正下驱动版本信息
lspci|grep - i eth
04:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
04:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)

Version-Release number of selected component (if applicable):
网卡驱动信息
driver: ixgbe
version: 5.1.0-k
firmware-version: 0x80000625
expansion-rom-version:
bus-info: 0000:04:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

Comment 9 pomegranate 2024-11-01 12:04:23 UTC

我们也遇到了相同问题，麒麟V10 sp2  intel x540 T2 驱动版本5.1.0-k。麻烦看到信息回复下。

Comment 10 djhwyh 2024-11-04 18:20:35 UTC

(In reply to pomegranate from comment #9)
> 我们也遇到了相同问题，麒麟V10 sp2  intel x540 T2 驱动版本5.1.0-k。麻烦看到信息回复下。
我们的总体进度和初步结论如下，仅供参考，不确定是否适用你的问题，或者说如果你也是X540问题，那么除非INTEL或你们的服务器厂家或者社区能提供特别支持，否则可能很难定位解决。当然如果你们后续有什么好的解决办法，也欢迎分享。从我个人建议，不要再在有问题的服务器上使用这种INTEL都EOL的X540网卡了。
“从2023年02月03日业务反馈到（当时最后一次处理）2023年3月2日，
 服务器和网卡过保，报修困难。X540网卡也已过了Intel官方的软件维护周期。

 尝试了通过龙蜥社区、浪潮研发两个渠道推动，都无法通过他们反向推动Intel为这个问题建立CASE。

 触发故障的根因和复现方法很难追查，目前线索指向和业务类型有关，和网卡驱动强相关

 基本判断和服务器品牌型号无关，已经出现报错的服务器品牌有浪潮、H3C等

因无法追到根因，不敢冒然指定解决方法。”

Comment 11 pomegranate 2024-11-05 15:20:59 UTC

最后我们的定位到了问题，是linux tc规则累计过多，导致了网络堵塞，引发了tx time out。