Menu
▾
▴
e1000-devel — Discussion of the Intel Ethernet out-of-tree drivers
You can subscribe to this list here.
| 2002 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
(2) |
Nov
(1) |
Dec
|
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2003 |
Jan
|
Feb
|
Mar
(1) |
Apr
(9) |
May
(3) |
Jun
|
Jul
(3) |
Aug
(6) |
Sep
|
Oct
(7) |
Nov
|
Dec
|
| 2004 |
Jan
|
Feb
(5) |
Mar
(10) |
Apr
(2) |
May
(22) |
Jun
(8) |
Jul
(4) |
Aug
(8) |
Sep
(3) |
Oct
|
Nov
(36) |
Dec
(52) |
| 2005 |
Jan
(9) |
Feb
(13) |
Mar
(9) |
Apr
|
May
(14) |
Jun
(5) |
Jul
(20) |
Aug
(31) |
Sep
(2) |
Oct
(3) |
Nov
(18) |
Dec
(18) |
| 2006 |
Jan
(36) |
Feb
(16) |
Mar
(76) |
Apr
(78) |
May
(32) |
Jun
(30) |
Jul
(67) |
Aug
(43) |
Sep
(54) |
Oct
(116) |
Nov
(223) |
Dec
(158) |
| 2007 |
Jan
(180) |
Feb
(71) |
Mar
(110) |
Apr
(114) |
May
(203) |
Jun
(100) |
Jul
(238) |
Aug
(191) |
Sep
(177) |
Oct
(171) |
Nov
(211) |
Dec
(159) |
| 2008 |
Jan
(227) |
Feb
(288) |
Mar
(197) |
Apr
(253) |
May
(132) |
Jun
(152) |
Jul
(109) |
Aug
(143) |
Sep
(157) |
Oct
(198) |
Nov
(121) |
Dec
(147) |
| 2009 |
Jan
(105) |
Feb
(61) |
Mar
(191) |
Apr
(161) |
May
(118) |
Jun
(172) |
Jul
(166) |
Aug
(67) |
Sep
(86) |
Oct
(79) |
Nov
(118) |
Dec
(181) |
| 2010 |
Jan
(136) |
Feb
(154) |
Mar
(92) |
Apr
(83) |
May
(101) |
Jun
(66) |
Jul
(118) |
Aug
(78) |
Sep
(134) |
Oct
(131) |
Nov
(132) |
Dec
(104) |
| 2011 |
Jan
(79) |
Feb
(104) |
Mar
(144) |
Apr
(145) |
May
(130) |
Jun
(169) |
Jul
(146) |
Aug
(76) |
Sep
(113) |
Oct
(82) |
Nov
(145) |
Dec
(122) |
| 2012 |
Jan
(132) |
Feb
(106) |
Mar
(145) |
Apr
(238) |
May
(140) |
Jun
(162) |
Jul
(166) |
Aug
(147) |
Sep
(80) |
Oct
(148) |
Nov
(192) |
Dec
(90) |
| 2013 |
Jan
(139) |
Feb
(162) |
Mar
(174) |
Apr
(81) |
May
(261) |
Jun
(301) |
Jul
(106) |
Aug
(175) |
Sep
(305) |
Oct
(222) |
Nov
(95) |
Dec
(120) |
| 2014 |
Jan
(196) |
Feb
(171) |
Mar
(146) |
Apr
(118) |
May
(127) |
Jun
(93) |
Jul
(175) |
Aug
(66) |
Sep
(85) |
Oct
(120) |
Nov
(81) |
Dec
(192) |
| 2015 |
Jan
(141) |
Feb
(133) |
Mar
(189) |
Apr
(126) |
May
(59) |
Jun
(117) |
Jul
(56) |
Aug
(97) |
Sep
(44) |
Oct
(48) |
Nov
(33) |
Dec
(87) |
| 2016 |
Jan
(37) |
Feb
(56) |
Mar
(72) |
Apr
(65) |
May
(66) |
Jun
(65) |
Jul
(98) |
Aug
(54) |
Sep
(84) |
Oct
(68) |
Nov
(69) |
Dec
(60) |
| 2017 |
Jan
(30) |
Feb
(38) |
Mar
(53) |
Apr
(6) |
May
(2) |
Jun
(5) |
Jul
(15) |
Aug
(15) |
Sep
(7) |
Oct
(18) |
Nov
(23) |
Dec
(6) |
| 2018 |
Jan
(39) |
Feb
(5) |
Mar
(34) |
Apr
(26) |
May
(27) |
Jun
(5) |
Jul
(12) |
Aug
(4) |
Sep
|
Oct
(4) |
Nov
(4) |
Dec
(4) |
| 2019 |
Jan
(7) |
Feb
(10) |
Mar
(21) |
Apr
(26) |
May
(4) |
Jun
(5) |
Jul
(11) |
Aug
(6) |
Sep
(7) |
Oct
(13) |
Nov
(3) |
Dec
(17) |
| 2020 |
Jan
|
Feb
(3) |
Mar
(3) |
Apr
(5) |
May
(2) |
Jun
(5) |
Jul
|
Aug
|
Sep
(6) |
Oct
(7) |
Nov
(2) |
Dec
(7) |
| 2021 |
Jan
(9) |
Feb
(10) |
Mar
(18) |
Apr
(1) |
May
(3) |
Jun
|
Jul
(16) |
Aug
(2) |
Sep
|
Oct
|
Nov
(9) |
Dec
(2) |
| 2022 |
Jan
(3) |
Feb
|
Mar
(9) |
Apr
(8) |
May
(5) |
Jun
(6) |
Jul
(1) |
Aug
|
Sep
(1) |
Oct
|
Nov
(7) |
Dec
(2) |
| 2023 |
Jan
(7) |
Feb
(2) |
Mar
(6) |
Apr
|
May
(4) |
Jun
(2) |
Jul
(4) |
Aug
(3) |
Sep
(4) |
Oct
(2) |
Nov
(4) |
Dec
(10) |
| 2024 |
Jan
(4) |
Feb
(2) |
Mar
(1) |
Apr
|
May
(1) |
Jun
(1) |
Jul
|
Aug
(1) |
Sep
|
Oct
|
Nov
|
Dec
|
| S | M | T | W | T | F | S |
|---|---|---|---|---|---|---|
|
1
|
2
(2) |
3
(3) |
4
(3) |
5
|
6
|
7
|
|
8
|
9
|
10
|
11
|
12
|
13
(5) |
14
|
|
15
|
16
(2) |
17
|
18
|
19
(2) |
20
|
21
|
|
22
|
23
|
24
|
25
|
26
|
27
|
28
|
|
29
|
30
|
31
|
|
|
|
|
|
From: Alexander D. <ale...@gm...> - 2019-12-19 21:43:18
|
On Thu, Dec 19, 2019 at 10:35 AM adam radford <ara...@gm...> wrote: > > e1000-devel list, > > Problem: > > With kernel 4.19.29 and igb 5.4.0-k on Intel E5-2618Lv4 and E5-2648Lv4 servers: <snip> > Using the default 8 MSI-X vectors (combined TX/RX queues), after 2 > days to 30 days of runtime we suddenly see unhandled interrupts, back > to back TX timeouts / Reset Adapter sequences, or both at the same > time on igb, i.e.: > > 2019-12-01T17:57:33.797+0000 controller- user.emerg kernel: > [919635.664612] do_IRQ: 13.53 No irq handler for vector > 2019-12-01T17:57:35.845+0000 controller- user.emerg kernel: > [919637.712587] do_IRQ: 13.53 No irq handler for vector > 2019-12-01T17:57:37.829+0000 controller- user.emerg kernel: > [919639.696569] do_IRQ: 13.53 No irq handler for vector > 2019-12-01T17:57:39.237+0000 controller- user.err kernel: > [919641.103011] igb 0000:01:00.1 eth1: Reset adapter > 2019-12-01T17:57:39.237+0000 controller- user.emerg kernel: > [919641.103021] do_IRQ: 13.53 No irq handler for vector > 2019-12-01T17:57:39.266+0000 controller- user.err kernel: > [919641.132142] igb 0000:01:00.2 eth2: Reset adapter > 2019-12-01T17:57:39.268+0000 controller- user.info kernel: > [919641.139341] igb 0000:01:00.2 eth2: igb: eth2 NIC Link is Up 1000 > Mbps Full Duplex, Flow Control: RX > 2019-12-01T17:57:39.346+0000 controller- user.emerg kernel: > [919641.212614] do_IRQ: 13.53 No irq handler for vector > 2019-12-01T17:57:39.363+0000 controller- user.info kernel: > [919641.234249] igb 0000:01:00.2 eth2: igb: eth2 NIC Link is Down > 2019-12-01T17:57:41.796+0000 controller- user.emerg kernel: > [919643.663562] do_IRQ: 13.53 No irq handler for vector > 2019-12-01T17:57:42.790+0000 controller- user.info kernel: > [919644.661800] igb 0000:01:00.2 eth2: igb: eth2 NIC Link is Up 1000 > Mbps Full Duplex, Flow Control: RX > 2019-12-01T17:57:43.841+0000 controller- user.info kernel: > [919645.712796] igb 0000:01:00.1 eth1: igb: eth1 NIC Link is Up 1000 > Mbps Full Duplex, Flow Control: RX > 2019-12-01T17:57:43.849+0000 controller- user.emerg kernel: > [919645.714693] do_IRQ: 13.53 No irq handler for vector > 2019-12-01T17:57:45.831+0000 controller- user.emerg kernel: > [919647.698208] do_IRQ: 13.53 No irq handler for vector So all of this points to an issue with the IRQ handler somehow not being associated with the vector on this CPU. I suspect not much has changed in the driver code itself. My thought is that something is causing the CPU to lose track of the IRQ handler for the vector. So the messages above are pointing to IRQ vector 53 on CPU 13. It makes me wonder if irqbalance is causing the interrupts to be bounced between CPUs and in the process of that migration it is somehow losing track of the IRQ handler. One thing you might try doing to determine if something like this is an issue is to disable irqbalance and then manually assign the IRQ affinity of the interrupts for the NIC ports. As long as that is working you wouldn't need to worry about it moving the interrupts to CPUs and if that is the issue then the interface is unlikely to lose the interrupt handler for its' vectors. Hope that helps. - Alex |
|
From: adam r. <ara...@gm...> - 2019-12-19 18:12:40
|
e1000-devel list,
Problem:
With kernel 4.19.29 and igb 5.4.0-k on Intel E5-2618Lv4 and E5-2648Lv4 servers:
# ethtool -i eth1
driver: igb
version: 5.4.0-k
firmware-version: 1.63, 0x800009fa
expansion-rom-version:
bus-info: 0000:01:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
[ 9.417249] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.4.0-k
[ 9.424543] igb: Copyright (c) 2007-2014 Intel Corporation.
[ 9.487321] igb 0000:01:00.0: added PHC on eth0
[ 9.492188] igb 0000:01:00.0: Intel(R) Gigabit Ethernet Network Connection
[ 9.499396] igb 0000:01:00.0: eth0: (PCIe:5.0Gb/s:Width x4) 00:50:cc:1d:bd:c6
[ 9.506939] igb 0000:01:00.0: eth0: PBA No: 106300-000
[ 9.512405] igb 0000:01:00.0: Using MSI-X interrupts. 8 rx
queue(s), 8 tx queue(s)
[ 9.597066] igb 0000:01:00.1: added PHC on eth1
[ 9.601934] igb 0000:01:00.1: Intel(R) Gigabit Ethernet Network Connection
[ 9.609148] igb 0000:01:00.1: eth1: (PCIe:5.0Gb/s:Width x4) 00:50:cc:1d:bd:c7
[ 9.616694] igb 0000:01:00.1: eth1: PBA No: 106300-000
[ 9.622162] igb 0000:01:00.1: Using MSI-X interrupts. 8 rx
queue(s), 8 tx queue(s)
[ 9.684412] igb 0000:01:00.2: added PHC on eth2
[ 9.689504] igb 0000:01:00.2: Intel(R) Gigabit Ethernet Network Connection
[ 9.696713] igb 0000:01:00.2: eth2: (PCIe:5.0Gb/s:Width x4) 00:50:cc:1d:bd:c8
[ 9.704256] igb 0000:01:00.2: eth2: PBA No: 106300-000
[ 9.709720] igb 0000:01:00.2: Using MSI-X interrupts. 8 rx
queue(s), 8 tx queue(s)
PCI config space from i.e. eth0:
01:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network
Connection (rev 01)
Subsystem: Seagate Technology PLC Device 8005
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 128 bytes
Interrupt: pin A routed to IRQ 16
NUMA node: 0
Region 0: Memory at 97840000 (32-bit, non-prefetchable) [size=128K]
Region 2: I/O ports at 2040 [size=32]
Region 3: Memory at 97868000 (32-bit, non-prefetchable) [size=16K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [70] MSI-X: Enable+ Count=10 Masked-
Vector table: BAR=3 offset=00000000
PBA: BAR=3 offset=00002000
Capabilities: [a0] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s
<512ns, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+
FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+
Unsupported+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+
AuxPwr+ TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1,
Exit Latency L0s <4us, L1 <32us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+,
LTR+, OBFF Not Supported
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms,
TimeoutDis-, LTR+, OBFF Disabled
AtomicOpsCtl: ReqEn-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt-
UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, ECRCGenCap+
ECRCGenEn- ECRCChkCap+ ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
Capabilities: [140 v1] Device Serial Number 00-50-cc-ff-ff-1d-bd-c6
Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 1
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-
IOVSta: Migration-
Initial VFs: 8, Total VFs: 8, Number of VFs: 0,
Function Dependency Link: 00
VF offset: 384, stride: 4, Device ID: 1520
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 0000183fffea0000 (64-bit, prefetchable)
Region 3: Memory at 0000183fffe80000 (64-bit, prefetchable)
VF Migration: offset: 00000000, BIR: 0
Capabilities: [1a0 v1] Transaction Processing Hints
Device specific mode supported
Steering table in TPH capability structure
Capabilities: [1c0 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Capabilities: [1d0 v1] Access Control Services
ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir-
UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir-
UpstreamFwd- EgressCtrl- DirectTrans-
Kernel driver in use: igb
Using the default 8 MSI-X vectors (combined TX/RX queues), after 2
days to 30 days of runtime we suddenly see unhandled interrupts, back
to back TX timeouts / Reset Adapter sequences, or both at the same
time on igb, i.e.:
2019-12-01T17:57:33.797+0000 controller- user.emerg kernel:
[919635.664612] do_IRQ: 13.53 No irq handler for vector
2019-12-01T17:57:35.845+0000 controller- user.emerg kernel:
[919637.712587] do_IRQ: 13.53 No irq handler for vector
2019-12-01T17:57:37.829+0000 controller- user.emerg kernel:
[919639.696569] do_IRQ: 13.53 No irq handler for vector
2019-12-01T17:57:39.237+0000 controller- user.err kernel:
[919641.103011] igb 0000:01:00.1 eth1: Reset adapter
2019-12-01T17:57:39.237+0000 controller- user.emerg kernel:
[919641.103021] do_IRQ: 13.53 No irq handler for vector
2019-12-01T17:57:39.266+0000 controller- user.err kernel:
[919641.132142] igb 0000:01:00.2 eth2: Reset adapter
2019-12-01T17:57:39.268+0000 controller- user.info kernel:
[919641.139341] igb 0000:01:00.2 eth2: igb: eth2 NIC Link is Up 1000
Mbps Full Duplex, Flow Control: RX
2019-12-01T17:57:39.346+0000 controller- user.emerg kernel:
[919641.212614] do_IRQ: 13.53 No irq handler for vector
2019-12-01T17:57:39.363+0000 controller- user.info kernel:
[919641.234249] igb 0000:01:00.2 eth2: igb: eth2 NIC Link is Down
2019-12-01T17:57:41.796+0000 controller- user.emerg kernel:
[919643.663562] do_IRQ: 13.53 No irq handler for vector
2019-12-01T17:57:42.790+0000 controller- user.info kernel:
[919644.661800] igb 0000:01:00.2 eth2: igb: eth2 NIC Link is Up 1000
Mbps Full Duplex, Flow Control: RX
2019-12-01T17:57:43.841+0000 controller- user.info kernel:
[919645.712796] igb 0000:01:00.1 eth1: igb: eth1 NIC Link is Up 1000
Mbps Full Duplex, Flow Control: RX
2019-12-01T17:57:43.849+0000 controller- user.emerg kernel:
[919645.714693] do_IRQ: 13.53 No irq handler for vector
2019-12-01T17:57:45.831+0000 controller- user.emerg kernel:
[919647.698208] do_IRQ: 13.53 No irq handler for vector
Additional observations:
1. The above unhandled MSI-X interrupt message above happens in most
cases every 2 seconds and it appears to be coming from
igb_watchdog_task(), i.e. from this commit:
commit 7a6ea550f2f7592742ac765e5a3b4b5d1461e0bd
Author: Alexander Duyck <ale...@in...>
Date: Tue Aug 26 04:25:03 2008 -0700
igb: force all queues to interrupt once every 2 seconds
2. We have found that when we ifconfig eth1 down / ifconfig eth1 up,
the unhandled interrupt problem stops but the back to back TX timeout
/ Adapter reset sequences do not stop.
3. We have found that rmmod igb / insmod igb makes both problems go
away temporarily.
4. We have found that for both the unhandled interrupt issue and the
tx timeout issue, if we look at the TX/RX interrupt MSI-X irq counts
for 1 igb interace in /proc/interrupts, one of them has stopped
incrementing, and never increments again (not every 2 seconds anymore,
and not even once after several days) i.e.:
# date ; cat /proc/interrupts | grep eth1 | grep "51:"
Thu Dec 19 17:41:04 GMT 2019
51: 0 19992 22218 40124 32935 28772
35734 48672 24207 49188 3171172 65812
36653 17586 46483 23624 40544 39936
37085 11671 PCI-MSI 526341-edge eth1-TxRx-4
# date ; cat /proc/interrupts | grep eth1 | grep "51:"
Thu Dec 19 17:41:12 GMT 2019
51: 0 19992 22218 40124 32935 28772
35734 48672 24207 49188 3171172 65812
36653 17586 46483 23624 40544 39936
37085 11671 PCI-MSI 526341-edge eth1-TxRx-4
The other TX/Rx vector counts keep incrementing on the igb interfaces.
5. We do not see either problem with our older igb kernel/driver i.e.
kernel 3.10.0-327.18.2.el7 (Centos) using igb driver 5.2.15-k.
6. We have several other devices using muti vector MSI-X on this
4.19.29 kernel/hardware without any issues.
7. We think PCIe ASPM has been disabled by the kernel at boot time:
[ 2.981858] ACPI FADT declares the system doesn't support PCIe
ASPM, so disable it
8. We have not performed any runtime PM operations:
# cat /sys/class/net/eth1/power/runtime_suspended_time
0
What could be causing these issues?
Thank you for your reply!
-Adam Radford
|
|
From: Nunley, N. D <nic...@in...> - 2019-12-16 23:35:03
|
I went through the configuration in i40e_add_vsi() and it looks correct to me, assuming it's updated to use VMDq2 VSIs. If you're observing that VNICs are receiving traffic with VMDq2 VSIs (albeit very slowly) then I wonder whether the issue is even in the HW VSI context configuration. If you want to double check the correctness of the illumos VSI configuration yourself then you should look at the Linux i40e driver as Alex suggested, though, since that's the de facto guide for proven hardware configurations. For an additional debugging datapoint you might try manually swapping around the ring-group-index-to-VSI mappings (via i40e_fill_rx_group(): infop->mgi_driver and I40E_DEF_VSI_SEID) and check whether the "good" behavior you see sticks around on the default HW VSI or the default group. I'm not sure if it works this way on illumos, but on later versions of Solaris I remember seeing a quirk where if you have multiple ring groups enabled then the GLD layer will use nonzero group indices even for the default traffic, so you may already be using the non-default VSIs in the working case. Otherwise, you could also try snooping the traffic on the VNIC to see if you notice anything noteworthy about the few packets you do receive. You might also want to verify that interrupts and/or polling are working as expected for all groups. I think the first step would be to try and see if you can isolate the configuration differences between the working and nonworking ring groups/VSIs, though, to see if that could explain any of the behavior. Unfortunately, there's not a lot we can directly help with here since illumos is not a supported OS. None of us have any experience with the illumos driver and there are simply too many ways it could be misconfiguring itself and/or the hardware to make any assumptions about which parts might be working. As others have mentioned, the only official advice we can give is to contact the illumos maintainers since they're the ones providing support for the driver. Thanks, Nick > -----Original Message----- > From: Youzhong Yang <yy...@ma...> > Sent: Monday, December 16, 2019 6:06 AM > To: Nunley, Nicholas D <nic...@in...>; Alexander Duyck > <ale...@gm...> > Cc: e10...@li... > Subject: RE: [E1000-devel] X710: very slow rx speed (a few MiB/second) > when using non-default VSI. > > Thanks Alex and Nick. It is nice that finally we come to the technical aspect of > it, and I really appreciate your input. > > Changing type PF to type VMDQ2 was the first thing I have tried when > debugging this issue. It didn't help. > > Yesterday I installed Solaris 11.4, create VNICs by setting "ring- > group=exclusive", ran my reproduction programs, there was no performance > issue. > > Obviously Intel knows how to do the proper initialization/configuration of > VMDq2 VSIs. I am wondering what that is. If you could shed some light on it, > that would be really helpful. > > Thanks, > > -Youzhong > > -----Original Message----- > From: Nunley, Nicholas D <nic...@in...> > Sent: Friday, December 13, 2019 4:10 PM > To: Alexander Duyck <ale...@gm...>; Youzhong Yang > <yy...@ma...> > Cc: e10...@li... > Subject: RE: [E1000-devel] X710: very slow rx speed (a few MiB/second) > when using non-default VSI. > > I have experience with the Solaris driver, but not Illumos, so I won't be able > to assist in any debug since the implementations differ too much, but I am > familiar with the interfaces and can say that in the official i40e Solaris driver > we've always used VMDq2 VSIs for the non-default ring groups, not PF VSIs. > > I would try changing the ring group configuration in i40e_add_vsi() to use > VMDq2 VSIs and see if that makes a difference. There shouldn't be any > reason Illumos would specifically need PF VSIs here. > > http://src.illumos.org/source/xref/illumos- > gate/usr/src/uts/common/io/i40e/i40e_main.c#2126 > change: > ctx.flags = I40E_AQ_VSI_TYPE_PF; > to: > ctx.flags = I40E_AQ_VSI_TYPE_VMDQ2; > > Thanks, > Nick > > > -----Original Message----- > > From: Alexander Duyck <ale...@gm...> > > Sent: Friday, December 13, 2019 11:27 AM > > To: Youzhong Yang <yy...@ma...> > > Cc: e10...@li... > > Subject: Re: [E1000-devel] X710: very slow rx speed (a few MiB/second) > > when using non-default VSI. > > > > Before you start complaining about what the driver is or isn't doing > > you should probably understand that both the Linux and FreeBSD drivers > > do have to create a PF VSI as that is the default VSI type if I am not > > mistaken. Do you know what the goal is in creating multiple PF VSIs? I > > don't see the reason for doing that versus allocating VMDq2 VSIs? > > > > If you take a look at the i40e Linux driver it supports a feature > > called ADq which can be used to create multiple VMDq2 VSIs. > > > > Nobody on this list likely has any experience with Illuminos. I would > > recommend getting in touch with the maintainer for the Illuminos i40e > > driver and working with them as they would know how that driver is > > supposed to work and why the functionality is implemented the way it is. > > > > Thanks. > > > > - Alex > > > > On Fri, Dec 13, 2019 at 11:04 AM Youzhong Yang <yy...@ma...> > > wrote: > > > > > > As I said, a supported OS (Linux and FreeBSD?) does not create the > > > kind of > > VSI (PF VSI) in its i40e driver, are you asking me to test nothing on > > your supported OS? > > > > > > "Intel(r) Ethernet Controller X710/ XXV710/XL710 Datasheet" claims > > > it > > supports PF VSIs, so why would you say it is not supported? Because > > your customer is not using Linux and FreeBSD? > > > > > > It does not make any sense. Sorry. > > > > > > -----Original Message----- > > > From: Fujinaka, Todd <tod...@in...> > > > Sent: Friday, December 13, 2019 1:45 PM > > > To: Youzhong Yang <yy...@ma...>; > > > e10...@li... > > > Subject: RE: X710: very slow rx speed (a few MiB/second) when using > > > non- > > default VSI. > > > > > > You can try official channels but you're just going to get an > > > official "Talk to > > Illuminos" from me. > > > > > > As I said, you can try installing a supported OS to see if it's a > > > hardware issue, > > but we have no support for Illuminos. > > > > > > Todd Fujinaka > > > Software Application Engineer > > > Datacenter Engineering Group > > > Intel Corporation > > > tod...@in... > > > > > > -----Original Message----- > > > From: Youzhong Yang <yy...@ma...> > > > Sent: Friday, December 13, 2019 10:11 AM > > > To: Fujinaka, Todd <tod...@in...>; > > > e10...@li... > > > Subject: RE: X710: very slow rx speed (a few MiB/second) when using > > > non- > > default VSI. > > > > > > This mailing list seems quite quiet. Any hope my issue could get > > > some > > attention? If it is a thing of 'Only Linux and FreeBSD are supported', > > is there any other communication channel through which this issue can be > escalated? > > Thanks a lot for your understanding. > > > > > > -----Original Message----- > > > From: Youzhong Yang <yy...@ma...> > > > Sent: Tuesday, December 3, 2019 12:21 PM > > > To: Fujinaka, Todd <tod...@in...>; > > > e10...@li... > > > Subject: RE: X710: very slow rx speed (a few MiB/second) when using > > > non- > > default VSI. > > > > > > Hello Todd, > > > > > > Thank you very much for replying so fast. I am sorry I missed your > > > e-mail as > > it was blocked by our company e-mail filters. > > > > > > I have contacted illumos dev twice but unfortunately there was no > > > reply, it > > seems the i40e driver developer is no longer active. I am trying to > > figure out how I can fix the issue by myself, I know how to do driver > > development but this particular issue seems to be very unique, and I > > suspect either something is missing when creating the VSIs, or there > > might be a bug (possibly in the Ethernet card firmware) in the way non- > default PF VSI is created. > > > > > > I have researched the Linux and FreeBSD i40e driver code, none of > > > them > > does the same kind of thing our illumos driver does. For each i40e > > instance, we create 31 PF VSIs in addition to the default one > > initiated by the card firmware. When receiving data through the RX > > queues of the default VSI, the performance is very good, but when > > receiving data through the RX queues of the non-default VSI, the > performance can be very poor. > > > > > > The "Intel(r) Ethernet Controller X710/ XXV710/XL710 Datasheet" > > > suggests > > there's nothing wrong by having 32 PF VSIs. Please refer to its page 20: > > > > > > "VSI Support > > > The X710/XXV710/XL710 supports a total of 384 VSIs. VSI assignment > > > is > > flexible, but the choice to support 384 VSIs is motivated by the > > following usage example: > > > * 256 VSIs for VFs or VMDq2 > > > * 32 VSIs for PFs" > > > > > > We purchased a bunch of XL710 cards, hoping it can deliver better > > performance than the ixgbe cards. Sadly we were hit by this > > performance issue, and I've tried troubleshooting/debugging for more > > than 2 months, but so far no luck. > > > > > > Your kind assistance would be highly appreciated. > > > > > > Sincerely, > > > > > > -Youzhong Yang > > > The MathWorks Inc. > > > > > > -----Original Message----- > > > From: Fujinaka, Todd <tod...@in...> > > > Sent: Monday, December 2, 2019 4:39 PM > > > To: Youzhong Yang <yy...@ma...>; > > > e10...@li... > > > Subject: RE: X710: very slow rx speed (a few MiB/second) when using > > > non- > > default VSI. > > > > > > I think you're going to have to ask Illumos about this. Can you > > > reproduce > > this on a supported OS such as Red Hat Linux or FreeBSD? > > > > > > Todd Fujinaka > > > Software Application Engineer > > > Datacenter Engineering Group > > > Intel Corporation > > > tod...@in... > > > > > > -----Original Message----- > > > From: Youzhong Yang <yy...@ma...> > > > Sent: Monday, December 2, 2019 12:19 PM > > > To: e10...@li... > > > Subject: [E1000-devel] X710: very slow rx speed (a few MiB/second) > > > when > > using non-default VSI. > > > > > > Hello everyone, > > > > > > I submitted a help request on Intel website and was redirected to > > > this > > mailing list. The request can be found here: > > > > > > X710: very slow rx speed (a few MiB/second) when using non-default > VSI. > > > > > > > > > I experienced very strange performance issue when sending data from > > > a > > client to a server with X710 card. Details about the investigation can > > be found > > here: > > > > > > https://github.com/youzhongyang/i40e_investigation > > > ub.com > > > > > > > > > > > > The server runs illumos (derived from open solaris), its i40e driver > > > can be > > found here: > > > > > > http://src.illumos.org/source/xref/illumos- > gate/usr/src/uts/common/io/. > > > illumos.org > > > i40e/ > > > > > > > > > > > > So for each i40e instance, we create 32 VSIs (including the default > > > one > > created by the firmware), each VSI is assigned 8 trq pairs. When a > > VIF/VNIC is created, the OS assigns a VSI to it, data receiving > > through the VNIC's IP will be done by the relevant VSI's 8 RX rings. > > However, data sending will always go through the default VSI's 8 TX rings. > > > > > > > > > > > > The function for creating non-default VSI is here: > > > > > > http://src.illumos.org/source/xref/illumos- > gate/usr/src/uts/common/io/. > > > illumos.org > > > i40e/i40e_main.c#2111 > > > > > > > > > > > > I've upgraded the firmware version to the latest, still the same > > > issue. Is this > > a known issue? or is there anything missing when setting up the > > non-default VSI? > > > > > > > > > > > > Your help/advice would be highly appreciated. > > > > > > Sincerely, > > > -Youzhong Yang > > > > > > > > > _______________________________________________ > > > E1000-devel mailing list > > > E10...@li... > > > https://lists.sourceforge.net/lists/listinfo/e1000-devel > > > s.sourceforge.net To learn more about Intel Ethernet, visit > > > https://forums.intel.com/s/topic/0TO0P00000018NbWAI/intel-ethernet > > > ms.intel.com > > > > > > > > > > > > _______________________________________________ > > > E1000-devel mailing list > > > E10...@li... > > > https://lists.sourceforge.net/lists/listinfo/e1000-devel > > > s.sourceforge.net To learn more about Intel Ethernet, visit > > > https://forums.intel.com/s/topic/0TO0P00000018NbWAI/intel-ethernet > > > ms.intel.com > > > > > > _______________________________________________ > > E1000-devel mailing list > > E10...@li... > > https://lists.sourceforge.net/lists/listinfo/e1000-devel. > > sourceforge.net To learn more about Intel Ethernet, visit > > https://forums.intel.com/s/topic/0TO0P00000018NbWAI/intel-ethernet |
|
From: Youzhong Y. <yy...@ma...> - 2019-12-16 14:06:13
|
Thanks Alex and Nick. It is nice that finally we come to the technical aspect of it, and I really appreciate your input. Changing type PF to type VMDQ2 was the first thing I have tried when debugging this issue. It didn't help. Yesterday I installed Solaris 11.4, create VNICs by setting "ring-group=exclusive", ran my reproduction programs, there was no performance issue. Obviously Intel knows how to do the proper initialization/configuration of VMDq2 VSIs. I am wondering what that is. If you could shed some light on it, that would be really helpful. Thanks, -Youzhong -----Original Message----- From: Nunley, Nicholas D <nic...@in...> Sent: Friday, December 13, 2019 4:10 PM To: Alexander Duyck <ale...@gm...>; Youzhong Yang <yy...@ma...> Cc: e10...@li... Subject: RE: [E1000-devel] X710: very slow rx speed (a few MiB/second) when using non-default VSI. I have experience with the Solaris driver, but not Illumos, so I won't be able to assist in any debug since the implementations differ too much, but I am familiar with the interfaces and can say that in the official i40e Solaris driver we've always used VMDq2 VSIs for the non-default ring groups, not PF VSIs. I would try changing the ring group configuration in i40e_add_vsi() to use VMDq2 VSIs and see if that makes a difference. There shouldn't be any reason Illumos would specifically need PF VSIs here. http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/io/i40e/i40e_main.c#2126 change: ctx.flags = I40E_AQ_VSI_TYPE_PF; to: ctx.flags = I40E_AQ_VSI_TYPE_VMDQ2; Thanks, Nick > -----Original Message----- > From: Alexander Duyck <ale...@gm...> > Sent: Friday, December 13, 2019 11:27 AM > To: Youzhong Yang <yy...@ma...> > Cc: e10...@li... > Subject: Re: [E1000-devel] X710: very slow rx speed (a few MiB/second) > when using non-default VSI. > > Before you start complaining about what the driver is or isn't doing > you should probably understand that both the Linux and FreeBSD drivers > do have to create a PF VSI as that is the default VSI type if I am not > mistaken. Do you know what the goal is in creating multiple PF VSIs? I > don't see the reason for doing that versus allocating VMDq2 VSIs? > > If you take a look at the i40e Linux driver it supports a feature > called ADq which can be used to create multiple VMDq2 VSIs. > > Nobody on this list likely has any experience with Illuminos. I would > recommend getting in touch with the maintainer for the Illuminos i40e > driver and working with them as they would know how that driver is > supposed to work and why the functionality is implemented the way it is. > > Thanks. > > - Alex > > On Fri, Dec 13, 2019 at 11:04 AM Youzhong Yang <yy...@ma...> > wrote: > > > > As I said, a supported OS (Linux and FreeBSD?) does not create the > > kind of > VSI (PF VSI) in its i40e driver, are you asking me to test nothing on > your supported OS? > > > > "Intel(r) Ethernet Controller X710/ XXV710/XL710 Datasheet" claims > > it > supports PF VSIs, so why would you say it is not supported? Because > your customer is not using Linux and FreeBSD? > > > > It does not make any sense. Sorry. > > > > -----Original Message----- > > From: Fujinaka, Todd <tod...@in...> > > Sent: Friday, December 13, 2019 1:45 PM > > To: Youzhong Yang <yy...@ma...>; > > e10...@li... > > Subject: RE: X710: very slow rx speed (a few MiB/second) when using > > non- > default VSI. > > > > You can try official channels but you're just going to get an > > official "Talk to > Illuminos" from me. > > > > As I said, you can try installing a supported OS to see if it's a > > hardware issue, > but we have no support for Illuminos. > > > > Todd Fujinaka > > Software Application Engineer > > Datacenter Engineering Group > > Intel Corporation > > tod...@in... > > > > -----Original Message----- > > From: Youzhong Yang <yy...@ma...> > > Sent: Friday, December 13, 2019 10:11 AM > > To: Fujinaka, Todd <tod...@in...>; > > e10...@li... > > Subject: RE: X710: very slow rx speed (a few MiB/second) when using > > non- > default VSI. > > > > This mailing list seems quite quiet. Any hope my issue could get > > some > attention? If it is a thing of 'Only Linux and FreeBSD are supported', > is there any other communication channel through which this issue can be escalated? > Thanks a lot for your understanding. > > > > -----Original Message----- > > From: Youzhong Yang <yy...@ma...> > > Sent: Tuesday, December 3, 2019 12:21 PM > > To: Fujinaka, Todd <tod...@in...>; > > e10...@li... > > Subject: RE: X710: very slow rx speed (a few MiB/second) when using > > non- > default VSI. > > > > Hello Todd, > > > > Thank you very much for replying so fast. I am sorry I missed your > > e-mail as > it was blocked by our company e-mail filters. > > > > I have contacted illumos dev twice but unfortunately there was no > > reply, it > seems the i40e driver developer is no longer active. I am trying to > figure out how I can fix the issue by myself, I know how to do driver > development but this particular issue seems to be very unique, and I > suspect either something is missing when creating the VSIs, or there > might be a bug (possibly in the Ethernet card firmware) in the way non-default PF VSI is created. > > > > I have researched the Linux and FreeBSD i40e driver code, none of > > them > does the same kind of thing our illumos driver does. For each i40e > instance, we create 31 PF VSIs in addition to the default one > initiated by the card firmware. When receiving data through the RX > queues of the default VSI, the performance is very good, but when > receiving data through the RX queues of the non-default VSI, the performance can be very poor. > > > > The "Intel(r) Ethernet Controller X710/ XXV710/XL710 Datasheet" > > suggests > there's nothing wrong by having 32 PF VSIs. Please refer to its page 20: > > > > "VSI Support > > The X710/XXV710/XL710 supports a total of 384 VSIs. VSI assignment > > is > flexible, but the choice to support 384 VSIs is motivated by the > following usage example: > > * 256 VSIs for VFs or VMDq2 > > * 32 VSIs for PFs" > > > > We purchased a bunch of XL710 cards, hoping it can deliver better > performance than the ixgbe cards. Sadly we were hit by this > performance issue, and I've tried troubleshooting/debugging for more > than 2 months, but so far no luck. > > > > Your kind assistance would be highly appreciated. > > > > Sincerely, > > > > -Youzhong Yang > > The MathWorks Inc. > > > > -----Original Message----- > > From: Fujinaka, Todd <tod...@in...> > > Sent: Monday, December 2, 2019 4:39 PM > > To: Youzhong Yang <yy...@ma...>; > > e10...@li... > > Subject: RE: X710: very slow rx speed (a few MiB/second) when using > > non- > default VSI. > > > > I think you're going to have to ask Illumos about this. Can you > > reproduce > this on a supported OS such as Red Hat Linux or FreeBSD? > > > > Todd Fujinaka > > Software Application Engineer > > Datacenter Engineering Group > > Intel Corporation > > tod...@in... > > > > -----Original Message----- > > From: Youzhong Yang <yy...@ma...> > > Sent: Monday, December 2, 2019 12:19 PM > > To: e10...@li... > > Subject: [E1000-devel] X710: very slow rx speed (a few MiB/second) > > when > using non-default VSI. > > > > Hello everyone, > > > > I submitted a help request on Intel website and was redirected to > > this > mailing list. The request can be found here: > > > > X710: very slow rx speed (a few MiB/second) when using non-default VSI. > > > > > > I experienced very strange performance issue when sending data from > > a > client to a server with X710 card. Details about the investigation can > be found > here: > > > > https://github.com/youzhongyang/i40e_investigation > > ub.com > > > > > > > > The server runs illumos (derived from open solaris), its i40e driver > > can be > found here: > > > > http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/io/. > > illumos.org > > i40e/ > > > > > > > > So for each i40e instance, we create 32 VSIs (including the default > > one > created by the firmware), each VSI is assigned 8 trq pairs. When a > VIF/VNIC is created, the OS assigns a VSI to it, data receiving > through the VNIC's IP will be done by the relevant VSI's 8 RX rings. > However, data sending will always go through the default VSI's 8 TX rings. > > > > > > > > The function for creating non-default VSI is here: > > > > http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/io/. > > illumos.org > > i40e/i40e_main.c#2111 > > > > > > > > I've upgraded the firmware version to the latest, still the same > > issue. Is this > a known issue? or is there anything missing when setting up the > non-default VSI? > > > > > > > > Your help/advice would be highly appreciated. > > > > Sincerely, > > -Youzhong Yang > > > > > > _______________________________________________ > > E1000-devel mailing list > > E10...@li... > > https://lists.sourceforge.net/lists/listinfo/e1000-devel > > s.sourceforge.net To learn more about Intel Ethernet, visit > > https://forums.intel.com/s/topic/0TO0P00000018NbWAI/intel-ethernet > > ms.intel.com > > > > > > > > _______________________________________________ > > E1000-devel mailing list > > E10...@li... > > https://lists.sourceforge.net/lists/listinfo/e1000-devel > > s.sourceforge.net To learn more about Intel Ethernet, visit > > https://forums.intel.com/s/topic/0TO0P00000018NbWAI/intel-ethernet > > ms.intel.com > > > _______________________________________________ > E1000-devel mailing list > E10...@li... > https://lists.sourceforge.net/lists/listinfo/e1000-devel. > sourceforge.net To learn more about Intel Ethernet, visit > https://forums.intel.com/s/topic/0TO0P00000018NbWAI/intel-ethernet |
|
From: Nunley, N. D <nic...@in...> - 2019-12-13 21:10:06
|
I have experience with the Solaris driver, but not Illumos, so I won't be able to assist in any debug since the implementations differ too much, but I am familiar with the interfaces and can say that in the official i40e Solaris driver we've always used VMDq2 VSIs for the non-default ring groups, not PF VSIs. I would try changing the ring group configuration in i40e_add_vsi() to use VMDq2 VSIs and see if that makes a difference. There shouldn't be any reason Illumos would specifically need PF VSIs here. http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/io/i40e/i40e_main.c#2126 change: ctx.flags = I40E_AQ_VSI_TYPE_PF; to: ctx.flags = I40E_AQ_VSI_TYPE_VMDQ2; Thanks, Nick > -----Original Message----- > From: Alexander Duyck <ale...@gm...> > Sent: Friday, December 13, 2019 11:27 AM > To: Youzhong Yang <yy...@ma...> > Cc: e10...@li... > Subject: Re: [E1000-devel] X710: very slow rx speed (a few MiB/second) > when using non-default VSI. > > Before you start complaining about what the driver is or isn't doing you > should probably understand that both the Linux and FreeBSD drivers do have > to create a PF VSI as that is the default VSI type if I am not mistaken. Do you > know what the goal is in creating multiple PF VSIs? I don't see the reason for > doing that versus allocating VMDq2 VSIs? > > If you take a look at the i40e Linux driver it supports a feature called ADq > which can be used to create multiple VMDq2 VSIs. > > Nobody on this list likely has any experience with Illuminos. I would > recommend getting in touch with the maintainer for the Illuminos i40e driver > and working with them as they would know how that driver is supposed to > work and why the functionality is implemented the way it is. > > Thanks. > > - Alex > > On Fri, Dec 13, 2019 at 11:04 AM Youzhong Yang <yy...@ma...> > wrote: > > > > As I said, a supported OS (Linux and FreeBSD?) does not create the kind of > VSI (PF VSI) in its i40e driver, are you asking me to test nothing on your > supported OS? > > > > "Intel(r) Ethernet Controller X710/ XXV710/XL710 Datasheet" claims it > supports PF VSIs, so why would you say it is not supported? Because your > customer is not using Linux and FreeBSD? > > > > It does not make any sense. Sorry. > > > > -----Original Message----- > > From: Fujinaka, Todd <tod...@in...> > > Sent: Friday, December 13, 2019 1:45 PM > > To: Youzhong Yang <yy...@ma...>; > > e10...@li... > > Subject: RE: X710: very slow rx speed (a few MiB/second) when using non- > default VSI. > > > > You can try official channels but you're just going to get an official "Talk to > Illuminos" from me. > > > > As I said, you can try installing a supported OS to see if it's a hardware issue, > but we have no support for Illuminos. > > > > Todd Fujinaka > > Software Application Engineer > > Datacenter Engineering Group > > Intel Corporation > > tod...@in... > > > > -----Original Message----- > > From: Youzhong Yang <yy...@ma...> > > Sent: Friday, December 13, 2019 10:11 AM > > To: Fujinaka, Todd <tod...@in...>; > > e10...@li... > > Subject: RE: X710: very slow rx speed (a few MiB/second) when using non- > default VSI. > > > > This mailing list seems quite quiet. Any hope my issue could get some > attention? If it is a thing of 'Only Linux and FreeBSD are supported', is there > any other communication channel through which this issue can be escalated? > Thanks a lot for your understanding. > > > > -----Original Message----- > > From: Youzhong Yang <yy...@ma...> > > Sent: Tuesday, December 3, 2019 12:21 PM > > To: Fujinaka, Todd <tod...@in...>; > > e10...@li... > > Subject: RE: X710: very slow rx speed (a few MiB/second) when using non- > default VSI. > > > > Hello Todd, > > > > Thank you very much for replying so fast. I am sorry I missed your e-mail as > it was blocked by our company e-mail filters. > > > > I have contacted illumos dev twice but unfortunately there was no reply, it > seems the i40e driver developer is no longer active. I am trying to figure out > how I can fix the issue by myself, I know how to do driver development but > this particular issue seems to be very unique, and I suspect either something > is missing when creating the VSIs, or there might be a bug (possibly in the > Ethernet card firmware) in the way non-default PF VSI is created. > > > > I have researched the Linux and FreeBSD i40e driver code, none of them > does the same kind of thing our illumos driver does. For each i40e instance, > we create 31 PF VSIs in addition to the default one initiated by the card > firmware. When receiving data through the RX queues of the default VSI, the > performance is very good, but when receiving data through the RX queues of > the non-default VSI, the performance can be very poor. > > > > The "Intel(r) Ethernet Controller X710/ XXV710/XL710 Datasheet" suggests > there's nothing wrong by having 32 PF VSIs. Please refer to its page 20: > > > > "VSI Support > > The X710/XXV710/XL710 supports a total of 384 VSIs. VSI assignment is > flexible, but the choice to support 384 VSIs is motivated by the following > usage example: > > * 256 VSIs for VFs or VMDq2 > > * 32 VSIs for PFs" > > > > We purchased a bunch of XL710 cards, hoping it can deliver better > performance than the ixgbe cards. Sadly we were hit by this performance > issue, and I've tried troubleshooting/debugging for more than 2 months, but > so far no luck. > > > > Your kind assistance would be highly appreciated. > > > > Sincerely, > > > > -Youzhong Yang > > The MathWorks Inc. > > > > -----Original Message----- > > From: Fujinaka, Todd <tod...@in...> > > Sent: Monday, December 2, 2019 4:39 PM > > To: Youzhong Yang <yy...@ma...>; > > e10...@li... > > Subject: RE: X710: very slow rx speed (a few MiB/second) when using non- > default VSI. > > > > I think you're going to have to ask Illumos about this. Can you reproduce > this on a supported OS such as Red Hat Linux or FreeBSD? > > > > Todd Fujinaka > > Software Application Engineer > > Datacenter Engineering Group > > Intel Corporation > > tod...@in... > > > > -----Original Message----- > > From: Youzhong Yang <yy...@ma...> > > Sent: Monday, December 2, 2019 12:19 PM > > To: e10...@li... > > Subject: [E1000-devel] X710: very slow rx speed (a few MiB/second) when > using non-default VSI. > > > > Hello everyone, > > > > I submitted a help request on Intel website and was redirected to this > mailing list. The request can be found here: > > > > X710: very slow rx speed (a few MiB/second) when using non-default VSI. > > > > > > I experienced very strange performance issue when sending data from a > client to a server with X710 card. Details about the investigation can be found > here: > > > > https://github.com/youzhongyang/i40e_investigation > > > > > > > > The server runs illumos (derived from open solaris), its i40e driver can be > found here: > > > > http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/io/ > > i40e/ > > > > > > > > So for each i40e instance, we create 32 VSIs (including the default one > created by the firmware), each VSI is assigned 8 trq pairs. When a VIF/VNIC is > created, the OS assigns a VSI to it, data receiving through the VNIC's IP will be > done by the relevant VSI's 8 RX rings. However, data sending will always go > through the default VSI's 8 TX rings. > > > > > > > > The function for creating non-default VSI is here: > > > > http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/io/ > > i40e/i40e_main.c#2111 > > > > > > > > I've upgraded the firmware version to the latest, still the same issue. Is this > a known issue? or is there anything missing when setting up the non-default > VSI? > > > > > > > > Your help/advice would be highly appreciated. > > > > Sincerely, > > -Youzhong Yang > > > > > > _______________________________________________ > > E1000-devel mailing list > > E10...@li... > > https://lists.sourceforge.net/lists/listinfo/e1000-devel > > To learn more about Intel Ethernet, visit > > https://forums.intel.com/s/topic/0TO0P00000018NbWAI/intel-ethernet > > > > > > > > _______________________________________________ > > E1000-devel mailing list > > E10...@li... > > https://lists.sourceforge.net/lists/listinfo/e1000-devel > > To learn more about Intel Ethernet, visit > > https://forums.intel.com/s/topic/0TO0P00000018NbWAI/intel-ethernet > > > _______________________________________________ > E1000-devel mailing list > E10...@li... > https://lists.sourceforge.net/lists/listinfo/e1000-devel > To learn more about Intel Ethernet, visit > https://forums.intel.com/s/topic/0TO0P00000018NbWAI/intel-ethernet |
|
From: Alexander D. <ale...@gm...> - 2019-12-13 19:26:49
|
Before you start complaining about what the driver is or isn't doing you should probably understand that both the Linux and FreeBSD drivers do have to create a PF VSI as that is the default VSI type if I am not mistaken. Do you know what the goal is in creating multiple PF VSIs? I don't see the reason for doing that versus allocating VMDq2 VSIs? If you take a look at the i40e Linux driver it supports a feature called ADq which can be used to create multiple VMDq2 VSIs. Nobody on this list likely has any experience with Illuminos. I would recommend getting in touch with the maintainer for the Illuminos i40e driver and working with them as they would know how that driver is supposed to work and why the functionality is implemented the way it is. Thanks. - Alex On Fri, Dec 13, 2019 at 11:04 AM Youzhong Yang <yy...@ma...> wrote: > > As I said, a supported OS (Linux and FreeBSD?) does not create the kind of VSI (PF VSI) in its i40e driver, are you asking me to test nothing on your supported OS? > > "Intel(r) Ethernet Controller X710/ XXV710/XL710 Datasheet" claims it supports PF VSIs, so why would you say it is not supported? Because your customer is not using Linux and FreeBSD? > > It does not make any sense. Sorry. > > -----Original Message----- > From: Fujinaka, Todd <tod...@in...> > Sent: Friday, December 13, 2019 1:45 PM > To: Youzhong Yang <yy...@ma...>; e10...@li... > Subject: RE: X710: very slow rx speed (a few MiB/second) when using non-default VSI. > > You can try official channels but you're just going to get an official "Talk to Illuminos" from me. > > As I said, you can try installing a supported OS to see if it's a hardware issue, but we have no support for Illuminos. > > Todd Fujinaka > Software Application Engineer > Datacenter Engineering Group > Intel Corporation > tod...@in... > > -----Original Message----- > From: Youzhong Yang <yy...@ma...> > Sent: Friday, December 13, 2019 10:11 AM > To: Fujinaka, Todd <tod...@in...>; e10...@li... > Subject: RE: X710: very slow rx speed (a few MiB/second) when using non-default VSI. > > This mailing list seems quite quiet. Any hope my issue could get some attention? If it is a thing of 'Only Linux and FreeBSD are supported', is there any other communication channel through which this issue can be escalated? Thanks a lot for your understanding. > > -----Original Message----- > From: Youzhong Yang <yy...@ma...> > Sent: Tuesday, December 3, 2019 12:21 PM > To: Fujinaka, Todd <tod...@in...>; e10...@li... > Subject: RE: X710: very slow rx speed (a few MiB/second) when using non-default VSI. > > Hello Todd, > > Thank you very much for replying so fast. I am sorry I missed your e-mail as it was blocked by our company e-mail filters. > > I have contacted illumos dev twice but unfortunately there was no reply, it seems the i40e driver developer is no longer active. I am trying to figure out how I can fix the issue by myself, I know how to do driver development but this particular issue seems to be very unique, and I suspect either something is missing when creating the VSIs, or there might be a bug (possibly in the Ethernet card firmware) in the way non-default PF VSI is created. > > I have researched the Linux and FreeBSD i40e driver code, none of them does the same kind of thing our illumos driver does. For each i40e instance, we create 31 PF VSIs in addition to the default one initiated by the card firmware. When receiving data through the RX queues of the default VSI, the performance is very good, but when receiving data through the RX queues of the non-default VSI, the performance can be very poor. > > The "Intel(r) Ethernet Controller X710/ XXV710/XL710 Datasheet" suggests there's nothing wrong by having 32 PF VSIs. Please refer to its page 20: > > "VSI Support > The X710/XXV710/XL710 supports a total of 384 VSIs. VSI assignment is flexible, but the choice to support 384 VSIs is motivated by the following usage example: > * 256 VSIs for VFs or VMDq2 > * 32 VSIs for PFs" > > We purchased a bunch of XL710 cards, hoping it can deliver better performance than the ixgbe cards. Sadly we were hit by this performance issue, and I've tried troubleshooting/debugging for more than 2 months, but so far no luck. > > Your kind assistance would be highly appreciated. > > Sincerely, > > -Youzhong Yang > The MathWorks Inc. > > -----Original Message----- > From: Fujinaka, Todd <tod...@in...> > Sent: Monday, December 2, 2019 4:39 PM > To: Youzhong Yang <yy...@ma...>; e10...@li... > Subject: RE: X710: very slow rx speed (a few MiB/second) when using non-default VSI. > > I think you're going to have to ask Illumos about this. Can you reproduce this on a supported OS such as Red Hat Linux or FreeBSD? > > Todd Fujinaka > Software Application Engineer > Datacenter Engineering Group > Intel Corporation > tod...@in... > > -----Original Message----- > From: Youzhong Yang <yy...@ma...> > Sent: Monday, December 2, 2019 12:19 PM > To: e10...@li... > Subject: [E1000-devel] X710: very slow rx speed (a few MiB/second) when using non-default VSI. > > Hello everyone, > > I submitted a help request on Intel website and was redirected to this mailing list. The request can be found here: > > X710: very slow rx speed (a few MiB/second) when using non-default VSI. > > > I experienced very strange performance issue when sending data from a client to a server with X710 card. Details about the investigation can be found here: > > https://github.com/youzhongyang/i40e_investigation > > > > The server runs illumos (derived from open solaris), its i40e driver can be found here: > > http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/io/i40e/ > > > > So for each i40e instance, we create 32 VSIs (including the default one created by the firmware), each VSI is assigned 8 trq pairs. When a VIF/VNIC is created, the OS assigns a VSI to it, data receiving through the VNIC's IP will be done by the relevant VSI's 8 RX rings. However, data sending will always go through the default VSI's 8 TX rings. > > > > The function for creating non-default VSI is here: > > http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/io/i40e/i40e_main.c#2111 > > > > I've upgraded the firmware version to the latest, still the same issue. Is this a known issue? or is there anything missing when setting up the non-default VSI? > > > > Your help/advice would be highly appreciated. > > Sincerely, > -Youzhong Yang > > > _______________________________________________ > E1000-devel mailing list > E10...@li... > https://lists.sourceforge.net/lists/listinfo/e1000-devel > To learn more about Intel Ethernet, visit https://forums.intel.com/s/topic/0TO0P00000018NbWAI/intel-ethernet > > > > _______________________________________________ > E1000-devel mailing list > E10...@li... > https://lists.sourceforge.net/lists/listinfo/e1000-devel > To learn more about Intel Ethernet, visit https://forums.intel.com/s/topic/0TO0P00000018NbWAI/intel-ethernet |
|
From: Youzhong Y. <yy...@ma...> - 2019-12-13 19:03:21
|
As I said, a supported OS (Linux and FreeBSD?) does not create the kind of VSI (PF VSI) in its i40e driver, are you asking me to test nothing on your supported OS? "Intel(r) Ethernet Controller X710/ XXV710/XL710 Datasheet" claims it supports PF VSIs, so why would you say it is not supported? Because your customer is not using Linux and FreeBSD? It does not make any sense. Sorry. -----Original Message----- From: Fujinaka, Todd <tod...@in...> Sent: Friday, December 13, 2019 1:45 PM To: Youzhong Yang <yy...@ma...>; e10...@li... Subject: RE: X710: very slow rx speed (a few MiB/second) when using non-default VSI. You can try official channels but you're just going to get an official "Talk to Illuminos" from me. As I said, you can try installing a supported OS to see if it's a hardware issue, but we have no support for Illuminos. Todd Fujinaka Software Application Engineer Datacenter Engineering Group Intel Corporation tod...@in... -----Original Message----- From: Youzhong Yang <yy...@ma...> Sent: Friday, December 13, 2019 10:11 AM To: Fujinaka, Todd <tod...@in...>; e10...@li... Subject: RE: X710: very slow rx speed (a few MiB/second) when using non-default VSI. This mailing list seems quite quiet. Any hope my issue could get some attention? If it is a thing of 'Only Linux and FreeBSD are supported', is there any other communication channel through which this issue can be escalated? Thanks a lot for your understanding. -----Original Message----- From: Youzhong Yang <yy...@ma...> Sent: Tuesday, December 3, 2019 12:21 PM To: Fujinaka, Todd <tod...@in...>; e10...@li... Subject: RE: X710: very slow rx speed (a few MiB/second) when using non-default VSI. Hello Todd, Thank you very much for replying so fast. I am sorry I missed your e-mail as it was blocked by our company e-mail filters. I have contacted illumos dev twice but unfortunately there was no reply, it seems the i40e driver developer is no longer active. I am trying to figure out how I can fix the issue by myself, I know how to do driver development but this particular issue seems to be very unique, and I suspect either something is missing when creating the VSIs, or there might be a bug (possibly in the Ethernet card firmware) in the way non-default PF VSI is created. I have researched the Linux and FreeBSD i40e driver code, none of them does the same kind of thing our illumos driver does. For each i40e instance, we create 31 PF VSIs in addition to the default one initiated by the card firmware. When receiving data through the RX queues of the default VSI, the performance is very good, but when receiving data through the RX queues of the non-default VSI, the performance can be very poor. The "Intel(r) Ethernet Controller X710/ XXV710/XL710 Datasheet" suggests there's nothing wrong by having 32 PF VSIs. Please refer to its page 20: "VSI Support The X710/XXV710/XL710 supports a total of 384 VSIs. VSI assignment is flexible, but the choice to support 384 VSIs is motivated by the following usage example: * 256 VSIs for VFs or VMDq2 * 32 VSIs for PFs" We purchased a bunch of XL710 cards, hoping it can deliver better performance than the ixgbe cards. Sadly we were hit by this performance issue, and I've tried troubleshooting/debugging for more than 2 months, but so far no luck. Your kind assistance would be highly appreciated. Sincerely, -Youzhong Yang The MathWorks Inc. -----Original Message----- From: Fujinaka, Todd <tod...@in...> Sent: Monday, December 2, 2019 4:39 PM To: Youzhong Yang <yy...@ma...>; e10...@li... Subject: RE: X710: very slow rx speed (a few MiB/second) when using non-default VSI. I think you're going to have to ask Illumos about this. Can you reproduce this on a supported OS such as Red Hat Linux or FreeBSD? Todd Fujinaka Software Application Engineer Datacenter Engineering Group Intel Corporation tod...@in... -----Original Message----- From: Youzhong Yang <yy...@ma...> Sent: Monday, December 2, 2019 12:19 PM To: e10...@li... Subject: [E1000-devel] X710: very slow rx speed (a few MiB/second) when using non-default VSI. Hello everyone, I submitted a help request on Intel website and was redirected to this mailing list. The request can be found here: X710: very slow rx speed (a few MiB/second) when using non-default VSI. I experienced very strange performance issue when sending data from a client to a server with X710 card. Details about the investigation can be found here: https://github.com/youzhongyang/i40e_investigation The server runs illumos (derived from open solaris), its i40e driver can be found here: http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/io/i40e/ So for each i40e instance, we create 32 VSIs (including the default one created by the firmware), each VSI is assigned 8 trq pairs. When a VIF/VNIC is created, the OS assigns a VSI to it, data receiving through the VNIC's IP will be done by the relevant VSI's 8 RX rings. However, data sending will always go through the default VSI's 8 TX rings. The function for creating non-default VSI is here: http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/io/i40e/i40e_main.c#2111 I've upgraded the firmware version to the latest, still the same issue. Is this a known issue? or is there anything missing when setting up the non-default VSI? Your help/advice would be highly appreciated. Sincerely, -Youzhong Yang _______________________________________________ E1000-devel mailing list E10...@li... https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel Ethernet, visit https://forums.intel.com/s/topic/0TO0P00000018NbWAI/intel-ethernet |
|
From: Fujinaka, T. <tod...@in...> - 2019-12-13 18:45:17
|
You can try official channels but you're just going to get an official "Talk to Illuminos" from me. As I said, you can try installing a supported OS to see if it's a hardware issue, but we have no support for Illuminos. Todd Fujinaka Software Application Engineer Datacenter Engineering Group Intel Corporation tod...@in... -----Original Message----- From: Youzhong Yang <yy...@ma...> Sent: Friday, December 13, 2019 10:11 AM To: Fujinaka, Todd <tod...@in...>; e10...@li... Subject: RE: X710: very slow rx speed (a few MiB/second) when using non-default VSI. This mailing list seems quite quiet. Any hope my issue could get some attention? If it is a thing of 'Only Linux and FreeBSD are supported', is there any other communication channel through which this issue can be escalated? Thanks a lot for your understanding. -----Original Message----- From: Youzhong Yang <yy...@ma...> Sent: Tuesday, December 3, 2019 12:21 PM To: Fujinaka, Todd <tod...@in...>; e10...@li... Subject: RE: X710: very slow rx speed (a few MiB/second) when using non-default VSI. Hello Todd, Thank you very much for replying so fast. I am sorry I missed your e-mail as it was blocked by our company e-mail filters. I have contacted illumos dev twice but unfortunately there was no reply, it seems the i40e driver developer is no longer active. I am trying to figure out how I can fix the issue by myself, I know how to do driver development but this particular issue seems to be very unique, and I suspect either something is missing when creating the VSIs, or there might be a bug (possibly in the Ethernet card firmware) in the way non-default PF VSI is created. I have researched the Linux and FreeBSD i40e driver code, none of them does the same kind of thing our illumos driver does. For each i40e instance, we create 31 PF VSIs in addition to the default one initiated by the card firmware. When receiving data through the RX queues of the default VSI, the performance is very good, but when receiving data through the RX queues of the non-default VSI, the performance can be very poor. The "Intel(r) Ethernet Controller X710/ XXV710/XL710 Datasheet" suggests there's nothing wrong by having 32 PF VSIs. Please refer to its page 20: "VSI Support The X710/XXV710/XL710 supports a total of 384 VSIs. VSI assignment is flexible, but the choice to support 384 VSIs is motivated by the following usage example: * 256 VSIs for VFs or VMDq2 * 32 VSIs for PFs" We purchased a bunch of XL710 cards, hoping it can deliver better performance than the ixgbe cards. Sadly we were hit by this performance issue, and I've tried troubleshooting/debugging for more than 2 months, but so far no luck. Your kind assistance would be highly appreciated. Sincerely, -Youzhong Yang The MathWorks Inc. -----Original Message----- From: Fujinaka, Todd <tod...@in...> Sent: Monday, December 2, 2019 4:39 PM To: Youzhong Yang <yy...@ma...>; e10...@li... Subject: RE: X710: very slow rx speed (a few MiB/second) when using non-default VSI. I think you're going to have to ask Illumos about this. Can you reproduce this on a supported OS such as Red Hat Linux or FreeBSD? Todd Fujinaka Software Application Engineer Datacenter Engineering Group Intel Corporation tod...@in... -----Original Message----- From: Youzhong Yang <yy...@ma...> Sent: Monday, December 2, 2019 12:19 PM To: e10...@li... Subject: [E1000-devel] X710: very slow rx speed (a few MiB/second) when using non-default VSI. Hello everyone, I submitted a help request on Intel website and was redirected to this mailing list. The request can be found here: X710: very slow rx speed (a few MiB/second) when using non-default VSI. I experienced very strange performance issue when sending data from a client to a server with X710 card. Details about the investigation can be found here: https://github.com/youzhongyang/i40e_investigation The server runs illumos (derived from open solaris), its i40e driver can be found here: http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/io/i40e/ So for each i40e instance, we create 32 VSIs (including the default one created by the firmware), each VSI is assigned 8 trq pairs. When a VIF/VNIC is created, the OS assigns a VSI to it, data receiving through the VNIC's IP will be done by the relevant VSI's 8 RX rings. However, data sending will always go through the default VSI's 8 TX rings. The function for creating non-default VSI is here: http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/io/i40e/i40e_main.c#2111 I've upgraded the firmware version to the latest, still the same issue. Is this a known issue? or is there anything missing when setting up the non-default VSI? Your help/advice would be highly appreciated. Sincerely, -Youzhong Yang _______________________________________________ E1000-devel mailing list E10...@li... https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel Ethernet, visit https://forums.intel.com/s/topic/0TO0P00000018NbWAI/intel-ethernet |
|
From: Youzhong Y. <yy...@ma...> - 2019-12-13 18:11:40
|
This mailing list seems quite quiet. Any hope my issue could get some attention? If it is a thing of 'Only Linux and FreeBSD are supported', is there any other communication channel through which this issue can be escalated? Thanks a lot for your understanding. -----Original Message----- From: Youzhong Yang <yy...@ma...> Sent: Tuesday, December 3, 2019 12:21 PM To: Fujinaka, Todd <tod...@in...>; e10...@li... Subject: RE: X710: very slow rx speed (a few MiB/second) when using non-default VSI. Hello Todd, Thank you very much for replying so fast. I am sorry I missed your e-mail as it was blocked by our company e-mail filters. I have contacted illumos dev twice but unfortunately there was no reply, it seems the i40e driver developer is no longer active. I am trying to figure out how I can fix the issue by myself, I know how to do driver development but this particular issue seems to be very unique, and I suspect either something is missing when creating the VSIs, or there might be a bug (possibly in the Ethernet card firmware) in the way non-default PF VSI is created. I have researched the Linux and FreeBSD i40e driver code, none of them does the same kind of thing our illumos driver does. For each i40e instance, we create 31 PF VSIs in addition to the default one initiated by the card firmware. When receiving data through the RX queues of the default VSI, the performance is very good, but when receiving data through the RX queues of the non-default VSI, the performance can be very poor. The "Intel(r) Ethernet Controller X710/ XXV710/XL710 Datasheet" suggests there's nothing wrong by having 32 PF VSIs. Please refer to its page 20: "VSI Support The X710/XXV710/XL710 supports a total of 384 VSIs. VSI assignment is flexible, but the choice to support 384 VSIs is motivated by the following usage example: * 256 VSIs for VFs or VMDq2 * 32 VSIs for PFs" We purchased a bunch of XL710 cards, hoping it can deliver better performance than the ixgbe cards. Sadly we were hit by this performance issue, and I've tried troubleshooting/debugging for more than 2 months, but so far no luck. Your kind assistance would be highly appreciated. Sincerely, -Youzhong Yang The MathWorks Inc. -----Original Message----- From: Fujinaka, Todd <tod...@in...> Sent: Monday, December 2, 2019 4:39 PM To: Youzhong Yang <yy...@ma...>; e10...@li... Subject: RE: X710: very slow rx speed (a few MiB/second) when using non-default VSI. I think you're going to have to ask Illumos about this. Can you reproduce this on a supported OS such as Red Hat Linux or FreeBSD? Todd Fujinaka Software Application Engineer Datacenter Engineering Group Intel Corporation tod...@in... -----Original Message----- From: Youzhong Yang <yy...@ma...> Sent: Monday, December 2, 2019 12:19 PM To: e10...@li... Subject: [E1000-devel] X710: very slow rx speed (a few MiB/second) when using non-default VSI. Hello everyone, I submitted a help request on Intel website and was redirected to this mailing list. The request can be found here: X710: very slow rx speed (a few MiB/second) when using non-default VSI. I experienced very strange performance issue when sending data from a client to a server with X710 card. Details about the investigation can be found here: https://github.com/youzhongyang/i40e_investigation The server runs illumos (derived from open solaris), its i40e driver can be found here: http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/io/i40e/ So for each i40e instance, we create 32 VSIs (including the default one created by the firmware), each VSI is assigned 8 trq pairs. When a VIF/VNIC is created, the OS assigns a VSI to it, data receiving through the VNIC's IP will be done by the relevant VSI's 8 RX rings. However, data sending will always go through the default VSI's 8 TX rings. The function for creating non-default VSI is here: http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/io/i40e/i40e_main.c#2111 I've upgraded the firmware version to the latest, still the same issue. Is this a known issue? or is there anything missing when setting up the non-default VSI? Your help/advice would be highly appreciated. Sincerely, -Youzhong Yang _______________________________________________ E1000-devel mailing list E10...@li... https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel Ethernet, visit https://forums.intel.com/s/topic/0TO0P00000018NbWAI/intel-ethernet |
|
From: Alexander D. <ale...@gm...> - 2019-12-04 19:26:37
|
On Wed, Dec 4, 2019 at 10:56 AM Eddy Geez <edd...@gm...> wrote: > > On Tue, Dec 3, 2019 at 6:22 PM Alexander Duyck > <ale...@gm...> wrote: > > On Tue, Dec 3, 2019 at 12:40 PM Eddy Geez <edd...@gm...> wrote: > > > > > > BLUF: > > > Is there any way to bring the interface physically up immediately > > > when the e1000e driver loads so auto-negotiation can start? > > > > > > Greetings! > > > > > > I am evaluating some Intel NUCs that will be diskless and rely on > > > PXE netbooting. > > > > > > I have everything working, but the boot process is significantly > > > delayed after the OS (in this case, Ubuntu 18.04.3) starts: > > > > > > From the dmesg output: > > > > > > [ 1.390993] e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k > > > [ 1.932005] e1000e 0000:00:1f.6 eno1: renamed from eth0 > > > [ 9.538232] e1000e: eno1 NIC Link is Up 1000 Mbps Full Duplex, Flow > > > Control: None > > > > > > So there's about 7 seconds where nothing is happening. > > > > > > I've searched this list and found messages about how auto-negotiation > > > can take 3-4 seconds, but there's more going on here. Specifically, > > > once iPXE is done and hands off control to the initrd, link on the NIC > > > goes away and does NOT come back up when the kernel loads the > > > e1000e driver. It isn't until the later in the boot process, when > > > the interface is configured, that link negotiation starts and the > > > LEDs on the NIC come back on. > > > > > > Is there any way to bring the interface physically up immediately > > > when the e1000e driver loads so auto-negotiation can start right > > > away? I'm guessing this simple change would take several seconds > > > off the boot time. (Incidentally, the auto-negotiation during PXE > > > boot happens very quickly.) > > > > > > FWIW, I've also built and installed the latest version of the driver: > > > > > > [ 1.410023] e1000e: Intel(R) PRO/1000 Network Driver - 3.6.0-NAPI > > > > > > and it didn't behave any differently. > > > > > > Thanks for any insight! > > > > So the issue is likely going to be that the physical interface for the > > NIC is powered down in e1000e_reset via e1000_power_down_phy and is > > not powered back on until e1000e_open calls e100_power_up_phy. As such > > the PHY has to be powered on and initialized before you can even start > > getting to the link negotiation process. > > > > Also I believe 3 to 4 seconds is the typical link time for 10/100Mbps > > Ethernet. If I recall correctly for 1Gbps you are looking at 4 to 5 > > seconds for link negotiation. > > > > Hope that helps. > > Thanks for the quick reply! > > So there's no way to get `e1000_power_up_phy` called sooner? > (I'm envisioning something like an E1000_PARAM that could be specified > as an option in /etc/modprobe.d/e1000e.conf) As far as I am aware nothing like that exists in the current driver. > As I mentioned, I had read previously on this list that auto-neg > can take 3-4 seconds, and that's why I want to start that process > ASAP after the e1000e module gets loaded by the kernel. Then, with > any luck, when the boot process gets around to configuring the network > interface it will hopefully be ready to go, letting the netboot process > continue that much sooner. > > Thanks again for any suggestions in speeding things up! If you are wanting to experiment, one thing you might try rebuilding the driver and modifying the function e1000_check_reset_block_ich8lan which should be in the ich8lan.c file for the driver. If you modified it so that it always returned the blocked value then that would likely disable any PHY resets and power cycles which may improve your boot time. Hope that helps. - Alex |
|
From: Eddy G. <edd...@gm...> - 2019-12-04 13:11:39
|
I'm also curious about these comments in the iPXE source code if anybody
has any insight. (The NIC comes up and establishes link very quickly in
iPXE.)
/** The i219 has a seriously broken reset mechanism */
#define INTEL_I219 ( INTEL_NO_PHY_RST | INTEL_RST_HANG )
from:
https://git.ipxe.org/ipxe.git/blob/master:/src/drivers/net/intel.h#l319
And to ease the code-spelunking, the code that handles INTEL_NO_PHY_RST
is at:
https://git.ipxe.org/ipxe.git/blob/master:/src/drivers/net/intel.c#l305
and INTEL_RST_HANG is at:
https://git.ipxe.org/ipxe.git/blob/master:/src/drivers/net/intel.c#l643
and mentions an "undocumented bit in FEXTNVM11 to work around an errata
in i219 devices that will otherwise cause a complete datapath hang at
the next device reset".
Thanks again for any insight, especially if there are similar things in
the e1000e kernel driver for the I219.
|
|
From: Eddy G. <edd...@gm...> - 2019-12-04 02:33:12
|
On Tue, Dec 3, 2019 at 6:22 PM Alexander Duyck <ale...@gm...> wrote: > On Tue, Dec 3, 2019 at 12:40 PM Eddy Geez <edd...@gm...> wrote: > > > > BLUF: > > Is there any way to bring the interface physically up immediately > > when the e1000e driver loads so auto-negotiation can start? > > > > Greetings! > > > > I am evaluating some Intel NUCs that will be diskless and rely on > > PXE netbooting. > > > > I have everything working, but the boot process is significantly > > delayed after the OS (in this case, Ubuntu 18.04.3) starts: > > > > From the dmesg output: > > > > [ 1.390993] e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k > > [ 1.932005] e1000e 0000:00:1f.6 eno1: renamed from eth0 > > [ 9.538232] e1000e: eno1 NIC Link is Up 1000 Mbps Full Duplex, Flow > > Control: None > > > > So there's about 7 seconds where nothing is happening. > > > > I've searched this list and found messages about how auto-negotiation > > can take 3-4 seconds, but there's more going on here. Specifically, > > once iPXE is done and hands off control to the initrd, link on the NIC > > goes away and does NOT come back up when the kernel loads the > > e1000e driver. It isn't until the later in the boot process, when > > the interface is configured, that link negotiation starts and the > > LEDs on the NIC come back on. > > > > Is there any way to bring the interface physically up immediately > > when the e1000e driver loads so auto-negotiation can start right > > away? I'm guessing this simple change would take several seconds > > off the boot time. (Incidentally, the auto-negotiation during PXE > > boot happens very quickly.) > > > > FWIW, I've also built and installed the latest version of the driver: > > > > [ 1.410023] e1000e: Intel(R) PRO/1000 Network Driver - 3.6.0-NAPI > > > > and it didn't behave any differently. > > > > Thanks for any insight! > > So the issue is likely going to be that the physical interface for the > NIC is powered down in e1000e_reset via e1000_power_down_phy and is > not powered back on until e1000e_open calls e100_power_up_phy. As such > the PHY has to be powered on and initialized before you can even start > getting to the link negotiation process. > > Also I believe 3 to 4 seconds is the typical link time for 10/100Mbps > Ethernet. If I recall correctly for 1Gbps you are looking at 4 to 5 > seconds for link negotiation. > > Hope that helps. Thanks for the quick reply! So there's no way to get `e1000_power_up_phy` called sooner? (I'm envisioning something like an E1000_PARAM that could be specified as an option in /etc/modprobe.d/e1000e.conf) As I mentioned, I had read previously on this list that auto-neg can take 3-4 seconds, and that's why I want to start that process ASAP after the e1000e module gets loaded by the kernel. Then, with any luck, when the boot process gets around to configuring the network interface it will hopefully be ready to go, letting the netboot process continue that much sooner. Thanks again for any suggestions in speeding things up! |
|
From: Alexander D. <ale...@gm...> - 2019-12-03 23:22:31
|
On Tue, Dec 3, 2019 at 12:40 PM Eddy Geez <edd...@gm...> wrote: > > BLUF: > Is there any way to bring the interface physically up immediately > when the e1000e driver loads so auto-negotiation can start? > > Greetings! > > I am evaluating some Intel NUCs that will be diskless and rely on > PXE netbooting. > > I have everything working, but the boot process is significantly > delayed after the OS (in this case, Ubuntu 18.04.3) starts: > > From the dmesg output: > > [ 1.390993] e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k > [ 1.932005] e1000e 0000:00:1f.6 eno1: renamed from eth0 > [ 9.538232] e1000e: eno1 NIC Link is Up 1000 Mbps Full Duplex, Flow > Control: None > > So there's about 7 seconds where nothing is happening. > > I've searched this list and found messages about how auto-negotiation > can take 3-4 seconds, but there's more going on here. Specifically, > once iPXE is done and hands off control to the initrd, link on the NIC > goes away and does NOT come back up when the kernel loads the > e1000e driver. It isn't until the later in the boot process, when > the interface is configured, that link negotiation starts and the > LEDs on the NIC come back on. > > Is there any way to bring the interface physically up immediately > when the e1000e driver loads so auto-negotiation can start right > away? I'm guessing this simple change would take several seconds > off the boot time. (Incidentally, the auto-negotiation during PXE > boot happens very quickly.) > > FWIW, I've also built and installed the latest version of the driver: > > [ 1.410023] e1000e: Intel(R) PRO/1000 Network Driver - 3.6.0-NAPI > > and it didn't behave any differently. > > Thanks for any insight! So the issue is likely going to be that the physical interface for the NIC is powered down in e1000e_reset via e1000_power_down_phy and is not powered back on until e1000e_open calls e100_power_up_phy. As such the PHY has to be powered on and initialized before you can even start getting to the link negotiation process. Also I believe 3 to 4 seconds is the typical link time for 10/100Mbps Ethernet. If I recall correctly for 1Gbps you are looking at 4 to 5 seconds for link negotiation. Hope that helps. - Alex |
|
From: Eddy G. <edd...@gm...> - 2019-12-03 20:37:44
|
BLUF: Is there any way to bring the interface physically up immediately when the e1000e driver loads so auto-negotiation can start? Greetings! I am evaluating some Intel NUCs that will be diskless and rely on PXE netbooting. I have everything working, but the boot process is significantly delayed after the OS (in this case, Ubuntu 18.04.3) starts: >From the dmesg output: [ 1.390993] e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k [ 1.932005] e1000e 0000:00:1f.6 eno1: renamed from eth0 [ 9.538232] e1000e: eno1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None So there's about 7 seconds where nothing is happening. I've searched this list and found messages about how auto-negotiation can take 3-4 seconds, but there's more going on here. Specifically, once iPXE is done and hands off control to the initrd, link on the NIC goes away and does NOT come back up when the kernel loads the e1000e driver. It isn't until the later in the boot process, when the interface is configured, that link negotiation starts and the LEDs on the NIC come back on. Is there any way to bring the interface physically up immediately when the e1000e driver loads so auto-negotiation can start right away? I'm guessing this simple change would take several seconds off the boot time. (Incidentally, the auto-negotiation during PXE boot happens very quickly.) FWIW, I've also built and installed the latest version of the driver: [ 1.410023] e1000e: Intel(R) PRO/1000 Network Driver - 3.6.0-NAPI and it didn't behave any differently. Thanks for any insight! |
|
From: Youzhong Y. <yy...@ma...> - 2019-12-03 17:36:14
|
Hello Todd, Thank you very much for replying so fast. I am sorry I missed your e-mail as it was blocked by our company e-mail filters. I have contacted illumos dev twice but unfortunately there was no reply, it seems the i40e driver developer is no longer active. I am trying to figure out how I can fix the issue by myself, I know how to do driver development but this particular issue seems to be very unique, and I suspect either something is missing when creating the VSIs, or there might be a bug (possibly in the Ethernet card firmware) in the way non-default PF VSI is created. I have researched the Linux and FreeBSD i40e driver code, none of them does the same kind of thing our illumos driver does. For each i40e instance, we create 31 PF VSIs in addition to the default one initiated by the card firmware. When receiving data through the RX queues of the default VSI, the performance is very good, but when receiving data through the RX queues of the non-default VSI, the performance can be very poor. The "Intel(r) Ethernet Controller X710/ XXV710/XL710 Datasheet" suggests there's nothing wrong by having 32 PF VSIs. Please refer to its page 20: "VSI Support The X710/XXV710/XL710 supports a total of 384 VSIs. VSI assignment is flexible, but the choice to support 384 VSIs is motivated by the following usage example: * 256 VSIs for VFs or VMDq2 * 32 VSIs for PFs" We purchased a bunch of XL710 cards, hoping it can deliver better performance than the ixgbe cards. Sadly we were hit by this performance issue, and I've tried troubleshooting/debugging for more than 2 months, but so far no luck. Your kind assistance would be highly appreciated. Sincerely, -Youzhong Yang The MathWorks Inc. -----Original Message----- From: Fujinaka, Todd <tod...@in...> Sent: Monday, December 2, 2019 4:39 PM To: Youzhong Yang <yy...@ma...>; e10...@li... Subject: RE: X710: very slow rx speed (a few MiB/second) when using non-default VSI. I think you're going to have to ask Illumos about this. Can you reproduce this on a supported OS such as Red Hat Linux or FreeBSD? Todd Fujinaka Software Application Engineer Datacenter Engineering Group Intel Corporation tod...@in... -----Original Message----- From: Youzhong Yang <yy...@ma...> Sent: Monday, December 2, 2019 12:19 PM To: e10...@li... Subject: [E1000-devel] X710: very slow rx speed (a few MiB/second) when using non-default VSI. Hello everyone, I submitted a help request on Intel website and was redirected to this mailing list. The request can be found here: X710: very slow rx speed (a few MiB/second) when using non-default VSI. I experienced very strange performance issue when sending data from a client to a server with X710 card. Details about the investigation can be found here: https://github.com/youzhongyang/i40e_investigation The server runs illumos (derived from open solaris), its i40e driver can be found here: http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/io/i40e/ So for each i40e instance, we create 32 VSIs (including the default one created by the firmware), each VSI is assigned 8 trq pairs. When a VIF/VNIC is created, the OS assigns a VSI to it, data receiving through the VNIC's IP will be done by the relevant VSI's 8 RX rings. However, data sending will always go through the default VSI's 8 TX rings. The function for creating non-default VSI is here: http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/io/i40e/i40e_main.c#2111 I've upgraded the firmware version to the latest, still the same issue. Is this a known issue? or is there anything missing when setting up the non-default VSI? Your help/advice would be highly appreciated. Sincerely, -Youzhong Yang _______________________________________________ E1000-devel mailing list E10...@li... https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel Ethernet, visit https://forums.intel.com/s/topic/0TO0P00000018NbWAI/intel-ethernet |
|
From: Fujinaka, T. <tod...@in...> - 2019-12-02 21:39:35
|
I think you're going to have to ask Illumos about this. Can you reproduce this on a supported OS such as Red Hat Linux or FreeBSD? Todd Fujinaka Software Application Engineer Datacenter Engineering Group Intel Corporation tod...@in... -----Original Message----- From: Youzhong Yang <yy...@ma...> Sent: Monday, December 2, 2019 12:19 PM To: e10...@li... Subject: [E1000-devel] X710: very slow rx speed (a few MiB/second) when using non-default VSI. Hello everyone, I submitted a help request on Intel website and was redirected to this mailing list. The request can be found here: X710: very slow rx speed (a few MiB/second) when using non-default VSI. I experienced very strange performance issue when sending data from a client to a server with X710 card. Details about the investigation can be found here: https://github.com/youzhongyang/i40e_investigation The server runs illumos (derived from open solaris), its i40e driver can be found here: http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/io/i40e/ So for each i40e instance, we create 32 VSIs (including the default one created by the firmware), each VSI is assigned 8 trq pairs. When a VIF/VNIC is created, the OS assigns a VSI to it, data receiving through the VNIC's IP will be done by the relevant VSI's 8 RX rings. However, data sending will always go through the default VSI's 8 TX rings. The function for creating non-default VSI is here: http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/io/i40e/i40e_main.c#2111 I've upgraded the firmware version to the latest, still the same issue. Is this a known issue? or is there anything missing when setting up the non-default VSI? Your help/advice would be highly appreciated. Sincerely, -Youzhong Yang _______________________________________________ E1000-devel mailing list E10...@li... https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel Ethernet, visit https://forums.intel.com/s/topic/0TO0P00000018NbWAI/intel-ethernet |
|
From: Youzhong Y. <yy...@ma...> - 2019-12-02 20:19:46
|
Hello everyone, I submitted a help request on Intel website and was redirected to this mailing list. The request can be found here: X710: very slow rx speed (a few MiB/second) when using non-default VSI. I experienced very strange performance issue when sending data from a client to a server with X710 card. Details about the investigation can be found here: https://github.com/youzhongyang/i40e_investigation The server runs illumos (derived from open solaris), its i40e driver can be found here: http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/io/i40e/ So for each i40e instance, we create 32 VSIs (including the default one created by the firmware), each VSI is assigned 8 trq pairs. When a VIF/VNIC is created, the OS assigns a VSI to it, data receiving through the VNIC's IP will be done by the relevant VSI's 8 RX rings. However, data sending will always go through the default VSI's 8 TX rings. The function for creating non-default VSI is here: http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/io/i40e/i40e_main.c#2111 I've upgraded the firmware version to the latest, still the same issue. Is this a known issue? or is there anything missing when setting up the non-default VSI? Your help/advice would be highly appreciated. Sincerely, -Youzhong Yang |
