e1000-devel Mailing List for Formerly Intel Ethernet Drivers

On Tue, May 21, 2019 at 04:22:17PM -0700, Alexander Duyck wrote:
> So we are either using 0 bits of the LUT or we are just not performing
> hashing because this is somehow being parsed into a type that doesn't
> support it.
> 
> I have attached 2 more patches we can test. The first enables hashing
> on what are called "OAM" packets, The thing is we shouldn't be
> identifying these packets as "OAM", Operations Administration &
> Management, as normally it would have to be recognized as a tunnel
> first and then have a specific flag set in either the GENEVE or
> VXLAN-GPE header. The second patch will dump the contents of the
> HREGION registers. They should all be 0, however I thought it best to
> dump the contents and verify that since I know that these registers
> can be used to change the traffic class of a given packet type and if
> we are encountering that it might be moving it to an uninitialized TC
> which would be using queue offset 0 with 0 bits of the LUT.
> 
> These last 2 patches would pretty much eliminate the entire RSS
> subsystem. If we don't see HREGION values set and the OAM flags have
> no effect I can only assume there is something going on with the
> parser in the NIC since it isn't recognizing the packet type.
> 
> Thanks.
> 
> - Alex

OK I applied those two patches and get this:

i40e: Intel(R) Ethernet Connection XL710 Network Driver - version 2.1.7-k
i40e: Copyright (c) 2013 - 2014 Intel Corporation.
i40e 0000:3d:00.0: fw 3.10.52896 api 1.6 nvm 4.00 0x80001577 1.1767.0
i40e 0000:3d:00.0: The driver for the device detected a newer version of the NVM image than expected. Please install the most recent version of the network driver.
i40e 0000:3d:00.0: MAC address: a4:bf:01:4e:0c:87
i40e 0000:3d:00.0: PFQF_HREGION[7]: 0x00000000
i40e 0000:3d:00.0: PFQF_HREGION[6]: 0x00000000
i40e 0000:3d:00.0: PFQF_HREGION[5]: 0x00000000
i40e 0000:3d:00.0: PFQF_HREGION[4]: 0x00000000
i40e 0000:3d:00.0: PFQF_HREGION[3]: 0x00000000
i40e 0000:3d:00.0: PFQF_HREGION[2]: 0x00000000
i40e 0000:3d:00.0: PFQF_HREGION[1]: 0x00000000
i40e 0000:3d:00.0: PFQF_HREGION[0]: 0x00000000
i40e 0000:3d:00.0: flow_type: 63 input_mask:0x0000000000004000
i40e 0000:3d:00.0: flow_type: 46 input_mask:0x0007fff800000000
i40e 0000:3d:00.0: flow_type: 45 input_mask:0x0007fff800000000
i40e 0000:3d:00.0: flow_type: 44 input_mask:0x0007ffff80000000
i40e 0000:3d:00.0: flow_type: 43 input_mask:0x0007fffe00000000
i40e 0000:3d:00.0: flow_type: 42 input_mask:0x0007fffe00000000
i40e 0000:3d:00.0: flow_type: 41 input_mask:0x0007fffe00000000
i40e 0000:3d:00.0: flow_type: 40 input_mask:0x0007fffe00000000
i40e 0000:3d:00.0: flow_type: 39 input_mask:0x0007fffe00000000
i40e 0000:3d:00.0: flow_type: 36 input_mask:0x0006060000000000
i40e 0000:3d:00.0: flow_type: 35 input_mask:0x0006060000000000
i40e 0000:3d:00.0: flow_type: 34 input_mask:0x0006060780000000
i40e 0000:3d:00.0: flow_type: 33 input_mask:0x0006060600000000
i40e 0000:3d:00.0: flow_type: 32 input_mask:0x0006060600000000
i40e 0000:3d:00.0: flow_type: 31 input_mask:0x0006060600000000
i40e 0000:3d:00.0: flow_type: 30 input_mask:0x0006060600000000
i40e 0000:3d:00.0: flow_type: 29 input_mask:0x0006060600000000
i40e 0000:3d:00.0: flow_type: 27 input_mask:0x00000000002c0000
i40e 0000:3d:00.0: flow_type: 26 input_mask:0x00000000002c0000
i40e 0000:3d:00.0: flow type: 36 update input mask from:0x0006060000000000, to:0x0001801800000000
i40e 0000:3d:00.0: flow type: 35 update input mask from:0x0006060000000000, to:0x0001801800000000
i40e 0000:3d:00.0: flow type: 34 update input mask from:0x0006060780000000, to:0x0001801f80000000
i40e 0000:3d:00.0: flow type: 33 update input mask from:0x0006060600000000, to:0x0001801e00000000
i40e 0000:3d:00.0: flow type: 32 update input mask from:0x0006060600000000, to:0x0001801e00000000
i40e 0000:3d:00.0: flow type: 31 update input mask from:0x0006060600000000, to:0x0001801e00000000
i40e 0000:3d:00.0: flow type: 30 update input mask from:0x0006060600000000, to:0x0001801e00000000
i40e 0000:3d:00.0: flow type: 29 update input mask from:0x0006060600000000, to:0x0001801e00000000

So seems the regions are all 0.

All ipsec packets still hitting queue 0.

-- 
Len Sorensen

On Tue, May 21, 2019 at 10:55 AM Lennart Sorensen
<lso...@cs...> wrote:
>
> On Tue, May 21, 2019 at 09:51:33AM -0700, Alexander Duyck wrote:
> > I think we need to narrow this down a bit more. Let's try forcing the
> > lookup table all to one value and see if traffic is still going to
> > queue 0.
> >
> > Specifically what we need to is run the following command to try and
> > force all RSS traffic to queue 8, you can verify the result with
> > "ethtool -x":
> > ethtool -X <iface> weight 0 0 0 0 0 0 0 0 1
> >
> > If that works and the IPSec traffic goes to queue 8 then we are likely
> > looking at some sort of input issue, either in the parsing or the
> > population of things like the input mask that we can then debug
> > further.
> >
> > If traffic still goes to queue 0 then that tells us the output of the
> > RSS hash and lookup table are being ignored, this would imply either
> > some other filter is rerouting the traffic or is directing us to limit
> > the queue index to 0 bits.
>
> # ethtool -x eth2
> RX flow hash indirection table for eth2 with 12 RX ring(s):
>     0:      7     7     7     7     7     7     7     7
>     8:      7     7     7     7     7     7     7     7
>    16:      7     7     7     7     7     7     7     7
>    24:      7     7     7     7     7     7     7     7
>    32:      7     7     7     7     7     7     7     7
> ...
>   472:      7     7     7     7     7     7     7     7
>   480:      7     7     7     7     7     7     7     7
>   488:      7     7     7     7     7     7     7     7
>   496:      7     7     7     7     7     7     7     7
>   504:      7     7     7     7     7     7     7     7
> RSS hash key:
> 0b:1f:ae:ed:60:04:7d:e5:8a:2b:43:3f:1d:ee:6c:99:89:29:94:b0:25:db:c7:4b:fa:da:4d:3f:e8:cc:bc:00:ad:32:01:d6:1c:30:3f:f8:79:3e:f4:48:04:1f:51:d2:5a:39:f0:90
> root@ECA:~# ethtool --show-priv-flags eth2
> Private flags for eth2:
> MFP              : off
> LinkPolling      : off
> flow-director-atr: off
> veb-stats        : off
> hw-atr-eviction  : on
> legacy-rx        : off
>
> All ipsec packets are still hitting queue 0.
>
> Seems it is completely ignoring RSS for these packets.  That is
> impressively weird.
>
> --
> Len Sorensen

So we are either using 0 bits of the LUT or we are just not performing
hashing because this is somehow being parsed into a type that doesn't
support it.

I have attached 2 more patches we can test. The first enables hashing
on what are called "OAM" packets, The thing is we shouldn't be
identifying these packets as "OAM", Operations Administration &
Management, as normally it would have to be recognized as a tunnel
first and then have a specific flag set in either the GENEVE or
VXLAN-GPE header. The second patch will dump the contents of the
HREGION registers. They should all be 0, however I thought it best to
dump the contents and verify that since I know that these registers
can be used to change the traffic class of a given packet type and if
we are encountering that it might be moving it to an uninitialized TC
which would be using queue offset 0 with 0 bits of the LUT.

These last 2 patches would pretty much eliminate the entire RSS
subsystem. If we don't see HREGION values set and the OAM flags have
no effect I can only assume there is something going on with the
parser in the NIC since it isn't recognizing the packet type.

Thanks.

- Alex

On Mon, 13 May 2019 07:36:41 +0000 Periyasamy wrote:
> Hi,
> 
> I’m trying to achieve PF passthrough of 40/10G ethernet interface (i40e) into guest VM running on qemu/kvm hypervisor and then create VFs on the PF inside the VM.
> This is to have a flexibility and better manageability of VFs inside the VM (for example, kubernetes worker node) itself and not on the host.
> 
> 
> The ethernet PCI device is seen inside the VM and bound to i40e driver. But I don’t see an option to create VFs. i.e. sriov_numvfs file is not seen under /sys/devices/pci0000:00/0000:00:02.1/0000:02:00.0 directory.

Hi Periyasamy,

The PCI space itself is not passed-through, it is completely fake and
generated by QEMU.

Do you know if anyone has ever gotten what you're trying to do to
work?  I don't think you can do what you're trying to do with using a
VM to spawn SR-IOV devices, at least I've not heard of it working.

Basically you have a scoping problem.  At it's core, the PCI space is
owned by the host, not the VM, and the hardware is literally in the
host PCI device space no matter where you pass it to.  The hardware
actually creates (starts decoding addresses and PCI space for) the new
PCI devices when you enable the device via sriov_numvfs.  Those devices
will appear in space reserved by the host, for SR-IOV devices to
"appear", but there is no guarantee that memory range will be passed
through to the VF, and again all the VM PCI devices are "fake" PCI
config space, so without some daemon monitoring and adding the devices
via virsh or something, I doubt the VM would ever see them even.


> Host versions:
> OS: Ubuntu 16.04.5 LTS, Kernel: 4.15.0-48-generic, libvirt: 4.0.0, qemu: 2.11.1
> i40e version: 2.1.14-k, firmware-version: 6.01 0x800034a3 1.1747.0
> 
> Guest versions:
> OS: CentOS 7 (Core) Kernel: 3.10.0-862.14.4.el7.x86_64
> i40e version: 2.1.14-k, firmware-version: 6.01 0x800034a3 1.1747.0
> 
> The VM libvirt xml configuration [1], PF configuration at host [2], PF configuration at VM [3] are attached.
> The lspci output line nos. 63-75 related to SRIOV Capabilities in host [2] are missing in VM which looks bit weird.

as per above, the PCI config space is completely virtualized by QEMU.

Hope this helps!
Jesse

Hi,

I’m trying to achieve PF passthrough of 40/10G ethernet interface (i40e) into guest VM running on qemu/kvm hypervisor and then create VFs on the PF inside the VM.
This is to have a flexibility and better manageability of VFs inside the VM (for example, kubernetes worker node) itself and not on the host.


The ethernet PCI device is seen inside the VM and bound to i40e driver. But I don’t see an option to create VFs. i.e. sriov_numvfs file is not seen under /sys/devices/pci0000:00/0000:00:02.1/0000:02:00.0 directory.

Host versions:
OS: Ubuntu 16.04.5 LTS, Kernel: 4.15.0-48-generic, libvirt: 4.0.0, qemu: 2.11.1
i40e version: 2.1.14-k, firmware-version: 6.01 0x800034a3 1.1747.0

Guest versions:
OS: CentOS 7 (Core) Kernel: 3.10.0-862.14.4.el7.x86_64
i40e version: 2.1.14-k, firmware-version: 6.01 0x800034a3 1.1747.0

The VM libvirt xml configuration [1], PF configuration at host [2], PF configuration at VM [3] are attached.
The lspci output line nos. 63-75 related to SRIOV Capabilities in host [2] are missing in VM which looks bit weird.

Could you please look into it and let us know what’s going wrong ?

[1] https://pastebin.ubuntu.com/p/wrDB6T68r3/
[2] https://pastebin.ubuntu.com/p/PW7Z4SxQPt/
[3] https://pastebin.ubuntu.com/p/JcmpMY48D6/

Thanks,
Periyasamy