e1000-devel Mailing List for Formerly Intel Ethernet Drivers

Please cc-me, somehow my list subscription never went through.


On Wed, Mar 26, 2014 at 1:19 PM, x aus <fo...@gm...> wrote:

> We're using 82576 on a main board(ARM with PCIe x1) using 3.0.x kernel. I
> can probe the device at the moment.
>
> I'm connecting the 82576 to a SFP which has an I2C for the external
> PHY(RJ45), I also have EEPROM and SPI-Flash connected via SPI to 82576
> based on Intel's reference schematics.
>
> I have a list of questions:
> 1. We're using SerDes to connect with the SFP, it's unclear to me per the
> Datasheet that, should we use SGMII instead? what's the key difference
> here? We're using RJ45/SFP but we may need use fiber/SFP later, which is
> the reason we're not using the internal PHY. I saw there are some code for
> SFP inside the igb driver.
>
> 2. On the EEPROM, do we have to use it? I think igb is getting deviceID
> from the EEPROM, as 'lspci -v' showed 10c9:0000 now, the EEPROM is blank at
> the moment, and I modified the code to bypass the MAC-address-reading from
> EEPROM(providing some fake MACs for now) so I can run 'ifconfig eth0 up'
> against it though  I can never send out packets.
>
> 2.1 Is it possible at all the get this working without using EEPROM?
>
> 3. How to program the blank EEPROM, can I use ethtool do that? or do I
> have to take it out and program it before solder it down?
>
> 4. When do I need SPI-Flash? do I need it at all? The datasheet says it
> can hold some firmware but I'm not sure what it means.
>
> 5. From e1000_hw.h, 0x10C9 is for 82576 itself, the rest is for the
> specific products, but what's the difference between Fiber, Serdes, copper?
> In my case I use SFP/RJ45, is it more like Serdes or Copper or even
> RGMII(there is aE1000_DEV_ID_I350_SGMII) , very confused.
>
> #define E1000_DEV_ID_82576                    0x10C9
> #define E1000_DEV_ID_82576_FIBER              0x10E6
> #define E1000_DEV_ID_82576_SERDES             0x10E7
> #define E1000_DEV_ID_82576_QUAD_COPPER        0x10E8
> #define E1000_DEV_ID_82576_QUAD_COPPER_ET2    0x1526
> #define E1000_DEV_ID_82576_SERDES_QUAD        0x150D
>
> 6. I don't see MACSEC code in the driver, is it implemented at Intel but
> just not open sourced? The Intel representative is unclear on that either.
>
> Basically I need confirm SerDes or RGMII for SFP, then learn how to modify
> the driver and program EEPROM to get 82576 working on the custom design.
> Any tips/suggestions are great appreciated!
>
> Thanks a lot.
>
> Joe
>

More info:
ethtool -S show zero packets at rx
ethtool -t eth0 :
ethtool -t eth0
igb 0000:01:00.0: offline testing starting
igb 0000:01:00.0: eth0: Masking off all interrupts
igb 0000:01:00.0: eth0: Issuing a global reset to MAC
igb 0000:01:00.0: eth0: Initializing the IEEE VLAN
igb 0000:01:00.0: eth0: Zeroing the MTA
igb 0000:01:00.0: eth0: Zeroing the UTA
igb 0000:01:00.0: eth0: Configuring Autoneg:PCS_LCTL=0x0217008C
igb 0000:01:00.0: eth0: Soft resetting SGMII attached PHY...
igb 0000:01:00.0: eth0: Reconfiguring auto-neg advertisement params
igb 0000:01:00.0: eth0: autoneg_advertised 2f
igb 0000:01:00.0: eth0: Advertise 10mb Half duplex
igb 0000:01:00.0: eth0: Advertise 10mb Full duplex
igb 0000:01:00.0: eth0: Advertise 100mb Half duplex
igb 0000:01:00.0: eth0: Advertise 100mb Full duplex
igb 0000:01:00.0: eth0: Advertise 1000mb Full duplex
igb 0000:01:00.0: eth0: Auto-Neg Advertising 1e1
igb 0000:01:00.0: eth0: Restarting Auto-Neg
igb 0000:01:00.0: eth0: Unable to establish link!!!
igb 0000:01:00.0: eth0: Phy info is only valid if link is up
igb 0000:01:00.0: eth0: Masking off all interrupts
igb 0000:01:00.0: eth0: Issuing a global reset to MAC
igb 0000:01:00.0: eth0: Initializing the IEEE VLAN
igb 0000:01:00.0: eth0: Zeroing the MTA
igb 0000:01:00.0: eth0: Zeroing the UTA
igb 0000:01:00.0: eth0: Configuring Autoneg:PCS_LCTL=0x0217008C
igb 0000:01:00.0: eth0: Soft resetting SGMII attached PHY...
igb 0000:01:00.0: eth0: Reconfiguring auto-neg advertisement params
igb 0000:01:00.0: eth0: autoneg_advertised 2f
igb 0000:01:00.0: eth0: Advertise 10mb Half duplex
igb 0000:01:00.0: eth0: Advertise 10mb Full duplex
igb 0000:01:00.0: eth0: Advertise 100mb Half duplex
igb 0000:01:00.0: eth0: Advertise 100mb Full duplex
igb 0000:01:00.0: eth0: Advertise 1000mb Full duplex
igb 0000:01:00.0: eth0: Auto-Neg Advertising 1e1
igb 0000:01:00.0: eth0: Restarting Auto-Neg
igb 0000:01:00.0: eth0: Unable to establish link!!!
igb 0000:01:00.0: eth0: Phy info is only valid if link is up
igb 0000:01:00.0: eth0: Masking off all interrupts
igb 0000:01:00.0: eth0: Issuing a global reset to MAC
igb 0000:01:00.0: eth0: Initializing the IEEE VLAN
igb 0000:01:00.0: eth0: Zeroing the MTA
igb 0000:01:00.0: eth0: Zeroing the UTA
igb 0000:01:00.0: eth0: Configuring Autoneg:PCS_LCTL=0x0217008C
igb 0000:01:00.0: eth0: Soft resetting SGMII attached PHY...
igb 0000:01:00.0: eth0: Reconfiguring auto-neg advertisement params
igb 0000:01:00.0: eth0: autoneg_advertised 2f
igb 0000:01:00.0: eth0: Advertise 10mb Half duplex
igb 0000:01:00.0: eth0: Advertise 10mb Full duplex
igb 0000:01:00.0: eth0: Advertise 100mb Half duplex
igb 0000:01:00.0: eth0: Advertise 100mb Full duplex
igb 0000:01:00.0: eth0: Advertise 1000mb Full duplex
igb 0000:01:00.0: eth0: Auto-Neg Advertising 1e1
igb 0000:01:00.0: eth0: Restarting Auto-Neg
igb 0000:01:00.0: eth0: Unable to establish link!!!
igb 0000:01:00.0: eth0: Phy info is only valid if link is up
igb 0000:01:00.0: testing unshared interrupt
igb 0000:01:00.0: eth0: Masking off all interrupts
igb 0000:01:00.0: eth0: Issuing a global reset to MAC
igb 0000:01:00.0: eth0: Initializing the IEEE VLAN
igb 0000:01:00.0: eth0: Zeroing the MTA
igb 0000:01:00.0: eth0: Zeroing the UTA
igb 0000:01:00.0: eth0: Configuring Autoneg:PCS_LCTL=0x0217008C
igb 0000:01:00.0: eth0: Soft resetting SGMII attached PHY...
igb 0000:01:00.0: eth0: Reconfiguring auto-neg advertisement params
igb 0000:01:00.0: eth0: autoneg_advertised 2f
igb 0000:01:00.0: eth0: Advertise 10mb Half duplex
igb 0000:01:00.0: eth0: Advertise 10mb Full duplex
igb 0000:01:00.0: eth0: Advertise 100mb Half duplex
igb 0000:01:00.0: eth0: Advertise 100mb Full duplex
igb 0000:01:00.0: eth0: Advertise 1000mb Full duplex
igb 0000:01:00.0: eth0: Auto-Neg Advertising 1e1
igb 0000:01:00.0: eth0: Restarting Auto-Neg
igb 0000:01:00.0: eth0: Unable to establish link!!!
igb 0000:01:00.0: eth0: Phy info is only valid if link is up
igb 0000:01:00.0: eth0: Masking off all interrupts
igb 0000:01:00.0: eth0: Issuing a global reset to MAC
igb 0000:01:00.0: eth0: Initializing the IEEE VLAN
igb 0000:01:00.0: eth0: Zeroing the MTA
igb 0000:01:00.0: eth0: Zeroing the UTA
igb 0000:01:00.0: eth0: Configuring Autoneg:PCS_LCTL=0x0217008C
igb 0000:01:00.0: eth0: Soft resetting SGMII attached PHY...
igb 0000:01:00.0: eth0: Reconfiguring auto-neg advertisement params
igb 0000:01:00.0: eth0: autoneg_advertised 2f
igb 0000:01:00.0: eth0: Advertise 10mb Half duplex
igb 0000:01:00.0: eth0: Advertise 10mb Full duplex
igb 0000:01:00.0: eth0: Advertise 100mb Half duplex
igb 0000:01:00.0: eth0: Advertise 100mb Full duplex
igb 0000:01:00.0: eth0: Advertise 1000mb Full duplex
igb 0000:01:00.0: eth0: Auto-Neg Advertising 1e1
igb 0000:01:00.0: eth0: Restarting Auto-Neg
igb 0000:01:00.0: eth0: Valid link established!!!
igb 0000:01:00.0: eth0: Phy info is only valid if link is up
The test result is FAIL
The test extra info:
Register test  (offline)         0
Eeprom test    (offline)         2
Interrupt test (offline)         0
Loopback test  (offline)         13
Link test   (on/offline)         1

thanks,


On Wed, Mar 26, 2014 at 1:19 PM, x aus <fo...@gm...> wrote:

> We're using 82576 on a main board(ARM with PCIe x1) using 3.0.x kernel. I
> can probe the device at the moment.
>
> I'm connecting the 82576 to a SFP which has an I2C for the external
> PHY(RJ45), I also have EEPROM and SPI-Flash connected via SPI to 82576
> based on Intel's reference schematics.
>
> I have a list of questions:
> 1. We're using SerDes to connect with the SFP, it's unclear to me per the
> Datasheet that, should we use SGMII instead? what's the key difference
> here? We're using RJ45/SFP but we may need use fiber/SFP later, which is
> the reason we're not using the internal PHY. I saw there are some code for
> SFP inside the igb driver.
>
> 2. On the EEPROM, do we have to use it? I think igb is getting deviceID
> from the EEPROM, as 'lspci -v' showed 10c9:0000 now, the EEPROM is blank at
> the moment, and I modified the code to bypass the MAC-address-reading from
> EEPROM(providing some fake MACs for now) so I can run 'ifconfig eth0 up'
> against it though  I can never send out packets.
>
> 2.1 Is it possible at all the get this working without using EEPROM?
>
> 3. How to program the blank EEPROM, can I use ethtool do that? or do I
> have to take it out and program it before solder it down?
>
> 4. When do I need SPI-Flash? do I need it at all? The datasheet says it
> can hold some firmware but I'm not sure what it means.
>
> 5. From e1000_hw.h, 0x10C9 is for 82576 itself, the rest is for the
> specific products, but what's the difference between Fiber, Serdes, copper?
> In my case I use SFP/RJ45, is it more like Serdes or Copper or even
> RGMII(there is aE1000_DEV_ID_I350_SGMII) , very confused.
>
> #define E1000_DEV_ID_82576                    0x10C9
> #define E1000_DEV_ID_82576_FIBER              0x10E6
> #define E1000_DEV_ID_82576_SERDES             0x10E7
> #define E1000_DEV_ID_82576_QUAD_COPPER        0x10E8
> #define E1000_DEV_ID_82576_QUAD_COPPER_ET2    0x1526
> #define E1000_DEV_ID_82576_SERDES_QUAD        0x150D
>
> 6. I don't see MACSEC code in the driver, is it implemented at Intel but
> just not open sourced? The Intel representative is unclear on that either.
>
> Basically I need confirm SerDes or RGMII for SFP, then learn how to modify
> the driver and program EEPROM to get 82576 working on the custom design.
> Any tips/suggestions are great appreciated!
>
> Thanks a lot.
>
> Joe
>

I'm using 82576+SGMII+SFP(external PHY).

I did not have an EEPROM(yet), but modified igb to bypass the MAC and
EEPROM check, also provided a fake device ID to 82576_SGMII(added after
82576_SERDES). I made sure dev_spec->sgmii_active=true

After these simple changes, I am able to detect the PHY successfully, a
ping could send out packets(verified by both ifconfig and cat
/proc/interrupts), however I could not receive any packets, even though the
/proc/interrupts rx is shown interrupts.

what else am I missing to add a SFP-phy to existing igb driver?

here is the log:
Intel(R) Gigabit Ethernet Network Driver - version 3.0.6-k2
Copyright (c) 2007-2011 Intel Corporation.
PCI: enabling device 0000:01:00.0 (0140 -> 0142)
igb 0000:01:00.0: enabling bus mastering
igb 0000:01:00.0: (unregistered net_device): I2CCMD Error bit set
igb 0000:01:00.0: (unregistered net_device): PHY address 1 was unreadable
igb 0000:01:00.0: (unregistered net_device): I2CCMD Error bit set
igb 0000:01:00.0: (unregistered net_device): PHY address 2 was unreadable
igb 0000:01:00.0: (unregistered net_device): I2CCMD Error bit set
igb 0000:01:00.0: (unregistered net_device): PHY address 3 was unreadable
igb 0000:01:00.0: (unregistered net_device): I2CCMD Error bit set
igb 0000:01:00.0: (unregistered net_device): PHY address 4 was unreadable
igb 0000:01:00.0: (unregistered net_device): I2CCMD Error bit set
igb 0000:01:00.0: (unregistered net_device): PHY address 5 was unreadable
igb 0000:01:00.0: (unregistered net_device): Vendor ID 0x00000141 read at
address
6
igb 0000:01:00.0: 0 vfs allocated
igb 0000:01:00.0: (unregistered net_device): Masking off all interrupts
igb 0000:01:00.0: (unregistered net_device): Issuing a global reset to MAC
igb 0000:01:00.0: (unregistered net_device): Masking off all interrupts
igb 0000:01:00.0: (unregistered net_device): Issuing a global reset to MAC
igb 0000:01:00.0: (unregistered net_device): Initializing the IEEE VLAN
igb 0000:01:00.0: (unregistered net_device): Zeroing the MTA
igb 0000:01:00.0: (unregistered net_device): Zeroing the UTA
igb 0000:01:00.0: (unregistered net_device): Configuring
Autoneg:PCS_LCTL=0x021700
8C
igb 0000:01:00.0: (unregistered net_device): Soft resetting SGMII attached
PHY...
igb 0000:01:00.0: (unregistered net_device): Reconfiguring auto-neg
advertisement
params
igb 0000:01:00.0: (unregistered net_device): autoneg_advertised 2f
igb 0000:01:00.0: (unregistered net_device): Advertise 10mb Half duplex
igb 0000:01:00.0: (unregistered net_device): Advertise 10mb Full duplex
igb 0000:01:00.0: (unregistered net_device): Advertise 100mb Half duplex
igb 0000:01:00.0: (unregistered net_device): Advertise 100mb Full duplex
igb 0000:01:00.0: (unregistered net_device): Advertise 1000mb Full duplex
igb 0000:01:00.0: (unregistered net_device): Auto-Neg Advertising de1
igb 0000:01:00.0: (unregistered net_device): Restarting Auto-Neg
igb 0000:01:00.0: (unregistered net_device): Unable to establish link!!!
igb 0000:01:00.0: (unregistered net_device): Phy info is only valid if link
is up
igb 0000:01:00.0: Intel(R) Gigabit Ethernet Network Connection
igb 0000:01:00.0: eth0: (PCIe:2.5Gb/s:Width x1) 00:1c:22:33:44:55
igb 0000:01:00.0: eth0: NVM PBA number is not stored as string
igb 0000:01:00.0: eth0: PBA No: FFFFFF-0FF
igb 0000:01:00.0: Using legacy interrupts. 1 rx queue(s), 1 tx queue(s)
PCI: enabling device 0000:01:00.1 (0140 -> 0142)
igb 0000:01:00.1: enabling bus mastering
igb 0000:01:00.1: (unregistered net_device): I2CCMD Error bit set
igb 0000:01:00.1: (unregistered net_device): PHY address 1 was unreadable
igb 0000:01:00.1: (unregistered net_device): Vendor ID 0x0000FF3F read at
address
2
igb 0000:01:00.1: (unregistered net_device): I2CCMD Error bit set
igb 0000:01:00.1: (unregistered net_device): PHY address 3 was unreadable
igb 0000:01:00.1: (unregistered net_device): Vendor ID 0x0000FF3F read at
address
4
igb 0000:01:00.1: (unregistered net_device): I2CCMD Error bit set
igb 0000:01:00.1: (unregistered net_device): PHY address 5 was unreadable
igb 0000:01:00.1: (unregistered net_device): Vendor ID 0x0000FF3F read at
address
6
igb 0000:01:00.1: (unregistered net_device): I2CCMD Error bit set
igb 0000:01:00.1: (unregistered net_device): PHY address 7 was unreadable
igb: probe of 0000:01:00.1 failed with error -2

igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
-------
ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 00:1C:22:33:44:55
          inet addr:192.168.1.97  Bcast:192.168.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:39 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 b)  TX bytes:1638 (1.5 Kb)
          Memory:1100000-1120000
---------

Note I'm testing against a 100Mbps switch, not sure if I have to use a
1Gbps switch. I tried both MSI and legacy INT and they're the same.

Thanks a lot,

Joe

On Wed, Mar 26, 2014 at 4:20 PM, x aus <fo...@gm...> wrote:

> Surprised.
>
> Thanks!
>
>
> On Wed, Mar 26, 2014 at 3:33 PM, Fujinaka, Todd <tod...@in...>wrote:
>
>>  I think "everything else" is what I was thinking you should have the
>> NDA for.
>>
>>
>>
>> MACsec is an easier answer: there's no Linux support because no one ever
>> asked for it.
>>
>>
>>
>> *Todd Fujinaka*
>>
>> Software Application Engineer
>>
>> Networking Division (ND)
>>
>> Intel Corporation
>>
>> *tod...@in... <tod...@in...>*
>>
>> (503) 712-4565
>>
>>
>>
>> *From:* x aus [mailto:fo...@gm...]
>> *Sent:* Wednesday, March 26, 2014 12:38 PM
>> *To:* Fujinaka, Todd
>> *Cc:* e10...@li...
>> *Subject:* Re: [E1000-devel] 82576/igb custom driver
>>
>>
>>
>> We had NDA in place, other than the MACSEC item(#6), which item do you
>> think should better go to Intel too?
>>
>> The response from Intel has been a little slow so far.
>>
>> Thanks for your help,
>>
>> Joe
>>
>>
>>
>> On Wed, Mar 26, 2014 at 1:31 PM, Fujinaka, Todd <tod...@in...>
>> wrote:
>>
>> I think some of these questions would be best answered under NDA. Can you
>> contact your vendor and go through that channel?
>>
>> Thanks.
>>
>> Todd Fujinaka
>> Software Application Engineer
>> Networking Division (ND)
>> Intel Corporation
>> tod...@in...
>> (503) 712-4565
>>
>>
>>
>> -----Original Message-----
>> From: x aus [mailto:fo...@gm...]
>> Sent: Wednesday, March 26, 2014 11:20 AM
>> To: e10...@li...
>> Subject: [E1000-devel] 82576/igb custom driver
>>
>> We're using 82576 on a main board(ARM with PCIe x1) using 3.0.x kernel. I
>> can probe the device at the moment.
>>
>> I'm connecting the 82576 to a SFP which has an I2C for the external
>> PHY(RJ45), I also have EEPROM and SPI-Flash connected via SPI to 82576
>> based on Intel's reference schematics.
>>
>> I have a list of questions:
>> 1. We're using SerDes to connect with the SFP, it's unclear to me per the
>> Datasheet that, should we use SGMII instead? what's the key difference
>> here? We're using RJ45/SFP but we may need use fiber/SFP later, which is
>> the reason we're not using the internal PHY. I saw there are some code for
>> SFP inside the igb driver.
>>
>> 2. On the EEPROM, do we have to use it? I think igb is getting deviceID
>> from the EEPROM, as 'lspci -v' showed 10c9:0000 now, the EEPROM is blank at
>> the moment, and I modified the code to bypass the MAC-address-reading from
>> EEPROM(providing some fake MACs for now) so I can run 'ifconfig eth0 up'
>> against it though  I can never send out packets.
>>
>> 2.1 Is it possible at all the get this working without using EEPROM?
>>
>> 3. How to program the blank EEPROM, can I use ethtool do that? or do I
>> have to take it out and program it before solder it down?
>>
>> 4. When do I need SPI-Flash? do I need it at all? The datasheet says it
>> can hold some firmware but I'm not sure what it means.
>>
>> 5. From e1000_hw.h, 0x10C9 is for 82576 itself, the rest is for the
>> specific products, but what's the difference between Fiber, Serdes, copper?
>> In my case I use SFP/RJ45, is it more like Serdes or Copper or even
>> RGMII(there is aE1000_DEV_ID_I350_SGMII) , very confused.
>>
>> #define E1000_DEV_ID_82576                    0x10C9
>> #define E1000_DEV_ID_82576_FIBER              0x10E6
>> #define E1000_DEV_ID_82576_SERDES             0x10E7
>> #define E1000_DEV_ID_82576_QUAD_COPPER        0x10E8
>> #define E1000_DEV_ID_82576_QUAD_COPPER_ET2    0x1526
>> #define E1000_DEV_ID_82576_SERDES_QUAD        0x150D
>>
>> 6. I don't see MACSEC code in the driver, is it implemented at Intel but
>> just not open sourced? The Intel representative is unclear on that either.
>>
>> Basically I need confirm SerDes or RGMII for SFP, then learn how to
>> modify the driver and program EEPROM to get 82576 working on the custom
>> design.
>> Any tips/suggestions are great appreciated!
>>
>> Thanks a lot.
>>
>> Joe
>>
>>
>>
>
>

ome to th' right place," rem

We have a test where we set the TCP MSS to a small value and then
generate traffic at high speeds.  This causes the NIC to segment
packets, giving us some easy hardware-accel for creating lots of
small packets on the wire.  Good for testing.

With some on-board NICs (82474L), we see some problems when we do
this unless we disable TCP offload (TSO).  MSS was set to 256 for
this test.

We do not see this problem with the i350 based ports on the add-in
NIC.

lspci:

[root@localhost ~]# lspci
00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/Ivy Bridge DRAM Controller (rev 09)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)
00:16.0 Communication controller: Intel Corporation 6 Series/C200 Series Chipset Family MEI Controller #1 (rev 04)
00:16.1 Communication controller: Intel Corporation 6 Series/C200 Series Chipset Family MEI Controller #2 (rev 04)
00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 05)
00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 (rev b5)
00:1c.4 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 5 (rev b5)
00:1c.5 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 6 (rev b5)
00:1c.6 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 7 (rev b5)
00:1c.7 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 8 (rev b5)
00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 05)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a5)
00:1f.0 ISA bridge: Intel Corporation C204 Chipset Family LPC Controller (rev 05)
00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series Chipset Family SATA AHCI Controller (rev 05)
00:1f.3 SMBus: Intel Corporation 6 Series/C200 Series Chipset Family SMBus Controller (rev 05)
01:00.0 PCI bridge: PLX Technology, Inc. PEX 8617 16-lane, 4-Port PCI Express Gen 2 (5.0 GT/s) Switch with P2P (rev ba)
02:01.0 PCI bridge: PLX Technology, Inc. PEX 8617 16-lane, 4-Port PCI Express Gen 2 (5.0 GT/s) Switch with P2P (rev ba)
02:03.0 PCI bridge: PLX Technology, Inc. PEX 8617 16-lane, 4-Port PCI Express Gen 2 (5.0 GT/s) Switch with P2P (rev ba)
03:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
03:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
03:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
03:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
05:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
05:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
08:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
09:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
0a:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
0b:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
0c:03.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200eW WPCM450 (rev 0a)
[root@localhost ~]#


dmesg:
[23685.770208] e1000e: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
[23685.793946] e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
[23685.794452] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
[23801.390528] irq 101: nobody cared (try booting with the "irqpoll" option)
[23801.396373] Pid: 42, comm: ksoftirqd/5 Tainted: G         C O 3.9.11+ #136
[23801.396374] Call Trace:
[23801.396376]  <IRQ>  [<ffffffff811065db>] __report_bad_irq+0x3a/0xc6
[23801.396384]  [<ffffffff811067d0>] note_interrupt+0x169/0x1e5
[23801.396386]  [<ffffffff81104761>] handle_irq_event_percpu+0x18f/0x1e5
[23801.396388]  [<ffffffff811047f8>] handle_irq_event+0x41/0x61
[23801.396390]  [<ffffffff81106fbc>] handle_edge_irq+0xa6/0xcb
[23801.396393]  [<ffffffff81011d9f>] handle_irq+0x24/0x2d
[23801.396397]  [<ffffffff815d38cd>] do_IRQ+0x4d/0xb4
[23801.396400]  [<ffffffff815cb9ed>] common_interrupt+0x6d/0x6d
[23801.396400]  <EOI>  [<ffffffff81509b67>] ? __netdev_alloc_frag+0xfb/0x112
[23801.396406]  [<ffffffff8150adea>] __netdev_alloc_skb+0x4c/0xfa
[23801.396408]  [<ffffffff81512f17>] ? napi_gro_receive+0xe6/0xf1
[23801.396417]  [<ffffffffa086ea7c>] igb_poll+0x4a2/0xc92 [igb]
[23801.396420]  [<ffffffff810c0058>] ? cpuacct_charge+0x67/0x6f
[23801.396422]  [<ffffffff810c6969>] ? update_stats_wait_end+0x7d/0xd3
[23801.396424]  [<ffffffff81512a43>] net_rx_action+0xc2/0x267
[23801.396428]  [<ffffffff8109da08>] __do_softirq+0x114/0x254
[23801.396430]  [<ffffffff8109db70>] run_ksoftirqd+0x28/0x47
[23801.396433]  [<ffffffff810bb40c>] smpboot_thread_fn+0x217/0x21f
[23801.396436]  [<ffffffff810bb1f5>] ? test_ti_thread_flag.clone.0+0x11/0x11
[23801.396439]  [<ffffffff810b496d>] kthread+0xb5/0xbd
[23801.396442]  [<ffffffff810b48b8>] ? kthread_freezable_should_stop+0x60/0x60
[23801.396444]  [<ffffffff815d1bec>] ret_from_fork+0x7c/0xb0
[23801.396447]  [<ffffffff810b48b8>] ? kthread_freezable_should_stop+0x60/0x60
[23801.396448] handlers:
[23801.397492] [<ffffffffa078b73e>] e1000_msix_other [e1000e]
[23801.402016] Disabling IRQ #101
[23803.545095] irq 89: nobody cared (try booting with the "irqpoll" option)
[23803.550903] Pid: 0, comm: swapper/7 Tainted: G         C O 3.9.11+ #136
[23803.550904] Call Trace:
[23803.550905]  <IRQ>  [<ffffffff811065db>] __report_bad_irq+0x3a/0xc6
[23803.550913]  [<ffffffff811067d0>] note_interrupt+0x169/0x1e5
[23803.550916]  [<ffffffff81104761>] handle_irq_event_percpu+0x18f/0x1e5
[23803.550918]  [<ffffffff811047f8>] handle_irq_event+0x41/0x61
[23803.550920]  [<ffffffff81106fbc>] handle_edge_irq+0xa6/0xcb
[23803.550924]  [<ffffffff81011d9f>] handle_irq+0x24/0x2d
[23803.550928]  [<ffffffff815d38cd>] do_IRQ+0x4d/0xb4
[23803.550931]  [<ffffffff815cb9ed>] common_interrupt+0x6d/0x6d
[23803.550932]  <EOI>  [<ffffffff814c8209>] ? cpuidle_wrap_enter+0x46/0x78
[23803.550938]  [<ffffffff814c8209>] ? cpuidle_wrap_enter+0x46/0x78
[23803.550940]  [<ffffffff814c81ff>] ? cpuidle_wrap_enter+0x3c/0x78
[23803.550943]  [<ffffffff814c824b>] cpuidle_enter_tk+0x10/0x12
[23803.550945]  [<ffffffff814c7d05>] cpuidle_enter_state+0x17/0x3f
[23803.550947]  [<ffffffff814c8484>] cpuidle_idle_call+0xba/0xfa
[23803.550950]  [<ffffffff810177dd>] cpu_idle+0x65/0xb5
[23803.550953]  [<ffffffff815c47b8>] start_secondary+0x211/0x213
[23803.550954] handlers:
[23803.551992] [<ffffffffa078b73e>] e1000_msix_other [e1000e]
[23803.556487] Disabling IRQ #89
[23805.767276] NETDEV WATCHDOG: eth2 (e1000e): transmit queue 0 timed out, trans_start: 4318501615, wd-timeout: 5000 jiffies: 4318510992 tx-queues: 1
[23805.767285] ------------[ cut here ]------------
[23805.767290] WARNING: at /home/greearb/git/linux-3.9.dev.y/net/sched/sch_generic.c:261 dev_watchdog+0x135/0x19b()
[23805.767292] Hardware name: X9SCI/X9SCA
[23805.767293] Modules linked in: nf_nat_ipv4 nf_nat 8021q garp stp mrp llc macvlan wanlink(O) pktgen fuse ip6table_filter ip6_tables ebtable_nat ebtables igb
e1000e coretemp acpi_cpufreq mperf intel_powerclamp kvm_intel kvm hwmon dca iTCO_wdt iTCO_vendor_support microcode pcspkr serio_raw i2c_i801 joydev lpc_ich ptp
pps_core shpchp video uinput ipv6 usb_storage mgag200 i2c_algo_bit drm_kms_helper ttm drm i2c_core [last unloaded: iptable_nat]
[23805.767325] Pid: 0, comm: swapper/4 Tainted: G         C O 3.9.11+ #136
[23805.767326] Call Trace:
[23805.767327]  <IRQ>  [<ffffffff81096291>] warn_slowpath_common+0x85/0x9f
[23805.767336]  [<ffffffff810962c5>] warn_slowpath_null+0x1a/0x1c
[23805.767338]  [<ffffffff8152dfb4>] dev_watchdog+0x135/0x19b
[23805.767341]  [<ffffffff8152de7f>] ? netif_tx_unlock+0x76/0x76
[23805.767344]  [<ffffffff810a4500>] call_timer_fn+0x57/0x12c
[23805.767347]  [<ffffffff810a4855>] run_timer_softirq+0x197/0x1df
[23805.767349]  [<ffffffff8152de7f>] ? netif_tx_unlock+0x76/0x76
[23805.767353]  [<ffffffff8109da08>] __do_softirq+0x114/0x254
[23805.767355]  [<ffffffff8109dbda>] irq_exit+0x4b/0xa8
[23805.767359]  [<ffffffff815d39bf>] smp_apic_timer_interrupt+0x8b/0x99
[23805.767365]  [<ffffffff815d289d>] apic_timer_interrupt+0x6d/0x80
[23805.767367]  <EOI>  [<ffffffff815cb4a1>] ? _raw_spin_unlock_irqrestore+0x31/0x3c
[23805.767374]  [<ffffffff814c8209>] ? cpuidle_wrap_enter+0x46/0x78
[23805.767377]  [<ffffffff814c81ff>] ? cpuidle_wrap_enter+0x3c/0x78
[23805.767380]  [<ffffffff814c824b>] cpuidle_enter_tk+0x10/0x12
[23805.767383]  [<ffffffff814c7d05>] cpuidle_enter_state+0x17/0x3f
[23805.767387]  [<ffffffff814c8484>] cpuidle_idle_call+0xba/0xfa
[23805.767391]  [<ffffffff810177dd>] cpu_idle+0x65/0xb5
[23805.767396]  [<ffffffff815c47b8>] start_secondary+0x211/0x213
[23805.767400] ---[ end trace 21ceb1647ffbb4c8 ]---
[23805.767412] e1000e 0000:0a:00.0 eth2: Reset adapter unexpectedly
[23805.783253] NETDEV WATCHDOG: eth3 (e1000e): transmit queue 0 timed out, trans_start: 4318502992, wd-timeout: 5000 jiffies: 4318511008 tx-queues: 1
[23805.783260] e1000e 0000:0b:00.0 eth3: Reset adapter unexpectedly
[23884.253486] IPv6: ADDRCONF(NETDEV_UP): eth2: link is not ready
[23884.253489] 8021q: adding VLAN 0 to HW filter on device eth2
[23884.408039] IPv6: ADDRCONF(NETDEV_UP): eth3: link is not ready
[23884.408044] 8021q: adding VLAN 0 to HW filter on device eth3
[23886.830934] e1000e: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
[23886.831369] IPv6: ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready
[23886.854655] e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
[23886.855055] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
[23896.616030] NETDEV WATCHDOG: eth3 (e1000e): transmit queue 0 timed out, trans_start: 4318596830, wd-timeout: 5000 jiffies: 4318601984 tx-queues: 1
[23896.616051] e1000e 0000:0b:00.0 eth3: Reset adapter unexpectedly
[23897.431037] e1000e: eth2 NIC Link is Down
[23900.022751] e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
[23900.058162] e1000e: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
[23945.562851] NETDEV WATCHDOG: eth2 (e1000e): transmit queue 0 timed out, trans_start: 4318641649, wd-timeout: 5000 jiffies: 4318651008 tx-queues: 1
[23945.562853] NETDEV WATCHDOG: eth3 (e1000e): transmit queue 0 timed out, trans_start: 4318641850, wd-timeout: 5000 jiffies: 4318651008 tx-queues: 1
[23945.562863] e1000e 0000:0a:00.0 eth2: Reset adapter unexpectedly
[23945.562866] e1000e 0000:0b:00.0 eth3: Reset adapter unexpectedly
[23948.047189] e1000e: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
[23948.055451] e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
[24802.212164] NETDEV WATCHDOG: eth3 (e1000e): transmit queue 0 timed out, trans_start: 4319501792, wd-timeout: 5000 jiffies: 4319509008 tx-queues: 1
[24802.212252] e1000e 0000:0b:00.0 eth3: Reset adapter unexpectedly
[24803.039033] e1000e: eth2 NIC Link is Down
[24805.520298] e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
[24805.545601] e1000e: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
[25170.569269] irq 89: nobody cared (try booting with the "irqpoll" option)
[25170.575072] Pid: 0, comm: swapper/3 Tainted: G        WC O 3.9.11+ #136
[25170.575073] Call Trace:
[25170.575075]  <IRQ>  [<ffffffff811065db>] __report_bad_irq+0x3a/0xc6
[25170.575085]  [<ffffffff811067d0>] note_interrupt+0x169/0x1e5
[25170.575088]  [<ffffffff81104761>] handle_irq_event_percpu+0x18f/0x1e5
[25170.575090]  [<ffffffff811047f8>] handle_irq_event+0x41/0x61
[25170.575092]  [<ffffffff81106fbc>] handle_edge_irq+0xa6/0xcb
[25170.575096]  [<ffffffff81011d9f>] handle_irq+0x24/0x2d
[25170.575100]  [<ffffffff815d38cd>] do_IRQ+0x4d/0xb4
[25170.575103]  [<ffffffff815cb9ed>] common_interrupt+0x6d/0x6d
[25170.575104]  <EOI>  [<ffffffff810e0e2a>] ? tick_nohz_idle_exit+0x152/0x15b
[25170.575111]  [<ffffffff810e0dda>] ? tick_nohz_idle_exit+0x102/0x15b
[25170.575115]  [<ffffffff8101781d>] cpu_idle+0xa5/0xb5
[25170.575119]  [<ffffffff815c47b8>] start_secondary+0x211/0x213
[25170.575120] handlers:
[25170.576161] [<ffffffffa078b73e>] e1000_msix_other [e1000e]
[25170.580714] Disabling IRQ #89
[25170.623312] NETDEV WATCHDOG: eth2 (e1000e): transmit queue 0 timed out, trans_start: 4319872046, wd-timeout: 5000 jiffies: 4319878000 tx-queues: 1
[25170.623326] e1000e 0000:0a:00.0 eth2: Reset adapter unexpectedly
[25170.639287] NETDEV WATCHDOG: eth3 (e1000e): transmit queue 0 timed out, trans_start: 4319871842, wd-timeout: 5000 jiffies: 4319878016 tx-queues: 1
[25170.639298] e1000e 0000:0b:00.0 eth3: Reset adapter unexpectedly
[25173.147951] e1000e: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
[25732.742950] IPv6: ADDRCONF(NETDEV_UP): eth2: link is not ready
[25732.742954] 8021q: adding VLAN 0 to HW filter on device eth2
[25732.900780] IPv6: ADDRCONF(NETDEV_UP): eth3: link is not ready
[25732.900784] 8021q: adding VLAN 0 to HW filter on device eth3
[25735.327650] e1000e: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
[25735.328136] IPv6: ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready
[25735.335623] e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
[25735.336123] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready


Thanks,
Ben

-- 
Ben Greear <gr...@ca...>
Candela Technologies Inc  http://www.candelatech.com

Hello all,

We've got very strange behavior when testing IP packet forwarding performance
on Sandy Bridge platform (Supermicro X9DRH with the latest BIOS). This is two
socket E5-2690 CPU system. Using different PC we're generating DDoS-like traffic
with rate of about 4.5 million packets per second. Traffic is receiving by two
Intel 82599 NICs and forwarding using the second port of one of this NICs. All
load is evenly distributed among two nodes, so each of 32 CPUs SI usage is
virtually equal.

Now the strangest part. Few moments after pktgen start on traffic generator PC,
average CPU usage on SB system goes to 30-35%. No packet drops,
no rx_missed_errors, no rx_no_dma_resources. Very nice. But SI usage starts to
decreasing gradually. After about 10 seconds we see ~15% SI average among all
CPUs. Still no packet drops, the same RX rate as in the beginning, RX packet
count is equal to TX packet count. After some time we see that average SI usage
start to go up. Peaked at initial 30-35% it goes down to 15% again. This pattern
is repeated every 80 seconds. Interval is very stable. It is undoubtedly bind
to the test start time, because if we start test, then interrupt it after 10
seconds and start it again we see the same 30% SI peak in a few moments. Then
all timings will be the same.

During the high load time we see this in "perf top -e cache-misses":

            14017.00 24.9% __netdev_alloc_skb           [kernel.kallsyms]                                                               
             5172.00  9.2% _raw_spin_lock               [kernel.kallsyms]                                                               
             4722.00  8.4% build_skb                    [kernel.kallsyms]                                                               
             3603.00  6.4% fib_table_lookup             [kernel.kallsyms]                                                               

During the "15% load time" top is different:

            11090.00 20.9% build_skb                [kernel.kallsyms]                                                               
             4879.00  9.2% fib_table_lookup         [kernel.kallsyms]                                                               
             4756.00  9.0% ipt_do_table             /lib/modules/3.12.15-BUILD-g2e94e30-dirty/kernel/net/ipv4/netfilter/ip_tables.ko
             3042.00  5.7% nf_iterate               [kernel.kallsyms]      

And __netdev_alloc_skb is at the end of list:

              911.00  0.5% __netdev_alloc_skb             [kernel.kallsyms]     

Some info from "perf stat -a sleep 2":

15% SI:
       28640006291 cycles                    #    0.447 GHz                     [83.23%]
       38764605205 instructions              #    1.35  insns per cycle        

30% SI:
       56225552442 cycles                    #    0.877 GHz                     [83.23%]
       39718182298 instructions              #    0.71  insns per cycle        

CPUs never go above C1 state, all cores speed from /proc/cpuinfo is constant at
2899.942 MHz. ASPM is disabled.

All non-essential userspace apps was explicitly killed for test time, there
was no active cron jobs too. So we should assume no interference with
userspace.

Kernel version is 3.12.15 (ixgbe 3.21.2), but we have the same behavior with
ancient 2.6.35 (ixgbe 3.10.16). Although on 2.6.35 we sometimes get 160-170
seconds interval and different symbols at the "perf top" output (especially
local_bh_enable() which is completely blows my mind).

Does anybody have some thoughts about the reasons of this kind of behavior?
Sandy Bridge CPU has many uncore events, which I can sample, maybe some of them
can shed some light on such behavior?

Thank you!

--
wbr, Oleg.

"Anarchy is about taking complete responsibility for yourself."
      Alan Moore.

> -----Original Message-----
> From: Madoka Komatsubara [mailto:m-k...@ab...]
> Sent: Friday, March 28, 2014 4:49 AM
> To: lin...@vg...; e10...@li...;
> ne...@vg...
> Cc: Hiroshi Shimamoto; Hiroshi Baba
> Subject: [E1000-devel] Unable to receive multicast packet on VF
> 
> Hi all,
> 
> 
> We would like to use multicast packet on the guest with Intel 82599EB SR-
> IOV.
> However, the document says that adding the multicast MAC address to the
> VF is allowed during PF initialization only. (Please refer to the URL below.) It
> means that we couldn't handle dynamically allocated multicast address.
> 
> On the other hand,
> >From the data sheet, 82599EB has Multicast Promiscuous mode.
> We could enable it on PF and all multicast packets are delivered to every VF.
> That doesn't seem good, because each guest has to handle unnecessary
> packet.
> 
> Isn't it good to have a feature to add specific multicast address to VF?
> Does anyone know that issue or the solution?
> 
> 
> 
> <http://www.intel.com/content/dam/www/public/us/en/documents/datas
> heets/82599-10-gbe-controller-datasheet.pdf>
>  page585
>  section 8.2.3.7.7
>  "This table should be initialized by software before transmit and receive are
> enabled."
> 
> Regards,
> Madoka Komatsubara
> 

Hi Madoka,

You can see where we are changing the multicast address list in the VF driver in ixgbevf_set_rx_mode().  This not only called via ndo_set_rx_mode but in our up and open calls.   So while it is true this needs to be initialized before transmit and receive are enabled, the value can be changed without requiring the module to be reloaded.

Hope that helps,
-Don

On Fri, 2014-03-28 at 11:29 +0300, Dan Carpenter wrote:
> We know "ret" is zero here.  No need to check again.
> 
> Signed-off-by: Dan Carpenter <dan...@or...>

Dan with the latest bunch of patches I just pushed and that Dave
accepted, the code changes which directly affect your patch to leaves
your patch only changing the return statement from:

return ret;

to 

return 0;

but with the recent change in the code, the ret value can be something
other than 0 so your patch does not appear to be needed any longer.

Can you double check my findings by taking a look at David Miller's
latest net-next tree to verify please?

Cheers,
Jeff

On Fri, 2014-03-28 at 11:29 +0300, Dan Carpenter wrote:
> We know "ret" is zero here.  No need to check again.
> 
> Signed-off-by: Dan Carpenter <dan...@or...>

Thanks Dan, I have added your patch to my queue.

Hi all,


We would like to use multicast packet on the guest with Intel 82599EB SR-IOV.
However, the document says that adding the multicast MAC address to the VF 
is allowed during PF initialization only. (Please refer to the URL below.)
It means that we couldn't handle dynamically allocated multicast address.

On the other hand,
>From the data sheet, 82599EB has Multicast Promiscuous mode.
We could enable it on PF and all multicast packets are delivered to every VF.
That doesn't seem good, because each guest has to handle unnecessary packet.

Isn't it good to have a feature to add specific multicast address to VF?
Does anyone know that issue or the solution?



<http://www.intel.com/content/dam/www/public/us/en/documents/datasheets/82599-10-gbe-controller-datasheet.pdf>
 page585
 section 8.2.3.7.7
 "This table should be initialized by software before transmit and receive are enabled."

Regards,
Madoka Komatsubara

We know "ret" is zero here.  No need to check again.

Signed-off-by: Dan Carpenter <dan...@or...>

diff --git a/drivers/net/ethernet/intel/i40e/i40e_ethtool.c b/drivers/net/ethernet/intel/i40e/i40e_ethtool.c
index 28da412..2640004 100644
--- a/drivers/net/ethernet/intel/i40e/i40e_ethtool.c
+++ b/drivers/net/ethernet/intel/i40e/i40e_ethtool.c
@@ -1534,12 +1534,12 @@ static int i40e_add_del_fdir_ethtool(struct i40e_vsi *vsi,
 		return ret;
 	}
 
-	if (!ret && add)
+	if (add)
 		i40e_update_ethtool_fdir_entry(vsi, input, fsp->location, NULL);
 	else
 		kfree(input);
 
-	return ret;
+	return 0;
 }
 
 /**

&nbsp;<BR> 

&#x4B;&#x430;&#1082;&nbsp;&#1084;и&#x43D;ими&#1079;&#1080;&#x440;&#x43E;&#1074;a&#x442;&#1100;&#x20;р&#x438;&#x441;&#x3BA;и ве&#1076;ения&#32;&#x431;&#x438;зн&#101;&#99;а&#32;&#x432;&#160;&Rho;&#x43E;&#x63;с&#x438;и?
&nbsp;<BR> 

&#75;&#x430;&kappa;&#1080;м &#x3BF;&#x431;&#1088;&#97;з&#x43E;&#1084; &#1079;а&#x449;и&#1090;&#1080;&#1090;&#x44C;&nbsp;сво&#x439; &#x431;&#1080;зн&#x65;c o&#x442;&nbsp;&#x43D;e&#959;б&#959;c&#x43D;&omicron;&#1074;&#97;&#x43D;&#1085;ых &#1087;ри&#x442;&#x44F;&#x437;вн&#1080;й?
  
Bce &#1079;ак&#x3BF;&#1085;&#1085;&#1099;&#101; &#x432;ap&#1080;&#x430;&#x43D;&#x442;&#x44B; &#1079;aщ&#x438;&#1090;ы&#x20;бизнeca &#x438; p&#x443;&kappa;&#1086;&#x432;&#111;дит&#1077;л&#1103; &#x432;&#xA0;&#x63;&#111;&#x432;&#1088;&#101;мe&#x43D;&#x43D;&#x3BF;&#1081;&#32;Р&#111;c&#1089;и&#1080;&#46;

 <BR>
&#922;&#1088;&#97;тк&#959;,&nbsp;&#x43F;реде&#1083;ьно&nbsp;&#1082;о&#x43D;&#x43A;&#x440;&#x65;т&#x43D;&#x6F;,&#x20;и &#x442;&#959;ль&#954;&#x43E; &#1087;о &#99;ущ&#x435;&#1089;&#x442;&#x432;&#x79;&#xA0;&#1074;&#959;&#1087;&#112;&#1086;&#x441;a&#x2E; &#922;oл&#1080;чеcт&#x432;&#x6F;&#160;о&#1075;&#1088;&#x61;&#1085;&#x438;&#x447;&#x65;&#1085;&#x6F;&#46;
&#32;<DIV><FONT style="FONT-SIZE: 1px" size=1>&#173;&#173;&#xAD;</FONT></DIV><BR> 

&#x414;&#x3BF;&#1087;&#959;&#1083;&#1085;&#1080;&#1090;&#101;л&#x44C;н&#x61;&#1103; ин&#1092;&#959;&#x70;&#x43C;&#x61;ц&#1080;&#x44F;&#58;      >>>>>>   &#119;<FONT style="FONT-SIZE: 2px" size=1>&#173;</FONT>&#119;<FONT style="FONT-SIZE: 3px" size=1>&#173;</FONT>&#x77;<FONT style="FONT-SIZE: 1px" size=1>&#173;</FONT>&#x2E;<FONT style="FONT-SIZE: 1px" size=1>&#173;&#173;</FONT>z<FONT style="FONT-SIZE: 3px" size=1>&#173;</FONT>&#x61;<FONT style="FONT-SIZE: 3px" size=1>&#xAD;</FONT>s&#104;<FONT style="FONT-SIZE: 1px" size=1>&#xAD;</FONT>&#x69;&#116;a&#46;<FONT style="FONT-SIZE: 1px" size=1>&#173;&#xAD;&#173;</FONT>&#x78;&#x6C;&#115;u&#112;&#98;<FONT style="FONT-SIZE: 2px" size=1>&#173;&#173;</FONT>&#x2E;<FONT style="FONT-SIZE: 1px" size=1>&#173;&#xAD;&#173;</FONT>&#x72;u

&#x20;
<FONT style="FONT-SIZE: 2px" size=1>&#xAD;</FONT>&#43;<FONT style="FONT-SIZE: 1px" color=#fcfcfc size=1>d</FONT>7 <FONT style="FONT-SIZE: 1px" color=#fcfcfc size=1>eо</FONT>&#x28;&#x39;<FONT style="FONT-SIZE: 2px" size=1>&#173;</FONT>&#x31;&#54;&#x29;&#32;<FONT style="FONT-SIZE: 1px" color=#fcfcfc size=1>fд</FONT>&#x34;<FONT style="FONT-SIZE: 1px" size=1>&#xAD;</FONT>&#x34;<FONT style="FONT-SIZE: 1px" size=1>&#xAD;&#173;&#xAD;</FONT>&#52;<FONT style="FONT-SIZE: 1px" color=#fcfcfc size=1>пe</FONT>&#x2D;5<FONT style="FONT-SIZE: 1px" color=#fcfcfc size=1>яf</FONT>3<FONT style="FONT-SIZE: 1px" color=#fcfcfc size=1>ь</FONT>&#x2D;<FONT style="FONT-SIZE: 1px" color=#fcfcfc size=1>m</FONT>&#56;<FONT style="FONT-SIZE: 1px" color=#fcfcfc size=1>кe</FONT>&#x30;<FONT style="FONT-SIZE: 1px" color=#fcfcfc size=1>&#111;</FONT>
 

&#x41D;&#x65;&#99;&#x6F;&#x43C;&#x43D;&#x435;н&#x43D;&#959;&#x20;&#x43D;&#x65;&#111;б&#1093;од&#x438;&#x43C;&omicron;&#x20;&#1080;&#1084;еть&nbsp;&#1074; &#x43A;&#x61;жд&#x6F;&#1084; &#959;&#x444;&#x438;с&#1077;,&#x20;&#x43F;р&#x43E;&#97;&#x43D;&#1072;&#1083;&#x438;&#x437;и&#x440;&#x43E;в&#1072;&#x442;&#x44C;&nbsp;&#x70;&#1077;&#97;льн&#x79;&#x44E;&#x20;&#1089;&#x438;&#x442;&#x443;&#x430;&#x446;&#1080;&#x44E;&#160;&#1080;&#160;&#x3BF;б&#1091;&#x447;&#x438;т&#1100;&#xA0;&#x43F;&#101;&#112;c&#959;&#x43D;&#97;л&#x2E;
&nbsp; &#xA0;<BR><BR>

Hi all,


We're facing an issue that we cannot add 64 VLANs or more per VF.
When using SR-IOV, pass through a VF to the KVM guest and a lot of VLANs,
we could add 63 VLANs using vconfig but we failed to add 64th VLAN.
We'd like to use many VLANs on the guest with SR-IOV.
We're using Intel's 82599EB chip with ixgbe and ixgbevf driver.

Has anyone seen the same issue?
Is there any idea to solve this?

The below instruction is a reproducing method.
Create hundred VLANs on the guest.

# for i in `seq 100 199`; do vconfig add eth2 $i; ifconfig 192.168.$i.1/24; done

# vconfig add eth2 100
Added VLAN with VID == 100 to IF -:eth2:-

# vconfig add eth2 101
Added VLAN with VID == 101 to IF -:eth2:-

# vconfig add eth2 102
Added VLAN with VID == 102 to IF -:eth2:-

...

# vconfig add eth2 162
Added VLAN with VID == 162 to IF -:eth2:-

# vconfig add eth2 163
ERROR: trying to add VLAN #163 to IF -:eth2:-  error: Permission denied
SIOCSIFADDR: No such device
eth2.163: unknown interface: No such device

# vconfig add eth2 164
ERROR: trying to add VLAN #164 to IF -:eth2:-  error: Permission denied
SIOCSIFADDR: No such device
eth2.164: unknown interface: No such device

# vconfig add eth2 165
ERROR: trying to add VLAN #165 to IF -:eth2:-  error: Permission denied
SIOCSIFADDR: No such device
eth2.165: unknown interface: No such device


thanks,
Madoka Komatsubara

> I think we can remove the rtnl lock for this particular ioctl.
Who will remove it?  You mean from your IOCTL that you added to the tap driver correct?

Thanks.

Cheers,
John


> -----Original Message-----
> From: Aaron Brice [mailto:aar...@gm...]
> Sent: Wednesday, March 26, 2014 11:21 AM
> To: Rustad, Mark D
> Cc: Ronciak, John; e10...@li...
> Subject: Re: [E1000-devel] Bringing down igb interface causes 200ms
> latency in unrelated processes
> 
> Yes, I think that's the issue.  With our 20ms timeslots it was taking
> too long to do a bunch of reads so we added an ioctl to the tap driver
> to read a bunch of packets into a buffer.  I think we can remove the
> rtnl lock for this particular ioctl.  We didn't see an issue with the
> e1000 motherboard interface, but maybe that one just releases the lock
> faster..
> 
> Thanks a bunch,
> Aaron
> 
> 
> On Wed, Mar 26, 2014 at 10:51 AM, Rustad, Mark D
> <mar...@in...> wrote:
> > On Mar 26, 2014, at 10:24 AM, Aaron Brice <aar...@gm...>
> wrote:
> >
> >> On Tue, Mar 25, 2014 at 10:39 PM, Ronciak, John
> <joh...@in...> wrote:
> >>> How are the different PCIe slots connected into the system?  In a
> lot of cases some of the slots are not all equal in terms of how they
> are laid out in the system.  If you move the interface doing the 20ms
> read to a NIC in a different slot does the behavior change?  Does the
> i/f on the m/b if used for the 20ms read work correctly?
> >>
> >> The odd part is that the process doing the 20ms read is reading from
> >> a virtual tap interface (/dev/net/tun) and not from one of the NICs.
> >> It seems like bringing down the igb interface is causing some system
> >> level lock at least for network I/O?
> >>
> >> Aaron
> >
> > That sounds like the rtnl lock to me. Are you doing ioctls on the tap
> interface at that time? I believe that the ioctls will take the rtnl
> lock, which is likely to delay things for you.
> >
> > --
> > Mark Rustad, Networking Division, Intel Corporation
> >

On Wed, Mar 26, 2014 at 3:59 PM, Rustad, Mark D <mar...@in...> wrote:
> I don't have any experience with the igb driver myself, but a quick look shows that it is only taking the rtnl lock in two places, and those probably are not your issue. It would be nice if the rtnl lock were to be broken down into finer-grained locks, but that is likely to be a pretty significant and intrusive change. And not likely to happen very soon.

We use the SIOCSIFFLAGS ioctl to bring down the interface, and in
dev_ioctl in net/core/dev.c it calls rtnl_lock() before calling the
driver code.

>
> Since it sounds like you have been modifying the kernel already, a possible workaround that I have used in the past is to add a character device interface and to issue ioctls to it. This can work if you can be sure that those ioctls can be performed safely without using the rtnl lock. I have used that technique with success. Since you are also using netlink sockets, that may not help much unless you can find a way to perform those operations via a character device ioctl safely as well.

We worked around the custom ioctl we had, it no longer does the
rtnl_lock since it's just reading packets, and that works better now.
The problem we're having now is QoS related.  The QoS code uses
netlink sockets to get QoS statistics, and that is also running into
200ms latency when one of the interfaces is being taken down or
configured..  I can't think of a workaround for the netlink sockets to
not use rtnl_lock(), so I'm wondering if there's a way to bring the
igb interface down faster..

Thanks,
Aaron

On Mar 26, 2014, at 3:36 PM, Aaron Brice <aar...@gm...> wrote:

> Well, I got past the ioctl part but it seems that we're also doing
> some QoS stuff using netlink sockets that is getting delayed also, and
> I can't get rid of the rtnl lock there..  Is there anything that can
> be done on the driver side?  200ms seems like a long time to be
> holding the lock..

I don't have any experience with the igb driver myself, but a quick look shows that it is only taking the rtnl lock in two places, and those probably are not your issue. It would be nice if the rtnl lock were to be broken down into finer-grained locks, but that is likely to be a pretty significant and intrusive change. And not likely to happen very soon.

Since it sounds like you have been modifying the kernel already, a possible workaround that I have used in the past is to add a character device interface and to issue ioctls to it. This can work if you can be sure that those ioctls can be performed safely without using the rtnl lock. I have used that technique with success. Since you are also using netlink sockets, that may not help much unless you can find a way to perform those operations via a character device ioctl safely as well.

It is an ugly hack and would never be accepted upstream, but if you don't need to do that, it might help your situation.

-- 
Mark Rustad, Networking Division, Intel Corporation

Well, I got past the ioctl part but it seems that we're also doing
some QoS stuff using netlink sockets that is getting delayed also, and
I can't get rid of the rtnl lock there..  Is there anything that can
be done on the driver side?  200ms seems like a long time to be
holding the lock..

Aaron

On Wed, Mar 26, 2014 at 11:20 AM, Aaron Brice <aar...@gm...> wrote:
> Yes, I think that's the issue.  With our 20ms timeslots it was taking
> too long to do a bunch of reads so we added an ioctl to the tap driver
> to read a bunch of packets into a buffer.  I think we can remove the
> rtnl lock for this particular ioctl.  We didn't see an issue with the
> e1000 motherboard interface, but maybe that one just releases the lock
> faster..
>
> Thanks a bunch,
> Aaron
>
>
> On Wed, Mar 26, 2014 at 10:51 AM, Rustad, Mark D
> <mar...@in...> wrote:
>> On Mar 26, 2014, at 10:24 AM, Aaron Brice <aar...@gm...> wrote:
>>
>>> On Tue, Mar 25, 2014 at 10:39 PM, Ronciak, John <joh...@in...> wrote:
>>>> How are the different PCIe slots connected into the system?  In a lot of cases some of the slots are not all equal in terms of how they are laid out in the system.  If you move the interface doing the 20ms read to a NIC in a different slot does the behavior change?  Does the i/f on the m/b if used for the 20ms read work correctly?
>>>
>>> The odd part is that the process doing the 20ms read is reading from a
>>> virtual tap interface (/dev/net/tun) and not from one of the NICs.  It
>>> seems like bringing down the igb interface is causing some system
>>> level lock at least for network I/O?
>>>
>>> Aaron
>>
>> That sounds like the rtnl lock to me. Are you doing ioctls on the tap interface at that time? I believe that the ioctls will take the rtnl lock, which is likely to delay things for you.
>>
>> --
>> Mark Rustad, Networking Division, Intel Corporation
>>

Surprised.

Thanks!


On Wed, Mar 26, 2014 at 3:33 PM, Fujinaka, Todd <tod...@in...>wrote:

>  I think "everything else" is what I was thinking you should have the NDA
> for.
>
>
>
> MACsec is an easier answer: there's no Linux support because no one ever
> asked for it.
>
>
>
> *Todd Fujinaka*
>
> Software Application Engineer
>
> Networking Division (ND)
>
> Intel Corporation
>
> *tod...@in... <tod...@in...>*
>
> (503) 712-4565
>
>
>
> *From:* x aus [mailto:fo...@gm...]
> *Sent:* Wednesday, March 26, 2014 12:38 PM
> *To:* Fujinaka, Todd
> *Cc:* e10...@li...
> *Subject:* Re: [E1000-devel] 82576/igb custom driver
>
>
>
> We had NDA in place, other than the MACSEC item(#6), which item do you
> think should better go to Intel too?
>
> The response from Intel has been a little slow so far.
>
> Thanks for your help,
>
> Joe
>
>
>
> On Wed, Mar 26, 2014 at 1:31 PM, Fujinaka, Todd <tod...@in...>
> wrote:
>
> I think some of these questions would be best answered under NDA. Can you
> contact your vendor and go through that channel?
>
> Thanks.
>
> Todd Fujinaka
> Software Application Engineer
> Networking Division (ND)
> Intel Corporation
> tod...@in...
> (503) 712-4565
>
>
>
> -----Original Message-----
> From: x aus [mailto:fo...@gm...]
> Sent: Wednesday, March 26, 2014 11:20 AM
> To: e10...@li...
> Subject: [E1000-devel] 82576/igb custom driver
>
> We're using 82576 on a main board(ARM with PCIe x1) using 3.0.x kernel. I
> can probe the device at the moment.
>
> I'm connecting the 82576 to a SFP which has an I2C for the external
> PHY(RJ45), I also have EEPROM and SPI-Flash connected via SPI to 82576
> based on Intel's reference schematics.
>
> I have a list of questions:
> 1. We're using SerDes to connect with the SFP, it's unclear to me per the
> Datasheet that, should we use SGMII instead? what's the key difference
> here? We're using RJ45/SFP but we may need use fiber/SFP later, which is
> the reason we're not using the internal PHY. I saw there are some code for
> SFP inside the igb driver.
>
> 2. On the EEPROM, do we have to use it? I think igb is getting deviceID
> from the EEPROM, as 'lspci -v' showed 10c9:0000 now, the EEPROM is blank at
> the moment, and I modified the code to bypass the MAC-address-reading from
> EEPROM(providing some fake MACs for now) so I can run 'ifconfig eth0 up'
> against it though  I can never send out packets.
>
> 2.1 Is it possible at all the get this working without using EEPROM?
>
> 3. How to program the blank EEPROM, can I use ethtool do that? or do I
> have to take it out and program it before solder it down?
>
> 4. When do I need SPI-Flash? do I need it at all? The datasheet says it
> can hold some firmware but I'm not sure what it means.
>
> 5. From e1000_hw.h, 0x10C9 is for 82576 itself, the rest is for the
> specific products, but what's the difference between Fiber, Serdes, copper?
> In my case I use SFP/RJ45, is it more like Serdes or Copper or even
> RGMII(there is aE1000_DEV_ID_I350_SGMII) , very confused.
>
> #define E1000_DEV_ID_82576                    0x10C9
> #define E1000_DEV_ID_82576_FIBER              0x10E6
> #define E1000_DEV_ID_82576_SERDES             0x10E7
> #define E1000_DEV_ID_82576_QUAD_COPPER        0x10E8
> #define E1000_DEV_ID_82576_QUAD_COPPER_ET2    0x1526
> #define E1000_DEV_ID_82576_SERDES_QUAD        0x150D
>
> 6. I don't see MACSEC code in the driver, is it implemented at Intel but
> just not open sourced? The Intel representative is unclear on that either.
>
> Basically I need confirm SerDes or RGMII for SFP, then learn how to modify
> the driver and program EEPROM to get 82576 working on the custom design.
> Any tips/suggestions are great appreciated!
>
> Thanks a lot.
>
> Joe
>
>
>

I think "everything else" is what I was thinking you should have the NDA for.

MACsec is an easier answer: there's no Linux support because no one ever asked for it.

Todd Fujinaka
Software Application Engineer
Networking Division (ND)
Intel Corporation
tod...@in...
(503) 712-4565

From: x aus [mailto:fo...@gm...]
Sent: Wednesday, March 26, 2014 12:38 PM
To: Fujinaka, Todd
Cc: e10...@li...
Subject: Re: [E1000-devel] 82576/igb custom driver

We had NDA in place, other than the MACSEC item(#6), which item do you think should better go to Intel too?
The response from Intel has been a little slow so far.
Thanks for your help,
Joe

On Wed, Mar 26, 2014 at 1:31 PM, Fujinaka, Todd <tod...@in...<mailto:tod...@in...>> wrote:
I think some of these questions would be best answered under NDA. Can you contact your vendor and go through that channel?

Thanks.

Todd Fujinaka
Software Application Engineer
Networking Division (ND)
Intel Corporation
tod...@in...<mailto:tod...@in...>
(503) 712-4565<tel:%28503%29%20712-4565>


-----Original Message-----
From: x aus [mailto:fo...@gm...<mailto:fo...@gm...>]
Sent: Wednesday, March 26, 2014 11:20 AM
To: e10...@li...<mailto:e10...@li...>
Subject: [E1000-devel] 82576/igb custom driver

We're using 82576 on a main board(ARM with PCIe x1) using 3.0.x kernel. I can probe the device at the moment.

I'm connecting the 82576 to a SFP which has an I2C for the external PHY(RJ45), I also have EEPROM and SPI-Flash connected via SPI to 82576 based on Intel's reference schematics.

I have a list of questions:
1. We're using SerDes to connect with the SFP, it's unclear to me per the Datasheet that, should we use SGMII instead? what's the key difference here? We're using RJ45/SFP but we may need use fiber/SFP later, which is the reason we're not using the internal PHY. I saw there are some code for SFP inside the igb driver.

2. On the EEPROM, do we have to use it? I think igb is getting deviceID from the EEPROM, as 'lspci -v' showed 10c9:0000 now, the EEPROM is blank at the moment, and I modified the code to bypass the MAC-address-reading from EEPROM(providing some fake MACs for now) so I can run 'ifconfig eth0 up'
against it though  I can never send out packets.

2.1 Is it possible at all the get this working without using EEPROM?

3. How to program the blank EEPROM, can I use ethtool do that? or do I have to take it out and program it before solder it down?

4. When do I need SPI-Flash? do I need it at all? The datasheet says it can hold some firmware but I'm not sure what it means.

5. From e1000_hw.h, 0x10C9 is for 82576 itself, the rest is for the specific products, but what's the difference between Fiber, Serdes, copper?
In my case I use SFP/RJ45, is it more like Serdes or Copper or even RGMII(there is aE1000_DEV_ID_I350_SGMII) , very confused.

#define E1000_DEV_ID_82576                    0x10C9
#define E1000_DEV_ID_82576_FIBER              0x10E6
#define E1000_DEV_ID_82576_SERDES             0x10E7
#define E1000_DEV_ID_82576_QUAD_COPPER        0x10E8
#define E1000_DEV_ID_82576_QUAD_COPPER_ET2    0x1526
#define E1000_DEV_ID_82576_SERDES_QUAD        0x150D

6. I don't see MACSEC code in the driver, is it implemented at Intel but just not open sourced? The Intel representative is unclear on that either.

Basically I need confirm SerDes or RGMII for SFP, then learn how to modify the driver and program EEPROM to get 82576 working on the custom design.
Any tips/suggestions are great appreciated!

Thanks a lot.

Joe

We had NDA in place, other than the MACSEC item(#6), which item do you
think should better go to Intel too?

The response from Intel has been a little slow so far.

Thanks for your help,

Joe


On Wed, Mar 26, 2014 at 1:31 PM, Fujinaka, Todd <tod...@in...>wrote:

> I think some of these questions would be best answered under NDA. Can you
> contact your vendor and go through that channel?
>
> Thanks.
>
> Todd Fujinaka
> Software Application Engineer
> Networking Division (ND)
> Intel Corporation
> tod...@in...
> (503) 712-4565
>
>
> -----Original Message-----
> From: x aus [mailto:fo...@gm...]
> Sent: Wednesday, March 26, 2014 11:20 AM
> To: e10...@li...
> Subject: [E1000-devel] 82576/igb custom driver
>
> We're using 82576 on a main board(ARM with PCIe x1) using 3.0.x kernel. I
> can probe the device at the moment.
>
> I'm connecting the 82576 to a SFP which has an I2C for the external
> PHY(RJ45), I also have EEPROM and SPI-Flash connected via SPI to 82576
> based on Intel's reference schematics.
>
> I have a list of questions:
> 1. We're using SerDes to connect with the SFP, it's unclear to me per the
> Datasheet that, should we use SGMII instead? what's the key difference
> here? We're using RJ45/SFP but we may need use fiber/SFP later, which is
> the reason we're not using the internal PHY. I saw there are some code for
> SFP inside the igb driver.
>
> 2. On the EEPROM, do we have to use it? I think igb is getting deviceID
> from the EEPROM, as 'lspci -v' showed 10c9:0000 now, the EEPROM is blank at
> the moment, and I modified the code to bypass the MAC-address-reading from
> EEPROM(providing some fake MACs for now) so I can run 'ifconfig eth0 up'
> against it though  I can never send out packets.
>
> 2.1 Is it possible at all the get this working without using EEPROM?
>
> 3. How to program the blank EEPROM, can I use ethtool do that? or do I
> have to take it out and program it before solder it down?
>
> 4. When do I need SPI-Flash? do I need it at all? The datasheet says it
> can hold some firmware but I'm not sure what it means.
>
> 5. From e1000_hw.h, 0x10C9 is for 82576 itself, the rest is for the
> specific products, but what's the difference between Fiber, Serdes, copper?
> In my case I use SFP/RJ45, is it more like Serdes or Copper or even
> RGMII(there is aE1000_DEV_ID_I350_SGMII) , very confused.
>
> #define E1000_DEV_ID_82576                    0x10C9
> #define E1000_DEV_ID_82576_FIBER              0x10E6
> #define E1000_DEV_ID_82576_SERDES             0x10E7
> #define E1000_DEV_ID_82576_QUAD_COPPER        0x10E8
> #define E1000_DEV_ID_82576_QUAD_COPPER_ET2    0x1526
> #define E1000_DEV_ID_82576_SERDES_QUAD        0x150D
>
> 6. I don't see MACSEC code in the driver, is it implemented at Intel but
> just not open sourced? The Intel representative is unclear on that either.
>
> Basically I need confirm SerDes or RGMII for SFP, then learn how to modify
> the driver and program EEPROM to get 82576 working on the custom design.
> Any tips/suggestions are great appreciated!
>
> Thanks a lot.
>
> Joe
>

I think some of these questions would be best answered under NDA. Can you contact your vendor and go through that channel?

Thanks.

Todd Fujinaka
Software Application Engineer
Networking Division (ND)
Intel Corporation
tod...@in...
(503) 712-4565


-----Original Message-----
From: x aus [mailto:fo...@gm...] 
Sent: Wednesday, March 26, 2014 11:20 AM
To: e10...@li...
Subject: [E1000-devel] 82576/igb custom driver

We're using 82576 on a main board(ARM with PCIe x1) using 3.0.x kernel. I can probe the device at the moment.

I'm connecting the 82576 to a SFP which has an I2C for the external PHY(RJ45), I also have EEPROM and SPI-Flash connected via SPI to 82576 based on Intel's reference schematics.

I have a list of questions:
1. We're using SerDes to connect with the SFP, it's unclear to me per the Datasheet that, should we use SGMII instead? what's the key difference here? We're using RJ45/SFP but we may need use fiber/SFP later, which is the reason we're not using the internal PHY. I saw there are some code for SFP inside the igb driver.

2. On the EEPROM, do we have to use it? I think igb is getting deviceID from the EEPROM, as 'lspci -v' showed 10c9:0000 now, the EEPROM is blank at the moment, and I modified the code to bypass the MAC-address-reading from EEPROM(providing some fake MACs for now) so I can run 'ifconfig eth0 up'
against it though  I can never send out packets.

2.1 Is it possible at all the get this working without using EEPROM?

3. How to program the blank EEPROM, can I use ethtool do that? or do I have to take it out and program it before solder it down?

4. When do I need SPI-Flash? do I need it at all? The datasheet says it can hold some firmware but I'm not sure what it means.

5. From e1000_hw.h, 0x10C9 is for 82576 itself, the rest is for the specific products, but what's the difference between Fiber, Serdes, copper?
In my case I use SFP/RJ45, is it more like Serdes or Copper or even RGMII(there is aE1000_DEV_ID_I350_SGMII) , very confused.

#define E1000_DEV_ID_82576                    0x10C9
#define E1000_DEV_ID_82576_FIBER              0x10E6
#define E1000_DEV_ID_82576_SERDES             0x10E7
#define E1000_DEV_ID_82576_QUAD_COPPER        0x10E8
#define E1000_DEV_ID_82576_QUAD_COPPER_ET2    0x1526
#define E1000_DEV_ID_82576_SERDES_QUAD        0x150D

6. I don't see MACSEC code in the driver, is it implemented at Intel but just not open sourced? The Intel representative is unclear on that either.

Basically I need confirm SerDes or RGMII for SFP, then learn how to modify the driver and program EEPROM to get 82576 working on the custom design.
Any tips/suggestions are great appreciated!

Thanks a lot.

Joe

Yes, I think that's the issue.  With our 20ms timeslots it was taking
too long to do a bunch of reads so we added an ioctl to the tap driver
to read a bunch of packets into a buffer.  I think we can remove the
rtnl lock for this particular ioctl.  We didn't see an issue with the
e1000 motherboard interface, but maybe that one just releases the lock
faster..

Thanks a bunch,
Aaron


On Wed, Mar 26, 2014 at 10:51 AM, Rustad, Mark D
<mar...@in...> wrote:
> On Mar 26, 2014, at 10:24 AM, Aaron Brice <aar...@gm...> wrote:
>
>> On Tue, Mar 25, 2014 at 10:39 PM, Ronciak, John <joh...@in...> wrote:
>>> How are the different PCIe slots connected into the system?  In a lot of cases some of the slots are not all equal in terms of how they are laid out in the system.  If you move the interface doing the 20ms read to a NIC in a different slot does the behavior change?  Does the i/f on the m/b if used for the 20ms read work correctly?
>>
>> The odd part is that the process doing the 20ms read is reading from a
>> virtual tap interface (/dev/net/tun) and not from one of the NICs.  It
>> seems like bringing down the igb interface is causing some system
>> level lock at least for network I/O?
>>
>> Aaron
>
> That sounds like the rtnl lock to me. Are you doing ioctls on the tap interface at that time? I believe that the ioctls will take the rtnl lock, which is likely to delay things for you.
>
> --
> Mark Rustad, Networking Division, Intel Corporation
>

We're using 82576 on a main board(ARM with PCIe x1) using 3.0.x kernel. I
can probe the device at the moment.

I'm connecting the 82576 to a SFP which has an I2C for the external
PHY(RJ45), I also have EEPROM and SPI-Flash connected via SPI to 82576
based on Intel's reference schematics.

I have a list of questions:
1. We're using SerDes to connect with the SFP, it's unclear to me per the
Datasheet that, should we use SGMII instead? what's the key difference
here? We're using RJ45/SFP but we may need use fiber/SFP later, which is
the reason we're not using the internal PHY. I saw there are some code for
SFP inside the igb driver.

2. On the EEPROM, do we have to use it? I think igb is getting deviceID
from the EEPROM, as 'lspci -v' showed 10c9:0000 now, the EEPROM is blank at
the moment, and I modified the code to bypass the MAC-address-reading from
EEPROM(providing some fake MACs for now) so I can run 'ifconfig eth0 up'
against it though  I can never send out packets.

2.1 Is it possible at all the get this working without using EEPROM?

3. How to program the blank EEPROM, can I use ethtool do that? or do I have
to take it out and program it before solder it down?

4. When do I need SPI-Flash? do I need it at all? The datasheet says it can
hold some firmware but I'm not sure what it means.

5. From e1000_hw.h, 0x10C9 is for 82576 itself, the rest is for the
specific products, but what's the difference between Fiber, Serdes, copper?
In my case I use SFP/RJ45, is it more like Serdes or Copper or even
RGMII(there is aE1000_DEV_ID_I350_SGMII) , very confused.

#define E1000_DEV_ID_82576                    0x10C9
#define E1000_DEV_ID_82576_FIBER              0x10E6
#define E1000_DEV_ID_82576_SERDES             0x10E7
#define E1000_DEV_ID_82576_QUAD_COPPER        0x10E8
#define E1000_DEV_ID_82576_QUAD_COPPER_ET2    0x1526
#define E1000_DEV_ID_82576_SERDES_QUAD        0x150D

6. I don't see MACSEC code in the driver, is it implemented at Intel but
just not open sourced? The Intel representative is unclear on that either.

Basically I need confirm SerDes or RGMII for SFP, then learn how to modify
the driver and program EEPROM to get 82576 working on the custom design.
Any tips/suggestions are great appreciated!

Thanks a lot.

Joe

On Mar 26, 2014, at 10:24 AM, Aaron Brice <aar...@gm...> wrote:

> On Tue, Mar 25, 2014 at 10:39 PM, Ronciak, John <joh...@in...> wrote:
>> How are the different PCIe slots connected into the system?  In a lot of cases some of the slots are not all equal in terms of how they are laid out in the system.  If you move the interface doing the 20ms read to a NIC in a different slot does the behavior change?  Does the i/f on the m/b if used for the 20ms read work correctly?
> 
> The odd part is that the process doing the 20ms read is reading from a
> virtual tap interface (/dev/net/tun) and not from one of the NICs.  It
> seems like bringing down the igb interface is causing some system
> level lock at least for network I/O?
> 
> Aaron

That sounds like the rtnl lock to me. Are you doing ioctls on the tap interface at that time? I believe that the ioctls will take the rtnl lock, which is likely to delay things for you.

-- 
Mark Rustad, Networking Division, Intel Corporation

Just an update.  A patch for this issue was submitted into our internal queue on Monday and is in review/testing.

Dave Ertman

-----Original Message-----
From: Thomas Gleixner [mailto:tg...@li...] 
Sent: Friday, March 21, 2014 3:01 PM
To: Ertman, DavidX M
Cc: Ronciak, John; e10...@li...; ne...@vg...
Subject: RE: [E1000-devel] i217-LM boot wreckage

On Fri, 21 Mar 2014, Ertman, DavidX M wrote:

> I have discovered the cause of this issue.  It is related to the 
> change in the application of a Si errata workaround.
> 
> I will be working on grok'ing the total impact of this issue and 
> writing up the patch over the weekend and will have something pushed 
> into our internal queue early next week.

Take your time and don't spoil your weekend!

It's not a show stopper, as we know how to work around the issue.

If you need testing of some early patch or help with gathering information, let me know. We have 4 reproducer systems handy.

Thanks,

	tglx