You can subscribe to this list here.
2002 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
(2) |
Nov
(1) |
Dec
|
---|---|---|---|---|---|---|---|---|---|---|---|---|
2003 |
Jan
|
Feb
|
Mar
(1) |
Apr
(9) |
May
(3) |
Jun
|
Jul
(3) |
Aug
(6) |
Sep
|
Oct
(7) |
Nov
|
Dec
|
2004 |
Jan
|
Feb
(5) |
Mar
(10) |
Apr
(2) |
May
(22) |
Jun
(8) |
Jul
(4) |
Aug
(8) |
Sep
(3) |
Oct
|
Nov
(36) |
Dec
(52) |
2005 |
Jan
(9) |
Feb
(13) |
Mar
(9) |
Apr
|
May
(14) |
Jun
(5) |
Jul
(20) |
Aug
(31) |
Sep
(2) |
Oct
(3) |
Nov
(18) |
Dec
(18) |
2006 |
Jan
(36) |
Feb
(16) |
Mar
(76) |
Apr
(78) |
May
(32) |
Jun
(30) |
Jul
(67) |
Aug
(43) |
Sep
(54) |
Oct
(116) |
Nov
(223) |
Dec
(158) |
2007 |
Jan
(180) |
Feb
(71) |
Mar
(110) |
Apr
(114) |
May
(203) |
Jun
(100) |
Jul
(238) |
Aug
(191) |
Sep
(177) |
Oct
(171) |
Nov
(211) |
Dec
(159) |
2008 |
Jan
(227) |
Feb
(288) |
Mar
(197) |
Apr
(253) |
May
(132) |
Jun
(152) |
Jul
(109) |
Aug
(143) |
Sep
(157) |
Oct
(198) |
Nov
(121) |
Dec
(147) |
2009 |
Jan
(105) |
Feb
(61) |
Mar
(191) |
Apr
(161) |
May
(118) |
Jun
(172) |
Jul
(166) |
Aug
(67) |
Sep
(86) |
Oct
(79) |
Nov
(118) |
Dec
(181) |
2010 |
Jan
(136) |
Feb
(154) |
Mar
(92) |
Apr
(83) |
May
(101) |
Jun
(66) |
Jul
(118) |
Aug
(78) |
Sep
(134) |
Oct
(131) |
Nov
(132) |
Dec
(104) |
2011 |
Jan
(79) |
Feb
(104) |
Mar
(144) |
Apr
(145) |
May
(130) |
Jun
(169) |
Jul
(146) |
Aug
(76) |
Sep
(113) |
Oct
(82) |
Nov
(145) |
Dec
(122) |
2012 |
Jan
(132) |
Feb
(106) |
Mar
(145) |
Apr
(238) |
May
(140) |
Jun
(162) |
Jul
(166) |
Aug
(147) |
Sep
(80) |
Oct
(148) |
Nov
(192) |
Dec
(90) |
2013 |
Jan
(139) |
Feb
(162) |
Mar
(174) |
Apr
(81) |
May
(261) |
Jun
(301) |
Jul
(106) |
Aug
(175) |
Sep
(305) |
Oct
(222) |
Nov
(95) |
Dec
(120) |
2014 |
Jan
(196) |
Feb
(171) |
Mar
(146) |
Apr
(118) |
May
(127) |
Jun
(93) |
Jul
(175) |
Aug
(66) |
Sep
(85) |
Oct
(120) |
Nov
(81) |
Dec
(192) |
2015 |
Jan
(141) |
Feb
(133) |
Mar
(189) |
Apr
(126) |
May
(59) |
Jun
(117) |
Jul
(56) |
Aug
(97) |
Sep
(44) |
Oct
(48) |
Nov
(33) |
Dec
(87) |
2016 |
Jan
(37) |
Feb
(56) |
Mar
(72) |
Apr
(65) |
May
(66) |
Jun
(65) |
Jul
(98) |
Aug
(54) |
Sep
(84) |
Oct
(68) |
Nov
(69) |
Dec
(60) |
2017 |
Jan
(30) |
Feb
(38) |
Mar
(53) |
Apr
(6) |
May
(2) |
Jun
(5) |
Jul
(15) |
Aug
(15) |
Sep
(7) |
Oct
(18) |
Nov
(23) |
Dec
(6) |
2018 |
Jan
(39) |
Feb
(5) |
Mar
(34) |
Apr
(26) |
May
(27) |
Jun
(5) |
Jul
(12) |
Aug
(4) |
Sep
|
Oct
(4) |
Nov
(4) |
Dec
(4) |
2019 |
Jan
(7) |
Feb
(10) |
Mar
(21) |
Apr
(26) |
May
(4) |
Jun
(5) |
Jul
(11) |
Aug
(6) |
Sep
(7) |
Oct
(13) |
Nov
(3) |
Dec
(17) |
2020 |
Jan
|
Feb
(3) |
Mar
(3) |
Apr
(5) |
May
(2) |
Jun
(5) |
Jul
|
Aug
|
Sep
(6) |
Oct
(7) |
Nov
(2) |
Dec
(7) |
2021 |
Jan
(9) |
Feb
(10) |
Mar
(18) |
Apr
(1) |
May
(3) |
Jun
|
Jul
(16) |
Aug
(2) |
Sep
|
Oct
|
Nov
(9) |
Dec
(2) |
2022 |
Jan
(3) |
Feb
|
Mar
(9) |
Apr
(8) |
May
(5) |
Jun
(6) |
Jul
(1) |
Aug
|
Sep
(1) |
Oct
|
Nov
(7) |
Dec
(2) |
2023 |
Jan
(7) |
Feb
(2) |
Mar
(6) |
Apr
|
May
(4) |
Jun
(2) |
Jul
(4) |
Aug
(3) |
Sep
(4) |
Oct
(2) |
Nov
(4) |
Dec
(10) |
2024 |
Jan
(4) |
Feb
(2) |
Mar
(1) |
Apr
|
May
(1) |
Jun
(1) |
Jul
|
Aug
(1) |
Sep
|
Oct
|
Nov
|
Dec
|
S | M | T | W | T | F | S |
---|---|---|---|---|---|---|
|
|
|
|
1
(1) |
2
|
3
(1) |
4
|
5
(2) |
6
(2) |
7
(9) |
8
(7) |
9
|
10
|
11
|
12
(1) |
13
(7) |
14
|
15
(2) |
16
(2) |
17
|
18
|
19
(1) |
20
(1) |
21
(1) |
22
|
23
(2) |
24
|
25
|
26
|
27
(1) |
28
(3) |
29
(5) |
30
|
31
|
From: Carsten A. <Car...@ae...> - 2015-10-29 10:50:05
|
Hi after filing https://sourceforge.net/p/e1000/bugs/496/ I thought it may help to subscribe here (again) to help out quickly if a question arises. If not, please feel free to scare me away with pitch forks ;) Cheers Carsten -- Dr. Carsten Aulbert, Atlas cluster administration Max Planck Institute for Gravitational Physics (Albert Einstein Institute) Callinstraße 38, 30167 Hannover, Germany Tel: +49 511 762 17185, Fax: +49 511 762 17193 |
From: Richard C. <ric...@gm...> - 2015-10-29 10:27:18
|
On Thu, Oct 29, 2015 at 09:47:36AM +0800, Gangfeng Huang wrote: > This patch is reference to the Intel Open-AVB project: > http://github.com/AVnu/Open-AVB/tree/master/kmod/igb You know that this kind of HW specific, chardev interface has very little chance of being accepted mainline, don't you? The idea of having a time based packet scheduler as a new interface in the networking stack is attractive, but it would need to be a generic solution. Also, this kind of work must be posted to the netdev list for proper review. Thanks, Richard |
From: Gangfeng H. <gan...@ni...> - 2015-10-29 02:27:12
|
This patch create a character device for Intel I210 Ethernet controller, it can be used for developing Audio/Video Bridging applications,Industrial Ethernet applications which require precise timing control over frame transmission, or test harneses for measuring system latencies and samping events. As the AVB queues (0,1) are mapped to a user-space application, typical LAN traffic must be steered away from these queues. For transmit, this driver implements one method registering an ndo_select_queue handler to map traffic to queue[3]. and set the register MRQC to receive all BE traffic to Rx queue[3]. This patch is reference to the Intel Open-AVB project: http://github.com/AVnu/Open-AVB/tree/master/kmod/igb Signed-off-by: Gangfeng Huang <gan...@ni...> --- drivers/net/ethernet/intel/igb/Makefile | 2 +- drivers/net/ethernet/intel/igb/e1000_defines.h | 1 + drivers/net/ethernet/intel/igb/igb.h | 14 +- drivers/net/ethernet/intel/igb/igb_cdev.c | 511 ++++++++++++++++++++++++ drivers/net/ethernet/intel/igb/igb_cdev.h | 45 +++ drivers/net/ethernet/intel/igb/igb_main.c | 104 ++++- 6 files changed, 664 insertions(+), 13 deletions(-) create mode 100644 drivers/net/ethernet/intel/igb/igb_cdev.c create mode 100644 drivers/net/ethernet/intel/igb/igb_cdev.h diff --git a/drivers/net/ethernet/intel/igb/Makefile b/drivers/net/ethernet/intel/igb/Makefile index 5bcb2de..3fee429 100644 --- a/drivers/net/ethernet/intel/igb/Makefile +++ b/drivers/net/ethernet/intel/igb/Makefile @@ -33,4 +33,4 @@ obj-$(CONFIG_IGB) += igb.o igb-objs := igb_main.o igb_ethtool.o e1000_82575.o \ e1000_mac.o e1000_nvm.o e1000_phy.o e1000_mbx.o \ - e1000_i210.o igb_ptp.o igb_hwmon.o + e1000_i210.o igb_ptp.o igb_hwmon.o igb_cdev.o diff --git a/drivers/net/ethernet/intel/igb/e1000_defines.h b/drivers/net/ethernet/intel/igb/e1000_defines.h index f09d016..6bf0e56 100644 --- a/drivers/net/ethernet/intel/igb/e1000_defines.h +++ b/drivers/net/ethernet/intel/igb/e1000_defines.h @@ -112,6 +112,7 @@ #define E1000_MRQC_RSS_FIELD_IPV6 0x00100000 #define E1000_MRQC_RSS_FIELD_IPV6_TCP 0x00200000 +#define E1000_MRQC_DEF_QUEUE_OFFSET 0x3 /* Management Control */ #define E1000_MANC_SMBUS_EN 0x00000001 /* SMBus Enabled - RO */ diff --git a/drivers/net/ethernet/intel/igb/igb.h b/drivers/net/ethernet/intel/igb/igb.h index b84a266..f661729 100644 --- a/drivers/net/ethernet/intel/igb/igb.h +++ b/drivers/net/ethernet/intel/igb/igb.h @@ -38,6 +38,8 @@ #include <linux/i2c-algo-bit.h> #include <linux/pci.h> #include <linux/mdio.h> +#include <linux/types.h> +#include <linux/cdev.h> struct igb_adapter; @@ -50,12 +52,12 @@ struct igb_adapter; #define IGB_70K_ITR 56 /* TX/RX descriptor defines */ -#define IGB_DEFAULT_TXD 256 +#define IGB_DEFAULT_TXD 1024 #define IGB_DEFAULT_TX_WORK 128 #define IGB_MIN_TXD 80 #define IGB_MAX_TXD 4096 -#define IGB_DEFAULT_RXD 256 +#define IGB_DEFAULT_RXD 1024 #define IGB_MIN_RXD 80 #define IGB_MAX_RXD 4096 @@ -468,6 +470,14 @@ struct igb_adapter { u16 eee_advert; bool qav_mode; + struct cdev char_dev; + struct list_head user_page_list; + struct mutex user_page_mutex; /* protect user_page_list */ + unsigned long tx_uring_init; + unsigned long rx_uring_init; + struct mutex user_ring_mutex; /* protect tx/rx_uring_init */ + bool cdev_in_use; + struct mutex cdev_mutex; /* protect cdev_in_use */ }; #define IGB_FLAG_HAS_MSI (1 << 0) diff --git a/drivers/net/ethernet/intel/igb/igb_cdev.c b/drivers/net/ethernet/intel/igb/igb_cdev.c new file mode 100644 index 0000000..df237c6 --- /dev/null +++ b/drivers/net/ethernet/intel/igb/igb_cdev.c @@ -0,0 +1,511 @@ +#include "igb.h" +#include "igb_cdev.h" + +#include <linux/pagemap.h> +#include <linux/bitops.h> +#include <linux/types.h> +#include <linux/cdev.h> + +/* TSN char dev */ +static DECLARE_BITMAP(cdev_minors, IGB_MAX_DEV_NUM); + +static int igb_major; +static struct class *igb_class; +static const char * const igb_class_name = "igb_tsn"; +static const char * const igb_dev_name = "igb_tsn_%s"; + +/* user-mode API forward definitions */ +static int igb_open_file(struct inode *inode, struct file *file); +static int igb_close_file(struct inode *inode, struct file *file); +static int igb_mmap(struct file *file, struct vm_area_struct *vma); +static long igb_ioctl_file(struct file *file, unsigned int cmd, + unsigned long arg); + +/* user-mode IO API registrations */ +static const struct file_operations igb_fops = { + .owner = THIS_MODULE, + .llseek = no_llseek, + .open = igb_open_file, + .release = igb_close_file, + .mmap = igb_mmap, + .unlocked_ioctl = igb_ioctl_file, +}; + +int igb_tsn_setup_all_tx_resources(struct igb_adapter *adapter) +{ + struct pci_dev *pdev = adapter->pdev; + int i, err = 0; + + for (i = 0; i < IGB_USER_TX_QUEUES; i++) { + err = igb_setup_tx_resources(adapter->tx_ring[i]); + if (err) { + dev_err(&pdev->dev, + "Allocation for Tx Queue %u failed\n", i); + for (i--; i >= 0; i--) + igb_free_tx_resources(adapter->tx_ring[i]); + break; + } + } + + return err; +} + +int igb_tsn_setup_all_rx_resources(struct igb_adapter *adapter) +{ + struct pci_dev *pdev = adapter->pdev; + int i, err = 0; + + for (i = 0; i < IGB_USER_RX_QUEUES; i++) { + err = igb_setup_rx_resources(adapter->rx_ring[i]); + if (err) { + dev_err(&pdev->dev, + "Allocation for Rx Queue %u failed\n", i); + for (i--; i >= 0; i--) + igb_free_rx_resources(adapter->rx_ring[i]); + break; + } + } + + return err; +} + +void igb_tsn_free_all_tx_resources(struct igb_adapter *adapter) +{ + int i; + + for (i = 0; i < IGB_USER_TX_QUEUES; i++) + igb_free_tx_resources(adapter->tx_ring[i]); +} + +void igb_tsn_free_all_rx_resources(struct igb_adapter *adapter) +{ + int i; + + for (i = 0; i < IGB_USER_RX_QUEUES; i++) + igb_free_rx_resources(adapter->rx_ring[i]); +} + +static int igb_bind(struct file *file, void __user *argp) +{ + struct igb_adapter *adapter; + u32 mmap_size; + + adapter = (struct igb_adapter *)file->private_data; + + if (NULL == adapter) + return -ENOENT; + + mmap_size = pci_resource_len(adapter->pdev, 0); + + if (copy_to_user(argp, &mmap_size, sizeof(mmap_size))) + return -EFAULT; + + return 0; +} + +static long igb_mapring(struct file *file, void __user *arg) +{ + struct igb_adapter *adapter; + struct igb_buf_cmd req; + int queue_size; + unsigned long *uring_init; + struct igb_ring *ring; + int err; + + if (copy_from_user(&req, arg, sizeof(req))) + return -EFAULT; + + if (req.flags != 0 && req.flags != 1) + return -EINVAL; + + adapter = file->private_data; + if (NULL == adapter) { + dev_err(&adapter->pdev->dev, "map to unbound device!\n"); + return -ENOENT; + } + + /* Req flags, Tx: 0, Rx: 1 */ + if (req.flags == 0) { + queue_size = IGB_USER_TX_QUEUES; + uring_init = &adapter->tx_uring_init; + ring = adapter->tx_ring[req.queue]; + } else { + queue_size = IGB_USER_RX_QUEUES; + uring_init = &adapter->rx_uring_init; + ring = adapter->rx_ring[req.queue]; + } + + mutex_lock(&adapter->user_ring_mutex); + if (test_bit(req.queue, uring_init)) { + dev_err(&adapter->pdev->dev, "the queue is in using\n"); + err = -EBUSY; + goto failed; + } + + if (req.queue >= queue_size) { + err = -EINVAL; + goto failed; + } + + set_pages_uc(virt_to_page(ring->desc), ring->size >> PAGE_SHIFT); + set_bit(req.queue, uring_init); + mutex_unlock(&adapter->user_ring_mutex); + + req.physaddr = ring->dma; + req.mmap_size = ring->size; + + if (copy_to_user(arg, &req, sizeof(req))) { + dev_err(&adapter->pdev->dev, "copyout to user failed\n"); + return -EFAULT; + } + + return 0; +failed: + mutex_unlock(&adapter->user_ring_mutex); + return err; +} + +static long igb_mapbuf(struct file *file, void __user *arg) +{ + struct igb_adapter *adapter; + struct igb_buf_cmd req; + struct page *page; + dma_addr_t page_dma; + struct igb_user_page *userpage; + int err = 0; + int direction; + + if (copy_from_user(&req, arg, sizeof(req))) + return -EFAULT; + + if (req.flags != 0 && req.flags != 1) + return -EINVAL; + + adapter = file->private_data; + if (NULL == adapter) { + dev_err(&adapter->pdev->dev, "map to unbound device!\n"); + return -ENOENT; + } + + userpage = kzalloc(sizeof(*userpage), GFP_KERNEL); + if (unlikely(!userpage)) + return -ENOMEM; + + page = alloc_page(GFP_KERNEL | __GFP_COLD); + if (unlikely(!page)) { + err = -ENOMEM; + goto failed; + } + + direction = req.flags ? DMA_FROM_DEVICE : DMA_TO_DEVICE; + page_dma = dma_map_page(&adapter->pdev->dev, page, + 0, PAGE_SIZE, direction); + + if (dma_mapping_error(&adapter->pdev->dev, page_dma)) { + put_page(page); + err = -ENOMEM; + goto failed; + } + + set_pages_uc(page, 1); + userpage->page = page; + userpage->page_dma = page_dma; + userpage->flags = req.flags; + + mutex_lock(&adapter->user_page_mutex); + list_add_tail(&userpage->page_node, &adapter->user_page_list); + mutex_unlock(&adapter->user_page_mutex); + + req.physaddr = page_dma; + req.mmap_size = PAGE_SIZE; + + if (copy_to_user(arg, &req, sizeof(req))) { + dev_err(&adapter->pdev->dev, "copyout to user failed\n"); + return -EFAULT; + } + return 0; + +failed: + kfree(userpage); + return err; +} + +static long igb_unmapring(struct file *file, void __user *arg) +{ + struct igb_adapter *adapter; + struct igb_buf_cmd req; + struct igb_ring *ring; + int queue_size; + unsigned long *uring_init; + int err; + + if (copy_from_user(&req, arg, sizeof(req))) + return -EFAULT; + + if (req.flags != 0 && req.flags != 1) + return -EINVAL; + + adapter = file->private_data; + if (NULL == adapter) { + dev_err(&adapter->pdev->dev, "map to unbound device!\n"); + return -ENOENT; + } + + if (req.flags == 0) { + queue_size = IGB_USER_TX_QUEUES; + uring_init = &adapter->tx_uring_init; + ring = adapter->tx_ring[req.queue]; + } else { + queue_size = IGB_USER_RX_QUEUES; + uring_init = &adapter->rx_uring_init; + ring = adapter->rx_ring[req.queue]; + } + + if (req.queue >= queue_size) + return -EINVAL; + + mutex_lock(&adapter->user_ring_mutex); + if (!test_bit(req.queue, uring_init)) { + dev_err(&adapter->pdev->dev, + "the ring is already unmap\n"); + err = -EINVAL; + goto failed; + } + + set_pages_wb(virt_to_page(ring->desc), ring->size >> PAGE_SHIFT); + clear_bit(req.queue, uring_init); + mutex_unlock(&adapter->user_ring_mutex); + + return 0; +failed: + mutex_unlock(&adapter->user_ring_mutex); + return err; +} + +static void igb_free_page(struct igb_adapter *adapter, + struct igb_user_page *userpage) +{ + int direction = userpage->flags ? DMA_FROM_DEVICE : DMA_TO_DEVICE; + + set_pages_wb(userpage->page, 1); + dma_unmap_page(&adapter->pdev->dev, + userpage->page_dma, + PAGE_SIZE, + direction); + + put_page(userpage->page); + list_del(&userpage->page_node); + kfree(userpage); + userpage = NULL; +} + +static long igb_unmapbuf(struct file *file, void __user *arg) +{ + int err = 0; + struct igb_adapter *adapter; + struct igb_buf_cmd req; + struct igb_user_page *userpage, *tmp; + + if (copy_from_user(&req, arg, sizeof(req))) + return -EFAULT; + + adapter = file->private_data; + if (NULL == adapter) { + dev_err(&adapter->pdev->dev, "map to unbound device!\n"); + return -ENOENT; + } + + mutex_lock(&adapter->user_page_mutex); + if (list_empty(&adapter->user_page_list)) { + err = -EINVAL; + goto failed; + } + + list_for_each_entry_safe(userpage, tmp, &adapter->user_page_list, + page_node) { + if (req.physaddr == userpage->page_dma) { + igb_free_page(adapter, userpage); + break; + } + } + mutex_unlock(&adapter->user_page_mutex); + + return 0; +failed: + mutex_unlock(&adapter->user_page_mutex); + return err; +} + +static long igb_ioctl_file(struct file *file, unsigned int cmd, + unsigned long arg) +{ + void __user *argp = (void __user *)arg; + int err; + + switch (cmd) { + case IGB_BIND: + err = igb_bind(file, argp); + break; + case IGB_MAPRING: + err = igb_mapring(file, argp); + break; + case IGB_MAPBUF: + err = igb_mapbuf(file, argp); + break; + case IGB_UNMAPRING: + err = igb_unmapring(file, argp); + break; + case IGB_UNMAPBUF: + err = igb_unmapbuf(file, argp); + break; + default: + err = -EINVAL; + break; + }; + + return err; +} + +static int igb_open_file(struct inode *inode, struct file *file) +{ + struct igb_adapter *adapter; + int err = 0; + + adapter = container_of(inode->i_cdev, struct igb_adapter, char_dev); + if (!adapter) + return -ENOENT; + + if (!adapter->qav_mode) + return -EPERM; + + mutex_lock(&adapter->cdev_mutex); + if (adapter->cdev_in_use) { + err = -EBUSY; + goto failed; + } + + file->private_data = adapter; + adapter->cdev_in_use = true; + mutex_unlock(&adapter->cdev_mutex); + + return 0; +failed: + mutex_unlock(&adapter->cdev_mutex); + return err; +} + +static int igb_close_file(struct inode *inode, struct file *file) +{ + struct igb_adapter *adapter = file->private_data; + + if (NULL == adapter) + return 0; + + mutex_lock(&adapter->cdev_mutex); + if (!adapter->cdev_in_use) + goto out; + + mutex_lock(&adapter->user_page_mutex); + if (!list_empty(&adapter->user_page_list)) { + struct igb_user_page *userpage, *tmp; + + list_for_each_entry_safe(userpage, tmp, + &adapter->user_page_list, page_node) { + if (userpage) + igb_free_page(adapter, userpage); + } + } + mutex_unlock(&adapter->user_page_mutex); + + file->private_data = NULL; + adapter->cdev_in_use = false; + adapter->tx_uring_init = 0; + adapter->rx_uring_init = 0; + +out: + mutex_unlock(&adapter->cdev_mutex); + return 0; +} + +static int igb_mmap(struct file *file, struct vm_area_struct *vma) +{ + struct igb_adapter *adapter = file->private_data; + unsigned long size = vma->vm_end - vma->vm_start; + dma_addr_t pgoff = vma->vm_pgoff; + dma_addr_t physaddr; + + if (NULL == adapter) + return -ENODEV; + + if (pgoff == 0) + physaddr = pci_resource_start(adapter->pdev, 0) >> PAGE_SHIFT; + else + physaddr = pgoff; + + vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); + + if (remap_pfn_range(vma, vma->vm_start, + physaddr, size, vma->vm_page_prot)) + return -EAGAIN; + + return 0; +} + +int igb_add_cdev(struct igb_adapter *adapter) +{ + int result = 0; + dev_t dev_num; + int igb_minor; + + igb_minor = find_first_zero_bit(cdev_minors, IGB_MAX_DEV_NUM); + if (igb_minor >= IGB_MAX_DEV_NUM) + return -EBUSY; + set_bit(igb_minor, cdev_minors); + + dev_num = MKDEV(igb_major, igb_minor); + cdev_init(&adapter->char_dev, &igb_fops); + adapter->char_dev.owner = THIS_MODULE; + adapter->char_dev.ops = &igb_fops; + result = cdev_add(&adapter->char_dev, dev_num, 1); + + if (result) { + dev_err(&adapter->pdev->dev, + "igb_tsn: add character device failed\n"); + return result; + } + + device_create(igb_class, NULL, dev_num, NULL, igb_dev_name, + adapter->netdev->name); + + return 0; +} + +void igb_remove_cdev(struct igb_adapter *adapter) +{ + device_destroy(igb_class, adapter->char_dev.dev); + cdev_del(&adapter->char_dev); +} + +int igb_cdev_init(char *igb_driver_name) +{ + dev_t dev_num; + int ret; + + ret = alloc_chrdev_region(&dev_num, 0, IGB_MAX_DEV_NUM, + igb_driver_name); + if (ret) + return ret; + igb_major = MAJOR(dev_num); + + igb_class = class_create(THIS_MODULE, igb_class_name); + if (IS_ERR(igb_class)) + pr_info("igb_tsn: create device class failed\n"); + + return ret; +} + +void igb_cdev_destroy(void) +{ + class_destroy(igb_class); + unregister_chrdev_region(MKDEV(igb_major, 0), IGB_MAX_DEV_NUM); +} diff --git a/drivers/net/ethernet/intel/igb/igb_cdev.h b/drivers/net/ethernet/intel/igb/igb_cdev.h new file mode 100644 index 0000000..07fc0b6 --- /dev/null +++ b/drivers/net/ethernet/intel/igb/igb_cdev.h @@ -0,0 +1,45 @@ +#ifndef _IGB_CDEV_H_ +#define _IGB_CDEV_H_ + +#include <asm/page.h> +#include <asm/ioctl.h> + +struct igb_adapter; +/* queues reserved for user mode */ +#define IGB_USER_TX_QUEUES 2 +#define IGB_USER_RX_QUEUES 2 +#define IGB_MAX_DEV_NUM 64 + +/* TSN char dev ioctls */ +#define IGB_BIND _IOW('E', 200, int) +#define IGB_MAPRING _IOW('E', 201, int) +#define IGB_UNMAPRING _IOW('E', 202, int) +#define IGB_MAPBUF _IOW('E', 203, int) +#define IGB_UNMAPBUF _IOW('E', 204, int) + +/* Used with both map/unmap ring & buf ioctls */ +struct igb_buf_cmd { + u64 physaddr; + u32 queue; + u32 mmap_size; + u32 flags; +}; + +struct igb_user_page { + struct list_head page_node; + struct page *page; + dma_addr_t page_dma; + u32 flags; +}; + +int igb_tsn_setup_all_tx_resources(struct igb_adapter *); +int igb_tsn_setup_all_rx_resources(struct igb_adapter *); +void igb_tsn_free_all_tx_resources(struct igb_adapter *); +void igb_tsn_free_all_rx_resources(struct igb_adapter *); + +int igb_add_cdev(struct igb_adapter *adapter); +void igb_remove_cdev(struct igb_adapter *adapter); +int igb_cdev_init(char *igb_driver_name); +void igb_cdev_destroy(void); + +#endif diff --git a/drivers/net/ethernet/intel/igb/igb_main.c b/drivers/net/ethernet/intel/igb/igb_main.c index 1d00f41..4193e58 100644 --- a/drivers/net/ethernet/intel/igb/igb_main.c +++ b/drivers/net/ethernet/intel/igb/igb_main.c @@ -55,6 +55,7 @@ #endif #include <linux/i2c.h> #include "igb.h" +#include "igb_cdev.h" #define MAJ 5 #define MIN 3 @@ -688,6 +689,11 @@ static int __init igb_init_module(void) #ifdef CONFIG_IGB_DCA dca_register_notify(&dca_notifier); #endif + + ret = igb_cdev_init(igb_driver_name); + if (ret) + return ret; + ret = pci_register_driver(&igb_driver); return ret; } @@ -706,6 +712,8 @@ static void __exit igb_exit_module(void) dca_unregister_notify(&dca_notifier); #endif pci_unregister_driver(&igb_driver); + + igb_cdev_destroy(); } module_exit(igb_exit_module); @@ -1629,7 +1637,8 @@ static void igb_configure(struct igb_adapter *adapter) * at least 1 descriptor unused to make sure * next_to_use != next_to_clean */ - for (i = 0; i < adapter->num_rx_queues; i++) { + i = adapter->qav_mode ? IGB_USER_RX_QUEUES : 0; + for (; i < adapter->num_rx_queues; i++) { struct igb_ring *ring = adapter->rx_ring[i]; igb_alloc_rx_buffers(ring, igb_desc_unused(ring)); } @@ -2077,10 +2086,24 @@ static int igb_set_features(struct net_device *netdev, return 0; } +static u16 igb_select_queue(struct net_device *netdev, + struct sk_buff *skb, + void *accel_priv, + select_queue_fallback_t fallback) +{ + struct igb_adapter *adapter = netdev_priv(netdev); + + if (adapter->qav_mode) + return adapter->num_tx_queues - 1; + else + return fallback(netdev, skb); +} + static const struct net_device_ops igb_netdev_ops = { .ndo_open = igb_open, .ndo_stop = igb_close, .ndo_start_xmit = igb_xmit_frame, + .ndo_select_queue = igb_select_queue, .ndo_get_stats64 = igb_get_stats64, .ndo_set_rx_mode = igb_set_rx_mode, .ndo_set_mac_address = igb_set_mac, @@ -2306,6 +2329,10 @@ static int igb_probe(struct pci_dev *pdev, const struct pci_device_id *ent) adapter->msg_enable = netif_msg_init(debug, DEFAULT_MSG_ENABLE); adapter->qav_mode = false; + adapter->tx_uring_init = 0; + adapter->rx_uring_init = 0; + adapter->cdev_in_use = false; + err = -EIO; hw->hw_addr = pci_iomap(pdev, 0, 0); if (!hw->hw_addr) @@ -2559,6 +2586,10 @@ static int igb_probe(struct pci_dev *pdev, const struct pci_device_id *ent) } } + err = igb_add_cdev(adapter); + if (err) + goto err_register; + /* carrier off reporting is important to ethtool even BEFORE open */ netif_carrier_off(netdev); @@ -2803,6 +2834,8 @@ static void igb_remove(struct pci_dev *pdev) struct igb_adapter *adapter = netdev_priv(netdev); struct e1000_hw *hw = &adapter->hw; + igb_remove_cdev(adapter); + pm_runtime_get_noresume(&pdev->dev); #ifdef CONFIG_IGB_HWMON igb_sysfs_exit(adapter); @@ -2985,6 +3018,12 @@ static int igb_sw_init(struct igb_adapter *adapter) adapter->min_frame_size = ETH_ZLEN + ETH_FCS_LEN; spin_lock_init(&adapter->stats64_lock); + + INIT_LIST_HEAD(&adapter->user_page_list); + mutex_init(&adapter->user_page_mutex); + mutex_init(&adapter->user_ring_mutex); + mutex_init(&adapter->cdev_mutex); + #ifdef CONFIG_PCI_IOV switch (hw->mac.type) { case e1000_82576: @@ -3231,7 +3270,8 @@ static int igb_setup_all_tx_resources(struct igb_adapter *adapter) struct pci_dev *pdev = adapter->pdev; int i, err = 0; - for (i = 0; i < adapter->num_tx_queues; i++) { + i = adapter->qav_mode ? IGB_USER_TX_QUEUES : 0; + for (; i < adapter->num_tx_queues; i++) { err = igb_setup_tx_resources(adapter->tx_ring[i]); if (err) { dev_err(&pdev->dev, @@ -3319,7 +3359,8 @@ static void igb_configure_tx(struct igb_adapter *adapter) { int i; - for (i = 0; i < adapter->num_tx_queues; i++) + i = adapter->qav_mode ? IGB_USER_TX_QUEUES : 0; + for (; i < adapter->num_tx_queues; i++) igb_configure_tx_ring(adapter, adapter->tx_ring[i]); } @@ -3374,7 +3415,8 @@ static int igb_setup_all_rx_resources(struct igb_adapter *adapter) struct pci_dev *pdev = adapter->pdev; int i, err = 0; - for (i = 0; i < adapter->num_rx_queues; i++) { + i = adapter->qav_mode ? IGB_USER_RX_QUEUES : 0; + for (; i < adapter->num_rx_queues; i++) { err = igb_setup_rx_resources(adapter->rx_ring[i]); if (err) { dev_err(&pdev->dev, @@ -3399,6 +3441,15 @@ static void igb_setup_mrqc(struct igb_adapter *adapter) u32 j, num_rx_queues; u32 rss_key[10]; + /* For TSN, kernel driver only create buffer for queue 2 and queue 3, + * by default receive all BE packets from queue 3. + */ + if (adapter->qav_mode) { + wr32(E1000_MRQC, (adapter->num_rx_queues - 1) + << E1000_MRQC_DEF_QUEUE_OFFSET); + return; + } + netdev_rss_key_fill(rss_key, sizeof(rss_key)); for (j = 0; j < 10; j++) wr32(E1000_RSSRK(j), rss_key[j]); @@ -3474,6 +3525,7 @@ static void igb_setup_mrqc(struct igb_adapter *adapter) if (hw->mac.type != e1000_i211) mrqc |= E1000_MRQC_ENABLE_RSS_4Q; } + igb_vmm_control(adapter); wr32(E1000_MRQC, mrqc); @@ -3701,7 +3753,8 @@ static void igb_configure_rx(struct igb_adapter *adapter) /* Setup the HW Rx Head and Tail Descriptor Pointers and * the Base and Length of the Rx Descriptor Ring */ - for (i = 0; i < adapter->num_rx_queues; i++) + i = adapter->qav_mode ? IGB_USER_RX_QUEUES : 0; + for (; i < adapter->num_rx_queues; i++) igb_configure_rx_ring(adapter, adapter->rx_ring[i]); } @@ -3737,8 +3790,8 @@ void igb_free_tx_resources(struct igb_ring *tx_ring) static void igb_free_all_tx_resources(struct igb_adapter *adapter) { int i; - - for (i = 0; i < adapter->num_tx_queues; i++) + i = adapter->qav_mode ? IGB_USER_TX_QUEUES : 0; + for (; i < adapter->num_tx_queues; i++) if (adapter->tx_ring[i]) igb_free_tx_resources(adapter->tx_ring[i]); } @@ -3804,7 +3857,8 @@ static void igb_clean_all_tx_rings(struct igb_adapter *adapter) { int i; - for (i = 0; i < adapter->num_tx_queues; i++) + i = adapter->qav_mode ? IGB_USER_TX_QUEUES : 0; + for (; i < adapter->num_tx_queues; i++) if (adapter->tx_ring[i]) igb_clean_tx_ring(adapter->tx_ring[i]); } @@ -3842,7 +3896,8 @@ static void igb_free_all_rx_resources(struct igb_adapter *adapter) { int i; - for (i = 0; i < adapter->num_rx_queues; i++) + i = adapter->qav_mode ? IGB_USER_RX_QUEUES : 0; + for (; i < adapter->num_rx_queues; i++) if (adapter->rx_ring[i]) igb_free_rx_resources(adapter->rx_ring[i]); } @@ -3898,7 +3953,8 @@ static void igb_clean_all_rx_rings(struct igb_adapter *adapter) { int i; - for (i = 0; i < adapter->num_rx_queues; i++) + i = adapter->qav_mode ? IGB_USER_TX_QUEUES : 0; + for (; i < adapter->num_rx_queues; i++) if (adapter->rx_ring[i]) igb_clean_rx_ring(adapter->rx_ring[i]); } @@ -6928,6 +6984,11 @@ static bool igb_clean_rx_irq(struct igb_q_vector *q_vector, const int budget) struct sk_buff *skb = rx_ring->skb; unsigned int total_bytes = 0, total_packets = 0; u16 cleaned_count = igb_desc_unused(rx_ring); + struct igb_adapter *adapter = netdev_priv(rx_ring->netdev); + + /* Don't service user (AVB) queues */ + if (adapter->qav_mode && rx_ring->queue_index < IGB_USER_RX_QUEUES) + return true; while (likely(total_packets < budget)) { union e1000_adv_rx_desc *rx_desc; @@ -7127,6 +7188,9 @@ static int igb_mii_ioctl(struct net_device *netdev, struct ifreq *ifr, int cmd) return 0; } +#define SIOSTXQUEUESELECT SIOCDEVPRIVATE +#define SIOSRXQUEUESELECT (SIOCDEVPRIVATE + 1) + /** * igb_ioctl - * @netdev: @@ -8188,6 +8252,9 @@ static int igb_change_mode(struct igb_adapter *adapter, int request_mode) if (request_mode == current_mode) return 0; + if (adapter->cdev_in_use) + return -EBUSY; + netdev = adapter->netdev; rtnl_lock(); @@ -8197,6 +8264,11 @@ static int igb_change_mode(struct igb_adapter *adapter, int request_mode) else igb_reset(adapter); + if (current_mode) { + igb_tsn_free_all_rx_resources(adapter); + igb_tsn_free_all_tx_resources(adapter); + } + igb_clear_interrupt_scheme(adapter); adapter->qav_mode = request_mode; @@ -8210,12 +8282,23 @@ static int igb_change_mode(struct igb_adapter *adapter, int request_mode) goto err_out; } + if (request_mode) { + err = igb_tsn_setup_all_tx_resources(adapter); + if (err) + goto err_out; + err = igb_tsn_setup_all_rx_resources(adapter); + if (err) + goto err_tsn_setup_rx; + } + if (netif_running(netdev)) igb_open(netdev); rtnl_unlock(); return err; +err_tsn_setup_rx: + igb_tsn_free_all_tx_resources(adapter); err_out: rtnl_unlock(); return err; @@ -8253,4 +8336,5 @@ static ssize_t igb_set_qav_mode(struct device *dev, return len; } + /* igb_main.c */ -- 1.7.9.5 |
From: Gangfeng H. <gan...@ni...> - 2015-10-29 02:11:50
|
I210 supports two transmit modes, legacy and Qav. The transmit mode is configured in TQAVCTRL.QavMode register. Before this patch igb driver only support legacy mode. This patch makes it possible to configure the transmit mode. Example: Get the transmit mode: $ echo /sys/class/net/eth0/qav_mode 0 Set transmit mode to qav mode $ echo 1 > /sys/class/net/eth0/qav_mode Tested: Setting /sys/class/net/eth0/qav_mode to Qav mode, 1) Switch back and forth between Qav mode and legacy mode 2) Send/recv packets in both mode. Signed-off-by: Gangfeng Huang <gan...@ni...> --- drivers/net/ethernet/intel/igb/e1000_defines.h | 21 +++ drivers/net/ethernet/intel/igb/e1000_regs.h | 7 + drivers/net/ethernet/intel/igb/igb.h | 5 + drivers/net/ethernet/intel/igb/igb_main.c | 182 +++++++++++++++++++++++- 4 files changed, 213 insertions(+), 2 deletions(-) diff --git a/drivers/net/ethernet/intel/igb/e1000_defines.h b/drivers/net/ethernet/intel/igb/e1000_defines.h index f8684aa..f09d016 100644 --- a/drivers/net/ethernet/intel/igb/e1000_defines.h +++ b/drivers/net/ethernet/intel/igb/e1000_defines.h @@ -359,6 +359,7 @@ #define MAX_JUMBO_FRAME_SIZE 0x3F00 /* PBA constants */ +#define E1000_PBA_32K 0x0020 #define E1000_PBA_34K 0x0022 #define E1000_PBA_64K 0x0040 /* 64KB */ @@ -1014,4 +1015,24 @@ #define E1000_RTTBCNRC_RF_INT_MASK \ (E1000_RTTBCNRC_RF_DEC_MASK << E1000_RTTBCNRC_RF_INT_SHIFT) +/* Queue mode, 0=strict, 1=SR mode */ +#define E1000_TQAVCC_QUEUEMODE 0x80000000 +/* Transmit mode, 0=legacy, 1=QAV */ +#define E1000_TQAVCTRL_TXMODE 0x00000001 +/* Report DMA time of tx packets */ +#define E1000_TQAVCTRL_1588_STAT_EN 0x00000004 +#define E1000_TQAVCTRL_DATA_FETCH_ARB 0x00000010 /* Data fetch arbitration */ +#define E1000_TQAVCTRL_DATA_TRAN_ARB 0x00000100 /* Data tx arbitration */ +#define E1000_TQAVCTRL_DATA_TRAN_TIM 0x00000200 /* Data launch time valid */ +/* Stall SP to guarantee SR */ +#define E1000_TQAVCTRL_SP_WAIT_SR 0x00000400 +#define E1000_TQAVCTRL_FETCH_TM_SHIFT (16) + +#define E1000_TXPBSIZE_TX0PB_SHIFT 0 +#define E1000_TXPBSIZE_TX1PB_SHIFT 6 +#define E1000_TXPBSIZE_TX2PB_SHIFT 12 +#define E1000_TXPBSIZE_TX3PB_SHIFT 18 + +#define E1000_DTXMXPKTSZ_DEFAULT 0x00000098 + #endif diff --git a/drivers/net/ethernet/intel/igb/e1000_regs.h b/drivers/net/ethernet/intel/igb/e1000_regs.h index 6f0490d..2c73e7f 100644 --- a/drivers/net/ethernet/intel/igb/e1000_regs.h +++ b/drivers/net/ethernet/intel/igb/e1000_regs.h @@ -135,6 +135,12 @@ #define E1000_FCRTC 0x02170 /* Flow Control Rx high watermark */ #define E1000_PCIEMISC 0x05BB8 /* PCIE misc config register */ +/* High credit registers where _n can be 0 or 1. */ +#define E1000_TQAVHC(_n) (0x300C + 0x40 * (_n)) +/* QAV Tx mode control registers where _n can be 0 or 1. */ +#define E1000_TQAVCC(_n) (0x3004 + 0x40 * (_n)) +#define E1000_TQAVCTRL 0x3570 /* Tx Qav Control registers */ + /* TX Rate Limit Registers */ #define E1000_RTTDQSEL 0x3604 /* Tx Desc Plane Queue Select - WO */ #define E1000_RTTBCNRM 0x3690 /* Tx BCN Rate-scheduler MMW */ @@ -201,6 +207,7 @@ #define E1000_TDFT 0x03418 /* TX Data FIFO Tail - RW */ #define E1000_TDFHS 0x03420 /* TX Data FIFO Head Saved - RW */ #define E1000_TDFPC 0x03430 /* TX Data FIFO Packet Count - RW */ +#define E1000_DTXMXPKT 0x0355C /* DMA TX Maximum Packet Size */ #define E1000_DTXCTL 0x03590 /* DMA TX Control - RW */ #define E1000_CRCERRS 0x04000 /* CRC Error Count - R/clr */ #define E1000_ALGNERRC 0x04004 /* Alignment Error Count - R/clr */ diff --git a/drivers/net/ethernet/intel/igb/igb.h b/drivers/net/ethernet/intel/igb/igb.h index c2bd4f9..b84a266 100644 --- a/drivers/net/ethernet/intel/igb/igb.h +++ b/drivers/net/ethernet/intel/igb/igb.h @@ -132,6 +132,9 @@ struct vf_data_storage { /* this is the size past which hardware will drop packets when setting LPE=0 */ #define MAXIMUM_ETHERNET_VLAN_SIZE 1522 +/* In qav mode, the maximum frame size is 1536 */ +#define IGB_MAX_QAV_FRAME_SIZE 1536 + /* Supported Rx Buffer Sizes */ #define IGB_RXBUFFER_256 256 #define IGB_RXBUFFER_2048 2048 @@ -463,6 +466,8 @@ struct igb_adapter { int copper_tries; struct e1000_info ei; u16 eee_advert; + + bool qav_mode; }; #define IGB_FLAG_HAS_MSI (1 << 0) diff --git a/drivers/net/ethernet/intel/igb/igb_main.c b/drivers/net/ethernet/intel/igb/igb_main.c index 41e2740..1d00f41 100644 --- a/drivers/net/ethernet/intel/igb/igb_main.c +++ b/drivers/net/ethernet/intel/igb/igb_main.c @@ -176,6 +176,17 @@ static int igb_ndo_get_vf_config(struct net_device *netdev, int vf, struct ifla_vf_info *ivi); static void igb_check_vf_rate_limit(struct igb_adapter *); +/* Switch qav mode and legacy mode by sysfs*/ +static void igb_setup_qav_mode(struct igb_adapter *adapter); +static void igb_setup_normal_mode(struct igb_adapter *adapter); +static ssize_t igb_get_qav_mode(struct device *dev, + struct device_attribute *attr, char *buf); +static ssize_t igb_set_qav_mode(struct device *dev, + struct device_attribute *attr, + const char *buf, size_t count); +static DEVICE_ATTR(qav_mode, S_IRUGO | S_IWUSR, + igb_get_qav_mode, igb_set_qav_mode); + #ifdef CONFIG_PCI_IOV static int igb_vf_configure(struct igb_adapter *adapter, int vf); static int igb_pci_enable_sriov(struct pci_dev *dev, int num_vfs); @@ -1600,6 +1611,11 @@ static void igb_configure(struct igb_adapter *adapter) igb_restore_vlan(adapter); + if (adapter->qav_mode) + igb_setup_qav_mode(adapter); + else + igb_setup_normal_mode(adapter); + igb_setup_tctl(adapter); igb_setup_mrqc(adapter); igb_setup_rctl(adapter); @@ -1873,8 +1889,10 @@ void igb_reset(struct igb_adapter *adapter) pba = rd32(E1000_RXPBS); pba &= E1000_RXPBS_SIZE_MASK_82576; break; - case e1000_82575: case e1000_i210: + pba = (adapter->qav_mode) ? E1000_PBA_32K : E1000_PBA_34K; + break; + case e1000_82575: case e1000_i211: default: pba = E1000_PBA_34K; @@ -2286,6 +2304,7 @@ static int igb_probe(struct pci_dev *pdev, const struct pci_device_id *ent) hw = &adapter->hw; hw->back = adapter; adapter->msg_enable = netif_msg_init(debug, DEFAULT_MSG_ENABLE); + adapter->qav_mode = false; err = -EIO; hw->hw_addr = pci_iomap(pdev, 0, 0); @@ -2531,6 +2550,15 @@ static int igb_probe(struct pci_dev *pdev, const struct pci_device_id *ent) if (err) goto err_register; + if (hw->mac.type == e1000_i210) { + err = sysfs_create_file(&netdev->dev.kobj, + &dev_attr_qav_mode.attr); + if (err) { + netdev_err(netdev, "error creating sysfs file\n"); + goto err_register; + } + } + /* carrier off reporting is important to ethtool even BEFORE open */ netif_carrier_off(netdev); @@ -2805,6 +2833,9 @@ static void igb_remove(struct pci_dev *pdev) */ igb_release_hw_control(adapter); + if (hw->mac.type == e1000_i210) + sysfs_remove_file(&netdev->dev.kobj, &dev_attr_qav_mode.attr); + unregister_netdev(netdev); igb_clear_interrupt_scheme(adapter); @@ -2886,7 +2917,12 @@ static void igb_init_queue_configuration(struct igb_adapter *adapter) break; } - adapter->rss_queues = min_t(u32, max_rss_queues, num_online_cpus()); + /* For QAV mode, always enable all queues */ + if (adapter->qav_mode) + adapter->rss_queues = max_rss_queues; + else + adapter->rss_queues = min_t(u32, max_rss_queues, + num_online_cpus()); /* Determine if we need to pair queues. */ switch (hw->mac.type) { @@ -5144,6 +5180,10 @@ static int igb_change_mtu(struct net_device *netdev, int new_mtu) return -EINVAL; } + /* For i210 Qav mode, the max frame is 1536 */ + if (adapter->qav_mode && max_frame > IGB_MAX_QAV_FRAME_SIZE) + return -EINVAL; + #define MAX_STD_JUMBO_FRAME_SIZE 9238 if (max_frame > MAX_STD_JUMBO_FRAME_SIZE) { dev_err(&pdev->dev, "MTU > 9216 not supported.\n"); @@ -8075,4 +8115,142 @@ int igb_reinit_queues(struct igb_adapter *adapter) return err; } + +static void igb_setup_qav_mode(struct igb_adapter *adapter) +{ + struct e1000_hw *hw = &adapter->hw; + u32 tqavctrl; + u32 tqavcc0, tqavcc1; + u32 tqavhc0, tqavhc1; + u32 txpbsize; + + /* reconfigure the tx packet buffer allocation */ + txpbsize = (8); + txpbsize |= (8) << E1000_TXPBSIZE_TX1PB_SHIFT; + txpbsize |= (4) << E1000_TXPBSIZE_TX2PB_SHIFT; + txpbsize |= (4) << E1000_TXPBSIZE_TX3PB_SHIFT; + + wr32(E1000_TXPBS, txpbsize); + + /* In Qav mode, the maximum sized frames of 1536 bytes */ + wr32(E1000_DTXMXPKT, IGB_MAX_QAV_FRAME_SIZE / 64); + + /* The I210 implements 4 queues, up to two queues are dedicated + * for stream reservation or priority, strict priority queuing + * while SR queue are subjected to launch time policy + */ + + tqavcc0 = E1000_TQAVCC_QUEUEMODE; /* no idle slope */ + tqavcc1 = E1000_TQAVCC_QUEUEMODE; /* no idle slope */ + tqavhc0 = 0xFFFFFFFF; /* unlimited credits */ + tqavhc1 = 0xFFFFFFFF; /* unlimited credits */ + + wr32(E1000_TQAVCC(0), tqavcc0); + wr32(E1000_TQAVCC(1), tqavcc1); + wr32(E1000_TQAVHC(0), tqavhc0); + wr32(E1000_TQAVHC(1), tqavhc1); + + tqavctrl = E1000_TQAVCTRL_TXMODE | + E1000_TQAVCTRL_DATA_FETCH_ARB | + E1000_TQAVCTRL_DATA_TRAN_TIM | + E1000_TQAVCTRL_SP_WAIT_SR; + + /* Default to a 10 usec prefetch delta from launch time - time for + * a 1500 byte rx frame to be received over the PCIe Gen1 x1 link. + */ + tqavctrl |= (10 << 5) << E1000_TQAVCTRL_FETCH_TM_SHIFT; + + wr32(E1000_TQAVCTRL, tqavctrl); +} + +static void igb_setup_normal_mode(struct igb_adapter *adapter) +{ + struct e1000_hw *hw = &adapter->hw; + + wr32(E1000_TXPBS, I210_TXPBSIZE_DEFAULT); + wr32(E1000_DTXMXPKT, E1000_DTXMXPKTSZ_DEFAULT); + wr32(E1000_TQAVCTRL, 0); +} + +static int igb_change_mode(struct igb_adapter *adapter, int request_mode) +{ + struct net_device *netdev; + int err = 0; + int current_mode; + + if (NULL == adapter) { + dev_err(&adapter->pdev->dev, "map to unbound device!\n"); + return -ENOENT; + } + + current_mode = adapter->qav_mode; + + if (request_mode == current_mode) + return 0; + + netdev = adapter->netdev; + + rtnl_lock(); + + if (netif_running(netdev)) + igb_close(netdev); + else + igb_reset(adapter); + + igb_clear_interrupt_scheme(adapter); + + adapter->qav_mode = request_mode; + + igb_init_queue_configuration(adapter); + + if (igb_init_interrupt_scheme(adapter, true)) { + dev_err(&adapter->pdev->dev, + "Unable to allocate memory for queues\n"); + err = -ENOMEM; + goto err_out; + } + + if (netif_running(netdev)) + igb_open(netdev); + + rtnl_unlock(); + + return err; +err_out: + rtnl_unlock(); + return err; +} + +static ssize_t igb_get_qav_mode(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct net_device *netdev = to_net_dev(dev); + struct igb_adapter *adapter = netdev_priv(netdev); + + return scnprintf(buf, PAGE_SIZE, "%d\n", adapter->qav_mode); +} + +static ssize_t igb_set_qav_mode(struct device *dev, + struct device_attribute *attr, + const char *buf, size_t len) +{ + struct net_device *netdev = to_net_dev(dev); + struct igb_adapter *adapter = netdev_priv(netdev); + int request_mode, err; + + if (!capable(CAP_NET_ADMIN)) + return -EPERM; + + if (0 > kstrtoint(buf, 0, &request_mode)) + return -EINVAL; + + if (request_mode != 0 && request_mode != 1) + return -EINVAL; + + err = igb_change_mode(adapter, request_mode); + if (err) + return err; + + return len; +} /* igb_main.c */ -- 1.7.9.5 |
From: Gangfeng H. <gan...@ni...> - 2015-10-29 02:01:42
|
The Intel Ethernet Server Adapter I210 supports IEEE 802.1Qav Audio-Video Bridging (AVB) for users requiring tightly controlled media stream synchronization, buffering, and reservation. The 802.1Qav is part of the AVB specification that provides a way to guarantee bounded latency and latency variation for time sensitive traffic. Reference: https://github.com/AVnu/Open-AVB/tree/master/kmod/igb Gangfeng Huang (2): igb: add function to set I210 transmit mode igb: add a character device to support AVB drivers/net/ethernet/intel/igb/Makefile | 2 +- drivers/net/ethernet/intel/igb/e1000_defines.h | 22 + drivers/net/ethernet/intel/igb/e1000_regs.h | 7 + drivers/net/ethernet/intel/igb/igb.h | 19 +- drivers/net/ethernet/intel/igb/igb_cdev.c | 511 ++++++++++++++++++++++++ drivers/net/ethernet/intel/igb/igb_cdev.h | 45 +++ drivers/net/ethernet/intel/igb/igb_main.c | 286 ++++++++++++- 7 files changed, 877 insertions(+), 15 deletions(-) create mode 100644 drivers/net/ethernet/intel/igb/igb_cdev.c create mode 100644 drivers/net/ethernet/intel/igb/igb_cdev.h -- 1.7.9.5 |
From: Christoph M. <era...@gm...> - 2015-10-28 12:48:31
|
Hi Raanan > The only way to query coalesce options is via the "ethtool -c" command. > I don't know if it helps you, but are you familiar with the driver param: InterruptThrottleRate. > Run: "modinfo e1000e" command, and you can also look at the param.c file. Thanks for the hint. I have used this for some time, but then we started to equip some computers with NICs using the igb driver. This driver no longer supports said parameters and so I switched over to ethtool completely. My "fix" is to set the coalesce options before I enable the interface. But this does of course nothing for the real problem, so others can still run into this trap. Christoph |
From: J.H <e10...@gm...> - 2015-10-28 12:08:22
|
Hi, I'm trying to do i82580 forced link up. (1G fiber) My current configuration is that RX port 0 is connected to IXIA packet generator tx port and TX port 0 remained not connected to anything. I wish the packets received from the RX port 0 to be transmitted to TX port 1 which is connected to rx port of IXIA generator. However, the RX port 1 is not connected to anything. I set the PCS link control registers in the two ports in order for their links to be up as forced method. PCS link control registers are set with bits FLV, FSV, FDV, FSD, Force link, etc. The links of the two port were up successfully 1G and Full-duplex without auto-negotiation, My problem is as follows : Data is well received from IXIA through RX port0, and my DPDK application says the data are sent through the TX port1. But IXIA generator cannot receive anything form the my TX port1. How can I fix this problem? Thanks in advance. J.H |
From: Avargil, R. <raa...@in...> - 2015-10-28 10:31:27
|
Hi Christoph, The only way to query coalesce options is via the "ethtool -c" command. I don't know if it helps you, but are you familiar with the driver param: InterruptThrottleRate. Run: "modinfo e1000e" command, and you can also look at the param.c file. Thanks, Raanan -----Original Message----- From: Christoph Mathys [mailto:era...@gm...] Sent: Tuesday, October 27, 2015 17:00 To: e10...@li... Subject: [E1000-devel] [e1000e] Mismatch between coalesce settings from ethtool and effective settings? Hi there Is there some way to read the actual values for the coalesce options back from the NIC? To be specific, I'm interested in the value of ITR on a 82574L chip. I suspect it does not match with the value that I get from "ethtool -c". Reading back from userspace, without compiling a lot of stuff if possible... Some background: I use ifupdown from Debian to configure an Ethernet interface. I want to disable all coalesce settings. What I've done is to call ethtool with the required parameter from a script in if-up.d, which is run right after the interface gets set to up (using something like ifconfig up). However, judging from my applications behaviour I think that the default "rx-usecs 3" is still active, even though ethtool -c tells me it is set to '0'. After changing it to "1" and then back to "0", my application works properly (change to one is required because ethtool will not write the same value again). I suspect some kind of race between the kernel thread bringing up the interface and my script calling ethtool, and I hope to verify this by seeing the actual values of coalesce that are used. I use a dated 3.12 with PREEMPT_RT patch. Christoph PS Please CC ------------------------------------------------------------------------------ _______________________________________________ E1000-devel mailing list E10...@li... https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired --------------------------------------------------------------------- Intel Israel (74) Limited This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. |
From: Christoph M. <era...@gm...> - 2015-10-27 14:59:44
|
Hi there Is there some way to read the actual values for the coalesce options back from the NIC? To be specific, I'm interested in the value of ITR on a 82574L chip. I suspect it does not match with the value that I get from "ethtool -c". Reading back from userspace, without compiling a lot of stuff if possible... Some background: I use ifupdown from Debian to configure an Ethernet interface. I want to disable all coalesce settings. What I've done is to call ethtool with the required parameter from a script in if-up.d, which is run right after the interface gets set to up (using something like ifconfig up). However, judging from my applications behaviour I think that the default "rx-usecs 3" is still active, even though ethtool -c tells me it is set to '0'. After changing it to "1" and then back to "0", my application works properly (change to one is required because ethtool will not write the same value again). I suspect some kind of race between the kernel thread bringing up the interface and my script calling ethtool, and I hope to verify this by seeing the actual values of coalesce that are used. I use a dated 3.12 with PREEMPT_RT patch. Christoph PS Please CC |
From: Sowmini V. <sow...@or...> - 2015-10-23 11:16:51
|
hi, I have a setup that uses ipsec/esp, and in theory, since the SPI falls in the same byte location as the tcp ports, one ought to be able to set up the rx-flow-hash etc based on the spi. But when I try something like this: # ethtool -N eth13 rx-flow-hash esp4 sdfn Cannot change RX network flow hashing options: Invalid argument I see that is error-ing out (for me) in ixgbe_set_rss_hash_opt() for nfc->flow_type == 4 (AH_ESP_V4_FLOW). But is there a significant technical reason why this cannot follow the same logic as TCP_V4_FLOW or TCP_V6_FLOW, i.e., use "sdfn" for classification (my reading of the code in that function is that tcp v4/v6 flows do sdfn by default). Related question that motivates this, has anyone looked into whether sdfn would improve perf for esp? --Sowmini |
From: Christian R. <id...@qa...> - 2015-10-23 07:34:13
|
On 2015-10-21 01:30, Rose, Gregory V wrote: >> -----Original Message----- >> From: Christian Ruppert [mailto:id...@qa...] >> Sent: Thursday, October 15, 2015 4:02 AM >> To: Rose, Gregory V >> Cc: Fujinaka, Todd; e10...@li... >> Subject: RE: [E1000-devel] X710 / i40e interface resets >> >> On 2015-10-08 17:05, Rose, Gregory V wrote: >> >> -----Original Message----- >> >> From: Christian Ruppert [mailto:id...@qa...] >> >> Sent: Thursday, October 08, 2015 7:34 AM >> >> To: Rose, Gregory V >> >> Cc: Fujinaka, Todd; e10...@li... >> >> Subject: RE: [E1000-devel] X710 / i40e interface resets >> > Please try turning off TSO/GSO to see if that helps. >> >> That seems to help. No more resets. Do you want me to enable TSO or >> GSO >> again to see which one exactly cause those resets? Or do you want me >> to do >> anything else? >> > > Sorry for the delayed response but I've been out of the office for the > last week. No problem! ;) > > Yes, please do try disabling TSO and GSO separately - I suspect that > internally to the driver it won't make a difference but if it does > then it would be good to know. You seem to be right, no resets, nothing unusual currently. > > Thanks! > > - Greg -- Regards, Christian Ruppert |
From: Rose, G. V <gre...@in...> - 2015-10-20 23:30:13
|
> -----Original Message----- > From: Christian Ruppert [mailto:id...@qa...] > Sent: Thursday, October 15, 2015 4:02 AM > To: Rose, Gregory V > Cc: Fujinaka, Todd; e10...@li... > Subject: RE: [E1000-devel] X710 / i40e interface resets > > On 2015-10-08 17:05, Rose, Gregory V wrote: > >> -----Original Message----- > >> From: Christian Ruppert [mailto:id...@qa...] > >> Sent: Thursday, October 08, 2015 7:34 AM > >> To: Rose, Gregory V > >> Cc: Fujinaka, Todd; e10...@li... > >> Subject: RE: [E1000-devel] X710 / i40e interface resets > > Please try turning off TSO/GSO to see if that helps. > > That seems to help. No more resets. Do you want me to enable TSO or GSO > again to see which one exactly cause those resets? Or do you want me to do > anything else? > Sorry for the delayed response but I've been out of the office for the last week. Yes, please do try disabling TSO and GSO separately - I suspect that internally to the driver it won't make a difference but if it does then it would be good to know. Thanks! - Greg |
From: Reverend F. G. <664...@qq...> - 2015-10-16 22:41:38
|
I am Reverend Frank Gorenc an attorney at law and personal lawyer to a late expatriate. I have a viable project which i intend to specifically discuss with you, get back to me so i can tell you more about it. Please give this information utmost priority. Sincerely Yours Reverend Frank Gorenc |
From: Mattias B. <mat...@gm...> - 2015-10-16 19:25:42
|
It works now! Thanks Alexander for your help! I really appreciate it and for me it has been very great value. On Tue, Oct 13, 2015 at 9:47 PM, Mattias Barthel <mat...@gm...> wrote: > Thanks again Alex, > It might be a good idea to upgrade the driver, who knows what more there > is in store for me using this one. > > Anyways, regarding the timestamping in each rx buffer I think I found it. > > From the I350 datasheet: > ----- > When the TSAUXC.Disable systime bit is cleared and the SRRCTL[n].Timestamp > bit is set to 1, packets received to the queue will be time stamped if > they meet one of the following conditions: > — Meet the criteria defined in the TSYNCRXCTL.Type field (See Section > 8.16.1 and Section 8.16.26). > — Match the value defined in one of the ETQF registers with the 1588 time > stamp bit set (See Section 7.1.2.4) if TSYNCRXCTL.Type field defines time > stamping of L2 packets. > > — Match a 2-tuple filter with the TTQF.1588 time stamp set (See Section > 7.1.2.5) if TSYNCRXCTL.Type field defines time stamping of L4 packets. > > When detecting a receive packet that should be time stamped, the I350 > will: > • Place a 64 bit timestamp, indicating the time a packet was received by > the MAC, at the beginning of the receive buffer before the received packet. > • Set the TSIP bit in the RDESC.STATUS field of the last receive > descriptor. > • Update the RDESC.Packet Type field in the last receive descriptor. Value > in this field enables identifying that this is a PTP (Precision Time > Protocol) packet (this indication is only relevant for L2 packets). > • Update the RDESC.HDR_LEN and RDESC.PKT_LEN values to include size of > timestamp. > ----- > In my driver I found this: > > if (hw->mac.type == e1000_82580) > srrctl |= E1000_SRRCTL_TIMESTAMP; > > So again, its only being done if its 82580. I will try to set this bit > tomorrow for the I350 before trying any driver upgrades which can be quite > painful. > > Thank again! > > On Tue, Oct 13, 2015 at 8:45 PM, Alexander Duyck < > ale...@gm...> wrote: > >> You can probably just download and use the latest driver from: >> http://sourceforge.net/projects/e1000/files/igb%20stable/ >> >> It should be able to build on a 3.0.8 kernel provided nothing has been >> back-ported that would change the APIs. >> >> - Alex >> >> On 10/13/2015 10:26 AM, Mattias Barthel wrote: >> >>> Thanks a lot Alex, there might still be some hope then. >>> Unfortunately Im bound to using the linux kernel 3.0.8. What out-of-tree >>> driver can I use with that, please? >>> >>> On Tuesday, October 13, 2015, Alexander Duyck <ale...@gm... >>> <mailto:ale...@gm...>> wrote: >>> >>> I don't know if it is available in that driver or not, but there >>> is an option in later igb drivers to support a time stamp being >>> written with the packet data. Based on the fact that you are >>> already seeing issues, you might want to try getting the >>> out-of-tree driver from e1000.sf.net <http://e1000.sf.net> and >>> >>> seeing if you can use that with the time stamp in packet option >>> that is enabled when you select to timestamp all incoming Rx packets. >>> >>> - Alex >>> >>> On 10/13/2015 10:01 AM, Mattias Barthel wrote: >>> >>> Yes, setting the cycles.shift to the same as on the 82580 and >>> also shifting the reads accordingly everything works. The sync >>> of shhwtstamps->syststamp too. >>> >>> My driver version is 3.0.6-k2 and linux is 3.0.8. >>> >>> Although this might not be very suited to my needs. >>> The incoming buffers wont be timestamped if you cant keep up >>> with reading the timestamping registers as I understand it. >>> This might be perfectly fine in the case of PTP where the >>> packet rate is fairly low. In my case Im looking at some >>> 80kpps that all would need to be timestamped. >>> There is a comment about this in the driver: >>> >>> *//* If this bit is set, then the RX registers contain the >>> time stamp. No/**/ other packet will be time stamped until we >>> read these registers, so/* */read the registers to make them >>> available again. Because only one/**/ packet can be time >>> stamped at a time, we know that the register/* */values must >>> belong to this one here and therefore we don't need to/**/ >>> compare any of the additional attributes stored for it./* */If >>> nothing went wrong, then it should have a shared tx_flags that >>> we/**/ can turn into a skb_shared_hwtstamps. *//* >>> *//* >>> *//* >>> >>> On Tuesday, October 13, 2015, Alexander Duyck >>> <ale...@gm... <mailto:ale...@gm...>> >>> wrote: >>> >>> I'm assuming you are using either an older kernel or an >>> out-of-tree driver do I have that right? I ask because >>> this code >>> doesn't resemble what is is currently in the latest Linux >>> kernel. >>> >>> Comments on what you have found are inline below. >>> >>> - Alex >>> >>> On 10/13/2015 12:53 AM, Mattias Barthel wrote: >>> >>> Hi, I think I managed to get around this problem. >>> >>> >>> in function init_hw_timer(): >>> ---- >>> case e1000_i350: >>> case e1000_82580: >>> >>> /* >>> * The 82580 timesync updates the system timer every >>> 8ns by 8ns >>> * and the value cannot be shifted. Instead we need to >>> shift >>> * the registers to generate a 64bit timer value. As a >>> result >>> * SYSTIMR/L/H, TXSTMPL/H, RXSTMPL/H all have to be >>> shifted by >>> * 24 in order to generate a larger value for >>> synchronization. >>> */ >>> adapter->cycles.shift = IGB_82580_TSYNC_SHIFT; >>> >>> ---- >>> >>> I have the I350 and I dont think this cycles shift was >>> meant >>> for my adapter. >>> IGB_82580_TSYNC_SHIFT is 24. >>> >>> Also when doing igb_read_clock, >>> this shifting is applied only for the 82580 and I guess >>> it >>> should not have been set in the first place for the I350: >>> -------- >>> /* >>> * The timestamp latches on lowest register read. For >>> the 82580 >>> * the lowest register is SYSTIMR instead of SYSTIML. >>> However >>> we never >>> * adjusted TIMINCA so SYSTIMR will just read as all 0s so >>> ignore it. >>> */ >>> if (hw->mac.type == e1000_82580) { >>> stamp = rd32(E1000_SYSTIMR) >> 8; >>> shift = IGB_82580_TSYNC_SHIFT; >>> } >>> >>> stamp |= (u64)rd32(E1000_SYSTIML) << shift; >>> stamp |= (u64)rd32(E1000_SYSTIMH) << (shift + 32); >>> ------------- >>> >>> >>> This is a bug in the code. The i350 should be shifted by >>> the same >>> value as the 82580. Then the result would be the same as >>> if you >>> had set the shift to 0. As I recall the reasoning behind the >>> change was to make use of as many bits of the time as >>> possible, >>> and on 82580 and newer the SYSTIMH portion of the clock only >>> contained 8 bits of actual clock data. >>> >>> Without that change I believe there is a risk of the >>> counter being >>> considered to have cycled too fast and causing errors as >>> the cycle >>> counter will roll over at 40 bits instead of 64. The >>> alternative >>> would be to update the cycle counter you are using so that >>> it is >>> aware that it is limited to 40 bits and then use an shift >>> of 0. >>> >>> Setting adapter->cycles.shift to 0 makes the clock run at >>> about the same speed as the system clock. >>> >>> Next problem I have is that the syststamp is not syncing >>> correctly against the system clock. >>> >>> Any help is much appreciated. >>> >>> Regards, >>> >>> Mattias >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> On Mon, Oct 12, 2015 at 5:04 PM, Mattias Barthel >>> <mat...@gm... >>> <mailto:mat...@gm...>> >>> wrote: >>> >>> Hi again, >>> >>> Started to try and timestamp all packets in the I350. >>> >>> Driver version: >>> >>> driver: igb >>> version: 3.0.6-k2 >>> firmware-version: 0.147-0 >>> >>> Doing an ioctl that turns on timestamping for all >>> packets. >>> >>> hwconfig.tx_type = HWTSTAMP_TX_OFF; >>> hwconfig.rx_filter = HWTSTAMP_FILTER_ALL; >>> hwconfig_requested = hwconfig; >>> ioctl(sock, SIOCSHWTSTAMP, &hwtstamp) >>> >>> The strangest thing is that although it works to >>> read the >>> timestamps the clock seems to advance very slowly >>> and/or >>> incorrectly. >>> >>> I just printout the HW timestamps alongside the >>> systemclock and >>> the hw stamping is advancing the nanoseconds when the >>> systemclock >>> is advancing us and seconds. >>> >>> Is there a know bug in this version of the driver >>> that maybe >>> initializes the clock incorrectly or maybe reads >>> incorrectly from >>> the clock? >>> >>> >>> [ 63.004116] igb_rx_hwtstamp >>> [ 63.004684] hwtstamp: 1444656871372825302, >>> syststamp: >>> 1444656871372845807 >>> [ 63.005721] system timestamp: tv_sec: 1444656793, >>> tv_usec: 683825 >>> [ 63.006631] sys hw timestamp: tstamp: >>> 1444656871372845807, >>> tv_sec: 1444656871, tv_usec: 372845 >>> [ 63.008044] hw timestamp: tstamp: >>> 1444656871372825302, >>> tv_sec: >>> 1444656871, tv_usec: 372825 >>> [ 63.009395] >>> [ 63.104111] igb_rx_hwtstamp >>> [ 63.104704] hwtstamp: 1444656871372825308, >>> syststamp: >>> 1444656871372845813 >>> [ 63.105747] system timestamp: tv_sec: 1444656793, >>> tv_usec: 783851 >>> [ 63.106669] sys hw timestamp: tstamp: >>> 1444656871372845813, >>> tv_sec: 1444656871, tv_usec: 372845 >>> [ 63.108075] hw timestamp: tstamp: >>> 1444656871372825308, >>> tv_sec: >>> 1444656871, tv_usec: 372825 >>> [ 63.109428] >>> [ 63.204109] igb_rx_hwtstamp >>> [ 63.204683] hwtstamp: 1444656871372825314, >>> syststamp: >>> 1444656871372845819 >>> [ 63.206018] system timestamp: tv_sec: 1444656793, >>> tv_usec: 884122 >>> [ 63.207242] sys hw timestamp: tstamp: >>> 1444656871372845819, >>> tv_sec: 1444656871, tv_usec: 372845 >>> [ 63.209096] hw timestamp: tstamp: >>> 1444656871372825314, >>> tv_sec: >>> 1444656871, tv_usec: 372825 >>> [ 63.210897] >>> [ 63.304068] igb_rx_hwtstamp >>> [ 63.304849] hwtstamp: 1444656871372825320, >>> syststamp: >>> 1444656871372845825 >>> [ 63.306176] system timestamp: tv_sec: 1444656793, >>> tv_usec: 984280 >>> [ 63.307391] sys hw timestamp: tstamp: >>> 1444656871372845825, >>> tv_sec: 1444656871, tv_usec: 372845 >>> [ 63.309252] hw timestamp: tstamp: >>> 1444656871372825320, >>> tv_sec: >>> 1444656871, tv_usec: 372825 >>> [ 63.311032] >>> [ 63.404107] igb_rx_hwtstamp >>> [ 63.404659] hwtstamp: 1444656871372825326, >>> syststamp: >>> 1444656871372845831 >>> [ 63.405695] system timestamp: tv_sec: 1444656794, >>> tv_usec: 83798 >>> [ 63.406635] sys hw timestamp: tstamp: >>> 1444656871372845831, >>> tv_sec: 1444656871, tv_usec: 372845 >>> [ 63.408044] hw timestamp: tstamp: >>> 1444656871372825326, >>> tv_sec: >>> 1444656871, tv_usec: 372825 >>> [ 63.409405] >>> [ 63.504084] igb_rx_hwtstamp >>> [ 63.504670] hwtstamp: 1444656871372825332, >>> syststamp: >>> 1444656871372845837 >>> [ 63.505715] system timestamp: tv_sec: 1444656794, >>> tv_usec: 183819 >>> [ 63.506657] sys hw timestamp: tstamp: >>> 1444656871372845837, >>> tv_sec: 1444656871, tv_usec: 372845 >>> [ 63.508090] hw timestamp: tstamp: >>> 1444656871372825332, >>> tv_sec: >>> 1444656871, tv_usec: 372825 >>> >>> >>> Any help will be very much appreciated. >>> >>> Best regards, >>> >>> Mattias >>> >>> >>> >>> >>> >>> >>> >>> >>> On Thu, Oct 8, 2015 at 6:01 PM, Alexander Duyck >>> <ale...@gm... >>> <mailto:ale...@gm...>> wrote: >>> >>> The i350 should support timestamping all >>> packets as I >>> recall. You should probably look into getting the >>> linuxptp package and >>> seeing if you can make use of the hwstamp_ctl >>> command to >>> enable timestamping for all incoming packets >>> on the >>> adapter. >>> >>> - Alex >>> >>> On 10/08/2015 12:33 AM, Mattias Barthel wrote: >>> >>> Hi >>> thanks for answer. >>> >>> Unfortunately, the udp/ip stream just as >>> the cfm >>> stream is >>> against only one peer. >>> Furthemore on the guest side I have only >>> one vcpu >>> tied to >>> two physical siblings to minimize L2 cache >>> misses >>> on ISRs >>> due to cache coherence. >>> >>> I had an idea this morning, maybe I can >>> make use >>> of the >>> ptp hw timestaming for this? This should give >>> better accuracy. >>> Is this possible or does it only work for >>> ethertype ptp? >>> >>> On Thursday, October 8, 2015, Alexander Duyck >>> <ale...@gm... >>> <mailto:ale...@gm...> >>> <mailto:ale...@gm... >>> <mailto:ale...@gm...>>> wrote: >>> >>> The difference could be Receive Side >>> Scaling >>> (RSS). The UDP/IP >>> packets >>> can be load balanced across all queues >>> via RSS >>> based >>> on a hash of the >>> source and destination addresses. So >>> if you have >>> multiple sources >>> generating these packets, or they are >>> being >>> received >>> on multiple IP >>> addresses then it is possible the load is >>> being spread >>> out over >>> multiple >>> CPUs. Your CFM packets are on an >>> Ethertype >>> other than >>> IPv4 or IPv6 so >>> all packets will end up on queue 0 if >>> I am not >>> mistaken. >>> >>> - Alex >>> >>> On 10/07/2015 10:21 PM, Mattias >>> Barthel wrote: >>> > Hi again! >>> > >>> > I realise my message might not have >>> been so >>> clear. >>> > >>> > I will try to clarify. I am doing PM >>> with >>> software >>> timestamping. >>> > >>> > igb versions: >>> > version: 3.0.6-k2 >>> > firmware-version: 0.147-0 >>> > >>> > I do timestamping (NTP format) of the >>> packets in the >>> same place >>> for all >>> > protocols. (in igb_clean_rx_irq_adv()). >>> > >>> > For these non ip packets like IEEE >>> 801.2ag CFM ETH-DM >>> (ethertype 0x8902) I >>> > get quite a lot worse accuracy in the >>> timestamping. >>> > >>> > For UDP/IP TWAMP I get for 10k pps a >>> max >>> delay of >>> around 140us >>> back to back. >>> > For CFM ETH DM I get for the same >>> packet >>> rate and >>> packet size a >>> max delay >>> > of around 900us. >>> > >>> > So my hypothesis was: >>> > Could FW be putting these L2 packets >>> in another >>> queue with different >>> > characteristics? >>> > >>> > I tried to add a ET-type filter as >>> written in my >>> previous mail >>> but it >>> > showed no difference. >>> > >>> > >>> > Thanks for any help given! >>> > >>> > >>> > >>> > On Mon, Oct 5, 2015 at 2:19 PM, >>> Mattias Barthel >>> <mat...@gm... >>> <mailto:mat...@gm...> >>> <javascript:;>> >>> >>> > wrote: >>> > >>> >> Greetings, >>> >> >>> >> Im getting quite bad latency when >>> using the igb >>> driver on i350 >>> on linux >>> >> regarding >>> >> ETH CFM packets. This compared to >>> TWAMP >>> (IPv4 UDP >>> packets). >>> >> The environment is KVM with the >>> i350 devices in >>> PCI-passthrough. >>> >> >>> >> So i figured I add an own rx-queue >>> (filter) for >>> those types of >>> ethernet >>> >> protocol packets. >>> >> >>> >> This is what i added: >>> >> >>> >> >>> >> wr32(E1000_ETQF(4), >>> >> (1 << 31) | /* queue enable */ >>> >> (E1000_ETQF_FILTER_ENABLE) | /* enable >>> filter */ >>> >> (1 << 29) | /* enable immediate >>> interrupt */ >>> >> (0x4 << 16) | /* queue no. 4 */ >>> >> (ETH_P_8021AG)); /* 0x8902 >>> CFM eth >>> protocol type */ >>> >> >>> >> >>> >> ETH_P_8021A is 0x8902 >>> >> >>> >> >>> >> >>> >> Is this a correct/good approach? >>> >> >>> >> Thanks in advance! >>> >> >>> >> -- >>> >> Mattias Barthel >>> >> >>> > >>> >>> >>> >>> ------------------------------------------------------------------------------ >>> _______________________________________________ >>> E1000-devel mailing list >>> E10...@li... >>> <mailto:E10...@li...> >>> <javascript:;> >>> https://lists.sourceforge.net/lists/listinfo/e1000-devel >>> To learn more about Intel® >>> Ethernet, visit >>> http://communities.intel.com/community/wired >>> >>> >>> Avis de confidentialité >>> >>> Les informations contenues dans le présent >>> message >>> et dans >>> toute pièce qui lui est jointe sont >>> confidentielles et >>> peuvent être protégées par le secret >>> professionnel. Ces >>> informations sont à l’usage exclusif de >>> son ou de ses >>> destinataires. Si vous recevez ce message >>> par erreur, >>> veuillez s’il vous plait communiquer >>> immédiatement >>> avec >>> l’expéditeur et en détruire tout >>> exemplaire. De >>> plus, il >>> vous est strictement interdit de le >>> divulguer, de le >>> distribuer ou de le reproduire sans >>> l’autorisation de >>> l’expéditeur. Merci. >>> >>> Confidentiality notice >>> >>> This e-mail message and any attachment >>> hereto contain >>> confidential information which may be >>> privileged >>> and which >>> is intended for the exclusive use of its >>> addressee(s). If >>> you receive this message in error, please >>> inform >>> sender >>> immediately and destroy any copy thereof. >>> Furthermore, any >>> disclosure, distribution or copying of this >>> message and/or >>> any attachment hereto without the consent >>> of the >>> sender is >>> strictly prohibited. Thank you. >>> >>> >>> >>> >>> >>> -- Mattias Barthel >>> >>> >>> >>> >>> -- Mattias Barthel >>> >>> >>> >>> Avis de confidentialité >>> >>> Les informations contenues dans le présent message et dans >>> toute pièce qui lui est jointe sont confidentielles et peuvent >>> être protégées par le secret professionnel. Ces informations >>> sont à l’usage exclusif de son ou de ses destinataires. Si >>> vous recevez ce message par erreur, veuillez s’il vous plait >>> communiquer immédiatement avec l’expéditeur et en détruire >>> tout exemplaire. De plus, il vous est strictement interdit de >>> le divulguer, de le distribuer ou de le reproduire sans >>> l’autorisation de l’expéditeur. Merci. >>> >>> Confidentiality notice >>> >>> This e-mail message and any attachment hereto contain >>> confidential information which may be privileged and which is >>> intended for the exclusive use of its addressee(s). If you >>> receive this message in error, please inform sender >>> immediately and destroy any copy thereof. Furthermore, any >>> disclosure, distribution or copying of this message and/or any >>> attachment hereto without the consent of the sender is >>> strictly prohibited. Thank you. >>> >>> >>> >>> Avis de confidentialité >>> >>> Les informations contenues dans le présent message et dans toute pièce >>> qui lui est jointe sont confidentielles et peuvent être protégées par le >>> secret professionnel. Ces informations sont à l’usage exclusif de son ou de >>> ses destinataires. Si vous recevez ce message par erreur, veuillez s’il >>> vous plait communiquer immédiatement avec l’expéditeur et en détruire tout >>> exemplaire. De plus, il vous est strictement interdit de le divulguer, de >>> le distribuer ou de le reproduire sans l’autorisation de l’expéditeur. >>> Merci. >>> >>> Confidentiality notice >>> >>> This e-mail message and any attachment hereto contain confidential >>> information which may be privileged and which is intended for the exclusive >>> use of its addressee(s). If you receive this message in error, please >>> inform sender immediately and destroy any copy thereof. Furthermore, any >>> disclosure, distribution or copying of this message and/or any attachment >>> hereto without the consent of the sender is strictly prohibited. Thank you. >>> >>> >> > > > -- > Mattias Barthel > -- Mattias Barthel |
From: Dementia 2. C. <ev...@gc...> - 2015-10-15 14:15:09
|
Dementia 2020: Transforming Care, Support & Research 12 April, 2016, Church House, Westminster 13 Confirmed Speakers Govconnect are please to announce its inaugural Dementia 2020 Conference will be held at Church House, Westminster on 12th April 2016. The Dementia 2020 strategy published on 21st Feb 2015; This conference will examine progress to date, and dissect the aspirations contained within the strategy. Govconnects Dementia 2020 series of conferences will develop a community of conference delegates online users and wider stakeholders who will seek to ensure that the vision to create a society by 2020 where every person with dementia in every area of the country receives high quality compassionate care from diagnosis through to end of life care. Register to Attend http://www.gconnect.org.uk/link.php?M=901713&N=78&L=11&F=T View Agenda http://www.gconnect.org.uk/link.php?M=901713&N=78&L=12&F=T Confirmed Speakers Include Professor Alistair Burns CBE National Clinical Director for Dementia NHS England Gillian Leng CBE Director for Health and Social Care National Institute for Health and Care Excellence (NICE) George McNamara Head of Policy and Public Affairs Alzheimer`s Society Martin Rossor National Director for Dementia Research National Institute for Health Research (NHIR) Dr Eric Karran Director of Research Alzheimer’s Research UK Andrea Sutcliffe Chief Inspector of Adult Social Care Care Quality Commission Rob Daniels, Director Tel: +44 (0) 161 246 6240 Govconnect, 5300 lakeside, SK8 3GP Tel: +44 (0) 161 246 6240 Registered Number 9688781 Useful Tip Adding our domain @gconnect.org.uk to your safe senders list will automatically ensure images are turned on in the emails you receive from us. Unsubscribe If you would like to opt-out of receiving emails at this email address, then please click http://www.gconnect.org.uk/link.php?M=901713&N=78&L=27&F=T to unsubscribe. |
From: Christian R. <id...@qa...> - 2015-10-15 11:02:09
|
On 2015-10-08 17:05, Rose, Gregory V wrote: >> -----Original Message----- >> From: Christian Ruppert [mailto:id...@qa...] >> Sent: Thursday, October 08, 2015 7:34 AM >> To: Rose, Gregory V >> Cc: Fujinaka, Todd; e10...@li... >> Subject: RE: [E1000-devel] X710 / i40e interface resets >> >> On 2015-10-08 16:23, Rose, Gregory V wrote: >> >> -----Original Message----- >> >> From: Christian Ruppert [mailto:id...@qa...] >> >> Sent: Wednesday, October 07, 2015 10:54 AM >> >> To: Rose, Gregory V >> >> Cc: Fujinaka, Todd; e10...@li... >> >> Subject: RE: [E1000-devel] X710 / i40e interface resets >> >> >> >> On 2015-10-07 18:49, Rose, Gregory V wrote: >> >> >> -----Original Message----- >> >> >> From: Christian Ruppert [mailto:id...@qa...] >> >> >> Sent: Wednesday, October 07, 2015 8:00 AM >> >> >> To: Rose, Gregory V >> >> >> Cc: Fujinaka, Todd; e10...@li... >> >> >> Subject: RE: [E1000-devel] X710 / i40e interface resets >> >> >> >> >> > >> > >> > [snip] >> > >> >> > Maybe, but first let me ask you if you're using the VEPA mode work >> >> > around for bonding? >> >> > >> >> >> >> No, just the basic bonding in 802.3ad mode. We don't use VEPA at all. >> > >> > Can you send me the output of the command "bridge link show"? >> >> 3: eth0 state UP : <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 >> master >> bond0 hwmode VEPA >> 5: eth1 state UP : <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 >> master >> bond0 hwmode VEPA >> >> hm, now that's interesting... > > That is exactly what we want to see. The XL710/X710 controllers have > an internal switch for when SR-IOV mode is used so that all the VFs on > that controller can communicate with each other across the bridge. If > the hwmode had been VEB then that would indicate a problem. However, > you're in the correct internal switch mode for LACP operation. > > Please try turning off TSO/GSO to see if that helps. That seems to help. No more resets. Do you want me to enable TSO or GSO again to see which one exactly cause those resets? Or do you want me to do anything else? > > If not can I get you to enter a bug ticket at the Intel SF site? > > Please include the following: > > Results of 'ethtool -i' on each X710 interface. > Results of 'ehtool' on each X710 interface. > Results of 'ip link show' and 'ip addr show'. > The system log obtained via 'dmesg' *after* the error has occurred. > > That should get us started. > > Thanks! > > - Greg > >> >> > >> > Thanks, >> > >> > - Greg >> >> -- >> Regards, >> Christian Ruppert -- Regards, Christian Ruppert |
From: Mattias B. <mat...@gm...> - 2015-10-13 19:47:16
|
Thanks again Alex, It might be a good idea to upgrade the driver, who knows what more there is in store for me using this one. Anyways, regarding the timestamping in each rx buffer I think I found it. >From the I350 datasheet: ----- When the TSAUXC.Disable systime bit is cleared and the SRRCTL[n].Timestamp bit is set to 1, packets received to the queue will be time stamped if they meet one of the following conditions: — Meet the criteria defined in the TSYNCRXCTL.Type field (See Section 8.16.1 and Section 8.16.26). — Match the value defined in one of the ETQF registers with the 1588 time stamp bit set (See Section 7.1.2.4) if TSYNCRXCTL.Type field defines time stamping of L2 packets. — Match a 2-tuple filter with the TTQF.1588 time stamp set (See Section 7.1.2.5) if TSYNCRXCTL.Type field defines time stamping of L4 packets. When detecting a receive packet that should be time stamped, the I350 will: • Place a 64 bit timestamp, indicating the time a packet was received by the MAC, at the beginning of the receive buffer before the received packet. • Set the TSIP bit in the RDESC.STATUS field of the last receive descriptor. • Update the RDESC.Packet Type field in the last receive descriptor. Value in this field enables identifying that this is a PTP (Precision Time Protocol) packet (this indication is only relevant for L2 packets). • Update the RDESC.HDR_LEN and RDESC.PKT_LEN values to include size of timestamp. ----- In my driver I found this: if (hw->mac.type == e1000_82580) srrctl |= E1000_SRRCTL_TIMESTAMP; So again, its only being done if its 82580. I will try to set this bit tomorrow for the I350 before trying any driver upgrades which can be quite painful. Thank again! On Tue, Oct 13, 2015 at 8:45 PM, Alexander Duyck <ale...@gm...> wrote: > You can probably just download and use the latest driver from: > http://sourceforge.net/projects/e1000/files/igb%20stable/ > > It should be able to build on a 3.0.8 kernel provided nothing has been > back-ported that would change the APIs. > > - Alex > > On 10/13/2015 10:26 AM, Mattias Barthel wrote: > >> Thanks a lot Alex, there might still be some hope then. >> Unfortunately Im bound to using the linux kernel 3.0.8. What out-of-tree >> driver can I use with that, please? >> >> On Tuesday, October 13, 2015, Alexander Duyck <ale...@gm... >> <mailto:ale...@gm...>> wrote: >> >> I don't know if it is available in that driver or not, but there >> is an option in later igb drivers to support a time stamp being >> written with the packet data. Based on the fact that you are >> already seeing issues, you might want to try getting the >> out-of-tree driver from e1000.sf.net <http://e1000.sf.net> and >> >> seeing if you can use that with the time stamp in packet option >> that is enabled when you select to timestamp all incoming Rx packets. >> >> - Alex >> >> On 10/13/2015 10:01 AM, Mattias Barthel wrote: >> >> Yes, setting the cycles.shift to the same as on the 82580 and >> also shifting the reads accordingly everything works. The sync >> of shhwtstamps->syststamp too. >> >> My driver version is 3.0.6-k2 and linux is 3.0.8. >> >> Although this might not be very suited to my needs. >> The incoming buffers wont be timestamped if you cant keep up >> with reading the timestamping registers as I understand it. >> This might be perfectly fine in the case of PTP where the >> packet rate is fairly low. In my case Im looking at some >> 80kpps that all would need to be timestamped. >> There is a comment about this in the driver: >> >> *//* If this bit is set, then the RX registers contain the >> time stamp. No/**/ other packet will be time stamped until we >> read these registers, so/* */read the registers to make them >> available again. Because only one/**/ packet can be time >> stamped at a time, we know that the register/* */values must >> belong to this one here and therefore we don't need to/**/ >> compare any of the additional attributes stored for it./* */If >> nothing went wrong, then it should have a shared tx_flags that >> we/**/ can turn into a skb_shared_hwtstamps. *//* >> *//* >> *//* >> >> On Tuesday, October 13, 2015, Alexander Duyck >> <ale...@gm... <mailto:ale...@gm...>> >> wrote: >> >> I'm assuming you are using either an older kernel or an >> out-of-tree driver do I have that right? I ask because >> this code >> doesn't resemble what is is currently in the latest Linux >> kernel. >> >> Comments on what you have found are inline below. >> >> - Alex >> >> On 10/13/2015 12:53 AM, Mattias Barthel wrote: >> >> Hi, I think I managed to get around this problem. >> >> >> in function init_hw_timer(): >> ---- >> case e1000_i350: >> case e1000_82580: >> >> /* >> * The 82580 timesync updates the system timer every >> 8ns by 8ns >> * and the value cannot be shifted. Instead we need to >> shift >> * the registers to generate a 64bit timer value. As a >> result >> * SYSTIMR/L/H, TXSTMPL/H, RXSTMPL/H all have to be >> shifted by >> * 24 in order to generate a larger value for >> synchronization. >> */ >> adapter->cycles.shift = IGB_82580_TSYNC_SHIFT; >> >> ---- >> >> I have the I350 and I dont think this cycles shift was >> meant >> for my adapter. >> IGB_82580_TSYNC_SHIFT is 24. >> >> Also when doing igb_read_clock, >> this shifting is applied only for the 82580 and I guess it >> should not have been set in the first place for the I350: >> -------- >> /* >> * The timestamp latches on lowest register read. For >> the 82580 >> * the lowest register is SYSTIMR instead of SYSTIML. >> However >> we never >> * adjusted TIMINCA so SYSTIMR will just read as all 0s so >> ignore it. >> */ >> if (hw->mac.type == e1000_82580) { >> stamp = rd32(E1000_SYSTIMR) >> 8; >> shift = IGB_82580_TSYNC_SHIFT; >> } >> >> stamp |= (u64)rd32(E1000_SYSTIML) << shift; >> stamp |= (u64)rd32(E1000_SYSTIMH) << (shift + 32); >> ------------- >> >> >> This is a bug in the code. The i350 should be shifted by >> the same >> value as the 82580. Then the result would be the same as >> if you >> had set the shift to 0. As I recall the reasoning behind the >> change was to make use of as many bits of the time as >> possible, >> and on 82580 and newer the SYSTIMH portion of the clock only >> contained 8 bits of actual clock data. >> >> Without that change I believe there is a risk of the >> counter being >> considered to have cycled too fast and causing errors as >> the cycle >> counter will roll over at 40 bits instead of 64. The >> alternative >> would be to update the cycle counter you are using so that >> it is >> aware that it is limited to 40 bits and then use an shift >> of 0. >> >> Setting adapter->cycles.shift to 0 makes the clock run at >> about the same speed as the system clock. >> >> Next problem I have is that the syststamp is not syncing >> correctly against the system clock. >> >> Any help is much appreciated. >> >> Regards, >> >> Mattias >> >> >> >> >> >> >> >> >> >> >> >> >> On Mon, Oct 12, 2015 at 5:04 PM, Mattias Barthel >> <mat...@gm... >> <mailto:mat...@gm...>> >> wrote: >> >> Hi again, >> >> Started to try and timestamp all packets in the I350. >> >> Driver version: >> >> driver: igb >> version: 3.0.6-k2 >> firmware-version: 0.147-0 >> >> Doing an ioctl that turns on timestamping for all >> packets. >> >> hwconfig.tx_type = HWTSTAMP_TX_OFF; >> hwconfig.rx_filter = HWTSTAMP_FILTER_ALL; >> hwconfig_requested = hwconfig; >> ioctl(sock, SIOCSHWTSTAMP, &hwtstamp) >> >> The strangest thing is that although it works to >> read the >> timestamps the clock seems to advance very slowly >> and/or >> incorrectly. >> >> I just printout the HW timestamps alongside the >> systemclock and >> the hw stamping is advancing the nanoseconds when the >> systemclock >> is advancing us and seconds. >> >> Is there a know bug in this version of the driver >> that maybe >> initializes the clock incorrectly or maybe reads >> incorrectly from >> the clock? >> >> >> [ 63.004116] igb_rx_hwtstamp >> [ 63.004684] hwtstamp: 1444656871372825302, syststamp: >> 1444656871372845807 >> [ 63.005721] system timestamp: tv_sec: 1444656793, >> tv_usec: 683825 >> [ 63.006631] sys hw timestamp: tstamp: >> 1444656871372845807, >> tv_sec: 1444656871, tv_usec: 372845 >> [ 63.008044] hw timestamp: tstamp: >> 1444656871372825302, >> tv_sec: >> 1444656871, tv_usec: 372825 >> [ 63.009395] >> [ 63.104111] igb_rx_hwtstamp >> [ 63.104704] hwtstamp: 1444656871372825308, syststamp: >> 1444656871372845813 >> [ 63.105747] system timestamp: tv_sec: 1444656793, >> tv_usec: 783851 >> [ 63.106669] sys hw timestamp: tstamp: >> 1444656871372845813, >> tv_sec: 1444656871, tv_usec: 372845 >> [ 63.108075] hw timestamp: tstamp: >> 1444656871372825308, >> tv_sec: >> 1444656871, tv_usec: 372825 >> [ 63.109428] >> [ 63.204109] igb_rx_hwtstamp >> [ 63.204683] hwtstamp: 1444656871372825314, syststamp: >> 1444656871372845819 >> [ 63.206018] system timestamp: tv_sec: 1444656793, >> tv_usec: 884122 >> [ 63.207242] sys hw timestamp: tstamp: >> 1444656871372845819, >> tv_sec: 1444656871, tv_usec: 372845 >> [ 63.209096] hw timestamp: tstamp: >> 1444656871372825314, >> tv_sec: >> 1444656871, tv_usec: 372825 >> [ 63.210897] >> [ 63.304068] igb_rx_hwtstamp >> [ 63.304849] hwtstamp: 1444656871372825320, syststamp: >> 1444656871372845825 >> [ 63.306176] system timestamp: tv_sec: 1444656793, >> tv_usec: 984280 >> [ 63.307391] sys hw timestamp: tstamp: >> 1444656871372845825, >> tv_sec: 1444656871, tv_usec: 372845 >> [ 63.309252] hw timestamp: tstamp: >> 1444656871372825320, >> tv_sec: >> 1444656871, tv_usec: 372825 >> [ 63.311032] >> [ 63.404107] igb_rx_hwtstamp >> [ 63.404659] hwtstamp: 1444656871372825326, syststamp: >> 1444656871372845831 >> [ 63.405695] system timestamp: tv_sec: 1444656794, >> tv_usec: 83798 >> [ 63.406635] sys hw timestamp: tstamp: >> 1444656871372845831, >> tv_sec: 1444656871, tv_usec: 372845 >> [ 63.408044] hw timestamp: tstamp: >> 1444656871372825326, >> tv_sec: >> 1444656871, tv_usec: 372825 >> [ 63.409405] >> [ 63.504084] igb_rx_hwtstamp >> [ 63.504670] hwtstamp: 1444656871372825332, syststamp: >> 1444656871372845837 >> [ 63.505715] system timestamp: tv_sec: 1444656794, >> tv_usec: 183819 >> [ 63.506657] sys hw timestamp: tstamp: >> 1444656871372845837, >> tv_sec: 1444656871, tv_usec: 372845 >> [ 63.508090] hw timestamp: tstamp: >> 1444656871372825332, >> tv_sec: >> 1444656871, tv_usec: 372825 >> >> >> Any help will be very much appreciated. >> >> Best regards, >> >> Mattias >> >> >> >> >> >> >> >> >> On Thu, Oct 8, 2015 at 6:01 PM, Alexander Duyck >> <ale...@gm... >> <mailto:ale...@gm...>> wrote: >> >> The i350 should support timestamping all >> packets as I >> recall. You should probably look into getting the >> linuxptp package and >> seeing if you can make use of the hwstamp_ctl >> command to >> enable timestamping for all incoming packets >> on the >> adapter. >> >> - Alex >> >> On 10/08/2015 12:33 AM, Mattias Barthel wrote: >> >> Hi >> thanks for answer. >> >> Unfortunately, the udp/ip stream just as >> the cfm >> stream is >> against only one peer. >> Furthemore on the guest side I have only >> one vcpu >> tied to >> two physical siblings to minimize L2 cache >> misses >> on ISRs >> due to cache coherence. >> >> I had an idea this morning, maybe I can >> make use >> of the >> ptp hw timestaming for this? This should give >> better accuracy. >> Is this possible or does it only work for >> ethertype ptp? >> >> On Thursday, October 8, 2015, Alexander Duyck >> <ale...@gm... >> <mailto:ale...@gm...> >> <mailto:ale...@gm... >> <mailto:ale...@gm...>>> wrote: >> >> The difference could be Receive Side >> Scaling >> (RSS). The UDP/IP >> packets >> can be load balanced across all queues >> via RSS >> based >> on a hash of the >> source and destination addresses. So >> if you have >> multiple sources >> generating these packets, or they are >> being >> received >> on multiple IP >> addresses then it is possible the load is >> being spread >> out over >> multiple >> CPUs. Your CFM packets are on an Ethertype >> other than >> IPv4 or IPv6 so >> all packets will end up on queue 0 if >> I am not >> mistaken. >> >> - Alex >> >> On 10/07/2015 10:21 PM, Mattias >> Barthel wrote: >> > Hi again! >> > >> > I realise my message might not have >> been so >> clear. >> > >> > I will try to clarify. I am doing PM >> with >> software >> timestamping. >> > >> > igb versions: >> > version: 3.0.6-k2 >> > firmware-version: 0.147-0 >> > >> > I do timestamping (NTP format) of the >> packets in the >> same place >> for all >> > protocols. (in igb_clean_rx_irq_adv()). >> > >> > For these non ip packets like IEEE >> 801.2ag CFM ETH-DM >> (ethertype 0x8902) I >> > get quite a lot worse accuracy in the >> timestamping. >> > >> > For UDP/IP TWAMP I get for 10k pps a max >> delay of >> around 140us >> back to back. >> > For CFM ETH DM I get for the same packet >> rate and >> packet size a >> max delay >> > of around 900us. >> > >> > So my hypothesis was: >> > Could FW be putting these L2 packets >> in another >> queue with different >> > characteristics? >> > >> > I tried to add a ET-type filter as >> written in my >> previous mail >> but it >> > showed no difference. >> > >> > >> > Thanks for any help given! >> > >> > >> > >> > On Mon, Oct 5, 2015 at 2:19 PM, >> Mattias Barthel >> <mat...@gm... >> <mailto:mat...@gm...> >> <javascript:;>> >> >> > wrote: >> > >> >> Greetings, >> >> >> >> Im getting quite bad latency when >> using the igb >> driver on i350 >> on linux >> >> regarding >> >> ETH CFM packets. This compared to TWAMP >> (IPv4 UDP >> packets). >> >> The environment is KVM with the >> i350 devices in >> PCI-passthrough. >> >> >> >> So i figured I add an own rx-queue >> (filter) for >> those types of >> ethernet >> >> protocol packets. >> >> >> >> This is what i added: >> >> >> >> >> >> wr32(E1000_ETQF(4), >> >> (1 << 31) | /* queue enable */ >> >> (E1000_ETQF_FILTER_ENABLE) | /* enable >> filter */ >> >> (1 << 29) | /* enable immediate >> interrupt */ >> >> (0x4 << 16) | /* queue no. 4 */ >> >> (ETH_P_8021AG)); /* 0x8902 >> CFM eth >> protocol type */ >> >> >> >> >> >> ETH_P_8021A is 0x8902 >> >> >> >> >> >> >> >> Is this a correct/good approach? >> >> >> >> Thanks in advance! >> >> >> >> -- >> >> Mattias Barthel >> >> >> > >> >> >> >> ------------------------------------------------------------------------------ >> _______________________________________________ >> E1000-devel mailing list >> E10...@li... >> <mailto:E10...@li...> >> <javascript:;> >> https://lists.sourceforge.net/lists/listinfo/e1000-devel >> To learn more about Intel® >> Ethernet, visit >> http://communities.intel.com/community/wired >> >> >> Avis de confidentialité >> >> Les informations contenues dans le présent >> message >> et dans >> toute pièce qui lui est jointe sont >> confidentielles et >> peuvent être protégées par le secret >> professionnel. Ces >> informations sont à l’usage exclusif de >> son ou de ses >> destinataires. Si vous recevez ce message >> par erreur, >> veuillez s’il vous plait communiquer >> immédiatement >> avec >> l’expéditeur et en détruire tout >> exemplaire. De >> plus, il >> vous est strictement interdit de le >> divulguer, de le >> distribuer ou de le reproduire sans >> l’autorisation de >> l’expéditeur. Merci. >> >> Confidentiality notice >> >> This e-mail message and any attachment >> hereto contain >> confidential information which may be >> privileged >> and which >> is intended for the exclusive use of its >> addressee(s). If >> you receive this message in error, please >> inform >> sender >> immediately and destroy any copy thereof. >> Furthermore, any >> disclosure, distribution or copying of this >> message and/or >> any attachment hereto without the consent >> of the >> sender is >> strictly prohibited. Thank you. >> >> >> >> >> >> -- Mattias Barthel >> >> >> >> >> -- Mattias Barthel >> >> >> >> Avis de confidentialité >> >> Les informations contenues dans le présent message et dans >> toute pièce qui lui est jointe sont confidentielles et peuvent >> être protégées par le secret professionnel. Ces informations >> sont à l’usage exclusif de son ou de ses destinataires. Si >> vous recevez ce message par erreur, veuillez s’il vous plait >> communiquer immédiatement avec l’expéditeur et en détruire >> tout exemplaire. De plus, il vous est strictement interdit de >> le divulguer, de le distribuer ou de le reproduire sans >> l’autorisation de l’expéditeur. Merci. >> >> Confidentiality notice >> >> This e-mail message and any attachment hereto contain >> confidential information which may be privileged and which is >> intended for the exclusive use of its addressee(s). If you >> receive this message in error, please inform sender >> immediately and destroy any copy thereof. Furthermore, any >> disclosure, distribution or copying of this message and/or any >> attachment hereto without the consent of the sender is >> strictly prohibited. Thank you. >> >> >> >> Avis de confidentialité >> >> Les informations contenues dans le présent message et dans toute pièce >> qui lui est jointe sont confidentielles et peuvent être protégées par le >> secret professionnel. Ces informations sont à l’usage exclusif de son ou de >> ses destinataires. Si vous recevez ce message par erreur, veuillez s’il >> vous plait communiquer immédiatement avec l’expéditeur et en détruire tout >> exemplaire. De plus, il vous est strictement interdit de le divulguer, de >> le distribuer ou de le reproduire sans l’autorisation de l’expéditeur. >> Merci. >> >> Confidentiality notice >> >> This e-mail message and any attachment hereto contain confidential >> information which may be privileged and which is intended for the exclusive >> use of its addressee(s). If you receive this message in error, please >> inform sender immediately and destroy any copy thereof. Furthermore, any >> disclosure, distribution or copying of this message and/or any attachment >> hereto without the consent of the sender is strictly prohibited. Thank you. >> >> > -- Mattias Barthel |
From: Alexander D. <ale...@gm...> - 2015-10-13 18:45:15
|
You can probably just download and use the latest driver from: http://sourceforge.net/projects/e1000/files/igb%20stable/ It should be able to build on a 3.0.8 kernel provided nothing has been back-ported that would change the APIs. - Alex On 10/13/2015 10:26 AM, Mattias Barthel wrote: > Thanks a lot Alex, there might still be some hope then. > Unfortunately Im bound to using the linux kernel 3.0.8. What > out-of-tree driver can I use with that, please? > > On Tuesday, October 13, 2015, Alexander Duyck > <ale...@gm... <mailto:ale...@gm...>> wrote: > > I don't know if it is available in that driver or not, but there > is an option in later igb drivers to support a time stamp being > written with the packet data. Based on the fact that you are > already seeing issues, you might want to try getting the > out-of-tree driver from e1000.sf.net <http://e1000.sf.net> and > seeing if you can use that with the time stamp in packet option > that is enabled when you select to timestamp all incoming Rx packets. > > - Alex > > On 10/13/2015 10:01 AM, Mattias Barthel wrote: > > Yes, setting the cycles.shift to the same as on the 82580 and > also shifting the reads accordingly everything works. The sync > of shhwtstamps->syststamp too. > > My driver version is 3.0.6-k2 and linux is 3.0.8. > > Although this might not be very suited to my needs. > The incoming buffers wont be timestamped if you cant keep up > with reading the timestamping registers as I understand it. > This might be perfectly fine in the case of PTP where the > packet rate is fairly low. In my case Im looking at some > 80kpps that all would need to be timestamped. > There is a comment about this in the driver: > > *//* If this bit is set, then the RX registers contain the > time stamp. No/**/ other packet will be time stamped until we > read these registers, so/* */read the registers to make them > available again. Because only one/**/ packet can be time > stamped at a time, we know that the register/* */values must > belong to this one here and therefore we don't need to/**/ > compare any of the additional attributes stored for it./* */If > nothing went wrong, then it should have a shared tx_flags that > we/**/ can turn into a skb_shared_hwtstamps. *//* > *//* > *//* > > On Tuesday, October 13, 2015, Alexander Duyck > <ale...@gm... <mailto:ale...@gm...>> > wrote: > > I'm assuming you are using either an older kernel or an > out-of-tree driver do I have that right? I ask because > this code > doesn't resemble what is is currently in the latest Linux > kernel. > > Comments on what you have found are inline below. > > - Alex > > On 10/13/2015 12:53 AM, Mattias Barthel wrote: > > Hi, I think I managed to get around this problem. > > > in function init_hw_timer(): > ---- > case e1000_i350: > case e1000_82580: > > /* > * The 82580 timesync updates the system timer every > 8ns by 8ns > * and the value cannot be shifted. Instead we need to > shift > * the registers to generate a 64bit timer value. As a > result > * SYSTIMR/L/H, TXSTMPL/H, RXSTMPL/H all have to be > shifted by > * 24 in order to generate a larger value for > synchronization. > */ > adapter->cycles.shift = IGB_82580_TSYNC_SHIFT; > > ---- > > I have the I350 and I dont think this cycles shift was > meant > for my adapter. > IGB_82580_TSYNC_SHIFT is 24. > > Also when doing igb_read_clock, > this shifting is applied only for the 82580 and I guess it > should not have been set in the first place for the I350: > -------- > /* > * The timestamp latches on lowest register read. For > the 82580 > * the lowest register is SYSTIMR instead of SYSTIML. > However > we never > * adjusted TIMINCA so SYSTIMR will just read as all 0s so > ignore it. > */ > if (hw->mac.type == e1000_82580) { > stamp = rd32(E1000_SYSTIMR) >> 8; > shift = IGB_82580_TSYNC_SHIFT; > } > > stamp |= (u64)rd32(E1000_SYSTIML) << shift; > stamp |= (u64)rd32(E1000_SYSTIMH) << (shift + 32); > ------------- > > > This is a bug in the code. The i350 should be shifted by > the same > value as the 82580. Then the result would be the same as > if you > had set the shift to 0. As I recall the reasoning behind the > change was to make use of as many bits of the time as > possible, > and on 82580 and newer the SYSTIMH portion of the clock only > contained 8 bits of actual clock data. > > Without that change I believe there is a risk of the > counter being > considered to have cycled too fast and causing errors as > the cycle > counter will roll over at 40 bits instead of 64. The > alternative > would be to update the cycle counter you are using so that > it is > aware that it is limited to 40 bits and then use an shift > of 0. > > Setting adapter->cycles.shift to 0 makes the clock run at > about the same speed as the system clock. > > Next problem I have is that the syststamp is not syncing > correctly against the system clock. > > Any help is much appreciated. > > Regards, > > Mattias > > > > > > > > > > > > > On Mon, Oct 12, 2015 at 5:04 PM, Mattias Barthel > <mat...@gm... > <mailto:mat...@gm...>> > wrote: > > Hi again, > > Started to try and timestamp all packets in the I350. > > Driver version: > > driver: igb > version: 3.0.6-k2 > firmware-version: 0.147-0 > > Doing an ioctl that turns on timestamping for all > packets. > > hwconfig.tx_type = HWTSTAMP_TX_OFF; > hwconfig.rx_filter = HWTSTAMP_FILTER_ALL; > hwconfig_requested = hwconfig; > ioctl(sock, SIOCSHWTSTAMP, &hwtstamp) > > The strangest thing is that although it works to > read the > timestamps the clock seems to advance very slowly > and/or > incorrectly. > > I just printout the HW timestamps alongside the > systemclock and > the hw stamping is advancing the nanoseconds when the > systemclock > is advancing us and seconds. > > Is there a know bug in this version of the driver > that maybe > initializes the clock incorrectly or maybe reads > incorrectly from > the clock? > > > [ 63.004116] igb_rx_hwtstamp > [ 63.004684] hwtstamp: 1444656871372825302, syststamp: > 1444656871372845807 > [ 63.005721] system timestamp: tv_sec: 1444656793, > tv_usec: 683825 > [ 63.006631] sys hw timestamp: tstamp: > 1444656871372845807, > tv_sec: 1444656871, tv_usec: 372845 > [ 63.008044] hw timestamp: tstamp: > 1444656871372825302, > tv_sec: > 1444656871, tv_usec: 372825 > [ 63.009395] > [ 63.104111] igb_rx_hwtstamp > [ 63.104704] hwtstamp: 1444656871372825308, syststamp: > 1444656871372845813 > [ 63.105747] system timestamp: tv_sec: 1444656793, > tv_usec: 783851 > [ 63.106669] sys hw timestamp: tstamp: > 1444656871372845813, > tv_sec: 1444656871, tv_usec: 372845 > [ 63.108075] hw timestamp: tstamp: > 1444656871372825308, > tv_sec: > 1444656871, tv_usec: 372825 > [ 63.109428] > [ 63.204109] igb_rx_hwtstamp > [ 63.204683] hwtstamp: 1444656871372825314, syststamp: > 1444656871372845819 > [ 63.206018] system timestamp: tv_sec: 1444656793, > tv_usec: 884122 > [ 63.207242] sys hw timestamp: tstamp: > 1444656871372845819, > tv_sec: 1444656871, tv_usec: 372845 > [ 63.209096] hw timestamp: tstamp: > 1444656871372825314, > tv_sec: > 1444656871, tv_usec: 372825 > [ 63.210897] > [ 63.304068] igb_rx_hwtstamp > [ 63.304849] hwtstamp: 1444656871372825320, syststamp: > 1444656871372845825 > [ 63.306176] system timestamp: tv_sec: 1444656793, > tv_usec: 984280 > [ 63.307391] sys hw timestamp: tstamp: > 1444656871372845825, > tv_sec: 1444656871, tv_usec: 372845 > [ 63.309252] hw timestamp: tstamp: > 1444656871372825320, > tv_sec: > 1444656871, tv_usec: 372825 > [ 63.311032] > [ 63.404107] igb_rx_hwtstamp > [ 63.404659] hwtstamp: 1444656871372825326, syststamp: > 1444656871372845831 > [ 63.405695] system timestamp: tv_sec: 1444656794, > tv_usec: 83798 > [ 63.406635] sys hw timestamp: tstamp: > 1444656871372845831, > tv_sec: 1444656871, tv_usec: 372845 > [ 63.408044] hw timestamp: tstamp: > 1444656871372825326, > tv_sec: > 1444656871, tv_usec: 372825 > [ 63.409405] > [ 63.504084] igb_rx_hwtstamp > [ 63.504670] hwtstamp: 1444656871372825332, syststamp: > 1444656871372845837 > [ 63.505715] system timestamp: tv_sec: 1444656794, > tv_usec: 183819 > [ 63.506657] sys hw timestamp: tstamp: > 1444656871372845837, > tv_sec: 1444656871, tv_usec: 372845 > [ 63.508090] hw timestamp: tstamp: > 1444656871372825332, > tv_sec: > 1444656871, tv_usec: 372825 > > > Any help will be very much appreciated. > > Best regards, > > Mattias > > > > > > > > > On Thu, Oct 8, 2015 at 6:01 PM, Alexander Duyck > <ale...@gm... > <mailto:ale...@gm...>> wrote: > > The i350 should support timestamping all > packets as I > recall. You should probably look into getting the > linuxptp package and > seeing if you can make use of the hwstamp_ctl > command to > enable timestamping for all incoming packets > on the > adapter. > > - Alex > > On 10/08/2015 12:33 AM, Mattias Barthel wrote: > > Hi > thanks for answer. > > Unfortunately, the udp/ip stream just as > the cfm > stream is > against only one peer. > Furthemore on the guest side I have only > one vcpu > tied to > two physical siblings to minimize L2 cache > misses > on ISRs > due to cache coherence. > > I had an idea this morning, maybe I can > make use > of the > ptp hw timestaming for this? This should give > better accuracy. > Is this possible or does it only work for > ethertype ptp? > > On Thursday, October 8, 2015, Alexander Duyck > <ale...@gm... > <mailto:ale...@gm...> > <mailto:ale...@gm... > <mailto:ale...@gm...>>> wrote: > > The difference could be Receive Side > Scaling > (RSS). The UDP/IP > packets > can be load balanced across all queues > via RSS > based > on a hash of the > source and destination addresses. So > if you have > multiple sources > generating these packets, or they are > being > received > on multiple IP > addresses then it is possible the load is > being spread > out over > multiple > CPUs. Your CFM packets are on an Ethertype > other than > IPv4 or IPv6 so > all packets will end up on queue 0 if > I am not > mistaken. > > - Alex > > On 10/07/2015 10:21 PM, Mattias > Barthel wrote: > > Hi again! > > > > I realise my message might not have > been so > clear. > > > > I will try to clarify. I am doing PM > with > software > timestamping. > > > > igb versions: > > version: 3.0.6-k2 > > firmware-version: 0.147-0 > > > > I do timestamping (NTP format) of the > packets in the > same place > for all > > protocols. (in igb_clean_rx_irq_adv()). > > > > For these non ip packets like IEEE > 801.2ag CFM ETH-DM > (ethertype 0x8902) I > > get quite a lot worse accuracy in the > timestamping. > > > > For UDP/IP TWAMP I get for 10k pps a max > delay of > around 140us > back to back. > > For CFM ETH DM I get for the same packet > rate and > packet size a > max delay > > of around 900us. > > > > So my hypothesis was: > > Could FW be putting these L2 packets > in another > queue with different > > characteristics? > > > > I tried to add a ET-type filter as > written in my > previous mail > but it > > showed no difference. > > > > > > Thanks for any help given! > > > > > > > > On Mon, Oct 5, 2015 at 2:19 PM, > Mattias Barthel > <mat...@gm... > <mailto:mat...@gm...> > <javascript:;>> > > > wrote: > > > >> Greetings, > >> > >> Im getting quite bad latency when > using the igb > driver on i350 > on linux > >> regarding > >> ETH CFM packets. This compared to TWAMP > (IPv4 UDP > packets). > >> The environment is KVM with the > i350 devices in > PCI-passthrough. > >> > >> So i figured I add an own rx-queue > (filter) for > those types of > ethernet > >> protocol packets. > >> > >> This is what i added: > >> > >> > >> wr32(E1000_ETQF(4), > >> (1 << 31) | /* queue enable */ > >> (E1000_ETQF_FILTER_ENABLE) | /* enable > filter */ > >> (1 << 29) | /* enable immediate > interrupt */ > >> (0x4 << 16) | /* queue no. 4 */ > >> (ETH_P_8021AG)); /* 0x8902 > CFM eth > protocol type */ > >> > >> > >> ETH_P_8021A is 0x8902 > >> > >> > >> > >> Is this a correct/good approach? > >> > >> Thanks in advance! > >> > >> -- > >> Mattias Barthel > >> > > > > > ------------------------------------------------------------------------------ > _______________________________________________ > E1000-devel mailing list > E10...@li... > <mailto:E10...@li...> > <javascript:;> > https://lists.sourceforge.net/lists/listinfo/e1000-devel > To learn more about Intel® > Ethernet, visit > http://communities.intel.com/community/wired > > > Avis de confidentialité > > Les informations contenues dans le présent > message > et dans > toute pièce qui lui est jointe sont > confidentielles et > peuvent être protégées par le secret > professionnel. Ces > informations sont à l’usage exclusif de > son ou de ses > destinataires. Si vous recevez ce message > par erreur, > veuillez s’il vous plait communiquer > immédiatement > avec > l’expéditeur et en détruire tout > exemplaire. De > plus, il > vous est strictement interdit de le > divulguer, de le > distribuer ou de le reproduire sans > l’autorisation de > l’expéditeur. Merci. > > Confidentiality notice > > This e-mail message and any attachment > hereto contain > confidential information which may be > privileged > and which > is intended for the exclusive use of its > addressee(s). If > you receive this message in error, please > inform > sender > immediately and destroy any copy thereof. > Furthermore, any > disclosure, distribution or copying of this > message and/or > any attachment hereto without the consent > of the > sender is > strictly prohibited. Thank you. > > > > > > -- Mattias Barthel > > > > > -- Mattias Barthel > > > > Avis de confidentialité > > Les informations contenues dans le présent message et dans > toute pièce qui lui est jointe sont confidentielles et peuvent > être protégées par le secret professionnel. Ces informations > sont à l’usage exclusif de son ou de ses destinataires. Si > vous recevez ce message par erreur, veuillez s’il vous plait > communiquer immédiatement avec l’expéditeur et en détruire > tout exemplaire. De plus, il vous est strictement interdit de > le divulguer, de le distribuer ou de le reproduire sans > l’autorisation de l’expéditeur. Merci. > > Confidentiality notice > > This e-mail message and any attachment hereto contain > confidential information which may be privileged and which is > intended for the exclusive use of its addressee(s). If you > receive this message in error, please inform sender > immediately and destroy any copy thereof. Furthermore, any > disclosure, distribution or copying of this message and/or any > attachment hereto without the consent of the sender is > strictly prohibited. Thank you. > > > > Avis de confidentialité > > Les informations contenues dans le présent message et dans toute pièce > qui lui est jointe sont confidentielles et peuvent être protégées par > le secret professionnel. Ces informations sont à l’usage exclusif de > son ou de ses destinataires. Si vous recevez ce message par erreur, > veuillez s’il vous plait communiquer immédiatement avec l’expéditeur > et en détruire tout exemplaire. De plus, il vous est strictement > interdit de le divulguer, de le distribuer ou de le reproduire sans > l’autorisation de l’expéditeur. Merci. > > Confidentiality notice > > This e-mail message and any attachment hereto contain confidential > information which may be privileged and which is intended for the > exclusive use of its addressee(s). If you receive this message in > error, please inform sender immediately and destroy any copy thereof. > Furthermore, any disclosure, distribution or copying of this message > and/or any attachment hereto without the consent of the sender is > strictly prohibited. Thank you. > |
From: Mattias B. <mba...@ac...> - 2015-10-13 17:27:01
|
Thanks a lot Alex, there might still be some hope then. Unfortunately Im bound to using the linux kernel 3.0.8. What out-of-tree driver can I use with that, please? On Tuesday, October 13, 2015, Alexander Duyck <ale...@gm...> wrote: > I don't know if it is available in that driver or not, but there is an > option in later igb drivers to support a time stamp being written with the > packet data. Based on the fact that you are already seeing issues, you > might want to try getting the out-of-tree driver from e1000.sf.net and > seeing if you can use that with the time stamp in packet option that is > enabled when you select to timestamp all incoming Rx packets. > > - Alex > > On 10/13/2015 10:01 AM, Mattias Barthel wrote: > >> Yes, setting the cycles.shift to the same as on the 82580 and also >> shifting the reads accordingly everything works. The sync of >> shhwtstamps->syststamp too. >> >> My driver version is 3.0.6-k2 and linux is 3.0.8. >> >> Although this might not be very suited to my needs. >> The incoming buffers wont be timestamped if you cant keep up with reading >> the timestamping registers as I understand it. >> This might be perfectly fine in the case of PTP where the packet rate is >> fairly low. In my case Im looking at some 80kpps that all would need to be >> timestamped. >> There is a comment about this in the driver: >> >> *//* If this bit is set, then the RX registers contain the time stamp. >> No/**/ other packet will be time stamped until we read these registers, >> so/* */read the registers to make them available again. Because only >> one/**/ packet can be time stamped at a time, we know that the register/* >> */values must belong to this one here and therefore we don't need to/**/ >> compare any of the additional attributes stored for it./* */If nothing went >> wrong, then it should have a shared tx_flags that we/**/ can turn into a >> skb_shared_hwtstamps. *//* >> *//* >> *//* >> >> On Tuesday, October 13, 2015, Alexander Duyck <ale...@gm... >> <mailto:ale...@gm...>> wrote: >> >> I'm assuming you are using either an older kernel or an >> out-of-tree driver do I have that right? I ask because this code >> doesn't resemble what is is currently in the latest Linux kernel. >> >> Comments on what you have found are inline below. >> >> - Alex >> >> On 10/13/2015 12:53 AM, Mattias Barthel wrote: >> >> Hi, I think I managed to get around this problem. >> >> >> in function init_hw_timer(): >> ---- >> case e1000_i350: >> case e1000_82580: >> >> /* >> * The 82580 timesync updates the system timer every 8ns by 8ns >> * and the value cannot be shifted. Instead we need to shift >> * the registers to generate a 64bit timer value. As a result >> * SYSTIMR/L/H, TXSTMPL/H, RXSTMPL/H all have to be shifted by >> * 24 in order to generate a larger value for synchronization. >> */ >> adapter->cycles.shift = IGB_82580_TSYNC_SHIFT; >> >> ---- >> >> I have the I350 and I dont think this cycles shift was meant >> for my adapter. >> IGB_82580_TSYNC_SHIFT is 24. >> >> Also when doing igb_read_clock, >> this shifting is applied only for the 82580 and I guess it >> should not have been set in the first place for the I350: >> -------- >> /* >> * The timestamp latches on lowest register read. For the 82580 >> * the lowest register is SYSTIMR instead of SYSTIML. However >> we never >> * adjusted TIMINCA so SYSTIMR will just read as all 0s so >> ignore it. >> */ >> if (hw->mac.type == e1000_82580) { >> stamp = rd32(E1000_SYSTIMR) >> 8; >> shift = IGB_82580_TSYNC_SHIFT; >> } >> >> stamp |= (u64)rd32(E1000_SYSTIML) << shift; >> stamp |= (u64)rd32(E1000_SYSTIMH) << (shift + 32); >> ------------- >> >> >> This is a bug in the code. The i350 should be shifted by the same >> value as the 82580. Then the result would be the same as if you >> had set the shift to 0. As I recall the reasoning behind the >> change was to make use of as many bits of the time as possible, >> and on 82580 and newer the SYSTIMH portion of the clock only >> contained 8 bits of actual clock data. >> >> Without that change I believe there is a risk of the counter being >> considered to have cycled too fast and causing errors as the cycle >> counter will roll over at 40 bits instead of 64. The alternative >> would be to update the cycle counter you are using so that it is >> aware that it is limited to 40 bits and then use an shift of 0. >> >> Setting adapter->cycles.shift to 0 makes the clock run at >> about the same speed as the system clock. >> >> Next problem I have is that the syststamp is not syncing >> correctly against the system clock. >> >> Any help is much appreciated. >> >> Regards, >> >> Mattias >> >> >> >> >> >> >> >> >> >> >> >> >> On Mon, Oct 12, 2015 at 5:04 PM, Mattias Barthel >> <mat...@gm... <mailto:mat...@gm...>> >> wrote: >> >> Hi again, >> >> Started to try and timestamp all packets in the I350. >> >> Driver version: >> >> driver: igb >> version: 3.0.6-k2 >> firmware-version: 0.147-0 >> >> Doing an ioctl that turns on timestamping for all packets. >> >> hwconfig.tx_type = HWTSTAMP_TX_OFF; >> hwconfig.rx_filter = HWTSTAMP_FILTER_ALL; >> hwconfig_requested = hwconfig; >> ioctl(sock, SIOCSHWTSTAMP, &hwtstamp) >> >> The strangest thing is that although it works to read the >> timestamps the clock seems to advance very slowly and/or >> incorrectly. >> >> I just printout the HW timestamps alongside the >> systemclock and >> the hw stamping is advancing the nanoseconds when the >> systemclock >> is advancing us and seconds. >> >> Is there a know bug in this version of the driver that maybe >> initializes the clock incorrectly or maybe reads >> incorrectly from >> the clock? >> >> >> [ 63.004116] igb_rx_hwtstamp >> [ 63.004684] hwtstamp: 1444656871372825302, syststamp: >> 1444656871372845807 >> [ 63.005721] system timestamp: tv_sec: 1444656793, >> tv_usec: 683825 >> [ 63.006631] sys hw timestamp: tstamp: 1444656871372845807, >> tv_sec: 1444656871, tv_usec: 372845 >> [ 63.008044] hw timestamp: tstamp: 1444656871372825302, >> tv_sec: >> 1444656871, tv_usec: 372825 >> [ 63.009395] >> [ 63.104111] igb_rx_hwtstamp >> [ 63.104704] hwtstamp: 1444656871372825308, syststamp: >> 1444656871372845813 >> [ 63.105747] system timestamp: tv_sec: 1444656793, >> tv_usec: 783851 >> [ 63.106669] sys hw timestamp: tstamp: 1444656871372845813, >> tv_sec: 1444656871, tv_usec: 372845 >> [ 63.108075] hw timestamp: tstamp: 1444656871372825308, >> tv_sec: >> 1444656871, tv_usec: 372825 >> [ 63.109428] >> [ 63.204109] igb_rx_hwtstamp >> [ 63.204683] hwtstamp: 1444656871372825314, syststamp: >> 1444656871372845819 >> [ 63.206018] system timestamp: tv_sec: 1444656793, >> tv_usec: 884122 >> [ 63.207242] sys hw timestamp: tstamp: 1444656871372845819, >> tv_sec: 1444656871, tv_usec: 372845 >> [ 63.209096] hw timestamp: tstamp: 1444656871372825314, >> tv_sec: >> 1444656871, tv_usec: 372825 >> [ 63.210897] >> [ 63.304068] igb_rx_hwtstamp >> [ 63.304849] hwtstamp: 1444656871372825320, syststamp: >> 1444656871372845825 >> [ 63.306176] system timestamp: tv_sec: 1444656793, >> tv_usec: 984280 >> [ 63.307391] sys hw timestamp: tstamp: 1444656871372845825, >> tv_sec: 1444656871, tv_usec: 372845 >> [ 63.309252] hw timestamp: tstamp: 1444656871372825320, >> tv_sec: >> 1444656871, tv_usec: 372825 >> [ 63.311032] >> [ 63.404107] igb_rx_hwtstamp >> [ 63.404659] hwtstamp: 1444656871372825326, syststamp: >> 1444656871372845831 >> [ 63.405695] system timestamp: tv_sec: 1444656794, >> tv_usec: 83798 >> [ 63.406635] sys hw timestamp: tstamp: 1444656871372845831, >> tv_sec: 1444656871, tv_usec: 372845 >> [ 63.408044] hw timestamp: tstamp: 1444656871372825326, >> tv_sec: >> 1444656871, tv_usec: 372825 >> [ 63.409405] >> [ 63.504084] igb_rx_hwtstamp >> [ 63.504670] hwtstamp: 1444656871372825332, syststamp: >> 1444656871372845837 >> [ 63.505715] system timestamp: tv_sec: 1444656794, >> tv_usec: 183819 >> [ 63.506657] sys hw timestamp: tstamp: 1444656871372845837, >> tv_sec: 1444656871, tv_usec: 372845 >> [ 63.508090] hw timestamp: tstamp: 1444656871372825332, >> tv_sec: >> 1444656871, tv_usec: 372825 >> >> >> Any help will be very much appreciated. >> >> Best regards, >> >> Mattias >> >> >> >> >> >> >> >> >> On Thu, Oct 8, 2015 at 6:01 PM, Alexander Duyck >> <ale...@gm... >> <mailto:ale...@gm...>> wrote: >> >> The i350 should support timestamping all packets as I >> recall. You should probably look into getting the >> linuxptp package and >> seeing if you can make use of the hwstamp_ctl command to >> enable timestamping for all incoming packets on the >> adapter. >> >> - Alex >> >> On 10/08/2015 12:33 AM, Mattias Barthel wrote: >> >> Hi >> thanks for answer. >> >> Unfortunately, the udp/ip stream just as the cfm >> stream is >> against only one peer. >> Furthemore on the guest side I have only one vcpu >> tied to >> two physical siblings to minimize L2 cache misses >> on ISRs >> due to cache coherence. >> >> I had an idea this morning, maybe I can make use >> of the >> ptp hw timestaming for this? This should give >> better accuracy. >> Is this possible or does it only work for >> ethertype ptp? >> >> On Thursday, October 8, 2015, Alexander Duyck >> <ale...@gm... >> <mailto:ale...@gm...> >> <mailto:ale...@gm... >> <mailto:ale...@gm...>>> wrote: >> >> The difference could be Receive Side Scaling >> (RSS). The UDP/IP >> packets >> can be load balanced across all queues via RSS >> based >> on a hash of the >> source and destination addresses. So if you have >> multiple sources >> generating these packets, or they are being >> received >> on multiple IP >> addresses then it is possible the load is >> being spread >> out over >> multiple >> CPUs. Your CFM packets are on an Ethertype >> other than >> IPv4 or IPv6 so >> all packets will end up on queue 0 if I am not >> mistaken. >> >> - Alex >> >> On 10/07/2015 10:21 PM, Mattias Barthel wrote: >> > Hi again! >> > >> > I realise my message might not have been so >> clear. >> > >> > I will try to clarify. I am doing PM with >> software >> timestamping. >> > >> > igb versions: >> > version: 3.0.6-k2 >> > firmware-version: 0.147-0 >> > >> > I do timestamping (NTP format) of the >> packets in the >> same place >> for all >> > protocols. (in igb_clean_rx_irq_adv()). >> > >> > For these non ip packets like IEEE 801.2ag >> CFM ETH-DM >> (ethertype 0x8902) I >> > get quite a lot worse accuracy in the >> timestamping. >> > >> > For UDP/IP TWAMP I get for 10k pps a max >> delay of >> around 140us >> back to back. >> > For CFM ETH DM I get for the same packet >> rate and >> packet size a >> max delay >> > of around 900us. >> > >> > So my hypothesis was: >> > Could FW be putting these L2 packets in another >> queue with different >> > characteristics? >> > >> > I tried to add a ET-type filter as written in my >> previous mail >> but it >> > showed no difference. >> > >> > >> > Thanks for any help given! >> > >> > >> > >> > On Mon, Oct 5, 2015 at 2:19 PM, Mattias Barthel >> <mat...@gm... >> <mailto:mat...@gm...> <javascript:;>> >> >> > wrote: >> > >> >> Greetings, >> >> >> >> Im getting quite bad latency when using the igb >> driver on i350 >> on linux >> >> regarding >> >> ETH CFM packets. This compared to TWAMP >> (IPv4 UDP >> packets). >> >> The environment is KVM with the i350 devices in >> PCI-passthrough. >> >> >> >> So i figured I add an own rx-queue (filter) for >> those types of >> ethernet >> >> protocol packets. >> >> >> >> This is what i added: >> >> >> >> >> >> wr32(E1000_ETQF(4), >> >> (1 << 31) | /* queue enable */ >> >> (E1000_ETQF_FILTER_ENABLE) | /* enable >> filter */ >> >> (1 << 29) | /* enable immediate >> interrupt */ >> >> (0x4 << 16) | /* queue no. 4 */ >> >> (ETH_P_8021AG)); /* 0x8902 CFM eth >> protocol type */ >> >> >> >> >> >> ETH_P_8021A is 0x8902 >> >> >> >> >> >> >> >> Is this a correct/good approach? >> >> >> >> Thanks in advance! >> >> >> >> -- >> >> Mattias Barthel >> >> >> > >> >> >> >> ------------------------------------------------------------------------------ >> _______________________________________________ >> E1000-devel mailing list >> E10...@li... >> <mailto:E10...@li...> >> <javascript:;> >> https://lists.sourceforge.net/lists/listinfo/e1000-devel >> To learn more about Intel® Ethernet, visit >> http://communities.intel.com/community/wired >> >> >> Avis de confidentialité >> >> Les informations contenues dans le présent message >> et dans >> toute pièce qui lui est jointe sont confidentielles et >> peuvent être protégées par le secret >> professionnel. Ces >> informations sont à l’usage exclusif de son ou de ses >> destinataires. Si vous recevez ce message par erreur, >> veuillez s’il vous plait communiquer immédiatement >> avec >> l’expéditeur et en détruire tout exemplaire. De >> plus, il >> vous est strictement interdit de le divulguer, de le >> distribuer ou de le reproduire sans l’autorisation de >> l’expéditeur. Merci. >> >> Confidentiality notice >> >> This e-mail message and any attachment hereto contain >> confidential information which may be privileged >> and which >> is intended for the exclusive use of its >> addressee(s). If >> you receive this message in error, please inform >> sender >> immediately and destroy any copy thereof. >> Furthermore, any >> disclosure, distribution or copying of this >> message and/or >> any attachment hereto without the consent of the >> sender is >> strictly prohibited. Thank you. >> >> >> >> >> >> -- Mattias Barthel >> >> >> >> >> -- Mattias Barthel >> >> >> >> Avis de confidentialité >> >> Les informations contenues dans le présent message et dans toute pièce >> qui lui est jointe sont confidentielles et peuvent être protégées par le >> secret professionnel. Ces informations sont à l’usage exclusif de son ou de >> ses destinataires. Si vous recevez ce message par erreur, veuillez s’il >> vous plait communiquer immédiatement avec l’expéditeur et en détruire tout >> exemplaire. De plus, il vous est strictement interdit de le divulguer, de >> le distribuer ou de le reproduire sans l’autorisation de l’expéditeur. >> Merci. >> >> Confidentiality notice >> >> This e-mail message and any attachment hereto contain confidential >> information which may be privileged and which is intended for the exclusive >> use of its addressee(s). If you receive this message in error, please >> inform sender immediately and destroy any copy thereof. Furthermore, any >> disclosure, distribution or copying of this message and/or any attachment >> hereto without the consent of the sender is strictly prohibited. Thank you. >> >> > -- Avis de confidentialité Les informations contenues dans le présent message et dans toute pièce qui lui est jointe sont confidentielles et peuvent être protégées par le secret professionnel. Ces informations sont à l’usage exclusif de son ou de ses destinataires. Si vous recevez ce message par erreur, veuillez s’il vous plait communiquer immédiatement avec l’expéditeur et en détruire tout exemplaire. De plus, il vous est strictement interdit de le divulguer, de le distribuer ou de le reproduire sans l’autorisation de l’expéditeur. Merci. Confidentiality notice This e-mail message and any attachment hereto contain confidential information which may be privileged and which is intended for the exclusive use of its addressee(s). If you receive this message in error, please inform sender immediately and destroy any copy thereof. Furthermore, any disclosure, distribution or copying of this message and/or any attachment hereto without the consent of the sender is strictly prohibited. Thank you. |
From: Alexander D. <ale...@gm...> - 2015-10-13 17:17:46
|
I don't know if it is available in that driver or not, but there is an option in later igb drivers to support a time stamp being written with the packet data. Based on the fact that you are already seeing issues, you might want to try getting the out-of-tree driver from e1000.sf.net and seeing if you can use that with the time stamp in packet option that is enabled when you select to timestamp all incoming Rx packets. - Alex On 10/13/2015 10:01 AM, Mattias Barthel wrote: > Yes, setting the cycles.shift to the same as on the 82580 and also > shifting the reads accordingly everything works. The sync of > shhwtstamps->syststamp too. > > My driver version is 3.0.6-k2 and linux is 3.0.8. > > Although this might not be very suited to my needs. > The incoming buffers wont be timestamped if you cant keep up with > reading the timestamping registers as I understand it. > This might be perfectly fine in the case of PTP where the packet rate > is fairly low. In my case Im looking at some 80kpps that all would > need to be timestamped. > There is a comment about this in the driver: > > *//* If this bit is set, then the RX registers contain the time stamp. > No/**/ other packet will be time stamped until we read these > registers, so/* */read the registers to make them available again. > Because only one/**/ packet can be time stamped at a time, we know > that the register/* */values must belong to this one here and > therefore we don't need to/**/ compare any of the additional > attributes stored for it./* */If nothing went wrong, then it should > have a shared tx_flags that we/**/ can turn into a > skb_shared_hwtstamps. *//* > *//* > *//* > > On Tuesday, October 13, 2015, Alexander Duyck > <ale...@gm... <mailto:ale...@gm...>> wrote: > > I'm assuming you are using either an older kernel or an > out-of-tree driver do I have that right? I ask because this code > doesn't resemble what is is currently in the latest Linux kernel. > > Comments on what you have found are inline below. > > - Alex > > On 10/13/2015 12:53 AM, Mattias Barthel wrote: > > Hi, I think I managed to get around this problem. > > > in function init_hw_timer(): > ---- > case e1000_i350: > case e1000_82580: > > /* > * The 82580 timesync updates the system timer every 8ns by 8ns > * and the value cannot be shifted. Instead we need to shift > * the registers to generate a 64bit timer value. As a result > * SYSTIMR/L/H, TXSTMPL/H, RXSTMPL/H all have to be shifted by > * 24 in order to generate a larger value for synchronization. > */ > adapter->cycles.shift = IGB_82580_TSYNC_SHIFT; > > ---- > > I have the I350 and I dont think this cycles shift was meant > for my adapter. > IGB_82580_TSYNC_SHIFT is 24. > > Also when doing igb_read_clock, > this shifting is applied only for the 82580 and I guess it > should not have been set in the first place for the I350: > -------- > /* > * The timestamp latches on lowest register read. For the 82580 > * the lowest register is SYSTIMR instead of SYSTIML. However > we never > * adjusted TIMINCA so SYSTIMR will just read as all 0s so > ignore it. > */ > if (hw->mac.type == e1000_82580) { > stamp = rd32(E1000_SYSTIMR) >> 8; > shift = IGB_82580_TSYNC_SHIFT; > } > > stamp |= (u64)rd32(E1000_SYSTIML) << shift; > stamp |= (u64)rd32(E1000_SYSTIMH) << (shift + 32); > ------------- > > > This is a bug in the code. The i350 should be shifted by the same > value as the 82580. Then the result would be the same as if you > had set the shift to 0. As I recall the reasoning behind the > change was to make use of as many bits of the time as possible, > and on 82580 and newer the SYSTIMH portion of the clock only > contained 8 bits of actual clock data. > > Without that change I believe there is a risk of the counter being > considered to have cycled too fast and causing errors as the cycle > counter will roll over at 40 bits instead of 64. The alternative > would be to update the cycle counter you are using so that it is > aware that it is limited to 40 bits and then use an shift of 0. > > Setting adapter->cycles.shift to 0 makes the clock run at > about the same speed as the system clock. > > Next problem I have is that the syststamp is not syncing > correctly against the system clock. > > Any help is much appreciated. > > Regards, > > Mattias > > > > > > > > > > > > > On Mon, Oct 12, 2015 at 5:04 PM, Mattias Barthel > <mat...@gm... <mailto:mat...@gm...>> > wrote: > > Hi again, > > Started to try and timestamp all packets in the I350. > > Driver version: > > driver: igb > version: 3.0.6-k2 > firmware-version: 0.147-0 > > Doing an ioctl that turns on timestamping for all packets. > > hwconfig.tx_type = HWTSTAMP_TX_OFF; > hwconfig.rx_filter = HWTSTAMP_FILTER_ALL; > hwconfig_requested = hwconfig; > ioctl(sock, SIOCSHWTSTAMP, &hwtstamp) > > The strangest thing is that although it works to read the > timestamps the clock seems to advance very slowly and/or > incorrectly. > > I just printout the HW timestamps alongside the > systemclock and > the hw stamping is advancing the nanoseconds when the > systemclock > is advancing us and seconds. > > Is there a know bug in this version of the driver that maybe > initializes the clock incorrectly or maybe reads > incorrectly from > the clock? > > > [ 63.004116] igb_rx_hwtstamp > [ 63.004684] hwtstamp: 1444656871372825302, syststamp: > 1444656871372845807 > [ 63.005721] system timestamp: tv_sec: 1444656793, > tv_usec: 683825 > [ 63.006631] sys hw timestamp: tstamp: 1444656871372845807, > tv_sec: 1444656871, tv_usec: 372845 > [ 63.008044] hw timestamp: tstamp: 1444656871372825302, > tv_sec: > 1444656871, tv_usec: 372825 > [ 63.009395] > [ 63.104111] igb_rx_hwtstamp > [ 63.104704] hwtstamp: 1444656871372825308, syststamp: > 1444656871372845813 > [ 63.105747] system timestamp: tv_sec: 1444656793, > tv_usec: 783851 > [ 63.106669] sys hw timestamp: tstamp: 1444656871372845813, > tv_sec: 1444656871, tv_usec: 372845 > [ 63.108075] hw timestamp: tstamp: 1444656871372825308, > tv_sec: > 1444656871, tv_usec: 372825 > [ 63.109428] > [ 63.204109] igb_rx_hwtstamp > [ 63.204683] hwtstamp: 1444656871372825314, syststamp: > 1444656871372845819 > [ 63.206018] system timestamp: tv_sec: 1444656793, > tv_usec: 884122 > [ 63.207242] sys hw timestamp: tstamp: 1444656871372845819, > tv_sec: 1444656871, tv_usec: 372845 > [ 63.209096] hw timestamp: tstamp: 1444656871372825314, > tv_sec: > 1444656871, tv_usec: 372825 > [ 63.210897] > [ 63.304068] igb_rx_hwtstamp > [ 63.304849] hwtstamp: 1444656871372825320, syststamp: > 1444656871372845825 > [ 63.306176] system timestamp: tv_sec: 1444656793, > tv_usec: 984280 > [ 63.307391] sys hw timestamp: tstamp: 1444656871372845825, > tv_sec: 1444656871, tv_usec: 372845 > [ 63.309252] hw timestamp: tstamp: 1444656871372825320, > tv_sec: > 1444656871, tv_usec: 372825 > [ 63.311032] > [ 63.404107] igb_rx_hwtstamp > [ 63.404659] hwtstamp: 1444656871372825326, syststamp: > 1444656871372845831 > [ 63.405695] system timestamp: tv_sec: 1444656794, > tv_usec: 83798 > [ 63.406635] sys hw timestamp: tstamp: 1444656871372845831, > tv_sec: 1444656871, tv_usec: 372845 > [ 63.408044] hw timestamp: tstamp: 1444656871372825326, > tv_sec: > 1444656871, tv_usec: 372825 > [ 63.409405] > [ 63.504084] igb_rx_hwtstamp > [ 63.504670] hwtstamp: 1444656871372825332, syststamp: > 1444656871372845837 > [ 63.505715] system timestamp: tv_sec: 1444656794, > tv_usec: 183819 > [ 63.506657] sys hw timestamp: tstamp: 1444656871372845837, > tv_sec: 1444656871, tv_usec: 372845 > [ 63.508090] hw timestamp: tstamp: 1444656871372825332, > tv_sec: > 1444656871, tv_usec: 372825 > > > Any help will be very much appreciated. > > Best regards, > > Mattias > > > > > > > > > On Thu, Oct 8, 2015 at 6:01 PM, Alexander Duyck > <ale...@gm... > <mailto:ale...@gm...>> wrote: > > The i350 should support timestamping all packets as I > recall. You should probably look into getting the > linuxptp package and > seeing if you can make use of the hwstamp_ctl command to > enable timestamping for all incoming packets on the > adapter. > > - Alex > > On 10/08/2015 12:33 AM, Mattias Barthel wrote: > > Hi > thanks for answer. > > Unfortunately, the udp/ip stream just as the cfm > stream is > against only one peer. > Furthemore on the guest side I have only one vcpu > tied to > two physical siblings to minimize L2 cache misses > on ISRs > due to cache coherence. > > I had an idea this morning, maybe I can make use > of the > ptp hw timestaming for this? This should give > better accuracy. > Is this possible or does it only work for > ethertype ptp? > > On Thursday, October 8, 2015, Alexander Duyck > <ale...@gm... > <mailto:ale...@gm...> > <mailto:ale...@gm... > <mailto:ale...@gm...>>> wrote: > > The difference could be Receive Side Scaling > (RSS). The UDP/IP > packets > can be load balanced across all queues via RSS > based > on a hash of the > source and destination addresses. So if you have > multiple sources > generating these packets, or they are being > received > on multiple IP > addresses then it is possible the load is > being spread > out over > multiple > CPUs. Your CFM packets are on an Ethertype > other than > IPv4 or IPv6 so > all packets will end up on queue 0 if I am not > mistaken. > > - Alex > > On 10/07/2015 10:21 PM, Mattias Barthel wrote: > > Hi again! > > > > I realise my message might not have been so > clear. > > > > I will try to clarify. I am doing PM with > software > timestamping. > > > > igb versions: > > version: 3.0.6-k2 > > firmware-version: 0.147-0 > > > > I do timestamping (NTP format) of the > packets in the > same place > for all > > protocols. (in igb_clean_rx_irq_adv()). > > > > For these non ip packets like IEEE 801.2ag > CFM ETH-DM > (ethertype 0x8902) I > > get quite a lot worse accuracy in the > timestamping. > > > > For UDP/IP TWAMP I get for 10k pps a max > delay of > around 140us > back to back. > > For CFM ETH DM I get for the same packet > rate and > packet size a > max delay > > of around 900us. > > > > So my hypothesis was: > > Could FW be putting these L2 packets in another > queue with different > > characteristics? > > > > I tried to add a ET-type filter as written in my > previous mail > but it > > showed no difference. > > > > > > Thanks for any help given! > > > > > > > > On Mon, Oct 5, 2015 at 2:19 PM, Mattias Barthel > <mat...@gm... > <mailto:mat...@gm...> <javascript:;>> > > > wrote: > > > >> Greetings, > >> > >> Im getting quite bad latency when using the igb > driver on i350 > on linux > >> regarding > >> ETH CFM packets. This compared to TWAMP > (IPv4 UDP > packets). > >> The environment is KVM with the i350 devices in > PCI-passthrough. > >> > >> So i figured I add an own rx-queue (filter) for > those types of > ethernet > >> protocol packets. > >> > >> This is what i added: > >> > >> > >> wr32(E1000_ETQF(4), > >> (1 << 31) | /* queue enable */ > >> (E1000_ETQF_FILTER_ENABLE) | /* enable > filter */ > >> (1 << 29) | /* enable immediate > interrupt */ > >> (0x4 << 16) | /* queue no. 4 */ > >> (ETH_P_8021AG)); /* 0x8902 CFM eth > protocol type */ > >> > >> > >> ETH_P_8021A is 0x8902 > >> > >> > >> > >> Is this a correct/good approach? > >> > >> Thanks in advance! > >> > >> -- > >> Mattias Barthel > >> > > > > > ------------------------------------------------------------------------------ > _______________________________________________ > E1000-devel mailing list > E10...@li... > <mailto:E10...@li...> > <javascript:;> > https://lists.sourceforge.net/lists/listinfo/e1000-devel > To learn more about Intel® Ethernet, visit > http://communities.intel.com/community/wired > > > Avis de confidentialité > > Les informations contenues dans le présent message > et dans > toute pièce qui lui est jointe sont confidentielles et > peuvent être protégées par le secret > professionnel. Ces > informations sont à l’usage exclusif de son ou de ses > destinataires. Si vous recevez ce message par erreur, > veuillez s’il vous plait communiquer immédiatement > avec > l’expéditeur et en détruire tout exemplaire. De > plus, il > vous est strictement interdit de le divulguer, de le > distribuer ou de le reproduire sans l’autorisation de > l’expéditeur. Merci. > > Confidentiality notice > > This e-mail message and any attachment hereto contain > confidential information which may be privileged > and which > is intended for the exclusive use of its > addressee(s). If > you receive this message in error, please inform > sender > immediately and destroy any copy thereof. > Furthermore, any > disclosure, distribution or copying of this > message and/or > any attachment hereto without the consent of the > sender is > strictly prohibited. Thank you. > > > > > > -- Mattias Barthel > > > > > -- > Mattias Barthel > > > > Avis de confidentialité > > Les informations contenues dans le présent message et dans toute pièce > qui lui est jointe sont confidentielles et peuvent être protégées par > le secret professionnel. Ces informations sont à l’usage exclusif de > son ou de ses destinataires. Si vous recevez ce message par erreur, > veuillez s’il vous plait communiquer immédiatement avec l’expéditeur > et en détruire tout exemplaire. De plus, il vous est strictement > interdit de le divulguer, de le distribuer ou de le reproduire sans > l’autorisation de l’expéditeur. Merci. > > Confidentiality notice > > This e-mail message and any attachment hereto contain confidential > information which may be privileged and which is intended for the > exclusive use of its addressee(s). If you receive this message in > error, please inform sender immediately and destroy any copy thereof. > Furthermore, any disclosure, distribution or copying of this message > and/or any attachment hereto without the consent of the sender is > strictly prohibited. Thank you. > |
From: Mattias B. <mba...@ac...> - 2015-10-13 17:01:21
|
Yes, setting the cycles.shift to the same as on the 82580 and also shifting the reads accordingly everything works. The sync of shhwtstamps->syststamp too. My driver version is 3.0.6-k2 and linux is 3.0.8. Although this might not be very suited to my needs. The incoming buffers wont be timestamped if you cant keep up with reading the timestamping registers as I understand it. This might be perfectly fine in the case of PTP where the packet rate is fairly low. In my case Im looking at some 80kpps that all would need to be timestamped. There is a comment about this in the driver: */* If this bit is set, then the RX registers contain the time stamp. No** other packet will be time stamped until we read these registers, so**read the registers to make them available again. Because only one** packet can be time stamped at a time, we know that the register**values must belong to this one here and therefore we don't need to** compare any of the additional attributes stored for it.**If nothing went wrong, then it should have a shared tx_flags that we** can turn into a skb_shared_hwtstamps. */* On Tuesday, October 13, 2015, Alexander Duyck <ale...@gm...> wrote: > I'm assuming you are using either an older kernel or an out-of-tree driver > do I have that right? I ask because this code doesn't resemble what is is > currently in the latest Linux kernel. > > Comments on what you have found are inline below. > > - Alex > > On 10/13/2015 12:53 AM, Mattias Barthel wrote: > >> Hi, I think I managed to get around this problem. >> >> >> in function init_hw_timer(): >> ---- >> case e1000_i350: >> case e1000_82580: >> >> /* >> * The 82580 timesync updates the system timer every 8ns by 8ns >> * and the value cannot be shifted. Instead we need to shift >> * the registers to generate a 64bit timer value. As a result >> * SYSTIMR/L/H, TXSTMPL/H, RXSTMPL/H all have to be shifted by >> * 24 in order to generate a larger value for synchronization. >> */ >> adapter->cycles.shift = IGB_82580_TSYNC_SHIFT; >> >> ---- >> >> I have the I350 and I dont think this cycles shift was meant for my >> adapter. >> IGB_82580_TSYNC_SHIFT is 24. >> >> Also when doing igb_read_clock, >> this shifting is applied only for the 82580 and I guess it should not >> have been set in the first place for the I350: >> -------- >> /* >> * The timestamp latches on lowest register read. For the 82580 >> * the lowest register is SYSTIMR instead of SYSTIML. However we never >> * adjusted TIMINCA so SYSTIMR will just read as all 0s so ignore it. >> */ >> if (hw->mac.type == e1000_82580) { >> stamp = rd32(E1000_SYSTIMR) >> 8; >> shift = IGB_82580_TSYNC_SHIFT; >> } >> >> stamp |= (u64)rd32(E1000_SYSTIML) << shift; >> stamp |= (u64)rd32(E1000_SYSTIMH) << (shift + 32); >> ------------- >> >> > This is a bug in the code. The i350 should be shifted by the same value > as the 82580. Then the result would be the same as if you had set the > shift to 0. As I recall the reasoning behind the change was to make use of > as many bits of the time as possible, and on 82580 and newer the SYSTIMH > portion of the clock only contained 8 bits of actual clock data. > > Without that change I believe there is a risk of the counter being > considered to have cycled too fast and causing errors as the cycle counter > will roll over at 40 bits instead of 64. The alternative would be to > update the cycle counter you are using so that it is aware that it is > limited to 40 bits and then use an shift of 0. > > Setting adapter->cycles.shift to 0 makes the clock run at about the same >> speed as the system clock. >> >> Next problem I have is that the syststamp is not syncing correctly >> against the system clock. >> >> Any help is much appreciated. >> >> Regards, >> >> Mattias >> >> >> >> >> >> >> >> >> >> >> >> >> On Mon, Oct 12, 2015 at 5:04 PM, Mattias Barthel < >> mat...@gm... <mailto:mat...@gm...>> wrote: >> >> Hi again, >> >> Started to try and timestamp all packets in the I350. >> >> Driver version: >> >> driver: igb >> version: 3.0.6-k2 >> firmware-version: 0.147-0 >> >> Doing an ioctl that turns on timestamping for all packets. >> >> hwconfig.tx_type = HWTSTAMP_TX_OFF; >> hwconfig.rx_filter = HWTSTAMP_FILTER_ALL; >> hwconfig_requested = hwconfig; >> ioctl(sock, SIOCSHWTSTAMP, &hwtstamp) >> >> The strangest thing is that although it works to read the >> timestamps the clock seems to advance very slowly and/or incorrectly. >> >> I just printout the HW timestamps alongside the systemclock and >> the hw stamping is advancing the nanoseconds when the systemclock >> is advancing us and seconds. >> >> Is there a know bug in this version of the driver that maybe >> initializes the clock incorrectly or maybe reads incorrectly from >> the clock? >> >> >> [ 63.004116] igb_rx_hwtstamp >> [ 63.004684] hwtstamp: 1444656871372825302, syststamp: >> 1444656871372845807 >> [ 63.005721] system timestamp: tv_sec: 1444656793, tv_usec: 683825 >> [ 63.006631] sys hw timestamp: tstamp: 1444656871372845807, >> tv_sec: 1444656871, tv_usec: 372845 >> [ 63.008044] hw timestamp: tstamp: 1444656871372825302, tv_sec: >> 1444656871, tv_usec: 372825 >> [ 63.009395] >> [ 63.104111] igb_rx_hwtstamp >> [ 63.104704] hwtstamp: 1444656871372825308, syststamp: >> 1444656871372845813 >> [ 63.105747] system timestamp: tv_sec: 1444656793, tv_usec: 783851 >> [ 63.106669] sys hw timestamp: tstamp: 1444656871372845813, >> tv_sec: 1444656871, tv_usec: 372845 >> [ 63.108075] hw timestamp: tstamp: 1444656871372825308, tv_sec: >> 1444656871, tv_usec: 372825 >> [ 63.109428] >> [ 63.204109] igb_rx_hwtstamp >> [ 63.204683] hwtstamp: 1444656871372825314, syststamp: >> 1444656871372845819 >> [ 63.206018] system timestamp: tv_sec: 1444656793, tv_usec: 884122 >> [ 63.207242] sys hw timestamp: tstamp: 1444656871372845819, >> tv_sec: 1444656871, tv_usec: 372845 >> [ 63.209096] hw timestamp: tstamp: 1444656871372825314, tv_sec: >> 1444656871, tv_usec: 372825 >> [ 63.210897] >> [ 63.304068] igb_rx_hwtstamp >> [ 63.304849] hwtstamp: 1444656871372825320, syststamp: >> 1444656871372845825 >> [ 63.306176] system timestamp: tv_sec: 1444656793, tv_usec: 984280 >> [ 63.307391] sys hw timestamp: tstamp: 1444656871372845825, >> tv_sec: 1444656871, tv_usec: 372845 >> [ 63.309252] hw timestamp: tstamp: 1444656871372825320, tv_sec: >> 1444656871, tv_usec: 372825 >> [ 63.311032] >> [ 63.404107] igb_rx_hwtstamp >> [ 63.404659] hwtstamp: 1444656871372825326, syststamp: >> 1444656871372845831 >> [ 63.405695] system timestamp: tv_sec: 1444656794, tv_usec: 83798 >> [ 63.406635] sys hw timestamp: tstamp: 1444656871372845831, >> tv_sec: 1444656871, tv_usec: 372845 >> [ 63.408044] hw timestamp: tstamp: 1444656871372825326, tv_sec: >> 1444656871, tv_usec: 372825 >> [ 63.409405] >> [ 63.504084] igb_rx_hwtstamp >> [ 63.504670] hwtstamp: 1444656871372825332, syststamp: >> 1444656871372845837 >> [ 63.505715] system timestamp: tv_sec: 1444656794, tv_usec: 183819 >> [ 63.506657] sys hw timestamp: tstamp: 1444656871372845837, >> tv_sec: 1444656871, tv_usec: 372845 >> [ 63.508090] hw timestamp: tstamp: 1444656871372825332, tv_sec: >> 1444656871, tv_usec: 372825 >> >> >> Any help will be very much appreciated. >> >> Best regards, >> >> Mattias >> >> >> >> >> >> >> >> >> On Thu, Oct 8, 2015 at 6:01 PM, Alexander Duyck >> <ale...@gm... <mailto:ale...@gm...>> wrote: >> >> The i350 should support timestamping all packets as I recall. >> You should probably look into getting the linuxptp package and >> seeing if you can make use of the hwstamp_ctl command to >> enable timestamping for all incoming packets on the adapter. >> >> - Alex >> >> On 10/08/2015 12:33 AM, Mattias Barthel wrote: >> >> Hi >> thanks for answer. >> >> Unfortunately, the udp/ip stream just as the cfm stream is >> against only one peer. >> Furthemore on the guest side I have only one vcpu tied to >> two physical siblings to minimize L2 cache misses on ISRs >> due to cache coherence. >> >> I had an idea this morning, maybe I can make use of the >> ptp hw timestaming for this? This should give better accuracy. >> Is this possible or does it only work for ethertype ptp? >> >> On Thursday, October 8, 2015, Alexander Duyck >> <ale...@gm... >> <mailto:ale...@gm...> >> <mailto:ale...@gm... >> <mailto:ale...@gm...>>> wrote: >> >> The difference could be Receive Side Scaling (RSS). >> The UDP/IP >> packets >> can be load balanced across all queues via RSS based >> on a hash of the >> source and destination addresses. So if you have >> multiple sources >> generating these packets, or they are being received >> on multiple IP >> addresses then it is possible the load is being spread >> out over >> multiple >> CPUs. Your CFM packets are on an Ethertype other than >> IPv4 or IPv6 so >> all packets will end up on queue 0 if I am not mistaken. >> >> - Alex >> >> On 10/07/2015 10:21 PM, Mattias Barthel wrote: >> > Hi again! >> > >> > I realise my message might not have been so clear. >> > >> > I will try to clarify. I am doing PM with software >> timestamping. >> > >> > igb versions: >> > version: 3.0.6-k2 >> > firmware-version: 0.147-0 >> > >> > I do timestamping (NTP format) of the packets in the >> same place >> for all >> > protocols. (in igb_clean_rx_irq_adv()). >> > >> > For these non ip packets like IEEE 801.2ag CFM ETH-DM >> (ethertype 0x8902) I >> > get quite a lot worse accuracy in the timestamping. >> > >> > For UDP/IP TWAMP I get for 10k pps a max delay of >> around 140us >> back to back. >> > For CFM ETH DM I get for the same packet rate and >> packet size a >> max delay >> > of around 900us. >> > >> > So my hypothesis was: >> > Could FW be putting these L2 packets in another >> queue with different >> > characteristics? >> > >> > I tried to add a ET-type filter as written in my >> previous mail >> but it >> > showed no difference. >> > >> > >> > Thanks for any help given! >> > >> > >> > >> > On Mon, Oct 5, 2015 at 2:19 PM, Mattias Barthel >> <mat...@gm... >> <mailto:mat...@gm...> <javascript:;>> >> >> > wrote: >> > >> >> Greetings, >> >> >> >> Im getting quite bad latency when using the igb >> driver on i350 >> on linux >> >> regarding >> >> ETH CFM packets. This compared to TWAMP (IPv4 UDP >> packets). >> >> The environment is KVM with the i350 devices in >> PCI-passthrough. >> >> >> >> So i figured I add an own rx-queue (filter) for >> those types of >> ethernet >> >> protocol packets. >> >> >> >> This is what i added: >> >> >> >> >> >> wr32(E1000_ETQF(4), >> >> (1 << 31) | /* queue enable */ >> >> (E1000_ETQF_FILTER_ENABLE) | /* enable filter */ >> >> (1 << 29) | /* enable immediate interrupt */ >> >> (0x4 << 16) | /* queue no. 4 */ >> >> (ETH_P_8021AG)); /* 0x8902 CFM eth >> protocol type */ >> >> >> >> >> >> ETH_P_8021A is 0x8902 >> >> >> >> >> >> >> >> Is this a correct/good approach? >> >> >> >> Thanks in advance! >> >> >> >> -- >> >> Mattias Barthel >> >> >> > >> >> >> >> ------------------------------------------------------------------------------ >> _______________________________________________ >> E1000-devel mailing list >> E10...@li... >> <mailto:E10...@li...> <javascript:;> >> https://lists.sourceforge.net/lists/listinfo/e1000-devel >> To learn more about Intel® Ethernet, visit >> http://communities.intel.com/community/wired >> >> >> Avis de confidentialité >> >> Les informations contenues dans le présent message et dans >> toute pièce qui lui est jointe sont confidentielles et >> peuvent être protégées par le secret professionnel. Ces >> informations sont à l’usage exclusif de son ou de ses >> destinataires. Si vous recevez ce message par erreur, >> veuillez s’il vous plait communiquer immédiatement avec >> l’expéditeur et en détruire tout exemplaire. De plus, il >> vous est strictement interdit de le divulguer, de le >> distribuer ou de le reproduire sans l’autorisation de >> l’expéditeur. Merci. >> >> Confidentiality notice >> >> This e-mail message and any attachment hereto contain >> confidential information which may be privileged and which >> is intended for the exclusive use of its addressee(s). If >> you receive this message in error, please inform sender >> immediately and destroy any copy thereof. Furthermore, any >> disclosure, distribution or copying of this message and/or >> any attachment hereto without the consent of the sender is >> strictly prohibited. Thank you. >> >> >> >> >> >> -- Mattias Barthel >> >> >> >> >> -- >> Mattias Barthel >> > > -- Avis de confidentialité Les informations contenues dans le présent message et dans toute pièce qui lui est jointe sont confidentielles et peuvent être protégées par le secret professionnel. Ces informations sont à l’usage exclusif de son ou de ses destinataires. Si vous recevez ce message par erreur, veuillez s’il vous plait communiquer immédiatement avec l’expéditeur et en détruire tout exemplaire. De plus, il vous est strictement interdit de le divulguer, de le distribuer ou de le reproduire sans l’autorisation de l’expéditeur. Merci. Confidentiality notice This e-mail message and any attachment hereto contain confidential information which may be privileged and which is intended for the exclusive use of its addressee(s). If you receive this message in error, please inform sender immediately and destroy any copy thereof. Furthermore, any disclosure, distribution or copying of this message and/or any attachment hereto without the consent of the sender is strictly prohibited. Thank you. |
From: Alexander D. <ale...@gm...> - 2015-10-13 15:51:51
|
I'm assuming you are using either an older kernel or an out-of-tree driver do I have that right? I ask because this code doesn't resemble what is is currently in the latest Linux kernel. Comments on what you have found are inline below. - Alex On 10/13/2015 12:53 AM, Mattias Barthel wrote: > Hi, I think I managed to get around this problem. > > > in function init_hw_timer(): > ---- > case e1000_i350: > case e1000_82580: > > /* > * The 82580 timesync updates the system timer every 8ns by 8ns > * and the value cannot be shifted. Instead we need to shift > * the registers to generate a 64bit timer value. As a result > * SYSTIMR/L/H, TXSTMPL/H, RXSTMPL/H all have to be shifted by > * 24 in order to generate a larger value for synchronization. > */ > adapter->cycles.shift = IGB_82580_TSYNC_SHIFT; > > ---- > > I have the I350 and I dont think this cycles shift was meant for my > adapter. > IGB_82580_TSYNC_SHIFT is 24. > > Also when doing igb_read_clock, > this shifting is applied only for the 82580 and I guess it should not > have been set in the first place for the I350: > -------- > /* > * The timestamp latches on lowest register read. For the 82580 > * the lowest register is SYSTIMR instead of SYSTIML. However we never > * adjusted TIMINCA so SYSTIMR will just read as all 0s so ignore it. > */ > if (hw->mac.type == e1000_82580) { > stamp = rd32(E1000_SYSTIMR) >> 8; > shift = IGB_82580_TSYNC_SHIFT; > } > > stamp |= (u64)rd32(E1000_SYSTIML) << shift; > stamp |= (u64)rd32(E1000_SYSTIMH) << (shift + 32); > ------------- > This is a bug in the code. The i350 should be shifted by the same value as the 82580. Then the result would be the same as if you had set the shift to 0. As I recall the reasoning behind the change was to make use of as many bits of the time as possible, and on 82580 and newer the SYSTIMH portion of the clock only contained 8 bits of actual clock data. Without that change I believe there is a risk of the counter being considered to have cycled too fast and causing errors as the cycle counter will roll over at 40 bits instead of 64. The alternative would be to update the cycle counter you are using so that it is aware that it is limited to 40 bits and then use an shift of 0. > Setting adapter->cycles.shift to 0 makes the clock run at about the > same speed as the system clock. > > Next problem I have is that the syststamp is not syncing correctly > against the system clock. > > Any help is much appreciated. > > Regards, > > Mattias > > > > > > > > > > > > > On Mon, Oct 12, 2015 at 5:04 PM, Mattias Barthel > <mat...@gm... <mailto:mat...@gm...>> wrote: > > Hi again, > > Started to try and timestamp all packets in the I350. > > Driver version: > > driver: igb > version: 3.0.6-k2 > firmware-version: 0.147-0 > > Doing an ioctl that turns on timestamping for all packets. > > hwconfig.tx_type = HWTSTAMP_TX_OFF; > hwconfig.rx_filter = HWTSTAMP_FILTER_ALL; > hwconfig_requested = hwconfig; > ioctl(sock, SIOCSHWTSTAMP, &hwtstamp) > > The strangest thing is that although it works to read the > timestamps the clock seems to advance very slowly and/or incorrectly. > > I just printout the HW timestamps alongside the systemclock and > the hw stamping is advancing the nanoseconds when the systemclock > is advancing us and seconds. > > Is there a know bug in this version of the driver that maybe > initializes the clock incorrectly or maybe reads incorrectly from > the clock? > > > [ 63.004116] igb_rx_hwtstamp > [ 63.004684] hwtstamp: 1444656871372825302, syststamp: > 1444656871372845807 > [ 63.005721] system timestamp: tv_sec: 1444656793, tv_usec: 683825 > [ 63.006631] sys hw timestamp: tstamp: 1444656871372845807, > tv_sec: 1444656871, tv_usec: 372845 > [ 63.008044] hw timestamp: tstamp: 1444656871372825302, tv_sec: > 1444656871, tv_usec: 372825 > [ 63.009395] > [ 63.104111] igb_rx_hwtstamp > [ 63.104704] hwtstamp: 1444656871372825308, syststamp: > 1444656871372845813 > [ 63.105747] system timestamp: tv_sec: 1444656793, tv_usec: 783851 > [ 63.106669] sys hw timestamp: tstamp: 1444656871372845813, > tv_sec: 1444656871, tv_usec: 372845 > [ 63.108075] hw timestamp: tstamp: 1444656871372825308, tv_sec: > 1444656871, tv_usec: 372825 > [ 63.109428] > [ 63.204109] igb_rx_hwtstamp > [ 63.204683] hwtstamp: 1444656871372825314, syststamp: > 1444656871372845819 > [ 63.206018] system timestamp: tv_sec: 1444656793, tv_usec: 884122 > [ 63.207242] sys hw timestamp: tstamp: 1444656871372845819, > tv_sec: 1444656871, tv_usec: 372845 > [ 63.209096] hw timestamp: tstamp: 1444656871372825314, tv_sec: > 1444656871, tv_usec: 372825 > [ 63.210897] > [ 63.304068] igb_rx_hwtstamp > [ 63.304849] hwtstamp: 1444656871372825320, syststamp: > 1444656871372845825 > [ 63.306176] system timestamp: tv_sec: 1444656793, tv_usec: 984280 > [ 63.307391] sys hw timestamp: tstamp: 1444656871372845825, > tv_sec: 1444656871, tv_usec: 372845 > [ 63.309252] hw timestamp: tstamp: 1444656871372825320, tv_sec: > 1444656871, tv_usec: 372825 > [ 63.311032] > [ 63.404107] igb_rx_hwtstamp > [ 63.404659] hwtstamp: 1444656871372825326, syststamp: > 1444656871372845831 > [ 63.405695] system timestamp: tv_sec: 1444656794, tv_usec: 83798 > [ 63.406635] sys hw timestamp: tstamp: 1444656871372845831, > tv_sec: 1444656871, tv_usec: 372845 > [ 63.408044] hw timestamp: tstamp: 1444656871372825326, tv_sec: > 1444656871, tv_usec: 372825 > [ 63.409405] > [ 63.504084] igb_rx_hwtstamp > [ 63.504670] hwtstamp: 1444656871372825332, syststamp: > 1444656871372845837 > [ 63.505715] system timestamp: tv_sec: 1444656794, tv_usec: 183819 > [ 63.506657] sys hw timestamp: tstamp: 1444656871372845837, > tv_sec: 1444656871, tv_usec: 372845 > [ 63.508090] hw timestamp: tstamp: 1444656871372825332, tv_sec: > 1444656871, tv_usec: 372825 > > > Any help will be very much appreciated. > > Best regards, > > Mattias > > > > > > > > > On Thu, Oct 8, 2015 at 6:01 PM, Alexander Duyck > <ale...@gm... <mailto:ale...@gm...>> wrote: > > The i350 should support timestamping all packets as I recall. > You should probably look into getting the linuxptp package and > seeing if you can make use of the hwstamp_ctl command to > enable timestamping for all incoming packets on the adapter. > > - Alex > > On 10/08/2015 12:33 AM, Mattias Barthel wrote: > > Hi > thanks for answer. > > Unfortunately, the udp/ip stream just as the cfm stream is > against only one peer. > Furthemore on the guest side I have only one vcpu tied to > two physical siblings to minimize L2 cache misses on ISRs > due to cache coherence. > > I had an idea this morning, maybe I can make use of the > ptp hw timestaming for this? This should give better accuracy. > Is this possible or does it only work for ethertype ptp? > > On Thursday, October 8, 2015, Alexander Duyck > <ale...@gm... > <mailto:ale...@gm...> > <mailto:ale...@gm... > <mailto:ale...@gm...>>> wrote: > > The difference could be Receive Side Scaling (RSS). > The UDP/IP > packets > can be load balanced across all queues via RSS based > on a hash of the > source and destination addresses. So if you have > multiple sources > generating these packets, or they are being received > on multiple IP > addresses then it is possible the load is being spread > out over > multiple > CPUs. Your CFM packets are on an Ethertype other than > IPv4 or IPv6 so > all packets will end up on queue 0 if I am not mistaken. > > - Alex > > On 10/07/2015 10:21 PM, Mattias Barthel wrote: > > Hi again! > > > > I realise my message might not have been so clear. > > > > I will try to clarify. I am doing PM with software > timestamping. > > > > igb versions: > > version: 3.0.6-k2 > > firmware-version: 0.147-0 > > > > I do timestamping (NTP format) of the packets in the > same place > for all > > protocols. (in igb_clean_rx_irq_adv()). > > > > For these non ip packets like IEEE 801.2ag CFM ETH-DM > (ethertype 0x8902) I > > get quite a lot worse accuracy in the timestamping. > > > > For UDP/IP TWAMP I get for 10k pps a max delay of > around 140us > back to back. > > For CFM ETH DM I get for the same packet rate and > packet size a > max delay > > of around 900us. > > > > So my hypothesis was: > > Could FW be putting these L2 packets in another > queue with different > > characteristics? > > > > I tried to add a ET-type filter as written in my > previous mail > but it > > showed no difference. > > > > > > Thanks for any help given! > > > > > > > > On Mon, Oct 5, 2015 at 2:19 PM, Mattias Barthel > <mat...@gm... > <mailto:mat...@gm...> <javascript:;>> > > > wrote: > > > >> Greetings, > >> > >> Im getting quite bad latency when using the igb > driver on i350 > on linux > >> regarding > >> ETH CFM packets. This compared to TWAMP (IPv4 UDP > packets). > >> The environment is KVM with the i350 devices in > PCI-passthrough. > >> > >> So i figured I add an own rx-queue (filter) for > those types of > ethernet > >> protocol packets. > >> > >> This is what i added: > >> > >> > >> wr32(E1000_ETQF(4), > >> (1 << 31) | /* queue enable */ > >> (E1000_ETQF_FILTER_ENABLE) | /* enable filter */ > >> (1 << 29) | /* enable immediate interrupt */ > >> (0x4 << 16) | /* queue no. 4 */ > >> (ETH_P_8021AG)); /* 0x8902 CFM eth > protocol type */ > >> > >> > >> ETH_P_8021A is 0x8902 > >> > >> > >> > >> Is this a correct/good approach? > >> > >> Thanks in advance! > >> > >> -- > >> Mattias Barthel > >> > > > > > ------------------------------------------------------------------------------ > _______________________________________________ > E1000-devel mailing list > E10...@li... > <mailto:E10...@li...> <javascript:;> > https://lists.sourceforge.net/lists/listinfo/e1000-devel > To learn more about Intel® Ethernet, visit > http://communities.intel.com/community/wired > > > Avis de confidentialité > > Les informations contenues dans le présent message et dans > toute pièce qui lui est jointe sont confidentielles et > peuvent être protégées par le secret professionnel. Ces > informations sont à l’usage exclusif de son ou de ses > destinataires. Si vous recevez ce message par erreur, > veuillez s’il vous plait communiquer immédiatement avec > l’expéditeur et en détruire tout exemplaire. De plus, il > vous est strictement interdit de le divulguer, de le > distribuer ou de le reproduire sans l’autorisation de > l’expéditeur. Merci. > > Confidentiality notice > > This e-mail message and any attachment hereto contain > confidential information which may be privileged and which > is intended for the exclusive use of its addressee(s). If > you receive this message in error, please inform sender > immediately and destroy any copy thereof. Furthermore, any > disclosure, distribution or copying of this message and/or > any attachment hereto without the consent of the sender is > strictly prohibited. Thank you. > > > > > > -- > Mattias Barthel > > > > > -- > Mattias Barthel |
From: Mattias B. <mat...@gm...> - 2015-10-13 07:53:29
|
Hi, I think I managed to get around this problem. in function init_hw_timer(): ---- case e1000_i350: case e1000_82580: /* * The 82580 timesync updates the system timer every 8ns by 8ns * and the value cannot be shifted. Instead we need to shift * the registers to generate a 64bit timer value. As a result * SYSTIMR/L/H, TXSTMPL/H, RXSTMPL/H all have to be shifted by * 24 in order to generate a larger value for synchronization. */ adapter->cycles.shift = IGB_82580_TSYNC_SHIFT; ---- I have the I350 and I dont think this cycles shift was meant for my adapter. IGB_82580_TSYNC_SHIFT is 24. Also when doing igb_read_clock, this shifting is applied only for the 82580 and I guess it should not have been set in the first place for the I350: -------- /* * The timestamp latches on lowest register read. For the 82580 * the lowest register is SYSTIMR instead of SYSTIML. However we never * adjusted TIMINCA so SYSTIMR will just read as all 0s so ignore it. */ if (hw->mac.type == e1000_82580) { stamp = rd32(E1000_SYSTIMR) >> 8; shift = IGB_82580_TSYNC_SHIFT; } stamp |= (u64)rd32(E1000_SYSTIML) << shift; stamp |= (u64)rd32(E1000_SYSTIMH) << (shift + 32); ------------- Setting adapter->cycles.shift to 0 makes the clock run at about the same speed as the system clock. Next problem I have is that the syststamp is not syncing correctly against the system clock. Any help is much appreciated. Regards, Mattias On Mon, Oct 12, 2015 at 5:04 PM, Mattias Barthel <mat...@gm...> wrote: > Hi again, > > Started to try and timestamp all packets in the I350. > > Driver version: > > driver: igb > version: 3.0.6-k2 > firmware-version: 0.147-0 > > Doing an ioctl that turns on timestamping for all packets. > > hwconfig.tx_type = HWTSTAMP_TX_OFF; > hwconfig.rx_filter = HWTSTAMP_FILTER_ALL; > hwconfig_requested = hwconfig; > ioctl(sock, SIOCSHWTSTAMP, &hwtstamp) > > The strangest thing is that although it works to read the timestamps the > clock seems to advance very slowly and/or incorrectly. > > I just printout the HW timestamps alongside the systemclock and the hw > stamping is advancing the nanoseconds when the systemclock is advancing us > and seconds. > > Is there a know bug in this version of the driver that maybe initializes > the clock incorrectly or maybe reads incorrectly from the clock? > > > [ 63.004116] igb_rx_hwtstamp > [ 63.004684] hwtstamp: 1444656871372825302, syststamp: > 1444656871372845807 > [ 63.005721] system timestamp: tv_sec: 1444656793, tv_usec: 683825 > [ 63.006631] sys hw timestamp: tstamp: 1444656871372845807, tv_sec: > 1444656871, tv_usec: 372845 > [ 63.008044] hw timestamp: tstamp: 1444656871372825302, tv_sec: > 1444656871, tv_usec: 372825 > [ 63.009395] > [ 63.104111] igb_rx_hwtstamp > [ 63.104704] hwtstamp: 1444656871372825308, syststamp: > 1444656871372845813 > [ 63.105747] system timestamp: tv_sec: 1444656793, tv_usec: 783851 > [ 63.106669] sys hw timestamp: tstamp: 1444656871372845813, tv_sec: > 1444656871, tv_usec: 372845 > [ 63.108075] hw timestamp: tstamp: 1444656871372825308, tv_sec: > 1444656871, tv_usec: 372825 > [ 63.109428] > [ 63.204109] igb_rx_hwtstamp > [ 63.204683] hwtstamp: 1444656871372825314, syststamp: > 1444656871372845819 > [ 63.206018] system timestamp: tv_sec: 1444656793, tv_usec: 884122 > [ 63.207242] sys hw timestamp: tstamp: 1444656871372845819, tv_sec: > 1444656871, tv_usec: 372845 > [ 63.209096] hw timestamp: tstamp: 1444656871372825314, tv_sec: > 1444656871, tv_usec: 372825 > [ 63.210897] > [ 63.304068] igb_rx_hwtstamp > [ 63.304849] hwtstamp: 1444656871372825320, syststamp: > 1444656871372845825 > [ 63.306176] system timestamp: tv_sec: 1444656793, tv_usec: 984280 > [ 63.307391] sys hw timestamp: tstamp: 1444656871372845825, tv_sec: > 1444656871, tv_usec: 372845 > [ 63.309252] hw timestamp: tstamp: 1444656871372825320, tv_sec: > 1444656871, tv_usec: 372825 > [ 63.311032] > [ 63.404107] igb_rx_hwtstamp > [ 63.404659] hwtstamp: 1444656871372825326, syststamp: > 1444656871372845831 > [ 63.405695] system timestamp: tv_sec: 1444656794, tv_usec: 83798 > [ 63.406635] sys hw timestamp: tstamp: 1444656871372845831, tv_sec: > 1444656871, tv_usec: 372845 > [ 63.408044] hw timestamp: tstamp: 1444656871372825326, tv_sec: > 1444656871, tv_usec: 372825 > [ 63.409405] > [ 63.504084] igb_rx_hwtstamp > [ 63.504670] hwtstamp: 1444656871372825332, syststamp: > 1444656871372845837 > [ 63.505715] system timestamp: tv_sec: 1444656794, tv_usec: 183819 > [ 63.506657] sys hw timestamp: tstamp: 1444656871372845837, tv_sec: > 1444656871, tv_usec: 372845 > [ 63.508090] hw timestamp: tstamp: 1444656871372825332, tv_sec: > 1444656871, tv_usec: 372825 > > > Any help will be very much appreciated. > > Best regards, > > Mattias > > > > > > > > > On Thu, Oct 8, 2015 at 6:01 PM, Alexander Duyck <ale...@gm... > > wrote: > >> The i350 should support timestamping all packets as I recall. You should >> probably look into getting the linuxptp package and seeing if you can make >> use of the hwstamp_ctl command to enable timestamping for all incoming >> packets on the adapter. >> >> - Alex >> >> On 10/08/2015 12:33 AM, Mattias Barthel wrote: >> >>> Hi >>> thanks for answer. >>> >>> Unfortunately, the udp/ip stream just as the cfm stream is against only >>> one peer. >>> Furthemore on the guest side I have only one vcpu tied to two physical >>> siblings to minimize L2 cache misses on ISRs due to cache coherence. >>> >>> I had an idea this morning, maybe I can make use of the ptp hw >>> timestaming for this? This should give better accuracy. >>> Is this possible or does it only work for ethertype ptp? >>> >>> On Thursday, October 8, 2015, Alexander Duyck <ale...@gm... >>> <mailto:ale...@gm...>> wrote: >>> >>> The difference could be Receive Side Scaling (RSS). The UDP/IP >>> packets >>> can be load balanced across all queues via RSS based on a hash of the >>> source and destination addresses. So if you have multiple sources >>> generating these packets, or they are being received on multiple IP >>> addresses then it is possible the load is being spread out over >>> multiple >>> CPUs. Your CFM packets are on an Ethertype other than IPv4 or IPv6 so >>> all packets will end up on queue 0 if I am not mistaken. >>> >>> - Alex >>> >>> On 10/07/2015 10:21 PM, Mattias Barthel wrote: >>> > Hi again! >>> > >>> > I realise my message might not have been so clear. >>> > >>> > I will try to clarify. I am doing PM with software timestamping. >>> > >>> > igb versions: >>> > version: 3.0.6-k2 >>> > firmware-version: 0.147-0 >>> > >>> > I do timestamping (NTP format) of the packets in the same place >>> for all >>> > protocols. (in igb_clean_rx_irq_adv()). >>> > >>> > For these non ip packets like IEEE 801.2ag CFM ETH-DM >>> (ethertype 0x8902) I >>> > get quite a lot worse accuracy in the timestamping. >>> > >>> > For UDP/IP TWAMP I get for 10k pps a max delay of around 140us >>> back to back. >>> > For CFM ETH DM I get for the same packet rate and packet size a >>> max delay >>> > of around 900us. >>> > >>> > So my hypothesis was: >>> > Could FW be putting these L2 packets in another queue with >>> different >>> > characteristics? >>> > >>> > I tried to add a ET-type filter as written in my previous mail >>> but it >>> > showed no difference. >>> > >>> > >>> > Thanks for any help given! >>> > >>> > >>> > >>> > On Mon, Oct 5, 2015 at 2:19 PM, Mattias Barthel >>> <mat...@gm... <javascript:;>> >>> >>> > wrote: >>> > >>> >> Greetings, >>> >> >>> >> Im getting quite bad latency when using the igb driver on i350 >>> on linux >>> >> regarding >>> >> ETH CFM packets. This compared to TWAMP (IPv4 UDP packets). >>> >> The environment is KVM with the i350 devices in PCI-passthrough. >>> >> >>> >> So i figured I add an own rx-queue (filter) for those types of >>> ethernet >>> >> protocol packets. >>> >> >>> >> This is what i added: >>> >> >>> >> >>> >> wr32(E1000_ETQF(4), >>> >> (1 << 31) | /* queue enable */ >>> >> (E1000_ETQF_FILTER_ENABLE) | /* enable filter */ >>> >> (1 << 29) | /* enable immediate interrupt */ >>> >> (0x4 << 16) | /* queue no. 4 */ >>> >> (ETH_P_8021AG)); /* 0x8902 CFM eth protocol type */ >>> >> >>> >> >>> >> ETH_P_8021A is 0x8902 >>> >> >>> >> >>> >> >>> >> Is this a correct/good approach? >>> >> >>> >> Thanks in advance! >>> >> >>> >> -- >>> >> Mattias Barthel >>> >> >>> > >>> >>> >>> >>> ------------------------------------------------------------------------------ >>> _______________________________________________ >>> E1000-devel mailing list >>> E10...@li... <javascript:;> >>> https://lists.sourceforge.net/lists/listinfo/e1000-devel >>> To learn more about Intel® Ethernet, visit >>> http://communities.intel.com/community/wired >>> >>> >>> Avis de confidentialité >>> >>> Les informations contenues dans le présent message et dans toute pièce >>> qui lui est jointe sont confidentielles et peuvent être protégées par le >>> secret professionnel. Ces informations sont à l’usage exclusif de son ou de >>> ses destinataires. Si vous recevez ce message par erreur, veuillez s’il >>> vous plait communiquer immédiatement avec l’expéditeur et en détruire tout >>> exemplaire. De plus, il vous est strictement interdit de le divulguer, de >>> le distribuer ou de le reproduire sans l’autorisation de l’expéditeur. >>> Merci. >>> >>> Confidentiality notice >>> >>> This e-mail message and any attachment hereto contain confidential >>> information which may be privileged and which is intended for the exclusive >>> use of its addressee(s). If you receive this message in error, please >>> inform sender immediately and destroy any copy thereof. Furthermore, any >>> disclosure, distribution or copying of this message and/or any attachment >>> hereto without the consent of the sender is strictly prohibited. Thank you. >>> >>> >> > > > -- > Mattias Barthel > -- Mattias Barthel |
From: Mattias B. <mat...@gm...> - 2015-10-12 15:05:03
|
Hi again, Started to try and timestamp all packets in the I350. Driver version: driver: igb version: 3.0.6-k2 firmware-version: 0.147-0 Doing an ioctl that turns on timestamping for all packets. hwconfig.tx_type = HWTSTAMP_TX_OFF; hwconfig.rx_filter = HWTSTAMP_FILTER_ALL; hwconfig_requested = hwconfig; ioctl(sock, SIOCSHWTSTAMP, &hwtstamp) The strangest thing is that although it works to read the timestamps the clock seems to advance very slowly and/or incorrectly. I just printout the HW timestamps alongside the systemclock and the hw stamping is advancing the nanoseconds when the systemclock is advancing us and seconds. Is there a know bug in this version of the driver that maybe initializes the clock incorrectly or maybe reads incorrectly from the clock? [ 63.004116] igb_rx_hwtstamp [ 63.004684] hwtstamp: 1444656871372825302, syststamp: 1444656871372845807 [ 63.005721] system timestamp: tv_sec: 1444656793, tv_usec: 683825 [ 63.006631] sys hw timestamp: tstamp: 1444656871372845807, tv_sec: 1444656871, tv_usec: 372845 [ 63.008044] hw timestamp: tstamp: 1444656871372825302, tv_sec: 1444656871, tv_usec: 372825 [ 63.009395] [ 63.104111] igb_rx_hwtstamp [ 63.104704] hwtstamp: 1444656871372825308, syststamp: 1444656871372845813 [ 63.105747] system timestamp: tv_sec: 1444656793, tv_usec: 783851 [ 63.106669] sys hw timestamp: tstamp: 1444656871372845813, tv_sec: 1444656871, tv_usec: 372845 [ 63.108075] hw timestamp: tstamp: 1444656871372825308, tv_sec: 1444656871, tv_usec: 372825 [ 63.109428] [ 63.204109] igb_rx_hwtstamp [ 63.204683] hwtstamp: 1444656871372825314, syststamp: 1444656871372845819 [ 63.206018] system timestamp: tv_sec: 1444656793, tv_usec: 884122 [ 63.207242] sys hw timestamp: tstamp: 1444656871372845819, tv_sec: 1444656871, tv_usec: 372845 [ 63.209096] hw timestamp: tstamp: 1444656871372825314, tv_sec: 1444656871, tv_usec: 372825 [ 63.210897] [ 63.304068] igb_rx_hwtstamp [ 63.304849] hwtstamp: 1444656871372825320, syststamp: 1444656871372845825 [ 63.306176] system timestamp: tv_sec: 1444656793, tv_usec: 984280 [ 63.307391] sys hw timestamp: tstamp: 1444656871372845825, tv_sec: 1444656871, tv_usec: 372845 [ 63.309252] hw timestamp: tstamp: 1444656871372825320, tv_sec: 1444656871, tv_usec: 372825 [ 63.311032] [ 63.404107] igb_rx_hwtstamp [ 63.404659] hwtstamp: 1444656871372825326, syststamp: 1444656871372845831 [ 63.405695] system timestamp: tv_sec: 1444656794, tv_usec: 83798 [ 63.406635] sys hw timestamp: tstamp: 1444656871372845831, tv_sec: 1444656871, tv_usec: 372845 [ 63.408044] hw timestamp: tstamp: 1444656871372825326, tv_sec: 1444656871, tv_usec: 372825 [ 63.409405] [ 63.504084] igb_rx_hwtstamp [ 63.504670] hwtstamp: 1444656871372825332, syststamp: 1444656871372845837 [ 63.505715] system timestamp: tv_sec: 1444656794, tv_usec: 183819 [ 63.506657] sys hw timestamp: tstamp: 1444656871372845837, tv_sec: 1444656871, tv_usec: 372845 [ 63.508090] hw timestamp: tstamp: 1444656871372825332, tv_sec: 1444656871, tv_usec: 372825 Any help will be very much appreciated. Best regards, Mattias On Thu, Oct 8, 2015 at 6:01 PM, Alexander Duyck <ale...@gm...> wrote: > The i350 should support timestamping all packets as I recall. You should > probably look into getting the linuxptp package and seeing if you can make > use of the hwstamp_ctl command to enable timestamping for all incoming > packets on the adapter. > > - Alex > > On 10/08/2015 12:33 AM, Mattias Barthel wrote: > >> Hi >> thanks for answer. >> >> Unfortunately, the udp/ip stream just as the cfm stream is against only >> one peer. >> Furthemore on the guest side I have only one vcpu tied to two physical >> siblings to minimize L2 cache misses on ISRs due to cache coherence. >> >> I had an idea this morning, maybe I can make use of the ptp hw >> timestaming for this? This should give better accuracy. >> Is this possible or does it only work for ethertype ptp? >> >> On Thursday, October 8, 2015, Alexander Duyck <ale...@gm... >> <mailto:ale...@gm...>> wrote: >> >> The difference could be Receive Side Scaling (RSS). The UDP/IP >> packets >> can be load balanced across all queues via RSS based on a hash of the >> source and destination addresses. So if you have multiple sources >> generating these packets, or they are being received on multiple IP >> addresses then it is possible the load is being spread out over >> multiple >> CPUs. Your CFM packets are on an Ethertype other than IPv4 or IPv6 so >> all packets will end up on queue 0 if I am not mistaken. >> >> - Alex >> >> On 10/07/2015 10:21 PM, Mattias Barthel wrote: >> > Hi again! >> > >> > I realise my message might not have been so clear. >> > >> > I will try to clarify. I am doing PM with software timestamping. >> > >> > igb versions: >> > version: 3.0.6-k2 >> > firmware-version: 0.147-0 >> > >> > I do timestamping (NTP format) of the packets in the same place >> for all >> > protocols. (in igb_clean_rx_irq_adv()). >> > >> > For these non ip packets like IEEE 801.2ag CFM ETH-DM >> (ethertype 0x8902) I >> > get quite a lot worse accuracy in the timestamping. >> > >> > For UDP/IP TWAMP I get for 10k pps a max delay of around 140us >> back to back. >> > For CFM ETH DM I get for the same packet rate and packet size a >> max delay >> > of around 900us. >> > >> > So my hypothesis was: >> > Could FW be putting these L2 packets in another queue with different >> > characteristics? >> > >> > I tried to add a ET-type filter as written in my previous mail >> but it >> > showed no difference. >> > >> > >> > Thanks for any help given! >> > >> > >> > >> > On Mon, Oct 5, 2015 at 2:19 PM, Mattias Barthel >> <mat...@gm... <javascript:;>> >> >> > wrote: >> > >> >> Greetings, >> >> >> >> Im getting quite bad latency when using the igb driver on i350 >> on linux >> >> regarding >> >> ETH CFM packets. This compared to TWAMP (IPv4 UDP packets). >> >> The environment is KVM with the i350 devices in PCI-passthrough. >> >> >> >> So i figured I add an own rx-queue (filter) for those types of >> ethernet >> >> protocol packets. >> >> >> >> This is what i added: >> >> >> >> >> >> wr32(E1000_ETQF(4), >> >> (1 << 31) | /* queue enable */ >> >> (E1000_ETQF_FILTER_ENABLE) | /* enable filter */ >> >> (1 << 29) | /* enable immediate interrupt */ >> >> (0x4 << 16) | /* queue no. 4 */ >> >> (ETH_P_8021AG)); /* 0x8902 CFM eth protocol type */ >> >> >> >> >> >> ETH_P_8021A is 0x8902 >> >> >> >> >> >> >> >> Is this a correct/good approach? >> >> >> >> Thanks in advance! >> >> >> >> -- >> >> Mattias Barthel >> >> >> > >> >> >> >> ------------------------------------------------------------------------------ >> _______________________________________________ >> E1000-devel mailing list >> E10...@li... <javascript:;> >> https://lists.sourceforge.net/lists/listinfo/e1000-devel >> To learn more about Intel® Ethernet, visit >> http://communities.intel.com/community/wired >> >> >> Avis de confidentialité >> >> Les informations contenues dans le présent message et dans toute pièce >> qui lui est jointe sont confidentielles et peuvent être protégées par le >> secret professionnel. Ces informations sont à l’usage exclusif de son ou de >> ses destinataires. Si vous recevez ce message par erreur, veuillez s’il >> vous plait communiquer immédiatement avec l’expéditeur et en détruire tout >> exemplaire. De plus, il vous est strictement interdit de le divulguer, de >> le distribuer ou de le reproduire sans l’autorisation de l’expéditeur. >> Merci. >> >> Confidentiality notice >> >> This e-mail message and any attachment hereto contain confidential >> information which may be privileged and which is intended for the exclusive >> use of its addressee(s). If you receive this message in error, please >> inform sender immediately and destroy any copy thereof. Furthermore, any >> disclosure, distribution or copying of this message and/or any attachment >> hereto without the consent of the sender is strictly prohibited. Thank you. >> >> > -- Mattias Barthel |