SlideShare una empresa de Scribd logo
1 de 20
Descargar para leer sin conexión
Ethernet Services
over IPoIB
Ali Ayoub, Mellanox Technologies
March, 2012

www.openfabrics.org

1
Background
• What is IPoIB?
– IP encapsulation over InfiniBand
– RFC 4391/4392
– Provides IP services over InfiniBand fabric

• Benefits:
– Acts like an data-link within the TCP/IP Stack
• Socket-based apps run transparently

– Allow users to run IP-based unmodified applications
on InfiniBand
– Supports NIC offloads: CSUM, TSO, etc.. And other
performance features such as: NAPI, TSS, RSS, etc..
www.openfabrics.org

2
IPoIB Packet Format
• IPoIB integrated as
Data Link layer
• Sockets API is not
affected

TCP/IP Stack

L5
Sockets API

L4
L3
L2 (IPoIB)
L1 (IB)

• Packet frame:
IB
Header

www.openfabrics.org

IPoIB
Header

IP
Header

Payload

3
Limitations
• IPoIB is limited for IP applications only
– No Ethernet Header encapsulation

• IPoIB network interface doesn’t act like a
standard Ethernet Network Device
•
•
•
•

Non-standard Ethernet interface utilities & characteristics
20 Bytes link-layer address
Ethernet link-layer (MAC) address is 6 Bytes
DHCP requires using dhcp-client identifier

– Host administrator must be aware of PKEYs
• While Ethernet interfaces support VLANs
• vconfig command cannot be used
www.openfabrics.org

4
Limitations cont.
• Link Layer setting, modification, migration
– IPoIB Link Layer address is based on QPN/GID
• Cannot be controlled by the user

– Host administrator cannot set or re-configure the link
layer address (MAC)

IPoIB Link Layer (RFC 4391)
www.openfabrics.org

5
Limitations cont.
• IPoIB cannot be used in fully-virtualized or
para-virtualized environments
– Hypervisor networking model normally use “bridged
mode”
– Virtual Switch (vSwitch) is Ethernet L2 switch
– IPoIB NIC cannot be enslaved to a vSwitch

• Promiscuous mode is unsupported
– Simplifies vSwitch functionality

www.openfabrics.org

6
IPoIB & Network Virtualization
VM “sees” an Ethernet
Network Device (e1000,
virtio, vmxnet, etc..)

Hypervisor

Packet sent out by the VM
includes an Ethernet
header

Virtual
Interface(s)
(vifX)
Virtual
Switch(s)
(vSwitchX)
Uplink(s)
(pifX)

vSwitch0

vSwitch1

pif0

Network Virtualization Model

vSwitch forwards the
Ethernet packet to the host
Network Interface
pif1

Native IPoIB driver serves
only IP packets (doesn’t
expect Ethernet Header)
Reference http://www.vmware.com/resources/techresources/997

www.openfabrics.org

7
Solution
• Create a new Ethernet Network Device
over IPoIB interface
• Managed by “eIPoIB” kernel module
• Registers a standard Ethernet Interface
into the Operating System
• Same “Look & Feel” as an Ethernet
NIC

TCP/IP
IP
Frame

Ethernet
Ethernet
Frame

eIPoIB
Shim Layer
Strip
Ethernet
Header

– ifconfig, vconfig, ethtool, promiscuous, etc..

• eIPoIB is a shim layer driver between
the TCP/IP stack and the native IPoIB
www.openfabrics.org

Push
Ethernet
Header

IP
Frame

Strip
IPoIB
Header

IPoIB
Push
IPoIB
Header

IPoIB
Frame

IB layer
8
eIPoIB vs. IPoIB Interface
• ifconfig ib0

www.openfabrics.org

9
eIPoIB vs. IPoIB Interface
• ifconfig eth3

www.openfabrics.org

10
eIPoIB Operations in a Nutshell
• Initialization:

TCP/IP

– Register Ethernet network interface (eth0)
– Map it to native IPoIB interface (ib0)

IP
Frame

• Transmit:
– Packet to be sent has Ethernet Header
– eIPoIB strips the Ethernet Header
– Pass the IP packet to the underlying IPoIB interface.

Ethernet
Ethernet
Frame

• Non IP/ARP/RARP packets are dropped

– Native IPoIB interface sends the packet out

eIPoIB
Shim Layer

• Receive:
–
–
–
–

Packet received is an IP packet
Native IPoIB handovers the IP packet to eIPoIB layer
eIPoIB pushes the Ethernet header
Packet forwarded to upper layers as regular Ethernet
frame

Push
Ethernet
Header

Strip
Ethernet
Header

IP
Frame

Strip
IPoIB
Header

IPoIB
Push
IPoIB
Header

IPoIB
Frame

IB layer
www.openfabrics.org

11
eIPoIB Frame Flow
IP
Header

IP Data
TCP/UDP

TCP/IP
IP
Frame

Ethernet
Header

Ethernet

Ethernet Data

Ethernet
Frame

IP
Header

IP Data
TCP/UDP

eIPoIB
Shim Layer
Strip
Ethernet
Header

IPoIB
Header

IPoIB

IP Data
Push
IPoIB
Header

www.openfabrics.org

IPoIB
Header

IP Data

CRC

InfiniBand
Header

IP
Frame

IPoIB
Frame

IB layer
12
eIPoIB/IPoIB Interoperability
TCP/IP

• Same wire protocol
• eIPoIB “talks” IPoIB

IP
Frame

Ethernet
Ethernet
Frame

TCP/IP

eIPoIB
Shim Layer
Strip
Ethernet
Header

IPoIB

www.openfabrics.org

IP
Frame

Strip
IPoIB
Header

IPoIB

InfiniBand
Header

IPoIB
Header

Ethernet Data

CRC

IB layer

Push
Ethernet
Header

Push
IPoIB
Header

IPoIB
Frame

IB layer
13
eIPoIB Networking Model (KVM)

eth0
ib0

Reference http://www.linux-kvm.org/wiki
www.openfabrics.org

14
eIPoIB Design (KVM)
vm1

vif0.1

vm2

vif0.2

vm.x

vif0.3

vif0.x

QEMU

TCP/IP

tap0.1

tap0.2

tap0.3

tap0.x

br0
eth0
ib0.0

ib0.1

ib0.2

CMA
ib0.3

ib0.x

ib0

IB port 0

www.openfabrics.org

15
eIPoIB – Closer Look
• Promiscuous mode
• MAC Translation

www.openfabrics.org

16
Promiscuous Mode
• Normally vSwitch uplink is put in
promiscuous mode
– InfiniBand doesn’t support promiscuous mode

• Promiscuous mode is simulated by:
a. Snooping the src.mac and vlan of outgoing packets
b. OS notifies the driver when a new MAC/VLAN need
to be “served”. For example: the driver can get a
notification from libvirt library (available for
KVM/XEN) when a new VM virtual NIC is created

• Multicast promiscuous support requires
IGMP/MLD Snooping in eIPoIB level
www.openfabrics.org

17
MAC Translation
• Requirements:
– eIPoIB exposes an Ethernet MAC
• Local Ethernet MAC (LEMAC) 6 bytes length

– eIPoIB neighbors are seen as Ethernet neighbors
• Remote Ethernet MAC (REMAC) 6 bytes length

– IPoIB Local MAC used on the wire
• Local IPoIB MAC (LIMAC) 20 bytes

– IPoIB neighbors’ MAC used on the wire
• Remote IPoIB MAC (RIMAC) 20 bytes

www.openfabrics.org

18
MAC Translation cont.
Receive Flow
• Receiver QP => dst.mac
• SQPN/SLID => src.mac
–

Remember QPN/LID to IPoIB-MAC
(QPN/GID) mapping

• IPoIB header => Ethernet
ethertype
• Replace IPoIB header by
Ethernet header

Transmit Flow
• src.mac => QP
(child interface)
– Source MAC normally controlled by the
host/hypervisor admin

• dst.mac => IPoIB-MAC
• Ethernet header ethertype
=> IPoIB Header
• Strip Ethernet header, and
handover packet to IPoIB

ARP/NDP packets are modified in TX/RX flow, so the MAC addresses
in the packet payload are updated accordingly
www.openfabrics.org

19
Questions?

www.openfabrics.org

20

Más contenido relacionado

La actualidad más candente

Route flow autoconf demo 2nd sdn world congress 2013
Route flow autoconf demo   2nd sdn world congress 2013Route flow autoconf demo   2nd sdn world congress 2013
Route flow autoconf demo 2nd sdn world congress 2013FIBRE Testbed
 
ONIE LinuxCon 2015
ONIE LinuxCon 2015ONIE LinuxCon 2015
ONIE LinuxCon 2015Curt Brune
 
ONIE: Open Network Install Environment @ OSDC 2014 Netways, Berlin
ONIE: Open Network Install Environment @ OSDC 2014 Netways, BerlinONIE: Open Network Install Environment @ OSDC 2014 Netways, Berlin
ONIE: Open Network Install Environment @ OSDC 2014 Netways, BerlinNat Morris
 
The Switch as a Server - PuppetConf 2014
The Switch as a Server - PuppetConf 2014The Switch as a Server - PuppetConf 2014
The Switch as a Server - PuppetConf 2014Puppet
 
OSDC 2014 ONIE by Nat Morris
OSDC 2014 ONIE by Nat MorrisOSDC 2014 ONIE by Nat Morris
OSDC 2014 ONIE by Nat MorrisCumulus Networks
 
Cisco's journey from Verbs to Libfabric
Cisco's journey from Verbs to LibfabricCisco's journey from Verbs to Libfabric
Cisco's journey from Verbs to LibfabricJeff Squyres
 
Understanding Open vSwitch
Understanding Open vSwitch Understanding Open vSwitch
Understanding Open vSwitch YongKi Kim
 
Cumulus networks conversion guide
Cumulus networks conversion guideCumulus networks conversion guide
Cumulus networks conversion guideScott Suehle
 
Introduction to nexux from zero to Hero
Introduction to nexux  from zero to HeroIntroduction to nexux  from zero to Hero
Introduction to nexux from zero to HeroDhruv Sharma
 
Kubernetes networking-made-easy-with-open-v switch
Kubernetes networking-made-easy-with-open-v switchKubernetes networking-made-easy-with-open-v switch
Kubernetes networking-made-easy-with-open-v switchInfraEngineer
 
Kamailio with Docker and Kubernetes
Kamailio with Docker and KubernetesKamailio with Docker and Kubernetes
Kamailio with Docker and KubernetesPaolo Visintin
 
Open Networking for Your OpenStack
Open Networking for Your OpenStackOpen Networking for Your OpenStack
Open Networking for Your OpenStackCumulus Networks
 
Openstack Neutron & Interconnections with BGP/MPLS VPNs
Openstack Neutron & Interconnections with BGP/MPLS VPNsOpenstack Neutron & Interconnections with BGP/MPLS VPNs
Openstack Neutron & Interconnections with BGP/MPLS VPNsThomas Morin
 
The linux kernel hidden inside windows 10
The linux kernel hidden inside windows 10The linux kernel hidden inside windows 10
The linux kernel hidden inside windows 10mark-smith
 
Interconnecting Neutron and Network Operators' BGP VPNs
Interconnecting Neutron and Network Operators' BGP VPNsInterconnecting Neutron and Network Operators' BGP VPNs
Interconnecting Neutron and Network Operators' BGP VPNsThomas Morin
 
Linux firmware for iRMC controller on Fujitsu Primergy servers
Linux firmware for iRMC controller on Fujitsu Primergy serversLinux firmware for iRMC controller on Fujitsu Primergy servers
Linux firmware for iRMC controller on Fujitsu Primergy serversVladimir Shakhov
 
20171010 multitenancy in openshift
20171010 multitenancy in openshift20171010 multitenancy in openshift
20171010 multitenancy in openshiftSmals
 

La actualidad más candente (20)

Route flow autoconf demo 2nd sdn world congress 2013
Route flow autoconf demo   2nd sdn world congress 2013Route flow autoconf demo   2nd sdn world congress 2013
Route flow autoconf demo 2nd sdn world congress 2013
 
ONIE LinuxCon 2015
ONIE LinuxCon 2015ONIE LinuxCon 2015
ONIE LinuxCon 2015
 
ONIE: Open Network Install Environment @ OSDC 2014 Netways, Berlin
ONIE: Open Network Install Environment @ OSDC 2014 Netways, BerlinONIE: Open Network Install Environment @ OSDC 2014 Netways, Berlin
ONIE: Open Network Install Environment @ OSDC 2014 Netways, Berlin
 
The Switch as a Server - PuppetConf 2014
The Switch as a Server - PuppetConf 2014The Switch as a Server - PuppetConf 2014
The Switch as a Server - PuppetConf 2014
 
OSDC 2014 ONIE by Nat Morris
OSDC 2014 ONIE by Nat MorrisOSDC 2014 ONIE by Nat Morris
OSDC 2014 ONIE by Nat Morris
 
Cisco's journey from Verbs to Libfabric
Cisco's journey from Verbs to LibfabricCisco's journey from Verbs to Libfabric
Cisco's journey from Verbs to Libfabric
 
Understanding Open vSwitch
Understanding Open vSwitch Understanding Open vSwitch
Understanding Open vSwitch
 
Report on OFELIA
Report on OFELIAReport on OFELIA
Report on OFELIA
 
Cumulus networks conversion guide
Cumulus networks conversion guideCumulus networks conversion guide
Cumulus networks conversion guide
 
Introduction to nexux from zero to Hero
Introduction to nexux  from zero to HeroIntroduction to nexux  from zero to Hero
Introduction to nexux from zero to Hero
 
Kubernetes networking-made-easy-with-open-v switch
Kubernetes networking-made-easy-with-open-v switchKubernetes networking-made-easy-with-open-v switch
Kubernetes networking-made-easy-with-open-v switch
 
Kamailio with Docker and Kubernetes
Kamailio with Docker and KubernetesKamailio with Docker and Kubernetes
Kamailio with Docker and Kubernetes
 
Open Networking for Your OpenStack
Open Networking for Your OpenStackOpen Networking for Your OpenStack
Open Networking for Your OpenStack
 
Openstack Neutron & Interconnections with BGP/MPLS VPNs
Openstack Neutron & Interconnections with BGP/MPLS VPNsOpenstack Neutron & Interconnections with BGP/MPLS VPNs
Openstack Neutron & Interconnections with BGP/MPLS VPNs
 
The linux kernel hidden inside windows 10
The linux kernel hidden inside windows 10The linux kernel hidden inside windows 10
The linux kernel hidden inside windows 10
 
Interconnecting Neutron and Network Operators' BGP VPNs
Interconnecting Neutron and Network Operators' BGP VPNsInterconnecting Neutron and Network Operators' BGP VPNs
Interconnecting Neutron and Network Operators' BGP VPNs
 
IPLOOK vEPC solution
IPLOOK vEPC solutionIPLOOK vEPC solution
IPLOOK vEPC solution
 
Linux firmware for iRMC controller on Fujitsu Primergy servers
Linux firmware for iRMC controller on Fujitsu Primergy serversLinux firmware for iRMC controller on Fujitsu Primergy servers
Linux firmware for iRMC controller on Fujitsu Primergy servers
 
Cumulus Linux 2.5.4
Cumulus Linux 2.5.4Cumulus Linux 2.5.4
Cumulus Linux 2.5.4
 
20171010 multitenancy in openshift
20171010 multitenancy in openshift20171010 multitenancy in openshift
20171010 multitenancy in openshift
 

Similar a 2012 workshop wed_ethernet_servicesoveri_poib

Practical Guide to Run an IEEE 802.15.4 Network with 6LoWPAN Under Linux
Practical Guide to Run an IEEE 802.15.4 Network with 6LoWPAN Under LinuxPractical Guide to Run an IEEE 802.15.4 Network with 6LoWPAN Under Linux
Practical Guide to Run an IEEE 802.15.4 Network with 6LoWPAN Under LinuxSamsung Open Source Group
 
Networking in Docker Containers
Networking in Docker ContainersNetworking in Docker Containers
Networking in Docker ContainersAttila Kanto
 
Adding IEEE 802.15.4 and 6LoWPAN to an Embedded Linux Device
Adding IEEE 802.15.4 and 6LoWPAN to an Embedded Linux DeviceAdding IEEE 802.15.4 and 6LoWPAN to an Embedded Linux Device
Adding IEEE 802.15.4 and 6LoWPAN to an Embedded Linux DeviceSamsung Open Source Group
 
Scaling the Container Dataplane
Scaling the Container Dataplane Scaling the Container Dataplane
Scaling the Container Dataplane Michelle Holley
 
SDN/OpenFlow #lspe
SDN/OpenFlow #lspeSDN/OpenFlow #lspe
SDN/OpenFlow #lspeChris Westin
 
LinuxConJapan2014_makita_0_MACVLAN.pdf
LinuxConJapan2014_makita_0_MACVLAN.pdfLinuxConJapan2014_makita_0_MACVLAN.pdf
LinuxConJapan2014_makita_0_MACVLAN.pdfDanielHanganu2
 
DPDK Summit - 08 Sept 2014 - Futurewei - Jun Xu - Revisit the IP Stack in Lin...
DPDK Summit - 08 Sept 2014 - Futurewei - Jun Xu - Revisit the IP Stack in Lin...DPDK Summit - 08 Sept 2014 - Futurewei - Jun Xu - Revisit the IP Stack in Lin...
DPDK Summit - 08 Sept 2014 - Futurewei - Jun Xu - Revisit the IP Stack in Lin...Jim St. Leger
 
"One network to rule them all" - OpenStack Summit Austin 2016
"One network to rule them all" - OpenStack Summit Austin 2016"One network to rule them all" - OpenStack Summit Austin 2016
"One network to rule them all" - OpenStack Summit Austin 2016Phil Estes
 
Flexible NFV WAN interconnections with Neutron BGP VPN
 Flexible NFV WAN interconnections with Neutron BGP VPN Flexible NFV WAN interconnections with Neutron BGP VPN
Flexible NFV WAN interconnections with Neutron BGP VPNThomas Morin
 
XPDS14 - Xen as High-Performance NFV Platform - Jun Nakajima, Intel
XPDS14 - Xen as High-Performance NFV Platform - Jun Nakajima, IntelXPDS14 - Xen as High-Performance NFV Platform - Jun Nakajima, Intel
XPDS14 - Xen as High-Performance NFV Platform - Jun Nakajima, IntelThe Linux Foundation
 
PLNOG 6: Konrad Plich, Robert Woźny - TPIX - How to connect two IXes?
PLNOG 6: Konrad Plich, Robert Woźny - TPIX - How to connect two IXes? PLNOG 6: Konrad Plich, Robert Woźny - TPIX - How to connect two IXes?
PLNOG 6: Konrad Plich, Robert Woźny - TPIX - How to connect two IXes? PROIDEA
 
Unikernels: Rise of the Library Hypervisor
Unikernels: Rise of the Library HypervisorUnikernels: Rise of the Library Hypervisor
Unikernels: Rise of the Library HypervisorAnil Madhavapeddy
 
OpenStack and OpenContrail for FreeBSD platform by Michał Dubiel
OpenStack and OpenContrail for FreeBSD platform by Michał DubielOpenStack and OpenContrail for FreeBSD platform by Michał Dubiel
OpenStack and OpenContrail for FreeBSD platform by Michał Dubieleurobsdcon
 
neutron_icehouse_update
neutron_icehouse_updateneutron_icehouse_update
neutron_icehouse_updateAkihiro Motoki
 
CommScope RUCKUS ICX Switching Configuration
CommScope RUCKUS ICX Switching ConfigurationCommScope RUCKUS ICX Switching Configuration
CommScope RUCKUS ICX Switching ConfigurationCarla Nadin
 
Learning how AWS implement AWS VPC CNI
Learning how AWS implement AWS VPC CNILearning how AWS implement AWS VPC CNI
Learning how AWS implement AWS VPC CNIHungWei Chiu
 
Unikernels: the rise of the library hypervisor in MirageOS
Unikernels: the rise of the library hypervisor in MirageOSUnikernels: the rise of the library hypervisor in MirageOS
Unikernels: the rise of the library hypervisor in MirageOSDocker, Inc.
 

Similar a 2012 workshop wed_ethernet_servicesoveri_poib (20)

Practical Guide to Run an IEEE 802.15.4 Network with 6LoWPAN Under Linux
Practical Guide to Run an IEEE 802.15.4 Network with 6LoWPAN Under LinuxPractical Guide to Run an IEEE 802.15.4 Network with 6LoWPAN Under Linux
Practical Guide to Run an IEEE 802.15.4 Network with 6LoWPAN Under Linux
 
Networking in Docker Containers
Networking in Docker ContainersNetworking in Docker Containers
Networking in Docker Containers
 
Adding IEEE 802.15.4 and 6LoWPAN to an Embedded Linux Device
Adding IEEE 802.15.4 and 6LoWPAN to an Embedded Linux DeviceAdding IEEE 802.15.4 and 6LoWPAN to an Embedded Linux Device
Adding IEEE 802.15.4 and 6LoWPAN to an Embedded Linux Device
 
Scaling the Container Dataplane
Scaling the Container Dataplane Scaling the Container Dataplane
Scaling the Container Dataplane
 
SDN/OpenFlow #lspe
SDN/OpenFlow #lspeSDN/OpenFlow #lspe
SDN/OpenFlow #lspe
 
Run Your Own 6LoWPAN Based IoT Network
Run Your Own 6LoWPAN Based IoT NetworkRun Your Own 6LoWPAN Based IoT Network
Run Your Own 6LoWPAN Based IoT Network
 
LinuxConJapan2014_makita_0_MACVLAN.pdf
LinuxConJapan2014_makita_0_MACVLAN.pdfLinuxConJapan2014_makita_0_MACVLAN.pdf
LinuxConJapan2014_makita_0_MACVLAN.pdf
 
DPDK Summit - 08 Sept 2014 - Futurewei - Jun Xu - Revisit the IP Stack in Lin...
DPDK Summit - 08 Sept 2014 - Futurewei - Jun Xu - Revisit the IP Stack in Lin...DPDK Summit - 08 Sept 2014 - Futurewei - Jun Xu - Revisit the IP Stack in Lin...
DPDK Summit - 08 Sept 2014 - Futurewei - Jun Xu - Revisit the IP Stack in Lin...
 
"One network to rule them all" - OpenStack Summit Austin 2016
"One network to rule them all" - OpenStack Summit Austin 2016"One network to rule them all" - OpenStack Summit Austin 2016
"One network to rule them all" - OpenStack Summit Austin 2016
 
Flexible NFV WAN interconnections with Neutron BGP VPN
 Flexible NFV WAN interconnections with Neutron BGP VPN Flexible NFV WAN interconnections with Neutron BGP VPN
Flexible NFV WAN interconnections with Neutron BGP VPN
 
XPDS14 - Xen as High-Performance NFV Platform - Jun Nakajima, Intel
XPDS14 - Xen as High-Performance NFV Platform - Jun Nakajima, IntelXPDS14 - Xen as High-Performance NFV Platform - Jun Nakajima, Intel
XPDS14 - Xen as High-Performance NFV Platform - Jun Nakajima, Intel
 
PLNOG 6: Konrad Plich, Robert Woźny - TPIX - How to connect two IXes?
PLNOG 6: Konrad Plich, Robert Woźny - TPIX - How to connect two IXes? PLNOG 6: Konrad Plich, Robert Woźny - TPIX - How to connect two IXes?
PLNOG 6: Konrad Plich, Robert Woźny - TPIX - How to connect two IXes?
 
Unikernels: Rise of the Library Hypervisor
Unikernels: Rise of the Library HypervisorUnikernels: Rise of the Library Hypervisor
Unikernels: Rise of the Library Hypervisor
 
OpenStack and OpenContrail for FreeBSD platform by Michał Dubiel
OpenStack and OpenContrail for FreeBSD platform by Michał DubielOpenStack and OpenContrail for FreeBSD platform by Michał Dubiel
OpenStack and OpenContrail for FreeBSD platform by Michał Dubiel
 
neutron_icehouse_update
neutron_icehouse_updateneutron_icehouse_update
neutron_icehouse_update
 
Network virtualization
Network virtualizationNetwork virtualization
Network virtualization
 
CommScope RUCKUS ICX Switching Configuration
CommScope RUCKUS ICX Switching ConfigurationCommScope RUCKUS ICX Switching Configuration
CommScope RUCKUS ICX Switching Configuration
 
Learning how AWS implement AWS VPC CNI
Learning how AWS implement AWS VPC CNILearning how AWS implement AWS VPC CNI
Learning how AWS implement AWS VPC CNI
 
OpenStack Quantum
OpenStack QuantumOpenStack Quantum
OpenStack Quantum
 
Unikernels: the rise of the library hypervisor in MirageOS
Unikernels: the rise of the library hypervisor in MirageOSUnikernels: the rise of the library hypervisor in MirageOS
Unikernels: the rise of the library hypervisor in MirageOS
 

Último

Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXTarek Kalaji
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfJamie (Taka) Wang
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarPrecisely
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfDaniel Santiago Silva Capera
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 

Último (20)

Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBX
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity Webinar
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 

2012 workshop wed_ethernet_servicesoveri_poib

  • 1. Ethernet Services over IPoIB Ali Ayoub, Mellanox Technologies March, 2012 www.openfabrics.org 1
  • 2. Background • What is IPoIB? – IP encapsulation over InfiniBand – RFC 4391/4392 – Provides IP services over InfiniBand fabric • Benefits: – Acts like an data-link within the TCP/IP Stack • Socket-based apps run transparently – Allow users to run IP-based unmodified applications on InfiniBand – Supports NIC offloads: CSUM, TSO, etc.. And other performance features such as: NAPI, TSS, RSS, etc.. www.openfabrics.org 2
  • 3. IPoIB Packet Format • IPoIB integrated as Data Link layer • Sockets API is not affected TCP/IP Stack L5 Sockets API L4 L3 L2 (IPoIB) L1 (IB) • Packet frame: IB Header www.openfabrics.org IPoIB Header IP Header Payload 3
  • 4. Limitations • IPoIB is limited for IP applications only – No Ethernet Header encapsulation • IPoIB network interface doesn’t act like a standard Ethernet Network Device • • • • Non-standard Ethernet interface utilities & characteristics 20 Bytes link-layer address Ethernet link-layer (MAC) address is 6 Bytes DHCP requires using dhcp-client identifier – Host administrator must be aware of PKEYs • While Ethernet interfaces support VLANs • vconfig command cannot be used www.openfabrics.org 4
  • 5. Limitations cont. • Link Layer setting, modification, migration – IPoIB Link Layer address is based on QPN/GID • Cannot be controlled by the user – Host administrator cannot set or re-configure the link layer address (MAC) IPoIB Link Layer (RFC 4391) www.openfabrics.org 5
  • 6. Limitations cont. • IPoIB cannot be used in fully-virtualized or para-virtualized environments – Hypervisor networking model normally use “bridged mode” – Virtual Switch (vSwitch) is Ethernet L2 switch – IPoIB NIC cannot be enslaved to a vSwitch • Promiscuous mode is unsupported – Simplifies vSwitch functionality www.openfabrics.org 6
  • 7. IPoIB & Network Virtualization VM “sees” an Ethernet Network Device (e1000, virtio, vmxnet, etc..) Hypervisor Packet sent out by the VM includes an Ethernet header Virtual Interface(s) (vifX) Virtual Switch(s) (vSwitchX) Uplink(s) (pifX) vSwitch0 vSwitch1 pif0 Network Virtualization Model vSwitch forwards the Ethernet packet to the host Network Interface pif1 Native IPoIB driver serves only IP packets (doesn’t expect Ethernet Header) Reference http://www.vmware.com/resources/techresources/997 www.openfabrics.org 7
  • 8. Solution • Create a new Ethernet Network Device over IPoIB interface • Managed by “eIPoIB” kernel module • Registers a standard Ethernet Interface into the Operating System • Same “Look & Feel” as an Ethernet NIC TCP/IP IP Frame Ethernet Ethernet Frame eIPoIB Shim Layer Strip Ethernet Header – ifconfig, vconfig, ethtool, promiscuous, etc.. • eIPoIB is a shim layer driver between the TCP/IP stack and the native IPoIB www.openfabrics.org Push Ethernet Header IP Frame Strip IPoIB Header IPoIB Push IPoIB Header IPoIB Frame IB layer 8
  • 9. eIPoIB vs. IPoIB Interface • ifconfig ib0 www.openfabrics.org 9
  • 10. eIPoIB vs. IPoIB Interface • ifconfig eth3 www.openfabrics.org 10
  • 11. eIPoIB Operations in a Nutshell • Initialization: TCP/IP – Register Ethernet network interface (eth0) – Map it to native IPoIB interface (ib0) IP Frame • Transmit: – Packet to be sent has Ethernet Header – eIPoIB strips the Ethernet Header – Pass the IP packet to the underlying IPoIB interface. Ethernet Ethernet Frame • Non IP/ARP/RARP packets are dropped – Native IPoIB interface sends the packet out eIPoIB Shim Layer • Receive: – – – – Packet received is an IP packet Native IPoIB handovers the IP packet to eIPoIB layer eIPoIB pushes the Ethernet header Packet forwarded to upper layers as regular Ethernet frame Push Ethernet Header Strip Ethernet Header IP Frame Strip IPoIB Header IPoIB Push IPoIB Header IPoIB Frame IB layer www.openfabrics.org 11
  • 12. eIPoIB Frame Flow IP Header IP Data TCP/UDP TCP/IP IP Frame Ethernet Header Ethernet Ethernet Data Ethernet Frame IP Header IP Data TCP/UDP eIPoIB Shim Layer Strip Ethernet Header IPoIB Header IPoIB IP Data Push IPoIB Header www.openfabrics.org IPoIB Header IP Data CRC InfiniBand Header IP Frame IPoIB Frame IB layer 12
  • 13. eIPoIB/IPoIB Interoperability TCP/IP • Same wire protocol • eIPoIB “talks” IPoIB IP Frame Ethernet Ethernet Frame TCP/IP eIPoIB Shim Layer Strip Ethernet Header IPoIB www.openfabrics.org IP Frame Strip IPoIB Header IPoIB InfiniBand Header IPoIB Header Ethernet Data CRC IB layer Push Ethernet Header Push IPoIB Header IPoIB Frame IB layer 13
  • 14. eIPoIB Networking Model (KVM) eth0 ib0 Reference http://www.linux-kvm.org/wiki www.openfabrics.org 14
  • 16. eIPoIB – Closer Look • Promiscuous mode • MAC Translation www.openfabrics.org 16
  • 17. Promiscuous Mode • Normally vSwitch uplink is put in promiscuous mode – InfiniBand doesn’t support promiscuous mode • Promiscuous mode is simulated by: a. Snooping the src.mac and vlan of outgoing packets b. OS notifies the driver when a new MAC/VLAN need to be “served”. For example: the driver can get a notification from libvirt library (available for KVM/XEN) when a new VM virtual NIC is created • Multicast promiscuous support requires IGMP/MLD Snooping in eIPoIB level www.openfabrics.org 17
  • 18. MAC Translation • Requirements: – eIPoIB exposes an Ethernet MAC • Local Ethernet MAC (LEMAC) 6 bytes length – eIPoIB neighbors are seen as Ethernet neighbors • Remote Ethernet MAC (REMAC) 6 bytes length – IPoIB Local MAC used on the wire • Local IPoIB MAC (LIMAC) 20 bytes – IPoIB neighbors’ MAC used on the wire • Remote IPoIB MAC (RIMAC) 20 bytes www.openfabrics.org 18
  • 19. MAC Translation cont. Receive Flow • Receiver QP => dst.mac • SQPN/SLID => src.mac – Remember QPN/LID to IPoIB-MAC (QPN/GID) mapping • IPoIB header => Ethernet ethertype • Replace IPoIB header by Ethernet header Transmit Flow • src.mac => QP (child interface) – Source MAC normally controlled by the host/hypervisor admin • dst.mac => IPoIB-MAC • Ethernet header ethertype => IPoIB Header • Strip Ethernet header, and handover packet to IPoIB ARP/NDP packets are modified in TX/RX flow, so the MAC addresses in the packet payload are updated accordingly www.openfabrics.org 19