It talks about very basic concepts such as frames, broadcast domain, IP addresses, routing and a bit of IPoIB.

1. <Insert Picture Here>
Network TOI for NAS (Basic)
2013.11.27
Hisao Tsujimura
Sr. Technical Support Engineer
Storage Technology Service Center, Oracle Tokyo

2. Agenda
!
Ethernet
!
ARP
!
IP
!
Routing
!
IPoIB (IP over Infiniband)
!2

3. 1. ETHERNET
!3

4. Ethernet
!
We call them “frames.”
!
It can be sent only to the same hub/switch.
!4

5. How Ethernet Works?
You check baggage claim tags (MAC Address) and pick up the bag
(frame) when the tag matches.
!5

6. Simplified Frame Format
•MAC address – 6 byte address (48bits), first 3 bytes indicate vendor. (OUI)
802.3 Ethernet II Frame
Destination
Source
Ethernet II
MAC Address MAC Address Payload (up to 1500 bytes)
802.1Q Frame (Tagged VLAN)
Destination
Source
VLAN ID Ethernet II
MAC Address MAC Address (4 bytes) Payload (up to 1500 bytes)
•Switch port settings have to match with what frame format to manage.
•The destination MAC address of all 1 (FF: FF: FF: FF: FF: FF) means broadcast.
•OUI search at IEEE < http://standards.ieee.org/develop/regauth/oui/public.html>
!6

7. Hubs
Frame 1:
Dst: 00:12:F0:00:00:10
Src: 00:12:F0:00:00:01
Node1 (Intel Card)
00:12:F0:00:00:01
Node2 (Intel Card)
00:12:F0:00:00:10
Node3 (Cisco Port)
00:05:9B:00:00:10
•Every node receives all frames
•Each node discards frames not addressed to them.
•If the destination is broadcast, all nodes process the frame.
!7

8. (2)
Switches
MAC table
Port #2 00:12:F0:00:00:10
Frame 1: (1)
Dst: 00:12:F0:00:00:10
Src: 00:12:F0:00:00:01
Frame 2: (3)
Dst: 00:12:F0:00:00:10
Src: 00:12:F0:00:00:01
Node1 (Intel Card)
00:12:F0:00:00:01
(1)
(3)
Node2 (Intel Card)
00:12:F0:00:00:10
(1)
Node3 (Cisco Port)
00:05:9B:00:00:10
•Every node receives the frames if the destination is not on switch’s mac table
•No other node receives frames if not addressed to them after the first frame.
•Frames with broadcast is sent to all ports.
!8

9. VLAN – Logical Partitioning of Switch
Frame 1: (VLAN2)
Dst: 00:12:F0:00:00:10
Src: 00:12:F0:00:00:01
VLAN 2
VLAN 3
Frame 2: (VLAN2)
Dst: 00:05:9B:00:00:10
Src: 00:12:F0:00:00:01
Node1 (Intel Card)
00:12:F0:00:00:01
Node2 (Intel Card)
00:12:F0:00:00:10
Node3 (Cisco Port)
00:05:9B:00:00:10
•Different VLAN ID means different logical switch.
•No way to go beyond the VLAN boundary unless routed.
•Therefore frame is invalid for transmission.
!9

10. Checking Mac Address
-bash-4.1$ more ifconfig-a.out
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
igb0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 5
inet 172.23.210.177 netmask ffffff80 broadcast 172.23.210.255
ether 0:10:e0:35:e0:5c
igb1: flags=1000842<BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4
inet 172.23.210.179 netmask ffffff80 broadcast 172.23.210.255
ether 0:10:e0:35:e0:5d
ipmp1: flags=8001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,IPMP> mtu 65520 index 6
inet 192.168.28.1 netmask fffffc00 broadcast 192.168.31.255
groupname ipmp1
lo0: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252 index 1
inet6 ::1/128
!10

11. 2. ARP
!11

12. ARP – Find MAC address from IP Address
Bean
!12

13. ARP – Find MAC address from IP Address
•Destination Mac Address is FF:FF:FF:FF:FF:FF
•Asks “Who is IP of aa.bb.cc.dd”
ARP Request Packet (Simplified)
Ethernet Header)
Destination
Source
Frame
MAC Address MAC Address Type
(All 1)
…
ARP Request
Source
Source
Destination
Destination
MAC Address IP Address MAC Address IP Address
(All 1)
!13

14. ARP
ARP Request
Who is 192.168.100.1?
Node1 (Intel Card)
00:12:F0:00:00:01
192.168.100.100
ARP Reply
00:05:9B:00:00:10
Node2 (Intel Card)
00:12:F0:00:00:10
192.168.100.101
Node3 (Cisco Port)
00:05:9B:00:00:10
192.168.100.1
•Node 1 “broadcasts” an ARP request asking 192.168.100.1
•Because the frames are addressed to every node, every node receives the frame.
•Node 3 with the target IP address sends back the ARP reply to node 1.
!14

15. Checking ARP Table
•No data in the supportbundle
-bash-4.1$ arp -a"
Net to Media Table: IPv4"
Device IP Address
Mask
Flags
Phys Addr"
------ -------------------- --------------- -------- ---------------"
net8 169.254.182.77
255.255.255.255 SPLA 02:21:28:57:47:17"
net8 169.254.182.76
255.255.255.255
02:21:28:57:47:16"
(….omitted….)
ipmp_vfdom0 amogg0005.us.oracle.com 255.255.255.255
00:10:e0:23:9c:84"
ipmp_vfdom0 amogg0006.us.oracle.com 255.255.255.255
00:10:e0:0e:09:1c
!15

16. 3. IP
!16

17. IP – Travelling to Final Destination
Germany
Office
Frankfurt
Direct flight
Indirect flight
Singapore
(Changi)
Indirect flight
Tokyo
(Narita)
My
House
!17

18. IPv4 Addressing
•IP Version 4 address is 4 byte address, delimited by dots e.g. 192.168.100.1.
•Subnet mask decides what part of IPv4 address to look at to identify if the IP
addresses belong to the same subnet.
•The network address is the beginning of the subnet.
•Subnet masks can be described in AA.BB.CC.DD or CIDR notation (/nn
format , where nn is the number of bits to mask from MSB.)
Examples:
(The below mean the same thing.)
"
IP: 192.168.0.1 Subnet mask: 255.255.255.0
IP: 192.168.0.1/24
!18

19. Do They Belong To The Same Subnet?
Q1:
(a) 192.168.0.1/24
(b) 192.168.1.1/24
Q2:
(a) 192.168.0.1/16
(b) 192.168.1.1/16
To identify the network address, we calculate “AND”
of IP address and subnet mask. /24 means FF:FF:FF:00.
The network addresses for each (a) and (b) are:
(a) 192.168.0.0
(b) 192.168.1.0
Therefore, they DO NOT belong to the same subnet.
To identify the network address, we calculate “AND”
of IP address and subnet mask. /24 means FF:FF:00:00.
The network addresses for each (a) and (b) are:
(a) 192.168.0.0
(b) 192.168.0.0
Therefore, they DO belong to the same subnet.
!19

20. How Is An IP Packet Delivered?
(1) ARP Request
Who is 192.168.100.1?
Node1 (Intel Card)
00:12:F0:00:00:01
192.168.100.100
(2) ARP Reply
00:05:9B:00:00:10
Node2 (Intel Card)
00:12:F0:00:00:10
192.168.100.101
Node3 (Cisco Port)
00:05:9B:00:00:10
192.168.100.1
(3) IP packet to 192.168.100.1
MAC address 00:05:9B:00:00:10
!20

21. 4. ROUTING
!21

22. !22

23. IP – Travelling to Final Destination
Germany
Office
Frankfurt
Direct flight
Indirect flight
Singapore
(Changi)
Indirect flight
Tokyo
(Narita)
My
House
!23

24. IP – Travelling to Final Destination
192.168.100.1
R3
Route 1
(192.168.10.0/24)
192.168.11.0/24
R2
R1
192.168.12.0/24
192.168.0.1
!24

25. Checking Routing Table
"
RRoouuttiinngg
TTaabbllee::
IIPPvv44
DDeessttiinnaattiioonn
GGaatteewwaayy
FFllaaggss
RReeff
UUssee
IInntteerrffaaccee
----------------------------------------
----------------------------------------
----------
----------
--------------------
------------------
ddeeffaauulltt
117722..2233..221100..225544
UUGG
55
88552244
iiggbb00
112277..00..00..11
112277..00..00..11
UUHH
66
118811334433
lloo00
117722..2233..221100..112288
117722..2233..221100..117777
UU
1100
664477771144
iiggbb00
119922..116688..2288..00
119922..116688..2288..11
UU
3322
1166118888664455
iippmmpp11
"
RRoouuttiinngg
TTaabbllee::
IIPPvv66
DDeessttiinnaattiioonn//MMaasskk
GGaatteewwaayy
FFllaaggss
RReeff
UUssee
IIff
------------------------------------------------------
------------------------------------------------------
----------
------
--------------
----------
::::11
::::11
UUHH
22
220000
lloo00
•The default gateway is used when no other routes are available.
•You have to look at the subnet mask as well. (Example in next page.)
!25

26. Routing Tables with Subnet Masks
"
RRoouuttiinngg
TTaabbllee::
IIPPvv44
DDeessttiinnaattiioonn
GGaatteewwaayy
SSuubbnneettmmaasskk
----------------------------------------
----------------------------------------
----------------------------
119922..116688..110000..11
119922..116688..110000..11
225555..225555..225555..00
119922..116688..110000..112299
119922..116688..110000..113300
225555..225555..225555..112288
•Routes with more bits in subnet mask have priority.
•So if you would like to connect to 192.168.100.131, you take
192.168.100.130 because subnet mask is 25 bits.
!26

27. How Packet is Delivered Over a Router/L3 SW
(1) ARP Request
Who is 192.168.200.1?
(2) ARP Reply
MAC: 00:05:9B:00:00:10
R1 Eth0
00:05:9B:00:00:10
192.168.100.254
Node1
00:12:F0:00:00:01
192.168.100.100
Node2
00:12:F0:00:00:10
192.168.100.101
(3) Packet to 192.168.200.1
MAC :00:05:9B:00:00:10
(4) Packet to 192.168.200.1
MAC :00:12:F0:00:00:10
R1
R1 Eth1
00:05:9B:00:00:11
192.168.200.254
Node3
00:12:F0:00:00:10
192.168.200.1
!27

28. 5. IPOIB
!28

29. IB Partitioning
PKEY:FFFF
Subnet
Manager
(Internal/External)
PKEY:8506
Node1 (PKEY: 8506)
10E00001328389
PKEY:8507
Node2 (PKEY:8506)
10E0000132838A
Node3 (PKEY: 8507)
10E0000132848C
•IB fabric is divided into “partitions” like VLANs
•We use PKEY instead of VLAN ID in IB partitioning.
•PKEY: FFFF means “connect to any port on fabric.”
•In the above example, Node 1 and 2 can talk, but not with node 3.
•Oracle supports only “on-switch” subnet manager with ZFSSA.
!29

30. Checking IB Partitioning
"
--bbaasshh--44..11$$
mmoorree
ddllaaddmm--sshhooww--iibb..oouutt
LLIINNKK
HHCCAAGGUUIIDD
PPOORRTTGGUUIIDD
PPOORRTT
SSTTAATTEE
PPKKEEYYSS
iibbpp00
1100EE0000000011332288338888
1100EE0000000011332288338899
11
uupp
FFFFFFFF,,88550033
iibbpp11
1100EE0000000011332288338888
1100EE00000000113322883388AA
22
uupp
FFFFFFFF,,88550033
"
--bbaasshh--44..11$$
mmoorree
ddllaaddmm--sshhooww--ppaarrtt..oouutt
LLIINNKK
PPKKEEYY
OOVVEERR
SSTTAATTEE
FFLLAAGGSS
pp88550033__iibbpp00
88550033
iibbpp00
uupp
ff------
pp88550033__iibbpp11
88550033
iibbpp11
uupp
ff------
"
--bbaasshh--44..11$$
mmoorree
iiffccoonnffiigg--aa..oouutt
lloo00::
ffllaaggss==22000011000000884499<<UUPP,,LLOOOOPPBBAACCKK,,RRUUNNNNIINNGG,,MMUULLTTIICCAASSTT,,IIPPvv44,,VVIIRRTTUUAALL>>
mmttuu
88223322
iinnddeexx
11
iinneett
112277..00..00..11
nneettmmaasskk
ffff000000000000
pp88550033__iibbpp00::
ffllaaggss==99004400884433<<UUPP,,BBRROOAADDCCAASSTT,,RRUUNNNNIINNGG,,MMUULLTTIICCAASSTT,,DDEEPPRREECCAATTEEDD,,IIPPvv44,,…>>
mmttuu
6655552200
iinnddeexx
22
iinneett
00..00..00..00
nneettmmaasskk
ffff000000000000
bbrrooaaddccaasstt
00..225555..225555..225555
ggrroouuppnnaammee
iippmmpp11
iippiibb
8800::00::00::44aa::ffee::8800::00::00::00::00::00::00::00::1100::ee00::00::11::3322::8833::8899
pp88550033__iibbpp11::
ffllaaggss==6699004400884433<<UUPP,,BBRROOAADDCCAASSTT,,RRUUNNNNIINNGG,,MMUULLTTIICCAASSTT,,DDEEPPRREECCAATTEEDD,,IIPPvv44,,…>>
mmttuu
6655552200
iinnddeexx
33
iinneett
00..00..00..00
nneettmmaasskk
ffff000000000000
bbrrooaaddccaasstt
00..225555..225555..225555
ggrroouuppnnaammee
iippmmpp11
iippiibb
8800::00::00::44bb::ffee::8800::00::00::00::00::00::00::00::1100::ee00::00::11::3322::8833::88aa
!30

31. References
•Ethernet Frame
< http://en.wikipedia.org/wiki/Ethernet_frame>
•802.1Q
< http://en.wikipedia.org/wiki/802.1Q>
•ARP(Address Resolution Protocol)
< http://en.wikipedia.org/wiki/Address_Resolution_Protocol>
•Subnetwork
<http://en.wikipedia.org/wiki/Subnetwork>
•
!31