Storage networking-technologies

EDUCATION
Networking Technologies: Concepts in Internal and External Networked Storage©
Copyright 2007 Howard Goldstein Associates, Inc. HGAI All Rights Reserved.
Networking Technologies:
Concepts in Internal and
External Networked Storage
Howard Goldstein
Howard Goldstein Associates, Inc.
Howard.Goldstein@HGAI.Com

2
EDUCATION
Storage Networking Concepts
• Objectives:
– Identify what networking is and how it relates to
storage
– Examine storage networking technology similarities
and differences
– Clarify and explain the language of networking:
Terminology
– Uniquely inform and hopefully entertain
Networking
Technologies
Networking Technologies: Concepts in Internal and External Networked Storage
© Copyright 2007 Howard Goldstein Associates, Inc. HGAI All Rights Reserved.

3
EDUCATION
Outline
Networking
Technologies

4
EDUCATION
Outline
• Network vs. Storage
• ANs and ASes
• Perspectives
• Virtual SCSI Cables
• Port vs. PHY
• Switch vs. Router vs.
Gateway
• Circuit Switch vs. Packet
Switch
• Connection-Oriented vs.
Connectionless
• Layered Protocols
• Flow Control
• Payload Integrity
• Names vs. Addresses
• Segmentation,
Convergence,
Reassembly & Offload
• Bit Rate, Bandwidth,
Throughput
• Semantic Analysm
• Future & Conclusions

5
EDUCATION
Network Vs. Storage
Network Considerations Storage Considerations
Information movement Information repository
Data over Distance Data over Time
Configuring and segmenting
network topologies
Formatting and partitioning hard
disks, Creating logical volumes
Hubs, Bridges, Switches, Routers,
Gateways
JBOD, RAID, Disk Controllers, Tape
Controllers
Client / Server Applications Initiator / Target Functions
Email, Web Browsing, File Mgt,
Network Interface
Backup, Archive, Mirror, Block Mgt,
I/O Interfaces

6
EDUCATION
Network Vs. Storage
Network Considerations Storage Considerations
Speed, Solid-State Capacity, Mechanical Movement
Media – Copper, Fiber, Wireless Media – Disk, Tape, Paper
Bits in Frames, CRC error checks
every frame
Bytes in Blocks, Parity check every
byte
Buffers and Link transfer rates –
Kbps, Mbps, Gbps
Cache and Disk/Tape I/O interface
transfer rates - MBps
Interconnectivity network and
device latencies - bottlenecks
Disk/Tape seek times and latencies
– ms, us, ns
Applications – HTTP, NFS, SMTP
Transport Protocols –
TCP/IP/Ethernet
Logical Device Protocols – SCSI,
ESCON, IDE/ATA
Physical Protocols – Parallel SCSI,
Fibre Channel,
iSCSI/TCP/IP/Ethernet

7
EDUCATION
Storage Data Units of Measure
Unit of
Measure
Abbreviation Approximate
Size
Byte Byte 8 bits
Block Blk 512 Bytes
Kilobyte KB 1,024 Bytes
Megabyte MB 1 Million
Bytes
Gigabyte GB 1,000 MB
Terabyte TB 1,000 GB
Petabyte PB 1,000 TB
Exabyte EB 1 Million TB
Zetabyte ZB 1,000 EB
Yottabyte YB 1,000 ZB
How long would it
take someone to
download a one-
yottabyte file using
a 56 Kbps modem?
5.5 trillion years.
ReallyReally BigBig NumbersNumbers

8
EDUCATION
Network Data Units of Measure
11
Unit of
Measure
Power
of 10
Number
milli - 3 One thousandth
micro - 6 One millionth
nano - 9 One billionth
pico - 12 One trillionth
femto - 15 One quadrillionth
atto - 18 One quintillionth
zepto - 21 One sextillionth
yocto - 24 One septillionth
ReallyReally SmallSmall NumbersNumbers

9
EDUCATION
Basic Storage Network Technology
Direct-Attached
Storage (DAS)
Application
File System
Storage

10
EDUCATION
Direct-Attached
Storage (DAS)
Application
File System
Storage
Direct-Attached
Storage (DAS)
Application
File System
Storage
Direct-Attached
Storage (DAS)
Application
File System
Storage

11
EDUCATION
Network-Attached
Storage (NAS)
Application
File System
Storage
Network
Application
File System
Storage
Direct-Attached
Storage (DAS)

12
EDUCATION
Network-Attached
Storage (NAS)
Application
File System
Storage
Application Application
Network

13
EDUCATION
Network-Attached
Storage (NAS)
Storage Area
Network (SAN)
Application
File System
Storage
Network
Application
File System
Storage
Direct-Attached
Storage (DAS)
Application
File SystemNetwork
Storage

14
EDUCATION
Storage Area
Network (SAN)
Application
File System
Storage
Network
Storage Storage
Application
File System
Application
File System

15
EDUCATION
Network-Attached
Storage (NAS)
Storage Area
Network (SAN)
Application
File System
Storage
Network
Application
File System
Storage
Direct-Attached
Storage (DAS)
Application
File System
Network
Storage

16
EDUCATION
Network-Attached
Storage (NAS)
Storage Area
Network (SAN)
Application
Storage
Application
File System
Network
Storage
Network
File System

17
EDUCATION
Network-Attached
Storage (NAS)
Storage Area
Network (SAN)
Application
File System
Storage
Network
Application
File SystemNetwork
Storage
NAS & SAN

18
EDUCATION
Application
File System
Storage
File Request File RequestFile
Block Request Block Request
Blocks
Blocks
File
DAS

19
EDUCATION
Network Attached Storage
NAS
•Messages
•Files
NFS/CIFS
Client
NFS/CIFS Server
LAN NAS
File
File Request
Blocks

20
EDUCATION
Storage Area Network
SERVER
NAS
Gateway
SAN
Storage
• Commands
• Blocks
• Control
• Status

21
EDUCATION
Storage Area Network – “External SAN”
SERVER
SAN
SCSI Initiator
Client
SCSI Target & LUN
Server
Read
Command
Request
Blocks
Status
“External SAN”
File
File Request
Application

22
EDUCATION
Network
Network
Network
LAN MAN WAN
Redundant Array of Independent Disks
Disk
Drives
Controller
A
Controller
B
Network“SmAN”

23
EDUCATION
LAN/SAN Benefits
• Greater Asset
Utilization
• Expanded connectivity
• Port consolidation
• Extended distance
• Higher bandwidth
• Potential performance
improvement
LAN
SAN
FC IP

24
EDUCATION
LAN & SAN Information Flow Requirements
LAN SAN
Ethernet
IP
TCP, UDP
iSCSI, HTTP,
SMTP, NFS,
FTP, DNS

25
EDUCATION
“External SAN” I/O Request to Storage Controller
Controller
Controller
SAN
External Read
Command
Request
Disk
Disk
Disk
Disk
Disk
Disk
External SCSI
Target & LUN
Blocks Blocks
Status

26
EDUCATION
“Internal SAN” I/O Requests in Storage Controller
Controller 2
Controller 1
Controller 1
Internal SCSI
Initiator Client
Internal SCSI
Target & LUN (Disk
Drive)
Controller 1
Internal Read
Command
Request
Blocks
Status
“Internal SAN”
RAID
Controller 1
Internal Read
Command
Request
Internal SCSI
Target & LUN
(Disk Drive)
External Read
Command
Request
Disk
Disk
Disk
Disk
Disk
Disk
Status
Blocks
External SCSI
Target & LUN
Blocks Blocks
Status
FC
Loop
Switch
SAN

27
EDUCATION
Controller 2
Controller 1
Internal SCSI
Initiator Client
Internal SCSI
Target & LUN (Disk
Drive)
Controller 1
Internal Read
Command
Request
Blocks
Status
“Internal SAN”
RAID
Controller 1
Internal Read
Command
Request
Internal SCSI
Target & LUN
(Disk Drive)
External Read
Command
Request
Disk
Disk
Disk
Disk
Disk
Disk
Status
Blocks
External SCSI
Target & LUN
Blocks Blocks
Status
FC
Loop
Switch

28
EDUCATION
Controller
Controller
Internal SCSI
Initiator Client
Internal SCSI
Target & LUN (Disk
Drive)
Internal Read
Command
Request
Blocks
Status
“Internal SAN”
RAID
Internal Read
Command
Request
Internal SCSI
Target & LUN
(Disk Drive)
External Read
Command
Request
Disk
Disk
Disk
Disk
Disk
Disk
Status
Blocks
External SCSI
Target & LUN
Blocks Blocks
Status
SAS
Expander

29
EDUCATION
Virtual SCSI Cables on SCSI Bus
Target
Logical Units
0 1 2 62 63
Peripheral
Controller
SCSI Bus
Logical Units
0 1 2 62 63
Peripheral
Controller
Host
Adapter
Host
Computer
Initiator
Host
Adapter
Host
Computer
Initiator
Host
Adapter
Host
Computer
Adapter
Adapter
Target
Initiator
P-SCSI

30
EDUCATION
Virtual SCSI Cables on SCSI Bus
Target
Logical Units
0 1 2 62 63
Peripheral
Controller
SCSI Bus
Logical Units
0 1 2 62 63
Peripheral
Controller
Host
Adapter
Host
Computer
Initiator
Host
Adapter
Host
Computer
Initiator
Host
Adapter
Host
Computer
Adapter
Adapter
Target
Initiator
SCSI Bus: Arbitrate ID, Select ID, Identify LUN
Blocks

31
EDUCATION
Virtual SCSI Cables Fibre Channel
FC: Fabric Login, N_Port Login, FC-4 Login
FC SAN
Host
Adapter
Host
Computer
Initiator
Host
Adapter
Host
Computer
Initiator
Host
Adapter
Host
Computer
Initiator
Logical Units
0 1 2 62 63
Peripheral
Controller
Adapter
Target
Logical Units
0 1 2 62 63
Peripheral
ControllerAdapter
Target

32
EDUCATION
Logical Units
0 1 2 62 63
Peripheral
Controller
Adapter
Target
Logical Units
0 1 2 62 63
Peripheral
ControllerAdapter
Target
Host
Adapter
Host
Computer
Initiator
Host
Adapter
Host
Computer
Initiator
Host
Adapter
Host
Computer
Initiator
FC Switch
FC SAN

33
EDUCATION
Logical Units
0 1 2 62 63
Peripheral
Controller
Adapter
Target
Logical Units
0 1 2 62 63
Peripheral
ControllerAdapter
Target
Host
Adapter
Host
Computer
Initiator
Host
Adapter
Host
Computer
Initiator
Host
Adapter
Host
Computer
Initiator
FC Switch
FC Point-to-Point:
Direct Routing
FC Loop: Arbitrate,
Open Loop Circuit
FC Fabric: Switch
Routing
Blocks

34
EDUCATION
Virtual SCSI Cables SAS
Target
Logical Units
0 1 2 62 63
Peripheral
Controller
Host
Adapter
Host
Computer
Initiator
Host
Adapter
Host
Computer
Initiator
Host
Adapter
Host
Computer
Initiator
Adapter
Target
Logical Units
0 1 2 62 63
Peripheral
Controller
Adapter
SAS

35
EDUCATION
Virtual SCSI Cables SAS
Target
Logical Units
0 1 2 62 63
Peripheral
Controller
Host
Adapter
Host
Computer
Initiator
Host
Adapter
Host
Computer
Initiator
Host
Adapter
Host
Computer
Initiator
Adapter
SAS Expander
Target
Logical Units
0 1 2 62 63
Peripheral
Controller
Adapter
SAS
Serial Attached SCSI: Open, Information, Close
Blocks

36
EDUCATION
Virtual SCSI Cables IP Storage (iSCSI)
Target
Logical Units
0 1 2 62 63
Peripheral
Controller
Host
Adapter
Host
Computer
Initiator
Host
Adapter
Host
Computer
Initiator
Host
Adapter
Host
Computer
Initiator
Adapter
Target
Logical Units
0 1 2 62 63
Peripheral
Controller
Adapter
IP Storage

37
EDUCATION
Virtual SCSI Cables IP Storage (iSCSI)
Target
Logical Units
0 1 2 62 63
Peripheral
Controller
Host
Adapter
Host
Computer
Initiator
Host
Adapter
Host
Computer
Initiator
Host
Adapter
Host
Computer
Initiator
Adapter
Ethernet Switch
Target
Logical Units
0 1 2 62 63
Peripheral
Controller
Adapter
Blocks

38
EDUCATION
M C Escher – Ascending & Descending Monks

39
EDUCATION
SAS Storage Controller Perspective
Host
HBA
Disks Root
Edge
Expanders
Fanout
Expander
SAS “Expanse”
Service Delivery Subsytem

40
EDUCATION
SAS Storage Network Perspective
HBA Disk
Edge
Expanders
Fanout
Expander
Root
SAS “Expanse”
Service Delivery Subsytem
End Devices Expanders

41
EDUCATION
SAS Storage Network: Host – Disk Perspective
HBAs Disks
Edge
Expanders
Fanout
Expander
Roots

42
EDUCATION
SAS Storage Network: Host – Disk Perspective

43
EDUCATION
Parts of a Physical Transport Network?
1. 2. 3.
Media or the
“appearance of”
media
Ports - 2 or more
(transceivers)
Protocol
?

44
EDUCATION
Port vs. PHY = Ports vs. Docks
Fibre Channel, Ethernet
1 Port – 1 PHY
SAS, Infiniband
1 Port – Many PHYs

45
EDUCATION
Striping the Transmission Word
1 Link - 4 Phy Lanes
0 1 2 30 1 2 3
0 1 2 3
PCIe, InfiniBand does Striping (4x, 12x)
SAS is not Striping – See connection path

46
EDUCATION
Striping the Transmission Word
1 Link - 4 Phy Lanes
0 1 2 30 1 2 3
0 1 2 3
10 Gbps Fibre Channel, PCIe, InfiniBand does Striping
SAS is not Striping – See connection path

47
EDUCATION
SAS Expander – Open, Arbitration, Open
Accept
Node Node
Node Node
Expander
ARB 02, 03
Frame Primitive
ID
03
ID
04
ID
01
ID
02OPEN Accept
OPEN 02, 03
AIP
AIP Done
OPEN 02, 03

48
EDUCATION
SAS Expander – Move
Node Node
Node Node
Expander
Frame Primitive
ID
03
ID
04
ID
01
ID
02

49
EDUCATION
SAS Expander – Move
Node Node
Node Node
ID
03
ID
04
ID
01
ID
02
Expander
Frame Primitive

50
EDUCATION
What is a Physical Transport Network?
• A group of connected systems - at least two!
– Host, Node, Computer, Server, Workstation, Printer, Router, Storage Device
Network Interface Cards (NICs), Host Bus Adapters
(HBAs), Storage Adapters (SAs)
• A media or the “appearance of” media
– Unshielded Twisted Pair(s) (UTP), Coaxial Cable(s), Multi-Mode Fiber(s),
Leased Line(s), RF
• A connection protocol or method
– Ethernet, Token Ring, PPP, Frame Relay, ATM, Fibre Channel, SONET/SDH,
DWDM
• Switches create an appearance of media
Router
File
Server
Switch

51
EDUCATION
Routing in Switches
• Routing
• Switch frame “steering” over one
“appearance of media”

52
EDUCATION
Routing in Routers
• Routing
• Router packet “forwarding”
across many “appearances of
media”
– A Day in the Life at Manhattan starts at
home:
– Walk to the bus stop
– Get on the bus
– Take the bus to the dock
– Get off the bus and walk on to the Ferry boat
– Sail across the river
– Get off the boat and hop in a cab.
– Drive to the office and get out of the cab.
– Run to the elevator and hop in
– Jump off the elevator and walk to your desk
– How many subnets? 9

53
EDUCATION
An Internetwork: Switch vs. Router
• Routers join appearances of media
File
Server
Router
Switch
File
Server
Router
Switch
File
Server
Switch
File
Server
Switch

54
EDUCATION
Router vs. Gateway
File
Server
Router
Switch
File
Server
Router
Switch
File
Server
Switch
File
Server
Switch
Gateway

55
EDUCATION
Circuit Switch – Packet Switch
• Fixed path whether moving or not versus routing
frames or packets on the current best path as they
arrive and leave

56
EDUCATION
Circuit Switch
• SAS Expanders Frame
• FC Class 1 Frame
• POTS (Plain Old
Telephone Network)
• Circuit Construction

57
EDUCATION
Packet (Frame) Switch
• FC Class 3
Frame (switch)
• IP Packet (router)
• VOIP
• Frame Direction

58
EDUCATION
Connection-Oriented vs. Connectionless
• Not path related!
• Connection-oriented requires a state exchange
to track communications
• Connectionless takes things as they come
Telephone Telegraph

59
EDUCATION
• Internet Protocol Suite
– TCP connections start with negotiation setup between hosts
• Similar to telephone connection
– Call, Answer, Talk, Hang up
• UDP connectionless
– IP Connectionless – Handles packets as they come
• Fibre Channel Class 2, 3 Switch Routing
– Connectionless – Handles frames as they come
TCP
TCP
IP
IP
IP
IP
Connection-Oriented vs. Connectionless

60
EDUCATION
Connection-Oriented FC Sessions - Login
Fabric
N-Port
Login
FC-4
Process
Login
Fabric
Login
Fabric
Login
Link
Active
Link
Active
Link
Active
Server Storage

61
EDUCATION
Layered Protocols
• Protocol Data Units (PDUs)
• They are both cake – Different ways to
provide similar function!

62
EDUCATION
OSI & IPS (TCP/IP)
Transport
Session
Presentation
Application
Physical
Data Link
Network
End User Data
Transport
Application
Services
Network
Interface
Sublayer
Internet
End User Data
Open Systems Interconnection Internet Protocol Suite

63
EDUCATION
OSI & Fibre Channel Architecture
Transport
Session
Presentation
Application
Physical
Data Link
Network
End User Data
FC0 - Physical
FC2 - Framing/Flow Control
FC1 - Encode/Decode
FC3 - Common Services
FC4 - Upper Layer Mapping
Channels
IPI, SCSI,
HIPPI, SBCCS
Networks
802.2 (LE),
IP, ATM
End User Data

64
EDUCATION
OSI & Fibre Channel Architecture
Transport
Session
Presentation
Application
Physical
Data Link
Network
End User Data
FC0 - Physical
FC1 - Encode/Decode
Channels
IPI, SCSI,
HIPPI, SBCCS
Networks
802.2 (LE),
IP, ATM
End User Data

65
EDUCATION
Fibre Channel & IPS
Internet Protocol SuiteFibre Channel
Transport
Application
Services
Network
Interface
Sublayer
Internet
FC0 - Physical
FC1 - Encode/Decode
Channels
IPI, SCSI,
HIPPI, SBCCS
Networks
802.2 (LE),
IP, ATM

66
EDUCATION
Flow Control
• Fibre Channel
– Credit-Based Fibre Channel
– Advertise Credit
• Internet Protocol Suite
– TCP Sliding Window
– IP Datagram Drop Congestion Control
– Advertise Larger Windows

67
EDUCATION
SAS Flow Control-
Expanders – Circuit Switches
HBA Switch SASwitch
Frame = ~1K Max
Credit R-RDY = 1 Dword

68
EDUCATION
Fibre Channel Buffer-to-Buffer (Class 2, 3)
N-Port E-Port
HBA Switch
N-Port
SA
F-Port
Switch
F-Port
Frame = ~2K Max
Credit R-RDY = 1 Word
B-B B-B B-B
E-Port

69
EDUCATION
Fibre Channel End-to-End (Class 1, 2)
Fabric
N-Port
E-Ports
N-PortF-PortF-Port
Frame
ACK Frame
Node Node NodeNode
E-E

70
EDUCATION
Internet - TCP Sliding Windows
• Sliding Windows
– Reduces overhead and delay of positive acknowledgments
– Allows sending of multiple segments before ACK is required
• Window Advertisements
– Receivers advertise window sizes
– Usually MSS multiple
– Maximum 65,535 Bytes
– Prevent “Silly window” syndrome
• Slow Start
– Congestion Window
– Increments till congestion detected

71
EDUCATION
TCP Window Scale Factor
• Multiplier for window field
• Window field only 16 bits for maximum of 65,535
bytes
• Both systems in a connection must agree to use
this
• Allows a maximum of 16 million bytes

72
EDUCATION
TCP Sliding Windows – TCP Segments
01 02 03 04 05 06 07 08 09 10
01 02 03 04 05 06 07 08 09 10
Send 6 Segments without
Acknowledgement
2 Acknowledgements
Send 2 More Segments

73
EDUCATION
TCP Sliding Windows
11 12 13 14 15 16 17 18 19 20
Increasing
11 12 13 14 15 16 17 18 19 20
Decreasing

74
EDUCATION
Payload Integrity
• Do you always want
payload integrity?
• Not necessarily with
Video and Voice
• Performance vs. Quality
tradeoffs
– A pixel in a online-rented
movie frame
– A medical imaging application
just prior to heart surgery
where you want to be sure the
speck of grey isn’t line noise!

75
EDUCATION
Names & Addresses
Name
Address
Route

76
EDUCATION
Name, Address & Route
• Addressing and Identification schemes
• Address is used to locate
• Names or identifiers uniquely represent
something or someone
Name
Address

77
EDUCATION
Name, Address & Route
• Name is a specific identification
– Machine, User, Application - Unique target for data
• Internet
– Fully Qualified Domain Names WWW.SNIA.ORG
– Media Access Control MAC ID – Physical Port Identifier
• Storage
– World Wide Node Names WWNN (Fibre Channel)
– World Wide Port Names WWPN (Fibre Channel)
– Enterprise Unique Identifier EUI (iSCSI)
– iSCSI Qualified Name IQN (iSCSI)
• Address identifies where the target is located
– Hardware, Network, Port, Socket
– Port ID FCID (Fibre Channel)
– IP Address
• Route identifies path to get to the destination

78
EDUCATION
Transport Addressing Ports
• Ports are used to deliver information to relevant
application services
• Well-known and “Not-so-well-known” Transient
• Comparable to Well known FCID Port ID Addresses
– FFFFFC Name Server
– FFFFFD Fabric Controller
– FFFFFE Fabric Port Login Server
Port Name Description
20 FTP-3DATA File Transfer (Data Channel)
21 FTP File Transfer (Control Channel)
22 TELNET Telnet
25 SMTP Simple Mail Transfer
53 DOMAIN Domain Name ServiceDNS Domain Name Services
3260 ISCSI Internet SCSI

79
EDUCATION
Comparative Address Ranges
• 1 Address = 1 Gram
• SATA Point-to-Point
– 20 = 1
• SATA II “Port Multiplier”
– 24 = 16
• SAS “Expanders”
– 214 = 128 x 128 = 16,384
• All Fibre Channel
– 224 = 16,777,216
• All IPv4
– 232 = 4,294,967,296
• All IPv6
– 2128 = 340,282,366,920,938,463,463,374,607,431,768,211,456
X 56
Billion

80
EDUCATION
Hierarchical, Routable Addressing
• IP addresses can be divided into three parts
– Network Address (NETID)
– Subnetwork Address using (Subnetwork Mask 255.255.255.0)
– Host IP Address
• Example 172.16.2.6
– Network 172.16.0.0, Subnetwork 172.16.2.0, Host IP Address
172.16.2.6
• FC addresses can be divided into three parts
– Domain (Domain ID)
– Area (Area ID)
– Device (Device ID)
• Example 0102EF
– Domain 01 (Switch 01), Area 02 (Port 02 on the Switch), Device EF
(ALPA of Loop Device)

81
EDUCATION
Address Assignment
• DHCP Mechanism for allocating and assigning IP
addresses to clients
– Automatic: Host requesting an address are provided with a
permanent IP address
– Dynamic: Hosts requesting an address are provided with a
temporary address
– Manual: Host IP addresses are manually configured and DHCP
just delivers these assignments
• Fibre Channel
– Loop Initialization LISM, LIFA, LIPA, LIHA, LISA, LIRP, LILP,
Close
• Select Master, Fabric Assigned, Previously
Assigned, Hard Assigned, Soft Assigned, Report
Position, List Position, Close
– Fabric Login Controller
– Point-to-Point - Lowest World Wide Port Name (WWPN) Assigns

82
EDUCATION
Name Service
• Internet
– Domain Name Services for IP
• Fully Qualified Domain Name FQDN to IP Address
• Running on distributed servers
• Storage
– Name Services for Fibre Channel
• WWPN to Fibre Channel Identifier FCID, Fabric
service running in the Switch
– iSNS Storage Name Service (iSNS) for iSCSI

83
EDUCATION
Segmentation & Offload
• Segmentation, Convergence and
Reassembly
• Offload for Performance Acceleration

84
EDUCATION
Traditional FC SAN
Host
Applications
Host
Applications
File ManagerFile Manager
Device DriverDevice Driver
SCSI ProtocolSCSI Protocol
Operating System
I/O Interface
HBA
FC2
FC1
FC3
FC4
FC0
Transport
Application
Services
Internet
Internet
Protocol Suite
Host NIC
Network
Interface
Sublayer
Ethernet
FC2

85
EDUCATION
Read 8K (6 FC frames with 4 - 2K Frames)
Host
HBA
FC2
FC1
FC3
FC4
FC0
FC2
Fibre Channel Multi-Frame Sequence Minimizes Server I/O interrupts - 2
Read Command
Data
Data
Data
Data
Status

86
EDUCATION
Traditional NAS
Host
Applications
Host
Applications
Operating System
I/O Interface
HBA
FC2
FC1
FC3
FC4
FC0
Transport
Application
Services
Internet
Internet
Protocol Suite
Host NIC
Network
Interface
Sublayer
Ethernet
NFS/CIFS
FC2

87
EDUCATION
Get 8K (7 Ethernet Frames with 6 - 1.5K Frames)
Host
Host IP Fragmentation - 6 IP Packet Fragments, 6 – 1500 byte
MTU Ethernet Frames, Maximizes Server I/O interrupts - 7
NIC
Network
Interface
Sublayer
Ethernet
Transport
Application
Services
Internet
Get File
Data
Data
Data
Data
Data
Data
NFS / CIFS / iSCSI

88
EDUCATION
iSCSI Full Offload
Host
Applications
Host
Applications
Operating System
I/O Interface
Transport
Application
Services
Internet
Internet
Protocol Suite
Host Intelligent
NIC/HBA
Network
Interface
Sublayer
Gigabit
Ethernet
i
S
C
S
I
TCP
IP
iSCSI

89
EDUCATION
Read 8K (8 Ethernet frames with 6 – 1.5K)
Host Offload IP Fragmentation
(Like FC Multi-Frame Sequence)
Minimizes Server I/O interrupts - 2
Read Command
Data
Data
Data
Status
Intelligent
NIC/HBA
Network
Interface
Sublayer
Gigabit
Ethernet
TCP
IP
iSCSI
Data
Data
Data

90
EDUCATION
Serial Scaling

91
EDUCATION
Serial Scaling

92
EDUCATION
Serial Link Bit Rate Scaling
40 Bits = Tword
1 Bit
1 Bit @ 1.5 Gbps = .666ns
Transmit Receive
1 Bit @ 3.0 Gbps = .333 ns
1 Bit
40 Bits = Tword
Assume
40 Folds

93
EDUCATION
Bandwidth

94
EDUCATION
iSCSI Overhead
iSCSI
Standard
Ethernet
Ethernet with Non-
Standard Jumbo
Frames
iSCSI/TCP/IP/ Standard
Ethernet Overhead
iSCSI/ TCP/IP/ Ethernet
Overhead with Non-
Standard Ethernet Jumbo
Frames
iSCSI Data Out or Data in PDU
Header Bytes 52 52
TCP Header Bytes 20 20
IPv4 Header Bytes 20 20
Preamble Bytes 7 7 7 7
SFD Bytes 1 1 1 1
Destination Address Bytes 6 6 6 6
Source Address Bytes 6 6 6 6
Length/Type Bytes 2 2 2 2
Maximum Data/Pad Bytes 1,500 9,000 1,500 9,000
FCS Bytes 4 4 4 4
Ethernet Frame overhead
including Preamble and SFD
fields Bytes 26 26 26 26
Total Overhead Bytes 52 52 144 144

95
EDUCATION
iSCSI Performance
iSCSI
Standard
Ethernet
Ethernet with Non-
Standard Jumbo
Frames
iSCSI/TCP/IP/ Standard
Ethernet Overhead
iSCSI/ TCP/IP/ Ethernet
Overhead with Non-
Standard Ethernet
Jumbo Frames
Total Overhead Bytes 52 52 144 144
Payload Bytes 1,500 9,000 1,500 9,000
Efficiency % 98.30% 99.43% 91.24% 98.43%
Bit Rate gbps 1.25 1.25 1.25 1.25
Data Rate MBps 122.87 124.28 114.05 123.03

96
EDUCATION
Fibre Channel Performance
Bytes Bytes Bytes
Fibre Channel
Fibre
Channel
Class 3
SCSI FC
DATA Out or
IN / Fibre
Channel
Class 3
SCSI FC
DATA Out or
IN / Fibre
Channel Class
2
SOF Bytes 4 4 4 4 4 4 4 4 4
Frame Header Bytes 24 24 24 24 24 24 24 24 24
Maximim Payload Bytes 2,112 2,048 2,048 2,112 2,048 2,048 2,112 2,048 2,048
CRC Bytes 4 4 4 4 4 4 4 4 4
EOF Bytes 4 4 4 4 4 4 4 4 4
InterFrame Word
Minimum Bytes Bytes 24 24 24 24 24 24 24 24 24
Acknowledgement
Frame Overhead 60 60 60
Total Overhead Bytes 60 60 120 60 60 120 60 60 120
Payload Bytes 2,112 2,048 2,048 2,112 2,048 2,048 2,112 2,048 2,048
Efficiency % 97.24% 97.15% 91.92% 97.24% 97.15% 91.92% 97.24% 97.15% 91.92%
Bit Rate gbps 1.0625 1.0625 1.0625 2.125 2.125 2.125 4.25 4.25 4.25
Data Rate MBps 103.31 103.23 97.67 206.63 206.45 195.33 413.26 412.90 390.66

97
EDUCATION
SAS Performance
Bytes Bytes
Serial Attached SCSI SAS SAS
SOF Bytes 4 4
Frame Header Bytes 24 24
Maximum Payload Bytes 1,024 1,024
CRC Bytes 4 4
EOF Bytes 4 4
InterFrame Word Minimum Bytes (1 Align every
2048 dwords) Bytes 2 2
R-Rdy Frame Overhead Bytes 4 4
Acknowledgement Frame Overhead Bytes 4 4
Total Overhead Bytes 46 46
Payload Bytes 1,024 1,024
Efficiency % 95.70% 95.70%
Bit Rate gbps 1.5 3
Data Rate MBps 143.55 287.10

98
EDUCATION
Bandwidth does not equal bit rate
Bit Rate Efficiency Data Rate Full Duplex
Standard Gigabit
Ethernet
1.25 Gbps 98.3% 123 MBps 246 MBps
iSCSI / TCP / IP /
Gigabit Ethernet
Standard MTU
1508
1.25 Gbps 91.2% 114 MBps 228 MBps
iSCSI / TCP / IP /
Gigabit Ethernet
Jumbo Frames
MTU 9000
1.25 Gbps 99.43% 124.28 MBps 248.56 MBps
Standard SCSI
FCP Fibre
Channel Class 3
1.0625 Gbps 97.2% 103.5 MBps 207 MBps
SAS 1.5 Gbps 95.7% 143.55 MBps 287.10 MBps
SAN Performance ComparisonSAN Performance Comparison

99
EDUCATION
Harry Newton’s Telecom Dictionary 17th
edition
• Delay
– The wait time between two events, such as the time from when a
signal is sent to the time it is received.
– There are all sorts of reasons for delays
• Propagation Delay
• Queuing Delay
• Processing Delay
• Rotational Delay
• Satellite Delay
• Latency
– A fancy term for waiting delay. The time it takes to get
information through a network.
– Accumulated delay

100
EDUCATION
Delay & Latency
• Delay
– A metric that measures how long
something is not happening
– (A packet waiting in a queue)
• Latency
– A metric that measures how long
it is between things happening
– (The time between when a
packet arrives at a device and
leaves a device
• Throughput
– Delay and Latency affect
throughput

101
EDUCATION
Throughput does not equal Bandwidth

102
EDUCATION
Network Components
• Network Topologies
– Data Networks
• Point-to-Point, Bus, Token Ring, Fabric
– Storage Networks
• Bus, Arbitrated Loop, Point-to-Point, Fabric
– Hybrids
• Network Types
– Local Area, Metropolitan Area, Wide Area, Storage Area
• Backbone Networks
• Network Interconnectivity
– Repeaters, Hubs, Bridges, Switches, Routers, Gateways

103
EDUCATION
Storage Network Choices
Integrated Device Electronics (IDE)
Small Computer Systems Interface (SCSI) Bus
Enterprise System Connectivity (ESCON)
Serial ATA, Serial Attached SCSI
Fibre Channel
IP Storage
InfiniBand
iSCSI / TCP / IP / GE
“Very Small “Very Skinny
Fibre Channel
Storage over IP
iSCSI / TCP / IP / GE
internet SCSI (iSCSI)
Transmission Control Protocol (TCP)
Internet Protocol (IP)
Gigabit Ethernet (GE)
internet Fibre Channel Protocol (iFCP)
Fibre Channel Internet Protocol (FCIP)
internet Storage Name Service (iSNS)
Infiniband
SRP
iSCSI Extension over RDMA (ISER)

104
EDUCATION
Semantic “Anal”yst: Beware of Semantics
Switch
Hub
Router
GatewayNetwork terminology
misuse is rampant!
• By the Storage
Network Industry
• By Vendors
• By Media
• By Educators
Confusing the
customer!

105
EDUCATION
Beware of Semantics
Switch
BROUTERSWOUTER
GROUTER
Hub
Router
Gateway

106
EDUCATION
Beware of Semantics
Director BridgeSwitch
BROUTERSWOUTER
GROUTER
HubRepeater Loop Switch
Router
Gateway
Smart
Intelligent
Enterprise
Storage
Cluster
Virtual

107
EDUCATION
Plumbing is Beautiful

108
EDUCATION
Storage Networking Futures
• Upper layer protocol changes
– Lazy and Hyper reads and writes
– System SAN awareness, Dynamic discovery
• Higher bandwidth rates for aggregation and
higher speed processors for servers, storage
and adapters
• Focus on PCI Express and Infiniband
• External SANs with Fibre Channel and
iSCSI/TCP/IP/GE
• Internal SANs with SAS and SATA
• NAS & SAN integration

109
EDUCATION
Networking Technologies: Concepts in Internal
and External Networked Storage
• Many
– Storage Network Solutions
– Storage Network Technologies
– Many Opportunities for Benefit
and Value
• Thanks for coming!
– Hope this helps!!
– Please fill out your evaluation
forms!!!
– Good luck with Storage Networks

110
EDUCATION
Outline
Networking
Technologies

111
EDUCATION
Q&A / Feedback
• Please send any questions or comments on this
presentation to the SNIA:
tracknetworking@snia.org
Many thanks to the following individuals
for their contributions to this tutorial.
SNIA Education Committee
Howard Goldstein – Tutorial Manager, Developer & Presenter
Greg Schulz – Tutorial Review
SW Worth – Tutorial Review

EDUCATION
Networking Technologies: Concepts in Internal and External Networked Storage©
Copyright 2007 Howard Goldstein Associates, Inc. HGAI All Rights Reserved.
HGAI Storage Networking
Web Links & Bibliography
For more info contact:
Howard.Goldstein@HGAI.Com

113
EDUCATION
Web Links
http://www.t10.org T10 (SCSI)
http://www.ncits.org/ The NCITS home page
http://www.symbios.com/t10.1/ The (former) T10.1 Task Group
(SSA) home page
http://www.t11.org/ The T11 Technical Committee
home page (Fibre Channel, HIPPI,
IPI, SBCON)
http://www.scsita.org The T13 Technical Committee
home page (ATA & ATAPI)
http://www.fibrechannel.com Fibre Channel Industry
Association (a trade association)

114
EDUCATION
Web Links
http://www.infostor.com Storage Industry information
http://www.snia.org Storage Networking Industry
Association (a trade
association)
http://www.1394ta.org/ IEEE 1394 Trade Association
http://www.infinibandta.org Infiniband Trade Association
http://www.ultra160-SCSI.com/ Information about Ultra160 SCSI
http://searchstorage.com Useful storage informaton
http://www.jobstor.com Jobs in the storage industry
http://www.asnp.org Association of Storage Network
Professionals
http://www.storagenetworking.org Storage Networking User
Groups

115
EDUCATION
Bibliography
• Fibre Channel – A Comprehensive Introduction Robert W. Kembel
• IP SANs– A Guide to iSCSI, iFCP, and FCIP Protocols for Storage Area
Networks Tom Clark
• Building Storage Networks Marc Farley
• iSCSI – The Universal Storage Connection John L. Hufferd
• Resilient Storage Networks – Designing Flexible Scalable Data
Infrastructures Greg Schulz
• Using SANs and NAS W. Curtis Preston
• Storage Area Network Essentials – A Complete Guide to
Understanding and Implementing SANs Richard Barker & Paul Massiglia
These texts were used in creating this course. Some of the course
material contains references from this material

116
EDUCATION
Bibliography
• Storage Security – Protecting SANs, NAS, and DAS John Chirillo &
Scott Blaul
• Inside Windows Storage – Server Storage Technologies for Windows
2000, Windows 2003, and Beyond Dilip C. Naik
• Successfully Implementing and Scaling SANs in the Enterprise - A
Planning Guide for Storage Networking Professionals
SearchStorage.Com
• The Holy Grail of Network Storage Management Jon William Toigo
• Storage Areas Networks for Dummies Christopher Poelker & Alex Nikitin
• Practical Storage Area Networking Daniel Pollack
• Designing Storage Area Networks – A Practical Reference for
Implementing Fibre Channel and IP SANs Tom Clark

117
EDUCATION
Bibliography
• Storage Area Networks – Designing and Implementing a Mass Storage
System Ralph H. Thornburgh & Barry J. Schoenborn
• Cisco Storage Networking Architecture Poster Cisco Press
• IP Storage Networking – Straight to the Core Gary Orenstein
• Fibre Channel for Mass Storage Ralph H. Thornburgh
• Making SCSI Work – A Practical Guide The Paralan Staff
• Fibre Channel for SANs Alan F. Benner
• The Book of SCSI – A Guide for Adventurers Peter M. Ridge
• The Holy Grail of Data Management – Modeling, Evaluating,
Comparing, Implementing, Storage Management Options, and
Solutions Jon William Toigo
• Storage Network Performance Analysis Huseyin Simitci

118
EDUCATION
Bibliography
• Fibre Channel – Gigabit Communications and I/O for Computer Networks Alan
F. Benner
• Building SANs with Brocade Fabric Switches – How to Design, Implement, and
Maintain Storage Area Networks (SANs) with Brocade Fabric Switches Chris
Beauchamp, Josh Judd, & Benjamin Kuo
• Multiprotocol Routing for SANs Josh Judd
• The Fibre Channel Bench Reference Jeffrey D. Stai
• The SCSI Bus & IDE Interface – Protocols, Applications & Programming
Friedhelm Schmidt
• Fibre Channel Switched Fabric Robert W. Kembel
• Fibre Channel Arbitrated Loop Robert W. Kembel
• Storage Networks Daniel Worden
These texts were used in creating this course. Some of the course material
contains references from this material

119
EDUCATION
Bibliography
• Disaster Recovery Planning – Preparing for the Unthinkable Jon
William Toigo
• Fibre Channel Volume 1: The Basics Gary R. Stephens & Jan V. Dedek
• Small Computer System Interface – An In-Depth Exploration Of ….
David Deming
• What is Fibre Channel? Ancot Corporation
• Fibre Channel Connection to the Future Fibre Channel Association FCA
• Introduction to Storage Area Network, SAN Ravi Kumar Khattar, Mark S.
Murphey, Giulio John Tarella, Kjell E. Nystrom

120
EDUCATION
Bibliography
• FireWire System Architecture Second Edition IEEE 1394a MindShare,
Inc. Don Anderson
• PCI-X System Architecture MindShare, Inc. Tom Shanley
• Storage Virtualization Technologies for Simplifying Data Storage and
Management Tom Clark
• Serial Attached SCSI: Beyond the Basics SAS Knowledge Tek Author:
Hugh Curley
• SAS Storage Architecture – Serial Atached SCSI MindShare, Inc. Mike
Jackson

121
EDUCATION
Bibliography
• Securing Storage A Practical Guide to SAN and NAS Security
Himanshu Dwivedi
• Storage Network Security – SNIA Technical Tutorial Roger Cummings &
Hugo Fruehauf
• Storage Virtualization – SNIA Technical Tutorial Frank Bunn, Nik
Simpson, Robert Peglar & Gene Nagle
• Storage Network Management – SNIA Technical Tutorial Roger
Cummings
• Network Storage Terms & Acronyms Storage Networking Industry
Association SNIA

122
EDUCATION
Bibliography
• Ethernet The Definitive Guide Charles E. Spurgeom
• Gigabit Ethernet for Metro Area Networks Paul Begell
• Gigabit Ethernet Networking David G. Cunningham, Ph.D. and William G.
Lane, Ph. D.
• High Performance TCP/IP Networking – Concepts, Issues, and
Solutions Mahbub Hassan, Raj Jain
• Gigabit Networks – Standards and Schemes for Next-Generation
Networking Paul Izzo

Storage networking-technologies

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Destacado

Destacado (20)

Similar a Storage networking-technologies

Similar a Storage networking-technologies (20)

Más de sagaroceanic11

Más de sagaroceanic11 (20)

Último

Último (20)

Storage networking-technologies