Brad Hedlund's speaking session at Interop Las Vegas 2012. Big Data clusters and SDN enabled clouds invite a new approach to data center networking. This session for data center architects will explore the transition from traditional scale-up chassis based Layer 2 centric networking, to the next generation of scale-out Layer 3 CLOS based fabrics of fixed switches.

1. Architecting Data Center
Networks in the era of Big Data
and Cloud
Brad Hedlund
Spring Interop—May 2012
VIDEO of this session:
http://bradhedlund.com/?p=3912
Global Marketing

2. Two approaches to DC Networking
THE SAME OLD
• Centralized, Scale-up Layer 2 networks
• Monstrous chassis switches
TRILL OpenFlow VEPA SPB
Or a Different Approach
Distributed, Scale-out Layer 3 fabrics
Efficient fixed switches
Open, industry standard protocols
2 Brad Hedlund
Global Marketing

3. Networks that suck for Cloud & Big Data
Core
Network
Dist Topology
Access
VM
Capacity
Topology
PARTITIONED CAPACITY
“Data center networks are in my way”
-James Hamilton, AWS
3 Brad Hedlund
Global Marketing

4. Networks that Don’t suck for Cloud & Big Data
Spine
Network
Topology
Leaf
VM
Capacity
Topology
UNIFORM CAPACITY
 All points equidistant
4 Brad Hedlund
Global Marketing

5. Big Data
TCP
switch switch
TCP
switch switch switch switch switch
Name Node Job Tracker Secondary NN Node
Node Node
Node Node Node Node Node
Node Node Node Node Node
Node Node Node Node Node
Node Node Node Client Client
TCP TCP
Rack 1 Rack 2 Rack 3 Rack 4 Rack N
World
• Inverse Virtualization
• Workloads orchestrated like cattle
• L2 or L3 network. Does it matter?
5 Brad Hedlund
Global Marketing

6. Basic requirements of Cloud (IaaS)
switch switch
Physical
Network
switch switch switch switch
FW LB
World Virtual
Network
VM VM VM VM
• Secure, Scalable Multi Tenancy
• Location independence
• On Demand virtual networks
6 Brad Hedlund
Global Marketing

7. Blend the Virtual and Physical Networks
VLAN 20
switch
VLAN 10
switch switch
VM VM
vSwitch vSwitch
VM VM VM VM VM VM
Host Host
• Tenant subnet = Network VLAN
7 Brad Hedlund
Global Marketing

8. Abstract the Virtual Network from Physical
Segment ID 20
switch
Segment ID 10
switch switch
VM VM
vSwitch vSwitch
VM VM VM VM VM VM
Host Host
• Network Virtualization Overlay
• Tenant subnet = Software VLAN
8 Brad Hedlund
Global Marketing

9. Scale-up centralized Layer 2
L3
• 2-post Rooted Architecture
L2
• Centralized L2/L3
• L2/L3/ARP table scale?
• Scale w/ Bigger Boxes
• Precious Pets
• VLAN Provisioning? vSwitch vSwitch
• Broadcasts
VM VM VM VM
9 Brad Hedlund
Global Marketing

10. Scale-out Layer 3 Leaf/Spine Fabric
(16) (8) (2)
(128)
L3
(64) (16)
L2
1980 Server ports
3072 Server ports
768
6144 Server ports
• Mesh from Leaf to Spine • Non-blocking Spine
• OSPF, ISIS, BGP, TRILL • 3:1 @ ToR
• ToR w/ 16 uplinks (ECMP) • 128 port 2RU Spine
10 Brad Hedlund
Global Marketing

11. Uniform fabric for Cloud & Big Data
(16) (8) (2) Storage Access
Hadoop
(128)
L3
(64) (16)
L2
Name Node Job Tracker
Secondary NN
Node Database ports
6144 Server
Node
Node Node Node
Node Node Node Node
vSwitch vSwitch
Node Node Block I/O
Node Node
Node Node Node Node
Node Node NAS
Node Node
VM VM VM VM
Client Client
Object
Node Node
Rack 1 Rack 2 Rack 3 Rack N
11 Brad Hedlund
Global Marketing

12. Attaching Services & North/South
(16) (8) (2)
(128)
L3
(64) (16)
L2
LB Firewall x86 Gateways
Name Node
Node
Job Tracker
Secondary NN
LB Firewall
Node Node
Node Node
Node Node
Node Node vswitch vswitch vswitch World
VM VM VM VM VM VM VM VM VM
Client Client
vswitch vswitch vswitch
Rack 1 Rack N VM VM VM VM VM VM VM VM VM
12 Brad Hedlund
Global Marketing

13. Generic Logical Architecture 1
World
• Overlay based L2 Fabric DC router
• Physical/Static FW
L3 L3
FW NAT FW NAT
L2
L2
LB LB L3
VM VM
L2 L2
VM VM VM Big Data
Green Co. Orange Co.
13 Brad Hedlund
Global Marketing

14. Generic Logical Architecture 2
World
• Overlay based L2
• Virtual/Mobile FW Fabric DC router
• Overlay Gateway Pub DMZ
L3 L3
FW NAT FW NAT
L2
L2
LB LB L3
VM VM
L2 L2
VM VM VM Big Data
Green Co. Orange Co.
14 Brad Hedlund
Global Marketing

15. Generic Logical Architecture 3
World
• No Overlays
• TRILL based L2 Fabric DC router
• Virtual/Mobile FW
Pub DMZ
TRILL
L3 L3
FW NAT FW NAT
L2
L2
LB LB L3
VM VM
L2 L2
VM VM VM Big Data
Green Co. Orange Co.
15 Brad Hedlund
Global Marketing

16. Density: Fixed vs. Chassis
10G per RU @ Line Rate (L3)
140
120
100
80
Chassis
60 Fixed
40
20
0
2008 2010 2012 2014
16 Brad Hedlund
Global Marketing

17. Power: Fixed vs. Chassis
18
Max Watts / Line Rate 10G (L3)
16
14
12
10
Chassis
8 Fixed
6
4
2
0
2010 2012 2014
17 Brad Hedlund
Global Marketing

18. What are the Challenges?
(16) (8) (2) Dell
Fabric Manager
 Design Templates
(128)  Validate deployment
 Automate fabric configuration L3
(16)
 Monitoring & Operations L2
• Deployment & Cabling • Layer 2 (TRILL?)
• Configuration & Policy • Design Best Practices
• Monitor & Troubleshoot Dell Fabric Manager
18 Brad Hedlund
Global Marketing

19. Webinar: CLOS Fabrics Explained
http://closfabric.eventbrite.com/
Wednesday, June 20, 2012 from 10:00 AM to 1:00 PM (ET)
HOST
CO-HOST
Brad Hedlund
Ivan Pepelnjak
19 Brad Hedlund
Global Marketing

20. Three Stage Layer 3 Leaf/Spine Fabric
(64) (8) (2)
/26
(128) 0/0
/26 /26
(512) 0/0
L3
L2
24,576 Server ports
• Non-blocking @ top tiers • Leaf+ToR mesh groups
• Default route @ ToR & Leaf • ~8usec worst case
20 Brad Hedlund
Global Marketing

21. The case for 40G QSFP switch ports
10G 10G 10G 10G 10G 10G 10G 10G
VS
SFP+ SFP+ SFP+ SFP+
QSFP
$1K $1K $1K $1K
$1,800
32 ToR
$230K $512K
21
Brad Hedlund
Global Marketing

22. Comparing Fabric efficiencies
of Fixed vs. Chassis designs
• Supplemental Section
Brad Hedlund
May 2012
Global Marketing

23. Total Fabric Power: Chassis vs. Fixed
KW
non-blocking
250
200
150
Chassis
100 Fixed
50
0
384 2048 4096 8192 Fabric size
23 Brad Hedlund
Global Marketing

24. Total Fabric RU: Chassis vs. Fixed
RU
non-blocking
700
600
500
400
Chassis
300 Fixed
200
100
0
384 2048 4096 8192 Fabric size
24 Brad Hedlund
Global Marketing

25. 8192 non-blocking Fabric
(64) (8) (2)
(128)
8192 non-blocking
• 384RU
• 153.6KW
• 8192 ISL
• 192 switches
25 Brad Hedlund
Global Marketing

26. 8192 non-blocking Fabric
(32)
(256)
8192 non-blocking
• 608RU
• 216.3KW
• 8192 ISL
• 288 switches
26 Brad Hedlund
Global Marketing

27. 4096 non-blocking Fabric
(32) (2)
(64)
4096 non-blocking
• 192RU
• 76.8KW
• 4096 ISL
• 96 switches
27 Brad Hedlund
Global Marketing

28. 4096 non-blocking Fabric
(16)
(128)
4096 non-blocking
• 304RU
• 108.1KW
• 4096 ISL
• 144 switches
28 Brad Hedlund
Global Marketing

29. 2048 non-blocking Fabric
(16) (2)
(32)
2048 non-blocking
• 96RU
• 38.4KW
• 512 ISL
• 48 switches
29 Brad Hedlund
Global Marketing

30. 2048 non-blocking Fabric
(8)
(64)
2048 non-blocking
• 152RU
• 54KW
• 2048 ISL
• 72 switches
30 Brad Hedlund
Global Marketing

31. 384 non-blocking Fabric
(4) (2)
(6)
384 non-blocking
• 20RU
• 8KW
• 96 ISL
• 10 switches
31 Brad Hedlund
Global Marketing

32. 384 non-blocking Fabric
(2)
(12)
384 non-blocking
• 26RU
• 18KW
• 384 ISL
• 14 switches
32 Brad Hedlund
Global Marketing

33. 256 non-blocking Fabric
(2)
(4)
256 non-blocking
• 12RU
• 4.8KW
• 64 ISL
• 6 switches
33 Brad Hedlund
Global Marketing

34. 256 non-blocking Fabric
(2)
(8)
256 non-blocking
• 22RU
• 6.7KW
• 256 ISL
• 10 switches
34 Brad Hedlund
Global Marketing

35. Total Fabric Power: Chassis vs. Fixed
KW
3:1 oversubscribed
80
70
60
50
40 Chassis
Fixed
30
20
10
0
768 1536 3072 6144 Fabric size
35 Brad Hedlund
Global Marketing

36. Total Fabric RU: Chassis vs. Fixed
RU
3:1 oversubscribed
250
200
150
Chassis
100 Fixed
50
0
768 1536 3072 6144 Fabric size
36 Brad Hedlund
Global Marketing

37. 768 @ 3:1 oversubscribed Fabric
(2)
(16)
768 @ 3:1
• 20RU
• 7.2KW
• 64 ISL
• 18 switches
37 Brad Hedlund
Global Marketing

38. 768 @ 3:1 non-blocking Fabric
(2)
(16)
256 non-blocking
• 30RU
• 7.5KW
• 256 ISL
• 18 switches
38 Brad Hedlund
Global Marketing

39. 1536 @ 3:1 oversubscribed Fabric
(4) (2)
(32)
1536 @ 3:1
• 40RU
• 14.4KW
• 128 ISL
• 36 switches
39 Brad Hedlund
Global Marketing

40. 1536 @ 3:1 oversubscribed Fabric
(2)
(32)
1536 @ 3:1
• 54RU
• 17KW
• 512 ISL
• 34 switches
40 Brad Hedlund
Global Marketing

41. 3072 @ 3:1 oversubscribed Fabric
(8) (2)
(64)
3072 @ 3:1
• 80RU
• 28.8KW
• 1024 ISL
• 72 switches
41 Brad Hedlund
Global Marketing

42. 3072 @ 3:1 oversubscribed Fabric
(4)
(64)
3072 @ 3:1
• 108RU
• 34KW
• 1024 ISL
• 68 switches
42 Brad Hedlund
Global Marketing

43. 6144 @ 3:1 oversubscribed Fabric
(16) (8) (2)
(128)
6144 @ 3:1
• 160RU
• 57.6KW
• 2048 ISL
• 144 switches
43 Brad Hedlund
Global Marketing

44. 6144 @ 3:1 oversubscribed Fabric
(8)
(128)
6144 @ 3:1
• 216RU
• 68KW
• 2048 ISL
• 136 switches
44 Brad Hedlund
Global Marketing

45. Total Fabric Power: Chassis vs. Fixed
KW
3:1 oversubscribed - Massive
1400
1200
1000
800
Chassis
600 Fixed
400
200
0
12288 24576 49152 98304 Fabric size
45 Brad Hedlund
Global Marketing

46. Total Fabric RU: Chassis vs. Fixed
RU
3:1 oversubscribed - Massive
5000
4500
4000
3500
3000
2500 Chassis
2000 Fixed
1500
1000
500
0
12288 24576 49152 98304 Fabric size
46 Brad Hedlund
Global Marketing

47. 12,288 @ 3:1 oversubscribed Fabric
(32)
(64) (8) (2)
(256)
12,288 @ 3:1
• 448RU
• 166.4KW
• 5120 ISL
• 352 switches
47 Brad Hedlund
Global Marketing

48. 12,288 @ 3:1 oversubscribed Fabric
(16)
(256)
12,288 @ 3:1
• 432RU
• 136.3KW
• 4096 ISL
• 272 switches
48 Brad Hedlund
Global Marketing

49. 24,576 @ 3:1 oversubscribed Fabric
(64)
(128) (8) (2)
(512)
24,576 @ 3:1
• 896RU
• 332.8KW
• 10,240 ISL
• 704 switches
49 Brad Hedlund
Global Marketing

50. 24,576 @ 3:1 oversubscribed Fabric
(32)
(256)
(512)
24,576 @ 3:1
• 1120RU
• 328.9KW
• 16,384 ISL
• 800 switches
50 Brad Hedlund
Global Marketing

51. 49,152 @ 3:1 oversubscribed Fabric
(128)
(256)
(1024)
49,152 @ 3:1
• 1792RU
• 665.6KW
• 20480 ISL
• 1408 switches
51 Brad Hedlund
Global Marketing

52. 49,152 @ 3:1 oversubscribed Fabric
(64)
(512)
(1024)
49,152 @ 3:1
• 2240RU
• 657.9KW
• 32,768 ISL
• 1600 switches
52 Brad Hedlund
Global Marketing

53. 98,304 @ 3:1 oversubscribed Fabric
(256)
(512)
(2048)
98,304 @ 3:1
• 3584RU
• 1331.2KW
• 40960 ISL
• 2816 switches
53 Brad Hedlund
Global Marketing

54. 98,304 @ 3:1 oversubscribed Fabric
(128)
(1024)
(2048)
98,304 @ 3:1
• 4480RU
• 1315.8KW
• 65,536 ISL
• 3200 switches
54 Brad Hedlund
Global Marketing

55. The power to do more
55