Google's Software Defined WAN Architecture for Efficient Data Center Interconnect

2013 OpenFlow Korea All Rights Reserved
The Great Transformation
Feb, 2013
OpenFlow Korea
(www.OPENFLOW.or.kr)
발표자 : 서영석
기술매니저
mr.seo@openflow.or.kr

OFA
Data Center Site 3
HW Tables
OFC RCS
Quagga
Site
Broker
Google’s Software Defined WAN
SDN
GatewayGlobal
Broker
iBGP / ISIS
Non-TE(ISIS) Path
Tunneling
App
OFA
Data Center Site 2
HW Tables
OFCRCS
Quagga
Site
Broker
Tunneling
App
PaxOS PaxOS
Cluster
Border Router
Cluster
Border Router
Flow
Manager
Path Allocation
Algorithm
Path
Selection
Topology
Manager
OFA
HW Tables
TE Server
Site Level TE Path
OFC
Tunneling
App
PaxOS
Quagga
Cluster
Border Router
Site
Broker
RCS
iBGP / ISIS
EBGP EBGP
Data Center Site 1
EBGP
Demand Matrix
{src, dst -> utility curve}
Abstract Path Assignment
{src, dst -> paths and weights}
Interface up/down
status
Per Site Path Manipulation
Commands
Site level edges with
RTT and Capacity
4가지 주요 기술
- Global Broker : 각 사이트의 필요 대역폭 산정
- TE Server : 필요 대역폭을 위한 경로 설정
- SDN GateWay : TE와 OFC 간 연동
- Quagga : 기존 네트워크와 연동
iBGP
/ ISIS
iBGP
/ ISIS

Google은 전세계에 13개의 Data Center를 가지고 있다.
ATLAS 2010 Traffic Report, Arbor Networks
• Data Growth
– Web expands/changes : billions of new / modified
pages every month
– Every few hours we crawl / refresh more than
entire Library of Congress
– YouTube gains 4+ billion views every day
7%

Google’s Data Center
• Two backbones
– I-Scale : Internet facing (user traffic)
– G-Scale : Datacenter traffic (Data Center간 traffic)
• Cloud Computing Requires Massive Bandwidth
– Low latency access
• Vast majority of data migrating to cloud
• Data must be replicated at multiple sites
• Computation & Storage migration to Data Centers
– Data storage : personal files, logs, company data
– Application execution : word processing, email, calendar
– Content retrieval : photos, music, video
– Web services : search, social, e-commerce
– Large-scale data processing : MapReduce, Hadoop

Google’s Data Center
• 기존 네트워크 Architecture의 치명적 한계에 직면
– Networks don’t function exactly as we would like
• Not deterministic
• Suboptimal behavior and failure handling
• Hard to configure and operate at scale
• Expensive, manual, error prone systems
– Networking is hard
• Performance
• Scale
• Failure Handling
• 혁신적인 네트워크 Architecture의 필요성 증가
– WAN을 하나의 system으로 구현하는 Architecture 필요
• 높은 네트워크 활용
• Routing 최적화 / Traffic Engineering
• non-equal cost /shortest path
• Global knowledge, simplicity, Ops friendly technology
• Multi-chassis architecture

WAN Fabrics Architecture
Gateway
Bandwidth
Broker
Flow
Manager
Path Allocation
Algorithm
Path
Selection
Topology
Manager
TE Server
Commands
Demand Matrix
RTT and Capacity
Interface
up/down status
Data Center Site 1
…
Data Center Site 2 Data Center Site 3 Data Center Site N

WAN Fabrics Network 구성 절차(1/5)
• 기존의 Data Center 연결 구조

• Data Center간 WAN Fabrics Network 구현을 위해 SDN 도입

• 단계적 SDN 구성

• SDN으로 WAN Fabrics 전체 연결

• Traffic Engineering을 통한 WAN Fabrics 구현

WAN Fabrics Network
• 평균 95% 이상의 회선 사용률

WAN Fabrics Network Hardware
• 수 백 개의 non-blocking 10GE port로 구성
• OpenFlow 지원
• BGP, ISIS를 위한 Open source routing stack을
지원
• 선별적 기능 구현
• Site별 Multiple chassis로 구성
– Fault tolerance
– Scale to multiple Tbps

OFA
Data Center Site 3
HW Tables
OFC RCS
Quagga
Site
Broker
SDN
GatewayGlobal
Broker
iBGP / ISIS
Non-TE(ISIS) Path
Tunneling
App
OFA
Data Center Site 2
HW Tables
OFCRCS
Quagga
Site
Broker
Tunneling
App
PaxOS PaxOS
Cluster
Border Router
Cluster
Border Router
Flow
Manager
Path Allocation
Algorithm
Path
Selection
Topology
Manager
OFA
HW Tables
TE Server
Site Level TE Path
OFC
Tunneling
App
PaxOS
Quagga
Cluster
Border Router
Site
Broker
RCS
iBGP / ISIS
EBGP EBGP
Data Center Site 1
EBGP
Demand Matrix
{src, dst -> utility curve}
{src, dst -> paths and weights}
Interface up/down
status
Commands
RTT and Capacity
4가지 주요 기술
- Global Broker : 각 사이트의 필요 대역폭 산정
- TE Server : 필요 대역폭을 위한 경로 설정
- SDN Gateway : TE와 OFC 간 연동
- Quagga : 기존 네트워크와 연동
iBGP
/ ISIS
iBGP
/ ISIS

Trust but Verify : Consistency Checks

High Level Architecture

B/W Broker
TE Server

• TE Server는 SDN Gateway에 Traffic
Engineering Service를 적용
• SDN Gateway는 모든 Site에 TE
Service를 적용하기 위해 각 Site의
Tunneling App과 연동
• Tunneling App을 통해 각 Site의 OFC는
TE Service가 적용된 flow table을
생성하여 OFA에게 전달
• 각 Site의 flow는 TE path를 통해 전달
• 일부 flow는 non-TE path를 통해 전달

• OFC는 두 가지 flow 정보를 이용하여
flow table을 생성
– Quagga를 통해 받은 Legacy Routing을 위한
Flow 정보
– Tunneling App을 통해 받은 TE Service가
적용된 Flow 정보
• OFC는 DataCenter Switch의 OFA에게
flow table을 전달
• OFA는 flow table을 이용하여 HW Table을
생성

Traffic Engineering Example
• West -> East demand
– 100Gb/s low latency
– 200Gb/s bulk transfer

TE Path Allocation
Flow group Allocation Paths and splits
CHS to MRN BE1 320 Gbps out of 320 Gbps
CHS-MRN : 75%
CHS-ATL-MRN : 25%
• Path Selection
– Find Static k shortest passible paths between src and dst
• Path Ordering and Grouping
– Group similar latency paths into path preference groups
– Sort paths preference group by latency
• Compute Flow Group Allocation :
– For each flow group, input:
• Sorted paths preference groups
• Demand with priority (utility function) from broker
– Exhaustive waterfill algorithm
• Fill preferred paths first

Convergence under Failures
• Without TE : Failure detection and convergence is slower :
– Delay ‘inside’ TE << timers for detecting and communicating failures (in ISIS)
– Fast failover may be milliseconds, but not guaranteed to be either accurate or “good”

Centralized TE의 이점
• Better efficiency with global visibility
• Converges faster to target optimum on failure
• Higher Efficiency
– allows for explicit definition of cost functions
– allows for in-house development of optimization algorithms
• Deterministic behavior
– Simplifies planning vs. over-provisioning for worst case variability
– Can directly mirror production event streams for testing
• Supports innovation and more robust SW development
• Controller uses modern server hardware
– Significantly higher performance

Conclusions
• Dramatic growth in WAN bandwidth requirements
– Every 10x, something breaks
– Existing software/hardware architectures make it impractical to deliver cheap
bandwidth globally
• Software Defined Networking enables
– Separation of hardware from software
– Efficient logically centralized control/management
– Innovation and flexibility
• Deployment experience with Google’s global SDN production WAN
– It’s real and it works
– This is just the beginning…

OpenFlow Korea
(www.OPENFLOW.or.kr)

Google's Software Defined WAN Architecture for Efficient Data Center Interconnect

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Google's Software Defined WAN Architecture for Efficient Data Center Interconnect

Similar a Google's Software Defined WAN Architecture for Efficient Data Center Interconnect (20)

Más de NAIM Networks, Inc.

Más de NAIM Networks, Inc. (20)

Último

Último (20)

Google's Software Defined WAN Architecture for Efficient Data Center Interconnect