如何用k8s打造國產5G NFV平臺？剖析經濟部5G核網技術的關鍵

如何用k8s打造國產5G NFV平臺？
剖析經濟部5G核網技術的關鍵
梁維恩 Jace Liang

About Me
版權宣告©工業技術研究院 2
• 梁維恩 Jace Liang
• 工研院資通所/資料中心架構與雲端應用軟體組 Div. X.
• What our team does
▪ NFV Performance Lab (cooperate with intel®)
▪ NFVI Architect of MOEA 5G Project
e-mail: jace.liang@itri.org.tw
Github: mJace

Outline
• 網路功能虛擬化帶來的衝擊
• VNF部屬於Kubernetes的優勢與挑戰
• 電信業者的5G Kubernetes Solution
• 國家5G計畫 NFVI 平台介紹
• 系統整合實例分享

網路功能虛擬化帶來的衝擊
The impact of NFV

Network Function Virtualization
CDN Session Border
Controller
WAN
Acceleration
DPI Carrier
Grade NAT
Tester / QoE
Monitor
SGSN / GGSN PE Router Firewall
Typical network appliances
General purpose
servers
Standard storage
NFV-based Approach
Standard switches
NFV is a process of decoupling the network function
from proprietary hardware appliances

The Impact of NFV
• The existing player need to invest and adopt new techs
▪ Help telecom increase service agility and efficiencies
• Easier to deploy service
• Able to reuse physical resources
• The new business potential
▪ Server providers
▪ VNF providers
▪ NFVI providers
▪ Solution providers
▪ Network operators

Evolving from VNFs to CNFs
Ref : https://events.linuxfoundation.org/wp-content/uploads/2017/11/2018-
OSSNA-CNF-Journey-in-Telecom-Seminar.pdf

Performance Issue in NFV
• Traffic hidden from monitoring
▪ Cisco estimates that about 73% of data-center traffic will come from within the data
center by 2019—most of this traffic is virtual machine to virtual machine (VM to VM)
communication
▪ The inability to isolate production from monitoring traffic
• No Solution to address performance uncertainty in NFV
▪ Where is the major bottleneck point of an NFV system
Container / POD
VNF Application
vCPUs vNICs
Container / POD
VNF Application
vCPUs vNICs
Host OS
(State transition overhead due to privileged operations and idleness)
Hardware Layer
(Resource exhaustion, buffer overflow or queue imbalance at NICs)
Virtual Switch Layer
(Less optimized packet processing engine)
Less optimized VNF implementations
Inter-Container communication overhead
Network I/O
Overhead
NFV Architecture with Major Bottleneck Points

VNF部署於k8s的優勢與挑戰
The Pros and Cons for VNF on k8s

Pros to run VNF on k8s
版權宣告©工業技術研究院
10
• K8S as container orchestrator
▪ Container grouping using pod
▪ Self-healing
▪ Auto-scalability
▪ DNS management
▪ Load balancing
▪ Rolling update or rollback
▪ Resource monitoring and logging
......

Challenge of Networking
• Kubernetes native CNI is not design for telco VNF
▪ Decoupling Control/Data plane is not easy in native K8S network.
• Only one NIC for each Pod.
▪ The network behavior is not suit for VNF.
• Whole traffic must go through Master or Load Balancer.

Challenge of Compute & Management
• Native K8S does not guarantee performance stability.
▪ It is very critical for VNF to require a stable and predictable performance.
• Native Kubernetes does not support NUMA aware and CPU Pinning.
• Native k8S is lack of detail hardware spec. info.
▪ When Deploy/Migrate/Scale VNF, must consider its required HW function.
• Native K8S don’t know if the machine supports certain CPU instruction and the NIC’s
spec.

電信業者的5G k8s Solution
The K8S solution from TelCo.

DANM
• From Nokia
▪ Support VXLan, IPAM
▪ Allowing pods to attach multiple network interfaces. Calico, Flannel, SR-IOV, etc
▪ a Kubernetes controller capable of centrally managing both VxLAN and VLAN
interfaces of all Kubernetes hosts
▪ another Kubernetes controller extending Kubernetes' Service-based service
discovery concept to work over all network interfaces of a Pod
https://github.com/nokia/danm

SONA
• Used by SK
• Extension work from OpenStack SONA
• Pure SDN Based overlay solution
▪ OvS L3 load balancing
▪ OvS L2/L3 connectivity
• Support OVS-DPDK, Smart NIC
• Support Multi NIC
https://github.com/sonaproject/sona-cni

CNI-Genie
• From Huawei
• Support Multi-NIC for Pod
https://github.com/huawei-cloudnative/CNI-Genie

Akraino
• Create an open source edge software stack for different
scenario
▪ IOT Edge, Edge on prem, AI Edge
▪ Nokia, AT&T, SAMSUNG, etc.
▪ https://www.lfedge.org/projects/akraino/

國家5G計畫 NFVI 平台介紹
K8S NFVI of MOEA 5G Project

How do we address these problems
• Kubernetes native CNI is not design for telco VNF
▪ Decoupling Control/Data plane is not easy in native K8S network.
• Only one NIC for each Pod.
▪ The network behavior is not suit for VNF.
• Whole traffic must go through Master or Load Balancer.
• Native K8S can not guarantee performance stability.
▪ It is very critical for VNF to require a stable and predictable performance.
• Native Kubernetes does not support NUMA aware and CPU Pinning.
• Native k8S does not own detail hardware spec. info.
▪ When Deploy/Migrate/Scale VNF, must consider its required HW function.
• Native K8S don’t know if the machine support certain CPU instruction or the NIC’s spec.

ITRI Open Source X-K8S solution
• Network
▪ Multus CNI
▪ SRIOV
▪ SmartNIC
▪ SRIOV Device Plugin
• Compute/Management
▪ CMK – CPU Management for
Kubernetes
▪ NFD – Node Feature Discovery
▪ VNF Monitor
▪ Performance Optimizer

Multus CNI
• Enables attaching multiple network interfaces to pods
▪ So that the VNF pod can separate the network by different usage, for example
• flannel as manage
• SRIOV VFs as Data in/out
▪ Support any CNI – SRIOV, Host-device, etc
• Most Popular
▪ Most used by community
▪ Already included in kubespray

SRIOV
• Single Root I/O Virtualization
▪ As a PCI-SIG Standard, SRIOV allowing PCIe I/O device to provide multiple virtual
function.
• SRIOV-CNI
▪ Allowing Pods to use SRIOV VF as it’s CNI
• SRIOV Device Plugin
▪ Allowing K8S scheduler to manage SRIOV NIC as device resource.
NIC
Pod Pod
vNIC vNIC

資通所重要營運資料，禁止複製、轉載、外流 ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTE -24-
SRIOV vs. OVS-DPDK
64 Bytes 128 Bytes 256 Bytes 512 Bytes 1024 Bytes 1280 Bytes
OVS-DPDK-1-Core 2.51 2.46 2.38 2.26 1.74 1.47
OVS-DPDK-2-Cores 6.31 6.1 5.64 4.98 3.71 3.04
OVS-DPDK-4-Cores 12.96 12.5 11.82 9.32 4.77 3.83
SRIOV 59.52 32.89 17.85 9.32 4.77 3.83
BM 59.52 32.89 17.85 9.32 4.77 3.83
0
10
20
30
40
50
60
70
Throughput(Mpps)
OVS-DPDK vs SRIOV vs Baremetal 4 Port - Throughput (L2-Fwd)
OVS-DPDK performance is similar to SRIOV
performance in large packet size scenario.

SRIOV – Example https://asciinema.org/a/263659

SmartNIC
• Smart NIC enables the functionality of OVS in a hardware
approach
▪ Improve network throughput
▪ Reduce CPU usage
NIC
Pod Pod
vNIC vNIC
OVS

Ericsson Cloud SDN & Netronome Agilio CX

The Mapping Between CPU and NIC
• The CPU core and NIC for packet generator should be in the same CPU socket, or
the QPI interface will be the performance bottleneck.
▪ 10GbE line rate (64bytes packet) = 10.00e9 bits / (8 bits * (64 + 20)bytes ) = 14.88e6 packets -> 67.2 nano-sec / pkt
▪ CPU and NIC in same socket, 10G port could generate 14.88 Mpps.
▪ CPU and NIC in different socket, 10G port could only generate 10 Mpps.
18 Core Intel(R) Xeon(R) CPU
E5-2695 v4 @ 2.10GHz
CPU 1
18 Core Intel(R) Xeon(R) CPU
E5-2695 v4 @ 2.10GHz
CPU 2
QPI
X710-DA4
adapter GEN
QPI interface will be the
performance bottleneck.

Result of NUMA misconfiguration
L2-Fwd Rx L2-Fwd Tx L3-Fwd Rx L3-Fwd Tx ACL Rx ACL Tx MPLS Rx MPLS Tx
BM 59.52 59.52 53.29 53.29 37.38 33.82 59.52 58.17
VM w/ NUMA 59.52 59.52 53.29 53.29 37.41 33.78 59.52 58.17
VM w/o NUMA 59.52 59.46 53.36 43.4 37.46 30.66 59.52 56.96
0
10
20
30
40
50
60
70
Throughput(Mpps) Baremetal vs SRIOV 4 Port - Throughput (64Bytes)
VM without NUMA-aware configuration
may decrease 10 – 20% performance.
VM performance with NUMA-aware configuration
is similar to Baremetal performance.

CMK & NFD
• CPU Manager for Kubernetes
▪ Pin Pods on specific CPU core.
▪ https://github.com/intel/CPU-Manager-for-Kubernetes
• Node Feature Discovery
▪ Detects hardware feature available on each node in k8s cluster.
▪ https://github.com/kubernetes-sigs/node-feature-discovery
*CMK is now a milestone for k8s v1.16 as a alpha version.

Compute / Management
• Compute / Management – node wise
▪ Support NUMA aware and CPU Pinning
▪ Enhanced Platform Awareness (EPA) VNF management
SRIOV NIC SmartNIC

• Compute / Management – node wise
SRIOV NIC SmartNIC
kubelet
api-server
CoreDns
Scheduler
kube-proxy controller
MME
SGW-C
SGW-U
PGW-C
PGW-U

• Compute / Management – cluster wise
kubelet
api-server
CoreDns
Scheduler
MME
PGW-C
SGW-U
PGW-C
PGW-U
SRIOV
Network
SmartNIC
Network
SGW-C
kubelet
api-server
CoreDns
Scheduler
SGW-C
PGW-USGW-U
SGW-U SGW-U
SGW-U

VNF Monitor
• How to get NUMA related info?
▪ Which CPU core that my pod runs on?
▪ NUMA node of NIC in my pod?
▪ Is core-affinity set?

Which CPU core that my container
runs on?
https://github.com/mJace/numacc

Developing feature.
▪ APM system for 5G VNF
• Customized APM system for 5G VNF
▪ Self performance optimization
▪ Data plane network slicing
• Network slicing based on the SRIOV data plane
▪ DNS / Service discovery for extra network interfaces

系統整合實例分享
End-to-End System Integration

EPC
38
• 完成3GPP R14版本 Container Based C/U Split vEPC
雛型系統，支援快速彈性擴充。每秒20個UE連線的情
況下約可乘載50K個使用者終端。
• 於 MME 實作 eMBMS 功能，支援車載網路群播應用。
並與Content Provider和商用E-UTRAN整合測試群播
功能完成
• 支持OVS-DPDK and SRIOV技術
• 透過SmartNIC兩種不同的數據路由，藉以實現大吞吐
量(接近10Gbps)，改善虛擬化後造成之網路傳輸延遲，
不需額外佔用伺服器計算資源，延遲時間可縮短至
300us以內。
• 完整利用SmartNIC效能進行GTP封包處理
• Future Work: SBA 5GC
X-Kubernetes

iMEC
發展 MEC 與 4G/5G 接取網路的整合，技術成果為 iMEC
iMEC 提供節省後端網路頻寬且低延遲雲端平台之解決方案
Enterprise
EPC Cloud
Wi-Fi AP
4G/5G RAN
FTTx
iMEC
企業私有網路終
端
設
備
網路&服務虛擬化智慧路由使用者辨識
多邊緣雲管理即時服務啟動系統效能水平擴展
一鍵自動部署
支援 OPNFV, Kubernetes及
DC/OS等多種 NFV 平台
支援網路本地分流功能(Local
breakout/Traffic Offload) ，
流量負載平衡
支援 UE Identify 功能，可辨
別垂直專網之企業用戶
一鍵自動部署iMEC系統與
ME APP
支援多邊緣雲互連架構，進
行服務佈建與管理
Just-in-Time 技術，可於0.2
秒快速啟動服務，並節省
edge cloud資源
可以動態增加User Plane
Function數量，使得效能線
性成長

End-to-End System Integration

5G Network Architecture on X-K8S
Key Features & Specification
• High Throughput
 10 / 25 / 40 / 100 Gbps
• CNF – Containerized Network Function / Services
• Performance Optimized
 SmartNIC, SRIOV, CMK, NFD, Multus
• Network Isolation
CNF / Services on X-K8S
• CNF : iMEC
 Data traffic offloading
• CNF : III-vEPC
 Use P4 SmartNICs as an accelerator for GTPU
• CNF : SON (Self-Organizing Network)
• APP : VR Community (III)

Q&A
Contact us.
jace.liang@itri.org.tw
Try our x-k8s
https://github.com/ITRI-ICL-Peregrine/x-k8s

如何用k8s打造國產5G NFV平臺？剖析經濟部5G核網技術的關鍵

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a 如何用k8s打造國產5G NFV平臺？剖析經濟部5G核網技術的關鍵

Similar a 如何用k8s打造國產5G NFV平臺？剖析經濟部5G核網技術的關鍵 (20)

Último

Último (20)