Carrier Ethernet and Segment Routing

1. Evolve Carrier Ethernet Architecture with SDN and
Segment Routing
Dennis Cai, Distinguished Engineer, Cisco Anna Wielosz, Sr. Product Manager, Cisco
Songbin Wei, Director of Product Marketing, Cisco
Abstract— Ethernet technology has been evolving to become the
main Wide Area Network transport technology. To address the rising
CAPEX and OPEX issues, service providers are deploying Unified
MPLS technology to consolidate various networks into one an
integrated carrier ethernet network. Although Unified MPLS has
simplified MPLS deployment in many aspects, it is still deemed
complex and it is not as agile as what clouding computing demands.
This paper proposes a further evolution of carrier ethernet
architecture by coupling the emerging segment routing and Software-
Defined Network (SDN) technologies. This new architecture will
significantly simplify the network infrastructure while providing rich
converged services with embedded high availability and agility.
Keywords— Carrier Ethernet , Segment Routing, SDN, MPLS,
Unified MPLS, RFC 3107, G.8032, Spanning-tree, IP Fast Re-
Route, L2VPN, VPLS, PBB-EVPN, E-VPN
I. CARRIER ETHERNET OVERVIEW
Since its debut in 1980s, Ethernet has gone through major
transformation. New technologies such as Ethernet OAM and
QoS have been developed to make Ethernet the de facto carrier
network transport technology. By migrating TDM to Ethernet,
service providers have improved bandwidth efficiency and
reduced costs, but they now face new challenges such as
network resiliency, and multi-service management and
maintenance complexity.
Carrier ethernet infrastructures are designed to transport
multiple services between the access, core and data center
networks. A typical carrier ethernet network consists of access,
aggregation, and service edge layers. The predominant
topologies – rings and hub-and-spoke, used in carrier ethernet
networks are generally dictated by existing physical fiber ring
layouts.
To maximize the efficiency of investment, service
providers are looking for technologies that are able to support
all services over a single infrastructure. Services driving carrier
ethernet growth today are:
 Mobile backhaul – rapid LTE growth is fueling
metro infrastructure build outs. The infrastructures
must also support earlier generations like 2G/3G,
which require hub-and-spoke connectivity with
TDM/ATM or Ethernet access. 4G/LTE is
Ethernet/IP based and, as opposed to hub-and-
spoke, introduces hub-to-hub flows with X2
interfaces. In addition, mobile streaming, MBMS
and eMBMS require multicast support. As the
number of mobile device grows exponentially,
there is a strong need to move to IPv6 based
deployments for these networks.
 Consumer video – traditional broadcast video to
consumer TVs is predominantly delivered using
IPTV techniques, such as E-Tree or IP/MPLS-
based Multicast. These services require strict
SLAs in terms of packet loss and jitter. As the
trend moves towards streaming to any devices,
unicast streams will become more important and
more efficient caching techniques need to be
implemented.
 Private cloud – XaaS implementations require
reliable and flexible transport to provide access to
distributed cloud resources. Cloud service
transport is a new, rapidly growing area for carrier
ethernet. Using L2VPN or L3VPN, it is paired
with SDN techniques like OpenFlow to redirect
traffic on demand to the best possible service
location. Cloud computing introduces new
requirements for flow-based forwarding
granularity where specific applications can be
forwarded to different locations for processing.
 Business VPN and Wholesale services rely on
MEF carrier ethernet 2.0 E-Line, E-LAN, E-
Access, or L3VPN options. These services require
strict SLA management through mechanisms such
as Ethernet or MPLS OAM.
The above services, which represent fast-growing
categories that all impose different requirements on the
underlying infrastructure. In addition, the infrastructure must
be flexible enough to adapt to emerging services. This is why
infrastructure ease of use and programmability play a crucial
role, as we will describe in the following chapters.
II. CARRIER ETHERNET SOLUTIONS AND CHALLENGES
Service providers are facing the challenge to provide
intelligent services that can quickly adapt to constantly
changing market needs. New services such as cloud transport
require unprecedented flexibility and elasticity from the
network. Increasing bandwidth demands and decreasing ARPU
put pressure on reducing network cost. At the same time,
services need to be deployed faster and more cost effectively to
stay competitive.

2. New carrier ethernet technologies need to fulfill the
following requirements to help overcome the challenges that
service providers are facing:
• Implement multiple services over a single
infrastructure with a unified operational model to achieve
CAPEX and OPEX optimizations.
• Scale multi-service deployments and maximize the
use of the deployed capacity.
• Increase network reliability to protect large failure
domains carrying multiple mission critical services.
• Deploy network elements and services faster to
improve time to market for new services, and shorten
service delivery times.
Carrier ethernet solutions have evolved from native Ethernet,
to static MPLS, and then to Unified MPLS to solve the above
challenges. This paper proposes further evolution delivering
significant simplification as compared to current solutions.
The Native Ethernet solution consisting of 802.1q, QinQ,
802.1ad is relatively simple to deploy as it involves a small
number of protocols and variables for which to plan. Carrier
grade high availability is achieved using G.8032 or Multi-
chassis ICCP-based LACP technologies. However, native L2
VLAN flooding domains are difficult to manage as they grow.
In addition, L2 domains are subject to loops and broadcast
storms that can take down an entire network.
While still being deployed in the access, native Ethernet
solutions are generally no longer deployed in the aggregation
layer, where they have been replaced by static or dynamic
MPLS.
MPLS-TP has been developed to replicate a SONET/SDH-
like approach for MPLS over Ethernet. LSPs are statically
provisioned via management systems; backup paths are
predetermined in advance to provide deterministic failover.
While the technology is attractive for service providers with a
predominantly optical background, it has a key shortcomings
of being based on point-to-point technology, therefore it’s sub-
optimal for multi-point and multicast services. In addition, the
introduction of new devices into the network requires rigorous
planning and establishment of multiple tunnels as opposed to a
plug-and-play dynamic approach.
The Unified MPLS approach consists of extending dynamic
MPLS to the access while achieving multiservice, multipoint,
and high availability benefits. To increase scalability,
hierarchical LSPs with BGP and RFC3107 deployed at the
access. While the solution solves multiple challenges, it is
perceived as very complex to manage because of the extension
of IGP, BGP, and MPLS LDP/T-LDP protocols to the access.
Cisco has introduced technologies like Auto-IP, Autonomic
Networking and LFA/Remote LFA[1]
to lower the deployment
complexity of MPLS networks with a large number of devices.
The following table (Figure-1) compares how existing
technologies address service provider requirements.
Figure 1: Comparison of existing carrier ethernet solutions
While Unified MPLS is the preferred solution to resolve
many of these challenges, areas exist which can be further
improved by the solutions discussed in this paper:
 How to accelerate deployment and reduce further
the complexity and maintenance by reducing the
number of protocols involved in infrastructure
management.
 How to accelerate service deployment by
eliminating device-by-device touch points.
 How to improve forwarding granularity to include
flow awareness necessary for emerging services.
III. THE CARRIER ETHERNET ARCHITECTURE EVOLUTION
WITH SEGMENT ROUTING (SR) AND SDN
Overall, we propose a new carrier ethernet architecture
solution that combine the advantages of the fully-distributed
control plane using the unified MPLS network solution and the
fully centralized control plane using the SDN/OpenFlow
solution. See Figure 2.
Figure 2: The balance between a fully-distributed control
plane unified MPLS solution and a fully-centralized control
plane SDN/OF solution
We will describe this proposal at two layers: the network
transport layer and the service layer.

3. A. Carrier ethernet Transport Evolution
This proposal uses SDN and segment routing technology to
evolve the carrier ethernet transport architecture. We will start
with a basic introduction of the segment routing technology,
followed by intra-domain, inter-domain transport operation,
and by SDN controlled traffic engineering at the end.
Segment Routing (SR): SR is currently described in the
IETF SR draft [2]
. The draft is driven by Cisco Systems and is
supported by leading telecom companies, WEB/OTT
companies, and other network equipment vendors. Segment
routing can have flexible control plane and data plane options.
In this proposal, we use IGP (ISIS or OSPF) as control plane,
and MPLS as data plane.
Here we provide a brief introduction to how segment
routing operates. Detailed information on segment routing
operation is available in the referenced IETF draft [2]
.
Fundamentally, segment routing uses the IGP (ISIS or
OSPF) extensions to distribute MPLS labels bindings without
requiring a separate protocol such as LDP. There are two types
of segment routing labels: a node label and an adjacency label.
The node label represents the router’s loopback prefix, and is
globally significant within the particular IGP/SR domain. The
adjacency label represents the link prefix, which is significant
only to the local router. Within the segment routing domain, all
routers need to install all other routers’ node label in its
forwarding table, however, router only needs to install its local
adjacency labels in its forwarding table.
From a forwarding point of view, the source router will
encode an ordered list of the labels in the packet header, which
will instruct the other routers in the network how to forward
the packet to the destination. Basically segment routing is
based on the concept of source routing technology. The source
router has all the intelligence to encode the labels into packet
header, while the other routers only need to do MPLS label
forwarding based on the label in the packet header.
Source router can dynamically learn the labels from IGP; it
can also get the labels from the external SDN controller.
Intelligent SDN Controller has a centralized view of the entire
network and can calculate the best path with certain
performance requirement like bandwidth, jitter, delay, etc., and
program this label stack to the source router.
There are many advantages to applying segment routing
technology into carrier ethernet networks. Here are two major
advantages:
 Simplicity: As shown in the Figure 2, segment
routing can remove the need for MPLS LDP, LDP
FRR, RSVP-TE and TE-FRR, RFC3107 BGP
transport. Fundamentally, the whole network runs
only one transport protocol suite: IGP/SR. With
fewer protocols, and more important, fewer
protocols interactions, this can dramatically reduce
the network complexity
 Application-enabled routing: segment routing can
be seamlessly integrated with SDN controllers or
other applications. The SDN controller can learn
the network topology and the real time state
information. Using this information, SDN
controller can calculate the best network path
based on requested criteria. With segment routing,
the SDN controller needs only to send an ordered
list of labels to the source router. It does not need
to program each individual router along the path
and the intermediate routers do not need to
maintain any state. This provides a much more
scalable and simple solution for traffic
engineering. Fundamentally, millions of
applications or flows can have millions of
different network paths. Achieving this is
impossible with traditional technologies like
RSVP-TE. Moreover, all changes applied to the
source router label stack are reflected
instantaneously in the traffic paths.
Intra-domain transport operation: For IGP/SR to work, it
requires some minimal configuration of IGP routing
parameters, router’s loopback address, link IP address, segment
routing node label, and segment routing global label range. For
the link IP address, it can either use IP-unnumbered interface
for IPv4, or use link local address for IPv6. For the other
parameters, the SDN controller can maintain an inventory
database, as shown in Figure 3. When a new router is inserted
into the network, this router will register with SDN controller
via some type of management network. The SDN controller
will assign the loopback address, node label, label range,
gateway label information via southbound API like
Netconf/Yang to provision the router.
For any-node-to-any-node communication within the same
IGP/SR domain, by default, the network will just use the IGP
shortest path. This is dynamic without needing any external
controller. It’s a single segment routing label, as shown in
Figure 3.
Figure 3: Intra-Domain segment routing overview
Inter-domain transport operation: One of the major
advantages of the unified MPLS solution is high scale. For a
large network, it uses hierarchical transport LSPs, as defined in
RFC3107. The whole network is divided into multiple

4. standalone IGP/LDP domains. For inter-domain traffic, it uses
BGP to advertise the network node’s loopback prefix to
achieve end-to-end network reachability. Basically. it uses two
levels of the transport LSP labels: IGP label for the reachability
within each IGP domain, and BGP label for the reachability
across domains.
With segment routing and SDN, this can be easily solved
with a label stack that includes local and remote gateway
labels, plus the end node label, as shown in Figure 4.
Figure 4: Inter-Domain segment routing
With this design, each of the IGP/SR domains are isolated:
there is no route redistribution or inter-domain prefix leaking
required. This method can remove the need for hierarchical
BGP LSP label as defined in RFC3107 for very large-scale
network deployments. With anycast labels, it can also provide
a much simpler node redundancy mechanisms. When the
primary gateway node fails, the IGP will re-converge and the
segment routing Topology Independent Fast Re-route can
provide 50msec protection. With the RFC 3107 approach,
convergence requires BGP and IGP interaction, which increase
the software and forwarding complexity. In addition to the
complexity, the convergence time will be much slower:
generally hundreds of million seconds or even into full seconds
in pathological cases.
As shown in Figure 4, the gateway router runs multiple IGP
processes. For example, one process is for CE1, the other
process is for Core network. Those IGP processes can program
the forwarding table independently.
The SDN controller is required to provision the segment
routing label stacks to the end nodes. The network node does
not need to keep the label or prefix outside of its domain. This
is a much simpler and more scalable solution. Only the SDN
controller has the information for all network nodes. Each node
has registered with SDN controller when it joins the network
initially. For a large networks, a single SDN controller may not
be sufficient. In these cases, operator can provision multiple
SDN controllers for different administrative domains. The
solution will require a method of information exchange
between controllers, which is outside of the scope of this paper.
Segment routing traffic engineering with intelligent SDN
controller: Aside from the simplicity, with the help of an
intelligent SDN controller, segment routing can provide the
flexibility of building traffic engineering tunnels on-demand
and on a per-flow level. This kind of flexibility is essential to
efficiently monetize the network bandwidth. Figure 5
illustrates how the SDN controller works together with
segment routing network to provide the bandwidth on-demand
services. In the picture, a particular application or flow requires
certain bandwidth between node A and Z. The best IGP path
has link congestion between C and D. With intelligence on the
SDN controller, it can calculate a new alternative path.
Subsequently, the SDN controller will send an explicit or loose
list of the segments to the head end node A. In this example,
the new path will avoid the link C and D. The head end router
A will impose this label lists. The path information will be
encoded in the packet header as MPLS label stacks, which
completely remove the need for the signaling of using RSVP-
TE. This provides a much simpler and more scalable traffic
engineering solution. How does the SDN controller learn the
network topology and running state information is outside of
the scope of this paper.
Figure 5: Segment routing traffic engineering using SDN
controller
B. Carrier ethernet service evolution
With the advent of SDN, NFV, and DC virtualization, more
and more advanced services are being moved to the cloud.
Carrier ethernet networks will retain the function for providing
the intelligent pipe between customers and the cloud. The
traffic pattern will be quite straightforward; most of the traffic
flow is between customer site and the data center site or the
service node with distributed data center. With this new trend,
we can simplify the carrier ethernet service by using a static
service label.
As shown in Figure 6 below, the SDN controller can
statically assign a service label for a particular service on both
ends: at the transport node and service node. When the access
node receives the packet, it can classify the packet based on
certain criteria such as VLAN ID, and then simply impose a
service label and segment routing labels. Advanced packet
classification could be done as well depends on the platform
hardware capability. Packet will be forwarded to the service
nodes based on either IGP path or explicit segment routing
label stacks, which are provisioned by SDN controller. When
the service node receives the packet, it will process the packet
based on the service label. The carrier ethernet network is now
kept very simple by only using a static service label. This
eliminates network protocols such as BGP, or Target-LDP for
the L2VPN and L3VPN services. The advanced services are
enabled on the service nodes. The service nodes, however,

5. could use traditional BGP or T-LDP network protocols
between themselves. This could include traditional multi-point
L2VPN service, L3VPN service, and residential broadband
services. It could also have some value-added service such as
DPI. Those services could be either in the data center, or
distributed locally on the SP POP site.
Service label can either be an MPLS PWE3 label or a
generic network service header as defined in the NSH draft[4]
.
The transport segment routing path could be between the
transport access node and the service node as shown, or it
could be between transport access node and the DC gateway
node as shown in Figure 6.
Figure 6: SDN-based static service label provisioning
This SDN-based service provisioning is similar to some
existing solutions like MPLS-TP where it uses NMS to
provision the MPLS-TP tunnel and PWE3 pseudowires
statically. While they do have some common attributes, an
SDN and the segment routing-based solution offer quite a few
advantages:
At network transport level, segment routing offers a self-
deployed, self-protected dynamic network infrastructure. It can
provide both IGP-based ECMP, as well as SDN-based on-
demand traffic engineering tunnel.
MPLS-TP solutions require static provisioning of both
transport and backup tunnels. This provisioning work must be
done on each node along the path – it’s not a plug-and-play
solution. When nodes are added or removed from the topology
these static paths must be re-provisioned for each node, which
increases operational complexity and the chance for costly
provisioning errors. Additionally, ECMP support is poor in
these topologies and there is no efficient method to
dynamically change network paths.
At the service level, both solutions use static label for
service delivery. However, the solution described in this paper
offers a key advantage of service node redundancy. MPLS-TP
solutions requires additional mechanism to monitor the primary
service node availability, and failover to the backup node
during failures. This will introduce not only additional
complexity, but also slow convergence.
Service node redundancy: SDN-based service provisioning
can provide much better service node redundancy.
Multiple service nodes could be part of the same
redundancy group. These nodes all share the same anycast
segment routing node label at transport level. At the service
level, the SDN controller will provision the same service label
on all redundant service nodes for a particular service.
Transport nodes will use an anycast segment routing label and
the common service label. The packet will be forwarded to the
service node based on the best IGP path, which could be load
balanced (based on the ayncast label method) to both service
nodes. When either of the service nodes receives the packet, it
will process the packet based on the common service label
assigned. The solution can provide active/active service load
balancing and fast convergence, which can’t easily be offered
by other solutions like MPLS-TP.
IV. SUMMARY
Service providers need to keep evolving their carrier
ethernet network architectures to meet the bandwidth demands,
as well as to reduce CAPEX and OPEX. Many service
providers are moving from native Ethernet solutions to Unified
MPLS solutions. Unified MPLS solutions provide a single
network infrastructure for converged services, and offers great
advantages of high scale, high availability, and optimized
forwarding. However, the Unified MPLS solution lacks parts
of the operational simplicity that is offered by the native
Ethernet solution. The segment routing and SDN-based carrier
ethernet architecture evolution, as described in this paper, can
provide the full advantages of Unified MPLS solution, and
with great simplicity and agility. This simple, agile, highly
scalable, and highly available carrier ethernet network solution
can help service providers grow profitably in the era of the
cloud-centric networking.
REFERENCES
[1] RFC 5286, Basic Specification for IP Fast Reroute: Loop-Free
Alternates http://datatracker.ietf.org/doc/rfc5286/
[2] Segment Routing: http://tools.ietf.org/html/draft-previdi-filsfils-isis-
segment-routing-02
[3] RFC 3107, Carrying Label Information in BGP-4 http://www.rfc-
base.org/rfc-3107.html
[4] NSH, http://datatracker.ietf.org/doc/draft-quinn-nsh/
Acronyms
ATM: Asynchronous Transfer Mode
CE: carrier ethernet
DC: Data Center
ECMP: Equal Cost Multi-Path

6. ICCP: inter-chassis communication protocol, defined in the IETF
LACP: Link Aggregation Control Protocol
LFA: Loop-Free Alternates
LTE: Long Term Evolution, a telephone and mobile broadband communication standard
LTE X2: X2 is a transport interface used to connect enodeBs together in a mobile LTE/4G network
MBMS: Multimedia Broadcast Multicast Service
eMBMS: evolved MBMS
MEF: Metro Ethernet Forum
E-LINE, E-LAN, E-ACCESS, E-TREE: those are MEF defined Ethernet service, please refer to: http://metroethernetforum.org/carrier-
ethernet/carrier-ethernet-services
MPLS: Multi-Protocol Label Switching
MPLS-TP: MPLS Transport Profile
NMS: Network management system
OAM: Operations Administration and Maintenance (Ethernet protocol)
PWE3: Pseudowire Emulation Edge-to-Edge
SONET: Synchronous Optical Network(ing)
SDH: Synchronous Optical Network/Synchronous Digital Hierarchy
SDN: Software Defined Network
SR: Segment Routing
TDM: Time-division multiplexing
VPN: Virtual Private Network
L2VPN: Layer2 VPN
E-VPN: Ethernet VPN
PBB-EVPN: Provider backbone bridging Ethernet VPN
VPLS: Virtual Private LAN Service
XaaS: Everything as a Service