High Speed Fiber Services and Challenges to the Core Network by Seiichi Kawamura

2. Copyright © 2014 BIGLOBE Inc.2 Some figures about BIGLOBE (AS2518) l 3 million consumer ISP users in Japan l 250Gbps total internet traffic p including enterprise, data center, transit traffic San Jose [vPOP] -‐‑‒Equinix IX(10G) LA [vPOP] -‐‑‒Coresite Any2(10G) Japan [Core] -‐‑‒ 3 million broadband customers -‐‑‒ 110 colocation customers -‐‑‒ 2 DCs fully owned by BIGLOBE -‐‑‒ Great connectivity -‐‑‒ Equinix(20G), JPIX(10G), BBIX(30G), JPNAP(30G) -‐‑‒ PNI with ALL major ISPs Hong Kong[POP] -‐‑‒HKIX(10G) Singapore [POP] -‐‑‒Equinix IX(10G) -‐‑‒ SGIX (1G)

4. Copyright © 2014 BIGLOBE Inc.4 Growth of Traffic Metro Network (Tokyo) Metro Network (Tokyo) Subscribers Peering Subscribers Peering Peering traffic grows 1.37x every year (Mobile Traffic 1.8x, broadband 1.2x)

6. Copyright © 2014 BIGLOBE Inc.6 Metro Network Metro Network Challenge1 : Keeping the metro growth down 28*10G Metro Network Metro Network 10*10G 5 years Core growth 1.2x every year Keeping growth down by •  Hot potato routing •  ECMP tuning (enhanced hash) •  Higher thresholds in capacity growth •  Studying flows and moving peers to a closer exit No MPLS Pure IP

7. Copyright © 2014 BIGLOBE Inc.7 Challenge 2: Streaming l 30% of FTTH traffic is streaming l We better have a cost effective way of handling streaming traffic (and all video traffic) l Cost effective peering was our answer p  Does not mean we love caches Tactics •  Use netflow and PeeringDB to find where we can pick up the traffic •  Remote peering (unfortunately) to keep costs low •  Extensive talks with content providers •  Disclosed peering policy to pick up peering traffic more effectively(more on this later) http://www.biglobe.co.jp/en/peering_policy.pdf

8. Copyright © 2014 BIGLOBE Inc.8 backbone Streaming traffic basic principle Minimize total cost of delivery and keep it from going to other links IX ports, transit ports Keep traffic from overflowing into ports with high utilization. Can be used as backup links, but should control traffic Try to peer at router closest to end user Lower backhaul cost Lower peering costsCosts are fixed But actual networks don’t look like this…

9. Copyright © 2014 BIGLOBE Inc.9 Actual Network Access network concentrator Core routing Peering Ideally, we want video traffic to be here, but can we peer where the access concentrators are? No. Can we have caches here? Yes, but GGC, AANP, Open Connect, and what next? A horrible cost model More flexible DC interconnects may help

10. Copyright © 2014 BIGLOBE Inc.10 Challenge3 : Peering l Domestic peering was REALLY an issue p IX port costs Ø 2 years ago, 10G ports cost 20,000 usd per month p the burden of peering, mistrust, lack of communication between peers Ø No place to talk about peering l We wanted a better relationship with content providers

11. Copyright © 2014 BIGLOBE Inc.11 Changes coming from Mobile and Cloud area l More mobile and cloud traffic p Broadband at 1.3x growth while mobile at 1.8x growth l Content providers have different requirements than ISPs p semi-full route p simple route servers p latency aware p mtu9000 p fast detection using BFD p DoS protection as a service

12. Copyright © 2014 BIGLOBE Inc.12 Evolution of peering community l Japan did not have a place to talk about peering until a few years ago p PAST: People disliked talking about peering at JANOG l Peering relied heavily on ISPs and Telcos •  BoFs, study councils, talks at NOGs, and joint efforts by the IXPs and the community –  more focus on content/dc and its requirements

13. Copyright © 2014 BIGLOBE Inc.13 community activities l Google Groups : Peering in Japan p local Japanese language only p discussion on latest peering issues p no IX personnel on list p host Peering BoFs l Tutorials p IXP provides low cost tutorials (available regularly) p free tutorials at JANOG (not always available) l CloudIX Study Council p Group of members(BGP operators) in BBIX doing technical experiments p sharing skills and helping out each other to peer

14. Copyright © 2014 BIGLOBE Inc.14 Challenge4 : Automation[ a work in progress ] l Our data center is well automated p in the past dc config change >>>>>>>>>> peering p dc requires orchestration with servers and other virtual devices l Since we started peering more (and more globally) we are now needing better automation tools l Current implementation p Simple html(form) +mysql+ jinja2 to generate configs via Web UI Ø Trying to shift to Django p Exscript to push configs to router

15. Copyright © 2014 BIGLOBE Inc.15 Other misc challenges l Unnecessary traffic avoidance p P2P services that have security problems are controlled at the edge p BCP38 to stop sending unnecessary traffic Ø  still in initial deployment phase l Higher density with better power efficiency p Working to keep PUE around 1.2-1.3 p Switching to higher density line cards l Measuring actual user experience p “1Gbps service” is more marketing talk than actual experience p Measuring response to popular sites (alexa) p Python script to crawl command and filter out noise data “hping -p 80 -c 5 --syn [url]” works well with unix, PSPing works well with Windows

17. Copyright © 2014 BIGLOBE Inc.17 Next steps for evolution l MPLS, TE, auto-bandwidth p Not necessary for domestic consumer service backbone, but need TE to control global network p We did not need this in the past since we did not provide VPN services (we may in the future) p we have a vxlan network running, so considering EVPN as one of the solutions l Better network management and security p BMP and other tools for better peering management p APIs to the router p BGP Flow Spec l Differentiation of services inside the backbone p Last mile QoS and internet fast lane?

18. Copyright © 2014 BIGLOBE Inc.18 Summary l Higher speed services to the consumers, means new challenges to the core network l Peering really helps, but it requires a good ecosystem to be helpful p diversity in IX services, fair pricing, carrier neutrality, open talks about peering l Handling streaming and video, is still a big headache p is there a better way than everybody has their own cdn? l Automation and tools are key to handling bigger traffic in a cost effective way l Measurements make a difference in user experience. “1Gbps marketing talks” don’t

High Speed Fiber Services and Challenges to the Core Network by Seiichi Kawamura

High Speed Fiber Services and Challenges to the Core Network by Seiichi Kawamura

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente(20)

Destacado

Destacado(20)

Similar a High Speed Fiber Services and Challenges to the Core Network by Seiichi Kawamura

Similar a High Speed Fiber Services and Challenges to the Core Network by Seiichi Kawamura(20)

Más de MyNOG

Más de MyNOG(20)

Último

Último(20)

High Speed Fiber Services and Challenges to the Core Network by Seiichi Kawamura