The presentation was delivered at the RIPE 64 Meeting on 16 – 20 April 2012 in Ljubljana by Alexander Asimov, Qrator network engineer at HLL. Alexander Asimov expressed his concerns about BGP Route Stability and it was suggested that RIPE Atlas may be able to address these issues.

1. BGP Route Stability
Alexander Asimov
<aa@highloadlab.com>
Highload Lab

2. Research history
• 2009-2010 – study of BGP convergence
process:
• Imitation testbed (BGPC + PRIME);
• BGP convergence time equations;
• Experiments with BGP timers;
• Since 2010 – BGP policy recovery model;
• Since 2011 – BGP route monitor.

3. Route instability
1. Router misconfiguration;
2. BGP route loops;
3. Link failure.

4. Default route
Default Route
Incomplete route Route loop at last
table upstream
No prefix at
loopback interface

5. Default route : Stat
• Prefixes affected 12514
• Vulnerability: could be used for DDoS
amplification on AS (~ TTL)
• Link exhaustion
• Billable bandwidth

6. DDoS amplifier
• Prefixes affected 688
• Vulnerability: exploit makes AS work
like DDoS amplifier.
• DDoS at multiple ASes at the same time
with N-time increase of attack magnitude

7. BGP Route Loops
Built-in defense for static loops
Dynamic loops!

8. BGP dynamic loops
• Route withdraw;
• Changing prepend policy;
• Changing LOCAL_PREF.

9. BGP convergence process
• 𝐸𝑇 𝑢𝑝 = 𝜃 𝑑 × 𝐸𝑡 𝑤𝑎𝑖𝑡 , where d –
diameter of AS graph;
• 𝐸𝑇 𝑑𝑜𝑤𝑛 = 𝜃 𝐷 × 𝐸𝑡 𝑤𝑎𝑖𝑡 , where D –
Hamiltonian path of AS graph;
Alternatives paths!

10. Route Withdraw
AS2 AS3
default AS-X
route
200
AS1 AS4
100 AS1 AS2 AS3 AS-X
AS2 AS3 AS-X
AS3 AS-X
AS4 AS-X

11. Route Withdraw
AS2 AS3
W
default AS-X
route
200
AS1 AS4
100 AS1 AS2 AS3 AS-X
AS2 AS3 AS-X
Loop AS2 – AS3
AS3 --
Packet loss AS1, AS2,AS3
AS4 AS-X

12. Route Withdraw
W
AS2 AS3
default AS-X
route
200
AS1 AS4
100 AS1 AS2 AS3 AS-X
AS2 --
Loop AS2 – AS3
AS3 --
Packet loss AS1, AS2,AS3
AS4 AS-X

13. Route Withdraw
AS2 AS3
AS-X
W A default
route
200
AS1 AS4
100 AS1 AS2 AS3 AS-X
AS2 AS1 AS4 AS-X
1 minute problem!
AS3 --
~𝜌 𝑐2𝑝 ∗ 30𝑠
AS4 AS-X

14. Route Flaps
One route withdraw could cause
unstable dynamic loop.
Multiple flaps could make your net
unreachable.

15. Prepend Policy
AS1
200
AS4 AS2 AS-X
AS3 AS1 AS-X
AS2 AS3 AS-X
AS3 AS-X
AS4 AS1 AS-X

16. Prepend Policy
AS1
200 A
AS4 AS2 AS-X
A
AS3 AS1 AS2 AS3 AS-X
AS2 AS3 AS-X
as-path prepend 5
AS3 AS4 AS1 AS-X
Loop AS1-AS2-AS3-AS4
AS4 AS1 AS-X

17. Prepend Policy
AS1
A
200
AS4 AS2 AS-X
W
A
AS3 AS1 AS2 AS3 AS-X
AS2 AS3 AS4 AS1 AS-X
Loop AS1-AS2-AS3-AS4
AS3 AS4 AS1 AS-X
AS4 AS1 AS2 AS3 AS-X

18. Prepend Policy
AS1
200 A
AS4 AS2 AS-X
A AS3 AS1 AS-X(5)
AS2 AS3 AS4 AS1 AS-X
More then 1 minute problem AS3 AS-X(5)
AS4 AS1 AS2 AS3 AS-X

19. Core of the problem
Traffic flow engineering looks
like…

20. Automated traffic flow
engineering
• No opportunity for traffic flow
prediction;
• No use of BGP convergence process;
• Could make networks partially
unavailable.

21. Route flaps : Stat
• Prefixes affected 1720
• Vulnerability:
• Packet loss;
• BGP message noise.

22. BGP Route Policy Loops
200
AS2 AS3
A
200
AS-X
200
AS1 AS4
A
200 AS1 --
AS2 --
AS3 AS-X
AS4 AS-X

23. BGP Route Policy Loops
200
AS2 A AS3
200
AS-X
A
200
AS1 AS4
200 AS1 --
AS2 AS3 AS-X
AS3 AS4 AS-X
AS4 AS-X

24. BGP Route Policy Loops
200
AS2 A AS3
200
AS-X
A
200
AS1 AS4
200 AS1 AS2 AS3 AS5 AS-X
AS2 AS3 AS4 AS5 AS-X
AS3 AS4 AS5 AS-X
AS4 AS5 AS-X

25. BGP Route Policy Loops
200
AS2 AS3
200
AS-X
A
200
AS1 A AS4
200 AS1 AS2 AS3 AS4 AS5 AS-X
AS2 AS3 AS4 AS5 AS-X
Loop AS1-AS2-AS3-AS4
AS3 AS4 AS5 AS-X
AS4 AS1 AS2 AS3 AS5 AS-X

26. BGP Route Policy Loops
200
AS2 AS3
200
AS-X
A
200
AS1 A AS4
200 AS1 AS2 AS3 AS4 AS5 AS-X
AS2 AS3 AS4 AS5 AS-X
AS3 AS5 AS-X
Again!
AS4 AS5 AS-X

27. BGP Route Policy Loops : Stat
• Prefixes affected 149
• Vulnerability:
• Packet loss
• BGP message noise

28. Breaking down
1. Route flap
2. Change prepend policy
AS-X AS-X
3. AS_PATH poisoning
AS-X AS2 AS-X
Makes AS2 ignore your route

29. Breaking down
1. Route flap
2. Change prepend policy
AS-X AS-X
3. AS_PATH poisoning
AS-X AS2 AS-X
Makes AS2 ignore your route
How not to fall into another loop?

30. One “killer” feature
Autonomous Systems Reverse
Map:
1. Traffic flow engineering;
2. AS architecture design;
3. BGP loop prediction.

31. AS Mapping projects
• Physical links discovery
No use of route policy;
• Macro map
Shows links that are used by somebody,
not by your AS;
• Route policy data
Outdated, incomplete.
• Model abstraction too far from reality

32. BGP Policy recovery project:
Current state
• Mathematical model for BGP route
policy recovery;
• Mathematical model for AS graph;
• New active verification methods;
• Working prototype;
• Negotiations with RIPE ATLAS.

33. They had problems…
AS174 AS3356 AS7018 AS6939 AS701
AS3549 AS209 AS4323 AS1239 AS12389
AS2848 AS3257 AS6461 AS2914 AS8468
AS23148 AS8447 AS20485 AS6830 AS8220
AS8928 AS3303 AS4589 AS42708 AS6453
AS6730 AS31130 AS3491 AS3320 AS8218
AS286 AS702 AS3561 AS20764 AS31323
AS20632 AS4766 AS680 AS29686 AS5089
AS10026 AS12350 AS2516 AS3786 AS12741
AS7575 AS1916 AS2273 AS9498 AS1785
And more then 4k other ASes!

34. Examples
AS174
• increases DDoS 17 times
• Default route: 25 prefixes affected
AS3356-AS3549
• increases DDoS 8 times
• Route flap: 12 prefixes affected
• Default route: 86 prefixes affected

35. Errors grouped by RR
ARIN
APNIC Errors in RR %
Errors in ASes %
RIPE
0 20 40 60

36. Duration
90,00%
80,00%
70,00%
60,00%
Flap
50,00%
Route loop
40,00%
Default route
30,00%
DDoS amplifier
20,00%
10,00%
0,00%
<1h <2h <4h <8h

37. Trends
16000
14000
12000
10000
8000
6000 Found Errors
4000 Fixed Errors
2000
0

38. Robin Hood
Hello.
During our research project we have detected IP address “*.*.*.*" in
AS***, which multiplies ICMP packets. 11 times is a mean value.
It is dangerous vulnerability because this IP could be used by attackers to
N-times increase DoS attack.
This could be harmful for your own network infrastructure and also break
down attacked network.
Could you tell me for what purpose your router was configured by such
way?
Hello
This is assigned to one of our customers. We cannot see how there
equipment is configured. If you believe this is in violation of our
Acceptable use policy you can email abuse@***.com to report that.

39. In the core of the Internet
BGP
2.2.2.0/24
4 second loop
=
DDoS*4
ASX2 Default ASX1
route

40. In the core of the Internet
Client ASes BGP
2.2.2.0/24
Max = 350*DDoS
> 50 amplifiers 4 second loop
Mean = 10*DDoS =
DDoS*4
ASX2 Default ASX1
route

41. In the core of the Internet
Client ASes BGP
2.2.2.0/24
Max = 350*DDoS
> 50 amplifiers 4 second loop
Mean = 10*DDoS =
DDoS*4
ASX2 Default ASX1
route
dst: aplifiers
src: 2.2.2.1

42. In the core of the Internet
Client ASes BGP
2.2.2.0/24
Max = 350*DDoS
> 50 amplifiers 4 second loop
Mean = 10*DDoS =
DDoS*4
ASX2 Default ASX1
route
dst: aplifiers
src: 2.2.2.1 Result = 40*DDoS

43. Results
• The problem in client network is your
problem;
• The problem in your upstream network is
your problem too;
• Unstable prefix is invisible from origin AS;
• BGP balancing without prediction could
make network unavailable;
• We have monitor. Community needs some
trusted mechanism for notification.

44. Thank you for
listening!
Questions?