AWS offers a variety of tools enabling users to create highly scalable, durable and resilient architectures and the user community has developed a broad range of best practices and frameworks to build rock-solid systems on top of AWS. Many customers such as Netflix, Airbnb, Smugmug and others, already demonstrated those practices in production systems. While the theoretical concepts of building resilient architectures are well established, the practices of maintaining such systems are less understood, mostly because of the unpredictability of production environments under stress conditions. To address this issue, many adopted the concept of Game Days, which consists of simulating unexpected failures to test the resilience, detect and fix flaws, and more importantly train the operation teams on emergency situations. This session covers the best practices learned from many AWS customers who implemented the Game Days practice and the different failure simulation techniques that can be used on AWS.

1. AWS Summit 2014
GAME DAYS
CRASH-TEST YOUR APPLICATION … AND YOUR TEAM
Carlos Conde
Chief EMEA Technical Evangelism
@caarlco

2. PLAY IN A
SIMULATED ENVIRONMENT

6. SIMULATION ENVIRONMENT

7. SIMULATION ENVIRONMENT
FOR CRISIS SITUATIONS

8. GOOD WEATHER DOESN’T MAKE GOOD SAILORS

9. PREPARE FOR
EXTREME SITUATIONS

10. THINGS ARE UNRELIABLE
UNTIL THEY ARE TESTED

11. HOW TO PLAY ?
GAME PREPARATION

12. SUPERVISORS & PLAYERS

13. MAKE A PLAN
ROLE PLAY & SCOPE DEFINITION

14. SET A DATE (DEADLINE)
MAKE IT FUN

15. CREATE THE SIMULATION
ENVIRONMENT

16. GAMEDAY
ENVIRONMENT
AWS ACCOUNT A
PRODUCTION
ENVIRONMENT
AWS ACCOUNT B

17. CLOUDFORMATION
TEMPLATE

18. USE AWS IAM
CONTROL WHO CAN DO WHAT
DURING THE GAME

19. ACTIVATE AWS CLOUDTRAIL
GAMEPLAY RECORDING & AUDITING

21. SIMULATE ACTIVITY

22. PRE-RECORDED NETWORK ACTIVITY

23. (THROTTLED) LOAD TEST
USE SPOT INSTANCES + TOOLS

25. USING AMAZON EC2 TO SIMULATE
2.4 MILLION PLAYERS

26. COMMUNICATION CHANNELS
IRC, SHARED DOCS, VIBER, …

27. PLAY !
THE FUN PART

28. SIMULATE FAILURES

30. TERMINATE RESOURCES
CHANGE SECURITY GROUPS
CHANGE IAM ROLES
DISABLE IAM USER
CHANGE /ETC/HOSTS FILE
AMAZON RDS FAIL-OVER TEST

32. RECORD & ANNOTATE
PLAYERS ACTIONS

33. IF YOU DON’T MEASURE IT,
IT DIDN’T HAPPEN

34. AFTER PARTY
CLOSING THE GAME

35. DEBRIEF
WHERE THE MAGIC HAPPENS

36. HOW LONG DOES IT TAKE TO
DETECT AN EVENT ?

37. NEW HIRE ?
GO TO THE SIMULATOR

38. KEY TEAM PLAYERS ?
SIMULATE “FAILURE” SITUATION

39. VALIDATE ASSUMPTIONS
PROVE YOUR ARCHITECTURE
KNOWLEDGE OF PROCEDURES

40. COMMUNICATION CHANEL
& CHAIN OF COMMAND

41. GOOD WEATHER DOESN’T MAKE GOOD SAILORS

42. TELL ME ABOUT YOUR GAMES!
CARLOSCO@AMAZON.LU

43. Thank You!
AWS EXPERT?
GET CERTIFIED!
Carlos Conde aws.amazon.com/certification
Chief EMEA Technical Evangelism
@caarlco