will explain about the devops culture for developing openstack in kakao[project name krane]

1. Krane, Devops and people
Andrew Yongjoon Kong
Cloud Computing Cell
Infra & Data platform team
Kakaocorp
Kakaocorp

2. KakaocorpAndrew. Yongjoon kong
• Cloud Technical Advisory for Government Broad Cast Agency
• Adjunct Prof. Ajou Univ
• Korea Data Base Agency Acting Professor for Bigdata
• Member of National Information Agency Bigdata Advisory committee
• Kakao à Daum Kakao à Kakaocorp, Cloud Computing Cell lead
Korean
Edition

3. KakaocorpSome Numbers about kakao openstack
1563 tenants
632 pull request since 2014.9
251 VMs are created/deleted per day
8703 vms

4. KakaocorpSome information about kakao openstack
openstack release from grizzly to Kilo
5 times upgraded
total 4Regionpg,seoulv2,seoulv1,busan
additional service Heat/Trove/Sahara

5. Kakaocorp
Cloud, we do have SDN, not have O
penflow, No others
eth0
Compute node
nova-compute
neutron-linuxbri
dge-agent
neutron-dhcp-a
gent
Gateway
10.10.100.1
linux bridge
vm
IP:10.10.100.2/32
Routing Table
1 10.100.10.2/32 via 192.1.1.201
BGP
192.1.1.202 BGP
Virtual Switch block
Virtual Process block
Neutron-l3-agent
vlan
vlan
Virtual Router
Service Route Table
1
Management Route Table
1
Practice Frugality to Boost Creativity

6. KakaocorpSPECIAL Network Resource
• Generally, Known as /32 network.
• No L2 (link) consideration needed anymore ( no subnet )
• With Dynamic Routing Protocol, it move every where.
• Simple IP planning ( Just think of IP ranges )
• It’s very Atomic Resource, it keeps its IP after migration through zones
10.0.0.1 / 32 or
IP 10.0.0.1 netmask 255.255.255.255

7. Kakaocorp
Integrated Information Service Bus
& EIP : Code Name KEMI
Enterprise Integration
• Topic based Data ETL
• Can cover every computing res
ource ( Physical Server, Virtual
instance, Container, Public Clo
ud )
• Abstracting “Data Center Infor
mation layer”
• Can make deep engineering ex
perience over every resources.
Physical
Servers
Virtual
Instances
Containers
External
Clouds
Others
(switches,
logs)
monitoring
KEMI
IMS
(kakao CMDB
API)
ESB
Rule
Engine
Notificati
on
ETL
Data Center Information abstraction layer
API
predicting
scheduling
Openstack
Heat
Other
Service
API
Data Center (or Service ) Management Activity
control

8. KakaocorpKEMI-stats:The packages and workflow

9. KakaocorpKEMI-log

10. KakaocorpCITE

11. KakaocorpDKOS

12. KakaocorpThe Problem

13. The Term : Devops
• Devops is no longer composite of two words
“Development + Operation”
• Cultural movement that changes
• how individuals think about their work,
• values the diversity of work done,
• supports intentional processes that accelerate the rate by
which businesses realize value, and measures the effect of
social and technical change
• Devops is about finding ways to adapt and innovate
social structure, culture, and technology together in
order to work more effectively

14. Software Development
Methodologies
• Waterfall
• Focus on sequential progression
• Large amount of time spent in requirements and design
phase.

15. Software Development
Methodologies
• Agile
• lightweight and flexible than previous methods such as
waterfall
• Agile Manifesto
• We are uncovering better ways of developing software by doing it and
helping others do it. Through this work we have come to value:
• individuals and interactions over processes and tools
• working software over comprehensive documentation
• customer collaboration over contract negotiation
• responding to change over following a plan

16. Software Development
Methodologies
• Scrum
• Ken Schwaber and Dr. Jeff Sutherland presents
• focuses on maximizing a development team’s ability to
quickly respond to changes in both project and customer
requirements.
• Sprints ( 1~4 weeks )
• Daily standup
• Restrospect
• What happened
• What went well/poorly ?

17. Devops Misconceptions
• Devops only for Developers and System administrators
• Devops is a team
• Devops is a Job Title
• Devops Means Doing All the Work with Half the People
• Devops is about Tool/Automation

18. Devops Anti-Patterns
• Blame Culture
• Prefer not to blame
• Silos
• Prefer share or usable product
• Root Cause Analysis
• Prefer quick/frequent response

19. Devops for Pillars
Collaboration Affinity Tools Scailing

20. Collaboration comprises
• Asynchronous code review
• Documentation
• Updating issues and bug report
• Demonstrating weekly progress
• Regular status updates
• Pairing

21. Collaboration: Kakao cloud
• Asynchronous code review: Github Pull Request
• Documentation: Source code or Wiki Page
• Updating issues and bug report:
• Jira Comment
• Githup and Jira Integration
• Demonstrating weekly progress: Sprint Review
• Regular status updates: Daily Standup, Auto Testing
• Pairing : Github Review/Merge

22. Collaboration: kakao cloud

23. Collaboration: kakao cloud
• Develop Github proxy.
• Sending Githup event to Kakao talk

24. Affinity
• Have to think about “What comprises Team”
• Have to fight against Diversity
• Diversity in Gerder/Race/Nationality/Age/Veteran
Status/Religion/Familial Status is OK and inevitable
• Way of Handling works
• Measurement of success
• Communication style in works
• Leadership
• The key is “From individiual to Teams”

25. Tools
Local Development
Environment
Version Control
Artifact Management Automated Test
Development
Tools

26. Tools: kakao cloud
Local Development Environment
KFIELD
Version Control
Github
Artifact Management
Chef + Cookbook version
Automated Test
Jenkins’ Github PR Trigger +
Rake+Kitchen + (openstack/docker)
Development Tools

27. Tools: Local Development
Environment, KField

28. Tools: Version Control, Github
• Pull Request based commit history management
• No Self Merge rule Applied

29. Artifact Management : Chef
/Cookbook version
• Two Species source code
• One is Openstack Code ( nova, neutron , cinder ….. )
• One is Openstack Deploying Code ( chef , infra-automation)
• From Chef code, it uses openstack’s certain branch
• Use cookbook metadata for artifact version.
• Automatic Engaged after testing and merged

30. Automated Test &Build
• 3 category
• On Demand Automated test and build
• Scheduled Test and build
• Triggered Automated Test and Build
• More category
• Smoke testing
• Regression Testing
• Usability Testing
• A/B Testing

31. Automated Test &Build : Jenkins Test
• Triggered
• Sometimes use on-demand test , “Test this please” comment on github pull request

32. Automated Test &Build : Jenkins Test

33. Tools is accelerator (in good/bad way)
• Tool is about which programming languages, IDEs, text
editor, shell, configuration management solution, or
chat program to use
• Standardization of Tools
• Improve communication
• Set boundaries
• Repair understanding within the scope of the devops compact
• Tools Impacting Communication
• Tool can reduce( or add ) the friction

34. References
• Effective devops (O’reilly)
• Test Driven infrastructure with chef (O’reilly)
• Infrastructure as a code (O’reilly)
• 클린 코드 Clean Code ( Acornpub)
• 리눅스 네트워크의 이해 Understanding Linux network
internals (Acornpub)
• Personal Experience( andrew, just joke)