Systems Management Forum 2013 (http://ac.nikkeibp.co.jp/nc/smf2013s/) の基調講演資料です。楽天では2012年4月から社内でPaaSを展開しており、その基盤の上でさまざまなサービスが開発、運用されています。講演では、楽天PaaSでのDevOpsの実践方法について具体的に紹介しました。
8. Rakuten Platform as a Service
• Everything should be automated.
• Developer’s IN Rakuten should focus on
business implementation in the latest
technologies.
• Platform as a Service infrastructure for
Developers in Rakuten, Inc.
• Like Google App Engine, Heroku, ..etc
• Abstraction Layer of On-Premise/Off-Premise
プライベート
クラウド?
9. Traditional Service Building Steps
• Too many human to
human workflows
• Too many approvals.
• Too many meetings.
• Too many documents.
Request Servers
Wait for
hardware/VM
creation
Setup Operating
System
Install Software
Setup monitoring
Deploy website
Check / Test
Add to Load
Balancer
...
Developer
Operators
複雑
ステークホルダー多い
申請書たくさん
11. Tricks – DevOps Layering
Service Devs
System Operators
Focus on Stable Infrastructure
Focus on Development
Ask
Response
依頼
12. Tricks – DevOps Layering
RPaaS API Service
Service DevOps
RPaaS DevOps
Focus on Infrastructure Features
Focus on Business App
Ask
Response
Consult
Consult
自動化された標準
一緒に
考える
22. Member Roles Definition
• Objectives:
• Clarify the responsibilities for each member.
• A member has more than one role.
Role In charge of:
Release
Manager
• Conducting the release operations by cooperation with
administrator.
System
Administrator
• Handling Daily Operation.
• Quick Fix on troubles.
• Reporting system stats to Team Leader.
System
Developer
• RPaaS functions from planning to operations.
• Negotiation with Team Leader and Managers to decide the
release line and scope.
Team Leader • Coordination of system developers and related people
• Evaluation for RPaaS.
Manager • Budget Control
• HR related things such as evaluation
“兼任できる”
ロールを明確にする
23. KPI Definition
• Objectives:
• Clarify the mission.
• Evaluate the mission result.
Role In charge of:
System
Administrator
• # of alerts
• Time to solve the alerts
Release
Manager
• # of releases.
• Time to release.
System
Developer
• # of features newly implemented
• # of solved issues
Team Leader • # of launched business services
ロールに目標をつけ、評価する
25. Communication in Code
• Objectives:
• Do not use manual ops.
• Anyone can do any ops
紙だけでコミュニケーションしない
26. Git-flow as Communication Protocol
• Objectives:
• Design & implementations in code repositories, not in meeting
rooms.
• Isolate each feature
feature/*
develop
release/*
master
hotfix/*
Administrator
Release Manager
Developer
Admin/RM
独自のルールは極力作らない
27. Standard Release Procedure
• Objectives:
• Reduce manual operations.
$ cd ~/rpaas-commander
$ cap setup-summary
$ cap setup-role -n
$ cap setup-role
$ cap service-in
手順書 = 動くコード
28. CI for infrastructure
• Objectives:
• Reduce the release tasks of which 80% are ‘task confirmation.
確認作業もコードで指示
29. Rolling Update
• Objectives:
• Easy to rollback
• A/B test for infrastructure
L7 Software LoadBalancer
Virtual Machine (v1)
Virtual Machine (v1)
Virtual Machine (v1)
Virtual Machine (v2)
Virtual Machine (v2)
Virtual Machine (v2)
アップデートではなく新規構築
30. Eat our own dog food
• Objectives:
• Stands on customer side.
• Understand pros and cons for our weapons.
自分の道具を自分で使う
31. Off topic?: Englishnization
• Team members from 7 countries
• Finding Specialists in the world
• Source Code / Diagram > English?
33. Key Factors for DevOps
• Everyone should have ‘developer role’.
• Tool is a tool.
• Everything should be automated.
34. Key Factors for DevOps
• Everyone should have ‘administration role’.
• Tool is a tool.
• Everything should be automated.
35. Everyone should have ‘developer role’
• De-facto standard communication method.
• Read source
• Find a bug
• Fix the bug
• Create a new application.
#!/usr/bin/env ruby
puts “Hello Developers!”
ソースコードいじりましょう
36. Tool is a tool
• Chef does not solve the barrier between Devs and Ops.
@@ -1,4 +1,4 @@
-package "apache" do
+package "nginx" do
action :install
end
I wanna use Nginx! Unacceptable.
Pull Request
Chef 使ってるかどうかは
問題ではない
37. Everything should be automated
• Everything / Uncompromising
• Automate after manual operations
“手作業” を使っていいのは1ヶ月に1度
手順書が必要な作業を実施したら、
自動化するまでが仕事