Cloud-based enterprise software platform for translation management

1. Continuous Deployment
by Smartling, Inc
#SmartlingDevLove

2. Andrey Akselrod
Smartling, Founder & CTO
@chelya
Max Sogin
Director, Software Development

3. What do we do?
● Cloud-based enterprise software platform for
translation management
● Used for websites and dynamic Web
applications, desktop & mobile apps,
business documents, and all kinds of digital
content

4. Clients

5. Smartling Operational Scale
3active data centers: US
East, US West, Ireland
400+servers
~200production
deployments a month
2B+page views/month
84K+metrics monitored
17K+triggers for alerts

6. Tech Team
New York: 20
Dnepropetrovsk, Ukraine: 29
Kiev, Ukraine: 3

7. Technologies
Java
+
Everything you can imagine

8. Continuous Deployment == Continuous
Releases
Continuous Deployment != Continuous
Integration

10. Life prior to Continuous
Deployment

12. ● A lot of code is being released
○ Bugs are pretty much guaranteed
○ It is difficult to understand which change caused
issues when a lot of new code is deployed.
○ It is difficult and sometimes impossible to roll code
back
● Quick fixes sometimes wait weeks to be deployed
● There is pressure to wrap up the feature at all costs
before scheduled deployment. Quality takes a hit.

13. ● No one works during the release day. Everyone
expects a sleepless night.
Can we deploy on Friday night?
|
NO!
|
But we only need…
|
HELL NO!

14. ● User feedback is coming late - after substantial
investment of time and effort into development
● The absence of early feedback increases the load
on planning. Heavy upfront planning makes
development process closer to waterfall rather than
agile
● Feedback results are waiting for the next release
which significantly slows down product
development cycle.

15. OODA Loop

16. Transition period - Team

17. Transition Challenges
● Management support is essential
o Transition will take time and resources
o Complete transition may take 6-8 months
● Team culture will change substantially
o Find champions and make sure you have their
support
o It is better to get support from other departments in
the company by explaining the benefits they get at
the end

18. Transition Challenges
● Excellent reason to create DevOps
● Do not wait until ideal implementation. Start with one
simple component and one button.
● Celebrate every single win

19. What we did
● Folks on the team were passive-aggressive and did not
want to do anything
● We were assuring, discussing and blackmailing the
team before starting Continuous Deployment
● Weekly releases got cancelled
● We were using stickers for every deployment: smiles
and asses. Smile was a successful deployment, ass -
was a deployment with a problem.
● At first folks were afraid of deploying, but eventually
they got used to it.

20. Smiles & Asses ;)

21. Continuous Deployment Rules

22. ● Master (git) or Trunk (svn) is always stable
● Every commit to master (trunk) is deployed to prod asap
● Continuous integration
● Automated testing => where is QA Team?
● Single button deployment
● Monitoring
● Zero downtime software update
● Communication. Every deployment is in chat. Every
deployment should be confirmed. Skype => Slack
● Developer is responsible for every feature end to end

23. QA
● Automation - as much as possible
o unit testing
o integration testing
o front end testing - balance is needed!
● Developers are responsible for manual
testing, QA is not involved!
● QA helps with automation

24. Continuous Integration
● Bamboo or Jenkins
● Every branch is built and tested on every
commit
● Only green builds could be deployed
● Build time is critical (fighting for < 10min)

25. Bamboo

26. CDeploy

27. Slack communication

28. Monitoring
● Examples:
o RPS
o avg resp time & percentile
o error rates
o CPU Utilization
o transaction rates
● Correlate Deployments with Metrics
● Splunk, Zabbix, Graphite, New Relic, Loggly

29. Splunk

30. ● Stack trace by errorid
Speeds up investigation of problems. Easy to communicate. Secure.
● “Log duty”
Keep logs clean, by continuous monitoring and elimination of errors. Noise
in logs makes monitoring difficult.
Splunk

31. New Relic

32. DB Updates
● Use DB version control system. DBMaintain
● Every change to DB is reviewed by two leads
● There is a single button deploy of DB changes
● Two phase DB updates
o Phase 1: new structures, backward compatible
 New columns always nullable, etc.
o Phase 2: cleanup of elements no longer in use

33. Results

34. ● Continuous Deployment becomes addictive
● Results of your work are immediately
available for your users to use. It is pretty
cool.
● Management is happy - there is a constant
flow of new features being deployed. Users
are happy with new features as well.

35. ● Quick feedback and quick iterations lead to a
product that is much better adapted to the
market
● Your competition does not understand what
happened and how you can move so quickly
● Other parts of the company respect your
results
● Near zero employee turnover in the team

36. Discussion
#SmartlingDevLove