Learn how to make development and operations work together while using Magento. This session will explore Magento dev-ops community resources, leading practices, common tools, and software. A quick look at server choices under today’s cloud environments will illustrate how to simplify the whole process while developing or deploying with Magento 1 or Magento 2.
3. Todays Plan
• Foundations: An overview of Magento Development and
Operations.
• Team: Discussing the culture of processes, technology and people
that are found throughout the Devops Lifecycle.
• Building Blocks: Examining the moving parts that make up
Devops.
• Creating a Blueprint: A brief look into the setting up Devops.
5. What is Devops?
“Devops is the practice of operations and development engineers participating
together in the entire service lifecycle, from design through the development
process to production support.”
From: http://bit.ly/1SVOYGU
6. Devops Goals
• Release on demand
• Prepare technology and processes
• Infrastructure as code
• Eliminate technical debt
• Fail without fear
• Measure and document everything
• Provide a knowledge and cultural feedback loop
7. Devops Foundation: CAMS Cycle
Term coined by John Willis in 2010
• http://bit.ly/20cIRRq
• This is a feedback loop which can be
built around simple building blocks
• Produces incremental growth of the
Devops culture, which creates more
automation, monitoring and sharing
8. CAMS: Culture
“People and process first. If you don’t have
culture, all automation attempts will be fruitless.”
9. CAMS: Automation
“Start to stitch together an automation fabric for
Devops. Tools for release management,
provisioning, configuration management, systems
integration, monitoring and control, and
orchestration become important pieces in building
a Devops fabric.”
10. CAMS: Measurement
“If you can’t measure, you can’t improve. A
successful Devops implementation will measure
everything it can as often as it can… performance
metrics, process metrics, and even people
metrics.”
11. CAMS: Sharing
“Sharing is the loopback in the CAMS cycle.
Creating a culture where people share ideas and
problems is critical.”
12. Team Devops
Discussing the processes, technology and
people that are found throughout the Devops
Lifecycle.
13. Team Devops
Each Devops role has people, processes and technologies that are
used to deliver the end product.
14. Devops Individual Roles
Roles may be more individualized for some of your “Star Players”
• Automation Architect
• Release Manager
• Security Engineer
• Experience/Quality Assurance
• Disruptive Technology Research Technician
15. Business / Operations
Business
• Sales
• Support
• Administration
• Business Operations
• Business Snowflake Logic
Operations
• “The Store” and any hardware or
software that is providing it
• Support for customers,
administrators and other users
• Secondary and tertiary
applications that support “The
Store”
16. Development / Engineering
• People who write and test code
• Tools that are used for Development
• Methods and Practices that are encouraged
• The development and testing infrastructure
17. Testing / Monitoring
• A wide variety of technologies and tools exist for testing
• Monitoring can turn up things tests missed
• Responding to monitoring is a must for it to be effective
• Testing and Monitoring should be throughout the Devops lifecycle
18. Customers / End Users
• Users of the store/website including administrators and customers
• Any service that consumes data from the website
• Processes that are related to “User Experience”
• Feedback from this group is critical to operational success
23. Infrastructure and Configuration as Code
Automated builds using Vagrant and Terraform are a great way to treat
infrastructure as code.
Configuration Management Tools:
• Ansible
• Chef
• Puppet
• Salt Stack
• And many others…
25. AWS Continuous Full Coverage
http://fbrnc.net/blog/2016/03/continuous-load-testing
26. Continuous Integration and Delivery Tools
Automated build, deploy, test and release.
CI/CD Tools that work well with Magento
• Amazon CodePipeline
• Bamboo
• Jenkins
• TravisCI
• And many others…
27. Continuous Testing
* Testing is a broad subject, deserving of more attention.
Continuous Integration tools like bamboo provides hooks to allow you to
test using several different methods.
Magento 2: Testing is built in and easy to tie into your CI/CD
Magento 1: More difficult, it is possible with some effort
29. Continuous Stats
See metrics from all of your apps,
tools & services in one place using
Datadog.
This can be tied to a variety of
inputs and outputs:
• Newrelic
• AWS Cloudwatch
• Slack
• VictorOps / Pagerduty
• And Many More
31. Logs
With the constant fragmentation of the infrastructure you end up with
some pretty incredible logging requirements.
Data sources of interest:
• Infrastructure metrics (CPU, Network IO)
• Business metrics (Sign ups, logins, placed orders)
• Events (deployments, auto-scaling activity)
• Test metrics (response times, error rates)
33. “The Devops Plan”
Define, Document & Empower Devops Culture
1. Identify and Document Processes
2. Find people within the Roles
3. Focus on building the foundation for feedback loop
Start Automating
1. Application build process
2. Development environment builds
3. UAT environments
4. Production environments
Build in monitoring, metrics and create feedback loop.
34. Small Simple Building Blocks
• Start with the git workflow
• Application build process
• Testing and peer review
• Release build process
• Automate production deployments
• Use operations and monitoring to
plan the next release
35. Keep it simple.
With so many choices for all of the technology, its important to focus on
small solutions that fit within your existing infrastructure that are easy to
implement and maintain.
37. Thanks!
• My Family
• Robofirm
• Mage Titans
• The Magento Community
• Fabrizio Branca
The people who have contributed
to the meta-culture that inspired
this talk.
Follow me on twitter:
@aepod
See my blog at http://aepod.com/
for a more detailed list of sources
and inspiration for this talk.
Notas del editor
Devops is a goal centric practice based on culture.
It’s a practice that builds upon itself over time
continuous improvement based on real measurements
A planned response, a reaction not just a plan
Devops is goal based on incremental improvements through a few key goals
Lack of Devops Problems
Maintaining unique snowflakes can be very difficult.
Low confidence in build and deploy
Failure is frustrating and fruitless
Infrastructure and Operations are Out of Scope
Configurations tend to drift over time
Lack of responsibility for critical parts of infrastructure
Training and support is difficult because processes are unique to the people working on the tasks
“There is a increase in feedback loops between business, all parts of the delivery process and operations… thanks to this feedback loops we increase the quality and speed up the flow”
- Patrick Debois (December 2009)
Without a strongly defined culture it’s near impossible to automate or measure or even share the results.
Magento has a strong Devops meta-culture, with many people participating in the discussion.
People, processes and technology
Work environment is the stage
Inclusive across the entire organization or product
Develop cultural growth:
Disrupt roles by pairing across disciplines
Document processes, actually document everything
“Build it and they will come” – You get the best people
Automation requires a Devops culture
Use simple tools, woven into an automation fabric
Automate one small piece at a time
Reducing human effort through automation will mean:
an increase in frequency and reliability of releases
simplified integrations for monitoring
less truck factor in your business
Define metrics and standards
Measure everything/anything as you can
Utilize tools to store, aggregate and visualize the data
Measurement and Testing is critical to Devops growth
It’s the foundation for sharing the information in the CAMS cycle
Sharing provides a direct feedback loop for Devops Culture
Share success and failure stories with others
Contribute to Open Source tools and solutions
Get involved in the community
Build a training program for clients and employees
Business / Owners
Development / Systems Engineering
Testing / Monitoring
Operations / “The Store”
Customers / End users
Devops will affect change throughout the entire operation, and the team roles help sort out how, who and what will be affected.
Devops functions best when roles are transparent
The Role groups are just examples of the potential boundaries
Integrating Devops into a existing process can be disruptive
Key players will have a more insight into how things work together
Business Includes people and processes from multiple companies
Agencies
Hosting
Store Owners
Licensed Software Support (Magento, Redhat, etc)
-----
Devops operations teams should focus on
Planned and Emergency Responses
Timely resolution of production issues
Horizontal visibility of production issues
Includes Systems Engineers and others such as
Senior Developers
System Architects
Support at hosting companies, agencies and other places that get involved in creating the website
A wide variety of technologies and tools exist for testing
Frontend: jMeter, gatling, Selenium and others
Code: Functional and Regression testing
Monitoring can turn up things tests missed
syslogd, snmp, newrelic and a ton of other tools exist to help this
Responding to monitoring is a must for it to be effective
Testing and Monitoring should be found throughout the Devops lifecycle
Devops brings all the separated groups together, acting like a hub and filling the gaps
Environments
Development: Individualized local boxes for developers. These can be Virtual Machines, containers, cloud boxes or even just a local OS. These are the only boxes where people “edit code”.
Testing: Build tests, user acceptance tests, and integration tests for production. Leading practice has this match your production and emulate any integrations the production site depends on. It is extremely beneficial to have branching setup so releases can be tested concurrently.
Production: Blue/Green deployed production site. Immutable infrastructure and code, and best if fully automated.
Devops Lifecycle
Using AWS as your Infrastructure
Continuous Toolsets
Continuous Toolsets:
Integration and Deployments
Testing
Monitoring
Metrics
Logs
Vagrant for local virtual machines
Terraform manages cloud resources:
Physical Machines
Virtual Machines (Cloud)
Network Switches and Firewalls
Containers
Plan/Test before you execute
Simple shell script and configuration, easy to use.
Amazon Web Services
Fast, reliable, scalable and amazing
Typical Services used:
EC2 Containers: These are your “boxes”
RDS: Relational Database service
ElastiCache: Redis & Memcache service
S3 Bucket: Scalable storage in the cloud
Route53: Rock solid DNS
Additional Services:
Elastic Load Balancing & Auto Scaling
Lambda: Noops response based computing
CloudWatch & CloudTrail: Monitoring and log collectioN
As Fabrizio says: Just stick Continuous in front of anything and it sounds cool.
“Continuous Integration doesn’t get rid of bugs, but it does make them dramatically easier to find and remove.”
- Martin Fowler, Chief Scientist, ThoughtWorks
M2: Performance, Selenium & Unit Tests all provided by the Magento Testing Framework (MTF)
M1: Performance Tests: Magento Performance Toolkit
Selenium Tests: Magiumlib can be utilized
Unit Tests: Typically using the EcomDev_PHPUnit tests.
SNMP: Simple Network Monitoring Protocol
Pagerduty: Endpoint for Alerts
Ties back to Jira automatically creating tickets on issues
Newrelic:
Real-time insight into performance across application and infrastructure
Allows for incredible horizontal visibility into metrics
Can be configured to collect a lot of application specific data
Has a robust API that can tie to other systems
Using ELK can really help sort this out, SaaS provider logz.io is a great place to start.
Continuous Load Testing and Monitoring: http://fbrnc.net/blog/2016/03/continuous-load-testing
Dev infrastructure builds first (simple and flat one per box)
UAT and staging environments (add branching and other challenges)
Production Environments (Autoscaling and other concerns are primary)
Rock Solid Magento Development – By Fabrizio Branca is a great way to get started with your workflow
The biggest downside of the Devops toolbox is that it’s has to be a bit of a Pandora's box, we are responsible for bringing in new tools to almost every aspect of the projects we work on.
Start small, work through the pieces one at a time and weaving each process into the devops fabric.