Continuous Localisation On A Massive Scale
•
0 recomendaciones•7 vistas
A crisis is brewing. There are hundreds of products to localise, and this number is growing rapidly. How does a small and efficient localisation team cope with increasing demand while keeping costs low and improving quality? In this session, Gary describes how an engineering team achieved this by taking automation to the next level with continuous localisation. You’ll get a technical deep dive into tools and techniques used to achieve near instantaneous translation in the hybrid cloud. This is a journey where you will learn about the challenges endured and overcome in the ballsy determination to localise an unlimited number of products.
Recomendados
Recomendados
Más contenido relacionado
Similar a Continuous Localisation On A Massive Scale
Similar a Continuous Localisation On A Massive Scale (20)
Último
Último (20)
Continuous Localisation On A Massive Scale
- 1. Continuous Localisation On a massive scale Gary Lefman Globalisation Architect LocWorld 47 Berlin 2022
- 2. Coming up… 01 Motivation What on earth possessed us? 02 Architecture We must be insane to handle so many variables! 03 Benefits Surely localisation is better, now?
- 4. Automation of the entire end-to-end localisation process with an uninterrupted platform for an unlimited number of products, achieving speed, accuracy, at low cost. Unavoidable scalability issues with a small, albeit efficient, localisation team facing an ever-growing portfolio of responsibility and challenging resource constraints. The motivation The solution
- 6. Collaboration hardware, firmware, and software. We Localise
- 9. Test service triggers vendor generated regression test tools. Webhook proxy enables a secure connection with third-party tools. Build service triggers independent in-house build tools. Core service connects development teams and resources. Four essential autonomous parts. Additional services enhance the continuous localisation platform. Components
- 10. Singleton Database and translation management systems. Observer Core, build and test services. Proxy Web hook proxy service and bots. Adaptor Build and test tools integration. Rich open-source libraries that get us there quickly. Strong in development operations and machine learning. API first and test-driven ethos. 100% Python
- 11. Microservices enable rapid, frequent, and reliable delivery of large, complex applications.* - Loosely coupled - Independently deployable - Organised around business capabilities - Owned by a small team Hybrid cloud is a secure integration of private and public cloud environments. * Definition: https://microservices.io Hybrid Cloud Microservices OpenShift Amazon Web Services Docker and Kubernetes API Gateway and Lambda
- 12. Creates and maintains localisation projects. - Source and target file changes trigger events - Handles communications Core Communication Tools Translation Management Systems Repository Service Core Service Notification Service
- 14. Handles incoming third-party webhooks from services outside the enterprise network. - Highly secure public access endpoints - Recycles API tokens - Discards unauthorised requests - Load balanced across multiple regions Webhook Proxy Webex Translation Management Systems Core Service Webhook Proxy Service
- 16. Automates creation of language packs and locale installers after localised files are updated. - Core service triggers a build event - Some products have custom build tools - Build storage service manages compilations Build Core Service Build Service Build Tool Build Tool Build Storage Service
- 17. Build Tool Build Tool Build Django Web User Interface Build Service API Locale Installer Test Tool Language Pack Tester Artifactory Build Storage Service API
- 18. Automates localisation regression testing after a language pack, locale installer, or product with integrated locales is built. - Localisation vendor maintains test tools - Test tools download locales from the build storage service Test Build Notification Service Core Service Test Service Test Tool Test Tool Build Storage Service
- 19. Build Notification Service API Test Tool Test Tool Test Test Service API Build Storage Service API Product Build Product Repository Locale Installer Product Build Tool
- 20. Overall Architecture Repository Service Repositories Webhook Proxy Service Build Service Build Tools Build Storage Service Build Notification Service Test Service Test Tools Third-Party Tools Core Service Notification Service
- 21. Need to know state of the platform constantly. - What is down? - Why did it go wrong? - Where are potential problems? - When are the bottlenecks happening? Custom node exporter for ephemeral tasks. Observability Metrics Logs Traces Prometheus Loki Tempo Grafana Coming soon
- 23. 0 1 2 3 4 5 6 2022-03-06 2022-03-07 2022-03-08 2022-03-09 Product A Product B Product C Modelling commit patterns. Predictive analysis: rate and velocity. Optimal day and time to start translations. Gives control back to programme managers. Commit Log Commits
- 24. Monday Tuesday Wednesday Thursday Friday Saturday Sunday Commit Days 0:00 4:48 9:36 14:24 19:12 0:00 Commit Times (UTC) 0 1 2 3 0 20 40 60 80 100 120 Words Per Commit New Words Changed Words Commit Rate Words Commits Total Commits Peak Commit Rate Total New Words Total Changed Words Mean Time Between Commits Most Frequent Commit Day Least Frequent Commit Day 52 3 399 97 2d 23m 54s Monday Saturday/Sunday M T W T F S S Commit Statistics
- 25. Don’t need a dedicated web application Everyone involved with a project is in the same space All communications and collaboration in one space Adaptive cards enable dynamic interaction with bots Know when developers commit changes, and when localisation is done Immediate access to build and test results
- 26. On-boarding Repository types Repositories Branches Resource files* Projects Products Unlimited scalability thanks to microservices with elastic processors, memory, and storage. Operating 24 x 7 x 365. * 3,666,660 files on 2022-04-23.
- 27. Locating files not stored in repositories. Discovered they are created after the product is compiled. Created a dedicated repository for compiled source files. Some repositories are outside corporate domain. Cloning and polling is resource intensive. Using web hooks and pulling is faster and cheaper. A few branches and pull request templates are changed or deleted without warning. Added a repository monitoring service to alert on unexpected changes. All teams do Agile differently. Some prefer a different translation frequency. Added a bot to provide control over translation readiness. Most on-boarded products have unique challenges. Must be agnostic to technology and human behaviour. Challenges
- 28. Engineer’s credentials were found in unexpected places. Attrition caused services to become inaccessible. Enforced use of generic Active Directory user accounts. Shared IT service maintenance windows occasionally interrupted workflows. Moved services to dedicated instances that we control. Used ngrok as a tunnelling protocol in the webhook proxy service, but its use was eventually banned, even in development. Replaced with a secure hybrid cloud environment. Jenkins was intended to be the core engine, but several limitations made this impossible. Created independent core and repository services. Tools and services beyond our control had to be worked around. We had to find or create new ways to maintain accessibility and security. However, we never wavered from the vision of continuous localisation. Challenges
- 29. 03 Benefits
- 30. We can achieve faster localisation… Annually with fewer hours wasted on manual tasks… while reducing operational costs* * Estimate (USD)
- 31. What’s Next? Localisation as a service for development teams to subscribe to and manage. Administration interface simplifies on-boarding and version transitions. Webex Assistant integration for voice control of projects. Artificial intelligence helps bots with natural language processing.
- 32. Ask me anything
Notas del editor
- A small and efficient localisation team in the Collaboration engineering organisation, distributed globally, timing is critical. Lots of products, merger with another localisation team, acquisitions, and organic growth. How will we cope? By 2014, industry talk of “Agile localisation” dried up. I was toying with the idea of CI/CD for localisation in 2016 and might have been the first to use that term at LocWorld Dublin.
- The scale of our challenge. The Collaboration organisation is the largest software group at Cisco.
- We localise all digital devices in these images… and all the services behind them that you don’t see.
- What we did and how we did it. Technical architecture and implementation. Complications and technical challenges.
- Each component of the continuous localisation platform works independently of the other components.
- API first. Test-driven. CI/CD to perform regression testing and deployment to DEV > STAGE > PROD.
- Microservices Small bits of code that wake up, perform a function, and are then destroyed. Incredibly fast and cheap to operate. Colour scheme Pink = Private cloud Yellow = Public cloud Blue = Third-party services Green = Independent internal tools
- Core service coordinates localisation projects. Repository service handles developer and localisation repositories. Notification service sends messages to communication tools.
- Developers commit resource files to their repositories. The repositories send webhooks to Jenkins repository listeners, which triggers the repository service. The repository service API triggers the core service to create a new localisation project and prepares translation memory. The state machine maintains the project state. The core is also responsible for passing a localisation project through pre- and post-localisation quality assurance filters. Project transitions are sent to the notifications service, which queues Webex and/or e-mail messages for recipients. Microsoft Adaptive Cards are coded into some Webex messages to provide forms that permit end user input. The database maintains all product and version information that is added during on-boarding, and localisation project data.
- Allows us to securely receive webhooks from services outside the organisation.
- TMS webhooks, Webex webhooks, bot service. Avoids network tunnelling.
- Locale installers intelligently apply language, region, and cultural content and configuration to a host. Language packs are not intelligent. They only dump files in a given path.
- Build tools are independent of continuous localisation. Adaptor design pattern provides a common interface for a variety of build tools. Traditional build tools are executed via remote SSH commands. Modern build tools have a REST API and are executed via HTTPS. Build storage service has a web user interface for manual download of builds.
- Test tools are hosted in a secure Extranet lab. Vendor advertises new test tools and command lines to execute tests. Test tools include automated screenshot analysis. Build notification service receives webhooks from product builds containing latest translations.
- Testing is triggered: by the completion of a locale installer/language pack, or by a completed product build with integrated locales.
- Database, state machine, and Jenkins (CI/CD) are excluded from this drawing. The focus is on services (pink and yellow icons).
- Metrics, logs, and traces are all linked by a common time series. This simplifies root cause analysis and speeds up problem resolution.
- Composite of several dashboards. We’re still refining metrics and dashboards to make sure they are meaningful in context.
- Commit logs measure the rate and velocity of commits. Useful in determining if the platform should send to translation immediately or aggregate commits. Programme managers can choose the best day and time to begin translation based on patterns.
- Noticed interesting commit patterns after 18 months of logging that have made aggregating commits for translation easy. Strict merge days (e.g., Tuesday only) is indicative of continuous integration pipelines Strict commit timeframe (e.g., China 09:00-17:00) is indicative of cultural behaviour Some product teams commit all days of the week at any time during a 24-hour period, which indicates instantaneous translation is preferable.
- Benefits of using Webex instead of a web user interface. Everyone in the organisation uses Webex. It keeps all product localisation communications in one room. This image is an adaptation of a real project. The product names have been replaced. Engineering details and participants have been removed.
- Our progress so far…
- Teams No such thing as one size fits all. We have needed to adapt tools and service along the way, even though the overall continuous localisation concept has never changed. Some teams don’t want high frequency rapid commits back to their repository – more work for them (allegedly), so we added a Webex bot to allow them to choose whether their commit is ready for translation or not. This has been well received by everyone. Repositories Bitbucket, GitHub, Gerrit. Some repositories are outside of the corporate domain (via acquisitions). Polling repositories does not scale well. It consumes resources that impacts other teams. Switching to webhooks alleviated this challenge. Cloning repositories is resource intensive. Pulling updates is less demanding. Branches Even though development teams are instructed to let us know if they make repository changes, but they rarely do. Ignoring pull requests causes problems later when future commits are translated. It creates a backlog. Files Two products were known to be challenging because source files are generated from a compiled and running instance.
- Jenkins Believed we could orchestrate everything in Jenkins but limitations such as (x, y, z) made it impossible, so we started to move things out and became less reliant on Jenkins Services controlled by other teams Impact of EngIT and Cisco IT on maintenance windows - Unexpected maintenance and extended maintenance timeframes Needed to move away from this reliance, so we created our own instances that we have better control over. Security Third-party tools and services outside the Cisco domain. We needed a webhook proxy service. Lack of technology to support our needs (used ngrok until SCI was available - it was expected at the beginning of the project but took over a year until it was generally available). Engineer’s credentials in places where they shouldn’t be - We lost some contract engineers along the way, which disrupted the platform because their accounts locked or disappeared.
- The outcome Unlimited Massive scale Benefits Faster localisation Lower costs Quality Kudos What happens next?
- Cost savings are estimated based on the aggregated average cost of human resource time less platform operational costs. <Estimated cost elided> Difficult to measure cost of human resource time, we pay for a few virtual machines, SCI, plus AWS/OpenShift resources, easy to view and manage in a dashboard. Now have time to spend on improving translation quality and linguistic test automation suites.
- A sample of major enhancements and the direction we’re heading.