1. Nubank is the largest digital bank in Latin America and is expanding internationally, which presents challenges in infrastructure, systems, and teams. 2. The initial international expansion involved manual setup of AWS accounts in other countries while the infrastructure was refactored to better support multiple regions. 3. The core systems also needed refactoring to separate country-specific from generic functionality to allow independent evolution in each market. 4. Team structures transitioned from single to distributed teams across countries, with most work done locally but platform teams providing shared capabilities.

1. Challenges to
internationalize
the largest
digital bank of
the world
O’Reilly Software Architecture Conference, New York, NY
February 26, 2020

2. Principal Software Engineer
@lucascs
Staff Software Engineer
@luiz_hespanha
@nubank
Who are we?

3. Nubank is the leading fintech in
Latin America, born to eliminate
complexity and empower
customers to take control of
their money.

4. Credit Card 2014
Account 2017
Lending 2019

5. GROWING RAPIDLY IN A COMPLEX DOMAIN
Engineers
500
Kafka Messages/day
1B
Microservices
350
Customers
20M
2014 2020
Millionsofcustomers
Teams
80
Deploys/day
10s
Shards
16
HTTP Requests/day
1B

6. Late 2018

7. September, 2019

8. Why talk about i18n?
WHY

9. Agenda
1
INFRASTRUCTURE JOURNEY
3
TEAM BUILDING JOURNEY
2
SYSTEM JOURNEY
4
CONCLUSION

10. Infrastructure
Journey

11. BACK IN 2014
Ruby Scripts and Cloud
Formation
Microservices, continuous delivery, immutable
infrastructure since the beginning

12. BACK IN 2015
Ruby Scripts and Cloud
Formation with Docker

13. BACK IN 2016
Infrastructure sharding
Whole infrastructure, per customer, not just the db
Copies of infrastructure with a global shard to do
customer agnostic operations or routing

14. SCALABILITY UNITS + GLOBAL ROUTING
SERVICE 1 SERVICE 2
SERVICE 3
SERVICE 1 SERVICE 2
SERVICE 3
shard S1 shard s2
SERVICE 4 SERVICE 5
global
SERVICE 6
purchase deposit
SERVICE 1 SERVICE 2
SERVICE 3
shard S0

15. BACK IN 2017
Ops automation
Clojure deployment scripts
Infrastructure EDN definition repository
NuCLI
Real time Monitoring

18. BACK IN 2018
Kubernetes
Better Infrastructure Definition repository
New shards happening too often

19. BACK IN 2019
Stability
16 shards, better monitoring, Prometheus,
Grafana, SLIs, SLOs, OpsGenie

21. DECISION
One AWS account per country
vs Single AWS Account

22. CHALLENGES
Kubernetes is not magic
It’s not only deploy in another AWS account

23. CHALLENGES
Legacy Ruby Scripts vs
Clojure Scripts
Resources that we only had to create once:
Network setup, memcached clusters, Zookeeper,
etc

24. CHALLENGES
Everything automated but lots
of BR specific scripts
Pointing directly to specific buckets, specific
resources, etc
Add br/ prefix in everything

25. CHALLENGES
Dealing with multiple AWS
credentials
NuCLI

26. CHALLENGES
Security keys and certificates
IAM, scopes and users
Global identity system

27. CHALLENGES
ETL
Aggregating data from multiple countries

28. Hack to launch Mexico while building a better
structure that will solve for any country
Manual AWS account setup took several weeks to run
Huge empirical work to figure out what was not properly i18n’ed

29. System
Journey

30. BACK IN 2014
Started as a front-end to
a Credit Card processor

31. BACK IN 2015-2016
Full Credit Card
processor in-house
Most of it reverse engineering the other system
Learned the domain on the go

32. BACK IN 2017
Multiple products: Credit
card and Checking
account
Redesign the app to support multiple products
Allow customers without Credit Card

33. BACK IN 2018-19
Rapid growth
Most of the development was to support it
All the corner cases happened
Tech debt only accruing interest
Optimization of individual flows

35. DECISION
Front-end of 3rd party again,
or adapt the current system?

36. CHALLENGES
Find the differences of the
Credit card product in MX
Credit card looks almost the same in all countries.
We didn’t know what was different

37. CHALLENGES
Obvious differences like:
Timezone, currency,
translations, id docs

38. SOLUTION
If locale = Brazil -> x
If locale = Mexico -> y
...

39. CHALLENGES
Not so obvious differences
like: Revolving credit
calculations, 3rd party
integrations, payment
integrations

40. SOLUTION
Microservices: replacing
some services by a different
one for MX, following the
same contract

41. CHALLENGES
Products need to be able to
evolve independently

42. SOLUTION

43. Platform services are generic
Product services are country specific
Good even for new products in Brazil

44. Team
Journey

45. BACK IN 2014
One Single team

46. BACK IN 2015-2016
Squads (Spotify Model)
Theme focused

47. BACK IN 2017
Packs (Project focused
teams)

48. BACK IN 2018
Tribes
Business Units
Groups of Squads

49. BACK IN 2019
Distributed teams
Teams in Berlin, São Paulo, CDMX
90% of the team in São Paulo

51. DECISION
Do all in BR or have a MX/i18n
team?

52. WRONG ASSUMPTION
Create a new i18n team that
would do all work and no
other team would be affected

53. PAINS
Onboarding was tough: not
enough documentation,
hidden BR product rules

54. I18n teams will mostly build product services,
and sometimes help growing the platforms

55. Conclusion

56. Infrastructure
Journey
Hack1 Launch2
Refactor to best
architecture
3

57. System Journey
Hack1 Launch2
Refactor to best
architecture
3

58. Team Journey
Hack1 Launch2
Refactor to best
team organization
3

59. CONCLUSION
If there is no
refactor, 3rd, 4th
countries should
be way harder
If the system was not built with i18n in mind, chances are
that it won’t be easy to do