Learn how GerritHub tackled and resolved the issues of migrating from Gerrit v2.14 to v2.15 with zero downtime and ping-pong availability zone swap

1. 0Gerrit User Summit 2018 – Palo Alto CA GerritForge.com 0
Migrating to Gerrit 2.15
A success story by GerritHub.io
Luca Milanesio
Gerrit Code Review Maintainer
GerritForge

2. 1Gerrit User Summit 2018 – Palo Alto CA GerritForge.com 1
About GerritForge
Founded in the
UK
HQ in London Committed to
OpenSource

3. 2Gerrit User Summit 2018 – Palo Alto CA GerritForge.com 2
About GerritHub.io
• 5 years of activity
• 37k Gerrit projects hosted
• 16k monthly active users
• 300+ organizations
• 99.9% availability

4. 3Gerrit User Summit 2018 – Palo Alto CA GerritForge.com 3
Multi-master / dual-site Gerrit setup
OVH - Canada Hetzner - Germany

5. 4Gerrit User Summit 2018 – Palo Alto CA GerritForge.com 4
Challenges when upgrading to 2.15
1.Zero-downtime
2.Accounts migration
3.NoteDb migration
4.Replication to GitHub

6. 5Gerrit User Summit 2018 – Palo Alto CA GerritForge.com 5
Challenge #1 zero downtime
OVH – Canada (v2.14) Hetzner – Germany
(v2.14 è v2.15 online reindex)
Incoming traffic

7. 6Gerrit User Summit 2018 – Palo Alto CA GerritForge.com 6
Challenge #1 zero downtime
OVH – Canada (v2.14)
+ readonly plugin
Hetzner – Germany
(reinit v2.15)
Incoming traffic
replication

8. 7Gerrit User Summit 2018 – Palo Alto CA GerritForge.com 7
Challenge #1 zero downtime
OVH – Canada (v2.14)
+ readonly plugin
Hetzner – Germany
(delta-reindex v2.15)
Incoming traffic
replication

9. 8Gerrit User Summit 2018 – Palo Alto CA GerritForge.com 8
Challenge #1 zero downtime
OVH – Canada
(v2.14 è v2.15)
Hetzner – Germany
(v2.15 on ReviewDb)
Incoming traffic

10. 9Gerrit User Summit 2018 – Palo Alto CA GerritForge.com 9
Challenge #1 zero downtime
OVH – Canada
(v2.15 delta reindex)
Hetzner – Germany
(v2.15 on ReviewDb)
Incoming traffic
replication

11. 10Gerrit User Summit 2018 – Palo Alto CA GerritForge.com 10
Challenge #2 accounts migration
Accounts table (DB)
All-Users (repo)
accounts index (lucene)
v2.14
All-Users (repo)
accounts index (lucene)
v2.15
• All-Users getting fragmented quickly è open files explosion
• accounts index misalignments in v2.14 (unnoticed)

12. 11Gerrit User Summit 2018 – Palo Alto CA GerritForge.com 11
Challenge #3 NoteDb migration
OVH – Canada
(v2.15 / ReviewDb)
Hetzner – Germany
(v2.15 ReviewDb è NoteDb)
Incoming traffic
replication

13. 12Gerrit User Summit 2018 – Palo Alto CA GerritForge.com 12
Challenge #3 NoteDb migration
OVH – Canada
(v2.15 restart with NoteDb cfg)
Hetzner – Germany
(v2.15 NoteDb)
Incoming traffic
replication

14. 13Gerrit User Summit 2018 – Palo Alto CA GerritForge.com 13
Challenge #3 NoteDb migration
OVH – Canada
(v2.15 NoteDb)
Hetzner – Germany
(v2.15 NoteDb + readonly)
Incoming traffic
replication

15. 14Gerrit User Summit 2018 – Palo Alto CA GerritForge.com 14
Challenge #3 NoteDb migration
OVH – Canada
(v2.15 NoteDb)
Hetzner – Germany
(v2.15 NoteDb)
Incoming traffic
replication

16. 15Gerrit User Summit 2018 – Palo Alto CA GerritForge.com 15
Challenge #4 replication to GitHub
NoteDb = refs (w/ ReviewDb) + Num (changes)
Git push to GitHub = HTTP Status 500 (boom !)

17. 16Gerrit User Summit 2018 – Palo Alto CA GerritForge.com 16
Poll: did you have similar problems?
Image from: http://cypp.rutgers.edu/ru-voting/political-information/public-opinion-polls/

18. 17Gerrit User Summit 2018 – Palo Alto CA GerritForge.com 17
Wants to know more?
GerritForge.com/contact