This presentation was given by us at Mongo Munich on 10th of October 2011. It covers the introduction and mostly the durability and robustness testing of MongoDB at AutoScout24 before launching a new site.

1. www.autoscout24.com IntroducingMongoDB in a HA multi-site Environment Munich| 10.10.2011 | Sebastian Geib, Jean-Charles Thomas

2. Seite 2 Jean-Charles Thomas Team Lead Unix Systems and Applications Sebastian Geib Database Administrator | Autoscout24 MongoDB HA environment| Sebastian Geib, J.C. Thomas

3. Index 1 About AutoScout24 2 Why MongoDB? 3 MongoDBarchitectureat AutoScout24 4 TestingMongoDB 5 MongoDBbackup/restore 6 MonitoringMongoDB 7 Conclusion | Autoscout24 MongoDB HA environment| Sebastian Geib, J.C. Thomas Seite 3

4. AutoScout24 Who arewe? 5.4 Mio Users 14 Countries 1.9 Mio Cars | Autoscout24 MongoDB HA environment| Sebastian Geib, J.C. Thomas Seite 4

5. AutoScout24 KPI Numbers, Data and Facts fortheoverall Autoscout24 IT Two seperated Data Centers in Germany >1000 Servers, 10 Loadbalancer, 25 Firewalls, 60 development servers 16 Storagesystemes with raw capacity of 800TB 11,2 Mrd. Total Requests (PI + Grabber and Bots) / Month 58 Mio. Image files for 1,9 Mio. Cars 2,1 Gbit/sec Peak Traffic 180TB Data Volume / Month Four Broadband Provider with in Total 13GBit/sec Seite 5 Internet AS44355 Global Traffic Mgmt. Loadbalancer Loadbalancer Firewall Firewall Loadbalancer Loadbalancer Backbone Router Backbone Router Applicationsserver Applicationsserver Database server Database server DC 2 DC 1 | Autoscout24 MongoDB HA environment| Sebastian Geib, J.C. Thomas

6. Index 1 About AutoScout24 2 WhyMongoDB 3 MongoDBarchitectureat AutoScout24 4 TestingMongoDB 5 MongoDBbackup/restore 6 MonitoringMongoDB 7 Conclusion | Autoscout24 MongoDB HA environment| Sebastian Geib, J.C. Thomas Seite 6

7. Why MongoDB at AutoScout24? New Product 2011: Portal for car inspection and services Complete new application development from scratch for the Front- and Back-ends Let‘s use what we dreamed! Initial Database Requirements Scale for large quantity of data Highly available across Data Centers Flexible database changes (avoid the DBAs as much as possible!) MapReduce functions Easy management MongoDB was choosen as the best Product Product Launch was September 2011 Seite 7 http://werkstatt.autoscout24.de/ | Autoscout24 MongoDB HA environment| Sebastian Geib, J.C. Thomas

8. Index 1 About AutoScout24 2 WhyMongoDB 3 MongoDBarchitectureat AutoScout24 4 TestingMongoDB 5 MongoDBbackup/restore 6 MonitoringMongoDB 7 Conclusion | Autoscout24 MongoDB HA environment| Sebastian Geib, J.C. Thomas Seite 8

9. Mongo Architecture Seite 8 | Autoscout24 MongoDB HA environment| Sebastian Geib, J.C. Thomas

10. Mongo Architecture Seite 9 Replica set across two data centers: primary and secondary. All four nodes are actively used by the application. Primary data center split in two fire areas. In the primary data center, both primary and secondary nodes can assume the role of a primary automatically. In the secondary data center, both secondary nodes can only be manually promoted to become a primary to avoid split brain situations. Currently running MongoDB1.8.1 All servers virtualized using Vmware ESX 4.1 2 Cores, 4 GB RAM, 100 GB HDD per server | Autoscout24 MongoDB HA environment| Sebastian Geib, J.C. Thomas

11. Index 1 About AutoScout24 2 WhyMongoDB 3 MongoDBarchitectureat AutoScout24 4 TestingMongoDB 5 MongoDBbackup/restore 6 MonitoringMongoDB 7 Conclusion | Autoscout24 MongoDB HA environment| Sebastian Geib, J.C. Thomas Seite 10

15. The failover takes 13 seconds with safe mode enabled.

16. After reboot the former primary becomes a working secondary.| Autoscout24 MongoDB HA environment| Sebastian Geib, J.C. Thomas

18. Majority remains intact and replica set is working properly.| Autoscout24 MongoDB HA environment| Sebastian Geib, J.C. Thomas

20. Cluster remains without primary.

22. Replica set still working fine due to majority being in place.| Autoscout24 MongoDB HA environment| Sebastian Geib, J.C. Thomas

23. Index 1 About AutoScout24 2 WhyMongoDB 3 MongoDBarchitectureat AutoScout24 4 TestingMongoDB 5 MongoDBbackup/restore 6 MonitoringMongoDB 7 Conclusion | Autoscout24 MongoDB HA environment| Sebastian Geib, J.C. Thomas Seite 15

25. Working fine. Restore a bitmorecomplicated.

26. Dump

27. Easiertorestoreandtoextractspecificdatafrom a restore.Forourcurrentdatavolumemongodumpisthebestchoiceforus. Locking an issue (verifyyourlocksarereleased after backuporyou‘llbe in trouble). | Autoscout24 MongoDB HA environment| Sebastian Geib, J.C. Thomas

29. With a restore in three 40 minute chunks a restore was possible

31. It took 30 minutes to get it back up and running| Autoscout24 MongoDB HA environment| Sebastian Geib, J.C. Thomas

32. Index 1 About AutoScout24 2 WhyMongoDB 3 MongoDBarchitectureat AutoScout24 4 TestingMongoDB 5 MongoDBbackup/restore 6 MonitoringMongoDB 7 Conclusion | Autoscout24 MongoDB HA environment| Sebastian Geib, J.C. Thomas Seite 18

34. Nagios-based

36. CPU, Load, Memory, Network, I/O, Disks

37. MongoDB Specific: Commands, Connections, Replica Set State, Flushing, Locking, Memory Consumption, Data file size| Autoscout24 MongoDB HA environment| Sebastian Geib, J.C. Thomas

39. XML-driven

40. Windows-based which needs to make heavy use of Cygwin to watch Linux servers.

41. Integrates with AutoScout24 platform monitoring.

43. CPU, Load, Memory, Network, I/O, Swap, Disks

44. MongoDB Specific: Availability, Commands, Connections, Replica Set State, Flushing, Locking, Memory Consumption| Autoscout24 MongoDB HA environment| Sebastian Geib, J.C. Thomas

45. Index 1 About AutoScout24 2 WhyMongoDB 3 MongoDBarchitectureat AutoScout24 4 TestingMongoDB 5 MongoDBbackup/restore 6 MonitoringMongoDB 7 Conclusion | Autoscout24 MongoDB HA environment| Sebastian Geib, J.C. Thomas Seite 21

46. MongoDB Conclusion What‘sbeenimportantforus Seite 22 Conclusion: As longasreplicasetsareworkingfinetheyaregreat. Watch theirhealth and youwon‘tgetintotrouble. Overall robustnesscouldbefurtherimprovedwithbettererrorhandling and reportingfromtheMongoDBserver. C# driverneedssomefurthertweakingtoavoidaccesses on arbiters. The currentreleasefixes this but hasn‘tbeenintroduced in productionyet. Whenourprimarydatacenteris down, noprimarycanbeelected in thesecondarydatacenter due tomissingmajority. This was our design choicetohavebettercontroloverprimaryelection. Permissionsneedtobeset in a moreatomicfashion. Most ofourteammembersarecomingfrom an Oracle background so theyareexpecting a lot. | Autoscout24 MongoDB HA environment| Sebastian Geib, J.C. Thomas

47. MongoDB Outlook Whatwearelookingforwardto Seite 23 Outlook: MongoDB2.0 looksreallypromising forus. Wearecurrentlywaitingforthefirstbugfixrelease and will thenstartourtesting. Improveddatacenterawareness a bigwinforus. Replicasetconfigurationwithminority in placeisreallyusefulforourfailoverscenario. | Autoscout24 MongoDB HA environment| Sebastian Geib, J.C. Thomas

48. Seite 24 Questions? jcthomas@autoscout24.de sgeib@autoscout24.de Looking for a great job as DBA in one of the largest internet companies in Europe? Great! We are looking to hire DBAs. Have a look on our homepage or contact us directly. | Autoscout24 MongoDB HA environment| Sebastian Geib, J.C. Thomas