SlideShare una empresa de Scribd logo
1 de 32
Descargar para leer sin conexión
High-Availability MySQL with DR:BD and Heartbeat: MTV Japan mobile services




©2008 MTV Networks Japan K.K.
Patrick Bolduan
                                      patrick.bolduan@mtvn.jp




                                   System Manager
                                MTV Networks Japan K.K.




©2008 MTV Networks Japan K.K.
Let’s get started


©2008 MTV Networks Japan K.K.
About MTVNJ
                                Company
                                Profile
                                Company Name    MTV Networks Japan K.K.


                                President      Peter Robert Bullard


                                Address        2-8-2 Jingumae, Shibuya-ku, Tokyo 150-0001
                                               TEL: 03-6434-3111 FAX: 03-6434-3150

                                Business       MTV Networks Japan K.K. (MTVNJ) operates the mobile and PC
                                Description    digital media services for all MTV Networks brands including MTV and
                                               Nickelodeon in Japan. In addition to digital media services, MTVNJ
                                               also operates a 24-hour Nickelodeon channel on CS/CATV, and the
                                               consumer products business for Nickelodeon's character franchises in
                                               Japan.




©2008 MTV Networks Japan K.K.
MTVNJ Mobile Brands




©2008 MTV Networks Japan K.K.
MTV Flux/MTV Mobile DB project intro




©2008 MTV Networks Japan K.K.
MTV Flux/MTV Mobile DB project
                                        DB Selection




                                       {database}


©2008 MTV Networks Japan K.K.
MTV Flux/MTV Mobile DB project
                                        DB Selection


            Project summary:
            Merge all the database services for the MTV Flux and
            MTV Mobile services on to a single database platform.


            Database requirements:
            • flexibility
            • performance
            • scalability
            • redundancy
            • support
            • cost

©2008 MTV Networks Japan K.K.
MTV Flux/MTV Mobile DB project
                                          DB Selection

                                                     MySQL       MySQL
                 reqs/DB           Oracle   MS SQL
                                                     Community   Enterprise

                 flexibility        △/ ×       ×         ○           ○

                 performance          ○        ○         ○           ○

                 scalability        ○ /△       ○         ○           ○

                 redundancy         ○ /△       ○         ○           ○

                 support              ○        ○         ×           ○

                 cost                 ×        ×         ◎          △
©2008 MTV Networks Japan K.K.
MTV Flux/MTV Mobile DB project
                                        DB Selection

                                   We chose MySQL Enterprise




©2008 MTV Networks Japan K.K.
MTV Flux/MTV Mobile DB project
                                     MySQL Architectures

                                MySQL Architectures:
                                • Single/multi Master
                                • Master/Slave
                                • Cluster
                                • DR:BD/Heartbeat HA Master

                                What we looked for:
                                • flexibility
                                • performance
                                • scalability
                                • redundancy
                                • support
                                • cost
©2008 MTV Networks Japan K.K.
MTV Flux/MTV Mobile DB project
                                         MySQL Architectures

            Single/multi Master
                     www www www                • cheap(!)
                                                • easy configuration
                                                                               ×
                                                • load ↑ ⇒ performance ↓
                                                • high risk (if master dies, no more DB)
                                Master




             flexibility performance scalability redundancy support cost

                  ○                  ○/△       ×            ×           ○      ◎

©2008 MTV Networks Japan K.K.
MTV Flux/MTV Mobile DB project
                                      MySQL Architectures

            Master/Slave
                     www www www
                                             • commonly used architecture
                                                                                    ×
                                             • some redundancy (read/writes separate)

                                             • still some risk (operator needed for outage recovery)
                                             • manual failover for outages
                  Master           Slave




             flexibility performance scalability redundancy support cost

                  ○               △         ○              △               ○        ◎

©2008 MTV Networks Japan K.K.
MTV Flux/MTV Mobile DB project
                                              MySQL Architectures

             Cluster
                      www www www                    • redundant!
                                                     • can scale
                                                                             △
                                                     • complicated configuration
                                                     • limited support in Japan (2007)


                                Cluster

             flexibility performance scalability redundancy support cost

                  ○                       ○         ◎         ◎       ○/△     △

©2008 MTV Networks Japan K.K.
MTV Flux/MTV Mobile DB project
                                      MySQL Architectures

            DR:BD/Heartbeat HA Master
                     www www www               • redundant (master fully redundant)
                                               • can scale (add salves)
                                               • reduced outage risk       ◎
                                               • can use failover for maintenance tasks
                 Primary          Secondary
                                               • complicated configuration


             flexibility performance scalability redundancy support cost

                  ◎              ○/△          ○/△        ◎           ○       ○

©2008 MTV Networks Japan K.K.
MTV Flux/MTV Mobile DB project
                                MySQL HA Master




©2008 MTV Networks Japan K.K.
MTV Flux/MTV Mobile DB project
                                                HA Master

                                Application                        Application



                                                      VIP



                                                    heartbeat




                /etc/my.cnf                                                      /etc/my.cnf
                    local                                                            local

                                   /var/lib/mysql    drbd       /var/lib/mysql
                                      DRBD                         DRBD
                                      Primary                     Secondary
©2008 MTV Networks Japan K.K.
MTV Flux/MTV Mobile DB project
                                              HA Master

            Application           Web site or whatever


                VIP               Manages/monitors DB process and mounts:
                                  • provides application connection point (Virtual IP)
                                  • monitors network status
            heartbeat             • watches the mysqld process
                                  • manages the /var/lib/mysql mount point




                                  mysqld (the database)




              DRBD                 Synchronous device layer data replication


©2008 MTV Networks Japan K.K.
MTV Flux/MTV Mobile DB project
                                                HA Master

                                Application                         3 – heartbeat does:
                                                                   Application
                                                                     • 4 – Failoverdir unmount
                                                                       pDB mysql complete!
                                                                     • DRBD2switch
  Failover
                                          ×
                                                                         1 – Application looses
                                                                               – heartbeat
                                                                     • sDB mysqlthe mount
                                                                          detects dir problem
                                                                         connection to mysqld
                                                      VIP
  process                                                            • sDB mysqld start
network or mysqld
      failure
                                                    heartbeat




                /etc/my.cnf                                                      /etc/my.cnf
                    local                                                            local

                                   /var/lib/mysql    drbd
                                                       drbd     /var/lib/mysql
                                      DRBD                         DRBD
                                      Primary                     Secondary
©2008 MTV Networks Japan K.K.
MTV Flux/MTV Mobile DB project
                                                HA Master

                                Application                        Application
                                                                      3 – heartbeat does:
                                                                      4 – Failover complete!
  Failover                                                            • sDB 32 – drbd mount
                                                                             mysql dir
                                                                              – heartbeat
                                                                      1 – primary server fails!
                                                                      • sDBdetects outage
                                                                             mysqld start
                                                      VIP                    fails over
  process
   server outage


                                                    heartbeat




                /etc/my.cnf                                                      /etc/my.cnf
                    local                                                            local

                                   /var/lib/mysql    drbd       /var/lib/mysql
                                      DRBD                         DRBD
                                      Primary                     Secondary
©2008 MTV Networks Japan K.K.
MTV Flux/MTV Mobile DB project
              HA MySQL install




©2008 MTV Networks Japan K.K.
MTV Flux/MTV Mobile DB project
                                              HA Master Install


  Old DB
  architecture




                                A Master          B Master        Master


                 local disk                local disk                      local disk


                            Dual Master                             Single Master

©2008 MTV Networks Japan K.K.
MTV Flux/MTV Mobile DB project
                                          HA Master Install


   New DB
   architecture
                                                    VIP



                                                  heartbeat




                /etc/my.cnf                                                    /etc/my.cnf
                    local                                                          local

                                 /var/lib/mysql    drbd       /var/lib/mysql
                                    DRBD                         DRBD
                                    Primary                     Secondary
©2008 MTV Networks Japan K.K.
MTV Flux/MTV Mobile DB project
                                       HA Master Install

           Database migration work plan:
           • stop production services
           • perform full data backup (mysqldump)
           • Replace MySQL architecture
           • restore data from backup
           • test database failover
           • test application
           • re-open production services




©2008 MTV Networks Japan K.K.
MTV Flux/MTV Mobile DB project
                                                HA Master Install



  Migration                                                                     2 – install/configure
                                                                                5 – test application
                                                                                      3 – install
                                                   1 – remove old
                                                         VIP                     4 – configure VIP
  process                                       database components
                                                                                  drbd and mysql
                                                                                    connectivity
                                                                                      heartbeat



                                                         heartbeat

                                A Master             B Master                  Master



                    ×
                /etc/my.cnf
                 local disk
                    local                     ×
                                            local disk

                                     /var/lib/mysql        drbd      /var/lib/mysql
                                                                                      /etc/my.cnf
                                                                                       local disk
                                                                                          local


                                        DRBD                            DRBD
                                           Primary                     Secondary
©2008 MTV Networks Japan K.K.
MTV Flux/MTV Mobile DB project
                                       HA Master Install

           Actual migration work times:
           •production services stop – 00:10
           • full data backup (mysqldump) – 00:05
           • MySQL architecture replacement – 01:30
           • data restore – 00:10
           • database failover test – 00:15
           • application test – 03:00
           • production services fully restored!




©2008 MTV Networks Japan K.K.
MTV Flux/MTV Mobile DB project
                                HA MySQL lessons learned




©2008 MTV Networks Japan K.K.
MTV Flux/MTV Mobile DB project
                                      Lessons learned


        Impact on our application:
        • essentially the same as a single master
        • can use same HA for:
            • single master - master/slave - multi-master
        • failover fast enough that DB outages don’t affect application

        Heartbeat:
        • no room for configuration mistakes
        • pay special attention to peer connectivity
        • run lots of tests before going to production!!!



©2008 MTV Networks Japan K.K.
MTV Flux/MTV Mobile DB project
                                      Lessons learned


        mysqld:
        • can take advantage of redundancy to swap out nodes
        • never try to run both primary and secondary at once!!
        • leave all the redundancy to heartbeat

        DR:BD:
        • requires separate partition (can be LVM)
        • like heartbeat – test your configurations!!!
        • always check your DRBD status after a failover
             • peers sometime require a manual re-sync


©2008 MTV Networks Japan K.K.
MTV Flux/MTV Mobile DB project
                                      Lessons learned

        Overall – good points:
        • we were able to deploy HA in a short maintenance window
        • commercial support available (MySQL – LINBIT)
            • MySQL consultative services were a big help
        • also usable with MySQL Community
        • can be part of other architectures
            • multi-master - master/slave

        Overall – weak points:
        • lots to learn, lots to test (heartbeat and DRBD)
        • configuration and failover is complex
        • mysqld restart requires some extra steps
        • all admins/vendors need to understand failover basics
©2008 MTV Networks Japan K.K.
HA MySQL in other MTVNJ environments




©2008 MTV Networks Japan K.K.
Questions?




     DRBD related links
     http://www.drbd.org/users-guide/s-heartbeat-r1.html
     http://www.drbd.org/users-guide/s-heartbeat-crm.html

     MTVNJ links
     http://www.mtvn.jp/en/index.html
     http://www.mtvn.jp/en/service.html

     Patrick Bolduan
     MTV Networks Japan KK
     2-8-2 Jingu-mae, Shibuya-ku, Tokyo 150-0001
     patrick.bolduan@mtvn.jp
     http://www.mtvjapan.com




©2008 MTV Networks Japan K.K.

Más contenido relacionado

La actualidad más candente

ApacheCon EU 2009 Tales from the front line - ActiveMQ ServiceMix and CXF
ApacheCon EU 2009 Tales from the front line - ActiveMQ ServiceMix and CXFApacheCon EU 2009 Tales from the front line - ActiveMQ ServiceMix and CXF
ApacheCon EU 2009 Tales from the front line - ActiveMQ ServiceMix and CXFAdrian Trenaman
 
Ottieni il massimo dalla virtualizzazione con le nuove piattaforme Intel® Xeon®.
Ottieni il massimo dalla virtualizzazione con le nuove piattaforme Intel® Xeon®.Ottieni il massimo dalla virtualizzazione con le nuove piattaforme Intel® Xeon®.
Ottieni il massimo dalla virtualizzazione con le nuove piattaforme Intel® Xeon®.FSCitalia
 
Track 2, Session 2, worlds most powerful intelligent and trusted storage syst...
Track 2, Session 2, worlds most powerful intelligent and trusted storage syst...Track 2, Session 2, worlds most powerful intelligent and trusted storage syst...
Track 2, Session 2, worlds most powerful intelligent and trusted storage syst...EMC Forum India
 
Do More with Oracle Environment with Open and Best of breed Technologies
Do More with Oracle Environment with Open and Best of breed TechnologiesDo More with Oracle Environment with Open and Best of breed Technologies
Do More with Oracle Environment with Open and Best of breed TechnologiesEMC Forum India
 
Achieving genuine elastic multitenancy with the Waratek Cloud VM for Java : J...
Achieving genuine elastic multitenancy with the Waratek Cloud VM for Java : J...Achieving genuine elastic multitenancy with the Waratek Cloud VM for Java : J...
Achieving genuine elastic multitenancy with the Waratek Cloud VM for Java : J...JAX London
 
Enhancing Live Migration Process for CPU and/or memory intensive VMs running...
Enhancing Live Migration Process for CPU and/or  memory intensive VMs running...Enhancing Live Migration Process for CPU and/or  memory intensive VMs running...
Enhancing Live Migration Process for CPU and/or memory intensive VMs running...Benoit Hudzia
 
2nd Eucalyptus Bay Area Meet Up with Rich Wolski
2nd Eucalyptus Bay Area Meet Up with Rich Wolski2nd Eucalyptus Bay Area Meet Up with Rich Wolski
2nd Eucalyptus Bay Area Meet Up with Rich WolskiEucalyptus Systems, Inc.
 
Overview and current topics in solid state storage
Overview and current topics in solid state storageOverview and current topics in solid state storage
Overview and current topics in solid state storageInterop
 
Advancing the Traditional Enterprise: An EA Story
Advancing the Traditional Enterprise: An EA Story Advancing the Traditional Enterprise: An EA Story
Advancing the Traditional Enterprise: An EA Story InnoTech
 

La actualidad más candente (10)

ApacheCon EU 2009 Tales from the front line - ActiveMQ ServiceMix and CXF
ApacheCon EU 2009 Tales from the front line - ActiveMQ ServiceMix and CXFApacheCon EU 2009 Tales from the front line - ActiveMQ ServiceMix and CXF
ApacheCon EU 2009 Tales from the front line - ActiveMQ ServiceMix and CXF
 
Ottieni il massimo dalla virtualizzazione con le nuove piattaforme Intel® Xeon®.
Ottieni il massimo dalla virtualizzazione con le nuove piattaforme Intel® Xeon®.Ottieni il massimo dalla virtualizzazione con le nuove piattaforme Intel® Xeon®.
Ottieni il massimo dalla virtualizzazione con le nuove piattaforme Intel® Xeon®.
 
Track 2, Session 2, worlds most powerful intelligent and trusted storage syst...
Track 2, Session 2, worlds most powerful intelligent and trusted storage syst...Track 2, Session 2, worlds most powerful intelligent and trusted storage syst...
Track 2, Session 2, worlds most powerful intelligent and trusted storage syst...
 
Do More with Oracle Environment with Open and Best of breed Technologies
Do More with Oracle Environment with Open and Best of breed TechnologiesDo More with Oracle Environment with Open and Best of breed Technologies
Do More with Oracle Environment with Open and Best of breed Technologies
 
Achieving genuine elastic multitenancy with the Waratek Cloud VM for Java : J...
Achieving genuine elastic multitenancy with the Waratek Cloud VM for Java : J...Achieving genuine elastic multitenancy with the Waratek Cloud VM for Java : J...
Achieving genuine elastic multitenancy with the Waratek Cloud VM for Java : J...
 
Enhancing Live Migration Process for CPU and/or memory intensive VMs running...
Enhancing Live Migration Process for CPU and/or  memory intensive VMs running...Enhancing Live Migration Process for CPU and/or  memory intensive VMs running...
Enhancing Live Migration Process for CPU and/or memory intensive VMs running...
 
2nd Eucalyptus Bay Area Meet Up with Rich Wolski
2nd Eucalyptus Bay Area Meet Up with Rich Wolski2nd Eucalyptus Bay Area Meet Up with Rich Wolski
2nd Eucalyptus Bay Area Meet Up with Rich Wolski
 
Overview and current topics in solid state storage
Overview and current topics in solid state storageOverview and current topics in solid state storage
Overview and current topics in solid state storage
 
Advancing the Traditional Enterprise: An EA Story
Advancing the Traditional Enterprise: An EA Story Advancing the Traditional Enterprise: An EA Story
Advancing the Traditional Enterprise: An EA Story
 
Tim Cramer, Eucaday
Tim Cramer, EucadayTim Cramer, Eucaday
Tim Cramer, Eucaday
 

Similar a High Availability MySQL with DRBD and Heartbeat MTV Japan Mobile Service

An introduction to Wireless Small Cell Networks
An introduction to Wireless Small Cell NetworksAn introduction to Wireless Small Cell Networks
An introduction to Wireless Small Cell NetworksMehdi Bennis
 
Telco-OTT: infrastructure challenges and solutions
Telco-OTT: infrastructure challenges and solutionsTelco-OTT: infrastructure challenges and solutions
Telco-OTT: infrastructure challenges and solutionsVictor Pascual Ávila
 
Implementing the Future of PostgreSQL Clustering with Tungsten
Implementing the Future of PostgreSQL Clustering with TungstenImplementing the Future of PostgreSQL Clustering with Tungsten
Implementing the Future of PostgreSQL Clustering with TungstenCommand Prompt., Inc
 
YugabyteDB - Distributed SQL Database on Kubernetes
YugabyteDB - Distributed SQL Database on KubernetesYugabyteDB - Distributed SQL Database on Kubernetes
YugabyteDB - Distributed SQL Database on KubernetesDoKC
 
My sql competitive update
My sql competitive updateMy sql competitive update
My sql competitive updatexKinAnx
 
Liberating Your Data From MySQL: Cross-Database Replication to the Rescue!
Liberating Your Data From MySQL: Cross-Database Replication to the Rescue!Liberating Your Data From MySQL: Cross-Database Replication to the Rescue!
Liberating Your Data From MySQL: Cross-Database Replication to the Rescue!Linas Virbalas
 
Scale-out Storage on Intel® Architecture Based Platforms: Characterizing and ...
Scale-out Storage on Intel® Architecture Based Platforms: Characterizing and ...Scale-out Storage on Intel® Architecture Based Platforms: Characterizing and ...
Scale-out Storage on Intel® Architecture Based Platforms: Characterizing and ...Odinot Stanislas
 
NoSQL and MySQL webinar - best of both worlds
NoSQL and MySQL webinar - best of both worldsNoSQL and MySQL webinar - best of both worlds
NoSQL and MySQL webinar - best of both worldsMat Keep
 
Newtec M6100 Broadcast Satellite Modulator Webinar Presentation
Newtec M6100 Broadcast Satellite Modulator Webinar PresentationNewtec M6100 Broadcast Satellite Modulator Webinar Presentation
Newtec M6100 Broadcast Satellite Modulator Webinar PresentationNewtec
 
GE Smallworld Network Inventory Overview
GE Smallworld Network Inventory OverviewGE Smallworld Network Inventory Overview
GE Smallworld Network Inventory Overviewcwilson5496
 
Ora mysql bothGetting the best of both worlds with Oracle 11g and MySQL Enter...
Ora mysql bothGetting the best of both worlds with Oracle 11g and MySQL Enter...Ora mysql bothGetting the best of both worlds with Oracle 11g and MySQL Enter...
Ora mysql bothGetting the best of both worlds with Oracle 11g and MySQL Enter...Ivan Zoratti
 
CDRLive & CDRInsight,CDR Verileri ile Đs Zekası ve Kullanım Örnekleri
CDRLive & CDRInsight,CDR Verileri ile Đs Zekası ve Kullanım ÖrnekleriCDRLive & CDRInsight,CDR Verileri ile Đs Zekası ve Kullanım Örnekleri
CDRLive & CDRInsight,CDR Verileri ile Đs Zekası ve Kullanım Örneklerididemtopuz
 
GE Smallworld Overview September2010
GE Smallworld Overview September2010GE Smallworld Overview September2010
GE Smallworld Overview September2010cwilson5496
 
Building managedprivatecloud kvh_vancouversummit
Building managedprivatecloud kvh_vancouversummitBuilding managedprivatecloud kvh_vancouversummit
Building managedprivatecloud kvh_vancouversummitmatsunota
 
LINE's Infrastructure Platform: How It Scales Massive Services and Maintains ...
LINE's Infrastructure Platform: How It Scales Massive Services and Maintains ...LINE's Infrastructure Platform: How It Scales Massive Services and Maintains ...
LINE's Infrastructure Platform: How It Scales Massive Services and Maintains ...LINE Corporation
 
London Breakfast Seminar
London Breakfast SeminarLondon Breakfast Seminar
London Breakfast SeminarNuoDB
 
2016-JAN-28 -- High Performance Production Databases on Ceph
2016-JAN-28 -- High Performance Production Databases on Ceph2016-JAN-28 -- High Performance Production Databases on Ceph
2016-JAN-28 -- High Performance Production Databases on CephCeph Community
 

Similar a High Availability MySQL with DRBD and Heartbeat MTV Japan Mobile Service (20)

An introduction to Wireless Small Cell Networks
An introduction to Wireless Small Cell NetworksAn introduction to Wireless Small Cell Networks
An introduction to Wireless Small Cell Networks
 
Telco-OTT: infrastructure challenges and solutions
Telco-OTT: infrastructure challenges and solutionsTelco-OTT: infrastructure challenges and solutions
Telco-OTT: infrastructure challenges and solutions
 
Implementing the Future of PostgreSQL Clustering with Tungsten
Implementing the Future of PostgreSQL Clustering with TungstenImplementing the Future of PostgreSQL Clustering with Tungsten
Implementing the Future of PostgreSQL Clustering with Tungsten
 
YugabyteDB - Distributed SQL Database on Kubernetes
YugabyteDB - Distributed SQL Database on KubernetesYugabyteDB - Distributed SQL Database on Kubernetes
YugabyteDB - Distributed SQL Database on Kubernetes
 
My sql competitive update
My sql competitive updateMy sql competitive update
My sql competitive update
 
Liberating Your Data From MySQL: Cross-Database Replication to the Rescue!
Liberating Your Data From MySQL: Cross-Database Replication to the Rescue!Liberating Your Data From MySQL: Cross-Database Replication to the Rescue!
Liberating Your Data From MySQL: Cross-Database Replication to the Rescue!
 
Was ist neu in Exchange 2013?
Was ist neu in Exchange 2013?Was ist neu in Exchange 2013?
Was ist neu in Exchange 2013?
 
Scale-out Storage on Intel® Architecture Based Platforms: Characterizing and ...
Scale-out Storage on Intel® Architecture Based Platforms: Characterizing and ...Scale-out Storage on Intel® Architecture Based Platforms: Characterizing and ...
Scale-out Storage on Intel® Architecture Based Platforms: Characterizing and ...
 
NoSQL and MySQL webinar - best of both worlds
NoSQL and MySQL webinar - best of both worldsNoSQL and MySQL webinar - best of both worlds
NoSQL and MySQL webinar - best of both worlds
 
Newtec M6100 Broadcast Satellite Modulator Webinar Presentation
Newtec M6100 Broadcast Satellite Modulator Webinar PresentationNewtec M6100 Broadcast Satellite Modulator Webinar Presentation
Newtec M6100 Broadcast Satellite Modulator Webinar Presentation
 
GE Smallworld Network Inventory Overview
GE Smallworld Network Inventory OverviewGE Smallworld Network Inventory Overview
GE Smallworld Network Inventory Overview
 
Ora mysql bothGetting the best of both worlds with Oracle 11g and MySQL Enter...
Ora mysql bothGetting the best of both worlds with Oracle 11g and MySQL Enter...Ora mysql bothGetting the best of both worlds with Oracle 11g and MySQL Enter...
Ora mysql bothGetting the best of both worlds with Oracle 11g and MySQL Enter...
 
CDRLive & CDRInsight,CDR Verileri ile Đs Zekası ve Kullanım Örnekleri
CDRLive & CDRInsight,CDR Verileri ile Đs Zekası ve Kullanım ÖrnekleriCDRLive & CDRInsight,CDR Verileri ile Đs Zekası ve Kullanım Örnekleri
CDRLive & CDRInsight,CDR Verileri ile Đs Zekası ve Kullanım Örnekleri
 
GE Smallworld Overview September2010
GE Smallworld Overview September2010GE Smallworld Overview September2010
GE Smallworld Overview September2010
 
Building managedprivatecloud kvh_vancouversummit
Building managedprivatecloud kvh_vancouversummitBuilding managedprivatecloud kvh_vancouversummit
Building managedprivatecloud kvh_vancouversummit
 
LINE's Infrastructure Platform: How It Scales Massive Services and Maintains ...
LINE's Infrastructure Platform: How It Scales Massive Services and Maintains ...LINE's Infrastructure Platform: How It Scales Massive Services and Maintains ...
LINE's Infrastructure Platform: How It Scales Massive Services and Maintains ...
 
Introduction to video compression
Introduction to video compressionIntroduction to video compression
Introduction to video compression
 
Chapter00 2
Chapter00 2Chapter00 2
Chapter00 2
 
London Breakfast Seminar
London Breakfast SeminarLondon Breakfast Seminar
London Breakfast Seminar
 
2016-JAN-28 -- High Performance Production Databases on Ceph
2016-JAN-28 -- High Performance Production Databases on Ceph2016-JAN-28 -- High Performance Production Databases on Ceph
2016-JAN-28 -- High Performance Production Databases on Ceph
 

Más de xlight

What does it take to make google work at scale
What does it take to make google work at scale What does it take to make google work at scale
What does it take to make google work at scale xlight
 
淘宝无线电子商务数据报告
淘宝无线电子商务数据报告淘宝无线电子商务数据报告
淘宝无线电子商务数据报告xlight
 
New zealand bloom filter
New zealand bloom filterNew zealand bloom filter
New zealand bloom filterxlight
 
Product manager-chrissyuan v1.0
Product manager-chrissyuan v1.0Product manager-chrissyuan v1.0
Product manager-chrissyuan v1.0xlight
 
Oracle ha
Oracle haOracle ha
Oracle haxlight
 
Oracle 高可用概述
Oracle 高可用概述Oracle 高可用概述
Oracle 高可用概述xlight
 
Stats partitioned table
Stats partitioned tableStats partitioned table
Stats partitioned tablexlight
 
Optimizing Drupal Performance Zend Acquia Whitepaper Feb2010
Optimizing Drupal Performance Zend Acquia Whitepaper Feb2010Optimizing Drupal Performance Zend Acquia Whitepaper Feb2010
Optimizing Drupal Performance Zend Acquia Whitepaper Feb2010xlight
 
C/C++与Lua混合编程
C/C++与Lua混合编程C/C++与Lua混合编程
C/C++与Lua混合编程xlight
 
Google: The Chubby Lock Service for Loosely-Coupled Distributed Systems
Google: The Chubby Lock Service for Loosely-Coupled Distributed SystemsGoogle: The Chubby Lock Service for Loosely-Coupled Distributed Systems
Google: The Chubby Lock Service for Loosely-Coupled Distributed Systemsxlight
 
Google: The Chubby Lock Service for Loosely-Coupled Distributed Systems
Google: The Chubby Lock Service for Loosely-Coupled Distributed SystemsGoogle: The Chubby Lock Service for Loosely-Coupled Distributed Systems
Google: The Chubby Lock Service for Loosely-Coupled Distributed Systemsxlight
 
PgSQL vs MySQL
PgSQL vs MySQLPgSQL vs MySQL
PgSQL vs MySQLxlight
 
SpeedGeeks
SpeedGeeksSpeedGeeks
SpeedGeeksxlight
 
GOOGLE: Designs, Lessons and Advice from Building Large Distributed Systems
GOOGLE: Designs, Lessons and Advice from Building Large   Distributed Systems GOOGLE: Designs, Lessons and Advice from Building Large   Distributed Systems
GOOGLE: Designs, Lessons and Advice from Building Large Distributed Systems xlight
 
sector-sphere
sector-spheresector-sphere
sector-spherexlight
 
Fixing Twitter Improving The Performance And Scalability Of The Worlds Most ...
Fixing Twitter  Improving The Performance And Scalability Of The Worlds Most ...Fixing Twitter  Improving The Performance And Scalability Of The Worlds Most ...
Fixing Twitter Improving The Performance And Scalability Of The Worlds Most ...xlight
 
Gfarm Fs Tatebe Tip2004
Gfarm Fs Tatebe Tip2004Gfarm Fs Tatebe Tip2004
Gfarm Fs Tatebe Tip2004xlight
 
Make Your web Work
Make Your web WorkMake Your web Work
Make Your web Workxlight
 
Capacity Management from Flickr
Capacity Management from FlickrCapacity Management from Flickr
Capacity Management from Flickrxlight
 

Más de xlight (20)

What does it take to make google work at scale
What does it take to make google work at scale What does it take to make google work at scale
What does it take to make google work at scale
 
淘宝无线电子商务数据报告
淘宝无线电子商务数据报告淘宝无线电子商务数据报告
淘宝无线电子商务数据报告
 
New zealand bloom filter
New zealand bloom filterNew zealand bloom filter
New zealand bloom filter
 
Product manager-chrissyuan v1.0
Product manager-chrissyuan v1.0Product manager-chrissyuan v1.0
Product manager-chrissyuan v1.0
 
Oracle ha
Oracle haOracle ha
Oracle ha
 
Oracle 高可用概述
Oracle 高可用概述Oracle 高可用概述
Oracle 高可用概述
 
Stats partitioned table
Stats partitioned tableStats partitioned table
Stats partitioned table
 
Optimizing Drupal Performance Zend Acquia Whitepaper Feb2010
Optimizing Drupal Performance Zend Acquia Whitepaper Feb2010Optimizing Drupal Performance Zend Acquia Whitepaper Feb2010
Optimizing Drupal Performance Zend Acquia Whitepaper Feb2010
 
C/C++与Lua混合编程
C/C++与Lua混合编程C/C++与Lua混合编程
C/C++与Lua混合编程
 
Google: The Chubby Lock Service for Loosely-Coupled Distributed Systems
Google: The Chubby Lock Service for Loosely-Coupled Distributed SystemsGoogle: The Chubby Lock Service for Loosely-Coupled Distributed Systems
Google: The Chubby Lock Service for Loosely-Coupled Distributed Systems
 
Google: The Chubby Lock Service for Loosely-Coupled Distributed Systems
Google: The Chubby Lock Service for Loosely-Coupled Distributed SystemsGoogle: The Chubby Lock Service for Loosely-Coupled Distributed Systems
Google: The Chubby Lock Service for Loosely-Coupled Distributed Systems
 
PgSQL vs MySQL
PgSQL vs MySQLPgSQL vs MySQL
PgSQL vs MySQL
 
SpeedGeeks
SpeedGeeksSpeedGeeks
SpeedGeeks
 
GOOGLE: Designs, Lessons and Advice from Building Large Distributed Systems
GOOGLE: Designs, Lessons and Advice from Building Large   Distributed Systems GOOGLE: Designs, Lessons and Advice from Building Large   Distributed Systems
GOOGLE: Designs, Lessons and Advice from Building Large Distributed Systems
 
UDT
UDTUDT
UDT
 
sector-sphere
sector-spheresector-sphere
sector-sphere
 
Fixing Twitter Improving The Performance And Scalability Of The Worlds Most ...
Fixing Twitter  Improving The Performance And Scalability Of The Worlds Most ...Fixing Twitter  Improving The Performance And Scalability Of The Worlds Most ...
Fixing Twitter Improving The Performance And Scalability Of The Worlds Most ...
 
Gfarm Fs Tatebe Tip2004
Gfarm Fs Tatebe Tip2004Gfarm Fs Tatebe Tip2004
Gfarm Fs Tatebe Tip2004
 
Make Your web Work
Make Your web WorkMake Your web Work
Make Your web Work
 
Capacity Management from Flickr
Capacity Management from FlickrCapacity Management from Flickr
Capacity Management from Flickr
 

Último

Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXTarek Kalaji
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfJamie (Taka) Wang
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 

Último (20)

Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBX
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 

High Availability MySQL with DRBD and Heartbeat MTV Japan Mobile Service

  • 1. High-Availability MySQL with DR:BD and Heartbeat: MTV Japan mobile services ©2008 MTV Networks Japan K.K.
  • 2. Patrick Bolduan patrick.bolduan@mtvn.jp System Manager MTV Networks Japan K.K. ©2008 MTV Networks Japan K.K.
  • 3. Let’s get started ©2008 MTV Networks Japan K.K.
  • 4. About MTVNJ Company Profile Company Name MTV Networks Japan K.K. President Peter Robert Bullard Address 2-8-2 Jingumae, Shibuya-ku, Tokyo 150-0001 TEL: 03-6434-3111 FAX: 03-6434-3150 Business MTV Networks Japan K.K. (MTVNJ) operates the mobile and PC Description digital media services for all MTV Networks brands including MTV and Nickelodeon in Japan. In addition to digital media services, MTVNJ also operates a 24-hour Nickelodeon channel on CS/CATV, and the consumer products business for Nickelodeon's character franchises in Japan. ©2008 MTV Networks Japan K.K.
  • 5. MTVNJ Mobile Brands ©2008 MTV Networks Japan K.K.
  • 6. MTV Flux/MTV Mobile DB project intro ©2008 MTV Networks Japan K.K.
  • 7. MTV Flux/MTV Mobile DB project DB Selection {database} ©2008 MTV Networks Japan K.K.
  • 8. MTV Flux/MTV Mobile DB project DB Selection Project summary: Merge all the database services for the MTV Flux and MTV Mobile services on to a single database platform. Database requirements: • flexibility • performance • scalability • redundancy • support • cost ©2008 MTV Networks Japan K.K.
  • 9. MTV Flux/MTV Mobile DB project DB Selection MySQL MySQL reqs/DB Oracle MS SQL Community Enterprise flexibility △/ × × ○ ○ performance ○ ○ ○ ○ scalability ○ /△ ○ ○ ○ redundancy ○ /△ ○ ○ ○ support ○ ○ × ○ cost × × ◎ △ ©2008 MTV Networks Japan K.K.
  • 10. MTV Flux/MTV Mobile DB project DB Selection We chose MySQL Enterprise ©2008 MTV Networks Japan K.K.
  • 11. MTV Flux/MTV Mobile DB project MySQL Architectures MySQL Architectures: • Single/multi Master • Master/Slave • Cluster • DR:BD/Heartbeat HA Master What we looked for: • flexibility • performance • scalability • redundancy • support • cost ©2008 MTV Networks Japan K.K.
  • 12. MTV Flux/MTV Mobile DB project MySQL Architectures Single/multi Master www www www • cheap(!) • easy configuration × • load ↑ ⇒ performance ↓ • high risk (if master dies, no more DB) Master flexibility performance scalability redundancy support cost ○ ○/△ × × ○ ◎ ©2008 MTV Networks Japan K.K.
  • 13. MTV Flux/MTV Mobile DB project MySQL Architectures Master/Slave www www www • commonly used architecture × • some redundancy (read/writes separate) • still some risk (operator needed for outage recovery) • manual failover for outages Master Slave flexibility performance scalability redundancy support cost ○ △ ○ △ ○ ◎ ©2008 MTV Networks Japan K.K.
  • 14. MTV Flux/MTV Mobile DB project MySQL Architectures Cluster www www www • redundant! • can scale △ • complicated configuration • limited support in Japan (2007) Cluster flexibility performance scalability redundancy support cost ○ ○ ◎ ◎ ○/△ △ ©2008 MTV Networks Japan K.K.
  • 15. MTV Flux/MTV Mobile DB project MySQL Architectures DR:BD/Heartbeat HA Master www www www • redundant (master fully redundant) • can scale (add salves) • reduced outage risk ◎ • can use failover for maintenance tasks Primary Secondary • complicated configuration flexibility performance scalability redundancy support cost ◎ ○/△ ○/△ ◎ ○ ○ ©2008 MTV Networks Japan K.K.
  • 16. MTV Flux/MTV Mobile DB project MySQL HA Master ©2008 MTV Networks Japan K.K.
  • 17. MTV Flux/MTV Mobile DB project HA Master Application Application VIP heartbeat /etc/my.cnf /etc/my.cnf local local /var/lib/mysql drbd /var/lib/mysql DRBD DRBD Primary Secondary ©2008 MTV Networks Japan K.K.
  • 18. MTV Flux/MTV Mobile DB project HA Master Application Web site or whatever VIP Manages/monitors DB process and mounts: • provides application connection point (Virtual IP) • monitors network status heartbeat • watches the mysqld process • manages the /var/lib/mysql mount point mysqld (the database) DRBD Synchronous device layer data replication ©2008 MTV Networks Japan K.K.
  • 19. MTV Flux/MTV Mobile DB project HA Master Application 3 – heartbeat does: Application • 4 – Failoverdir unmount pDB mysql complete! • DRBD2switch Failover × 1 – Application looses – heartbeat • sDB mysqlthe mount detects dir problem connection to mysqld VIP process • sDB mysqld start network or mysqld failure heartbeat /etc/my.cnf /etc/my.cnf local local /var/lib/mysql drbd drbd /var/lib/mysql DRBD DRBD Primary Secondary ©2008 MTV Networks Japan K.K.
  • 20. MTV Flux/MTV Mobile DB project HA Master Application Application 3 – heartbeat does: 4 – Failover complete! Failover • sDB 32 – drbd mount mysql dir – heartbeat 1 – primary server fails! • sDBdetects outage mysqld start VIP fails over process server outage heartbeat /etc/my.cnf /etc/my.cnf local local /var/lib/mysql drbd /var/lib/mysql DRBD DRBD Primary Secondary ©2008 MTV Networks Japan K.K.
  • 21. MTV Flux/MTV Mobile DB project HA MySQL install ©2008 MTV Networks Japan K.K.
  • 22. MTV Flux/MTV Mobile DB project HA Master Install Old DB architecture A Master B Master Master local disk local disk local disk Dual Master Single Master ©2008 MTV Networks Japan K.K.
  • 23. MTV Flux/MTV Mobile DB project HA Master Install New DB architecture VIP heartbeat /etc/my.cnf /etc/my.cnf local local /var/lib/mysql drbd /var/lib/mysql DRBD DRBD Primary Secondary ©2008 MTV Networks Japan K.K.
  • 24. MTV Flux/MTV Mobile DB project HA Master Install Database migration work plan: • stop production services • perform full data backup (mysqldump) • Replace MySQL architecture • restore data from backup • test database failover • test application • re-open production services ©2008 MTV Networks Japan K.K.
  • 25. MTV Flux/MTV Mobile DB project HA Master Install Migration 2 – install/configure 5 – test application 3 – install 1 – remove old VIP 4 – configure VIP process database components drbd and mysql connectivity heartbeat heartbeat A Master B Master Master × /etc/my.cnf local disk local × local disk /var/lib/mysql drbd /var/lib/mysql /etc/my.cnf local disk local DRBD DRBD Primary Secondary ©2008 MTV Networks Japan K.K.
  • 26. MTV Flux/MTV Mobile DB project HA Master Install Actual migration work times: •production services stop – 00:10 • full data backup (mysqldump) – 00:05 • MySQL architecture replacement – 01:30 • data restore – 00:10 • database failover test – 00:15 • application test – 03:00 • production services fully restored! ©2008 MTV Networks Japan K.K.
  • 27. MTV Flux/MTV Mobile DB project HA MySQL lessons learned ©2008 MTV Networks Japan K.K.
  • 28. MTV Flux/MTV Mobile DB project Lessons learned Impact on our application: • essentially the same as a single master • can use same HA for: • single master - master/slave - multi-master • failover fast enough that DB outages don’t affect application Heartbeat: • no room for configuration mistakes • pay special attention to peer connectivity • run lots of tests before going to production!!! ©2008 MTV Networks Japan K.K.
  • 29. MTV Flux/MTV Mobile DB project Lessons learned mysqld: • can take advantage of redundancy to swap out nodes • never try to run both primary and secondary at once!! • leave all the redundancy to heartbeat DR:BD: • requires separate partition (can be LVM) • like heartbeat – test your configurations!!! • always check your DRBD status after a failover • peers sometime require a manual re-sync ©2008 MTV Networks Japan K.K.
  • 30. MTV Flux/MTV Mobile DB project Lessons learned Overall – good points: • we were able to deploy HA in a short maintenance window • commercial support available (MySQL – LINBIT) • MySQL consultative services were a big help • also usable with MySQL Community • can be part of other architectures • multi-master - master/slave Overall – weak points: • lots to learn, lots to test (heartbeat and DRBD) • configuration and failover is complex • mysqld restart requires some extra steps • all admins/vendors need to understand failover basics ©2008 MTV Networks Japan K.K.
  • 31. HA MySQL in other MTVNJ environments ©2008 MTV Networks Japan K.K.
  • 32. Questions? DRBD related links http://www.drbd.org/users-guide/s-heartbeat-r1.html http://www.drbd.org/users-guide/s-heartbeat-crm.html MTVNJ links http://www.mtvn.jp/en/index.html http://www.mtvn.jp/en/service.html Patrick Bolduan MTV Networks Japan KK 2-8-2 Jingu-mae, Shibuya-ku, Tokyo 150-0001 patrick.bolduan@mtvn.jp http://www.mtvjapan.com ©2008 MTV Networks Japan K.K.