SlideShare una empresa de Scribd logo
1 de 32
Descargar para leer sin conexión
A Clever Way to Scale-out
    a Web Application
       Cybozu Labs, Inc.
         Kazuho Oku
RDB sharding

    denormalization is inevitable

               uid:1-2000
                 uid:2001-4000
                      uid:4001-6000

                  tweet
                          tweet
                           tweet

                following
                     following
                        following
                                                                                                ...
              followed_by
                   followed_by
                       followed_by

                timeline
                      timeline
                         timeline



    when uid:123 tweets, write his tweet, read uids of his followers, and
    update the timeline table of his followers

Sep 11 2009                      A Clever Way to Scale-out a Web Application                           2
Two methods to update the shards

    eventual consistency
         asynchonous updates using worker processes
         pros: fast response, high scalability
         cons: hard to maintain
    2-phase commit
         synchronous updates
         pros: synchronous, doesn't require external
          daemon
         cons: slow response
Sep 11 2009           A Clever Way to Scale-out a Web Application   3
The problems

    complex queries
         reading from / writing to multiple DB nodes
         cannot use secondary indexes
               need to maintain per-user views (denormalized tables)

    maintain consistency between the nodes
         when using eventual consistency model
    dynamic scaling
         adding new nodes without stopping the service

Sep 11 2009               A Clever Way to Scale-out a Web Application   4
Incline




Sep 11 2009   A Clever Way to Scale-out a Web Application   5
Incline

    solution for the two problems of
     eventual consistency:
         complex update queries
         maintenance of the denormalized tables
    basic idea
         do not let app. developers write denormalization
          logic
         handle denormalization below the SQL layer
               by using triggers and queue tables

Sep 11 2009               A Clever Way to Scale-out a Web Application   6
Incline – illustrated

    insert / update / delete rows of related
     tables automatically
               uid:1-2000
                 uid:2001-4000
                      uid:4001-6000

                  tweet
                          tweet
                           tweet

               following
                      following
                        following

              followed_by
                   followed_by
                       followed_by
    ...
                timeline
                       timeline
                        timeline

                        queue
                            queue
                       queue



    when uid:123 tweets, write only to his tweet table. Incline updates
    other tables automatically

Sep 11 2009                      A Clever Way to Scale-out a Web Application                           7
Incline – illustrated (cont'd)

    insert / update / delete rows of related
     tables automatically
               uid:1-2000
                 uid:2001-4000
                      uid:4001-6000

                  tweet
                          tweet
                           tweet

                following
                     following
                        following

              followed_by
                   followed_by
                       followed_by
    ...
                timeline
                       timeline
                        timeline

                        queue
                            queue
                       queue




    when uid:2431 starts following uid:940 only write to his following table

Sep 11 2009                      A Clever Way to Scale-out a Web Application                           8
Incline – details

    triggers generated from def. files
    sync. updates within each node
    async. updates between the nodes
         each DB node has a queue table
         helper program (C++) applies the queued events
          to other nodes
         uses a fault tolerant algorithm
    application only needs to write to the
     user's shard
Sep 11 2009          A Clever Way to Scale-out a Web Application   9
Incline – the commands
   # create queue tables
   % incline --mode=shard --rdbms=mysql --database=microblog 
    --host=10.0.200.10 --source=microblog.json --shard-source=shard.json 
    create-queue

   # create triggers
   % incline --mode=shard --rdbms=mysql --database=microblog 
    --host=10.0.200.10 --source=microblog.json --shard-source=shard.json 
    create-trigger

   # run forwarder (transfers data from specified host to other shards)
   % incline --mode=shard --rdbms=mysql --database=microblog 
    --host=10.0.200.10 --source=microblog.json --shard-source=shard.json 
    forward




Sep 11 2009             A Clever Way to Scale-out a Web Application     10
Incline – the definition files
   # view def. file
                                               # shard def. file
   [
                                                              {
      {
                                                              "algorithm" : "range-int",
          "source"      : [ "tweet", "followed_by" ],
                  "map"      : {
          "destination" : "timeline",
                                    "1"    : {
          "pk_columns" : {
                                                "host"     : "10.0.200.10",
            "followed_by.follower_id" : "user_id",
                        "username" : "pac1251781019"
            "tweet.user_id"           : "tweet_user_id",
                },
            "tweet.tweet_id"          : "tweet_id"
                      "2001" : {
          },
                                                               "host"     : "10.0.200.11",
          "npk_columns" : {
                                                "username" : "pac1251781332"
            "tweet.ctime" : "ctime"
                                     },
          },
                                                            "4001" : {
          "merge"       : {
                                                "host" : "10.0.200.12",
            "tweet.user_id" : "followed_by.user_id"
                        "username" : "pac1251781408"
          },
                                                            }
          "shard-key"   : "user_id"
                               }
        }, {
          "source"      : "following",
          "destination" : "followed_by",
          "pk_columns" : {
              "following.following_id" : "user_id",
              "following.user_id"      : "follower_id"
          },
          "shard-key"    : "user_id"
        }
   ]


Sep 11 2009                              A Clever Way to Scale-out a Web Application                        11
Incline – FYI the generated triggers
   CREATE TRIGGER _INCLINE_followed_by_INSERT AFTER INSERT ON followed_by FOR EACH          NEW.following_id,NEW.user_id,'I';
         ROW BEGIN
                                                                     END IF;
     IF (((1<=NEW.follower_id AND NEW.follower_id<2001))) THEN
                       ENDCREATE TRIGGER _INCLINE_following_DELETE AFTER DELETE ON following FOR EACH
       INSERT INTO timeline (user_id,ctime,tweet_id,tweet_user_id) SELECT                   ROW BEGIN
         NEW.follower_id,tweet.ctime,tweet.tweet_id,tweet.user_id FROM tweet WHERE      IF (((1<=OLD.following_id AND OLD.following_id<2001))) THEN
         tweet.user_id=NEW.user_id;
                                                                                          DELETE FROM followed_by WHERE followed_by.user_id=OLD.following_id AND
     ELSE
                                                                                  followed_by.follower_id=OLD.user_id;
       INSERT INTO _iq_timeline (user_id,ctime,tweet_id,tweet_user_id,_iq_action)       ELSE
         SELECT NEW.follower_id,tweet.ctime,tweet.tweet_id,tweet.user_id,'I' FROM
                                                                                           INSERT INTO _iq_followed_by (user_id,follower_id,_iq_action) SELECT
         tweet WHERE tweet.user_id=NEW.user_id;
                                                                                            OLD.following_id,OLD.user_id,'D';
     END IF;
                                                                           END IF;
   END
                                                                               END
   CREATE TRIGGER _INCLINE_followed_by_UPDATE AFTER UPDATE ON followed_by FOR EACH    CREATE TRIGGER _INCLINE_tweet_INSERT AFTER INSERT ON tweet FOR EACH ROW BEGIN
          ROW BEGIN
                                                                                        INSERT INTO timeline (user_id,ctime,tweet_id,tweet_user_id) SELECT
     IF (((1<=NEW.follower_id AND NEW.follower_id<2001))) THEN
                             followed_by.follower_id,NEW.ctime,NEW.tweet_id,NEW.user_id FROM
        REPLACE INTO timeline (user_id,ctime,tweet_id,tweet_user_id) SELECT                 followed_by WHERE ((1<=followed_by.follower_id AND
          NEW.follower_id,tweet.ctime,tweet.tweet_id,tweet.user_id FROM tweet WHERE         followed_by.follower_id<2001)) AND NEW.user_id=followed_by.user_id;
          tweet.user_id=NEW.user_id;
                                                                                        INSERT INTO _iq_timeline (user_id,ctime,tweet_id,tweet_user_id,_iq_action)
     ELSE
                                                                                  SELECT followed_by.follower_id,NEW.ctime,NEW.tweet_id,NEW.user_id,'I' FROM
        INSERT INTO _iq_timeline (user_id,ctime,tweet_id,tweet_user_id,_iq_action)          followed_by WHERE NOT (((1<=followed_by.follower_id AND
          SELECT NEW.follower_id,tweet.ctime,tweet.tweet_id,tweet.user_id,'U' FROM          followed_by.follower_id<2001))) AND NEW.user_id=followed_by.user_id;
          tweet WHERE tweet.user_id=NEW.user_id;
                                     END
     END IF;
                                                                         CREATE TRIGGER _INCLINE_tweet_UPDATE AFTER UPDATE ON tweet FOR EACH ROW BEGIN
   END
                                                                                 REPLACE INTO timeline (user_id,ctime,tweet_id,tweet_user_id) SELECT
   CREATE TRIGGER _INCLINE_followed_by_DELETE AFTER DELETE ON followed_by FOR EACH          followed_by.follower_id,NEW.ctime,NEW.tweet_id,NEW.user_id FROM
          ROW BEGIN
                                                                        followed_by WHERE ((1<=followed_by.follower_id AND
     IF (((1<=OLD.follower_id AND OLD.follower_id<2001))) THEN
                             followed_by.follower_id<2001)) AND NEW.user_id=followed_by.user_id;
        DELETE FROM timeline WHERE timeline.user_id=OLD.follower_id AND                 INSERT INTO _iq_timeline (user_id,ctime,tweet_id,tweet_user_id,_iq_action)
          tweet_user_id=OLD.user_id;
                                                       SELECT followed_by.follower_id,NEW.ctime,NEW.tweet_id,NEW.user_id,'U' FROM
     ELSE
                                                                                  followed_by WHERE NOT (((1<=followed_by.follower_id AND
                                                                                            followed_by.follower_id<2001))) AND NEW.user_id=followed_by.user_id;
       INSERT INTO _iq_timeline (user_id,tweet_id,tweet_user_id,_iq_action) SELECT
         OLD.follower_id,tweet.tweet_id,tweet.user_id,'D' FROM tweet WHERE            END
         tweet.user_id=OLD.user_id;
                                                  CREATE TRIGGER _INCLINE_tweet_DELETE AFTER DELETE ON tweet FOR EACH ROW BEGIN
     END IF;
                                                                           DELETE FROM timeline WHERE timeline.tweet_id=OLD.tweet_id AND
                                                                                            timeline.tweet_user_id=OLD.user_id;
   END
                                                                                        INSERT INTO _iq_timeline (tweet_id,tweet_user_id,user_id,_iq_action) SELECT
   CREATE TRIGGER _INCLINE_following_INSERT AFTER INSERT ON following FOR EACH ROW
                                                                                            OLD.tweet_id,OLD.user_id,followed_by.follower_id,'D' FROM followed_by
          BEGIN
                                                                                            WHERE OLD.user_id=followed_by.user_id AND NOT
     IF (((1<=NEW.following_id AND NEW.following_id<2001))) THEN
                           (((1<=followed_by.follower_id AND followed_by.follower_id<2001)));
        INSERT INTO followed_by (user_id,follower_id) SELECT
                                                                                      END
          NEW.following_id,NEW.user_id;
     ELSE
       INSERT INTO _iq_followed_by (user_id,follower_id,_iq_action) SELECT


Sep 11 2009                                          A Clever Way to Scale-out a Web Application                                                                   12
Pacific




Sep 11 2009   A Clever Way to Scale-out a Web Application   13
Range-based sharding vs. hash-based

    Range-based sharding is better
         range queries are sometimes necessary
         manual tuning is easy
         number of nodes increase continuously
               with hash-based sharding, you have to add
                1,2,4,8,16,32,64,... servers at once




Sep 11 2009               A Clever Way to Scale-out a Web Application   14
Pacific

    utility programs for dynamic scaling
         mysqld_jumpstart
         pacific_divide




Sep 11 2009         A Clever Way to Scale-out a Web Application   15
mysqld_jumpstart – summary

    create a mysqld instance in a single
     command
         service automatically started by daemontools
         setup of primary nodes and slaves
         auto-generated backup script: install_dir/etc/
          backup.sh
               uses XtraBackup for hot-backup 




Sep 11 2009               A Clever Way to Scale-out a Web Application   16
mysql_jumpstart – the commands
   # create and start a master database
   % mysqld_jumpstart --mysql-install-db=/usr/local/mysql/bin/
      mysql_install_db --mysqld=/usr/local/mysql/libexec/mysqld --base-
      dir=/var/servicedb --server-id=1252619462 --socket=/tmp/mysql-
      servicedb.sock --service-dir=/service/mysql-servicedb --replication-
      network='10.0.200.0/255.255.255.0'

   # backup
   % /var/servicedb/etc/backup.sh /var/backup/servicedb.backup.20090911

   # create and start a slave database
   % mysqld_jumpstart --mysql-install-db=/usr/local/mysql/bin/
      mysql_install_db --mysqld=/usr/local/mysql/libexec/mysqld --base-
      dir=/var/servicedb --server-id=1252619493 --socket=/tmp/mysql-
      servicedb.sock --service-dir=/service/mysql-servicedb --replication-
      network='10.0.200.0/255.255.255.0' --master-host=10.0.200.1 --from-
      innobackupex



Sep 11 2009              A Clever Way to Scale-out a Web Application       17
Splitting a MySQL shard

    use replication to prepare, then upgrade
     a slave to master
         Before:

                        1 2,000
           2,001 4,000
                            4,001 6,000

                                   replication
                                                               slave


              After:

                        1 2,000
           2,001 3,000
             3,001 4,000
   4,001 6,000




Sep 11 2009                        A Clever Way to Scale-out a Web Application                    18
Problems in splitting a shard

    speed vs. safety
         downtime should be minimum
         guarantee that all the application servers write to
          the new node
               reads may switch to the new node eventually




Sep 11 2009               A Clever Way to Scale-out a Web Application   19
Pacific_divide – the blurbs

    fail-safe
         application servers using the old sharding
          definition cannot access the split nodes
               app. servers reload the definition upon such case

    minimum impact on users
         no read-locks during division
               in eventual-consistency mode
         acquires write lock only against the dividing node
         write lock time < 10 seconds
               if no delay in replication
Sep 11 2009                 A Clever Way to Scale-out a Web Application   20
Pacific_divide – the split algorithm

   1.         create a new slave node
   2.         drop write privileges of existing username on the dividing
              node
   3.         wait until the new node becomes in sync.
   4.         update incline triggers
   5.         create new user and give read / write privileges
   6.         update shard def.
   7.         drop read privileges granted to the old username




Sep 11 2009                 A Clever Way to Scale-out a Web Application   21
Pacific_divide – the comand 
   #   upgrade 10.0.200.18 to a master with range uid:3,000-
   #   
   #   when instructed by pacific_divide, transmit shard.json to all
   #   application servers and mysql shards (or you may use nfs, etc.)

   % pacific_divide --shard-def=shard.json --database=microblog --new-
      host=10.0.200.18 --from-id=3000 --incline-source=microblog.json



                 Before:

                            1 2,000
          2,001 4,000
                           4,001 6,000


                                       replication
                                                             slave



                  After:

                            1 2,000
          2,001 3,000
            3,001 4,000
   4,001 6,000




Sep 11 2009                     A Clever Way to Scale-out a Web Application                         22
Pacific_divide – how the shard def. changes
    # before
                                                      # after

    {
                                                             {
         "algorithm" : "range-int",
                                    "algorithm" : "range-int",
         "map"       : {
                                               "map"       : {
          "1"      : {
                                                  "1"      : {
            "host"      : "10.0.200.10",
                                  "host"      : "10.0.200.10",
            "username" : "pac1251781019"
                                  "username" : "pac1251781019"
          },
                                                            },
          "2001" : {
                                                    "2001" : {
             "host"     : "10.0.200.11",
                                  "host"     : "10.0.200.11",
             "username" : "pac1251781332"
                                 "username" : "pac1252624011"
          },
                                                            },
          "4001" : {
                                                    "3001" : {
             "host" : "10.0.200.12",
                                      "host"     : "10.0.200.18",
             "username" : "pac1251781408"
                                 "username" : "pac1252624011"
          }
                                                             },
    }
                                                                   "4001" : {
                                                                           "host" : "10.0.200.12",
                                                                           "username" : "pac1251781408"
                                                                         }
                                                                   }




Sep 11 2009                              A Clever Way to Scale-out a Web Application                       23
DBIx::ShardManager




Sep 11 2009    A Clever Way to Scale-out a Web Application   24
DBIx::ShardManager – the code
   # create manager object
   my $mgr = DBIx::ShardManager->new(
       definition => DBIx::ShardManager::Definition::JSON->new(
           file         => 'etc/user_shard_def.json',
           auto_reload => 1,
       ),
       connector => DBIx::ShardManager::Connector::DBI->new(
           driver => 'mysql',
           dbname => 'microblog',
           attr    => {
                mysql_enable_utf8 => 1,
                RaiseError => 1,
           },
       ),
   );



Sep 11 2009             A Clever Way to Scale-out a Web Application   25
DBIx::ShardManager – the code (cont'd)
   # read user's timeline

   # first, read my timeline table
   my $timeline = $mgr->rw_handle($user_id)->selectall_arrayref(
       'SELECT * FROM timeline WHERE user_id=? ORDER BY ctime DESC LIMIT
      20',
       { Slice => {} },
       $user_id,
   );
   # fetch the tweets using (tweet_user_id,tweet_id) from other shards
   $mgr->shard_inner_join(
       $timeline,
       tweet_user_id => {
            'tweet.tweet_id' => 'tweet_id',
       },
   }


Sep 11 2009             A Clever Way to Scale-out a Web Application        26
DBIx::ShardManager – blurbs

    access to raw DBI handles
         easy to use ORM above DBIx::ShardManager
    detects changes and reloads shard def.
         but may throw exceptions on writes during node
          divisions by pacific_divide
               display maintenance error, and let the user retry

    shard_join to be optimized
         with Net::Drizzle, or mycached


Sep 11 2009                A Clever Way to Scale-out a Web Application   27
Conclusion




Sep 11 2009   A Clever Way to Scale-out a Web Application   28
Conclusion

    RDB sharding is not difficult when using
     Incline, Pacific, DBIx::ShardManager
         IMO it is as easy as writing code for a standalone
          database system
    app. developers can use 2-phase commit
     if necessary
         or rely on Incline for async. updates



Sep 11 2009           A Clever Way to Scale-out a Web Application   29
Current Status & ToDo

    Incline - early beta
         ToDo: add support for multiple shard keys, add
          recovery support on data-loss
    Pacific - early beta
         ToDo: make it a distribution
    DBIx::ShardManager - still alpha
         ToDo: write more join functions, concurrent
          access, etc.


Sep 11 2009           A Clever Way to Scale-out a Web Application   30
Miscellaneous

    Mycached
         currently in alpha status
         access MySQL tables using memcached protocol
         higher concurrency (thousands of connections)
         higher throughput (2x SQL)




Sep 11 2009          A Clever Way to Scale-out a Web Application   31
For more information

    see my blog
     http://developer.cybozu.co.jp/kazuho/
         DBIx::ShardManager is in coderepos.org/share/
          lang/perl
    come to BPStudy #25 on 9/25
         2h30m talk on Incline, Pacific,
          DBIx::ShardManager (hopefully including demos)



Sep 11 2009          A Clever Way to Scale-out a Web Application   32

Más contenido relacionado

Destacado

St. Mark’S Libraries – Tech Talk
St. Mark’S Libraries – Tech TalkSt. Mark’S Libraries – Tech Talk
St. Mark’S Libraries – Tech Talkguestf47073
 
Niedziela W Supermarkecie
Niedziela W SupermarkecieNiedziela W Supermarkecie
Niedziela W Supermarkecieagata stanisz
 
8.5 Y1 Passes Tu Tes Vacances En France
8.5 Y1 Passes Tu Tes Vacances En France8.5 Y1 Passes Tu Tes Vacances En France
8.5 Y1 Passes Tu Tes Vacances En Franceguestc60dc6
 
Печатные издания → цифровые журналы на iPad
Печатные издания → цифровые журналы на iPadПечатные издания → цифровые журналы на iPad
Печатные издания → цифровые журналы на iPadMaria Podolyak
 
Cloudcamp- The World Wide Cloud
Cloudcamp- The World Wide CloudCloudcamp- The World Wide Cloud
Cloudcamp- The World Wide CloudReuven Cohen
 
Ailehekimligianketi
AilehekimligianketiAilehekimligianketi
Ailehekimligianketianttab
 
Mobile Development 101
Mobile Development 101Mobile Development 101
Mobile Development 101Michael Galpin
 
'The university as a hackerspace'
'The university as a hackerspace''The university as a hackerspace'
'The university as a hackerspace'Joss Winn
 
Slavery Module: Lesson three
Slavery Module: Lesson threeSlavery Module: Lesson three
Slavery Module: Lesson threeTerri Weiss
 
Design Patterns for Tablets and Smartphones
Design Patterns for Tablets and SmartphonesDesign Patterns for Tablets and Smartphones
Design Patterns for Tablets and SmartphonesMichael Galpin
 
Lyddie: Unit3 lesson4
Lyddie:  Unit3 lesson4Lyddie:  Unit3 lesson4
Lyddie: Unit3 lesson4Terri Weiss
 
Focus on What Matters
Focus on What MattersFocus on What Matters
Focus on What MattersJennifer Orr
 
Presentation bulgaria o_drzavi
Presentation bulgaria o_drzaviPresentation bulgaria o_drzavi
Presentation bulgaria o_drzaviGavranica
 
Bærum kommune - ny kommunikasjonsstrategi 2009
Bærum kommune - ny kommunikasjonsstrategi 2009Bærum kommune - ny kommunikasjonsstrategi 2009
Bærum kommune - ny kommunikasjonsstrategi 2009Pål Hivand
 
Apprendimento collaborativo e apprendimento esperienziale in Second Life
Apprendimento collaborativo e apprendimento esperienziale in Second LifeApprendimento collaborativo e apprendimento esperienziale in Second Life
Apprendimento collaborativo e apprendimento esperienziale in Second LifeCommunication Village
 

Destacado (20)

St. Mark’S Libraries – Tech Talk
St. Mark’S Libraries – Tech TalkSt. Mark’S Libraries – Tech Talk
St. Mark’S Libraries – Tech Talk
 
Niedziela W Supermarkecie
Niedziela W SupermarkecieNiedziela W Supermarkecie
Niedziela W Supermarkecie
 
8.5 Y1 Passes Tu Tes Vacances En France
8.5 Y1 Passes Tu Tes Vacances En France8.5 Y1 Passes Tu Tes Vacances En France
8.5 Y1 Passes Tu Tes Vacances En France
 
Печатные издания → цифровые журналы на iPad
Печатные издания → цифровые журналы на iPadПечатные издания → цифровые журналы на iPad
Печатные издания → цифровые журналы на iPad
 
Palimpsest Maps
Palimpsest MapsPalimpsest Maps
Palimpsest Maps
 
Cloudcamp- The World Wide Cloud
Cloudcamp- The World Wide CloudCloudcamp- The World Wide Cloud
Cloudcamp- The World Wide Cloud
 
Ailehekimligianketi
AilehekimligianketiAilehekimligianketi
Ailehekimligianketi
 
Online presentatie MvdV
Online presentatie MvdVOnline presentatie MvdV
Online presentatie MvdV
 
Mobile Development 101
Mobile Development 101Mobile Development 101
Mobile Development 101
 
'The university as a hackerspace'
'The university as a hackerspace''The university as a hackerspace'
'The university as a hackerspace'
 
Slavery Module: Lesson three
Slavery Module: Lesson threeSlavery Module: Lesson three
Slavery Module: Lesson three
 
Design Patterns for Tablets and Smartphones
Design Patterns for Tablets and SmartphonesDesign Patterns for Tablets and Smartphones
Design Patterns for Tablets and Smartphones
 
Lyddie: Unit3 lesson4
Lyddie:  Unit3 lesson4Lyddie:  Unit3 lesson4
Lyddie: Unit3 lesson4
 
Focus on What Matters
Focus on What MattersFocus on What Matters
Focus on What Matters
 
Life Of An IPL Question
Life Of An IPL QuestionLife Of An IPL Question
Life Of An IPL Question
 
Origen, filosofía
Origen, filosofíaOrigen, filosofía
Origen, filosofía
 
Presentation bulgaria o_drzavi
Presentation bulgaria o_drzaviPresentation bulgaria o_drzavi
Presentation bulgaria o_drzavi
 
Bærum kommune - ny kommunikasjonsstrategi 2009
Bærum kommune - ny kommunikasjonsstrategi 2009Bærum kommune - ny kommunikasjonsstrategi 2009
Bærum kommune - ny kommunikasjonsstrategi 2009
 
Apprendimento collaborativo e apprendimento esperienziale in Second Life
Apprendimento collaborativo e apprendimento esperienziale in Second LifeApprendimento collaborativo e apprendimento esperienziale in Second Life
Apprendimento collaborativo e apprendimento esperienziale in Second Life
 
Eclipse @eBay 2009
Eclipse @eBay 2009Eclipse @eBay 2009
Eclipse @eBay 2009
 

Similar a A Clever Way to Scale-out a Web Application

Projet d'accès aux résultats des étudiant via client mobile
Projet d'accès aux résultats des étudiant via client mobile Projet d'accès aux résultats des étudiant via client mobile
Projet d'accès aux résultats des étudiant via client mobile Patrick Bashizi
 
Grokking Engineering - Data Analytics Infrastructure at Viki - Huy Nguyen
Grokking Engineering - Data Analytics Infrastructure at Viki - Huy NguyenGrokking Engineering - Data Analytics Infrastructure at Viki - Huy Nguyen
Grokking Engineering - Data Analytics Infrastructure at Viki - Huy NguyenHuy Nguyen
 
Vertically Scaled Design Patters
Vertically Scaled Design PattersVertically Scaled Design Patters
Vertically Scaled Design PattersJeff Malnick
 
Over-the-Air: How we Remotely Compromised the Gateway, BCM, and Autopilot ECU...
Over-the-Air: How we Remotely Compromised the Gateway, BCM, and Autopilot ECU...Over-the-Air: How we Remotely Compromised the Gateway, BCM, and Autopilot ECU...
Over-the-Air: How we Remotely Compromised the Gateway, BCM, and Autopilot ECU...Priyanka Aash
 
Deploying Next Gen Systems with Zero Downtime
Deploying Next Gen Systems with Zero DowntimeDeploying Next Gen Systems with Zero Downtime
Deploying Next Gen Systems with Zero DowntimeTwilio Inc
 
Manchester Serverless Meetup - July 2018
Manchester Serverless Meetup - July 2018Manchester Serverless Meetup - July 2018
Manchester Serverless Meetup - July 2018Jonathan Vines
 
Building Your First App with MongoDB Stitch
Building Your First App with MongoDB StitchBuilding Your First App with MongoDB Stitch
Building Your First App with MongoDB StitchMongoDB
 
Cross the Streams! Creating Streaming Data Pipelines with Apache Flink + Apac...
Cross the Streams! Creating Streaming Data Pipelines with Apache Flink + Apac...Cross the Streams! Creating Streaming Data Pipelines with Apache Flink + Apac...
Cross the Streams! Creating Streaming Data Pipelines with Apache Flink + Apac...StreamNative
 
Caffe studying 2017
Caffe studying 2017Caffe studying 2017
Caffe studying 2017Te-Yen Liu
 
ОЛЕКСАНДР ЛИПКО «Graceful Shutdown Node.js + k8s» Online WDDay 2021
ОЛЕКСАНДР ЛИПКО «Graceful Shutdown Node.js + k8s» Online WDDay 2021ОЛЕКСАНДР ЛИПКО «Graceful Shutdown Node.js + k8s» Online WDDay 2021
ОЛЕКСАНДР ЛИПКО «Graceful Shutdown Node.js + k8s» Online WDDay 2021WDDay
 
1 Database Security Lab 2 – Virtual Private Database.docx
1 Database Security Lab 2 – Virtual Private Database.docx1 Database Security Lab 2 – Virtual Private Database.docx
1 Database Security Lab 2 – Virtual Private Database.docxjeremylockett77
 
Tutorial: Building Your First App with MongoDB Stitch
Tutorial: Building Your First App with MongoDB StitchTutorial: Building Your First App with MongoDB Stitch
Tutorial: Building Your First App with MongoDB StitchMongoDB
 
[CB16] Esoteric Web Application Vulnerabilities by Andrés Riancho
[CB16] Esoteric Web Application Vulnerabilities by Andrés Riancho[CB16] Esoteric Web Application Vulnerabilities by Andrés Riancho
[CB16] Esoteric Web Application Vulnerabilities by Andrés RianchoCODE BLUE
 
JSON SQL Injection and the Lessons Learned
JSON SQL Injection and the Lessons LearnedJSON SQL Injection and the Lessons Learned
JSON SQL Injection and the Lessons LearnedKazuho Oku
 
An intro to Docker, Terraform, and Amazon ECS
An intro to Docker, Terraform, and Amazon ECSAn intro to Docker, Terraform, and Amazon ECS
An intro to Docker, Terraform, and Amazon ECSYevgeniy Brikman
 

Similar a A Clever Way to Scale-out a Web Application (20)

Projet d'accès aux résultats des étudiant via client mobile
Projet d'accès aux résultats des étudiant via client mobile Projet d'accès aux résultats des étudiant via client mobile
Projet d'accès aux résultats des étudiant via client mobile
 
Grokking Engineering - Data Analytics Infrastructure at Viki - Huy Nguyen
Grokking Engineering - Data Analytics Infrastructure at Viki - Huy NguyenGrokking Engineering - Data Analytics Infrastructure at Viki - Huy Nguyen
Grokking Engineering - Data Analytics Infrastructure at Viki - Huy Nguyen
 
Vertically Scaled Design Patters
Vertically Scaled Design PattersVertically Scaled Design Patters
Vertically Scaled Design Patters
 
Not so blind SQL Injection
Not so blind SQL InjectionNot so blind SQL Injection
Not so blind SQL Injection
 
Over-the-Air: How we Remotely Compromised the Gateway, BCM, and Autopilot ECU...
Over-the-Air: How we Remotely Compromised the Gateway, BCM, and Autopilot ECU...Over-the-Air: How we Remotely Compromised the Gateway, BCM, and Autopilot ECU...
Over-the-Air: How we Remotely Compromised the Gateway, BCM, and Autopilot ECU...
 
Deploying Next Gen Systems with Zero Downtime
Deploying Next Gen Systems with Zero DowntimeDeploying Next Gen Systems with Zero Downtime
Deploying Next Gen Systems with Zero Downtime
 
Manchester Serverless Meetup - July 2018
Manchester Serverless Meetup - July 2018Manchester Serverless Meetup - July 2018
Manchester Serverless Meetup - July 2018
 
Building Your First App with MongoDB Stitch
Building Your First App with MongoDB StitchBuilding Your First App with MongoDB Stitch
Building Your First App with MongoDB Stitch
 
Cross the Streams! Creating Streaming Data Pipelines with Apache Flink + Apac...
Cross the Streams! Creating Streaming Data Pipelines with Apache Flink + Apac...Cross the Streams! Creating Streaming Data Pipelines with Apache Flink + Apac...
Cross the Streams! Creating Streaming Data Pipelines with Apache Flink + Apac...
 
The Art of Grey-Box Attack
The Art of Grey-Box AttackThe Art of Grey-Box Attack
The Art of Grey-Box Attack
 
Mojolicious
MojoliciousMojolicious
Mojolicious
 
Caffe studying 2017
Caffe studying 2017Caffe studying 2017
Caffe studying 2017
 
ОЛЕКСАНДР ЛИПКО «Graceful Shutdown Node.js + k8s» Online WDDay 2021
ОЛЕКСАНДР ЛИПКО «Graceful Shutdown Node.js + k8s» Online WDDay 2021ОЛЕКСАНДР ЛИПКО «Graceful Shutdown Node.js + k8s» Online WDDay 2021
ОЛЕКСАНДР ЛИПКО «Graceful Shutdown Node.js + k8s» Online WDDay 2021
 
1 Database Security Lab 2 – Virtual Private Database.docx
1 Database Security Lab 2 – Virtual Private Database.docx1 Database Security Lab 2 – Virtual Private Database.docx
1 Database Security Lab 2 – Virtual Private Database.docx
 
Load testing with Blitz
Load testing with BlitzLoad testing with Blitz
Load testing with Blitz
 
Tutorial: Building Your First App with MongoDB Stitch
Tutorial: Building Your First App with MongoDB StitchTutorial: Building Your First App with MongoDB Stitch
Tutorial: Building Your First App with MongoDB Stitch
 
[CB16] Esoteric Web Application Vulnerabilities by Andrés Riancho
[CB16] Esoteric Web Application Vulnerabilities by Andrés Riancho[CB16] Esoteric Web Application Vulnerabilities by Andrés Riancho
[CB16] Esoteric Web Application Vulnerabilities by Andrés Riancho
 
Hybrid Tips & Tricks
Hybrid Tips & TricksHybrid Tips & Tricks
Hybrid Tips & Tricks
 
JSON SQL Injection and the Lessons Learned
JSON SQL Injection and the Lessons LearnedJSON SQL Injection and the Lessons Learned
JSON SQL Injection and the Lessons Learned
 
An intro to Docker, Terraform, and Amazon ECS
An intro to Docker, Terraform, and Amazon ECSAn intro to Docker, Terraform, and Amazon ECS
An intro to Docker, Terraform, and Amazon ECS
 

Más de Kazuho Oku

HTTP/2で 速くなるとき ならないとき
HTTP/2で 速くなるとき ならないときHTTP/2で 速くなるとき ならないとき
HTTP/2で 速くなるとき ならないときKazuho Oku
 
QUIC標準化動向 〜2017/7
QUIC標準化動向 〜2017/7QUIC標準化動向 〜2017/7
QUIC標準化動向 〜2017/7Kazuho Oku
 
HTTP/2の課題と将来
HTTP/2の課題と将来HTTP/2の課題と将来
HTTP/2の課題と将来Kazuho Oku
 
TLS 1.3 と 0-RTT のこわ〜い話
TLS 1.3 と 0-RTT のこわ〜い話TLS 1.3 と 0-RTT のこわ〜い話
TLS 1.3 と 0-RTT のこわ〜い話Kazuho Oku
 
Reorganizing Website Architecture for HTTP/2 and Beyond
Reorganizing Website Architecture for HTTP/2 and BeyondReorganizing Website Architecture for HTTP/2 and Beyond
Reorganizing Website Architecture for HTTP/2 and BeyondKazuho Oku
 
Recent Advances in HTTP, controlling them using ruby
Recent Advances in HTTP, controlling them using rubyRecent Advances in HTTP, controlling them using ruby
Recent Advances in HTTP, controlling them using rubyKazuho Oku
 
Programming TCP for responsiveness
Programming TCP for responsivenessProgramming TCP for responsiveness
Programming TCP for responsivenessKazuho Oku
 
Programming TCP for responsiveness
Programming TCP for responsivenessProgramming TCP for responsiveness
Programming TCP for responsivenessKazuho Oku
 
Developing the fastest HTTP/2 server
Developing the fastest HTTP/2 serverDeveloping the fastest HTTP/2 server
Developing the fastest HTTP/2 serverKazuho Oku
 
TLS & LURK @ IETF 95
TLS & LURK @ IETF 95TLS & LURK @ IETF 95
TLS & LURK @ IETF 95Kazuho Oku
 
HTTPとサーバ技術の最新動向
HTTPとサーバ技術の最新動向HTTPとサーバ技術の最新動向
HTTPとサーバ技術の最新動向Kazuho Oku
 
ウェブを速くするためにDeNAがやっていること - HTTP/2と、さらにその先
ウェブを速くするためにDeNAがやっていること - HTTP/2と、さらにその先ウェブを速くするためにDeNAがやっていること - HTTP/2と、さらにその先
ウェブを速くするためにDeNAがやっていること - HTTP/2と、さらにその先Kazuho Oku
 
Cache aware-server-push in H2O version 1.5
Cache aware-server-push in H2O version 1.5Cache aware-server-push in H2O version 1.5
Cache aware-server-push in H2O version 1.5Kazuho Oku
 
HTTP/2時代のウェブサイト設計
HTTP/2時代のウェブサイト設計HTTP/2時代のウェブサイト設計
HTTP/2時代のウェブサイト設計Kazuho Oku
 
H2O - making the Web faster
H2O - making the Web fasterH2O - making the Web faster
H2O - making the Web fasterKazuho Oku
 
H2O - making HTTP better
H2O - making HTTP betterH2O - making HTTP better
H2O - making HTTP betterKazuho Oku
 
H2O - the optimized HTTP server
H2O - the optimized HTTP serverH2O - the optimized HTTP server
H2O - the optimized HTTP serverKazuho Oku
 
JSX 速さの秘密 - 高速なJavaScriptを書く方法
JSX 速さの秘密 - 高速なJavaScriptを書く方法JSX 速さの秘密 - 高速なJavaScriptを書く方法
JSX 速さの秘密 - 高速なJavaScriptを書く方法Kazuho Oku
 
JSX の現在と未来 - Oct 26 2013
JSX の現在と未来 - Oct 26 2013JSX の現在と未来 - Oct 26 2013
JSX の現在と未来 - Oct 26 2013Kazuho Oku
 
Using the Power to Prove
Using the Power to ProveUsing the Power to Prove
Using the Power to ProveKazuho Oku
 

Más de Kazuho Oku (20)

HTTP/2で 速くなるとき ならないとき
HTTP/2で 速くなるとき ならないときHTTP/2で 速くなるとき ならないとき
HTTP/2で 速くなるとき ならないとき
 
QUIC標準化動向 〜2017/7
QUIC標準化動向 〜2017/7QUIC標準化動向 〜2017/7
QUIC標準化動向 〜2017/7
 
HTTP/2の課題と将来
HTTP/2の課題と将来HTTP/2の課題と将来
HTTP/2の課題と将来
 
TLS 1.3 と 0-RTT のこわ〜い話
TLS 1.3 と 0-RTT のこわ〜い話TLS 1.3 と 0-RTT のこわ〜い話
TLS 1.3 と 0-RTT のこわ〜い話
 
Reorganizing Website Architecture for HTTP/2 and Beyond
Reorganizing Website Architecture for HTTP/2 and BeyondReorganizing Website Architecture for HTTP/2 and Beyond
Reorganizing Website Architecture for HTTP/2 and Beyond
 
Recent Advances in HTTP, controlling them using ruby
Recent Advances in HTTP, controlling them using rubyRecent Advances in HTTP, controlling them using ruby
Recent Advances in HTTP, controlling them using ruby
 
Programming TCP for responsiveness
Programming TCP for responsivenessProgramming TCP for responsiveness
Programming TCP for responsiveness
 
Programming TCP for responsiveness
Programming TCP for responsivenessProgramming TCP for responsiveness
Programming TCP for responsiveness
 
Developing the fastest HTTP/2 server
Developing the fastest HTTP/2 serverDeveloping the fastest HTTP/2 server
Developing the fastest HTTP/2 server
 
TLS & LURK @ IETF 95
TLS & LURK @ IETF 95TLS & LURK @ IETF 95
TLS & LURK @ IETF 95
 
HTTPとサーバ技術の最新動向
HTTPとサーバ技術の最新動向HTTPとサーバ技術の最新動向
HTTPとサーバ技術の最新動向
 
ウェブを速くするためにDeNAがやっていること - HTTP/2と、さらにその先
ウェブを速くするためにDeNAがやっていること - HTTP/2と、さらにその先ウェブを速くするためにDeNAがやっていること - HTTP/2と、さらにその先
ウェブを速くするためにDeNAがやっていること - HTTP/2と、さらにその先
 
Cache aware-server-push in H2O version 1.5
Cache aware-server-push in H2O version 1.5Cache aware-server-push in H2O version 1.5
Cache aware-server-push in H2O version 1.5
 
HTTP/2時代のウェブサイト設計
HTTP/2時代のウェブサイト設計HTTP/2時代のウェブサイト設計
HTTP/2時代のウェブサイト設計
 
H2O - making the Web faster
H2O - making the Web fasterH2O - making the Web faster
H2O - making the Web faster
 
H2O - making HTTP better
H2O - making HTTP betterH2O - making HTTP better
H2O - making HTTP better
 
H2O - the optimized HTTP server
H2O - the optimized HTTP serverH2O - the optimized HTTP server
H2O - the optimized HTTP server
 
JSX 速さの秘密 - 高速なJavaScriptを書く方法
JSX 速さの秘密 - 高速なJavaScriptを書く方法JSX 速さの秘密 - 高速なJavaScriptを書く方法
JSX 速さの秘密 - 高速なJavaScriptを書く方法
 
JSX の現在と未来 - Oct 26 2013
JSX の現在と未来 - Oct 26 2013JSX の現在と未来 - Oct 26 2013
JSX の現在と未来 - Oct 26 2013
 
Using the Power to Prove
Using the Power to ProveUsing the Power to Prove
Using the Power to Prove
 

Último

[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdf[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdfSandro Moreira
 
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...Angeliki Cooney
 
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...DianaGray10
 
Vector Search -An Introduction in Oracle Database 23ai.pptx
Vector Search -An Introduction in Oracle Database 23ai.pptxVector Search -An Introduction in Oracle Database 23ai.pptx
Vector Search -An Introduction in Oracle Database 23ai.pptxRemote DBA Services
 
Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...apidays
 
MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MIND CTI
 
Introduction to Multilingual Retrieval Augmented Generation (RAG)
Introduction to Multilingual Retrieval Augmented Generation (RAG)Introduction to Multilingual Retrieval Augmented Generation (RAG)
Introduction to Multilingual Retrieval Augmented Generation (RAG)Zilliz
 
WSO2's API Vision: Unifying Control, Empowering Developers
WSO2's API Vision: Unifying Control, Empowering DevelopersWSO2's API Vision: Unifying Control, Empowering Developers
WSO2's API Vision: Unifying Control, Empowering DevelopersWSO2
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAndrey Devyatkin
 
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businessWhy Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businesspanagenda
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoffsammart93
 
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...Orbitshub
 
Platformless Horizons for Digital Adaptability
Platformless Horizons for Digital AdaptabilityPlatformless Horizons for Digital Adaptability
Platformless Horizons for Digital AdaptabilityWSO2
 
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWEREMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWERMadyBayot
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century educationjfdjdjcjdnsjd
 
FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024The Digital Insurer
 
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingRepurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingEdi Saputra
 
CNIC Information System with Pakdata Cf In Pakistan
CNIC Information System with Pakdata Cf In PakistanCNIC Information System with Pakdata Cf In Pakistan
CNIC Information System with Pakdata Cf In Pakistandanishmna97
 

Último (20)

[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdf[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdf
 
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
 
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
 
Vector Search -An Introduction in Oracle Database 23ai.pptx
Vector Search -An Introduction in Oracle Database 23ai.pptxVector Search -An Introduction in Oracle Database 23ai.pptx
Vector Search -An Introduction in Oracle Database 23ai.pptx
 
Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...
 
MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024
 
Introduction to Multilingual Retrieval Augmented Generation (RAG)
Introduction to Multilingual Retrieval Augmented Generation (RAG)Introduction to Multilingual Retrieval Augmented Generation (RAG)
Introduction to Multilingual Retrieval Augmented Generation (RAG)
 
WSO2's API Vision: Unifying Control, Empowering Developers
WSO2's API Vision: Unifying Control, Empowering DevelopersWSO2's API Vision: Unifying Control, Empowering Developers
WSO2's API Vision: Unifying Control, Empowering Developers
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
 
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businessWhy Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire business
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
 
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
 
Platformless Horizons for Digital Adaptability
Platformless Horizons for Digital AdaptabilityPlatformless Horizons for Digital Adaptability
Platformless Horizons for Digital Adaptability
 
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWEREMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century education
 
Understanding the FAA Part 107 License ..
Understanding the FAA Part 107 License ..Understanding the FAA Part 107 License ..
Understanding the FAA Part 107 License ..
 
FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024
 
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingRepurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
 
CNIC Information System with Pakdata Cf In Pakistan
CNIC Information System with Pakdata Cf In PakistanCNIC Information System with Pakdata Cf In Pakistan
CNIC Information System with Pakdata Cf In Pakistan
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
 

A Clever Way to Scale-out a Web Application

  • 1. A Clever Way to Scale-out a Web Application Cybozu Labs, Inc. Kazuho Oku
  • 2. RDB sharding  denormalization is inevitable uid:1-2000 uid:2001-4000 uid:4001-6000 tweet tweet tweet following following following ... followed_by followed_by followed_by timeline timeline timeline when uid:123 tweets, write his tweet, read uids of his followers, and update the timeline table of his followers Sep 11 2009 A Clever Way to Scale-out a Web Application 2
  • 3. Two methods to update the shards  eventual consistency  asynchonous updates using worker processes  pros: fast response, high scalability  cons: hard to maintain  2-phase commit  synchronous updates  pros: synchronous, doesn't require external daemon  cons: slow response Sep 11 2009 A Clever Way to Scale-out a Web Application 3
  • 4. The problems  complex queries  reading from / writing to multiple DB nodes  cannot use secondary indexes  need to maintain per-user views (denormalized tables)  maintain consistency between the nodes  when using eventual consistency model  dynamic scaling  adding new nodes without stopping the service Sep 11 2009 A Clever Way to Scale-out a Web Application 4
  • 5. Incline Sep 11 2009 A Clever Way to Scale-out a Web Application 5
  • 6. Incline  solution for the two problems of eventual consistency:  complex update queries  maintenance of the denormalized tables  basic idea  do not let app. developers write denormalization logic  handle denormalization below the SQL layer  by using triggers and queue tables Sep 11 2009 A Clever Way to Scale-out a Web Application 6
  • 7. Incline – illustrated  insert / update / delete rows of related tables automatically uid:1-2000 uid:2001-4000 uid:4001-6000 tweet tweet tweet following following following followed_by followed_by followed_by ... timeline timeline timeline queue queue queue when uid:123 tweets, write only to his tweet table. Incline updates other tables automatically Sep 11 2009 A Clever Way to Scale-out a Web Application 7
  • 8. Incline – illustrated (cont'd)  insert / update / delete rows of related tables automatically uid:1-2000 uid:2001-4000 uid:4001-6000 tweet tweet tweet following following following followed_by followed_by followed_by ... timeline timeline timeline queue queue queue when uid:2431 starts following uid:940 only write to his following table Sep 11 2009 A Clever Way to Scale-out a Web Application 8
  • 9. Incline – details  triggers generated from def. files  sync. updates within each node  async. updates between the nodes  each DB node has a queue table  helper program (C++) applies the queued events to other nodes  uses a fault tolerant algorithm  application only needs to write to the user's shard Sep 11 2009 A Clever Way to Scale-out a Web Application 9
  • 10. Incline – the commands # create queue tables % incline --mode=shard --rdbms=mysql --database=microblog --host=10.0.200.10 --source=microblog.json --shard-source=shard.json create-queue # create triggers % incline --mode=shard --rdbms=mysql --database=microblog --host=10.0.200.10 --source=microblog.json --shard-source=shard.json create-trigger # run forwarder (transfers data from specified host to other shards) % incline --mode=shard --rdbms=mysql --database=microblog --host=10.0.200.10 --source=microblog.json --shard-source=shard.json forward Sep 11 2009 A Clever Way to Scale-out a Web Application 10
  • 11. Incline – the definition files # view def. file # shard def. file [ { { "algorithm" : "range-int", "source" : [ "tweet", "followed_by" ], "map" : { "destination" : "timeline", "1" : { "pk_columns" : { "host" : "10.0.200.10", "followed_by.follower_id" : "user_id", "username" : "pac1251781019" "tweet.user_id" : "tweet_user_id", }, "tweet.tweet_id" : "tweet_id" "2001" : { }, "host" : "10.0.200.11", "npk_columns" : { "username" : "pac1251781332" "tweet.ctime" : "ctime" }, }, "4001" : { "merge" : { "host" : "10.0.200.12", "tweet.user_id" : "followed_by.user_id" "username" : "pac1251781408" }, } "shard-key" : "user_id" } }, { "source" : "following", "destination" : "followed_by", "pk_columns" : { "following.following_id" : "user_id", "following.user_id" : "follower_id" }, "shard-key" : "user_id" } ] Sep 11 2009 A Clever Way to Scale-out a Web Application 11
  • 12. Incline – FYI the generated triggers CREATE TRIGGER _INCLINE_followed_by_INSERT AFTER INSERT ON followed_by FOR EACH NEW.following_id,NEW.user_id,'I'; ROW BEGIN END IF; IF (((1<=NEW.follower_id AND NEW.follower_id<2001))) THEN ENDCREATE TRIGGER _INCLINE_following_DELETE AFTER DELETE ON following FOR EACH INSERT INTO timeline (user_id,ctime,tweet_id,tweet_user_id) SELECT ROW BEGIN NEW.follower_id,tweet.ctime,tweet.tweet_id,tweet.user_id FROM tweet WHERE IF (((1<=OLD.following_id AND OLD.following_id<2001))) THEN tweet.user_id=NEW.user_id; DELETE FROM followed_by WHERE followed_by.user_id=OLD.following_id AND ELSE followed_by.follower_id=OLD.user_id; INSERT INTO _iq_timeline (user_id,ctime,tweet_id,tweet_user_id,_iq_action) ELSE SELECT NEW.follower_id,tweet.ctime,tweet.tweet_id,tweet.user_id,'I' FROM INSERT INTO _iq_followed_by (user_id,follower_id,_iq_action) SELECT tweet WHERE tweet.user_id=NEW.user_id; OLD.following_id,OLD.user_id,'D'; END IF; END IF; END END CREATE TRIGGER _INCLINE_followed_by_UPDATE AFTER UPDATE ON followed_by FOR EACH CREATE TRIGGER _INCLINE_tweet_INSERT AFTER INSERT ON tweet FOR EACH ROW BEGIN ROW BEGIN INSERT INTO timeline (user_id,ctime,tweet_id,tweet_user_id) SELECT IF (((1<=NEW.follower_id AND NEW.follower_id<2001))) THEN followed_by.follower_id,NEW.ctime,NEW.tweet_id,NEW.user_id FROM REPLACE INTO timeline (user_id,ctime,tweet_id,tweet_user_id) SELECT followed_by WHERE ((1<=followed_by.follower_id AND NEW.follower_id,tweet.ctime,tweet.tweet_id,tweet.user_id FROM tweet WHERE followed_by.follower_id<2001)) AND NEW.user_id=followed_by.user_id; tweet.user_id=NEW.user_id; INSERT INTO _iq_timeline (user_id,ctime,tweet_id,tweet_user_id,_iq_action) ELSE SELECT followed_by.follower_id,NEW.ctime,NEW.tweet_id,NEW.user_id,'I' FROM INSERT INTO _iq_timeline (user_id,ctime,tweet_id,tweet_user_id,_iq_action) followed_by WHERE NOT (((1<=followed_by.follower_id AND SELECT NEW.follower_id,tweet.ctime,tweet.tweet_id,tweet.user_id,'U' FROM followed_by.follower_id<2001))) AND NEW.user_id=followed_by.user_id; tweet WHERE tweet.user_id=NEW.user_id; END END IF; CREATE TRIGGER _INCLINE_tweet_UPDATE AFTER UPDATE ON tweet FOR EACH ROW BEGIN END REPLACE INTO timeline (user_id,ctime,tweet_id,tweet_user_id) SELECT CREATE TRIGGER _INCLINE_followed_by_DELETE AFTER DELETE ON followed_by FOR EACH followed_by.follower_id,NEW.ctime,NEW.tweet_id,NEW.user_id FROM ROW BEGIN followed_by WHERE ((1<=followed_by.follower_id AND IF (((1<=OLD.follower_id AND OLD.follower_id<2001))) THEN followed_by.follower_id<2001)) AND NEW.user_id=followed_by.user_id; DELETE FROM timeline WHERE timeline.user_id=OLD.follower_id AND INSERT INTO _iq_timeline (user_id,ctime,tweet_id,tweet_user_id,_iq_action) tweet_user_id=OLD.user_id; SELECT followed_by.follower_id,NEW.ctime,NEW.tweet_id,NEW.user_id,'U' FROM ELSE followed_by WHERE NOT (((1<=followed_by.follower_id AND followed_by.follower_id<2001))) AND NEW.user_id=followed_by.user_id; INSERT INTO _iq_timeline (user_id,tweet_id,tweet_user_id,_iq_action) SELECT OLD.follower_id,tweet.tweet_id,tweet.user_id,'D' FROM tweet WHERE END tweet.user_id=OLD.user_id; CREATE TRIGGER _INCLINE_tweet_DELETE AFTER DELETE ON tweet FOR EACH ROW BEGIN END IF; DELETE FROM timeline WHERE timeline.tweet_id=OLD.tweet_id AND timeline.tweet_user_id=OLD.user_id; END INSERT INTO _iq_timeline (tweet_id,tweet_user_id,user_id,_iq_action) SELECT CREATE TRIGGER _INCLINE_following_INSERT AFTER INSERT ON following FOR EACH ROW OLD.tweet_id,OLD.user_id,followed_by.follower_id,'D' FROM followed_by BEGIN WHERE OLD.user_id=followed_by.user_id AND NOT IF (((1<=NEW.following_id AND NEW.following_id<2001))) THEN (((1<=followed_by.follower_id AND followed_by.follower_id<2001))); INSERT INTO followed_by (user_id,follower_id) SELECT END NEW.following_id,NEW.user_id; ELSE INSERT INTO _iq_followed_by (user_id,follower_id,_iq_action) SELECT Sep 11 2009 A Clever Way to Scale-out a Web Application 12
  • 13. Pacific Sep 11 2009 A Clever Way to Scale-out a Web Application 13
  • 14. Range-based sharding vs. hash-based  Range-based sharding is better  range queries are sometimes necessary  manual tuning is easy  number of nodes increase continuously  with hash-based sharding, you have to add 1,2,4,8,16,32,64,... servers at once Sep 11 2009 A Clever Way to Scale-out a Web Application 14
  • 15. Pacific  utility programs for dynamic scaling  mysqld_jumpstart  pacific_divide Sep 11 2009 A Clever Way to Scale-out a Web Application 15
  • 16. mysqld_jumpstart – summary  create a mysqld instance in a single command  service automatically started by daemontools  setup of primary nodes and slaves  auto-generated backup script: install_dir/etc/ backup.sh  uses XtraBackup for hot-backup Sep 11 2009 A Clever Way to Scale-out a Web Application 16
  • 17. mysql_jumpstart – the commands # create and start a master database % mysqld_jumpstart --mysql-install-db=/usr/local/mysql/bin/ mysql_install_db --mysqld=/usr/local/mysql/libexec/mysqld --base- dir=/var/servicedb --server-id=1252619462 --socket=/tmp/mysql- servicedb.sock --service-dir=/service/mysql-servicedb --replication- network='10.0.200.0/255.255.255.0' # backup % /var/servicedb/etc/backup.sh /var/backup/servicedb.backup.20090911 # create and start a slave database % mysqld_jumpstart --mysql-install-db=/usr/local/mysql/bin/ mysql_install_db --mysqld=/usr/local/mysql/libexec/mysqld --base- dir=/var/servicedb --server-id=1252619493 --socket=/tmp/mysql- servicedb.sock --service-dir=/service/mysql-servicedb --replication- network='10.0.200.0/255.255.255.0' --master-host=10.0.200.1 --from- innobackupex Sep 11 2009 A Clever Way to Scale-out a Web Application 17
  • 18. Splitting a MySQL shard  use replication to prepare, then upgrade a slave to master Before: 1 2,000 2,001 4,000 4,001 6,000 replication slave After: 1 2,000 2,001 3,000 3,001 4,000 4,001 6,000 Sep 11 2009 A Clever Way to Scale-out a Web Application 18
  • 19. Problems in splitting a shard  speed vs. safety  downtime should be minimum  guarantee that all the application servers write to the new node  reads may switch to the new node eventually Sep 11 2009 A Clever Way to Scale-out a Web Application 19
  • 20. Pacific_divide – the blurbs  fail-safe  application servers using the old sharding definition cannot access the split nodes  app. servers reload the definition upon such case  minimum impact on users  no read-locks during division  in eventual-consistency mode  acquires write lock only against the dividing node  write lock time < 10 seconds  if no delay in replication Sep 11 2009 A Clever Way to Scale-out a Web Application 20
  • 21. Pacific_divide – the split algorithm 1.  create a new slave node 2.  drop write privileges of existing username on the dividing node 3.  wait until the new node becomes in sync. 4.  update incline triggers 5.  create new user and give read / write privileges 6.  update shard def. 7.  drop read privileges granted to the old username Sep 11 2009 A Clever Way to Scale-out a Web Application 21
  • 22. Pacific_divide – the comand # upgrade 10.0.200.18 to a master with range uid:3,000- # # when instructed by pacific_divide, transmit shard.json to all # application servers and mysql shards (or you may use nfs, etc.) % pacific_divide --shard-def=shard.json --database=microblog --new- host=10.0.200.18 --from-id=3000 --incline-source=microblog.json Before: 1 2,000 2,001 4,000 4,001 6,000 replication slave After: 1 2,000 2,001 3,000 3,001 4,000 4,001 6,000 Sep 11 2009 A Clever Way to Scale-out a Web Application 22
  • 23. Pacific_divide – how the shard def. changes # before # after { { "algorithm" : "range-int", "algorithm" : "range-int", "map" : { "map" : { "1" : { "1" : { "host" : "10.0.200.10", "host" : "10.0.200.10", "username" : "pac1251781019" "username" : "pac1251781019" }, }, "2001" : { "2001" : { "host" : "10.0.200.11", "host" : "10.0.200.11", "username" : "pac1251781332" "username" : "pac1252624011" }, }, "4001" : { "3001" : { "host" : "10.0.200.12", "host" : "10.0.200.18", "username" : "pac1251781408" "username" : "pac1252624011" } }, } "4001" : { "host" : "10.0.200.12", "username" : "pac1251781408" } } Sep 11 2009 A Clever Way to Scale-out a Web Application 23
  • 24. DBIx::ShardManager Sep 11 2009 A Clever Way to Scale-out a Web Application 24
  • 25. DBIx::ShardManager – the code # create manager object my $mgr = DBIx::ShardManager->new( definition => DBIx::ShardManager::Definition::JSON->new( file => 'etc/user_shard_def.json', auto_reload => 1, ), connector => DBIx::ShardManager::Connector::DBI->new( driver => 'mysql', dbname => 'microblog', attr => { mysql_enable_utf8 => 1, RaiseError => 1, }, ), ); Sep 11 2009 A Clever Way to Scale-out a Web Application 25
  • 26. DBIx::ShardManager – the code (cont'd) # read user's timeline # first, read my timeline table my $timeline = $mgr->rw_handle($user_id)->selectall_arrayref( 'SELECT * FROM timeline WHERE user_id=? ORDER BY ctime DESC LIMIT 20', { Slice => {} }, $user_id, ); # fetch the tweets using (tweet_user_id,tweet_id) from other shards $mgr->shard_inner_join( $timeline, tweet_user_id => { 'tweet.tweet_id' => 'tweet_id', }, } Sep 11 2009 A Clever Way to Scale-out a Web Application 26
  • 27. DBIx::ShardManager – blurbs  access to raw DBI handles  easy to use ORM above DBIx::ShardManager  detects changes and reloads shard def.  but may throw exceptions on writes during node divisions by pacific_divide  display maintenance error, and let the user retry  shard_join to be optimized  with Net::Drizzle, or mycached Sep 11 2009 A Clever Way to Scale-out a Web Application 27
  • 28. Conclusion Sep 11 2009 A Clever Way to Scale-out a Web Application 28
  • 29. Conclusion  RDB sharding is not difficult when using Incline, Pacific, DBIx::ShardManager  IMO it is as easy as writing code for a standalone database system  app. developers can use 2-phase commit if necessary  or rely on Incline for async. updates Sep 11 2009 A Clever Way to Scale-out a Web Application 29
  • 30. Current Status & ToDo  Incline - early beta  ToDo: add support for multiple shard keys, add recovery support on data-loss  Pacific - early beta  ToDo: make it a distribution  DBIx::ShardManager - still alpha  ToDo: write more join functions, concurrent access, etc. Sep 11 2009 A Clever Way to Scale-out a Web Application 30
  • 31. Miscellaneous  Mycached  currently in alpha status  access MySQL tables using memcached protocol  higher concurrency (thousands of connections)  higher throughput (2x SQL) Sep 11 2009 A Clever Way to Scale-out a Web Application 31
  • 32. For more information  see my blog http://developer.cybozu.co.jp/kazuho/  DBIx::ShardManager is in coderepos.org/share/ lang/perl  come to BPStudy #25 on 9/25  2h30m talk on Incline, Pacific, DBIx::ShardManager (hopefully including demos) Sep 11 2009 A Clever Way to Scale-out a Web Application 32