SlideShare una empresa de Scribd logo
1 de 63
Descargar para leer sin conexión
r man
          Gea
Sunday, 25 October 2009
Gearman: a technology
                     for distributed
                       computing
                               Giuseppe Maxia
                          MySQL Community Team Lead
                              Sun Microsystems




Sunday, 25 October 2009
Mainframe

                                                             terminal
                          Mainframe

                                                             terminal

                operating system
                                                                        client
                                                             terminal
                          hardware

                           application
                                                             terminal
    0                                                                   100
                                         USER FRIENDLINESS
Sunday, 25 October 2009
Mini computers
               Mainframe                      Mini           terminal


                                                             terminal
         operating system                    Mini
                                                                        client
                          hardware                           terminal

                           application
                                                             terminal
    0                                                                   100
                                         USER FRIENDLINESS
Sunday, 25 October 2009
Networks
                   server                                   personal
                                                           computer

                                                            personal
       operating operating                                 computer
        system    system
                                                            personal   client
          hardware          hardware                       computer
                                       hardware
                                                            personal
                      application
    0                                                      computer    100
                                       USER FRIENDLINESS
Sunday, 25 October 2009
Web applications
              web server                                browser




                                             INTERNET
                                                        browser
       operating operating
                        operating
        system system systemoperating
                             system
                                                                  client
                                                        browser
        hardware hardware
                            hardware

                      application                       browser
    0                                                             100
                                    USER FRIENDLINESS
Sunday, 25 October 2009
service
                          Cloud applications
        provider service
                 provider
              service
             provider
         service                                              browser
        provider                         web




                                                   INTERNET
                                        server
                                                              browser
       operating operating
                        operating
        system system system
                                                                        client
                                                              browser
          hardware        hardware
                                     hardware

              application applicationapplication              browser
    0                                                                   100
                                     USER FRIENDLINESS
Sunday, 25 October 2009
Some actors

                • memcached
                • gearman

                          Used in production by Facebook and Digg



Sunday, 25 October 2009
GEARMAN


Sunday, 25 October 2009
!= G ER M A N


Sunday, 25 October 2009
G E A R M A N?


Sunday, 25 October 2009
MANAGER

Sunday, 25 October 2009
job
                                   task
                                                   request




            server        worker          client

                  http://gearman.org
Sunday, 25 October 2009
Distributed




Sunday, 25 October 2009
Multiple operating systems




Sunday, 25 October 2009
multiple languages




Sunday, 25 October 2009
freedom of
                             choice



Sunday, 25 October 2009
redundancy



Sunday, 25 October 2009
USING GEARMAN
                    • Server: gearmand
                    • Client libraries:
                     • C/C++
                     • Java
                     • Perl
                     • PHP
                     • Python
Sunday, 25 October 2009
Simple usage
                            GEARMAN


                    • Command line client and worker



Sunday, 25 October 2009
starting the server
  /usr/local/sbin/gearmand -d

  # started as daemon.
  # No feedback given on the command line




Sunday, 25 October 2009
starting the server (2)
  /usr/local/sbin/gearmand -v -v
   INFO Starting up
   INFO Listening on :::4730 (5)
   INFO Listening on 0.0.0.0:4730 (6)

  # started as normal application
  # verbose output requested




Sunday, 25 October 2009
starting the worker
  gearman -w -h hostname -p 4730 
    -f conta wc

  #      -w               =   act as worker
  #      -f               =   function
  #      conta            =   function name
  #      wc               =   command to execute when function
  #                           'conta' is called




Sunday, 25 October 2009
what the server says
  /usr/local/sbin/gearmand -v -v
   INFO Starting up
   INFO Listening on :::4730 (5)
   INFO Listening on 0.0.0.0:4730 (6)
   …

     INFO Accepted connection from 127.0.0.1:4994
     INFO [   0]       127.0.0.1:4994 Connected




Sunday, 25 October 2009
starting the client
  gearman -h hostname -p 4730 
    -f conta < ~/.bashrc
        57     135    2149   # <- output
                             # from worker

  # -f                      = function
  # conta                   = function name
  # < ~/.bashrc             = input data




Sunday, 25 October 2009
what the server says
  /usr/local/sbin/gearmand -v -v
   INFO Starting up
   INFO Listening on :::4730 (5)
   INFO Listening on 0.0.0.0:4730 (6)
   …

     INFO            Accepted connection from 127.0.0.1:4994
     INFO            [   0]       127.0.0.1:4994 Connected
     …
     INFO            Accepted connection from 127.0.0.1:5181
     INFO            [   0]       127.0.0.1:5181 Connected
     INFO            [   0]       127.0.0.1:5181 Disconnected




Sunday, 25 October 2009
What happened
            1             server start


                          listen to port 4730




Sunday, 25 October 2009
What happened
            2              worker starts


                          registers function 'conta' to server




Sunday, 25 October 2009
What happened
            3             client starts


                          requires function 'conta' from server

                                  provides input data


Sunday, 25 October 2009
What happened
            4             server sends client request to worker


                              passes all input data to worker




Sunday, 25 October 2009
What happened
            5             worker receives request and data


                                  processes input




Sunday, 25 October 2009
What happened
            6             worker returns processed data


                             server passes it to client




Sunday, 25 October 2009
What happened
            7             client receives processed data


                               client displays result




Sunday, 25 October 2009
A simple Perl worker



Sunday, 25 October 2009
A simple perl worker

                    1. add server
                    2. add function
                    3. loop
                    4. function definition




Sunday, 25 October 2009
simple worker (1)
  use           strict;
  use           warnings;
  use           Gearman::XS qw(:constants);
  use           Gearman::XS::Worker;

  my $host = '127.0.0.1';
  my $port = 4730;

  my $worker = new Gearman::XS::Worker;

  my $ret = $worker->add_server($host, $port);
  if ($ret != GEARMAN_SUCCESS) {
    printf(STDERR "%sn", $worker->error());
    exit(1);
  }
Sunday, 25 October 2009
simple worker (2)
  my $options = '';

  $ret = $worker->add_function(
    "reverse",   # public function name
    0,           # timeout
    &myreverse, # reference to function
    $options);   # function arguments
  if ($ret != GEARMAN_SUCCESS) {
    printf(STDERR "%sn", $worker->error());
  }




Sunday, 25 October 2009
simple worker (3)
  while (1) {
    my $ret = $worker->work();
    if ($ret != GEARMAN_SUCCESS) {
      printf(STDERR "%sn", $worker->error());
    }
  }




Sunday, 25 October 2009
simple worker (4)
  sub myreverse {
    my ($job) = @_;

         my $workload = $job->workload();
         my $result   = reverse($workload);

    printf(
  "Job=%s F_Name=%s Workload=%s Result=%sn",
       $job->handle(),
       $job->function_name(),
       $job->workload(),
       $result);
    return $result;
  }

Sunday, 25 October 2009
A simple Perl client



Sunday, 25 October 2009
A simple perl client


                    • add server
                    • run a task


Sunday, 25 October 2009
simple client (1)
  use           strict;
  use           warnings;
  use           Gearman::XS qw(:constants);
  use           Gearman::XS::Client;

  my $client = new Gearman::XS::Client;

  my      $host = '127.0.0.1';
  my      $port = 4730;
  my      $ret = $client->add_server($host, $port);
  if      ($ret != GEARMAN_SUCCESS) {
         printf(STDERR "%sn", $client->error());
         exit(1);
  }

Sunday, 25 October 2009
simple client (2)
  my $input = shift || 'teststring';

  my ($return, $result) =
    $client->do("reverse", $input);
  if ($return == GEARMAN_SUCCESS) {
    printf("Result=%sn", $result);
  }




Sunday, 25 October 2009
A sample run



Sunday, 25 October 2009
host 1
  perl worker.pl




Sunday, 25 October 2009
host 2
  perl client.pl
  Result=gnirtstset




Sunday, 25 October 2009
host 1
  perl worker.pl
  Job=H:gmac3.local:4 F_Name=reverse
  Workload=teststring Result=gnirtstset




Sunday, 25 October 2009
more client functions

                    • do_background
                    • add_task
                    • run_tasks


Sunday, 25 October 2009
Some advanced usage

                    • DBIx::SQLCrosstab
                    • Data cubes
                    • Perl only


Sunday, 25 October 2009
Sunday, 25 October 2009
Image processing

                    • CPU intensive
                    • Large storage needed
                    • Application is OS specific


Sunday, 25 October 2009
Sunday, 25 October 2009
Sunday, 25 October 2009
Remote sandboxes
               Using GEARMAN
                     client server architecture
                     N servers
                     N workers
                     N clients
                     Supports multiple languages




Sunday, 25 October 2009
host1                            worker1                    host2
                                                   worker2
                                         func1
     worker0                                         func1
                                          func2         func3
        func1
         func3
                                                           worker3
                                                            func1
                           Gearman
                            server                            func3




        worker4
             func1
               func3                  Gearman
                                       server     client
                          worker5
                            func1
    host3                    func3                              host4
Sunday, 25 October 2009
host1                                           worker1                 host2
                                                             worker2
                                                  func1
    worker0                                                    func1
                                                   func2          func3
       func1
        func3
                                                                     worker3
                                                                      func1
                            Gearman              Gearman
 Master                      server               server    Slave1      func3




       worker4
            func1
                              Gearman
              func3
                               server
                                                            client
                                      worker5
                                        func1
  host3                   Slave2         func3                            host4

Sunday, 25 October 2009
Remote sandbox
                             worker

                   •add server
                   •add function
                   •loop
                   •function definition


Sunday, 25 October 2009
remote sandbox worker (2)
  $ret = $worker->add_function(
    "make_sandbox",
    0,
    &make_sandbox,
    '');
  if ($ret != GEARMAN_SUCCESS) {
       printf(STDERR "%sn", $worker->error());
  }




Sunday, 25 October 2009
remote sandbox worker (4)
  sub make_sandbox {
      my ($job, $options) = @_;

                my $command     = $job->workload();
                chomp $command;
                print STDERR "<$command>n";
                my $result = qx(make_sandbox $command) ;
                if ($?) {
                    print $result, "n";
                    return "Error ($!)n";
                }
                else {
                    return "okn"
                }
  }
Sunday, 25 October 2009
Client to install
                          Remote sandboxes

                   •add server
                   •run remote installation on master
                   •run remote installation on each slave
                   •run configuration queries on slaves


Sunday, 25 October 2009
sample call to remote worker
  my $ret =
    qx(echo "$MASTER_SANDBOX" |
    gearman -h $master_ip -f make_sandbox ) ;

  # using the gearman command line utility




Sunday, 25 October 2009
More on similar matters

     • FOSS.MY, Kuala Lumpur, Malaysia, 25-Oct-2009
     • CodeBits, Lisbon, Portugal, 3-4-5-Dec-2009
     http://datacharmer.blogspot.com

                   http://gearman.org
Sunday, 25 October 2009
THANKS
                                                                              Let's talk!




This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License. To view a copy of this license, visit http://
creativecommons.org/licenses/by-sa/3.0/ or send a letter to Creative Commons, 171 Second Street, Suite 300, San Francisco, California, 94105, USA.
Sunday, 25 October 2009

Más contenido relacionado

Destacado

Gearman Introduction
Gearman IntroductionGearman Introduction
Gearman IntroductionGreen Wang
 
MapReduce Using Perl and Gearman
MapReduce Using Perl and GearmanMapReduce Using Perl and Gearman
MapReduce Using Perl and GearmanJamie Pitts
 
Scalability In PHP
Scalability In PHPScalability In PHP
Scalability In PHPIan Selby
 
Distributed Applications with Perl & Gearman
Distributed Applications with Perl & GearmanDistributed Applications with Perl & Gearman
Distributed Applications with Perl & GearmanIssac Goldstand
 
Scale like an ant, distribute the workload - DPC, Amsterdam, 2011
Scale like an ant, distribute the workload - DPC, Amsterdam,  2011Scale like an ant, distribute the workload - DPC, Amsterdam,  2011
Scale like an ant, distribute the workload - DPC, Amsterdam, 2011Helgi Þormar Þorbjörnsson
 
Gearman work queue in php
Gearman work queue in phpGearman work queue in php
Gearman work queue in phpBo-Yi Wu
 
Building a Scalable Web Crawler with Hadoop
Building a Scalable Web Crawler with HadoopBuilding a Scalable Web Crawler with Hadoop
Building a Scalable Web Crawler with HadoopHadoop User Group
 

Destacado (9)

Gearman Introduction
Gearman IntroductionGearman Introduction
Gearman Introduction
 
MapReduce Using Perl and Gearman
MapReduce Using Perl and GearmanMapReduce Using Perl and Gearman
MapReduce Using Perl and Gearman
 
Scalability In PHP
Scalability In PHPScalability In PHP
Scalability In PHP
 
Gearman and Perl
Gearman and PerlGearman and Perl
Gearman and Perl
 
Distributed Applications with Perl & Gearman
Distributed Applications with Perl & GearmanDistributed Applications with Perl & Gearman
Distributed Applications with Perl & Gearman
 
Scale like an ant, distribute the workload - DPC, Amsterdam, 2011
Scale like an ant, distribute the workload - DPC, Amsterdam,  2011Scale like an ant, distribute the workload - DPC, Amsterdam,  2011
Scale like an ant, distribute the workload - DPC, Amsterdam, 2011
 
Gearman work queue in php
Gearman work queue in phpGearman work queue in php
Gearman work queue in php
 
MapReduce入門
MapReduce入門MapReduce入門
MapReduce入門
 
Building a Scalable Web Crawler with Hadoop
Building a Scalable Web Crawler with HadoopBuilding a Scalable Web Crawler with Hadoop
Building a Scalable Web Crawler with Hadoop
 

Similar a Gearman For Beginners

Quick Introduction to Gearman
Quick Introduction to GearmanQuick Introduction to Gearman
Quick Introduction to GearmanGiuseppe Maxia
 
EclipseCon Europe 2011 m2m workshop
EclipseCon Europe 2011 m2m workshopEclipseCon Europe 2011 m2m workshop
EclipseCon Europe 2011 m2m workshopThibault Cantegrel
 
Cloud 101 - Workshop from Gov2.0 in DC, May 2010
Cloud 101 - Workshop from Gov2.0 in DC, May 2010Cloud 101 - Workshop from Gov2.0 in DC, May 2010
Cloud 101 - Workshop from Gov2.0 in DC, May 2010Alistair Croll
 
Cloudify - Scalability On Demand
Cloudify - Scalability On DemandCloudify - Scalability On Demand
Cloudify - Scalability On DemandFederico Feroldi
 
NGSoft General Overview
NGSoft General OverviewNGSoft General Overview
NGSoft General OverviewMichael Starr
 
ITCamp 2012 - Tudor Damian - Private Cloud with Hyper-V 3 and SCVMM 2012
ITCamp 2012 - Tudor Damian - Private Cloud with Hyper-V 3 and SCVMM 2012ITCamp 2012 - Tudor Damian - Private Cloud with Hyper-V 3 and SCVMM 2012
ITCamp 2012 - Tudor Damian - Private Cloud with Hyper-V 3 and SCVMM 2012ITCamp
 
Gambit communications mimic–snmp based simulation
Gambit communications mimic–snmp based simulationGambit communications mimic–snmp based simulation
Gambit communications mimic–snmp based simulationGambit Communications
 
MIMIC–SNMP Based Simulation of Enterprise Networks
MIMIC–SNMP Based Simulation of Enterprise NetworksMIMIC–SNMP Based Simulation of Enterprise Networks
MIMIC–SNMP Based Simulation of Enterprise NetworksGambit Communications
 
Cloud computing
Cloud computingCloud computing
Cloud computingvdvennen
 
Windows Azure: Is Azure right for you?
Windows Azure: Is Azure right for you?Windows Azure: Is Azure right for you?
Windows Azure: Is Azure right for you?Intergen
 
Hisham Dalle - Zero client computing - taking the desktop into the cloud
Hisham Dalle - Zero client computing - taking the desktop into the cloudHisham Dalle - Zero client computing - taking the desktop into the cloud
Hisham Dalle - Zero client computing - taking the desktop into the cloudnooralmousa
 
Sun/Oracle Desktop Virtualization
Sun/Oracle Desktop VirtualizationSun/Oracle Desktop Virtualization
Sun/Oracle Desktop Virtualizationselghaly
 
Linux, Virtualisation, and Clouds
Linux, Virtualisation, and CloudsLinux, Virtualisation, and Clouds
Linux, Virtualisation, and CloudsRobert Sutor
 
Windows azure uk universities overview march 2012
Windows azure uk universities overview march 2012Windows azure uk universities overview march 2012
Windows azure uk universities overview march 2012Lee Stott
 
Cloud Computing Security
Cloud Computing SecurityCloud Computing Security
Cloud Computing SecurityPiyush Mittal
 
Customer presentation: Trisys, Introduction to AWS, Cambridge
Customer presentation: Trisys, Introduction to AWS, CambridgeCustomer presentation: Trisys, Introduction to AWS, Cambridge
Customer presentation: Trisys, Introduction to AWS, CambridgeAmazon Web Services
 
InduSoft Web Studio and DCS Conversion and Integration Webinar
InduSoft Web Studio and DCS Conversion and Integration WebinarInduSoft Web Studio and DCS Conversion and Integration Webinar
InduSoft Web Studio and DCS Conversion and Integration WebinarAVEVA
 

Similar a Gearman For Beginners (20)

Quick Introduction to Gearman
Quick Introduction to GearmanQuick Introduction to Gearman
Quick Introduction to Gearman
 
EclipseCon Europe 2011 m2m workshop
EclipseCon Europe 2011 m2m workshopEclipseCon Europe 2011 m2m workshop
EclipseCon Europe 2011 m2m workshop
 
Cloud 101 - Workshop from Gov2.0 in DC, May 2010
Cloud 101 - Workshop from Gov2.0 in DC, May 2010Cloud 101 - Workshop from Gov2.0 in DC, May 2010
Cloud 101 - Workshop from Gov2.0 in DC, May 2010
 
Cloudify - Scalability On Demand
Cloudify - Scalability On DemandCloudify - Scalability On Demand
Cloudify - Scalability On Demand
 
NGSoft General Overview
NGSoft General OverviewNGSoft General Overview
NGSoft General Overview
 
DESKTOP VIRTUALIZZATIONS
DESKTOP VIRTUALIZZATIONSDESKTOP VIRTUALIZZATIONS
DESKTOP VIRTUALIZZATIONS
 
ITCamp 2012 - Tudor Damian - Private Cloud with Hyper-V 3 and SCVMM 2012
ITCamp 2012 - Tudor Damian - Private Cloud with Hyper-V 3 and SCVMM 2012ITCamp 2012 - Tudor Damian - Private Cloud with Hyper-V 3 and SCVMM 2012
ITCamp 2012 - Tudor Damian - Private Cloud with Hyper-V 3 and SCVMM 2012
 
Gambit communications mimic–snmp based simulation
Gambit communications mimic–snmp based simulationGambit communications mimic–snmp based simulation
Gambit communications mimic–snmp based simulation
 
MIMIC–SNMP Based Simulation of Enterprise Networks
MIMIC–SNMP Based Simulation of Enterprise NetworksMIMIC–SNMP Based Simulation of Enterprise Networks
MIMIC–SNMP Based Simulation of Enterprise Networks
 
Usenix Invited Talk
Usenix Invited TalkUsenix Invited Talk
Usenix Invited Talk
 
ThinClient
ThinClientThinClient
ThinClient
 
Cloud computing
Cloud computingCloud computing
Cloud computing
 
Windows Azure: Is Azure right for you?
Windows Azure: Is Azure right for you?Windows Azure: Is Azure right for you?
Windows Azure: Is Azure right for you?
 
Hisham Dalle - Zero client computing - taking the desktop into the cloud
Hisham Dalle - Zero client computing - taking the desktop into the cloudHisham Dalle - Zero client computing - taking the desktop into the cloud
Hisham Dalle - Zero client computing - taking the desktop into the cloud
 
Sun/Oracle Desktop Virtualization
Sun/Oracle Desktop VirtualizationSun/Oracle Desktop Virtualization
Sun/Oracle Desktop Virtualization
 
Linux, Virtualisation, and Clouds
Linux, Virtualisation, and CloudsLinux, Virtualisation, and Clouds
Linux, Virtualisation, and Clouds
 
Windows azure uk universities overview march 2012
Windows azure uk universities overview march 2012Windows azure uk universities overview march 2012
Windows azure uk universities overview march 2012
 
Cloud Computing Security
Cloud Computing SecurityCloud Computing Security
Cloud Computing Security
 
Customer presentation: Trisys, Introduction to AWS, Cambridge
Customer presentation: Trisys, Introduction to AWS, CambridgeCustomer presentation: Trisys, Introduction to AWS, Cambridge
Customer presentation: Trisys, Introduction to AWS, Cambridge
 
InduSoft Web Studio and DCS Conversion and Integration Webinar
InduSoft Web Studio and DCS Conversion and Integration WebinarInduSoft Web Studio and DCS Conversion and Integration Webinar
InduSoft Web Studio and DCS Conversion and Integration Webinar
 

Más de Giuseppe Maxia

MySQL NDB 8.0 clusters in your laptop with dbdeployer
MySQL NDB 8.0 clusters in your laptop with dbdeployerMySQL NDB 8.0 clusters in your laptop with dbdeployer
MySQL NDB 8.0 clusters in your laptop with dbdeployerGiuseppe Maxia
 
Dbdeployer, the universal installer
Dbdeployer, the universal installerDbdeployer, the universal installer
Dbdeployer, the universal installerGiuseppe Maxia
 
Test complex database systems in your laptop with dbdeployer
Test complex database systems in your laptop with dbdeployerTest complex database systems in your laptop with dbdeployer
Test complex database systems in your laptop with dbdeployerGiuseppe Maxia
 
A quick tour of Mysql 8 roles
A quick tour of Mysql 8 rolesA quick tour of Mysql 8 roles
A quick tour of Mysql 8 rolesGiuseppe Maxia
 
Synchronise your data between MySQL and MongoDB
Synchronise your data between MySQL and MongoDBSynchronise your data between MySQL and MongoDB
Synchronise your data between MySQL and MongoDBGiuseppe Maxia
 
Juggle your data with Tungsten Replicator
Juggle your data with Tungsten ReplicatorJuggle your data with Tungsten Replicator
Juggle your data with Tungsten ReplicatorGiuseppe Maxia
 
Tungsten Replicator tutorial
Tungsten Replicator tutorialTungsten Replicator tutorial
Tungsten Replicator tutorialGiuseppe Maxia
 
Preventing multi master conflicts with tungsten
Preventing multi master conflicts with tungstenPreventing multi master conflicts with tungsten
Preventing multi master conflicts with tungstenGiuseppe Maxia
 
MySQL high availability power and usability
MySQL high availability power and usabilityMySQL high availability power and usability
MySQL high availability power and usabilityGiuseppe Maxia
 
Solving MySQL replication problems with Tungsten
Solving MySQL replication problems with TungstenSolving MySQL replication problems with Tungsten
Solving MySQL replication problems with TungstenGiuseppe Maxia
 
State of the art of MySQL replication and clustering
State of the art of MySQL replication and clusteringState of the art of MySQL replication and clustering
State of the art of MySQL replication and clusteringGiuseppe Maxia
 
Testing mysql creatively in a sandbox
Testing mysql creatively in a sandboxTesting mysql creatively in a sandbox
Testing mysql creatively in a sandboxGiuseppe Maxia
 
Mysql 5.5 and 5.6 replication
Mysql 5.5 and 5.6 replicationMysql 5.5 and 5.6 replication
Mysql 5.5 and 5.6 replicationGiuseppe Maxia
 

Más de Giuseppe Maxia (20)

MySQL NDB 8.0 clusters in your laptop with dbdeployer
MySQL NDB 8.0 clusters in your laptop with dbdeployerMySQL NDB 8.0 clusters in your laptop with dbdeployer
MySQL NDB 8.0 clusters in your laptop with dbdeployer
 
Test like a_boss
Test like a_bossTest like a_boss
Test like a_boss
 
Dbdeployer, the universal installer
Dbdeployer, the universal installerDbdeployer, the universal installer
Dbdeployer, the universal installer
 
Test complex database systems in your laptop with dbdeployer
Test complex database systems in your laptop with dbdeployerTest complex database systems in your laptop with dbdeployer
Test complex database systems in your laptop with dbdeployer
 
Dbdeployer
DbdeployerDbdeployer
Dbdeployer
 
Dbdeployer
DbdeployerDbdeployer
Dbdeployer
 
A quick tour of Mysql 8 roles
A quick tour of Mysql 8 rolesA quick tour of Mysql 8 roles
A quick tour of Mysql 8 roles
 
MySQL document_store
MySQL document_storeMySQL document_store
MySQL document_store
 
Replication skeptic
Replication skepticReplication skeptic
Replication skeptic
 
Synchronise your data between MySQL and MongoDB
Synchronise your data between MySQL and MongoDBSynchronise your data between MySQL and MongoDB
Synchronise your data between MySQL and MongoDB
 
Juggle your data with Tungsten Replicator
Juggle your data with Tungsten ReplicatorJuggle your data with Tungsten Replicator
Juggle your data with Tungsten Replicator
 
MySQL in your laptop
MySQL in your laptopMySQL in your laptop
MySQL in your laptop
 
Script it
Script itScript it
Script it
 
Tungsten Replicator tutorial
Tungsten Replicator tutorialTungsten Replicator tutorial
Tungsten Replicator tutorial
 
Preventing multi master conflicts with tungsten
Preventing multi master conflicts with tungstenPreventing multi master conflicts with tungsten
Preventing multi master conflicts with tungsten
 
MySQL high availability power and usability
MySQL high availability power and usabilityMySQL high availability power and usability
MySQL high availability power and usability
 
Solving MySQL replication problems with Tungsten
Solving MySQL replication problems with TungstenSolving MySQL replication problems with Tungsten
Solving MySQL replication problems with Tungsten
 
State of the art of MySQL replication and clustering
State of the art of MySQL replication and clusteringState of the art of MySQL replication and clustering
State of the art of MySQL replication and clustering
 
Testing mysql creatively in a sandbox
Testing mysql creatively in a sandboxTesting mysql creatively in a sandbox
Testing mysql creatively in a sandbox
 
Mysql 5.5 and 5.6 replication
Mysql 5.5 and 5.6 replicationMysql 5.5 and 5.6 replication
Mysql 5.5 and 5.6 replication
 

Último

UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
99.99% of Your Traces Are (Probably) Trash (SRECon NA 2024).pdf
99.99% of Your Traces  Are (Probably) Trash (SRECon NA 2024).pdf99.99% of Your Traces  Are (Probably) Trash (SRECon NA 2024).pdf
99.99% of Your Traces Are (Probably) Trash (SRECon NA 2024).pdfPaige Cruz
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
The Kubernetes Gateway API and its role in Cloud Native API Management
The Kubernetes Gateway API and its role in Cloud Native API ManagementThe Kubernetes Gateway API and its role in Cloud Native API Management
The Kubernetes Gateway API and its role in Cloud Native API ManagementNuwan Dias
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfJamie (Taka) Wang
 
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...Daniel Zivkovic
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
IEEE Computer Society’s Strategic Activities and Products including SWEBOK Guide
IEEE Computer Society’s Strategic Activities and Products including SWEBOK GuideIEEE Computer Society’s Strategic Activities and Products including SWEBOK Guide
IEEE Computer Society’s Strategic Activities and Products including SWEBOK GuideHironori Washizaki
 
Governance in SharePoint Premium:What's in the box?
Governance in SharePoint Premium:What's in the box?Governance in SharePoint Premium:What's in the box?
Governance in SharePoint Premium:What's in the box?Juan Carlos Gonzalez
 
100+ ChatGPT Prompts for SEO Optimization
100+ ChatGPT Prompts for SEO Optimization100+ ChatGPT Prompts for SEO Optimization
100+ ChatGPT Prompts for SEO Optimizationarrow10202532yuvraj
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 

Último (20)

UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
99.99% of Your Traces Are (Probably) Trash (SRECon NA 2024).pdf
99.99% of Your Traces  Are (Probably) Trash (SRECon NA 2024).pdf99.99% of Your Traces  Are (Probably) Trash (SRECon NA 2024).pdf
99.99% of Your Traces Are (Probably) Trash (SRECon NA 2024).pdf
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
The Kubernetes Gateway API and its role in Cloud Native API Management
The Kubernetes Gateway API and its role in Cloud Native API ManagementThe Kubernetes Gateway API and its role in Cloud Native API Management
The Kubernetes Gateway API and its role in Cloud Native API Management
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
 
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
IEEE Computer Society’s Strategic Activities and Products including SWEBOK Guide
IEEE Computer Society’s Strategic Activities and Products including SWEBOK GuideIEEE Computer Society’s Strategic Activities and Products including SWEBOK Guide
IEEE Computer Society’s Strategic Activities and Products including SWEBOK Guide
 
Governance in SharePoint Premium:What's in the box?
Governance in SharePoint Premium:What's in the box?Governance in SharePoint Premium:What's in the box?
Governance in SharePoint Premium:What's in the box?
 
100+ ChatGPT Prompts for SEO Optimization
100+ ChatGPT Prompts for SEO Optimization100+ ChatGPT Prompts for SEO Optimization
100+ ChatGPT Prompts for SEO Optimization
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 

Gearman For Beginners

  • 1. r man Gea Sunday, 25 October 2009
  • 2. Gearman: a technology for distributed computing Giuseppe Maxia MySQL Community Team Lead Sun Microsystems Sunday, 25 October 2009
  • 3. Mainframe terminal Mainframe terminal operating system client terminal hardware application terminal 0 100 USER FRIENDLINESS Sunday, 25 October 2009
  • 4. Mini computers Mainframe Mini terminal terminal operating system Mini client hardware terminal application terminal 0 100 USER FRIENDLINESS Sunday, 25 October 2009
  • 5. Networks server personal computer personal operating operating computer system system personal client hardware hardware computer hardware personal application 0 computer 100 USER FRIENDLINESS Sunday, 25 October 2009
  • 6. Web applications web server browser INTERNET browser operating operating operating system system systemoperating system client browser hardware hardware hardware application browser 0 100 USER FRIENDLINESS Sunday, 25 October 2009
  • 7. service Cloud applications provider service provider service provider service browser provider web INTERNET server browser operating operating operating system system system client browser hardware hardware hardware application applicationapplication browser 0 100 USER FRIENDLINESS Sunday, 25 October 2009
  • 8. Some actors • memcached • gearman Used in production by Facebook and Digg Sunday, 25 October 2009
  • 10. != G ER M A N Sunday, 25 October 2009
  • 11. G E A R M A N? Sunday, 25 October 2009
  • 13. job task request server worker client http://gearman.org Sunday, 25 October 2009
  • 17. freedom of choice Sunday, 25 October 2009
  • 19. USING GEARMAN • Server: gearmand • Client libraries: • C/C++ • Java • Perl • PHP • Python Sunday, 25 October 2009
  • 20. Simple usage GEARMAN • Command line client and worker Sunday, 25 October 2009
  • 21. starting the server /usr/local/sbin/gearmand -d # started as daemon. # No feedback given on the command line Sunday, 25 October 2009
  • 22. starting the server (2) /usr/local/sbin/gearmand -v -v INFO Starting up INFO Listening on :::4730 (5) INFO Listening on 0.0.0.0:4730 (6) # started as normal application # verbose output requested Sunday, 25 October 2009
  • 23. starting the worker gearman -w -h hostname -p 4730 -f conta wc # -w = act as worker # -f = function # conta = function name # wc = command to execute when function # 'conta' is called Sunday, 25 October 2009
  • 24. what the server says /usr/local/sbin/gearmand -v -v INFO Starting up INFO Listening on :::4730 (5) INFO Listening on 0.0.0.0:4730 (6) … INFO Accepted connection from 127.0.0.1:4994 INFO [ 0] 127.0.0.1:4994 Connected Sunday, 25 October 2009
  • 25. starting the client gearman -h hostname -p 4730 -f conta < ~/.bashrc 57 135 2149 # <- output # from worker # -f = function # conta = function name # < ~/.bashrc = input data Sunday, 25 October 2009
  • 26. what the server says /usr/local/sbin/gearmand -v -v INFO Starting up INFO Listening on :::4730 (5) INFO Listening on 0.0.0.0:4730 (6) … INFO Accepted connection from 127.0.0.1:4994 INFO [ 0] 127.0.0.1:4994 Connected … INFO Accepted connection from 127.0.0.1:5181 INFO [ 0] 127.0.0.1:5181 Connected INFO [ 0] 127.0.0.1:5181 Disconnected Sunday, 25 October 2009
  • 27. What happened 1 server start listen to port 4730 Sunday, 25 October 2009
  • 28. What happened 2 worker starts registers function 'conta' to server Sunday, 25 October 2009
  • 29. What happened 3 client starts requires function 'conta' from server provides input data Sunday, 25 October 2009
  • 30. What happened 4 server sends client request to worker passes all input data to worker Sunday, 25 October 2009
  • 31. What happened 5 worker receives request and data processes input Sunday, 25 October 2009
  • 32. What happened 6 worker returns processed data server passes it to client Sunday, 25 October 2009
  • 33. What happened 7 client receives processed data client displays result Sunday, 25 October 2009
  • 34. A simple Perl worker Sunday, 25 October 2009
  • 35. A simple perl worker 1. add server 2. add function 3. loop 4. function definition Sunday, 25 October 2009
  • 36. simple worker (1) use strict; use warnings; use Gearman::XS qw(:constants); use Gearman::XS::Worker; my $host = '127.0.0.1'; my $port = 4730; my $worker = new Gearman::XS::Worker; my $ret = $worker->add_server($host, $port); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $worker->error()); exit(1); } Sunday, 25 October 2009
  • 37. simple worker (2) my $options = ''; $ret = $worker->add_function( "reverse", # public function name 0, # timeout &myreverse, # reference to function $options); # function arguments if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $worker->error()); } Sunday, 25 October 2009
  • 38. simple worker (3) while (1) { my $ret = $worker->work(); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $worker->error()); } } Sunday, 25 October 2009
  • 39. simple worker (4) sub myreverse { my ($job) = @_; my $workload = $job->workload(); my $result = reverse($workload); printf( "Job=%s F_Name=%s Workload=%s Result=%sn", $job->handle(), $job->function_name(), $job->workload(), $result); return $result; } Sunday, 25 October 2009
  • 40. A simple Perl client Sunday, 25 October 2009
  • 41. A simple perl client • add server • run a task Sunday, 25 October 2009
  • 42. simple client (1) use strict; use warnings; use Gearman::XS qw(:constants); use Gearman::XS::Client; my $client = new Gearman::XS::Client; my $host = '127.0.0.1'; my $port = 4730; my $ret = $client->add_server($host, $port); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $client->error()); exit(1); } Sunday, 25 October 2009
  • 43. simple client (2) my $input = shift || 'teststring'; my ($return, $result) = $client->do("reverse", $input); if ($return == GEARMAN_SUCCESS) { printf("Result=%sn", $result); } Sunday, 25 October 2009
  • 44. A sample run Sunday, 25 October 2009
  • 45. host 1 perl worker.pl Sunday, 25 October 2009
  • 46. host 2 perl client.pl Result=gnirtstset Sunday, 25 October 2009
  • 47. host 1 perl worker.pl Job=H:gmac3.local:4 F_Name=reverse Workload=teststring Result=gnirtstset Sunday, 25 October 2009
  • 48. more client functions • do_background • add_task • run_tasks Sunday, 25 October 2009
  • 49. Some advanced usage • DBIx::SQLCrosstab • Data cubes • Perl only Sunday, 25 October 2009
  • 51. Image processing • CPU intensive • Large storage needed • Application is OS specific Sunday, 25 October 2009
  • 54. Remote sandboxes Using GEARMAN client server architecture N servers N workers N clients Supports multiple languages Sunday, 25 October 2009
  • 55. host1 worker1 host2 worker2 func1 worker0 func1 func2 func3 func1 func3 worker3 func1 Gearman server func3 worker4 func1 func3 Gearman server client worker5 func1 host3 func3 host4 Sunday, 25 October 2009
  • 56. host1 worker1 host2 worker2 func1 worker0 func1 func2 func3 func1 func3 worker3 func1 Gearman Gearman Master server server Slave1 func3 worker4 func1 Gearman func3 server client worker5 func1 host3 Slave2 func3 host4 Sunday, 25 October 2009
  • 57. Remote sandbox worker •add server •add function •loop •function definition Sunday, 25 October 2009
  • 58. remote sandbox worker (2) $ret = $worker->add_function( "make_sandbox", 0, &make_sandbox, ''); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $worker->error()); } Sunday, 25 October 2009
  • 59. remote sandbox worker (4) sub make_sandbox { my ($job, $options) = @_; my $command = $job->workload(); chomp $command; print STDERR "<$command>n"; my $result = qx(make_sandbox $command) ; if ($?) { print $result, "n"; return "Error ($!)n"; } else { return "okn" } } Sunday, 25 October 2009
  • 60. Client to install Remote sandboxes •add server •run remote installation on master •run remote installation on each slave •run configuration queries on slaves Sunday, 25 October 2009
  • 61. sample call to remote worker my $ret = qx(echo "$MASTER_SANDBOX" | gearman -h $master_ip -f make_sandbox ) ; # using the gearman command line utility Sunday, 25 October 2009
  • 62. More on similar matters • FOSS.MY, Kuala Lumpur, Malaysia, 25-Oct-2009 • CodeBits, Lisbon, Portugal, 3-4-5-Dec-2009 http://datacharmer.blogspot.com http://gearman.org Sunday, 25 October 2009
  • 63. THANKS Let's talk! This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License. To view a copy of this license, visit http:// creativecommons.org/licenses/by-sa/3.0/ or send a letter to Creative Commons, 171 Second Street, Suite 300, San Francisco, California, 94105, USA. Sunday, 25 October 2009