With Gearman, a client/server infrastructure for generic tasks, you can work on distributed servers easily, with little worry about the details.
No matter what language you speak, Gearman can meet your needs in C, PHP, Perl, Ruby, shell scripting, and several more.
2. Gearman: a technology
for distributed
computing
Giuseppe Maxia
MySQL Community Team Lead
Sun Microsystems
Sunday, 25 October 2009
3. Mainframe
terminal
Mainframe
terminal
operating system
client
terminal
hardware
application
terminal
0 100
USER FRIENDLINESS
Sunday, 25 October 2009
4. Mini computers
Mainframe Mini terminal
terminal
operating system Mini
client
hardware terminal
application
terminal
0 100
USER FRIENDLINESS
Sunday, 25 October 2009
5. Networks
server personal
computer
personal
operating operating computer
system system
personal client
hardware hardware computer
hardware
personal
application
0 computer 100
USER FRIENDLINESS
Sunday, 25 October 2009
6. Web applications
web server browser
INTERNET
browser
operating operating
operating
system system systemoperating
system
client
browser
hardware hardware
hardware
application browser
0 100
USER FRIENDLINESS
Sunday, 25 October 2009
7. service
Cloud applications
provider service
provider
service
provider
service browser
provider web
INTERNET
server
browser
operating operating
operating
system system system
client
browser
hardware hardware
hardware
application applicationapplication browser
0 100
USER FRIENDLINESS
Sunday, 25 October 2009
8. Some actors
• memcached
• gearman
Used in production by Facebook and Digg
Sunday, 25 October 2009
20. Simple usage
GEARMAN
• Command line client and worker
Sunday, 25 October 2009
21. starting the server
/usr/local/sbin/gearmand -d
# started as daemon.
# No feedback given on the command line
Sunday, 25 October 2009
22. starting the server (2)
/usr/local/sbin/gearmand -v -v
INFO Starting up
INFO Listening on :::4730 (5)
INFO Listening on 0.0.0.0:4730 (6)
# started as normal application
# verbose output requested
Sunday, 25 October 2009
23. starting the worker
gearman -w -h hostname -p 4730
-f conta wc
# -w = act as worker
# -f = function
# conta = function name
# wc = command to execute when function
# 'conta' is called
Sunday, 25 October 2009
24. what the server says
/usr/local/sbin/gearmand -v -v
INFO Starting up
INFO Listening on :::4730 (5)
INFO Listening on 0.0.0.0:4730 (6)
…
INFO Accepted connection from 127.0.0.1:4994
INFO [ 0] 127.0.0.1:4994 Connected
Sunday, 25 October 2009
25. starting the client
gearman -h hostname -p 4730
-f conta < ~/.bashrc
57 135 2149 # <- output
# from worker
# -f = function
# conta = function name
# < ~/.bashrc = input data
Sunday, 25 October 2009
26. what the server says
/usr/local/sbin/gearmand -v -v
INFO Starting up
INFO Listening on :::4730 (5)
INFO Listening on 0.0.0.0:4730 (6)
…
INFO Accepted connection from 127.0.0.1:4994
INFO [ 0] 127.0.0.1:4994 Connected
…
INFO Accepted connection from 127.0.0.1:5181
INFO [ 0] 127.0.0.1:5181 Connected
INFO [ 0] 127.0.0.1:5181 Disconnected
Sunday, 25 October 2009
27. What happened
1 server start
listen to port 4730
Sunday, 25 October 2009
28. What happened
2 worker starts
registers function 'conta' to server
Sunday, 25 October 2009
29. What happened
3 client starts
requires function 'conta' from server
provides input data
Sunday, 25 October 2009
30. What happened
4 server sends client request to worker
passes all input data to worker
Sunday, 25 October 2009
31. What happened
5 worker receives request and data
processes input
Sunday, 25 October 2009
32. What happened
6 worker returns processed data
server passes it to client
Sunday, 25 October 2009
33. What happened
7 client receives processed data
client displays result
Sunday, 25 October 2009
35. A simple perl worker
1. add server
2. add function
3. loop
4. function definition
Sunday, 25 October 2009
36. simple worker (1)
use strict;
use warnings;
use Gearman::XS qw(:constants);
use Gearman::XS::Worker;
my $host = '127.0.0.1';
my $port = 4730;
my $worker = new Gearman::XS::Worker;
my $ret = $worker->add_server($host, $port);
if ($ret != GEARMAN_SUCCESS) {
printf(STDERR "%sn", $worker->error());
exit(1);
}
Sunday, 25 October 2009
37. simple worker (2)
my $options = '';
$ret = $worker->add_function(
"reverse", # public function name
0, # timeout
&myreverse, # reference to function
$options); # function arguments
if ($ret != GEARMAN_SUCCESS) {
printf(STDERR "%sn", $worker->error());
}
Sunday, 25 October 2009
38. simple worker (3)
while (1) {
my $ret = $worker->work();
if ($ret != GEARMAN_SUCCESS) {
printf(STDERR "%sn", $worker->error());
}
}
Sunday, 25 October 2009
39. simple worker (4)
sub myreverse {
my ($job) = @_;
my $workload = $job->workload();
my $result = reverse($workload);
printf(
"Job=%s F_Name=%s Workload=%s Result=%sn",
$job->handle(),
$job->function_name(),
$job->workload(),
$result);
return $result;
}
Sunday, 25 October 2009
41. A simple perl client
• add server
• run a task
Sunday, 25 October 2009
42. simple client (1)
use strict;
use warnings;
use Gearman::XS qw(:constants);
use Gearman::XS::Client;
my $client = new Gearman::XS::Client;
my $host = '127.0.0.1';
my $port = 4730;
my $ret = $client->add_server($host, $port);
if ($ret != GEARMAN_SUCCESS) {
printf(STDERR "%sn", $client->error());
exit(1);
}
Sunday, 25 October 2009
43. simple client (2)
my $input = shift || 'teststring';
my ($return, $result) =
$client->do("reverse", $input);
if ($return == GEARMAN_SUCCESS) {
printf("Result=%sn", $result);
}
Sunday, 25 October 2009
59. remote sandbox worker (4)
sub make_sandbox {
my ($job, $options) = @_;
my $command = $job->workload();
chomp $command;
print STDERR "<$command>n";
my $result = qx(make_sandbox $command) ;
if ($?) {
print $result, "n";
return "Error ($!)n";
}
else {
return "okn"
}
}
Sunday, 25 October 2009
60. Client to install
Remote sandboxes
•add server
•run remote installation on master
•run remote installation on each slave
•run configuration queries on slaves
Sunday, 25 October 2009
61. sample call to remote worker
my $ret =
qx(echo "$MASTER_SANDBOX" |
gearman -h $master_ip -f make_sandbox ) ;
# using the gearman command line utility
Sunday, 25 October 2009
62. More on similar matters
• FOSS.MY, Kuala Lumpur, Malaysia, 25-Oct-2009
• CodeBits, Lisbon, Portugal, 3-4-5-Dec-2009
http://datacharmer.blogspot.com
http://gearman.org
Sunday, 25 October 2009
63. THANKS
Let's talk!
This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License. To view a copy of this license, visit http://
creativecommons.org/licenses/by-sa/3.0/ or send a letter to Creative Commons, 171 Second Street, Suite 300, San Francisco, California, 94105, USA.
Sunday, 25 October 2009