Slides from my talk at Open Source Bridges

1. Gearman
From the Worker's Perspective
Tuesday, June 21, 2011

2. /usr/bin/whoami
Brian Aker
CTO/Founder Data Differential
Previously MySQL, Slashdot, Sun Microsystems
Tuesday, June 21, 2011

3. What is Gearman?
Tuesday, June 21, 2011

4. “The way I like to think of Gearman is a
massively distributed fork mechanism”
-Joe Stump, Digg
“The Not Mechanical Turk”
-Don MacAskill, SmugMug
Tuesday, June 21, 2011

5. resize_image()
do (“resize_image”) {
…
return $image;
}
Tuesday, June 21, 2011

6. !"#$%#&'()#*+
! !
Tuesday, June 21, 2011

7. !"#$%&$'()&$*&+,-./
! 01"2)3+&$4$3)&51)6-5+3$7+12"-&$&8&5+9
! :45-14.$."43$64.47;)7<
" ="1>+1&$41+$7"5),)+3$473$4&>$,"1$#"1>?$7"5$,"1;+3
! @-.5)A.47<-4<+$)75+<145)"7
! B)&51)6-5+$C1";+&&)7<
" 0"&&)6.8$;."&+1$5"$3454
! D87;(1"7"-&$473$4&87;(1"7"-&$E-+-+&
! !
Tuesday, June 21, 2011

8. Server
Provides Asynchronous and Synchronous Requests
Restarts Work
Durable Requests
Gearman Protocol/HTTP
Epoch Scheduling
Logging
Tuesday, June 21, 2011

9. Client
# Create our client object.
$gmclient= new GearmanClient();
# Add default server (localhost).
$gmclient->addServer();
$result= $gmclient->do("reverse", "Hello!");
echo "Success: $resultn";
Tuesday, June 21, 2011

10. Worker
# Create our worker object.
$gmw= new GearmanWorker();
# Add default server (localhost).
$gmw->addServer();
$gmw->addFunction("reverse", "reverse_fn");
while ($gmworker->work()) {…}
Tuesday, June 21, 2011

11. Worker Function
function reverse_fn($job)
{
$workload= $job->workload();
$result= strrev($workload);
return $result;
}
Tuesday, June 21, 2011

12. Lots of functions...
$gmw->addFunction("resize", "resize_fn");
$gmw->addFunction("grep", "grep_fn");
$gmw->addFunction("fetch_url", "fetch_url");
Tuesday, June 21, 2011

13. Function
gearman_return_t fetch_url(gearman_job_st *job, void*)
{
const char *workload= gearman_job_workload(job);
size_t workload_size= gearman_job_workload_size(job);
gearman_job_send_status(job, 0, 100);
…
gearman_job_send_data(job, chunk, sizeofchunk);
…
gearman_job_send_status(job, 50,100);
… if (issue_warning)
gearman_job_warning(job, “I'm sorry, Dave. I'm afraid I can't do that.”, size);
return GEARMAN_SUCCESS;
}
Tuesday, June 21, 2011

14. Worker Return
GEARMAN_SUCCESS
GEARMAN_FATAL
GEARMAN_ERROR
GEARMAN_SHUTDOWN
Tuesday, June 21, 2011

15. reduce() {…}
map(list[…],
reduce()); map() {…}
reduce() {…}
reduce() {…}
Tuesday, June 21, 2011

16. Map @#$@# ?
Tuesday, June 21, 2011

17. find()
Partitioning
find()
{A...K}
{L...Q}
{R...Z} find()
Tuesday, June 21, 2011

18. Partitioning
gearman_return_t split_worker(gearman_job_st *job, void* /* context */)
{
const char *workload= gearman_job_workload(job);
size_t workload_size= gearman_job_workload_size(job);
const char *chunk_begin= workload;
for (size_t x= 0; x < workload_size; x++)
{
if (workload[x] == 0 or workload[x] == ' ')
{
gearman_job_send_data(job, chunk_begin,
workload +x -chunk_begin);
chunk_begin= workload +x +1;
}
}
return GEARMAN_SUCCESS;
}
Tuesday, June 21, 2011

19. $result
Aggregation
$result
+ result
+ result
+ result
$result
= sum result
Tuesday, June 21, 2011

20. Aggregation
gearman_return_t cat_aggregator (gearman_aggregator_st *,
gearman_task_st *task,
gearman_result_st *result)
{
std::string string_value;
do
{
gearman_result_st *result_ptr= gearman_task_result(task);
string_value.append(gearman_result_value(result_ptr),
gearman_result_size(result_ptr));
} while ((task= gearman_next(task)));
gearman_result_store_value(result, string_value.c_str(),
string_value.size());
return GEARMAN_SUCCESS;
}
Tuesday, June 21, 2011

21. Do we have to
partition?
(What other tricks exist!)
Tuesday, June 21, 2011

22. Pipeline
Store() Resize() Publish()
Tuesday, June 21, 2011

23. CPU? poll()
Tuesday, June 21, 2011

24. Tuesday, June 21, 2011

25. multiple languages
Tuesday, June 21, 2011

26. Namespaces
Foo::resize_image() Acme::resize_image()
{ {
… …
return $image; return $image;
} }
Tuesday, June 21, 2011

27. Future
0.22 Released
Custom Logging Plugins
Client/Worker Configuration
Extended Administrative Commands
SSL
Job Result Cache
Uplift!
Tuesday, June 21, 2011

28. • Gearman.info (up to date)
• Gearman.org (...)
• http://launchpad.net/gearmand/
• twitter: brianaker
• blog: blog.krow.net
Tuesday, June 21, 2011