gearman

1. Gearman
'The manager'
”since it dispatches jobs to be done,
but does not do anything useful itself.”

2. Presentation done using info from...
http://www.slideshare.net/pcdinh/gearman-and-asynchronous-processing-in-php-applications-6135047
http://assets.en.oreilly.com/1/event/45/The%20Gearman%20Cookbook%20Presentation.pdf
http://www.gearman.org
http://nz.php.net/manual/en/book.gearman.php
others...

3. Scalable Solutions..

More Hardware

Caching

Precalculated Data

Load Balancing

Multi-tier application

Job Queue

4. History

Created by Danga Interactive.

Some company that developed Memcache.

Original implementation in perl (2005).

2008 rewriteen in C by Brian Aker

PHP Extension by James Luedke

5. Used by
Digg: 45+ Servers, 400K jobs/day
Yahoo: 60+ servers, 6M jobs/day
And many others..

6. Installing
Compiling:
tar xzf gearmand-X.Y.tar.gz
cd gearmand-X.Y
./configure
make
make install
Starting server:
$ gearmand -d
Pecl extension:
tar xzf gearman-X.Y.tgz
cd gearman-X.Y
phpize
./configure
make
make install
To add to the php.ini:
extension="gearman.so"

7. Terminology

Client: Create jobs to be run and send them to a job server.

Worker: Run jobs given from the job server.

Job Server: Handle the job queue form clients to workers.

8. Gearman is...
“A massively distributed,
massively fault tolerant
fork mechanism.”
- Joe Stump, SimpleGeo

9. Features

Open Source.

Simple & Fast.

Multi-language.

Flexible application design.

Load Balancing.

No single point of failure.

10. Client Client Client Client
Job Server Job Server
Worker Worker Worker Worker

11. Queue Options

Memory

Memcached

Mysql/Drizzle

PostgreSQL

SQLite

Tokio Cabinet

12. Foreground
(synchronus)
Or
Background
(asynchronus)

13. Gearman Client
Fishpond_Controller_Front::getResource('gearman')
->getGearmanClient()
->doBackground("updateCompetitorPrice", $this->_barcode);
->do("updateCompetitorPrice", $this->_barcode);

GearmanClient::do() - Run a single task and return a result

GearmanClient::doLow() - Run a single low priority task

GearmanClient::doBackground() - Run a task in the background

GearmanClient::doHighBackground() - Run a high priority task in the background

GearmanClient::doLowBackground() - Run a low priority task in the background

14. Strategies

Scatter / Gather.

Map / Reduce.

Asynchronus Queues.

Pipeline Processing.

15. Scatter / Gather
Client
Product Detail Recomendations
Price Calculation Image Resize

16. $client = Fishpond_Controller_Front::getResource('gearman')
->getGearmanClient();
//adding gearman tasks
$client->addTask("getProductDetail", $barcode);
$client->addTask("getPrice", $barcode);
$client->addTask("resizeImage", serialize($barcode,100,100));
$client->addTask("getRecomendations", $barcode);
//callbacks to know when this finish
$client->setCompleteCallback(array($this, "complete"));
//runing tasks
$client->runTasks();
/**
* Callback when task is complete
*
*/
public function complete($task)
{
$data = $task->data();
}

17. Task Methods

GearmanClient::addTaskHigh() - Add a high priority task to run in parallel

GearmanClient::addTaskLow() - Add a low priority task to run in parallel

GearmanClient::addTaskBackground() - Add a background task to be run in parallel

GearmanClient::addTaskHighBackground() - Add a high priority background task to be run in parallel

GearmanClient::addTaskLowBackground() - Add a low priority background task to be run in parallel

GearmanClient::runTasks() - Run a list of tasks in parallel

18. Client Callback
GearmanClient::setDataCallback() - Callback function when there is a data packet for a task
GearmanClient::setCompleteCallback() - Set a function to be called on task completion
GearmanClient::setCreatedCallback() - Set a callback for when a task is queued.
GearmanClient::setExceptionCallback() - Set a callback for worker exceptions.
GearmanClient::setFailCallback() - Set callback for job failure.
GearmanClient::setStatusCallback() - Set a callback for collecting task status.
GearmanClient::setWarningCallback() - Set a callback for worker warnings.
GearmanClient::setWorkloadCallback() - Set a callback for accepting incremental data updates

19. Scatter / Gather

Concurrent tasks with different workers.

All tasks run in the time for longest running.

Must have enough workers available.

20. Map/Reduce
Client
Task T
Task T-0 Task T-1 Task T-2 Task T-3
Task T-00 Task T-01 Task T-02

21. Asynchronous Queues
No everyting need inmediate procesing..

Competitor pricing.

Emails.

Whole price engine.

Loging.

Etc.
Example:
$gearmanClient = Fishpond_Controller_Front::getResource('gearman')->getGearmanClient();
$gearmanClient->doBackground("updateCompetitorPrice", $this->_barcode);

22. Loging
<VirtualHost *:80>
ServerName example.com
DocumentRoot /var/www/
CustomLog “| gearman -n -f looger” common (client)
</VirtualHost>

23. Pipeline Procesing
Client
Task T
Worker Worker Worker
Operation 1 Operation 2 Operation 3
Output

24. Questions ?