Gearman work queue in php 用 Gearman 打造 PHP 排程

1. Gearman work queue in php
用 Gearman 打造 PHP 排程
2013.10.05
@appleboy

2. 2
Who am I
Bo-Yi Wu @appleboy
http://blog.wu-boy.com
https://github.com/appleboy
任職於瑞昱半導體 RealTek(IC Design House)

3. 3
今日來聊聊排程系統

4. 4
排程使用時機 ?

5. 5
不需要即時回覆
都可以使用排程

6. 6
哪些應用需要排程 ?

7. 7
使用者註冊系統

8. 8
使用者註冊
填寫表單完成
寄發認證信
使用者收到回覆
未使用排程
處理資料
透過第三方寄信
Amazone SES

9. 9
缺點 Response time 過長
使用者不需要等伺服器寄信
( 來自知名插畫家彎彎 )

10. 10
使用者註冊
填寫表單完成
寄發認證信
使用者收到回覆
使用排程
處理資料
寫入排程

11. 11
Email Queue?
Github Social Coding

12. 12
Application Server
Database Database Database
Work Queue

13. 13
執行排程
Linux crontab
Ref: http://www.xenstreet.com/2013/03/choosing-the-right-linux/

14. 14
<?php
$link = mysqli_connect("host", "user", "passw", "db");
$query = 'SELECT * FROM jobs where status = "on"';
$result = $link->query($query);
while($row = mysqli_fecth_array($result)) {
// …....
// do work
// …....
$sql = 'update jobs set status = "off" where id = {$row["id"]}';
$link->query($sql);
}

15. 15
每分鐘執行一次
*/1 * * * * root cron.sh

16. 16
優點
●
容易瞭解
●
任何程式語言都可以實現
●
快速 Deploy 到任何機器

17. 17
缺點
●
每分鐘只能執行一次
●
Race condition
●
浪費 Database 資源

18. 18
換個寫法來試試看

19. 19
每 10 秒執行一次
<?php
$link = mysqli_connect("myhost","myuser","mypassw","mybd");
$query = 'SELECT * FROM jobs where status = "on"';
$done = false;
while(!$done) {
$result = $link->query($query);
if ($result->num_rows > 0) {
// do work
$sql = 'update jobs set status = "off" where id = {$row["id"]}';
$link->query($sql);
} else {
sleep(10);
}
}

20. 20
缺點
●
真的 10 秒執行一次 ?
●
Race condition
●
更浪費 Database 資源

21. 21
如果沒有 Race Condition

22. 22
#!/bin/sh
(sleep 5 && /path/script.sh) &
(sleep 15 && /path/script.sh) &
(sleep 25 && /path/script.sh) &
(sleep 35 && /path/script.sh) &
(sleep 45 && /path/script.sh) &
(sleep 55 && /path/script.sh) &
每 10 秒執行一次

23. 23
如何分配處理這些問題 ?

24. 24
Using Gearman

25. 25
What is Gearman?
It allows you to do work in
parallel, to load balance
processing, and to call
functions between languages.

26. 26
Client in
Perl,Python,PHP
Java,C#(.Net)

27. 27
even
MySQL,PostgreSQL

28. 28
How Does Gearman Work?

29. 29
Gearman 優點

30. 30
Work Queues
●
淺顯易懂
Got job do job repeat→ →
Application
Gearmand
Worker
Got job
Do job

31. 31
Work Queues
●
淺顯易懂
Get job do job repeat→ →
●
跨語言
任何平台只要能存入字串即可

32. 32
Multiple Language
Application
Gearmand
Application
Worker Worker

33. 33
Work Queues
●
淺顯易懂
Get job do job repeat→ →
●
跨語言
任何平台只要能存入字串即可
●
跨平台
當 Client 寫入工作時， Worker 不需要啟動

34. 34
跨平台
Application
Gearmand
Application
Gearmand

35. 35
Work Queues
●
淺顯易懂
Get job do job repeat→ →
●
跨語言
任何平台只要能存入字串即可
●
跨平台
當 Client 寫入工作時， Worker 不需要啟動
●
即時性
可以為同步或非同步

36. 36
Gearman Architecture
佈署架構

37. 37
Application
Gearmand
Job

38. 38
Application
Gearmand
Job
Worker
Job

39. 39
Application
Gearmand
Job
Worker
Job Response

40. 40
Application
Gearmand
Job
Worker
Job Response
Response

41. 41
Application
Gearmand
Worker
Worker
Worker

42. 42
Application
Gearmand
Worker
Worker
Gearmand

43. 43
Application
Gearmand
Worker
Worker
Gearmand
Application

44. 44
結論 :Gearman 擴充性佳

45. 45
不論要幾台 gearmand
跟數個 worker 都可以

46. 46
Application 只會跟
Gearmand 溝通

47. 47
多個 Application 不管
丟出多少個 Jobs

48. 48
最終由多個 Job Server
分配工作

49. 49
Synchronous
vs
Asynchronous

50. 50
Synchronous
Application
Gearmand
Job
Worker
Job Response
Response

51. 51
Asynchronous
Application
Gearmand
Job
Worker
Job Done!

52. 52
Installation
安裝方式

53. 53
Installation
●
Job Server (C)
●
Client library (PHP,Python ...)
●
Worker library (PHP,Python ...)

54. 54
Job Server
安裝方式

55. 55
Ubuntu/Debian
aptitude -y install gearman gearman-job-server libgearman-dev libdrizzle0

56. 56
CentOS/Fedora
yum -y install libgearman gearmand libgearman-devel libdrizzle

57. 57
Tarball install
https://gist.github.com/appleboy/6774735

58. 58
Gearmand enabled using
MySQL(Mariadb)
Postgresql
SQL Lite
Memcached

59. 59
MySQL Example
/etc/default/gearman-job-server
-q mysql
--mysql-host=localhost
--mysql-user=xx
--mysql-password=xx
--mysql-db=ovoq
--mysql-table=gearman_queue

60. 60
Client Library
pecl install channel://pecl.php.net/gearman-1.1.2
echo 'extension=gearman.so' > /path/gearman.ini

61. 61
Quick Test
php -i | grep 'gearman'

62. 62
Gearman in PHP Client

63. 63
PHP Client
# Create a client object
$gm = new GearmanClient();
# Add default server, default is localhost:4730
$gm->addServer();
# Do Job
$result = $gm->doNormal("reverse", "Hello!");
echo "Success: $resultn";

64. 64
Do background Job
$job = $gm->doBackground("reverse", "apple");

65. 65
Gearman in PHP Worker

66. 66
PHP Worker
# Define worker job function
function reverse_fn($job) { //.... }
# fetch message from client
function reverse_fn($job) {
$data = $job->workload();
}

67. 67
PHP Worker
# Create our worker object.
$gm= new GearmanWorker();
# Add default server (localhost).
$gm->addServer();
# Register function "reverse"
$gm->addFunction("reverse", "reverse_fn");
# Waiting for job...
while($gm->work());

68. 68
Multiple Jobs
$gm->addFunction("reverse", "reverse_fn");
$gm->addFunction("resize", "resize_fn");
$gm->addFunction("queue", "queue_fn");
…...

69. 69
Anonymous function
$gm->addFunction("reverse", function($job){
$data = $job->workload();
});

70. 70
Gearman in CodeIgniter

71. 71
CodeIgniter Gearman Library
https://github.com/appleboy/CodeIgniter-Gearman-Library

72. 72
Installation via Spark
php tools/spark install -v1.0.1 codeigniter-gearman

73. 73
Setup Gearmand Server and Port
config/gearman.php
$config['gearman_server'] = array('localhost');
$config['gearman_port'] = array('4730');

74. 74
CodeIgniter Client

75. 75
Implement Image Resize Job
create app controller

76. 76
class App extends CI_Controller
{
public function resize_sync()
public function resize_async()
}

77. 77
public function resize_sync()
{
$data = json_encode(array('file_name' => 'test.png'));
$client = $this->lib_gearman->gearman_client();
do {
$this->lib_gearman->do_job_foreground('resize', $data);
switch($client->returnCode()) { //.... }
} while ($client->returnCode() != GREARMAN_SUCCESS);
}

78. 78
class App extends CI_Controller
{
public function resize_sync()
public function resize_async()
}

79. 79
public function resize_async()
{
$data = json_encode(array('file_name' => 'test.png'));
$client = $this->lib_gearman->gearman_client();
$this->lib_gearman->do_job_background('resize', $data));
}

80. 80
CodeIgniter Worker
create cli controller

81. 81
class Cli extends CI_Controller
{
public static function image_resize()
public static function email_queue()
public function worker()
}

82. 82
public static function image_resize($job)
{
$data = json_decode($job->workload(), true);
$ci =& get_instance();
$ci->image_lib->initialize($config);
$ci->image_lib->resize();
$ci->image_lib->clear();
}

83. 83
Define Worker

84. 84
public function worker()
{
$worker = $this->lib_gearman->gearman_worker();
$this->lib_gearman
->add_worker_function('resize', 'Cli::image_resize');
$this->lib_gearman
->add_worker_function('queue', 'Cli::email_queue');
while ($this->lib_gearman->work());
}

85. 85
Start Worker
$ php index.php cli worker

86. 86
Running workers via script
#!/bin/sh
script="index.php cli worker"
folder="/script_path/"
max_worker="5"
php_bin="/usr/bin/php"
count=0
current_count=$(ps aux | grep "index.php cli worker" | grep -v "grep" | wc -l)
if test $current_count -lt $max_worker
then
cd $folder
$php_bin $folder$script &
else
echo "The worker count has been limited."
fi
https://gist.github.com/appleboy/6806323

87. 87
About Job Data
Don't pass 2GB object.
Gearman is not data center.

88. 88
Keep your message data small
Amazone SQS message siez: 256K
http://goo.gl/0MBq8W

89. 89
Scalability

90. 90
Client Client
Job Server
Client
ClientWorker Worker Worker Worker
Job Server
Client

91. 91
you can scale out your
clients and workers
as needed

92. 92
Client Setting
Client1, Client2: Gearman1
Client3, Client4: Gearman2
…...

93. 93
Worker Setting
Worker[1-4]:Server[1-2]
…...

94. 94
Add new Job Server...

95. 95
Reconfigure all setting
(client and worker program)

96. 96
Reconfigure all setting
(client and worker program)
even
Deploy Tool

97. 97
如何控制易怒的情緒?

98. 98
How to improve your
architecture?

99. 99
Client Client
Job Server
Client
ClientWorker Worker Worker Worker
Job Server
Client
Proxy (Haproxy ...)

100. 100
Live Demo
Email Queue.

101. 101
Any Questions?

102. 102
Thanks
全體工作人員和聽眾