Building an Incredible Machine with Generators and Pipelines in PHP

1. Building a J-Meter
With Generators and Pipelines in PHP
by Dan Leech
@dantleech

2. g
b

4. PHPBench

5. /**
* @Revs(1000)
* @Iterations(10)
*/
class HashingBench
{
public function benchMd5()
{
md5('hello world');
}
public function benchSha1()
{
sha1('hello world');
}
}

7. No way to interpolate tests

8. Three Benchmarks

9. Linear Execution

10. Linear Execution

11. Interpolated Execution
The results are now consistently inconsistent
PHPBench will report a high standard deviation, and you will
know to start again.

12. We want more
● Concurrent scheduling (load tests)
● Parameter / data generation
● HTTP sampling
● Even better progress loggers

14. Can I do this with Iterators?

15. Can I do this with Generators?

16. Generators

17. PHP 5.5.0
20 June 2013

18. Generators are Iterators

20. interface Iterator extends Traversable {
public function current();
public function next();
public function key();
public function valid();
public function rewind();
}

21. class PointlessIterator implements Iterator {
private $array;
public function __construct(array $array) {
$this->array = $array;
}
function rewind() {
return reset($this->array);
}
function current() {
return current($this->array);
}
function key() {
return key($this->array);
}
function next() {
return next($this->array);
}
function valid() {
return key($this->array) !== null;
}

22. $pointless = [ 'a' => 1, 'b' => 2 ];
$pointless = new ArrayIterator($pointless);
foreach ($pointless as $key => $value) {
echo "$key = $value";
}
// outputs
// a = 1
// b = 2

23. Generators are easy Iterators
That you cannot rewind
But that's OK, because they do cool stuff

24. final class Generator implements Iterator {
function rewind() {}
function valid() {}
function current() {}
function key() {}
function next() {}
function send($value) {}
function throw(Exception $exception) {}
function getReturn() {}
}

25. final class Generator implements Iterator {
function rewind() {}
function valid() {}
function current() {}
function key() {}
function next() {}
function send($value) {}
function throw(Exception $exception) {}
function getReturn() {}
}
Iterator

26. final class Generator implements Iterator {
function rewind() {}
function valid() {}
function current() {}
function key() {}
function next() {}
function send($value) {}
function throw(Exception $exception) {}
function getReturn() {}
}
Generator

27. YIELD

28. function numbers(): Generator
{
yield "1";
yield "2";
yield "3";
}
var_dump(numbers() instanceof Iterator);
// true
foreach (numbers() as $number) {
echo $number;
}
// 123

29. $generator = new ArrayIterator([1, 2, 3]);
foreach (numbers() as $number) {
echo $number;
}
// 123

30. Yielding control

31. <?php
function lines($fileName)
{
$h = fopen($fileName, 'r');
$lines = [];
while (false !== $line = fgets($h)) {
$lines[] = $line;
}
fclose($h);
return $lines;
}
foreach (lines() as $line) {
echo $line;
}

32. <?php
function lines($fileName)
{
$h = fopen($fileName, 'r');
while (false !== $line = fgets($h)) {
yield $line;
}
fclose($h);
}
$generator = lines();
foreach ($generator as $line) {
echo $line;
// do something else
}

33. Generators make functions interruptible

34. final class Generator implements Iterator {
function rewind() {}
function valid() {}
function current() {}
function key() {}
function next() {}
function send($value) {}
function throw(Exception $exception) {}
function getReturn() {}
}
Generator

35. send()

36. send()
function myGenerator(): Generator
{
$data = yield;
yield $data;
}
echo myGenerator()->send('Goodbye');
// Goodbye
Fast forwards to first yield returns
the sent value

37. send()
<?php
function echoGenerator(): Generator
{
$data = yield;
$data = yield $data;
}
$generator = echoGenerator();
var_dump($generator->send('Hello'));
var_dump($generator->send('Goodbye'));
var_dump($generator->send('Adios'));

38. send()
string(5) "Hello"
NULL
NULL

39. throw()

40. throw()
function myGenerator(): Generator
{
try {
yield;
} catch (MyException $e) {
echo 'Hello!';
}
}
echo myGenerator()->throw(new MyException());
Exception thrown here

41. invalid generator
function myGenerator(): Generator
{
try {
yield;
} catch (MyException $e) {
echo 'Hello!';
}
}
$generator = myGenerator();
$generator->next();
echo myGenerator()->throw(new MyException());
Exception thrown here

42. getReturn()

43. getReturn
<?php
function hello(): Generator
{
yield 'hello';
yield 'goodbye';
return 'ciao';
}
$generator = hello();
echo $generator->getReturn();

45. PHP Fatal error: Uncaught Exception: Cannot get
return value of a generator that hasn't returned

46. getReturn
<?php
function hello(): Generator
{
yield 'hello';
yield 'goodbye';
return 'ciao';
}
$generator = hello();
$generator->next()
$generator->next();
echo $generator->getReturn ();
// ciao

47. Generators will not return the return value
Unless you call getReturn()
After the generator has returned

48. Asynchronous Operations

50. Async Operations
function requestUrl($url): Generator {
$process = new Process("curl -I $url");
$process->start();
while ($process->isRunning()) {
echo $url . "n";
yield false;
}
yield $process->getOutput();
}

51. Async Operations
$generators = [
requestUrl('https://inviqa.com'),
requestUrl('https://bbc.co.uk'),
];
while ($generators) {
foreach ($generators as $i => $gen) {
if (false !== $result = $gen->current()) {
var_dump($result);
}
$task->next();
if (false === $gen->valid()) {
unset($generators[$i]);
}
}
}

52. https://www.inviqa.com
https://www.inviqa.com
http://www.bbc.com
http://www.bbc.com
https://www.inviqa.com
http://www.bbc.com
https://www.inviqa.com
http://www.bbc.com
https://www.inviqa.com
http://www.bbc.com
https://www.inviqa.com
http://www.bbc.com
https://www.inviqa.com
http://www.bbc.com
https://www.inviqa.com
http://www.bbc.com
https://www.inviqa.com
http://www.bbc.com
https://www.inviqa.com
http://www.bbc.com
https://www.inviqa.com

53. string(344) "HTTP/1.1 301 Moved Permanently
Date: Tue, 20 Feb 2018 21:21:23 GMT
Connection: keep-alive
Cache-Control: max-age=3600
Expires: Tue, 20 Feb 2018 22:21:23 GMT
Location: http://inviqa.com/
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-
cgi/beacon/expect-ct"
Server: cloudflare
CF-RAY: 3f0483ebdb3a2d6b-TXL
string(612) "HTTP/1.1 200 OK
Server: Apache
Content-Type: text/html
Expires: Tue, 20 Feb 2018 21:21:32 GMT
Content-Language: en
Etag: "25e4bf9a0519a562c8fb0a6379044365"
X-PAL-Host: pal143.back.live.telhc.local:80
Content-Length: 209756
Date: Tue, 20 Feb 2018 21:21:23 GMT
Connection: keep-alive
Set-Cookie: BBC-
UID=055ac8cca981056306da9a3821c959a1b1d799b4875494361aa037e2fe86c6080curl/7.55.1;
expires=Sat, 19-Feb-22 21:21:23 GMT; path=/; domain=.bbc.com
Cache-Control: private, max-age=60
X-Cache-Action: HIT
X-C"...

54. Go and look at event loops

55. AMP
Amp is a non-blocking concurrency
framework for PHP. It provides an event
loop, promises and streams as a base for
asynchronous programming.
Promises in combination with generators
are used to build coroutines, which allow
writing asynchronous code just like
synchronous code, without any callbacks.
https://github.com/amphp/amp

56. AMP
Amp is a non-blocking concurrency
framework for PHP. It provides an event
loop, promises and streams as a base for
asynchronous programming.
Promises in combination with generators
are used to build coroutines, which allow
writing asynchronous code just like
synchronous code, without any callbacks.
https://github.com/amphp/amp

57. Infinite Potential

58. while (true)

59. <?php
function numbers($number = 0): Generator
{
while (true) {
yield $number++;
}
}
foreach (numbers() as $number) {
echo $number . ' ';
}
// 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37
38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55
56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73
74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91
92 93 94 95 96 97 98 99 100 101 102 103 104 105 106
107 108 109 110 111 112 113 114 115 116 117 118 ...

60. The number of iterations is no longer the concern
of the generator

61. Recursion
function myGenerator(): Generator
{
$count = yield;
while (true) {
$count = $count * 2;
$count = yield $count;
}
}
$generator = myGenerator();
$count = 1;
while (true) {
$count = $generator->send($count);
}

64. Pipelines

65. A pipeline is a sequence of stages.
Each stage has a specific job to do.

66. 1)Make each program do one thing well. To do a new job, build afresh
rather than complicate old programs by adding new "features".
2)Expect the output of every program to become the input to another, as yet
unknown, program. Don't clutter output with extraneous information. Avoid
stringently columnar or binary input formats. Don't insist on interactive
input.
3)Design and build software, even operating systems, to be tried early,
ideally within weeks. Don't hesitate to throw away the clumsy parts and
rebuild them.
4)Use tools in preference to unskilled help to lighten a programming task,
even if you have to detour to build the tools and expect to throw some of
them out after you've finished using them.
Unix Philosophy

67. 1)Make each program do one thing well. To do a new job, build afresh
rather than complicate old programs by adding new "features".
2)Expect the output of every program to become the input to another, as yet
unknown, program. Don't clutter output with extraneous information. Avoid
stringently columnar or binary input formats. Don't insist on interactive
input.
3)Design and build software, even operating systems, to be tried early,
ideally within weeks. Don't hesitate to throw away the clumsy parts and
rebuild them.
4)Use tools in preference to unskilled help to lighten a programming task,
even if you have to detour to build the tools and expect to throw some of
them out after you've finished using them.
Unix Philosophy

68. Unix Pipes

69. $ echo '<?php echo "hello";' | php
hello

70. This is a simple phraze this is
Spell Checker
$ echo 'This is a a simple phraze this is'
Command
Output

71. This
is
a
simple
phraze
this
is
Spell Checker
$ echo 'This is a simple phraze this is' |
words
Command
Output

72. this
is
a
simple
phraze
this
is
Spell Checker
$ echo 'This is a simple phraze' |
words |
lowercase
Command
Output

73. a
is
is
phraze
simple
this
this
Spell Checker
$ echo 'This is a simple phraze' |
words |
lowercase |
sort
Command
Output

74. a
is
phraze
simple
this
Spell Checker
$ echo 'This is a simple phraze' |
words |
lowercase |
sort |
unique
Command
Output

75. phraze
Spell Checker
$ echo 'This is a simple phraze' |
words |
lowercase |
sort |
unique |
mismatch dictionary.txt
Command
Output

76. GNU is not Unix
Unix GNU
words fmt -1
lowercase tr '[:upper:]' '[:lower:]'
sort sort
unique uniq
mismatch dictionary.txt comm -23 - dictionary.txt

77. Working together + Specialisation

78. Pipelines in PHP

79. Input
Callable
Stage
Output
Stage
Stage

80. Input
Callable
Callable
Output
Callable
Callable function ($value) { return $value * 2; }
10
20
function ($value) { return $value +1; }
21
function ($value) { return $value * 2; }
42

81. $pipeline = (new Pipeline)
->pipe(function ($value) {
return $value * 2;
})
->pipe(function ($value) {
return $value +1;
})
->pipe(function ($value) {
return $value * 2;
})
;
$pipeline->process(10);

82. class TimesTwoStage {
public function __invoke($value) {
return $value * 2;
}
}
class AddOneStage {
public function __invoke($value) {
return $value + 1;
}
}

83. $pipeline = (new Pipeline)
->pipe(new TimeTwoStage())
->pipe(new AddOneStage())
->pipe(new TimeTwoStage());
$pipeline->process(10);

84. Stages can be Generators

85. function Stage1(): Generator
{
$payload = yield;
while (true) {
$data = yield data;
}
}

87. Let's build a thing!

89. ● Performance Testing
● Load Testing
● Acceptance Testing
● Cache Testing

90. In the beginning there was the
Test Plan

92. Samplers
<update screenshots , samplers should be
introduced before threads>

95. Threads
a.k.a Users

97. Variables

101. Assertions

104. Live Reports

107. Let's Build a J-Meter
with Generators and Pipelines
in PHP

112. Blog Posts
●
https://nikic.github.io/2012/12/22/Cooperative-multitasking-using-coroutines-in-
PHP.html
● https://blog.ircmaxell.com/2012/07/what-generators-can-do-for-you.html
●
https://speakerdeck.com/chrispitt/cooperative-multitasking-with-generators
– https://www.youtube.com/watch?v=cY8FUhZvK7w
● https://medium.com/@aaronweatherall/the-pipeline-pattern-for-fun-and-profit-
9b5f43a98130
●
https://www.cise.ufl.edu/research/ParallelPatterns/PatternLanguage/Algorithm
Structure/Pipeline.htm
● https://markbakeruk.net/2016/01/19/a-functional-guide-to-cat-herding-with-php-
generators/
●
http://sergeyzhuk.me/2018/02/15/amp-coroutines/

113. Videos
● Cooperative multitasking with Generators:
https://www.youtube.com/watch?v=cY8FUhZvK7w
● Pushing the limits of PHP with React:
https://www.youtube.com/watch?
v=fQxxm4vD8Ok&feature=youtu.be
● Herding Cats: https://vimeo.com/189755262
● What the heck is the event loop anyway?:
https://www.youtube.com/watch?v=8aGhZQkoFbQ

114. Thanks