This document discusses a patch written by Shota Fukumori to enable parallel testing in Ruby. The patch allows the "make test-all" command to spawn multiple worker processes to run tests concurrently. It overcame challenges like traps in tests, modifying stdin/out, and duplicate filenames. Performance tests showed the patch significantly reduced test runtime by utilizing additional processor cores. The patch was accepted and merged into Ruby.

1. Parallel running
`make test-all` of Ruby
Shota Fukumori (sora_h)

2. Self Introduction
• Ruby Committer (I became by this patch!)
• Rubyist
• 8th grade student; (in Japan JHS 2nd)
• see http://sorah.cosmio.net/ or
http://about.me/sorah

3. Agenda
• Background
• Implementation
• Result
• How to use

4. Background
• `make test-all` is slow
• Can’t run parallel
• I started writing a patch

5. Background
• Today I talk about between

6. Background

7. Background
• 4 days

8. Background
• 80% Project.

9. Windows

10. Windows
• In Windows, Ruby can’t use Kernel.fork
• Asked at IRC

11. Windows
sora_h: test-all
Windows
(Will you reject a patch doesn’t support
Windows if I wrote a patch?)

12. Windows
unak:
(Yes I’ll reject.)

13. Windows
• ... It’ll be rejected if I don’t support Windows.
• So I used IO.popen
• but currently this feature doesn’t work at
Windows. I’m working with it.

14. Traps in test

15. Trap?
• Doesn’t work well with parallel running
• Modify STDIN/OUT
• Duplicate filename/port number in 2+
testcases

16. ruby/test_signal.rb
• def have_fork?; ... Process.fork{} ...; end
• test/unit at_exit, at_exit, at_exit
• Added flag for at_exit

17. ruby/test_process.rb
• Modifies STDIN/OUT
• duplicate and save backup in parallel.rb

18. net/http/test_https.rb
• Port number is same as net/http/test_http.rb
--- a/test/net/http/test_https.rb
+++ b/test/net/http/test_https.rb
@@ -24,7 +24,7 @@ class TestNetHTTPS < Test::Unit::TestCase
CONFIG = {
'host' => '127.0.0.1',
- 'port' => 10081,
+ 'port' => 10082,
'proxy_host' => nil,
'proxy_port' => nil,
'ssl_enable' => true,

19. csv
• csv have duplicate temporary file name
--- a/test/csv/test_serialization.rb
+++ b/test/csv/test_serialization.rb
@@ -131,7 +131,7 @@ class TestCSV::Serialization < TestCSV
def test_io
test_class_dump
- data_file = File.join(File.dirname(__FILE__), "temp_test_data.csv")
+ data_file = File.join(File.dirname(__FILE__), "serialization_test_data.csv")
CSV.dump(@names, File.open(data_file, "wb"))
assert(File.exist?(data_file))

20. Implementation

21. Implementation
• http://github.com/sorah/ruby/blob/
parallel_test
• patch on test/unit
• because minitest/unit is outer project
• and minitest/unit doesn’t support multiple
file running?

22. How does it work
1. Load list of files.
2. Spawn workers.
3. Shift file from files and throw to worker
which is ready while files isn’t empty
4. Retry testcase failed at worker in master
5. Output results.

23. How does it work
• Master: `make test-all`
Worker: will be spawned by master process
ruby parallel.rb
worker
master IO.popen …
worker

24. How does it work
• Master: `make test-all`
Worker: will be spawned by master process
master
workers

25. How does it work
• Worker and Master are connected via
STDIN,OUT = IO.pipe, IO.pipe
worker
master pipe …
worker

26. How does it work
File List I’m ready
Run test.rb
master worker
Done <result>
I’m ready

27. How does it work
File List
test.rb 1 Fail worker
master worker Disk
……
worker

28. How does it work
File List
1 Fail worker
master
test.rb worker Disk
……
worker

29. How does it work
File List
test.rb
1 Fail worker
master worker Disk
……
worker

30. How does it work
File List
test.rb
1 Fail worker
master worker Disk
……
worker

31. How does it work
File List
test.rb
1 Fail worker
master worker Disk
……
worker

32. How does it work
File List
test.rb
1 Fail worker
master worker Disk
……
worker

33. Demo

34. Performance
TOTAL TESTCASES TESTS/S
150.0
112.5
75.0
37.5
0
no -j -j1 -j2 -j3 -j5 -j8 -j13
https://gist.github.com/834848 - thanks mrkn!

35. Performance
Fraction
TOTAL TESTCASES TESTS/S
3.00
2.50
2.00
1.50
1.00
0.50
0
no -j -j1 -j2 -j3 -j5 -j8 -j13
https://gist.github.com/834848 - thanks mrkn!

36. Discussion
• [ruby-dev:43222]
• [ruby-core:35294]
• #4415 [ruby-dev:43226]
• And I become a committer!
• Imported at r30939.

37. Testing
• Build ruby trunk (svn head)
• make TESTS=’-j N -v’ test-all

38. Testing
• If you found bug, please submit to
http://redmine.ruby-lang.org/
With assigning to “Shota Fukumori”.
• I’ll see and response to the ticket.

39. Todos
• Many bugs
• Windows
• Anyone can test?
• I can’t test on Win because Windows
machine (MacBook)’s HDD is strange... ;’(
• Refactoring (I’m working)

40. Apology
• Thanks mrkn (Kenta Murata)
• Providing Hints to me
• Taking performance data

41. See Also
• http://www.slideshare.net/mrkn/ruby-
workshop-sapporo16mrkn
• Ruby - Kenta Murata a.k.a. mrkn
• (Japanese)

42. Thanks!