Text
Page: 1
Portable and Fast -
How to implement a
parallel test runner
Tsutomu Katsube
RubyKaigi 2026
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 2
Tsutomu Katsube
✓ Job: Software Engineer
✓ Hobby: OSS Activities
✓ GitHub and X: @tikkss
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 3
Starting point
きっかけ
About 2 years ago
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 4
Red Data Tools
✓ A community to provide data processing
tools for Ruby
Ruby 用のデータ処理ツールを提供するコミュニティ
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 5
Red Datasets
✓ A RubyGem for easy use to common (ML)
datasets
機械学習などでよく使われるデータセットを簡単に使える gem
✓ Iris, MNIST, CIFAR, etc.
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 6
Slow test suite
✓ Tests became slower as the supported
datasets grew
サポートされるデータセットが増えるにつれて、テストが遅くなっていた
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 7
Discussion
✓ How to speed up the test suite such as
parallelizing them?
テストを並列実行するなどして高速化できないか？
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 8
First step
✓ Speed up each test
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 9
Why?
✓ Test2 is bottleneck even with parallelization
並列化したとしても、遅いテスト（Test2）がボトルネックになるから
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 10
Next step
✓ Parallelize the tests
Today’s topic
テストの並列実行です
✓ test-unit natively adds support for parallel test
running
test-unit 本体でテストの並列実行をサポートすることにした
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 11
Why?
✓ Sutou Kouhei
✓ Founder of Red Data Tools
✓ Maintainer of test-unit (as a bundled gem)
Day3 11:30 - 12:00 Small Hall
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 12
Plan
✓ [x] Thread
✓ [x] Multiprocess
Today’s topic
✓ [ ] Ractor
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 13
Basics
What is a parallel test
running?
並列テスト実行とはなんでしょうか？
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 14
Sequential running
Processing time
CPU1
Portable and Fast - How to implement a parallel test runner
Test1
Test2
Test3
Powered by Rabbit 4.0.1
Page: 15
Parallel running
Processing time
CPU1Test1
CPU2Test2
CPU3Test3
Portable and Fast - How to implement a parallel test runner
Free time!
Powered by Rabbit 4.0.1
Page: 16
Why portable?
✓ For working across environments
色々な環境で動いてほしいため
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 17
Why fast?
✓ For faster feedback loop
より速いフィードバックループのため
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 18
Topics
✓ Issues
課題
✓ Solutions
解決策
✓ Demo
デモ
✓ Future
今後のこと
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 19
Issues
✓ Portable
✓ Fast
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 20
Portable Requirements (1)
✓ Don’t depend on Unix-specific features such
as fork
fork のような Unix 固有の機能に依存しない
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 21
Launch a child process
✓ Kernel.#fork
✓ Kernel.#spawn
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 22
Portability
✓ Kernel.#fork
✓ Works only on Unix-based platforms
fork は Unix 系のみで動く
✓ Kernel.#spawn
✓ Also works on Windows
spawn は Windows でも動く
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 23
Overhead of launching a process
Processing time
CPU1
CPU2
CPU3
Portable and Fast - How to implement a parallel test runner
Test1
Over
head
Test2
Free
time!
Test3
Powered by Rabbit 4.0.1
Page: 24
Overhead
✓ Kernel.#spawn has more overhead than
Kernel.#fork
spawn は fork よりもオーバーヘッドが大きいので、速さは少々犠牲になる
✓ Because Kernel.#fork uses the COW (Copy on
Write)
fork はコピーオンライトを利用するので
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 25
Kernel.#fork
✓ Creates a copy of the process
fork はプロセスのコピーを作る
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 26
Kernel.#spawn
✓ Creates a clean environment (similar to
Ruby::Box)
spawn はまっさらな環境を作る（Ruby::Box.new に似ている）
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 27
Issues (1)
How to restore the
state of a parent
process?
親プロセスの状態をどうやって復元する？
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 28
Portable Requirements (2)
✓ Don’t depend on external libraries
外部ライブラリに依存しない
✓ Including default/bundled gems such as drb
drb のようなデフォルト/バンドル gem も含む
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 29
Multiprocess programming
✓ Network communication is required
ネットワーク通信が必要
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 30
How to communicate (1)
✓ Cannot be referenced across processes
プロセスを超えて参照できない
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 31
How to communicate (2)
✓ Share data over the network
ネットワークを経由してデータを共有する
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 32
drb (as a bundled gem)
✓ Can be referenced like an in-process
同じプロセス内のように参照できる
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 33
Lately
最近
✓ Standard Library is reduced
標準ライブラリは減らされて
✓ Extract them to default/bundled gems
デフォルト/バンドル gem に分離されている
Core features
(no require)
Portable and Fast - How to implement a parallel test runner
Default gems
Bundled gems
(with require)
Powered by Rabbit 4.0.1
Page: 34
Issues (2)
✓ How to design network communication
protocol?
通信プロトコルをどう設計する？
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 35
Portable Requirements (3)
✓ Don’t break backward compatibility
後方互換性を壊さない
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 36
Testing framework
✓ Expected to work on various environments
色々な環境で動くことが期待されている
✓ test-unit of CI: CRuby 2.1 ~ head, JRuby,
TruffleRuby
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 37
If breaks backward compatibility
✓ It’s unhappy for users
ユーザーにとっては嬉しくない
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 38
Version up
✓ Users may need to change their codes and
test at once
ユーザーは自身のコードとテストを同時に変更する必要があるかもしれない
✓ For new Ruby and testing framework
新しい Ruby とテスティングフレームワークに対応するために
✓ This makes it harder to identify the cause of
errors
エラーの原因を特定することが難しくなる
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 39
Issues (3)
✓ How to support parallelization with
backward compatibility?
後方互換性を維持しながら、どうやって並列実行をサポートする？
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 40
Issues
✓ Portable
✓ Fast
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 41
Fast Requirements
✓ Keep busy and reduce heavy fixture
暇をさせずに、重たい準備と後片付けを減らす
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 42
Idle
Processing time
CPU1
CPU2
Test1
Test4
Test2
Idle
CPU3
Portable and Fast - How to implement a parallel test runner
Test3
Powered by Rabbit 4.0.1
Page: 43
Keep busy
Processing time
CPU1
CPU2
CPU3
Portable and Fast - How to implement a parallel test runner
Test1
Test2
Test4
Test4
Test3
Powered by Rabbit 4.0.1
Page: 44
Fast Requirements
✓ Keep busy and reduce heavy fixture
暇をさせずに、重たい準備と後片付けを減らす
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 45
Fixture
✓ Sets up/tears down the state needed to run
consistently
テストを一貫して実行するために必要な準備や後片付けを行う
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 46
Test level fixture
✓ #setup/#teardown are called before/after
each test
テスト前後に setup と teardown が呼ばれる
class TestA < Test::Unit::TestCase
def setup; end
def teardown; end
def test1; end
def test2; end
def test3; end
end
# setup -> test1 -> teardown
# setup -> test2 -> teardown
# setup -> test3 -> teardown
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 47
Test case level fixture
✓ .startup/.shutdown are called before/after
each test case
テストケース前後に startup と shutdown が呼ばれる
✓ Tends to be heavy such as preparing database
データベースの準備など、処理が重くなりがち
class TestA < Test::Unit::TestCase
class << self
def startup; end
def shutdown; end
end
def test1; end
def test2; end
def test3; end
end
# startup -> test1 -> test2 -> test3 -> shutdown
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 48
One by one
✓ Most overhead, and bad balance
Processing time
CPU1startupTest1shutdown
CPU2startupTest2shutdown
Portable and Fast - How to implement a parallel test runner
startup
Test3
shutdown
Powered by Rabbit 4.0.1
Page: 49
All at once
✓ Less overhead, but bad balance
Processing time
CPU1
startup
Test1
Test2
Test3
shutdown
CPU2
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 50
Balance
✓ More overhead, but good balance
Processing time
CPU1startupTest1Test2
CPU2startupTest3shutdown
Portable and Fast - How to implement a parallel test runner
shutdown
Powered by Rabbit 4.0.1
Page: 51
Issues (4)
✓ How to keep busy and reduce heavy
fixtures?
どうやって暇をさせずに、重たい準備や後片付けを減らす？
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 52
Issues Summary
1. Use spawn for課題のまとめ
Windows: How to restore
state?
spawn は Windows でも動くけどまっさらな環境を作る。状態を復元するに
は？
2. Don’t depend on drb: Communication
protocol?
drb に依存しない。通信プロトコルはどうする？
3. Keep backward compatibility: How to
support parallelism?
後方互換性を維持しながら並列実行をサポートするには？
4. How to keep busy and reduce heavy
fixtures?
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 54
1. Use spawn for
Windows: How
to restore state?
spawn は Windows でも動くけどまっさらな環境を作る。状
態を復元するには？
Page: 55
test-unit’s features
✓ Collect tests
テストを集める
✓ Run tests
テストを実行する
✓ Report results
テスト結果をレポートする
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 56
Collect tests (1)
✓ Example: collects tests under a test directory
例: test ディレクトリ配下のテストを集める
$ tree test
test
└── test-a.rb
class TestA < Test::Unit::TestCase
def test1; end
def test2; end
end
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 57
tests
✓ test-unitCollect
builds a test
suite(2)
tree under given
base directory
test-unit は指定されたベースディレクトリ配下からテストスイートのツリーを構築する
$ test-unit test
test
TestA
test1
Portable and Fast - How to implement a parallel test runner
test2
Powered by Rabbit 4.0.1
Page: 58
Passing to child processes
✓ Pass given base directory to command
line arguments in spawn
指定されたベースディレクトリを spawn のコマンドライン引数経由で子プロセス
に渡す
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 59
Append to $LOAD_PATH
✓ Users can append directories to $LOAD_PATH
ユーザーは $LOAD_PATH にディレクトリを追加できる
✓ Also pass given $LOAD_PATH to command line
arguments in spawn
指定された $LOAD_PATH をspawn のコマンドライン引数経由で子プロセスに渡
す
$ test-unit test -I lib
or
$ test-unit test --load-path lib
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 60
Done: Issues (1)
✓ A test suite tree can be rebuild in child
processes
テストスイートのツリーが子プロセスで再構築できた
✓ Load all required files in each child process
各子プロセスで必要なファイルをすべてロードしている
✓ Wasteful but likely has minimal impact on total
speed
無駄が多いけど全体の速度にはほとんど影響なさそう
✓ Because collecting tests is not a bottleneck
テストを集めるのはボトルネックではないから
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 61
2. Don’t depend
on drb:
Communication
protocol?
drb に依存しない。通信プロトコルはどうする？
Page: 62
IPC (Inter-Process Communication)
✓ Pipe
✓ Portable/Faster/Local
✓ Unix domain socket
✓ Unix only/Fast/Local
✓ TCP/IP socket
✓ Portable/Slow/Local and Remote
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 63
Creates a pipe
✓ IO.pipe: Creates a pair of pipes
IO.pipe はパイプのペアを作成する
✓ Pipes provide an undirectional IPC
パイプは一方向の IPC を提供する
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 64
Bidirectional IPC
✓ Needs two pipes
Parent process
Child process
fd fd
fd fd
Pipe
Pipe
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 65
Run tests
✓ Run tests by traversing a test suite tree
テストスイートのツリーをたどってテストを実行する
test
TestA
test1
Portable and Fast - How to implement a parallel test runner
test2
Powered by Rabbit 4.0.1
Page: 66
Run a test
✓ Know path to the leaf of a test suite tree
テストスイートのツリーの葉までのパスを知っていれば
✓ Can find test in a test suite tree and run a test
テストを探して実行できる
✓ Path to the leaf
葉までの経路
✓ TestA test1
✓ TestA test2
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 67
Serializing
✓ Test name strings
テスト名の文字列
✓ test1(TestA), test2(TestA)
✓ Test result object
テスト結果のオブジェクト
✓ failures, errors, summary
失敗、エラー、サマリー
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 68
Marshalling
✓ Marshal.#dump
✓ Can serialize an object
オブジェクトをシリアライズできる
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 69
Unmarshalling
✓ Marshal.#load
✓ Can deserialize an object
オブジェクトをデシリアライズできる
✓ We do not need to know the size of data
読み込むデータのバイト数を知る必要がない
✓ Blocks process until data is readable
データが読み込み可能になるまで処理をブロックする
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 70
Role
✓ A child process named Worker
✓ Collects tests
テストを集める
✓ Run tests
テストを実行する
✓ A main process named Main
✓ Report results
テスト結果をレポートする
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 71
Single worker
✓ Also sends notify and ready signal to one
another
実行してー。終了してー。や、準備完了の合図もお互いに送りあう
Notify signal, Test name
Main
Worker
Ready signal, Test result
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 72
Multiplesafely
workers
✓ Must be exchanged
between multiple
processes
複数のプロセス間で安全に交換される必要がある
✓ We cannot use Thread::Queue because it’s not
multithread programming
マルチスレッドプログラミングではないので Thread::Queue を使えない
Notify signal, Test name
Worker1
Ready signal, Test result
Notify signal, Test name
Main
Worker2
Ready signal, Test result
Notify signal, Test name
Ready signal, Test result
Portable and Fast - How to implement a parallel test runner
Worker3
Powered by Rabbit 4.0.1
Page: 73
A possible solution
✓ Share pipes like Thread::Queue
Thread::Queue のような共有パイプ
✓ Can be exchanged safely if it is one byte writing/
reading
1 バイトの読み書きであれば安全に交換できる
✓ Run, Finish, worker id (~255)
✓ OS guarantees atomic operation
OS がアトミックな操作を保証してくれる
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 74
Shared pipes
: shared pipe
Worker1
Notify
signal
Main
Worker2
Ready
signal
Worker3
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 75
Data pipes
: data pipe
Test name
Test result
Main
Worker1
Worker2
Worker3
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 76
All pipes
: shared pipe
: data pipe
Test name
Test result
Worker1
Notify
signal
Main
Worker2
Ready
signal
Worker3
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 77
Pipe communication protocol
Main
NotifiedPipe
2
ReadyPipe
DataPipe
Workers
Spawn workers
1
Write `R`, `F`
loop
[Repeat until all tests are finished]
Read `R`
3
Write a worker id
Read a worker id
4
5
Write a test name
6
7
Read a test name
Find and run a test
8
Read `F`
9
Write result
Read result sequentially
Main
Portable and Fast - How to implement a parallel test runner
NotifiedPipe
ReadyPipe
10
11
DataPipe
Workers
Powered by Rabbit 4.0.1
Page: 78
Real-time reporting
✓ Shows test results after all tests have
finished
すべてのテストが終わってからテスト結果を表示する
✓ We want to show failures or errors
immediately
テスト実行中、すぐに失敗やエラーを表示したい
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 79
IO.select
✓ Allows multiplexing by monitoring multiple
IO objects
複数の IO オブジェクトを監視することで、 I/O の多重化を可能にする
✓ Can remove shared pipes
共有パイプを削除できる
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 80
A solution
: data pipe
Test name
Notify signal
Worker1
Test result
Ready signal
Main
Worker2
IO.
select
Worker3
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 81
Pipe communication protocol (1)
Main
DataPipe
Workers
Spawn workers
1
loop
[Repeat until all tests are distributed]
Write `ready` or `result`
Read (IO.select)
alt
2
3
[`ready`]
4
Write a test name
5
Read a test name
Find and run a test
6
[`result`]
Process a result
7
Main
Portable and Fast - How to implement a parallel test runner
DataPipe
Workers
Powered by Rabbit 4.0.1
Page: 82
Pipe communication protocol (2)
Main
DataPipe
Workers
Write `nil`
1
loop
[Repeat until all tests are finished]
Write `result` or `done`
Read (IO.select)
alt
2
3
[`result`]
Process a result
4
[`done`]
5
Write a `nil`
6
Main
Portable and Fast - How to implement a parallel test runner
DataPipe
Read `nil`
Workers
Powered by Rabbit 4.0.1
Page: 83
Dumping Issue
✓ Cannot dump an object includes Proc or IO
Proc や IO を含むオブジェクトはダンプできない
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 84
Test::Unit::TestCase object
✓ Should also write to pipe for hook point
Test::Unit::TestCase オブジェクトもフック用にパイプへ書き込む必要があっ
た
✓ Users can set any instance variables in test
ユーザーはテスト内で任意のインスタンス変数を設定できる
class TestA < Test::Unit::TestCase
def test1
@input = IO.new(0)
end
end
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 85
Customize marshal behavior
✓ Hook points for marshaling
マーシャルのためのフックポイント
✓ Object#marshal_dump
✓ Object#marshal_load
✓ Marshaling only required information
必須な情報だけをマーシャリングする
✓ Can now perform marshaling
マーシャリングできるようになった
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 86
Cannot dump anonymous class
無名クラスをダンプできない
✓ Users can be grouping tests using
sub_test_case
ユーザーは sub_test_case を使ってテストをグループ化できる
✓ Like context or describe in RSpec
✓ sub_test_case creates an anonymous
class
sub_test_case は無名クラスを作る
class TestA < Test::Unit::TestCase
sub_test_case("Foo Context") do
def test1
end
end
end
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 87
Assign a constant
✓ Assign an anonymous class to a constant
無名クラスを定数に割り当てる
✓ No longer anonymous class
無名クラスじゃなくなる
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 88
Safe constant name
✓ Marshal.#load cannot restore diffrerent
class definitions
Marshal.#load はクラス定義が違うと復元できない
✓ Must be a unique constant name
ユニークな定数名じゃないといけない
✓ Even across processes
プロセスを超えても同じ
✓ Safe as a constant name
定数名として安全な名前
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 89
Base64 encoding
✓ We solved by encoding a sub-test-case name
with Base64
Base64 でエンコードすることで解決した
✓ object_id is different across processes
オブジェクト ID は別プロセスだと違う値になる
# We can't use "\n", "=", "+" and "/" in base64 as class name.
encoded_name = [sub_test_case.name].pack("m").delete("\n=+/")
const_set(:"TEST_#{encoded_name}", sub_test_case)
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 90
Done:
Issues (2)
Page: 91
3. Keep backward
compatibility:
How to support
parallelism?
後方互換性を維持しながら並列実行をサポートするには？
Page: 92
Overview of running test suites
Focus
AutoRunner
TestCase
@runner_options
run
run
.run((snip))
@method_name
@internal_data
#run(result)
UI::TestRunnerMediator
UI::TestRunner
@suite
@options
TestSuite
UI::Console::TestRunner
run
#initialize(suite, options)
#initialize(suite)
#run()
#run_suite(result)
run
@tests
run
#run(result)
.run(suite, options)
#initialize(suite, options)
#start()
under Test::Unit module
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 93
Overview of running test suites
(focus)
TestCase
UI::TestRunnerMediator
#initialize(suite)
#run()
#run_suite(result)
TestSuite
run
@tests
#run(result)
run
@method_name
@internal_data
#run(result)
run
under Test::Unit module
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 94
Abstract #run from TestSuite to
TestSuiteRunner
UI::TestRunnerMediator
#initialize(suite)
#run()
TestSuiteRunner
TestSuite
run
#run_suite(result)
@tests
run
run
#run(result)
@test_suite
TestCase
run
@method_name
@internal_data
#initialize(test_suite)
#run(result)
#run(result)
under Test::Unit module
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 95
Introduce runner switching
TestRunContext
initialize
@runner_class : reader
#initialize(runner_class)
TestCase
TestSuiteRunner
@test_suite
run
run_all_tests
UI::TestRunnerMediator
@method_name
@internal_data
.run_all_tests(result, options)
#run(result, run_context:)
#initialize(test_suite)
#run(result, run_context:)
run
#initialize(suite)
#run()
run
TestSuite
#run_suite(result, run_context:)
run
under Test::Unit module
Portable and Fast - How to implement a parallel test runner
@tests
#run(result, run_context:)
Powered by Rabbit 4.0.1
Page: 96
Introduce WorkerContext
TestRunContext
initialize
@runner_class : reader
#initialize(runner_class)
TestSuiteRunner
@test_suite
run_all_tests
TestCase
.run_all_tests(result, options)
#initialize(test_suite)
#run(worker_context)
run
@method_name
@internal_data
#run(worker_context)
WorkerContext
UI::TestRunnerMediator
#initialize(suite)
#run()
#run_suite(worker_context)
under Test::Unit module
Portable and Fast - How to implement a parallel test runner
initialize
run
run
@id : reader
@run_context : reader
@result : reader
#initialize(id, run_context, result)
run
TestSuite
@tests
#run(worker_context)
Powered by Rabbit 4.0.1
Page: 97
Introduce parallel runner
TestProcessRunContext < TestRunContext
initialize
@test_names : reader
#initialize(runner_class)
TestSuiteProcessRunner < TestSuiteRunner
run_all_tests
TestCase
.run_all_tests(result, options)
#run(worker_context)
run
@method_name
@internal_data
#run(worker_context)
WorkerContext
UI::TestRunnerMediator
#initialize(suite)
initialize
#run()
#run_suite(worker_context)
under Test::Unit module
run
run
@id : reader
@run_context : reader
@result : reader
#initialize(id, run_context, result)
run
TestSuite
@tests
#run(worker_context)
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 98
Done:
Issues (3)
Page: 99
4. How to keep
busy and reduce
heavy fixtures?
暇をさせずに、重たい準備や後片付けを減らすには？
Page: 100
Producer-Consumer model
✓ Producer: produces tests
プロデューサーはテストを生成する
✓ Main Process (1)
✓ Consumer: consumes tests
コンシューマーはテストを消費する
✓ Worker Processes (N)
We want consumers to keep busy.
コンシューマーを暇させないようにしたい
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 101
Distribute styles
✓ Push style:
✓ Producer pushes tests to consumers
プロデューサーがコンシューマーにテストをプッシュする
✓ Pull style:
✓ Consumers pull a next test from producer
コンシューマーがプロデューサーから次のテストをプルする
push
Producer
Consumer1
pull
pull
Consumer2
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 102
Which leads?
どっちがリードする？
✓ Producer does not know consumers are
busy
プロデューサーはコンシューマーが忙しいのか知らない
✓ Consumers know own is busy
コンシューマーは自身が忙しいのか知っている
✓ So start with consumers and then keep them busy
なので、コンシューマーが起点となって暇かどうかを教えてくれれば暇をさせ
にくくできる
We use pull style.
我々はプルスタイルを使う
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 103
Balance variable workloads
負荷を均す
✓ Reduce heavy fixtures as much as possible
重たい準備と後片付けはできるだけ減らしたい
✓ Heavy fixtures are test case level fixtures
重たい準備と後片付けは、テスケースレベルのフィクスチャーのことでしたね
✓ However, we want to assign tests to another
idle consumer
でも別のコンシューマーが暇しているときは、そちらにテストを割り振りたい
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 104
Pass a test one by one
テストを一つずつ渡す
fixtures are called multiple
✓ Test case level
times
テストケースレベルのフィクスチャーは複数回呼び出される
Processing time
CPU1startupTest1shutdown
CPU2startupTest2shutdown
CPU3startupTest3shutdown
Test4
Test5
Test6
: Have test case level fixtures
: Haven't test case level fixtures
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 105
A possible solution
✓ Pass each test case level
テストケースレベルで渡す
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 106
Solution
✓ Pass each test case level if test case level
fixtures exist
テストケースレベルのフィクスチャーがあるときだけ、テストケースレベルで渡
す
Processing time
CPU1
startup
CPU2
CPU3
Test1
Test2
Test3
shutdown
Test4
Test5
Test6
: Have test case level fixtures
: Haven't test case level fixtures
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 107
Done:
Issues (4)
Page: 108
Works fine!!!
✓ But on CI of Windows
しかし、Windows の CI で
✓ Occurred 'Kernel#spawn': wrong file
descriptor error
ファイルディスクリプターのエラーが発生した
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 109
File descriptors
✓ 0, 1 and 2 are used by default
✓ 0: Standard Input
✓ 1: Standard Output
✓ 2: Standard Error Output
✓ Pass 3 and 4 for data pipes to child
processes
データ用パイプのために 3, 4 を子プロセスに渡してる
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 110
File descriptors on Windows
✓ Cannot pass 3 and above to child processes
3 以上を子プロセスに渡せない
✓ Avoid changing stdin/stdout/stderr as much
as possible
標準入力/標準出力/標準エラー出力を変更するのはできるだけしたくない
✓ Because tests may use them
テストがそれらを使うかもしれないから
We solved by using TCP/IP socket instead of
pipe if Windows
Windows の場合、パイプの代わりに TCP/IP のソケットを使うことで解決した
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 111
Solutions Summary
1. State? Rebuild解決案のまとめ
a test suite tree
状態は？テストスイートのツリーを再構築する
2. Communication? Use pipe, Marshal,
IO.select
通信は？パイプ、Marshal、IO.selectを使う
3. Backward compatibility? Abstract runner
and switch it
後方互換性は？実行するところを抽象化して切り替える
4. Fast? Use pull style. Pass a test suite if
test case level fixtures exist
速さは？Pull スタイルを使う。テストケースレベルのフィクスチャーがあるとき
Powered by Rabbit 4.0.1
はまとめて渡す
Portable and Fast - How to implement a parallel test runner
Page: 112
Demo
✓ --parallel=process option is available
--parallel=process オプションでマルチプロセスベースの並列実行を有効に
できる
✓ --n-workers=N option is available (default: the
number of available processors)
--n-workers=N オプションで並列実行するワーカーの数を指定できる
✓ Parallel running output looks like a regular
test-unit
並列実行の出力は、通常の test-unit と同様
✓ Show the test result in real time
テスト結果をリアルタイムに表示する
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 114
Ractor runner
✓ Now implementing!!!
✓ State: Already loaded, but object sharing is limited
状態: 既にロード済みだが、オブジェクトの共有は制限されている
✓ Communication: Use Ractor::Port
通信: Ractor::Port を使う
✓ Backword compatibility: Switching backend
後方互換性: バックエンドを切り替える
✓ Pull style: Ractor.receive not Ractor.select
Pull スタイル: Ractor.select ではなく Ractor.receive を使う
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 115
Ractor::IsolationError
✓ We met a lot of
Ractor::IsolationError
多くの Ractor::IsolationError が発生した
✓ Allow reading shareable class variables from non-
main Ractors
非メイン Ractor から共有可能なクラス変数を読み取ることを許可する
✓ https://bugs.ruby-lang.org/issues/21942
✓ Ractor
Ruby::Box ?
Ractor と Ruby::Box は相性が良い？
✓ Allow accessing unshareable objects within a
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 116
Conclusion
まとめ
We can run portable
and fast parallel test
running
by test-unit!!!
test-unit を使うと、ポータブルで速い並列テストを実行できる!!!
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 117
Acknowledgement
✓ Sutou Kouhei
✓ Thank you for launching the Red Data Tools
Red Data Tools を立ち上げてくれてありがとう
✓ Thank you for always polite explaining
いつも丁寧に説明してくれありがとう
✓ I wouldn’t be standing here today without him
彼なしではこの場に立つことはできなかった
✓ Naoto Ono
✓ Thank you for working with us
一緒に開発してくれてありがとう
✓ Your honest feedback are always helpful
率直な意見にいつも助けられています
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
Page: 118
Join us!
Red Data Tools
Portable and Fast - How to implement a parallel test runner
Powered by Rabbit 4.0.1
