Airframe: Lightweight Building Blocks for Scala - Scale By The Bay 2018
1. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
Taro L. Saito, Ph.D.
GitHub: @xerial
Arm Treasure Data
Airframe: Lightweight Building
Blocks for Scala
November 16th, 2018
Scale By The Bay 2018 - Unconference
1
2. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
About Me: Taro L. Saito (Leo)
● An Engineer with Research Background
● Ph.D., University of Tokyo
● DBMS & Genome Science
● Leading Query Engine Team
● Active OSS Developer
● airframe
● sqlite-jdbc
■ More than 1000 GitHub stars
● snappy-java
■ Compression library used in
Spark, Parquet
● sbt-sonatype
■ Used in 2000+ Scala projects
● ...
2
3. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
Airframe
● Lightweight Building Blocks for Scala
● Essential libraries for building any applications
● Used in production for 2+ years
● Based on my code collection since 2009
● Initially written in Java
● Gradually migrated to Scala
● Repackaged into wvlet.airframe in 2016
● For maintainability
● 18 Modules
● Simplifying your daily programming in Scala
3
4. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
18 Airframe Modules
● Bootstrap
● airframe-config Configuration loader
● airframe-launcher Command-line program launcher
● Object Serialization
● airframe-codec encoder/decoder SPI + standard codecs
● airframe-msgpack pure-Scala MessagePack implementation
● airframe-tablet CSV/TSV/JSON/JDBC ResultSet <-> Object
● Monitoring & Debugging
● airframe-log Logging
● airframe-metrics Human-readable metrics for time, date, data size, etc
● airframe-jmx Object metrics provider through JMX
● Building Service Objects
● airframe Dependency injection
● airframe-surface Object type inspector
● Misc:
● airframe-control, airframe-jdbc, airframe-json, airframe-http, etc.
4
5. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
Utilities for Debugging Applications
5
6. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
Debugging Applications: Airframe Log
● Airframe Log: A Modern Logging Library for Scala (Medium Blog)
● ANSI color, source code location support
6
7. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
Airframe Metrics
● Human Readable Data Format (Duration, DataSize, etc.)
● Handy Time Window String Support
7
“-1d”
“-1w”
“-7d”
8. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
Airframe JMX
● Checking the internal states of remote JVM processes
● JMX clients
● jconsole has JMX metric monitor
● Airframe JMX -> DataDog -> Dashboard
8
9. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
Airframe Launcher
● Using Class & Function Command Line Programs
9
10. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
Dependency Injection with Airframe
10
11. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
Today’s Goals
● Lean How to Use Airframe DI (Dependency Injection)
● Understand what can be simplified with DI
● Learn 5 Airframe DI Design Patterns
● That improve the thought processes in programming
11
12. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
What is Dependency Injection (DI)?
● Many Articles…
● Inversion of Control Containers and the Dependency Injection pattern. Martin
Fowler (2004)
● StackOverflow, Wikipedia, …
● Many Frameworks...
● Spring, Google Guice, Scaldi, Macwire, Grafter, Weld, etc.
● No framework approaches also exist (Pure-Scala DI)
● Recent Definition:
● Dependency Injection is the process of creating the static, stateless graph of
service objects, where each service is parameterised by its dependencies.
■ What is Dependency Injection? by Adam Warski (2018)
● However, it’s still difficult to understand what is DI
12
13. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
Simplifying DI with Airframe
● Airframe Usage
● import wvlet.airframe._
● Simple 3 Step DI
● bind
● design
● build
13
14. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
Design Patterns In Airframe
● Pattern 0 (basic): Building Service Objects with Auto-Wiring
● Pattern 1: Configuring Modules
● Pattern 2: Switching Bindings for Tests
● Pattern 3: Managing Lifecycle in FILO order
● Pattern 4: Bundling Service Traits
● Flower-bundle pattern
● Pattern 5: Binding Factory
14
15. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
Constructor vs In-Trait Injections
● Constructor Injection
● In-Trait Injection
15
16. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
Building Service Objects
● When coding A and B
● You can focus on only direct dependencies
● You can forget about indirect dependencies
● Airframe DI builds A, B, and direct/indirect dependencies on your behalf.
A
DB
Connection
Pool
DB
Client
DB Monitor
Fluentd
Logger
HttpClient
B
16
You can forget this part
17. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
Code Example
● Focus on Logic
17
A
DB
Client
Fluentd
Logger
B
18. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
Hand Wiring vs Auto Wiring
18
19. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
Configuring Modules
● How to pass configuration objects to corresponding modules?
● new A(new DBClient(new ConnectionPool(new DB(new DBConfig(...)), new
ConnectionPoolConfig(...), new DBClientConfig(...)))
● Things You Need to Remember
● Argument orders (or argument names in Scala) of individual modules
● How to instantiate modules
A
DB
Connection
Pool
DB
Client
DB Monitor
Fluentd
Logger
HttpClient
B
19
DB
Config
Connection
Pool Config
HttpClient
Config
20. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
Pattern 1: Adding Config
● Put config to the closest place in the code
20
21. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
Pattern 2: Switching Bindings For Testing
● In Airframe Design
● You can replace DB and FluentdLogger to In-Memory Impl
● How to build A and B differs, but the same code can be used
A
Memory
DB
Connection
Pool
DB
Client
DB Monitor
Fluentd
Logger
In-memory
Logger
B
21
Overriding Design for Testing
22. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
Pattern 2: Switching Bindings For Testing
22
23. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
Pattern 3: Managing Lifecycle
● onStart, onShutdown hooks
● JSR-250 annotations
23
24. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
Complex FILO Order Resource Management
● FILO := First-In Last-Out
● Airframe registers onStart and onShutdown lifecycle hooks when creating
instances
● When closing sessions, onShutdown will be called in the reverse order
● Dependencies forms DAG
● Dependencies will be generated when creating new service objects
A
DB
Connection
Pool
DB
Client
DB Monitor
Fluentd
Logger
HttpClient
B
134
56
7
2
8
Shared
Resource
24
25. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
Pattern 4: Bundling Service Traits
● Flower-Bundle Pattern
● Create composable services
25
26. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
Pattern 5: Bind Factory
● Create a factory that change partial dependencies
● e.g., creating differently configured instances of Databases
26
27. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
3 Things You Can Forget With Airframe DI
● 1. How to Build Service Objects
● config, auto-wiring, flower bundle pattern
● 2. How to Manage Resource Lifecycle
● FILO order
● 3. How to Use DI Itself (!!)
● Only need to understand bind, design, and build.
27
28. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
Summary: Reducing Code Complexity with Airframe DI
● You can effectively forget about:
● How to build service objects
● How to manage resources in FILO order
● How to use DI itself
A
DB
Connection
Pool
DB
Client
DB Monitor
Fluentd
Logger
HttpClient
B
134
56
7
2
8
28
Implementation Details
29. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
Airframe Internals
29
30. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
Details
● bind[X]
● X becomes singleton
● Usually there is no need to declare bind[X].toSingleton
■ Unless you want to initialize X early with production mode
● bindInstance[X]
● When you need to call new X(d1, d2, …) every time
● bindFactory[A1 => X]
● For customizing A1 in X
● Design
● Immutable & Serializable
● design1 + design2 + ….
■ Overriding the previous design (adding order is important)
● Session
● withLazyMode/withProductionMode
● noLifeCycleLogging
30
31. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
Injecting Airframe Session with Scala Macros
31
32. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
Airframe Surface: Object Shape Inspector
● Reading Type Signatures From ScalaSig
● Scala compiler embeds Scala Type Signatures (ScalaSig) to class files
● Surface supports
● type alias, tagged type
● higher-kinded types
class A (data:List[B])
class A
data: List[java.lang.Object]
class A
data: List[java.lang.Object]
ScalaSig: data:List[B]
javac
scalac
Surface.of[A]
data: List[B]
scala.reflect.runtime.
universe.TypeTag
Type Erasure
32
???
33. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
Future Work & Summary
33
34. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
Current State of Airframe
● Version 0.73 (As of November 2018)
● We already had 40+ releases in 2018
● Automated Release
● Cross building libraries for Scala 2.11, 2.12, 2.13, and Scala.js
● ‘sbt release’ command took 3 hours
■ Sequential steps:
○ compile -> test -> package -> upload x 18 modules x 4 Scala versions
● Now a new version can be released in 10 minutes on Travis CI
● Blog
● 3 Tips for Maintaining Scala Projects
● Future Work
● Adding child session support
● Support multiple constructor argument blocks
34
35. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
Philosophy: Simplicity By Design
● “Simplicity” by Philippe Dufour
● A clock made by a legendary
watchmaker in Switzerland
● Every part of the clock is built
by himself
● Airframe
● Provides simplicity for
application developers
35
36. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
Summary
● To understand DI, think about what you can
simplify
● How to build objects
● How to manage resources (FILO)
● Learning DI framework itself
● 5 Airframe Design Patterns
● Pattern 1: Configuring Modules
● Pattern 2: Overriding Bindings for Tests
● Pattern 3: Managing Lifecycle in FILO order
● Pattern 4: Bundling Service Traits
■ Flower-bundle pattern
● Pattern 5: Binding Factory
Don’t Forget Adding GitHub Star!
wvlet/airframe
36