How to write application in Java 8 that do not waste resources and which can maximize effective utilization of CPU/RAM. Comparison of blocking and non-blocking approach for I/O and application services. Based on microservices implementing simple business logic in security/cryptography/payments domain. Demonstration of following aspects:
* NIO at all edges of application
* popular libraries that support NIO
* single instance scalability
* performance metrics (incl. throughput and latency)
* resources utilization
* code readability with CompletableFuture
* application maintenance and debugging
All above based on our experiences gathered during development of software platforms at Oberthur Technologies R&D Poland.
2. Cloud of microservices for secure IoT
gateway backing
service
core
service
gateway
gateway
core
service
backing
service
3. The demand and characteristics of our domain
Crowded IoT environment
Slow, long lived connections
Big amount of concurrent connections
Scalability and resilience
Rather I/O intensive than CPU intensive
5. OTA Gateway – Use Case
TSM
Trusted
Service
Manager
Security Module
6. OTA Gateway – Use Case
OTA
(Over-the-Air)
Gateway
TSM
Trusted
Service
Manager
Security Module
DB
7. OTA Gateway – Use Case
OTA
(Over-the-Air)
Gateway
TSM
Trusted
Service
Manager
Security Module
DB
HTTP
1. submit
scripts
8. OTA Gateway – Use Case
OTA
(Over-the-Air)
Gateway
TSM
Trusted
Service
Manager
Security Module
DB
HTTP
1. submit
scripts
HTTP
2. encrypt
scripts
9. OTA Gateway – Use Case
OTA
(Over-the-Air)
Gateway
TSM
Trusted
Service
Manager
Security Module
DB
HTTP
1. submit
scripts
HTTP
2. encrypt
scripts
TCP
3. store
scripts
10. OTA Gateway – Use Case
OTA
(Over-the-Air)
Gateway
TSM
Trusted
Service
Manager
Security Module
DB
HTTP
1. submit
scripts
HTTP
2. encrypt
scripts
TCP
4. submition
response
3. store
scripts
11. OTA Gateway – Use Case
OTA
(Over-the-Air)
Gateway
TSM
Trusted
Service
Manager
Security Module
DB
HTTP
1. submit
scripts
HTTP
2. encrypt
scripts
TCP
4. submition
response
HTTP
5. poll
scripts
3. store
scripts
12. OTA Gateway – Use Case
OTA
(Over-the-Air)
Gateway
TSM
Trusted
Service
Manager
Security Module
DB
HTTP
1. submit
scripts
HTTP
2. encrypt
scripts
TCP
3. store
scripts
4. submition
response
HTTP
5. poll
scripts
6. search
scripts
13. OTA Gateway – Use Case
OTA
(Over-the-Air)
Gateway
TSM
Trusted
Service
Manager
Security Module
DB
HTTP
1. submit
scripts
HTTP
2. encrypt
scripts
TCP
3. store
scripts
4. submition
response
HTTP
5. poll
scripts
6. search
scripts
7. response
with scripts
14. OTA Gateway – Use Case
OTA
(Over-the-Air)
Gateway
TSM
Trusted
Service
Manager
Security Module
DB
HTTP HTTP
HTTP
TCP
15. OTA Gateway – Use Case
OTA
(Over-the-Air)
Gateway
TSM
Trusted
Service
Manager
Security Module
DB
Log
Storage
HTTP HTTP
HTTP
TCP File I/OMonitoring
System
HTTP
17. OTA Gateway Blocking - technologies
25 February 2017 17
TSM
DB Log
Storage
HTTP
TCP STDOUT
OTA
Gateway
JAX-RS
Logback
appender
Security Module
JAX-RS
Jedis
JAX-RS Client
18. OTA Gateway – Blocking - test
OTA
(Over-the-Air)
Gateway
send
2 scripts per
request
Security Module
DB
Log
Storage
HTTP HTTP
HTTP
TCP File I/O
emulated
latency
200 ms
expected
latency
< 450 ms
verified with
throughput
over 7k req/s
19. Blocking – 1k threads
OTA
(Over-the-Air)
Gateway
send
2 scripts per
request
Security Module
DB
Log
Storage
HTTP HTTP
HTTP
TCP File I/O
emulated
latency
200 ms
max 1000
connections
max 1000
threads
expected
latency
< 450 ms
21. The drawbacks of classic synchronous I/O
One thread per connection
Threads waiting instead of running
Context switches
Resource wasting (~1 MB per thread - 64bit)
23. OTA Gateway Non-blocking - technologies
25 February 2017 23
TSM
DB Log
Storage
HTTP
TCP STDOUT
OTA
Gateway
JAX-RS 2.0
Logback
async
appender
Async Http Client 2
(Netty)
Security Module
JAX-RS 2.0
Lettuce
(Netty)
24. Non-blocking – 16 threads
OTA
(Over-the-Air)
Gateway
send
2 scripts per
request
Security Module
DB
Log
Storage
HTTP HTTP
HTTP
TCP File I/O
emulated
latency
200 ms
no limit for
connections
max 16
threads
expected
latency
< 450 ms
28. OTA Gateway Blocking – sequence diagram
OTA
Gateway
TSM
Security
Module DB Logs
loop
1. submit
scripts
2. encrypt
script
3a. store
script
4. submition
response
3b. count
scripts
29. OTA Gateway Non-blocking – sequence diagram
OTA
Gateway
TSM
Security
Module DB Logs
loop
1. submit
scripts
2. encrypt
script
3a. store
script
4. submition
response
3b. count
scripts
30. OTA Non-blocking – realityOTA
Gateway
TSM
Security
Module DB Logs
„loopedprocessingchain”
1. submit
scripts
4. submition
response
3b. count
scripts
HTTP
Server
2. encrypt
script
3a. store
script
Logging
DB
Client
Security
Client
31. Code. Bird view
3125 February 2017
package org.demo.ota.blocking.rest;
@Path("se")
public class ScriptSubmissionResource extends Application {
private static final Logger log = LoggerFactory.getLogger(ScriptSubmissionResource.class);
private static final ResourceMetrics METRICS = new ResourceMetrics("ota_submission");
private SecureModuleClient secureModuleClient = SecureModuleClient.instance();
private ScriptStorageClient scriptStorageClient = ScriptStorageClient.instance();
@POST
@Path("/{seId}/scripts")
@Consumes(MediaType.APPLICATION_JSON)
@Produces(MediaType.TEXT_PLAIN)
public long submitScripts(@PathParam("seId") String seId, List<Script> scripts) {
MDC.put("flow", "submission");
MDC.put("se", seId);
return METRICS.instrument(() -> {
log.debug("Processing {} scripts submission", scripts.size(), seId);
for (int i = 0; i < scripts.size(); i++) {
final Script script = scripts.get(i);
log.debug("Encrypting {} script", i);
final String encryptedPayload = secureModuleClient.encrypt(seId, script.getPayload());
script.setPayload(encryptedPayload);
log.debug("Storing encrypted script {}", i);
scriptStorageClient.storeScript(seId, script);
}
long numberOfScripts = scriptStorageClient.numberOfScriptsForSe(seId);
log.debug("Request processed", seId);
return numberOfScripts;
});
}
@Override
public Set<Object> getSingletons() {
return Collections.singleton(this);
}
}
package org.demo.ota.nonblocking.rest;
@Path("se")
public class ScriptSubmissionResource extends Application {
private static final Logger log = LoggerFactory.getLogger(ScriptSubmissionResource.class);
private static final ResourceMetrics METRICS = new ResourceMetrics("ota_submission");
private SecureModuleClient secureModuleClient = SecureModuleClient.instance();
private ScriptStorageClient scriptStorageClient = ScriptStorageClient.instance();
@POST
@Path("/{seId}/scripts")
@Consumes(MediaType.APPLICATION_JSON)
@Produces(MediaType.TEXT_PLAIN)
public void submitScripts(
@PathParam("seId") String seId,
List<Script> scripts,
@Suspended final AsyncResponse asyncResponse
) {
final DiagnosticContext diagnosticContext = new DiagnosticContext("submission", seId);
METRICS.instrumentStage(() -> {
log.debug("{} Processing {} scripts submission", diagnosticContext, scripts.size());
return
encryptAndStoreAllScripts(diagnosticContext, seId, scripts)
.thenCompose(
ignore -> scriptStorageClient.numberOfScriptsForSe(seId)
);
})
.whenComplete((numberOfScripts, e) -> {
if (e != null) {
asyncResponse.resume(e);
} else {
log.debug("{} Request processed", diagnosticContext);
asyncResponse.resume(numberOfScripts);
}
});
}
private CompletionStage<Void> encryptAndStoreAllScripts(
final DiagnosticContext diagnosticContext,
final String seId,
final List<Script> scripts
) {
CompletionStage<Void> stage = null; // <- non final field, potential concurrent access bug!
for (int i = 0; i < scripts.size(); i++) {
final int scriptIndex = i;
final Script script = scripts.get(scriptIndex);
if (stage == null) {
stage = encryptAndStoreSingleScript(diagnosticContext, seId, scriptIndex, script);
} else {
stage = stage.thenCompose(ignore ->
encryptAndStoreSingleScript(diagnosticContext, seId, scriptIndex, script));
}
}
return stage;
}
private CompletionStage<Void> encryptAndStoreSingleScript(
final DiagnosticContext diagnosticContext,
final String seId,
final int scriptIndex,
final Script script
) {
log.debug("{} Encrypting script {}", diagnosticContext, scriptIndex);
return secureModuleClient
.encrypt(seId, script.getPayload())
.thenCompose(
encryptedPayload -> {
log.debug("{} Storing encrypted script {}", diagnosticContext, scriptIndex);
return scriptStorageClient.storeScript(seId, new Script(encryptedPayload));
}
);
}
@Override
public Set<Object> getSingletons() {
return new HashSet<>(Collections.singletonList(this));
}
}
32. Code. Blocking. Submission 1
25 February 2017 32
@POST
@Path("/{seId}/scripts")
@Consumes(MediaType.APPLICATION_JSON)
@Produces(MediaType.TEXT_PLAIN)
public
long submitScripts(@PathParam("seId") String seId, List<Script> scripts) {
MDC.put("flow", "submission"); // Setting diagnostic context
MDC.put("se", seId);
return METRICS.instrument(() -> { // Instrumenting with metrics
//...
33. Code. Blocking. Submission 2
25 February 2017 33
log.debug("Processing {} scripts submission", scripts.size(), seId);
for (int i = 0; i < scripts.size(); i++) { Cycle through the
scripts
final Script script = scripts.get(i);
log.debug("Encrypting {} script", i);
final String encryptedPayload = secureModuleClient
.encrypt(seId, script.getPayload()); Encrypting the script
script.setPayload(encryptedPayload);
log.debug("Storing encrypted script {}", i);
scriptStorageClient.storeScript(seId, script); Saving the script into
DB
}
long numberOfScripts =
scriptStorageClient.numberOfScriptsForSe(seId); Getting current number
of scripts in DB
log.debug("Request processed", seId);
return numberOfScripts;
34. Code. Non-blocking. Submission 1
25 February 2017 34
@POST
@Path("/{seId}/scripts")
@Consumes(MediaType.APPLICATION_JSON)
@Produces(MediaType.TEXT_PLAIN)
public void submitScripts(
@PathParam("seId") String seId,
List<Script> scripts,
@Suspended final AsyncResponse asyncResponse
) {
final DiagnosticContext diagnosticContext =
new DiagnosticContext("submission", seId); Creating diagnostic
context
METRICS.instrumentStage(() -> { Instrumenting with metrics
35. Code. Non-blocking. Submission 1
25 February 2017 35
@POST
@Path("/{seId}/scripts")
@Consumes(MediaType.APPLICATION_JSON)
@Produces(MediaType.TEXT_PLAIN)
public void submitScripts(
@PathParam("seId") String seId,
List<Script> scripts,
@Suspended final AsyncResponse asyncResponse
) {
final DiagnosticContext diagnosticContext =
new DiagnosticContext("submission", seId); Creating diagnostic
context
METRICS.instrumentStage(() -> { Instrumenting with metrics
37. Code. Non-blocking. Submission 3
25 February 2017 37
private CompletionStage<Void> encryptAndStoreAllScripts(
final DiagnosticContext diagnosticContext,
final String seId,
final List<Script> scripts
) {
CompletionStage<Void> stage = null; // <- non final field, potential
// concurrent access bug!
for (int i = 0; i < scripts.size(); i++) { Cycle through the
scripts
final int scriptIndex = i;
final Script script = scripts.get(scriptIndex);
if (stage == null) {
stage = encryptAndStoreSingleScript(
diagnosticContext, seId, scriptIndex, script);
} else {
stage = stage.thenCompose(ignore ->
encryptAndStoreSingleScript(
diagnosticContext, seId, scriptIndex, script));
}
}
return stage;
}
38. Code. Non-blocking. Submission 4
25 February 2017 38
private CompletionStage<Void> encryptAndStoreSingleScript(
final DiagnosticContext diagnosticContext,
final String seId,
final int scriptIndex,
final Script script
) {
log.debug("{} Encrypting script {}", diagnosticContext, scriptIndex);
return secureModuleClient
.encrypt(seId, script.getPayload()) Encrypting the script
.thenCompose(
encryptedPayload -> {
log.debug(
"{} Storing encrypted script {}",
diagnosticContext,
scriptIndex);
return
scriptStorageClient.storeScript( Saving the script into seId,
the DB
new Script(encryptedPayload));
}
);
}
39. Code. Blocking. Integration
25 February 2017 39
private final JedisPool pool;
public void storeScript(String seId, Script script) {
try (Jedis jedis = pool.getResource()) {
jedis.rpush(seId, script.getPayload());
}
}
public long numberOfScriptsForSe(String seId) {
try (Jedis jedis = pool.getResource()) {
return jedis.llen(seId);
}
}
public Optional<Script> nextScript(String seId) {
try (Jedis jedis = pool.getResource()) {
return Optional.ofNullable(jedis.lpop(seId)).map(Script::new);
}
}
public String encrypt(String keyDiversifier, String payload) {
// ...
}
40. Code. Non-Blocking. Integration
25 February 2017 40
private final RedisAsyncCommands<String, String> commands;
public CompletionStage<Void> storeScript(String seId, Script script) {
return commands
.rpush(seId, script.getPayload())
.thenApply(ignore -> null);
}
public CompletionStage<Long> numberOfScriptsForSe(String seId) {
return commands.llen(seId);
}
public CompletionStage<Optional<String>> nextScript(String seId) {
return commands.lpop(seId).thenApply(Optional::ofNullable);
}
public CompletionStage<String> encrypt(String keyDiversifier, String payload) {
// ...
}
41. Diagnostics in non-blocking systems
No clear stack traces, need for good logs
Name your threads properly
MDC becomes useless (thread locals)
Explicitly pass debug context to trace flows
Be prepared for debuging non-obvious errors
43. Lessons learned, part 1
Vanila Java 8 for NIO µ-services
Netty best for custom protocols in NIO
Unit tests should be synchronous
Load/stress testing is a must
Make bulkheading and plan your resources
44. Lessons learned, part 2
Functional programming patterns for readability
Immutability as 1-st class citizen
Scala may be good choice ;-)
45. Conclusion
Non-blocking processing can really
save your resources (== money)
But!
Weight all pros and cons and use non-blocking processing
only if you really need it.
46. Thank you.
Michał Baliński
m.balinski@oberthur.com
Oleksandr Goldobin
o.goldobin@oberthur.com
goldobin
@goldobin
balonus
@MichalBalinski
Readings:
• https://github.com/balonus/blocking-vs-nonblocking-demo
• C10k Problem, C10M Problem, Asynchronous I/O
• Boost application performance using asynchronous I/O (M. Tim Jones, 2006)
• Zuul 2 : The Netflix Journey to Asynchronous, Non-Blocking Systems (Netflix, 2016)
• Thousands of Threads and Blocking I/O: The Old Way to Write Java Servers Is New
Again (and Way Better) (Paul Tyma, 2008)
• Why Non-Blocking? (Bozhidar Bozhanov, 2011)
Notas del editor
Michał
AlexOur products are operating in pretty "crowded" environment
The clients of our services are:
from the one site: service providers like MNO or Banks
from the other site: crowds of devices like secure elements in mobile and IoT devices
those devices maintaining SLOOOW, long living connections
The demand of the market is to process pretty big amount of concurrent connections in a secure way
We also should be scalable and resilient -> micro-services architecture
The microservices in IT are generally performing some kind of gateway role:
receive request
request other web services/database for some data
synchronize and process responses
respond (or not) to the caller
So primarily they are IO intensive rather then CPU intensive
The most CPU intensive part is actually marshalling/unmarshaling of requests/responses
Michał
Michał
Michał
Michał
Michał
Michał
Michał
Michał
Michał
Michał
Michał
Alex
Alex
Alex
Alex
Michał
AlexTODO reconsider order of slides
Your are forced to have one thread per connection
Those threads actually waiting for the data instead of perform real work
This leads to big amount of context switches
And also this leads to resource wasting because threads are expensive
For example you can't create more than 4000 threads (on the latest Mac Book Pro -> OOM )
Christopher Batey - $8 per year on AWS EC2
Alex
Alex
Alex
Alex
Michał
TODO kick out?
Michał
Michał
Michał
Alex
AlexIt is harder to debug:
no clear stack traces;
you're more relying on logs;
It is harder to trace business flows:
MDC becomes useless (thread locals);
To trace business flows you need to pass (or close on) some immutable (!!!) debugging context;
Michał
Servers:
JaxRS 2.x implementation like RestEasy 3.x with @Suspended annotation support
Netty
Akka (IO, Http) – different story, still better used with Scala (sorry )
Clients:
Async Http Client 2.0
Latest database clients:
Lettuce for Redis
Cassandra
Mongo DB
JDBC drivers by nature blocking :(
Akka (IO, Http)
Alex + Michał (last one)
JDK 8 + some libraries like Netty (vanilla Java) are sufficient to implement non-blocking processing and build micro-services (no additional huge frameworks really required)
Async code can be readable but it requires to apply functional programming patterns which are not common for classic approach
As soon as you're using several threads to process your flows immutability becomes 1-st class citizen
Unit tests should be synchronous -- no concurrency at all or fully synchronized on test's execution thread (which can be challenging sometimes)
Sometimes you need to develop your own tools to make tests locking like scenarios and not as complex asynchronous code;
Load/stress testing is a must - you really need to stress your system to check that it behaves properly
Michał + Alex (last one)
Do not use MDC use some context passing technique (like we showed in example);
Name your threads properly: it will help you to investigate/profile issues;
Make bulkheading and plan your resources utilization in advance, adjust configuration while performing load/stress tests;
Be prepared to solve some non-obvious errors (like direct memory OOM or virtual memory OOM);
Implementing graceful shutdown is tricky, design it from the beginning;
Netty is a best library to implement custom protocols in non-blocking way in Java;
Technically Scala still provides better toolset for non-blocking asynchronous data processing (standard library and/or Akka) -- but it is different topic ;)