More Related Content
Similar to Spring Batch - Lessons Learned out of a real life banking system. (20)
Spring Batch - Lessons Learned out of a real life banking system.
- 1. Spring Batch
Lessons Learned out of a real life banking system.
Michael Beer
Senior Consultant
Raffael Schmid
Consultant
BASEL
1
BERN
BRUGG
LAUSANNE
ZUERICH
DUESSELDORF
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
FRANKFURT A.M.
FREIBURG I.BR.
HAMBURG
MUNICH
STUTTGART
VIENNA
- 2. About us
Michael Beer
Senior Consultant
Trivadis since 2001
Trivadis since 2010
design and development of
web based applications
design and development of
web based applications
part of the Trivadis APM
team
2
Raffael Schmid
Consultant
interested in performance
related things on the JVM
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
- 5. Initial and target system context
Web Interface
Automatic load out of thirdparty
systems
Rule Engine
multiple times per day
export / import as CSV file
Manual load by user
common usage
at irregular time intervals
Migration load due to thirdparty
system decommissioning
Thirdparty system
File Interface
Thirdparty system
5
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
run once
different volumes
- 6. Why we used Spring Batch?
It was already part of the technology stack within the customers
environment.
There were no real (free) alternatives at that time (or at least we didn‘t
know any).
Exceptional permit was needed for Spring Integration because it was
shipped with Spring Batch Admin.
6
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
- 7. Terminology of Spring Batch jobs
*
Job
Step
*
Job Instance
*
Job Parameter
*
*
Job Execution
*
Job Execution
Context
7
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
Step Execution
Step Execution
Context
*
Tasklet
- 9. Requirement: Performance
Performance
Reprocess
failed items
Summary mail
Gather detailed
job information
Trigger jobs
Control load
Deactivate jobs
Inter-job
dependencies
parallelized on multiple threads
order of execution matters
number of threads can be changed before or after job execution
9
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
- 10. Requirement: Reprocess failed items
Performance
Reprocess
failed items
Summary mail
Gather detailed
job information
Trigger jobs
Control load
Deactivate jobs
Inter-job
dependencies
rerunning a job processes failed items (only)
process state of item therefore needs to be maintained
10
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
- 11. Requirement: Summary mail
Performance
Reprocess
failed items
Summary mail
Gather detailed
job information
Trigger jobs
Control load
Deactivate jobs
Inter-job
dependencies
write a detailed execution log that contains
Exceptions
messages out of the Rule Engine
aggregate to summary
Number of errors, warnings, etc.
worst status level
send summary to submitter and operator
11
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
- 12. Requirement: Gather detailed job information
Performance
Reprocess
failed items
Summary mail
Gather detailed
job information
Trigger jobs
Control load
Deactivate jobs
Inter-job
dependencies
rerun failed single item in trace mode
collect diagnostic information
e.g. out of the Rule Engine
12
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
- 13. Requirement: Trigger jobs
Performance
Reprocess
failed items
Summary mail
Gather detailed
job information
Trigger jobs
Control load
Deactivate jobs
Inter-job
dependencies
periodically
either fixed delay or fixed rate
on event
e.g. new data arrived in database
13
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
- 14. Requirement: Control load
Performance
Reprocess
failed items
Summary mail
Gather detailed
job information
Trigger jobs
Control load
Deactivate jobs
Inter-job
dependencies
prevent too many jobs in parallel
conditions might prevent job execution
requests should prevent system shutdown
14
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
- 15. Requirement: Deactivate jobs
Performance
Reprocess
failed items
Summary mail
Gather detailed
job information
Trigger jobs
Control load
Deactivate jobs
Inter-job
dependencies
deactivate job execution
set job execution on hold
15
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
- 17. Requirements grouped into five different topics
Performance
Partitioning
Error
Reprocess
failed items
Handling
Trigger jobs
Control load
Summary mail
Monitoring
Job Control
Deactivate jobs
Sourcing
17
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
Gather detailed
& Tracing
job information
Inter-job
dependencies
- 19. Topics we will cover
Monitoring
& Tracing
depends on
Partitioning
depends on
supervising
Sourcing
firing trigger
depends on
Error
Handling
19
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
Job Control
- 21. Sourcing
Load data into staging table
CAT_ID
310
310
311
|
|
|
|
PERSON_ID
2908
1608
1608
|
|
|
|
AMOUNT
200000
100000
100000
|
|
|
|
CURRENCY
USD
CHF
CHF
|
|
|
|
VALID_FORM
03.12.2013
03.12.2013
03.12.2013
|
|
|
|
LAST_CHANGE|
02.02.2013 |
02.01.2013 |
02.01.2013 |
PARTITION
LINE
TYPE
STATE
FIELD01
FIELD02
FIELD03
FIELD04
PT_000122
171
D
02
441
1804
400000
EUR
PT_000122
172
D
02
441
1002
221000
EUR
PT_000123
1
H
01
CAT_ID
PERSON_ID
AMOUNT
CURRENCY
PT_000123
2
D
01
310
2908
200000
USD
PT_000123
3
D
01
310
1608
100000
CHF
PT_000123
4
D
01
311
1608
100000
CHF
21
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
- 22. Allows partitioning, single record execution
Sourcing
PARTITION
LINE
TYPE
STATE
FIELD01
FIELD02
FIELD03
FIELD04
PT_000123
1
H
01
CAT_ID
PERSON_ID
AMOUNT
CURRENCY
PT_000123
2
D
01
310
2908
200000
USD
PT_000123
3
D
01
310
1410
100000
CHF
Partitioning out of the box
SELECT * FROM LOAD
WHERE PARTITION =‘PT_000123’
AND LINE BETWEEN 0 AND 99;
Reexcution of failed records
SELECT * FROM LOAD
WHERE PARTITION =‘PT_000123’
AND STATE = ‘01’
Execution of single record
SELECT * FROM LOAD
WHERE PARTITION =‘PT_000123’
AND LINE = 8
22
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
- 23. Job Control
Monitoring
& Tracing
depends on
Partitioning
depends on
supervising
Sourcing
firing trigger
depends on
Error
Handling
23
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
Job Control
- 24. Job Control
Simplest way of triggering a job
0 0/5 * * * ?
Trigger
job-requests
Spring Batch
(SimpleJobLauncher)
Request Factory
Define polling channel adapter (CronTrigger)
<int:inbound-channel-adapter method="create" channel="job-requests"
ref="requestFactory">
<int:poller cron="0 0/5 * * * ?" />
</int:inbound-channel-adapter>
Or even simpler (PeriodicTrigger)
<int:inbound-channel-adapter method="create" channel="job-requests"
ref="requestFactory">
<int:poller fixed-rate="300000"/>
</int:inbound-channel-adapter>
24
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
- 25. Job Control
Persist job launch requests into database
Data import job
(database)
job queue
Periodic jobs
*JobQueueMessageHandler
0 0/5 * * * ?
persist
*JobLaunchingMessageHandler
job launch
request
jobrequests
Guard
verification
update status
save job dependencies
25
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
*JobLauncher
- 27. Partitioning
Performance
Type
Description
Multi-threaded Step
Local
A step is multithreaded (TaskExecutor)
Parallel Steps
Local
Executes steps in parallel using multithreading
Partitioning Step
Local
Remote
Partitions data and splits up processing
Remote Chunking
27
Local/Remote
Remote
Distributed chunk processing to remote nodes
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
- 30. Partitioning
Characteristics of data
There might be dependencies between records. The order of execution
matters at some point.
CAT_ID
310
310
311
312
313
310
390
310
|
|
|
|
|
|
|
|
|
PERSON_ID
2908
1608
1608
1608
1608
1410
1108
2908
|
|
|
|
|
|
|
|
|
AMOUNT
200000
100000
100000
100000
100000
100000
100000
500000
| CURRENCY
| Insert
USD
| CHF
| CHF
| CHF
| CHF
| CHF
| CHF
|Update
USD
Read
30
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
|
|
|
|
|
|
|
|
|
VALID_FORM
03.12.2013
03.12.2013
03.12.2013
03.12.2013
03.12.2013
03.12.2013
03.12.2013
04.12.2013
Write
Process / Write
|
|
|
|
|
|
|
|
|
LAST_CHANGE|
02.02.2013 |
02.01.2013 |
02.01.2013 |
02.01.2013 |
02.01.2013 |
02.01.2013 |
02.01.2013 |
03.02.2013 |
- 31. Partitioning detail – Spring Batch Admin
31
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
Partitioning
- 32. Performance – Read / Write / Process
Partitioning
Begin Step
Open
Transaction
For each item
For each item
ItemReader
Item read()
ItemProcessor
Process(Item)
RuntimeException
RuntimeException
ItemWriter
Write(List<Item>)
Commit
Transaction
RuntimeException
Rollback
Transaction
Finish Step
32
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
- 33. Partitioning
Performance
No error
Error on each item
• ~ 22 sec
• 0 Rollbacks
•
•
•
•
33
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
~ 50 sec
1‘100 Rollbacks
5‘500 Filter
1‘000 Write Skips
- 34. Error Handling
Monitoring
& Tracing
depends on
Partitioning
depends on
supervising
Sourcing
firing trigger
depends on
Error
Handling
34
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
Job Control
- 36. Error
Handling
Restartability configuration
initial (failed) execution
Job
restarted execution
serialized
context
Step
initialization
Step
process
Step
initialization
Execution
Context
Step
finalize
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
Step
process
Step
finalize
cleanup
36
Job
- 37. Error
Handling
Item based error handling
Feature
What?
Where?
Skip
For nonfatal
exceptions
Keeps processing for an
incorrect item
Chunk-oriented step
Retry
37
When?
For transient
exceptions
Makes new attemps on an
operation
Chunk-oriented step,
application code
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
- 39. Monitoring – Spring Batch Admin
39
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
Monitoring &
Tracing
- 40. Monitoring – logging to a database
Step
Monitoring &
Tracing
Listener
Before
Read
OnSkip
[WARN]
Process
LoggerService
@Transactional(propagation =
Propagation.REQUIRES_NEW)
OnError
[ERROR]
Write
After
[INFO]
2 calls on error
40
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
- 41. Monitoring &
Tracing
Monitoring - Mail
Job
Step
ReadProcessWrite
FAILED
OK Mail
41
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
Error Mail
- 42. Monitoring &
Tracing
Tracing – Explain plan
explicit job start
Operator
Step
ReadProcessWrite
Partitioner
Read
Process
Writer
1 Record
1 Record
Debug mode
does nothing
Step
…
42
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
- 43. Monitoring &
Tracing
Monitoring / Tracing summary
Mail
OK/ERROR
Mail
ERROR
UI
Log
UI
Explain plan
User
Spring Batch
Admin
43
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
Operator
- 45. Challenges – Error Handling
Fire and forget
Commit until first error
- Skip
- Retry
Depending on the characteristic of data
Rollback
- Whole job
- Partition
- Chunk
Job flow
Deciders
Operator takes care about the errors
45
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
- 46. Why we still use Spring Batch
Out of 155 SE tests in the JSR-352 TCK, Spring Batch 3.0 (milestone 1)
passes 70.
The terminology stays more or less the same: Job, Step, Chunk, Item,
ItemProcessor, JobInstance, JobExecution.
Spring Batch
JSR 352
Tasklet
Batchlet
ItemReader / ItemStream
ItemReader
JSR-352‘s ItemReader includes Spring Batchs
ItemStream capabilities
ItemWriter / ItemStream
ItemWriter
JSR-352‘s ItemReader includes Spring Batchs
ItemStream capabilities
JobExecutionListener
JobListener
StepExecutionListener
StepListener
46
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
Comments
- 47. Suitable for different applications
Spring Batch supports implementing reliable batch jobs with small effort.
In few cases even no Java code is necessary.
It’s very easy to customize existing parts at your needs. Different level of
customization is possible:
configuration
47
partial
implementation
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
job control
- 48. Questions and answers ...
Michael Beer
michael.beer@trivadis.com
+41 58 459 51 90
Raffael Schmid
raffael.schmid@trivadis.com
+41 58 459 52 34
BASEL
48
BERN
BRUGG
LAUSANNE
ZUERICH
DUESSELDORF
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
FRANKFURT A.M.
FREIBURG I.BR.
HAMBURG
MUNICH
STUTTGART
VIENNA