This document discusses Spring Cloud Data Flow, which provides tools for creating complex data pipelines for streaming and batch processes. It describes how Spring Cloud Data Flow orchestrates Spring Cloud Stream for event streaming and Spring Cloud Task for short-lived microservices. The document also references challenges around integration, event streaming, and orchestrating short-lived tasks that Spring Cloud Data Flow addresses.

1. © Copyright 2018 Pivotal Software, Inc. All rights Reserved.
Abhinav Rau
Senior Platform Architect
arau@pivotal.io
@abhinavrau
https://github.com/abhinavrau/scdf-demo
Spring Cloud Data Flow Overview

7. Sadly, we add Complexity to
what’s complicated

8. Lack of Cohesion
Hard to Deploy
Excessive Configuration

9. What Complexity?

10. Avoid Tight Coupling
What Complexity?

11. Continuous Delivery with
Zero Downtime
Avoid Tight Coupling
What Complexity?

12. Scaling
Avoid Tight Coupling
Continuous Delivery with
Zero Downtime
What Complexity?

13. Test Surface Area
What Complexity?

16. (Loosely)
Story based on True Events..

19. ??

21. Let’s Dive In…

22. Test Surface Area
• Unit Tests: 60+ Pre-built and Unit Tested Sources, Sinks and
Processors
• Integration Tests: Applications have a defined input/output
• Component Tests: Application is the Component boundary
• Contract Tests: External services are clearly visible and defined
• End to End Tests: This is where testing is needed.

23. Integration
Problem

24. Spring Integration

25. Spring Cloud Stream

26. Spring Boot

27. Spring Boot Data Pipeline

28. Microservice Data Pipeline

29. Microservice Data Pipeline

30. Function Data Pipeline

31. Spring Cloud Data Flow

32. Spring Cloud Data Flow

33. Spring Cloud Data Flow
Tools to create complex topologies
for streaming and batch
data pipelines

34. Spring Cloud Data Flow
Spring Cloud Stream
Spring Cloud Task

35. framework for building highly
scalable event-driven
microservices connected with
shared messaging systems
Spring Cloud Stream

36. Spring Cloud Stream
Rabbit MQ Apache Kafka Google PubSub
Amazon Kinesis Azure Event Hubs Solace
Pluggable Binder
Implementations

37. Spring Cloud Stream
Pluggable Binder
Implementations
Imperative + Functional
Programming Model
public class TransferServiceImpl implements TransferService {
public TransferServiceImpl(AccountRepository ar) {
this.accountRepository = ar;
}

38. Spring Cloud Stream
@EnableBinding(Processor.class)
public class Application {
@StreamListener("foo")
@SendTo("bar")
public String replaceStringMsgHandler(String payload) {
return StringUtils.replace(payload, "foo", "bar");
}
}
B
I
N
D
I
N
G
E
V
E
N
T
S
foo
channel
B
I
N
D
I
N
G
channel
bar
C
O
N
S
U
M
E
R
S
Message Channel Abstraction
Programming Model

39. Spring Cloud Stream
@EnableBinding(Processor.class)
public class Application {
@StreamListener("foo")
@SendTo("bar")
public Flux<Average> sensorAverage(Flux<Sensor> data) {
return . .;
}
}
E
V
E
N
T
S
B
I
N
D
I
N
G
foo
channel
bar
B
I
N
D
I
N
G
channel
C
O
N
S
U
M
E
R
S
Native Reactor Flux’s
Programming Model

40. Spring Cloud Stream
Programming Model
@EnableBinding(Processor.class)
public class Application {
@StreamListener("foo")
@SendTo("bar")
public KStream<Object, Foo> handler(KStream<Object,
Event> input){
return . .;
}
}
Native Kafka Streams
E
V
E
N
T
S
B
I
N
D
I
N
G
foo
channel
bar
B
I
N
D
I
N
G channel
C
O
N
S
U
M
E
R
S

41. Spring Cloud Stream
Programming Model
@EnableBinding(Processor.class)
public class Application {
@Bean
public Function<String, String> toUpperCase() {
return s -> s.toUpperCase();
}
}
E
V
E
N
T
S
B
I
N
D
I
N
G
foo
channel
bar
B
I
N
D
I
N
G channel
C
O
N
S
U
M
E
R
S
Plain Old Java Functions

42. Spring Cloud Stream
Pluggable Binder
Implementations
Stream
Partitions
Consumer
Groups
Message
Headers
Testing
Framework
Content-type
Negotiation
Imperative + Functional
Programming Model
public class TransferServiceImpl implements TransferService {
public TransferServiceImpl(AccountRepository ar) {
this.accountRepository = ar;
}

43. Spring Cloud Data Flow
Spring Cloud Stream
Spring Cloud Task

44. Spring Cloud Task
makes it easy to
orchestrate short-lived
microservices.

45. @EnableTask
public class TimestampTask {
@Bean
public TimestampTask timeStampTask() {
return new TimestampTask();
}
public static class TimestampTask implements CommandLineRunner {
@Override
public void run(String... strings) throws Exception {
DateFormat dateFormat = . .
logger.info(dateFormat.format(new Date()));
}
}
}
Database
R
E
P
O
S
I
T
O
R
Y
An arbitrary business-logic as short-lived Application
@EnableTask
Spring Cloud Task

46. @EnableTask
@EnableBatchProcessing
public class BatchJobApplication {
@Bean
public Step extractStep() { // extract business logic }
@Bean
public Step transformStep() { // transformation logic }
@Bean
public Step loadStep() { // persistence logic }
@Bean
public Job etlJob() {
return this.jobBuilderFactory.get("etlJob")
.start(extractStep())
.next(transformStep())
.next(loadStep())
.build();
}
}
Database
R
E
P
O
S
I
T
O
R
Y
Spring Batch Job as short-lived Application
Spring Cloud Task

47. Spring Cloud Task
Lifecycle Management Transactions
Bookkeeping for Restarts/Replay Historical Representation
Remote
Partitions

48. Spring Cloud Data Flow
Spring Cloud Stream
Spring Cloud Task

49. Spring Cloud Stream
Spring Cloud Task
Dem
o

51. Spring Cloud Stream -> Task
SFTP Source TaskLauncher ETL Job/Task
Database
Orchestrated by Spring Cloud Data Flow
SFTP Server
poll for
new files
publish each file launch task for each file
persist
parsed data

52. Spring Cloud Task -> Stream
SFTP Source TaskLauncher Batch Job/Task
Database
SFTP Server
poll for
new files
publish each file launch task for each file
persist
parsed data
:task-eventsTaskLauncher

53. Spring Cloud Data Flow
Integration Problems
Event Streaming
Orchestrating Short Lived Tasks

56. References
• Software developers are now more valuable to companies than money - CNBC
• Testing Strategies in a Microservice Architecture by Martin Fowler and Toby
Clemson
• Presentation Patterns by Neal Ford, Matthew McCullough, and Nate Schutta
• Don't Walk Away from Complexity, Run • Venkat Subramaniam
• Distributed Sagas: A Protocol for Coordinating Microservices - Caitie McCaffrey
• Dave Syer, Pivotal—Choosing Software Abstractions, SpringOne Platform
2018

57. References (con’t)
• Code for Demos: https://github.com/abhinavrau/scdf-demo
• Spring Cloud Data Flow Microsite Reference Doc
• Spring Cloud Stream Samples | Gitter | StackOverflow
• Spring Cloud Task Samples | Gitter | StackOverflow
• Spring Cloud Skipper Samples | Gitter | StackOverflow

58. Transforming How The World Builds Software
© Copyright 2019 Pivotal Software, Inc. All rights Reserved.