Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.
Data Integration and Real-time
Data Processing with
Spring Boot
Sabby Anandan
@sabbyanandan
- Introduce Spring projects
- Discuss how they address
Data Integration / Data
Processing Challenges
Goal:
A toolkit for building
data integration, real-
time streaming, and
batch data processing
pipelines.
Spring Cloud Data Flow
A toolkit for building
data integration, real-
time streaming, and
batch data processing
pipelines.
Spring Cloud Data Flow
- Decentralization (no ESB)
- Lightweight applications that contain
integration logic
- Loose coupling through message
cha...
A toolkit for building
data integration, real-
time streaming, and
batch data processing
pipelines.
Spring Cloud Data Flow
Stream processing is much
more than just EIP
Spring Cloud Stream
a event-driven microservices framework
@EnableBinding(Processor.class)
public class Application {
@Str...
Spring Cloud Stream
a event-driven microservices framework
@EnableBinding(Processor.class)
public class Application {
@Str...
Spring Cloud Stream
a event-driven microservices framework
@EnableBinding(Processor.class)
public class Application {
@Str...
Spring Cloud Stream
a event-driven microservices framework
@EnableBinding(Processor.class)
public class Application {
@Bea...
Spring Cloud Stream
a event-driven microservices framework
Pluggable Binder
Implementations
Stream
Partitions
Consumer
Gro...
Spring Cloud Stream
a event-driven microservices framework
Pluggable Binder
Implementations
Rabbit MQ Apache Kafka Google ...
Spring Cloud Stream
a event-driven microservices framework
Pluggable Binder
Implementations
Stream
Partitions
Consumer
Gro...
Demo #1
Events and Data Intensive Applications
:UserCreated
:UserCreated
:UserNameChanged
:UserActivated
User activity in
the last 30s
Users created in the
last 2 mins
User interaction by
region in the last
1hr window
1
3
2
1
2
3
Websocket Connection
REST request/response
REST request/response
A toolkit for building
data integration, real-
time streaming, and
batch data
processing pipelines.
Spring Cloud Data Flow
JSR 352
- Closely related processing steps that
perform a discrete business process
- Deployable unit comprised of one or
more job...
Spring Cloud Task
a short-lived microservices framework
@EnableTask
@EnableBatchProcessing
public class BatchJobApplicatio...
Spring Cloud Task
a short-lived microservices framework
@EnableTask
public class TimestampTask {
@Bean
public TimestampTas...
Spring Cloud Task
a short-lived microservices framework
Lifecycle Management Transactions
Bookkeeping for Restarts/Replay ...
Demo #2
24/7 ETL: Cloud-native File Ingest
SFTP Source TaskLauncher ETL Job/Task
Database
Orchestrated by Spring Cloud Data FlowSFTP Server
poll for
new files
publis...
A toolkit for building
data integration, real-
time streaming, and
batch data processing
pipelines.
Spring Cloud Data Flow
A toolkit for building
data integration, real-
time streaming, and
batch data processing
pipelines.
Spring Cloud Data Flow...
Developer Focus
Ideation Implementation Production
Demo #3
CI/CD for Data Pipelines
Mask each Payload
111-22-3333
444-55-6666
777-88-9999
. . .
The Security Number = xxx-xx-3333
The Security Number = xxx-xx...
Let’s Recap
Spring Cloud Stream
Build highly scalable
event-driven
microservices
connected with shared
messaging systems.
Spring Cloud...
Next
Spring Boot 2.1 Compatibility: Stream, Task, Skipper, and SCDF
Function Composition / Function Chaining
OAuth2 + Open...
Resources
Spring Cloud Stream Samples | Gitter | StackOverflow
Spring Cloud Task Samples | Gitter | StackOverflow
Spring C...
Q+A
Data Integration and Real-Time Data Processing with Spring Boot
Data Integration and Real-Time Data Processing with Spring Boot
Data Integration and Real-Time Data Processing with Spring Boot
Próxima SlideShare
Cargando en…5
×

Data Integration and Real-Time Data Processing with Spring Boot

1.449 visualizaciones

Publicado el

SF JUG talk by Sabby Anandan at Pivotal Software.
https://www.meetup.com/sfjava/events/256850303/

Publicado en: Tecnología
  • Sé el primero en comentar

Data Integration and Real-Time Data Processing with Spring Boot

  1. 1. Data Integration and Real-time Data Processing with Spring Boot Sabby Anandan @sabbyanandan
  2. 2. - Introduce Spring projects - Discuss how they address Data Integration / Data Processing Challenges Goal:
  3. 3. A toolkit for building data integration, real- time streaming, and batch data processing pipelines. Spring Cloud Data Flow
  4. 4. A toolkit for building data integration, real- time streaming, and batch data processing pipelines. Spring Cloud Data Flow
  5. 5. - Decentralization (no ESB) - Lightweight applications that contain integration logic - Loose coupling through message channels Spring Integration
  6. 6. A toolkit for building data integration, real- time streaming, and batch data processing pipelines. Spring Cloud Data Flow
  7. 7. Stream processing is much more than just EIP
  8. 8. Spring Cloud Stream a event-driven microservices framework @EnableBinding(Processor.class) public class Application { @StreamListener("foo") @SendTo("bar") public String replaceStringMsgHandler(String payload) { return StringUtils.replace(payload, "foo", "bar"); } } B I N D I N G E V E N T S foo channel B I N D I N G channel bar C O N S U M E R S Programming Model: Message Channel Abstraction
  9. 9. Spring Cloud Stream a event-driven microservices framework @EnableBinding(Processor.class) public class Application { @StreamListener("foo") @SendTo("bar") public KStream<Object, Foo> handler(KStream<Object, Event> input){ return . .; } } E V E N T S B I N D I N G foo channel bar B I N D I N G channel C O N S U M E R S Programming Model: Native Kafka Streams
  10. 10. Spring Cloud Stream a event-driven microservices framework @EnableBinding(Processor.class) public class Application { @StreamListener("foo") @SendTo("bar") public Flux<Average> sensorAverage(Flux<Sensor> data) { return . .; } } E V E N T S B I N D I N G foo channel bar B I N D I N G channel C O N S U M E R S Programming Model: Native Reactor Flux’s
  11. 11. Spring Cloud Stream a event-driven microservices framework @EnableBinding(Processor.class) public class Application { @Bean public Function<String, String> toUpperCase() { return s -> s.toUpperCase(); } } E V E N T S B I N D I N G foo channel bar B I N D I N G channel C O N S U M E R S Programming Model: Plain Old Java Functions
  12. 12. Spring Cloud Stream a event-driven microservices framework Pluggable Binder Implementations Stream Partitions Consumer Groups Message Headers Testing Framework Content-type Negotiation Imperative + Functional Programming Model public class TransferServiceImpl implements TransferService { public TransferServiceImpl(AccountRepository ar) { this.accountRepository = ar; }
  13. 13. Spring Cloud Stream a event-driven microservices framework Pluggable Binder Implementations Rabbit MQ Apache Kafka Google PubSub Amazon Kinesis Azure Event Hubs Solace Same code + Same test-harness Drop-in replacement for a variety of Messaging SystemsOpportunities:
  14. 14. Spring Cloud Stream a event-driven microservices framework Pluggable Binder Implementations Stream Partitions Consumer Groups Message Headers Testing Framework Content-type Negotiation Imperative + Functional Programming Model public class TransferServiceImpl implements TransferService { public TransferServiceImpl(AccountRepository ar) { this.accountRepository = ar; }
  15. 15. Demo #1 Events and Data Intensive Applications
  16. 16. :UserCreated :UserCreated :UserNameChanged :UserActivated
  17. 17. User activity in the last 30s Users created in the last 2 mins User interaction by region in the last 1hr window
  18. 18. 1 3 2 1 2 3 Websocket Connection REST request/response REST request/response
  19. 19. A toolkit for building data integration, real- time streaming, and batch data processing pipelines. Spring Cloud Data Flow
  20. 20. JSR 352
  21. 21. - Closely related processing steps that perform a discrete business process - Deployable unit comprised of one or more job steps - Lifecycle management for jobs/steps Spring Batch JSR 352
  22. 22. Spring Cloud Task a short-lived microservices framework @EnableTask @EnableBatchProcessing public class BatchJobApplication { @Bean public Step extractStep() { // extract business logic } @Bean public Step transformStep() { // transformation logic } @Bean public Step loadStep() { // persistence logic } @Bean public Job etlJob() { return this.jobBuilderFactory.get("etlJob") .start(extractStep()) .next(transformStep()) .next(loadStep()) .build(); } } Database R E P O S I T O R Y Programming Model: Spring Batch Job as Short-lived Application
  23. 23. Spring Cloud Task a short-lived microservices framework @EnableTask public class TimestampTask { @Bean public TimestampTask timeStampTask() { return new TimestampTask(); } public static class TimestampTask implements CommandLineRunner { @Override public void run(String... strings) throws Exception { DateFormat dateFormat = . . logger.info(dateFormat.format(new Date())); } } } Database R E P O S I T O R Y Programming Model: An arbitrary business-logic as Short-lived Application
  24. 24. Spring Cloud Task a short-lived microservices framework Lifecycle Management Transactions Bookkeeping for Restarts/Replay Historical Representation Remote Partitions
  25. 25. Demo #2 24/7 ETL: Cloud-native File Ingest
  26. 26. SFTP Source TaskLauncher ETL Job/Task Database Orchestrated by Spring Cloud Data FlowSFTP Server poll for new files publish each file launch task for each file persist parsed data
  27. 27. A toolkit for building data integration, real- time streaming, and batch data processing pipelines. Spring Cloud Data Flow
  28. 28. A toolkit for building data integration, real- time streaming, and batch data processing pipelines. Spring Cloud Data Flow But wait, there’s more!
  29. 29. Developer Focus Ideation Implementation Production
  30. 30. Demo #3 CI/CD for Data Pipelines
  31. 31. Mask each Payload 111-22-3333 444-55-6666 777-88-9999 . . . The Security Number = xxx-xx-3333 The Security Number = xxx-xx-6666 The Security Number = xxx-xx-9999 . . . Don’t Disturb Don’t Disturb Fix This!
  32. 32. Let’s Recap
  33. 33. Spring Cloud Stream Build highly scalable event-driven microservices connected with shared messaging systems. Spring Cloud Task Build short-lived microservices to perform data processing locally or in the cloud. Spring Cloud Skipper Discover applications and manage their lifecycle on multiple Cloud Platforms. Spring Cloud Data Flow Orchestrate data pipelines made of Spring Cloud Stream or Spring Cloud Task microservices. Consolidate Development and Testing Practices Standardize CI/CD Tooling & AutomationOpportunities:
  34. 34. Next Spring Boot 2.1 Compatibility: Stream, Task, Skipper, and SCDF Function Composition / Function Chaining OAuth2 + OpenID Connect by Default Deeper Integration with Micrometer for Metrics/Monitoring New Data Integration Apps
  35. 35. Resources Spring Cloud Stream Samples | Gitter | StackOverflow Spring Cloud Task Samples | Gitter | StackOverflow Spring Cloud Skipper Samples | Gitter | StackOverflow Spring Cloud Data Flow Samples | Gitter | StackOverflow Demo #1: Events + Kafka Streams Demo #2: File-ingest Demo #3: CI/CD for Data Pipelines
  36. 36. Q+A

×