This talk will discuss the state of the Apache NiFi Ecosystem & Community. Apache NiFi is an integrated data logistics platform for automating the movement of data between disparate systems. It provides real-time control that makes it easy to manage the movement of data between any source and any destination. It is data source agnostic, supporting disparate and distributed sources of differing formats, schemas, protocols, speeds and sizes such as machines, geo location devices, click streams, files, social feeds, log files and videos and more. It is configurable plumbing for moving data around, similar to how Fedex, UPS or other courier delivery services move parcels around. And just like those services, Apache NiFi allows you to trace your data in real time, just like you could trace a delivery.

1. 1 © Hortonworks Inc. 2011–2018. All rights reserved.
State of the Apache NiFi Ecosystem &
Community
Aldrin Piri
@aldrinpiri

2. 2 © Hortonworks Inc. 2011–2018. All rights reserved.
About Me
• Sadly, not an Accumulo expert
• I was using cbshell before it went mainstream, though
• Apache NiFi PMC Member and Committer
• Hortonworks

3. 3 © Hortonworks Inc. 2011–2018. All rights reserved.
Agenda
• Apache NiFi
• Apache NiFi Registry
• Apache NiFi - MiNiFi
• Community

4. 4 © Hortonworks Inc. 2011–2018. All rights reserved.
Apache NiFi – A Primer

5. 5 © Hortonworks Inc. 2011–2018. All rights reserved.
The Problem at Hand
Producers A.K.A Things
Anything
AND
Everything
Internet!
Consumers
• User
• Storage
• System
• …More Things

6. 6 © Hortonworks Inc. 2011–2018. All rights reserved.
Moving data effectively is hard
Standards: http://xkcd.com/927/

7. 7 © Hortonworks Inc. 2011–2018. All rights reserved.
Data Pipeline: http://xkcd.com/927/
Empower Users to Reason About Their Data Flow

8. 8 © Hortonworks Inc. 2011–2018. All rights reserved.
8
Apache NiFi
Key Features and Principles
• Guaranteed delivery
• Data buffering
- Backpressure
- Pressure release
• Prioritized queuing
• Flow specific QoS
- Latency vs. throughput
- Loss tolerance
• Data provenance
• Recovery/recording
a rolling log of fine-grained
history
• Visual command and
control
• Pluggable/multi-role
security
• Designed for extension
• Clustering

9. 9 © Hortonworks Inc. 2011–2018. All rights reserved.
NiFi & Data Agnosticism
• NiFi is data agnostic!
• But, NiFi was designed understanding that users
can care about specifics and provides tooling
to interact with specific formats, protocols, etc.
ISO 8601 - http://xkcd.com/1179/
Robustness principle
Be conservative in what you do,
be liberal in what you accept from others“

10. 10 © Hortonworks Inc. 2011–2018. All rights reserved.
1
0

11. 11 © Hortonworks Inc. 2011–2018. All rights reserved.

12. 12 © Hortonworks Inc. 2011–2018. All rights reserved.
Apache NiFi – 0.x to 1.x
and Beyond

13. 13 © Hortonworks Inc. 2011–2018. All rights reserved.
Integrate All the Systems: 274+ Processors,
57 Controller Services
Hash
Extract
Merge
Duplicate
Scan
GeoEnrich
Replace
ConvertSplit
Translate
Route Content
Route Context
Route Text
Control Rate
Distribute Load
Generate Table Fetch
Jolt Transform JSON
Prioritized Delivery
Encrypt
Tail
Evaluate
Execute
All Apache project logos are trademarks of the ASF and the respective projects.
Fetch
HTTP
Syslog
Email
HTML
Image
HL7
FTP
UDP
XML
SFTP
AMQP
WebSocket
Parse Records Convert Records

14. 14 © Hortonworks Inc. 2011–2018. All rights reserved.
1.X Clustering Architecture
OS/Host
JVM
Flow Controller
Web Server
Processor 1 Extension N
FlowFile
Repository
Content
Repository
Provenance
Repository
Local Storage
Standalone
Cluster

15. 15 © Hortonworks Inc. 2011–2018. All rights reserved.
• Now flowfile content can contain many “record”
elements
• Read and write with *Reader and *Writer Controller
Services
• Perform lookups, routing, conversion, SQL queries,
validation, and more…
• 1 + 1 flowfiles = 2 flowfiles
Record Parsing

16. 16 © Hortonworks Inc. 2011–2018. All rights reserved.
New and Upcoming Features
• Enhanced support for containerization
• Auto load-balancing in clustered environments (NIFI-5516)
• Node decommissioning
• Enhanced compatibility with Java 9+
• Introduction of the Apache NiFi Flow Design System
• Common UI/UX across the ecosystem

17. 17 © Hortonworks Inc. 2011–2018. All rights reserved.
Evolving the NiFi
Platform

18. 18 © Hortonworks Inc. 2011–2018. All rights reserved.
• Version my flows?
• Drive CI/CD processes?
• Migrate flows between environments?
• Provision distributions of NiFi with a set of components?
• Make reference datasets/extensions available to the entirety of my data
flow?
• Certify / Audit / Sign-off on flows as compliant per regulations?
How can I … How do I ... What about ...
Listening to our community

19. 19 © Hortonworks Inc. 2011–2018. All rights reserved.
Capturing the essence of a flow in your organization
• The n-dimensions of data flow
• Consider a flowfile to be a singular event at a given juncture in its processing
 A flow is the directed graph of processing at a given point in time
 With each component’s:
 Configuration
 Version
 Referenced Assets

20. 20 © Hortonworks Inc. 2011–2018. All rights reserved.
Apache NiFi Registry

21. 21 © Hortonworks Inc. 2011–2018. All rights reserved.
Operator Experience

22. 22 © Hortonworks Inc. 2011–2018. All rights reserved.
Challenges
• Templates
• Updates/replacement
• Sensitive property replacement
• flow.xml.gz migration
• Key synchronization
• Environment promotion
• Approval processes
• Verifiability

23. 23 © Hortonworks Inc. 2011–2018. All rights reserved.
Template Replacement
• Export a new version of template
• Transfer (somehow)
• Verify?
• Import onto canvas side-by-side existing flow
• Stop processors
• Empty queues
• Reconnect queues
• Start
• Pray?

24. 24 © Hortonworks Inc. 2011–2018. All rights reserved.
Template Replacement

25. 25 © Hortonworks Inc. 2011–2018. All rights reserved.
• Previously, flows were exported via XML templates
• Didn’t contain sensitive values
• Couldn’t be updated in-place
• No tracking system
• NiFi Registry brings asset management as first-class citizen to NiFi
• Flows can be versioned
Introducing Apache NiFi Registry 0.3.0
NiFi Registry for Dataflows

26. 26 © Hortonworks Inc. 2011–2018. All rights reserved.
Flows can be promoted between environments
• Connect multiple NiFi instances to a NiFi Registry instance
• Communicate between multiple NiFi Registry instances
• via multiple Registry Clients
• via NiFi CLI

27. 27 © Hortonworks Inc. 2011–2018. All rights reserved.
Extensibility
• Git-backed persistence
• Share flows via GitHub, etc.
• Commit hooks
• Register a hook & action
• “When a new version of the flow is committed to QA
Registry, email the QA team and post in the QA Deploy
Slack channel”
• Pluggable DB implementations

28. 28 © Hortonworks Inc. 2011–2018. All rights reserved.

29. 29 © Hortonworks Inc. 2011–2018. All rights reserved.

30. 30 © Hortonworks Inc. 2011–2018. All rights reserved.

31. 31 © Hortonworks Inc. 2011–2018. All rights reserved.

32. 32 © Hortonworks Inc. 2011–2018. All rights reserved.
Registry acts as an enabler
• SDLC
• Manage variables, sensitive properties for environments
• Extension Registry
• Association/tagging of data with the flow that created it

33. 33 © Hortonworks Inc. 2011–2018. All rights reserved.
Apache NiFi – MiNiFi

34. 34 © Hortonworks Inc. 2011–2018. All rights reserved.
Apache NiFi - MiNiFi
• Let me get the key parts of NiFi close to where data begins
• Bidirectional data transfer
• Greater illuminate journey with provenance
• NiFi lives in the data center. Give it an enterprise server or a cluster of them.
• MiNiFi lives as close to where data is born and is a guest on that device or
system

35. 35 © Hortonworks Inc. 2011–2018. All rights reserved.
Apache NiFi - MiNiFi: Scoping
• Go small: Java – Write once, run anywhere*
• Feature parity and reuse of core NiFi libraries
• Go smaller: C/C++ – Write once**, run anywhere
• Go smallest: Write n-many times, embed, run anywhere
Language libraries to support tagging, FlowFile format, Site to Site
protocol, and provenance generation without a full processing framework
– Language SDKs, Mobile Platforms
Provide all the key principles of NiFi in varying, smaller footprints

36. 36 © Hortonworks Inc. 2011–2018. All rights reserved.
Apache NiFi - MiNiFi: Centralized Command & Control (C2)
• Provide flow updates, information and assets to instances where
they live
• Act as a gateway to/from network enclaves
• https://s.apache.org/minifi-c2-design-proposal
Extend the reach of user experience and operations

37. 37 © Hortonworks Inc. 2011–2018. All rights reserved.
Community &
Ecosystem

38. 38 © Hortonworks Inc. 2011–2018. All rights reserved.
The Evolution of Apache NiFi
• Our core substrate for data flow is NiFi & MiNiFi
• Command and Control facilitates operations and management of components
• Registry for common tasks with disparate resources across the NiFi ecosystem

39. 39 © Hortonworks Inc. 2011–2018. All rights reserved.
Managing data flow for a courier service
Physical Store
Gateway
Server
Mobile Devices
Registers
Server Cluster
Distribution Center
Kafka
Core Data Center at HQ
Server Cluster
Others
Storm / Spark /
Flink / Apex
Kafka
Storm / Spark / Flink / Apex
On Delivery Routes
Trucks Deliverers
Delivery Truck: Creative Stall, https://thenounproject.com/creativestall/
Deliverer: Rigo Peter, https://thenounproject.com/rigo/
Cash Register: Sergey Patutin, https://thenounproject.com/bdesign.by/
Hand Scanner: Eric Pearson, https://thenounproject.com/epearson001/
Client
Libraries
Client
Libraries
MiNiFi
MiNiFi
NiFi NiFi NiFi NiFi NiFi NiFi
Client
Libraries

40. 40 © Hortonworks Inc. 2011–2018. All rights reserved.
Why the Apache NiFi Ecosystem?
• Moving data is multifaceted in its challenges and these are
present in different contexts at varying scopes
• Provide components and a platform with common tooling
and extensions that are commonly needed but be flexible
for extension in all aspects
• Allow organizations to integrate with their existing infrastructure
• Empower folks managing your infrastructure to make
changes and reason about issues that are occurring
• Data Provenance to show context and data’s journey
• User Interface/Experience a key component

41. 41 © Hortonworks Inc. 2011–2018. All rights reserved.
Community Health

42. 42 © Hortonworks Inc. 2011–2018. All rights reserved.
Project Sites
• NiFi: https://nifi.apache.org
• Subproject MiNiFi: https://nifi.apache.org/minifi/
• Subproject Registry: http://nifi.apache.org/registry.html
Subscribe to and collaborate at
• http://nifi.apache.org/mailing_lists.html
Follow us on Twitter
@apachenifi
Apache NiFi Community and Ecosystem Resources
Learn More and Join Us!

43. 43 © Hortonworks Inc. 2011–2018. All rights reserved.
Thanks! Questions?