Community Update of Apache Flink as presented at the Flink Meetup in Berlin, Germany

1. Berlin Apache Flink Meetup #12
Community Update & 2015 – The Year in Review
December 2015
Robert Metzger
Committer and PMC Member
rmetzger@apache.org
@rmetzger_

2. What happened?
• 0.10.0 and 0.10.1 released
– Event-time streaming (out of order streams, sessions)
– Master HA  no single point of failure
– DataStream API graduated
– New monitoring interface
– Offheap memory
– Outer joins
– Gelly Scala API
• Otto released a Flink testing framework:
https://github.com/ottogroup/flink-spector
flink.apache.org 1

3. Now in master (1.0-SNAPSHOT)
flink.apache.org 2
• Kafka Connector improvements: Multiple topics,
keys, deleted messages, periodic offset commit
• Flink in firewalled setups
• Out of core state backend for JDBC
• Major improvements in the flink-storm module
• RabbitMQ Source with exactly-once
• Many other operational and usability
improvements

4. Pending Pull Requests
• Savepoints: Trigger state checkpoints manually
and restore a job from a savepoint
• Submit Flink jobs through JobManager web
interface
• Add range partitioner with sampling (FLINK-7)
• “stop” signal for streaming topologies
• Monitoring: JMX beans & checkpoint metrics
flink.apache.org 3

5. Reading List
• Realtime dashboards with Flink and ElasticSearch
https://www.elastic.co/blog/building-real-time-dashboard-applications-with-apache-flink-elasticsearch-and-kibana
• Introduction to streaming windows
http://flink.apache.org/news/2015/12/04/Introducing-windows.html
• Flink + Storm: Run Storm topologies on Flink
http://flink.apache.org/news/2015/12/11/storm-compatibility.html
• New streaming use cases
http://data-artisans.com/how-apache-flink-enables-new-streaming-applications-part-1/
• Streaming with Flink Intro (reading twitter data)
http://blog.brakmic.com/stream-processing-with-apache-flink/
flink.apache.org 4

6. Flink 2015: The journey to
mainstream
2015 – The Year in Review
flink.apache.org 5

7. Let the numbers speak:
GitHub
flink.apache.org 6
February 2015
December 2015

8. Let the numbers speak:
Meetup Groups
flink.apache.org 7
From one Meetup group to a global community

9. Let the facts speak:
Organizations at Flink Forward 2015
flink.apache.org 88

10. Let the facts speak:
Articles about Flink featured in
9

11. Let the facts speak:
Hadoop Summit 2016: 17 Flink talks
10

12. flink.apache.org 11
• Check out the slides: http://flink-
forward.org/?post_type=session
• Video recordings on YouTube, “Flink
Forward” channel
Apropos

13. Feature Timeline 2015
• January
– Flink 0.8
• February
– Gelly
– Table API
• March
– Flink-Google Data Flow
Runner (batch)
– FlinkML
• April
– Python API
• May
– Static Code Analysis
– Zeppelin Integration
• June
– 0.9: Akka, batch
exchange, exactly-once
• July
– New dashboard
– first HA features
flink.apache.org 12

14. Feature Timeline 2015
• August
– Gelly Scala API
– DataSet.sample()
• September
– Cascading on Flink
– Scala 2.11 (and 2.10)
– Offheap memory
– Reworked windowing
– Rolling file sink
– Elasticsearch sink
– High Availability
flink.apache.org 13
• October
– Reworked Kafka
Consumer
– Streaming API
graduation
• November
– Flink 0.10
– JDBC backend
• December
– Flink behind firewalls

15. git Stats 2015
git log --pretty=oneline --after=1/1/2015 | wc -l
2201 commits in 2015
flink.apache.org 14
Lines of code
Commits per day
Commits per hour

16. We are not done yet
• Flink Forward 2016
• Roadmap 2016 (pending community discussion)
– Autoscaling
– Stream SQL
– Managed Memory State
… much more
Let’s make 2016 the Year of Flink!
flink.apache.org 15