The presentation discusses how technological advances have enabled real-time data processing capabilities that allow companies to leverage data streams to drive core business processes. It covers the evolution of data storage, computation and networking technologies. It also discusses stream processing concepts and architectures like Kappa that combine stream and batch processing for analytics. The presentation argues that the most profitable companies are those that allow customers to create their own data flows.

1. The most profitable companies are leveraging
real-time data flows to drive core business
processes.
Presentation by Devin Bost, Senior Data Engineer at Overstock
Contact: dbost@overstock.com
linkedin.com/in/devinbost/

2. Others from my team are
here to answer questions
after the meetup

3. We’re hiring.

4. Good data requires seeing
the end from the beginning

5. “More than 90% of startups
fail, due primarily to self-
destruction”
Marmer, M., Herrmann, B. L., Dogrultan, E., Berman, R., Eesley,
C., & Blank, S. (2011). Startup genome report extra: Premature
scaling. Startup Genome, 10, 1-56.

6. Why do companies fail?

7. “Managers tend to start by building
solutions, but we emphasize the need to
first deeply understand the problem.”
― Nathan Furr, The Innovator's Method: Bringing the Lean Start-up into Your Organization

8. Image from: http://weekendblitz.com/top-7-longest-domestic-us-flights/

9. “[O]rganizations enter the state of decline when they fail
to:
anticipate,
recognize,
avoid,
neutralize or
adapt
to external or internal pressures that threaten the
organization's long-term survival.” (Amankwah-Amoah, J.
(2016). An integrative process model of organisational
failure. Journal of Business Research, 69(9), 3388-3397)

10. “[O]rganizations enter the state of decline when they fail
to:
anticipate,
recognize,
avoid,
neutralize or
adapt
to external or internal pressures that threaten the
organization's long-term survival.” (Amankwah-Amoah, J.
(2016). An integrative process model of organisational
failure. Journal of Business Research, 69(9), 3388-3397)

11. “[O]rganizations enter the state of decline when they fail
to:
anticipate,
recognize,
avoid,
neutralize or
adapt
to external or internal pressures that threaten the
organization's long-term survival.” (Amankwah-Amoah, J.
(2016). An integrative process model of organisational
failure. Journal of Business Research, 69(9), 3388-3397)

12. “[O]rganizations enter the state of decline when they fail
to:
anticipate,
recognize,
avoid,
neutralize or
adapt
to external or internal pressures that threaten the
organization's long-term survival.” (Amankwah-Amoah, J.
(2016). An integrative process model of organisational
failure. Journal of Business Research, 69(9), 3388-3397)

13. “[O]rganizations enter the state of decline when they fail
to:
anticipate,
recognize,
avoid,
neutralize or
adapt
to external or internal pressures that threaten the
organization's long-term survival.” (Amankwah-Amoah, J.
(2016). An integrative process model of organisational
failure. Journal of Business Research, 69(9), 3388-3397)

14. “[O]rganizations enter the state of decline when they fail
to:
anticipate,
recognize,
avoid,
neutralize or
adapt
to external or internal pressures that threaten the
organization's long-term survival.” (Amankwah-Amoah, J.
(2016). An integrative process model of organisational
failure. Journal of Business Research, 69(9), 3388-3397)

15. What is the greatest pressure
on companies today?

16. Technological adaptation

17. It is unavoidable.

18. Society has advanced

19. Data storage:

20. Image from: https://www.quora.com/Why-does-technology-advance-rapidly

21. Image from: https://www.metmuseum.org/toah/works-of-art/14.1.531/

23. Piles of papers

24. Computation also
advanced:

25. Picture of Roman abacus by photographer: Mike Cowlishaw (aus der englischen Wikipedia) - Photographer: Mike Cowlishaw
(aus der englischen Wikipedia), CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=146646

26. Image of slide rule CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=647104

27. Image of punch card by Herman Hollerith - Railroad Gazette, Public Domain, https://commons.wikimedia.org/w/index.php?curid=5034979

28. Then, switching
became automated:

29. Image from: https://www.reddit.com/r/interestingasfuck/comments/8lv16x/the_turing_machine_was_created_out_of_necessity/
Touring Machine, world’s first “computer”

30. allowing storage to
improve further:

31. Image of 5 MB hard drive that weighed over 1 ton from: PC World

32. Image from: PC World

33. Image from: https://www.newegg.com/Product/Product.aspx?Item=N82E16820167458
Sequential reads up to 2.5 GB/s

34. Transport also
evolved:

36. By George Chernilevsky - Own work, Public Domain, https://commons.wikimedia.org/w/index.php?curid=6963942

37. Image from: https://blogs.ufv.ca/cmns/2016/03/20/how-did-we-get-here-a-short-history-of-social-media/
ARPA Network, the birthplace of the internet

38. From: https://www.computerhope.com/jargon/d/dialup.htm
Dial up modem

39. Image from: https://www.motioncontrolonline.org/blog-article.cfm/8-Qualities-of-an-Ideal-Ethernet-Network-for-Motion-Control/33
Ethernet with managed switches

40. Fiber optics

41. Data management
also changed.

42. Image from: https://micheleberger.wordpress.com/2014/05/12/spring-cleaning-for-the-creative-life/

43. VisiCalc spreadsheet on an Apple II
By User:Gortu - apple2history.org, Public Domain, https://commons.wikimedia.org/w/index.php?curid=342925

44. Microsoft Excel
Image from: https://www.excel-easy.com/examples/percent-change.html

45. Relational Database
Image from: https://medium.com/cracking-the-data-science-interview/relational-database-101-a8ace25c12a

46. Relational Database Entity Relations
Image from: https://docs.microsoft.com/en-us/azure/architecture/data-guide/relational-data/

47. Web technology
emerged

48. Model View Controller (MVC) pattern
Image from: https://stackoverflow.com/questions/42695467/is-the-view-in-mvc-supposed-to-know-the-about-the-model-data
(presentation)
(logic)

49. Batch analytics
emerged

50. Introduction of Apache Hadoop:
Image from: https://www.hostingadvice.com/how-to/what-is-hadoop/

51. Then, limitations were
discovered.

52. MVC doesn’t scale
well.
Example 1:

53. Problems with MVC
Image from Facebook, at: https://www.youtube.com/watch?v=nYkdrAPrdcw

54. Data lakes & data
warehouses can begin to
resemble that same pattern.
Example 2:

55. Valuable data that’s scattered
around:
Example A:

56. https://www.mprnews.org/story/2014/08/24/california-earthquake

57. Lots of collections, not very
organized:
Example B:

58. https://www.businessinsider.my/jb-supermarket-shopping-frenzy-backed-johor-crown-prince-apocalyptic-aftermath-aeon-tebrau-city-closed/

59. Data that’s just trash:
Example C:

61. What happens when we use
data like that?

62. Image from: http://serendipics.blogspot.com/2009/06/computer-trash.html

63. A good data lake / data
warehouse / streaming
platform should look like this:

64. Image from: https://www.containerstore.com/s/our-starter-kits/pantry-shelves-starter-kit/1d?productId=11006559

65. Remember, the technology is
the tool, not the solution.

66. Why do data disasters occur?

67. It can happen when we lack
data governance.

68. Data governance problems
aren’t created overnight.

69. LinkedIn had one of these
problems. It looked like this:

70. In 2010:

71. So, LinkedIn invented Kafka:

73. Image from: https://kafka.apache.org/0110/documentation/streams/developer-guide

74. Web technology got
smarter with flows too

75. Image from Facebook, at: https://www.youtube.com/watch?v=nYkdrAPrdcw
Facebook Flux architecture

76. Architectures emerged
to combine stream and
batch processing
paradigms

77. Lambda architecture
By Textractor - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=34963985

78. But, what is stream
processing?

79. “Event streaming”
“Stream analytics”
“Real-time”
“Near-real-time”
“Event-driven”
“Reactive”

80. “Event streaming”
“Stream analytics”
“Real-time”
“Near-real-time”
“Event-driven”
“Reactive”

81. “Event streaming”
“Stream analytics”
“Real-time”
“Near-real-time”
“Event-driven”
“Reactive”

82. “Event streaming”
“Stream analytics”
“Real-time”
“Near-real-time”
“Event-driven”
“Reactive”

83. “Event streaming”
“Stream analytics”
“Real-time”
“Near-real-time”
“Event-driven”
“Reactive”

84. “Event streaming”
“Stream analytics”
“Real-time”
“Near-real-time”
“Event-driven”
“Reactive”

85. “Event streaming”
“Stream analytics”
“Real-time”
“Near-real-time”
“Event-driven”
“Reactive”
“Proactive”

86. “Event streaming”
“Stream analytics”
“Real-time”
“Near-real-time”
“Event-driven”
“Reactive”
“Proactive”
“Predictive”

87. Let’s consider some
examples.

88. Credit
Card
fraud
Immediate
decision vs
after-the-
fact

89. Real-time customer support

90. Financial decision support
Real-time
pricing
decisions

91. Blockchain everything
Image from: http://investors.overstock.com/static-files/b3363f4a-479b-49b8-84b8-0172b39af959

92. Have you needed to change your plans because you received
critical information too late?

93. What do the most
profitable internet
companies have in
common?

95. Self-service data
streams –as-a-
platform

97. “The most profitable companies are
figuring out ways to allow their
customers to create data flows that drive
core business processes.”
From: https://towardsdatascience.com/is-your-big-data-thinking-backwards-e4176413a801

98. “The emergence of digital data streams
is creating strategic opportunities for
existing firms and enabling the
formation of new enterprises.”
Pigni, F., Piccoli, G., & Watson, R. (2016). Digital Data Streams:
Creating value from the real-time flow of big data. California
Management Review, 58(3), 5–25. Bold added. Retrieved
from https://dds.cct.lsu.edu/ddslab/pdf/digitalDataStream.pdf

99. Dimensional Research
surveyed 307 IT decision
makers responsible for
corporate data solutions.

100. “How important is it to your
organization that you are
able to process and act on
data as quickly as possible?”
Research Question:

101. https://streaml.io/pdf/Speed-of-Data-Survey-Report.pdf

102. “How many say it’s
critically important?”
Research Question:

103. https://streaml.io/pdf/Speed-of-Data-Survey-Report.pdf

104. If it takes your company more
than hours to answer data
questions, your company may be
in serious danger.

105. “For what types of applications
would your organization benefit
from processing data immediately
rather than in batch?”
Research Question:

106. https://streaml.io/pdf/Speed-of-Data-Survey-Report.pdf

107. “What benefits can your organization
gain by processing data immediately as it
arrives rather than waiting to process it
periodically in batches?”
Research Question:

108. https://streaml.io/pdf/Speed-of-Data-Survey-Report.pdf

109. “Which of the following additional
technology capabilities would your
organization find valuable?"
Research Question:

110. https://streaml.io/pdf/Speed-of-Data-Survey-Report.pdf

111. “How would the technology
capabilities from the previous
question benefit your
organization?"
Research Question:

112. https://streaml.io/pdf/Speed-of-Data-Survey-Report.pdf

113. Image from: http://gotocon.com/aarhus-2012/thebigdatavaluecontinuum

114. Image from: http://gotocon.com/aarhus-2012/thebigdatavaluecontinuum
Milliseconds Hundreds of milliseconds Minutes Hours

115. What if we could combine
them?

116. Image from: http://gotocon.com/aarhus-2012/thebigdatavaluecontinuum
Milliseconds

117. Can we do that with
stream processing?
Can we do that with
stream processing?
Can we do that with
stream processing?
Can we do that with
stream processing?

118. Stream processing is key
to accomplishing this.

119. Kappa architecture
From: https://towardsdatascience.com/a-brief-introduction-to-two-data-processing-architectures-lambda-and-kappa-for-big-data-4f35c28005bb

120. From: https://medium.com/@devin.bost/the-kappa-architecture-8105a3c10f98

121. But, streaming isn’t just
for analytics!

122. Serverless + Streaming

123. From: http://lukeangel.co/cross-platform/docker-servless-faas-functions-as-a-service/
Capability
driven by
streaming
data

124. Functions
as a
service
Image from: https://blog.algorithmia.com/serverless-computing/

125. Now, what if we could
combine all of these
ideas?

126. Featuring the next-
generation streaming
engine

128. Inspired by:

136. So, what does Pulsar
offer?

137. Multi-tenancy

138. Geo-replication

139. Security

140. Guaranteed
delivery

141. Guaranteed
delivery
Picture from:
https://community.ebay.com/t5/Selling/Packag
e-Allegedly-Damaged-by-USPS/qaq-p/27683454

142. Persistent
Stateful
Storage

143. Extremely fast

144. Designed for
latencies < 5 ms at
scale

145. Horizontal
scaling
f(x)f(x)f(x)f(x)f(x)f(x)f(x)f(x)f(x)f(x)f(x)f(x)f(x)f(x)
f(x)f(x)f(x)f(x)
f(x)f(x)f(x)f(x)f(x)f(x)f(x)f(x)
f(x) f(x)f(x)f(x)
Time

146. Image from: https://oceanservice.noaa.gov/facts/riversnotsalty.html Image from: https://i.ytimg.com/vi/wHlvTKoXOIk/maxresdefault.jpg
Stream
organization

147. Pulsar
Functions

148. From: https://www.forbes.com/sites/gilpress/2016/03/23/data-preparation-most-time-consuming-least-enjoyable-data-science-task-survey-says/#5294ca156f63

149. “Real-time processing
of streaming big data is crucial to
finding meaningful data and
reacting to it.”
Avinash, B. M., & Babu, S. H. (2018). BIG DATA TECHNOLOGIES FOR E-
BUSINESS--FUTURE OPPORTUNITIES, CHALLENGES AHEAD AND
GROWING TRENDS. International Journal of Advanced Research in
Computer Science, 9(2).

150. “[P]reprocessing . . . can improve
performance by many orders of
magnitude.”
https://medium.com/@devin.bost/the-kappa-architecture-8105a3c10f98

151. Good data Great machine
learning ROI

152. Lesson learned:
Drive your market, not
your cost optimization

153. Lesson learned:
Streaming and ML applications will be
significantly more successful if they directly
enhance the customer experience.

154. We’re hiring.

155. The most profitable companies are leveraging
real-time data flows to drive core business
processes.
Presentation by Devin Bost, Senior Data Engineer at Overstock
Contact: dbost@overstock.com
linkedin.com/in/devinbost/