my talk prepared for IT-Arena 2021

1. MÖDERN BIG DATA
TRENDS & TECHNOLOGIES
BORIS TROFIMOV
@b0ris_1

2. CUSTOMERS ARCHITECTURES TECHNOLOGIES MATURITY
OF DATA AND MAN INNOVATIONS
A G E N D A

3. OF DATA AND MEN

4. EPIC SPICE WITH BIG DATA
YOUR SOLUTION

5. EPIC FAIL WITH BIG DATA
Y
O
U
R
S
O
L
U
T
I
O
N

6. SOLUTION PERSPECTIVE
⎼⎼

7. UNDER HOOD
⎼⎼

8. UNDER HOOD
⎼⎼

9. UNDER HOOD
⎼⎼

10. TECH PERSPECTIVE
⎼⎼

11. UNDER HOOD
⎼⎼

12. UNDER HOOD
⎼⎼

13. WHO ARE PEOPLE WHO BUILD DATA
PLATFORMS?

14. Big Data

15. DATA ENGINEERING HAS COME
DATA ENGINEERS ARE GETTING
TO PLAY MORE ACTIVE ROLE
COMPANIES HIRE DATA ENGINEERS
MORE IMPACT FROM
BIG DATA TECHNOLOGIES
[KAFKA, SPARK, ETC.]
BLESSING FROM
ST. MARTIN FOWLER
CROSSFUNCTIONAL TEAMS
PER SUBDOMAIN

16. CUSTOMERS

17. Picture

18. AND GIVE US ANOTHER YEAR
SPARK
AWS SCALA

19. SHORT ETA AND TTM

20. MOOOARRR…
LOWER COST OF CHANGE
Changes are local, Easier to make change
Easy to deploy
LOWER MAINTANANCE COSTS
Lower OPEX
Less routine efforts from developers
EASY TO STAFF AND REPLACE
DEVS
Python or Node.js over Scala

21. ARCHITECTURES

22. BYE LAMBDA (1/3)

23. BYE LAMBDA – HI KAPPA (2/3)

24. BYE LAMBDA – HI DELTA (3/3)

26. GREAT SPLIT

27. MONOLYTHIC DATA PLATFORMS

28. DATA MESH VALUES
• Domain ownership
• Data as a product
• Self-serve data platform
• Federated computational
governance

29. ETL VS ELT (1/2)
ETL
• Separates computational
concerns
• Bigger latency before data is
ready for quering

30. ETL VS ELT (2/2)
ELT
• Leverages DWH capabilities
to run transformation
• Data is ready asap
• Relies on dwh capabilities
like data views

31. TECHNOLOGIES

32. THE SQL IS DEAD LONG LIVE THE SQL
3 SQL QUERIES WALK INTO A
NoSQL BAR …
SOME MINUTES LATER THEY WALK OUT
AGAIN BECAUSE THEY COULDN’T FIND A
TABLE.

33. BACK TO SQL (1/2)

34. BACK TO SQL (2/2)
Trend – develop a component where SQL would be a DSL
LEVERAGE CLOUD CAPABILITIES
• Well written SQL code (?!) easier to maintain and reduce complexity
• Lower cost of change
ADVANTAGES

35. MORE SERVERLESS
• Faster TTM
• Less maintenance efforts
• Cheaper solution [in some cases]
ADVANTAGES
Transforming Storage Data
Governance
Data Analytics

36. MATURITY

37. BIG DATA TECHNOLOGY LANDSCAPE

38. DATA GOVERNANCE

40. PRODUCTIZING BIG DATA
Reaching understanding what can
be simplified and automated

41. UPSOLVER
⎼⎼
■ ETL + Data Lake
■ Fully managed platform
■ SQL driven transformations
■ Enterprise - enabled
■ Scalable
■ Cloud-driven

42. INNOVATIONS

43. DATA LAKE HOUSES (1/2)
1ST GENERATION 2nd GENERATION

44. DATA LAKE HOUSES (2/2)
Separating processing (stateless
computational clusters) and persistence
Allows scalable ELT Single infrastructure for DWH and data lake (hello
DataMesh)

46. AI-DRIVEN DATA ANALYTICS (1/2)
INSIGHT GENERATION AUTOMATION NATURAL LANGUAGE
INTERACTION
ADVANCED
ANALYTICS

47. AI-DRIVEN DATA
ANALYTICS (2/2)
• Covering more stacks and
expiriences – better
analytics
• Augmented analytics as a
tradeoff

48. THANK YOU

49. WHY BIG DATA MIGHT MATTER TO YOU
• How does big data influence on ordinary engineering teams?
• More chances to see big data on a projects
• Some big data technologies
• Martin Fawler? Cross functional team

50. • Target audience
• Managers, engineers
• ETA
• Why this talk might have interest to other engineers. No hooks for them
• How big data influeces engineering teams?
• Message
• Message is to give you guys understanding in brief what big data looks like,
latest trends there and who knows maybe you will change your religion
• Give outlook of for people outside big data world what is going on.

51. ideas
• Multiple tiers
• technology
• align things with experience from sigma software (demand from
clients) – simplicity of maintanance, ETA,
• Source
• Maturity of big data
• Challenges
• A way to split this talk into stories
• Kind of great methafor where all sections would be included
somehow

52. • - dbt зараз дуже стала можна штука (https://www.getdbt.com/)
• - Datalake related : Dremio + LakeFS
• - тестування даних https://greatexpectations.io/
• - тренди типу Lakehouse + Data Mesh (і ще тут
https://www.infoq.com/articles/ai-ml-data-engineering-trends-2021/)
• - data discovery https://www.amundsen.io/ +
https://github.com/linkedin/datahub
• а ще тут може щось знайдеш корисне, я сюди майже кожного дня
пишу (https://t.me/bigdatadriven)

53. SUMMARY
• data engineers and leverage big data experitise

54. MORE SCRIPTING
Python
NodeJS

55. trends
• Data Lake Houses
• 1 generation
• 2 generation (snowflake, dreamio)
• Back to SQL (DBT)
• Everywhere on all tiers from execution, storage
• ETL productization
• More cloud
• More PaaS
• Data mesh