Many organizations are immature when it comes to data and analytics use. The answer lies in delivering a greater level of insight from data, straight to the point of need. There are so many Data Architecture best practices today, accumulated from years of practice. In this webinar, William will look at some Data Architecture best practices that he believes have emerged in the past two years and are not worked into many enterprise data programs yet. These are keepers and will be required to move towards, by one means or another, so it’s best to mindfully work them into the environment.

1. Data Architecture
Best Practices for
Advanced Analytics
Presented by: William McKnight
“#1 Global Influencer in Big Data” Thinkers360
President, McKnight Consulting Group
A 2 time Inc. 5000 Company
@williammcknight
www.mcknightcg.com
(214) 514-1444
Second Thursday of Every Month, at 2:00 ET
With William McKnight

2. Image Goes
Here
Know Better®
May 12, 2022

3. © 2022 ChaosSearch, Inc.
ChaosSearch Activates the Data Lake
for Analytics at Scale
The ChaosSearch Data Lake Platform
eliminates complexity and helps organizations
overcome the challenges posed by costly,
siloed analytics solutions.
Users perform both search and SQL queries
concurrently and in situ from their cloud
object storage, without data pipelining,
transformation, or movement.
The end result: Dramatic reductions in time,
cost and complexity.
FEATURED CUSTOMERS

4. © 2022 ChaosSearch, Inc.
The ChaosSearch Approach: Connect Seamlessly, Eliminate Complexity
Constantly Scale and Reduce Cost
3
* This is a roadmap item and subject to change.
BENEFICIAL OUTCOMES
✓ One unified data lake for analytics at scale
✓ Combined search, SQL insights (PLG, security, etc.)
✓ No data pipelines or data movement
✓ No schema management, sharding or managing
server clusters
✓ Simple, open API access to analytics tools of choice
✓ Scale and performance for analytic workloads with up
to 80% cost savings
✓ Unprecedented simplicity

5. © 2022 ChaosSearch, Inc.
Optimize Cloud Apps, Infrastructure and Security
Optimize CloudOps/DevOps
• Unlimited retention to optimize troubleshooting and
performance of increasingly complex cloud architectures
• Better log coverage to shorten time to resolution
• Eliminate administrative toil, reduce operational costs
4
Optimize SecOps
• Affordable long-term retention for in-depth forensics
• Centralize logs in a security data lake for end-to-end
visibility and monitoring
• Simpler, more cost-effective compliance

6. “With the move to ChaosSearch, 98% of all
operational burdens have been lifted from us,
allowing us to focus on Blackboard-specific
tasks.”
Joel Snook, Director, DevOps Engineering

7. © 2022 ChaosSearch, Inc. 6
Single Access Pane on Global Deployment
Case Study
Challenge
Need to reduce Elasticsearch costs & improve
stability
• Architectural challenges at scale caused system
downtime requiring support
• Challenges with infrastructure planning due to
unpredictable log spikes -- especially during
COVID-19 when demand grew 3,000%
• SREs spent 10-15+ hours per week maintaining
environment
Solution
• GDPR-compliant ChaosSearch log analytics solution provides
single access pane to storage across multiple environments
• True managed service with 99.999% availability
• Use cases: visibility of cloud environments at scale, long-term
app troubleshooting & alerting, root cause analysis
Impact
• Increased uptime with decreased management overhead
• Retention more than doubled at 50% cost
• No more data duplication
• Querying activities scale on demand without manual
reconfigurations or cost increases
• SREs spend time on value-add work vs. ELK stack
maintenance
Need access to long-term data for resolving
academic disputes, compliance purposes, etc.

8. © 2022 ChaosSearch, Inc.
Before: Amazon ES
Single Access Pane on Global Deployment
© 2021 ChaosSearch, Inc. 7
Case Study
• Multiple environments per region per team (3 teams)
• Unstable environment with massive admin pain & multiple silos with failure points
(master nodes)
• $2M per year
• 1-3 days retention
AZ2
AZ1
…
…
ca-central-1
AZ2
AZ1
…
…
us-east-2
AZ2
AZ1
…
…
ap-northeast-2
AZ2
AZ1
…
…
eu-central-1
AZ2
AZ1
…
…
AZ2
AZ1
…
Master
Master
Master
Data
Node
Data
Node
Data
Node
Data
Node
Master
Master
Master
Data
Node
Data
Node
Data
Node
Data
Node
Master
Master
Master
Data
Node
Data
Node
Data
Node
Data
Node
Master
Master
Master
Data
Node
Data
Node
Data
Node
Data
Node
Master
Master
Master
Data
Node
Data
Node
Data
Node
Data
Node
Master
Master
Master
Data
Node
Data
Node
Data
Node
Data
Node …
ap-southeast-2
ap-southeast-1
With ChaosSearch
Single access pane,
multiple environments for
storage & access
isolation (e.g., GDPR)
$950k per year
(50%+ discount)
True managed service
with 5 9s availability
7 days retention
(could be unlimited)

9. © 2022 ChaosSearch, Inc.
A Question for Our Audience Today…
8
What are your greatest challenges when it comes to deriving insight from all your
data?
• Expertise
• Resources
• Technology
• Time
• Cost
• Other

10. Thank you!
Learn more
● ChaosSearch.io
● Free Trial - chaossearch.io/trial
● Demo Webinar - chaossearch.io/demo-webinar
● 2022 Cloud Data & Analytics Survey Report

11. © 2022 ChaosSearch, Inc. 10
Connect to any
and all data in
your cloud object
storage
Index into highly
compressed,
unified
representation of
data that never
leaves your
storage
Prepare your
data views for
discovery and
analytics with no
data movement
Use your existing
tools for
• Log analytics
• Operational BI
queries
• Dashboards
• Monitoring and
alerting
ChaosSearch is Built for Operational Analytics at Scale
Index
2 Refine
3 Analyze
4
Store
1

12. © 2022 ChaosSearch, Inc.
Log Analytics Transformed
Before: Elasticsearch (ELK stack)
DevO
ps
SecO
ps
LOB
???
• Limited retention
• Expensive to scale
• Management and
configuration
challenges
• Downtime created by
instability at scale
• Multiple data silos
created due to the
limits above
Cloud Object Storage
i.e., Google GCS, AWS S3
Dev
Ops
Sec
Ops
LOB ???
PUBLISHED
ELASTIC API
One unified data lake
Unlimited scale and retention.
Save up to 80% on Managed Service with 99.99% uptime.
With ChaosSearch
11

13. Appendix

14. © 2022 ChaosSearch, Inc. 13
Partially In-Situ architectures do
work and add incremental value.
But fall short of a true
breakthrough.
In-Situ
INFORMATION
In-Situ
PROCESSING
• Underlying representation
of data is native to use
case
• Multi-model access
• Distributed
processing
• Leveraging the
power of cloud
computing
elasticity
In-Situ
ALGORITHM
• Low memory
execution
• Stream based
operations
A fully In-Situ Architecture is needed to
unlock the full spectrum of In-Situ advantages

15. A Unified Data Lake Architecture for Log and SQL Analytics
14
ChaosSearch uniquely solves for known and unknown data and queries
* Source: Gartner. DW = data warehouse
The Zone of Confusion Within the Data
and Analytics Infrastructure Model
Expanding
Understanding and
Investigating
Founational
Core
Innovation
and Exploration
Establish
Value
Traditional DW
Data Lake
Zone of Confusion
Questions
Known Unknown
Data
Unknown
Known
Bring together search and relational analytics
• Eliminate pipelines, ETL, data movements
• Faster insights
A unified data lake architecture that supports:
• Innovation & exploration
• Investigative queries
• Operational analytics
Disrupts the economics of Big Data

16. A Unified Data Lake Architecture for Log and SQL Analytics
15
ChaosSearch uniquely solves for broadest scope of analytics needs
Bring together search and relational analytics
• Eliminate pipelines, ETL, data movements
• Faster insights
A unified data lake architecture that supports:
• Innovation & exploration
• Investigative queries
• Operational analytics
Disrupts the economics of Big Data
* Source: Gartner. DW = data warehouse
The Zone of Confusion Within the Data
and Analytics Infrastructure Model
Expanding
Understanding and
Investigating
Founational
Core
Innovation
and Exploration
Establish
Value
Traditional DW
Data Lake
Questions
Known Unknown
Data
Unknown
Known

17. William McKnight
President, McKnight Consulting Group
• Consulted to Pfizer, Scotiabank, Fidelity, TD
Ameritrade, Teva Pharmaceuticals, Verizon, and many
other Global 1000 companies
• Frequent keynote speaker and trainer internationally
• Hundreds of articles, blogs and white papers in
publication
• Focused on delivering business value and solving
business problems utilizing proven, streamlined
approaches to information management
• Former Database Engineer, Fortune 50 Information
Technology executive and Ernst&Young Entrepreneur
of Year Finalist
• Owner/consultant: Data strategy and implementation
consulting firm
William McKnight
The Savvy Manager’s Guide
The
Savvy
Manager’s
Guide
Information
Management
Information Management
Strategies for Gaining a
Competitive Advantage with Data
2

18. 3
All Data Under
Management
Best Practice:
Get all data
under
management

19. Data is Under Management when it is…
• In a leveragable platform
• In an appropriate platform for its profile and
usage
• With high non-functionals (Availability,
performance, scalability, stability, durability,
secure)
• Data is captured at the most granular level
• Data is at a data quality standard (as
defined by Data Governance)
• Enables self-service
4
Best Practice:
Enpower
everyone with
true self-service

20. “80% of analysts’ time
is spent simply discovering
and preparing data.”
What’s Your Data Strategy, Thomas Davenport, HBR 2017
Best Practice:
Start getting
concerned with
the tools and
processes of the
analyst

21. The Relational Database Data Page
© McKnight Consulting Group, 2010
Page Header
Page
Footer
Row IDs
Records
1120MCG William McKnight President
214-514-1444 wmcknight@mcknightcg.com
1121Stolt Offshore MS Ltd Joe Tyron Director 226-5555-
1269 jtyron@stoltoffshore.com
1122Medtronic, Inc. Mark Smith Principle Database Administrator
763-555-2557
mark.smith@medtronic.com

22. Columnar Orientation
7
Best Practice:
Make all
analytic
structure
columnar

23. Data Lakes
• P
8
Parquet format
Best Practice:
Put big data in
data lakes
Best Practice:
Index the data
lake

24. Data Lakes
• Common & centralized storage for the enterprise
• No defined data model into which the data is
formed
• No relationships between the datasets
• Historical data retention
• All data formats
• For big data
• Analytical processing
• Data scientists and analysts
• Less governance/quality than data warehouse
– Focus: Ingestion
9

25. Graph Databases
Bridge
vertex
Bridge
vertex
10
• Subject: John R Peterson Predicate: Knows Object: Frank T
Smith
• Subject: Triple #1 Predicate: Confidence Percent Object: 70
• Subject: Triple #1 Predicate: Provenance Object: Mary L Jones
Best Practice:
Use graph
databases for
sizable
connected data

26. Data Virtualization
“The right answer is not
always to centralize the
data. Data Virtualization
will be of utmost
importance as the
‘perpetual short-term’
solution to the need.”
11
Data Warehouses
Marts & Cubes Operational
Data Stores Transactional
Sources
File Systems
Big Data
Enterprise Data
Virtualization
Best Practice:
Enable data
virtualization for
edge and
temporary
needs

27. Enterprise Analytic Stack
• Dedicated Compute
• Storage
• Data Integration
• Streaming
• Analytics
• Data Exploration
• Data Lake
• Business Intelligence
• Machine Learning
• Identity Management
• Data Catalog
• Data Virtualization
Best Practice:
Leverage best of
breed for your
analytics stack

28. • Autonomous Administration
• Lack of Platform Features Leads to Increased
Configuration and Management
– stored procedures, referential integrity and uniqueness
capabilities
– mission critical options for backup and disaster recovery, which
typically includes a standby database
– full ANSI-SQL compliance
• Performance
Total Cost of Ownership is More Than Just
Cloud Costs
Best Practice:
Get a strong
handle on your
cloud costs

29. Capabilities for Data Integration for
Enterprise Data
• Comprehensive Native Connectivity
• Multi-Latency Data Ingestion
• Data Integration (in ETL, ELT, Streaming)
• Data Quality and Data Governance
• Data Cataloging and Metadata
Management
• Enterprise Trust, Enterprise Scale (or Class)
• AI Intelligence and Automation
• Ecosystem and Multi-cloud

30. Data Integration Options
Project Technical Environments Recommended For
Consideration Project Scope
Heterogenous:
Cloudera Any Any
IBM Any Any
Informatica Any Any
Talend Any Any
Specialist:
AWS (Glue) Environments on AWS with core of Redshift, EMR Any
Azure (Azure Data Factory) Environments on Azure with core of Synapse, HDInsight Any
FiveTran Any Contained scope
Google Environments on GCP with core of BQ, DataProc Any
Matillion Any Contained scope
Oracle Environments with Oracle database Any
SAP SAP-only environments SAP projects
Best Practice:
Use fit-for-
purpose data
integration

31. Competitive Analytic
Architectures

32. Architecture Component Needs
• Security and Privacy
• Governance and Compliance
• Availability
• Backup and Recovery
• Performance
• Scalability
• Licensing
17

33. Analytics Reference Architecture
Logs
(Apps, Web,
Devices)
User tracking
Operational
Metrics
Offload
data
Raw Data Topics
JSON, AVRO
Processed
Data Topics
Sensors
and
/ or
Transactiona
l/ Context
Data
OLTP/ODS
ETL
Or
EL with
T in Spark
Batch
Low
Latency
Applications
Files
In-
database
analytics
Reach
through
or ETL/ELT
or
Stream
Processing
or
Stream
Processing
Q
Q
Data
Warehouse
Data Lake

34. Data Lakehouse
19
and
/ or
In-
database
analytics
Reach
through
Q
Q
Data
Warehouse
Data Lake

35. Data Mesh
Logs
(Apps, Web,
Devices)
User tracking
Operational
Metrics
Offload
data
Sensors
and
/ or
Transactiona
l/ Context
Data
OLTP/ODS
Batch
Low
Latency
Applications
Files
In-
database
analytics
Reach
through
or ETL/ELT
or
Stream
Processing
or
Stream
Processing
Q
Q
Data
Warehouse
Data Lake
ETL
Or
EL with
T in Spark
Raw Data Topics
JSON, AVRO
Processed
Data Topics

36. Data Fabric
Logs
(Apps, Web,
Devices)
User tracking
Operational
Metrics
Raw Data Topics
JSON, AVRO
Processed
Data Topics
Sensors
and
/ or
Transactiona
l/ Context
Data
OLTP/ODS
ETL
Or
EL with
T in Spark
Batch
Low
Latency
Applications
Files
In-
database
analytics
or
Stream
Processing
Data
Warehouse
Data Lake

37. Best Practice: Pursue
mesh and fabric
architectures to the
degree possible

38. Data Cloud (Snowflake)
Logs
(Apps, Web,
Devices)
User tracking
Operational
Metrics
Raw Data Topics
JSON, AVRO
Processed
Data Topics
Sensors
and
/ or
Transactiona
l/ Context
Data
OLTP/ODS
ETL
Or
EL with
T in Spark
Batch
Low
Latency
Applications
Files
In-
database
analytics
or
Stream
Processing
Data
Warehouse
Data Lake

39. Summary
• Get all enterprise data under management
• RDBMS (/columnar), Cloud Storage/Parquet, Graph cover most analytic
platform needs
• Cost of ownership is more than the cloud costs
• Data Integration is vital to Data Architecture for Modern Analytics
• The Data Mesh and Data Fabric are decentralizing the architecture
24

40. Upcoming Topics
• Is Our Information Management Mature?
• The Future based on AI & Analytics
• Organizational Change Management for Data & Analytics
Driven Projects
• Graph Database Use Cases
• Assessing New Databases: Translytical Use Cases
25
Second Thursday of Every Month, at 2:00 ET

41. Data Architecture
Best Practices for
Advanced Analytics
Presented by: William McKnight
“#1 Global Influencer in Big Data” Thinkers360
President, McKnight Consulting Group
A 2 time Inc. 5000 Company
@williammcknight
www.mcknightcg.com
(214) 514-1444
Second Thursday of Every Month, at 2:00 ET
#AdvAnalytics