As cyber threats continuously grow in sophistication and frequency, companies need to quickly acclimate to effectively detect, respond, and protect their environments. At Intel, we’ve addressed this need by implementing a modern, scalable Cyber Intelligence Platform (CIP) based on Splunk and Apache Kafka. We believe that CIP positions us for the best defense against cyber threats well into the future.
Our CIP ingests tens of terabytes of data each day and transforms it into actionable insights through streams processing, context-smart applications, and advanced analytics techniques. Kafka serves as a massive data pipeline within the platform. It provides us the ability to operate on data in-stream, enabling us to reduce Mean Time to Detect (MTTD) and Mean Time to Respond (MTTR). Faster detection and response ultimately leads to better prevention.
In our session, we’ll discuss the details described in the IT@Intel white paper that was published in Nov 2020 with same title.
3. IT@Intel 3
Jac Noel has over 25 years of Information Technology and
Cyber Security experience across the military, government,
and corporate environments.
He started his technical career in the United States Air Force
supporting defense intelligence systems for the AF mission in
EMEA. He has spent the past 20 years serving in various
technical roles in Intel’s IT organization. He’s currently serving
as a Security Solutions Architect focusing on security
intelligence and response capabilities. He’s the lead architect
for Intel’s Cyber Intelligence Platform (CIP), which is a next-
gen architecture combining a data lake, message bus, stream
processing, machine-learning, orchestration, and workflow
automation into a single platform.
Jac holds a Bachelor of Science degree from Chico State
University and has earned numerous professional certifications
over the years, including CISSP, GCFW, CCNA, and MCSE.
He’s also a proud inventor, patent holder, and author of several
white papers.
Jac Noel
Security Solutions Architect
4. IT@Intel 4
Intel Information Security’s Mission
4
Our mission is to keep Intel
legal and secure.
This mission is never
“done.”
Best ways to measure our success:
Reduce Mean Time to Detect (MTTD)
and Mean Time to Respond (MTTR)
Identify and implement more effective
preventative controls
Improve our agility to respond to new and
changing threats and regulations
5. IT@Intel 5
API Data Virtualization Layer
Information Security
Business Role
Incident Response
Vulnerability
Management
Compliance
Enforcement
Data Protection
Threat Intelligence
Common Work
Surface Layer
Query
Search
Reporting
Dashboards
Visualizations
Analytics Workbench
Workflow Automation
Infrastructure
Clients
Servers
Network
Infrastructure
Other Data
Sources
Data
Blueprint
Security
Data Lake
Control Layer
Security Event Management
User Event Behavior Analytics
Vulnerability Scanning
Threat Intelligence
Advanced Analytics
Deceptions
Intrusion Detection
Firewalls
Intrusion Prevention
Endpoint Detection and Response
Data Loss Prevention
Intrusion Scanning
Connectors
Enterprise Security Message Bus
Topics, Publish/Subscribe, Transform, Enrich, Filter, Join
CyberIntelligencePlatform-ReferenceArchitecture
A platform that supports our entire InfoSec organization
5
6. IT@Intel 6
High Performance Compute & Storage
BU
Partners
IT
Ops
Partners
Confluent Platform
Message Bus
Stream Processing
Cyber Intelligence Platform - Solution Stack
Our partners produce and consume data, too!
6
8. IT@Intel 8
The Power of the Kafka Bus
No Message Bus
Point to point, complex
Slow to implement
Increased technical debt due to tightly-coupled solutions and brittle integrations
No orchestration (custom-code it, multiple times)
No transformation (custom-code it, multiple times)
Slow to move data between multiple capabilities
Harder to monitor and govern
With Message Bus
Data Transformation (enrich, aggregate, normalize)
Near real-time integration (streaming)
Resilient, robust, scalable, available
Orchestrate multiple activities in one place
Cross-capability consumption
Platform independent, plug and play
Apps loosely coupled but tightly integrated
Common architectural element for large enterprises
App App App App App App
App App App App App App
App App App App App App
App App App App App App
Message Bus
Abstraction, Resiliency, Scalability, Availability
Transform Orchestrate
9. IT@Intel 9
Improving Data Availability with Confluent MRC
9
Single Cluster
Data Center 3
Producers Consumers
Streaming Apps
Consumers Producers
Data Center 1
Leaders (ISR)
Zookeeper 1
Zookeeper 2
Broker n
Broker 2
Broker 1
Broker 3
…
Mirroring
Data Center 2
Observers
Zookeeper 3
Zookeeper 4
Broker n
Broker 2
Broker 1
Broker 3
…
Zookeeper 5
10. IT@Intel 10
Asynchronous Replication for Faster Recovery
10
Single Cluster
Data Center 3
Producers Consumers
Streaming Apps
Consumers Producers
Data Center 1
Zookeeper 1
Zookeeper 2
Broker n
Broker 2
Broker 1
Broker 3
…
Mirroring
Data Center 2
Leaders (ISR)
Zookeeper 3
Zookeeper 4
Broker n
Broker 2
Broker 1
Broker 3
…
Zookeeper 5
Confluent Platform with Multi Region Clusters
12. IT@Intel 12
Monitoring Our Kafka Clusters
12
Our C3 server requires Intel 2nd gen Xeon processors for high-performance compute
and Intel Optane DC SSDs for low latency and high-endurance storage.
Kafka
Admins
All-in-One Kafka Cluster
Confluent Control Center Server (C3)
(Broker, ZooKeeper, Connect, Kafka Streams)
Kafka Streams App
“Stream Processor”
C3 Web App
Consumers UI
Trouble-
shooting
Producers
Producers
Kafka
Production
Monitoring Data
Metrics Data
Metrics
Reporter
Monitoring
Interceptor
Topics
Topics
Topics
Topics
Topics
Topics
Topics
Topics
Topics
Topics
Topics
Topics
Topics
Topics
Topics
Topics
Topics
Consumers
Metrics Topic
Monitoring Topic Transformed Topics
Health
Monitoring
13. IT@Intel 13
Managing Vulnerabilities with Stream Processing
13
Confluent Platform
Producers Kafka Streams API
Stream Processing
Kafka Bus
Vulnerability
Topic Filter
Vulnerabilities by
Business Unit
IP Address
Range Topic
Join Asset
Asset Inventory
Topic
Ownership with Consumers
Vulnerable Assets
BU #1’s
Vulnerabilities Topic
Data Lake
BU Partners
BU #2’s
Vulnerabilities Topic
IT Partners
BU #3’s
Vulnerabilities Topic
SIEM
Vulnerabilities
with Owners Topic Enforcement
SOAR
Scanning
Engine
IP Address
Management
Asset Management
Inventory
Vulnerabilities
Asset configuration, CVEs, CVSS
IP Address Ranges
Ownership, Business Units
Asset Ownership
14. IT@Intel 14
Kafka Maturity
Timeline
14
Acquire once-consume many
Integration efficiency
Remove the noise, and
duplication
Cost savings for downstream consumers
Join multiple sources
Contextually rich + clean data downstream
ACQUIRE
DATA
FILTERING
ENRICHMENT
SUMMARIZATION
ADVANCED Autonomous Actions
e.g. Cluster analysis, ML
Produce summary statistics
State information, performance benefit
and downstream cost savings
15. IT@Intel 15
Kafka By The Numbers
15
20+
TB/DAY
135+
32+
CONSUMERS DATA
SOURCES
320+
TOPICS
90+
PRODUCERS
>18B
EVENTS/DAY
Kafka
by the
Numbers
~8 trillion events indexed by Splunk in 2020
16. IT@Intel 16
Kafka - Benefits to Intel
16
KAFKA LEADERSHIP
THROUGH CONFLUENT
EXPERTISE
GENERATES
CONTEXTUALLY RICH
DATA
MODERN
ARCHITECTURE WITH
THRIVING COMMUNITY
GLOBAL
SCALE AND REACH
OPERATE ON DATA
IN STREAM
ECONOMIES
OF SCALE
REDUCE TECHNICAL
DEBT AND
DOWNSTREAM COSTS
ALWAYS
ON
17. IT@Intel 17
People + Technology + Data
Transforming How Information Security Works
17
Reduced Risk
to Intel
Greater Insight
and Tighter
Collaboration
Highly
Integrated
and
Automated
A Force
Multiplier
Faster
Detection and
Response
Speaking a
Common
Language
A Platform
for the Future
People + Technology + DataTransforming How Information Security Works
Abstraction Layer
Economies of Scale via acquire data once consume many
Operate on Data In Stream – near real time identification and response to threats
Reduce downstream costs, e.g. filtering data and transforming data (contextually rich) in kafka before applications and data lakes like Splunk, consumes
Reduce technical Debt by eliminating custom connectors
Generates Contextually rich data
Global Scale and Reach – distributed bus technology that connects to cloud, IOT , other buses, kafka in backpack because records even when elements of assets are offline/separate
Always On – no downtime, producers and consumers do not impact each other, kafka in backpack because it brings the data back online
Modern Architecture with Thriving Community – great minds working across many distributed systems, data types, message bus systems, new APIs, always innovating
Kafka leadership Through Confluent expertise – Confluent is technology leader and partnering with Intel to innovate