SlideShare a Scribd company logo
1 of 15
Capturing Network Traffic into
Database
Key Words: Sniffer, Network Analyzer,
Wireshark, MySQL, Database, PCAP
to MySQL
How to Store Packets into Database
(for example, MySQL)
• Having packets in database can be
very convenient:
– More performance
– Parallel writing
– Quick analysis
– Data Mining (if you want)
– Long time storage
How???

PCAP
(or real-time
capturing)

XML output

MySQL
What do we need?
• tshark (supplied with WireShark)
• PHP with XML, Xpath support
• MySQL database
My workstation
• All examples here are done in Windows
7, but if you want, you will not need a
lot of time to make them for Linux
• Our task: capture TCP packets (IP from,
IP to, port from, port to, length,
sequence) into database. Example can
be any, for instance, checking for
network scanning.
Distributed
• This can be distributed, no problem,
but you need to use extra network or
filters (otherwise, you will hang your
system: 1 sniffed packet sent make 1
more, and so on).
Distributed
Remote SQL Server

WorkStation

WorkStation
WorkStation
Getting traffic XML format
• tshark -r "D:test.pcap" -T pdml > D:test_T.xml
– Converting pcap into XML

Or
• tshark -T pdml | you_application.exe
– Real-Time
Output XML example
Warning
• Such converting to XML consume a lot of
space (50x)! PCAP file from 200 Kb grew into
10 Mb XML!!!
• In this case you might find useful to divide one
big pcap file into several of smaller size
• Also filtering is good idea, so you can throw
out fields useless for you.
XML output file structure
• It is very simple (I crossed out trivial parts, so real lines are
bigger):
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="pdml2html.xsl"?>
<pdml >
<packet>
Packet info
</packet>
<packet>
Packet info
</packet>
And so on.
XML structure – packets (Example,
DNS query)
<packet>
<proto name="geninfo" pos="0" showname="General information" size="73">
</proto>
<proto name="frame" showname="Frame 1: 73 bytes on wire (584 bits), 73 bytes
captured (584 bits)" size="73" pos="0">
</proto>
<proto name="eth">
</proto>
<proto name="ip" >
</proto>
<proto name="udp" showname="User Datagram Protocol, Src Port: 58150 (58150),
Dst Port: domain (53)" size="8" pos="34">
</proto>
<proto name="dns" showname="Domain Name System (query)" size="31" pos="42">
</proto>
</packet>
(Child elements and attributes of proto are not shown here)
XML to MySQL
• You can use LOTS of options: C++/Java,etc.
• I used SimpleXML and XPath with PHP:
$file = "test_T.xml";
$my_file = simplexml_load_file($file );
foreach ($my_file >xpath('//packet') as $packet)
{
$packet_type = $packet->proto[4];
echo $packet_type['name']; //protocol
}
And putting into database
function LoadToDataBase($con)
{
$stmt =$con->prepare("INSERT INTO tcp (capture_order, from_ip,
to_ip, from_port, to_port, tcp_length, tcp_stream, tcp_stream_text,
tcp_sequence_dec) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)");

$stmt->bind_param('sssiiiisi', $this->capture_order,$this->from_ip,
$this->to_ip,
$this->from_port,$this->to_port,
$this->tcp_length,$this->tcp_stream, $this->tcp_stream_text,
$this->tcp_sequence_dec);
$stmt->execute();
}
Here $con is open connection to mysql, and all this vars I got in cycle.
Please, refer to full code.
Thank you
• I hope you find this useful.

• Full code is available at
http://tigrantsat.me/randd/pcaptomysql/

More Related Content

What's hot

Semantic web meetup – sparql tutorial
Semantic web meetup – sparql tutorialSemantic web meetup – sparql tutorial
Semantic web meetup – sparql tutorialAdonisDamian
 
An overview of reference architectures for Postgres
An overview of reference architectures for PostgresAn overview of reference architectures for Postgres
An overview of reference architectures for PostgresEDB
 
Care and Feeding of Catalyst Optimizer
Care and Feeding of Catalyst OptimizerCare and Feeding of Catalyst Optimizer
Care and Feeding of Catalyst OptimizerDatabricks
 
Svetlin Nakov - Database Transactions
Svetlin Nakov - Database TransactionsSvetlin Nakov - Database Transactions
Svetlin Nakov - Database TransactionsSvetlin Nakov
 
Vxlan control plane and routing
Vxlan control plane and routingVxlan control plane and routing
Vxlan control plane and routingWilfredzeng
 
Shipping Data from Postgres to Clickhouse, by Murat Kabilov, Adjust
Shipping Data from Postgres to Clickhouse, by Murat Kabilov, AdjustShipping Data from Postgres to Clickhouse, by Murat Kabilov, Adjust
Shipping Data from Postgres to Clickhouse, by Murat Kabilov, AdjustAltinity Ltd
 
C* Summit 2013: The World's Next Top Data Model by Patrick McFadin
C* Summit 2013: The World's Next Top Data Model by Patrick McFadinC* Summit 2013: The World's Next Top Data Model by Patrick McFadin
C* Summit 2013: The World's Next Top Data Model by Patrick McFadinDataStax Academy
 
Cics program, interval and task control commands-session 5
Cics program, interval and task control commands-session 5Cics program, interval and task control commands-session 5
Cics program, interval and task control commands-session 5Srinimf-Slides
 
Beyond SQL: Speeding up Spark with DataFrames
Beyond SQL: Speeding up Spark with DataFramesBeyond SQL: Speeding up Spark with DataFrames
Beyond SQL: Speeding up Spark with DataFramesDatabricks
 
Building Modern Data Streaming Apps with Python
Building Modern Data Streaming Apps with PythonBuilding Modern Data Streaming Apps with Python
Building Modern Data Streaming Apps with PythonTimothy Spann
 
An Insider’s Guide to Maximizing Spark SQL Performance
 An Insider’s Guide to Maximizing Spark SQL Performance An Insider’s Guide to Maximizing Spark SQL Performance
An Insider’s Guide to Maximizing Spark SQL PerformanceTakuya UESHIN
 
Kafka replication apachecon_2013
Kafka replication apachecon_2013Kafka replication apachecon_2013
Kafka replication apachecon_2013Jun Rao
 
InnoDB Internal
InnoDB InternalInnoDB Internal
InnoDB Internalmysqlops
 
IBM i client partitions concepts and implementation
IBM i client partitions concepts and implementationIBM i client partitions concepts and implementation
IBM i client partitions concepts and implementationCOMMON Europe
 
Websphere MQ (MQSeries) fundamentals
Websphere MQ (MQSeries) fundamentalsWebsphere MQ (MQSeries) fundamentals
Websphere MQ (MQSeries) fundamentalsBiju Nair
 
InnoDB Flushing and Checkpoints
InnoDB Flushing and CheckpointsInnoDB Flushing and Checkpoints
InnoDB Flushing and CheckpointsMIJIN AN
 
Everyday I'm Shuffling - Tips for Writing Better Spark Programs, Strata San J...
Everyday I'm Shuffling - Tips for Writing Better Spark Programs, Strata San J...Everyday I'm Shuffling - Tips for Writing Better Spark Programs, Strata San J...
Everyday I'm Shuffling - Tips for Writing Better Spark Programs, Strata San J...Databricks
 

What's hot (20)

Semantic web meetup – sparql tutorial
Semantic web meetup – sparql tutorialSemantic web meetup – sparql tutorial
Semantic web meetup – sparql tutorial
 
An overview of reference architectures for Postgres
An overview of reference architectures for PostgresAn overview of reference architectures for Postgres
An overview of reference architectures for Postgres
 
Care and Feeding of Catalyst Optimizer
Care and Feeding of Catalyst OptimizerCare and Feeding of Catalyst Optimizer
Care and Feeding of Catalyst Optimizer
 
Svetlin Nakov - Database Transactions
Svetlin Nakov - Database TransactionsSvetlin Nakov - Database Transactions
Svetlin Nakov - Database Transactions
 
Vxlan control plane and routing
Vxlan control plane and routingVxlan control plane and routing
Vxlan control plane and routing
 
Shipping Data from Postgres to Clickhouse, by Murat Kabilov, Adjust
Shipping Data from Postgres to Clickhouse, by Murat Kabilov, AdjustShipping Data from Postgres to Clickhouse, by Murat Kabilov, Adjust
Shipping Data from Postgres to Clickhouse, by Murat Kabilov, Adjust
 
C* Summit 2013: The World's Next Top Data Model by Patrick McFadin
C* Summit 2013: The World's Next Top Data Model by Patrick McFadinC* Summit 2013: The World's Next Top Data Model by Patrick McFadin
C* Summit 2013: The World's Next Top Data Model by Patrick McFadin
 
Cics program, interval and task control commands-session 5
Cics program, interval and task control commands-session 5Cics program, interval and task control commands-session 5
Cics program, interval and task control commands-session 5
 
RDF and OWL
RDF and OWLRDF and OWL
RDF and OWL
 
Beyond SQL: Speeding up Spark with DataFrames
Beyond SQL: Speeding up Spark with DataFramesBeyond SQL: Speeding up Spark with DataFrames
Beyond SQL: Speeding up Spark with DataFrames
 
Building Modern Data Streaming Apps with Python
Building Modern Data Streaming Apps with PythonBuilding Modern Data Streaming Apps with Python
Building Modern Data Streaming Apps with Python
 
SPARQL Cheat Sheet
SPARQL Cheat SheetSPARQL Cheat Sheet
SPARQL Cheat Sheet
 
An Insider’s Guide to Maximizing Spark SQL Performance
 An Insider’s Guide to Maximizing Spark SQL Performance An Insider’s Guide to Maximizing Spark SQL Performance
An Insider’s Guide to Maximizing Spark SQL Performance
 
Kafka replication apachecon_2013
Kafka replication apachecon_2013Kafka replication apachecon_2013
Kafka replication apachecon_2013
 
InnoDB Internal
InnoDB InternalInnoDB Internal
InnoDB Internal
 
IBM i client partitions concepts and implementation
IBM i client partitions concepts and implementationIBM i client partitions concepts and implementation
IBM i client partitions concepts and implementation
 
Websphere MQ (MQSeries) fundamentals
Websphere MQ (MQSeries) fundamentalsWebsphere MQ (MQSeries) fundamentals
Websphere MQ (MQSeries) fundamentals
 
InnoDB Flushing and Checkpoints
InnoDB Flushing and CheckpointsInnoDB Flushing and Checkpoints
InnoDB Flushing and Checkpoints
 
VXLAN Practice Guide
VXLAN Practice GuideVXLAN Practice Guide
VXLAN Practice Guide
 
Everyday I'm Shuffling - Tips for Writing Better Spark Programs, Strata San J...
Everyday I'm Shuffling - Tips for Writing Better Spark Programs, Strata San J...Everyday I'm Shuffling - Tips for Writing Better Spark Programs, Strata San J...
Everyday I'm Shuffling - Tips for Writing Better Spark Programs, Strata San J...
 

Viewers also liked

Ranking Merco 2014: EAE Business School, segunda escuela de negocios más repu...
Ranking Merco 2014: EAE Business School, segunda escuela de negocios más repu...Ranking Merco 2014: EAE Business School, segunda escuela de negocios más repu...
Ranking Merco 2014: EAE Business School, segunda escuela de negocios más repu...EAE Business School
 
Open source network forensics and advanced pcap analysis
Open source network forensics and advanced pcap analysisOpen source network forensics and advanced pcap analysis
Open source network forensics and advanced pcap analysisGTKlondike
 
MySQL/MariaDB query optimizer tuning tutorial from Percona Live 2013
MySQL/MariaDB query optimizer tuning tutorial from Percona Live 2013MySQL/MariaDB query optimizer tuning tutorial from Percona Live 2013
MySQL/MariaDB query optimizer tuning tutorial from Percona Live 2013Sergey Petrunya
 
CapAnalysis - Deep Packet Inspection
CapAnalysis - Deep Packet InspectionCapAnalysis - Deep Packet Inspection
CapAnalysis - Deep Packet InspectionChris Harrington
 
Wireshar training
Wireshar trainingWireshar training
Wireshar trainingLuke Luo
 
Authentication, authorization, accounting(aaa) slides
Authentication, authorization, accounting(aaa) slidesAuthentication, authorization, accounting(aaa) slides
Authentication, authorization, accounting(aaa) slidesrahul kundu
 
Radius server,PAP and CHAP Protocols
Radius server,PAP and CHAP ProtocolsRadius server,PAP and CHAP Protocols
Radius server,PAP and CHAP ProtocolsDhananjay Aloorkar
 
Implementing Cisco AAA
Implementing Cisco AAAImplementing Cisco AAA
Implementing Cisco AAAdkaya
 
What is PCRF? – Detailed PCRF architecture and functioning
What is PCRF? – Detailed PCRF architecture and functioningWhat is PCRF? – Detailed PCRF architecture and functioning
What is PCRF? – Detailed PCRF architecture and functioningMahindra Comviva
 
Wireshark
WiresharkWireshark
Wiresharkbtohara
 
Radius Protocol
Radius ProtocolRadius Protocol
Radius ProtocolNetwax Lab
 
AAA & RADIUS Protocols
AAA & RADIUS ProtocolsAAA & RADIUS Protocols
AAA & RADIUS ProtocolsPeter R. Egli
 

Viewers also liked (20)

Traffic types in internet
Traffic types in internetTraffic types in internet
Traffic types in internet
 
Ranking Merco 2014: EAE Business School, segunda escuela de negocios más repu...
Ranking Merco 2014: EAE Business School, segunda escuela de negocios más repu...Ranking Merco 2014: EAE Business School, segunda escuela de negocios más repu...
Ranking Merco 2014: EAE Business School, segunda escuela de negocios más repu...
 
Open source network forensics and advanced pcap analysis
Open source network forensics and advanced pcap analysisOpen source network forensics and advanced pcap analysis
Open source network forensics and advanced pcap analysis
 
MySQL/MariaDB query optimizer tuning tutorial from Percona Live 2013
MySQL/MariaDB query optimizer tuning tutorial from Percona Live 2013MySQL/MariaDB query optimizer tuning tutorial from Percona Live 2013
MySQL/MariaDB query optimizer tuning tutorial from Percona Live 2013
 
CapAnalysis - Deep Packet Inspection
CapAnalysis - Deep Packet InspectionCapAnalysis - Deep Packet Inspection
CapAnalysis - Deep Packet Inspection
 
AfterGlow
AfterGlowAfterGlow
AfterGlow
 
Wireshar training
Wireshar trainingWireshar training
Wireshar training
 
AAA in a nutshell
AAA in a nutshellAAA in a nutshell
AAA in a nutshell
 
Authentication, authorization, accounting(aaa) slides
Authentication, authorization, accounting(aaa) slidesAuthentication, authorization, accounting(aaa) slides
Authentication, authorization, accounting(aaa) slides
 
Radius server,PAP and CHAP Protocols
Radius server,PAP and CHAP ProtocolsRadius server,PAP and CHAP Protocols
Radius server,PAP and CHAP Protocols
 
RADIUS
RADIUSRADIUS
RADIUS
 
Quality of Service
Quality of ServiceQuality of Service
Quality of Service
 
Implementing Cisco AAA
Implementing Cisco AAAImplementing Cisco AAA
Implementing Cisco AAA
 
What is PCRF? – Detailed PCRF architecture and functioning
What is PCRF? – Detailed PCRF architecture and functioningWhat is PCRF? – Detailed PCRF architecture and functioning
What is PCRF? – Detailed PCRF architecture and functioning
 
AAA Protocol
AAA ProtocolAAA Protocol
AAA Protocol
 
Wireshark
WiresharkWireshark
Wireshark
 
Introduction to Diameter Protocol - Part1
Introduction to Diameter Protocol - Part1Introduction to Diameter Protocol - Part1
Introduction to Diameter Protocol - Part1
 
Radius Protocol
Radius ProtocolRadius Protocol
Radius Protocol
 
Wireshark
WiresharkWireshark
Wireshark
 
AAA & RADIUS Protocols
AAA & RADIUS ProtocolsAAA & RADIUS Protocols
AAA & RADIUS Protocols
 

Similar to Capturing Network Traffic into Database

Project Tungsten Phase II: Joining a Billion Rows per Second on a Laptop
Project Tungsten Phase II: Joining a Billion Rows per Second on a LaptopProject Tungsten Phase II: Joining a Billion Rows per Second on a Laptop
Project Tungsten Phase II: Joining a Billion Rows per Second on a LaptopDatabricks
 
High Performance, Scalable MongoDB in a Bare Metal Cloud
High Performance, Scalable MongoDB in a Bare Metal CloudHigh Performance, Scalable MongoDB in a Bare Metal Cloud
High Performance, Scalable MongoDB in a Bare Metal CloudMongoDB
 
User-space Network Processing
User-space Network ProcessingUser-space Network Processing
User-space Network ProcessingRyousei Takano
 
High performace network of Cloud Native Taiwan User Group
High performace network of Cloud Native Taiwan User GroupHigh performace network of Cloud Native Taiwan User Group
High performace network of Cloud Native Taiwan User GroupHungWei Chiu
 
Attack monitoring using ElasticSearch Logstash and Kibana
Attack monitoring using ElasticSearch Logstash and KibanaAttack monitoring using ElasticSearch Logstash and Kibana
Attack monitoring using ElasticSearch Logstash and KibanaPrajal Kulkarni
 
Ga4 gh meeting at the the sanger institute
Ga4 gh meeting at the the sanger instituteGa4 gh meeting at the the sanger institute
Ga4 gh meeting at the the sanger instituteMatt Massie
 
Explore big data at speed of thought with Spark 2.0 and Snappydata
Explore big data at speed of thought with Spark 2.0 and SnappydataExplore big data at speed of thought with Spark 2.0 and Snappydata
Explore big data at speed of thought with Spark 2.0 and SnappydataData Con LA
 
Project Tungsten: Bringing Spark Closer to Bare Metal
Project Tungsten: Bringing Spark Closer to Bare MetalProject Tungsten: Bringing Spark Closer to Bare Metal
Project Tungsten: Bringing Spark Closer to Bare MetalDatabricks
 
Event Processing and Integration with IAS Data Processors
Event Processing and Integration with IAS Data ProcessorsEvent Processing and Integration with IAS Data Processors
Event Processing and Integration with IAS Data ProcessorsInvenire Aude
 
End to End Processing of 3.7 Million Telemetry Events per Second using Lambda...
End to End Processing of 3.7 Million Telemetry Events per Second using Lambda...End to End Processing of 3.7 Million Telemetry Events per Second using Lambda...
End to End Processing of 3.7 Million Telemetry Events per Second using Lambda...DataWorks Summit/Hadoop Summit
 
osdi20-slides_zhao.pptx
osdi20-slides_zhao.pptxosdi20-slides_zhao.pptx
osdi20-slides_zhao.pptxCive1971
 
Hardware Provisioning
Hardware ProvisioningHardware Provisioning
Hardware ProvisioningMongoDB
 
KSQL Performance Tuning for Fun and Profit ( Nick Dearden, Confluent) Kafka S...
KSQL Performance Tuning for Fun and Profit ( Nick Dearden, Confluent) Kafka S...KSQL Performance Tuning for Fun and Profit ( Nick Dearden, Confluent) Kafka S...
KSQL Performance Tuning for Fun and Profit ( Nick Dearden, Confluent) Kafka S...confluent
 
DPDK Summit 2015 - Aspera - Charles Shiflett
DPDK Summit 2015 - Aspera - Charles ShiflettDPDK Summit 2015 - Aspera - Charles Shiflett
DPDK Summit 2015 - Aspera - Charles ShiflettJim St. Leger
 
Practical 7 - Using Wireshark Tutorial and Hands-on
Practical 7 - Using Wireshark Tutorial and Hands-onPractical 7 - Using Wireshark Tutorial and Hands-on
Practical 7 - Using Wireshark Tutorial and Hands-onQaisSaifQassim
 
seed block algorithm
seed block algorithmseed block algorithm
seed block algorithmDipak Badhe
 
Multitenancy: Kafka clusters for everyone at LINE
Multitenancy: Kafka clusters for everyone at LINEMultitenancy: Kafka clusters for everyone at LINE
Multitenancy: Kafka clusters for everyone at LINEkawamuray
 
Python (Jinja2) Templates for Network Automation
Python (Jinja2) Templates for Network AutomationPython (Jinja2) Templates for Network Automation
Python (Jinja2) Templates for Network AutomationRick Sherman
 
Big data analytics with Spark & Cassandra
Big data analytics with Spark & Cassandra Big data analytics with Spark & Cassandra
Big data analytics with Spark & Cassandra Matthias Niehoff
 

Similar to Capturing Network Traffic into Database (20)

Project Tungsten Phase II: Joining a Billion Rows per Second on a Laptop
Project Tungsten Phase II: Joining a Billion Rows per Second on a LaptopProject Tungsten Phase II: Joining a Billion Rows per Second on a Laptop
Project Tungsten Phase II: Joining a Billion Rows per Second on a Laptop
 
High Performance, Scalable MongoDB in a Bare Metal Cloud
High Performance, Scalable MongoDB in a Bare Metal CloudHigh Performance, Scalable MongoDB in a Bare Metal Cloud
High Performance, Scalable MongoDB in a Bare Metal Cloud
 
User-space Network Processing
User-space Network ProcessingUser-space Network Processing
User-space Network Processing
 
High performace network of Cloud Native Taiwan User Group
High performace network of Cloud Native Taiwan User GroupHigh performace network of Cloud Native Taiwan User Group
High performace network of Cloud Native Taiwan User Group
 
Jumbo Mumbo in OpenStack
Jumbo Mumbo in OpenStackJumbo Mumbo in OpenStack
Jumbo Mumbo in OpenStack
 
Attack monitoring using ElasticSearch Logstash and Kibana
Attack monitoring using ElasticSearch Logstash and KibanaAttack monitoring using ElasticSearch Logstash and Kibana
Attack monitoring using ElasticSearch Logstash and Kibana
 
Ga4 gh meeting at the the sanger institute
Ga4 gh meeting at the the sanger instituteGa4 gh meeting at the the sanger institute
Ga4 gh meeting at the the sanger institute
 
Explore big data at speed of thought with Spark 2.0 and Snappydata
Explore big data at speed of thought with Spark 2.0 and SnappydataExplore big data at speed of thought with Spark 2.0 and Snappydata
Explore big data at speed of thought with Spark 2.0 and Snappydata
 
Project Tungsten: Bringing Spark Closer to Bare Metal
Project Tungsten: Bringing Spark Closer to Bare MetalProject Tungsten: Bringing Spark Closer to Bare Metal
Project Tungsten: Bringing Spark Closer to Bare Metal
 
Event Processing and Integration with IAS Data Processors
Event Processing and Integration with IAS Data ProcessorsEvent Processing and Integration with IAS Data Processors
Event Processing and Integration with IAS Data Processors
 
End to End Processing of 3.7 Million Telemetry Events per Second using Lambda...
End to End Processing of 3.7 Million Telemetry Events per Second using Lambda...End to End Processing of 3.7 Million Telemetry Events per Second using Lambda...
End to End Processing of 3.7 Million Telemetry Events per Second using Lambda...
 
osdi20-slides_zhao.pptx
osdi20-slides_zhao.pptxosdi20-slides_zhao.pptx
osdi20-slides_zhao.pptx
 
Hardware Provisioning
Hardware ProvisioningHardware Provisioning
Hardware Provisioning
 
KSQL Performance Tuning for Fun and Profit ( Nick Dearden, Confluent) Kafka S...
KSQL Performance Tuning for Fun and Profit ( Nick Dearden, Confluent) Kafka S...KSQL Performance Tuning for Fun and Profit ( Nick Dearden, Confluent) Kafka S...
KSQL Performance Tuning for Fun and Profit ( Nick Dearden, Confluent) Kafka S...
 
DPDK Summit 2015 - Aspera - Charles Shiflett
DPDK Summit 2015 - Aspera - Charles ShiflettDPDK Summit 2015 - Aspera - Charles Shiflett
DPDK Summit 2015 - Aspera - Charles Shiflett
 
Practical 7 - Using Wireshark Tutorial and Hands-on
Practical 7 - Using Wireshark Tutorial and Hands-onPractical 7 - Using Wireshark Tutorial and Hands-on
Practical 7 - Using Wireshark Tutorial and Hands-on
 
seed block algorithm
seed block algorithmseed block algorithm
seed block algorithm
 
Multitenancy: Kafka clusters for everyone at LINE
Multitenancy: Kafka clusters for everyone at LINEMultitenancy: Kafka clusters for everyone at LINE
Multitenancy: Kafka clusters for everyone at LINE
 
Python (Jinja2) Templates for Network Automation
Python (Jinja2) Templates for Network AutomationPython (Jinja2) Templates for Network Automation
Python (Jinja2) Templates for Network Automation
 
Big data analytics with Spark & Cassandra
Big data analytics with Spark & Cassandra Big data analytics with Spark & Cassandra
Big data analytics with Spark & Cassandra
 

Recently uploaded

Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024Scott Keck-Warren
 
Artificial intelligence in cctv survelliance.pptx
Artificial intelligence in cctv survelliance.pptxArtificial intelligence in cctv survelliance.pptx
Artificial intelligence in cctv survelliance.pptxhariprasad279825
 
Commit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easyCommit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easyAlfredo García Lavilla
 
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationBeyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationSafe Software
 
Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024Enterprise Knowledge
 
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfAddepto
 
My Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 PresentationMy Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 PresentationRidwan Fadjar
 
What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024Stephanie Beckett
 
Streamlining Python Development: A Guide to a Modern Project Setup
Streamlining Python Development: A Guide to a Modern Project SetupStreamlining Python Development: A Guide to a Modern Project Setup
Streamlining Python Development: A Guide to a Modern Project SetupFlorian Wilhelm
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr BaganFwdays
 
Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfUnraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfAlex Barbosa Coqueiro
 
Unleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding ClubUnleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding ClubKalema Edgar
 
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brandgvaughan
 
SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024Lorenzo Miniero
 
Vector Databases 101 - An introduction to the world of Vector Databases
Vector Databases 101 - An introduction to the world of Vector DatabasesVector Databases 101 - An introduction to the world of Vector Databases
Vector Databases 101 - An introduction to the world of Vector DatabasesZilliz
 
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmaticsKotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmaticscarlostorres15106
 
Story boards and shot lists for my a level piece
Story boards and shot lists for my a level pieceStory boards and shot lists for my a level piece
Story boards and shot lists for my a level piececharlottematthew16
 
CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):comworks
 
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek SchlawackFwdays
 

Recently uploaded (20)

Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024
 
Artificial intelligence in cctv survelliance.pptx
Artificial intelligence in cctv survelliance.pptxArtificial intelligence in cctv survelliance.pptx
Artificial intelligence in cctv survelliance.pptx
 
Commit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easyCommit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easy
 
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationBeyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
 
Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024
 
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdf
 
DMCC Future of Trade Web3 - Special Edition
DMCC Future of Trade Web3 - Special EditionDMCC Future of Trade Web3 - Special Edition
DMCC Future of Trade Web3 - Special Edition
 
My Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 PresentationMy Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 Presentation
 
What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024
 
Streamlining Python Development: A Guide to a Modern Project Setup
Streamlining Python Development: A Guide to a Modern Project SetupStreamlining Python Development: A Guide to a Modern Project Setup
Streamlining Python Development: A Guide to a Modern Project Setup
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan
 
Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfUnraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdf
 
Unleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding ClubUnleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding Club
 
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brand
 
SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024
 
Vector Databases 101 - An introduction to the world of Vector Databases
Vector Databases 101 - An introduction to the world of Vector DatabasesVector Databases 101 - An introduction to the world of Vector Databases
Vector Databases 101 - An introduction to the world of Vector Databases
 
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmaticsKotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
 
Story boards and shot lists for my a level piece
Story boards and shot lists for my a level pieceStory boards and shot lists for my a level piece
Story boards and shot lists for my a level piece
 
CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):
 
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
 

Capturing Network Traffic into Database

  • 1. Capturing Network Traffic into Database Key Words: Sniffer, Network Analyzer, Wireshark, MySQL, Database, PCAP to MySQL
  • 2. How to Store Packets into Database (for example, MySQL) • Having packets in database can be very convenient: – More performance – Parallel writing – Quick analysis – Data Mining (if you want) – Long time storage
  • 4. What do we need? • tshark (supplied with WireShark) • PHP with XML, Xpath support • MySQL database
  • 5. My workstation • All examples here are done in Windows 7, but if you want, you will not need a lot of time to make them for Linux • Our task: capture TCP packets (IP from, IP to, port from, port to, length, sequence) into database. Example can be any, for instance, checking for network scanning.
  • 6. Distributed • This can be distributed, no problem, but you need to use extra network or filters (otherwise, you will hang your system: 1 sniffed packet sent make 1 more, and so on).
  • 8. Getting traffic XML format • tshark -r "D:test.pcap" -T pdml > D:test_T.xml – Converting pcap into XML Or • tshark -T pdml | you_application.exe – Real-Time
  • 10. Warning • Such converting to XML consume a lot of space (50x)! PCAP file from 200 Kb grew into 10 Mb XML!!! • In this case you might find useful to divide one big pcap file into several of smaller size • Also filtering is good idea, so you can throw out fields useless for you.
  • 11. XML output file structure • It is very simple (I crossed out trivial parts, so real lines are bigger): <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="pdml2html.xsl"?> <pdml > <packet> Packet info </packet> <packet> Packet info </packet> And so on.
  • 12. XML structure – packets (Example, DNS query) <packet> <proto name="geninfo" pos="0" showname="General information" size="73"> </proto> <proto name="frame" showname="Frame 1: 73 bytes on wire (584 bits), 73 bytes captured (584 bits)" size="73" pos="0"> </proto> <proto name="eth"> </proto> <proto name="ip" > </proto> <proto name="udp" showname="User Datagram Protocol, Src Port: 58150 (58150), Dst Port: domain (53)" size="8" pos="34"> </proto> <proto name="dns" showname="Domain Name System (query)" size="31" pos="42"> </proto> </packet> (Child elements and attributes of proto are not shown here)
  • 13. XML to MySQL • You can use LOTS of options: C++/Java,etc. • I used SimpleXML and XPath with PHP: $file = "test_T.xml"; $my_file = simplexml_load_file($file ); foreach ($my_file >xpath('//packet') as $packet) { $packet_type = $packet->proto[4]; echo $packet_type['name']; //protocol }
  • 14. And putting into database function LoadToDataBase($con) { $stmt =$con->prepare("INSERT INTO tcp (capture_order, from_ip, to_ip, from_port, to_port, tcp_length, tcp_stream, tcp_stream_text, tcp_sequence_dec) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)"); $stmt->bind_param('sssiiiisi', $this->capture_order,$this->from_ip, $this->to_ip, $this->from_port,$this->to_port, $this->tcp_length,$this->tcp_stream, $this->tcp_stream_text, $this->tcp_sequence_dec); $stmt->execute(); } Here $con is open connection to mysql, and all this vars I got in cycle. Please, refer to full code.
  • 15. Thank you • I hope you find this useful. • Full code is available at http://tigrantsat.me/randd/pcaptomysql/