https://github.com/valtech/ros_hadoop - One PC/NAS can have a hard time with large Rosbag files. Hadoop provides distributed storage and parallel processing. InputFormat allows to process Rosbag files via Hadoop interfaces.
2. Working with large Rosbag files
One PC/NAS can have a hard
time with large Rosbag files
Hadoop provides distributed
storage and parallel processing
Hadoop InputFormat allows to
process Rosbag files via Hadoop
interfaces:
https://github.com/valtech/ros_hadoop
2
3. Hadoop InputFormat for ROS
Benefits
+ Analyse ROS data with 3rd-party tools like Spark via Hadoop
interface
+ Run parallel jobs on ROS data in Hadoop with MapReduce
+ Save time because data conversion not needed
+ Save storage because data duplication not needed
3
4. Search & Analysis
+ Hadoop InputFormat and
Record Reader for Rosbag
+ Process Rosbag with Spark,
Yarn, MapReduce, Hadoop
Streaming API, …
+ Spark RDD are cached and
optimized for analysis
Large
Rosbag
Processing
Engine
Computer
Network
Storage
Advanced
Analytics
RDD
Record
Reader
RDD
DataFrame, DataSet
SQL, Spark APIs
NumPy
Ros
Msg
5. Training & Evaluation
+ TensorFlow Record Reader
+ Protocol Buffers to serialize
records
+ Save time because data
conversion not needed
+ Save storage because data
duplication not needed
Training
Engine
Machine
Learning
Computer
Network
Storage
Record
Reader
Ros
msg
Large
Rosbag
6. Re-Simulation & Testing
+ Use Spark for preprocessing,
transformation, cleansing,
aggregation, time window
selection before publish to ROS
topics
+ Use Re-Simulation framework
of choice to subscribe to the
ROS topics
Engine
Re-Simulation
with framework
of choice
Computer
Network
Storage
Ros
topic
core
subscribe
publish
Large
Rosbag