1. DataFinder - A Data Management Application
for Grids and Clouds
Why develop the How are Cloud and Grid
DataFinder? connected with the
Department
DataFinder?
Employee
When conducting experiments
Simulation Use Cases:
large datasets are produced. The same user interface is Using a grid to
The DataFinder supports Geometry
provided for all data stores, so calculate and execute
therefore: users have uniform access to Grid Generation Geometry, Flow
• organizing large data sets backends for Grid and Cloud Flow Solution
Simulation,… and also
• handling describing meta storage. This allows easy data storing Data in Grids or
Visualisation Clouds
data management throughout the
• execution of scripts for the whole virtual storage space,
automation of common for example copying and
procedures, like simulations moving data between Grid
and visualizations of and Cloud resources. Wizards:
Help to configure Use
experiment conditions
Cases and manage the
With the DataFinder the What kind of backends Data Access
produced datasets are easily does the Datafinder
accessible and processable. support?
Who could use the A major concept of
DataFinder? DataFinder is to store data on
different backends: Application:
A typical use case is • WebDAV server Datafinder (User
archiving of data on Interface)
• FTP und GridFTP server For executing scripts in
Cloud storage. Even if the • local file system grids and for accessing
computation producing the • Tivoli Storage Manager and managing data
data is performed locally (TSM) stored on different
or in the Grid, storing on • for higher flexibility: backends
a lightweighted storage Amazon Simple Storage
backend is possible. Service (S3)
The DataFinder could be The user has the option
especially interesting for to either store his data on
smaller companies or a classical server, on Grid
institutions which do not resources, or in the Amazon
want to maintain their own Cloud. External Medias
(CD, DVD,…) Meta Data Server
archiving infrastructure.
Storage System:
All possible
Example Use Case Backends, that can
D-Grid Project: „AeroGrid“
be used, including
• Technical and scientific data File System
Grids and Clouds
management on distributed resources ����
(Grid and Cloud)
• Documentation of CFD simulations
���� FTP Server
• Assure quality of results and trace S3 Cloud- Storage System
errors
WebDAV Server
• Repeatability of simulations
• User Front End „Datafinder“ (Python/
PyQT)
• SimulationCode: Cluster optimized Storage Resource
CFD code „Trace“ (C++) GridFTP Server Broker
• Jobs with Globus and UNICORE Tivoli Storage
Manager
Deutsches Zentrum Miriam Ney
für Luft- und Raumfahrt e.V. Miriam.Ney@dlr.de
in der Helmholtz-Gemeinschaft
Andreas Schreiber
Simulations- und Softwaretechnik Andreas.Schreiber@dlr.de
Köln/ Braunschweig/ Berlin
www.dlr.de/sc/datafinder Simulation and
Software Technology