HDF

HDF
HDF/HDF-EOS Workshop III
Sept. 14-16, 1999
Mike Folk, HDF Group
http://hdf.ncsa.uiuc.edu/
National Center for Supercomputing Applications
University of Illinois at Urbana-Champaign
NCSA/Univ of Illinois at Urbana-Champaign

HDF

1

Topics

I. Overview
II. NCSA HDF Activities
III. HDF5
IV. HDF4 vs. HDF5


HDF

2

I. HDF Overview


HDF

3

HDF Mission

To develop, promote, deploy, and support
open and free technologies that facilitate
scientific data storage, exchange, access,
analysis and discovery.


HDF

4

What is HDF?
• Scientific data file format & supporting
software
• For images, arrays, tables, other structures
• Features
– Portability across architectures
• I/O library
• Files

– Efficient I/O
– Efficient storage
HDF


5

Why use HDF?
•
•
•
•
•
•

Manage data
Share data
Use software that understands HDF
Improve I/O performance
Improve storage efficiency
Use an open standard


HDF

6

An HDF File: A Collection of
Scientific Data Objects

HDF file containing four 3-D arrays


HDF

7

Mixing HDF Objects in One File

3-D array

group

Raster image

palette

HDF file
3-D array

Raster
image

Lat lon temp
---- ---- ----12
23
3.1
15
24
4.2
17
21
3.6
16
35
5.7

Table

HDF

8

HDF Software
Utilities and applications for
manipulating, viewing, and
analyzing data.

General Applications
Application
Programming
Interfaces
Low-level
Interface
HDF
file

}

HDF I/O library
– High-level, object-specific APIs.
– Low-level API for I/O to files, etc.
File or other data source.

HDF

9

HDF Applications Software
• Free software
– NCSA HDF library and utilities
– Other software

• Commercial/other software that “understands”
– all of HDF (Noesys, IDL, HDF Explorer)
– certain HDF objects (MATLAB, WebWinds)
– certain HDF applications (SHARP, WIM)
• http://hdf.ncsa.uiuc.edu/tools.html

HDF

10

What platforms does HDF run on?
• Sun: Solaris
• SGI: Indy, Power Challenge, Origin, Cray C90, YMP, T3E
• HP9000, HP-Convex Exemplar
• IBM: RS6000, SP2
• DEC: Alpha/Digital UNIX, OpenVMS
VAX: OpenVMS
• Intel: Solarisx86, Linux, FreeBSD, Windows NT/98
• PowerPC: Mac-OS

University

HDF

11

A Sampling of HDF Users
NCSA-affiliated Science teams

Visualization, data exch, fast I/O, ...

Mathworks, Fortner Software,
Research Systems Inc., etc.

Format supported by vendors of vis
and data analysis software

Boeing

Space-time change detection in images

Distributed Oceanographic Data
System (DODS)

Remote access to earth science data

Army Research Lab

Network distributed global memory

Center for Analysis & Prediction
of Storms

Fast parallel I/O, portability,
multi-resolution grids

TRAPPIST
(Euro consortium)

Exchange, analysis & visualization of
non-destructive testing data

HDF

12

Major User #1: EOSDIS
• ESDIS Project
– open standard exchange format and I/O library for EOSDIS
– EOS applications

• HDF requirements
–
–
–
–
–

Earth science data types (HDF-EOS, etc,)
User support for scientists, data producers, etc.
Library and file structure improvements
HDF tools, utilities, access software
Software maintenance and QA

HDF

13

Major User #2: ASCI
• ASCI Data Models and Formats (DMF) Group
– open standard exchange format and I/O library for ASCI
– DOE tri-lab ASCI applications

• HDF requirements
–
–
–
–

large datasets (> a terabyte)
ASCI data types, especially meshes
good performance in massive parallel environments
primarily HDF 5


HDF

14

II. NCSA HDF Activities


HDF

15

Java applications
• HDF APIs
– Basis for tools that access HDF

• HDF Viewers
– HDF browser/visualizer

• HDF4 Data Server Prototype
– Lessons learned about remote access to


HDF

16

Remote Data Access

• The SDB: Web-based Server-side Data
Browser
• Java for remote access
• WP-ESIP: DODS project
• Computational Grids (Globus/GASS)


HDF

17

HDF Standardization
• To share files, users must organize them similarly.
• HDF user groups create standard profiles
– Ways to organize data in HDF files.
– Metadata
– API

• Examples: HDF-EOS, ASCI DMF


HDF

18

HDF-EOS software layers
HDF-EOS Applications
HDF-EOS
profiles

General Applications
HDF-EOS API
Application
Programming
Interfaces
Low-level
Interface
HDF
file

HDF

19

“HDF Configuration Record” (HCR)

• To simplify the tasks of defining, comparing,
and producing HDF-EOS files
• Formal (ODL) descriptions of HDF-EOS
objects


HDF

20

HCR of Swath
/* Project XYZ */
/* First version defined on June 10th, 1998 */
OBJECT = SWATH
NAME = SCAN1
OBJECT = Dimension
NAME = GeoTrack
Size = 1200
END_OBJECT = Dimension
OBJECT = Dimension
NAME = GeoCrossTrack
Size = 205
OBJECT = Dimension
NAME = DataX
Size = 2410
END_OBJECT = SWATH
END


HDF

21

HCR
• HCR Utilities:
– Converters: HCR ↔ HDF-EOS
– Edit HCR and HDF-EOS
– Compare HCR with HDF-EOS file

• Current projects:
– Extend HCR converters to all of HDF4
– Similar work with HDF5
– XML too

HDF

22

III. HDF5


HDF

23

Why HDF5?
• HDF shortcomings
exposed by EOSDIS, ASCI and others...
–
–
–
–
–

Limits on object & file size (<2GB)
Limited number of of objects (<20K)
Rigid data models
I/O performance
Aging software infrastructure (code entropy)

HDF

24

• …new Demands...
– Bigger, faster machines and storage systems
• massive parallelism, parallel file systems
• teraflop speeds, terabyte storage

– Greater complexity
• complex data structures
• complex subsetting

– More emphasis on remote & distributed access

HDF

25

• … and ASCI Requirements
–
–
–
–

Compatibility with vector bundle model
Compatibility with MPI-IO
Ability to transform data between memory & storage
Parallel file systems: PIOFS, HPSS, etc.


HDF

26

New HDF5 Features
• More scalable
– Larger arrays and files
– More objects

• Improved data model
– New datatypes
– Single comprehensive dataset object

• Improved software
– More flexible, robust library
– More flexible API
– More I/O options

HDF

27

HDF5 data model

• Two primary objects
• Dataset
– multidimensional array of elements
– rich variety of datatypes

• group
– directory-like structure
– contains datasets, groups, other objects

HDF

28

Dataset components
• multidimensional array
• header with metadata
–
–
–
–

datatype
dataspace
attributes
storage properties


HDF

29

Simple datatypes
•
•
•
•
•
•

The usual scalars: integer & float
user-defined scalars (e.g. 13-bit integers)
variable length (e.g. strings)
pointers to objects or regions of datasets
enumeration
opaque


HDF

30

Compound datatypes

•
•
•
•

User-defined
Comparable to C structs
Members can be simple or compound types
Members can be multidimensional


HDF

31

Data Spaces

• How data are organized to form a dataset
– rank
– dimensions

• Subsetting during I/O operations
– What subset of data is to be moved
– In-memory organization of data
– In-file organization of data

HDF

32

HDF5 dataset: array of records
int8

int4

int16

Datatype:

float32

Dimensionality: 5 x 3
Record

3

5


HDF

33

Dataspaces
Reading Dataset into Memory from File
File

Memory

2D array of integers

3D array of floats

Read


HDF

34

Selection: Examples of mappings between file selections
and memory selections.

(a) A hyperslab from a 2D array to the
corner of a smaller 2D array

(c) A sequence of points from a 2D array to
a sequence of points in a 3D array.

(b) A regular series of blocks from a 2D
array to a contiguous sequence at a
certain offset in a 1D array

(d) Union of slabs in file to union of slabs in
memory. No. of elements must be equal.


HDF

35

Attributes
• Named pieces of data
• Stored in a dataset or group header
• Operations are scaled-down versions of the
dataset operations
– Not extendible
– No compression
– No partial I/O

HDF

36

Property list

• Properties of objects or operations
• Describe how to create, store, access and
transfer data


HDF

37

Some Properties
• chunked

Better subsetting
access time;
extendable

• compressed

Improves storage
efficiency,
transmission speed

• extendable

Datasets can be
extended in any
direction

• split file

Dataset “Fred”

File A

HDF

Metadata for Fred

File B

Metadata in one file,
raw data in another.

Data for Fred

38

Dataset components
Dataset
Metadata

Data
Attributes
time = 32.4
pressure = 987
temp = 56

Dataspace
Datatype
Dim_3=2

Rank=2

Dim_2=4
Dim_1=5

int16

Storage properties
Chunked; compressed


HDF

39

Groups

•
•
•
•
•

Structures for organizing the file
Like Vgroups in HDF4
Like directories in hierarchical file system
Every file starts with a root group
Groups have attributes


HDF

40

Groups
• A mechanism for collections of
related objects
• Every file starts with a
root group
• Can have attributes
• Like directories
in Unix, but a graph,
rather than a tree

“root”


HDF

41

Groups
Groups and members of groups can be shared
root


HDF

42

Mounting
File A

File B

root

root

mount!

HDF

43

Reading & writing with HDF5

• Set properties
• Describe the data
– datatypes
– rank and dimensions
– mapping between file and memory

• Read/write

HDF

44

Files needn’t be files - Virtual File Layer
VFL: A public API for writing I/O drivers
Hid_t

“File” Handle
VFL: Virtual File I/O Layer
stdio

mpio

memory

network I/O drivers

“Storage”

Files

HDF

Memory

Network


45

HDF5 tools
• Current
– hdf5ls - lists contents of HDF5 file
– h5dumper - higher level view
– hdf5 hdf4 converter

• Future
–
–
–
–
–
HDF

Convert HDF5 ↔ ascii, binary, GIFF, etc
Convert HDF4 HDF5
Java tools - VisAD, etc.
File/code generation from DDL description
Talking to vendors
46

Other HDF5 activities

•
•
•
•

Performance tuning
Object model
Fortran and C++ API
Thread-safe HDF5


HDF

47

IV. HDF4 vs. HDF5


HDF

48

HDF4 vs. HDF5
• HDF4

• HDF5 - successor to HDF4

– Original format and library
– Compatible with all earlier
versions
– 6 primary objects
•
•
•
•
•

multidim array of scalars
raster image, palette
table
annotation
group

– Biggest current user: Earth
Observing System Data and
Info System (EOSDIS)

– New format and library
– Not compatible with earlier
versions
– 2 primary objects
• multidim. array of records
• group

– Biggest current user: Accelerated
Strategic Computing Initiative
(ASCI)

HDF

49

HDF4 object types can be derived from
HDF5 datasets and groups

HDF5 group

HDF5 dataset

HDF4 Vgroup
lat
12
15
17
23
25

lon
23
24
21
35
31

temp
3.1
4.2
3.6
7.2
6.3

HDF4 Vdata
1-dim array
of records

HDF

HDF4 SDS
n-dim array
of scalars

2-dim array of
multi-component
scalars

HDF4
8-bit raster

March 15, 1990.
Simulation with k=10.0,
beta=1.22e3. Calculate
the magnitude ...

03
-3
45
45

04
72
77
67

43
44
34
87

43
50
23
00

43
34
57
45

HDF4 NCSA/Univ of Illinois at Urbana-Champaign
24-bit raster

50

Status of HDF4 vs. HDF5
• HDF4 is still an EOS standard
• HDF5 likely also
• HDF4 maintenance
– Maintained as long as EOS needs it
– Minimal new feature

• New applications: use HDF5 if possible!
– New features, performance improvements, etc.

HDF

51

HDF Information
• HDF Information Center
– http://hdf.ncsa.uiuc.edu/

• HDF Help email address
– hdfhelp@ncsa.uiuc.edu

• HDF users mailing list
– hdfnews@ncsa.uiuc.edu


HDF

52

HDF

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a HDF

Similar a HDF (20)

Más de The HDF-EOS Tools and Information Center

Más de The HDF-EOS Tools and Information Center (20)

Último

Último (20)

HDF

Notas del editor