HDF and HDF-EOS Workshop XVII (2014) Mike Folk

1. 1July 8 – 11, 2014
HDF and
Augmentation
Mike Folk
The HDF Group
ESIP 2014

2. www.hdfgroup.orgESIP Summer MeetingJuly 8 – 11, 2014 2
*
*shoes that remove
ice and snow

3. www.hdfgroup.orgESIP Summer MeetingJuly 8 – 11, 2014 3
*
*shoes that remove
ice and snow
Augment:
Add to, amplify, extend, enhance.
thesaurus.com

4. www.hdfgroup.orgESIP Summer Meeting
Augmentation
An HDF design philosophy
July 8 – 11, 2014 4

5. www.hdfgroup.orgESIP Summer Meeting
Why we augment
July 8 – 11, 2014 5
• Serve intended purpose better.
• Whole new purpose.
• Available to new tools and applications
• More “standard,” hence more widely accessible.
• Preservation

6. www.hdfgroup.orgESIP Summer Meeting
HDF augmentation
• Preserve the original use
• Add needed extra information
• Rename things
• Move things around
• Reduce complexity
• Hide things
July 8 – 11, 2014 6

7. July 8 – 11, 2014 7
AUGMENTING
HDF-EOS5 FILES
FOR NETCDF-4
ACCESS

8. www.hdfgroup.orgESIP Summer Meeting
HDF5
HDF-EOS5
HDF-EOS5
file
netCDF4 to access HDF-EOS5 files
• HDF-EOS5 uses HDF5 as
its storage layer
• Adds EOS structural and
content metadata
July 8 – 11, 2014 8

9. www.hdfgroup.orgESIP Summer Meeting
HDF5
HDF-EOS5
HDF-EOS5
file HDF5
NetCDF4
NetCDF-4
file
netCDF4 to access HDF-EOS5 files
• netCDF-4 uses HDF5 as
its storage layer
• Adds netCDF-4
structural and content
metadata
July 8 – 11, 2014 9

10. www.hdfgroup.orgESIP Summer Meeting
HDF5
HDF-EOS5
HDF-EOS5
file HDF5
NetCDF4
NetCDF-4
file
Augmented
HDF-EOS5
file
netCDF4 to access HDF-EOS5 files
• netCDF-4 metadata added to HDF-EOS5 file
• EOS5 use not affected.
July 8 – 11, 2014 10

11. July 8 – 11, 2014 11
H5AUGJPSS

12. www.hdfgroup.orgESIP Summer Meeting
IDV
• IDV
• Based on netCDF classic model (no groups)
• IDV required attributes and variables
• Scale factors, offsets
• Geolocation metadata
• “Radiance” variable
July 8 – 11, 2014 12

13. www.hdfgroup.orgESIP Summer Meeting
H5augjpss will …
• Make a JPSS HDF5 file look like netCDF-4
• Hide HDF5 elements not supported by netCDF
• Add coordinate variables
• Remove groups from view to conform to netCDF
classic
• Add information needed by tools such as IDV
• Geolocation metadata
• “Radiance” variable
July 8 – 11, 2014 13

14. HDF4 FILE CONTENT MAPS

15. www.hdfgroup.orgESIP Summer Meeting
HDF4 File Contents – User View
July 8 – 11, 2014 15

16. www.hdfgroup.orgESIP Summer Meeting
HDF4 File Contents – User View
Objects & Relationships
July 8 – 11, 2014 16

17. www.hdfgroup.orgESIP Summer Meeting
HDF4 File Contents – User View
User Metadata
July 8 – 11, 2014 17

18. www.hdfgroup.orgESIP Summer Meeting
HDF4 File Contents – User View
Object Data
July 8 – 11, 2014 18

19. www.hdfgroup.orgESIP Summer Meeting
HDF4 files are complex
July 8 – 11, 2014 19

20. www.hdfgroup.orgESIP Summer Meeting
Project Motivation
HDF4 file
HDF4
Library
HDFView

21. www.hdfgroup.orgESIP Summer Meeting
Project Motivation
HDF4 file
HDF4
Library
HDFView
X

22. www.hdfgroup.orgESIP Summer Meeting
HDF4 mapping workflow
HDF4 File
HDF4 Mapping File
(XML document)
hmap
linked with
HDF4 library
July 8 – 11, 2014 22

23. www.hdfgroup.orgESIP Summer Meeting
HDF4 mapping workflow
HDF4 File
HDF4 Mapping File
(XML document)
hmap
linked with
HDF4 library
Reader
program
Object Data Groups, Data Objects,
Structural and Application
Metadata;
Locations of Object Data
July 8 – 11, 2014 23

24. July 8 – 11, 2014 24