1. Aggregation/Subsetting
Use Case: Unidata IDD/LDM Data
Ethan Davis
UCAR Unidata

2. Unidata Use Case
• Serving data from IDD/LDM data streams
• Real-time data: model, satellite, radar, station,
profiles, etc.
– A lot of data, e.g., several radar data records per
second
– delete after 7, 30, 45 days depending on data and
server

3. Starting from netCDF data model
(array index space)
• netCDF and OPeNDAP data models
don't understand coordinate systems
– Arrays and index space
– Sequences with constraints
• Lots of limitations when dealing with array index space
– Types of aggregation
• Join on an Existing dimension
• Join on a New Dimension
• Union

4. Problems with Array Index based
Aggregation
• Data access/subsetting:
– Client WANTS to deal with coordinate systems
– Client must do some heavy lifting
– rolling archive means the mapping between
index space and coordinate space is potentially
time dependent
• Aggregation:
– Brittle: Data must be VERY homogeneous (any
variation breaks things … and there's always
variation in real-time data)

5. Coordinate System and Data Type
Aggregation/Subsetting
• Aggregation
– Higher-level understanding of datasets allows for improved
aggregation.
• Not as brittle.
• Better understanding of needed metadata changes
• Subsetting
– Higher-level understanding of datasets allows for services that don't
require as much work by client
• Grid: OGC WCS and WMS
• Point, station, profile:
– TDS NCSS, etc.
– OGC SOS and WFS (* Outside implementations)
– Advantages:
• Easier for users/clients
• Can better handle real-time/changing datasets

6. GRIB storage

7. netCDF storage

8. GRIB Rectilyzationologicment
• Turn unordered collection of 2D slices into 3-6D multidimensional array
• Each GRIB record (2D slice) is independent
• There is no overall schema to describe what its supposed to be
 there is, but not able to be encoded in GRIB