1. Geoscience
Data
Analysis
and
Visualization
Tools
from
NCAR
HDF/HDF-EOS Workshop XI
Nov 6-8, 2007
David Brown!

2. Mesa
Lab
·
Boulder,
Colorado
Visualization
and
Enabling
Technologies
Section
National Center for Atmospheric Research
Photos: Lynda Lester

3. Topics
•
•
•
•
•
Two interfaces: the same capabilities (mostly)
– NCL - a self-contained scripting language
– PyNGL and PyNIO - Python modules
Quick overview: visualization and analysis
Data model
State of our I/O component development
Enabling access to HDF 5, HDFEOS 5, and
NetCDF 4
HDF/HDEOS
Workshop
XI,
November
2007

4. PyNIO and PyNGL
July 2006
(Numeric 24.x/NumPy 1.0)
PyNIO
2005
PyNGL
2004
NIO
2005
2000-present:
30+ NCL Workshops
1995: NCL Conference
1992: NCARG Conference
Original Fortran
Late 60s
GSUN - Late 90s
PyHLU
2003
NCL
Mid 90s
C interface
Mid 90s
HLUs
Mid 90s
GKS/CGM/Fortran 77/UNIX conversion
NCAR Graphics
1980s

5. NCAR Command Language (NCL)
A scripting language designed for
scientific data analysis and visualization
1. Simple, robust file
input and output
2. Hundreds of
analysis functions
3. Visualizations
(2D) are world
class and highly
customizable
http://www.ncl.ucar.edu/
HDF/HDEOS
Workshop
XI,
November
2007

6. NCL: Data analysis
• Array-based syntax and operators
• Hundreds of functions
-
-
-
-
Spherical harmonics
Scalar and vector regridding
Vertical interpolation
EOFs
• Many tailored to geosciences
• Most handle missing data
• Can call C and Fortran routines
HDF/HDEOS
Workshop
XI,
November
2007

7. NCL: Visualization
• High-quality and customizable visualizations
• Contours, XY, vectors, streamlines
•
Maps with most common map projections
• Handles data on regular and irregular grids,
triangular meshes
• Specialized scripts for skew-T, wind roses,
histograms, Taylor diagrams, panels, bar charts
• GSUN interface: simplifies visualization
• Over 1,400 visualization options available
HDF/HDEOS
Workshop
XI,
November
2007

8. HDF/HDEOS
Workshop
XI,
November
2007

9. PyNGL - Python module
•
•
•
•
Python NCL Graphics Library
Python version of popular GSUN interface
Same publication-quality graphics as NCL
Utilizes existing Python modules and
development tools (swig, Numeric, NumPy)
• Contains some climate-specific data
analysis functions
• Extensive and updated documentation
http://www.pyngl.ucar.edu/
HDF/HDEOS
Workshop
XI,
November
2007

10. NCL Data Model
• Based on netCDF 3
• Language variables can have:
• Named dimensions
• Attributes
• Coordinates variables
• A convention in NetCDF but a language feature in
NCL
• 1d array with the same name as a dimension
• Contains coordinate values for dimension elements
HDF/HDEOS
Workshop
XI,
November
2007

11. Supported file format I/O
• One function reads all supported data
formats:
• NetCDF (now including NetCDF4 classic),
HDF4, HDF-EOS
2, GRIB 1 and 2
• Writes NetCDF and HDF4
• All file formats massaged into same
model: file variables have basically the
same features as internal variables
• Flat name space: HDF group names are
appended to variable names
HDF/HDEOS
Workshop
XI,
November
2007

12. PyNIO - Python module
•
•
•
•
•
Same I/O library as NCL (libnio)
Reads and writes same formats as NCL
Same NetCDF-like view of all formats
But interfaces with NumPy for data access
Modeled on the Scientific NetCDF module
– NioVariable type is a reference to the variable in
the file (metadata is attached to variable)
– Dereferencing it results in a NumPy variable with
no metadata
• *Numpy is a Python array processing module
http://www.pyngl.ucar.edu/Nio.shtml
HDF/HDEOS
Workshop
XI,
November
2007

13. 2D Coordinates
• Traditional NetCDF coordinate variables are
1D vectors
• However, satellite data and model data on
modern grids require 2D coordinate grids
• 2D coordinate variables created on the fly for
GRIB grids and HDFEOS Grid-type data
• Added value variables that allow data to be
plotted on any map projection
HDF/HDEOS
Workshop
XI,
November
2007

14. Enabling NetCDF 4 ‘classic’
• Recompile with NetCDF 4 and HDF 5 1.8
beta (we also included szip)
• Add options for specifying format and
compression level
• Tests using ncl_convert2nc on GRIB files
show reduction in file size by ~1/2 over GRIB.
• Caveats:
– Beta version NetCDF 4 not yet supported on some
architectures (64 bit mainly)
– Not supported for OPeNDAP NetCDF client library
HDF/HDEOS
Workshop
XI,
November
2007

15. Path to HDF, HDFEOS 5 and
full NetCDF 4 support
• size_t for dimension sizes to enable large
variables on 64-bit hosts
• Add support for more atomic types (int64)
• Support for groups in the file context
• User control of chunking
• Compound and variable length data types
• Components to read HDF 5 and HDFEOS 5
• Current NetCDF module extended to handle v4
• More support for aggregation
HDF/HDEOS
Workshop
XI,
November
2007

16. Current status
• NCL 5.0.0 released this week
– NCSA-style Open Source license (finally)
– Binaries available under slightly more
restrictive license (because of included 3rd
party software)
• PyNGL/PyNIO 1.2.0 available
– Supports NumPy 1.0.x and Python 2.5
– Binaries (easy installation) available for:
• Various flavors of Linux, Mac OSX, Cygwin
• Other Unix systems, 32 and 64 bit
HDF/HDEOS
Workshop
XI,
November
2007

17. Distribution of NCL sites
Distribution of PyNGL/PyNIO sites

18. Support
• Websites with extensive documentation
• Tutorials
• Hundreds of examples with
downloadable scripts
• Active email lists
– ncl-talk@ucar.edu, pyngl-talk@ucar.edu
• Hands-on training workshops
HDF/HDEOS
Workshop
XI,
November
2007

19. Download locations
• NCL
http://www.ncl.ucar.edu/download.html
• PyNGL and PyNIO
http://www.pyngl.ucar.edu/download.html
HDF/HDEOS
Workshop
XI,
November
2007

20. Questions? Email me: dbrown@ucar.edu
HDF/HDEOS
Workshop
XI,
November
2007

21. education
What kind of research
monsoons
(from our ESG download website)
rainfall
cloud modeling
climate change
data conversion
lightning
oceanography
marine studies
air quality
planetary
atmospheres
dust
atmospheric research
astronomy
weather forecasting
paleoclimatology
utility forecasting
pollution
acoustics
salmon abundance
glaciers
hydrology
medical
flood risk mgmt
fire modeling
physics
agriculture
HDF/HDEOS
Workshop
XI,
November
2007
microbiology
electric power
storms
aerosol light
absorption

22. Sample GRIB output
from PyNIO
filename:
ruc2.T00Z.bgrbanl
path: /Users/dbrown/grib/ruc2.T00Z.bgrbanl.grb
file global attributes:
dimensions:
gridx_236 = 113
gridy_236 = 151
lv_HYBL2 = 40
lv_DBLL3 = 5
variables:
float PRES_236_TRO ( gridx_236, gridy_236 )
center :
US National Weather Service - NCEP (WMC)
long_name : Pressure
units :
Pa
_FillValue : -999
coordinates : gridlat_236 gridlon_236
level_indicator :
7
grid_number : 236
parameter_number : 1
model :
RUC Model from FSL (isentropic; scale: 20km at 40N)
forecast_time :
0
initial_time : 05/11/2000 (00:00)
…
float gridlat_236 ( gridx_236, gridy_236 )
corners :
ARRAY
long_name : latitude
GridType : Lambert Conformal Secant or Tangent, Conical or bipolar
…
units :
degrees_north

23. Resources (a.k.a. attributes)
• Same control mechanism for NCL and
PyNGL
• Detailed control of the appearance of a
visualization
• Example using POP grid ocean current
data
HDF/HDEOS
Workshop
XI,
November
2007

24. HDF/HDEOS
Workshop
XI,
November
2007

25. HDF/HDEOS
Workshop
XI,
November
2007

26. HDF/HDEOS
Workshop
XI,
November
2007

27. HDF/HDEOS
Workshop
XI,
November
2007

28. HDF/HDEOS
Workshop
XI,
November
2007

29. NCL and PyNGL visualization
examples
• scatter and line plots, contours, vectors,
skew-t, meteograms, maps, wind barbs,
bar charts, streamlines, trajectories,
filled polygons, paneled visualizations
HDF/HDEOS
Workshop
XI,
November
2007

32. Skew-T graphic
Courtesy of
Dennis Shea
A Skew-T plot is
used by
meteorologists to
analyze data from
a balloon
sounding.

36. Based on a visualization
of Adam Phillips

37. Based on a visualization
of Joel Norris (Scripps)
using dummy data

38. Data from Climate Analysis Section
Christian Guillemot

39. GRIB 2 TIGGE (THORPEX Interactive Grand Global Ensemble) data
Japan Meteorological Agency
!
THORPEX - an international research and development
program responding to the weather related challenges of
the 21st century to accelerate improvements in the
accuracy of 1-day to 2- week high impact weather
forecasts for the benefit of society, the economy and the
!
environment.

40. Courtesy Jeff Yin
NCAR CGD

42. Image courtesy of Nan
Rosenbloom, CGD

43. First two map databases built-in; high-resolution available as simple download

44. Christophe Cassou
(CNRS/CERFACS)
This grid could be
described as a tripole
grid that is further
modified by the
arbitrary displacement
of some portions of the
grid to achieve finer
resolution over areas
of interest (typically,
ocean areas).

45. Courtesy Mark Stevens, NCAR CGD

46. Brett D. Estrada
Naval Research Laboratory

47. Grid from Tom Gross
NOAA/NOS/CSDL/MMAP

48. Special grids you can contour
Soon to include vectors and streamlines
• Grids already on a triangular mesh
• Grids with “cell boundary” information
• Grids represented by one-dimensional arrays
of x, y, and z values
HDF/HDEOS
Workshop
XI,
November
2007

49. HDF/HDEOS
Workshop
XI,
November
2007
http://isccp.giss.nasa.gov/

50. HDF/HDEOS
Workshop
XI,
November
2007
http://isccp.giss.nasa.gov/

51. ARPEGE GRID
(used by Meteo-France)
Christophe Cassou
(CNRS/CERFACS)
Similar to ISCCP grid,
but with somewhat finer
resolution. Grid is rotated
with respect to the globe
so as to put its poles
somewhere other than at
the North and South
Pole.
HDF/HDEOS
Workshop
XI,
November
2007

52. Data from Dave Randall,
Todd Ringler,
Ross Heikes of CSU
Most geodesic grids
appear to be formed by
elaborating an
icosahedron; each of the
20 faces of the
icosahedron is
subdivided into smaller
triangles in a more or less
obvious way.

53. Data from Dave Randall,
Todd Ringler,
Ross Heikes of CSU

54. Station data:
contouring on a
triangular mesh

56. Triangular mesh from Brett D.
Estrada, Naval Research Laboratory
HDF/HDEOS
Workshop
XI,
November
2007
Grid from Brett D. Estrada, Naval Research Laboratory

57. Triangular mesh from Tom
Gross
NOAA/NOS/CSDL/MMAP

58. Samples from VASPACKT
Vectors and streamlines on a triangular mesh
•
•
•
•
•
Developed by Dave Kennison
Simple vectors
Curly vectors
Streamlines
Examples use dummy data
HDF/HDEOS
Workshop
XI,
November
2007

59. Simple
vectors

60. Curly vectors

61. Streamlines