Status and future plans of raster support in GeoTools and GeoServer open source projects

1. Raster data in GeoServer and GeoTools:
Achievements, issues and future developments
Ing. Simone Giannecchini, GeoSolutions
Ing. Daniele Romagnoli
FOSS4G 2011, Denver
12th-16th September 2011

2. Outline
 Who we are
 Raster Developments
 JAITools
 ImageIO-Ext
 GeoTools
 GeoServer
 GeoBatch
FOSS4G 2011, Denver
12th-16th September 2011

3. GeoSolutions
 Founded in Italy in 2006
 Expertise
• Image Processing, GeoSpatial Data Fusion
• Java, Java Enterprise, C++, Python
• JPEG2000, JPIP, Advanced 2D visualization
 Supporting/Developing FOSS4G projects
 GeoTools, GeoServer
 GeoBatch, GeoNetwork
 Clients
 Public Agencies
 Private Companies
 http://www.geo-solutions.it
FOSS4G 2011, Denver
12th-16th September 2011

4. JAITools
 High performance raster processing
 Extends, complements, replaces Oracle Java Advanced
Imaging (JAI) Library
 Open Source, BSD license
 Low Level Operations
 e.g. RangeLookup
 Higher Level Operations
 Vectorize, Contour
 Works mostly in pure raster space (Image Processing)
 Stable release is 1.2.0
FOSS4G 2011, Denver
12th-16th September 2011

5. JAITools – Low Level Operations
 Basic Extensions to JAI processing
 RangeLookup
 Raster (re)classification
 Raster Statistics
 ZonalStats
 ClassifiedStats*
 KernelStats
 High Perf ColorMap*
 ColorMap Extraction/Inversion
 High Perf VectorBinarize
 Vector to raster conversion
FOSS4G 2011, Denver
12th-16th September 2011

6. JAITools – High Level Operations
 Vectorize
 Polygon Extraction
 Contour
 Isolines extraction
 Raster Algebra (Jiffle)
 MapCalc like extension
 Generic Raster Processing
FOSS4G 2011, Denver
12th-16th September 2011

7. ImageIO-Ext
 High Performance raster I/O
 Extends, complements, replaces Oracle JAI ImageIO
Library
 Open Source, LGPL and BSD licenses
 Plugins:
 BigTiff*, netCDF-CF,GriB1,MatFile 5, HDF
 JP2000 (Kakadu)
 GDAL integration via ImageIO-ext
 JPEG2000, MrSID, ECW
 BigTIFF, ERDAS Image, HDF4
 Complete list here
FOSS4G 2011, Denver
12th-16th September 2011

8. ImageIO-Ext – Improving Tiff
 Improved standard Tiff reader and writer
 Support for r/w double and float data type
 JP2 compressor/decompressor*
 Improved metadata Management
 More aggressive caching
 Based on soft references
 BigTiff support
 Read/write
 Reduced resolution page support
 Overviews!
FOSS4G 2011, Denver
12th-16th September 2011

9. ImageIO-Ext – JPEG2000
 Kakadu based reader/writer
 Wraps kakadu JNI APIs as ImageIO
ImageReader/ImageWriter
 Support for GeoJP2
 Supporting additional JPEG2000 code params
 Tiling, Corder, Tile Marker, Part Tile Marker,
Sprofile , etc…
 Some of them needed for NITF support
 Fine grain control over JPEG2000 codestream
creation
FOSS4G 2011, Denver
12th-16th September 2011

10. ImageIO-Ext – GDAL
 Wraps GDAL behind ImageIO API
 Mainly Read Capabilities
 Write support experimental (JP2Kak, GeoTiff)
 Geotiff, JP2Kak, ECW, MrSID, JP2MrSid, Erdas, IDRIS,
Esri Binary Grid, Envisat, NITF, DTED, DOQ1, DOQ2, etc..
 Expose Generic IIOMetadata for basic Dataset info
 Georeferencing, Band layout, CRS, etc..
 Expose Specific IIoMetadata for Driver specific
info
 E.g. MrSid metadata
 1.1.0 supports GDAL 1.7.3 directly (no custom bindings!)
FOSS4G 2011, Denver
12th-16th September 2011

11. GeoTools
 The Open Source Java GIS Toolkit
 LGPL
 Management of Geospatial data
 CRUD Operations
 Raster and vector
 Rendering
 GDAL Support
 Based on best-of-breed Open Source projects
 Java Advanced Imaging (JAI), ImageIO
 Java Topology Suite (JTS)
FOSS4G 2011, Denver
12th-16th September 2011

12. GeoTools - JPEG2K Improvements
 Leverage on ImageIO API
 Read/Write Capabilities
 Fine grain support for writing with code params
 GeoJP2 support
 Other capabilities
 Re-Tiling on the fly
 ColorMasking
 Immediate (Java2D) VS Deferred Mode (JAI)
 Multithreaded loading (per tile)
FOSS4G 2011, Denver
12th-16th September 2011

13. GeoTools - Tiff & BigTiff
 Leverages on ImageIO API
 Read/Write Capabilities
 Internal/External Overviews support
 Improved Metadata management
 Less memory
 Wise caching – more speed
 Heavy Refactor ongoing
 Re-Tiling on the fly
 ColorMasking
 Immediate (Java2D) VS Deferred Mode (JAI)
FOSS4G 2011, Denver
12th-16th September 2011

14. GeoTools - Raster Processes
 Basic Building Blocks for GeoServer WPS
 Clip and Ship (raster and vector), buffering, vector
intersection, vector union, etc…
 JAITOOLs
 Vectorize, Contour
 Statistics
 GDAL based*
 gdal_translate wrapper
 gdalwarp wrapper
 org2ogr wrapper
FOSS4G 2011, Denver
12th-16th September 2011

15. GeoTools - ImageMosaic
 Wraps multiple homogeneus raster as a single one
 Improvements
 Support for footprint on single tiles
 Support for external catalog of tiles
 Autoindexing of directories
 Support for TIME, ELEVATION
 Support for OGC & CQL Filtering
 Multithreaded loading
 Support for heterogeneous overviews layout
 Supports any format with an ImageIO reader
 Re-Tiling on the fly
 ColorMasking on the fly
 Immediate VS Deferred Mode
FOSS4G 2011, Denver
12th-16th September 2011

16. GeoTools - ImagePyramid
 Multiresolution/Multiscale tile based raster
plugin
 Take a large raster  split in small
chunks/tiles  mosaic and
downsample start over until happy
 Upon request
 Select right resolution level  select needed
chunks/tiles  assemble  do some magic and return
 Improvements
 Automatically recognize output of gdal_retile
 Automatic tile indexing
 Other improvements inherited from ImageMosaic
FOSS4G 2011, Denver
12th-16th September 2011

17. GeoTools – Optimized Reprojection
 Standard Approach  Accurate Transformation
 Each pixel is back mapped using the full transformation
 Grid to world on the target
 Backwards projection towards the source
 World to grid on the source
 Slow!, cannot be optimized in hardware
 Optimization  Approximate Transformation
 Local Linear Approximation using Grid Warps
 Use a set of known positions and their transformations
 Everything else is linearly interpolated between the known
positions
 Estimation through iterative process (speed vs quality)
FOSS4G 2011, Denver
12th-16th September 2011

18. GeoTools – Optimized Reprojection
Linearly interpolate on the top and bottom lines
Linearly interpolate between the two resulting points
FOSS4G 2011, Denver
12th-16th September 2011

19. GeoTools – Optimized Reprojection
FOSS4G 2011, Denver
12th-16th September 2011

20. GeoServer
 GeoSpatial enterprise gateway
 Java Enterprise
 Management of raster and vector data
 Standards compliant
 OGC WCS 1.0, 1.1.1 (RI)
 OGC WFS 1.0, 1.1 (RI), 2.0
 OGC WMS 1.1.1, 1.3
 OGC WPS 1.0.0
 Google Earth/Maps support
 KML, GeoSearch, etc..
FOSS4G 2011, Denver
12th-16th September 2011

21. GeoServer - Rendering Transformations
 In Place Transformations during rendering
Specified in SLD docs
 Applied right before the rest of the
directives
 Can be used to:
 Extract and draw contours on
the fly
 Extract and draw polygons
on the fly
 Extract and draw symbols (e.g. arrows) on the fly
 Transform geometries on the fly
 (Add your own!)
FOSS4G 2011, Denver
12th-16th September 2011

22. GeoServer - Rendering Transformations
On-the-fly
Polygonalization
On-the-fly
Contouring
Band
Raster to Combine +
point vector Symbology
FOSS4G 2011, Denver
12th-16th September 2011

23. GeoServer – Improved Raster Rendering
 Hit JAVA2D Bottleneck – Scalability Issue
 http://bit.ly/qJcZBi
 http://bit.ly/oe0CHo
 Created direct raster rendering path with JAI
On-the-fly
Polygonalization
 Drop-in replacement (1 raster at time)
On-the-fly
 2x speedup
Contouring
 2x/3x scalability improvement under heavy load
 Enabled/Disabled via Java Switch
Band
Raster to Combine +
point vector Symbology
FOSS4G 2011, Denver
12th-16th September 2011

24. GeoServer – Improved Raster Chains
Performance
 Improved internal JAI Concurrency
 Introduced and Improved JAI Tile Recycling
 Reduced depth of JAI Processing Chains
 More aggressive factory caching On-the-fly
 Added a few specific high perf JAI operations
Polygonalization
 e.g. crop On-the-fly
Contouring
 Created Specific high perf ROI class
 Improving Tile Caching
 Work ongoing

Band
Transparent File vs Memory Caching when writing
Raster to Combine +
raster point vector Symbology
FOSS4G 2011, Denver
12th-16th September 2011

25. GeoServer – More Raster WPS Processes
 Relies on GeoTools Processes
 Vectorize
 Contouring
 RasterClassification On-the-fly
Polygonalization
 ZonalStats On-the-fly
Contouring
 GDAL Processes
 gdal_translate
 gdalwarp
Band
 ogr2ogr Raster to Combine +
point vector Symbology
FOSS4G 2011, Denver
12th-16th September 2011

26. GeoServer – Support for TIME &
ELEVATION dimensions + Filters
 WMS TIME and ELEVATION dimensions
 …&TIME=20110914T00:00:000Z&ELEVATION=10
 Implemented thanks to ImageMosaic
 Support for the Current keyword
On-the-fly
 Support for Default values Polygonalization
 Support for On-the-fly
 Contouring
TimeInstant
 Discrete Range Tb/Te/Res, Eb/Ee/Res
 Continuous Range Tb/Te, Eb/Ee
 Any combination of the above
 Support for OGC and CQL filters Band
 Raster to
Filter Mosaic granules based on index Combine +
attributes
point vector Symbology
FOSS4G 2011, Denver
12th-16th September 2011

27. GeoBatch
 Real-time produced data
 Remote Sensing, In situ, Models, etc..
 Recognizable, isolate data streams
 Flow
 Incoming series of geospatial data
 Specific format
 Defined rules for preprocessing
 Defined rules for ingestion and/or exploitation
 Automatic Ingestion and preprocessing  No manual intervention
 Example, GeoTiff preprocessing
GeoNetwork
Producer Tile Overviews Publish
GeoServer
FOSS4G 2011, Denver
12th-16th September 2011

28. GeoBatch – GeoTiff Preprocessing
 GeoTiff serving optimizations
 Internal Tiling
 Compression
 Internal/External Overviews
 Mosaicking On-the-fly
Polygonalization
 Pyramiding

On-the-fly
GeoServer publishing
Contouring
 2 Options
 Pure Java based on ImageIO and GeoTools
 External Processes with calls to GDAL
Band
Raster to Combine +
point vector Symbology
FOSS4G 2011, Denver
12th-16th September 2011

29. GeoBatch – MetOc Data Preprocessing
 Optimization of MetOc for WMS/WCS serving
 Conversion from
 GRiB 1
 netCDF-CF
 COAMPS On-the-fly
Polygonalization
 BUFr*
On-the-fly
 To Contouring
 Multiple 2D geotiff (Internal Overviews)
 Indexing with ImageMosaic
 More preprocessing
 Fixing No Data Band
 Change UoM Raster to Combine +
point vector Symbology
 Band Processing Wind(U,V)  Wind ( mod, theta)
FOSS4G 2011, Denver
12th-16th September 2011

30. The End
Questions?
simone.giannecchini@geo-solutions.it
FOSS4G 2011, Denver
12th-16th September 2011

31. GeoTools - ImageMosaic
FOSS4G 2011, Denver
12th-16th September 2011

32. GeoTools – Optimized Reprojection
A raster reprojection
involves chaining
togheter the standard
reprojection with the grid
to world transformations
of source and target
FOSS4G 2011, Denver
12th-16th September 2011

33. GeoTools – Optimized Reprojection
Set of evaluation vectors used in the
Recursive algorithm
Test distance between actual position of the reprojected point
and the predicted by a linear interpolation
FOSS4G 2011, Denver
12th-16th September 2011

34. GeoTools – Optimized Reprojection
FOSS4G 2011, Denver
12th-16th September 2011