Presentation by Tony Kent on Managing data interoperability with FME at GIS Ireland 11/10/12

1. Managing Data
Interoperability with FME
Tony Kent
Applications Engineer
IMGS

2. IMGS
We deliver innovative spatial solutions
For the desktop, web and mobile
Built on our partner’s technology
Designed to meet the challenges of Government,
Mapping Agencies, and Utility & Communications
Customers

3. Safe Software
Powering the flow of spatial data with
FME
Enabling people to use their spatial data where, when and how they want to
Most Used Spatial Interoperable Solution in Ireland
3

4. Why Spatial ETL?
Significant proliferation of different spatial data
formats and types
Hundreds of formats, with more added each year
Multiple types of data stored in multiple systems
Unique data model requirements for each application
4

5. Why Spatial ETL?
Traditional approaches to data translation
and data model manipulation are not viable
Complex, inefficient and time-consuming
5

6. Why Spatial ETL?
Increasing pressure for access to spatial data
More users, beyond traditional GIS users
Expectations of real-time custom data views, 24x7
6

7. FME Capabilities
 The only complete spatial ETL solution
 Translate spatial data from one
format to another
 Transform spatial data into the
precise data model you need
 Integrate different data types
into a single data model
 Distribute spatial data to users
where, when and how they need it
7

8. FME Desktop
Flexible and powerful spatial ETL toolset
Translate, transform and integrate data in hundreds of
formats
Graphical authoring environment
Step 1 - Extract
Select and add the source
dataset(s)
Step 2 - Transform
Add transformers to manipulate
the data as it moves from source
to destination
Step 3 – Load
Load the transformed data into a
destination format and source
8

9. FME Workbench
Use simple
point and click
to easily define
spatial data
flows to
translate,
transform and
integrate your
data
9

10. Examples
Automating Ordnance Survey data updates
Pushing NTF data to multiple GIS platforms
Stripping out unnecessary data
Adding custom styling and symbology – CAD
E.g. Eircom, ESB, Fingal County Council
Publishing data to internal public portals
Bulk and transactional updates
Fire wall Friendly – use selected port
Completely automated
E.g. Dublin City Council

11. Open Data Challenge
You want to meet Open data requirements, but
your data is organized rather differently
?

12. What FME does …
?
Build data bridges
to your SDI

13. SDI Harmonization
Core Concepts
Harmonization: implied requirement for building
an SDI
Disparate sources must be mapped to a common
destination data model
Core to the harmonization workflow is a process
called schema mapping.
Delivered by services based on open standards

14. Harmonization Principles
Typical stages:
1. Evaluation
2. Assembly
3. Transformation
4. Validation
5. Publication
Based on the Spatial ETL concept (Extract,
Transform and Load), as applied to INSPIRE SDI’s

15. Evaluation
Assess destination schema and data
requirements
Assess source datasets and schema
Consider fundamental differences in
representation, resolution
Closely inspect actual representative
datasets

16. Data Assembly
Assess the diversity of source data types:
vector, raster, CAD, GIS, database, text, XML,
web, 3D, sensor and non-spatial
Review format and semantic translation
needs
Decide how to perform necessary joins
ID joins, spatial relates, nearest neighbor, one to
many relationships
Goal is to build a data structure that corresponds with your
published standard

17. Metadata – Data about data
Describes data structures
tables
geometry types
data types
fields
Describes data content
coordinate system
extent
modification date
quality, ownership, etc.

18. Metadata - Purpose

19. Key FME Metadata Capabilities
Reading
Writing
Updating
Harvesting
Validating
Integration with web services

20. CSW: Catalog Service for the Web
Deegree
GeoNetwork
OpenGIS
TerraCatalog

21. Data Transformation - Schema
Reshape source data
to match required
destination schema
Schema mapping
feature type
attribute name
new attribute
creation
code lists
conditional value
mappings

22. Schema Mapping in FME
Feature Type Mapping in FME Workbench
Attribute Mapping in FME Workbench

23. FME Data Model Restructuring:
Attribute Names & Values
 Value Mapping

24. FME SchemaMapper:
INSPIRE geographic names
FME Workspace
Name mapping
Name &
value mapping

25. Transformation: Geometry
Non-spatial to spatial
Geometry extraction (spatial to GML)
Representation transform: CAD drawing lines with labels to
GIS polygonal features with attributes
Coordinate System Reprojection (ED50 to ETRF89)
Simple to complex geometry
Source point and polygon data to multiple geometric
representations (city as point / area, river as line / area)
Generalization and interpolation
Highly granular national and regional datasets often require thinning
to be usable on pan-European scales

26. Validation
Schema validation i.e. INSPIRE (xsds)
Data integrity
Unique IDs
Geometric integrity (closed polygons)
Null values (nullable?)
Valid values: ranges and domain codes
Data gaps
Bounds
Network integrity
Custom validity rules specific to domain
Validation automation via FME Server upload
Ensure data quality throughout the data transformation process

27. Publication
Produce INSPIRE compliant GML
Provide discovery, view or download
services, for WxS, GML and other desired
formats
Publish with FME Server or integrate with
your geo web server of choice:
Spatial Data Services

28. Publication with FME Server
Publish workspace to FME Server
Store the workspace in a central repository
Make your FME workspaces available to others –over
the web
Register the workspace with one or more services (Data
Streaming, Data Download, etc.)
28

29. FME Tools for INSPIRE
Format translation
Schema mapping
String and list manipulation
Data validation
Database load and extract
XML,GML,WFS: reading, validation,
publication
Web services: WFS, WMS,
integration with others
Metadata support
Enterprise services with FME Server

30. Other ETL Software Options
Intergraph GeoMedia Fusion
GeoKettle – Open Source
Snowflake Software
Spatial Data Integrator – Open Source

31. Summary
FME can provide all the tools to help build support your
data sharing needs:
Integrate your data sources
Manage your meta data catalogues
Transform your data to standard schemas
Publish the data in the required formats

32. Thank You
For more information:
Email: ckirk@imgs.ie or tkent@imgs.ie
Twitter: @Ciarankirk
Web: www.imgs.ie