Data integration makes it possible to deliver intelligence and keep decision makers, first responders, and civilians informed. For over 20 years, FME has been trusted by federal governments to move data from nearly any source to the target destination, while saving time and budget resources.
With FME, federal governments can deliver open data, improve emergency & disaster response, enhance land management, turn public safety and defense into actionable results, and integrate & deliver location intelligence.
Using Data Integration to Deliver Intelligence to Anyone, Anywhere
1. Using Data Integration to Deliver
Intelligence to Anyone, Anywhere
Craig Hantke, Dean Hintz
2. Dean Hintz
Senior Applications Analyst
Meet the experts
Craig Hantke
Account Manager
Use the GoToWebinar Control Panel to chat in questions
Erin Lemky
Senior Product
Marketing Manager
3. “The world’s most valuable resource
is no longer oil, but data.”
THE ECONOMIST
5. ● 66% of enterprises rank Location
Intelligence as either critical or very
important to ongoing revenue growth
strategies (Forbes)
● Gain insight into your key assets &
security risks
● Integrate geospatial into your
intelligence
Importance of Location
8. SOLUTION: DATA INTEGRATION
“The discipline of data integration comprises the
practices, architectural techniques and tools for
achieving the consistent access and delivery of
data across the spectrum of data subject areas and
data structure types in the enterprise to meet the
data consumption requirements of all applications
and business processes.”
- Gartner
9. Years of solving data challenges
25
Safe So ware
COMPANY PROFILE
10,000
Organizations trusting us worldwide
Our mission is to help
you maximize the
value of your data.
Partners supporting our network
150
128
Countries with FME customers
11. Connect Your Data Sources Transform Your Data Automate Your Workflows
12. CONNECT
CAD
GIS
FME release by year
NumberofsupportedformatsinFME
1995
10500100300
20192015201020052000
TABULAR
DATABASE
RASTER
POINT
CLOUD
BIM
3D
WEB
XML
CLOUD
BIG
DATA
IOT
BI
AR / VR
AI / ML
INDOOR
MAPPING
GAMING
14. FME and Intelligence: Conceptual Approach
There are several key areas where FME can support deriving and delivering
key intelligence resources to your decision makers:
● Extraction / collection
● Integration / fusion
● Filtering / cleaning
● Enrichment / analysis
● Distribution
With all the above workflows, the key to scalability of both volume and
complexity is automation
17. CUSTOMER STORY
“We love FME.
We’ve been using it for about 20 years.”
- Piet Nooij, Fortis BC
PROJECT
Assess the current wildfire threat to
assets.
SOLUTION
Integrate active wildfire data from
provincial government with their GIS.
RESULTS
● Workflow automatically runs at
same interval as source dataset
updates.
● Notifications & reports are
immediately sent to Operations
Managers who can coordinate
with Emergency Services.
FORTIS BC >
23. Powerlink is a leading Australian provider of
high voltage electricity transmission network
services, combining innovation with insight to
deliver safe, cost effective and reliable
solutions.
About Powerlink
The Emergency Readiness Project Team
Mark West – Manager of Technical Information Services
Aaron Ambler – Information & Services Team Leader
Alexei Radun – Data & Systems Team Leader
Matthew Cooper – Information Analyst
John Mockler – Information Analyst
Tim Mackay – Spatial Coordinator
Ron Innes – Project Server Coordinator
Kyle Morris – Information Support Specialist
John Marshall – FME Developer
A Queensland Government Owned
Corporation that owns, develops, operates
and maintains the transmission network that
extends 1,700km from north of Cairns to the
New South Wales border, and comprises
15,337 circuit kilometres of transmission lines
and 140 substations.
24. Emergency Spatial Journey
2019
2018
2017
2015
2011
2013
Pre 2010
Fully Automated, Data driven approach to Emergency awareness
Optimised data & information management
Expanded data access to increase reliability
Full automation of data downloads FME Server
Introduced manual reactive mapping & data retrieval & creation
No internal emergency data or Information Support
Development of FME mapping automation
Patent
Pendin
g
26. Bushfire
Workflows
Fire started
Analysis Risk levels
Bushfire Low Risk
Bushfire Medium Risk
Bushfire High Risk
Gas Wells
QRFS
Planned Burns
QRFS
Sentinel
>=50 Power
and
>=80 Confidence
QRFS
Sentinel – all Sentinel
less than business
criteria
Gas Wells
Sentinel QRFS
Planned Burns
Legend
PQ Easement Buffered Area
27. BOM
JTWC
BOM
Cyclone
Workflows
BOM Responsible Area FIJI Responsible Area
TL
Early
Warning
Entered
Australian
Region
Cyclone
Preparedness
Cyclone
Monitor
Stage 1 & 2
Active
Impact
Post Event
Damaging
Destructive
Very
Destructive
Wind Areas
Asset Wind Rating Analysis with Wind Areas
FIJI
4
Predictive
Analysis
4
4
TL
31. Summary
FME Server
Average 2,500 jobs/day
Max 3,500 jobs/day addition due to emergency
event
Emergency Workflows = 50% of scheduled load
Service
Now
Emai
l
SM
S
Notifications
sent
3,500 past year
Mapping
Products
8,000 past year
Faults
900 past year
Workflows
52 In total
What was days of effort now typically down to hours per event
34. EU INSPIRE: Related Themes
•Addresses
•Administrative Areas
•Networks: Pipelines, electricity
•Transportation
•Hydrography
•Elevation
•Natural Risk Zones
•Environmental monitoring
•Land use
•Land cover
•Ortho imagery
•Buildings
Data Integration Example
35. Integration: EU INSPIRE
• Common, open data standard (complex schema)
• Multi-domain, multi-agency, & trans-national
• OGC open standards based data and services
• OGC formats: GML, WFS, WMS, WCS, GeoTIFF,
NetCDF
37. Satellite and UAV Remote Sensing:
Common Tasks
Workflows and requirements for Satellite and UAV
● Sensor platforms
● Raster data types
● Selection, conversion
● Enhancement, classification
● Preparation, enrichment, distribution
● UAV specific workflows
40. Satellite Imagery: Cloud Sources
Landsat (NASA/USGS) Sentinel 2 (ESA) Planet UrtheCast (Future)
Resolution 15/30/100m 10/20/60m 3-5m* 0.5 - 5 m
Status Free Free Paid Paid
Frequency Monthly Weekly (or better) Daily A few times a day
42. Example: Landsat and Sentinel Selector
● User selects location
of interest, platform
and cloud cover
● Returns list of
available tiles that
meet criteria
● User accesses results
on Amazon S3
44. White Rock Pier Storm Dec 20, 2018
● One of the worst storms in BC
Hydro history
● JRCC coordinated rescue from
collapsed pier (CAF, RCMSAR)
https://www.peacearchnews.com/news/white-rock-pier-damaged-by-storm/
45. Damage Assessment - WR Pier: Open Drone Map GeoTIFF
UAV survey JPGs to mosaicked, georeferenced GeoTIFF orthophoto using FME & ODM docker
46. Damage Assessment - WR Pier: Open Drone Map LAS
UAV survey JPGs to georeferenced 3D LAS point cloud using FME & ODM docker
47. OpenDroneMapCaller, ODMMosaicker
Global Medic use case: problem
processing 500 images offline
FME OpenDroneMapCaller from
FMEHub (FME calls ODM on docker)
● Compute approximate centroid
of each tile from exif tags
● Copy source tiles into folder by
tile name
● Call ODM on docker for each
folder to generate tile GeoTIFF
● Use FME to mosaic
georeferenced ODM tiles into
one large mosaic
Group source images by exif location
51. Integrating AI to Support SAR
Intelligence Automation
● Data enrichment to support search and rescue (SAR) operations
● MS Cognitive Services API - ID target type with confidence metrics
● FME allows for easy integration with third party systems - in this
case via REST (see FMEHub for other web connections)
● Originally presented at 2019 World Maritime Rescue Congress
FME Workflow:
● automates the feed of UAV imagery to Cognitive Services API
● filters results and compiles reports showing targets of interest,
with levels of confidence
52. AI: Vision Analysis - MS Cognitive Services API
FME Workspace reads directory of images and calls REST API (HTTPCaller) for each
58. CUSTOMER STORY
“FME Server plays an important role in
scheduling processes, publishing data,
and process monitoring.”
- Yann Rebois, ICRC
PROJECT
Integrate disparate datasets from internal
departments and external sources, both
globally and locally.
SOLUTION
Implemented FME within existing GIS
architecture to enable data integration,
management, and automation.
RESULTS
● Automate and simplify complex
workflows.
● Feed data to Tableau & ArcGIS.
● Publish data to web applications via
MapBox, CartoDB & PDF reports.
ICRC
INTERNATIONAL COMMITTEE OF THE RED CROSS
>
59. “... to protect the lives and dignity of
victims of war and internal violence and
to provide them with assistance.”
60. “GIS officers are based worldwide, need to
get information on a daily basis, and be able
to use processes without having any FME
technical knowledge.”
– Régis Longchamp, INSER
61. ● Data management
○ Quantity
○ Quality
○ Heterogeneity
○ Applications
● Security policies
● Global scope
Challenges
62. 6 Jobs for FME
at ICRC
1. Integrating source data
2. Data cleaning
3. Simplifying geometry
4. Site reporting
5. Supplying data to Tableau
6. Gazetteer – Population Kiosk
63. #1 Integrating
Source Data
Integrating source data from
3rd-party web services etc.
Examples:
1. Healthsite.io Free & open
healthcare location data.
2. ACLED – armed conflict
location & event data.
64. #2 Data Cleaning
Mixed data needs to be
harmonized and cleaned.
Types of cleaning done by FME:
● ArcSDEGridSapper
● Self-intersection
● Minimum area
● OGC tests
● ArcPy repair
65. #3 Simplifying
Geometry
A single SDE database is
behind:
● large scale paper maps
● cross-border maps
● web applications
● Importing to Tableau
66. #4 Site Reports
Essential for safety and
security of ICRC teams
● Premises Management web app
● PDF creation for site reports
(offline, mobile)
● Raster backgrounds from Google
or Esri services, where available
67. #5 Supplying Data
to Tableau
Tableau used for real time
analysis of operations and
programs.
● Preparing spatial data for Tableau
● Joining business data to spatial data
68. #6 Population Kiosk
Reference site for risks to local
populations: disease, conflict
Helps answers questions like:
1. What is the affected
population?
2. In what villages have abuses
been committed?
69. Geospatial Environment
FME’s role & key results:
● Integrate diverse range of
data sources, including
non-GIS (facilities)
● Feed data to Tableau &
ArcGIS
● Publish data to web via
MapBox, CartoDB & PDF
70. Key transformers used by Régis of
INSER in this work
1. HTTPCaller
2. JSONFlattener
3. PythonCaller
4. WorkspaceRunner & FMEServerJobSubmitter
5. CSMapReprojector
71. Benefits of FME to ICRC
✓ Flexibility
○ Extract information from any data source as needed
✓ Workflow maintenance
○ Easier than maintaining many python scripts
✓ Versatility
○ Feed other systems for additional analysis and publication
“Thank you @inser team and Safe Software for the Grant Program!”
-- Yann Rebois ICRC
74. ● Britain’s mapping service for government, businesses, and citizens.
● Geospatial data serves the national interest by enabling a safe, healthy
and prosperous society.
● Vision: to ensure Britain can build a world-leading digital and
connected economy of the future.
75. Ordnance Survey: Objectives and Challenges
Objectives:
● Provide customers with more detail about Britain’s
national landscape and building types
● Automatically detect roof type using deep learning
● Improve public safety, planning (energy, 5G),
building management and 3D
Challenges:
● Better management of large data volumes
● 20,000 updates / day to 500 million geospatial
feature database
76. Solution
1. Building footprints of all structures in Great Britain
2. Use crowdsourcing and deep learning to classify roof types
3. Add attributes + clip buildings
4. FME adds roof type attributes to polygons and clips buildings from raster satellite
imagery
Key was to quickly create different patch types, test lots of data for the patches and
process patches to the cloud
81. RESULTS
● Results on three classes:
(hipped, gabled, flat)
● 90% accuracy
● Trained on the
geographically diverse
data set
82. “We could have created a python script, but it was quicker
and simpler to utilize FME, and saved us money on
processing costs. ”
-- Charis Doidge, Ordnance Survey
“FME was quick to set up and use, which was handy
when we had several tests we wanted to run on the
patch types.”
88. Indoor Mapping
● Great opportunity
○ bring the blue dot inside
● Important area for public safety
○ responders and civilians
● Massive challenge
○ volume of buildings
○ updates
89. Indoor Mapping Challenges
● Integrate multiple sources to produce an indoor map.
○ GeoJSON, Revit, IFC, CAD (Autodesk, Bentley), Civil 3D, Esri Geodatabase, databases,
CityGML …
● Must clean and transform inconsistent data: schema and geometry.
● Must comply with indoor format specifications, e.g. IMDF, HERE, ArcGIS
Indoors, IndoorGML.
○ Strict data models and explicit spatial relationships.
● Venues constantly change => automatic updates
● Logical vs physical areas - security zones
● Public safety - hazard, damaged areas, wall and door materials, real time
90. Production: Build Indoor Mapping Datasets Using FME
● Convert floor plans and ancillary data
into indoor mapping formats.
● Validate against specifications to ensure
data meets standards.
● No coding involved. FME workflows are
created using a visual interface.
91. Indoor Mapping: OGC Indoor GML Pilot
● OGC Indoor Pilot sponsored by NIST, Dept of Commerce
● Goal: LIDAR scans -> Indoor Mapping and Navigation
● Responsible for the navigation modeller component
● Consume CityGML Public Safety (PS) ADE and produce
IndoorGML PS extension
● Project presented by OGC at recent Dept of Commerce
conference
95. CUSTOMER STORY
Improving operations and passenger
experience to become a top airport in
Europe.
PROJECT
Unify data across departments into a
central GIS database.
SOLUTION
Integrated asset and infrastructure data,
enhanced it, validated it, and distributed it
across teams.
RESULTS
● Synchronized GIS database.
● Open APIs available to developers.
● Foundation for building digital twin.
● Indoor mapping in Apple Maps (IMDF)
and a custom augmented reality app.
AMS
Amsterdam Airport Schiphol
>
97. ● Converting data for indoor mapping can be a challenge.
● CAD standards help, but more feature info is needed.
● Big win going between standards, e.g. TRIRIGA, BIM to IMDF;
CityGML to OGC IndoorGML
● Leverage existing tools, e.g. import for ArcGIS Indoor
● Build a multi-step workflow, enriching indoor data at each
step.
● Extend internal data model based on indoor requirements
(e.g. doors)
Lessons Learned
98. CUSTOMER STORY
“FME is the cherry on top the ice cream sundae
that helps bring data together and customize
it for your needs.”
- David Runneals, Iowa DOT
PROJECT
Provide road conditions and plow
information to the public.
SOLUTION
Use FME to integrate plow locations,
plow cams, and road conditions.
RESULTS
● Data is retrieved 1x/min from AWS
to populate Oracle database.
● FME automatically delivers a KML
file to Windows Azure for AGOL.
● API delivers the data to local TV
stations for their on-air software.
IOWA DOT >
99. ● Automation and data integration => scalability
● Rule / AI based filtering and analysis
● Public safety / disaster response - similar
patterns across defense, security sectors
● Need for a dynamic, flexible platform that
enables rapid innovation
● Enterprise wide service integration to span
silos
CHALLENGE: INTELLIGENCE BASED DECISIONS
SOLUTION: DATA INTEGRATION
100. FREE RESOURCES
WE’RE ALL ABOUT
safe.com
Getting Started
Online Courses
On Demand
Live chat >
Tutorials and webinars >
Instructor led >
Live and hands-on >
Video courses & demos >
Knowledge Base >
"FME sets the
standard for
support and is the
leader by far".
- Brad
Very helpful and
went above and
beyond to help
find a solution to
my request! A+"
- Justin
"Thank you!
Answered all my
questions - AGAIN.
You folks always
seem to have a
way of doing that."
- Ray
101. Thank you!
Any questions?
You can reach us at:
● @SafeSoftware on Twitter
● Live chat and free trial at safe.com
● Post on knowledge.safe.com