Natural Resources Canada (NRCan) uses FME software since 1998. Initially, it was used exclusively as a translation tools. As NRCan gained experience with the FME, it began to integrate it within its cartographic production processes. Today, FME is a centerpiece for validation, integration and dissemination of spatial data from NRCan. This presentation will describe the infrastructure (feature catalog, constraints management, database, etc.) surrounding the use of FME at NRCan and it will also present some interesting validation and spatial data management solutions.

1. 1
FME within NRCan
Marie-EveMartin
Jean-MarcPrévost

2. 2
Agenda
! Context
! System architecture
! Programming standards, Quality assurance,
testing methods, etc.
! Validations processes and dependencies
! FME developments at NRCan
! Safe Software collaboration

3. 3
Agenda
! Context
! Who are we?
! Where do we come from?
! Where are we now?
! Where are we going?
! System architecture
! Programming standards, Quality assurance, testing methods, etc.
! Validations processes and dependencies
! FME developments at NRCan
! Safe Software collaboration

4. 4
Natural Resources Canada
Mission and Outcomes

5. 5
NRCan
Earth
Science
Sector
Geoma4cs
Canada
CCMEO
/
CCCOT
Energy
Sector
Canadian
Forest
Service
Direc4on
&
Coordina4on
Minerals
&
Metal
Sector
Science
&
Policy
Integra4on
Energy
Technology
&
Programs
Sector
Corporate
Management
&
Services
Sector
NRCan structure overview
CCMEO
Canada
Centre
for
Mapping
and
Earth
Observa5on
CCCOT
Centre
Canadien
de
Cartographie
et
d’
Observa5on
de
la
Terre
CCMEO
total
number
of
employees:
319
227
employees
for
Mapping
92
employees
for
Earth
Observa5on
Sherbrooke
OGawa

6. 6
Agenda
! Context
! Who are we?
! Where do we come from?
! Where are we now?
! Where are we going?
! System architecture
! Programming standards, Quality assurance, testing methods, etc.
! Validations processes and dependencies
! FME developments at NRCan
! Safe Software collaboration

7. 7
FME history at CCMEO
NTDB
specs
vectoriza4on
File
management
CanVec,
CanTopo,
GeoBase,
etc.
era
Database
management
FGP
2015
2010
2005
2000
1995
1990
1985
Transla4ons
-­‐
Workbench
-­‐
Data
manipula4on
&
valida4on
Custom
transformers
-­‐
Programming
standards
-­‐
Tes4ng
methodology
≈700

8. 8
Agenda
! Context
! Who are we?
! Where do we come from?
! Where are we now?
! Where are we going?
! System architecture
! Programming standards, Quality assurance, testing methods, etc.
! Validations processes and dependencies
! FME developments at NRCan
! Safe Software collaboration

9. 9
Distribu?on
Management
System architecture at CCMEO
Aquisi?on
Condi4onning
Partners
Data
acquisition
Others
Data
Access
QA

10. 10
What do we produce ?
! Main products :
! Canvec
! Toporama
! CDED
! 200 collections of various nature
! Statistics for last year :
! 8 500 000 downloads
! 3 000 mail and phone requests

11. 11
Where to find our products ?
! Data availability :
! Geogratis : geogratis.gc.ca
! Geobase : www.geobase.ca
! WMS services
! Licence : Open Government Licence

12. 12
Data extraction tool
http://geogratis.gc.ca/site/eng/extraction

13. 13
Agenda
! Context
! Who are we?
! Where do we come from?
! Where are we now?
! Where are we going?
! System architecture
! Programming standards, Quality assurance, testing methods, etc.
! Validations processes and dependencies
! FME developments at NRCan
! Safe Software collaboration

14. 14

15. 15
Agenda
! Context
! System architecture
! Programming standards, Quality assurance,
testing methods, etc.
! Validations processes and dependencies
! FME developments at NRCan
! Safe Software collaboration

16. 16
System architecture
FME,
Python,
.EXE,
SQL,
etc.
Windows
servers
Produc4on:
64
processes
(32bits)
Distribu4on:
56
processes
(64bits)
Oracle
11g,
PostgreSQL
9.2
ArcSDE
10.1,
PostGIS
2.0
400
TB
Instruc?on
files

17. 17
Agenda
! Context
! System architecture
! Programming standards, Quality assurance,
testing methods, etc.
! Validations processes and dependencies
! FME developments at NRCan
! Safe Software collaboration

18. 18
Worbenches – Last saved build
Programs
s4ll
in
opera4on
!

19. 19
Taking control
! 2007 problematic:
! Mix of FME versions on computers
! Everyone wanted the lasted build
! Increasing number of workbenches
! Workbenches were transferred to support team
! Solution :
! FME upgrades management
! Programming standards

20. 20
FME upgrades management
! Upgrade rules:
! Everyone use one common build
! Upgrade only to official builds
! At least one upgrade per year
(idealy no more than two)

21. 21
Programming standards

22. 22
Programming standards
! Documentation (.docx and annotations)
! Parameter naming standards
! Known bad practices review
! Unit tests
! Peer review

23. 23
Programming standards
Examples of bad practices
! Missing Rounders
! Missing Terminators

24. 24
Programming standards
Unit tests
! Each workbench is managed in a source code
control environment
! Each workbench is saved with the bat file,
sources and expected results files necessary
to test it
! Batch testing of all workbenches before every
upgrade.

25. 25
Programming standards
Unit tests
! Fail rate is used to plan upgrades
Source Destination Fail rate
2013 SP1 2013 SP4 2%
2013 SP4 2014 SP1 10%

26. 26
Programming standards
Peer review
! FME is easy to learn without taking classes
! FME always offer various way of doing
something
! Peer review allows:
! Improved shared knowledge
! Programming uniformity
! Identifying repetitive patterns

27. 27
Agenda
! Context
! System architecture
! Programming standards, Quality assurance,
testing methods, etc.
! Validations processes and dependencies
! FME developments at NRCan
! Safe Software collaboration

28. 28
Validation processes
! At CCMEO, FME is used for :
! Feature translation
! Feature creation
! Feature validation
Catalog

29. 29
Distribu?on
Management
Validation processes
Aquisi?on
Condi4onning
Partners
Data
acquisition
Others
Data
Access
QA

30. 30
Validation Catalog
! ISO-19110 compliant
XML HTMLDatabase
Users

31. 31
Catalog based validations
! Spatial relations validation
! Domain attribute validation
! Proximity validation
! Minimal dimension validation
! Segmentation validation
! Data clipping validation

32. 32
Catalog based validations
Example
Spatial relations validation
! Based on Egenhofer-Clementini DE-9IM masks

33. 33
Catalog based validations
Examples
! Spatial relations
! Attribute value

34. 34
Catalog validation within FME
ISO
19110
XML
List
friendly
format
Generic
reader
Generic
writer
Valida4on
Error
forma4ng
Valida4on
rules
extrac4on
Valida4on
Error
forma4ng
Valida4on
rules
extrac4on
One
custom
transformer
per
valida5on
concept!

35. 35
Catalog validation within FME
! Real program example:

36. 36
Agenda
! Context
! System architecture
! Programming standards, Quality assurance,
testing methods, etc.
! Validations processes and dependencies
! FME developments at NRCan
! Safe Software collaboration

37. 37
Utility customs - Examples
! TerminatorRecorder
! Record all the errors to a .ffs
file before causing the
workbench to fail
! TerminatorMinMax
! Cause the workbench to fail if
the number of feature is not
as expected

38. 38
! Matcher_CCMEO
! Match features if
start/end points
are within tolerance
! SnapToNeatline
! Snap features within tolerance without
collapsing
Utility customs - Examples

39. 39
Utility customs - Examples
! SchemaMapper_CCMEO
! Similar to SchemaMapper but :
! Allow mapping with a variable number of
conditions
! Conditions may be ranges of values
! Conditions may use any operator (>, <, != , etc)
! Conditions are validated and produce report if
badly formatted
TYPE=[11:13];PRECISION<=20;NAME=Lake

40. 40
Validation customs
Minimal dimension
Voronoi
Side
lines
Centerline
Area
Width
Length

41. 41
Validation customs
Proximity
What
are
we
looking
for?
Valida4on
Process
Op4misa4on
Process

42. 42
Agenda
! Context
! System architecture
! Programming standards, Quality assurance,
testing methods, etc.
! Validations processes and dependencies
! FME developments at NRCan
! Safe Software collaboration

43. 43
Safe Software collaboration
! Motivations for collaboration
! Simplifies our workbenches
! Faster processing
! Maintenance under Safe’s responsibility

44. 44
Safe Software collaboration
Generalization
Hydro network
AngularityCalculator
SherbendGeneralizer
StreamOrderCalculator
StreamPriorityCalculator
NetworkFlowOrientor
BoundingBoxAccumulator

45. 45
Safe Software collaboration
VoronoiDiagrammer

46. 46
QUESTIONS ?!?
Marie-Eve.Martin@RNCan-NRCan.gc.ca
Jean-Marc.Prevost@RNCan-NRCan.gc.ca

47. 47
Amalgamator

48. 48
ConvexityFilter

49. 49
Generalizer
Wang
The
Wang
algorithm
will
itera5vely
combine,
eliminate
and
exaggerate
bends
un5l
the
input
line
feature
has
no
bend
that
is
smaller
than
the
given
tolerance
value.
McMaster
/
McMaster
Weighted
Distance
The
McMaster
algorithm
calculates
a
new
loca5on
for
each
point
by
first
taking
the
average
value
of
the
x
and
y
coordinates
of
the
point
and
a
number
of
neighboring
points.
It
then
slides
the
averaged
point
towards
the
original
point
according
to
a
specified
displacement
value.
The
overall
effect
is
that
each
point
will
be
pulled
towards
its
neighboring
points.
NURBfit
(B-­‐Spline)
The
NURBfit
algorithm
will
fit
lines
using
B-­‐Spline
curves
of
given
polynomial
degree.
The
resul5ng
lines
will
follow
these
curves
with
given
segment
length.
The
higher
the
degree,
the
smoother
the
line.
An
example
of
usage
is
smoothing
contour
lines
in
order
to
remove
sparks
and
simulate
the
work
of
a
cartographic
cra[sman.
Inflec?on
Points
The
Inflec5on
algorithm
will
calculate
the
loca5on
of
the
inflec5on
points
along
a
line
and
return
the
list
of
these
points.
Inflec5on
points
are
measures
of
the
sinuosity
of
a
line.

50. 50
SherbendGeneralizer
SherbendGeneralizer

51. 51
NetworkFlowOrientor
NetworkFlowOrientor

52. 52
StreamPriorityCalculator
StreamPriorityCalculator

53. 53
Densifier

54. 54
Chopper

55. 55
Chopper use case scenario
chopper
dissolve
chopper chopper
dissolve
dissolve
Data
extraction
Group by metadata No group by