By Stephen C. Philips IT Innovation GRAVITATE participating @ Understanding Europe – Promoting the European Public and Cultural Space

1. Geometric and Semantic
Matching for Cultural Heritage
Artefacts
Stephen C. Phillips
IT Innovation

2. https://www.flickr.com/photos/lunamodule/
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016
2

3. © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016
3

4. © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016
4

5. © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016
5

6. Front Back
Faded
Cracked
Eroded
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016

7. British Museum, London Cyprus Museum, Cyprus
Ashmolean Museum, Oxford Fitzwilliam Museum, Cambridge
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016

8. Jigsaws
in 3D
most pieces missing
all the edges worn off
many puzzles jumbled together
pieces spread across many countries
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016

9. Re-Assembly
Re-Unification
Re-Association
_______________
Similarity Search
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016
9

10. Re-Assembly
You have some (eroded) pieces.
The computer fits them together.
This is hard in 2 dimensions, never mind in 3D!
Localised geometry
3D descriptors
Localised detail
Colour descriptors
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016
10

11. Re-Unification
Head in the
British Museum
Statue in the
Cyprus Museum
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016
11
 Re-unify 3D models of the statue
 3D print copies for re-unification
 Re-unify statue in a virtual
museum

12. Re-Association
Artefact with semantic description
Material, decoration, size, shape, period, style, glaze, …
Data in another museum catalogue
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016
12

13. Re-Association
Find similarities leading to
new insights about past cultures
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016
13

14. How?
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016
14

15. What’s the data?
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016
15

16. © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016
16
Fragments of Terracotta Statues from Salamis

17. © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016
17
We need to Gather all the Knowledge
Catalogue data
with text
descriptions
Archaeological
papers
Excavation
notes
Chemical
analysis
X-ray
fluorescence
3D scanning
 models

18. © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016
18
We need to Gather all the Knowledge
Catalogue data
with text
descriptions
Archaeological
papers
Excavation
notes
Chemical
analysis
X-ray
fluorescence
3D scanning
 models

19. © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016
19
We need to Gather all the Knowledge
Catalogue data
with text
descriptions
Archaeological
papers
Excavation
notes
The British Museum has 2.5M objects
described in CIDOC CRM
Catalogue data from Ashmolean, Fitzwilliam
and Cyprus museums is now mapped to the
same data model
Object  hasNote 
“…lots of free-form text added by the curator”
Using Natural Language Processing to extract
and encode meaning from this text:
• References to papers, to catalogue entries
• Parts and features
• Conservation condition
• Measurements
• …
 CIDOC CRM / CRMarchaeo / etc

20. © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016
20
We need to Gather all the Knowledge
Chemical
analysis
X-ray
fluorescence
3D scanning
 models
Volume, area
thickness,
curvature, …
Colour, texture,
distance from
convex hull, …
Feature
detection
Faceting:
front / back
/ fracture
Part
annotation
CRMdig
CH Artefact
Partonomy
All fragments (211)
are scanned, in a
variety of resolutions
and with a variety of
scanners

21. Faceting
 Assume local fold
model
 Measure local
fold angle
 Select locally salient
angles
 Connect to form facet
curves
 Determine the facet
types
 Robust for our
fragments
 Very few parameters
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016
21
90ᵒ

22. © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016
22
Feature Detection
 Morphological and
stylistic feature
detection and
characterization
 Reasoning on similarity
among fragments
−1 −0.5 0 0.5 1
−0.4
−0.2
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
x
y

23. © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016
23
Using CH Artefact Partonomy within CIDOC-CRM

24. How do I use it?
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016
24

25. © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016
25
Six views in the dashboard
 Inspection
 Users can inspect 3D fragments and metadata and run a search
 Fragment
 3D assets can be processed (faceting, geometric characterisation,
feature identification) and annotated
 ReAssembly
 Exploration
 Datasets can be explored according to specific selected properties
 History
 To preserve the list of operations performed in the session
 Clipboard
 To save all the data the user is interested in (e.g. notes, models,
annotation)

26. © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016
26
Six views in the dashboard
 Inspection
 Users can inspect 3D fragments and metadata and run a search
 Fragment
 3D assets can be processed (faceting, geometric characterisation,
feature identification) and annotated
 ReAssembly
 Exploration
 Datasets can be explored according to specific selected properties
 History
 To preserve the list of operations performed in the session
 Clipboard
 To save all the data the user is interested in (e.g. notes, models,
annotation)

27. © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016
27
Six views in the dashboard
 Inspection
 Users can inspect 3D fragments and metadata and run a search
 Fragment
 3D assets can be processed (faceting, geometric characterisation,
feature identification) and annotated
 ReAssembly
 Exploration
 Datasets can be explored according to specific selected properties
 History
 To preserve the list of operations performed in the session
 Clipboard
 To save all the data the user is interested in (e.g. notes, models,
annotation)

28. © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016
28
Six views in the dashboard
 Inspection
 Users can inspect 3D fragments and metadata and run a search
 Fragment
 3D assets can be processed (faceting, geometric characterisation,
feature identification) and annotated
 ReAssembly
 Exploration
 Datasets can be explored according to specific selected properties
 History
 To preserve the list of operations performed in the session
 Clipboard
 To save all the data the user is interested in (e.g. notes, models,
annotation)

29. © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016
29
Six views in the dashboard
 Inspection
 Users can inspect 3D fragments and metadata and run a search
 Fragment
 3D assets can be processed (faceting, geometric characterisation,
feature identification) and annotated
 ReAssembly
 Exploration
 Datasets can be explored according to specific selected properties
 History
 To preserve the list of operations performed in the session
 Provenance and argumentation
 Clipboard
 To save all the data the user is interested in (e.g. notes, models,
annotation)

30. © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016

31. © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016
31
Prototype

32. And Finally: ReAssembly
Use all the data to guide the process
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016
32
Selection
by
Similarity
Positioning
clues
Matching Mating
Jigsaws
in 3D
most pieces missing
all the edges worn off
many puzzles jumbled together
pieces spread across many countries

33. © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016
33
Mannequin

34. Matching: geometry-based
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016
34

35. © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016
35
Mating
 Mating digitally mimics gluing of the
fragments
 We place selected fragments in their
optimal relative position based on:
 Geometric complementarity
 Skin pattern continuity
 Semantic constraints
 Global alignment
 Mathematical morphology used
 Final approval of proposed assemblage is
requested of the user
(SotA mating from predecessor PRESIOUS)

36. GRAVITATE-PROJECT.EU
Coordinator:
Stephen C Phillips
scp@it-innovation.soton.ac.uk