The update report on Benchmarking framework of vision-based spatial registration and tracking methods for MAR (ISO/IEC CD 18520) was presented in ISO IEC/JTC 1/SC 24 meetings (2017/8/7-8).

1. Benchmarking framework of
vision-based spatial registration and
tracking methods for MAR
(ISO/IEC CD 18520)
Takeshi Kurata
AIST, Japan
ISO IEC/JTC 1/SC 24 (2017/8/7-8)

2. Contents before CD ballot
• Main Body
– Terms and Definitions
– Benchmarking processes
– Benchmark indicators
– Trial set for benchmarking
– Conformance
2
• Annex A: Benchmarking organizations and activities
• Annex B: Tracking competitions in ISMAR
• Annex C: Conceptual relationship between this document and
other benchmarking standards

3. Contents after CD ballot
• Main Body (Remove proper/individual names, from
academic paper style to specification style, include self-
benchmarking more)
– Terms and Definitions
– Benchmarking processes (More narrative descriptions)
– Benchmark indicators
– Trial set for benchmarking
– Conformance (More specific: conformance example with check
sheets)
3
• Annex A: Use case examples (More compact)
• Annex B: Conceptual relationship between this document and
other benchmarking standards (More narrative descriptions)

4. Benchmarking framework
vSRT: Vision-based spatial registration and tracking

5. Example of stakeholders and their roles

6. Benchmark indicators
vSRT: Vision-based spatial registration and tracking

7. Benchmark indicators
PEVO: Projection error of virtual objects, which is the most direct and intuitive indicator for
vSRT methods for MAR
NOTE: Indicators for off-site benchmarking can also be used for on-site benchmarking.

8. Benchmark indicators
PEVO: Projection error of virtual objects, which is the most direct and intuitive indicator for
vSRT methods for MAR
NOTE: Indicators for off-site benchmarking can also be used for on-site benchmarking.
ISMAR 2015 Tracking competition

9. Trial set for benchmarking
vSRT: Vision-based spatial registration and tracking

10. Trial set for benchmarking

11. Trial set for benchmarking
TrakMark

12. Trial set for benchmarking
Metaio

13. Trial set for benchmarking
The City of Sights:
An Augmented Reality Stage Set

14. Trial set for benchmarking
ISMAR 2015 Tracking competition

15. Trial set for benchmarking
ISMAR 2014 Tracking
competition

16. Trial set for benchmarking
ISMAR 2015 Tracking competition

17. Meetings after SC 24 meetings in Beijing
• Editing meetings in WG 9
– 2017/01/18-19 (WG 9 in Seoul), 2017/02/09
• Drafting meetings in WG 9 Japanese subcommittee
– 2016/10/18, 2016/11/16, 2016/12/26, 2017/03/08,
2017/06/20, 2017/07/19
– Members
• T. Kurata (AIST/Univ. of Tsukuba)
• M. Aono (Toyohashi Univ. of Tech.)
• T. Kondo (The Open Univ. of Japan)
• F. Shibata (Ritsumeikan Univ.)
• T. Taketomi (NAIST)
• H. Uchiyama (Kyushu Univ.)
• S. Mori (Keio Univ./Graz University of Technology)
• K. Makita (Canon/AIST) (Expert)
17

18. Next Step: 40.00: DIS registered
18

19. TODO for DIS registration
• More narrative
• Make conformance check sheets
• Compact annexes
• Target dates
• Due dates
19
DIS FDIS IS
9/17 1/18 7/18
DIS FDIS IS
12/17 6/18 12/18

20. Conformance
check sheet
(tentative)
Process
Target (T)/
Input (I)/
Output (O)/
Organized
storage (S)
Reliability
Temporality
Variety
Contents
[ ] Image sequences: ____________________________
[ ] Intrinsic/extrinsic camera parameters: ___________
[ ] Challenge points: _____________________________
[ ] Optional contents: ____________________________
Metadata
[ ] Scenario: ____________________________________
[ ] Camera motion type: __________________________
[ ] Camera configuration: _________________________
[ ] Image quality: ________________________________
Contents [ ] Physical objects: ______________________________
Metadata [ ] How to find the physical objects: ________________
Trial set
format
Dataset
Physical
object
Process
flow
[ ] develop vSRT methods and/or MAR systems: ______________________
[ ] gather vSRT methods and/or MAR systems: _______________________
[ ] prepare and conduct benchmarking: ______________________________
[ ] provide and maintain benchmarking instruments: ___________________
[ ] provide and maintain benchmarking repositories: ___________________
[ ] share benchmarking results: _____________________________________
[ ] vSRT method: _________________________________________________
[ ] MAR system: __________________________________________________
[ ] trial sets and physical objects: ___________________________________
[ ] benchmarking instruments: ______________________________________
[ ] benchmarking results: __________________________________________
[ ] benchmarking surveys: _________________________________________
[ ] benchmarking repository: ________________________________________
[ ] external repositories: ____________________________________________
Indicator
formura
[ ] PEVO: ________________________________________________________
[ ] Reprojection error of image features: _____________________________
[ ] Position and posture errors of a camera: __________________________
[ ] Completeness of a trial: _________________________________________
[ ] Throughput: ___________________________________________________
[ ] Latency: ______________________________________________________
[ ] Time for trial completion: ________________________________________
[ ] Number of datasets/trials: ________________________________________
[ ] Variety on properties of datasets/trials: _____________________________