Ivana Tilca and Bruno Capuano discuss lessons learned from creating a multiplatform AI project for Azure Kinect and Hololens 2. They cover topics like face detection, object recognition, preparing and training custom machine learning models, and using the Custom Vision API to build a flag detector. They demonstrate examples using Python, Unity, and WPF with the Azure Kinect. While there is no official SDK, it is possible to access the WinRT API directly from Unity scripts with some limitations.

1. GLOBAL XR
TALKS
@elbruno
http://elbruno.com
Ivana.Tilca@3xmgroup.com
@ivanatilca
Ivana Tilca
Quality Manager 3XM Group
Microsoft MVP - AI
Bruno Capuano
Innovation Lead Canada
Microsoft MVP - AI
Lessons Learned creating a
multiplatform AI project for
Azure Kinect and Hololens 2

2. Face Detection / Object Recognition
FACE
DETECTION
DETECT FACIAL
FEATURES
FACE
RECOGNITION
OBJECT
RECOGNITION

3. Face Detection / Object Recognition
FACE
DETECTION
DETECT FACIAL
FEATURES
FACE
RECOGNITION
OBJECT
RECOGNITION

4. Prepare Your Data Build & Train Run
Build your own custom machine learning
models

6. Custom Vision: Flag Detector

7. Custom Vision: Flag Detector

8. Custom Vision: Flag Detector

9. Custom Vision: Flag Detector

10. Custom Vision: Flag Detector

11. Custom Vision: Flag Detector

12. Custom Vision: Flag Detector

13. Flag Detector in Python Labs

14. Flag Detector in Unity3 with MRTK

15. Flag Detector in Unity3 with MRTK

16. And Azure Kinect ?

17. Azure Kinect – WPF – Face detection

18. 1MP depth sensor 7-mic array
12 MP RGB video camera
Accelerometer and gyroscope (IMU)
External sync pins

19. Azure Kinect – Youtube - holostream
Takashi Yoshinaga

20. Azure Kinect – Images...

22. Universal Windows Apps!
We can do this!
Universal Windows Platform: WinRT API in C# scripts
It is possible to use WinRT API directly in Unity scripts
. However there are limitations and requirements for
this:

23. The type or namespace name 'Windows'
does not exist in the namespace 'Microsoft'

24. The type or namespace name 'Media' does not
exist in the namespace 'System.Windows'

25. .Net HTTP Client does not work … 

26. No BGRA, go for JPG

27. Recap
• No official SDK
• Custom code will be required
• You will test your patience
• And …. Have fun !

28. GLOBAL XR
TALKS
@elbruno
http://elbruno.com
Ivana.Tilca@3xmgroup.com
@ivanatilca
Ivana Tilca
Quality Manager 3XM Group
Microsoft MVP - AI
Bruno Capuano
Innovation Lead Avanade
Microsoft MVP - AI
Q&A
Lessons Learned creating a
multiplatform AI project for
Azure Kinect and Hololens 2