Se ha denunciado esta presentación.
https://buildAR.com - image credit
Web Standards for AR
An intro to the latest developments
on the Augmented Web Platform
Who's making these outlandish claims?
Rob Manson @nambor
CEO of MOB-labs the creators of buildAR.com
Chair of the W3C's Au...
What is the Augmented Web?
https://buildAR.com
It's what comes after HTML5
https://buildAR.com
https://buildAR.com - image credit
Key Concept:
1992-2000
WWW: Page based websites
key innovation: HREF
2000-2013
Web 2.0: AJAX driven social services
key in...
https://buildAR.com - image credit
https://buildAR.com
Key Concept:
https://buildAR.com
JSARToolkit demo
https://buildAR.com
FAST algorithm feature detection demo
Test your own device at: theAWEsomeWEB.com
https://buildAR.com
https://buildAR.com - image credit
First lets look at the broad
Stream processing pipeline concept
https://buildAR.com
Key Concept:
https://buildAR.com
So what the hell is a
Stream processing pipeline?
https://buildAR.com
The vision we proposed in 2010 is now here!
ARStanda...
https://buildAR.com
Stream processing pipelines
1. Get Stream
2. Connect to a Stream pipeline
A way to connect a Stream to...
Now lets look at the more specific
MediaStream processing pipeline
https://buildAR.com
https://buildAR.com
MediaStream processing pipelines
1. Get MediaStream
MediaStream from getUserMedia or RTCPeerConnection...
https://buildAR.com
This is where the APIs are still evolving
1. Get MediaStream
MediaStream from getUserMedia or RTCPeerC...
https://buildAR.com
This is where the Augmented Web comes to life
1. Get MediaStream
MediaStream from getUserMedia or RTCP...
https://buildAR.com
This is where the fun is!
1. Get MediaStream
MediaStream from getUserMedia or RTCPeerConnection
2. Con...
webcamtoy.com
https://buildAR.com
https://buildAR.com - image credit
Key Concept:
#$!%@#%@#$@$@!!!!!
https://buildAR.com
https://buildAR.com
Browser's have always had a “binary barrier”
The Video element
stream -> <video>
https://buildAR.com
Now we've broken through the “binary barrier”
The Video/Canvas pipeline
stream -> <video> -> <canvas> ...
https://buildAR.com
Now we've broken through the “binary barrier”
The Video/Canvas pipeline
stream -> <video> -> <canvas> ...
https://buildAR.com
Now we've broken through the “binary barrier”
The Video/Canvas pipeline
stream -> <video> -> <canvas> ...
https://buildAR.com
Now we've broken through the “binary barrier”
The Video/Canvas pipeline
stream -> <video> -> <canvas> ...
Now lets take a closer look at the
Video/Canvas MediaStream pipeline
https://buildAR.com
https://buildAR.com
The Video/Canvas MediaStream pipeline
Access the camera (or some other stream) source
a. getUserMedia(...
https://buildAR.com
The Video/Canvas MediaStream pipeline
Follow along on github: github.com/buildar/getting_started_with_...
Now lets compare that to the
WebGLSL based Video/Shader pipeline
https://buildAR.com
https://buildAR.com
The Video/Shader MediaStream pipeline
Access the camera (or some other stream) source
a. getUserMedia(...
https://buildAR.com - image credit
Key Concept:
ArrayBuffers and minimising copies
are the key to an efficient walk
https://buildAR.com
https://buildAR.com
Views are moveable windows
into the raw data streams
1. Get stream
MediaStream from getUserMedia or RT...
https://buildAR.com - image credit
Key Summary:
Where is the Augmented Web
up to right now?
https://buildAR.com
https://buildAR.com
Mobile AR vs. the Augmented Web
Mobile AR at end of 2009 Augmented Web at end of 2013
https://buildAR.com
Mobile AR vs. the Augmented Web
Apple had not yet opened up
camera access on iOS through
objective-c.
...
https://buildAR.com
Mobile AR vs. the Augmented Web
Apple had not yet opened up
camera access on iOS through
objective-c.
...
https://buildAR.com
Mobile AR vs. the Augmented Web
Apple had not yet opened up
camera access on iOS through
objective-c.
...
https://buildAR.com
Mobile AR vs. the Augmented Web
Apple had not yet opened up
camera access on iOS through
objective-c.
...
https://buildAR.com
Mobile AR vs. the Augmented Web
Apple had not yet opened up
camera access on iOS through
objective-c.
...
https://buildAR.com
Mobile AR vs. the Augmented Web
Apple had not yet opened up
camera access on iOS through
objective-c.
...
https://buildAR.com
Mobile AR vs. the Augmented Web
Apple had not yet opened up
camera access on iOS through
objective-c.
...
https://buildAR.com
What's in the near future?
Integrating WebRTC and Visual Search
Using WebGL/GLSL to utilise GPU parall...
https://buildAR.com
What's in the near future?
Integrating WebRTC and Visual Search
Using WebGL/GLSL to utilise GPU parall...
https://buildAR.com
What's in the near future?
Integrating WebRTC and Visual Search
Using WebGL/GLSL to utilise GPU parall...
https://buildAR.com
What's in the near future?
Integrating WebRTC and Visual Search
Using WebGL/GLSL to utilise GPU parall...
https://buildAR.com
Join the W3C's
Augmented Web Community Group
http://w3.org/community/ar
https://buildAR.com - image credit
Web Standards for AR
An intro to the latest developments
on the Augmented Web Platform
Próxima SlideShare
Cargando en…5
×

Web Standards for AR workshop at ISMAR13

4.078 visualizaciones

Publicado el

This work was presented at the Open Standards session at the IEEE ISMAR 2013 event. It provides a detailed overview and working examples that show exactly where Augmented Reality and Computer Vision are up to on the Web Platform.

This presentation also provides a detailed description of how to define exactly what the Augmented Web is.

Publicado en: Tecnología, Empresariales
  • Sé el primero en comentar

Web Standards for AR workshop at ISMAR13

  1. 1. https://buildAR.com - image credit Web Standards for AR An intro to the latest developments on the Augmented Web Platform
  2. 2. Who's making these outlandish claims? Rob Manson @nambor CEO of MOB-labs the creators of buildAR.com Chair of the W3C's Augmented Web Community Group Invited Expert with the ISO, W3C & the Khronos Group Author of “Getting started with WebRTC” - (Packt Publishing) https://buildAR.com
  3. 3. What is the Augmented Web? https://buildAR.com
  4. 4. It's what comes after HTML5 https://buildAR.com
  5. 5. https://buildAR.com - image credit
  6. 6. Key Concept: 1992-2000 WWW: Page based websites key innovation: HREF 2000-2013 Web 2.0: AJAX driven social services key innovation: XHR 2013-... AWE: Sensor driven real-time web of things key innovation: ArrayBuffer https://buildAR.com
  7. 7. https://buildAR.com - image credit
  8. 8. https://buildAR.com Key Concept:
  9. 9. https://buildAR.com JSARToolkit demo
  10. 10. https://buildAR.com FAST algorithm feature detection demo
  11. 11. Test your own device at: theAWEsomeWEB.com https://buildAR.com
  12. 12. https://buildAR.com - image credit
  13. 13. First lets look at the broad Stream processing pipeline concept https://buildAR.com
  14. 14. Key Concept: https://buildAR.com
  15. 15. So what the hell is a Stream processing pipeline? https://buildAR.com The vision we proposed in 2010 is now here! ARStandards Workshop in Seoul 2010 – Rob Manson
  16. 16. https://buildAR.com Stream processing pipelines 1. Get Stream 2. Connect to a Stream pipeline A way to connect a Stream to an ArrayBuffer 3. Get ArrayBuffer 5. Populate a scene Array with Array Buffer Views 6. Move through Views to process data 7. Output events and metadata 8. Update UI and/or send requests
  17. 17. Now lets look at the more specific MediaStream processing pipeline https://buildAR.com
  18. 18. https://buildAR.com MediaStream processing pipelines 1. Get MediaStream MediaStream from getUserMedia or RTCPeerConnection 2. Connect to a MediaStream pipeline Canvas/Video, Image Capture, Recording, ScriptNodePorcessor or Video/Shader pipelines 3. Get ArrayBuffer new ArrayBuffer() 5. Populate a scene Array with Array Buffer Views var scene = [new Uint8ClampedArray(buffer), ...] 6. Move through Views to process data for each (var view in scene) { …process(view)... } 7. Output events and metadata return results 8. Update UI and/or send requests Wave hands inspriationally here!
  19. 19. https://buildAR.com This is where the APIs are still evolving 1. Get MediaStream MediaStream from getUserMedia or RTCPeerConnection 2. Connect to a MediaStream pipeline Canvas/Video, Image Capture, Recording, ScriptNodePorcessor or Video/Shader pipelines 3. Get ArrayBuffer new ArrayBuffer() 6. Move through Views to process data for each (var view in scene) { …process(view)... } 7. Output events and metadata return results 8. Update UI and/or send requests Wave hands inspriationally here! 5. Populate a scene Array with Array Buffer Views var scene = [new Uint8ClampedArray(buffer), ...]
  20. 20. https://buildAR.com This is where the Augmented Web comes to life 1. Get MediaStream MediaStream from getUserMedia or RTCPeerConnection 2. Connect to a MediaStream pipeline Canvas/Video, Image Capture, Recording, ScriptNodePorcessor or Video/Shader pipelines 3. Get ArrayBuffer new ArrayBuffer() 6. Move through Views to process data for each (var view in scene) { …process(view)... } 7. Output events and metadata return results 8. Update UI and/or send requests Wave hands inspriationally here! 5. Populate a scene Array with Array Buffer Views var scene = [new Uint8ClampedArray(buffer), ...]
  21. 21. https://buildAR.com This is where the fun is! 1. Get MediaStream MediaStream from getUserMedia or RTCPeerConnection 2. Connect to a MediaStream pipeline Canvas/Video, Image Capture, Recording, ScriptNodePorcessor or Video/Shader pipelines 3. Get ArrayBuffer new ArrayBuffer() 6. Move through Views to process data for each (var view in scene) { …process(view)... } 7. Output events and metadata return results 8. Update UI and/or send requests Wave hands inspriationally here! 5. Populate a scene Array with Array Buffer Views var scene = [new Uint8ClampedArray(buffer), ...]
  22. 22. webcamtoy.com https://buildAR.com
  23. 23. https://buildAR.com - image credit
  24. 24. Key Concept: #$!%@#%@#$@$@!!!!! https://buildAR.com
  25. 25. https://buildAR.com Browser's have always had a “binary barrier” The Video element stream -> <video>
  26. 26. https://buildAR.com Now we've broken through the “binary barrier” The Video/Canvas pipeline stream -> <video> -> <canvas> -> image data -> array buffer -> process -> output
  27. 27. https://buildAR.com Now we've broken through the “binary barrier” The Video/Canvas pipeline stream -> <video> -> <canvas> -> image data -> array buffer -> process -> output The Mediastream Image Capture API pipeline stream -> track -> image capture -> image data -> array buffer -> process -> output
  28. 28. https://buildAR.com Now we've broken through the “binary barrier” The Video/Canvas pipeline stream -> <video> -> <canvas> -> image data -> array buffer -> process -> output The Mediastream Image Capture API pipeline stream -> track -> image capture -> image data -> array buffer -> process -> output The MediaStream Recording API pipeline stream -> recorder -> blob -> file reader -> array buffer -> process -> output
  29. 29. https://buildAR.com Now we've broken through the “binary barrier” The Video/Canvas pipeline stream -> <video> -> <canvas> -> image data -> array buffer -> process -> output The Mediastream Image Capture API pipeline stream -> track -> image capture -> image data -> array buffer -> process -> output The MediaStream Recording API pipeline stream -> recorder -> blob -> file reader -> array buffer -> process -> output Plus other non-MediaStream pipelines: ????? -> Web Sockets/XHR/File/JS -> array buffer -> process -> output
  30. 30. Now lets take a closer look at the Video/Canvas MediaStream pipeline https://buildAR.com
  31. 31. https://buildAR.com The Video/Canvas MediaStream pipeline Access the camera (or some other stream) source a. getUserMedia() Setup <video> element in the DOM a. declaratively then via getElementById or similar b. createElement(“video”) (no need to appendChild()) Pipe camera stream into <video> a. video.src = stream Setup <canvas> element in the DOM a. declaratively then via getElementById or similar b. createElement(“canvas”) then appendChild() Get 2D drawing context a. canvas.getContext('2d'); Draw <video> frame onto <canvas> a. canvas.drawImage(video, top, left, width, height); Get RGBA Uint8ClampedArray of the pixels a. context.getImageData(top, left, width, height).data.buffer; Burn CPU (not GPU) cycles a. fore each(var obj in view) { … } NOTE: Integrate other streams & sensor data here Render results a. using HTML/JS/CSS b. using another <canvas> and drawImage() c. using WebGL d. a combination of all 1. Get MediaStream MediaStream from getUserMedia or RTCPeerConnection 2. Connect to a MediaStream pipeline Canvas/Video, Image Capture or Recording pipelines 3. Get ArrayBuffer new ArrayBuffer() 5. Populate a scene with Array Buffer Views var scene = [new Uint8ClampedArray(buffer), ...] 6. Move through Views to process data for each (var view in scene) { …process(view)... } 7. Output events and metadata return results 8. Update UI and/or send requests Wave hands inspriationally here! Follow along on github: github.com/buildar/getting_started_with_webrtc
  32. 32. https://buildAR.com The Video/Canvas MediaStream pipeline Follow along on github: github.com/buildar/getting_started_with_webrtc Access the camera (or some other stream) source a. getUserMedia() Setup <video> element in the DOM a. declaratively then via getElementById or similar b. createElement(“video”) (no need to appendChild()) Pipe camera stream into <video> a. video.src = stream Setup <canvas> element in the DOM a. declaratively then via getElementById or similar b. createElement(“canvas”) then appendChild() Get 2D drawing context a. canvas.getContext('2d'); Draw <video> frame onto <canvas> a. canvas.drawImage(video, top, left, width, height); Get RGBA Uint8ClampedArray of the pixels a. context.getImageData(top, left, width, height).data.buffer; Burn CPU (not GPU) cycles a. fore each(var obj in view) { … } NOTE: Integrate other streams & sensor data here Render results a. using HTML/JS/CSS b. using another <canvas> and drawImage() c. using WebGL d. a combination of all setInterval() RequestAnimationFrame() setTimeout()
  33. 33. Now lets compare that to the WebGLSL based Video/Shader pipeline https://buildAR.com
  34. 34. https://buildAR.com The Video/Shader MediaStream pipeline Access the camera (or some other stream) source a. getUserMedia() Setup <video> element in the DOM a. declaratively then via getElementById or similar b. createElement(“video”) (no need to appendChild()) Pipe camera stream into <video> a. video.src = stream Compile shaders from text in elements in the DOM a. declaratively then via getElementById or similar b. createElement(“canvas”) then appendChild() Get 3D drawing context a. canvas.getContext('webgl|experimental-webgl'); Load <video> frame into fragment shader a. gl.texImage2D(..., video) Burn GPU cycles a. for each(var obj in view) { … } NOTE: Integrate other streams & sensor data here Render results a. using HTML/JS/CSS b. using another <canvas> and drawImage() c. using WebGL d. a combination of all setInterval() RequestAnimationFrame() setTimeout()
  35. 35. https://buildAR.com - image credit
  36. 36. Key Concept: ArrayBuffers and minimising copies are the key to an efficient walk https://buildAR.com
  37. 37. https://buildAR.com Views are moveable windows into the raw data streams 1. Get stream MediaStream from getUserMedia or RTCPeerConnection 2. Connect to a MediaStream pipeline Canvas/Video, Image Capture or Recording pipelines 3. Get ArrayBuffer new ArrayBuffer() 6. Move through Views to process data for each (var view in scene) { …process(view)... } 7. Output events and metadata return results 8. Update UI and/or send requests Wave hands inspriationally here! 5. Populate a scene Array with Array Buffer Views var scene = [new Uint8ClampedArray(buffer), ...]
  38. 38. https://buildAR.com - image credit
  39. 39. Key Summary: Where is the Augmented Web up to right now? https://buildAR.com
  40. 40. https://buildAR.com Mobile AR vs. the Augmented Web Mobile AR at end of 2009 Augmented Web at end of 2013
  41. 41. https://buildAR.com Mobile AR vs. the Augmented Web Apple had not yet opened up camera access on iOS through objective-c. See the Letter from Ori Mobile AR at end of 2009 Augmented Web at end of 2013
  42. 42. https://buildAR.com Mobile AR vs. the Augmented Web Apple had not yet opened up camera access on iOS through objective-c. See the Letter from Ori Apple has not yet opened up camera access by adopting getUserMedia()/WebRTC. See 1.0 spec within ~5mths Mobile AR at end of 2009 Augmented Web at end of 2013
  43. 43. https://buildAR.com Mobile AR vs. the Augmented Web Apple had not yet opened up camera access on iOS through objective-c. See the Letter from Ori QR codes, Geo AR and Fiducial Marker tracking were commonly available but Natural Feature Tracking was still evolving. Apple has not yet opened up camera access by adopting getUserMedia()/WebRTC. See 1.0 spec within ~5mths Mobile AR at end of 2009 Augmented Web at end of 2013
  44. 44. https://buildAR.com Mobile AR vs. the Augmented Web Apple had not yet opened up camera access on iOS through objective-c. See the Letter from Ori QR codes, Geo AR and Fiducial Marker tracking were commonly available but Natural Feature Tracking was still evolving. Apple has not yet opened up camera access by adopting getUserMedia()/WebRTC. See 1.0 spec within ~5mths QR codes, Geo AR and Fiducial Marker tracking are now available and Natural Feature Tracking is quickly evolving in our R&D. Mobile AR at end of 2009 Augmented Web at end of 2013
  45. 45. https://buildAR.com Mobile AR vs. the Augmented Web Apple had not yet opened up camera access on iOS through objective-c. See the Letter from Ori QR codes, Geo AR and Fiducial Marker tracking were commonly available but Natural Feature Tracking was still evolving. There was limited if any support for rich 3D on all the mainstream AR browsers. Apple has not yet opened up camera access by adopting getUserMedia()/WebRTC. See 1.0 spec within ~5mths QR codes, Geo AR and Fiducial Marker tracking are now available and Natural Feature Tracking is quickly evolving in our R&D. Mobile AR at end of 2009 Augmented Web at end of 2013
  46. 46. https://buildAR.com Mobile AR vs. the Augmented Web Apple had not yet opened up camera access on iOS through objective-c. See the Letter from Ori QR codes, Geo AR and Fiducial Marker tracking were commonly available but Natural Feature Tracking was still evolving. There was limited if any support for rich 3D on all the mainstream AR browsers. Apple has not yet opened up camera access by adopting getUserMedia()/WebRTC. See 1.0 spec within ~5mths QR codes, Geo AR and Fiducial Marker tracking are now available and Natural Feature Tracking is quickly evolving in our R&D. Rich 3D is supported in Chrome and Firefox on Android. Safari supports WebGL, and iOS devices support OpenGL, but iSafari does not have WebGL enabled Mobile AR at end of 2009 Augmented Web at end of 2013
  47. 47. https://buildAR.com Mobile AR vs. the Augmented Web Apple had not yet opened up camera access on iOS through objective-c. See the Letter from Ori QR codes, Geo AR and Fiducial Marker tracking were commonly available but Natural Feature Tracking was still evolving. There was limited if any support for rich 3D on all the mainstream AR browsers. Apple has not yet opened up camera access by adopting getUserMedia()/WebRTC. See 1.0 spec within ~5mths QR codes, Geo AR and Fiducial Marker tracking are now available and Natural Feature Tracking is quickly evolving in our R&D. Rich 3D is supported in Chrome and Firefox on Android. Safari supports WebGL, and iOS devices support OpenGL, but iSafari does not have WebGL enabled (except for advertisers!!!?). Mobile AR at end of 2009 Augmented Web at end of 2013
  48. 48. https://buildAR.com What's in the near future? Integrating WebRTC and Visual Search Using WebGL/GLSL to utilise GPU parallelism Khronos Group's OpenVX Khronos Group's Camera Working Group Lots more demos to share! 8)
  49. 49. https://buildAR.com What's in the near future? Integrating WebRTC and Visual Search Using WebGL/GLSL to utilise GPU parallelism Khronos Group's OpenVX Khronos Group's Camera Working Group Lots more demos to share! 8)
  50. 50. https://buildAR.com What's in the near future? Integrating WebRTC and Visual Search Using WebGL/GLSL to utilise GPU parallelism Khronos Group's OpenVX Khronos Group's Camera Working Group Lots more demos to share! 8)
  51. 51. https://buildAR.com What's in the near future? Integrating WebRTC and Visual Search Using WebGL/GLSL to utilise GPU parallelism Khronos Group's OpenVX Khronos Group's Camera Working Group Lots more demos to share! 8)
  52. 52. https://buildAR.com Join the W3C's Augmented Web Community Group http://w3.org/community/ar
  53. 53. https://buildAR.com - image credit Web Standards for AR An intro to the latest developments on the Augmented Web Platform

×