This document discusses challenges and solutions related to audio management in the Firefox OS operating system. It covers issues like multi-channel audio, background audio playback, competing audio streams, and ensuring alarms can still sound when other audio is muted. The document proposes attributes for <audio> tags, different volume settings for notification, content, and alarm audio, and an "interrupt" event for notifying when audio is interrupted by a higher priority stream. It also addresses related topics like silent mode policies, video element audio, and ensuring compliance with standards for safe audio output levels.

1. Web Developer or OS
Developer?
Experience from FirefoxOS
Alive
COSCUP 2013

2. me
@alivedise
http://alivedise.github.io
http://github.com/alivedise
alive@mozilla.com
alegnadise@gmail.com
~2.5year Frontend Exp.
聽說COSCUP一定要放

3. Operating System
?
http://www.flickr.com/photos/klaikong/6041270516/

4. Operating System

5. FirefoxOS….OS?
Storage
Management
fileSystem(yet)
DeviceStorageWatcher
Memory
Management
PageVisibility
mozAppType
User
Interface
Multi Window
Management
Statusbar
System Overlay
App Overlay
Web Activity
Keyboard
Hardware
Resource
Management
Wi-Fi
Radio
GPS
Bluetooth
Power
CPU WakeLock
Audio/Media
Multi-Tasking
mozBrowser
remote=true
Card view

6. • Multi Channel
• Audio Competing
• Background Play
Audio Management

7. The beginning of everything..
http://bugzil.la/805333

8. 805333

9. Why do we need multi-
channel on OS?

10. Sounds within FxOS - <audio/>

11. Sounds within FxOS - radio

12. Sounds within FxOS - alarm

13. Sounds within FxOS - voice

14. The world of single channel..
Peace

15. But..
Too peaceful.

16. This means:
When you mute the device, your
alarm also is muted. So..

17. Overslept ♪♪♪~
Zzzzz..

18. How does OS avoid that?
Different settings for audios on
different purposes.

19. On the other hand..
Competing

20. On Desktop browser, we don’t need
competing.

21. How about mobile OS?

22. Cases
• one <audio/> in one app.
• two <audio/> in one app.
• two <audio/> in two apps.
• one <audio/> and one not-audio-sound in two
apps.

23. It’s all about UX.
User wants to listen music in background..
User has difficulty to mute the background app..
User needs the alarm to sound anyway..

24. Android solution: audio focus
• http://developer.android.com/training/managing
-audio/audio-focus.html
• App is responsible to request the audio focus on
demand and release the focus actively.

25. Problems
• How to request audio focus in web?
• Volume control of different type of audio source.
• Telephony is not using audio tag.
• FM is not using audio tag.
• …

26. Proposal
• New attribute for media tags.
• Background play and competing needs permission
check.
• Expose current occupied media info to system.

27. Page Visibility
• HTML5 Page Visibility
• The page visibility state is simple on desktop
browsers by toggling tabs.
• A little complicated on FirefoxOS
• Window(App) has opening/closing transitions.
• Screen off means page visibility is hidden, too.
• Background page has higher possibility to be
killed. (Memory Management!)

28. mozAudioChannelType
telephony
alarm
notification
content
normal(default)

29. Multi channel volume settings
• ringer, desktop-notification
audio.volume.notification
• music, FMRadio
audio.volume.content
• alarm
audio.volume.alarm
• telephony
audio.volume.voice
• Bluetooth SCO
audio.volume.bt_sco

30. Usage
OR
manifest.webapp
app.html
app.js

31. Interrupted Event

32. Current Competing Policies
• Channel with higher priority interrupts the
lower channel.
• Foreground page is always playable. But
interruption occurs or not depends on
the channel of the media element.
• When screen is off, the current app or
page is not brought to background now if
it’s playing normal channel audio now.
(Memory and Power concern!)
• Audio elements being interrupted could
know by ‘mozinterruptbegin’ and
‘mozinterruptend’.
content
normal

33. Gecko side
mozAudioChannelType
audio
App
nsIDOMHTMLAudioElement
nsIDOMHTMLMediaElement
nsHTMLMediaElement
nsIAudioChannelAgentCallback
nsIAudioChannelAgent
nsBuiltinDecoder
nsMediaOmxDecoder
nsMediaDecoder
DataSource
MediaStreamSource
MediaResource
- SetReadMode()
create
IOMX
OMX
MediaExtractorOMXCodec
MediaSource - Tell()
- Seek()
- Read()
- GetLength()
hw codec
sw codec
MediaBuffer
- CanPlayChanged()
- mozinterruptbegin event
- mozinterruptend event
OmxDecoder
nsHTMLAudioElement
nsIChannel
create
nsMediaOmxReader
nsBuiltinDecoderReader
create
create
MediaOmxStateMachine
nsBuiltinDecoderStateMachine
nsDecoderStateMachine
create
nsIThread
nsIThread
control
create
nsDocument
- NotifyOwnerDocumentActivityChanged()
FileMediaResource
ChannelMediaResource
nsAudioStream
nsNativeAudioStream
libsydneyaudio
AudioTrack
AudioFlinger
- Write()
- Pause()
- Resume()
- IsPaused()
- Drain()
- GetMinWriteSize()
- GetPosition()
- SetVolume()
platform: Firefox OS(gonk)
branch: Firefox OS 1.01 (b2g18)
will be replaced to libcubeb in v1.2
Audio Streams
Diagram
https://github.com/sotaroikeda/firefox-diagrams

34. Since API is implemented..
World peace?

35. No…
Don’t forget the dependency tree.

36. Incoming Issues
• Silent mode under multi channels
• Background play ability of existing web pages
• Background play and competing for video
• Short playing sound is hard to adjust.
• DTMF tones
• Power consumptions.
• Bluetooth earphone has its own profile.
• …

37. Policy for Silent mode
• Easy for single channel
• Multi channels
• The default channel for volume rockers is
‘notification’. (Same as Android)
• Silent mode interesting channels
• notification
• content/normal
• Alarm is not affected by silent mode.

38. Policy for video element
• video doesn’t need to play at background.
• Currently, use ‘visibilitychange’ event.
• video by default shall be able to compete the
background playing content channel.
• Separate background play and audio competing
for video element.

39. Background play issues
• Existing web pages: “Hey, I’m a music site!”
• grooveshark
• youtube
• …
• How to play when screen is off without adding
mozAudioChannelType to each web site?
• Depart screen off state from invisible page
visibility state.

40. CE Certification
• EN 50332-1
• Output level through earphone
should be under 85 dBA.
• Prompt a warning when earphone is
plugged and the volume is greater
than 85 dBA.
• The cool down time of CE warning
popup is continuous 20 hours music
listening.

41. OS level policy for CE
• Only apply to content/normal channel.
• Output power is hardware-dependent. Exactly,
we don’t know the db value for specific devices.
• The CE maximum value is customizable for
different OEM via default settings.

42. After all..

43. UX logic in backend and frontend
backend
frontend

44. Find out solutions for existing web sites without
new API.
• As a platform, it’s difficult to make decisions.
• New API takes time to be mature.

45. The scope is undefined before you look
into it.

46. To be honest, I wrote few
code in these issues.
But lots of online/offline discussions.

47. The position of Frontend
https://speakerdeck.com/josephj/f2e-the-keystone

48. • It doesn’t matter you are making a web site, a
web app or an operating system.
• Frontend is the one connecting UX and Backend,
and struggle for ideal design and real
requirement.

49. Thanks!
Not the end, to be continued..