Introduction to AV Foundation (CocoaConf, Aug '11)

Introduction to AV Foundation
Chris Adamson — @invalidname — http://www.subfurther.com/blog

CocoaConf ’11 — Columbus, OH — August 12, 2011

Tuesday, August 23, 11

Road Map

✤ Fundamentals of Dynamic Media

✤ iOS Media Frameworks

✤ Playback

✤ Capture

✤ Editing

✤ Advanced Stuff


The Fundamentals


Analog

✤ Having a measurable value that is continuously variable

✤ Contrast to digital or “discrete” signals


Audio

✤ Phonograph — Grooves
vibrate a needle, which is
ampliﬁed to a speaker

✤ Telephone — Voice vibrates
microphone membrane,
vibration is transmitted as
voltage and reproduced by
vibrating headset speaker

✤ Radio — Audio signal
modulated on a carrier wave


Film

✤ Light projected through
translucent ﬁlm frames onto
screen

✤ Each frame held in place brieﬂy
(! 1/24 sec)

✤ Eye sees a moving image due
to “persistence of vision”

✤ Sound may be out-of-band


Television

✤ Chrominance and luminance
sent as continuous AM signal

✤ CRT gun sweeps across screen
in zig-zag pattern, illuminating
phosphors

✤ Sound is FM in adjacent
spectrum


Digital Media

✤ Represents a continuous signal numerically

✤ Audio — sample the signal at some frequency

✤ Video — digital images as frames

✤ Other kinds of samples — text (captions, subtitles), metadata, web
links, executable code, etc.


Encoding

✤ How do we turn a digital signal into numbers

✤ Audio — Pulse Code Modulation (PCM). Each sample represents
amplitude of audio signal at a speciﬁc time.

✤ Compressed audio — Lossless and lossy transformations to and from
PCM

✤ Video — Series of images (e.g., M-JPEG), or keyframe image (i-frame)
followed by deltas (p-frames and b-frames)

✤ Other media — Text samples are just strings


Containers

✤ Transport and/or storage of encoded streams

✤ Examples: MP3, AIFF, QuickTime Movie, Core Audio Format, .mp4,
MPEG-2 transport stream

✤ Containers may be optimized for streaming, editing, end-user
delivery, etc.


Metadata

✤ Information related to the audio data other than the signal itself

✤ Song title/album/artist, movie title, TV episode title/series, etc.

✤ Some containers support metadata, otherwise it is provided out-of-
band


Keep in mind…

✤ Different codecs may go in different containers

✤ A network stream is a container

✤ A media stream and a network stream are two different things

✤ Containers can contain multiple media streams

✤ A stream’s data is not necessarily in the container ﬁle

✤ Media samples may be in distinct places, or interleaved


Media frameworks on iPhone OS /
iOS


iPhone 2 Media Frameworks

Core Audio /
Low-level audio streaming
OpenAL

Media Player Full-screen video player

Obj-C wrapper for audio
AV Foundation
playback (2.2 only)


iPhone 3 Media Frameworks

Core Audio /
OpenAL

Media Player iPod library search/playback

Obj-C wrapper for audio
AV Foundation
playback, recording


iOS 4 Media Frameworks

Core Audio /
OpenAL

Media Player iPod library search/playback

Audio / Video capture, editing,
AV Foundation
playback, export…

Core Video Quartz effects on moving images

Objects for representing media
Core Media
times, formats, buffers


AV Foundation in iPhone 2.2


AV Foundation in iPhone 3.2


AV Foundation in iOS 4


Size is relative

AV android. QuickTime
QT Kit
Foundation media for Java

Classes 56 40 24 576

Methods 460 280 360 >10,000


How do media frameworks work?


“Boom Box” APIs

✤ Simple API for playback, sometimes
recording

✤ Little or no support for editing,
mixing, metadata, etc.

✤ Example: HTML 5 <audio> tag


“Streaming” APIs

✤ Use “stream of audio” metaphor

✤ Strong support for mixing, effects,
other real-time operations

✤ Example: Core Audio


“Streaming” APIs

✤ Use “stream of audio” metaphor

✤ Strong support for mixing, effects,
other real-time operations

✤ Example: Core Audio
and AV Foundation (capture)


“Document” APIs

✤ Use “media document” metaphor

✤ Strong support for editing

✤ Mixing may be a special case of
editing

✤ Example: QuickTime


“Document” APIs

✤ Use “media document” metaphor

✤ Strong support for editing

✤ Mixing may be a special case of
editing

✤ Example: QuickTime
and AV Foundation (playback and editing)


AV Foundation Classes

✤ Capture

✤ Assets and compositions

✤ Playback, editing, and export

✤ Legacy classes


Assets and Players


Demo
VTM_AVPlayer


AVAsset

✤ A collection of time-based media data

✤ Sound, video, text (closed captions, subtitles, etc.)

✤ Each distinct media type is contained in a track

✤ An asset represents the arrangement of the tracks. The tracks
represent the traits of the media’s presentation (volume, pan, afﬁne
transforms, opacity, etc.).

✤ Asset ≠ media. Track ≠ media. Media = media.

✤ Also contains metadata (where common to all tracks)


AVAsset subclasses

✤ AVURLAsset — An asset created from a URL, such as a song or
movie ﬁle or network document/stream

✤ AVComposition — An asset created from assets in multiple ﬁles, used
to combine and present media together.

✤ Used for editing


AVPlayer

✤ Provides the ability to play an asset

✤ play, pause, seekToTime: methods; currentTime, rate properties

✤ Init with URL or with AVPlayerItem
NSURL *url = [NSURL URLWithString:
@"http://www.subfurther.com/video/running-start-
iphone.m4v"];
AVURLAsset *asset = [AVURLAsset URLAssetWithURL:url
! ! ! ! ! ! ! ! options:nil];
AVPlayerItem *playerItem = [AVPlayerItem
playerItemWithAsset:asset];
player = [[AVPlayer playerWithPlayerItem:playerItem]
retain];


AVPlayerLayer (or not)

✤ CALayer used to display video from a player

✤ Check that the media has video

NSArray *visualTracks = [asset tracksWithMediaCharacteristic:
AVMediaCharacteristicVisual];
if ((!visualTracks) ||
! ([visualTracks count] == 0)) {
! playerView.hidden = YES;
! noVideoLabel.hidden = NO;
}


AVPlayerLayer (no really)

✤ If you have video, create AVPlayerLayer from AVPlayer.

✤ Set bounds and video “gravity” (bounds-ﬁlling behavior)
else {
! playerView.hidden = NO;
! noVideoLabel.hidden = YES;
! AVPlayerLayer *playerLayer = [AVPlayerLayer
playerLayerWithPlayer:player];
! [playerView.layer addSublayer:playerLayer];
! playerLayer.frame = playerView.layer.bounds;
! playerLayer.videoGravity =
AVLayerVideoGravityResizeAspect;
}


A brief sidetrack…


HTTP Live
Streaming


HTTP Live Streaming

✤ Audio / Video network streaming standard developed by Apple

✤ Replaces RTP/RTSP

✤ Built-in support in iOS (AV Framework, Media Player) and Mac OS X
10.6 (QTKit)

✤ Required for apps that stream more than 10 MB over cellular network


How HTTP Live Streaming works

✤ Segmenting server splits source media into separate ﬁles
(usually .m4a for audio-only, .ts for A/V), usually of about 10 seconds
each, and creates an .m3u8 playlist ﬁle

✤ Playlist may point to bandwidth-appropriate playlists

✤ Clients download the playlist, fetch the segments, queue them up

✤ Server updates playlist periodically with latest segments; clients
refresh playlist, fetch and queue new segments


Steve’s .m3u8 playlists
#EXTM3U
#EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=688301
http://qthttp.apple.com.edgesuite.net/
11piubpwiqubf06/0640_vod.m3u8

#EXTM3U
#EXT-X-TARGETDURATION:8
#EXT-X-MEDIA-SEQUENCE:0
#EXTINF:8,
0640/06400.ts
#EXTINF:8,
0640/06401.ts
#EXTINF:8,
0640/06402.ts
#EXTINF:8,
0640/06403.ts

HTTP Live Streaming wins

✤ Works with existing ﬁle servers and content delivery networks

✤ Port 80 is never blocked

✤ Adapts to changes in available bandwidth

✤ Can be encrypted

✤ Has been submitted as a proposed IETF standard

✤ http://tools.ietf.org/html/draft-pantos-http-live-streaming-04


HTTP Live Streaming fails

✤ Not really “live” when buffer can be a minute long

✤ Can’t watch a game on TV and listen to HLS web radio for the
audio

✤ Modest adoption outside of the Apple world

✤ VLC 1.2, third-party Silverlight project, partial support in Android
3.0


HLS in AVF

NSURL *url = [NSURL URLWithString:@"http://
qthttp.apple.com.edgesuite.net/11piubpwiqubf06/sl_vod.m3u8"];
AVURLAsset *asset = [AVURLAsset URLAssetWithURL:url
! ! ! ! ! ! ! ! options:nil];

✤ Just use the .m3u8 to create an AVAsset like any other NSURL

✤ Do not check for presence of visual tracks immediately

✤ AVF needs to start parsing the stream before it knows what kinds
of tracks it has


Back to AV Foundation…


Media Capture

✤ AV Foundation capture classes for audio / video capture, along with
still image capture

✤ Programmatic control of white balance, autofocus, zoom, etc.

✤ Does not exist on the simulator. AV Foundation capture apps can
only be compiled for and run on the device.

✤ API design is borrowed from QTKit on the Mac


Capture Classes Seem Familiar?
QT Kit AV Foundation
QTCaptureAudioPreviewOutput AVCaptureAudioDataOutput
QTCaptureConnection AVCaptureConnection
QTCaptureDecompressedAudioOutput AVCaptureDevice
QTCaptureDecompressedVideoOutput AVCaptureFileOutput
QTCaptureDevice AVCaptureInput
QTCaptureDeviceInput AVCaptureMovieFileOutput
QTCaptureFileOutput AVCaptureOutput
QTCaptureInput AVCaptureSession
QTCaptureLayer AVCaptureStillImageOutput
QTCaptureMovieFileOutput AVCaptureVideoDataOutput
QTCaptureOutput AVCaptureVideoPreviewLayer
QTCaptureSession
QTCaptureVideoPreviewOutput
QTCaptureView


Capture basics

✤ Create an AVCaptureSession to coordinate the capture

✤ Investigate available AVCaptureDevices

✤ Create AVCaptureDeviceInput and connect it to the session

✤ Optional: set up an AVCaptureVideoPreviewLayer

✤ Optional: connect AVCaptureOutputs

✤ Tell the session to start recording


Demo
VTM_AVRecPlay


Getting capture device and input

AVCaptureDevice *videoDevice = [AVCaptureDevice
defaultDeviceWithMediaType: AVMediaTypeVideo];
if (videoDevice) {
! NSLog (@"got videoDevice");
! AVCaptureDeviceInput *videoInput = [AVCaptureDeviceInput
deviceInputWithDevice:videoDevice
! ! ! ! ! ! ! ! error:&setUpError];
! if (videoInput) {
! ! [captureSession addInput: videoInput];
! }
}

Note 1: You may also want to check for AVMediaTypeMuxed
Note 2: Do not assume devices based on model (c.f. iPad
Camera Connection Kit)


Creating a video preview layer

AVCaptureVideoPreviewLayer *previewLayer =
[AVCaptureVideoPreviewLayer
layerWithSession:captureSession];
previewLayer.frame = captureView.layer.bounds;
previewLayer.videoGravity =
AVLayerVideoGravityResizeAspect;
[captureView.layer addSublayer:previewLayer];

Keep in mind that the iPhone cameras have a
portrait orientation


Setting an output

captureMovieOutput = [[AVCaptureMovieFileOutput alloc] init];
if (! captureMovieURL) {
! captureMoviePath = [getCaptureMoviePath() retain];
! captureMovieURL = [[NSURL alloc]
initFileURLWithPath:captureMoviePath];
}
NSLog (@"recording to %@", captureMovieURL);
[captureSession addOutput:captureMovieOutput];

We’ll use the captureMovieURL later…


Start capturing

[captureSession startRunning];
recordButton.selected = YES;
if ([[NSFileManager defaultManager]
fileExistsAtPath:captureMoviePath]) {
! [[NSFileManager defaultManager]
removeItemAtPath:captureMoviePath error:nil];
}
// note: must have a delegate
[captureMovieOutput
startRecordingToOutputFileURL:captureMovieURL
! ! ! ! ! ! ! ! recordingDelegate:self];


Capture delegate callbacks

- (void)captureOutput:(AVCaptureFileOutput *)captureOutput
didStartRecordingToOutputFileAtURL:(NSURL *)fileURL
fromConnections:(NSArray *)connections {
! NSLog (@"started recording to %@", fileURL);
}

- (void)captureOutput:(AVCaptureFileOutput *)captureOutput
didFinishRecordingToOutputFileAtURL:(NSURL *)outputFileURL
fromConnections:(NSArray *)connections
error:(NSError *)error {
! if (error) {
! ! NSLog (@"failed to record: %@", error);
! } else {
! ! NSLog (@"finished recording to %@", outputFileURL);
! }
}


More fun with capture

✤ Can analyze video data coming off the camera with the
AVCaptureVideoDataOutput class

✤ Can provide uncompressed frames to your
AVCaptureVideoDataOutputSampleBufferDelegate

✤ The callback provides you with a CMSampleBufferRef


Surprise! Core Media!


Core Media

✤ C-based framework containing structures that represent media
samples and media timing

✤ Opaque types: CMBlockBuffer, CMBufferQueue,
CMFormatDescription, CMSampleBuffer, CMTime, CMTimeRange

✤ Handful of convenience functions to work with these

✤ Buffer types provide wrappers around possibly-fragmented memory,
time types provide timing at arbitrary precision


Media Editing


Video Editing? On iPhone?
Really?

Comparison specs from everymac.com

Really?
1999:
Power Mac G4 500 AGP


Really?
1999:
Power Mac G4 500 AGP

CPU: 500 MHz G4
RAM: 256 MB
Storage: 20 GB HDD

Really?
1999: 2010:
Power Mac G4 500 AGP iPhone 4

CPU: 500 MHz G4
RAM: 256 MB
Storage: 20 GB HDD

Really?
1999: 2010:
Power Mac G4 500 AGP iPhone 4

CPU: 500 MHz G4 CPU: 800 MHz Apple A4
RAM: 256 MB RAM: 512 MB
Storage: 20 GB HDD Storage: 16 GB Flash

Demo
VTM_AVEditor


AVComposition

✤ An AVAsset that gets its tracks from multiple ﬁle-based sources

✤ To create a movie, you typically use an AVMutableComposition

composition = [[AVMutableComposition alloc] init];


Copying from another asset

✤ -[AVMutableComposition insertTimeRange:ofAsset:atTime:error:]

CMTime inTime = CMTimeMakeWithSeconds(inSeconds, 600);
CMTime outTime = CMTimeMakeWithSeconds(outSeconds, 600);
CMTime duration = CMTimeSubtract(outTime, inTime);
CMTimeRange editRange = CMTimeRangeMake(inTime, duration);
NSError *editError = nil;

[targetController.composition insertTimeRange:editRange
! ! ! ! ! ! ! ofAsset:sourceAsset
atTime:targetController.composition.duration
! ! ! ! ! ! ! error:&editError];


CMTime

✤ CMTime contains a value and a timescale (similar to QuickTime)

✤ Time scale is how the time is measured: “nths of a second”

✤ Time in seconds = value / timescale

✤ Allows for exact timing of any kind of media

✤ Different tracks of an asset can and will have different timescales

✤ Convert with CMTimeConvertScale()


Track-level editing

✤ Add an AVMutableCompositionTrack to the composition

✤ Call -[AVMutableCompositionTrack
insertTimeRange:ofTrack:atTime:error]

musicAsset = [AVURLAsset URLAssetWithURL: musicURL
! ! ! ! ! ! ! ! options: nil];
CMTimeRange musicRange = CMTimeRangeMake(
CMTimeMakeWithSeconds (0,600),
composition.duration);
AVAssetTrack *musicAssetTrack = [[musicAsset
tracksWithMediaType:AVMediaTypeAudio] objectAtIndex: 0];
[musicTrack insertTimeRange:musicRange
! ! ! ! ! ofTrack:musicAssetTrack
! ! ! ! ! atTime:CMTimeMakeWithSeconds(0,600)
! ! ! ! ! error:&exportError];

Export

✤ Create an AVAssetExportSession

✤ Must set outputURL and outputFileType properties

✤ Inspect possible types with supportedFileTypes property (list of
AVFileType… strings in docs)

✤ Begin export with exportAsynchronouslyWithCompletionHandler:

✤ This takes a block, which will be called on completion, failure,
cancellation, etc.


Export
AVAssetExportSession *exportSession =
[[AVAssetExportSession alloc] initWithAsset:composition
! ! presetName:AVAssetExportPresetMediumQuality];
exportSession.outputURL = exportURL;
exportSession.outputFileType = @"com.apple.quicktime-movie";
[exportSession exportAsynchronouslyWithCompletionHandler:^{
! NSLog (@"i is in your block, exportin. status is %d",
! ! exportSession.status);
! switch (exportSession.status) {
! ! case AVAssetExportSessionStatusFailed:
! ! case AVAssetExportSessionStatusCompleted: {
! ! ! [self performSelectorOnMainThread:
@selector (doPostExportUICleanup:)
! ! ! ! ! ! ! ! withObject:nil
! ! ! ! ! ! ! ! waitUntilDone:NO];
! ! ! break;
! ! }
! };
}];

Coming Up in Advanced…

✤ Capture-time data processing

✤ Writing raw samples

✤ Reading raw samples

✤ Multi-track editing with effects


Q&A Time

✤ Cleaned-up code will be available on my blog:

✤ http://www.subfurther.com/blog

✤ invalidname [at] gmail.com

✤ @invalidname

✤ Check out the Rough Cut of my Core Audio book too.

✤ Get the WWDC sessions on AV Foundation


Introduction to AV Foundation (CocoaConf, Aug '11)

Recomendados

Recomendados

Más contenido relacionado

Similar a Introduction to AV Foundation (CocoaConf, Aug '11)

Similar a Introduction to AV Foundation (CocoaConf, Aug '11) (20)

Más de Chris Adamson

Más de Chris Adamson (20)

Último

Último (20)

Introduction to AV Foundation (CocoaConf, Aug '11)