Core Image talk at Toronto Area Cocoa and WebObjects Developers Group

1. Core Image
Mark Pavlidis
Co-founder
Flixel Photos Inc.
Toronto Cocoa and WebObjects Developers Group
2013/01/08

2. Motivation
Why use Core Image?
• Image processing and analysis
• Fast, efficient filters
• Auto enhancement
• Feature detection
• Still and “real-time” video
• Simple Objective-C API

3. Definitions
• Image Filter
• a single transform or effect
• built-in or loaded from an Image Unit plugin (OS X only)
• Framework
<CoreImage/CoreImage.h>
• Key Classes
CIImage, CIFilter, CIContext

4. Overview
Apply an image filter to the source pixel data
Monochrome
Filter
• Pixel Accurate
• Non-destructive
• Filters can be chained together

5. Core Image Runtime
How it works

6. Core Image Runtime
How it works
• Operates on image data types from:
• Core Graphics
• Core Video
• Image I/O

7. Core Image Runtime
How it works
• Built-in or plugin filters apply effects
• Written in the Core Image Kernel Language
• JIT complier assembles instruction pipeline

8. Core Image Runtime
How it works
• Executed on the GPU or CPU
• Low-level graphics processing encapsulated by CI
• Performance: GLSL capabilities of the GPU or
processing power of the CPU

9. Core Image Inputs
• Photo Library and Files
imageWithContentsOfURL:
• Live Video Capture
imageWithCVPixelBuffer:
imageWithCVImageBuffer:
• In Memory
imageWithCGImage:
• GL Texture
imageWithTexture:size:flipped:colorSpace:

10. Core Image Outputs
• CGImageRef
• Output to UIImage, ImageIO, ALAssetLibrary
createCGImage:fromRect:
• CAEAGLLayer
drawImage:inRect:fromRect:
• CVPixelBufferRef
render:toCVPixelBuffer:
• Bitmap
render:toBitmap:rowBytes:bounds:format:colorSpace:

11. Simple Filter
// Create a CIImage
CIImage *ciImage = [CIImage imageWithContentsOfURL:myURL];
// Create a CIFilter
CIFilter *filter = [CIFilter
filterWithName:@”CIColorMonochrome”];
[filter setValue:ciImage forKey:kCIInputImageKey];
[filter setValue:@(0.5) forKey:@”inputIntensity];
// Render the filter output image into a UIImage
UIImage *uiImage = [UIImage
imageWithCIImage:filter.outputImage];

12. Filter Chains
// Filter 1
CIImage *output = [CIFilter
filterWithName:@”CIColorMonochrome”]
keysAndValues:
kCIInputImageKey, ciImage,
@”inputIntensity, @(0.5),
nil].outputImage;
// Filter 2
output = [CIFilter
filterWithName:@”CIVignette”] keysAndValues:
kCIInputImageKey, output,
@”inputIntensity, @(0.5),
@”inputRadius, @(1.5),
nil].outputImage;
•CI defers pixel processing until render is requested
•CI optimizes the render graph (e.g., sepia then scale)

13. Math!
• An image filter is kernel
• Output is the convolution of the source & kernel
Source Output
Kernel
2 2 2 1 1 1
2 1 3 -1 2 1 6
2 2 1 -1 -1 1
• Commutative, associative, distributive, associative
with scalar multiplication, multiplicative identity

14. Demo - QuartzComposer

15. Demo - Simple filter

16. Tips and Best Practices
• CIImage and CIFilter are autoreleased
• Use autorelease pools
• CIImage, CIContext are immutable and thread-safe,
CIFilter is not
• CIContext is expensive
• Stores a lot of state information so reuse them
• Avoid CA animations when rendering on GPU
• Use smaller images when possible
• Use Core Animation to upscale view, texture, framebuffer
• Disable colour management
• If real-time performance and/or reduced quality is
unnoticeable

17. Face Detection
CIDetector
• Identifies rectangles that contain human faces
• Feature detection
• eyes & mouth position, tracking ID and frame count (video)
CIContext *context = [CIContext
contextWithOptions:nil];
NSDictionary *opts = @{CIDetectorAccuracy,
! ! ! ! ! ! ! CIDetectorAccuracyHigh};
CIDetector *detector = [CIDetector
! ! ! ! ! ! detectorOfType:CIDetectorTypeFace
! ! ! ! ! ! ! ! context:context
! ! ! ! ! ! ! ! options:opts];
NSArray *features = [detector featuresInImage:myImage
! ! ! ! ! ! options:opts];

18. Demo - Redacted Faces

19. OS X and iOS Differences
Portability
• Some CIImage create methods differ
• Has more built-in filters
• Permits custom filter plugins
• filter.outputImage is iOS only, use
[filter valueForKey:kCIOutputImage]
• Remember to [filter setDefaults]
• All input parameter keys have strings defined

20. Alternatives
• GPUImage (iOS)
✓Hand-tuned OpenGL ES shaders that are much faster than CI
✓Custom filters
- Lacks advanced features of Core Image
• Aviary
✓ Everyone is using it (Twitter, Flickr)
- Everyone is using it
• Only still images
• Hardcore - Write your own GL shaders
• Don’t be silly

21. Summary
Core Image
• Fast and efficient image processing framework
• Simple Objective-C API
• Encapsulates the low-level instruction details
• Many built-in filters
• Advanced processing features:
• Face Detection
• Auto Enhancement
• Real-time video filters

22. More Information
• Core Image Programming Guide
• WWDC 2012 Videos
• Session 510 - Getting Started with Core Image
• Session 511 - Core Image Techniques
• Sample Code
• https://github.com/mhpavl/FilterDemo
• https://github.com/mhpavl/RedactedFaces
• Core Image Funhouse (TBD)

23. [self release];
Mark Pavlidis
mark@pavlidis.ca
Twitter: @mhp
ADN: @mhp