How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
Sumit& archit osi nov-2011-displays-in-mobile-devices
1. Displays in Mobile Devices
OSI Days 2011, Bangalore
Archit Taneja (archit@ti.com)
Sumit Semwal (sumit.semwal@ti.com)
1
Linux
Development Centre
2. Agenda
• Historical look at mobile devices
– Non-phone mobile devices
• Types of display panels
• Display Subsystems
– In PC World
– SoC World
– PC vs SoC
– Example Data Flow
• Mobile Display Interface Standards
• Interesting Display Use Cases
• Example of SoC DSS:
– OMAP2+ Display Subsystem
– Current software design – DSS2 framework
• Linux Kernel Display Frameworks
• What Next?
• Q&A
2
Linux
Development Centre
3. Under CC License from http://xkcd.com/732/
Linux
Development Centre
4. A Historical Look at Mobile Displays
LG Optimus 3D
3D Stereoscopic LCD;
800x480; 4.3” [4]
Nokia 9000 Communicator:
First 'business phone' Google Galaxy Nexus
Monochrome display, Wide Super AMOLED plus
640x200: [3] 720x1280; 4.65”
Nokia 1011
First mass
One of the first
produced
color screen
GSM phone
phones
Monochrome
Sony Ericsson T68i
[1]
101x80, 256color
[2]
Apple iPhone 4
Google Nexus One “Retina” display
AMOLED: 480x800, 3.7” 960 x 640; 3.5” [5]
All trademarks and copyrights are properties of and attributed to their respective companies
Sources: in references
5. 'non-phone' Mobile Displays
Device Positioning Screen Resolution Technology
size
Samsung Tablet 5.3” 1280 x 800 HD sAMOLED
Galaxy Note Smartphone
Dell Phone 5” 800 x 480 TFT LCD
Tablet 7” 800 x 480 Capacitative
10.1” 1366 x 768 TFT
Amazon eBook Reader 6” 600 x 800 eInk, 16 level
Kindle 3 grayscale
B&N Nook eBook Reader 6” 800 x 600 eInk, 16 level
grayscale
Archos PMP 7” 800 x 480 TFT
Tablet 8” 1024 x 768
Tablet 10.1” 1024 x 600
Apple iPad Tablet 9.7” 1024 x 768 IPS LED
Samsung Tablet 7” 1024 x 600 TFT
Galaxy Tab 8.9” 1280 x 800
10.1” 1280 x 800
Only an indicative list; certainly not exhaustive!
All trademarks and copyrights are properties of and attributed to their respective companies
Linux
Sources: in references Development Centre
6. Types of Display Panels
• Retina – Apple iPhone 4
– In Plane Switching (IPS) LCD
– Highest density: 326 ppi (3.5” screen, 960x640)
• AMOLED (Active Matrix Organic LED)
– No Backlight needed
• Higher contrast ratios, more power efficient.
– Pen Tile arrangement – RGBG arrangement of sub-pixels, where G sub-
pixel is shared by 2 pixels. So the display has only 2 sub-pixels per real
pixel.
• Super AMOLED
– Same as AMOLED, but touchscreen is integrated onto the same glass, thus
increasing visibility on the screen.
All trademarks and copyrights are properties of and attributed to their respective companies
Linux
Sources: in references Development Centre
7. Types of Display Panels
• Super AMOLED Plus
– Integrated touchscreen, and no Pen tile arrangement – much brighter,
thinner and more efficient.
– regular RGB, so 50% more subtiles, hence lesser resolution.
• eInk Panel
– Based on actual electronic ink that is coated on some ultra-thin plastic.
– eInk contains microcapsules filled with ink, and pigmented, charged chips.
– Each microcapsule acts as a pixel; when an electric charge is applied, either
the light or dark pigmented chip will 'come on top', thus 'switching' the pixel
on or off.
All trademarks and copyrights are properties of and attributed to their respective companies
Linux
Sources: in references Development Centre
8. Display Subsystems: PC World
2D/3D HDMI
Accelerators
Pixel Gen 1
Display
Port
Port
Video
Pixel Gen 2
Decoders
Interconnect Display(s)
LVDS
Scene information
Overlay / Pixel Gen M
or Analog
Blending
Encoded frames Video
GPU Display I/F
Display Subsystem
Shared/Dedicated
Memory
Linux
Development Centre
9. Generic SoC Hardware Blocks
Video Display Sub
MPU GPU
Accelerator System Display(s)
interconnect
EMIF DSP Imaging Other
Peripherals
Memory
Only display related blocks are shown; there are many other blocks too, which are out of scope
for this discussion.
Linux
Development Centre
10. Display Subsystem in SoCs
Serial
Plane 1 Compositor 1 Interface
& (MIPI DSI)
Pixel Gen
Plane 2 Parallel
Interface
(MIPI DPI)
Compositor 2
Plane 3 &
Interconnect Remote
Display(s)
Pixel Gen framebuffer
Interface
(MIPI DBI)
GPU composed frames
or
Decoded Video frames Compositor M HDMI/
& Display Port
Plane N Pixel Gen
Overlay Display I/F
Display Subsystem
Linux
Development Centre
11. Display Subsystems: PC vs SoC
• Differences:
– PC: GPUs are generally responsible for composing different windows/layers to get one single frame.
Display Subsystem only fetches and display this composed frame .
– SoC: Display Subsystem is 'also' responsible for composing layers generated by the GPU and Video
accelerators.
– PC: Display Subsystem may only fetch frame data from the dedicated memory for the graphics
card/controller.
– SoC: Display Subsystem fetches frame data from the main memory connected to the SoC.
• Reasons:
– Legacy: GPUs and Video Accelerators have existed in SoCs as separate entities for a while. Making it
important for the Display Subsystems to compose the output of each into one frame.
– Area: SoCs (and mobile devices) can't afford dedicated memory for GPUs.
– IP reuse: Different Vendors of GPUs and Display Subsytems in SoCs, easier to interface with a standard
interconnect.
– Power: Overlaying different layers is more optimal for SoCs in some cases.
Linux
Development Centre
12. Example data flow
3. Display frame on panel
1. Video frames to decode
1. Commands to draw UI
2.Done
Video Display Sub
MPU GPU Accelerator System
2. Done
interconnect
2. Soc 2.
3. Display
Control path
Data path
Memory
Only display related blocks are shown; there are many other blocks too, which are out of scope
Linux
for this discussion. Development Centre
13. Mobile Display Interface Standards
MIPI
- Display Working Group
Primary (fixed) panels:
- Display Serial Interface (DSI):
- serializes all pixel data, commands, and events.
- low power(SLVS)
- Works with Smart / Dumb Panels.
- Display Parallel Interface (DPI):
- Legacy parallel interface.
- Lots of lanes needed for pixel data and control events.
- used with Dumb Panels.
- Display Bus Interface (DBI):
- Parallel Interface for smart panels.
Pluggable panels:
- HDMI
- Display Port
- SDTV / Composite
- DVI
All trademarks and copyrights are properties of and attributed to their respective companies
Linux
Sources: in references Development Centre
14. Interesting Display Use cases
• Device Dock
• Dynamic configurability, parallel usage
– Switch output paths, using multiple paths simultaneously
• Handle audio interfacing additionally
– HDMI support
– DSI support in future
• Support for 'virtual' pipelines – when we run out of video pipelines
– Co-use of 2D/3D graphics accelerator.
• Multi-display configurations
– Clone, Virtual.
●
Display security
14
Linux
Development Centre
15. OMAP2+ DSS hardware: Overview
OMAP2,3,4
Display
OMAP3,4
Panels
OMAP4
DISPC DSI1 encoder
GFX ●
Gfx, vid1-vid3 are
Primary DSI overlays, which color-
convert, scale each frame
VID1 LCD1 RFBI
mgr
RFBI
●
Managers can compose,
VID2 overlay multiple streams
LCD2
DSI2 encoder into one.
VID3 mgr Secondary
DSI ●
Not all paths are shown:
DPI All overlays connect to all
TV
overlay managers.
mgr DPI
HDMI encoder
VENC HDMI
Display
Interfaces SDTV
DSS hardware
15
Linux
Development Centre
16. Current software design – DSS2 framework
UI // Graphics
UI Graphics Video playback
Video playback
Userspace app Userspace app Control
Userspace app Userspace app
User
space Framebuffer APIs V4L2 APIs
/dev/fb0 /dev/video1
sysfs
/dev/fb1 /dev/video2
Panel
omapfb driver drivers omap_vout driver
Kernel
space
gfx vid1 vid2 vid3
DSS2
DSS2
lcd1-mgr lcd2-mgr tv-mgr
OMAP DSS hardware
Display
hardware HDMI TV LCD panels
16
Linux
Development Centre
17. Linux Kernel Display Frameworks
• Frame Buffer
– Oldest of them all - first public release in 1999.
– Linux Kernel Abstraction for graphics hardware – represents framebuffer of some
video hardware, so the software doesn't need to know the low-level (hardware) stuff.
– allows emulate text console without hardware text-mode support.
– is a possible graphic output method for X server; independent of video adapter
hardware and drivers.
– allows 'direct' graphics programs without X; popular in embedded world [GTK+, Qt
Extended are some examples]
– No inherent window or buffer management
• DirectFB [6]
– Specifically designed for embedded systems.
– Is a thin library providing hardware graphics acceleration.
– Integrated windowing system with support for translucent windows and multiple
display layers.
– Input device handling.
– Is a complete hardware abstraction layer, with software fallbacks to every graphics
operation not supported by an underlying hardware.
– Provides for X-less graphical systems: gtk+, LinuxTV, HP WebOS to name a few.
Linux
Development Centre
18. Linux Kernel Display Frameworks
• Video For Linux (V4L)
– Kernel interface for streaming capture and output drivers.
– Provides framework for buffer management and streaming – very relevant for streaming
kind of use-cases [video record, playback, TV, even analog radio! :)]
– Buffers are tied to each device, and allocated before streaming starts.
• Direct Rendering Manager (DRM)
– The modern Linux (and *BSD) display driver framework
– The kernel component of DRI (Direct Rendering Infrastructure)
– But not just DRI:
• Hotplug
• XRandR/KMS
• EDID parsing
– Kernel Mode Setting (KMS)
• Setting screen resolution/timings
• Hotplug, and retrieving EDID
• Multi-display configurations
– Buffers are independent of overlays
– Security inherent
Linux
Development Centre
19. What Next?
- HDMI 3D
- eInk Color (Triton) [7]
Linux
Development Centre
24. Acronyms
• SoC: System on Chip
• DSI: Display Serial Interface: a MIPI standard interface for serial display panels.
• DPI: Display Pixel Interface: a MIPI standard interface for parallel display panels.
• RFBI: Remote Frame Buffer Interface: a MIPI standard interface for parallel display panels
supporting remote frame buffer mechanism.
• HDMI: High Definition Multimedia Interface: a standard compact AV interface to transmit
uncompressed digital data.
• VENC: Video ENCoder: standard interface to display content on standard definition TVs, using
TV formats like NTSC / PAL.
24
Linux
Development Centre
25. Features of OMAP2+ DSS
Feature OMAP2430 OMAP3430 OMAP4430
GFX overlay 1 1 1
VID overlay 2 2 3
WB overlay - - 1
LCD overlay 1 1 2
manager
TV overlay 1 1 1
manager
Rotation Via VRFB* Via VRFB* Via TILER*
(per overlay)
OMAP3/4 support ARGB color formats.
OMAP4 additionally has native support for NV12, configurable Z-order.
Overlay perspective
GFX overlay supports CLUT/gamma table, anti-flicker, replication logic – primarily meant for UI.
VID overlays support Multiple color formats(RGB, YUV, NV12 based on OMAP version), colorspace conversion, independent horizontal
and vertical polyphase filter scaling, and VC-1 range mapping.
WB overlay, in
mem-to-mem mode: colorspace conversion, rescaling, compositing processes.
Capture mode: allows composited encoded pixel data to be stored back into memory.
Overlay perspective
Each overlay supports: overlaying [fixed-order for OMAP2/3, z-order per pipeline for OMAP4], transparency color-keys [source and
destination] and global and pixel alpha blending [OMAP3/4], Gamma correction, Sync buffers.
LCD overlays additionally support Active / Passive matrix dithering, color phase rotation.
25
* Read more about Virtual Rotated Frame Buffer (VRFB) and TILER in the 3430 public TRM and 4430 public TRM respectively
Linux
Development Centre
