The Linux kernel has adapted in recent years to cope with the arrival of the embedded platforms using more and more advanced graphics capabilities. This presentation will explore the past challenges, but also the present and future, focusing on the current situation, the problems we face and the solutions that are currently proposed and considered.

1. Display and Kernel
Review and Future
Kernel Recipes 2013
Paris
Laurent Pinchart
laurent.pinchart@ideasonboard.com

3. Problem Definition

4. display /

5. / graphics /

6. / video

7. Problem - Purpose
display / graphics /
video

8. format

9. memory /
deep pipeline

10. device / CPU

11. Problem - Source
format
memory / deep pipeline
device / CPU

12. rotation /

13. / scaling /

14. / composing

15. Problem - Processing
rotation
scaling
composing

16. X11 /

17. / Wayland /

18. / MIR /

19. / SDL /

20. / DirectFB /

21. / Raw API

22. Problem - Stack
X11
Wayland
SDL
DirectFB
Raw API

23. DRM /

24. / FBDEV /

25. / V4L2

26. Problem - API
DRM
FBDEV
V4L2

27. Display – Scanout
Frame Buffer

28. Display – Composition
CRTC
Plane(s)
Composition

34. Origins

35. Origins – FBDEV
FBD
EV,Linux
1.3.94
1996 2012
4C
C
Form
ats
2000
Blanking

36. Origins – V4L2
V4L,Linux
2.2.0
1999 2002 2006
V4L2,Linux
2.4.0
V4L
deprecated
2008
V4L2
subdev
2011
M
edia
controller
videobuf2
2012
D
M
ABU
F

37. Origins – DRM/KMS
2000
D
R
M
,Linux
2.2.18
2009
KM
S,TTM
2008
G
EM
Planes,D
M
ABU
F
2011 2012
D
um
b
Buffers

38. Features

39. Memory Management
DRM FB V4L2
Dynamic Allocation Yes No Yes
Multiple Buffers Yes panning Yes
Import dmabuf No userptr
Export
dmabuf
mmap
mmap mmap

40. Mode Setting
DRM FB V4L2
Formats 4CC
RGB
4CC
4CC
Enumeration Planes No Yes
Negotiation No No Yes
Atomicity Yes No No

41. Transformations
DRM FB V4L2
Overlays Yes No Yes
Rotation Yes No Yes
Scaling Yes No Yes
Cropping/Panning Yes Yes Yes

42. Documentation
api core drivers
drm
fb
v4l2

43. DRM API Documentation
The DRM core exports several interfaces to applications,
generally intended to be used through corresponding libdrm
wrapper functions. In addition, drivers export device-specific
interfaces for use by userspace drivers & device-aware
applications through ioctls and sysfs files.
External interfaces include: memory mapping, context
management, DMA operations, AGP management, vblank
control, fence management, memory management, and output
management.
Cover generic ioctls and sysfs layout here. We only need high-
level info, since man pages should cover the rest.
“
„

44. Documentation
api core drivers
drm
fb
v4l2

45. Code Size
api core drivers
drm
fb
v4l2

46. Cumulative Changes - API
0
1000
2000
3000
4000
5000
6000
drm
fb
v4l2
drm
fb
v4l2

47. Cumulative Changes - Core
0
5000
10000
15000
20000
25000
30000
35000
40000
45000
50000
drm
fb
v4l2
drm
fb
v4l2

48. Cumulative Changes - Drivers
0
200000
400000
600000
800000
1000000
1200000
1400000
1600000
1800000
drm
fb
v4l2
drm
fb
v4l2

49. Mailing List Traffic
0
200
400
600
800
1000
1200
1400
1600
1800
drm
fb
v4l2
drm+bug

50. Device Model – FBDEV
Frame Buffer
(memory)
Display
Device

51. Device Model – DRM/KMS
Frame Buffer
(memory)
CRTC Encoder Connector
Planes
(memory)
Encoder
Connector
Connector

52. Device Model – V4L2
Buffers
(memory)
Display
Device
Connector

53. Device Model – V4L2/MC
V4L2
subdev
0
1
2
V4L2
subdev
0 1
V4L2
queue
0

54. Device Model – V4L2/MC

55. Use Cases

56. Use Cases - FBDEV

57. Use Cases - FBDEV
(that's it...)

58. Use Cases - V4L2
Video

59. Use Cases – DRM/KMS
Everything else

60. FB vs. DRM - sloccount
fbdev
0
500
1000
1500
2000
2500
2370

61. FB vs. DRM - sloccount
fbdev drm
0
500
1000
1500
2000
2500
2370
1659

62. DRM/KMS API
drm
kms
exynos
i810
i915
mga
nouveau
r128
radeon
savage
sis
via

63. KMS - Device Model
Frame Buffer
(memory)
CRTC Encoder Connector
Plane
(memory)
SoCMemory Off-Chip
Encoder Connector

64. KMS – Frame Buffer
CRTC
Frame Buffer
GEM
Object(s)
Memory
● width
● height
● format
● pitches
● offsets
Properties

65. DRM/KMS – GEM Object
CRTC
Frame Buffer
GEM Object
● width
● height
● bpp
● pitch
● size
Properties
Memory

66. DRM – Handles
Process A
Local
Handle
GEM
Object
Process B
Send FD
SCM_RIGHTS
Global
FD
1 2
Global
FD
4
Local
Handle
3

67. Active
Area
KMS – Modes (1/2)
sync
back porch
front porch
active area

68. Active
Area
KMS – Modes (2/2)
sync
back porch
front porch
active area
hdisplay
hsync_start
htotal
hsync_end

69. KMS – Mode Setting
crtc
fb
Active
Area
x
y
mode.hdisplay
mode.vdisplay
*connectors
num_connectors
mode
struct drm_mode_set {
struct drm_framebuffer *fb;
struct drm_crtc *crtc;
struct drm_display_mode *mode;
uint32_t x;
uint32_t y;
struct drm_connector **connectors;
size_t num_connectors;
};

70. Source: http://www.flickr.com/photos/buckaroobay/3721809183/

71. WIP – Display Framework
Common Display
Framework
http://lwn.net/Articles/512363/

72. Entity
Entity
Link Port
CDF - Entity Model

73. CDF - Device Tree
hdmi_encoder {
ports {
#address-cells = <1>;
#size-cells = <0>;
port@0 {
hdmi_input: endpoint@0 {
remote = <&display_output>;
};
};
port@1 {
endpoint@0 { ... };
endpoint@1 { ... };
};
};
};

74. Display
Controller
Video
Entity
Pipeline Controller
controlcontrol
Entity Entity
CDF - Configuration Model

75. Display
Controller
Video
Entity
streaming control
streaming control
Entity Entity
CDF - Streaming Control

76. CDF - Integration
Display
Controller
Driver
Control Bus
API
FBDEV DRM
SPI/I2C Mem Map
Panel Driver
DT
pdata
DSI

77. • dri-devel@listsfreedesktop.org
• linux-fbdev@vger.kernel.org
• linux-media@vger.kernel.org
• laurent.pinchart@ideasonboard.com
Contact

78. ? !

79. merci.