Patanjali Somayaji, Ex-Director of Engineering, Motorola Mobility talk about The Evolution of Embedded Linux.

1. The
Evolu*on
of
Embedded
Linux
Patanjali
Somayaji
(patanjalisomayaji@gmail.com)

2. Embedded
Linux
TiVo
Roku
Leapster
Explorer
Samsung
Smart
TV
Robonaut
2

3. The
Early
Days
• Tradi*onally,
Embedded
Systems
used
“roll-­‐your-­‐own”,
“homegrown”
or
“in-­‐house”
RTOSes
• 1980:
Arrival
of
the
first
general
purpose
commercial
embedded
OS
– VRTX
(from
Hunter
and
Ready)
– Used
in
the
Hubble
space
telescope
and
soWware-­‐controlled
radio
in
early
Motorola
Mobile
phones)
• Mid-­‐1980s:
Increased
adop*on
of
commercial
RTOSes
– VxWorks
from
WindRiver
became
the
market
leader
– Other
solu*ons
–
QNX,
Greenhills
Integrity
– Embedded
developer
value
add
was
in
the
App
– Time-­‐to-­‐market
increasingly
important
• Linux
v0.1
in
1991,
by
1995
supported
Alpha,
i386,
MIPS
and
Sparc
architectures;
and
by
1996
had
support
for
m68K
and
PPC

4. The
Early
Days
• 1995:
Busybox
ini*al
release
to
enable
a
small
Linux
image
to
fit
on
a
1.44
MB
floppy
– Started
by
Bruce
Perens
to
use
as
a
Debian
bootable
install
and
rescue
disk
• 1997:
Linux
Router
Project
–
first
embedded
Linux
project
• 1999:
Commercial
RTOS
industry
was
USD
1
Billion
• 1999:
The
first
year
you
could
buy
a
commercial
off
the
shelf
Embedded
Linux
Distribu*on
–
Montavista’s
Red
Hat
Linux
based
“Hard
Hat
Linux”
• 1999:
Linux
based
TiVo
DVR
announced
(PPC
403,
16
MB
RAM,
13
GB
HDD)
• 2000:
Embedded
Linux
Consor*um
(ELC)
–
now
transi*oned
to
the
Linux
Founda*on
star*ng
2005
• CE
Linux
Forum
(CELF)
–
now
CE
working
group
of
the
Linux
Foumda*on

5. Why
did
Embedded
Linux
grow
?
• “given
enough
eyeballs,
all
bugs
are
shallow”
-­‐
ESR
– So
reliability,
which
a
concern
in
the
beginning,
faded
away
• Linux
supported
MMUs
providing
memory
par**oning
and
virtual
memory
– App
crashes
do
not
crash
the
system
– Became
important
as
embedded
systems
grew
in
complexity
– MMUs
became
common
in
low-­‐cost
32-­‐bit
embedded
systems
– uCLinux
developed
for
MMU-­‐less
high
volume,
cost-­‐sensi*ve
markets
• Royalty
free
– No
license
free
per
unit
deployed,
unlike
commercial
RTOSes
– Commercial
linux
is
usually
sold
on
a
developer-­‐seat
license
basis
– Important
for
large
scale
embedded
deployment
which
is
cost
sensi*ve
• No
dependency
on
a
single
OS
vendor
for
a
cri*cal
component
of
embedded
applica*on

6. Why
did
Embedded
Linux
grow
?
• Networking
was
usually
a
paid-­‐for
extra
on
commercial
RTOSes.
Linux
was
built
with
networking
support
from
the
ground
up.
• Open
source
alterna*ve
–
Net
BSD
– NetBSD
was
mature,
open
source
and
gemng
established
at
the
same
*me
– Linux
however
was
based
on
GPLv2
which
triggered
a
lot
of
give-­‐back
– More
drivers
and
architectures
started
to
get
support,
and
there
was
a
massive
Network
effect
• ARM
processors
were
heavily
adopted,
and
Linux
was
ported
to
various
SoCs
and
development
planorms
• Linux
development
in
other
areas
e.g.
Servers
were
leveraged
in
Embedded
systems
e.g.
Networking,
CPU
Hotplug

7. Embedded
Linux
-­‐
Networking
• The
Linux
Router
Project
(LRP)
–
the
first
embedded
Linux
project
?
• Pioneering
features
– Small
base
OS
footprint
– A
simplified
packaging
system
– Menu
based
system
and
package
configura*on
– Strict
separa*on
of
vola*le,
non-­‐
vola*le,
Read
Only,
and
Read/Write
areas
of
the
root
hierarchy
– Unpacked
and
run
from
ramdisk
or
run
directly
from
flash
– A
system
to
commit
configura*on
changes
to
a
non-­‐vola*le
medium
(Disk/Flash)
• Spawned
a
deluge
of
Embedded
Linux
Networking
Devices
Cisco
Linksys
WRT54GL
Wireless-­‐G
Broadband
Linux
Router

8. “Every
age
had
its
own
kind
of
war,
its
own
limi*ng
condi*ons,
and
its
own
peculiar
preconcep*ons”
–
Carl
von
Clausewitz,
German-­‐Prussian
Soldier
&
Military
Strategist

9. Real-­‐*me
Linux
• Different
approaches
– TimeSys
introduces
proprietary
Linux/RT
1.0
– FSM
Labs
RTLinux
(commercially
licensed)
• Real-­‐*me
kernel
running
Linux
as
a
low-­‐priority
thread
• Posix
API
to
run
real-­‐*me
threads
from
User
Space
• Under
15
micro-­‐second
response
(Year
2000)
• Acquired
by
WindRiver
in
2007
– MontaVista
worked
on
kernel
pre-­‐emp*on
• Meanwhile
the
Linux
Kernel
developers
were
working
on
their
solu*on

10. Real-­‐*me
Linux
• 2000
–
Ingo
Molnar
and
Andrew
Morton
start
work
on
voluntary
pre-­‐empt
“CONFIG_PREEMPT_VOLUNTARY”
– Reduce
worst-­‐case
interrupt
latencies
– Break
lengthy
kernel
cycles
into
smaller
chunks
and
add
checks
to
common
causes
of
long
latencies
– Rescheduling
if
an
interrupt
handler
has
set
the
“need_resched”
flag
– Eventually
merged
in
Linux
Kernel
2.6
• 2001
–
Robert
Love
began
work
on
the
kernel
pre-­‐emp*on
“CONFIG_PREEMPT”
– Ini*ally
based
on
MontaVista
patch
– Target
low-­‐latency
apps
and
smooth
user
interac*on
– Makes
kernel
pre-­‐emp*ble
like
user-­‐space,
allowing
higher
priority
tasks
to
run
-­‐
all
kernel
code
outside
of
spinlock-­‐protected
regions
and
interrupt
handlers
eligible
for
pre-­‐emp*on
– Adopted
by
MontaVista
in
MVL
4.0
and
merged
with
Linux
Kernel
2.6

11. Controlling
a
laser
with
Linux
is
crazy,
but
everyone
in
this
room
is
crazy
in
his
own
way.
So
if
you
want
to
use
Linux
to
control
an
industrial
welding
laser,
I
have
no
problem
with
your
using
PREEMPT_RT.”
-­‐
Linus
Torvalds

12. Real-­‐*me
Linux
• 2005
–
Ingo
Molnar
and
others
began
work
on
CONFIG_PREEMPT_RT
– Convert
Linux
Kernel
into
a
fully
pre-­‐emp*ble
kernel
• In-­‐kernel
locking
primi*ves
made
pre-­‐emp*ble
• Cri*cal
sec*ons
made
pre-­‐emp*ble
(crea*on
of
non-­‐pre-­‐
emp*ble
sec*ons
s*ll
possible)
• Implemented
priority
inheritance
for
in-­‐kernel
spinlocks
and
semaphores
• Converted
interrupt
handlers
to
pre-­‐emp*ble
kernel
threads
• Converted
Linux
*mer
API
into
separate
infrastructures
for
high
resolu*on
*mers
and
*meouts
• 10
to
30
micro-­‐second
scheduler
and
interrupt
latencies

13. Real-­‐*me
Linux
• Patches
for
Linux
2.6
Kernel
suppor*ng
Real-­‐*me
– Preemp*ble
Interrupt
Handlers
in
Thread
Context
– IRQ-­‐Disable
Virtualiza*on
for
Drivers
• IRQ
threads
disabled
without
masking
hardware
– Integrated
Kernel
Mutex
with
Priority
Inheritance
(PI)
• Protec*on
of
Kernel
Cri*cal
Sec*ons
by
Preemp*ble
PI
Mutex
– PI
Mutex
Subs*tuted
for
Non-­‐Preemp*ble
Kernel
(SMP)
Locks
• Big
Kernel
Lock
(BKL)
converted
to
PI
Mutex
• Spin-­‐Locks
converted
to
PI
Mutex
• Read-­‐Write
Locks
converted
to
PI
Mutex
• RCU
Preemp*on
Enhancements
– High
Resolu*on
Timers
– User-­‐Space
Mutex
• Robustness/Dead-­‐owner/Priority
Queueing

14. ARM
is
Everywhere
• Smartphones
• Inexpensive
evalua*on
boards
– BeagleBoards
– BeagleBones
– Raspberry
Pi
• Now
also
coming
for
Servers
– AMD
to
release
its
first
“Seaule”
ARM
Server
Chip
in
2014
Source:
hup://arstechnica.com/business/2012/08/from-­‐
altair-­‐to-­‐ipad-­‐35-­‐years-­‐of-­‐personal-­‐computer-­‐market-­‐share/

15. ARM
Linux
• Started
out
in
1994
as
a
port
of
the
1.0.x
kernel
to
the
Acorn
5000
• 1999
Linux
2.2.0
–
ARM
architecture
added
to
mainline
• Variety
of
architectures,
Kernel
Patches
and
drivers
– Hardware
varies
from
SoC
to
SoC
– Development
planorm
divergence
• “ARM
Wrestling”
– Grew
to
be
a
massive
collec*on
of
“subplanorms”
– Managed
independently
by
different
developers
– “ARM
support
is
a
labyrinthine
maze
of
wheels
within
reinvented
wheels”
• Linaro
formed
in
2010
to
drive
common
development
and
integra*on

16. “I
think
the
more
serious
long-­‐term
issue
we
have
in
the
kernel
is
the
wild
and
crazy
embedded
planorm
code
(and
mostly
ARM
-­‐
not
because
ARM
is
in
any
way
fundamentally
crazier,
but
because
ARM
is
clearly
the
most
successful
embedded
planorm
by
far).
The
embedded
world
has
always
tended
to
eschew
standardized
planorms:
they've
been
resource
constrained
etc,
so
they've
done
very
tailored
chip
and
board
solu*ons,
and
felt
that
they
couldn't
afford
a
lot
of
planorm
abstrac*on.”
-­‐
Linus
Torvalds
–
May
2011

17. Mobile
Linux
…
2001/2002:
Sharp
Zaurus
SL-­‐5500
running
OpenZaurus
and
OPIE
SA-­‐1110
CPU
@
206
MHz,
64
MiB
RAM,
16
MiB
NOR
2003:
Motorola
A760
ARM
925
chip
(an
Intel
PXA270)
at
316
MHz,
48
MiB
flash
storage,
QVGA
(240
x
320)
touch
screen
2008:
HTC/T-­‐Mobile
G1
Qualcomm
MSM7210A
(ARM11)
processor
running
@
528
MHz,
256
MiB
NAND
flash
storage,
192
MiB
RAM,
3.2-­‐inch
touchscreen
(320
x
480).
Launched
with
Android
1.0
based
on
modified
Linux
2.6.25
2005:
Nokia
N770
Tablet
TI
OMAP
1710
@
252
MHz
64
MB
RAM,
128
MB
Flash
4.1
inch
touchscreen
(800
x
480).
Launched
on
Maemo/
Linux

18. …
and
Mobile
Linux
Consor*ums

19. Brief
Interlude:
Embedded
Filesystems
• Ini*al
configura*ons
– Expensive
NOR
Flash
for
Cri*cal
storage
– Cheaper
and
denser
NAND
Flash
for
larger
capaci*es
– NAND
flash
used
in
consumer
devices
• Flash
usage
needed
“wear-­‐leveling”
to
spread
the
block
erases
evenly
• Ini*al
Linux
support
for
read-­‐only
FS
(no
need
for
wear-­‐levelling)
– CramFS
&
SquishFS
• Since
Flash
connected
as
raw
memory
devices,
Linux
needed
to
handle
intelligently
for
RW
via
the
Linux
MTD
Interface
(Memory
Technology
Device)
– JFFS2,
YAFFS2,
LogFS,
UBIFS
• Smartphones
evolved
to
using
eMMC
– Large
volume
of
NAND
storage
with
a
controller
– Connect
as
block
devices
– Use
flash
vendor’s
flash
wear-­‐leveling
algorithms
• 2012:
F2FS
– SoWware
layer
to
handle
“smartness”
outside
the
Flash
Transla*on
Layer
(FTL)
– Targeted
at
SSDs,
eMMC,
SD
Cards
and
other
Flash
storage
with
FTL
– Flash
wear-­‐leveling
in
controller

20. Brief
Interlude:
Power
Management
• Embedded
Linux
–
started
off
with
Routers,
etc.
“plugged
in”
devices
so
Power
Management
was
not
a
priority
• PM
focused
mostly
on
Desktop/Laptop/Server
support
– APM
(1992
Intel
&
MicrosoW)
• PM
in
BIOS,
BIOS
HW
specific
• Reduce
CPU,
turn-­‐off
Display,
disable
Harddisk
aWer
*me-­‐out
– ACPI
(1996
v1.0
open
standard
industry
specifica*on)
• OS
controls
PM,
BIOS
provides
API
• Power
Management
cri*cal
for
Mobile
Phones
–
supported
on
ARM
SoCs
with
the
usual
fragmenta*on
• Integra*ng
Power
Management
in
drivers
and
kernel
is
a
major
ac*vity
for
the
Mobile
OEM

21. Brief
Interlude:
Power
Management
• Power
Management
mechanics
– System
Suspend
–
User
forces
system
to
sleep
– CPU
idle
–
Idle
thread
triggers
sleep
states
– CPU
freq
–
CPU
frequency
scaling
– Run*me
PM
–
Device
Power
Management
– CPU
Hotplug
–
Remove
a
CPU
from
the
running
system
(ini*ally
used
for
Server
SMP
Linux
became
important
for
mul*-­‐core
ARM)
• Due
to
a
lack
of
a
standard
like
ACPI,
ARM
SoCs
support
is
fragmented
– ARM
SoCs
expose
a
lot
of
HW
knobs
to
SoWware,
directly
controlled
by
the
OS
drivers
– Each
SoC
vendor
exposes
a
superset
of
standard
ARM
power
states
for
fine-­‐
grained
control,
increasing
complexity
of
SoC
enablement
code
and
peripheral
drivers
– Different
approaches
to
OS
Power
Management
frameworks
due
to
a
lack
of
design
pauerns,
lack
of
common
infrastructure
and
intrinsic
HW
differences
– Linaro
approach
to
integrate
and
unify

22. hups://wiki.linaro.org/WorkingGroups/
PowerManagement/Doc/Architecture

23. Android
“wakelock”
• Default
state
is
to
suspend
• Kernel
implementa*on
and
Android
user-­‐space
APIs
– When
a
wakelock
of
type
“WAKE_LOCK_IDLE”
is
in
effect,
system
does
not
enter
Idle
(low
power)
– When
a
wakelock
of
type
“WAKE_LOCK_SUSPEND”
is
in
effect,
system
will
not
suspend
• Highly
controversial
feature,
mul*ple
patch
submissions
and
mailing
list
discussions
over
many
years
• Compa*ble
solu*on
being
mainlined
in
later
Kernel
3.x
series

24. Is
Android
the
new
“Embedded
Linux”
?

25. Android
“The
interes*ng
thing
about
Android’s
design
is
how
liule
we
modified
the
kernel.
Most
embedded
systems
on
which
I
have
worked
have
made
dras*c
changes
to
the
kernel,
only
to
leave
user-­‐space
alone
—
for
example,
a
heavily-­‐modified
“real*me”
kernel
but
X11
for
a
GUI.
Android
is
the
opposite:
Only
minimal
changes
to
the
kernel,
but
a
user-­‐space
wholly
unlike
that
of
any
other
Unix
system.
In
fact,
Android’s
user-­‐space
is
so
different
from
stock
Linux,
you
can
easily
say
that
Android
is
not
in
any
way
a
Linux
system,
except
for
the
kernel.”
-­‐
Robert
Love,
Google
Engineer
and
Kernel
Contributor
hup://www.forbes.com/sites/quora/2013/05/13/what-­‐are-­‐the-­‐major-­‐changes-­‐that-­‐android-­‐made-­‐to-­‐the-­‐linux-­‐kernel/

26. Android
• ashmem
(Android
Shared
Memory),
a
file-­‐based
shared
memory
system
• Binder,
an
inter-­‐process
communica*on
(IPC)
and
remote
procedure
call
(RPC)
system
• logger,
a
high-­‐speed
in-­‐kernel
logging
mechanism
op*mized
for
writes
• Paranoid
Networking,
a
mechanism
to
restrict
network
I/O
to
certain
processes
• pmem
(Physical
Memory),
a
driver
for
mapping
large
chunks
of
physical
memory
into
user-­‐space
• Viking
Killer,
a
replacement
OOM
killer
that
implements
Android’s
“kill
least
recently
used
process”
logic
under
low
memory
condi*ons
• wakelocks,
Android’s
unique
power
management
solu*on,
in
which
the
default
state
of
the
device
is
sleep
and
explicit
ac*on
is
required
(via
a
wakelock)
to
prevent
that
• The
usual
assortment
of
drivers,
ARM
architecture
ports,
and
other
associated
low-­‐level
code
necessary
to
support
Android
on
any
given
device
hup://www.forbes.com/sites/quora/2013/05/13/what-­‐are-­‐the-­‐major-­‐changes-­‐that-­‐android-­‐made-­‐to-­‐the-­‐linux-­‐kernel/

27. Android
• Android
is
the
most
popular
Embedded
Linux
“distro”
today
• From
hos*lity
to
gradual
adop*on
– E.g.
mainlining
of
selec*ve
Android
features
in
Linux
Kernel
– Ubuntu
“MIR”
display
server
across
Desktop/
Tablet/Phones
leverages
Android
display
drivers
and
architecture
– We
all
love
“adb”

29. Looking
ahead
…

30. Linaro
• Not-­‐for-­‐profit
engineering
organiza*on
consolida*ng
and
op*mizing
open
source
Linux
soWware
and
tools
for
the
ARM
architecture
• Focus
on
generic
ARM
technology
common
to
all
ARM
SoC
vendors
• Focus
on
areas
that
interact
directly
with
silicon
like
Mul*media,
Graphics,
Power
Management,
Kernel
and
Boot
• Maintenance,
up-­‐streaming
and
implementa*on
of
standards
• Established
in
2010
by
ARM,
Freescale,
IBM,
Samsung,
ST-­‐Ericsson
and
Texas
Instruments

31. Yocto
Project
• Not
an
embedded
Linux
distribu*on
–
Yocto
provides
tools
to
build
a
custom
embedded
Linux
distribu*on
• Based
on
Poky,
OpenEmbedded
• Provides
Linux
Kernel,
System
Libraries,
UI
Framework
• Op*onal
Sato
user
interface
based
on
GNOME
Mobile
Stack

32. Yocto
Project

33. Genivi
• Industry
non-­‐profit
alliance
to
drive
adop*on
of
open
source
In-­‐vehicle
Infotainment
development
planorm
• Launched
in
Mar
2009
• Linux-­‐based
kernel,
core
services,
middleware
and
open
applica*on
interfaces
• Genivi
compliant
–
Tizen,
MontaVista
IVI,
Ubuntu
Remix
IVI,
WindRiver
IVI

34. Tizen
• History
–
builds
on
SLP
from
LiMo,
formed
when
Intel
leW
Meego
to
join
Tizen
• Open-­‐source,
standards
based
planorm
for
Smartphones,
Tablets,
In-­‐
vehicle
Infotainment
Devices,
Smart
TVs
• Part
of
the
Linux
Founda*on
• Devices
from
Samsung
“later
in
2013”

35. Ubuntu
for
Smartphones
• Built
on
Ubuntu
Linux,
leverages
Android
Tech
• HTML5
and
Na*ve
Apps
are
equal
ci*zens
• Same
OS
across
devices,
same
OS
across
same
device
in
different
form
factors
• OEM
Customizable
• Phones
in
4Q2013/
early
2014

36. If
you’re
confused
…

37. Source:
hups://github.com/kumadasu/*zen-­‐history/blob/master/*zen-­‐history.pdf

38. Not
to
forget
…

39. Firefox
Mobile
• “Boot
to
Gecko”
-­‐>
Linux
+
Mozilla
Gecko
• Web
as
the
Applica*on
Planorm
• Leverages
HTML5
and
other
Web
Standards
• Devices
launching
“soon”

40. Sources
Google
and
Wolfram
Alpha
hup://www.embedded-­‐linux.co.uk/embedded-­‐history
hup://www.arm.linux.org.uk/docs/history.php
hup://lwn.net/Ar*cles/437162/
hup://www.linuxjournal.com/ar*cle/5755
hup://en.wikipedia.org/wiki/Carl_von_Clausewitz
hups://rt.wiki.kernel.org/index.php/Main_Page
hup://elinux.org/images/b/ba/Elc2013_Rostedt.pdf
hups://lwn.net/Ar*cles/518988/
hup://en.wikipedia.org/wiki/Sharp_Zaurus
hup://en.wikipedia.org/wiki/Nokia_770_Internet_Tablet
hups://wiki.linaro.org/WorkingGroups/
PowerManagement
hups://www.kernel.org/doc/Documenta*on/cpu-­‐
hotplug.txt
hup://elinux.org/Power_Management
hup://www.itworld.com/mobile-­‐wireless/175829/arm-­‐
and-­‐linux-­‐major-­‐construc*on-­‐ahead
hup://arstechnica.com/business/2012/08/from-­‐altair-­‐to-­‐
ipad-­‐35-­‐years-­‐of-­‐personal-­‐computer-­‐market-­‐share/
hup://www.pcworld.com/ar*cle/2042267/amd-­‐slates-­‐
first-­‐arm-­‐server-­‐chip-­‐seaule-­‐for-­‐2014.html
hup://www.slideshare.net/opersys/is-­‐android-­‐the-­‐new-­‐
embedded-­‐embedded-­‐linux-­‐at-­‐embedded-­‐wordl-­‐2013
hups://wiki.linaro.org/WorkingGroups/
PowerManagement/Doc/Architecture
hup://www.forbes.com/sites/quora/2013/05/13/what-­‐
are-­‐the-­‐major-­‐changes-­‐that-­‐android-­‐made-­‐to-­‐the-­‐linux-­‐
kernel/
hup://training.linuxfounda*on.org/free-­‐linux-­‐training/
download-­‐training-­‐materials/history-­‐of-­‐embedded-­‐linux-­‐
and-­‐best-­‐prac*ces-­‐for-­‐gemng-­‐started
hup://en.wikipedia.org/wiki/Linux_Router_Project
hup://www.extremetech.com/extreme/155392-­‐
interna*onal-­‐space-­‐sta*on-­‐switches-­‐from-­‐windows-­‐to-­‐
linux-­‐for-­‐improved-­‐reliability
hup://www.linaro.org
hup://yoctoproject.org
hup://*zen.org
hup://www.ubuntu.com/phone
hups://github.com/kumadasu/*zen-­‐history/blob/
master/*zen-­‐history.pdf
hup://www.mozilla.org/en-­‐US/firefox/os/