Coscup2018 itri android-in-cloud

Copyright 2016 ITRI 工業技術研究院 0
Android-in-cloud, a combination of
android-x86 open source project and
Linux container technology
COSCUP 2018
工業技術研究院
資料中心系統軟體組
吳天健 (tcwu2005@gmail.com)

1Copyright 2018 ITRI 工業技術研究院
講者自介
With over 15 years of experience in
developing assembly, firmware, embedded
software and video encoding algorithm on
variants of CPUs including x86, Arm, and
proprietary processor.
I joined ITRI Cloud Computing center of
Mobile Application (CCMA) as a system
software engineer in 2014. And I've been
working on the project "Cloud Execution
Platform for Android Containers” since
2016.
My research interest includes Hypervisor,
Containerization, and Linux kernel.

回顧
❖ We were speakers of
COSCUP 2017
• https://youtu.be/9W_ZMyany-
0?t=42m27s
❖ In January, 2018, ITRI ICL
released the virtual
smartphone implementation
as “CLONDROID” open
source project on github.
The project enables one
Google Pixel XL handset to
run multiple tailored Android
containers on it.
Android Container on
Google Pixel
https://github.com/clondroid

Clondroid: Multiple Your Mobile Phone
Linux Kernel + LXC + Drivers
Virtual Devices namespace-aware binder
Shared service invocation
Unshared service invocation
Linux Kernel + LXC + Drivers
Surface Flinger buffer
Host Android
Shared Framework
Services
Surface Flinger
Window Manager
Android n
Window Manager
A container-based virtual
smartphone open source project
initiated by ITRI
You are welcome to visit
https://github.com/clondroid/cba
-aosp-marlin-7

Clondroid
Display Switching between Containers
• Layers of different Android
containers in SurfaceFlinger
• SurfaceFlinger groups layers
according to their container
ID
Assign layers of host container to
layersSortedByZ of primary display
primary display
compose
Assign layers of container 1 to
layersSortedByZ of primary display
primary display
compose

App Streaming in Gaming Industry

The Needs for App Streaming Cloud
Infrastructure
❖ Virtual Android in The Cloud Enables
➢ VMI as a better solution for the BYOD
➢ Mobile cloud-based Gaming-on-Demand services
➢ Cloud gaming bot - online 24/7
❖ A Case Study - Redfinger Cloud Phone
➢ 5 million subscribers in 2016
➢ Services and Charge
• $9.95 USD / per month for 8 GB Storage, 4 GB RAM
❖ Flexible and Cost Effective Infrastructure
Is The Key

LXC meets Android
❖ To host Android VMs in cloud
➢ Hypervisor
➢ Emulator
➢ LXC + Android
❖ LXC
➢ LXC (Linux Container) = Linux namespace + Cgroup
❖ Android and Its Software Stack Comprises
❖ To containerize Android for cloud, it covers
➢ Linux system
➢ Complete Android-namespaces that Linux namespaces
does not
➢ Relative cloud system design
(LXC介紹摘錄自wiki)
目前的LXC使用下列內核功能來控制行程：
• 內核命名空間（行程間通訊(IPC)、
uts、mount、pid、network和user）
• AppArmor和SELinux配置
• Seccomp策略
• chroot（使用pivot_root）
• Kernel Capibilities
• 控制組（cgroups）
因此，LXC通常被認為介於「加強版」的
chroot和完全成熟的虛擬機器之間的技術。
LXC的目標是建立一個儘可能與標準安裝的
Linux相同但又不需要分離內核的環境。

Linux vs Android in Software Stack
• The Android operating system
has its own C library (為了輕薄
短小)called Bionic whereas
Linux systems use GNU C
library.
• Android owns its specific
kernel driver, Binder (see
kernel/drivers/staging/android/)
這個目錄還有其他的如lmk,ashmem,alarm
• Application execution model
(i.e. Dalvik VM)
• Rootfs (Android’s startup
script beginning from init.rc)
Source, Aug7,2018 from http://svholla.net/conndots.html

Why BINDER is crucial in Android?
• The existing model is called
Android native code
• The new one, 依賴BINDER的
Server-client model
• Applications and Services may run in
separate processes but must communicate
and share data.
IPC的花費鉅大
• The solution is binder driver to facilitate inter-
process communication (IPC)
• BINDER not only provide software
service
• Starting in Android O, the Android framework
and HALs now communicate with each other
using binder
參考1, Studying Main Differences between Android & Linux Operating Systems Hadeel Tariq Al-Rayes M.Sc. Computer Science Basic
Education College-Computer Science Department Diyala University
參考2,The Device Driver Structure for Android with Linux Kernel Driver and Android HAL, William W.-Y. Liang (梁文耀),

Binder Namespace
❖ Device Namespace
➢ Laadan and his colleagues at Cellrox have put together a set of patches
based on the 3.4 kernel for the Android emulator (goldfish)
➢ https://github.com/Cellrox/devns-
patches/blob/master/dev_ns__adopt_the_android_binder_driver
❖ Google’s Android binder patch
➢ https://lkml.org/lkml/2016/10/24/339. (patch)
➢ https://sourceforge.net/p/android-x86/kernel/ci/kernel-
4.4/tree/drivers/android/. (directory)
Device namespaces, https://lwn.net/Articles/564854/,LWN ,Jake Edge, August 28, 2013
From Martijn Coenen <>
Subject [PATCH 06/10] android: binder: support multiple /dev instances.
Date Mon, 24 Oct 2016 15:20:34 +0200
share
Add a new module parameter 'devices', that can be
used to specify the names of the binder device
nodes we want to populate in /dev.

Android Device (HAL)
❖ Android HAL Architecture
Linux Device Drivers
The new comer:
Android HAL
參考, The key issues for teaching or learning Android and Linux Kernel 梁文耀, CTO 和沛移動股份有限公司

Android Device Virtualization
❖ Device/HALs
➢ Audio
➢ Bluetooth
➢ GPS
➢ Camera
➢ Sensors
➢ Ethernet
➢ Backlight
➢ …族繁不及備載…
 Why Android-x86 is a good
baseline of Android
Containerization
• Sensor example
https://sourceforge.net/p/android-
x86/hardware_libsensors/ci/nougat-
x86/tree/
❖Audio Device Emulation
• https://github.com/SVMP/android_de
vice_mitre_svmp/tree/svmp-
2.x/libaudio

Container1
App App App
Android Framework
Surface
flinger
Windows
Manager
Service
Manager
Input
Manager
Container2
App App App
Android Framework
Surface
flinger
Windows
Manager
Service
Manager
Input
Manager
Container3
App App App
Android Framework
Surface
flinger
Windows
Manager
Service
Manager
Input
Manager
Android as Host
LXCTools
namespaces
Graphic Driver Binder Driver
Keyboard Driver
Mouse Driver
Touch-Panel
Other
Device Drivers
Display
Virtualization
Binder (IPC)
Virtualization
Input
Virtualization
…...
Device
Emulati
on
…...
Device
Emulati
on
…...
Device
Emulati
on
❖ Android-in-cloud DEMO
➢ https://youtu.be/UkxE0e7m1D4
❖ Android-in-cloud software stacks
Android-in-cloud

Android scheduling
❖ Android’s foreground/background process
scheduling
14
/dev/cpuctl/tasks
/dev/cpuctl/bg_non_interactive/tasks
cpu.shares=1024
cpu.shares=50
ActiveManager
jni/android_util_
process.cpp
libcutils/sched_
policy.c
How set_sched_policy() is called

❖ Senario
➢ cpu.shares
• specifies a relative share of CPU time available to the tasks in a cgroup
➢ Android assigns system services to bg_non_interactive group
➢ In order to maintain fame rate @ specific level, Android assigns
@ least 5% of CPU resources to bg_non_interactive group
❖ Symptom
➢ As the # of containers increases, App UI update
frequency is not stable
❖ Reason
➢
16ms
UI Update (ms/per frame)
2 containers
system service process cpu.share
s
Host
only
system_server, com.android.systemui 5%
+ con1 system_server, com.android.systemui
system_server, com.android.systemui
5%
+ con2 system_server, com.android.systemui
5%
But there are other Androids…

Container-aware Scheduler
❖ Solutions
➢ Improve schedule policy, make it to
understand there are containers
➢ Assign separated subgroup and
shares value to each android
container
16ms
before
UI Update (ms/per frame), 2 containers
after
system service process cpu.shares
Host
only
system_server,
com.android.systemui
5%
+ con1 system_server,
system_server,
10%
+ con2 system_server,
system_server,
system_server,
15% DEMO@
https://youtu.be/e7
FpZojOj5c

Network topology design in
container-based cloud
❖ Linux bridge , a pure
software switch
❖ A simple block of script to
enable Linux bridge
br0
eth0 veth1 veth2
Android Container 1
eth0
Android Container 2
eth0
DHCP
Server
SVMP
Server
Firewall
LAN
Linux Bridge
FOUND=`grep br0 /proc/net/dev`
if [ -n "$FOUND" ] ; then
echo "br0 exists, skip"
else
brctl addbr br0
ifconfig eth0 0.0.0.0
brctl addif br0 eth0
dhcptool br0
fi

Pitfalls of Linux bridge
❖ You don’t need to enable IP forwarding
❖ Unexpected bridge netfilter behavior
❖ Be sure in promiscuous mode
### NOT NECESSARY TO ENABLE ROUTING ###
###echo 1 > /proc/sys/net/ipv4/ip_forward
### IF THERE IS BRIDGE NETFILTER, DISABLE THEM
###
for f in /proc/sys/net/bridge/bridge-nf-* ; do echo 0 >$f ; done
make sure promiscuous mode enables
● in hypervisor, check network setting
● in physical machine
For promiscuous mode, IFF_PROMISC, 0x100 should be set.
$ cat /sys/devices/virtual/net/veth0/flags
0x1303 # 0001 001[1] 0000 0011 # device is in promiscuous mode.
$ cat /sys/devices/virtual/net/br0/flags
0x1003 # 0001 000[0] 0000 0011 # device is not in promiscuous mode.

GetMAC()
{
if [ -n "$1" ]; then
##OID="02:16:3e"
OID_PREFIX="02"
OID_SUFIX=$(cat /sys/class/net/peth0/address | md5sum | sed
's/(..)(..).*/1:2/')
OID="$OID_PREFIX:$OID_SUFIX"
echo OID=$OID
RAND=$(echo $1 | md5sum | sed 's/(..)(..)(..).*/1:2:3/')
echo "$OID:$RAND"
else
echo "ERROR: please supply hostname to create MAC address from, e.g.:"
echo " $FUNCNAME myhost"
fi
}
Mac address assignment
In a container-based cloud, mac addresses of each container
should be
 Pre-assigned
 Lower address (br0 automatically take away new enslaved nic card’s
mac address if it’s lower)
 Manageable (設想你有上百甚至上千個容器要管理,要生成,指派,出錯的時
候大略判定mac所屬的區段)
tcwu2005@itri-29:~$ GetMAC con1
02:9f:8b:8e:52:8e
02:9f:8b:17:52:5b
02:9f:8b:37:1b:c9

X86 Server GPU Test
container# avg.fps sum.fps
1 60 60
5 60 300
10 34.6 346
15 24.2 363
20 18.5 370
30 13.97 419
• Hardware: rack mount server MI1200
• GPU: Intel embedded core ‘gt4e’, 2/3 of
area 210-220mm2 is for gpu
• In each container , run test app ‘seascape’ ,
a small size (4MB) GPU benchmark with
highly realistic dynamic ocean render using
OpenGL ES 2.0 and specific glsl shader
features
0
20
40
60
80
0 10 20 30 40
avg.fps
0
50
100
150
200
250
300
350
400
450
0 5 10 15 20 25 30 35
sum.fps
1. Container numbers of 1 and 5 , average fps is 60
and no more than that, guess it’s fixed to system
(system fps = 1/vsynv_period)
2. More benchmark applications executed
simultaneously, the total bandwidth of gpu
reaches around four hundreds of frames per
second

ITRI OPEN SOURCE PROJECTS
❖ Cuju: An Open Source
Project for QEMU/KVM
based Fault Tolerance
❖ CLONDROID: A Container
based Virtual Smartphone
Open Source Project to allow
you to run multiple Android-OS
on single Google Pixel XL.
Separate work and personal data
easily.
https://github.com/clondroid

工研院 (ITRI) COSCUP 2018 徵才活動
# 徵才ing
❖ 工研院資通所目前正在尋求具有以下
經驗跟能力的人才：
- 系統安全
- 軟體安全
- 虛擬化系統
- Container (LXC)
- IaaS / PaaS / SaaS
- Android 作業系統
- 軟體定義式儲存系統
- 快閃記憶體磁碟陣列系統
- 深度學習
- DevOps
- UI / UX engineering
- I/O 虛擬化
- CEPH Software Storage Platform
❖ 請留下您的聯絡方式，並告訴我們您
有興趣的職缺。
# We are hiring!
❖ We are ITRI Information and
Communication Labs (ICL). We need
people who have the following
talents:
- System security
- Software security
- Virtualization
- Container (LXC)
- IaaS / PaaS / SaaS
- Android operating system
- Software-defined storage
- All flash array
- Deep learning
- DevOps
- UI / UX engineering
- I/O virtualization
- CEPH Software Storage Platform

Container-Based-Android Team
Members
Tzi-cker Chiueh,
General Director of ICL/ITRI
Victor Hsu Tian-Jian Wu Te-Yu Tsai I-Fan WangIan TsaiSting Cheng

Coscup2018 itri android-in-cloud

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Coscup2018 itri android-in-cloud

Similar a Coscup2018 itri android-in-cloud (20)

Último

Último (20)

Coscup2018 itri android-in-cloud