This document describes DeepRemote, a smart remote controller that uses deep learning for intuitive home appliance selection and control. It consists of a control unit with a camera and buttons and a deep learning unit for appliance recognition. The system was tested for classification accuracy of over 80% on average, response time of under 2 seconds, and faster control times than traditional remotes in user tests. Overall, DeepRemote demonstrates an effective deep learning approach for selecting and controlling home appliances intuitively with a single remote controller.

1. DeepRemote: A Smart Remote Controller for
Intuitive Control through Home Appliances
Recognition by Deep Learning
〇Yuta Takahashi1, Naoki Shirakura1, Kenta Toyoshima1, Takuro
Amako1, Ryota Isobe1, Jun Takamatsu1 and Keiichi Yasumoto1
1. Nara Institute of Science and Technology
The Tenth International Conference on Mobile
Computing and Ubiquitous Networking (ICMU2017)
S7: Application, October 5, 2017

2. Background
❖Increasing home appliances
2
❖Many IR remote controllers
▪ Need to learn how to use
▪ High management costs
Increase user’s burden

3. Unification of remote controllers
3
Network Multiple IR controller
Increase the number of
appliances
The interface becomes
complicated
Method for selecting home appliance is important!
✔ Management cost

4. Home appliance selection
4
❖With special attachments
❖Without special attachments
✔ Accurate selection
Cost of devices/markers
Selectable distance
Voice [Pan 2010] Vision [Kong 2016]
Intuitiveness ✔ Intuitiveness
IR control
Burden for wearing
• IR LED [Neßelrath 2011]
• IR transmitter [Tsukada 2004]
• QR [Ullah 2012]
Dedicated device is needed

5. DeepRemote (Proposed)
5
❖Home appliance selection
Object recognition by deep learning
→ Intuitive and robust
❖Two units
▪ Control unit
▪ Deep learning unit
❖Network
Home network (ROS)

6. Control unit
6
✓ Hand-held-type remote controller
❖Main processor
▪ Raspberry Pi3 (with Wi-Fi module)
❖Capturing a home appliance
▪ Front camera
❖Control interfaces
▪ Four buttons
▪ Gestures (right & left rotation)

7. Deep learning unit
7
✓ Image recognition
❖Main processor
▪ Laptop PC with Core i5 (Ubuntu)
▪ Distribute the calculation load of control unit
❖Recognition Model
▪ VGG16 [Simonyan 2014]
TV
Recognition
result
Deep
learning unit
Control unit

8. Technique of training
❖Requirements of deep learning with zero-base
8
Huge dataset A lot of time
❖Fine tuning
Trained model Arranging model
Re-training

9. Experiment
9

10. Experimental environment
10
✓Living room in smart-home facility
❖Five home appliances
▪ Fan (IR)
▪ Air conditioner (IR)
▪ TV (Network)
▪ Audio player (IR)
▪ Air purifier (Network)
❖Training data
▪ Took 20 images each appliances at P1, P2 and P3
▪ 20 × 5 (appliances) × 3 (positions) = 300 images

11. Example of training data
11
P1
P2
P3
Air
purifier
Air
conditioner
Audio
player Fan TV

12. Training
❖Model
▪ VGG16 trained ImageNet
❖Dataset
▪ Three hundred images
▪ Five classes
12
❖Optimizer
▪ SDG
❖Loss function
▪ Categorical cross entropy

13. Evaluations
1. Classification accuracy
▪ Captured 50 images of each appliance in each position
▪ 50 (images) × 5 (appliances) × 3 (positions) = 750 images
13
2. Response time
▪ Measured time since pushing button until return the result
▪ Evaluated at the same time as 1.
3. User test
▪ Verified control time of home appliances

14. Appliance Precision [%] Recall [%] F-measure [%]
P1
Air purifier 100.00 68.00 80.95
Audio player 83.33 90.00 86.54
TV 96.15 100.00 98.04
Air conditioner 57.47 100.00 72.99
Fan 86.96 40.00 54.80
Average 84.78 79.60 78.66
P2
Air purifier 81.63 80.00 80.81
Audio player 100.00 98.00 98.99
TV 92.59 100.00 96.15
Air conditioner 79.25 84.00 81.55
Fan 100.00 90.00 94.74
Average 90.69 90.40 90.45
P3
Air purifier 76.36 84.00 80.00
Audio player 100.00 58.00 73.42
TV 71.43 80.00 75.47
Air conditioner 53.17 84.00 65.12
Fan 100.00 62.00 76.54
Average 80.19 73.60 74.11
1. Classification accuracy of home appliances
14
P1:Effect of
black door?
P3: Too near?

15. Evaluations
1. Classification accuracy
▪ Captured 50 images of each appliance in each position
▪ 50 (images) × 5 (appliances) × 3 (positions) = 750 images
15
2. Response time
▪ Measured time since pushing button until return the result
▪ Evaluated at the same time as 1.
3. User test
▪ Verified control time of home appliances

16. 2. Response time
16
(n=750)
Maximum time: 3.07 [sec]
Minimum time: 1.72 [sec]
Stable recognition about
two seconds

17. Evaluations
1. Classification accuracy
▪ Captured 50 images of each appliance in each position
▪ 50 (images) × 5 (appliances) × 3 (positions) = 750 images
17
2. Response time
▪ Measured time since pushing button until return the result
▪ Evaluated at the same time as 1.
3. User test
▪ Verified control time of home appliances

18. 3. User test to assess control time (1/3)
Experimental conditions
Five participants
Position: P1
Targets: Fan, Air conditioner, TV and Audio player
18
❖Control time ❖Comparison
Holding Power on DeepRemote Original
vs

19. 3. User test to assess control time (2/3)
19
F:86% F:98% F:77% F:55%
High → ✔ about 5 seconds
Lower than the 85% → over 10 seconds
Accuracy
Accuracy requires
85% over

20. 3. User test to assess control time (3/3)
20
DeepRemote: Sum of each test > All
Original: Sum of each test < All
Changing time
DeepRemote < Original
Sum Sum
31.533.7
11.7
7.7
(- 2.1)
(+ 3.0)

21. Conclusions
❖DeepRemote
▪ Smart device for intuitively control the home appliances
▪ Deep learning method for home appliance selection
❖Results of evaluation
▪ 81.07% classification accuracy on average
▪ Average of response time is 1.97 seconds
▪ Time of power on an appliance takes 5 seconds
▪ 85% accuracy is required
▪ Time of changing target is lower than original remote
controller
21

22. Appendix
22

23. Future works
❖Improving recognition accuracy
▪ Cropping object
❖Investigating energy consumption
❖Accuracy of distinguishing similar appliances
❖Experiment in long-term usage
▪ User teaches correct label when the system recognizes
wrong
23

24. Detail conditions of user test
❖Compare DeepRemote and original remote controller
▪ measure control time
▪ Remove “air conditioner” (no original controller)
▪ Five participants (males in the 20s)
▪ Perform three times in each measurement (one participant performs
30 times)
▪ Participant’s position: P1
24
1) Measure one appliance control time
(start) → (power on an appliance)
2) Measure all appliances control time
(start) → (power on the fan)
→・ ・ ・ → (power on the audio player)

25. Confusion matrix of P1
25
Classified
Air purifier
Air
conditioner
Audio
player
Fan TV
True
Air purifier 34 6 9 0 1
Air
conditioner 0 50 0 0 0
Audio player 0 1 45 3 1
Fan 0 30 0 20 0
TV 0 0 0 0 50

26. Confusion matrix of P2
26
Classified
Air purifier
Air
conditioner
Audio
player
Fan TV
True
Air purifier 40 6 0 0 4
Air
conditioner 8 42 0 0 0
Audio player 1 0 49 0 0
Fan 0 5 0 45 0
TV 0 0 0 0 50

27. Confusion matrix of P3
27
Classified
Air purifier
Air
conditioner
Audio
player
Fan TV
True
Air purifier 42 5 0 0 3
Air
conditioner 7 42 0 0 1
Audio player 3 6 29 0 12
Fan 0 19 0 31 0
TV 3 7 0 0 40