Heating, Ventilation and Air Conditioning (HVAC) is responsible for keeping us warm and comfortable indoors. In a modern building, data such as weather, occupancy, and equipment use are collected routinely and can be used to optimise energy use. Reinforcement learning (RL) is a good fit as it can learn patterns in the data and identify strategies to control the system so as to reduce energy consumption to up to 20%. In this session we will be training a reinforcement learning algorithm with Amazon SageMaker using an open source simulator, EnergyPlus, to train an agent to optimise energy usage. We will then publish this model in AWS Marketplace for other customers to reuse your machine learning solution.

1. S U M M I T
SYDNEY

2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Optimise energy usage using Amazon
SageMaker reinforcement learning and publish
your model in AWS Marketplace
Aparna Elangovan
Solutions Architect
Amazon Web Services

3. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Agenda
• The problem of energy optimisation for heating, ventilation, and air
conditioning (HVAC)
• Introduce reinforcement learning
• Reinforcement learning on Amazon SageMaker to optimise energy
usage in HVAC
• AWS Marketplace

4. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Business objective: Optimise energy in HVAC
HVAC consumes 40% of building
energy
1. Decrease energy usage
2. Improve occupant’s comfort

5. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

6. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Reinforcement Learning (RL)

7. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
How agents learn - A simplistic intuition
So what can we do?
• Randomly try different actions
• Compute average reward
• Assign higher transition probabilities to high
value actions
R =
-10000
Do
nothing

Reality mostly model free!
• State transitions, intermediate rewards,
probabilities are not known

8. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
How to simulate building energy consumption?
• Building type
• Building materials
• …..
• Human activity
• Lights, bulbs,…..
Input
• EnergyPlus
Simulator
(Environment)
• Bulb1 300 watts
per hour
• Cooler temp
• …
Output – Energy
consumed +
Current State
Environment

9. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
What about action and state?
Action
Space (At)
• Cooler set point
• Heater set point
Observation
Space (St)
• Zone temp
• Outside temp
• Relative humidity

10. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Reward function
Designing a reward function requires domain knowledge
Aim to maximise total reward in the long run
Environment

11. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
The agent
Learns the right action based on state and reward
Map transition probability from state to action (aka policy)
How? Using reinforcement algorithms
Environment

12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
The coding- RL frameworks on Amazon SageMaker
Coach
• Agent RL algorithms
• Compatible with OpenAI
Gym Environments
OpenAI Gym
• Provides unified interface
for the environment
• Your own agent
RLlib
• Scalable Reinforcement
Learning
Supported on SageMaker
RL deep learning - TensorFlow, MxNet
Preconfigured RL libraries
Docker based - built-in or BYO
Hyperparameter tuning
Support distributed training

13. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Challenges using RL
Simulate
real world
Model the
environment
Represent the
observable
environment
State
Align reward
with real
world
Long term vs
short term
rewards
Reward
function

15. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

16. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
What we heard from our customers
RE IN VENTING T HE WHE E L
Significant time and valuable
resources are spent in developing
ML solutions to problems that are
already solved by others
POOR S E L E CTION
It is hard to find, evaluate, and qualify
trustworthy algorithms
and models for enterprise use cases
L OS T OPPORT UNITY
Difficulty in deployment, version
management, and reproducibility
led to delay in time to market
L ACK OF DAT A/IP S E CURIT Y
No way to ensure data security,
compliance needs, or regulatory
standards for customer’s data
and intellectual property

17. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
AWS ML Marketplace - What can you buy or sell?
Algorithms let you
train a custom model
using your own data;
You can also tune
your algorithm using
hyperparameters
Model packages
are pretrained
by your seller
and ready-to-use;
you can run an
inference in
real-time or
batch mode

18. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Machine Learning on AWS algorithm/model
pricing and licensing
Hourly
Free trials
Free
Metered
Paid

19. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Summary
Energy optimisation has significant financial and environmental reward
Assessing the implementation complexity is key to successful AI implementation
AWS SageMaker reduces some of the technical challenges and supports open source frameworks
Before you build look for what is out there , such as AWS Marketplace AI Application Services,
built-in Algorithms and models
Quick wins are well studied, well implemented AI solutions

20. Thank you!
S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Aparna Elangovan