hybriData IIoT Workshop for AAPG Short Course

AAPG Short Course:
Improving Modeling and Predicting Reservoir Behavior
January 15th, 2019
Marathon Oil Tower - Houston, TX
Execute Faster + Lower Total Cost of
Ownership:
A Hands on Energy Domain Based IIoT
Serverless Coding Workshop
CJ Ejimuda-MS
Hybrid Data Solutions, Los Angeles, CA, USA
email: cj@hybridata.us

CJ Ejimuda
Full Stack Data Scientist / Principal
Hybrid Data Solutions
Mine more value leveraging AI, IIoT, Big Data
Domain Expertise in Reservoir and Production Engineering
ExxonMobil, Aera Energy
2
A little about us
AAPG Short Course: Improving Modeling and Predicting Reservoir Behavior
January 15, 2019 - Houston, TX

Course Outline
● Objectives
● Introduction
● Limitations and Benefits of Serverless
● Use cases
● Hands on Coding Session:
- MAC / Window PC / Unix environment set up
- Hardware and sensors local configuration
- Telemetry data streaming local validation
- IIoT Device security registration, configuration and connection
- Serverless environment configuration and cloud resources provisioning - AWS IoT, Lambda, S3,
DynamoDB, Kinesis, Api Gateway, IAM, Cloudwatch, Elasticsearch, Kibana
- Telemetry Data Ingestion to AWS
● IIoT Device Data Management, Analytics, Visualization and Exploration
● References

4
Objectives
● Learn how to configure IIoT sensors onto gateway devices
● Ingest, manage, analyze, search and visualize associated telemetry
data on the cloud serverlessly

5
Introduction
● IIoT - Use of network-connected devices, separated or embedded in
an industrial equipment to record, monitor, control and improve an
existing industrial process or to enable a new capability previously
unrealized
Config. 1 Config. 2

6
● Serverless - cloud computing execution model in which the cloud
provider acts as the server, dynamically managing the allocation of
machine resources
Introduction

7
Limitations of Serverless
● Handling big data computations and algorithms are ineffective
● Expensive for continuous real time computation
● Cloud provider lock in

8
Benefits of Serverless
● Little to negligible upfront investment
● Faster time to market
● Converts CAPEX to OPEX
● Pay for only services you use

9
Use cases
● G&G:
- Task: Generate 4-D seismic image of target reservoir
- Monitor / Record: Geophones , hydrophones
- Control: Vibrator trucks, airguns
- Improve: Estimate the likelihood of commercial hydrocarbon quantity
● Reservoir / Production/ Drilling / Facility Engineer:
- Reservoir surveillance / simulation
- Production optimization
- Facility capacity checks
- Predrill, while drilling and post drilling analysis

10
Hands on Coding Session

11
IIoT Gateway Setup - Raspberry Pi
- Download and install an SD card formatter such as SD Memory Card Formatter or PiBakery. Insert the
SD card into your computer. Start the program and choose the drive where you have inserted your SD
card. You can perform a quick format of the SD card
- Download the Raspbian Stretch operating system as a .zip file
- Using an SD card-writing tool (such as Etcher), follow the tool's instructions to flash the downloaded zip
file onto the SD card. Because the operating system image is large, this step might take some time.
Eject your SD card from your computer, and insert the microSD card into your Raspberry Pi
- For the first boot, we recommend that you connect the Raspberry Pi to a monitor (through HDMI), a
keyboard, and a mouse. Next, connect your Pi to a micro USB power source and the Raspbian
operating system should start up

12
- You might want to configure the Pi's keyboard layout before you continue. To do so, choose the
Raspberry icon in the upper-right, choose Preferences and then choose Mouse and Keyboard Settings.
Next, on the Keyboard tab, choose Keyboard Layout, and then choose an appropriate keyboard variant
- Next, connect your Raspberry Pi to the internet through a Wi-Fi network or an Ethernet cable
NB: Connect your Raspberry Pi to the same network that your computer is connected to, and be sure that both your computer and Raspberry Pi have
internet access before you continue. If you're in a work environment or behind a firewall, you might need to connect your Pi and your computer to the
guest network to get both devices on the same network. However, this approach might disconnect your computer from local network resources, such as
your intranet. One solution is to connect the Pi to the guest Wi-Fi network and to connect your computer to the guest Wi-Fi network and your local network
through an Ethernet cable. In this configuration, your computer should be able to connect to the Raspberry Pi through the guest Wi-Fi network and your
local network resources through the Ethernet cable.

13
- You must set up SSH on your Pi to remotely connect to it. On your Raspberry Pi, open a terminal
window and run the following command: sudo raspi-config
- You should see the following:
- Scroll down and choose Interfacing Options and then choose P2 SSH. When prompted, choose Yes.
(Use the Tab key followed by Enter). SSH should now be enabled. Choose OK. Use the Tab key to
choose Finish and then press Enter. Lastly, reboot your Pi by running the following command:
sudo reboot
- On your Raspberry Pi, run the following command in the terminal: hostname -I
This returns the IP address of your Raspberry Pi. For the following, if you receive an ECDSA key fingerprint message (Are you sure you want to continue
connecting (yes/no)?), enter yes. The default password for the Raspberry Pi is raspberry.
If you are using macOS, open a terminal window and enter the following: ssh pi@IP-address

14
- IP-address is the IP address of your Raspberry Pi that you obtained by using the hostname -I command.
If you are using Windows, you need to install and configure PuTTY. Expand Connection, choose Data,
and make sure that Prompt is selected:

15
- Next, choose Session, enter the IP address of the Raspberry Pi, and then choose Open using default
settings.

16
- If a PuTTY security alert is displayed, choose Yes.The default Raspberry Pi login and password are pi
and raspberry, respectively.
NB: If your computer is connected to a remote network using VPN, you might have difficulty connecting from the computer to the
Raspberry Pi using SSH.

17
- If you are running Raspbian Jessie, run the following commands to update the Linux kernel version of
your Raspberry Pi:
sudo apt-get install rpi-update
sudo rpi-update b81a11258fc911170b40a0b09bbd63c84bc5ad59
- To activate the new firmware, reboot your Raspberry Pi:
sudo reboot
- After about a minute, reconnect to the Raspberry Pi using SSH. Next, run the following command to
ensure you have the correct kernel version: uname -a
- You should receive output similar to the following. In this example, the Linux Raspberry Pi version
information is 4.9.30:

18
- To improve security on the Pi device, enable hardlink and softlink protection on the operating system at
startup
- Navigate to the 98-rpi.conf file.
> cd /etc/sysctl.d
> ls
- If you don't see the 98-rpi.conf file, follow the instructions in the README.sysctl file.
- Use a text editor (such as Leafpad, GNU nano, or vi) to add the following two lines to the end of the file. You might need to
use the sudo command to edit as root (for example, sudo nano 98-rpi.conf)
fs.protected_hardlinks = 1
fs.protected_symlinks = 1
- Reboot the Pi : sudo reboot
- After about a minute, connect to the Pi using SSH and then run the following command to confirm the change:
sudo sysctl -a 2> /dev/null | grep fs.protected
- You should see fs.protected_hardlinks = 1 and fs.protected_symlinks = 1.

19
Serverless Environment Setup
- Sign up for an AWS Account
- AWS CLI installation: Follow the link for MAC or Windows to install
- Configure AWS CLI
- Terminal configuration for Windows users:
Windows 10 users linux terminal configuration follow this link
Windows 10 or Less follow this link
You may need to install and configure PuTTY for Windows users (Optional) - Old
- Install npm
- Install Serverless framework: npm install -g serverless

20
- Configure Serverless framework with AWS account from here
- git clone https://github.com/cjhD/aapg_hD_IIoT_shortcourse
- cd aapg_hD_IIoT_shortcourse
- Run command to create certificate, private and public key pair to be used in serverless configuration file and rpi device
> aws iot create-keys-and-certificate --set-as-active --certificate-pem-outfile ./awsIoTConf/cert.pem
--public-key-outfile ./awsIoTConf/publickey.pem --private-key-outfile ./awsIoTConf/privatekey.pem --output text
--query certificateArn > certArn.txt
- Copy certArn.txt value to iotThingCertArn object in iiot_arn_config.yml file

21
- Get the root CA certificate from here. Copy the root CA text from the browser, paste it into a file
- Move the rootCA certificate to the awsIoTConf folder
- Grab os.environ['AWS_IOT_ENDPOINT'] in read_data.py: aws iot describe-endpoint --endpoint-type iot:Data-ATS
- npm init; accept all defaults ; npm install --only=dev serverless-python-requirements
- Install virtualenv: pip install virtualenv and setup the environment using: virtualenv -p python3 aapg_iiot; source
aapg_iiot/bin/activate
- Run: pip install -r requirements.txt
- Edit .env file with AWS_ACCOUNT_ID and run: source .env
- Provision AWS cloud resources using: sls deploy -v

22
Gateway and Sensors Configuration
- We would be configuring DHT11 Temperature and Humidity Sensors to a Raspberry Pi 3 B gateway device
- GND pin: first connection, Signal / Data pin: second connection, VCC pin: last connection

23
IIoT Device Configuration
- ssh into raspberry pi - sudo ssh pi@raspberryPi(*IP_Addr)
- Move certificate, private key and root CA certificate from config folder on local machine into rpi - Mac /
Linux
sudo scp -r ./awsIoTconf pi@raspberrypi:/home/pi/
- Move certificate, private and public key files from config folder on local machine into rpi - Windows
Head to: https://filezilla-project.org/
Download the client - Not server version
Install following the installation wizard

24
IIoT Device Configuration
Within Raspberry Pi Device:
- Install AWS IoT Python SDK:
> git clone https://github.com/aws/aws-iot-device-sdk-python
> cd aws-iot-device-sdk-python
> sudo python setup.py install
- Download DHT11 sensor python library:
> git clone https://github.com/szazo/DHT11_Python
- Navigate to the folder: cd DHT11_Python
- Copy read_data.py to DHT11_Python folder on rpi:
> sudo scp read_data.py pi@raspberrypi:/home/pi/DHT11_Python
> Change os.environ['AWS_IOT_ENDPOINT'] to AWS IOT MQTT broker address obtained earlier
> Specify correct full path for certificate and keys on line 16
> Run: python read_data.py - telemetry data should start streaming

25
IIoT Device Management and Analytics
- Edit .env file with AWS_ACCOUNT_ID and run: source .env
- Map IIoT telemetry data by running: sls invoke local -f aapghdiiot -l (successful if “null” is returned)
- Install aws-es-proxy:
Mac: brew install aws-es-proxy
Windows(Choose processor type): link
- Configure AWS IAM credentials:
export AWS_ACCESS_KEY_ID=MY_ACCESS_KEY
export AWS_SECRET_ACCESS_KEY=MY_SECRET_KEY
. - Get elasticsearch domain url: echo https://$ENDPOINT
- Run: aws-es-proxy -endpoint AWS_ELASTIC_SEARCH_DOMAIN_URL
- Navigate to http://localhost:9200 on the browser

26
IIoT Data Visualization and Exploration
- Navigate to Kibana using: http://localhost:9200/_plugin/kibana/
- Setting up Kibana:
- Select management

27
- Type in index value defined on serverless.yml file “iotdata”

28
- Select your time filter field name as timestamp

29
- What you see after hitting the create index pattern button

30
- Select your type of charts as a line chart

31
- Set up your y-axis fields as temperature and humidity while aggregating their values as average

32
- Similarly, select using the Date Histogram for aggregation and field as timestamp for the Buckets section (x-axis) and hit the
highlighted play button on the right of Panel Settings

33
- IIoT Data streamed after 2 hours

34
- IIoT Data streamed after 4 hours

35
Deleting cloud resources and services
- On a separate terminal: source aapg_iiot/bin/activate ; source .env
- Configure AWS IAM credentials:
export AWS_ACCESS_KEY_ID=MY_ACCESS_KEY
export AWS_SECRET_ACCESS_KEY=MY_SECRET_KEY
. - Stop IIoT device from streaming data: ctrl + c
- Delete elasticsearch data and index: curl -X DELETE ‘http://localhost:9200/_all’
- Remove all cloud resources and services: sls remove -v
- One more to go: Navigate to dynamoDB console and delete table created
- And that’s a wrap!

36
References
● https://docs.aws.amazon.com/greengrass/latest/developerguide/module1.html
● https://docs.amazonaws.cn/en_us/iot/latest/developerguide/iot-dg.pdf
● https://serverless.com/framework/docs/providers/aws/
● https://techblog.calvinboey.com/raspberrypi-aws-iot-python/
● https://serverless.com/blog/build-geosearch-graphql-api-aws-appsync-elasticsearch/
● https://aws.amazon.com/iot-core/features/

37
Questions?
email: cj@hybriData.us
web: https://hybriData.us
Tell me what you think:
https://www.surveymonkey.com/r/HJ9FK7N

38
Thank you!

39
Back Up

hybriData IIoT Workshop for AAPG Short Course

Recommended

Recommended

More Related Content

Similar to hybriData IIoT Workshop for AAPG Short Course

Similar to hybriData IIoT Workshop for AAPG Short Course (20)

More from Chijioke “CJ” Ejimuda

More from Chijioke “CJ” Ejimuda (13)

Recently uploaded

Recently uploaded (20)

hybriData IIoT Workshop for AAPG Short Course