1-hr tech talk introducing ML and the GCP ML APIs

1. Google Cloud developer tools + an
Easyier path to machine learning
Wesley Chun
Developer Advocate, Google
G Suite Dev Show
goo.gl/JpBQ40
About the speaker (not a data scientist!)
Developer Advocate, Google Cloud
● Mission: enable current and future
developers everywhere to be
successful using Google Cloud and
other Google developer tools & APIs
● Videos: host of the G Suite Dev Show
on YouTube
● Blogs: developers.googleblog.com &
gsuite-developers.googleblog.com
● Twitters: @wescpy, @GoogleDevs,
@GSuiteDevs
Previous experience / background
● Software engineer & architect for 20+ years
● One of the original Yahoo!Mail engineers
● Author of bestselling "Core Python" books
(corepython.com)
● Technical trainer, teacher, instructor since
1983 (Computer Science, C, Linux, Python)
● Fellow of the Python Software Foundation
● AB (Math/CS) & CMP (Music/Piano), UC
Berkeley and MSCS, UC Santa Barbara
● Adjunct Computer Science Faculty, Foothill
College (Silicon Valley)

2. Why and Agenda
● Big data is everywhere now
● Need the power of AI to help analyze
● Requires certain level of math/statistics background
● AI/ML has somewhat steep learning curve
● APIs powered by ML helps ease this burden
● If you can call APIs, you can use ML!
1
Intro to machine
learning
2
Intro to Google
Cloud
3
Google APIs
4
Cloud ML APIs
5
Other APIs to
consider
6
All of Cloud
(inspiration)
7
Summary &
wrap-up
What is machine learning?
AI, ML, and making computers smarter; to help us
understand more and get more insights than before
1

3. AI
Make code solve
problems commonly
associated with
human intelligence
ML
Make code learn
from experience
instead of explicit
programming
DL
ML using deep neural
networks… make
code learn to be
even better/smarter

4. AI & Machine Learning
Puppy or muffin?
Machine learning is learning
from rules plus experience.

6. ML @ Google
How has ML improved our products?
20%

7. Google Translate
Google Photos
Did you ever stop
to notice this app
has a search bar?!?

8. How to get started
Enough talk, let's think about first steps
Lots of data
Complex mathematics in
multidimensional spaces
Magical results
Popular imagination of what Machine Learning is

9. Organize data
Use machines to
flesh out the
model from data
Collect
data
Create model
Deploy fleshed
out model
In reality what ML is
Rules
Data
Traditional
Programming
Answers
Answers
Data
RulesMachine
Learning

10. Fashion MNIST
● 70k grayscale images
○ 60k training set
○ 10k testing set
● 10 categories
● Images: 28x28 pixels
● Go train a neural net!
tensorflow.org/tutorials/
keras/classification

11. import tensorflow as tf
from tensorflow import keras
fashion_mnist = keras.datasets.fashion_mnist
(train_images, train_labels), (test_images, test_labels) = fashion_mnist.load_data()
09 09 = ankle boot;
踝靴;
アンクルブーツ;
Bróg rúitín
Training Phase
Answers
Data
Rules/modelMachine
Learning
Model
Data Predictions
Inference Phase

12. Your steps
1. Import MNIST dataset
2. Explore/preprocess data
3. Build model
a. Setup layers
b. Compile model
4. Train model
5. Evaluate accuracy
6. Make predictions
7. (Have fun!)
2 Introduction to
Google Cloud
GCP and G Suite tools & APIs

13. GCP Machine Learning APIs
● Gain insights from data using GCP's
pre-trained machine learning models
● Leverage the same technology as
Google Translate, Photos, and Assistant
● Requires ZERO prior knowledge of ML
● If you can call an API, you can use AI/ML!
Vision Video
Intelligence
Speech
(S2T & T2S)
Natural
Language
Translation

14. Full Spectrum of AI & ML Offerings
App developer
Data Scientist
Data Scientist,
Researcher w/access to
infrastructure, GPUs...
Use pre-built models
Use/extend OSS SDK,
build models, manage
training infrastructure
ML Engine
Auto ML
Build custom models,
use/extend OSS SDK
ML APIs
App developer,
data scientist
Use/customize pre-built
models
3
Google (REST) APIs
What are they? How do you use them?

15. Cloud/GCP console
console.cloud.google.com
● Hub of all developer activity
● Applications == projects
○ New project for new apps
○ Projects have a billing acct
● Manage billing accounts
○ Financial instrument required
○ Personal or corporate credit cards,
Free Trial, and education grants
● Access GCP product settings
● Manage users & security
● Manage APIs in devconsole
● View application statistics
● En-/disable Google APIs
● Obtain application credentials
Using Google APIs
goo.gl/RbyTFD
API manager aka Developers Console (devconsole)
console.developers.google.com

16. SIMPLE
AUTHORIZED
Which do you choose?
4
Cloud ML APIs
Easier path to ML by simply calling APIs!

17. Machine Learning: Cloud Vision
Google Cloud Vision API
cloud
labeling = VISION.images().annotate(body=body).execute().get('responses')
for labels in labeling:
if 'labelAnnotations' in labels:
print('** Labels detected (and confidence score):')
for label in labels['labelAnnotations']:
print(('%.2f%%' % (
label['score']*100.)).ljust(10), label['description'])
if 'faceAnnotations' in labels:
print('n** Facial features detected (and likelihood):')
for label, value in labels['faceAnnotations'][0].items():
if label.endswith('Likelihood'):
print(label.split('Likelihood')[0].ljust(16),
value.lower().replace('_', ' '))
Vision: image analysis & metadata extraction

18. $ python viz_demo.py
** Labels detected (and confidence score):
89.94% Sitting
86.09% Interior design
82.08% Furniture
81.52% Table
80.85% Room
79.04% White-collar worker
76.19% Office
68.18% Conversation
60.96% Window
60.07% Desk
** Facial features detected (and likelihood):
anger very unlikely
joy very likely
underExposed very unlikely
sorrow very unlikely
surprise very unlikely
headwear very unlikely
blurred very unlikely
Vision: image analysis & metadata extraction
Higher-level GCP SDK & API client libraries
1. Bad news: Just showed you the "harder
way" of using Google Cloud Platform APIs
2. Good news: it's even easier with the GCP
SDK and higher-level client libraries
3. Why (not)? Not all Google APIs have high-
level client libraries. Lower-level serves as
"LCD" for accessing more Google APIs
cloud.google.com/sdk
cloud.google.com/apis/docs

19. from google.cloud import vision
image_uri = 'gs://cloud-samples-data/vision/using_curl/shanghai.jpeg'
client = vision.ImageAnnotatorClient()
image = vision.types.Image()
image.source.image_uri = image_uri
response = client.label_detection(image=image)
print('Labels (and confidence score):')
print('=' * 30)
for label in response.label_annotations:
print(f'{label.description} ({label.score*100.:.2f}%)')
Vision: label annotation/object detection
$ python3 label-detect.py
Labels (and confidence score):
==============================
People (95.05%)
Street (89.12%)
Mode of transport (89.09%)
Transport (85.13%)
Vehicle (84.69%)
Snapshot (84.11%)
Urban area (80.29%)
Infrastructure (73.14%)
Road (72.74%)
Pedestrian (68.90%)
Vision: label annotation/object detection
codelabs.developers.google.com/codelabs/cloud-vision-api-python#6

20. from google.cloud import vision
image_uri = 'gs://cloud-vision-codelab/otter_crossing.jpg'
client = vision.ImageAnnotatorClient()
image = vision.types.Image()
image.source.image_uri = image_uri
response = client.text_detection(image=image)
for text in response.text_annotations:
print('=' * 30)
print(f'"{text.description}"')
vertices = [f'({v.x},{v.y})' for v in text.bounding_poly.vertices]
print(f'bounds: {",".join(vertices)}')
Vision: OCR, text detection/extraction
$ python3 text-detect.py
==============================
"CAUTION
Otters crossing
for next 6 miles
"
bounds: (61,243),(251,243),(251,340),(61,340)
==============================
"CAUTION"
bounds: (75,245),(235,243),(235,269),(75,271)
==============================
"Otters"
bounds: (65,296),(140,297),(140,315),(65,314)
==============================
"crossing"
bounds: (151,294),(247,295),(247,317),(151,316)
:
Vision: OCR, text detection/extraction
codelabs.developers.google.com/codelabs/cloud-vision-api-python#7

21. from google.cloud import vision
image_uri = 'gs://cloud-vision-codelab/eiffel_tower.jpg'
client = vision.ImageAnnotatorClient()
image = vision.types.Image()
image.source.image_uri = image_uri
response = client.landmark_detection(image=image)
for landmark in response.landmark_annotations:
print('=' * 30)
print(landmark)
Vision: landmark detection, entity extraction
$ python3 landmark-detect.py
==============================
mid: "/g/120xtw6z"
description: "Trocad303251ro Gardens"
score: 0.9368728995323181
:
==============================
mid: "/m/02j81"
description: "Eiffel Tower"
score: 0.30917829275131226
bounding_poly {
vertices {
x: 400
y: 40
:
}
Vision: landmark detection, entity extraction
codelabs.developers.google.com/codelabs/cloud-vision-api-python#8

22. Cloud Vision demo
quickdraw.withgoogle.com
Machine Learning: Cloud Natural Language
Google Cloud Natural Language API
cloud

23. Simple sentiment & classification analysis
TEXT = '''Google, headquartered in Mountain View, unveiled the new
Android phone at the Consumer Electronics Show. Sundar Pichai said
in his keynote that users love their new Android phones.'''
print('TEXT:', TEXT)
data = {'type': 'PLAIN_TEXT', 'content': TEXT}
NL = discovery.build('language', 'v1', developerKey=API_KEY)
# sentiment analysis
sent = NL.documents().analyzeSentiment(
body={'document': data}).execute().get('documentSentiment')
print('nSENTIMENT: score (%s), magnitude (%s)' % (sent['score'], sent['magnitude']))
# content classification
print('nCATEGORIES:')
cats = NL.documents().classifyText(body={'document': data}).execute().get('categories')
for cat in cats:
print('* %s (%s)' % (cat['name'][1:], cat['confidence']))
Simple sentiment & classification analysis
$ python nl_sent_simple.py
TEXT: Google, headquartered in Mountain View, unveiled the new Android
phone at the Consumer Electronics Show. Sundar Pichai said in
his keynote that users love their new Android phones.
SENTIMENT: score (0.3), magnitude (0.6)
CATEGORIES:
* Internet & Telecom (0.76)
* Computers & Electronics (0.64)
* News (0.56)

24. Machine Learning: Cloud Speech
Google Cloud Speech APIs
cloud
cloud
Machine Learning: Cloud Video Intelligence
Google Cloud Video Intelligence
API
cloud

25. Machine Learning: Cloud Translation
Google Translate
cloud
Machine Learning: AutoML
AutoML:
cloud
cloud

26. ● General steps
a. Prep your training data
b. Create dataset
c. Import items into dataset
d. Create/train model
e. Evaluate/validate model
f. Make predictions
Cloud AutoML: how to use
Machine Learning: Cloud ML Engine
Google Cloud Machine Learning Engine
cloud

27. Machine Learning: Cloud TPUs
Google Cloud TPU API
cloud
Other APIs to consider
These may also be helpful5

28. Storing and Analyzing Data: BigQuery
Google BigQuery
cloud
BigQuery: querying Shakespeare words
TITLE = "The top 10 most common words in all of Shakespeare's works"
QUERY = '''
SELECT LOWER(word) AS word, sum(word_count) AS count
FROM [bigquery-public-data:samples.shakespeare]
GROUP BY word ORDER BY count DESC LIMIT 10
'''
rsp = BQ.query(body={'query': QUERY}, projectId=PROJ_ID).execute()
print('n*** Results for %r:n' % TITLE)
for col in rsp['schema']['fields']: # HEADERS
print(col['name'].upper(), end='t')
print()
for row in rsp['rows']: # DATA
for col in row['f']:
print(col['v'], end='t')
print()

29. Top 10 most common Shakespeare words
$ python bq_shake.py
*** Results for "The most common words in all of Shakespeare's works":
WORD COUNT
the 29801
and 27529
i 21029
to 20957
of 18514
a 15370
you 14010
my 12936
in 11722
that 11519
Running Code: Compute Engine
>
Google Compute Engine
cloud

30. Running Code: App Engine
Google App Engine
we
>
cloud
Running Code: Cloud Functions
Google Cloud Functions
cloud
firebase

31. G Suite: Google Sheets
Sheets API
developers
Try our Node.js customized reporting tool codelab:
g.co/codelabs/sheets
Why use the Sheets API?
data visualization
customized reports
Sheets as a data source

32. Migrate SQL data to a Sheet
# read SQL data then create new spreadsheet & add rows into it
FIELDS = ('ID', 'Customer Name', 'Product Code',
'Units Ordered', 'Unit Price', 'Status')
cxn = sqlite3.connect('db.sqlite')
cur = cxn.cursor()
rows = cur.execute('SELECT * FROM orders').fetchall()
cxn.close()
rows.insert(0, FIELDS)
DATA = {'properties': {'title': 'Customer orders'}}
SHEET_ID = SHEETS.spreadsheets().create(body=DATA,
fields='spreadsheetId').execute().get('spreadsheetId')
SHEETS.spreadsheets().values().update(spreadsheetId=SHEET_ID, range='A1',
body={'values': rows}, valueInputOption='RAW').execute()
Migrate SQL data
to Sheets
goo.gl/N1RPwC
G Suite: Google Slides
Slide API
create
manage
developers

33. Try our Node.js BigQuery GitHub license analyzer codelab:
g.co/codelabs/slides
Why use the Slides API?
data visualization
presentable reports
Import Sheets cells & charts into Slides
requests = [
# create Slides table (for Sheets cell data)
{'createTable': {
'elementProperties': {'pageObjectId': tableSlideID},
'rows': len(orders),
'columns': len(orders[0])q,
}},
# bold row 1
{'createSheetsChart': {
'spreadsheetId': sheetID,
'chartId': chartID,
'linkingMode': 'LINKED',
}},
]
SLIDES.presentations().batchUpdate(body={'requests': requests},
presentationId=DECK_ID, fields='').execute()
Sheets ⇒ Slides
goo.gl/Yb06ZC

34. G Suite: Apps Script
Apps Script
developers
“Hello World!” in Apps Script

35. Sheets-bound “Hello World!”
Apps Script intro
goo.gl/1sXeuD
What can you do with this?

36. Accessing maps from
spreadsheets?!?
goo.gl/oAzBN9
This… with help from Google Maps & Gmail
function sendMap() {
var sheet = SpreadsheetApp.getActiveSheet();
var address = sheet.getRange("A2").getValue();
var map = Maps.newStaticMap().addMarker(address);
GmailApp.sendEmail('friend@example.com', 'Map',
'See below.', {attachments:[map]});
}
JS

37. Simple sentiment & classification analysis
● Analyze sentiment in
Google Docs
● Use simple API call to
Natual Language API
● Call with Apps Script
UrlFetch service
● Build this app yourself at
g.co/codelabs/nlp-docs
[simple API/API key sample]
Simple sentiment & classification analysis
function getSentiment(text) {
var apiKey = YOUR_API_KEY;
var apiEndpoint =
'https://language.googleapis.com/v1/documents:analyzeSentiment?key=' + apiKey;
// NL API metadata JSON object
var nlData = {
document: {
language: 'en',
type: 'PLAIN_TEXT',
content: text
},
encodingType: 'UTF8'
};

38. [simple API/API key sample]
Simple sentiment & classification analysis
// Create API payload
var nlOptions = {
method: 'POST',
contentType: 'application/json',
payload: JSON.stringify(nlData)
};
// Make API call via UrlFetch (when no object available)
var response = UrlFetchApp.fetch(apiEndpoint, nlOptions);
var data = JSON.parse(response);
var sentiment = 0.0;
if (data && data.documentSentiment && data.documentSentiment.score) {
sentiment = data.documentSentiment.score;
}
Logger.log(sentiment);
return sentiment;
}
● Extend functionality of G Suite editors
● Embed your app within ours!
● 2014: Google Docs, Sheets, Forms
● 2017 Q3: Google Slides
● 2017 Q4: Gmail
● 2018 Q1: Hangouts Chat bots
● Apps Script also powers App Maker,
Google Data Studio community
connectors, and Google Ads scripts
Apps Script powers add-ons… and more!

39. 6 All of Cloud
(inspiration)
Build powerful solutions with both
GCP and G Suite
Custom intelligence in Gmail
Analyze G Suite data with GCP

40. Gmail message processing with GCP
Gmail
Cloud
Pub/Sub
Cloud
Functions
Cloud
Vision
G Suite GCP
Star
message
Message
notification
Trigger
function
Extract
images
Categorize
images

41. Inbox augmented with Cloud Function
● Gmail API: sets up notification forwarding to Cloud Pub/Sub
● developers.google.com/gmail/api/guides/push
● Pub/Sub: triggers logic hosted by Cloud Functions
● cloud.google.com/functions/docs/calling/pubsub
● Cloud Functions: "orchestrator" accessing GCP APIs
● Combine all of the above to add custom intelligence to Gmail
● Deep dive code blog post
● cloud.google.com/blog/products/application-development/
adding-custom-intelligence-to-gmail-with-serverless-on-gcp
● Application source code
● github.com/GoogleCloudPlatform/cloud-functions-gmail-nodejs
App summary

42. Big data analysis to slide presentation
Access GCP tools from G Suite
Big data analysis

43. Store big data results

44. Visualize big data results
Ingest data from Sheets

45. Link to chart in Sheets

46. Supercharge G Suite with GCP
G Suite GCP
BigQuery
Apps Script
Slides Sheets
Application
request
Big data
analytics
App summary
● Leverage GCP and build the "final mile" with G Suite
● Driven by Google Apps Script
● Google BigQuery for data analysis
● Google Sheets for visualization
● Google Slides for presentable results
● "Glued" together w/G Suite serverless
● Build this app (codelab)
● g.co/codelabs/bigquery-sheets-slides
● Video and blog post
● bit.ly/2OcptaG
● Application source code
● github.com/googlecodelabs/bigquery-sheets-slides
● Presented at Google Cloud NEXT (Jul 2018 [DEV229] & Apr 2019 [DEV212])
● cloud.withgoogle.com/next18/sf/sessions/session/156878
● cloud.withgoogle.com/next/sf/sessions?session=DEV212

47. 7
Wrap-up
Summary and resources
Session Summary
● What is machine learning again?
○ Solving harder problems by making computers smarter
● How do you machine learning again?
○ Have lots of data (with "labels")
○ Build and train your model then validate it
○ Use your model to make predictions on new data
● Do you need lots of machine learning experience to get started?
○ No: use pre-trained models available through APIs
○ Google Apps Script provides an easy way to do it w/API keys

48. References
● G Suite, Google Apps Script documentation & open source repos
○ developers.google.com/gsuite
○ developers.google.com/apps-script
● Google Cloud Platform (GCP) documentation & open source repos
○ cloud.google.com/bigquery
○ cloud.google.com/vision
○ cloud.google.com/language
○ cloud.google.com/video-intelligence
○ cloud.google.com/speech and cloud.google.com/text-to-speech
● Your next steps… further train our models by customizing them
○ By using the AutoML-enabled ML APIs
○ cloud.google.com/automl
Thank you!
Wesley Chun
@wescpy@
Progress bars: goo.gl/69EJVw
Slides: bit.ly/