Smart homes, smart phones, even smart stones… Today users expect everything to be smart and web sites to be tailored to their needs, and intelligent enough to serve within less taps. The huge advancements in machine learning and big data in recent years made that possible. One of the most complete and advanced services that is a step in front of the competition, and allows developers to add AI to their products, is Azure Cognitive Services. This session will be about how computer vision, natural language processing, speech and intent recognition could allow building smart apps with enhanced experience and be more engaging, personal and relevant.

1. Nov 17, 2018Sofia
var title = “Building Smart Web Apps
with Azure and AI as a Service”;
var info = {
name: “Ivelin Andreev”,
otherOptional: “Building apps for enhanced,
personal and relevant UX”
};

2. Nov 17, 2018
• Software Architect @
o 16+ years professional experience
• Microsoft Azure MVP
• External Expert Horizon 2020
• External Expert Eurostars-Eureka, InnoFund Denmark
• Business Interests
o Web Development, SOA, Integration
o IoT, Machine Learning, Computer Intelligence
o Security & Performance Optimization
• Contact
ivelin.andreev@icb.bg
www.linkedin.com/in/ivelin
www.slideshare.net/ivoandreev
Who is in front of you?

3. Nov 17, 2018
Thanks to our Sponsors:
General Sponsor:
Gold Sponsors:
Technological Partner:
Hosting Partner:
Silver Sponsors:
Lunch Sponsor:
Innovation Sponsor:
Bronze Sponsors:
Infotainment Sponsor:

4. Nov 17, 2018
agenda();
• Artificial Intelligence as a Service
• Computer Vision
• Custom Vision
• Language Understanding
• Custom Decision
• Other Cognitive Services
• Demo

5. Nov 17, 2018
Users’ expectations:
• Engaging experience
• Effortless interaction
• High performance
• Relevant content
Businesses aim:
• Provide high value
• Faster and at low cost
o Data science talent
o Powerful infrastructure
o Continuous improvement
The developer role is to
bridge the gap:

6. Nov 17, 2018
Artificial Intelligence as a Service (AIaaS)
Def: Artificial intelligence off the shelf
• Bots and NLP – commands and guidance
• Cognitive APIs – speech, vision, translation, knowledge
• ML frameworks – build own model w/o infrastructure (i.e. Azure ML Service)
• Fully managed ML – templates, deployment, drag-drop (i.e. Azure ML Studio)
• Innovation w/o upfront costs and expertise
• Usability – easy learning curve
• Scalability – start PoC, grow big
• Flexi cost – know what you pay for
• Share data with vendors
• Data regulations (i.e. GDPR)
• Reduced transparency
• Breaking changes

7. Nov 17, 2018
AIaaS market expected to
grow from $1.5Bil (2018) to
10.9Bil (2023)
(ResearchAndMarket Apr’ 2018)
1 year ago this was not as
achievable as it is now.

8. Nov 17, 2018
• Microsoft and open source
• Contribute in Code
https://opensource.microsoft.com/
• Contribute in Localization
o Democratic mode and voting
o Crowdsourcing with tool support
https://github.com/Microsoft/Localization/wiki
Microsoft are Open (more than ever before)
830+ Projects
o VS Code
o VSCode tools for AI
o SQL tools on Linux
o TypeScript
o CNTK
o .NET Core
o SignalR
o ONNX

9. Nov 17, 2018
Computer Vision – Why?
What it does
• Advanced algorithms for processing images for information
Use cases
• Tag visual features (2000+ objects and actions)
• Description in human readable language
• Face detection with tagging of age, gender and bounding rectangle
• OCR in 25 languages (no Bulgarian ), Handwriting (preview)
• Categorization on hierarchical taxonomies
• Domain modeling – recognize domain members (i.e. celebrity, family )
Pricing
• Free - 5000 req./month, $1-$2.5 per 1’000 req. (10 TPS)

10. Nov 17, 2018
Sample: Computer Vision for Fire Detection
Cloud Vision
(Google)
Computer Vision
(Microsoft)

11. Nov 17, 2018
Face API, Verify API, Identify API – Why?
What it does
• Detect and analyze facial attributes
• Authentication against two faces or group
Use cases
• Security - verify the same person (Verify API)
• Person identification – find closest match (Identify API)
• Find known persons - similar faces (Face API)
• Image tagging – age, gender, emotion, hair, makeup, accessories (Face API)
Pricing
• Free – 30’000 req/month (20 TPM)
• Standard - $1 per 1’000 req. (10 TPS)

12. Nov 17, 2018
Identify API – How?
Create person group
• Create group (i.e. Family) and get unique personGroupID
Create person
• Create person (name, userData) on personGroupID
Add face to person in group
• Upload image (URL and byte[] supported) for personGroupID and personID
Train group
• Train prior to identification; retrain on changes (!= Status.Running)
Identify
• Upload image and get unique faceID
• Identify candidates on personGroupID and faceID

13. Nov 17, 2018
[{"faceId": "abaff639-9ed4-43d6-b519-
670804552941",
faceRectangle": {
"width": 220,
"height": 242,
"left": 256,
"top": 256
},
"faceLandmarks": {…},
"faceAttributes": {
"age": 67.0,
"gender": "male",
"smile": 0.0,
"facialHair": {
"moustache": 0.9,
"beard": 0.9,
"sideburns": 0.9
},
"glasses": "sunglasses",
"headPose": {
"roll": -0.8,
"yaw": -11.7,
"pitch": 0.0
},
"emotion": {
"anger": 0.0,
"contempt": 0.0,
"disgust": 0.0,
"fear": 0.0,
"happiness": 0.0,
"neutral": 0.673,
"sadness": 0,
"surprise": 0.326
},
"hair": {
"bald": 0.0,
"invisible": true,
"hairColor": []
},
"makeup": {
"eyeMakeup": false,
"lipMakeup": false
},
"occlusion": {
"foreheadOccluded": true,
"eyeOccluded": false,
"mouthOccluded": false
},
"accessories": [
{"type":"headWear","confidence":1.0},
{"type":"glasses","confidence":1.0}],
…
}
}
]
Face API - Sample Response

14. Nov 17, 2018
Computer Vision is an easy to
use pretrained image classifier.
Use Custom Vision to create
custom classifications.

15. Nov 17, 2018
customvision.ai (preview) – Why?
What it does
• Build, deploy, improve custom image classifiers (on tags)
Use cases
• Image classification – assign tags to each image
• Object detection – return the coordinates of object in image
• Execute on mobile device (ONNX, CoreML, Tensorflow)
Pricing
• Free (2 TPS, 2 projects, 5’000 images/project)
• Standard (10 TPS; 100 projects; $1 per 1’000 req., SLA 99.9%)
API- and SDK-based (train, predict, test)

16. Nov 17, 2018
Domains (General, Food, Landmarks, Retails, Adult, Compact)
Create project
Upload and tag
• It is easier to upload by the tags (50+ images/tag)
• Use different angles, background, size and balance
Train the classifier
• As easy as
• Set training iteration as default
Performance
• Precision – correct prediction TP / (TP + FP)
• Recall – proper identification TP / (TP + FN)
customvision.ai (preview) – What?

17. Nov 17, 2018
Prediction API Request
customvision.ai (preview) – How?
Negative image handling
• Resemblance: Automatic negative image handling (0% score expected)
• No resemblance: Build class “Other”
Prediction API Response

18. Nov 17, 2018
Language Understanding

19. Nov 17, 2018
LUIS.ai – Why?
What it does
• Apply custom ML intelligence to conversational natural language
• Predict meaning to complete a task / command
Use cases
• Chat bot; Intent recognition
LUISGen
• CMD tool for strongly typed interfaces in C# and TS
Pricing
• Free (5 TPS)
• Standard (50 TPS; Text - $1.50 per 1’000; Speech - $5.50 per 1’000)
12 languages supported

20. Nov 17, 2018
LUIS.ai – What?
Concepts
• Intents – tasks or actions user wants to perform
• Utterances – input from user the app needs to interpret
• Entities – key words and phrases for domain (prebuilt, simple, list, hierarchy, composite, RegEx)
o Hierarchies – i.e. Name::FirstName, Name::LastName
• Roles – contextual subtypes of entity for patterns (i.e. {Location:origin} to {Location:destination})
Build LUIS Model
• Start with 10-15 training utterances per intent (Authoring API or LUIS portal)
Publish (incl. sentiment)
Active Learning
• Utterances added to review list (low top score, top 2 are too close)

21. Nov 17, 2018
LUIS.ai – How? (Best Practices)
Different intents, different vocabulary (i.e. Book flight / Book hotel)
Phrase lists
• {Jan, Feb, Mar…}; Synonyms (i.e. want, require, need)
Patterns
“None” intent for subject outside of application domain
Don’t add examples of the same format
• Buy a ticket to Seattle
• Buy a ticket to Paris -> Reserve a seat to Paris
Don’t create intent for different entities (i.e. every destination)

22. Nov 17, 2018
LUIS.ai – How?
Endpoint
https://<region>.api.cognitive.microsoft.com/luis/v2.0/apps/<appID>?verbose=true&subscription-
key=<YOUR_KEY>&<optional-name-value-pairs>&q=<user-utterance-text>
• Send encoded utterance with “q” parameter (i.e. Here+is+my+c.v.+for+the+data+scientist+job)
JSON
{
"query": "Here is my c.v. for the data scientist job",
"topScoringIntent": {
"intent": "ApplyForJob",
"score": 0.9826467
},
"intents": [
{"intent": "ApplyForJob",
"score": 0.9826467},
{"intent": "GetJobInformation",
"score": 0.0218927357}
…
],
"entities": [
{
"entity": “data scientist",
"type": "Job",
"startIndex": 24,
"endIndex": 37,
"score": 0.5714122
}
]
}

23. Nov 17, 2018
Custom Decision

24. Nov 17, 2018
Custom Decision (experimental) – Why?
What it does
• Learn behavioural patterns of users
• Recommend relevant experience or content
Use cases
• Personalize product / website
• Personalize video content
• Optimize Ad placement on pages
• Rank shopping item recommendations
18 languages supported

25. Nov 17, 2018
Custom Decision (experimental) – What?
Handles end-to-end ML process with reinforcement learning
• Explore and collect data
• Train and deploy models
User is presented with a list of content items (i.e. on front page)
• Specify an RSS feed to define Action Set and refresh time
• Convert content into ML features (text, images, video, sentiment)
• Use other services at the backend (Entity Linking, Text Analytics, Emotion/Face, Vision)
• Get all content and returning in ranked order (Ranking)
• Callback function (defined by developer) renders the content in UI in ranked order
When a user makes selection
• Clicks are reported by custom code and measure response (Reward)

26. Nov 17, 2018
• Register application to receive App ID
• Define RSS feed
Ranking API
• JSONP formatted response to bypass Same-origin-policy
Reward API
Custom Decision (experimental) – How?

27. Nov 17, 2018
Cognitive Labs - in Preview
Gesture
o Productivity SDK to allow interaction with technology using gestures
o Requires Intel RealSense camera ($89)
Conversation Learner
o Build and teach bots and intelligent agents from sample interaction
o Based on LSTM NN and learn based on prototypical dialogs
o Custom code attached to callbacks
Entity Linking
o Recognize named entities based on context and provide Wikipedia ref.
o Part of Text Analytics API
https://labs.cognitive.microsoft.com/

28. Nov 17, 2018
Cognitive Labs - Private Preview
Anomaly Finder
o Monitor data over time, adapt and detect anomalies based on statistical models
URL Preview
o JSON with description and a link to preview image
o isFamilyFriendly flag – indicates adult, pirated or illegal content
Local Insights
o Score the attractiveness of a location based on proximity to amenities
(public transport, parks, cinemas or other)
Answer Search
o Answer question in natural language with information from the web
Event Tracking
o Cluster news that report things happening on Earth to event

29. Nov 17, 2018
Takeaways
• Cognitive services documentation
https://docs.microsoft.com/en-us/azure/cognitive-services/welcome
• Custom Decision service documentation
https://docs.microsoft.com/en-us/azure/cognitive-services/custom-decision-service
• LUIS Samples
https://github.com/Microsoft/LUIS-Samples/tree/master/examples
• Cognitive Services Online Demos
https://azure.microsoft.com/en-us/services/cognitive-services/directory/vision/
• Computer Vision (Sample JS Code)
https://github.com/Azure-Samples/cognitive-services-javascript-computer-vision-tutorial

30. Nov 17, 2018
Thanks to our Sponsors:
General Sponsor:
Gold Sponsors:
Technological Partner:
Hosting Partner:
Silver Sponsors:
Lunch Sponsor:
Innovation Sponsor:
Bronze Sponsors:
Infotainment Sponsor: