by Zignal Labs Today, machine learning solves a range of everyday business challenges. Companies are leveraging machine learning to understand how their brands are perceived in the marketplace across key stakeholder segments. How does the brand resonate with customers and the media? What product feedback and enhancements can be learned? By harnessing the power of machine learning, Zignal monitors and analyzes – in real-time – brand conversations across social, broadcast, digital and traditional media channels. In this session, learn how Zignal leverages Amazon SageMaker, Amazon Mechanical Turk, AWS Code Pipeline and AWS Lambda to accurately measure the brand health of major enterprises such as NVIDIA and Airbnb. Zignal will dive deep into how Amazon SageMaker and these services work together on machine learning models in a real-time media environment.

1. A Sentiment Pipeline with AWS and
Amazon SageMaker
Jeff Fenchel

2. Starbucks - Racial Profiling
Shutdown for racial bias training estimated to cost an
additional 16.7 million in lost revenue.

3. Agenda
1. Why are we building yet another sentiment API?
2. How we leverage Amazon Mechanical Turk to collect labeled
data
3. Utilizing Amazon SageMaker to regularly retrain and update
models in a resilient fashion
www.linkedin.com/in/jeffreyfenchel

4. Why Sentiment?
Option A Option B

5. Customer Feedback
“The sentiment is too neutral.”
“I have removed sentiment from all my reports.”
“I spend hours doing manual sentiment overrides.”
“Why was tweet X labeled neutral/positive/negative.”

6. What Happened

7. Zignal Labs

8. Rule Based Sentiment
● Positive if it:
○ mentions and no dissatisfaction is expressed
○ portrays the company as being sustainable
○ Is introducing a new executive
● Negative if it:
○ Equates the company to something negative e.g world
hunger
● Neutral if it:
○ focuses on a new facility being opened

9. Reputation Polarity
"Polarity for reputation: Does the information (facts, opinions) in the text have positive, negative,
or neutral implications for the image of the company? This problem is related to sentiment
analysis and opinion mining, but has substantial differences with the mainstream research in that
areas: polar facts are ubiquitous (for instance, “Lehmann Brothers goes bankrupt” is a fact with
negative implications for reputation), perspective plays a key role. The same information may
have negative implications from the point of view of clients and positive from the point of view of
investors, negative sentiments may have positive polarity for reputation (for example, “R.I.P.
Michael Jackson. We’ll miss you” has a negative associated sentiment - sadness -, but a positive
implication for the reputation of Michael Jackson.)”
-- RepLab 2012

10. Reputation Polarity
"Polarity for reputation: Does the information (facts, opinions) in the text have positive, negative,
or neutral implications for the image of the company? This problem is related to sentiment
analysis and opinion mining, but has substantial differences with the mainstream research in that
areas: polar facts are ubiquitous (for instance, “Lehmann Brothers goes bankrupt” is a fact with
negative implications for reputation), perspective plays a key role. The same information may
have negative implications from the point of view of clients and positive from the point of view of
investors, negative sentiments may have positive polarity for reputation (for example, “R.I.P.
Michael Jackson. We’ll miss you” has a negative associated sentiment - sadness -, but a positive
implication for the reputation of Michael Jackson.)”
-- RepLab 2012
Negative Neutral Positive

11. Data Skew and Model Variance

12. Transparency And Trust
Predicted Sentiment
Actual
Sentiment
Negative Neutral Positive
Negative 102 18 13
Neutral 16 39 16
Positive 6 2 33

13. The Plan

14. Mechanical Turk for Human Intelligence Tasks (HITs)
On Demand Workforce

15. Where do we start?
$ click_
https://github.com/pallets/click

16. Quality Control
● Fleiss’ Kappa Agreement
● Worker quality and bias assessment
with expectation maximization
● Qualification test and training
○ 21% pass rate

17. Continuous Labeling
Complete Records are critical
including:
● Raw assignment answers
from Mturk + HIT info
● Computed worker
evaluations (quality + bias +
support)
● Best fit answers

18. Continuous Labeling
Complete Records are critical
including:
● Raw assignment answers
from Mturk + HIT info
● Computed worker
evaluations (quality + bias +
support)
● Best fit answers

19. Quality in Test != Quality at Scale
● 92% -> 73% Label accuracy
● We get repeat workers!
CrossPolarityErrorRate
Worker Score [0,1]
Number of batches with contribution
Ratioofworkers
Workers by Repeat work Cumulative Histogram
Worker Score vs Cross Polarity Error

20. Investment in Workers
● Transparent feedback
● Automatic exclusions

21. Iteration is Key

22. Deployable Project Pattern
sultan/
├── deploy/
│ ├── templates/
│ │ ├── __init__.py
│ │ ├── pipeline.py
│ │ └── sultan.py
│ ├── __init__.py
│ └──requirements.txt
├── functions/
│ ├── create_sentiment_questions/
│ ├── provide_feedback/
│ └── manage_sentiment_results/
├── sultan/
├── setup.py
└── requirements.py
Troposphere
https://github.com/cloudtools/troposphere
ZignalLib
- Cloudformation in Python
- Common Deployment Patterns

23. The Pipeline Code

24. The Twitter Model
● Current models use Keras + NLTK (for longform)
● BYOE approach

25. The Longform Model
● Two phase training: sentence level and document level

26. The Twitter Model

27. The Twitter Model

28. The Twitter Model

29. The Twitter Model

30. Amazon SageMaker
Optimized For:
Also supports:

31. Model Lifecycle
Training jobs:
● Longform
● Twitter
Endpoints:
● Twitter
○ Variant 1 - <Date Trained> - 95%
Traffic
○ Variant 2 - <Date Trained> - 5%
Traffic
● Longform

32. Model Training
model/
├── config.json
├── model.hdf5
├── output_encoder.pickle
├── stats.json
└── tokenizer.pickle

33. Model Training

34. Model Training

35. Model Training

36. Model Deployment
● New models introduced as a 5%
variant
1) Deploy
2) Promote ● Promoted if 5xx replies < 10%

37. Model Deployment

38. Model Deployment

39. Model Deployment

40. Model Deployment

41. Model Deployment

42. Model Deployment (Review)
● New models introduced as a 5%
variant
1) Deploy
2) Promote ● Promoted if 5xx replies < 10%

43. Model Promotion

44. Model Promotion

45. Model Promotion

46. Model Promotion

47. Model Deployment (Review)
● New models introduced as a 5%
variant
1) Deploy
2) Promote ● Promoted if 5xx replies < 10%

48. Serving Model Architecture
● Amazon SageMaker provided framework
https://github.com/awslabs/amazon-sagemaker-examples/tree/master/advan
ced_functionality/scikit_bring_your_own
● API consumer side batching

49. Summary
● Continuously gather labeled data from Mechanical Turk
● Leverage Amazon SageMaker to retrain daily and provide an
endpoint for our real time data pipeline
○ Serverless
○ Provides architecture patterns
● Received positive feedback in a trials with numerous
customers especially around sentiment directionality
● Future Work
○ Explore Hyper Parameter Tuning
○ Improve the inclusion of relevance in sentiment analysis

50. Zignal Labs
https://zignallabs.com/
www.linkedin.com/in/jeffreyfenchel
jfenchel@zignallabs.com