SmartNews uses various SaaS products like New Relic, Datadog, and Chartio to monitor applications and infrastructure, collect metrics, and create dashboards. SaaS allows SmartNews to focus on its core product without having to dedicate engineering resources to tasks like monitoring and visualization. It provides built-in integrations, easy setup processes, and expert support so SmartNews can move fast without having to develop and maintain these capabilities internally. SaaS makes operations more efficient and allows SmartNews to gather insights that help optimize performance and troubleshoot issues.

1. Why and How
SmartNews uses SaaS?
@takus
2015.06.26
SmartNews Official Character
Chikyu-Kun

2. Takumi Sakamoto
(@takus)

4. SmartNews Crosses 10 Million Downloads in Global
Source: App Annie

5. SmartNews has No.1 MAU in News Apps, Japan
Comparison of the other News Apps
4.3M
MAU
Source: Nielsen Japan

6. Ranking of Active Users
* 2 5 1 1 199
-‐‑‒ E .
1
2
3
4
5
6
7
8
9
10
1 B 1
2 A 0 1 5

7. Long Time Spent
Facebook
messenger
LINE Facebook Gmail Youtube COOKPAD News App
A
SmartNews News App
B
News App
C
※ Spending minutes/person, month
Source: Nielsen Japan

8. Why SaaS?
Use cases
Relationship
Summary

9. We want to focus on
OUR PRODUCT

11. We must move fast

13. Engineering resources
are
limited

14. Efficient
Ads
System
Document
Classification
Site
Reliability
Engineering
High
Performance
API Server
News
Crawler
Realtime
Ranking
Algorithm
Data
Management
Platform
UI/UX

15. Can we assign
excellent engineers
to every positions?

16. No, probably

17. The answer is

19. Why SaaS?
• Solves specific issues for us
• monitoring, alerting, visualising etc...
• Less bootstrap process
• tiny setup process (a java agent, a chef recipe etc...)
• built-in user friendly web Interface
• No dedicated person is needed to develop/operate
• experts in an area do it well
• easy to manage with programable API

20. Why SaaS?
Use cases
Relationship
Summary

22. NewRelic
• (Application) Monitoring as a Service
• easy setup for popular language (e.g. Java)
• add method traces by annotation
• trace SQL queries automatically
• records deployments

23. Setup for Tomcat
• deploy newrelic archives to $CATALINA_HOME
• edit newrelic.yml
• application name
• add -javaagent
• $CATALINA_HOME/newrelic/newrelic.jar
• add environment if necessary
• -Dnewrelic.environment=xxx

24. Customize for each Apps
# add method tracer
@Trace(metricName="API/yourMethod", dispatcher=true)
public Response yourMethod() {
...
...
...
}
# add custom metric
MetricAggregator aggregator = NewRelic.getAgent().getMetricAggregator();
aggregator.recordMetric(key, value);
# record exception
NewRelic.noticeError(ex);

25. Trace methods & queries

26. Record Deployments

27. from fabric.api run
from fabric.decorators import runs_once, parallel
@parallel
@task
def deploy():
do_your_task()
notify()
@runs_once
def notify():
ver = release_version()
run("java -jar newrelic.jar deployment --appname=xxxx --revision=%s" % ver)
Record deployment by fabric
https://docs.newrelic.com/docs/agents/java-agent/instrumentation/recording-deployments-java-agent

28. NewRelic Insights
Query to custom metrics via NRQL
SmartNews seems top 10 user of Insights

29. What's great?
• Developer can monitor their own app by themselves
• adding custom tracer, custom metrics
• Don't have to prepare profiler for each language
• Java, Scala, Ruby, (Golang), etc...
• suit for Microservices

31. Datadog
• (System) Monitoring as a Service
• easy to setup with Chef
• 100+ built in integration (AWS, MySQL, Docker...)
• correlate metrics with events
• tag-based flexible visualization
• build your own interactive dashboards

32. Setup dd-agent with Chef
# Install cookbook
knife cookbook site install datadog
# Set Datadog-specific attributes
node.default['datadog']['api_key'] = "xxxxxxxxxx"
# Add a recipe in your run list
"run_list": ["recipe[datadog::dd-agent]"]

33. BTW, What is dd-agent?
http://help.datadoghq.com/hc/en-us/articles/203034929-What-is-the-Datadog-Agent-What-resources-does-it-consume-

34. Built-in Integrations

35. Setup MySQL Monitor
# Edit conf.d/mysql.yaml
instances:
- server: localhost
user: datadog
pass: xxxxxxxx
tags:
- dbtype:master
options:
replication: 0

36. Correlate metrics w/ deployment

37. New Relic Integration

38. sum:apache.status_2xx_count{role:web} by {env}
EC2
Classic
VPC
Tag-based flexible visualization

39. http://qiita.com/takus/items/c1d71bfcc231d0c24e0a

40. Migrate from EC2 Classic to VPC with Route53

41. role = web
env = classic
role = web
env = vpc
Provision clusters with Chef

42. role = web
env = classic
role = web
env = vpc
Aggregate metrics with 'env' tag

43. Check clusters by Host Maps
https://www.datadoghq.com/blog/2015/03/introducing-host-maps-know-thy-infrastructure/

44. Build your own dashboards

45. PagerDuty Integration
Only pager for related teams (members)

46. What's great?
• No dedicated Ops required for system monitoring
• chef cookbook & built-in integration
• customisable dashboard for each developers
• Find the root cause quickly
• gather all of metrics/events/alerting at one place
• tag-based flexible visualization
• correlating metrics & events

48. Chartio
• Dashboard as a Service
• create beautiful dashboards
• provide connectors to various data sources
• MySQL, Redshift, Google Analytics, BigQuery, etc...
• Presto with Prestogres
• non-engineer can create dashboard w/o SQL

50. How to create charts?
1. Build a query to data source
(drag & drop / SQL)
2. Modify data
(filter、sort、add column)
3. Visualise it
(table, pie chart, bar chart)

51. Connecting to Presto
https://github.com/treasure-data/prestogreshttp://docs.treasuredata.com/articles/chartio

52. Join BigQuery & MySQL
Layer1 BigQuery Result
article_id country pv
1 JP 1000000
2 US 2000000
3 GB 3000000
4 JP 4000000
article_id url title
1 http://aaa Article A
2 http://bbb Article B
3 http://ccc Article C
4 http://ddd Article D
Layer2 MySQL Result
Join
https://support.chartio.com/docs/charts/#merging-data-joins

53. Join BigQuery & MySQL
article_id country pv url title
1 JP 1000000 http://aaa Article A
2 US 2000000 http://bbb Article B
3 GB 3000000 http://ccc Article C
4 JP 4000000 http://ddd Article D
Joined Layer : BigQuery + MySQL
https://support.chartio.com/docs/charts/#merging-data-joins
CSV
Graph
Mail

54. Categorical Drop down
Could you give me the ios's article ranking of bbc.co.uk in UK?
e.g. : A request from BizDev team

55. How is the result of a A/B test in California?
It seems that iPhone 6's user complaints about new features.
Can I compare with the result of each devices?
e.g. : A request from client developers
Categorical Drop down

56. What's great?
• No dedicated SQL Monkey required for visualising
• everyone can be SQL Monkey
• non-engineer can do by WebUI
• Easy to drill down from overview to each users
• overall summaries
• each OS, country, region, language, etc...

57. c.f. About Our Data Platform
https://goo.gl/Rc1uuO http://goo.gl/KL1fpL

59. OneLogin
• Identity as a Service
• provides single sign on to various SaaS
• SAML / Auto complete form with browser extension
• non-frustrated 2FA with mobile app

62. Why SaaS?
Use cases
Relationship
Summary

63. Send a Pull Request

64. Send a Pull Request
• Wrote a fluentd - datadog integration
• only aiming for meeting our requirements
• It became an official integration
• https://www.datadoghq.com/2015/02/monitor-
fluentd-datadog/

65. Hold an User Meetup
http://blog.takus.me/2015/02/26/dd-sushi/

66. Gifts from SaaS

67. Discuss products in conferences

68. Discuss products in conferences
• Fortunately, I had some chances to attend them
• Velocity Conference 2013, 2015
• Amazon re:Invent 2014
• Great time to talk with core developers
• learn new services & new features
• explain our feature requests by myself
• ask their product roadmap

69. New Relic's Service Map

70. Anomaly detection by SumoLogic

71. Future of SaaS
• SaaS is becoming hot area in startup
• 2014/12/11: NewRelic raised $115M in IPO
• 2015/01/29: Datadog raised $31M in Funding
• 2015/06/01: SumoLogic raised $80M in Funding
• SaaS will evolve rapidly with a large amount of funds?
• probably in my opinion

72. Why SaaS?
Use cases
Relationship
Summary

73. SaaS makes life easier
• The real world is complex. We have a lot of tasks to do
• Professionals solve specific issues in behalf of us
• No dedicated engineer is needed for it
• We can focus on our own problems

74. We’re hiring!
Site reliability engineer
iOS/Android engineer
Web application engineer
Productivity engineer
ML/NLP engineer
Growth hack engineer
Ad engineer
http://about.smartnews.com/en/careers/

75. One more thing...

76. Join Our Free Lunch!
We love talking about Technologies, Products, Medias, etc...