Spark is used to perform in-memory transformations on customer data collected by Totango to generate analytics and insights. Luigi is used as a workflow engine to manage dependencies between batch processing tasks like metrics generation, health scoring, and alerting. The tasks are run on Spark and output to S3. A custom Gameboy controller provides monitoring and management of the Luigi workflow.

1. Spark Workflow Management
Romi Kuntsman
Senior Big Data Engineer @ Totango
romi@totango.com
https://il.linkedin.com/in/romik
„Big things are happening here“ Meetup
2015-04-29

2. Agenda
●
Totango and Customer Success
●
Totango architecture overview
●
Apache Spark computing framework
●
Luigi workflow Engine
●
Luigi in Totango

3. Totango and Customer Success
Your customers' success is your success

4. SaaS Customer Journey
DECREASE VALUE
DECREASE VALUE
CHURN
CHURN
GROW VALUE
FIRST VALUE
START
INCREASE USERS
INCREASE USAGE
EXPAND FUNCTIONALITY
CHURN
ONGOING VALUE

5. Customer Success Platform
●
Analytics for SaaS companies
●
Clear view of customer journey
●
Proactively prevent churn
●
Increase upsale
●
Track feature, module and total usage
●
Health score based on usages pattern
●
Improve conversion from trial to paying

6. Health Console

7. Module Statistics

8. Feature Adoption

9. About Totango
●
Founded in 2010
●
Size: ~50 (half R&D)
●
Offices in Tel Aviv, San Mateo CA
●
120+ customers
●
~70 million events per day
●
~1.5 billion indexed documents per month
●
Hosted on Amazon Web Services

10. Totango Architecture Overview
From usage information to actionable analytics

11. Terminology
●
Service – Totango's customer (e.g. Zendesk)
●
Account – Service's (Zendesk's) customer
●
SDR (Service Data Record) – User activity
event (e.g. user Joe from account Acme did
activity Login in module Application)

12. SDR reception
●
Clients send SDRs to the gateway, where they
are collected, filtered, packaged and finally
stored in S3 for daily/hourly batch processing.
●
Realtime processing also notified.

13. Batch Workflow

14. Account Data Flow
1) Raw Data (SDRs)
2) Account Aging (MySQL - legacy)
3) Activity Aggregations (Hadoop – legacy)
4) Metrics (Spark)
5) Health (Spark)
6) Alerts (Spark)
7) Indexing to Elasticsearch

15. Data Structure
●
Account documents stored on Amazon S3
●
Hierarchial directory structure per task param:
e.g. /s-1234/prod/2015-04-27/account/metrics
●
Documents have a predefined JSON schema.
JSON mapped directly to Java document class
●
Each file is an immutable collection of documents
One object per line – easily partitioned by lines

16. Apache Spark
One tool to rule all data transformations

17. Resilient Distributed Datasets
●
RDDs – distributed memory abstraction that lets
programmers perform in-memory computations
on large clusters in a fault-tolerant way
●
Initial RDD created from stable storage
●
Programmer defines a transformation from an
immutable input object to a new output object
●
Transformation function class can (read: should!)
be built and tested separately from Spark

18. Transformation flow
Read: inputRows = sparkContext.textFile(inputPath)
Decode: inputDocuments = inputRows.map(new
jsonToAccountDocument())
Trasform: docsWithHealth = inputDocuments.map(new
augmentDocumentWithHealth(healthCalcMetadata))
… other transformations may be done, all in memory …
Encode: outputRows = docsWithHealth.map(new
accountDocumentToJson())
Write: outputRows.saveAsTextFile(outputPath)

19. Examples (Java)
Class AugmentDocumentWithHealth implements
Function<AccountDocument, AccountDocument>
AccountDocument call(final AccountDocument document)
throws Exception { … return document with health … }
Class AccountHealthToAlerts implements
FlatMapFunction<AccountDocument, EventDocument>
Iterable<EventDocument> call(final AccountDocument
document) throws Exception { … generate alerts … }

20. Transformation function
●
Passed as parameter to Spark transformation:
map, reduce, filter, flatMap, mapPartitions
●
Can (read: should!!) be checked in Unit Tests
●
Serializable – sent to Spark worker serialized
●
Function must be idempotent!
●
May be passed immutable metadata

21. Luigi Workflow Engine
You build the tasks, it takes care of the plumbing

22. Why a workflow engine?
●
Managing many ETL jobs
●
Dependencies between jobs
●
Continue pipeline from point of failure
●
Separate workflow per service per date
●
Overview and drill-down status Web UI
●
Manual intervention

23. Workflow engines
●
Azkaban, by LinkedIn (mostly for Hadoop)
●
Oozie, by Apache (only for Hadoop)
●
Amazon Simple Workflow Service (too generic)
●
Amazon Data Pipeline (deeply tied to AWS)
●
Luigi, by Spotify (customizable) – our choice!

24. What is Luigi
●
Like Makefile – but in Python, and for data
●
Dependencies are managed directly in code
●
Generic and easily extendable
●
Visualization of task status and dependency
●
Command-line interface

25. Luigi Task Structure
●
Extend luigi.Task
Implement 4 methods:
●
def input(self) (optional)
●
def output(self)
●
def depends(self)
●
def run(self)

26. Luigi Task Example

27. Luigi Predefined Tasks
●
HadoopJobTask
●
SparkSubmitTask
●
CopyToIndex (ES)
●
HiveQueryTask
●
PigJobTask
●
CopyToTable (RDMS)
●
… many others

28. Luigi Task Parameters

29. Luigi Command-line

30. Luigi Task List

31. Luigi Dependency Graph

32. Luigi Dependency Graph

33. Luigi in Totango
This is how we do it

34. Our codebase is in Java
Java class is called inside the task run method

35. Jenkins for Luigi

36. Gameboy
●
Totango-specific controller for Luigi
●
Provides high level overview
●
Enable manual re-run of specific tasks
●
Monitor progress, performance, run time,
queue, worker load etc

37. Gameboy

38. Gameboy

39. Gameboy

40. Summary
●
Typical data flow – from raw data to insights
●
We use Spark for fast in-memory
transformations, all code is in Java
●
Our batch processing pipeline consist of a
series of tasks, which are managed in Luigi
●
We don't use all of Luigi's python abilities, and
we've added some new management abilities

41. Questions?
The end is only the beginning