SlideShare una empresa de Scribd logo
1 de 70
Descargar para leer sin conexión
Quanta: Quora’s Hierarchical
Counting System On HBase
Nikhil Garg Chun-Ho Hung
@nikhilgarg28 @chhung6
@Quora HBaseCon 6/12/17
HBaseCon2017 Quanta: Quora's hierarchical counting system on HBase
To Grow And Share World’s Knowledge
HBaseCon2017 Quanta: Quora's hierarchical counting system on HBase
HBaseCon2017 Quanta: Quora's hierarchical counting system on HBase
HBaseCon2017 Quanta: Quora's hierarchical counting system on HBase
Around 200 million monthly uniques
Millions of questions & answers
In hundreds of thousands of topics
Supported by < 100 engineers
Quora’s Big Data In Cloud
● All infrastructure is on AWS from t = 0
● MySQL and HBase are online production databases
● Redshift: ds2.8xlarge boxes, hundreds of terabytes of data
● Spark: d2.2xlarge, runs HDFS + YARN_MR + Spark, hundreds of terabytes of data
Quanta
Over 1 Billion Realtime Updates/Reads Everyday
● All features based on content views
● All Ads monitoring and tracking
● Exploring:
○ Internal Dashboards
○ Deduplication service
○ Count based ML Features
○ Monitoring native app
performance/reliability
Users of Quanta
1. Design Constraints
2. Update Algorithm
3. Macro Architecture
4. Towards Horizontal Scalability
5. Alternatives
1. Design Constraints
2. Update Algorithm
3. Macro Architecture
4. Towards Horizontal Scalability
5. Alternatives
HBaseCon2017 Quanta: Quora's hierarchical counting system on HBase
Design Constraints
● An explicit commit point after which updates are never lost.
● Maintain billions of counters -- not bound by a single machine.
● Very high volume of writes (> 100K/sec) and reads (>1M/sec).
Design Constraints
● An explicit commit point after which updates are never lost.
⇒ Persistent on disk, replicated
● Maintain billions of counters -- not bound by a single machine.
⇒ Distributed on multiple machines
● Very high volume of writes (> 100K/sec) and reads (>1M/sec).
⇒ Append-only writes
Counters
Design Constraints
● Low latency reads on a single counter and some “related” counters (~1ms)
Design Constraints
● Low latency reads on a single counter and some “related” counters (~1ms)
⇒ avoid random reads in the same request
⇒ heavy in-memory caching
⇒ storing in range-sorted format
Design Constraints
● Low latency reads on a single counter and some “related” counters (~1ms).
⇒ avoid random reads in the same request
⇒ heavy in-memory caching
⇒ storing in range-sorted format
● Writes lag behind by at most a couple of minutes.
Design Constraints
● Low latency reads on a single counter and some “related” counters (~1ms).
⇒ avoid random reads in the same request
⇒ heavy in-memory caching
⇒ storing in range-sorted format
● Writes lag behind by at most a couple of minutes.
⇒ can do asynchronous writes
Design Constraints
● Should be highly available for both reads and writes.
● Handle 2x the load with 2x the resources, should scale to 20-50x.
● Should be as cheap as possible.
Time Windows
Time Series
Counter
Arbitrary Time Bucketing
● Should support counting in rolling windows -- “How many times did this happen in last X”?
● Should support storing timeseries -- “How many times did this happen every X in the last
Y?”
● X and Y can be arbitrary time units.
Configurable Rolling Errors
Sunday Monday Tuesday Wednesday Thursday Friday
Last Week
Saturday
Sunday Monday Tuesday Wednesday Thursday Friday
Last Week
Saturday Sunday
Aggregation
Aggregation Groups
Dynamic Hierarchical Aggregation
aid2
uid
aid1
Increments on aid1 and aid2 should be propagated along the edges to uid
direct: 1
total: 1
direct: 1
total: 1
direct: 0
total: 2
aid2
uid
aid1
direct: 1
total: 1
direct: 2
total: 2
direct: 0
total: 3
Dynamic Hierarchical Aggregation
aid2
uid
aid1
When edge aid2 → uid is removed, we should pull back all previously propagated counts
direct: 1
total: 1
direct: 1
total: 1
direct: 0
total: 2
aid2
uid
aid1
direct: 1
total: 1
direct: 1
total: 1
direct: 0
total: 1
Arbitrary DAGs
1. Design Constraints
2. Update Algorithm
3. Macro Architecture
4. Towards Horizontal Scalability
5. Alternatives
Lazy Vs Eager Updates
Lazy Updates
● We don’t do anything when the graph changes.
● At query time, we find out all the ancestors of a counter and return the sum of their values.
Eager Updates
● When the graph changes, we change the counter information right away.
● At query time, we don’t have to read the graph -- can just return the value of the counter.
We Need Eager Updates
● Doing a lot of processing at query time isn’t acceptable for latency.
● However it’s okay if the writes take longer, they are aysnc after all.
● Graph updates are extremely rare compared to queries.
Increment (u, delta)
● Increment counter u by delta.
● Read descendents of u from
the graph.
● Increment all these
descendents by delta as well.
Update Algorithm V1
Add Edge (u, v)
● Read all direct counts of u.
● Read all descendants of v for
the graph.
● Add direct counts of u to all
descendents of v.
● Add edge u -> v in the graph.
Update Algorithm V1 On Multi-Level Graphs
A
B
C
A
B
C
Add Edge (u, v)
● Read all direct counts of u.
● Read all descendants of v for
the graph.
● Add direct counts of u to all
descendents of v.
● Add edge u -> v in the graph.
direct: 1
total: 1
direct: 1
total: 2
direct: 1
total: 1
direct: 1
total: 1
direct: 1
total: 2
direct: 1
total: ?
Update Algorithm V1 Fails On Multi-Level Graphs
A
B
C
A
B
C
Add Edge (u, v)
● Read all direct counts of u.
● Read all descendants of v for
the graph.
● Add direct counts of u to all
descendents of v.
● Add edge u -> v in the graph.
direct: 1
total: 1
direct: 1
total: 2
direct: 1
total: 1
direct: 1
total: 1
direct: 1
total: 2
direct: 1
total: 2
Update Algorithm V2
A
B
C
A
B
C
direct: 1
total: 1
direct: 1
total: 2
direct: 1
total: 1
direct: 1
total: 1
direct: 1
total: 2
direct: 1
total: 3
Add Edge (u, v)
● Read all direct counts of all
ancestors of u.
● Read all descendants of v for
the graph.
● Add direct counts of ancestors
of u to all descendents of v.
● Add edge u -> v in the graph.
Update Algorithm V2 Fails On “Diamonds”
Aa, a
B
b, a+b
C
C
c, a+c
Dd, a+b+d
Aa, a
B
b, a+b
C
C
c, a+c
Dd, ?
Add Edge (u, v)
● Read all direct counts of all
ancestors of u.
● Read all descendants of v for
the graph.
● Add direct counts of ancestors
of u to all descendents of v.
● Add edge u -> v in the graph.
Update Algorithm V2 Fails On “Diamonds”
Aa, a
B
b, a+b
C
C
c, a+c
Dd, a+b+d
Aa, a
B
b, a+b
C
C
c, a+c
Dd, (a+b+d)+a+c
Add Edge (u, v)
● Read all direct counts of all
ancestors of u.
● Read all descendants of v for
the graph.
● Add direct counts of ancestors
of u to all descendents of v.
● Add edge u -> v in the graph.
Update Algorithm V3
Aa, a
B
b, a+b
C
C
c, a+c
Dd, a+b+d
Aa, a
B
b, a+b
C
C
c, a+c
Dd, a+b+c+d
Add Edge (u, v)
● Read all direct counts of all
ancestors of u.
● Read all descendants of v for
the graph.
● Add direct counts of ancestors
of u to all descendents of v if
not already added.
● ...
Final Graph Update Algorithm
Add Edge (u, v)
for a in ancestors(u):
for d in descendants(v):
if d not in descendants(a):
total[d] += direct[a]
add_edge_in_graph(u, v)
ancestors(u)
descendants(v)
Final Graph Update Algorithm: Ordering
● Commutative with increments → can process increments and graph updates separately
● Can reorder graph updates as long as the final graph structure is correct.
● Can run the graph algorithm in a batched mode
1. Design Constraints
2. Update Algorithm
3. Macro Architecture
4. Towards Horizontal Scalability
5. Alternatives
Web
Journal
Graph
Storage
Counter
Storage
Processor
Quanta Client
Commit
Read
Pull
Read/Write
Read/Write
Quanta Client
Web
Journal
Graph
Storage
Counter
Storage
Processor
Client
Commit
Read
Pull
Read/Write
Read/Write
● Very thin client, knows how to talk to
Journal and Counter storage.
● Commits updates to Journal.
● All updates are thrift messages, so clients
can be written in any language.
● Reads count data directly from ‘Counter
Storage’
Journal
● Just a replicated persistent commit log.
Replication makes writes available.
● Stores updates temporarily till processor
pulls them to process
● At-least-once processing semantics.
● Any persistent commit log can work as
Journal. Currently using Scribe but soon
moving to Kafka.
Web
Journal
Graph
Storage
Counter
Storage
Processor
Client
Commit
Read
Pull
Read/Write
Read/Write
Processor
● Reads updates from Journal, and processes
them in batches by reading/updating
graph/counter storage.
● Graph updates also processed in batches --
two rounds of BFS to “download” graph
● Processing updates asynchronously in
batches is the single most important idea in
this architecture.
Web
Journal
Graph
Storage
Counter
Storage
Processor
Client
Commit
Read
Pull
Read/Write
Read/Write
Processor: Batched Asynchronous Updates
● Processor becomes stateless → don’t have
to worry about its availability.
● Can easily absorb spikes
● Reduce volume by deduplicating increments
● Can batch IO requests.
Web
Journal
Graph
Storage
Counter
Storage
Processor
Client
Commit
Read
Pull
Read/Write
Read/Write
Processor: Batched Asynchronous Updates
● Does some optimizations based on the graph
structure
● Currently implemented as system of stateless
minutely crons, on top of our distributed cron
architecture
● Lag of few minutes comes from implementing
on top of minutely crons
● Can reduce lag by processing data in smaller
batches
Web
Journal
Graph
Storage
Counter
Storage
Processor
Client
Commit
Read
Pull
Read/Write
Read/Write
Counter Storage
● Implemented on top of HBase.
● Each row denotes a single counter.
● Column family corresponds to a time window
(e.g “last week”)
● Columns in this column family correspond to
time buckets (either rolling or static).
Web
Journal
Graph
Storage
Counter
Storage
Processor
Client
Commit
Read
Pull
Read/Write
Read/Write
Counter Storage - HBase Schema
counter_key
Last day column family Last Week column family
d:11
Row Key
d:12 d:13 d:14 d:15 d:16
Counter Storage: Why HBase
● Hbase supports high write throughput
● Allows range scans -- can get data for related
counters in a sequential disk read
● Supports variable number of columns -- great
for both time-series and rolling windows
● Supports automatic garbage collection of old
data by setting TTL per column family.
Web
Journal
Graph
Storage
Counter
Storage
Processor
Client
Commit
Read
Pull
Read/Write
Read/Write
Counter Storage: Why HBase
● Bulk upload is very useful to backfill data into
Quanta
● High availability of Quanta reads (due to
HBase availability from multi-way replication)
Web
Journal
Graph
Storage
Counter
Storage
Processor
Client
Commit
Read
Pull
Read/Write
Read/Write
Graph Storage
● Built a general purpose graph datastore called
Graphiq on top of HBase.
● Data stored as sparse adjacency matrix.
● Schema -- 2 column families, one for
incoming edges, and one for outgoing edges
● HBase can serve negative queries through
Bloom Filters → good fit for graph traversals
Web
Journal
Graph
Storage
Counter
Storage
Processor
Client
Commit
Read
Pull
Read/Write
Read/Write
1. Design Constraints
2. Update Algorithm
3. Macro Architecture
4. Towards Horizontal Scalability
5. Alternatives
Sharding Quanta Processor
● Must shard processor to reach the desired throughput
● But sharding isn’t easy because there can be be race conditions between graph updates
and counter updates.
Increment (u, delta)
● Increment counter u by delta.
● Read descendents of u from the graph.
● Increment all these descendents by delta as
well.
Revisiting The Algorithm
Add Edge (u, v)
● Read all counter data of u.
● Read all descendants of v for
the graph.
● Add counter data of u to all
descendents of v.
● Add edge u -> v in the graph.
Increment (u, delta)
● Increment counter u by delta.
● Read descendents of u from the graph.
● Increment all these descendents by delta as
well.
Race Conditions
Add Edge (u, v)
● Read all counter data of u.
● Read all descendants of v for
the graph.
● Add counter data of u to all
descendents of v.
● Add edge u -> v in the graph.
● Counter updates don’t create a race condition with any other counter updates (assuming
increments are “atomic”)
● Volume of counter updates is 2-3 orders of magnitude higher than graph updates.
● Sharding Solution --
○ Partition processor in two independent parts -- one for graph and other for counter
○ Shard counter processor in as many shards as required
○ Synchronize all processors using an instance wide shared/exclusive lock
More Observations
Design -- single graph processor, sharded counter processor, shared/exclusive lock to synchronize
● Graph update volume is low → could scale to 20x on this design.
● Lock starvation for graph processor as ‘counter’ shards grow in the number
● Would ideally prefer to eliminate locks completely.
Design 1: Sharding With Shared/Exclusive Locks
Design -- single graph processor, sharded counter processor, shared/exclusive lock to synchronize
● Graph update volume is low → could scale to 20x on this design.
● Lock starvation for graph processor as ‘counter’ shards grow in the number
● Would ideally prefer to eliminate locks completely.
Decided to find ways of sharding without these problems.
Sharding Quanta Processor With Shared/Exclusive Locks
Observation: Connected Components Are Isolated
A
B C
D
P
Q R
S
Processor 1 Processor 2
Co-sharding counter/graph on connected components
A
B C
D
P
Q R
S
Processor 1 Processor 2
● Application owners define a shard key (by
overriding a function) for their graphs.
● We can now run counter/graph updates of a
shared sequentially and eliminate locks.
● However, connected components is a much
stronger condition than required.
Design 2: Co-Sharding Counter/Graph Processor
aid1
aid2
aid3
uid
● Can provide out-of-box support for co-sharding without users telling us the shard key.
● Key idea -- propagating information in the graph level-by-level.
● Will free up Quanta from closely managing partitioning and delegate it to someone else.
Could then easily move onto Spark.
● Not yet implemented, still on our TODO list.
Design 3: Out Of Box Co-Sharding
1. Design Constraints
2. Update Algorithm
3. Macro Architecture
4. Towards Horizontal Scalability
5. Alternatives
Alternatives: Twitter’s RainBird
● Distributed counting with aggregation and
temporal bucketing -- similar to Quanta.
● No dynamic hierarchical aggregation, only static
● Latency guarantees at ~100ms (Quanta is ~1ms)
● Built atop Cassandra
Alternatives: LinkedIn’s Pinot
● Powers LinkedIn’s product features
that are related to profile views.
● SQL-like language support --
aggregation, filtering, group by etc.
● Lots of moving parts &
configuration state -- high
operational cost.
Alternatives: Facebook’s Insights System
● Almost identical to Quanta -- does similar
things, built atop HBase, Scribe etc..
● Except, doesn’t support dynamic
hierarchical aggregation, only static
● Many of our design decisions were
inspired by this system.
Summary
● Quanta is a service built on top of HBase to do counting on dynamic graphs.
● Supports defining arbitrary time windows and configurable rolling error.
● Super low latency: 1ms.
● Currently doing 1 Billion updates/reads everyday. Built to scale to 20-50x.
● Powering all ads reporting as well as features based on content views. Many
more users lined up.
Nikhil Garg Chun-Ho Hung
@nikhilgarg28 @chhung6
Thank you!

Más contenido relacionado

La actualidad más candente

HBaseCon2017 Highly-Available HBase
HBaseCon2017 Highly-Available HBaseHBaseCon2017 Highly-Available HBase
HBaseCon2017 Highly-Available HBaseHBaseCon
 
HBaseCon2017 Data Product at AirBnB
HBaseCon2017 Data Product at AirBnBHBaseCon2017 Data Product at AirBnB
HBaseCon2017 Data Product at AirBnBHBaseCon
 
Apache Samza: Reliable Stream Processing Atop Apache Kafka and Hadoop YARN
Apache Samza: Reliable Stream Processing Atop Apache Kafka and Hadoop YARNApache Samza: Reliable Stream Processing Atop Apache Kafka and Hadoop YARN
Apache Samza: Reliable Stream Processing Atop Apache Kafka and Hadoop YARNblueboxtraveler
 
Will it Scale? The Secrets behind Scaling Stream Processing Applications
Will it Scale? The Secrets behind Scaling Stream Processing ApplicationsWill it Scale? The Secrets behind Scaling Stream Processing Applications
Will it Scale? The Secrets behind Scaling Stream Processing ApplicationsNavina Ramesh
 
Flink Forward Berlin 2017: Robert Metzger - Keep it going - How to reliably a...
Flink Forward Berlin 2017: Robert Metzger - Keep it going - How to reliably a...Flink Forward Berlin 2017: Robert Metzger - Keep it going - How to reliably a...
Flink Forward Berlin 2017: Robert Metzger - Keep it going - How to reliably a...Flink Forward
 
Kafka Summit NYC 2017 - Introducing Exactly Once Semantics in Apache Kafka
Kafka Summit NYC 2017 - Introducing Exactly Once Semantics in Apache KafkaKafka Summit NYC 2017 - Introducing Exactly Once Semantics in Apache Kafka
Kafka Summit NYC 2017 - Introducing Exactly Once Semantics in Apache Kafkaconfluent
 
S3, Cassandra or Outer Space? Dumping Time Series Data using Spark - Demi Ben...
S3, Cassandra or Outer Space? Dumping Time Series Data using Spark - Demi Ben...S3, Cassandra or Outer Space? Dumping Time Series Data using Spark - Demi Ben...
S3, Cassandra or Outer Space? Dumping Time Series Data using Spark - Demi Ben...Codemotion Tel Aviv
 
Scylla Summit 2018: Scylla Feature Talks - Scylla Streaming and Repair Updates
Scylla Summit 2018: Scylla Feature Talks - Scylla Streaming and Repair UpdatesScylla Summit 2018: Scylla Feature Talks - Scylla Streaming and Repair Updates
Scylla Summit 2018: Scylla Feature Talks - Scylla Streaming and Repair UpdatesScyllaDB
 
What's the time? ...and why? (Mattias Sax, Confluent) Kafka Summit SF 2019
What's the time? ...and why? (Mattias Sax, Confluent) Kafka Summit SF 2019What's the time? ...and why? (Mattias Sax, Confluent) Kafka Summit SF 2019
What's the time? ...and why? (Mattias Sax, Confluent) Kafka Summit SF 2019confluent
 
Uber Real Time Data Analytics
Uber Real Time Data AnalyticsUber Real Time Data Analytics
Uber Real Time Data AnalyticsAnkur Bansal
 
Scylla Summit 2018: Worry-free ingestion - flow-control of writes in Scylla
Scylla Summit 2018: Worry-free ingestion - flow-control of writes in ScyllaScylla Summit 2018: Worry-free ingestion - flow-control of writes in Scylla
Scylla Summit 2018: Worry-free ingestion - flow-control of writes in ScyllaScyllaDB
 
SignalFx Kafka Consumer Optimization
SignalFx Kafka Consumer OptimizationSignalFx Kafka Consumer Optimization
SignalFx Kafka Consumer OptimizationSignalFx
 
QCon London 2016 - Patterns of reliable in-stream processing @ Scale
QCon London 2016 - Patterns of reliable in-stream processing @ ScaleQCon London 2016 - Patterns of reliable in-stream processing @ Scale
QCon London 2016 - Patterns of reliable in-stream processing @ ScaleAlexey Kharlamov
 
Flink Forward Berlin 2017: Stefan Richter - A look at Flink's internal data s...
Flink Forward Berlin 2017: Stefan Richter - A look at Flink's internal data s...Flink Forward Berlin 2017: Stefan Richter - A look at Flink's internal data s...
Flink Forward Berlin 2017: Stefan Richter - A look at Flink's internal data s...Flink Forward
 
Flink Forward San Francisco 2018: Steven Wu - "Scaling Flink in Cloud"
Flink Forward San Francisco 2018: Steven Wu - "Scaling Flink in Cloud" Flink Forward San Francisco 2018: Steven Wu - "Scaling Flink in Cloud"
Flink Forward San Francisco 2018: Steven Wu - "Scaling Flink in Cloud" Flink Forward
 
Unified Stream Processing at Scale with Apache Samza - BDS2017
Unified Stream Processing at Scale with Apache Samza - BDS2017Unified Stream Processing at Scale with Apache Samza - BDS2017
Unified Stream Processing at Scale with Apache Samza - BDS2017Jacob Maes
 
stream-processing-at-linkedin-with-apache-samza
stream-processing-at-linkedin-with-apache-samzastream-processing-at-linkedin-with-apache-samza
stream-processing-at-linkedin-with-apache-samzaAbhishek Shivanna
 
Kafka Summit SF 2017 - Infrastructure for Streaming Applications
Kafka Summit SF 2017 - Infrastructure for Streaming Applications Kafka Summit SF 2017 - Infrastructure for Streaming Applications
Kafka Summit SF 2017 - Infrastructure for Streaming Applications confluent
 
The Flux Capacitor of Kafka Streams and ksqlDB (Matthias J. Sax, Confluent) K...
The Flux Capacitor of Kafka Streams and ksqlDB (Matthias J. Sax, Confluent) K...The Flux Capacitor of Kafka Streams and ksqlDB (Matthias J. Sax, Confluent) K...
The Flux Capacitor of Kafka Streams and ksqlDB (Matthias J. Sax, Confluent) K...HostedbyConfluent
 

La actualidad más candente (20)

HBaseCon2017 Highly-Available HBase
HBaseCon2017 Highly-Available HBaseHBaseCon2017 Highly-Available HBase
HBaseCon2017 Highly-Available HBase
 
HBaseCon2017 Data Product at AirBnB
HBaseCon2017 Data Product at AirBnBHBaseCon2017 Data Product at AirBnB
HBaseCon2017 Data Product at AirBnB
 
ApacheCon BigData Europe 2015
ApacheCon BigData Europe 2015 ApacheCon BigData Europe 2015
ApacheCon BigData Europe 2015
 
Apache Samza: Reliable Stream Processing Atop Apache Kafka and Hadoop YARN
Apache Samza: Reliable Stream Processing Atop Apache Kafka and Hadoop YARNApache Samza: Reliable Stream Processing Atop Apache Kafka and Hadoop YARN
Apache Samza: Reliable Stream Processing Atop Apache Kafka and Hadoop YARN
 
Will it Scale? The Secrets behind Scaling Stream Processing Applications
Will it Scale? The Secrets behind Scaling Stream Processing ApplicationsWill it Scale? The Secrets behind Scaling Stream Processing Applications
Will it Scale? The Secrets behind Scaling Stream Processing Applications
 
Flink Forward Berlin 2017: Robert Metzger - Keep it going - How to reliably a...
Flink Forward Berlin 2017: Robert Metzger - Keep it going - How to reliably a...Flink Forward Berlin 2017: Robert Metzger - Keep it going - How to reliably a...
Flink Forward Berlin 2017: Robert Metzger - Keep it going - How to reliably a...
 
Kafka Summit NYC 2017 - Introducing Exactly Once Semantics in Apache Kafka
Kafka Summit NYC 2017 - Introducing Exactly Once Semantics in Apache KafkaKafka Summit NYC 2017 - Introducing Exactly Once Semantics in Apache Kafka
Kafka Summit NYC 2017 - Introducing Exactly Once Semantics in Apache Kafka
 
S3, Cassandra or Outer Space? Dumping Time Series Data using Spark - Demi Ben...
S3, Cassandra or Outer Space? Dumping Time Series Data using Spark - Demi Ben...S3, Cassandra or Outer Space? Dumping Time Series Data using Spark - Demi Ben...
S3, Cassandra or Outer Space? Dumping Time Series Data using Spark - Demi Ben...
 
Scylla Summit 2018: Scylla Feature Talks - Scylla Streaming and Repair Updates
Scylla Summit 2018: Scylla Feature Talks - Scylla Streaming and Repair UpdatesScylla Summit 2018: Scylla Feature Talks - Scylla Streaming and Repair Updates
Scylla Summit 2018: Scylla Feature Talks - Scylla Streaming and Repair Updates
 
What's the time? ...and why? (Mattias Sax, Confluent) Kafka Summit SF 2019
What's the time? ...and why? (Mattias Sax, Confluent) Kafka Summit SF 2019What's the time? ...and why? (Mattias Sax, Confluent) Kafka Summit SF 2019
What's the time? ...and why? (Mattias Sax, Confluent) Kafka Summit SF 2019
 
Uber Real Time Data Analytics
Uber Real Time Data AnalyticsUber Real Time Data Analytics
Uber Real Time Data Analytics
 
Scylla Summit 2018: Worry-free ingestion - flow-control of writes in Scylla
Scylla Summit 2018: Worry-free ingestion - flow-control of writes in ScyllaScylla Summit 2018: Worry-free ingestion - flow-control of writes in Scylla
Scylla Summit 2018: Worry-free ingestion - flow-control of writes in Scylla
 
SignalFx Kafka Consumer Optimization
SignalFx Kafka Consumer OptimizationSignalFx Kafka Consumer Optimization
SignalFx Kafka Consumer Optimization
 
QCon London 2016 - Patterns of reliable in-stream processing @ Scale
QCon London 2016 - Patterns of reliable in-stream processing @ ScaleQCon London 2016 - Patterns of reliable in-stream processing @ Scale
QCon London 2016 - Patterns of reliable in-stream processing @ Scale
 
Flink Forward Berlin 2017: Stefan Richter - A look at Flink's internal data s...
Flink Forward Berlin 2017: Stefan Richter - A look at Flink's internal data s...Flink Forward Berlin 2017: Stefan Richter - A look at Flink's internal data s...
Flink Forward Berlin 2017: Stefan Richter - A look at Flink's internal data s...
 
Flink Forward San Francisco 2018: Steven Wu - "Scaling Flink in Cloud"
Flink Forward San Francisco 2018: Steven Wu - "Scaling Flink in Cloud" Flink Forward San Francisco 2018: Steven Wu - "Scaling Flink in Cloud"
Flink Forward San Francisco 2018: Steven Wu - "Scaling Flink in Cloud"
 
Unified Stream Processing at Scale with Apache Samza - BDS2017
Unified Stream Processing at Scale with Apache Samza - BDS2017Unified Stream Processing at Scale with Apache Samza - BDS2017
Unified Stream Processing at Scale with Apache Samza - BDS2017
 
stream-processing-at-linkedin-with-apache-samza
stream-processing-at-linkedin-with-apache-samzastream-processing-at-linkedin-with-apache-samza
stream-processing-at-linkedin-with-apache-samza
 
Kafka Summit SF 2017 - Infrastructure for Streaming Applications
Kafka Summit SF 2017 - Infrastructure for Streaming Applications Kafka Summit SF 2017 - Infrastructure for Streaming Applications
Kafka Summit SF 2017 - Infrastructure for Streaming Applications
 
The Flux Capacitor of Kafka Streams and ksqlDB (Matthias J. Sax, Confluent) K...
The Flux Capacitor of Kafka Streams and ksqlDB (Matthias J. Sax, Confluent) K...The Flux Capacitor of Kafka Streams and ksqlDB (Matthias J. Sax, Confluent) K...
The Flux Capacitor of Kafka Streams and ksqlDB (Matthias J. Sax, Confluent) K...
 

Similar a HBaseCon2017 Quanta: Quora's hierarchical counting system on HBase

FOSDEM 2019: M3, Prometheus and Graphite with metrics and monitoring in an in...
FOSDEM 2019: M3, Prometheus and Graphite with metrics and monitoring in an in...FOSDEM 2019: M3, Prometheus and Graphite with metrics and monitoring in an in...
FOSDEM 2019: M3, Prometheus and Graphite with metrics and monitoring in an in...Rob Skillington
 
Approximate Queries and Graph Streams on Apache Flink - Theodore Vasiloudis -...
Approximate Queries and Graph Streams on Apache Flink - Theodore Vasiloudis -...Approximate Queries and Graph Streams on Apache Flink - Theodore Vasiloudis -...
Approximate Queries and Graph Streams on Apache Flink - Theodore Vasiloudis -...Seattle Apache Flink Meetup
 
Approximate queries and graph streams on Flink, theodore vasiloudis, seattle...
Approximate queries and graph streams on Flink, theodore vasiloudis,  seattle...Approximate queries and graph streams on Flink, theodore vasiloudis,  seattle...
Approximate queries and graph streams on Flink, theodore vasiloudis, seattle...Bowen Li
 
Scalable real-time processing techniques
Scalable real-time processing techniquesScalable real-time processing techniques
Scalable real-time processing techniquesLars Albertsson
 
Kafka used at scale to deliver real-time notifications
Kafka used at scale to deliver real-time notificationsKafka used at scale to deliver real-time notifications
Kafka used at scale to deliver real-time notificationsSérgio Nunes
 
Graph processing
Graph processingGraph processing
Graph processingyeahjs
 
Data Science in the Cloud @StitchFix
Data Science in the Cloud @StitchFixData Science in the Cloud @StitchFix
Data Science in the Cloud @StitchFixC4Media
 
Introduction to Akka Streams
Introduction to Akka StreamsIntroduction to Akka Streams
Introduction to Akka StreamsKnoldus Inc.
 
Cliff Click Explains GBM at Netflix October 10 2013
Cliff Click Explains GBM at Netflix October 10 2013Cliff Click Explains GBM at Netflix October 10 2013
Cliff Click Explains GBM at Netflix October 10 2013Sri Ambati
 
How Netflix Uses Amazon Kinesis Streams to Monitor and Optimize Large-scale N...
How Netflix Uses Amazon Kinesis Streams to Monitor and Optimize Large-scale N...How Netflix Uses Amazon Kinesis Streams to Monitor and Optimize Large-scale N...
How Netflix Uses Amazon Kinesis Streams to Monitor and Optimize Large-scale N...Amazon Web Services
 
Netflix Open Source Meetup Season 4 Episode 2
Netflix Open Source Meetup Season 4 Episode 2Netflix Open Source Meetup Season 4 Episode 2
Netflix Open Source Meetup Season 4 Episode 2aspyker
 
Keynote: Building and Operating A Serverless Streaming Runtime for Apache Bea...
Keynote: Building and Operating A Serverless Streaming Runtime for Apache Bea...Keynote: Building and Operating A Serverless Streaming Runtime for Apache Bea...
Keynote: Building and Operating A Serverless Streaming Runtime for Apache Bea...Flink Forward
 
Using Event Streams in Serverless Applications
Using Event Streams in Serverless ApplicationsUsing Event Streams in Serverless Applications
Using Event Streams in Serverless ApplicationsJonathan Dee
 
Empowering the AWS DynamoDB™ application developer with Alternator
Empowering the AWS DynamoDB™ application developer with AlternatorEmpowering the AWS DynamoDB™ application developer with Alternator
Empowering the AWS DynamoDB™ application developer with AlternatorScyllaDB
 
Computer Graphics - Lecture 01 - 3D Programming I
Computer Graphics - Lecture 01 - 3D Programming IComputer Graphics - Lecture 01 - 3D Programming I
Computer Graphics - Lecture 01 - 3D Programming I💻 Anton Gerdelan
 
Flink Forward SF 2017: Srikanth Satya & Tom Kaitchuck - Pravega: Storage Rei...
Flink Forward SF 2017: Srikanth Satya & Tom Kaitchuck -  Pravega: Storage Rei...Flink Forward SF 2017: Srikanth Satya & Tom Kaitchuck -  Pravega: Storage Rei...
Flink Forward SF 2017: Srikanth Satya & Tom Kaitchuck - Pravega: Storage Rei...Flink Forward
 
Financial Cash-Flow Scripting: Beyond Valuation by Antoine Savine
Financial Cash-Flow Scripting: Beyond Valuation by Antoine SavineFinancial Cash-Flow Scripting: Beyond Valuation by Antoine Savine
Financial Cash-Flow Scripting: Beyond Valuation by Antoine SavineAntoine Savine
 
Malo Denielou - No shard left behind: Dynamic work rebalancing in Apache Beam
Malo Denielou - No shard left behind: Dynamic work rebalancing in Apache BeamMalo Denielou - No shard left behind: Dynamic work rebalancing in Apache Beam
Malo Denielou - No shard left behind: Dynamic work rebalancing in Apache BeamFlink Forward
 
A Graph Database That Scales - ArangoDB 3.7 Release Webinar
A Graph Database That Scales - ArangoDB 3.7 Release WebinarA Graph Database That Scales - ArangoDB 3.7 Release Webinar
A Graph Database That Scales - ArangoDB 3.7 Release WebinarArangoDB Database
 

Similar a HBaseCon2017 Quanta: Quora's hierarchical counting system on HBase (20)

FOSDEM 2019: M3, Prometheus and Graphite with metrics and monitoring in an in...
FOSDEM 2019: M3, Prometheus and Graphite with metrics and monitoring in an in...FOSDEM 2019: M3, Prometheus and Graphite with metrics and monitoring in an in...
FOSDEM 2019: M3, Prometheus and Graphite with metrics and monitoring in an in...
 
Approximate Queries and Graph Streams on Apache Flink - Theodore Vasiloudis -...
Approximate Queries and Graph Streams on Apache Flink - Theodore Vasiloudis -...Approximate Queries and Graph Streams on Apache Flink - Theodore Vasiloudis -...
Approximate Queries and Graph Streams on Apache Flink - Theodore Vasiloudis -...
 
Approximate queries and graph streams on Flink, theodore vasiloudis, seattle...
Approximate queries and graph streams on Flink, theodore vasiloudis,  seattle...Approximate queries and graph streams on Flink, theodore vasiloudis,  seattle...
Approximate queries and graph streams on Flink, theodore vasiloudis, seattle...
 
Scalable real-time processing techniques
Scalable real-time processing techniquesScalable real-time processing techniques
Scalable real-time processing techniques
 
Kafka used at scale to deliver real-time notifications
Kafka used at scale to deliver real-time notificationsKafka used at scale to deliver real-time notifications
Kafka used at scale to deliver real-time notifications
 
Graph processing
Graph processingGraph processing
Graph processing
 
Data Science in the Cloud @StitchFix
Data Science in the Cloud @StitchFixData Science in the Cloud @StitchFix
Data Science in the Cloud @StitchFix
 
Introduction to Akka Streams
Introduction to Akka StreamsIntroduction to Akka Streams
Introduction to Akka Streams
 
Cliff Click Explains GBM at Netflix October 10 2013
Cliff Click Explains GBM at Netflix October 10 2013Cliff Click Explains GBM at Netflix October 10 2013
Cliff Click Explains GBM at Netflix October 10 2013
 
How Netflix Uses Amazon Kinesis Streams to Monitor and Optimize Large-scale N...
How Netflix Uses Amazon Kinesis Streams to Monitor and Optimize Large-scale N...How Netflix Uses Amazon Kinesis Streams to Monitor and Optimize Large-scale N...
How Netflix Uses Amazon Kinesis Streams to Monitor and Optimize Large-scale N...
 
Netflix Open Source Meetup Season 4 Episode 2
Netflix Open Source Meetup Season 4 Episode 2Netflix Open Source Meetup Season 4 Episode 2
Netflix Open Source Meetup Season 4 Episode 2
 
Keynote: Building and Operating A Serverless Streaming Runtime for Apache Bea...
Keynote: Building and Operating A Serverless Streaming Runtime for Apache Bea...Keynote: Building and Operating A Serverless Streaming Runtime for Apache Bea...
Keynote: Building and Operating A Serverless Streaming Runtime for Apache Bea...
 
Using Event Streams in Serverless Applications
Using Event Streams in Serverless ApplicationsUsing Event Streams in Serverless Applications
Using Event Streams in Serverless Applications
 
Empowering the AWS DynamoDB™ application developer with Alternator
Empowering the AWS DynamoDB™ application developer with AlternatorEmpowering the AWS DynamoDB™ application developer with Alternator
Empowering the AWS DynamoDB™ application developer with Alternator
 
Computer Graphics - Lecture 01 - 3D Programming I
Computer Graphics - Lecture 01 - 3D Programming IComputer Graphics - Lecture 01 - 3D Programming I
Computer Graphics - Lecture 01 - 3D Programming I
 
Flink Forward SF 2017: Srikanth Satya & Tom Kaitchuck - Pravega: Storage Rei...
Flink Forward SF 2017: Srikanth Satya & Tom Kaitchuck -  Pravega: Storage Rei...Flink Forward SF 2017: Srikanth Satya & Tom Kaitchuck -  Pravega: Storage Rei...
Flink Forward SF 2017: Srikanth Satya & Tom Kaitchuck - Pravega: Storage Rei...
 
Financial Cash-Flow Scripting: Beyond Valuation by Antoine Savine
Financial Cash-Flow Scripting: Beyond Valuation by Antoine SavineFinancial Cash-Flow Scripting: Beyond Valuation by Antoine Savine
Financial Cash-Flow Scripting: Beyond Valuation by Antoine Savine
 
Malo Denielou - No shard left behind: Dynamic work rebalancing in Apache Beam
Malo Denielou - No shard left behind: Dynamic work rebalancing in Apache BeamMalo Denielou - No shard left behind: Dynamic work rebalancing in Apache Beam
Malo Denielou - No shard left behind: Dynamic work rebalancing in Apache Beam
 
Akka streams
Akka streamsAkka streams
Akka streams
 
A Graph Database That Scales - ArangoDB 3.7 Release Webinar
A Graph Database That Scales - ArangoDB 3.7 Release WebinarA Graph Database That Scales - ArangoDB 3.7 Release Webinar
A Graph Database That Scales - ArangoDB 3.7 Release Webinar
 

Más de HBaseCon

hbaseconasia2017: HBase on Beam
hbaseconasia2017: HBase on Beamhbaseconasia2017: HBase on Beam
hbaseconasia2017: HBase on BeamHBaseCon
 
hbaseconasia2017: HBase Disaster Recovery Solution at Huawei
hbaseconasia2017: HBase Disaster Recovery Solution at Huaweihbaseconasia2017: HBase Disaster Recovery Solution at Huawei
hbaseconasia2017: HBase Disaster Recovery Solution at HuaweiHBaseCon
 
hbaseconasia2017: Removable singularity: a story of HBase upgrade in Pinterest
hbaseconasia2017: Removable singularity: a story of HBase upgrade in Pinteresthbaseconasia2017: Removable singularity: a story of HBase upgrade in Pinterest
hbaseconasia2017: Removable singularity: a story of HBase upgrade in PinterestHBaseCon
 
hbaseconasia2017: HareQL:快速HBase查詢工具的發展過程
hbaseconasia2017: HareQL:快速HBase查詢工具的發展過程hbaseconasia2017: HareQL:快速HBase查詢工具的發展過程
hbaseconasia2017: HareQL:快速HBase查詢工具的發展過程HBaseCon
 
hbaseconasia2017: Apache HBase at Netease
hbaseconasia2017: Apache HBase at Neteasehbaseconasia2017: Apache HBase at Netease
hbaseconasia2017: Apache HBase at NeteaseHBaseCon
 
hbaseconasia2017: HBase在Hulu的使用和实践
hbaseconasia2017: HBase在Hulu的使用和实践hbaseconasia2017: HBase在Hulu的使用和实践
hbaseconasia2017: HBase在Hulu的使用和实践HBaseCon
 
hbaseconasia2017: 基于HBase的企业级大数据平台
hbaseconasia2017: 基于HBase的企业级大数据平台hbaseconasia2017: 基于HBase的企业级大数据平台
hbaseconasia2017: 基于HBase的企业级大数据平台HBaseCon
 
hbaseconasia2017: HBase at JD.com
hbaseconasia2017: HBase at JD.comhbaseconasia2017: HBase at JD.com
hbaseconasia2017: HBase at JD.comHBaseCon
 
hbaseconasia2017: Large scale data near-line loading method and architecture
hbaseconasia2017: Large scale data near-line loading method and architecturehbaseconasia2017: Large scale data near-line loading method and architecture
hbaseconasia2017: Large scale data near-line loading method and architectureHBaseCon
 
hbaseconasia2017: Ecosystems with HBase and CloudTable service at Huawei
hbaseconasia2017: Ecosystems with HBase and CloudTable service at Huaweihbaseconasia2017: Ecosystems with HBase and CloudTable service at Huawei
hbaseconasia2017: Ecosystems with HBase and CloudTable service at HuaweiHBaseCon
 
hbaseconasia2017: HBase Practice At XiaoMi
hbaseconasia2017: HBase Practice At XiaoMihbaseconasia2017: HBase Practice At XiaoMi
hbaseconasia2017: HBase Practice At XiaoMiHBaseCon
 
hbaseconasia2017: hbase-2.0.0
hbaseconasia2017: hbase-2.0.0hbaseconasia2017: hbase-2.0.0
hbaseconasia2017: hbase-2.0.0HBaseCon
 
HBaseCon2017 Democratizing HBase
HBaseCon2017 Democratizing HBaseHBaseCon2017 Democratizing HBase
HBaseCon2017 Democratizing HBaseHBaseCon
 
HBaseCon2017 Apache HBase at Didi
HBaseCon2017 Apache HBase at DidiHBaseCon2017 Apache HBase at Didi
HBaseCon2017 Apache HBase at DidiHBaseCon
 
HBaseCon2017 gohbase: Pure Go HBase Client
HBaseCon2017 gohbase: Pure Go HBase ClientHBaseCon2017 gohbase: Pure Go HBase Client
HBaseCon2017 gohbase: Pure Go HBase ClientHBaseCon
 
HBaseCon2017 Spark HBase Connector: Feature Rich and Efficient Access to HBas...
HBaseCon2017 Spark HBase Connector: Feature Rich and Efficient Access to HBas...HBaseCon2017 Spark HBase Connector: Feature Rich and Efficient Access to HBas...
HBaseCon2017 Spark HBase Connector: Feature Rich and Efficient Access to HBas...HBaseCon
 
HBaseCon2017 Efficient and portable data processing with Apache Beam and HBase
HBaseCon2017 Efficient and portable data processing with Apache Beam and HBaseHBaseCon2017 Efficient and portable data processing with Apache Beam and HBase
HBaseCon2017 Efficient and portable data processing with Apache Beam and HBaseHBaseCon
 
HBaseCon2017 HBase/Phoenix @ Scale @ Salesforce
HBaseCon2017 HBase/Phoenix @ Scale @ SalesforceHBaseCon2017 HBase/Phoenix @ Scale @ Salesforce
HBaseCon2017 HBase/Phoenix @ Scale @ SalesforceHBaseCon
 
HBaseCon2017 Community-Driven Graphs with JanusGraph
HBaseCon2017 Community-Driven Graphs with JanusGraphHBaseCon2017 Community-Driven Graphs with JanusGraph
HBaseCon2017 Community-Driven Graphs with JanusGraphHBaseCon
 
HBaseCon2017 Warp 10, a novel approach to managing and analyzing time series ...
HBaseCon2017 Warp 10, a novel approach to managing and analyzing time series ...HBaseCon2017 Warp 10, a novel approach to managing and analyzing time series ...
HBaseCon2017 Warp 10, a novel approach to managing and analyzing time series ...HBaseCon
 

Más de HBaseCon (20)

hbaseconasia2017: HBase on Beam
hbaseconasia2017: HBase on Beamhbaseconasia2017: HBase on Beam
hbaseconasia2017: HBase on Beam
 
hbaseconasia2017: HBase Disaster Recovery Solution at Huawei
hbaseconasia2017: HBase Disaster Recovery Solution at Huaweihbaseconasia2017: HBase Disaster Recovery Solution at Huawei
hbaseconasia2017: HBase Disaster Recovery Solution at Huawei
 
hbaseconasia2017: Removable singularity: a story of HBase upgrade in Pinterest
hbaseconasia2017: Removable singularity: a story of HBase upgrade in Pinteresthbaseconasia2017: Removable singularity: a story of HBase upgrade in Pinterest
hbaseconasia2017: Removable singularity: a story of HBase upgrade in Pinterest
 
hbaseconasia2017: HareQL:快速HBase查詢工具的發展過程
hbaseconasia2017: HareQL:快速HBase查詢工具的發展過程hbaseconasia2017: HareQL:快速HBase查詢工具的發展過程
hbaseconasia2017: HareQL:快速HBase查詢工具的發展過程
 
hbaseconasia2017: Apache HBase at Netease
hbaseconasia2017: Apache HBase at Neteasehbaseconasia2017: Apache HBase at Netease
hbaseconasia2017: Apache HBase at Netease
 
hbaseconasia2017: HBase在Hulu的使用和实践
hbaseconasia2017: HBase在Hulu的使用和实践hbaseconasia2017: HBase在Hulu的使用和实践
hbaseconasia2017: HBase在Hulu的使用和实践
 
hbaseconasia2017: 基于HBase的企业级大数据平台
hbaseconasia2017: 基于HBase的企业级大数据平台hbaseconasia2017: 基于HBase的企业级大数据平台
hbaseconasia2017: 基于HBase的企业级大数据平台
 
hbaseconasia2017: HBase at JD.com
hbaseconasia2017: HBase at JD.comhbaseconasia2017: HBase at JD.com
hbaseconasia2017: HBase at JD.com
 
hbaseconasia2017: Large scale data near-line loading method and architecture
hbaseconasia2017: Large scale data near-line loading method and architecturehbaseconasia2017: Large scale data near-line loading method and architecture
hbaseconasia2017: Large scale data near-line loading method and architecture
 
hbaseconasia2017: Ecosystems with HBase and CloudTable service at Huawei
hbaseconasia2017: Ecosystems with HBase and CloudTable service at Huaweihbaseconasia2017: Ecosystems with HBase and CloudTable service at Huawei
hbaseconasia2017: Ecosystems with HBase and CloudTable service at Huawei
 
hbaseconasia2017: HBase Practice At XiaoMi
hbaseconasia2017: HBase Practice At XiaoMihbaseconasia2017: HBase Practice At XiaoMi
hbaseconasia2017: HBase Practice At XiaoMi
 
hbaseconasia2017: hbase-2.0.0
hbaseconasia2017: hbase-2.0.0hbaseconasia2017: hbase-2.0.0
hbaseconasia2017: hbase-2.0.0
 
HBaseCon2017 Democratizing HBase
HBaseCon2017 Democratizing HBaseHBaseCon2017 Democratizing HBase
HBaseCon2017 Democratizing HBase
 
HBaseCon2017 Apache HBase at Didi
HBaseCon2017 Apache HBase at DidiHBaseCon2017 Apache HBase at Didi
HBaseCon2017 Apache HBase at Didi
 
HBaseCon2017 gohbase: Pure Go HBase Client
HBaseCon2017 gohbase: Pure Go HBase ClientHBaseCon2017 gohbase: Pure Go HBase Client
HBaseCon2017 gohbase: Pure Go HBase Client
 
HBaseCon2017 Spark HBase Connector: Feature Rich and Efficient Access to HBas...
HBaseCon2017 Spark HBase Connector: Feature Rich and Efficient Access to HBas...HBaseCon2017 Spark HBase Connector: Feature Rich and Efficient Access to HBas...
HBaseCon2017 Spark HBase Connector: Feature Rich and Efficient Access to HBas...
 
HBaseCon2017 Efficient and portable data processing with Apache Beam and HBase
HBaseCon2017 Efficient and portable data processing with Apache Beam and HBaseHBaseCon2017 Efficient and portable data processing with Apache Beam and HBase
HBaseCon2017 Efficient and portable data processing with Apache Beam and HBase
 
HBaseCon2017 HBase/Phoenix @ Scale @ Salesforce
HBaseCon2017 HBase/Phoenix @ Scale @ SalesforceHBaseCon2017 HBase/Phoenix @ Scale @ Salesforce
HBaseCon2017 HBase/Phoenix @ Scale @ Salesforce
 
HBaseCon2017 Community-Driven Graphs with JanusGraph
HBaseCon2017 Community-Driven Graphs with JanusGraphHBaseCon2017 Community-Driven Graphs with JanusGraph
HBaseCon2017 Community-Driven Graphs with JanusGraph
 
HBaseCon2017 Warp 10, a novel approach to managing and analyzing time series ...
HBaseCon2017 Warp 10, a novel approach to managing and analyzing time series ...HBaseCon2017 Warp 10, a novel approach to managing and analyzing time series ...
HBaseCon2017 Warp 10, a novel approach to managing and analyzing time series ...
 

Último

UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxUdaiappa Ramachandran
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXTarek Kalaji
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Brian Pichman
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarPrecisely
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfJamie (Taka) Wang
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 

Último (20)

UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptx
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBX
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity Webinar
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
201610817 - edge part1
201610817 - edge part1201610817 - edge part1
201610817 - edge part1
 

HBaseCon2017 Quanta: Quora's hierarchical counting system on HBase

  • 1. Quanta: Quora’s Hierarchical Counting System On HBase Nikhil Garg Chun-Ho Hung @nikhilgarg28 @chhung6 @Quora HBaseCon 6/12/17
  • 3. To Grow And Share World’s Knowledge
  • 7. Around 200 million monthly uniques Millions of questions & answers In hundreds of thousands of topics Supported by < 100 engineers
  • 8. Quora’s Big Data In Cloud ● All infrastructure is on AWS from t = 0 ● MySQL and HBase are online production databases ● Redshift: ds2.8xlarge boxes, hundreds of terabytes of data ● Spark: d2.2xlarge, runs HDFS + YARN_MR + Spark, hundreds of terabytes of data
  • 9. Quanta Over 1 Billion Realtime Updates/Reads Everyday
  • 10. ● All features based on content views ● All Ads monitoring and tracking ● Exploring: ○ Internal Dashboards ○ Deduplication service ○ Count based ML Features ○ Monitoring native app performance/reliability Users of Quanta
  • 11. 1. Design Constraints 2. Update Algorithm 3. Macro Architecture 4. Towards Horizontal Scalability 5. Alternatives
  • 12. 1. Design Constraints 2. Update Algorithm 3. Macro Architecture 4. Towards Horizontal Scalability 5. Alternatives
  • 14. Design Constraints ● An explicit commit point after which updates are never lost. ● Maintain billions of counters -- not bound by a single machine. ● Very high volume of writes (> 100K/sec) and reads (>1M/sec).
  • 15. Design Constraints ● An explicit commit point after which updates are never lost. ⇒ Persistent on disk, replicated ● Maintain billions of counters -- not bound by a single machine. ⇒ Distributed on multiple machines ● Very high volume of writes (> 100K/sec) and reads (>1M/sec). ⇒ Append-only writes
  • 17. Design Constraints ● Low latency reads on a single counter and some “related” counters (~1ms)
  • 18. Design Constraints ● Low latency reads on a single counter and some “related” counters (~1ms) ⇒ avoid random reads in the same request ⇒ heavy in-memory caching ⇒ storing in range-sorted format
  • 19. Design Constraints ● Low latency reads on a single counter and some “related” counters (~1ms). ⇒ avoid random reads in the same request ⇒ heavy in-memory caching ⇒ storing in range-sorted format ● Writes lag behind by at most a couple of minutes.
  • 20. Design Constraints ● Low latency reads on a single counter and some “related” counters (~1ms). ⇒ avoid random reads in the same request ⇒ heavy in-memory caching ⇒ storing in range-sorted format ● Writes lag behind by at most a couple of minutes. ⇒ can do asynchronous writes
  • 21. Design Constraints ● Should be highly available for both reads and writes. ● Handle 2x the load with 2x the resources, should scale to 20-50x. ● Should be as cheap as possible.
  • 23. Arbitrary Time Bucketing ● Should support counting in rolling windows -- “How many times did this happen in last X”? ● Should support storing timeseries -- “How many times did this happen every X in the last Y?” ● X and Y can be arbitrary time units.
  • 24. Configurable Rolling Errors Sunday Monday Tuesday Wednesday Thursday Friday Last Week Saturday Sunday Monday Tuesday Wednesday Thursday Friday Last Week Saturday Sunday
  • 26. Dynamic Hierarchical Aggregation aid2 uid aid1 Increments on aid1 and aid2 should be propagated along the edges to uid direct: 1 total: 1 direct: 1 total: 1 direct: 0 total: 2 aid2 uid aid1 direct: 1 total: 1 direct: 2 total: 2 direct: 0 total: 3
  • 27. Dynamic Hierarchical Aggregation aid2 uid aid1 When edge aid2 → uid is removed, we should pull back all previously propagated counts direct: 1 total: 1 direct: 1 total: 1 direct: 0 total: 2 aid2 uid aid1 direct: 1 total: 1 direct: 1 total: 1 direct: 0 total: 1
  • 29. 1. Design Constraints 2. Update Algorithm 3. Macro Architecture 4. Towards Horizontal Scalability 5. Alternatives
  • 30. Lazy Vs Eager Updates Lazy Updates ● We don’t do anything when the graph changes. ● At query time, we find out all the ancestors of a counter and return the sum of their values. Eager Updates ● When the graph changes, we change the counter information right away. ● At query time, we don’t have to read the graph -- can just return the value of the counter.
  • 31. We Need Eager Updates ● Doing a lot of processing at query time isn’t acceptable for latency. ● However it’s okay if the writes take longer, they are aysnc after all. ● Graph updates are extremely rare compared to queries.
  • 32. Increment (u, delta) ● Increment counter u by delta. ● Read descendents of u from the graph. ● Increment all these descendents by delta as well. Update Algorithm V1 Add Edge (u, v) ● Read all direct counts of u. ● Read all descendants of v for the graph. ● Add direct counts of u to all descendents of v. ● Add edge u -> v in the graph.
  • 33. Update Algorithm V1 On Multi-Level Graphs A B C A B C Add Edge (u, v) ● Read all direct counts of u. ● Read all descendants of v for the graph. ● Add direct counts of u to all descendents of v. ● Add edge u -> v in the graph. direct: 1 total: 1 direct: 1 total: 2 direct: 1 total: 1 direct: 1 total: 1 direct: 1 total: 2 direct: 1 total: ?
  • 34. Update Algorithm V1 Fails On Multi-Level Graphs A B C A B C Add Edge (u, v) ● Read all direct counts of u. ● Read all descendants of v for the graph. ● Add direct counts of u to all descendents of v. ● Add edge u -> v in the graph. direct: 1 total: 1 direct: 1 total: 2 direct: 1 total: 1 direct: 1 total: 1 direct: 1 total: 2 direct: 1 total: 2
  • 35. Update Algorithm V2 A B C A B C direct: 1 total: 1 direct: 1 total: 2 direct: 1 total: 1 direct: 1 total: 1 direct: 1 total: 2 direct: 1 total: 3 Add Edge (u, v) ● Read all direct counts of all ancestors of u. ● Read all descendants of v for the graph. ● Add direct counts of ancestors of u to all descendents of v. ● Add edge u -> v in the graph.
  • 36. Update Algorithm V2 Fails On “Diamonds” Aa, a B b, a+b C C c, a+c Dd, a+b+d Aa, a B b, a+b C C c, a+c Dd, ? Add Edge (u, v) ● Read all direct counts of all ancestors of u. ● Read all descendants of v for the graph. ● Add direct counts of ancestors of u to all descendents of v. ● Add edge u -> v in the graph.
  • 37. Update Algorithm V2 Fails On “Diamonds” Aa, a B b, a+b C C c, a+c Dd, a+b+d Aa, a B b, a+b C C c, a+c Dd, (a+b+d)+a+c Add Edge (u, v) ● Read all direct counts of all ancestors of u. ● Read all descendants of v for the graph. ● Add direct counts of ancestors of u to all descendents of v. ● Add edge u -> v in the graph.
  • 38. Update Algorithm V3 Aa, a B b, a+b C C c, a+c Dd, a+b+d Aa, a B b, a+b C C c, a+c Dd, a+b+c+d Add Edge (u, v) ● Read all direct counts of all ancestors of u. ● Read all descendants of v for the graph. ● Add direct counts of ancestors of u to all descendents of v if not already added. ● ...
  • 39. Final Graph Update Algorithm Add Edge (u, v) for a in ancestors(u): for d in descendants(v): if d not in descendants(a): total[d] += direct[a] add_edge_in_graph(u, v) ancestors(u) descendants(v)
  • 40. Final Graph Update Algorithm: Ordering ● Commutative with increments → can process increments and graph updates separately ● Can reorder graph updates as long as the final graph structure is correct. ● Can run the graph algorithm in a batched mode
  • 41. 1. Design Constraints 2. Update Algorithm 3. Macro Architecture 4. Towards Horizontal Scalability 5. Alternatives
  • 43. Quanta Client Web Journal Graph Storage Counter Storage Processor Client Commit Read Pull Read/Write Read/Write ● Very thin client, knows how to talk to Journal and Counter storage. ● Commits updates to Journal. ● All updates are thrift messages, so clients can be written in any language. ● Reads count data directly from ‘Counter Storage’
  • 44. Journal ● Just a replicated persistent commit log. Replication makes writes available. ● Stores updates temporarily till processor pulls them to process ● At-least-once processing semantics. ● Any persistent commit log can work as Journal. Currently using Scribe but soon moving to Kafka. Web Journal Graph Storage Counter Storage Processor Client Commit Read Pull Read/Write Read/Write
  • 45. Processor ● Reads updates from Journal, and processes them in batches by reading/updating graph/counter storage. ● Graph updates also processed in batches -- two rounds of BFS to “download” graph ● Processing updates asynchronously in batches is the single most important idea in this architecture. Web Journal Graph Storage Counter Storage Processor Client Commit Read Pull Read/Write Read/Write
  • 46. Processor: Batched Asynchronous Updates ● Processor becomes stateless → don’t have to worry about its availability. ● Can easily absorb spikes ● Reduce volume by deduplicating increments ● Can batch IO requests. Web Journal Graph Storage Counter Storage Processor Client Commit Read Pull Read/Write Read/Write
  • 47. Processor: Batched Asynchronous Updates ● Does some optimizations based on the graph structure ● Currently implemented as system of stateless minutely crons, on top of our distributed cron architecture ● Lag of few minutes comes from implementing on top of minutely crons ● Can reduce lag by processing data in smaller batches Web Journal Graph Storage Counter Storage Processor Client Commit Read Pull Read/Write Read/Write
  • 48. Counter Storage ● Implemented on top of HBase. ● Each row denotes a single counter. ● Column family corresponds to a time window (e.g “last week”) ● Columns in this column family correspond to time buckets (either rolling or static). Web Journal Graph Storage Counter Storage Processor Client Commit Read Pull Read/Write Read/Write
  • 49. Counter Storage - HBase Schema counter_key Last day column family Last Week column family d:11 Row Key d:12 d:13 d:14 d:15 d:16
  • 50. Counter Storage: Why HBase ● Hbase supports high write throughput ● Allows range scans -- can get data for related counters in a sequential disk read ● Supports variable number of columns -- great for both time-series and rolling windows ● Supports automatic garbage collection of old data by setting TTL per column family. Web Journal Graph Storage Counter Storage Processor Client Commit Read Pull Read/Write Read/Write
  • 51. Counter Storage: Why HBase ● Bulk upload is very useful to backfill data into Quanta ● High availability of Quanta reads (due to HBase availability from multi-way replication) Web Journal Graph Storage Counter Storage Processor Client Commit Read Pull Read/Write Read/Write
  • 52. Graph Storage ● Built a general purpose graph datastore called Graphiq on top of HBase. ● Data stored as sparse adjacency matrix. ● Schema -- 2 column families, one for incoming edges, and one for outgoing edges ● HBase can serve negative queries through Bloom Filters → good fit for graph traversals Web Journal Graph Storage Counter Storage Processor Client Commit Read Pull Read/Write Read/Write
  • 53. 1. Design Constraints 2. Update Algorithm 3. Macro Architecture 4. Towards Horizontal Scalability 5. Alternatives
  • 54. Sharding Quanta Processor ● Must shard processor to reach the desired throughput ● But sharding isn’t easy because there can be be race conditions between graph updates and counter updates.
  • 55. Increment (u, delta) ● Increment counter u by delta. ● Read descendents of u from the graph. ● Increment all these descendents by delta as well. Revisiting The Algorithm Add Edge (u, v) ● Read all counter data of u. ● Read all descendants of v for the graph. ● Add counter data of u to all descendents of v. ● Add edge u -> v in the graph.
  • 56. Increment (u, delta) ● Increment counter u by delta. ● Read descendents of u from the graph. ● Increment all these descendents by delta as well. Race Conditions Add Edge (u, v) ● Read all counter data of u. ● Read all descendants of v for the graph. ● Add counter data of u to all descendents of v. ● Add edge u -> v in the graph.
  • 57. ● Counter updates don’t create a race condition with any other counter updates (assuming increments are “atomic”) ● Volume of counter updates is 2-3 orders of magnitude higher than graph updates. ● Sharding Solution -- ○ Partition processor in two independent parts -- one for graph and other for counter ○ Shard counter processor in as many shards as required ○ Synchronize all processors using an instance wide shared/exclusive lock More Observations
  • 58. Design -- single graph processor, sharded counter processor, shared/exclusive lock to synchronize ● Graph update volume is low → could scale to 20x on this design. ● Lock starvation for graph processor as ‘counter’ shards grow in the number ● Would ideally prefer to eliminate locks completely. Design 1: Sharding With Shared/Exclusive Locks
  • 59. Design -- single graph processor, sharded counter processor, shared/exclusive lock to synchronize ● Graph update volume is low → could scale to 20x on this design. ● Lock starvation for graph processor as ‘counter’ shards grow in the number ● Would ideally prefer to eliminate locks completely. Decided to find ways of sharding without these problems. Sharding Quanta Processor With Shared/Exclusive Locks
  • 60. Observation: Connected Components Are Isolated A B C D P Q R S Processor 1 Processor 2
  • 61. Co-sharding counter/graph on connected components A B C D P Q R S Processor 1 Processor 2
  • 62. ● Application owners define a shard key (by overriding a function) for their graphs. ● We can now run counter/graph updates of a shared sequentially and eliminate locks. ● However, connected components is a much stronger condition than required. Design 2: Co-Sharding Counter/Graph Processor aid1 aid2 aid3 uid
  • 63. ● Can provide out-of-box support for co-sharding without users telling us the shard key. ● Key idea -- propagating information in the graph level-by-level. ● Will free up Quanta from closely managing partitioning and delegate it to someone else. Could then easily move onto Spark. ● Not yet implemented, still on our TODO list. Design 3: Out Of Box Co-Sharding
  • 64. 1. Design Constraints 2. Update Algorithm 3. Macro Architecture 4. Towards Horizontal Scalability 5. Alternatives
  • 65. Alternatives: Twitter’s RainBird ● Distributed counting with aggregation and temporal bucketing -- similar to Quanta. ● No dynamic hierarchical aggregation, only static ● Latency guarantees at ~100ms (Quanta is ~1ms) ● Built atop Cassandra
  • 66. Alternatives: LinkedIn’s Pinot ● Powers LinkedIn’s product features that are related to profile views. ● SQL-like language support -- aggregation, filtering, group by etc. ● Lots of moving parts & configuration state -- high operational cost.
  • 67. Alternatives: Facebook’s Insights System ● Almost identical to Quanta -- does similar things, built atop HBase, Scribe etc.. ● Except, doesn’t support dynamic hierarchical aggregation, only static ● Many of our design decisions were inspired by this system.
  • 69. ● Quanta is a service built on top of HBase to do counting on dynamic graphs. ● Supports defining arbitrary time windows and configurable rolling error. ● Super low latency: 1ms. ● Currently doing 1 Billion updates/reads everyday. Built to scale to 20-50x. ● Powering all ads reporting as well as features based on content views. Many more users lined up.
  • 70. Nikhil Garg Chun-Ho Hung @nikhilgarg28 @chhung6 Thank you!