An Introduction to Google Percolator, what is it and what is it used for ? How is it different from Apache Hadoop MapReduce ?

1. Google Percolator
● What is it ?
● What is it used for ?
● Percolator Vs MapReduce
● Architecture
www.semtech-solutions.co.nz info@semtech-solutions.co.nz

2. Percolator – What is it ?
● Incremental updates to Big Data
● Developed by Google
● Based on Google File System ( GFS )
● Provides transactions and locking
● Faster than comparable Map Reduce
● Developed by Google due to MapReduce limitations
www.semtech-solutions.co.nz info@semtech-solutions.co.nz

3. Percolator – What is it used for ?
● Iterative updates
● No need to batch process
● Update as data received
● Data in multi petabyte range
● Strong consistency needed
● Improved latency ( 100 x )
● Reduced document age ( 50 % )
● Random access to big data repository
www.semtech-solutions.co.nz info@semtech-solutions.co.nz

4. Percolator Vs MapReduce
Map Reduce
● Batch Processing
● No transactions
● Latency A
● Run time scales with data
● Code in C++
● Open source
● Uses HDFS
Percolator
– Iterative
– Transactions
– Latency 100 x A
– Incremental updates
– Code in Java ( mainly )
– Google owned
– Uses GFS
www.semtech-solutions.co.nz info@semtech-solutions.co.nz

5. Percolator – Architecture
● Applications are a sequence of observers
● An observer is called via a notification
● A notification is triggered when table data changes
● Application calls TabletServer via RPC
● TabletServer calls GFS ChunkServer

6. Percolator – Architecture
● Applications
– Series of observers
● Observer
– Completes task
– Updates table
● Next Observer called
– Via notification
● Percolator worker
– Scans for changes
– Sends notifications

7. Percolator – Architecture
Actual worker diagram including time stamping and locking
via Chubby lock server

8. Contact Us
● Feel free to contact us at
– www.semtech-solutions.co.nz
– info@semtech-solutions.co.nz
● We offer IT project consultancy
● We are happy to hear about your problems
● You can just pay for those hours that you need
● To solve your problems