2. One of the Things to Watch at
Strata
TechCrunch article:
“… An interesting item that made the top
terms list is “alluxio,” which is the recently
renamed Tachyon project. Alluxio is a virtual
distributed storage system, and it has a
memory-centric architecture that enables
data sharing across clusters at memory
speed. … “
2
3. Who Are We?
• Calvin Jia
• SWE @ Alluxio, Inc.
• #1 Alluxio contributor
• Twitter: @JiaCalvin
• Jiri Simsa
• SWE @ Alluxio, Inc
• CMU Ph.D. & Google
• Twitter: @jsimsa
3
4. Alluxio Inc.
• Founded by Alluxio creators and top
committers
• Formerly Tachyon Nexus, Inc.
• $7.5 million Series A by Andreessen Horowitz
• Committed to the Alluxio Open Source
Project
• Company Website: http://www.alluxio.com
4
7. Memory Speed
• Memory-centric architecture designed for memory I/O
Virtual
• Abstracts persistent storage from applications
Distributed
• Designed to scale with nothing but commodity hardware
Open Source
• One of the fastest growing project communities
7
13. Simple Examples
• Data sharing between frameworks
• Data resilience during application crashes
• Consolidate memory usage and alleviate
GC issues
13
14. Spark Job
Spark
Memory
block 1
block 3
Hadoop MR Job
YARN
HDFS / Amazon S3
block 1
block 3
block 2
block 4
storage engine &
execution engine
same process
Data Sharing Between Frameworks
Inter-process sharing slowed down by network and/or disk I/O
14
15. Data Sharing Between Frameworks
Spark Job
Spark Memory
Hadoop MR Job
YARN
HDFS / Amazon S3
block 1
block 3
block 2
block 4
HDFS
disk
block 1
block 3
block 2
block 4
Alluxio
In-Memory
block 1
block 3 block 4
storage engine &
execution engine
same process
Inter-process sharing can happen at memory speed
15
16. Data Resilience during Crashes
Spark Task
Spark Memory
block manager
block 1
block 3
HDFS / Amazon S3
block 1
block 3
block 2
block 4
storage engine &
execution engine
same process
Process crash requires network and/or disk I/O to re-read the data
16
17. Data Resilience during Crashes
Crash
Spark Memory
block manager
block 1
block 3
HDFS / Amazon S3
block 1
block 3
block 2
block 4
storage engine &
execution engine
same process
Process crash requires network and/or disk I/O to re-read the data
17
18. HDFS / Amazon S3
Data Resilience during Crashes
block 1
block 3
block 2
block 4
Crash
storage engine &
execution engine
same process
Process crash requires network and/or disk I/O to re-read the data
18
19. Data Resilience during Crashes
Spark Task
Spark Memory
block manager
storage engine &
execution engine
same process
HDFS
disk
block 1
block 3
block 2
block 4
Alluxio
In-Memory
block 1
block 3 block 4
Process crash only needs memory I/O to re-read the data
19
20. Data Resilience during Crashes
Crash
storage engine &
execution engine
same process
Process crash only needs memory I/O to re-read the data
HDFS
disk
block 1
block 3
block 2
block 4
Alluxio
In-Memory
block 1
block 3 block 4
20
22. Consolidating Memory
Spark Job1
Spark mem
Spark Job2
Spark mem
HDFS / Amazon S3
block 1
block 3
block 2
block 4
storage engine &
execution engine
same process
HDFS
disk
block 1
block 3
block 2
block 4
Alluxio
In-Memory
block 1
block 3 block 4
Data not duplicated at memory-level
22
23. Case Study: Barclays
Making the Impossible Possible with Tachyon: Accelerate Spark
Jobs from Hours to Seconds
• Application: SparkSQL + Spark RDDs
• Alluxio Storage Layer: MEM
• Backend Storage: None
• Result: Speeding up Spark jobs from hours to seconds
23
24. Common Questions
– Memory speed sharing among distributed applications
HDFS interface compatible
– GC overhead introduced by in-memory caching
Off-Heap Memory Management
– Data set could be larger than available memory
Tiered storage
24
26. Motivation
• Memory resources are still constrained
• Alluxio data management logic is not
limited to memory
• Storage resources available on compute
clusters
26
28. Tiered Storage
• Extends Alluxio with support for SSDs and/or
HDDs storage
• Different tiers have different characteristics
– Keep hot data in fast but limited storage
– Keep warm data in slower but abundant storage
• Workers manage their own storage
• Data allocation and eviction is driven by
application access
28
31. Automatic Data Migration
• Data can be evicted to lower layers if it is “cooling down”
• Data can be promoted to upper layers if it is “warming
up”
Evict stale data to
lower tier
Promote hot data to
upper tier
31
32. Pluggable Policies
• Policies can be customized to suit
workloads
• Defaults provided for general scenarios
• Advanced users can optimize with
additional knowledge
– For example: Optimize for iterations
32
33. Case Study: Baidu
Baidu Queries Data 30 Times Faster with Alluxio
• Application: Spark
• Alluxio Storage: MEM + HDD
• Backend Storage: Baidu’s File System
• 200+ nodes deployment, 2PB+ managed space
• Result: Speeding up data querying by 30x
33
38. Motivation
• At large organizations, data spans many storage
systems (object storage, network / distributed file
systems, DBs)
• Application logic needs to integrate with different types
of storage systems
• Data needs to be moved around to work around
application limitations
• In-house storage layers are built to address limitations
of legacy storage systems
38
39. Transparent Naming
• Applications can transparently and efficiently interact
with remote storage through Alluxio.
• Applications do not need to use different APIs for
interacting with different storage systems.
alluxio://host:port/
data users
reports sales alice bob
s3n://bucket/directory
data users
reports sales alice bob
Alluxio Storage System
39
40. Single Namespace
• Applications can read and write different storage
systems.
• Decouples data location from application
alluxio://host:port/
data users
reports sales alice bob
hdfs://host:port/
users
alice bob
s3n://bucket/directory
reports sales
Alluxio Storage System A
Storage System B
40
42. Alluxio Benefits
42
• Enable new workloads across storage systems
• Work with the framework of your choice
• Scale storage and compute independently
Good afternoon everyone, and welcome to the Alluxio features talk. We will give an introduction to Alluxio and specifically go over two fundamental features in Alluxio. By the end of the talk, you should have a good idea as to why we believe Alluxio is qualified as a “Data Innovation”. First, could I get a show of hands of who’s already attended the Alluxio talk early today? Great, you guys will have a lot more insight if you watch the recording of that talk after this one.
I want to start off by introducing us. I’m Calvin, the top contributor to the project I’ve been working on the Alluxio project for a little over 3 years now. I’m currently a software engineer at Alluxio, Inc. Joining me for this talk is my colleague, Jiri. He’s also a software engineer at Alluxio, Inc and has experience working at Google as well as a PhD from CMU. Both of our twitter accounts are here if you want to follow us for the latest news about the project.
I mentioned we are both working at Alluxio Inc, which is a company dedicated to growing and building the Alluxio open source project. We were formerly known as Tachyon Nexus and are backed by A16Z. If you are interested in learning more about us, our company site is alluxio.com. And of course, if we’ve impressed you enough and you want to work with us, we are hiring!
Now let’s dive into the talk. There will be three sections, the first of which is an introduction to Alluxio.
Alluxio – Open Source Memory Speed Virtual Distributed Storage. That’s a lot of adjectives, is probably the first thing you thought. The second might be, Hey that sounds really familiar, isn’t that Tachyon? Much like how the company was originally Tachyon Nexus, the Tachyon project has recently become Alluxio with the 1.0 release.
More importantly, you are probably wondering what all those adjectives meant. Let’s start with Open Source, this means the system’s source code is available for anyone to download, look at, or contribute to. We have a large community working together on the project and are growing at a rapid pace. Memory speed is referring to the system architecture designed to take advantage of the growing amounts of memory in machines. Virtual describes the abstraction Alluxio provides to storage systems and applications, essentially allowing the two layers to be separate from each other. And finally distributed refers to the fact Alluxio can scale to many machines as long as you have more commodity hardware to throw at it.
Here is a more visual representation of where Alluxio sits and its function in the big data stack. Above Alluxio are many compute frameworks, such as map reduce and spark. These are connected by Alluxio to various storage systems which may not necessarily need to be file systems. However, Alluxio is more than just a connection layer, it provides great benefits by acting as a storage system which provides a view of all your data but only holds what is necessary.
The previous diagram implied that Alluxio is something new in the stack, not a replacement for anything. Why would a new layer emerge or be useful, don’t applications just directly communicate with storage? To see the answer to this question, we need to take a look at technologies trends, in particular, memory. Memory is awesome, its super performant and allows workloads to run at blazing speeds. In the past decade, we’ve seen a exponential growth in RAM throughput and steadily declining costs. Ands its not just Alluxio which has realized this direction, many compute frameworks have embraced the idea of being memory centric to achieve impressive results.
The add to the exponential throughput improvements of memory I/O are the steadily declining costs. The two factors generate the perfect situation for commodity technologies to be seriously designed with memory in mind.
I’ll go through some simple examples of the high level point I mentioned. Data sharing between frameworks, Data resilience during application crashes, and Consolidate memory usage and reduce GC
