This document describes CFS, a Cassandra-backed storage system for Hadoop. It discusses the motivations for building such a system given Cassandra's strengths in scalability and real-time data but lack of support for ad-hoc queries. The solution presented uses Cassandra to store Hadoop file metadata and block data, allowing tasks to run locally for data locality. It describes how files are written by splitting them into blocks and updating column families for inode metadata and block storage (sblocks). Reads check the inode to determine blocks and request local or remote data as needed.

1. CFS
Cassandra-backed storage for Hadoop
Nick Bailey
@nickmbailey
nick@datastax.com

2. ©2012 DataStax
Motivation
2

3. ©2012 DataStax
Help me Cassandra, you’re my
only hope
3

4. ©2012 DataStax
Cassandra
• Distributed architecture
• No SPOF
• Scalable
• Real time data
• No ad-hoc query support
4

5. ©2012 DataStax
Cassandra, why can’t you...
5

6. ©2012 DataStax
...do the things Hadoop was
built for.
6

7. ©2012 DataStax
Cassandra + Hadoop = <3
7

8. ©2012 DataStax
The Solution
• InputFormat/OutputFormat
• Unfortunately, still need a DFS
• Run tasktrackers/datanodes locally
• Data Locality FTW!
• Run namenode/jobtracker somewhere
• Since Cassandra 0.6 (the dark ages)
8

9. ©2012 DataStax
Ok, but what about these parts
that suck...
9

10. ©2012 DataStax
Do not want...
• Multiple hadoop stacks?
• SPOF?
• 3 JVMS?
10

11. ©2012 DataStax
CFS
11

12. ©2012 DataStax
Cassandra Data model in 1
minute
12

13. ©2012 DataStax
Column Families
• Column Family ~= Table
• Row Key + columns
• Columns are sparse
13

14. ©2012 DataStax
Static - Users Column Family
14
Row Key
nickmbailey password: * name: Nick
zznate password: * name: Nate phone: 512-7777

15. ©2012 DataStax
Select * from Users where name=Nick;
Secondary Indexes
15

16. ©2012 DataStax
Dynamic - Friends
16
Row Key
nickmbailey zznate: thobbs:
zznate jbeiber: thobbs: steve_watt:

17. ©2012 DataStax
So what about CFS...
17

18. ©2012 DataStax
Simple...
18

19. ©2012 DataStax 19

20. ©2012 DataStax
CF: inode
• Essentially, namenode replacement
• File metadata
20

21. ©2012 DataStax 21

22. ©2012 DataStax
CF: inode
• Row Key = UUID
• Allows for file renames
• Secondary indexes for file browsing
• Columns:
22
Column
filename /home/nick/data.txt
parent_path /home/nick/
attributes nick:nick:777
TimeUUID1 <block metadata>
TimeUUID2 <block metadata>
TimeUUID3 <block metadata>
...

23. ©2012 DataStax 23

24. ©2012 DataStax
CF: sblocks
• Essentially, datanode replacement
• Stores actual contents of files
• Each row is an hdfs block
• Row Key = Block ID
24
Column
TimeUUID1 <compressed file data>
TimeUUID2 <compressed file data>
TimeUUID3 <compressed file data>
...

25. ©2012 DataStax 25

26. ©2012 DataStax
Writes
• Write file metadata
• Split into blocks
• Still controlled by ‘dfs.block.size’
• also ‘cfs.local.subblock.size’
• Read in a block
• split into sub blocks
• Update inode, sblocks
• rinse, repeat
26

27. ©2012 DataStax 27

28. ©2012 DataStax
Reads
• Check for file in inode
• Determine appropriate blocks
• Request blocks via thrift
• If data is local...
• ...get location on local filesystem
• If data is remote...
• ...get actual file content via thrift
28

29. ©2012 DataStax
What Else?
• Current Implementation: 1.0.4
• <property>
<name>fs.cfs.impl</name>
<value>com.datastax.bdp.hadoop.cfs.CassandraFileSystem</value>
</property>
• Supports HDFS append()
• Immutability makes things easy
• See the first incarnation
• https://github.com/riptano/brisk
29

30. Want a job?
nick@datastax.com

31. Questions?