This document discusses integrating the BlobSeer data management system with the HGMDS distributed metadata management system to build a global file system deployed across multiple sites. Two approaches are considered: having multiple BlobSeer instances, one per site, or having a single BlobSeer-WAN instance spanning sites. The authors propose the latter approach with distributed version managers and provider managers to leverage data locality. Preliminary evaluations show BlobSeer-WAN performance on two sites is similar to vanilla BlobSeer on a single site. Next steps include further evaluation, integrating BlobSeer-WAN with HGMDS, and submitting a co-authored paper.

1. Towards a Scalable File System
Progress on adapting BlobSeer to WAN scale
for the HGMDS distributed metadata system
Viet-Trung Tran, Gabriel Antoniu, Alexandru Costan (INRIA - Rennes)
In collaboration with Kohei Hiraga, Osamu Tatebe (U Tsukuba)
FP3C meeting
Bordeaux, 2 – 3 September 2011

2. Plan
1. Background and context
2. Goal
3. Approach and solution
4. Preliminary evaluation
5. Conclusion
FP3C meeting – Bordeaux, 2-3 September 2011 -2

3. 1
Background
BlobSeer & HGMDS
FP3C meeting – Bordeaux, 2-3 September 2011 -3

4. BlobSeer: A large-scale data management
service
Generic data-management platform for huge, unstructured data
• Huge data (TB) : BLOBs
• Highly concurrent, fine-grain access (MB): R/W/A
• Prototype available
Key design features
• Decentralized metadata management
• Beyond MVCC: multiversioning exposed to the user
• Lock-free write access through versioning
A back-end for higher-level, sophisticated data management systems
FP3C meeting – Bordeaux, 2-3 September 2011 -4

5. BlobSeer: Architecture
Clients Providers
• Perform fine grain blob accesses
Providers
• Store the pages of the blob
Provider manager
• Monitors the providers
• Favours data load balancing Provider
Clients manager
Metadata providers
• Store information about page location Version
Version manager manager
• Ensures concurrency control
Metadata providers
FP3C meeting – Bordeaux, 2-3 September 2011 -5

6. HGMDS: A distributed metadata
management system for global file systems
• Multi-master file system
The
Internet
metadata server (MDS).
Site A
Site B
• Managing inode structure. File system Clients
• High latency networks don't
affect metadata operation
HGMD HGMD
performance. S
S
- Both reading and writing.
• One MDS per site.
• Metadata versioning using mkdir/rmdir/ Propagate
create/stat/ updates in
vector clocks for collision unlink
background
detection. Site C
• Automatic collision resolution
by system side.
FP3C meeting – Bordeaux, 2-3 September 2011 -6

7. 2
Goal
A joint architecture integrating BlobSeer and HGMDS
FP3C meeting – Bordeaux, 2-3 September 2011 -7

8. Goal
BlobSeer HGMDS
Data management Metadata management
Typically on a single site Global scale, multiple sites
Idea: build a global file system deployed on multiple site by integrating
BlobSeer to HGMDS
Potential benefits:
• HGMDS: efficient multi-site file metadata management
• BlobSeer: concurrency-optimized access to globally shared data
FP3C meeting – Bordeaux, 2-3 September 2011 -8

9. 3
Our approach and solution
FP3C meeting – Bordeaux, 2-3 September 2011 -9

10. Two approaches
Multiple BlobSeer instances
• One BlobSeer / site
One single BlobSeer-WAN over distributed geographic
sites
FP3C meeting – Bordeaux, 2-3 September 2011 - 10

11. 1st approach: 1 BlobSeer instance / site
Client
FP3C meeting – Bordeaux, 2-3 September 2011 - 11

12. 1st approach: Zoom
High latency when accessing remote BLOBs:
• Too many remote requests for small metadata
EMETTEUR - NOM DE LA PRESENTATION - 12

13. 2nd approach: 1 BlobSeer-WAN instance
over distributed geographic sites
Multiple version managers
• 1 version manager/site
Multiple provider managers
• 1 provider manager/site
On each site
• Multiple data providers and metadata servers
• Data providers are under control of local provider manager
EMETTEUR - NOM DE LA PRESENTATION - 13

14. Idea: leverage locality
for remote metadata accesses
2
Metadata I/O is resolved locally
EMETTEUR - NOM DE LA PRESENTATION - 14

15. 2nd approach: I/O scheme in BlobSeer-WAN
Writing
• Publish version on local version manager
• Locally write metadata on local metadata servers
• Locally write data on local data providers
Reading (Read your write in many cases)
• Ask a version to local version manager
• Local metadata accesses
• Access remote/local providers if necessary
FP3C meeting – Bordeaux, 2-3 September 2011 - 15

16. Vector clocks and optimistic metadata
replication
FP3C meeting – Bordeaux, 2-3 September 2011 - 16

17. Expected benefits
• On WAN: BlobSeer coordinates with HGMDS to provide a
global versioning file system
- Low latency metadata I/O
- Eventually consistency model
- Load balancing/fault tolerance
• On LAN:
- Distributed version management
- Load balancing/fault tolerance
FP3C meeting – Bordeaux, 2-3 September 2011 - 17

18. 4
Preliminary evaluation
BlobSeer-WAN on G5K
FP3C meeting – Bordeaux, 2-3 September 2011 - 18

19. Testbed
Using 2 sites of G5K
• Rennes: 40 nodes
• 30 nodes reserved for BlobSeer services
• 10 nodes for clients
• Grenoble: 40 nodes
• 30 nodes reserved for BlobSeer services
• 10 nodes for clients
• Interconnect network between sites 10 Gbps
FP3C meeting – Bordeaux, 2-3 September 2011 - 19

20. Concurrent appending: 512 MB/client
FP3C meeting – Bordeaux, 2-3 September 2011 - 20

21. 5
Conclusion
On going work
FP3C meeting – Bordeaux, 2-3 September 2011 - 21

22. Summary
Discussed the integration of BlobSeer and HGMDS:
• BlobSeer-WAN extension is required
BlobSeer-WAN
• Preliminary results look encouraging
• Performance of BlobSeer-WAN on two sites similar to that of
vanilla BlobSeer on a single site
• Prototype available at BlobSeer’s repository/branches/
BlobSeer-WAN-dev/
HGMDS
• Implementation almost done
• Works on multi-sites
• Collisions automatically solved by a rule
FP3C meeting – Bordeaux, 2-3 September 2011 - 22

23. Next steps
• A more extensive evaluation for BlobSeer-WAN
• Integrate BlobSeer-WAN to HGMDS
• Preliminary evaluation of HGMDS BlobSeer-WAN on
Grid5000 and on the Japanese Clusters
• Submit co-authored paper by Spring 2012
• Next internships: Kohei @Inria Rennes
FP3C meeting – Bordeaux, 2-3 September 2011 - 23

24. Thank you!
FP3C meeting
2 – 3 September 2011