Slides for LinuxCon / Open Cloud Japan 2013

1. OCDET Activity and Glusterfs
May 31, 2013
Gluster Workshop @ LinuxCon / OpenCloud Japan 2013
Copyright (c) OCDET All Rights Reserved.
May 31, 2013
OCDET / NTT DATA CORPORATION
Masanori Itoh

2. Who I am
Masanori Itoh
– Professional Affiliation
• NTT DATA CORPORATION
– A Japanese System Integrator (NOT a Telco ! ☺)
– Activities
• OCDET (Steering Committee) Today, I’m on behalf of
Copyright (c) OCDET All Rights Reserved. 1
• OCDET (Steering Committee)
• Japan OpenStack Users Group
• etc.
– Speciality
• Operating System
– Filesystem, Networking, …
• Trouble Shooting … :o
Today, I’m on behalf of
OCDET !

3. Agenda
1. What is OCDET ?
2. OSS IaaS with Glusterfs
3. Future Expectations and Ideas for Glusterfs
4. Summary
Copyright (c) OCDET All Rights Reserved. 2

4. 1. What is OCDET ?1. What is OCDET ?
3

5. What is OCDET?
Home Page : http://ocdet.org/ (Sorry, No English page… )
Copyright (c) OCDET All Rights Reserved. 4

6. What is OCDET?
OCDET stands for :
Open Cloud Demonstration Experiment Taskforce
オープン クラウド 実証 実験 タスクフォース
BTW, “Demonstration Experiment” ?
Copyright (c) OCDET All Rights Reserved. 5
BTW, “Demonstration Experiment” ?
– In Japanese,
• “実証実験” (Jissho-Jikken)
– In Short,
• An Activity on POC : Proof-of-Concept (of Something)

7. What is OCDET? : the Goal
the Goal
To Promote “Open Source Cloud Computing Components”
• From the Lowest Layer to Upper Layer things.
• Even including Hardware (!?)
Prove that we can Build Mature Infrastructure Using Them
By doing various POC activities
Copyright (c) OCDET All Rights Reserved. 6
By doing various POC activities
Under Fully Volunteered and Non-Profit Organization
And Share the Results Publicly
• Documentation
– Installation/Configuration Procedures, System Design
Guidelines, Knowhows , … etc.
• Tools, Sample Configuration Files, …

8. What is OCDET? : Overview
Formed on:
Feb. 3, 2012
Participants:
24 Organization Members
Founding Organizations : Hokkaido University,
NTT Communications, Cloud Business Association, CUPA
Copyright (c) OCDET All Rights Reserved. 7
NTT Communications, Cloud Business Association, CUPA
68 Active Volunteer Members (May 2013)
Organization
Steering Committee
Study Groups / Working Groups

9. What is OCDET? : Press Coverage
Study Groups / Working Groups
http://ocdet.org/information.html#pageLink02
Copyright (c) OCDET All Rights Reserved. 8

10. What is OCDET? : Organization
Study Groups / Working Groups
http://ocdet.org/information.html#pageLink02
Copyright (c) OCDET All Rights Reserved. 9

11. What is OCDET? : Organization
Study Groups / Working Groups
Open IaaS Study Group
• OpenStack WG
• CloudStack WG
• System Management WG
Open Storage Study Group
Orchestration SIG
Copyright (c) OCDET All Rights Reserved. 10
Open Storage Study Group
• Swift, Glusterfs, etc.
Open Network Study Group
Open PaaS Study Group
• CloudFoundry <--> Japan CloudFoundry User Group
Open Facility Study Group
• OpenCompute WG <--> OCP Japan

12. 2. IaaS Software and Glusterfs2. IaaS Software and Glusterfs
11

13. IaaS Software and Glusterfs : IaaS at a Glance
Provide Infrastructure Level Resources on Demand
Compute Resource
CPU, Memory, …
Network Resource
L2 Network, IP address Management, Router, Firewall, Load
Balancer, VPN, …
Copyright (c) OCDET All Rights Reserved. 12
Balancer, VPN, …
Storage Resource
File Access Store
Block Access Store
Object Storage Store

14. IaaS Software and Glusterfs : Examples
Examples of Public IaaS Service
Amazon Web Services
RackSpace Cloud
HP Cloud Services
etc.
Copyright (c) OCDET All Rights Reserved. 13
Examples of Open Source Software for IaaS
OpenStack
CloudStack
Eucalyptus
OpenNebula
etc.

15. IaaS Software and Glusterfs : Typical Use Cases
VM Images Store (Not always Expected being Persistent)
Image Templates and Image Instances of Running VMs
Need to be Either of File or Block Access I/F
External(/Additional) Volumes
Typically, Expected as ‘Persistent Storage’ for Database
and to have Block Storage I/F (to the guest OS instances)
Copyright (c) OCDET All Rights Reserved. 14
and to have Block Storage I/F (to the guest OS instances)
VM Image/External Volumes Snapshot Store
(Tenant) Backup / Bulk Data Store
XXX as a backend of HDFS (gluster already has!)
Object Storage I/F Fits for this Purpose Best. (Swift API?)
etc.

16. Test Case 1 : OpenStack Configuration Example
Intranet
Internet /
Intranet
Zone #1
External Network
End User PC
Firewall
Amazon EC2 type Network Model
NW Gateway Node aggregates
tenant external traffic
(*1) Cloud Controller
API, Scheduler, RabbitMQ, RDBMS etc.
Courtesy of Internet Initiative Japan Inc.
Copyright (c) OCDET All Rights Reserved.
Quantum
agent
(L2/L3)
Cloud
Controller
Glance
(Swift)
VM
Image
Nova Compute
Quantum Agent
V
M
V
M
…
Internal Network (VLAN/GRE etc.）
Management/Storage Network
DB
(MySQL)
Nova Compute
Quantum Agent
V
M
V
M
ovs ovs
Management/Storage Network
NW Gateway
Hypervisors
gluser
gluser
gluster
. . .
gluster

17. Test Case 1 : VM Image Store w/OpenStack
Shared FS by
Gluster for VM
Images
volume: gvol1
V
M
V
M
V
M
V
M
VM Service Network
OpenStack Folsom + CentOS 6.3 (x86_64) + gluster 3.3
gluster
Copyright (c) OCDET All Rights Reserved.
Management
/mnt/sdb1 /mnt/sdb1 /mnt/sdb1
volume: gvol1
/mnt/gluster1 /mnt/gluster1 /mnt/gluster1
NFS mount
bricks

18. Test Case 2 : CloudStack Configuration Example
Intranet
Internet /
Intranet
Zone/Pod
External Network
End User PC
Traffic for
Virtual Routers
DNAT based Network Model
Per-tenant Virtual Router VM will
be created and it proxies tenant
traffic.
Courtesy of DataHotel Co., Ltd.
for Secondary
Stoage
Copyright (c) OCDET All Rights Reserved.
Mgmt.
Server.
NFS
VM
Image
Compute
V
M
…
DB
Compute
V
M
Compute
Virtual
Router
Compute
Virtual
Router
vnic0
vnic1
V
M
vnic0
vnic1
SS
VM
CP
VM
Management/Storage Network
Management/Storage Network
gluster

19. Test Case 2 : VM Image Store w/CloudStack
Shared FS by
Gluster for VM
Images
V
M
V
M
V
M
V
M
VM Service Network
CloudStack 4.0 + CentOS 6.4(x86_64) + gluster 3.3 (4nodes)
Copyright (c) OCDET All Rights Reserved.
Management / Storage Traffic
/mnt/sdb1 /mnt/sdb1 /mnt/sdb1
volume: gvol1 /mnt/gluster1 /mnt/gluster1 /mnt/gluster1
NFS mount
brick
. . .

20. Summary of our (current) POC Experiences
VM Image Store
Glusterfs is QUITE EASY to install/configure and WORKS
FAIRLY WELL
With both OpenStack and CloudStack
Without almost No Pitfalls except trivial configuration mistakes
Forgot to add iptables ACCEPT rule for NFS etc. :o
Copyright (c) OCDET All Rights Reserved. 19
Forgot to add iptables ACCEPT rule for NFS etc. :o
Bandwidth Bottleneck when Many VMs launched at once
etc.
Object Storage
TBD
Volume Backend
TBD

21. OCDET (Possible) Future Plan
Large Scale Evaluation Using STARBED
Reserved 140 nodes (Now, Swift evaluation is ongoing)
Gluster Swift API Evaluation
OpenStack BlockStorage (Cinder) driver Evaluation
InfiniBand Support Evaluation (?)
Copyright (c) OCDET All Rights Reserved. 20
InfiniBand Support Evaluation (?)
etc.

22. 3. Future Expectations and (Stupid?) Ideas3. Future Expectations and (Stupid?) Ideas
for Glusterfs
21

23. Future Expectations and (Stupid?) Ideas
Enhancements of OpenStack Cinder Backend Driver
Add Volume Snapshot Feature
Snapshot Support (versioning?)
Some Handoffs with Local Filesystems? (e.g. btrfs?)
libvirt support of qemu-gluster integration
Copyright (c) OCDET All Rights Reserved. 22
libvirt support of qemu-gluster integration
Improvement of InfiniBand Support Quality
I saw a gluster-rdma system hung up under heavy I/O
load (using iozone) using gluster 3.3beta2 formerly…
Linux Upstream Code Merge(?)

24. Future Expectations and (Stupid?) Ideas
Erasure Coding Support (Like the case of HDFS) ?
To Reduce Required Physical Capacity without Losing
Redundancy Level
To Reduce Storage I/O Traffic
Replica Locality Control ??
Copyright (c) OCDET All Rights Reserved. 23
Replica Locality Control ??
To Reduce Replication Traffic
NFS/RDMA on top of Glustefs ?
Geologically Distributed “Synchronous” Replication?
For DR. In this sense, Single Master makes sense …

25. Natural Expectations…
http://office.microsoft.com/en-
us/images/similar.aspx#ai:MP900442518|
Copyright (c) OCDET All Rights Reserved. 24

26. 4. Summary
25

27. Conclusion and Summary
Glusterfs is a Fairly Good Solution for IaaS
Deployments ☺
OCDET is Further Evaluating Glusterfs
and Sharing our Results Publicly
Join us ! and Let’s Play with Glusterfs together !☺
Copyright (c) OCDET All Rights Reserved. 26
Join us ! and Let’s Play with Glusterfs together !☺

28. Thank You !
Copyright (c) OCDET All Rights Reserved. 27
Thank You !