The document discusses providing easy access to OpenStack object storage through proxy servers like ftp-cloudfs and sftp-cloudfs. These open source projects allow users to access OpenStack storage via FTP and SFTP protocols, abstracting the underlying REST API. They have gained popularity with over 1,000 monthly downloads. The proxies help customers use OpenStack storage like other services by providing familiar access methods and tools to move data in and out while scaling easily on low-cost VMs.

1. Easy Access to OpenStack Object Storage
Juan J. Martínez
@reidrac
http://www.usebox.net/jjm/
Merry OpenStack, London
Dec 2013

2. What is Swift?
Open Source software for creating redundant, scalable object storage using
clusters of standardized servers to store petabytes of accessible data.
Object Storage
●
●
●
●
Not a real file system
RESTful HTTP API
Blob and metadata (name,
content type, date-time, size,
MD5)
Support for pseudo-hierarchical
directories
Real File System
●
●
●
●
●
open, read, write, seek, close
chdir, mkdir, rmdir
remove, rename
chmod, chown, stat, lock
etc

3. CUSTOMERS

4. What we have

5. What we sell

6. What customers want

7. Solution: Proxies

8. Back in 2011...
●
●
●
Building a product based on OpenStack Object Storage
Started working with Bexar
Went live with Diablo
●
Not many end-user tools using OpenStack Object Storage API
(actually, it was Rackspace Cloud Files™)
●
●
●
Customers won’t start building things straight away
Internal tools WIP (snapshot VM, re-image from snapshot)
It was hard to explain the product*:
What is Amazon S3? I want Dropbox!
* disclaimer: I’m a developer

9. Proxies: FTP
ftp-cloudfs
●
●
●
●
Personal project of Chmouel Boudjnah (now @ https://github.com/cloudfs/ftp-cloudfs)
Open Source
Very cool, “almost working” (Jan 2011)
fs.py abstraction:
Swift API -> FTP
Memset started contributing to make it “production ready”:
●
●
●
●
●
Support for OpenStack Object Storage (was Cloud Files only)
Missing bits in fs.py, more tests, introduced some new bugs
Fixed performance problems: memcache all the things!
Finally migrated from python-cloudfiles to python-swiftclient (kept the name, it’s cool)
Keystone auth, resume downloads, listing +10k files, large file support (>5GB)

10. Proxies: SFTP
sftp-cloudfs
●
●
●
●
●
●
●
Open Source sponsored by Memset Hosting Ltd
https://github.com/Memset/sftpcloudfs
ftp-cloudfs’ fs.py + paramiko (ssh2 protocol for Python)
Secure: client -> SSH -> sftp-cloudfs -> SSL -> swift
Not as performant as FTP (1.6 MB/s vs pyftpdlib 4.2 MB/s), CPU intensive
Only password based authentication
Partial SCP support contributed by Koert van der Veer
+ everything supported by ftp-cloudfs

11. Bigger than the API
●
●
●
●
●
●
+2 TB / mo traffic (in > out)
Small VMs running ftp-cloudfs, sftp-cloudfs and shared cache (memcache)
Load-balanced with LVS
Easy to scale (add more VMs!)
Use cases: backups, CMS plugins (CDN support), remote storage, ???
FTP > SFTP > API
Useful Open Source projects:
●
●
●
Numbers from PyPI
ftp-cloudfs: 1895 downloads last month
sftp-cloudfs: 1107 downloads last month
Patches from several contributors (including hosting companies)
“Stable” and almost feature complete, a new release every now and then

12. Easy Access to OpenStack Object Storage
Questions?
Juan J. Martínez
@reidrac
http://www.usebox.net/jjm/
Merry OpenStack, London
Dec 2013