YALMS Basics: A Group Drive API for Cloud-Based Classrooms

.

.

YALMS Basics

M.Zhanikeev, H.Koide -- maratishe@gmail.com --

YALMS: A Group Drive API for Cloud-Based Classrooms -- http://tinyurl.com/kyutech131121 ---

2 /26
2/26

.

.

YALMS Basics
.
YALMS stands for ...
.
nother ecture
. et

Y A
•

L

Management System

next generation digital classroom 02

robots

• several
: video annotation, content packing, multimedia storage, etc.
• can run in clouds and cloud-based classrooms
◦ specifically, in VM-per-student environments
• knowledge extraction and social components

02 "Homepage of the YALMS software presented in this paper" https://github.com/maratishe/yalms (2013)


3 /26
3/26

.

.

YALMS : The Big Picture
2013.07・ET研

YALMS: Intro

2013.11・IN研

YALMS.cloud
VM sync groupdrive
content
2013.09・ET研

YALMS.annotation
PDF

video

2014.01・LOIS研

slices

YALMS.format

2014.01・ET研

stream record
replay storage

YALMS.streaming
DASH Adaptive

HTML5

2013.10・ITE-MMS研

YALMS.hiperf
SSD
multicore

YALMS.done!


4 /26
4/26

.

.

YALMS.groupdrive : The Specs



5 /26
5/26

.

.

YALMS.groupdrive : Overall Idea
Master
Copy

Slave
Copy


Slave
Copy

Slave
Copy


6 /26
6/26

.

.

YALMS.groupdrive : The Specs
1.

realtime async. asymm. P2P sync -- but not
distributed storage

fully distributed, can work without a master
3. smart and adaptive, fit for any environment
2.

◦ remote master, fully distributed, throughput-limited, clouds, etc.

4. efficient with
5. API with

large files -- a sub-file data grain

cross-platform implementation

◦ Linux, Windows, VMs



7 /26
7/26

.

.

YALMS.groupdrive : Existing (bad)
Solutions



8 /26
8/26

.

.

Bad Solution : BitTorrent
bad solution
needs central registry -- Tracker
good only with many peers, too much overhead with few
◦ rarest-first policy is not fit for the problem
good for download, but what about upload?

• a very
•
•
•



9 /26
9/26

.

.

Bad Solution : DSync
• a paper in 05 -- the closest rival
• aggregation from

multiple sources while optimizing load

• not really distributed -•

single point prospective -- o2m or o2o

no practical considerations -- no measurement, ordered lists, etc.

05 H.Pucha+3 "Adaptive File Transfers for Diverse Environments" USENIX Annual Technical Conference (2008)


10 /26
10/26

.

.

Bad Solution : P2P Streaming
…
Share

video
streaming 04

• good for
06

• again, bad for

bi-directional

Content
Provider
(origin)

syncs
P2P Network

04 myself+0 "Multi-Source Stream Aggregation in the Cloud" Wiley Book on Content Delivery (ACDN) (2013)
06 myself+0 "Extremely Scalable and Low Demand Live P2P Streaming based on Variable Bitrate" CANDAR (2013)


11 /26
11/26

.

.

Bad Solution : Group Virtualization
Service
traﬃc

Service

Application

• specific to

remote
services 07

Service
Provider

Virtual
Wireless
User

Wireless
User

Wireless
User

Wireless
User

• does not work

fully
local syncs
not specific to
well with

•

syncs

07 myself "Virtual Wireless User: A Practical Design for Parallel MultiConnect Using WiFi Direct..." Mobiquitous (2013)


12 /26
12/26

.

.

Proposal and Design



13 /26
13/26

.

.

Proposal : Fundamentals
.
Objective
.
+
.

API Implementation of the proposed group drive
• standard

RESTful API

• implemented in PHP
• cross-platform

e2e throughput -- response time
avoid congesting the master

• adaptive to
•



14 /26
14/26

.

.

Design : PUSH/PULL Basics
• simple: 1 PULL and

.
The PULL
.
• regularly

X PUSH rounds, X peers in each round -- X is scale

ask peers for changes, new content comes in replies

• do not push the newly recieved content
.
.
The PUSH
.
• push to
for quick distribution
.
.
Conflicts
.
... ignored for now, treat like in GoogleDrive, etc.

X peers in X rounds



15 /26
15/26

.

.

Design : The MFFxR Unit
• basic
•
•

list unit for PUSHes and PULLs -- ordered by decreasing

throughput
Random Peer is basically for measurement/discovery
scale is a global parameter

Self

List:M


List:MF

List:MFF

List:MFF+R


16 /26
16/26

.

.

Design : Peer Discovery Roadmap
• dynamically incorporate

new peers
Master
REP( peers)

REQ()


Update
peers

Done


17 /26
17/26

.

.

Design : PULL Roadmap
• a

roadmap rather than state machine
Error,
failure,
timeout

REP( null)
REP( block, epoch)

REQ( state)
Scale (s)

Align
peers

MFFF…
+R

Notify

Replace

(if new block)

Done


18 /26
18/26

.

.

Design : PUSH Roadmap
REP( OK)
REP( Collision)

REQ(block)
Scale (s)

Align
peers

MFFF…
+R

Pick next
from MFFF…
for 2nd round

I already
have it!

(double up)

Done

Notify

If M or R


19 /26
19/26

.

.

Experiments



20 /26
20/26

.

.

Experiments
Traditional (10 clients)
Traditional (1 client)
YALMS.groupdrive Remote Master (10 peers and scale 2)
YALMS.groupdrive Local Only (10 peers and scale 2)
1

•

has the file
• try various

environments

0.8
Peer completion ratio

10Gb file
initial state: only master

• one

0.6

0.4

0.2

0
0

0.5

1

1.5

2
2.5
log( time)

3

3.5


4

4.5

21 /26
21/26

.

.

Implementation and API



22 /26
22/26

.

.

Implementation : Basic Unit of Info.
Exchange
• standard unit of data exchange -- JSON -- part of API 03
• some parts are not used -- blob only used for PUSHes/PULLs
Datatype

Parameter
Comments
name

string relpath

Relative to top folder

string filename
long blockpos
int blocksize
double mtime
string abspath
blob content

Position of the head byte of the block in file
Fixed for each session
Epoch time of last update for that block at Master
Path for this block at Master. Not used by Slave Copies.
Blob: binary string. Used when needed.

03 "Homepage of the YALMS.groupdrive project" https://github.com/maratishe/yalms.groupdrive (2013)


23 /26
23/26

.

.

Implementation : Actual API
MFFxR is outside of the API (part of implementation)
state is a flat JSON of items (keys) and MD5 digests (values) 03
APIs are standard RESTfull HTTP request/reply routines

• per-peer,
•
•

API1: pull(state) − −reply(dataunit)
• API2: push(dataunit) − −reply(status)
•



24 /26
24/26

.

.

Implementation : Software Design
Linux but will work on Windows using portable TinyWeb 03
Web
Server

Web
Server

OS, System

Reps

Reqs

OS, System

• prefers

wget

wget

Reps

Reqs



25 /26
25/26

.

.

That’s all, thank you ...



26 /26
26/26

.

.

[01]

(2013)
The enPiT Project
http://www.enpit.jp

[02]

(2013)
Homepage of the YALMS software presented in this paper
https://github.com/maratishe/yalms

[03]

(2013)
Homepage of the YALMS.groupdrive project
https://github.com/maratishe/yalms.groupdrive

[04] myself+0 (2013)
Multi-Source Stream Aggregation in the Cloud
Wiley Book on Content Delivery (ACDN)
[05] H.Pucha+3 (2008)
Adaptive File Transfers for Diverse Environments
USENIX Annual Technical Conference


26 /26
26/26

.

.

[06] myself+0 (2013)
Extremely Scalable and Low Demand Live P2P Streaming based on Variable Bitrate
CANDAR
[07] myself (2013)
Virtual Wireless User: A Practical Design for Parallel MultiConnect Using WiFi
Direct...
Mobiquitous
[08]

(2013)
Dropbox Core API
https://www.dropbox.com/developers/core



26 /26
26/26

YALMS Basics: A Group Drive API for Cloud-Based Classrooms

Recomendados

Recomendados

Más contenido relacionado

Destacado

Destacado (8)

Similar a YALMS Basics: A Group Drive API for Cloud-Based Classrooms

Similar a YALMS Basics: A Group Drive API for Cloud-Based Classrooms (20)

Más de Tokyo University of Science

Más de Tokyo University of Science (20)

Último

Último (20)

YALMS Basics: A Group Drive API for Cloud-Based Classrooms