SlideShare una empresa de Scribd logo
1 de 51
Descargar para leer sin conexión
Beolink.org
RestFS Internals
Fabrizio Manfredi Furuholmen
Federico Mosca
Beolink.org
Europython 2013
2
Agenda
 Introduction
 Goals
 Principals
 RestFS
 Architecture
 Internals
 Sub project
 Conclusion
 Developments
Beolink.orgIntroduction
3
Europython 2013
Zetabyte
10007 bytes
1021 bytes
1,000,000,000,000,000,000,000 bytes
All of the data on Earth today 150GB of data per person
2% of the data on Earth in 2020
Beolink.org
4
GOAL 1/2
Europython 2013
Create a free available Cloud Storage Software
Beolink.org
5
GOAL 2/2
Create a
framework for
testing a new
technologies and
paradigm
Europython 2013
Beolink.orgPrinciple 1/3
6
“Moving Computation
is
Cheaper than Moving Data”
Europython 2013
Beolink.orgPrinciple 2/3
7
“There is always a failure waiting
around the corner”
Europython 2013
*Werner Vogel
Beolink.orgPrinciple 3/3
8
“Decompose into small loosely
coupled, stateless building
blocks”
Europython 2013
*’ Leaving a Legacy System Revisited’ Chad Fowler
Beolink.org
9
RestFS
Europython 2013
Beolink.org
10
RestFS Key Words
RestFS
Cell
collection of servers
Bucket
virtual container,
hosted by one or
more server
Object
entity (file, dir, …)
contained in a
Bucket
Europython 2013
Beolink.org
11
RestFS Components
Europython 2013
Cell
Bucket N
Bucket X
Objects
Objects
Bucket/Cell Cells Object
S3:
bucket_name.mydomain.com/object_name
RestFS:
bucket_name.mydomain.com
Rpc :
bucket_name
object_name
object
Bucket
Cell
Beolink.orgFive main areas
12
Objects
• Separation btw data and
metadata
• Each element is marked
with a revision
• Each element is marked
with an hash.
Cache
• Client side
• Callback/Notify
• Persistent
Transmission
• Parallel operation
• Http like protocol
• Compression
• Transfer by difference
Distribution
• Resource discovery by
DNS
• Data spread on multi
node cluster
• Decentralize
• Independents cluster
• Data Replication
Security
• Secure connection
• Encryption client side,
• Extend ACL
• Delegation/Federation
• Admin Delegation
Europython 2013
Beolink.orgBucket Discovery
13
Client
DNS
Lookup
Cell 1
Cell 2
N server
N server
Bucket name Cell RL
IP list
Bucket name
Server list +
Load info
Server Priority Type
IP 1
.. …
Server list
priority List
Europython 2013
Beolink.orgObject
14
Data Metadata
Segments
Object
Attributes
set by user
Europython 2013
Properties
ACL
Ext Properties
Block 1
Block 2
Block n
Block …
HashHashHashHash
SerialSerialSerialSerialSerial
Beolink.orgCell Interaction
15
Client
Metadata
Client Cache
Europython 2013
Cell
Resource
Locator
Beolink.orgCentral Service for Domain
16
Client
DNS
Lookup
SrvCell
Create
Bucket
Cell
Service
Service name Cell RL
IP list
Server Priority Type
IP 1
.. …
Server list
priority List
Europython 2013
Beolink.org
17
Cache client side
DNS
RestFS Metadata
RestFS Block
Federated Auth
Callbacks
Metadata
cache
Block
cache
RestFS Block
RestFS Block
Persistent
Cache Resource Locator
Europython 2013
ServerList
Tokens
Pub/Sub
List
Temporary
Locks
Beolink.org
18
Server Architecture
S3
Service
Storage
Mgr
Auth
Manager
Meta
Mgr
Storage
Driver
Token
Driver
RestFS
RPC
Resource
Manager
Distributed Cache
Callbacks
Manager
Meta
Driver
Auth
Driver
Callbacks
Driver
Auth
InterfaceManagersDrivers
Plugin
Resource
Locator
Backends
Europython 2013
Token Sub/Pu
b
Token
Manager
Resource
Driver
MetaService
RLService
CallbackService
AuthService
TokenService
BlockService
Locks
Mgr
Locks
Driver
LocksService
Beolink.org
19
Backends
Europython 2013
Service
• DNS
• SQL
Auth
• SQL
Token
• NoSQL
• Distributed Cache
RL
• SQL
• Memory
Meta
• NoSQL
• Distributed Cache
CallBack
• NoSQL
Locks
• NoSQL
• Distributed Cache
MultiqueryVsSingleQuery
- Dedicated storage
infrastructure per
Service
- Distributed Memory
cache
- One or more DB per
Driver
Beolink.org
20
Bucket
Europython 2013
Beolink.org
21
Bucket
Europython 2013
 Cell Ips
The bucket is stored in the DNS for
lookup, the ip address returned by DNS
are the Cell RL address
 Property
The property element is a collection of object information, with
this element you can retrieve the default value for the bucket
(logging level, security level, ect).
- Property
- Property Ext
- Property ACL
- Property Stats
Object Serialized
Backends agnostic on information stored
Bucket Name
zebra
Property
segment_size= 512
block_size = 16k
max_read’=1000
storage_class=STANDARD
compression= none
…
Beolink.org
22
Bucket Type
Europython 2013
Filesystm
The bucket is used as a filesystem
Logging
Logging operation done on the specific
Bucket
Replica RO
Bucket shadow replication
…
Custom definition
Beolink.org
23
Objects
Europython 2013
Beolink.org
24
Object
Europython 2013
Object Property
Object Property Ext
Object Stats
Object ACL
Segments
Object
zebra.c1d2197420bd41ef24fc665f228e2c76e98da247
Segment-id
1:16db0420c9cc29a9d89ff89cd191bd2045e47378
2:9bcf720b1d5aa9b78eb1bcdbf3d14c353517986c
3:158aa47df63f79fd5bc227d32d52a97e1451828c
4:1ee794c0785c7991f986afc199a6eee1fa4
5:c3c662928ac93e206e025a1b08b14ad02e77b29d
…
vers:1335519328.091779
Property
segment_size= 512
block_size = 16k
content_type =
md5=ab86d732d11beb65ed0183d6a87b9b0
max_read’=1000
storage_class=STANDARD
compression= none
…
Beolink.org
25
Object Type
Europython 2013
Data
Contains files
Folder
Special object that contain others
objects
Mount point
Contains the name of the buckets
Link
Contains the name of the objects
Immutable
Gold image
Custom
Defined by the users
Cell
Bucket N
Objects
Cell
Bucket N
Objects
Beolink.org
26
Object Properties
Europython 2013
Key Value Pair
Key for everything
- Metadata: BUCKET_NAME.UUID
- Block: BUCKET_NAME.UUID
Serial
Each element has a version which is identified by a serial.
Object Serialized
Backends agnostic on information stored
Default Root Object
For each bucket is defined a default root object with object id
ROOT
Nosql Storage
Key: serialized object
Object
zebra.c1d2197420bd41ef24fc665f228e2c76e98da247
Segment-id
1:16db0420c9cc29a9d89ff89cd191bd2045e47378
2:9bcf720b1d5aa9b78eb1bcdbf3d14c353517986c
3:158aa47df63f79fd5bc227d32d52a97e1451828c
4:1ee794c0785c7991f986afc199a6eee1fa4
5:c3c662928ac93e206e025a1b08b14ad02e77b29d
…
vers:1335519328.091779
Property
segment_size= 512
block_size = 16k
content_type =
md5=ab86d732d11beb65ed0183d6a87b9b0
max_read’=1000
storage_class=STANDARD
compression= none
…
Beolink.org
27
Segments
Europython 2013
Beolink.orgSegments
28
Segment 1
Europython 2013
Pos 1 : Hash
Serial
Pos 2 : Hash
Pos 3 : Hash
Pos n : Hash
Segment 1
Pos 1 : Hash
Serial
Pos 2 : Hash
Pos 3 : Hash
Pos n : Hash
Segment n
Pos 1 : Hash
Serial
Pos 2 : Hash
Pos 3 : Hash
Pos n : Hash
N Segment = (Object Size/block size)/segment size
Beolink.orgSegments
29
“Serial vs Hash”
Europython 2013
Beolink.org
30
Object Versioning
Europython 2013
Cell
Bucket N
Objects
Objects
Objects
Object Pointer
Object point to the previous one
Only CreateBlock
Client has to use only createBlock
operation
New ID for the old Object
Segment Difference
Beolink.org
31
Protocols
Europython 2013
Beolink.orgProtocols
32
 DNS
 RestFS
 S3 Interface
Europython 2013
Beolink.org
33
RestFS Protocol
WebSocket
is a web technology for multiplexing bi-directional, full-
duplex communications channels over a single TCP
connection.
This is made possible by providing a standardized way
for the server to send content to the browser without
being solicited by the client, and allowing for messages
to be passed back and forth while keeping the
connection open…
JSON-RPC
is lightweight remote procedure call protocol similar to
XML-RPC. It's designed to be simple
BSON
short for Binary JSON,
is a binary-encoded serialization of JSON-like
documents. Like JSON, BSON supports the embedding
of documents and arrays within other documents and
arrays.
BSON can be compared to binary interchange formats
Only Primitives
No objects or list
{"hello": "world"}
→
"x16x00x00x00x02hellox00
x06x00x00x00worldx00x00"
Europython 2013
--> { "method": ”readBlock", "params": [”…"], "id": 1}
<-- { "result": [..], "error": null, "id": 1}
GET /mychat HTTP/1.1
Host: server.example.com
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: x3JJHMbDL1EzLkh9GBhXDw==
Sec-WebSocket-Protocol: chat
Sec-WebSocket-Version: 13
Origin: http://example.com
HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: HSmrc0sMlYUkAGmm5OPpG2HaGWk=
Sec-WebSocket-Protocol: chat
Beolink.org
34
RestFS Protocol
Meta Operation
- Bucket
- Object id
- Operation
- List elements
Operation Packets
Collect operation to single segment
Parallel
 Single channel to meta data
server
 Parallel channel to block
storage (one per segment)
Europython 2013
Beolink.org
35
Locks
Europython 2013
Beolink.org
36
Locks vs No consistency
Europython 2013
Beolink.org
37
Locks
Europython 2013
OpLocks
Time base
Token base
Ordering
Conflict are management with the serial
property
Beolink.org
38
Cache
Europython 2013
Beolink.org
39
Cache
Publish–subscribe
“… is a messaging pattern where senders of messages,
called publishers, do not program the messages to be sent
directly to specific receivers, called subscribers. Published
messages are characterized into classes, without
knowledge of what, if any, subscribers there may be.
Subscribers express interest in one or more classes, and
only receive messages that are of interest, without
knowledge of what, if any, publishers there are… “
Wikipedia
Pattern matching
Clients may subscribe to glob-style patterns in order to
receive all the messages sent to channel names matching
a given pattern.
Distributed Cache Server Side
For server side the server share information over
distributed cache to reduce the use of backend
Client Cache
Pre allocated block with circular cache
write-through cache
Europython 2013
Demo http://www.websocket.org/echo.html
Use case A: 1,000
Use case B: 10,000
Use case C: 100,000
Beolink.org
40
Block Storage
Europython 2013
Beolink.org
41
Backend: Storage
Kademlia's XOR distance is easier to calculate.
Kademlia's routing tables makes routing table
management a bit easier.
Each node in the network keeps contact information
for only log n other nodes
Kademlia implements a "least recently seen"
eviction policy, removing contacts that have not
been heard from for the longest period of time.
Key/value pair is stored on the node whose 160-bit
nodeID is closest to the key
Closest node, send a copy to neighbor
Europython 2013
Beolink.org
42
Code
Europython 2013
Beolink.org
43
Pluggable
Europython 2013
Protocol
• Connection Handler
• Data transcoding
Service
• High level Operations across multiple functions
(like locking)
• Integrity operations/transaction
Manager
• Operations handler for specific area (ex.
metadata)
• Split info in sub info
Driver
• Read and write operation to storage system,
agnostic operation
Beolink.org
44
NoSQL as much as Possible
Key Value
- Key in memory
- Value on disc
Example of benchmark result
The test was done with 50 simultaneous clients
performing 100000 requests.
The value SET and GET is a 256 bytes string.
The Linux box is running Linux 2.6, it's Xeon
X3320 2.5 GHz.
Text executed using the loopback interface
(127.0.0.1).
Connections
Transaction
Cluster
Multi-master
Auto recovery
Europython 2013
Beolink.org
45
What we are using
Module Software
Storage Filesystem, DHT (kademlia, Pastry*)
Metadata SQL(mysql,sqlite), Nosql (Redis)
Auth Oauth(google, twitter, facebook), kerberos*,
internal
Protocol Websocket
Message
Format
JSON-RPC 2.0, Amazon S3
Encoding Plain, bson
CallBack Subscribe/Publish Websocket/Redis, Async I/O
TornadoWeb, ZeroMQ*
HASH Sha-XXX, MD5-XXX, AES
Encryptio
n
SSL, ciphers supported by crypto++
Discovery DNS, file base* are planned
Europython 2013
Beolink.orgWhat is it good for ?
46
User
• Home directory
• Remote/Internet disks
Application
• Object storage
• Shared space
• Virtual Machine
Distribution
• CDN (Multimedia)
• Data replication
• Disaster Recovery
Europython 2013
Beolink.orgAdvantages
47
High
reliability
Distributed
Decentralized
Data replication
Nearly
unlimited
scalability
Horizontal scalability
Multi tier scalability
Cost-
efficient
Cheap HW
Optimized resource usage
Flexible User Property
Extended values and info
Enhanced
security
Extended ACL
OAUTH / Federation
Encryption
Token for single device
Simple to
Extend
Plugin
Bricks
Europython 2013
Beolink.orgRoadmap
48
 0.1
Single server on storage (No DHT)
S3 Interface
Federated Authentication
 0.2 Release (coming soon)
DHT on storage
Storage Encryption and compression
FUSE
 0.3 Release TBD (codename WorstFS++)
Deduplication
pub/sub
ACL
…
 Next
Disconnected operation, Logging, Locks, Dlocks,
Bucket automate provisioning, Distribution algorithms, Load balancing,
samba module, more async i/o, block replication control, negative cache, index, user
defined index
Europython 2013
Beolink.orgSupport
49
Europython 2013
Beolink.org
Thank you
http://restfs.beolink.org
manfred.furuholmen@gmail.com
fege85@gmail.com
Beolink.org
51
Backend: Storage
Transport Layer ZeroMQ
Storage Compressed DAta
Europython 2013

Más contenido relacionado

La actualidad más candente

Finding target for hacking on internet is now easier
Finding target for hacking on internet is now easierFinding target for hacking on internet is now easier
Finding target for hacking on internet is now easierDavid Thomas
 
OpenStack networking
OpenStack networkingOpenStack networking
OpenStack networkingSim Janghoon
 
Yeti DNS - Experimenting at the root
Yeti DNS - Experimenting at the rootYeti DNS - Experimenting at the root
Yeti DNS - Experimenting at the rootMen and Mice
 
MySQL async message subscription platform
MySQL async message subscription platformMySQL async message subscription platform
MySQL async message subscription platformLouis liu
 
Part 2 - Local Name Resolution in Windows Networks
Part 2 - Local Name Resolution in Windows NetworksPart 2 - Local Name Resolution in Windows Networks
Part 2 - Local Name Resolution in Windows NetworksMen and Mice
 
How to upgrade to MongoDB 4.0 - Percona Europe 2018
How to upgrade to MongoDB 4.0 - Percona Europe 2018How to upgrade to MongoDB 4.0 - Percona Europe 2018
How to upgrade to MongoDB 4.0 - Percona Europe 2018Antonios Giannopoulos
 
DNS High-Availability Tools - Open-Source Load Balancing Solutions
DNS High-Availability Tools - Open-Source Load Balancing SolutionsDNS High-Availability Tools - Open-Source Load Balancing Solutions
DNS High-Availability Tools - Open-Source Load Balancing SolutionsMen and Mice
 
RIPE 71 and IETF 94 reports webinar
RIPE 71 and IETF 94 reports webinarRIPE 71 and IETF 94 reports webinar
RIPE 71 and IETF 94 reports webinarMen and Mice
 
Managing data and operation distribution in MongoDB
Managing data and operation distribution in MongoDBManaging data and operation distribution in MongoDB
Managing data and operation distribution in MongoDBAntonios Giannopoulos
 
BIND 9 logging best practices
BIND 9 logging best practicesBIND 9 logging best practices
BIND 9 logging best practicesMen and Mice
 
Troubleshooting containerized triple o deployment
Troubleshooting containerized triple o deploymentTroubleshooting containerized triple o deployment
Troubleshooting containerized triple o deploymentSadique Puthen
 
青云CoreOS虚拟机部署kubernetes
青云CoreOS虚拟机部署kubernetes 青云CoreOS虚拟机部署kubernetes
青云CoreOS虚拟机部署kubernetes Zhichao Liang
 
Designate - Operators Deep Dive
Designate - Operators Deep DiveDesignate - Operators Deep Dive
Designate - Operators Deep DiveGraham Hayes
 
The DNSSEC KSK of the root rolls
The DNSSEC KSK of the root rollsThe DNSSEC KSK of the root rolls
The DNSSEC KSK of the root rollsMen and Mice
 
Software Packaging for Cross OS Distribution
Software Packaging for Cross OS DistributionSoftware Packaging for Cross OS Distribution
Software Packaging for Cross OS DistributionJian-Hong Pan
 
DNSSEC signing Tutorial
DNSSEC signing Tutorial DNSSEC signing Tutorial
DNSSEC signing Tutorial Men and Mice
 
Sharding in MongoDB 4.2 #what_is_new
 Sharding in MongoDB 4.2 #what_is_new Sharding in MongoDB 4.2 #what_is_new
Sharding in MongoDB 4.2 #what_is_newAntonios Giannopoulos
 
SANS Holiday Hack 2013 – Investigation Timeline
SANS Holiday Hack 2013 – Investigation TimelineSANS Holiday Hack 2013 – Investigation Timeline
SANS Holiday Hack 2013 – Investigation Timelinegiacomo83m
 
Python twisted
Python twistedPython twisted
Python twistedMahendra M
 

La actualidad más candente (20)

Finding target for hacking on internet is now easier
Finding target for hacking on internet is now easierFinding target for hacking on internet is now easier
Finding target for hacking on internet is now easier
 
OpenStack networking
OpenStack networkingOpenStack networking
OpenStack networking
 
Yeti DNS - Experimenting at the root
Yeti DNS - Experimenting at the rootYeti DNS - Experimenting at the root
Yeti DNS - Experimenting at the root
 
MySQL async message subscription platform
MySQL async message subscription platformMySQL async message subscription platform
MySQL async message subscription platform
 
Part 2 - Local Name Resolution in Windows Networks
Part 2 - Local Name Resolution in Windows NetworksPart 2 - Local Name Resolution in Windows Networks
Part 2 - Local Name Resolution in Windows Networks
 
How to upgrade to MongoDB 4.0 - Percona Europe 2018
How to upgrade to MongoDB 4.0 - Percona Europe 2018How to upgrade to MongoDB 4.0 - Percona Europe 2018
How to upgrade to MongoDB 4.0 - Percona Europe 2018
 
DNS High-Availability Tools - Open-Source Load Balancing Solutions
DNS High-Availability Tools - Open-Source Load Balancing SolutionsDNS High-Availability Tools - Open-Source Load Balancing Solutions
DNS High-Availability Tools - Open-Source Load Balancing Solutions
 
RIPE 71 and IETF 94 reports webinar
RIPE 71 and IETF 94 reports webinarRIPE 71 and IETF 94 reports webinar
RIPE 71 and IETF 94 reports webinar
 
Managing data and operation distribution in MongoDB
Managing data and operation distribution in MongoDBManaging data and operation distribution in MongoDB
Managing data and operation distribution in MongoDB
 
BIND 9 logging best practices
BIND 9 logging best practicesBIND 9 logging best practices
BIND 9 logging best practices
 
Troubleshooting containerized triple o deployment
Troubleshooting containerized triple o deploymentTroubleshooting containerized triple o deployment
Troubleshooting containerized triple o deployment
 
青云CoreOS虚拟机部署kubernetes
青云CoreOS虚拟机部署kubernetes 青云CoreOS虚拟机部署kubernetes
青云CoreOS虚拟机部署kubernetes
 
Designate - Operators Deep Dive
Designate - Operators Deep DiveDesignate - Operators Deep Dive
Designate - Operators Deep Dive
 
DNSTap Webinar
DNSTap WebinarDNSTap Webinar
DNSTap Webinar
 
The DNSSEC KSK of the root rolls
The DNSSEC KSK of the root rollsThe DNSSEC KSK of the root rolls
The DNSSEC KSK of the root rolls
 
Software Packaging for Cross OS Distribution
Software Packaging for Cross OS DistributionSoftware Packaging for Cross OS Distribution
Software Packaging for Cross OS Distribution
 
DNSSEC signing Tutorial
DNSSEC signing Tutorial DNSSEC signing Tutorial
DNSSEC signing Tutorial
 
Sharding in MongoDB 4.2 #what_is_new
 Sharding in MongoDB 4.2 #what_is_new Sharding in MongoDB 4.2 #what_is_new
Sharding in MongoDB 4.2 #what_is_new
 
SANS Holiday Hack 2013 – Investigation Timeline
SANS Holiday Hack 2013 – Investigation TimelineSANS Holiday Hack 2013 – Investigation Timeline
SANS Holiday Hack 2013 – Investigation Timeline
 
Python twisted
Python twistedPython twisted
Python twisted
 

Similar a Restfs internals

FPC for the Masses - CoRIIN 2018
FPC for the Masses - CoRIIN 2018FPC for the Masses - CoRIIN 2018
FPC for the Masses - CoRIIN 2018Xavier Mertens
 
Check the version with fixes. Link in description
Check the version with fixes. Link in descriptionCheck the version with fixes. Link in description
Check the version with fixes. Link in descriptionPrzemyslaw Koltermann
 
PASTE: A Network Programming Interface for Non-Volatile Main Memory
PASTE: A Network Programming Interface for Non-Volatile Main MemoryPASTE: A Network Programming Interface for Non-Volatile Main Memory
PASTE: A Network Programming Interface for Non-Volatile Main Memorymicchie
 
The post release technologies of Crysis 3 (Slides Only) - Stewart Needham
The post release technologies of Crysis 3 (Slides Only) - Stewart NeedhamThe post release technologies of Crysis 3 (Slides Only) - Stewart Needham
The post release technologies of Crysis 3 (Slides Only) - Stewart NeedhamStewart Needham
 
Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I s...
Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I s...Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I s...
Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I s...Anne Nicolas
 
Splunk app for stream
Splunk app for stream Splunk app for stream
Splunk app for stream csching
 
LISA18: Hidden Linux Metrics with Prometheus eBPF Exporter
LISA18: Hidden Linux Metrics with Prometheus eBPF ExporterLISA18: Hidden Linux Metrics with Prometheus eBPF Exporter
LISA18: Hidden Linux Metrics with Prometheus eBPF ExporterIvan Babrou
 
Oracle Basics and Architecture
Oracle Basics and ArchitectureOracle Basics and Architecture
Oracle Basics and ArchitectureSidney Chen
 
Elasticsearch on Kubernetes
Elasticsearch on KubernetesElasticsearch on Kubernetes
Elasticsearch on KubernetesJoerg Henning
 
Debugging linux issues with eBPF
Debugging linux issues with eBPFDebugging linux issues with eBPF
Debugging linux issues with eBPFIvan Babrou
 
Caching Methodology & Strategies
Caching Methodology & StrategiesCaching Methodology & Strategies
Caching Methodology & StrategiesTiệp Vũ
 
Caching methodology and strategies
Caching methodology and strategiesCaching methodology and strategies
Caching methodology and strategiesTiep Vu
 
Troubleshooting .net core on linux
Troubleshooting .net core on linuxTroubleshooting .net core on linux
Troubleshooting .net core on linuxPavel Klimiankou
 
OSCON 2011 Learning CouchDB
OSCON 2011 Learning CouchDBOSCON 2011 Learning CouchDB
OSCON 2011 Learning CouchDBBradley Holt
 
Java File I/O Performance Analysis - Part I - JCConf 2018
Java File I/O Performance Analysis - Part I - JCConf 2018Java File I/O Performance Analysis - Part I - JCConf 2018
Java File I/O Performance Analysis - Part I - JCConf 2018Michael Fong
 
A New Framework for Detection
A New Framework for DetectionA New Framework for Detection
A New Framework for DetectionSourcefire VRT
 
Sourcefire Vulnerability Research Team Labs
Sourcefire Vulnerability Research Team LabsSourcefire Vulnerability Research Team Labs
Sourcefire Vulnerability Research Team Labslosalamos
 
Best Practices for performance evaluation and diagnosis of Java Applications ...
Best Practices for performance evaluation and diagnosis of Java Applications ...Best Practices for performance evaluation and diagnosis of Java Applications ...
Best Practices for performance evaluation and diagnosis of Java Applications ...IndicThreads
 

Similar a Restfs internals (20)

FPC for the Masses - CoRIIN 2018
FPC for the Masses - CoRIIN 2018FPC for the Masses - CoRIIN 2018
FPC for the Masses - CoRIIN 2018
 
Check the version with fixes. Link in description
Check the version with fixes. Link in descriptionCheck the version with fixes. Link in description
Check the version with fixes. Link in description
 
PASTE: A Network Programming Interface for Non-Volatile Main Memory
PASTE: A Network Programming Interface for Non-Volatile Main MemoryPASTE: A Network Programming Interface for Non-Volatile Main Memory
PASTE: A Network Programming Interface for Non-Volatile Main Memory
 
The post release technologies of Crysis 3 (Slides Only) - Stewart Needham
The post release technologies of Crysis 3 (Slides Only) - Stewart NeedhamThe post release technologies of Crysis 3 (Slides Only) - Stewart Needham
The post release technologies of Crysis 3 (Slides Only) - Stewart Needham
 
Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I s...
Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I s...Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I s...
Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I s...
 
Splunk app for stream
Splunk app for stream Splunk app for stream
Splunk app for stream
 
LISA18: Hidden Linux Metrics with Prometheus eBPF Exporter
LISA18: Hidden Linux Metrics with Prometheus eBPF ExporterLISA18: Hidden Linux Metrics with Prometheus eBPF Exporter
LISA18: Hidden Linux Metrics with Prometheus eBPF Exporter
 
Oracle Basics and Architecture
Oracle Basics and ArchitectureOracle Basics and Architecture
Oracle Basics and Architecture
 
Elasticsearch on Kubernetes
Elasticsearch on KubernetesElasticsearch on Kubernetes
Elasticsearch on Kubernetes
 
Debugging linux issues with eBPF
Debugging linux issues with eBPFDebugging linux issues with eBPF
Debugging linux issues with eBPF
 
Caching Methodology & Strategies
Caching Methodology & StrategiesCaching Methodology & Strategies
Caching Methodology & Strategies
 
Caching methodology and strategies
Caching methodology and strategiesCaching methodology and strategies
Caching methodology and strategies
 
Troubleshooting .net core on linux
Troubleshooting .net core on linuxTroubleshooting .net core on linux
Troubleshooting .net core on linux
 
SQL Server On SANs
SQL Server On SANsSQL Server On SANs
SQL Server On SANs
 
OSCON 2011 Learning CouchDB
OSCON 2011 Learning CouchDBOSCON 2011 Learning CouchDB
OSCON 2011 Learning CouchDB
 
Java File I/O Performance Analysis - Part I - JCConf 2018
Java File I/O Performance Analysis - Part I - JCConf 2018Java File I/O Performance Analysis - Part I - JCConf 2018
Java File I/O Performance Analysis - Part I - JCConf 2018
 
A New Framework for Detection
A New Framework for DetectionA New Framework for Detection
A New Framework for Detection
 
Sourcefire Vulnerability Research Team Labs
Sourcefire Vulnerability Research Team LabsSourcefire Vulnerability Research Team Labs
Sourcefire Vulnerability Research Team Labs
 
Best Practices for performance evaluation and diagnosis of Java Applications ...
Best Practices for performance evaluation and diagnosis of Java Applications ...Best Practices for performance evaluation and diagnosis of Java Applications ...
Best Practices for performance evaluation and diagnosis of Java Applications ...
 
Docker as an every day work tool
Docker as an every day work toolDocker as an every day work tool
Docker as an every day work tool
 

Más de Manfred Furuholmen (20)

Pisa
PisaPisa
Pisa
 
Samba4 Introduction
Samba4 IntroductionSamba4 Introduction
Samba4 Introduction
 
Introduction to message_queue
Introduction to message_queueIntroduction to message_queue
Introduction to message_queue
 
Restfs
RestfsRestfs
Restfs
 
Winbind as Identity Management Connector
Winbind as Identity Management ConnectorWinbind as Identity Management Connector
Winbind as Identity Management Connector
 
Use Distributed Filesystem as a Storage Tier
Use Distributed Filesystem as a Storage TierUse Distributed Filesystem as a Storage Tier
Use Distributed Filesystem as a Storage Tier
 
Managing OpenAFS users with OpenIDM
Managing OpenAFS users with OpenIDMManaging OpenAFS users with OpenIDM
Managing OpenAFS users with OpenIDM
 
Afs manager
Afs managerAfs manager
Afs manager
 
Pt server ng
Pt server ngPt server ng
Pt server ng
 
Best Practices to create High Load Websites
Best Practices to create High Load WebsitesBest Practices to create High Load Websites
Best Practices to create High Load Websites
 
Be lazy... make automation
Be lazy... make automationBe lazy... make automation
Be lazy... make automation
 
Disaster recovery
Disaster recoveryDisaster recovery
Disaster recovery
 
Domestic cloud
Domestic cloudDomestic cloud
Domestic cloud
 
Inexpensive storage
Inexpensive storageInexpensive storage
Inexpensive storage
 
Samba management Console
Samba management ConsoleSamba management Console
Samba management Console
 
Link Samba to Cloud Storage
Link Samba to Cloud StorageLink Samba to Cloud Storage
Link Samba to Cloud Storage
 
Samba as a gateway to OpenAFS
Samba as a gateway to OpenAFSSamba as a gateway to OpenAFS
Samba as a gateway to OpenAFS
 
Samba distributed env
Samba distributed envSamba distributed env
Samba distributed env
 
AFS introduction
AFS introductionAFS introduction
AFS introduction
 
AFS case study
AFS case studyAFS case study
AFS case study
 

Último

Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataSafe Software
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
PicPay - GenAI Finance Assistant - ChatGPT for Customer Service
PicPay - GenAI Finance Assistant - ChatGPT for Customer ServicePicPay - GenAI Finance Assistant - ChatGPT for Customer Service
PicPay - GenAI Finance Assistant - ChatGPT for Customer ServiceRenan Moreira de Oliveira
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
Babel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxBabel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxYounusS2
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
RAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIRAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIUdaiappa Ramachandran
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdfJamie (Taka) Wang
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 

Último (20)

Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
PicPay - GenAI Finance Assistant - ChatGPT for Customer Service
PicPay - GenAI Finance Assistant - ChatGPT for Customer ServicePicPay - GenAI Finance Assistant - ChatGPT for Customer Service
PicPay - GenAI Finance Assistant - ChatGPT for Customer Service
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
Babel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxBabel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptx
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
RAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIRAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AI
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 

Restfs internals

Notas del editor

  1. The session devided in three main parts, a small introduction on ecosystem, the presentation of the Goals and architecture of Restfs and a small demo of same basic capabilties of the RestFS
  2. I don’t know in which point of the curve we are, you can understand the end of all tecnologies is coming But on the other hand we can’t find mature tecnologies we still in innovation
  3. For this reason we start a RestFS project, because we want re thinking the storage,Last but not the least Today you can’t find cloud implementation that you can setup, only service are avaible no software but also make testing and experiment on new tecnologieFrom one side we want create scalable system On other side we want more framework for test.
  4. For this reason we start a RestFS project, because we want re thinking the storage,Last but not the least Today you can’t find cloud implementation that you can setup, only service are avaible no software but also make testing and experiment on new tecnologieFrom one side we want create scalable system On other side we want more framework for test.
  5. The principle used for all decicion in design and to identify the right or best solution is base on the idea behind hadoop or GFS and it is
  6. The principle used for all decicion in design and to identify the right or best solution is base on the idea behind hadoop or GFS and it is
  7. The principle used for all decicion in design and to identify the right or best solution is base on the idea behind hadoop or GFS and it is
  8. For this reason we start a RestFS project, because we want re thinking the storage,Last but not the least Today you can’t find cloud implementation that you can setup, only service are avaible no software but also make testing and experiment on new tecnologieFrom one side we want create scalable system On other side we want more framework for test.
  9. Now I will describe a little bit more RestFS, functionality and architcture
  10. Now I will describe a little bit more RestFS, functionality and architcture
  11. Wedevide the design in 5 areas
  12. Hash for change by clientSerial for keep sequence and history (updated by server)
  13. Hash for change by clientSerial for keep sequence and history (updated by server)
  14. Hash for change by clientSerial for keep sequence and history (updated by server)
  15. Hash for change by clientSerial for keep sequence and history (updated by server)
  16. Hash for change by clientSerial for keep sequence and history (updated by server)
  17. Hash for change by clientSerial for keep sequence and history (updated by server)
  18. BSON [bee · sahn], short for Bin­ary JSON, is a bin­ary-en­coded seri­al­iz­a­tion of JSON-like doc­u­ments. Like JSON, BSON sup­ports the em­bed­ding of doc­u­ments and ar­rays with­in oth­er doc­u­ments and ar­rays. BSON also con­tains ex­ten­sions that al­low rep­res­ent­a­tion of data types that are not part of the JSON spec. For ex­ample, BSON has a Date type and a BinData type.BSON can be com­pared to bin­ary inter­change for­mats, like Proto­col Buf­fers. BSON is more &quot;schema-less&quot; than Proto­col Buf­fers, which can give it an ad­vant­age in flex­ib­il­ity but also a slight dis­ad­vant­age in space ef­fi­ciency (BSON has over­head for field names with­in the seri­al­ized data).
  19. BSON [bee · sahn], short for Bin­ary JSON, is a bin­ary-en­coded seri­al­iz­a­tion of JSON-like doc­u­ments. Like JSON, BSON sup­ports the em­bed­ding of doc­u­ments and ar­rays with­in oth­er doc­u­ments and ar­rays. BSON also con­tains ex­ten­sions that al­low rep­res­ent­a­tion of data types that are not part of the JSON spec. For ex­ample, BSON has a Date type and a BinData type.BSON can be com­pared to bin­ary inter­change for­mats, like Proto­col Buf­fers. BSON is more &quot;schema-less&quot; than Proto­col Buf­fers, which can give it an ad­vant­age in flex­ib­il­ity but also a slight dis­ad­vant­age in space ef­fi­ciency (BSON has over­head for field names with­in the seri­al­ized data).
  20. Hash for change by clientSerial for keep sequence and history (updated by server)
  21. Hash for change by clientSerial for keep sequence and history (updated by server)
  22. Hash for change by clientSerial for keep sequence and history (updated by server)
  23. Hash for change by clientSerial for keep sequence and history (updated by server)
  24. Hash for change by clientSerial for keep sequence and history (updated by server)