This document describes DeepTorrent, a distributed storage system based on Bittorrent that aims to provide permanent storage. It discusses issues with cloud storage like high energy consumption, limited security, and legal restrictions. DeepTorrent uses Python and C libraries and Redis for control flow. It allows uploading and downloading files and managing storers. DeepTorrent is being tested for archiving production logs and medical imaging data. Future work includes access control and distributed control flow using DHT.

1. DeepTorrent
Stockage distribué
perenne basé sur
Bittorrent
Gil Utard
Ugloo SAS/MIS UPJV

2. Digital Revolution

4. Backup

5. 20 TB
100 PB
Inflation des données

6. CLOUD

7. CLOUD
?

8. + High Energy Consumpton
+ Limited Security Flow & Storage
+ Legal Restrictons (GRDP, USA, Germany…)
FORECAST
AUDIT
RADIOLOGYQUANTITY &
COST
CLOUD STORAGE

9. FORECAST
AUDIT
RADIOLOGYQUANTITY &
COST
65% HARDWARE CAPACITY NOT USED
=
200 DATACENTERS IN FRANCE
EDGE STORAGE

10. Stockage P2P (put)

11. Récupération (get)

12. Regénération

13. Bittorrent
Seeder
Leecher
Leetcher Leetcher
Leetcher
File → Swarm

14. Bit Torrent Deep Torrent→

15. Deep Torrent
●
Python + C
●
Librairies :
– Libtorrent (BSD) : http://libtorrent.org
– ISA-L (Intel) : https://sofware.intel.com/
●
Flot de contrôle :
– Redis : http://redis.io
●
Multi-plateforme :
– Deb, MSI

16. Commande Client
up2p --help
up2p.
Usage:
up2p put <source> <uuid> [-a] [--timeout=<timeout>]
up2p get <uuid> <destination> [-a] [--timeout=<timeout>]
up2p delete <uuid>
up2p status <task_id>
up2p wait <task_id> [--timeout=<timeout>]
up2p storer <storer> (empty-trash|list-trash)
up2p storer <storer> torrents [--info]
up2p storer-status <storer> <task_id>
Options:
-h --help Show this screen.
-a, --async Run the command in async mode
--timeout=<timeout> Set timeout.

17. Commande Système
root@client:/shared# up2p-tools --help
up2p-tools.
Usage:
up2p-tools list-storers [<critical_time>]
up2p-tools (look|get-torrent-info|get-info-hash) <torrent_uuid>
up2p-tools check-queues
up2p-tools (auto-config|apply-config)
up2p-tools prepare-config <ip_address>
up2p-tools node-uuid
up2p-tools hash-file <source>
up2p-tools state [<source>]
Options:
<critical_time> Set critical time.
<torrent_uuid> The torrent uuid.
<ip_address> The IP ip_address of the client.

18. Commande Storer
root@client:/shared# up2pstorer-config --help
up2pstorer config.
Usage:
up2pstorer-config [--download_rate_limit=<download_rate_limit>] [--
upload_rate_limit=<upload_rate_limit>] [(--percent_free_disk=<percent_free_disk>
--minimum_free_disk=<minimum_free_disk>)]
up2pstorer-config --banish-me
up2pstorer-config --unbanish-me
Options:
-h --help Show this screen.
--banish-me Ban the current storer from up2p (cannot download anymore).
--unbanish-me Unban the current storer from up2p (can download again).
--download_rate_limit=<download_rate_limit> Set the download limit rate.
--upload_rate_limit=<upload_rate_limit> Set the upload limit rate.
--percent_free_disk=<percent_free_disk> Set the free space disk in percent.
--minimum_free_disk=<minimum_free_disk> Set the free space disk in Go.

19. Interfaces
●
Amazon S3 :
– Minio : http://minio.io
●
Posix :
– Fuse
●
Compatibilité :
– Duplicity, Veeam, Rubrick, Spectrum, ...

20. Travaux en cours
●
Control d’accès anonyme :
– « An Anonymous Data Control Access Protocole in Distributed Storage Systems », M. Ben Fadl & G. Utard,
4th IEEE Workshop on Security and Privacy in Cloud, Bejing, June 2018
●
Distribution du flot de contrôle : DHT
– « U-RPC : a Protocol for Microservices in DHT » , G. Utard, H.C. Le & T.T Tran, SoICT 17, Dec. 2017

21. Conclusion
●
Deep Torrent : bientôt disponible sur GitHub (AGPL)
●
Déploiement :
– Procter & Gamble : archivage log production (traçabilité)
– CHU Amiens : Archivage Imagerie Médicale (>100 To)
●
Projet Suivant :
– Blockchain : marché de stockage (Sia, Storj like)