Denser containers with PF cache - Pavel Emelyanov

•Descargar como ODP, PDF•

0 recomendaciones•612 vistas

This document discusses using PFCache to improve storage density for container files by deduplicating identical data. PFCache uses a cache area to store deduplicated file contents, referenced via cache links in container image files. Evaluation showed PFCache improved storage density. Future work includes upstreaming PLoop for containers and pursuing IO deduplication in the Linux kernel for additional benefits.

Software

Denser containers with PFCacheDenser containers with PFCache
Pavel Emelyanov
ContainerCon, Seattle, 2015

AgendaAgenda
• How to store container files
• Why shared template matters
• What can be deduplicated and what should be
• PFCache
• Q&A
2

How to store container filesHow to store container files
3
Filesystem
Container
processes

How to store container filesHow to store container files
4
Filesystem
Container
processes
Block device
NetworkHost
Filesystem
Host
block device
Hardware

How to store container files (1)How to store container files (1)
5
Filesystem
Container
processes
Block device
NetworkHost
Filesystem
Host
block device
Hardware
Chroot()
Union FS

How to store container files (2)How to store container files (2)
6
Filesystem
Container
processes
Block device
NetworkHost
Filesystem
Host
block device
Hardware
Loop device
ZFS ZVol
BTRFS subvolume
PLoop

What's PLoopWhat's PLoop
• Loop device plus
– AIO for better performance
– Snapshots
– QCOW2-like format for thin provisioning
– Thin provisionong itself
• Upstreaming work in progress
7

How to store container files (3)How to store container files (3)
8
Filesystem
Container
processes
Block device
NetworkHost
Filesystem
Host
block device
Hardware
LVM
DM-thin

How to store container files (4)How to store container files (4)
9
Filesystem
Container
processes
Block device
NetworkHost
Filesystem
Host
block device
Hardware
NBD
Ceph RBD
iSCSI

How to store container files (5)How to store container files (5)
10
Filesystem
Container
processes
Block device
NetworkHost
Filesystem
Host
block device
Hardware
NFS
GFS2
OCFS
Ceph

Containers vs TemplatesContainers vs Templates
• Containers ...
– are massively cloned from pre-created “templates”
– do not have direct access to the underlying (block) storage
• Identical data can be effectively deduplicated
– Higher density
– Lower IO and/or memory consumption
11

Who can do shared templatesWho can do shared templates
12
Storage OpenVZ Docker LXC
Union FSs + + +
Btrfs +
DM-thin +
PLoop +
Ceph
ZFS +

What can be de-duplicatedWhat can be de-duplicated
13
Filesystem
Container
processes
Block device Network

What can be de-duplicatedWhat can be de-duplicated
14
Filesystem
Container
processes
Block device Network
Page cache
Cached pages

What can be de-duplicatedWhat can be de-duplicated
15
Filesystem
Container
processes
Block device Network
Page cache
Cached pages
IO flow

What is deduplicatedWhat is deduplicated
16
Storage Memory IO
Union FSs + +
Btrfs +/-
DM-thin
PLoop + +
Ceph
ZFS

Additional OpenVZ constraintsAdditional OpenVZ constraints
• Containers disks are independent image files
– Can be easily copied across nodes
– No single (shared) point of failure
• Deduplicated data is volatile
– “Templates” can be lost (e.g. while migrating)
– Too big pool with shared data can be easily shrunk
17

Virtuozzo IO stackVirtuozzo IO stack
18
Ext4
Container
processes
PLoop device
Image file

PF-CachePF-Cache
19
Ext4
PLoop device
Cache area
Cache link (xattr)Ext4
Container
processes
PLoop device
Image file Image file

Cache and cache link behaviorCache and cache link behavior
• Cache area
– target file name is sha1 sum of the contents
– files are created by user-space daemon
– cache size is limited by ploop
• Cache link
– created automatically upon file creation
– dropped when file is opened for writing
– Is kept during metadata update (chown/chmod)
20

Future workFuture work
• PLoop is available in OpenVZ & Virtuozzo
– Upstream WIP
• IO deduplication in the upstream
– Issue raied at 2013'th LSFMM
– DM-thin/btrfs IO dedup for containers
– KSM++ for VM-s
22

Thank you
xemul@odin.com
Thank you
xemul@odin.com

Más contenido relacionado

La actualidad más candente

Comparison between OCFS2 and GFS2

Gang He

Gsummit apis-2013

Gluster.org

Introduction to file system and OCFS2

Gang He

Hdfs internals

Bhupesh Chawda

Glusterfs session #2 1 layer above disk filesystems

Pranith Karampuri

redis basics

Manoj Kumar

[Nvidia] Divide Your Depots

Perforce

Containers from scratch

Rakuten Group, Inc.

OSBConf 2015 | Scale out backups with bareos and gluster by niels de vos

NETWAYS

Merge2013 mwarren-presentation1.pptx(pv6)

Mark Warren

Level 101 for Presto: What is PrestoDB?

Ali LeClerc

MongoDB

jsterce

Filesystem Comparison: NFS vs GFS2 vs OCFS2

Giuseppe Paterno'

Redis, a 2 minutes introduction

Mirko Calvaresi

The Care + Feeding of a Mongodb Cluster

Unix 6 en

HDFS Deep Dive

Redis

Backup / Restore to Cloud Storage with esXpress and CloudArray software

TwinStrata

Writing file system in CPython

delimitry

La actualidad más candente (20)

Comparison between OCFS2 and GFS2

Gsummit apis-2013

Introduction to file system and OCFS2

Hdfs internals

Glusterfs session #2 1 layer above disk filesystems

redis basics

[Nvidia] Divide Your Depots

Containers from scratch

OSBConf 2015 | Scale out backups with bareos and gluster by niels de vos

Merge2013 mwarren-presentation1.pptx(pv6)

Level 101 for Presto: What is PrestoDB?

MongoDB

Filesystem Comparison: NFS vs GFS2 vs OCFS2

Redis, a 2 minutes introduction

The Care + Feeding of a Mongodb Cluster

Unix 6 en

HDFS Deep Dive

Redis

Backup / Restore to Cloud Storage with esXpress and CloudArray software

Writing file system in CPython

Similar a Denser containers with PF cache - Pavel Emelyanov

Oracle Open World 2014: Lies, Damned Lies, and I/O Statistics [ CON3671]

Kyle Hailey

Ioppt

arun sharma

PostgreSQL + ZFS best practices

Sean Chittenden

ZFS provides several advantages over traditional block-based filesystems when used with PostgreSQL, including preventing bitrot, improved compression ratios, and write locality. ZFS uses copy-on-write and transactional semantics to ensure data integrity and allow for snapshots and clones. Proper configuration such as enabling compression and using ZFS features like intent logging can optimize performance when used with PostgreSQL's workloads.

002-Storage Basics and Application Environments V1.0.pptx

DrewMe1

Storage Basics and Application Environments is a document that discusses storage concepts, hardware, protocols, and data protection basics. It begins by defining storage and describing different types including block storage, file storage, and object storage. It then covers basic concepts of storage hardware such as disks, disk arrays, controllers, enclosures, and I/O modules. Storage protocols like SCSI, NVMe, iSCSI, and Fibre Channel are also introduced. Additional concepts like RAID, LUNs, multipathing, and file systems are explained. The document provides a high-level overview of fundamental storage topics.

Case study of BtrFS: A fault tolerant File system

Kumar Amit Mehta

UKOUG, Lies, Damn Lies and I/O Statistics

Kyle Hailey

1. Many factors can cause storage performance anomalies that make benchmarking difficult. Caching, shared infrastructure, I/O consolidation and fragmentation, and tiered storage are some of the top issues. 2. It is important to use real workloads, capture latency histograms rather than just averages, ensure results are reproducible, and run tests long enough to reach steady state. 3. Proper testing methodology is required to accurately characterize storage performance and avoid anomalies. Tools like FIO can help simulate real workloads.

M3 nas architecture-3.1.1

MrudulaJoshi10

This document discusses network attached storage (NAS) and file systems for network storage. It defines NAS as storage units that are on the network and accessed as files from file systems supported by NAS servers. The document outlines different types of storage arrays including SAN, NAS, and unified storage. It also describes NAS architectures like the NAS server and NAS gateway configurations and how file systems manage blocks of disk to provide files to applications. The document discusses scaling NAS by deploying multiple NAS arrays or using a scale-out NAS cluster with a single global namespace.

It's the End of Data Storage As We Know It (And I Feel Fine)

Stephen Foskett

Technological change is finally coming to storage, and it will wipe away the architecture we've come to know over the last few decades. Say goodbye to the "do it all" Fibre Channel SAN storage array and get ready for converged infrastructure, distributed storage, alternative attachments like PCIe, and top-of-rack flash! In this session, Stephen Foskett will explain why this change is inevitable and how it will shake out. You won't recognize what's coming, but it will be faster, cheaper, and more integrated than ever! Delivered at

Kfs presentation

Petrovici Florin

KFS aka Kosmos FS is a distributed file system written in C++ that is modeled after HDFS. It was originally developed by Kosmix, which was later acquired by Walmart. Some key points: - KFS uses a master/chunkserver architecture where the metadata is stored on a master node and file data is stored in chunks on chunkservers. - It supports features like replication, data integrity checks, and rebalancing of chunks. - While still in early stages, it provides alternatives to the Hadoop ecosystem through its C++ implementation and bindings for other languages like Java and Python. - The documentation provides instructions for building, deploying, and accessing KFS, though some functionality

Containers in a File

OpenVZ

Containers are typically managed by having each container chroot to its own subdirectory of the host filesystem. This leads to problems like journal bottlenecks and inefficient small file I/O. The proposed solution is to manage each container's filesystem within a virtual block device represented by a file (container-in-a-file). This avoids journal bottlenecks and allows efficient operations like backup, migration and snapshots through copy-on-write images. It provides flexibility in filesystem choice and management while solving storage and I/O issues. Future work includes optimizing the design and integrating it into the Linux kernel.

Containers in a file

OpenVZ

009709863.pdf

KalsoomTahir2

This document provides an overview of file system topics. It begins with an introduction to file systems and their relationship to operating system architecture. It then discusses the Virtual File System (VFS) interface and key metadata components like super blocks, inodes, and directory entries. The document reviews common file system optimizations based on memory hierarchy and storage characteristics. Examples of specific file systems are given, including Ext4, NTFS, ZFS, NFS, and Google File System. The document concludes by soliciting any questions.

Learning from ZFS to Scale Storage on and under Containers

inside-BigData.com

Evan Powell presented this deck at the MSST 2107 Mass Storage Conference. "What is so new about the container environment that a new class of storage software is emerging to address these use cases? And can container orchestration systems themselves be part of the solution? As is often the case in storage, metadata matters here. We are implementing in the open source OpenEBS.io some approaches that are in some regards inspired by ZFS to enable much more efficient scale out block storage for containers that itself is containerized. The goal is to enable storage to be treated in many regards as just another application while, of course, also providing storage services to stateful applications in the environment." Watch the video: http://wp.me/p3RLHQ-gPs Learn more: blog.openebs.io and http://storageconference.us Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

Storage in hadoop

Puneet Tripathi

Vancouver bug enterprise storage and zfs

Rami Jebara

This document provides an overview of enterprise storage needs and introduces the ZFS file system. It discusses typical enterprise storage requirements, components, tiers, and concerns. It then provides a brief history of ZFS, introduces key concepts like storage pools, datasets, snapshots, replication, and more. It offers tips for preparing ZFS for production and highlights emerging trends, resources for learning more, and examples of tools that use ZFS.

Tuning Solr & Pipeline for Logs

Sematext Group, Inc.

An updated talk about how to use Solr for logs and other time-series data, like metrics and social media. In 2016, Solr, its ecosystem, and the operating systems it runs on have evolved quite a lot, so we can now show new techniques to scale and new knobs to tune. We'll start by looking at how to scale SolrCloud through a hybrid approach using a combination of time- and size-based indices, and also how to divide the cluster in tiers in order to handle the potentially spiky load in real-time. Then, we'll look at tuning individual nodes. We'll cover everything from commits, buffers, merge policies and doc values to OS settings like disk scheduler, SSD caching, and huge pages. Finally, we'll take a look at the pipeline of getting the logs to Solr and how to make it fast and reliable: where should buffers live, which protocols to use, where should the heavy processing be done (like parsing unstructured data), and which tools from the ecosystem can help.

Tuning Solr and its Pipeline for Logs: Presented by Rafał Kuć & Radu Gheorghe...

Lucidworks

The document summarizes key points from a presentation on optimizing Solr and log pipelines for time-series data. The presentation covered using time-based Solr collections that rotate based on size, tiering hot and cold clusters, tuning OS and Solr settings, parsing logs, buffering pipelines, and shipping logs using protocols like UDP, TCP, and Kafka. The overall conclusions were that tuning segments per tier and max merged segment size improved indexing throughput, and that simple, reliable pipelines like Filebeat to Kafka or rsyslog over UNIX sockets generally work best.

LAS16-400: Mini Conference 3 AOSP (Session 1)

Linaro

LAS16-400: Mini Conference 3 AOSP (Session 1) Speakers: Thomas Gall, Bernhard Rosenkränzer Date: September 29, 2016 ★ Session Description ★ The Android Open Source Project is one community which is strategic to Linaro and it’s members. The purpose of this mini conference is to gather fellow Android engineers together from the community, member companies, and Linaro to discuss engineering activities and improve collaboration across different groups. Within this mini conference we encourage discussion and presentations to advance engineering topics, forge consensus and educate each other. The tentative agenda for this mini conference includes : - Quick introduction - Filesystems - Between requirements for encryption and standing concerns about degrading performance as an Android file system age, let’s have some discussion involving current data, known issues and towards improvements in this area for Android. - HAL consolidation - Review current status and discuss next steps to work on. One build for many devices: device/build configuration. Next features and platforms to add. Gaps in HiKey support vs. AOSP build. - Graphics - YUV support in mesa and hwc. - WiFi and sensor HAL status and next steps - New developments with AOSP + the Kernel - With regards to the Google Common Kernel tree and upstream Linux kernel activities related to Android, there are a few topics up for discussion: - - Updates on HiKey in AOSP - - EAS in common.git & integration with AOSP userspace - - New Sync API in 4.6+ kernels, and how it will affects graphics drivers - AOSP transition to clang - As everyone knows GCC in AOSP has been deprecated. Let’s cover current status, issues and next steps. Let’s also discuss the elephant in the room, building the kernel with clang. - Out of tree AOSP User space Patches - This is a discussion with the goal of organized action to see forward progress on AOSP user space patches that aren’t in AOSP for whatever reason. - Android is used in some environments where booting can be frequent and affect the product experience. Do you want to wait for a minute while your car boots? We’ll spend time brainstorming on improving Android boot time. ★ Resources ★ Etherpad: pad.linaro.org/p/las16-400 Presentations & Videos: http://connect.linaro.org/resource/las16/las16-400/ ★ Event Details ★ Linaro Connect Las Vegas 2016 – #LAS16 September 26-30, 2016 http://www.linaro.org http://connect.linaro.org

State of the Art Thin Provisioning

Stephen Foskett

Thin provisioning allows storage arrays to provision more capacity than is physically available by only allocating space as it is used. This improves efficiency but can lead to issues if overprovisioned storage runs out. There are challenges to thin provisioning across different layers including file systems, virtualization, and storage arrays. For thin provisioning to be effective, all layers must work together to monitor capacity usage and free space accurately at a fine granularity.

Tachyon-2014-11-21-amp-camp5

Haoyuan Li

Tachyon is a memory-centric distributed storage system that provides reliable data sharing at memory speed across various cluster computing frameworks. It addresses issues with current storage systems like slow data sharing due to disk writes, cache loss when processes crash, and in-memory data duplication. Tachyon keeps only one copy of data in memory, tracks data lineage for fault tolerance, and enables fast sharing of data within and across frameworks and jobs. It provides a simple API and allows frameworks like Spark and MapReduce to access data reliably from memory without code changes.

Similar a Denser containers with PF cache - Pavel Emelyanov (20)

Oracle Open World 2014: Lies, Damned Lies, and I/O Statistics [ CON3671]

Ioppt

PostgreSQL + ZFS best practices

002-Storage Basics and Application Environments V1.0.pptx

Case study of BtrFS: A fault tolerant File system

UKOUG, Lies, Damn Lies and I/O Statistics

M3 nas architecture-3.1.1

It's the End of Data Storage As We Know It (And I Feel Fine)

Kfs presentation

Containers in a File

Containers in a file

009709863.pdf

Learning from ZFS to Scale Storage on and under Containers

Storage in hadoop

Vancouver bug enterprise storage and zfs

Tuning Solr & Pipeline for Logs

Tuning Solr and its Pipeline for Logs: Presented by Rafał Kuć & Radu Gheorghe...

LAS16-400: Mini Conference 3 AOSP (Session 1)

State of the Art Thin Provisioning

Tachyon-2014-11-21-amp-camp5

Más de OpenVZ

PFcache - LinuxCon 2015

OpenVZ

Speeding up ps and top

OpenVZ

This document proposes a new interface called task_diag to provide process information in a more efficient way compared to the current /proc interface. Task_diag would use a netlink message format to allow querying process attributes in groups and splitting the response across multiple messages. Performance tests show task_diag can provide process information 2-8 times faster than parsing /proc. The goal is to speed up tools like ps and top.

Live migration: pros, cons and gotchas -- Pavel Emelyanov

OpenVZ

Live migrating containers has pros like load balancing and updating hardware, but also cons. It is complex due to needing to save a container's state, transfer it, and restore it on another host while tasks are frozen. This can be done with memory pre-copy, where memory is copied iteratively while tasks run, or post-copy, where memory is transferred after tasks resume. P.Haul uses CRIU to handle the critical state save and restore functions needed for live migrating containers between hosts.

Live migrating a container: pros, cons and gotchas -- Pavel Emelyanov

OpenVZ

Live migrating a container: pros, cons and gotchas Monday, November 16 • 17:20 - 18:05 Pavel Emelyanov Principal Engineer, Odin Principal engineer at Odin Server Virtualization team, creator and maintainer of the CRIU project. Joined Parallels in 2004 as junior Linux kernel developer, later became kernel team leader. Now works on architecture of the Odin Server products. | | Pavel tweets at @xemulp. http://dockerconeu2015.sched.org/event/62e6d2ea7380442a48fafaeee26c9842

CRIU: time and space travel for Linux containers -- Kir Kolyshkin

OpenVZ

Тестирование ПО, основанного на сторонних компонентах - Денис Силаков, SECR 2015

OpenVZ

В своей презентации мы на примере дистрибутива Linux расскажем об опыте организации процесса тестирования продукта, существенная часть (более 90%) кода которого создается независимыми от компании разработчиками. https://www.youtube.com/watch?v=AstgrnE7_dI

Живая миграция: плюсы, минусы и подводные камни - Павел Емельянов

OpenVZ

What's missing from upstream kernel containers? - Sergey Bronnikov

OpenVZ

Проблема фрагментации виртуальных дисков и способы её решения -- Дмитрий Монахов

OpenVZ

Развёртывание приложений Docker в контейнерах Virtuozzo -- Павел Тихомиров

OpenVZ

CRIU: ускорение запуска PHP в CloudLinux OS -- Руслан Купреев

OpenVZ

LibCT и контейнеры на уровне приложений -- Александр Бурлука

OpenVZ

Управление памятью контейнеров в проекте OpenVZ -- Владимир Давыдов

OpenVZ

Живая миграция контейнеров: плюсы, минусы, подводные камни -- Павел Емельянов

OpenVZ

LibCT: one lib to rule them all -- Andrey Vagin

OpenVZ

LibCT is a C library that allows building containerized applications by configuring namespaces and cgroups to provide isolation. It aims to simplify the complex low-level container APIs and support different container types like Linux containers, OpenVZ, Solaris Zones, and BSD jails. LibCT hides low-level API changes and provides bindings for other languages. It also serves as an alternative to Libcontainer which is written in Go. The presentation covered the history of Linux containers, namespaces, cgroups, LibCT's API, examples of use, and future integration plans.

CGroups kernel memory controller -- Pavel Emelyanov

OpenVZ

What's missing from upstream kernel containers? - Kir Kolyshkin, Sergey Bronn...

OpenVZ

This document discusses containers in the upstream Linux kernel compared to the Virtuozzo (VZ) kernel. It notes that OpenVZ has contributed many features to the Linux kernel over time, with about 60% of the VZ kernel now upstreamed. While much progress has been made, there are still areas needing to be added to the upstream kernel to fully replace the VZ kernel, including memory management and accounting features, network and I/O optimizations, and legacy containerization tools. The presenters from OpenVZ are available to discuss any questions.

Not so brief history of Linux Containers - Kir Kolyshkin

OpenVZ

Linux containers have evolved from initial ideas in 1999 to widespread use today. Early container technologies included Virtuozzo in 1999-2000 and the introduction of namespaces in 2000-2001. The Linux-VServer project in 2001 helped advance containers. Checkpointing and live migration were implemented in 2002-2003. OpenVZ in 2005 helped popularize containers. Developments from 2006-2010 included new namespaces and use of cgroups. Docker in 2013 and projects like CoreOS and LXC further advanced containers. CRIU, introduced in 2011, helped enable checkpoint/restore in userspace. Containers are now widely used in projects like OpenStack.

Openvz booth

OpenVZ

This document compares virtual machines (VMs) to containers and discusses their differences. It notes that VMs have higher overhead than containers and can support more instances per server. Containers offer near-native performance and scale better. The document also outlines ongoing and future work to further integrate container technologies into the Linux kernel to provide capabilities like checkpoint/restore and live migration of containers across servers.

Управление ресурсами в Linux и OpenVZ

OpenVZ

Управление ресурсами в Linux и OpenVZ Кирилл Колышкин kir@openvz.org http://openvz.org/ Отчет - http://yourcmc.ru/wiki/RootConf_2009:_%D0%9E%D1%82%D1%87%D1%91%D1%82_%D0%92%D0%B8%D1%82%D0%B0%D0%BB%D0%B8%D1%8F_%D0%A4%D0%B8%D0%BB%D0%B8%D0%BF%D0%BF%D0%BE%D0%B2%D0%B0#.D0.A3.D0.BF.D1.80.D0.B0.D0.B2.D0.BB.D0.B5.D0.BD.D0.B8.D0.B5_.D1.80.D0.B5.D1.81.D1.83.D1.80.D1.81.D0.B0.D0.BC.D0.B8_.D0.B2_Linux_.D0.B8_OpenVZ_.28.D0.B7.D0.B0.D1.87.D1.91.D1.82.21.29

Más de OpenVZ (20)