The NixOS project and deploying systems declaratively

The NixOS project and deploying systems
declaratively
Sander van der Burg
March 12, 2016
Sander van der Burg The NixOS project and deploying systems declaratively

On being declarative
A declarative sentence makes a statement. It is punctuated by a
period:
The dog in the neighbor’s yard is barking.
(Source: http://www.slideshare.net/luigi a97/parts-of-a-sentence-8862361)

On being imperative
An imperative sentence is a command or polite request:
(Source: https://xkcd.com/149/)

On being declarative in programming
A style of building the structure and elements of computer
programs – that expresses the logic of a computation without
describing its control ﬂow
(Source: https://en.wikipedia.org/wiki/Declarative programming)

Declarative: describing ”what” is to be computed rather than
”how” to compute the result/behavior
Imperative: a description of a computation that involves
implicit eﬀects, usually mutable state and input/output.
(Source:
http://wcook.blogspot.com/2013/05/declarative-versus-imperative.html)

Declarative: describing ”what” is to be computed rather than
”how” to compute the result/behavior
Imperative: a description of a computation that involves
implicit eﬀects, usually mutable state and input/output.
(Source:
http://wcook.blogspot.com/2013/05/declarative-versus-imperative.html)
Declarative
“declarative” is a spectrum – hard to draw a hard line
between “what” and “how”.
Imperative is not necessarily the opposite of
declarative.

Example: HTML and CSS
<!DOCTYPE html>
<html>
<head>
<title>Test</title>
<link rel=”stylesheet” href=”style.css” type=”text/css”>
</head>
<body>
<div id=”outer”>
<div id=”inner”>
<p>HTML and CSS are declarative and so cool!</p>
</div>
</div>
</body>
</html>
#outer {
margin−left: auto;
margin−right: auto;
width: 20%;
border−style: solid;
}
#inner {
width: 500px;
}

Example: HTML and CSS

Deployment: What do we want?

Deployment: Activities
Building
Packaging
Transferring packages from producer to consumer site
Activating
Deactivating
Modifying conﬁguration ﬁles
Upgrading

Deployment complexity
Diverse technology imposes many kinds of deployment procedures:
Diﬀerent operating systems, diﬀerent dependencies, many
variants

Deployment may need to be done on a large scale:

How to update the deployment frequently?
How not to break the system while upgrading?
How to minimize downtimes?
How to roll back in case of a failure?

Deployment automation
To deal with deployment complexities automation is needed!

Deployment automation
To deal with deployment complexities automation is needed!
Many automated deployment solutions available
Automation is typically driven by a specification
Some solutions have been developed for specific kinds of
technology:
Apache Felix (for OSGi components)
Some solutions are general:
Chef
Puppet
CFEngine
Nix
Some solutions use declarative deployment specifications

On being declarative in deployment
Declare what system you want to run in the consumer environment,
not the activities that need to be executed to accomplish it!

Chef: convergent declarative deployment
wordpress_latest = Chef::Config[:file_cache_path] + "/wordpress-latest.tar.gz"
remote_file wordpress_latest do
source "http://wordpress.org/latest.tar.gz"
mode "0644"
end
directory node["phpapp"]["path"] do
owner "root"
group "root"
mode "0755"
action :create
recursive true
end
execute "untar-wordpress" do
cwd node[’phpapp’][’path’]
command "tar --strip-components 1 -xzf " + wordpress_latest
creates node[’phpapp’][’path’] + "/wp-settings.php"
end
(Source: http://gettingstartedwithchef.com/first-steps-with-chef.html)

Chef: convergent declarative deployment
wordpress_latest = Chef::Config[:file_cache_path] + "/wordpress-latest.tar.gz"
remote_file wordpress_latest do
source "http://wordpress.org/latest.tar.gz"
mode "0644"
end
directory node["phpapp"]["path"] do
owner "root"
group "root"
mode "0755"
action :create
recursive true
end
execute "untar-wordpress" do
cwd node[’phpapp’][’path’]
command "tar --strip-components 1 -xzf " + wordpress_latest
creates node[’phpapp’][’path’] + "/wp-settings.php"
end
(Source: http://gettingstartedwithchef.com/first-steps-with-chef.html)
Declarative
The specification captures the outcome of a set of
changes as a fixpoint. Chef converges to the outcome.
Specification applies to set of machines – but does not
guarantee that an entire machine’s configuration can
be reproduced elsewhere
How to roll back to a previous configuration?
How to mimimize downtime?

NixOS
NixOS: A GNU/Linux distribution using the Nix package manager

NixOS conﬁguration
/etc/nixos/configuration.nix
{pkgs, ...}:
{
boot.loader.grub.device = "/dev/sda";
fileSystems = [ { mountPoint = "/"; device = "/dev/sda2"; } ];
swapDevices = [ { device = "/dev/sda1"; } ];
services = {
openssh.enable = true;
xserver = {
enable = true;
desktopManager.kde4.enable = true;
};
};
environment.systemPackages = [ pkgs.mc pkgs.firefox ];
}

NixOS configuration
nixos-rebuild switch
Nix package manager builds a complete system configuration
Includes all packages and generates all configuration files, e.g.
OpenSSH configuration
Upgrades are (almost) atomic
Components are stored safely next to each other, due to hashes
No files are automatically removed or overwritten
Users can switch to older generations of system configurations
not garbage collected yet

NixOS bootloader

Nix store
Main idea: store all packages
in isolation from each other:
/nix/store/rpdqxnilb0cg...
-firefox-3.5.4
Paths contain a 160-bit
cryptographic hash of all
inputs used to build the
package:
Sources
Libraries
Compilers
Build scripts
. . .
/nix/store
l9w6773m1msy...-openssh-4.6p1
bin
ssh
sbin
sshd
smkabrbibqv7...-openssl-0.9.8e
lib
libssl.so.0.9.8
c6jbqm2mc0a7...-zlib-1.2.3
lib
libz.so.1.2.3
im276akmsrhv...-glibc-2.5
lib
libc.so.6

Nix expressions
openssh.nix
{ stdenv, fetchurl, openssl, zlib }:
stdenv.mkDerivation {
name = "openssh-4.6p1";
src = fetchurl {
url = http://.../openssh-4.6p1.tar.gz;
sha256 = "0fpjlr3bfind0y94bk442x2p...";
};
buildCommand = ’’
tar xjf $src
./configure --prefix=$out --with-openssl=${openssl}
make; make install
’’;
}

Nix expressions
all-packages.nix
openssh = import ../tools/networking/openssh {
inherit fetchurl stdenv openssl zlib;
};
openssl = import ../development/libraries/openssl {
inherit fetchurl stdenv perl;
};
stdenv = ...;
openssl = ...;
zlib = ...;
perl = ...;
nix-env -f all-packages.nix -iA openssh
Produces a /nix/store/l9w6773m1msy...-openssh-4.6p1
package in the Nix store.

User environments
Users can have
different sets of
installed applications.
PATH
/nix/.../profiles
current
42
/nix/store
pp56i0a01si5...-user-env
bin
firefox
ssh
bin
ssh
rpdqxnilb0cg...-firefox-3.5.4
bin
firefox

User environments
Users can have
different sets of
nix-env operations
create new user
environments in the
store.
PATH
/nix/.../profiles
current
42
/nix/store
bin
firefox
ssh
bin
ssh
bin
firefox
aqn3wygq9jzk...-openssh-5.2p1
bin
ssh
(nix-env -u openssh)

User environments
Users can have
different sets of
nix-env operations
create new user
environments in the
store.
PATH
/nix/.../profiles
current
42
/nix/store
bin
firefox
ssh
bin
ssh
bin
firefox
bin
ssh
i3d9vh6d8ip1...-user-env
bin
ssh
firefox

User environments
Users can have
different sets of
nix-env operations
create new user
environments in the
store.
PATH
/nix/.../profiles
current
42
43
/nix/store
bin
firefox
ssh
bin
ssh
bin
firefox
bin
ssh
bin
ssh
firefox

User environments
Users can have
different sets of
nix-env operations
create new user
environments in the
store.
We can atomically
switch between them.
PATH
/nix/.../profiles
current
42
43
/nix/store
bin
firefox
ssh
bin
ssh
bin
firefox
bin
ssh
bin
ssh
firefox

User environments
Users can have
different sets of
nix-env operations
create new user
environments in the
store.
We can atomically
These are roots of the
garbage collector.
PATH
/nix/.../profiles
current
43
/nix/store
bin
firefox
ssh
bin
ssh
bin
firefox
bin
ssh
bin
ssh
firefox
(nix-env --remove-generations old)

User environments
Users can have
different sets of
nix-env operations
create new user
environments in the
store.
We can atomically
These are roots of the
garbage collector.
PATH
/nix/.../profiles
current
43
/nix/store
bin
firefox
bin
ssh
bin
ssh
firefox
(nix-collect-garbage)

NixOS
In NixOS, all packages including the Linux kernel and
conﬁguration ﬁles are managed by Nix.
NixOS does not have directories such as: /lib and /usr
NixOS has a minimal /bin and /etc

Distributed deployment
NixOS has good properties for deployment of a single system
Can we extend these properties to distributed systems?

Motivating example: Trac

Motivating example: Trac
Trac can be deployed in a distributed environment:
Subversion server
Database server
Web server

Distributed NixOS conﬁguration
network.nix
{ storage = {pkgs, ...}:
{
services.nfsKernel.server.enable = true; ...
};
postgresql = {pkgs, ...}:
{
services.postgresql.enable = true; ...
};
webserver = {pkgs, ...}:
{
fileSystems = [
{ mountPoint = "/repos"; device = "storage:/repos"; } ];
services.httpd.enable = true;
services.httpd.extraSubservices = [ { serviceType = "trac"; } ]; ...
};
...
}

Distributed deployment
$ nixops create network.nix -d production
$ nixops deploy -d production
Build system configurations by the Nix package manager
Transfer complete system and all dependencies to target
machines in the network
Efficient: only missing store paths must be transferred
Safe: Existing configuration is not affected, because no files
are overwritten or removed
Activate new system configuration
In case of a failure, roll back all configurations
Relatively cheap operation, because old configuration is stored
next to new configuration

The Nix project
Tools part of the Nix-project: http://nixos.org:
Nix. A purely functional package manager
NixOS. Nix based GNU/Linux distribution
Hydra. Nix based continuous build and integration server
Disnix. Nix based distributed service deployment
NixOps. NixOS-based multi-cloud deployment tool

The Nix project
Automated deployment using declarative speciﬁcations with the
following properties:
Generic. Can be used with many programming languages,
component technologies, and operating systems.
Reproducible. (Almost) no impurities – if inputs are the same,
result should be the same regardless of its location
Reliable. Dependency completeness, (almost) atomic
upgrades and rollbacks.
Eﬃcient. Only the required deployment activities are
executed.

Nix-related tools: how declarative are they?
Nix-related tools solve problems in a technical domain:
e.g. deployment of packages, machines, services, ...
What about your domain?

A real world example: Conference Compass
Conference Compass provides a service to improve the way
people experience events
Most visible part of the service: apps for conference attendees
Each customer basically gets “their own” app.

We have a product-line using a Nix-based build infrastructure,
including Hydra, driven by simple app speciﬁc conﬁgurations:
{
name = "wroclove.rb 2016";
homepage = "http://www.wrocloverb.com";
iconSet = ./icons;
backgroundImage" = ./background.png;
...
}

The app’s contents is customizable with a configurator service
allowing organizers to create and update their content
Apps connect to a configurator to retrieve the data to be
displayed and other configuration settings
Integration with third party information systems is also
possible

{
wrocloverb = {
eventName = "wroclove.rb 2016";
domain = "http://www.wrocloverb.com";
channels = [ "wrocloverb" ];
};
otherevent = ...;
yetanotherevent = ...;
...
}
We have developed a formalism to concisely model such
configurations and to automatically deploy them
Tool figures out which machines to configure, what services to
deploy etc.
If underlying implementation and technology evolves,
specifications (probably) remains the same.

Conclusions
I have illustated a declarative deployment vision
I have demonstrated NixOS and the Nix package manager
I have explained that domain speciﬁc deployment tools can be
built on top of tools from the Nix project

References
NixOS project homepage: http://nixos.org
Software available under free and open-source licenses
(LGPL/X11)
Nix package manager can be used on any Linux system, Mac
OS X, and (in some extent) Cygwin and FreeBSD.

Questions

The NixOS project and deploying systems declaratively

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a The NixOS project and deploying systems declaratively

Similar a The NixOS project and deploying systems declaratively (20)

Más de Sander van der Burg

Más de Sander van der Burg (13)

Último

Último (20)

The NixOS project and deploying systems declaratively