1. Linux Disaster Recovery
Made Easy
Schlomo Schapiro Jeff Lindholm
Systems Architect / Open Source Evangelist Technology Sales Specialist – Linux
Immobilien Scout GmbH Novell, Inc.
schlomo.schapiro@immobilienscout24.de jlindholm@novell.com
License: http://creativecommons.org/licenses/by-nc-nd/3.0/

2. Agenda
Linux and disaster recovery
Solutions: relax and recover
Secure disaster recovery for enterprise environments
SLES 11 SP1 HA – ReaR availability and support
Live demo and use cases
Questions and answers
2 © Novell, Inc. All rights reserved.

3. Backup != Restore/Recovery
3 © Novell, Inc. All rights reserved.

4. Linux and Disaster Recovery

5. Disaster Recovery Strategies
Copy Files and Store Disk Disk Imaging
Layout and Boot Info
+ Online -> no problem + Offline -> no open
+ Backup independent of disk file issues
layout and sizes + Simple to restore
– More effort required to – Online -> very
restore (can be scripted) problematic
– Maybe consistency – No “perfect” open
problems, but should be source tool for
solved by backup solution Linux available
5 © Novell, Inc. All rights reserved.

6. Advantages of a Linux System
All information is stored in files; all files always readable
Operation system and applications can be “slim” -
600MB enough for complete standard server
Open system – open methods and procedures
All steps of an installation can be scripts:
partitioning, file systems, boot loader etc.
6 © Novell, Inc. All rights reserved.

7. Disaster Recovery – Media
• Most important: external storage!
• Bootable media: CD/DVD, USB key, LAN, tape ...
• Media usually combination boot and backup media:
– Bootable CD/DVD, USB key with backup data on it
– LAN boot (PXE) with backup data via CIFS, NFS ...
– Bootable tapes - HP OBDR (CD emulation)
• Separation between boot media and backup data
– Boot the system from a (small) USB key, CD/DVD or LAN
– Recover the system with backup software, tar, rsync ...
7 © Novell, Inc. All rights reserved.

8. Disaster Recovery – How It Works
• Store the disk layout
– Partitioning, LVM and RAID configuration
– File systems, file system labels ...
– Boot loader (GRUB, LILO, ELILO)
• Store the files (tgz, rsync, through backup software ...)
• Create bootable rescue media with system
configuration (and backup data)
• Can be done online
– No business interruption
– 100% compatible with original systems hard- and software
8 © Novell, Inc. All rights reserved.

9. Disaster Recovery – Rescue Media
• Create “rescue linux” from running system
• Optimally compatible “tool box”
• Clone the system environment
– Linux kernel and modules
– Device driver configuration
– Network configuration
– Basic system software and tools
• Operate entirely in RAM (initrd)
9 © Novell, Inc. All rights reserved.

10. Disaster Recovery – In Action
• Boot system from rescue media
• Restore disk layout
– Create partitions, RAID configuration and LVM
– Create file systems (mkfs, mkswap)
– Configure file systems (labels, mount points)
• Restore the backup data
• Restore the boot loader
• Reboot
• Done!
10 © Novell, Inc. All rights reserved.

11. Solutions: Relax and Recover

12. Proprietary Solutions
• Usually part of backup software
• Extra licenses and can be costly (up to 1000€ / server)
• Some are free, but are the any good?
• Level of integration into backup software varies
• Not all backup software actually supports
disaster recovery
• Cover only simple setups
• Often static boot media, inflexible with regard to drivers
• Certification on server hardware missing or lagging
behind procurement cycles
12 © Novell, Inc. All rights reserved.

13. Open Source Solutions
• Several software projects on the internet:
– Relax and recover
– mkCDrec
– Mondo Rescue
– CRU
– BACULA
– Your-Favorite-Selfmade-Script
– <something you know but I never heard about>
• In 2006, all of them where lacking something...
13 © Novell, Inc. All rights reserved.

14. Relax and Recover – ReaR
http://rear.sourceforge.net
• GPL Software – Developers in Germany and Belgium
• 100% Bash script – no GUI and no dependencies
• Utilize kernel, modules, binaries of host (kernel ≥ 2.6)
• Support any combination of SW/HW RAID, LVM
• Internal backup on CIFS, NFS ...
• Boot media on CD/DVD, USB key and LAN (PXE)
• Successor of mkCDrec
14 © Novell, Inc. All rights reserved.

15. ReaR – Features
• Focus on disaster recovery and not backup
• Tight integration with common backup software –
delegate file backup to backup infrastructure
• Simple full backup integrated
• Complements backup software:
– Backup software: Data storage and retrieval
– ReaR: Recover system layout and make it work again
– ReaR utilizes the backup software to restore the backup data
• Use the best tool for the job
15 © Novell, Inc. All rights reserved.

16. ReaR – Backup Software
• Supported solutions include:
– CommVault Galaxy
– IBM Tivoli Storage Manager
– Veritas NetBackup
– HP Data Protector
– Rsync and other “external” methods
– tar.gz archive on NAS share – CIFS, NFS, NCP ...
• Very transparent integration
• Quick implementation: 2-3 PT programming
• Can be easily extended to support other vendors
16 © Novell, Inc. All rights reserved.

17. ReaR – Network Integration
• Disaster recovery as part of network infrastructure
– Backup software – file-level backup storage
– ReaR – system environment
– Boot rescue media via PXE – no physical media required
– Very scalable – automated installation of entire disaster recovery
data center
ReaR
Boot Files Software
Backup
PXE Boot
Backup
Network Storage
infrastructure
17 © Novell, Inc. All rights reserved.

18. ReaR – Status
• Stable software
– i386 and x86_64 well tested
– ia64 and ppc experimental
• Regular releases (RPM, DEB, TGZ)
• Major Linux distributions ship ReaR:
– SUSE Linux Enterprise High Availability Extension 11 SP1
®
– openSUSE 11.2 and Fedora 11
®
• Community and commercial support available
• Regular patch submissions from ReaR users
18 © Novell, Inc. All rights reserved.

19. ReaR - Development
• Open Source development model:
– Submit patches and feedback – “field testing”
– Sponsoring
• Modular design:
– ReaR is a framework to plug together many small bash scripts
– Maximize code reusability
– Simple development model (vi works fine)
– Little to no “interferences” between different areas of code
• Documentation on project homepage
– Hierarchical structure
– References to source code
19 © Novell, Inc. All rights reserved.

20. ReaR – Configuration
• Shell-script style configuration file
• Usually: Set backup software and boot media
System definition:
ARCH = Linux-x86_64
OS = GNU/Linux
OS_VENDOR = SUSE_LINUX
OS_VERSION = 11
OS_VENDOR_ARCH = SUSE_LINUX/x86_64
OS_VENDOR_VERSION = SUSE_LINUX/11
OS_VENDOR_VERSION_ARCH = SUSE_LINUX/11/x86_64
Configuration tree:
Linux-x86_64.conf : OK
GNU/Linux.conf : OK
SUSE_LINUX.conf : missing/empty
SUSE_LINUX/x86_64.conf : missing/empty
SUSE_LINUX/11.conf : missing/empty
SUSE_LINUX/11/x86_64.conf : missing/empty
site.conf : OK
local.conf : OK
20 © Novell, Inc. All rights reserved.

21. ReaR – Operation
rear [Options] <command> [command options ...]
Relax & Recover Version 1.7.23.2 / 2010-03-19
Build: b80d32a97e41a6ec8a5b19cd6399e444
Copyright (C) 2006-2009
Schlomo Schapiro
Gratien D'haese, IT3 Consultants
Relax & Recover comes with ABSOLUTELY NO WARRANTY; for details
see the GNU General Public License at
http://www.gnu.org/licenses/gpl.html
Available Options:
-V version information
-d debug mode
-D debugscript mode
-S Step-by-step mode
-s Simulation mode (shows the scripts
included)
-q Quiet mode
-r a.b.c-xx-yy kernel version to use (current:
2.6.32.8-0.3-default)
21 © Novell, Inc. All rights reserved.

22. ReaR – Operation (continued)
List of commands:
dump Dump configuration and system information
help print out usage
mkbackup Create rescue media and backup system.
mkbackuponly Backup system without creating a (new)
rescue media.
mkdeb Create DEB packages with this rear version
mkdist Create distribution tar archive with this
rear version
mkrescue Create rescue media only
mkrpm Create RPM packages with this rear version
mktar Create tar archive with this rear
installation
mkvendorrpm Create vendor RPM with this rear version
recover Recover the system
validate Submit validation information
22 © Novell, Inc. All rights reserved.

23. ReaR – Example Configuration
# cat /etc/rear/local.conf
OUTPUT=ISO
#BACKUP=TSM|NBU|DP|GALAXY|EXTERNAL|...
BACKUP=NETFS
NETFS_URL=nfs://server.domain/backup-path
MODULES_LOAD=( vmxnet )
23 © Novell, Inc. All rights reserved.

24. ReaR – Example Execution
# rear mkbackup
Relax & Recover Version 1.7.23.2 / 2010-03-19
The preparation phase OK
Physical devices that will be recovered: /dev/sda /dev/sdb
Creating root FS layout OK
Copy files and directories OK
Copy program files & libraries OK
Copy kernel modules OK
Create initramfs OK
Creating archive 'nfs://server.domain/backup-path/rear/backup.tar.gz'
Transferred 297 MB in 254 seconds [1197 KB/sec]
Making ISO image OK
Wrote ISO Image /tmp/ReaR.iso (19M)
Copying resulting files to network location OK
The cleanup phase OK
Finished in 258 seconds.
24 © Novell, Inc. All rights reserved.

25. ReaR – Disk Layout
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/system-root
2,5G 927M 1,4G 40% /
devtmpfs 244M 128K 244M 1% /dev
tmpfs 247M 0 247M 0% /dev/shm
/dev/sda1 84M 19M 61M 24% /boot
/dev/mapper/system-home
524M 17M 481M 4% /home
/dev/md0 680M 55M 591M 9% /var
25 © Novell, Inc. All rights reserved.

26. ReaR – Disk Layout
# pvs
PV VG Fmt Attr PSize PFree
/dev/sda3 system lvm2 a- 3,21G 0
/dev/sdb2 system lvm2 a- 312,00M 0
# lvs
LV VG Attr LSize Origin Snap%
home system -wi-ao 532,00M
root system -wi-ao 2,44G
swap system -wi-ao 572,00M
# cat /proc/mdstat
Personalities : [linear] [raid1]
md0 : active raid1 sda2[0] sdb1[1]
706816 blocks super 1.0 [2/2] [UU]
26 © Novell, Inc. All rights reserved.

27. ReaR – Recovery Example
# rear recover
Relax & Recover Version 1.7.23.2 / 2010-03-19
Backup archive size is 298M
Initializing physical devices OK
Creating Software RAID devices OK
Creating the LVM2 devices OK
Creating file systems OK
Recreated this filesystem layout:
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/system-root
2.5G 68M 2.3G 3% /mnt/local
/dev/sda1 84M 5.6M 74M 7% /mnt/local/boot
/dev/mapper/system-home
524M 17M 481M 4% /mnt/local/home
/dev/md0 680M 17M 629M 3% /mnt/local/var
Restoring from 'nfs://laptop/dev/shm/nfs/rear/backup.tar.gz' OK
27 © Novell, Inc. All rights reserved.

28. ReaR – Recovery Example (continued)
Restoring the bootloader (SuSE style)
Creating swap files and partitions OK
Finished recovering your system
You can explore it under /mnt/local
Thank you for using Relax & Recover Version
1.7.23.2
Finished in 150 seconds.
28 © Novell, Inc. All rights reserved.

29. ReaR – Vision
• Do more than just disaster recovery
– System cloning and provisioning
– System migration (P2V, V2P, V2V, P2P)
– Generic system tool-kit for offline tasks and rescue media
• Centralized information and management:
– rear-server
– 1.0: Gather and aggregate information about ReaR
> System protection status
> Rescue media boot files
– 2.0: Centrally manage configuration and control ReaR
> Site-wide and system-local configuration store
> Trigger and control disaster recovery tasks
– 3.0: Integrate into 3rd party solutions
29 © Novell, Inc. All rights reserved.

30. Secure Disaster Recovery
Enterprise Environments

31. Enterprise DR – Requirements
Standardization
Scalability
Automation
Integration in existing system environment
Security
Coverage over all systems in use
Support and maintenance
Education and certification
Affordable licensing – site license
31 © Novell, Inc. All rights reserved.

32. Enterprise DR – Relax and Recover
• Standardization through global configuration:
– /etc/rear/site.conf
• Scalability and automation:
– rear mkrescue can be run as a cron job
– Automated processing of rescue boot media fairly simple
– Rescue boot media also stored with backup software
– LAN boot – extreme scalability and automated DR possible
• Integration in existing system environment
– Support internal and external backup solutions
– E-mail reports
32 © Novell, Inc. All rights reserved.

33. Enterprise DR – Relax and Recover
• Security considerations
– No “secret” data in rescue media
– Secure export of rescue media through backup software
and E-mail
– Remote access to rescue system only with SSH keys
• Coverage
– i386 and x86_64 stable
– Enterprise Linux (Novell , Red Hat, Ubuntu) stable
®
– Community distributions also work, less tested
• Maintenance and support, education
– Please contact Novell and the authors
• Site license: Included with GPL
33 © Novell, Inc. All rights reserved.

34. Security Aspects – Disaster Recovery
Backup server DR boot server
DR client
Transfer of Backup Data Transfer of Boot Riles
• How secure is the • Insecure (anonymous):
backup software? – NFS
• Insecure: NFS2, NFS3 – CIFS
• Secure (with – SSH (keys)
authentication): – RCP
– NFS4, CIFS • Manipulation proof:
– SSH (rsync) – SMTP
34 © Novell, Inc. All rights reserved.

35. Enterprise Linux Disaster Recovery
• Linux – Simple Disaster Recovery
• Integration: Disaster recovery – backup software
• Automated creation of disaster recovery information
• Network boot into rescue system
• Disaster recovery becomes part of infrastructure
• Scales well even for very large data centers
• Disaster recovery becomes a valid option for clients,
not only for servers
35 © Novell, Inc. All rights reserved.

36. SUSE Linux Enterprise High Availability
®
Extension 11 Service Pack 11 –
ReaR Availability and Support

37. SUSE Linux Enterprise
®
Modular Portfolio
SUSE Linux Enterprise
Common Code Level 3 Support Runs on x86,
Base foundation x86_64, Itanium,
Power, and s390x
SUSE Linux SLES for SAP,
Enterprise SLED JeOS SLE SDK SLE POS Expanded
Server Support
Server Desktop Custom Developer Bundles Support
OS OS OS Tools SKUs
High Availability Real Time Extension Mono Extension
Optional extensions for high availability clustering, Pay only for
deterministic computing, and running .NET applications what you need
37 © Novell, Inc. All rights reserved.

38. SUSE Linux Enterprise High Availability Extension 11
®
Key Features
• Flexible, policy driven • Continuous data
clustering solution replication
– OpenAIS messaging and – Distributed Replicated Block
membership layer Device (DRBD8)
– Pacemaker cluster
resource manager • GUI and CLI tools
– Graphical user interface
• Cluster-aware file system – Unified command line interface
and volume manager
– Oracle Cluster File • IP load balancing
System (OCFS2)
– Clustered Logical Volume
• Resource agents
Manager (cLVM2)
• Bare Metal Recovery (ReaR)
• Virtualization aware
38 © Novell, Inc. All rights reserved.

39. SUSE Linux Enterprise High Availability Extension 11
®
Differentiators
• Affordable
Costs less than other Linux clustering solutions, with tools
and resource agents included at no extra charge
• Virtualization-aware
Supports hybrid virtual and physical clusters
• Open Source
No proprietary modules or user space applications means your
Linux clustering solution is secure and easily kept up to date
• Integrated
Streamlined support and an optimized solution from a single
vendor that delivers both the OS and clustering software
39 © Novell, Inc. All rights reserved.

40. SUSE Linux Enterprise High Availability Extension 11
®
New Features/Enhancements SP1
ReaR – Relax and Recover
• ReaR will be included with SUSE Linux Entreprise
Server 11 SP1 HA Add-on
• ReaR is considered to be an extensible framework, allowing
for custom scripting and extension of the recovery toolset
• ReaR RPM packages will be supported on x86 and x86_64
• SLE11 High Availability Add-on inherits service level
agreement from the SLES Subscription
– Priority (24x7)
– Standard (12x5)
• Customization is not covered by Novell Technical Support
®
– Integration Partners are available (contact Novell Sales)
40 © Novell, Inc. All rights reserved.
•

41. SUSE Linux Enterprise High Availability Extension 11
®
More Information - SP1
For more information on what is new with SLE11 HA SP1:
ELS307 Building High Availability Clusters with SUSE
Linux Enterprise High Availability Extension
ELS302 SUSE Linux Enterprise High Availability
Extension 11: Support and Troubleshooting
ATT307 Quickly Configure a High Availability Cluster with
SUSE Linux Enterprise 11 High Availability Extension
41 © Novell, Inc. All rights reserved.

42. Live Demo/Use Cases

43. Relax and Recover – Live Demo
Demo (~10 min):
• SUSE Linux ®
Enterprise High
Availability Extension
11 SP1
• ReaR creates backup
and boot CD
• “PEBKAC” at work
• Boot system from CD
• Recover system
• Happy End
43 © Novell, Inc. All rights reserved.

44. Questions and Answers

46. Unpublished Work of Novell, Inc. All Rights Reserved.
This work is an unpublished work and contains confidential, proprietary, and trade secret information of Novell, Inc.
Access to this work is restricted to Novell employees who have a need to know to perform tasks within the scope
of their assignments. No part of this work may be practiced, performed, copied, distributed, revised, modified,
translated, abridged, condensed, expanded, collected, or adapted without the prior written consent of Novell, Inc.
Any use or exploitation of this work without authorization could subject the perpetrator to criminal and civil liability.
General Disclaimer
This document is not to be construed as a promise by any participating company to develop, deliver, or market a
product. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in
making purchasing decisions. Novell, Inc. makes no representations or warranties with respect to the contents
of this document, and specifically disclaims any express or implied warranties of merchantability or fitness for any
particular purpose. The development, release, and timing of features or functionality described for Novell products
remains at the sole discretion of Novell. Further, Novell, Inc. reserves the right to revise this document and to
make changes to its content, at any time, without obligation to notify any person or entity of such revisions or
changes. All Novell marks referenced in this presentation are trademarks or registered trademarks of Novell, Inc.
in the United States and other countries. All third-party trademarks are the property of their respective owners.