Bareos, ZFS and Puppet: Christian Reiß Symgenius

Bareos, ZFS and Puppet
Christian Reiß
Symgenius
About Me
● Christian Reiß
● 37 years old
● System Administrator at Symgenius
2
About Me
● Linux since 1995'ish (Kernel 1.3)

● B* evangelist
● Love Puppet
3
Agenda
● Motivation and a little(!) history,

● Bacula (Alternatives),
● ZFS Introduction,
● The Puppet Module.
4
A Promise
● I’ll try to do an ‘easy listening’,

● No code snippets.
5
About Symgenius
● Symgenius
● Brand ‘Power-Netz.de’
● Mid sized hosting company,

● Usual web-stack,
● Project hosting and housing
6
Motivation – The situation
● We do webhosting.
● Several hundred Servers,

● Several hundred Customers each,
● Thousands of files per webhost,

● Thousands of files per Maildir,
● And of course: mysql.
7
Motivation - History
● We used to backup with tar,

● Full backups of root into a tar located
on a nfs share,
● Incrementals with a “find -ctime 1”.
● Everything piped through
compression, ie. pigz or pbzip2.
8
Motivation - History
● It did it’s job.
● Compression on client-side,
● Monitoring was a nightmare,
● But Restoring was hell.
9
Motivation - Alternatives
We tried several alternatives:
● BackupPC
● Rsnapshot
● Rsnapshot + ZFS Snaps
● Bacula
10
Motivation - Alternatives
● BackupPC and rsnapshot failed.

● Backup times were 13+ hours due to
amount of files,
● Job deletion even took longer.
● Backup + Job deletion exceeded 24
hours → Daily jobs piled up.
11
Enter Bacula
Bacula solved several issues:
● Backups are blazing fast even with

high file counts,
● Backups are written inside a single
archive volume,
● Restoring is bliss.
12
Enter Bacula
… But also:
● No heavy CPU load on client side

(think compression)
● You could do Backups during work
hours without Clients noticing (Well,
except for MySQL Dumps…).
13
Enter Bacula
Client issue is solved,

but what about the server side?
14
Enter Bacula
The Server needs:
● Protection against Hardware faults:

- RAID(-alike)
- Bit-Rot
● Other things:
- Transparent compression
- Quotas per Server
15
Enter Bacula
Also:
● Should be hassle- and maintenance

free (mostly).
● Needs to perform fast on ‘normal’
Hardware.
16
ZFS - Overview
All is solved with ZFS
17
ZFS - Overview
All is solved with ZFS

and Bareos.
18
ZFS - Features
The features of ZFS include:

● […] protection against data
corruption,
● continuous integrity checking and
automatic repair [...]
19
ZFS - Features

● […] RAID-Z [...]
20
ZFS - Features

● […] support for high storage
capacities, [...]
21
ZFS - Features

● […] efficient data compression, [...]
22
ZFS - Features

● […] snapshots and copy-on-write
clones, [...]
23
ZFS - Availability
Apart from Sun OS ZFS is available
● In Ubuntu 16.04 and up,

● In (future) Debian releases,
● And for all other Distributions via
zfsonlinux.org.
● FreeNAS using ZFS as main storage
24
ZFS - Install
● You install the package for your OS,

● Reboot,
● You do a “zpool create”,
● You might want to “zfs set” some
stuff,
● Done.
This does not count as ‘code snippet’.
25
ZFS – Live Example
Our SD (also Dir) consists out of:
● Intel Xeon CPU L5420 @ 2.50GHz

● 8 total cores
● 32gb Memory
● SSD for System and MySQL Data,
● 22x 4TB ‘NAS Grade’ WD Reds.
● (currently) 1Gb NIC
26
We backup 35 servers simultaneously.

Both the CPU and NIC are maxed.
That’s ~900Mbit of data transparently

compressed, RAID’ed and
checksummed right there.
With 8 cores.
27
ZFS – System Load
28
[root@bertha d125.powernetz.de]# pwd
/bacula/storage/d125.powernetz.de
[root@bertha d125.powernetz.de]# df h .
Filesystem Size Used Avail Use% Mounted on
bacula/d125.powernetz.de 4.0T 992G 3.1T 25%
/bacula/storage/d125.powernetz.de
29
[root@bertha d125.powernetz.de]# ls lhat
total 992G
drwxrx 268 bacula bacula 268 Jul 27 15:05 ..
rwr 1 bacula bacula 12G Jul 27 01:43 d125daily1420
rwr 1 bacula bacula 44G Jul 24 02:17 d125weekly1932
[...]
[...]
rwr 1 bacula bacula 362G Jul 11 05:10 d125monthly5778
rwr 1 bacula bacula 45G Jun 24 01:13 d125weekly2846
rwr 1 bacula tape 359G Jun 11 04:38 d125monthly3633
30
[root@bertha ~]# zfs get compressratio
NAME PROPERTY VALUE
[...]
bacula/d136.powernetz.de compressratio 1.79x
[...]
31
[root@bertha d125.powernetz.de]# du h d125monthly3633 apparentsize
359G d125monthly3633
[root@bertha d125.powernetz.de]# du h d125monthly3633
264G d125monthly
Net Save of 95GB, that’s ~30%!
32
On a sidenote:
Deduplication does not work.
33
ZFS - Advantages
● No more RAID controllers,

● Defense against Bit-Rot,
● Defense against Drive Faults*,
● Compression is done on Server,
● Quotas are possible,
● No hacks or workarounds.
● Everything is done transparently!
34
ZFS / Puppet
Puppet.
35
Puppet Module
The module does the usual:
● It configures the director,

● The storage daemon
● And of course the FD’s.
→ Backups work :)
36
Puppet
Each Server has:
● An independent Pool for Full Backups,

● An independent Pool for Differentials,
● An independent Pool for Incrementals.
In its own ZFS-Dataset.
37
Puppet
Each Server:
● Has limited bconsole shell access

(using ACL's)
● Has its own device in bacula-sd
● Is fully monitored by Icinga/Nagios
38
Puppet
● We wanted a fully automated System
● Non-destructive Installation
● Managing new Servers, adding ZFS

● Soft removal of retired Servers
39
Puppet Module
In addition to that:
● One-File-per-Job (Volume)
● Monitor last Backup,
● Monitor MySQL Dumps,
● We check for a valid full-backup,
● We check for /bacula-restores dir.
40
Puppet Module
… Also:
● Random Passwords generated

● bareos-fd.conf copy saved on dir
● Option to override IPs
41
Puppet Module
Set & override by hiera:
● Accurate flag (default yes),

● Exclude of files,
● Include of files,
● Onefs flag (default yes).
42
Puppet
43
Puppet Module
… and of course it will install the webui!
44
The End.
Thank you for your attention! \o/
45
The End.
As promised:
Bareos-Puppet Module available:

https://github.com/christianreiss/al_bareos
Contact me via:
[email protected]
Questions?
46

Bareos, ZFS and Puppet: Christian Reiß Symgenius

Uploaded by

Copyright:

Available Formats

Bareos, ZFS and Puppet: Christian Reiß Symgenius

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Bareos, ZFS and Puppet: Christian Reiß Symgenius

Uploaded by

Copyright:

Available Formats

Bareos, ZFS and Puppet

● Linux since 1995'ish (Kernel 1.3)

● Motivation and a little(!) history,

● I’ll try to do an ‘easy listening’,

● Mid sized hosting company,

● Several hundred Servers,

● Thousands of files per webhost,

● We used to backup with tar,

● It did it’s job.

We tried several alternatives:

● BackupPC and rsnapshot failed.

Bacula solved several issues:

● Backups are blazing fast even with

● No heavy CPU load on client side

Client issue is solved,

The Server needs:

● Protection against Hardware faults:

● Should be hassle- and maintenance

All is solved with ZFS

All is solved with ZFS

The features of ZFS include:

The features of ZFS include:

The features of ZFS include:

The features of ZFS include:

The features of ZFS include:

Apart from Sun OS ZFS is available

● In Ubuntu 16.04 and up,

● You install the package for your OS,

Our SD (also Dir) consists out of:

● Intel Xeon CPU L5420 @ 2.50GHz

We backup 35 servers simultaneously.

That’s ~900Mbit of data transparently

Deduplication does not work.

● No more RAID controllers,

The module does the usual:

● It configures the director,

Each Server has:

● An independent Pool for Full Backups,

In its own ZFS-Dataset.

● Has limited bconsole shell access

● We wanted a fully automated System

● Managing new Servers, adding ZFS

● Random Passwords generated

Set & override by hiera:

● Accurate flag (default yes),

… and of course it will install the webui!

Thank you for your attention! \o/

Bareos-Puppet Module available:

You might also like