Bare-metal, Virtual Machines and Containers in

OpenStack
Charalampos Gavriil Kominos Nicolas Seyvet and Konstantinos Vandikas
Department of Computer Science Management and Operations of Complex Systems
Uppsala University Ericsson Research
Uppsala, Sweden Kista, Sweden
Email: h’lastname’@gmail.com Email: [email protected]

Abstract—Cloud computing is an on-demand access model for • Hybrid cloud: Which is a combination of the public and
computing resources most notably embodied by the OpenStack private models.
project. As of release Liberty, OpenStack supports provisioning
Bare-metal, Virtual machine (VM) and container based hosts. Generally it is assumed that the origins of Cloud computing
These different hosts incur different overheads. Consequently, the can be traced back to the 1960s when applications had to
main goal of this paper is to empirically quantify that overhead share system resources provided by mainframe computers.
through a series of experiments. IBM then invested a lot of time and effort in developing robust
The following drivers are leveraged in this process: Ironic for
time-sharing solutions to improve efficiency between users and
Bare-metal or Metal as a Service (MaaS), nova-compute for VM-
based hosts, and nova-docker for Docker based containers. expensive shared computer resources. Today, the best way to
We make use of a private-cloud in order to compare the improve resource utilization, and at the same time simplify
different options. This cloud is then used to compare the different data center management, is through virtualization.
hosts in terms of performance (CPU, networking, disk I/O and Virtualization refers to the act of creating a virtual (rather
RAM) by using various open-source benchmarking tools. We than actual) version of something. It abstracts the physical
also measure boot-up times. The output of these benchmarks is
collected and results are compared. hardware formed by clusters of servers into large aggregated
In this paper we discuss our learnings as well as the different pools of logical resources. This, in turn, can then be divided
configurations and fine-tuning that we implemented. As a result, and offered back to users in the form of VMs. Today, Infras-
we provide a set of recommendations based on the advantages tructure as a Service (IaaS) is largely synonymous with VMs.
and disadvantages of each host in present and future cloud Within IaaS, the OpenStack project is a well-known open
deployments.
source software. OpenStack begun in 2010 as a joint project
between Rackspace Hosting and Anso Labs (contracting for
I. I NTRODUCTION NASA). Its first release, code name ”Austin”, is launched in
There is no doubt that cloud computing brings an entire July 2010. Since then, hundreds of companies have pledged
new set of value propositions to enterprise computing en- support to the project, and, at the time when this paper is
vironments. Cloud computing offers a huge set of benefits written, there have been thirteen additional releases, the latest
such as application scalability, operational flexibility, improved being ”Mitaka”. The OpenStack mission is to produce the
economies of scale, reduced costs, resource efficiency, agility ubiquitous Open Source Cloud Computing platform that will
improvement and more. The National Institute of Standards meet the needs of both public and private clouds regardless
and Technology [1] defines Cloud as a model for enabling of size, by being simple to implement and massively scalable
ubiquitous, convenient, on-demand network access to a shared [2].
pool of configurable computing resources (e.g., networks, When deciding how to deploy an application in a private
servers, storage, applications, and services) that can be cloud using OpenStack a developer can provision three differ-
rapidly provisioned and released with minimal management ent hosts:
effort or service provider interaction. • Bare-metal: No virtualization, the hardware is fully ded-
Cloud computing is, at this date, based on four (4) main icated, delivered on demand.
deployment models [1]: • Virtual Machine: Traditional virtualization where the
• Private cloud: The cloud infrastructure is provisioned for machine appears as a self-contained computer, boots a
exclusive use by a single organization. standard OS kernel, and runs an unmodified application
• Community cloud: The cloud infrastructure is provi- process on top of a Hypervisor layer (see Figure 1).
sioned for exclusive use by a specific community of • Containers: A light-weight approach to isolating re-
consumers from organizations that have shared concerns. sources, where applications share a common kernel.
• Public cloud: The cloud infrastructure is provisioned for OpenStack supports these hosts through three different
open use by the general public. drivers/projects:
 • Ironic [3]: To provision bare-metal machines via PXE • Type two hypervisor: VMs lie on top of a physical
and Intelligent Platform Management Interface (IPMI) host which is running its own operating system (e.g.
aka Lights Out Management (LOM). VirtualBox).
• Nova-compute [4]: To provision VMs using KVM [5] as Nova-compute uses KVM by default; a type one hypervisor
a hypervisor within an OpenStack compute host. [9] and more specifically the KVM-QEMU pair. Nova-libvirt
• Nova-docker [6]: To provision Docker containers within ( [10] and [11]) driver manages the communication between
an OpenStack compute host. OpenStack and KVM.
The key focus of this paper is to evaluate the different While VMs excel at isolation, data exchanges between
hosts that can be provisioned by means of the aforementioned guests and hypervisors are slower. KVM uses hardware vir-
drivers in an empirical manner; by comparing their perfor- tualization features in the processors to limit overheads, and
mance in terms of CPU, Network, Disk I/O, memory and supports paravirtual devices via Virtio [12], a virtualization
boot-up times. standard for network and disk device drivers where only
In this research space we encounter similar work done by the guest’s device driver ”knows” it is running in a virtual
Felter et. al. [7]. The main difference between our work and environment, and cooperates with the hypervisor. Both features
theirs is that we place our emphasis on OpenStack and on tend to reduce virtualization overheads.
different kinds of hosts that can be provisioned as opposed Rather than running a full OS in a VM, containers achieve
to performing a straight-cut comparison of hypervisors vs similar process isolation by modifying a shared OS (operating-
containers. Moreover, in the context of OpenStack there is system-level virtualization). This eliminates the need for a
similar work done by Boden [8]. Our work improves on that Guest OS; applications run in their own isolated user space;
by extending that comparison with Ironic and a higher version a sandbox which is manifested via Linux kernel features such
of Docker, both of which were not available at the time when as namespaces and cgroups. The entirety of these user spaces
that comparison took place. is managed by the container engine which gains access to
physical resources through the host operating system. The
II. BACKGROUND AND RELATED WORK elimination of Guest OS in containers enables them to be more
lightweight and faster than VMs thus significantly reducing
In this paper, the OpenStack project is used as an IaaS boot time.
provider. OpenStack supports three fundamentally different The nova-docker driver spawns Docker [13] containers.
types of computing resources that can be requested by a user: Docker containers wrap a piece of software in a complete file-
bare-metal, VM and container based. These different options system that contains everything needed to run an application:
are illustrated in Figure 1. Code, runtime, system tools, system libraries anything that
Ironic (bare-metal) allocates the whole server hardware to can be installed on a server.
the load. Consequently, applications can run natively on the The remaining sections of the paper are structured as
host and fully utilize the underlying hardware. However, this follows: The Methodology section describes our setup and
is a single tenant option as unused hardware resources cannot the different configurations we have implemented in order to
be shared or re-used by others within the data center. As a provision the individual hosts. In addition, it describes the
result bare-metal tends to decrease overall utilization rates, different assumptions we made and the expectations we had
and is often not a cost-effective option. Additionally, from for each test. The Evaluation section presents and analyzes
an Ironic project perspective, special extensions are need to the different tests that have been performed for each kind of
allow access to hardware resources. Implementation of these host for the following resources: CPU, Network, Memory, disk
extensions limits the list of supported hardware and leads to I/O and booting times. Finally, the paper finishes off with the
increase the cost of developing and maintaining the OpenStack conclusions that can be derived from this work and a set of
Ironic driver. further steps that can be taken as part of Future Work.
The other two types of hosts addressed in this paper over-
come the aforementioned limitations by introducing a level III. M ETHODOLOGY
of abstraction by means of a hypervisor and the container
engine respectively. This addition comes at the expense of In this section we describe the hardware setup, software
performance. tools and configuration choices that have been made in our
The hypervisor spawns VMs, and presents the guest op- evaluation. Before we do that we start by giving more specific
erating system (OS) with a list of ideal virtual resources definitions to a set of terms that are used extensively through-
through which to manage all interactions with the physical out this paper.
resources. As a result, it achieves load isolation and hardware
independence from the guest OS perspective. A. Terminology
There are two types of hypervisors: Within the scope of this paper, the following terms are used:
• Type one hypervisor or bare-metal hypervisor: VMs lie • A Compute host or host is a node managed by OpenStack
directly on top of the hypervisor (e.g. ESXi, Xen, KVM) to instantiate a VM, or a container. It runs various
 Fig. 1. Different hosts in OpenStack

OpenStack services such as the Neutron agent, ceilome- whether the machine is powered on, or if an operating system
ter, a nova agent, openvSwitch [14], a messaging client is installed or functional.
(RabbitMQ), etc.
• An Ironic host is a bare-metal node managed by Open- C. IaaS software
Stack. The physical server is entirely allocated to the load We used the OpenStack Liberty release, and Docker v1.11.
and does not run any OpenStack services. Those were the latest available versions at the time.
• A docker container running within a compute host is
referred to as Docker. D. Benchmarking tools
• A VM running within a compute host is referred to as Our evaluation focuses on CPU, memory, disk I/O, network
VM. (latency and bandwidth) and boot-up times. To stress each
B. Hardware resource, the following open-source tools have been selected:
• CPU: the PXZ program has been selected. [16] This
Our setup consists of five servers located in the same rack.
tool, which is widely available in Ubuntu and Fedora,
Each physical server is a Hewlett Packard G6 blade equipped
implements the classic Lempel-Ziv-Markov compression
with two Intel(R) Xeon(R) CPU E5540, 72 GB of RAM and
algorithm [17]. PXZ is a parallel loss-less compression
two 160 GB 10000 rpm hard disks in RAID0 configuration.
tool and can be easily configured to run in any number of
The servers are connected via two NICs to a 1000/1000
cores. As input for PXZ a 1GB Wikipedia data dump has
network through an Top of Rack (ToR) Extreme Networks
been selected [18]. The file is fed into the PXZ algorithm
X450a-48t Switch. One of the servers is dedicated to be the
while varying the number of allocated cores. The wall
OpenStack controller and the four others are compute/ironic
time that PXZ takes to compress the file is then recorded.
hosts.
• Network: Network throughput and latency were tested
Note that during the evaluation phase, a particular server is
singled out to avoid variations. This is the server under test. In using two different tools:
cases where communication to a remote VM or container is – Nuttcp [19] is used to measure network throughput.
required, then a different server connected to the ToR is used This tool works in a client-server fashion. The system
in order to limit the number of network hops to one. to be tested acts as client and an external system in
Networking in OpenStack uses Neutron and VLANs. The the cloud acts as a server. Nuttcp traffic is generated
PXE network (NIC1, untagged) is required for Fuel to control between the client and server. The throughput is
and deploy the physical servers. The Public network (NIC1, a variable and tests are made between 650 and
1200) is used to distribute public IPs to VMs, containers and 1100 Mbps. We measure both inbound and outbound
hosts. Storage (NIC1, 1062) and management (NIC1, 1061) throughput.
are separate internal networks used by OpenStack manage- – Netperf [20] is used to measure network latency. It
ment. The ”private” tags (NIC2, 1063-1080) are used for also works in a client-server model although in this
isolating each virtual network. The bare-metal network (NIC2, case the machine to be tested acts as a server. The
1082) is reserved for Ironic provisioned instances. Standard client is then used to generate a packet of fixed length
tuning is applied for 1Gbps network. (200 bytes). When that packet reaches the server, the
Since the Ironic conductor achieves bare-metal provision- server sends back a response of fixed length (200
ing through Lights Out Management (LOM) protocol, the bytes). Therefore only one packet is in flight at any
servers are communicating via the Integrated Lights Out point in time. We measure how many packets are
(ILO) interface. ILO is HP’s version of LOM. LOM allows a sent within a fixed period of time.
system administrator to monitor and manage servers and other • Memory: In the tested system four memory levels are
network-attached equipment by remote control regardless of available (L1 cache, L2 cache, L3 cache, RAM). In order
 Fig. 2. Compute node networking using Neutron (VLAN) in Liberty release [15]

to test all these resources the open-source bandwidth [21] multi-layered Unification FileSystem, AUFS). Since our intent
tool was selected. In this tool memory chunks of different is not to compare AUFS vs VM block based storage, but to
sizes are copied into memory. The tool starts from 256 compare how different hosts affect the access to a resource,
bit and continues until the chunks are 512 MB in size. this approach permits mounting a data volume to access the
• Disk I/O: In order to test system disk I/O two different local Compute host disk and thus to bypass AUFS. Similarly
tools were used: the VM is using the ”mount host-file” option to access the local
– SysBench [22]. host hard drive. One of the caveats of using Docker directly
– Linux binary copy (dd) utility. was security issues with App Armor.
Disk IO tools operate by copying and reading from and For CPU measurements of KVM VMs, the CPU allocation
to big chunks of data. For such benchmarking, it is model is set to ”host-passthrough” [23], With this mode, the
important to prevent RAM involvement (buffer caches CPU is visible to the guest VM is exactly the same as the
at OS level). To achieve that: host physical machine CPU. By default, nova-compute does
not optimize KVM. Consequently, CPU pinning is not enabled.
– VMs and containers are limited to 8 GB of RAM.
– For Ironic and Host experiments, the physical server
is stripped to two DIMMs. IV. E VALUATION
– The benchmark files will have a size of 16 GB (i.e. The different experiments for each resource performed in
twice the size of the RAM). our evaluation are repeated ten times. Mean and standard de-
viation are then computed. The standard deviation is displayed
Tool Version Resource
ParallelPXZ 4.999.9beta CPU as error bar in the diagrams. When the error bar is not visible,
Nuttcp 6.1.2 Network assume the deviation to be less than 1%.
Netperf 2.6.0 Network
Bandwidth 1.3.1 Memory
The results for each experiment are presented and analyzed
SysBench 0.4.12 Disk in the following subsections.
dd 8.25 Disk
TABLE I
B ENCHMARKING TOOLS A. CPU
For CPU tests, four different OpenStack VMs are created
Table I summarizes the different tools used in our evalua- respectively with 1,2,4,8 vCPUs. We also leverage PXZ’s
tion. ability configure the number of cores. For example a VM with
2 vCPUs is compared with a bare-metal machine where PXZ
E. Configuration choices is limited to 2 cores. The results for the different configurations
For I/O tests instead of using the nova-docker driver, we are shown in Tables II and III.
spawned a standard Ubuntu 14.04 container directly from the From this evaluation the following observations can be
CLI. The reason is that in a standard container, disk access is made:
not to the device but to a copy-on-write file-system (Advanced • Bare-metal offers best performance.
 Resource 1 vCPU 2 vCPU
time (s) std % time (s) std %
Bare-metal 670.8 6.9 -2% 360.4 4.3 -7%
Compute host 683.8 2.8 0% 388.9 3.8 0%
Docker 685.2 3.9 0% 387.0 7.6 0%
VM 709.6 7.8 +3% 393.6 3.4 +1%
TABLE II
1-2 V CPU PERFORMANCE

Resource 4 vCPU 8 vCPU

time (s) std % time (s) std %
Bare-metal 192.0 0.8 -9% 109.5 1.0 -16%
Compute host 211.3 2.1 0% 131.4 2.1 0%
Container 211.7 2.5 0% 131.2 1.4 0%
VM 223.9 1.2 +5% 141.1 1.0 +7%
TABLE III
4-8 V CPU PERFORMANCE
Fig. 3. Network Bandwidth

• As highlighted by the difference between bare-metal increases, the delta between containers and VMs also increases
and Compute hosts, OpenStack services consume a non- in favor of containers.
negligible part of the CPU.
• The worst performer is the VM. This is in part due to C. Networking
the hypervisor as its main task is to schedule time in 1) Bandwidth: Nuttcp [19] uses a client/server approach
the physical CPU for the guests processes. For example, where the client is the system under test, and the server another
when the hypervisor requests 4 vCPU cores for a limited node within the cluster. It is configured to generate traffic at
amount of time, it must wait until these resources become different rates ranging from 500 to 1100 Mbps.
available. As expected, in our measurements, the link saturates just
• The Docker container performs on par with the compute below 1000 Mbps. This value is the expected maximum rate
host. of the network(switch and NIC) when considering packet
• Surprisingly, when the number of vCPUs allocated to headers.
a VM increases, the overall performance degrades. I.e. The following observations can be made:
while an improvement in overall computation time is
measured, the difference with running on bare-metal • Bare-metal based host is closer to maximum throughput
increases. The number of allocated vCPUs by the VM than other options,
is a major factor when trying to approximate hypervisor • All other options are quite close to bare-metal except the
overhead. Our tests indicate a performance drop of about Docker case.
2%, 5% and 7% when allocating respectively 1, 2, 4 • The Docker container based host did not perform as
and 8 vCPUs to a VM. The more CPU cycles that well in nearly every case despite several re-configuration
the hypervisor schedules for, the more the performance attempts.
drops percentage-wise. KVM CPU pinning could have A partial explanation to above results is that the OpenStack
improved the VM performance significantly as indicated internal communication being based on TCP as a protocol, a
by [8]. small part of the network traffic in all cases except Ironic is
used by the internal messaging system.
B. CPU contention In the Docker case, when taken out of the box a 50%
Contention is the situation where different processes com- drop in bandwidth was initially observed. It is only after
pete for a single resource. To assess this we created up to 4 lowering the Maximum Transmission Unit (MTU) within
VMs and Containers with 4 vCPUs/CPUs on a single server the container from 1500 to 1420 that a more reasonable
and then ran PXZ. performance was achieved. In our setup, both nova-docker and
Neutron cooperate to provide networking within the container.
# of instancesVM Container % diff During the instantiation, the container is assigned a networking
1 216s 203s 6.0%
2 252s 236s 6.3% namespace belonging the host that is connected to the tap port
3 367s 339s 7.6% provided by Neutron (see Figure 2). In our experiments, we
4 483s 445s 7.9% observed that the tap ports received traffic with an MTU set to
TABLE IV
CPU CONTENTION : PXZ EXECUTION TIME IN SECONDS 1420 while the nova-docker daemon expected 1500. This may
also be a consequences of issues within Docker 1.11( [24]).
This tuning of the container was only possible from within
As shown in Table IV, running within Docker containers is the host, and a normal OpenStack nova-docker user would
slightly faster than running within a hypervisor. As contention have been unable to manipulate the stack to reach such a
result which is a significant drawback. Further research and
experimentation is advised with newer Docker releases.
Resource TCP Reverse TCP UDP Reverse UDP
Bare-metal 923 929 941 931
Compute host 923 – 937 —
Container 899 327 932 939
VM 912 931 939 939
TABLE V
TCP & UDP MAXIMUM PERFORMANCE

Table V displays the findings for maximum throughput

achieved in TCP and UDP traffic in both directions. Reverse
TCP and UDP traffic were not measured successfully in the
time set and were omitted. Fig. 4. Memory access patterns: Random read
2) Latency: Netperf [20] was used for testing the system’s
latency. For this tool, the machine that is to be tested acts
as a server and another machine outside the cloud acts as
the client. In order to make a good approximation of system
latency we performed a fixed time test (30 sec). In this time
frame, the tool calculates how many packets were sent and
received. Therefore we can approximate the time it takes each
packet to reach its destination.
We use formula 1 from [25] to approximate system latency.
1 106 µs
∗
T ransf erRate(Req/s) 1s
(1)
2

Setup UDP UDPUDP TCP TCP TCP Fig. 5. Memory access patterns: Random writes
(req/s) stdlatency (req/s) std latency
(µs) (µs)
Bare-metal 1917 100 260 1944 102 257
Compute host 2016 59 250 1992 87 250 • Read performance does not show significant fluctuations
Container 1673 67 298 1724 93 289 regardless of the system tested.
VM 1380 64 362 1430 43 349 • RAM write performance in the VM shows a performance
TABLE VI
M EASURED TCP & UDP LATENCY drop of approximately 30%.
• Bare-metal has faster memory access than Compute host.
The OpenStack services are adding an overhead of at least
It can be observed from Table VI that while both Ironic and 5% for all tests.
the Compute host show similar performance results, the extra
networking layers added by Neutron (Figure 2) introduce some E. Disk I/O
40 µs latency both for Docker and VM. In the VM’s case, there Two different tools are used to stress disk I/O and measure
is an extra 60 µs inferred by the hypervisor bringing the total first number of requests/s, then MB/s.
to 100 µs when compared to a bare-metal option. 1) SysBench: We use Sysbench [22] to create a 16GB set
of files and an internal 70%/30% distribution of read/write to
D. Memory stress the I/O. Since Sysbench cannot force data to always
The Bandwidth tool [21] uses a random data batch of be stored on disk, the test length is extended to minimize the
variable size (from 256bit to 512 MB) to stress system effects of data caching in RAM. As such, we measured I/O
memory. It is used to test the random read and write speeds requests over a 20 minute period.
which are typical access types for applications. As indicated by Figure 7, While bare-metal, compute and
The Intel Xeon E5440 CPU used in our setup has L1 Docker achieve similar behavior, the VM handles only 35%
(data/instruction) cache of 32KB, L2 of 256KB, and L3 of of I/O requests.
8192KB. The different memory levels are clearly visible in 2) Binary Copy: A simple binary copy with the Linux’s dd
Figures (4, 5, 6). And, as expected, each level outperforms is done to create a 4.3GB file filed with zeros. The results are
the previous by a considerable factor. compiled in Table VII.
Figure 6 illustrates our data set partitioned in four data- Bare-metal and compute host exhibit similar performance.
clusters (256b, 48KB, 768KB, 512MB), one for each memory While Container experiences a 6% performance degradation
level. From this figure we observe: and VM performance drops by 30%.
 s MB/s std
Bare-metal 28.0 148 2.5
Compute host 28.7 149 1.4
Container 31.2 138 1.9
VM 82.0 104 2.4
TABLE VII
B INARY COPY PERFORMANCE

Boot-up time (s)

Bare-metal 600
Container 5
VM 105
TABLE VIII
B OOT- UP TIMES

Fig. 6. Random Read example

instantiation is quick, it does not imply that it is capable of
handling application load. For our measurements, we assume
that a computing resource is ready when it can be ssh-ed in
and not just pinged or visible in the horizon interface.
Note that the measured time includes transferring the image
from the Glance repository to the server, This impacts Bare-
Metal worse than VMs or Container as it is a mandatory step
during provisioning while other options can use host based
caching mechanisms on the compute.

V. C ONCLUSIONS
The main goal of this paper is to offer an evaluation of
different hosts that can be provisioned in OpenStack. Within
this scope, we have evaluated bare-metal hosts, VM based
Fig. 7. File Test I/O hosts and Docker container based hosts in terms of CPU,
Networking, Memory, Disk I/O and boot-up times. We also
It seems that in cases of full load, the KVM/QEMU hyper- evaluated the impacts of running OpenStack services on a
visor slows down the data transfer considerably. We speculate compute host.
that with further KVM tuning higher Input/Output(IO) speeds As expected, bare-metal outperforms all other options in our
can be achieved. However, as mentioned in [7] because each tests. As such it is recommended for high intensity workloads.
request must go through QEMU, the system will require more Deploying bare-metal via Ironic introduces no overhead and
IO Operations per second (IOPS) and CPU instructions per applications can run native on the hardware. However, the
second for the same absolute performance. This leaves fewer absence of overhead also means a lack of flexibility, and
resources available for useful application work. an overall lower utilization rate as no resources are shared.
For example, there is no network isolation as all bare-metal
F. Boot-up time instances share the same network address range. Note also that
To measure boot-up times, the OpenStack logs are used. while the Ironic project is a recent addition to OpenStack we
An instance is said to be available when it reaches the ”login” encountered no issues using it during our experimentation.
stage. Although this method is not overly accurate, it is a On the opposite-end in our evaluation we have VM based
good enough estimation of the time required for a resource to hosts. We observe that VM overhead varies and depends
become usable. on both the workload and the quantity of assigned virtual
Depending on the job execution length, the boot-up time resources. More specifically, our CPU tests indicate a strong
for a computing resource to become available to a user may correlation between the number of resources requested (vCPU)
or may not be relevant. If the job is going to run for days or and the measured overhead where performance degrades from
months then the above values are not important. But for short 1-2% (negligible) to 8-10% (important). Beyond CPU, VM
jobs (< 1-2 h), then the differences are significant enough to based hosts exhibit degraded performance when it comes to
favor container based setups. Memory and Disk I/O. VM networking performance is on par
While the Boden report [8] indicates boot-up times of 5s with the other alternatives.
and 3s respectively for VMs and containers, we did not see Docker container based hosts have the fastest boot-time
any optimization that would justify such improvements. We and an overall performance on par with bare-metal with the
assume that the authors used the time taken to allocate a one exception being networking bandwidth. This is a strong
resource, not the time where it becomes usable. While a VM indication that the nova-docker project is viable and allows
OpenStack to combine the speed of Containers with the ultra compact and as a result compete in performance with
flexibility and isolation guarantees of OpenStack’s Neutron containers.
based networking. There remains some restrictions to deploy
ACKNOWLEDGMENT
nova-docker at scale. First there is this issue with the MTU
setting in the container, then nova-docker specifies a hard- The authors would like to thank the following people from
coded space limit of 26GB per node for running containers. the IT group that helped us setup the hardware and network,
These two limitations make traditional VM-based hosts easier particularly Patrik Usher, Christopher Lange and Rajive Yadav,
to setup and maintain over the long term. as well as all the OpenStack developers present in in the IRC
Finally, a surprising result is the non-negligible CPU and channels #fuel, #nova-docker, #openstack-ironic, #openstack-
networking overheads introduced by the OpenStack services neutron for taking the time to answer all our questions.
running on all the compute nodes of the cluster. R EFERENCES
VI. F UTURE W ORK [1] “Final version of nist cloud computing definition published.”
[Online]. Available: https://www.nist.gov/news-events/news/2011/10/
Overall, OpenStack can be setup to control and deploy final-version-nist-cloud-computing-definition-published
different options from bare-metal, to containers, and VMs. [2] O. Sefraoui, M. Aissaoui, and M. Eleuldj, “Openstack: toward an open-
source solution for cloud computing,” International Journal of Computer
Such an environment presents an interesting solution to a Applications, vol. 55, no. 3, 2012.
complete data center deployment. [3] “Ironic project: Openstack bare metal provisioning program.” [Online].
In the scope of Future work it would be interesting to repeat Available: https://wiki.openstack.org/wiki/Ironic
[4] “Openstack nova.” [Online]. Available: https://wiki.openstack.org/wiki/
this evaluation and enable certain features and optimization Nova
that may yield better performance such as KVM CPU pinning [5] “Kvm.” [Online]. Available: http://www.linux-kvm.org/page/Main Page
with regards to CPU performance for VM-based hosts. In [6] “Docker driver for openstack nova.” [Online]. Available: https:
//github.com/openstack/nova-docker
addition it is important to further investigate the issues behind [7] W. Felter, A. Ferreira, R. Rajamony, and J. Rubio, “An updated
the poor networking performance in Docker despite our efforts performance comparison of virtual machines and linux containers,” in
for configuring the MTU. Another interesting dimension to in- Performance Analysis of Systems and Software (ISPASS), 2015 IEEE
International Symposium On. IEEE, 2015, pp. 171–172.
vestigate is that of the AUFS available in Docker which could [8] “Passive benchmarking with docker lxc, kvm & openstack v
also yield better Disk I/O performance in certain scenarios 2.0,” April 2014. [Online]. Available: http://www.slideshare.net/
since it is based on copy-on-write. BodenRussell/kvm-and-docker-lxc-benchmarking-with-openstack
[9] G. J. Popek and R. P. Goldberg, “Formal requirements for virtualizable
Moreover, there are efforts to integrate the advantages pro- third generation architectures,” ACM, vol. 17, no. 7, pp. 412–421, 1974.
vided by combining bare-metal and container technology. One [10] “Libvirt: The virtualization api.” [Online]. Available: http://libvirt.org/
such project named Kolla [26] provides production-ready con- [11] “Under the hood with nova, libvirt and kvm,” OpenStack
Summit, 2014. [Online]. Available: https://www.openstack.org/assets/
tainers and deployment tools for operating OpenStack. Kolla presentation-media/OSSummitAtlanta2014-NovaLibvirtKVM2.pdf
uses containers to instantiate and manage the deployment of [12] “Paravirtualized drivers for kvm/linux.” [Online]. Available: http:
OpenStack. The result is similar to a traditional IaaS where //www.linux-kvm.org/page/Virtio
[13] “Docker: Build, ship, run.” [Online]. Available: https://www.docker.com/
users can spawn VMs, bare-metal or containers. Another inter- [14] “Openvswitch.” [Online]. Available: http://openvswitch.org/
esting project is Magnum [27] that targets running container [15] “Welcome to openstack documentation.” [Online]. Available: http:
orchestration engines such as Docker Swarm, Kubernetes, and //docs.openstack.org/liberty/
[16] “Devstack project.” [Online]. Available: http://docs.openstack.org/
Apache Mesos as first class resources in OpenStack. Magnum developer/devstack/
uses Heat to orchestrate an OS image which contains Docker [17] “Lempelzivmarkov chain algorithm.” [Online]. Available: https://en.
and Kubernetes and runs that image in either VMs or bare wikipedia.org/wiki/LempelZivMarkov chain algorithm
[18] “Hutter.” [Online]. Available: https://cs.fit.edu/∼mmahoney/
metal in a cluster configuration. compression/textdata.html
Overall, we observe a considerable pressure to move away [19] “Nuttcp.” [Online]. Available: http://www.nuttcp.net/
from pure VM based clouds to more flexible computing envi- [20] “Netperf.” [Online]. Available: http://www.netperf.org/netperf/
[21] “Bandwidth: a memory bandwidth benchmark.” [Online]. Available:
ronments that make better use of available hardware resources, https://zsmith.co/bandwidth.html
and improve the revenues/server ratio for cloud providers. [22] “Sysbench.” [Online]. Available: https://github.com/akopytov/sysbench
In parallel, cloud computing continues to evolve with [23] “Hostpassthrough.” [Online]. Available: https://libvirt.or/formatdomain.
html\#elementsCPUAllocation
new approaches to application deployment. Two such newly [24] “containers in docker 1.11 does not get same mtu as host.” [Online].
emerging options are serverless computing and unikernels. Available: https://github.com/docker/docker/issues/22297
Serverless computing completely removes the need to care for [25] “Ibmnetperf.” [Online]. Available: http://www.ibm.com/support/
knowledgecenter/SSQPD3 2.4.0/com.ibm.wllm.doc/runnetpe
servers (including virtual ones) by adding another layer of ab- [26] “Welcome to kollas documentation!” [Online]. Available: http:
straction atop the cloud infrastructure. Running an application //docs.openstack.org/developer/kolla/
becomes only about purposefully built code, and a number of [27] “Magnum.” [Online]. Available: https://wiki.openstack.org/wiki/
Magnum
operations to perform i.e. cycles. Unikernels re-use the concept
of minimalistic operating systems introduced by Containers. A
unikernel is single-user, single-process, specialized operating
systems containing the full application stack. While deployed
on top an hypervisor, they benefit from being light and