oVirt Architecture

Itamar Heim
Director, RHEV-M Engineering, Red Hat

oVirt Engine Architecture 1

oVirt Engine

Large scale, centralized

management for server and
desktop virtualization
Based on leading performance,
scalability and security
infrastructure technologies

oVirt Engine Architecture 2

Kenrel-based Virtual Machine (KVM)
● Included in Linux kernel since 2006

● Runs Linux, Windows and other

operating system guests

● Advanced features
● Live migration
● Memory page sharing
● Thin provisioning
● PCI Pass-through

● KVM architecture provides high

“feature-velocity” – leverages the
power of Linux

oVirt Engine Architecture 3

Linux as a Hypervisor?
● What makes up a hypervisor ?

● Hardware management
● Device drivers
● I/O Stack
● Resource Management
● Scheduling
● Access Control
● Power Management
● Memory Manager
● Device Model (emulation)
● Virtual Machine Monitor

oVirt Engine Architecture 4

Linux as a Hypervisor?

What makes up a hypervisor ?

}
●

● Hardware management
● Device drivers
● I/O Stack
● Resource Management Operating System Kernel
● Scheduling
● Access Control
● Power Management
● Memory Manager
● Device Model (emulation)
● Virtual Machine Monitor

oVirt Engine Architecture 5

Linux as a Hypervisor?

How well does Linux perform as a hypervisor?

Isn't Linux a general purpose operating system?

Linux is architected to scale from the smallest embedded

systems through to the largest multi-socket servers
● From cell phones through to mainframes

KVM benefits from mature, time tested infrastructure

● Powerful, scalable memory manager
● Robust security infrastructure
● High performance network stack
● Versatile storage infrastructure – iSCSI, FC, NAS, multipath, etc
● Rich ecosystem of supported hardware systems
oVirt Engine Architecture 6
Linux as a Hypervisor?

How well does Linux perform as a hypervisor?

Isn't Linux a general purpose operating system?

Over the last 4 years features have been added to Linux to

provide a
better infrastructure for a hypervisor
● Scheduler enhancements
Improved scalability and reduced latency
● Enhancements to memory manager
Advanced features such as memory page sharing and
compression
● Improvements to Block I/O subsystem
● Better performance, automated alignment, etc
oVirt Engine Architecture 7
Red Hat Enterprise Virtualization
Performance and Scalability
SPECvirt_sc2010
Vendor neutral virtualizaztion benchmarks
Comprised of application specific benchmarks running inside “tiles”
Each tile runs 6 virtual machines
● Application Server
● Database Server
● Mail Server
● Web Server
● Infrastructure Server
● Idle Server

Each VM runs a benchmark, eg SpecWeb, SPECjAppServer,SPECmail

and must meet specifi

oVirt Engine Architecture 8

Red Hat Enterprise Virtualization
Performance and Scalability
SPECvirt_sc2010

KVM leads the pack in 2, 4, 8 socket systems for SPECvirt

Including the largest benchmark results with over 400 Vms

Score : 7067 @ 432 VMs (72 tiles)

Processor: Intel Xeon E7-4870 (80 cores, 8 chips, 10 cores/chip, 2 threads/core)
Memory: 2 TB (128 x 16 GB, Quad Rank x4 PC3-8500 CL7 ECC DDR3 1066MHz LP RDIMM)

http://www.spec.org/virt_sc2010/

oVirt Engine Architecture 9

Red Hat Enterprise Virtualization
Competitive Landscape
● InfoWorld “shootout” 2011
– Independent analysis of
leading virtualization
platforms
– After <18 months Red Hat
has overtaken Citrix &
Microsoft in performance and
functionality

http://bit.ly/virtshootout

oVirt Engine Architecture 10

Security
oVirt inherits the security features of
Linux
SELinux security policy infrastructure
Provides protection and isolation
for virtual machines and host
Compromised virtual machine
cannot access other VMs or host
sVirt Project
Sub-project of NSA's SELinux
community. Provides “hardened”
hypervisors
Multilevel security. Isolate guests
Contain any hypervisor breaches

oVirt Engine Architecture 11

oVirt Node

● Standalone hypervisor
● Small footprint < 100MB
● Customized 'spin' of Fedora +
KVM
● 'Just enough' Fedora to run virtual
machines
● Runs on all RHEL hardware
with Intel VT/AMD-V CPUs
● Easy to install, configure and upgrade
● PXE boot, USB boot, CD or Hard
drive

oVirt Engine Architecture 12

oVirt Node vs. Full Host

oVirt Node
•
Less than 100 MB
•
Pre-configured, no Linux skills
needed.

Full Host
•
Flexible
•
Add monitoring agents, scripts
etc. Leverage existing Fedora
infrastructure.
•
Hybrid mode capable

oVirt Engine Architecture 13

Ovirt Engine

oVirt Engine Architecture 14

Management Features
Feature Description

High Availability Restart guest VMs from failed hosts automatically on other hosts

Live Migration Move running VM between hosts with zero downtime

System Scheduler Continuously load balance VMs based on resource usage/policies

Power Saver Concentrate virtual machines on fewer servers during off-peak hours
No downtime for virtual machines during planned maintenance windows.
Maintenance Manager Hypervisor patching

Image Management Template based provisioning, thin provisioning and snapshots

Monitoring & Reporting For all objects in system – VM guests, hosts, networking, storage etc.

OVF Import/Export Import and export VMs and templates using OVF files

V2V Convert VMs from VMware and RHEL/Xen to RHEV

oVirt Engine Architecture 15

High Availability

• Build a highly available enterprise

infrastructure
• Continually monitor host systems and
virtual machines
• Automatically restart virtual machines in
case of host failure
● Restart virtual machine on another
node in the cluster
• Use live migration to “fail-back” a VM to
it's original host when the server is
restored

oVirt Engine Architecture 16

Live Migration

• Dynamically move virtual machines

between hosts
● No service interruption
● Applications continue to run
• Migrate even I/O intensive workloads
such as databases
• Perform hardware maintenance
without application downtime
• Dynamically balance workloads
between host systems

oVirt Engine Architecture 17

System Scheduler

• Dynamically balance
workloads in the data center.
• Automatically live migrate
virtual machines based on
resources
• Define custom policies for
distribution of virtual machines

Maintain consistent resource usage across the

enterprise data center

oVirt Engine Architecture 18

Power Saver

Define policies to optimize workload on a fewer number of servers during “off-

peak” hours

oVirt Engine Architecture 19

Management Features
Feature Description

High Availability Restart guest VMs from failed hosts automatically on other hosts

Live Migration Move running VM between hosts with zero downtime

System Scheduler Continuously load balance VMs based on resource usage/policies

Power Saver Concentrate virtual machines on fewer servers during off-peak hours
No downtime for virtual machines during planned maintenance windows.
Maintenance Manager Hypervisor patching

Image Management Template based provisioning, thin provisioning and snapshots

Monitoring & Reporting For all objects in system – VM guests, hosts, networking, storage etc.

OVF Import/Export Import and export VMs and templates using OVF files

V2V Convert VMs from VMware and RHEL/Xen to RHEV

oVirt Engine Architecture 20

Virtual Desktop Infrastructure (VDI)

Centralized management,
security and policy enforcement
Virtual desktops with user
experience of a physical PC
Multiple monitors
HD quality video
Bi-directional audio/video for
VoIP or video-conferencing
Smartcard support
USB support
Industry leading density of virtual
desktops/server

oVirt Engine Architecture 21

Red Hat Enterprise Virtualization
RHEV 3.0 Key Initiatives
● Move from proprietary to open technologies
● Remove dependency on Windows
(But maintain interoperability with Windows)
● Deliver new features and releases in parallel
● Build Open Source community project around open
virtualization

oVirt Engine Architecture 22

History

● Qumranet
● KVM
● SPICE
● SolidICE --> RHEV-M C# --> RHEV-M Java --> oVirt
● C# --> Java
● using automatic conversion approach for core and UI[1]
● VDSM
● oVirt Node

[1] http://lpeer.blogspot.com/2010/04/switching-from-c-to-java.html
oVirt Engine Architecture 23
 Things have changed

Things have evolved

There are a lot of good ideas

There is a lot to refactor/change/do

This is where we are

Let's get to work...

oVirt Engine Architecture 24

Admin Portal

oVirt Engine Architecture 25

User Portal

oVirt Engine Architecture 26

Power User Portal - VM's

oVirt Engine Architecture 27

Add Virtual Machine

oVirt Engine Architecture 28

Power User Portal - Resources

oVirt Engine Architecture 29

oVirt High Level Architecture

Postgres
Postgres Admin
AdminPortal
Portal
gwt
gwt

AD
AD oVirt
oVirt RR
EE SDK/CLI
SDK/CLI
Engine
Engine
SS python
python
Java
Java
IPA
IPA TT
User
UserPortal
Portal
gwt
gwt

Guest
Guestagent
agent Guest
Guestagent
agent
SPICE
Shared
SharedStorage
Storage Linux
LinuxVM
VM Win
WinVM
VM Linux/Windows
Linux/Windows
FC/iSCSI/NFS
FC/iSCSI/NFS client
client
libvirt
libvirt
VDSM
VDSM
Host
Host| |Node
Node

Local
LocalStorage
Storage
oVirt Engine Architecture 30
Engine Core (Backend)

VMVM&&Template
TemplateLife
LifeCycle
Cycle Load
Load HA
HA
create,
create,schedule,
schedule,snapshot
snapshot Balancing
Balancing

Storage
Storage
Configuration
Configuration&&Monitoring
Monitoring

Network
Network
Configuration
Configuration&&Monitoring
Monitoring

Host
Host Host
Host Host
Host Host
Host
Register/Install
Register/Install Monitoring
Monitoring Maintenance
Maintenance Fencing
Fencing

Authentication,
Authentication,Authorization
Authorization Inventory
Inventory
Audit
Audit

oVirt Engine Architecture 31

Authentication

● Builtin user admin@internal

● AD, IPA integration
● Kerberos authentication
● LDAP - user info, group membership
● Multiple domains, trusts, etc.
● Cached for searches, not for login
● Next
● Open LDAP (patch ready)
● Internal users (picketlink?)
● Linux users?

oVirt Engine Architecture 32

Multi Level Admin

● Users
● Groups
● Roles
● Permissions

oVirt Engine Architecture 33

Multi Level Admin

● Users
● Groups
● Roles
● Permissions

oVirt Engine Architecture 34

Multi Level Admin

● Users
● Groups
● Roles
● Permissions

oVirt Engine Architecture 35

Multi Level Admin

● Users
● Groups
● Roles
● Permissions

oVirt Engine Architecture 36

Multi Level Admin

● Users
● Groups
● Roles
● Permissions

oVirt Engine Architecture 37

Multi Level Admin

● Users
● Groups
● Roles
● Permissions

oVirt Engine Architecture 38

Multi Level Admin

● Users
● Groups
● Roles
● Permissions

oVirt Engine Architecture 39

Database

● Moved from SQL Server to Postgres

● JDBC based
● Next
● Hibernate
● Scheme upgrade management

oVirt Engine Architecture 40

REST API

● New RESTful API for integration with oVirt Engine

● REST interface exposed for all API functions
● Developed in upstream RHEV-M API project (before oVirt)

oVirt Engine Architecture 41

RHEVM-API Upstream Project

Community project to deliver RESTful API for RHEV 2.2

https://fedorahosted.org/rhevm-api/

● Provides preview of 3.0 RESTful API

● Draft implementation of new API
● Runs on RHEL 2.2 wraps PowerShell
● Allows early testing of API for customers and partners
● 3.0 Implementation based on Java backend engine
● Will be consolidated into oVirt

oVirt Engine Architecture 42

oVirt Engine Architecture 43
RESTful Web Service

● Stands for Representational State Transfer

● Modeling entity actions around HTTP verbs
● GET
● PUT
● POST
● DELETE
● Still uses 'actions' for some state changes
● Self describes – entity navigation and actions

oVirt Engine Architecture 44

Welcome

oVirt Engine Architecture 45

Hosts Collection

oVirt Engine Architecture 46

Host networks collection

oVirt Engine Architecture 47

Create a Virtual Machine from a Template

POST http://10.35.1.1/rhevm-api/vms
<vm>
<name>my_new_vm</name>
<cluster id="99408929-82cf-4dc7-a532-9d998063fa95" />
<template id="00000000-0000-0000-0000-000000000000" />
</vm>

curl -v -u "[email protected]"
-H "Content-type: application/xml"
-d '<vm><name>my_new_vm</name><cluster id="99408929-82cf-4dc7-
a532-9d998063fa95" /><template id="00000000-0000-0000-0000-
000000000000"/></vm>'
'http://10.35.1.1/rhevm-api/vms'

oVirt Engine Architecture 48

Changing a property

PUT http://10.35.1.1/rhevm-api/vms/2496a177-e7c8-
4f82-bf3d-2d0f73444990
<vm>
<name>test_vm_new_name</name>
</vm>

echo "<vm><name>test_vm_new_name1</name></vm>" > /tmp/upload.xml

curl -v -u "[email protected]"
-H "Content-type: application/xml"
-T /tmp/upload.xml
'http://10.35.1.1/rhevm-api/vms/2496a177-e7c8-4f82-bf3d-2d0f73444990'

oVirt Engine Architecture 49

Adding a Virtual Disk

POST
http://10.35.1.1/rhevm-api/vms/2496a177-e7c8-4f82-bf3d-2
<disk>
<storage_domain id="3e1c96f0-8667-4a80-9689-af1337395dea" href="/rhevm-
api/storagedomains/3e1c96f0-8667-4a80-9689-af1337395dea" />
<size>1073741824</size>
<type>system</type>
<interface>virtio</interface>
<format>raw</format>
<sparse>true</sparse>
<bootable>true</bootable>
<wipe_after_delete>false</wipe_after_delete>
<propagate_errors>false</propagate_errors>
</disk>

● curl -v -u "[email protected]"
-H "Content-type: application/xml"
-d '<disk>...</disk>' http://...

oVirt Engine Architecture 50

What Else?

● Data warehouse
● Reports (based on jasperforge.org)
● Tools
● Notifications
● Config
● Iso uploader
● Log collector

oVirt Engine Architecture 51

oVirt Data Warehouse

● ETL based on talendforge.org

● Periodic polling from operational DB
● Types of data
● Config with version tracking
● Statistics – aggregated hourly/daily
● API is view based

oVirt Engine Architecture 52

Talend Studio

oVirt Engine Architecture 53

oVirt Reports

● Jasper allows to import/export reports definitions

● Rich reporting engine
● Report scheduling
● Filters
● Export to various formats
● Report creation studio
● Next
● Integrated in web admin

oVirt Engine Architecture 54

oVirt Reports

oVirt Engine Architecture 55

oVirt Reports

oVirt Engine Architecture 56

Notification Service

● oVirt allows registration to certain audit events

● The notification service sends emails per audit
message to relevant users
● Also monitors engine itself

oVirt Engine Architecture 57

Configuration tool

● The configuration utility allows changing oVirt

advanced configuration options
● Sample commands
● engine-config --list
● engine-config --get <key_name>
● engine-config -all
● engine-config --set <key_name>=<value>
● Special config for authentication domains:
manage-domains

oVirt Engine Architecture 58

ISO Uploader

● Iso uploader is a utility to upload iso files to the iso

domain, to allow bootstrapping guests from them
● Admin can just copy the files to the iso domain
● Supports both scp and nfs based copies
● Integrates with the REST API to allow using storage
domain name instead of specific NFS path

oVirt Engine Architecture 59

Log Collector

● The log collector utility helps collecting logs and

configuration data for troubleshooting
● Written as a linux script launching sos plugins
● Collects the data from engine and nodes

oVirt Engine Architecture 60

oVirt Guest Agent

● The guest agent provides additional information to

oVirt Engine, such as guest memory usage, guest ip
address, installed applications and sso.
● Python code, available for both linux and windows
guests
● Communication is done over virtio-serial
● SSO for windows is based on a gina module for XP
and a credential provider for windows 7
● SSO for RHEL 6 is based on a PAM module with
support for both KDE and Gnome

oVirt Engine Architecture 61

Guest

SSO
SSO

balloon
balloon Virtio-
Virtio- Virtio-
Virtio- USB
USB Spice
Spice guest
guest
net
net block
block driver
driver Agent
Agent

oVirt Engine Architecture 62

RHEV-M Guest Agent - SSO for RHEL

oVirt Engine Architecture 63

oVirt Host Agent - VDSM

● Covers all functionality required by oVirt Engine

● Configures host, networking and shared storage
● Uses libvirt for VM life cycle operations

oVirt Engine Architecture 64

oVirt Host Agent - VDSM

virto-serial

Guest
GuestAgent
Agent

QEMU/KVM
QEMU/KVM

libvirt
libvirt

hooks
hooks

Host
Host Storage
Storage Network
Network VM
VM Auto
Auto
Config
Config&& Config
Config&& Config
Config&& Config
Config&& Register
Register
Monitor
Monitor Monitor
Monitor Monitor
Monitor Monitor
Monitor

KSM
KSM

oVirt Engine Architecture 65

oVirt Storage


VDSM manages a Storage Pool, comprised of Storage Domains

Storage Pool - a VM repository that contains meta data about
storage domains, storage tasks, VMs, locks, etc.

Storage Domain - a disk image repository

Disk Image - a collection of volumes (chain of snapshots)

Volume - stored as files in NFS, and as Logical Volumes for
FC/iScsi

Thin provisioning for SAN supported (storage mailbox based)

oVirt Engine Architecture 66

Storage Pool Manager
The SPM runs on an arbitrary host (chosen by oVirt Engine)
oVirt Engine requires SPM to be running in order to add storage
If SPM host dies/disappears, RHEV-M causes SPM to start on a
different host

oVirt Engine Architecture 67

oVirt Storage “Clustering”


A Storage Pool is implemented as a managed cluster

Manager is oVirt Engine, running on a node external to
the hosts using the storage pool

Heartbeats and fencing are used in case of node failures

Storage based leased locks used as another layer of
protection

Clustering wise - VMs are mostly single reader/writer -
locks mostly needed to handle failures

Can easily create a cluster of >100 nodes

oVirt Engine Architecture 68

Hooks

● “Hook” mechanism for customization

● Allows administrator to define scripts to modify VM operation
● eg. Add extra options such as CPU pinning, watchdog device,
direct LUN access, etc
● Allows oVirt to be extended for new KVM features before full
integration is done
● An easy way to test a new kvm/libvirt/linux feature

oVirt Engine Architecture 69

Hooks

oVirt Engine Architecture 70

Hooks

oVirt Engine Architecture 71

Hooks
● Hook scripts are called at specific VM lifecycle events
● VDSM (management agent) Start
● Before VM start
● After VM start
● Before VM migration in/out
● After VM migration in/out
● Before and After VM Pause
● Before and After VM Continue
● Before and After VM Hibernate
● Before and After VM resume from hibernate
● On VM stop
● On VDSM Stop
● Hooks can modify a virtual machines XML definition before VM start
● Hooks can run system commands – eg. Apply firewall rule to VM

oVirt Engine Architecture 72

Hooks

oVirt Engine Architecture 73

Hooks
Hooks installed in /usr/libexec/vdsm/hooks

oVirt Engine Architecture 74

Hooks

oVirt Engine Architecture 75

Hooks

oVirt Engine Architecture 76

Sample Hooks
● CPU pinning ● Fileinject
● SR/IOV ● Floppy
● Smart card ● Hostusb
● Direct LUN ● Isolatedprivatevlan
● Hugepages ● Numa
● Promiscuous mode network interface ● Qos
● Cisco VN-Link ● Scratchpad
● smbios

oVirt Engine Architecture 77

On the Horizon - Infra

● Engine – JBoss AS 7, modular lighter engine

● Engine – custom hooks
● Engine – vdsm communication protocol and transport
● API – non admin api
● Reports – integrated in web admin
● Code cleanups, refactoring, unitests, etc

oVirt Engine Architecture 78

On the Horizon - Features

● Live snapshots ● Qbg/Qbh

● Live storage migration ● virt-resize, pv-resize
● Quotas ● Progress bars
● Hot plug ● Stable pci addresses
● Multiple storage domains ● Network types
● Shared disks ● Backup API
● iScsi disk ● SLA
● Shared file system ● SDM
support ● Many many more...
● Storage array integration
oVirt Engine Architecture 79
THANK YOU !

http://www.ovirt.org

oVirt Engine Architecture 80