VM boot sequence


Creating and booting new virtual machine is quire complex process. This document describes whole sequence in CoreCluster.

API Call /api/vm/create/

Whole process starts from here. The API call is authorized by CoreCluster's @register decorator. If credentials (user token) is valid, then the main function is executed. API function checks if given template and image IDs are valid, and if given template fits into user's quota:

def create(context, name, description,
           base_image_id = None):
    template = Template.get(template_id)
    base_image = None
    if base_image_id != None:
        base_image = Image.get(context.user.id, base_image_id)
        if base_image.state != 'ok':
            raise CMException('image_wrong_state')

Above code was simplified. For whole code, check it in repository, at GitHub. If credentials and requested resources are ok, then it performs creating VM object instance in database.

Adding hardware to libvirt template

Some aspects of VM’s hardware are defined by template and primary virtual drive (vCPU, memory and disk controller+disk contents). This is defined in Libvirt's VM template. Some other parameters are more sophisticated and it is necessary to define it outside the main XML. Such things are:

Thus, the basic xml template requires to be extended each time something is added or removed from virtual machine. To handle this, the Devices database model is used. Each instance has some entries in Devices table with above definitions. Adding new network card, disk or just mouse creates new entries in this table with XML descriptions of those devices for Libvirt. To create this XML descriptions, entries from config/templates/devices" are used.

Above mechanism allows to concatenate one libvirt template from many small parts (except the overal structure). Each time, when libvirt xml template is updated and new Devices entry is created, libvirt_redefine method should be called on VM instance (usually proper place for this action is Agent thread, not API!):

vm = VM.get(user_id, vm_id)

Virtual machine, which is updated should be in stopped state. This is especially important, because running virtual machines cannot be updated. Also machines, which weren’t created yet cannot be redefined (it doesn’t exist in libvirt). So, the stopped state of instance is a time, when agents could do any changes on instance.

After connecting all hardware, including external disks and network leases, virtual machine could be started.

VM-related tasks

If VM object has been successfully added to database, a few tasks are created and added into queues:

This tasks are stored in task queues and no more jobs is executed by API function. All further work is done by agents.

Preparing virtual drive

First task copies virtual machine from common storage into node’s hard drive. This operation takes long time and delays all next tasks until it is not finished. If this task fails, all other tasks related to such virtual machine or node are blocked. In some cases and configuration setups, node is blocked too.

VM’s hard drive image is being copied between two Libvirt storage pools - the storage's pool and images pool, at computing node. It is possible to use another Libvirt backend as storage pool or to replace whole node agent to use non-standard storages. Final result of this task is new libvirt volume in images pool. By default, this image is placed in /images directory with vm-id name.

The storage pool is usually mounted in /var/lib/CloudOver/[storage name]. This is valid until storage is handled by NFS libvirt pool. Definition of this resource is in config/templates/pools/netfs.xml.

The images pool is usually mounted in /images directory and is ordinary directory in Linux. Basic images of all virtual machines are stored in this pool to provide better performance of storage. Large I/O of multiple virtual machines kept in common storage might cause large traffic in your cluster's network and thus storage. To avoid this problem base image for each instance is copied to its node. The definition of Libvirt pool for images storage is in config/templates/pools/images.xml

Creating VM

Next task creates virtual machine in libvirt. During this operation, all devices are injected into libvirt’s xml. Once defined machine is ready and stopped until vm.start task is not executed.

Before start

After hard drive imvage has been transfered to node, all other tasks could be executed. This could be external disk or network lease attachment. Like before, any fail of those tasks block all next tasks. The exception are tasks with ignore_errors flag set to true.

After vm is created in libvirt (all tasks created by api call /api/vm/create/ are done), user could start virtual machine by calling /api/vm/start/. This api method creates new task: vm:start_vm.


The hook mechanism allows one to put any additional action before and after each task. Such actions are used by extensions, e.g. corevpn. You could find more about this mechanisms on hook reference missing article page.

VM start and shutdown

Instance start and shutdown are done via task, by libvirt call from Core to Node, however there is one more thing worth to mention. vm:start_vm task sets status of vm to starting. The final update of this status is always done by libvirt’s hook on node. Dedicated script connects from node to CoreCluster CI Api and notifies Core, that machine is running now.

By default CoreCluster uses only two scripts: to notify vm start and vm shutdown. Unless the first script could seem to be useless, the second is very important. Imagine user is shutting down VM from shell, by calling shutdown -h now command. Without hook nobody would know about that fact. This situation could be undesirable, because accounging plugins could still charge user for that machine as running (by status). Using hook subsystem prevents such situations.

The second purpose of hook scripts is to handle VM start/stop by cloud extensions. Some of them might require to provide some additional actions on task is started or finished. Such extension is for example CoreVPN, which has to close openvpn process on isolated network is removed.

VM cleanup

Api call /api/vm/cleanup/ executes the cleanup method on VM entity (database object). This function is used by administrator panel functions too. vm.cleanup method adds set of new tasks related to virtual machine. All those tasks in sequence remove attached hardware, deletes vm from libvirt and finally cleans up the base image on node.

< Go back     Author: Maciej Nabozny Published: Jan. 12, 2017, 3:07 p.m.