VM Monitoring
The purpose of this spec is to describe a method for monitoring the health of the VMs without access to the VMs's internals. Change-Id: I82ccb4ae64f48ca154c5450641ed41e04fee9d17
This commit is contained in:
sampathP
Adam Spiers
parent
b61b7b70ef
commit
468d526263
295
specs/newton/approved/newton-instance-ha-vm-monitoring-spec.rst
Normal file
295
specs/newton/approved/newton-instance-ha-vm-monitoring-spec.rst
Normal file
@ -0,0 +1,295 @@
|
||||
..
|
||||
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||
License.
|
||||
|
||||
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
|
||||
==========================================
|
||||
VM Monitoring
|
||||
==========================================
|
||||
|
||||
The purpose of this spec is to describe a method for monitoring the
|
||||
health of OpenStack VM instances without access to the VMs' internals.
|
||||
|
||||
Problem description
|
||||
===================
|
||||
|
||||
Monitoring VM health is essential for providing high availability for
|
||||
the VMs. Typically cloud operators cannot access inside VMs in order
|
||||
to monitor their health, because this would violate the contract
|
||||
between cloud operators and users that users have complete autonomy
|
||||
over the contents of their VMs and all actions are performed inside
|
||||
them. Operators cannot assume any knowledge of the software stack
|
||||
inside the VM or make any changes to it. Therefore, VM health
|
||||
monitoring must be done externally. This VM monitor must be able to
|
||||
detect VM crashes, hangs (e.g. due to I/O errors) and so on.
|
||||
|
||||
Use Cases
|
||||
---------
|
||||
|
||||
As a cloud operator, I would like to provide my users with highly
|
||||
available VMs to meet high SLA requirements. Therefore, I need my VMs
|
||||
automatically monitored for sudden stops, crashes, IO failures and
|
||||
similar. Any VM failure event detected needs to be passed to a VM
|
||||
recovery workflow service which takes the appropriate actions to
|
||||
recover the VM. For example:
|
||||
|
||||
- If a VM crashes, the recovery service will try to restart it,
|
||||
possibly on the same host at first, and then on a different host if
|
||||
it fails to restart or if it restarts successfully but then crashes
|
||||
a second time on the original host.
|
||||
|
||||
- If a VM receives an I/O error, the recovery service may prefer to
|
||||
immediately contact ``nova-api`` to centrally disable the
|
||||
``nova-compute`` service on that host (so that no new VMs are
|
||||
scheduled on the host) and restart the VM on a different host. It
|
||||
could also potentially live-migrate all other VMs off that host, in
|
||||
order to pre-empt an further I/O errors.
|
||||
|
||||
Proposed change
|
||||
===============
|
||||
|
||||
VM monitoring can be done at the hypervisor level without accessing
|
||||
inside the VMs. In particular, |libvirt|_ provides a mechanism for
|
||||
monitoring its event stream via an event loop. We need to filter the
|
||||
required events and pass them to a recovery workflow service. In
|
||||
order to eliminate redundancy and improve extensibility, these event
|
||||
filters must be configurable.
|
||||
|
||||
.. |libvirt| replace:: `libvirt`
|
||||
.. _libvirt: https://libvirt.org/
|
||||
|
||||
Potential advantages:
|
||||
|
||||
- Catching events at their source (the hypervisor layer) means that we
|
||||
don't have to rely on ``nova`` having knowledge of those events.
|
||||
For example, ``libvirtd`` can output errors when a VM's I/O layer
|
||||
encounters issues, but ``nova`` doesn't emit corresponding events for
|
||||
this.
|
||||
- It should be relatively easy to support a configurable event filter.
|
||||
- The VM instance monitor can be run on each compute node, so it should
|
||||
scale well as the number of compute nodes increases.
|
||||
- The VM instance monitors could be managed by `pacemaker_remote`__ via a
|
||||
new `OCF RA (resource agent)`__.
|
||||
|
||||
__ http://clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Pacemaker_Remote/
|
||||
__ http://www.linux-ha.org/wiki/OCF_Resource_Agents
|
||||
|
||||
Alternatives
|
||||
------------
|
||||
|
||||
There are three alternatives to the proposed change:
|
||||
|
||||
1. Listen for VM status change events on message queue.
|
||||
|
||||
Potential disadvantages:
|
||||
|
||||
- It might be less reliable, if for some reason the
|
||||
message queue introduced latency or was lossy.
|
||||
|
||||
- There also might be some gaps in which events are propagated to
|
||||
the queue; if so, we could submit a ``nova`` spec to plug the gaps.
|
||||
|
||||
- If we listen for events from the control plane, it won't scale as
|
||||
well to large numbers of compute nodes, and then would be awkward
|
||||
to trigger recovery via Pacemaker.
|
||||
|
||||
2. Write a new ``nova-libvirt`` OCF RA
|
||||
|
||||
It would compare ``nova``'s expectations of which VMs should be running
|
||||
on the compute node with the reality. Any differences between the
|
||||
two would send appropriate failure events to the recovery workflow
|
||||
service.
|
||||
|
||||
Potential disadvantages:
|
||||
|
||||
- This is more complexity than is expected to run inside an RA.
|
||||
RAs are supposed to be lightweight components which simply start,
|
||||
stop, and monitor services, whereas this would require abusing
|
||||
that model by pretending there is a separate monitoring service
|
||||
when there isn't. The ``monitor`` action would need to fail when
|
||||
any differences as mentioned above were detected, and then the
|
||||
``stop`` or ``start`` action would need to send the failure
|
||||
events.
|
||||
|
||||
- Within this "fake service" model, it's not clear how to avoid
|
||||
sending the same failure events over and over again until the
|
||||
failures were corrected.
|
||||
|
||||
- Typically RAs are implemented in ``bash``. This is not a hard
|
||||
requirement, but something of this complexity would be much
|
||||
better coded in Python, resulting in either a mix of languages
|
||||
within the `openstack-resource-agents`_ repository
|
||||
|
||||
3. Same as 2. above, but as part of the NovaCompute_ RA
|
||||
|
||||
- This has all the disadvantages of 2., but even more so, since
|
||||
new functionality would have to be mixed alongside the existing
|
||||
NovaCompute_ functionality.
|
||||
|
||||
.. _openstack-resource-agents: https://launchpad.net/openstack-resource-agents
|
||||
.. _NovaCompute: https://github.com/openstack/openstack-resource-agents/blob/master/ocf/NovaCompute
|
||||
|
||||
Data model impact
|
||||
-----------------
|
||||
|
||||
None
|
||||
|
||||
API impact
|
||||
----------
|
||||
|
||||
The HTTP API of the VM recovery workflow service needs to be able to
|
||||
receive events in the format they are sent by this instance monitor.
|
||||
|
||||
Security impact
|
||||
---------------
|
||||
|
||||
Ideally it should be possible for the instance monitor to send
|
||||
instance event data securely to the recovery workflow service
|
||||
(e.g. via TLS), without relying on the security of the admin network
|
||||
over which the data is sent.
|
||||
|
||||
Other end user impact
|
||||
---------------------
|
||||
|
||||
None
|
||||
|
||||
Performance Impact
|
||||
------------------
|
||||
|
||||
There will be a small amount of extra RAM and CPU required on each
|
||||
compute node for running the instance monitor. However it's a
|
||||
relatively simple service, so this should not have significant impact
|
||||
on the node.
|
||||
|
||||
Other deployer impact
|
||||
---------------------
|
||||
|
||||
Distributions need to package and deploy an extra service on each
|
||||
compute node. However the existing `instance monitor`_ implementation
|
||||
in masakari_ already provides files to simplify packaging on the Linux
|
||||
distributions most commonly used for OpenStack infrastructure.
|
||||
|
||||
.. _masakari: https://github.com/ntt-sic/masakari
|
||||
.. _`instance monitor`:
|
||||
https://github.com/ntt-sic/masakari/tree/master/masakari-instancemonitor/
|
||||
|
||||
Developer impact
|
||||
----------------
|
||||
|
||||
Nothing other than the listed work items below.
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
``libvirtd`` uses `QMP (QEMU Machine Protocol)`__ via UNIX domain
|
||||
socket (``/var/lib/libvirt/qemu/xxxx.monitor``) to communicate with
|
||||
the VM domain. ``libvirt`` catches the failure events and passes them
|
||||
to the VM monitor. The VM monitor filters the events and passes them
|
||||
to an external recovery workflow via HTTP, which then takes the action
|
||||
required to recover the VM.
|
||||
|
||||
__ http://wiki.qemu.org/QMP
|
||||
|
||||
::
|
||||
|
||||
+-----------------------+
|
||||
| +----------------+ |
|
||||
| | VM | |
|
||||
| | (qemu Process) | |
|
||||
| +---------^------+ |
|
||||
| | |QMP |
|
||||
| +-----v----------+ |
|
||||
| | libvirtd | |
|
||||
| +---------^------+ |
|
||||
| | | |
|
||||
| +-----v----------+ | +-----------------------+
|
||||
| | VM Monitor +------------>+ VM recovery workflow |
|
||||
| +----------------+ | +-----------------------+
|
||||
| |
|
||||
| Compute Node |
|
||||
+-----------------------+
|
||||
|
||||
We can almost certainly reuse the `instance monitor`_ provided
|
||||
by masakari_.
|
||||
|
||||
**FIXME**:
|
||||
|
||||
- Need to detail how and in which format the event data should
|
||||
be sent over HTTP. **This should allow for support for other
|
||||
hypervisors not based on** ``libvirt`` **being added in the future.**
|
||||
- Need to give details of in which exact ways the service can
|
||||
be configured.
|
||||
|
||||
- How should event filtering be configurable?
|
||||
|
||||
- Where should the configuration live? With `masakari`, it
|
||||
lives in ``/etc/masakari-instancemonitor.conf``.
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Primary assignee:
|
||||
<launchpad-id or None>
|
||||
|
||||
Other contributors:
|
||||
<launchpad-id or None>
|
||||
|
||||
Work Items
|
||||
----------
|
||||
|
||||
- Package `masakari`_'s `instance monitor`_ for SLES (`aspiers`)
|
||||
- Add documentation to the |ha-guide|_ (`beekhof`)
|
||||
- Look into libvirt-test-API_
|
||||
- Write test suite
|
||||
|
||||
.. |ha-guide| replace:: OpenStack High Availability Guide
|
||||
.. _ha-guide: http://docs.openstack.org/ha-guide/
|
||||
.. _libvirt-test-API: https://libvirt.org/testapi.html
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
- `libvirt <https://libvirt.org/>`_
|
||||
- `libvirt's Python bindings <https://libvirt.org/python.html>`_
|
||||
|
||||
Testing
|
||||
=======
|
||||
|
||||
It may be possible to write a test suite using libvirt-test-API_ or
|
||||
at least some of its components.
|
||||
|
||||
Documentation Impact
|
||||
====================
|
||||
|
||||
The service should be documented in the |ha-guide|_.
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
- `Instance HA etherpad started at Newton Design Summit in Austin
|
||||
<https://etherpad.openstack.org/p/newton-instance-ha>`_
|
||||
|
||||
- `"High Availability for Virtual Machines" user story
|
||||
<http://specs.openstack.org/openstack/openstack-user-stories/user-stories/proposed/ha_vm.html>`_
|
||||
|
||||
- `video of "HA for Pets and Hypervisors" presentation at OpenStack conference in Austin
|
||||
<https://youtu.be/lddtWUP_IKQ>`_
|
||||
|
||||
- `automatic-evacuation etherpad
|
||||
<https://etherpad.openstack.org/p/automatic-evacuation>`_
|
||||
|
||||
- `Instance auto-evacuation cross project spec (WIP)
|
||||
<https://review.openstack.org/#/c/257809>`_
|
||||
|
||||
|
||||
History
|
||||
=======
|
||||
|
||||
.. list-table:: Revisions
|
||||
:header-rows: 1
|
||||
|
||||
* - Release Name
|
||||
- Description
|
||||
* - Newton
|
||||
- Introduced
|
||||
Loading…
x
Reference in New Issue
Block a user