airship/drydock
Code Issues Proposed changes

Node storage configuration support

Browse Source 
- Refactor YAML schema for storage specification
- Add Drydock models for HostVolume/List, HostVolumeGroup/List
  HostStorageDevice/List, HostPartition/List
- Add MAAS API models for block device, partition, volume group
- Add implementation of ApplyNodeStorage driver task
- Add documentation for authoring storage configuration
- Add unit tests for YAML parsing
- Add unit tests for size calculation

Change-Id: I94fa00b2f2bcaff1607b645a421f7e54e6d1f11e
This commit is contained in:
Scott Hussey 2017-09-08 16:26:25 -05:00
parent 907d08699d
commit 689445280e
25 changed files with 2270 additions and 579 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

132
docs/topology.rst Normal file
View File

@ -0,0 +1,132 @@
=======================
Authoring Site Topology
=======================
Drydock uses a YAML-formatted site topology definition to configure
downstream drivers to provision baremetal nodes. This topology describes
the networking configuration of a site as well as the set of node configurations
that will be deployed. A node configuration consists of network attachment,
network addressing, local storage, kernel selection and configuration and
metadata.
The best source for a sample of the YAML schema for a topology is the unit
test input source_ /tests/yaml_samples/fullsite.yaml in tests/yaml_samples/fullsite.yaml.
Defining Node Storage
=====================
Storage can be defined in the `storage` stanza of either a HostProfile or BaremetalNode
document. The storage configuration can describe creation of partitions on physical disks,
the assignment of physical disks and/or partitions to volume groups, and the creation of
logical volumes. Drydock will make a best effort to parse out system-level storage such
as the root filesystem or boot filesystem and take appropriate steps to configure them in
the active node provisioning driver.
Example YAML schema of the `storage` stanza::
storage:
physical_devices:
sda:
labels:
bootdrive: true
partitions:
- name: 'root'
size: '10g'
bootable: true
filesystem:
mountpoint: '/'
fstype: 'ext4'
mount_options: 'defaults'
- name: 'boot'
size: '1g'
filesystem:
mountpoint: '/boot'
fstype: 'ext4'
mount_options: 'defaults'
sdb:
volume_group: 'log_vg'
volume_groups:
log_vg:
logical_volumes:
- name: 'log_lv'
size: '500m'
filesystem:
mountpoint: '/var/log'
fstype: 'xfs'
mount_options: 'defaults'
Schema
------
The `storage` stanza can contain two top level keys: `physical_devices` and
`volume_groups`. The latter is optional.
Physical Devices and Partitions
-------------------------------
A physical device can either be carved up in partitions (including a single partition
consuming the entire device) or added to a volume group as a physical volume. Each
key in the `physical_devices` mapping represents a device on a node. The key should either
be a device alias defined in the HardwareProfile or the name of the device published
by the OS. The value of each key must be a mapping with the following keys
* `labels`: A mapping of key/value strings providing generic labels for the device
* `partitions`: A sequence of mappings listing the partitions to be created on the device.
The mapping is described below. Incompatible with the `volume_group` specification.
* `volume_group`: A volume group name to add the device to as a physical volume. Incompatible
with the `partitions` specification.
Partition
~~~~~~~~~
A partition mapping describes a GPT partition on a physical disk. It can left as a raw
block device or formatted and mounted as a filesystem
* `name`: Metadata describing the partition in the topology
* `size`: The size of the partition. See the *Size Format* section below
* `bootable`: Boolean whether this partition should be the bootable device
* `part_uuid`: A UUID4 formatted UUID to assign to the partition. If not specified one will be generated
* `filesystem`: A optional mapping describing how the partition should be formatted and mounted
* `mountpoint`: Where the filesystem should be mounted. If not specified the partition will be left as a raw deice
* `fstype`: The format of the filesyste. Defaults to ext4
* `mount_options`: fstab style mount options. Default is 'defaults'
* `fs_uuid`: A UUID4 formatted UUID to assign to the filesystem. If not specified one will be generated
* `fs_label`: A filesystem label to assign to the filesystem. Optional.
Size Format
~~~~~~~~~~~
The size specification for a partition or logical volume is formed from three parts
* The first character can optionally be `>` indicating that the size specified is a minimum and the
calculated size should be at least the minimum and should take the rest of the available space on
the physical device or volume group.
* The second part is the numeric portion and must be an integer
* The third part is a label
* `m`\|`M`\|`mb`\|`MB`: Megabytes or 10^6 * the numeric
* `g`\|`G`\|`gb`\|`GB`: Gigabytes or 10^9 * the numeric
* `t`\|`T`\|`tb`\|`TB`: Terabytes or 10^12 * the numeric
* `%`: The percentage of total device or volume group space
Volume Groups and Logical Volumes
---------------------------------
Logical volumes can be used to create RAID-0 volumes spanning multiple physical disks or partitions.
Each key in the `volume_groups` mapping is a name assigned to a volume group. This name must be specified
as the `volume_group` attribute on one or more physical devices or partitions, or the configuration is invalid.
Each mapping value is another mapping describing the volume group.
* `vg_uuid`: A UUID4 format uuid applied to the volume group. If not specified, one is generated
* `logical_volumes`: A sequence of mappings listing the logical volumes to be created in the volume group
Logical Volume
~~~~~~~~~~~~~~
A logical volume is a RAID-0 volume. Using logical volumes for `/` and `/boot` is supported
* `name`: Required field. Used as the logical volume name.
* `size`: The logical volume size. See *Size Format* above for details.
* `lv_uuid`: A UUID4 format uuid applied to the logical volume: If not specified, one is generated
* `filesystem`: A mapping specifying how the logical volume should be formatted and mounted. See the
*Partition* section above for filesystem details.

4
drydock_provisioner/config.py
View File

@ -128,6 +128,10 @@ class DrydockConfig(object):
'apply_node_networking',
default=5,
help='Timeout in minutes for configuring node networking'),
cfg.IntOpt(
'apply_node_storage',
default=5,
help='Timeout in minutes for configuring node storage'),
cfg.IntOpt(
'apply_node_platform',
default=5,

2
drydock_provisioner/control/bootdata.py
View File

@ -90,7 +90,7 @@ class BootdataResource(StatefulResource):
r"""[Unit]
Description=Promenade Initialization Service
Documentation=http://github.com/att-comdev/drydock
After=network.target local-fs.target
After=network-online.target local-fs.target
ConditionPathExists=!/var/lib/prom.done
[Service]

4
drydock_provisioner/drivers/node/__init__.py
View File

@ -52,5 +52,5 @@ class NodeDriver(ProviderDriver):
if task_action in self.supported_actions:
return
else:
raise DriverError("Unsupported action %s for driver %s" %
(task_action, self.driver_desc))
raise errors.DriverError("Unsupported action %s for driver %s" %
(task_action, self.driver_desc))

13
drydock_provisioner/drivers/node/maasdriver/api_client.py
View File

@ -18,6 +18,8 @@ import requests
import requests.auth as req_auth
import base64
import drydock_provisioner.error as errors
class MaasOauth(req_auth.AuthBase):
def __init__(self, apikey):
@ -74,7 +76,7 @@ class MaasRequestFactory(object):
def test_connectivity(self):
try:
resp = self.get('version/')
except requests.Timeout(ex):
except requests.Timeout as ex:
raise errors.TransientDriverError("Timeout connection to MaaS")
if resp.status_code in [500, 503]:
@ -89,10 +91,11 @@ class MaasRequestFactory(object):
def test_authentication(self):
try:
resp = self.get('account/', op='list_authorisation_tokens')
except requests.Timeout(ex):
except requests.Timeout as ex:
raise errors.TransientDriverError("Timeout connection to MaaS")
except:
raise errors.PersistentDriverError("Error accessing MaaS")
except Exception as ex:
raise errors.PersistentDriverError(
"Error accessing MaaS: %s" % str(ex))
if resp.status_code in [401, 403]:
raise errors.PersistentDriverError(
@ -172,4 +175,6 @@ class MaasRequestFactory(object):
% (prepared_req.method, prepared_req.url,
str(prepared_req.body).replace('\\r\\n', '\n'),
resp.status_code, resp.text))
raise errors.DriverError("MAAS Error: %s - %s" % (resp.status_code,
resp.text))
return resp

706
drydock_provisioner/drivers/node/maasdriver/driver.py
View File

@ -18,6 +18,8 @@ import logging
import traceback
import sys
import uuid
import re
import math
from oslo_config import cfg
@ -25,6 +27,7 @@ import drydock_provisioner.error as errors
import drydock_provisioner.drivers as drivers
import drydock_provisioner.objects.fields as hd_fields
import drydock_provisioner.objects.task as task_model
import drydock_provisioner.objects.hostprofile as hostprofile
from drydock_provisioner.drivers.node import NodeDriver
from drydock_provisioner.drivers.node.maasdriver.api_client import MaasRequestFactory
@ -37,6 +40,8 @@ import drydock_provisioner.drivers.node.maasdriver.models.tag as maas_tag
import drydock_provisioner.drivers.node.maasdriver.models.sshkey as maas_keys
import drydock_provisioner.drivers.node.maasdriver.models.boot_resource as maas_boot_res
import drydock_provisioner.drivers.node.maasdriver.models.rack_controller as maas_rack
import drydock_provisioner.drivers.node.maasdriver.models.partition as maas_partition
import drydock_provisioner.drivers.node.maasdriver.models.volumegroup as maas_vg
class MaasNodeDriver(NodeDriver):
@ -168,8 +173,6 @@ class MaasNodeDriver(NodeDriver):
self.orchestrator.task_field_update(
task.get_id(), status=hd_fields.TaskStatus.Running)
site_design = self.orchestrator.get_effective_site(design_id)
if task.action == hd_fields.OrchestratorAction.CreateNetworkTemplate:
self.orchestrator.task_field_update(
@ -529,6 +532,99 @@ class MaasNodeDriver(NodeDriver):
else:
result = hd_fields.ActionResult.Failure
self.orchestrator.task_field_update(
task.get_id(),
status=hd_fields.TaskStatus.Complete,
result=result,
result_detail=result_detail)
elif task.action == hd_fields.OrchestratorAction.ApplyNodeStorage:
self.orchestrator.task_field_update(
task.get_id(), status=hd_fields.TaskStatus.Running)
self.logger.debug(
"Starting subtask to configure the storage on %s nodes." %
(len(task.node_list)))
subtasks = []
result_detail = {
'detail': [],
'failed_nodes': [],
'successful_nodes': [],
}
for n in task.node_list:
subtask = self.orchestrator.create_task(
task_model.DriverTask,
parent_task_id=task.get_id(),
design_id=design_id,
action=hd_fields.OrchestratorAction.ApplyNodeStorage,
task_scope={'node_names': [n]})
runner = MaasTaskRunner(
state_manager=self.state_manager,
orchestrator=self.orchestrator,
task_id=subtask.get_id())
self.logger.info(
"Starting thread for task %s to config node %s storage" %
(subtask.get_id(), n))
runner.start()
subtasks.append(subtask.get_id())
cleaned_subtasks = []
attempts = 0
max_attempts = cfg.CONF.timeouts.apply_node_storage * (
60 // cfg.CONF.poll_interval)
worked = failed = False
self.logger.debug(
"Polling for subtask completion every %d seconds, a max of %d polls."
% (cfg.CONF.poll_interval, max_attempts))
while len(cleaned_subtasks) < len(
subtasks) and attempts < max_attempts:
for t in subtasks:
if t in cleaned_subtasks:
continue
subtask = self.state_manager.get_task(t)
if subtask.status == hd_fields.TaskStatus.Complete:
self.logger.info(
"Task %s to configure node storage complete - status %s"
% (subtask.get_id(), subtask.get_result()))
cleaned_subtasks.append(t)
if subtask.result == hd_fields.ActionResult.Success:
result_detail['successful_nodes'].extend(
subtask.node_list)
worked = True
elif subtask.result == hd_fields.ActionResult.Failure:
result_detail['failed_nodes'].extend(
subtask.node_list)
failed = True
elif subtask.result == hd_fields.ActionResult.PartialSuccess:
worked = failed = True
time.sleep(cfg.CONF.poll_interval)
attempts = attempts + 1
if len(cleaned_subtasks) < len(subtasks):
self.logger.warning(
"Time out for task %s before all subtask threads complete"
% (task.get_id()))
result = hd_fields.ActionResult.DependentFailure
result_detail['detail'].append(
'Some subtasks did not complete before the timeout threshold'
)
elif worked and failed:
result = hd_fields.ActionResult.PartialSuccess
elif worked:
result = hd_fields.ActionResult.Success
else:
result = hd_fields.ActionResult.Failure
self.orchestrator.task_field_update(
task.get_id(),
status=hd_fields.TaskStatus.Complete,
@ -719,260 +815,6 @@ class MaasNodeDriver(NodeDriver):
status=hd_fields.TaskStatus.Complete,
result=result,
result_detail=result_detail)
elif task.action == hd_fields.OrchestratorAction.ApplyNodeNetworking:
self.orchestrator.task_field_update(
task.get_id(), status=hd_fields.TaskStatus.Running)
self.logger.debug(
"Starting subtask to configure networking on %s nodes." %
(len(task.node_list)))
subtasks = []
result_detail = {
'detail': [],
'failed_nodes': [],
'successful_nodes': [],
}
for n in task.node_list:
subtask = self.orchestrator.create_task(
task_model.DriverTask,
parent_task_id=task.get_id(),
design_id=design_id,
action=hd_fields.OrchestratorAction.ApplyNodeNetworking,
site_name=task.site_name,
task_scope={'site': task.site_name,
'node_names': [n]})
runner = MaasTaskRunner(
state_manager=self.state_manager,
orchestrator=self.orchestrator,
task_id=subtask.get_id())
self.logger.info(
"Starting thread for task %s to configure networking on node %s"
% (subtask.get_id(), n))
runner.start()
subtasks.append(subtask.get_id())
running_subtasks = len(subtasks)
attempts = 0
worked = failed = False
while running_subtasks > 0 and attempts < cfg.CONF.timeouts.apply_node_networking:
for t in subtasks:
subtask = self.state_manager.get_task(t)
if subtask.status == hd_fields.TaskStatus.Complete:
self.logger.info(
"Task %s to apply networking on node %s complete - status %s"
% (subtask.get_id(), n, subtask.get_result()))
running_subtasks = running_subtasks - 1
if subtask.result == hd_fields.ActionResult.Success:
result_detail['successful_nodes'].extend(
subtask.node_list)
worked = True
elif subtask.result == hd_fields.ActionResult.Failure:
result_detail['failed_nodes'].extend(
subtask.node_list)
failed = True
elif subtask.result == hd_fields.ActionResult.PartialSuccess:
worked = failed = True
time.sleep(1 * 60)
attempts = attempts + 1
if running_subtasks > 0:
self.logger.warning(
"Time out for task %s before all subtask threads complete"
% (task.get_id()))
result = hd_fields.ActionResult.DependentFailure
result_detail['detail'].append(
'Some subtasks did not complete before the timeout threshold'
)
elif worked and failed:
result = hd_fields.ActionResult.PartialSuccess
elif worked:
result = hd_fields.ActionResult.Success
else:
result = hd_fields.ActionResult.Failure
self.orchestrator.task_field_update(
task.get_id(),
status=hd_fields.TaskStatus.Complete,
result=result,
result_detail=result_detail)
elif task.action == hd_fields.OrchestratorAction.ApplyNodePlatform:
self.orchestrator.task_field_update(
task.get_id(), status=hd_fields.TaskStatus.Running)
self.logger.debug(
"Starting subtask to configure the platform on %s nodes." %
(len(task.node_list)))
subtasks = []
result_detail = {
'detail': [],
'failed_nodes': [],
'successful_nodes': [],
}
for n in task.node_list:
subtask = self.orchestrator.create_task(
task_model.DriverTask,
parent_task_id=task.get_id(),
design_id=design_id,
action=hd_fields.OrchestratorAction.ApplyNodePlatform,
site_name=task.site_name,
task_scope={'site': task.site_name,
'node_names': [n]})
runner = MaasTaskRunner(
state_manager=self.state_manager,
orchestrator=self.orchestrator,
task_id=subtask.get_id())
self.logger.info(
"Starting thread for task %s to config node %s platform" %
(subtask.get_id(), n))
runner.start()
subtasks.append(subtask.get_id())
running_subtasks = len(subtasks)
attempts = 0
worked = failed = False
while running_subtasks > 0 and attempts < cfg.CONF.timeouts.apply_node_platform:
for t in subtasks:
subtask = self.state_manager.get_task(t)
if subtask.status == hd_fields.TaskStatus.Complete:
self.logger.info(
"Task %s to configure node %s platform complete - status %s"
% (subtask.get_id(), n, subtask.get_result()))
running_subtasks = running_subtasks - 1
if subtask.result == hd_fields.ActionResult.Success:
result_detail['successful_nodes'].extend(
subtask.node_list)
worked = True
elif subtask.result == hd_fields.ActionResult.Failure:
result_detail['failed_nodes'].extend(
subtask.node_list)
failed = True
elif subtask.result == hd_fields.ActionResult.PartialSuccess:
worked = failed = True
time.sleep(1 * 60)
attempts = attempts + 1
if running_subtasks > 0:
self.logger.warning(
"Time out for task %s before all subtask threads complete"
% (task.get_id()))
result = hd_fields.ActionResult.DependentFailure
result_detail['detail'].append(
'Some subtasks did not complete before the timeout threshold'
)
elif worked and failed:
result = hd_fields.ActionResult.PartialSuccess
elif worked:
result = hd_fields.ActionResult.Success
else:
result = hd_fields.ActionResult.Failure
self.orchestrator.task_field_update(
task.get_id(),
status=hd_fields.TaskStatus.Complete,
result=result,
result_detail=result_detail)
elif task.action == hd_fields.OrchestratorAction.DeployNode:
self.orchestrator.task_field_update(
task.get_id(), status=hd_fields.TaskStatus.Running)
self.logger.debug("Starting subtask to deploy %s nodes." %
(len(task.node_list)))
subtasks = []
result_detail = {
'detail': [],
'failed_nodes': [],
'successful_nodes': [],
}
for n in task.node_list:
subtask = self.orchestrator.create_task(
task_model.DriverTask,
parent_task_id=task.get_id(),
design_id=design_id,
action=hd_fields.OrchestratorAction.DeployNode,
site_name=task.site_name,
task_scope={'site': task.site_name,
'node_names': [n]})
runner = MaasTaskRunner(
state_manager=self.state_manager,
orchestrator=self.orchestrator,
task_id=subtask.get_id())
self.logger.info(
"Starting thread for task %s to deploy node %s" %
(subtask.get_id(), n))
runner.start()
subtasks.append(subtask.get_id())
running_subtasks = len(subtasks)
attempts = 0
worked = failed = False
while running_subtasks > 0 and attempts < cfg.CONF.timeouts.deploy_node:
for t in subtasks:
subtask = self.state_manager.get_task(t)
if subtask.status == hd_fields.TaskStatus.Complete:
self.logger.info(
"Task %s to deploy node %s complete - status %s" %
(subtask.get_id(), n, subtask.get_result()))
running_subtasks = running_subtasks - 1
if subtask.result == hd_fields.ActionResult.Success:
result_detail['successful_nodes'].extend(
subtask.node_list)
worked = True
elif subtask.result == hd_fields.ActionResult.Failure:
result_detail['failed_nodes'].extend(
subtask.node_list)
failed = True
elif subtask.result == hd_fields.ActionResult.PartialSuccess:
worked = failed = True
time.sleep(1 * 60)
attempts = attempts + 1
if running_subtasks > 0:
self.logger.warning(
"Time out for task %s before all subtask threads complete"
% (task.get_id()))
result = hd_fields.ActionResult.DependentFailure
result_detail['detail'].append(
'Some subtasks did not complete before the timeout threshold'
)
elif worked and failed:
result = hd_fields.ActionResult.PartialSuccess
elif worked:
result = hd_fields.ActionResult.Success
else:
result = hd_fields.ActionResult.Failure
self.orchestrator.task_field_update(
task.get_id(),
status=hd_fields.TaskStatus.Complete,
result=result,
result_detail=result_detail)
class MaasTaskRunner(drivers.DriverTaskRunner):
@ -1060,7 +902,8 @@ class MaasTaskRunner(drivers.DriverTaskRunner):
# Ensure that the MTU of the untagged VLAN on the fabric
# matches that on the NetworkLink config
vlan_list = maas_vlan.Vlans(self.maas_client, fabric_id=link_fabric.resource_id)
vlan_list = maas_vlan.Vlans(
self.maas_client, fabric_id=link_fabric.resource_id)
vlan_list.refresh()
vlan = vlan_list.singleton({'vid': 0})
vlan.mtu = l.mtu
@ -1126,7 +969,7 @@ class MaasTaskRunner(drivers.DriverTaskRunner):
self.maas_client,
name=n.name,
cidr=n.cidr,
dns_servers = n.dns_servers,
dns_servers=n.dns_servers,
fabric=fabric.resource_id,
vlan=vlan.resource_id,
gateway_ip=n.get_default_gateway())
@ -1202,53 +1045,62 @@ class MaasTaskRunner(drivers.DriverTaskRunner):
"DHCP enabled for subnet %s, activating in MaaS"
% (subnet.name))
rack_ctlrs = maas_rack.RackControllers(self.maas_client)
rack_ctlrs = maas_rack.RackControllers(
self.maas_client)
rack_ctlrs.refresh()
dhcp_config_set=False
dhcp_config_set = False
for r in rack_ctlrs:
if n.dhcp_relay_upstream_target is not None:
if r.interface_for_ip(n.dhcp_relay_upstream_target):
iface = r.interface_for_ip(n.dhcp_relay_upstream_target)
if r.interface_for_ip(
n.dhcp_relay_upstream_target):
iface = r.interface_for_ip(
n.dhcp_relay_upstream_target)
vlan.relay_vlan = iface.vlan
self.logger.debug(
"Relaying DHCP on vlan %s to vlan %s" % (vlan.resource_id, vlan.relay_vlan)
)
"Relaying DHCP on vlan %s to vlan %s"
% (vlan.resource_id,
vlan.relay_vlan))
result_detail['detail'].append(
"Relaying DHCP on vlan %s to vlan %s" % (vlan.resource_id, vlan.relay_vlan))
"Relaying DHCP on vlan %s to vlan %s"
% (vlan.resource_id,
vlan.relay_vlan))
vlan.update()
dhcp_config_set=True
dhcp_config_set = True
break
else:
for i in r.interfaces:
if i.vlan == vlan.resource_id:
self.logger.debug(
"Rack controller %s has interface on vlan %s" %
(r.resource_id, vlan.resource_id))
"Rack controller %s has interface on vlan %s"
% (r.resource_id,
vlan.resource_id))
rackctl_id = r.resource_id
vlan.dhcp_on = True
vlan.primary_rack = rackctl_id
self.logger.debug(
"Enabling DHCP on VLAN %s managed by rack ctlr %s"
% (vlan.resource_id, rackctl_id))
% (vlan.resource_id,
rackctl_id))
result_detail['detail'].append(
"Enabling DHCP on VLAN %s managed by rack ctlr %s"
% (vlan.resource_id, rackctl_id))
% (vlan.resource_id,
rackctl_id))
vlan.update()
dhcp_config_set=True
dhcp_config_set = True
break
if dhcp_config_set:
break
if not dhcp_config_set:
self.logger.error(
"Network %s requires DHCP, but could not locate a rack controller to serve it." %
(n.name))
"Network %s requires DHCP, but could not locate a rack controller to serve it."
% (n.name))
result_detail['detail'].append(
"Network %s requires DHCP, but could not locate a rack controller to serve it." %
(n.name))
"Network %s requires DHCP, but could not locate a rack controller to serve it."
% (n.name))
elif dhcp_on and vlan.dhcp_on:
self.logger.info(
@ -1465,7 +1317,8 @@ class MaasTaskRunner(drivers.DriverTaskRunner):
except:
self.logger.warning(
"Error updating node %s status during commissioning, will re-attempt."
% (n))
% (n),
exc_info=True)
if machine.status_name == 'Ready':
self.logger.info("Node %s commissioned." % (n))
result_detail['detail'].append(
@ -1611,8 +1464,8 @@ class MaasTaskRunner(drivers.DriverTaskRunner):
if iface.effective_mtu != nl.mtu:
self.logger.debug(
"Updating interface %s MTU to %s"
% (i.device_name, nl.mtu))
"Updating interface %s MTU to %s" %
(i.device_name, nl.mtu))
iface.set_mtu(nl.mtu)
for iface_net in getattr(i, 'networks', []):
@ -1886,6 +1739,247 @@ class MaasTaskRunner(drivers.DriverTaskRunner):
else:
final_result = hd_fields.ActionResult.Success
self.orchestrator.task_field_update(
self.task.get_id(),
status=hd_fields.TaskStatus.Complete,
result=final_result,
result_detail=result_detail)
elif task_action == hd_fields.OrchestratorAction.ApplyNodeStorage:
try:
machine_list = maas_machine.Machines(self.maas_client)
machine_list.refresh()
except Exception as ex:
self.logger.error(
"Error configuring node storage, cannot access MaaS: %s" %
str(ex))
traceback.print_tb(sys.last_traceback)
self.orchestrator.task_field_update(
self.task.get_id(),
status=hd_fields.TaskStatus.Complete,
result=hd_fields.ActionResult.Failure,
result_detail={
'detail': 'Error accessing MaaS API',
'retry': True
})
return
nodes = self.task.node_list
result_detail = {'detail': []}
worked = failed = False
for n in nodes:
try:
self.logger.debug(
"Locating node %s for storage configuration" % (n))
node = site_design.get_baremetal_node(n)
machine = machine_list.identify_baremetal_node(
node, update_name=False)
if machine is None:
self.logger.warning(
"Could not locate machine for node %s" % n)
result_detail['detail'].append(
"Could not locate machine for node %s" % n)
failed = True
continue
except Exception as ex1:
failed = True
self.logger.error(
"Error locating machine for node %s: %s" % (n,
str(ex1)))
result_detail['detail'].append(
"Error locating machine for node %s" % (n))
continue
try:
"""
1. Clear VGs
2. Clear partitions
3. Apply partitioning
4. Create VGs
5. Create logical volumes
"""
self.logger.debug(
"Clearing current storage layout on node %s." %
node.name)
machine.reset_storage_config()
(root_dev, root_block) = node.find_fs_block_device('/')
(boot_dev, boot_block) = node.find_fs_block_device('/boot')
storage_layout = dict()
if isinstance(root_block, hostprofile.HostPartition):
storage_layout['layout_type'] = 'flat'
storage_layout['root_device'] = root_dev.name
storage_layout['root_size'] = root_block.size
elif isinstance(root_block, hostprofile.HostVolume):
storage_layout['layout_type'] = 'lvm'
if len(root_dev.physical_devices) != 1:
msg = "Root LV in VG with multiple physical devices on node %s" % (
node.name)
self.logger.error(msg)
result_detail['detail'].append(msg)
failed = True
continue
storage_layout[
'root_device'] = root_dev.physical_devices[0]
storage_layout['root_lv_size'] = root_block.size
storage_layout['root_lv_name'] = root_block.name
storage_layout['root_vg_name'] = root_dev.name
if boot_block is not None:
storage_layout['boot_size'] = boot_block.size
self.logger.debug(
"Setting node %s root storage layout: %s" %
(node.name, str(storage_layout)))
machine.set_storage_layout(**storage_layout)
vg_devs = {}
for d in node.storage_devices:
maas_dev = machine.block_devices.singleton({
'name':
d.name
})
if maas_dev is None:
self.logger.warning("Dev %s not found on node %s" %
(d.name, node.name))
continue
if d.volume_group is not None:
self.logger.debug(
"Adding dev %s to volume group %s" %
(d.name, d.volume_group))
if d.volume_group not in vg_devs:
vg_devs[d.volume_group] = {'b': [], 'p': []}
vg_devs[d.volume_group]['b'].append(
maas_dev.resource_id)
continue
self.logger.debug("Partitioning dev %s on node %s" %
(d.name, node.name))
for p in d.partitions:
if p.is_sys():
self.logger.debug(
"Skipping manually configuring a system partition."
)
continue
maas_dev.refresh()
size = MaasTaskRunner.calculate_bytes(
size_str=p.size, context=maas_dev)
part = maas_partition.Partition(
self.maas_client,
size=size,
bootable=p.bootable)
if p.part_uuid is not None:
part.uuid = p.part_uuid
self.logger.debug(
"Creating partition %s on dev %s" % (p.name,
d.name))
part = maas_dev.create_partition(part)
if p.volume_group is not None:
self.logger.debug(
"Adding partition %s to volume group %s" %
(p.name, p.volume_group))
if p.volume_group not in vg_devs:
vg_devs[p.volume_group] = {
'b': [],
'p': []
}
vg_devs[p.volume_group]['p'].append(
part.resource_id)
if p.mountpoint is not None:
format_opts = {'fstype': p.fstype}
if p.fs_uuid is not None:
format_opts['uuid'] = str(p.fs_uuid)
if p.fs_label is not None:
format_opts['label'] = p.fs_label
self.logger.debug(
"Formatting partition %s as %s" %
(p.name, p.fstype))
part.format(**format_opts)
mount_opts = {
'mount_point': p.mountpoint,
'mount_options': p.mount_options,
}
self.logger.debug(
"Mounting partition %s on %s" % (p.name,
p.mount))
part.mount(**mount_opts)
self.logger.debug(
"Finished configuring node %s partitions" % node.name)
for v in node.volume_groups:
if v.is_sys():
self.logger.debug(
"Skipping manually configuraing system VG.")
continue
if v.name not in vg_devs:
self.logger.warning(
"No physical volumes defined for VG %s, skipping."
% (v.name))
continue
maas_volgroup = maas_vg.VolumeGroup(
self.maas_client, name=v.name)
if v.vg_uuid is not None:
maas_volgroup.uuid = v.vg_uuid
if len(vg_devs[v.name]['b']) > 0:
maas_volgroup.block_devices = ','.join(
[str(x) for x in vg_devs[v.name]['b']])
if len(vg_devs[v.name]['p']) > 0:
maas_volgroup.partitions = ','.join(
[str(x) for x in vg_devs[v.name]['p']])
self.logger.debug(
"Creating volume group %s on node %s" %
(v.name, node.name))
maas_volgroup = machine.volume_groups.add(
maas_volgroup)
maas_volgroup.refresh()
for lv in v.logical_volumes:
calc_size = MaasTaskRunner.calculate_bytes(size_str=lv.size, context=maas_volgroup)
bd_id = maas_volgroup.create_lv(
name=lv.name,
uuid_str=lv.lv_uuid,
size=calc_size)
if lv.mountpoint is not None:
machine.refresh()
maas_lv = machine.block_devices.select(bd_id)
self.logger.debug(
"Formatting LV %s as filesystem on node %s."
% (lv.name, node.name))
maas_lv.format(
fstype=lv.fstype, uuid_str=lv.fs_uuid)
self.logger.debug(
"Mounting LV %s at %s on node %s." %
(lv.name, lv.mountpoint, node.name))
maas_lv.mount(
mount_point=lv.mountpoint,
mount_options=lv.mount_options)
except Exception as ex:
raise errors.DriverError(str(ex))
if worked and failed:
final_result = hd_fields.ActionResult.PartialSuccess
elif failed:
final_result = hd_fields.ActionResult.Failure
else:
final_result = hd_fields.ActionResult.Success
self.orchestrator.task_field_update(
self.task.get_id(),
status=hd_fields.TaskStatus.Complete,
@ -2018,6 +2112,62 @@ class MaasTaskRunner(drivers.DriverTaskRunner):
result=final_result,
result_detail=result_detail)
@classmethod
def calculate_bytes(cls, size_str=None, context=None):
"""Calculate the size on bytes of a size_str.
Calculate the size as specified in size_str in the context of the provided
blockdev or vg. Valid size_str format below.
#m or #M or #mb or #MB = # * 1024 * 1024
#g or #G or #gb or #GB = # * 1024 * 1024 * 1024
#t or #T or #tb or #TB = # * 1024 * 1024 * 1024 * 1024
#% = Percentage of the total storage in the context
Prepend '>' to the above to note the size as a minimum and the calculated size being the
remaining storage available above the minimum
If the calculated size is not available in the context, a NotEnoughStorage exception is
raised.
:param size_str: A string representing the desired size
:param context: An instance of maasdriver.models.blockdev.BlockDevice or
instance of maasdriver.models.volumegroup.VolumeGroup. The
size_str is interpreted in the context of this device
:return size: The calculated size in bytes
"""
pattern = '(>?)(\d+)([mMbBgGtT%]{1,2})'
regex = re.compile(pattern)
match = regex.match(size_str)
if not match:
raise errors.InvalidSizeFormat(
"Invalid size string format: %s" % size_str)
if ((match.group(1) == '>' or match.group(3) == '%') and not context):
raise errors.InvalidSizeFormat(
'Sizes using the ">" or "%" format must specify a '
'block device or volume group context')
base_size = int(match.group(2))
if match.group(3) in ['m', 'M', 'mb', 'MB']:
computed_size = base_size * (1000 * 1000)
elif match.group(3) in ['g', 'G', 'gb', 'GB']:
computed_size = base_size * (1000 * 1000 * 1000)
elif match.group(3) in ['t', 'T', 'tb', 'TB']:
computed_size = base_size * (1000 * 1000 * 1000 * 1000)
elif match.group(3) == '%':
computed_size = math.floor((base_size / 100) * int(context.size))
if computed_size > int(context.available_size):
raise errors.NotEnoughStorage()
if match.group(1) == '>':
computed_size = int(context.available_size)
return computed_size
def list_opts():
return {MaasNodeDriver.driver_key: MaasNodeDriver.maasdriver_options}

37
drydock_provisioner/drivers/node/maasdriver/models/base.py
View File

@ -11,16 +11,17 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""A representation of a MaaS REST resource.
Should be subclassed for different resources and
augmented with operations specific to those resources
"""
import json
import re
import logging
import drydock_provisioner.error as errors
"""
A representation of a MaaS REST resource. Should be subclassed
for different resources and augmented with operations specific
to those resources
"""
class ResourceBase(object):
@ -46,10 +47,16 @@ class ResourceBase(object):
resp = self.api_client.get(url)
updated_fields = resp.json()
updated_model = self.from_dict(self.api_client, updated_fields)
for f in self.fields:
if f in updated_fields.keys():
setattr(self, f, updated_fields.get(f))
if hasattr(updated_model, f):
setattr(self, f, getattr(updated_model, f))
def delete(self):
"""Delete this resource in MaaS."""
url = self.interpolate_url()
resp = self.api_client.delete(url)
"""
Parse URL for placeholders and replace them with current
@ -157,8 +164,7 @@ class ResourceBase(object):
class ResourceCollectionBase(object):
"""
A collection of MaaS resources.
"""A collection of MaaS resources.
Rather than a simple list, we will key the collection on resource
ID for more efficient access.
@ -175,10 +181,7 @@ class ResourceCollectionBase(object):
self.logger = logging.getLogger('drydock.nodedriver.maasdriver')
def interpolate_url(self):
"""
Parse URL for placeholders and replace them with current
instance values
"""
"""Parse URL for placeholders and replace them with current instance values."""
pattern = '\{([a-z_]+)\}'
regex = re.compile(pattern)
start = 0
@ -273,8 +276,7 @@ class ResourceCollectionBase(object):
return result
def singleton(self, query):
"""
A query that requires a single item response
"""A query that requires a single item response.
:param query: A dict of k:v pairs defining the query parameters
"""
@ -298,11 +300,8 @@ class ResourceCollectionBase(object):
else:
return None
"""
Iterate over the resources in the collection
"""
def __iter__(self):
"""Iterate over the resources in the collection."""
return iter(self.resources.values())
"""

270
drydock_provisioner/drivers/node/maasdriver/models/blockdev.py Normal file
View File

@ -0,0 +1,270 @@
# Copyright 2017 AT&T Intellectual Property. All other rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""API model for MaaS node block device resource."""
import uuid
from . import base as model_base
from . import partition as maas_partition
import drydock_provisioner.error as errors
class BlockDevice(model_base.ResourceBase):
resource_url = 'nodes/{system_id}/blockdevices/{resource_id}/'
fields = [
'resource_id',
'system_id',
'name',
'path',
'size',
'type',
'path',
'partitions',
'uuid',
'filesystem',
'tags',
'serial',
'model',
'id_path',
'bootable',
'available_size',
]
json_fields = [
'name',
]
"""Filesystem dictionary fields:
mount_point: the mount point on the system directory hierarchy
fstype: The filesystem format, defaults to ext4
mount_options: The mount options specified in /etc/fstab, defaults to 'defaults'
label: The filesystem lab
uuid: The filesystem uuid
"""
def __init__(self, api_client, **kwargs):
super().__init__(api_client, **kwargs)
if getattr(self, 'resource_id', None) is not None:
try:
self.partitions = maas_partition.Partitions(
api_client,
system_id=self.system_id,
device_id=self.resource_id)
self.partitions.refresh()
except Exception as ex:
self.logger.warning(
"Could not load partitions on node %s block device %s" %
(self.system_id, self.resource_id))
else:
self.partitions = None
def format(self, fstype='ext4', uuid_str=None, label=None):
"""Format this block device with a filesystem.
:param fstype: String of the filesystem format to use, defaults to ext4
:param uuid: String of the UUID to assign to the filesystem. One will be
generated if this is left as None
"""
try:
data = {'fstype': fstype}
if uuid_str:
data['uuid'] = str(uuid_str)
else:
data['uuid'] = str(uuid.uuid4())
url = self.interpolate_url()
self.logger.debug(
"Formatting device %s on node %s as filesystem: fstype=%s, uuid=%s"
% (self.name, self.system_id, fstype, uuid))
resp = self.api_client.post(url, op='format', files=data)
if not resp.ok:
raise Exception("MAAS error: %s - %s" % (resp.status_code,
resp.text))
self.refresh()
except Exception as ex:
msg = "Error: format of device %s on node %s failed: %s" \
% (self.name, self.system_id, str(ex))
self.logger.error(msg)
raise errors.DriverError(msg)
def unformat(self):
"""Unformat this block device.
Will attempt to unmount the device first.
"""
try:
self.refresh()
if self.filesystem is None:
self.logger.debug(
"Device %s not currently formatted, skipping unformat." %
(self.name))
return
if self.filesystem.get('mount_pount', None) is not None:
self.unmount()
url = self.interpolate_url()
self.logger.debug("Unformatting device %s on node %s" %
(self.name, self.system_id))
resp = self.api_client.post(url, op='unformat')
if not resp.ok:
raise Exception("MAAS error: %s - %s" % (resp.status_code,
resp.text))
self.refresh()
except Exception as ex:
msg = "Error: unformat of device %s on node %s failed: %s" \
% (self.name, self.system_id, str(ex))
self.logger.error(msg)
raise errors.DriverError(msg)
def mount(self, mount_point=None, mount_options='defaults'):
"""Mount this block device with a filesystem.
:param mount_point: The mountpoint on the system
:param mount_options: fstab style mount options, defaults to 'defaults'
"""
try:
if mount_point is None:
raise errors.DriverError(
"Cannot mount a block device on an empty mount point.")
data = {'mount_point': mount_point, 'mount_options': mount_options}
url = self.interpolate_url()
self.logger.debug(
"Mounting device %s on node %s at mount point %s" %
(self.resource_id, self.system_id, mount_point))
resp = self.api_client.post(url, op='mount', files=data)
if not resp.ok:
raise Exception("MAAS error: %s - %s" % (resp.status_code,
resp.text))
self.refresh()
except Exception as ex:
msg = "Error: mount of device %s on node %s failed: %s" \
% (self.name, self.system_id, str(ex))
self.logger.error(msg)
raise errors.DriverError(msg)
def unmount(self):
"""Unmount this block device."""
try:
self.refresh()
if self.filesystem is None or self.filesystem.get(
'mount_point', None) is None:
self.logger.debug(
"Device %s not currently mounted, skipping unmount." %
(self.name))
url = self.interpolate_url()
self.logger.debug("Unmounting device %s on node %s" %
(self.name, self.system_id))
resp = self.api_client.post(url, op='unmount')
if not resp.ok:
raise Exception("MAAS error: %s - %s" % (resp.status_code,
resp.text))
self.refresh()
except Exception as ex:
msg = "Error: unmount of device %s on node %s failed: %s" \
% (self.name, self.system_id, str(ex))
self.logger.error(msg)
raise errors.DriverError(msg)
def set_bootable(self):
"""Set this disk as the system bootdisk."""
try:
url = self.interpolate_url()
self.logger.debug("Setting device %s on node %s as bootable." %
(self.resource_id, self.system_id))
resp = self.api_client.post(url, op='set_boot_disk')
if not resp.ok:
raise Exception("MAAS error: %s - %s" % (resp.status_code,
resp.text))
self.refresh()
except Exception as ex:
msg = "Error: setting device %s on node %s to boot failed: %s" \
% (self.name, self.system_id, str(ex))
self.logger.error(msg)
raise errors.DriverError(msg)
def create_partition(self, partition):
"""Create a partition on this block device.
:param partition: Instance of models.partition.Partition to be carved out of this block device
"""
if self.type == 'physical':
if self.partitions is not None:
partition = self.partitions.add(partition)
self.partitions.refresh()
return self.partitions.select(partition.resource_id)
else:
msg = "Error: could not access device %s partition list" % self.name
self.logger.error(msg)
raise errors.DriverError(msg)
else:
msg = "Error: cannot partition non-physical device %s." % (
self.name)
self.logger.error(msg)
raise errors.DriverError(msg)
def delete_partition(self, partition_id):
if self.partitions is not None:
part = self.partitions.select(partition_id)
if part is not None:
part.delete()
self.refresh()
def clear_partitions(self):
for p in getattr(self, 'partitions', []):
p.delete()
self.refresh()
@classmethod
def from_dict(cls, api_client, obj_dict):
"""Instantiate this model from a dictionary.
Because MaaS decides to replace the resource ids with the
representation of the resource, we must reverse it for a true
representation of the block device
"""
refined_dict = {k: obj_dict.get(k, None) for k in cls.fields}
if 'id' in obj_dict.keys():
refined_dict['resource_id'] = obj_dict.get('id')
i = cls(api_client, **refined_dict)
return i
class BlockDevices(model_base.ResourceCollectionBase):
collection_url = 'nodes/{system_id}/blockdevices/'
collection_resource = BlockDevice
def __init__(self, api_client, **kwargs):
super().__init__(api_client)
self.system_id = kwargs.get('system_id', None)

19
drydock_provisioner/drivers/node/maasdriver/models/interface.py
View File

@ -27,11 +27,24 @@ class Interface(model_base.ResourceBase):
resource_url = 'nodes/{system_id}/interfaces/{resource_id}/'
fields = [
'resource_id', 'system_id', 'name', 'type', 'mac_address', 'vlan',
'links', 'effective_mtu', 'fabric_id', 'mtu',
'resource_id',
'system_id',
'name',
'type',
'mac_address',
'vlan',
'links',
'effective_mtu',
'fabric_id',
'mtu',
]
json_fields = [
'name', 'type', 'mac_address', 'vlan', 'links', 'mtu',
'name',
'type',
'mac_address',
'vlan',
'links',
'mtu',
]
def __init__(self, api_client, **kwargs):

132
drydock_provisioner/drivers/node/maasdriver/models/machine.py
View File

@ -11,22 +11,36 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Model representing MAAS node/machine resource."""
import drydock_provisioner.error as errors
import drydock_provisioner.drivers.node.maasdriver.models.base as model_base
import drydock_provisioner.drivers.node.maasdriver.models.interface as maas_interface
import drydock_provisioner.drivers.node.maasdriver.models.blockdev as maas_blockdev
import drydock_provisioner.drivers.node.maasdriver.models.volumegroup as maas_vg
import bson
import yaml
class Machine(model_base.ResourceBase):
resource_url = 'machines/{resource_id}/'
fields = [
'resource_id', 'hostname', 'power_type', 'power_state',
'power_parameters', 'interfaces', 'boot_interface', 'memory',
'cpu_count', 'tag_names', 'status_name', 'boot_mac', 'owner_data'
'resource_id',
'hostname',
'power_type',
'power_state',
'power_parameters',
'interfaces',
'boot_interface',
'memory',
'cpu_count',
'tag_names',
'status_name',
'boot_mac',
'owner_data',
'block_devices',
'volume_groups',
]
json_fields = ['hostname', 'power_type']
@ -38,8 +52,24 @@ class Machine(model_base.ResourceBase):
self.interfaces = maas_interface.Interfaces(
api_client, system_id=self.resource_id)
self.interfaces.refresh()
try:
self.block_devices = maas_blockdev.BlockDevices(
api_client, system_id=self.resource_id)
self.block_devices.refresh()
except Exception as ex:
self.logger.warning("Failed loading node %s block devices." %
(self.resource_id))
try:
self.volume_groups = maas_vg.VolumeGroups(
api_client, system_id=self.resource_id)
self.volume_groups.refresh()
except Exception as ex:
self.logger.warning("Failed load node %s volume groups." %
(self.resource_id))
else:
self.interfaces = None
self.block_devices = None
self.volume_groups = None
def interface_for_ip(self, ip_address):
"""Find the machine interface that will respond to ip_address.
@ -61,6 +91,100 @@ class Machine(model_base.ResourceBase):
if resp.status_code == 200:
self.power_parameters = resp.json()
def reset_storage_config(self):
"""Reset storage config on this machine.
Removes all the volume groups/logical volumes and all the physical
device partitions on this machine.
"""
self.logger.info("Resetting storage configuration on node %s" %
(self.resource_id))
if self.volume_groups is not None and self.volume_groups.len() > 0:
for vg in self.volume_groups:
self.logger.debug("Removing VG %s" % vg.name)
vg.delete()
else:
self.logger.debug("No VGs configured on node %s" %
(self.resource_id))
if self.block_devices is not None:
for d in self.block_devices:
if d.partitions is not None and d.partitions.len() > 0:
self.logger.debug(
"Clearing partitions on device %s" % d.name)
d.clear_partitions()
else:
self.logger.debug(
"No partitions found on device %s" % d.name)
else:
self.logger.debug("No block devices found on node %s" %
(self.resource_id))
def set_storage_layout(self,
layout_type='flat',
root_device=None,
root_size=None,
boot_size=None,
root_lv_size=None,
root_vg_name=None,
root_lv_name=None):
"""Set machine storage layout for the root disk.
:param layout_type: Whether to use 'flat' (partitions) or 'lvm' for the root filesystem
:param root_device: Name of the block device to place the root partition on
:param root_size: Size of the root partition in bytes
:param boot_size: Size of the boot partition in bytes
:param root_lv_size: Size of the root logical volume in bytes for LVM layout
:param root_vg_name: Name of the volume group with root LV
:param root_lv_name: Name of the root LV
"""
try:
url = self.interpolate_url()
self.block_devices.refresh()
root_dev = self.block_devices.singleton({'name': root_device})
if root_dev is None:
msg = "Error: cannot find storage device %s to set as root device" % root_device
self.logger.error(msg)
raise errors.DriverError(msg)
root_dev.set_bootable()
data = {
'storage_layout': layout_type,
'root_device': root_dev.resource_id,
}
self.logger.debug("Setting node %s storage layout to %s" %
(self.hostname, layout_type))
if root_size:
data['root_size'] = root_size
if boot_size:
data['boot_size'] = boot_size
if layout_type == 'lvm':
if root_lv_size:
data['lv_size'] = root_lv_size
if root_vg_name:
data['vg_name'] = root_vg_name
if root_lv_name:
data['lv_name'] = root_lv_name
resp = self.api_client.post(
url, op='set_storage_layout', files=data)
if not resp.ok:
raise Exception("MAAS Error: %s - %s" % (resp.status_code,
resp.text))
except Exception as ex:
msg = "Error: failed configuring node %s storage layout: %s" % (
self.resource_id, str(ex))
self.logger.error(msg)
raise errors.DriverError(msg)
def commission(self, debug=False):
url = self.interpolate_url()

216
drydock_provisioner/drivers/node/maasdriver/models/partition.py Normal file
View File

@ -0,0 +1,216 @@
# Copyright 2017 AT&T Intellectual Property. All other rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""API model for MaaS node storage partition resource."""
import uuid
from . import base as model_base
import drydock_provisioner.error as errors
class Partition(model_base.ResourceBase):
resource_url = 'nodes/{system_id}/blockdevices/{device_id}/partition/{resource_id}'
fields = [
'resource_id',
'system_id',
'device_id',
'name',
'path',
'size',
'type',
'uuid',
'filesystem',
'bootable',
]
json_fields = [
'size',
'uuid',
'bootable',
]
"""Filesystem dictionary fields:
mount_point: the mount point on the system directory hierarchy
fstype: The filesystem format, defaults to ext4
mount_options: The mount options specified in /etc/fstab, defaults to 'defaults'
label: The filesystem lab
uuid: The filesystem uuid
"""
def __init__(self, api_client, **kwargs):
super().__init__(api_client, **kwargs)
def format(self, fstype='ext4', uuid_str=None, fs_label=None):
"""Format this partition with a filesystem.
:param fstype: String of the filesystem format to use, defaults to ext4
:param uuid: String of the UUID to assign to the filesystem. One will be
generated if this is left as None
"""
try:
data = {'fstype': fstype}
if uuid_str:
data['uuid'] = str(uuid_str)
else:
data['uuid'] = str(uuid.uuid4())
if fs_label is not None:
data['label'] = fs_label
url = self.interpolate_url()
self.logger.debug(
"Formatting device %s on node %s as filesystem: %s" %
(self.name, self.system_id, data))
resp = self.api_client.post(url, op='format', files=data)
if not resp.ok:
raise Exception("MAAS error: %s - %s" % (resp.status_code,
resp.text))
self.refresh()
except Exception as ex:
msg = "Error: format of device %s on node %s failed: %s" \
% (self.name, self.system_id, str(ex))
self.logger.error(msg)
raise errors.DriverError(msg)
def unformat(self):
"""Unformat this block device.
Will attempt to unmount the device first.
"""
try:
self.refresh()
if self.filesystem is None:
self.logger.debug(
"Device %s not currently formatted, skipping unformat." %
(self.name))
return
if self.filesystem.get('mount_pount', None) is not None:
self.unmount()
url = self.interpolate_url()
self.logger.debug("Unformatting device %s on node %s" %
(self.name, self.system_id))
resp = self.api_client.post(url, op='unformat')
if not resp.ok:
raise Exception("MAAS error: %s - %s" % (resp.status_code,
resp.text))
self.refresh()
except Exception as ex:
msg = "Error: unformat of device %s on node %s failed: %s" \
% (self.name, self.system_id, str(ex))
self.logger.error(msg)
raise errors.DriverError(msg)
def mount(self, mount_point=None, mount_options='defaults'):
"""Mount this block device with a filesystem.
:param mount_point: The mountpoint on the system
:param mount_options: fstab style mount options, defaults to 'defaults'
"""
try:
if mount_point is None:
raise errors.DriverError(
"Cannot mount a block device on an empty mount point.")
data = {'mount_point': mount_point, 'mount_options': mount_options}
url = self.interpolate_url()
self.logger.debug(
"Mounting device %s on node %s at mount point %s" %
(self.resource_id, self.system_id, mount_point))
resp = self.api_client.post(url, op='mount', files=data)
if not resp.ok:
raise Exception("MAAS error: %s - %s" % (resp.status_code,
resp.text))
self.refresh()
except Exception as ex:
msg = "Error: mount of device %s on node %s failed: %s" \
% (self.name, self.system_id, str(ex))
self.logger.error(msg)
raise errors.DriverError(msg)
def unmount(self):
"""Unmount this block device."""
try:
self.refresh()
if self.filesystem is None or self.filesystem.get(
'mount_point', None) is None:
self.logger.debug(
"Device %s not currently mounted, skipping unmount." %
(self.name))
url = self.interpolate_url()
self.logger.debug("Unmounting device %s on node %s" %
(self.name, self.system_id))
resp = self.api_client.post(url, op='unmount')
if not resp.ok:
raise Exception("MAAS error: %s - %s" % (resp.status_code,
resp.text))
self.refresh()
except Exception as ex:
msg = "Error: unmount of device %s on node %s failed: %s" \
% (self.name, self.system_id, str(ex))
self.logger.error(msg)
raise errors.DriverError(msg)
def set_bootable(self):
"""Set this disk as the system bootdisk."""
try:
url = self.interpolate_url()
self.logger.debug("Setting device %s on node %s as bootable." %
(self.resource_id, self.system_id))
resp = self.api_client.post(url, op='set_boot_disk')
if not resp.ok:
raise Exception("MAAS error: %s - %s" % (resp.status_code,
resp.text))
self.refresh()
except Exception as ex:
msg = "Error: setting device %s on node %s to boot failed: %s" \
% (self.name, self.system_id, str(ex))
self.logger.error(msg)
raise errors.DriverError(msg)
@classmethod
def from_dict(cls, api_client, obj_dict):
"""Instantiate this model from a dictionary.
Because MaaS decides to replace the resource ids with the
representation of the resource, we must reverse it for a true
representation of the block device
"""
refined_dict = {k: obj_dict.get(k, None) for k in cls.fields}
if 'id' in obj_dict.keys():
refined_dict['resource_id'] = obj_dict.get('id')
i = cls(api_client, **refined_dict)
return i
class Partitions(model_base.ResourceCollectionBase):
collection_url = 'nodes/{system_id}/blockdevices/{device_id}/partitions/'
collection_resource = Partition
def __init__(self, api_client, **kwargs):
super().__init__(api_client)
self.system_id = kwargs.get('system_id', None)
self.device_id = kwargs.get('device_id', None)

22
drydock_provisioner/drivers/node/maasdriver/models/vlan.py
View File

@ -21,12 +21,26 @@ class Vlan(model_base.ResourceBase):
resource_url = 'fabrics/{fabric_id}/vlans/{api_id}/'
fields = [
'resource_id', 'name', 'description', 'vid', 'fabric_id', 'dhcp_on',
'mtu', 'primary_rack', 'secondary_rack', 'relay_vlan',
'resource_id',
'name',
'description',
'vid',
'fabric_id',
'dhcp_on',
'mtu',
'primary_rack',
'secondary_rack',
'relay_vlan',
]
json_fields = [
'name', 'description', 'vid', 'dhcp_on', 'mtu', 'primary_rack',
'secondary_rack', 'relay_vlan',
'name',
'description',
'vid',
'dhcp_on',
'mtu',
'primary_rack',
'secondary_rack',
'relay_vlan',
]
def __init__(self, api_client, **kwargs):

153
drydock_provisioner/drivers/node/maasdriver/models/volumegroup.py Normal file
View File

@ -0,0 +1,153 @@
# Copyright 2017 AT&T Intellectual Property. All other rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""API model for MaaS node volume group resource."""
import uuid
from . import base as model_base
import drydock_provisioner.error as errors
class VolumeGroup(model_base.ResourceBase):
resource_url = 'nodes/{system_id}/volume-group/{resource_id}/'
fields = [
'resource_id',
'system_id',
'name',
'size',
'available_size',
'uuid',
'logical_volumes',
'block_devices',
'partitions',
]
json_fields = [
'name',
'size',
'uuid',
'block_devices',
'partitions',
]
def create_lv(self, name=None, uuid_str=None, size=None):
"""Create a logical volume in this volume group.
:param name: Name of the logical volume
:param uuid_str: A UUID4-format string specifying the LV uuid. Will be generated if left as None
:param size: The size of the logical volume
"""
try:
if name is None or size is None:
raise Exception(
"Cannot create logical volume without specified name and size"
)
if uuid_str is None:
uuid_str = str(uuid.uuid4())
data = {'name': name, 'uuid': uuid_str, 'size': size}
self.logger.debug(
"Creating logical volume %s in VG %s on node %s" %
(name, self.name, self.system_id))
url = self.interpolate_url()
resp = self.api_client.post(
url, op='create_logical_volume', files=data)
if not resp.ok:
raise Exception("MAAS error - %s - %s" % (resp.status_code,
resp.txt))
res = resp.json()
if 'id' in res:
return res['id']
except Exception as ex:
msg = "Error: Could not create logical volume: %s" % str(ex)
self.logger.error(msg)
raise errors.DriverError(msg)
def delete_lv(self, lv_id=None, lv_name=None):
"""Delete a logical volume from this volume group.
:param lv_id: Resource ID of the logical volume
:param lv_name: Name of the logical volume, only referenced if no lv_id is specified
"""
try:
self.refresh()
if self.logical_volumes is not None:
if lv_id and lv_id in self.logical_volumes.values():
target_lv = lv_id
elif lv_name and lv_name in self.logical_volumes:
target_lv = self.logical_volumes[lv_name]
else:
raise Exception(
"lv_id %s and lv_name %s not found in VG %s" %
(lv_id, lv_name, self.name))
url = self.interpolate_url()
resp = self.api_client.post(
url, op='delete_logical_volume', files={'id': target_lv})
if not resp.ok:
raise Exception("MAAS error - %s - %s" % (resp.status_code,
resp.text))
else:
raise Exception("VG %s has no logical volumes" % self.name)
except Exception as ex:
msg = "Error: Could not delete logical volume: %s" % str(ex)
self.logger.error(msg)
raise errors.DriverError(msg)
@classmethod
def from_dict(cls, api_client, obj_dict):
"""Instantiate this model from a dictionary.
Because MaaS decides to replace the resource ids with the
representation of the resource, we must reverse it for a true
representation of the block device
"""
refined_dict = {k: obj_dict.get(k, None) for k in cls.fields}
if 'id' in obj_dict:
refined_dict['resource_id'] = obj_dict.get('id')
if 'logical_volumes' in refined_dict and isinstance(
refined_dict.get('logical_volumes'), list):
lvs = {}
for v in refined_dict.get('logical_volumes'):
lvs[v.get('name')] = v.get('id')
refined_dict['logical_volumes'] = lvs
i = cls(api_client, **refined_dict)
return i
class VolumeGroups(model_base.ResourceCollectionBase):
collection_url = 'nodes/{system_id}/volume-groups/'
collection_resource = VolumeGroup
def __init__(self, api_client, **kwargs):
super().__init__(api_client)
self.system_id = kwargs.get('system_id', None)
def add(self, res):
res = super().add(res)
res.system_id = self.system_id
return res

10
drydock_provisioner/error.py
View File

@ -46,6 +46,14 @@ class PersistentDriverError(DriverError):
pass
class NotEnoughStorage(DriverError):
pass
class InvalidSizeFormat(DriverError):
pass
class ApiError(Exception):
def __init__(self, msg, code=500):
super().__init__(msg)
@ -53,7 +61,7 @@ class ApiError(Exception):
self.status_code = code
def to_json(self):
err_dict = {'error': msg, 'type': self.__class__.__name__}
err_dict = {'error': self.message, 'type': self.__class__.__name__}
return json.dumps(err_dict)

101
drydock_provisioner/ingester/plugins/yaml.py
View File

@ -11,8 +11,8 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""YAML Ingester.
This data ingester will consume YAML site topology documents."""
"""This data ingester will consume YAML site topology documents."""
import yaml
import logging
import base64
@ -336,36 +336,83 @@ class YamlIngester(IngesterPlugin):
model.oob_parameters[k] = v
storage = spec.get('storage', {})
model.storage_layout = storage.get('layout', 'lvm')
bootdisk = storage.get('bootdisk', {})
model.bootdisk_device = bootdisk.get(
'device', None)
model.bootdisk_root_size = bootdisk.get(
'root_size', None)
model.bootdisk_boot_size = bootdisk.get(
'boot_size', None)
phys_devs = storage.get('physical_devices', {})
partitions = storage.get('partitions', [])
model.partitions = objects.HostPartitionList()
model.storage_devices = objects.HostStorageDeviceList(
)
for p in partitions:
part_model = objects.HostPartition()
for k, v in phys_devs.items():
sd = objects.HostStorageDevice(name=k)
sd.source = hd_fields.ModelSource.Designed
part_model.name = p.get('name', None)
part_model.source = hd_fields.ModelSource.Designed
part_model.device = p.get('device', None)
part_model.part_uuid = p.get('part_uuid', None)
part_model.size = p.get('size', None)
part_model.mountpoint = p.get(
'mountpoint', None)
part_model.fstype = p.get('fstype', 'ext4')
part_model.mount_options = p.get(
'mount_options', 'defaults')
part_model.fs_uuid = p.get('fs_uuid', None)
part_model.fs_label = p.get('fs_label', None)
if 'labels' in v:
sd.labels = v.get('labels').copy()
model.partitions.append(part_model)
if 'volume_group' in v:
vg = v.get('volume_group')
sd.volume_group = vg
elif 'partitions' in v:
sd.partitions = objects.HostPartitionList()
for vv in v.get('partitions', []):
part_model = objects.HostPartition()
part_model.name = vv.get('name')
part_model.source = hd_fields.ModelSource.Designed
part_model.part_uuid = vv.get(
'part_uuid', None)
part_model.size = vv.get('size', None)
if 'labels' in vv:
part_model.labels = vv.get(
'labels').copy()
if 'volume_group' in vv:
part_model.volume_group = vv.get(
'vg')
elif 'filesystem' in vv:
fs_info = vv.get('filesystem', {})
part_model.mountpoint = fs_info.get(
'mountpoint', None)
part_model.fstype = fs_info.get(
'fstype', 'ext4')
part_model.mount_options = fs_info.get(
'mount_options', 'defaults')
part_model.fs_uuid = fs_info.get(
'fs_uuid', None)
part_model.fs_label = fs_info.get(
'fs_label', None)
sd.partitions.append(part_model)
model.storage_devices.append(sd)
model.volume_groups = objects.HostVolumeGroupList()
vol_groups = storage.get('volume_groups', {})
for k, v in vol_groups.items():
vg = objects.HostVolumeGroup(name=k)
vg.vg_uuid = v.get('vg_uuid', None)
vg.logical_volumes = objects.HostVolumeList()
model.volume_groups.append(vg)
for vv in v.get('logical_volumes', []):
lv = objects.HostVolume(
name=vv.get('name'))
lv.size = vv.get('size', None)
lv.lv_uuid = vv.get('lv_uuid', None)
if 'filesystem' in vv:
fs_info = vv.get('filesystem', {})
lv.mountpoint = fs_info.get(
'mountpoint', None)
lv.fstype = fs_info.get(
'fstype', 'ext4')
lv.mount_options = fs_info.get(
'mount_options', 'defaults')
lv.fs_uuid = fs_info.get(
'fs_uuid', None)
lv.fs_label = fs_info.get(
'fs_label', None)
vg.logical_volumes.append(lv)
interfaces = spec.get('interfaces', [])
model.interfaces = objects.HostInterfaceList()

10
drydock_provisioner/objects/__init__.py
View File

@ -89,6 +89,11 @@ class Utils(object):
@staticmethod
def merge_lists(child_list, parent_list):
if child_list is None:
return parent_list
if parent_list is None:
return child_list
effective_list = []
@ -123,6 +128,11 @@ class Utils(object):
@staticmethod
def merge_dicts(child_dict, parent_dict):
if child_dict is None:
return parent_dict
if parent_dict is None:
return child_dict
effective_dict = {}

3
drydock_provisioner/objects/base.py
View File

@ -104,6 +104,9 @@ class DrydockObjectListBase(base.ObjectListBase):
@classmethod
def from_basic_list(cls, obj_list):
if obj_list is None:
return None
model_list = cls()
for o in obj_list:

479
drydock_provisioner/objects/hostprofile.py
View File

@ -11,7 +11,8 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
"""Models representing host profiles and constituent parts."""
from copy import deepcopy
import oslo_versionedobjects.fields as obj_fields
@ -27,30 +28,42 @@ class HostProfile(base.DrydockPersistentObject, base.DrydockObject):
VERSION = '1.0'
fields = {
'name': obj_fields.StringField(nullable=False),
'site': obj_fields.StringField(nullable=False),
'source': hd_fields.ModelSourceField(nullable=False),
'parent_profile': obj_fields.StringField(nullable=True),
'hardware_profile': obj_fields.StringField(nullable=True),
'oob_type': obj_fields.StringField(nullable=True),
'oob_parameters': obj_fields.DictOfStringsField(nullable=True),
'storage_layout': obj_fields.StringField(nullable=True),
'bootdisk_device': obj_fields.StringField(nullable=True),
# Consider a custom field for storage size
'bootdisk_root_size': obj_fields.StringField(nullable=True),
'bootdisk_boot_size': obj_fields.StringField(nullable=True),
'partitions': obj_fields.ObjectField(
'HostPartitionList', nullable=True),
'interfaces': obj_fields.ObjectField(
'HostInterfaceList', nullable=True),
'tags': obj_fields.ListOfStringsField(nullable=True),
'owner_data': obj_fields.DictOfStringsField(nullable=True),
'rack': obj_fields.StringField(nullable=True),
'base_os': obj_fields.StringField(nullable=True),
'image': obj_fields.StringField(nullable=True),
'kernel': obj_fields.StringField(nullable=True),
'kernel_params': obj_fields.DictOfStringsField(nullable=True),
'primary_network': obj_fields.StringField(nullable=True),
'name':
obj_fields.StringField(nullable=False),
'site':
obj_fields.StringField(nullable=False),
'source':
hd_fields.ModelSourceField(nullable=False),
'parent_profile':
obj_fields.StringField(nullable=True),
'hardware_profile':
obj_fields.StringField(nullable=True),
'oob_type':
obj_fields.StringField(nullable=True),
'oob_parameters':
obj_fields.DictOfStringsField(nullable=True),
'storage_devices':
obj_fields.ObjectField('HostStorageDeviceList', nullable=True),
'volume_groups':
obj_fields.ObjectField('HostVolumeGroupList', nullable=True),
'interfaces':
obj_fields.ObjectField('HostInterfaceList', nullable=True),
'tags':
obj_fields.ListOfStringsField(nullable=True),
'owner_data':
obj_fields.DictOfStringsField(nullable=True),
'rack':
obj_fields.StringField(nullable=True),
'base_os':
obj_fields.StringField(nullable=True),
'image':
obj_fields.StringField(nullable=True),
'kernel':
obj_fields.StringField(nullable=True),
'kernel_params':
obj_fields.DictOfStringsField(nullable=True),
'primary_network':
obj_fields.StringField(nullable=True),
}
def __init__(self, **kwargs):
@ -114,12 +127,17 @@ class HostProfile(base.DrydockPersistentObject, base.DrydockObject):
self.kernel_params = objects.Utils.merge_dicts(self.kernel_params,
parent.kernel_params)
self.storage_devices = HostStorageDeviceList.from_basic_list(
HostStorageDevice.merge_lists(self.storage_devices,
parent.storage_devices))
self.volume_groups = HostVolumeGroupList.from_basic_list(
HostVolumeGroup.merge_lists(self.volume_groups,
parent.volume_groups))
self.interfaces = HostInterfaceList.from_basic_list(
HostInterface.merge_lists(self.interfaces, parent.interfaces))
self.partitions = HostPartitionList.from_basic_list(
HostPartition.merge_lists(self.partitions, parent.partitions))
self.source = hd_fields.ModelSource.Compiled
return
@ -194,6 +212,12 @@ class HostInterface(base.DrydockObject):
@staticmethod
def merge_lists(child_list, parent_list):
if child_list is None:
return parent_list
if parent_list is None:
return child_list
effective_list = []
if len(child_list) == 0 and len(parent_list) > 0:
@ -281,8 +305,236 @@ class HostInterfaceList(base.DrydockObjectListBase, base.DrydockObject):
fields = {'objects': obj_fields.ListOfObjectsField('HostInterface')}
@base.DrydockObjectRegistry.register
class HostVolumeGroup(base.DrydockObject):
"""Model representing a host volume group."""
VERSION = '1.0'
fields = {
'name': obj_fields.StringField(),
'vg_uuid': obj_fields.StringField(nullable=True),
'logical_volumes': obj_fields.ObjectField(
'HostVolumeList', nullable=True),
}
def __init__(self, **kwargs):
super().__init__(**kwargs)
self.physical_devices = []
def get_name(self):
return self.name
def get_id(self):
return self.name
def add_pv(self, pv):
self.physical_devices.append(pv)
def is_sys(self):
"""Is this the VG for root and/or boot?"""
for lv in getattr(self, 'logical_volumes', []):
if lv.is_sys():
return True
return False
@staticmethod
def merge_lists(child_list, parent_list):
if child_list is None:
return parent_list
if parent_list is None:
return child_list
effective_list = []
if len(child_list) == 0 and len(parent_list) > 0:
for p in parent_list:
pp = deepcopy(p)
pp.source = hd_fields.ModelSource.Compiled
effective_list.append(pp)
elif len(parent_list) == 0 and len(child_list) > 0:
for i in child_list:
if i.get_name().startswith('!'):
continue
else:
ii = deepcopy(i)
ii.source = hd_fields.ModelSource.Compiled
effective_list.append(ii)
elif len(parent_list) > 0 and len(child_list) > 0:
parent_devs = []
for i in parent_list:
parent_name = i.get_name()
parent_devs.append(parent_name)
add = True
for j in child_list:
if j.get_name() == ("!" + parent_name):
add = False
break
elif j.get_name() == parent_name:
p = objects.HostVolumeGroup()
p.name = j.get_name()
inheritable_field_list = ['vg_uuid']
for f in inheritable_field_list:
setattr(p, f,
objects.Utils.apply_field_inheritance(
getattr(j, f, None),
getattr(i, f, None)))
p.partitions = HostPartitionList.from_basic_list(
HostPartition.merge_lists(
getattr(j, 'logical_volumes', None),
getattr(i, 'logical_volumes', None)))
add = False
p.source = hd_fields.ModelSource.Compiled
effective_list.append(p)
if add:
ii = deepcopy(i)
ii.source = hd_fields.ModelSource.Compiled
effective_list.append(ii)
for j in child_list:
if (j.get_name() not in parent_devs
and not j.get_name().startswith("!")):
jj = deepcopy(j)
jj.source = hd_fields.ModelSource.Compiled
effective_list.append(jj)
return effective_list
@base.DrydockObjectRegistry.register
class HostVolumeGroupList(base.DrydockObjectListBase, base.DrydockObject):
VERSION = '1.0'
fields = {'objects': obj_fields.ListOfObjectsField('HostVolumeGroup')}
def add_device_to_vg(self, vg_name, device_name):
for vg in self.objects:
if vg.name == vg_name:
vg.add_pv(device_name)
return
vg = objects.HostVolumeGroup(name=vg_name)
vg.add_pv(device_name)
self.objects.append(vg)
return
@base.DrydockObjectRegistry.register
class HostStorageDevice(base.DrydockObject):
"""Model representing a host physical storage device."""
VERSION = '1.0'
fields = {
'name': obj_fields.StringField(),
'volume_group': obj_fields.StringField(nullable=True),
'labels': obj_fields.DictOfStringsField(nullable=True),
'partitions': obj_fields.ObjectField(
'HostPartitionList', nullable=True),
}
def __init__(self, **kwargs):
super().__init__(**kwargs)
self.physical_devices = []
def get_name(self):
return self.name
def get_id(self):
return self.name
def add_partition(self, partition):
self.partitions.append(partition)
@staticmethod
def merge_lists(child_list, parent_list):
if child_list is None:
return parent_list
if parent_list is None:
return child_list
effective_list = []
if len(child_list) == 0 and len(parent_list) > 0:
for p in parent_list:
pp = deepcopy(p)
pp.source = hd_fields.ModelSource.Compiled
effective_list.append(pp)
elif len(parent_list) == 0 and len(child_list) > 0:
for i in child_list:
if i.get_name().startswith('!'):
continue
else:
ii = deepcopy(i)
ii.source = hd_fields.ModelSource.Compiled
effective_list.append(ii)
elif len(parent_list) > 0 and len(child_list) > 0:
parent_devs = []
for i in parent_list:
parent_name = i.get_name()
parent_devs.append(parent_name)
add = True
for j in child_list:
if j.get_name() == ("!" + parent_name):
add = False
break
elif j.get_name() == parent_name:
p = objects.HostStorageDevice()
p.name = j.get_name()
inherit_field_list = ['volume_group']
for f in inherit_field_list:
setattr(p, f,
objects.Utils.apply_field_inheritance(
getattr(j, f, None),
getattr(i, f, None)))
p.labels = objects.Utils.merge_dicts(
getattr(j, 'labels', None),
getattr(i, 'labels', None))
p.partitions = HostPartitionList.from_basic_list(
HostPartition.merge_lists(
getattr(j, 'partitions', None),
getattr(i, 'partitions', None)))
add = False
p.source = hd_fields.ModelSource.Compiled
effective_list.append(p)
if add:
ii = deepcopy(i)
ii.source = hd_fields.ModelSource.Compiled
effective_list.append(ii)
for j in child_list:
if (j.get_name() not in parent_devs
and not j.get_name().startswith("!")):
jj = deepcopy(j)
jj.source = hd_fields.ModelSource.Compiled
effective_list.append(jj)
return effective_list
@base.DrydockObjectRegistry.register
class HostStorageDeviceList(base.DrydockObjectListBase, base.DrydockObject):
"""Model representing a list of host physical storage devices."""
VERSION = '1.0'
fields = {'objects': obj_fields.ListOfObjectsField('HostStorageDevice')}
@base.DrydockObjectRegistry.register
class HostPartition(base.DrydockObject):
"""Model representing a host GPT partition."""
VERSION = '1.0'
@ -291,7 +543,9 @@ class HostPartition(base.DrydockObject):
obj_fields.StringField(),
'source':
hd_fields.ModelSourceField(),
'device':
'bootable':
obj_fields.BooleanField(default=False),
'volume_group':
obj_fields.StringField(nullable=True),
'part_uuid':
obj_fields.UUIDField(nullable=True),
@ -307,12 +561,10 @@ class HostPartition(base.DrydockObject):
obj_fields.UUIDField(nullable=True),
'fs_label':
obj_fields.StringField(nullable=True),
'selector':
obj_fields.ObjectField('HardwareDeviceSelector', nullable=True),
}
def __init__(self, **kwargs):
super(HostPartition, self).__init__(**kwargs)
super().__init__(**kwargs)
def get_device(self):
return self.device
@ -324,17 +576,11 @@ class HostPartition(base.DrydockObject):
def get_name(self):
return self.name
# The device attribute may be hardware alias that translates to a
# physical device address. If the device attribute does not match an
# alias, we assume it directly identifies a OS device name. When the
# apply_hardware_profile method is called on the parent Node of this
# device, the selector will be decided and applied
def set_selector(self, selector):
self.selector = selector
def get_selector(self):
return self.selector
def is_sys(self):
"""Is this partition for root and/or boot?"""
if self.mountpoint is not None and self.mountpoint in ['/', '/boot']:
return True
return False
"""
Merge two lists of HostPartition models with child_list taking
@ -345,6 +591,12 @@ class HostPartition(base.DrydockObject):
@staticmethod
def merge_lists(child_list, parent_list):
if child_list is None:
return parent_list
if parent_list is None:
return child_list
effective_list = []
if len(child_list) == 0 and len(parent_list) > 0:
@ -362,8 +614,16 @@ class HostPartition(base.DrydockObject):
effective_list.append(ii)
elif len(parent_list) > 0 and len(child_list) > 0:
inherit_field_list = [
"device", "part_uuid", "size", "mountpoint", "fstype",
"mount_options", "fs_uuid", "fs_label"
"device",
"part_uuid",
"size",
"mountpoint",
"fstype",
"mount_options",
"fs_uuid",
"fs_label",
"volume_group",
"bootable",
]
parent_partitions = []
for i in parent_list:
@ -392,7 +652,7 @@ class HostPartition(base.DrydockObject):
effective_list.append(ii)
for j in child_list:
if (j.get_name() not in parent_list
if (j.get_name() not in parent_partitions
and not j.get_name().startswith("!")):
jj = deepcopy(j)
jj.source = hd_fields.ModelSource.Compiled
@ -407,3 +667,130 @@ class HostPartitionList(base.DrydockObjectListBase, base.DrydockObject):
VERSION = '1.0'
fields = {'objects': obj_fields.ListOfObjectsField('HostPartition')}
@base.DrydockObjectRegistry.register
class HostVolume(base.DrydockObject):
"""Model representing a host logical volume."""
VERSION = '1.0'
fields = {
'name':
obj_fields.StringField(),
'source':
hd_fields.ModelSourceField(),
'lv_uuid':
obj_fields.UUIDField(nullable=True),
'size':
obj_fields.StringField(nullable=True),
'mountpoint':
obj_fields.StringField(nullable=True),
'fstype':
obj_fields.StringField(nullable=True, default='ext4'),
'mount_options':
obj_fields.StringField(nullable=True, default='defaults'),
'fs_uuid':
obj_fields.UUIDField(nullable=True),
'fs_label':
obj_fields.StringField(nullable=True),
}
def __init__(self, **kwargs):
super().__init__(**kwargs)
# HostVolume keyed by name
def get_id(self):
return self.get_name()
def get_name(self):
return self.name
def is_sys(self):
"""Is this LV for root and/or boot?"""
if self.mountpoint is not None and self.mountpoint in ['/', '/boot']:
return True
return False
"""
Merge two lists of HostVolume models with child_list taking
priority when conflicts. If a member of child_list has a name
beginning with '!' it indicates that HostPartition should be
removed from the merged list
"""
@staticmethod
def merge_lists(child_list, parent_list):
if child_list is None:
return parent_list
if parent_list is None:
return child_list
effective_list = []
if len(child_list) == 0 and len(parent_list) > 0:
for p in parent_list:
pp = deepcopy(p)
pp.source = hd_fields.ModelSource.Compiled
effective_list.append(pp)
elif len(parent_list) == 0 and len(child_list) > 0:
for i in child_list:
if i.get_name().startswith('!'):
continue
else:
ii = deepcopy(i)
ii.source = hd_fields.ModelSource.Compiled
effective_list.append(ii)
elif len(parent_list) > 0 and len(child_list) > 0:
inherit_field_list = [
"lv_uuid",
"size",
"mountpoint",
"fstype",
"mount_options",
"fs_uuid",
"fs_label",
]
parent_volumes = []
for i in parent_list:
parent_name = i.get_name()
parent_volumes.append(parent_name)
add = True
for j in child_list:
if j.get_name() == ("!" + parent_name):
add = False
break
elif j.get_name() == parent_name:
p = objects.HostPartition()
p.name = j.get_name()
for f in inherit_field_list:
setattr(p, f,
objects.Utils.apply_field_inheritance(
getattr(j, f, None),
getattr(i, f, None)))
add = False
p.source = hd_fields.ModelSource.Compiled
effective_list.append(p)
if add:
ii = deepcopy(i)
ii.source = hd_fields.ModelSource.Compiled
effective_list.append(ii)
for j in child_list:
if (j.get_name() not in parent_volumes
and not j.get_name().startswith("!")):
jj = deepcopy(j)
jj.source = hd_fields.ModelSource.Compiled
effective_list.append(jj)
return effective_list
@base.DrydockObjectRegistry.register
class HostVolumeList(base.DrydockObjectListBase, base.DrydockObject):
VERSION = '1.0'
fields = {'objects': obj_fields.ListOfObjectsField('HostVolume')}

22
drydock_provisioner/objects/node.py
View File

@ -14,9 +14,7 @@
#
# Models for drydock_provisioner
#
import logging
from copy import deepcopy
"""Drydock model of a baremetal node."""
from oslo_versionedobjects import fields as ovo_fields
@ -96,6 +94,24 @@ class BaremetalNode(drydock_provisioner.objects.hostprofile.HostProfile):
return None
def find_fs_block_device(self, fs_mount=None):
if not fs_mount:
return (None, None)
if self.volume_groups is not None:
for vg in self.volume_groups:
if vg.logical_volumes is not None:
for lv in vg.logical_volumes:
if lv.mountpoint is not None and lv.mountpoint == fs_mount:
return (vg, lv)
if self.storage_devices is not None:
for sd in self.storage_devices:
if sd.partitions is not None:
for p in sd.partitions:
if p.mountpoint is not None and p.mountpoint == fs_mount:
return (sd, p)
return (None, None)
@base.DrydockObjectRegistry.register
class BaremetalNodeList(base.DrydockObjectListBase, base.DrydockObject):

130
drydock_provisioner/orchestrator/__init__.py
View File

@ -464,7 +464,6 @@ class Orchestrator(object):
hd_fields.ActionResult.PartialSuccess,
hd_fields.ActionResult.Failure
]:
# TODO(sh8121att) This threshold should be a configurable default and tunable by task API
if node_identify_attempts > max_attempts:
failed = True
break
@ -580,12 +579,55 @@ class Orchestrator(object):
]:
failed = True
node_storage_task = None
if len(node_networking_task.result_detail['successful_nodes']) > 0:
self.logger.info(
"Found %s successfully networked nodes, configuring platform."
"Found %s successfully networked nodes, configuring storage."
% (len(node_networking_task.result_detail[
'successful_nodes'])))
node_storage_task = self.create_task(
tasks.DriverTask,
parent_Task_id=task.get_id(),
design_id=design_id,
action=hd_fields.OrchestratorAction.ApplyNodeStorage,
task_scope={
'node_names':
node_networking_task.result_detail['successful_nodes']
})
self.logger.info(
"Starting node driver task %s to configure node storage." %
(node_storage_task.get_id()))
node_driver.execute_task(node_storage_task.get_id())
node_storage_task = self.state_manager.get_task(
node_storage_task.get_id())
if node_storage_task.get_result() in [
hd_fields.ActionResult.Success,
hd_fields.ActionResult.PartialSuccess
]:
worked = True
elif node_storage_task.get_result() in [
hd_fields.ActionResult.Failure,
hd_fields.ActionResult.PartialSuccess
]:
failed = True
else:
self.logger.warning(
"No nodes successfully networked, skipping storage configuration subtask."
)
node_platform_task = None
if (node_storage_task is not None and
len(node_storage_task.result_detail['successful_nodes']) >
0):
self.logger.info(
"Configured storage on %s nodes, configuring platform." %
(len(node_storage_task.result_detail['successful_nodes'])))
node_platform_task = self.create_task(
tasks.DriverTask,
parent_task_id=task.get_id(),
@ -593,7 +635,7 @@ class Orchestrator(object):
action=hd_fields.OrchestratorAction.ApplyNodePlatform,
task_scope={
'node_names':
node_networking_task.result_detail['successful_nodes']
node_storage_task.result_detail['successful_nodes']
})
self.logger.info(
"Starting node driver task %s to configure node platform."
@ -614,49 +656,49 @@ class Orchestrator(object):
hd_fields.ActionResult.PartialSuccess
]:
failed = True
if len(node_platform_task.result_detail['successful_nodes']
) > 0:
self.logger.info(
"Configured platform on %s nodes, starting deployment."
% (len(node_platform_task.result_detail[
'successful_nodes'])))
node_deploy_task = self.create_task(
tasks.DriverTask,
parent_task_id=task.get_id(),
design_id=design_id,
action=hd_fields.OrchestratorAction.DeployNode,
task_scope={
'node_names':
node_platform_task.result_detail[
'successful_nodes']
})
self.logger.info(
"Starting node driver task %s to deploy nodes." %
(node_deploy_task.get_id()))
node_driver.execute_task(node_deploy_task.get_id())
node_deploy_task = self.state_manager.get_task(
node_deploy_task.get_id())
if node_deploy_task.get_result() in [
hd_fields.ActionResult.Success,
hd_fields.ActionResult.PartialSuccess
]:
worked = True
elif node_deploy_task.get_result() in [
hd_fields.ActionResult.Failure,
hd_fields.ActionResult.PartialSuccess
]:
failed = True
else:
self.logger.warning(
"Unable to configure platform on any nodes, skipping deploy subtask"
)
else:
self.logger.warning(
"No nodes successfully networked, skipping platform configuration subtask"
"No nodes with storage configuration, skipping platform configuration subtask."
)
node_deploy_task = None
if node_platform_task is not None and len(
node_platform_task.result_detail['successful_nodes']) > 0:
self.logger.info(
"Configured platform on %s nodes, starting deployment." %
(len(node_platform_task.result_detail['successful_nodes'])
))
node_deploy_task = self.create_task(
tasks.DriverTask,
parent_task_id=task.get_id(),
design_id=design_id,
action=hd_fields.OrchestratorAction.DeployNode,
task_scope={
'node_names':
node_platform_task.result_detail['successful_nodes']
})
self.logger.info(
"Starting node driver task %s to deploy nodes." %
(node_deploy_task.get_id()))
node_driver.execute_task(node_deploy_task.get_id())
node_deploy_task = self.state_manager.get_task(
node_deploy_task.get_id())
if node_deploy_task.get_result() in [
hd_fields.ActionResult.Success,
hd_fields.ActionResult.PartialSuccess
]:
worked = True
elif node_deploy_task.get_result() in [
hd_fields.ActionResult.Failure,
hd_fields.ActionResult.PartialSuccess
]:
failed = True
else:
self.logger.warning(
"Unable to configure platform on any nodes, skipping deploy subtask"
)
final_result = None

22
drydock_provisioner/orchestrator/readme.md
View File

@ -24,13 +24,19 @@ is compatible with the physical state of the site.
#### Validations ####
* All baremetal nodes have an address, either static or DHCP, for all networks they are attached to.
* No static IP assignments are duplicated
* No static IP assignments are outside of the network they are targetted for
* All IP assignments are within declared ranges on the network
* Networks assigned to each node's interface are within the set of of the attached link's allowed_networks
* No network is allowed on multiple network links
* Boot drive is above minimum size
* Networking
** No static IP assignments are duplicated
** No static IP assignments are outside of the network they are targetted for
** All IP assignments are within declared ranges on the network
** Networks assigned to each node's interface are within the set of of the attached link's allowed\_networks
** No network is allowed on multiple network links
** Network MTU is equal or less than NetworkLink MTU
** MTU values are sane
* Storage
** Boot drive is above minimum size
** Root drive is above minimum size
** No physical device specifies a target VG and a partition list
** No partition specifies a target VG and a filesystem
### VerifySite ###
@ -102,4 +108,4 @@ Destroy current node configuration and rebootstrap from scratch
Based on the requested task and the current known state of a node
the orchestrator will call the enabled downstream drivers with one
or more tasks. Each call will provide the driver with the desired
state (the applied model) and current known state (the build model).
state (the applied model) and current known state (the build model).

196
tests/unit/test_maasdriver_calculate_bytes.py Normal file
View File

@ -0,0 +1,196 @@
# Copyright 2017 AT&T Intellectual Property. All other rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
'''Tests for the maasdriver calculate_bytes routine.'''
import pytest
import math
from drydock_provisioner import error
from drydock_provisioner.drivers.node.maasdriver.driver import MaasTaskRunner
from drydock_provisioner.drivers.node.maasdriver.models.blockdev import BlockDevice
from drydock_provisioner.drivers.node.maasdriver.models.volumegroup import VolumeGroup
class TestCalculateBytes():
def test_calculate_m_label(self):
'''Convert megabyte labels to x * 10^6 bytes.'''
size_str = '15m'
drive_size = 20 * 1000 * 1000
drive = BlockDevice(None, size=drive_size, available_size=drive_size)
calc_size = MaasTaskRunner.calculate_bytes(size_str=size_str, context=drive)
assert calc_size == 15 * 1000 * 1000
def test_calculate_mb_label(self):
'''Convert megabyte labels to x * 10^6 bytes.'''
size_str = '15mb'
drive_size = 20 * 1000 * 1000
drive = BlockDevice(None, size=drive_size, available_size=drive_size)
calc_size = MaasTaskRunner.calculate_bytes(size_str=size_str, context=drive)
assert calc_size == 15 * 1000 * 1000
def test_calculate_M_label(self):
'''Convert megabyte labels to x * 10^6 bytes.'''
size_str = '15M'
drive_size = 20 * 1000 * 1000
drive = BlockDevice(None, size=drive_size, available_size=drive_size)
calc_size = MaasTaskRunner.calculate_bytes(size_str=size_str, context=drive)
assert calc_size == 15 * 1000 * 1000
def test_calculate_MB_label(self):
'''Convert megabyte labels to x * 10^6 bytes.'''
size_str = '15MB'
drive_size = 20 * 1000 * 1000
drive = BlockDevice(None, size=drive_size, available_size=drive_size)
calc_size = MaasTaskRunner.calculate_bytes(size_str=size_str, context=drive)
assert calc_size == 15 * 1000 * 1000
def test_calculate_g_label(self):
'''Convert gigabyte labels to x * 10^9 bytes.'''
size_str = '15g'
drive_size = 20 * 1000 * 1000 * 1000
drive = BlockDevice(None, size=drive_size, available_size=drive_size)
calc_size = MaasTaskRunner.calculate_bytes(size_str=size_str, context=drive)
assert calc_size == 15 * 1000 * 1000 * 1000
def test_calculate_gb_label(self):
'''Convert gigabyte labels to x * 10^9 bytes.'''
size_str = '15gb'
drive_size = 20 * 1000 * 1000 * 1000
drive = BlockDevice(None, size=drive_size, available_size=drive_size)
calc_size = MaasTaskRunner.calculate_bytes(size_str=size_str, context=drive)
assert calc_size == 15 * 1000 * 1000 * 1000
def test_calculate_G_label(self):
'''Convert gigabyte labels to x * 10^9 bytes.'''
size_str = '15G'
drive_size = 20 * 1000 * 1000 * 1000
drive = BlockDevice(None, size=drive_size, available_size=drive_size)
calc_size = MaasTaskRunner.calculate_bytes(size_str=size_str, context=drive)
assert calc_size == 15 * 1000 * 1000 * 1000
def test_calculate_GB_label(self):
'''Convert gigabyte labels to x * 10^9 bytes.'''
size_str = '15GB'
drive_size = 20 * 1000 * 1000 * 1000
drive = BlockDevice(None, size=drive_size, available_size=drive_size)
calc_size = MaasTaskRunner.calculate_bytes(size_str=size_str, context=drive)
assert calc_size == 15 * 1000 * 1000 * 1000
def test_calculate_t_label(self):
'''Convert terabyte labels to x * 10^12 bytes.'''
size_str = '15t'
drive_size = 20 * 1000 * 1000 * 1000 * 1000
drive = BlockDevice(None, size=drive_size, available_size=drive_size)
calc_size = MaasTaskRunner.calculate_bytes(size_str=size_str, context=drive)
assert calc_size == 15 * 1000 * 1000 * 1000 * 1000
def test_calculate_tb_label(self):
'''Convert terabyte labels to x * 10^12 bytes.'''
size_str = '15tb'
drive_size = 20 * 1000 * 1000 * 1000 * 1000
drive = BlockDevice(None, size=drive_size, available_size=drive_size)
calc_size = MaasTaskRunner.calculate_bytes(size_str=size_str, context=drive)
assert calc_size == 15 * 1000 * 1000 * 1000 * 1000
def test_calculate_T_label(self):
'''Convert terabyte labels to x * 10^12 bytes.'''
size_str = '15T'
drive_size = 20 * 1000 * 1000 * 1000 * 1000
drive = BlockDevice(None, size=drive_size, available_size=drive_size)
calc_size = MaasTaskRunner.calculate_bytes(size_str=size_str, context=drive)
assert calc_size == 15 * 1000 * 1000 * 1000 * 1000
def test_calculate_TB_label(self):
'''Convert terabyte labels to x * 10^12 bytes.'''
size_str = '15TB'
drive_size = 20 * 1000 * 1000 * 1000 * 1000
drive = BlockDevice(None, size=drive_size, available_size=drive_size)
calc_size = MaasTaskRunner.calculate_bytes(size_str=size_str, context=drive)
assert calc_size == 15 * 1000 * 1000 * 1000 * 1000
def test_calculate_percent_blockdev(self):
'''Convert a percent of total blockdev space to explicit byte count.'''
drive_size = 20 * 1000 * 1000 # 20 mb drive
part_size = math.floor(.2 * drive_size) # calculate 20% of drive size
size_str = '20%'
drive = BlockDevice(None, size=drive_size, available_size=drive_size)
calc_size = MaasTaskRunner.calculate_bytes(size_str=size_str, context=drive)
assert calc_size == part_size
def test_calculate_percent_vg(self):
'''Convert a percent of total blockdev space to explicit byte count.'''
vg_size = 20 * 1000 * 1000 # 20 mb drive
lv_size = math.floor(.2 * vg_size) # calculate 20% of drive size
size_str = '20%'
vg = VolumeGroup(None, size=vg_size, available_size=vg_size)
calc_size = MaasTaskRunner.calculate_bytes(size_str=size_str, context=vg)
assert calc_size == lv_size
def test_calculate_overprovision(self):
'''When calculated space is higher than available space, raise an exception.'''
vg_size = 20 * 1000 * 1000 # 20 mb drive
vg_available = 10 # 10 bytes available
lv_size = math.floor(.8 * vg_size) # calculate 80% of drive size
size_str = '80%'
vg = VolumeGroup(None, size=vg_size, available_size=vg_available)
with pytest.raises(error.NotEnoughStorage):
calc_size = MaasTaskRunner.calculate_bytes(size_str=size_str, context=vg)
def test_calculate_min_label(self):
'''Adding the min marker '>' should provision all available space.'''
vg_size = 20 * 1000 * 1000 # 20 mb drive
vg_available = 15 * 1000 * 1000
lv_size = math.floor(.1 * vg_size) # calculate 20% of drive size
size_str = '>10%'
vg = VolumeGroup(None, size=vg_size, available_size=vg_available)
calc_size = MaasTaskRunner.calculate_bytes(size_str=size_str, context=vg)
assert calc_size == vg_available

107
tests/unit/test_orch_oob.py
View File

@ -1,107 +0,0 @@
# Copyright 2017 AT&T Intellectual Property. All other rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# Generic testing for the orchestrator
#
import pytest
#from pytest_mock import mocker
#import mock
import os
import shutil
import uuid
from drydock_provisioner.ingester import Ingester
import drydock_provisioner.orchestrator as orch
import drydock_provisioner.objects.fields as hd_fields
import drydock_provisioner.statemgmt as statemgmt
import drydock_provisioner.objects as objects
import drydock_provisioner.objects.task as task
import drydock_provisioner.drivers as drivers
import drydock_provisioner.ingester.plugins.yaml as yaml_ingester
class TestClass(object):
design_id = str(uuid.uuid4())
# sthussey None of these work right until I figure out correct
# mocking of pyghmi
def test_oob_verify_all_node(self, loaded_design):
#mocker.patch('pyghmi.ipmi.private.session.Session')
#mocker.patch.object('pyghmi.ipmi.command.Command','get_asset_tag')
orchestrator = orch.Orchestrator(state_manager=loaded_design,
enabled_drivers={'oob': 'drydock_provisioner.drivers.oob.pyghmi_driver.PyghmiDriver'})
orch_task = orchestrator.create_task(task.OrchestratorTask,
site='sitename',
design_id=self.design_id,
action=hd_fields.OrchestratorAction.VerifyNode)
orchestrator.execute_task(orch_task.get_id())
orch_task = loaded_design.get_task(orch_task.get_id())
assert True
"""
def test_oob_prepare_all_nodes(self, loaded_design):
#mocker.patch('pyghmi.ipmi.private.session.Session')
#mocker.patch.object('pyghmi.ipmi.command.Command','set_bootdev')
orchestrator = orch.Orchestrator(state_manager=loaded_design,
enabled_drivers={'oob': 'drydock_provisioner.drivers.oob.pyghmi_driver.PyghmiDriver'})
orch_task = orchestrator.create_task(task.OrchestratorTask,
site='sitename',
action=enum.OrchestratorAction.PrepareNode)
orchestrator.execute_task(orch_task.get_id())
#assert pyghmi.ipmi.command.Command.set_bootdev.call_count == 3
#assert pyghmi.ipmi.command.Command.set_power.call_count == 6
"""
@pytest.fixture(scope='module')
def loaded_design(self, input_files):
objects.register_all()
input_file = input_files.join("oob.yaml")
design_state = statemgmt.DesignState()
design_data = objects.SiteDesign(id=self.design_id)
design_state.post_design(design_data)
ingester = Ingester()
ingester.enable_plugins([yaml_ingester.YamlIngester])
ingester.ingest_data(plugin_name='yaml', design_state=design_state,
design_id=self.design_id, filenames=[str(input_file)])
return design_state
@pytest.fixture(scope='module')
def input_files(self, tmpdir_factory, request):
tmpdir = tmpdir_factory.mktemp('data')
samples_dir = os.path.dirname(str(request.fspath)) + "../yaml_samples"
samples = os.listdir(samples_dir)
for f in samples:
src_file = samples_dir + "/" + f
dst_file = str(tmpdir) + "/" + f
shutil.copyfile(src_file, dst_file)
return tmpdir

57
tests/yaml_samples/fullsite.yaml
View File

@ -299,33 +299,36 @@ spec:
credential: admin
# Specify storage layout of base OS. Ceph out of scope
storage:
# How storage should be carved up: lvm (logical volumes), flat
# (single partition)
layout: lvm
# Info specific to the boot and root disk/partitions
bootdisk:
# Device will specify an alias defined in hwdefinition.yaml
device: primary_boot
# For LVM, the size of the partition added to VG as a PV
# For flat, the size of the partition formatted as ext4
root_size: 50g
# The /boot partition. If not specified, /boot will in root
boot_size: 2g
# Info for additional partitions. Need to balance between
# flexibility and complexity
partitions:
- name: logs
device: primary_boot
# Partition uuid if needed
part_uuid: 84db9664-f45e-11e6-823d-080027ef795a
size: 10g
# Optional, can carve up unformatted block devices
mountpoint: /var/log
fstype: ext4
mount_options: defaults
# Filesystem UUID or label can be specified. UUID recommended
fs_uuid: cdb74f1c-9e50-4e51-be1d-068b0e9ff69e
fs_label: logs
physical_devices:
sda:
labels:
role: rootdisk
partitions:
- name: root
size: 20g
bootable: true
filesystem:
mountpoint: '/'
fstype: 'ext4'
mount_options: 'defaults'
- name: boot
size: 1g
bootable: false
filesystem:
mountpoint: '/boot'
fstype: 'ext4'
mount_options: 'defaults'
sdb:
volume_group: 'log_vg'
volume_groups:
log_vg:
logical_volumes:
- name: 'log_lv'
size: '500m'
filesystem:
mountpoint: '/var/log'
fstype: 'xfs'
mount_options: 'defaults'
# Platform (Operating System) settings
platform:
image: ubuntu_16.04

2
tox.ini
View File

@ -33,6 +33,6 @@ commands = flake8 \
{posargs}
[flake8]
ignore=E302,H306,D101,D102,D103,D104
ignore=E302,H306,H304,D101,D102,D103,D104
exclude= venv,.venv,.git,.idea,.tox,*.egg-info,*.eggs,bin,dist,./build/
max-line-length=119