x/solar
Code Issues Proposed changes

Merge branch 'master' into docs_handler_ansible

Browse Source
This commit is contained in:
Jedrzej Nowak 2015-12-01 16:56:20 +01:00
commit cc98cc4213
9 changed files with 192 additions and 76 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
docs/index.rst
View File

@ -14,6 +14,7 @@ Contents:
source/glossary
source/resource
source/orchestration
source/transports
source/handler_ansible
source/examples
source/deployment_plan

34
docs/source/deployment_plan.rst
View File

@ -3,8 +3,9 @@
Preparing deployment plan
=========================
Solar allows to make transitions between different versions of infrastructure
based on changes found by solar control plane and configured events between this changes.
Solar allows you to make transitions between different versions of
infrastructure based on changes found by solar control plane and events
configured between this changes.
Required information
----------------------
@ -27,11 +28,11 @@ user (human/program). Examples of changes are:
Staged changes
---------------
On demand solar runs procedure that will find all changed from last deployment
resources, and using transitions from solar state machine will determine
list of actions.
On demand solar runs procedure that will find all resources changed from last
deployment and will determine list of actions using transitions from solar
state machine .
This procedure performed by ::
This procedure is performed by ::
solar changes stage -d
@ -58,7 +59,7 @@ It prints information like ::
++ collation: utf8_general_ci
++ location_id: 694b35afa622da857f95e14a21599d81
At this point information is stored as a list, and doensnt know anything
At this point information is stored as a list, and user doesn't know anything
about dependencies between found changes.
Events usage
@ -66,21 +67,22 @@ Events usage
For events definition see :ref:`res-event-term`.
Events used during this procedure to insert dependencies between found changes,
and add new actions that are reactions for changes.
Events are used during this procedure to insert dependencies between found
changes, and to add new actions that are reactions for changes.
Example of dependency between changes would be *nova service* that depends
on successful creation of *database*.
And for removal we might add dependencies that will allow reverse
order, e.g when removing *nova service* and *database*, *database* will be
removed only after successfull *nova service*.
Can be specified as ::
For removal we might add dependencies that will allow reverse order, e.g. when
removing *nova service* and *database*, *database* will be removed only after
successful *nova service* removal.
This can be specified as ::
Dependency database1.run -> nova1.run
Dependency nova1.remove -> database1.remove
Reaction allows to construct more advacned worklows that will take into
Reaction allows to construct more advanced workflows that will take into
account not only changes, but also arbitrary actions for resources in solar.
Good example of usage is provisioning procedure, where reboot must be
@ -100,9 +102,9 @@ of deployment plan for current version of infrastructure ::
solar changes process
After this command plan can be viewed by ::
After this deployment command plan can be viewed by ::
# graphviz representataion
# graphviz representation
solar orch dg last
# text report

39
docs/source/examples.rst
View File

@ -6,29 +6,30 @@ Examples
Create resource for the puppet handler
--------------------------------------
Let's create an example :ref:`resource-term` for the puppet :ref:`res-handler-term`
version 1 [#]_. The resource should install and configure OpenStack Nova API
service.
Let's create an example :ref:`resource-term` for the puppet
:ref:`res-handler-term` version 1 [#]_. The resource should install and
configure OpenStack Nova API service.
.. [#] There is also puppet handler version 2 but it is out of the scope
of this example.
Step 1: Find an apropriate puppet module
Step 1: Find an appropriate puppet module
++++++++++++++++++++++++++++++++++++++++
The `Puppet OpenStack <https://wiki.openstack.org/wiki/Puppet>`_
module for `Nova <https://github.com/openstack/puppet-nova>`_
provides all of the required functionality.
Step 2: Define granularity scope for a resource
Step 2: Define granularity level for a resource
+++++++++++++++++++++++++++++++++++++++++++++++
One may want to implement resources as atomic entities doing each its only
task, like running one and only puppet manifest [#]_. Or as a single entity doing
all required tasks instead. In order to configure and run the Nova API service
at least two manifests should be executed: the
One may want to implement resources as atomic entities doing their only single
task, like running one and only puppet manifest [#]_. Other option might be
single entity doing all required tasks instead. In order to configure and run
the Nova API service at least two manifests should be executed:
`init.pp <https://github.com/openstack/puppet-nova/blob/master/manifests/init.pp>`_
and the `api.pp <https://github.com/openstack/puppet-nova/blob/master/manifests/api.pp>`_ [#]_.
and
`api.pp <https://github.com/openstack/puppet-nova/blob/master/manifests/api.pp>`_ [#]_.
.. [#] Puppet manifests may contain references to externally defined classess
or services in the catalog. Keep that in mind then designing the resource.
@ -40,26 +41,26 @@ Assuming the atomic tasks approach, the example resource for Nova API service
should only use the `api.pp` manifest. Note that the puppet handler is normally
executed in its own isolated puppet catalog with its specific hiera data only.
This assumes every puppet manifest called by every action to be executed as a
separate puppet run and shares nothing.
separate puppet run and shares nothing with other tasks.
Step 3: Define resource inputs
++++++++++++++++++++++++++++++
Once the granularity scope of the resource was clearly defined, one should
Once the granularity level of the resource is clearly defined, one should
define the resource's :ref:`res-input-term` data. The puppet class `nova::api`
contains a lots of parameters. It looks reasonable to use them as the resource
contains lots of parameters. It looks reasonable to use them as the resource
inputs as is.
.. note ::
There is a `helper script <https://github.com/bogdando/convert_puppet_parameters>`_
to convert a puppet class parameters into the format expeted by the
to convert a puppet class parameters into the format expected by the
`meta.yaml` inputs file.
Step 4: Implement the basic action run
Step 4: Implement basic action run
++++++++++++++++++++++++++++++++++++++
Each resource should have all of the mandatory actions defined. In this example
we define only the :ref:`ref-action-term` `run`. For the example Nova API
we define only the :ref:`ref-action-term` `run`. With the example of Nova API
resource, the action run should:
- fetch the resource inputs from the hiera [#]_ ::
@ -86,7 +87,7 @@ resource, the action run should:
}
.. note ::
Otherwise, the called class would assume the package and exec are
Otherwise, called class would assume the package and exec are
already included in the catalog by the `init.pp`. And would fail as
there is no `class { 'nova': }` call expected for the Nova API resource
action run.
@ -101,7 +102,7 @@ Step 5: Think of the rest of the resource actions
One should also design other actions for the resource. Mandatory are only
`run`, `update` and `remove`. There might be additional ones like `on-fail`,
`on-retry` or whichever are actually required to implement expected behavior.
For the given API resource there is none specific actions expected and there
For the given API resource there are no specific actions expected and there
is nothing to do for the action remove. For the action update, it is likely
the same steps should be done as for the action run.
@ -116,7 +117,7 @@ Step 7: Think of the deployment composition
The deployment composition is which resources should be used and in which order
it should be executed to achive the expected result, which is a successfull
:ref:`deploy-plan-term`. For the given example, the composition may be as the
:ref:`deploy-plan-term`. For the given example, the composition may be as
following:
- Install and configure MySQL DB [#]_

51
docs/source/glossary.rst
View File

@ -9,28 +9,33 @@ Solar Glossary
Resource
========
You can learn more about it in :ref:`Resource Details <resource_details>`
Resource is an abstraction of item in system managed by Solar. It's a basic
building block used to assemble your system. Almost every entity in Solar
is a resource.
You can learn more about it in :ref:`resource details <resource_details>`
.. _res-input-term:
Input
-----
Resource configuration that will be used in actions, handlers and orchestration.
All known inputs should be provided in meta.yaml
Resource configuration that will be used in actions, handlers and
orchestration. All known inputs for a resource should be defined in meta.yaml
.. _res-connection-term:
Connection
----------
Allows to build hierarchy between inputs of several resources,
parent value will be always used in child while connection is created.
If connection will be removed - original value of child will be preserved.
Allows to build hierarchy between inputs of several resources, parent value
will be always used in child while connection is created. If connection is
removed - original value of child will be preserved.
.. _res-action-term:
Action
------
Solar wraps deployment code into actions with specific names.
Solar wraps deployment code into actions with specific names. Actions are
executed from the resource.
.. _res-tag-term:
@ -44,20 +49,20 @@ used for different user operations.
Handler
=======
Layer that responsible for action execution and tracking result.
Layer responsible for action execution and tracking results.
.. _res-transports-term:
Transport
=========
Used in handlers to communicate with managed by solar hosts. List of transports
should be added to a node. Transports will be added to a resource by means
of transports id.
Used in handlers to communicate with hosts managed by Solar.
Two different types of transports are used: run and sync.
Run transport - reponsible for running command on remote host.
Sync transport - uploads required information.
.. seealso::
:ref:`More details about transports <transports_details>`
.. _location-id-term:
location_id
-----------
@ -65,12 +70,15 @@ Used in transport layer to find ip address of a node. ::
'location_id': '96bc779540d832284680785ecd948a2d'
.. _transports-id-term:
transports_id
-------------
Used to find transports array that will be used for transport selection. ::
'transports_id': '3889e1790e68b80b4f255cf0e13494b1'
BAT transport
-------------
According to preferences solar will choose best available transport for
@ -85,12 +93,11 @@ Used in solar to describe all possible transitions between resources changes.
Each event allows to specify two points of transitions, condition of this
transition and type of event.
Right now we are supporting 2 types of events.
Right now we are supporting 2 types of events:
1. Dependency
Inserts edge between 2 changes into the deployment plan.
2. Reaction
Inserts change specified in reaction and makes edge between parent and child.
1. Dependency - inserts edge between 2 changes into the deployment plan.
2. Reaction - inserts change specified in reaction and makes edge between
parent and child.
Example ::
@ -106,7 +113,7 @@ Example ::
Virtual resource/template
=========================
Composition layer that allows to:
Composition layer that allows user to:
- group resources
- specify connections between inputs
@ -130,9 +137,9 @@ History
After action that is related to change will be executed - it will be moved to
history with same uuid.
Commited resource data
Committed resource data
----------------------
After each succesfull change commited copy of resource data will be updated
After each successful change committed copy of resource data will be updated
with diff of that change.
.. _orch-term:

24
docs/source/orchestration.rst
View File

@ -5,7 +5,6 @@ Deployment operations
Stage changes
-------------
After user created all required resource - it is possible to automatically
detect which resource requires changes with ::
@ -14,30 +13,30 @@ detect which resource requires changes with ::
History
-------
After changes are staged - they will be used to populate history which can be seen
with command (*n* option used to limit number of items, -1 will return all changes) ::
After changes are staged - they will be used to populate history which can be
previewed with command (*n* option used to limit number of items, -1 will
return all changes) ::
solar changes history -n 5
Prepare deployment plan
-----------------------
User is able to generate deployment scenario based on changes found by system log. ::
User is able to generate deployment scenario based on changes found by system
log. ::
solar changes process
This command will prepare deployment graph, and return uid of deployment graph to
work with.
This command will prepare deployment graph, and return uid of deployment graph
to work with.
All commands that are able to manipulate deployment graph located in
*orch* namespace.
.. tip::
Solar writes returned deployment graph uid into special file (`.solar_cli_uids`), it
allows you to use `last` instead of full returned uid:
`solar orch report <uid>` becomes `solar orch report last`
Solar writes returned deployment graph uid into special file
(`.solar_cli_uids`), it allows you to use `last` instead of full returned
uid: `solar orch report <uid>` becomes `solar orch report last`
Report
@ -84,6 +83,7 @@ All tasks will be returned to PENDING state, and deployment will be restarted ::
Retry deployment
----------------
Orchestrator will reset all ERROR tasks to PENDING state and restart deployment ::
Orchestrator will reset all ERROR tasks to PENDING state and restart
deployment ::
solar orch retry <uid>

49
docs/source/resource.rst
View File

@ -3,7 +3,15 @@
Resource
========
Resource is one of the key Solar components. Resoruce definition takes place in ``meta.yaml`` file.
Resource is one of the key Solar components, almost every entity in Solar is a
resource. Examples are:
* packages
* services
Resources are defined in ``meta.yaml`` file. This file is responsible for basic
configuration of given resource. Below is an explanation what constitutes
typical resource.
Basic resource structure
@ -23,14 +31,34 @@ Handler
.. TODO: add link to handlers doc there
Layer that is responsible for action execution. You need to specify handler per resource, definition in ``meta.yaml`` looks like ::
Pluggable layer that is responsible for executing an action on resource. You
need to specify handler per every resource. Handler is defined in ``meta.yaml``
as below ::
handler: puppet
Solar currently supports following handlers:
* puppet - first version of puppet handler (legacy, will be deprecated soon)
* puppetv2 - second, improved version of puppet, supporting hiera integration
* ansible_playbook - first version of ansible handler (legacy, will be
deprecated soon)
* ansible_template - second generation of ansible implementation, includes
transport support
Handlers are pluggable, so you can write your own easily to extend
functionality of Solar. Interesting examples might be Chef, SaltStack,
CFEngine etc. Using handlers allows Solar to be quickly implemented in various
environments and integrate with already used configuration management tools.
Input
-----
Treat them as values that your resouce have. All needed inputs should be provided in ``meta.yaml`` for example ::
Inputs are essentially values that given resource can accept. Exact usage
depends on handler and actions implementation. If your handler is puppet,
inputs are basically parameters that can be accepted by puppet manifest
underneath.
All needed inputs should be defined in ``meta.yaml`` for example: ::
input:
keystone_password:
@ -48,7 +76,9 @@ Treat them as values that your resouce have. All needed inputs should be provide
Input schema
~~~~~~~~~~~~
It allows to validate if all values are correct. ``!`` at the end of a type means that this is required (``null`` value is not valid).
Input definition contains basic schema validation that allows to validate if
all values are correct. ``!`` at the end of a type means that it is required
(``null`` value is not valid).
* string type ``str``, ``str!``
* integer type ``int``, ``int!``
@ -63,14 +93,18 @@ It allows to validate if all values are correct. ``!`` at the end of a type mean
Action
------
Solar wraps deployment code into actions with specific names. Actions are executed by :ref:`res-handler-term`
Solar wraps deployment code into actions with specific names. Actions are
executed by :ref:`res-handler-term`
Several actions of resource are mandatory:
- run
- remove
- update
You can just put files into ``actions`` subdir in your resource or solar will detect them automaticaly, you can also provide actions in ``meta.yaml`` ::
You can just put files into ``actions`` subdir in your resource and solar will
detect them automatically based on their names, or you can also customize
action file names in ``meta.yaml`` ::
actions:
run: run.pp
@ -79,6 +113,7 @@ You can just put files into ``actions`` subdir in your resource or solar will de
Tag
---
You can attach as many tags to resource as you want, later you can use those tags for grouping etc ::
Tags are used for flexible grouping of resources. You can attach as many tags
to resource as you want, later you can use those tags for grouping etc ::
tags: [resource=hosts_file, tag_name=tag_value, just_some_label]

50
docs/source/transports.rst Normal file
View File

@ -0,0 +1,50 @@
.. _transports_details:
Transports
==========
Transports are used by Solar to communicate with managed nodes.
Transports are also resources, so they have all resources features and
flexibility.
Transports should be added to a node, but if you need you can add different
transports for different resources.
How it works
------------
Each resource in solar has a random :ref:`transports-id-term` generated,
when resources are connected to each other. Solar will ensure that correct
`transport_id` is used. Then using this `transport_id` a correct real value is
fetched. Changing transports contents will not cause `resource.update` action
for related resources.
Sync transport
--------------
This transport uploads required information to target node.
Currently there are following sync transports available:
* ssh
* rsync
* solar_agent
* torrent
Run transport
-------------
This transport is responsible for running commands on remote host.
Currently there are following run transports available:
* ssh
* solar_agent
BAT transport
-------------
A transport that will automatically select best available transport (BAT) that
is available for a given resource. Currently it's default transport in the
system, so when you add more transports, everything should configure
automatically.

6
tox.ini
View File

@ -21,6 +21,12 @@ deps = -r{toxinidir}/requirements.txt
sphinx
commands = {posargs:}
[testenv:cover]
commands =
coverage erase
python setup.py testr --coverage \
--testr-args='--concurrency=1 {posargs}'
[testenv:devenv]
envdir = devenv
usedevelop = True

14
utils/docker_build.sh Executable file
View File

@ -0,0 +1,14 @@
#!/bin/bash
set -e
# should be executed from directory with required Dockerfile
name_w_tags=$1
if [[ -z "$name_w_tags" ]]; then
name_w_tags='solarproject/solar-celery:latest'
fi
echo "Building image with name $name_w_tags"
docker build -t "$name_w_tags" .
docker push "$name_w_tags"