diff --git a/docs/index.rst b/docs/index.rst index 3f31f801..eb452a53 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -14,6 +14,7 @@ Contents: source/glossary source/resource source/orchestration + source/transports source/handler_ansible source/examples source/deployment_plan diff --git a/docs/source/deployment_plan.rst b/docs/source/deployment_plan.rst index e4654f39..b974c262 100644 --- a/docs/source/deployment_plan.rst +++ b/docs/source/deployment_plan.rst @@ -3,8 +3,9 @@ Preparing deployment plan ========================= -Solar allows to make transitions between different versions of infrastructure -based on changes found by solar control plane and configured events between this changes. +Solar allows you to make transitions between different versions of +infrastructure based on changes found by solar control plane and events +configured between this changes. Required information ---------------------- @@ -27,11 +28,11 @@ user (human/program). Examples of changes are: Staged changes --------------- -On demand solar runs procedure that will find all changed from last deployment -resources, and using transitions from solar state machine will determine -list of actions. +On demand solar runs procedure that will find all resources changed from last +deployment and will determine list of actions using transitions from solar +state machine . -This procedure performed by :: +This procedure is performed by :: solar changes stage -d @@ -58,7 +59,7 @@ It prints information like :: ++ collation: utf8_general_ci ++ location_id: 694b35afa622da857f95e14a21599d81 -At this point information is stored as a list, and doensnt know anything +At this point information is stored as a list, and user doesn't know anything about dependencies between found changes. Events usage @@ -66,21 +67,22 @@ Events usage For events definition see :ref:`res-event-term`. -Events used during this procedure to insert dependencies between found changes, -and add new actions that are reactions for changes. +Events are used during this procedure to insert dependencies between found +changes, and to add new actions that are reactions for changes. Example of dependency between changes would be *nova service* that depends on successful creation of *database*. -And for removal we might add dependencies that will allow reverse -order, e.g when removing *nova service* and *database*, *database* will be -removed only after successfull *nova service*. -Can be specified as :: +For removal we might add dependencies that will allow reverse order, e.g. when +removing *nova service* and *database*, *database* will be removed only after +successful *nova service* removal. + +This can be specified as :: Dependency database1.run -> nova1.run Dependency nova1.remove -> database1.remove -Reaction allows to construct more advacned worklows that will take into +Reaction allows to construct more advanced workflows that will take into account not only changes, but also arbitrary actions for resources in solar. Good example of usage is provisioning procedure, where reboot must be @@ -100,9 +102,9 @@ of deployment plan for current version of infrastructure :: solar changes process -After this command plan can be viewed by :: +After this deployment command plan can be viewed by :: - # graphviz representataion + # graphviz representation solar orch dg last # text report diff --git a/docs/source/examples.rst b/docs/source/examples.rst index 50288832..d41800e1 100644 --- a/docs/source/examples.rst +++ b/docs/source/examples.rst @@ -6,29 +6,30 @@ Examples Create resource for the puppet handler -------------------------------------- -Let's create an example :ref:`resource-term` for the puppet :ref:`res-handler-term` -version 1 [#]_. The resource should install and configure OpenStack Nova API -service. +Let's create an example :ref:`resource-term` for the puppet +:ref:`res-handler-term` version 1 [#]_. The resource should install and +configure OpenStack Nova API service. .. [#] There is also puppet handler version 2 but it is out of the scope of this example. -Step 1: Find an apropriate puppet module +Step 1: Find an appropriate puppet module ++++++++++++++++++++++++++++++++++++++++ The `Puppet OpenStack `_ module for `Nova `_ provides all of the required functionality. -Step 2: Define granularity scope for a resource +Step 2: Define granularity level for a resource +++++++++++++++++++++++++++++++++++++++++++++++ -One may want to implement resources as atomic entities doing each its only -task, like running one and only puppet manifest [#]_. Or as a single entity doing -all required tasks instead. In order to configure and run the Nova API service -at least two manifests should be executed: the +One may want to implement resources as atomic entities doing their only single +task, like running one and only puppet manifest [#]_. Other option might be +single entity doing all required tasks instead. In order to configure and run +the Nova API service at least two manifests should be executed: `init.pp `_ -and the `api.pp `_ [#]_. +and +`api.pp `_ [#]_. .. [#] Puppet manifests may contain references to externally defined classess or services in the catalog. Keep that in mind then designing the resource. @@ -40,26 +41,26 @@ Assuming the atomic tasks approach, the example resource for Nova API service should only use the `api.pp` manifest. Note that the puppet handler is normally executed in its own isolated puppet catalog with its specific hiera data only. This assumes every puppet manifest called by every action to be executed as a -separate puppet run and shares nothing. +separate puppet run and shares nothing with other tasks. Step 3: Define resource inputs ++++++++++++++++++++++++++++++ -Once the granularity scope of the resource was clearly defined, one should +Once the granularity level of the resource is clearly defined, one should define the resource's :ref:`res-input-term` data. The puppet class `nova::api` -contains a lots of parameters. It looks reasonable to use them as the resource +contains lots of parameters. It looks reasonable to use them as the resource inputs as is. .. note :: There is a `helper script `_ - to convert a puppet class parameters into the format expeted by the + to convert a puppet class parameters into the format expected by the `meta.yaml` inputs file. -Step 4: Implement the basic action run +Step 4: Implement basic action run ++++++++++++++++++++++++++++++++++++++ Each resource should have all of the mandatory actions defined. In this example -we define only the :ref:`ref-action-term` `run`. For the example Nova API +we define only the :ref:`ref-action-term` `run`. With the example of Nova API resource, the action run should: - fetch the resource inputs from the hiera [#]_ :: @@ -86,7 +87,7 @@ resource, the action run should: } .. note :: - Otherwise, the called class would assume the package and exec are + Otherwise, called class would assume the package and exec are already included in the catalog by the `init.pp`. And would fail as there is no `class { 'nova': }` call expected for the Nova API resource action run. @@ -101,7 +102,7 @@ Step 5: Think of the rest of the resource actions One should also design other actions for the resource. Mandatory are only `run`, `update` and `remove`. There might be additional ones like `on-fail`, `on-retry` or whichever are actually required to implement expected behavior. -For the given API resource there is none specific actions expected and there +For the given API resource there are no specific actions expected and there is nothing to do for the action remove. For the action update, it is likely the same steps should be done as for the action run. @@ -116,7 +117,7 @@ Step 7: Think of the deployment composition The deployment composition is which resources should be used and in which order it should be executed to achive the expected result, which is a successfull -:ref:`deploy-plan-term`. For the given example, the composition may be as the +:ref:`deploy-plan-term`. For the given example, the composition may be as following: - Install and configure MySQL DB [#]_ diff --git a/docs/source/glossary.rst b/docs/source/glossary.rst index 1f0581d9..6ca15a82 100644 --- a/docs/source/glossary.rst +++ b/docs/source/glossary.rst @@ -9,28 +9,33 @@ Solar Glossary Resource ======== -You can learn more about it in :ref:`Resource Details ` +Resource is an abstraction of item in system managed by Solar. It's a basic +building block used to assemble your system. Almost every entity in Solar +is a resource. + +You can learn more about it in :ref:`resource details ` .. _res-input-term: Input ----- -Resource configuration that will be used in actions, handlers and orchestration. -All known inputs should be provided in meta.yaml +Resource configuration that will be used in actions, handlers and +orchestration. All known inputs for a resource should be defined in meta.yaml .. _res-connection-term: Connection ---------- -Allows to build hierarchy between inputs of several resources, -parent value will be always used in child while connection is created. -If connection will be removed - original value of child will be preserved. +Allows to build hierarchy between inputs of several resources, parent value +will be always used in child while connection is created. If connection is +removed - original value of child will be preserved. .. _res-action-term: Action ------ -Solar wraps deployment code into actions with specific names. +Solar wraps deployment code into actions with specific names. Actions are +executed from the resource. .. _res-tag-term: @@ -44,20 +49,20 @@ used for different user operations. Handler ======= -Layer that responsible for action execution and tracking result. +Layer responsible for action execution and tracking results. .. _res-transports-term: Transport ========= -Used in handlers to communicate with managed by solar hosts. List of transports -should be added to a node. Transports will be added to a resource by means -of transports id. +Used in handlers to communicate with hosts managed by Solar. -Two different types of transports are used: run and sync. -Run transport - reponsible for running command on remote host. -Sync transport - uploads required information. +.. seealso:: + + :ref:`More details about transports ` + +.. _location-id-term: location_id ----------- @@ -65,12 +70,15 @@ Used in transport layer to find ip address of a node. :: 'location_id': '96bc779540d832284680785ecd948a2d' +.. _transports-id-term: + transports_id ------------- Used to find transports array that will be used for transport selection. :: 'transports_id': '3889e1790e68b80b4f255cf0e13494b1' + BAT transport ------------- According to preferences solar will choose best available transport for @@ -85,12 +93,11 @@ Used in solar to describe all possible transitions between resources changes. Each event allows to specify two points of transitions, condition of this transition and type of event. -Right now we are supporting 2 types of events. +Right now we are supporting 2 types of events: -1. Dependency -Inserts edge between 2 changes into the deployment plan. -2. Reaction -Inserts change specified in reaction and makes edge between parent and child. +1. Dependency - inserts edge between 2 changes into the deployment plan. +2. Reaction - inserts change specified in reaction and makes edge between +parent and child. Example :: @@ -106,7 +113,7 @@ Example :: Virtual resource/template ========================= -Composition layer that allows to: +Composition layer that allows user to: - group resources - specify connections between inputs @@ -130,9 +137,9 @@ History After action that is related to change will be executed - it will be moved to history with same uuid. -Commited resource data +Committed resource data ---------------------- -After each succesfull change commited copy of resource data will be updated +After each successful change committed copy of resource data will be updated with diff of that change. .. _orch-term: diff --git a/docs/source/orchestration.rst b/docs/source/orchestration.rst index 5ea0f761..c9e36503 100644 --- a/docs/source/orchestration.rst +++ b/docs/source/orchestration.rst @@ -5,7 +5,6 @@ Deployment operations Stage changes ------------- - After user created all required resource - it is possible to automatically detect which resource requires changes with :: @@ -14,30 +13,30 @@ detect which resource requires changes with :: History ------- -After changes are staged - they will be used to populate history which can be seen -with command (*n* option used to limit number of items, -1 will return all changes) :: +After changes are staged - they will be used to populate history which can be +previewed with command (*n* option used to limit number of items, -1 will +return all changes) :: solar changes history -n 5 Prepare deployment plan ----------------------- - -User is able to generate deployment scenario based on changes found by system log. :: +User is able to generate deployment scenario based on changes found by system +log. :: solar changes process - -This command will prepare deployment graph, and return uid of deployment graph to -work with. +This command will prepare deployment graph, and return uid of deployment graph +to work with. All commands that are able to manipulate deployment graph located in *orch* namespace. .. tip:: - Solar writes returned deployment graph uid into special file (`.solar_cli_uids`), it - allows you to use `last` instead of full returned uid: - `solar orch report ` becomes `solar orch report last` + Solar writes returned deployment graph uid into special file + (`.solar_cli_uids`), it allows you to use `last` instead of full returned + uid: `solar orch report ` becomes `solar orch report last` Report @@ -84,6 +83,7 @@ All tasks will be returned to PENDING state, and deployment will be restarted :: Retry deployment ---------------- -Orchestrator will reset all ERROR tasks to PENDING state and restart deployment :: +Orchestrator will reset all ERROR tasks to PENDING state and restart +deployment :: solar orch retry diff --git a/docs/source/resource.rst b/docs/source/resource.rst index 2597dd6c..2b4acca4 100644 --- a/docs/source/resource.rst +++ b/docs/source/resource.rst @@ -3,7 +3,15 @@ Resource ======== -Resource is one of the key Solar components. Resoruce definition takes place in ``meta.yaml`` file. +Resource is one of the key Solar components, almost every entity in Solar is a +resource. Examples are: + +* packages +* services + +Resources are defined in ``meta.yaml`` file. This file is responsible for basic +configuration of given resource. Below is an explanation what constitutes +typical resource. Basic resource structure @@ -23,14 +31,34 @@ Handler .. TODO: add link to handlers doc there -Layer that is responsible for action execution. You need to specify handler per resource, definition in ``meta.yaml`` looks like :: +Pluggable layer that is responsible for executing an action on resource. You +need to specify handler per every resource. Handler is defined in ``meta.yaml`` +as below :: handler: puppet +Solar currently supports following handlers: + +* puppet - first version of puppet handler (legacy, will be deprecated soon) +* puppetv2 - second, improved version of puppet, supporting hiera integration +* ansible_playbook - first version of ansible handler (legacy, will be +deprecated soon) +* ansible_template - second generation of ansible implementation, includes +transport support + +Handlers are pluggable, so you can write your own easily to extend +functionality of Solar. Interesting examples might be Chef, SaltStack, +CFEngine etc. Using handlers allows Solar to be quickly implemented in various +environments and integrate with already used configuration management tools. Input ----- -Treat them as values that your resouce have. All needed inputs should be provided in ``meta.yaml`` for example :: +Inputs are essentially values that given resource can accept. Exact usage +depends on handler and actions implementation. If your handler is puppet, +inputs are basically parameters that can be accepted by puppet manifest +underneath. + +All needed inputs should be defined in ``meta.yaml`` for example: :: input: keystone_password: @@ -48,7 +76,9 @@ Treat them as values that your resouce have. All needed inputs should be provide Input schema ~~~~~~~~~~~~ -It allows to validate if all values are correct. ``!`` at the end of a type means that this is required (``null`` value is not valid). +Input definition contains basic schema validation that allows to validate if +all values are correct. ``!`` at the end of a type means that it is required +(``null`` value is not valid). * string type ``str``, ``str!`` * integer type ``int``, ``int!`` @@ -63,14 +93,18 @@ It allows to validate if all values are correct. ``!`` at the end of a type mean Action ------ -Solar wraps deployment code into actions with specific names. Actions are executed by :ref:`res-handler-term` +Solar wraps deployment code into actions with specific names. Actions are +executed by :ref:`res-handler-term` Several actions of resource are mandatory: + - run - remove - update -You can just put files into ``actions`` subdir in your resource or solar will detect them automaticaly, you can also provide actions in ``meta.yaml`` :: +You can just put files into ``actions`` subdir in your resource and solar will +detect them automatically based on their names, or you can also customize +action file names in ``meta.yaml`` :: actions: run: run.pp @@ -79,6 +113,7 @@ You can just put files into ``actions`` subdir in your resource or solar will de Tag --- -You can attach as many tags to resource as you want, later you can use those tags for grouping etc :: +Tags are used for flexible grouping of resources. You can attach as many tags +to resource as you want, later you can use those tags for grouping etc :: tags: [resource=hosts_file, tag_name=tag_value, just_some_label] diff --git a/docs/source/transports.rst b/docs/source/transports.rst new file mode 100644 index 00000000..950a681e --- /dev/null +++ b/docs/source/transports.rst @@ -0,0 +1,50 @@ +.. _transports_details: + + +Transports +========== + +Transports are used by Solar to communicate with managed nodes. +Transports are also resources, so they have all resources features and +flexibility. +Transports should be added to a node, but if you need you can add different +transports for different resources. + +How it works +------------ + +Each resource in solar has a random :ref:`transports-id-term` generated, +when resources are connected to each other. Solar will ensure that correct +`transport_id` is used. Then using this `transport_id` a correct real value is +fetched. Changing transports contents will not cause `resource.update` action +for related resources. + +Sync transport +-------------- + +This transport uploads required information to target node. + +Currently there are following sync transports available: + +* ssh +* rsync +* solar_agent +* torrent + +Run transport +------------- + +This transport is responsible for running commands on remote host. + +Currently there are following run transports available: + +* ssh +* solar_agent + +BAT transport +------------- + +A transport that will automatically select best available transport (BAT) that +is available for a given resource. Currently it's default transport in the +system, so when you add more transports, everything should configure +automatically. diff --git a/tox.ini b/tox.ini index f2478e3b..2a38889f 100644 --- a/tox.ini +++ b/tox.ini @@ -21,6 +21,12 @@ deps = -r{toxinidir}/requirements.txt sphinx commands = {posargs:} +[testenv:cover] +commands = + coverage erase + python setup.py testr --coverage \ + --testr-args='--concurrency=1 {posargs}' + [testenv:devenv] envdir = devenv usedevelop = True diff --git a/utils/docker_build.sh b/utils/docker_build.sh new file mode 100755 index 00000000..423516e6 --- /dev/null +++ b/utils/docker_build.sh @@ -0,0 +1,14 @@ +#!/bin/bash +set -e + +# should be executed from directory with required Dockerfile +name_w_tags=$1 + +if [[ -z "$name_w_tags" ]]; then + name_w_tags='solarproject/solar-celery:latest' +fi + +echo "Building image with name $name_w_tags" +docker build -t "$name_w_tags" . +docker push "$name_w_tags" +