airship/deckhand
Code Issues Proposed changes

Merge "Revamp Deckhand documentation"

Browse Source
This commit is contained in:
Anthony Lin 2017-10-18 16:34:40 -04:00 committed by Gerrit Code Review
commit 8541266e14
14 changed files with 1396 additions and 1338 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

77
README.rst
View File

@ -1,7 +1,25 @@
========
Deckhand
========
A foundational python REST YAML processing engine providing data and secrets
management to other platform services.
Deckhand is a document-based configuration storage service built with
auditability and validation in mind.
Core Responsibilities
=====================
* layering - helps reduce duplication in configuration while maintaining
auditability across many sites
* substitution - provides separation between secret data and other
configuration data, while allowing a simple interface for clients
* revision history - improves auditability and enables services to provide
functional validation of a well-defined collection of documents that are
meant to operate together
* validation - allows services to implement and register different kinds of
validations and report errors
Getting Started
===============
To generate a configuration file automatically::
@ -36,3 +54,58 @@ To run locally in a development environment::
$ sudo pip install .
$ sudo python setup.py install
$ uwsgi --http :9000 -w deckhand.cmd --callable deckhand_callable --enable-threads -L
Testing
-------
Automated Testing
^^^^^^^^^^^^^^^^^
To run unit tests using sqlite, execute:
::
$ tox -epy27
$ tox -epy35
against a py27- or py35-backed environment, respectively. To run individual
unit tests, run:
::
$ tox -e py27 -- deckhand.tests.unit.db.test_revisions
for example.
To run unit tests using postgresql, execute:
::
$ tox -epy27-postgresql
$ tox -epy35-postgresql
To run functional tests:
::
$ tox -e functional
You can also run a subset of tests via a regex:
::
$ tox -e functional -- gabbi.suitemaker.test_gabbi_document-crud-success-multi-bucket
Manual Testing
^^^^^^^^^^^^^^
Document creation can be tested locally using (from root deckhand directory):
.. code-block:: console
$ curl -i -X PUT localhost:9000/api/v1.0/bucket/{bucket_name}/documents \
-H "Content-Type: application/x-yaml" \
--data-binary "@deckhand/tests/unit/resources/sample_document.yaml"
# revision_id copy/pasted from previous response.
$ curl -i -X GET localhost:9000/api/v1.0/revisions/1

167
deckhand/control/README.rst
View File

@ -1,167 +0,0 @@
Control
=======
This is the external-facing API service to operate on and query
Deckhand-managed data.
v1.0 Endpoints
--------------
POST `/documents`
~~~~~~~~~~~~~~~~~
Accepts a multi-document YAML body and creates a new revision which adds
those documents. Updates are detected based on exact match to an existing
document of `schema` + `metadata.name`. Documents are "deleted" by including
documents with the tombstone metadata schema, such as:
```yaml
---
schema: any-namespace/AnyKind/v1
metadata:
schema: metadata/Tombstone/v1
name: name-to-delete
...
```
This endpoint is the only way to add, update, and delete documents. This
triggers Deckhand's internal schema validations for all documents.
If no changes are detected, a new revision should not be created. This allows
services to periodically re-register their schemas without creating
unnecessary revisions.
Sample response:
```yaml
---
created_at: '2017-07-31T14:46:46.119853'
data:
path:
to:
merge:
into:
ignored: {data: here}
parent: {foo: bar}
substitution: {target: null}
deleted: false
deleted_at: null
id: f99630d9-a89c-4aad-9aaa-7c44462047c1
metadata:
labels: {genesis: enabled, master: enabled}
layeringDefinition:
abstract: false
actions:
- {method: merge, path: .path.to.merge.into.parent}
- {method: delete, path: .path.to.delete}
layer: region
parentSelector: {required_key_a: required_label_a, required_key_b: required_label_b}
name: unique-name-given-schema
schema: metadata/Document/v1
storagePolicy: cleartext
substitutions:
- dest: {path: .substitution.target}
src: {name: name-of-source-document, path: .source.path, schema: another-service/SourceType/v1}
name: unique-name-given-schema
revision_id: 0206088a-c9e9-48e1-8725-c9bdac15d6b7
schema: some-service/ResourceType/v1
updated_at: '2017-07-31T14:46:46.119858'
```
GET `/revisions`
~~~~~~~~~~~~~~~~
Lists existing revisions and reports basic details including a summary of
validation status for each `deckhand/ValidationPolicy` that is part of that
revision.
Sample response:
```yaml
---
count: 7
next: https://deckhand/api/v1.0/revisions?limit=2&offset=2
prev: null
results:
- id: 0
url: https://deckhand/api/v1.0/revisions/0
createdAt: 2017-07-14T21:23Z
validationPolicies:
site-deploy-validation:
status: failed
- id: 1
url: https://deckhand/api/v1.0/revisions/1
createdAt: 2017-07-16T01:15Z
validationPolicies:
site-deploy-validation:
status: succeeded
...
```
GET `/revisions/{revision_id}/documents`
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Returns a multi-document YAML response containing all the documents matching
the filters specified via query string parameters. Returned documents will be
as originally posted with no substitutions or layering applied.
Supported query string parameters:
* `schema` - string, optional - The top-level `schema` field to select. This
may be partially specified by section, e.g., `schema=promenade` would select all
`kind` and `version` schemas owned by promenade, or `schema=promenade/Node`
which would select all versions of `promenade/Node` documents. One may not
partially specify the namespace or kind, so `schema=promenade/No` would not
select `promenade/Node/v1` documents, and `schema=prom` would not select
`promenade` documents.
* `metadata.name` - string, optional
* `metadata.layeringDefinition.abstract` - string, optional - Valid values are
the "true" and "false".
* `metadata.layeringDefinition.layer` - string, optional - Only return documents from
the specified layer.
* `metadata.label` - string, optional, repeatable - Uses the format
`metadata.label=key=value`. Repeating this parameter indicates all
requested labels must apply (AND not OR).
Sample response:
```yaml
created_at: '2017-07-31T14:36:00.352701'
data: {foo: bar}
deleted: false
deleted_at: null
id: ffba233a-326b-4eed-9b21-079ebd2a53f0
metadata:
labels: {genesis: enabled, master: enabled}
layeringDefinition:
abstract: false
actions:
- {method: merge, path: .path.to.merge.into.parent}
- {method: delete, path: .path.to.delete}
layer: region
parentSelector: {required_key_a: required_label_a, required_key_b: required_label_b}
name: foo-name-given-schema
schema: metadata/Document/v1
storagePolicy: cleartext
substitutions:
- dest: {path: .substitution.target}
src: {name: name-of-source-document, path: .source.path, schema: another-service/SourceType/v1}
name: foo-name-given-schema
revision_id: d3428d6a-d8c4-4a5b-8006-aba974cc36a2
schema: some-service/ResourceType/v1
updated_at: '2017-07-31T14:36:00.352705'
```
Testing
-------
Document creation can be tested locally using (from root deckhand directory):
.. code-block:: console
$ curl -i -X PUT localhost:9000/api/v1.0/bucket/{bucket_name}/documents \
-H "Content-Type: application/x-yaml" \
--data-binary "@deckhand/tests/unit/resources/sample_document.yaml"
# revision_id copy/pasted from previous response.
$ curl -i -X GET localhost:9000/api/v1.0/revisions/1

1137
doc/design.md
View File

File diff suppressed because it is too large Load Diff

499
doc/source/api_ref.rst Normal file
View File

@ -0,0 +1,499 @@
..
Copyright 2017 AT&T Intellectual Property.
All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
.. _api-ref:
Deckhand API Documentation
==========================
API
---
This API will only support YAML as a serialization format. Since the IETF
does not provide an official media type for YAML, this API will use
``application/x-yaml``.
This is a description of the ``v1.0`` API. Documented paths are considered
relative to ``/api/v1.0``.
PUT ``/bucket/{bucket_name}/documents``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Accepts a multi-document YAML body and creates a new revision that updates the
contents of the ``bucket_name`` bucket. Documents from the specified bucket that
exist in previous revisions, but are absent from the request are removed from
that revision (though still accessible via older revisions).
Documents in other buckets are not changed and will be included in queries for
documents of the newly created revision.
Updates are detected based on exact match to an existing document of ``schema`` +
``metadata.name``. It is an error that responds with ``409 Conflict`` to attempt
to PUT a document with the same ``schema`` + ``metadata.name`` as an existing
document from a different bucket in the most-recent revision.
This endpoint is the only way to add, update, and delete documents. This
triggers Deckhand's internal schema validations for all documents.
If no changes are detected, a new revision should not be created. This allows
services to periodically re-register their schemas without creating
unnecessary revisions.
GET ``/revisions/{revision_id}/documents``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Returns a multi-document YAML response containing all the documents matching
the filters specified via query string parameters. Returned documents will be
as originally added with no substitutions or layering applied.
Supported query string parameters:
* ``schema`` - string, optional - The top-level ``schema`` field to select. This
may be partially specified by section, e.g., ``schema=promenade`` would select all
``kind`` and ``version`` schemas owned by promenade, or ``schema=promenade/Node``
which would select all versions of ``promenade/Node`` documents. One may not
partially specify the namespace or kind, so ``schema=promenade/No`` would not
select ``promenade/Node/v1`` documents, and ``schema=prom`` would not select
``promenade`` documents.
* ``metadata.name`` - string, optional
* ``metadata.layeringDefinition.abstract`` - string, optional - Valid values are
the "true" and "false".
* ``metadata.layeringDefinition.layer`` - string, optional - Only return documents from
the specified layer.
* ``metadata.label`` - string, optional, repeatable - Uses the format
``metadata.label=key=value``. Repeating this parameter indicates all
requested labels must apply (AND not OR).
* ``sort`` - string, optional, repeatable - Defines the sort order for returning
results. Default is by creation date. Repeating this parameter indicates use
of multi-column sort with the most significant sorting column applied first.
* ``status.bucket`` - string, optional, repeatable - Used to select documents
only from a particular bucket. Repeating this parameter indicates documents
from any of the specified buckets should be returned.
GET ``/revisions/{revision_id}/rendered-documents``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Returns a multi-document YAML of fully layered and substituted documents. No
abstract documents will be returned. This is the primary endpoint that
consumers will interact with for their configuration.
Valid query parameters are the same as for
``/revisions/{revision_id}/documents``, minus the paremters in
``metadata.layeringDetinition``, which are not supported.
GET ``/revisions``
^^^^^^^^^^^^^^^^^^
Lists existing revisions and reports basic details including a summary of
validation status for each ``deckhand/ValidationPolicy`` that is part of that
revision.
Supported query string parameters:
* ``tag`` - string, optional, repeatable - Used to select revisions that have
been tagged with particular tags.
Sample response:
.. code-block:: yaml
---
count: 7
next: https://deckhand/api/v1.0/revisions?limit=2&offset=2
prev: null
results:
- id: 1
url: https://deckhand/api/v1.0/revisions/1
createdAt: 2017-07-14T21:23Z
buckets: [mop]
tags: [a, b, c]
validationPolicies:
site-deploy-validation:
status: failure
- id: 2
url: https://deckhand/api/v1.0/revisions/2
createdAt: 2017-07-16T01:15Z
buckets: [flop, mop]
tags: [b]
validationPolicies:
site-deploy-validation:
status: success
DELETE ``/revisions``
^^^^^^^^^^^^^^^^^^^^^
Permanently delete all documents.
.. warning::
This removes all revisions and resets the data store.
GET ``/revisions/{{revision_id}}``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Get a detailed description of a particular revision. The status of each
``ValidationPolicy`` belonging to the revision is also included. Valid values
for the status of each validation policy are:
* ``success`` - All validations associated with the policy are ``success``.
* ``failure`` - Any validation associated with the policy has status ``failure``,
``expired`` or ``missing``.
Sample response:
.. code-block:: yaml
---
id: 1
url: https://deckhand/api/v1.0/revisions/1
createdAt: 2017-07-14T021:23Z
buckets: [mop]
tags:
a:
name: a
url: https://deckhand/api/v1.0/revisions/1/tags/a
validationPolicies:
site-deploy-validation:
url: https://deckhand/api/v1.0/revisions/1/documents?schema=deckhand/ValidationPolicy/v1&name=site-deploy-validation
status: failure
validations:
- name: deckhand-schema-validation
url: https://deckhand/api/v1.0/revisions/1/validations/deckhand-schema-validation/0
status: success
- name: drydock-site-validation
status: missing
- name: promenade-site-validation
url: https://deckhand/api/v1.0/revisions/1/validations/promenade-site-validation/0
status: expired
- name: armada-deployability-validation
url: https://deckhand/api/v1.0/revisions/1/validations/armada-deployability-validation/0
status: failure
Validation status is always for the most recent entry for a given validation.
A status of ``missing`` indicates that no entries have been created. A status
of ``expired`` indicates that the validation had succeeded, but the
``expiresAfter`` limit specified in the ``ValidationPolicy`` has been exceeded.
GET ``/revisions/{{revision_id}}/diff/{{comparison_revision_id}}``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
This endpoint provides a basic comparison of revisions in terms of how the
buckets involved have changed. Only buckets with existing documents in either
of the two revisions in question will be reported; buckets with documents that
are only present in revisions between the two being compared are omitted from
this report. That is, buckets with documents that were accidentally created
(and then deleted to rectify the mistake) that are not directly present in
the two revisions being compared are omitted.
The response will contain a status of ``created``, ``deleted``, ``modified``, or
``unmodified`` for each bucket.
The ordering of the two revision ids is not important.
For the purposes of diffing, the ``revision_id`` "0" is treated as a revision
with no documents, so queries comparing revision "0" to any other revision will
report "created" for each bucket in the compared revision.
Diffing a revision against itself will respond with the each of the buckets in
the revision as ``unmodified``.
Diffing revision "0" against itself results in an empty dictionary as the response.
Examples
""""""""
A response for a typical case, ``GET /api/v1.0/revisions/6/diff/3`` (or
equivalently ``GET /api/v1.0/revisions/3/diff/6``).
.. code-block:: yaml
---
bucket_a: created
bucket_b: deleted
bucket_c: modified
bucket_d: unmodified
A response for diffing against an empty revision, ``GET /api/v1.0/revisions/0/diff/6``:
.. code-block:: yaml
---
bucket_a: created
bucket_c: created
bucket_d: created
A response for diffing a revision against itself, ``GET /api/v1.0/revisions/6/diff/6``:
.. code-block:: yaml
---
bucket_a: unmodified
bucket_c: unmodified
bucket_d: unmodified
Diffing two revisions that contain the same documents, ``GET /api/v1.0/revisions/8/diff/11``:
.. code-block:: yaml
---
bucket_e: unmodified
bucket_f: unmodified
bucket_d: unmodified
Diffing revision zero with itself, ``GET /api/v1.0/revisions/0/diff/0``:
.. code-block:: yaml
---
{}
POST ``/revisions/{{revision_id}}/validations/{{name}}``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Add the results of a validation for a particular revision.
An example ``POST`` request body indicating validation success:
.. code-block:: yaml
---
status: success
validator:
name: promenade
version: 1.1.2
An example ``POST`` request indicating validation failure:
::
POST /api/v1.0/revisions/3/validations/promenade-site-validation
Content-Type: application/x-yaml
---
status: failure
errors:
- documents:
- schema: promenade/Node/v1
name: node-document-name
- schema: promenade/Masters/v1
name: kubernetes-masters
message: Node has master role, but not included in cluster masters list.
validator:
name: promenade
version: 1.1.2
GET ``/revisions/{{revision_id}}/validations``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Gets the list of validations which have been reported for this revision.
Sample response:
.. code-block:: yaml
---
count: 2
next: null
prev: null
results:
- name: deckhand-schema-validation
url: https://deckhand/api/v1.0/revisions/4/validations/deckhand-schema-validation
status: success
- name: promenade-site-validation
url: https://deckhand/api/v1.0/revisions/4/validations/promenade-site-validation
status: failure
GET ``/revisions/{{revision_id}}/validations/{{name}}``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Gets the list of validation entry summaries that have been posted.
Sample response:
.. code-block:: yaml
---
count: 1
next: null
prev: null
results:
- id: 0
url: https://deckhand/api/v1.0/revisions/4/validations/promenade-site-validation/0/entries/0
status: failure
GET ``/revisions/{{revision_id}}/validations/{{name}}/entries/{{entry_id}}``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Gets the full details of a particular validation entry, including all posted
error details.
Sample response:
.. code-block:: yaml
---
name: promenade-site-validation
url: https://deckhand/api/v1.0/revisions/4/validations/promenade-site-validation/entries/0
status: failure
createdAt: 2017-07-16T02:03Z
expiresAfter: null
expiresAt: null
errors:
- documents:
- schema: promenade/Node/v1
name: node-document-name
- schema: promenade/Masters/v1
name: kubernetes-masters
message: Node has master role, but not included in cluster masters list.
POST ``/revisions/{{revision_id}}/tags/{{tag}}``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Associate the revision with a collection of metadata, if provided, by way of
a tag. The tag itself can be used to label the revision.
Sample request with body:
::
POST ``/revisions/0615b731-7f3e-478d-8ba8-a223eab4757e/tags/foobar``
Content-Type: application/x-yaml
---
thing: bar
Sample response:
::
Content-Type: application/x-yaml
HTTP/1.1 201 Created
Location: https://deckhand/api/v1.0/revisions/0615b731-7f3e-478d-8ba8-a223eab4757e/tags/foobar
---
tag: foobar
data:
thing: bar
Sample request without body:
::
POST ``/revisions/0615b731-7f3e-478d-8ba8-a223eab4757e/tags/foobar``
Content-Type: application/x-yaml
Sample response:
::
Content-Type: application/x-yaml
HTTP/1.1 201 Created
Location: https://deckhand/api/v1.0/revisions/0615b731-7f3e-478d-8ba8-a223eab4757e/tags/foobar
---
tag: foobar
data: {}
GET ``/revisions/{{revision_id}}/tags``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
List the tags associated with a revision.
Sample request with body:
::
GET ``/revisions/0615b731-7f3e-478d-8ba8-a223eab4757e/tags``
Sample response:
::
Content-Type: application/x-yaml
HTTP/1.1 200 OK
---
- tag: foo
data:
thing: bar
- tag: baz
data:
thing: qux
GET ``/revisions/{{revision_id}}/tags/{{tag}}``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Show tag details for tag associated with a revision.
Sample request with body:
::
GET ``/revisions/0615b731-7f3e-478d-8ba8-a223eab4757e/tags/foo``
Sample response:
::
Content-Type: application/x-yaml
HTTP/1.1 200 OK
---
tag: foo
data:
thing: bar
DELETE ``/revisions/{{revision_id}}/tags/{{tag}}``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Delete tag associated with a revision.
Sample request with body:
::
GET ``/revisions/0615b731-7f3e-478d-8ba8-a223eab4757e/tags/foo``
Sample response:
::
Content-Type: application/x-yaml
HTTP/1.1 204 No Content
DELETE ``/revisions/{{revision_id}}/tags``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Delete all tags associated with a revision.
Sample request with body:
::
GET ``/revisions/0615b731-7f3e-478d-8ba8-a223eab4757e/tags``
Sample response:
::
Content-Type: application/x-yaml
HTTP/1.1 204 No Content
POST ``/rollback/{target_revision_id}``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Creates a new revision that contains exactly the same set of documents as the
revision specified by ``target_revision_id``.

32
doc/source/buckets.rst Normal file
View File

@ -0,0 +1,32 @@
..
Copyright 2017 AT&T Intellectual Property.
All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
.. _bucket:
Buckets
=======
Collections of documents, called buckets, are managed together. All documents
belong to a bucket and all documents that are part of a bucket must be fully
specified together.
To create or update a new document in, e.g. bucket ``mop``, one must PUT the
entire set of documents already in ``mop`` along with the new or modified
document. Any documents not included in that PUT will be automatically
deleted in the created revision.
This feature allows the separation of concerns when delivering different
categories of documents, while making the delivered payload more declarative.

177
doc/source/document_types.rst Normal file
View File

@ -0,0 +1,177 @@
..
Copyright 2017 AT&T Intellectual Property.
All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
.. _document-types:
Document Types
==============
Control Documents
-----------------
Control documents (documents which have ``metadata.schema=metadata/Control/v1``),
are special, and are used to control the behavior of Deckhand at runtime. Only
the following types of control documents are allowed.
DataSchema
^^^^^^^^^^
``DataSchema`` documents are used by various services to register new schemas
that Deckhand can use for validation. No ``DataSchema`` documents with names
beginning with ``deckhand/`` or ``metadata/`` are allowed. Tme ``metadata.name``
field of each ``DataSchema`` document specifies the top level ``schema`` that it
is used to validate.
The contents of its ``data`` key are expected to be the json schema definition
for the target document type from the target's top level ``data`` key down.
.. TODO: give valid, tiny schema as example
.. code-block:: yaml
---
schema: deckhand/DataSchema/v1 # This specifies the official JSON schema meta-schema.
metadata:
schema: metadata/Control/v1
name: promenade/Node/v1 # Specifies the documents to be used for validation.
labels:
application: promenade
data: # Valid JSON Schema is expected here.
$schema: http://blah
...
LayeringPolicy
^^^^^^^^^^^^^^
Only one ``LayeringPolicy`` document can exist within the system at any time.
It is an error to attempt to insert a new ``LayeringPolicy`` document if it has
a different ``metadata.name`` than the existing document. If the names match,
it is treated as an update to the existing document.
This document defines the strict order in which documents are merged together
from their component parts. It should result in a validation error if a
document refers to a layer not specified in the ``LayeringPolicy``.
.. code-block:: yaml
---
schema: deckhand/LayeringPolicy/v1
metadata:
schema: metadata/Control/v1
name: layering-policy
data:
layerOrder:
- global
- site-type
- region
- site
- force
...
ValidationPolicy
^^^^^^^^^^^^^^^^
Unlike ``LayeringPolicy``, many ``ValidationPolicy`` documents are allowed. This
allows services to check whether a particular revision (described below) of
documents meets a configurable set of validations without having to know up
front the complete list of validations.
Each validation ``name`` specified here is a reference to data that is postable
by other services. Names beginning with ``deckhand`` are reserved for internal
use. See the Validation section below for more details.
Since validations may indicate interactions with external and changing
circumstances, an optional ``expiresAfter`` key may be specified for each
validation as an ISO8601 duration. If no ``expiresAfter`` is specified, a
successful validation does not expire. Note that expirations are specific to
the combination of ``ValidationPolicy`` and validation, not to each validation
by itself.
.. code-block:: yaml
---
schema: deckhand/ValidationPolicy/v1
metadata:
schema: metadata/Control/v1
name: site-deploy-ready
data:
validations:
- name: deckhand-schema-validation
- name: drydock-site-validation
expiresAfter: P1W
- name: promenade-site-validation
expiresAfter: P1W
- name: armada-deployability-validation
...
Provided Utility Document Kinds
-------------------------------
These are documents that use the ``Document`` metadata schema, but live in the
``deckhand`` namespace.
Certificate
^^^^^^^^^^^
.. code-block:: yaml
---
schema: deckhand/Certificate/v1
metadata:
schema: metadata/Document/v1
name: application-api
storagePolicy: cleartext
data: |-
-----BEGIN CERTIFICATE-----
MIIDYDCCAkigAwIBAgIUKG41PW4VtiphzASAMY4/3hL8OtAwDQYJKoZIhvcNAQEL
...snip...
P3WT9CfFARnsw2nKjnglQcwKkKLYip0WY2wh3FE7nrQZP6xKNaSRlh6p2pCGwwwH
HkvVwA==
-----END CERTIFICATE-----
...
CertificateKey
^^^^^^^^^^^^^^
.. code-block:: yaml
---
schema: deckhand/CertificateKey/v1
metadata:
schema: metadata/Document/v1
name: application-api
storagePolicy: encrypted
data: |-
-----BEGIN RSA PRIVATE KEY-----
MIIEpQIBAAKCAQEAx+m1+ao7uTVEs+I/Sie9YsXL0B9mOXFlzEdHX8P8x4nx78/T
...snip...
Zf3ykIG8l71pIs4TGsPlnyeO6LzCWP5WRSh+BHnyXXjzx/uxMOpQ/6I=
-----END RSA PRIVATE KEY-----
...
Passphrase
^^^^^^^^^^
.. code-block:: yaml
---
schema: deckhand/Passphrase/v1
metadata:
schema: metadata/Document/v1
name: application-admin-password
storagePolicy: encrypted
data: some-password
...

164
doc/source/documents.rst Normal file
View File

@ -0,0 +1,164 @@
..
Copyright 2017 AT&T Intellectual Property.
All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
Documents
=========
All configuration data is stored entirely as structured documents, for which
schemas must be registered. Documents satisfy the following use cases:
* layering - helps reduce duplication in configuration while maintaining
auditability across many sites
* substitution - provides separation between secret data and other
configuration data, while allowing a simple interface for clients
* revision history - improves auditability and enables services to provide
functional validation of a well-defined collection of documents that are
meant to operate together
* validation - allows services to implement and register different kinds of
validations and report errors
Detailed documentation for :ref:`layering`, :ref:`substitution`,
:ref:`revision-history` and :ref:`validation` should be reviewed for a more
thorough understanding of each concept.
Document Format
---------------
The document format is modeled loosely after Kubernetes practices. The top
level of each document is a dictionary with 3 keys: ``schema``, ``metadata``,
and ``data``.
* ``schema`` - Defines the name of the JSON schema to be used for validation.
Must have the form: ``<namespace>/<kind>/<version>``, where the meaning of
each component is:
* ``namespace`` - Identifies the owner of this type of document. The
values `deckhand` and `metadata` are reserved for internal use.
* ``kind`` - Identifies a type of configuration resource in the namespace.
* ``version`` - Describe the version of this resource, e.g. "v1".
* ``metadata`` - Defines details that Deckhand will inspect and understand.
There are multiple schemas for this section as discussed below. All the
various types of metadata include a `name` field which must be unique for
each document `schema`.
* ``data`` - Data to be validated by the schema described by the ``schema``
field. Deckhand only interacts with content here as instructed to do so by
the ``metadata`` section. The form of this section is considered to be
completely owned by the ``namespace`` in the ``schema``.
Below is a fictitious example of a complete document, which illustrates all the
valid fields in the ``metadata`` section:
.. code-block:: yaml
---
schema: some-service/ResourceType/v1
metadata:
schema: metadata/Document/v1
name: unique-name-given-schema
storagePolicy: cleartext
labels:
genesis: enabled
master: enabled
layeringDefinition:
abstract: true
layer: region
parentSelector:
required_key_a: required_label_a
required_key_b: required_label_b
actions:
- method: merge
path: .path.to.merge.into.parent
- method: delete
path: .path.to.delete
substitutions:
- dest:
path: .substitution.target
src:
schema: another-service/SourceType/v1
name: name-of-source-document
path: .source.path
data:
path:
to:
merge:
into:
parent:
foo: bar
ignored:
data: here
substitution:
target: null
Document Metadata
^^^^^^^^^^^^^^^^^
There are 2 supported kinds of document metadata. Documents with `Document`
metadata are the most common, and are used for normal configuration data.
Documents with `Control` metadata are used to customize the behavior of
Deckhand.
schema: metadata/Document/v1
""""""""""""""""""""""""""""
This type of metadata allows the following metadata hierarchy:
* ``name`` - string, required - Unique within a revision for a given ``schema``.
* ``storagePolicy`` - string, required - Either ``cleartext`` or ``encrypted``. If
``encyrpted`` is specified, then the ``data`` section of the document will be
stored in an secure backend (likely via OpenStack Barbican). ``metadata`` and
``schema`` fields are always stored in cleartext.
* ``layeringDefinition`` - dict, required - Specifies layering details. See the
Layering section below for details.
* ``abstract`` - boolean, required - An abstract document is not expected to
pass schema validation after layering and substitution are applied.
Non-abstract (concrete) documents are.
* ``layer`` - string, required - References a layer in the ``LayeringPolicy``
control document.
* ``parentSelector`` - labels, optional - Used to construct document chains for
executing merges.
* ``actions`` - list, optional - A sequence of actions to apply this documents
data during the merge process.
* ``method`` - string, required - How to layer this content.
* ``path`` - string, required - What content in this document to layer onto
parent content.
* ``substitutions`` - list, optional - A sequence of substitutions to apply. See
the Substitutions section for additional details.
* ``dest`` - dict, required - A description of the inserted content destination.
* ``path`` - string, required - The JSON path where the data will be placed
into the ``data`` section of this document.
* ``pattern`` - string, optional - A regex to search for in the string
specified at ``path`` in this document and replace with the source data
* ``src`` - dict, required - A description of the inserted content source.
* ``schema`` - string, required - The ``schema`` of the source document.
* ``name`` - string, required - The ``metadata.name`` of the source document.
* ``path`` - string, required - The JSON path from which to extract data in
the source document relative to its ``data`` section.
schema: metadata/Control/v1
"""""""""""""""""""""""""""
This schema is the same as the ``Document`` schema, except it omits the
``storagePolicy``, ``layeringDefinition``, and ``substitutions`` keys, as these
actions are not supported on ``Control`` documents.
The complete list of valid ``Control`` document kinds is specified below along
with descriptions of each document kind.

32
doc/source/index.rst
View File

@ -1,18 +1,18 @@
..
Copyright 2017 AT&T Intellectual Property.
All Rights Reserved.
Copyright 2017 AT&T Intellectual Property.
All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
====================================
Welcome to Deckhand's documentation!
@ -38,7 +38,14 @@ User's Guide
.. toctree::
:maxdepth: 2
policy-enforcement
documents
document_types
buckets
validation
substitution
layering
revision_history
api_ref
Developer's Guide
=================
@ -47,6 +54,7 @@ Developer's Guide
:maxdepth: 2
HACKING
policy-enforcement
testing
Glossary

151
doc/source/layering.rst Normal file
View File

@ -0,0 +1,151 @@
..
Copyright 2017 AT&T Intellectual Property.
All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
.. _layering:
Document Layering
=================
Layering provides a restricted data inheritance model intended to help reduce
duplication in configuration. Documents with different ``schema``'s are never
layered together (see the :ref:`substitution` section if you need to combine data
from multiple types of documents).
Layering is controlled in two places:
1. The ``LayeringPolicy`` control document (described below), which defines the
valid layers and their order of precedence.
2. In the ``metadata.layeringDefinition`` section of normal
(``metadata.schema=metadata/Document/v1``) documents.
When rendering a particular document, you resolve the chain of parents upward
through the layers, and begin working back down each layer rendering at each
document in the chain.
When rendering each layer, the parent document is used as the starting point,
so the entire contents of the parent are brought forward. Then the list of
`actions` will be applied in order. Supported actions are:
* ``merge`` - "deep" merge child data at the specified path into the existing data
* ``replace`` - overwrite existing data with child data at the specified path
* ``delete`` - remove the existing data at the specified path
After actions are applied for a given layer, substitutions are applied (see
the Substitution section for details).
Selection of document parents is controlled by the ``parentSelector`` field and
works as follows. A given document, ``C``, that specifies a ``parentSelector``
will have exactly one parent, ``P``. Document ``P`` will be the highest
precedence (i.e. part of the lowest layer defined in the ``layerOrder`` list
from the ``LayeringPolicy``) document that has the labels indicated by the
``parentSelector`` (and possibly additional labels) from the set of all
documents of the same ``schema`` as ``C`` that are in layers above the layer ``C``
is in. For example, consider the following sample documents:
.. code-block:: yaml
---
schema: deckhand/LayeringPolicy/v1
metadata:
schema: metadata/Control/v1
name: layering-policy
data:
layerOrder:
- global
- region
- site
---
schema: example/Kind/v1
metadata:
schema: metadata/Document/v1
name: global-1234
labels:
key1: value1
layeringDefinition:
abstract: true
layer: global
data:
a:
x: 1
y: 2
---
schema: example/Kind/v1
metadata:
schema: metadata/Document/v1
name: region-1234
labels:
key1: value1
layeringDefinition:
abstract: true
layer: region
parentSelector:
key1: value1
actions:
- method: replace
path: .a
data:
a:
z: 3
---
schema: example/Kind/v1
metadata:
schema: metadata/Document/v1
name: site-1234
layeringDefinition:
layer: site
parentSelector:
key1: value1
actions:
- method: merge
path: .
data:
b: 4
When rendering, the parent chosen for ``site-1234`` will be ``region-1234``,
since it is the highest precedence document that matches the label selector
defined by ``parentSelector``, and the parent chosen for ``region-1234`` will be
``global-1234`` for the same reason. The rendered result for ``site-1234`` would
be:
.. code-block:: yaml
---
schema: example/Kind/v1
metadata:
name: site-1234
data:
a:
z: 3
b: 4
If ``region-1234`` were later removed, then the parent chosen for `site-1234`
would become ``global-1234``, and the rendered result would become:
.. code-block:: yaml
---
schema: example/Kind/v1
metadata:
name: site-1234
data:
a:
x: 1
y: 2
b: 4
.. TODO: Add figures for this example, with region present, have site point
.. with dotted line at global and indicate in caption (or something) that it's
.. selected for but ignored, because there's a higher-precedence layer to select

20
doc/source/policy-enforcement.rst
View File

@ -1,17 +1,17 @@
..
Copyright 2017 AT&T Intellectual Property. All other rights reserved.
Copyright 2017 AT&T Intellectual Property. All other rights reserved.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Rest API Policy Enforcement

57
doc/source/revision_history.rst Normal file
View File

@ -0,0 +1,57 @@
..
Copyright 2017 AT&T Intellectual Property.
All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
.. _revision-history:
Revision History
================
Revision History
----------------
Documents will be ingested in batches which will be given a revision index.
This provides a common language for describing complex validations on sets of
documents.
Revisions can be thought of as commits in a linear git history, thus looking
at a revision includes all content from previous revisions.
Revision Diffing
----------------
By maintaining a linear history of all the documents in each revision, Deckhand
is able to diff different revisions together to report what has changed
across revisions, allowing external services to determine whether the Deckhand
configuration undergone any changes since the service last queried the Deckhand
API.
The revision difference is calculated by comparing the `overall` difference
across all the documents in the buckets associated with the two revisions that
are diffed. For example, if a bucket shared between two revisions contains two
documents, and between the first revision and the second revision, if only
one of those two documents has been modified, the bucket itself is tagged
as ``modified``. For more information about revision diffing, please reference
the :ref:`api-ref`.
Revision Rollback
-----------------
As all the changes to documents are maintained via revisions, it is possible to
rollback the latest revision in Deckhand to a prior revision. This behavior can
be loosely compared to a ``git rebase`` in which it is possible to squash the
latest revision in order to go back to the previous revision. This behavior is
useful for undoing accidental changes and returning to a stable internal
configuration.

140
doc/source/substitution.rst Normal file
View File

@ -0,0 +1,140 @@
..
Copyright 2017 AT&T Intellectual Property.
All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
.. _substitution:
Secret Substitution
===================
Substitution is primarily designed as a mechanism for inserting secrets into
configuration documents, but works for unencrypted source documents as well.
Substitution is applied at each layer after all merge actions occur. Further,
substitution is only applied to the ``data`` section of a document.
Concrete (non-abstract) documents can be used as a source of substitution
into other documents. This substitution is layer-independent, so given the 3
layer example above, which includes ``global``, ``region`` and ``site`` layers,
a document in the ``region`` layer could insert data from a document in the
``site`` layer.
Here is a sample set of documents demonstrating substitution:
.. code-block:: yaml
---
schema: deckhand/Certificate/v1
metadata:
name: example-cert
storagePolicy: cleartext
layeringDefinition:
layer: site
data: |
CERTIFICATE DATA
---
schema: deckhand/CertificateKey/v1
metadata:
name: example-key
storagePolicy: encrypted
layeringDefinition:
layer: site
data: |
KEY DATA
---
schema: deckhand/Passphrase/v1
metadata:
name: example-password
storagePolicy: encrypted
layeringDefinition:
layer: site
data: my-secret-password
---
schema: armada/Chart/v1
metadata:
name: example-chart-01
storagePolicy: cleartext
layeringDefinition:
layer: region
substitutions:
- dest:
path: .chart.values.tls.certificate
src:
schema: deckhand/Certificate/v1
name: example-cert
path: .
- dest:
path: .chart.values.tls.key
src:
schema: deckhand/CertificateKey/v1
name: example-key
path: .
- dest:
path: .chart.values.some_url
pattern: INSERT_[A-Z]+_HERE
src:
schema: deckhand/Passphrase/v1
name: example-password
path: .
data:
chart:
details:
data: here
values:
some_url: http://admin:INSERT_PASSWORD_HERE@service-name:8080/v1
...
The rendered document will look like:
.. code-block:: yaml
---
schema: armada/Chart/v1
metadata:
name: example-chart-01
storagePolicy: cleartext
layeringDefinition:
layer: region
substitutions:
- dest:
path: .chart.values.tls.certificate
src:
schema: deckhand/Certificate/v1
name: example-cert
path: .
- dest:
path: .chart.values.tls.key
src:
schema: deckhand/CertificateKey/v1
name: example-key
path: .
- dest:
path: .chart.values.some_url
pattern: INSERT_[A-Z]+_HERE
src:
schema: deckhand/Passphrase/v1
name: example-password
path: .
data:
chart:
details:
data: here
values:
some_url: http://admin:my-secret-password@service-name:8080/v1
tls:
certificate: |
CERTIFICATE DATA
key: |
KEY DATA
...

20
doc/source/testing.rst
View File

@ -1,17 +1,17 @@
..
Copyright 2017 AT&T Intellectual Property. All other rights reserved.
Copyright 2017 AT&T Intellectual Property. All other rights reserved.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
=======
Testing

61
doc/source/validation.rst Normal file
View File

@ -0,0 +1,61 @@
..
Copyright 2017 AT&T Intellectual Property.
All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
.. _validation:
Document Validation
===================
Validations
-----------
The validation system provides a unified approach to complex validations that
require coordination of multiple documents and business logic that resides in
consumer services.
Services can report success or failure of named validations for a given
revision. Those validations can then be referenced by many ``ValidationPolicy``
control documents. The intended purpose use is to allow a simple mapping that
enables consuming services to be able to quickly check whether the
configuration in Deckhand is in a valid state for performing a specific
action.
Deckhand-Provided Validations
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
In addition to allowing 3rd party services to report configurable validation
statuses, Deckhand provides a few internal validations which are made
available immediately upon document ingestion.
Here is a list of internal validations:
* ``deckhand-document-schema-validation`` - All concrete documents in the
revision successfully pass their JSON schema validations. Will cause
this to report an error.
* ``deckhand-policy-validation`` - All required policy documents are in-place,
and existing documents conform to those policies. E.g. if a 3rd party
document specifies a ``layer`` that is not present in the layering policy,
that will cause this validation to report an error.
Externally Provided Validations
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
As mentioned, other services can report whether named validations that have
been registered by those services as success or failure. ``DataSchema`` control
documents are used to register a new validation mapping that other services
can reference to verify whether a Deckhand bucket is in a valid configuration.
For more information, refer to the ``DataSchema`` section in
:ref:`document-types`.