airship/shipyard
Code Issues Proposed changes
shipyard/docs/source/CLI.rst
Anthony Lin 80fc8095ae Shipyard Documentation for Logs Step 
Documentation for Shipyard API and CLI that is used for retrieval
of logs for a particular Airflow workflow step.

Change-Id: I89c479b164bcbf72e971bb0fc756720b096989e6
2018-04-13 09:54:31 -04:00

767 lines
22 KiB
ReStructuredText
Raw Blame History

..
Copyright 2017 AT&T Intellectual Property.
All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
.. _shipyard_cli:
Shipyard CLI
============
Environment Variables
---------------------
All commands will utilize the following environment variables to
determine necessary information for execution, unless otherwise noted.
OpenStack Keystone Authorization environment variables
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The Shipyard CLI/API Client will check for the presence of appropriate
environment setup to do authentication on behalf of the user. The openrc
variables that will be used are as follows:
- OS_PROJECT_DOMAIN_NAME ("default" if not specified)
- OS_USER_DOMAIN_NAME ("default" if not specified)
- OS_PROJECT_NAME
- OS_USERNAME
- OS_PASSWORD
- OS_AUTH_URL The fully qualified identity endpoint. E.g. http://keystone.ucp.fully.qualified.name:80/v3
OpenStack Keystone Authorization environment variables *not* used
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
These OpenStack identity variables are not supported by shipyard.
- OS_IDENTITY_API_VERSION
This value will be ignored as Shipyard only supports version 3 at this time
Shipyard command options
------------------------
The base shipyard command supports options that determine cross-CLI behaviors.
These options are positionally immediately following the shipyard command as
shown here:
::
shipyard <--these options> subcommands...
shipyard
[--context-marker=<uuid>]
[--debug/--no-debug]
[--os-{various}=<value>]
[--output-format=[format | raw | cli]] (default = cli)
<subcommands, as noted in this document>
\--context-marker=<uuid>
Specifies a UUID (8-4-4-4-12 format) that will be used to correlate logs,
transactions, etc... in downstream activities triggered by this interaction.
If not specified, Shipyard will supply a new UUID to serve as this marker.
(optional)
\--debug | --no-debug
Enable/disable debugging of this CLI and API client. Defaults to no debug
\--os-<various>=<value>
See supported OpenStack Keystone Authorization Environment variables above
for the list of supported names, converting to a downcase version of the
environment variable. E.g.: --os-auth-url=http://keystone.ucp:80/v3
If not specified, the environment variables matching these options will be
used instead. The Keystone os-auth-url should reference the exposed
keystone:port for the target Shipyard environment, as this Keystone will be
used to discover the instance of Shipyard. For most invocations other than
help, a valid combination of values must be resolved to authenticate and
authorize the user's invocation.
\--output-format=<format | raw | cli>
Specifies the desired output formatting such that:
- format
Display the raw output from the invoked Shipyard API in a column
restricted mode.
- raw
Display the result from the invoked Shipyard API as-is, without
modification.
- cli (default)
Display results in a plain text interpretation of the response from the
invoked Shipyard API.
Commit Commands
---------------
commit configdocs
~~~~~~~~~~~~~~~~~
Attempts to commit the Shipyard Buffer documents, first invoking validation by
downstream components.
::
shipyard commit configdocs
[--force]
Example:
shipyard commit configdocs
\--force
Force the commit to occur, even if validations fail.
\--dryrun
Retrieve validation status for the contents of the buffer without committing.
Sample
^^^^^^
::
TBD
Control commands
----------------
pause, unpause, stop
~~~~~~~~~~~~~~~~~~~~
Three separate commands with a common format that allow for controlling
the processing of actions created in Shipyard.
pause
pause something in progress e.g. an executing action
unpause
unpause something paused e.g. a paused action
stop
stops an executing or paused item e.g. an action
::
shipyard pause
<type>
<id>
shipyard unpause
<type>
<id>
shipyard stop
<type>
<id>
shipyard
pause|unpause|stop
<qualified name>
Example:
shipyard pause action 01BTG32JW87G0YKA1K29TKNAFX
shipyard unpause action 01BTG32JW87G0YKA1K29TKNAFX
shipyard stop action 01BTG32JW87G0YKA1K29TKNAFX
shipyard pause action/01BTG32JW87G0YKA1K29TKNAFX
<type>
The type of entity to take action upon. Currently supports: action
<id>
The id of the entity to take action upon.
<qualified name>
The qualified name of the item to take the specified action upon
Sample
^^^^^^
::
$ shipyard pause action/01BZZMEXAVYGG7BT0BMA3RHYY7
pause successfully submitted for action 01BZZMEXAVYGG7BT0BMA3RHYY7
A failed command:
::
$ shipyard pause action/01BZZK07NF04XPC5F4SCTHNPKN
Error: Unable to pause action
Reason: dag_run state must be running, but is failed
- Error: dag_run state must be running, but is failed
Create Commands
---------------
create action
~~~~~~~~~~~~~
Invokes the specified workflow through Shipyard. Returns the
id of the action invoked so that it can be queried subsequently.
::
shipyard create action
<action_command>
--param=<parameter> (repeatable)
Example:
shipyard create action redeploy_server --param="server-name=mcp"
<action_command>
The action to invoke.
\--param=<parameter>
A parameter to be provided to the action being invoked. (repeatable)
Sample
^^^^^^
::
$ shipyard create action deploy_site
Name Action Lifecycle
deploy_site action/01BZZK07NF04XPC5F4SCTHNPKN None
create configdocs
~~~~~~~~~~~~~~~~~
Load documents into the Shipyard Buffer. The use of one or more filenames
or one or more directory options must be specified.
::
shipyard create configdocs
<collection>
[--append | --replace]
--filename=<filename> (repeatable)
|
--directory=<directory> (repeatable)
Example:
shipyard create configdocs design --append --filename=site_design.yaml
.. note::
If neither append nor replace are specified, the Shipyard API default value
of rejectoncontents will be used.
.. note::
Either --filename or --directory must be specified, but both may not be
specified for the same invocation of shipyard.
<collection>
The collection to load.
\--append
Add the collection to the Shipyard Buffer. This will fail if the collection
already exists.
\--replace
Clear the shipyard buffer and replace it with the specified contents.
\--filename=<filename>
The file name to use as the contents of the collection. (repeatable) If
any documents specified fail basic validation, all of the documents will
be rejected. Use of filename parameters may not be used in conjunction
with the directory parameter.
\--directory=<directory>
A directory containing documents that will be joined and loaded as a
collection. (Repeatable) Any documents that fail basic validation will reject the
whole set. Use of the directory parameter may not be used with the
filename parameter.
\--recurse
Recursively search through all directories for sub-directories that
contain yaml files.
Sample
^^^^^^
::
$ shipyard create configdocs coll1 --filename=/home/ubuntu/yaml/coll1.yaml
Configuration documents added.
Status: Validations succeeded
Reason: Validation
Attempting to load the same collection into the uncommitted buffer.
::
$ shipyard create configdocs coll1 --filename=/home/ubuntu/yaml/coll1.yaml
Error: Invalid collection specified for buffer
Reason: Buffermode : rejectoncontents
- Error: Buffer is either not empty or the collection already exists in buffer. Setting a different buffermode may provide the desired functionality
Replace the buffer with --replace
::
$ shipyard create configdocs coll1 --replace --filename=/home/ubuntu/yaml/coll1.yaml
Configuration documents added.
Status: Validations succeeded
Reason: Validation
Describe Commands
-----------------
describe
~~~~~~~~
Retrieves the detailed information about the supplied namespaced item
::
shipyard describe
<namespaced_item>
Example:
shipyard describe action/01BTG32JW87G0YKA1K29TKNAFX
Equivalent to:
shipyard describe action 01BTG32JW87G0YKA1K29TKNAFX
shipyard describe step/01BTG32JW87G0YKA1K29TKNAFX/preflight
Equivalent to:
shipyard describe step preflight --action=01BTG32JW87G0YKA1K29TKNAFX
shipyard describe validation/01BTG32JW87G0YKA1K29TKNAFX/01BTG3PKBS15KCKFZ56XXXBGF2
Equivalent to:
shipyard describe validation 01BTG3PKBS15KCKFZ56XXXBGF2 \
--action=01BTG32JW87G0YKA1K29TKNAFX
shipyard describe workflow/deploy_site__2017-01-01T12:34:56.123456
Equivalent to:
shipyard describe workflow deploy_site__2017-01-01T12:34:56.123456
describe action
~~~~~~~~~~~~~~~
Retrieves the detailed information about the supplied action id.
::
shipyard describe action
<action_id>
Example:
shipyard describe action 01BTG32JW87G0YKA1K29TKNAFX
Sample
^^^^^^
::
$ shipyard describe action/01BZZK07NF04XPC5F4SCTHNPKN
Name: deploy_site
Action: action/01BZZK07NF04XPC5F4SCTHNPKN
Lifecycle: Failed
Parameters: {}
Datetime: 2017-11-27 20:34:24.610604+00:00
Dag Status: failed
Context Marker: 71d4112e-8b6d-44e8-9617-d9587231ffba
User: shipyard
Steps Index State
step/01BZZK07NF04XPC5F4SCTHNPKN/action_xcom 1 success
step/01BZZK07NF04XPC5F4SCTHNPKN/dag_concurrency_check 2 success
step/01BZZK07NF04XPC5F4SCTHNPKN/deckhand_get_design_version 3 failed
step/01BZZK07NF04XPC5F4SCTHNPKN/validate_site_design 4 None
step/01BZZK07NF04XPC5F4SCTHNPKN/deckhand_get_design_version 5 failed
step/01BZZK07NF04XPC5F4SCTHNPKN/deckhand_get_design_version 6 failed
step/01BZZK07NF04XPC5F4SCTHNPKN/drydock_build 7 None
Commands User Datetime
invoke shipyard 2017-11-27 20:34:34.443053+00:00
Validations: None
describe step
~~~~~~~~~~~~~
Retrieves the step details associated with an action and step.
::
shipyard describe step
<step_id>
--action=<action id>
Example:
shipyard describe step preflight --action=01BTG32JW87G0YKA1K29TKNAFX
<step id>
The id of the step found in the describe action response.
\--action=<action id>
The action id that provides the context for this step.
Sample
^^^^^^
::
$ shipyard describe step/01BZZK07NF04XPC5F4SCTHNPKN/action_xcom
Name: action_xcom
Task ID: step/01BZZK07NF04XPC5F4SCTHNPKN/action_xcom
Index: 1
State: success
Start Date: 2017-11-27 20:34:45.604109
End Date: 2017-11-27 20:34:45.818946
Duration: 0.214837
Try Number: 1
Operator: PythonOperator
describe validation
~~~~~~~~~~~~~~~~~~~
Retrieves the validation details associated with an action and
validation id
::
shipyard describe validation
<validation_id>
--action=<action_id>
Example:
shipyard describe validation 01BTG3PKBS15KCKFZ56XXXBGF2 \
--action=01BTG32JW87G0YKA1K29TKNAFX
<validation_id>
The id of the validation found in the describe action response.
\--action=<action_id>
The action id that provides the context for this validation.
Sample
^^^^^^
::
TBD
describe workflow
~~~~~~~~~~~~~~~~~
Retrieves the details for a workflow that is running or has run in the
workflow engine.
::
shipyard describe workflow
<workflow_id>
Example:
shipyard describe workflow deploy_site__2017-01-01T12:34:56.123456
<workflow_id>
The id of the workflow found in the get workflows response.
Sample
^^^^^^
::
$ shipyard describe workflow deploy_site__2017-11-27T20:34:33.000000
Workflow: deploy_site__2017-11-27T20:34:33.000000
State: failed
Dag ID: deploy_site
Execution Date: 2017-11-27 20:34:33
Start Date: 2017-11-27 20:34:33.979594
End Date: None
External Trigger: True
Steps State
action_xcom success
dag_concurrency_check success
deckhand_get_design_version failed
validate_site_design None
deckhand_get_design_version failed
deckhand_get_design_version failed
drydock_build None
Subworkflows:
Workflow: deploy_site.deckhand_get_design_version__2017-11-27T20:34:33.000000
State: failed
Dag ID: deploy_site.deckhand_get_design_version
Execution Date: 2017-11-27 20:34:33
Start Date: 2017-11-27 20:35:06.281825
End Date: None
External Trigger: False
Workflow: deploy_site.deckhand_get_design_version.deckhand_get_design_version__2017-11-27T20:34:33.000000
State: failed
Dag ID: deploy_site.deckhand_get_design_version.deckhand_get_design_version
Execution Date: 2017-11-27 20:34:33
Start Date: 2017-11-27 20:35:20.725506
End Date: None
External Trigger: False
Get Commands
------------
get actions
~~~~~~~~~~~
Lists the actions that have been invoked.
::
shipyard get actions
Sample
^^^^^^
::
$ shipyard get actions
Name Action Lifecycle
deploy_site action/01BZZK07NF04XPC5F4SCTHNPKN Failed
update_site action/01BZZKMW60DV2CJZ858QZ93HRS Processing
get configdocs
~~~~~~~~~~~~~~
Retrieve documents loaded into Shipyard, either committed or from the
Shipyard Buffer.
::
shipyard get configdocs
[<collection>]
[--committed | --buffer]
Example:
shipyard get configdocs design
[<collection>]
The collection to retrieve for viewing. If no collection is entered, the
status of the collections in the buffer and those that are committed will be
displayed.
\--committed
Retrieve the documents that have last been committed for this collection
\--buffer
Retrive the documents that have been loaded into Shipyard since the
prior commit. If no documents have been loaded into the buffer for this
collection, this will return an empty response (default)
Samples
^^^^^^^
::
$ shipyard get configdocs
Collection Committed Buffer
coll1 present unmodified
coll2 not present created
::
$ shipyard get configdocs coll1
data:
chart_groups: [kubernetes-proxy, container-networking, dns, kubernetes, kubernetes-rbac]
release_prefix: ucp
id: 1
metadata:
layeringDefinition: {abstract: false, layer: site}
name: cluster-bootstrap-1
schema: metadata/Document/v1.0
storagePolicy: cleartext
schema: armada/Manifest/v1.0
status: {bucket: coll1, revision: 1}
get renderedconfigdocs
~~~~~~~~~~~~~~~~~~~~~~
Retrieve the rendered version of documents loaded into Shipyard.
Rendered documents are the "final" version of the documents after
applying Deckhand layering and substitution.
::
shipyard get renderedconfigdocs
[--committed | --buffer]
Example:
shipyard get renderedconfigdocs
\--committed
Retrieve the documents that have last been committed.
\--buffer
Retrieve the documents that have been loaded into Shipyard since the
prior commit. (default)
Sample
^^^^^^
::
$ shipyard get renderedconfigdocs
data:
chart_groups: [kubernetes-proxy, container-networking, dns, kubernetes, kubernetes-rbac]
release_prefix: ucp
id: 1
metadata:
layeringDefinition: {abstract: false, layer: site}
name: cluster-bootstrap-1
schema: metadata/Document/v1.0
storagePolicy: cleartext
schema: armada/Manifest/v1.0
status: {bucket: coll1, revision: 1}
get workflows
~~~~~~~~~~~~~
Retrieve workflows that are running or have run in the workflow engine.
This includes processes that may not have been started as an action
(e.g. scheduled tasks).
::
shipyard get workflows
[--since=<date>]
Example:
shipyard get workflows
shipyard get workflows --since=2017-01-01T12:34:56.123456
\--since=<date>
The historical cutoff date to limit the results of of this response.
Sample
^^^^^^
::
$ shipyard get workflows
Workflows State
deploy_site__2017-11-27T20:34:33.000000 failed
update_site__2017-11-27T20:45:47.000000 running
Logs Commands
-------------
logs
~~~~
Retrieves the logs of the supplied namespaced item
::
shipyard logs
<namespaced_item>
Example:
shipyard logs step/01BTG32JW87G0YKA1K29TKNAFX/drydock_validate_site_design
Equivalent to:
shipyard logs step drydock_validate_site_design --action=01BTG32JW87G0YKA1K29TKNAFX
shipyard logs step/01BTG32JW87G0YKA1K29TKNAFX/drydock_validate_site_design/2
Equivalent to:
shipyard logs step drydock_validate_site_design --action=01BTG32JW87G0YKA1K29TKNAFX --try=2
logs step
~~~~~~~~~
Retrieves the logs for a particular workflow step. Note that 'try'
is an optional parameter.
::
shipyard logs step
<step_id> --action=<action_name> [--try=<try>]
Example:
shipyard logs step drydock_validate_site_design --action=01BTG32JW87G0YKA1K29TKNAFX
shipyard logs step drydock_validate_site_design --action=01BTG32JW87G0YKA1K29TKNAFX --try=2
Sample
^^^^^^
::
$ shipyard logs step/01C9VVQSCFS7V9QB5GBS3WFVSE/action_xcom
[2018-04-11 07:30:41,945] {{cli.py:374}} INFO - Running on host airflow-worker-0.airflow-worker-discovery.ucp.svc.cluster.local
[2018-04-11 07:30:41,991] {{models.py:1197}} INFO - Dependencies all met for <TaskInstance: deploy_site.action_xcom 2018-04-11 07:30:37 [queued]>
[2018-04-11 07:30:42,001] {{models.py:1197}} INFO - Dependencies all met for <TaskInstance: deploy_site.action_xcom 2018-04-11 07:30:37 [queued]>
[2018-04-11 07:30:42,001] {{models.py:1407}} INFO -
--------------------------------------------------------------------------------
Starting attempt 1 of 1
--------------------------------------------------------------------------------
[2018-04-11 07:30:42,022] {{models.py:1428}} INFO - Executing <Task(PythonOperator): action_xcom> on 2018-04-11 07:30:37
[2018-04-11 07:30:42,023] {{base_task_runner.py:115}} INFO - Running: ['bash', '-c', 'airflow run deploy_site action_xcom 2018-04-11T07:30:37 --job_id 2 --raw -sd DAGS_FOLDER/deploy_site.py']
[2018-04-11 07:30:42,606] {{base_task_runner.py:98}} INFO - Subtask: [2018-04-11 07:30:42,606] {{driver.py:120}} INFO - Generating grammar tables from /usr/lib/python3.5/lib2to3/Grammar.txt
[2018-04-11 07:30:42,635] {{base_task_runner.py:98}} INFO - Subtask: [2018-04-11 07:30:42,634] {{driver.py:120}} INFO - Generating grammar tables from /usr/lib/python3.5/lib2to3/PatternGrammar.txt
[2018-04-11 07:30:43,515] {{base_task_runner.py:98}} INFO - Subtask: [2018-04-11 07:30:43,515] {{configuration.py:206}} WARNING - section/key [celery/celery_ssl_active] not found in config
[2018-04-11 07:30:43,516] {{base_task_runner.py:98}} INFO - Subtask: [2018-04-11 07:30:43,515] {{default_celery.py:41}} WARNING - Celery Executor will run without SSL
[2018-04-11 07:30:43,517] {{base_task_runner.py:98}} INFO - Subtask: [2018-04-11 07:30:43,516] {{__init__.py:45}} INFO - Using executor CeleryExecutor
[2018-04-11 07:30:43,822] {{base_task_runner.py:98}} INFO - Subtask: [2018-04-11 07:30:43,821] {{models.py:189}} INFO - Filling up the DagBag from /usr/local/airflow/dags/deploy_site.py
[2018-04-11 07:30:43,892] {{cli.py:374}} INFO - Running on host airflow-worker-0.airflow-worker-discovery.ucp.svc.cluster.local
[2018-04-11 07:30:43,945] {{base_task_runner.py:98}} INFO - Subtask: [2018-04-11 07:30:43,944] {{python_operator.py:90}} INFO - Done. Returned value was: None
[2018-04-11 07:30:43,992] {{base_task_runner.py:98}} INFO - Subtask: """)
Help Commands
-------------
help
~~~~
Provides topical help for shipyard.
.. note::
--help will provide more specific command help.
::
shipyard help
[<topic>]
Example:
shipyard help configdocs
<topic>
The topic of the help to be displayed. If this parameter is not
specified the list of available topics will be displayed.
Sample
^^^^^^
::
$ shipyard help
THE SHIPYARD COMMAND
The base shipyard command supports options that determine cross-CLI behaviors.
FORMAT
shipyard [--context-marker=<uuid>] [--os_{various}=<value>]
[--debug/--no-debug] [--output-format] <subcommands>
Please Note: --os_auth_url is required for every command except shipyard help
<topic>.
TOPICS
For information of the following topics, run shipyard help <topic>
actions
configdocs
View Git Blame Copy Permalink