airship/spyglass
Code Issues Proposed changes

Spyglass Docs Update

Browse Source 
Changed paths for all examples, removed references to Tugboat
and Formation

Change-Id: Ibeb80a1c239169bd4f4ecb124e86671891bef99a
This commit is contained in:
Ryan Schroder 2019-11-21 10:47:55 -06:00
parent 9d983e4874
commit 161528365d
6 changed files with 105 additions and 79 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
.zuul.yaml
View File

@ -84,6 +84,7 @@
timeout: 1800 timeout: 1800
run: tools/gate/playbooks/docker-image-build.yaml run: tools/gate/playbooks/docker-image-build.yaml
nodeset: spyglass-single-node nodeset: spyglass-single-node
voting: false
vars: vars:
publish: false publish: false
distro: opensuse_15 distro: opensuse_15
@ -132,6 +133,7 @@
nodeset: spyglass-single-node nodeset: spyglass-single-node
secrets: secrets:
- airship_spyglass_quay_creds - airship_spyglass_quay_creds
voting: false
vars: vars:
publish: true publish: true
distro: opensuse_15 distro: opensuse_15

33
README.rst
View File

@ -6,27 +6,26 @@ Spyglass is the data extractor tool which can interface with
different input data sources to generate site manifest YAML files. different input data sources to generate site manifest YAML files.
The data sources will provide all the configuration data needed The data sources will provide all the configuration data needed
for a site deployment. These site manifest YAML files generated for a site deployment. These site manifest YAML files generated
by spyglass will be saved in a Git repository, from where Pegleg by Spyglass will be saved in a Git repository, from where Pegleg
can access and aggregate them. This aggregated file can then be can access and aggregate them. This aggregated file can then be
fed to shipyard for site deployment / updates. fed to Shipyard for site deployment/updates.
Spyglass follows plugin model to support multiple input data sources. Spyglass follows a plugin model to support multiple input data sources.
Current supported plugins are formation-plugin and Tugboat. Formation The currently supported plugin is the Spyglass Excel plugin
is a rest API based service which will be the source of information (`spyglass-plugin-xls`_).
related to hardware, networking, site data. Formation plugin will
interact with Formation API to gather necessary configuration. The Spyglass Excel plugin accepts an engineering spec in the form of a
Similarly Tugboat accepts engineering spec which is in the form of spreadsheet and an index file to read the spreadsheet as inputs and
spreadsheet and an index file to read spreadsheet as inputs and generates site level manifests. As an optional step, it can generate an
generates the site level manifests. intermediary YAML which contains all the information that will be rendered to
As an optional step it can generate an intermediary yaml which contain generate Airship site manifests. This optional step will help the deployment
all the information that will be rendered to generate Airship site engineer modify any data if required.
manifests. This optional step will help the deployment engineer to
modify any data if required.
Getting Started Getting Started
--------------- ---------------
For more detailed installation and setup information, please refer to the For more detailed information, please refer to the `Index`_ to easily navigate
`Getting Started`_ guide. the Getting Started, Developer Quickstart, and Command Line Interface guides.
.. _`Getting Started`: ./doc/source/getting_started.rst .. _spyglass-plugin-xls: https://opendev.org/airship/spyglass-plugin-xls
.. _`Index`: ./doc/source/index.rst

89
doc/source/cli.rst
View File

@ -28,19 +28,19 @@ The Spyglass CLI is used in conjunction with the script ``tools/spyglass.sh``.
CLI Options CLI Options
=========== ===========
**-v / --verbose** (Optional). False by default. **-v / \\-\\-verbose** (Optional). False by default.
Enable debug logging. Enable debug logging.
Excel Plugin Excel Plugin
************ ************
Commands available under the excel plugin package. Commands available under the Excel plugin package.
Generate Intermediary Generate Intermediary
--------------------- ---------------------
Generates an intermediary file from passed excel data. Generates an intermediary file from passed Excel data.
.. code-block:: bash .. code-block:: bash
@ -52,42 +52,42 @@ Generates an intermediary file from passed excel data.
Options Options
^^^^^^^ ^^^^^^^
**-d / --intermediary-dir** (Optional). **-d / \\-\\-intermediary-dir** (Optional).
Path where the intermediary file will be created. Must be a writeable Path where the intermediary file will be created. Must be a writeable
directory. directory.
**-x / --excel-file** (Required for "tugboat" plugin). **-x / \\-\\-excel-file** (Required for Excel plugin).
Path to the engineering excel file. Multiple files can be included, provided Path to the engineering Excel file. Multiple files can be included, provided
they follow the same specification. Must be readable file(s) in a Microsoft they follow the same specification. Must be readable file(s) in a Microsoft
Excel supported format (.xls, .xslx, etc...). Excel supported format (.xls, .xslx, etc...).
**-e / --excel-spec** (Required for "tugboat" plugin). **-e / \\-\\-excel-spec** (Required for Excel plugin).
Path to the specification YAML that defines the content of the provided Path to the specification YAML that defines the content of the provided
engineering excel files. Must be a readable file in YAML format. engineering Excel files. Must be a readable file in YAML format.
**-c / --site-configuration** (Optional). **-c / \\-\\-site-configuration** (Optional).
Path to site specific configuration YAML. Must be a readable file. Path to site specific configuration YAML. Must be a readable file.
**--intermediary-schema** (Optional). **\\-\\-intermediary-schema** (Optional).
Path to the intermediary schema to be used for validation. Path to the intermediary schema to be used for validation.
**--no-validation** (Optional). **\\-\\-no-validation** (Optional).
Skips validation on generated intermediary data. Skips validation on generated intermediary data.
**-s / --site-name** (Optional). **-s / \\-\\-site-name** (Optional).
Name of the site for which the intermediary is generated. Name of the site for which the intermediary is generated.
Generate Manifests Generate Manifests
------------------ ------------------
Generates manifests from intermediary file created from passed excel data. Generates manifests from intermediary file created from passed Excel data.
Intermediary data is always generated, but will not be saved unless specified. Intermediary data is always generated, but will not be saved unless specified.
.. code-block:: bash .. code-block:: bash
@ -99,48 +99,48 @@ Intermediary data is always generated, but will not be saved unless specified.
Options Options
^^^^^^^ ^^^^^^^
**-i / --generate-intermediary** (Optional). False by default. **-i / \\-\\-generate-intermediary** (Optional). False by default.
Saves the intermediary file used to make the manifests created by the command. Saves the intermediary file used to make the manifests created by the command.
**-d / --intermediary-dir** (Optional). **-d / \\-\\-intermediary-dir** (Optional).
Path where the intermediary file will be created. Must be a writeable Path where the intermediary file will be created. Must be a writeable
directory. directory.
**-x / --excel-file** (Required for "tugboat" plugin). **-x / \\-\\-excel-file** (Required for Excel plugin).
Path to the engineering excel file. Multiple files can be included, provided Path to the engineering Excel file. Multiple files can be included, provided
they follow the same specification. Must be readable file(s) in a Microsoft they follow the same specification. Must be readable file(s) in a Microsoft
Excel supported format (.xls, .xslx, etc...). Excel supported format (.xls, .xslx, etc...).
**-e / --excel-spec** (Required for "tugboat" plugin). **-e / \\-\\-excel-spec** (Required for Excel plugin).
Path to the specification YAML that defines the content of the provided Path to the specification YAML that defines the content of the provided
engineering excel files. Must be a readable file in YAML format. engineering Excel files. Must be a readable file in YAML format.
**-c / --site-configuration** (Optional). **-c / \\-\\-site-configuration** (Optional).
Path to site specific configuration YAML. Must be a readable file. Path to site specific configuration YAML. Must be a readable file.
**--intermediary-schema** (Optional). **\\-\\-intermediary-schema** (Optional).
Path to the intermediary schema to be used for validation. Path to the intermediary schema to be used for validation.
**--no-validation** (Optional). **\\-\\-no-validation** (Optional).
Skips validation on generated intermediary data. Skips validation on generated intermediary data.
**-s / --site-name** (Optional). **-s / \\-\\-site-name** (Optional).
Name of the site for which the intermediary is generated. Name of the site for which the intermediary is generated.
**-t / --template-dir** (Required). **-t / \\-\\-template-dir** (Required).
Path to the Jinja2 template files that will be used to generate manifest files. Path to the Jinja2 template files that will be used to generate manifest files.
Must be a readable directory with Jinja2 files using the .j2 extension. Must be a readable directory with Jinja2 files using the .j2 extension.
**-m / --manifest-dir** (Optional). **-m / \\-\\-manifest-dir** (Optional).
Path where generated manifest files should be written. Must be a writeable Path where generated manifest files should be written. Must be a writeable
directory. directory.
@ -169,17 +169,17 @@ manifests.
Options Options
^^^^^^^ ^^^^^^^
**-t / --template-dir** (Required). **-t / \\-\\-template-dir** (Required).
Path to the Jinja2 template files that will be used to generate manifest files. Path to the Jinja2 template files that will be used to generate manifest files.
Must be a readable directory with Jinja2 files using the .j2 extension. Must be a readable directory with Jinja2 files using the .j2 extension.
**-m / --manifest-dir** (Optional). **-m / \\-\\-manifest-dir** (Optional).
Path where generated manifest files should be written. Must be a writeable Path where generated manifest files should be written. Must be a writeable
directory. directory.
**--force** (Optional). **\\-\\-force** (Optional).
Forces manifests to be written, regardless of undefined data. Forces manifests to be written, regardless of undefined data.
@ -195,11 +195,11 @@ Validates pegleg documents against their schema.
Options Options
^^^^^^^ ^^^^^^^
**-d / --document-path** **-d / \\-\\-document-path**
Path to the document(s) to validate. Path to the document(s) to validate.
**-p / --schema-path** **-p / \\-\\-schema-path**
Path to a schema or directory of schema files used to validate documents in Path to a schema or directory of schema files used to validate documents in
document path. document path.
@ -220,27 +220,33 @@ Generating intermediary and manifests
.. code-block:: bash .. code-block:: bash
spyglass excel documents -i -x SiteDesignSpec_v0.1.xlsx \ spyglass excel documents -i \
-e excel_spec_upstream.yaml -c site_config.yaml \ -x ../spyglass-plugin-xls/spyglass_plugin_xls/examples/SiteDesignSpec_v0.1.xlsx \
-s airship-seaworthy -t <j2 template dir> -e ../spyglass-plugin-xls/spyglass_plugin_xls/examples/excel_spec.yaml \
-c spyglass/examples/site_config.yaml \
-s airship-seaworthy -t spyglass/examples/templates/
Generating intermediary without manifests Generating intermediary without manifests
----------------------------------------- -----------------------------------------
.. code-block:: bash .. code-block:: bash
spyglass excel intermediary -x SiteDesignSpec_v0.1.xlsx \ spyglass excel intermediary \
-e excel_spec_upstream.yaml -c site_config.yaml \ -x ../spyglass-plugin-xls/spyglass_plugin_xls/examples/SiteDesignSpec_v0.1.xlsx \
-s airship-seaworthy -e ../spyglass-plugin-xls/spyglass_plugin_xls/examples/excel_spec.yaml \
-c spyglass/examples/site_config.yaml \
-s airship-seaworthy
Generating manifests without intermediary Generating manifests without intermediary
----------------------------------------- -----------------------------------------
.. code-block:: bash .. code-block:: bash
spyglass excel documents -x SiteDesignSpec_v0.1.xlsx \ spyglass excel documents \
-e excel_spec_upstream.yaml -c site_config.yaml \ -x ../spyglass-plugin-xls/spyglass_plugin_xls/examples/SiteDesignSpec_v0.1.xlsx \
-s airship-seaworthy -t <j2 template dir> -e ../spyglass-plugin-xls/spyglass_plugin_xls/examples/excel_spec.yaml \
-c spyglass/examples/site_config.yaml \
-s airship-seaworthy -t spyglass/examples/templates/
Generating manifests using intermediary Generating manifests using intermediary
*************************************** ***************************************
@ -249,8 +255,9 @@ Generating manifests using intermediary
spyglass mi <intermediary.yaml> -t <j2 template dir> spyglass mi <intermediary.yaml> -t <j2 template dir>
Where sample 'excel_spec_upstream.yaml', 'SiteDesignSpec_v0.1.xlsx' Where sample `excel_spec.yaml` and `SiteDesignSpec_v0.1.xlsx` can be found in
'site_config.yaml' and J2 templates can be found under 'spyglass/examples' spyglass-plugin-xls in the `spyglass_plugin_xls/examples` folder. The Jinja2
templates and `site_config.yaml` can be found in the `spyglass/examples`
folder. folder.
Validate Documents Validate Documents

34
doc/source/developer_quickstart.rst
View File

@ -18,52 +18,56 @@
Developer Quickstart Guide Developer Quickstart Guide
========================== ==========================
To run your first spyglass job, follow these steps from inside the 1. Clone the Spyglass directory. (Perform the following steps from inside the
airship-spyglass directory. spyglass directory)
1. Install external dependencies if not already installed. .. code-block:: console
git clone https://opendev.org/airship/spyglass.git
2. Install external dependencies if not already installed.
.. code-block:: console .. code-block:: console
sudo apt install -y python3-pip sudo apt install -y python3-pip
sudo apt install -y tox sudo apt install -y tox
2. Install Pipenv. 3. Install Pipenv.
.. code-block:: console .. code-block:: console
pip3 install pipenv pip3 install pipenv
2. Set up an environment with Pipenv 4. Set up an environment with Pipenv
.. code-block:: console .. code-block:: console
pipenv install pipenv install
3. Enter the Pipenv environment. 5. Enter the Pipenv environment.
.. code-block:: console .. code-block:: console
pipenv shell pipenv shell
4. Install spyglass in the tox environment. 6. Install spyglass in the tox environment.
.. code-block:: console .. code-block:: console
pip3 install . pip3 install .
5. Run spyglass on the example files to generate an intermediate document. 7. Run spyglass on the example files to generate an intermediate document.
.. code-block:: console .. code-block:: console
mkdir intermediate mkdir intermediate
spyglass excel documents -s airship-seaworthy -d intermediate -i \ spyglass excel documents -s airship-seaworthy -d intermediate -i \
--excel-spec spyglass/examples/excel_spec.yaml \ --excel-spec ../spyglass-plugin-xls/spyglass_plugin_xls/examples/excel_spec.yaml \
--excel-file spyglass/examples/SiteDesignSpec_v0.1.xlsx \ --excel-file ../spyglass-plugin-xls/spyglass_plugin_xls/examples/SiteDesignSpec_v0.1.xlsx \
--site-configuration spyglass/examples/site_config.yaml \ --site-configuration spyglass/examples/site_config.yaml \
--template-dir spyglass/examples/templates/ --template-dir spyglass/examples/templates/
6. Run spyglass on the intermediate document to generate manifests. 8. Run spyglass on the intermediate document to generate manifests.
.. code-block:: console .. code-block:: console

14
doc/source/getting_started.rst
View File

@ -25,10 +25,10 @@ Spyglass is a data extraction tool which can interface with
different input data sources to generate site manifest YAML files. different input data sources to generate site manifest YAML files.
The data sources will provide all the configuration data needed The data sources will provide all the configuration data needed
for a site deployment. These site manifest YAML files generated for a site deployment. These site manifest YAML files generated
by spyglass will be saved in a Git repository, from where Pegleg by Spyglass will be saved in a Git repository, from where Pegleg
can access and aggregate them. This aggregated file can then be can access and aggregate them. This aggregated file can then be
fed to Shipyard for site deployment / updates. fed to Shipyard for site deployment / updates.
Reference: https://airshipit.readthedocs.io/projects/specs/en/latest/specs/approved/data_config_generator.html Reference: `airship-specs`_
Architecture Architecture
------------ ------------
@ -39,7 +39,7 @@ Architecture
| | | +-------+ | | | | +-------+ |
| | +------>| |Generic| | | | +------>| |Generic| |
+-----------+ | | | |Object | | +-----------+ | | | |Object | |
|Tugboat(Xl)| I | | | +-------+ | |Excel | I | | | +-------+ |
|Plugin | N | | | | | |Plugin | N | | | | |
+-----------+ T | | | | | +-----------+ T | | | | |
| E | | | +------+ | | E | | | +------+ |
@ -65,7 +65,7 @@ Architecture
Supported Features Supported Features
------------------ ------------------
1. Spyglass XLS Plugin: https://opendev.org/airship/spyglass-plugin-xls 1. Spyglass Excel Plugin: https://opendev.org/airship/spyglass-plugin-xls
Future Work Future Work
----------- -----------
@ -76,7 +76,7 @@ managed by considering a mapping of j2 templates with schemas and site type.
List of Generated Site Manifests: List of Generated Site Manifests:
--------------------------------- ---------------------------------
The spyglass uses the plugin data source to generate the following site The Spyglass uses the plugin data source to generate the following site
manifests: manifests:
- site-definition.yaml - site-definition.yaml
@ -126,8 +126,10 @@ Before using Spyglass you must:
git clone https://opendev.org/airship/spyglass.git git clone https://opendev.org/airship/spyglass.git
2. Install the required packages in spyglass: 2. Install the required packages in Spyglass:
.. code-block:: console .. code-block:: console
pip3 install pipenv && pipenv install pip3 install pipenv && pipenv install
.. _airship-specs: https://airshipit.readthedocs.io/projects/specs/en/latest/specs/1.x/approved/data_config_generator.html

12
doc/source/index.rst
View File

@ -28,9 +28,21 @@ by spyglass will be saved in a Git repository, from where Pegleg
can access and aggregate them. This aggregated file can then be can access and aggregate them. This aggregated file can then be
fed to Shipyard for site deployment / updates. fed to Shipyard for site deployment / updates.
Spyglass follows a plugin model to support multiple input data sources.
The currently supported plugin is the Spyglass Excel plugin
(`spyglass-plugin-xls`_).
The Spyglass Excel plugin accepts an engineering spec in the form of a
spreadsheet and an index file to read the spreadsheet as inputs and
generates site level manifests. As an optional step, it can generate an
intermediary YAML which contains all the information that will be rendered to
generate Airship site manifests. This optional step will help the deployment
engineer modify any data if required.
.. toctree:: .. toctree::
:maxdepth: 2 :maxdepth: 2
getting_started getting_started
developer_quickstart developer_quickstart
cli cli
.. _spyglass-plugin-xls: https://opendev.org/airship/spyglass-plugin-xls