This patch set expands on the unit test coverage for lint checks
in test_selectable_linting which only covers a small subset of
the lint checks handled by Pegleg. This logic should be properly
tested as linting is fundamental to Pegleg functionality.
Change-Id: I6a59295982abd22bba8036827cefd4186b68e2fb
This patch set adds hacking rule to pegleg and fixes outstanding non-
docstring related violations.
Change-Id: I5bb5e78c211f24cf95669124bfcf9603bea8bf15
Signed-off-by: Tin Lam <tin@irrational.io>
This PS enables Pegleg to upload documents directly to Shipyard
thus ensuring that unencrypted data never gets stored in disk.
The flow for this new CLI command is as follows:
- Collect documents as per the provided site repository
- Decrypt the collected documets(TODO)
- Upload document to Shipyard:
- one collection per repository will be uploaded to Shipyard
Eg-
pegleg site -r /opt/aic-clcp-site-manifests \
-e global=/opt/aic-clcp-manifests upload <site-name>
Two collections will be created in shipyard since there are two
repositories provided. The name of the collections will be the
name of repositories provided.
- Commit the documents in shipyard buffer.
Change-Id: I6275252b044ebb82d8bb2009c0bea6ebf7033bce
This patch set sanitizes and changes the default proxy URLs.
Change-Id: Ibaeb501b59eeea8427fa812078236cfc4fe39623
Signed-off-by: Tin Lam <tin@irrational.io>
This patch set changes Pegleg in two similar ways:
1) Ignore certain types of files altogether:
- those located in hidden folders
- those prefixed with "." (files like .zuul.yaml)
2) Only read Deckhand-formatted documents for lint/collect/etc.
commands as Pegleg need not consider other types of documents
(it separately reads the site-definition.yaml for internal
processing still).
The tools/ subfolder is also ignored as it can contain
.yaml files which are not Deckhand-formatted documents,
so need not be processed by pegleg.engine.
Change-Id: I8996b5d430cf893122af648ef8e5805b36c1bfd9
1. Added the method to decrypt a secret file and return its contents
as a list of documents (instead of printing out the file content).
2. Added clarifications for a encrypt and decrypt commands.
Change-Id: I77bce21be214c880c8413f5e6a2d0c2d1993fc8e
This patch set drops unused lint code P002 from Pegleg. The
variable for P002 is called:
DECKHAND_RENDERING_INCOMPLETE_FLAG
It is not clear what this even means, so the assumption here is
that it is a mishap so there is nothing to implement behind the
scenes. The recourse is to remove it.
Change-Id: I3bc99e5f48aa1fa9ea652f073fd4e147b4283e10
This patch set refactors uses of tempfile.mkdtemp() used
throughout many tests in Pegleg which leaves lingering temporary
directories around. Recently a fixture was introduced in [0] which
automatically cleans up after itself. This patch set applies the
fixture everywhere possible to minimize the testing footprint.
[0] https://review.openstack.org/#/c/609818/20/tests/unit/fixtures.py
Change-Id: Id4c1195c4f248b974a5396a429d651943a84ee83
This patch set replaces raising NotADirectoryError after trying
to parse a repository for its root path (normalize_repo_path in
pegleg.engine.util.git) with a better exception
(exceptions.GitInvalidRepoException). It is better because a
folder can still not be a repo, so raising the first exception
isn't apropos.
Next, this patch set changes where the exception is raised --
which is in normalize_repo_path itself, which is more appropriate
as the function is used in many places and so there should be
intrinsic error handling so as to avoid having to wrap it every
time.
Change-Id: I918d8c293f1140eb80c83499dba2c23af232b79e
This patch set adds additional logic to properly handle parsing
out the revision from an SSH repo url. The issue was being
masked by unit tests whose automated logic for calculating
the expected revision mirrored the actual implementation.
Thus, the unit tests have also been refactored to take in
hardcoded expected values to ensure that the assertions are
foolproof around validating expected revisions.
Change-Id: I7aacb4792f6b2dfc08d3a7bb4c3f18bbcfc95b8a
Pegleg's config module is used for basically keeping track
of CLI context information within global memory namespace.
None of the functions therein are properly documented which
can lead to confusion. This patch set adds clarifying
documentation for all the functions.
Change-Id: I93545331ffc3a83b593f654ee90fb6af3d067402
This patch set fixes a recent issue that was introducing
following a refactor Pegleg table output. The related
CLI unit tests were changed to stop checking for an exit_code
of 0 due to improper mocking causing an exit_code of -1
to be returned instead.
This patch set corrects the issue by refactoring the code to
allow for exit_code 0 to always be checked.
Further refactoring was done in places to reduce code
duplication via helpers.
Change-Id: Ib9cf7c71b02d74de152b7e34911c867d2251a18a
As it currently stands, Pegleg clones site repositories into the /tmp
directory. Even if the site repository already exists in the /tmp
directory it is still cloned there which results in wasted disk space.
This commit allows users to pass in a `clone_path` (-p) option to Pegleg
CLI commands that specify where to clone a site repository. If the clone
path matches the path of an existing repository, then a error message is
logged stating so. If the repository already exists in the clone path, the
user can either specify to use that local repo by passing it as the site
repository or they proceed by passing in a different clone path.
This commit also updates the logic that deletes the copy of the repo that
is created in the temporary folder to also delete the parent folder that
contains the copied repo. This scenario happens when using a local
repository as the site repository.
Addionally, this commit adds a cleanup fixture that removes files and
directories created in the temporary folder by the unit tests.
Change-Id: I1b2943493b8f201f337ea60006c009973dd941b3
This patch set enables Pegleg to support repos like Airship in a
Bottle -- those that have site/ type/ global/ folders nested
under deployment_files/. Very particular logic is needed in order
to handle that. CLI unit tests included for validation/regression.
Change-Id: I9f13f59738599f07329ad3e3274eb4590e8638f9
If a repository remote ends with `.git` Pegleg is successfully able
to return the repository name. The same is not true in Pegleg if the
repository name does not end in `.git`. This commit allows Pegleg to
support repository names that do and do not end in `.git`.
Change-Id: I45fd15790677e003af7be584221903551022b7d7
This commit leverages python prettytable to create tables and output
CLI information for the following:
- site:
- lint
- list
- show
- type:
- list
- repo:
- lint
Addtionally, this commit changes the verbosity settings for pegleg CLI.
When verbosity is not set, only error logs will be shown as this would
be useful to users when errors do occur. Otherwise, no logs should be
shown in the CLI output unless the user passes the `verbose` flag.
Change-Id: Ic7782e9e383a1d6a7e31ff7cce025beb53c7db01
Currently Pegleg really needs CLI tests that test the CLI workflow
end to end. This continues off the initial legwork for it here:
https://review.openstack.org/#/c/605289/
This adds positive tests for all CLI site actions. Negative tests
will be added in the future.
Change-Id: Icf6737c6204edf518f2b384dd8369de793cb303a
With the implementation of revisioned repository to the CLI in
https://review.openstack.org/#/c/577886 there was a change to
the lint command [0], which changed it from being a global lint
to a site-level (targetted lint)... kind of: Only the CLI logic
was modified to support targetted single-site linting. Thus,
the first issue this patch set addresses is implementing the
back-end logic to realize targetted, single-site linting.
The second issue this patch set addresses is re-supporting global
linting (linting all sites within a repository) which means that
this partially reverts [0] which had (kind of) replaced global
linting with per-site linting.
So, this patch set:
1) Implements targetted, single-site linting back-end logic
2) Re-implements global linting for all sites in a repo
3) Adds unit tests for both
4) Adds some helper functions to util.engine.definition to
help with 1) and 2)
[0] https://review.openstack.org/#/c/577886/4/src/bin/pegleg/pegleg/cli.py@191
Change-Id: I5147282556763d93dfaf06912d2c4c876e1bd69f
This patch set adds support for passing in URLs ending in .git
via the --site-repository (-r) flag, e.g.:
pegleg site -r https://github.com/openstack/project.git@foo [...]
Unit test added to validate behavior.
Change-Id: If822ed195f07dc25e0590e42d046730632fdaafd
This patch set tidies up collect output file names for cloned
repos. As an example, previously the output file could be named
something like tmp5ak7409aic-clcp-security-manifests.yaml -- after
this change it will be aic-clcp-security-manifests.yaml instead
which is much cleaner.
Pegleg "collect" looks at the last piece in a file path to
determine the filename to write to: [0]
This means that changing the clone path here from something like:
/tmp/tmp54d13yairship-pegleg (which will result in collection
output to tmp54d13yairship-pegleg.yaml)
To:
/tmp/tmp54d13y/airship-pegleg (which will result in collection
output to airship-pegleg.yaml)
[0] 2ea1a55a2d/pegleg/engine/site.py (L71)
Change-Id: I2198e299d392e24d376ccfa53bef57fcabf0d41b
This patch set fixes an issue where currently -r flag won't
work with remote repository URLs. site_repository.process_repositories
will blow up with an error (for example):
Error: https://github.com/openstack/airship-treasuremap/site/
airship-seaworthy/site-definition.yaml not found. Pegleg must be
run from the root of a configuration repository.
It is apparent that the URL should not be treated as a local path
which is what is happening behind the scenes with this bug.
Also, CLI unit tests are added to validate the intended behavior.
They will be expanded on in future patch sets.
Change-Id: I618465841f1e455c8f00f046b3c5d22348b99396
This patch set increases test coverage for some missing git
functions and adds some additional edge case/exception handling
logic to those functions.
Change-Id: I54348599ea959691254c678bc836025cf8df8722
This patch set standardizes the Pegleg directory
structure because of the following reasons:
1) src/bin/pegleg is not necessary and only makes
building (e.g. documentation building) and running
of tox targets unnecessarily difficult.
2) src/bin/pegleg is a Java-like standard that
bears no relevance to Python.
Change-Id: I37d39d3d6186b92f8fbfe234221c9e44da48cf10
