This PS replaces old airskiff integration gate
with new kubeadm based airskiff integration gate.
The main goal of this gate is to test new deckhand
image and chart together with other Airship
components.
Change-Id: I4221b7be64e9a7e07964fa817d42c21a3f172db0
Upgrading htk to version 0.2.55, which deprecates the ingress class
annotation (kubernetes.io/ingress.class) with .spec.ingressClassName
https://review.opendev.org/c/openstack/openstack-helm-infra/+/891720
Change-Id: I573a926ab6fb07f10c0c4d9020746ba80e6d8dbd
Signed-off-by: Anselme, Schubert (sa246v) <sa246v@att.com>
This PS updates python modules and code to match Airflow 2.6.2:
- bionic py36 gates were removed
- python code corrected to match new modules versions
- selection of python modules versions was perfrmed based on
airflow-2.6.2 constraints
- airskiff deploy pipeline was aligned with latest in treasuremap v1.9
- postgresql image updated to 14.8
Change-Id: I65a1b86473ee3e988aae353b59fb5473d75851f9
This PS makes the following changes:
- uses deploy-k8s.sh from treasuremap
- makes sure the airskiff-deploy playbook is using 80Gb partition if
available
- adds available security updates to docker images
Change-Id: I0f330cb15ec32b12703f0bc6620b3f3c797a25bb
This PS restores image build for ubuntu_bionic and adds appropriate
gates to keep it tested by appropriate functional and integrational
tests.
Also the latest osh-infra commit was used with kubernetes 1.27.1 for
integration tests.
Change-Id: Ia2c951f27b96774b553e7c0c7c6809172312f753
- adjusted .gitignore to keep fresh egg-info and omit build artifacts
- fresh egg-info data is needed for promenade that depends on Deckhand
- restored deckhand-functional-uwsgi-py38 gate
- restored deckhand-integration-uwsgi-py38 gate
- made deckhand-airskiff-deployment gate voting ( treasuremap project
has been updated)
- removed bionic gates
- updated focal dockerfile
- added more binary deps into bindep.txt
- updated deckhand chart values to latest images - focal and wallaby
- fixed python code to compy with CVE's found by fresh version of bandit
- implemented pip freeze approach
- added tox -e freeze profile to manage it
- requirements-frozen.txt is now main file with requirements
- requirements-direct.txt is the file to control deps
- updated setup.cfg to adjust to newer version of setuptools
- fixed airskiff-deploy gate
- fixed docker-image-build playbook to restore Quay repo image publish
- updated other playbooks to include roles from zuul/base-jobs in order
to setup build hosts properly
- removed workaround with hardcoded dns resolver ip 10.96.0.10 as it
became obsolette due to recent fix in openstack-helm-infra
- adjusted tools/whitespace-linter.sh script
- tox.ini has been brought to compliance with tox4 requirements
- replaced str() calls with six.text_type() according to D325 Deckhand specific
commandment from Hacking.rst
- locked python-barbicanclient version with 5.2.0 because of breaking
changes in the upper versions
Change-Id: I1cd3c97e83569c4db7e958b3400bdd4b7ea5e668
update dockerfile for python deckhand install
add deckhand version to chart 1.0
add chart version 0.2.0
update all packages to latest in requirements.txt
update zuul jobs for focal and python 3.8
remove zuul job functional-uwsgi-py38 in favor of functional-docker-py38
update tox config
typecast to string in re.sub() function
add stestr to test-requirements.txt
add SQLAlchemy jsonpickle sphinx-rtd-theme stestr to requirements.txt
deprecated function: BarbicanException -> BarbicanClientException
fix mock import using unittest
fix import collections to collections.abc
fix for collections modules for older than python 3.10 versions.
deprecated function: json -> to_json
deprecated function: werkzeug.contrib.profiler ->
werkzeug.middleware.profiler
deprecated function: falcon.AIP -> falcon.App
deprecation warning: switch from resp.body to resp.text
rename fixtures to dh_fixtures because there is an imported module
fixtures
switch from stream.read to bounded_stream.read
deprecated function: falcon process_response needed additional parameter
deprecated function: falcon default_exception_handler changed parameter
order
move from MagicMock object to falcon test generated object to fix
incompatability with upgraded Falcon module.
Adjust gabbi tests to fix incompatability with upgraded DeepDiff module
update Makefile to execute ubuntu_focal
update HTK (helmtoolkit)
unpin barbican to pass integration tests
Use helm 3 in chart build.
`helm serve` is removed in helm 3 so this moves
to using local `file://` dependencies [0] instead.
Change-Id: I180416f480edea1b8968d80c993b3e1fcc95c08d
Unpin python3-six for Opensuse image build.
Update helm-toolkit stable commit to merge of this change:
https://review.opendev.org/#/c/803654/
Update the helm installation script to download and install v2.17.0
Fix integration tests by pinning Barbican to stable commit.
Pinn jsonschema to 4.0.0a2 to fix tox tests
Change-Id: I2badd0e2f6c934098f0c9f5ef7e52354756c12e0
When pip is upgraded to 20.3, the pip dependency resolver is much more
strict and will no longer install a combination of packages that is mutually
inconsistent[0].
These changes account for the fact that Shipyard imports Armada, Drydock,
Promenade, and Deckhand. Having said that, with pip 20.3, the pip
packages amongst those projects cannot conflict. A follow-up change may
be needed if more conflicts are found.
[0] https://pip.pypa.io/en/latest/user_guide/#changes-to-the-pip-dependency-resolver-in-20-2-2020
Change-Id: Id75acea82ddf5d915a8b8805e076dac49cab800f
Use pip3 in event system has both pip2 and pip3 installed.
Use apt to install setuptools for Ansible's consumption.
Change-Id: I6929ecb0cce2ec8ac70e9261acb9f87dc7031153
Co-authored-by: Alexander Hughes <Alexander.Hughes@pm.me>
- With Ubuntu bionic based deckhand docker image, uswsi crashes with
segmentation fault when it tries to load psycopg2 library, causing
the deckhand service become unavailable.
The root cause of this problem seems to be that uwsgi and psycopg2
binary wheels are built with different ssl libraries.
To address this issue, upgrading psycopg2 to the latest release to
the latest binary release.
- For opensuse 15.1 image build, python3-six was updated to address
package not found issue.
- Updated gating ansbile playbooks to address non-voting gate failures.
Change-Id: I7be920e16e6114eb2bdbc052a6761f29008baf81
- Added ability to build deckhand docker image, using either the
Ubuntu xenial or the Ubuntu bionic base image.
- Made the bionic base image the default base image for deckhand
docker image build.
Change-Id: I26657de34a233ee3223a7f93fc667e734ac9140b
tox is failing because oslo.messaging 9.1.1 installs kombu 4.6.7, which
requires amqp >=2.5.2, but it is pinned to 2.5.1. Additionally, the
deckhand-integration-docker are failing because barbican is not being
deployed correctly.
This change updated the pinned version of amqp to 2.5.2, and uses the
barbican deployment script in openstack-helm.
Depends-On: I020632ad6a33d29a657164cf0e32baef6b670d5a
Change-Id: Idafe35d1496cd880e2d6a93c96afa724fd1c2e89
Fix problems with integration test zuul jobs which were made non-voting
by [0] and restore them to voting status. These jobs exercise some
secret-management functions that test integration between Deckhand and
Barbican that doesn't happen in the Airskiff jobs, so they still have
some test value.
- Some integration test scripts were pointing to old versions of
openstack-helm scripts that are no longer maintained. This PS
updates those scripts to point to current OSH scripts.
- Uplift openstack-helm-infra pin in helm_tk.sh to a recent
version
- Add build-images to uwsgi test jobs to ensure airship_deckhand_path
is set
[0]: https://review.opendev.org/660738
Change-Id: If83dead126307a98b44ad5d8f367a972c48a30d0
The airskiff pipeline attempts to build the Deckhand Docker image before
Docker is installed. This change moves the order of the airskiff
pipeline tasks to build the Deckhand image AFTER Docker has been
installed by the OpenStack-Helm minikube script.
Change-Id: I78158c8de9c4fe1f06499c1a5d917e31db9c6b4f
Signed-off-by: Drew Walters <andrew.walters@att.com>
Airskiff dependencies are installed from a script in the OpenStack-Helm
repository now, so the script was removed from treasuremap [0]. This
change removes a call to the deleted "000-install-packages.sh" script
that causes the Airskiff job to fail on changes submitted to the
Deckhand repository.
[0] https://review.opendev.org/672540
Change-Id: Iacb6131babffe794e9d3a18ff68f3cd6756a0b2c
Signed-off-by: Drew Walters <andrew.walters@att.com>
The location of the Airskiff site recently changed [0], causing the
Airskiff job to fail. This change updates the "reduce site" playbook to
match the job in treasuremap, thereby avoiding the directory which no
longer exists.
[0] https://review.opendev.org/674963
Change-Id: I4d82093cb8042e60f947b6e53a7817bebad518d8
Signed-off-by: Drew Walters <andrew.walters@att.com>
This removes several of the the required-projects from the Airskiff
gate as the scripts for the gate can already clone down the necessary
repositories and the required-projects can overwrite any pins that may
have been included in the Airskiff scripts.
This requires a new nodeset as well, because the Airskiff gate does not
presently work with ubuntu-bionic
Change-Id: Idff16a450af1e3f9fe0ee49496fd7d72df69b595
Adding DISTRO parameter for makefile to invoke distribution specific
Dockerfile and build image accordingly.
Updated logic for existing jobs to have distro specific logic
for building and publishing images.
Added multiple distro specific document in operator section.
Change-Id: I415ab28b06ea17b21e76d28ccb3e284041c8072d
This commit introduces a non-voting job to lint Helm charts against the
latest version of Helm toolkit from OpenStack-Helm Infra and a gate for
a pinned version of Helm toolkit.
Change-Id: Ia98209745641db011469d54220777110b2f6ae3a
- Zuul updated ansible to 2.7, no longer allows missing variables.
- Using default value when it isn't available.
- airship_deckhand_path isn't working, but is a non-voting gate.
Based on Aaron Sheffield's PS for Pegleg: https://review.openstack.org/#/c/645631/
Change-Id: I6fe13d2043480cd6de6785203373f35d83545196
Deckhand's integration job relies on OpenStack-Helm-Infra to deploy
Kubernetes using Kubeadm. Since [0] was merged, Calico requires AppArmor
to be deployed. This commit adds the same role to deploy AppArmor that
exists in OpenStack-Helm-Infra to prevent inadvertent gate failures.
[0] https://review.openstack.org/614805/
This also includes gate fix in https://review.openstack.org/#/c/627906/
because of a circular dependency; that change in question fixes
failing CI jobs for all non-integration jobs.
Change-Id: Ie3fb04ea0fbe0487ae743033315e2a4211248d32
Added container image tagging on quay.io with commit id on every merge,
when certain files but not the code are changed (docs, charts, etc.).
Previously we were building updated container images, adding tags and
uploading images onto quay.io only when the code has been changed. The
main reason for not re-building images for every single merged commit
was a long waiting time in Zuul `post` queue for a node (VM)
assignment for the job to run on.
With this change we will have image tag for every merged commit,
including documentation commits, and those tags would be added almost
immediately (within ~1 min), as job runs directly on Zuul and does not
wait in Zuul queue for a node (VM) assignment.
Change-Id: I9692947b74a16de0eb3e8100d30cd4310069abfe
1) Use OCI Image Specs for labels instead of custom 'commit-id=xxxxx'
or legacy "Label Schema"
2) Fix missing git commit id labels on images (.revision)
3) Add human-readable title (.title) of the image, URL (.url), and
a few other properties (annotations) according to the latest Specs
Change-Id: Iebb37edd003204d3adc41aa9af76612ab419993a
To cut down on Zuul nodes, this patchset combines the integration
docker job with the airship-deckhand-ubuntu job together, because
the former essentially subsumes the latter: integration docker
job sets up the host, deploys docker, deploys k8s, generates
the Deckhand chart using Helm, builds the Deckhand image using
Docker, deploys Deckhand, PostgreSQL and other Keystone/Barbican
dependencies using OSH, then finally runs integration tests
against the deployment.
This, then, validates everything that airship-deckhand-ubuntu
did deployment-wise.
Also, this patchset switches the 2 uwsgi jobs over to voting: false
as the Docker jobs are already voting.
Change-Id: I10b15f5a45a32413dc904ee32bdbbb4e9cdf98a2
This patchset adds integration tests job to .zuul.yaml so that
they can be run against Zuul per each commit.
The number of jobs that are executed in the gate pipeline
have been pruned because only *-docker jobs are of critical
importance. We can safely skip over *-uwsgi jobs in order
to save on Zuul resources.
Change-Id: I98ce49163b2db64cbdf18fe00b82c99fecd4139d
This patchset updates osh-infra-deploy-docker.yaml playbook
to align with role rename change in osh-infra causing pre-run
to fail: https://review.openstack.org/#/c/578703/6
Change-Id: I241a2692d232efdb1f15d23d57d53a7933a4bc98
This patchset fixes a schema bug to allow for single source multi
dest substitution to work (it is currently rendering successfully
but failing on validation as the base Deckhand validation schema
was not updated).
This adds a functional test to avoid regression.
Also fixes failed_when condition for functional-tests
and integration-tests roles to check if 'commands failed'
appears in stdout which only happens when pytest runner
fails.
Change-Id: I9bb0c51b3374bbaf9fd34d5f93f1a1777bb3b1e8
This patch set adds a trigger on post to trigger a RTD job to start
building the documentation post-merge.
Change-Id: Icd0ad1670b0d239e601a4456f84cc46e2f28aaf2
Moving the whitespace linting job as part of PEP8 job to reduce the need to
create a VM just to find whitespaces in none *.py files. Python files'
whitespace violation is covered as part of PEP8 job.
Change-Id: I3bc6047c9ab65e8f55f91a949a6c05ef4e38ea16
Signed-off-by: Tin Lam <tin@irrational.io>
This patchset allows Deckhand image to be built behind proxy
by adding appropriate variables to Dockerfile and to the Ansible
build image script.
Change-Id: Idff4adb4762b8062072a2da066044ecd7782ed9d
