Also add centos-platforms for the zookeeper test matrix since it
appears the role may have support for redhat but is not being
tested in that environment.
Change-Id: I4ab0e4fe42eab46ff8286760518ea9ed280b6733
This role ensures that a specific pip-installable command is
available.
Example usage:
- role: ensure-python-command
vars:
ensure_python_command_name: poetry
ensure_python_command_version: ==1.8.5 # omit to install latest
In this case, if the `poetry` command is not already available, pip will
install it in a new venv. Either way, after running this role, the
`ensure_python_command_executable` variable will hold the full path to
the command.
We already have similar roles for specific commands:
- ensure-nox
- ensure-poetry
- ensure-pyproject-build
- ensure-tox
- ensure-twine
- ensure-uv
These roles are essentially copies of each other with different command
names. This new role consolidates that code. The existing roles now act
as wrappers that just set variables and call the new role.
> Note: The `ensure-tox` role has not been refactored due to exclusive
> legacy code related to Python 2, which must be removed first.
The new role introduces three variables to replace the overloaded
`ensure_<command>_executable` variable from the other roles:
- `ensure_python_command_name` (input, command name)
- `ensure_python_command_existing` (input, existing path for the command)
- `ensure_python_command_executable` (output, detected/installed path)
This separation avoids using the same variable as both input and output,
which can cause issues due to Ansible's variable precedence rules:
https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_variables.html
Understanding variable precedence
...
19. set_facts / registered vars
20. role (and include_role) params
...
Since we use `set_fact` inside the role, it is ineffective when the same
variable is also passed as a role parameter :/
I'm not adding tests for the new role because its functionality is
already covered by the existing tests for all the refactored roles:
- test-playbooks/ensure-nox.yaml
- test-playbooks/ensure-poetry.yaml
- test-playbooks/ensure-pyproject-build.yaml
- test-playbooks/ensure-twine.yaml
- test-playbooks/ensure-uv.yaml
Change-Id: Idd970cb31bd928576bca3602ce96fbc491ecdb60
This adds a role similar to upload-image-swift to upload dib images
to s3 for use by zuul-launcher.
Change-Id: Ie802a0221717e6d5d5dcaa91771d07f8d0321889
Install twine into a venv and set appropriate environment
variables. Also added tests.
Based on commit adding `ensure-nox` (77b1b24) role.
Related-bug: #2095514
Change-Id: Ibb4e89f79879b4d0ae0294440c9c0b79fc57a7fa
Uv (https://docs.astral.sh/uv/) is not declared as a dependency for a
Python project, it must be available somehow in the system. This role
installs it if missing.
- Latest version is installed, unless `ensure_uv_version` is
informed.
- The installed executable path is set as the `uv_executable` fact.
- The `/usr/local/bin/uv` symlink can also be created if
`ensure_uv_global_symlink: true`.
This new role is a verbatim copy of the `ensure-poetry` role, just doing
a `s/poetry/uv/g`. Even this commit is a replay of the commit adding
that role: 524b7e7b95dcd6adc311e74dd7f0e6da8a3cce58.
Change-Id: I55bc5e1d273045d0978b09f719bf79a875336e30
pyproject-build (https://build.pypa.io/) is used as a pyproject (PEP
517) build frontend. This role installs it if missing.
This new role is basically a copy of the `ensure-poetry` role, in
turn copied from other roles in this repository.
Change-Id: If6e3970d995256beea170cad039d7dba9538d191
This adds a role (and job) to mirror container images from one
registry to another.
Also, disable the name[template] ansible-lint check because it
greatly reduces the utility of including templates in task names.
Change-Id: Id01295c51b67ffb7e98637c6cdcc4e7a14c92b22
This adds new style mirror_info handling to use-docker-mirror to give us
greater control over whether or not docker hub should be mirrored. We
ignore old style configuration if new style is present which gives us
this control. Otherwise we fallback to the old behavior.
We also update the ensure-docker test jobs to be triggered by updates to
the use-docker-mirror roles as ensure-docker includes this role. We
should get decent functional testing coverage this way.
Change-Id: Ia1b216a6dd68bcafbe599777037c5d7b1b3e8201
If you need to run native arm64 builds, you can take advantage
of this change which will rely on the remote builders in order
to build things natively giving a significant speed up in
container build time.
Change-Id: I962bb2357a2c458d5e72b334b4fe36b55b034864
The ensure-nodejs role defaults to install nodejs 6 which produces this
error currently:
Failed to update apt cache: W:The repository
'https://deb.nodesource.com/node_6.x noble Release' does not have a
Release file., W:Data from such a repository can't be authenticated
and is therefore potentially dangerous to use.
We need to make a few changes to bring this ensure-nodejs role up to
modern expectations for nodesource usage. First we drop the default
nodejs version from ensure-nodejs. Everyone is already setting this
value to make this role work or they are broken and will need to change
something anyway. This gets us off of the nodejs update treadmill in
this role.
Then with nodejs 16 and newer there is a new gpg key and no deb-src
packages so we need to change the apt configuration if using 16 and
newer. We make these changes to match the corresponding setup_16.x etc
scripts from nodesource.
Change-Id: I0d5c93e4fbcee0be2cc477bf9f625e419a2b9bd1
Previously we pinned to 1.28/stable due to a bug that prevented
1.29/stable from working. Now we've hit a new issue with 1.28/stable on
bookworm. The fix for that appears to simply be to upgrade to
1.31/stable so we do so here. More details can be found in this GitHub
issue:
https://github.com/canonical/microk8s/issues/4361
The new version appears to return from the snap installation before the
k8s installation is fully ready to deal with add-on installation. This
occasionally produces errors like:
subprocess.CalledProcessError:
Command '('/snap/microk8s/7178/microk8s-kubectl.wrapper', 'get',
'all,ingress', '--all-namespaces')'
returned non-zero exit status 1.
Work around that with `microk8s status --wait-ready` to ensure that k8s
is up before adding addons.
While we are at it we also update the collect-kubernetes-logs role to
collect microk8s inspect output as that would've enabled us to debug the
above issue without holding nodes. We also update test jobs to trigger
when the collect-kubernetes-logs and collect-container-logs roles are
updated to ensure we get coverage from those jobs when updating these
roles.
Change-Id: I60022ec6468c2cadd723a71bbc583f20096b27dc
Fstrings are not supported in python3.5 which is in use on Xenial.
We don't claim to support Xenial, but this is an easy regression
to avoid.
Also, add test jobs for this role so that we get feedback before
copying it to the prod roles.
Also, add a xenial test job to exercise it since we still have
Xenial nodes available.
Change-Id: Ifc773aa688adb1a01cfe691b3bdca0b3086658cd
This adds a role convert-diskimage which uses the qemu-img tool to
convert diskimages from one format to another. Currently supported image
formats are raw and qcow2.
Change-Id: I4770af04c37f39e0cce23d5dd59ead744bed7d74
* This adds some extra options to the ensure-kubernetes role:
* podman + cri-o can now be used for testing
* This mode seems to be slightly more supported than the
current profiles.
* The location for minikube install can be moved.
* The use-buildset-registry role needed slight updates in order
to populate the kubernetes registry config early.
Change-Id: Ia578f1e00432eec5d81304f70db649e420786a02
* It looks like zuul-jobs-test-registry-buildset-registry-k8s-crio
is busted with Ubuntu Jammy + cri-o installed from kubic, with
errors like https://github.com/cri-o/ocicni/issues/77
(also, kubic has been wound down and cri-o has been spun off)
* cri-o in Noble uninstalls docker-ce, in a follow-up we should
clean that up and switch to a pure podman profile
* This minikube configuration is not supported, but it seems that
upstream cri-o might have made some fixes that makes it work
* Update the job to use Ubuntu Noble instead of Jammy
* Update ensure-podman for Ubuntu Noble
(podman is now part of the Ubuntu distro)
* Update the cri-o install in ensure-minikube for Ubuntu Noble and later
(cri-o is now part of k8s)
Other miscellaneous fixes and workarounds:
* k8s.gcr.io is being sunsetted, updated the test image:
https://kubernetes.io/blog/2023/03/10/image-registry-redirect/
* Relaxed the security to run minikube from /tmp (in future,
we should set the default to /usr/local/bin)
* Updated the microk8s check-distro task for Noble
Change-Id: I3b0cbac5c72c31577797ba294de8b8c025f8c2c3
The podman socket is owned by root by default, so add a podman group
(like the docker group) to allow the zuul/ansible user to access it.
Also, add support for Ubuntu noble.
Change-Id: I653d9c313c69298da00b139a791a6177d37475cd
Poetry (https://python-poetry.org) is not declared as a dependency for a
Python project, it must be available somehow in the system. This role
installs it if missing.
- Latest version is installed, unless `ensure_poetry_version` is
informed.
- The installed executable path is set as the `poetry_executable` fact.
- The `/usr/local/bin/poetry` symlink can also be created if
`ensure_poetry_global_symlink: true`.
This new role is basically a copy of the `ensure-nox` role with the
symlink creation snippet taken from the `ensure-tox` role.
The commit adding `ensure-nox` (77b1b24) has been taken as an example of
the necessary changes when adding a new role.
Change-Id: I5592d38d415a9d74055348406653b69f110541ae
CentOS 8 Stream is EOL and these jobs are no longer functional. We would
like to clean up the test nodes entirely as a result of this. Part of
this cleanup is either deleting existing jobs that rely on CentOS 8
Stream or converting them to run on a different platform. In this case
we update the fips role job to run on CentOS 9 Stream.
Change-Id: Ifb7d6deaf529e21c257590d63938026584bee7ae
We have support for installing python from pyenv, but it currently
requires setting the python version explicitly as an argument. If
the repo in question has a .python-version file, we shouldn't need to
require the user to provide that version a second time. Instead, we can
read from the file for the install step.
Change-Id: Ic4c2d3fc7f55169cec5211010fc3a9622fa324d1
Python 3.12 is a thing now and is present in a stable Linux distro
release (Ubuntu Noble). It seems reasonable for people to want to run
nox and tox against py312 targets. Let's make it easy for them and add
jobs to do that for them.
Change-Id: I9d644cfbe65b92207a5fe2ad6dd950093bda87dd
We were pinning ansible to 2.8 for unittesting but Zuul currently only
supports Ansible 8 and 9. Pin Ansible to 8 in order to better test what
we expect people are using in the wild. This will also enable the
testing of Zuul and Ansible and Zuul Jobs with newer python versions
like 3.12.
In order to run testing with Ansible 8 intsead of 2.8 we drop testing
against python2.7 and python3.8 as neither of them can install an
Ansible version this new.
Change-Id: Icd563def65dcfd40b174218cc4e2b94e0230c374
The package repos for centos-8-stream are empty now, so it can no longer
be deployed. Ubuntu Bionic is EOL and about to be deleted from the
opendev infrastructure, too.
Change-Id: I5e14957f2c866fc3e337ac4051692c68b450ceba
OpenDev has recently added ubuntu-noble nodes, so let's test our
standard library on that platform now.
Change-Id: I8d49696812e6b0c67723fba9242ea6a4f5cd5026
As a followup to I4d05f9b187f9e40c3dcb2597e08c5bb50c261b17
We switch buildset-registry jobs to debian bookworm which has new enough
golang to build the latest skopeo version. Latest skopeo is used in
order to get api version negotiation behavior which is necessary for
talking to modern docker (version 25 or newer).
Change-Id: Ie673ef6724b0a40e3cfb2ba83e90d566e1f1837c
Co-Authored-By: Clark Boylan <cboylan@sapwetik.org>
This reverts *parts* of commit 9519fafd102017e54f4d543f25ca4398f1ec67b8.
Specifically, this reverts the main fix regarding DOCKER_MIN_API_VERSION.
Reason for revert: Docker 26.0.0 is out, which drops the old API
This patch additionally modifies the Zuul config in two ways:
* disables buildset-registry jobs because they are children of
opendev-buildset-registry (a base-job in opendev) and thus
do not pick up new changes - they are reenabled in a followup
* disables crio jobs because they have their repo seemingly broken
Change-Id: I4d05f9b187f9e40c3dcb2597e08c5bb50c261b17
Co-Authored-By: Radosław Piliszek <radek@piliszek.it>
CentOS 7 is going EOL in the near future. Due to a lack of demand/need
within OpenDev and the upcoming EOL OpenDev has decided to drop CentOS 7
test nodes entirely. This will free up room for OpenDev to build and
mirror distro releases that are more relevant today. In preparation for
this removal we drop the zuul-jobs CentOS 7 testing.
Change-Id: Id49d28dd426ecbb3470e234d3b67875a6b097112
The prepare_workspace_sync_required_projects_only variable allows
users to define which projects to sync to the node. This can prevent
syncing of unnecessary repositories. For some builds e.g. the
depends-on repositories dont need to be synced. The projects are
filtered based on the 'required' flag present in each zuul.project
entry and the required projects list also does not contain projects
which are present due to Depends-On or gate queue sequencing.
Having unnecessary repos in the workspace can for example also break
the analysis phase of bazel.
Change-Id: I3cc36cbfc60c81956caf5137da63973aeade4e21
Co-Authored-By: James E. Blair <jim@acmegating.com>
Co-Authored-By: Bernhard Berg <bernhard.berg@bearingpoint.com>
Debian buster is the old old release of Debian (it has been replaced by
bullseye and bookworm). Drop testing of this release so that OpenDev can
drop image builds and mirroring of content.
Change-Id: I7244c045d3346ff8c222ab5afc77dfbe05420cae
The OpenDev team is planning to remove OpenSUSE LEAP 15 images as our
node builds and mirrors are for 15.2 which is ancient and no one is
currently working to modernize these test environments. On top of that
LEAP is apparently going away in the future and will be replaced with
another distro.
Change-Id: I7b35561f2577d0d2a3f951199e10c509bf7de421
Skopeo hardcoded the docker api version for image manipulation to
version 1.22 of the api until very recently. Docker 25 sets 1.24 as a
minimum version by default introducing an incompatibility between the
tools. It isn't straightforward to install an updated skopeo everywhere
we need it (due to golang requirements). As a workaround we override
the min version to 1.22 when installing the docker daemon. This should
work until Docker 26 is released and removed the override option.
Note we also pin microk8s from latest/stable (which is currently
1.29/stable) to 1.28/stable to workaround
https://github.com/canonical/microk8s/issues/4361. This is necessary to
get the CI jobs for docker/registry/k8s testing working in order to land
this fixup.
Change-Id: I377ac84d532749eba578c4b32eb2ed6a5ce7a0c9
The openshift testing relied on openshift 3.11 as this version of
openshift can be deployed locally without using baremetal management.
Unfortunately, docker images for this version of openshift are no longer
available leading to errors like:
pod_workers.go:186] Error syncing pod b14b1b90-cb64-11ee-816f-bc764e2013fc ("openshift-service-cert-signer-operator-6d477f986b-qx6cs_openshift-core-operators(b14b1b90-cb64-11ee-816f-bc764e2013fc)"), skipping: failed to "StartContainer" for "operator" with ImagePullBackOff: "Back-off pulling image \"openshift/origin-service-serving-cert-signer:v3.11\""
When deploying this version of openshift. Since openshift 4 is not
generally installable in a similar manner we drop the testing. The
testing of the registry with other versions of kubernetes should give us
reasonable coverage.
Change-Id: I6ac87edee2123a5105f3b8138157faa7ecd94b0d
