pyproject-build (https://build.pypa.io/) is used as a pyproject (PEP
517) build frontend. This role installs it if missing.
This new role is basically a copy of the `ensure-poetry` role, in
turn copied from other roles in this repository.
Change-Id: If6e3970d995256beea170cad039d7dba9538d191
Ansible-core 2.16.4 appears to have a behavior change where it
will include the implicit localhost in hostvars, which means that
any location we iterate over hostvars and assume it's a real host
could throw an exception. To avoid that, add checks that the
variables we are about to access on the host exist.
Change-Id: Iff89da761e5f6748b454610a64c2fdd4f5e56a77
This adds a role (and job) to mirror container images from one
registry to another.
Also, disable the name[template] ansible-lint check because it
greatly reduces the utility of including templates in task names.
Change-Id: Id01295c51b67ffb7e98637c6cdcc4e7a14c92b22
the openvswitch.openvswitch collection is removed from Ansible packages
starting with Ansible 11. This causes ansible-lint to correctly not find
the openvswitch_bridge module when ansible-lint runs with Ansible 11.
Workaround this by capping Ansible used by ansible-lint to <10 and leave
a note about the module going away where we use it.
Change-Id: Id2d4e4f59c7d7e595c5458bc8717146c2326c573
This job that tests the zuul-jobs tox role runs various tox targets
including the docs target. This means we need to install dependencies
for doc building. On Ubuntu Noble this include libjpeg-dev for Pillow
wheel building and we fail without this profile installed.
We keep the default profiles of compile and test because we also run
other targets that likely depend on these profiles.
Change-Id: Ifa3495488f35b1fbe4fc665c4d0ac5ed8adb33aa
Fstrings are not supported in python3.5 which is in use on Xenial.
We don't claim to support Xenial, but this is an easy regression
to avoid.
Also, add test jobs for this role so that we get feedback before
copying it to the prod roles.
Also, add a xenial test job to exercise it since we still have
Xenial nodes available.
Change-Id: Ifc773aa688adb1a01cfe691b3bdca0b3086658cd
This adds a role convert-diskimage which uses the qemu-img tool to
convert diskimages from one format to another. Currently supported image
formats are raw and qcow2.
Change-Id: I4770af04c37f39e0cce23d5dd59ead744bed7d74
This adds a role variable to configure the diskimage-builder environment.
This allows users a choice of using the Ansible "environment" argument,
or using a variable. The variable may be particularly useful since it
allows full configuration of the role from a Zuul job definition.
Change-Id: I68542f13454b4f2e2e9bb8d356feefddba23d8f2
* This adds some extra options to the ensure-kubernetes role:
* podman + cri-o can now be used for testing
* This mode seems to be slightly more supported than the
current profiles.
* The location for minikube install can be moved.
* The use-buildset-registry role needed slight updates in order
to populate the kubernetes registry config early.
Change-Id: Ia578f1e00432eec5d81304f70db649e420786a02
* It looks like zuul-jobs-test-registry-buildset-registry-k8s-crio
is busted with Ubuntu Jammy + cri-o installed from kubic, with
errors like https://github.com/cri-o/ocicni/issues/77
(also, kubic has been wound down and cri-o has been spun off)
* cri-o in Noble uninstalls docker-ce, in a follow-up we should
clean that up and switch to a pure podman profile
* This minikube configuration is not supported, but it seems that
upstream cri-o might have made some fixes that makes it work
* Update the job to use Ubuntu Noble instead of Jammy
* Update ensure-podman for Ubuntu Noble
(podman is now part of the Ubuntu distro)
* Update the cri-o install in ensure-minikube for Ubuntu Noble and later
(cri-o is now part of k8s)
Other miscellaneous fixes and workarounds:
* k8s.gcr.io is being sunsetted, updated the test image:
https://kubernetes.io/blog/2023/03/10/image-registry-redirect/
* Relaxed the security to run minikube from /tmp (in future,
we should set the default to /usr/local/bin)
* Updated the microk8s check-distro task for Noble
Change-Id: I3b0cbac5c72c31577797ba294de8b8c025f8c2c3
Poetry (https://python-poetry.org) is not declared as a dependency for a
Python project, it must be available somehow in the system. This role
installs it if missing.
- Latest version is installed, unless `ensure_poetry_version` is
informed.
- The installed executable path is set as the `poetry_executable` fact.
- The `/usr/local/bin/poetry` symlink can also be created if
`ensure_poetry_global_symlink: true`.
This new role is basically a copy of the `ensure-nox` role with the
symlink creation snippet taken from the `ensure-tox` role.
The commit adding `ensure-nox` (77b1b24) has been taken as an example of
the necessary changes when adding a new role.
Change-Id: I5592d38d415a9d74055348406653b69f110541ae
We have support for installing python from pyenv, but it currently
requires setting the python version explicitly as an argument. If
the repo in question has a .python-version file, we shouldn't need to
require the user to provide that version a second time. Instead, we can
read from the file for the install step.
Change-Id: Ic4c2d3fc7f55169cec5211010fc3a9622fa324d1
There are a number of updates we make for Ubuntu Noble in this commit.
1. Remove python2-dev from bindep for Noble as Noble appears to have no
python2 runtime options.
2. Add libjpeg-dev to bindep for noble because Pillow doesn't build
python3.12 wheels for Pillow<10 which we currently depend on. This
means we need to build from source and that depends on libjpeg-dev.
3. We remove double bracket wrappers from ansible vars in ansible
assertion blocks. Having them results in errors like:
Conditional is marked as unsafe, and cannot be evaluated.
4. We update rust testing to explicitly install pkg-config before
building python cryptography. This tool is required to build
cryptography from source and is no longer being pulled in either
by the base images or build-essential meta pacakge.
5. Add an Ubuntu-24.04 tasks file for the ensure-skopeo roles so that
we try to install skopeo using distro packages or build from source
and don't use Kubic which only has packages for old Ubuntu releases.
Change-Id: I388710ce40dc757ada4de819a9c3c59fc32fb07a
As a followup to I4d05f9b187f9e40c3dcb2597e08c5bb50c261b17
We switch buildset-registry jobs to debian bookworm which has new enough
golang to build the latest skopeo version. Latest skopeo is used in
order to get api version negotiation behavior which is necessary for
talking to modern docker (version 25 or newer).
Change-Id: Ie673ef6724b0a40e3cfb2ba83e90d566e1f1837c
Co-Authored-By: Clark Boylan <cboylan@sapwetik.org>
The prepare_workspace_sync_required_projects_only variable allows
users to define which projects to sync to the node. This can prevent
syncing of unnecessary repositories. For some builds e.g. the
depends-on repositories dont need to be synced. The projects are
filtered based on the 'required' flag present in each zuul.project
entry and the required projects list also does not contain projects
which are present due to Depends-On or gate queue sequencing.
Having unnecessary repos in the workspace can for example also break
the analysis phase of bazel.
Change-Id: I3cc36cbfc60c81956caf5137da63973aeade4e21
Co-Authored-By: James E. Blair <jim@acmegating.com>
Co-Authored-By: Bernhard Berg <bernhard.berg@bearingpoint.com>
This is no longer present in Ansible 9.
Removing these upsets ansible-lint, so those errors are ignored.
The base roles job has bitrotted on centos-7 and bionic due to
a bad voluptuous release used in an stestr test. That is fixed in
this change as well.
Change-Id: I67886d5ad82ab590979f82bd102d6f974b9d4421
We can now run untrusted code on the executor, so we can re-enable
these tests.
This reverts commit 44d2187e7fcea0683c75db530458088b8371bb4e.
Change-Id: I65960af09a3cf3da1e780a96b6d13ddba3ce73d8
This test pins stestr (reason unclear) but does not pin python-subunit.
Python-subunit has made a release incompatible with that version of
stestr, so pin it.
Separately, it may be useful to see if we can unpin both.
Change-Id: Ia7cc45dc53ff0697e7ec84479c08c93e7d872a76
The container roles assume a full image url path and do not work with
shorted names like the docker roles do. The buildx path used the shorted
path when it should use the full path. Fix this.
Change-Id: Icdeee0ebb1c0d280968f425f8560cc5208ebd18d
This ended up calling into push-to-intermediate-registry with both
docker_images *and* container_images variable set.
This hid from testing that push-to-intermeidate-registry was not
working with only the container_images variable set.
Split these calls up so we don't have both variables defined.
Change-Id: If84b039852f2afc4df66c98e64fcce6f30f51246
This role uses skopeo to perform image operations.
Also update the container roles docs to add missing documentation
for the already existing upload-container-image role. Clarify
some ambiguity about the registry and repository attributes of
the container images data structure.
Change-Id: Ib66c85daf0edacf0dd797ab34b0d629f99c7111b
Co-Authored-By: James E. Blair <jim@acmegating.com>
This warns about pulling unversioned things from git as it is not
idempotent. Ignore in this case.
Change-Id: I3eb58fb1ee12a82223c63f19e591d908f45cfacc
Use pipefail in some shell commands. In this case I don't think we
can really be fooled, but not a bad idea to fail if the first command
errors.
Change-Id: I25750c4edfe815af9e9d9ee47639b315e7133aa2
This adds names to blocks and includes for consistency. We've done
this before (e.g. Ia7e490aaba99da9694a6f3fdb1bca9838221b30a) but I
guess 6.12.0 is finding more...
Change-Id: Ib451f6d3c5a18047873e63aa0a1aa2b425846fec
These all trigger command-instead-of-shell for ansible-lint 6.12.0.
It seems a few were ignored with warnings with
I4e415cbd34f0f4cb15857051bf95458e0316de86.
I don't see why these can't be command: for consistency
Change-Id: Ib0f590b461d2a5a7d9bb8bdddcbbfb2230cc3d1c
This is currently failing as buildx is incomaptible with the old
version of skopeo.
Switch to jammy nodes and install an updated skopeo for testing.
Change-Id: I40b9134200bcbbbe469acab3aedbea2eaf4c0f14
To test current buildx we need a later skopeo that understands the OCI
manifest formats. c.f. the zuul-executor image updates with
Iab667a92a5b6e6f8591db2aa435a782913d9d34f.
A recent enough version (1.9) isn't packaged for any LTS distros. So
add an upstream option, but it's only implemented (for now) on Jammy,
where we will test with it.
Change-Id: I206a3cbfb16575f409771d96c2b7e49929e61a49
This enables microk8s/containerd to pull through the intermediate zuul
registry. This is tested with the new
zuul-jobs-test-registry-buildset-registry-k8s-microk8s job.
Change-Id: I5a6c0d63a6ba0acf94ab9f0ef94777fab58fec6e
Add microk8s support to the ensure-kubernetes role. This installs via
a snap, and is currently only implemented for Ubuntu Jammy.
Mostly this is a straight-forward installation. I did notice though
it needs a little more time to be stable in the test, so the timeout
is bumped slightly.
microk8s is the Ubuntu "blessed" way of doing things. This should be
a better choice for Ubuntu platforms, because minikube is tightly tied
to cri-o, which is only packaged through kubic, which is currently in
some sort of deprecated but best-effort supported mode [1]. This was
inspired by an outage where the kubic gpg expired. This appears fixed
now.
[1] https://kubic.opensuse.org/blog/2022-06-10-kubic-retired/
Change-Id: Id3e31c70a35dde218e35e7c50964f8a3c0348150
The kubernetes + docker jobs are failing because the ensure-kubernetes
role no longer works with the docker runtime. It will be updated to
use microk8s in a later change, and we will deprecate its use with
docker.
Change-Id: Ia0a6d470ddfe594810ad761ed3494884f56cdb46
Some projects are converting from tox to nox for driving tests. This
means there isn't a tox env to find testr/stestr in. Update
fetch-subunit-output to look for nox envs as well.
Change-Id: I051c4b27d22921f1f0c3a44dc4eaccdbb50afa29
