This PS has airflow upgrade script that makes sure
that the airflow-worker that is hosting currently
running instance of the script will be deleted at
last. This makes sure that all airflow-workers
will be restarted to apply new config or image.
Change-Id: Iacf6e63e0247ae836e7eb5b7ff24d0369d5e5191
This PS applies a fix for airflow dags state
command output filter.
Also applied several zuul gates fixes and bumps up
kubectl to 1.29.5 due to CVE vulnerability
Change-Id: Ic623f27ef8c0f856f2a80ea73fe48c992cd8936b
This PS is mainly fixing SubDAGs timing issues when they
started along with main DAG and not at the time the main
DAG needs them. In Airflow 2.6.2 SubDAGs are deprecated
in favor of TaskGroups. So in this PS all SubDAGS were
replaced with TaskGroups.
Also task level logging config was extended by adding
py-console to obtain logs from tasks like it was
configured in Airflow 1.10.5.
Change-Id: I3f6d3961b1511e3b7cd2f7aab9810d033cfc14a3
This PS updates python modules and code to match Airflow 2.6.2 as well
as deploys new Airflow:
- bionic py36 gates were removed
- python code corrected to match new modules versions
- selection of python modules versions was performed based on
airflow-2.6.2 constraints
- airskiff deploy pipeline was aligned with latest in treasuremap v1.9
- shipyard chart was corrected to match new airflow cli, configuration
items and their default values
- added new celery configuration items and their values
- updated airflow runtime logging config
- disabled deprecation and future python warnings in airflow images
- added celery to the list of airflow providers
- adjusted airflow runtime scripts to match new cli
- shipyard SQL queries to airflow DB were adjusted to match new SQL
schema of the db
- shipyard_airflow and shipyard_client unit tests were updated to match
new DB structure and new cli
- airflow db sync job is using db upgrade command
- helm version uplifted to v3.12.2
Change-Id: Ife88e53ce0dd8dc77bf267de1f5e6b8361ca76fd
This PS makes the following changes:
- makes sure the airskiff-deploy playbook is using 80Gb partition if
available
- adds available security updates to docker images
Change-Id: Ibcb8fcddb319c3032483fdb73b55b725ea7737d4
This PS restores image build for ubuntu_bionic and adds appropriate
gates to keep it tested by appropriate functional and integrational
tests.
Change-Id: I4c4c7186e67c771f8249b2017b2507aac17760b7
This PS is created to :
- roll back apache-airflow back to 1.10.5
- change default chart values from bionic to ubuntu_focal
- save freshly generated config and policy files samples
- in focal Dockerfile postgres client version is pubped up to v15
- change airflow docker image requirements from limited to fully frozen for shipyard-airflow project
- adjusted requirements-direct.txt for shipyard_airflow
- adjusted requirements-direct.txt for shipyard_client
- regenerated requirements-frozen for both projects
- fixed unit tests after upgrading click module
- gen_all tox profile processing has been moved over to py8 gate because it requires focal zuul node to run
- upgraded airskiff gate playbook to include latest treasuremap patchset with updated airskiff site admada manifests
Change-Id: I47e44f5cfa19b2649697e7cc5a31557a6f4fcfea
- upgraded Airflow to 1.10.15 -
https://airflow.apache.org/docs/apache-airflow/1.10.15/changelog.html
- disabled xenial, bionic and opensuse images build gates
- added focal image build gate
- added focal zuul build node
- adjusted Makefile for focal
- added bindep.txt to utilize bindep zuul base role for zuul build node
pre-setup
- added focal Dockerfile
- implemented freeze requirements.txt approach like in other Airship
projects
- removed specific requirements.txt for airflow in favor of using
requirements-frozen.txt from shipyard_airflow project when building
airflow docker image
- fixed docker image publishing to Quay
- replaces deprecated LOG.warn with new LOG.warning call
- replaced deprecated body attribute in responce wiht responce.text
attribute
- update of falcon module deprecated .API call - replaced wiht
falcon.App call
- deprecated routing.create_http_method_map method replaced with
routing.map_http_methods
- re-formatted code tabulations based on yapf recommendations
- replaced deprecated protocol attribute in Pytest create_environ() with
http_version attribute
- replaced deprecated app attribute in Pytest create_environ() with
root_path attribute
- fixed airflow CLI commands to match 1.10.15 version
- updated zuul gates to work on focal nodes and added focal specific
node setup items by adding appriate ansible tasks and roles
- uplifted Helm to 3.9.4
- uplifted stable HTK commit id
- updated tox.in to with with tox v4
- uplifted dependences references to other Airship projects
- common python dependences were syncronized with other Airship
projects(Promenade, Deckhand, Armada, Drydock)
- fixed airskiff deployment gate
- fixed genconfig* profiles in shipyard-airflow tox.ini responsible for
maintanance of policy.yaml.sample and shipyard.conf.sample
Change-Id: I0c85187dc9bacf0849382563dd5ff7e9b2814c59
Connections from airflow-worker can be long-lived (e.g. when applying
large Armada manifests), and are prone to disconnection due to network
inactivity.
This change uplifts the Armada client to enable socket keepalives in
order to avoid this issue. (Promenade is also uplifted to address a
requirements conflict.)
Change-Id: I067a5edbbdff06f45394e772ebb16d5012987b6c
* Fixed typo for jsonschema version
* Setuptools version forced to <58 to allow usage of use_2to3, which
some dependencies require
* Fixed scripting error that allowed Airflow installation to silently
fail
Change-Id: I237801488795df07f2d4de1cde6cd4ec41182e31
* Install older version of pip<21.0 for ubuntu_xenial images
* Install setuptools via pip for ubuntu_xenial images
* Pin typing-extensions to 3.7.2 and apache-airflow to 1.10.5
* Move promenade Dependencies under UCP components in requirements.txt
* With apache-airflow=1.10.5, strip ANSI escape sequences
in test_deployment_group_manager.py
* Update tox.ini to support apache-airflow=1.10.5
* airskiff gate fixes
- Pin treasuremap to v1.9 branch
- Pin openstack-helm-infra to master
- Remove openstack client setup as it's not used
Change-Id: Iee4ce59fdceacb165120a69d11c44e6e47feaea8
When pip is upgraded to 20.3, the pip dependency resolver is much more
strict and will no longer install a combination of packages that is mutually
inconsistent[0].
These changes account for the fact that Shipyard imports Armada, Drydock,
Promenade, and Deckhand. Having said that, with pip 20.3, the pip
packages amongst those projects cannot conflict. A follow-up change may
be needed if more conflicts are found.
Change-Id: Idd9ea0d57b5be063b133036cfc9ebaa69956f4fc
Updates Dockerfiles to build the LibYAML library, which can provide much
faster YAML parsing and emitting than the native Python library.
https://pyyaml.org/wiki/LibYAML
Change-Id: I6ebafa1eb2af8841b86b64fd119dcb581e6c18a7
airflow quicktest is throwing an error when trying to start the
webserver:
can't open file '_cmd': [Errno 2] No such file or directory
Change-Id: I3fddc1cdd3daeacc09a5924287d711a72878c5e3
1. Locked the WTForms to 2.2.1 to address the import issue with
wtforms.widgets.HTMLString. WTForms 2.3.0 was released on
April 21/2020. This release causes shipyard gate fail with
import error for wtforms.widgets.HTMLString.
2. Deleted psycopg2==2.7.7, which is installed as a dependency of
apache-airflow extra package postgres, and resoted the newer
release psycopg2-binary==2.8.4, to be used instead.
Change-Id: I303a2c94ec409e97af1192ae892b8148fcdbb8d5
- With bionic image based shipyard docker images, uwsgi crashes
with segmentation fault, when it tries to load the psycopg2 library,
causing the api become unreachable on both shipyard docker images.
This happens because psycopg2 2.7.x and uwsgi binary wheels are built
with incompatible ssl libraries. This patch upgrades psycopg2 to the
latest release to address this issue.
- The existing image build script cannot run in a docker or a pod,
based pipeline because of two reasons:
- The build script runs a docker (docker-in-docker) and mounts a
volume.
In a dind case, volume bind mounts will not work, because the nested
container will need the host file system's path for the source path.
- The shipyard service listens to its exposed service port in the
nested docker network namespace, which is not reachable from the host
pod/container.
This patch address both of the above issues. It first creates the
container, copies needed config files to the container and then starts
it. Also it execs into the nested docker to access the shipyard services
in a dind (docker-in-dcoker) case.
Change-Id: Ifdfed539babab01608bfaef37001bb79cd3a080d
Added support to buid shipyard and airflow images using either a xenial
or Ubuntu bionic base image.
The default base image is set to bionic.
Change-Id: I6ad4d42dede081586b3ccea87a42e250979ac106
Updated airflow config and Dockerfiles for apache airflow 1.10.4
For details see: https://github.com/apache/airflow/blob/master/UPDATING.md
Also updated the kubectl to 1.16.2, as part of kubernetes 1.16 uplift.
Change-Id: Ib24ff0304b6279ff0be749141854d6a604473597
Change URLs from git.openstack.org & github.com to opendev.org due to the
migration; wrap multiple LABELs into one.
Change-Id: I240fa6f746bd1e424e5a2b7fd381903c46059ae5
Updated deckhand commit id, because apache airflow release 1.10.3 and
later use flask >=1.1.0, which needs Werkzeug library version >=0.15.0.
see: https://issues.apache.org/jira/browse/AIRFLOW-4900
The updated deckhand commit lets the Werkzeug package float to >=0.15.0.
Change-Id: I62d7f4e5eecb2f05035a1c9552544aa65e70ee3f
- Allow a Docker build-arg to specify the source of the Airflow
python package in any format supported by pip
Change-Id: Ifa2dd62d91570035cff91ff07868b0fcf659add8
The current logrotate logic deletes logs that are
more than X days old in the Airflow log path, however
the Airflow log archive may still reach 100%
usage and cause the airflow-worker to crashloop.
This PS adds logic to logrotate.sh to delete the oldest
logs and empty dirs when the Airflow log archive
reaches the max usage specified in values.yaml.
Change-Id: I3dcb80901d7dd36da6812850a1f54e7ebf3b1cf2
Adding DISTRO parameter for makefile to invoke distribution specific
Dockerfile and build image accordingly.
Also qualifying image job name to distinguish ubuntu vs opensuse
check jobs and passing distro specific variable.
Similar logic for pushing distro specific images.
Added new document section for multiple distro support.
Change-Id: I215f8a107cab8770181c0472e0cc29053b2b978e
A recent Shipyard change [0] introduced user-context tracing support in
Shipyard, which requires the passing of an additional parameter to the
Deckhand and Drydock API clients. The Deckhand [1] and Drydock [2]
clients were updated to handle the additional parameter, but the
versions of those components used by Shipyard was not updated. This
change uplifts the versions of Deckhand and Drydock in requirements.txt
for the Airflow component of Shipyard to a version that supports the
aforementioned, additional, user-context parameter.
[0] https://review.openstack.org/633873
[1] https://review.openstack.org/634068
[2] https://review.openstack.org/638115
Change-Id: I096191f436c5123b6c263fe92c0d69b9e4085c66
The current script is fragile to the introduction of a DEBUG logging
level. This change addresses the specifics of logging records being
returned when invoking the airflow command enough to make the dag_status
result end up being the last result as the script desires, so the script
is returned to working order, even if the logging level is set to DEBUG.
This change goes a little further to the point of removing all records
that indicate a logging level of DEBUG|INFO|WARN|ERROR.
Change-Id: Ic72358097e3a476e20fa3713008d0d0bcd35463e
The airflow image has a requirements file that is separate from the one
located in src/bin/shipyard_airflow, which was missed in a previous
uplift patch [0]. This change updates all versions in the image
requirements file to match the source code requirements.
[0] https://review.openstack.org/652736
Change-Id: I238bba4038e5cd8364711c2c5d1f78f51fc76948
Updates the dependencies on Armada, Drydock, and Deckhand to current
values. The primary reason for this is to take advantage of a new set of
Armada requirements that allows for a reduced image size (no longer
includes grpc tools)
Change-Id: I8f208ab91664479a7af96c543a4e72d8f6d154ef
Change to use ubuntu base image instead of python
Refactor Shipyard Dockerfile to reduce image size significantly
BREAKING CHANGE: The `make images` PYTHON_BASE_IMAGE arg is now renamed to BASE_IMAGE.
Change-Id: I3338dfbbb91b5514fa4fd205bdfc4136d0abc2e5
Changes to make the docker image build to include the workflows from
Shipyard, rather than adding them to the container during Helm install
of Shipyard. This also removes the "prod" switch, as it is now always
built the same way, with the workflows in place.
Change-Id: I4acd6195cbec32193e15621e75ccaeb9879455f5
Updates to Airflow 1.10.1; See (1), (2) for some notes
Related, and additionally: configures Airflow to restore logging of
workflow steps to a console/sdtout logger, supporting the desired
ability to attach logging and monitoring to standard container
mechanisms. This does not change the behavior of also logging to the
airflow-arranged log files for steps and DAG runs.
A side effect of updating to 1.10.1 includes a major decrease in
resource usage by the Airflow scheudler process (reducing from ~ 1 core
fully consumed to less than 5% of a core consumed YMMV, but significant)
Additional adjustment downward of resources allocated, threads produced,
and frequency of polling leads to an overall significant reduction in
resource usage.
Airship note:
Because Airflow 1.10.0 and 1.10.1 use compatible versions of celery and
dag_run information, updating from 1.10.0 - 1.10.1 in place is possible
if airflow-worker pods are allowed to continue to run.
(1) https://github.com/apache/incubator-airflow/blob/master/UPDATING.md
(2) https://github.com/apache/incubator-airflow/releases/tag/1.10.1
Change-Id: I9b024e3996c528c7b74e2888191d48c7a45a1f04
This PS enables shipyard to use PBR as rest
of the other OpenStack project.
Currently if shipyard is used as a requirement in another
project, the dependencies for shipyard doesn't get installed.
This PS fixes this bug.
Change-Id: I34d452649a886af9d865d4ad51eeab70399fe395
Now that Deckhand supports rolling back to revision 0 with
https://review.openstack.org/#/c/592725/ it is preferrable to
exercise this functionality instead of deleting all revisions
in Deckhand's database, so that previous revision history
isn't lost.
Deckhand is fast-forwarded to
0b5aa2e98a1ab5ab8a58c9dec3c1f88ef00d17a9 which contains
the commit required for SY to be able to make use of this
new change in Deckhand's client, as well as to align with
Promenade's Deckhand fast-forward of Deckhand in
https://review.openstack.org/#/c/604459/
Depends-On: https://review.openstack.org/#/c/592725/
Change-Id: Iebbfeb5e6c74b3e2f09b2137ec14482cc8fa4f44
Updates the image building to Airflow 1.10, including necessary
configuration changes and a general update of dependencies.
Airflow 1.10 includes many enhancements and bugfixes since 1.9 [0]
This change introduces many "unused" configuration parameters to satsify
Airflow's expectations[1]. An ugly, but likely harmless change to the
log output with interleaved newline characters from Airflow steps[2].
Changes to the chart and other dependendencies have also been introduced
to match this update.
[0] https://github.com/apache/incubator-airflow/blob/master/CHANGELOG.txt
[1] https://issues.apache.org/jira/browse/AIRFLOW-3099
[2] https://issues.apache.org/jira/browse/AIRFLOW-1917
Change-Id: I179dcf1f0369650b8c4519f704abb7fb495f4248
1) UCP -> Airship
2) readthedocs.org -> readthedocs.io (there is redirect)
3) http -> https
4) attcomdev -> airshipit (repo on quay.io)
5) att-comdev -> openstack/airship-* (repo on github/openstack git)
6) many URLs have been verified and adjusted to be current
7) no need for 'en/latest/' path in URL of the RTD
8) added more info to some setup.cfg and setup.py files
9) ucp-integration docs are now in airship-in-a-bottle
10) various other minor fixes
Change-Id: I4b8cc6ddf491e35d600a83f5f82d7717108e31dd
1) Use OCI Image Specs for labels instead of custom 'commit-id=xxxxx'
or legacy "Label Schema"
2) Fix missing git commit id labels on images (.revision)
3) Add human-readable title (.title) of the image, URL (.url), and
a few other properties (annotations) according to the latest Specs
Change-Id: Iedd64fa70016880d5df241d27d8dad101e10130b
