system-config/zuul.d/infra-prod.yaml
Ian Wienand d616ec9d9a Bootstrap-bridge as top-level job
The idea here is

 * all prod jobs are parented to the boostrap-bridge job (they have a
   hard dependency on this job).

 * the bootstrap-bridge job checks out the system-config source to the
   right place (the commit for a change, master HEAD for periodic). This
   was actually implemented in a prior change. We're just taking full
   advantage of it here.

 * bootstrap-bridge pauses once bridge is setup to the right place

 * the child jobs now don't have to worry about cloning system-config;
   they can be sure that it's at the right place for them.  they just
   need keys so their executor can log into bridge and run the
   playbooks against the production hosts

 * the bootstrap-bridge job is paused with a semaphore stopping any
   other runs jumping in.  in deployment, zuul is ordering it for us
   anyway.  so really this is stopping conflicts with the periodic
   jobs.

 * in theory - all the child production jobs could run in parallel
   while the boostrap jobs waits for them (modulo dependencies they
   have expressed; e.g. needing letsencyrpt or backup jobs to have
   run). To begin with we limit this with a second semaphore with a
   limit of 1. We can roll this out and check things mostly operate
   as they did before then bump the max value on this semaphore upwards
   to run things in parallel.

 * does this work?  I have no idea :) it seems difficult to test
   outside production because in the testing side everything is its
   own little world, there's no overarching bootstrap job.

Depends-On: https://review.opendev.org/c/opendev/base-jobs/+/942740
Change-Id: I7d2c4737f900c9b964855c4d03ca58a2de2d60b3
2025-02-25 14:17:14 -08:00

704 lines
22 KiB
YAML

# NOTE: job dependencies keep this running in parallel and are defined
# in projects.yaml because it's easier to keep an overall view of
# what's happening in there.
# Make sure only one run happens at a time. The deploy pipeline
# should keep things in order, but this is to stop perodic jobs
# jumping in.
- semaphore:
name: infra-prod-deployment
max: 1
# This semaphore limits the total number of production playbook
# jobs that can run on bridge at one time. We want things to run in
# parallel but we have a lot of jobs (particularly in the periodic
# pipeline) that we don't want to run all at once.
- semaphore:
name: infra-prod-playbook-limit
# TODO(clarkb) this semaphore allows us to stage the rollout of
# parallel infra-prod job exceution in two steps. First we reorganize
# everything but roughly keep the same behaviors as before (max: 1).
# When we are happy with that we can bump this to 2 or higher and see
# things run in parallel.
max: 1
- job:
name: infra-prod-bootstrap-bridge
parent: opendev-infra-prod-setup-src
semaphores: infra-prod-deployment
description: |
Configure the bastion host (bridge)
This job does minimal configuration on the bastion host
(bridge.openstack.org) to allow it to run system-config
playbooks against our production hosts. It sets up Ansible
and root keys on the host. It also synchronizes the
system-config repo from the executor to the bastion.
Note that this is separate to infra-prod-service-bridge;
bridge in it's role as the bastion host actaully runs that
against itself; it includes things not strictly needed to make
the host able to deploy system-config.
This job is the parent of all deployment jobs, and will pause
until they finish. This prevents conflicts between deployment
jobs from changes and periodic runs (which use HEAD of
master).
run: playbooks/zuul/run-production-bootstrap-bridge.yaml
# Do not set file matchers on this job. We must always run this job
# before any other infra-prod jobs to ensure system-config is up to
# date on bridge before we run our playbooks.
nodeset:
nodes: []
- job:
name: infra-prod-playbook
parent: opendev-infra-prod-setup-keys
semaphores: infra-prod-playbook-limit
description: |
Run specified playbook against productions hosts.
This is a parent job designed to be inherited to enabled
CD deployment of our infrastructure. Set playbook_name to
specify the playbook relative to
/home/zuul/src/opendev.org/opendev/system-config/playbooks
on the bastion host.
abstract: true
run: playbooks/zuul/run-production-playbook.yaml
post-run: playbooks/zuul/run-production-playbook-post.yaml
required-projects:
- opendev/system-config
vars:
infra_prod_ansible_forks: 10
infra_prod_playbook_collect_log: false
infra_prod_playbook_encrypt_log: true
nodeset:
nodes: []
dependencies:
- name: infra-prod-bootstrap-bridge
# This is a hard dependency because we require the bootstrap job to
# have run before we start any playbook jobs, otherwise our buildset
# would not hold the bridge semaphore and we may not have the correct
# system-config state on bridge.
- job:
name: infra-prod-base
parent: infra-prod-playbook
description: Run the base playbook everywhere.
vars:
playbook_name: base.yaml
infra_prod_ansible_forks: 50
files:
- inventory/
- inventory/service/host_vars/
- inventory/service/group_vars/
- playbooks/base.yaml
- playbooks/roles/base/
- job:
name: infra-prod-letsencrypt
parent: infra-prod-playbook
description: Run letsencrypt.yaml playbook.
vars:
playbook_name: letsencrypt.yaml
files:
- inventory/
- playbooks/letsencrypt.yaml
# Any touching of host_vars or group_vars can substantively
# change the certs we're doing, so be greedy here.
- inventory/service/host_vars/
- inventory/service/group_vars/
- playbooks/roles/letsencrypt
- playbooks/roles/logrotate/
- job:
name: infra-prod-manage-projects
parent: infra-prod-playbook
timeout: 4800
description: |
Create and update projects in gerrit and gitea.
allowed-projects:
- opendev/system-config
- openstack/project-config
required-projects:
- opendev/system-config
- openstack/project-config
vars:
playbook_name: manage-projects.yaml
infra_prod_ansible_forks: 10
infra_prod_playbook_collect_log: true
- job:
name: infra-prod-service-base
parent: infra-prod-playbook
description: Base job for most service playbooks.
abstract: true
- job:
name: infra-prod-service-bridge
parent: infra-prod-service-base
description: Run service-bridge.yaml playbook.
vars:
playbook_name: service-bridge.yaml
files:
- inventory/base
- playbooks/service-bridge.yaml
- inventory/service/group_vars/bastion.yaml
- playbooks/roles/logrotate/
- playbooks/roles/edit-secrets-script/
- playbooks/roles/install-kubectl/
- playbooks/roles/iptables/
- playbooks/roles/configure-kubectl/
- playbooks/roles/configure-openstacksdk/
- playbooks/templates/clouds/bridge_all_clouds.yaml.j2
- job:
name: infra-prod-service-gitea-lb
parent: infra-prod-service-base
description: Run service-gitea-lb.yaml playbook.
vars:
playbook_name: service-gitea-lb.yaml
files:
- inventory/base
- playbooks/service-gitea-lb.yaml
- inventory/service/group_vars/gitea-lb.yaml
- playbooks/roles/pip3/
- playbooks/roles/iptables/
- playbooks/roles/install-docker/
- playbooks/roles/haproxy/
- job:
name: infra-prod-service-nameserver
parent: infra-prod-service-base
description: Run service-nameserver.yaml playbook.
vars:
playbook_name: service-nameserver.yaml
files:
- inventory/base
- playbooks/service-nameserver.yaml
- inventory/service/group_vars/adns.yaml
- inventory/service/group_vars/adns-primary.yaml
- inventory/service/group_vars/adns-secondary.yaml
- playbooks/roles/master-nameserver/
- playbooks/roles/nameserver/
- playbooks/roles/iptables/
- job:
name: infra-prod-service-nodepool
parent: infra-prod-service-base
description: Run service-nodepool.yaml playbook.
vars:
playbook_name: service-nodepool.yaml
required-projects:
- opendev/system-config
- openstack/project-config
files:
- inventory/base
- playbooks/service-nodepool.yaml
- inventory/service/host_vars/nb
- inventory/service/host_vars/nl
- inventory/service/group_vars/nodepool
- playbooks/roles/configure-kubectl/
- playbooks/roles/configure-openstacksdk/
- playbooks/roles/install-docker/
- playbooks/roles/iptables/
- playbooks/roles/nodepool
- playbooks/templates/clouds/nodepool_
- job:
name: infra-prod-service-etherpad
parent: infra-prod-service-base
description: Run service-etherpad.yaml playbook.
vars:
playbook_name: service-etherpad.yaml
files:
- inventory/base
- playbooks/service-etherpad.yaml
- inventory/service/group_vars/etherpad.yaml
- playbooks/roles/install-docker/
- playbooks/roles/pip3/
- playbooks/roles/etherpad
- playbooks/roles/logrotate
- playbooks/roles/iptables/
- docker/etherpad/
- job:
name: infra-prod-service-keycloak
parent: infra-prod-service-base
description: Run service-keycloak.yaml playbook.
vars:
playbook_name: service-keycloak.yaml
files:
- inventory/base
- playbooks/service-keycloak.yaml
- inventory/service/group_vars/keycloak.yaml
- playbooks/roles/keycloak/
- playbooks/roles/install-docker/
- playbooks/roles/iptables/
- job:
name: infra-prod-service-meetpad
parent: infra-prod-service-base
description: Run service-meetpad.yaml playbook.
vars:
playbook_name: service-meetpad.yaml
files:
- inventory/base
- playbooks/service-meetpad.yaml
- inventory/service/host_vars/meetpad01.opendev.org.yaml
- inventory/service/group_vars/meetpad.yaml
- playbooks/roles/pip3/
- playbooks/roles/install-docker/
- playbooks/roles/iptables/
- playbooks/roles/jitsi-meet/
- job:
name: infra-prod-service-mirror-update
parent: infra-prod-service-base
description: Run service-mirror-update.yaml playbook.
vars:
playbook_name: service-mirror-update.yaml
files:
- inventory/base
- inventory/service/group_vars/mirror.yaml
- inventory/service/host_vars/mirror
- playbooks/service-mirror-update.yaml
- playbooks/roles/mirror-update/
- playbooks/roles/reprepro/
- playbooks/roles/iptables/
- playbooks/roles/logrotate/
- roles/kerberos-client/
- roles/openafs-client/
- job:
name: infra-prod-service-mirror
parent: infra-prod-service-base
description: Run service-mirror.yaml playbook.
vars:
playbook_name: service-mirror.yaml
files:
- inventory/base
- playbooks/service-mirror.yaml
- inventory/service/host_vars/mirror
- inventory/service/group_vars/mirror.yaml
- playbooks/roles/mirror/
- playbooks/roles/afs-release/
- playbooks/roles/afsmon/
- playbooks/roles/iptables/
- playbooks/roles/logrotate/
- roles/openafs-client/
- job:
name: infra-prod-service-paste
parent: infra-prod-service-base
description: Run service-paste.yaml playbook.
vars:
playbook_name: service-paste.yaml
files:
- inventory/base
- playbooks/service-paste.yaml
- inventory/service/group_vars/paste.yaml
- playbooks/roles/install-docker/
- playbooks/roles/pip3/
- playbooks/roles/lodgeit/
- playbooks/roles/iptables/
- job:
name: infra-prod-service-static
parent: infra-prod-service-base
description: Run service-static.yaml playbook.
vars:
playbook_name: service-static.yaml
files:
- inventory/base
- playbooks/service-static.yaml
- inventory/service/group_vars/static.yaml
- playbooks/roles/apache-ua-filter/
- playbooks/roles/iptables/
- playbooks/roles/static/
- playbooks/roles/zuul-user/
- roles/openafs-client/
- job:
name: infra-prod-service-tracing
parent: infra-prod-service-base
description: Run service-tracing.yaml playbook.
vars:
playbook_name: service-tracing.yaml
files:
- inventory/base
- playbooks/service-tracing.yaml
- inventory/service/group_vars/tracing.yaml
- playbooks/roles/jaeger/
- playbooks/roles/install-docker/
- playbooks/roles/iptables/
- job:
name: infra-prod-service-borg-backup
parent: infra-prod-service-base
description: Run service-borg-backup.yaml playbook.
vars:
playbook_name: service-borg-backup.yaml
files:
- inventory/base
- inventory/service/groups.yaml
- inventory/service/host_vars/backup02.ca-ymq-1.vexxhost.opendev.org.yaml
- inventory/service/host_vars/backup01.ord.rax.opendev.org.yaml
- inventory/service/group_vars/borg-backup.yaml
- inventory/service/group_vars/borg-backup-server.yaml
- playbooks/service-borg-backup.yaml
- playbooks/roles/install-borg/
- playbooks/roles/borg-backup/
- playbooks/roles/borg-backup-server/
- playbooks/roles/iptables/
- job:
name: infra-prod-service-registry
parent: infra-prod-service-base
description: Run service-registry.yaml playbook.
vars:
playbook_name: service-registry.yaml
files:
- inventory/base
- playbooks/service-registry.yaml
- inventory/service/group_vars/registry.yaml
- playbooks/roles/pip3/
- playbooks/roles/install-docker/
- playbooks/roles/iptables/
- playbooks/roles/registry/
- job:
name: infra-prod-service-zuul-preview
parent: infra-prod-service-base
description: Run service-zuul-preview.yaml playbook.
vars:
playbook_name: service-zuul-preview.yaml
files:
- inventory/base
- playbooks/service-zuul-preview.yaml
- inventory/service/group_vars/zuul-preview.yaml
- playbooks/roles/pip3/
- playbooks/roles/install-docker/
- playbooks/roles/iptables/
- playbooks/roles/zuul-preview/
- job:
name: infra-prod-service-zookeeper
parent: infra-prod-service-base
description: Run service-zookeeper.yaml playbook.
vars:
playbook_name: service-zookeeper.yaml
files:
- inventory/base
- inventory/service/group_vars/zookeeper.yaml
- ^inventory/service/host_vars/zk\d+\..*
- playbooks/roles/pip3/
- playbooks/roles/install-docker/
- playbooks/roles/iptables/
- playbooks/roles/zookeeper/
- job:
name: infra-prod-service-zuul
parent: infra-prod-service-base
description: |
Run service-zuul.yaml playbook.
This configures the main Zuul cluster. It will perform a
smart-reconfigure of the scheduler if the tenant configuration
is changed.
vars:
playbook_name: service-zuul.yaml
files:
- inventory/base
- playbooks/service-zuul.yaml
- inventory/service/group_vars/zuul
- inventory/service/group_vars/zookeeper.yaml
- inventory/service/host_vars/zk\d+
- inventory/service/host_vars/zuul\d+.opendev.org
- playbooks/roles/install-docker/
- playbooks/roles/iptables/
- playbooks/roles/zookeeper/
- playbooks/roles/zuul
- roles/kerberos-client/
- roles/openafs-client/
- job:
name: infra-prod-service-zuul-db
parent: infra-prod-service-base
description: Run service-zuul-db.yaml playbook.
vars:
playbook_name: service-zuul-db.yaml
files:
- inventory/base
- playbooks/service-zuul-db.yaml
- inventory/service/group_vars/zuul-db.yaml
- playbooks/roles/iptables/
- playbooks/roles/install-docker/
- playbooks/roles/mariadb/
- job:
name: infra-prod-service-zuul-lb
parent: infra-prod-service-base
description: Run service-zuul-lb.yaml playbook.
vars:
playbook_name: service-zuul-lb.yaml
files:
- inventory/base
- playbooks/service-zuul-lb.yaml
- inventory/service/group_vars/zuul-lb.yaml
- playbooks/roles/pip3/
- playbooks/roles/iptables/
- playbooks/roles/install-docker/
- playbooks/roles/haproxy/
- job:
name: infra-prod-service-review
parent: infra-prod-service-base
description: Run service-review.yaml playbook.
vars:
playbook_name: service-review.yaml
files:
- inventory/base
- playbooks/service-review.yaml
- inventory/service/group_vars/review.yaml
- inventory/service/host_vars/review02.opendev.org.yaml
- playbooks/roles/pip3/
- playbooks/roles/install-docker/
- playbooks/roles/iptables/
- playbooks/roles/gerrit/
- zuul.d/docker-images/gerrit.yaml
- job:
name: infra-prod-service-refstack
parent: infra-prod-service-base
description: Run service-refstack.yaml playbook.
vars:
playbook_name: service-refstack.yaml
files:
- inventory/base
- playbooks/service-refstack.yaml
- inventory/service/group_vars/refstack.yaml
- inventory/service/host_vars/refstack[0-9][0-9]
- playbooks/roles/install-docker/
- playbooks/roles/pip3/
- playbooks/roles/refstack/
- playbooks/roles/iptables/
- playbooks/roles/logrotate/
- docker/refstack
- docker/python-base/
- job:
name: infra-prod-service-gitea
parent: infra-prod-service-base
description: Run service-gitea.yaml playbook.
vars:
playbook_name: service-gitea.yaml
files:
- inventory/base
- playbooks/service-gitea.yaml
- inventory/service/group_vars/gitea.yaml
- inventory/service/host_vars/gitea[0-9][0-9]
- playbooks/roles/apache-ua-filter/
- playbooks/roles/install-docker/
- playbooks/roles/pip3/
- playbooks/roles/gitea/
- playbooks/roles/iptables/
- playbooks/roles/logrotate/
- docker/gitea/
- docker/gitea-init/
- docker/jinja-init/
- docker/python-base/
- job:
name: infra-prod-service-eavesdrop
parent: infra-prod-service-base
description: Run service-eavesdrop.yaml playbook.
required-projects:
- opendev/system-config
- openstack/project-config
vars:
playbook_name: service-eavesdrop.yaml
files: &infra_prod_eavesdrop_files
- inventory/base
- playbooks/service-eavesdrop.yaml
- playbooks/run-accessbot.yaml
- inventory/service/group_vars/eavesdrop.yaml
- playbooks/roles/install-docker
- playbooks/roles/iptables/
- playbooks/roles/accessbot
- playbooks/roles/limnoria
- playbooks/roles/ptgbot
- playbooks/roles/statusbot
- playbooks/roles/logrotate
- playbooks/roles/matrix-eavesdrop
- playbooks/roles/matrix-gerritbot
- playbooks/zuul/templates/group_vars/eavesdrop.yaml.j2
- docker/accessbot/
- docker/ircbot
- docker/matrix-eavesdrop
- job:
name: infra-prod-run-accessbot
parent: infra-prod-service-base
description: Run run-accessbot.yaml playbook.
required-projects:
- opendev/system-config
- openstack/project-config
vars:
playbook_name: run-accessbot.yaml
files:
- accessbot/channels.yaml
- playbooks/run-accessbot.yaml
- playbooks/roles/accessbot
- docker/accessbot/
- job:
name: infra-prod-service-codesearch
parent: infra-prod-service-base
description: Run service-codesearch.yaml playbook.
vars:
playbook_name: service-codesearch.yaml
files:
- docker/hound/
- inventory/base
- playbooks/service-codesearch.yaml
- inventory/service/host_vars/codesearch02.opendev.yaml
- inventory/service/group_vars/codesearch
- playbooks/roles/install-docker/
- playbooks/roles/pip3/
- playbooks/roles/codesearch
- playbooks/roles/logrotate
- playbooks/roles/iptables
- job:
name: infra-prod-service-grafana
parent: infra-prod-service-base
description: Run service-grafana.yaml playbook.
vars:
playbook_name: service-grafana.yaml
files:
- inventory/base
- playbooks/service-grafana.yaml
- inventory/service/host_vars/grafana02.org.yaml
- inventory/service/group_vars/grafana
- playbooks/roles/install-docker/
- playbooks/roles/pip3/
- playbooks/roles/grafana
- playbooks/roles/logrotate
- playbooks/roles/iptables/
- job:
name: infra-prod-service-graphite
parent: infra-prod-service-base
description: Run service-graphite.yaml playbook.
vars:
playbook_name: service-graphite.yaml
files:
- inventory/base
- playbooks/service-graphite.yaml
- inventory/service/host_vars/graphite02.opendev.org.yaml
- inventory/service/group_vars/graphite
- playbooks/roles/install-docker/
- playbooks/roles/pip3/
- playbooks/roles/graphite/
- playbooks/roles/iptables/
- job:
name: infra-prod-service-lists3
parent: infra-prod-service-base
description: Run service-lists3.yaml playbook.
vars:
playbook_name: service-lists3.yaml
files:
- docker/mailman
- inventory/base
- inventory/service/group_vars/mailman3.yaml
- playbooks/roles/iptables/
- playbooks/roles/base/exim
- playbooks/roles/mailman3/
- playbooks/service-lists3.yaml
# Run AFS changes separately so we can make sure to only do one at a time
# (turns out quorum is nice to have)
- job:
name: infra-prod-service-afs
parent: infra-prod-service-base
description: Run AFS playbook.
vars:
playbook_name: service-afs.yaml
infra_prod_ansible_forks: 1
required-projects:
- opendev/system-config
files:
- inventory/base
- playbooks/service-afs.yaml
- inventory/service/group_vars/afs
- inventory/service/group_vars/mirror-update
- playbooks/roles/iptables/
- playbooks/roles/vos-release/
- playbooks/roles/openafs-server/
- modules/
- manifests/
- roles/kerberos-client/
- roles/openafs-client/
- job:
name: infra-prod-service-kerberos
parent: infra-prod-service-base
description: Run Kerberos playbook.
vars:
playbook_name: service-kerberos.yaml
infra_prod_ansible_forks: 1
required-projects:
- opendev/system-config
files:
- inventory/base
- playbooks/service-kerberos.yaml
- inventory/service/group_vars/kerberos-kdc.yaml
- playbooks/roles/kerberos-kdc/
- roles/kerberos-client/
- playbooks/roles/iptables/
- job:
name: infra-prod-remote-puppet-else
parent: infra-prod-service-base
description: Run remote-puppet-else.yaml playbook.
vars:
playbook_name: remote_puppet_else.yaml
infra_prod_ansible_forks: 50
required-projects:
- opendev/ansible-role-puppet
- opendev/system-config
files:
- Gemfile
- Rakefile
- modules.env
- install_modules.sh
- hiera/
- inventory/
- roles/puppet-install/
- playbooks/install_puppet.yaml
- playbooks/update_puppet_version.yaml
- playbooks/remote_puppet_else.yaml
- playbooks/roles/puppet-run/
- playbooks/roles/install-ansible-roles/
- playbooks/roles/disable-puppet-agent/
- playbooks/roles/puppet-setup-ansible/
- playbooks/roles/iptables/
- modules/
- manifests/
- job:
name: infra-prod-run-cloud-launcher
parent: infra-prod-service-base
description: Run cloud launcher playbook
vars:
playbook_name: run_cloud_launcher.yaml
infra_prod_ansible_forks: 1
required-projects:
- opendev/ansible-role-cloud-launcher
- opendev/system-config
files:
- playbooks/run_cloud_launcher.yaml
- inventory/service/group_vars/bastion.yaml