Reparent the bootstrap-bridge job onto a job that sets up git repos

A while back there was a big refactor where initial infra-prod setup was
updated to differentiate between parent jobs that configure bridge
within zuul then update source repos on bridge and those that only
configure bridge within zuul. The idea being we could have a single job
update git on bridge then allow many ansible playbooks for service setup
to run concurrently.

As part of this refactor the infra-prod-bootstrap-bridge got parented
to the only set up bridge in the executor job which is a problem because
bootstrap-bridge configures known_hosts on bridge which requires up to
date git repos. Correct this mistake by reparenting to the job that does
both things.

This results in a rough job dependency heirarchy that looks like this:

  infra-prod-bootstrap-bridge
    ^
    |
  infra-prod-base
    ^
    |
  infra-prod-letsencrypt | infra-prod-service-that-doesnt-le
    ^
    |
  infra-prod-service-that-does-le

Eventually we should be able to convert everything but
infra-prod-bootstrap-bridge to use the executor setup only variant of
the parent job. That would optimize the amount of git repo setup we are
doing.

To ensure the switch in this change is safe and avoids multiple jobs
attempting to update git repos at the same time we update dependencies
for the zuul-db, zuul-lb, and gitea-lb jobs as they weren't depending on
infra-prod-base previously. They probably should depend on
infra-prod-base anyway as that ensures users, packages, and firewalls
are up to date for example.

Finally, for extra belts and suspenders to avoid multiple simultaneous
synchronizations of the system-config repo we apply the
infra-prod-playbook semaphore to the bootstrap-bridge job. Eventually
this should get cleaned up if we start allowing concurrent ansible runs.

Change-Id: If18c109ed177b34efde00b1097ce0a1d7a4906e2

zuul.d/infra-prod.yaml

@@ -33,19 +33,27 @@
 
 - job:
     name: infra-prod-bootstrap-bridge
-    parent: opendev-infra-prod-setup-keys
+    parent: opendev-infra-prod-setup-src
     description: |
         Configure the bastion host (bridge)
 
         This job does minimal configuration on the bastion host
         (bridge.openstack.org) to allow it to run system-config
         playbooks against our production hosts.  It sets up Ansible
-        and root keys on the host.
+        and root keys on the host. It also synchronizes the system-config
+        repo from the executor to the bastion. This is necessary to
+        emit an up to date known_hosts file when adding new hosts to
+        the inventory.
 
         Note that this is separate to infra-prod-service-bridge;
         bridge in it's role as the bastion host actaully runs that
         against itself; it includes things not strictly needed to make
         the host able to deploy system-config.
+    # While we don't run the infra-prod-playbook in this job we do run
+    # system-config git repo updates. Until we're ready to stop running
+    # system-config updates in every job we use this semaphore to ensure
+    # exclusivity.
+    semaphores: infra-prod-playbook
     run: playbooks/zuul/run-production-bootstrap-bridge.yaml
     files:
       - playbooks/bootstrap-bridge.yaml

zuul.d/project.yaml

@@ -399,6 +399,27 @@
               - name: infra-prod-base
                 soft: true
 
+        #
+        # Only depends on base, or image promotion.
+        #
+        - infra-prod-service-gitea-lb: &infra-prod-service-gitea-lb
+            dependencies:
+              - name: infra-prod-base
+                soft: true
+              - name: system-config-promote-image-haproxy-statsd
+                soft: true
+
+        - infra-prod-service-zuul-db: &infra-prod-service-zuul-db
+            dependencies:
+              - name: infra-prod-base
+                soft: true
+        - infra-prod-service-zuul-lb: &infra-prod-service-zuul-lb
+            dependencies:
+              - name: infra-prod-base
+                soft: true
+              - name: system-config-promote-image-haproxy-statsd
+                soft: true
+
         #
         # Hosts using certificates and backups
         #
@@ -455,10 +476,6 @@
                 soft: true
               - name: system-config-promote-image-gitea
                 soft: true
-        - infra-prod-service-gitea-lb: &infra-prod-service-gitea-lb
-            dependencies:
-              - name: system-config-promote-image-haproxy-statsd
-                soft: true
         - infra-prod-service-grafana: &infra-prod-service-grafana
             dependencies:
               - name: infra-prod-letsencrypt
@@ -540,11 +557,6 @@
               # should reconfigure after any project updates
               - name: infra-prod-manage-projects
                 soft: true
-        - infra-prod-service-zuul-db
-        - infra-prod-service-zuul-lb: &infra-prod-service-zuul-lb
-            dependencies:
-              - name: system-config-promote-image-haproxy-statsd
-                soft: true
         - infra-prod-service-zuul-preview: &infra-prod-service-zuul-preview
             dependencies:
               - name: infra-prod-letsencrypt
@@ -630,13 +642,15 @@
         - infra-prod-service-afs: *infra-prod-service-afs
         - infra-prod-service-nameserver: *infra-prod-service-nameserver
         - infra-prod-service-mirror-update: *infra-prod-service-mirror-update
+        - infra-prod-service-gitea-lb: *infra-prod-service-gitea-lb
+        - infra-prod-service-zuul-db: *infra-prod-service-zuul-db
+        - infra-prod-service-zuul-lb: *infra-prod-service-zuul-lb
         - infra-prod-service-borg-backup: *infra-prod-service-borg-backup
         - infra-prod-letsencrypt: *infra-prod-letsencrypt
         - infra-prod-service-codesearch: *infra-prod-service-codesearch
         - infra-prod-service-eavesdrop: *infra-prod-service-eavesdrop
         - infra-prod-service-etherpad: *infra-prod-service-etherpad
         - infra-prod-service-gitea: *infra-prod-service-gitea
-        - infra-prod-service-gitea-lb: *infra-prod-service-gitea-lb
         - infra-prod-service-grafana: *infra-prod-service-grafana
         - infra-prod-service-graphite: *infra-prod-service-graphite
         - infra-prod-service-keycloak: *infra-prod-service-keycloak
@@ -652,8 +666,6 @@
         - infra-prod-service-tracing: *infra-prod-service-tracing
         - infra-prod-service-zookeeper: *infra-prod-service-zookeeper
         - infra-prod-service-zuul: *infra-prod-service-zuul
-        - infra-prod-service-zuul-db
-        - infra-prod-service-zuul-lb: *infra-prod-service-zuul-lb
         - infra-prod-service-zuul-preview: *infra-prod-service-zuul-preview
         - infra-prod-run-accessbot: *infra-prod-run-accessbot
         - infra-prod-manage-projects: *infra-prod-manage-projects