diff --git a/doc/source/base.rst b/doc/source/base.rst
new file mode 100644
index 0000000..dc32985
--- /dev/null
+++ b/doc/source/base.rst
@@ -0,0 +1,6 @@
+Base Jobs
+=========
+
+.. zuul:autojob:: base
+
+.. zuul:autojob:: base-test
diff --git a/doc/source/conf.py b/doc/source/conf.py
index e9958ba..9346347 100755
--- a/doc/source/conf.py
+++ b/doc/source/conf.py
@@ -19,6 +19,8 @@
extensions = [
# 'sphinx.ext.autodoc',
# 'sphinx.ext.intersphinx',
+ 'sphinxcontrib.blockdiag',
+ 'sphinxcontrib.seqdiag',
'zuul_sphinx',
]
diff --git a/doc/source/docker-image.rst b/doc/source/docker-image.rst
new file mode 100644
index 0000000..88ebbe0
--- /dev/null
+++ b/doc/source/docker-image.rst
@@ -0,0 +1,165 @@
+Container Images
+================
+
+The jobs described in this section all work together to handle the
+full gating process for continuously deployed container images. They
+can be used to build or test images which rely on other images using
+the full power of Zuul's speculative execution.
+
+There are a few key concepts to keep in mind:
+
+A *buildset* is a group of jobs all running on the same change.
+
+A *buildset registry* is a container image registry which is used to
+store speculatively built images for the use of jobs in a single
+buildset. It holds the differences between the current state of the
+world and the future state if the change in question (and all of its
+dependent changes) were to merge. It must be started by one of the
+jobs in a buildset, and it ceases to exist once that job is complete.
+
+An *intermediate registry* is a long-running registry that is used to
+store images created for unmerged changes for use by other unmerged
+changes. It is not publicly accessible and is intended only to be
+used by Zuul in order to transfer artifacts from one buildset to
+another. OpenDev maintains such a registry.
+
+With these concepts in mind, the jobs described below implement the
+following workflow for a single change:
+
+.. _buildset_image_transfer:
+
+.. seqdiag::
+ :caption: Buildset registry image transfer
+
+ seqdiag image_transfer {
+ Ireg [label="Intermediate\nRegistry"];
+ Breg [label="Buildset\nRegistry"];
+ Bjob [label="Image Build Job"];
+ Djob [label="Deployment Test Job"];
+
+ Ireg -> Breg [label='Images from previous changes'];
+ Breg -> Bjob [label='Images from previous changes'];
+ Breg <- Bjob [label='Current image'];
+ Ireg <- Breg [noactivate, label='Current image'];
+ Breg -> Djob [label='Current and previous images'];
+ Breg <- Djob [style=none];
+ Ireg <- Breg [style=none];
+ }
+
+The intermediate registry is always running and the buildset registry
+is started by a job running on a change. The "Image Build" and
+"Deployment Test" jobs are example jobs which might be running on a
+change. Essentially, these are image producer or consumer jobs
+respectively.
+
+There are two ways to use the jobs described below:
+
+A Repository with Producers and Consumers
+-----------------------------------------
+
+The first is in a repository where images are both produced and
+consumed. In this case, we can expect that there will be at least one
+image build job, and at least one job which uses that image (for
+example, by performing a test deployment of the image). In this case
+we need to construct a job graph with dependencies as follows:
+
+.. blockdiag::
+
+ blockdiag dependencies {
+ obr [label='opendev-\nbuildset-registry'];
+ bi [label='build-image'];
+ ti [label='test-image'];
+
+ obr <- bi <- ti;
+ }
+
+The :zuul:job:`opendev-buildset-registry` job will run first and
+automatically start a buildset registry populated with images built
+from any changes which appear ahead of the current change. It will
+then return its connection information to Zuul and pause and continue
+running until the completion of the build and test jobs.
+
+The build-image job should inherit from
+:zuul:job:`opendev-build-docker-image`, which will ensure that it is
+automatically configured to use the buildset registry.
+
+The test-image job is something that you will create yourself. There
+is no standard way to test or deploy an image, that depends on your
+application. However, there is one thing you will need to do in your
+job to take advantage of the buildset registry. In a pre-run playbook,
+use the `use-buildset-registry
+<https://zuul-ci.org/docs/zuul-jobs/roles.html#role-use-buildset-registry>`_
+role:
+
+.. code:: yaml
+
+ - hosts: all
+ roles:
+ - use-buildset-registry
+
+That will configure the docker daemon on the host to use the buildset
+registry so that it will use the newly built version of any required
+images.
+
+A Repository with Only Producers
+--------------------------------
+
+The second way to use these jobs is in a repository where an image is
+merely built, but not deployed. In this case, there are no consumers
+of the buildset registry other than the image build job, and so the
+registry can be run on the job itself. In this case, you may omit the
+:zuul:job:`opendev-buildset-registry` job and run only the
+:zuul:job:`opendev-build-docker-image` job.
+
+Publishing an Image
+-------------------
+
+So far we've covered the image building process. This system also
+provides two more jobs that are used in publishing images to Docker
+Hub.
+
+The :zuul:job:`opendev-upload-docker-image` job does everything the
+:zuul:job:`opendev-build-docker-image` job does, but it also uploads
+the built image to Docker Hub using an automatically-generated and
+temporary tag. The "build" job is designed to be used in the
+*check* pipeline, while the "upload" job is designed to take its
+place in the *gate* pipeline. By front-loading the upload to Docker
+Hub, we reduce the chance that a credential or network error will
+prevent us from publishing an image after a change lands.
+
+The :zuul:job:`opendev-promote-docker-image` jobs is designed to be
+used in the *promote* pipeline and simply re-tags the image on Docker
+Hub after the change lands.
+
+Keeping in mind that everything described above in
+:ref:`buildset_image_transfer` applies to the
+:zuul:job:`opendev-upload-docker-image` job, the following illustrates
+the additional tasks performed by the "upload" and "promote" jobs:
+
+.. seqdiag::
+
+ seqdiag image_transfer {
+ DH [activated, label="Docker Hub"];
+ Ujob [label="upload-image"];
+ Pjob [label="promote-image"];
+
+ DH -> Ujob [style=none];
+ DH <- Ujob [label='Current image with temporary tag'];
+ DH -> Pjob [label='Current image manifest with temporary tag',
+ note='Only the manifest
+ is transferred,
+ not the actual
+ image layers.'];
+ DH <- Pjob [label='Current image manifest with final tag'];
+ }
+
+Jobs
+----
+
+.. zuul:autojob:: opendev-buildset-registry
+
+.. zuul:autojob:: opendev-build-docker-image
+
+.. zuul:autojob:: opendev-upload-docker-image
+
+.. zuul:autojob:: opendev-promote-docker-image
diff --git a/doc/source/index.rst b/doc/source/index.rst
index 282aa08..1aeeb1c 100644
--- a/doc/source/index.rst
+++ b/doc/source/index.rst
@@ -1,7 +1,7 @@
.. include:: ../../README.rst
.. toctree::
- :maxdepth: 1
+ :maxdepth: 2
jobs
roles
diff --git a/doc/source/jobs.rst b/doc/source/jobs.rst
index 1d722cb..247cb99 100644
--- a/doc/source/jobs.rst
+++ b/doc/source/jobs.rst
@@ -1,5 +1,10 @@
Jobs
-=====
+====
-.. zuul:autojobs::
+Documentation for jobs is grouped by subject area:
+.. toctree::
+ :maxdepth: 1
+
+ base
+ docker-image
diff --git a/test-requirements.txt b/test-requirements.txt
index 7300381..c6ca123 100644
--- a/test-requirements.txt
+++ b/test-requirements.txt
@@ -17,3 +17,5 @@ ansible-lint<4
bashate>=0.2
zuul-sphinx>=0.1.1
stestr>=1.0.0 # Apache-2.0
+sphinxcontrib-blockdiag>=1.1.0
+sphinxcontrib-seqdiag
diff --git a/zuul.yaml b/zuul.yaml
index 08d1926..7f4597c 100644
--- a/zuul.yaml
+++ b/zuul.yaml
@@ -164,9 +164,16 @@
- job:
name: opendev-buildset-registry
description: |
- Inherit from this job to get a buildset registry which interacts
- with the intermediate CI registry to share speculative container
- images between projects.
+ Starts a buildset registry which interacts with the intermediate
+ CI registry to share speculative container images between
+ projects.
+
+ Configure any jobs which require the use of a buildset registry
+ to depend on this job using the "dependencies" job attribute.
+
+ This job will pause after starting the registry so that it is
+ available to any jobs which depend on it. Once all such jobs
+ are complete, this job will finish.
pre-run: playbooks/buildset-registry/pre.yaml
run: playbooks/buildset-registry/run.yaml
post-run: playbooks/buildset-registry/post.yaml
@@ -179,7 +186,10 @@
name: opendev-build-docker-image
parent: opendev-buildset-registry
description: |
- Starts a buildset registry and builds one or more docker images.
+ Starts a buildset registry (if one has not already been started,
+ e.g., by invoking :zuul:job:`opendev-buildset-registry` and
+ specifying it as a dependency) and builds one or more docker
+ images.
Analog of build-docker-image job, but with a buildset registry.