123bdfbb16
This commit adds the schema info for the subunit gearman workers to the firehose docs. Change-Id: Id98b1f7e735cee8a9932776520884b19c25649a7
152 lines
4.9 KiB
ReStructuredText
152 lines
4.9 KiB
ReStructuredText
:title: Firehose Schema
|
|
|
|
.. _firehose_schema:
|
|
|
|
Firehose Schema
|
|
###############
|
|
|
|
This attempts to document the topic and payload schema for all the services
|
|
reporting to the firehose. However since much of what is reported to firehose
|
|
is dynamically generated it is possible this document misses a case.
|
|
|
|
Gerrit
|
|
======
|
|
|
|
Messages on firehose for gerrit are generated using the `germqtt`_ project. For
|
|
the most part these are basically identical to what gerrit returns on it's
|
|
native event stream except over MQTT.
|
|
|
|
.. _germqtt: http://git.openstack.org/cgit/openstack-infra/germqtt/
|
|
|
|
Topics
|
|
------
|
|
|
|
The topics for gerrit are generated dynamically. However, they follow a fairly
|
|
straightforward pattern. The basic formula for this is::
|
|
|
|
gerrit/<git namespace>/<repo name>/<gerrit event>
|
|
|
|
So for example a typical topic would be::
|
|
|
|
gerrit/openstack/nova/comment-added
|
|
|
|
The ``git namespace`` and ``repo name`` are pretty self explanatory and are just
|
|
from the git repository the change in gerrit is for. The event is defined in the gerrit event stream. You can see the full reference for topics in the Gerrit
|
|
docs for `Gerrit events`_. However, for simplicity the possible values are:
|
|
|
|
* change-abandoned
|
|
* change-merged
|
|
* change-restored
|
|
* comment-added
|
|
* draft-published
|
|
* hashtags-changed
|
|
* merge-failed
|
|
* patchset-created
|
|
* ref-updated
|
|
* reviewer-added
|
|
* topic-changed
|
|
|
|
Payload
|
|
-------
|
|
The payload for gerrit messages are basically the same JSON that gets returned
|
|
by gerrit's event stream command. Instead of repeating the entire gerrit schema
|
|
doc here just refer to gerrit's docs on the `JSON payload`_ which documents the
|
|
contents of each JSON object and refer to the doc on `Gerrit events`_ for which
|
|
JSON objects are included with which event type.
|
|
|
|
.. _JSON payload: https://review.openstack.org/Documentation/json.html
|
|
.. _Gerrit events: https://review.openstack.org/Documentation/cmd-stream-events.html#events
|
|
|
|
Launchpad
|
|
=========
|
|
The messages sent to firehose for launchpad are generated using `lpmqtt`_
|
|
|
|
.. _lpmqtt: http://git.openstack.org/cgit/openstack-infra/lpmqtt/
|
|
|
|
Topics
|
|
------
|
|
|
|
The topics for lpmqtt follow a pretty simple formula::
|
|
|
|
launchpad/<project>/<event type>/<bug number>
|
|
|
|
the ``project`` is the launchpad project name, ``event type`` will always be
|
|
"bug" (or not present). The intent of this was to be "bug" or "blueprint", but
|
|
due to limitations in launchpad getting notifications from blueprints is not
|
|
possible. The flexibility was left in the schema just in case this ever changes.
|
|
The ``bug number`` is obviously the bug number from launchpad.
|
|
|
|
It's also worth noting that only the base topic is a guaranteed field. Depending
|
|
on the notification email from launchpad some of the other fields may not be
|
|
present. In those cases the topic will be populated left to right until a
|
|
missing field is encountered.
|
|
|
|
Payload
|
|
-------
|
|
|
|
The payload of messages is dynamically generated and dependent on the
|
|
notification recieved from launchpad, and launchpad isn't always consistent in
|
|
what fields are present in those notifications.
|
|
|
|
However, for bug event types there is a standard format. The fields which
|
|
are always present for bugs (which should normally be the only message for
|
|
firehose) are:
|
|
|
|
* commenters
|
|
* bug-reporter
|
|
* bug-modifier
|
|
* bug-number
|
|
* event-type
|
|
|
|
The remaining fields are dynamic and depend on launchpad. An example message
|
|
payload (with the body trimmed) for a bug is::
|
|
|
|
{
|
|
"status": "Triaged",
|
|
"project": "octavia",
|
|
"assignee": "email@fakedomain.com",
|
|
"bug-reporter": "Full name (username)",
|
|
"event-type": "bug",
|
|
"bug-number": "1680938",
|
|
"commenters": ["username"]
|
|
"tags": ["rfe"],
|
|
"importance": "Medium",
|
|
"bug-modifier": "Full Name (username)",
|
|
"body": "notification body, often is just bug comment or summary",
|
|
}
|
|
|
|
Subunit Workers
|
|
===============
|
|
|
|
The messages for the subunit workers are generated directly in the
|
|
`subunit gearman worker scripts`_.
|
|
|
|
.. _subunit gearman worker scripts: http://git.openstack.org/cgit/openstack-infra/puppet-subunit2sql/tree/files/subunit-gearman-worker.py
|
|
|
|
Topics
|
|
------
|
|
|
|
The topics for the subunit workers follow a simple pattern::
|
|
|
|
gearman-subunit/<worker hostname>/<git namespace/<repo name>/<change number>
|
|
|
|
Where ``worker hostname`` is the host which processed the subunit file, as
|
|
of right now there are 2, subunit-worker01 and subunit-worker02, but there may
|
|
be more (or fewer) in the future. The ``git namespace`` and ``repo name`` are
|
|
pretty self explanatory, and are just for the git repo under test that the
|
|
subunit was emitted from. ``change number`` is the gerrit change number for the
|
|
job that launched the tests the subunit is for.
|
|
|
|
Payload
|
|
-------
|
|
The payload for the messages from the subunit workers is pretty straightforward
|
|
json that contains 3 fields: ``status``, ``build_uuid``, and ``source_url``.
|
|
|
|
An example is::
|
|
|
|
{
|
|
'status': 'success',
|
|
'build_uuid': '45f7c1ddbfd74c6aba94662623bd61b8'
|
|
'source_url': 'A url',
|
|
}
|