From c7b7cf7876059d3d07b28a8fc550e4d0748287d4 Mon Sep 17 00:00:00 2001 From: Levi Blackstone Date: Fri, 3 Apr 2015 09:37:31 -0500 Subject: [PATCH] Correct a bunch of typos in docs and add contributor link Change-Id: I18769378a3c14ca0771da9c1fbd5d3fb81959e3b --- docs/about.html | 20 ++++++++++---------- docs/contribute.html | 2 +- docs/install.html | 8 ++++---- 3 files changed, 15 insertions(+), 15 deletions(-) diff --git a/docs/about.html b/docs/about.html index 08aa864..8b3cd7b 100644 --- a/docs/about.html +++ b/docs/about.html @@ -151,7 +151,7 @@ durable = True max_messages = 100 -

The important part of this configuration is the [event_worker] section. This says we want to use the RabbitMQ data source. The RabbitMQ connectivity information is stored in the [rabbit_broker] section. The name of each rabbitmq queue to consume from is specified in the [consumers] section. For every queue you define there, you will need a [consumer:<queue_name>] section. This last section is where there real magic happens. Beyond defining the exchange, routing_key and durability characteristics, it defines the chain of Yagi Handlers that will run on every notification that gets consumed.

+

The important part of this configuration is the [event_worker] section. This says we want to use the RabbitMQ data source. The RabbitMQ connectivity information is stored in the [rabbit_broker] section. The name of each RabbitMQ queue to consume from is specified in the [consumers] section. For every queue you define there, you will need a [consumer:<queue_name>] section. This last section is where there real magic happens. Beyond defining the exchange, routing_key and durability characteristics, it defines the chain of Yagi Handlers that will run on every notification that gets consumed.

You can write your own Yagi handlers if you like, but there are a number that ship with StackTach.v3 to do some interesting things. The most important of these is the winchester.yagi_handler:WinchesterHandler. This handler is your entry point into StackTach.v3 stream processing. But first, we need to convert those messy notifications into events ...

Distilling Notifications to Events

Now we have notifications coming into Winchester. But, as we hinted at above, we need to take the larger notification and distill it down into a, more manageable, event. The stack-distiller module makes this happen. Within StackTach.v3, this is part of winchester.yagi_handler:WinchesterHandler.

@@ -269,7 +269,7 @@ max_messages = 100 Boolean support - Coersed to Integer + Coerced to Integer Millisecond resolution datetime support @@ -301,7 +301,7 @@ pipeline_handlers: notabene: winchester.pipeline_handler:NotabeneHandler -

The first thing you'll notice is the database connection string. But then you'll notice that the Winchester module needs three other configuration files. The distiller config file we've already covered. The other two require a little more explaination. They define your Triggers and your Pipelines.

+

The first thing you'll notice is the database connection string. But then you'll notice that the Winchester module needs three other configuration files. The distiller config file we've already covered. The other two require a little more explanation. They define your Triggers and your Pipelines.

@@ -398,7 +398,7 @@ my_expire_pipeline: -

During pipeline processing each handler is called to process the events in the stream. A handler has three methods: handle_events(), commit() and rollback(). The handle_events() method is called for each handler in the order they're defined. If they all succeed, the commit() method of each handler is called. Otherwise, the rollback() method of each handler is called. No work should be performed in the handle_events() method. The data should be pre-computed and stored, but not actioned until in the commit() method. In the case of errors, the handle_event() method could be called many times. So, to ensure at-most-once functionality, non-reversable operations should be reserved for the commit() call. Things like, publishing new notifications, emitting metrics, sending emails, etc. should be done in commit(). rollback() is a last chance for you to unwind any work you may have performed.

+

During pipeline processing each handler is called to process the events in the stream. A handler has three methods: handle_events(), commit() and rollback(). The handle_events() method is called for each handler in the order they're defined. If they all succeed, the commit() method of each handler is called. Otherwise, the rollback() method of each handler is called. No work should be performed in the handle_events() method. The data should be pre-computed and stored, but not actioned until in the commit() method. In the case of errors, the handle_event() method could be called many times. So, to ensure at-most-once functionality, non-reversible operations should be reserved for the commit() call. Things like, publishing new notifications, emitting metrics, sending emails, etc. should be done in commit(). rollback() is a last chance for you to unwind any work you may have performed.

Stream debugging

@@ -515,7 +515,7 @@ winchester.debugging[INFO line: 161] ---------------------------- most popular OpenStack operations.

The UsageHandler

The UsageHandler is a pipeline handler for determining the daily usage of every instance with an OpenStack Nova deployment. The usage handler is cells-aware so it can support large deployments.

-

The useage handler requires a stream per instance per day. It triggers when the compute.instance.exists event is seen. Audit notifications should be enabled within Nova. See the samples for an example of a usage stream definition.

+

The usage handler requires a stream per instance per day. It triggers when the compute.instance.exists event is seen. Audit notifications should be enabled within Nova. See the samples for an example of a usage stream definition.

Once triggered, the usage handler will compare the daily transactional events for every instance against the various .exists records for that instance. If nothing happens to an instance within that 24-hour period, an end-of-day .exists notification is sent from Nova. Nova operations that change the launched_at date for an instance will issue additional .exists records. These include create, delete, resize and rebuild operations. If the transactional events for the instance match the values in the .exists event, a compute.instance.exists.verified notification is created, otherwise a compute.instance.exists.failed and/or compute.instance.exists.warnings notifications are created. When coupled with the NotabeneHandler, these new notifications can be republished to the queue for subsequent processing.

The schema of these new notifications are as follows:

@@ -532,7 +532,7 @@ winchester.debugging[INFO line: 161] ---------------------------- 'audit_period_beginning': start datetime of audit period 'audit_period_ending': ending datetime of audit period 'launched_at': datetime this instance was launched - 'deleted_at': datatime this instance was deleted + 'deleted_at': datetime this instance was deleted 'instance_id': instance uuid 'tenant_id': tenant id 'display_name': instance display name @@ -564,13 +564,13 @@ winchester.debugging[INFO line: 161] ---------------------------- 'timestamp': datetime this notification was generated at source 'stream_id': stream id 'original_message_id': message_id of .exists event - 'error': human readable explaination for verification failure + 'error': human readable explanation for verification failure 'error_code': numeric error code (see below) 'payload': { 'audit_period_beginning': start datetime of audit period 'audit_period_ending': ending datetime of audit period 'launched_at': datetime this instance was launched - 'deleted_at': datatime this instance was deleted + 'deleted_at': datetime this instance was deleted 'instance_id': instance uuid 'tenant_id': tenant id 'display_name': instance display name @@ -647,8 +647,8 @@ winchester.debugging[INFO line: 161] ----------------------------

The NotabeneHandler

-

The NotabeneHandler will take any new notifications (not events) it finds in the pipeline Environment variable and publish them to the rabbitmq exchange specified. The handler will look ofor a key/value in the pipeline environment (passed into the handler on the handle_events() call).

-

In your pipeline definition, you can set the configuration for the NotabeneHandler as shown below. Note how the enviroment variable keys are defined by the env_keys value. This can be a list of keys. Any new notifications this handler finds in those variables will get published to the RabbitMQ exchange specified in the rest of the configuration. The queue_name is also critical so we know which topic to publish to. In OpenStack, the routing key is the queue name. The notabene handler does connection pooling to the various queues, so specifying many different servers is not expensive.

+

The NotabeneHandler will take any new notifications (not events) it finds in the pipeline Environment variable and publish them to the RabbitMQ exchange specified. The handler will look for a key/value in the pipeline environment (passed into the handler on the handle_events() call).

+

In your pipeline definition, you can set the configuration for the NotabeneHandler as shown below. Note how the environment variable keys are defined by the env_keys value. This can be a list of keys. Any new notifications this handler finds in those variables will get published to the RabbitMQ exchange specified in the rest of the configuration. The queue_name is also critical so we know which topic to publish to. In OpenStack, the routing key is the queue name. The notabene handler does connection pooling to the various queues, so specifying many different servers is not expensive.

Because these environment keys have to be set before the notabene handler is called, it has to be one of the last handlers in the pipeline. The UsageHandler adds new notifications to the usage_notifications key. If the notabene handler is not part of the pipeline, these new notifications are dropped when the pipeline is finished.

diff --git a/docs/contribute.html b/docs/contribute.html index 52e4e3f..8ced293 100644 --- a/docs/contribute.html +++ b/docs/contribute.html @@ -60,7 +60,7 @@

StackTach.v3 is licensed under the Apache 2.0 license

All the source repos for StackTach.v3 (and .v2) are available on StackForge. Details on contributing to StackForge projects are available here

The core developers are available on Freenode IRC in the #stacktach channel

-

These docs are available in the Sandbox repo. Patches welcome!

+

These docs are available in the Sandbox repo. Patches welcome!