How StackTach.v3 is deployed at Rackspace

For Rackspace Public Cloud, OpenStack is deployed in many different regions and each region is comprised of many cells.

In this deployment scenario, we have a set of "worker" boxes that run the yagi-event workers (to consume notifications from RabbitMQ). We run one yagi-event worker per OpenStack cell. We also run the Winchester pipeline-workers on this box. The pipeline-worker manages pipeline processing for ready streams. We run as many pipeline-workers as needed to keep each core busy. Both the yagi-event and pipeline-worker processes are single threaded so we spawn more to keep the host busy. We add more servers like these to handle the load in the pipeline processing and/or the source queues.

Additionally, we have servers for the API nodes and the MySQL database. Load balancing the API servers and making MySQL highly available is left to the deployer.

The Ansible deployment scripts

In the StackTach Sandbox repo, there is an ansible directory that contains roles for the workers, the API nodes and the database. There is a fourth "common" role that is used by the other roles.

These scripts assume a multi-cell deployment where there is one StackTach.v3 deployment per region (each with it's own API servers and database). If you are not running with cells, pretend it's a single cell.

The Build script

The StackTach Sandbox repo is the umbrella repo for all the other StackTach modules. It:

• can create a local dev environment for anyone wanting to contribute to StackTach or just play around with it
• can package StackTach into a python virtual environment and tarball it up for larger scale deployment. The ansible scripts use this tarball approach.
• git clones all the dependent python modules so you can develop locally and see your changes immediately and easily.

The sandbox readme will get you started, but the gory details are in the build.sh script. There are essentially three options:

• ./build.sh - run the local StackTach.v3 dev environment in a screen session.
• ./build.sh -t - run the tests on all the StackTach.v3 repos.
• ./build.sh -p - install StackTach.v3 into the virtual env and package it up into a tarball for deployment.
• ./build.sh -p -d - package and deploy the tarball. Currently this just lists the ansible incantation required to do the deployment.

The screencasts cover using the sandbox for local dev in more detail.