124 lines
4.1 KiB
Markdown
124 lines
4.1 KiB
Markdown
tomograph
|
|
=========
|
|
|
|
A library to help distributed applications send trace information to
|
|
metrics backends like [Zipkin][zipkin] and [Statsd][statsd].
|
|
|
|
Data Model
|
|
----------
|
|
|
|
A request to a distributed application is modeled as a trace. Each
|
|
trace consists of a set of spans, and a span is a set of notes.
|
|
|
|
Each span's extent is defined by its first and last notes. Any number
|
|
of additional notes can be added in between -- for example in a
|
|
handler for ERROR-level logging.
|
|
|
|
The tomograph data model is basically the Dapper/Zipkin data model.
|
|
For translation to statsd, we emit the length of the span as a timer
|
|
metric, and each note gets emitted individually as a counter metric.
|
|
|
|
For example, here is a basic client/server interaction. It is one
|
|
trace, with two spans, each with two notes -- their beginning and end:
|
|
|
|
|
|
|
|
This is the same data as it would be viewed in using the statsd
|
|
backend with graphite:
|
|
|
|
|
|
|
|
|
|
Tracing Your Application
|
|
------------------------
|
|
|
|
There are a few basic ways to add tracing to your application. The
|
|
lowest level one is to call start, stop, and annotate yourself:
|
|
|
|
import tomograph
|
|
|
|
tomograph.start('my service', 'a query', '127.0.0.1', 80)
|
|
(...)
|
|
tomograph.annotate('something happened')
|
|
tomograph.tag('key', 'value')
|
|
(...)
|
|
tomograph.stop('a query')
|
|
|
|
Each start/stop pair defines a span. Spans can be arbitrarily nested
|
|
using this interface as long they stay on a single thread: tomograph
|
|
keeps the current span stack in thread local storage.
|
|
|
|
When continuing a trace from one thread to another, you must grab the
|
|
trace token from tomograph and pass it:
|
|
|
|
token = tomograph.get_trace_info()
|
|
(...)
|
|
tomograph.start('my service', 'a query', '127.0.0.1', 80, token)
|
|
(...)
|
|
|
|
That will enable tomograph to connect all of the spans into one trace.
|
|
|
|
Helpers
|
|
-------
|
|
|
|
There are some slightly higher level interfaces to help you add
|
|
tracing. For HTTP, add_trace_info_header() will add an X-Trace-Info
|
|
header to a dict on the client side, and start_http() will consume
|
|
that header on the server side:
|
|
|
|
def traced_http_client(url, body, headers):
|
|
tomograph.start('client', 'http request', socket.gethostname(), 0)
|
|
tomograph.add_trace_info_header(headers)
|
|
http_request(url, body, headers)
|
|
tomograph.stop('http request')
|
|
|
|
|
|
def traced_http_server(request):
|
|
tomograph.start_http('server', 'http response', request)
|
|
(...)
|
|
tomograph.stop('http response')
|
|
|
|
There's no need to call start and stop yourself -- you can use the
|
|
@tomograph.traced decorator:
|
|
|
|
@tomograph.traced('My Server', 'myfunc')
|
|
def myfunc(yadda):
|
|
dosomething()
|
|
|
|
For WSGI pipelines, there's the class tomograph.Middleware that will
|
|
consume the X-Trace-Info header. It can be added to a paste pipeline
|
|
like so:
|
|
|
|
[pipeline:foo]
|
|
pipeline = tomo foo bar baz...
|
|
|
|
[filter:tomo]
|
|
paste.filter_factory = tomograph:Middleware.factory
|
|
service_name = glance-registry
|
|
|
|
If you use [SQL Alchemy][sql alchemy] in your application, there are
|
|
some event listeners available that will trace SQL statement
|
|
execution:
|
|
|
|
_ENGINE = sqlalchemy.create_engine(FLAGS.sql_connection, **engine_args)
|
|
|
|
sqlalchemy.event.listen(_ENGINE, 'before_execute', tomograph.before_execute('my app'))
|
|
sqlalchemy.event.listen(_ENGINE, 'after_execute', tomograph.after_execute('my app'))
|
|
sqlalchemy.event.listen(_ENGINE, 'dbapi_error', tomograph.dbapi_error('my app'))
|
|
|
|
|
|
Screenshots
|
|
-----------
|
|
|
|
Here is a slightly more involved example -- a glance image list
|
|
command in [Openstack][openstack]. It uses SQL statement tracing and
|
|
the tomograph middleware:
|
|
|
|
|
|
|
|
|
|
[openstack]: http://www.openstack.org/
|
|
[statsd]: https://github.com/etsy/statsd
|
|
[zipkin]: http://twitter.github.com/zipkin/
|
|
[sql alchemy]: http://www.sqlalchemy.org/
|