x/deb-os-testr
Code Issues Proposed changes

Improved docs for os-testr commands

Browse Source 
This commit adds real docs for using the 3 currently packaged commands
in the os-testr project. As part of this change it also updates the
sphinx config to generate proper man pages for all the tooling. At a
future stage we need to ensure that these get installed properly.

Change-Id: I0055c7961203a094590f6cfeb136a2236a2f65cc
This commit is contained in:
Matthew Treinish 2015-07-09 21:22:25 -04:00
parent 16330c0526
commit 2f29ca51be
No known key found for this signature in database
GPG Key ID: FD12A0F214C9E177
6 changed files with 388 additions and 8 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
README.rst
View File

@ -21,6 +21,14 @@ Features
Release Notes
=============
0.2.0
-----
* Adds support for comments in a blacklist file and printing comments for each
exclude
* Several bugfixes for subunit-trace
* Switched subunit-trace behavior to disable printing percent change
in run time by default and make it optional
0.1.0
-----
* First release which includes: ostestr, subunit-trace, and subunit2html

15
doc/source/conf.py
View File

@ -22,7 +22,7 @@ sys.path.insert(0, os.path.abspath('../..'))
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
extensions = [
'sphinx.ext.autodoc',
#'sphinx.ext.intersphinx',
# 'sphinx.ext.intersphinx',
'oslosphinx'
]
@ -38,7 +38,7 @@ master_doc = 'index'
# General information about the project.
project = u'os-testr'
copyright = u'2013, OpenStack Foundation'
copyright = u'2015, Matthew Treinish'
# If true, '()' will be appended to :func: etc. cross-reference text.
add_function_parentheses = True
@ -68,8 +68,15 @@ latex_documents = [
('index',
'%s.tex' % project,
u'%s Documentation' % project,
u'OpenStack Foundation', 'manual'),
u'Matthew Treinish', 'manual'),
]
man_pages = [('ostestr', 'ostestr', 'tooling to run OpenStack tests',
['Matthew Treinish'], 1),
('subunit_trace', 'subunit-trace', 'pretty output filter for '
'subunit streams', ['Matthew Treinish'], 1),
('subunit2html', 'subunit2html', 'generate a html results page '
'from a subunit stream', ['Matthew Treinish'], 1)]
# Example configuration for intersphinx: refer to the Python standard library.
#intersphinx_mapping = {'http://docs.python.org/': None}
# intersphinx_mapping = {'http://docs.python.org/': None}

217
doc/source/ostestr.rst Normal file
View File

@ -0,0 +1,217 @@
.. _ostestr:
ostestr
=======
The ostestr command provides a wrapper around the testr command included in
the testrepository package. It's designed to build on the functionality
included in testr and workaround several UI bugs in the short term. By default
it also has output that is much more useful for OpenStack's test suites which
are lengthy in both runtime and number of tests. Please note that the CLI
semantics are still a work in progress as the project is quite young, so
default behavior might change in future version.
Summary
-------
ostestr [-b|--blacklist_file <blacklist_file>] [-r|--regex REGEX]
[-p|--pretty] [--no-pretty] [-s|--subunit] [-l|--list]
[-n|--no-discover <test_id>] [--slowest] [--no-slowest]
[--pdb <test_id>] [--parallel] [--serial]
[-c|--concurrency <workers>] [--until-failure] [--print-exclude]
Options
-------
--blacklist_file BLACKLIST_FILE, -b BLACKLIST_FILE
Path to a blacklist file, this file contains a
separate regex exclude on each newline
--regex REGEX, -r REGEX
A normal testr selection regex. If a blacklist file is
specified, the regex will be appended to the end of
the generated regex from that file
--pretty, -p
Print pretty output from subunit-trace. This is
mutually exclusive with --subunit
--no-pretty
Disable the pretty output with subunit-trace
--subunit, -s
output the raw subunit v2 from the test run this is
mutuall exclusive with --pretty
--list, -l
List all the tests which will be run.
--no-discover TEST_ID, -n TEST_ID
Takes in a single test to bypasses test discover and
just excute the test specified
--slowest
After the test run print the slowest tests
--no-slowest
After the test run don't print the slowest tests
--pdb TEST_ID
Run a single test that has pdb traces added
--parallel
Run tests in parallel (this is the default)
--serial
Run tests serially
--concurrency WORKERS, -c WORKERS
The number of workers to use when running in parallel.
By default this is the number of cpus
--until-failure
Run the tests in a loop until a failure is
encountered. Running with subunit or prettyoutput
enable will force the loop to run testsserially
--print-exclude
If an exclude file is used this option will prints the
comment from the same line and all skipped tests
before the test run
Running Tests
-------------
os-testr is primarily for running tests at it's basic level you just invoke
ostestr to run a test suite for a project. (assuming it's setup to run tests
using testr already) For example::
$ ostestr
This will run tests in parallel (with the number of workers matching the number
of CPUs) and with subunit-trace output. If you need to run tests in serial you
can use the serial option::
$ ostestr --serial
Or if you need to adjust the concurrency but still run in parallel you can use
-c/--concurrency::
$ ostestr --concurrency 2
If you only want to run an individual test module or more specific (a single
class, or test) and parallel execution doesn't matter, you can use the
-n/--no-discover to skip test discovery and just directly calls subunit.run on
the tests under the covers. Bypassing discovery is desirable when running a
small subset of tests in a larger test suite because the discovery time can
often far exceed the total run time of the tests.
For example::
$ ostestr --no-discover test.test_thing.TestThing.test_thing_method
Additionally, if you need to run a single test module, class, or single test
with pdb enabled you can use --pdb to directly call testtools.run under the
covers which works with pdb. For example::
$ ostestr --pdb tests.test_thing.TestThing.test_thing_method
Test Selection
--------------
ostestr is designed to build on top of the test selection in testr. testr only
exposed a regex option to select tests. This equivalent is functionality is
exposed via the --regex option. For example::
$ ostestr --regex 'magic\.regex'
This will do a straight passthrough of the provided regex to testr.
Additionally, ostestr allows you to specify a a blacklist file to define a set
of regexes to exclude. You can specify a blacklist file with the
--blacklist-file/-b option, for example::
$ ostestr --blacklist_file $path_to_file
The format for the file is line separated regex, with '#' used to signify the
start of a comment on a line. For example::
# Blacklist File
^regex1 # Excludes these tests
.*regex2 # exclude those tests
Will generate a regex to pass to testr which will exclude both any tests
matching '^regex1' and '.*regex2'. If a blacklist file is used in conjunction
with the --regex option the regex specified with --regex will be appended to
the generated output from the --blacklist_file. Also it's worth noting that the
regex test selection options can not be used in conjunction with the
--no-discover or --pdb options described in the previous section. This is
because the regex selection requires using testr under the covers to actually
do the filtering, and those 2 options do not use testr.
It's also worth noting that you can use the test list option to dry run any
selection arguments you are using. You just need to use --list/-l with your
selection options to do this, for example::
$ ostestr --regex 'regex3.*' --blacklist_file blacklist.txt --list
This will list all the tests which will be run by ostestr using that combination
of arguments.
Please not that all of this selection functionality will be expanded on in the
future and a default grammar for selecting multiple tests will be chosen in a
future release. However as of right now all current arguments (which have
guarantees on always remaining in place) are still required to perform any
selection logic while this functionality is still under development.
Output Options
--------------
By default ostestr will use subunit-trace as the output filter on the test
run. It will also print the slowest tests from the run after the run is
concluded. You can disable the printing the slowest tests with the --no-slowest
flag, for example::
$ ostestr --no-slowest
If you'd like to disable the subunit-trace output you can do this using
--no-pretty::
$ ostestr --no-pretty
ostestr also provides the option to just output the raw subunit stream on
STDOUT with --subunit/-s. Note if you want to use this you also have to
specify --no-pretty as the subunit-trace output and the raw subunit output
are mutually exclusive. For example, to get raw subunit output the arguments
would be::
$ ostestr --no-pretty --subunit
An additional option on top of the blacklist file is --print-exclude option.
When this option is specified when using a blacklist file before the tests are
run ostestr will print all the tests it will be excluding from the blacklist
file. If a line in the blacklist file has a comment that will be printed before
listing the tests which will be excluded by that line's regex. If no comment is
present on a line the regex from that line will be used instead. For example,
if you were using the example blacklist file from the previous section the
output before the regular test run output would be::
$ ostestr -b blacklist-file blacklist.txt --print-exclude
Excludes these tests
regex1_match
regex1_exclude
exclude those tests
regex2_match
regex2_exclude
...
Notes for running with tox
--------------------------
If you use `tox`_ for running your tests and call ostestr as the test command
.. _tox: https://tox.readthedocs.org/en/latest/
it's recommended that you set a posargs following ostestr on the commands
stanza. For example::
[testenv]
commands = ostestr {posargs}
this will enable end users to pass args to configure the output, use the
selection logic, or any other options directly from the tox cli. This will let
tox take care of the venv management and the environment separation but enable
direct access to all of the ostestr options to easily customize your test run.
For example, assuming the above posargs usage you would be to do::
$ tox -epy34 -- --regex ^regex1
or to skip discovery::
$ tox -epy34 -- -n test.test_thing.TestThing.test_thing_method

33
doc/source/subunit2html.rst Normal file
View File

@ -0,0 +1,33 @@
.. _subunit2html:
subunit2html
============
subunit2html is a tool that takes in a subunit stream file and will output an
html page
Summary
-------
subunit2html subunit_stream [output]
Usage
-----
subunit2html takes in 1 mandatory argument. This is used to specify the location
of the subunit stream file. For example::
$ subunit2html subunit_stream
By default subunit2html will store the generated html results file at
results.html file in the current working directory.
An optional second argument can be provided to set the output path of the html
results file that is generated. If it is provided this will be the output path
for saving the generated file, otherwise results.html in the current working
directory will be used. For example::
$ subunit2html subunit_stream test_results.html
will write the generated html results file to test_results.html in the current
working directory

110
doc/source/subunit_trace.rst Normal file
View File

@ -0,0 +1,110 @@
.. _subunit_trace:
subunit-trace
=============
subunit-trace is an output filter for subunit streams. It is often used in
conjunction with test runners that emit subunit to enable a consistent and
useful realtime output from a test run.
Summary
-------
subunit-trace [--fails|-f] [--failonly] [--perc-diff|-d] [--no-summary]
[--diff-threshold|-t <threshold>]
Options
-------
--no-failure-debug, -n
Disable printing failure debug information in realtime
--fails, -f
Print failure debug information after the stream is
proccesed
--failonly
Don't print success items
--perc-diff, -d
Print percent change in run time on each test
--diff-threshold THRESHOLD, -t THRESHOLD
Threshold to use for displaying percent change from the
avg run time. If one is not specified the percent
change will always be displayed.
--no-summary
Don't print the summary of the test run after completes
Usage
-----
subunit-trace will take a subunit stream in via STDIN. This is the only input
into the tool. It will then print on STDOUT the formatted test result output
for the test run information contained in the stream.
A subunit v2 stream must be passed into subunit-trace. If only a subunit v1
stream is available you must use the subunit-1to2 utility to convert it before
passing the stream into subunit-trace. For example this can be done by chaining
pipes::
$ cat subunit_v1 | subunit-1to2 | subunit-trace
Adjusting per test output
^^^^^^^^^^^^^^^^^^^^^^^^^
subunit-trace provides several options to customize it's output. This allows
users to customize the output from subunit-trace to suit their needs. The output
from subunit-trace basically comes in 2 parts, the per test output, and the
summary at the end. By default subunit-trace will print failure messages during
the per test output, meaning when a test fails it will also print the message
and any traceback and other attachments at that time. However this can be
disabled by using --no-failure-debug, -n. For example::
$ testr run --subunit | subunit-trace --no-failure-debug
Rhere is also the option to print all failures together at the end of the test
run before the summary view. This is done using the --fails/-f option. For
example::
$ testr run --subunit | subunit-trace --fails
Often the --fails and --no-failure-debug options are used in conjunction to
only print failures at the end of a test run. This is useful for large test
suites where an error message might be lost in the noise. To do this ::
$ testr run --subunit | subunit-trace --fails --no-failure-debug
By default subunit-trace will print a line for each test after it completes with
the test status. However, if you only want to see the run time output for
failures and not any other test status you can use the --failonly option. For
example::
$ testr run --subunit | subunit-trace --failonly
The last output option provided by subunit-trace is to diable the summary view
of the test run which is normally displayed at the end of a run. You can do
this using the --no-summary option. For example::
$ testr run --subunit | subunit-trace --no-summary
Show per test run time percent change
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
subunit-trace provides an option to display the percent change in run time
from the previous run. To do this subunit-trace leverages the testr internals
a bit. It uses the times.dbm database which, the file repository type in
testrepository will create, to get the previous run time for a test. If testr
hasn't ever been used before or for whatever reason subunit-trace is unable to
find the times.dbm file from testr no percentages will be displayed even if it's
enabled. Additionally, if a test is run which does not have an entry in the
times.dbm file will not have a percentage printed for it.
To enable this feature you use --perc-diff/-d, for example::
$ testr run --subunit | subunit-trace --perc-diff
There is also the option to set a threshold value for this option. If used it
acts as an absolute value and only percentage changes that exceed it will be
printed. Use the --diff-threshold/-t option to set a threshold, for example::
$ testr run --subunit | subunit-trace --perc-diff --threshold 45
This will only display percent differences when the change in run time is either
>=45% faster or <=45% slower.

13
doc/source/usage.rst
View File

@ -1,7 +1,12 @@
========
=====
Usage
========
=====
To use os-testr in a project::
This section contains the documentation for each of tools packaged in os-testr
import os_testr
.. toctree::
:maxdepth: 2
ostestr
subunit_trace
subunit2html