Currently tests are sorted alphabetically by test class name regardless
of the pass/fail status. When troubleshooting failed unit tests in
jenkins jobs with many unit tests (> 11242 in nova), one must scroll
potentially a lot or wait as the page loads to search for the word
'fail' to get to the details of the failed test.
This sorts failed (and error) tests at the top of the HTML page for
easier troubleshooting (sorted failed tests + sorted passed tests).
Change-Id: I0b575e77c4a3e1cc73e60ea48ca5b9bf2e84d76f
The regex building logic is independently useful and will likely start
to be used in other places. This commits splits it out from the ostestr
cli and makes it an independent module that just contains the pieces
necessary for building a selection regex.
Change-Id: Ic8494d0f54357fdafd650b40219e6ad7fd5a65ad
This commit splits out the actual output generation functionality from
the main function. It also adds parameters for specifying a stdin and
stdout file object. This enables external access to the output
generation within python, which could be useful for integrating
subunit-trace output with your own test runner.
Change-Id: Id9fc13b9b70c5b2cc47b48eee6426e7e312eff7f
This commit adds a version option to os-testr commands. Users would like
to confirm the version when then face unexpected behavior of the command
like a bug.
Change-Id: I30ee9623c25f2ce2da0fab0b5e0a2795b46a2cd6
This commit updates one edge case where output from xfail was treated
like a failure instead of like a success.
Change-Id: I073302bc9aea6751906cb4955055c55b7c5dc3fc
This commit enables passing through any unknown arguments to testr run.
There are several features in testr's run cli which aren't exposed in
ostestr. Instead of adding a duplicate argument for each of these this
commit will capture any unidentified arguments and will pass them
through to the testr run subprocess call.
Change-Id: I26eb5c6a5908258c3035b93d72924646bb834d40
We already support comments in blacklist files. we should
do the same for whitelist files as well.
Change-Id: I5ad0c113cdb04dd1bdaa4d8bfdd1a3ab169fb0af
This commit adds a generate_subunit.py script which is used to
generate a subunit stream with a single result for a period of time.
It takes 2 mandatory args and 2 optional to specify the start time,
stop time, and optionally the status (it defaults to success) and an
id for test (if one isn't provided 'devstack' is used) The resulting
stream is written to STDOUT. There is some overlap with subunit-output
from tool from python-subunit, but this is a much smaller scope to just
just do a test_id, status, and timestamps. subunit-output doesn't support
timestamps. Eventually it'll be good to add the missing pieces to
subunit-output at which point we can likely deprecate and remove this.
The intent here is to leverage this to inject 'test results' into the
subunit2sql db to reflect failures that occur before tempest (or any
other test suite) is run. This is necessary for the openstack-health
dashboard. (otherwise it makes our failure rate look much better than
it is) This is only needed until we get a zuul mysql reporter in place
which can give us the higher level run information.
Change-Id: Icc7df33e4d73ba6322af38fbdf3aea230f2fcf4d
This commit adds a new flag to the ostestr cli, --color, which
is a passthrough option to subunit-trace to enabled colorized
output.
Change-Id: Ic38e008982d8f5bca78c52f51c69b5333744ecbc
Currently only v1 correctly parses out the class information from the test_id.
This changes the default behavior to always parse the class and module from test_id.
Closes-Bug: #1517229
Change-Id: I6340797064f9289a7e861f853c42a763c5f5e2c2
This commit enables that subunit-trace shows colored output. Because
there are some people who like colored output. While this doesn't work
for the OpenStack gate for local testing it would be nice to appease
those folks and have a --color flag that adds different colors on the
console output from subunit-trace.
Change-Id: Iac48591786d10c908d63e6df94b979ded7d8dbe1
Closes-Bug: #1519044
This commit adds another sanity check to ensure we didn't have a test
run with just skipped tests.
Change-Id: Ie4bd185c209d9e9230ca8a3f54e8a433e82d9c14
On certain environments if the locale is set to ASCII when printing an
attachment things will explode because it's potentially trying to use
unicode chars outside of ASCII. This commit forces the attachment
output to be unicode encoded to avoid this issue.
Change-Id: I79ff7482ec3b1fd3ce83b8acf3137119b3db39a9
Closes-Bug: #1501415
So in Nova, i wanted to add a text file with a list of
test ids like so:
nova.tests.unit.api.ec2.test_api.ApiEc2TestCase
nova.tests.unit.api.ec2.test_apirequest.APIRequestTestCase
nova.tests.unit.api.ec2.test_cinder_cloud.CinderCloudTestCase
nova.tests.unit.api.ec2.test_cloud.CloudTestCase
functional
to skip those specific test cases and any functional tests and
the current code would not work, so digging through the
regexp(s) on stack overflow and found this:
http://stackoverflow.com/questions/406230/regular-expression-to-match-line-that-doesnt-contain-a-word
Works like a charm!
Change-Id: I0e947c599ab19276f35150749d559487d9790028
A whitelist files allows the user to specify a list of regexes
that ostestr will expand and run. It is mutually exclusive with
blacklist, but not mutually exclusive with --regex/path.
The value of --regex/path is appended to the whitelist.
Co-Authored-By: Davanum Srinivas <davanum@gmail.com>
Change-Id: Ic397212ef3127c176a5870676f67d8a5e0de0441
This commit fixes a couple of bugs in the blacklist file regex
generator. The first where a comment line was accidently made
mandatory. If a comment wasn't specified an IndexError would be
raised. The second was related to the variable naming in the
function's blacklist file if branch. Variable reuse there was causing
the last blacklist regex to take precendence over a regex passed
in and would be used instead.
Change-Id: Ib80a0c1781db7c8c9e4449b4773258fe3348411a
Closes-Bug: #1488700
The use of -n or --pdb was broken if a user
specified the test by path. The problem is
that even though in the case of -n we parse
out path and convert to import notation, we
weren't using the parsed our variable in the
call_subunit_run call.
Also, it turns out that --pdb and -n are really
both work just fine calling call_subunit_run. So
we can remove the def call_testtools_run and just
consolidate pdb and -n into a single statement and
runner call.
I looked at using a secondary exclusive group
between no-discover and pdb, but it it shouldn't
matter if a user tries to specify both, the or
statement should evaluate correctly and things
should just work even if it's pointless to type
in `tox -epy27 -- -n --pdb project.tests.test_name`.
Change-Id: Icd61d9fb710807a508c95e8080865429a1f34fb4
Closes-Bug: #1499891
While working on some other patches and trying to add
some unit tests there were a few things in os_testr that
made it difficult to test.
Rather than try and be super tricky, just refactor os_testr
a bit.
* Refactor get_parser to handle parse_args for us
We'll need to explicitly pass in argv here
* Move the logic of selecting/calling the test
runner into its own helper method (_select_call_runner)
* Modify existing tests to pass args in to get_parser
* Add a new test for runner selection
This cleans up the logic a bit and makes testing a good
deal easier.
Change-Id: I40527601613e6064cf6220f218bef1876ec69cda
This commit fixes a bug in subunit-trace where if a test is skipped
but does not have a reason associated with the skip it would stack
trace. This checks for the existence of a reason before trying to
use it.
Change-Id: I14dd9fbb40a8c232431e5042aa46f7e521e15311
This change makes use of argparse groups to generate error
messages when the user specifies invalid command line option
combinations. For example if the user specified --pretty
and --no-pretty before it would not complain, but now it
will.
Change-Id: I07e1edb5c43ff7b0a81879f388770fb732fed43b
Add the capability to the --no-discover option to convert file
names into regular sexpressions. Also add the --path option that
converts a file or directory to a regular expression. Create a
mutually exclusive argparse group for --regex, --path, and
--no-discover.
This change will allow the user to specify a file name instead of a
regular expression to match a particular set of tests e.g:
tox -e py27 -- --path os_testr/tests/test_os_testr.py
Will run os_testr.tests.test_os_testr but you can use tab complete
to generate the name.
Change-Id: Ibfca2bc023aed44b1b87a0444559ab2a00303a70
If a subunit stream is aborted in the middle (like in the case of a
segfault) this causes subunit-trace to emit a stacktrace. This commit
attempts to handle this edge case gracefully.
Related-Bug: #1482230
Change-Id: I1a8a0a8e2ab65e637c6a5212e324670b7d95d28d
This commit disables printing the percent change in run time by
default. This was a cool experiment, but isn't really useful in
practice. Especially in the gate, things are so noisy and there
is so much variance looking at the change for a single doesn't
actually mean anything. It makes more sense to make this feature
opt-in because it's not generally useful.
Change-Id: Iecb153452edfe1d7b55757d022ae0331ac563b35
Python 3 renames StringIO -> io. Use six to deal
with this change.
Introduces new test dependency on six for StringIO.
Change-Id: Ia875b7fcbb976599053970ef79ed3f3474626bad
The database I am getting back is a gdm database and it does
not have a get method. Catch the exception and try something
else. If that blows up, ignore because we'd rather see the
results of our tests.
Change-Id: I2882e19d49f2fb3471669f5eb8a017c5d5ac98c2
When a subunit stream for a testcase doesn't contain start end
enddate, the duration can't be calculated which leads to a:
ValueError: could not convert string to float
Check now if the duration is an empty string and add basic test
coverage based on ddt for the subunit_trace command.
Change-Id: I9953019794ba53fcfcb20e32fecbe94da22c9565
This commit adds the support to have comments in the exclude file.
After this commit anything after a '#' will be treated as a comment
and ignored for matching. A new option --print-exclude is added which
will print out what is being skipped by the exclude file with any
comments in the file.
Change-Id: Ia357730f918e0a57cf2ac51cd8195e2721667511
This commit fixes pep8 issues that were added when subunit2html was
copied into the repo. As part of this the subunit2html was missing a
license header for the additions made to it on top of the original
HTMLTestRunner script it was based on. The addition of the copyright
header mirrors project-config change:
I6fc16d316012fd3e1c196f74df25d725a310f6dc
Change-Id: Ieb42d5baddb3e1446fcd50197136f605324323aa
This commit adds another utility to os-testr, subunit2html, which is
used to generate html output from a subunit stream. This utility is
currently being used after OpenStack test jobs to generate the
testr_results.html page. The previous home for this file was in
the openstack-infra project config repo as a jenkins slave script.
This commit adds a missing option to calling subunit trace inside the
until-failure loop. Previously, when a test failed it would not print
the failure output which would make it impossible to debug why it had
failed. This fixes that oversight.
This commit adds an --until-failure option to ostestr which basically
calls through to --until-failure on testr. The key difference is
because of open issues with testr you can't use --until-failure with
subunit output enabled. This would break just a straight passthrough
if either pretty output or subunit output are enabled. This
works around this by manually reimplementing this functionality by
generating a test list and looping over it and running the tests with
the desired output forever until a failure is encountered. The tradeoff
here is that to do this the test operations are serialized.
As part of this to make the pretty output not overly verbose an option
is added to subunit-trace to disable the summary view from being
printed.
This commit adds some basic unit tests which runs ostestr in some
basic configurations against a fake test suite to ensure that the
ostestr always exits with a 0 on success and 1 on errors when running
tests. This is invaluable for using ostestr in ci systems.
This commit adds a percent change to the duration on individual test
output lines. This is conditionally displayed based on the presence
of a testrepository times.dbm file and data in that file for the test
being run. If there is useable data subunit-trace will now use the
runtimes from there to display how the current run has changed from
the dbm file. A new threshold option is added to optionally specify a
minimum percent change to be used to determine whether to display the
value or not.
Change-Id: I3d68425f48114531a78cab08c353111648ce3911
This commit switches from using subprocess to call testtools.run and
subunit.run with python -m to directly calling the methods being run.
This should make the non-default cases when using subunit.run and
testtools.run faster, and it simplifies the code. As part of this, the
code around call_subunit is fixed to make sure the it works as expected
given different argument combinations.
ostestr will still call subprocess to run testr, because the interface
is more complex, and when subunit.run is used with subunit-trace,
because the stdin handling is tricky. The subunit.run with subunit-trace
case will be handled in a later patch.
This commit adds options to ostestr to control the parallelism and
concurrency of testr. By default it will run in parallel with
concurrency equal to the number of cpus on the system. This commit
also modifies the metavar values on the no-discover and pdb options
to make them more descriptive.
The subprocess.call() to run testr init if the .testrepository dir
hasn't been created yet was setting passing the actual call assuming
shell=True, which it wasn't. This commit fixes this oversight to make
the call actually work.
