This commit adds a new flag to the ostestr cli, --color, which
is a passthrough option to subunit-trace to enabled colorized
output.
Change-Id: Ic38e008982d8f5bca78c52f51c69b5333744ecbc
So in Nova, i wanted to add a text file with a list of
test ids like so:
nova.tests.unit.api.ec2.test_api.ApiEc2TestCase
nova.tests.unit.api.ec2.test_apirequest.APIRequestTestCase
nova.tests.unit.api.ec2.test_cinder_cloud.CinderCloudTestCase
nova.tests.unit.api.ec2.test_cloud.CloudTestCase
functional
to skip those specific test cases and any functional tests and
the current code would not work, so digging through the
regexp(s) on stack overflow and found this:
http://stackoverflow.com/questions/406230/regular-expression-to-match-line-that-doesnt-contain-a-word
Works like a charm!
Change-Id: I0e947c599ab19276f35150749d559487d9790028
A whitelist files allows the user to specify a list of regexes
that ostestr will expand and run. It is mutually exclusive with
blacklist, but not mutually exclusive with --regex/path.
The value of --regex/path is appended to the whitelist.
Co-Authored-By: Davanum Srinivas <davanum@gmail.com>
Change-Id: Ic397212ef3127c176a5870676f67d8a5e0de0441
This commit fixes a couple of bugs in the blacklist file regex
generator. The first where a comment line was accidently made
mandatory. If a comment wasn't specified an IndexError would be
raised. The second was related to the variable naming in the
function's blacklist file if branch. Variable reuse there was causing
the last blacklist regex to take precendence over a regex passed
in and would be used instead.
Change-Id: Ib80a0c1781db7c8c9e4449b4773258fe3348411a
Closes-Bug: #1488700
The use of -n or --pdb was broken if a user
specified the test by path. The problem is
that even though in the case of -n we parse
out path and convert to import notation, we
weren't using the parsed our variable in the
call_subunit_run call.
Also, it turns out that --pdb and -n are really
both work just fine calling call_subunit_run. So
we can remove the def call_testtools_run and just
consolidate pdb and -n into a single statement and
runner call.
I looked at using a secondary exclusive group
between no-discover and pdb, but it it shouldn't
matter if a user tries to specify both, the or
statement should evaluate correctly and things
should just work even if it's pointless to type
in `tox -epy27 -- -n --pdb project.tests.test_name`.
Change-Id: Icd61d9fb710807a508c95e8080865429a1f34fb4
Closes-Bug: #1499891
While working on some other patches and trying to add
some unit tests there were a few things in os_testr that
made it difficult to test.
Rather than try and be super tricky, just refactor os_testr
a bit.
* Refactor get_parser to handle parse_args for us
We'll need to explicitly pass in argv here
* Move the logic of selecting/calling the test
runner into its own helper method (_select_call_runner)
* Modify existing tests to pass args in to get_parser
* Add a new test for runner selection
This cleans up the logic a bit and makes testing a good
deal easier.
Change-Id: I40527601613e6064cf6220f218bef1876ec69cda
This change makes use of argparse groups to generate error
messages when the user specifies invalid command line option
combinations. For example if the user specified --pretty
and --no-pretty before it would not complain, but now it
will.
Change-Id: I07e1edb5c43ff7b0a81879f388770fb732fed43b
Add the capability to the --no-discover option to convert file
names into regular sexpressions. Also add the --path option that
converts a file or directory to a regular expression. Create a
mutually exclusive argparse group for --regex, --path, and
--no-discover.
This change will allow the user to specify a file name instead of a
regular expression to match a particular set of tests e.g:
tox -e py27 -- --path os_testr/tests/test_os_testr.py
Will run os_testr.tests.test_os_testr but you can use tab complete
to generate the name.
Change-Id: Ibfca2bc023aed44b1b87a0444559ab2a00303a70
This commit adds the support to have comments in the exclude file.
After this commit anything after a '#' will be treated as a comment
and ignored for matching. A new option --print-exclude is added which
will print out what is being skipped by the exclude file with any
comments in the file.
Change-Id: Ia357730f918e0a57cf2ac51cd8195e2721667511
This commit adds a missing option to calling subunit trace inside the
until-failure loop. Previously, when a test failed it would not print
the failure output which would make it impossible to debug why it had
failed. This fixes that oversight.
This commit adds an --until-failure option to ostestr which basically
calls through to --until-failure on testr. The key difference is
because of open issues with testr you can't use --until-failure with
subunit output enabled. This would break just a straight passthrough
if either pretty output or subunit output are enabled. This
works around this by manually reimplementing this functionality by
generating a test list and looping over it and running the tests with
the desired output forever until a failure is encountered. The tradeoff
here is that to do this the test operations are serialized.
As part of this to make the pretty output not overly verbose an option
is added to subunit-trace to disable the summary view from being
printed.
This commit switches from using subprocess to call testtools.run and
subunit.run with python -m to directly calling the methods being run.
This should make the non-default cases when using subunit.run and
testtools.run faster, and it simplifies the code. As part of this, the
code around call_subunit is fixed to make sure the it works as expected
given different argument combinations.
ostestr will still call subprocess to run testr, because the interface
is more complex, and when subunit.run is used with subunit-trace,
because the stdin handling is tricky. The subunit.run with subunit-trace
case will be handled in a later patch.
This commit adds options to ostestr to control the parallelism and
concurrency of testr. By default it will run in parallel with
concurrency equal to the number of cpus on the system. This commit
also modifies the metavar values on the no-discover and pdb options
to make them more descriptive.
The subprocess.call() to run testr init if the .testrepository dir
hasn't been created yet was setting passing the actual call assuming
shell=True, which it wasn't. This commit fixes this oversight to make
the call actually work.
This commit fixes a couple of issues from running flake8 for the first
time. Mostly minor whitespace fixes, but we'll need this for when we
move this repo over into openstack's ci system.
This commit adds a --pdb flag which is similar to --no-discover except
that instead of running subunit.run test_id to bypass test discovery,
testtools.run test_id is used so that pdb traces set in the code are
useable. As part of this several fixes in the --no-discover path were
added since the codes is similar.
This commit migrates the current subunit trace commit from tempest-lib
with the commits:
d7c3f6b Merge "Summarize expected failures"
e29ec71 Summarize expected failures
21e3f6a Enable stdout passthrough for subunit-trace
d588748 Default the worker number to 0 not NaN
87c1442 Fix subunit-trace on python < 2.7
b73b9eb bring over fail only functionality from nova
5715fd6 Switch to elapsed time in subunit-trace summary
d2e4040 Setup subunit-trace as an entry point
and also adds the start of the ostestr command to wrap testr.
