I noticed this by accident when I ran ansible-lint over this repo from
an outside context; it didn't use the .yamllint in here and started
compalining about eof whitespace.
After scratching my head for a bit as to why this didn't fail here, I
realised we've allowed various newlines since the initial commit
I936fe2c997597972d884c5fc62655d28e8aaf8c5.
Remove this and just use the default eof rules, and fixup the
whitespace as required. This is fairly unimportant, but is nice for
consistency.
Change-Id: Idb46a1f39ba798b0bf70eaa27b4c6b4758ce3d26
The below error is coming on mol-centos7 jobs while
installing sibling for python2 packeges on centos7
Error: TypeError: initial_value must be unicode or None, not str
This patch handled the TypeError while installing sibling
for python packeges for centos7.
Closes-Bug: #1946641
Change-Id: Ie8058cca92d099e50af19b95b4c417c5a665da0d
With the original verbose output support in change
Iafeb88eaf9a596603ad4d2134a4574345d5189ab we looked for lines from
tox --showconfig output starting with an opening '[' but verbosity
also causes output from pip install activity to be included if a
tox.requires entry causes it to install packages before continuing
and these lines are prefixed by a process ID number wrapped in
brackets with the command string after that, which prematurely
triggered our search for the start of the INI content. Add a
stipulation that the first INI output line also end in ']' in order
to skip over those additional prepended lines.
Change-Id: If29e5a9abe3b92a145d87f5efc1b93350ea3908a
Unfortunately, when tox combines --showconfig with verbosity options
like -vv, some non-config output gets streamed to stdout before the
configuration is emitted. Filter this preamble in
tox_install_sibling_packages by discarding any initial lines of
output before the first section heading.
Also extend get_envlist() to deal with the fact that additional
verbosity adds a [tox] section in the --showconfig output, which it
was previously relying on to determine whether the config had been
filtered to a subset of env sections. Instead also check the
tox.args string to determine whether a -e option was passed on the
command line.
Change-Id: Iafeb88eaf9a596603ad4d2134a4574345d5189ab
This allows projects, that use tox but may not have a setup.cfg file
still use tox siblings. We do this to allow non-python project, to use
tox as an entry point for testing, and still have depends-on
requirements work in zuul.
Change-Id: I9b37117b27ff6b7e436d456b6cbae39ccb9b968c
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
There's a couple of places that call subprocess.check_output that
might raise and exception (I'm seeing one finding the "--name"); catch
the CalledProcessError and include it's output to give more info.
Change-Id: I53cfb9c1b37bd4dbf9e5bf04fbab83d3cdad795d
siblings: python2.7 ConfigParser has no __getitem__
Constrain soupsieve to <2 for python2.7
Add python2-dev to bindep for subprocess32
Change-Id: If9d6a0ae1a62a94dcec11f6bf637ffee7f0f4fc9
Also we can parse the output from --showconfig to se which
testenvs we should act upon when installing siblings.
Change-Id: I438035c92d62102c80db9bed81e9f290e4c941b2
Since tox_envlist has a default value it cannot be undefined
so the fail task will never run. Instead handle the case when
tox_envlist is an empty string by getting the default configured
envlist from tox. Also handle the casewhen tox_envlist is 'ALL'.
This also updates tox_install_sibling_packages to correctly
handle multiple testenvs and uses configuration supplied by
'tox --showconfig -e <envlist>' instead of guessing where the
envdir and logdir are located.
We also cannot run tox inside python because it gets complicated
to know which tox_executable we should call during the python test
cases so run these commands in ansible and pass the output to
tox_install_sibling_packages.
Since role params have higher precedence than set_fact we set an
internal _tox_envlist fact that is a comma separated list of testenvs
that should be run.
Change-Id: I9e5a1b041f653cbcff7b8ed62e4a95a0a040fdd7
This reverts commit 53f2444dbba473c22554af1689dfc444f3441018.
Revert to avoid regression:
https://review.opendev.org/710208 just gave a lot of new linting warnings that look new "py38: DeprecationWarning: inspect.getargspec() is deprecated since Python 3.0, use inspect.signature() or inspect.getfullargspec()"
Change-Id: I26c862255cce6b6c99a69d9e3f5f4354db90f953
This is needed to be able to parse output from ansible-lint.
Removes a testcase with a faulty assumption that if the column is
not present the output must be a warning and should not be returned.
It is probably better to allow the user to configure their linting
output themselves.
Also we don't need a separate matcher for sphinx.
Change-Id: I6d06dce921348d3c6fb5a56cdc1e4df675d685c2
Adds yamllint to the linters with a minimal configuration, some
rules are disabled to allow us to fix them in follow-ups, if
we agree on them.
Fixes invalid YAML file containing characters inside block.
Fixes few minor linting issues.
Change-Id: I936fe2c997597972d884c5fc62655d28e8aaf8c5
The sphinx regex is picking up python warnings. We can disambiguate
the two because sphinx doesn't emit a space before the message but
the python warnings module does. This adjusts the regex to do that
and adds a test file that includes warnings output.
If we need/want to make this more robust, we could also simply filter
the sphinx output for messages that start with " ?\w+Warning:".
We could also decide that we want to include python warnings.
Change-Id: Ib58cc1a2d4f673a0ce3d2a9cae306004559f5b52
We get many false positives on the .tox directory, ignore it.
Even better would be to only handle files under version control but that
is more involved, so use the simple heuristic for .tox for now since
that catches 90+ per cent of the cases.
Change-Id: Ibb17958454a4d6bf156020fe4d9f588f3c666cc7
After stripping the work dir there should not be any absolute path
left that can be related to the repo. Thus filter them so they don't
produce mapping warnings in zuul.
Change-Id: Iadeac9b1faea63aa6009e2cfbc215853695abbe6
Some programs like sphinx emit absolute paths in error messages. Those
cannot be processed by zuul so filter them if they're part of
zuul.project.
Change-Id: Ie8af91017c26f39c3aa9ca098d4230f0688351ce
We shouldn't catch exceptions during parsing without having any
log. This makes debugging impossible. Since errors are now ignored we
can stop silently catching any exception and take the chance to harden
the parsing when we hit exceptions in the future.
Change-Id: Ie58acc4c7fbfcb7e44eef49883fb69f3bccef994
The regex used to filter for line comments needs to be fine tuned for
different linters. Instead having a huge complicated regex add support
for more specialized regexes. Start with a regex that matches spinx
output.
Change-Id: I516086bba315b8bb03a1103e1b33f1a819853be7
keystoneauth has some unit tests which output lines like:
stdin:5:1: K333 'from oslo import utils' must be used instead of 'from oslo import utils'.
The stdin isn't a file, but we try to report it back.
Pass in zuul_work_dir and use it to check that the "file"
is a file that exists.
Change-Id: I2c2aeb4e68f7d50f15ea6c6128fec6050d476c64
To make it clear which tox command a given comment came from
in the case where more than one job has messages, prepend the
tox_envlist parameter to the inline message.
Change-Id: I70228baa285c4e676dc1348c354d4bbf3a3825bc
The `tox_parse_output` currently logs the entire output of tox
into the syslog as the natural Ansible behaviour, as part of doing
that, it also encodes it inside utf-8. This results in errors
in the case that the the data contains surrogates and crashes the
entire job.
Given that this issue really occurs during logging and we don't
have a reason to log all of the tox output to the syslog, this
patch sets the module parameter to `no_log` which should skip the
encoding and logging for that parameter[0].
[0]: https://github.com/ansible/ansible/blob/stable-2.9/lib/ansible/module_utils/basic.py#L1945-L1957
Change-Id: I470cd35111a63662c43f779fc9e119aa8d78c838
We need to pin enum34 on python2 because upstream broke it.
Also, we should not install zuul in python2, because it's not targetted
at python2. It's only here so that linters can deal with zuul_return
and other things from its library, so it should be fine to not install
it for python2 unittests.
GitPython was in here as a pin to the versions that support python3
for zuul. Since we're constraining zuul to only python3, we don't need
to mention GitPython at all.
Update the siblings unit test to use requests as the package we expect
to be already installed since we're no longer installing zuul for
python2.
Fedora 29 got archived upstream:
http://mirror.sjc1.vexxhost.openstack.org/fedora/releases/29/README
So we need to switch to fedora 30.
Change-Id: I69e0b03f624ba45889916f89c0912df667aaf096
This fixes a byte vs str problem on python3 in the tox role's sibling
library.
As part of this fix add unit testing of the function that was failing
under python3 and functional testing of the role that covers siblings.
Change-Id: I2df4ddce91dfd53f5c4177a19288e97120bc3e3b
Under Python3, subprocess.check_output returns binary data,
convert to utf-8 so that it runs under both Python2 and 3.
Using decode works with both Py2 and Py3.
Note that in other places, we already convert the output of check_output
with decode, this change follows that pattern.
Change-Id: I727a2368233473f57978ba24947f6d861b02f7da
Some projects have a setup.cfg included, but don't have metadata with
the package name in it. In this case, skip on to finding it via
setup.py
Change-Id: I9339ac86832994b9f7f706c99e07441fc120b25a
This patch is a follow up for patch:
I3573c067f63a09f22377612f53c80c80252965e4
Comment explains the string matching lines for the pip freeze output.
Change-Id: Ifc5b2ae002859c1ca7405c8e1f64380b311e983d
In pip 19.0 release [1] the
Editable Git installs without a remote now freeze as editable. (#4759) [2]
bug is fixed. Now the output of pip freeze for such package looks like:
# Editable Git install with no remote (neutron==10.0.8.dev66)
-e /home/zuul/src/git.openstack.org/openstack/neutron
instead of:
## !! Could not determine repository location
neutron==10.0.8.dev66
Since for listing the package names the module uses the lines that contains
'==' the new behaviour leaves some garbage in front of the package name. So
in case the string contains '(' then remove the first part of the string.
[1] https://pip.pypa.io/en/stable/news/#id15
[2] https://github.com/pypa/pip/issues/4759
Change-Id: I3573c067f63a09f22377612f53c80c80252965e4
Move the zuul requirement from test-requirements into the linters
section of tox.ini because we need it as a src install in the tox
env, so that we can reference the ansible library location.
Change-Id: I089c69b539107bdbc25791f5730502a4f46e7cf6
Add a unit testing framework for python roles. Thanks to Matthew
Treinish for the suggestion of how to perform discovery (and much of the
code which is copied from Tempest).
Change-Id: Iec95dd1026a41614def57c65c3faa0516a682a5a
Some openstack packages show up in the requirements list under a
hyphenated name while their package metadata name contains only
underscores. (For example, the glance_store library is listed in
requirements as 'glance-store'.) This is causing problems for the
tox_install_sibling_packages task run by zuul. Pip is able to handle
this situation [0]; add code so the ansible task can handle it too.
[0] http://paste.openstack.org/show/722298/
Change-Id: I89ad9926647a011174815761e79372f2d7d43609
Needed-by: https://review.openstack.org/#/c/569225/
Closes-bug: #1774030
When python setup.py --name is run in ansible/ansible currently, a
warning is printed:
UserWarning: Normalizing '2.7.0dev0' to '2.7.0.dev0' normalized_version
With stderr=stdout, this goes into the package name causing sibling
detection to not work.
The stderr is not interesting to us from this command, so ignore it.
Change-Id: I38ea21960932dbddcac1b828e95a0cf963c3b94c
As of pip10 pip is no longer promising external stable apis (they never
really did anyways just failed to annotate the source as such). As a
result our use of direct pip internals breaks under pip10.
We replace this with a fork and exec of pip instead as the command line
is a stable api. We were already doing this anyways so shouldn't cause
anything to run slower and should be more reliable as an api.
Note that we also remove the import of pip as this does an unexpected
thing and uses pip outside of the virtualenv rather than pip inside the
virtualenv. Since we are only forking pip inside the virtualenv now we
don't need to check for external pip.
Change-Id: Ib2628003995530881d297782f044dfc8a1c72d08
The current logic uninstalls all detected siblings first, then it
executes a single install with all of the detected sibling git repos.
If one sibling (A) depends on another sibling (B), then the commands:
pip uninstall A
pip uninstall B
pip install ../A ../B
Will result in pip finding B in the requirements list for ../A and
re-installing the released version, then considering the requirement
satisfied when it gets to ../B and not installing it.
To solve that, do the uninstall and reinstall in a single-project loop
for each sibling. This results in:
pip uninstall A
pip install ../A
pip will find B in ../A's requirements, but B will already be installed
because it hasn't been uninstalled yet, so nothing will happen. Then:
pip uninstall B
pip install ../B
Which will result in both ../A and ../B being installed.
As if that wasn't enough, if A and B happen to appear in the opposite
order, doing:
pip uninstall B
pip install ../B
pip uninstall A
pip install ../A
Will wind up with pip uninstalling ../B and installing B if the version
requested for B is greater than the version reported by ../B (which is
the case for openstack projects operating from master after a release
but before the first release of the next cycle)
In order to combat that, do a second installation pass with --no-deps.
This way the first pass will be sure to get any transitive dependency
changes, but the second pass will ensure all the siblings are installed
instead of their non-git versions.
pip uninstall B
pip install ../B
pip uninstall A
pip install ../A
pip install --no-deps ../B
pip install --no-deps ../A
Change-Id: I060e188313391de7847adf851566468c4b032342
ansible/ansible doesn't use pbr, so we can't inspect its setup.cfg to
drive tox-nstall-siblings.
In case we find a setup.py but no name in setup.cfg, fallback to running
setup.py --name.
Change-Id: I6d234247e3d45befd264e159c8b0fe77f01c7fdc
When sibling installation fails, we only get MODULE_FAILURE and no
feedback. Wrap the guts in an exception handler so we can figure out
what's wrong.
Change-Id: I42b61ad58415bacbd0d7b220fb47534745ddaa26
There is a bunch of duplicate logic between tox and tox-siblings, the
tox-siblings logic is already protected with a flag - and having them be
split already led to not doing the right thing with constraints files in
siblings when we did it right in tox itself.
Combine them into one role.
This means the siblings code will run as part of run instead of as part
of pre- but that's how things have worked for years anyway until
siblings was introduced - and is also the behavior is the siblings flag
is false -so it's more consistent overall.
Leave a no-op tox-siblings role so that we can gracefully remove the use of
tox-siblings from castellan, python-openstacksdk, shade and tacker.
Change-Id: Id61ae52d48b28cfc2221cb556a1c1f7c6dfd60dd
