From e13eab706a440fdb3d4b0b10961a435676c6ca51 Mon Sep 17 00:00:00 2001 From: Mark McLoughlin Date: Tue, 2 Jul 2013 14:11:11 +0100 Subject: [PATCH] Ignore --find-links lines in requirement checks Fixes bug #1194807 In quantum, we currently have a URL based requirement: http://.../oslo.config-1.2.0a2.tar.gz#egg=oslo.config-1.2.0a2 The requirements check currently ignores this. It turns out that pip has a bug which doesn't where you can end up with the oslo.config 1.1.1 code installed. This is because oslo.config>=1.1.0 gets pulled in as a transitive dep and pip gets confused. You can reproduce with e.g. $> pip install \ http://.../oslo.config-1.2.0a3.tar.gz#egg=oslo.config-1.2.0a3 \ python-keystoneclient $> pip freeze | grep oslo.config oslo.config-1.2.0a3 $> python -c 'from oslo.config.cfg import DeprecatedOpt' Traceback (most recent call last): File "", line 1, in ImportError: cannot import name DeprecatedOpt This is because of a bug with pip where it sees oslo.config-1.2.0a3 and oslo.config as two unrelated things. It should strip the version part of the egg= fragment before using it as a package name, but it doesn't. However, we can simply use the -f/--find-links pip option in our requirements.txt to add the tarball URL to the list of URLs considered and also add the oslo.config>=1.2.0a3 dependency: $> pip install \ -f http://.../oslo.config-1.2.0a3.tar.gz#egg=oslo.config-1.2.0a3 \ 'oslo.config>=1.2.0a3' \ python-keystoneclient $> pip freeze | grep oslo.config oslo.config-1.2.0a3 $> python -c 'from oslo.config.cfg import DeprecatedOpt' This is actually exactly the semantics we want and we go to great lengths in pbr to get these semantics while using a single tarball URL. The only downside to this --find-links strategy is that we gain an extra line in our requirements.txt ... but it does work around the pip bug. I think it makes sense for the requirements check to just ignore --find-links lines for now like it does for URLs and -editable lines. Using this method means we actually do require new versions of libraries consumed this way to be approved into openstack/requirements first since we have an explicit 'oslo.config>=1.2.0a3' listed rather than that being derived from a URL. It may make sense in future to have automation around checking which find-links URLs are allowed ... but the same can be true for normal dependency URLs. This change allows us to move forward and use latest oslo.config in Nova, Neutron, etc. without falling foul of the pip bug. Change-Id: I6f3eb5fd2c75615d9a1cae172aed859b36b27d4c Reviewed-on: https://review.openstack.org/35296 Reviewed-by: James E. Blair Approved: Monty Taylor Reviewed-by: Monty Taylor Tested-by: Jenkins --- .../files/slave_scripts/project-requirements-change.py | 9 ++++----- .../files/jenkins_job_builder/config/requirements.yaml | 2 +- 2 files changed, 5 insertions(+), 6 deletions(-) diff --git a/modules/jenkins/files/slave_scripts/project-requirements-change.py b/modules/jenkins/files/slave_scripts/project-requirements-change.py index 2c2fbc6dcb..45c2f5949e 100755 --- a/modules/jenkins/files/slave_scripts/project-requirements-change.py +++ b/modules/jenkins/files/slave_scripts/project-requirements-change.py @@ -41,11 +41,10 @@ class RequirementsList(object): for line in open(fn): line = line[:line.find('#')] line = line.strip() - if not line: - continue - if line.startswith('http'): - continue - if line.startswith('-e'): + if (not line or + line.startswith('http') or + line.startswith('-e') or + line.startswith('-f')): continue req = pkg_resources.Requirement.parse(line) if req.project_name.lower() in self.reqs: diff --git a/modules/openstack_project/files/jenkins_job_builder/config/requirements.yaml b/modules/openstack_project/files/jenkins_job_builder/config/requirements.yaml index d3262e35ce..481fe8b00f 100644 --- a/modules/openstack_project/files/jenkins_job_builder/config/requirements.yaml +++ b/modules/openstack_project/files/jenkins_job_builder/config/requirements.yaml @@ -15,7 +15,7 @@ if [ -e $FILE ] then # Ignore lines beginning with https?:// just as the mirror script does. - sed -e '/https\?:\/\//d' $FILE > $FILE.clean + sed -e '/^https\?:\/\//d' $FILE > $FILE.clean PIP_ARGS="$PIP_ARGS -r $FILE.clean" fi done