infra-manual/doc/source/creators.rst

title

Project Creator's Guide

Project Creator's Guide

Before You Start

This is a long document. It's long because it has to be, not because we want it to be. If you follow it, everything will be fine.

It is important that you perform all of the steps, in the order they are given here. Don't skip any steps. Don't try to do things in parallel. Don't jump around.

Choosing a Good Name for Your Project

It is important to choose a descriptive name that does not conflict with other projects. There are several places you'll need to look to ensure uniqueness and suitability of the name.

Note

If you encounter any issues establishing a valid unique name across all of the tools we use, consult with the Release Manager before going any further.

Character Set

We prefer to use names made up of lower case ASCII letters and the - punctuation symbol to avoid issues with various installation tools.

git repository

The base name of the repository should be unique across all of the namespace directories for git repositories under https://git.openstack.org/cgit. That is, it is not sufficient to have openstack/foo and openstack-dev/foo because that prevents us from moving those two projects into the same namespace at some point.

Launchpad

It is preferred but not absolutely necessary for your project name on https://launchpad.net to be the same as your git repository name. Try "python-" as a prefix if necessary (for example, "python-stevedore").

PyPI

Python projects need to have a unique name on the Python Package Index (https://pypi.python.org) so we can publish source distributions to be installed via pip.

It is best to name the project and the top level Python package the same when possible so that the name used to install the dist and the name used to import the package in source files match. Try "python-" as a prefix if necessary (for example, "python-stevedore").

Project Team Rules

Some OpenStack project teams have naming conventions that must be followed. For example, the Oslo team has instructions for choosing a name for new Oslo libraries.

Set up Launchpad

OpenStack uses https://launchpad.net for project management tasks such as release planning and bug tracking. The first step to importing your project is to make sure you have the right project management tools configured.

Create a new Launchpad Project

1. Visit https://launchpad.net/projects/+new and fill in the details.
2. Name your project using the same name you plan to use for the git repository, unless that is taken. Try "python-" as a prefix if necessary (for example, "python-stevedore"). If that name is also taken, consult with the Release Manager before going any further.

Put Your New Project in the Correct Project Group

1. From the Overview page of your project, select "Change Details" from the right sidebar (e.g., http://launchpad.net/oslo.foo/+edit).
2. Find the "Part of" field and set the value to "openstack" for integrated projects and "oslo" for Oslo libraries.
3. Save your changes.

Create Bug Tracker

1. From the Overview page for your project, click the "Bugs" link at the top of the page. Launchpad should suggest that you set up bug tracking.

2. Choose "In launchpad".

3. Check the box labeled "Expire 'Incomplete' bug reports when they become inactive"

4. Check the box labeled "Search for possible duplicate bugs when a new bug is filed"

5. Set the "Bug supervisor" field to "<projectname>-bugs" (for example, "oslo-bugs").

   Note

   You may need to create the bug management team in Launchpad. If you do so, set the owner of the team to the "OpenStack Administrators team" called "openstack-admins".

6. Save your changes.

Create Blueprint Tracker

If your project uses Launchpad blueprints to track new feature work, you should set up the blueprint tracker now. Otherwise, skip this step.

1. From the Overview page for your project, click the "Blueprints" link at the top of the page. Launchpad should suggest that you set up blueprint tracking.
2. Choose "Launchpad".
3. Save your changes.

Set up Supervisors for your Project

From the Overview page for your project, click the pencil "edit" icon next to the Maintainer field. Replace your name with the <projectname>-drivers team (for example, "oslo-drivers").

Note

You may need to create the drivers team. If you do so, set the owner of the team to 'openstack-administrators'.

From the Overview page for your project, click the pencil "edit" icon next to the Drivers field. Replace your name with the project drivers team.

Note

If either of these steps makes it so you cannot edit the project, stop and ask someone in the drivers group to help you before proceeding.

Give OpenStack Permission to Publish Releases

New projects without any releases need to be manually registered on PyPI.

If you do not have PyPI credentials, you should create them at https://pypi.python.org/pypi?%3Aaction=register_form as they are required for the next step.

Once you have PyPI credentials visit https://pypi.python.org/pypi?%3Aaction=submit_form and fill in only the required fields.

Next your project needs to be updated so the "openstackci" user has "Owner" permissions.

Visit https://pypi.python.org/pypi?:action=role_form&package_name=<projectname> and add "openstackci" in the "User Name" field, set the role to "Owner", and click "Add Role".

Add Project to the Governance Repository

Each project managed by an official OpenStack project team needs to be listed in reference/projects.yaml in the openstack/governance repository to indicate who owns the project so we know where ATCs voting rights extend.

If your project is under the stackforge section of the git repository structure, you can skip this step.

Find the appropriate section in reference/projects.yaml and add the new project to the list. For example, to add a new Oslo library edit the "Oslo" section:

Oslo:
  ptl: Doug Hellmann (dhellmann)
  service: Common libraries
  mission:
    To produce a set of python libraries containing code shared by OpenStack
    projects. The APIs provided by these libraries should be high quality,
    stable, consistent, documented and generally applicable.
  url: https://wiki.openstack.org/wiki/Oslo
  projects:
    - openstack/oslo-incubator
    - openstack/oslo.config
    - openstack/oslo.messaging
    - openstack/oslo.rootwrap
    - openstack/oslo.sphinx
    - openstack/oslo.version
    - openstack-dev/cookiecutter
    - openstack-dev/hacking
    - openstack-dev/pbr

Submit this change and note your change-Id. When you later create the change to the project-config project (the following steps outline how), you will make the project-config change dependent on the governance change by including a reference to the governance change in the commit message of the project-config change.

Adding the Repository to the CI System

To add a repository to the CI System, you need to modify some infrastructure configuration files using git and the OpenStack gerrit review server.

All of the changes described in this section should be submitted together as one patchset to the openstack-infra/project-config repository.

Add the project to the master project list

1. Edit gerrit/projects.yaml to add a new section like:

   - project: openstack/<projectname>
     description: Latest and greatest cloud stuff.

2. Provide a very brief description of the library.

3. If you have an existing repository that you want to import (for example, when graduating an Oslo library or bringing a project into gerrit from github), set the "upstream" field to the URL of the publicly reachable repository and also read the information in setup_review:

   - project: openstack/<projectname>
     description: Latest and greatest cloud stuff.
     upstream: git://github.com/awesumsauce/<projectname>.git

   Note

   If the git repository short name does not match the Launchpad project name, you need to add a "groups" list to provide the mapping. The groups list is also used by Storyboard to be able to present grouped views of stories and tasks across multiple related projects.

   For example, Oslo projects should use "oslo" to ensure that they are associated with the https://launchpad.net/oslo project group for tracking bugs and milestones:

   - project: openstack/<projectname>
     description: Latest and greatest cloud stuff.
     upstream: git://github.com/awesumsauce/<projectname>.git
     groups:
        - oslo

Add Gerrit permissions

Each project should have two gerrit groups. The first, "<projectname>-core", is the normal core group, with permission to +2 changes. The second, "<projectname>-release" is a small group of the primary maintainers with permission to push tags to trigger releases.

Create gerrit/acls/openstack/<projectname>.config:

[access "refs/heads/*"]
abandon = group <projectname>-core
label-Code-Review = -2..+2 group <projectname>-core
label-Workflow = -1..+1 group <projectname>-core

[access "refs/tags/*"]
pushSignedTag = group <projectname>-release

[receive]
requireChangeId = true
requireContributorAgreement = true

[submit]
mergeContent = true

See other files in the same directory for examples.

Add Basic Jenkins Jobs

Test jobs run through Jenkins, and the jobs are defined using jenkins-job-builder configuration files.

Note

Different projects will need different jobs, depending on their nature, implementation language, etc. This example shows how to set up a new Python code project because that is our most common case. If you are working on another type of project, you will want to choose different jobs or job templates to include in the "jobs" list.

Edit jenkins/jobs/projects.yaml to add your project. There are several sections, designated in comments, for different types of projects. Find the right section and then add a new stanza like:

- project:
   name: <projectname>
   node: 'bare-precise || bare-trusty'
   tarball-site: tarballs.openstack.org
   doc-publisher-site: docs.openstack.org
   jobs:
     - python-jobs
     - openstack-publish-jobs
     - pypi-jobs

Configure Zuul to Run Jobs

Zuul is the gate keeper. It watches for changes in gerrit to trigger the appropriate jobs. To start, establish the rules for the jobs you need.

Note

Different projects will need different jobs, depending on their nature, implementation language, etc. This example shows how to set up the full set of gate jobs for a new Python code project because that is our most common case. If you are working on another type of project, you will want to choose different jobs or job templates to include here.

Edit zuul/layout.yaml to add your project. There are several sections, designated in comments, for different types of projects. Find the right section and then add a new stanza like:

- name: openstack/<projectname>
  template:
    - name: merge-check
    - name: python-jobs
    - name: openstack-server-publish-jobs
    - name: check-requirements
    - name: integrated-gate
    - name: publish-to-pypi
    - name: python3-jobs
    - name: translation-jobs

You can find more info about job templates in the beginning of zuul/layout.yaml in the section starting with "project-templates:".

Note

If you use pypi-jobs and publish-to-pypi, please ensure your project's namespace is registered on https://pypi.python.org as described in register-pypi. This will be required before your change is merged.

If you are not ready to run any tests yet and did not configure python-jobs in jenkins/jobs/projects.yaml, the entry for zuul/layout.yaml should look like this instead:

- name: openstack/<projectname>
  template:
    - name: merge-check
    - name: noop-jobs

Configure GerritBot to Announce Changes

If you want changes proposed and merged to your project to be announced on IRC, edit gerritbot/channels.yaml to add your new repository to the list of projects. For example, to announce changes related to an Oslo library in the #openstack-oslo channel, add it to the openstack-oslo section:

openstack-oslo:
  events:
    - patchset-created
    - x-vrif-minus-2
  projects:
    - openstack/cliff
    - openstack/oslo.config
    - openstack/oslo-incubator
    - openstack/oslo.messaging
    - openstack/oslo.rootwrap
    - openstack/oslosphinx
    - openstack/oslo-specs
    - openstack/oslo.test
    - openstack/oslo.version
    - openstack/oslo.vmware
    - openstack/stevedore
    - openstack/taskflow
    - openstack-dev/cookiecutter
    - openstack-dev/hacking
    - openstack-dev/oslo-cookiecutter
    - openstack-dev/pbr
  branches:
    - master

If you're adding a new IRC channel, see the IRC services documentation.

Submitting Infra Change for Review

In your commit message, include a reference to the governance change to make your project-config change depend on it:

Depends-On: <Gerrit Change-Id>

When submitting the change to openstack-infra/project-config for review, use the "new-project" topic so it receives the appropriate attention:

$ git review -t new-project

Wait Here

The rest of the process needs this initial import to finish, so coordinate with the Infra team, and read ahead, but don't do any of these other steps until the import is complete and the new repository is configured.

The Infra team can be contacted via IRC on Freenode in the #openstack-infra channel or via email to the openstack-infra mail list.

Update the Gerrit Group Members

After the review is approved and groups are created, ask the Infra team to add you to both groups in gerrit, and then you can add other members.

The project PTL, at least, should be added to "<projectname>-release", and other developers who understand the release process can volunteer to be added as well.

Updating devstack-vm-gate-wrap.sh

The devstack-gate tools let us install OpenStack projects in a consistent way so they can all be tested with a common configuration. If your project will not need to be installed for devstack gate jobs, you can skip this step.

Check out openstack-infra/devstack-gate and edit devstack-vm-gate-wrap.sh to add the new project:

PROJECTS="openstack/<projectname> $PROJECTS"

Keep the list in alphabetical order.

Add Project to the Requirements List

The global requirements repository (openstack/requirements) controls which dependencies can be added to a project to ensure that all of OpenStack can be installed together on a single system without conflicts. It also automatically contributes updates to the requirements lists for OpenStack projects when the global requirements change.

If your project is not going to participate in this requirements management, you can skip this step.

Edit the projects.txt file to add the new library, adding "openstack/<projectname>" in the appropriate place in alphabetical order.

Preparing a New Git Repository using cookiecutter

All OpenStack projects should use one of our cookiecutter templates for creating an initial repository to hold the source for the project.

If you had an existing repository ready for import when you submitted the change to project-config, you can skip this section.

Start by checking out a copy of the new repository for your project:

$ git clone git://git.openstack.org/openstack/<projectname>

$ pip install cookiecutter

Choosing the Right cookiecutter Template

The template in openstack-dev/cookiecutter is suitable for most projects.

$ cookiecutter https://git.openstack.org/openstack-dev/cookiecutter

The template in openstack-dev/oslo-cookiecutter should be used for Oslo libraries.

$ cookiecutter https://git.openstack.org/openstack-dev/oslo-cookiecutter

Applying the Template

Running cookiecutter will prompt you for several settings, based on the template's configuration. It will then update your project with a project skeleton, ready to have your other project files added.

$ cd <projectname>
$ git review

If you configured all of the tests for the project when it was created in the previous section, you will have to ensure that all of the tests pass before the cookiecutter change will merge. You can run most of the tests locally using tox to verify that they pass.

Verify That Gerrit and the Test Jobs are Working

The next step is to verify that you can submit a change request for the repository, have it pass the test jobs, approve it, and then have it merge.

Configure git review

If the new project you have added has a specified upstream you will need to add a .gitreview file to the project once it has been created. This new file will allow you to use git review.

The basic process is clone your new repository, add file, push to Gerrit, review and approve:

$ git clone https://git.openstack.org/openstack/<projectname>
$ cd <projectname>
$ git checkout -b add-gitreview
$ cat > .gitreview <<EOF
[gerrit]
host=review.openstack.org
port=29418
project=openstack/<projectname>.git
EOF
$ git review -s
$ git add .gitreview
$ git commit -m 'Add .gitreview file'
$ git review

Verify that the Tests Pass

If you configure tests for an imported project, ensure that all of the tests pass successfully before importing. Otherwise your first change needs to fix all test failures. You can run most of the tests locally using tox to verify that they pass.

Verify the Gerrit Review Permissions

When your project is added to gerrit, the groups defined in the ACLs file (see add-gerrit-permissions) are created, but they are empty by default. Someone on the infrastructure team with gerrit administrator privileges will need to add you to each group. After that point, you can add other members.

To check the membership of the groups, visit https://review.openstack.org/#/admin/projects/openstack/<projectname>,access -- for example, https://review.openstack.org/#/admin/projects/openstack-infra/infra-manual,access -- and then click on the group names displayed on that page to review their membership.

Prepare an Initial Release

Make Your Project Useful

Before going any farther, make the project do something useful.

If you are importing an existing project with features, you can go ahead.

If you are creating a brand new project, add some code and tests to provide some minimal functionality.

Provide Basic Developer Documentation

Update the README.rst file to include a paragraph describing the new project.

Update the rest of the documentation under doc/source with information about the public API, tips on adopting the tool, instructions for running the tests, etc.

Tagging a Release

To verify that the release machinery works, push a signed tag to the "gerrit" remote. Use the smallest version number possible. If this is the first release, use "0.1.0". If other releases of the project exist, choose an appropriate next version number.

Note

You must have GnuPG installed and an OpenPGP key configured for this step.

Run:

$ git tag -s -m "descriptive message" $version
$ git push gerrit $version

Wait a little while for the pypi job to run and publish the release.

If you need to check the logs, you can use the git-os-job command:

$ git os-job $version

Allowing Other OpenStack Projects to Use Your Library

OpenStack projects share a common global requirements list so that all components can be installed together on the same system. If you are importing a new library project, you need to update that list to allow other projects to use your library.

Update the Global Requirements List

Check out the openstack/requirements git repository and modify global-requirements.txt to:

1. add the new library
2. add any of the library's direct dependencies that are not already listed

Setting up Gate Testing

The devstack gate jobs install all OpenStack projects from source so that the appropriate git revisions (head, or revisions in the merge queue) are tested together. To include the new library in these tests, it needs to be included in the list of projects in the devstack gate wrapper script. For the same feature to work for developers outside of the gate, the project needs to be added to the appropriate library file of devstack.

Updating devstack

1. Check out openstack-dev/devstack.

2. Edit the appropriate project file under lib to add a variable defining where the source should go. For example, when adding a new Oslo library add it to lib/oslo:

   <PROJECTNAME>_DIR=$DEST/<projectname>

3. Edit the installation function in the same file to add commands to check out the repository. For example, when adding an Oslo library, change install_oslo in lib/oslo.

   When adding the new item, consider the installation order. Dependencies installed from source need to be processed in order so that the lower-level packages are installed first (this avoids having a library installed from a package and then re-installed from source as a dependency of something else):

   function install_oslo() {
     ...
     _do_install_oslo_lib "<projectname>"
     ...
   }

4. Edit stackrc to add the other variables needed for configuring the new library:

   # new-project
   <PROJECTNAME>_REPO=${<PROJECTNAME>_REPO:-${GIT_BASE}/openstack/<projectname>.git}
   <PROJECTNAME>_BRANCH=${<PROJECTNAME>_BRANCH:-master}

Add Link to Your Developer Documentation

Update the http://docs.openstack.org/developer/openstack-projects.html page with a link to your documentation by checking out the openstack/openstack-manuals repository and editing www/developer/openstack-projects.html.

Skip this step if your project is under stackforge.