69db7f81fa
Unusual documents are documents with different data types for the data field. The data types include: object, array, string and integer. This PS makes necessary ORM model and schema changes needed to support the different data types. The ORM data type for the data column has been changed to JSONB for PostgreSQL. Thus, DH now only supports PostgreSQL. As a result, the tox jobs have been updated to only use postgre. Change-Id: I53694d56bef71adacb5eb79162678be73acb4ad8
127 lines
4.3 KiB
ReStructuredText
127 lines
4.3 KiB
ReStructuredText
..
|
|
Copyright 2017 AT&T Intellectual Property. All other rights reserved.
|
|
|
|
Licensed under the Apache License, Version 2.0 (the "License");
|
|
you may not use this file except in compliance with the License.
|
|
You may obtain a copy of the License at
|
|
|
|
http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
Unless required by applicable law or agreed to in writing, software
|
|
distributed under the License is distributed on an "AS IS" BASIS,
|
|
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
See the License for the specific language governing permissions and
|
|
limitations under the License.
|
|
|
|
=======
|
|
Testing
|
|
=======
|
|
|
|
Unit testing
|
|
============
|
|
|
|
Prerequisites
|
|
-------------
|
|
|
|
`pifpaf <https://github.com/jd/pifpaf>`_ is used to spin up a temporary
|
|
postgresql database for unit tests. The DB URL is set up as an environment
|
|
variable via ``PIFPAF_URL`` which is referenced by Deckhand's unit test suite.
|
|
|
|
Guide
|
|
-----
|
|
|
|
Unit testing currently uses an in-memory sqlite database. Since Deckhand's
|
|
primary function is to serve as the back-end storage for UCP, the majority
|
|
of unit tests perform actual database operations. Mocking is used sparingly
|
|
because Deckhand is a fairly insular application that lives at the bottom
|
|
of a very deep stack; Deckhand only communicates with Keystone and Barbican.
|
|
As such, validating database operations is paramount to correctly testing
|
|
Deckhand.
|
|
|
|
To run unit tests using PostgreSQL, execute::
|
|
|
|
$ tox -epy27
|
|
$ tox -epy35
|
|
|
|
against a py27- or py35-backed environment, respectively. To run individual
|
|
unit tests, run::
|
|
|
|
$ tox -e py27 -- deckhand.tests.unit.db.test_revisions
|
|
|
|
for example.
|
|
|
|
.. warning::
|
|
|
|
It is **not** recommended to run postgresql-backed unit tests concurrently.
|
|
Only run them serially. This is because, to guarantee true test isolation,
|
|
the DB tables are re-created each test run. Only one instance of PostgreSQL
|
|
is created across all threads, thus causing major conflicts if concurrency
|
|
> 1.
|
|
|
|
Functional testing
|
|
==================
|
|
|
|
Prerequisites
|
|
-------------
|
|
Deckhand requires Docker to run its functional tests. A basic installation
|
|
guide for Docker for Ubuntu can be found
|
|
`here <https://docs.docker.com/engine/installation/linux/docker-ce/ubuntu/>`_.
|
|
|
|
Overview
|
|
--------
|
|
Deckhand uses `gabbi <https://github.com/cdent/gabbi>`_ as its functional
|
|
testing framework. Functional tests can be executed via::
|
|
|
|
$ tox -e functional
|
|
|
|
You can also run a subset of tests via a regex::
|
|
|
|
$ tox -e functional -- gabbi.suitemaker.test_gabbi_document-crud-success-multi-bucket
|
|
|
|
The command executes ``tools/functional-tests.sh`` which:
|
|
|
|
1) Launches Postgresql inside a Docker container.
|
|
2) Sets up a basic Deckhand configuration file that uses Postgresql
|
|
in its ``oslo_db`` connection string.
|
|
3) Sets up a custom policy file with very liberal permissions so that
|
|
gabbi can talk to Deckhand without having to authenticate against
|
|
Keystone and pass an admin token to Deckhand.
|
|
4) Instantiates Deckhand via ``uwisgi``.
|
|
5) Calls gabbi which runs a battery of functional tests.
|
|
6) An HTML report that visualizes the result of the test run is output to
|
|
``results/index.html``.
|
|
|
|
At this time, there are no functional tests for policy enforcement
|
|
verification. Negative tests will be added at a later date to confirm that
|
|
a 403 Forbidden is raised for each endpoint that does policy enforcement
|
|
absent necessary permissions.
|
|
|
|
CICD
|
|
----
|
|
Since it is important to validate the Deckhand image itself, CICD:
|
|
|
|
* Generates the Deckhand image from the new patchset
|
|
* Runs functional tests against the just-produced Deckhand image
|
|
|
|
Deckhand uses the same script -- ``tools/functional-tests.sh`` -- for CICD
|
|
testing. To test Deckhand against a containerized image, run, for example:
|
|
|
|
::
|
|
|
|
export DECKHAND_IMAGE=quay.io/attcomdev/deckhand:latest
|
|
tox -e functional
|
|
|
|
Which will result in the following script output:
|
|
|
|
::
|
|
|
|
Running Deckhand via Docker
|
|
+ sleep 5
|
|
+ sudo docker run --rm --net=host -p 9000:9000 -v /opt/stack/deckhand/tmp.oBJ6XScFgC:/etc/deckhand quay.io/attcomdev/deckhand:latest
|
|
|
|
.. warning::
|
|
|
|
For testing dev changes, it is **not** recommended to follow this approach,
|
|
as the most up-to-date code is located in the repository itself. Running tests
|
|
against a remote image will likely result in false positives.
|