diff --git a/README.rst b/README.rst index 1cbda34..5b5173a 100644 --- a/README.rst +++ b/README.rst @@ -1,34 +1,42 @@ Shaker ====== -Shake VMs with our sheer-class tests! +The distributed data-plane testing tool for OpenStack. -Installation +Features +-------- + + * User-defined topology via Heat templates + * Simultaneously test execution on multiple instances + * Pluggable tools + * Interactive report with stats and charts + +Requirements ------------ -The tool consists of a single server running on master node and set of dynamically -provisioned agents. In order to run the server needs to know OpenStack parameters -(credentials, auth_url), they can be set via environment (e.g. by sourcing openrc file) -or via parameters. + * Shaker server routable from OpenStack cloud + * Admin-user access to OpenStack API -To install: - 1. ``git clone git://git.openstack.org/stackforge/shaker`` - 2. ``python setup.py install`` - installs the tool and all its python dependencies - 3. ``shaker-image-builder`` - builds image for agent VMs inside OpenStack +Setup +----- -Note: image builder is able to create Nova flavor optimized for the image and this requires -admin user privileges. However if the flavor is already exists then it can be provided via -``flavor-name`` config parameter and the tool executed from an ordinary user. + 1. ``pip install pyshaker`` - installs the tool and all its python dependencies + 2. ``shaker-image-builder`` - builds shaker image and stores it in Glance -How to run ----------- - 1. ``shaker --server-endpoint : --scenario --report `` -During the run the tool deploys topology, spawns instances, distributes -tasks among instances and generates report in HTML format. +Run +--- + + ``shaker --server-endpoint --scenario --report `` + + where: + * ```` - address of machine where Shaker is deployed and any free port + * ```` - the scenario to execute; L2, L3 east-west and L3 north-south already included + * ```` - file to store the report + Links ----- * Launchpad - https://launchpad.net/shaker/ - * Docs - http://shaker-docs.readthedocs.org/ + * Docs - http://pyshaker.readthedocs.org/ * PyPi - https://pypi.python.org/pypi/pyshaker/ diff --git a/doc/source/architecture.rst b/doc/source/architecture.rst new file mode 100644 index 0000000..fcc5159 --- /dev/null +++ b/doc/source/architecture.rst @@ -0,0 +1,50 @@ +============ +Architecture +============ + +Shaker tool consists of server and agent modules. The server is executed by ``shaker`` command +and is responsible for deployment of instances, execution of tests as specified in the scenario, +for results processing and report generation. The agent is light-weight and polls tasks from +the server and replies with the results. Agents have connectivity to the server, but the server does not +(so it is easy to keep agents behind NAT). + +.. image:: images/architecture.svg + + +Under the Hood +^^^^^^^^^^^^^^ + +Scenario execution involves the following steps: + + 1. User launches shaker with the following minimum set of parameters:: + + shaker --server-endpoint --scenario --report + + where: + * host:port - address of the machine where Shaker is installed and port is some arbitrary free port to bind the server to; + * scenario - file name of the scenario (yaml file); + * report - file name where report will be saved. + + 2. Shaker verifies connection to OpenStack. The parameters are taken from set of os-* params or from the env (``openrc``). + + 3. Based on ``accommodation`` parameter the list of agents is generated. + + 4. The topology is deployed with help of Heat. The list of agents is extended with IP addresses and instance names. + + 5. Shaker waits for all agents to connect. Once all agents are alive it means that the Quorum is full and everyone ready to execute the tests. + + 6. Shaker starts tests one by one in order they listed in the scenario. The test definition is converted into the actual command that will be + executed by agent. Shaker schedules the command to be started at the same time on all agents. For networking testing only agents in ``master`` role + are involved. Slave agents are used as back-end for corresponding commands (i.e. they run iperf in server mode). + + 7. Agents send their results to the server. Once all replies are received the test execution meant to be finished. If some agent didn't make it in + dedicated time it is marked as lost. + + 8. Once all tests are executed Shaker can output the raw result in JSON format (option ``--output``). + + 9. Shaker clears the topology with help of Heat. + + 10. Shaker calculates statistics and aggregated charts. If there are any SLA statements they are also evaluated, the result can be stored in subunit format + (option ``--subunit``). + + 11. Shaker generates report in HTML format. diff --git a/doc/source/contributing.rst b/doc/source/contributing.rst new file mode 100644 index 0000000..26c470f --- /dev/null +++ b/doc/source/contributing.rst @@ -0,0 +1,78 @@ +============ +Contributing +============ + +Contribute to Shaker +-------------------- + +Shaker follows standard OpenStack contribution workflow as described at http://docs.openstack.org/infra/manual/developers.html + + +Start working +^^^^^^^^^^^^^ + +1. Clone the repo:: + + $ git clone git://git.openstack.org/stackforge/shaker + +2. From the root of your workspace, check out a new branch to work on:: + + $ git checkout -b + +3. Implement your code + + +Before Commit +^^^^^^^^^^^^^ + +4. Make sure your code works by running the tests:: + + $ tox + +By default tox executes the same set of tests as configured in Jenkins, i.e.: py34 and py27 unit tests, +pep8 style check and documentation build. + +5. If there are any changes in config parameters, also do:: + + $ tox -egenconfig + +This job updates sample config file as well as documentation on CLI utils. + + +Submit Review +^^^^^^^^^^^^^ + +6. Commit the code:: + + $ git commit -a + +Commit message should indicate what the change is, for a bug fix commit it needs to contain reference to Launchpad bug number. + +7. Submit the review:: + + $ git review + +8. If the code is approved with a +2 review, Gerrit will automatically merge your code. + + +Bug Tracking +------------ + +Bugs are tracked at Launchpad: + + https://bugs.launchpad.net/shaker + + +Developer's Guide of OpenStack +------------------------------ + +If you would like to contribute to the development of OpenStack, you must follow the steps in this page: + + http://docs.openstack.org/infra/manual/developers.html + +Once those steps have been completed, changes to OpenStack should be submitted for review via the Gerrit tool, following the workflow documented at: + + http://docs.openstack.org/infra/manual/developers.html#development-workflow + +Pull requests submitted through GitHub will be ignored. + diff --git a/doc/source/examples.rst b/doc/source/examples.rst new file mode 100644 index 0000000..aa0dc3d --- /dev/null +++ b/doc/source/examples.rst @@ -0,0 +1,67 @@ +======== +Examples +======== + +L2 Same Domain +-------------- + +This scenario tests the bandwidth between pairs of instances in the same virtual network (L2 domain). +Each instance is deployed on own compute node. The test increases the load from 1 pair until all +available instances are used. + +.. image:: images/topology_l2.png + +How To Run +^^^^^^^^^^ +.. code:: + + shaker --server-endpoint --scenario --report + +Scenario +^^^^^^^^ + +.. literalinclude:: ../../scenarios/networking/full_l2.yaml + + +L3 East-West +------------ + +This scenario tests the bandwidth between pairs of instances deployed in different virtual networks +plugged into the same router. Each instance is deployed on its own compute node. The test increases the load +from 1 pair pair until all available instances are used. + +.. image:: images/topology_l3_east_west.png + +How To Run +^^^^^^^^^^ +.. code:: + + shaker --server-endpoint --scenario --report + +Scenario +^^^^^^^^ + +.. literalinclude:: ../../scenarios/networking/full_l3_east_west.yaml + + + +L3 North-South +-------------- + +This scenario tests the bandwidth between pairs of instances deployed in different virtual networks. Instances +with master agents are located in one network, instances with slave agents are reached via floating IP. +Each instance is deployed on its own compute node. The test increases the load from 1 pair pair until +all available instances are used. + +.. image:: images/topology_l3_north_south.png + +How To Run +^^^^^^^^^^ +.. code:: + + shaker --server-endpoint --scenario --report + +Scenario +^^^^^^^^ + +.. literalinclude:: ../../scenarios/networking/full_l3_north_south.yaml diff --git a/doc/source/images/accommodation_double_room.svg b/doc/source/images/accommodation_double_room.svg new file mode 100644 index 0000000..4af89c5 --- /dev/null +++ b/doc/source/images/accommodation_double_room.svg @@ -0,0 +1,359 @@ + + + + + + + + + + + + + + + + + + + + + + + image/svg+xml + + + + + + + + accomodation: [pair double_room] + + compute node-1 + + master_1 + + slave_1 + + + + compute node-2 + + master_2 + + slave_2 + + compute node-N + + master_N + + slave_N + + diff --git a/doc/source/images/accommodation_single_room.svg b/doc/source/images/accommodation_single_room.svg new file mode 100644 index 0000000..659c1f3 --- /dev/null +++ b/doc/source/images/accommodation_single_room.svg @@ -0,0 +1,337 @@ + + + + + + + + + + + + + + + + + + + + + + + image/svg+xml + + + + + + + + accomodation: [pair single room] + + compute node-1 + + master_1 + + compute node-2 + + slave_1 + + compute node-N-1 + + master_M + + compute node-N + + slave_M + + + + diff --git a/doc/source/images/architecture.svg b/doc/source/images/architecture.svg new file mode 100644 index 0000000..24ab856 --- /dev/null +++ b/doc/source/images/architecture.svg @@ -0,0 +1,475 @@ + + + + + + + + + + + + + + + + + + + + image/svg+xml + + + + + + + + master_1 + master_1 + + master_2 + master_1 + + slave_1 + master_1 + + slave_2 + + + + shaker + + + + + + + OpenStack + + + network A + network B + + + + virtual router + + + tenant network A + + tenant network B + + shaker agents traffic + + iperf/netperf traffic + + diff --git a/doc/source/images/topology_l2.png b/doc/source/images/topology_l2.png new file mode 100644 index 0000000..bf4569f Binary files /dev/null and b/doc/source/images/topology_l2.png differ diff --git a/doc/source/images/topology_l3_east_west.png b/doc/source/images/topology_l3_east_west.png new file mode 100644 index 0000000..4600cba Binary files /dev/null and b/doc/source/images/topology_l3_east_west.png differ diff --git a/doc/source/images/topology_l3_north_south.png b/doc/source/images/topology_l3_north_south.png new file mode 100644 index 0000000..ce913e7 Binary files /dev/null and b/doc/source/images/topology_l3_north_south.png differ diff --git a/doc/source/index.rst b/doc/source/index.rst index 9981f70..575e4c0 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -1,6 +1,22 @@ Welcome to Shaker! ================== -Shake VMs with our sheer-class tests! +The distributed data-plane testing tool for OpenStack. + +.. toctree:: + :maxdepth: 2 + + installation + usage + examples + tools + architecture + contributing + +.. Indices and tables +.. ================== + +.. * :ref:`genindex` +.. * :ref:`modindex` +.. * :ref:`search` -.. include:: tools.rst diff --git a/doc/source/installation.rst b/doc/source/installation.rst new file mode 100644 index 0000000..0fcc26b --- /dev/null +++ b/doc/source/installation.rst @@ -0,0 +1,54 @@ +============ +Installation +============ + +Installation in Python environment +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Shaker is distributed as Python package and available through PyPi (https://pypi.python.org/pypi/pyshaker/). +It is recommended to be installed inside virtualenv. + +.. code:: + + $ virtualenv venv + $ . venv/bin/activate + $ pip install pyshaker + + +Installation on Ubuntu Cloud Image +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Installation on fresh system requires additional libraries needed by some of dependencies. + +.. code:: + + $ sudo apt-add-repository "deb http://nova.clouds.archive.ubuntu.com/ubuntu/ trusty multiverse" + $ sudo apt-get update + $ sudo apt-get -y install python-dev libzmq-dev + $ wget -O get-pip.py https://bootstrap.pypa.io/get-pip.py && sudo python get-pip.py + $ sudo pip install pbr pyshaker + $ shaker --help + + +Deployment +^^^^^^^^^^ + +.. image:: images/architecture.svg + +Requirements: + + * User with admin privileges OpenStack cloud + * Computer where shaker is executed should be routable from OpenStack instances + * Open port to accept connections from agents running on instances + + +First Run +^^^^^^^^^ + +Build the master image. The process downloads Ubuntu cloud image, installs all necessary packages and stores +snapshot into Glance. This snapshot is used by ``shaker`` as base of instances. + +.. code:: + + $ shaker-image-builder + diff --git a/doc/source/tools.rst b/doc/source/tools.rst index 85d24d8..e8fc6ed 100644 --- a/doc/source/tools.rst +++ b/doc/source/tools.rst @@ -1,28 +1,40 @@ +=================== +CLI Tools Reference +=================== + shaker ------ -The tool runs specified scenario +Executes specified scenario in OpenStack cloud, stores results and generates HTML report. .. literalinclude:: tools/shaker.txt + +shaker-image-builder +-------------------- + +Builds master image in OpenStack cloud. The image is based on Ubuntu cloud image distro and +configured to run ``shaker-agent``. + +.. literalinclude:: tools/shaker-image-builder.txt + shaker-agent ------------ +Client-side process that is run inside pre-configured image. + .. literalinclude:: tools/shaker-agent.txt shaker-report ------------- +Generates report based on raw results stored in JSON format. + .. literalinclude:: tools/shaker-report.txt -shaker-image-builder --------------------- - -The tool that creates master image in OpenStack cloud. - -.. literalinclude:: tools/shaker-image-builder.txt - shaker-cleanup -------------- +Removes master image from OpenStack cloud. + .. literalinclude:: tools/shaker-cleanup.txt diff --git a/doc/source/usage.rst b/doc/source/usage.rst new file mode 100644 index 0000000..730f6f4 --- /dev/null +++ b/doc/source/usage.rst @@ -0,0 +1,82 @@ +===== +Usage +===== + +Configuration +------------- + +The connection to OpenStack is configured using standard ``openrc`` file. (Refer to +http://docs.openstack.org/cli-reference/content/cli_openrc.html on how to retrieve it) + +The config can be passed to Shaker rather by sourcing into system env ``source openrc`` +or via set of CLI parameters ``--os-tenant-name``, ``--os-username``, ``--os-password``, +``--os-auth-url`` and ``--os-region-name``. Note that Shaker requires a user with admin privileges. + + +Scenario Explained +------------------ + +Shaker scenario is file in YAML format. It describes how agents are deployed +(at OpenStack instances or statically) and sequence of tests to execute. When agents +are deployed at OpenStack instances a reference to Heat template is provided. + +.. code:: + + description: + This scenario launches pairs of VMs in the same private network. Every VM is + hosted on a separate compute node. + + deployment: + template: l2.hot + accommodation: [pair, single_room] + + execution: + size: quadratic_progression + tests: + - + title: Iperf TCP + class: iperf_graph + time: 60 + +Deployment +^^^^^^^^^^ + +By default Shaker spawns instances on every available compute node. The distribution +of instances is configured by parameter ``accommodation``. There are several instructions +that allow control the scheduling precisely: + + * ``pair`` - instances are grouped in pairs, meaning that one can be used as source of traffic and the other as a consumer (needed for networking tests) + * ``single_room`` - 1 instance per compute node + * ``double_room`` - 2 instances per compute node + * ``density: N`` - the multiplier for number of instances per compute node + * ``compute_nodes: N`` - how many compute nodes should be used (by default Shaker use all of them) + +Examples: + +.. image:: images/accommodation_single_room.svg + +.. image:: images/accommodation_double_room.svg + +As result of deployment the set of agents is produced. For networking testing this set contains +agents in ``master`` and ``slave`` roles. Master agents are controlled by ``shaker`` tool and execute commands. +Slaves are used as back-ends and do not receive any commands directly. + +Execution +^^^^^^^^^ + +The execution part of scenario contains a list of tests that are executed one by one. By default Shaker runs the test +simultaneously on all available agents. The level of concurrency can be controlled by option ``size``. There are +3 values available: + + * ``size: linear_progression`` - the execution starts with 1 agent and increases by 1 until all agents are involved; + * ``size: quadratic_progression`` - the execution starts with 1 agent (or 1 pair) and doubles until all agents are involved. + +Tests are executed in order of definition. The exact action is defined by option ``class``, additional attributes are provided +by respective parameters. The following classes are available: + + * ``iperf_graph`` - runs ``iperf`` tool and shows chart and statistics + * ``netperf_wrapper`` - runs ``netperf-wrapper`` and shows chart and statistics + * ``iperf`` - runs ``iperf`` tool and shows plain output + * ``netperf`` - runs ``netpers`` tool and shows plain output + * ``shell`` - runs any shell command or process and shows plain output + diff --git a/shaker/resources/report_template.html b/shaker/resources/report_template.html index fbd7078..d0f0af4 100644 --- a/shaker/resources/report_template.html +++ b/shaker/resources/report_template.html @@ -4,7 +4,7 @@ - Shaker | The Report + Shaker | The distributed data-plane testing tool for OpenStack