performa/shaker
Code Issues Proposed changes
shaker/doc/source/catalog.rst
Oded Le'Sage abe9fbd877 Enhance Shaker to provide the ability to define support templates 
Issue: At AT&T we have large complex test stacks that make putting
everything into a single heat template and environment file very
cumbersome. Large monolithic templates make it harder to debug failures,
maintain, extend, and organize these tests. In order to solve this issue
we have enhanced Shaker to support specifying support templates with
environment files.

This commit enhances Shaker to add the ability to define
support_templates with env_files in test definitions.

Support templates spin up "support type" resources before the actual
test stack is spun up. This could range from networks, to volumes, to
anything Heat can create. The support resources do not have any reliance
on resources created in the test stack, they set up a "foundation" for
the test stack. The test stack can then reference these resources by
name. (e.g. assume they exist by the time the test stack is spun up)

While the example provided with this commit is simple, and the support
networks that get created are not directly used in the test, it
shows the basic principles of how support templates work.

As a real-world example and to give an idea of the complexity this
enhancement is trying to solve, we have a test definition that looks
like this:

  support_templates:
  -
    Base:
      template: templates/module_1_base.yaml
      env_file: env/module_1_base.env
  -
   SI_L2:
      template: templates/module_2_si_l2.yaml
      env_file: env/module_2_si_l2.env
  -
   SI_L3:
      template: templates/module_3_si_l3.yaml
      env_file: env/module_3_si_l3.env

  template: templates/module_4_master_servant.yaml
  env_file: env/module_4_master_servant.env

The first support stack (module_1) sets up some "base" network
resources. This stack provides some network resources used by the SI_L2
and SI_L3 support stacks.

SI_L2 is a support stack with 2 VMs that do Contrail service chaining
on an L2 network.

SI_L3 is a support stack with 2 VMs that do Contrail service chaining
on an L3 network.

Then the test stack (module 4) gets spun up on N amount of computes and
runs traffic across the SI_L2 and SI_L3 service chained networks.

After the test run all stacks are cleaned up

Using the concept of support stacks allows us to beter organize and
maintain our complex tests and allows for faster debugging due to the
"layered" nature of the setup.

Support templates also allow us to spin up more Shaker test threads
that use the same support templates simultaneously to better simulate
real-world network traffic. It also reduces the set up time of certain
tests we have since the support stacks already exist.

This enhancement does not alter existing Shaker functionality and
is fully backwards compatible.

Change-Id: Ife51bc55874c6ec4faac221bab8f9f0eea175fdc
2018-11-05 11:12:14 -06:00

28 KiB
Raw Blame History

Scenario Catalog

Scenarios

OpenStack instances metadata query

In this scenario Shaker launches ten instances on a single compute node and asks instances to retrieve the metadata. The scenario is used to load metadata processes.

To use this scenario specify parameter --scenario misc/instance_metadata. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/misc/instance_metadata.yaml

OpenStack L2 Cross-AZ

In this scenario Shaker launches pairs of instances in the same tenant network. Every instance is hosted on a separate compute node, all available compute nodes are utilized. The master and slave instances are in different availability zones. The scenario is used to test throughput between nova and vcenter zones. The traffic goes within the tenant network (L2 domain).

To use this scenario specify parameter --scenario openstack/cross_az/full_l2. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/cross_az/full_l2.yaml

OpenStack L3 East-West Cross-AZ

In this scenario Shaker launches pairs of instances, each instance on its own compute node. All available compute nodes are utilized. Instances are connected to one of 2 tenant networks, which plugged into single router. The traffic goes from one network to the other (L3 east-west). The master and slave instances are in different availability zones. The scenario is used to test throughput between nova and vcenter zones.

To use this scenario specify parameter --scenario openstack/cross_az/full_l3_east_west. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/cross_az/full_l3_east_west.yaml

OpenStack L3 North-South Cross-AZ

In this scenario Shaker launches pairs of instances on different compute nodes. All available compute nodes are utilized. Instances are in different networks connected to different routers, master accesses slave by floating ip. The traffic goes from one network via external network to the other network. The master and slave instances are in different availability zones. The scenario is used to test throughput between nova and vcenter zones.

To use this scenario specify parameter --scenario openstack/cross_az/full_l3_north_south. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/cross_az/full_l3_north_south.yaml

OpenStack L2 Cross-AZ Performance

In this scenario Shaker launches 1 pair of instances in the same tenant network. Each instance is hosted on a separate compute node. The master and slave instances are in different availability zones. The scenario is used to test throughput between nova and vcenter zones.

To use this scenario specify parameter --scenario openstack/cross_az/perf_l2. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/cross_az/perf_l2.yaml

OpenStack L3 East-West Cross-AZ Performance

In this scenario Shaker launches 1 pair of instances, each instance on its own compute node. Instances are connected to one of 2 tenant networks, which plugged into single router. The traffic goes from one network to the other (L3 east-west). The master and slave instances are in different availability zones. The scenario is used to test throughput between nova and vcenter zones.

To use this scenario specify parameter --scenario openstack/cross_az/perf_l3_east_west. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/cross_az/perf_l3_east_west.yaml

OpenStack L3 North-South Cross-AZ Performance

In this scenario Shaker launches 1 pair of instances on different compute nodes. Instances are in different networks connected to different routers, master accesses slave by floating ip. The traffic goes from one network via external network to the other network. The master and slave instances are in different availability zones. The scenario is used to test throughput between nova and vcenter zones.

To use this scenario specify parameter --scenario openstack/cross_az/perf_l3_north_south. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/cross_az/perf_l3_north_south.yaml

OpenStack L2 Cross-AZ UDP

In this scenario Shaker launches pairs of instances in the same tenant network. Every instance is hosted on a separate compute node. The load is generated by UDP traffic. The master and slave instances are in different availability zones. The scenario is used to test throughput between nova and vcenter zones.

To use this scenario specify parameter --scenario openstack/cross_az/udp_l2. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/cross_az/udp_l2.yaml

OpenStack L2 Cross-AZ UDP Jumbo

In this scenario Shaker launches pairs of instances in the same tenant network. Every instance is hosted on a separate compute node. The load is generated by UDP traffic and jumbo packets. The master and slave instances are in different availability zones. The scenario is used to test throughput between nova and vcenter zones.

To use this scenario specify parameter --scenario openstack/cross_az/udp_l2_mss8950. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/cross_az/udp_l2_mss8950.yaml

OpenStack L3 East-West Cross-AZ UDP

In this scenario Shaker launches pairs of instances, each instance on its own compute node. Instances are connected to one of 2 tenant networks, which plugged into single router. The traffic goes from one network to the other (L3 east-west). The load is generated by UDP traffic. The master and slave instances are in different availability zones. The scenario is used to test throughput between nova and vcenter zones.

To use this scenario specify parameter --scenario openstack/cross_az/udp_l3_east_west. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/cross_az/udp_l3_east_west.yaml

OpenStack L2 Dense

In this scenario Shaker launches several pairs of instances on a single compute node. Instances are plugged into the same tenant network. The traffic goes within the tenant network (L2 domain).

To use this scenario specify parameter --scenario openstack/dense_l2. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/dense_l2.yaml

OpenStack L3 East-West Dense

In this scenario Shaker launches pairs of instances on the same compute node. Instances are connected to different tenant networks connected to one router. The traffic goes from one network to the other (L3 east-west).

To use this scenario specify parameter --scenario openstack/dense_l3_east_west. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/dense_l3_east_west.yaml

OpenStack L3 North-South Dense

In this scenario Shaker launches pairs of instances on the same compute node. Instances are connected to different tenant networks, each connected to own router. Instances in one of networks have floating IPs. The traffic goes from one network via external network to the other network.

To use this scenario specify parameter --scenario openstack/dense_l3_north_south. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/dense_l3_north_south.yaml

OpenStack L3 North-South Dense to external target

In this scenario Shaker launches instances on one compute node in a tenant network connected to external network. The traffic is sent to and from external host. The host name needs to be provided as command-line parameter, e.g. --matrix "{host: 172.10.1.2}".

To use this scenario specify parameter --scenario openstack/external/dense_l3_north_south_no_fip. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/external/dense_l3_north_south_no_fip.yaml

OpenStack L3 North-South Dense to external target with floating IP

In this scenario Shaker launches instances on one compute node in a tenant network connected to external network. All instances have floating IPs. The traffic is sent to and from external host. The host name needs to be provided as command-line parameter, e.g. --matrix "{host: 172.10.1.2}".

To use this scenario specify parameter --scenario openstack/external/dense_l3_north_south_with_fip. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/external/dense_l3_north_south_with_fip.yaml

OpenStack L3 North-South to external target

In this scenario Shaker launches instances in a tenant network connected to external network. Every instance is hosted on dedicated compute node. All available compute nodes are utilized. The traffic is sent to and from external host (L3 north-south). The host name needs to be provided as command-line parameter, e.g. --matrix "{host: 172.10.1.2}".

To use this scenario specify parameter --scenario openstack/external/full_l3_north_south_no_fip. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/external/full_l3_north_south_no_fip.yaml

OpenStack L3 North-South to external target with floating IP

In this scenario Shaker launches instances in a tenant network connected to external network. Every instance is hosted on dedicated compute node. All available compute nodes are utilized. All instances have floating IPs. The traffic is sent to and from external host (L3 north-south). The host name needs to be provided as command-line parameter, e.g. --matrix "{host: 172.10.1.2}".

To use this scenario specify parameter --scenario openstack/external/full_l3_north_south_with_fip. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/external/full_l3_north_south_with_fip.yaml

OpenStack L3 North-South Performance to external target

In this scenario Shaker launches instance in a tenant network connected to external network. The traffic is sent to and from external host. By default one of public iperf3 servers is used, to override this the target host can be provided as command-line parameter, e.g. --matrix "{host: 172.10.1.2}".

To use this scenario specify parameter --scenario openstack/external/perf_l3_north_south_no_fip. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/external/perf_l3_north_south_no_fip.yaml

OpenStack L3 North-South performance to external target with floating IP

In this scenario Shaker launches instance in a tenant network connected to external network. The instance has floating IP. The traffic is sent to and from external host. By default one of public iperf3 servers is used, to override this the target host can be provided as command-line parameter, e.g. --matrix "{host: 172.10.1.2}".

To use this scenario specify parameter --scenario openstack/external/perf_l3_north_south_with_fip. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/external/perf_l3_north_south_with_fip.yaml

OpenStack L2

In this scenario Shaker launches pairs of instances in the same tenant network. Every instance is hosted on a separate compute node, all available compute nodes are utilized. The traffic goes within the tenant network (L2 domain).

To use this scenario specify parameter --scenario openstack/full_l2. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/full_l2.yaml

OpenStack L3 East-West

In this scenario Shaker launches pairs of instances, each instance on its own compute node. All available compute nodes are utilized. Instances are connected to one of 2 tenant networks, which plugged into single router. The traffic goes from one network to the other (L3 east-west).

To use this scenario specify parameter --scenario openstack/full_l3_east_west. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/full_l3_east_west.yaml

OpenStack L3 North-South

In this scenario Shaker launches pairs of instances on different compute nodes. All available compute nodes are utilized. Instances are in different networks connected to different routers, master accesses slave by floating ip. The traffic goes from one network via external network to the other network.

To use this scenario specify parameter --scenario openstack/full_l3_north_south. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/full_l3_north_south.yaml

OpenStack L2 Performance

In this scenario Shaker launches 1 pair of instances in the same tenant network. Each instance is hosted on a separate compute node. The traffic goes within the tenant network (L2 domain).

To use this scenario specify parameter --scenario openstack/perf_l2. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/perf_l2.yaml

OpenStack L3 East-West Performance

In this scenario Shaker launches 1 pair of instances, each instance on its own compute node. Instances are connected to one of 2 tenant networks, which plugged into single router. The traffic goes from one network to the other (L3 east-west).

To use this scenario specify parameter --scenario openstack/perf_l3_east_west. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/perf_l3_east_west.yaml

OpenStack L3 North-South Performance

In this scenario Shaker launches 1 pair of instances on different compute nodes. Instances are in different networks connected to different routers, master accesses slave by floating ip. The traffic goes from one network via external network to the other network.

To use this scenario specify parameter --scenario openstack/perf_l3_north_south. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/perf_l3_north_south.yaml

OpenStack L2 QoS Performance

In this scenario Shaker launches 1 pair of instances in the same tenant network. Each instance is hosted on a separate compute node. The traffic goes within the tenant network (L2 domain). Neutron QoS feature is used to limit traffic throughput to 10 Mbit/s.

To use this scenario specify parameter --scenario openstack/qos/perf_l2. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/qos/perf_l2.yaml

OpenStack L2 UDP

In this scenario Shaker launches pairs of instances in the same tenant network. Every instance is hosted on a separate compute node. The traffic goes within the tenant network (L2 domain). The load is generated by UDP traffic.

To use this scenario specify parameter --scenario openstack/udp_l2. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/udp_l2.yaml

OpenStack L3 East-West UDP

In this scenario Shaker launches pairs of instances, each instance on its own compute node. Instances are connected to one of 2 tenant networks, which plugged into single router. The traffic goes from one network to the other (L3 east-west). The load is generated by UDP traffic.

To use this scenario specify parameter --scenario openstack/udp_l3_east_west. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/udp_l3_east_west.yaml

OpenStack L3 North-South UDP

In this scenario Shaker launches pairs of instances on different compute nodes. Instances are in different networks connected to different routers, master accesses slave by floating ip. The traffic goes from one network via external network to the other network. The load is generated by UDP traffic.

To use this scenario specify parameter --scenario openstack/udp_l3_north_south. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/udp_l3_north_south.yaml

Ping

This scenario uses ping to measure the latency between the local host and the remote. The remote host can be provided via command-line, it defaults to 8.8.8.8. The scenario verifies SLA and expects the latency to be at most 30ms. The destination host can be overridden by command-line parameter, e.g. --matrix "{host: 172.10.1.2}".

To use this scenario specify parameter --scenario spot/ping. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/spot/ping.yaml

TCP bandwidth

This scenario uses iperf3 to measure TCP throughput between local host and ping.online.net (or against hosts provided via CLI). SLA check is verified and expects the speed to be at least 90Mbit and at most 20 retransmitts. The destination host can be overridden by command-line parameter, e.g. --matrix "{host: 172.10.1.2}".

To use this scenario specify parameter --scenario spot/tcp. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/spot/tcp.yaml

UDP bandwidth

This scenario uses iperf3 to measure UDP throughput between local host and ping.online.net (or against hosts provided via CLI). SLA check is verified and requires at least 10 000 packets per second. The destination host can be overridden by command-line parameter, e.g. --matrix "{host: 172.10.1.2}".

To use this scenario specify parameter --scenario spot/udp. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/spot/udp.yaml

Sample TCP Test with Environment File

This test definition demonstrates the use of an environment file. In this scenario Shaker launches pairs of instances in the same tenant network. Every instance is hosted on a separate compute node, 1 compute node is utilized. The traffic goes within the tenant network (L2 domain)

To use this scenario specify parameter --scenario test/sample_with_env. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/test/sample_with_env.yaml

Sample TCP Test with Support Stacks

This test definition demonstrates the use of support stacks In this scenario Shaker launches pairs of instances in the same tenant network. Each test VM is also connected to a previously launched support network. The support neworks are part of their own support heat stack. Every instance is hosted on a separate compute node, 1 compute node is utilized. The traffic goes within the tenant network (L2 domain)

To use this scenario specify parameter --scenario test/sample_with_support_stacks. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/test/sample_with_support_stacks.yaml

Static agents

In this scenario Shaker runs tests in spot mode. The scenario can be used for Shaker integration testing.

To use this scenario specify parameter --scenario test/spot. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/test/spot.yaml

Static agents

In this scenario Shaker runs tests on pre-deployed static agents. The scenario can be used for Shaker integration testing.

To use this scenario specify parameter --scenario test/static_agent. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/test/static_agent.yaml

Paired static agents

In this scenario Shaker runs tests on pre-deployed pair of static agents. The scenario can be used for Shaker integration testing.

To use this scenario specify parameter --scenario test/static_agents_pair. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/test/static_agents_pair.yaml

Heat Templates

misc/instance_metadata

Heat template creates a new Neutron network, a router to the external network, plugs instances into this network and assigns floating ips

Template source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/misc/instance_metadata.hot

openstack/cross_az/l2

This Heat template creates a new Neutron network, a router to the external network and plugs instances into this new network. All instances are located in the same L2 domain.

Template source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/cross_az/l2.hot

openstack/cross_az/l3_east_west

This Heat template creates a pair of networks plugged into the same router. Master instances and slave instances are connected into different networks.

Template source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/cross_az/l3_east_west.hot

openstack/cross_az/l3_north_south

This Heat template creates a new Neutron network plus a north_router to the external network. The template also assigns floating IP addresses to each instance so they are routable from the external network.

Template source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/cross_az/l3_north_south.hot

openstack/external/l3_north_south_no_fip

This Heat template creates a new Neutron network plugged into a router connected to the external network, and boots an instance in that network.

Template source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/external/l3_north_south_no_fip.hot

openstack/external/l3_north_south_with_fip

This Heat template creates a new Neutron network plugged into a router connected to the external network, and boots an instance in that network. The instance has floating IP.

Template source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/external/l3_north_south_with_fip.hot

openstack/l2

This Heat template creates a new Neutron network, a router to the external network and plugs instances into this new network. All instances are located in the same L2 domain.

Template source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/l2.hot

openstack/l3_east_west

This Heat template creates a pair of networks plugged into the same router. Master instances and slave instances are connected into different networks.

Template source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/l3_east_west.hot

openstack/l3_north_south

This Heat template creates a new Neutron network plus a north_router to the external network. The template also assigns floating IP addresses to each instance so they are routable from the external network.

Template source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/l3_north_south.hot

openstack/qos/l2_qos

This Heat template creates a new Neutron network, a router to the external network and plugs instances into this new network. All instances are located in the same L2 domain.

Template source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/qos/l2_qos.hot

test/l2_with_env

This Heat template creates a new Neutron network, a router to the external network and plugs instances into this new network. All instances are located in the same L2 domain.

Template source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/test/l2_with_env.hot

test/templates/l2_with_support

This Heat template creates a new Neutron network, a router to the external network and plugs instances into this new network. All instances are located in the same L2 domain. The VMs are also connected to support networks that should exist before this template is spun up.

Template source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/test/templates/l2_with_support.hot

test/templates/support_network

This Heat template creates a new Neutron network. This is used to demonstrate a support stack in Shaker.

Template source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/test/templates/support_network.hot