Add Support for DPDK Bonding

This patch will allow you to setup DPDK bonds within a given OVS
bridge using DPDK-accelerated interfaces. A new provider network
key, network_bond_interfaces, and related keys, have been
introduced.

Co-Authored-By: James Denton <james.denton@rackspace.com>

Change-Id: I3fc2846a0c2b6579e4cdb54c3a7c36620700cd44

doc/source/app-openvswitch-dpdk.rst

@@ -27,7 +27,7 @@ We recommend that you read the following documents before proceeding:
 * Open vSwitch with DPDK datapath:
   `<https://docs.openstack.org/neutron/latest/admin/config-ovs-dpdk.html>`_
 * Getting the best performance from DPDK:
-  `<https://doc.dpdk.org/guides-16.04/linux_gsg/nic_perf_intel_platform.html>`_
+  `<https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html>`_
 * OpenStack documentation on hugepages:
   `<https://docs.openstack.org/nova/latest/admin/huge-pages.html>`_
 
@@ -156,11 +156,13 @@ The remaining cores can be reserved for virtual machine instances.
 
 In this example, the breakdown would resemble the following:
 
+```
 | Reserved Cores         | Purpose               | node0     | node1 |
 | ---------------------- | --------------------- | --------- | ----- |
 | 0,8,16,24              | Host Operating System | 0,16      | 8,24  |
 | 1,9,17,25              | DPDK PMDs             | 1,17      | 9,25  |
 | 2-7,18-23              | Virtual Machines      | 2-7,18-23 | N/A   |
+```
 
 The variables are overrides used to define this configuration are discussed
 in the following sections.
@@ -280,9 +282,11 @@ to a hexidecimal mask and defined using the ``ovs_dpdk_lcore_mask`` override.
 To convert to a hex mask you must first establish the binary mask of chosen
 cores using the following table:
 
+```
 | 31 | 30 | . | 24 | 23 | . | 17 | 16 | 15 | . | 9  | 8  | 7  | . | 1  | 0  |
 | -- | -- | - | -- | -- | - | -- | -- | -- | - | -- | -- | -- | - | -- | -- |
 | 0  | 0  | . | 1  | 0  | . | 0  | 1  | 0  | . | 0  | 1  | 0  | . | 0  | 1  |
+```
 
 The ellipses represent cores not shown. The binary mask for cores 0,8,16,24
 can be determined in the following way:
@@ -302,9 +306,11 @@ file or ``openstack_user_config.yml``:
 The mask for cores 1,9,17,25 reserved for DPDK PMDs can be determined in
 a similar fashion. The table would resemble the following:
 
+```
 | 31 | 30 | . | 25 | 24 | . | 17 | 16 | 15 | . | 9  | 8  | 7  | . | 1  | 0  |
 | -- | -- | - | -- | -- | - | -- | -- | -- | - | -- | -- | -- | - | -- | -- |
 | 0  | 0  | . | 1  | 0  | . | 1  | 0  | 0  | . | 1  | 0  | 0  | . | 1  | 0  |
+```
 
 The ellipses represent cores not shown. The binary mask for cores 1,9,17,254
 can be determined in the following way:
@@ -401,6 +407,12 @@ Specify provider network definitions in your
 ``/etc/openstack_deploy/openstack_user_config.yml`` that define one or more
 Neutron provider bridges and related configuration:
 
+.. note::
+
+  Bridges specified here will be created automatically. If *network_interface*
+  is defined, the interface will be placed into the bridge automatically
+  as a *DPDK-accelerated* interface.
+
 .. code-block:: yaml
 
   - network:
@@ -409,13 +421,55 @@ Neutron provider bridges and related configuration:
       type: "vlan"
       range: "101:200,301:400"
       net_name: "physnet1"
+      network_interface: "eno49"
       group_binds:
         - neutron_openvswitch_agent
 
-.. note::
+A *DPDK-accelerated* **bond** interface can be created by specifying a list
+of member interfaces using `network_bond_interfaces`. The bond port will
+be created automatically and added to the respective bridge in OVS:
 
-  A single DPDK interface can be connected to an OVS provider bridge, and
+.. code-block:: yaml
-  must be done using the ``ovs-vsctl`` command as a post-installation step.
+
+  - network:
+      container_bridge: "br-provider"
+      container_type: "veth"
+      type: "vlan"
+      range: "101:200,301:400"
+      net_name: "physnet1"
+      network_bond_interfaces:
+        - "0000:04:00.0"
+        - "0000:04:00.1"
+      group_binds:
+        - neutron_openvswitch_agent
+
+Additional OVS bond parameters can be specified using the following keys:
+
+* bond_mode (Default: active-backup)
+* lacp (Default: off)
+* bond_downdelay (Default: 100)
+* bond_updelay (Default: 100)
+
+.. code-block:: yaml
+
+  - network:
+      container_bridge: "br-provider"
+      container_type: "veth"
+      type: "vlan"
+      range: "101:200,301:400"
+      net_name: "physnet1"
+      network_bond_interfaces:
+        - "0000:04:00.0"
+        - "0000:04:00.1"
+      bond_mode: balance-tcp
+      lacp: active
+      bond_downdelay: 200
+      bond_updelay: 200
+      group_binds:
+        - neutron_openvswitch_agent
+
+For more information on possible values, visit:
+`<https://docs.ansible.com/ansible/latest/collections/openvswitch/openvswitch/openvswitch_bond_module.html>`_
 
 Set the following user variables in your
 ``/etc/openstack_deploy/user_variables.yml`` to enable the Open vSwitch driver
@@ -430,7 +484,9 @@ and DPDK support:
   ovs_dpdk_support: True
 
   # Add these overrides or set on per-host basis in openstack_user_config.yml
-  ovs_dpdk_pci_addresses: "0000:03:00.0"
+  ovs_dpdk_pci_addresses:
+    - "0000:04:00.0"
+    - "0000:04:00.1"
   ovs_dpdk_lcore_mask: 1010101
   ovs_dpdk_pmd_cpu_mask: 2020202
   ovs_dpdk_socket_mem: "1024,1024"
@@ -442,20 +498,29 @@ and DPDK support:
 Post-installation
 ~~~~~~~~~~~~~~~~~
 
-Once the playbooks have been run and OVS/DPDK has been configured, it will be
+Once the playbooks have been run and OVS/DPDK has been configured, it may be
 necessary to add a physical interface to the provider bridge before networking
-can be fully established.
+can be fully established *if* `network_interface` or `network_bond_interfaces`
+have not been defined.
 
 On compute nodes, the following command can be used to attach a NIC port
-``0000:03:00.0`` to the provider bridge ``br-provider``:
+``0000:04:00.0`` to the provider bridge ``br-provider``:
 
 .. code-block:: console
 
-  ovs-vsctl add-port br-provider 0000:03:00.0 -- set Interface 0000:03:00.0 type=dpdk options:dpdk-devargs=0000:03:00.0
+  ovs-vsctl add-port br-provider 0000:04:00.0 -- set interface 0000:04:00.0 type=dpdk options:dpdk-devargs=0000:04:00.0
 
-The command can be adjusted according to your configuration.
+Additionally, it may be necessary to make post-installation adjustments to
+interface queues or other parameters to avoid errors within Open vSwitch:
+
+.. code-block:: console
+
+  ovs-vsctl set interface 0000:04:00.0 options:n_txq=5
+  ovs-vsctl set interface 0000:04:00.0 options:n_rxq=5
+
+The command(s) can be adjusted according to your configuration.
 
 .. warning::
 
-  Adding multiple ports to the bridge may result in bridging loops unless
+  Adding multiple ports to a bridge may result in bridging loops unless
-  bonding is configured. DPDK bonding is outside the scope of this guide.
+  bonding is configured.

library/provider_networks

@@ -23,7 +23,7 @@ from ansible.module_utils.basic import AnsibleModule
 DOCUMENTATION = """
 ---
 module: provider_networks
-version_added: "1.8.6"
+version_added: "1.8.7"
 short_description:
     - Parse a list of networks and return data that Ansible can use
 description:
@@ -116,6 +116,22 @@ EXAMPLES = """
 #     group_binds:
 #       - neutron_openvswitch_agent
 # - network:
+#     container_bridge: "br-provider"
+#     container_type: "veth"
+#     container_interface: "eth11"
+#     network_bond_interfaces:
+#       - "0000:02:00.0"
+#       - "0000:02:00.1"
+#     bond_mode: balance-tcp
+#     bond_updelay: 100
+#     bond_downdelay: 100
+#     lacp: active
+#     type: "vlan"
+#     range: "1:1, 101:101"
+#     net_name: "physnet1"
+#     group_binds:
+#       - neutron_openvswitch_agent
+# - network:
 #     container_bridge: "br-storage"
 #     container_type: "veth"
 #     container_interface: "eth2"
@@ -157,6 +173,14 @@ EXAMPLES = """
 #             "flat:brx-eth12",
 #             "vlan:brx-eth11"
 #         ],
+#         "network_bond_interfaces_mappings": [{
+#             'bridge': 'br-provider',
+#             'interfaces': ['0000:02:00.0', '0000:02:00.1'],
+#             'bond_mode': 'balance-tcp',
+#             'lacp': 'active',
+#             'bond_updelay': 100,
+#             'bond_downdelay': 100
+#         }],
 #         "network_sriov_mappings": "physnet1:p1p1,physnet1:p1p2",
 #         "network_sriov_mappings_list": [
 #             "physnet1:p1p1"
@@ -202,6 +226,7 @@ class ProviderNetworksParsing(object):
         self.network_types = list()
         self.network_sriov_mappings = list()
         self.network_interface_mappings = list()
+        self.network_bond_interfaces_mappings = list()
 
     def load_networks(self, provider_networks, is_metal=False,
                       bind_prefix=None, group_names=None):
@@ -299,6 +324,29 @@ class ProviderNetworksParsing(object):
                                 )
                             )
 
+                        # Builds a list of provider bridge to physical
+                        # interface (bond member) mappings and is used
+                        # when constructing OVS bonds
+                        if 'network_bond_interfaces' in net['network']:
+                            self.network_bond_interfaces_mappings.append({
+                                'bridge': net['network'][
+                                    'container_bridge'
+                                ],
+                                'interfaces': net['network'][
+                                    'network_bond_interfaces'
+                                ],
+                                'bond_mode': net['network'].get(
+                                    'bond_mode', 'active-backup'
+                                ),
+                                'lacp': net['network'].get('lacp', 'off'),
+                                'bond_updelay': net['network'].get(
+                                    'bond_updelay', 100
+                                ),
+                                'bond_downdelay': net['network'].get(
+                                    'bond_downdelay', 100
+                                )
+                            })
+
                         # SR-IOV interface mappings
                         if 'sriov_host_interfaces' in net['network']:
                             host_interfaces = \
@@ -367,7 +415,12 @@ def main():
             'network_interface_mappings': ','.join(
                 pnp.network_interface_mappings
             ),
-            'network_interface_mappings_list': pnp.network_interface_mappings
+            'network_interface_mappings_list': pnp.network_interface_mappings,
+            'network_bond_interfaces_mappings': ','.join(
+                map(str, pnp.network_bond_interfaces_mappings)
+            ),
+            'network_bond_interfaces_mappings_list':
+                pnp.network_bond_interfaces_mappings
         }
 
         module.exit_json(changed=True, **resp)

releasenotes/notes/neutron-dpdk-bonds-4dd98fc0b341ebfb.yaml

@@ -0,0 +1,10 @@
+---
+features:
+  - |
+    The ``provider_networks`` library has been updated to support the
+    definition of bond member interfaces that can automatically be added as
+    bond ports to OVS provider bridges setup during a deployment. This
+    feature is currently limited to DPDK-based deployments. To activate this
+    feature, add the ``network_bond_interfaces`` key to the respective provider
+    network definition in ``openstack_user_config.yml``. For more information,
+    refer to the latest Open vSwitch w/ DPDK deployment guide.

tasks/providers/setup_ovs_dpdk.yml

@@ -28,6 +28,7 @@
     dest: "/etc/dpdk/interfaces"
     owner: "root"
     group: "root"
+    mode: "0640"
   when:
     - neutron_services['neutron-openvswitch-agent']['group'] in group_names
     - '"nova_compute" in group_names'
@@ -38,6 +39,7 @@
     dest: "/etc/dpdk/dpdk.conf"
     owner: "root"
     group: "root"
+    mode: "0640"
   when:
     - neutron_services['neutron-openvswitch-agent']['group'] in group_names
     - '"nova_compute" in group_names'
@@ -111,7 +113,7 @@
     - neutron_plugin_type in ['ml2.ovs', 'ml2.ovs.dvr']
     - neutron_provider_networks.network_mappings is defined
 
-# (todo) Loop thru ints or build a bond with ints. TBD.
+# Adds a single host interface to an OVS bridge
 - name: Add ports to Network Provider Bridges
   openvswitch_port:
     bridge: "{{ interface_mapping.split(':')[0] }}"
@@ -125,3 +127,23 @@
     - neutron_services['neutron-openvswitch-agent']['group'] in group_names
     - neutron_plugin_type in ['ml2.ovs', 'ml2.ovs.dvr']
     - neutron_provider_networks.network_interface_mappings is defined and (neutron_provider_networks.network_interface_mappings|length > 0)
+
+# Adds a DPDK-accelerated bond interface to an OVS bridge
+- name: Add Bonds to Network Provider Bridges
+  openvswitch.openvswitch.openvswitch_bond:
+    bridge: "{{ bond_interfaces_mapping.bridge }}"
+    port: "{{ bond_interfaces_mapping.bridge }}-dpdkbond"
+    interfaces: "{{ bond_interfaces_mapping.interfaces }}"
+    bond_mode: "{{ bond_interfaces_mapping.bond_mode | default('active-backup') }}"
+    lacp: "{{ bond_interfaces_mapping.lacp | default('off') }}"
+    bond_updelay: "{{ bond_interfaces_mapping.bond_updelay | default(100) }}"
+    bond_downdelay: "{{ bond_interfaces_mapping.bond_downdelay | default(100) }}"
+    set: "{% for interface in bond_interfaces_mapping.interfaces %}interface {{ interface }} type=dpdk options:dpdk-devargs='{{ interface }}'{% if not loop.last %},{% endif %}{% endfor %}"
+    state: present
+  with_items: "{{ neutron_provider_networks.network_bond_interfaces_mappings }}"
+  loop_control:
+    loop_var: bond_interfaces_mapping
+  when:
+    - neutron_services['neutron-openvswitch-agent']['group'] in group_names
+    - neutron_plugin_type in ['ml2.ovs', 'ml2.ovs.dvr']
+    - neutron_provider_networks.network_bond_interfaces_mappings is defined and (neutron_provider_networks.network_bond_interfaces_mappings|length > 0)