78c47acdea
Add new appendix to detail how to use port hardware offloading with Mellanox ConnectX-5 based NICs. Change-Id: I25c702236b01593d903f1da88dfe7a32b62b7a4b
127 lines
4.6 KiB
ReStructuredText
127 lines
4.6 KiB
ReStructuredText
================================
|
|
Appendix S: NIC hardware offload
|
|
================================
|
|
|
|
Overview
|
|
--------
|
|
|
|
As of the 20.05 release, the OpenStack charms support configuration of Open vSwitch
|
|
hardware offloading with Mellanox ConnectX-5 NICs. Hardware offloading can be used
|
|
to accelerate VLAN and VXLAN networking using the capabilities of the underlying
|
|
network card to achieve much higher performance than with virtio based VM ports.
|
|
|
|
See the Neutron documentation on `OVS hardware offload`_ for background information.
|
|
|
|
.. warning::
|
|
|
|
Hardware offloading cannot be used with either SR-IOV or DPDK networking support
|
|
as provided by the OpenStack charms.
|
|
|
|
Prerequisites
|
|
-------------
|
|
|
|
- Ubuntu 18.04 LTS or later
|
|
- Linux kernel >= 5.3
|
|
- Open vSwitch >= 2.11
|
|
- OpenStack Stein or later
|
|
- Mellanox ConnectX-5 NICs using recent firmware (>= 16.26.4012)
|
|
|
|
.. note::
|
|
|
|
Hardware offload does not currently support offloading of Neutron Security
|
|
Group rules - experimental support is expected in Open vSwitch 2.13 when used
|
|
with Linux >= 5.4 and as yet unreleased NIC firmware. It is recommended that
|
|
port security is disabled on Neutron networks being used for hardware
|
|
offloading use cases due to the performance overhead of enforcing
|
|
security group rules in userspace.
|
|
|
|
Deployment
|
|
----------
|
|
|
|
MAAS - Hardware Enablement Kernel
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
As a more recent Linux kernel than that provided as part of Ubuntu 18.04 LTS
|
|
is required to support this feature, machines with compatible network cards
|
|
must be commissioned and configured in MAAS to use the Ubuntu 18.04 LTS
|
|
Hardware Enablement (HWE) kernel rather than the standard release kernel.
|
|
|
|
MAAS - SR-IOV VF-LAG
|
|
~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Mellanox `SR-IOV VF-LAG`_ provides hardware link-aggregation (LAG) for
|
|
hardware offloaded ports and is recommended for deployment as it avoids the
|
|
need to pass two hardware offloaded ports to each VM for resilience. This
|
|
feature is configured in the underlying NIC using standard Linux bonding as
|
|
configured through MAAS.
|
|
|
|
.. note::
|
|
|
|
VF-LAG can only be used with NIC ports that reside on the same underlying
|
|
NIC.
|
|
|
|
.. note::
|
|
|
|
Use of VF-LAG halfs the offloaded port capacity of the card and as
|
|
such VF representor port configuration should only be made on the first
|
|
network port on the card.
|
|
|
|
Charm configuration
|
|
~~~~~~~~~~~~~~~~~~~
|
|
|
|
Hardware offload support is enabled using the ``enable-hardware-offload`` option
|
|
provided by the neutron-api and neutron-openvswitch charms.
|
|
|
|
Enabling hardware offloading requires configuration of VF representator ports on
|
|
the NICs supporting the hardware offload - these are used to route network packets
|
|
without flow rules to the OVS userspace daemon for handling and subsequent
|
|
programming into the hardware offloaded flows. This is supported via use of
|
|
the ``sriov-numvfs`` option provided by the neutron-openvswitch charm.
|
|
|
|
Finally the ``openvswitch`` firewall driver must be used with hardware offloading.
|
|
Eventually it will be possible to offload security group rules using this driver
|
|
(see note above).
|
|
|
|
The following overlay may be used with the OpenStack base deployment bundle:
|
|
|
|
.. code-block:: yaml
|
|
|
|
series: bionic
|
|
applications:
|
|
neutron-openvswitch:
|
|
charm: cs:neutron-openvswitch
|
|
options:
|
|
enable-hardware-offload: true
|
|
sriov-numvfs: "enp3s0f0:64 enp3s0f1:0"
|
|
firewall-driver: openvswitch
|
|
neutron-api:
|
|
charm: cs:neutron-api
|
|
options:
|
|
enable-hardware-offload: true
|
|
|
|
In this overlay ``enp3s0f0`` and ``enp3s0f1`` are two ports on the same
|
|
Mellanox ConnectX-5 card and are configured as a Linux bond ``bond1``
|
|
to enable VF-LAG for resilience and performance. ``bond1`` is also configured
|
|
with the network interface used for VXLAN overlay traffic to allow full
|
|
offloading of networks of this type.
|
|
|
|
Creating hardware offloaded ports
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Hardware offloaded ports must be created via Neutron and then passed to Nova
|
|
for use by VM's:
|
|
|
|
.. code-block:: none
|
|
|
|
openstack port create --network private --vnic-type=direct \
|
|
--binding-profile '{"capabilities": ["switchdev"]}' direct_port1
|
|
openstack server create --flavor m1.small --image bionic \
|
|
--nic port-id=direct_port1 vm1
|
|
|
|
The image used for the VM must include the Mellanox kernel driver. Ubuntu 18.04
|
|
LTS (or later) cloud images include this driver by default.
|
|
|
|
.. LINKS
|
|
.. _OVS hardware offload: https://docs.openstack.org/neutron/stein/admin/config-ovs-offload.html
|
|
.. _SR-IOV VF-LAG: https://www.mellanox.com/related-docs/prod_software/ASAP2_Hardware_Offloading_for_vSwitches_Release_Notes_v4.4.pdf
|