In this post, I describe the necessary steps to integrate OpenStack Ussuri and a single node Hyper-V compute with Open vSwitch provider+VXLAN networking. Open vSwitch is mandatory when you plan to use VXLAN networking.

Lab environment

Setup of the following components is not covered by this blogpost.

  • Server with two network card. First network card will be used as management interface, the second one will be handled by Open vSwitch.
  • Operational OpenStack controller stack, provider and self-service network.
  • Existing Windows domain infrastructure.

Installing Windows Server

Windows Server version

This blogpost is written using english Windows Server Core 2019 Datacenter Evaluation.

I recommend either Windows Server Core 2019 or Hyper-V Server 2019. You can also install Windows Server 2019 Desktop Experience if you are more familiar with the full Windows desktop. The default install is Server Core with command line focused graphical interface.

Different server software editions, costs, virtualization restrictions and guest licensing are summarized in the following table. Comparison is based on the Comparison of Standard and Datacenter editions of Windows Server 2019 page.

Windows version Licensing Max VMs Licence for VMs
Windows Server 2019 Standard paid license 2 per licence no
Windows Server 2019 Datacenter paid license unlimited yes (AVMA)
Hyper-V Server 2019 free unlimited no

Localized Windows Server (or Hyper-V Server) versions can be affected by various incompatibility problems, thus I gently advise you to choose english version.

Basic configuration

Log on to the freshly installed server, set password and start configuration using command line.

You can install hardware drivers using pnputil as required.

Server Core and Hyper-V Servers can be configured using the sconfig utility.

Microsoft (R) Windows Script Host Version 5.812
Copyright (C) Microsoft Corporation. All rights reserved.

Inspecting system...


===============================================================================
                         Server Configuration
===============================================================================

1) Domain/Workgroup:                    Domain:  example.net
2) Computer Name:                       OSHV2K19
3) Add Local Administrator
4) Configure Remote Management          Enabled

5) Windows Update Settings:             DownloadOnly
6) Download and Install Updates
7) Remote Desktop:                      Enabled (all clients)

8) Network Settings
9) Date and Time
10) Telemetry settings                  Unknown
11) Windows Activation

12) Log Off User
13) Restart Server
14) Shut Down Server
15) Exit to Command Line

Enter number to select an option:
  1. Assign an IP address to the choosen management network interface: 8) Network Settings
  2. Enable remote desktop: 7) Remote Desktop
  3. Apply Windows updates: 6) Download and Install Updates
  4. Join a Windows domain because Windows remote management can be simpler: 1) Domain/Workgroup

Configure NTP if the server is not domain member.

net stop w32time
w32tm /config "/manualpeerlist:pool.ntp.org,0x8" /syncfromflags:MANUAL
net start w32time

Firewall can be switched off in a lab environment.

netsh advfirewall set allprofiles state off

Start PowerShell in a new window.

start powershell

You need to install Hyper-V role on Windows Server 2019. Hyper-V Server 2019 comes with these roles built-in.

PS > Add-WindowsFeature Hyper-V
PS > Add-WindowsFeature Hyper-V-PowerShell

Enable remote PowerShell for remote Hyper-V management.

PS > Enable-PSRemoting

Open vSwitch

Cloudbase provides signed Open vSwitch builds for Windows, but the latest downloadable stable version is 2.10.0 at time of writing. This is an old version and I have experienced some serious packet loss with that.

Finally, I built my own Open vSwitch MSI package from source tree and I installed that. Fortunatelly, I have not had to install a Windows build platform but I utilized AppVeyor autobuild service. Just fork the ovs repository under your GitHub account, link the forked repository to your AppVeyor account, create a new build and download the MSI installer. The latest commit was f0e4a73 in the repository when my MSI installer was made.

Needed dependencies to be installed on the target Hyper-V can be read from appveyor.yml.

Install Visual Studio 2013/2019 runtime.

vcredist_x64.exe

Install OpenSSL 1.0.2u.

Win64OpenSSL-1_0_2u.exe

AppVeyor Open vSwitch builds are not signed and therefore you have to configure Windows to accept unsigned Open vSwitch drivers. Run the following commands in command line and reboot Windows.

bcdedit /set LOADOPTIONS DISABLE_INTEGRITY_CHECKS
bcdedit /set TESTSIGNING ON
bcdedit /set nointegritychecks ON
shutdown /r

Install your Open vSwitch build. Enabling installer logging (/l*v ovs-install-log.txt) is not mandatory but can be useful at a later time.

msiexec /i OpenvSwitch-Release.msi /qr /l*v ovs-install-log.txt

Enable listening on TCP/6640.

PS > sc.exe config ovsdb-server binPath= --% "\"C:\Program Files\Open vSwitch\bin\ovsdb-server.exe\" --log-file=\"C:\Program Files\Open vSwitch\logs\ovsdb-server.log\" --pidfile --service --service-monitor --unixctl=\"C:\ProgramData\openvswitch\ovsdb-server.ctl\" --remote=punix:\"C:\ProgramData\openvswitch\db.sock\" --remote=ptcp:6640 \"C:\Program Files\Open vSwitch\conf\conf.db\""
PS > sc.exe stop ovs-vswitchd
PS > sc.exe stop ovsdb-server
PS > sc.exe start ovsdb-server
PS > sc.exe start ovs-vswitchd

Logoff and re-logon to take effect the new value of the %PATH% variable.

Create VMswitch. Open vSwitch has a Hyper-V virtual switch extension.

Get-NetAdapter | select "Name"

Name
----
management
Ethernet

New-VMSwitch -Name vSwitchOVS -NetAdapterName "Ethernet" -AllowManagementOS $true
Get-VMSwitchExtension -VMSwitchName vSwitchOVS -Name "Open vSwitch Extension"
Enable-VMSwitchExtension -VMSwitchName vSwitchOVS -Name "Open vSwitch Extension"

Create basic Open vSwitch configuration.

ovs-vsctl.exe add-br br-ext
ovs-vsctl.exe add-port br-ext Ethernet

Assign a VXLAN endpoint IP address to br-ext.

Enable-NetAdapter br-ext
New-NetIPAddress -IPAddress <VXLAN_IPaddr> -InterfaceAlias br-ext -PrefixLength <example: 24>

OpenStack Nova compute installer

Install OpenStack Nova compute Hyper-V agent. The installer contains also Neutron OVS agent. Enabling installer logging (/l*v nova-compute-install-log.txt) is not mandatory but can be useful at a later time.

msiexec /i HyperVNovaCompute_Ussuri_21_0_0.msi /l*v nova-compute-install-log.txt

Go through the installer wizard.

  1. Custom Setup: Unselect Neutron Hyper-V Agent feature to install. Select OpenStack Nova Compute, Neutron OVS Agent, FreeRDP for Hyper-V features to install.
  2. Virtual Switch (Neutron OVS Agent): Choose existing existing virtual switch: vSwitchOVS
  3. Basic Configuration: RDP console URL is the hypervisor itself in single node setup, AMQP Transport URL points to a RabbitMQ server.
  4. Placement API: Fill the form according to your OpenStack setup.
  5. Neutron: Fill the form according to your OpenStack setup.
  6. Neutron Agents Configuration: OVS Network Type: vxlan, Tunneling local IP: <VXLAN_IPaddr>.

Alternatively, silent or unattended install is also available. Manual install process can be found in OpenStack Hyper-V virtualization platform document.

Patching nova compute hyperv agent

VM virtual network interface plugging is not handled correctly in my lab environment. After a VM is launched, it has not network connectivity and ovs-vsctl show reports error.

ovs-vsctl show
...
Port "932cb602-8ec0-43be-bed5-b690a8b1d99a"
    Interface "932cb602-8ec0-43be-bed5-b690a8b1d99a"
        error: "could not open network device 932cb602-8ec0-43be-bed5-b690a8b1d99a (No such device)"
...

Relevant code can be found in C:\Program Files\Cloudbase Solutions\OpenStack\Nova\Python\Lib\site-packages\compute_hyperv\nova\vmops.py. Two functions, spawn() and power_on() initiate vif plugging. According to the code, the virtual network interface is added to the OVS bridge before Hyper-V creates it upon VM power on. But the right behavior is that nova compute agent must wait until VM power on (not only wait until VM creation) before plugging in virtual network interface to the OVS bridge.

I wrote a small patch which powers on VM first, then add virtual network interface to the OVS bridge. Apply the following patch on vmops.py and restart nova-compute service.

--- orig/vmops.py    2020-10-06 02:15:08.000000000 +0200
+++ updated/vmops.py 2021-11-08 08:46:11.044979827 +0100
@@ -312,13 +312,10 @@
         self._create_ephemerals(instance, block_device_info['ephemerals'])

         try:
-            with self.wait_vif_plug_events(instance, network_info):
-                # waiting will occur after the instance is created.
-                self.create_instance(context, instance, network_info,
-                                     block_device_info, vm_gen, image_meta)
-                # This is supported starting from OVS version 2.5
-                self.plug_vifs(instance, network_info)
-
+            LOG.debug("Creating instance")
+            self.create_instance(context, instance, network_info,
+                                 block_device_info, vm_gen, image_meta)
+            LOG.debug("Updating device metadata")
             self.update_device_metadata(context, instance)

             if configdrive.required_by(instance):
@@ -337,6 +334,11 @@
                 self.power_on(instance,
                               network_info=network_info,
                               should_plug_vifs=False)
+                self.pause(instance)
+                with self.wait_vif_plug_events(instance, network_info):
+                    LOG.debug("Plugging vifs")
+                    self.plug_vifs(instance, network_info)
+                self.unpause(instance)
         except Exception:
             with excutils.save_and_reraise_exception():
                 self.destroy(instance, network_info, block_device_info)
@@ -1018,9 +1020,18 @@
             self._volumeops.fix_instance_volume_disk_paths(instance.name,
                                                            block_device_info)

+        self._set_vm_state(instance, os_win_const.HYPERV_VM_STATE_ENABLED)
         if should_plug_vifs:
+            LOG.debug("Pause instance", instance=instance)
+            self._set_vm_state(instance,
+                               os_win_const.HYPERV_VM_STATE_PAUSED)
+            LOG.debug("Unplug instance vif(s)", instance=instance)
+            self.unplug_vifs(instance, network_info)
+            LOG.debug("Plug instance vif(s)", instance=instance)
             self.plug_vifs(instance, network_info)
-        self._set_vm_state(instance, os_win_const.HYPERV_VM_STATE_ENABLED)
+            LOG.debug("Unpause instance", instance=instance)
+            self._set_vm_state(instance,
+                               os_win_const.HYPERV_VM_STATE_ENABLED)

     def _set_vm_state(self, instance, req_state):
         instance_name = instance.name

Provider network configuration in neutron agent

The MSI installer does not configure provider network bridge mappings. Edit C:\Program Files\Cloudbase Solutions\OpenStack\Nova\etc\neutron_ovs_agent.conf according to the followings.

[ovs]
...
bridge_mappings = 'provider:br-ext'
...

FreeRDP WebConnect setup

Install FreeRDP WebConnect which is a RDP-HTML5 proxy that provides access to a virtual machine console from a web browser.

Go through the installer wizzard.

  1. FreeRDP-WebConnect configuration: You can enable Redirect HTTP connections to HTTPS if you create a proper HTTPS certificate and overwrite C:\Program Files\Cloudbase Solutions\FreeRDP-WebConnect\etc\server.cer.
  2. OpenStack configuration: Configure OpenStack integration.

The generated configuration file is located at C:\Program Files\Cloudbase Solutions\FreeRDP-WebConnect\etc\wsgate.ini.

VM console URL can be obtained from OpenStack command line.

nova get-rdp-console <VM instance> rdp-html5

OpenStack controller settings

Discover new hypervisor node on the controller.

su -s /bin/sh -c "nova-manage cell_v2 discover_hosts --verbose" nova

Cleaning OVS bridge

Orphan, erroneous virtual network interfaces can be removed from OVS bridge after system restart or neutron agent restart using the following PowerShell snippet.

$interfaces = ovs-vsctl --format=csv find interface ofport="-1" | ConvertFrom-Csv | Select "name"
Foreach($i in $interfaces)
{
  ovs-vsctl del-port br-int "$($i.name)"
}

Further steps

  • Open vSwitch code signing.
  • Multiple network interface: teaming or bonding.
  • Configure Windows clustering.
  • Configure VM live migration.
  • Cinder installation.

Installing Ubuntu on cheap Acer Aspire notebooks

Installing Ubuntu on cheap Acer Aspire notebooks can be a bit tricky but not impossible.

Replacing battery in Eaton 5P 1150 UPS and re-flashing firmware

I had to made a firmware re-flashing of my Eaton 5P 1150 after replacing batteries.

Open vSwitch and network configuration tools on Ubuntu 18.04: netplan, ifupdown, systemd-networkd

Setting up static IP configuration, DNS resolver and Open vSwitch on Ubuntu 18.04: netplan, ifupdown, systemd-networkd.

NUT serial support on OpenWrt

Using a serial port connected UPS on OpenWrt with NUT.

Disable Intel Hyper-Threading on Linux

Disable Intel Hyper-Threading on Linux in software, not in BIOS/UEFI.

Migration from BIND/OpenDNSSEC to PowerDNS with DNSSEC

Migration of BIND and OpenDNSSEC to PowerDNS 4 with DNSSEC.

Installing Altera Quartus II 14.0 and ModelSim on Ubuntu Linux 14.10 system

Installation of Altera Quartus II 14.0 and setting up of ModelSim Altera Edition on Ubuntu 14.10

Remote FPGA JTAG programming via SSH tunnel with Altera Quartus II

JTAG programming of an Altera FPGA, through an SSH tunnel with Quartus II. The device is on the network, connected to a remote machine through USB Blaster.

Altera Quartus II and NIOS II command line tutorial

Altera Quartus II and NIOS II command line