Clear Linux + KVM + OpenVswitch

anowak · April 30, 2023, 11:50pm

Hi All,
Note: I’m new to Clear Linux - just downloaded yesterday

Traditionally I ran Debian 11 + KVM + Openvswitch but found that on a Intel NUC it randomly crashes … spent hours trying to troubleshoot without success. I then stumbled on a few sites suggesting an Intel optimised Linux - so here I am.

I have installed the OS and the packages.

sudo swupd bundle-add kernel-kvm kvm-host

Install OpenvSwitch.

sudo swupd bundle-add openvswitch

and some default networking

sudo swupd bundle-add network-basic

The KVM part seems to be working. I just can’t seem to get openvswitch configured as I would have in Debian.

My setup is relatively simple. I use DHCP to obtain an address but can use static. I just can’t find any documentation to help. I found this command to start ovs

sudo /usr/share/openvswitch/scripts/ovs-ctl --start

But after every reboot it decided to stop … obviouly I need to load it as a service?

I also configure my nework by running

sudo ovs-vsctl add-br br0
sudo ovs-vsctl add-port eno1 br0
sudo ip link set br0 up

I don’t know how to restart the network or rerun dhcp … things seem to be missing coming from Debian? Looks like I am running systemd-network, but don’t see the service or associated files?
This is brand new install and updated to 38990.

Looking for some help or documentation that may get me up and running.

Regards

anowak · May 1, 2023, 12:04am

I’ve had a little progress with the OpenvSwitch. Debian did this for me, so it was a matter of searching.

sudo systemctl start openvswitch.service
sudo systemctl enable openvswitch.service
sudo systemctl status openvswitch.service

Now I just need to workout the networking part. I did find the folder /usr/lib/systemd/network. So guessing in here I would create me config file. In Debian it looked like.

# The loopback network interface
auto lo
iface lo inet loopback

# Required "auto ovs-br0" since Debian 11.5
auto ovs-br0
allow-ovs ovs-br0
iface ovs-br0 inet dhcp
 ovs_type OVSBridge
 ovs_ports enp5s0

# The primary network interface
auto enp5s0
allow-ovs-br0 enp5s0
iface enp5s0 inet manual
 ovs_bridge ovs-br0
 ovs_type OVSPort

Ta.

anowak · May 1, 2023, 12:37am

Hi All,

Think I’ve got it working - happy to be corrected, if this is not the incorrect way.

modified the file /usr/lib/systemd/network/80-dhcp.network

[Match]
Name=en*

[Network]
DHCP=yes

To

[Match]
Name=en*

and created a new file 70-ovs.network

[Match]
Name=br0

[Network]
DHCP=yes

I also enabled the networkd service just in case…

sudo systemctl enable systemd-networkd.service

Network now seems to be up over reboots. DHCP address is allocating to the br0 MAC which is mirrored from eno1.

Ta.

anowak · May 1, 2023, 2:59am

Hi All,

So defined my networking network.xml

<network>
    <name>ovs-network</name>
    <forward mode='bridge' />
    <bridge name='br0' />
    <model type='virtio'/>
    <virtualport type='openvswitch' />
    <portgroup name='native' default='yes' />
    <portgroup name='device_vl20'>
        <vlan>
            <tag id='20' />
        </vlan>
    </portgroup>
</network>

Moved a couple VMs off my Debian KVM and tried starting it up, with no luck. I get the error.

error: Failed to start domain 'testvm'
error: /usr/libexec/qemu-bridge-helper --use-vnet --br=br0 --fd=26: failed to communicate with bridge helper: stderr=failed to parse default acl file `/etc/qemu/bridge.conf'
: Transport endpoint is not connected

So I manage to find some information that the files needs to be created.

sudo mkdir -p /etc/qemu
touch bridge.conf
sudo chmod u+r /etc/qemu/bridge.conf

sudo vim bridge.conf - add the bridge adapter.

allow br0

Now I get the next error.

error: Failed to start domain 'testvm'
error: /usr/libexec/qemu-bridge-helper --use-vnet --br=br0 --fd=26: failed to communicate with bridge helper: stderr=failed to add interface `tap0' to bridge `br0': Operation not supported
: Transport endpoint is not connected

I can create a tunnel using

sudo openvpn --mktun --dev tap0

and add it to the bridge br0

sudo ovs-vsctl add-port br0 tap0

sudo ovs-vsctl show

442085b9-7c92-4f25-bdc5-dd8cad6a1efa
    Bridge br0
        Port eno1
            Interface eno1
        Port tap0
            Interface tap0
        Port br0
            Interface br0
                type: internal
    ovs_version: "3.1.0"

Gorian · May 3, 2023, 9:12pm

@anowak apologies that I didn’t see this sooner! I’m using the same setup in regards to Open vSwitch and systemd-networkd, and that’s roughly the same configuration that I’m using. I haven’t quite got around to getting KVM setup yet, but it’s on my todo list.

btwarden · May 3, 2023, 10:17pm

Files in /usr/lib are owned/managed by swupd and will be replaced on update or swupd repair. You need to override the system-owned file by making (and editing) a copy at /etc/systemd/network/80-dhcp.network.

For more information:

anowak · May 3, 2023, 10:22pm

Hi Gorian,

Well do let me know if you manage to get it installed as I am stuck with the following message, and can’t find a reason.

error: Failed to start domain 'testvm'
error: /usr/libexec/qemu-bridge-helper --use-vnet --br=br0 --fd=26: failed to communicate with bridge helper: stderr=failed to add interface `tap0' to bridge `br0': Operation not supported
: Transport endpoint is not connected

I read that systemd-networkd does not integrate with openvswitch, but these posts were from years ago. It looks like openvswitch is working… as I can see my connections come up correctly?

anowak · May 3, 2023, 10:24pm

Hi btwarden,

Ah, that would explain it.
Thank you for sharing so quickly. I’ll have a read and post back.

Regards

anowak · May 4, 2023, 10:29pm

Hi Btwarden,

Got over the first issue by following your advice. I created /etc/systemd/network and created two files in here.

ovs-br0.network

[Match]
Name=ovs-br0

[Network]
DHCP=yes

80-dhcp.network

[Match]
Name=eno1

[Network]
Bridge=ovs-br0

If I start the VM I get

sudo virt-install --name test --memory 4096 --disk /var/lib/libvirt/images/test.qcow2,bus=virtio,size=10 --vcpus 2 --os-variant debian11 --network network=ovs-network,model=virtio --graphic spice,listen='0.0.0.0' --boot hd --noautoconsole --features kvm_hidden=on --autostart --cdrom /opt/kvm/iso/debian-11.7.0-amd64-netinst.iso

Starting install...
Allocating 'test.qcow2'                                                                                                                                                                                                               |    0 B  00:00:00 ...
Removing disk 'test.qcow2'                                                                                                                                                                                                            |    0 B  00:00:00
ERROR    Network not found: no network with matching name 'ovs-network'
Domain installation does not appear to have been successful.
If it was, you can restart your domain by running:
  virsh --connect qemu:///system start test
otherwise, please restart your installation.

defined network

<network>
    <name>ovs-network</name>
    <forward mode='bridge' />
    <bridge name='ovs-br0' />
    <model type='virtio'/>
    <virtualport type='openvswitch' />
    <portgroup name='native' default='yes' />
    <portgroup name='device_vl20'>
        <vlan>
            <tag id='20' />
        </vlan>
    </portgroup>
</network>

Network Active - Does it require a .network file created?

 virsh net-list --all
 Name          State    Autostart   Persistent
------------------------------------------------
 ovs-network   active   yes         yes

Bear with me - new to this OS

Regards

btwarden · May 4, 2023, 10:52pm

You’re beyond anything I’ve tried to do so far, but I suspect at minimum you also need to create the bridge netdev. See this example: systemd.network

anowak · May 4, 2023, 11:59pm

Hi Btwarden,

Can’t create a netdev. I tried this already and it breaks the ovs, as it tries to create two bridges of the same name.

Basically you can’t do

ovs-br0.netdev

[NetDev]
Name=ovs-br0
Kind=bridge

and

sudo ovs-vsctl add-br ovs-br0
sudo ovs-vsctl add-port ovs-br0 eno1

You get an error under ovs-vsctl show that the bridge already exists. This is stored in the ovs db.
Sticking to the second part and creating the bridge and ports and using systemd-networkd just to allocate networks works.

not sure where the error is generated? as it’s part of the KVM defined networks.

ERROR    Network not found: no network with matching name 'ovs-network'

anowak · May 5, 2023, 12:13am

Ok, I can see issue - just need to find resolve.
If I use sudo during virt-install it complains. Checking this shows the issue. Now need to find permissions to give my user.

sudo virsh net-list --all
Password:
 Name   State   Autostart   Persistent
----------------------------------------

virsh net-list --all
 Name          State    Autostart   Persistent
------------------------------------------------
 ovs-network   active   yes         yes

anowak · May 5, 2023, 6:02am

Hi all.

So looks like it is a permissions and location issue, still trying to workout where?

If I create the ovs switch as sudo and use sudo virt-install it works?

sudo virsh net-define ~/network.xml
sudo virsh net-start ovs-nework
sudo virsh net-autostart ovs-nework

sudo virsh net-list --all
 Name          State    Autostart   Persistent
------------------------------------------------
 ovs-network   active   yes         yes

sudo virt-install --name test --memory 4096 --disk /opt/kvm/vms/test.qcow2,bus=virtio,size=10 --vcpus 2 --os-variant debian11 --network=bridge:ovs-br0,model=virtio,virtualport_type=openvswitch --graphic spice,listen='0.0.0.0' --boot hd --noautoconsole --features kvm_hidden=on --autostart --cdrom /opt/kvm/iso/debian-11.7.0-amd64-netinst.iso

Starting install...
Allocating 'test.qcow2'                                                                                                                                                                                                               |    0 B  00:00:00 ...
Creating domain...                                                                                                                                                                                                                    |    0 B  00:00:00

Domain is still running. Installation may be in progress.
You can reconnect to the console to complete the installation process.

ip addr
...
...
19: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq master ovs-system state UNKNOWN group default qlen 1000
    link/ether fe:54:00:9e:fb:90 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fe9e:fb90/64 scope link
       valid_lft forever preferred_lft forever

anowak · May 6, 2023, 9:00am

So if I am using my standard user account: say kvmuser, the KVM components appear under

~/.config/libvirt/

Nothing seems to have access to this location hence the errors.

If I use sudo, the KVM files get created under

/etc/libvirt/

I don’t really want to be storing VMs under user directory. If I delete ~/.config/libvirt it just gets recreated.

Regards

anowak · May 9, 2023, 8:52am

Hi All,

Still I’m no closer to getting KVM with openvswitch to work.

I have libvirt scattered in 3 places.

~/.config/libvirt/qemu - The location it will used when using virsh as my user account (although no access to this folder to qemu service).

/var/lib/libvirt/qemu - Not sure what this is used for?

/etc/libvirt/qemu - Used by root where everything works correctly? Is this what is required to get KVM and openvswitch working? run everything as root?

Still get the following error due to some permission when creating a VM under the user account

ERROR    /usr/libexec/qemu-bridge-helper --use-vnet --br=ovs-br0 --fd=28: failed to communicate with bridge helper: stderr=failed to add interface `tap0' to bridge `ovs-br0': Operation not supported

Any help would be greatly appreciated.

Kind Regards

Gorian · May 9, 2023, 5:47pm

FYI, ping in the future, it’s hard to keep up otherwise I’ll try to find some time to dig into this soon, as it’s on my todo list

As far as kvm/qemu - I’ve only ever ran it as root, and not as a specific user. Generally, it seems like that’s a better design for global services - the service should be able to de-escalate if needed.

The issues you are running into is one of many reasons why - need access to lots of devices, making network changes, etc

Is there a reason you don’t want to run as root? Generally speaking, you will run applications as a specific user (usually yourself) but services as root - things like DNS, nginx, apache, etc. run as root and then they de-escalate after they’ve bound to things that need root permissions (like using ports 80 and 443 for example)

Here’s a good explanation I found on google

anowak · May 10, 2023, 12:30am

Thanks Gorian for reaching out… didn’t want to directly trouble anyone with this issue, I was hoping someone had come across this before. Noticed there are no documents for using openvswitch in Clear Linux anywhere - I scoured through lots of articles, but nothing that’s helped. Every distribution has a different method of doing things.

To answer your original question, yes, I could run everything as root, although, if I want to give a user access to administer KVM - start/stop/modify vms etc (hopefully build and add networks)? I’d have to give them root privileges to the system, which is not great. What I am used to, at least with my experience in Debian, was being able to manage VMs via the user account. New VMs were created with elevate permissions and then the VMs were managed by user using virsh.

In Clear Linux, when I try this method, it doesn’t work as expected. If I create a VM using root - as a user, I am unable to see the virtual machine. If I create a VM under the user (without network) the root user is unable to see the VMs. Everything is separated under a user context. This is probably a good thing. Now going off how the system works - I want to run KVM as only the user - this is where I get stuck as the openvswitch permissions are causing an issue… I can’t seem to find where I can allocate these permissions.

Examples
Here I create a VM under the user (without network)

virt-install --name debian --memory 4096 --disk debian.qcow2,pool=vms,bus=sata,size=10 --vcpus 2 --os-variant debian11  --graphic spice,listen='0.0.0.0' --boot hd --noautoconsole --features kvm_hidden=on --autostart --cdrom /opt/kvm/iso/debian-11.7.0-amd64-netinst.iso

# As User
>virsh list --all
 Id   Name     State
------------------------
 1    debian   running

# As root
> sudo virsh list --all
 Id   Name   State
--------------------

The files are created under the user.

 ls ~/.config/libvirt/qemu/
autostart  channel  checkpoint  debian.xml  dump  lib  networks  nvram  ram  save  snapshot

This is in reverse when root create the VM.

If I define anything as root eg. network or pool. I cannot see it as any user.
I tried symbolic link to ~/.config/libvirt → /etc/libvirt although this does not work… you can see the VM but they shows as down under the user context, when in fact as root it’s up.

Kind Regards

anowak · May 10, 2023, 9:29am

“FYI, ping in the future, it’s hard to keep up otherwise”

How do I “ping”?

Regards

marioroy · May 10, 2023, 5:34pm

I recall adding myself to various groups in /etc/group and had zero problems running KVM as a user. Notice that the qemu user is also a member of the kvm group.

qemu:x:200:mario
kvm:x:201:qemu,mario
libvirt:x:203:nova,mario

Gorian · May 10, 2023, 6:53pm

ah! My bad. Basically, “at” someone - like @anowak, it should give you a notification of some sort that someone has “mentioned” you