LTS Kernel entry in boot manager remains. How to remove?

Hi. I removed the LTS Kernel bundles but the entry in the boot manager still remains.

$ sudo swupd bundle-remove kernel-lts

Warning: Bundle "kernel-lts" is not installed, skipping it...

Failed to remove 1 of 1 bundles

$ sudo swupd bundle-remove kernel-lts2018

Warning: Bundle "kernel-lts2018" is not installed, skipping it...

Failed to remove 1 of 1 bundles

$ sudo swupd clean
0 files removed

$ sudo clr-boot-manager list-kernels
* org.clearlinux.native.5.5.8-917
  org.clearlinux.lts2018.4.19.105-115

$ sudo clr-boot-manager update
[ERROR] cbm (../src/bootman/update.c:L238): Could not find default kernel for type lts2018, using highest relno

$ swupd search kernel -a | grep installed | tail -n+2
	 kernel-install                 	- Installs kernel, initrd, kernel config, system map, and creates a bootloader entry for the new kernel. (installed)
	 man-pages                      	- Linux man-pages to document kernel and C library interfaces for user-space programs.  (26MB) (installed)
	 bootloader                     	- Loads kernel from disk and boots the system.  (5MB) (installed)
	 fuse                           	- Filesystem in Userspace (FUSE) to export a virtual filesystem to Linux kernel, or mount filesystem implementation for non-privileged users. (installed)
	 linux-firmware                 	- Firmware for kernel drivers.  (302MB) (installed)
	 kernel-native                  	- Run the native kernel.  (159MB) (installed)
	 linux-firmware-extras          	- Extra firmware for the kernel.  (39MB) (installed)
	 syslinux                       	- Kernel loader which uses a FAT, ext2/3 or iso9660 filesystem or a PXE network  (66MB) (installed)
	 linux-dev                      	- Files for building modules against the native kernel.  (410MB) (installed)
	 sysadmin-basic                 	- Run common utilities, useful for managing a system.  (107MB) (installed)

$ ls -la /usr/lib/kernel/
total 165612
drwxr-xr-x  3 root root     4096 Mar  8 11:56 .
drwxr-xr-x 40 root root     4096 Mar  8 11:25 ..
-rw-r--r--  1 root root      256 Feb 20 00:09 cmdline-4.19.105-115.lts2018
-rw-r--r--  1 root root      282 Mar  6 00:15 cmdline-5.5.8-917.native
-rw-r--r--  1 root root   188855 Feb 20 00:09 config-4.19.105-115.lts2018
-rw-r--r--  1 root root   211256 Mar  6 00:15 config-5.5.8-917.native
lrwxrwxrwx  1 root root       31 Mar  6 00:16 default-native -> org.clearlinux.native.5.5.8-917
-rw-r--r--  1 root root    79296 Sep 27 00:29 initrd-org.clearlinux.current.5.3.1-11
-rw-r--r--  1 root root    81288 Feb 15 01:11 initrd-org.clearlinux.current.5.6.0.rc1-12
-rw-r--r--  1 root root    81300 Feb 18 05:48 initrd-org.clearlinux.current.5.6.0.rc2-13
-rw-r--r--  1 root root    81256 Mar  6 00:36 initrd-org.clearlinux.current.5.6.0.rc4-14
-rw-r--r--  1 root root    68216 Feb  5 16:06 initrd-org.clearlinux.lts2017.4.14.170-92
-rw-r--r--  1 root root    68208 Feb 12 00:25 initrd-org.clearlinux.lts2017.4.14.170-93
-rw-r--r--  1 root root    68276 Feb 15 00:12 initrd-org.clearlinux.lts2017.4.14.171-94
-rw-r--r--  1 root root    68560 Mar  6 00:21 initrd-org.clearlinux.lts2017.4.14.172-99
-rw-r--r--  1 root root    73360 Feb  5 16:05 initrd-org.clearlinux.lts2018.4.19.102-111
-rw-r--r--  1 root root    73432 Feb 12 00:20 initrd-org.clearlinux.lts2018.4.19.103-113
-rw-r--r--  1 root root    73404 Feb 15 00:12 initrd-org.clearlinux.lts2018.4.19.104-114
-rw-r--r--  1 root root    73468 Feb 20 00:10 initrd-org.clearlinux.lts2018.4.19.105-115
-rw-r--r--  1 root root    73764 Mar  5 22:36 initrd-org.clearlinux.lts2018.4.19.108-120
-rw-r--r--  1 root root    78784 Sep 27 00:27 initrd-org.clearlinux.mainline.5.3.0-10
-rw-r--r--  1 root root    80772 Feb 15 04:03 initrd-org.clearlinux.mainline.5.6.0.rc1-11
-rw-r--r--  1 root root    80748 Feb 18 05:52 initrd-org.clearlinux.mainline.5.6.0.rc2-12
-rw-r--r--  1 root root    80752 Mar  6 00:47 initrd-org.clearlinux.mainline.5.6.0.rc4-14
-rw-r--r--  1 root root    80040 Mar  6 00:16 initrd-org.clearlinux.native.5.5.8-917
drwxr-xr-x  2 root root     4096 Feb  9 15:46 install.d
-rwxr-xr-x  1 root root 12875424 Feb 20 00:09 org.clearlinux.lts2018.4.19.105-115
-rwxr-xr-x  1 root root 14575776 Mar  6 00:15 org.clearlinux.native.5.5.8-917
-rw-r--r--  1 root root  5085432 Feb 20 00:09 System.map-4.19.105-115.lts2018
-rw-r--r--  1 root root  6271427 Mar  6 00:15 System.map-5.5.8-917.native
-rw-r--r--  1 root root 52023416 Feb 20 00:09 vmlinux-4.19.105-115.lts2018
-rw-r--r--  1 root root 76925456 Mar  6 00:15 vmlinux-5.5.8-917.native

CL is using the correct Kernel.

$ uname -r
5.5.8-917.native

Any hint on how to remove the LTS Kernel(s) the correct way?

Thank you.

Have you tried rebooting after running sudo clr-boot-manager update You could also try sudo swupd repair --picky which will verify and remove any files that are not associated with bundles but are in the path where bundles are installed. You could also try adding the LTS bundle back and then removing again.

If that doesn’t work you could forcefully remove it by sudo start boot.mount and then go into you /boot/EFI/org.clearlinux/ folder and delete it. Be careful with this option though.

This is a gap between swupd and clr-boot-manager. I’ll open a bug on it but you need to manually remove the content in /usr/lib/{kernel, modules} of the kernel you want to delete yourself.

Thank you @eadams for the suggestions, but…

Ofcourse. :slight_smile:


Done, but it didn’t help. (output shortened)

$ sudo swupd repair --picky
Password: 
Diagnosing version 32550
Downloading missing manifests...

Checking for corrupt files
 [100%]

Validate downloaded files
 [100%]

Starting download of remaining update content. This may take a while...
 [100%]


Adding any missing files
 [100%]


Repairing corrupt files
 [100%]


Removing extraneous files
 [100%]


Removing extra files under /usr
 -> ...
Inspected 380676 files
  80 files found which should be deleted
	80 of 80 files were deleted
	0 of 80 files were not deleted

Calling post-update helper scripts
External command: none
External command: [ERROR] cbm (../src/bootman/update.c:L238): Could not find default kernel for type lts2018, using highest relno

Repair successful
 [100%]

Also done, but also didn’t help.


Is the command line correct?

$ sudo start boot.mount
sudo: start: command not found

@william.douglas: Thank you, i’ll do.

Oh and make sure you run clr-boot-manager update after doing that.

Done.

  • deleted everything LTS under kernel and modules with: for i in $(ls -1 *lts*); do rm -r $i; done
  • ran clr-boot-manager update
  • ran reboot

After the reboot, the system got stuck but booted fine after a hard shutdown. I then ran swupd repair --picky which created LTS initrd symlinks.

$ ls /usr/lib/kernel/
cmdline-5.5.8-917.native
config-5.5.8-917.native
default-native
initrd-org.clearlinux.current.5.3.1-11
initrd-org.clearlinux.current.5.6.0.rc1-12
initrd-org.clearlinux.current.5.6.0.rc2-13
initrd-org.clearlinux.current.5.6.0.rc4-14
initrd-org.clearlinux.lts2017.4.14.172-99
initrd-org.clearlinux.lts2018.4.19.108-120
initrd-org.clearlinux.mainline.5.3.0-10
initrd-org.clearlinux.mainline.5.6.0.rc1-11
initrd-org.clearlinux.mainline.5.6.0.rc2-12
initrd-org.clearlinux.mainline.5.6.0.rc4-14
initrd-org.clearlinux.native.5.5.8-917
install.d
org.clearlinux.native.5.5.8-917
System.map-5.5.8-917.native
vmlinux-5.5.8-917.native

$ ls /usr/lib/modules/
5.5.8-917.native


$ sudo clr-boot-manager list-kernels
* org.clearlinux.native.5.5.8-917

I’m not sure if the current and mainline initrd-org files can also be deleted or if they are needed. Suggestion?

Sorry, I meant to say sudo systemctl start boot.mount That will let you get into the editing the boot menu. The other thing to try if you want to see if you can get this fixed would be to.

  1. Add lts bundle again
  2. Set kernel to lts kernel
  3. reboot and verify lts kernel set
  4. set kernel back to 5.x kernel
  5. reboot and verify 5.x kernel is set
  6. remove lts bundle
  7. run sudo swupd repair --picky --force

Do you know how you might have gotten into this state? I wonder if the lts kernel was still set while the bundle was removed causing the entry to still be left in /boot I am going to test this now.

Edit: I tried that and it switched back to the 5.x kernel without issue. I re-added the lts bundle and then sudo systemctl start boot.mount and edit /boot/loader/loader.conf to set the default kernel to the 5.x kernel and it booted that without issue. This folder is where you can do the cleanup. Just for fun I edited the loader.conf with some bad garbage in it and ran sudo swupd repair --picky --fix and it fixed it so I am not sure why yours has artifacts left.

I currently reinstall the LTS packages, which i had installed before and will then try your suggestions.

Not really, it might be happen while trying to get nvidia drivers to work. At least that was the time when i installed the LTS and LTS_DKMS kernels.


EDIT: After reinstalling the LTS bundles, applying LTS and back to Native i’m where i started. No LTS in clr-boot-manager list-kernels but a lot of stuff in /usr/lib/kernel and also 4.19.108-120.lts2018 in /usr/lib/modules.

Also, i have to be a bit more careful with swupd repair --picky --force, because this removes my printer/scanner drivers. :slight_smile:

@eadams somehow my whole system is a bit messed up know. I’m not able to shutdown or reboot without the system getting stuck, i have to power off it the hard way. Also journalctl -p 3 -xb gives me some errors.

-- Logs begin at Mon 2020-03-09 21:21:20 CET, end at Mon 2020-03-09 21:46:22 CET. --
Mar 09 21:44:20 clr-a564e0b263f64af281a30de9f04d3957 systemd-udevd[409]: /usr/lib/udev/rules.d/64-md-raid-assembly.rules:30 Invalid value "/sbin/mdadm --incremental --export $devnode --offroot ${DEVLINKS}" for IMPORT (char 55: invalid substitution type), ignoring, but please fix it.
Mar 09 21:44:20 clr-a564e0b263f64af281a30de9f04d3957 systemd-udevd[409]: /usr/lib/udev/rules.d/84-nm-drivers.rules:10 Invalid value "/bin/sh -c 'ethtool -i $1 | sed -n s/^driver:\ //p' -- $env{INTERFACE}" for PROGRAM (char 24: invalid substitution type), ignoring, but please fix it.
Mar 09 21:44:20 clr-a564e0b263f64af281a30de9f04d3957 systemd-tty-ask-password-agent[450]: Failed to acquire /dev/ttyS0: Inappropriate ioctl for device
Mar 09 21:44:20 clr-a564e0b263f64af281a30de9f04d3957 systemd-tty-ask-password-agent[450]: Failed to show password: Inappropriate ioctl for device
Mar 09 21:44:29 dellicious kernel: nvidiafb: unknown NV_ARCH
Mar 09 21:44:30 dellicious gdm-autologin][681]: gkr-pam: unable to locate daemon control file
Mar 09 21:44:30 dellicious kernel: nouveau 0000:01:00.0: bus: MMIO read of 00000000 FAULT at 409800 [ TIMEOUT ]
Mar 09 21:44:32 dellicious kernel: nouveau 0000:01:00.0: gr: init failed, -16
Mar 09 21:44:40 dellicious kernel: [drm:intel_pipe_update_end] *ERROR* Atomic update failure on pipe A (start=62 end=63) time 2684 us, min 2146, max 2159, scanline start 2104, end 168
Mar 09 21:45:02 dellicious kernel: nouveau 0000:01:00.0: gr: FECS falcon already acquired by gr!
Mar 09 21:45:02 dellicious kernel: nouveau 0000:01:00.0: gr: init failed, -16
Mar 09 21:45:08 dellicious kernel: [drm:intel_pipe_update_end] *ERROR* Atomic update failure on pipe A (start=12 end=13) time 2910 us, min 2146, max 2159, scanline start 1817, end 2170
Mar 09 21:45:08 dellicious kernel: [drm:intel_pipe_update_end] *ERROR* Atomic update failure on pipe A (start=9 end=10) time 2778 us, min 2146, max 2159, scanline start 1816, end 2150
Mar 09 21:45:11 dellicious kernel: [drm:intel_pipe_update_end] *ERROR* Atomic update failure on pipe A (start=22 end=23) time 2645 us, min 2146, max 2159, scanline start 1827, end 2145
Mar 09 21:45:12 dellicious kernel: [drm:intel_pipe_update_end] *ERROR* Atomic update failure on pipe A (start=65 end=66) time 3174 us, min 2146, max 2159, scanline start 2041, end 172
Mar 09 21:45:17 dellicious kernel: [drm:intel_pipe_update_end] *ERROR* Atomic update failure on pipe A (start=10 end=11) time 2682 us, min 2146, max 2159, scanline start 2068, end 133
Mar 09 21:46:07 dellicious kernel: [TTM] Buffer eviction failed
Mar 09 21:46:22 dellicious kernel: nouveau 0000:01:00.0: DRM: failed to idle channel 1 [DRM]

EDIT: blacklisted nouveau and nvidiafb, reboot and shutdown works again. This also reduced the errors:

-- Logs begin at Mon 2020-03-09 22:02:12 CET, end at Mon 2020-03-09 22:14:37 CET. --
Mar 09 22:10:52 clr-f0e07f34c58545739db4c2840dfa559a systemd-udevd[410]: /usr/lib/udev/rules.d/64-md-raid-assembly.rules:30 Invalid value "/sbin/mdadm --incremental --export $devnode --offroot ${DEVLINKS}" for IMPORT (char 55: invalid sub>
Mar 09 22:10:52 clr-f0e07f34c58545739db4c2840dfa559a systemd-udevd[410]: /usr/lib/udev/rules.d/84-nm-drivers.rules:10 Invalid value "/bin/sh -c 'ethtool -i $1 | sed -n s/^driver:\ //p' -- $env{INTERFACE}" for PROGRAM (char 24: invalid sub>
Mar 09 22:10:52 clr-f0e07f34c58545739db4c2840dfa559a systemd-tty-ask-password-agent[448]: Failed to acquire /dev/ttyS0: Inappropriate ioctl for device
Mar 09 22:10:52 clr-f0e07f34c58545739db4c2840dfa559a systemd-tty-ask-password-agent[448]: Failed to show password: Inappropriate ioctl for device
Mar 09 22:11:00 dellicious gdm-autologin][680]: gkr-pam: unable to locate daemon control file
Mar 09 22:11:11 dellicious kernel: ath10k_pci 0000:3b:00.0: Could not init core: -110

If you can come up with reproduction steps and file it at Issues · clearlinux/distribution · GitHub that would help a lot. I have not been able to reproduce this. I was trying this on version 32400.

After todays update (32580), the bundle linux-lts2018-dev was installed again (i removed it before). I bundle-remove’d it and as far as i can tell everything looks good now. Thank you for your help, also to @william.douglas

1 Like

This bug still exists. I installed two LTS kernels yesterday, and after removing them with swupd and doing a clr-boot-manager update, they still appeared in the boot menu and the kernel and config files were still in the EFI partition. I deleted them manually and did another clr-boot-manager update and that corrected the problem.