Unable to boot after update - flashing cursor only

As a quick reference, to boot the previous kernel, smash the spacebar repeatedly during early boot to get into the EFI boot manager, then scroll to the appropriate kernel entry.

Ok but this is for desktop and physically accessible machines, we keep forgetting, how is this possible on a remote Clear Linux bare metal server accessible only via ssh when the TCP/IP stack gets loaded???

I feel like there is a large chasm between having this feature and people knowing of this feature. It is a tremendous feature (and I couldnā€™t switch to a distro that doesnā€™t do this as itā€™s so important), but hardly anyone knows that it exists, or how to use it till after they run into an issue and end up needing support.

Havenā€™t a clue how to solve without adding a timeout on the bootmenu by default.

Those systems tend to use ipmi style controls or other vendor specific management frameworks to enable remote bios management and console access.

It is a hard default to get right. For some use cases a timeout ideal but for others it is really just a useless slowdown. It is an unfortunately large chasm between these. This does bring up a feature being discussed on clr-boot-manager though. @dorileo what do you think about setting the efi nextboot for the kernel when we update and then having the service that tests boot make it permanent if the system came up?

Yes they do but they wonā€™t give the user the opportunity to hit the space bar a million times for them to bring up some menu to select a previous kernelā€¦The network latency on IPMI is too high and easily miss this opportunityā€¦

Doesnā€™t the Linux world have kernel cache same like on macOS?

So it would appear that setting the timeout manually makes sense for this use case.

In some sense yes but consider kernel cache, the system already has a reliable record of kernel extensions previously used to successfully boot the system based on last boot, if a new updated kernel fails to boot then look at the kernel cache, resolve dependency issues and boot and or point to the last kernel and kernel extension cache and bootā€¦all without user intervention.

I tried your solution. I was able to mount the partitions appropriately, but swupd reported it ran out of space on the device when downloading the necessary files and aborted. Iā€™m assuming that this is because the rootfs when booting from the iso is only about 500mb.

Any suggestions now?

Ah sorry, youā€™ll want to pass -S /mnt/var/lib/swupd to the swupd command as well to tell it where the statedir is.

swupd worked perfectly once I added in that -S flag. Thanks very much. Iā€™m back up and running.

I successfully updated to 32600 with no issues. Iā€™ll wait here for a bit and update in a few versions.

I was thinking about this a bit more. When you said LUKS encryption of the rootfs is not part of the test scope, I would advocate that you either add it to your test scope or make it more clear in the installer that it is a potentially unstable option. LUKS is not an uncommon use case and the installer offers it with a very visible checkbox. If it is not part of your test scope I was thinking that should be mentioned when installing, or simply donā€™t offer LUKS in the installer and let the user do it themselves, so they understand implicitly that it is not necessarily going to be checked for stability.

I have had a similar issue hereā€¦ However I canā€™t even know what the issue is! It gets stuck on checking sda1. Iā€™m reinstalling to see if the issue gets solved.

Did you try booting from a Live USB and then downgrading as per the instructions above? Might save you the reinstall.

So I was actually wrong about this. It is something we test currentlyā€¦ and those tests pass. Unfortunately this looks to be related to something we canā€™t easily replicate without specific hardware that is a higher bar for our automated testing to account for.

I am sadly also wrong about having a fix today. We are having some troubles with making a good release for other content reasons. I am trying to get something reasonable in place for tomorrows build but I canā€™t make any guarantees.

Just an FYI we are making a release for 32640, this does not have the fix for the LUKS issue but has a lot of changes we were building up and our default policy is that if we donā€™t have regressions over the previously release we release. The next build has a fix weā€™ve done testing on and confirmed works for the LUKS issue but it has other changes in it so it may run into QA problems tomorrow. The next release after 32640 will have this issue resolved regardless of it is 32650 or not though.

Thanks very much for letting us know when the LUKS issue will be resolved. Iā€™ll make sure to catch the update after 32640!

Actually after some testing I believe the problem is encryption. What I did:
Using an older installer I was able to install with an encrypted partition, but after updating I had the same issue. Then I tried a newer image, I was able to install with an encrypted partition, but after typing the diskā€™s password Clear wouldnā€™t boot. Lastly I tried reinstalling the latest version and everything worked, I hope this gets fixed in the futureā€¦