Swupd 41340 upgrade automatically deleted contents of /root

General Discussion
1

i SSH-ed into my machine this morning after stopping work last night, to find the contents of my /root directory completely wiped. this is absolutely insane and i’m still trying to figure out how this is possible.

there were no prints in dmesg, nothing in journalctl besides an update from “version 41300 to version 41340”, the disk and partition are fine besides the root directory. only the following remains “. … .bash_history .ssh”.

the box is on a home LAN. the idea of intrusion makes no sense. (also no external network connections open).

for the life of me i can’t even guess as to what happened? maybe the swupd update??

2

the only user i have on the system, that home directory was untouched. and the root directory was the only directory on the system that was deleted or changed.

it’s basically just an inert home dev box that nothing runs on besides when i’m SSH-ed in doing dev work. so i’m becoming more and more convinced that there must be some bug in swupd that wiped the root directory when it upgraded from 6.7.9-1414.native to 6.8.1-1419.native.

thank god no critical data loss because i work out of a Dropbox folder so there are copies on multiple machines and the cloud of all the data that matters… but there were many other random things in my root folder now destroyed.

3

Same here. The contents of /root were wiped away. I restored .vimrc in the /root folder.

Last night, I updated the system to 41340 (regression also in 41350). Gnome Terminal is unusable on Xorg due to random 1 second delays for keyboard input. So, I rolled back to 41300 wondering if a 41340 regression. Maybe a Gnome 46 regression. I rolled back to 41270.

4

See also “File that should be deleted: /root” when updating to 41340 ? · Issue #3083 · clearlinux/distribution (github.com)

5

This caused me to lose all of my documentation of how I had set up my system :sob:

Sure, I should have had it backed up but this is just a homelab testing system I am trying out clear linux on.

Anyone have any pointers on how I might recover a small text file that got deleted?

6

@btwarden posted a write up for this issue as part of the github issue thread -

We’ve taken actions to block this release and to prevent a recurrence, while we investigate deeper how the problem made it through various quality checks.

We’re deeply sorry for the issues this caused, and are actively bolstering code paths and test procedures at multiple levels to prevent this or similar issues from happening again.

Background:
I made a change a few weeks ago to remove an extraneous entry for /root from the configuration used by systemd-tmpfiles to create and maintain various directories on the platform. It should have been harmless, but instead exposed some weak points in our development and release processes that culminated with deleting the /root directory from updated systems.

We are putting the following measures and failsafes in place to ensure this and a multitude of similar issues are prevented in the future:

  • We’ve already automated tests to catch this type of failure very early in our development process (creating the base files), prior to generating content for even an update candidate.
  • We’re currently adding more complex tests for the update content itself (manifests), to fail on an attempt to delete protected content.
  • We identified code in swupd itself that was supposed to prevent touching files outside of /usr, but wasn’t quite correct, and have fixed it and added more specific protections.
  • We’re reworking how swupd update works with files it doesn’t know about. Specifically it will no longer remove unknown files or directories. Only swupd repair --extra-files-only will remove files that swupd doesn’t know about (with documented caveats). As always, the source code for swupd-client is available for your review.
  • And finally, we’re improving our ability to block a release that’s already in the wild if we find out there’s a serious problem that slipped through.
7

I am sorry this has happened to all involved. I have significantly reduced my risk of my system being “bricked” by controlling when swupd is allowed to update my main local server only after I have verified
that my rescue USB drives have updated successfully. I have a pair of rescue USB thumb drives that update and check that the necessary daemon’s run with out issues. If the update fails on one of the USB rescue drives I don’t update my server. If the swupd update is not successful, I use dd to reimage the updated USB drive back to the last known working update.

I only update my system only after one of the rescue USB drive updates successfully.