Why does ClearLinux use `swupd` and not apt/deb/rpm/...?

Hi folks - We’ve had a few articles posted on some journalism websites and we’re happy to see the attention. Generally we (the ClearLinux team) don’t really go out and partake in the discussion on these forums as we … are just too busy handling the ones that we do communicate!

However, we do read the user comments on e.g. Phoronix and Ars Technica (and others, we admit!) and we often see the same questions posed here, and sometimes people writing comments based on inaccurate or incomplete data. It would be too costly to reply everywhere, so, instead, I’m going to try and write up a few of these “FAQ” type of questions and write out a reaction from the Clear team.

We see plenty of users struggle with swupd. Why couldn’t the ClearLinux team use a packaging mechanism that already exists?

To answer this question, we have to go down to the mechanics of update versus installation. Most packaging systems that exist are not update mechanics, and essentially are doing installation all the time. This sounds reasonable because an update is nothing more than the removal of the old, plus the installation of the new content. Everyone can follow that.

What is painfully evident though is the fact that an install of any package, in order to update it, is a waste of bandwidth, disk space, and processing power. In many cases, only small bits of a package change with each update, but most Linux package management systems do not optimize for updates. (Note: things have changed in recent years, please bear in mind that swupd predates some of them, and swupd predates Clear Linux even).

Second, classic package management requires that the client device perform full dependency resolution. That’s because you can endlessly combine components, and, components may actually conflict! This makes updating complex and error prone.

Third, most package systems allow you to partially update your system. And those upgrades are not done in a way that is reliable - there is no update control point that you must step through to make sure that going from update 1 to update 5 skips a step in point 3 that is needed to keep everything working.

The list isn’t exhaustive here - I might be forgetting some minor points of friction - after all package management is cause for plenty heated debate :).

We didn’t like any of these problems because they all fundamentally interfere with the stability and long-term usability of the OS. All of these problems cause people to “reinstall” the OS - something we want to avoid. Worse, it causes people to stop updating their OS because they know it will break, and now they are no longer installing security updates.

We opted to not inherit all these problems and it was exactly why swupd was created to begin with.

  • swupd does not allow you to partially upgrade your OS - you can add/remove parts, but only from the same version.
  • conflicts are not allowed!
  • an update is designed to be close to ideal size - the less changed, the smaller the update. The downloader only needs to fetch the exact parts that changed, at the diff level (made using binary deltas).
  • dependency management is done by and large at the server level, through the creation of bundle Manifests. That makes applying the updates a simple linear task, without requiring a complex solver.
  • updates are safe from any moment in the past and pass through update “checkpoints” to make sure that the system can upgrade through ABI breaking changes without any issues.

Now, this doesn’t mean we’re (overly) critical of other package management systems - each has flaws and design issues, and we haven’t stopped working on swupd since forever, but it does mean that we’re not trying to invent “the same thing” here - swupd is fundamentally not compatible or replacing something that didn’t exist. It did something we needed it to do differently than existing solutions.

The other thing in mind is to keep historical perspective: You can, in the case of a Linux OS, not just ask everyone to reinstall because you’re switching package management tooling. We’ve stuck with swupd and any criticism on the design and its effects is fine, but changing to another system is a serious thing that is going to be prohibitive in most cases.

We intend to keep ClearLinux OS going rolling forward - that’s what swupd was for and so far it’s done exactly that.

PS: I will update this article as needed. If you have corrections, please share them.
[edit 2/18 mention update checkpoints, changed some wording]

8 Likes

Thanks for the explanation. It’s incredible that even in the Linux world, some people want the same old thing🤔

I’m not sure I understand what your point is, or what “same thing” you are referring to - can you elaborate?

2/17/2020

After reading arstechnica comments on swupd i think it might take more than a explanation to change their minds.

perplexed

From what i’ve seen distro’s that make life easier for the end user will always win , take Mint for example it’s so easy to use that they are currently on $25,000 a month in donations and Manjaro is about to become a limited partnership business entity for the amount of support it gets for making Arch really easy to use.

Distro’s that make life easier for the devs but harder for the end user with a new set of technobabble commands to learn will always come up against criticism for trying to re-invent the wheel and often end up nothing more than a limited niche distro with a few dedicated followers rather than being mainstream.

Your arguments are valid, but i don’t think it was Intel’s original intention (if at all ?) to create a mainstream disto.
Publicity is great especially with the likes of Phoronix etc, but it creates often unrealistic expectations !
My Linux background is a Gentoo user and often in the forums it was a case to try and educate new users and bring them down from cloud 9 or help them out of the depths of hell :wink:

There is a counter argument to my point about re-inventing the wheel that i tried to avoid , systemd seems to be doing pretty well.

hopefully clearlinux can do the same , intentionally or unintentionally :slight_smile:

My point is that they are arguing for the familiar package manager, because that what they’ve been doing, my point is that Linux should always be pushing forward I know swupd isn’t new but the concept is different thant apt and Pacman, and that people who use Linux have a gripe about that, well makes me scratch my head?

I’m not writing this per se to change their minds - I’m writing this so that others, who read the thread, have a place to go to and find out more information for themselves from the source.

Thank you for explaining this aspect of CL. I would suggest it is put in the documentation and/or FAQ so that as many as possible get to read and understand the approach of CL.
I appreciate your effort to do this via the forum.Thanks!

When I have switched several month ago I was also not so very pleased by swupd. Because I was expecting ok its commandline install like apt,yum…whatever:
so I will just have to type swupd install tar …but this was not the case. After reading and using it a bit I have adopted the concept but still there are issues if you want to have chrome/ brave what so ever or ffmpeg…

Luckily there exists flatpak and you can use rpms so this makes CL usable as universal distro for daily tasks otherwise I would have ditched it.

I like swupd…and not forgot that the CL is rolling release,I dont know apt/deb/rpm rolling release distros.

Hi Folks,

I’d like to say that I love the way ClearLinux does things. I am a developer and I’ve got multiple tailored mixes to suit my needs at my company, at home and in a club. Swupd is my friend.

1 Like

I Would love to hear more about what specifically you’re doing here!

2 Likes

@ahkok At home I am using it in VMs as a playground to test things. At work I use it as a video player appliance with a modified mpv. The video player appliance plays back uncompressed video on LED walls and needs to be fast. I found Clear Linux to be fast and very helpful in creating a readily customized distribution that can be deployed onto new boxes to get a running system with minimal user intervention. In a club I’m using it for a POI terminal that solely needs to run a web browser in full screen. The POI terminal and the video player appliance both need to be immune to sudden power loss, so I’ve tweaked my customized distribution to use a read only rootfs. Once I got started with mixer, I found it to be the easiest tool I know to create a customized distribution variation. For some tasks there might be other distributions that better fit my need, but Clear Linux works well, too, and I just need to learn about one set of tools.

1 Like

@hgdahmen have you looked at using the new mixer which adds content as a keyword used in the bundles configuration? It basically lets you define a path to a directory as a bundle rather then a set of packages. This might be useful for some of your content (it was added for 3rd-party bundles so it isn’t something used in upstream Clear Linux at this time).

@william.douglas Could you elaborate a bit more please? I use Clear Linux similarly to @hgdahmen with my own mixes and bundle sets for different scenarios (VMs and dedicated devices but also desktops etc.) and I am very happy with it so far. The only thing missing is a way to automate distribution and updates of (portable) tools that are not distributed with Clear Linux. I began designing a simple scripted solution based on rsync for this purpose but if Clear Linux already has something in place that could be used instead I would be glad to find out more about it. Is there something publicly online about this new feature, how to use it etc.?

https://docs.01.org/clearlinux/latest/guides/clear/swupd-3rd-party.html#create-bundle-of-3rd-party-app-with-mixer