What makes Clear Linux perform much better compared to other distros?

[unix_man]

What makes Clear Linux perform much better compared to other distros?

I know for a fact that it does use special optimisation flags when compiling the packages into binaries, it uses some power tweak tools or something; I am also aware that it patches packages or something ~ I was wondering if somebody could elaborate on what they mean by patching packages please?

Is there anything else Clear Linux does to help optimise the distro?

[SPAstef]

Another thing I like is its “almost-stateless” system. I say almost because it’s really up to the user to keep it that way, and sometimes it’s not as easy as one may think. Anyway, enforcing bundles updates with system updates really helps with bugs (e.g. "which version of PHP do you have? Ok, and what about nginx? Ok, and MariaDB? Good, now tell me your Firefox version…), and the cool thing is that other than having optimized software, it usually has the most up-to-date software out there! So, more performance, less clutter, less coupling between user stuff/system stuff, always up-to-date software… Anything else?

[unix_man]

@SPAstef

and the cool thing is that other than having optimized software, it usually has the most up-to-date software out there!

So with Gentoo, people tend to use optimised flags to build their packages. So if people chose to use the most up-to-date packages on Gentoo, why is it that it still underperforms over Clear Linux?

@SPAstef

Anything else?

Ok so truth be told, I really like Clear Linux and I want to use it, but I find that it takes up way quite a bit of RAM, storage space, and its package manager seems more complicated to use over other package managers.

According to a Gentoo user as to why Clear Linux uses a lot of space:

My understanding of Clear Linux is that its binaries are big because it does run time CPU detection to provide the performance gains over other binary distros.
That’s what makes it big because it installs the same code for lots of different CPUs but only ever uses one.
Its all a binary distro can do, where it doesn’t know what CPU it will be run on at build time.

Gentoo does CPU optimisation at build time, so you only ever build code for one CPU, unless you choose not to.

Is this true?

If so, then is it possible to use Gentoo and to build Clear Linux but only keep the packages that I need?

[SPAstef]

@unix_man

why is it that it still underperforms over Clear Linux?

I think it might be because CL has also optimized system calls, not just packages. I never used Gentoo and I’m not a CL developer, so I cannot say for sure… But having an optimized OS, and not just packages, really should help IMO.

@unix_man

it takes up way quite a bit of RAM, storage space, and its package manager seems more complicated to use.

Complicated? Nah, If you ever used apt, and I’m pretty sure you did, THAT is complicated. On CL you basically have 4 commands:

1. swupd update → because of the stateless architecture, this command searches for both system and package updates and automatically installs all of them! Yay, no more apt update && upgrade (and even dist-upgrade)!
2. swupd bundle-add → Install a bundle. You dont have to install a dozillion packages if you want to run a web-server or even build the entire OS: just do swupd bundle-add something-dev and you have all you need to build "something (be it GNOME, the OS itself, or something else…) (Note: I didn’t try to do this, but I’m pretty confident it should work as intended)
3. swupd bundle-remove → remove a bundle
4. swupd-repair → can actually repair a lot of user-error done on system partitions. Obviously not everything. This is a bit harder to use, but fixing errors is by definition an hard task, isn’t it?

Memory usage on my PC is usually 4GB out of 8GB of RAM. Disk usage is actually quite high, because for some reason CL comes with 500MB of fonts (including ancient Sanskrit, biblic Hebrew and probably Aztec) and other 500MB of themes and icons. Basically, summing all things up, you have almost 2GB of custom/optional stuff… But fear not, they said they removed most of this clutter in the latest GNOME 3.36 update (should be coming any moment this week).

[unix_man]

@SPAstef

Complicated? Nah, If you ever used apt, and I’m pretty sure you did, THAT is complicated. On CL you basically have 4 commands:

I have used apt before and sure it is a bit confusing but I have used pacman far more.

@SPAstef

swupd update → because of the stateless architecture

What is a stateless architecture?

@SPAstef

swupd bundle-add → Install a bundle.

Is a bundle where you can build a package, sorry I don’t quite know?

swupd-repair → can actually repair a lot of user-error done on system partitions. Obviously not everything. This is a bit harder to use, but fixing errors is by definition an hard task, isn’t it?

Wait, I thought swupd is a package manager? Why would it also be able to fix up system partitions?

@SPAstef

Memory usage on my PC is usually 4GB out of 8GB of RAM.

That is quite a lot, Is there like a skinned down version of Clear Linux where its kept very minimal and only the required packages and stuff runs?

@SPAstef

Disk usage is actually quite high, because for some reason CL comes with 500MB of fonts (including ancient Sanskrit, biblic Hebrew and probably Aztec) and other 500MB of themes and icons. Basically, summing all things up, you have almost 2GB of custom/optional stuff… But fear not, they said they removed most of this clutter in the latest GNOME 3.36 update (should be coming any moment this week).

Hopefully, but I would want to use a tilling window manager anyways.

[SPAstef]

Stateless achitecture is the fancy term CL team uses to say that system updates and package updates are basically the same thing (you can look up in the CL docs for a more detailed explanation: Clear Linux Stateless.
The 4GB of memory usage I’m talking about are when doing normal work (GNOME, Firefox, VS code, Telegram… stuff like that), not on idle. I chose to tell you this usage because unless I misunderstood your use-case and you’re talking about a server, nobody uses a PC just to stare at the wallpaper, and idle memory usage really depends on what you actually mean by “idle”.
If you have some familiarity with Object Oriented Programming, a package is the equivalent of a primitive data type (meaning it’s in some way “atomic”), while bundles are collections of packages and possibly other bundles.
By fixing system partitions I meant fixing problems like overwriting system files and stuff like that. It does not fix filesystem issues.
I think they distribute a “desktopless” version of CL, where you can download any DE you want later, just it’s a bit longer to setup and not really ready-to-use for a normal user.

[unix_man]

@SPAstef

Stateless achitecture is the fancy term CL team uses to say that system updates and package updates are basically the same thing (you can look up in the CL docs for a more detailed explanation: Clear Linux Stateless.

I see.

@SPAstef

The 4GB of memory usage I’m talking about are when doing normal work (GNOME, Firefox, VS code, Telegram… stuff like that), not on idle.

Oh I see I thought you meant like when you just startup the PC.

@SPAstef

I chose to tell you this usage because unless I misunderstood your use-case and you’re talking about a server, nobody uses a PC just to stare at the wallpaper, and idle memory usage really depends on what you actually mean by “idle”.

Oh lol.

@SPAstef

If you have some familiarity with Object Oriented Programming, a package is the equivalent of a primitive data type (meaning it’s in some way “atomic”), while bundles are collections of packages and possibly other bundles.

Oh so is it like the equivilant to adding ppas if I typed swupd bundle-add ?

@SPAstef

By fixing system partitions I meant fixing problems like overwriting system files and stuff like that. It does not fix filesystem issues.

Oh I see.

@SPAstef

I think they distribute a “desktopless” version of CL, where you can download any DE you want later, just it’s a bit longer to setup and not really ready-to-use for a normal user.

I see, I don’t want a DE, I just want a tiling window manager so that might work for me just fine.

[SPAstef]

Not exactly. PPAs are repositories (swupd has beta 3rd-party repositories support, but there are none that I know of). Bundles are… Let’s say “purpose-packages”. Conceptually, a package contains something (especially libraries) that is not enough to do what you actually want to do. A bundle is a super-package that contains all the packages to do something (build GNOME, setup a webserver, compile and debug C programs…), So you don’t have to install a ton of packages manually, because installing a bundle automatically installs all the packages you need.

[anon93869965]

OT

@SPAstef

swupd has beta 3rd-party repositories support, but there are none that I know of

At least:

$ sudo swupd 3rd-party list 
Swupd 3rd-party repositories:
 - greginator: https://clear.greginator.xyz/

Total: 1

He does a great job so far with this repo, so it is worth mentioning.

[SPAstef]

I actually did know about that, I even added it, but last week it was offline and I’ve not been able to query it (it wasn’t swupd fault), so I thought he just shut it down…

[anon93869965]

Maybe because of the update.

Testing 3rd-party bundles (including ffmpeg)

Update 17-04-2020: to get the latest information keep track of my GitHub I will keep updating the readme and I will add more guides (like how to build your own 3rd-party repo) in the future. The GitHub organization that I created also contains all the package configs that I used. The last few days I have been working on getting some 3rd-party repositories working on my local machine and now I have hosted them on my server for anyone to enjoy, please note some workarounds are still needed to get…

[doct0rHu]

As for RAM usage, I’ve 32 GB of RAM, with no foreground applications running, it uses about 3-4 GB. I’m using Gnome and just a few extensions. The only costly background service I can think of is Dropbox.

I’m quite okay about the RAM usage. But since I haven’t run any ‘minimal’ Linux distribution before, I cannot say how this compares to others.

[puneetse]

It’s a good question to dig into. We’ve been working on adding a page to the documentation to elaborate. It should be published any day now I’ll post a link here once it is.

In short, Clear Linux favors runtime performance.That’s not to say we’re not conscious about other aspects though.

@unix_man

I know for a fact that it does use special optimisation flags

This is true we tend to use more aggressive flags. Also the compilers and low level libraries themselves are also patched.

• GitHub - clearlinux-pkgs/gcc
• GitHub - clearlinux-pkgs/glibc
• GitHub - clearlinux-pkgs/llvm

@unix_man

I was wondering if somebody could elaborate on what they mean by patching packages please

It depends on the package - could be simple config changes, changing default buffers sizes, changing the order of loading things,etc… In general the approach is to upstream any enhancements if possible

[sunnyflunk]

@SPAstef

and the cool thing is that other than having optimized software, it usually has the most up-to-date software out there!

So with Gentoo, people tend to use optimised flags to build their packages. So if people chose to use the most up-to-date packages on Gentoo, why is it that it still underperforms over Clear Linux?

What evidence do you have of this? I’ve only seen one result lately comparing a phoronix CL result with their properly configured gentoo (same CPU and similar hardware) where their numbers were much better (albeit it was only a couple of tests which isn’t representative). Any time phoronix uses a gentoo based system, it’s not even compiled for the host CPU (so same as a generic distro)

Ok so truth be told, I really like Clear Linux and I want to use it, but I find that it takes up way quite a bit of RAM, storage space, and its package manager seems more complicated to use over other package managers.

According to a Gentoo user as to why Clear Linux uses a lot of space:

My understanding of Clear Linux is that its binaries are big because it does run time CPU detection to provide the performance gains over other binary distros.

That’s what makes it big because it installs the same code for lots of different CPUs but only ever uses one.
Its all a binary distro can do, where it doesn’t know what CPU it will be run on at build time.

Gentoo does CPU optimisation at build time, so you only ever build code for one CPU, unless you choose not to.

Is this true?

If so, then is it possible to use Gentoo and to build Clear Linux but only keep the packages that I need?

It does duplicate some libraries and binaries, but that won’t change RAM usage and only 5-10% extra disk space as it’s only done for a handful of packages. You can get slightly higher RAM use from compile flags making binaries larger and including features that you don’t need, which happens with all distros, but possibly more due to bundles including more packages than you might want. The use of bundles rather than packages results in greater disk space as you can’t minimize for your use cases well. In gentoo you would be able to compile out completely support for functions you don’t need (for example printing or samba shares), at the cost of compilng your own system, but would use a lot of disk space in compiling the packages and fetching the sources!

The feature Gentoo lacks is build support for PGO, which is used sparingly in CL and not in combination with AVX2/AVX512 builds last time I looked.

[unix_man]

Hi,

@puneetse

Also the compilers and low level libraries themselves are also patched.

When you say the packages are patched, do you mean the Clear Linux project forks gcc, glibc and llvm and modify those packages?

[unix_man]

@sunnyflunk

What evidence do you have of this? I’ve only seen one result lately comparing a phoronix CL result with their properly configured gentoo (same CPU and similar hardware) where their numbers were much better (albeit it was only a couple of tests which isn’t representative). Any time phoronix uses a gentoo based system, it’s not even compiled for the host CPU (so same as a generic distro)

I saw one phronix test and Gentoo was way behind CL and somebody from reddit mentioned that Clear Linux’s optimisation mainly comes from patching the packages. But I guess they were not optimised for that particular hardware, as you have mentioned. Do you have the link to the test that you saw?

@sunnyflunk

The feature Gentoo lacks is build support for PGO

Sorry what is a PGO?

[doct0rHu]

One example:

clearlinux-pkgs/gcc/blob/b7478716e6049d1e79668d5219c351041294e7ae/optimize.patch

[sunnyflunk]

@unix_man

I saw one phronix test and Gentoo

It probably used Calculate Linux (a derivative) which is distributed compiled like every other generic distro. Packages were certainly not compiled for the system CPU or tweaked for optimization. Also you need to understand the phoronix tests, many don’t test the distribution provided package so comparing the results across distributions (to determine the speed of the distribution) is actually insane!

@unix_man

Do you have the link to the test that you saw?

I was thinking of this comment

Turns out it wasn’t the same CPU, it was actually a weaker CPU in the gentoo system that scores 2/3rds of the one tested in CL according to passmark. But it does compromise the comparison nonetheless.

There hasn’t been a proper benchmark done comparing an optimized gentoo vs CL (I’d argue there hasn’t been a proper benchmark comparing any distributions, certainly not by phoronix).

@unix_man

Sorry what is a PGO?

Profile guided optimization.

[unix_man]

@sunnyflunk

It probably used Calculate Linux (a derivative) which is distributed compiled like every other generic distro. Packages were certainly not compiled for the system CPU or tweaked for optimization. Also you need to understand the phoronix tests, many don’t test the distribution provided package so comparing the results across distributions (to determine the speed of the distribution) is actually insane!

I see that makes more sense.

@sunnyflunk

Turns out it wasn’t the same CPU, it was actually a weaker CPU in the gentoo system that scores 2/3rds of the one tested in CL according to passmark. But it does compromise the comparison nonetheless.

I don’t get why don’t they just test on the same hardware?

@sunnyflunk

There hasn’t been a proper benchmark done comparing an optimized gentoo vs CL (I’d argue there hasn’t been a proper benchmark comparing any distributions, certainly not by phoronix).

Yeah you are right about that.

@sunnyflunk

Profile guided optimization.

Is this some flag for building applications or something?

[doct0rHu]

PGO is an optimization technique that based on profiling. It first compile the code, then profile it(run it in test cases), then according to heuristics, comiple again with optimisation.