Clr-install fails with custom mix on own mirror

hgdahmen · January 10, 2020, 11:58am

Hi,

I have created a custom mix and it fails to install where it has worked before. I re-created my install
USB sticks using the guide on the website (live server image on usb) and now it does not work anymore. When trying to use my installer config YAML file that specifies my own mirror with my own mix it fails on the swupd os-install run.

The log file shows that it copies offline content even though I specified that it should not. Then it says that my mirror version is too far behind upstream (my mix has version 120) and forces a much higher version. Then it complains that it can not download the manifests for this version, of course.

What do I need to do to make this work again?

2020/01/10 11:57:33 [INF] Overriding version from latest to 32050 to enable offline install
2020/01/10 11:57:33 [DBG] Overwriting stateDir with offline content
2020/01/10 11:57:33 [DBG] losetup -j /mnt/media/images/rootfs.img -O NAME -n
2020/01/10 11:57:33 [DBG] Using default shellProxy.DefaultGetProxyValue
2020/01/10 11:57:33 [DBG] [Previous line repeated 4 times]
2020/01/10 11:57:33 [DBG] cmd.Env: []
2020/01/10 11:57:33 [DBG] Extracting offline content in squashfs to target media
2020/01/10 11:57:33 [DBG] unsquashfs -d /tmp/install-438710953 -f /dev/loop0 -e /var/lib/clr-installer/offline-content
2020/01/10 11:57:33 [DBG] Using default shellProxy.DefaultGetProxyValue
2020/01/10 11:57:33 [DBG] [Previous line repeated 4 times]
2020/01/10 11:57:33 [DBG] cmd.Env: []
2020/01/10 11:57:35 [DBG] Parallel unsquashfs: Using 8 processors
2020/01/10 11:57:35 [DBG] 33452 inodes (42385 blocks) to write
2020/01/10 11:57:35 [DBG] ^M[=====|                                                      ]  3600/42385   8%!!(MISSING)
(MISSING)
2020/01/10 11:57:40 [DBG] ^M[===========================================================|] 42385/42385 100%!!(MISSING)
(MISSING)
2020/01/10 11:57:40 [DBG] created 32663 files
2020/01/10 11:57:40 [DBG] created 26 directories
2020/01/10 11:57:40 [DBG] created 730 symlinks
2020/01/10 11:57:40 [DBG] created 0 devices
2020/01/10 11:57:40 [DBG] created 0 fifos
2020/01/10 11:57:40 [INF] Installing base OS and configured bundles
2020/01/10 11:57:40 [DBG] Installing bundles: os-core, os-core-update, bootloader, freitag-all, NetworkManager, sysadmin-basic, kbd, kernel-native
2020/01/10 11:57:40 [DBG] Bundle os-core was already installed with the core bundles, skipping
2020/01/10 11:57:40 [DBG] Bundle os-core-update was already installed with the core bundles, skipping
2020/01/10 11:57:40 [DBG] swupd os-install --certpath=freitag-swupd-root.pem --statedir-cache=/var/lib/clr-installer/offline-content --url=https://192.168.1.203 --path=/tmp/install-438710953 --statedir=/tmp/install-438710953/var/lib/swupd -V 32050 --force --no-boot-update --json-output -B os-core-update,openssh-server,bootloader,freitag-all,NetworkManager,sysadmin-basic,kbd,kernel-native
2020/01/10 11:57:40 [DBG] Using default shellProxy.DefaultGetProxyValue
2020/01/10 11:57:40 [DBG] [Previous line repeated 4 times]
2020/01/10 11:57:40 [DBG] cmd.Env: []
2020/01/10 11:57:40 [DBG] [
2020/01/10 11:57:40 [DBG] { "type" : "start", "section" : "os-install" },
2020/01/10 11:57:40 [DBG] { "type" : "warning", "msg" : "/tmp/install-438710953//usr/share/clear/versionstamp does not exist!" },
2020/01/10 11:57:40 [DBG] { "type" : "progress", "currentStep" : 1, "totalSteps" : 9, "stepCompletion" : -1, "stepDescription" : "load_manifests" },
2020/01/10 11:57:40 [DBG] Target OS: : Setting indeterminate progress for task load_manifests
2020/01/10 11:57:40 [DBG] { "type" : "info", "msg" : "Installing OS version 32050" },
2020/01/10 11:57:40 [DBG] { "type" : "error", "msg" : "Certificate verification error - unable to get local issuer certificate" },
2020/01/10 11:57:40 [DBG] [Previous line repeated 1 time]
2020/01/10 11:57:40 [DBG] { "type" : "warning", "msg" : "Removing corrupt Manifest.MoM artifacts and re-downloading..." },
2020/01/10 11:57:40 [DBG] { "type" : "error", "msg" : "Failed to retrieve 32050 MoM manifest" },
2020/01/10 11:57:40 [DBG] { "type" : "error", "msg" : "Unable to download/verify 32050 Manifest.MoM" },
2020/01/10 11:57:40 [DBG] { "type" : "info", "msg" : " Installation failed" },
2020/01/10 11:57:40 [DBG] { "type" : "progress", "currentStep" : 1, "totalSteps" : 9, "stepCompletion" : 100, "stepDescription" : "load_manifests" },
2020/01/10 11:57:40 [DBG] Target OS: : Task load_manifests completed
2020/01/10 11:57:40 [DBG] { "type" : "end", "section" : "os-install", "status" : 4 }
2020/01/10 11:57:40 [DBG] ]
2020/01/10 11:57:40 [ERR] An error occurred executing command: "swupd os-install --certpath=freitag-swupd-root.pem --statedir-cache=/var/lib/clr-installer/offline-content --url=https://192.168.1.203 --path=/tmp/install-438710953 --statedir=/tmp/install-438710953/var/lib/swupd -V 32050 --force --no-boot-update --json-output -B os-core-update,openssh-server,bootloader,freitag-all,NetworkManager,sysadmin-basic,kbd,kernel-native". Error: exit status 4

bun.k.tan · January 10, 2020, 7:53pm

Greetings @hgdahmen!

swupd error code 4 says: Unable to download or read MoM manifest

Before you start the installation, can you please verify you have connectivity to your HTTP server hosting the bundles?

hgdahmen · January 11, 2020, 12:16pm

Hi,
of course I have checked this already. The problem is exactly what I described in the first post. It starts with this:
2020/01/10 11:57:33 [INF] Overriding version from latest to 32050 to enable offline install`

I did some digging and it can be traced back to the control flow starting at the linked line of code:

github.com

clearlinux/clr-installer/blob/8a7e4b486cce3367f7cfefe06c4523460f3f9963/controller/controller.go#L522


      
          			}
          		}
          	}
          
          
	bundles := md.Bundles
          
          
	if md.Kernel.Bundle != "none" {
          		bundles = append(bundles, md.Kernel.Bundle)
          	}
          
          
	if swupd.IsOfflineContent() {
          		msg := utils.Locale.Get("Copying cached content to target media")
          		prg = progress.NewLoop(msg)
          		log.Info(msg)
          
          
		if err := utils.ParseOSClearVersion(); err != nil {
          			return prg, err
          		}
          		log.Info("Overriding version from %s to %s to enable offline install", version, utils.ClearVersion)
          		version = utils.ClearVersion

The problem with the live server image is that the directory “/var/lib/clr-installer/offline-content” is populated.
This is the default directory which is checked by the function here

github.com

clearlinux/clr-installer/blob/8a7e4b486cce3367f7cfefe06c4523460f3f9963/swupd/swupd.go#L229


      
          func IsCoreBundle(bundle string) bool {
          	for _, curr := range CoreBundles {
          		if curr == bundle {
          			return true
          		}
          	}
          	return false
          }
          
          
// IsOfflineContent checks that the offline content dir is valid for swupd to consume
          func IsOfflineContent() bool {
          	info, err := os.Stat(conf.OfflineContentDir)
          	if err != nil {
          		return false
          	}
          
          
	// Check required permissions for offline content directory
          	if !info.IsDir() || info.Mode().Perm() != os.FileMode(0700) {
          		return false
          	}

to determine if cached offline content should be installed. The two linked pieces of code in conjunction with the special setup of the live server image (which is recommended for USB install sticks) is causing the behaviour that ignores the YAML setting regarding offline installs and leads to the erraneous behaviour that tries to download a non-existent version’s manifest file from my own mix.

The quick fix is to rm -rf the offending directory from the live stick prior to installation.

Regarding clr-install, may this be considered as a bug or is it intentional? If it’s a bug then I’d file an issue at github.

hgdahmen · January 11, 2020, 1:49pm

After further inspection it looks like the offline setting in the YAML file is not meant to control if the current installation is an offline install, but that the resulting target system should do offline installations. So the behaviour maybe isn’t that erraneous, but it still is problematic to just override the install version when the mirror doesn’t host that version. It essentially means that I must create my own USB installer image variation when I want to install my own mix on real hardware or use the quick fix mentioned in my last post.

mhorn · January 13, 2020, 6:58pm

@hgdahmen The “offline” in the YAML file is used to instruct the image build to include offline content in the image so that the network is not needed for a baseline installation.

github.com

clearlinux/clr-installer/blob/8a7e4b486cce3367f7cfefe06c4523460f3f9963/scripts/InstallerYAMLSyntax.md

# Installer YAML Syntax

This document describes the syntax for constructing a clr-installer configuration file.

## Environment Variables
Environment variables can be defined which will be used when installation commands are executed. These are most commonly used for `pre-install` and `post-install` hooks.
```yaml
env:
  <variable>: <value>
```

## Device Aliases
To avoid changing a device name in multiple locations in the `targetMedia`, device aliases can be used to simply change between image files and physical devices.
```yaml
# switch between aliases in order to install to an actual block device
# i.e /dev/sda
block-devices: [
   {name: "bdevice", file: "os-image.img"}
]
```

This file has been truncated. show original

The published installer images are intended for the mainstream installation only. You are correct if you create your own mix, and you want an installer image for that mix, then you need to create your own installer image. All of the needed files (YAML and post/pre scripts) are published for each release of Clear Linux OS:
https://download.clearlinux.org/current/config/image/

hgdahmen · January 14, 2020, 11:16am

@mhorn No I don’t need to create my own installer image, I can just do:
rm -rf /var/lib/clr-installer/offline-content to turn the official installer image into a generic one and use my YAML file to install my own mix.

Is there any chance a YAML or cmdline setting could be made to achieve the same behaviour (disable using offline content)?

mhorn · January 14, 2020, 7:11pm

@hgdahmen Modifying a release installer image to force install your own mix is not a supported method. There are several critical dependencies between the installer, swupd (package manager) and the image in which they run. If you need an installer image for your own mix, the supported method is to create a new installer based on your mix – which will then need to contain all of the install required bundles.

If you want to take the approach of modifying a validated image that is your choice, but it is not something we will plan to support. If you really want to delete the offline content, then you could use a post-processing hook in the YAML (documented in the InstallerYAMLSyntax.md.

hgdahmen · January 15, 2020, 11:08am

@mhorn I think we are talking about different things. I don’t want any offline content or build or even host an image, I just want to use the clr-install utility to do an online install, which was possible with the Intel supplied USB installer image from roughly a year ago. The offline functionality broke my use-case. Nevermind though, I’ve got my problem solved.