[AWS] ClearLinux customized AMI cannot be used as "base" for other AMIs

ClearLinux has caught my attention for its incredibly lean and fast boot/build/compile times, and its unbeatable speed/performance on Intel-based x86 EC2s.

For that reason, I’ve spent the last few months experimenting with Clear Linux on EC2 in my free time.

After spending 2 months tinkering with Clear Linux on AWS, I am having a few problems I can’t seem to figure out.


Problem #1: Packer + ClearLinux
It seems as though you cannot “bake” a custom Clear Linux AMI, and then use THAT custom baked AMI as a base to build a “derived” image.

Let’s say that I custom-build an AMI from the ClearLinux 37000 on the AWS Marketplace, and I name that new AMI “Image One.”

Now, let’s say that I want to use “Image One” as a base to build another custom image, named “Image Two.”

When I build Image One, and create an EC2 instance from that AMI → everything is fine.

All my packages and changes are there. But when I use Image One as the base Packer base for Image Two, NONE of my packages or configurations from Image One are present on the Packer builder’s Launched EC2 instance. It’s almost as if the AMI has been somehow “reset” or “wiped.”

Simple Example (full example is included further down below…)

  1. Bake a custom Clear Linux AMI from the latest Clear Linux AMI on AWS…
  2. Install rpm into the custom image with swupd bundle-add package-utils during Packer build
  3. Packer build completes → get back new AMI-ID → create EC2 from AMI
  4. Access the EC2 and check for rpm → rpm => command exists
  5. Create a new Packer builder, and use the previous ^^ AMI as its base
  6. Packer Shell provisioner reports rpm: command not found when attempting to install a custom .rpm file generated with autospec
  7. Access the Packer EC2 (using -debug mode in Packer) --> check if the bundle and/or binary for rpm are present...but they are not. The rpm` binary is nowhere to be found…in fact, none of the packages I installed or changes I made to my custom AMI “base” image are present.

I’ve tried everything in order to get this to work, and I have no idea why I can’t build a “second-phase” or “derived” AMI from a customized “base” Clear Linux AMI.

I suspect this has something to do with Clear Linux’s “Stateless” principles, or maybe ucd is doing something I don’t understand…but I sincerely would appreciate help with this one.


Packer Stuff (Detailed)

  1. Derive a “base” image (aka “Golden Image”) from the ClearLinux AMI
    - This image is basically just my systems baseline.
    - Install a bunch of swupd bundles
    - Disable swupd autoupdates
    - Add some default /etc/profile and /etc/environment items (NOT SURE IF THESE ARE IN THE RIGHT PLACE)
    - Stop/mask tallow (ALL systems in our infrastructure are managed and remotely accessed only through the aws-ssm-agent, so we don’t need tallow)
    "variables": {
        "source_ami_arch": "x86_64",
        "source_ami_name": "clear-3*",
        "source_ami_owner": "679593333241",
        "product-code": "4ks0lssia3phyyrbckdk9xgba",
        "source_ami_ssh_user": "clear"

      "type": "shell",
      "expect_disconnect": true,
      "inline": [
        "SWUPD=$(sudo pidof swupd); while ps -p ${SWUPD} > /dev/null; do echo 'Waiting for swupd...'; sleep 1; done;",
        "sudo swupd autoupdate --disable --no-progress",
        "sudo systemctl stop tallow",
        "sudo systemctl mask tallow",
        "sudo systemctl daemon-reload",
        "sudo mkdir -p ~/usr/bin",
        "sudo mkdir -p ~/usr/lib64",
        "sudo touch /etc/environment",
        "sudo touch /etc/profile",
        "echo 'PATH=$PATH:~/usr/bin' | sudo tee --append /etc/environment",
        "echo 'PATH=$PATH:~/usr/bin' | sudo tee --append /etc/profile",
        "echo 'export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:~/usr/lib64' | sudo tee --append /etc/environment",
        "echo 'export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:~/usr/lib64' | sudo tee --append /etc/profile",
        "sudo swupd bundle-add c-basic c-basic-legacy cloud-api cloud-control cloud-native-basic cpio cronie curl devpkg-expat devpkg-libnetfilter_conntrack devpkg-LVM2 devpkg-openssl devpkg-systemd flatpak go-basic-dev iperf ipvsadm jq kernel-aws-dkms logrotate network-basic nfs-utils openssl os-cloudguest-aws os-core os-core-search package-utils parted python-extras runtime-libs-boost rust-basic socat service-os-dev sysadmin-basic time-server-basic unzip wget yq",
        "sudo shred -u /etc/ssh/*_key /etc/ssh/*_key.pub",
        "shred -u ~/.ssh/authorized_keys",
        "sudo shred -u /var/lib/cloud/aws-user-data",
        "sudo shred -u /etc/machine-id"

^^ The above produces an AMI with the expected packages present (rpm, dnf, ifconfig...)

But when this image (the resulting AMI) is used later in a different Packer builder as the source_ami: for some other build, the Packer Shell provisioner reports rpm: command not found.

ANY help or guidance would be much appreciated. Thank you again for all of your work!

Reading through this I’m a little confused. Is it possible at step 5 you’ve used the wrong ID? At step 4 everything seems to be according to expectations. Aside that I don’t know packer well enough to understand what should happen and where it could go wrong. However, as clearlinux isn’t an RPM based distribution for end users, I’m wondering if this is a factor that is causing issues to using clearlinux with Packer.

ucd doesn’t wipe a system image. It merely fetches (if the system hasn’t done this before) the user/metadata and executes it. In most cases, running it again won’t change anything and it certainly will not uninstall software.