PVM Host Kernel Boots into Initramfs with module.sig_enforce=1

Symptom

After installing the PVM host kernel from the kernel-release-* Releases and running deploy/pvm/grub/host_grub_config.sh, the next boot drops to the initramfs shell instead of the running system:

Gave up waiting for root file system device.  Common problems:
 - Boot args (cat /proc/cmdline)
   - Check rootdelay= (did the system wait long enough?)
 - Missing modules (cat /proc/modules; ls /dev)
ALERT!  /dev/mapper/openstack--dev--vg-root does not exist.  Dropping to a shell

BusyBox v1.37.0 (Debian 1:1.37.0-6+b9) built-in shell (ash)
(initramfs)

The previous (distribution) kernel still boots fine via the GRUB "Advanced options" submenu.

Environment

• Cube Sandbox version: v0.7.x (deploy scripts at the time of writing)
• Deployment mode: PVM (no /dev/kvm; the PVM host kernel provides KVM)
• Host OS / kernel: Debian 13 (trixie) on VMware — verified case below; ESXi reports the same pattern (#1949)
• Related components: deploy/pvm/grub/host_grub_config.sh

Root Cause

Two facts combine into the failure:

1. The community release kernels ship unsigned modules. The pvm_host kernel config has CONFIG_MODULE_SIG=y but CONFIG_MODULE_SIG_ALL is not set and CONFIG_SYSTEM_TRUSTED_KEYS="", so the build never signs the shipped .ko files (modinfo -F signer returns empty for dm-mod.ko, kvm.ko, scsi_mod.ko, mptspi.ko, vmw_pvscsi.ko, ...).
2. host_grub_config.sh enables module.sig_enforce=1 on the kernel command line (by design). With signature enforcement on, the kernel refuses to load every unsigned module, so storage drivers (e.g. mptspi on VMware, dm-mod for LVM) never come up and the root device cannot be found.

Verified case: Debian 13 (trixie) on VMware

Host: Debian 13 (trixie) VMware VM, no nested virtualization, disks on an LSI 53c1030 Fusion-MPT controller (mptspi), root on LVM (/dev/mapper/openstack--dev--vg-root).

Kernel	Source	Module signing
6.12.113+deb13-amd64 (distribution)	Debian	signed — signer='Build time autogenerated kernel key', compressed .ko.xz
6.6.117-opencloudos9.cubesandbox.pvm.host-gbc75224fca7e	kernel-release-261012-1 deb	unsigned — modinfo -F signer empty for all modules

Relevant config of the released PVM kernel (queried from /boot/config-<kver> and /proc/config.gz):

CONFIG_MODULE_SIG=y
# CONFIG_MODULE_SIG_FORCE is not set
# CONFIG_MODULE_SIG_ALL is not set
CONFIG_MODULE_SIG_KEY="certs/signing_key.pem"
CONFIG_MODULE_SIG_HASH="sha512"
CONFIG_SYSTEM_TRUSTED_KEYRING=y
CONFIG_SYSTEM_TRUSTED_KEYS=""

Module signature check after installing the deb:

$ modinfo -F signer /lib/modules/6.6.117-...pvm.host/kernel/drivers/md/dm-mod.ko
(empty)
$ modinfo -F signer /lib/modules/6.12.113+deb13-amd64/kernel/drivers/md/dm-mod.ko.xz
Build time autogenerated kernel key

Recovery (exact steps used):

1. Reboot → GRUB → Advanced options for Debian GNU/Linux → boot 6.12.113+deb13-amd64.

2. Remove the parameter and regenerate the grub config:

   sed -i 's/ module\.sig_enforce=1//' /etc/default/grub
   update-grub
   reboot

3. After reboot into the PVM kernel: kvm_pvm loads, /dev/kvm appears, and Cube Sandbox installs and runs normally.

Tip: on this platform the disk order (sda/sdb) can change between reboots. Use UUID= entries in /etc/fstab for mounts other than the LVM root.

Verified case: openEuler 24.03 LTS-SP4 on VMware

Host: openEuler 24.03 LTS-SP4 VMware VM, no nested virtualization, 100 GiB empty data disk for /data/cubelet, root on LVM (ext4).

Kernel	Source	Module signing
6.6.0-159.4.3.154.oe2403sp4.x86_64 (distribution)	openEuler	signed — signer='openEuler kernel ICA 1', compressed .ko.xz
6.6.117-opencloudos9.cubesandbox.pvm.host-gbc75224fca7e	kernel-release-261012-1 rpm	unsigned — same artifact as the Debian case

The stock openEuler kernel embeds its own certificate, which is exactly the pattern the PVM kernel build would need:

# openEuler stock kernel
CONFIG_MODULE_SIG=y
# CONFIG_MODULE_SIG_FORCE is not set
CONFIG_MODULE_SIG_KEY=""
CONFIG_SYSTEM_TRUSTED_KEYS="certs/openeuler-cert.pem"

# PVM release kernel
CONFIG_MODULE_SIG=y
# CONFIG_MODULE_SIG_ALL is not set
CONFIG_MODULE_SIG_KEY="certs/signing_key.pem"
CONFIG_SYSTEM_TRUSTED_KEYS=""

Module signature check:

$ modinfo -F signer /lib/modules/6.6.0-159.../kernel/drivers/md/dm-mod.ko.xz
openEuler kernel ICA 1
$ modinfo -F signer /lib/modules/6.6.117-...pvm.host/kernel/drivers/md/dm-mod.ko
(empty)

Conclusion: the PVM kernel artifact is distro-independent — the unsigned modules and the module.sig_enforce=1 risk apply identically on RPM-based hosts (openEuler, CentOS, ...). Recovery and mitigation are the same.

Resolution

1. Recover the host

Follow the recovery steps in the verified case above: boot the previous distribution kernel from GRUB, then remove module.sig_enforce=1 from /etc/default/grub and regenerate the grub config.

2. Fix the kernel build, keep the grub script enforcing

PR #2053 does not weaken host_grub_config.sh: it keeps appending module.sig_enforce=1 unconditionally. The root cause is fixed in the kernel build instead, by enabling module signing in deploy/pvm/configs/pvm_host:

CONFIG_MODULE_SIG=y
CONFIG_MODULE_SIG_ALL=y                        # sign every in-tree module at modules_install
CONFIG_MODULE_SIG_KEY="certs/signing_key.pem"  # private key + matching X.509 certificate
CONFIG_SYSTEM_TRUSTED_KEYS=""                  # the signing key's public cert is embedded automatically

With CONFIG_MODULE_SIG_ALL=y, make modules_install signs every in-tree module with CONFIG_MODULE_SIG_KEY. The build also extracts the matching public certificate (certs/signing_key.x509) and embeds it in the kernel's built-in trusted keyring, so the kernel trusts exactly the modules it ships — that is what makes the unconditional module.sig_enforce=1 safe.

Rebuild and publish the kernel-release-* artifacts from this config. A host that still runs an older (unsigned) release kernel must install the rebuilt signed kernel before re-running host_grub_config.sh, otherwise enforcement rejects its modules again. This was validated with self-built kernels from the updated config: both a Debian 13 deb and an openEuler 24.03 rpm boot with module.sig_enforce=1, load kvm_pvm//dev/kvm, and reject unsigned modules while accepting modules signed with the build key.

Two operational notes when rolling this out:

• After installing the rebuilt kernel, remove the old unsigned PVM kernel package (or never select its GRUB entry). module.sig_enforce=1 is global, so booting the old entry would drop to the initramfs again.
• Publish the rebuilt artifacts under a new kernel-release-* tag and bump the package version. This config change does not alter the kernel release string, so RPM will not upgrade an installed package with the same version automatically; a localversion suffix or an explicit package version bump makes the signed kernel an actual upgrade.

3. Quick risk check for any host

Check every installed PVM kernel, not only the running one: the recovery path boots the distribution kernel, so uname -r is not necessarily the kernel the grub script configures.

for kdir in /lib/modules/*pvm.host*; do
  [ -d "$kdir" ] || continue
  kver=$(basename "$kdir")
  echo "== $kver"
  grep -qw "module.sig_enforce=1" /proc/cmdline && echo "  enforcement: ON"
  grep -E "^CONFIG_MODULE_SIG_FORCE=" "/boot/config-$kver" 2>/dev/null \
    || zgrep -E "^CONFIG_MODULE_SIG_FORCE=" /proc/config.gz 2>/dev/null
  # Stop at the first module without a signer: enforcement would reject it.
  find "$kdir" \( -type f -o -type l \) -name '*.ko*' 2>/dev/null |
  while IFS= read -r m; do
    [ -n "$(modinfo -F signer "$m" 2>/dev/null)" ] || { echo "  unsigned: $m"; break; }
  done
done

High risk = enforcement is on (cmdline, grub config, or CONFIG_MODULE_SIG_FORCE=y) while any module of an installed PVM kernel reports an empty signer or cannot be read by modinfo. In that state the next boot will drop to the initramfs again. Note that modinfo -F signer only shows that a signature is present, not that the kernel trusts the signing key: modules signed by an untrusted key are still rejected under enforcement, and that case cannot be detected from userspace before boot.

4. Out-of-tree and DKMS modules

Any out-of-tree/DKMS module that has to load on the host must be signed with the same key the kernel trusts (scripts/sign-file), or enforcement rejects it. Modules that cannot be signed with the release signing key should be built into the kernel instead.

Note: kernel-release-* artifacts published before this change still ship unsigned modules. They have to be rebuilt from the updated deploy/pvm/configs/pvm_host before the unconditional module.sig_enforce=1 is safe on hosts running them.

References

• Related PR: #2053
• Possibly related issue: #1949
• Kernel docs: Module Signing