---
url: >-
  https://cubesandbox.com/guide/troubleshooting/pvm-host-boot-unsigned-modules.md
---

# PVM Host Kernel Boots into Initramfs with module.sig\_enforce=1

## Symptom

After installing the PVM host kernel from the `kernel-release-*` Releases and
running `deploy/pvm/grub/host_grub_config.sh`, the next boot drops to the
initramfs shell instead of the running system:

```text
Gave up waiting for root file system device.  Common problems:
 - Boot args (cat /proc/cmdline)
   - Check rootdelay= (did the system wait long enough?)
 - Missing modules (cat /proc/modules; ls /dev)
ALERT!  /dev/mapper/openstack--dev--vg-root does not exist.  Dropping to a shell

BusyBox v1.37.0 (Debian 1:1.37.0-6+b9) built-in shell (ash)
(initramfs)
```

The previous (distribution) kernel still boots fine via the GRUB
"Advanced options" submenu.

## Environment

* Cube Sandbox version: v0.7.x (deploy scripts at the time of writing)
* Deployment mode: PVM (no `/dev/kvm`; the PVM host kernel provides KVM)
* Host OS / kernel: Debian 13 (trixie) on VMware — verified case below;
  ESXi reports the same pattern
  ([#1949](https://github.com/TencentCloud/CubeSandbox/issues/1949))
* Related components: `deploy/pvm/grub/host_grub_config.sh`

## Root Cause

Two facts combine into the failure:

1. The community release kernels ship **unsigned** modules. The pvm\_host
   kernel config has `CONFIG_MODULE_SIG=y` but
   `CONFIG_MODULE_SIG_ALL` is not set and `CONFIG_SYSTEM_TRUSTED_KEYS=""`,
   so the build never signs the shipped `.ko` files
   (`modinfo -F signer` returns empty for `dm-mod.ko`, `kvm.ko`,
   `scsi_mod.ko`, `mptspi.ko`, `vmw_pvscsi.ko`, ...).
2. `host_grub_config.sh` enables `module.sig_enforce=1` on the kernel
   command line (by design). With signature enforcement on, the kernel
   refuses to load **every** unsigned module, so storage drivers (e.g.
   `mptspi` on VMware, `dm-mod` for LVM) never come up and the root device
   cannot be found.

## Verified case: Debian 13 (trixie) on VMware

Host: Debian 13 (trixie) VMware VM, no nested virtualization, disks on an
LSI 53c1030 Fusion-MPT controller (`mptspi`), root on LVM
(`/dev/mapper/openstack--dev--vg-root`).

| Kernel | Source | Module signing |
| --- | --- | --- |
| `6.12.113+deb13-amd64` (distribution) | Debian | signed — `signer='Build time autogenerated kernel key'`, compressed `.ko.xz` |
| `6.6.117-opencloudos9.cubesandbox.pvm.host-gbc75224fca7e` | `kernel-release-261012-1` deb | **unsigned** — `modinfo -F signer` empty for all modules |

Relevant config of the released PVM kernel (queried from
`/boot/config-<kver>` and `/proc/config.gz`):

```text
CONFIG_MODULE_SIG=y
# CONFIG_MODULE_SIG_FORCE is not set
# CONFIG_MODULE_SIG_ALL is not set
CONFIG_MODULE_SIG_KEY="certs/signing_key.pem"
CONFIG_MODULE_SIG_HASH="sha512"
CONFIG_SYSTEM_TRUSTED_KEYRING=y
CONFIG_SYSTEM_TRUSTED_KEYS=""
```

Module signature check after installing the deb:

```console
$ modinfo -F signer /lib/modules/6.6.117-...pvm.host/kernel/drivers/md/dm-mod.ko
(empty)
$ modinfo -F signer /lib/modules/6.12.113+deb13-amd64/kernel/drivers/md/dm-mod.ko.xz
Build time autogenerated kernel key
```

Recovery (exact steps used):

1. Reboot → GRUB → `Advanced options for Debian GNU/Linux` → boot
   `6.12.113+deb13-amd64`.

2. Remove the parameter and regenerate the grub config:

   ```bash
   sed -i 's/ module\.sig_enforce=1//' /etc/default/grub
   update-grub
   reboot
   ```

3. After reboot into the PVM kernel: `kvm_pvm` loads, `/dev/kvm` appears,
   and Cube Sandbox installs and runs normally.

> Tip: on this platform the disk order (sda/sdb) can change between reboots.
> Use `UUID=` entries in `/etc/fstab` for mounts other than the LVM root.

## Verified case: openEuler 24.03 LTS-SP4 on VMware

Host: openEuler 24.03 LTS-SP4 VMware VM, no nested virtualization, 100 GiB
empty data disk for `/data/cubelet`, root on LVM (ext4).

| Kernel | Source | Module signing |
| --- | --- | --- |
| `6.6.0-159.4.3.154.oe2403sp4.x86_64` (distribution) | openEuler | signed — `signer='openEuler kernel ICA 1'`, compressed `.ko.xz` |
| `6.6.117-opencloudos9.cubesandbox.pvm.host-gbc75224fca7e` | `kernel-release-261012-1` rpm | **unsigned** — same artifact as the Debian case |

The stock openEuler kernel embeds its own certificate, which is exactly the
pattern the PVM kernel build would need:

```text
# openEuler stock kernel
CONFIG_MODULE_SIG=y
# CONFIG_MODULE_SIG_FORCE is not set
CONFIG_MODULE_SIG_KEY=""
CONFIG_SYSTEM_TRUSTED_KEYS="certs/openeuler-cert.pem"

# PVM release kernel
CONFIG_MODULE_SIG=y
# CONFIG_MODULE_SIG_ALL is not set
CONFIG_MODULE_SIG_KEY="certs/signing_key.pem"
CONFIG_SYSTEM_TRUSTED_KEYS=""
```

Module signature check:

```console
$ modinfo -F signer /lib/modules/6.6.0-159.../kernel/drivers/md/dm-mod.ko.xz
openEuler kernel ICA 1
$ modinfo -F signer /lib/modules/6.6.117-...pvm.host/kernel/drivers/md/dm-mod.ko
(empty)
```

Conclusion: the PVM kernel artifact is distro-independent — the unsigned
modules and the `module.sig_enforce=1` risk apply identically on RPM-based
hosts (openEuler, CentOS, ...). Recovery and mitigation are the same.

## Resolution

### 1. Recover the host

Follow the recovery steps in the verified case above: boot the previous
distribution kernel from GRUB, then remove `module.sig_enforce=1` from
`/etc/default/grub` and regenerate the grub config.

### 2. Fix the kernel build, keep the grub script enforcing

[PR #2053](https://github.com/TencentCloud/CubeSandbox/pull/2053) does not
weaken `host_grub_config.sh`: it keeps appending `module.sig_enforce=1`
unconditionally. The root cause is fixed in the kernel build instead, by
enabling module signing in `deploy/pvm/configs/pvm_host`:

```text
CONFIG_MODULE_SIG=y
CONFIG_MODULE_SIG_ALL=y                        # sign every in-tree module at modules_install
CONFIG_MODULE_SIG_KEY="certs/signing_key.pem"  # private key + matching X.509 certificate
CONFIG_SYSTEM_TRUSTED_KEYS=""                  # the signing key's public cert is embedded automatically
```

With `CONFIG_MODULE_SIG_ALL=y`, `make modules_install` signs every in-tree
module with `CONFIG_MODULE_SIG_KEY`. The build also extracts the matching
public certificate (`certs/signing_key.x509`) and embeds it in the kernel's
built-in trusted keyring, so the kernel trusts exactly the modules it ships —
that is what makes the unconditional `module.sig_enforce=1` safe.

Rebuild and publish the `kernel-release-*` artifacts from this config. A host
that still runs an older (unsigned) release kernel must install the rebuilt
signed kernel **before** re-running `host_grub_config.sh`, otherwise
enforcement rejects its modules again. This was validated with self-built
kernels from the updated config: both a Debian 13 `deb` and an openEuler 24.03
`rpm` boot with `module.sig_enforce=1`, load `kvm_pvm`/`/dev/kvm`, and reject
unsigned modules while accepting modules signed with the build key.

Two operational notes when rolling this out:

* After installing the rebuilt kernel, remove the old unsigned PVM kernel
  package (or never select its GRUB entry). `module.sig_enforce=1` is global,
  so booting the old entry would drop to the initramfs again.
* Publish the rebuilt artifacts under a new `kernel-release-*` tag and bump
  the package version. This config change does not alter the kernel release
  string, so RPM will not upgrade an installed package with the same version
  automatically; a localversion suffix or an explicit package version bump
  makes the signed kernel an actual upgrade.

### 3. Quick risk check for any host

Check every installed PVM kernel, not only the running one: the recovery path
boots the distribution kernel, so `uname -r` is not necessarily the kernel the
grub script configures.

```bash
for kdir in /lib/modules/*pvm.host*; do
  [ -d "$kdir" ] || continue
  kver=$(basename "$kdir")
  echo "== $kver"
  grep -qw "module.sig_enforce=1" /proc/cmdline && echo "  enforcement: ON"
  grep -E "^CONFIG_MODULE_SIG_FORCE=" "/boot/config-$kver" 2>/dev/null \
    || zgrep -E "^CONFIG_MODULE_SIG_FORCE=" /proc/config.gz 2>/dev/null
  # Stop at the first module without a signer: enforcement would reject it.
  find "$kdir" \( -type f -o -type l \) -name '*.ko*' 2>/dev/null |
  while IFS= read -r m; do
    [ -n "$(modinfo -F signer "$m" 2>/dev/null)" ] || { echo "  unsigned: $m"; break; }
  done
done
```

High risk = enforcement is on (cmdline, grub config, or
`CONFIG_MODULE_SIG_FORCE=y`) while **any** module of an installed PVM kernel
reports an empty signer or cannot be read by `modinfo`. In that state the next
boot will drop to the initramfs again. Note that `modinfo -F signer` only shows
that a signature is present, not that the kernel trusts the signing key:
modules signed by an untrusted key are still rejected under enforcement, and
that case cannot be detected from userspace before boot.

### 4. Out-of-tree and DKMS modules

Any out-of-tree/DKMS module that has to load on the host must be signed with
the same key the kernel trusts (`scripts/sign-file`), or enforcement rejects
it. Modules that cannot be signed with the release signing key should be built
into the kernel instead.

> Note: `kernel-release-*` artifacts published before this change still ship
> unsigned modules. They have to be rebuilt from the updated
> `deploy/pvm/configs/pvm_host` before the unconditional
> `module.sig_enforce=1` is safe on hosts running them.

## References

* Related PR: [#2053](https://github.com/TencentCloud/CubeSandbox/pull/2053)
* Possibly related issue: [#1949](https://github.com/TencentCloud/CubeSandbox/issues/1949)
* Kernel docs: [Module Signing](https://docs.kernel.org/admin-guide/module-signing.html)