Multi-Node Cluster Deployment

This guide explains how to expand a single-node Cube Sandbox deployment into a multi-node cluster by adding compute nodes. Compute nodes run only the sandbox runtime components (Cubelet with the embedded network runtime, CubeShim) and register themselves to the control plane on the first machine.

Production Use

If you plan to use Cube Sandbox in a production environment, please refer to the Network Hardening guide to secure your deployment before exposing services to untrusted networks.

Prerequisite

You must have a working control node deployed via the Self-Build Deployment Guide before adding compute nodes.

Architecture Overview

┌─────────────────────────────────────────┐
│           Control Node                  │
│  CubeMaster, CubeOps, cube-api,         │
│  CubeProxy, CoreDNS, MySQL, Redis,      │
│  Cubelet (network runtime)              │
└──────────────────┬──────────────────────┘
                   │  /internal/v1/node-agent API
       ┌───────────┼───────────┐
       ▼           ▼           ▼
┌────────────┐┌────────────┐┌────────────┐
│ Compute #1 ││ Compute #2 ││ Compute #N │
│ Cubelet    ││ Cubelet    ││ Cubelet    │
│ net runtime││ net runtime││ net runtime│
└────────────┘└────────────┘└────────────┘

• The control node runs the full stack: orchestration (CubeMaster), node management (CubeOps), API gateway (cube-api), proxy (CubeProxy + CoreDNS), databases (MySQL + Redis), the bundled MinIO S3 volume store, and also acts as a compute node itself.
• Each compute node runs only Cubelet with its embedded network runtime. It registers to the control-plane CubeOps and receives sandbox scheduling requests from CubeMaster.

Prerequisites

Each compute node must meet the same hardware and software requirements as the control node:

• Physical machine or bare-metal server (nested virtualization is not supported)
• x86_64 or aarch64 (ARM64) architecture with KVM enabled (ls /dev/kvm)
• Docker installed and running
• Network connectivity to the control node (specifically to CubeOps on port 3010 for node registration, and to the S3 endpoint — port 9000 with the bundled MinIO)

For the full requirements list, see Self-Build Deployment — Prerequisites.

Step 1: Prepare the Release Bundle

Use the same release bundle that was built for the control node. Copy it to the compute node and extract:

tar -xzf cube-sandbox-one-click-<version>.tar.gz
cd cube-sandbox-one-click-<version>

Step 2: Configure Environment Variables

cp env.example .env

Edit .env and set the following variables:

ONE_CLICK_DEPLOY_ROLE=compute
CUBE_SANDBOX_NODE_IP=<current-node-ip>
ONE_CLICK_CONTROL_PLANE_IP=<control-plane-ip>

# CUBE_S3_*: optional but strongly recommended. If missing, install proceeds
# with a warning and the S3 volume plugin stays disabled. See the tip below
# for how to fetch the values. With the bundled MinIO this looks like:
CUBE_S3_ENDPOINT=http://<control-plane-ip>:9000
CUBE_S3_ACCESS_KEY_ID=<from control node>
CUBE_S3_SECRET_ACCESS_KEY=<from control node>
CUBE_S3_BUCKET=cube-volumes
CUBE_S3_S3FS_EXTRA_OPTS=-ouse_path_request_style

Variable	Description
ONE_CLICK_DEPLOY_ROLE	Must be set to compute for compute-only nodes
CUBE_SANDBOX_NODE_IP	This node's primary network interface IP
ONE_CLICK_CONTROL_PLANE_IP	The control node's IP; automatically expanded to <ip>:3010 for CubeOps node registration
CUBE_S3_*	Optional but strongly recommended. The volume plugin depends on S3; if missing, install warns and the S3 volume plugin stays disabled. See the tip below for how to fetch these values.

Warns when missing

install-compute.sh checks CUBE_S3_ENDPOINT; if it is missing the installer prints a prominent yellow warning and continues. The node deploys normally, but the S3 volume plugin stays disabled until you set CUBE_S3_* and re-run the installer.

To fetch the CUBE_S3_* values backfilled on the control node:

1. On the control node, run:

   grep '^CUBE_S3_' /usr/local/services/cubetoolbox/.one-click.env

   Empty output means the control node itself has no S3 backend configured.
2. Paste the output into this node's .env.
3. Re-run sudo ./install-compute.sh.

With the bundled MinIO, also allow TCP 9000 from this node to the control node.

You can also specify the CubeOps endpoint explicitly if it uses a non-default port:

ONE_CLICK_CONTROL_PLANE_CUBEOPS_ADDR=<control-plane-ip>:3010

ONE_CLICK_CONTROL_PLANE_CUBEOPS_ADDR takes precedence over ONE_CLICK_CONTROL_PLANE_IP when both are set.

Step 3: Install

sudo ./install-compute.sh

The compute-node install script will:

1. Install only Cubelet with the embedded network runtime, cube-shim, cube-image, cube-kernel-scf, and the runtime scripts
2. Start only the host process cubelet
3. Automatically point Cubelet's meta_server_endpoint to the control-plane CubeOps
4. Register the node and report status through the control plane /internal/v1/node-agent API to CubeOps

Verifying the Deployment

Health Check

sudo ./smoke.sh

In compute-node mode, quickcheck.sh verifies:

• Local Cubelet and embedded network runtime health
• Reachability of the control-plane CubeOps
• That the current node appears under /internal/v1/nodes/{node_id} on the control plane

Verify from the Control Node

On the control node, you can confirm the compute node has registered:

curl http://127.0.0.1:3010/internal/v1/nodes

The response should include the compute node's IP and a healthy status.

Configure CubeMaster Scheduler Scoring

For multi-node deployments, configure CubeMaster's scheduler.score on the control node. If scoring is omitted, CubeMaster filters eligible nodes and then selects from the filtered node order, which can concentrate new sandboxes on the first eligible node until resource filters push traffic elsewhere.

Merge the following scheduler fields into the existing scheduler section of cubemaster.yaml. Keep your existing filter, timeout, overcommit, and other scheduler settings.

scheduler:
  # Keep your existing filter, timeout, overcommit, and other scheduler settings.
  priority_select_num: 3
  score:
    enable_scorers:
      - real_time_weighted_average
    resource_weights:
      mvm_num: 2
      local_create_num: 3
      quota_cpu_usage: 1
      quota_mem_usage: 1
    plugin_conf:
      real_time_weighted_average:
        weight: 1.0
        enable_weight_factors:
          - mvm_num
          - local_create_num
          - quota_cpu_usage
          - quota_mem_usage

For multi-node clusters, set scheduler.priority_select_num to a value greater than 1 so CubeMaster randomly selects from the top scored nodes. The shipped default config uses priority_select_num: 1, which means scoring only determines which single node receives the next sandbox. Use 3 as a starting point for small clusters and tune it based on your node count. scheduler.least_select_name defaults to random, so it usually does not need to be set explicitly.

For the complete CubeMaster scheduler reference, including Cubelet node reports, quota / label / concurrency effects, and template redo after adding compute nodes, see CubeMaster Scheduler Configuration.

After updating cubemaster.yaml, restart CubeMaster with your normal deployment procedure so the scheduler loads the new scoring configuration.

Connect Clients to the Cluster

Client applications need the CubeAPI control-plane address and a route to sandbox services through CubeProxy. Choose the simplest data-plane access method that fits your client:

Method	Best for	Wildcard DNS	Extra component
CubeSandbox SDK with CUBE_PROXY_NODE_IP	Python, Go, and Node.js SDKs	No	No
CubeProxy path mode	curl, backend services, generic HTTP clients	No	No
Wildcard DNS	Production, browsers, SPAs, official E2B SDK	Yes	No
E2B dev sidecar	Official E2B SDK in local development without DNS	No	Yes

CubeSandbox SDK: direct CubeProxy access

The CubeSandbox SDKs can connect directly to a CubeProxy IP while preserving the virtual Host used to route requests to the correct sandbox. This avoids wildcard DNS:

export CUBE_API_URL="http://<control-plane-ip>:3000"
export CUBE_PROXY_NODE_IP="<cubeproxy-node-ip>"
export CUBE_PROXY_PORT_HTTP=80
export CUBE_TEMPLATE_ID="<your-template-id-or-alias>"

Use the SDK normally after setting these variables. Control-plane requests go to CubeAPI; data-plane requests connect directly to CubeProxy.

Generic HTTP clients: path mode

Any HTTP client can access a sandbox service through the CubeProxy path prefix:

http://<cubeproxy-host>:<http-port>/sandbox/<sandbox-id>/<container-port>/<path>

For example:

curl http://10.0.0.5/sandbox/abc123/49999/health

Path mode requires no DNS or certificate setup and supports WebSocket upgrades. It is not suitable for SPAs that load assets from root-absolute paths such as /static/app.js; use wildcard DNS for those applications.

Production and browser access: wildcard DNS

Host-based routing uses sandbox domains in the form <port>-<sandbox-id>.<domain>. Configure a wildcard A record that points to CubeProxy:

*.cube.example.com  →  <CubeProxy public or private IP>

Configure CubeAPI with the same base domain:

export CUBE_API_SANDBOX_DOMAIN=cube.example.com

The one-click deployment includes CoreDNS for local *.cube.app resolution. It is intended for local use; production and shared multi-machine environments should use managed DNS or an internal DNS service. /etc/hosts cannot provide wildcard records.

See HTTPS Certificates & Domain Resolution for TLS and DNS configuration details.

Official E2B SDK without wildcard DNS: dev sidecar

The official E2B SDK does not expose the CubeSandbox SDK's direct-IP option. When wildcard DNS is unavailable in local development, use the E2B dev sidecar example:

cd examples/e2b-dev-sidecar
pip install -r requirements.txt
cp env.example .env

For a remote cluster, configure:

E2B_API_URL="http://<control-plane-ip>:3000"
CUBE_REMOTE_PROXY_BASE="https://<cubeproxy-node-ip>:443"
E2B_API_KEY="<api-key>"
CUBE_TEMPLATE_ID="<your-template-id-or-alias>"

Then run:

python demo.py

CUBE_REMOTE_PROXY_BASE must point to CubeProxy, not to the sidecar's own listening address. When authentication is enabled, replace the placeholder API key with a valid key.

Common Operations

Stop Compute Node Services

sudo ./down.sh

In compute-node mode, this only stops cubelet. It does not affect the control plane or other compute nodes.

Reinstall

To reinstall a compute node, simply run install-compute.sh again. The script automatically stops the existing deployment before installing.

View Logs

Component	Log Path
Cubelet	/data/log/Cubelet/
CubeShim	/data/log/CubeShim/
Hypervisor (VMM)	/data/log/CubeVmm/
Runtime PID files	/var/run/cube-sandbox-one-click/
Process stdout/stderr	/var/log/cube-sandbox-one-click/

For control-node log paths, see Self-Build Deployment — View Logs.

Configuration Reference

Compute nodes use the same .env file format. The following variables are specific to or particularly relevant for compute-node deployments:

Variable	Default	Description
ONE_CLICK_DEPLOY_ROLE	control	Must be set to compute
ONE_CLICK_CONTROL_PLANE_IP	empty	Control-plane host IP; expanded to <ip>:3010 by default
ONE_CLICK_CONTROL_PLANE_CUBEOPS_ADDR	empty	Explicit CubeOps address; takes precedence over ONE_CLICK_CONTROL_PLANE_IP
CUBE_SANDBOX_NODE_IP	10.0.0.10	Required. This node's primary network interface IP
CUBE_SANDBOX_NETWORK_CIDR	192.168.0.0/18 (from config.toml)	cubevs local network CIDR. Should match the control-plane value. IPv4 CIDR format (e.g., 10.100.0.0/18), mask range /16–/24. Auto-detected for host network conflicts at install time.
CUBE_SANDBOX_NETWORK_CIDR_SKIP_CONFLICT_CHECK	0	Set to 1 to skip CIDR conflict detection (not recommended).
ONE_CLICK_RUN_QUICKCHECK	1	Run health check after installation
CUBE_S3_*	empty / filled by control MinIO	Optional but strongly recommended. The volume plugin depends on S3; if missing, install warns and the S3 volume plugin stays disabled. Copy from the control node's /usr/local/services/cubetoolbox/.one-click.env (fetch steps in Step 2 above); ENDPOINT / ACCESS_KEY_ID / SECRET_ACCESS_KEY / BUCKET have no usable default.

For the full configuration reference (build-time options, database, proxy, etc.), see Self-Build Deployment — Configuration Reference.

Troubleshooting

Compute Node Cannot Reach CubeOps

Verify network connectivity:

curl http://<control-plane-ip>:3010/internal/v1/nodes

If this fails, check:

• Firewall rules on the control node (port 3010 must be accessible)
• The ONE_CLICK_CONTROL_PLANE_IP or ONE_CLICK_CONTROL_PLANE_CUBEOPS_ADDR value in .env

Node Not Appearing in Control Plane

If smoke.sh passes locally but the node does not appear on the control plane:

1. Check Cubelet logs: /data/log/Cubelet/
2. Verify meta_server_endpoint in the Cubelet config points to the correct CubeOps address
3. Ensure CUBE_SANDBOX_NODE_IP is correctly set to a routable IP (not 127.0.0.1)

For general troubleshooting (Docker, KVM, DNS, etc.), see Self-Build Deployment — Troubleshooting.