Service Management & Logs

This page is for users who have already installed CubeSandbox and want to keep the stack healthy in day-to-day operation.

After reading this page you will know:

Which systemd services run on the host and how they depend on each other
Which service to restart after editing a config file
How to debug a service that keeps failing
Where to find runtime logs, startup logs and in-container logs — and the boundaries between them
How to stop / restart the whole stack cleanly

Scope

This page targets the systemd-managed one-click installer. If your machine still uses the legacy up-with-deps.sh / down-with-deps.sh scripts as the daily entry point, that is the pre-systemd version — re-running the latest one-click installer will migrate it to systemd automatically (the installer detects and takes over the old layout).

TL;DR cheat-sheet

bash

# 1. Are all cube-sandbox services still alive?
sudo systemctl --no-legend list-units 'cube-sandbox-*'

# 2. Edited a config -> restart the matching service
sudo systemctl restart cube-sandbox-cube-api.service
sudo systemctl restart cube-sandbox-cubemaster.service
sudo systemctl restart cube-sandbox-cubelet.service

# 3. Runtime logs (requests / stats / audit / VMM) live under /data/log/, NOT in journalctl
sudo tail -F /data/log/Cubelet/Cubelet-req.log
sudo tail -F /data/log/CubeMaster/cubemaster-req.log
sudo tail -F /data/log/CubeAPI/cube-api-$(date +%F).log
sudo tail -F /data/log/CubeVmm/vmm.log              # sandbox VMM lifecycle
sudo tail -F /data/log/cube-proxy/error.log         # proxy errors

# 4. Startup failures / process exit reasons -> journalctl
sudo journalctl -u cube-sandbox-cube-api.service -n 200 --no-pager

# 5. One-shot diagnostic bundle (tails of /data/log + configs + dmesg + process snapshot)
sudo /usr/local/services/cubetoolbox/scripts/cube-diag/collect-logs.sh

Runtime logs are at /data/log/, NOT journalctl

This is the most common pitfall for new operators: each component only sends startup-time stdout/stderr to journal. Request / scheduling / stat / audit / VMM creation logs are written directly to /data/log/<Module>/. To find "who created a sandbox in the last hour", look at /data/log/, not journalctl.

Service overview

The one-click installer registers 14 systemd units under /etc/systemd/system/ and aggregates them into two role-specific targets.

Role targets

Target	Purpose	Role
`cube-sandbox-control.target`	All control-plane services (default all-in-one)	`control`
`cube-sandbox-compute.target`	Minimum subset for compute-only nodes	`compute`

How aggregation works

The target lists its child services via Wants=; each service declares membership via PartOf=. So systemctl stop cube-sandbox-control.target stops every PartOf=cube-sandbox-control.target service in one shot — no need to spell out the long list of unit names.

Service catalog

Unit	Process form	Port / listen	Present on	Upstream deps
`cube-sandbox-mysql.service`	Docker container	`3306`	control	docker
`cube-sandbox-redis.service`	Docker container	`6379`	control	docker
`cube-sandbox-cubemaster.service`	Host process	`8089`	control	mysql, redis
`cube-sandbox-cube-api.service`	Host process	`3000` (E2B-compatible API)	control	cubemaster
`cube-sandbox-cubelet.service`	Host process	`9999` (gRPC), HTTP diagnostics	control / compute	embedded network runtime + `/data/cubelet` (XFS)
`cube-sandbox-coredns.service`	Docker container	`127.0.0.54:53` or `169.254.254.53:53`	control	docker
`cube-sandbox-cube-proxy.service`	Docker container	`443` (TLS) / `80` / `9090` (gRPC)	control	docker, redis
`cube-sandbox-dns.service`	oneshot (no daemon)	—	control	coredns (`BindsTo`)
`cube-sandbox-webui.service`	Docker container	`12088`	control	docker, cube-api

Startup dependency map (control node)

text

docker.service
   ├─ mysql.service ─┐
   ├─ redis.service ─┼─ cubemaster.service ─ cube-api.service ─ webui.service
   │                 └─ cube-proxy.service
   └─ coredns.service ─ dns.service (oneshot, BindsTo coredns)

network-online.target
   └─ cubelet.service (embedded network runtime)

Dependencies only express startup ordering via After= / Wants=. If an upstream service crashes at runtime, downstreams are not automatically restarted — cubelet won't be cycled just because cube-api died, and vice versa.

Restarting services

Scenario A: edited a config and want it to take effect

The two most common config entry points:

Top-level env: /usr/local/services/cubetoolbox/.one-click.env
Per-component: Cubelet/config/config.toml, Cubelet/dynamicconf/conf.yaml, CubeMaster/conf.yaml, cubeproxy/global.conf, coredns/Corefile

Restart the service that consumes that config:

bash

# Cubelet config
sudo systemctl restart cube-sandbox-cubelet.service

# CubeMaster config
sudo systemctl restart cube-sandbox-cubemaster.service

# CUBE_API_* in .one-click.env
sudo systemctl restart cube-sandbox-cube-api.service

# cubeproxy/global.conf
sudo systemctl restart cube-sandbox-cube-proxy.service

# coredns/Corefile
sudo systemctl restart cube-sandbox-coredns.service

Editing the systemd unit file itself

If you change /etc/systemd/system/cube-sandbox-*.service, run daemon-reload so systemd picks up the new content:

bash

sudo systemctl daemon-reload
sudo systemctl restart cube-sandbox-<service>.service

If you only edited the helper script (/usr/local/services/cubetoolbox/scripts/systemd/*.sh), daemon-reload is not needed — the next restart re-invokes the script.

CubeMaster settings

Path: /usr/local/services/cubetoolbox/CubeMaster/conf.yaml (from configs/single-node/cubemaster.yaml in one-click bundles).

Under cubelet_conf:

Key	Purpose
`default_timeout_insec`	Server default sandbox idle TTL (seconds) when the client omits `timeout`. Unset or `<= 0` means no cluster-wide idle timeout (sandboxes never time out from idle unless the client sets `timeout`). The repository ships `-1` for this “no default” behavior. Set a positive value (e.g. `300`) in production if you want automatic reclamation of sandboxes created without an explicit TTL.
`create_timeout_insec`	Create/scheduling RPC deadline only — not sandbox idle TTL. Defaults to `300` when unset.
`common_timeout_insec`	Generic CubeMaster→Cubelet RPC timeout for non-create paths.

After changing default_timeout_insec, restart CubeMaster and read Sandbox lifecycle — Operational Notes for client-visible behavior. For node selection, quota, labels, scheduler scoring, or template redo after adding compute nodes, see CubeMaster Scheduler Configuration.

Scenario B: a service is failing or restart-looping

Every service has Restart=on-failure, so a single crash is auto-recovered. If the unit is restart-looping, find the root cause first.

1. Inspect current state

bash

sudo systemctl status cube-sandbox-cube-proxy.service --no-pager

Watch for:

Active: failed / Active: activating (start-post) (still trying)
Restart Counter climbing rapidly (restart loop)
The last 10 journal lines printed at the bottom

2. Read startup logs

bash

sudo journalctl -u cube-sandbox-cube-proxy.service -n 200 --no-pager

Best for: scripting bugs, ExecStart failures, docker pull errors, apk / apt network errors, ExecStartPost health-check timeouts.

3. Read runtime logs

If the service starts but misbehaves, runtime logs live under /data/log/, not in journal:

bash

sudo tail -200 /data/log/Cubelet/Cubelet-req.log
sudo tail -200 /data/log/CubeMaster/cubemaster-req.log
sudo tail -200 /data/log/CubeAPI/cube-api-$(date +%F).log

4. Reset the failed counter and restart

bash

sudo systemctl reset-failed cube-sandbox-cube-proxy.service
sudo systemctl restart cube-sandbox-cube-proxy.service

Scenario C: full restart / post-maintenance recovery

bash

# Control node
sudo systemctl restart cube-sandbox-control.target

# Compute node
sudo systemctl restart cube-sandbox-compute.target

Or from the release-bundle directory:

bash

sudo ./down.sh
sudo systemctl start cube-sandbox-control.target

Restarting the target = ordered restart of every PartOf service

A target has no process of its own. Restarting it makes systemd cycle every PartOf=cube-sandbox-control.target service in dependency order — a shorthand for "restart everything".

Scenario D: full shutdown

bash

# Recommended: use the bundled script (auto-detects role)
sudo /root/cube-sandbox-one-click-<version>/down.sh

# Equivalent
sudo systemctl stop cube-sandbox-control.target   # control node
sudo systemctl stop cube-sandbox-compute.target   # compute node

down.sh does not delete data: MySQL / Redis volumes, /data/cubelet, /data/log/... are all preserved and resumed on the next start.

Reading logs

CubeSandbox has multiple log sources, including component-specific in-container logs. The two primary host-side entry points are:

Source	Contains	How to read
Runtime logs (primary entry point)	requests, scheduling decisions, stats, audit, VMM creation	`/data/log/<Module>/`
Startup logs	systemd start / hooks / ExecStartPost / exit codes / container build output	`journalctl -u <unit>`

`/data/log/` runtime logs (primary)

⚠️ Cubelet / CubeMaster / CubeAPI / CubeShim / VMM all write business request + stat + audit + VMM lifecycle logs to /data/log/. They do not show up in journalctl — read the files directly.

Module	Directory	Main files
Cubelet	`/data/log/Cubelet/`	`Cubelet-req.log` (requests) `Cubelet-stat.log` (metrics/stats)
CubeMaster	`/data/log/CubeMaster/`	`cubemaster-req.log`
CubeAPI	`/data/log/CubeAPI/`	`cube-api-YYYY-MM-DD.log` (daily-rotated)
CubeShim	`/data/log/CubeShim/`	`cube-shim-req.log`, `cube-shim-stat.log`
Hypervisor (VMM)	`/data/log/CubeVmm/`	`vmm.log` (one entry per sandbox creation)
cube-proxy	`/data/log/cube-proxy/`	`error.log`, `access.log` (see below)

Common commands:

bash

# Follow Cubelet requests
sudo tail -F /data/log/Cubelet/Cubelet-req.log

# Follow daily-rotated CubeAPI log (E2B-compatible layer)
sudo tail -F /data/log/CubeAPI/cube-api-$(date +%F).log

# Slow / failing sandbox start: check VMM log
sudo tail -200 /data/log/CubeVmm/vmm.log

Rotating CubeShim and VMM logs

CubeShim and the VMM keep their log files open while they run. CubeShim reopens its files on an internal 30-minute rotation event. The VMM control thread owns a monotonic timerfd and emits the existing LOG_CTRL_REOPEN control record once per hour. Reopen is schedule-driven: an external rename + create is picked up at the next scheduled reopen rather than detected immediately on an arbitrary write. The timer is owned by the VMM control thread, not by deferred logger initialization or a vCPU/API thread. The host-side policy should run hourly and use rename + create; do not use copytruncate.

For example, install the following as /etc/logrotate.d/cubesandbox and make sure the host invokes logrotate hourly:

text

/data/log/CubeVmm/vmm.log
/data/log/CubeShim/*.log {
    hourly
    rotate 24
    missingok
    notifempty
    compress
    delaycompress
    create 0640 root root
}

rotate 24 retains 24 hourly files; adjust it to the required retention period. delaycompress keeps the newest rotated file uncompressed for one cycle because a writer may still use the old descriptor until its next scheduled reopen. CubeShim's internal event runs every 30 minutes, while the VMM control thread sends a reopen control event every hour. No postrotate signal or service restart is required. This policy guarantees bounded retention on the host, but does not promise immediate handling of an arbitrary manual rotation; it does not support copytruncate.

The example uses root root because the bundled one-click systemd services run as root. If CubeShim or the VMM run under another account, set the create owner and group to that account; otherwise a newly-created file may not be reopenable. The 0640 mode in this example applies to files created by logrotate; the CubeShim and VMM writers themselves continue to use the process umask.

`journalctl` startup logs

journalctl captures stdout/stderr from when systemd starts the process until it stabilizes (or exits), useful for:

Startup failure exit codes / error messages
Output from ExecStart / ExecStartPost / ExecStop hooks
docker pull / docker build / apk update failures
Auto-restart counter and reasons

bash

# Last 200 lines
sudo journalctl -u cube-sandbox-cubelet.service -n 200 --no-pager

# Live tail
sudo journalctl -u cube-sandbox-cubemaster.service -f

# Everything since the last boot
sudo journalctl -u cube-sandbox-cube-api.service -b

No business request logs in journalctl

Once a process is stable, its stdout/stderr volume is tiny because each component writes business logs straight to /data/log/<Module>/. To find "which sandboxes were created in the last hour", journalctl is the wrong place — go to /data/log/CubeMaster/cubemaster-req.log or /data/log/Cubelet/Cubelet-req.log.

`cube-proxy` host logs

cube-proxy is an OpenResty/nginx container. The one-click deployment bind-mounts the host directory /data/log/cube-proxy/ into the container at the same path, so the logs remain available across container restarts and can be read directly from the host:

bash

sudo tail -200 /data/log/cube-proxy/error.log
sudo tail -200 /data/log/cube-proxy/access.log

The same directory is mounted at /data/log/cube-proxy/ inside the container; no image rebuild is required.

One-shot diagnostic bundle

Use the bundled diagnostic collector when sharing logs with the community or filing issues:

bash

sudo /usr/local/services/cubetoolbox/scripts/cube-diag/collect-logs.sh

It collects everything into cube-diag-<timestamp>/:

Tails of /data/log/CubeMaster|Cubelet|CubeAPI|CubeShim|CubeVmm/
/data/log/cube-proxy/ access/error logs
dmesg / process list / ports / mounts / cgroup / cpuinfo
Major config files (with secrets redacted)

Pack and share:

bash

tar czf cube-diag-<ts>.tar.gz cube-diag-<ts>/

Selective collection — e.g. only cubelet + dmesg:

bash

sudo /usr/local/services/cubetoolbox/scripts/cube-diag/collect-logs.sh \
  --module cubelet --module dmesg --lines 500

See --help for full options.

Operations cheat-sheet

Goal	Command
List all cube services on this node	`systemctl --no-legend list-units 'cube-sandbox-*'`
Status of one service	`systemctl status cube-sandbox-<service>.service`
Dependency tree of a target	`systemctl list-dependencies cube-sandbox-control.target`
Start / stop / restart one service	`systemctl {start\|stop\|restart} cube-sandbox-<service>.service`
Start / stop / restart the whole stack	`systemctl {start\|stop\|restart} cube-sandbox-{control,compute}.target`
Why did the service fail	`journalctl -u cube-sandbox-<service>.service -n 200 --no-pager`
Live tail startup output	`journalctl -u cube-sandbox-<service>.service -f`
Reset failed counter	`systemctl reset-failed cube-sandbox-<service>.service`
Run health check	`sudo /root/cube-sandbox-one-click-*/smoke.sh` or `sudo /usr/local/services/cubetoolbox/scripts/one-click/quickcheck.sh`
Full shutdown	`sudo /root/cube-sandbox-one-click-*/down.sh`
Collect diagnostic bundle	`sudo /usr/local/services/cubetoolbox/scripts/cube-diag/collect-logs.sh`

Typical troubleshooting flows

Sandbox creation fails / times out

Walk through the layers in order:

Is the role target active?

bash

sudo systemctl status cube-sandbox-control.target

Run the health check

bash

sudo /root/cube-sandbox-one-click-*/smoke.sh

Did CubeAPI receive the request?

bash

sudo tail -F /data/log/CubeAPI/cube-api-$(date +%F).log

CubeMaster scheduling chain

bash

sudo tail -F /data/log/CubeMaster/cubemaster-req.log

Is the node (Cubelet) online and being scheduled?

bash

curl http://127.0.0.1:3010/internal/v1/nodes
sudo tail -F /data/log/Cubelet/Cubelet-req.log

VMM startup errors

bash

sudo tail -200 /data/log/CubeVmm/vmm.log

A service is stuck in `activating (start-post)`

bash

sudo systemctl status cube-sandbox-<service>.service
sudo journalctl -u cube-sandbox-<service>.service -n 200 --no-pager

Common root causes:

Container build needs the network (e.g. cube-proxy's apk update) and the upstream mirror is flaky — see Deployment Troubleshooting
ExecStartPost health probe timeout (port already in use, upstream not yet ready)
For cube-sandbox-cube-proxy.service, CUBE_PROXY_HTTP_PORT and CUBE_PROXY_GRPC_PORT are the nginx listeners checked by the post-start TCP probe. CUBE_PROXY_HOST_PORT is deprecated and ignored; set CUBE_PROXY_HTTP_PORT instead if you need a non-default HTTP check port.
/data/log or /data/cubelet missing / wrong permissions / XFS not mounted

Dashboard / API unreachable

bash

# WebUI container
sudo systemctl status cube-sandbox-webui.service
sudo ss -lntp 'sport = :12088'

# CubeAPI listener
sudo systemctl status cube-sandbox-cube-api.service
sudo ss -lntp 'sport = :3000'

Appendix

Path quick-reference

Use	Path
Install root	`/usr/local/services/cubetoolbox/`
Runtime env file	`/usr/local/services/cubetoolbox/.one-click.env`
systemd unit install dir	`/etc/systemd/system/cube-sandbox-*`
systemd helper scripts	`/usr/local/services/cubetoolbox/scripts/systemd/*.sh`
Runtime logs (primary)	`/data/log/<Module>/`
Cubelet container layer (XFS)	`/data/cubelet/`
Sandbox images / snapshots	`/data/cube-shim/disks/`, `/data/snapshot_pack/disks/`
systemd PID files	`/run/cube-sandbox-systemd/`

Role / service matrix

Service	`control` node	`compute` node
`mysql` / `redis`	✅	—
`cubemaster`	✅	—
`cube-api`	✅	—
`webui`	✅	—
`cube-proxy` / `coredns` / `dns`	✅	—
`cubelet`	✅	✅

Kubernetes Deployment

Framework & Tool Integrations

Case Studies

Service Management & Logs

TL;DR cheat-sheet

Service overview

Role targets

Service catalog

Startup dependency map (control node)

Restarting services

Scenario A: edited a config and want it to take effect

CubeMaster settings

Scenario B: a service is failing or restart-looping

1. Inspect current state

2. Read startup logs

3. Read runtime logs

4. Reset the failed counter and restart

Scenario C: full restart / post-maintenance recovery

Scenario D: full shutdown

Reading logs

`/data/log/` runtime logs (primary)

Rotating CubeShim and VMM logs

`journalctl` startup logs

`cube-proxy` host logs

One-shot diagnostic bundle

Operations cheat-sheet

Typical troubleshooting flows

Sandbox creation fails / times out

A service is stuck in `activating (start-post)`

Dashboard / API unreachable

Appendix

Path quick-reference

Role / service matrix

See also

Service Management & Logs ​

TL;DR cheat-sheet ​

Service overview ​

Role targets ​

Service catalog ​

Startup dependency map (control node) ​

Restarting services ​

Scenario A: edited a config and want it to take effect ​

CubeMaster settings ​

Scenario B: a service is failing or restart-looping ​

1. Inspect current state ​

2. Read startup logs ​

3. Read runtime logs ​

4. Reset the failed counter and restart ​

Scenario C: full restart / post-maintenance recovery ​

Scenario D: full shutdown ​

Reading logs ​

/data/log/ runtime logs (primary) ​

Rotating CubeShim and VMM logs ​

journalctl startup logs ​

cube-proxy host logs ​

One-shot diagnostic bundle ​

Operations cheat-sheet ​

Typical troubleshooting flows ​

Sandbox creation fails / times out ​

A service is stuck in activating (start-post) ​

Dashboard / API unreachable ​

Appendix ​

Path quick-reference ​

Role / service matrix ​

See also ​

Service Management & Logs

TL;DR cheat-sheet

Service overview

Role targets

Service catalog

Startup dependency map (control node)

Restarting services

Scenario A: edited a config and want it to take effect

CubeMaster settings

Scenario B: a service is failing or restart-looping

1. Inspect current state

2. Read startup logs

3. Read runtime logs

4. Reset the failed counter and restart

Scenario C: full restart / post-maintenance recovery

Scenario D: full shutdown

Reading logs

`/data/log/` runtime logs (primary)

Rotating CubeShim and VMM logs

`journalctl` startup logs

`cube-proxy` host logs

One-shot diagnostic bundle

Operations cheat-sheet

Typical troubleshooting flows

Sandbox creation fails / times out

A service is stuck in `activating (start-post)`

Dashboard / API unreachable

Appendix

Path quick-reference

Role / service matrix

See also