CubeObservability

CubeSandbox sandbox observability deployment and operations.

English | 简体中文

Observability is opt-in for one-click and Kubernetes installs. Enable it in the one-click environment file or in Helm values; there are no extra installer flags.

Collected by default:

• Metrics: the cubesandbox_* resource metrics from Cubelet at 127.0.0.1:9998/v1/metrics/resource.
• Logs: sandbox stdout and stderr at /data/cubelet/log/<sandbox-id>/{stdout,stderr}.

Helm can additionally collect CRI logs for an allowlist of CubeSandbox containers under /var/log/pods. This optional Collector DaemonSet is disabled by default and is available only through the Helm chart, not the one-click/systemd installer.

Prometheus, Grafana, and one log backend (Loki by default, or Elasticsearch) ship with both installers. Both can also export to external backends.

Ownership:

• Grafana image and dashboards: CubeObservability/;
• one-click CVM/systemd: deploy/one-click/observability/;
• Helm Collector and backends: deploy/kubernetes/chart/.

1. Capabilities, versions, and preflight

1.1 Cubelet capability baseline

This solution uses the following data from the current Cubelet:

• Sandbox CPU and memory metrics from /v1/metrics/resource. Each per-sandbox series carries sandbox_id; guest_workload series also carry container_id.
• The node-level cubesandbox_node_sandbox_count and cubesandbox_node_sandbox_available_count gauges. Both return 0 when no sandbox is active.
• Sandbox stdout and stderr logs from /data/cubelet/log/<sandbox-id>/{stdout,stderr}.

For the complete metric names, types, units, and labels, see the CubeSandbox resource metrics guide on GitHub. The host_sandbox series carry sandbox_id; guest_workload series carry both sandbox_id and container_id. The Cubelet Prometheus source is the implementation reference. Node-count gauges have no dynamic labels and do not contain a list of Sandbox IDs.

Preflight

Run these checks after Cubelet is installed. For a new one-click environment, complete the first-node steps in 2.1 first:

# Cubelet version and service
/usr/local/services/cubetoolbox/Cubelet/bin/cubecli version
sudo systemctl is-active cube-sandbox-cubelet.service

# Sandbox resource endpoint and per-sandbox ID label
curl -fsS http://127.0.0.1:9998/v1/metrics/resource | \
  grep -E '^cubesandbox_(host_sandbox|guest_workload|node_sandbox_)'
curl -fsS http://127.0.0.1:9998/v1/metrics/resource | \
  grep 'sandbox_id='

# Standard sandbox stdout/stderr log path while a sandbox is active
SANDBOX_ID=019abc-example
ls -l "/data/cubelet/log/${SANDBOX_ID}"/{stdout,stderr}

The sandbox_id check can be empty when no sandbox is active; both cubesandbox_node_sandbox_* gauges should still return 0. A disabled plugin or pending first sample can also produce an empty result, so interpret it together with the plugin configuration and service state.

1.2 Installer versions and external compatibility baselines

Loki and Elasticsearch are mutually exclusive sandbox log backends. Loki is installed by default; Elasticsearch is installed only when explicitly selected with CUBE_LOGS_BACKEND=elasticsearch (or observability.logs.backend=elasticsearch for Helm). Only one log backend is active in a deployment, and logs are not duplicated.

Component	Installer-pinned and validated version	External/self-managed baseline	Default port	Role and compatibility notes
OTel Collector Contrib	0.162.0	Prefer 0.162.x	127.0.0.1:13133 (health)	Per-node collector. The generated YAML uses the OTLP/HTTP and Elasticsearch exporter configuration from this release. Arbitrary later releases are not implicitly supported; rerun otelcol validate after any upgrade.
Prometheus	one-click 3.13.4; Helm v3.2.1	Prometheus 3.x with OTLP and Remote Write receivers	9092	Sandbox metrics backend. --web.enable-otlp-receiver, --web.enable-remote-write-receiver, and the documented otlp configuration are required.
Loki (default log backend; choose one)	3.7.9	Loki 3.x with native OTLP and structured metadata enabled	3100	Default sandbox logs backend. The Collector writes to /otlp; Grafana queries it with LogQL.
Elasticsearch (optional log backend; choose one)	8.17.3	Elasticsearch 8.x	9200	Replaces Loki when explicitly selected; uses raw mapping and the Bulk API. Validate other major releases separately.
Grafana OSS	10.4.18	>= 10.4 with dashboard schema 39 support	3001	Metrics and logs UI. Validate the selected log data source, three metrics dashboards, and logs dashboard before upgrading.

Pinned versions keep bundled installs reproducible. An external baseline is not a guarantee for every future release.

Upstream configuration references: Prometheus receiver, filelog receiver, Elasticsearch exporter, Prometheus OTLP ingestion, and Loki OTLP ingestion.

The bundled CVM/systemd installers require 64-bit Linux, systemd, curl, tar, and python3; Python is used to extract and validate the Loki release archive safely. Kubernetes and Helm do not require these host binaries.

1.3 Loki or Elasticsearch

• Loki (default): lower resource overhead, suited to quick deployment and primarily label-based log queries.
• Elasticsearch (optional): suited to teams with an existing ES operations model or a need for full-text search.
• Choose one. In production, prefer an existing team-managed or hosted log platform.

Select the backend in .env for a new installation:

CUBE_LOGS_BACKEND=loki
# Or:
# CUBE_LOGS_BACKEND=elasticsearch

Learn more in the Loki documentation, Loki OTLP ingestion guide, Elasticsearch logs documentation, and Elasticsearch Query DSL.

2. Enable with a standard deployment

Deployment	Entrypoint	Enable
first one-click node (all-in-one by default)	deploy/one-click/install.sh	Set ONE_CLICK_ENABLE_OBSERVABILITY=1, then run the normal installer
one-click compute node	deploy/one-click/install-compute.sh	Set ONE_CLICK_ENABLE_OBSERVABILITY=1 and the first-node address; only the node Collector is installed
existing one-click deployment	original install.sh / install-compute.sh	Update the original .env, then run with --mode=upgrade --yes; see 2.1 C
Kubernetes Helm	deploy/kubernetes/chart	Pass --set observability.enabled=true

2.1 one-click

one-click keeps using its existing .env and installer entrypoints, and observability remains disabled by default. The first node is all-in-one by default. Additional compute nodes send metrics and logs to that first node; there is no separate monitoring control plane.

A. Install the first all-in-one node

1. Enter deploy/one-click in a source checkout, or the one-click root of an extracted release. Copy the template only for a fresh installation:

# Source checkout:
cd deploy/one-click
# Extracted release:
# cd /path/to/cube-sandbox-one-click
[[ -f .env ]] || cp env.example .env
chmod 600 .env

2. Choose a private IP reachable from later compute nodes, such as 10.0.0.10, and set:

# .env
CUBE_SANDBOX_NODE_IP=10.0.0.10
ONE_CLICK_ENABLE_OBSERVABILITY=1
CUBE_OBSERVABILITY_CLUSTER_ID=cube-prod
CUBE_LOGS_BACKEND=loki
CUBE_GRAFANA_ADMIN_PASSWORD='<random-password-at-least-16-chars>'

The installer rejects an unset or shorter Grafana password. Set CUBE_LOGS_BACKEND=elasticsearch to use Elasticsearch. The installer can detect CUBE_SANDBOX_NODE_IP when it is omitted, but confirm the resulting address after installation.

3. Run the normal installer:

sudo ./install.sh

4. Record the final first-node address. Additional compute nodes must use this reachable private IP:

grep '^CUBE_SANDBOX_NODE_IP=' \
  /usr/local/services/cubetoolbox/.one-click.env

5. Allow only the required network paths:

   • compute nodes to the first node: TCP 9092, plus 3100 for Loki or 9200 for Elasticsearch;
   • administrators to the first node: Grafana 3001;
   • never expose 9092/3100/9200 directly to the public Internet.

6. Verify the first node:

sudo /usr/local/services/cubetoolbox/observability/install.sh status
sudo systemctl --no-pager --full status cube-sandbox-otelcol.service
curl -fsS http://127.0.0.1:9092/-/ready
curl -fsS http://127.0.0.1:3100/ready
# Elasticsearch users replace the previous line with:
# curl -fsS http://127.0.0.1:9200/

Open Grafana at http://10.0.0.10:3001 and check the CubeSandbox sandbox resources and CubeSandbox sandbox logs dashboards. Per-sandbox series and logs can be empty when no sandbox is active.

B. Install additional compute nodes

1. Prepare the same one-click release and an .env on each compute node:

[[ -f .env ]] || cp env.example .env
chmod 600 .env

2. Set the first-node address and keep the cluster ID and log backend identical to the first node:

# .env
ONE_CLICK_CONTROL_PLANE_IP=10.0.0.10
ONE_CLICK_ENABLE_OBSERVABILITY=1
CUBE_OBSERVABILITY_CLUSTER_ID=cube-prod
CUBE_LOGS_BACKEND=loki

ONE_CLICK_CONTROL_PLANE_IP is the first CubeSandbox node's private IP, not a newly created monitoring node. install-compute.sh exports ONE_CLICK_DEPLOY_ROLE=compute automatically, so do not repeat it in .env. With no explicit external endpoints, the installer derives the Prometheus and selected log-backend addresses from this IP.

3. Install and inspect the local Collector:

sudo ./install-compute.sh
sudo systemctl --no-pager --full status cube-sandbox-otelcol.service
sudo journalctl -u cube-sandbox-otelcol.service -n 100 --no-pager

4. Create a test sandbox, then query by node_id and sandbox_id in Grafana on the first node. If no data arrives, first check local 127.0.0.1:9998, the log path, and connectivity to the required first-node ports.

C. Enable observability in an existing one-click deployment

Use a one-click release that contains the observability deployment files and keep maintaining the .env in the original installation directory. Do not overwrite it with env.example, and do not edit the generated .one-click.env directly.

1. Add or update these values in .env on the first node:

ONE_CLICK_ENABLE_OBSERVABILITY=1
CUBE_OBSERVABILITY_CLUSTER_ID=cube-prod
CUBE_LOGS_BACKEND=loki
CUBE_GRAFANA_ADMIN_PASSWORD='<random-password-at-least-16-chars>'

2. Run the original one-click entrypoint in upgrade mode:

sudo ./install.sh --mode=upgrade --yes

3. After verifying the first node, add or update these values in .env on every compute node:

ONE_CLICK_CONTROL_PLANE_IP=10.0.0.10
ONE_CLICK_ENABLE_OBSERVABILITY=1
CUBE_OBSERVABILITY_CLUSTER_ID=cube-prod
CUBE_LOGS_BACKEND=loki

sudo ./install-compute.sh --mode=upgrade --yes

Replace 10.0.0.10 with the first node's reachable private IP, and use the same cluster ID and log backend on every node. These commands reconcile the one-click stack and observability components from the original .env; back up the configuration through the normal operations process before running them. On upgrade, a wholesale copy of env.example with its default ONE_CLICK_ENABLE_OBSERVABILITY=0 does not disable an already enabled stack. To explicitly disable it, run sudo env ONE_CLICK_ENABLE_OBSERVABILITY=0 ./install.sh --mode=upgrade --yes (or use ./install-compute.sh on a compute node).

D. Online installation

Online installation follows the same environment-variable forwarding style as bundled MinIO:

# First all-in-one node. Bundled Grafana requires an admin password of at least
# 16 characters; store the generated value in your password manager.
GRAFANA_ADMIN_PASSWORD="$(openssl rand -hex 16)"
echo "Grafana admin password: ${GRAFANA_ADMIN_PASSWORD}"
curl -fsSL https://raw.githubusercontent.com/tencentcloud/CubeSandbox/master/deploy/one-click/online-install.sh \
  | sudo env ONE_CLICK_ENABLE_OBSERVABILITY=1 \
      CUBE_OBSERVABILITY_CLUSTER_ID=cube-prod \
      CUBE_LOGS_BACKEND=loki \
      CUBE_GRAFANA_ADMIN_PASSWORD="${GRAFANA_ADMIN_PASSWORD}" \
      bash

# Compute node
curl -fsSL https://raw.githubusercontent.com/tencentcloud/CubeSandbox/master/deploy/one-click/online-install.sh \
  | sudo env ONE_CLICK_DEPLOY_ROLE=compute \
      ONE_CLICK_CONTROL_PLANE_IP=10.0.0.10 \
      ONE_CLICK_ENABLE_OBSERVABILITY=1 \
      CUBE_OBSERVABILITY_CLUSTER_ID=cube-prod \
      CUBE_LOGS_BACKEND=loki \
      bash

online-install.sh downloads the release and invokes the same install.sh; it is not a separate deployment topology. For authentication tokens and other secrets, prefer downloading the release and storing values in a mode-0600.env rather than leaving secrets in shell history.

E. External backends

ONE_CLICK_OBSERVABILITY_ROLE=auto
CUBE_LOGS_BACKEND=loki
CUBE_METRICS_OTLP_HTTP_ENDPOINT=http://10.0.0.10:9092/api/v1/otlp
CUBE_LOGS_OTLP_HTTP_ENDPOINT=http://10.0.0.10:3100/otlp

With explicit external endpoints, auto installs only the node Collector, not local backends. See 4.1 for external-backend authentication.

2.2 Kubernetes Helm

Deploy the in-cluster backend and the compute-node Collector:

helm upgrade --install cube ./deploy/kubernetes/chart \
  -n cube-system --create-namespace \
  -f /path/to/runtime-values.yaml \
  --set observability.enabled=true \
  --set observability.clusterId=k8s-prod \
  --set observability.logs.backend=loki

The chart generates a release-scoped Grafana admin Secret with a random password and reuses it on upgrades. Read it from the cluster instead of passing credentials on the command line:

kubectl get secret -n cube-system \
  -l app.kubernetes.io/component=grafana-admin \
  -o jsonpath='{.items[0].data.admin-password}' | base64 --decode; echo

Set observability.backend.grafana.existingSecret to a Secret with admin-user and admin-password to manage credentials outside Helm. An inline adminPassword must be at least 16 characters.

To also collect component container logs, add --set observability.logs.components.enabled=true, or set it in runtime-values.yaml:

observability:
  logs:
    components:
      enabled: true

This runs a second Collector that tails only the allowlisted containers (observability.logs.components.containers, default: master, API, Ops, TemplateCenter, Cubelet, proxy, lifecycle manager) in the release namespace, from a read-only /var/log/pods mount, on nodes matching observability.logs.components.nodeSelector, affinity, and tolerations. Collection starts at the file end, so existing history is not replayed. Query it in Grafana with service_name="cubesandbox-cube-master", or use the Component logs row of the logs dashboard. Check component logs for secrets before enabling, and add redaction when needed.

Collector, Prometheus, log backend, and Grafana Pods carry readiness/liveness probes, so Kubernetes gates rollout and restarts unhealthy monitoring Pods. This is not alerting; no Alertmanager rules are installed.

Deploy only the Collector and send data to existing backends:

helm upgrade --install cube ./deploy/kubernetes/chart \
  -n cube-system --create-namespace \
  -f /path/to/runtime-values.yaml \
  --set observability.enabled=true \
  --set observability.backend.enabled=false \
  --set observability.clusterId=k8s-prod \
  --set observability.metrics.otlpEndpoint="http://10.0.0.10:9092/api/v1/otlp" \
  --set observability.logs.backend=loki \
  --set observability.logs.otlpEndpoint="http://10.0.0.10:3100/otlp"

Use the same runtime-values.yaml as a normal chart installation; it must contain non-default database/Redis credentials and placement settings. Observability values are additions to that existing install command. For Elasticsearch, set observability.logs.backend=elasticsearch and, for an external backend, observability.logs.esEndpoint. Do not set both log endpoints.

For authenticated external backends, create a Secret in the release namespace with your existing secret-management workflow and reference it from runtime-values.yaml. Store the complete Authorization header value (for example Bearer ... or Elasticsearch ApiKey ...) and do not pass credentials through --set:

# Example only. Create these Secrets through the cluster's secret-management workflow.
apiVersion: v1
kind: Secret
metadata:
  name: cube-observability-auth
  namespace: cube-system
type: Opaque
stringData:
  metrics-authorization: "Bearer <metrics-token>"
  metrics-tenant-id: "cube-prod"
  logs-authorization: "Bearer <logs-token>"
  logs-tenant-id: "cube-prod"
---
apiVersion: v1
kind: Secret
metadata:
  name: cube-grafana-admin
  namespace: cube-system
type: Opaque
stringData:
  admin-user: "admin"
  admin-password: "<strong-password>"

For an Elasticsearch API key, set logs-authorization to ApiKey <key>. Remove a tenant key when the backend does not require that header.

Reference those Secret keys from runtime-values.yaml:

observability:
  metrics:
    auth:
      existingSecret: cube-observability-auth
      authorizationKey: metrics-authorization
      tenantIdKey: metrics-tenant-id # optional; empty disables X-Scope-OrgID
  logs:
    auth:
      existingSecret: cube-observability-auth
      authorizationKey: logs-authorization
      tenantIdKey: logs-tenant-id # set "" when the backend needs no tenant header
  backend:
    grafana:
      existingSecret: cube-grafana-admin
      adminUserKey: admin-user
      adminPasswordKey: admin-password

HTTPS endpoints use the container system CA. The chart does not yet mount a custom CA Secret; extend the CA mount for a private-CA deployment instead of disabling certificate verification.

The Helm Collector always uses the host network and scrapes the same-node Cubelet at 127.0.0.1:9998. The built-in topology therefore requires the default cubeNode.hostNetwork=true. The chart does not currently discover a reliable per-node metrics target when Cubelet is moved to the Pod network.

Common values:

Value	Default	Meaning
observability.enabled	false	Observability master switch
observability.clusterId	cubesandbox	Cluster label for metrics and logs
observability.otelCollector.scrapeInterval	15s	Cubelet scrape interval; it must not be shorter than the fixed 10s scrape timeout
observability.otelCollector.cubeletMetricsTarget	127.0.0.1:9998	Node-local Cubelet endpoint; the default requires cubeNode.hostNetwork=true
observability.otelCollector.stateHostPath	/var/lib/cube-sandbox/otelcol-k8s	Per-node persistent filelog offsets, queues, and WAL
observability.backend.enabled	true	Deploy Prometheus, the selected log backend, and Grafana with the Collector
observability.metrics.otlpEndpoint	""	External Prometheus OTLP/HTTP base endpoint
observability.metrics.remoteWriteUrl	""	External Prometheus Remote Write URL; alternative to OTLP
observability.metrics.auth.existingSecret	""	Secret containing metrics Authorization / tenant headers
observability.logs.enabled	true	Collect sandbox stdout/stderr logs
observability.logs.components.enabled	false	Collect allowlisted component CRI logs from /var/log/pods with a second DaemonSet
observability.logs.components.hostPath	/var/log/pods	Absolute, non-root host directory holding the pod log tree
observability.logs.components.containers	Cube component list	Container-name allowlist; only these directories are tailed
observability.logs.components.nodeSelector / affinity / tolerations	Linux nodes; control/compute labels and taints	Placement of the component-log Collector
observability.logs.backend	loki	Log backend: loki or elasticsearch
observability.logs.otlpEndpoint	""	External OTLP/HTTP base endpoint for backend=loki
observability.logs.esEndpoint	""	External Elasticsearch endpoint for backend=elasticsearch
observability.logs.auth.existingSecret	""	Secret containing logs Authorization / tenant headers
observability.backend.<component>.persistence.enabled	true	Use a PVC for built-in Prometheus, log backend, and Grafana
observability.backend.grafana.adminPassword	""	Empty generates a random password in a release-scoped Secret; inline values need 16+ characters
observability.backend.grafana.existingSecret	""	Grafana admin Secret (admin-user / admin-password); preferred in production

3. Standalone installation (optional)

The observability components can also be installed without a one-click or Helm release. The standalone installer manages cube-sandbox-otelcol, Prometheus, Loki/Elasticsearch, and Grafana, and does not touch CubeMaster, Cubelet, or other services. Run the preflight first, then pick a backend node reachable from the compute nodes.

3.1 Single node or first backend node

If the backend node also runs sandboxes, install both the backend and its local Collector:

sudo env \
  CUBE_OBSERVABILITY_CLUSTER_ID=cube-prod \
  CUBE_LOGS_BACKEND=loki \
  CUBE_GRAFANA_ADMIN_PASSWORD='REPLACE_WITH_STRONG_PASSWORD' \
  ./deploy/one-click/observability/install.sh all

If that node hosts only the backend and runs no sandboxes, omit the local Collector:

sudo env \
  CUBE_OBSERVABILITY_CLUSTER_ID=cube-prod \
  CUBE_LOGS_BACKEND=loki \
  CUBE_GRAFANA_ADMIN_PASSWORD='REPLACE_WITH_STRONG_PASSWORD' \
  ./deploy/one-click/observability/install.sh backend

Replace the Grafana password before running either command. To use Elasticsearch, set CUBE_LOGS_BACKEND=elasticsearch. Restrict the backend ports as described in 2.1 A.

3.2 Other CVM or bare-metal compute nodes

Install the Collector on each compute node, replacing 10.0.0.10 with the backend node's private IP:

sudo env \
  CUBE_OBSERVABILITY_CLUSTER_ID=cube-prod \
  CUBE_METRICS_OTLP_HTTP_ENDPOINT=http://10.0.0.10:9092/api/v1/otlp \
  CUBE_LOGS_OTLP_HTTP_ENDPOINT=http://10.0.0.10:3100/otlp \
  ./deploy/one-click/observability/install.sh cvm

For Elasticsearch, remove CUBE_LOGS_OTLP_HTTP_ENDPOINT and use:

sudo env \
  CUBE_OBSERVABILITY_CLUSTER_ID=cube-prod \
  CUBE_LOGS_BACKEND=elasticsearch \
  CUBE_METRICS_OTLP_HTTP_ENDPOINT=http://10.0.0.10:9092/api/v1/otlp \
  CUBE_LOGS_ES_ENDPOINT=http://10.0.0.10:9200 \
  ./deploy/one-click/observability/install.sh cvm

Add the variables from 4.1 when the external backend requires a token, tenant header, ES API key, or Basic Auth.

3.3 Verification and uninstall

# Full systemd all-in-one installation
sudo ./deploy/one-click/observability/install.sh status

# Collector-only systemd node
sudo systemctl --no-pager --full status cube-sandbox-otelcol.service

# Remove independently installed systemd components; data directories are retained.
sudo ./deploy/one-click/observability/install.sh uninstall all

For a Kubernetes release, check the Collector DaemonSets instead (names are prefixed with the release fullname, cube below):

kubectl -n cube-system rollout status daemonset/cube-node-otelcol
# Only with observability.logs.components.enabled=true
kubectl -n cube-system rollout status daemonset/cube-component-otelcol

4. OTel Collector configuration

Each node runs otelcol-contrib 0.162.0 and generates up to two independent pipelines according to the configured destinations:

Pipeline	Receiver	Input	Exporter
metrics	prometheus	Cubelet /v1/metrics/resource	otlp_http/metrics or prometheus_remote_write
logs/sandbox	filelog/sandbox	/data/cubelet/log/*/{stdout,stderr}	elasticsearch/logs or otlp_http/logs

These names come from the systemd generator. Helm uses the equivalent prometheus/cubelet_resource_metrics, filelog/sandbox_stdio, and logs names, plus one logs/component_<container> pipeline per allowlisted container in the separate component-log Collector. Implementation details are documented in the OTel design.

4.1 Core environment variables and export authentication

Variable	Default	Example	Meaning
CUBE_OBSERVABILITY_CLUSTER_ID	cubesandbox	cube-prod	Cluster label written to metrics and logs
CUBE_LOGS_BACKEND	loki	elasticsearch	Log-backend selector for one-click/systemd
CUBE_SANDBOX_NODE_IP	auto-detected	10.0.0.20	systemd only: node label; set it when auto-detection is unsuitable
CUBE_METRICS_OTLP_HTTP_ENDPOINT	empty	http://10.0.0.10:9092/api/v1/otlp	OTLP/HTTP base endpoint; omit /v1/metrics
CUBE_METRICS_REMOTE_WRITE_URL	empty	http://10.0.0.10:9092/api/v1/write	Prometheus Remote Write URL; alternative to metrics OTLP
CUBE_METRICS_AUTHORIZATION / CUBE_METRICS_REMOTE_WRITE_TOKEN	empty	Bearer ...	Metrics authentication for one-click/systemd; choose a complete header or a bare token
CUBE_METRICS_TENANT_ID	empty	cube-prod	Metrics X-Scope-OrgID
CUBE_LOGS_ES_ENDPOINT	empty	http://10.0.0.10:9200	Elasticsearch endpoint
CUBE_LOGS_ES_INDEX	cubesandbox-logs	cube-prod-logs	Elasticsearch index
CUBE_LOGS_ES_USER / CUBE_LOGS_ES_PASSWORD	empty	cube / ...	one-click/systemd: ES Basic Auth; both values are required
CUBE_LOGS_ES_API_KEY	empty	...	one-click/systemd: ES API key; alternative to Basic Auth
CUBE_LOGS_ES_TLS_INSECURE_SKIP_VERIFY	false	true	Skip ES TLS verification for temporary diagnostics only
CUBE_LOGS_OTLP_HTTP_ENDPOINT	empty	http://10.0.0.10:3100/otlp	OTLP/HTTP logs base endpoint; alternative to Elasticsearch
CUBE_LOGS_AUTHORIZATION / CUBE_LOGS_TOKEN	empty	Bearer ...	Log authentication for one-click/systemd; choose a complete header or a bare token
CUBE_LOGS_TENANT_ID	empty	cube-prod	Logs X-Scope-OrgID
CUBE_LOGS_SANDBOX_ID	*	019abc-example	Optional sandbox ID filter for the systemd collector
CUBE_LOGS_FILE_START_AT	beginning	end	systemd only: initial file position, either beginning or end

The bundled Grafana provisioning reuses these metrics/log credentials for its datasources. Running grafana/install.sh dashboards creates or updates datasources by UID.

Helm uses the Kubernetes Node name as the node label and does not read these systemd-only variables. Its fixed start_at: beginning applies only when no saved offset exists. Collector state is persisted per node at /var/lib/cube-sandbox/otelcol-k8s, so Pod replacement on the same node resumes from file_storage instead of rereading existing files.

One-click users can use the Observability section already present in deploy/one-click/env.example. After copying it to .env as shown in 2.1, uncomment optional settings only when needed:

ONE_CLICK_ENABLE_OBSERVABILITY=1
CUBE_OBSERVABILITY_CLUSTER_ID=cube-prod
CUBE_LOGS_BACKEND=loki

# The standard all-in-one/compute topology derives these automatically.
# Set them only for a self-managed or external backend.
CUBE_METRICS_OTLP_HTTP_ENDPOINT=http://10.0.0.10:9092/api/v1/otlp
CUBE_LOGS_OTLP_HTTP_ENDPOINT=http://10.0.0.10:3100/otlp

Configure at least one metrics or logs destination. Do not configure both metrics destinations or both logs destinations at the same time. The standard one-click topology fills in its bundled backend endpoints automatically. For Elasticsearch, use CUBE_LOGS_BACKEND=elasticsearch, CUBE_LOGS_ES_ENDPOINT=http://10.0.0.10:9200, and optionally CUBE_LOGS_ES_INDEX.

Authentication in every topology

The supported one-click and Kubernetes workflows can export to authenticated external metrics and log backends. A minimal Bearer example is:

CUBE_METRICS_OTLP_HTTP_ENDPOINT=https://prometheus.example.com/api/v1/otlp
CUBE_METRICS_AUTHORIZATION='Bearer <metrics-token>'
CUBE_LOGS_OTLP_HTTP_ENDPOINT=https://loki.example.com/otlp
CUBE_LOGS_AUTHORIZATION='Bearer <logs-token>'

• the one-click/systemd installer reads these variables from the node's .env file;
• Helm references a Secret through observability.metrics.auth.existingSecret and observability.logs.auth.existingSecret; see 2.2;
• Elasticsearch uses CUBE_LOGS_ES_API_KEY, or the paired CUBE_LOGS_ES_USER and CUBE_LOGS_ES_PASSWORD.

An *_AUTHORIZATION value is the complete single-line header. An *_TOKEN value is a bare token that the installer converts to a Bearer header; do not set both forms. These settings configure Collector/Grafana client headers. They do not enable server-side authentication on bundled Prometheus, Loki, or Elasticsearch. Put an authentication gateway or reverse proxy in front of a bundled backend, or use a managed backend.

Official references: OTLP HTTP exporter, Prometheus Remote Write exporter, Loki authentication, Elasticsearch API keys, and Kubernetes Secrets.

Never commit real credentials. Set .env to mode 0600. HTTPS uses the system CA by default; install or mount a private CA instead of disabling verification.

4.2 Config locations and rendering

render and installation use the same generator. Values already configured in the environment are written into the generated YAML, so the YAML does not need a second manual edit. The one-click wrapper loads .env automatically; for a standalone run, export the variables first or pass them with env.

Topology	Installed configuration	Render with configured values
systemd	/etc/cube-sandbox/otelcol/otelcol.yaml	./deploy/one-click/observability/install.sh render cvm
Helm	ConfigMaps <release>-node-otelcol-config and <release>-component-otelcol-config	Use the helm template command below

Helm uses observability.* values from values.yaml or --set; it does not read CUBE_* environment variables automatically.

Even a render-only Helm command must satisfy the chart-wide credential validation with non-default placeholders:

helm template cube ./deploy/kubernetes/chart \
  --set observability.enabled=true \
  --set-string mysql.password=render-only \
  --set-string mysql.rootPassword=render-only \
  --set-string redis.password=render-only \
  -s templates/observability.yaml

Use render-only only for local rendering. Supply real credentials through secret values for an actual installation.

4.3 Generated configuration example

The excerpt below assumes the example values from 4.1 and node address 10.0.0.20, sending metrics to Prometheus and logs to Loki. It omits some exporter discover/queue/retry and service.telemetry fields for readability; use render for the complete output. Comments explain environment-variable mappings and are not guaranteed to appear in the generated file.

This is generated output, not a second configuration source. render and install substitute the configured environment values automatically. Rerun the installation after changing those values.

receivers:
  # Scrape sandbox resource metrics from the local Cubelet only.
  prometheus:
    trim_metric_suffixes: false
    config:
      scrape_configs:
        - job_name: cubesandbox-resource
          scrape_interval: 5s
          scrape_timeout: 3s
          metrics_path: /v1/metrics/resource
          static_configs:
            - targets: ["127.0.0.1:9998"]
  # CUBE_LOGS_SANDBOX_ID replaces "*"; the default collects every sandbox.
  filelog/sandbox:
    include:
      - /data/cubelet/log/*/stdout
      - /data/cubelet/log/*/stderr
    start_at: beginning # CUBE_LOGS_FILE_START_AT
    storage: file_storage
    include_file_path: true
    include_file_name: false
    max_log_size: 1MiB
    operators:
      - type: regex_parser
        parse_from: attributes["log.file.path"]
        regex: '^/data/cubelet/log/(?P<sandbox_id>[^/]+)/(?P<stream>stdout|stderr)$'
      - type: copy
        from: attributes["log.file.path"]
        to: attributes.log_file_path
      - type: copy
        from: attributes.sandbox_id
        to: attributes.cubesandbox_sandbox_id
      - type: move
        from: attributes.sandbox_id
        to: resource["cubesandbox.sandbox.id"]
      - type: copy
        from: attributes.stream
        to: attributes.log_iostream
      - type: move
        from: attributes.stream
        to: attributes["log.iostream"]

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 256
  # Values come from CUBE_OBSERVABILITY_CLUSTER_ID and
  # CUBE_SANDBOX_NODE_IP (auto-detected when unset).
  resource/metrics:
    attributes:
      - {key: cluster, value: "cube-prod", action: upsert}
      - {key: node, value: "10.0.0.20", action: upsert}
      - {key: cubesandbox.cluster.id, value: "cube-prod", action: upsert}
      - {key: host.name, value: "10.0.0.20", action: upsert}
  resource/logs_common:
    attributes:
      - {key: host.name, value: "10.0.0.20", action: upsert}
      - {key: cubesandbox.cluster.id, value: "cube-prod", action: upsert}
  attributes/logs_common:
    actions:
      - {key: host_name, value: "10.0.0.20", action: upsert}
      - {key: cubesandbox_cluster_id, value: "cube-prod", action: upsert}
      - {key: cubesandbox_sandbox_id, value: "none", action: insert}
      - {key: log_iostream, value: "none", action: insert}
  resource/sandbox:
    attributes:
      - {key: service.name, value: "cubesandbox-sandbox", action: upsert}
      - {key: cubesandbox.log.source, value: "sandbox-init", action: upsert}
  attributes/sandbox:
    actions:
      - {key: service_name, value: "cubesandbox-sandbox", action: upsert}
      - {key: cubesandbox_log_source, value: "sandbox-init", action: upsert}
  batch:
    timeout: 2s
    send_batch_size: 1024

exporters:
  otlp_http/metrics:
    # CUBE_METRICS_OTLP_HTTP_ENDPOINT; the Collector appends /v1/metrics.
    endpoint: "http://10.0.0.10:9092/api/v1/otlp"
    compression: gzip
    sending_queue:
      enabled: true
      storage: file_storage
    retry_on_failure:
      enabled: true
      max_elapsed_time: 0
  otlp_http/logs:
    # CUBE_LOGS_OTLP_HTTP_ENDPOINT; the Collector appends /v1/logs.
    endpoint: "http://10.0.0.10:3100/otlp"
    compression: gzip
    sending_queue:
      enabled: true
      storage: file_storage
    retry_on_failure:
      enabled: true
      max_elapsed_time: 0

extensions:
  health_check:
    endpoint: 127.0.0.1:13133
  # Persist file offsets and the OTLP sending queue across process restarts.
  file_storage:
    directory: /var/lib/cube-sandbox/otelcol/storage
    create_directory: true

service:
  extensions: [health_check, file_storage]
  pipelines:
    # Metrics and logs are independent pipelines.
    metrics:
      receivers: [prometheus]
      processors: [memory_limiter, resource/metrics, batch]
      exporters: [otlp_http/metrics]
    logs/sandbox:
      receivers: [filelog/sandbox]
      processors: [memory_limiter, resource/logs_common, resource/sandbox, attributes/logs_common, attributes/sandbox, batch]
      exporters: [otlp_http/logs]

With Elasticsearch selected, the log exporter becomes elasticsearch/logs; its endpoint and index come from CUBE_LOGS_ES_ENDPOINT and CUBE_LOGS_ES_INDEX. The receiver, field extraction, and pipeline stay the same.

4.4 Supported rules and recommended settings

Metric and log collection rules

Type	Supported rule	Configuration
Sandbox metrics	Scrape only the local Cubelet /v1/metrics/resource; includes sandbox CPU, memory, and node sandbox counts	Cluster/node labels and either an OTLP or Remote Write destination; Helm also exposes observability.otelCollector.scrapeInterval
Sandbox logs	Read /data/cubelet/log/<sandbox-id>/{stdout,stderr} line by line and extract the sandbox ID and stream from the path	Send to Loki over OTLP by default, or to Elasticsearch; systemd also exposes CUBE_LOGS_SANDBOX_ID and CUBE_LOGS_FILE_START_AT
Component logs (Helm opt-in)	Read CRI/containerd logs only for observability.logs.components.containers from /var/log/pods; start at end and use one pipeline per component	Separate read-only component-log Collector DaemonSet on matching control/compute nodes; one-click does not support this option
State and retry	Use file_storage for file offsets and OTLP queues, a WAL for Remote Write, and an in-memory queue with at most five retries for Elasticsearch	File state lives on the host for systemd and at /var/lib/cube-sandbox/otelcol-k8s per Kubernetes node; component logs use a separate -components host path

For the systemd collector, each original log line is kept in Body, with a maximum log-record size of 1MiB.

Both backends retain these semantic fields (Loki normalizes . in OTel attribute names to _):

• service_name=cubesandbox-sandbox
• cubesandbox_log_source=sandbox-init
• cubesandbox_cluster_id
• host_name
• cubesandbox_sandbox_id
• log_iostream=stdout|stderr
• log_file_path
• the log body and timestamp

Component CRI records use service_name=cubesandbox-<container>, cubesandbox_log_source=component, and cubesandbox_component_name=<container>; they do not contain the sandbox-only ID and stream fields above.

Loki indexes only the low-cardinality service_name, cubesandbox_cluster_id, and host_name labels. cubesandbox_sandbox_id, log_iostream, and the remaining fields stay as structured metadata. This avoids creating one high-cardinality stream per sandbox. Common LogQL filters:

# All sandbox stderr
{service_name="cubesandbox-sandbox"} | log_iostream="stderr"

# Error-like messages from one sandbox
{service_name="cubesandbox-sandbox"}
  | cubesandbox_sandbox_id="<sandbox-id>"
  |~ "(?i)error|fatal|panic"

Elasticsearch documents additionally contain Body and @timestamp. Common Lucene filters:

# All sandbox stderr
service_name:"cubesandbox-sandbox" AND log_iostream:"stderr"

  # Error-like messages from sandbox stdout/stderr
cubesandbox_log_source:"sandbox-init" AND Body:(ERROR OR FATAL OR panic)

# One sandbox
cubesandbox_sandbox_id:"<sandbox-id>"

Recommended Collector settings

Setting	Built-in value	Recommendation
Scrape interval / timeout	systemd: 5s / 3s; Helm: 15s / 10s	Keep script defaults; Helm intervals must not be shorter than the fixed 10s timeout, and larger deployments should keep the default 15s
CUBE_LOGS_FILE_START_AT (systemd only)	beginning	Keep beginning to read the current files in full; use end to collect only records written after the Collector starts
CUBE_LOGS_SANDBOX_ID (systemd only)	*	Keep * normally; specify one ID only for focused troubleshooting or temporary volume reduction
Memory limiter	Fixed at 256MiB; Kubernetes memory request/limit 128Mi / 512Mi	Start with the defaults; raising only the Pod limit does not change the limiter, which currently has no installer setting
Batch	1024 records; timeout 2s for scripts and 5s for Helm	Keep the defaults for normal low-latency delivery
OTLP queue	1000, disk-backed with file_storage, retry without an elapsed-time limit	Ensure the host state directory has enough capacity
Elasticsearch queue	In-memory 1000, 10 consumers, at most 5 retries	Sustained backend outages can eventually drop logs; monitor Collector logs and backend availability
Remote Write	WAL-backed	Ensure the WAL directory has enough capacity and monitor remote-write failures
Prometheus retention	one-click: 15d / 40GB; Helm: 15d	For one-click use CUBE_PROMETHEUS_RETENTION_TIME / CUBE_PROMETHEUS_RETENTION_SIZE; for Helm use observability.backend.prometheus.retention, which currently has no size value
Loki retention	one-click and Helm: 15d	Use CUBE_LOKI_RETENTION_PERIOD for one-click or observability.backend.loki.retention for Helm
Elasticsearch heap	one-click and Helm: 1g	Tune CUBE_ES_HEAP_SIZE or observability.backend.elasticsearch.javaOpts, normally keeping heap below 50% of available memory

Except for settings with an environment variable or Helm value listed above, scrape timeout, memory limiter, batch, and queue values are installer-owned defaults and should not be maintained by editing generated files.

Helm's bundled Prometheus, selected log backend, and Grafana use PVCs by default. Configure each component's persistence.storageClassName, existingClaim, or size under observability.backend; set persistence.enabled=false only for disposable validation. Use an external managed backend when high availability is required. Bundled Loki defaults to 15d; configure Elasticsearch ILM separately when Elasticsearch is selected.

Suggested monitoring expressions

The installer provisions dashboards but does not install Alertmanager, Prometheus alert rules, Grafana alerts, or Elasticsearch ILM. Suggested PromQL starting points are shown below. Use for: 5m for the CPU, throttling, memory ratio, and unavailable-sample conditions. Fire the memory-failure counter immediately or use only a short delay.

# CPU usage above 80% of a finite sandbox CPU limit
100 * rate(cubesandbox_host_sandbox_cpu_usage_seconds_total[5m])
  / clamp_min(cubesandbox_host_sandbox_cpu_limit_cores, 0.001) > 80

# CPU throttled periods above 20%
100 * rate(cubesandbox_host_sandbox_cpu_throttled_periods_total[5m])
  / clamp_min(rate(cubesandbox_host_sandbox_cpu_periods_total[5m]), 0.001) > 20

# Memory usage above 85% of a finite sandbox memory limit
100 * cubesandbox_host_sandbox_memory_current_bytes
  / clamp_min(cubesandbox_host_sandbox_memory_limit_bytes, 1) > 85

# Any memory-limit failure during the last five minutes
increase(cubesandbox_host_sandbox_memory_failures_total[5m]) > 0

# A sandbox exists but has no available resource sample
cubesandbox_node_sandbox_count
  - cubesandbox_node_sandbox_available_count > 0

CPU and memory ratios only apply when a finite resource-limit series exists.

4.5 Extending collection to component logs

Helm already collects allowlisted component container logs (see 2.2); deploy/kubernetes/chart/templates/observability.yaml is the reference implementation. The one-click/systemd installer collects sandbox logs only.

Never edit the installed /etc/cube-sandbox/otelcol/otelcol.yaml or the ConfigMap: installation regenerates them. Change the generator instead:

• systemd: write_otel_config() in deploy/one-click/observability/otelcol/install.sh;
• Helm: the Collector ConfigMap and volumes in deploy/kubernetes/chart/templates/observability.yaml, behind a values.yaml switch that stays off by default.

Use one receiver and pipeline per component (filelog/component_cubelet, logs/component_cubelet) so unrelated formats never share a parser, and set both the OTel resource attribute and the flat Elasticsearch field:

• service.name / service_name: stable low-cardinality name, e.g. cubesandbox-cubelet;
• cubesandbox.log.source / cubesandbox_log_source: component;
• cubesandbox.component.name / cubesandbox_component_name: e.g. cubelet;
• keep the existing cubesandbox.cluster.id, host.name, log.file.path, and timestamp fields.

Constraints for any extension:

• read files with start_at: end and a dedicated parser for JSON or multiline formats; a parser failure must keep the original Body and must not block other pipelines;
• index only stable labels; keep PID, request ID, and sandbox ID as structured metadata;
• mount only the required host directories, read-only; never the host root;
• exclude credentials, tokens, and headers, adding a redaction or transform processor when needed;
• verify file_storage offsets across log rotation, restarts, and backend outages;
• update the generator, both documentation languages, the dashboards, and the Collector validate tests, and state which topologies stay unsupported.

5. Backends and dashboards

The bundled backend is Prometheus, Grafana, and one log backend. Grafana provisions three separate metrics dashboards plus one logs dashboard:

• cubesandbox-cluster.json: sandbox resource totals for one cluster.
• cubesandbox-node.json: sandbox resource totals for one node in that cluster.
• cubesandbox-sandbox.json: one sandbox's Host Sandbox and Guest Workload metrics, with an optional guest-container filter.
• cubesandbox-logs.json (Loki, default) or cubesandbox-logs-elasticsearch.json (Elasticsearch), both under the UID cubesandbox-logs: stdout/stderr volume, log query, and a collapsed Component logs row (Helm opt-in).

Metrics selectors cascade from cluster to node, sandbox_id, and container_id. All 24 currently exported metric families are covered; aggregate CPU rates are cores, and memory limits are sums of exported finite limits, not physical host capacity. Host and Guest measurements must not be added together. Missing data is not zero, and guest rates omit windows with observed epoch changes. The Loki sandbox filter remains a regex text input for historical IDs; Elasticsearch selectors cascade from cluster to node to sandbox.

See 1.2 for backend versions, default ports, and external-component baselines.

After a one-click/systemd installation, run sudo ./deploy/one-click/observability/install.sh status to check the Collector and backends. Check the default Loki with curl -fsS http://127.0.0.1:3100/ready; for Elasticsearch, use curl -fsS 'http://127.0.0.1:9200/cubesandbox-logs*/_count'. A sandbox removal deletes its local log directory, so the Collector must continuously tail and export logs while the sandbox exists.