WeKnora: Building a Persistent Agent Runtime on CubeSandbox

By｜Tencent Technical Experts, WeKnora Project Maintainers · Chen Yang, Zhao Hailong

Editor's note: WeKnora is Tencent's open-source enterprise LLM knowledge platform. Since open-sourcing in August 2025, its popularity has kept climbing, now at 24.4k stars. It turns enterprise document assets into RAG Q&A, ReAct agents, and a self-maintaining wiki.

Its sandbox layer supports three backends — Cube Sandbox, E2B, and Docker — with Cube Sandbox already running the full chain of session binding, skill snapshots, pause/resume, template management, and network policies. This article records how WeKnora designed and built on Cube Sandbox, and answers a more fundamental question: what role should a sandbox play in an Agent platform?

1. The Full Picture of Cube Sandbox in WeKnora

WeKnora's latest release, v0.8.0, centers on the skill sandbox runtime: session-resident Docker/E2B/Cube sandbox backends with per-space network policies. In WeKnora, the sandbox is not an auxiliary executor for one-off commands — it is the runtime environment for Agent skills.

Docker / E2B (including E2B Cloud and any E2B-compatible control plane) / Cube are three parallel backends. The application layer doesn't branch by vendor; it goes through the same RemoteSandboxClient + session binding, and deployers choose by isolation strength and operational form. On the CubeSandbox path, WeKnora's usage falls into seven main areas:

Usage	Cube Mechanism	What It Solves
Session-persistent sandbox	Create + metadata session binding	One session binds to one sandbox; packages installed, files created, and services started in the last turn are still there in the next
Skill snapshotting	CreateSnapshot (snapshot ID as TemplateID)	A sandbox with skills installed is snapshotted; new sessions are created directly from the snapshot, skills ready in seconds
Pause/resume	onTimeout=pause + autoResume	Auto-pause when idle, auto-resume on the next Connect, memory state preserved
Session filesystem	envd Files API	Attachments, artifacts, and the cross-tool-call workspace all live in /workspace
Command execution	Commands API + stdin injection	The Agent runs scripts inside the session with traceable results
Template management	Template CRUD + dedicated image variant	weknora-sandbox:*-cube images carrying the envd data plane
Network policies	Outbound/inbound dual switches + L7 rules	Per-space control over whether a sandbox can reach the internet or be reached publicly

The three backends in WeKnora's sandbox layer can be swapped without changing a single line of upper-layer business code. Two layers of design support this constraint:

• The first layer is the neutral interface RemoteSandboxClient: six lifecycle methods — Create, Connect, Get, List, Delete, Exec — plus four auxiliary interfaces for files, snapshots, templates, and inbound tokens. Cube-specific types, error codes, and HTTP semantics are all translated into neutral DTOs and RemoteErrors with stable Kinds; not a single line leaks outward.
• The second layer is capability advertisement: the application layer never writes if cube; instead, it queries capability accessors at runtime — in-session command execution, session filesystem, turn markers, install commands. If an accessor returns nil, the capability is unavailable under the current configuration, and the application layer decides which features to register accordingly.

Figure 1: WeKnora x Cube Sandbox integration architecture

CubeRemoteClient carries all the adaptation code for the Cube backend, consolidated in a single file; it implements four interfaces at once, and all calls are divided into two planes:

Plane	Call Contents	Transport Path
Control plane	Create / Connect / List / Template CRUD / Snapshot CRUD	Through a shared gateway connection pool, reusing the same transport as the E2B backend
Data plane (envd)	Files operations (Write/Read/List/MakeDir/Remove/Stat) + Commands execution	Preserves the SDK's proxy dial rewriting (ProxyNodeIP/Port/Scheme), wrapped in a dedicated transport

Currently, WeKnora integrates Cube in three main forms: bare metal, PVM, and K8s (still in preview).

2. The Sandbox's Role Shift: From "Run and Destroy" to "Persistent Runtime"

These capability and architecture designs in the usage panorama only evolved into their current shape after several real-world problems.

The initial backend was Docker, implemented as docker run --rm every time — run and destroy. It exposed three problems: 1) missing session state — packages installed in one turn were gone the next; 2) shell execution, attachment staging, and artifact collection couldn't be registered in the capability matrix; 3) timeouts killed the client process while the container kept running in the background. Only after switching to "one session, one long-lived container" did behavior align with Cube/E2B.

But Docker's ceiling was also obvious: it couldn't deliver three of our underlying requirements — cross-host scheduling, kernel-level isolation, and memory-state snapshots. For scenarios running model-generated code, a shared-kernel isolation boundary wasn't enough either. These three gaps pushed our selection toward remote sandbox services.

For remote sandbox integration, WeKnora chose the E2B protocol. The turning point came when the skill persistence requirement appeared. Skill environments need to be built, baked into images, and version-managed — these are control-plane capabilities. The E2B protocol also covers template building and network control; the difference is granularity: snapshotting a running sandbox directly, using a snapshot ID as a template, and outbound control extending allowOut/denyOut into L7 rules. These are exactly the capabilities the skill-image approach needs, and Cube's control-plane API maps to them one by one. This is why WeKnora maintains a dedicated Cube adapter alongside the E2B adapter.

WeKnora's initial design for skill persistence was "one skill, one volume" — each skill mounted into the sandbox as a volume. Because E2B Volumes was still in private beta at the time, our first release switched to Cube, and before writing any code we swapped the whole approach from volume mounts to snapshots. This forced detour later proved to be the better path.

At this point in the evolution, the sandbox's role had fundamentally changed. It is no longer an execution container for "running a command in a sandbox" — it is the persistent runtime environment for Agent skills: skills are pre-installed inside, sessions happen inside, and files "live" inside.

3. Skill Persistence: Turning Snapshots into Releases

The cost model of skill installation dictates that it cannot be amortized across sessions.

What does installing a skill involve? Parsing SKILL.md, installing system packages and Python/Node dependencies, running install verification. That alone is a multi-minute agent conversation. Having users wait for a live install when they start a session is too slow; installing one copy per session is too wasteful. So the skill environment must be frozen into an immutable artifact, and sessions start directly from that artifact.

Why snapshots instead of volumes? The two represent opposite models. Volumes are shared and writable — good for datasets. What a skill environment needs is "verified and immutable": after installation, the skills directory is owned by root and read-only; ad-hoc pip install in a session can only land in the /workspace overlay and cannot pollute the image. For skills that depend on system packages, the volume approach is hard to support and carries the risk of installed skills being modified by the agent.

This path works because of four properties of Cube snapshots:

• A snapshot ID can be used directly as the TemplateID in CreateOptions: the session side never knows the word "skill" — it simply swaps to a different template.
• Snapshots can be taken of a running sandbox: the install process is ordinary exec plus file writes, with no separate image-build pipeline needed.
• XFS reflink/CoW keeps the storage density and incremental cost of "base template + one layer of installed dependencies" reasonable.
• Snapshots share the same lifecycle semantics as sandbox keep-alive, with complete supporting tooling.

The full install chain: create a sandbox from the base template → the installer runs the installation → verification passes → the ledger is persisted first, then CreateSnapshot is called → the SnapshotID is recorded. All subsequent new sessions are created with the snapshot ID in place of the template ID — skills ready in seconds.

Snapshot "ownership" is guarded by a fingerprint mechanism. The fingerprint is SHA-256(provider + APIKey + APIURL) — identifying the provider account the snapshot lives in. The install path, snapshot-validity checks, and config parsing all compute the fingerprint from the same inputs, guaranteeing consistent judgment. After credential rotation, fingerprints mismatch, old snapshots silently invalidate, and sessions automatically fall back to the base template — instead of starting up with an image ID that no longer exists. When the fingerprint is empty, the install flow refuses to record the snapshot outright; pointers without an owner are discarded at session startup.

Choosing the snapshot approach also has its costs, mainly two. Cube returns snapshots and regular templates from the same list API, so WeKnora's settings page must filter out the snap- prefix and the snapshot list — otherwise admins will pick a skill image as the base template. Additionally, as skills accumulate in snapshots with each install, the image slowly bloats.

4. Session Persistence: Keep-Alive and /workspace

Agent sessions last minutes to hours, interspersed with thinking, waiting on users, and waiting on retrieval. If the sandbox is killed when the TTL hits, the user's next message pays a cold start, and intermediate files in /workspace, overlay packages, and uncollected artifacts all vanish.

So session sandboxes are created with onTimeout=pause + autoResume: when idle, the MicroVM is frozen — saving compute while preserving memory state — and the next Connect wakes it automatically. From the user's perspective: the conversation is still there, and so is the environment.

The Docker backend's handling makes an interesting contrast. Docker's pause still holds host memory, so freezing means not reclaiming. The Docker backend therefore kills on idle, and the binding layer rebuilds on next use — its aligned semantics are "if it's gone, treat it as rebuildable," not keep-alive. Different backends choosing their own semantics under the same abstract interface is exactly the value of the neutral interface layer.

The other half of the environment is the filesystem. If envd's read/write were only for the installer, the Agent would be "blind" inside sessions. WeKnora shaped the Files API into a session filesystem, solving four things:

• Attachments: object storage is the source of truth; at session start they're restored to /workspace/input (read-only by convention), and skills get this directory via environment variables;
• Artifacts: scripts write to /workspace/output; at the end of a turn, they're collected and downloaded as the same tree — no "model reads the file and pastes it into chat," and no context window consumed;
• Model-written files: generated scripts land directly on disk instead of being stuffed into shell command heredocs;
• Cross-tool-call workspace: multiple executions within the same session see the same tree. This is the part of the "persistent runtime" that is visible to the model.

The skill image is a read-only "release"; /workspace is the writable "machine for this turn." Two paths, two sets of permissions — the Files API is the formal entry point for the latter.

5. Identity Persistence: The Redis Binding Layer and Orphan Reclamation

Cube's TTL plus AutoPause manages the MicroVM's own lifetime: pause when idle, resume on the next Connect. It knows nothing about WeKnora's sessions, tenants, replicas, or skill generations. All of these application semantics sit on the Redis binding layer:

What Cube Provides	What Redis Adds
Sandbox ID, TTL, pause	Authoritative session → sandbox binding (multi-replica WeKnora must share it; in-memory binding only works for single-process development)
Single Create	Lifecycle lock: create/recover/replace/delete serialized across processes, preventing one session from starting two VMs
metadata (we tag tenant/session/config)	After binding loss, claim via metadata; only create new if unclaimable. TTL/pause can't save you from "the binding write was lost"
Sandbox expires on its own	Bindings never expire (SET NX, TTL=0). If the session lives on and the sandbox is paused, the binding must exist so you resume instead of buying another machine
No concept of "image has changed"	StaleAt marker + turn lease (see below)
Paused still occupies snapshot storage and money	Orphan reclamation: after a binding is overwritten or Redis is lost, paused sandboxes occupy disk. Cube won't delete based on "does WeKnora still recognize it"

The most typical problem this layer faces is billing. Session sandboxes use onTimeout=pause; once a binding is lost, the Cube side holds a paused VM that occupies storage indefinitely. So a reaper layer is required: periodically reconcile by tenant metadata and delete unbound instances (including paused ones).

The division of labor between the two layers is therefore crystal clear: Cube keeps the sandbox alive; Redis governs "which session this sandbox belongs to, whether it can be torn down, and whether the image should be swapped."

6. The Turn Lease: Admin Installs a Skill While the User Is Mid-Conversation

Persistent environments bring a unique conflict: the admin is installing a skill while the user is mid-conversation — they're contending for the same VM.

The concrete scenario: one Agent turn resolves the sandbox many times — staging attachments, several command executions, running skill scripts, collecting artifacts — all assuming /workspace and processes still exist. Skill installation often happens mid-conversation: after the first tool call, the admin finishes the install, and the second tool call tears down the VM — this turn's drafts, packages installed into the overlay, and running execs all disappear. From the model's perspective: "the files that were just there are gone."

Not tearing down doesn't work either. After a skill installs successfully, if existing sessions aren't marked, the skill the user just installed stays invisible in the current conversation forever — the pointer switch only affects newly created sandboxes.

WeKnora's solution separates "declaring" from "acting":

• StaleAt is the declaration: the image has changed, but no action yet.
• rebuild=1 at BeginTurn: the first resolve of this turn is allowed to tear down and rebuild; ConsumeTurnRebuild immediately resets it to zero. Subsequent resolves in the same turn keep using the current sandbox even if still stale.
• With no lease (no AgentQA in flight), stale still rebuilds immediately — background tasks and idle sessions don't need to wait.
• When the Redis lease read fails, treat it as a turn in flight — don't tear down.

"Rebuild only once" guards against another scenario: multiple resolves within one turn plus another install arriving in between, tearing down the VM repeatedly. Once the ticket is consumed, no more rebuilds this turn. Turn markers leaked by process crashes are expired by a 30-minute TTL backstop — rebuilding a stale image at that point is expected behavior.

7. What Was Pre-Designed, What Was Learned the Hard Way, and a Checklist for Those Who Follow

Among the designs above, some were laid down in advance during architecture design:

• Backend-agnostic RemoteSandboxClient, capabilities advertised via capability, never if cube.
• Session-level sandbox + Redis binding + lifecycle lock.
• SSRF protection for tenant-controlled URLs: validate at save time + validate again at dial time (against DNS rebinding); even when private networks are allowed, link-local / cloud metadata endpoints are blocked.
• Skills installed into snapshots, sessions started from snapshots; ledger written before CreateSnapshot.
• Scripts and non-root user (uid 1000) execution; installation goes through a separate root interface.
• Working directory locked to /workspace, skill tree at /opt/weknora/tenant/skills, scratch cleaned before snapshotting.

Among the pre-designed items, network policy handling deserves its own explanation. Cube's network control has two switches governing two orthogonal dimensions: allowInternetAccess governs outbound — whether the sandbox can actively reach the public internet, with curl/pip connectivity determined by it layered with allowOut/denyOut/L7 rules; allowPublicTraffic governs inbound — whether the sandbox's public URL is publicly reachable; when off, all inbound traffic must carry a traffic token or gets 403. WeKnora explicitly sets both switches when creating every sandbox, guarding against default drift — omit them in config, and the server falls back to template defaults; change the template, and production behavior changes with it. allowPublicTraffic explicitly set to true is a hard requirement: WeKnora's files, execution, and terminal all go through CubeProxy data-plane URLs — turn it off, and our own access gets 403 first. Currently, outbound is fully open by default — a deliberate starting choice, since skill installation needs to pull packages, so connectivity comes first; fine-grained per-space/per-session outbound allowlists are still being tightened.

The other part is experience distilled from pitfalls we stepped into during operation:

• Cube templates must carry envd, otherwise the :49983/health probe gets connection refused — hence the dedicated -cube image variant.
• Snapshots mixed into the template list — the settings page must filter the snap- prefix.
• ListSnapshots pagination tokens repeating in a loop causes failure — skill orphan cleanup once deadlocked on this.
• Generic E2B data-plane compatibility: envd requires Basic auth, an X-User-ID header, and multipart uploads while the SDK sends raw octet-stream — E2B Cloud is lenient; other implementations return 401/500 directly.
• Saving from the settings page once wiped the SkillImage pointer — skills "in the list but not in sessions" — the update API now forbids clients from touching snapshot fields and endpoints.
• Tearing down a VM mid-turn destroys /workspace — the turn lease.
• Named-config field-level inheritance once silently dialed to 127.0.0.1 — changed to named configs being self-contained, not inheriting the deployment baseline's endpoint.
• A command blacklist blocked even the recovery suggestion for pip install — changed to making skills read-only after installation with kernel-level write denial, and only then prompting the overlay path on failure.

For teams also building Agent platforms, we recommend verifying these eight things first (along the "skill persistence" path, not "as long as exec works"):

1. Can a snapshot ID be used directly as a template to create new sandboxes? Are snapshots mixed into the template list?
2. When snapshotting a running instance, does the original sandbox remain usable? Is snapshotting during pause stable?
3. With pause + Connect auto-resume, are the filesystem and memory still there? Are paused instances still billed? How do you List them?
4. Can metadata set at Create be Listed back by tenant/session — without this, multi-replica and crash recovery are left to the luck of your own binding store;
5. Do the two outbound switches really take effect orthogonally as documented? Does DNS work inside private networks?
6. The data-plane envd contract — Basic auth, multipart upload, non-root account. Don't test only against the official gateway; it's more lenient;
7. Does the image include envd? Without it, health checks fail immediately;
8. Pagination, delete idempotency, name echo — skill images are billed resources; when a process dies in the "created but not yet persisted" window, you must be able to claim by name or reconcile by List.

At the architecture level, capabilities WeKnora still lacks include: mounting volumes at Create (Cube's CreateOptions has no volume-mount field yet — there's currently no path for shared datasets or hot-updating large files); separating snapshots and templates into different directories (mixing them pollutes the "pick a base template" product experience and adds mis-deletion risk); application-layer callbacks for pause/timeout (today we can only scan List ourselves — WeKnora isn't notified when TTL hits, so orphans can only be reconciled periodically); and first-class session identity — Cube manages VMs, not "which session this is," so bindings, leases, staleness, and orphans all live on the application side. If the control plane could natively recognize metadata ownership and reclaim unclaimed instances by policy, the Redis layer could slim down significantly.

WeKnora project: https://github.com/Tencent/WeKnora