Plugin sandbox on Linux — operator runbook

Sprint 20260501-052 PLG-LINUX-OVL-06. Operator-facing companion to the architecture in docs/modules/release-orchestrator/modules/plugin-system.md“Workspace layout seam”. Architecture-level rationale lives there; this document covers what an operator needs to know to deploy, configure, diagnose, and recover the Linux plugin sandbox.

Audience

Site reliability engineers, platform operators, and security responders who own a Stella Ops host running Linux plugin workloads. Plugin authors should continue to follow the manifest/SDK guidance in the Plugin module dossier; this document is only about the host filesystem and process boundary.

Strategy selection

The Linux workspace seam (IPluginSandboxWorkspaceFactory, DefaultPluginSandboxWorkspaceFactory.Prepare) materializes exactly one strategy per sandbox start. The three Linux-relevant strategies are:

1. Kernel overlayfs (PluginSandboxWorkspaceStrategy.KernelOverlay, strategy name kernel-overlay) — runs mount -t overlay overlay -o lowerdir=<lower>,upperdir=<upper>,workdir=<work> <merged>. Requires a kernel that advertises overlay in /proc/filesystems, CAP_SYS_ADMIN in the effective set of the Stella Ops host process, and a sandbox root on a filesystem the kernel accepts as overlay-able (avoid tmpfs upper layers on RHEL family hosts; ext4/xfs/btrfs upper is fine).
2. fuse-overlayfs (PluginSandboxWorkspaceStrategy.FuseOverlay, strategy name fuse-overlayfs) — runs fuse-overlayfs -o lowerdir=<lower>,upperdir=<upper>,workdir=<work> <merged>. Requires the fuse-overlayfs binary in PATH and the fuse kernel module loaded. Useful inside rootless containers.
3. Materialized copy (compatibility) (PluginSandboxWorkspaceStrategy.MaterializedCopy, strategy name materialized-copy) — the host copies the verified plugin install directory into merged and exposes the copy. Reads and writes both target the copy; lower is never mounted. This is flagged degraded (PluginSandboxWorkspace.IsDegraded == true) whenever the sandbox mode requires an overlay filesystem (OutOfProcessOverlay/OutOfProcessConfined).

Reality check (verified against DefaultPluginSandboxWorkspaceFactory.ResolveStrategy, 2026-05-29). There is no automatic kernel→fuse→copy priority fallback in the current code. ResolveStrategy returns the strategy named in SandboxWorkspaceOptions.ForcedStrategy when one is set, and otherwise always returns MaterializedCopy. The kernel-overlay and fuse-overlayfs paths are only reached when the caller explicitly sets ForcedStrategy. Materialized copy therefore remains the default even for OutOfProcessConfined/Untrusted workloads — it is marked degraded in that case but is not yet suppressed. An auto-selecting fallback chain (probe kernel overlay → fall back to fuse-overlayfs → fail closed) is roadmap, not shipped. Treat the numbered list above as the set of available strategies, not an automatic priority order.

The one safety guard that is enforced: when the mode requires an overlay filesystem and SandboxWorkspaceOptions.AllowMaterializedCopyFallback is set to false while no ForcedStrategy is supplied, ResolveStrategy fails closed with plugin.workspace.overlay_required rather than silently copying. AllowMaterializedCopyFallback defaults to true.

Selection is deterministic: the same options always choose the same strategy. When a forced kernel-overlay or fuse-overlayfs mount is rejected by the kernel, the sandbox fails closed with plugin.workspace.mount_failed rather than degrading to a copy.

Workspace layout

Every sandbox start materializes a fresh, per-invocation upper/work/merged triple under <sandbox-root>/<sandbox-id>/. The lower layer is not a subdirectory of the sandbox root — it is the verified plugin install directory itself, referenced in place as the read-only overlay lower layer:

lower   -> the verified plugin install directory (the directory containing the
           plugin assembly), referenced read-only — NOT copied or relocated
           under the sandbox root

<sandbox-root>/<sandbox-id>/
  upper/    -> writable scratch layer (per-invocation, deleted on cleanup)
  work/     -> overlayfs work directory (per-invocation, deleted on cleanup)
  merged/   -> mountpoint exposed to the plugin process as its working directory

(PluginSandboxWorkspaceLayout records Lower = the plugin install directory and Upper/Work/Merged = <root>/<sandbox-id>/{upper,work,merged}.)

The host process never mutates lower directly. Under the overlay strategies all writes from the plugin land in upper. After the sandbox stops, merged is unmounted and the per-sandbox root (upper/work/merged) is removed; the plugin install directory is untouched.

Materialized-copy exception. Under the materialized-copy strategy lower is not mounted at all: the install directory is recursively copied into merged and the plugin reads and writes the copy directly. There is no read-only lower layer protecting the original in that mode — the install directory stays untouched only because the copy is a separate tree.

The factory validates paths before any directory is created: it rejects non-absolute paths, parent-directory traversal (..), source/workspace overlap, and (when RejectReparsePoints is true, the default) symlinks or reparse points anywhere inside the plugin install directory.

Operator settings

Caveat — these are code-level configuration properties, not environment variables (verified 2026-05-29). The sandbox reads its workspace and AppArmor settings from SandboxConfiguration / SandboxWorkspaceOptions / AppArmorOptions objects constructed in process (see SandboxFactory). There is no environment-variable binding layer in the sandbox library: the STELLAOPS_PLUGIN_SANDBOX_* names (…_LINUX_STRATEGY, …_ROOT, …_ALLOW_COPY_FALLBACK, …_OS_CONFINEMENT) are not read by any sandbox code and do not configure the sandbox. (One CI test reads STELLAOPS_PLUGIN_SANDBOX_ROOT to locate a runner-provisioned sandbox root, but the production factory does not.) An env-var/host-config binding for these properties is roadmap.

Persisted operator policy (whether Untrusted plugins may run on hosts without verified OS confinement) is not an environment variable and must not be set by env. It lives in the platform settings store; default production behavior is fail-closed.

The factory also exports workspace metadata to the spawned plugin process through these environment variables (set by ProcessSandbox, read by the plugin host, not operator-tunable): STELLAOPS_SANDBOX_WORKSPACE_STRATEGY, STELLAOPS_SANDBOX_WORKSPACE_DEGRADED, STELLAOPS_SANDBOX_WORKSPACE_LOWER, STELLAOPS_SANDBOX_WORKSPACE_UPPER, STELLAOPS_SANDBOX_WORKSPACE_WORK, and STELLAOPS_SANDBOX_WORKSPACE_MERGED.

Degraded-posture warning

ProcessSandbox.StartAsync emits a structured warning log line (not a structured event) when the prepared workspace is flagged degraded (PluginSandboxWorkspace.IsDegraded). In practice that is the materialized-copy strategy running under an overlay-requiring mode (OutOfProcessOverlay/OutOfProcessConfined); a materialized-copy workspace for a non-overlay mode is not flagged degraded and does not log this line. The log line includes:

• Id — sandbox correlation id;
• Strategy — e.g. MaterializedCopy;
• Lower, Upper, Work, Merged — full paths.

Verified against ProcessSandbox.StartAsync (2026-05-29): the warning log carries no reason field (there is no opt-in vs prerequisite-missing distinction in the emitted line), and it fires off IsDegraded, not off “every materialized-copy start”. The previous wording on both points was incorrect.

The structured confinement-event subsystem from sprint 054 PLG-OS-CONF-05 has landed (IPluginSandboxEventEmitter, PluginSandboxEvent, and the event-name catalog including plugin.sandbox.workspace.degraded_fallback). However, the workspace factory does not yet emit through it — it has no event-emitter dependency, so the degraded-workspace signal is still only the ProcessSandbox warning log above. (The AppArmor attach path and the Windows Job Object path do emit structured events; the workspace-strategy path does not.) Operators should alert on this log line and treat any production occurrence as an incident-grade degradation, not a normal operating mode.

AppArmor kernel-denial events (auditd scraper)

Sprint 20260504-002 KENF-AA-02.

When SandboxConfiguration.AppArmor.EnableAuditLogScraping is set to true, the hosting layer is expected to construct an ApparmorAuditLogScraper and call StartAsync; the scraper tails SandboxConfiguration.AppArmor.AuditLogPath (default /var/log/audit/audit.log). For every line carrying the kernel signature apparmor="DENIED" that also has a parseable msg=audit(...) prefix and a non-empty profile= and operation=, the scraper emits one plugin.sandbox.confinement.profile_denied event via the standard IPluginSandboxEventEmitter surface.

No profile-name filtering (verified against ApparmorAuditLineParser + ApparmorAuditLogScraper.EmitDenied, 2026-05-29). The scraper does not match the profile= field against the configured Stella profile name — it emits for every parseable AppArmor denial in the tailed log, carrying the denying profile in Diagnostics["profile_name"]. On a host whose auditd also covers non-Stella AppArmor profiles, expect denials from those profiles to surface as events too. Per-sandbox attribution comes from the PID registry (below), not from the profile name.

Not auto-wired. The toggle is read from configuration, but neither AddPluginSandbox (DI) nor SandboxFactory constructs or starts the scraper today — it is a standalone library type. A host that wants kernel denials mirrored to events must instantiate ApparmorAuditLogScraper itself, supply the emitter and (for correlation) the shared ISandboxPidRegistry, and call StartAsync.

Operator notes:

• Failure posture is no-op, not throw. If the audit log does not exist, is not readable by the host process, or disappears mid-tail, the scraper logs a single warning and stops emitting. It never throws out of StartAsync and never aborts the host. Use the IsNoOp property in a health-check probe if you want to surface this as a visible degradation.
• Pre-existing denials are NOT replayed. The scraper seeks to the end of the audit log on start, so historical denials accumulated before the host process was running do not surface as events. Operators investigating past incidents must read the audit log directly.
• PID correlation requires registration — and is not wired today. Per-sandbox attribution needs the spawner to call ISandboxPidRegistry.Register(pid, sandboxId) at start and Unregister(pid) at exit against the same registry instance the scraper resolves against. In the current code ProcessSandbox does not register child PIDs, and ApparmorAuditLogScraper defaults to a fresh SandboxPidRegistry() when none is injected — so without explicit host wiring every emitted event carries sandboxId = "sandbox:unknown". That is still useful for SOC investigation (the denying profile, PID, path, and masks are all in Diagnostics) but loses the per-sandbox join. To get the join, a host must share one registry instance between the process spawner and the scraper and perform the register/unregister calls itself.
• Container/runner caveat. /var/log/audit/audit.log is not present in default container images and is typically read-only outside the host’s auditd group. Hosts running plugins inside containers should either bind-mount the log read-only or leave the toggle off and rely on the host’s own auditd consumer.
• No rate-limiting in v1. A misbehaving plugin can drive thousands of denials per second. The events contract permits aggregation (docs/modules/plugin/sandbox-confinement-events.md §9) but the v1 scraper does not implement it; if the host is hit by storm-pattern denials, disable the toggle and treat the underlying plugin as compromised.

Cleanup behavior

A normal sandbox stop performs, in order:

1. Graceful plugin shutdown via the gRPC bridge.
2. Process exit (graceful, with kill-after-timeout fallback).
3. umount <merged> (kernel overlay or fuse-overlayfs strategy).
4. rm -rf <upper> <work> <merged> constrained to the sandbox root.
5. Removal of the per-sandbox directory under <sandbox-root>.

Materialized-copy strategy skips the unmount step (it has no mount); cleanup is a plain recursive delete of the per-sandbox root.

If step 3 fails on a Linux overlay strategy (busy mount, EBUSY because a child is still holding the view, kernel refuses to release), the unmount cleanup raises a PluginSandboxWorkspaceException with code plugin.workspace.unmount_failed carrying the merged path, the unmount exit code, and stderr. (The plugin.workspace.cleanup_failed code is raised by the Windows junction path’s bounded-retry root delete, not by the Linux unmount path — verified against PluginSandboxWorkspace.cs, 2026-05-29.) Operators must:

• terminate any holding processes (see “Diagnosing stale mounts” below);
• unmount manually, or run the stale-mount sweep (LinuxStaleMountSweeper, sprint 052 PLG-LINUX-OVL-04 — implemented and integration-tested, but not yet auto-invoked at host start; see “Open follow-ups”);
• only after the path is unmounted, delete the leftover scratch state.

The sandbox must not force-unmount with umount -l (lazy) on busy mounts — that hides held file descriptors and lets a misbehaving plugin keep reading from a “deleted” workspace. Lazy unmount is reserved for emergency operator intervention and is documented separately below.

Diagnosing stale mounts

When the host crashes or a previous sandbox shut down uncleanly, mount points may be left behind under <sandbox-root>. Symptoms:

• a new sandbox start that reuses the same <sandbox-id> deletes the leftover root before recreating it (Directory.Delete(root, recursive:true) in PrepareLinuxMountWorkspace); a stale mount beneath that root can make the delete fail and surface a start error;
• mount | grep <sandbox-root> shows entries that are not owned by any running plugin process;
• disk usage under the sandbox root grows unexpectedly between restarts.

Note on workspace_in_use: the current workspace factory does not probe for stale mounts on start and does not raise a plugin.workspace.workspace_in_use error — that string exists only as a structured event name (plugin.sandbox.workspace.workspace_in_use) / reason code in the confinement-event catalog, reserved for the host-start stale-mount sweep once it is wired in. The earlier claim that a new start “fails with plugin.workspace.workspace_in_use” did not match the code.

Triage commands (run as the Stella Ops service account or root):

# 1. enumerate Stella-owned mounts under the sandbox root only.
#    Set SANDBOX_ROOT to the configured SandboxWorkspaceOptions.RootDirectory
#    for this host (the platform does NOT export it; the default when unset is
#    <temp>/stellaops-sandbox, i.e. typically /tmp/stellaops-sandbox).
SANDBOX_ROOT=${SANDBOX_ROOT:-/var/lib/stellaops/plugin-sandbox}
findmnt --raw --output TARGET,SOURCE,FSTYPE,OPTIONS \
  | awk -v root="$SANDBOX_ROOT" 'index($1, root) == 1'

# 2. find processes holding the mount or any path beneath it
sudo lsof +D "$SANDBOX_ROOT" 2>/dev/null

# 3. unmount one stale entry (only after lsof shows no live holder)
sudo umount /var/lib/stellaops/plugin-sandbox/<sandbox-id>/merged

# 4. emergency only — when umount refuses and no live holder exists
#    (e.g., kernel module bug). Use lazy unmount sparingly:
sudo umount -l /var/lib/stellaops/plugin-sandbox/<sandbox-id>/merged

Never unmount paths above <sandbox-root>. The future stale-mount sweep (PLG-LINUX-OVL-04) will enforce this constraint in code; operator muscle memory should match.

Troubleshooting matrix

Capability prerequisites

For kernel overlay strategy (recommended for production):

• kernel ≥ 4.18 (overlayfs metacopy and redirect_dir behavior stable);
• overlay advertised in /proc/filesystems;
• service host has CAP_SYS_ADMIN in its effective set (systemd: AmbientCapabilities=CAP_SYS_ADMIN);
• sandbox root resides on a filesystem the kernel accepts as upper layer (ext4, xfs, btrfs — not tmpfs on RHEL/CentOS family, which the kernel rejects with EINVAL).

For fuse-overlayfs strategy (rootless / unprivileged):

• fuse-overlayfs ≥ 1.10 (older versions miss xattr passthrough);
• fuse kernel module loaded;
• /dev/fuse accessible to the service account;
• user_allow_other set in /etc/fuse.conf only when sandbox root is shared with non-Stella consumers (uncommon).

Related sprints and docs

• Architecture: docs/modules/release-orchestrator/modules/plugin-system.md “Workspace layout seam” and “Linux AppArmor confinement” sections.
• Structured event contract: docs/modules/plugin/sandbox-confinement-events.md (plugin.sandbox.workspace.*, plugin.sandbox.confinement.*; §9 covers the optional aggregation the v1 scraper does not yet implement).
• Sprint 052 (overlay sandbox — archived): docs-archive/implplan/SPRINT_20260501_052_Plugin_linux_overlay_sandbox.md.
• Sprint 054 (OS confinement that consumes this layout — archived): docs-archive/implplan/SPRINT_20260501_054_Plugin_os_process_confinement.md.
• Sprint 20260504-002 (AppArmor kernel enforcement + auditd scraper — archived): docs-archive/implplan/SPRINT_20260504_002_Plugin_apparmor_kernel_enforcement_followup.md.
• Threat model: docs/security/plugin-sandbox-threat-model.md (sprint 054 PLG-OS-CONF-06).
• CI lanes: .gitea/workflows/plugin-linux-overlay-sandbox.yml (overlay) and .gitea/workflows/plugin-linux-os-confinement.yml (AppArmor).

Open follow-ups

Status reconciled against the archived sprint trackers and src/ on 2026-05-29. The overlay/fuse/sweeper code shipped and was validated; remaining gaps are about host-start wiring and an env/config binding layer.

• Strategy auto-selection / env binding (not shipped). The factory has no kernel→fuse→copy priority fallback and no STELLAOPS_PLUGIN_SANDBOX_* environment-variable binding. Strategy is MaterializedCopy unless the host sets SandboxWorkspaceOptions.ForcedStrategy in code. Wiring host config / env into these properties (and suppressing the copy default for Untrusted) is the open item.
• PLG-LINUX-OVL-02/03 — DONE. Kernel overlay and fuse-overlayfs paths shipped and were exercised with real mounts on a WSL2 runner (writes-land-in-upper + lower-immutable proven) and on the Gitea Linux lane. The earlier “only unit-level command builders, no real mounts” caveat is obsolete.
• PLG-LINUX-OVL-04 — code DONE, host-start wiring open. LinuxStaleMountSweeper (parse /proc/self/mountinfo, filter strictly to the sandbox root, unmount + scratch-remove) is implemented and integration-tested, but it is not yet invoked at host start (no DI registration, no IHostedService). Until that wiring lands, operators run the manual triage commands above.
• PLG-OS-CONF-02/03/05 — DONE. AppArmor attach (fail-closed for Untrusted), the SELinux sample policy (advisory), and the structured confinement-event subsystem all shipped. KENF-AA-01/02 then proved kernel-enforced denials and the auditd profile_denied scraper on the apparmor-host Gitea runner. Residual: the workspace-strategy path still emits only the ProcessSandbox degraded-posture log line (not the plugin.sandbox.workspace.degraded_fallback structured event), and the auditd scraper is not auto-wired — see the relevant sections above.