Entropy Analysis for Executable Layers

Imposed rule: Entropy evidence must be included in scan exports and DSSE attestations; opaque regions without provenance cannot be whitelisted without an explicit policy waiver. Status: Partially implemented; target-state sections are retained below (verified 2026-07-19) Owners: Scanner Guild · Policy Guild · UI Guild · Docs Guild

Current implementation boundary (2026-07-19)

• The Worker registers EntropyStageExecutor and runs it before SurfaceManifestStageExecutor. It discovers eligible files from the materialized root filesystem or resolves virtual source-workspace entries without permitting paths to escape the declared root.
• The implemented analyzer computes 4 KiB/1 KiB sliding-window Shannon entropy over whole eligible files (ELF, PE, Mach-O, and binary blobs over 16 KiB). It does not parse ELF/PE/Mach-O section tables or detect UPX/ASPack section names. PE HIGH_ENTROPY_VA is an ASLR hardening flag, not content entropy.
• Current Worker output is the canonical Surface Manifest payload pair entropy.report and entropy.layer-summary. It is not yet projected into a policy entropy penalty or automatically posted into WebService scan state.
• WebService currently accepts a separate operator/internal POST /api/v1/scans/{scanId}/entropy ratio snapshot and returns that snapshot from the normal scan-status response. There is no implemented GET /api/v1/scans/{id}/entropy endpoint or Worker-to-WebService attach client.
• The DSSE/export, trust-lattice penalty, CLI, UI heatmap, direct-download, and notification behavior below is target state and must not be treated as shipped.

1. Overview

Entropy analysis highlights opaque regions inside container layers (packed binaries, stripped blobs, embedded firmware) so Stella Ops can prioritise artefacts that are hard to audit. The scanner computes per-file entropy metrics, reports opaque ratios per layer, and feeds penalties into the trust algebra.

2. Scanner pipeline (SCAN-ENTROPY-186-011/012)

• Target files: ELF, PE/COFF, Mach-O executables and large raw blobs (>16 KB). Archive formats (zip/tar) are unpacked by existing analyzers before entropy processing.

• Section analysis:

  • ELF – .text, .rodata, .data, custom sections.
  • PE – section table entries (IMAGE_SECTION_HEADER).
  • Mach-O – LC_SEGMENT/LC_SEGMENT_64 sections.

• Sliding window: 4 KB window with 1 KB stride. Entropy calculated using Shannon entropy:

  [ H = -\sum_{i=0}^{255} p_i \log_2 p_i ]

  Windows with H ≥ 7.2 bits/byte are marked “opaque”.

• Heuristics & hints:

  • Flag entire files with no symbols or stripped debug info.
  • Detect known packer section names (.UPX*, .aspack, etc.).
  • Record offsets, window sizes, and entropy values to support explainability.

• Outputs (all canonical, UTF-8, sorted keys):

  • entropy.report.json (per-file details, windows, hints; schema stellaops.entropy/report@1).
  • layer_summary.json (opaque byte ratios per layer and overall image; schema stellaops.entropy/layer-summary@1).
  • entropy_penalty scalar injected into trust lattice inputs.

• All outputs are signed within the scan DSSE bundle and exported in Offline/Replay kits.

All JSON output is canonical (sorted keys, UTF-8) and included in DSSE attestations/replay bundles.

3. JSON Schemas

3.1 entropy.report.json

{
  "schema": "stellaops.entropy/report@1",
  "generatedAt": "2025-11-26T12:00:00Z",
  "imageDigest": "sha256:…",
  "layerDigest": "sha256:…",
  "files": [
    {
      "path": "/opt/app/libblob.so",
      "size": 5242880,
      "opaqueBytes": 1342177,
      "opaqueRatio": 0.25,
      "flags": ["stripped", "section:.UPX0"],
      "windows": [
        { "offset": 0, "length": 4096, "entropy": 7.45 },
        { "offset": 1024, "length": 4096, "entropy": 7.38 }
      ]
    }
  ]
}

3.2 layer_summary.json

{
  "schema": "stellaops.entropy/layer-summary@1",
  "generatedAt": "2025-11-26T12:00:00Z",
  "imageDigest": "sha256:…",
  "layers": [
    {
      "digest": "sha256:layer4…",
      "opaqueBytes": 2306867,
      "totalBytes": 10485760,
      "opaqueRatio": 0.22,
      "indicators": ["packed", "no-symbols"]
    }
  ],
  "imageOpaqueRatio": 0.18,
  "entropyPenalty": 0.12
}

4. Policy integration (POLICY-RISK-90-001)

• Policy Engine receives entropy_penalty and per-layer ratios via scan evidence.
• Default thresholds (tenant-overridable):
  • Block when imageOpaqueRatio > 0.15 and provenance unknown.
  • Warn when any executable has opaqueRatio > 0.30.
  • Suppress penalty when symbols are present and provenance attested.
• Trust lattice mapping:
  • entropy_penalty feeds the risk lattice alongside reachability, provenance, and exploitability signals; capped at 0.3.
  • Policy explanations include highest-entropy files, offsets, and reason codes (packed, no symbols, runtime reachable).

5. UI experience (UI-ENTROPY-40-001/002)

• Heatmaps: render entropy along the file timeline (green → red).
• Layer donut: show opaque % per layer with tooltips linking to file list.
• “Why risky?” chips: highlight triggers such as Packed-like, Stripped, No symbols.
• Policy banners explain configured thresholds and mitigation (add provenance, unpack, or accept risk).
• Provide direct download links to entropy.report.json for audits.

6. CLI / API hooks

• CLI – stella scan artifacts --entropy --threshold 0.15 --top 10 prints top opaque files and penalty; exits non-zero when penalty exceeds threshold.
• CLI – stella scan export --include entropy bundles entropy reports with SBOM/VEX for Offline kits.
• API – GET /api/v1/scans/{id}/entropy serves summary + evidence references; supports Accept: application/json or NDJSON stream.
• Notify templates can include entropy penalties to escalate opaque images (channel: Ops/Sec).

7. Trust algebra

The penalty is computed as:

[ \text{entropyPenalty} = \min\Bigg(0.3,; K \sum_{\text{layers}} \big( \frac{\text{opaqueBytes}}{\text{totalBytes}} \times \frac{\text{layerBytes}}{\text{imageBytes}} \big)\Bigg) ]

• Default K = 0.5; tenants can override via policy lattice config.
• If symbols are present and attested, apply a 0.5 multiplier to the per-layer contribution.
• Combine with reachability and provenance weights before final risk verdict.

8. Implementation checklist

Update this document as thresholds change or additional packer signatures are introduced.