Multi-Tenant Policy Rollout Flow

Implementation status (reconciled against src/Policy). The cross-tenant, percentage-staged, auto-canary rollout orchestration described in this document is NOT IMPLEMENTED as a single feature. There is no rollout engine, no impact-analysis pre-flight, no percentage/scan-volume tenant selection, no automatic-rollback trigger evaluation, and no blue-green / canary-traffic / feature-flag strategy executor anywhere in the Policy module. The JSON/YAML rollout-plan, impact-analysis, and stage-metrics payloads below are forward-looking design drafts, not contracts emitted by any running service.

What does exist today (verified in source) and composes most of this story:

• Policy Engine HTTP surface for the per-pack lifecycle: POST /api/policy/packs (create pack), POST .../revisions (Draft or Approved), POST .../revisions/{v}/bundle (compile + sign), POST .../revisions/{v}/evaluate, and POST .../revisions/{v}:activate with two-person-approval enforcement. Revision lifecycle states are exactly Draft -> Approved -> Active (StellaOps.Policy.Engine/Domain/PolicyPackRecord.cs).
• Policy Registry library (StellaOps.Policy.Registry, no HTTP host of its own — consumed via the CLI GitOps path and the typed PolicyRegistryClient): environment promotion bindings (Dev/Stage/Prod) with modes Manual, AutomaticOnApproval, Scheduled, Canary; rollback by binding/steps; promotion history; publish pipeline (signing/attestation); and a review workflow (submit / approve / reject / request-changes). The only IPolicyPackStore implementation shipped in the library is the test-harness InMemoryPolicyPackStore; production registration is store-agnostic (AddPolicyPackStore<TStore>) but no durable (e.g. Postgres) store exists in this repo, so in practice the Registry is in-memory only.

Sections that map onto real code are tagged [IMPLEMENTED]; sections that describe unbuilt orchestration are tagged [DRAFT / NOT IMPLEMENTED].

Overview

The Multi-Tenant Policy Rollout Flow describes how StellaOps intends to propagate policy changes across multiple tenants in a controlled, auditable manner. The target design supports staged rollouts, canary deployments, and rollback capabilities for enterprise policy governance.

Note on terminology: today the building blocks are per-pack revisions and per-environment promotion bindings (Dev -> Stage -> Prod), not percentage-staged multi-tenant cohorts. “Promotion” in the codebase moves one pack version into one environment for one tenant; it is not a fan-out across a tenant population.

Audience: policy administrators and platform engineers managing policy-pack lifecycle and environment promotion. Read the implementation-status banner above first — the staged multi-tenant rollout orchestration is forward-looking design; the live surface is the Implemented Lifecycle.

Business Value: Centralized policy management with controlled rollout reduces risk of policy changes breaking production workflows while ensuring consistent security standards across the organization.

Actors

Tenant hierarchy / inheritance — NOT IMPLEMENTED. There is no organization -> team -> project tenant tree in Authority and no policy inherits_from resolution anywhere in src/. The tenant model is flat: a tenant is an opaque identifier carried in the envelope-bound stellaops:tenant claim. The Authority “tenant hierarchy” actor role and the Scheduler “Get tenant hierarchy” call in the diagram below are aspirational. Policy notification events do carry an effective_scope block (tenants / projects / purl_patterns / cpe_patterns / tags — see Notifications/PolicyProfileNotificationModels.cs), but that is a flat scope filter, not an inheritance hierarchy.

Prerequisites

• Multi-tenant environment configured (flat tenant catalog in Authority)
• Caller holds the relevant policy:* scopes (see Authorization Scopes)

[DRAFT / NOT IMPLEMENTED] The original prerequisites below assume features that do not exist yet:

• Tenant hierarchy defined (org → teams → projects) — no hierarchy exists.
• Policy inheritance rules established — no inherits_from resolution exists.
• Rollout approval workflow configured — there is a per-revision two-person activation workflow and a per-pack review workflow, but no rollout-level approval object.

Tenant Hierarchy

[DRAFT / NOT IMPLEMENTED] — illustrative only; Authority stores a flat tenant list, not the tree shown here.

Organization (acme-corp)
├── Team: Platform Engineering
│   ├── Project: core-services
│   └── Project: infrastructure
├── Team: Product Development
│   ├── Project: web-app
│   ├── Project: mobile-api
│   └── Project: data-pipeline
└── Team: Security
    └── Project: security-tools

Flow Diagram

[DRAFT / NOT IMPLEMENTED] — the staged-rollout choreography (Scheduler-driven tenant-tree fetch, canary stage, 25%/100% cohorts, monitor windows) is a target design. No service emits or consumes these messages today. For the implemented sequence see Implemented Lifecycle below.

┌─────────────────────────────────────────────────────────────────────────────────┐
│              Multi-Tenant Policy Rollout Flow (TARGET DESIGN)                     │
└─────────────────────────────────────────────────────────────────────────────────┘

┌──────────┐  ┌─────────┐  ┌───────────┐  ┌──────────┐  ┌────────┐  ┌────────┐
│  Policy  │  │ Policy  │  │ Scheduler │  │ Authority│  │ Policy │  │ Notify │
│  Admin   │  │ Store   │  │           │  │          │  │ Engine │  │        │
└────┬─────┘  └────┬────┘  └─────┬─────┘  └────┬─────┘  └───┬────┘  └───┬────┘
     │             │             │             │            │           │
     │ Create      │             │             │            │           │
     │ policy v2   │             │             │            │           │
     │────────────>│             │             │            │           │
     │             │             │             │            │           │
     │             │ Store as    │             │            │           │
     │             │ draft       │             │            │           │
     │             │───┐         │             │            │           │
     │             │   │         │             │            │           │
     │             │<──┘         │             │            │           │
     │             │             │             │            │           │
     │ Define      │             │             │            │           │
     │ rollout     │             │             │            │           │
     │────────────────────────────>            │            │           │
     │             │             │             │            │           │
     │             │             │ Get tenant  │            │           │
     │             │             │ hierarchy   │            │           │
     │             │             │────────────>│            │           │
     │             │             │             │            │           │
     │             │             │ Tenant tree │            │           │
     │             │             │<────────────│            │           │
     │             │             │             │            │           │
     │ Rollout     │             │             │            │           │
     │ plan        │             │             │            │           │
     │<────────────────────────────            │            │           │
     │             │             │             │            │           │
     │ Approve     │             │             │            │           │
     │────────────────────────────>            │            │           │
     │             │             │             │            │           │
     │             │             │ Stage 1:    │            │           │
     │             │             │ Canary      │            │           │
     │             │             │─────────────────────────>│           │
     │             │             │             │            │           │
     │             │             │             │            │ Apply to  │
     │             │             │             │            │ canary    │
     │             │             │             │            │ tenant    │
     │             │             │             │            │───┐       │
     │             │             │             │            │   │       │
     │             │             │             │            │<──┘       │
     │             │             │             │            │           │
     │             │             │ Monitor     │            │           │
     │             │             │ (24h)       │            │           │
     │             │             │───┐         │            │           │
     │             │             │   │         │            │           │
     │             │             │<──┘         │            │           │
     │             │             │             │            │           │
     │             │             │ Stage 2:    │            │           │
     │             │             │ 25% tenants │            │           │
     │             │             │─────────────────────────>│           │
     │             │             │             │            │           │
     │             │             │ ...         │            │           │
     │             │             │             │            │           │
     │             │             │ Stage N:    │            │           │
     │             │             │ 100%        │            │           │
     │             │             │─────────────────────────>│           │
     │             │             │             │            │           │
     │             │             │ Complete    │            │           │
     │             │             │───────────────────────────────────────>
     │             │             │             │            │           │
     │ Rollout     │             │             │            │           │ Notify
     │ complete    │             │             │            │           │ admins
     │<────────────────────────────────────────────────────────────────────
     │             │             │             │            │           │

Authorization Scopes

[IMPLEMENTED] — canonical scope claim values from StellaOps.Auth.Abstractions/StellaOpsScopes.cs. These are the scope claim values (some use :, some use .); ASP.NET policy names may differ from the prose. Verify against the scope constant when wiring callers.

There is no dedicated policy:rollout scope, because no rollout orchestration endpoint exists. A hypothetical rollout engine would most likely compose policy:promote + policy:activate + authority:tenants.read.

Implemented Lifecycle [IMPLEMENTED]

The flow that real services execute today:

1. Author — POST /api/policy/packs creates a pack container (policy:edit), then POST /api/policy/packs/{packId}/revisions upserts a revision with InitialStatus of Draft or Approved and an optional RequiresTwoPersonApproval flag.
2. Compile + sign — POST /api/policy/packs/{packId}/revisions/{v}/bundle compiles the DSL into an immutable, signed bundle (fails fast on syntax errors / rule conflicts; returns the bundle digest).
3. Evaluate (dry run) — POST /api/policy/packs/{packId}/revisions/{v}/evaluate runs the compiled bundle against an advisory/VEX/SBOM context deterministically (policy:read).
4. Activate — POST /api/policy/packs/{packId}/revisions/{v}:activate (policy:activate) promotes an Approved revision to Active. When two-person approval is required it returns 202 Accepted with pending_second_approval until a distinct second approver signs off, then 200 OK with activated. Activation is audited (AuditActions.Policy.ActivateRevision).
5. (Library) Environment promotion — via the StellaOps.Policy.Registry library: a PromotionBinding moves a pack version into an environment (Dev/Stage/Prod) under mode Manual, AutomaticOnApproval, Scheduled, or Canary. PromotionBindingRules gate this with RequiredApprovers, MinimumApprovals, RequireSuccessfulSimulation, MinimumSimulationInputs, MinimumSoakPeriod, and AllowedSourceEnvironments. RollbackAsync restores a prior binding by id or by steps-back (RollbackRequest.TargetBindingId / StepsBack). The only shipped IPolicyPackStore is the test-harness InMemoryPolicyPackStore — no durable store implementation exists, and none of these Registry services (IPromotionService, IReviewWorkflowService, IPublishPipelineService) is referenced by the StellaOps.Policy.Engine host, so the promotion/review/publish surface is not exposed as a Policy HTTP route.

The per-pack review workflow (IReviewWorkflowService) and publish pipeline (IPublishPipelineService) likewise live in the Registry library (Pending -> InReview -> ChangesRequested/Approved/Rejected/Cancelled; publish states Published/Revoked/Superseded). They are not wired to a running Policy service host in this repo.

Step-by-Step (Target Design) [DRAFT / NOT IMPLEMENTED]

Everything in this section describes a rollout engine that does not exist. The payloads are illustrative drafts. The closest implemented analogues are listed in Implemented Lifecycle above.

1. Policy Creation

A policy revision is authored in the DSL (the syntax "stella-dsl@1" version string is real and validated in PolicyParser.cs). The richer rollout-plan metadata fields (version_tag, changes_from_v1) are not part of any compiled contract:

# Policy Set: production-v2
version: "stella-dsl@1"
name: production
version_tag: "v2.0.0"
description: "Updated production policy with KEV blocking"

changes_from_v1:
  - added: block-kev-vulnerabilities
  - modified: critical-threshold (9.0 → 8.5)
  - removed: legacy-exception-rule

rules:
  - name: block-kev-vulnerabilities
    description: Block any KEV-listed vulnerability
    condition: kev == true
    action: FAIL
    severity: critical

  - name: no-critical-reachable
    condition: |
      severity == 'critical' AND
      cvss >= 8.5 AND
      reachability IN ['SR', 'RO', 'CR']
    action: FAIL

2. Rollout Plan Definition

Platform Admin defines rollout strategy:

{
  "rollout_id": "rollout-789",
  "policy_set": "production",
  "from_version": "v1.0.0",
  "to_version": "v2.0.0",
  "strategy": "staged",
  "stages": [
    {
      "name": "canary",
      "description": "Single low-risk tenant",
      "tenants": ["platform-eng-core-services"],
      "duration": "24h",
      "success_criteria": {
        "max_new_failures": 5,
        "max_failure_rate_increase": 0.05
      },
      "auto_proceed": false
    },
    {
      "name": "early-adopters",
      "description": "25% of tenants (by scan volume)",
      "selection": {
        "method": "percentage",
        "value": 25,
        "weight_by": "scan_volume"
      },
      "duration": "48h",
      "success_criteria": {
        "max_new_failures": 20,
        "max_failure_rate_increase": 0.10
      },
      "auto_proceed": true
    },
    {
      "name": "majority",
      "description": "75% of tenants",
      "selection": {
        "method": "percentage",
        "value": 75
      },
      "duration": "24h",
      "auto_proceed": true
    },
    {
      "name": "full",
      "description": "100% of tenants",
      "selection": {
        "method": "all"
      }
    }
  ],
  "rollback": {
    "automatic": true,
    "triggers": [
      {"metric": "failure_rate_increase", "threshold": 0.20},
      {"metric": "new_critical_blocks", "threshold": 50}
    ]
  }
}

3. Impact Analysis

Before approval, system analyzes potential impact:

{
  "impact_analysis": {
    "rollout_id": "rollout-789",
    "analysis_date": "2024-12-29T10:00:00Z",
    "historical_data_range": "30d",
    "results": {
      "total_scans_analyzed": 15234,
      "predicted_new_failures": 127,
      "predicted_failure_rate_change": "+0.83%",
      "affected_images": 89,
      "by_team": [
        {"team": "Product Development", "new_failures": 78},
        {"team": "Platform Engineering", "new_failures": 31},
        {"team": "Security", "new_failures": 18}
      ],
      "top_triggered_rules": [
        {"rule": "block-kev-vulnerabilities", "count": 45},
        {"rule": "no-critical-reachable", "count": 82}
      ],
      "recommendation": "PROCEED_WITH_CAUTION"
    }
  }
}

4. Approval and Initiation

Policy Admin approves rollout after review:

{
  "approval": {
    "rollout_id": "rollout-789",
    "approved_by": "policy-admin@acme.com",
    "approved_at": "2024-12-29T11:00:00Z",
    "approval_notes": "Impact acceptable. Notified affected teams.",
    "notifications_sent": [
      {"channel": "slack", "target": "#platform-engineering"},
      {"channel": "email", "target": "team-leads@acme.com"}
    ]
  }
}

5. Staged Execution

Scheduler executes each stage:

Stage 1: Canary

{
  "stage_execution": {
    "rollout_id": "rollout-789",
    "stage": "canary",
    "started_at": "2024-12-29T11:00:00Z",
    "tenants_activated": ["platform-eng-core-services"],
    "status": "monitoring"
  }
}

Stage Monitoring

{
  "stage_metrics": {
    "rollout_id": "rollout-789",
    "stage": "canary",
    "monitored_period": "24h",
    "metrics": {
      "scans_evaluated": 234,
      "new_failures": 3,
      "failure_rate_before": 0.12,
      "failure_rate_after": 0.13,
      "success_criteria_met": true
    }
  }
}

6. Progressive Rollout

After canary success, proceed to next stages:

{
  "stage_progression": {
    "rollout_id": "rollout-789",
    "completed_stages": ["canary", "early-adopters", "majority"],
    "current_stage": "full",
    "tenants_on_v2": 47,
    "tenants_on_v1": 0,
    "total_rollout_duration": "96h",
    "status": "completed"
  }
}

7. Rollback (If Needed)

If success criteria not met, automatic rollback:

{
  "rollback": {
    "rollout_id": "rollout-789",
    "triggered_at": "2024-12-30T15:30:00Z",
    "trigger_reason": "failure_rate_increase exceeded 0.20 threshold",
    "rollback_stage": "early-adopters",
    "tenants_rolled_back": 12,
    "action": "reverted to v1.0.0",
    "notifications_sent": true
  }
}

Rollout Strategies [DRAFT / NOT IMPLEMENTED]

No strategy executor exists. The only strategy enum in source is PromotionBindingMode (Manual, AutomaticOnApproval, Scheduled, Canary) on a single per-environment binding — there is no blue-green, canary-traffic-split, or feature-flag engine. The YAML blocks below are design sketches only.

Blue-Green

strategy: blue_green
config:
  parallel_evaluation: true  # Both versions evaluated
  comparison_period: 24h
  switch_threshold:
    verdict_match_rate: 0.95

Canary with Traffic Split

strategy: canary_traffic
config:
  initial_percentage: 5
  increment: 10
  increment_interval: 4h
  max_error_rate: 0.01

Feature Flag

strategy: feature_flag
config:
  flag_name: "policy-v2-enabled"
  default: false
  overrides:
    - tenant: "security-team"
      value: true

Data Contracts [DRAFT / NOT IMPLEMENTED]

These TypeScript interfaces do not correspond to any emitted contract. The real, implemented contracts are the C# records in StellaOps.Policy.Registry/Services/ (PromotionBinding, PromoteRequest, RollbackRequest, PromotionBindingRules, PublishResult, ReviewRequest, etc.) and the Policy Engine PolicyPackDto / PolicyRevisionDto / PolicyRevisionActivationResponse records in Endpoints/PolicyPackEndpoints.cs. See Implemented Lifecycle.

Rollout Plan Schema (draft)

interface RolloutPlan {
  rollout_id: string;
  policy_set: string;
  from_version: string;
  to_version: string;
  strategy: 'staged' | 'blue_green' | 'canary_traffic' | 'feature_flag';
  stages: Stage[];
  rollback: {
    automatic: boolean;
    triggers: RollbackTrigger[];
  };
  notifications: NotificationConfig[];
}

interface Stage {
  name: string;
  description?: string;
  tenants?: string[];
  selection?: TenantSelection;
  duration?: string;  // ISO-8601 duration
  success_criteria?: SuccessCriteria;
  auto_proceed?: boolean;
}

Rollout Status Schema (draft)

interface RolloutStatus {
  rollout_id: string;
  status: 'pending' | 'in_progress' | 'paused' | 'completed' | 'rolled_back' | 'failed';
  current_stage?: string;
  stages: Array<{
    name: string;
    status: 'pending' | 'active' | 'monitoring' | 'completed' | 'failed';
    started_at?: string;
    completed_at?: string;
    metrics?: StageMetrics;
  }>;
  tenant_status: Array<{
    tenant_id: string;
    policy_version: string;
    activated_at?: string;
  }>;
}

Policy Inheritance [DRAFT / NOT IMPLEMENTED]

No policy inheritance / inherits_from resolution exists in src/. There is no org/team/project tenant tree and no cascading override resolver. The Risk Profile model does have a single-level Extends field (profile.Extends in RiskProfileModel) that one profile can use to extend another profile by id — but that is per-profile inheritance, not the tenant-hierarchy cascade depicted here. Treat the diagram below as a roadmap sketch.

Organization Policy (base)
    └── inherits_from: stellaops-default

Team Policy (override)
    └── inherits_from: organization
    └── overrides: [severity-thresholds]

Project Policy (final)
    └── inherits_from: team
    └── overrides: [specific-exceptions]

Resolution order (target): Project → Team → Organization → Platform Default

Error Handling [DRAFT / NOT IMPLEMENTED]

Stage-level recovery requires the rollout engine, which does not exist. The implemented error paths are per-operation: revision activation rejects non-Approved revisions (400 NotApproved), duplicate approvals (400 DuplicateApproval), and missing pack/revision (404); the Registry RollbackAsync returns RollbackResult.Success = false with an Error when a prior binding cannot be restored.

Observability

Metrics

[DRAFT / NOT IMPLEMENTED] — the policy_rollout_* and policy_version_active metrics below are not emitted by any service.

[IMPLEMENTED] — the metrics actually emitted today come from the StellaOps.Policy.Registry library (meter StellaOps.Policy.Registry, see Telemetry/PolicyRegistryMetrics.cs):

Key Log Events

[DRAFT / NOT IMPLEMENTED] — the rollout.* events below are not emitted; no rollout engine exists.

[IMPLEMENTED] — the notification/audit events that exist today:

Related Flows

• Policy Evaluation Flow - Policy application
• Exception Approval Workflow - Exception handling
• Notification Flow - Rollout alerts