DCM User Flows

    graph TB
    User([User])
    Admin[Admin]
    CM[Catalog Manager]
    CDB[(Catalog DB)]
    PM[Placement Manager]
    POL[Policy Manager / OPA]
    SPRM[SP Resource Manager]
    AR[(Agent Registry)]
    DB[(Placement DB)]
    MS[Messaging System / NATS]

    AG1[Agent - Environment 1]
    AG2[Agent - Environment 2]
    SP1[KubeVirt SP]
    SP2[K8s Container SP]
    SP3[ACM Cluster SP]

    User --> CM
    Admin --> CM
    Admin --> POL
    CM --> CDB
    CM --> PM
    PM --> POL
    PM --> SPRM
    PM --> DB
    SPRM --> AR
    SPRM -->|publish requests| MS
    MS -->|deliver requests| AG1
    MS -->|deliver requests| AG2
    AG1 --> SP1
    AG1 --> SP2
    AG2 --> SP3
    AG1 -.->|registration & heartbeat| DCM_API
    AG2 -.->|registration & heartbeat| DCM_API
    DCM_API[DCM API]
    DCM_API --> AR

    SP1 -->|status events| MS
    SP2 -->|status events| MS
    SP3 -->|status events| MS
    MS -->|status updates| SPRM
    SPRM -->|status updates| CM
  

    sequenceDiagram
    actor Admin
    participant PM as Policy Manager
    participant DB as Policy DB
    participant OPA as OPA Engine

    Admin->>PM: POST /api/v1/policies<br/>{name, type, priority, label_selector, rego_code}
    PM->>PM: Validate name & priority uniqueness (at policy type level)
    alt Validation fails
        PM-->>Admin: 400 Bad Request (duplicate name or priority)
    end
    PM->>PM: Generate UUID, parse PackageName from REGO
    PM->>DB: Store policy metadata<br/>(UUID, name, package_name, label_selector, type, priority)
    PM->>OPA: Push REGO code (keyed by UUID)
    alt Compilation fails
        OPA-->>PM: Compilation error
        PM->>DB: Rollback policy record
        PM-->>Admin: 400 Bad Request (REGO compilation error)
    end
    OPA-->>PM: Compilation success
    PM-->>Admin: 201 Created {policy_id: UUID}
  

    sequenceDiagram
    participant PM as Placement Manager
    participant PE as Policy Manager
    participant DB as Policy DB
    participant OPA as OPA Engine

    PM->>PE: POST /api/v1alpha1/policies:evaluateRequest<br/>{service_instance: {spec}, available_agents}

    PE->>DB: Fetch enabled policies matching request via label selector
    PE->>PE: Sort by Level (Global→Tenant→User), then Priority (asc)

    loop For each policy in sorted order
        PE->>OPA: Evaluate policy with:<br/>{spec, agent, constraints, agent_constraints}
        OPA-->>PE: {rejected, patch, constraints,<br/>selected_agent, agent_constraints}

        alt rejected == true
            PE-->>PM: 406 Not Acceptable (rejection_reason)
        end

        PE->>PE: Validate constraints<br/>(lower-level cannot unlock higher-level locks)
        alt Constraint conflict
            PE-->>PM: 409 Conflict (policy conflict error)
        end

        PE->>PE: Merge constraints into ConstraintContext
        PE->>PE: Merge agent_constraints
        PE->>PE: Validate & apply patches against constraints
        PE->>PE: Validate selected_agent against agent constraints
    end

    PE-->>PM: 200 OK {evaluated_service_instance, selected_agent, status}
  

    graph LR
    subgraph CommonFields[Common Fields]
        ST[service_type: string]
        MD[metadata: name, labels]
        PH[provider_hints: provider-specific config]
    end

    subgraph ServiceTypeSchemas[ServiceType Schemas]
        VM[VM Schema]
        CT[Container Schema]
        DBS[Database Schema]
        CL[Cluster Schema]
        STG[Storage Schema]
    end

    CommonFields --> VM
    CommonFields --> CT
    CommonFields --> DBS
    CommonFields --> CL
    CommonFields --> STG
  

    sequenceDiagram
    actor Admin
    participant CM as Catalog Manager

    Admin->>CM: POST /api/v1/catalog-items
    Note right of CM: Payload includes:<br/>- service_type reference<br/>- field definitions with:<br/>  - path (e.g., "resources.cpu")<br/>  - default value<br/>  - editable flag<br/>  - validation_schema

    CM->>CM: Validate CatalogItem schema
    CM-->>Admin: 201 Created {catalog_item_id}
  

    sequenceDiagram
    actor User
    participant UI as UI / CLI
    participant CM as Catalog Manager
    participant PM as Placement Manager

    User->>UI: Select CatalogItem "production-postgres"
    UI->>UI: Render form with editable fields,<br/>defaults, and validation rules
    User->>UI: Customize editable fields<br/>(e.g., version="16", cpu=8)
    UI->>UI: Client-side validation against validation_schema
    User->>CM: POST /api/v1/catalog-item-instances<br/>{catalog_item_id, user_values}
    CM->>CM: Validate input against validation_schema
    CM->>CM: Merge defaults + user input → ServiceType payload
    Note right of CM: Result:<br/>{service_type: "database",<br/> engine: "postgresql",<br/> version: "16",<br/> resources: {cpu: 8, memory: "16GB", storage: "100GB"}}
    CM->>PM: POST /api/v1/resources<br/>{CatalogItemInstance, spec}
    PM-->>CM: 202 Accepted
    CM-->>User: Instance created (provisioning)
  

    sequenceDiagram
    participant SP as Service Provider
    participant AG as Agent
    participant DCM as DCM Control Plane
    participant DB as Database

    Note over AG: Embedded SPs registered<br/>internally at startup

    SP->>AG: POST /api/v1/providers<br/>{name, display_name, endpoint, service_type, metadata}

    alt Service type already served by another SP
        AG-->>SP: 409 Conflict<br/>{error: "service type X already served by provider Y"}
    else Name does not exist
        AG->>AG: Create new SP entry, generate provider_id
        AG-->>SP: 201 Created {id, name, status: "registered"}
    else Name exists, same provider_id
        AG->>AG: Update existing entry
        AG-->>SP: 200 OK {id, name, status: "registered"}
    else Name exists, different provider_id
        AG-->>SP: 409 Conflict
    end

    alt Service type list changed AND agent registered to DCM
        AG->>DCM: POST /api/v1/agents<br/>{name, environment, service_types, cost, topic_name}
        DCM->>DB: Update agent registration
        DCM-->>AG: 200 OK
    end

    Note over SP,AG: SP periodically re-registers<br/>to maintain lease
  

    sequenceDiagram
    autonumber
    participant AG as Agent
    participant MS as Messaging System
    participant DCM as DCM Control Plane
    participant DB as Database

    AG->>MS: Create topics (main + retry)
    Note over AG: Wait for at least 1 SP<br/>(embedded or external) to register<br/>and be healthy

    AG->>DCM: POST /api/v1/agents<br/>{name, environment, service_types,<br/>resources_available, cost, topic_name}
    DCM->>DB: Store agent registration
    DCM-->>AG: 201 Created {agent_id}
  

    stateDiagram-v2
    [*] --> Ready: Registered with Agent

    Ready --> FailureCount: Failed
    FailureCount --> Ready: OK (reset)
    FailureCount --> Unavailable: Threshold reached

    Ready --> Unhealthy: status: unhealthy
    Unhealthy --> Ready: status: healthy
    Unhealthy --> Unavailable: Timeout/error threshold

    Unavailable --> Ready: OK (recover)
  

    sequenceDiagram
    participant AG as Agent
    participant SP as External SP

    Note over AG: Embedded SPs: health<br/>checked in-process<br/>(no network call)

    loop Every {health_check_interval} seconds (external SPs only)
        AG->>SP: GET /health
        alt 200 OK, status: healthy
            SP-->>AG: {status: "healthy"}
            AG->>AG: Reset failure counter, mark Ready
        else 200 OK, status: unhealthy
            SP-->>AG: {status: "unhealthy"}
            AG->>AG: Mark Unhealthy<br/>Stop routing, hold requests
        else Timeout or error
            SP-->>AG: Error / Timeout
            AG->>AG: Increment failure counter
            alt Failures >= threshold
                AG->>AG: Mark Unavailable<br/>Remove service type, update DCM
            end
        end
    end
  

    sequenceDiagram
    participant AG as Agent
    participant DCM as DCM Control Plane

    loop Every {heartbeat_interval} seconds
        AG->>DCM: PUT /api/v1/agents/{agent_id}/heartbeat<br/>{timestamp, consumer_lag}
        DCM->>DCM: Update heartbeat, check lag
        DCM-->>AG: 200 OK
    end

    Note over DCM: No heartbeat within threshold
    DCM->>DCM: Mark agent Unavailable
  

    stateDiagram-v2
    [*] --> Ready: Agent registers
    Ready --> Congested: lag >= threshold
    Congested --> Ready: lag < threshold
    Ready --> Unavailable: Heartbeat timeout
    Congested --> Unavailable: Heartbeat timeout
    Unavailable --> Ready: Agent re-registers
  

    sequenceDiagram
    participant Platform as Underlying Platform<br/>(K8s, KubeVirt, ACM)
    participant SP as Service Provider
    participant MS as Messaging System (NATS)
    participant DCM as DCM Core Service
    participant DB as Status DB

    Platform->>SP: State change event<br/>(via informer watch or polling)
    SP->>SP: Map platform status → DCM status
    SP->>SP: Build CloudEvent
    SP->>MS: Publish to NATS subject<br/>dcm.{service_type}<br/>

    MS->>DCM: Deliver event
    DCM->>DCM: Validate CloudEvent schema
    alt Valid
        DCM->>DB: UPSERT instance status
    else Invalid
        DCM->>DCM: Log error, discard
    end
  

    sequenceDiagram
    actor User
    participant CM as Catalog Manager
    participant PM as Placement Manager
    participant DB as Placement DB
    participant PE as Policy Manager
    participant SPRM as SP Resource Manager
    participant AR as Agent Registry
    participant MS as Messaging System
    participant AG as Agent
    participant SP as Service Provider

    User->>CM: Request CatalogItemInstance<br/>(select CatalogItem + customize fields)
    CM->>CM: Validate input, merge with defaults
    CM->>PM: POST /api/v1/resources<br/>{CatalogItemInstance: UUID, spec}

    PM->>DB: Store original request (intent)

    PM->>AR: Fetch available agents<br/>(healthy, not Congested, matching service_type)
    AR-->>PM: available_agents list

    PM->>PE: POST /api/v1alpha1/policies:evaluateRequest<br/>{service_instance: {spec}, available_agents}
    PE->>PE: Evaluate policy chain<br/>(validate, mutate, select Agent)

    alt Policy rejects
        PE-->>PM: 406 Not Acceptable
        PM->>DB: Delete intent record
        PM-->>CM: Error (policy rejected)
        CM-->>User: Request denied
    end

    PE-->>PM: 200 OK<br/>{evaluated_service_instance, selected_agent, status}
    PM->>DB: Store validated request with agent_name

    PM->>SPRM: POST /api/v1/service-type-instances<br/>{agent_name, service_type, spec}

    SPRM->>AR: Lookup agent, get topic_name
    alt Agent not found or unhealthy/Congested
        SPRM-->>PM: Error (404/503)
        PM->>DB: Delete records
        PM-->>CM: Error
        CM-->>User: Agent unavailable
    end

    SPRM->>MS: PUBLISH CloudEvent<br/>topic: {topic_name}<br/>{resource_id, service_type, spec}
    SPRM->>DB: Create instance record
    SPRM-->>PM: 202 Accepted {instance_id, agent_name, status: PENDING}
    PM-->>CM: 201 Created
    CM-->>User: Instance created (PENDING)

    Note over MS,AG: Async processing
    MS->>AG: Deliver creation request
    AG->>AG: Validate service type, select SP
    AG->>SP: POST {sp_endpoint}/api/v1/{service_type}<br/>{spec}
    SP-->>AG: {instance_id, status: PROVISIONING}
    AG->>MS: PUBLISH CloudEvent<br/>topic: dcm.agents.responses<br/>{resource_id, agent_name, topic_name,<br/>status: PROVISIONING}
    MS->>SPRM: Deliver response
    SPRM->>DB: Update instance: PROVISIONING

    opt Agent queues request (SP Unhealthy)
        AG->>MS: PUBLISH CloudEvent<br/>topic: dcm.agents.responses<br/>{resource_id, status: QUEUED}
        MS->>SPRM: Deliver QUEUED response
        SPRM->>SPRM: Update instance: QUEUED
        SPRM->>PM: Notify: instance QUEUED

        Note over PM: Start queued_request_timeout
        alt Timeout or timeout = 0
            PM->>SPRM: DELETE instance
            PM->>PE: Re-evaluate excluding agent
            Note over PM: Route to alternative agent<br/>or return error if none available
        end
    end

    Note over SP,MS: Status reporting (unchanged)
    SP->>MS: Publish status CloudEvents
    MS->>SPRM: Deliver status updates
  

    flowchart TD
    A[Receive request from Catalog Manager] --> B[Store original request in Placement DB]
    B --> C[Fetch available agents from Agent Registry]
    C --> D[Send to Policy Manager for evaluation<br/>with available_agents]
    D --> E{Policy approved?}
    E -->|No| F[Delete intent record]
    F --> G[Return error to Catalog Manager]
    E -->|Yes| H[Store validated request with agent_name]
    H --> I[Forward to SP Resource Manager<br/>with agent_name, service_type, spec]
    I --> J{SPRM response?}
    J -->|Error| K[Delete records from Placement DB]
    K --> G
    J -->|202 Accepted| L[Return 201 Created<br/>to Catalog Manager]
    J -->|QUEUED| M[Start queued_request_timeout timer]
    M --> N{Timeout?}
    N -->|Yes| O[Send DELETE to SPRM<br/>Re-evaluate excluding agent]
    O --> P{Alternative agent?}
    P -->|Yes| I
    P -->|No| K
  

    flowchart TD
    A[Receive request from Placement Manager<br/>agent_name + service_type + spec] --> B[Query Agent Registry<br/>by agent_name]
    B --> C{Agent found?}
    C -->|No| D[Return 404 Not Found]
    C -->|Yes| E{Agent healthy<br/>and not Congested?}
    E -->|No| F[Return 503 Service Unavailable]
    E -->|Yes| G[Publish CloudEvent to agent topic<br/>via Messaging System]
    G --> H[Create instance record in DB]
    H --> I[Return 202 Accepted<br/>instance_id, agent_name, status: PENDING]
  

    flowchart LR
    subgraph KubeVirtSP[KubeVirt SP]
        A1[Receive VM spec] --> A2[Create VirtualMachine CR]
        A2 --> A3[Return instance_id - PROVISIONING]
    end

    subgraph K8sContainerSP[K8s Container SP]
        B1[Receive Container spec] --> B2[Create Deployment and Service]
        B2 --> B3[Return request_id - PENDING]
    end

    subgraph ACMClusterSP[ACM Cluster SP]
        C1[Receive Cluster spec] --> C2[Create HostedCluster and NodePool]
        C2 --> C3[Return request_id - PENDING]
    end

    subgraph K8sStorageSP[K8s Storage SP]
        D1[Receive Storage spec] --> D2[Create PersistentVolumeClaim]
        D2 --> D3[Return request_id - PROVISIONING]
    end
  

    flowchart TD
    subgraph Service Provider
        A[Platform event detected<br/>via Informer watch or polling]
        A --> B[Map platform status<br/>to DCM status enum]
        B --> C[Build CloudEvent v1.0]
        C --> D["Publish to NATS<br/>dcm.{service_type}"]
    end

    subgraph DCM Core
        D --> E[Receive CloudEvent]
        E --> F{Valid schema?}
        F -->|Yes| G[UPSERT instance status in DB]
        F -->|No| H[Log error, discard]
    end

    subgraph Monitoring Approaches
        I[Event-Driven Streaming<br/>Preferred - K8s Informers]
        J[Polling<br/>Fallback - Legacy APIs]
    end
  

    graph LR
    subgraph KubeVirt VMI Phase
        VP1[Pending/Scheduling/Scheduled]
        VP2[Running]
        VP3[Succeeded]
        VP4[Failed/Unknown]
        VP5[Not Found]
    end

    subgraph DCM VM Status
        DS1[PROVISIONING]
        DS2[RUNNING]
        DS3[STOPPING]
        DS4[FAILED]
        DS5[DELETED]
    end

    VP1 --> DS1
    VP2 --> DS2
    VP3 --> DS3
    VP4 --> DS4
    VP5 --> DS5
  

    graph LR
    subgraph K8s Pod Phase
        KP1[Pending/ContainerCreating]
        KP2[Running]
        KP3[Succeeded]
        KP4[Failed/CrashLoopBackOff]
        KP5[Unknown - node lost]
    end

    subgraph DCM Container Status
        CS1[PENDING]
        CS2[RUNNING]
        CS3[SUCCEEDED]
        CS4[FAILED]
        CS5[UNKNOWN]
    end

    KP1 --> CS1
    KP2 --> CS2
    KP3 --> CS3
    KP4 --> CS4
    KP5 --> CS5
  

    graph LR
    subgraph HostedCluster Conditions
        HC1[Progressing=Unknown]
        HC2[Progressing=True, Available=False]
        HC3[Available=True, Progressing=False]
        HC4[Degraded=True]
        HC5[Not Found]
    end

    subgraph DCM Cluster Status
        DC1[PENDING]
        DC2[PROVISIONING]
        DC3[READY]
        DC4[FAILED]
        DC5[DELETED]
    end

    HC1 --> DC1
    HC2 --> DC2
    HC3 --> DC3
    HC4 --> DC4
    HC5 --> DC5
  

    graph LR
    subgraph K8s PVC Phase
        PVC1[Pending]
        PVC2[Bound - resizing]
        PVC3[Bound]
        PVC4[Lost]
        PVC5[deletionTimestamp set]
        PVC6[Not Found]
    end

    subgraph DCM Storage Status
        SS1[PROVISIONING]
        SS2[PROVISIONING]
        SS3[RUNNING]
        SS4[FAILED]
        SS5[DELETING]
        SS6[DELETED]
    end

    PVC1 --> SS1
    PVC2 --> SS2
    PVC3 --> SS3
    PVC4 --> SS4
    PVC5 --> SS5
    PVC6 --> SS6
  

    sequenceDiagram
    actor User
    participant CM as Catalog Manager
    participant PM as Placement Manager
    participant DB as Placement DB
    participant SPRM as SP Resource Manager
    participant MS as Messaging System
    participant AG as Agent
    participant SP as Service Provider

    User->>CM: Delete CatalogItemInstance
    CM->>PM: DELETE /api/v1/resources/{resource_id}
    PM->>DB: Lookup resource (agent_name, service_type, instance_id)

    PM->>SPRM: DELETE /api/v1/service-type-instances/{instance_id}
    SPRM->>MS: PUBLISH CloudEvent<br/>topic: {topic_name}<br/>type: dcm.request.delete<br/>{resource_id, service_type}
    SPRM-->>PM: 202 Accepted

    MS->>AG: Deliver deletion request
    AG->>SP: DELETE {sp_endpoint}/api/v1/{service_type}/{resource_id}
    SP-->>AG: {status: DELETING}
    AG->>MS: PUBLISH CloudEvent<br/>topic: dcm.agents.responses<br/>{resource_id, agent_name, topic_name,<br/>status: DELETING}

    opt Agent queues request (SP Unhealthy)
        AG->>MS: PUBLISH CloudEvent<br/>topic: dcm.agents.responses<br/>{resource_id, status: QUEUED}
        MS->>SPRM: Deliver QUEUED response
        SPRM->>SPRM: Update instance: QUEUED
        SPRM->>PM: Notify: deletion QUEUED

        Note over PM: Resource stays DELETING.<br/>Deletion cannot be re-routed.<br/>Agent holds the request in its<br/>retry topic for automatic resolution.

        alt SP recovers — Agent processes held deletion
            AG->>SP: DELETE {sp_endpoint}/api/v1/{service_type}/{resource_id}
            SP-->>AG: {status: DELETING}
            AG->>MS: PUBLISH CloudEvent<br/>{resource_id, status: DELETING}
        else SP becomes Unavailable — Agent rejects
            AG->>MS: PUBLISH CloudEvent<br/>{resource_id, error: "SP unavailable"}
            MS->>SPRM: Deliver error
            Note over SPRM: Enqueue in cleanup queue<br/>for deferred retry.<br/>Resource stays DELETING.
        end
    end

    Note over SP: SP manages deletion<br/>and reports final status
    SP->>MS: CloudEvent {status: DELETED}
    MS->>SPRM: Status update
    SPRM->>DB: Update status: DELETED