Skip to content

Slice 002 - Discovery Runs (MVP)

Goal

Run discovery via Discovery Connectors and store results summaries and normalized facts.

Scope

  • Run now
  • Run status lifecycle (queued/running/succeeded/partial/failed)
  • Run history list
  • Error mapping to actionable remediation
  • Output normalized entities + relationships + provenance
  • Accept streaming discovery outputs in batches or full-result payloads
  • Queue-backed execution with isolated workers

Acceptance Criteria

  • A run produces a run summary and facts compliant with canonical model contract
  • Partial runs show partial results and list failures
  • Audit events exist for run start/completion
  • Streaming connectors can emit batches and the service persists incrementally
  • Runs execute via queue-backed workers with per-run isolation

Tenancy & Isolation Requirements

  • Each run is scoped to one tenant_id + workspace_id and one Connection.
  • Queue payloads and worker execution context must carry tenant/workspace identifiers.
  • Worker fetches of configuration/scope must enforce caller and target scope match.
  • Run history and facts queries must enforce tenant/workspace visibility.
  • Cross-tenant run access is out of scope for v0 product flows.

Depends on

  • 04-contracts/canonical-model.md
  • 04-contracts/connector-spec.md
  • ADR 0001
  • ADR 0004 (Discovery Execution Architecture)

Definition of Done

  • Run history visible per connection
  • Provenance present on all outputs
  • Runs can scale horizontally based on queue depth

Execution Architecture (AWS)

Discovery runs are coordinated by the Discovery service (control plane) and executed by isolated workers (data plane).

See ADR 0004 for the accepted architecture.

flowchart LR
  client["Client"] --> api["Discovery Service (Control Plane)"]
  api --> db["Postgres (Runs + Facts)"]
  api --> queue["SQS Run Queue"]
  queue --> worker["Discovery Worker (Fargate)"]
  worker --> integ["Integrations Service (Internal)"]
  worker --> db
  worker --> audit["Audit Service (Outbox)"]

Control plane: - Receives run requests and writes discovery_runs. - Enqueues a run job to SQS (no secrets in the message).

Worker plane: - Fargate workers poll SQS and execute one run at a time. - Workers fetch connection configuration + scope from the Integrations service via an internal, token-protected endpoint. - Workers extend SQS visibility while a run is active to avoid duplicate execution.

Security notes: - Secrets never transit SQS. - Workers only receive tenant/workspace scoped inputs from the Integrations service. - Separate task role for workers limits permissions vs. API services.

Scaling notes: - Worker service auto-scales based on SQS queue depth.