ADR 0004 - Discovery Execution Architecture (Queue + Workers)

Date: 2026-02-07 Status: Accepted

Context

Discovery runs are long-running and resource intensive. The system must: - Maintain strong tenant/workspace isolation. - Scale horizontally under bursty load. - Avoid blocking API request threads for discovery execution. - Preserve auditability and provenance.

Decision

Adopt a control-plane + worker-plane architecture: - The Discovery service enqueues run jobs to a queue. - A worker service (Fargate) polls the queue and executes one run per task. - Workers fetch connection configuration from Integrations via a token-protected internal endpoint. - Queue messages do not contain secrets. - Workers extend queue visibility while a run is active to prevent duplicate execution.

Consequences

• Improved isolation posture: execution happens outside the API service.
• Horizontal scaling based on queue depth.
• Requires queue infrastructure and worker service deployment.
• Introduces internal service authentication for fetching connection configuration.

Alternatives considered

• In-process execution in Discovery service (rejected due to isolation and scalability concerns).
• Direct RunTask invocation for each run (deferred; higher orchestration complexity).