Orchestration Pipeline
Deconstructing fanned concurrency, stream delta propagation, and evaluator synthesis steps.
The core engine of Arbiter is lib/orchestrator/runPipeline.ts. It handles the synchronization of concurrent stream outputs, intercepts stream metadata, and pipes results directly into the evaluation step.
Fanned Execution Phase
When a prompt arrives, the orchestrator maps all selected workers and fires their stream processes concurrently using Promise.all. As chunks arrive from each vendor API, they are tagged with their provider code and pushed to the HTTP write buffer as SSE events.
The server tracks metrics like Time-to-First-Token (TTFT) by catching the timestamp of the first chunk delta. When a worker stream concludes, its usage statistics are saved to a temporary tracker.
Evaluator Synthesis Phase
Once all fanned worker streams complete successfully, the orchestrator checks if an evaluator is selected. If active, it resolves the evaluator adapter and feeds:
- The original user query
- Historical chat messages
- All worker responses (labeled with model IDs and completion statuses)
The evaluator streams a consolidated response back to the user, identifying points of agreement or contradiction. If no evaluator is selected, the client falls back to displaying the first successful worker response.
Interactive SSE Stream Simulator
Test the SSE event emission sequences and cost accumulators below.
SSE Event Stream Orchestrator Simulator
Visualize server-to-client chunk flows and Drizzle transaction commits.