Known Limitations & Rough Edges
An honest engineering look at scale constraints and pipeline vulnerabilities.
No system is perfect. Here are the known trade-offs and limits currently present in the codebase.
1. In-Memory Sliding Rate Limiting
The rate limiter implemented in lib/orchestrator/rateLimiter.ts uses a simple in-memory JavaScript Map. While this works perfectly for single-instance applications, it does not share state across multiple server instances (e.g. Vercel Serverless or multi-node Docker deployments). Under high loads or autoscaling groups, users can bypass limits. Upgrading this to Redis is a high priority.
2. Evaluator History Truncation
To prevent hitting strict context boundaries on the evaluator (which receives fanned responses from multiple workers plus the conversation history), we strip empty lines and limit historical context to standard turns. If worker answers are excessively verbose, the evaluator request can suffer from latency bloat or truncation issues.
3. Database Connection pool bounds
Since serverless API endpoints open connection pools on every fanned pipeline call, database connection spikes can occur under heavy concurrent user requests if the pool bounds are not throttled at the database level.