FLAGSHIP · APPLIED AI ENGINEERING
When is more agent complexity actually worth it?
Agentic Analytics Lab is an implemented analytics-agent baseline and a test bed for architecture decisions.
Problem
Analytics agents can return plausible answers without respecting metric definitions or the data boundary.
Engineering question
When does additional agent complexity improve correctness enough to justify its cost?
What I built
A Python analytics-agent baseline using synthetic operational data, dataset-scoped read-only ClickHouse tooling, semantic metric guards, evaluation infrastructure, three deliberately separate architecture lanes, automated regression coverage, CI, and CodeQL.
Architecture
The project includes a measured single-agent baseline plus an implemented oracle_metadata_routed_v0 experiment. The routed experiment uses frozen benchmark metadata to select deterministic handlers for structured cases while preserving the existing local model path where model reasoning is required. This isolates the value of specialized execution when task type is already known. It does not evaluate natural-language route inference.
Evaluation
The recorded October 5, 2026 CI run for main commit 58add5e passed 289 tests on both Python 3.11 and 3.12. The fixed benchmark separates execution, correctness, task success, tool grounding, evidence quality, route selection, and worker execution.
Guardrails and boundaries
Tools are read-only and dataset-scoped. Semantic guards protect metric meaning. The data is synthetic.
Failure evidence
The accepted frozen single-agent comparison baseline produced repeatable failure evidence rather than a single benchmark score. Q1, Q2, Q4, and Q6 passed consistently across three unchanged runs, while Q3 and Q5 failed consistently. Those stable failures became the basis for a controlled execution-layer experiment rather than an assumption that more agent complexity would automatically help.
Tradeoffs
Under oracle task metadata, specialized deterministic execution corrected the stable Q3 and Q5 baseline failures and substantially reduced model work in the reviewed experiment. That result has an important boundary: route selection was supplied by frozen benchmark metadata. The experiment therefore demonstrates the value of different execution strategies for known task categories, not the reliability of a deployable natural-language router. The routed design also adds control-plane and handler complexity, so the architectural question remains whether those benefits survive realistic route inference.
Current limitations
The dataset is synthetic, and the project is an evaluation lab rather than a production analytics service. The routed experiment uses oracle benchmark metadata and therefore does not establish natural-language route inference, a general multi-agent advantage, production performance, or real organizational outcomes. The natural-language development set is a regression fixture set, not held-out evidence of generalization.
Current research status
As of October 28, later benchmark and evidence-capture engineering includes inactive preparation under review. Live capture qualification remains deferred, evaluator acceptance is open, the statistical method is not frozen, and final evaluation has not begun. Engineering checks do not establish new scientific results or acceptance of those gates.
What comes next
Qualify the operational evidence and accept the evaluator, common-runtime comparison, and statistical protocol before making a new comparison claim. A runnable strong single-agent implementation exists; its existence alone does not establish an accepted comparison anchor. Held-out route-inference evaluation remains separate from the development fixtures.