FLAGSHIP · APPLIED AI ENGINEERING
When is more agent complexity actually worth it?
Agentic Analytics Lab is an implemented analytics-agent baseline and a test bed for architecture decisions.
Problem
Analytics agents can return plausible answers without respecting metric definitions or the data boundary.
Engineering question
When does additional agent complexity improve correctness enough to justify its cost?
What I built
A Python analytics-agent baseline using synthetic operational data, dataset-scoped read-only ClickHouse tooling, semantic metric guards, evaluation infrastructure, automated regression coverage, CI, and CodeQL.
Architecture
The project includes a measured single-agent baseline plus an implemented oracle_metadata_routed_v0 experiment. The routed experiment uses frozen benchmark metadata to select deterministic handlers for structured cases while preserving the existing local model path where model reasoning is required. This isolates the value of specialized execution when task type is already known. It does not evaluate natural-language route inference.
Evaluation
The baseline is tested on a fixed question set. Execution, correctness, task success, tool grounding, and evidence quality are separate measures.
Guardrails and boundaries
Tools are read-only and dataset-scoped. Semantic guards protect metric meaning. The data is synthetic.
Failure evidence
The accepted frozen single-agent comparison baseline produced repeatable failure evidence rather than a single benchmark score. Q1, Q2, Q4, and Q6 passed consistently across three unchanged runs, while Q3 and Q5 failed consistently. Those stable failures became the basis for a controlled execution-layer experiment rather than an assumption that more agent complexity would automatically help.
Tradeoffs
Under oracle task metadata, specialized deterministic execution corrected the stable Q3 and Q5 baseline failures and substantially reduced model work in the reviewed experiment. That result has an important boundary: route selection was supplied by frozen benchmark metadata. The experiment therefore demonstrates the value of different execution strategies for known task categories, not the reliability of a deployable natural-language router. The routed design also adds control-plane and handler complexity, so the architectural question remains whether those benefits survive realistic route inference.
Current limitations
The dataset is synthetic, and the project is an evaluation lab rather than a production analytics service. The routed experiment uses oracle benchmark metadata and therefore does not establish natural-language route inference, a general multi-agent advantage, production performance, or real organizational outcomes. A fresh three-run single-agent benchmark on current main is still needed before treating historical benchmark percentages as current-system performance.
What comes next
Re-run the frozen single-agent benchmark on current main and use the resulting artifact provenance as the next canonical evidence point. Only then decide whether a natural-language routing experiment is justified by a concrete remaining failure pattern.