How a Small Fine-Tuned Model Beats the Frontier

Results

In one of our pilot engagements, we trained a custom retrieval agent to replace an internal agent running on a frontier model, using a small open model for that one task. The result is 80.5% Pass@1 on held-out questions the model never saw in training, against 75.5% for Opus 4.8 on the same evaluation, at roughly 25x lower cost per output token.

Cost at published API pricing. Opus 4.8 and Haiku 4.5 tokens/sec from Artificial Analysis. Qwen figures measured single-stream on our serving stack; base and finetuned share the same architecture and run at the same speed.

The base model starts at 67.1% and training adds 13.4 points. That puts a model with roughly 3B active parameters above every frontier baseline on this task. In our comparison, the best performing model is not the largest model trained on the most data, it is the one purpose trained for that job. Cost per output token is roughly $1 versus $25 per million for a model that is also more accurate and generates roughly three times as fast.

The rest of this post covers the task, the training setup, the environment design that makes the task learnable in the first place, and the ablations that show why it works. You can read the broader argument for why we train open-source models for vertical agent solutions like this one in our manifesto.

The task

The system is an internal retrieval tool built for high throughput. It answers operational questions across a real business, covering staffing and scheduling, daily operations, financial analysis, SOP and policy lookup, inventory, HR, and support tickets. The underlying data lives in Notion, Airtable, HubSpot, and BigQuery, and most questions require reasoning across systems rather than a single lookup. Think "which locations are understaffed next week relative to their forecasted volume", not "what's the wifi password".

This agent lives inside a larger organization, where hundreds of queries hit it every hour across multiple systems. A frontier model reaches the accuracy, but at that volume its per-token cost does not hold. This is the regime where task-specific training pays for itself. High volume, well-defined scope, verifiable answers.

The setup

The base model is Qwen3.6-35B-a3b, a sparse MoE with roughly 3B active parameters per token. Two architectural details matter here. The sparsity means we get 35B-class representations at 3B-class inference cost, which reduces latency. And the hybrid attention (most layers use Gated DeltaNet-style linear attention rather than full softmax attention, see Yang et al., 2024) keeps long agentic trajectories cheap, since cost doesn't grow quadratically as the context fills up with tool results.

Training data is 850 single-hop and multi-hop questions over the business. We optimize with a variant of GSPO (Zheng et al., 2025), the sequence-level cousin of GRPO (Shao et al., 2024), further adapted to align not just the final answer but also the amount and quality of the model's thinking process. The sequence-level importance ratios matter for us specifically because the base model is a MoE. Token-level ratios interact badly with expert routing, which shifts between rollout and gradient computation in asynchronous disaggregated training, and GSPO's formulation sidesteps that instability. The reward follows the standard post-R1 recipe of RL with verifiable rewards (DeepSeek-AI, 2025), a rubric scored by an LLM judge (Zheng et al., 2023) and decomposed into groundedness and correctness of the final answer, so a fluent-sounding answer only earns reward if it is actually backed by what the model retrieved.

Evaluation is 150 held-out samples, Pass@1 reported as the mean of 4 rollouts per sample. All models, ours and the frontier baselines, run through the same harness, the same tools, and the same judge.

An environment built for production, with RL in mind

RL puts demands on an environment that production systems cannot meet. Training runs thousands of concurrent rollouts, and every step of every rollout calls tools; pointed at live APIs, that load trips rate limits far below what RL requires, runs up per-call charges, and lets network latency dominate wall-clock time. The data also has to be frozen, since a task's ground-truth answer must not drift while the policy trains on it, or the reward signal turns to noise. And the agent has to be free to explore without mutating live business data. All three requirements point to the same design. The agent trains against an isolated environment built on a frozen snapshot of the real systems, not against the systems themselves.

A context caching layer sits on top of that snapshot, and because it's local, tool calls return in milliseconds instead of API round-trips, which is what makes millions of rollout steps affordable in the first place. And because we own the tool surface, we can build functionality the source APIs do not offer, like bounded document reads, top-n retrieval slices, and grep across everything. The model can read lines 40 through 80 instead of swallowing a whole document, and for an agent doing many tool calls inside a bounded context window, that is the difference between fitting and running out of context.

The last piece is a single, standardized interface in place of four separate vendor APIs. The agent never talks to Notion, Airtable, HubSpot, or BigQuery directly; it talks to one API that exposes SQL, semantic search, grep, document reads, and table listing uniformly over all four systems. That collapses four pagination schemes and four response formats into one, so the model spends its capacity on the question rather than on API idiosyncrasies.

How much of the model's performance actually comes from this environment design, rather than the model itself? To isolate the interface, we trained a second model with the identical recipe. Same base model, same frozen snapshot, same questions, same reward. The only change is that this agent works through a mock mirroring the same MCP functionality Notion, Airtable, HubSpot, and BigQuery usually offer, so it gets whatever the source APIs return and cannot slice or bound its reads. Average context per task grows from 5.8k to 10.5k tokens, roughly 1.8x on the mean; the finetuned model through the caching layer gets that down to 2.4k, roughly 0.4x. The mock-interface model barely improves over the base: it often runs out of context before reaching an answer. Same data, same training, different interface. The gap isn't the model getting smarter, it's the agent being able to learn how to manage its own context.

Did we break anything?

The standard worry with task-specific training is that you win the task and quietly lose the model's other general capabilities. We track a canary panel of held-out general-capability benchmarks alongside training to catch regressions on out-of-domain behavior, spanning math, knowledge, instruction following, phrasing quality, and agentic tool use. After training, every delta falls within noise: gsm8k, mmlu-pro, and antislop drift up, ifeval and the τ² domains drift down a touch.

All rows are pass rates on fixed, seeded question sets, except antislop, which is a reward score where higher is better. Parenthesized τ² values are pass⁴: the share of tasks solved in all four independent trials. "Within noise" means the delta sits inside the eval's run-to-run spread at this sample size.

gsm8k (Cobbe et al., 2021) tracks grade-school math word problems, a canary for multi-step arithmetic reasoning.

mmlu-pro (Wang et al., 2024) covers harder multiple-choice knowledge and reasoning questions across 14 domains.

ifeval (Zhou et al., 2023) tests instruction following with programmatically verifiable constraints, like "answer in exactly three sentences".

antislop (Paech et al., 2025) scores generations for degenerate stock phrasing and repetition; higher reward means less slop.

τ²-bench (Barres et al., 2025) runs conversational tool-use agents in retail, airline, and telecom customer-service domains, the closest canary to our actual workload. Pass⁴ (Yao et al., 2024) is the stricter of the two numbers: it only credits tasks the model solves reliably, not just occasionally.

Serving is the other half

This post is about model quality, but a dedicated model that sits on its own GPUs would win on accuracy and lose on cost, since real traffic is bursty and dedicated hardware swings between idle and backed up. We serve the finetune as a low-rank adapter (Hu et al., 2021) in a shared pool in the style of S-LoRA (Sheng et al., 2023), so many customer finetunes ride the same base weights and the per-token cost holds at the ~$1 figure above. The adapter gives up essentially nothing in quality against a full finetune, including in RL post-training (Schulman et al., 2025). We cover the serving setup in the serving post.

Work with us

None of this is specific to this customer. The same approach works for any task with enough call volume and a verifiable answer. We put a small model behind a context layer over your systems, train it on your own signal, and evaluate on held-out data until it beats the frontier on accuracy and cost. You provide the workflow and the data; we run the training loop, the evaluation, and keep the model current as your task shifts.

Reach out at hello@constructlabs.com.

Book an intro call

---

References

• Hu et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685
• Cobbe et al. (2021). Training Verifiers to Solve Math Word Problems. arXiv:2110.14168
• Zheng et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv:2306.05685
• Sheng et al. (2023). S-LoRA: Serving Thousands of Concurrent LoRA Adapters. arXiv:2311.03285
• Zhou et al. (2023). Instruction-Following Evaluation for Large Language Models. arXiv:2311.07911
• Shao et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300
• Wang et al. (2024). MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark. arXiv:2406.01574
• Yao et al. (2024). τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv:2406.12045
• Yang, Kautz & Hatamizadeh (2024). Gated Delta Networks: Improving Mamba2 with Delta Rule. arXiv:2412.06464
• DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948
• Chu et al. (2025). SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-Training. arXiv:2501.17161
• Barres et al. (2025). τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment. arXiv:2506.07982
• Zheng et al. (2025). Group Sequence Policy Optimization. arXiv:2507.18071
• Shenfeld, Pari & Agrawal (2025). RL's Razor: Why Online Reinforcement Learning Forgets Less. arXiv:2509.04259
• Schulman et al. (2025). LoRA Without Regret. Thinking Machines Lab
• Paech et al. (2025). Antislop: A Comprehensive Framework for Identifying and Eliminating Repetitive Patterns in Language Models. arXiv:2510.15061