Research ·
Ibraheem Shaikh · Urjit Chakraborty
AI customer support agents operate within well-defined policies and handle recurring categories of inquiries. At scale, running them on frontier models becomes expensive. Smaller, specialized models can significantly reduce that cost while preserving the capabilities the job requires.
We tested this approach on the Telecom domain of Sierra’s τ²-Bench, where the agent must solve a simulated customer’s phone issue (no service, slow data, and so on) by asking questions and working in the customer’s carrier portal.
These tasks can require several coordinated fixes. The agent must identify each problem, obtain any required approval, guide the customer through the device fix, and verify that service works. The benchmark checks whether the required outcome was reached, including a correct human handoff when appropriate.
We generated training data with Gemini 3.8 Flash at medium reasoning as the teacher. It ran the 74 training tasks, and its conversations, including customer messages, reasoning summaries, agent responses, and tool calls, became the training set. The remaining 40 tasks were held back for evaluation.
We then fine-tuned the student, OpenAI’s open-weight gpt-oss-20b at high reasoning, to predict the teacher’s next response or tool call from the conversation so far. Training took under an hour and taught the model what to do across diverse scenarios: a line suspended over an unpaid bill, a data allowance that has run out, a phone stuck on the wrong network, or picture messages that won’t send abroad.
We ran the final model three times on each of the 40 evaluation tasks, 120 attempts in all, and report average task success and inference cost per attempt. Costs for the base and fine-tuned gpt-oss-20b use Tinker’s published per-token rates.
Our final model achieved 86.7% task success, passing 104 of 120 attempts across three trials of 40 Telecom tasks. Before fine-tuning, the same gpt-oss-20b scored 60.2% at 3.09¢ per attempt, so fine-tuning raised task success by 26.5 points and cut cost by 22%. The base model often tried to change settings on the customer’s phone itself, using tools it doesn’t have, and its reasoning went in circles. The fine-tuned model asks the customer to make those changes and spends less time on unnecessary reasoning, which is also where the cost saving comes from.
Our model finished within 1.9 percentage points of Claude’s published score at approximately 65× lower inference cost.
Taken together, these results demonstrate the value of specialization and fine-tuning: a 20B-parameter model trained on 74 customer support tasks achieved performance competitive with frontier models at a fraction of the inference cost.
For our customers, Rudimint derives training tasks from production traces of their workflows, keeps the ones worth learning from, and fine-tunes a model on a stronger teacher’s behavior on those tasks. After deployment, we track where the model still falls short and generate new training data aimed at those gaps, so it keeps improving on the traffic it actually serves.