Az IBM az élre repítette ügyfélszolgálati ágensét modellfrissítés nélkül

Learned memory takes a basic ReAct agent with GPT-4.1 to the top τ²-bench tests policy-bound customer service Guidelines improve capability and reliability The best amount of guidance depends on the domain Better performance costs more tokens, not more turns Learned guidelines also improve agents using frontier models ALTK-Evolve tops the leaderboard comparison Agents can improve without weight updates Links: ALTK-Evolve learns and applies a playbook using GPT-4.1, outperforming the retail entries on τ²-bench, a customer service agent benchmark—without updating model weights. Using GPT-4.1, an older, non-reasoning model, a basic customer-service agent reaches 67.5% Pass^4 on τ²-bench retail — meaning it solves the task on all four independent attempts for 67.5% of tasks. It scores above every retail entry in our October 12, 2026 leaderboard snapshot. It runs on the benchmark's stock ReAct scaffold, with learned guidelines appended to its system prompt. ALTK-Evolve builds that playbook using GPT-4.1. It analyzes the agent's training-task attempts and uses the model to distill successes and failures into short behavioral rules: how to authenticate a customer, confirm a transaction, or handle a return. Those guidelines are then added to the agent's prompt for subsequent conversations. The same model powers both guideline extraction and the agent; no weight updates or human annotation are needed. Our previous work demonstrated this approach on coding agents. Here, we test it on customer service, where success requires following domain policies, understanding a customer's request, and executing the right tool actions. τ²-bench (from Sierra) measures how well an agent handles customer-service conversations under real constraints. Each task pairs the agent with an LLM-simulated customer who has a goal in mind, a set of domain tools (look up an order, issue a refund, rebook a flight), and a written policy document the agent must follow. The agent has to draw out the right details, stay within policy, and take the correct — often irreversible — actions; a task counts as solved only when the final database state matches the expected outcome, a strict binary reward. We use its two customer-service domains, retail and airline. We run τ²-bench's own implementation of a basic ReAct agent with a two-phase THINK → ACT loop, using gpt-4.1 as the agent model and gpt-5.5 as the customer-simulator model; 4 trials per task. The only change we make is a block of learned guidelines appended to the system prompt — everything else (policy, tools, scaffold) is stock τ²-bench. Concretely, the learned guidelines are tacked on after the domain policy as a plain markdown list: We extract guidelines from the agent's trajectories on τ²-bench train tasks using gpt-4.1, the same model that powers the agent, then apply them unchanged on the held-out test set. In a prior post, we talked about how different models / tasks require different dosage of guidance. Inline with the idea, we also evaluate different amounts of guidelines provided to the model. Each guideline carries a support count — the number of distinct training tasks it was extracted from, so higher support means a rule that generalized across more tasks rather than a one-off. We compare three configurations: baseline (no guidelines), sup3 (keep only guidelines with support ≥ 3 — a small, high-confidence core), and lossless (keep every guideline retained after consolidation, i.e. support ≥ 1). Table 1. Guideline configurations and retained guideline counts by domain. The metrics we track. Every task is attempted 4 times, and we read those trials two ways: Figure 1. Mean Accuracy and Pass^4 on held-out retail and airline tasks for agents using gpt-4.1 across four trials per task. Agent model: gpt-4.1 (user simulator: gpt-5.5). Table 2. Held-out agent performance using gpt-4.1 across four trials per task. Parentheses show percentage-point changes from baseline. Mean Accuracy climbs in both