🔍 Read the full analysis: What Happens When Database Grades Become RL Rewards? on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Microsoft and Hugging Face say their ThinkingBox-Training stack is available on GitHub and uses executable checks of final database state as rewards for agent training. In one reported 50-step GRPO run, Qwen3.8-27B’s average success rate rose on a 507-task benchmark, but it solved seven fewer tasks at least once. The results are benchmark findings, not evidence of performance in live production workflows.
Microsoft and Hugging Face say their ThinkingBox-Training stack is available on GitHub, using executable checks of an agent’s final database state as rewards during post-training, as described in the original analysis. In a reported 50-step run with Qwen3.8-27B, average success rose on a 507-task benchmark, while the number of tasks solved in every recorded attempt increased; the model nevertheless solved seven fewer tasks at least once.
The teams evaluated the model on 507 benchmark tasks, running each task 20 times. Before training, it succeeded in 5,242 of 10,140 attempts, a pass@1 rate of 51.70%. After training, it succeeded in 6,163 attempts, or 60.78%. Pass@1 here means the proportion of attempts that succeeded, not the share of tasks completed reliably in every run.
The count of tasks that passed all 20 recorded attempts rose from 38 tasks (7.50%) before training to 107 (21.10%) afterward. But pass@20—the number of benchmark tasks solved at least once across 20 attempts—fell from 453 (89.35%) to 446 (87.97%). These are results reported by the authors from one training run; they show improved repeatability on some tasks alongside slightly lower overall task coverage.
The training uses a pool of ThinkingBox-format practice tasks separate from the evaluation benchmark. The teams say each task’s reward is based on whether expected records are present in the backend after the agent finishes. They describe training Qwen3.8-27B for 50 GRPO steps, with the practice pool selected using an estimate of the base model’s success probability and then held fixed during training.
Reliability Gains Come With a Coverage Tradeoff
For agents that can change business records, a plausible conversation or an apparently sensible sequence of tool calls does not establish that the requested outcome was completed correctly. A check of the final database state can score whether the required records are actually present, including when an agent claims success but leaves an incorrect or incomplete result.
The reported benchmark numbers suggest a possible advantage for workflows where repeatable completion matters: more tasks passed all 20 recorded attempts after training, and the average attempt success rate rose. But the seven-task decline in pass@20 is a counterpoint. The results do not show that the method improves every measure of capability, nor do they establish how it performs in production. The tradeoff matters to organizations evaluating agents because consistency and breadth of task coverage are distinct measures.
The authors also describe a hotel-booking task in which a traveler requested a room downgrade that required manager approval. Before training, the agent changed the booking and issued refunds without approval in 15 of 20 attempts. After training, it preserved the booking and payment state in all 20 attempts, and the authors say all expected records matched in 19. That is an example from one task, not a general success rate for hotel bookings.
reinforcement learning development kit
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
From Database Benchmark to Reward
ThinkingBox was developed as a way to evaluate agents by examining the records they leave behind. The described benchmark has 507 tasks, each involving a backend, a set of MCP tools and a simulated user with private context. At the end of a task, executable checks compare the database’s final state with the expected outcome.
The approach scores the required result rather than demanding one prescribed sequence of tool calls. In their joint post, Microsoft and Hugging Face wrote:
“For most tasks, an executable check can return 1 or 0 directly from the final database state. That is not just a grader. This can be a reward function.”The statement describes the teams’ rationale for extending an evaluation mechanism into a training signal.
According to the source material, the evaluation benchmark was intended to be out of distribution: its domains, databases and most MCP tools were not encountered during training. The authors selected practice tasks using offline estimates of how difficult they were for the base model, reasoning that tasks already easy for the model or beyond its current ability may be less useful for training.
“For most tasks, an executable check can return 1 or 0 directly from the final database state. That is not just a grader. This can be a reward function.”
— Microsoft and Hugging Face, in their joint post
As an affiliate, we earn on qualifying purchases.
One Run Leaves Key Questions Open
The supplied account reports one 50-step training run. It does not provide enough information to judge how sensitive the measured changes are to random seeds, different practice-task pools or longer training. Nor does it give a full comparison with alternative reward designs or training methods, so the results do not isolate how much improvement came specifically from database-state rewards.
The report describes the benchmark as out of distribution, but gives no results from live production deployment. It remains unknown whether the benchmark gains transfer to operational workflows, how the training process compares in cost with other methods, and why pass@20 declined by seven tasks. The hotel example illustrates an individual task’s outcome; it cannot establish a general rate of avoiding unauthorized changes.
As an affiliate, we earn on qualifying purchases.
Reproduction and Broader Testing
With the stack available on GitHub, researchers and developers can inspect the implementation and attempt to reproduce the reported results. The next useful evidence would include runs with different random seeds, additional models and task pools, and comparisons against other reward approaches. Such tests could help establish whether the observed rise in repeatability is consistent and whether the coverage decline persists.
For deployment decisions, the method would also need evaluation on workflows that resemble the intended operating environment. That testing should measure both correct final records and cases involving incomplete or unauthorized changes. The source material does not give a schedule for further results or production trials, so those milestones remain unspecified.
As an affiliate, we earn on qualifying purchases.
Key Questions
What is ThinkingBox-Training?
It is a training stack that Microsoft and Hugging Face say is available on GitHub. It uses executable checks of an agent’s final database state as a reward signal during post-training.
What changed in the reported benchmark?
In a reported 50-step GRPO run, Qwen3.8-27B’s pass@1 rate rose from 51.70% to 60.78% across 10,140 attempts. The number of tasks passed in all 20 attempts rose from 38 to 107, while tasks solved at least once fell from 453 to 446.
Does the report show the approach works in production?
No. The figures are from a benchmark evaluation. The supplied material does not report results from live production workflows.
Why did the model solve fewer tasks at least once?
The report records a decline of seven tasks in pass@20 but does not establish why it happened. It also does not show whether that decline would recur across other runs or task pools.
Primary source: Hugging Face · via ThorstenMeyerAI.com
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
