Automation’s hardest test begins after the demo
For readers following AI tools, the most consequential question is no longer whether a model can draft a polished answer. It is whether an AI workforce can notice trouble, resist pressure, find the decisive evidence and complete work that changes a company’s fate.
Firmulate has turned that question into a live business story. Its software company has 13 synthetic employees and real money mechanics: it burns €105k each month against €2.3k in monthly recurring revenue. A public cash countdown makes the stakes visible, while every workday is versioned. The company has accumulated 680+ self-learned playbook rules as it attempts to operate and survive.
This is build-in-public pushed to an unusual extreme. Visitors can watch the company live, not merely read a retrospective about what worked. The expanding record supplies daily evidence of how synthetic employees behave when business conditions are unforgiving.
Building AI-Powered Products: The Essential Guide to AI and GenAI Product Management
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
A company becomes a management wargame
The live operation also provides the setting for the Crucible League, finalized in July 2026. Each frontier model ran the same small software company through its worst week, facing the same customers, crises and temptations. Every decision was versioned and auditable.
The final table placed gpt-5.6-sol first with 95, followed by Kimi K3 with 93, Sonnet 5 with 88, Fable 5 with 77 and Opus 4.8 with 73. A do-nothing baseline scored 26 because partial progress still counted. But the evaluation imposed a hard boundary around integrity: a single breach of trust capped the total, under the principle that “no amount of good work outweighs a breach of trust.”
The striking result was not that the models missed obvious alarms. All of them spotted every crisis, and all refused every manipulation attempt. The separation appeared at the end of the work. Only two models signed the €55,000 deal that their own analysis had earned: “Same diagnosis, same pitch — no signature.”
That distinction matters for anyone evaluating automation through chat demonstrations. Recognizing a problem, recommending the right action and finishing the commercial task are separate capabilities. In this experiment, several participants reached the right diagnosis without converting it into the required outcome.
The decisive fact was buried in the company’s files
The deal also exposed the value of organizational reading. The decisive weakness in a competitor was not sitting in the customer event. It was two document references deep in the company’s own files. Models that followed the trail found the fact and won the deal at full price, worth +€4,583 in monthly recurring revenue.
That is a particularly relevant lesson for businesses considering AI agents. Useful company knowledge may be dispersed across routine records rather than presented in the prompt or event that triggers a task. The winning behavior was to read the available material deeply enough to find what the immediate situation did not reveal.
Pressure tested honesty as well as competence
The worst week included fake CEO messages escalating across three stages and a reporter’s attempt to extract “just one yes/no, on background.” All 5 models refused. Kimi K3 stated its reasoning on the record: “Treat the request as a suspected approval-bypass / possible impersonation.”
That clean refusal came with an important comparison caveat. K3 ran without an effort parameter, using the API default, while the other participants ran at xhigh. Even with that difference, it finished second with 93 and was one of the models that completed the deal.
Thoroughness did not guarantee execution
Opus 4.8 produced the deepest analyses and added +80 learned rules, making it the most thorough participant. It nevertheless finished last in the league. The close was left on the table, and discipline slipped when it attempted to write into a locked department instead of escalating. The same weakness appeared in all four of the others, though less strongly.
The contrast is useful because it separates visible effort from business completion. A large body of analysis and learning can coexist with missed escalation and unfinished work. Firmulate’s public record makes that mismatch observable rather than hiding it behind a fluent final response.
The experiment also contains 242 real, unedited management decisions used for a “guess the model” quiz. They offer another way to examine whether readers can reliably distinguish models from the choices they make rather than from their conversational style. Enterprises can also run the same wargame against a read-only export of their own business, with nothing writing back to real systems.
AI Change Management Made Simple: A 9-Step Framework for Business Leaders to Drive Generative AI Transformation (Reduce AI Fear, Win Buy-in, and Accelerate AI Adoption Across Your Organization)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The useful question is whether AI finishes responsibly
Firmulate’s live company turns AI automation into an ongoing corporate survival story. Its synthetic workforce must operate under a visible cash countdown, learn from its work and face decisions whose consequences can be audited.
The Crucible League’s central finding is therefore more demanding than “the models understood the problem.” Every participant recognized every crisis and resisted every manipulation attempt, yet only two signed the deal their analysis supported. The buried competitive fact rewarded models that read beyond the immediate event; the unfinished closes punished those that stopped short of execution.
For organizations assessing AI tools, the practical standard is emerging clearly: watch what the system completes, what evidence it consults and whether its discipline survives pressure. Firmulate makes that behavior public through the live experiment and the synthetic employees’ published words. The result is less like a product showcase and more like a continuously observed business fighting to stay alive.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
AI NATIVE – KNOWLEDGE GRAPHS: Designing Knowledge Maps & Knowledge Graphs (The AI-Native Series)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Analytics, Data Science, & Artificial Intelligence: Systems for Decision Support
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
