🔍 Read the full analysis: The Fine Print To Consider As OpenAI Trains Agents In Your Software on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
OpenAI described training a frontier model in hosted copies of Ironclad’s contract-management software, using 11 legal, commercial and procurement tasks. GPT-6 Astra met an average 55% of task criteria, while its reported time savings were simulated estimates, not measured customer results.
OpenAI said on Oct. 6 that it trained its frontier model GPT-6 Astra in hosted copies of Ironclad’s contract-management software, testing whether an AI agent could carry out multi-step legal, commercial and procurement work. The results point to a possible new way of training agents on professional software, but Astra met an average of 55% of task criteria and the reported completion times were simulations, not measured customer savings.
OpenAI and Ironclad selected 11 tasks with staff who use the contract platform. Examples included setting up nondisclosure agreements, creating procurement approval processes and changing a reusable contract clause based on a requester’s jurisdiction. OpenAI estimated that an experienced user would take 30 to 40 minutes for each task.
Each task was evaluated against 8 to 50 criteria, depending on its complexity. OpenAI reported that GPT-6 Astra met an average of 55% of those criteria, compared with 41.6% for GPT-5.6 Sol in a high-reasoning setting. An internal model used in Astra’s development reached 63.7%. Astra met about 94% of the criteria on one showcase task, but that example is not the overall result.
OpenAI reported estimated attempt times of 19.2 minutes for Astra and 37 minutes for GPT-5.6 Sol. The company’s footnote says those figures are simulated from assumed processing and generation speeds. They are not observed customer time savings and apply to the 11 research tasks, not to Ironclad workflows broadly.
OpenAI is training agents inside your software. Read the fine print on Ironclad.
Several AI trackers guessed “Ironclad” was a hardened agent framework. It’s a contract-management software company — and the post describes OpenAI training its frontier model inside a vendor’s real product, then inviting other vendors to do the same.
legal, commercial & procurement — e.g. NDAs, approval flows, jurisdiction clauses
per task, experienced user (OpenAI estimate)
criteria per task — a rubric, not pass/fail
public SEC filings; no customer or non-public Ironclad data
The average share of rubric criteria met — not tasks completed. In contracting, partial credit isn’t partial value: a workflow that skips one required approval is the exact failure the system exists to prevent.
37.0 → 19.2 minutes are “simulated estimates … not measured customer time savings,” per OpenAI’s own footnote. Credit to OpenAI for saying so plainly.
Its hardest customer problems get built into the next frontier model; agents that work well in its product make the product more valuable.
Every improvement makes the model better at operating the vendor’s interface. Taken far enough, the agent becomes the interface.
Averages hide missed approvals.
Narrowest access; no self-escalation.
METR found agents spoofing tool-call records.
Measure the whole loop.
Public filings, not your contracts.
Modest numbers, significant method. A frontier lab is moving from general computer use to training inside specialised business software, with the vendor’s help — agents learning their trade the way people do. Today: just over half of a contracting workflow’s requirements, in simulated time, on 11 research tasks.Software vendors are becoming training grounds for the agents that may one day operate their products for them.
Why Partial Workflow Accuracy Matters
The reported 55% is the average share of criteria met, not the percentage of tasks completed successfully. That distinction matters in contract and procurement work, where missing one requirement can invalidate an otherwise polished result. For example, a procurement process may need Finance approval above a spending threshold, Security review for certain requests and Legal review for unusual terms. Missing one control can route a purchase incorrectly.
OpenAI’s report recognizes that an agent can lose track of a business rule during a task and says human oversight remains important. Ironclad CTO Sunita Verma likewise emphasized that agents must preserve “the controls teams rely on.” The test suggests progress on complex software tasks, but does not establish that agents are ready to handle consequential workflows without people checking their work.
For software vendors, the partnership model could help expose where agents struggle in real products and give vendors a role in shaping future capabilities. It may also change how customers use software: if agents increasingly act through a product, a vendor’s lasting value may depend less on its screens and more on its business rules, data, audit records and controls.
AI legal contract management software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
How Ironclad Became a Test Environment
OpenAI’s post, titled “Advancing computer use with Ironclad,” describes training and evaluation inside a specialist software product rather than a general computer-use benchmark. Ironclad is a contract-management software company, not the name of a new agent framework. The tasks were drawn from legal, commercial and procurement workflows selected by people familiar with the product.
For practice, Ironclad supplied hosted copies of its product. OpenAI says it created synthetic training tasks from publicly filed contracts in the U.S. Securities and Exchange Commission’s EDGAR database, with personal information filtered out. The company says it did not use OpenAI customer data, OpenAI internal contracts or non-public Ironclad customer data.
OpenAI is asking a small number of software companies to partner on similar research. It says prospective partners should bring a concrete task agents cannot reliably complete, people with detailed knowledge of the work, a secure test environment and data that can safely be used for research. The report presents this as a way to study agents in specialized software, not as evidence that the tested agent is ready for general deployment.
AI workflow automation tools for business
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
What the Test Does Not Establish
The results do not show how often Astra would complete a full workflow to a standard that a company could accept without review. The 55% figure averages rubric criteria, and the report’s headline metric does not by itself show which requirements were missed or how serious each miss was. A high score on one example does not settle performance across the other tasks.
It is also unclear whether the results would hold across different companies, software configurations, contract types or live customer environments. OpenAI’s time figures are simulated, and the test covered 11 selected research tasks. The material does not establish measured productivity gains, error rates in production or the level of human review needed for each workflow.
OpenAI says it excluded specified customer and internal data from training, but the report does not, in the supplied material, detail every security, retention or access arrangement for hosted testing. Companies considering agent use should seek task-level evaluation results and written information about data handling and controls rather than treating an aggregate score as a deployment guarantee.
AI-powered contract review software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The Next Test Is Vendor Adoption
OpenAI says it is inviting a small number of software companies to work on tasks current agents cannot reliably finish. The next developments to watch are which vendors participate, what tasks they select and whether future reports include clearer task-by-task results, production testing and independently observed time savings.
For buyers, the immediate next step is to ask vendors how an agent’s work is evaluated and supervised before enabling it in systems that handle contracts, finances or customer records. Useful details include the specific criteria it failed, how approval controls are enforced, what actions require human sign-off and how activity is recorded. Until those answers are available, OpenAI’s Ironclad study is best read as a research demonstration with incomplete performance, not proof of hands-off automation.
As an affiliate, we earn on qualifying purchases.
Key Questions
What did OpenAI test with Ironclad?
OpenAI evaluated GPT-6 Astra on 11 legal, commercial and procurement tasks in hosted copies of Ironclad’s contract-management software.
Does the 55% score mean Astra completed 55% of the tasks?
No. It is the average share of rubric criteria met across the tasks, not the percentage of tasks completed. The figure also does not show, on its own, which requirements were missed.
Did Astra cut customer task times in half?
That has not been shown. OpenAI reported a simulated estimate of 19.2 minutes per attempt for Astra, compared with 37 minutes for GPT-5.6 Sol. The company says these were not measured customer time savings.
Did OpenAI use Ironclad customer contracts to train the model?
OpenAI says it used synthetic tasks based on publicly filed contracts in the SEC’s EDGAR database and did not use non-public Ironclad customer data, OpenAI customer data or OpenAI internal contracts.
Can companies rely on agents to run these workflows without review?
The report does not establish that. Astra met an average of 55% of task criteria, and OpenAI says human oversight remains important. Companies would need more task-specific evidence and clear controls before relying on agents for consequential work.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
