TL;DR
Z.ai’s GLM-5.3-FlashX-Legal recorded a 19.17% all-criteria pass rate on Vals AI’s Harvey Legal Agent Benchmark, placing seventh among 81 models on the archived leaderboard. The same run passed 92.5% of individual rubric criteria, but the benchmark counts a task as complete only when every required criterion passes. Z.ai’s claims about open-source leadership and a 12.5-point improvement need to be read alongside the benchmark’s scoring rules and the absence of published weights for the legal model in the source material.
Z.ai’s GLM-5.3-FlashX-Legal scored 19.17% on the all-criteria pass measure of the Harvey Legal Agent Benchmark, ranking seventh of 81 models on Vals AI’s archived leaderboard. The result indicates that the model completed every requirement on about one in five benchmark tasks; the benchmark gives no partial credit at the task level, even though the same run passed 92.5% of individual criteria.
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
Z.ai describes GLM-5.3-FlashX-Legal as a model for professional legal workflows, including document understanding, legal reasoning and task execution. Its developer documentation says the system is intended to support research, contract review and drafting, including initial drafts. The model is built on GLM-5.3-Flash, according to Z.ai, and the company recommends using it as an agent connected to services for laws, cases and company data.
Z.ai says the model received legal-domain mid-training and post-training, including reinforcement learning across tens of thousands of task environments modeled on legal workflows. The company’s documentation reports a 19.2% all-criteria pass rate on Harvey’s benchmark and says this is 12.5 percentage points above the base model. Vals AI’s leaderboard lists the legal model at 19.17%, seventh of 81, with a listed test cost of $0.15. Z.ai’s pricing page separately lists $0.37 per million input tokens and $1.25 per million output tokens.
The two percentages describe different levels of scoring. Vals AI’s result means the model met every requirement on 19.17% of benchmark tasks, while the reported 92.5% individual-criterion pass rate counts criteria separately. A single missed citation, deadline or other required item can make an entire task fail under the benchmark’s all-pass rule.
Understanding Z.ai’s Legal AI: Low Pass Rate, High Benchmark Ranking
GLM-5.3-FlashX-Legal ranked seventh on Vals AI’s archived Harvey leaderboard. Its 19.17% score measures fully completed tasks, while its 92.5% criterion pass rate counts individual rubric checks.
One model, two levels of scoring
A high rank and a low all-pass rate can coexist: the leaderboard compares task completion, while rubric accuracy counts each requirement separately.
What the headline means: The model met every required criterion on about one in five benchmark tasks. One missed item can make a whole task fail, even when most of its rubric checks pass.
Criteria are not completed tasks
Harvey’s all-pass rule is deliberately demanding. It measures whether an agent delivered a complete work product under the benchmark rubric.
Reported performance · scale 0–100%
From legal matter to leaderboard score
The public set spans legal work; the leaderboard uses a separate private holdout and averages task pass rates from two judges.
Task + matter
An instruction is paired with a client matter and its relevant materials.
Work product
The agent produces a result for review, such as research, analysis, or a draft.
Rubric checks
More than 75,000 expert-written binary criteria assess facts, citations, deadlines, and more.
All-pass result
Every criterion must pass for a task to count as complete; holdout grading averages two judges.
“A task is marked complete only if every criterion passes.”
Harvey · Benchmark grading rule
What Z.ai says the model is built to do
The company positions GLM-5.3-FlashX-Legal for professional legal workflows, with human review still central to use.
Research & drafting
Z.ai describes support for document understanding, legal reasoning, research, contract review, and initial drafts.
Legal-domain tuning
The company says the model builds on GLM-5.3-Flash and received legal mid-training and post-training, including reinforcement learning across tens of thousands of task environments.
Connected agent
Z.ai recommends connecting the model to services for laws, cases, and company data. The supplied material does not detail data handling safeguards.
What the available sources establish
Treat the leaderboard, company statements, and a small spot check as different kinds of evidence.
| Evidence | What it says | What remains open |
|---|---|---|
| Vals AI leaderboard | 19.17%; seventh of 81; $0.15 listed test cost | Rank and score reflect this benchmark’s holdout and grading method. |
| Z.ai documentation | Reports 19.2% and a 12.5 percentage-point gain over the base model | No published weights for the legal model appear in the supplied material; the open-source comparison set is unspecified. |
| Small spot check | Four tasks reportedly included unfindable rulings cited by the legal model | Too small to estimate error rates or compare systems; not a benchmark result. |
| Pricing page | $0.37 per million input tokens; $1.25 per million output tokens | Token pricing is separate from the listed benchmark test cost. |
Careful interpretation: Z.ai’s “state of the art among open-source models” description cannot be independently assessed from the supplied material because its comparison set is not specified. The source material also does not establish that model output can be relied on without lawyer review.
Questions firms should answer before use
A benchmark result is a starting point for evaluation. Local testing should reflect the firm’s matters, review process, and data obligations.
How much correction?
Measure lawyer edits on representative tasks, including the frequency and severity of errors across practice areas.
Can sources be verified?
Check that cited authorities exist, support the stated proposition, and are reviewed before client work leaves the firm.
Where does client data go?
Establish who processes data, which rules apply to integrations, and what safeguards govern connected services.
Availability note: Z.ai announced an application-based beta on October 13, 2026; Vals AI lists October 12 as the release date. The supplied sources provide no public release schedule or confirmed date for further results.
Quick answers
The score is best understood as a strict measure of complete task execution, not a general verdict on legal usefulness.
What does 19.17% mean?
It is the share of benchmark tasks on which the model passed every required criterion, according to Vals AI’s leaderboard.
How can 92.5% criteria mean 19.17% tasks?
Criteria are counted individually, but a task fails if even one required check fails. Several small misses can lower the whole-task rate.
Does this prove it is the best open-source legal AI?
No. The supplied source material does not specify the comparison set behind Z.ai’s claim, and it reports no published weights for the legal model.
Does a strong rank prove safe legal use?
No. Firms still need to verify citations, drafts, correction effort, and client-data handling in their own review process.
Why Task Completion Matters
The difference between criterion accuracy and full task completion matters to law firms evaluating whether an AI agent can prepare work for professional review. A system may satisfy most rubric items across a set of tasks while still producing a fully compliant work product relatively infrequently under an all-or-nothing scoring method. The benchmark therefore tests a demanding form of reliability, but its task pass rate alone does not describe how useful partial outputs might be in a lawyer’s workflow.
For legal teams, the result points to practical questions beyond leaderboard position: how often lawyers must correct drafts, how citations are checked, and what client information is processed when the model is connected to legal or company data. Z.ai presents the model as a tool for legal professionals and initial drafting. The source material does not establish that its output can be relied on without review.
A small spot check described by Thorsten Meyer AI adds a caution but is not a benchmark: across four tasks, the author said the legal model cited rulings that could not be found. The author also reported that two Claude models cited a real ruling for the opposite of its holding. These observations are limited and do not establish comparative error rates, but they underline why citation verification remains relevant even when a model posts a strong benchmark rank.
Products Worth Considering
Samsung 27-inch Odyssey OLED G6 QHD Gaming Monitor
Bose QuietComfort Wireless Headphones – Twilight Blue (Limited Edition)
Apple Watch Series 12 GPS 42mm
As an affiliate, we earn on qualifying purchases.
How Harvey Scores Legal Agents
Harvey released the Legal Agent Benchmark as open source on May 6, 2026. Its public set contains 1,250 tasks across 24 legal practice areas. Each task pairs an instruction with a client matter containing relevant materials and asks an agent to produce a work product for review. Harvey says the benchmark uses more than 75,000 expert-written, binary rubric criteria covering matters such as facts, conclusions, citations, deadlines, recommendations and formatting.
Under Harvey’s all-pass grading rule, a task counts as complete only if every criterion passes. The company’s example is a deal-team report that identifies eight of ten risks: it would not receive an 80% task score, because the report remains incomplete. That rule explains how a model can pass 92.5% of individual criteria yet pass fewer than one in five whole tasks.
The public leaderboard uses a private holdout set that Vals AI describes as containing 120 tasks, with internet access disabled. Vals says two language-model judges, GPT-5.5 and Claude Sonnet 4.6, grade each submission; the reported result averages the judges’ task pass rates. Z.ai’s AutoClaw blog announced an application-based beta on October 13, 2026, while Vals lists the release date as October 12. The developer documentation containing the benchmark claim is undated.
“A task is marked complete only if every criterion passes.”
— Harvey, describing its benchmark grading rule
Open Questions on Reliability
The source material does not specify the comparison set behind Z.ai’s description of the result as state of the art among open-source models. It also reports no published weights for GLM-5.3-FlashX-Legal. The 12.5-percentage-point improvement is Z.ai’s claim and is described as checking out against the base model on the same leaderboard table, but the underlying scores and evaluation details are not reproduced here.
It remains unclear how the benchmark result translates to routine legal work, including performance across practice areas, the frequency and severity of errors, and the amount of lawyer correction needed. The source material also does not establish how client data is handled when the model is connected to external services, or what safeguards apply to those integrations. The four-task spot check is too small to answer broader questions about citation accuracy.
What Firms Should Verify
Z.ai has opened a beta by application, according to its October 13 AutoClaw post. The available source material does not give a public release schedule or say when further benchmark results will be published. Firms considering access can compare the model’s performance on their own review tasks and check its citations and drafts before using them in client work.
Procurement and deployment reviews will also need to establish who processes client data, which rules govern that processing, and how outputs are checked before they leave the firm. Any later leaderboard updates, published model weights or more detailed documentation could clarify the model’s availability and performance, but none is confirmed in the material provided.
Key Questions
What does the 19.17% score mean?
It is the share of benchmark tasks on which GLM-5.3-FlashX-Legal passed every required criterion, according to Vals AI’s leaderboard. A task with even one failed criterion does not count as complete.
How can the model pass 92.5% of criteria but only 19.17% of tasks?
The 92.5% figure counts individual rubric criteria, while 19.17% counts whole tasks. Since every criterion must pass for a task to count, several small misses can reduce the task-level score.
Did Z.ai prove that its model is the best open-source legal AI?
No such conclusion is established by the material provided. Z.ai describes the result as state of the art among open-source models, but the source does not define the comparison set, and it reports no published weights for the legal model.
Is GLM-5.3-FlashX-Legal available?
Z.ai’s AutoClaw blog announced an application-based beta on October 13, 2026. The source material does not specify a broader availability date.
Can lawyers use the model’s output without checking it?
The source describes the model as supporting legal professionals and producing initial drafts. It does not establish that outputs are suitable for use without review; the reported spot check also raised citation-verification concerns, though it covered only four tasks.
Source: Thorsten Meyer AI
