TL;DR
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
GLM-5.3 FlashX Legal scored 8 out of 100 on Lexassay, a test of 15 legal questions answered without internet access or tools. The test author says it fully solved none of the tasks and gave no answer on two; the small, single-run test is a spot check, not a definitive ranking of legal AI systems.
Lexassay tested 42 model configurations on the same 15 questions spanning German, US and EU law. The models answered without internet access or other tools. GLM-5.3 FlashX Legal scored 8 points; its base model scored 3. Europe’s Mistral Large 4 scored 10, while the top listed result was Claude Opus 5.5 at 79. Meyer says the test is a spot check: each model and task received one answer, so small score differences may reflect noise.
The questions include calculating court deadlines, identifying recent legal changes and leading decisions, and spotting false premises. Answers are graded against written criteria, and cited laws and court decisions are checked against official sources. The scoring rules deduct points for invented or misattributed citations and for answers that omit a legal-advice disclaimer or referral to a lawyer. Meyer reports that across published runs, Lexassay checked 2,644 citations as real and found 804 invented or misattributed. Those totals cover all runs, not only the law-focused model.
One example illustrates why the test examines detail. Two German civil cases were served on April 27, 2027, in Munich and Berlin. The appeal deadline differs because Corpus Christi is a public holiday in Bavaria but not Berlin: May 28 in Munich and May 27 in Berlin. Meyer says most strong models answered correctly, while one moved Berlin’s deadline a day late and another put the appeal-grounds deadline a month late. These are test examples, not advice about an actual case.
Why Did the Law-Built AI Model Score Just 8 Out of 100?
GLM-5.3 FlashX Legal scored 8 on Lexassay’s test of 15 legal questions. The result is a one-run spot check, not a definitive ranking of legal AI.
What the test found
Models answered questions about German, US and EU law without internet access or other tools.
No task fully solved
Test creator Thorsten Meyer says the model fully solved none of the questions and gave no answer on two.
Scores varied widely
The base model scored 3, Mistral Large 4 scored 10, and Claude Opus 5.5 led the listed results at 79.
Why legal details matter
A location-specific holiday can change a deadline, and an incorrect citation can make an answer look well supported.
- Corpus Christi is a public holiday in Bavaria.
- Reported appeal deadline: May 28.
- Corpus Christi is not a public holiday in Berlin.
- Reported appeal deadline: May 27.
How to read the score
Lexassay records one answer per model and task, so small score differences may reflect noise.
The test does not show consistency across repeated attempts or performance in real legal workflows.
Questions cover deadlines, legal changes, court decisions and false premises; answers and citations are checked against written criteria.
Models had no research tools, and a separate benchmark produced a different ranking.
Why Legal Deadlines Expose Errors
A wrong date or citation can have consequences beyond an inaccurate answer. Meyer notes that under German civil procedure, a late appeal is rejected as inadmissible. The example shows how a seemingly minor location-specific detail can change a deadline, while an incorrect citation can make an answer appear better supported than it is.
The findings bear on how readers should use general-purpose and law-focused AI tools. The test suggests that a product description promising legal specialization does not by itself establish that a model can reliably handle legal work. It also shows why any score needs its method and limitations alongside it: these models answered without retrieval tools, and performance may differ when a system can search supplied or official documents.
Meyer advises readers to treat chatbot output as a starting point, verify each cited authority in its source, and consult a lawyer when a deadline, money or communication with an authority is at stake. The test does not establish how the models perform in professional workflows with human review.
Products Worth Considering
Bose TV Speaker (2nd Gen) Compact Soundbar
Samsung 34-Inch Odyssey G5 Ultra-Wide Curved Gaming Monitor
LG UltraGear 34GX90SA-W 34-inch WQHD Curved OLED Gaming Monitor
As an affiliate, we earn on qualifying purchases.
How Lexassay Tested Legal Models
Lexassay, run by Meyer, uses 10 published questions and five hidden questions. The hidden-task scores are published, but the questions are withheld, according to the source material. Tasks cover multiple legal systems and include questions designed to test whether a model challenges an incorrect premise rather than accepting it.
Each answer is compared with a reference answer using three to six weighted criteria. Citations are classified as real, misattributed, invented or not findable. The source says the answers were graded blind by several Claude agents, and that public answers, reference answers and citation checks are published for review. Meyer discloses that Thorsten Meyer AI runs Lexassay and that a Claude model ranks first, a potential conflict readers should consider when interpreting the results.
The leaderboard is not the only available comparison. Meyer cites a separate Vals AI legal-agent benchmark in which models work with supplied documents: Mistral Large 4 ranks eighth there, ahead of Claude Opus 5.5 in 21st place and GPT-6 Astra in 24th. The differing results reflect different test designs and do not establish which system is best across legal tasks.
““It is a spot check, not a benchmark in the strict sense: one answer per model and task, so a few points of difference are noise.””
— Thorsten Meyer, Lexassay creator
What the Scores Cannot Establish
The results do not show how consistently each model would answer the same question across repeated attempts, because Lexassay records one answer per model configuration and task. The source describes the exercise as a spot check, and the published material does not establish performance across a larger set of real client matters or workflows involving lawyers and research tools.
The test also measures answers without internet access or tools. It is not clear from the reported results how much retrieval from official legal sources would change scores, or whether the same models would perform differently when given case documents. The conflicting positions in Lexassay and Vals AI rankings underline that benchmark results depend on task design.
Finally, Meyer says Claude agents graded the answers and discloses that a Claude model ranked first. The source says the grading was blind and that public materials are available for checking, but the reported results alone do not independently resolve how grading choices affected scores.
Verification Before Legal Use
The source material does not announce a date for a new Lexassay run or a response from Z.ai about the score. Meyer says the test’s method, public answers, reference answers and citation checks are available for scrutiny; readers can use those materials to examine how the published grades were reached.
For anyone using AI on a legal question, the immediate next step is to check cited statutes and rulings against official sources and confirm time-sensitive deadlines with a lawyer. The reported test offers no assurance that a model will catch a jurisdiction-specific holiday, apply current law or flag a false premise in an individual case.
Key Questions
What score did GLM-5.3 FlashX Legal receive?
It scored 8 out of 100 on Lexassay’s 15-task test, according to test creator Thorsten Meyer. He says it fully solved none of the tasks and gave no answer on two.
What did the test measure?
Models answered 15 questions about German, US and EU law without internet access or tools. The questions included deadlines, legal changes, court decisions and false premises; answers and citations were graded against reference materials.
Does the result prove the model is unreliable in every legal setting?
No. Meyer describes Lexassay as a spot check with one answer per model and task. It does not establish performance across repeated runs, real legal matters or workflows that include research tools and human review.
Why can a legal deadline depend on the city?
In Meyer’s example, the cases were heard in Munich and Berlin, and Corpus Christi is a public holiday in Bavaria but not Berlin. The reported appeal deadline therefore fell on different dates. Readers should verify actual deadlines with a lawyer.
How should readers treat AI legal answers?
Meyer recommends using them as a starting point, checking each cited law or ruling at its source, and asking a lawyer when a deadline, money or an authority is involved. The article says it is not legal advice.
Source: Thorsten Meyer AI
