🔍 Read the full analysis: Testing Weight-Space AI At Home: Legal Scores Remained At Zero on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
A home experiment edited the weights of the local model Ornith 1.5 35B-A3B to reduce repetitive looping. On the 10 published Lexassay tasks, it answered all 10 instead of four in the comparison setup, but its legal score remained zero. The report says the experiment was small and the edited answers contained 55 wrong or invented citations.
A home test of a weight edit helped the local AI model Ornith 1.5 35B-A3B finish all 10 published Lexassay legal benchmark tasks, up from four in the comparison setup, but its legal score remained zero. The experiment’s author said the edit reduced looping rather than adding legal knowledge; the 10 edited answers contained 55 wrong or invented citations.
The test was inspired by a report from four MIT CSAIL researchers about “weight-space programming,” in which an AI system edits another model’s weights directly. The home experiment used a different, simpler method: a Claude Opus 5.5 session wrote code to run Ornith, identify a repetitive internal direction, and remove that direction at half strength from the output weights of 27 of the model’s 40 layers. The report says there was no additional training, no new legal data and no model reading the weights as text.
On the same setup used for the edited run, the unedited model answered 4 of 10 tasks; the edited model answered 10 of 10. Yet the report says the edited answers met only 7 of 50 checked criteria, compared with three before editing, and the legal score stayed at zero. The author also reports that, using the model card’s recommended sampling settings without any weight edit, the unchanged model answered nine of 10 tasks on its original setup.
The published figures cover 10 Lexassay tasks, not the benchmark’s full question set. Lexassay includes 15 questions on German, US and EU law, according to the report; five are kept hidden. The runs, answers, grades and citation checks are publicly available on Lexassay.com. The article says an OpenAI GPT-6 Astra session in Codex reviewed the code, results and claims and identified faults. It was a review of the experiment, not a second competing test.
Infographic 01 · What we changed
One loop direction, weakened at half strength
Inspired by the MIT post, but a simpler, classic technique (directional ablation): code written by a Claude Opus 5.5 session measured one loop direction by running the model and projected it out of the weights. No training, and no AI reading weights as text.
The six steps
- Write test material that is not the benchmark: 24 legal development questions plus 8 general controls, 0 overlap with benchmark prompts
- Find the true loop onset at token level: from there, at least 90% of the next 1,500 positions repeat earlier 32-token sequences
- Run the model and measure one loop direction per layer: average internal signal inside the loop minus the reasoning just before it
- Weaken it in the weights:
W ← W − α·r·(rᵀW)on the MLP output matrices of all 256 experts and the shared expert - Bake an edited copy; the original files stay untouched and the edited weights stay internal
- Benchmark original and edit on the same local setup, graded blind by three Claude agents; figures cover the 10 published tasks
Quick test: share of pure copying
Restart 20 tokens before a known loop onset, 8 cases, 3,000 tokens. Lower is better.
The cost of the edit
- Full strength: 23 of 24 legal questions finished, but control answers broke: tokens dropped out of short formulas, and month counting went wrong
- Half strength: 23 of 24 finished, 9 of 9 checked control facts right, and one German control question was answered in Chinese once
- Before the edit: 8 of 24 legal development questions did not finish; all 8 controls did
One run per setting. The loop detector was revised three times during the experiment, so the development numbers come from a moving yardstick. The direction is a statistical description, not a single “loop neuron”.
Sources: our experiment records, 4–5 Nov 2026 (executing Claude Opus 5.5 session; review by GPT-6 Astra in Codex); Ornith-1.5-35B-A3B model card.
Powered by Thorsten Meyer AI
More Finished Answers, Same Legal Score
The result illustrates a distinction that matters when evaluating AI tools: completion is not correctness. A model that stops repeating itself may produce more responses, but that does not mean those responses are legally reliable. In this test, the rise in completed tasks did not translate into a passing legal score, and the reported citation errors point to a substantial accuracy problem.
That distinction is especially relevant for legal use, where a fabricated or incorrect authority can undermine an otherwise plausible answer. The report describes Lexassay as checking citations against primary sources and says the edited answers included 55 wrong or invented citations. The experiment offers no evidence that the edited model is suitable for legal advice or professional work; the source explicitly says the article is not legal advice.
The findings are also a reminder to compare an intervention with appropriate baselines. The model answered nine tasks with recommended sampling settings and no edit on its original setup, while it answered only four in the separate comparison setup. Because those setups differ, the figures cannot by themselves establish how much of the apparent gain came from the weight edit rather than configuration.
Products Worth Considering
Logitech MX Mechanical Wireless Illuminated Keyboard, Tactile Quiet, Graphite
Logitech MX Ergo S Advanced Wireless Trackball Mouse USB-C – Graphite
Samsung Galaxy Tab A11+ 8GB RAM, 256GB Storage, Gray
As an affiliate, we earn on qualifying purchases.
A Looping Model Meets Lexassay
The experiment followed a report by Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab of MIT CSAIL. Their work described an AI reasoning about another network’s weights and changing them without a conventional training loop. In a Qwen example, the researchers reported that a model’s looping stopped and its math score rose from 7.9% to 48.1%. Their stated limit was that an edit might remove an obstacle to an existing ability but could not supply an ability the model lacked.
The home test did not reproduce that direct weight-reading approach. It used directional ablation, a known technique that records internal signals during model runs and removes a calculated direction. The source says this approach has been used in research, including work on removing refusal behavior. Here, the target was repetitive output from Ornith, a mixture-of-experts model whose publisher describes it as built for coding and agent work, rather than law.
According to the report, Ornith scored 1 out of 100 in the benchmark and two of its answers on the published tasks did not finish before a 62,000-token output limit. The author says the final portions of those cut-off transcripts showed substantial repetition. Other tested local models either did not loop or were not suitable for this test, so Ornith was selected as the candidate for an edit aimed at looping.
“The model finishes is not the same as the answer is right.”
— Thorsten Meyer, author of the home experiment
Limits of the Home Test
The evidence is limited to one run per setting and the 10 published tasks. The source says the loop detector changed during the experiment, the development set was small, and three graders came from the same model family. GPT-6 Astra’s independent review found faults, according to the author, but the report does not provide a separate repeated evaluation that would show whether the results are stable.
The comparison also has a configuration caveat: the unedited model answered eight of 10 tasks in its published run but four of 10 on the setup used for the edited comparison. A separate run with recommended sampling settings answered nine of 10 without an edit. The source says these local setups are not directly comparable, so the contribution of the weight change cannot be cleanly isolated from the available figures.
It remains unclear whether the edit would have the same effect across repeated runs, other models, different legal questions or the five hidden benchmark tasks. The report provides no evidence that the change improves legal accuracy beyond the small increase in checked criteria, and its zero legal score means it did not demonstrate a useful legal capability.
Public Runs and Further Testing
The experiment’s answers, grading and citation checks are available on Lexassay.com, alongside the original, sampling-settings, edited-model and edit-control runs, according to the report. Readers can inspect the public task results, but those records do not replace further testing across more runs and the hidden tasks.
The next useful evidence would include repeated trials under matched settings, a stable loop-detection method and evaluation on the benchmark’s hidden questions. Until such results are available, the reported outcome supports a narrow conclusion: this edit was associated with more completed answers in one comparison, while the legal score remained zero.
Key Questions
What changed after the weight edit?
The edited Ornith model answered 10 of 10 published tasks, compared with four in the same comparison setup before editing. Its legal score stayed at zero.
Did the edit teach the model law?
The report gives no evidence that it did. The edit targeted repetitive looping, and the legal score remained zero; the edited answers contained 55 wrong or invented citations.
Was this the same method used by the MIT researchers?
No. The home test used code to measure an internal direction during model runs and remove it from some weights. The source says the MIT work involved an AI reasoning directly about another model’s weights.
How broad was the test?
The reported figures cover 10 published Lexassay tasks and one run per setting. Lexassay has 15 questions, with five kept hidden, according to the report.
Can the edited model be relied on for legal advice?
The experiment does not establish that. The report says the legal score remained zero and explicitly states that the article is not legal advice.
Source: ThorstenMeyerAI.com
