🔍 Read the full analysis: Why My Favorite AI Test Result Isn’t Running My News Pipeline on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Thorsten Meyer reports that a test of 71 model configurations for news writing found GPT-6 Astra produced the highest-scoring articles, while GPT-6 Luna offered a much cheaper production option. The evaluation also found failures in two fallback models, including invented quotes, which Meyer says were removed from the pipeline.
Thorsten Meyer says he tested 71 AI model configurations on real news-writing tasks and now uses GPT-6 Luna in his production pipeline rather than the higher-scoring GPT-6 Astra. In the evaluation, Astra scored best, but Meyer said its roughly $0.12-per-article cost was too high for routine news production; Luna cost about $0.0012 per article and delivered lower, but consistent, scores.
The evaluation ran from September 8 to October 13, 2026, using Meyer’s production prompt and real news sources. It covered API, subscription, free-tier and locally run open-weight models. Each source was used to produce an English and a German article, which were assessed by GPT-6.1 Sol and Claude Opus 5.5 on source fidelity, fabrication, clarity about uncertainty, news value, structure, polish and German language quality.
Meyer reports that GPT-6 Astra scored 4.00 out of 5 in both languages in the initial test, with perfect source-fidelity scores, and averaged 3.93 across a 12-article robustness run. GPT-6 Luna scored 3.67 in English and 3.68 in German in the initial test; it averaged 3.67 in the robustness run, with source fidelity of 4.71 out of 5. Meyer says every Luna output was valid under the pipeline’s existing repair step. Astra took about 50 seconds per article, compared with about 17 seconds for Luna.
The test also found that higher reasoning settings did not produce a reliable quality gain for Luna. At maximum effort, the reported score was 3.58 in English and 3.57 in German, versus 3.67 and 3.68 at the no-effort production setting. Maximum effort cost about 12 times as much and took about 12 times as long. Meyer also reports that two former fallbacks failed on source fidelity or output validity, and that he removed them from the route on October 7.
Infographic 01 · Who writes the best news article?
The best writer is not the one in production
71 model configurations, one real job: turn a news source into an article for an English and a German edition. GPT-6 Astra writes the best articles. GPT-6 Luna writes them for about a hundredth of the price.
Robustness run: 12 real news sources, English edition
Dot = mean score (1–5), bar = 95% bootstrap interval. German edition mean below each score. Overlapping bars are not a real difference.
Two judges, two caveats: GPT-6.1 Sol and Claude Opus 5.5 grade every article and were also tested as writers, so they partly grade themselves. Claude models ran with effort medium unless noted.
Source: Thorsten Meyer AI model-scout benchmark, 8 Sep – 13 Oct 2026. Task: write a news article from a real source with the production prompt, English and German editions. Judges: GPT-6.1 Sol (reasoning high) and Claude Opus 5.5, both with the source in context; score 1–5, mean of both judges. Robustness run: 2 test articles plus 10 more real sources, mostly short like production; mean with 95% bootstrap interval.
Powered by Thorsten Meyer AI
Cost and Reliability Shape the Choice
The results highlight a practical distinction between the highest-scoring model and the model a publisher can afford to run at volume. Meyer’s test puts Astra ahead on measured writing quality, but Luna’s much lower reported cost and faster completion made it his production choice. That trade-off matters to publishers weighing article quality against the expense and speed of generating many pieces.
The fallback findings point to a separate operational risk: a pipeline can produce unreliable or unusable articles when its primary model is unavailable. Meyer says one backup generated invented quotations, while another produced no valid articles in four attempts. A model’s reputation or broad benchmark rank, the report argues, does not establish that it can safely perform a specific editorial task.
These are results from one publisher’s setup, not a general ranking of all models or news operations. Still, the reported method—testing actual prompts and source material, including short sources, and checking whether outputs pass production validation—offers a direct way to examine whether a model fits a particular workflow.
Products Worth Considering
Amazon Kindle Scribe (3rd Generation), 64GB, 11-inch Display, Graphite, with Premium Pen
Apple AirPods Pro (2nd Generation) Wireless Earbuds with USB‑C Charging, Up to 2X More Active Noise Cancelling, Transparency Mode, Adaptive
Apple AirPods Pro (1st Generation) with MagSafe Charging Case
As an affiliate, we earn on qualifying purchases.
A Test Built Around Production Articles
Meyer says his benchmark used two source lengths: a 6,340-character product filing and a 619-character political news item. He describes short-source writing as especially representative of his production work. The leading configurations were also tested on 10 additional real sources, mostly short ones, in what he calls a robustness run.
Two AI judges scored the articles, and Meyer reports 95% bootstrap intervals for the robustness results. He cautions that with only two articles in the initial test, score gaps below about 0.3 points could be noise, and treats the larger robustness run as a more dependable ranking. He reports that Astra’s interval, 3.83–4.02, did not overlap Luna’s, 3.59–3.76, in that run.
The comparison also illustrates why performance on one task may not carry over to another. Meyer says Claude Opus 5.5, which he uses for software development and describes as a leading coding model on a separate benchmark, scored 3.46 in English and 3.39 in German in this news-writing test. Its robustness score was 3.52. Those figures are Meyer’s evaluation results, not an independent assessment of the coding benchmark.
“A general intelligence score cannot answer the question I actually have to answer: which model should write the news articles my sites publish every day?”
— Thorsten Meyer, in the October 13 report
Limits of the Reported Results
The findings come from one publisher’s prompt, sources, languages and validation process. The source material does not provide the full article set, scoring sheets or judge-by-judge results, so readers cannot independently verify every reported score from the information presented here. AI judges supplied the ratings; the report does not describe a separate human-editor scoring panel.
The initial comparison used only two source items, while the follow-up included 10 more. Meyer says the latter is more reliable, but that remains a limited sample of production material. The report also does not establish whether the same cost, speed and quality results would hold for other prompts, subject areas, model versions or languages. The stated costs and model availability are tied to the test period and could change.
Meyer reports that GPT-4.1 nano and gpt-oss-120b were removed from the pipeline on October 7. The supplied material does not detail the full current failover configuration, beyond stating that Luna now runs first and then repeating Luna as its fallback. It is not clear whether that repeated name reflects a deliberate setup or a reporting error.
Further Testing Would Check Durability
The report describes a production choice already made: GPT-6 Luna remains the first model in the news route, while the two identified fallbacks were removed on October 7. Meyer says the production setting uses no additional reasoning effort, based on the cost and timing results and the absence of a clear quality advantage at higher settings.
The next useful evidence would be continued testing on a larger, varied set of real sources, especially short items, with transparent output samples and more detail on how failures and corrections are counted. It would also help to see how the selected model performs over time as source types, model versions and prices change. The report does not give a date for a further evaluation or announce a replacement fallback.
Key Questions
Which model scored highest in the test?
GPT-6 Astra scored 4.00 out of 5 in both English and German in the initial test. It averaged 3.93 across the 12-article robustness run, according to Meyer.
Why did Meyer choose GPT-6 Luna for production?
Meyer says Luna’s reported quality was consistent enough for his needs while costing about $0.0012 per article, compared with about $0.12 for Astra. Luna also took about 17 seconds per article, versus about 50 seconds for Astra.
Did higher reasoning effort improve Luna’s articles?
Not reliably in the reported test. Meyer says the score differences across effort settings were within the noise, while maximum effort cost and took about 12 times more than the no-effort setting.
What went wrong with the former fallback models?
Meyer reports that GPT-4.1 nano scored 1.00 out of 5 for source fidelity and produced invented quotes. He says locally run gpt-oss-120b produced zero valid articles out of four and also invented attributed quotes. Both were removed from the news route on October 7.
Do these results show which AI model is best for all newsrooms?
No. The results reflect one publisher’s prompts, sources, languages and pipeline checks. They show how models performed in that particular evaluation; other newsrooms would need to test models against their own editorial requirements and production conditions.
Source: ThorstenMeyerAI.com
