📊 Full opportunity report: Minerva. The opposite path. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
Italy’s Minerva project trained a large, open-source LLM from scratch with 50% Italian data, outperforming multilingual models but scoring poorly on Italian exams. This reveals scaling challenges in sovereign-LLM development.
Italy’s Minerva-3B, a large open-source language model trained from scratch on 2.5 trillion tokens, scored only 4.9% on the INVALSI Italian school-exam benchmark, highlighting significant challenges in achieving country-specific language understanding despite massive investment.
The Minerva project, led by Sapienza University of Rome and funded through Italy’s national AI strategy, involved training models with approximately 50% Italian content on a dataset of 2.5 trillion tokens. Despite outperforming comparable multilingual models on Italian benchmarks, Minerva-3B’s performance on the INVALSI exams was near chance, at just 4.9%, a surprising and revealing result.
Researchers emphasize that the overall dataset size and parameter scale are critical for complex language tasks. The results suggest that even large-scale native-language training may not suffice at current parameter levels, raising questions about the required investment to develop truly country-knowledgeable AI models.
Minerva.
The opposite
path.
Italy spent years building a European sovereign LLM from scratch. Then Minerva-3B scored 4.9% on the INVALSI Italian school exam.
Where AMÁLIA layered Portuguese specialization onto a multilingual foundation, Minerva trained from scratch on 2.5 trillion tokens with approximately 50% Italian content. Where AMÁLIA’s weights are not yet public, Minerva published weights, training data, and code as truly-open from day one. By every institutional measure, the Italian approach worked. But the empirical results contain a finding the press coverage has been quiet about — and it has implications that extend well beyond Italy.
Same problem. Opposite path.
European sovereign-LLM development has two primary architectural approaches. Italy chose from scratch with substantial native-language foundation. Portugal chose continuation pre-training of a multilingual model. The structural comparison surfaces what each commitment actually requires operationally.
The comparison is not “Italy did it better than Portugal.” Both projects respond to the same structural problem with different architectural strategies under different institutional and economic constraints. Italy’s national-AI investment is structurally larger by an order of magnitude — and Minerva is the visible artifact of that scale.
Large Language Models (LLMs)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
4.9% on INVALSI. The bitter lesson surfaces.
In June 2024, researchers evaluated Minerva-3B on the Italian school-exam benchmark. The result was unambiguous. This is not a critique of Minerva — it is a critique of the public discourse around what Minerva’s empirical results actually demonstrate.
AI Systems Performance Engineering: Optimizing Model Training and Inference Workloads with GPUs, CUDA, and PyTorch
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
350M to 7B. Four parameter scales, one architecture.
The Minerva model family covers four parameter tiers, each with specific training corpora. Each scale level reveals what the from-scratch path actually requires at different operating points.
Italian + English
100B English
~50% English
+ 200B code
AI Engineering: Building Applications with Foundation Models
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three answers. Same question.
Minerva, AMÁLIA, and OpenEuroLLM represent the three operational answers to the European sovereign-LLM question. Each makes different architectural and institutional bets. The strategic discourse benefits from treating all three as data points in the same empirical experiment.
Mastering Small Language Models: A Practical Guide to Building Lightweight NLP Systems with Python, Transformers, and Quantization Techniques
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three standards the movement should adopt.
The structural critique generalizes beyond Minerva. The European sovereign-LLM movement benefits from internalizing these lessons across every subsequent national project. Italy modeled the openness standard; the movement should adopt it as norm.
Minerva is one valid answer to the European sovereign-LLM question. AMÁLIA is another. OpenEuroLLM is potentially a third. The strategic discourse benefits from treating all three as data points in the same empirical experiment rather than as competing national-prestige projects. More analysis like this is needed. Not less.
Implications of Minerva’s Performance on Sovereign-LLM Strategies
The findings challenge the assumption that larger native-language datasets and models automatically yield deeper country-specific knowledge. For European nations investing heavily in sovereign AI, this underscores the need to consider not just scale but also the quality and scope of training data and architecture. The results imply that current investments may be insufficient to produce models with meaningful academic or practical understanding of national languages and content, impacting future policy and research directions.
Background on Italy’s Sovereign-Language Model Development
Italy’s Minerva project represents a major effort to develop a European sovereign LLM from scratch, trained on a dataset of 2.5 trillion tokens, roughly half Italian, utilizing Italy’s top supercomputing infrastructure and a dedicated research team. Unlike approaches that adapt multilingual models via continuation training, Minerva was built from the ground up, with open weights and data released publicly. The project aims to demonstrate the feasibility of national AI infrastructure and to produce models tailored to Italian language and content.
Previous European efforts, such as Portugal’s AMÁLIA, layered specialization onto multilingual foundations, whereas Minerva chose a comprehensive native-language training path. Despite technical successes, early empirical results reveal significant limitations in achieving high-level academic and language understanding, raising broader questions about the scalability and investment needed for effective sovereign AI models.
Unresolved Questions About Scaling and Data Quality
It remains unclear what the precise scale and data quality thresholds are for achieving meaningful country-specific knowledge in LLMs. The current results from Minerva suggest that even with massive datasets and parameters, models may still fall short of expected performance on complex, real-world tasks. Further research is needed to determine whether alternative architectures, training methods, or data curation can bridge this gap.
Next Steps in European Sovereign-LLM Development
The Minerva team plans to continue iterating on their models, including ongoing experiments with continual training and larger parameter scales. Future benchmarks and real-world tests will help clarify whether scaling alone can overcome current limitations. European policymakers and researchers will need to reassess investment strategies, balancing scale with data quality and architectural innovation to develop truly effective national AI models.
Key Questions
Why did Minerva perform poorly on the Italian exams despite large-scale training?
The results suggest that dataset size and parameters alone are insufficient; the quality, diversity, and relevance of training data, along with model architecture, are critical factors in achieving high-level language understanding.
How does Minerva compare to other European sovereign LLM projects?
Unlike Portugal’s AMÁLIA, which layered specialization onto multilingual models, Minerva was trained from scratch with a focus on native Italian content, demonstrating impressive technical scope but facing similar performance challenges on complex tasks.
What are the implications for European AI policy?
The findings indicate that simply scaling native-language datasets and models may not be enough; strategic investments must also prioritize data quality, architecture, and task-specific training to build effective country-specific AI systems.
Will future models improve performance on academic benchmarks?
Future iterations with increased scale, refined training methods, and better data curation are expected, but it remains uncertain whether these will fully overcome current limitations.
What does this mean for the broader European sovereign-LLM movement?
The results highlight the need for a nuanced understanding of scale versus quality, suggesting that European projects may need to rethink their strategies to achieve truly effective national AI models.
Source: ThorstenMeyerAI.com
