TL;DR
Alex Wang’s essay explains linear layers, MLPs and self-attention as different forms of associative memory, comparing how well each can store and retrieve key-value pairs. Under the post’s stated assumptions, it estimates capacity at about 64 memories for a linear store, 2,700 for an MLP and more than a million for attention with 64-dimensional keys.
Alex Wang has published an explanatory essay that recasts linear layers, multilayer perceptrons and self-attention as different kinds of associative memory. Using a measure of interference during recall, Wang estimates that with 64-dimensional keys, a basic linear store can hold about 64 memories, an MLP about 2,700, and self-attention more than a million under the post’s assumptions.
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
Wang’s central question is how models retrieve information after it has been stored. The essay describes a memory as a key-value pair: a key is used to look up information, while its paired value is the information to retrieve. A query recalls a value based on how similar the query is to the stored keys. In technical literature, Wang notes, this arrangement is called associative memory.
The simplest example is a linear layer whose weights accumulate an outer product for each key-value pair. When queried, the layer returns a weighted sum of stored values. The target value contributes the desired signal, but other memories also contribute when their keys resemble the query. Wang calls this added contribution interference and defines capacity as the number of memories that can be stored before interference becomes as large as the target signal.
For normalized random keys and values, Wang’s back-of-the-envelope estimate puts a linear store’s capacity at roughly the key dimension: about 64 memories for 64-dimensional keys. The post then presents MLPs as a way to reduce interference and raise capacity, estimating about 2,700 memories, before describing attention as a store with strong recall and potentially unbounded capacity. Its estimate of more than one million memories for attention also assumes 64-dimensional keys. These are theoretical heuristics in the essay, not reported measurements of a particular model.
A Practical Lens on Model Memory
The framework offers readers a way to connect neural-network design with a familiar trade-off: storing information is not enough if the system cannot retrieve the intended item reliably. Wang’s focus on signal versus interference makes that trade-off explicit and gives a shared vocabulary for comparing layers that are often taught as separate mathematical operations.
The estimates may help explain why attention is useful when a model needs to select relevant information from many stored items. But the figures do not establish how many facts a deployed language model can actually remember. Real models involve learned representations, finite precision, training effects and other components not captured by the simplified calculation. The essay is best read as an intuition-building analysis, rather than a benchmark or a direct measurement of model knowledge.
Products Worth Considering
Apple AirPods Pro 3 Wireless Earbuds with Active Noise Cancellation
Bose QuietComfort Wireless Over-Ear Noise-Cancelling Headphones, Sandstone
Apple Watch Series 12 GPS 42mm Smartwatch
As an affiliate, we earn on qualifying purchases.
From Weight Updates to Recall
Wang begins from the everyday observation that language models can answer questions about many subjects, then asks how memory might work inside their components. In the post’s simplified setup, each stored item has a key vector and a value vector. A system is judged not just by how many pairs it can encode, but by whether a query for one pair can recover its value without other pairs overwhelming it.
The essay also links memory storage to training: for a linear layer, a gradient-descent update changes the weights by an outer product of an input and a gradient-derived value. Wang presents this as a way to see learning updates as writing memories into a layer. The supplied excerpt develops the linear-store calculation and introduces the broader comparison with MLPs and attention; it does not include the full derivations for those later capacity estimates.
“The key property that characterizes a “good” memory store is its ability to recall what it stored.”
— Alex Wang, in the essay
Limits of the Capacity Estimates
The supplied source does not state the essay’s publication date, and it does not provide experimental validation for the headline capacity figures. The linear-layer estimate is derived under simplifying assumptions about normalized, random keys and values; actual model representations may not follow those assumptions. The excerpt also does not show the detailed derivations behind the MLP and attention estimates, so their precise conditions and limits cannot be fully assessed from the material provided.
It is also unclear how these estimates translate into usable factual recall in a trained language model. A theoretical store’s capacity is not the same as the number of facts a model can answer correctly, retain across prompts or retrieve under real-world conditions.
Testing the Memory-Store View
The next step for readers is to examine the full essay’s derivations and assumptions, particularly for the MLP and attention comparisons. Empirical tests could then check whether measured recall and interference in trained networks follow the proposed capacity patterns. The supplied material announces no such experiment, follow-up publication or benchmark, so any confirmation beyond the essay’s analysis remains outstanding.
Key Questions
What is the essay’s main idea?
It treats linear layers, MLPs and self-attention as memory stores that associate keys with values, then compares how reliably each can retrieve stored information.
What does capacity mean in this analysis?
Wang defines capacity as the number of stored memories a system can hold before interference from other memories becomes as large as the signal from the item being queried.
Are the capacity figures measured in a language model?
No such measurement is reported in the supplied material. The numbers are estimates from a theoretical analysis under stated simplifying assumptions, not a benchmark of a deployed model.
Why does the essay give attention a much higher estimate?
Wang describes self-attention as having very strong recall and potentially unbounded storage, and estimates more than a million memories with 64-dimensional keys. The supplied excerpt does not include the detailed derivation, so the estimate’s full conditions cannot be checked here.
Source: hn
