Can 32GB Of RAM Power MoE Inference For 180B Parameters In Transformers?
AIThis post was created with the assistance of artificial intelligence (AI).

🔍 Read the full analysis: Can 32GB Of RAM Power MoE Inference For 180B Parameters In Transformers? on ThorstenMeyerAI.com

Buying for a business?Offer from Amazon

Get business pricing on tech for your team

  • Business-only prices and quantity discounts
  • Tax-exempt purchasing
  • Multiple users, one account, clear invoices
As an affiliate, we earn on qualifying purchases.

TL;DR

A Hugging Face developer has proposed an experimental Transformers feature that streams selected mixture-of-experts weights from storage into a bounded cache. In an OLMoE test, it reached 34.6 tokens per second, near the reported 38.1-token baseline; results for a 180B model on a 32 GB Mac are not available.

A Hugging Face developer has proposed an experimental SSD-backed expert-streaming option for Transformers, reporting 34.6 tokens per second on an OLMoE test, compared with 38.1 for the standard Transformers path, in the original analysis. The work aims to run mixture-of-experts models without keeping all their weights in fast memory, but the source does not report a completed test of a 180B-parameter model on a Mac with 32 GB of unified memory.

The approach targets mixture-of-experts (MoE) models, which route each token through a subset of a model’s experts rather than using every expert for every token. The developer says experts make up more than 90% of the model weights. That structure may allow an inference system to keep selected experts close to the processor while fetching others from RAM or storage when needed.

In the reported experiment, the developer first tested OLMoE-1B-7B with a basic Python loop that ran experts individually. That version produced 10.1 tokens per second even when the experts were already cached, compared with about 32 for vanilla Transformers. The developer then used a shared, preallocated device tensor and a table mapping expert IDs to cache slots. This let Transformers’ existing grouped matrix-multiplication kernel process selected experts together, bringing the reported rate to 34.6 tokens per second, against 38.1 for the baseline.

The developer says the change was submitted to Transformers as pull request #50717, which adds an option to the from_pretrained flow. The source also points to a Hugging Face Space, tarekziade/moe-expert-streaming, with project code. It describes implementation work across Transformers, a model-specific package and Metal kernels, and says AI tools were used to build code; the author says their contribution included choosing experiments, reviewing profiles and deciding what to retain.

At a glance
reportWhen: Experimental implementation reported; p…
The developmentA developer proposed SSD-backed expert streaming for Transformers and reported near-baseline OLMoE speed, while the larger 180B-on-32-GB-Mac test remains unreported.
AI Inference · Briefing

Streaming MoE Experts With Limited Memory

An experimental Transformers change streams selected experts from SSD into a bounded cache. The reported OLMoE test ran close to vanilla speed, but the 180B model on a 32 GB Mac remains untested in the source.

34.6 tokens/s
OLMoE with expert streaming
38.1 tokens/s
Vanilla Transformers comparison
90%+
Model weights attributed to experts

How the cache works

The router selects experts for each token; the experiment fetches those experts into a shared cache.

1
Route each token

Choose a subset of experts for the token.

2
Load selected experts

Stream them from storage into available device cache slots.

3
Compute together

Use Transformers’ grouped matrix multiplication kernel.

What the test shows

On OLMoE-1B-7B, the shared cache kept the reported speed near the standard path.

Earlier approach
  • Python loop ran experts one at a time.
  • Reached 10.1 tokens per second, even with experts cached.
Shared cache
  • Preallocated tensor mapped expert IDs to cache slots.
  • Reached 34.6 tokens per second versus 38.1 baseline.

The larger device test is open

The 180B model on a 32 GB Mac is a motivating target, not a measured result in the supplied source.

Evidence

Promising engineering

The cache design brought the reported small-model test close to vanilla Transformers speed.

Open question

SSD performance

Storage traffic and cache misses could change speed when experts must be fetched repeatedly.

Next evidence

Measure the target setup

Report speed and memory use on the larger model, with storage, cache settings and workload specified.

StatusThe change was submitted as Transformers pull request #50717; the source does not say whether it was merged or released.
The takeaway
Expert streaming looks close to baseline on the reported OLMoE test, while performance on the 180B model and 32 GB Mac remains unproven.

The Promise of Smaller Memory

If the technique performs reliably on models that exceed available memory, it could make some large MoE models usable on laptops or single-GPU systems that cannot hold every expert in memory. That could broaden local inference options for developers and researchers who face memory limits, while relying on the model’s routing pattern to load only the experts needed for a given token.

The reported speed result is encouraging but narrow: it shows that the shared-cache design ran close to the baseline in the stated OLMoE test. It does not establish how quickly the method works when experts must be fetched from an SSD during generation. Storage speed, cache capacity, routing changes and data movement could all affect performance. The practical question is not only whether the model fits, but whether expert transfers can keep up with token generation.

Products Worth Considering

Apple AirPods Pro 3 Wireless Earbuds with Active Noise Cancellation

Apple AirPods Pro 3 Wireless Earbuds with Active Noise Cancellation

As an affiliate, we earn on qualifying purchases.

How Expert Streaming Works

MoE inference systems manage a large set of experts even though each token uses only some of them. Approaches described in the source place experts in different tiers, including device memory, system RAM and disk. The proposed Transformers experiment focuses on keeping a bounded set of experts in a shared cache and retrieving selected experts as the router requests them, rather than processing experts one by one in Python.

The source also reports comparisons on a different setup: Qwen3.6-35B-A3B running on an A10G GPU with 24 GB of memory and ample system RAM. In that configuration, it says Strata reached 82–88 tokens per second with Q8_0, while llama.cpp with Q8_0 and its fit option reached 63–64. These figures belong to that model and hardware setup; they do not predict performance on a 32 GB Mac. The source characterizes other offloading approaches differently, but gives no same-device comparison between them and the new Transformers experiment.

“34.6 tokens per second, compared with 38.1 for vanilla Transformers.”

— The developer, describing the OLMoE test

The 32 GB Mac Test

The source identifies a 180B-parameter model on a 32 GB Mac as a motivating use case, but does not provide a measured result for that model or device. It also does not specify the storage type, cache size, prompt conditions or repeated-run results for the OLMoE comparison, limiting how much can be inferred about performance across workloads.

It remains unclear whether pull request #50717 has been accepted, merged or included in a release. The report also does not show how the cache performs when routing shifts rapidly or compare the new implementation with Strata on matching hardware and a matching model. Near-baseline speed on the reported OLMoE test does not by itself show that SSD reads will remain fast enough for a much larger model.

Results Needed for the Larger Model

The next milestone is the status of pull request #50717 and whether the experimental option becomes available in a published Transformers release. The linked Hugging Face Space gives readers a place to inspect the project code, according to the source, but does not substitute for a full benchmark.

To test the use case behind the work, future results would need to report generation speed and memory use for the intended 180B model on a 32 GB Mac, along with the storage device, cache settings, workload and test conditions. Repeated runs and a comparison with alternatives on the same hardware would help show whether the approach remains responsive when expert weights must be fetched during inference. Until those results are reported, the larger-device claim remains a target rather than a demonstrated outcome.

Key Questions

Did the experiment run a 180B model on a 32 GB Mac?

No such benchmark is reported in the source. That setup is described as a motivating target, not a completed test result.

What speed did the Transformers experiment report?

The developer reported 34.6 tokens per second on OLMoE, compared with 38.1 for the standard Transformers path. The source does not provide enough test details to generalize that result to other models or devices.

How does expert streaming aim to reduce memory use?

It keeps a bounded set of selected MoE experts in a cache and fetches other experts from storage as needed, rather than keeping all expert weights in fast memory at once.

Is the feature already part of Transformers?

The developer says the change was submitted as pull request #50717. The source does not say whether it was accepted, merged or released.

Does the reported OLMoE speed prove SSD streaming is fast on larger models?

No. The result shows near-baseline speed in the reported OLMoE test, but the source does not establish comparable performance when a larger model must fetch experts from an SSD during generation.

Primary source: Hugging Face · via ThorstenMeyerAI.com

You May Also Like

Will ChatGPT Be #2 Free App In The US Apple App Store On September 4?

A new Polymarket market suggests ChatGPT could become the second most downloaded free app in the US Apple App Store on September 4, with a 50% probability implied.

9 AI Startups To Watch In 2026

Discover the nine most promising AI startups shaping the future in 2026, their innovations, and potential impact on industry and society.

Portfolio. The synthesis.

A comprehensive analysis of six European institutional AI projects reveals a strategic framework for compliance ahead of August 2, 2026 enforcement under the EU AI Act.

How Small Language Models Change Edge Deployment Strategy

How small language models revolutionize edge deployment strategies by enabling faster, more private, and resource-efficient AI solutions that transform industries and user experiences.