Skip to content

Experiment/Melasistema, Luca Visciola

S-MoE

Seismic Mixture of Experts

A model-agnostic inference engine for fine-grained Mixture-of-Experts LLMs on Apple Silicon. It streams experts from NVMe, so the model never has to fit in RAM. A 235B runs from 32 GB up.

Get startedRead the docs
235B
parameters, streamed from a laptop SSD
32 GB
of unified memory is where the frontier starts
0 bytes
of RAM held by the ~117 GB expert vault

01/The Monolithic Delusion

Silicon Valley has a bucket problem.

To run a frontier model, the industry insists you must hold the entire mathematical mountain in RAM at once. For Qwen3-235B that is 235 billion parameters, about 470 GB at bfloat16. Rent the cloud, or buy a workstation that costs more than a car.

But in a Mixture of Experts, 95% of the model is silent at any given millisecond. Only a handful of experts fire for each word. The rest is cold stone, and standard runtimes keep all of it in memory anyway. Just in case.

470 GB
what the monolith demands in RAM
~5%
what actually fires for a token

What if we only load what fires, and load it before it fires?

02/The Phonon Metaphor

You cannot see inside the rock. So listen to it.

In 2022, Ing. Filippo Biondi published a way to image the interior of the Great Pyramid of Giza from orbit. Radar cannot penetrate stone. But when the pulse strikes the surface it generates phonons, acoustic vibrations that travel through the rock, and the surface answers with sub-nanometre movements a satellite can measure.

S-MoE is a philosophical translation of that intuition. Stop fighting the barrier. Measure what the barrier does to the signal.

SeismicS-MoE
Deep rock strataThe expert weights, cold on the NVMe SSD
EM surface pulseThe dense backbone running the current token
Generated phononsThe router gates’ exact activation echo
Acoustic receiversThe async I/O streamer and its retained ring
Subsurface imageThe generated token

What S-MoE borrows is the physical principle; the method and the genius are his. Read the paper

03/The architecture

Shatter the mountain.

One offline command slices a monolithic checkpoint into a self-describing vault. From then on, memory is a transient execution medium, not a storage pool.

  1. 01The resident backbone

    In memory

    Embeddings, attention, norms and routing gates stay in unified memory: about 16 GB for the 235B. At every layer of every token it evaluates the real router gate, one small matrix multiply that names the exact experts to stream.

  2. 02The deep strata

    On the SSD

    Every routed expert, shattered into 4-bit binary segments aligned to Apple Silicon’s 16 KB page boundary. The whole vault rests cold on NVMe and occupies zero bytes of RAM.

  3. 03The acoustic receiver

    In between

    Background workers read the named experts with Direct I/O, bypassing the OS page cache, into a lock-free ring a fraction of a second before the Metal GPU runs them. The ring doubles as an LRU cache.

Shatter
make shatter MODEL=./checkpoints/qwen3-235b-instruct OUT=./vault

04/Exact routing, streamed

Summoned, not loaded.

Inference is a streaming problem, not a loading problem. At every layer of every token, the backbone evaluates the real router gate on the true hidden state. It is a measurement, not a prediction, so it cannot be wrong.

The experts it names are pulled from NVMe into the ring a fraction of a second before the GPU runs them. Adjacent tokens share 46.4% of their experts, so nearly half are already there and are never read again.

token
0001
layer
01/30
ring hits
46%
streamed
0

Readout of the animation behind this page, not of a running engine.

05/Current status

235B, streamed from a laptop SSD.

No cloud. No H100. No half-terabyte workstation. Qwen3-235B at 4-bit, measured on a standard 48 GB Apple Silicon MacBook, since the latency campaign began.

Cold time to first token

>3×

92.7 s~26 s

Decode speed

>3.8×

0.48 tok/s1.84 tok/s

~14 s
warm follow-up turn, flat however long the conversation
~16 GB
resident: the dense backbone, not the whole model
46.4%
of experts reused by the next token, already in the ring
~150 GB/s
dequant bandwidth through coalesced Metal kernels

Every step verified bit-exact or coherence-checked against the reference path. 32 GB is the practical floor for the 235B, because its dense backbone alone weighs about 16 GB. More RAM buys speed, never quality.

06/Model-agnostic

One engine. Any fine-grained MoE.

The 235B is the proof of force, not the whole story. Point the shatter script at a compatible checkpoint and the model’s own mathematics travel inside the vault, in a 128-byte arch block. No recompilation, no per-model configuration. Vaults trade places live, from any model picker.

The frontier

Verified

Qwen3-235B-A22B

Vault (Q4)
~117 GB, streamed from SSD
Decode
1.84 tok/s
Topology
94 MoE layers × 128 experts

The daily driver

Verified

Qwen3-30B-A3B

Vault (Q4)
~14 GB, fits almost entirely in the ring
Decode
~13.7 tok/s
Topology
48 MoE layers × 128 experts

The next summit

Next

GLM MoE 477B

Vault
Not shattered yet
Decode
Unknown
Topology
Read from the vault, like the others

Same engine, same binary, same laptop. Supported models and hardware projections

07/Multi-protocol server

It speaks every protocol that matters.

One file, one process, one port, zero new dependencies. Anything that speaks OpenAI or Anthropic talks to the vault fleet on your laptop, unmodified: Open WebUI, LibreChat, editor plugins, both SDKs, Claude Code. 

.venv/bin/python serve.py --port 8000

# then open http://127.0.0.1:8000/ and start typing
  • POST
    /v1/chat/completions

    OpenAI Chat Completions, streaming or not

  • POST
    /v1/messages

    Anthropic Messages, streaming or not

  • GET
    /v1/models

    Every vault on disk. Name another to switch engines

  • GET
    /

    Built-in, dependency-free web chat console

08/The manifesto

Speed is a privilege.
Intelligence is not.

A 32 GB MacBook and a 512 GB Mac Pro receive the same 235 billion parameters. The same knowledge, the same reasoning depth. One waits less, one waits more. Both get the answer.

We are not opposing technology. We are opposing artificial scarcity. The barrier is not physical. It is architectural: the assumption, never questioned, that inference is a loading problem.

The honest amendment

The original claim said 16 GB. Reality audited it: the frontier’s dense backbone alone weighs about 16 GB, so the floor for the 235B moved to 32. A 16 GB Mac runs the same architecture with smaller fine-grained MoE models. We publish the audit, because refusing to correct yourself is their habit, not ours.

The rebel constraints

  1. 01

    Zero runtime heap allocations

    malloc, new and std::vector::resize are illegal inside the token loop. Every buffer is carved at startup.

  2. 02

    Direct I/O only

    F_NOCACHE on every file descriptor. SSD to DMA to unified memory to GPU. No copies, no page cache.

  3. 03

    Atomic synchronisation only

    No mutexes, no condition variables. The I/O thread and the GPU thread cannot block each other.

  4. 04

    Decoupled routing

    The gates measure, the streamer moves bytes, the kernel multiplies. None knows what the others do.

Read the manifesto

09/A curious builder’s diary

The mistakes are public too.

S-MoE is an experiment, not a product. It was built by a self-taught web developer and an AI agent: the direction came from human curiosity, the implementation depth from the collaboration. Every chapter is written down, including the ideas that got fired.

  1. 01Rock, Paper, SiliconHow a web developer used a satellite hack and an AI agent to ask a ridiculous question about frontier AI.
  2. 02“Hello, World!”A 235-billion-parameter frontier model just spoke on a 48 GB MacBook.
  3. 03The Mountain Learns to ConverseTeaching a 235B model not to keep you waiting.
  4. 04The End of ProphecyHow we doubled the speed of a 235B model.
  5. 05The Mountain Predicts ItselfWe fired our best idea, and the model got faster and smarter.
  6. 06The Mountain RunsI don’t know if we’re chasing something or being chased, but the speed won’t stop climbing.
  7. 07S-MoE Started to Speak OpenAIPoint any chat app at your MacBook and watch a 235B model think.
  8. 08The Mountain ExhalesRiding the 30B wind before the 477B avalanche.

10/Begin

The mountain is already vibrating. You just have to listen.

Clone and bootstrap
git clone https://github.com/melasistema/s-moe.git
cd s-moe
make setup
  • Apple Silicon
  • macOS 14+
  • 32 GB+ for the 235B
  • 16 GB for smaller vaults
  • ~150 GB free NVMe
Setup guideGitHubDocumentation