Slipstream is an open-source fork of llama.cpp that streams a Mixture-of-Experts model's expert weights from your SSD on demand, instead of loading the whole model into RAM. This lets a 16–36 GB Apple-Silicon Mac run 35B–480B MoE coding models locally.

What hardware do I need?

An Apple-Silicon Mac (M-series) with 16 GB of RAM or more, and a fast SSD. Larger XL models (240–466 GB on disk) need a big, fast external SSD; the interactive 35B model needs about 20 GB of disk.

A 35B MoE coder runs at ~13–19 tokens/sec on a 36 GB Mac. A 118B model that does not fit in RAM runs at ~2.8 tokens/sec, which is usable for batch coding. Speed scales with SSD bandwidth and available RAM cache.

Is it free and private?

Yes. Slipstream is MIT-licensed and local inference stays on your machine. Optional remote mesh inference is off by default and encrypted in transit, but the selected worker decrypts and sees the plaintext prompt to run inference. Sensitive and secret work stays local by default.

Local · Private · Apple Silicon

Run coding models bigger than your Mac's RAM

Slipstream is a llama.cpp fork that streams Mixture-of-Experts weights from your SSD. A 16–36 GB Mac runs 35B–480B MoE coders locally, and stays usable while it does it.

↓ Download for macOS View on GitHub

Free · MIT · engine bundled, nothing to compile

expert cache · live resident hitstreaming from SSD

Illustrative of the app's live view. Green cells are experts resident in the RAM cache, amber cells are streaming from SSD. In the app it's drawn from your actual hit-rate.

480Blargest MoE that runs

16 GBminimum Mac RAM

3.9×optimization speedup

100%local by default

New in v0.3.4

Every click lands, and the app says what it is doing.

The four first-run path fields now focus when you click their label, the tab strip reports which tab is current, and every message — including the first-run “choose a model folder” error — is announced instead of silent. Found by driving the real interface, not by reading the source.

Nothing about the engines changed in this release. The v0.3.3 ad-hoc bundle seal and the measured v0.3.2 gains remain: llama.cpp caps prompt cache RAM at 512 MiB and warms the exact tool schema only when enabled, cutting first visible tool TTFT from 23.36 s to 4.73 s without swap growth. Slipstream remains pure llama.cpp/PGRN + oMLX/PGRN with no Ollama runtime.

01 — The core finding

I moved one file, the streamed experts, onto the fastest disk. That single change was 2.7×. Not a clever kernel. Storage placement.

A MoE model activates only a few experts per token, so most weights sit idle. Slipstream keeps the always-needed weights resident and streams the routed experts off the SSD as they come up, into a bounded cache sized against your RAM. It refuses to load anything that would swap your Mac.

02 — How it works

An engine designed around one bottleneck: SSD fetch.

Decode is ~78–92% fetch-bound. Every mechanism below exists to keep the Mac responsive and hide that fetch behind compute.

pread + F_NOCACHE

SSD expert streaming

A binary sidecar (PGRN) holds the stacked experts. Reads bypass the OS page cache, so it never balloons and fights the RAM budget.

CLOCK-LRU-K

Layer-partitioned arena

One bounded cache tier per layer. Cross-layer eviction is impossible by construction, so streaming behaviour stays predictable.

admission gate

Memory-health guard

The engine refuses a cache that would push the Mac into swap. "The Mac stays usable" is a hard invariant, not a hope.

MTP · DFlash

Speculative decoding

Drafts several tokens per target pass: MTP for Qwen, a DFlash draft model for Laguna. Each draft amortizes the expert fetches behind it.

PGCT1 · PGCC1

Async prefetch

A background thread warms the next layer's experts while the current one computes. Wired end-to-end; the predictor is active research.

NMSE = 0

Parity gate

A test asserts streamed output is bit-identical to fully-resident output. Optimizations that only reorder eviction can't change results.

03 — Measured, on a 36 GB Mac

Measured on my own machine, losses included.

Real runs, with the methodology written down. No hand-waved 5× miracle.

Qwen3.6-35B-A3B (Q4), the interactive workhorse

RAM cache	Decode	Cache hit-rate	Fits
2 GiB	5.5 tok/s	21%	16 GB Mac
10 GiB	~13 tok/s	78%	36 GB, interactive
14 GiB	~19 tok/s	86%	mostly-idle Mac

Prefill of a ~30k-token agent prompt: 75 → 208 tok/s (2.7×) with ubatch 2048 + parallel I/O threads.

Laguna S 2.1 (118B-A8B, Q4), larger than 36 GB of RAM

Configuration (each row adds one lever)	Decode	vs. baseline
PGRN on external USB SSD, no draft	0.72 tok/s	1.0×
PGRN on internal NVMe (storage split)	1.95 tok/s	2.7×
+ DFlash speculative draft	2.36 tok/s	3.3×
+ larger cache	2.83 tok/s	3.9×

A model that doesn't fit in RAM, made runnable. At ~2.8 tok/s that's batch coding, not chat.

What to expect on your Mac

Your Mac	Sweet-spot model	Experience
16 GB	Qwen3.6-35B-A3B · small cache	runnable, more SSD reads
24–36 GB	35B · 10–14 GiB cache	interactive · ~13–19 tok/s
36 GB + fast SSD	118B (Laguna) for hard tasks	batch quality · ~2.8 tok/s
0.5–2 TB fast NVMe	XL tier, up to 480B	demo / batch coding

This is guidance. The app works out the real settings and an expected speed for your exact machine. Got real numbers? share them →

04 — What did not work

The negatives, recorded honestly.

The rejected experiments say more about an engine than the wins do. So does going back and re-checking a rejection at the cache size it was meant for.

One verdict flipped. I rejected the compact / zero-copy expert path at a 2 GiB cache, which is a size almost nobody runs and where there's nothing to save. Re-measured across the cache sizes the app actually recommends, it's +13–24% faster (peak +24% at 6 GiB), swap-safe, and bit-exact. It's on by default now. The lesson I kept: re-measure a rejection at the size it was meant for before believing it.

A win, but only where it earns it. For structured output (JSON / tool calls) the grammar itself is a draft source: wherever it forces the next character, I feed that in as a pre-accepted, target-verified token — fewer model forwards, so fewer expert fetches off the SSD. It's lossless (byte-identical output) and, on a streamed 118B with rigid schemas, +45% tok/s. An adaptive guard keeps it neutral on easy JSON the draft model already predicts well, so it's on by default. Honest scope: it helps grammar-constrained output, not free-form code — a targeted win, not a universal one.

OS page-cache instead of the bounded arena unsafe
~75% faster, but 1000+ swapouts. It breaks “the Mac stays usable.” Rejected.
Speculative prefetch predictors −8%
Static + online co-activation, at real cache sizes. Prediction isn’t accurate enough to beat the cost of the speculative reads on a fetch-bound path. The async machinery is wired; a predictor that wins is still open research.
HOT/WARM tier reservation −1…−6%
Even on top of compact. Pinning “hot” experts shrinks the general working set → lower hit-rate. Pure LRU-K stays best.
Dual-SSD striping slower
tok/s-negative on internal NVMe + slow USB (shared bus). It's a genuine capacity feature for two fast disks, so it stays opt-in.
ANE / MLX compute paths wrong regime
Real 1.2–1.4× gains, but only for models that fit in RAM. They don’t help the >RAM streaming case, which is the whole point of Slipstream.

05 — Standing on the shoulders of

The research I read.

The design leans on published work in MoE offloading, flash-memory inference and speculative decoding.

2312.11514

LLM in a flash (Apple)

Efficient LLM inference with limited memory by streaming weights from flash. This is the idea Slipstream builds on for Apple Silicon.

2510.10302

SP-MoE

Speculative decoding and prefetching to accelerate MoE inference. Validates coupling draft tokens to expert prefetch.

2508.21706

Hiding Offloading Latency with Speculative Decoding

Uses the draft pass to hide expert-fetch latency, which is the lever behind the async prefetch trigger here.

2602.06036

DFlash: Block Diffusion for Flash Speculative Decoding

The DFlash speculative approach used for Laguna's draft model.

2401.15077

EAGLE

Feature-level speculative sampling. Reference for lossless draft-and-verify decoding.

2503.06823

eMoE

Task-aware, memory-efficient MoE inference under a tight memory budget.

2505.05950

FloE

On-the-fly MoE inference on memory-constrained hardware. A comparable offloading strategy.

2401.10774

Medusa

Multiple decoding heads for parallel token prediction. The lineage behind MTP.

06 — Get started

A 480B coding model.
On your MacBook.

The engine is bundled inside the app. Download, drag to Applications, open. Point your coding assistant at localhost and go.

↓ Download the .dmg Read the docs

Drag Slipstream.app to Applications. First launch: right-click → Open (not yet notarized).

Open it. It detects your Mac and proposes settings for it. Pick a model, click Start.

Point Kilo / Cline / Cursor / OpenCode at 127.0.0.1:8080, or use the one-click patch.

Requires an Apple-Silicon Mac, 16 GB+ RAM. macOS 11+.