TL;DR: You should probably not use this technique. You should definitely not use this model for programming; Qwen3.8-Flash-Next and Qwen3.8-27B are simply better in all the ways that matter other than openness. Right now the approach I outline below is bad. But it is fun & fascinating and I think I’ll keep at it: a working implementation of SeedLM, the Apple/Meta idea of storing LLM weights as seeds of a pseudo-random generator, running on an ordinary Mac CPU & GPU instead of an FPGA. I applied it to the experts of IFM/K2-Horizon-MoVA-36B-A4B. The result is a 36B mixture-of-experts model in 22.76 GB that fits a 32 GB Mac and decodes at about 50 tok/s on an M4 Max. At 26.53 GB, the same size as a 4-bit affine quant, the seed version has 16% lower KL divergence than that quant. But it’s also slower, and truly calibrated imatrix quants are probably better. Nevertheless, I thought it was an experiment worth doing! So the code & model are linked at the bottom.
Where this started
I first heard about SeedLM at Apple’s machine learning summit in the spring of 2025. I ended up in this session with a bunch of machine learning nerds talking way over my head about math I didn’t understand because, hey, I’m a high-school graduate with some college and self-taught for most of this compSci stuff. But one presentation in particular stuck with me, and for the past year and a half I’ve been noodling off and on with how I might create an open-source engine that demonstrates this alternative, pseudo-random-number-generator based approach toward model inference.
The pitch is simple, but a little bit crazy. Take a weight matrix. Cut it into blocks of 8 weights. For each block, find a 16-bit seed for a linear feedback shift register (LFSR), the cheap pseudo-random generator that lives on most hardware. Run the LFSR from that seed and you get a small random basis matrix. Store the seed, plus three or four tiny int4 coefficients and a shared exponent, and the block becomes “this random basis, times these coefficients.” At inference time you never load the weights. You regenerate them from the seed.
Why would anyone do this? Because LLM decode is memory-bound. The GPU sits idle waiting for weights to stream in from RAM. SeedLM trades memory reads for arithmetic: fewer bytes per weight, more ALU work to rebuild them. The paper shows it on an FPGA (an AMD Virtex-7) where the LFSR is literally a few flip-flops, and reports matrix-vector speed-ups that approach 4x. It is also data-free: no calibration set is needed to find the seeds.
What the paper does not show is that this works on hardware normal people own. Unified memory machines (Macs, Strix Halo, DGX Spark) live and die by memory capacity and bandwidth. That is exactly where “fewer bytes per weight” should matter.
My thought process is that if we can reduce the bandwidth requirement of an LLM, it would free up the GPU and memory bandwidth for other kinds of operations simultaneously. Think, for instance, ASR-LLM-TTS pipelines for interactive voice chat. Or running classifier models simultaneously against inputs and outputs for safety and privacy. That kind of thing.
So anyway, I started blasting.
Attempt one: failed prototype (spring and summer 2025)
My first NanoSeedLM was Python and MLX on TinyLlama and Qwen2.5-0.5B. I got very excited at the headline when I started working through the approach. My earliest results showed something like 23.3x compression, 777% better than published results.
I was stunned. This seemed ludicrous. It was.
- The block size was 65,536 weights, not 8. Of course it compressed 23x. It also destroyed the model.
- The dequantization divided by 2^e where it should have multiplied.
- And the kicker: chat worked by decompressing the seeds back to safetensors and loading those in mlx-lm. Zero memory saved at inference. The two major selling points of the technique — reduced RAM usage and reduced bandwidth requirements — gone.
I wrote up a post-mortem to teach myself more about how this worked with a fresh design in July 2025, with the paper’s corrected parameters (C=8, P=3) and a more realistic hill-climb: up to 8x compression (if the model started at FP32). But I got frustrated and moved on to other projects while this nibbled on the back of my mind, the design doc sitting on my NVMe for a bit while I dallied elsewhere.
Attempt two: C++ and a wall (July 2025)
The second NanoSeedLM was an optimistic C++20 effort with Clang vector extensions, GGUF and safetensors readers, Python bindings, and a little web demo. But I found Seed Searching to be, frankly, hella slow. So I only searched the first 1,000 of 65,535 seeds, ran the seed search on the CPU, and had a fairly relaxed error target just to try to get through the seed search in a timely fashion. A lot of the commit history from that week is CI and linting as I worked through the tortuous process of teaching myself better C++.
The real cause of death of this effort was my naive attempt to use the CPU for seed search, which made it way too slow for larger models. I had parallelised it as a full set of OS threads per 8-weight block, each doing its own least-squares solve. For a 7B model, that felt hopeless, and for a 36B MoE, I’d have been waiting until the heat death of the universe for it to finish. So I gave up on that attempt again in mid-July 2025. In December I wrote a brief to restart it on gpt-oss with MXFP4 — my favorite model at the time — but that waited in the wings while I went to work on other things, including eventually writing K2-Horizon support in oMLX from what I’d learned about how models work.
And of course, large language models were getting better the whole time. The capability of Opus 4.5 in December 2025 at long-horizon tasks and just getting stuff done was groundbreaking at the time, and I remember showing off to family & friends how easy it was to vibe-code dumb little games. I was blown away: these SOTA language models were finally capable of real work, and not just code completions or spellcheck.
What I changed: September 2026
Ten months later I came back with a different set of rules. I’d learned hard lessons. Like the king in the swamp in Monty Python and the Holy Grail, I’d built this castle twice and by gum, THIS TIME it still stands! The basic rules were focused on leveraging the GPU for the seed search as well as during inference. And this time I chose C instead of C++ or Python, because I simply found the Metal docs and Obj-C to be much easier to reason about for LFSR and PRNG than high-level abstractions. A few lessons:
- Search over blocks, not seeds. Each GPU thread owns a block and walks all 65,535 seeds. No thread-per-seed, no OS threads per block. The full 0.9B search went from hours on the CPU to minutes on the GPU.
- Bit-matched scalar reference. Every LFSR generator, Metal kernel, and search path equated to a plain C function. This helped me — and, by this time, Claude — catch a number of real bugs. Like the fact that Metal fuses multiply-adds unless I tell it not to (
#pragma clang fp contract(off)). Ugh. That sucked a lot of time. - Seed semantics from the paper. The seed is the initial register state. I’d woefully misunderstood how this actually could work in my prior attempts — my lack of a strong math background working against me — but this time I had a competent LLM buddy who could check my homework.
- Measure against BF16. Last year, I didn’t even know what “KL Divergence” was. But this time, as a cheap check I could verify the BF16 model’s logits on held-out text and on chat, plus a bunch of generated outputs judged blind by LLM judges. I’ve also gotten really hot about behavior-driven development, and it turns out modern LLMs are incredibly talented if given a specific numeric goal to achieve: they can arrange a host of experiments and run autonomously overnight trying them for marginal gains. Codex, Qwen, Claude, Kimi K3, IFM/K2-Horizon and GLM-5.3 all contributed enormously to the hill-climb!
- Compress what matters. In a mixture-of-experts model, the experts are most of the bytes, and each one is used rarely. That, I decided, is where I should focus on using seeds. Every test I ran trying to use 16-bit LFSR seeds for attention, routers, norms and embeddings was kind of a disaster, so those stayed in normal formats. I suspect there is a seed out there at higher bits for the LFSR that might better approximate those and do less damage. But given how long seed search takes even with a fairly optimized GPU approach, I’m not willing to spend the time searching that space 🙂
- Never decompress. This one was really tough to convince Codex and Claude to avoid. Both basically wanted to “rehydrate” the pseudo-random numbers as BF16 weights to “speed up” inference. SeedLM is not an idea that’s really got much weight in the training corpora, I suppose. Anyway, to get the savings right, weights are rebuilt inside the GPU kernels, in registers, every time.
Why K2-Horizon-MoVA
K2-Horizon-MoVA-36B-A4B is not a typical MoE. Besides 100 routed FFN experts (top 8) per layer, it has Mixture-of-Values attention: the value projection in attention is replaced by 64 routed “value experts” (top 4). There’s a softplus gate on the attention output, grouped RMSNorm, and sigmoid routers whose bias only selects experts. Of its 37.4B stored weights, the routed FFN experts alone are 71%. That’s a lot of rarely used matrices, which IMHO made it an ideal target for seeds.
IFM also vibes with my open-source leanings. They publish everything: the training data, the training code and every intermediate checkpoint. While working through seed search, it occurred to me that I could actually train a model toward seeds instead of fitting seeds after the fact.
Mind. Blown. IFM/K2-Horizon is a rare open-source AND open-weight family where I could do just that… assuming I have the budget for time on an actual GPU cluster for training.
I had a less technical reason too. The K2-Horizon family is, at present, kinda’ obscure. I wrote the oMLX support for the model family, so my thought process there is basically that there is little competition for PRs; my dumb little experiment will not break anything people depend on.
And I like the model’s “bedside manner”. It’s got that eager-puppy energy Anthropic lost from Claude in 4.6 and beyond. Unfortunately, it’s not nearly as competent as some of its competition, but for the purpose of this experiment I figured that’s OK. IFM/K2-Horizon is not as task-focused and matter-of-fact as Qwen, not as concerned with “policy” as gpt-oss, and much easier to experiment with than Gemma 4 when I have all the training data. I just wanted to prove NSLM could work on commodity hardware without an FPGA, and I’ve done that now.
The sprint
The hill-climb of optimization was a fun experiment in Karpathy’s auto-research idea. I’d finalized the skeleton of the project and basic approach in C by Sunday morning on October 4, and then language models together worked through optimizations and testing autonomously for 42 hours of wall time, all on one 16-inch MacBook Pro with an M4 Max and 128 GB.
My setup for these rounds of autoresearch had what I call the “peanut gallery” alongside it. Lunaroute.com generously donated a two-week trial, and I thoroughly abused glm-5.3 on pi to keep providing commentary to the primary model driving the research (Claude Opus 5.5 in Claude Code in a tmux session). Another window ran Codex with GPT-6 Astra, focused on reading the paper and associated research and keeping the coding agent on task to not “helpfully modify” SeedLM’s LFSR approach. It turns out that this is surprisingly challenging, as Claude Opus 5.5 would keep trying to pursue a local maximum on a dead-end path, and Astra kept reminding it after a certain number of failed trials that it should choose a different Seed-based approach to try to optimize so that I could select from a variety of approaches.
I thought at first that I’d need to rent some GPU time, but it turned out OK once I stopped trying to do the seed search on CPU. One machine did everything: seed searches, BF16 reference runs, KLD scoring, benchmarks and generations. The seed searches alone took 11.4 hours of compute. Only about 7.7 hours of that went into the results I kept. The rest was re-runs, pursuing local maxima, finding their peaks, and giving up to try another way. With KLD runs, judging runs and benchmarks on top, the laptop fan was howling in my office for most of those two days.
It turns out, I really missed my MacBook Pro that whole time, which taught me the limits of my tolerance of being without my handy sidekick-box. It sat on the desk, fans screaming, pretending to be a very expensive space heater for a hill-climbing loop, and I could not use it as a laptop. The irony was not lost on me that a piece of “consumer hardware” was not available while developing a project intended for consumer hardware.
Anyway, here’s a retracing of the hill-climbing efforts:
| Step | What | Outcome |
|---|---|---|
| 1 | K2-Horizon-0.9B, all weights as 4-bit seeds | Fail. KLD 0.38-0.44 against a Q4_K_M bar of 0.084 |
| 2 | GPU search, activation weighting, fast kernels | Search ~25x faster. Seed decode faster than BF16 on 0.9B. Quality still short |
| 3 | MoVA: seeds for expert gate/up only | Generations hold. KLD worse than Q4 |
| 4 | All routed experts as seeds, rest Q8 (24.87 GB) | Two coherence failures in 80 outputs |
| 5 | A C99 + Metal engine for MoVA, matched to MLX | Matches MLX quality. Faster than MLX for BF16 and Q8 |
| 6 | Kernel work on seed decode | Seeds 8-11% behind Q4 decode speed |
| 7 | 4.5-bit blocks with four coefficients (P=4) | Same size as Q4, lower KLD, all five rules pass |
| 8 | Value experts moved to Q4 | 22.76 GB, fits a 32 GB Mac, one prompt loops |
The Travelogue of Tragic Misunderstandings
“LFSR is free”
Yeah, this assumption on my part was kind of bullshit. Sure, on an FPGA, LFSR might be free. On a GPU it is not. To rebuild one weight, you step a 16-bit shift register several times, pull out bit fields, subtract an offset, multiply by an int4 coefficient, sum, and scale. My first bit-exact kernels spent 15-17 operations per weight. The plan had assumed 6-7. Meanwhile a 4-bit affine weight costs about two.
This is what I called the “ALU wall”: decode on Apple Silicon is memory-bound only as long as you do very little math per byte. Do enough math and you are compute-bound, and you have traded a memory problem for a compute problem.
This finding alone made me slightly regret returning the DGX Spark I bought in February. It’s profoundly memory-bandwidth-bound, but has GPU to spare. This also makes me think that even though my “target” for this project is unified-memory systems, commodity hardware like an RTX 5070 with DDR4 or DDR5 RAM might benefit, too, if it’s doing expert-offloading to RAM already. This experiment remains to be run, but I kinda’ suspect in such a tragically memory-speed-constrained environment that an optimized kernel on GPU might actually speed things up a bit under some circumstances.
Climbing the ALU wall was a challenge:
- Stream tables. The LFSR is linear over GF(2). The 24 or 32 bits that follow any seed are a linear function of the seed. So a 256 KB table (one 32-bit word per seed) replaces stepping the register. States become shifts of a 64-bit word.
- Two-operation states. Put the 16 state bits into the mantissa of 1.0 with one bit-insert. That gives
1 + state/65536as a float with no integer-to-float convert. Fold the “1” into a precomputed sum of the inputs, and fold the 2^16 into the exponent. - Convert-free coefficients.
(q ^ 8) - 8sign-extends a nibble without a convert.
After all that, the kernel stopped being ALU-bound and became load- and latency-bound. Removing the table lookup entirely saved 0%. But the 4-bit affine kernels got faster too: their throughput rose from 233 to 377 GB/s. The gap stayed at 8-11% for P=3 and 17-18% for P=4.
“Four bits is four bits”
The paper’s 4-bit setting is P=3: three coefficients per 8 weights, 4.0 bits per weight. Affine Q4 with group 64 is really 4.5 bits per weight once you count scales and biases. I misunderstood them — thinking they were the same thing — for longer than I care to admit. The whole peanut gallery struggled with this concept, and so did this user.
But eventually a data-free probe on 3,072 MoVA expert blocks made it clear:
| Format | Bits per weight | Relative error |
|---|---|---|
| Seeds, P=3 | 4.0 | 0.01467 |
| Seeds, P=3, 5-bit coefficients | 4.375 | 0.01116 |
| Affine Q4, group 64 | 4.5 | 0.00834 |
| Seeds, P=4 | 4.5 | 0.00692 |
At equal bits, four coefficients beat affine Q4 by 17% on reconstruction error. Once we thought to run that probe, it only took a few minutes but changed the project trajectory.
Found some B.S.: “Better KLD means better text”
Yeah, I’ve long suspected that KL Divergence is a poor substitute for benchmark results. It can often be deceiving. On the 0.9B model I tried GPTQ-style error feedback across seed blocks. KLD improved to 0.0659, the best number I had. But generations got worse: 9 coherence failures and 29 regressions in 80 outputs. KL divergence is a useful tool, sure. But it’s not the thing the LLM does; it’s a proxy for quality. After repeatedly measuring that lower KLD would often still result in worse text generations, I decided my Peanut Gallery would need to include various models testing every result in the project as blind judges.
“Speculative decoding is free speed”
Prompt-lookup decoding is lossless, and it helps dense models. On MoVA it gave x1.01 to x1.04: noise. Verifying a few draft tokens at once means routing all of them, and different tokens pick different experts. A 4-token verify touched 1.3-2.3x as many expert matrices as one token. For an MoE whose cost is “how many experts do I read,” that largely canceled the gain. As a result, I haven’t attempted to port UNO diffusion models to Seeds yet.
“GPU search == CPU search”
No, it certainly did not. The Metal search and the C search disagreed on 307 blocks. It wasn’t even a near-tie. The cause was fused multiply-add contraction and float-float arithmetic that did not reproduce double precision exactly. The eventual fix — contraction off, exact double emulation — cost 33% in search speed but had bit-identical results. Worth.
Smaller dead ends
- Calibration on chat text instead of web text? No real difference.
- Weighting the search by squared routing weights? Worse.
- Multi-row decode kernels, threadgroup LFSR tables, GQA-shared decode attention, more key splits? Indistinguishable from noise or slower.
- Prefill attention through threadgroup memory? This one seemed promising but got stuck at about 2 TFLOP/s. Meanwhile, register fragments with BF16 operands reached 7.4 TFLOP/s. :shrug:
- Four search workers were slower than two on my hardware. I think the GPU queue starved, so lesson learned about parallelism on my laptop!
Results
Apple M4 Max, using my engine instead of oMLX. Because apparently late 2026 is the season for everybody to build their own inference engine; who am I to buck the trend?
| Model | Size | KLD held-out / chat | Decode 1k / 4k ctx |
|---|---|---|---|
| BF16 | 74.89 GB | reference | 35.8 / 34.0 tok/s |
| Affine Q4 experts, Q8 rest | 26.53 GB | 0.0233 / 0.0110 | 60.2 / 55.9 tok/s |
| Seeds P=4 experts, Q8 rest | 26.53 GB | 0.0195 / 0.0095 | 49.5 / 46.5 tok/s |
| Seeds P=4, value experts Q4 | 22.76 GB | 0.0260 / 0.0126 | 50.9 / 48.0 tok/s |
Conclusions:
- Against BF16, NanoSeedLM is 3.3x smaller and much faster.
- Against affine Q4 at the same size, seeds give lower KLD, and in my blind-judged set, fewer output problems. But we’re paying a 15% to 20% decode tax using Seeds, and I’m not sure where to hill-climb to try to make this approach actually faster than affine Q4. It’s slower on my hardware, but doesn’t use as much memory bandwidth, instead maxing out the GPU.
- Against Q4 at the same quality, seeds give a smaller model. This is the real benefit, I suppose: The 22.76 GB version is the one that matters for a 32 GB Mac. macOS gives the GPU about 24 GB by default, and this model uses 23.3-23.9 GB in my engine up to about 4k of context. The Q4 model at 26.53 GB would not fit. If we were more aggressive about using seeds for layers that we currently don’t seed, we could bring it down further.
Which leads me to my nothingburger of a conclusion: If you have plenty of memory, use a normal quant and enjoy the speed. If you are right at the edge of what your machine can hold, seeds let you trade some speed for a few gigabytes less model size in RAM. Sometimes those gigabytes decide whether the model runs at all.
My recent experiments with the “Strata” model engine suggest that in marginal conditions, that trade might be worth making on systems with, say, 12 GB GPUs. Reducing a few gigabytes really matters when you don’t have all that many to work with, and thus would (in theory) result in a giant speedup compared to paging in experts from RAM, NVMe, or SSD for Flash-class models like Qwen3.8-Flash-Next.
Why calibrated quants (mostly) win
Since the paper came out, the quantization world has not stood still. Importance-matrix (imatrix) calibrated quantization is very good. In my tests it beats seeds under many conditions.
On the 0.9B model, an imatrix dynamic quant at 538.7 MB scored KLD 0.0833. My best activation-weighted seed model at 540 MB scored 0.0919. IFM’s own Q4_K_M, which as far as I can tell was made without an imatrix, set the bar at 0.0842. On MoVA I compared seeds only against plain affine Q4 without calibration. An imatrix-calibrated 4-bit MoVA would probably close the gap, and might pass seeds. But I’ve messed around with this long enough and I figure I should just release it so that for GPU-RAM-constrained and/or memory-bandwidth-constrained devices the LocalLlama community has something else to noodle on in our quest to run frontier models on our rusty old gaming PCs and unified-RAM DGX Sparks, AMD Strix Halo, or Macs.
So anyway, my conclusion is mixed. SeedLM on consumer GPUs works. It seems like it might be a niche tool for extremely memory-constrained (bandwidth/capacity) setups.
But it’s not a free lunch, and unless you fit into that niche, it is probably not the best 4-bit format for most people today.
Is anyone else doing this?
Eight months ago I searched and found nothing. Now there is a little more, but still almost nothing that runs inference from seeds:
- SeedLM: the paper. No official code.
- seedlm-repro: a NumPy replication that measures reconstruction error.
- SeedLM+O: seeds plus an outlier side channel; a measured negative result.
- Seed-Q: sensitivity-aware LFSR seed quantization, with ASIC synthesis.
- AWSRC: seeded residuals on top of an INT4 backbone. I’d like to dig into this a bit more. Q4 plus a small seeded correction could be interesting.
- PRANC and NOLA: whole networks and LoRA adapters as combinations of seeded random bases.
- QTIP and ik_llama.cpp’s trellis quants: weights computed by cheap integer hashes instead of looked up. They are the closest working relatives.
- The Hacker News thread from April 2025 has good arguments on both sides.
As far as I can find, mine seems like it might be the first open implementation that runs SeedLM inference inside an engine on a consumer GPU, on a mixture-of-experts model. If you know of another, tell me. I would love a little company in this very small subculture of people who for bizarre reasons want to experiment using pseudo-random seeds instead of weights 🙂
What’s next
- Seed-aware training. IFM published the data and the checkpoints. A short fine-tune that pulls the experts toward seed-representable weights sounds like fun if I could figure out how to fund the training time.
- Seeded residuals on Q4. The AWSRC direction: keep Q4, add a tiny seeded correction where it matters.
- Bandwidth-starved chips. The M4 Max has a lot of memory bandwidth, which is the worst case for seeds. A base M-series chip with a third of the bandwidth might tell a different story. Also to-do: apply NSLM to Qwen3.8-Flash-Next models using something like Strata as a base to see if, by further reducing VRAM load on the lone, tiny GPU, we could actually get a performance story out of SeedLM, not just a capability.
- Bigger models. The seed search time may be extreme, but I’d like to try this next against Flash-class models to see how well we can bring more of the “big boys” of the open-source LLM world down to VRAM and RAM configurations mere mortals can afford. Time to buy some time on a real GPU instead of doing it all on my Mac; I suspect these jobs that take hours should finish in a few minutes.
Try it
- Engine (C99 + Metal, MIT): github.com/mbarnson/nanoseedlm
- Model (22.76 GB, Apache-2.0 like the base): txgsync/K2-Horizon-MoVA-36B-A4B-NSLM-p4mx-q4v
- oMLX support: You can simply enable “Trust Remote Code” in the oMLX options panel. Since this is experimental I didn’t feel like putting in a pull request again for this model family.
git clone https://github.com/mbarnson/nanoseedlm && cd nanoseedlm && make
hf download txgsync/K2-Horizon-MoVA-36B-A4B-NSLM-p4mx-q4v --local-dir K2-NSLM
out/bin/nslm-chat --model K2-NSLM --effort low "Why do tide pools matter?"NanoSeedLM was built with help from a rotating cast of LLM coding assistants, which deserve a collective thank-you:
- Lunaroute.com for their generous free trial. The idea of priority queues and batch queues for a low flat rate is really appealing, and I’m sorely tempted to replace one of my AI subscriptions with their $99 tier.
- Claude.com because Opus 5.5 is peak at hill-climbing to a /goal.
- Openai.com because Astra 6 is the best freaking bug-hunter I can pay $500 a month for.
- Deepseek.com because DwarfStar4 and the new DeepSeek-4.1-flash model showed me how good custom inference engines can be on constrained hardware.
- Openrouter.com because it has all the models. Gemma-4-31B is really pleasant to talk to, and then I can run /model in pi to take that pleasant conversation context to a plan with a model that doesn’t fail basic tool calls.
- Apple.com for the SeedLM paper that inspired this year-and-a-half journey.
- The /r/Localllama community on Reddit for being a cadre of privacy and data sovereignty freaks dealing with the infestation of bots as best we can.