FoveatedKV

https://github.com/samfurr/foveated_kv

I’ve been running local LLMs on my Mac a lot lately, and I kept hitting the same wall. The bottleneck is memory bandwidth. Every decode step has to read the entire KV cache from memory. At long context on an 8GB machine, that’s gigabytes moving per token. The model is mostly just sitting there waiting for data.

The usual fixes are to either evict tokens (fast, but you’re literally deleting information and breaking the softmax denominator) or quantize everything uniformly (which treats the answer to your question with the same precision as filler text). Both bugged me. Attention focuses. On any given step, a handful of tokens get nearly all the weight. So why store the unimportant ones at the same precision?

The VR headset thing

This got me thinking about foveated rendering. VR headsets figured this out a long time ago. Your eye can only resolve detail at the center of your gaze, maybe 2° of your visual field. Everything else is peripheral blur. So they render the center sharp and blur the rest. You save 50-70% of GPU work and nobody can tell the difference.

I applied the same idea to LLM attention:

| VR Foveated Rendering | FoveatedKV | | Eye position | Query vector | | Sharp center of gaze | Near tier: top 10% of tokens at fp16 | | Blurry periphery | Far tier: bottom 90% at fp8 keys + INT4 values | | Full-res frame buffer on disk | NVMe mmap archive of the fp16 originals | | Eye tracker detects gaze shift | Metal kernel spike detection (falls out of softmax for free) | | Re-render at new gaze point | C++ worker promotes exact fp16 into near tier |

The important bit: every token still contributes to attention. The softmax denominator stays correct. Nothing gets evicted. The only approximation is quantization noise on tokens the model isn’t paying attention to, and that noise is bounded, non-accumulating, and recoverable.

For the far tier I use fp8 E4M3 for keys and INT4 for values. Multiple recent papers (KIVI, LeanKV, KV-AdaQuant, AsymKV) have independently found that keys need higher precision than values because key error gets amplified through exp() in softmax. My ablation confirms it. INT4 keys produce 34x more error than fp8.

Making it fast

The quantization strategy is established prior work. Where I spent most of my time was making it actually fast on Apple Silicon. I wrote a fused split-K Metal kernel that handles both precision tiers plus the decode buffer in a single GPU dispatch. The query gets pre-scaled by 1/√D at load time. fp8 decoding uses a 256-entry LUT in threadgroup memory, so one shared-memory read instead of 10+ ALU ops per element. Score-gated value loading skips INT4 dequant entirely when a token’s score falls below exp(−16) (basically zero contribution). Online softmax computes each exp() exactly once.

Spike detection falls right out of the softmax bookkeeping with no extra work. When the kernel notices a far-tier token scoring higher than near-tier tokens, a C++ background worker reads the exact fp16 original from an NVMe-backed mmap archive and swaps it into the near tier. Next kernel dispatch, it’s just an ordinary full-precision token. Raw spike rate is high (~95% per head-layer slot), but aggressive filtering (cooldown, dedup, budget cap, GQA dedup) brings the effective promotion rate down to about 3.1%.

In a multi-fact needle-in-haystack retrieval test, promotion recovered 24 additional facts out of 64, bringing accuracy from 50% all the way up to 88%. That matches the full fp16 baseline exactly. Pretty satisfying.

Results

Everything benchmarked on my M2 with 8GB. Three model families: Qwen2.5-0.5B, Qwen2.5-7B, and Llama-3.2-1B (all 4-bit weight quantized).

• 2.3x Faster 7B decode at 512 context on 8GB. Compressed cache reduces memory pressure
• 0.995+ Cosine fidelity vs exact fp16 attention, stable from 512 to 32K
• 2.0x Memory compression at 10% near tier, consistent across all context lengths
• 88% Multi-fact retrieval with promotion, matching full fp16 baseline

The really fun result was on the 7B model. On 8GB hardware, FoveatedKV is actually faster than standard because the compressed cache reduces memory pressure enough to make the whole pipeline flow better. Standard OOMs at around 3K context. Foveated keeps going.

| CONTEXT | FP16 SDPA | FUSED KERNEL | SPEEDUP | | 1K | 1.12 ms | 0.94 ms | 1.19x | | 4K | 2.33 ms | 1.46 ms | 1.60x | | 8K | 4.05 ms | 2.22 ms | 1.82x | | 16K | 7.68 ms | 3.73 ms | 2.06x | | 32K | 15.72 ms | 6.81 ms | 2.31x |

Fused kernel vs Apple SDPA, 7B GQA shapes (H_kv=8, H_q=32, D=128). Break-even around 512 tokens.

What’s next

The benchmarks so far are all on 8GB Apple Silicon, which is the most constrained and most interesting target. The obvious next steps are a llama.cpp head-to-head, full-length LongBench runs on 16GB+ hardware, and scaling validation on bigger models. The promotion pipeline also opens up some interesting possibilities around speculative caching. Predicting which tokens will spike before they actually do.

The code, benchmarks, and a draft technical paper are all on GitHub. 73 tests, 8 benchmark suites, MIT licensed. Built with significant Claude assistance on the iteration and benchmarking side.