Post

Five kernels against SGLang: my Muse Spark Open Kernel Challenge

Five kernels against SGLang: my Muse Spark Open Kernel Challenge

For two weeks (22 October to 4 November 2026) I took part in the kernel track of the Muse Spark Open Kernel Challenge. The task was to write GPU kernels for one decoder layer of Muse Spark Open and beat SGLang’s own kernels on an H200.

TL;DR

  • The whole decoder layer ran 1.75× faster than SGLang (22.62 ms → 12.91 ms per step).
  • MoE was my best board at 2.03× (13.45 ms → 6.63 ms).
  • All five boards finished between 1.26× and 2.03×, for rank 691 of 2,014 overall.
  • That took 665 judged runs, all remote on their H200.
  • The biggest single lesson: one night auditing the hot paths I already had beat a week of new kernel designs.

How the judge worked

Each problem was one piece of the model, as a single GPU of an 8×H200 server runs it.

  • Submission. One Python file per problem with a kernel_fn(data, state) entry point. Inline CUDA was allowed, and an untimed prepare() step could pack weights or tune kernels ahead of time. Anything SGLang ships (Triton, CUTLASS, FlashInfer, DeepGEMM) was fair game.
  • Correctness. Every run used fresh random inputs, checked against an FP32 reference within a tolerance. Lower precision and lossy weights were banned, so every speedup had to come from doing the same math faster.
  • Timing. My kernel and SGLang’s took turns on the same GPU with the same inputs. Each turn ran as a CUDA graph, the way SGLang runs in production, and the L2 cache was flushed between turns so neither side benefited from data the other left behind.
  • Score. Each problem was timed under several loads: plain decode with 1, 16 and 32 users, and speculative verify, where each user checks an 8-token draft in one step (8, 128 and 256 tokens). The score is the geometric mean of the speedups across those loads, so each load counts equally.

Results

ProblemSGLangMineSpeedup
Whole layer (muse_decode_layer)22.62 ms12.91 ms1.75×
MoE (muse_moe_w4a16)13.45 ms6.63 ms2.03×
Attention (muse_attention_decode)3.77 ms2.06 ms1.83×
Dense projections (muse_bf16_proj)4.83 ms3.85 ms1.26×
Prefill layer (muse_prefill_layer)387.4 ms300.2 ms1.29×

Times are the judge’s time per step, averaged over all loads. Being honest about where that lands: it’s the middle of the pack. I beat the field median on MoE and dense projections and fell below it on the other three. The leaders show how much was left.

Speedup over SGLang, against the field
mine field median board leader

The five problems

Pick a problem to see which part of the layer it covers and how the speedup splits across loads.

Dense projections (muse_bf16_proj, 1.26×)

The layer’s five weight matmuls (x @ W.T): the qkv_gate, o, pooler and unpooler projections in BF16, plus the router in FP32, at 1 to 256 tokens per step. With few tokens there is almost no math, so the GPU spends its time streaming weights from memory. The only way past cuBLAS is to read fewer bytes.

  • 12-bit lossless weights. Most BF16 weights fall in a narrow range of exponents, so each one fits in its low byte plus a 4-bit sign-and-exponent code. The ~0.01% that don’t fit go in a separate correction list. That’s 25% fewer bytes with bit-identical output.
  • gv5, a GEMV kernel that unpacks the 12-bit stream straight into tensor-core (mma.sync) inputs. With no H200 to test on, I first checked the unpacking bit-for-bit with a CPU emulation.
  • The router as two BF16 halves (hi + lo), so the FP32 router can use the fast BF16 tensor cores and still pass the FP32 accuracy check: 1.4–2.6× on every load.
  • A wgmma kernel for large batches. From m ≥ 128 the work becomes compute-bound and cuBLAS is hard to beat. Mine only matched it, and that’s where the leaders found their extra 0.4×.

Attention decode (muse_attention_decode, 1.83×)

Each new token attends over an FP8 KV cache of about 80K tokens. Sliding-window layers read only the most recent part of that cache; full layers read all of it. Both are timed in plain decode and in 8-token speculative verify.

  • swa, a fused sliding-window kernel. A cluster of thread blocks splits the window and merges their partial softmax results in shared memory instead of going through HBM. Batch-1 decode went from 25.0 to 10.5 µs.
  • attn12 for full layers. It splits the long context across many CTAs with FP8 wgmma, and a small combine kernel merges the partial results.
  • A hot-path audit. I swapped the library exp2f for the hardware ex2.approx, sized ring buffers as powers of two so every % became a bit mask, and moved setup out of the inner loop. That alone took the board from 1.57× to 1.71×.
  • A per-shape tuner in prepare() that tests my kernels against FA3 (SGLang’s kernel) and keeps the fastest one that passes an FP32 check.

MoE (muse_moe_w4a16, 2.03×)

256 experts with NVFP4 (4-bit) weights, split across 8 GPUs so each holds 32. Every token goes to its top 8 experts, so each step means: route tokens to experts, run two GEMMs per expert, then combine the results.

  • moe4 runs that pipeline (route → GEMM1 → GEMM2 → finalize), decoding FP4 to BF16 exactly as the weights stream in.
  • Independent warps. Each warp keeps its own cp.async load queue and never waits on the rest of its block. From batch 32 up, the kernel runs as fast as the weight loads alone, which is the memory limit.
  • Early GEMM2 loads. GEMM2’s weights don’t depend on GEMM1’s output, so they start loading before GEMM1 finishes, saving 2–7 µs per call.
  • A per-batch config tuner, plus a faster routing kernel (6.0 → 4.3 µs).

Whole decoder layer (muse_decode_layer, 1.75×)

The full layer: the projections, attention and MoE above, plus the RMSNorms and residual adds between them. My kernels from the other problems are its parts; the rest of the win comes from shrinking the gaps between kernels.

  • Reusing the kernels above (gv5, swa/attn12, moe4), each picked per shape.
  • Fused boundaries. I ported SGLang’s faster norm-residual kernel (4.5 → 2.5 µs) and folded one RMSNorm into GEMM1, so it runs while the input loads.
  • Overlap. Expert routing runs on a side stream next to the pooler, and PDL (programmatic dependent launch) starts each kernel before the previous one finishes, hiding its setup.
  • A race I introduced. One early-launched kernel read its input before the previous kernel had finished writing it. Tests still passed because the buffer held identical values from the last call. I fixed it and confirmed the score hadn’t depended on it.

Prefill layer (muse_prefill_layer, 1.29×)

The same layer on whole prompts: 753-token turns and 16K-token first turns. With this many tokens the MoE GEMMs dominate. I set it aside after day one and only came back on day 8.

  • moe15, a grouped wgmma GEMM that decodes each expert’s weights once per call and reuses them for all of its tokens. It replaced Marlin, which had been about 65% of the time.
  • At 16K tokens, dequantize all experts to BF16 once, then run DeepGEMM’s grouped GEMM.
  • Custom routing in place of PyTorch’s argsort and scatter: the MoE part at 16K went from 5.5 to 5.0 ms.

Two weeks of progress

The best judged score so far on each board. The shaded band marks the hot-path audit. Before it came a week of new kernel designs that mostly didn’t beat what I had.

Each point is a judged run: both scored runs, which post to the board, and unscored checks, which test code without posting. Run-to-run noise is about ±2%, so a curve can end slightly above its final board score.

Where the time goes

This is one decoder layer at batch 1 (a single user), on a sliding-window layer with 80K tokens of context. For every shape, the judge also reports a speed of light: the time it would take just to read every byte the step needs at full H200 memory bandwidth. A single-user decode step is almost all weight reads, so the distance from that line is the honest measure of what’s left.

streaming weights (projections, experts) other kernels
Kernel group (from the judge’s profile)Time
Dense projections (qkv_gate, o, pooler, unpooler)39.0 µs
MoE GEMMs (this GPU’s experts)26.3 µs
Expert routing + finalize (sort tokens to experts, combine their outputs)8.4 µs
Router matmul (scores every expert for each token)7.0 µs
Attention (swa)6.1 µs
Norm + residual5.1 µs

The kernels add up to about 92 µs, yet the measured step is 86 µs: with PDL, each kernel starts its setup (reading parameters, issuing its first memory loads) while the previous one is still finishing. About three quarters of the time goes to streaming weights. The rest is a dozen kernels of only a few µs each, where launch overhead and the time to get memory requests flowing cost as much as the data itself.

Lessons

  1. Audit before you redesign. Small inefficiencies in the inner loop cost more than I expected: a library exp2f where the hardware ex2.approx would do, a % or / on every ring-buffer index, and accumulators indexed at runtime that spilled to local memory. Fixing them in one night took attention from 1.57× to 1.71×, more than a week of new kernel designs had.
  2. The noise is ±2%. The same code scored anywhere from 1.247 to 1.268 on different runs, while most late ideas were worth about 1%. Judge a change by per-shape microseconds over several runs, never by one score.
  3. Bandwidth needs bytes in flight. Memory latency is high, so full bandwidth means keeping many reads outstanding at once. One 8-warp thread block per SM streamed only 2.3–2.6 TB/s. Giving each warp its own cp.async or TMA load queue, with no block-wide barriers holding it back, reached 4+ TB/s.
  4. PDL is free speed until it’s a race. PDL lets a kernel start before the previous one finishes, but it must not read that kernel’s output before griddepcontrol.wait. If it does, reused buffers can still hold the right values from the last call, so every test passes anyway.
  5. Clusters can’t span GPCs. The H200’s 132 SMs are grouped into 8 GPCs of 16–17, and a thread-block cluster must fit inside one GPC. A grid of 33 four-block clusters left the last one waiting for a second wave, which doubled the kernel time.
  6. One crash loses the whole run. An illegal memory access in any candidate the autotuner tries corrupts the CUDA context, and then even the fallback kernels fail. A tuner protects you from slow candidates, not broken ones.

Ideas that didn’t make it

IdeaWhat happened
TMA multicast of xLoad the input once per cluster and broadcast it to every block, to cut memory traffic. It worked, but each multicast cost ~0.3 µs: +50% time at cluster size 2, +150–200% at 4–8.
12-bit weights for wgmmaUnpack 12-bit weights into shared memory so the faster wgmma instructions could use them at large batches. The unpacked tiles left too little shared memory for the load pipeline: 27–31 µs vs 20–22 µs for plain BF16.
Self-routing MoEBuild the expert routing table inside every GEMM block, to skip one kernel launch. Rebuilding it in every block cost more than the launch saved: 73.4 vs 70.8 µs.
Fused attention mergeMerge the partial results from the split context inside the same kernel, using a grid-wide barrier, to drop the separate combine kernel. Slower (21.9 vs 20.2 µs), and the barrier hung when reused across launches with different grid sizes.
Early PDL triggers everywhereLet every kernel signal the next one to start as early as possible. It caused occasional NaNs in full-context verify, and without a clear cause I didn’t ship it.
MegaMoEDeepGEMM’s fused all-in-one MoE kernels. The judge image’s builds only supported Blackwell (SM100) or FP8, and lower precision was banned.