CUDA Kernel Lab
RMSNorm fusion remains the strongest supported win, while H200 matmul autotune now bounds the Tensor Core gap against PyTorch/cuBLAS.
- Device
- A10G + H200
- Correctness
- Selected rows pass
Method
Optimization loop
Every kernel claim moves through the same loop before it becomes a decision.
Profile first
Start from matched PyTorch/Triton rows, correctness checks, and Nsight counters when the benchmark is interesting enough to explain.
Identify bottleneck
Classify the limit as memory bandwidth, launch overhead, occupancy, register pressure, Tensor Core utilization, or traffic from unfused intermediates.
Optimize that bottleneck
Make one bounded change: fuse rows, change block size, compare reduction strategy, sweep matmul tiles, or reduce decode replay overhead.
Re-profile
Compare the next run against the same control and promote only supported, caveated, rejected, or pending claims.
Decisions
Kernel Optimization Decisions
Fusion wins
Use fusion for RMSNorm because it removes enough framework and intermediate-tensor overhead to beat PyTorch across the measured shape sweep.
fp16 reached 5.901x at 4096x8192; the 4096x4096 rerun reached 5.599x.
Treat elementwise fusion as a strong path before deeper matmul work.
fp32 0.4547 ms Triton vs 1.418 ms PyTorch; 3.119x speedup.
Memory/reduction boundaries
Do not broaden simple launch sweeps before profiler counters explain the PyTorch lead.
Triton vector_add profile shows 91.58% DRAM throughput rather than an obvious block-size fix. Next: Prioritize fusion or reuse changes before wider simple-memory launch sweeps.
Keep reduction variants as an active track, but do not claim a PyTorch win yet.
Both Triton reduction variants passed correctness and streamed well, but still trailed the PyTorch baseline. Next: Target launch/finalization overhead or a different reduction structure.
Do not present the current row-softmax kernel as a portfolio win.
PyTorch beat the current Triton fused row-softmax kernel in both float16 and float32.
Matmul/Tensor Core gaps
Treat Triton tiling as an active optimization track, not a portfolio win over cuBLAS yet.
Best Triton tile reached 25.74 TFLOP/s; PyTorch/cuBLAS stayed around 30-31 TFLOP/s. Next: Profile the current best tile before the next tuning pass.
Keep H200 matmul as an active gap-closing track; standard tiled-dot is currently best, while persistent-wave scheduling is measured but not yet useful.
Clean focused H200 timing reached 470.7 TFLOP/s bfloat16 and 462.0 TFLOP/s float16, about 88-90% of PyTorch/cuBLAS; the latest persistent-wave sweep kept standard Triton ahead at 471.4 TFLOP/s bfloat16. Next: Run Tensor Core counter profiles on an H200 host with NVIDIA performance-counter access; current RunPod profile reports hit permission failures.
Use counters to explain benchmark results instead of treating latency alone as the answer.
RMSNorm profile shows 90.91% DRAM throughput and 93.12% occupancy; profiled matmul shows 45.19% Tensor Core utilization.
Decode replay caveats
Use resident-KV same-stream piecewise CUDA Graph replay as a measured synthetic upper bound, not an end-to-end serving claim.
Round 12 reached 0.1375 ms fixed-shape p50 and about 0.156 ms dynamic p50 / 0.230 ms p95. Next: Turn timing probes back on to explain the remaining p95 tail.
Evidence archive
Experiments
The measured runs behind the decisions — each detail page keeps the run shape, commands, and result evidence.
Kernel memory traffic
Memory Primitive Bandwidth
Kernel reductions
Reduction Strategy Comparison
Kernel fusion
Normalization Fusion
Kernel fusion
SwiGLU Elementwise Fusion
Kernel fusion
Row Softmax Fusion
Kernel tiling
Matmul Tile Sweep
Tensor Core matmul
H200 Matmul Autotune
Decode scheduling
Decode Step Graph Replay
Profiler evidence
Profiler Validation