https://l1-ca0.github.io/tags/hopper/ Recent content in Hopper on Li Cao's Blog Hugo -- 0.148.0 en-us Sat, 29 Aug 2026 20:12:39 +0000 https://l1-ca0.github.io/posts/flash-attention-3-on-hopper/ Sat, 29 Aug 2026 20:12:39 +0000 https://l1-ca0.github.io/posts/flash-attention-3-on-hopper/ <p><a id="contents"></a></p> <h2 id="contents">Contents</h2> <ul> <li><a href="#ch-intro">1 Introduction and Motivation</a> <ul> <li><a href="#section-1.1">1.1 Why attention is the bottleneck</a></li> <li><a href="#section-1.2">1.2 The Hopper problem: FA2 leaves most of the H100 idle</a></li> <li><a href="#sec-three-ideas">1.3 The three ideas of FlashAttention-3</a></li> <li><a href="#sec-codebase">1.4 The code base: from CUTLASS C++ to the CuTe DSL</a></li> <li><a href="#section-1.5">1.5 Organization of this post</a></li> </ul> </li> <li><a href="#ch-hopper">2 The Hopper GPU Architecture: Features Used by FA3</a> <ul> <li><a href="#section-2.1">2.1 Chip-level anatomy of the H100</a></li> <li><a href="#sec-sm-anatomy">2.2 The SM: warps, warpgroups, and the asynchronous datapaths</a></li> <li><a href="#sec-tma">2.3 TMA: the Tensor Memory Accelerator</a> <ul> <li><a href="#subsection-2.3.1">2.3.1 Multicast and clusters</a></li> </ul> </li> <li><a href="#sec-mbarrier">2.4 mbarriers: arrive/wait synchronization in shared memory</a></li> <li><a href="#sec-wgmma">2.5 WGMMA: asynchronous warpgroup matrix multiply</a></li> <li><a href="#sec-async-proxy">2.6 The async proxy and memory fences</a></li> <li><a href="#sec-setmaxnreg">2.7 Register reallocation: <code>setmaxnreg</code></a></li> <li><a href="#section-2.8">2.8 Threadblock clusters, persistent kernels, and occupancy</a></li> <li><a href="#sec-sfu">2.9 The SFU: why softmax is expensive</a></li> <li><a href="#section-2.10">2.10 Putting it together: the FA3 hardware checklist</a></li> </ul> </li> <li><a href="#ch-background">3 Background: Attention, Online Softmax, FA1/FA2</a> <ul> <li><a href="#section-3.1">3.1 Attention and the quadratic intermediate</a></li> <li><a href="#sec-online-softmax">3.2 Safe softmax and its streaming state</a> <ul> <li><a href="#subsection-3.2.1">3.2.1 The block-combine interpretation</a></li> <li><a href="#subsection-3.2.2">3.2.2 Consequences for implementation</a></li> </ul> </li> <li><a href="#sec-fa1-background">3.3 FlashAttention-1: IO-aware exact attention</a> <ul> <li><a href="#subsection-3.3.1">3.3.1 Backward recomputation</a></li> <li><a href="#subsection-3.3.2">3.3.2 First-generation work partitioning</a></li> </ul> </li> <li><a href="#sec-fa2-background">3.4 FlashAttention-2: parallelism and work partitioning</a> <ul> <li><a href="#subsection-3.4.1">3.4.1 Fewer non-matrix operations</a></li> <li><a href="#subsection-3.4.2">3.4.2 A query block becomes an independent thread-block task</a></li> <li><a href="#subsection-3.4.3">3.4.3 Split-Q warp partitioning</a></li> <li><a href="#subsection-3.4.4">3.4.4 Performance envelope and the remaining Hopper gap</a></li> </ul> </li> <li><a href="#section-3.5">3.5 FA1, FA2, and FA3 in one comparison</a></li> <li><a href="#section-3.6">3.6 Kernel anatomy inherited by FA3</a></li> </ul> </li> <li><a href="#ch-fwd-algo">4 The FA3 Forward Pass: Algorithm</a> <ul> <li><a href="#section-4.1">4.1 Work decomposition</a></li> <li><a href="#sec-fwd-online-invariant">4.2 Online-softmax state and invariant</a></li> <li><a href="#sec-producer-consumer">4.3 Producer–consumer warp specialization</a> <ul> <li><a href="#subsection-4.3.1">4.3.1 Pipeline-stage ownership</a></li> </ul> </li> <li><a href="#sec-pingpong">4.4 Pingpong scheduling: hiding softmax between warpgroups</a></li> <li><a href="#sec-intra-wg">4.5 Intra-warpgroup overlapping: the 2-stage GEMM–softmax pipeline</a> <ul> <li><a href="#subsection-4.5.1">4.5.1 Prologue, steady state, and drain</a></li> </ul> </li> <li><a href="#sec-fwd-softmax-details">4.6 Softmax details: rescaling, correction, and LSE</a></li> <li><a href="#sec-fwd-causal">4.7 Causal masking and the split loop</a></li> <li><a href="#sec-fwd-tile-accounting">4.8 Per-tile computation and data movement</a></li> <li><a href="#sec-fwd-tiles">4.9 Tile sizes, head-dimension variants, and register budgets</a></li> <li><a href="#section-4.10">4.10 Epilogue</a></li> </ul> </li> <li><a href="#ch-fwd-impl">5 The FA3 Forward Pass: CuTe DSL Implementation</a> <ul> <li><a href="#section-5.1">5.1 A note on the CuTe DSL</a></li> <li><a href="#sec-fwd-impl-map">5.2 Implementation map</a></li> <li><a href="#sec-fwd-config">5.3 Kernel configuration</a> <ul> <li><a href="#subsection-5.3.1">5.3.1 Compile-time variant selection</a></li> </ul> </li> <li><a href="#sec-fwd-smem">5.4 Shared memory plan</a> <ul> <li><a href="#sec-fwd-tensor-partitions">5.4.1 Tensor views and WGMMA partitions</a></li> </ul> </li> <li><a href="#section-5.5">5.5 Kernel entry: pipelines and role dispatch</a> <ul> <li><a href="#sec-fwd-pipeline-cursors">5.5.1 Pipeline cursors and phase changes</a></li> </ul> </li> <li><a href="#sec-fwd-producer">5.6 The producer: <code>load</code></a> <ul> <li><a href="#subsection-5.6.1">5.6.1 From a scheduled tile to copy coordinates</a></li> </ul> </li> <li><a href="#sec-fwd-consumer">5.7 The consumer: <code>mma</code></a> <ul> <li><a href="#subsection-5.7.1">5.7.1 Accumulator ownership and initialization</a></li> </ul> </li> <li><a href="#sec-fwd-inner-step">5.8 Core operation: one \(n\)-block with intra-warpgroup overlap</a> <ul> <li><a href="#subsection-5.8.1">5.8.1 Why each wait and release is safe</a></li> </ul> </li> <li><a href="#section-5.9">5.9 Pingpong in code: the scheduler barriers</a></li> <li><a href="#sec-fwd-softmax-code">5.10 Softmax module</a></li> <li><a href="#section-5.11">5.11 Epilogue in code</a></li> <li><a href="#sec-cpp-vs-dsl">5.12 Differences from the paper&rsquo;s CUTLASS C++ kernels</a></li> </ul> </li> <li><a href="#ch-fp8">6 FP8 Forward: Layouts, Block Quantization, Incoherent Processing</a> <ul> <li><a href="#section-6.1">6.1 FP8 formats and accumulation</a></li> <li><a href="#sec-fp8-transpose">6.2 The layout problem: K-major operands and the in-kernel V transpose</a></li> <li><a href="#sec-block-quant">6.3 Block quantization</a></li> <li><a href="#sec-hadamard">6.4 Incoherent processing: the Hadamard transform</a></li> <li><a href="#section-6.5">6.5 FP8 kernel schedule and performance shape</a></li> </ul> </li> <li><a href="#ch-bwd-algo">7 The FA3 Backward Pass: Algorithm</a> <ul> <li><a href="#sec-bwd-math">7.1 Gradient math</a> <ul> <li><a href="#subsection-7.1.1">7.1.1 The softmax Jacobian, step by step</a></li> <li><a href="#subsection-7.1.2">7.1.2 Why \(D\) can be computed without \(\mathbf{P}\)</a></li> <li><a href="#subsection-7.1.3">7.1.3 Why LSE alone suffices to recompute \(\mathbf{P}\)</a></li> <li><a href="#sec-dlse">7.1.4 Gradients through LSE itself</a></li> <li><a href="#subsection-7.1.5">7.1.5 Where the scale \(\alpha\) is applied</a></li> </ul> </li> <li><a href="#sec-bwd-tiling">7.2 Tiling and the dQ problem</a> <ul> <li><a href="#subsection-7.2.1">7.2.1 Why KV-parallel and not Q-parallel</a></li> <li><a href="#subsection-7.2.2">7.2.2 Causal masking: trapezoidal iteration ranges</a></li> <li><a href="#subsection-7.2.3">7.2.3 Three ways to resolve the dQ race</a></li> </ul> </li> <li><a href="#sec-bwd-preprocess">7.3 The preprocessing and postprocessing kernels</a> <ul> <li><a href="#subsection-7.3.1">7.3.1 Preprocess: D, LSElog2, and zeroing dQaccum</a></li> <li><a href="#subsection-7.3.2">7.3.2 Postprocess: dQaccum \(\to\) dQ</a></li> </ul> </li> <li><a href="#sec-bwd-warpspec">7.4 Warp specialization in the backward</a> <ul> <li><a href="#subsection-7.4.1">7.4.1 Named barriers of the backward</a></li> <li><a href="#subsection-7.4.2">7.4.2 SMEM budget</a></li> </ul> </li> <li><a href="#sec-bwd-schedule">7.5 Scheduling the five GEMMs</a> <ul> <li><a href="#subsection-7.5.1">7.5.1 Operand layouts: who is SS, who is RS</a></li> <li><a href="#sec-swapab">7.5.2 swapAB: transposing the whole GEMM instead of the data</a></li> </ul> </li> <li><a href="#sec-bwd-dq-algo">7.6 dQ accumulation across thread blocks</a></li> <li><a href="#section-7.7">7.7 Backward tile configurations</a></li> <li><a href="#sec-bwd-features">7.8 Causal masking, varlen, and other mainloop features</a></li> <li><a href="#sec-bwd-perf">7.9 Backward performance</a></li> </ul> </li> <li><a href="#ch-bwd-impl">8 The FA3 Backward Pass: CuTe DSL Implementation</a> <ul> <li><a href="#section-8.1">8.1 Configuration surface</a></li> <li><a href="#sec-bwd-smem-layouts">8.2 SMEM layouts supporting two access orientations</a></li> <li><a href="#section-8.3">8.3 Host-side: TMA atoms, schedulers, semaphores, PDL</a></li> <li><a href="#section-8.4">8.4 Role dispatch: two producer warps, two consumer warpgroups</a></li> <li><a href="#section-8.5">8.5 The producer load loop</a></li> <li><a href="#sec-bwd-inner-code">8.6 The consumer inner loop: five GEMMs and two pointwise stages</a></li> <li><a href="#sec-dq-accum">8.7 The dQaccum store agent</a></li> <li><a href="#section-8.8">8.8 Epilogue: dK/dV stores, and the GQA accumulation path</a></li> <li><a href="#section-8.9">8.9 The postprocessing kernel</a></li> <li><a href="#section-8.10">8.10 Preprocess kernel code</a></li> </ul> </li> <li><a href="#ch-scheduling">9 Tile Scheduling, Variable-Length Sequences, and Masking</a> <ul> <li><a href="#section-9.1">9.1 What the tile scheduler controls</a> <ul> <li><a href="#subsection-9.1.1">9.1.1 Where scheduling appears in the kernel</a></li> </ul> </li> <li><a href="#section-9.2">9.2 The scheduling problem: causal masking skews tile cost</a> <ul> <li><a href="#subsection-9.2.1">9.2.1 Static, persistent, and dynamic scheduling are not synonyms</a></li> </ul> </li> <li><a href="#sec-varlen">9.3 Variable-length sequences (varlen)</a> <ul> <li><a href="#subsection-9.3.1">9.3.1 How the varlen coordinate map works</a></li> </ul> </li> <li><a href="#section-9.4">9.4 Scheduling and masking: division of responsibility</a> <ul> <li><a href="#subsection-9.4.1">9.4.1 Correctness invariants for scheduler changes</a></li> </ul> </li> <li><a href="#section-9.5">9.5 GQA and PackGQA</a></li> <li><a href="#sec-masking">9.6 Masking machinery</a></li> <li><a href="#sec-paged-kv">9.7 Paged KV and other serving features</a></li> </ul> </li> <li><a href="#ch-splitkv">10 Flash-Decoding: Split-KV and the Combine Kernel</a> <ul> <li><a href="#section-10.1">10.1 Why decoding underutilizes the GPU</a></li> <li><a href="#sec-splitkv-idea">10.2 Parallelization over the KV sequence</a> <ul> <li><a href="#subsection-10.2.1">10.2.1 The combine math</a></li> </ul> </li> <li><a href="#sec-combine-kernel">10.3 The CuTe DSL combine kernel</a></li> <li><a href="#section-10.4">10.4 How the mainloop produces splits</a></li> <li><a href="#sec-num-splits">10.5 Choosing <code>num_splits</code></a></li> <li><a href="#section-10.6">10.6 Interaction with GQA, paged KV, and varlen</a></li> <li><a href="#section-10.7">10.7 Performance character and caveats</a></li> </ul> </li> <li><a href="#ch-packgqa">11 PackGQA: Packing Query Heads into the Query Tile</a> <ul> <li><a href="#section-11.1">11.1 Motivation: filling the M tile</a> <ul> <li><a href="#subsection-11.1.1">11.1.1 What packing changes, and what it preserves</a></li> </ul> </li> <li><a href="#sec-packgqa-layout">11.2 The packed layout: a zero-copy CuTe transformation</a> <ul> <li><a href="#subsection-11.2.1">11.2.1 TMA with a packed mode</a></li> </ul> </li> <li><a href="#sec-packgqa-copy">11.3 The gather/scatter path: <code>PackGQA</code></a></li> <li><a href="#sec-packgqa-mask">11.4 Masking and per-row head indices</a></li> <li><a href="#sec-packgqa-heur">11.5 Enablement criteria for PackGQA</a></li> <li><a href="#section-11.6">11.6 Composition with split-KV, and provenance</a></li> </ul> </li> <li><a href="#ch-performance">12 Performance Analysis and Benchmarks</a> <ul> <li><a href="#section-12.1">12.1 Forward pass, BF16</a></li> <li><a href="#section-12.2">12.2 Backward pass</a></li> <li><a href="#sec-fp8-perf">12.3 FP8 forward</a></li> <li><a href="#section-12.4">12.4 An issue-time consistency check</a></li> <li><a href="#section-12.5">12.5 Interpreting throughput curves</a></li> <li><a href="#section-12.6">12.6 End-to-end implications of attention speedups</a></li> </ul> </li> <li><a href="#ch-caveats">13 Caveats and Pitfalls</a> <ul> <li><a href="#section-13.1">13.1 Numerical caveats</a></li> <li><a href="#section-13.2">13.2 Resource and performance caveats</a></li> <li><a href="#section-13.3">13.3 Minimum validation matrix for kernel changes</a></li> </ul> </li> <li><a href="#ch-refs">14 Sources and Further Reading</a> <ul> <li><a href="#sec-star-14-0">How the sources were used</a></li> </ul> </li> </ul> <p><a id="ch-intro"></a></p>