参考
Linear Layouts: Robust Code Generation of Efficient Tensor Computation Using F₂
A Formal Analysis of the NVIDIA PTX Memory Consistency Model by Lustig et al., 2019
Dissecting the Turing GPU Architecture through Microbenchmarking (Jia et al., 2019)
PTX Documentation: Link
SASS Instruction List: Link
https://mlc.ai/modern-gpu-programming-for-mlsys/chapter_background/index.html
FlashAttention-4: Faster Attention with Asynchrony and Low-precision
Parallel Random Numbers: As Easy as 1, 2, 3 (Salmon et al., 2011)
Dropout: A Simple Way to Prevent Neural Networks from Overfitting (Srivastava et al., JMLR 2014)
Layer Normalization (Ba, Kiros, Hinton, 2016)
FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning (Dao, 2023)
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (Dao et al., 2022)
Self-attention Does Not Need O(n^2) Memory (Rabe and Staats, 2021)
Triton Tutorial - Libdevice function;
tl.extra.libdevice;triton-lang/tritonProfiling Deep Learning Inference with Nsight Systems and nvprof: A Practical Guide
NVIDIA PTX ISA:Asynchronous Warpgroup Level Matrix Instructions