How I Beat PyTorch's Flash Attention: A CUDA Optimization Journey
From 1.75 to 22.6 TFLOPS - a deep dive into CUDA tensor cores, PTX instructions, and the optimization techniques that let my implementation outperform PyTorch's cuDNN at long sequence lengths.