Hugging Face published 'Profiling in PyTorch (Part 3): Attention is all you profile' on 2026-07-10. It shows how to read GPU profiler traces across attention backends — naive, math, memory-efficient, FlashAttention-2, and cuDNN — to find which wins.
The takeaway: 'inefficient'-looking fused kernels like FlashAttention-2 are often fastest, and trace visualization exposes hidden costs (memory copies, CPU overhead) that occupancy metrics miss.



