Skip to main content

Writing CUDA kernels for attention by hand

00:00:36:79

This write-up is in progress. I’m putting together the full story of building the kernels: the ideas behind each one, the benchmarks, and the mistakes I made measuring them.

Until then, the code and the full results are on GitHub.