- Published on
- |Views: 1332|40 min read
On bottlenecks in attention, kv caching, long-context decoding, attention variants, and how DeepSeek MLA came to be. Part 1 of the FlashMLA blog series.
All Posts
All Posts
- cuda (2)
- writing (2)
- llm (2)
- tensors (2)
- pytorch (2)
- linear-algebra (2)
- machine-learning (2)
- metal (1)
- gpu (1)
- parallel-programming (1)
- nvcc (1)
- colab (1)
- pretraining (1)
- midtraining (1)
- posttraining (1)
- data-quality (1)
- synthetic-data (1)
- dpo (1)
- personal-growth (1)
- webdev (1)
- side-project (1)
- attention (1)
- transformers (1)
- deepseek (1)
- mla (1)
- kv-cache (1)
- inference (1)
- Published on
- |Views: 1107|35 min read
Notes on Microsoft phi-4 data pipeline for pre-training, 'mid-training', supervised fine-tuning and preference optimization- Published on
- |Views: 4107|24 min read
Notes on the Apple Silicon GPUs: Architecture, Memory Hierarchy, and the Metal Programming framework, and how it compares to NVIDIA CUDA.- Published on
- |Views: 1941|13 min read
A recap of my day debugging issues with nvcc and nvcc4jupyter on Google Colab's free T4 GPUs, with brief notes on CUDA backward compatibility and compute capability- Published on
- |Views: 382|17 min read
Part 2 of my Tensor Puzzles Walkthrough series: optimizing solutions to fit the puzzle constraints, and comparing notes with the author.