Big news from DeepSeek! The company has officially launched its first open-source repository, leveraging CUDA Kernels to enhance the speed and efficiency of LLMs. At the heart of this update is FlashMLA<\/strong>, an advanced multi-latent attention (MLA) decoding kernel, specifically optimized for Hopper GPUs. This technology handles variable-length sequences more efficiently, making AI model hosting smoother and faster.<\/p>\n \ud83d\ude80 Day 1 of #OpenSourceWeek<\/a>: FlashMLA<\/p>\n Honored to share FlashMLA \u2013 our efficient MLA decoding kernel for Hopper GPUs, optimized for variable-length sequences and now in production.<\/p>\n \u2705 BF16 support \u2014 DeepSeek (@deepseek_ai) February 24, 2025<\/a><\/p><\/blockquote>\n<\/div>\n<\/figure>\n Key Highlights of the Release:<\/strong><\/p>\n These optimizations deliver up to 3000 GB\/s in memory-bound configurations and 580 TFLOPS in computation-bound scenarios when running on H800 SXM5 GPUs with CUDA 12.6.<\/p>\n With this level of performance, AI inference just got a major upgrade! Sounds intriguing, right?<\/p>\n Note: Earlier, MLA was used in DeepSeek Models and now FlashMLA using CUDA Kernels makes hosting DeepSeek AI\u2019s R1 + V3 faster!<\/em><\/p>\n FlashMLA is an optimised MLA decoding kernel designed specifically for Hopper GPUs, NVIDIA\u2019s next-generation architecture. Built with performance in mind, it embodies Deepseek\u2019s commitment to accelerating AI models at scale. FlashMLA ensures faster, more efficient processing where every millisecond counts.<\/p>\n FlashMLA is designed to run on high-performance GPUs, specifically Hopper architecture GPUs such as the H800 SXM5. It requires CUDA 12.3+ and PyTorch 2.0+ for optimal performance.<\/p>\n Based on results from its official GitHub repository<\/strong><\/a>, FlashMLA delivers impressive performance:<\/p>\n This combination of high memory bandwidth, efficient caching, and exceptional computational throughput makes FlashMLA a powerful choice for AI workloads requiring extreme performance.<\/p>\n If this is all gibberish for you then, don\u2019t worry I will be explaining this in depth. Let\u2019s start with Multi-head Latent Attention (MLA)<\/p>\n The Multi-head Latent Attention (MLA) was introduced with the release of the DeepSeek-V2 a variant of multi-head attention (MHA). It belongs to a family of techniques designed to address a key challenge in scaling large models: reducing the KV-cache size, which can become a major memory bottleneck. Other methods in this category include Group-Query Attention and Multi-Query Attention. While these approaches help lower memory usage, they often come with a tradeoff\u2014sacrificing some performance in exchange for greater scalability.<\/p>\n MLA takes a different approach by using a low-rank factorized projection matrix, which works somewhat like multi-query attention. However, instead of simply repeating a single head multiple times, it decompresses a latent vector to generate a unique and appropriate K and V head for each Q head. According to DeepSeek, this method not only reduces memory overhead but actually enhances the model\u2019s performance rather than compromising it.<\/p>\n Multi-head attention (MHA) enhances a model\u2019s ability to capture diverse relationships in data by processing queries, keys, and values independently across multiple attention heads. However, this flexibility comes at a cost, especially during inference. The KV cache, which stores keys and values from previous tokens, expands linearly with sequence length. This quickly becomes a bottleneck, consuming significant GPU memory for long sequences.<\/p>\n For a model with n_h<\/strong> attention heads and a head dimension of d_h<\/strong>, the KV cache size is calculated as:<\/p>\n For large sequence lengths, this can exceed memory limits, restricting model scalability and efficiency.<\/p>\n Memory Latent Attention (MLA)<\/strong> addresses this challenge by introducing a more compact way to store KV information. Instead of directly caching keys and values, MLA compresses them into a latent vector c_t<\/strong> for each token t<\/strong>, significantly reducing storage requirements. The process works as follows:<\/p>\n Here, W^{UK}<\/strong> and W^{UV}<\/strong> are transformation matrices mapping d_c<\/strong> back to n_h * d_h<\/strong>.<\/p>\n This approach drastically cuts memory usage\u2014DeepSeek-V2 demonstrates up to 93.3% reduction<\/strong>, allowing for longer context handling and more efficient processing.<\/p>\n By leveraging MLA, models can achieve longer context understanding while keeping hardware requirements manageable, unlocking new possibilities for efficient large-scale AI applications.<\/p>\n To understand this in detail read: <\/p>\n Key-value (KV) caching is a powerful optimization technique that accelerates the autoregressive decoding process by storing and reusing previously computed key-value pairs, rather than recalculating them at each step.<\/p>\n This method primarily serves during inference, as training still requires processing the entire input sequence simultaneously. By leveraging KV caching, we avoid redundant computations, significantly improving efficiency.<\/p>\n KV caching typically operates as a rolling buffer. During each decoding step:<\/p>\n This approach reduces computational overhead, making autoregressive models more efficient. However, it comes with a trade-off: Increased memory usage<\/strong>. Since the KV cache scales proportionally with factors like batch size, sequence length, hidden size, and the number of attention heads, it can quickly become a memory bottleneck\u2014especially for large batches or long sequences.<\/p>\n To tackle these memory constraints, two key strategies have emerged:<\/p>\n By integrating these techniques, KV caching enables faster and more scalable inference, making it an essential component in modern transformer-based architectures.<\/p>\n DeepSeek\u2019s models leverage FlashMLA<\/strong> to achieve remarkable efficiency and scalability in the following models.<\/p>\n By integrating FlashMLA<\/strong>, DeepSeek is pushing the boundaries of AI efficiency and economic feasibility.<\/p>\n Now, let\u2019s talk about the NVIDIA Hopper.\u00a0<\/p>\n NVIDIA Hopper is a revolutionary GPU architecture designed to supercharge artificial intelligence (AI) and high-performance computing (HPC) workloads. Named after the pioneering computer scientist Grace Hopper<\/strong>, this cutting-edge technology is built to handle large-scale parallel processing with exceptional memory efficiency. It empowers researchers, developers, and enterprises to achieve breakthrough speeds in AI, machine learning, and deep learning applications.<\/p>\n The NVIDIA Hopper architecture is packed with over 80 billion transistors, built on TSMC\u2019s advanced 4N process. It incorporates key innovations such as NVLink Switch, Confidential Computing, the Transformer Engine, and Second-Generation MIG (Multi-Instance GPU). These technologies fuel the power of NVIDIA\u2019s H100 and H200 GPUs, making them the ultimate choice for AI workloads\u2014from training and inference to generative AI and deep learning.<\/p>\n Whether you\u2019re tackling massive datasets, training sophisticated AI models, or running complex simulations, NVIDIA Hopper delivers the speed, scalability, and efficiency needed to push the boundaries of AI and computing.<\/p>\n The optimized CUDA Kernels in DeepSeek AI\u2019s implementation are achieving an actual performance of 580 TFLOPS (trillion floating-point operations per second) for BF16 (bfloat16) matrix multiplication\u2014which is more than double the theoretical peak of 260 TFLOPS for the H800 GPU.<\/p>\n The fact that DeepSeek\u2019s optimizations are more than doubling the expected performance suggests an extremely efficient use of the GPU\u2019s computational power, making AI workloads much faster than conventional implementations.<\/p>\n DeepSeek\u2019s release of FlashMLA marks a significant breakthrough in AI inference efficiency, particularly for Hopper GPUs. By introducing Multi-Latent Attention (MLA), DeepSeek optimizes memory usage while maintaining or even enhancing model performance. The paged KV cache and BF16 support allow for high-speed processing, with memory bandwidth reaching 3000 GB\/s and computational performance up to 580 TFLOPS on H800 SXM5 GPUs.<\/p>\n MLA drastically reduces KV cache size\u2014by up to 93.3%\u2014making large-scale AI models more efficient and cost-effective. This innovation is central to DeepSeek-V2 and V3, enabling longer context handling, faster inference, and lower training costs. With FlashMLA, DeepSeek is pushing the limits of AI scalability, making large-scale AI more accessible and practical while setting new standards in model efficiency and economic viability.<\/p>\n Stay tuned to Analytics Vidhya Blog <\/a>for our detailed analysis on DeepSeek\u2019s Day 2 release! <\/p>\n\n
\u2705 Paged KV cache (block size 64)
\u26a1 3000 GB\/s memory-bound & 580 TFLOPS\u2026<\/p>\n\n
What is FlashMLA?<\/h2>\n
Hardware Requirements<\/strong><\/h4>\n
Precision and Optimization<\/strong><\/h4>\n
\n
Performance Benchmarks<\/strong><\/h4>\n
\n
Brief About Multi-head Latent Attention (MLA)<\/h2>\n
![\"\"](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/Brief-About-Multi-head-Latent-Attention-MLA.webp\")
Standard Multi-Head Attention and Its Limitations<\/h3>\n
![\"\"](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/KV-cache-size.webp\")
<\/figure>\nHow MLA Optimizes Memory Usage?<\/h3>\n
![\"\"](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/How-MLA-Optimizes-Memory-Usage_.webp\")
\n
\n
![\"\"](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/unnamed-3.webp\")
<\/figure>\n\n
\n
Key-Value Caching: Enhancing Autoregressive Decoding<\/h2>\n
How KV Caching Works?<\/h3>\n
\n
Overcoming the Memory Challenge<\/h3>\n
![\"\"](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/Overcoming-the-Memory-Challenge.webp\")
\n
FlashMLA: Powering DeepSeek\u2019s Cutting-Edge Models<\/h3>\n
What is NVIDIA Hopper?<\/h2>\n
Inside the NVIDIA Hopper Architecture<\/h4>\n
The Performance<\/h2>\n
What this Implies?<\/h3>\n
\n
\n
\n
\n
Conclusion<\/h2>\n
![\"Pankaj](\"https:\/\/av-eks-lekhak.s3.amazonaws.com\/media\/lekhak-profile-images\/converted_image_Lb7Lh0T.webp\")
<\/p>\n