Tech ONTAP Podcast Episode 404 – NetApp DataOps Toolkit
Managing large amounts of data can be challenging, especially when you also have to manage the storage system where the data lives. So to make life easier for storage administrators, NetApp offers a data ops toolkit that can offload some of the work associated with storage management. On this Tech ONTAP Podcast episode, we welcome the creator of the toolkit, Michael Oglesby, to talk about the latest updates and where best to use it.
Mike also has a number of blogs on using ONTAP for AI use cases, as well as other video interviews:
Finding the podcast
Check it out here, like and subscribe and all that jazz (now hosted on the NetApp YouTube channel!):
If you prefer audio only, I also still offer that:
https://soundcloud.com/techontap_podcast/episode-404-dataops-toolkit
You can also find the Tech ONTAP Podcast on:
Transcription
The following transcript was generated using Descript’s speech to text service and then further edited. As it is AI generated, YMMV.
Tech ONTAP Podcast Episode 404 – Data Ops Toolkit
===
[00:00:00]
Justin Parisi: I’m here in my studio, of course in the basement as you can see well lit basement. I’m here with Mike Oglesby, he’s actually in the office.
‘Cause you know, we thrive in the office. So Mike what do you do here at NetApp and how to reach you?
Michael Oglesby: Yeah, I’m here in the RTP office with a blank white wall behind me in a conference room. But yeah. I’m Mike Oglesby. I’m a software engineer here at NetApp, focused on AI solutions specifically.
So this would be essentially anything outside of the core product that involves using our products for AI use cases, so reference architectures, tooling, integrations, develop code, write a technical reports, solution, documentation, all that good stuff.
And you can reach me either by email, [email protected] or LinkedIn is the best social media to find me [00:01:00] on. I don’t really do much on the other social medias.
Justin Parisi: Yeah, I don’t think many people do anymore. Yeah, it’s mostly all LinkedIn, which is kind of funny ’cause you know, five, 10 years ago, nobody would’ve been like, oh yeah, find me on LinkedIn.
But now, yeah.
Michael Oglesby: Yeah, yeah. Yeah. I can’t even remember the last time I opened Facebook. I occasionally open Twitter/X and then quickly regret it.
Justin Parisi: Yeah. Yeah. It’s definitely become kind of a cesspool. Yeah. But hey, you know, it is what it is. So you mentioned that you work on AI workflows, you work on AI use cases, and one of those includes the large language model. So I know that’s a hot topic these days with generative AI and all the other AI things floating around, out there. Tell me a little more about large language models , what they are and some of the general workflows.
Michael Oglesby: Yeah, so, LLMs are the 800 pound gorilla in the AI space these days. A lot of people, when they say ai, they basically [00:02:00] mean LLMs. That’s what they’re, that’s what they’re talking about. Yeah. They mean gen AI with LLMs, it’s kind of become almost synonymous with the term ai. It’s been the hot topic ever since chat GPT came out. But for anybody who’s not familiar with the terminology I’m sure you’re familiar with chat GPT and Gemini and Claude, and basically those are chatbots that are powered by LLMs. A large language model is the type of AI model that sits behind those chatbots. You submit a prompt, it’s fed through this LLM, the LLM responds to your prompt, and the chat bot prints the prompt out to you.
So the LLMs are the brains of all these chatbots that are out there.
Justin Parisi: Yeah, it’s kind of the librarian, right? So you’ve got this big old catalog of books and information and knowledge out there, and the LLMs job is to find the specific section [00:03:00] of the specific book that you were asking about.
And it’s really only as good as your prompts. So you had to be really specific.
Michael Oglesby: Yeah. Technically speaking, what it actually does is look at all the previous words in a sequence and predict the next word, essentially. The prompt is divided into tokens and then it predicts the next token.
A token might not match one to one to a word, but you can think about it as you submit a prompt, a sequence of words, and then one by one it predicts each of the next words based on all the content it was trained on, which for most of the LLMs out there is essentially the entirety of the public internet plus a bunch of public domain books, plus a bunch of books that were picked up at a used bookstore by an anthropic employee.
Stuff like that. Yeah,
Justin Parisi: yeah. And like, like any database or data store, it’s only as good as the information it’s fed. So that becomes a really big part of it when you’re trying [00:04:00] to build a le, a large language model that. Does what it’s supposed to do.
Michael Oglesby: Yeah. Yeah. So that’s a good segue to deploying LLMs in an enterprise setting.
These LLMs were trained on like general public stuff, right? So the public internet books, just anything that the researchers could get their hands on, that was a corpus of text they used for training. But that doesn’t include your private data within an enterprise. And so since chat GPT was released, basically every enterprise everywhere has been scrambling to figure out how to apply LLMs within their enterprise and augment them with their private data. And really there’s three primary ways you can do this. For unstructured data, meaning just like lLMs work on text.
So when I say unstructured data, I mean unstructured text. Basically you [00:05:00] can either fine tune the model, so you can take a foundation model. Typically you would use an open source model if you were fine tuning like one of Meta’s Llama models or one of the MUSTAR models or one of the deep seek or Quinn models.
And you would essentially modify the last few layers of it. By training it on your private data, that’s kind of hard. You need a certain amount of GPU horsepower and you need some AI researcher skills to be able to do that effectively. There’s frameworks you can use to help you, but it’s still not a small undertaking and then you have the headache of keeping it up to date. As your source data changes, you have to re-fine tune your model periodically to keep it up to date with your source data. So what most enterprises private organizations are doing is implementing RAG or retrieval augmented generation. Basically this was the hot topic a [00:06:00] year or two ago. And there were POCs running everywhere and now these types of deployments are actually starting to go into production. I know we have, we have a pretty extensive RAG deployment running in production here at NetApp, serving internal NetApp employees. But basically you take a corpus of source data. So this could be your private code bases, your private Wiki pages, your private knowledge base entries, any unstructured text, and you vectorize it, essentially, you run it through what’s called an embedding model to generate vector representations of the text, which you store in a vector database, and then anytime a prompt comes in to the LLM, you can take that prompt, vectorize it, perform a similarity search on the vector database, return any similar vectors, and essentially append them to the end of your prompt or augment your prompt with these [00:07:00] similar vectors, hence the name retrieval augmented generation.
You retrieve them from the vector database and then augment your prompt. And it takes some DevOps skills, some platform engineering skills to deploy something like this. You typically have to code up a simple pipeline to keep the vector embeddings up to date, stitch a few tools together, but it’s not as daunting as fine-tuning a model and continually keeping that up to date and you can use standard off the shelf tools.
Open source tools like VLLM or proprietary tools like NVIDIA’s, suite of NIMS to deploy these LLMs with off the shelf configs. It’s much more approachable. So this is what most enterprises are doing. But then there’s a third way to augment LLMs with private data, and that one applies more to structured data. So I’ve been talking about unstructured data when I talk about fine tuning [00:08:00] and RAG unstructured texts specifically. But let’s say you’ve got a bunch of databases with relevant data in them.
You’ve got maybe a data lakehouse with tables in it. You’ve got Salesforce data sitting behind an API, you’ve got all this structured data or even semi-structured data that you want an LLM to be able to access. Well, how do you give it access to that? And this is where agentic AI has a solution in terms of this new emerging, agentic Tool use model. So this new model where you have AI agents, which are essentially LLMs with a wrapper that have a set of tools available to them. And so basically you could have a text to SQL and a SQL query tool that you present to an AI agent. So you could have an AI agent that takes a prompt, calls its text to SQL tool or uses the LLM itself to generate a [00:09:00] SQL query from the prompt and then calls its query tool to go query your database, gets the data back from the database, the results of the query, and then feeds those in with the prompt to the LLM and then the LLM can submit out a response.
So there’s this third agentic tool use model that’s becoming the standard for augmenting LLMs with private structured data. And these two things are kind of merging together as everybody moves more towards this agentic AI model. Now you have one AI agent, so A LLM with some orchestration coding around it that tells it what tools it has access to, what its role is, what it can do that could kind of bounds the prompt. And then it might have access to a SQL Query tool. So it can query structured data. It might have access to a Salesforce API call tool. It can go call the [00:10:00] Salesforce API it might have access to an SAP API tool.
It can go talk to SAP. And then it also has access to vector database where it can perform a similarity search. So that’s RAG, but now RAG is presented to the AI agent a s just one tool and an arsenal of tools that the agent has access to. So when you hear about Agentic ai , it’s LLMs with an orchestration wrapper that have access to a set of tools, and those tools could be tools that give it access to private data.
Justin Parisi: So would you say that a agentic AI can be run basically on a per workstation basis, whereas something like a RAG.
Would run on a set of servers that people access individually?
Michael Oglesby: It could be, but it depends on what you’re doing. There’s still often a need to use bigger LLMs that basically need a bunch of GPUs to run on. Oftentimes these agentic [00:11:00] AI systems are following the model of a client application where basically like the scaffolding, the orchestration code sits, and then it talks to an LLM via an API. Essentially, it crafts a prompt, feeds it into the LLM. The LLM comes back and says, Hey, okay, here, call this tool and here’s the parameters to pass.
And so the orchestration could sit on a laptop, it could also be sitting in a platform like Salesforce or in Microsoft Copilot Studio. A gentic AI is a very broad term, but oftentimes the LLM itself. Depending on how big of a model you need to run, how much smarts you need in quotes. The LLM itself is often sitting behind an API. So that could be a VLLM instance that you host on, say, a NetApp AI pod. Or it could be a NIM that you host on a NetApp AI pod. Or it could be [00:12:00] like the Azure Open AI service. The industry has standardized on the open AI API spec.
And so all of these tools and API services I just mentioned actually serve the same API and so that makes them somewhat interchangeable in terms of being able to plug them into tools. Obviously there’s models that are exclusive to certain providers and that sort of thing. But that’s the most typical setup is the client agentic AI app might be on a laptop, might be on a general purpose server or computing environment, and it’s talking to an LLM that’s hosted somewhere else via an API.
Justin Parisi: So I guess there’s a standard, but not a standard at this time. Right? It’s kind of a moving target at this point.
Michael Oglesby: Yeah. It depends on the use case really. If you just want to do some simple tasks on your laptop, you’ve got a MacBook Pro or something like [00:13:00] that, you could deploy a 8 billion parameter model on your MacBook Pro and it’s gonna be plenty good for searching your file system or editing some text files or small things like that. But if you’re trying to really do something sophisticated, you’re gonna need a bigger model and you’re gonna run outta horsepower on the edge device.
Justin Parisi: Right. And that’s when you basically come into servers or storage solutions like a NetApp.
Michael Oglesby: Yeah. So that’s where you could subscribe to an API and pay per token. You could subscribe to OpenAI or the Azure OpenAI service, or the Vertex ai, API service or the Gemini API and pay per token. Or you could self-host, if you were seeing heavy usage and you wanted to optimize, you buy some GPUs. You buy the NetApp AI pod reference architecture either the NetApp AI POD for DGX, where we have a validated [00:14:00] certified reference architecture that pairs NetApp AFF systems with Nvidia DGX servers, or the NetApp AI pod with Lenovo, which not quite as powerful as the DGX, it’s L40s GPUs as opposed to the cutting edge B200 GPUs. So you could buy one of these, you could deploy a Nvidia NIM on it to host an open source LLM. And then you’ve got your API you can talk to.
Justin Parisi: And really it just comes down to, how much do you need? What do you wanna spend? Right. How important is it? And all those questions, I guess are, are kind of hard to answer sometimes.
Michael Oglesby: Yeah. Yeah. And honestly the beauty of our storage platforms here at NetApp is they work with open protocols, right?
And we have services in each of the major clouds and we have the ability to seamlessly and efficiently move data across different environments. So we can work with you either way. If you don’t want to [00:15:00] buy and host your own GPUs, or you don’t have the power in your data center to do that, or maybe you don’t even have a data center. You can still hook your NetApp storage either on-prem or in the cloud up to your AI application and use one of these cloud APIs or subscribe to an API service for your LLM. So the beauty of NetApp is, you have options.
Justin Parisi: Yeah. You’re not just buying Hardware, you can definitely leverage rental type of scenarios with as a service models type of things.
Michael Oglesby: Exactly.
Justin Parisi: So I know that you work specifically on the data ops toolkit for NetApp and I’d like you to tell me a little bit about what that is, if it incorporates some of the stuff that you’re talking about and if it doesn’t, what else it does.
Michael Oglesby: Yeah. So this is a good time to revisit the data ops toolkit because there’s some new stuff we’re developing that’s very relevant to this new agentic AI model. So quick [00:16:00] refresher, the data ops toolkit is a Python module. It’s open source, the code’s on GitHub, and then we also package it and push it to the PyPy repo, so you can install it with pip.
There’s two different packages. One package for Kubernetes and one package for basically any other environment. So you run pip install netapp-dataops-traditional or pip install netapp-dataops-k8s. And basically what you get when you install it is a Library with a set of simple functions that you could import into a Jupyter Notebook.
If you’re a data scientist that wants to, let’s say on demand, take a snapshot of your training data set for traceability purposes or just for backup purposes, or let’s say you’re a platform engineer putting together some automation, you could import these functions into an automation script that you’re developing [00:17:00] in Python.
These are just super simple python functions that make it easy to consume some of our key capabilities in ONTAP. So I mentioned creating a snapshot. There’s a create snapshot function. There’s a create volume function. So you could have an automation script that anytime data scientist requests a workspace, it goes and creates a volume and mounts it on some sort of client machine and gives the data scientist access to the workspace. There’s also a simple function for cloning a volume, so let’s say you have a massive data set and you’re using it to train or fine tune an AI model. You can call this function to near instantaneously using our FlexClone technology, clone that data set. So let’s say you’ve got a gold source data set and then you’re running a bunch of different experiments.
You can clone it on demand, run your experiment. Your gold source is still preserved. You could have five different people cloning it and running five different [00:18:00] experiments. In the Kubernetes flavor of the toolkit, there’s also, because it has access to the Kubernetes scheduler, it has the ability to provision apps essentially on top of the volumes.
And so it has all these same create volume, clone volume, create snapshot capabilities. But the objects it creates are Kubernetes native objects. So if you’re familiar with Kubernetes, when you create a volume, it creates a persistent volume object and a persistent volume claim object on your Kubernetes cluster.
It uses Trident, our CSI driver under the hood. And so it has all the same create volume, clone volume, create snapshot capabilities, but it also has the ability to create Jupyter Lab workspaces, which are basically it’s a web interface where data scientists and data engineers can work with Jupyter Notebooks interactively.
And so with one function call or one command [00:19:00] create a Jupyter Lab workspace that’s backed by a NetApp volume. So a data scientist could say, create me a 500 terabyte Jupyter Lab Workspace. A few seconds they get a URL they can drop into their browser. They can pull all their data into this workspace, and it looks to them like they’re working with any other Jupyter Lab workspace, but it’s all backed by NetApp.
So anything they pull in is saved on a Trident persistent volume, and then they can clone the workspace on demand. They can snapshot the workspace for backup or traceability. So because it has access to the the Kubernetes scheduler, it can provision apps like Jupyter Lab on top of the storage.
I’ve been talking about functions. We also package a Simple CLI with the toolkit. So in addition to a create volume function, you can import into any Python program or block of code, there’s a CLI and if you’re in a terminal, [00:20:00] you could run netapp-data-ops cli create volume. Mm-hmm. And it’s a quick and simple way to create a volume. So that’s what the data ops toolkit is. We originally created it as a simple way for the data scientists, data engineers, ML platform engineers of the world to take advantage of our capabilities without having to learn the ins and outs of ONTAP.
Justin Parisi: Right?
Michael Oglesby: The storage admins can carve them out an SVM or a sandbox for them to do their work in and then give them the credentials to it and they can set up their their credentials file for the DataOps toolkit, and then do whatever they want within that sandbox.
So that was the original conception of the DataOps toolkit and how it’s being used today. Where it becomes relevant to this world of agentic AI is, I’ve talked about how [00:21:00] agentic AI follows this model where you have AI agents that have access to a set of tools, right? So the DataOps toolkit has these very simple wrapper functions for administering volumes or snapshots that lend themselves to being tools that an AI agent can consume. And so we are now working on building an MCP server into the data ops toolkit that’s now gonna let an AI agent call these functions. So, this AI agent could be as simple as GitHub co-pilot chat.
So GitHub co-pilot chat supports the MCP protocol. So you could add this MCP server to your GitHub co-pilot chat running in your, vs. Code on your laptop, and you could type in the chat, create a new volume [00:22:00] for me on my NetApp system. And mount it on my server and it’ll go see that it has this create volume tool available to it from the DataOps toolkit.
It’ll invoke it, that’ll go create a volume on ONTAP and then mount it locally on the machine the user’s working on. And likewise you could say, okay, I have this volume I’m working with where I have a data set. Clone this volume for me. I want to clone my volume. And it would just go off and call that tool and clone the volume.
You could say, Hey, I’m at a stopping point. Create a snapshot of my volume for me for traceability. And it would go do that for you. If you had access to a Kubernetes cluster and you had your kubeconfig file on your laptop, you could say, Hey, go create me a Jupyter Lab workspace. It goes invokes that create Jupyter Lab tool and the LLM will come back and say, okay, I created it for you. Here’s your URL [00:23:00] you can put into your browser to access your workspace. So essentially it’s the same functions now presented as tools to a AI agent that can be prompted in natural language. So the GitHub copilot chat is the simple use case. Copilot studio, chatting with that, setting up a simple chat bot chat in front of that is another simple use case. But it could be as sophisticated as you could set up some autonomous AI agents, you know, maybe you’re using something like Salesforce and a ticket comes in and then the AI agent looks at the ticket figures out, I need to create a volume to fulfill this ticket. It has access to that MCP server. It just automatically goes and creates a volume and then it goes and fulfills the ticket without anybody ever having to take a look at it. And so you could get as sophisticated as these autonomous AI agents that are working in an [00:24:00] environment and responding to different requests that come in or different events that happen. And now they have access to some volume and snapshot administration tools. This is how we’re evolving the DataOps toolkit to now make it relevant to this new world of AI agents and I can go ahead and circle back to MCP since I mentioned this is gonna take the form of an MCP server.
So what is MCP? MCP is the model context protocol, and it was originally created by anthropic, seems like a lifetime ago, but it was earlier this year. It’s taken the world by storm. Everybody’s building an MCP server now. And it essentially it’s a common framework for defining tools and exposing them to AI agents.
So there’s a bunch of client applications out there, agentic AI tools that support MCP. So I mentioned GitHub Copilot Chat, [00:25:00] copilot studio. There’s, agent, SDK frameworks like Nvidia, agent iq land Graph, and others that support MCP. Claude Desktop is another application. Anything LLM. There’s a ton of them. And so it’s really been quickly adopted by the ecosystem as a standard way to present a set of tools defined in code to AI agent.
Justin Parisi: So I know the data ops toolkit is intended for AI use cases.
Could it extend out into other use cases like version control, source control type of stuff?
Michael Oglesby: Absolutely. Yeah. And we actually have a couple customers using it for those types of use cases. We intentionally called it the data ops toolkit and not the ML ops toolkit or AI ops toolkit because we knew there would probably be some applicability outside of AI use cases. These AI use cases were the inspiration for it. But we’ve got a customer in [00:26:00] the semiconductor space actually that’s using it to manage dev workspaces and build workspaces. And then we have a customer in the public sector space that’s using it to manage dev workspaces and that kind of thing. We’ll see more of that because this MCP server that we’re developing can expose some of these capabilities to really any type of AI agent.
Justin Parisi: Yeah, actually I talked to a customer today that this probably would fit, so I’ll point them your way.
Michael Oglesby: Yeah, sounds good. Yeah, and in the initial release of this MCP server, the only tools available will be these simple volume, snapshot clone type functions that are available in the data Ops toolkit today, but we do plan to build out more tools and we’re actively soliciting feedback from customers on Right tools they’d like to see. So would love to have a conversation with them.
Justin Parisi: Yeah, absolutely.
Michael Oglesby: Or, and anybody feel free to reach out. Let me know [00:27:00] what tools you’d like to see.
Justin Parisi: Are you leveraging consistency groups at all where you can consolidate snapshots across multiple volumes?
Michael Oglesby: No, we’re not doing that now, but that’s a good idea for another tool to add.
Justin Parisi: Yeah. There you go. Yeah. So I know that one of the challenges to these workloads is performance and some of that performance is outta your control ’cause it’s basically your infrastructure. Some of it is in your control and there’s ways to get around some of those issues through things like caching and that sort of thing.
Tell me a little bit about the work you’ve done in that area.
Michael Oglesby: Yeah, so optimizing LLM inference performance is a hot topic these days because nobody can get their hands on enough GPUs. And so everybody’s trying to make sure they’re fully utilizing and optimizing the GPUs that they have.
And so, you want to make sure you’re optimizing your workloads so that you can use as few GPUs as [00:28:00] possible to host them. So what that means for LLM inference is there’s things you can do. VLLM has become very popular as an inference framework because of its performance. It’s very performant. And then there’s also parameters you can tune like the batch size and the max sequence link, depending on your use case to optimize utilization. But then another topic that that’s become very popular recently is KV cache offloading. Basically, when an LLM is performing inference. So when it’s processing a prompt for each new token that comes in, it has to calculate what’s called an attention value. And we could probably spend a whole hour talking about what attention is and all the different parts of an LLM.
The important thing to know is just for every new token, there’s an attention value that needs to be calculated and to [00:29:00] calculate that attention value, you need all the previous attention values from all the previous tokens in the sequence. So let’s say you submit a long prompt, and then the LLM spits out a long response to it. You know, for each additional token in your prompt, as it’s processing your prompt, it has to take the previous attention values into account. And then when it starts generating its response, there’s an attention value for each new token. And so recalculating all these attention values that have already been calculated, that would be a lot of wasted compute resources. So you want to store these somewhere, right? So you can reuse them for each new token in the sequence. And so the concept that has become the standard is something called a KV cache. What folks have been doing is basically carving out a section of GPU [00:30:00] memory for a key value cache, and essentially it’s as simple as it sounds. It’s a bunch of values with keys associated with them, and so all those previous attention values are stored in this KV cache and GPU memory, and for each new token that comes in, they’re quickly retrieved so that the new attention value can be calculated.
But there’s a massive drawback here, and that’s the GPU memory is limited, so you are sacrificing some of your GPU memory. And using it to store this KV cache in order to save on compute intensity. So it’s kind of a trade off between GPU memory and compute intensity.
GPU memory is important because when you perform inference on an LLM, the entire model has to be loaded into memory and these models can get pretty big. If we’re talking. A 70 billion parameter model, you often need to [00:31:00] host that across like four GPUs to fit it in VAM, maybe even more depending on how you tune your parameters. The more of your memory you dedicate to a KV cache, then that’s less memory that you can use to host your model. If you were hosting a model without a KV cache, maybe you use two GPUs. You know, if we’re talking about a small model, when you introduce a KV cache, you might now need four GPUs. Between compute intensity and GPU memory utilization. And so using the KV cache might make the LLM run faster and you need it to make the performance acceptable. But it causes you to need more GPUs. And so there’s been an interest in being able to shrink the amount of GPU memory or VRAM that’s used for the KV cache and essentially offload the KV cache or a portion of it to an [00:32:00] an off GPU target.
So that could be CPU memory, that’s something that some folks are doing, but it could also be really fast external storage. Mm-hmm. That’s also something that a lot of folks are interested in doing or have already implemented. And so I mentioned VLLM which has emerged as a popular inference framework because of its performance characteristics. There’s a project called LM Cache which you can think of as a plugin for VLLM. And they have become the standard for essentially adding KV cache offloading to VLLM. There’s a bunch of different offload targets that they support. The two most interesting in my opinion, are CPU Memory and a GDS backend. GDS stands for GPU Direct storage. That’s NVIDIA’s framework for allowing a GPU to talk directly to [00:33:00] external storage, bypassing the CPU. It uses RDMA. They support some other external storage targets, but this one’s particularly interesting because when you use GPU direct storage with really fast external storage, there’s not that much latency. what’s interesting for us here at NetApp is it supports any GPU direct storage file system that’s mounted on the client and at NetApp we support GPU direct storage over NFS without any proprietary client, any sort of proprietary file system client or drop in replacement NFS client.
You can use Linux, you can use the standard NFS client. Just set the appropriate mount options, enable the right settings on your ONTAP box, and you can use GPU direct storage to talk to an ONTAP system over an NFS Mount. And so [00:34:00] what this means is with VLLM and LM cache we can support KV cache offloading using industry standard tools and protocols.
No weird one off proprietary or potentially unstable file system clients. Just industry standard tooling on both the inference and storage side, and it just works. And so this is something we recently validated and that I wrote up a blog on. So you can find that on the Tech ONTAP community blog site.
It just walks through how to implement KV cache offloading to ONTAP using GDS with VLLM and LM cache. It’s pretty easy. You just sit, sounds pretty easy. Yeah, that’s the cool thing about it. And it just works.
Justin Parisi: Would the KV cache be something that’s worth snapshotting or is that so ephemeral that it doesn’t really matter?
Michael Oglesby: You know, it’s a good [00:35:00] question and that’s something I’ve thought about. I have not seen that use case come up yet, but I could see it being worthwhile. Especially in a world of AI agents that have ongoing conversations with an LLM basically, where the context can get pretty long, but it’s all part of the same conversation. At least snapshotting it for DR purposes could probably be useful if something goes down.
You could fail over to a different Region, bring the whole environment back up and the KV cache is already there. Or maybe if something gets funky, you can restore your whole environment. You could even have something like Trident Protect running in your Kubernetes cluster, backing up your VLLM instance.
And if something gets wonky, you just restore, the snapshot for the volume that the KV cache is being offloaded to, and [00:36:00] it just picks it right back up where it left off. And a new API request comes in that KV cache is already there.
Justin Parisi: What about something like a flexcache that hangs off of that volume that can localize those KV caches to cloud instances?
Michael Oglesby: Yeah, that could also be interesting as we get into distributed inference environments where perhaps there’s different GPU farms all serving the same API endpoint or something like that where you want to share that KV cache so that you can route requests to wherever you have GPUs available. I haven’t seen this stuff being used in real life yet, but I think there’s a bunch of different interesting things you could do and I think we’re kind of just scratching the surface here. So I look forward to seeing all this stuff get built out as our customers move further down this path.
Justin Parisi: Is there any advantage to leveraging block devices for this? Or is it just kind of like you want to have a shared storage like [00:37:00] NFS?
Michael Oglesby: You really want to have a shared storage because when you have a multi-node deployment, you want that same KV, especially if you’re splitting one model across multiple nodes, you want that same KV cache to be available on both nodes.
And that’s one of the benefits of offloading to external storage versus CPU memory is you can share that KV cache across nodes. And so you know, you don’t have to unnecessarily recalculate attention values that have already been calculated on a different node.
Justin Parisi: Okay. Makes sense. You can share all the values across all the clients.
Michael Oglesby: Yeah.
Justin Parisi: With Flex Clone, you could also spread that workload across multiple CPUs and your cluster, right? So that’s if your volume is FlexVol and it’s on a single node, and that’s where your KV cache is, you could have KV cache copies across all the nodes. That way you have redundancy, you have better performance [00:38:00] potentially.
Michael Oglesby: Yeah. For the validation, we used FlexGroup with PNFS over RDMA.
Justin Parisi: Right.
Michael Oglesby: And so to make sure we were fully optimizing performance on the ONTAP side. If you’re familiar with the AI pod for DGX reference architecture, we use the mount options that are recommended in the best practices section of that reference architecture. You have to use RDMA to use GDS. So that is one drawback. You have to have an environment that supports RoCE, RDMA over converged ethernet. If you don’t have that, you can’t use the GDS backend and LM cache.
There’s probably other ways you could implement the offloading. Your performance probably won’t be quite as good ’cause the GPU’s not gonna be able to talk directly to the storage.
Justin Parisi: How big do these caches get?
Michael Oglesby: So they can get pretty massive when we’re talking about a consumer service like chat GPT. There aren’t a lot of hard [00:39:00] numbers coming out of those guys. But they’re maintaining a lot of simultaneous conversations. I know I go into chat GPT and I keep asking things in the same thread, and then I might go away and then I come back to it and I ask a follow up.
And so with those big consumer oriented deployments they can get pretty massive. If you have a long context and you’re including a large amount of source material in your prompt to the LLM, then they can get fairly big there too.
But we’re still probably talking about a different scale than a chat GPT. The tricky thing is we haven’t really seen how big they can get because we’ve been bound by GPU memory. And so what folks have been doing is just throwing out old stuff and then they have to recalculate it, so as we get into optimizing things further and offloading KV caches in production, I think we’ll see these [00:40:00] things probably grow quite large.
Justin Parisi: So is a KV cache associated with a single token or multiple tokens. Is it like a one-to-one or..?
Michael Oglesby: An attention value is associated with a single token. Okay. But the KV cache contains all the previous attention values.
Justin Parisi: Okay. So it’s not a, for all
Michael Oglesby: the previous tokens.
Justin Parisi: Is it per session? Like I say, I open up a chat GPT session, do I get a KV cache with that and then someone else opens up a chat GPT session somewhere else? Is that an additional KV cache or are they sharing that same cache?
Michael Oglesby: So there is some sharing that goes on. But for the most part, because each attention value is calculated with all the previous attention values, your attention values typically aren’t useful in someone else’s conversation. Unless they ask the exact same thing.
Justin Parisi: Right.
Michael Oglesby: Have the exact same conversation, so while they all might be stored in the same KV cache, what’s gonna be useful in processing your [00:41:00] stream is all the previous attention values from your stream, that are in that KV cache.
Justin Parisi: Okay. So the data ops toolkit incorporates KV cache offloading, I assume, with the all the pieces like the NFS over our RDMA and all that stuff.
Right?
Michael Oglesby: You could use the DataOps toolkit to provision the volume that you want to offload your KV cache to, And you could also use it to mount that volume if you wanted to. It can take custom mount options, and you could use it to clone that volume if you wanted to clone it, to set up another deployment somewhere.
But what someone would typically do is deploy their VLLM using helm and just specify a specific PVC that they want mounted to it, and then specify all the right parameters in their config file to offload the KV cache to that PVC.
And then that PV C’s gonna remain, at least for that specific deployment. [00:42:00] Fairly static for the life of the the VLLM instance. Gotcha.
Justin Parisi: So it sounds like when you say start up a chat GPT instance, it spins up a Kubernetes pod that creates a persistent volume claim that creates a volume and use the KV cache and that then it goes on its merry way.
Is that kind of the workflow there?
Michael Oglesby: Yeah. Yeah. If they were using VLLM, that’s how it would work. There’s not a lot of details out there on how chat GPT itself works, but yeah your typical enterprise deployment is gonna have either VLLM or NIM deployed in a Kubernetes cluster. If it’s a multi-user deployment where usage really scales up and down, they’ve probably got a horizontal pod autoscaler that’ll scale it out a s more users are hitting it. And then essentially, yeah, there would be a volume that’s mounted in each of those pods, and it would be part of the pod spec where the KV cache would be offloaded to, and each new pod that comes up would have [00:43:00] access to the existing KV cache.
Justin Parisi: Then when they’re done, it tears it down and then deletes the volume and they just keep going.
Michael Oglesby: Yeah. If you tore down the whole deployment, but typically it would scale down to some small amount of pods. Right? Like maybe one volume still there. Gotcha. Typically these things don’t go away. They just scale up and down based on usage.
Justin Parisi: All right. Sounds like a lot of information there.
I don’t know that I’ve absorbed it all. Luckily we’ll have a transcript. You can go in and we could probably feed it into an LLM and then it can make it nice and pretty and all that good stuff.
Michael Oglesby: Yeah. The, the LLM could probably explain it better and more concisely than me. Yeah.
Justin Parisi: We’re gonna do that. We’re gonna feed it into an LLM. Say, Hey, dumb this down for me. Right. So if I wanted to find the DataOps toolkit or the blog, where’s the best place to the Tech ONTAP community, right?
Michael Oglesby: Yeah, so the Tech ONTAP community is where you can find our LLM Inference blog series, which is an ongoing series.
But we’ve got some recent posts on how to [00:44:00] deploy VLLM on Kubernetes with a chatbot front end, and then how to add KV cache offloading to ONTAP with GDS, how to add that to that deployment. So that’s out there on the tech ONTAP community site and then the DataOps toolkit if you just google NetApp DataOps toolkit, the GitHub repo should be your first result.
Justin Parisi: Okay.
Michael Oglesby: And so you just go out to our GitHub repo. The first thing you’ll see is the Read Me and that has all the information you’ll need to install it and start using it.
If you already have an ONTAP system, or one of our cloud services, then you could go install it and use it. It’s open source. It’s free,
Justin Parisi: yeah. Excellent. And the support for it, is that kind of like ad hoc, like how does that work?
Michael Oglesby: It’s community support via GitHub.
Yeah. So if you have an issue, just submit it to us on GitHub. We monitor it every day, we’ll get back to you.
Justin Parisi: Okay. Excellent. And are you on the NetApp discord as [00:45:00] well or are you not?
Michael Oglesby: I am on the NetApp Discord. Yeah. I primarily hang out in the artificial intelligence channel on there, so if you go there, you’ll see some posts from me there.
And feel free to reach out on there. And as we continue to post blogs in this LLM Inference blog series, I typically drop the link in there as well.
Justin Parisi: I assume we’ll see you at Insight.
Michael Oglesby: We’ll see. We’ll see. We shall, I assume so. I assume so.
Justin Parisi: We haven’t decided who’s going, but I assume that both of us will be there.
Michael Oglesby: Yeah, I assume so.
Justin Parisi: Alright, excellent. Well, Mike, again, thank you for coming by and talking to us and being on video and all that good stuff. And if you wanna check out the data ops toolkit, feel free to do that. And of course techontappodcast.com. Yeah, that’s all I got.
So thanks again, Mike.
Michael Oglesby: Yeah. Enjoyed it.
