Large Language Models (LLMs) are the heart of Agentic systems and RAG systems. And building with LLMs is exciting until the scale makes them expensive. There is always a tradeoff for cost vs quality, but in this article we will explore the 10 best ways according to me that can slash costs for the LLM usage while focusing on maintaining the quality of the system. Also note I\u2019ll be using OpenAI API for the inference but the techniques could be applied to other model providers as well. So without any further ado let\u2019s understand the cost equation and see ways of LLM cost optimization.\u00a0<\/p>\n
Prerequisite: Understanding the Cost Equation<\/h2>\n
Before we start, it\u2019s better we get better versed with about costs, tokens and context window:\u00a0<\/p>\n
- \n
- Tokens:<\/strong> These are the small units of the text. For all practical purposes you can assume 1,000 tokens is roughly 750 words.\u00a0\u00a0<\/li>\n
- Prompt Tokens: <\/strong>These are the input tokens that we send to the model. They are generally cheaper.\u00a0<\/li>\n
- Completion Tokens: <\/strong>These are the tokens generated by the model. They are often 3-4 times more expensive than input tokens.\u00a0<\/li>\n
- Context Window:<\/strong> This is like a short-term memory (It can include the old Inputs + Outputs). If you exceed this limit, the model leaves out the earlier parts of the conversation. If you send 10 previous messages in the context window, then those count as Input Tokens for the current request and will add to the costs.\u00a0<\/li>\n
- Total Cost:<\/strong> (Input Tokens x Per Input Token Cost) + (Output Tokens x Per Output Token Cost)\u00a0<\/li>\n<\/ul>\n
Note<\/strong>: For OpenAI you can use the billing dashboard to track costs: https:\/\/platform.openai.com\/settings\/organization\/billing\/overview<\/a>\u00a0<\/p>\n
To learn how to get the OpenAI API read this<\/a> article. <\/p>\n
1. Route Requests to the Right Model<\/h2>\n
Not every task requires the best, state-of-the-art model, you can experiment with a cheaper model or try using a few-shot prompting with a cheaper model to replicate a bigger model.\u00a0\u00a0<\/p>\n
Configure the API key\u00a0\u00a0<\/h4>\n
from google.colab import userdata\u00a0\nimport os\u00a0\n\nos.environ['OPENAI_API_KEY']=userdata.get('OPENAI_API_KEY')\u00a0<\/code><\/pre>\nDefine the functions\u00a0<\/h4>\n
from openai import OpenAI\u00a0\n\nclient = OpenAI()\u00a0\nSYSTEM_PROMPT = \"You are a concise, helpful assistant. You answer in 25-30 words\"\u00a0\n\ndef generate_examples(questions, n=3):\u00a0\n\u00a0\u00a0\u00a0examples = []\u00a0\n\u00a0\u00a0\u00a0for q in questions[:n]:\u00a0\n\u00a0\u00a0\u00a0 response = client.chat.completions.create(\u00a0\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 model=\"gpt-5.1\",\u00a0\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 messages=[{\"role\": \"system\", \"content\": SYSTEM_PROMPT},\u00a0\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0{\"role\": \"user\", \"content\": q}]\u00a0\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0)\u00a0\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0examples.append({\"q\": q, \"a\": response.choices[0].message.content})\u00a0\n\n\u00a0\u00a0\u00a0return examples<\/code><\/pre>\nThis function uses the larger GPT-5.1<\/a> and answers the question in 25-30 words.\u00a0\u00a0<\/p>\n
# Example usage\u00a0\n\nquestions = [\u00a0\n\u00a0\u00a0\u00a0\"What is overfitting?\",\u00a0\n\u00a0\u00a0\u00a0\"What is a confusion matrix?\",\u00a0\n\u00a0\u00a0\u00a0\"What is gradient descent?\"\u00a0\n]\u00a0\n\nfew_shot = generate_examples(questions, n=3)<\/code><\/pre>\n\n![\"Few_shot](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/12\/word_media_image1-1-1.webp\")
<\/figure>\n<\/div>\nGreat, we got our question-answer pairs.\u00a0\u00a0<\/p>\n
def build_prompt(examples, question):\u00a0\n\n\u00a0\u00a0\u00a0prompt = \"\"\u00a0\n\u00a0\u00a0\u00a0for ex in examples:\u00a0\n\u00a0\u00a0\u00a0 prompt += f\"Q: {ex['q']}\\nA: {ex['a']}\\n\\n\"\u00a0\n\u00a0\u00a0\u00a0return prompt + f\"Q: {question}\\nA:\"\u00a0\n\ndef ask_small_model(examples, question):\u00a0\n\n\u00a0\u00a0\u00a0prompt = build_prompt(examples, question)\u00a0\n\u00a0\u00a0\u00a0response = client.chat.completions.create(\u00a0\n\n\u00a0\u00a0\u00a0 model=\"gpt-5-nano\",\u00a0\n\u00a0\u00a0\u00a0 messages=[{\"role\": \"system\", \"content\": SYSTEM_PROMPT},\u00a0\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0{\"role\": \"user\", \"content\": prompt}]\u00a0\n\u00a0\u00a0\u00a0)\u00a0\n\n\u00a0\u00a0\u00a0return response.choices[0].message.content<\/code><\/pre>\nHere, we have a function that uses smaller \u2018gpt-5-nano\u2019 and another function that makes the prompt using the question-answer pairs for the model.\u00a0\u00a0<\/p>\n
answer = ask_small_model(few_shot, \"Explain regularization in ML.\")\u00a0\n\nprint(answer)<\/code><\/pre>\nLet\u2019s pass a question to the model.\u00a0<\/p>\n
Output<\/strong>:<\/p>\n
Regularization adds a penalty to the loss for model complexity to reduce overfitting. Common forms include L1 (lasso) promoting sparsity and L2 (ridge) shrinking weights; elastic net blends.\u00a0<\/pre>\n
Great! We have used a much cheaper model (gpt-5-nano) to get our output, but surely we can\u2019t use the cheaper model for every task.\u00a0\u00a0<\/p>\n
2. Use Models according to the task<\/h2>\n
The idea here is to use a smaller model for routine tasks, and using the larger models only for complex reasoning. So how do we do this? Here we will define a classifier that returns \u201csimple\u201d or \u201ccomplex\u201d and route the queries accordingly. This is help us save costs on routine costs.\u00a0\u00a0<\/p>\n
Example:\u00a0<\/strong><\/p>\n
from openai import OpenAI\u00a0\n\nclient = OpenAI()\u00a0\n\ndef get_complexity(question):\u00a0\n\n\u00a0\u00a0\u00a0prompt = f\"Rate the complexity of the question from 1 to 10 for an LLM to answer. Provide only the number.\\nQuestion: {question}\"\u00a0\n\n\u00a0\u00a0\u00a0res = client.chat.completions.create(\u00a0\n\u00a0\u00a0\u00a0 model=\"gpt-5.1\",\u00a0\n\u00a0\u00a0\u00a0 messages=[{\"role\": \"user\", \"content\": prompt}],\u00a0\n\u00a0\u00a0\u00a0\u00a0)\u00a0\n\n\u00a0\u00a0\u00a0return int(res.choices[0].message.content.strip())\u00a0\n\nprint(get_complexity(\"Explain convolutional neural networks\"))<\/code><\/pre>\nOutput:<\/strong><\/p>\n
4\u00a0<\/pre>\n
So our classifier says the complexity is 4, don\u2019t worry about the extra LLM call as this is generating only a single number. This complexity number can be used to route the tasks,\u00a0like: complexity < 7<\/em> then route to a smaller model, else a larger model.\u00a0\u00a0\u00a0<\/p>\n
3. Using Prompt Caching<\/h2>\n
If the LLM-system uses bulky system instructions or lots of few-shot examples across many calls, then make sure to place them at the start<\/em> of your message.<\/p>\n
Few important points here:\u00a0<\/p>\n
- \n
- Ensure the prefix is exactly identical across requests (including all the characters, whitespace included).\u00a0<\/li>\n
- According to OpenAI the supported models will automatically benefit from Caching but the prompt has to be longer than 1,024 tokens.\u00a0<\/li>\n
- Requests using Prompt Caching have a
cached_tokens<\/code> value as a part of the response.\u00a0<\/li>\n<\/ul>\n4. Use the Batch API for Tasks that can wait<\/h2>\n
Many tasks don\u2019t require immediate responses, this is where we can use the asynchronous Batch endpoint for the inference. By submitting a file of requests and giving OpenAI upto 24 hours time to process them, will reduce 50% costs on token costs compared to the usual OpenAI API calls.\u00a0<\/p>\n
5. Trim the Outputs with max_tokens and Stops parameters<\/h2>\n
What we\u2019re trying to do here is stop the untrolled token generation, Let\u2019s say you need a 75-word summary or a specific JSON object, don\u2019t let the model keep generating unnecessary text. Instead we can make use of the parameters:<\/p>\n
Example:<\/strong><\/p>\n
from openai import OpenAI\u00a0\nclient = OpenAI()\u00a0\n\nresponse = client.chat.completions.create(\u00a0\n\u00a0\u00a0\u00a0model=\"gpt-5.1\",\u00a0\n\u00a0\u00a0\u00a0messages=[\u00a0\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0{\u00a0\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\"role\": \"system\",\u00a0\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\"content\": \"You are a data extractor. Output only raw JSON.\"\u00a0\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0}\u00a0\n\u00a0\u00a0\u00a0],\u00a0\n\u00a0\u00a0\u00a0max_tokens=100,\u00a0\n\u00a0\u00a0\u00a0stop=[\"\\n\\n\", \"}\"]\u00a0\n)<\/code><\/pre>\nWe have set max_tokens as 100 as it\u2019s roughly 75 words.\u00a0\u00a0<\/p>\n
6. Make Use of RAG<\/h2>\n
Instead of flooding the context window, we can use Retrieval-Augmented Generation. This will help convert the knowledge base into embeddings and store them in a vector database. When a user queries, then all the context won\u2019t be in the context window but the retrieved top few relevant text chunks will be passed for context.\u00a0\u00a0<\/p>\n
\n![\"RAG](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/12\/word_media_image2-1-1.webp\")
<\/figure>\n<\/div>\n7. Always Manage the Conversation History<\/h2>\n
Here our focus is on the conversation history where we pass the older inputs and outputs. Instead of iteratively adding the conversations we can implement a \u201csliding window\u201d approach.\u00a0\u00a0<\/p>\n
Here we drop the oldest messages once the context gets too long (set a threshold), or summarize previous turns into a single system message before continuing. Ensure that the active context window is not too long as it\u2019s crucial for long-running sessions.\u00a0<\/p>\n
Function for summarization\u00a0<\/h3>\n
from openai import OpenAI\u00a0\n\nclient = OpenAI()\u00a0\n\nSYSTEM_PROMPT = \"You are a concise assistant. Summarize the chat history in 30-40 words.\"\u00a0\n\ndef summarize_chat(history_text):\u00a0\n\n\u00a0\u00a0\u00a0response = client.chat.completions.create(\u00a0\n\u00a0\u00a0\u00a0 model=\"gpt-5.1\",\u00a0\n\u00a0\u00a0\u00a0 messages=[\u00a0\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0{\"role\": \"system\", \"content\": SYSTEM_PROMPT},\u00a0\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0{\"role\": \"user\", \"content\": history_text}\u00a0\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0]\u00a0\n\u00a0\u00a0\u00a0)\u00a0\n\n\u00a0\u00a0\u00a0return response.choices[0].message.content<\/code><\/pre>\nInference\u00a0<\/h3>\n
chat_history = \"\"\"\u00a0\n\nUser: Hi, I'm trying to understand how embeddings work.\u00a0\nAssistant: Embeddings turn text into numeric vectors.\u00a0\n\nUser: Can I use them for similarity search?\nAssistant: Yes, that\u2019s a common use case.\u00a0\n\nUser: Nice, show me simple code.\u00a0\nAssistant: Sure, here's a short example...\u00a0\n\n\"\"\"\u00a0\n\nsummary = summarize_chat(chat_history)<\/code><\/pre>\nUser asked what embeddings are; assistant explained they convert text to numeric vectors. User then asked about using embeddings for similarity search; assistant confirmed and provided a short example code snippet demonstrating basic similarity search.\u00a0<\/p>\n
We now have a summary which can be added to the model\u2019s context window when the input tokens are above a defined threshold.\u00a0<\/p>\n
8. Upgrade to Efficient Model Modes<\/h2>\n
OpenAI frequently releases optimized versions of their models. Always check for newer \u201cMini,\u201d or \u201cNano\u201d variants of the latest models. These are specifically made for efficiency, often delivering similar performance for certain tasks at a fraction of the cost.<\/p>\n
\n![\"Upgrade](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/12\/word_media_image3-1.webp\")
<\/figure>\n<\/div>\n9. Enforce Structured Outputs (JSON)<\/h2>\n
When you need data extracted or formatted. Defining a strict schema forces the model to cut the unnecessary tokens and returns only the exact data fields requested. Denser responses mean fewer generated tokens on your bill.\u00a0<\/p>\n
Imports and Structure Definition\u00a0\u00a0<\/h3>\n
from openai import OpenAI\u00a0\n\nimport json\u00a0\n\nclient = OpenAI()\u00a0\n\nprompt = \"\"\"\u00a0\nYou are an extraction engine. Output ONLY valid JSON.\u00a0\nNo explanations. No natural language. No extra keys.\u00a0\n\nExtract these fields:\u00a0\n\n- title (string)\n- date (string, format: YYYY-MM-DD)\u00a0\n- entities (array of strings)\u00a0\n\nText:\u00a0\n\n\"On 2025-12-05, OpenAI introduced Structured Outputs, allowing developers to enforce strict JSON schemas. This improved reliability was welcomed by many engineers.\"\u00a0\n\nReturn JSON in this exact format:\u00a0\n\n{\u00a0\n\u00a0\"title\": \"\",\u00a0\n\u00a0\"date\": \"\",\u00a0\n\u00a0\"entities\": []\u00a0\n}\u00a0\n\n\"\"\"<\/code><\/pre>\nInference\u00a0<\/h3>\n
response = client.chat.completions.create(\u00a0\n\u00a0\u00a0\u00a0model=\"gpt-5.1\",\u00a0\n\u00a0\u00a0\u00a0messages=[{\"role\": \"user\", \"content\": prompt}]\u00a0\n)\u00a0\n\ndata = response.choices[0].message.content\u00a0\n\njson_data = json.loads(data)\u00a0\n\nprint(json_data)<\/code><\/pre>\nOutput:<\/strong><\/p>\n
{'title': 'OpenAI Introduces Structured Outputs', 'date': '2025-12-05', 'entities': ['OpenAI', 'Structured Outputs', 'JSON', 'developers', 'engineers']}\u00a0<\/pre>\nAs we can see only the required dictionary with the required details is returned. Also the output is neatly structured as key-value pairs.\u00a0<\/p>\n
10. Cache Queries<\/h2>\n
Unlike our earlier idea of caching, this is quite different. If the users frequently ask the exact same questions, cache the LLM\u2019s response in your own database. Check this database before calling the API. This cached response is faster for the user and is practically free. Also if working with LangGraph for Agents then you can explore this for Node-level-caching:\u00a0Caching in LangGraph<\/a><\/p>\n
Conclusion\u00a0<\/h2>\n
Building with LLMs is powerful but the scale can quickly make them expensive, so understanding the cost equation becomes essential.By applying the right mix of model routing, caching, structured outputs, RAG, and efficient context management, we can significantly slash inference costs. These techniques help maintain the quality of the system while ensuring the overall LLM usage remains practical and cost-effective. Don\u2019t forget to take check the billing dashboard for the costs after implementing each technique.<\/p>\n
Frequently Asked Questions\u00a0<\/h2>\n
\nQ1. What is a token in the context of LLMs?<\/strong><\/strong> <\/p>\nA. A token is a small unit of text, where roughly 1,000 tokens correspond to about 750 words.\u00a0<\/p>\n<\/p><\/div>\n
Q2. Why are completion tokens more costly than prompt tokens?<\/strong><\/strong> <\/p>\nA. Because output tokens (from the model) are often several times more expensive per token than input (prompt) tokens.\u00a0<\/p>\n<\/p><\/div>\n
Q3. What is the \u201ccontext window\u201d and why does it matter for cost?<\/strong><\/strong> <\/p>\nA. The context window is the short-term memory (previous inputs and outputs) sent to the model; a longer context increases token usage and thus cost.\u00a0<\/p>\n<\/p><\/div>\n<\/p><\/div>\n
- Prompt Tokens: <\/strong>These are the input tokens that we send to the model. They are generally cheaper.\u00a0<\/li>\n
![\"Mounish](\"https:\/\/av-eks-lekhak.s3.amazonaws.com\/media\/lekhak-profile-images\/converted_image_ZFxQ96b.webp\")
<\/p>\n