Imagine you are building a customer support AI<\/a> that needs to answer questions about your product. Sometimes it needs to pull information from your documentation, while other times it needs to search the web for the latest updates. Agentic RAG systems<\/a> come in handy in such types of complex AI applications. Think of them as smart research assistants who not only know your internal documentation but also decide when to go to search the web. In this guide, we will walk through the process of building an agentic QA RAG system using the Haystack framework.<\/p>\n This article was published as a part of the\u00a0<\/strong><\/em>Data Science Blogathon.<\/strong><\/em><\/a><\/p>\n An agentic LLM is an AI system that can autonomously make decisions and take actions based on its understanding of the task. Unlike traditional LLMs that mainly generate text responses, an agentic LLM can do a lot more. It can think, plan, and act with minimal human input. It assesses its knowledge, recognizing when it needs more information or external tools. <\/span>Agentic LLMs don\u2019t rely on static data or indexed knowledge, instead, they decide which sources to trust and how to gather the best insights.<\/span><\/p>\n This type of system can also pick the right tools for the job. It can decide when it needs to retrieve documents, run calculations, or automate tasks. What sets them apart is its ability to break down complex problems into steps and execute them independently which makes it valuable for research, analysis, and workflow automation.<\/span><\/p>\n Traditional RAG systems follow a linear process. When a query is received, the system first identifies the key elements within the request. It then searches the knowledge base, scanning for relevant information that can help design an accurate response. Once the relevant information or data is retrieved, the system processes it to generate a meaningful and contextually relevant response.<\/span><\/p>\n You can understand the processes easily by the below diagram.<\/span><\/p>\n Now, an agentic RAG system enhances this process by:<\/p>\n The key difference<\/a> lies in the system\u2019s ability to make intelligent decisions about how to handle queries, rather than following a fixed retrieval-generation pattern.<\/p>\n Haystack is an open-source framework for building production-ready AI, LLM applications, RAG pipelines, and search systems. It offers a powerful and flexible framework for building LLM applications. It allows you to integrate models from various platforms such as Huggingface, OpenAI, CoHere, Mistral, and Local Ollama. You can also deploy models on cloud services like AWS SageMaker<\/a>, BedRock<\/a>, Azure<\/a>, and GCP<\/a>. <\/span><\/p>\n Haystack provides robust document stores for efficient data management. It also comes with a comprehensive set of tools for evaluation, monitoring, and data integration which ensure smooth performance across all layers of your application. It also has strong community collaboration which makes new service integration from various service providers periodically.<\/span><\/p>\n Haystack has two primary concepts for building fully functional GenAI LLM systems \u2013 components and pipelines. Let\u2019s understand them with a simple example of RAG on Japanese Anime Characters<\/p>\n Components are the core building blocks of Haystack. They can perform tasks such as document storing, document retrieval, text generation, and embedding. Haystack has many components you can use directly after installation, it also provides APIs for making your own components by writing a Python class.<\/p>\n There is a collection of integration from partner companies and the community.<\/p>\n Install Libraries and set Ollama<\/a><\/strong><\/p>\n Import some components<\/b><\/p>\n Create a document and document store<\/p>\n Pipelines are the backbone of Haystack\u2019s framework. They define the flow of data between different components. Pipelines are essentially a Directed Acyclic Graph (DAG). A single component with multiple outputs can connect to another single component with multiple inputs.<\/p>\n You can define pipeline by<\/p>\n You can visualize the pipeline<\/p>\n The pipeline provides:<\/p>\n Nodes are the basic processing units that can be connected in a pipeline these nodes are the components that perform specific tasks.<\/p>\n Examples of nodes from the above pipeline<\/p>\n The connection\u00a0graph defines how components interact.<\/p>\n From the above pipeline, you can visualize the connection graph.<\/p>\n The connection graph of the anime pipeline<\/p>\n This graph structure:<\/p>\n Now we can query our anime knowledge base using the prompt.<\/p>\n Create a prompt template<\/strong><\/p>\n This prompt will provide an answer taking information from the document base.<\/p>\n Query using prompt and retriever<\/p>\n Response:<\/strong><\/p>\n This RAG is simple yet conceptually valuable to the newcomer. Now that we have understood most of the concepts of Haystack frameworks, we can deep dive into our main project. If any new thing comes up I will explain along the way.<\/p>\n We will build an NCERT Physics books-based Question Answer RAG for higher secondary students. It will provide answers to the query by taking information from the NCERT books, and If the information is not there it will search the web to get that information. I use a free, totally localized system.<\/p>\n We will setup a conda env Python 3.12<\/p>\n Now create a project directory named qagent<\/b>.<\/p>\n You can use plain Python files for the project or Jupyter Notebook for the project it does not matter. I will use a plain Python file.<\/p>\n Create a main.py<\/b> file on the project root.<\/p>\n Document store is the most important here we will store our embedding for retrieval, we use ChromaDB <\/b>for the embedding store, and as you may see in the earlier example, we use InMemoryDocumentStore for fast retrieval because then our data was tiny but for a robust system of retrieval we don\u2019t rely on the InMemoryStore, it will hog the memory and we will have creat embeddings every time we start the system.<\/p>\n The solution is a Vector database such as Pinecode, Weaviate, Postgres Vector DB, or ChromaDB. I use ChromaDB because free, open-source, easy to use, and robust.<\/p>\n persist_path<\/b> is where you want to store your embedding.<\/p>\n PDF files path<\/b><\/p>\n It will create a list of files from the data folder which consists of our PDF files.<\/p>\n We will use Haystack\u2019s built-in document preprocessor such as cleaner, splitter, and file converter, and then use a writer to write the data into the store.<\/p>\n Cleaner: <\/b>It will clean the extra space, repeated lines, empty lines, etc from the documents.<\/p>\n Splitter: <\/b>It will split the document in various ways such as words, sentences, para, pages.<\/p>\n File Converter: <\/b>It will use the pypdf to convert the pdf to documents.<\/p>\n Writer: <\/b>It will store the document where you want to store the documents and for duplicate documents, it will overwrite with previous one.<\/p>\n Now set the embedder for document indexing.<\/p>\n Embedder: Nomic Embed Text<\/b><\/p>\n We will use nomic-embed-text embedder which is very effective and free inHuggingface and Ollama.<\/p>\n Before you run your indexing pipeline open your terminal and type below to Pull the nomic-embed-text and llama3.2:3b model from the Ollama model store<\/p>\n and start Ollama by typing the command ollama serve <\/b>in your terminal<\/p>\n now embedder component<\/p>\n We use OllamaDocumentEmbedder <\/b>component for embedding documents, but if you want to embed the text string then you have to use OllamaTextEmbedder.<\/b><\/p>\n Like our previous toy RAG example, we will start by initiating the Pipeline class.<\/p>\n Now we will add the components to our pipeline one by one<\/p>\n Adding components to the pipeline does not care about order so, you can add components in any order. but connecting is what matters.<\/p>\n Here, order matters, because how you connect the component tells the pipeline how the data will flow through the pipeline. It is like, It doesn\u2019t matter in which order or from where you buy your plumbing items but how to put them together will decide whether you get your water or not.<\/p>\n The converter converts the PDFs and sends them to clean for cleaning. Then the cleaner sends the cleaned documents to the splitter for chunking. Those chunks will then pass to the embedded for vectorization, and the last embedded will hand over these embeddings to the writer for storage.<\/p>\n Understand! Ok, let me give you a visual graph of the indexing so you can inspect the data flow.<\/p>\n Yeah, you can create a nice mermaid graph from the haystack pipeline easily.<\/p>\n Graph of Indexing Pipeline<\/strong><\/p>\n I assume now you have fully grasped the idea behind the Haystack Pipeline. Give a thank to you Plumber.<\/p>\n Now, we need to create a router to route the data through a different path. In this case, we\u2019ll use a conditional router which will do our routing job on certain conditions. The conditional router will evaluate conditions based on component output. It will direct data flow through different pipeline branches which enables dynamic decision-making. It will also have robust fallback strategies.<\/span><\/p>\n When the system gets no_answer replies from the embedding store context, then it will go to the web search tools for collecting relevant data from the internet.<\/p>\n For web search, we will use Duckduckgo API or Tavily, here I have used Duckduckgo.<\/p>\n Ok, most of the heavy lifting has been done. Now, time for prompt engineering<\/p>\n We will use the Haystack PromptBuilder component for building prompts from the template<\/p>\n First, we will create a prompt for qa<\/p>\n It will take the context from the document and try to answer the question. But if it does not find relevant context in the documents it will reply no_answer.<\/p>\n Now, in the second prompt after getting no_answer from the LLM, the system will use the web search tools for gathering context from the internet.<\/p>\n Duckduckgo prompt template<\/strong><\/p>\n It will facilitate the system to go to the web search and try to answer the query.<\/p>\n Creating prompt using PromptBuilder from Haystack<\/strong><\/p>\n We will use Haystack prompt joiner to join to branches of the prompt together.<\/p>\n The query pipeline will be embedding the query gathering contextual resources from the embeddings and answering our query using LLM or Web Search tool.<\/p>\n It is similar to the indexing pipeline.<\/p>\n Adding components to the query pipeline<\/p>\n Here, for LLM generation we use the OllamaGenerator component for generating answers using Llama3.2:3b or 1b or whatever LLM you like with tools calling.<\/p>\n Connecting all the components together for query flow and answer generation<\/p>\n In summary of the above connection:<\/p>\n Why not see for yourself?<\/p>\n Query Graph<\/b><\/p>\n I know it is a huge graph but it will show you exactly what is going on under the belly of the beast.<\/p>\n Now it is time to enjoy the fruit of our hard work.<\/p>\n Create a function for easy querying.<\/p>\n It is an easy simple function for answer generation.<\/p>\n Now run your main script for indexing the NCERT physics book<\/p>\n It is a one-time job, after indexing you must comment on this line otherwise it will start re-indexing the books.<\/p>\n and the bottom of the file we write our driver code for the query<\/p>\n MCQ on resistivity from the book\u2019s knowledge<\/p>\n Another question that is not in the book<\/p>\n Output<\/strong><\/p>\n Let\u2019s try another question.<\/p>\n So, it\u2019s working! We can use more data, books, or PDFs for embedding which will generate more contextual-aware answers. Also, LLMs such as GPT-4o, Anthropic\u2019s Claude, or other cloud LLMs will do the job even better.<\/p>\n Our agentic RAG system demonstrates the flexibility and robustness of the Haystack framework with its power of combining components and pipelines. This RAG can be made production-ready by deploying to the web service platform and also using better paid LLM such as OpenAI, and nthropic. You can build a UI using Streamlit or React-based web SPA for a better user experience.<\/p>\n You can find all the code used in the article, here.<\/a><\/p>\n The media shown in this article is not owned by Analytics Vidhya and is used at the Author\u2019s discretion<\/strong>.<\/p>\nLearning Objectives<\/h4>\n
\n
What is an Agentic LLM?<\/h2>\n
RAG vs Agentic RAG<\/h3>\n
![\"How](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/RAG.webp\")
\n
Understanding Haystack Framework Components<\/h2>\n
What Can You Build Using Haystack?<\/h3>\n
\n
Haystack Building Blocks<\/h2>\n
Components<\/h3>\n
$ pip install haystack-ai ollama-haystack\n\n# On you system download Ollama and install LLM\n\nollama pull llama3.2:3b\n\nollama pull nomic-embed-text\n\n\n# And then start ollama server\nollama serve<\/code><\/pre>\nfrom haystack import Document, Pipeline\nfrom haystack.components.builders.prompt_builder import PromptBuilder\nfrom haystack.components.retrievers.in_memory import InMemoryBM25Retriever\nfrom haystack.document_stores.in_memory import InMemoryDocumentStore\nfrom haystack_integrations.components.generators.ollama import OllamaGenerator<\/code><\/pre>\ndocument_store = InMemoryDocumentStore()\ndocuments = [\n Document(\n content=\"Naruto Uzumaki is a ninja from the Hidden Leaf Village and aspires to become Hokage.\"\n ),\n Document(\n content=\"Luffy is the captain of the Straw Hat Pirates and dreams of finding the One Piece.\"\n ),\n Document(\n content=\"Goku, a Saiyan warrior, has defended Earth from numerous powerful enemies like Frieza and Cell.\"\n ),\n Document(\n content=\"Light Yagami finds a mysterious Death Note, which allows him to eliminate people by writing their names.\"\n ),\n Document(\n content=\"Levi Ackerman is humanity\u2019s strongest soldier, fighting against the Titans to protect mankind.\"\n ),\n]<\/code><\/pre>\nPipeline<\/h3>\n
pipe = Pipeline()\n\npipe.add_component(\"retriever\", InMemoryBM25Retriever(document_store=document_store))\npipe.add_component(\"prompt_builder\", PromptBuilder(template=template))\npipe.add_component(\n \"llm\", OllamaGenerator(model=\"llama3.2:1b\", url=\"http:\/\/localhost:11434\")\n)\npipe.connect(\"retriever\", \"prompt_builder.documents\")\npipe.connect(\"prompt_builder\", \"llm\")<\/code><\/pre>\nimage_param = {\n \"format\": \"img\",\n \"type\": \"png\",\n \"theme\": \"forest\",\n \"bgColor\": \"f2f3f4\",\n}\npipe.show(params=image_param)<\/code><\/pre>\n\n
Nodes<\/h3>\n
pipe.add_component(\"retriever\", InMemoryBM25Retriever(document_store=document_store))\npipe.add_component(\"prompt_builder\", PromptBuilder(template=template))\npipe.add_component(\n \"llm\", OllamaGenerator(model=\"llama3.2:1b\", url=\"http:\/\/localhost:11434\")\n)\n<\/code><\/pre>\nConnection Graph<\/h3>\n
image_param = {\n \"format\": \"img\",\n \"type\": \"png\",\n \"theme\": \"forest\",\n \"bgColor\": \"f2f3f4\",\n}\npipe.show(params=image_param)<\/code><\/pre>\n![\"Building](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/image_Yq64Z4B.webp\")
<\/figure>\n\n
template = \"\"\"\nGiven only the following information, answer the question.\nIgnore your own knowledge.\n\nContext:\n{% for document in documents %}\n {{ document.content }}\n{% endfor %}\n\nQuestion: {{ query }}?\n\"\"\"<\/code><\/pre>\nquery = \"How Goku eliminate people?\"\nresponse = pipe.run({\"prompt_builder\": {\"query\": query}, \"retriever\": {\"query\": query}})\nprint(response[\"llm\"][\"replies\"])<\/code><\/pre>\n![\"RAG](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/goku_res.webp\")
<\/figure>\nQuestion-Answer RAG Project for Higher Secondary Physics<\/h2>\n
For this, I will use:<\/p>\n\n
Setting Up the Developer Environment<\/h3>\n
$conda create --name agenticlm python=3.12\n\n$conda activate agenticlm<\/code><\/pre>\nInstall Necessary Package<\/h4>\n
$pip install haystack-ai ollama-haystack pypdf\n\n$pip install chroma-haystack duckduckgo-api-haystack<\/code><\/pre>\n$md qagent # create dir\n\n$cd qagent # change to dir\n\n$ code . # open folder in vscode<\/code><\/pre>\nImporting Necessary Libraries<\/h4>\n
\n
# System packages\nimport os\nfrom pathlib import Path<\/code><\/pre>\n# Core haystack components\nfrom haystack import Pipeline\nfrom haystack.components.writers import DocumentWriter\nfrom haystack.components.joiners import BranchJoiner\nfrom haystack.document_stores.types import DuplicatePolicy\nfrom haystack.components.converters import PyPDFToDocument\nfrom haystack.components.routers import ConditionalRouter\nfrom haystack.components.builders.prompt_builder import PromptBuilder\nfrom haystack.components.preprocessors import DocumentCleaner, DocumentSplitter<\/code><\/pre>\n# ChromaDB integration\nfrom haystack_integrations.document_stores.chroma import ChromaDocumentStore\nfrom haystack_integrations.components.retrievers.chroma import (\n ChromaEmbeddingRetriever,\n)<\/code><\/pre>\n# Ollama integration\nfrom haystack_integrations.components.embedders.ollama.document_embedder import (\n OllamaDocumentEmbedder,\n)\nfrom haystack_integrations.components.embedders.ollama.text_embedder import (\n OllamaTextEmbedder,\n)\nfrom haystack_integrations.components.generators.ollama import OllamaGenerator<\/code><\/pre>\n# Duckduckgo search integration\nfrom duckduckgo_api_haystack import DuckduckgoApiWebSearch<\/code><\/pre>\nCreating a Document Store<\/h4>\n
# Chroma DB integration component for document(embedding) store\n\ndocument_store = ChromaDocumentStore(persist_path=\"qagent\/embeddings\")<\/code><\/pre>\nHERE = Path(__file__).resolve().parent\nfile_path = [HERE \/ \"data\" \/ Path(name) for name in os.listdir(\"QApipeline\/data\")]<\/code><\/pre>\nDocument Preprocessing Components<\/h3>\n
cleaner = DocumentCleaner()<\/code><\/pre>\nsplitter = DocumentSplitter()<\/code><\/pre>\nfile_converter = PyPDFToDocument()<\/code><\/pre>\nwriter = DocumentWriter(document_store=document_store, policy=DuplicatePolicy.OVERWRITE)<\/code><\/pre>\n$ ollama pull nomic-embed-text\n\n$ ollama pull llama3.2:3b<\/code><\/pre>\nembedder = OllamaDocumentEmbedder(\n model=\"nomic-embed-text\", url=\"http:\/\/localhost:11434\"\n)<\/code><\/pre>\nCreating Indexing Pipeline<\/h3>\n
indexing_pipeline = Pipeline()<\/code><\/pre>\nindexing_pipeline.add_component(\"embedder\", embedder)\nindexing_pipeline.add_component(\"converter\", file_converter)\nindexing_pipeline.add_component(\"cleaner\", cleaner)\nindexing_pipeline.add_component(\"splitter\", splitter)\nindexing_pipeline.add_component(\"writer\", writer)<\/code><\/pre>\nConnecting Components to the Pipeline Graph<\/h4>\n
indexing_pipeline.connect(\"converter\", \"cleaner\")\nindexing_pipeline.connect(\"cleaner\", \"splitter\")\nindexing_pipeline.connect(\"splitter\", \"embedder\")\nindexing_pipeline.connect(\"embedder\", \"writer\")<\/code><\/pre>\nDraw Indexing Pipeline<\/h4>\n
image_param = {\n \"format\": \"img\",\n \"type\": \"png\",\n \"theme\": \"forest\",\n \"bgColor\": \"f2f3f4\",\n}\n\nindexing_pipeline.draw(\"indexing_pipeline.png\", params=image_param) # type: ignore<\/code><\/pre>\n![\"Building](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/indexing_pipeline.webp\")
<\/figure>\nImplement a Router<\/h3>\n
# Conditions for routing\nroutes = [\n {\n \"condition\": \"{{'no_answer' in replies[0]}}\",\n \"output\": \"{{query}}\",\n \"output_name\": \"go_to_websearch\",\n \"output_type\": str,\n },\n {\n \"condition\": \"{{'no_answer' not in replies[0]}}\",\n \"output\": \"{{replies[0]}}\",\n \"output_name\": \"answer\",\n \"output_type\": str,\n },\n]\n\n\n# router component\n\nrouter = ConditionalRouter(routes=routes)<\/code><\/pre>\nwebsearch = DuckduckgoApiWebSearch(top_k=5)<\/code><\/pre>\nCreate Prompt Templates<\/h3>\n
template_qa = \"\"\"\nGiven ONLY the following information, answer the question.\nIf the answer is not contained within the documents reply with \"no_answer.\nIf the answer is contained within the documents, start the answer with \"FROM THE KNOWLEDGE BASE: \".\n\nContext:\n{% for document in documents %}\n {{ document.content }}\n{% endfor %}\n\nQuestion: {{ query }}?\n\n\"\"\"<\/code><\/pre>\ntemplate_websearch = \"\"\"\nAnswer the following query given the documents retrieved from the web.\nStart the answer with \"FROM THE WEB: \".\n\nDocuments:\n{% for document in documents %}\n {{ document.content }}\n{% endfor %}\n\nQuery: {{query}}\n\n\"\"\"<\/code><\/pre>\nprompt_qa = PromptBuilder(template=template_qa)\n\nprompt_builder_websearch = PromptBuilder(template=template_websearch)<\/code><\/pre>\nprompt_joiner = BranchJoiner(str)<\/code><\/pre>\nImplement Query Pipeline<\/h3>\n
Initiating Pipeline<\/h4>\n
query_pipeline = Pipeline()<\/code><\/pre>\nquery_pipeline.add_component(\"text_embedder\", OllamaTextEmbedder())\nquery_pipeline.add_component(\n \"retriever\", ChromaEmbeddingRetriever(document_store=document_store)\n)\nquery_pipeline.add_component(\"prompt_builder\", prompt_qa)\nquery_pipeline.add_component(\"prompt_joiner\", prompt_joiner)\nquery_pipeline.add_component(\n \"llm\",\n OllamaGenerator(model=\"llama3.2:3b\", timeout=500, url=\"http:\/\/localhost:11434\"),\n)\nquery_pipeline.add_component(\"router\", router)\nquery_pipeline.add_component(\"websearch\", websearch)\nquery_pipeline.add_component(\"prompt_builder_websearch\", prompt_builder_websearch)<\/code><\/pre>\nquery_pipeline.connect(\"text_embedder.embedding\", \"retriever.query_embedding\")\nquery_pipeline.connect(\"retriever\", \"prompt_builder.documents\")\nquery_pipeline.connect(\"prompt_builder\", \"prompt_joiner\")\nquery_pipeline.connect(\"prompt_joiner\", \"llm\")\nquery_pipeline.connect(\"llm.replies\", \"router.replies\")\nquery_pipeline.connect(\"router.go_to_websearch\", \"websearch.query\")\nquery_pipeline.connect(\"router.go_to_websearch\", \"prompt_builder_websearch.query\")\nquery_pipeline.connect(\"websearch.documents\", \"prompt_builder_websearch.documents\")\nquery_pipeline.connect(\"prompt_builder_websearch\", \"prompt_joiner\")<\/code><\/pre>\n\n
Draw Query Pipeline Graph<\/h3>\n
query_pipeline.draw(\"agentic_qa_pipeline.png\", params=image_param) # type: ignore<\/code><\/pre>\n![\"Building](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/agentic_qa_pipeline.webp\")
<\/figure>\ndef get_answer(query: str):\n response = query_pipeline.run(\n {\n \"text_embedder\": {\"text\": query},\n \"prompt_builder\": {\"query\": query},\n \"router\": {\"query\": query},\n }\n )\n return response[\"router\"][\"answer\"]<\/code><\/pre>\nindexing_pipeline.run({\"converter\": {\"sources\": file_path}})<\/code><\/pre>\nif __name__ == \"__main__\":\n query = \"Give me 5 MCQ on resistivity?\"\n print(get_answer(query))<\/code><\/pre>\n![\"RAG](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/image_8zfl6k1.webp\")
<\/figure>\nif __name__ == \"__main__\":\n query = \"What is Photosynthesis?\"\n print(get_answer(query))<\/code><\/pre>\n![\"Output](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/image_6tu9tAI.webp\")
<\/figure>\nif __name__ == \"__main__\":\n query = (\n \"Tell me what is DRIFT OF ELECTRONS AND THE ORIGIN OF RESISTIVITY from the book\"\n )\n print(get_answer(query))<\/code><\/pre>\n![\"Building](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/image_a0bJG0P.webp\")
<\/figure>\nConclusion<\/h2>\n
Key Takeaways<\/h4>\n
\n
Frequently Asked Question<\/h2>\n
![\"Avijit](\"https:\/\/av-eks-lekhak.s3.amazonaws.com\/media\/lekhak-profile-images\/converted_image_9z6Gys1.webp\")
<\/p>\n