In this tutorial, we explore how to set up and execute a sophisticated retrieval-augmented generation (RAG) pipeline in Google Colab. We leverage multiple state-of-the-art tools and libraries-including Gemma 3 for language and vision tasks, Docling for document conversion, LangChain for chain-of-thought orchestration, and Milvus as our vector database build a multimodal system that understands and processes text, tables, and images. Let\u2019s dive into each component and see how they work together.<\/p>\n
What is Multimodal RAG?<\/h2>\n
Multimodal RAG (Retrieval-Augmented Generation) extends traditional text-based RAG systems by integrating multiple data modalities, in this case, text, tables, and images. This means that the pipeline not only processes and retrieves text but also leverages vision models to understand and describe image content, making the solution more comprehensive. This multimodal approach is particularly beneficial for documents like annual reports that often contain visual elements, such as charts and diagrams.<\/p>\n
Proposed Architecture of Multimodal RAG with Gemma 3<\/h2>\n\n![\"architecture\"](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/03\/image-122.webp\")
<\/a>Source: Author<\/figcaption><\/figure>\n<\/div>\nThe aim of this project is to build a robust multimodal RAG<\/a> pipeline that can ingest documents (like PDFs), process text and images, store document embeddings in a vector database, and answer queries by retrieving relevant information. This setup is particularly useful for applications such as analyzing annual reports, extracting financial statements, or summarizing technical papers. By integrating various libraries and tools, we combine the power of language models with document conversion and vector search to create a comprehensive end-to-end solution.<\/p>\nThe pipeline uses several key libraries and tools:<\/p>\n
\n- Colab-Xterm Extension<\/strong>: This adds terminal support in Colab, allowing us to run shell commands and manage the environment efficiently.<\/li>\n
- Ollama Models<\/strong>: Provides pre-trained models such as Gemma3, which are used for both language and vision tasks.<\/li>\n
- Transformers<\/strong>: From Hugging Face, for model loading and tokenization.<\/li>\n
- LangChain<\/strong>: Orchestrates the chain of processing steps, from prompt creation to document retrieval and generation.<\/li>\n
- Docling<\/strong>: Converts PDF documents into structured formats, enabling extraction of text, tables, and images.<\/li>\n
- Milvus<\/strong>: A vector database that stores document embeddings and supports efficient similarity search.<\/li>\n
- Hugging Face CLI<\/strong>: Used for logging into Hugging Face to access certain models.<\/li>\n
- Additional Utilities<\/strong>: Such as Pillow for image processing and IPython for display functionalities.<\/li>\n<\/ul>\n
Also read: A Comprehensive Guide to Building Multimodal RAG Systems<\/a><\/p>\nBuilding Multimodal Rag with Gemma 3<\/h2>\n
We are building multimodal rag: This approach improves contextual understanding, accuracy, and relevance, especially in fields like healthcare, research, and media analysis. By leveraging cross-modal embeddings, hybrid retrieval strategies, and vision-language models, multimodal RAG systems can provide richer and more insightful responses. The key challenge lies in efficiently integrating and retrieving multimodal data while maintaining coherence and scalability. As AI progresses, developing optimized architectures and retrieval strategies will be crucial for unlocking the full potential of multimodal intelligence. <\/p>\n
Terminal Setup with Colab-Xterm<\/h3>\n
First, we install the colab-xterm extension to bring a terminal environment directly into Colab. This allows us to run system commands, install packages, and manage our session more flexibly.<\/p>\n
!pip install colab-xterm\u00a0 # Install colab-xterm\n%load_ext colabxterm \u00a0 \u00a0 # Load the xterm extension\n%xterm\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 # Launch an xterm terminal session in Colab<\/code><\/pre>\n![\"xtream\"](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/03\/image-123.webp\")
<\/figure>\nThis terminal support is especially useful for installing additional dependencies or managing background processes.<\/p>\n
Installing and Managing Ollama Models<\/h3>\n
We pull specific Ollama<\/a> models into our environment using simple shell commands. For example:<\/p>\n!ollama pull gemma3:4b\n!ollama pull llama3.2\n!ollama list<\/code><\/pre>\nThese commands ensure that we have the necessary language and vision models available, such as the powerful Gemma 3 model<\/a>, which is central to our multimodal processing.<\/p>\nInstalling Essential Python Packages<\/h3>\n
The next step involves installing a host of packages required for our pipeline. This includes libraries for deep learning, text processing, and document handling:<\/p>\n
! pip install transformers pillow langchain_community langchain_huggingface langchain_milvus docling langchain_ollama<\/code><\/pre>\nBy installing these packages, we prepare the environment for everything from document conversion to retrieval-augmented generation.<\/p>\n
Logging and Hugging Face Authentication<\/h3>\n
Setting up logging is crucial for monitoring pipeline operations:<\/p>\n
import logging\nlogging.basicConfig(level=logging.INFO)<\/code><\/pre>\nWe also log in to Hugging Face using their CLI to access certain pre-trained models:<\/p>\n
!huggingface-cli login<\/code><\/pre>\nThis authentication step is necessary for fetching model artifacts and ensuring smooth integration with Hugging Face\u2019s ecosystem.<\/p>\n
Configuring Vision and Language Models (Gemma 3)<\/h3>\n
The pipeline leverages the Gemma 3 model for both vision and language tasks. For the language side, we set up the model and tokenizer:<\/p>\n
This dual setup enables the system to generate textual descriptions from images, making the pipeline truly multimodal.<\/p>\n
Document Conversion with Docling<\/h3>\n1<\/strong>. Converting PDFs to Structured Documents<\/h4>\nWe employ Docling\u2019s DocumentConverter to convert PDFs into structured documents. The conversion process involves extracting text, tables, and images from the source PDFs:<\/p>\n
from docling.document_converter import DocumentConverter, PdfFormatOption\nfrom docling.datamodel.base_models import InputFormat\nfrom docling.datamodel.pipeline_options import PdfPipelineOptions\npdf_pipeline_options = PdfPipelineOptions(\n\u00a0\u00a0\u00a0\u00a0do_ocr=False,\n\u00a0\u00a0\u00a0\u00a0generate_picture_images=True,\n)\nformat_options = { InputFormat.PDF: PdfFormatOption(pipeline_options=pdf_pipeline_options) }\nconverter = DocumentConverter(format_options=format_options)\n# Define the sources (URLs) of the documents to be converted.\n# \"https:\/\/arxiv.org\/pdf\/1706.03762\"\nsources = [\n\u00a0\"https:\/\/www.pwc.com\/jm\/en\/research-publications\/pdf\/basic-understanding-of-a-companys-financials.pdf\"\n]\n# Convert the PDF documents from the sources into an internal document format.\nconversions = { source: converter.convert(source=source).document for source in sources }<\/code><\/pre>\nInput File<\/h4>\n
We\u2019ll be using PwC\u2019s publicly available financial statements. I\u2019ve included the PDF link, and you\u2019re welcome to add your own source links as well!<\/p>\n
2. Extracting and Chunking Content<\/h4>\n
After conversion, we chunk the document into manageable pieces, separating text from tables and images. This segmentation allows each component to be processed independently:<\/p>\n
from docling_core.transforms.chunker.hybrid_chunker import HybridChunker\nfrom langchain_core.documents import Document\n# Process text chunks (excluding pure table segments)\ntexts: list[Document] = []\nfor source, docling_document in conversions.items():\n\u00a0\u00a0\u00a0\u00a0for chunk in HybridChunker(tokenizer=embeddings_tokenizer).chunk(docling_document):\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0# Skip table-only chunks; process tables separately\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0if len(chunk.meta.doc_items) == 1:\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0continue\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0document = Document(\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0page_content=chunk.text,\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0metadata={\"source\": source, \"ref\": \"reference details\"}\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0)\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0texts.append(document)<\/code><\/pre>\nThis approach not only improves processing efficiency but also facilitates more precise vector storage and retrieval later.<\/p>\n
Image Processing and Encoding<\/h3>\n
Images from the documents are processed using Pillow. We convert images into base64-encoded strings that can be embedded directly into prompts:<\/p>\n
import base64, io, PIL.Image, PIL.ImageOps\ndef encode_image(image: PIL.Image.Image, format: str = \"png\") -> str:\n\u00a0\u00a0\u00a0\u00a0image = PIL.ImageOps.exif_transpose(image) or image\n\u00a0\u00a0\u00a0\u00a0image = image.convert(\"RGB\")\n\u00a0\u00a0\u00a0\u00a0buffer = io.BytesIO()\n\u00a0\u00a0\u00a0\u00a0image.save(buffer, format)\n\u00a0\u00a0\u00a0\u00a0encoding = base64.b64encode(buffer.getvalue()).decode(\"utf-8\")\n\u00a0\u00a0\u00a0\u00a0return f\"data:image\/{format};base64,{encoding}\"<\/code><\/pre>\nSubsequently, these images are fed into our vision model to generate descriptive text, enhancing the multimodal capabilities of our pipeline.<\/p>\n
Creating a Vector Database with Milvus<\/h3>\n
To enable fast and accurate retrieval of document embeddings, we set up Milvus as our vector store:<\/p>\n
import tempfile\nfrom langchain_core.vectorstores import VectorStore\nfrom langchain_milvus import Milvus\ndb_file = tempfile.NamedTemporaryFile(prefix=\"vectorstore_\", suffix=\".db\", delete=False).name\nvector_db: VectorStore = Milvus(\n\u00a0\u00a0\u00a0\u00a0embedding_function=embeddings_model,\n\u00a0\u00a0\u00a0\u00a0connection_args={\"uri\": db_file},\n\u00a0\u00a0\u00a0\u00a0auto_id=True,\n\u00a0\u00a0\u00a0\u00a0enable_dynamic_field=True,\n\u00a0\u00a0\u00a0\u00a0index_params={\"index_type\": \"AUTOINDEX\"},\n)<\/code><\/pre>\nDocuments\u2014whether text, tables, or image descriptions\u2014are then added to the vector database, enabling fast and accurate similarity searches during query execution.<\/p>\n
Building the Retrieval-Augmented Generation (RAG) Chain<\/h3>\n1. Prompt Creation and Document Wrapping<\/h4>\n
Using LangChain\u2019s prompt templates, we create custom prompts to feed context and queries into our language model:<\/p>\n
from langchain.prompts import PromptTemplate\nprompt = \"{input} Given the context: {context}\"\nprompt_template = PromptTemplate.from_template(template=prompt)<\/code><\/pre>\nEach retrieved document is wrapped using a document prompt template, ensuring that the model understands the structure of the input context.<\/p>\n
2. Assembling the RAG Pipeline<\/h4>\n
We combine the prompt with the vector store to create a retrieval chain that first fetches relevant documents and then uses them to generate a coherent answer:<\/p>\n
from langchain.chains.retrieval import create_retrieval_chain\nfrom langchain.chains.combine_documents import create_stuff_documents_chain\ncombine_docs_chain = create_stuff_documents_chain(\n\u00a0\u00a0\u00a0\u00a0llm=model,\n\u00a0\u00a0\u00a0\u00a0prompt=prompt_template,\n\u00a0\u00a0\u00a0\u00a0document_prompt=PromptTemplate.from_template(template=\"\"\"\\\nDocument {doc_id}\n{page_content}\"\"\"),\n\u00a0\u00a0\u00a0\u00a0document_separator=\"\\n\\n\",\n)\nrag_chain = create_retrieval_chain(\n\u00a0\u00a0\u00a0\u00a0retriever=vector_db.as_retriever(),\n\u00a0\u00a0\u00a0\u00a0combine_docs_chain=combine_docs_chain,\n)<\/code><\/pre>\nQueries are then executed against this chain, retrieving context and generating responses based on both the query and the stored document embeddings.<\/p>\n
Executing Queries and Retrieving Information<\/h3>\n
Once the RAG chain is established, you can run queries to retrieve relevant information from your document database. For example:<\/p>\n
query = \"Explain Three Key Financial Statements Notes\"\noutputs = rag_chain.invoke({\"input\": query})\nMarkdown(outputs['answer'])<\/code><\/pre>\n![\"Output\"](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/03\/unnamed-2025-03-28T155139.917.webp\")
<\/figure>\nquery = \"tell me the Contents of an annual report\"\noutputs = rag_chain.invoke({\"input\": query})\nMarkdown(outputs['answer'])<\/code><\/pre>\n![\"Output\"](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/03\/unnamed-2025-03-28T155212.803.webp\")
<\/figure>\nquery = \"what are the benefits of an annual report?\"\noutputs = rag_chain.invoke({\"input\": query})\nMarkdown(outputs['answer'])<\/code><\/pre>\n![\"output\"](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/03\/unnamed-2025-03-28T155257.690.webp\")
<\/figure>\nThe same process can be applied for various queries, such as explaining financial statement notes or summarizing an annual report, thereby demonstrating the versatility of the pipeline.<\/p>\n
Here\u2019s the full code: AV-multimodal-gemma3-rag<\/a><\/p>\nUse Cases<\/h2>\n
This pipeline has numerous applications:<\/p>\n
\n- Financial Reporting<\/strong>: Automatically extract and summarize key financial statements, cash flow elements, and annual report details.<\/li>\n
- Document Analysis<\/strong>: Convert PDFs into structured data for further analysis or machine learning tasks.<\/li>\n
- Multimodal Search<\/strong>: Enable search and retrieval from mixed media documents, combining textual and visual content.<\/li>\n
- Business Intelligence<\/strong>: Provide quick insights into complex documents by aggregating and synthesizing information across modalities.<\/li>\n<\/ul>\n
Conclusion<\/h2>\n
In this tutorial, we demonstrated how to build a multimodal RAG with Gemma 3 in Google Colab. By integrating tools like Colab-Xterm, Ollama models (Gemma 3), Docling<\/a>, LangChain<\/a>, and Milvus<\/a>, we created a system capable of processing text, tables, and images. This powerful setup not only enables effective document retrieval but also supports complex query answering and analysis in diverse applications. Whether you\u2019re dealing with financial reports, research papers, or business intelligence tasks, this pipeline offers a versatile and scalable solution.<\/p>\nHappy coding, and enjoy exploring the possibilities of multimodal retrieval-augmented generation!<\/p>\n
<\/a>The aim of this project is to build a robust multimodal RAG<\/a> pipeline that can ingest documents (like PDFs), process text and images, store document embeddings in a vector database, and answer queries by retrieving relevant information. This setup is particularly useful for applications such as analyzing annual reports, extracting financial statements, or summarizing technical papers. By integrating various libraries and tools, we combine the power of language models with document conversion and vector search to create a comprehensive end-to-end solution.<\/p>\n The pipeline uses several key libraries and tools:<\/p>\n Also read: A Comprehensive Guide to Building Multimodal RAG Systems<\/a><\/p>\n We are building multimodal rag: This approach improves contextual understanding, accuracy, and relevance, especially in fields like healthcare, research, and media analysis. By leveraging cross-modal embeddings, hybrid retrieval strategies, and vision-language models, multimodal RAG systems can provide richer and more insightful responses. The key challenge lies in efficiently integrating and retrieving multimodal data while maintaining coherence and scalability. As AI progresses, developing optimized architectures and retrieval strategies will be crucial for unlocking the full potential of multimodal intelligence. <\/p>\n First, we install the colab-xterm extension to bring a terminal environment directly into Colab. This allows us to run system commands, install packages, and manage our session more flexibly.<\/p>\n This terminal support is especially useful for installing additional dependencies or managing background processes.<\/p>\n We pull specific Ollama<\/a> models into our environment using simple shell commands. For example:<\/p>\n These commands ensure that we have the necessary language and vision models available, such as the powerful Gemma 3 model<\/a>, which is central to our multimodal processing.<\/p>\n The next step involves installing a host of packages required for our pipeline. This includes libraries for deep learning, text processing, and document handling:<\/p>\n By installing these packages, we prepare the environment for everything from document conversion to retrieval-augmented generation.<\/p>\n Setting up logging is crucial for monitoring pipeline operations:<\/p>\n We also log in to Hugging Face using their CLI to access certain pre-trained models:<\/p>\n This authentication step is necessary for fetching model artifacts and ensuring smooth integration with Hugging Face\u2019s ecosystem.<\/p>\n The pipeline leverages the Gemma 3 model for both vision and language tasks. For the language side, we set up the model and tokenizer:<\/p>\n This dual setup enables the system to generate textual descriptions from images, making the pipeline truly multimodal.<\/p>\n We employ Docling\u2019s DocumentConverter to convert PDFs into structured documents. The conversion process involves extracting text, tables, and images from the source PDFs:<\/p>\n We\u2019ll be using PwC\u2019s publicly available financial statements. I\u2019ve included the PDF link, and you\u2019re welcome to add your own source links as well!<\/p>\n After conversion, we chunk the document into manageable pieces, separating text from tables and images. This segmentation allows each component to be processed independently:<\/p>\n This approach not only improves processing efficiency but also facilitates more precise vector storage and retrieval later.<\/p>\n Images from the documents are processed using Pillow. We convert images into base64-encoded strings that can be embedded directly into prompts:<\/p>\n Subsequently, these images are fed into our vision model to generate descriptive text, enhancing the multimodal capabilities of our pipeline.<\/p>\n To enable fast and accurate retrieval of document embeddings, we set up Milvus as our vector store:<\/p>\n Documents\u2014whether text, tables, or image descriptions\u2014are then added to the vector database, enabling fast and accurate similarity searches during query execution.<\/p>\n Using LangChain\u2019s prompt templates, we create custom prompts to feed context and queries into our language model:<\/p>\n Each retrieved document is wrapped using a document prompt template, ensuring that the model understands the structure of the input context.<\/p>\n We combine the prompt with the vector store to create a retrieval chain that first fetches relevant documents and then uses them to generate a coherent answer:<\/p>\n Queries are then executed against this chain, retrieving context and generating responses based on both the query and the stored document embeddings.<\/p>\n Once the RAG chain is established, you can run queries to retrieve relevant information from your document database. For example:<\/p>\n The same process can be applied for various queries, such as explaining financial statement notes or summarizing an annual report, thereby demonstrating the versatility of the pipeline.<\/p>\n Here\u2019s the full code: AV-multimodal-gemma3-rag<\/a><\/p>\n This pipeline has numerous applications:<\/p>\n In this tutorial, we demonstrated how to build a multimodal RAG with Gemma 3 in Google Colab. By integrating tools like Colab-Xterm, Ollama models (Gemma 3), Docling<\/a>, LangChain<\/a>, and Milvus<\/a>, we created a system capable of processing text, tables, and images. This powerful setup not only enables effective document retrieval but also supports complex query answering and analysis in diverse applications. Whether you\u2019re dealing with financial reports, research papers, or business intelligence tasks, this pipeline offers a versatile and scalable solution.<\/p>\n Happy coding, and enjoy exploring the possibilities of multimodal retrieval-augmented generation!<\/p>\n\n
Building Multimodal Rag with Gemma 3<\/h2>\n
Terminal Setup with Colab-Xterm<\/h3>\n
!pip install colab-xterm\u00a0 # Install colab-xterm\n%load_ext colabxterm \u00a0 \u00a0 # Load the xterm extension\n%xterm\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 # Launch an xterm terminal session in Colab<\/code><\/pre>\n<\/figure>\nInstalling and Managing Ollama Models<\/h3>\n
!ollama pull gemma3:4b\n!ollama pull llama3.2\n!ollama list<\/code><\/pre>\nInstalling Essential Python Packages<\/h3>\n
! pip install transformers pillow langchain_community langchain_huggingface langchain_milvus docling langchain_ollama<\/code><\/pre>\nLogging and Hugging Face Authentication<\/h3>\n
import logging\nlogging.basicConfig(level=logging.INFO)<\/code><\/pre>\n!huggingface-cli login<\/code><\/pre>\nConfiguring Vision and Language Models (Gemma 3)<\/h3>\n
Document Conversion with Docling<\/h3>\n
1<\/strong>. Converting PDFs to Structured Documents<\/h4>\n
from docling.document_converter import DocumentConverter, PdfFormatOption\nfrom docling.datamodel.base_models import InputFormat\nfrom docling.datamodel.pipeline_options import PdfPipelineOptions\npdf_pipeline_options = PdfPipelineOptions(\n\u00a0\u00a0\u00a0\u00a0do_ocr=False,\n\u00a0\u00a0\u00a0\u00a0generate_picture_images=True,\n)\nformat_options = { InputFormat.PDF: PdfFormatOption(pipeline_options=pdf_pipeline_options) }\nconverter = DocumentConverter(format_options=format_options)\n# Define the sources (URLs) of the documents to be converted.\n# \"https:\/\/arxiv.org\/pdf\/1706.03762\"\nsources = [\n\u00a0\"https:\/\/www.pwc.com\/jm\/en\/research-publications\/pdf\/basic-understanding-of-a-companys-financials.pdf\"\n]\n# Convert the PDF documents from the sources into an internal document format.\nconversions = { source: converter.convert(source=source).document for source in sources }<\/code><\/pre>\nInput File<\/h4>\n
2. Extracting and Chunking Content<\/h4>\n
from docling_core.transforms.chunker.hybrid_chunker import HybridChunker\nfrom langchain_core.documents import Document\n# Process text chunks (excluding pure table segments)\ntexts: list[Document] = []\nfor source, docling_document in conversions.items():\n\u00a0\u00a0\u00a0\u00a0for chunk in HybridChunker(tokenizer=embeddings_tokenizer).chunk(docling_document):\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0# Skip table-only chunks; process tables separately\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0if len(chunk.meta.doc_items) == 1:\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0continue\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0document = Document(\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0page_content=chunk.text,\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0metadata={\"source\": source, \"ref\": \"reference details\"}\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0)\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0texts.append(document)<\/code><\/pre>\nImage Processing and Encoding<\/h3>\n
import base64, io, PIL.Image, PIL.ImageOps\ndef encode_image(image: PIL.Image.Image, format: str = \"png\") -> str:\n\u00a0\u00a0\u00a0\u00a0image = PIL.ImageOps.exif_transpose(image) or image\n\u00a0\u00a0\u00a0\u00a0image = image.convert(\"RGB\")\n\u00a0\u00a0\u00a0\u00a0buffer = io.BytesIO()\n\u00a0\u00a0\u00a0\u00a0image.save(buffer, format)\n\u00a0\u00a0\u00a0\u00a0encoding = base64.b64encode(buffer.getvalue()).decode(\"utf-8\")\n\u00a0\u00a0\u00a0\u00a0return f\"data:image\/{format};base64,{encoding}\"<\/code><\/pre>\nCreating a Vector Database with Milvus<\/h3>\n
import tempfile\nfrom langchain_core.vectorstores import VectorStore\nfrom langchain_milvus import Milvus\ndb_file = tempfile.NamedTemporaryFile(prefix=\"vectorstore_\", suffix=\".db\", delete=False).name\nvector_db: VectorStore = Milvus(\n\u00a0\u00a0\u00a0\u00a0embedding_function=embeddings_model,\n\u00a0\u00a0\u00a0\u00a0connection_args={\"uri\": db_file},\n\u00a0\u00a0\u00a0\u00a0auto_id=True,\n\u00a0\u00a0\u00a0\u00a0enable_dynamic_field=True,\n\u00a0\u00a0\u00a0\u00a0index_params={\"index_type\": \"AUTOINDEX\"},\n)<\/code><\/pre>\nBuilding the Retrieval-Augmented Generation (RAG) Chain<\/h3>\n
1. Prompt Creation and Document Wrapping<\/h4>\n
from langchain.prompts import PromptTemplate\nprompt = \"{input} Given the context: {context}\"\nprompt_template = PromptTemplate.from_template(template=prompt)<\/code><\/pre>\n2. Assembling the RAG Pipeline<\/h4>\n
from langchain.chains.retrieval import create_retrieval_chain\nfrom langchain.chains.combine_documents import create_stuff_documents_chain\ncombine_docs_chain = create_stuff_documents_chain(\n\u00a0\u00a0\u00a0\u00a0llm=model,\n\u00a0\u00a0\u00a0\u00a0prompt=prompt_template,\n\u00a0\u00a0\u00a0\u00a0document_prompt=PromptTemplate.from_template(template=\"\"\"\\\nDocument {doc_id}\n{page_content}\"\"\"),\n\u00a0\u00a0\u00a0\u00a0document_separator=\"\\n\\n\",\n)\nrag_chain = create_retrieval_chain(\n\u00a0\u00a0\u00a0\u00a0retriever=vector_db.as_retriever(),\n\u00a0\u00a0\u00a0\u00a0combine_docs_chain=combine_docs_chain,\n)<\/code><\/pre>\nExecuting Queries and Retrieving Information<\/h3>\n
query = \"Explain Three Key Financial Statements Notes\"\noutputs = rag_chain.invoke({\"input\": query})\nMarkdown(outputs['answer'])<\/code><\/pre>\n<\/figure>\nquery = \"tell me the Contents of an annual report\"\noutputs = rag_chain.invoke({\"input\": query})\nMarkdown(outputs['answer'])<\/code><\/pre>\n<\/figure>\nquery = \"what are the benefits of an annual report?\"\noutputs = rag_chain.invoke({\"input\": query})\nMarkdown(outputs['answer'])<\/code><\/pre>\n<\/figure>\nUse Cases<\/h2>\n
\n
Conclusion<\/h2>\n