Retrieval Augmented Generation (RAG<\/a>) has revolutionized how large language models access external data, but traditional approaches are limited to text. With the rise of multimodal data, integrating text and visual information is crucial for comprehensive analysis, especially in complex fields like finance and research. Multimodal RAG addresses this by enabling models to process both text and images for better knowledge retrieval and reasoning. This article explores building a multimodal RAG system using Google\u2019s Gemini<\/a> models, Vertex AI<\/a>, and LangChain<\/a>, guiding you through environment setup, data processing, embedding generation, and constructing a robust document search engine.<\/p>\n Multimodal RAG models combine visual and printed information to supply more strong and context-aware yields. Not at all like conventional Cloth models, which exclusively depend on content, multimodal Clothes are outlined to get and consolidate visual substance such as graphs, charts, and pictures. This dual-processing capability is particularly valuable for analyzing complex records where visuals are as enlightening as content, such as money-related reports, logical papers, or client manuals.<\/p>\n By preparing content and pictures, the show offers a more profound understanding of the substance, driving to more precise and smart reactions. This integration relieves the chance of producing deceiving or relevantly erroneous data (commonly known as visualization in machine learning), coming about in more dependable yields for decision-making and investigation.<\/p>\n Here\u2019s a summary of each key technology:<\/p>\n The architecture of a multimodal RAG system involves:<\/p>\n Now let\u2019s get into the actual coding part. In this section, I will guide you through the steps of building a multimodal RAG system for content and images, using Google Gemini, Vertex AI, and LangChain.<\/p>\n \u00a0Let\u2019s begin by setting up the environment.<\/p>\n The %pip install command installs all the necessary Python libraries, including google-cloud-aiplatform, langchain, and various document-processing libraries like pypdf.<\/p>\n Add the code to authenticate and initialize the Vertex AI environment PROJECT_ID = \u201cYOUR_PROJECT_ID\u201d # @param {type:\u201dstring\u201d}<\/p>\n Add the code for constructing the document repository and integrating LangChain: \u2022 Load data into Document Store and Vector Store<\/p>\n Multimodal RAG (Retrieval-Augmented Generation) combines text and visual data to enhance information retrieval, enabling more contextually accurate and comprehensive AI responses. By leveraging tools like Gemini, Vertex AI, and LangChain, developers can build intelligent systems that efficiently process both textual and visual data.<\/p>\n Gemini enables understanding of diverse data types, while Vertex AI supports scalable model deployment for real-time applications. LangChain streamlines integration with external APIs and databases, allowing seamless interaction with multiple data sources. Together, these technologies provide powerful capabilities for creating context-aware, data-rich systems for use in areas like content generation, personalized recommendations, and interactive AI assistants.<\/p>\n A. Multimodal RAG (Retrieval Augmented Generation) combines text and visual data to improve the accuracy and context of information retrieval, allowing AI systems to provide more comprehensive and relevant responses.<\/p>\n<\/p><\/div>\nLearning Objectives<\/h4>\n
\n
Multimodal RAG Model: An Overview<\/h2>\n
![\"multimodal](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/1_eTqZBKyd1EMXUcDxfFNWNg.webp\")
Key Technologies Used<\/h2>\n
\n
Model Architecture Explained<\/h2>\n
\n
![\"building](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/1718492204884.webp\")
<\/figure>\nBuilding a Multimodal RAG System with Vertex AI, Gemini, and LangChain<\/h2>\n
Step 1: Setting Up Your Development Environment<\/h3>\n
1. Install necessary packages<\/h4>\n
%pip install -U -q google-cloud-aiplatform langchain-core langchain-google-vertexai langchain-text-splitters langchain-community \"unstructured[all-docs]\" pypdf pydantic lxml pillow matplotlib opencv-python tiktoken<\/code><\/pre>\n2. Restart the runtime to make sure new packages are accessible<\/h4>\n
import IPython\n\napp = IPython.Application.instance()\napp.kernel.do_shutdown(True)<\/code><\/pre>\n3. Authenticate the notebook environment (Google Colab only)<\/h4>\n
The auth.authenticate_user() function is used for authenticating your Google Cloud account in Google Colab.<\/p>\nimport sys\n\n# Additional authentication is required for Google Colab\nif \"google.colab\" in sys.modules:\n # Authenticate user to Google Cloud\n from google.colab import auth\n\n auth.authenticate_user()<\/code><\/pre>\nStep 2: Define Google Cloud Project Information<\/h3>\n
\n
PROJECT_ID = \"YOUR_PROJECT_ID\" # @param {type:\"string\"}\nLOCATION = \"us-central1\" # @param {type:\"string\"}\n\n# For Vector Search Staging\nGCS_BUCKET = \"YOUR_BUCKET_NAME\" # @param {type:\"string\"}\nGCS_BUCKET_URI = f\"gs:\/\/{GCS_BUCKET}\"<\/code><\/pre>\nStep 3: Initialize the Vertex AI SDK<\/h3>\n
from google.cloud import aiplatform\n\naiplatform.init(project=PROJECT_ID, location=LOCATION, staging_bucket=GCS_BUCKET_URI)<\/code><\/pre>\nStep 4: Import Necessary Libraries<\/h3>\n
Imports various libraries like langchain, IPython, pillow, and others needed for the retrieval and processing pipeline.<\/p>\nimport base64\nimport os\nimport re\nimport uuid\n\nfrom IPython.display import Image, Markdown, display\nfrom langchain.prompts import PromptTemplate\nfrom langchain.retrievers.multi_vector import MultiVectorRetriever\nfrom langchain.storage import InMemoryStore\nfrom langchain_core.documents import Document\nfrom langchain_core.messages import AIMessage, HumanMessage\nfrom langchain_core.output_parsers import StrOutputParser\nfrom langchain_core.runnables import RunnableLambda, RunnablePassthrough\nfrom langchain_google_vertexai import (\n ChatVertexAI,\n VectorSearchVectorStore,\n VertexAI,\n VertexAIEmbeddings,\n)\nfrom langchain_text_splitters import CharacterTextSplitter\nfrom unstructured.partition.pdf import partition_pdf\n\n# from langchain_community.vectorstores import Chroma # Optional<\/code><\/pre>\nStep 5: Define Model Information<\/h3>\n
MODEL_NAME = \"gemini-1.5-flash\"\nGEMINI_OUTPUT_TOKEN_LIMIT = 8192\n\nEMBEDDING_MODEL_NAME = \"text-embedding-004\"\nEMBEDDING_TOKEN_LIMIT = 2048\n\nTOKEN_LIMIT = min(GEMINI_OUTPUT_TOKEN_LIMIT, EMBEDDING_TOKEN_LIMIT)<\/code><\/pre>\nStep 6: Load the Data<\/h3>\n
1. Get documents and images from GCS<\/h4>\n
# Download documents and images used in this notebook\n!gsutil -m rsync -r gs:\/\/github-repo\/rag\/intro_multimodal_rag\/ .\nprint(\"Download completed\")<\/code><\/pre>\n2. Extract images, tables, and chunk text from a PDF file<\/h4>\n
\n
pdf_folder_path = \"\/content\/data\/\" if \"google.colab\" in sys.modules else \"data\/\"\npdf_file_name = \"google-10k-sample-14pages.pdf\"\n\n# Extract images, tables, and chunk text from a PDF file.\nraw_pdf_elements = partition_pdf(\n filename=pdf_file_name,\n extract_images_in_pdf=False,\n infer_table_structure=True,\n chunking_strategy=\"by_title\",\n max_characters=4000,\n new_after_n_chars=3800,\n combine_text_under_n_chars=2000,\n image_output_dir_path=pdf_folder_path,\n)\n\n# Categorize extracted elements from a PDF into tables and texts.\ntables = []\ntexts = []\nfor element in raw_pdf_elements:\n if \"unstructured.documents.elements.Table\" in str(type(element)):\n tables.append(str(element))\n elif \"unstructured.documents.elements.CompositeElement\" in str(type(element)):\n texts.append(str(element))\n\n# Optional: Enforce a specific token size for texts\ntext_splitter = CharacterTextSplitter.from_tiktoken_encoder(\n chunk_size=10000, chunk_overlap=0\n)\njoined_texts = \" \".join(texts)\ntexts_4k_token = text_splitter.split_text(joined_texts)<\/code><\/pre>\n\n
def generate_text_summaries(\n texts: list[str], tables: list[str], summarize_texts: bool = False\n) -> tuple[list, list]:\n \"\"\"\n Summarize text elements\n texts: List of str\n tables: List of str\n summarize_texts: Bool to summarize texts\n \"\"\"\n\n # Prompt\n prompt_text = \"\"\"You are an assistant tasked with summarizing tables and text for retrieval. \\\n These summaries will be embedded and used to retrieve the raw text or table elements. \\\n Give a concise summary of the table or text that is well optimized for retrieval. Table or text: {element} \"\"\"\n prompt = PromptTemplate.from_template(prompt_text)\n empty_response = RunnableLambda(\n lambda x: AIMessage(content=\"Error processing document\")\n )\n # Text summary chain\n model = VertexAI(\n temperature=0, model_name=MODEL_NAME, max_output_tokens=TOKEN_LIMIT\n ).with_fallbacks([empty_response])\n summarize_chain = {\"element\": lambda x: x} | prompt | model | StrOutputParser()\n\n # Initialize empty summaries\n text_summaries = []\n table_summaries = []\n\n # Apply to text if texts are provided and summarization is requested\n if texts:\n if summarize_texts:\n text_summaries = summarize_chain.batch(texts, {\"max_concurrency\": 1})\n else:\n text_summaries = texts\n\n # Apply to tables if tables are provided\n if tables:\n table_summaries = summarize_chain.batch(tables, {\"max_concurrency\": 1})\n\n return text_summaries, table_summaries\n\n\n# Get text, table summaries\ntext_summaries, table_summaries = generate_text_summaries(\n texts_4k_token, tables, summarize_texts=True\n)<\/code><\/pre>\ndef encode_image(image_path: str) -> str:\n \"\"\"Getting the base64 string\"\"\"\n with open(image_path, \"rb\") as image_file:\n return base64.b64encode(image_file.read()).decode(\"utf-8\")\n\n\ndef image_summarize(model: ChatVertexAI, base64_image: str, prompt: str) -> str:\n \"\"\"Make image summary\"\"\"\n msg = model.invoke(\n [\n HumanMessage(\n content=[\n {\"type\": \"text\", \"text\": prompt},\n {\n \"type\": \"image_url\",\n \"image_url\": {\"url\": f\"data:image\/png;base64,{base64_image}\"},\n },\n ]\n )\n ]\n )\n return msg.content\n\n\ndef generate_img_summaries(path: str) -> tuple[list[str], list[str]]:\n \"\"\"\n Generate summaries and base64 encoded strings for images\n path: Path to list of .jpg files extracted by Unstructured\n \"\"\"\n\n # Store base64 encoded images\n img_base64_list = []\n\n # Store image summaries\n image_summaries = []\n\n # Prompt\n prompt = \"\"\"You are an assistant tasked with summarizing images for retrieval. \\\n These summaries will be embedded and used to retrieve the raw image. \\\n Give a concise summary of the image that is well optimized for retrieval.\n If it's a table, extract all elements of the table.\n If it's a graph, explain the findings in the graph.\n Do not include any numbers that are not mentioned in the image.\n \"\"\"\n\n model = ChatVertexAI(model_name=MODEL_NAME, max_output_tokens=TOKEN_LIMIT)\n\n # Apply to images\n for img_file in sorted(os.listdir(path)):\n if img_file.endswith(\".png\"):\n base64_image = encode_image(os.path.join(path, img_file))\n img_base64_list.append(base64_image)\n image_summaries.append(image_summarize(model, base64_image, prompt))\n\n return img_base64_list, image_summaries\n\n\n# Image summaries\nimg_base64_list, image_summaries = generate_img_summaries(\".\")<\/code><\/pre>\nStep 7: Create and Deploy a Vertex AI Vector Search Index and Endpoint<\/h3>\n
# https:\/\/cloud.google.com\/vertex-ai\/generative-ai\/docs\/model-reference\/text-embeddings\nDIMENSIONS = 768 # Dimensions output from textembedding-gecko\n\nindex = aiplatform.MatchingEngineIndex.create_tree_ah_index(\n display_name=\"mm_rag_langchain_index\",\n dimensions=DIMENSIONS,\n approximate_neighbors_count=150,\n leaf_node_embedding_count=500,\n leaf_nodes_to_search_percent=7,\n description=\"Multimodal RAG LangChain Index\",\n index_update_method=\"STREAM_UPDATE\",\n)<\/code><\/pre>\nDEPLOYED_INDEX_ID = \"mm_rag_langchain_index_endpoint\"\n\nindex_endpoint = aiplatform.MatchingEngineIndexEndpoint.create(\n display_name=DEPLOYED_INDEX_ID,\n description=\"Multimodal RAG LangChain Index Endpoint\",\n public_endpoint_enabled=True,\n)<\/code><\/pre>\n\n
index_endpoint = index_endpoint.deploy_index(\n index=index, deployed_index_id=\"mm_rag_langchain_deployed_index\"\n)\nindex_endpoint.deployed_indexes<\/code><\/pre>\nStep 8: Create Retriever and Load Documents<\/h3>\n
# The vectorstore to use to index the summaries\nvectorstore = VectorSearchVectorStore.from_components(\n project_id=PROJECT_ID,\n region=LOCATION,\n gcs_bucket_name=GCS_BUCKET,\n index_id=index.name,\n endpoint_id=index_endpoint.name,\n embedding=VertexAIEmbeddings(model_name=EMBEDDING_MODEL_NAME),\n stream_update=True,\n)<\/code><\/pre>\ndocstore = InMemoryStore()\n\nid_key = \"doc_id\"\n# Create the multi-vector retriever\nretriever_multi_vector_img = MultiVectorRetriever(\n vectorstore=vectorstore,\n docstore=docstore,\n id_key=id_key,\n)<\/code><\/pre>\n# Raw Document Contents\ndoc_contents = texts + tables + img_base64_list\n\ndoc_ids = [str(uuid.uuid4()) for _ in doc_contents]\nsummary_docs = [\n Document(page_content=s, metadata={id_key: doc_ids[i]})\n for i, s in enumerate(text_summaries + table_summaries + image_summaries)\n]\n\nretriever_multi_vector_img.docstore.mset(list(zip(doc_ids, doc_contents)))\n\n# If using Vertex AI Vector Search, this will take a while to complete.\n# You can cancel this cell and continue later.\nretriever_multi_vector_img.vectorstore.add_documents(summary_docs)<\/code><\/pre>\nStep 9: Create Chain with Retriever and Gemini LLM<\/h3>\n
def looks_like_base64(sb):\n \"\"\"Check if the string looks like base64\"\"\"\n return re.match(\"^[A-Za-z0-9+\/]+[=]{0,2}$\", sb) is not None\n\n\ndef is_image_data(b64data):\n \"\"\"\n Check if the base64 data is an image by looking at the start of the data\n \"\"\"\n image_signatures = {\n b\"\\xFF\\xD8\\xFF\": \"jpg\",\n b\"\\x89\\x50\\x4E\\x47\\x0D\\x0A\\x1A\\x0A\": \"png\",\n b\"\\x47\\x49\\x46\\x38\": \"gif\",\n b\"\\x52\\x49\\x46\\x46\": \"webp\",\n }\n try:\n header = base64.b64decode(b64data)[:8] # Decode and get the first 8 bytes\n for sig, format in image_signatures.items():\n if header.startswith(sig):\n return True\n return False\n except Exception:\n return False\n\n\ndef split_image_text_types(docs):\n \"\"\"\n Split base64-encoded images and texts\n \"\"\"\n b64_images = []\n texts = []\n for doc in docs:\n # Check if the document is of type Document and extract page_content if so\n if isinstance(doc, Document):\n doc = doc.page_content\n if looks_like_base64(doc) and is_image_data(doc):\n b64_images.append(doc)\n else:\n texts.append(doc)\n return {\"images\": b64_images, \"texts\": texts}\n\n\ndef img_prompt_func(data_dict):\n \"\"\"\n Join the context into a single string\n \"\"\"\n formatted_texts = \"\\n\".join(data_dict[\"context\"][\"texts\"])\n messages = [\n {\n \"type\": \"text\",\n \"text\": (\n \"You are financial analyst tasking with providing investment advice.\\n\"\n \"You will be given a mix of text, tables, and image(s) usually of charts or graphs.\\n\"\n \"Use this information to provide investment advice related to the user's question. \\n\"\n f\"User-provided question: {data_dict['question']}\\n\\n\"\n \"Text and \/ or tables:\\n\"\n f\"{formatted_texts}\"\n ),\n }\n ]\n\n # Adding image(s) to the messages if present\n if data_dict[\"context\"][\"images\"]:\n for image in data_dict[\"context\"][\"images\"]:\n messages.append(\n {\n \"type\": \"image_url\",\n \"image_url\": {\"url\": f\"data:image\/jpeg;base64,{image}\"},\n }\n )\n return [HumanMessage(content=messages)]\n\n\n# Create RAG chain\nchain_multimodal_rag = (\n {\n \"context\": retriever_multi_vector_img | RunnableLambda(split_image_text_types),\n \"question\": RunnablePassthrough(),\n }\n | RunnableLambda(img_prompt_func)\n | ChatVertexAI(\n temperature=0,\n model_name=MODEL_NAME,\n max_output_tokens=TOKEN_LIMIT,\n ) # Multi-modal LLM\n | StrOutputParser()\n)<\/code><\/pre>\nStep 10: Test the Model<\/h3>\n
1. Process User Query<\/h4>\n
query = \"What are the EV \/ NTM and NTM rev growth for MongoDB, Cloudflare, and Datadog?\n\"<\/code><\/pre>\n2. Get Retrieved documents<\/h4>\n
# List of source documents\ndocs = retriever_multi_vector_img.get_relevant_documents(query, limit=1)\n\n# We get relevant docs\nlen(docs)\n\ndocs<\/code><\/pre>\n![\"RAG](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/74ecaca749ae459a_856.webp\")
<\/figure>\n3. Get generative response<\/h4>\n
plt_img_base64(docs[3])<\/code><\/pre>\n![\"EV](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/989ad388127f5d60_856.webp\")
<\/figure>\nresult = chain_multimodal_rag.invoke(query)\n\nfrom IPython.display import Markdown as md\nmd(result)<\/code><\/pre>\n![\"Vertex](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/989ad388127f5d60_856_1.webp\")
<\/figure>\nPractical Applications<\/h2>\n
\n
Conclusion<\/h2>\n
Key Takeaways<\/h4>\n
\n
Frequently Asked Questions<\/h2>\n
![\"Soumyadarshan](\"https:\/\/av-eks-lekhak.s3.amazonaws.com\/media\/lekhak-profile-images\/converted_image_0GnayIY.webp\")
<\/p>\n