The intersection of artificial intelligence<\/a> and data processing has evolved significantly with the rise of multimodal Retrieval-Augmented Generation<\/a> systems. Multimodal RAG goes beyond traditional models that focus only on text. It integrates various data types like text, images, audio, and video. This allows for more nuanced and context-aware responses. A key innovation is Nomic vision embeddings. They create a unified space for both visual and textual data. This enables seamless interaction across different formats. By using advanced models to generate high-quality embeddings, multimodal RAG improves information retrieval. It bridges the gap between different content forms. The result is richer and more informative user experiences.<\/p>\n This article was published as a part of the\u00a0<\/strong><\/em>Data Science Blogathon.<\/strong><\/em><\/a><\/p>\n Multimodal RAG represents a significant advancement in artificial intelligence. It is build upon traditional RAG systems by incorporating diverse data types such as text, images, audio, and video. Unlike conventional RAG systems that primarily process textual information, multimodal RAG is designed to handle and integrate multiple forms of data simultaneously. This capability allows for more comprehensive understanding and generation of responses that are context-aware across different modalities. <\/p>\n Key Components of Multimodal RAG<\/strong><\/p>\n A significant innovation in this field of multimodal embeddings is the incorporation of Nomic vision embeddings, which create a cohesive embedding space for both visual and textual data.\u00a0<\/p>\n Nomic Embed Vision v1 and v1.5\u00a0<\/a>are both high-quality vision embedding models developed by Nomic AI, designed to share the same latent space as their corresponding text embedding models,\u00a0Nomic Embed Text v1 and v1.5, respectively.\u00a0It operates within the same space as Nomic Embed Text, making it well-suited for multimodal tasks such as text-to-image retrieval. With a vision encoder comprising only 92M parameters, Nomic Embed Vision is well-suited for high-volume production applications, complementing the 137M parameters of Nomic Embed Text.<\/p>\n CLIP models suffer in unimodal tasks<\/strong><\/p>\n Multimodal models such as CLIP demonstrate remarkable zero-shot capabilities across different modalities. However, CLIP\u2019s text encoders struggle with tasks beyond image retrieval, as seen in benchmarks like MTEB, which evaluates the effectiveness of text embedding models. Nomic Embed Vision aims to address these limitations by aligning a vision encoder with the existing Nomic Embed Text latent space.<\/p>\n To tackle the issue of underperformance on unimodal tasks, such as semantic similarity, Nomic Embed Vision, a vision encoder, was trained alongside Nomic Embed Text, a long-context text encoder. The training method involved freezing the text encoder and training the vision encoder on image-text pairs. This approach not only produced optimal results but also ensured backward compatibility with the embeddings from Nomic Embed Text.<\/p>\n As mentioned earlier, existing multimodal models such as CLIP exhibit impressive zero-shot capabilities across different modalities. However, the performance of CLIP\u2019s text encoders is subpar outside of tasks like image retrieval, as evidenced by benchmarks like MTEB, which evaluates the quality of text embedding models. Nomic Embed Vision is specifically designed to address these shortcomings by aligning a vision encoder with the existing Nomic Embed Text latent space. This alignment results in a unified multimodal latent space that delivers strong performance on image, text, and multimodal tasks, as demonstrated by the Imagenet Zero-Shot, MTEB, and Datacomp benchmarks.<\/p>\n In this tutorial, we will build a multimodal RAG system that can efficiently retrieve information from a PDF containing both textual and visual content. We will build this on Google Colab using T4 GPU (Free tier).<\/p>\n Install all required Python libraries, including OpenAI, Qdrant, Transformers, Torch, and PyMuPDF.<\/p>\n Set up the OpenAI API key and import essential libraries like PyMuPDF, PIL, LangChain, and OpenAI.<\/p>\n Set up the OpenAI API key and import essential libraries like PyMuPDF, PIL, LangChain, and OpenAI.<\/p>\n Use PyMuPDF to extract text from all pages of the PDF and store it in a list.<\/p>\n Save images in the \u201ctest\u201d directory and extract text for further processing.<\/p>\n We use this PDF <\/a>that has both text and images or charts to test the multimodal RAG.\u00a0<\/p>\n We save the images extracted from the PDF using the\u00a0PyMuPDF library in the \u201ctest\u201d directory. In the next steps, create embeddings of these images so as to be able to retrieve information from them in future based on a user query.<\/p>\n Split extracted text into smaller chunks using LangChain\u2019s RecursiveCharacterTextSplitter.<\/p>\n Load Nomic\u2019s text and vision embedding models using Hugging Face\u2019s Transformers library.<\/p>\n Convert text and images into vector embeddings for efficient retrieval.<\/p>\n Qdrant\u00a0is\u00a0an open-source vector database and\u00a0search engine\u00a0designed to efficiently store, manage, and query high-dimensional vectors.We save our embeddings in this vector DB.<\/p>\n Save image embeddings in a separate Qdrant collection for multimodal retrieval.<\/p>\n Retrieve the most relevant text and image embeddings based on a user query.<\/p>\n Use LangChain to process retrieved text and images, generating context-aware responses using GPT-4o.<\/p>\n Let us now query our multimodal RAG system with different queries to test its multimodal capability,\u00a0<\/p>\n Output<\/b>:<\/p>\n The response to this query is only present in the following chart (Fig 4) in the PDF <\/a>and not in any text. So our, multimodal RAG is able to retrieve this information accurately.<\/p>\n Output<\/b>:<\/p>\n The response to this query as well is only present in the following diagram (Fig 3) in the PDF <\/a>and not in any text. So our, multimodal RAG is able to retrieve this information accurately.<\/p>\nLearning Objectives<\/h4>\n
\n
What is Multimodal RAG?<\/h2>\n
\n
Nomic Vision Embeddings<\/h2>\n
Performance Benchmarks of Nomic Vision Embeddings<\/h2>\n
Hands on Python Implementation of MultiModal RAG with Nomic Vision Embeddings<\/h2>\n
Step 1: Installing Necessary Libraries<\/h3>\n
!pip install openai==1.55.3 httpx==0.27.2 \n!pip install qdrant_client\n!pip install transformers\n!pip install transformers torch pillow\n!pip install --upgrade nltk\n!pip install sentence-transformers\n!pip install --upgrade qdrant-client fastembed Pillow\n!pip install PyMuPDF<\/code><\/pre>\nStep 2: Setting OpenAI API key and Importing Necessary Libraries<\/h3>\n
from openai import ChatCompletion\nimport openai\nimport os\nfrom openai import AzureOpenAI\nfrom PIL import Image\nimport torch\nimport numpy as np\nimport fitz # PyMuPDF\nimport os\nimport time\nfrom langchain_text_splitters import RecursiveCharacterTextSplitter\nfrom langchain_core.messages import HumanMessage\nfrom langchain_openai import ChatOpenAI\nfrom openai import ChatCompletion\nimport openai\nimport base64\nfrom base64 import b64decode\n\nos.environ[\"OPENAI_API_KEY\"] = ''<\/code><\/pre>\n#images\n\ndef extract_images_from_pdf(pdf_path, output_folder):\n pdf_document = fitz.open(pdf_path)\n os.makedirs(output_folder, exist_ok=True)\n #Iterating throught the pages in the PDF\n for page_number in range(len(pdf_document)):\n page = pdf_document[page_number]\n #Function For Getting Images From the PDF Pages\n images = page.get_images(full=True)\n\n for image_index, img in enumerate(images):\n xref = img[0]\n base_image = pdf_document.extract_image(xref)\n image_bytes = base_image[\"image\"]\n image_ext = base_image[\"ext\"]\n image_filename = f\"page_{page_number+1}_image_{image_index+1}.{image_ext}\"\n image_path = os.path.join(output_folder, image_filename)\n with open(image_path, \"wb\") as image_file:\n image_file.write(image_bytes)\n pdf_document.close()<\/code><\/pre>\ndef extract_text_pdf(path):\n \"\"\"Extracts text from a PDF using PyMuPDF.\"\"\"\n doc = fitz.open(path)\n text_results = []\n for page in doc:\n text = page.get_text()\n text_results.append(text)\n return text_results<\/code><\/pre>\nStep 5: Saving Extracted Text and Images From PDF<\/h3>\n
def get_contents(pdf_path, output_directory):\n \"\"\"Extracts text and images from a PDF, saves images, and returns text and elapsed time.\"\"\"\n\n extract_images_from_pdf(pdf_path, output_directory)\n text_results=extract_text_pdf(pdf_path)\n return(text_results)\n \npdf_path = \"\/content\/retailcoffee.pdf\"\noutput_directory = \"\/content\/test\"\ntext_results=get_contents(pdf_path, output_directory)<\/code><\/pre>\nStep 6. Chunking Text Data For RAG<\/h3>\n
text_splitter = RecursiveCharacterTextSplitter(\n chunk_size=2048,\n chunk_overlap=50,\n length_function=len,\n is_separator_regex=False,\n separators=[\n \"\\n\\n\",\n \"\\n\",\n \" \",\n \".\",\n \",\",\n \"\\u200b\", # Zero-width space\n \"\\uff0c\", # Fullwidth comma\n \"\\u3001\", # Ideographic comma\n \"\\uff0e\", # Fullwidth full stop\n \"\\u3002\", # Ideographic full stop\n \"\",\n ],\n )\n\ndoc_texts = text_splitter.create_documents(text_results)<\/code><\/pre>\nStep 7: Loading Nomic Text Embedding Model and\u00a0Nomic Vision Embedding Model<\/h3>\n
from transformers import AutoTokenizer, AutoModel\n\n# Load the tokenizer and model\ntext_tokenizer = AutoTokenizer.from_pretrained(\"nomic-ai\/nomic-embed-text-v1.5\", trust_remote_code=True)\ntext_model = AutoModel.from_pretrained(\"nomic-ai\/nomic-embed-text-v1.5\", trust_remote_code=True)\n\ndef text_embeddings(text):\n inputs = text_tokenizer(text, return_tensors=\"pt\", padding=True, truncation=True)\n outputs = text_model(**inputs)\n embeddings = outputs.last_hidden_state.mean(dim=1)\n return embeddings[0].detach().numpy()\n \nfrom transformers import AutoModel, AutoProcessor\nfrom PIL import Image\nimport torch\nmodel = AutoModel.from_pretrained(\"nomic-ai\/nomic-embed-vision-v1.5\", trust_remote_code=True)\nprocessor = AutoProcessor.from_pretrained(\"nomic-ai\/nomic-embed-vision-v1.5\")<\/code><\/pre>\nStep 8: Generating Text and Image Embeddings For Our Data<\/h3>\n
#Text Embeddins\ntexts_embeded = [text_embeddings(document.page_content) for document in doc_texts]\n\n#Image Embeddings\nimage_embeddings = []\nfor img in image_files:\n try:\n image = Image.open(os.path.join(output_directory, img))\n inputs = processor(images=image, return_tensors=\"pt\")\n with torch.no_grad():\n outputs = model(**inputs)\n embeddings = outputs.last_hidden_state\n if embeddings.size(0) > 0: # Ensure the batch size is non-zero\n\n image_embedding = embeddings.mean(dim=1).squeeze().cpu().numpy()\n image_embeddings.append(image_embedding)\n else:\n print(f\"No Embeddings For {img}\")\n\n except Exception as e:\n print(e)\n\n#SIZE OF Text & Image Embeddings\ntext_embeddings_size=len(texts_embeded[0])\nimage_embeddings_size=len(image_embeddings[0])<\/code><\/pre>\nStep 9: Storing Text Embeddings in Qdrant\u00a0<\/h3>\n
from qdrant_client import QdrantClient, models\n\nclient = QdrantClient(\":memory:\")\n\nif not client.collection_exists(\"text1\"): #creating a Collection\n client.create_collection(\n collection_name =\"text1\",\n vectors_config=models.VectorParams(\n size=text_embeddings_size, # Vector size is defined by used model\n distance=models.Distance.COSINE,\n ),\n )\n \n client.upload_points(\n collection_name=\"text1\",\n points=[\n models.PointStruct(\n id=str(uuid.uuid4()),\n vector=np.array(texts_embeded[idx]),\n payload={\n \"metadata\": doc.metadata,\n \"content\": doc.page_content\n }\n )\n for idx, doc in enumerate(doc_texts)\n ]\n)<\/code><\/pre>\nStep 10: Storing Image Embeddings in Qdrant\u00a0<\/h3>\n
if not client.collection_exists(\"images1\"):\n client.create_collection(\n collection_name=\"images1\",\n vectors_config=models.VectorParams(\n size=image_embeddings_size, # Vector size is defined by used model\n distance=models.Distance.COSINE,\n ),\n )\n \n# Ensure that image_embeddings are not empty\nif len(image_embeddings) > 0:\n client.upload_points(\n collection_name=\"images1\",\n points=[\n models.PointStruct(\n id=str(uuid.uuid4()), # unique id\n vector= np.array(image_embeddings[idx]) ,\n payload={\"image_path\": output_directory+'\/'+str(image_files[idx])} # Image path as metadata\n )\n for idx in range(len(image_embeddings)) \n )\nelse:\n print(\"No embeddings found\")\n<\/code><\/pre>\nStep 11: Creating a MultiModal Retriever For Retrieving Images and Text<\/h3>\n
def MultiModalRetriever(query):\n\n query = text_embeddings(query)\n\n # Retrieve text hits\n text_hits = client.query_points(\n collection_name=\"text1\",\n query=query,\n limit=3,3\n ).points\n # Retrieve image hits\n Image_hits = client.query_points(\n collection_name=\"images1\",\n query=query,\n limit=5,\n ).points\n\n return text_hits, Image_hits\n<\/code><\/pre>\nStep 12: Creating a MultiModal RAG using LangChain<\/h3>\n
def MultiModalRAG(context,images,user_query,model): \n # Helper function to encode an image as a base64 string\n def encode_image(image_path):\n if image_path:\n with open(image_path, \"rb\") as image_file:\n return base64.b64encode(image_file.read()).decode()\n return None\n\n\n image_paths = images \n #three images based on retrived images\n img_base64 = encode_image(image_paths[0]) \n img_base641 = encode_image(image_paths[1]) \n img_base642 = encode_image(image_paths[2]) \n\n message = HumanMessage(\n content=[\n {\"type\": \"text\", \"text\": \"BASED ON RETRIEVED CONTEXT %s ONLY, ANSWER THE FOLLOWING QUERY %s. Context can be tables, texts or Images\"%(context,user_query)},\n {\n \"type\": \"image_url\",\n \"image_url\": {\"url\": f\"data:image\/jpeg;base64,{img_base64}\"},\n },\n {\n \"type\": \"image_url\",\n \"image_url\": {\"url\": f\"data:image\/jpeg;base64,{img_base641}\"},\n },\n {\n \"type\": \"image_url\",\n \"image_url\": {\"url\": f\"data:image\/jpeg;base64,{img_base642}\"},\n },\n ],)\n\n model = ChatOpenAI(model=model) \n response = model.invoke([message])\n return response.content\n\n\ndef RAG(query):\n text_hits, Image_hits=MultiModalRetriever(query)\n\n retrieved_images=[i.payload['image_path'] for i in Image_hits]\n print(retrieved_images)\n answer=MultiModalRAG(text_hits,retrieved_images,query,\"gpt-4o\")\n return(answer)<\/code><\/pre>\nQuerying the Model<\/h2>\n
RAG(\"Revenue of Starbucks in billion dollars of Food in 2020?\")<\/code><\/pre>\n'Based on the chart showing Starbucks' revenue by product for 2020, the revenue from
![\"response](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/image_88_7KLzHqg.webp\")
<\/figure>\nRAG(\"Explain what the Ansoff Matrix is for Starbucks.\")<\/pre>\n
![\"output\"](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/image_90.webp\")
<\/figure>\nRAG(\"Global coffee consumption in 2017\")<\/pre>\n
![\"coffee](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/image_53WBphV.webp\")
<\/figure>\n![\"Nibedita](\"https:\/\/av-eks-lekhak.s3.amazonaws.com\/media\/lekhak-profile-images\/converted_image_ZkJo4gb.webp\")
<\/p>\n