Imagine a world where finding information in a document is as easy as asking a question\u2014and getting a response that combines both text and images seamlessly. In this guide, we dive into building a Multimodal Retrieval-Augmented Generation<\/a> pipeline that can do just that. You\u2019ll learn how to parse text and images from a PDF slide deck using tools like LlamaParse, create contextual summaries for enhanced retrieval, and feed this data into advanced models like GPT-4 for query answering. Along the way, we\u2019ll explore how contextual retrieval improves accuracy, optimize costs with prompt caching, and compare results between baseline and enhanced pipelines. Get ready to unlock the potential of RAG with this step-by-step walkthrough! <\/p>\n This article was published as a part of the\u00a0<\/strong><\/em>Data Science Blogathon.<\/strong><\/em><\/a><\/p>\n Contextual retrieval was initially introduced in this Anthropic\u00a0blog post<\/a>. The high-level intuition is that every chunk is given a concise summary of where that chunk fits in with respect to the overall summary of the document. This allows insertion of high-level concepts\/keywords that enable this chunk to be better retrieved for different types of queries.<\/p>\n These LLM<\/a> calls are expensive. Contextual retrieval depends on\u00a0prompt caching\u00a0in order to be efficient.<\/p>\n In this notebook, we use Claude 3.5-Sonnet <\/a>to generate contextual summaries. We cache the document as text tokens, but generate contextual summaries by feeding in the parsed text chunk.<\/p>\n We feed both the text and image chunks into the final multimodal RAG pipeline to generate the response.<\/p>\n In a Retrieval-Augmented Generation (RAG) pipeline, we typically:<\/p>\n Contextual Retrieval is a neat enhancement to standard RAG. Each chunk of text is annotated with a short summary that situates it within the broader document context. This helps the retriever pick the chunk more accurately for queries that might not match the exact words but relate to the overall topic or concept.<\/p>\n We\u2019ll demonstrate how to build a Multimodal RAG <\/a>pipeline over a PDF slide deck, using:<\/p>\n We will also show how to cache LLM calls to minimize costs, since Contextual Retrieval can generate a lot of prompt calls.<\/p>\n You\u2019ll need to install or upgrade a few packages:<\/p>\n Additionally<\/b>:<\/p>\n We setup an integration with LlamaTrace (integration with Arize).<\/p>\n If you haven\u2019t already done so, make sure to create an account here:\u00a0https:\/\/llamatrace.com\/login<\/a>. Then create an API key and put it in the\u00a0PHOENIX_API_KEY\u00a0variable below.<\/p>\n Voyage AI utilizes API keys to monitor usage and manage permissions. To obtain your key, please sign in with your Voyage AI account and click the \u201cCreate new API key\u201d button in the\u00a0dashboard<\/a>. Add Payment details as well , but still Your first 200 million tokens are still free for Voyage series 3 models.<\/p>\n Phoenix API key can be obtained by signing up for LlamaTrace here<\/a> , then navigate to the bottom left panel and click on \u2018Keys\u2019 where you should find your\u00a0 API key.<\/p>\n In our example, we\u2019ll parse the ICONIQ 2024 State of AI Report. This PDF is publicly available at the URL below. If you prefer, you can replace it with any PDF you have.<\/p>\n Let\u2019s set up the core components required to build and implement our Multimodal RAG pipeline effectively.<\/p>\n In this example, use LlamaParse to parse both the text and images from the document.<\/p>\n We parse out the text with LlamaParse premium.<\/p>\n NOTE: The report has 40 pages, and at ~5c per page, this will cost you $2. Just a heads up!<\/p>\n For obtaining the LlamaCloud API key, click on the \u2018Get started\u2019 here\u00a0https:\/\/www.llamaindex.ai\/contact , and login. Once redirected to the LlamaCloud dashboard<\/a>, generate a new API key by navigating to the API pane on the left.<\/p>\n Multimodal nodes are the building blocks that allow us to process and integrate diverse data types like text and images. Here, we\u2019ll construct nodes to parse, embed, and index chunks from a PDF slide deck, setting the foundation for a robust retrieval system.<\/p>\n Each PDF page corresponds to one \u201cnode\u201d containing:<\/p>\n In this step, we\u2019ll split the PDF pages into smaller, manageable text nodes. This ensures efficient embedding and retrieval by breaking down the content into meaningful chunks for precise contextual analysis.<\/p>\n Contextual retrieval attaches a short, high-level summary to each chunk, describing where it fits into the overall document. We\u2019ll use the LLM to generate these short summaries and store them in each node\u2019s metadata[\u201ccontext\u201d].<\/p>\n We\u2019ll now embed these summarized chunks and store them in a vector store for retrieval. LlamaIndex can persist indices locally or integrate with 40+ external vector databases.<\/p>\n We\u2019ll also build a \u201cbaseline\u201d index on the original text nodes (without the contextual summaries) to compare the difference in retrieval quality.<\/p>\n We want a RAG pipeline that:<\/p>\n Let\u2019s query our new pipeline about AI usage by department.<\/p>\n A typical response might look like this:<\/p>\n 1. **AI, Machine Learning, and Data Science** with a score of 4.5. These scores are derived from a survey where respondents rated the level of In terms of how these departments are using generative AI:<\/p> - **AI, Machine Learning, and Data Science**: While specific use cases for this - **IT**: The IT department is using generative AI for several impactful use cases, The information about the departments and their use cases comes from the parsed Comparatively, if we run the same query on the baseline index:<\/p>\n You\u2019ll see the baseline might have fewer details or slightly different retrieval results. Contextual retrieval gives more precise context around the IT usage specifically. The response would look like:<\/p>\n 1. **AI, Machine Learning, and Data Science** - This department has the highest 2. **IT** - With a score of 4.0, the IT department is also a leading user of GenAI. 3. **Engineering \/ R&D** - This department has a score of 3.9. While specific use The information is derived from the parsed markdown text, which provides a detailed Here\u2019s another example query,\u00a0In this next question, the same sources are retrieved with and without contextual retrieval, and the answer is correct for both approaches. This is thanks for LlamaParse Premium\u2019s ability to comprehend graphs.<\/p>\n From the parsed markdown text, we can gather the following insights related to 1. **Deployment Environments**: Enterprises are primarily hosting generative AI 2. **Cloud Service Providers**: The most utilized cloud service providers for These insights are derived from the parsed markdown text, specifically from the Now, lets try with the baseline approach:<\/p>\n Therefore, based on the parsed markdown text provided, there are no relevant insights available from the \u2018deep dive on infrastructure\u2019 section regarding model preferences, cost, or deployment environments. If there were any images associated with this section, they were not provided, and thus no additional insights could be derived from them.<\/p>\n This conclusion is drawn from the parsed markdown text, which lacks any specific information on model preferences, cost, or deployment environments in that section. The image confirms this, as it only shows the title and a graphic without additional details.<\/p>\n If you need insights on these topics, you might want to refer to other sections or slides that specifically address model preferences, costs, or deployment environments.<\/p>\n Comparing both answers helps demonstrate that those short \u201ccontextual summaries\u201d in your metadata often lead to more relevant retrieval.<\/p>\n A big thanks to Jerry Liu from LlamaIndex for creating this amazing pipeline.<\/p>\n In this tutorial, we explored the process of parsing a PDF slide deck using LlamaParse to extract both text and images, enriching each text chunk with contextual summaries to enhance retrieval accuracy. We demonstrated how to build a Multimodal RAG pipeline with LlamaIndex, integrating both textual and visual data into a powerful model like GPT-4, showcasing the potential of multimodal LLMs. Finally, we compared results from a baseline index to a contextual index, highlighting the significant improvements in retrieval precision and relevance achieved through the contextual approach. This comprehensive guide equips you with the tools and techniques to build effective multimodal AI solutions.<\/p>\n With these steps, you can adapt the approach to any <\/b>PDF or external data source\u2014whether it\u2019s a huge enterprise knowledge base, marketing materials, or your company\u2019s internal documentation.<\/p>\n A. Contextual Retrieval is an approach where each chunk of text in your dataset has a concise summary that situates it within the broader document. This helps your retriever better match relevant chunks\u2014especially for queries that rely on thematic or conceptual overlaps rather than exact keyword matches.<\/p>\n<\/p><\/div>\n A. In a Multimodal RAG pipeline, you not only retrieve and feed text chunks into the LLM but also related images, audio, or other modalities. This is especially useful when your data sources are slide decks, PDFs with charts, or any materials that mix text with images. It allows the model to reference both textual and visual content for a more comprehensive answer.<\/p>\n<\/p><\/div>\n A. LlamaParse is a parsing utility that can extract both text and images from a PDF. Traditional PDF extractors often only get the text or struggle with embedded charts and diagrams. With LlamaParse, you can create \u201cnodes\u201d that include a reference to each PDF page\u2019s image file\u2014enabling genuine multimodal retrieval.<\/p>\n<\/p><\/div>\n A. No, it isn\u2019t mandatory, but it\u2019s a great way to benchmark the difference. Having a baseline index helps you see how retrieval results change when you add contextual summaries.<\/p>\n<\/p><\/div>\n<\/p><\/div>\n This article was published as a part of the\u00a0<\/strong><\/em>Data Science Blogathon.<\/strong><\/em><\/a><\/p>\nLearning Objectives<\/h3>\n
\n
Building a Contextual Multimodal RAG Pipeline<\/h2>\n
\n
Overview of the Multimodal RAG Pipeline<\/h3>\n
\n
Environment Setup and Dependencies<\/h2>\n
!pip install -U llama-index llama-parse\n!pip install -U llama-index-callbacks-arize-phoenix<\/code><\/pre>\n\n
Setup Observability with LlamaTrace (Arize Integration)<\/h3>\n
import os\nimport nest_asyncio\n\nnest_asyncio.apply()\n\n# Arize Phoenix\nPHOENIX_API_KEY = \"Load and Parse the PDF Slides<\/h2>\n
!mkdir data\n!mkdir data_images_iconiq\n!wget \"https:\/\/cdn.prod.website-files.com\/65e1d7fb19a3e64b5c36fb38\/66eb856e019e59758ef73759_ICONIQ%20Analytics%20%2B%20Insights%20-%20State%20of%20AI%20Sep24.pdf\" -O data\/iconiq_report.pdf<\/code><\/pre>\nModel Setup<\/h3>\n
import os\nfrom llama_index.llms.anthropic import Anthropic\nfrom llama_index.embeddings.voyageai import VoyageEmbedding\nfrom llama_index.core import Settings\n\n# Replace with your actual keys\nos.environ[\"ANTHROPIC_API_KEY\"] = \"sk-...\"\nos.environ[\"VOYAGE_API_KEY\"] = \"...\"\n\nllm = Anthropic(model=\"claude-3-5-sonnet-20240620\")\nembed_model = VoyageEmbedding(model_name=\"voyage-3\")\n\nSettings.llm = llm\nSettings.embed_model = embed_model<\/code><\/pre>\nParse Text and Images with LlamaParse<\/h3>\n
from llama_parse import LlamaParse\n\nparser = LlamaParse(\n result_type=\"markdown\",\n premium_mode=True,\n # invalidate_cache=True # Uncomment if you want to force a fresh parse\n api_key = 'LlamaCloud-API-Key'\n)\nprint(\"Parsing text...\")\nmd_json_objs = parser.get_json_result(\"data\/iconiq_report.pdf\")\nmd_json_list = md_json_objs[0][\"pages\"]\n\nimage_dicts = parser.get_images(md_json_objs, download_path=\"data_images_iconiq\")<\/code><\/pre>\nBuild Multimodal Nodes<\/h2>\n
\n
Split Pages into Text Nodes<\/h3>\n
from pathlib import Path\nfrom llama_index.core.schema import TextNode\nfrom typing import Optional\nimport re\n\ndef get_page_number(file_name):\n match = re.search(r\"-page_(\\d+)\\.jpg$\", str(file_name))\n if match:\n return int(match.group(1))\n return 0\n\ndef _get_sorted_image_files(image_dir):\n raw_files = [\n f for f in list(Path(image_dir).iterdir()) if f.is_file() and \"-page\" in str(f)\n ]\n return sorted(raw_files, key=get_page_number)\n\ndef get_text_nodes(image_dir, json_dicts):\n nodes = []\n image_files = _get_sorted_image_files(image_dir)\n md_texts = [d[\"md\"] for d in json_dicts]\n\n for idx, md_text in enumerate(md_texts):\n chunk_metadata = {\n \"page_num\": idx + 1,\n \"image_path\": str(image_files[idx]),\n \"parsed_text_markdown\": md_text,\n }\n node = TextNode(text=\"\", metadata=chunk_metadata)\n nodes.append(node)\n\n return nodes\n\ntext_nodes = get_text_nodes(\"data_images_iconiq\", md_json_list)<\/code><\/pre>\nAdd Contextual Summaries<\/h2>\n
from copy import deepcopy\nfrom llama_index.core.llms import ChatMessage\nfrom llama_index.core.prompts import ChatPromptTemplate\nimport time\n\n\nwhole_doc_text = \"\"\"\\\nHere is the entire document.\nBuild and Persist the Index<\/h2>\n
import os\nfrom llama_index.core import (\n StorageContext,\n VectorStoreIndex,\n load_index_from_storage,\n)\n\nif not os.path.exists(\"storage_nodes_iconiq\"):\n index = VectorStoreIndex(new_text_nodes, embed_model=embed_model)\n index.set_index_id(\"vector_index\")\n index.storage_context.persist(\".\/storage_nodes_iconiq\")\nelse:\n storage_context = StorageContext.from_defaults(persist_dir=\"storage_nodes_iconiq\")\n index = load_index_from_storage(storage_context, index_id=\"vector_index\")\n\nretriever = index.as_retriever()<\/code><\/pre>\nBaseline Index (Without Summaries)<\/h3>\n
if not os.path.exists(\"storage_nodes_iconiq_base\"):\n base_index = VectorStoreIndex(text_nodes, embed_model=embed_model)\n base_index.set_index_id(\"vector_index\")\n base_index.storage_context.persist(\".\/storage_nodes_iconiq_base\")\nelse:\n storage_context = StorageContext.from_defaults(\n persist_dir=\"storage_nodes_iconiq_base\"\n )\n base_index = load_index_from_storage(storage_context, index_id=\"vector_index\")<\/code><\/pre>\nBuild a Multimodal Query Engine<\/h2>\n
\n
import base64\nimport openai\nimport os\nfrom typing import Optional, List\n\nfrom llama_index.core.query_engine import CustomQueryEngine\nfrom llama_index.core.base.response.schema import Response\nfrom llama_index.core.retrievers import BaseRetriever\nfrom llama_index.core.prompts import PromptTemplate\nfrom llama_index.core.schema import NodeWithScore, MetadataMode\n\nQA_PROMPT_TMPL = \"\"\"\\\nBelow we give parsed text from slides, as well as images.\n\n---------------------\n{context_str}\n---------------------\n\nGiven the context information and no prior knowledge, please answer the query:\n\nQuery: {query_str}\nAnswer:\n\"\"\"\n\nQA_PROMPT = PromptTemplate(QA_PROMPT_TMPL)\n\ndef encode_image(image_path: str) -> str:\n \"\"\"If you want to inline a local image in base64.\"\"\"\n with open(image_path, \"rb\") as f:\n return base64.b64encode(f.read()).decode(\"utf-8\")\n\n\nclass MultimodalQueryEngine(CustomQueryEngine):\n \"\"\"\n Custom multimodal Query Engine that retrieves text nodes,\n then sends them + image(s) to the new Vision-capable API as documented.\n \"\"\"\n\n def __init__(\n self,\n retriever: BaseRetriever,\n model_name: str = \"gpt-4o\",\n qa_prompt: Optional[PromptTemplate] = None,\n ) -> None:\n super().__init__(qa_prompt=qa_prompt or QA_PROMPT)\n self.retriever = retriever\n self.model_name = model_name\n\n def custom_query(self, query_str: str) -> Response:\n # 1) Retrieve text nodes\n node_with_scores: List[NodeWithScore] = self.retriever.retrieve(query_str)\n\n # 2) Build context\n context_str = \"\\n\\n\".join(\n[nws.node.get_content(metadata_mode=MetadataMode.LLM) for nws in node_with_scores]\n\n )\n\n # 3) Format the final prompt\n formatted_prompt_text = self._qa_prompt.format(\n context_str=context_str,\n query_str=query_str,\n )\n\n # 4) Build the user message with text + images\n user_message_content = [\n {\n \"type\": \"text\",\n \"text\": formatted_prompt_text,\n }\n ]\n\n for nws in node_with_scores:\n image_path = nws.node.metadata.get(\"image_path\", \"\")\n if image_path:\n base64_data = encode_image(image_path)\n image_url = f\"data:image\/jpeg;base64,{base64_data}\"\n user_message_content.append(\n {\n \"type\": \"image_url\",\n \"image_url\": {\n \"url\": image_url,\n \"detail\": \"auto\"\n },\n }\n )\n\n messages = [\n {\n \"role\": \"user\",\n \"content\": user_message_content,\n }\n ]\n\n # 5) Call your Vision model\n response = openai.ChatCompletion.create(\n model=self.model_name,\n messages=messages,\n max_tokens=500,\n )\n\n # 6) Return a Response object\n return Response(\n response=response.choices[0].message.content,\n source_nodes=node_with_scores,\n metadata={},\n )\n \n # 2) Create a query engine\nquery_engine = MultimodalQueryEngine(\n retriever=index.as_retriever(similarity_top_k=3),\n model_name=\"gpt-4o\", # or \"gpt-4o-mini\", \"gpt-4-turbo\", etc.\n)\n\nbase_query_engine = MultimodalQueryEngine(\n retriever=base_index.as_retriever(similarity_top_k=3),\n model_name=\"gpt-4o\",\n)<\/code><\/pre>\nTrying Out Queries<\/h2>\n
response = query_engine.query(\n \"Which departments use GenAI the most and how are they using it?\"\n)\nprint(str(response))<\/code><\/pre>\nBased on the parsed markdown text provided, the departments\/teams that use
base_response = base_query_engine.query(\n \"Which departments use GenAI the most and how are they using it?\"\n)\nprint(str(base_response))<\/code><\/pre>\nBased on the parsed markdown text provided, the departments that use Generative AI
Observing the Benefits of Contextual Retrieval<\/h2>\n
query = \"what are relevant insights from the 'deep dive on infrastructure' section in terms of model preferences, cost, deployment environments?\"\n\nresponse = query_engine.query(query)\nprint(str(response))<\/code><\/pre>\nOutput<\/h3>\n
The \"Deep Dive on Infrastructure\" section from the ICONIQ Growth report provides
base_response = base_query_engine.query(query)\nprint(str(base_response))<\/code><\/pre>\nOutput<\/h3>\n
The parsed text from the slides does not provide specific insights regarding model preferences, cost, or deployment environments in the 'deep dive on infrastructure' section. The slide titled \"Deep Dive on Infrastructure\" (page 24) only contains the title, the ICONIQ Growth branding, and confidentiality and copyright notices. There is no detailed information or data presented in the parsed text for this section.<\/code><\/pre>\n\n
Conclusion<\/h2>\n
Key Takeaways<\/h3>\n
\n
Frequently Asked Questions<\/h2>\n
![\"Adarsh](\"https:\/\/av-eks-lekhak.s3.amazonaws.com\/media\/lekhak-profile-images\/converted_image_tHXFGNS.webp\")
<\/p>\n