here <\/a>, then navigate to the bottom left panel and click on \u2018Keys\u2019 where you should find your\u00a0 API key.<\/p>\n
PHOENIX_API_KEY = \"\"\nos.environ[\"OTEL_EXPORTER_OTLP_HEADERS\"] = f\"api_key={PHOENIX_API_KEY}\"\nllama_index.core.set_global_handler(\n \"arize_phoenix\", endpoint=\"https:\/\/llamatrace.com\/v1\/traces\"\n)<\/phoenix_api_key><\/code><\/pre>\nStep 3: Load the data \u2013 Obtain Your Slide Deck<\/h3>\n For demonstration, we use ConocoPhillips\u2019 2023 investor meeting slide deck. We download the PDF:<\/p>\n
import os\nimport requests\n\n# Create the directories (ignore errors if they already exist)\nos.makedirs(\"data\", exist_ok=True)\nos.makedirs(\"data_images\", exist_ok=True)\n\n# URL of the PDF\nurl = \"https:\/\/static.conocophillips.com\/files\/2023-conocophillips-aim-presentation.pdf\"\n\n# Download and save to data\/conocophillips.pdf\nresponse = requests.get(url)\nwith open(\"data\/conocophillips.pdf\", \"wb\") as f:\n f.write(response.content)\n\nprint(\"PDF downloaded to data\/conocophillips.pdf\")<\/code><\/pre>\nCheck if the pdf slide deck is in the data folder, if not place it in the data folder and name it as you want.<\/p>\n
Step 4: Set Up Models<\/h3>\n You need an embedding model and an LLM. In this example:<\/p>\n
from llama_index.llms.openai import OpenAI\nfrom llama_index.embeddings.openai import OpenAIEmbedding\nembed_model = OpenAIEmbedding(model=\"text-embedding-3-large\")\nllm = OpenAI(model=\"gpt-4o\")<\/code><\/pre>\nNext, you register these as the default for LlamaIndex:<\/p>\n
from llama_index.core import Settings\nSettings.embed_model = embed_model\nSettings.llm = llm<\/code><\/pre>\nStep 5: Parse the Document with LlamaParse<\/h3>\n LlamaParse can extract text and images (via a multimodal large model). For each PDF page, it returns:<\/p>\n
\nMarkdown text<\/b> (with tables, headings, bullet points, etc.)<\/li>\nA rendered image<\/b> (saved locally)<\/li>\n<\/ul>\nprint(f\"Parsing slide deck...\")\nmd_json_objs = parser.get_json_result(\"data\/conocophillips.pdf\")\nmd_json_list = md_json_objs[0][\"pages\"]<\/code><\/pre>\nprint(md_json_list[10][\"md\"])<\/code><\/pre>\nprint(md_json_list[1].keys())<\/code><\/pre>\nimage_dicts = parser.get_images(md_json_objs, download_path=\"data_images\")<\/code><\/pre>\nStep 6: Associate Text and Images<\/h3>\n We create a list of TextNode<\/i><\/b> objects (LlamaIndex\u2019s data structure) for each page. Each node has metadata about the page number and the corresponding image file path:<\/p>\nfrom llama_index.core.schema import TextNode\nfrom typing import Optional\n\n# get pages loaded through llamaparse\nimport re\n\n\ndef get_page_number(file_name):\n match = re.search(r\"-page-(\\d+)\\.jpg$\", str(file_name))\n if match:\n return int(match.group(1))\n return 0\n\n\ndef _get_sorted_image_files(image_dir):\n \"\"\"Get image files sorted by page.\"\"\"\n raw_files = [f for f in list(Path(image_dir).iterdir()) if f.is_file()]\n sorted_files = sorted(raw_files, key=get_page_number)\n return sorted_files\n \nfrom copy import deepcopy\nfrom pathlib import Path\n\n\n# attach image metadata to the text nodes\ndef get_text_nodes(json_dicts, image_dir=None):\n \"\"\"Split docs into nodes, by separator.\"\"\"\n nodes = []\n\n image_files = _get_sorted_image_files(image_dir) if image_dir is not None else None\n md_texts = [d[\"md\"] for d in json_dicts]\n\n for idx, md_text in enumerate(md_texts):\n chunk_metadata = {\"page_num\": idx + 1}\n if image_files is not None:\n image_file = image_files[idx]\n chunk_metadata[\"image_path\"] = str(image_file)\n chunk_metadata[\"parsed_text_markdown\"] = md_text\n node = TextNode(\n text=\"\",\n metadata=chunk_metadata,\n )\n nodes.append(node)\n\n return nodes\n \n# this will split into pages\ntext_nodes = get_text_nodes(md_json_list, image_dir=\"data_images\")\n\nprint(text_nodes[10].get_content(metadata_mode=\"all\"))<\/code><\/pre>\nStep 7: Build a Summary Index<\/h3>\n With these text nodes in hand, you can create a SummaryIndex:<\/p>\n
import os\nfrom llama_index.core import (\n StorageContext,\n SummaryIndex,\n load_index_from_storage,\n)\n\nif not os.path.exists(\"storage_nodes_summary\"):\n index = SummaryIndex(text_nodes)\n # save index to disk\n index.set_index_id(\"summary_index\")\n index.storage_context.persist(\".\/storage_nodes_summary\")\nelse:\n # rebuild storage context\n storage_context = StorageContext.from_defaults(persist_dir=\"storage_nodes_summary\")\n # load index\n index = load_index_from_storage(storage_context, index_id=\"summary_index\")<\/code><\/pre>\nThe SummaryIndex ensures you can easily retrieve or generate high-level summaries over the entire document.<\/p>\n
Step 8: Define a Structured Output Schema<\/h3>\n Our pipeline aims to produce a final output with interleaved text blocks and image blocks. For that, we create a custom Pydantic model (using Pydantic v2 or ensuring compatibility) with two block types\u2014TextBlock<\/i><\/b> and ImageBlock<\/i><\/b>\u2014and a parent model ReportOutput<\/i><\/b>:\u00a0\u00a0<\/p>\nfrom llama_index.llms.openai import OpenAI\nfrom pydantic import BaseModel, Field\nfrom typing import List\nfrom IPython.display import display, Markdown, Image\nfrom typing import Union\n\n\nclass TextBlock(BaseModel):\n \"\"\"Text block.\"\"\"\n\n text: str = Field(..., description=\"The text for this block.\")\n\n\nclass ImageBlock(BaseModel):\n \"\"\"Image block.\"\"\"\n\n file_path: str = Field(..., description=\"File path to the image.\")\n\n\nclass ReportOutput(BaseModel):\n \"\"\"Data model for a report.\n\n Can contain a mix of text and image blocks. MUST contain at least one image block.\n\n \"\"\"\n\n blocks: List[Union[TextBlock, ImageBlock]] = Field(\n ..., description=\"A list of text and image blocks.\"\n )\n\n def render(self) -> None:\n \"\"\"Render as HTML on the page.\"\"\"\n for b in self.blocks:\n if isinstance(b, TextBlock):\n display(Markdown(b.text))\n else:\n display(Image(filename=b.file_path))\n\n\nsystem_prompt = \"\"\"\\\nYou are a report generation assistant tasked with producing a well-formatted context given parsed context.\n\nYou will be given context from one or more reports that take the form of parsed text.\n\nYou are responsible for producing a report with interleaving text and images - in the format of interleaving text and \"image\" blocks.\nSince you cannot directly produce an image, the image block takes in a file path - you should write in the file path of the image instead.\n\nHow do you know which image to generate? Each context chunk will contain metadata including an image render of the source chunk, given as a file path. \nInclude ONLY the images from the chunks that have heavy visual elements (you can get a hint of this if the parsed text contains a lot of tables).\nYou MUST include at least one image block in the output.\n\nYou MUST output your response as a tool call in order to adhere to the required output format. Do NOT give back normal text.\n\n\"\"\"\n\n\nllm = OpenAI(model=\"gpt-4o\", api_key=\"OpenAI_API_KEY\", system_prompt=system_prompt)\nsllm = llm.as_structured_llm(output_cls=ReportOutput)<\/code><\/pre>\nThe key point: ReportOutput<\/i><\/b> requires at least one image block, ensuring the final answer is multimodal.<\/p>\nStep 9: Create a Structured Query Engine<\/h3>\n
LlamaIndex allows you to use a \u201cstructured LLM\u201d (i.e., an LLM whose output is automatically parsed into a specific schema). Here\u2019s how:\u00a0\u00a0<\/p>\n
query_engine = index.as_query_engine(\n similarity_top_k=10,\n llm=sllm,\n # response_mode=\"tree_summarize\"\n response_mode=\"compact\",\n)\n\nresponse = query_engine.query(\n \"Give me a summary of the financial performance of the Alaska\/International segment vs. the lower 48 segment\"\n)\n\nresponse.response.render()<\/code><\/pre>\n# Output\nThe financial performance of ConocoPhillips' Alaska\/International segment and the Lower 48 segment can be compared based on several key metrics such as capital expenditure, production, and free cash flow over the next decade.\n\nAlaska\/International Segment\nCapital Expenditure: The Alaska\/International segment is projected to have capital expenditures of $3.7 billion in 2023, averaging $4.4 billion from 2024 to 2028, and $3.0 billion from 2029 to 2032.\nProduction: Production is expected to be around 750 MBOED in 2023, increasing to an average of 870 MBOED from 2024 to 2028, and reaching 1080 MBOED from 2029 to 2032.\nFree Cash Flow (FCF): The segment is anticipated to generate $5.5 billion in FCF in 2023, with an average of $6.5 billion from 2024 to 2028, and $15.0 billion from 2029 to 2032.\nLower 48 Segment\nCapital Expenditure: The Lower 48 segment is expected to have capital expenditures of $6.3 billion in 2023, averaging $6.5 billion from 2024 to 2028, and $8.0 billion from 2029 to 2032.\nProduction: Production is projected to be approximately 1050 MBOED in 2023, increasing to an average of 1200 MBOED from 2024 to 2028, and reaching 1500 MBOED from 2029 to 2032.\nFree Cash Flow (FCF): The segment is expected to generate $7 billion in FCF in 2023, with an average of $8.5 billion from 2024 to 2028, and $13 billion from 2029 to 2032.\nOverall, the Lower 48 segment shows higher capital expenditure and production levels compared to the Alaska\/International segment, but both segments are projected to generate significant free cash flow over the next decade.<\/code><\/pre>\n# Trying another query\nresponse = query_engine.query(\n \"Give me a summary of whether you think the financial projections are stable, and if not, what are the potential risk factors. \"\n \"Support your research with sources.\"\n)\n\nresponse.response.render()<\/code><\/pre>\nConclusion<\/h2>\n By combining LlamaIndex, LlamaParse, and OpenAI, you can build a multimodal report generator that processes an entire PDF (with text, tables, and images) into a structured output. This approach delivers richer, more visually informative results\u2014exactly what stakeholders need to glean critical insights from complex corporate or technical documents.<\/p>\n
Feel free to adapt this pipeline to your own documents, add a retrieval step for large archives, or integrate domain-specific models for analyzing the underlying images. With the foundations laid out here, you can create dynamic, interactive, and visually rich reports that go far beyond simple text-based queries.<\/p>\n
A big thanks to Jerry Liu from LlamaIndex for developing this amazing pipeline.<\/p>\n
![\"Parse](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/01\/Parse-the-Document-with-LlamaParse.webp\")
<\/figure>\n![\"parsing\"](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/01\/image_gAQhz2v.webp\")
<\/figure>\n![\"dict](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/01\/dict-output.webp\")
<\/figure>\n![\"image](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/01\/image-dict.webp\")
<\/figure>\n![\"Associate](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/01\/Associate-Text-and-Images.webp\")
<\/figure>\n![\"Multimodal](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/01\/Multimodal-Output-Retrieval.webp\")
<\/figure>\n![\"Part](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/01\/Part-of-the-Output-Response.webp\")
<\/figure>\n![\"output:](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/01\/image_V8L3rvO.webp\")
<\/figure>\n![\"Retrieved](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/01\/Retrieved-Image-Image.webp\")
<\/figure>\n![\"Adarsh](\"https:\/\/av-eks-lekhak.s3.amazonaws.com\/media\/lekhak-profile-images\/converted_image_tHXFGNS.webp\")
<\/p>\n