Multimodal Retrieval-Augmented Generation (RAG) is a transformative innovation in AI, enabling systems to process and integrate diverse data types such as text, images, audio, and video. This capability is crucial in addressing the challenge of unstructured enterprise data, which predominantly consists of multimodal formats. By leveraging multimodal inputs, RAG enhances contextual understanding, improves accuracy, and expands AI\u2019s applicability across industries like healthcare, customer support, and education.\u00a0Docling is an open-source toolkit developed by IBM to streamline document processing for generative AI applications. We will build Multimodal RAG Capabilities Using Docling.<\/p>\n
It converts diverse formats like PDFs, DOCX, and images into structured outputs such as JSON and Markdown, enabling seamless integration with AI frameworks like LangChain and LlamaIndex. By facilitating the extraction of unstructured data and supporting advanced layout analysis, Docling empowers multimodal Retrieval-Augmented Generation (RAG)<\/a> by making complex enterprise data machine-readable and accessible for AI-driven insights<\/p>\n This article was published as a part of the\u00a0<\/strong><\/em>Data Science Blogathon.<\/strong><\/em><\/a><\/p>\n Docling is an open-source document processing toolkit developed by IBM<\/a>, designed to convert unstructured files like PDFs, DOCX, and images into structured formats such as JSON and Markdown. Powered by advanced AI models like DocLayNet for layout analysis and TableFormer for table recognition, it enables accurate extraction of text, tables, and images while preserving document structure. With seamless integration into generative AI frameworks like LangChain and LlamaIndex, Docling supports applications such as Retrieval-Augmented Generation (RAG) and question-answering systems. Its lightweight architecture allows efficient performance on standard hardware, making it a cost-effective alternative to SaaS-based solutions for enterprises seeking control over data privacy.<\/p>\n Docling implements a linear pipeline of operations, which execute sequentially on each given document (as shown in the above Figure). Each document is first parsed by a PDF backend, which retrieves the programmatic text tokens, consisting of string content and its coordinates on the page, and also renders a bitmap image of each page to support downstream operations. Then, the standard model pipeline applies a sequence of AI models independently on every page in the document to extract features and content, such as layout and table structures. Finally, the results from all pages are aggregated and passed through a post-processing stage, which augments metadata, detects the document language, infers reading order and eventually assembles a typed document object which can be serialized to JSON or Markdown.\u00a0<\/p>\n Traditionally, developers have depended on optical character recognition (OCR)<\/a> for converting documents into digital formats. However, this technology can be slow and prone to errors due to the heavy computational power required. Docling avoids OCR whenever possible, instead using computer vision models that are specifically trained to identify and categorize the visual components of a page.<\/p>\n Docling is based on two models developed by IBM researchers.<\/p>\n The layout analysis model functions as an object detector, predicting the bounding boxes and categories of various elements within an image of a given page. Its design is based on RT-DETR<\/a> and has been re-trained using DocLayNet, our well-known human-annotated dataset for document layout analysis, along with other proprietary datasets. DocLayNet<\/a> is a human-annotated document layout segmentation dataset containing 80863 pages from a broad variety of document sources.<\/p>\n This model utilizes object detection techniques to examine the layout of documents, ranging from machine manuals to annual reports. It then identifies and classifies elements such as blocks of text, images, tables, captions, and more. The Docling pipeline processes page images at a resolution of 72 dpi, enabling them to be handled by a single CPU.<\/p>\n The TableFormer model, initially introduced in 2022 and subsequently enhanced with a custom token structure language, is a vision-transformer model designed for recovering the structure of tables. It can predict the logical organization of rows and columns in a table based on an input image, identifying which cells belong to column headers, row headers, or the main body of the table. Unlike previous methods, TableFormer effectively handles various table complexities, including partial or absent borders, empty cells, missing rows or columns, cell spans, hierarchical structures in both column and row headings, as well as inconsistencies in indentation or alignment.<\/p>\n Here are the features:<\/p>\n Docling provides optional support for OCR, for example, to cover scanned PDFs or content in In this article, we will first extract all kinds of data \u2013 text, images, and tables from a PDF using Docling. For extracted images, we will use a vision language model to generate the description of the images and save these text descriptions of the images in our VectorDB along with the text data from the original text contents and text from extracted Tables in the PDF. Post this, we will build a RAG system using the vector DB for retrieval along with an LLM (Phi 4) through Ollama for querying from the PDF document.<\/p>\n You can find the Colab Notebook which has all the steps here<\/a>.<\/p>\n We first start with installing the necessary libraries<\/p>\n This code prepares a document converter to process PDF files without OCR but with image generation. It then applies this conversion to a specified PDF file, storing the results in a dictionary.<\/p>\n We use this PDF<\/a> (we save it in the current working directory as \u2018accenture.pdf\u2019) which has a lot of charts to test the multimodal retrieval using Docling.<\/p>\n The code below is for the document processing pipeline. It takes converted documents from the previous step and breaks them down into smaller chunks, excluding tables (which is processed separately later). Each chunk is then wrapped into a\u00a0Document\u00a0object with specific metadata. The\u00a0code processes converted documents by splitting them into chunks, skipping tables, and creating new\u00a0Document\u00a0objects with metadata for each chunk.<\/p>\n The code below is designed to process tables from converted documents. It extracts tables, converts them into Markdown format, and wraps each table into a\u00a0Document\u00a0object with specific metadata.<\/p>\n We will use a vision language model from Ollama to analyse the extracted images from the PDF and generate a description for each of the images. To facilitate the use of Ollama models, we install the following libraries and start up the Ollama server before pulling the model as described below in the code.<\/p>\n The code below is designed to process images from converted documents. It extracts images, uses\u00a0a vision model (llama3.2-vision through Ollama) to generate descriptive text for each image, and wraps this text into\u00a0a\u00a0Document\u00a0object with\u00a0specific metadata. Here\u2019s a detailed explanation:<\/p>\n Pulling the \u201cllama3.2-vision\u201d model from Ollama.<\/p>\nLearning Objectives<\/h3>\n
\n
Docling For Unstructured Data<\/h2>\n
Docling Pipeline<\/h2>\n
![\"docling](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/03\/image_gPlLqbz.webp\")
Key AI Models Behind Docling<\/h2>\n
Layout Analysis Model<\/h3>\n
Table Former Model<\/h3>\n
Some Key Features of Docling<\/h3>\n
\n
bitmap images embedded on a page. Docling relies on EasyOCR, a popular third-party OCR library with support for many languages. These features make Docling a comprehensive solution for document parsing and preparation in generative AI workflows.<\/p>\nBuilding a Multimodal RAG System using Docling<\/h2>\n
Hands-On Python Implementation on Google Colab using T4 GPU (Free Tier)<\/h2>\n
Step 1. Installing Libraries<\/h3>\n
!pip install docling\n\n#Following code added to avoid an error in installation - can be removed if not needed\nimport locale\ndef getpreferredencoding(do_setlocale = True):\n return \"UTF-8\"\nlocale.getpreferredencoding = getpreferredencoding\n\n!pip install langchain-huggingface<\/code><\/pre>\nStep 2. Loading the Converter Object<\/h3>\n
from docling.document_converter import DocumentConverter, PdfFormatOption\nfrom docling.datamodel.base_models import InputFormat\nfrom docling.datamodel.pipeline_options import PdfPipelineOptions\n\npdf_pipeline_options = PdfPipelineOptions(do_ocr=False,generate_picture_images=True,)\nformat_options = {InputFormat.PDF: PdfFormatOption(pipeline_options=pdf_pipeline_options)}\nconverter = DocumentConverter(format_options=format_options)\n\nsources = [ \"\/content\/accenture.pdf\",]\nconversions = {source: converter.convert(source=source).document for source in sources}<\/code><\/pre>\nStep 3. Loading the Model For Embedding Text<\/h3>\n
from langchain_huggingface.embeddings import HuggingFaceEmbeddings\nfrom transformers import *\n\nembeddings_model_path = \"ibm-granite\/granite-embedding-30m-english\"\nembeddings_model = HuggingFaceEmbeddings(model_name=embeddings_model_path,)\nembeddings_tokenizer = AutoTokenizer.from_pretrained(embeddings_model_path)<\/code><\/pre>\nStep 4. Chunking the Texts in the Document<\/h3>\n
from docling_core.transforms.chunker.hybrid_chunker import HybridChunker\nfrom docling_core.types.doc.document import TableItem\nfrom langchain_core.documents import Document\n\n\ndoc_id = 0\ntexts: list[Document] = []\n\nfor source, docling_document in conversions.items():\n for chunk in HybridChunker(tokenizer=embeddings_tokenizer).chunk(docling_document):\n items = chunk.meta.doc_items\n if len(items) == 1 and isinstance(items[0], TableItem):\n continue # we will process tables later\n refs = \" \".join(map(lambda item: item.get_ref().cref, items))\n text = chunk.text\n document = Document(page_content=text,metadata={\"doc_id\": (doc_id:=doc_id+1),\"source\": source,\"ref\": refs,},)\n texts.append(document)\n\n\n\nprint(f\"{len(texts)} text document chunks created\")<\/code><\/pre>\nStep 5. Processing the Tables in the Document<\/h3>\n
from docling_core.types.doc.labels import DocItemLabel\n\ndoc_id = len(texts)\ntables: list[Document] = []\n\nfor source, docling_document in conversions.items():\n for table in docling_document.tables:\n if table.label in [DocItemLabel.TABLE]:\n ref = table.get_ref().cref\n text = table.export_to_markdown()\n document = Document(\n page_content=text,\n metadata={\n \"doc_id\": (doc_id:=doc_id+1),\n \"source\": source,\n \"ref\": ref\n },\n )\n tables.append(document)\n\n\nprint(f\"{len(tables)} table documents created\")<\/code><\/pre>\nStep 6. Defining Function For Converting Images From PDF to base64 form<\/h3>\n
import base64\nimport io\nimport PIL.Image\nimport PIL.ImageOps\nfrom IPython.display import display\n\ndef encode_image(image: PIL.Image.Image, format: str = \"png\") -> str:\n image = PIL.ImageOps.exif_transpose(image) or image\n image = image.convert(\"RGB\")\n buffer = io.BytesIO()\n image.save(buffer, format)\n encoding = base64.b64encode(buffer.getvalue()).decode(\"utf-8\")\n return encoding<\/code><\/pre>\nStep 7. Pulling Model From Ollama For Analysing Images from the PDF<\/h3>\n
!sudo apt update\n!sudo apt install -y pciutils\n!pip install langchain-ollama\n!curl -fsSL https:\/\/ollama.com\/install.sh | sh\n!pip install ollama==0.4.2\n!pip install langchain-community\n\n\n#Enabling threading to start ollama server in a non blocking manner\nimport threading\nimport subprocess\nimport time\n\ndef run_ollama_serve():\n subprocess.Popen([\"ollama\", \"serve\"])\n\nthread = threading.Thread(target=run_ollama_serve)\nthread.start()\ntime.sleep(5)<\/code><\/pre>\n!ollama pull llama3.2-vision<\/code><\/pre>\ndef encode_image(image: PIL.Image.Image, format: str = \"png\") -> str:\n\n image = PIL.ImageOps.exif_transpose(image) or image\n image = image.convert(\"RGB\")\n buffer = io.BytesIO()\n image.save(buffer, format)\n encoding = base64.b64encode(buffer.getvalue()).decode(\"utf-8\")\n return encoding<\/code><\/pre>\nimport ollama\n\npictures: list[Document] = []\ndoc_id = len(texts) + len(tables)\n\nfor source, docling_document in conversions.items():\n for picture in docling_document.pictures:\n ref = picture.get_ref().cref\n image = picture.get_image(docling_document)\n if image:\n print(image)\n response = ollama.chat(\n model=\"llama3.2-vision\",\n messages=[{\n \"role\": \"user\",\n \"content\": \"Describe this image?\",\n \"images\": [encode_image(image)]\n }],\n )\n text = response['message']['content'].strip()\n document = Document(\n page_content=text,\n metadata={\n\n \"doc_id\": (doc_id:=doc_id+1),\n\n \"source\": source,\n\n \"ref\": ref,\n\n },\n\n )\n\n pictures.append(document)\n\nprint(f\"{len(pictures)} image descriptions created\")<\/code><\/pre>\n![\"Output\"](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/03\/image-67.webp\")
<\/figure>\nimport itertools\nfrom docling_core.types.doc.document import RefItem\n\n# Print all created documents\nfor document in itertools.chain(texts, tables):\n print(f\"Document ID: {document.metadata['doc_id']}\")\n print(f\"Source: {document.metadata['source']}\")\n print(f\"Content:\\n{document.page_content}\")\n print(\"=\" * 80) # Separator for clarity\n\nfor document in pictures:\n print(f\"Document ID: {document.metadata['doc_id']}\")\n source = document.metadata['source']\n print(f\"Source: {source}\")\n print(f\"Content:\\n{document.page_content}\")\n docling_document = conversions[source]\n ref = document.metadata['ref']\n picture = RefItem(cref=ref).resolve(docling_document)\n image = picture.get_image(docling_document)\n print(\"Image:\")\n display(image)\n print(\"=\" * 80) # Separator for clarity<\/code><\/pre>\n![\"Output\"](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/03\/image-66.webp\")
<\/figure>\nStep 9. Storing in Milvus Vector DB<\/h3>\n
![\"The](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/03\/image_MAHzhCB.webp\")
![\"](\"https:\/\/av-eks-lekhak.s3.amazonaws.com\/media\/__sized__\/article_images\/image_PzkfwwJ-thumbnail_webp-600x300.webp\")
![\"](\"https:\/\/av-eks-lekhak.s3.amazonaws.com\/media\/__sized__\/article_images\/image_A4d4cij-thumbnail_webp-600x300.webp\")
![\"Nibedita](\"https:\/\/av-eks-lekhak.s3.amazonaws.com\/media\/lekhak-profile-images\/converted_image_ZkJo4gb.webp\")
<\/p>\n