{"id":50167,"date":"2025-01-27T05:04:03","date_gmt":"2025-01-27T05:04:03","guid":{"rendered":"https:\/\/peraltafinancing.com\/analytics\/a-journey-into-multimodal-llms-part-1\/"},"modified":"2025-01-27T05:04:03","modified_gmt":"2025-01-27T05:04:03","slug":"a-journey-into-multimodal-llms-part-1","status":"publish","type":"post","link":"https:\/\/fivemor.com\/?p=50167","title":{"rendered":"A Journey into Multimodal LLMs Part 1"},"content":{"rendered":"


\n<\/p>\n

\n

The human mind naturally perceives language, vision, smell, and touch, enabling us to understand our surroundings. We are particularly inclined toward linguistic thought and visual memory. As GenAI models continue to grow, researchers are now working on extending their capabilities by incorporating multimodality. Large Language models (LLMs)<\/a> only accept text as input and produce text as output, which means these models do not process or generate data from other modalities such as images, videos, or voice. LLMs have excelled in handling tasks such as question-answering, text summarization, translation, information retrieval, code generation, and reasoning. However, integrating other modalities with LLMs (Multimodal LLMs) enhances the potential of GenAI models. For instance, training a model by combining text and images solves problems such as visual Q&A, image segmentation, and object detection. Likewise, we can add videos in the same model for more advanced media-related analysis.<\/p>\n

Introduction to Multimodal LLMs<\/h2>\n

Generative AI<\/a> is a subsection of machine learning models allowing for new content generation. We can generate new text after feeding input as text to the model known as text-to-text. However, after extending the capabilities of LLMs with other modalities, we can open the solution to a wide range of use cases such as text-to-image, text-to-video, text-to-speech, image-to-image, and image-to-video. We call such models Large multimodal models (Multimodal LLMs). Training such models happens on large datasets containing text and other modalities so that algorithms can learn the relationships among all the input types. Intuitively, these models are not limited to a single input or output type; they can be adapted to handle inputs from any modality and generate output accordingly. In this way, multimodal LLMs can be seen as providing the system with the ability to process and understand different types of sensory inputs.<\/p>\n

This blog is split into two sections; in the first part, I will explore the applications of multimodal LLMs and various architectures, while in the second part, I will train a small vision model.<\/p>\n

Datasets<\/h2>\n

While combining different input types to create multimodal LLMs may appear straightforward, it becomes more complex when processing data from 1D, 2D, and 3D together. It is a multi-step problem that needs to be solved sequentially in a step-by-step manner, and the data must be carefully curated to enhance the problem-solving capabilities of such models.<\/p>\n

For now, we will limit our discussion to text and images. Unlike text, images and videos come in varying sizes and resolutions, so a robust pre-processing technique is needed to standardize all inputs into a single framework. Furthermore, inputs like images, videos, prompts, and metadata should be prepared in a way that helps models build coherent thought processes and maintain logical consistency during inference. Models trained with text, image, and video data are called Large Vision-Language Models (LVLMs).<\/p>\n

Application of Multimodal LLMs<\/h2>\n

The following image is taken from a Qwen2-VL paper<\/a> where researchers trained a vision model based on Qwen2 LLM<\/a> that can solve multiple visual use cases.<\/p>\n

\n
\"Qwen2-VL\"
Source: Qwen2-VL<\/figcaption><\/figure>\n<\/div>\n

The figure below demonstrates how a Multimodal Language Model (MMLM)<\/a> processes different types of input data (image, text, audio, video) to achieve various objectives. The core part of the diagram, the MMLM, integrates all the different modalities (image, text, audio, video) to process them in combination.<\/p>\n

\n
\"A
A generic understanding of the Input and output flow of MMLMs.<\/figcaption><\/figure>\n<\/div>\n

Let\u2019s proceed further and understand the different applications of vision models. The complete code used in this blog is stored in GitHub<\/a>.<\/p>\n

1. Image captioning<\/h3>\n

It is the task of describing the features of images in words. People are using this feature to generate descriptions of images and innovating a range of engaging captions and relevant hashtags for their social media posts to improve visibility.<\/p>\n

image_path = \"quantum.jpg\"\nwith open(image_path, 'rb') as image_file:\n    image_data = image_file.read()\n    \nimage_data = base64.b64encode(image_data).decode(\"utf-8\")\n\nprompt=\"\"\"explain this image\"\"\"\nmessage = HumanMessage(\n    content=[\n        {\"type\": \"text\", \"text\": prompt},\n        {\n            \"type\": \"image_url\",\n            \"image_url\": {\"url\": f\"data:image\/jpeg;base64,{image_data}\"},\n        },\n    ],\n)\nresponse = llm.invoke([message])\nprint(response.content)<\/code><\/pre>\n
Information extraction is another application for vision models where we expect the model to retrieve features or data points from the images. For example, we can question the model to identify underlying objects\u2019 colour, text, or feature. Contemporary models use function calling or JSON parsing techniques to extract structured data points from the images.<\/p>\n
from langchain.output_parsers import PydanticOutputParser\nfrom langchain_core.prompts import ChatPromptTemplate\nfrom pydantic import BaseModel, Field\nimport json\n\nclass Retrieval(BaseModel):\n    Description: str = Field(description=\"Describe the image\")\n    Machine: str = Field(description=\"Explain what is the machine about\")\n    Color: str = Field(description=\"What are the color used in the image\")\n    People: str = Field(description=\"Count how many male and female are standing their\")\n\nparser = PydanticOutputParser(pydantic_object=Retrieval)\n\nprompt = ChatPromptTemplate.from_messages([\n    (\"system\", \"Extract the requested details as per the given details.\\n'{struct_format}'\\n\"),\n    (\"human\", [\n        {\n            \"type\": \"image_url\",\n            \"image_url\": {\"url\": \"data:image\/jpeg;base64,{image_data}\"},\n        },\n    ]),\n])\n\nchain = prompt | llm | parser\n\nimage_path = \"quantum.jpg\"\nwith open(image_path, 'rb') as image_file:\n    image_data = image_file.read()\n    \nimage_data = base64.b64encode(image_data).decode(\"utf-8\")\n\n\nresponse = chain.invoke({\n    \"struct_format\": parser.get_format_instructions(),\n    \"image_data\": image_data\n})\n\ndata = json.loads(response.model_dump_json())\n\nfor k,v in data.items():\n    print(f\"{k}: {v}\")<\/code><\/pre>\n
3. Visual Interpretation & Reasoning<\/h3>\n
It is a use case for a vision model to analyze the image and perform reasoning tasks. For example, the model can interpret the underlying information in images, diagrams, and graphical representations, create step-by-step analyses, and conclude.<\/p>\n
4. OCR\u2019ing<\/h3>\n
It is one of the most important use cases in the area of Document AI where models convert and extract text data from images for downstream tasks.<\/p>\n
image_path = \"qubits.png\"\nwith open(image_path, 'rb') as image_file:\n    image_data = image_file.read()\n    \nimage_data = base64.b64encode(image_data).decode(\"utf-8\")\n\nprompt=\"\"\"Extract all the text from the image\"\"\"\nmessage = HumanMessage(\n    content=[\n        {\"type\": \"text\", \"text\": prompt},\n        {\n            \"type\": \"image_url\",\n            \"image_url\": {\"url\": f\"data:image\/jpeg;base64,{image_data}\"},\n        },\n    ],\n)\nresponse = llm.invoke([message])\nprint(response.content)<\/code><\/pre>\n
5. Object Detection & Segmentation<\/h3>\n
Vision models are capable of identifying objects in the images and classifying them into defined categories. Mainly in the case of object detection models can locate the objects and classify them whereas in the case of segmentation, vision models can divide the images into different regions based on surrounding pixel values.<\/p>\n
from langchain.output_parsers import PydanticOutputParser\nfrom langchain_core.prompts import ChatPromptTemplate\nfrom pydantic import BaseModel, Field\nfrom typing import List\n\nimport json\n\nclass Segmentation(BaseModel):\n    Object: List[str] = Field(description=\"Identify the object and give a name\")\n    Bounding_box: List[List[int]] = Field(description=\"Extract the bounding boxes\")\n\nparser = PydanticOutputParser(pydantic_object=Segmentation)\n\nprompt = ChatPromptTemplate.from_messages([\n    (\"system\", \"Extract all the image objects and their bounding boxes. You must always return valid JSON.\\n'{struct_format}'\\n\"),\n    (\"human\", [\n        {\n            \"type\": \"image_url\",\n            \"image_url\": {\"url\": \"data:image\/jpeg;base64,{image_data}\"},\n        },\n    ]),\n])\n\nchain = prompt | llm | parser\n\nimage_path = \"quantum.jpg\"\nwith open(image_path, 'rb') as image_file:\n    image_data = image_file.read()\n    \nimage_data = base64.b64encode(image_data).decode(\"utf-8\")\n\nresponse = chain.invoke({\n    \"struct_format\": parser.get_format_instructions(),\n    \"image_data\": image_data\n})\n\ndata = json.loads(response.model_dump_json())\n\nfor k,v in data.items():\n    print(f\"{k}: {v}\")\n\n## Complete code is available in GitHub\nplot_bounding_boxes(im=img,labels=data['Object'], bounding_boxes=data['Bounding_box'])<\/code><\/pre>\n
Vision models have a wide range of use cases across various industries and are increasingly being integrated into different platforms like Canva, Fireflies, Instagram, and YouTube.<\/p>\n
Architecture of Large Vision-Language Models (LVLMs)<\/h2>\n
The primary purpose of developing vision models is to unify features from images, videos, and text. Researchers are exploring different architectures to pretrain Large Vision-Language Models (LVLMs).
Typically, encoders are employed to extract image features, while text data can be processed using an encoder, a decoder, or a combination of both. Modality projectors, sometimes called connectors, are dense neural networks used to align image features with text representations.<\/p>\n
Below is the general overview of common network designs.<\/p>\n
1. Two-Tower VLM<\/h3>\n
The figure below represents the simplest architecture where images and text are encoded separately and trained under a common objective. Here\u2019s a breakdown of the components:<\/p>\n
\n
\"Two-Tower
Two-Tower VLM<\/figcaption><\/figure>\n<\/div>\n