{"id":299618,"date":"2025-06-17T23:21:09","date_gmt":"2025-06-17T23:21:09","guid":{"rendered":"https:\/\/peraltafinancing.com\/analytics\/how-does-a-multimodal-llm-work-the-vision-story\/"},"modified":"2025-06-17T23:21:09","modified_gmt":"2025-06-17T23:21:09","slug":"how-does-a-multimodal-llm-work-the-vision-story","status":"publish","type":"post","link":"https:\/\/fivemor.com\/?p=299618","title":{"rendered":"How Does A Multimodal LLM Work? The Vision Story"},"content":{"rendered":"


\n<\/p>\n

\n

Multimodal Large Language Models (MLLMs) have lately become the talk of the AI universe. It is dynamically reshaping how AI systems understand and interact with our complex, multi-sensory world. These multi-sensory inputs that we get can also be coined as our different modalities (images, audio, etc.). From Google\u2019s latest Veo 3, generating state-of-the-art videos to ElevenLabs creating incredibly realistic AI voice overs, these systems are demonstrating capabilities that were once considered to be science fiction.<\/span><\/p>\n

This comprehensive guide is the first part of a two-part series exploring the intricate world of multimodal LLMs. The second part of this series will explore how these models generate multimodal content and their practical applications across various industries.<\/span><\/p>\n

Challenges of Multimodality<\/span><\/h2>\n

Multimodality is definitely one of the greatest capabilities and advancements in AI models. However, when we deal with several modalities, there will be certain challenges that need to be curbed.<\/span> Here are two major challenges we face in this regard:<\/p>\n

    \n
  1. How to represent our information?<\/b>
    One of the main challenges of multimodal LLMs is when it comes to representing different types of information. It is how to represent and summarize these multimodal data in a common space which we need to train our multimodal models.<\/span><\/li>\n
  2. How do we align our different modalities?<\/b>
    <\/b>We have to ensure we identify direct relations between similar elements from different modalities. This is done in two ways:<\/span>\n
      \n
    1. Explicit Alignment:<\/strong> Here, we directly find correspondences between elements of different modalities. For this, we have to train our model across various modalities like audio, text, image, etc. This <\/span>supervised or <\/span>rule-based alignment<\/span> is implemented using algorithms like Dynamic Time Warping (DTW), Attention with supervision, or alignment matrices.<\/span><\/li>\n
    2. Implicit Alignment:<\/strong> This uses internally latent alignment of modalities to better solve different problems. Allowing the model to figure it out itself. Models use techniques like <\/span>self-attention, <\/span>contrastive learning, or <\/span>co-attention mechanisms to learn which parts of one modality relate to another.<\/span><\/li>\n<\/ol>\n<\/li>\n<\/ol>\n
      \"Multimodal
      Source \u2013 <\/span>Medium<\/a><\/figcaption><\/figure>\n

      Let\u2019s understand this with a small example:<\/span><\/p>\n

      Since we need to represent the term \u201ccat\u201d whether it\u2019s in the form of text, image, or speech as closely as possible, we should make sure other terms like \u201ddog\u201d are far from the vicinity of the term \u201ccat\u201d. These embeddings from various modalities need to be correctly aligned across the shared dimensional space.<\/span><\/p>\n

      \"How
      Source \u2013 Media2.dev<\/a><\/figcaption><\/figure>\n

      Representation Learning<\/span><\/h2>\n

      The solution to our first problem on \u201chow to represent information\u201d can be solved by representation learning. There are 2 types of representations-based learning through which multimodal information could be understood by these multimodal models. These are: Joint Representation and Coordinated Representation<\/span>.<\/p>\n

      Joint Representation<\/span><\/h3>\n

      Joint representation could be defined as a single unified representation of different types of information which could be text, image, video, audio, etc. We combine the embeddings of each modality in a single embedding dimension space.<\/span><\/p>\n

      \"Joint
      Source \u2013 <\/span>Medium<\/span><\/a><\/figcaption><\/figure>\n

      Here, in this approach, we will pass each modality across its respective encoders. Basically, Text will be passed through a Text Encoder (e.g. BERT) and image across an Image Encoder (e.g. VIT) likewise for other modalities.<\/span><\/p>\n

      \"List
      Source \u2013 <\/span>Medium<\/a><\/figcaption><\/figure>\n

      We get the embeddings for each modality. Later, these embedding representations merge using a concatenation technique. Then, a projection layer or multimodal attention mechanism will assign certain importance to certain features. The resulting joint embedding will contain the complete semantics of all the input modalities.<\/span><\/p>\n

      This <\/span>entire system is trained. The individual modality encoders, the fusion mechanism, and the final task-specific layers are all optimized together using a <\/span>single loss function. This unified training setup allows the model to learn cross-modal correlations more effectively, especially when the modalities are strongly interdependent (e.g. image and its caption like in the COCO dataset).<\/span><\/p>\n

      These joint embeddings are particularly useful when the input modalities are closely aligned or when the available training data is limited, as shared representations help in regularizing the learning process and extracting richer, semantically meaningful features from the combined input.<\/span><\/p>\n

      Read more about the Evolution of Embeddings.<\/a><\/span><\/p>\n

      Coordinated Representation<\/span><\/h3>\n

      Coordinated Representation learning on the other side has a completely different approach. Here, we learn independent representations alone and then coordinate (or align) them together in the fusion stage. In this approach, each modality (text, image, audio, etc.) is handled by its dedicated model, which is trained separately and may also have its loss function and objective.<\/span><\/p>\n

      \"coordinated
      Source \u2013 <\/span>Medium<\/span><\/a><\/figcaption><\/figure>\n

      Once these models are trained, their individual output embeddings are combined using a <\/span>coordinated fusion mechanism like late fusion (simple concatenation), cross-modal attention, or statistical alignment methods such as Canonical Correlation Analysis (CCA). The coordination phase focuses on ensuring that the separate embeddings are semantically aligned with each other so that they can jointly contribute to the final prediction. Unlike joint embeddings, coordinated embeddings allow each modality to <\/span>preserve its own feature structure without being forced into a shared representation space prematurely.<\/span><\/p>\n

      This method is highly effective when modalities are somewhat <\/span>independent or loosely coupled, when there is <\/span>abundant modality-specific data, or when computational resources allow for more extensive pre-training. Coordinated embeddings also offer greater flexibility in model architecture and training pipelines, as each modality can be improved independently before coordination.<\/span><\/p>\n

      Explicit vs Implicit Alignment<\/h4>\n

      Let\u2019s try to tabulate our understanding here:<\/span><\/p>\n

      \n\n\n\n\n\n\n\n\n\n\n
      Feature<\/b><\/td>\nExplicit Alignment<\/b><\/td>\nImplicit Alignment<\/b><\/td>\n<\/tr>\n
      Nature<\/b><\/td>\nSupervised \/ Annotated<\/span><\/td>\nUnsupervised \/ Learned during training<\/span><\/td>\n<\/tr>\n
      Need for Labels<\/b><\/td>\nRequires aligned or annotated data<\/span><\/td>\nDoes not require explicit alignments<\/span><\/td>\n<\/tr>\n
      Approach<\/b><\/td>\nManual or rule-based mapping<\/span><\/td>\nLearned via attention or contrastive loss<\/span><\/td>\n<\/tr>\n
      Example Tasks<\/b><\/td>\nImage captioning with bounding boxes<\/span><\/td>\nCLIP, VQA with unsupervised attention<\/span><\/td>\n<\/tr>\n
      Advantages<\/b><\/td>\nHigh precision, interpretable<\/span><\/td>\nScalable, flexible, learn fine-grained links<\/span><\/td>\n<\/tr>\n
      Challenges<\/b><\/td>\nExpensive to label, less flexible<\/span><\/td>\nCan be less interpretable, data-hungry<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

      We will now try to understand another important term that we used in the above section named \u201cfusion\u201d next.<\/span><\/p>\n

      If you want to understand how implicit alignment can be done, read <\/span>this<\/span><\/a>. In this research paper, the model embeds fragments of images (objects in the image) and fragments of sentences (typed dependency tree relations) into a common space.<\/span><\/p>\n

      \"how<\/figure>\n

      Let\u2019s dive a little deeper into this.<\/p>\n

      The Concept of Fusion in Multimodal LLMs<\/h2>\n

      The cornerstone of multimodal learning lies in understanding how different types of data can be combined effectively. In other words, it serves as a way to accurately align our different modalities across a unified dimensional space. Fusion strategies determine when and how information from different modalities is integrated, fundamentally shaping the model\u2019s ability to understand complex multimodal inputs<\/span>.<\/p>\n

      Fusion refers to the <\/span>integration of information from multiple modalities such as text, image, and audio into a <\/span>unified representation. It plays a critical role in enabling models to <\/span>leverage complementary information from each modality. The goal is to combine features in such a way that the model can make more informed predictions.<\/span> It\u2019s pretty similar to the concept of fusion that we use in Deep Learning.<\/span><\/p>\n

      There are two widely used strategies for fusion: <\/span>Early Fusion<\/b> and <\/span>Late Fusion<\/b>.<\/span><\/p>\n

      \"Early
      Source \u2013 <\/span>Medium<\/a><\/figcaption><\/figure>\n

      There also exists a third category \u2013 mid-fusion, about which I will explain in a while.<\/p>\n

      1. Early Fusion<\/span><\/h3>\n

      Early Fusion represents the simplest approach to multimodal integration, here the raw data from different modalities is combined at the input level itself before any processing occurs. In early fusion systems, data from various sources such as pixel values from images and tokenized text are concatenated or combined through simple operations at the very beginning of the processing pipeline. This approach allows for comprehensive interaction between modalities from the earliest stages of computation, enabling the model to capture subtle correlations and dependencies that might be lost in later-stage fusion approaches.\u00a0<\/span><\/p>\n