In today\u2019s fast-paced business world, leveraging cutting-edge technology like Generative AI <\/a>can significantly elevate operations of a business. Vision-language models, such as PaliGemma 2 Mix, offer businesses a powerful way to bridge the gap between visual and textual data. By combining the advanced SigLIP vision model and Gemma 2 language models<\/a>, PaliGemma 2 Mix excels at tasks like image captioning, visual question answering, OCR, object detection, and segmentation, all with exceptional accuracy. What sets PaliGemma 2 Mix apart is its plug-and-play capability. Unlike previous models that required extensive fine-tuning, this tool is ready for immediate application across various tasks. Available in multiple configurations (3B, 10B, and 28B parameters) and resolutions (224\u00d7224 and 448\u00d7448), it gives us the flexibility to align computational power with specific business needs.\u00a0\u00a0<\/p>\n This article was published as a part of the\u00a0<\/strong><\/em>Data Science Blogathon.<\/strong><\/em><\/a><\/p>\n PaliGemma 2, released by Google in December 2024, was an iteration of the\u00a0PaliGemma vision language model<\/a>. PaliGemma 2 connects the powerful SigLIP image encoder with the\u00a0Gemma 2<\/a>\u00a0language model.<\/p>\n Let us understand the key components of PaliGemma 2:<\/p>\n SigLIP is a vision encoder that processes visual data, such as images or videos, by breaking them down into analyzable features. It extracts visual tokens from images and uses them for tasks like image classification, object detection, and OCR. SigLIP has evolved into SigLIP 2, which offers improved performance and new variants for dynamic resolution.<\/p>\n PaliGemma 2 is a vision-language model (VLM) that integrates the SigLIP vision encoder with the Gemma 2 language model. It combines visual and textual data to perform tasks such as image captioning, visual question answering, and OCR, leveraging both the SigLIP encoder for visual analysis and the Gemma 2 model for text understanding.<\/p>\n PaliGemma 2 has been trained on a wide range of datasets to support its diverse capabilities. These include WebLI, a multilingual image-text dataset for tasks like visual semantics and object localization; CC3M-35L, which features image-alt-text pairs in multiple languages; and VQ\u00b2A-CC3M-35L, a subset with question-answer pairs related to images. Additionally, it uses OpenImages for detection tasks and object-aware Q&A pairs, and WIT, a dataset derived from Wikipedia with images and corresponding text. Together, these datasets equip PaliGemma 2 for tasks such as image understanding and multilingual text interpretation.<\/p>\n Let us now explore PaliGemma 2 Mix below:<\/p>\n While both models, PaliGemma 2 and PaliGemma 2 Mix, share a similar architecture, PaliGemma 2 Mix optimizes performance for immediate use across multiple tasks without requiring fine-tuning. This makes it more convenient for developers to quickly integrate vision-language capabilities into their applications.<\/p>\n PaliGemma 2 Mix is available in several variants, each differing in model size and input resolution. These variations allow users to choose the best model for their specific needs based on computational resources and task complexity.<\/p>\n Model Sizes:<\/b><\/p>\n Resolutions:<\/strong><\/p>\n The PaliGemma 2 mix models are capable of handling a wide range of tasks. These tasks can be grouped into the following categories based on their subtasks:<\/p>\n In the following tutorial, we will create a query system to extract information from medical prescriptions using PaliGemma 2 Mix model. We will see how it performs on extracting information from different scanned doctor\u2019s prescriptions. We can run the following code on Google colab with T4 GPU (free tier). The whole code is given in this Colab Notebook.\u00a0<\/a><\/p>\n Let us install necessary libraries first.<\/p>\n The code installs or updates two Python libraries, bitsandbytes and transformers. Bitsandbytes is a library that optimizes memory usage for machine learning models, especially for quantization tasks. The transformers model will be used for fetching the models\u00a0from Hugging Face.<\/p>\n Next step is to import all required libraries;<\/p>\n We import all the libraries needed to run the next blocks of code here.<\/p>\n Since this model is in a gated repo on Hugging Face, we need to create a fine grained access token on Hugging Face and setting the \u201cRead access to contents of all public gated repos you can access\u201d.<\/p>\n We can define this API token in the above code before running the next steps.<\/p>\n \u00a0We load the model google\/paligemma2-10b-mix-448<\/a> here which was fine-tuned on a mixture of academic tasks using 448\u00d7448 input images.\u00a0\u00a0<\/p>\n We fetch a sample scanned document from a URL, convert it to RGB format if needed, and display it for processing.<\/p>\n We load this scanned document, which is a sample masked Aadhar document. Then we will try to extract the name from this document using the model.<\/p>\n We create a text prompt, process the input image and text, and generate a response using the model to extract prescription details.<\/p>\n The above code processes an input prompt and an image, then feeds them into the model to generate a response. It first processes the text prompt along with the image, inputs them into the model, and generates a response based on the given context. Finally, it decodes the output and prints it as a readable text answer.<\/p>\n Output<\/b><\/p>\n As we can see from the output above, the medicine name has been extracted correctly from the document<\/p>\n Query 2<\/b><\/p>\n Input Image<\/b><\/p>\n Output<\/b><\/p>\n The output above shows that the model correctly extracted two diseases, Diabetes and Hypertension, from the document. However, it failed to extract \u201ccholesterol\u201d accurately.<\/p>\n Query 3<\/b><\/p>\nLearning Objectives<\/h3>\n
\n
Understanding PaliGemma 2 and Its Architecture<\/h2>\n
Key Components of PaliGemma 2<\/h3>\n
\n
How is PaliGemma 2 Different from SigLIP?<\/h2>\n
Training Data For PaliGemma 2<\/h3>\n
PaliGemma 2 Mix and Its Key Differentiating Features\u00a0<\/h2>\n
![\"Finetuned](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/03\/Finetuned-Pali-Gemma-2.webp\")
<\/figure>\n\n
\n
Range of Tasks With PaliGemma 2 Mix<\/h2>\n
\n
Building a Medical Prescription Scanner using PaliGemma 2 Mix<\/h2>\n
Step1: Install Necessary Libraries<\/h3>\n
!pip install -U bitsandbytes -U transformers -q<\/code><\/pre>\nStep2: Import Necessary Libraries<\/h3>\n
import torch\nimport pandas as pd\nfrom transformers import PaliGemmaForConditionalGeneration, PaliGemmaProcessor, BitsAndBytesConfig\nfrom transformers import BitsAndBytesConfig\nfrom PIL import Image\nfrom transformers.image_utils import load_image\nimport requests\nfrom io import BytesIO<\/code><\/pre>\nStep3: Setting Hugging Face API Token<\/h3>\n
import os\nos.environ[\"HF_TOKEN\"]=\"\"<\/code><\/pre>\nStep5: Loading the Model<\/h3>\n
model_id = \"google\/paligemma2-10b-mix-448\" \nbnb_config = BitsAndBytesConfig(\n load_in_8bit=True, # Change to load_in_4bit=True for even lower memory usage\n llm_int8_threshold=6.0,\n)\n\n# Load model with quantization\nmodel = PaliGemmaForConditionalGeneration.from_pretrained(\n model_id, quantization_config=bnb_config\n).eval()\nprocessor = PaliGemmaProcessor.from_pretrained(model_id)\n\n#Set the following for avoiding an error on \" Dynamic control flow is not supported at the moment\"\n\nmodel.forward = torch.compile(model.forward, mode=\"reduce-overhead\", fullgraph=False)\n<\/code><\/pre>\nStep6: Loading the Image<\/h3>\n
# URL of the image\nurl = \"https:\/\/assets.isu.pub\/document-structure\/230725104448-236aeacced7d7abcdafb3f9f2caf21c3\/v1\/a61879b5c46195fd5526fe6fe4e15fc8.jpeg\"\n\n# Send a GET request to the URL\nresponse = requests.get(url)\n\n# Check if the request was successful\nif response.status_code == 200:\n # Load the image from the response content\n img = Image.open(BytesIO(response.content))\n img.show()\nelse:\n print(\"Failed to retrieve the image.\")\n\n\ndef ensure_rgb(image: Image.Image) -> Image.Image:\n if image.mode != \"RGB\":\n image = image.convert(\"RGB\")\n return image<\/code><\/pre>\n![\"inputimage;](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/03\/inputimage.webp\")
Step7: Querying from Scanned Document\u00a0<\/h3>\n
prompt = \"Answer en Which medicines are recommended in the prescription\"\nmodel_inputs = processor(text=prompt, images=ensure_rgb(img), return_tensors=\"pt\").to(torch.bfloat16).to(model.device)\ninput_len = model_inputs[\"input_ids\"].shape[-1]\n\nwith torch.inference_mode():\n generation = model.generate(**model_inputs, max_new_tokens=100, do_sample=False)\n generation = generation[0][input_len:]\n decoded = processor.decode(generation, skip_special_tokens=True)\n print(decoded)<\/code><\/pre>\n![\"output\"](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/03\/output-6.webp\")
<\/figure>\nTesting on other Queries<\/h3>\n
# URL of the image\nurl = \"https:\/\/ars.els-cdn.com\/content\/image\/1-s2.0-S2468502X21000334-gr6.jpg\"\n\n# Send a GET request to the URL\nresponse = requests.get(url)\n\n# Check if the request was successful\nif response.status_code == 200:\n # Load the image from the response content\n img = Image.open(BytesIO(response.content))\n img.show()\nelse:\n print(\"Failed to retrieve the image.\")\n \nprompt = \"Answer en Which diseases are mentioned in the prescription\"\nmodel_inputs = processor(text=prompt, images=ensure_rgb(img), return_tensors=\"pt\").to(torch.bfloat16).to(model.device)\ninput_len = model_inputs[\"input_ids\"].shape[-1]\n\nwith torch.inference_mode():\n generation = model.generate(**model_inputs, max_new_tokens=100, do_sample=False)\n generation = generation[0][input_len:]\n decoded = processor.decode(generation, skip_special_tokens=True)\n print(decoded)<\/code><\/pre>\n![\"query2:](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/03\/query2.webp\")
![\"output](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/03\/image_0wJwlEw.webp\")
<\/figure>\n# URL of the image\nurl = \"https:\/\/www.madeformedical.com\/wp-content\/uploads\/2018\/07\/vio-4.jpg\"\n\n# Send a GET request to the URL\nresponse = requests.get(url)\n\n# Check if the request was successful\nif response.status_code == 200:\n # Load the image from the response content\n img = Image.open(BytesIO(response.content))\n img.show()\nelse:\n print(\"Failed to retrieve the image.\")\n \nprompt = \"Answer en Which medicines are mentioned in the prescription\"\nmodel_inputs = processor(text=prompt, images=ensure_rgb(img), return_tensors=\"pt\").to(torch.bfloat16).to(model.device)\ninput_len = model_inputs[\"input_ids\"].shape[-1]\n\nwith torch.inference_mode():\n generation = model.generate(**model_inputs, max_new_tokens=100, do_sample=False)\n generation = generation[0][input_len:]\n decoded = processor.decode(generation, skip_special_tokens=True)\n print(decoded)<\/code><\/pre>\n
![\"query3;](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/03\/query3.webp\")
![\"output](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/03\/output-query3.webp\")
<\/figure>\n![\"query4;](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/03\/query4.webp\")
![\"output](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/03\/output-query4.webp\")
<\/figure>\n![\"Nibedita](\"https:\/\/av-eks-lekhak.s3.amazonaws.com\/media\/lekhak-profile-images\/converted_image_ZkJo4gb.webp\")
<\/p>\n