DeepSeek Janus Pro 1B, launched on January 27, 2025, is an advanced multimodal AI model built to process and generate images from textual prompts. With its ability to comprehend and create images based on text, this 1 billion parameter version (1B) delivers efficient performance for a wide range of applications, including text-to-image generation and image understanding. Additionally, it excels at producing detailed captions from photos, making it a versatile tool for both creative and analytical tasks.<\/p>\n
Learning Objectives<\/h3>\n\n- Analyzing its architecture and key features that enhance its capabilities.<\/li>\n
- Exploring the underlying design and its impact on performance.<\/li>\n
- A step-by-step guide to building a Retrieval-Augmented Generation (RAG) system.<\/li>\n
- Utilizing the DeepSeek Janus Pro 1 billion model for real-world applications.<\/li>\n
- Understanding how DeepSeek Janus Pro optimizes AI-driven solutions.<\/li>\n<\/ul>\n
This article was published as a part of the\u00a0<\/strong><\/em>Data Science Blogathon.<\/strong><\/em><\/a><\/p>\nWhat is DeepSeek Janus Pro?<\/h2>\n
DeepSeek<\/a> Janus Pro is a multimodal AI model that integrates text and image processing, capable of understanding and generating images from text prompts. The 1 billion parameter version (1B) is designed for efficient performance across applications like text-to-image generation and image understanding tasks.<\/p>\nUnder DeepSeek\u2019s Janus Pro series, the primary models available are\u00a0\u201cJanus Pro 1B\u201d and \u201cJanus Pro 7B\u201d,<\/b> which differ mainly in their parameter size, with the 7B model being significantly larger and offering improved performance in text-to-image generation tasks;\u00a0both are considered multimodal models capable of handling both visual understanding and text generation based on visual context.<\/p>\nKey Features and Design Aspects of Janus Pro 1B<\/h3>\n\n- Architecture<\/b>: Janus Pro uses a unified transformer architecture but decouples visual encoding into separate pathways to improve performance in both image understanding and creation tasks.<\/li>\n
- Capabilities<\/b>: It excels in tasks related to both understanding of images and the generation of new ones based on text prompts. It supports 384\u00d7384 image inputs.<\/li>\n
- Image Encoders<\/b>: For image understanding tasks, Janus uses SigLIP to encode images. SigLIP is an image embedding model that uses CLIP\u2019s framework but replaces the loss function with a pairwise sigmoid loss. For image generation, Janus uses an existing encoder from LlamaGen, an autoregressive image generation mode. LlamaGen is a family of image-generation models that applies the next-token prediction paradigm of large language models to a visual generation<\/li>\n
- Open Source:<\/b> It is available on GitHub under the MIT License, with model usage governed by the DeepSeek Model License.<\/li>\n<\/ul>\n
Also read: How to Access DeepSeek Janus Pro 7B?<\/a><\/p>\nDecoupled Architecture For Image Understanding & Generation<\/h2>\n\n![\"Architectural](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/image_O4tMZCG.webp\")
Architectural Features of Deepsee<\/figcaption><\/figure>\n<\/div>\nJanus-Pro diverges from previous multimodal models by employing separate, specialized pathways for visual encoding, rather than relying on a single visual encoder for both image understanding and generation.<\/p>\n
\n- Image Understanding Encoder.<\/b> This pathway extracts semantic features from images.<\/li>\n
- Image Generation Encoder.<\/b> This pathway synthesizes images based on text descriptions.<\/li>\n<\/ul>\n
This decoupled architecture facilitates task-specific optimizations, mitigating conflicts between interpretation and creative synthesis. The independent encoders interpret input features which are then processed by a unified autoregressive transformer. This allows both multimodal understanding and generation components to independently select their most suitable encoding methods.<\/p>\n
Also read: How DeepSeek\u2019s Janus Pro Stacks Up Against DALL-E 3?<\/a><\/p>\nKey Features of Model Architecture<\/h3>\n1. Dual-pathway architecture for visual understanding & generation<\/h3>\n\n- Visual Understanding Pathway:<\/b>\u00a0For multimodal understanding tasks, Janus Pro uses SigLIP-L as the visual encoder, which supports image inputs of up to 384\u00d7384 resolution. This high-resolution support allows the model to capture more image details, thereby improving the accuracy of visual understanding.\u00a0\u00a0<\/li>\n
- Visual Generation Pathway<\/b>:\u00a0For image generation tasks, Janus Pro uses LlamaGen Tokenizer with a downsampling rate of 16 to generate more detailed images.\u00a0\u00a0<\/li>\n<\/ul>\n
![\"DeepSeek](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/image_RdlmqXs.webp\")
Fig 1. The architecture of our Janus-Pro. We decouple visual encoding for multimodal understanding and visual generation. \u201cUnd. Encoder\u201d and \u201cGen. Encoder\u201d are abbreviations for \u201cUnderstanding Encoder\u201d and \u201cGeneration Encoder\u201d, respectively. Source: DeepSeek Janus-Pro<\/a><\/figcaption><\/figure>\n2. Unified Transformer Architecture<\/h3>\n
A shared transformer backbone is used\u00a0for\u00a0text and image feature fusion. The independent encoding methods to convert the raw inputs into features are processed by a unified autoregressive transformer.\u00a0\u00a0<\/p>\n
3. Optimized Training Strategy<\/h3>\n
In Previous Janus training, there was a three-stage training process for the model. The first stage focused on training the adaptors and the image head. The second stage handled unified pretraining, during which all components except the understanding encoder and the generation encoder have their parameters updated. Stage III covered supervised fine-tuning, building upon Stage II by further unlocking the parameters of the understanding encoder during training.<\/p>\n
This was improved in Janus Pro:<\/p>\n
\n- By increasing the training steps in Stage I, allowing sufficient training on the ImageNet dataset.<\/li>\n
- Additionally, in Stage II, for text-to-image generation training, the ImageNet data was dropped completely. Instead normal text-to-image data was utilized to train the model to generate images based on dense descriptions. This was found to improve the training efficiency and overall performance.<\/li>\n<\/ul>\n
Now, lets build Multimodal RAG with Deepseek Janus Pro:<\/p>\n
Multimodal RAG with Deepseek Janus Pro 1B model<\/h2>\n
In the following steps, we will build a multimodal RAG system to query on images based on the Deepseek Janus Pro 1B model.<\/p>\n
Step 1. Install Necessary Libraries<\/h3>\n!pip install byaldi ollama pdf2image\n!sudo apt-get install -y poppler-utils\n!git clone https:\/\/github.com\/deepseek-ai\/Janus.git\n!pip install -e .\/Janus<\/code><\/pre>\nStep 2. Model For Saving Image Embeddings<\/h3>\nimport os\nfrom pathlib import Path\nfrom byaldi import RAGMultiModalModel\nimport ollama\n# Initialize RAGMultiModalModel\nmodel1 = RAGMultiModalModel.from_pretrained(\"vidore\/colqwen2-v0.1\")<\/code><\/pre>\nByaldi gives an easy-to-use framework for setting up multimodal RAG systems. As seen from the above code, we load Colqwen2, which is a model designed for efficient document indexing using visual features.\u00a0<\/p>\n
Step 3. Loading the Image PDF<\/h3>\n# Use ColQwen2 to index and store the presentation\nindex_name = \"image_index\"\nmodel1.index(input_path=Path(\"\/content\/PublicWaterMassMailing.pdf\"),\n index_name=index_name,\n store_collection_with_index=True, # Stores base64 images along with the vectors\n overwrite=True\n)<\/code><\/pre>\nWe use this PDF<\/a> to query and build an RAG system on in the next steps. In the above code, we store the image PDF along with the vectors.<\/p>\nStep 4. Querying & Retrieval From Saved Images<\/h3>\nquery = \"How many clients drive more than 50% revenue?\"\nreturned_page = model1.search(query, k=1)[0]\nimport base64\n# Example Base64 string (truncated for brevity)\nbase64_string = returned_page['base64']\n\n# Decode the Base64 string\nimage_data = base64.b64decode(base64_string)\nwith open('output_image.png', 'wb') as image_file:\n image_file.write(image_data)<\/code><\/pre>\nThe relevant page from the pages of the PDF is retrieved and saved as output_image.png based on the query.<\/p>\n
Step 5. Load Janus Pro Model<\/h3>\nimport os\nos.chdir(r\"\/content\/Janus\")\n\nfrom janus.models import VLChatProcessor\nfrom transformers import AutoConfig, AutoModelForCausalLM\nimport torch\nfrom janus.utils.io import load_pil_images\nfrom PIL import Image\n\nprocessor= VLChatProcessor.from_pretrained(\"deepseek-ai\/Janus-Pro-1B\")\ntokenizer = processor.tokenizer\nvl_gpt = AutoModelForCausalLM.from_pretrained(\n \"deepseek-ai\/Janus-Pro-1B\", trust_remote_code=True\n)\n\nconversation = [\n {\n \"role\": \"\",\n \"content\": f\"\\n{query}\",\n \"images\": ['\/content\/output_image.png'],\n },\n {\"role\": \"\", \"content\": \"\"},\n]\n\n# load images and prepare for inputs\npil_images = load_pil_images(conversation)\ninputs = processor(conversations=conversation, images=pil_images)\n\n# # run image encoder to get the image embeddings\ninputs_embeds = vl_gpt.prepare_inputs_embeds(**inputs)<\/image_placeholder><\/code><\/pre>\n\n- VLChatProcessor.from_pretrained(\u201cdeepseek-ai\/Janus-Pro-1B\u201d)<\/b> loads a pretrained processor for handling multimodal inputs (images and text). This processor will process and prepare input data (like text and images) for the model.<\/li>\n
- The tokenizer is extracted from the VLChatProcessor. It will tokenize the text input, converting text into a format suitable for the model.<\/li>\n
- AutoModelForCausalLM.from_pretrained(\u201cdeepseek-ai\/Janus-Pro-1B\u201d) <\/b>loads the pre-trained Janus Pro model, specifically for causal language modelling.<\/li>\n
- Also, a multimodal conversation format <\/b>is set up where the user inputs both text and an image.<\/li>\n
- The load_pil_images(conversation) <\/b>is a function that likely loads the images listed in the conversation object and converts them into PIL Image format, which is commonly used for image processing in Python.<\/li>\n
- The processor<\/b> here is an instance of a multimodal processor (the VLChatProcessor <\/b>from the DeepSeek Janus Pro model), which takes both text and image data as input.<\/li>\n
- prepare_inputs_embeds(inputs)<\/b> is a method that takes the processed inputs (inputs contain both the text and image)\u00a0, and prepares the embeddings required for the model to generate a response.<\/li>\n<\/ul>\n
Step 6. Output Generation<\/h3>\noutputs = vl_gpt.language_model.generate(\n inputs_embeds=inputs_embeds,\n attention_mask=inputs.attention_mask,\n pad_token_id=tokenizer.eos_token_id,\n bos_token_id=tokenizer.bos_token_id,\n eos_token_id=tokenizer.eos_token_id,\n max_new_tokens=512,\n do_sample=False,\n use_cache=True,\n)\n\nanswer = tokenizer.decode(outputs[0].cpu().tolist(), skip_special_tokens=True)\nprint(answer)<\/code><\/pre>\nThe code generates a response from the DeepSeek Janus Pro 1B model using the prepared input embeddings (text and image). It uses several configuration settings like padding, start\/end tokens, max token length, and whether to use caching and sampling. After the response is generated, it decodes the token IDs back into human-readable text using the tokenizer. The decoded output is stored in the answer variable.\u00a0<\/p>\n
The whole code is present in this colab notebook<\/a>.\u00a0\u00a0<\/p>\nOutput For the Query<\/h4>\n![\"output\"](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/image_76_YG7oUtq.webp\")
<\/figure>\nOutput For Another Query<\/h4>\n
\u201cWhat has been the revenue in France?\u201d<\/em><\/p>\n![\"output\"](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/image_mJHQSUu.webp\")
<\/figure>\nThe above response is not accurate even though the relevant page was retrieved by the\u00a0colqwen2 retriever, the DeepSeek Janus Pro 1B model could not generate the accurate answer from the page. The exact answer should be $2B.<\/p>\n
Output For Another Query<\/h4>\n
\u201c\u201dWhat has been the number of promotions since beginning of FY20?\u201d<\/em><\/p>\n![\"output\"](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/image_Hsk2vXk.webp\")
<\/figure>\nThe above response is correct as it matches with the text mentioned in the PDF.<\/p>\n
Conclusions<\/h2>\n
In conclusion, the DeepSeek Janus Pro 1B model represents a significant advancement in multimodal AI, with its decoupled architecture that optimizes both image understanding and generation tasks. By utilizing separate visual encoders for these tasks and refining its training strategy, Janus Pro offers enhanced performance in text-to-image generation and image analysis. This innovative approach (Multimodal RAG with Deepseek Janus Pro), combined with its open-source accessibility, makes it a powerful tool for various applications in AI-driven visual comprehension and creation.<\/p>\n
Key Takeaways<\/h2>\n\n- Multimodal AI with Dual Pathways<\/b>: Janus Pro 1B integrates both text and image processing, using separate encoders for image understanding (SigLIP) and image generation (LlamaGen), enhancing task-specific performance.<\/li>\n
- Decoupled Architecture:<\/b> The model separates visual encoding into distinct pathways, enabling independent optimization for image understanding and generation, thus minimizing conflicts in processing tasks.<\/li>\n
- Unified Transformer Backbone<\/b>: A shared transformer architecture merges the features of text and images, streamlining multimodal data fusion for more effective AI performance.<\/li>\n
- Improved Training Strategy:<\/b> Janus Pro\u2019s optimized training approach includes increased steps in Stage I and the use of specialized text-to-image data in Stage II, significantly boosting training efficiency and output quality.<\/li>\n
- Open-Source Accessibility: <\/b>Janus Pro 1B is available on GitHub under the MIT License, encouraging widespread use and adaptation in various AI-driven applications.<\/li>\n<\/ol>\n
The media shown in this article is not owned by Analytics Vidhya and is used at the Author\u2019s discretion.<\/strong><\/p>\nFrequently Asked Questions<\/h2>\n\nQ1. What is DeepSeek Janus Pro 1B?<\/strong> <\/p>\nAns. DeepSeek Janus Pro 1B is a multimodal AI model designed to integrate both text and image processing, capable of understanding and generating images from text descriptions. It features 1 billion parameters for efficient performance in tasks like text-to-image generation and image understanding.<\/p>\n<\/p><\/div>\n
This article was published as a part of the\u00a0<\/strong><\/em>Data Science Blogathon.<\/strong><\/em><\/a><\/p>\n DeepSeek<\/a> Janus Pro is a multimodal AI model that integrates text and image processing, capable of understanding and generating images from text prompts. The 1 billion parameter version (1B) is designed for efficient performance across applications like text-to-image generation and image understanding tasks.<\/p>\n Under DeepSeek\u2019s Janus Pro series, the primary models available are\u00a0\u201cJanus Pro 1B\u201d and \u201cJanus Pro 7B\u201d,<\/b> which differ mainly in their parameter size, with the 7B model being significantly larger and offering improved performance in text-to-image generation tasks;\u00a0both are considered multimodal models capable of handling both visual understanding and text generation based on visual context.<\/p>\n Also read: How to Access DeepSeek Janus Pro 7B?<\/a><\/p>\n Janus-Pro diverges from previous multimodal models by employing separate, specialized pathways for visual encoding, rather than relying on a single visual encoder for both image understanding and generation.<\/p>\n This decoupled architecture facilitates task-specific optimizations, mitigating conflicts between interpretation and creative synthesis. The independent encoders interpret input features which are then processed by a unified autoregressive transformer. This allows both multimodal understanding and generation components to independently select their most suitable encoding methods.<\/p>\n Also read: How DeepSeek\u2019s Janus Pro Stacks Up Against DALL-E 3?<\/a><\/p>\n A shared transformer backbone is used\u00a0for\u00a0text and image feature fusion. The independent encoding methods to convert the raw inputs into features are processed by a unified autoregressive transformer.\u00a0\u00a0<\/p>\n In Previous Janus training, there was a three-stage training process for the model. The first stage focused on training the adaptors and the image head. The second stage handled unified pretraining, during which all components except the understanding encoder and the generation encoder have their parameters updated. Stage III covered supervised fine-tuning, building upon Stage II by further unlocking the parameters of the understanding encoder during training.<\/p>\n This was improved in Janus Pro:<\/p>\n Now, lets build Multimodal RAG with Deepseek Janus Pro:<\/p>\n In the following steps, we will build a multimodal RAG system to query on images based on the Deepseek Janus Pro 1B model.<\/p>\n Byaldi gives an easy-to-use framework for setting up multimodal RAG systems. As seen from the above code, we load Colqwen2, which is a model designed for efficient document indexing using visual features.\u00a0<\/p>\n We use this PDF<\/a> to query and build an RAG system on in the next steps. In the above code, we store the image PDF along with the vectors.<\/p>\n The relevant page from the pages of the PDF is retrieved and saved as output_image.png based on the query.<\/p>\n The code generates a response from the DeepSeek Janus Pro 1B model using the prepared input embeddings (text and image). It uses several configuration settings like padding, start\/end tokens, max token length, and whether to use caching and sampling. After the response is generated, it decodes the token IDs back into human-readable text using the tokenizer. The decoded output is stored in the answer variable.\u00a0<\/p>\n The whole code is present in this colab notebook<\/a>.\u00a0\u00a0<\/p>\n \u201cWhat has been the revenue in France?\u201d<\/em><\/p>\n The above response is not accurate even though the relevant page was retrieved by the\u00a0colqwen2 retriever, the DeepSeek Janus Pro 1B model could not generate the accurate answer from the page. The exact answer should be $2B.<\/p>\n \u201c\u201dWhat has been the number of promotions since beginning of FY20?\u201d<\/em><\/p>\n The above response is correct as it matches with the text mentioned in the PDF.<\/p>\n In conclusion, the DeepSeek Janus Pro 1B model represents a significant advancement in multimodal AI, with its decoupled architecture that optimizes both image understanding and generation tasks. By utilizing separate visual encoders for these tasks and refining its training strategy, Janus Pro offers enhanced performance in text-to-image generation and image analysis. This innovative approach (Multimodal RAG with Deepseek Janus Pro), combined with its open-source accessibility, makes it a powerful tool for various applications in AI-driven visual comprehension and creation.<\/p>\n The media shown in this article is not owned by Analytics Vidhya and is used at the Author\u2019s discretion.<\/strong><\/p>\n Ans. DeepSeek Janus Pro 1B is a multimodal AI model designed to integrate both text and image processing, capable of understanding and generating images from text descriptions. It features 1 billion parameters for efficient performance in tasks like text-to-image generation and image understanding.<\/p>\n<\/p><\/div>\nWhat is DeepSeek Janus Pro?<\/h2>\n
Key Features and Design Aspects of Janus Pro 1B<\/h3>\n
\n
Decoupled Architecture For Image Understanding & Generation<\/h2>\n
\n
Key Features of Model Architecture<\/h3>\n
1. Dual-pathway architecture for visual understanding & generation<\/h3>\n
\n
2. Unified Transformer Architecture<\/h3>\n
3. Optimized Training Strategy<\/h3>\n
\n
Multimodal RAG with Deepseek Janus Pro 1B model<\/h2>\n
Step 1. Install Necessary Libraries<\/h3>\n
!pip install byaldi ollama pdf2image\n!sudo apt-get install -y poppler-utils\n!git clone https:\/\/github.com\/deepseek-ai\/Janus.git\n!pip install -e .\/Janus<\/code><\/pre>\nStep 2. Model For Saving Image Embeddings<\/h3>\n
import os\nfrom pathlib import Path\nfrom byaldi import RAGMultiModalModel\nimport ollama\n# Initialize RAGMultiModalModel\nmodel1 = RAGMultiModalModel.from_pretrained(\"vidore\/colqwen2-v0.1\")<\/code><\/pre>\nStep 3. Loading the Image PDF<\/h3>\n
# Use ColQwen2 to index and store the presentation\nindex_name = \"image_index\"\nmodel1.index(input_path=Path(\"\/content\/PublicWaterMassMailing.pdf\"),\n index_name=index_name,\n store_collection_with_index=True, # Stores base64 images along with the vectors\n overwrite=True\n)<\/code><\/pre>\nStep 4. Querying & Retrieval From Saved Images<\/h3>\n
query = \"How many clients drive more than 50% revenue?\"\nreturned_page = model1.search(query, k=1)[0]\nimport base64\n# Example Base64 string (truncated for brevity)\nbase64_string = returned_page['base64']\n\n# Decode the Base64 string\nimage_data = base64.b64decode(base64_string)\nwith open('output_image.png', 'wb') as image_file:\n image_file.write(image_data)<\/code><\/pre>\nStep 5. Load Janus Pro Model<\/h3>\n
import os\nos.chdir(r\"\/content\/Janus\")\n\nfrom janus.models import VLChatProcessor\nfrom transformers import AutoConfig, AutoModelForCausalLM\nimport torch\nfrom janus.utils.io import load_pil_images\nfrom PIL import Image\n\nprocessor= VLChatProcessor.from_pretrained(\"deepseek-ai\/Janus-Pro-1B\")\ntokenizer = processor.tokenizer\nvl_gpt = AutoModelForCausalLM.from_pretrained(\n \"deepseek-ai\/Janus-Pro-1B\", trust_remote_code=True\n)\n\nconversation = [\n {\n \"role\": \"\",\n \"content\": f\"\n
Step 6. Output Generation<\/h3>\n
outputs = vl_gpt.language_model.generate(\n inputs_embeds=inputs_embeds,\n attention_mask=inputs.attention_mask,\n pad_token_id=tokenizer.eos_token_id,\n bos_token_id=tokenizer.bos_token_id,\n eos_token_id=tokenizer.eos_token_id,\n max_new_tokens=512,\n do_sample=False,\n use_cache=True,\n)\n\nanswer = tokenizer.decode(outputs[0].cpu().tolist(), skip_special_tokens=True)\nprint(answer)<\/code><\/pre>\nOutput For the Query<\/h4>\n
<\/figure>\nOutput For Another Query<\/h4>\n
<\/figure>\nOutput For Another Query<\/h4>\n
<\/figure>\nConclusions<\/h2>\n
Key Takeaways<\/h2>\n
\n
Frequently Asked Questions<\/h2>\n
![\"Nibedita](\"https:\/\/av-eks-lekhak.s3.amazonaws.com\/media\/lekhak-profile-images\/converted_image_ZkJo4gb.webp\")
<\/p>\n