Can AI detect and fix coding errors just by analyzing a screenshot? With a Multi-Agent System for Automatic Code Error Detection, the answer is yes. This innovative approach uses artificial intelligence and reasoning to identify coding mistakes from images, propose accurate solutions, and explain the logic behind them. At the core is a decentralized Multi-Agent System, where autonomous agents\u2014such as AI models, tools, or services\u2014work collaboratively. Each agent gathers data, makes localized decisions, and contributes to solving complex debugging tasks. By automating this process, developers can save time, improve accuracy, and avoid the manual hassle of searching for solutions online.<\/p>\n
Learning Objectives<\/h3>\n\n- Understand the Multi-Agent System with Reasoning and how it automates error detection and solution generation from screenshots.<\/li>\n
- Explore the role of artificial intelligence in enhancing the efficiency of a Multi-Agent System with Reasoning for software debugging.<\/li>\n
- Learn how Griptape simplifies the development of multi-agent systems with modular workflows.<\/li>\n
- Implement a multi-agent system for detecting coding errors from screenshots using AI models.<\/li>\n
- Utilize vision language models and reasoning-based LLMs<\/a> for automated error detection and explanation.<\/li>\n
- Build and deploy AI agents specialized in web searching, reasoning, and image analysis.<\/li>\n
- Develop structured workflows to extract, analyze, and resolve coding errors efficiently.<\/li>\n
- Optimize security, scalability, and reliability in multi-agent system implementations.<\/li>\n<\/ul>\n
This article was published as a part of the\u00a0<\/strong><\/em>Data Science Blogathon.<\/strong><\/em><\/a><\/p>\nMulti Agentic Systems: A brief Introduction<\/h2>\n
Multi-Agent Systems (MAS)\u00a0represent intricate frameworks consisting of numerous interactive intelligent agents, each possessing unique skills and objectives. These agents can take various forms, including software applications, robotic entities, drones, sensors, and even humans, or a blend of these elements. The primary purpose of MAS is to address challenges that individual agents struggle to manage independently by harnessing the power of collective intelligence, collaboration, and coordinated efforts among the agents.<\/p>\n
Distinctive Features of Multi-Agent Systems<\/h3>\n\n- Autonomy<\/b>: Each agent functions with a level of self-governance, making choices based on its localized understanding of the surroundings.<\/li>\n
- Decentralization<\/b>: Authority is spread across the agents, enabling the system to maintain operations even if certain parts fail.<\/li>\n
- Self-Organization: <\/b>Agents possess the ability to adjust and arrange themselves according to emergent behaviors, resulting in effective task distribution and conflict management.<\/li>\n
- Real-Time Functionality: <\/b>MAS can swiftly react to dynamic conditions without requiring human oversight, which makes them ideal for scenarios such as emergency response and traffic regulation.<\/li>\n<\/ul>\n
Some Practical Examples of Multi Agent Systems<\/h2>\n
Multi-agent systems are transforming various industries by enabling intelligent collaboration among autonomous agents. Here are some practical examples showcasing their real-world applications.<\/p>\n
\n- Agent For Resolving Queries Dynamically: <\/b>This is a sophisticated multi-agent system designed to address customer inquiries effectively. It begins by tapping into its extensive knowledge base and, when necessary, retrieves pertinent information from integrated tools to deliver precise answers.<\/li>\n
- Dynamic Assignment of Tickets: <\/b>This advanced multi-agent system streamlines the ticket management workflow within the Customer Support division by automatically directing incoming support tickets to the most appropriate agents. Utilizing generative AI, it evaluates each ticket based on established criteria such as category, severity, and agent specialization.<\/li>\n
- Agent For Analyzing Gaps in Knowledge Base:\u00a0<\/b>This specialized multi-agent system aims to enhance the efficacy of the knowledge base by pinpointing recurring support challenges that require better coverage in existing articles. Through the use of generative AI, this agent examines trends in support tickets and customer inquiries to identify areas needing improvement.<\/li>\n<\/ul>\n
Building Multi-Agent Systems with Griptape<\/h2>\n
The Griptape framework streamlines the development of collaborative AI agents by balancing predictability and creativity through modular design and secure workflows.<\/p>\n
Agent Specialization and Coordination<\/h3>\n
Griptape enables developers to define agents with distinct roles, such as:<\/p>\n
\n- Research Agents: <\/strong>Gather data using tools like web search and scraping<\/li>\n
- Writer Agents:<\/strong> Transform insights into narratives tailored to specific audiences<\/li>\n
- Analytical Agents:<\/strong> Validate outputs against predefined schemas or business rules<\/li>\n<\/ul>\n
Agents interact through workflows that parallelize tasks while maintaining dependencies. For example, a research agent\u2019s findings can trigger multiple writer agents to generate content simultaneously<\/p>\n
Workflow Design<\/h3>\n
The framework supports two approaches:<\/p>\n
\n- Sequential Pipelines:<\/strong> For linear task execution (e.g., data ingestion \u2192 analysis \u2192 reporting)<\/li>\n
- DAG-Based Workflows<\/strong>: For complex, branching logic where agents dynamically adjust based on intermediate outputs<\/li>\n<\/ul>\n
Security and Scalability<\/h3>\n
Key safeguards include:<\/p>\n
\n- Off-prompt data handling: <\/strong>Minimizes exposure of sensitive information during LLM interactions<\/li>\n
- Permission controls: <\/strong>Restricts tool usage based on agent roles<\/li>\n
- Cloud integration:<\/strong> Deploys agents independently via services like Griptape Cloud for horizontal scaling<\/li>\n<\/ul>\n
Implementation Best Practices<\/h3>\n\n- Use Rulesets to enforce agent behavior (e.g., formatting standards, ethical guidelines)<\/li>\n
- Leverage Memory types: Short-term for creative tasks, long-term for structured processes<\/li>\n
- Test workflows locally before deploying to distributed environments<\/li>\n<\/ul>\n
Griptape\u2019s modular architecture reduces reliance on prompt engineering by prioritizing Python code for logic definition, making it ideal for enterprise-grade applications like customer support automation and real-time data analysis pipelines<\/p>\n
Hands on Implementation of Developing a Multi-agent system for Resolution of Coding Errors<\/h2>\n
In this tutorial, we will be creating a multi-agent system aimed at automatically detecting errors from coding screenshots, specifically using Python for our example. This system will not only identify errors but also offer users clear explanations on how to resolve them. Throughout this process, we will utilize vision language models in conjunction with reasoning-based large language models to enhance the functionality of our multi-agent framework.<\/p>\n
Step1: Installing and Importing Necessary Libraries<\/h3>\n
First we will install all required libraries below:<\/p>\n
!pip install griptape\n!sudo apt update\n!sudo apt install -y pciutils\n!pip install langchain-ollama\n!curl -fsSL https:\/\/ollama.com\/install.sh | sh\n!pip install ollama==0.4.2\n!pip install \"duckduckgo-search>=7.0.1\"\n\n\nimport os\nfrom griptape.drivers.prompt.ollama import OllamaPromptDriver\nimport requests\nfrom griptape.drivers.file_manager.local import LocalFileManagerDriver\nfrom griptape.drivers.prompt.openai import OpenAiChatPromptDriver\nfrom griptape.loaders import ImageLoader\nfrom griptape.structures import Agent\nfrom griptape.tools import FileManagerTool, ImageQueryTool\nfrom griptape.tasks import PromptTask, StructureRunTask\nfrom griptape.drivers.structure_run.local import LocalStructureRunDriver\nfrom griptape.structures import Agent, Workflow\n\nfrom griptape.drivers.web_search.duck_duck_go import DuckDuckGoWebSearchDriver\nfrom griptape.structures import Agent\nfrom griptape.tools import PromptSummaryTool, WebSearchTool<\/code><\/pre>\nStep2:\u00a0Running the Ollama Server and Pulling the Models<\/h3>\n
The following code starts the ollama server. We also pull \u201cminicpm-v\u201d model from ollama so that this vision model can be used to extract text from handwritten notes.<\/p>\n
import threading\nimport subprocess\nimport time\n\ndef run_ollama_serve():\n subprocess.Popen([\"ollama\", \"serve\"])\n\nthread = threading.Thread(target=run_ollama_serve)\nthread.start()\ntime.sleep(5)\n\n!ollama pull minicpm-v<\/code><\/pre>\nNow we also set the Open AI API key below which is needed to chat with the Ollama Model on Griptape<\/p>\n
import os\nos.environ[\"OPENAI_API_KEY\"] = \"\"<\/code><\/pre>\nWe will also leverage a powerful LLM to assist with reasoning by explaining the coding error and the provided solution with sufficient context. For this, we pull the phi4-mini model.<\/p>\n
!ollama pull phi4-mini<\/code><\/pre>\nStep3: Creating an Agent to Analyze Screenshots<\/h3>\n
We start with creating an agent to analyze screenshots of Python Coding Errors. This agent leverages a vision language model (minicpm-v) in the backend.<\/p>\n
images_dir = os.getcwd()\n\n\ndef analyze_screenshots():\n driver = LocalFileManagerDriver(workdir=images_dir)\n return Agent(\n tools=[\n FileManagerTool(file_manager_driver=driver),\n ImageQueryTool(\n prompt_driver=OllamaPromptDriver(model=\"minicpm-v\"), image_loader=ImageLoader(file_manager_driver=driver)\n ),\n ])<\/code><\/pre>\nStep4: Creating Agents For Web searching and Reasoning<\/h3>\n
We then create two agents, one for web searching on possible solutions of the coding error and another for reasoning behind the error and its found solutions.<\/p>\n
def websearching_agent():\n return Agent(\n tools=[WebSearchTool(web_search_driver=DuckDuckGoWebSearchDriver()), PromptSummaryTool(off_prompt=False)],\n )\n \n \ndef reasoning_agent():\n\t return Agent(\n\t prompt_driver=OllamaPromptDriver(\n\t model=\"phi4-mini\",\n\t ))<\/code><\/pre>\nStep5: Defining Tasks For Analyzing Screenshots, Finding Solutions and Providing with Reasoning<\/h3>\n
We use screenshots<\/a> like this for automatic evaluation. We save it in our current working directory as \u201csample.jpg\u201d. Its a handwritten answer sheet. This agentic system will first extract errors from coding screenshots and identify possible solutions. It will then provide sufficient reasoning behind the errors and their solutions.<\/p>\nimage_file_name = \"pythonerror1.jpg\"\nteam = Workflow()\nscreenshotanalysis_task= StructureRunTask(\n (\n \"\"\"Extract IN TEXT FORMAT ALL THE LINES FROM THE GIVEN SCREEN SHOT %s\"\"\"%(image_file_name),\n ),\n id=\"research\",\n structure_run_driver=LocalStructureRunDriver(\n create_structure=analyze_screenshots,\n ),\n )\n\nfindingsolution_task =StructureRunTask(\n (\n \"\"\"FIND SOLUTION TO ONLY THE CODING ERRORS FOUND in the TEXT {{ parent_outputs[\"research\"] }}. DO NOT INCLUDE ANY ADDITIONAL JUNK NON CODING LINES WHILE FINDING THE SOLUTION.\n\n \"\"\",\n ),id=\"evaluate\",\n structure_run_driver=LocalStructureRunDriver(\n create_structure=websearching_agent,\n )\n )\n\nreasoningsolution_task = StructureRunTask(\n (\n \"\"\"ADD TO THE PREVIOUS OUTPUT, EXPANDED VERSION OF REASONING ON HOW TO SOLVE THE ERROR BASED ON {{ parent_outputs[\"evaluate\"] }}.\n DO INCLUDE THE WHOLE OUTPUT FROM THE PREVIOUS AGENT {{ parent_outputs[\"evaluate\"] }} AS WELL IN THE FINAL OUTPUT.\n \n \n \"\"\",\n ),\n structure_run_driver=LocalStructureRunDriver(\n create_structure=reasoning_agent,\n )\n )\n <\/code><\/pre>\nStep6: Executing the Workflow\u00a0<\/h3>\n
Now we will execute the workflow.<\/p>\n
screenshotanalysis_task.add_child(findingsolution_task)\nfindingsolution_task.add_child(reasoningsolution_task)\nscreenshotanalysis_task.add_child(reasoningsolution_task) \n \nteam = Workflow(\n tasks=[screenshotanalysis_task,findingsolution_task,reasoningsolution_task],\n)\nanswer = team.run()\nprint(answer.output)<\/code><\/pre>\nInput Screenshot<\/b><\/p>\n![\"input](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/03\/input-snap.webp\")
<\/figure>\nOutput From Agentic System<\/h4>\n```python\")As seen from the above output, not just the error is correctly explained with sufficient reasoning but the solution is also provided with enough reasoning.<\/p>\n
Analyzing with Other Queries<\/h2>\n
This article was published as a part of the\u00a0<\/strong><\/em>Data Science Blogathon.<\/strong><\/em><\/a><\/p>\n Multi-Agent Systems (MAS)\u00a0represent intricate frameworks consisting of numerous interactive intelligent agents, each possessing unique skills and objectives. These agents can take various forms, including software applications, robotic entities, drones, sensors, and even humans, or a blend of these elements. The primary purpose of MAS is to address challenges that individual agents struggle to manage independently by harnessing the power of collective intelligence, collaboration, and coordinated efforts among the agents.<\/p>\n Multi-agent systems are transforming various industries by enabling intelligent collaboration among autonomous agents. Here are some practical examples showcasing their real-world applications.<\/p>\n The Griptape framework streamlines the development of collaborative AI agents by balancing predictability and creativity through modular design and secure workflows.<\/p>\n Griptape enables developers to define agents with distinct roles, such as:<\/p>\n Agents interact through workflows that parallelize tasks while maintaining dependencies. For example, a research agent\u2019s findings can trigger multiple writer agents to generate content simultaneously<\/p>\n The framework supports two approaches:<\/p>\n Key safeguards include:<\/p>\n Griptape\u2019s modular architecture reduces reliance on prompt engineering by prioritizing Python code for logic definition, making it ideal for enterprise-grade applications like customer support automation and real-time data analysis pipelines<\/p>\n In this tutorial, we will be creating a multi-agent system aimed at automatically detecting errors from coding screenshots, specifically using Python for our example. This system will not only identify errors but also offer users clear explanations on how to resolve them. Throughout this process, we will utilize vision language models in conjunction with reasoning-based large language models to enhance the functionality of our multi-agent framework.<\/p>\n First we will install all required libraries below:<\/p>\n The following code starts the ollama server. We also pull \u201cminicpm-v\u201d model from ollama so that this vision model can be used to extract text from handwritten notes.<\/p>\n Now we also set the Open AI API key below which is needed to chat with the Ollama Model on Griptape<\/p>\n We will also leverage a powerful LLM to assist with reasoning by explaining the coding error and the provided solution with sufficient context. For this, we pull the phi4-mini model.<\/p>\n We start with creating an agent to analyze screenshots of Python Coding Errors. This agent leverages a vision language model (minicpm-v) in the backend.<\/p>\n We then create two agents, one for web searching on possible solutions of the coding error and another for reasoning behind the error and its found solutions.<\/p>\n We use screenshots<\/a> like this for automatic evaluation. We save it in our current working directory as \u201csample.jpg\u201d. Its a handwritten answer sheet. This agentic system will first extract errors from coding screenshots and identify possible solutions. It will then provide sufficient reasoning behind the errors and their solutions.<\/p>\n Now we will execute the workflow.<\/p>\n Input Screenshot<\/b><\/p>\n ```python As seen from the above output, not just the error is correctly explained with sufficient reasoning but the solution is also provided with enough reasoning.<\/p>\nMulti Agentic Systems: A brief Introduction<\/h2>\n
Distinctive Features of Multi-Agent Systems<\/h3>\n
\n
Some Practical Examples of Multi Agent Systems<\/h2>\n
\n
Building Multi-Agent Systems with Griptape<\/h2>\n
Agent Specialization and Coordination<\/h3>\n
\n
Workflow Design<\/h3>\n
\n
Security and Scalability<\/h3>\n
\n
Implementation Best Practices<\/h3>\n
\n
Hands on Implementation of Developing a Multi-agent system for Resolution of Coding Errors<\/h2>\n
Step1: Installing and Importing Necessary Libraries<\/h3>\n
!pip install griptape\n!sudo apt update\n!sudo apt install -y pciutils\n!pip install langchain-ollama\n!curl -fsSL https:\/\/ollama.com\/install.sh | sh\n!pip install ollama==0.4.2\n!pip install \"duckduckgo-search>=7.0.1\"\n\n\nimport os\nfrom griptape.drivers.prompt.ollama import OllamaPromptDriver\nimport requests\nfrom griptape.drivers.file_manager.local import LocalFileManagerDriver\nfrom griptape.drivers.prompt.openai import OpenAiChatPromptDriver\nfrom griptape.loaders import ImageLoader\nfrom griptape.structures import Agent\nfrom griptape.tools import FileManagerTool, ImageQueryTool\nfrom griptape.tasks import PromptTask, StructureRunTask\nfrom griptape.drivers.structure_run.local import LocalStructureRunDriver\nfrom griptape.structures import Agent, Workflow\n\nfrom griptape.drivers.web_search.duck_duck_go import DuckDuckGoWebSearchDriver\nfrom griptape.structures import Agent\nfrom griptape.tools import PromptSummaryTool, WebSearchTool<\/code><\/pre>\nStep2:\u00a0Running the Ollama Server and Pulling the Models<\/h3>\n
import threading\nimport subprocess\nimport time\n\ndef run_ollama_serve():\n subprocess.Popen([\"ollama\", \"serve\"])\n\nthread = threading.Thread(target=run_ollama_serve)\nthread.start()\ntime.sleep(5)\n\n!ollama pull minicpm-v<\/code><\/pre>\nimport os\nos.environ[\"OPENAI_API_KEY\"] = \"\"<\/code><\/pre>\n!ollama pull phi4-mini<\/code><\/pre>\nStep3: Creating an Agent to Analyze Screenshots<\/h3>\n
images_dir = os.getcwd()\n\n\ndef analyze_screenshots():\n driver = LocalFileManagerDriver(workdir=images_dir)\n return Agent(\n tools=[\n FileManagerTool(file_manager_driver=driver),\n ImageQueryTool(\n prompt_driver=OllamaPromptDriver(model=\"minicpm-v\"), image_loader=ImageLoader(file_manager_driver=driver)\n ),\n ])<\/code><\/pre>\nStep4: Creating Agents For Web searching and Reasoning<\/h3>\n
def websearching_agent():\n return Agent(\n tools=[WebSearchTool(web_search_driver=DuckDuckGoWebSearchDriver()), PromptSummaryTool(off_prompt=False)],\n )\n \n \ndef reasoning_agent():\n\t return Agent(\n\t prompt_driver=OllamaPromptDriver(\n\t model=\"phi4-mini\",\n\t ))<\/code><\/pre>\nStep5: Defining Tasks For Analyzing Screenshots, Finding Solutions and Providing with Reasoning<\/h3>\n
image_file_name = \"pythonerror1.jpg\"\nteam = Workflow()\nscreenshotanalysis_task= StructureRunTask(\n (\n \"\"\"Extract IN TEXT FORMAT ALL THE LINES FROM THE GIVEN SCREEN SHOT %s\"\"\"%(image_file_name),\n ),\n id=\"research\",\n structure_run_driver=LocalStructureRunDriver(\n create_structure=analyze_screenshots,\n ),\n )\n\nfindingsolution_task =StructureRunTask(\n (\n \"\"\"FIND SOLUTION TO ONLY THE CODING ERRORS FOUND in the TEXT {{ parent_outputs[\"research\"] }}. DO NOT INCLUDE ANY ADDITIONAL JUNK NON CODING LINES WHILE FINDING THE SOLUTION.\n\n \"\"\",\n ),id=\"evaluate\",\n structure_run_driver=LocalStructureRunDriver(\n create_structure=websearching_agent,\n )\n )\n\nreasoningsolution_task = StructureRunTask(\n (\n \"\"\"ADD TO THE PREVIOUS OUTPUT, EXPANDED VERSION OF REASONING ON HOW TO SOLVE THE ERROR BASED ON {{ parent_outputs[\"evaluate\"] }}.\n DO INCLUDE THE WHOLE OUTPUT FROM THE PREVIOUS AGENT {{ parent_outputs[\"evaluate\"] }} AS WELL IN THE FINAL OUTPUT.\n \n \n \"\"\",\n ),\n structure_run_driver=LocalStructureRunDriver(\n create_structure=reasoning_agent,\n )\n )\n <\/code><\/pre>\nStep6: Executing the Workflow\u00a0<\/h3>\n
screenshotanalysis_task.add_child(findingsolution_task)\nfindingsolution_task.add_child(reasoningsolution_task)\nscreenshotanalysis_task.add_child(reasoningsolution_task) \n \nteam = Workflow(\n tasks=[screenshotanalysis_task,findingsolution_task,reasoningsolution_task],\n)\nanswer = team.run()\nprint(answer.output)<\/code><\/pre>\n<\/figure>\nOutput From Agentic System<\/h4>\n
Analyzing with Other Queries<\/h2>\n
![\"imput2;](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/03\/image_rCCqtFF.webp\")
<\/figure>\n![\"input3](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/03\/input3-python-code-error.webp\")
<\/figure>\n![\"Nibedita](\"https:\/\/av-eks-lekhak.s3.amazonaws.com\/media\/lekhak-profile-images\/converted_image_ZkJo4gb.webp\")
<\/p>\n