{"id":7029847,"date":"2026-08-06T08:09:07","date_gmt":"2026-08-06T08:09:07","guid":{"rendered":"https:\/\/peraltafinancing.com\/analytics\/does-ai-find-real-ui-problems-or-just-hallucinations-measuringu\/"},"modified":"2026-08-06T08:09:07","modified_gmt":"2026-08-06T08:09:07","slug":"does-ai-find-real-ui-problems-or-just-hallucinations-measuringu","status":"publish","type":"post","link":"https:\/\/fivemor.com\/?p=7029847","title":{"rendered":"Does AI Find Real UI Problems or Just Hallucinations? \u2013 MeasuringU"},"content":{"rendered":"


\n<\/p>\n

\n

\"Feature<\/a>In a previous\u00a0experiment<\/a>, AI identified roughly half the usability problems that trained researchers found in a video of a usability test session.<\/p>\n

That sounds promising. If AI can find usability issues, it can substantially increase the amount of usability testing that research teams can conduct.<\/p>\n

But in our analysis of that video, AI generated nearly as many additional<\/em> problems that humans never flagged. Are these problems hidden gems missed by multiple researchers, or just AI hallucinations?<\/p>\n

For this article, we classified all the unique problems the AIs generated into one of three categories:<\/p>\n

    \n
  1. a real problem humans missed<\/li>\n
  2. a false alarm (a true observation misread as a usability problem)<\/li>\n
  3. a hallucination (something the AI reported that simply never happened)<\/li>\n<\/ol>\n

    What we found suggests that the new AI problems are mostly false alarms, but there are some notable exceptions.<\/p>\n

    Experimental Design: Four Researchers, Two LLMs, and One Video<\/h2>\n

    For this study, we had four humans (professional UX researchers working at MeasuringU) review a video from a previous usability benchmark study of online dining reservation websites. Each researcher independently created a list of the usability issues they observed in the six-minute video.<\/p>\n

    We ran the video through two LLMs (ChatGPT-5.4 Thinking and Gemini 3 Flash Thinking) four times using the same prompt<\/a> each time.<\/p>\n

    So in this study, we held constant the video, the key elements of the prompt, and the LLM versions\/settings\u2014variables that we plan to vary in future studies. This time, we varied only the type of analyst: human, ChatGPT, and Gemini.<\/p>\n

    Gemini Finds a Jewel; ChatGPT Goes on a Tangent<\/h2>\n

    Using the human-generated and -verified problem lists as the \u201cgold standard,\u201d Figure 1 shows a summary of what we found.<\/p>\n

    \"Venn<\/a><\/p>\n

    Figure 1: <\/strong>Venn diagram of usability problem discovery by humans, ChatGPT, and Gemini.<\/p>\n

    We know Venn diagrams can generate some bad high school math memories, so here\u2019s a summary for all our sanity:<\/p>\n