We can talk about hypotheticals, or we can actually try it out. We tried it out.<\/p>\n In our first study<\/a>, we started small. We compared human and AI reviews of a video taken from a usability test on making dinner reservations with OpenTable. We used the standard ChatGPT and Gemini interfaces (standard Claude doesn\u2019t support video review).<\/p>\n The good:<\/strong> AI identified roughly half<\/strong> the usability problems identified by human UX researchers. It even uncovered one genuine problem not found by any of the four human evaluators.<\/p>\n The bad:<\/strong> AI identified seven false alarms. They were true statements but not connected to the participants\u2019 actions or utterances.<\/p>\n The ugly:<\/strong> Three problems were straight-up hallucinations. AI described events that just didn\u2019t happen.<\/p>\n So, of the eleven problems the two AIs reported that no human flagged, only one was a genuine problem. That\u2019s a useful number to keep in mind: 91% of the AI-only problems in this study required either correction or dismissal.<\/p>\n A case study like that, however, is limited. There was one video, two AIs (ChatGPT, Gemini), and one prompt. Replication helps with generalization.<\/p>\n To move beyond that first case study, we kept all experimental variables the same and conducted the same type of analysis on a similar video from a different usability study.<\/p>\n For this research, one UX researcher with decades of experience (Jim, the lead researcher from the previous study) reviewed a similar video from a previous usability benchmark study of online pet websites multiple times, creating a timeline of key events and a list of observed usability problems for this participant (referred to as Participant A; see Appendix A for the details of the timeline).<\/p>\n The characteristics the new video shared with the first one were:<\/p>\n Using the same prompt each time, we then ran the video four times each through the same two AIs used in the previous case study (ChatGPT-5.4 Thinking and Gemini 3 Flash Thinking).<\/p>\n So, in this study, we held constant the videos, the key elements of the prompt, and the AI versions\/settings\u2014variables that we eventually plan to vary. This time, as in the first case study, we only varied the type of analyst: human, ChatGPT, and Gemini.<\/p>\n During the previous usability benchmark study conducted at MeasuringU in 2019, participants used the PetSmart website to start the process of booking a grooming appointment in Glendale, CO, for a bath and full haircut for an English Springer Spaniel older than six months. The task was successfully completed if the participant found that specific grooming option and reported the listed price of $61. For the step-by-step details of the \u201chappy path\u201d to complete this task, see Appendix B.<\/p>\n The prompt we used for this study was:<\/p>\n During a usability test, the facilitator must keep track of participant behaviors as they navigate through tasks on a website, mobile app, software program, etc. We\u2019d like you to watch a video of a usability test where participants were asked to book a grooming reservation for their dog. As you\u2019re watching, please look for problems the participant has while attempting to complete the task. For example, you can document the path users take, describe issues they encounter as well as what on the website might be causing problems. The task has been successfully completed if the participant finds the target service (\u201cBath & Full Haircut\u201d which costs $61; not \u201cBath & Full Haircut with FURminator\u201d which costs $74). If you understand these instructions, let me know and I\u2019ll drag the video in for you to review. Are you ready for the video?<\/em><\/p>\n<\/blockquote>\n This was based on the prompt we used for the OpenTable case study with slight modifications. The task details are necessarily different because there was only one correct choice for this new task (compared to many correct choices for the OpenTable task), so we specified the end task details required for successful completion.<\/p>\n Even though the participant successfully completed the task in the video, a total of ten usability problems were identified by the AIs and the human researcher.<\/p>\n Usability problems aren\u2019t like observing a visual defect in a product. They require judgement, so it\u2019s worth digging into what these problems are because they\u2019re at the crux of how AI may or may not be able to effectively emulate UX researcher judgement (for now). So, let\u2019s dig into what we saw.<\/p>\n Participant A started with a few clicks not on the happy path, but on the third click got the task started, so these two early clicks could be considered minor usability problems that were quickly corrected.<\/p>\n There were two more impactful usability problems, both associated with breed selection (Figure 1). First, there was no visible indication that the dropdown list could be filtered by typing over the word \u201cbreed\u201d; the participant scrolled through the entire list. Second, the order of presentation of the breeds in the dropdown list was alphabetically inconsistent (e.g., \u201cEnglish Toy Spaniel\u201d started with E; \u201cSpringer Spaniel \u2013 English\u201d started with S). Despite this, the participant successfully completed the task, just not on the most efficient path.<\/p>\n a:<\/strong> Breed list at boundary of D and E\u2014\u201dEnglish Toy Spaniel\u201d but no English Springer Spaniel<\/p>\n b<\/strong>: Location of \u201cSpringer Spaniel \u2013 English\u201d<\/p>\n Figure 1: <\/strong>Participant A\u2019s usability issues with breed selection.<\/p>\n Table 1 shows the four problems discovered by the human researcher (three of which were also identified by the AIs) and the six additional problems reported only by the AIs. We looked to see if the problems identified only by the AIs were false alarms (an event happened but not really a usability problem) or hallucinations (the event just didn\u2019t happen).<\/p>\n The good news was that none of the six problems were hallucinations. The bad news is that they were all determined to be false alarms. Either the participant never actually noticed the issue flagged by the AI or wasn\u2019t affected by it (e.g., the location prompt, the below-the-fold item), or it was just normal, expected system behavior rather than a flaw (e.g., the ZIP search returning multiple locations, the menu reloading).<\/p>\n![\"Feature](\"https:\/\/measuringu.com\/wp-content\/uploads\/2026\/08\/081126-FeatureImage-1-scaled.jpg\")
<\/a>If AI can code problems<\/a> and potentially moderate a structured interview<\/a>, can it also reliably find usability issues from watching a video?<\/p>\nExperimental Design: One Human Researcher, Two AIs, and One Video<\/span><\/h2>\n
\n
The Task<\/span><\/h3>\n
The Prompt<\/span><\/h3>\n
\n
Major Findings<\/span><\/h2>\n
![\"Image](\"https:\/\/measuringu.com\/wp-content\/uploads\/2026\/08\/081126-F1a.png\")
<\/a><\/p>\n![\"Breed](\"https:\/\/measuringu.com\/wp-content\/uploads\/2026\/08\/081126-F1b.png\")
<\/a><\/p>\nSome Agreement Between Human and AI, No Hallucinations, Six False Alarms<\/span><\/h3>\n
![\"Venn](\"https:\/\/measuringu.com\/wp-content\/uploads\/2026\/08\/081126-F2.png\")
<\/a><\/p>\n![\"Venn](\"https:\/\/measuringu.com\/wp-content\/uploads\/2026\/08\/081126-F3.png\")
<\/a><\/p>\n![\"\"](\"https:\/\/measuringu.com\/wp-content\/uploads\/2026\/08\/081126-Figure-1-1024x525.jpg\")
<\/a><\/p>\n