We\u2019ve seen that AI can do lower-level assistant work reasonably well (e.g., coding comments<\/a>). But when it comes to high-level \u201canalyst\u201d tasks like discovering usability problems, we\u2019ve seen mixed results so far.<\/p>\n There is a lot of literature on the challenges of consistently and effectively discovering usability problems<\/a>. And that\u2019s for humans! We need a similar thorough analysis when comparing AI to human performance. So, we\u2019re doing that, one video<\/a> at a time.<\/p>\n So far, we\u2019ve reviewed two videos. Each has a participant attempting a task. In one, a participant successfully made\u00a0a dinner reservation<\/a> with OpenTable. In the second video, a different participant successfully booked\u00a0a dog grooming session<\/a> with PetSmart.<\/p>\n To identify lists of human-identified usability problems, we had four researchers independently review the first video and one researcher review the second video. We then had ChatGPT and Gemini analyze the same videos to identify usability problems.<\/p>\n We found similar results with both videos. Most metrics comparing the overlap of usability problems were within ten percentage points of each other (e.g., 40% vs. 45% of human-identified problems found by AI). The one real divergence was in the type of AI-only errors: the PetSmart video had more false alarms (60% vs. 35%) but zero hallucinations (compared to three in the OpenTable case study).<\/p>\n In both case studies, participants encountered some usability issues but were ultimately successful. In this article, we extend the scope of the case studies to a PetSmart video in which the respondent was ultimately unsuccessful.<\/p>\n Repeating our method from the previous case study, one UX researcher with decades of experience reviewed a similar video from a previous usability benchmark study of online pet websites multiple times, creating a timeline of key events and a list of observed usability problems for this participant (referred to as Participant B in this article; see Appendix A for the details of the timeline).<\/p>\n The characteristics the new video shared with the first one were:<\/p>\n Using the same prompt each time, we ran the video four times through the same two AIs used in the previous case study (ChatGPT-5.4 Thinking and Gemini 3 Flash Thinking).<\/p>\n In this study, we held constant the key elements of the prompt and the AI versions\/settings (all variables that we eventually plan to vary). This time, as in the first two case studies, we only varied the type of analyst: human, ChatGPT, and Gemini.<\/p>\n During a usability benchmark study MeasuringU conducted in 2019, participants used the PetSmart website to start the process of booking a grooming appointment in Glendale, CO, for a bath and full haircut for a one-year-old English Springer Spaniel. The task was successfully completed if the participant found that specific grooming option and reported the listed price of $61. For the step-by-step details of the \u201chappy\u201d path to complete this task, see Appendix B.<\/p>\n The prompt we used for this study was:<\/p>\n During a usability test, the facilitator must keep track of participant behaviors as they navigate through tasks on a website, mobile app, software program, etc. We\u2019d like you to watch a video of a usability test where participants were asked to book a grooming reservation for their dog. As you\u2019re watching, please look for problems the participant has while attempting to complete the task. For example, you can document the path users take, describe issues they encounter as well as what on the website might be causing problems. The task has been successfully completed if the participant finds the target service (\u201cBath & Full Haircut\u201d which costs $61; not \u201cBath & Full Haircut with FURminator\u201d which costs $74). If you understand these instructions, let me know and I\u2019ll drag the video in for you to review. Are you ready for the video?<\/em><\/p>\n A total of 17 unique usability problems were identified by the AIs and the human researcher.<\/p>\n Usability problems aren\u2019t like observing a visual defect in a product. They require judgment, so it\u2019s worth digging into what these problems are, because such judgments are at the crux of how AI may or may not be able to effectively emulate UX researchers (for now). So, let\u2019s follow this path to task failure.<\/p>\n Participant B started on the correct path by clicking Pet Services, then Grooming Salon. Then things rapidly went downhill.<\/p>\n Figure 1 shows a particularly problematic sequence of events that caused numerous downstream usability problems due to a critical latency issue.<\/p>\n![\"Feature](\"https:\/\/measuringu.com\/wp-content\/uploads\/2026\/10\/100626-FeatureImage1.jpg\")
<\/a>Can AI replace UX researchers? Well, UX researchers do a lot (they certainly do at MeasuringU).<\/p>\nExperimental Design: One Human Researcher, Two AIs, One Video<\/h2>\n
\n
The Task<\/h3>\n
The Prompt<\/h3>\n
Major Findings<\/h2>\n