That\u2019s a good question. It\u2019d be good to have data to help answer that. Instead, we have claims from vendors selling AI moderator technology. That\u2019s not nothing, but it\u2019s hardly the objective data you\u2019d want before you retire your research team from moderating.<\/p>\n When we encounter interesting claims like that, we generally take the following approach:<\/p>\n We completed step two in our previous article<\/a>, where we reviewed the published literature. There wasn\u2019t much literature to review, and some of the findings may be a bit dated. But even newer work is lacking because most studies compared AI moderators to surveys or static follow-up questions, not to a live human moderator. Only two studies actually put an AI moderator head-to-head against a human one.<\/p>\n One was a small classroom-style study (Wuttke et al., 2025<\/a>). Student pairs each ran one AI-led and one human-led interview on politics and democracy, netting just five AI-led and six human-led sessions, with students standing in for trained moderators.<\/p>\n The other (Zhu et al., 2026<\/a>) was a properly powered randomized control trial (RCT) with 30 sessions per condition, pitting an agentic voice moderator against a human moderator in think-aloud usability testing of a note-taking app. It\u2019s a stronger study, but think-aloud moderation is a lower-demand job than in-depth-interview moderation. In a think-aloud session, the moderator\u2019s main task is prompting someone to keep narrating what they\u2019re already doing. It\u2019s not that far removed from unmoderated think-aloud, where there\u2019s no moderator at all, just a periodic \u201ckeep talking\u201d nudge or \u201ctell me more.\u201d A semi-structured or in-depth interview asks much more of a moderator: building rapport<\/a> from a cold start, deciding in real time whether an offhand comment is worth five extra minutes, and steering a conversation that has no task to anchor it.<\/p>\n In the Zhu et al. study, they found no difference in the number of words participants produced, but the AI<\/strong> moderator talked more, faster, and more often <\/strong>during the session itself. The AI moderator also scored significantly worse than the human moderator on context-aware follow-up and on building trust, even though the two were indistinguishable on procedural adherence.<\/p>\n That review left us with five questions that the literature couldn\u2019t settle:<\/p>\n Each of those questions needs its own analysis, and all require more data. To help us (and the industry) answer these questions, we built our own AI moderator and ran a controlled experiment. In this article, we\u2019ll cover the setup of that experiment and address the first question of whether participants say less, more, or about the same to AI moderators as human moderators. Of course, speaking more isn\u2019t necessarily better, but if humans systematically say LESS to an AI moderator, that would create fewer opportunities for uncovering insights.<\/p>\n To test an AI moderator, you need a few things: an AI moderator, human moderators, a study goal and discussion guide, and human participants. The last three are staples of UX research; AI moderators are the new additions.<\/p>\n There are AI moderators on the market, but we wanted to build our own for two reasons. First, we wanted to understand how prompting and programming decisions affect the output around probing decisions, especially as models change. Second, we wanted the AI moderator to look as realistic as possible (and uncannily similar to the real human moderators in the study). We built our AI moderator using an AI agent and created photorealistic avatars with voices of two of our researchers (see Video 1).<\/p>\n![\"Feature](\"https:\/\/measuringu.com\/wp-content\/uploads\/2026\/09\/091526-FeatureImage1.jpg\")
<\/a>\u201cAre AI moderators as good as human moderators?\u201d<\/p>\n\n
\n
Experimental Study Setup<\/h2>\n
AI Moderator<\/h3>\n