Measuring the user experience means assessing both what people do and what people think (a mix of action and attitudinal<\/a> metrics). For attitudes, standardized questionnaires like the UX-Lite\u00ae<\/sup> are a good place to start, but they\u2019re not diagnostic on their own and won\u2019t tell you why Trust is low or what\u2019s driving Anxiety.<\/p>\n Building a standardized questionnaire involves assessing validity, reliability, and sensitivity<\/a>. Practically, this involves six broad steps:<\/p>\n We did step one in our previous article<\/a>, where we identified 34 candidate items across six constructs (AI Productivity, AI Trust, AI Dependency, AI Anxiety, AI Personification, and Early Adoption) based on our reading and experience with these products.<\/p>\n In this article, we\u2019ll cover the next five steps: collecting data from real users of real products and seeing how well those 34 items perform.<\/p>\n In May 2026, we conducted a retrospective study of four AI-based chat software products<\/a> with 420 U.S.-based panel participants. This study included the metrics we typically collect in our\u00a0standard UX and NPS study of consumer software<\/a>.<\/p>\n There was a roughly equal gender split (52% female, 47% male). Respondents tended to be younger, with 65% under the age of 40. Participants were asked to reflect on their most recent experiences with the software and complete several existing questionnaires, including the\u00a0NPS<\/a>,\u00a0SUS<\/a>,\u00a0UX-Lite<\/a>, and\u00a0TAC-10<\/a>\u2122. In addition to these existing questionnaires, respondents completed 34 new items focused on different aspects of the UX of AI. The AI-based chat products and sample sizes were:<\/p>\n The sample sizes are modest but\u00a0adequate to establish baselines and identify medium-sized differences\u00a0<\/a>relative to each other and other software products we measure (e.g., \u00b1 5 for 0\u2013100-point rating scales). With a combined n<\/em> = 420, the sample size is also large enough to support advanced analyses<\/a> (e.g., factor analysis, regression analysis, reliability analysis, ANOVA).<\/p>\n We used the multivariate technique of factor analysis to determine how well the candidate items map to their intended construct. The output of a factor analysis is factor loadings<\/em>, numbers that range from \u22121 to +1, which are interpreted like a correlation. To review the factor loadings for all 34 items, see Appendix Figure 1.<\/p>\n All but one of the items (\u201cI often rely on AI chatbots to perform tasks that I would otherwise do myself\u201d) didn\u2019t map well to its intended construct of Trust (it loaded as high on Productivity), so this was a good candidate to exclude. The remaining 33 items strongly loaded on their intended constructs\u2014solid evidence of construct validity<\/strong>.<\/p>\n A common approach to item selection in the psychometric process of developing a standardized questionnaire is to retain the items with the highest loadings on their associated factors. In our current practice, we enhance that method by examining item means and beta weights from regression models with a key outcome variable.<\/p>\n While still paying attention to the magnitude of item loadings, we also try to select items that vary in their observed means to better differentiate low and high levels of the construct. We also created six regression models, one for each construct, to see by examining beta weights which items accounted for larger amounts of variation in a key outcome metric: the likelihood to continue using the product.<\/p>\n In the following sections, we present the three criteria for all the items for each target construct to guide the items we selected for inclusion in a final, streamlined questionnaire. For easier interpretation, the means in the tables were converted from their original five-point scale to a scale ranging from 0 to 100<\/a>. Our typical target for measuring a construct at this stage of questionnaire development is to select two to three items per construct with the goal of achieving scale reliability (measured with coefficient alpha) of at least 0.70 for each construct.<\/p>\n![\"\"](\"https:\/\/measuringu.com\/wp-content\/uploads\/2026\/08\/081926-FeatureImage-2.jpg\")
<\/a>There\u2019s nothing quite the same as chatting with AI. But it\u2019s just like every other interface in one important way: we experience it. And what we experience, we can measure.<\/p>\n\n
AI-Based Chat Software Benchmark Study<\/span><\/h2>\n
\n
Factor Analysis: Items (Almost) Perfectly Aligned with Target Constructs<\/span><\/h2>\n
Item Analyses: Selecting the Best Items for a Streamlined Questionnaire<\/span><\/h2>\n