We\u2019re celebrating our 20th anniversary at MeasuringU (2005\u20132025). For us, it\u2019s less about popping the champagne and more about reflecting on how the UX industry has changed and how we have helped shape some of that change through measurement.<\/p>\n Some things have changed a lot, while others haven\u2019t. We looked back at key moments, reviewing influential publications and events to describe the story of how our company and the industry have evolved.<\/p>\n We\u2019ve divided the MeasuringU timeline into three epochs. In each epoch, we briefly describe the industry trends, our company milestones, and the state of the art, including our contributions to enduring UX topics of sample size estimation, online UX tools, usability testing, UX data analysis, and UX metrics.<\/p>\n In our first article<\/a> on the history of MeasuringU, we covered the foundational years from 1998 to 2008. This period included the founding of the company, the launching of the website, and the first of many collaborations and publications between Jeff Sauro and Jim Lewis, including the recommendation of adjusting binary confidence intervals for more accurate analysis.<\/p>\n In our second article<\/a>, we reviewed how the expansion of UX research teams led to the need for well-documented ways of UX measurement. This period culminated in several books, including A Practical Guide to Measuring Usability<\/em>, Quantifying the User Experience<\/em>, and Customer Analytics for Dummies<\/em>.<\/p>\n In this final article, we cover the years from 2016 to the present, a period in which we made major investments in our research platform (MUiQ\u00ae<\/sup>), grew our research team, and developed new research in UX methods and metrics.<\/p>\n The world economy saw moderate but steady growth while the U.S. experienced one of its longest periods of economic expansion. The U.S. unemployment rate dropped to 3.5% in 2019, but wage growth remained slow. Companies began to invest heavily in AI to reduce the need for low-skilled labor.<\/p>\n Major trends in UX research were affected by an increased use of web analytics, automated usability testing platforms, conversational interactions (e.g., chatbots, voice agents), and standardized UX metrics (e.g., SUS, NPS, SUPR-Q).<\/p>\n In 2016, we released the second edition of Quantifying the User Experience<\/em><\/a> with a new chapter on correlation, regression, and analysis of variance. MeasuringU outgrew its original space and built a new office with custom labs, including one-way mirrors<\/a> (Figure 1). While one-way mirrors were becoming less common as much UX research moved online, they proved invaluable a few years later when testing new physical products, including AR and VR devices.<\/p>\n Figure 1:<\/strong> MeasuringU lab with a one-way mirror.<\/p>\n As more UX teams began to measure their user experience with benchmarking, we got more calls to help with planning and analysis. Across these requests, we saw a lot of the same questions and challenges related to benchmarking. What sample size, what metrics, how many tasks? How do you assess task completion rates when you can\u2019t directly observe the participants? To answer those questions, in 2018 Jeff wrote Benchmarking the User Experience<\/a><\/em> (Figure 2), which was the first of a series of books by the newly created MeasuringU Press.<\/p>\n Figure 2:<\/strong> Cover of Benchmarking the User Experience<\/em>.<\/p>\n We continued to collect more SUPR-Q\u00ae<\/sup> data, periodically updating the percentiles, and we developed a version focused on mobile experiences, the SUPR-Qm\u00ae<\/sup> (more on that below). We published three papers on the SUS and Jim\u2019s book, Using the PSSUQ and CSUQ in User Experience Research and Practice<\/em><\/a>. We also developed and published the PURE usability inspection method (Practical Usability Rating by Experts<\/a>) and a more scientific approach to UX maturity measurement<\/a>.<\/p>\n Dissatisfied with the existing online UX research platforms, we made major investments in MUiQ to extend its capabilities from analysis to online data collection. The MUiQ platform became an integral part of our annual UX measurement bootcamps.<\/p>\n Starting in 2017, we had over a million views of our blog articles a year, demonstrating our increasing reach to the UX community through this educational channel.<\/p>\n With the release of the second edition of Quantifying the User Experience<\/em><\/a>, we added sample size estimation methods for correlations and simple regression. In Chapter 6 of Benchmarking the User Experience<\/em><\/a>, Jeff provided numerous tables to simplify sample size planning (many of which are also available in MeasuringU blog articles published during this time).<\/p>\n During this period, we conducted many studies that pushed the limits of existing online UX research platforms. In 2016 we enhanced the MUiQ platform, originally just an analytical platform, to enable UX data collection, and in 2017 we introduced the first licensable version. After that, we continued to enhance the MUiQ platform with features like website intercepts and URL-based tasks with screen recording.<\/p>\n Because finding and fixing usability problems is the cornerstone of usability testing, we published blog articles in 2016 that discussed seven ways to uncover usability problems<\/a>, how to assign severity ratings<\/a>, and steps for conducting an effective expert review<\/a>. We focused on facilitation and prototype testing in 2017<\/a>. Usability testing topics in 2018<\/a> included the cost of usability tests, how to build a dedicated lab, and further investigation of the evaluator effect.<\/p>\n At CHI 2016, Jeff (in collaboration with Christian Rohrer and other UX researchers from Intel Security) published the first description of PURE<\/a> (Practical Usability Rating by Experts), a new usability inspection method that included analysis of data across multiple expert inspectors (Figure 3). They validated the method with data from usability tests on three software products. Following up on this research, Jeff wrote additional blog articles in 2017<\/a> and 2018<\/a> on the PURE method, including an overview<\/a>, practical tips<\/a>, and predicting UX metrics<\/a>.<\/p>\n Figure 3:<\/strong> Example PURE scorecard.<\/p>\n In this period, we conducted significant research on the SUPR-Qm, SUS, and NPS.<\/p>\n In 2017, Jeff and Paree Zarolia published \u201cSUPR-Qm: A Questionnaire to Measure the Mobile App Experience<\/a>.\u201d The paper described the development of the SUPR-Qm, a 16-item instrument that assesses a user\u2019s experience of a mobile application. They used Rasch analysis to assess the psychometric properties of items collected from four independent surveys (N<\/em> = 1,046) with ratings on 174 unique apps. The final version had very high reliability, high concurrent validity, and good predictive validity (its scores significantly correlated with the number of app reviews in the Google Play Store and Apple\u2019s App Store).<\/p>\n Using their very large dataset of completed SUS questionnaires, in 2017 and 2018 Jeff and Jim published three papers in the Journal of User Experience:<\/em> \u201cRevisiting the Factor Structure of the System Usability Scale<\/a>,\u201d \u201cCan I Leave This One Out? The Effect of Dropping an Item from the SUS<\/a>,\u201d and \u201cItem Benchmarks for the System Usability Scale<\/a>.\u201d The first paper presented evidence that an earlier finding of Usability and Learnability subscales for the SUS were artifacts caused by the mixed tone of the SUS items. The second paper demonstrated that removing any one item from the SUS has little effect on its score as long as it\u2019s properly interpolated to a 0\u2013100-point scale. The third paper presented benchmarks for the interpretation of individual SUS items. Jeff also published one of our most popular blog articles on five ways to interpret the SUS<\/a>.<\/p>\n Between 2016 and 2019, the NPS maintained significant popularity as a key metric for assessing customer loyalty (nearly half the firms surveyed by Forrester in 2019 used the NPS). During this time, a backlash against the NPS emerged in the UX community, with some legitimate complaints about corporate misuse but also some over-the-top polemics claiming the NPS was not only useless but harmful and should be replaced by a generally unspecified \u201csomething else\u201d without any proof that the \u201csomething else\u201d would be better.<\/p>\n A quick review of the NPS: People rate their likelihood to recommend on a 0\u201310 point scale and are classified as detractors (0\u20136), passives (7\u20138), or promoters (9\u201310). The NPS is the percentage of promoters minus the percentage of detractors.<\/p>\n As a backlash to its popularity, the NPS controversies began to get loud, and the first of many screaming screeds proclaiming the uselessness of the NPS started to appear. We were never necessarily strong advocates of using the NPS, but we generally saw the value in having corporate commitment to measuring likelihood-to-recommend, which is the basis of the NPS. So, it was a bit surprising to read these strong condemnations from prominent UX voices like Jared Spool.<\/p>\n Jeff was working at Intuit in 2003 when the NPS was introduced. The founder and former CEO had come from Bayne and quickly introduced the NPS at Intuit. The software division Jeff was working in at the time was in nonprofit and state\/local government accounting software. Despite working on products where recommendation behavior seemed unlikely, Jeff\u2019s team got an edict from upper management to start using the NPS. The content of the question didn\u2019t seem right\u2014who would recommend government accounting software to friends? But, to Jeff\u2019s surprise, it correlated highly with the measures of satisfaction and loyalty his team was using, and they were able to slightly modify the wording so it made sense in their research context (e.g., \u201cIf a friend or colleague asked about your experience with government accounting software, how likely would you be to recommend the xyz product?\u201d).<\/p>\n Upon even mild scrutiny, some of the basic reasons provided for why the NPS was harmful were easily dismissed. We realized this was less about data than about diatribes (Figure 4). Some people simply saw (and continue to see) an opportunity to make a name for themselves in social media by signaling that they were in the anti-NPS tribe.<\/p>\n Figure 4:<\/strong> Graphic from MeasuringU\u2019s three webinars<\/a> on NPS Data, Drivers, and Diatribes.<\/p>\n This led to a lot of MeasuringU research (primary and secondary) into claims about the NPS claims. For example, people might reference an article as \u201cproof\u201d that the NPS was debunked, but careful reading of the article showed the data did not support that conclusion. Several articles showed that the NPS wasn\u2019t necessarily superior, but it performed about as well as other similar metrics (e.g., customer satisfaction). It was clear that more data were needed.<\/p>\n To address that need for more data, from 2018<\/a> to 2019<\/a> Jeff published 13 blog articles on the NPS. Some key findings from that body of research are:<\/p>\n The COVID-19 pandemic led to significant economic contraction from 2020\u20132022, including spikes in unemployment worldwide. With the rollout of vaccines and easing of restrictions, economies began to recover in 2022 and experienced robust growth rates, particularly in sectors like technology and healthcare, though with the specter of rising inflation. The pandemic accelerated the adoption of digital technologies, leading to growth in e-commerce, remote work, and digital services.<\/p>\n In UX, designers and researchers began exploring how to incorporate new generative AI tools into their work processes, striving to understand what would and wouldn\u2019t improve their efficiency and quality, given the power and limitations of this new technology. This period also saw significant advancement in remote unmoderated testing.<\/p>\n From the beginning, Jim had been MeasuringU adjacent while working at IBM, collaborating with Jeff on publishing research and books, with some joint teaching of tutorials and workshops at major conferences (e.g., HCII, CHI, UXPA). After 40 years at IBM, Jim retired on December 31, 2019, remained retired for one day, then joined MeasuringU on January 2, 2020. During this period, he and Jeff published six peer-reviewed journal articles, a chapter in the 5th<\/sup> edition of Handbook of Human Factors and Ergonomics<\/em> (Usability and User Experience: Design and Evaluation<\/a>), and, in 2024, published their latest book, Surveying the User Experience<\/em><\/a> (Figure 5).<\/p>\n Figure 5:<\/strong> Cover of Surveying the User Experience<\/em>.<\/p>\n The pandemic put an end to our in-person training, including our popular UX measurement bootcamp. We shifted to teaching over Zoom in 2020 (extending our training to a worldwide audience) and began creating courses for our MeasuringUniversity<\/a>\u00ae<\/sup> online educational service. Our current bootcamps have a flipped-classroom format<\/a>, reducing the time spent on live lectures for more dynamic interaction during the scheduled live sessions.<\/p>\n Over this period, our org chart really expanded. The research team doubled to ten researchers and managers, the software development team grew from two to ten developers and managers, and the customer engagement, research operations, and project management staff grew, too.<\/p>\n We published 37 industry reports and continued investing in the development of standardized UX metrics and methods (including careful incorporation of generative AI) and our MUiQ platform (details below).<\/p>\n In this time period, we analyzed our historical data to estimate typical standard deviations for rating scales and standardized UX questionnaires, then used those estimates to create sample size tables to make it easier for UX researchers to do fundamental sample size estimation (for the SUS in 2022<\/a> and more generally in 2023<\/a>). We also worked out sample size estimation methods for the NPS<\/a>. Other key blog articles on this topic in 2024 were that one sample size does not fit all usability studies<\/a> and an analysis of what you get with specific sample sizes<\/a> in UX problem discovery studies.<\/p>\n To better serve our clients, we began licensing an enterprise-ready version of the MUiQ. We expanded its feature set to include mobile studies, click tests, and most recently, moderated studies.<\/p>\n In 2022<\/a> and 2023<\/a>, we published 12 blog articles about think-aloud usability testing<\/a>, both in-lab and remote unmoderated. Some key findings were that asking participants to think aloud increases dropout rates<\/a> and task times<\/a> but leads to the discovery of more problems<\/a>.<\/p>\n Using the new click-test capabilities of MUiQ<\/a>, we found, with some caveats, that click tests were reasonably predictive of live site<\/a> and product page<\/a> clicks.<\/p>\n Our initial exploration in 2023 of using ChatGPT-4 to support usability testing found variability in multiple AI classifications of user comments, but only slightly lower interrater reliabilities<\/a> between human coders and ChatGPT than between human coders alone. In 2024, we compared ChatGPT-4\u2019s ability to sort items into groups<\/a> with the outputs of multiple human researchers, finding moderate to substantial interrater reliability. In a separate experiment, we found that ChatGPT-4 was not good at estimating how well humans find items in a tree test<\/a> but was reasonably good at predicting human ease ratings of this task.<\/p>\n An outstanding problem for the NPS was a lack of statistical methods for estimating confidence intervals, comparing scores with a benchmark, or comparing two scores. In 2021, we published nine blog articles<\/a> describing how to do these computations. We also published articles on how to combine significance testing and confidence interval estimation to go beyond simple statistical significance to the evaluation of practical significance<\/a>.<\/p>\n In 2020<\/a> and 2021<\/a>, we made major investments in studying the effects of item formats on the rating behaviors of participants in UX research studies. We published 19 blog articles on various topics like labeling scale points, using sliders instead of radio buttons, and using stars or emojis instead of scale numbers. Most of these alterations had little effect on respondent behavior<\/a>. We researched SEQ\u00ae<\/sup> variants<\/a> and published a new way to interpret SEQ scores<\/a> based on an adjective scale.<\/p>\n We published \u201cMeasuring User Experience With 3, 5, 7, or 11 Points: Does It Matter?<\/a>\u201d in Human Factors<\/em> (answer\u2014don\u2019t use three points; otherwise it doesn\u2019t much matter) and \u201cComparison of Select-All-That-Apply Items with Yes\/No Forced Choice Items<\/a>\u201d in the Journal of User Experience <\/em>(SATA is better). In 2024, we published a new standardized questionnaire designed to assess the perceived clutter of websites<\/a>.<\/p>\n During this time, we developed three especially useful new standardized UX metrics: UX-Lite, TAC-10, and SUPR-Qm V2.<\/p>\n The UX-Lite\u00ae<\/sup> has its roots in the UMUX-LITE<\/a>, which was itself derived in 2013 from the UMUX (2010). It\u2019s a two-item questionnaire that is essentially a miniature version of the Technology Acceptance Model (TAM), assessing the perceived ease-of-use and perceived usefulness of products and services, and it is becoming an increasingly popular UX metric. From 2020 to 2024, we published 15 articles on the UX-Lite, many of which explored different ways to phrase the \u201cusefulness\u201d item because its original wording was overly complex. In addition to demonstrating the reliability and validity of the UX-Lite, it has also proved to be useful in regression and structural equation modeling of higher-level outcome metrics like ratings of overall experience, likelihood-to-recommend and likelihood to reuse, and actual user behaviors (2020: \u201cPerceived Usability and the Modified Technology Acceptance Model<\/a>,\u201d 2023: \u201cEffect of Perceived Ease of Use and Usefulness on UX and Behavioral Outcomes<\/a>\u201c).<\/p>\n We presented the TAC-10, a new measure of tech savviness, at UXPA 2024 based on research conducted at MeasuringU from 2015 through 2023. We published six blog articles in 2023<\/a> detailing its development, including why there was a need for a measure of tech savviness<\/a> in UX research (to enable discrimination of interface and participant characteristics when analyzing UX data) and how to use the TAC-10 to classify participants into different levels of tech savviness<\/a>.<\/p>\n In 2025, we published the second version of the SUPR-Qm<\/a> in the Journal of User Experience.<\/em> In this paper, we described our analysis of large-sample datasets to (1) replicate the Rasch model reported in the initial publication of the SUPR-Qm in 2017, (2) identify redundant items from the original 16-item SUPR-Qm that could be removed to create a streamlined five-item version (SUPR-Qm V2), (3) demonstrate the stability of the SUPR-Qm and SUPR-Qm V2 over two multiyear periods in which data were collected for 155 mobile apps across 23 industries, and (4) develop interpretive norms (including curved grading scales) for the SUPR-Qm and SUPR-Qm V2. This new version enhances the usefulness of the SUPR-Qm for UX practitioners and researchers who need a standardized questionnaire that provides a quick five-item measure of the UX of mobile apps. The SUPR-Qm V2 is easy to interpret with norms that should remain stable for many years.<\/p>\n Table 1 summarizes the key topics for the MeasuringU timeline from its genesis to the present day.<\/p>\n![\"Feature](\"https:\/\/measuringu.com\/wp-content\/uploads\/2025\/05\/052725-FeatureImage-1.png\")
<\/a>Who cares what happened 15 or 20 years ago? While technology changes fast, some of the most important questions in UX research are more enduring. Preparing for the future means understanding the past.<\/p>\nIndustry Trends<\/h2>\n
Moderate but Steady Economic Growth<\/em><\/h3>\n
Company Milestones<\/h2>\n
The MeasuringU Press Keeps Printing<\/em><\/h3>\n
<\/a><\/p>\n<\/a><\/p>\nSample Size<\/h2>\n
Sample Sizes in Practice<\/em><\/h3>\n
UX Online Tools<\/h2>\n
MUiQ Gets Its First Licensed Customers<\/em><\/h3>\n
Usability Testing<\/h2>\n
Facilitation and Facilities<\/em><\/h3>\n
Data Analysis<\/h2>\n
PURE Fun<\/em><\/h3>\n
<\/a><\/p>\nUX Metrics<\/h2>\n
Innovating and Separating the Data from the Diatribes<\/em><\/h3>\n
SUPR-Qm<\/h4>\n
SUS<\/h4>\n
NPS<\/h4>\n
<\/a><\/p>\nIndustry Trends<\/h2>\n
COVID Impacts Led to Zero Interest Rate, Fueling Growth in Tech and UX<\/em><\/h3>\n
Company Milestones<\/h2>\n
Jim Lewis Joins MeasuringU<\/em><\/h3>\n
<\/a><\/p>\nSample Size<\/h2>\n
Rating Scales Need Only about \u00bc the Sample Size as Binary Metrics<\/em><\/h3>\n
UX Online Tools<\/h2>\n
MUiQ Goes Moderated<\/em><\/h3>\n
Usability Testing<\/h2>\n
Thinking Aloud Does Affect Behavior<\/em><\/h3>\n
Data Analysis<\/h2>\n
Statistical Analysis of the NPS<\/em><\/h3>\n
UX Metrics<\/h2>\n
From the SUS to the UX Lite: UX Metrics that Span the Generations<\/em><\/h3>\n
UX-Lite<\/h4>\n
TAC-10<\/h4>\n
SUPR-Qm V2<\/h4>\n