{"id":187520,"date":"2025-04-16T06:56:00","date_gmt":"2025-04-16T06:56:00","guid":{"rendered":"https:\/\/peraltafinancing.com\/analytics\/how-stable-is-the-supr-qm-after-8-years-measuringu\/"},"modified":"2025-04-16T06:56:00","modified_gmt":"2025-04-16T06:56:00","slug":"how-stable-is-the-supr-qm-after-8-years-measuringu","status":"publish","type":"post","link":"https:\/\/fivemor.com\/?p=187520","title":{"rendered":"How Stable is the SUPR-Qm After 8 Years? \u2013 MeasuringU"},"content":{"rendered":"


\n<\/p>\n

\n

\"feature<\/a>We spend a lot of time using our phones every day<\/a>, but we hardly make any calls. A lot of our time is spent interacting with various mobile apps<\/a>. Mobile apps are different enough from websites and hardware products for us to believe they deserve their own UX measurement instrument. We aren\u2019t the only ones who think so, as there have been previous efforts<\/a> to measure the mobile app experience using standardized questionnaires.<\/p>\n

Building on prior work<\/a>, we developed the SUPR-Qm\u00ae<\/sup> in 2017 to focus on the quality of the mobile app experience.<\/p>\n

But a lot has changed since 2017, and mobile apps aren\u2019t the same as they were. How stable is the SUPR-Qm model we generated over eight years ago? In this article, we review our latest results from a large-scale data collection effort to assess the stability of the SUPR-Qm.<\/p>\n

Between February 2019 and May 2023, we collected retrospective mobile app data from 4,149 participants using our MUiQ\u00ae<\/sup> platform<\/a>. The participants came from a U.S.-based panel with a mix of gender (48% male, 50% female) and age (42% less than 30 years old, and 58% 30 years or older). The data collection effort was part of our rolling SUPR-Q industry surveys, so we also asked participants about their website usage and items specific to an industry (such as dating, pets, and office supplies). In all, we collected ratings for 155 mobile apps across 23 industries.<\/p>\n

In these surveys, all SUPR-Qm items had initially been randomly assigned to one of two eight-item grids and then randomized within those grids for each participant.<\/p>\n

We used Rasch analysis<\/a>, part of the Item Response Theory (IRT) methods, to analyze the new data for comparison with the original analysis of the SUPR-Qm. Most questionnaires in UX research (including the SUPR-Q) were developed using Classical Test Theory (CTT). But when the measurement goal is a unidimensional measure, which can have a large number of items focusing on the measurement of individual differences, the more appropriate method is Item Response Theory.<\/p>\n

Interpreting a Wright Map<\/h2>\n

One key output of Rasch analysis is a Wright map (also called an item-person map), which places the difficulty of the items (how hard it was for respondents to agree with them) on the same measurement scale as the participants\u2019 ratings. Each # represents the number of participants on the left side, and the label shows each item\u2019s location on the right side of the map.<\/p>\n

A Wright map is organized as two vertical histograms with the items and respondents (persons) arranged from easiest (most likely to agree) on the bottom to most difficult (least likely to agree) on the top. For example, most participants agreed or strongly agreed (4s and 5s) to the items \u201cEasy\u201d and \u201cEasyNav.\u201d In contrast, few participants highly rated apps in the item \u201cAppBest.\u201d<\/p>\n

On the left side, the Wright map shows the mean (M) and two standard deviation points (S = one SD and T = two SD) for the measurement of participants\u2019 tendency to agree. On the right side of the map, the mean difficulty of the items (M) and two standard deviation points (S = one SD and T = two SD) for the items are shown.<\/p>\n

Original and Replication Wright Maps<\/h2>\n

Figure 1 shows the results from the original analysis conducted in 2017 on the left and the results from this replication study on the right. The item wording and the mapping between the labels that appear in Figure 1 also appear in Table 1.<\/p>\n\n\n\n
\"Original<\/a><\/td>\n\"Replication<\/a><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

\u00a0<\/p>\n

Figure 1<\/strong>: Original (left) and replication (right) Wright maps of the SUPR-Qm.<\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Original Label<\/th>\nReplication Label<\/th>\nItem Wording<\/th>\n<\/tr>\n<\/thead>\n
CantLiveWo<\/td>\nCantLiveWithout<\/td>\nI can\u2019t live without this app on my phone.<\/td>\n<\/tr>\n
AppBest<\/td>\nAppBest<\/td>\nThe app is the best app I\u2019ve ever used.<\/td>\n<\/tr>\n
CantImagineBetter<\/td>\nCantImagineBetter<\/td>\nI can\u2019t imagine a better app than this one.<\/td>\n<\/tr>\n
NeverDelete<\/td>\nNeverDelete<\/td>\nI would never delete the app.<\/td>\n<\/tr>\n
EveryoneHave<\/td>\nEveryoneHave<\/td>\nEveryone should have the app.<\/td>\n<\/tr>\n
Discover<\/td>\nDiscover<\/td>\nI like discovering new features on the app.<\/td>\n<\/tr>\n
AllEverWant<\/td>\nAllEverWant<\/td>\nThe app has all the features and functions you could ever want.<\/td>\n<\/tr>\n
Delightful<\/td>\nDelightful<\/td>\nThe app is delightful.<\/td>\n<\/tr>\n
Integrates<\/td>\nIntegrates<\/td>\nThe app integrates well with the other features of my mobile phone.<\/td>\n<\/tr>\n
UseFreq<\/td>\nUseFreq<\/td>\nI like to use the app frequently.<\/td>\n<\/tr>\n
DefFuture<\/td>\nDefFuture<\/td>\nI will definitely use this app many times in the future.<\/td>\n<\/tr>\n
AppAttractive<\/td>\nAppAttractive<\/td>\nI find the app to be attractive.<\/td>\n<\/tr>\n
FindInfo<\/td>\nFindInfo<\/td>\nThe design of this app makes it easy for me to find the information I\u2019m looking for.<\/td>\n<\/tr>\n
AppMeetsNeeds<\/td>\nAppMeetsNeeds<\/td>\nThe app\u2019s features meet my needs.<\/td>\n<\/tr>\n
EasyNav<\/td>\nEasyNav<\/td>\nIt is easy to navigate within the app.<\/td>\n<\/tr>\n
Easy<\/td>\nEasy<\/td>\nThe app is easy to use.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

<\/p>\n

Table 1:<\/strong> Mapping between Figure 1 labels and item wording.<\/p>\n

As shown in the left Wright map in Figure 1, the alignment of the original 16 items on the logit scale ranged between \u22122 (Easy and EasyNav) on the very easy end of the scale and +2 (AppBest and CantLiveWo) on the very difficult end. There were three clusters of items in between those extremes:<\/p>\n