{"id":240845,"date":"2025-05-14T05:17:54","date_gmt":"2025-05-14T05:17:54","guid":{"rendered":"https:\/\/peraltafinancing.com\/analytics\/what-happens-when-you-test-a-mobile-prototype-on-desktops-measuringu\/"},"modified":"2025-05-14T05:17:54","modified_gmt":"2025-05-14T05:17:54","slug":"what-happens-when-you-test-a-mobile-prototype-on-desktops-measuringu","status":"publish","type":"post","link":"https:\/\/fivemor.com\/?p=240845","title":{"rendered":"What Happens When You Test a Mobile Prototype on Desktops? \u2013 MeasuringU"},"content":{"rendered":"


\n<\/p>\n

\n

\"Feature<\/a>Early and often is not just advice for voting in Chicago<\/a>\u2014it\u2019s also one of the key principles for designing for a usable experience<\/a>.<\/p>\n

Testing an experience while it\u2019s still in its prototype stage allows you to find and fix problems before they become difficult and expensive to fix. User experiences with prototypes (even low fidelity ones) tend to be reasonable predictors<\/a> of the final experience.<\/p>\n

Not surprisingly, then, building and testing prototypes are common activities<\/a> in the development process<\/a> for websites, software, physical products, and mobile apps.<\/p>\n

With rapidly moving development timelines, designers often want to get feedback as soon as possible. When testing prototypes for mobile (websites or apps), it can often be easier and faster to recruit and test users who participate in online research with desktop computers or laptops. The two main reasons are that participants don\u2019t have to download an app if you want to record screens or clicks and that it\u2019s easier to display longer task instructions that remain visible on a desktop (e.g., using the MUiQ\u00ae<\/sup> platform\u2014see Figure 1).\u200b<\/p>\n

\"Desktop<\/a><\/p>\n

Figure 1<\/strong>: Desktop and mobile presentations of task instructions.<\/p>\n

Figure 1 shows the simultaneous presentation of the prototype, task description, and End Task button on a desktop screen in Panel 1. Panel 2 shows the task presentation modifications needed for mobile devices, with the task instructions presented in a pop-up over the prototype screen that participants can close by clicking \u201cProceed with the task\u201d and re-open by clicking Task Info in the lower-right corner of the screen (after which End Task is enabled).<\/p>\n

But how realistic is testing a mobile prototype on desktop? A few things to consider about desktops:<\/p>\n

    \n
  1. They have bigger screens than mobile devices.<\/li>\n
  2. You use a mouse rather than gestures such as tapping and swiping.<\/li>\n
  3. They lack haptic feedback (e.g., vibrations), which is common on mobile devices.<\/li>\n
  4. They can\u2019t mimic the contextual interactions of using mobile devices, such as walking, driving, or one-handed use.<\/li>\n<\/ol>\n

    So, while it may be easier to collect data on desktop, how realistic is that data relative to data collected on mobile devices?<\/p>\n

    There is surprisingly little published research comparing assessments of the same mobile application on desktops and mobile devices. In this article, we\u2019ll review what we found on published differences in testing as well as report on the findings from an experiment we conducted.<\/p>\n

    In 2015, we compared the UX of mobile versus desktop<\/a> presentations of seven live websites (n<\/em> = 3,740). We expected that the richer UX of a desktop would lead to higher SUPR-Q<\/a>\u00ae<\/sup> scores. Instead, six of the seven websites had higher SUPR-Q <\/strong>scores for the mobile presentations.<\/p>\n

    In an earlier published study, Betiol and Cybis (2005<\/a>) used the mobile usability testing technologies available at the time. They had participants complete seven tasks for a mobile portal app using a mobile emulator on a computer screen, a mobile device fixed to a tripod in a usability lab, and a mobile device fixed to a sled with a wireless camera used outside the lab (12 participants per context). Despite expecting these different contexts to have dramatic effects on user performance and problem identification, they concluded, \u201cThe results \u2026 showed the existence of more similarities than significant differences\u201d (p. 470).<\/p>\n

    These examples illustrate the importance of conducting this type of research. Even when you logically expect one result, \u201cthe only sure way to have knowledge of a context variable is to vary it\u201d (Abelson\u2019s 7th<\/sup> law<\/a>: You can\u2019t see the dust if you don\u2019t move the couch).<\/p>\n

    Clearly, more data is needed, as we\u2019ve observed many studies testing mobile prototypes in a desktop setting. So, we conducted an experiment to measure the differences in objective and subjective usability metrics when evaluating a mobile prototype with a desktop emulator or on a mobile device.<\/p>\n

    In January and February 2025, we used Figma to build a fake banking app called Capital Two (yes, creative; we know). We had 100 participants attempt three tasks using our MUiQ platform<\/a>. Half the participants were assigned to take the study on a desktop with a mobile emulator, and the other half used the mobile version of MUiQ on their Android or iPhone (which required downloading our MUiQ app to record behavior). With 50 participants in each group, at p<\/em> SEQ\u00ae<\/sup>) and about 27% in binary metrics (e.g., success rates).<\/p>\n

    The Three Tasks<\/h2>\n

    Participants were told they would be asked to complete three tasks on a mobile prototype where some links or features may not be functional. The task orders were randomized for each participant.<\/p>\n

    The three task scenarios and validation criteria were:<\/p>\n

    Task 1:<\/em> Check Balance <\/strong><\/p>\n

    Description:<\/em> Find the checking account balance and begin the bill payment process.<\/p>\n

    Validation:<\/em> Selecting the correct balance amount and the first word in the placeholder text for the terms and conditions.<\/p>\n

    Task 2:<\/em> Card Transaction <\/strong><\/p>\n

    Description:<\/em> Locate the Jan 7 transaction for the Savor credit card.<\/p>\n

    Validation:<\/em> Identify the correct first word from a transaction description.<\/p>\n

    Task 3:<\/em> Credit Score<\/strong><\/p>\n

    Description:<\/em> Find and note the credit score range displayed in the app.<\/p>\n

    Validation:<\/em> Choose the correct score range from multiple-choice options.<\/p>\n

    Figure 2 shows the starting state for the prototype app and two ways to display controls that are not visible at the start. Those controls are the necessary first steps to successfully complete the tasks.\u200b<\/p>\n

    \"Screenshots<\/a><\/p>\n

    Figure 2:<\/strong> Initial paths to successful task completion for all three tasks in the study. From the initial state in Panel 1, users can display the controls below the fold by scrolling or dragging (Panel 2) or\u00a0 clicking on the hamburger menu (Panel 3).<\/span><\/p>\n

    Task 1: Check Balance<\/h2>\n

    Figure 3 shows the flow for efficient completion of this task, assuming the user has selected Check Balance from the starting screen or Checking Account from the hamburger menu. To complete the task, the participant notes the balance in Panel 1, selects Pay Bills from the options in Panel 1, notes the first word in the terms and conditions (Panel 2), scrolls\/drags to the bottom of that page, then selects the checkbox to agree (Panel 3).<\/p>\n

    \"Screenshots<\/a><\/p>\n

    Figure 3:<\/strong> Flow for the Check Balance task.<\/p>\n

    Task 2: Card Transaction<\/h2>\n

    Figure 4 shows the flow for efficient completion of this task, assuming the user has selected View Cards from the starting screen or Credit Card Transactions from the hamburger menu. To complete the task, the participant swipes or clicks an arrow control to switch from the Platinum to the Savor card (Panel 1), then notes the first word of the final transaction (Panel 2).<\/p>\n

    \"Screenshots<\/a><\/p>\n

    Figure 4:<\/strong> Flow for the Card Transaction task.<\/p>\n

    Task 3: Credit Score<\/h2>\n

    Figure 5 shows the final screen for the efficient completion of this task (assuming the user has selected View Score from the starting screen or Check Credit Score from the hamburger menu). To complete the task, the participant notes the credit score range.<\/p>\n

    \"Screenshot<\/a><\/p>\n

    Figure 5:<\/strong> The completion screen of the Credit Score task.<\/p>\n

    Task-Level Metrics<\/h2>\n

    To compare performance, we had four task-level metrics: two objective and two attitudinal metrics<\/a>. Successful completion rates were assessed with multiple-choice questions, task time was automatically collected by the MUiQ platform, perceived task ease was assessed with the SEQ, and confidence in having completed the task correctly was assessed with our standard confidence item (Figure 6).\u00a0\"Screenshots<\/a><\/p>\n

    Figure 6:<\/strong> Our standard task completion confidence item.<\/p>\n

    Study-Level Metrics<\/h2>\n

    At the study level, we measured perceived ease and usefulness with the UX-Lite<\/a>\u00ae<\/sup> and the extent to which participants thought the prototype they evaluated was realistic (Figure 7).<\/p>\n

    \u200b\"Screenshots<\/a><\/p>\n

    Figure 7:<\/strong> The realism item.<\/p>\n

    Across most metrics collected, mobile tended to have slightly more favorable scores,<\/em> with some reaching statistical significance. Prior to digging into the individual metrics, we compared the composition of the samples, finding no significant differences in the distributions of participant age or gender for the two experimental conditions.<\/p>\n

    Successful Task Completions<\/h2>\n

    All success rates tended to be high (over 80%), with two out of three completion rates higher on mobile. Only the Credit Score difference was statistically significant (higher for the mobile condition) as shown in Table 1.<\/p>\n\n\n\n\n\n\n\n
    Task<\/th>\nDesktop<\/th>\nMobile<\/th>\nDifference<\/th>\np<\/i><\/th>\n<\/tr>\n<\/thead>\n
    Check Balance<\/td>\n90%<\/td>\n84%<\/td>\n\u2007\u2002\u200a6%<\/td>\n0.375<\/td>\n<\/tr>\n
    Card Transaction<\/td>\n86%<\/td>\n94%<\/td>\n\u2007\u22128%<\/td>\n0.185<\/td>\n<\/tr>\n
    Credit Score<\/td>\n82%<\/td>\n96%<\/td>\n\u221214%<\/td>\n0.026<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    Table 1:<\/strong> Successful task completions with p<\/em>-values generated using N<\/em>\u22121 Two-Proportion tests<\/a>.<\/p>\n

    To understand the differences in completion rates, we reviewed the task videos (captured in MUiQ) of the participants who failed the Credit Score task (nine in the desktop condition and two in the mobile condition), focusing on the properties of the screens shown in Figure 8.<\/p>\n

    \"Screenshots<\/a><\/p>\n

    Figure 8:<\/strong> Two potential sources of error in the Credit Score task.<\/p>\n

    As shown in Panel 1 of Figure 8, the area on the home screen required to complete the first step of the Credit Score task is completely hidden from view (below the fold<\/a>) and includes an image that participants might interpret as the target for the task. We were curious about whether there were differences in the error paths for the desktop and mobile conditions. We hypothesized several different potential error paths:<\/p>\n