{"id":27625,"date":"2025-01-15T03:02:01","date_gmt":"2025-01-15T03:02:01","guid":{"rendered":"https:\/\/peraltafinancing.com\/analytics\/using-observed-power-in-online-a-b-tests\/"},"modified":"2025-01-15T03:02:01","modified_gmt":"2025-01-15T03:02:01","slug":"using-observed-power-in-online-a-b-tests","status":"publish","type":"post","link":"https:\/\/fivemor.com\/?p=27625","title":{"rendered":"Using Observed Power in Online A\/B Tests"},"content":{"rendered":"


\n<\/p>\n

\n

Observed power, often referred to as \u201cpost hoc power\u201d and \u201cretrospective power\u201d is the statistical power of a test to detect a true effect equal to the observed effect size. \u201cDetect\u201d in the context of a statistical hypothesis test means to result in a statistically significant outcome.<\/p>\n

Some calculators aimed at A\/B testing practitioners use observed power to guide users as to what the sample size of their test should be<\/strong>. When performing statistical significance calculations, these tools present something like \u201cRequired sample size\u201d or \u201cHow long to run the test for\u201d next to their results. Those numbers are apparently supposed to tell if the test needs to run for longer or not, or at the very least to indicate if a test is underpowered or overpowered.<\/p>\n

Despite several attempts over the years to nudge the authors of said tools to remove this harmful \u201cfunctionality\u201d from their calculators, it continues to exist. Given the relative popularity of these tools among less experienced practitioners, the use of observed power may be doing noticeable harm to the online experimentation field, hence this article.<\/p>\n

The following topics are covered below:<\/strong><\/p>\n

    \n
  1. Using observed power in practice<\/a><\/li>\n
  2. Requiring statistical significance AND high observed power<\/a><\/li>\n
  3. Fishing for high observed power<\/a>
    3.1     The simulation<\/a>
    3.2     Results from peeking through observed power<\/a><\/li>\n
  4. Takeaways<\/a><\/li>\n<\/ol>\n

    Using observed power in practice<\/h2>\n

    There are three possible scenarios for using observed power in any one of these calculators:<\/p>\n

      \n
    1. The outcome of an A\/B test is calculated as being statistically significant<\/strong>, and the observed power is higher than whatever threshold was chosen (80%, 90%, etc.). In this case the calculator will almost always say that the test was overpowered<\/strong>. In other words, one has \u201cwaited for too long\u201d and has exposed too many users to the test. For example, the sample size may have been 10,000 users per variant, but the \u201cRequired sample size\u201d field may say only 8,000 were needed.<\/li>\n
    2. The outcome may be statistically significant, but the observed power is lower<\/strong> than the chosen threshold, which will (wrongly) suggest the test was underpowered<\/strong>. One might interpret this to mean the test is untrustworthy and cause them to ignore a perfectly valid statistically significant outcome. A different interpretation may be to treat the low observed power as a license to continue the test at least until the \u201cRequired sample size\u201d is reached and perhaps even for longer if it happens to need even more data after that first extension. Both of these routes lead to very different outcomes.<\/li>\n
    3. Finally, the outcome may not be statistically significant<\/strong>, in which case the calculator will always <\/em>suggest the A\/B test is significantly underpowered<\/strong>. For example, a user may have tested with 10,000 users per variant, but the \u201cRequired sample size\u201d output will inform them they need multiple times larger sample size, with numbers 5x or 10x not being uncommon. In the same example that would mean the tools suggest exposing 40,000 or 90,000 more users per variant than had initially been planned (assuming there was a plan) and taking five to ten times longer.<\/li>\n<\/ol>\n

      Notably, one is almost never going to see that their test is well-powered based on observed power calculations.<\/strong> The math is bound to almost always label an A\/B test as either underpowered (more often), or overpowered (typically less often). My comprehensive guide on observed power<\/a> would be a great read for anyone keen on understanding the matter in detail, and it explains why the above scenarios encapsulate all that can happen. It also includes a calculator which lets you thoroughly explore observed power.<\/p>\n

      In the first scenario many experimenters would typically just walk away with a statistically significant finding. What happens in each of the other two scenarios is far more interesting. Let us deal with them one by one.<\/p>\n

      Requiring statistical significance AND high observed power<\/h2>\n

      In scenario two some practitioners will just ignore the warning about their test \u201crequiring more users\u201d or \u201cmore time\u201d since the result is statistically significant. However, some might be swayed into accepting a test\u2019s significant outcome only if the outcome is both statistically significant and \u201cwell-powered\u201d or \u201coverpowered\u201d (based on observed power). This kind of behavior leads to multiple times more stringent type I error control than the nominal threshold sets.<\/p>\n

      The exact type I error which would be achieved can be estimated easily by plotting p-values and observed power on a graph, and checking what p-value corresponds to the chosen power level. This can also be done with the post hoc power calculator<\/a> I\u2019ve created for educational purposes.<\/p>\n

      \n
      \"Actual
      Figure 1<\/strong>: Actual vs nominal threshold if requiring both high observed power and significance<\/figcaption><\/figure>\n<\/div>\n

      For example, using \u201ctextbook\u201d values of 0.05 and 0.80 as the cut-off for significance and for observed power (80% power of the design), one would in fact achieve a type I error rate of about 0.6%, or more than eight times smaller<\/strong> than the target 5%. Note that the same is true if one requires the observed effect to be greater than the MDE.<\/p>\n

      If using 0.05 significance threshold and 90% power, the actual type I error rate would be 0.17%, or about thirty times smaller<\/strong> than the target of 5%. If using a stricter significance threshold of 0.01 with 80% power, the actual type I rate is 0.076%, or about thirteen times smaller, and so on. The stricter each of the thresholds is, the smaller the actual type I error attained through this approach.<\/p>\n

      Additionally I have confirmed all of the above independently through simulations.<\/p>\n

      Such a practice has a crippling effect on the achieved statistical power, which instead of being 80% or 90%, is reduced significantly. If it is not obvious why that is a big <\/em>issue, consider The importance of statistical power in online A\/B testing<\/a>.<\/p>\n

      A different problem faces those who might choose to continue the initially significant but seemingly underpowered test until it is eventually both statistically significant and well-powered. Interestingly, my simulations show that such A\/B tests will not suffer from an inflation of their type I error. However, they should expect to commit to an eightfold increase in sample size<\/strong>, on average, at least in the cases where there is no true effect or it is of minimal size.<\/p>\n

      Fishing for high observed power<\/h2>\n

      In scenario three, some may interpret the fact that their sample size is many times lower than the \u201crequired sample size\u201d as a license to continue testing until the recommended sample size is reached. After all, if a statistically insignificant test seems \u201cunderpowered\u201d, it sounds like there is a true effect waiting to be found, if one just extends their sample size to what\u2019s needed to make it \u201cpowerful enough\u201d. Why then not to continue testing for however long it takes to reach the \u201crequired sample size\u201d suggested by the calculator?<\/p>\n

      However, examining A\/B test data with an intent to stop based on the outcome constitutes peeking<\/a> and is a violation of crucial assumptions behind the tests used in these calculators. Classic peeking is sometimes called \u201cfishing for significance\u201d so by analogy we may call this approach \u201cfishing for observed power\u201d. Just like its cousin, peeking through observed power results in an inflated false positive rate (type I error rate). In a typical scenario of a one-sided test<\/a>, if the true effect is zero or negative, statistically significant outcomes would be \u201cfound\u201d multiple times more often than the nominal p-value or confidence level would suggest.<\/p>\n

      Unlike classic peeking, how much type I error inflation one is bound to get by fishing for high observed power is unknown, at least as far as I am aware. Given this, a simulation is a straightforward way to emulate the behavior of peeking through observed power and to estimate what it leads to.<\/p>\n

      The simulation<\/h3>\n

      The simulation below was performed under the following assumptions:<\/p>\n