{"id":22566,"date":"2025-01-11T09:46:15","date_gmt":"2025-01-11T09:46:15","guid":{"rendered":"https:\/\/peraltafinancing.com\/analytics\/a-comprehensive-guide-to-observed-power-post-hoc-power\/"},"modified":"2025-01-11T09:46:15","modified_gmt":"2025-01-11T09:46:15","slug":"a-comprehensive-guide-to-observed-power-post-hoc-power","status":"publish","type":"post","link":"https:\/\/fivemor.com\/?p=22566","title":{"rendered":"A Comprehensive Guide to Observed Power (Post Hoc Power)"},"content":{"rendered":"


\n<\/p>\n

\n

\u201cObserved power\u201d, \u201cPost hoc Power\u201d, and \u201cRetrospective power\u201d all refer to the statistical power of a statistical significance test to detect a true effect equal to the observed effect. In a broader sense these terms may also describe any power analysis performed after an experiment has completed. Importantly, it is the first, narrower sense that will be used throughout the article.<\/p>\n

Here are three typical scenarios of using observed power<\/strong> in online A\/B testing and in more general applications of statistical tests:<\/p>\n

    \n
  1. An experiment concludes with a statistically non-significant outcome. Observed power is compared to the target power to determine if the test was underpowered. If that is the case it is argued that there is a true effect which remains undetected. This can lead to a larger follow-up experiment or in some cases to extending the experiment until it achieves whatever level of observed power level is deemed sufficient;<\/li>\n
  2. An experiment concludes with a statistically significant outcome, but the observed power is smaller than the target power level of, say, 80% or 90%. This is seen as the test being underpowered and therefore questionable or outright unreliable;<\/li>\n
  3. An experiment concludes with a statistically significant outcome, but the observed effect size is lower than the target MDE (minimum detectable effect), and that is seen as cause for concern for its trustworthiness;<\/li>\n<\/ol>\n

    While these might seem reasonable, there are unsurmountable issues in using post hoc power. These have been covered with increasing frequency in the scientific literature [1][2][3][4][5]<\/sup>, yet the problem persists in statistical practice.<\/p>\n

    This article takes certain novel approaches to debunking observed power, including using a post hoc power calculator as an educational device. Through a comprehensive examination of the rationale behind observed power all possible use cases are shown to rest on valid, but unsound chains of reasoning<\/strong>, which might explain their surface appeal.<\/p>\n

    This and the two companion articles further explore the different harms caused by using observed power<\/strong> in scenarios readers should find familiar, in a uniquely comprehensive manner.<\/p>\n

    Table of Contents:<\/strong><\/p>\n

      \n
    1. Observed power as a proxy for true power<\/a>
      1.1.       High true power is not a requirement for a good test<\/a>
      1.2.       Observed power as an estimate of true power<\/a><\/li>\n
    2. Viewing the power level as a threshold to be met<\/a>
      2.1.       Low observed power does not mean a test is underpowered<\/a><\/li>\n
    3. Observed power is a direct product of the p-value<\/a>
      3.1.       Post hoc power calculator<\/a><\/li>\n
    4. Observed power with a non-significant outcome<\/a>
      4.1.       Does peeking through observed power inflate the false positive rate?<\/a><\/li>\n
    5. Observed power with a statistically significant outcome?<\/a>
      5.1.       MDE as a threshold<\/a>
      5.2.       Viewing the observed effect size as the one a test\u2019s conclusion is about<\/a><\/li>\n
    6. All power is propter hoc power<\/a><\/li>\n
    7. Takeaways<\/a><\/li>\n<\/ol>\n

      Observed power as a proxy for true power<\/h2>\n

      A major line of reasoning behind the use of observed power is what may be called \u201cthe search for true power\u201d<\/strong><\/em>.<\/p>\n

      The logic of searching for \u201ctrue power\u201d*<\/strong> goes like this:<\/p>\n

        \n
      1. \u201cGood\u201d or \u201ctrustworthy\u201d statistical tests should have high \u201ctrue power\u201d: high probability of detecting the true effect, whatever it may be;<\/li>\n
      2. The true effect size is unknown and there is often little data to produce good pre-test predictions about where it lies;<\/li>\n
      3. Post data, the observed effect (or an appropriate point estimate), is our best guess for the true effect size;<\/li>\n
      4. Substituting the true effect for the observed effect in a power calculation should result in a good estimate of the \u201ctrue power\u201d of a test.<\/li>\n<\/ol>\n

        These premises lead to the following conclusion:<\/p>\n

        \n

        To ensure a test is \u201cgood\u201d or \u201ctrustworthy\u201d, aim to achieve high power versus the observed effect as the best proxy for the true statistical power of the test.<\/p>\n<\/blockquote>\n

        While the above is a valid argument for using observed power, it is not a sound one since premises one and four can be shown to be false.<\/p>\n

        \n* side remark on \u201ctrue power\u201d<\/summary>\n

        The focus some people have on \u201ctrue power\u201d may stem from poor and incomplete definitions of power found in multiple textbooks and articles. Namely, one can often see power described as \u201cthe probability of finding an effect, if there is one\u201d. However, this is a misleading definition since it skips the all important \u201d of a given size\u201d instead of the easily assumed \u201cof any size\u201d. Such a confusion result may lead to searches of \u201ctrue power\u201d instead of acceptance of its hypothetical nature. Compare the above shorthand to the definition given in \u201cStatistical Methods in Online A\/B Testing\u201d: \u201cThe statistical power of a statistical test is defined as the probability of observing a p-value statistically significant at a certain threshold \u03b1 if a true effect of a certain magnitude \u03bc1 is in fact present.[\u2026]\u201d<\/p>\n<\/details>\n

        High true power is not a requirement for a good test<\/h3>\n

        When planning an A\/B test or another type of experiment, one often does not know the true effect size and may have very limited information to make predictions about where it may lie. Planning a test with a primary motivation of achieving a high true power results in committing to as large a sample size as practically achievable and in choosing as low a significance threshold as possible. This would maximize the chance of detecting even tiny true effects.<\/p>\n

        However, such an approach would ignore the trade-offs<\/strong> between running a test for longer \/ with more users, and the potential benefits that may be realized (for details see \u201cStatistical Power, MDE, and Designing Statistical Tests\u201d<\/a>, risk-reward analysis<\/a> and its references). True effects below a given value are simply not worth pursuing due to the risks of running the test<\/a> no longer being justified by the potential benefits.<\/p>\n

        The pivotal value where that happens is called a minimum detectable effect (MDE) or a minimum effect of interest (MEI). It is the smallest effect size one wants to detect with high probability, where \u201cdetect\u201d means a statistically significant outcome at a specified significance threshold. It is the smallest effect size one would not want to miss. Aiming for high power for any value below the MEI would, by definition, achieve poorer business outcomes from A\/B testing<\/strong>.<\/p>\n

        Even if it is almost sure<\/em> what the true effect size is, it would rarely match the minimum effect of interest. The framework for choosing optimal sample sizes and significance thresholds deployed at Analytics Toolkit\u2019s A\/B testing hub<\/a> often arrives at optimal parameters under which the MDE (at a high power level such as 80% or 90%) is larger than the expected effect size. This decision framework is most fully explained in the relevant chapter of \u201cStatistical Methods in Online A\/B Testing\u201d<\/a>.<\/p>\n

        \n
        \"\"