{"id":88925,"date":"2025-02-15T03:27:46","date_gmt":"2025-02-15T03:27:46","guid":{"rendered":"https:\/\/peraltafinancing.com\/analytics\/simple-way-to-measure-a-b-test-flicker-impact\/"},"modified":"2025-02-15T03:27:46","modified_gmt":"2025-02-15T03:27:46","slug":"simple-way-to-measure-a-b-test-flicker-impact","status":"publish","type":"post","link":"https:\/\/fivemor.com\/?p=88925","title":{"rendered":"Simple Way To Measure A\/B Test Flicker Impact"},"content":{"rendered":"


\n<\/p>\n

\n

\u201cFlickering\u201d or \u201cFlash Of Original Content\u201d (FOOC<\/a>) is a phenomenon where there\u2019s a (typically) slight but observable delay in the browser updating the site or element layout if the user is included in a variant group for experimentation. This manifests in the original, unmodified element being rendered in the visible portion of the page before the experiment library updates it with the variant.<\/p>\n

There are ways to mitigate the flicker:<\/p>\n

    \n
  1. Add the A\/B testing library directly into the page template and don\u2019t load it via some other, asynchronously loaded dependency (e.g. Google Tag Manager<\/a>).<\/li>\n
  2. Load the A\/B testing library synchronously, and have it hide the element that is being tested until the library is loaded.<\/li>\n
  3. Utilize some kind of anti-flicker tech<\/a>.<\/li>\n
  4. Run the experiments server-side, and render content with the variant in place.<\/li>\n<\/ol>\n
    \n

    <\/p>\n

    \"Flicker<\/p>\n

    <\/a><\/p>\n<\/div>\n

    Typically, the only non-intrusive and consistent way to avoid the flicker is to look into server-side rendering for your experiments. For example, tools like Conductrics<\/a> offer a robust set of APIs to do all the decision-making logic in your server. Then there are tools like Google Optimize<\/a> that require you to do the variant selection and assignment manually, but the tool can then handle the data collection and reporting.<\/p>\n

    However, the reason you\u2019ve read thus far is probably because you\u2019re worried about client-side testing<\/strong>.<\/p>\n


    \n X<\/span><\/p>\n

    \n
    \n The Simmer Newsletter<\/strong>\n <\/p>\n

    \n Subscribe to the Simmer newsletter<\/a> to get the latest news and content from Simo Ahava into your email inbox!\n <\/p>\n

    <\/span><\/p>\n

    Introducing the problem<\/h2>\n

    With JavaScript-based experimentation libraries, you\u2019re subject to the rules and limitations of the page render in the browser. The flicker<\/strong> happens because the page with the modified element is being rendered from the page HTML source, but the experimentation library needs to wait for an opening to allow the browser to process the experiment data.<\/p>\n

    This is most often a problem when you\u2019re running scripts asynchronously<\/strong>. Async load means that once the browser starts to download the library, it doesn\u2019t wait for the download to complete. Instead, it proceeds with the page render. Once the download is complete, and as soon as the browser has an available slot in its single thread of execution, it will start parsing and executing the JavaScript within the library.<\/p>\n

    By moving from asynchronous to synchronous loading, you solve part of this issue. However, it\u2019s not like synchronous loading actually fixes anything automatically. Since the library is loaded at the top of , a synchronously loaded library doesn\u2019t have access to the elements it\u2019s designed to modify (since those elements are created in the , which hasn\u2019t yet been generated).<\/p>\n

    Instead, libraries like Google Optimize, when loaded synchronously, actually hide<\/strong> the element that\u2019s being tested. They inject a style declaration<\/strong> that sets the visibility of all elements matching the CSS selector of the experimentation targets to hidden<\/code>. Only once the element has been actually added to the page, can Optimize then modify it and unhide it. This is fairly elegant but it might introduce a slight flicker of another kind, where the element seems to \u201cpop\u201d into place out of sequence with the rest of the render.<\/p>\n

    \n

    <\/p>\n

    \"optimize-hides.jpg\"\/<\/p>\n

    <\/a><\/p>\n<\/div>\n

    A similar solution is anti-flicker<\/strong> JavaScript. The purpose here is to actually hide the entire page<\/strong> until the experimentation library has loaded. This is, and has been, my biggest objection about how A\/B-testing tools are implemented. I simply can\u2019t fathom the logic behind potentially sacrificing the usability of the entire page just to get better data quality for your experimentation.<\/p>\n

    \n

    Considering how crucial page performance and perceived<\/em> page performance is these days, I steer clear of anti-flicker snippets that hide the entire page. It doesn\u2019t matter if there are mitigations in place for ad blockers and download errors. If the endpoint is unresponsive or lags, Google Optimize\u2019s default anti-flicker snippet<\/a> has the page wait for a maximum of 4 seconds<\/strong> (this is adjustable) before revealing the content. Naturally, if the container loads before that, the page is revealed faster. But still, OUCH!<\/p>\n<\/blockquote>\n

    Measuring the impact of flicker<\/h2>\n

    So, let\u2019s assume the situation is as follows:<\/p>\n

    You\u2019ve got an experiment running that treats a home page element, which is visible above the fold if the page is loaded without a scroll threshold in place.<\/p>\n

    You\u2019ve deployed Google Optimize using the new snippet<\/a>. You\u2019ve deployed the asynchronous<\/strong> snippet, and you are not<\/strong> using the anti-flicker JavaScript, so there\u2019s a visible and measurable flicker in place.<\/p>\n

    \n

    <\/p>\n

    \"Flicker<\/p>\n

    <\/a><\/p>\n

    Flicker of the original (grey background) before the variant (red background)<\/span><\/p>\n<\/div>\n

    In order to measure<\/strong> the severity of this flicker, we need to collect a number of timings:<\/p>\n

      \n
    1. Time when the original element was added to the page,<\/li>\n
    2. Time when the original element became visible in the viewport,<\/li>\n
    3. Time when the experimentation library was loaded,<\/li>\n
    4. Time when the experiment change was applied to the page.<\/li>\n<\/ol>\n

      The flicker<\/strong> is the time delta between (2) and (4). If the element isn\u2019t visible in the viewport, or if the experiment is applied before<\/em> the base element becomes visible, the flicker is not a problem. (3) is interesting metadata about how the experimentation library itself works, and how fast it manages to apply the change after loading.<\/p>\n

      Introduction to the JavaScript we\u2019ll need<\/h3>\n

      The solution will rely on two pieces of JavaScript<\/strong> code running directly in the page template<\/strong>. You can\u2019t<\/strong> execute this code reliably through a dependency like Google Tag Manager, because Google Tag Manager in many cases loads after<\/em> all steps (1)-(4) have already happened, meaning you won\u2019t get accurate measurements.<\/p>\n

      The first bit of JavaScript is run at the very top of , even before the Optimize snippet. This script uses the optimize.callback<\/code><\/a> API to collect the timestamp of the experimentation library load. This is timing number (3) in the list above.<\/p>\n

      The second JavaScript snippet is added to the top of <\/strong>, because the observers need access to document.body<\/code>. Here\u2019s what it does:<\/p>\n