A/B Testing a Static Docusaurus Site

It's common to use static site generators like Jekyll or Docusaurus for marketing or documentation websites. However, it's not always easy to run A/B tests when using these tools.

Prefab makes it simple. In this post we'll show how to setup an A/B test on a statically-generated Docusaurus website. We'll also show you how to send your experiment exposures to an analytics tool. We'll be using Posthog, but the process should be very similar for any analytics tool that has a JS client.

Installing Prefab

This step is the same as for adding Prefab to any other React project.

npm

npm install @prefab-cloud/prefab-cloud-react

yarn

yarn add @prefab-cloud/prefab-cloud-react

Initializing Prefab in Docusaurus

We recommend using the PrefabProvider component from our React library. In a normal React application, you would insert this component somewhere near the top level of your app. For a Docusuarus site, the easiest place to add it is in the Root component. That way Prefab will be available for experimentation on any page of your site.

tip

If you haven't already swizzled the Root component, here's a link to the Docusaurus docs for how to do it: https://docusaurus.io/docs/swizzling#wrapper-your-site-with-root

Everything that we're going to do here needs to run client side, so we'll start by adding the Docusaurus useIsBrowser hook to our Root component.

import React from "react";
import useIsBrowser from "@docusaurus/useIsBrowser";

export default function Root({ children }) {
  const isBrowser = useIsBrowser();

  if (isBrowser) {
    // do client stuff
  }

  return <>{children}</>;
}

This is the basic initialization for the Prefab client.

import React from "react";
import useIsBrowser from "@docusaurus/useIsBrowser";
import { PrefabProvider } from "@prefab-cloud/prefab-cloud-react";

export default function Root({ children }) {
  const isBrowser = useIsBrowser();

  if (isBrowser) {
    const onError = (error) => {
      console.log(error);
    };

    return (
      <PrefabProvider apiKey={"YOUR_CLIENT_API_KEY"} onError={onError}>
        {children}
      </PrefabProvider>
    );
  }

  return <>{children}</>;
}

Adding Context for Consistent Bucketing

Often A/B tests are bucketed based on users. To do that, we need some consistent way to identify the user, even if they're not logged in...which is usually the case for a static site. Luckily you can probably get an identifier from whatever analytics tool you have installed, or you can generate one yourself.

const uniqueIdentifier = window.posthog?.get_distinct_id();

Once you have the identifier, you can pass it to the Prefab client as context.

const contextAttributes = {
  user: { key: uniqueIdentifier },
};

<PrefabProvider
  ...
  contextAttributes={contextAttributes}
  ...
>
  {children}
</PrefabProvider>

tip

We have some opinions about why you might want to generate your own unique tracking ID.

Tracking Experiment Exposures

Your experiment is only going to be useful if you have data to analyze. Prefab is designed to work with whatever analysis tool you already have, so you don't have a competing source of truth. To do this we make it easy to forward exposure events to your tool of choice.

Typically you will have initialized your tracking library as part of the Docusaurus config. You can then provide an afterEvaluationCallback wrapper function to the Prefab client. This will be called after each use of isEnabled or get to record the flag evaluation and resulting value. In this example we're using the Posthog analytics platform.

<PrefabProvider
  ...
  afterEvaluationCallback={(key, value) => {
    window.posthog?.capture("Feature Flag Evaluation", {
      key, // this is the feature flag name, e.g. "my-experiment"
      value, // this is the active flag variant, e.g. true, "control", etc.
    });
  }}
  ...
>
  {children}
</PrefabProvider>

Here's an example chart from Posthog showing an experiment funnel going from experiment exposure to viewing any other page.

tip

Prefab also provides evaluation charts for each feature flag, which you can find under the Evaluations tab on the flag detail page. This telemetry is opt-in, so you need to pass collectEvaluationSummaries={true} to PrefabProvider if you want the data collected. While these are lossy and not a substite for analysis in your analytics tool of choice, they can be useful for troubleshooting experiment setup. Below is an example of an experiment with a 30/70 split.

Setting up Your Experiment Code

Congrats, now you're ready to use Prefab from any Docusuarus JSX page or component. Import the usePrefab hook and use it to get a value for your experiment.

import React from "react";
import Layout from "@theme/Layout";
import { usePrefab } from "@prefab-cloud/prefab-cloud-react";

export default function Hello() {
  const { isEnabled } = usePrefab();

  return (
    <Layout title="Hello" description="Hello React Page">
      {isEnabled("my-experiment") && (
        <div>
          <p>"Some experimental copy..."</p>
        </div>
      )}
    </Layout>
  );
}

tip

The usePrefab hook also provides a get function for accessing non-boolean feature flags.

Is it Fast?

The Prefab client loads feature flag data via our CDN to ensure minimal impact on your page load speed. It also caches flag data after the initial load. You can read more about the Prefab client architecture in our docs.

Will it Flicker?

There's a catch here, which is not specific to using Prefab. Since Docusaurus is a static site generator, it does not execute any server-side logic when pages are requested. There are more details in the Docusaurus static site generation docs.

This means that the page will first render the static version, which means no access to cookies or to the Prefab flags data. Once your React code runs client-side, it will render again with the correct feature flag values from Prefab.

So in the example above, the page will initially load without your experiment content. Then it will pop-in on the re-render. You'll have to make a judgement call on whether this negatively impacts the user experience, depending on where the experiment is on the page and how it affects the layout of other page elements.

The alternative is to render a loading state on the initial render, then display the actual content once the Prefab client has loaded.

const MyComponent () => {
  const {get, loading} = usePrefab();

  if (loading) {
    return <MySpinnerComponent />
  }

  switch (get("my-experiment")) {
    case "experiment-on":
      return (<div>Render the experiment UI...</div>);
    case "control":
    default:
      return (<div>Render the control UI...</div>);
  }
}

You can read a more in-depth discussion of handling loading states in the Prefab React client docs.

Configuring the Experiment in the Prefab Dashboard

I wrote a detailed walkthrough of creating flags in the Prefab UI in a previous blog post.

For a simple experiment with only a control and an experiment treatment, you'll want to create a boolean feature flag. The important part for making it an experiment is defining a rollout rule for targeting. Notice that we are setting user.key as the "sticky property". This means that Prefab will use the unique identifier we passed in for segmenting users into the two experiment variants.