Experimentation / A/B Testing

The Experimentation capability answers the three questions every A/B-testing backend agrees on:

1. Which variant of experiment X should user U see right now? (deterministic per-user assignment) 2. Record that we showed variant V to user U at time T. (exposure tracking — the denominator in the final stats) 3. Record that user U did metric M with value V at time T. (outcome tracking — the numerator in the final stats)

The trait is vendor-neutral. Local-hash backends, GrowthBook, Statsig, PostHog, and Optimizely all plug into the same three methods.

Feature Flags vs Experiments

Some commercial platforms bundle both, and it is tempting to merge them into a single trait. bext keeps them separate because the surfaces differ: feature flags return bool / String variants and never emit analytics; experiments return a Variant plus demand exposure and outcome writes. A project using GrowthBook for both can still install one plugin that exposes both traits — they just stay distinct at the call site.

The Trait

pub trait ExperimentPlugin: Send + Sync {
    fn name(&self) -> &str;

    fn assign(
        &self,
        experiment: &str,
        ctx: &ExperimentContext,
    ) -> Result<Variant, ExperimentError>;

    fn record_exposure(
        &self,
        exposure: ExposureRecord,
    ) -> Result<(), ExperimentError>;

    fn record_outcome(
        &self,
        outcome: OutcomeRecord,
    ) -> Result<(), ExperimentError>;
}

assign is pure. Calling it a thousand times with the same context returns the same variant and affects nothing. Exposure is the side effect that moves the statistics. Keeping them separate lets callers assign, gate on some other condition, and only then expose — the standard server-rendered pattern.

Key Types

user_id is optional everywhere. Anonymous users still participate — they default to the control variant, and exposure / outcome records key on session_id inside attributes if the backend supports it.

Outcome Value Shapes

Reference Implementation: @bext/experiment-hash

@bext/experiment-hash is a local, deterministic backend that needs no network. Assignment is an FNV-1a hash of experiment_id + ":" + user_id:

- The hash is reduced to a value in [0, 1).

- That value picks a variant based on the configured split.

- Anonymous users (no user_id) always get the control variant.

Deterministic means the same (experiment, user_id) pair lands on the same variant for the lifetime of the experiment — essential for consistent UX across requests.

[[plugins]]
name = "experiment-hash"
source = "@bext/experiment-hash"

[[plugins.experiments]]
id = "checkout_button_color"
variants = [
    { id = "control",   weight = 50, is_control = true },
    { id = "treatment", weight = 50, is_control = false },
]

Example: Assign, Expose, Record Outcome

use bext_plugin_api::experiment::*;
use chrono::Utc;

let plugin: &dyn ExperimentPlugin = /* @bext/experiment-hash */;

// 1. Deterministic assignment (pure, no side effects).
let ctx = ExperimentContext::for_user("alice")
    .with_attribute("country", "FR")
    .with_attribute("plan", "pro");
let variant = plugin.assign("checkout_button_color", &ctx)?;

// 2. Render the page and tell the backend the user saw this variant.
let html = render_with_variant(&variant);
plugin.record_exposure(ExposureRecord {
    experiment: "checkout_button_color".into(),
    variant_id: variant.variant_id.clone(),
    user_id: ctx.user_id.clone(),
    timestamp: Utc::now(),
})?;

// 3. Later — the user converts. Record the outcome.
plugin.record_outcome(OutcomeRecord {
    experiment: "checkout_button_color".into(),
    metric: "signup".into(),
    value: OutcomeValue::Bool(true),
    user_id: ctx.user_id,
    timestamp: Utc::now(),
})?;

Exposure Before Outcome, Always

The invariant every experimentation backend expects: record exposure before outcomes. The exposure is what makes the user part of the experiment; outcomes attributed to a user who was never exposed are thrown away by the analysis engine. bext does not enforce the order at the trait level — it is up to the caller — but the order matters for the numbers to make sense.

Feature Flag

None. The Experiment trait and types live in bext-plugin-api and are always available; no cargo feature gates them.

See Also

- @bext/experiment-hash — local deterministic backend.

- Feature Flag — code-path selection (rollout, kill switch) distinct from experiment measurement.

- Session — where the stable user_id for deterministic bucketing typically comes from.

- Tracer — correlate experiment variant assignments with distributed traces.

- Capabilities overview — the full list of pluggable capabilities.