Experiments
Experiments are at the heart of A vs B. Each one compares two or more versions of a page, or a user flow, against a goal you can measure. This section covers everything: from your first experiment to managing ones that are already live.
How experiments work
An experiment splits your visitors into groups. Each group sees a different version of your page, called a variation. One group always sees the original, unchanged page: this is the Control. The other groups see changed versions: these are the Variants. A vs B tracks a metric for each group. Then it uses statistics to work out which version wins.
The five-step builder
Creating an experiment in A vs B uses a guided five-step builder:
- Targeting: Choose which pages the experiment runs on, and which visitors can take part.
- Variations: Define the control and variant(s), write their CSS and JS, and set traffic splits.
- Metrics: Select the goals you are measuring.
- Analysis: Choose the stats engine and confidence level. You can also seal an analysis plan that locks in your primary metric and guardrails before you launch.
- Review: Pre-flight checks and the publish button.
Pages in this section
Creating an ExperimentStart a new experiment and understand the five-step builder flow.
TargetingURL rules and audience selection: control exactly which visitors see your experiment.
VariationsSet up control and variants, write CSS/JS, and set traffic splits.
Split URL ExperimentsSend a slice of traffic to a different page or funnel, instead of changing the one they're on.
MetricsSelect the metrics that determine which variation wins.
Review & PublishPre-flight checks and publishing your experiment to production.
Variance Reduction (CUPED)How CUPED uses what visitors did before the test to shrink noise, so you need less traffic to see a real effect.
Managing Running ExperimentsPause, stop, edit, and manage experiments that are already live.
Experiment StatusesDraft, Scheduled, Running, Paused, and Completed: what each status means, plus archiving.
Scheduling ExperimentsLaunch and end an experiment at a set time. Times show in your own timezone, with catch-up handling built in.
Bayesian EngineThe default engine: read results as a probability of beating Control, plus a credible interval.
Frequentist EngineHow p-values, confidence intervals, and significance work, and when to pick this engine.
Sequential EnginePeek as often as you like and stop the moment the evidence is in. The trade is a slightly wider interval.
Choosing a Stats EngineA plain-English comparison of Bayesian, Frequentist, and Sequential, and when each is the right call.
Comparing enginesSee any experiment's results under a different stats engine, without changing the official one.
Analysis plans & pre-registrationLock the primary metric, guardrails, engine, and win bar before launch. Every later change goes through a documented amendment.
Early stopping & peek protectionWhy stopping a Frequentist experiment early inflates false positives. A vs B protects you with a banner, a modal, and an audit stamp.