Go to platform

Your First Experiment

A complete walkthrough of your first A/B test. By the end you'll have a live experiment splitting visitors between two versions and tracking which wins.

Our example: does changing the homepage call-to-action button from grey to green get more clicks? The original (grey) is the Control. The new version (green) is the Variant. Half of visitors see each, and A vs B tracks clicks in both groups.

Build it

1

Create the experiment

On your dashboard, click Experiments in the sidebar, then Create Experiment. Pick Custom Code as the experiment type, since this walkthrough writes CSS by hand, then click Continue. Name it (e.g. "Homepage CTA Colour"), add an optional description, and click Continue again. Leave the URL blank on the last screen (you'll set the exact page next) and click Create Experiment. It's saved as a draft and opens the builder on the Targeting step.

2

Step 1: Targeting (where it runs)

Click Add URL rule, set the match type to Path Match, and enter / to run only on your homepage. Leave the audience on Everyone so all visitors are eligible, then click Next: Variations.

Step 1: set URL rules and choose who's eligible.
3

Step 2: Variations (what each group sees)

You start with Control and Variant A. The control is your unchanged page; the variant is what you're testing. On Variant A, click Code Editor, open the CSS file, and paste:

CSS
/* Change the CTA button to green */.cta-button {  background-color: #22c55e;  border-color: #16a34a;}
CSS5 lines

Leave the split at 50/50 and click Next: Metrics.

Step 2: set the split and open the code editor for each variation.
4

Step 3: Metrics & Goals (what you're measuring)

Pick the metric that decides the winner. If you don't have one yet, click Create a new metric, choose Click, name it (e.g. "CTA Button Click"), enter the button's CSS selector (.cta-button), and click Create metric.

That puts the metric in your project's library. It is not measuring anything yet: attaching it to this experiment is three short steps, and the step list still reads "No metrics attached" until you finish them. Click Select next to the metric and a dialog opens:

  1. Step 1 of 3, Pick a metric: the one you clicked Select on.
  2. Step 2 of 3, Pick a measure: how it is counted. For a click metric, Unique conversions per visitor is the usual choice.
  3. Step 3 of 3, Configure measure: which direction counts as better, with a live preview of the number. Then click Attach metric.

The metric now shows in the attached list, marked Main goal and Primary. The first metric you attach is the primary one: it is what decides the winner. Click Next: Analysis.

Step 3: attach the metric that decides the winner.
5

Step 4: Analysis & Engine (how results are judged)

A vs B already picked a statistics engine for you: Bayesian, the project default. The defaults are fine for your first experiment, so just click Next: Review. This choice locks once the experiment launches, since switching it mid-flight would invalidate the result.

Step 4: the analysis choices are usually fine left at their defaults.
6

Step 5: Review & Publish

A vs B runs a pre-launch checklist: the snippet detected on your site, a metric attached, and the traffic split adding up to 100%. You can also schedule the experiment to start later from this screen, but for your first test, publishing right away is simplest. When every check is green, click Publish, then confirm. It's live.

Step 5: the pre-launch checklist must be green before you publish.
More ways to target pages

Path Match compares the visitor's page path exactly, so / matches only your homepage. Use Exact Match when you need the complete address, protocol included, like https://example.com/pricing. Use Substring when you want one rule to catch text anywhere in the address, such as /blog matching every page under your blog. It can also catch pages you didn't intend, if that same text shows up elsewhere in the address.

Two ways to write variation code

  • Platform editor: write CSS and JavaScript right in A vs B's built-in editor. Best for quick changes.

  • CLI + Chrome extension: clone the experiment with avsb clone, write TypeScript and SCSS in your own IDE with hot reload, then avsb push the compiled code back. See Local Development.

You can test up to four variations total: the control plus three variants.

Watch the results

After publishing you land on the results page. It's empty at first. That's normal. As visitors load your site and get bucketed, you'll see the number of visitors per variation, each one's conversion rate, and a significance indicator once there's enough data to call a winner. Here's what a healthy, running experiment looks like once the data is in:

The results page once visitors arrive: winning probability, observed lift, and a conversion-rate time series.
Give it time

A/B tests need data. Aim for at least a few hundred conversions per variation before trusting the result; don't stop after a day or two.

You don't have to sit and watch

The first time an experiment in this project records real results, A vs B emails everyone who can manage the project, with a link straight to those results. It is a one-off "you're live" note rather than a running report: once per project, when the data actually arrives.

Where to go next

Was this helpful?