Creating an Experiment

Creating an experiment in A vs B takes you through a five-step guided builder. Each step focuses on one aspect of the experiment: where it runs, what each group sees, how success is measured, how the data will be analyzed, and a final review before going live.

The Create Experiment dialog over the experiments list, asking for the experiment type with three options: Visual (point-and-click edits), Custom Code (JavaScript and CSS per variation), and Split URL (send traffic to different URLs). — The first choice when creating an experiment: Visual, Custom Code, or Split URL.

Before you start

Before creating an experiment, make sure:

The A vs B snippet is installed and verified on your website. Without the snippet, experiments cannot run.
You have at least one metric created for this project. You will need to select a metric in Step 3. You can create one during the experiment flow, but it is easier to set them up in advance from the Metrics section.
You have a clear hypothesis: what are you changing, why do you think it will improve things, and how will you measure success?

Starting the builder

The Create Experiment drawer walks you through a few quick questions before it hands you off to the five-step builder proper.

Go to the Experiments page

In the left sidebar, click Experiments. This opens the experiments list for your current project.

Click 'Create Experiment'

Click the Create Experiment button in the top right corner of the experiments list. A drawer opens, starting with Experiment type.

Pick an experiment type

Choose Visual, Custom Code, or Split URL (see below for what each one means), then select Continue.

Enter a name and description

Give your experiment a descriptive name that explains what you are testing and where; a description is optional. Good names make it easy to identify experiments later. Select Continue.

Confirm the target URL, then create

A Visual experiment needs a URL: the visual editor loads that page so you can edit it directly. A Custom Code experiment can leave this blank to apply anywhere. The URL field already shows your project's site address, so you only need to change it if this experiment targets a different page. Press Enter at any point in the drawer to create the experiment right away, or select Create Experiment. A Split URL experiment skips this screen entirely and creates as soon as you confirm the name. Either way, you land in the five-step builder: on Targeting for Visual and Custom Code, or straight on Variations for Split URL, where its destination URLs live.

Examples of descriptive experiment names:

"Homepage hero headline: formal vs casual"
"Pricing page CTA button color: grey vs green"
"Checkout flow: single-page vs multi-step"

The three experiment types

Visual: the most common choice. Make changes directly on your page using the point-and-click visual editor. No code required.
Custom Code: write JavaScript and CSS for each variation by hand. Use this when you need fine-grained control over behaviour that the visual editor cannot express. To A/B test product recommendations, use a Custom Code experiment whose variation code calls the recommendations API; see the Recommendations Quickstart.
Split URL: redirect a portion of qualifying traffic from one page to a separate destination URL. Use this when the change is too large for in-place editing, or when you want to compare two fully independent pages. See Split URL Experiments for the full guide.

The choice is locked once the experiment is created: pick the right one up front, or delete and recreate if you change your mind.

The five-step builder

Once the experiment exists, the builder itself has five steps, in this order: Targeting, Variations, Metrics & Goals, Analysis, then Review & Publish. You can click ahead to any step in the sidebar at any time, so you're never stuck waiting on an earlier one. A vs B just won't let you publish until every step's pre-flight checks pass, so an incomplete step catches up with you at Step 5.

The experiment builder's left sidebar, listing five steps in order (Targeting, Variations, Metrics & Goals, Analysis, Review & Publish) plus a History item below them, with Targeting currently active. — The five-step builder sidebar. You can jump to any step; A vs B only blocks you at Publish if something is incomplete.

Step 1: Targeting

Define which pages this experiment runs on and which visitors are eligible. You set URL targeting rules and optionally select audience segments. See Targeting for the full guide to this step.

Step 2: Variations

Set up your control and variant(s). Write the CSS and JavaScript that define what each variant looks like and does. Configure the traffic split between variations. See Variations for the full guide.

Step 3: Metrics & Goals

Choose which metrics determine the winner. The first metric you select becomes the primary metric: the one that drives statistical significance. You can add more metrics to monitor as secondary goals. See Metrics for the full guide.

Step 4: Analysis

Choose the stats engine (Bayesian, Frequentist, or Sequential), the confidence level, the variance-reduction mode, and any per-experiment analysis overrides. If pre-registration is enabled for the project, seal an analysis plan here before launch. See Choosing a stats engine and Analysis plans for the full background on these decisions.

Step 5: Review & Publish

A vs B runs pre-flight checks to verify the experiment is configured correctly. Once everything passes, you can publish the experiment and it goes live immediately. See Review & Publish for the full guide.

You can save a draft at any point and come back to finish it later. Experiments stay in Draft status until you explicitly publish them. See Experiment Statuses for details on the experiment lifecycle.

After creating

Once your experiment is published, it begins running immediately. You can monitor its progress on the experiment's results page. Results start accumulating as visitors load your website and are bucketed into variations.