Profit Metrics
Revenue tells you how much money came in. Profit tells you how much you kept, after what the products cost you to source. A vs B computes two profit metrics for each experiment variation (one specific version being tested, control or a challenger), once you supply your product costs: Profit per visitor and Profit per order.
How profit is calculated
For each attributed order, A vs B computes:
profit = net revenue − cost of goods sold − (optional) shipping costIn detail:
- Net revenue is the order total after refunds. Discounts are already embedded in the order total (it is the discounted grand total your checkout records), so they are not subtracted again.
- Cost of goods sold (COGS) is the sum of
quantity × unit costacross all line items on the order that have a known cost. - Shipping cost can be subtracted too. A per-project switch called Take shipping cost off profit controls it, on the Profit settings card at the bottom of Commerce → Orders & attribution. It is off by default, which matches the most common margin definition (contribution margin before logistics). Flipping it applies from the next nightly profit run onward. Orders already worked out keep the profit they were given.
- Profit can be negative. That is a real result, not a data error.
How refunds affect COGS
Net revenue already has refunds removed. For COGS, a proportional adjustment is applied: if 30% of the order value was refunded, COGS is reduced by 30% as well. This keeps the profit figure consistent. You are not left paying phantom costs on revenue that never arrived.
What is NOT subtracted
Discounts are not re-subtracted. The order total A vs B records is the discounted amount the customer actually paid. Subtracting discounts a second time would understate profit.
Providing your product costs
There are two ways to give A vs B the cost of each product SKU.
Option 1: Upload a costs file
Create a dataset with the reserved slug product-costs on the Datasets page. The dataset type must be TABLE, keyed by sku. Each row is a JSON object with two fields: your product's sku, and its cost:
{"sku": "SKU-001", "cost": 12.50}The sku field name matters: A vs B uses it as the row's key. A row with any other field name (for example key instead of sku) fails to upload.
cost is a decimal number in major currency units (dollars, pounds, euros: the same units you use everywhere else). No cents conversion is needed at upload time.
Upload the file as NDJSON (one JSON object per line), commit the version, and activate it. Profit metrics become available for any experiment in that project the morning after the first nightly job runs.
You can only create one dataset with the slug product-costs per project, and it must be a TABLE dataset. If you try to create a dataset with this slug as a different type, the dashboard will reject it. The slug is reserved so the nightly profit job can find your costs reliably.
Option 2: Connect Shopify and sync automatically
If your project has the A vs B Shopify app installed, you can let A vs B pull unit_cost directly from your Shopify inventory. A nightly sync job fetches the cost for every variant (one specific size, colour, or option of a product) in your store that has a cost set. It runs at 2 AM UTC, builds the product-costs dataset version automatically, and activates it. That's one hour before the profit job runs, at 3 AM UTC.
The sync runs per project, for every project with an active Shopify connection. To turn it off, switch Keep product costs in step with Shopify off on the Profit settings card at the bottom of Commerce → Orders & attribution. That is what to do if you want a cost list you uploaded yourself to stay the one that counts.
If you uploaded a costs file manually and then connect Shopify, the next nightly sync creates a new version of product-costs from Shopify data and activates it. Your manual upload stays in the version history, and you can reactivate it from the Datasets page. But Shopify's version is live from that point, and the sync runs again every night after.
USER vs SYSTEM costs: coexistence rules
A dataset has a source: USER if you created it via the dashboard upload flow, or SYSTEM if A vs B's Shopify sync created it first. Both types use the same product-costs dataset. There is only ever one per project. Each new upload or sync run adds a new version. Whichever version is activated serves.
A SYSTEM-sourced product-costs dataset cannot be deleted from the dashboard (it is managed by the sync). A USER-sourced one can be deleted. Deleting it stops profit metrics until a new costs dataset is created, or the sync creates one.
Cost coverage
Not every SKU in every order necessarily has a cost row. When items are missing costs, A vs B tracks cost coverage: the percentage of line items on attributed orders that had a known cost.
By default, only fully-covered orders (every line item has a cost) are included in profit calculations. This is the safest default. An order missing half its costs would understate COGS and overstate profit.
The Results page shows:
- Cost coverage: N%: the proportion of attributed line items with known costs, across all variations.
- N orders excluded (incomplete cost data): when the covered-only policy is active and some orders were excluded.
You can switch to include-all mode using the coverage toggle on the Results page. In include-all mode, orders with partial coverage are included. Missing items contribute zero to COGS, so their revenue counts toward profit with no offsetting cost. This is useful for quickly checking whether the missing costs skew results significantly.
Orders recorded without any line items (for example, a webhook (an automatic message A vs B receives when your store reports something) that arrived with no items array) are treated as fully covered with zero COGS. Their profit equals their net revenue. They are not excluded under covered-only mode.
- The Covered only / Include all toggle switches which orders count.
- Cost coverage shows what share of line items had a known cost.
The nightly snapshot limitation
Profit figures reflect the cost version that was active when the nightly job ran. They are not a historical per-order cost ledger. If you change a product's cost today, old orders are repriced at the new cost the next time the job runs.
Specifically: the job runs at 3 AM UTC, reprices all orders from the last 90 days, and records which cost dataset version it used. If you activated a new cost version yesterday evening, tonight's job picks it up and updates the numbers.
The practical implication: if costs change significantly mid-experiment, profit results from before and after the change reflect different cost assumptions. The Results page does not yet show which night's job priced the numbers you're looking at. To check when your active product-costs version was committed, look at its version history on the Datasets page.
Where profit appears in results
Once product-costs is active and the first nightly job has run, the Results page shows a Profit section below the existing revenue cards. It contains:
- Profit per visitor: total profit divided by visitors exposed (actually counted in the experiment, usually because they saw the thing being tested) to each variation.
- Profit per order: total profit divided by the number of orders attributed to each variation.
How profit per visitor is analysed
Profit per visitor gets the same continuous per-visitor treatment as revenue per visitor. Every exposed visitor's profit feeds the statistical engine (zero for a visitor who didn't buy). The engine reports the mean profit per visitor with a confidence interval (a range built so the true result would fall inside it most of the time, if you repeated the experiment) at 95%, plus a profit lift versus the control. The headline number is always the profit lift, never the conversion-rate lift. That way, a variation that sells fewer, higher-margin orders is judged on the margin it actually earns.
Outlier capping is on by default for profit per visitor. Per-variation profit values above the 99th percentile are capped down before analysis, a technique called winsorization: it stops one unusually large order from swinging the result. The cap is configurable per metric.
Cost coverage interacts with the confidence interval. The interval is computed on whichever orders the active coverage policy admits. Under covered-only (the default), orders with incomplete cost data are excluded. With sparse cost coverage, that means fewer data points. Read a narrow interval alongside the coverage percentage shown on the panel: a tight range computed from 40% of your orders is honest about the maths, but only speaks for the covered orders. Switching to include-all widens the data set, but treats un-costed items as zero cost, which overstates profit.
Profit per order is a ratio metric (one number divided by another, here profit divided by order count), and uses the same delta-method maths as other ratio metrics on the platform. Both profit metrics support CUPED (a technique that uses a visitor's pre-experiment behaviour to reduce noise, so the same effect shows up with less traffic), just like their revenue counterparts.
You do not need to add profit metrics to an experiment manually. As soon as product-costs has an active version and the nightly job has run at least once, profit cards appear on the Results page for any experiment that already tracks revenue.