Notification Keywords

Notification keywords are a small list of organization-wide words that decide which experiments are allowed to notify anyone. Name an experiment so it begins with one of those keywords, and it fires notifications. Name it anything else, and it stays silent. No one has to configure routing by hand for every experiment.

This keeps QA experiments, internal A/A tests, and exploratory work out of production channels. The experiments that matter, like launches and optimization rollouts, still flow through automatically. They reach Slack, Teams, Jira, and webhooks (an automatic HTTP request A vs B sends to your own server when something happens).

Built-in keywords: LIVE and PROD

LIVE and PROD are recognized in every organization automatically, no setup required:

• LIVE: for production-impacting experiments your team wants to broadcast.
• PROD: alternate naming convention for teams that prefer it.

They are always active and can't be turned off, so a production-named experiment can never be accidentally silenced. You can add your own keywords on top (see below); the built-ins keep working alongside them.

How matching works

Matching is a case-insensitive prefix check with a separator. An experiment name matches a keyword when three things are all true:

1. The name starts with the keyword, in any mix of upper and lower case.
2. The very next character is a separator: a space, dash (-), underscore (_), or colon (:).
3. There is at least one more character after that separator.

Here is what that means in practice, using the default keywords LIVE and PROD:

Configuring keywords

Open Organization Settings → Integrations. The Notification keywords panel sits at the top. Keywords are organization-wide: they apply to every project under the organization.

1. The Notification keywords panel heading, at the top of the Integrations tab.
2. A keyword row, with its Edit and Remove buttons.
3. The Add keyword input and button.

Adding a keyword

Click Add keyword, type the keyword (letters and numbers only, up to 32 characters, no spaces or punctuation), and save. The new keyword takes effect immediately for every event fired afterward.

Editing a keyword

Hover the keyword row and click Edit. The same validation rules apply: letters and numbers only, 32 characters maximum. Editing changes which experiment names match. Running experiments whose names no longer match go silent on the next event; ones whose names now match start firing.

Removing a keyword (with impact preview)

Click Remove on a keyword row. A confirmation modal titled "Remove notification keyword" opens with the impact preview: how many running experiments will be silenced if you confirm. An optional link lists them by name. Review it, then confirm the removal.

If the impact cannot be worked out, the modal says so and offers Try again. You can still confirm the removal without it: the summary is a courtesy, not a condition of removing the keyword.

Safety events bypass the gate

A guardrail is a metric you're not trying to improve but don't want to make worse, like page load time. Two events ignore the keyword convention and always fire, no matter the experiment's name: SRM (short for sample ratio mismatch) failing, and a guardrail breaching.

• SRM failed: fires when the traffic split between variations (one specific version being tested, control or one of the challengers) doesn't match what you configured. It usually means a bug rather than a real result, so it needs attention even on a QA experiment.
• Guardrail breached: fires when a guardrail crosses the line you set for it. Catching this early on any experiment, gated or not, is the safer default.

Both events are flagged as bypassedGate=true in the delivery record so you can filter them in the delivery log.

The Notifications badge

Every experiment list row and detail page header displays a small Notifications badge:

• Notifications: On: the experiment name matches a keyword. Hovering shows which keyword it matched.
• Notifications: Off: the experiment name does not match any keyword. Hovering shows the current keyword list and suggests renaming to enable.

The badge updates in real time as you rename an experiment, no page refresh required.

Troubleshooting

"I renamed the experiment but notifications still aren't firing"

Most common causes:

• Name doesn't start with the keyword. The keyword has to be at the very beginning. Homepage LIVE test does not match; LIVE Homepage test does.
• Missing separator. LIVECheckoutcopy does not match: there's no space, dash, underscore, or colon after the keyword.
• Nothing after the separator. LIVE - alone does not match.
• Destination disabled or has no route for the event. Check the destination card on the project's Notifications tab. The badge can read "On" while a specific destination is paused.
• Keyword was just added. Events fired before the keyword existed are not retroactively delivered.

"I'm seeing notifications for a QA experiment: why?"

Two possibilities:

• The name accidentally starts with a keyword. Something like Live-test-thing matches the built-in LIVE. Rename the experiment: LIVE / PROD are built in and can't be removed. For a custom keyword, you can instead remove it from the org config if no one is using it intentionally.
• It was a safety event. SRM and guardrail events always fire. The delivery row will show bypassedGate=true, that's by design.

"We haven't added any custom keywords"

You don't need to. LIVE and PROD are built in, so experiments named like LIVE - Checkout notify out of the box. With no custom keywords added, every other experiment stays silent, except safety events, which always fire. Add custom keywords in the Keywords panel to extend the convention.

Related