Reading results

An experiment's report exists to answer one question — is a variant beating control on the metric you chose? — and to be explicit about how sure that answer is. Everything else on the page is context for that answer, not a second verdict.

The primary metric

Every experiment names one primary metric at creation: conversion rate (orders per measured session), revenue per visitor, or average order value. The leader, the verdict, and the end-of-experiment decision all follow this metric; the others are still reported, but as context. The primary metric is editable while the experiment is a draft and locks once it launches — re-reading data that was collected under one question as if it had been another is how experiments get retold rather than read.

When sessions cannot be measured directly, conversion rate falls back to a labeled proxy — the label is on the number, so you always know which read you are looking at.

Two statistical reads, side by side

• Bayesian: for each variant, the probability it beats control, the probability it is the best variant overall, and a 95% credible interval on the difference. This read is always available and is what the verdict uses.
• Frequentist: a classical significance test shown alongside — but only when the counts are large enough for it to be valid. When it is absent, that absence is the information: the sample is too small for this kind of claim, and Proa would rather show nothing than a p-value that lies.

Two schools agreeing is a strong signal. Two schools disagreeing is a reason to keep the experiment running.

Enough data, or not yet

Before you start, a duration estimate — seeded from your own site's traffic — tells you roughly how long an answer should take. While running, the report says plainly whether there is enough data for a decision. The report reads a window you choose — lifetime, the last 7, 30, or 90 days, or a custom range — and states the exact dates it covers; numbers are aggregated periodically, not live.

The funnel

Below the headline numbers, a funnel per variant: sessions, how many reached the cart, and how many ordered. On stores with the Proa Shopify app it gains two steps from the checkout itself — Started checkout and Payment intent — counting only checkouts the shopper's privacy consent allowed the app to see, and saying so instead of guessing. The funnel is descriptive — there is no per-step verdict — but it shows where a variant wins or loses. Steps that cannot be measured are marked as such, never silently zero.

Coverage among paid orders

On stores with the Proa Shopify app, a section below the funnel answers a question most reports skip: how much of the checkout is actually being measured? For each checkout step it shows how many of the paid orders in the window were also seen at that step, and the share. Shoppers who decline analytics consent are invisible to the checkout measurement, and this number shows the size of that gap among the orders you actually received — it is not an estimate of abandoned checkouts. While the sample is small, the report says so on the number.

Slices and context

• By audience: the same numbers broken down by device, country, source channel, source site, and landing page. Small buckets fold into "Other". These slices are descriptive; running twenty slices and celebrating the one that turned green is the oldest false positive in the book, and the report will not do it for you.
• Currencies never mix; revenue is reported per currency.
• Attribution reliability — unattributed order share — sits next to the revenue numbers; see Revenue attribution for how to read it.
• Every view exports to CSV, so your own analyst can re-derive everything.

Asking questions

Analyse this experiment produces a written reading of the current numbers, and the follow-up chat answers questions about the same window the report shows — the same data, in prose. It cites what the report says; it does not have access to anything the report hides from you, because the report hides nothing.