Statistical significance in Jurni Tests

Statistical significance helps you evaluate whether a test result is strong enough to act on. In Jurni Tests, you can choose a Control variant, set a confidence target, and see how each alternative performs against that Control.

This guide explains how to set up significance, read your results, and decide when to end a test.

Statistical significance currently uses conversion rate (CVR). Other metrics, including revenue and average order value, remain available for context but do not determine statistical confidence.

Key terms

A higher CVR alone does not mean a variant has reached your confidence target. Read lift and confidence together to understand both the size of the difference and the strength of the result.

Set up a new test

1. Go to Tests → New Test.

2. Under Setup → Goal, select CVR as your success metric.

3. Choose your confidence target: 85%, 90%, 95%, or a custom value.

4. Add your variants under Variants.

5. Mark one variant as the Control.

6. Complete your test setup and launch it.

Choose a Control that gives you a useful baseline. For example, if you are testing a new landing page against your current page, use the current page as the Control.

Choose a confidence target

Your target determines how much statistical confidence you require before treating a result as conclusive for your test.

• Lower targets accept more uncertainty and may be reached sooner.

• Higher targets require stronger evidence and may take longer to reach.

Choose the target before launching, based on the importance of the decision and your tolerance for uncertainty. Avoid lowering it simply because a preferred variant has not reached the original target.

Enable significance on an existing test

Existing tests continue to show their performance data. To enable significance:

1. Open the test.

2. Under Variants, mark one variant as the Control.

3. Review the success metric and confidence target under Goal.

4. Save your changes and open Results.

Without a Control, Jurni can display test data but cannot calculate statistical significance.

Each variant needs at least 500 sessions before confidence is shown. This is the minimum for displaying confidence, not a guarantee that the result will reach your target.

Read the Test Status card

The Test Status card gives you a quick view of where a live test stands.

An early lead is a reason to keep watching the test. It is not yet a result that meets your chosen confidence target.

Understand the Results view

Open Results to review each variant’s:

• CVR — its conversion rate.

• Lift vs. Control — how its CVR compares with the baseline.

• Confidence — the reported statistical confidence for the comparison.

• State and progress — where the result stands relative to your target.

For example, “Confidence 66% of 85% required” means the current confidence is 66%, while your target is 85%. The target has not been reached. This is not a countdown or an estimate of how much longer the test needs to run.

Example

Suppose your test shows these illustrative results:

Variant B’s CVR is 0.4 percentage points higher, which is a 10% relative lift over the Control.

That tells you the size of the observed improvement. Check the confidence separately: if your target is 90% and the result is below 90%, it has not yet met your decision threshold.

Every variant is compared against the Control. A variant beating the Control does not, by itself, establish that it beats every other alternative in the test.

End a test and declare a winner

When you are ready to make a decision, open Results and use End & Declare to end the test and declare a winner.

Before doing so, check:

• Whether the result has reached the confidence target you selected.

• Whether the lift is meaningful for your business.

• Whether other metrics, such as revenue and AOV, support the decision.

Tests do not end automatically when the confidence target is reached. Jurni also does not currently send an alert when the threshold is reached, so review results in the platform.

You can end a test before it reaches the target. If you do, treat the outcome as a decision made with less evidence, rather than a result that met your original threshold.

Frequently asked questions

Why is confidence not showing?

Check that a Control is selected and each variant has reached the minimum of 500 sessions. Without a Control or sufficient data, confidence is not shown.

How long does a test need to run?

There is no fixed test duration. Reaching 500 sessions per variant enables confidence to be shown, but the test may need more data to reach your selected target. Some tests may remain inconclusive.

Can I use revenue, AOV, or add-to-cart as the success metric?

Not currently. Statistical significance is calculated using CVR only. Other metrics remain visible for context. Additional success metrics are planned, but no release date is committed.

Does reaching confidence guarantee the same performance in the future?

No. Statistical confidence helps you interpret the evidence from the test. It does not guarantee that the same lift will continue after the test ends.

What if no variant reaches my confidence target?

You can continue collecting data or end the test without a conclusive result. Avoid treating the variant with the highest observed CVR as a confirmed improvement simply because it is ahead.

Do I need to calculate significance in another tool?

No. Once a Control is selected and the minimum data requirement is met, Jurni displays CVR, lift, confidence, and progress toward your target directly in Results.