App Onboarding
Part of Mobile growth programme reviews
Planning experiments around the weakest journey stage
Check a weak app journey stage, define a focused change and outcome, protect the comparison and decide after its window closes.
Test a weak journey stage only after confirming it is measured correctly, its users are eligible, and improving it could help the final task. The largest percentage drop is not automatically the best test. A confirmed blocking defect may call for repair first.
Qualify the stage
For each candidate, record eligible people who reached it, confirmed completions and the time allowed. Check whether another valid route bypasses the measured step. A displayed gap can therefore reflect configuration or a valid route, not only a product problem.
Consider the number of affected people, the consequence of failure and whether the team can change the stage. A rare payment error blocking a vital task may deserve direct repair even if another stage has a larger measured drop.
Write one experiment question
Suppose a hypothetical appointment app finds that eligible people view available times but fewer confirm a booking. Its question might be: “Does showing the appointment location before time selection increase confirmed bookings?” The change is the earlier location display; the primary outcome is a confirmed booking.
Errors, cancellations and support contacts could reveal a worse experience despite a higher booking rate. This is a proposed test, not a result.
| Field | Rule to set before launch |
|---|---|
| Eligibility | Who can encounter either version and genuinely book? |
| Assignment | What stable person or app-instance rule assigns a group? |
| Change | What differs between baseline and variant? |
| Outcome | Which verified state counts, and by when? |
| Safeguards | What could worsen while the primary rate rises? |
| Decision | What finding would support keeping, revising or leaving the change unresolved? |
Protect the comparison
Assign eligible participants before the experience can affect whether they remain in the analysis. Count everyone assigned under the defined rule, including people who leave before seeing the change. Comparing only variant viewers with everyone assigned to the baseline would select people who stayed long enough to view it.
Firebase Remote Config A/B Testing is one possible implementation. Its documented variant assignment uses a hash of the experiment ID and Firebase installation ID for users meeting the targeting conditions. Google Analytics must be enabled for experiment outcome data.
Installation-level assignment does not automatically answer an account-level question across devices. Firebase also cautions that changing app behaviour during a running experiment can affect results. The team must check whether this implementation fits its actual task and identity rules.
Decide after the window closes
Allow the outcome and safeguard windows to finish. Compare counts and rates, check whether assigned groups received their intended versions, and inspect relevant platform or app-version differences.
If a variant improves the selected step but not the final task, investigate whether the step was a poor proxy or another obstacle remains. If the evidence is too thin, record the result as unresolved.
Keep the stage definition, assignment rule, change, dates, concurrent changes and decision together so the next experiment can build on what was learned.
