App Store Listings
Part of App store listing experiments
Testing an app icon without changing every listing asset
Compare an app icon on the App Store or Google Play while keeping the treatment identifiable and reading the result within its limits.
To test an app icon, make the icon the identifiable difference between the current listing and its variant. Keep screenshots, previews and copy steady where the setup permits. The result covers that icon comparison among eligible store visitors; it does not assess a wider listing redesign.
Choose the icon question
State what the alternative might help a visitor recognise at a small size. A simpler symbol, for example, might make the app’s main task clearer.
Save the current and proposed icon files and the reason for the design change. If both the symbol and colour change, treat the alternative as one complete icon; the result cannot isolate either feature.
Check that the alternative accurately represents the app and works as an installed icon, not just as a store image. More variants are possible, but they divide available experimental traffic.
App Icon Test: Original vs. Alternative
- Original Icon
- Current app icon used in the App Store listing
- Alternative Icon
- Simplified symbol with updated colour; designed for clarity at small sizes
- Design Rationale
- Improves recognisability and task clarity at small sizes on device home screens
Set up the relevant store test
On Apple’s App Store, Product Page Optimization treatments begin as copies of the original product page. Unedited screenshots and previews stay original.
Test icons must be included in the binary of the app version currently on the App Store, with supported alternate icons. A user who downloads from an icon treatment sees that icon during download and on their device.
Note any concurrent release, screenshot refresh or campaign change. Apple warns that a new version containing tested assets or metadata may affect a running test.
Set Up an App Icon Test on Apple App Store
- Prepare Icon FilesSave both original and alternative icon files in supported formats (e.g., PNG, SVG)
- Add Alternate Icons to BinaryInclude tested icons in the app binary version currently on the App Store
- Create Product Page Optimization TestUse App Store Connect to set up a treatment that mirrors the original page but replaces only the icon
- Monitor for ConflictsAvoid concurrent app updates, screenshot refreshes or campaign changes during testing
Read the result as an icon comparison
Choose the store outcome before launch. Apple reports estimated conversion, lift and confidence against the original product page. A store result does not show whether people complete a useful task after opening the app.
Thin traffic may leave the choice unresolved. Apple’s test results appear after at least five first-time downloads are attributed to the test and can later be labelled “Likely to be Inconclusive”. Keep the status and uncertainty beside the icon decision.
Apply the alternative only if the result and the product’s current identity support it. Otherwise, retain the original or plan a clearer follow-up.
Key Metrics from App Icon Test Results
- Estimated Conversion Rate
- Measured against original product page
- Lift in Conversions
- Percentage improvement over original icon (positive or negative)
- Confidence Level
- Statistical confidence in the result (e.g., high, medium, low)
- First-Time Downloads Attributed
- Minimum of 5 required before results are considered valid
