Android Friday: How should you distribute test builds of your app?


Your company is developing an Android app. Along the way you want to distribute test builds of the app. The designer needs to do user testing out in the field, the backend developer wants to end-to-end-test the whole system from the app… oh and by the way: the CEO wants to check in periodically as well.

If you were building a web-app it would be easy: just email a link to everyone.  Distribution happens every time they reload their browser. However, with native apps you need to periodically push out new versions of the app to their phones.

You could do it manually: just tell people to swing by your desk, plug their phone to your laptop and deploy the app directly. This scales poorly, though. It takes time, is error-prone, interrupts the development work and does not work for people who are offsite.

Better to automate it, then. It’s easy to configure a continuous integration system such as Bitrise, where you can set up automated releases to your team, your stakeholders and test users. You should aim to be in a situation where new app builds are automatically distributed to testers every time you push your code to a stable release branch.

There are multiple solutions that solve the actual distribution step, services that help you push the app out to actual user phones without having to publish a fully public app in Google Play. Two mainstream approaches are Google Play closed beta, as well as Fabric Beta (formerly called Crashlytics).

Which one should you use? Well, from the developer side it does not really matter. It’s easy to integrate any of these options with CI systems, there’s just minor differences in how quickly you can get new app distributions up and running.

However, there are significant tradeoffs in terms of the user experience for the end user/tester of your app.

Fabric Beta

  • No manual setup for each new app you start distributing: you don’t have to set up a Google Play account for it, just push the app to Fabric and it becomes available for distribution.
  • Since the app is distributed outside Google Play, the users do not need to tie it to their Google account, they can just give you an arbitrary email where they want to receive notifications about new app versions.
  • The tester gets download access to all previous builds of the app through the unifying Beta app, making it easy to go back and forth through app versions. If you develop multiple apps (or flavors of the app), they can access all of them in a single place through the Fabric Beta app.
  • Unfortunately, since Fabric Beta sidesteps Google Play, getting users set up is a little quirky. Android phones will not just their owners install random .apk app files directly, unless the user opts in to some somewhat scary-looking popup dialogs to set up exceptions for your app. Fabric tries to onboard new users as smoothly as they can, but you may still have to help external stakeholders and non-technical users get started with this. The user experience of installing apps this way may feel a little alien and confusing to them.
  • The test app is not automatically updated when you push out new versions: Fabric prompts the testers with email and push messages telling them that a new version is available (with a one or two-click installation process).

Google Play closed beta

  • You will need to set up a Google Play account for the app, so each new app that you want to distribute this way needs a little bit of manual work up front.
  • Users will need to supply their Google account to get access to the app.
  • Users only see the latest version of the app.
  • New versions of the app are auto-installed without need for users to actively do it themselves.
  • The users will not receive the auto-updated new versions of your app at the same time, since auto-update is influenced both by Googles content distribution network and how quickly the user phone decides to upgrade the app once available.
  • The user experience of getting started is less confusing and “scary” for non-technical users.

Which one should you pick?

Fabric Beta is good for internal testing within your own company, for quick and disposable demos and prototypes, and situations where you work quite closely with the people who do the testing.

Google Play closed beta is good when you need to distribute to external stakeholders and non-technical people, and when you need to put your best foot forward with easy onboarding and a familiar user experience.

And finally: nothing stops you from using both these approaches at the same time!