Release management: successful launches and updates on Google Play (Google I/O ’18)
Articles Blog

Release management: successful launches and updates on Google Play (Google I/O ’18)

August 8, 2019


[MUSIC PLAYING] MIKE DAVIS: Hi. Welcome, I’m Mike, an
engineering manager on the Google Play Console. And I’m very happy
to be here today to talk to you about
a subject that’s particularly important
to my team and I, and that is enabling
success on Android through safe, controlled,
effective app releases. We know from yourselves,
from our own experience, and the experience of
other teams at Google, that releasing new
apps and updates is what you might describe
as an exciting time. It’s the time when
all that hard work the whole team has
been putting in finally gets into the hands
of your users. And that is an
incredible feeling. Users are going to be delighted
by your new functionality and improved performance. Business metrics are going
to go through the roof. And everyone is
going to be catching on to that good
kind of excitement. In essence, for a
good release, we need to delight users
with new experiences. We need to maintain and improve
those important business metrics. And, of course, we need to
get that tech right to deliver a stable, high quality product. Simple, right? But releases, they also present
us with some challenges. We need to keep our
existing users happy. We need to make sure
we’re actually improving those key performance
indicators, and of course, not introduce performance
regressions or bugs. And if we do get
this wrong, it can lead to an altogether
different kind of excitement. A miscalculated release may
result in crashes and bad user experience, and this can have
a prompt and measurable impact. We crunched some
numbers on the subject, and our data shows users are
26% more likely to give one star reviews on the days when
your app is at its crashiest For games, this increases to
a staggering 52% more one star ratings. Similarly, in
reviews, for reviews given on those
crashiest days, users are 38% more likely to
mention stability and bugs. And as you might expect,
this is not just limited to ratings in the store. Key performance indicators
are also affected. Engagement metrics
and in-store numbers also suffer when
your app is crashing. On top of all this,
let’s take a moment to think about the ecosystem
in which you’re operating here. At more than 2 billion
active devices, Android is, in fact, the
largest software and computing platform on the planet. Of course, this presents
phenomenal opportunity for scale and reach to influence
an incredibly large diverse set of users. However, it also presents
some unique challenges when it comes to
software deployment. That incredible
diversity in users also comes with unprecedented
diversity in device types, with thousands of
different makes and models across a variety
of different form factors. And so to have great
releases and to keep with that good
kind of excitement, we need to listen and validate
functionality improvements by collecting feedback
from a representative set of engaged, enthusiastic users. We need to measure
to make sure we’re having the right impact on those
key user and business metrics. And we need to optimize
that technology by detecting those bugs
and stability issues. Not only this, but
ideally, we want to do all this as
early as possible. The sooner we catch
problems, the better. Be them bugs, usability issues,
or business metric regressions. The sooner we catch
problems, the more scalable our processes are, the more
effective our teams can be, and the faster we can ship
even more great features. And this is where the
right tools and release process comes in. For the next part
of this session, we’ll walk you through
the suite of tools that the Google
Play Console makes available to help you handle
this process confidently and effectively. We’ll look at deploying
your release to increasingly large audiences,
and what you can hope to achieve at
each phase to reach the ultimate goal of
deploying a great new app or update to everybody. And after that, we’ll welcome
Radha from the Android system health team to give us a deep
dive on the Google app release process. We’ll see how teams at Google– big and small– are making
great use of these tools to scale their
release processes. And so to begin
all that, I’d like to welcome on stage, Serge, an
engineer on the Play Console, to talk to us about the first
phase of our release, testing. [APPLAUSE] SERGE BEAUCHAMP: Thanks, Mike. First, I’d like to
do a show of hands, who amongst you have published
an app in the Play Console already? Wow. And please, keep your
hands up if you also have been using the alpha
and beta track for your app. Well, you guys are great. I think you’ll love this talk. Of course, for those
who haven’t yet published an app in
the Play Console, a track is a way to
define a group of users to whom you can
publish your app. For example, there’s
the production track which targets all the over
2 billion Android users. And then, there is the
alpha and beta track which targets a much
smaller subset of users. Of course, before you publish
your app to the world, testing your app with
that small group of users is really important. This is for catching
bugs, issues, but also to scale and move faster. Key to this is distributing
your test version of your app to a small group of users. This could be either your wider
team, product managers, a QA team, or whoever can provide
you with early feedback. There’s lots of ways to do that. One way would be to simply
email APKs to your users, but that’s quite cumbersome
and actually that puts a lot of burden
on their shoulders. While the alpha
and beta track can be used for
distributing test APKs, we heard from you
that, actually, the alpha and beta track
are too slow for that kind of fast, iterative testing. And this is because–
as we’ll see later– the alpha and beta track are
designed for scale testing with potentially
millions of users. And the APKs published
to those tracks goes through processing
that can take several hours. To enable fast,
iterative testing, we introduced the
internal test track. The internal test track is a
new track in the Play Console that is designed to distribute
test versions of your app to a small group of users
within your organization. Let me show you how it works. Switching to the
computer, here, I have my test app
that they created for this demo called I/O demo. And I have it here
in the Play Console. This is the App Release’s page. And we can see I
have the production track, the beta track, the alpha
track, and a new internal test track. You can see here
I have version– actually, I have version
49 already published in a release in this track– version 49, here. I have one tester for this app– switching to the device. And you can see this– can you see on the device? Very soon, yes. So this is my testers
device, and you can see I have this I/O
demo app in the launcher. When I open it, there’s
a single activity, and it shows the version code. I’m guessing that your app
is a bit more sophisticated. If I go in the Play
Console, I can see that I don’t have any updates. So the version code of my
APK on my tester’s device is the same one that
is in the release that I rolled out in
the internal test track. Now, Mike here just told
me that actually my app doesn’t conform with
the official Google I/O branding for the color scheme. So I will go and
fix this and quickly distribute an app to my tester. I’m going back to the computer. So we’ll go to Android Studio. Here, in Android Studio,
I have my project already open with my app. I also have installed a
Gradle Play Publisher plugin, which is set up so that
it automatically published the Play Console updates
and builds I make of my app. I have a gradle task here– publish APK release–
and when I invoke it, it automatically pushes the new
test track to build on my app. So I will create a new version– version 50. And then, I will change my
color to, what was it again? Space pineapple. So what will happen is that
when I invoke this gradle task, it will automatically
build the APK and then upload it
to the Play Console, to the internal test track. Once the upload completes,
I will start a timer here. I have another
device with a timer, and I set the timer
for two hours. I hope you guys
are not in a hurry. This is because this is
how long you should expect it to take if I was to
publish this APK to the alpha and beta track. Then, I will go to the Play
Store on my testers device, and I will check for an update. And as soon as the
update is available, I will stop the timer, and
we’ll see how long it takes. So I start a task. So let’s switch to
the device, please. So Android Studio
is building the APK and has started to upload it. Of course, I don’t usually
do this in a Wi-Fi network in a tent shared
by 2,000 people, but hopefully it still works. So now the upload has completed. If I go in my tester’s
device, or I start the timer. I check for an update. Oh, my update is here already. So actually, that took
less than five seconds. So if I update my app,
it installs the update. I open it. And I have my new color scheme
with my new version code. [APPLAUSE] This is dramatically faster
than publishing to the alpha and beta track, as you can see. Actually, it’s quite amazing. I’ve been advised against
doing a live demo on stage at Google I/O with
a real device, but you guys deserve it. So we’ve seen how you can
publish to the Play Console using a gradle plug-in
in Android Studio. But of course, you can
also use the Play Console UI to do just that. You can set up testers– sorry– skipped a slide. So you can set up
testers by specifying a list of email addresses. The testers opt in
with a single URL. Then, the testers use the
Play Store app on the device to install and update your app. It’s super simple for
testers to set up. It works on any Android
device with the Play Store. And there’s no special
app or permission. It’s fast, safe, and simple. Sorry– there’s two buttons here
and 50% chance of missing it. In order to get
APKs out quickly, we distribute them to
testers before the processing of the APK has completed. So although the publishing
of the APKs is instant, the APKs and bundles published
in the internal test track are still subject
to Play policies. The update for internal
testers is also different than from alpha beta. So users can be either
alpha beta testers or internal testers. There’s also a cap of 100
testers in the internal test track per app. The new internal test track is
supported by the Play Console publisher API and by many Play
Console publisher plug-ins. So that means that you
can automate publishing to the internal test track
very easily from your existing toolchains and workflows. You can get up and running
with very little change. The internal test track is
great for quick iteration during development, for
catching bugs early, for getting fast
feedback from your team, and also for testing
Play integration. Because your app is
published through Play, it supports all
the Play features– LVL testing, in-app purchases,
and most importantly, the new Android app bundle and
dynamic developer features. So if you’re thinking about
trying out those features, the internal test track
is a great way to do that. Once you have your
app tested internally with the internal test track,
you’re ready for real users. Although, typically,
you’re not yet ready to open up to everyone. You might be looking for
trusted feedback on maybe a confidential set of features. Closed testing allows you
to scale up your testing, while maintaining control
of who can get your app. The alpha and beta track have
been supporting closed testing for some time now,
but we heard feedback that as your app scales,
the alpha and beta track don’t scale well with
various testing scenarios. What if, for example,
you would like to test your app with a track
for each feature branch? Or what if you would like
to create a new track for new events or milestones? Well, I’m delighted to
announce that now you can create new, additional
closed testing tracks, enabling all those scenarios. You now have one production
track, one open testing track, any number of closed
testing track, and an internal test track. When you want to create an
additional closed testing track, you can just click on
the button, Create Closed Track. Choose any name you want,
and that name is also used in the API. From our early
adopters, we have seen that this is great
for user studies, country-specific
variants, printing feature branches, and testing in
incrementally larger groups. I’m just scratching
the surface of how those new tracks can be used. For those who have been using
the alpha and beta track, you may have heard
of pre-launch report. It takes APKs– APKs that you publish
to the Play Console– and install them on 10 different
unique devices in test forms. It runs your app
for five minutes at a time with robo for nearly
an hour of testing in total. It shows crash reports,
performance issues, and security vulnerabilities. Pre-launch report
also added new ways to control the crawl,
like robo scripts and demo loops for games. They are announcing
exciting new features. Go see their awesome
talk tomorrow at 9:30. Be sure to set an
alarm for that. Or if you’re like
me and you still wake up on a different
time zone, you’ll be set. As we’ve seen, closed
testing is great for once you want to test your app with
a set of trusted testers, but typically, the
problem with those users is that they’re not very
representative of production users. Obviously, production users
have a wider set of devices, but they might not be as
motivated as trusted testers and might engage with a
different set of features on your app. When you want to have broad
coverage of your code, find technical issues, or get
more relevant business metrics, you can use open testing. Releasing to open testing– allow any user to discover and
join your testing community directly from the Google Play
Store app on their device. And these users can provide
you private feedback that you can review
in the Play Console. And those feedbacks won’t
affect your public ratings. This is a great way to
scale up and to make sure that the new version
of your app provides a great experience to
users before you launch it to the world. If you need to you can also set
a cap on the number of testers or of users that join that
testing program to keep things manageable. Of course, open testing
is only as valuable as users are willing
to participate. Today, there are over
230 million users who already opted in open test,
with two and a half million users opting in each week. So it’s really up to you
to put it to good use. Late last year, we also
introduced a new country targeting for testing tracks. With country
targeting testing, you can now control the country
availability for each track individually. This can be used for both
open and closed testing, and allows you to run a test
in a different set of countries to those in which a
production app is live. We have seen some
of the features Play provides to help
you test your app. It’s time to go to the next
step in the release process. And for that, I’m going
to hand it over to Nick, another engineer on the
Play Console, who will talk to you about rollouts. [APPLAUSE] NICHOLAS LATIVY: Thanks, Serge. So you’ve tested internally,
then in closed groups, and finally, using open testing. You’ve polished the
user experience, optimized those
business metrics, and ironed out those bugs
and performance issues. You’re confident this
new release has it all. The only thing left to do is
roll it out to the public. Last I/O, we unveiled a
new gated release flow for managing your release. Thank you for all the
feedback you provided. Based on this input,
we have further streamlined the process,
making it faster to provide translated
release notes and enhancing the
APK upload process so you can drag and drop all
your APKs into your release in a single step. We have also introduced support
for Instant Apps, Instant Games, Actions on Android, and
the new Android App Bundle. In preparation for the
new Android App Bundle, we have introduced
a new UI to help you understand the contents
of your APK or bundle. Even if you aren’t using
Android App Bundle, this provides useful insight
into the sizes of your APKs. Keeping your APKs as small
as possible means users can download and install faster
and have more free space left for their photos and music. On average, apps will
see a 20% size reduction by using Android App Bundle,
and the Bundle Explorer can show you exactly
the size saving you can achieve for your app. We recommend all developers
use stage rollouts when rolling to production. Stage rollouts allow you to
control the percentage of users eligible to update and
gradually increase this as your confidence
in the release grows. In the unlikely event
that something goes wrong, you can also halt the release
where only a small percentage of users are affected. We’re really happy to see
that since introducing the new gated release
flow, twice as many of you are making use of
stage rollouts. Following on from this, we
have extended stage rollouts with the new country
targeted rollout option. This allows you to rollout
per country in addition to by percentage. This is really useful to
start your release in a market that you have a deep
understanding of, making it easier to understand
your business metrics as the rollout progresses. Once your release
is rolling out, you can use the release
dashboard in the Play Console to monitor key metrics, such as
update rates and app ratings, as well as crashes and ANRs. These metrics help you
evaluate your release and ensure it remains healthy
as you proceed with the rollout. The Statistics view now
shows release events, making it even
easier to understand the impact of a rollout. For a more detailed
view, you can use the Vitals page
to check metrics like battery usage, jank,
wait lock usage, and others. While we’ve introduced
many exciting new features for releasing your app
through the Play Console, we know that many of you
automate your APK deployment with continuous
integration systems or using gradle plug-ins–
as Serge did earlier. So we’ve updated
the publishing API, and from today, you
can use these features in your automated workflows. Rolling out a
release via the API now uses the same
release model you’re familiar with from
the Play Console. This allows you to
name your release and more easily
modify release notes. In addition, we’ve
relaxed the constraints on what you can rollout via the
API to match the Play Console. The new testing
tracks Serge mentioned earlier are, of course, also
now supported by the API. This is great for configuring
your continuous integration system to push through
your internal test track whenever you have a
new build ready for QA. The new release model
also unlocks some features that were previously, only
available via the Play Console. For instance, it’s now possible
to halt a release using the publishing API. This allows you to fully
automate your release process, including automatically
responding to problems. We’ve heard from many
of you that while you’re comfortable rolling out
test builds via the API, you like the control of
coming to the console when rolling out to production. So with the updated API,
we’ve added the ability to create and manage draft
releases via the API. This means you can
stage a draft release from your continuous
integration system and then have your
product manager login, check everything looks good,
and hit CONFIRM and Rollout. You can find the full details
of the new API changes and the associated client
libraries on our Developer pages in GitHub. To end this session
off, I would like to welcome Radha to
the stage to tell us about how the tools
we have looked are put to use at Google. [APPLAUSE] RADHA NARAYAN: Hi, everyone. My name is Radha, and I
run product operations for Android System Health. My team is responsible
for making sure that all of Google’s
Android developers can build excellent
apps quickly. I’m here to talk to you
about some of the lessons we’ve learned in working with
Google’s many app developer teams, how we use some of the
tools you’ve heard about today, and why we love them. Google has many Android apps. Some of them you may
have heard about, like Gmail, Maps,
YouTube, or Search, which serve billions of users. But we actually have over
150 apps, many of which are just starting out and have
a few hundred or a few thousand users. Some of these apps have
small developer teams, up to 10 people total, while
others have hundreds or even thousands of Googlers
working on them. In this room as well,
there’s probably a similar diversity in
terms of how big teams are and how many users you serve. Smaller apps are
looking for ways to scale rapidly to
a growing user base. And larger apps are
looking for ways to innovate safely and quickly. In working with
Google’s teams, we found that even this
vast range of apps have a few things in common. Each of these apps is
constantly evolving. Each is looking to add
new and exciting features and adapt quickly
to user feedback. And they would all rather
avoid that bad update and all of the implications
that we heard about earlier. What I’ll be walking
through today is how we’ve come to align
on some common tools and best practices that scale from the
smallest to the largest apps. So if we go back to our
key points on releases, we need to make sure
that we can listen to our users and the
feedback they give us. We need to make sure we can
maintain, validate, and improve the key user and
business metrics. And we need to
optimize the technology to avoid bugs and
performance issues and to support the incredible
diversity of users and devices out there. How do we map this to Google? And how do we scale? And as teams grow in
size and ambition, how do we continue to achieve
good release attributes, while still moving quickly? Well, we’ve developed some
rules of the road for how we can do testing and app updates. Using the tools we’ve
heard about today, we’ve developed a few
strategies to create a consistent process that
scales, even to the deployment of our largest apps. I’ll be going over
some of the strategies that Google App developers
are taking advantage of to test out new features and
to release the next versions of their apps. First, let’s take
a look at testing. We used to test
by always putting the raw, most bleeding edge
features into our alpha track. This ensured that the team could
stay in tune with each other and with the latest and
greatest as we made progress. Unfortunately, as the team
grew, this led to an issue that we called, you
did what to my VP demo? Teams need to be
able to move quickly, to experiment, and to
occasionally, break stuff. But product managers need
fresh, at most a day-old, stable builds, to show off the
latest features to executives and other stakeholders. We might need a stable
binary, let’s say, to do an I/O demo, that has
a lot of the new features. With more testing
tracks, we’re now able to give individual
teams their own builds where they can experiment
independently of each other and independently of
the release process. They can deploy daily or
even hourly, if they wish, to share the latest
features among themselves. We can also set the country
availability of the testing tracks to global to allow
Googlers in any country to test our apps, even if
the released product is not yet available globally. More tracks also allow us to
have a more thorough testing process at release time. Thanks to individual
team tracks, new features have already been
tested independently, even before we cut our release. Once we’re ready to
release, we promote our build through multiple
stages of testing, starting with a team-wide
test of tens of people, to a Google-wide test
with thousands of users, and then a public, open beta
with potentially millions of test users. Our open betas
allow external users to try our latest
and greatest features if they’re willing to give us
private feedback on if they find they like it
or if they find the ride is a little bumpy. At first, we imagined that we
might have a hard time getting users to sign up
for this, but users can join direct from the store
listing in the Play Store app. And some of our apps
now have millions of users in their open betas. With additional testing
tracks at our disposal, more and more Google Apps
are setting up an open beta. This quote shows how the
Google Search app is benefiting from the open beta program. It also highlights that
testing is not just about mitigating the
risk of bugs and crashes, but it’s also a chance to
get feedback on features and to ensure that the new
features have the right impact on key performance indicators. You might be
wondering, what we get from all of these various
stages of testing? Why not just go
straight to open beta? Well, we might want to test out
some confidential new features before they are visible
to external users. Why do we have both a team-wide
test and a Google-wide test? At Google, we have a
tradition of dog fooding our own products,
and we’d rather not break Gmail for all
Googlers in our tests. So why do we still
need the open beta if we have so many
Googlers testing? Well, it’s because even if
a Google-wide test can reach thousands of users,
by and large, these users tend to have a
somewhat narrow set of Android devices. An open beta gives
us much more scale and a device set that is
much more representative of our production users. So that’s testing. We’ve validated our
features and our app update. Let’s move on to talk
about how we actually rollout these updates
and features to users. A staged rollout is critically
important to Google teams. Rather than updating 2
billion devices in one go, we always start small and
ramp up incrementally, keeping a close eye on business,
user, and crash metrics. At the first sign of any issues,
we can always halt a release. But how do we actually
manage these releases? One method we tried for
managing an app release was to wait until all
the features scheduled for that release were
ready, then cut and rollout the release. This worked great at
first, but unfortunately, features sometimes– just sometimes– hit
a snag causing delays to all the other features
scheduled for the release. Soon, the May release
becomes the June release. And now you’re pulling
more and more features into that single release. More features means
more testing, more bugs, more release
candidates, more changes that make it hard to
track down exactly what caused any particular bug
or performance regression. And since the next release
might be a while away, developers scramble
to get their features on board the release train. And then, in the
meantime, you discover you need to ship an urgent
bug fix that just can’t wait. And as you can see, as the team
scales, this tends to spiral. Waiting for all
features to be ready slows down innovation and adds a
lot of risk to the app release. We found that
counter-intuitively faster is safer. Keeping a continuous and
predictable release train makes each release smaller,
safer, and easier to manage. Those urgent bug fixes can
be sure to go out on time. And interestingly, knowing
that the next release is only a few weeks
away, developers tend to avoid cramming their
almost, but not quite finished, features into a given release. So this is a great
philosophy, but how does it work in practice? How do we keep the
trains running on time? And how do we ensure that
new features don’t go live before they’re ready? To get features out
safely in an app release, we use a technique called
flag-guarded feature development. Whenever possible, we try to
avoid shipping new features directly with an app update. We put each feature
behind a flag that can be toggled on
and off from the server. First, we used the stage rollout
to release the app update with all the flags turned off. By first deploying such
a changed neutral binary, it’s easier and safer to
get this update to all 100% of our users. This also allows us to evaluate
the application binary itself, for things like stability
and system health, as it installs on the
user’s device separately from the feature and how
the feature is doing. Once this change-neutral app
update is safely deployed, we can then slowly
enable each feature by turning on the flags. Again, carefully
keeping an eye on all of those important user,
business, and crash metrics to detect any issues. If we find an issue, we
always have the option to disable the feature
by turning off the flag. Importantly, keeping these
features behind flags and first deploying a
change-neutral binary means that any issue
with one feature won’t adversely affect all
the other features in the app update. I like to think of this
as satellite deployment, not that I’ve ever actually
deployed a satellite, but step one is to get
the thing into orbit. Once it’s safely
in orbit, then you can unpack each
instrument, one-by-one. You can achieve this in your
own app using Firebase Remote Config. The Google Search app is
one of many Google teams that have scaled
their release process and their feature launches
using flag-guarded development. So you might be wondering
how long releases take if we’re going through
each of these stages of testing and ramp up. Because the release
train stays predictable, we’re usually able to deliver
most app updates to users within a couple of weeks, while
having the ability to speed up a release if
absolutely necessary. For the larger apps– the ones that have more
than a billion users– we’re able to
collect enough data to have confidence
in our releases within a couple of days. These release best
practices help us keep each of our 150-plus
apps at Google high quality without ever putting the brakes
on how fast we can innovate and bring fresh and exciting
features to our billions of users worldwide. So that was a tour through
the Google release process. I hope you found this
useful and that there are some ideas that you can
put to use in your own release process. And I’d like to
welcome Mike back on stage to close things off. [APPLAUSE] MIKE DAVIS: Thanks, Radha. So to recap, we’ve seen why
releasing is an exciting time, and how we like to
think about what constitutes a great release– a release that delights users,
improves business metrics, and is stable and perfomant. After this, we took a look
through the suite of tools that the Google
Play Console makes available to help you achieve
efficient, well understood, risk-free app releases. We looked to the
new internal testing track for fast,
iterative testing with your team and testers. We looked at closed testing,
where you– the developer– get to control exactly
who can test your app. We’ve seen how Google Play
is going beyond alpha beta with additional
closed testing tracks. And, of course,
open testing, where any user can opt in to test your
app direct from the Google Play Store. We’ve seen how this can bring
huge scale to your testing. We also looked at rolling
out your releases, with improvements to the
releases UI, improvements to stage rollout, and the
new improved publishing API, for more automation
of your release, using gradle and continuous
integration tools. And we took a whirlwind tour
through Google’s approach to creating a consistent
release process that scales from the smallest
to the largest teams. We hope you find
this talk useful. Please, do give
these tools a go, and let us know what you think. We look forward to hearing
your feedback as you use these tools more and more. If you’d like to
share your experiences or you have thoughts
or ideas on how we can make these
tools even better, please, do come and see us in
the Android and Play Sandbox in Dome C. Thank you. [MUSIC PLAYING]

Only registered users can comment.

  1. Thanks for improvements in Publisher APIs.
    Would appreciate guided write-ups for the same in medium or developer blog.
    Eagerly waiting…!

Leave a Reply

Your email address will not be published. Required fields are marked *