Bring the best of Google Photos to your app (Google I/O ’18)
Articles Blog

Bring the best of Google Photos to your app (Google I/O ’18)

August 9, 2019


[MUSIC PLAYING] RUCHI BEZOLES: Good
morning, everybody. I’m Ruchi, and this is Ben. And we are from the
Google Photos team. BEN GREENWOOD: We
built Google Photos to help people
solve the problems they were having
storing, organizing, and sharing their memories. And now, three years in,
we’re delighted that hundreds of millions of people
around the world call the product home for
their photos and videos. Here at I/O, we’re
announcing a new way for you, as developers, to be
able to use this platform to build photo and video
experiences that are smarter, faster, and easier to use. In the session
today, we’re going to dive into some of
those new technologies and show how you can use them to
bring the best of Google Photos into your own apps. Now, we designed
Google Photos and we built the app for the way people
use photos and videos today. This means users get
unlimited storage, so, yes, we can
do the inevitable and take 15 photos to
capture the perfect sunset, or automatic backup, so we
never lose our memories, even as we change
devices over time. And of course, we’ve
built the product with machine
learning at its core. And you heard Sundar
speak yesterday in the keynote about
the ways in which we’re applying AI to help
people organize and bring their memories to life. And one of my favorites that
the team’s been working on is what we call Color Pop. What this means is the
Google Photos Assistant can help you automatically
create images like this. And you can see how
this effect really elevates the subject in vivid
color, fading the background. And if you look closely,
the AI has even identified and segmented the flower
that this little girl has in her hair. It’s pretty neat. Now, these are just
a few examples. But of course, we all
use our photos and videos in different ways. When we look across the
apps and devices that are important to
people, we see a lot of workflows that are difficult
and tedious, like if you’ve ever tried to set up
a digital photo frame and you’re moving
files between devices. It’s a real chore just
to keep up to date. Or how after you go
on a holiday trip, your family memories
are handed to you on a USB drive or even a DVD. Like, what do I
even do with that? And we’re all constantly
downloading and re-uploading content between services. Maybe you’re posting about
your latest adventures, and you have some
awesome drone footage that you want to include. But it’s in 4K, and
you have to download and re-upload over mobile data. And so we think it should
be much easier for people to seamlessly use
their photos and videos across all the apps and devices
that are important to them. But to solve these problems,
we can’t do it alone. And that’s why, today, we’re
announcing the Google Photos partner program and
giving you, as developers, the tools and the
APIs you need to build amazing photo and video
experiences for your users. Now, as a developer, we know you
want to make your apps engaging and user friendly. And look, photos are a
great way to do that. But often, this involves
complex infrastructure for serving and having
to pay for storage just to duplicate the
user’s own content. So we think that
should be easier, too. And so at the core of
the partner program is what we call the
Google Photos Library API. This makes available some of
our best technology to you to be able to bring the
best of Google Photos and AI to help solve real
problems for your users. And here’s Ruchi
to tell you more. RUCHI BEZOLES: Thank you, Ben. All right. Now, I’m going to
show you how you can use the API to build
more engaging experiences for your users. There are three
main functionalities that the API enables. The first is that it
will allow your users to access their library of
photos in your products. The second is that you will
be able to upload photos to a user’s Google
Photos library. And third, it will
provide the infrastructure you need to enable sharing
of photos within your apps. It goes without
saying, but all of this requires explicit user consent. So let’s hone in on connect. One of the main things we wanted
to solve when building this API is to make it incredibly
simple for users to access and experience their
photos in their favorite apps and devices, many
of which you build. This could be in Timehop,
where once you’ve integrated your library and
given permission to use it, you’ll get more personalized and
creative reminders of memories to look back on,
or on HP Sprocket. It’s a really cute,
portable printer that will let you print
photos from your own library as stickers on the go. While this functionality
is fantastic, to provide your users access to
their library in your products, we also want to help developers
surface the right photos a user might need in context. And you can do this thanks
to the filters functionality. There are four ways you can
filter a library of photos. The first is content
categories, and that’s categories of photos
that a user has, whether it’s pet photos,
selfies, landscape photos, landmark photos, so on. Then there’s the date of
when the photos were created, so the time period
of photos and videos. And then there’s the date– then there’s the
media type, excuse me. So that’s whether it’s a photo
or a video or something else. You can pull those out, too. And finally, a
personal album a user already has created
in their account. OK. Time for a question. How many of you here have
ever had to do expenses? Raise your hands. OK, full room. Keep them up. Keep them up. How many of you would say that
you have loved doing expenses? OK, sad state. Yeah. Nobody really likes
doing expenses. Neither do I. But Xero, an app that
helps us do expenses, is now integrating
with the Library API. So what this means is they
can apply these filters I referenced earlier, and
they can pull receipt images in a certain time range and
surface them as suggestions of expenses I should do. So completing expenses
is as simple as viewing the suggestion and
hitting Tab to submit. BEN GREENWOOD: Pretty amazing. Thanks, Ruchi. RUCHI BEZOLES: Yeah. BEN GREENWOOD: So using
these smart filters is a really powerful
way that you can make it easier
for your users to find just the content
they’re looking for. And I’m going to show
you a few code examples. And as the library
API is a RESTful API, you can play along whether you
are a mobile, back end, or web developer. The first category–
or the first filter, excuse me, that we
were most excited about are content categories. And as Ruchi mentioned, these
use Google’s computer vision to figure out what
an image is about so you can filter and narrow
down results in your app. The API offers many
categories to choose from. And as you can see
from this example, I can also exclude
categories that might not make sense for my use case. For example, an
app like Timehop, which is there to really remind
me of my fondest memories, may not want to include pictures
of old bills and receipts and other documents
from years ago. And filters are
a great way to be able to exclude this content. Another way that users explore
their content is based on time. And sometimes, you’ll want to
present photos chronologically so you can scroll back in time. But there are other
use cases where being able to just jump
to the right moment can make it much easier for your
users to find their content. As you can see, you can filter
by specific dates or date ranges, and you can also
combine filters together to be even more expressive. Say I wanted to find all of my
birthday photos from years past and put them together
in a collage. I could sit there and scroll
through decades of photos to try and find
just the right pics, or I could combine a
date and content category filter like this. You can see how we’ve
emitted the year here and left it as a
wildcard so it’ll match any days on my birthday
and included a content category filter to find those
great selfies and birthday celebration moments. It’s pretty neat. Of course, you can also
filter by other kinds of media types,
so photos, videos, or other kinds of
content, like VR. And you can select photos
that are already in an album that a user has curated
for some purpose. But regardless of
the filters you use, you’ll get back a list of what
we refer to as media items. And here’s what one looks like. Google Photos supports a wide
variety of media formats, from the obvious photos
and videos to animations and even complex VR
and 360 degree images. So for each media item, you’ll
get back this consistent JSON object. And there are a couple of
interesting attributes here. The first is the
base URL, and this is what your application
uses to access the image bytes themselves so
you can show them in your app. A subtle but really
powerful feature of this URL is that it has
built-in resizing. So with a simple
parameter, you can ask Google Photos to return an
image that fits the dimensions or aspect ratio for
your device, saving you a lot of boilerplate code. It also makes it really
easy to reduce data transfer for users on mobile. The second URL here
is a product URL. And what this is is a deep
link back into Google Photos. This can be really
helpful if you want to provide a complete
and comprehensive UX with common tasks like cropping,
rotating, and even viewing complicated formats, such as VR. You can just link
out to Google Photos rather than rebuilding all
of that code in your own app. And then, of course,
there’s other metadata, depending on the media
format you’re dealing with. And what’s powerful
about this is Google Photos will extract
this from a variety of media formats, so you don’t have to go
through and parse it yourself. You’ll have it returned in
this consistent JSON object. And so with all of
this data together, you’re able to build and
include the user’s content inside of your user experience
and make it seamless and fast for them to bring photos and
videos into your application, just like the examples
we showed earlier. RUCHI BEZOLES: Thank you, Ben. All right. So Ben talked about how users
can access their library of photos in your products. Now, what about when you need
to send users their photos? The upload functionality
can help you do this. So with the Library
API, developers will be able to send
full-resolution photos and videos to a user by adding
a directory into their Google Photos account. Let me walk you through
how this would work. So this is actually
one of my photo albums that I have at home, but
accessing all my albums can be extremely
difficult. They’re buried in dusty
cupboards in my house, or they are thousands of miles
away at my parent’s home. They live in
Germany, so it’s not going to be very frequent
when I get to see them. But I decided to get this fixed. I gathered my albums and
sent them to Legacy Republic. They specialize in digitizing
physical albums and old media formats, like VHS tapes. I hope people have heard
about VHS tapes in the room. OK. So once they had digitized
it and it was ready, Legacy Republic was able
to use the Library API, create an album, and upload
more than 600 photos they just digitized into my
account almost instantly. That means at the
end of the day, I had access to all of
my childhood memories and some embarrassing photos
that I won’t pull up here. But they were all in
the palm of my hands, right in the Google Photos app. One thing my physical
albums do have, and this is a scrapbook that
I had from when I was younger, is a rich amount of information. They don’t just hold photos,
but they tell stories, at least for me. I scribbled little descriptions
of things I would think of, the places where the
photos were taken, or when they were captured. With the Library
API, Legacy Republic was able to take
this information and bring it into a digital
album in Google Photos. So a scrapbook like what
you saw turned into this. So having hundreds of photos
in one place is really great, but we also want to help users
find that exact photo they’re talking about in a story. And thanks to the machine
intelligence in Google Photos, we can do that, too. So let’s say I want
to show you my photos from the last few weeks. I’ll just type in “me”
in the search bar, and I’ll get photos of me. You don’t look that impressed. I guess, yes, those photos
are in the top of my grid, so doesn’t seem as hard. But let’s say I wanted to
tell you I was a little chubby toddler when I was a baby. I won’t tell you
how many years ago, but the next example will
tell it to you anyway. I could type in “me in 1990”
and, in less than a second, get photos of me as a baby. So I think, Ben,
it’s time for you to show some of your
embarrassing photos. BEN GREENWOOD: I don’t
think anyone wants to see those, Ruchi. Now, I’m going to
show you how we can create our own rich
albums using the API. But before we do
that, a quick poll. How many of you out
there have a friend who’s an outdoors person? Go ahead and raise your hand. You know, the type to go
camping and running and hiking. And you’ll know they’re an
outdoors person, because they will absolutely tell you about
it at every waking moment. You’ll come back into the
office on Monday morning, and they’ll regale
to you the epic miles they covered and cliffs
and snakes and bears that they had to fight with just
to get in for Monday morning standups. I have a confession to make. In my team, that guy is me. And so to prove this
to my colleagues, a few months ago,
I went for a hike in central Outback Australia. It was a pretty memorable
trip, so I built a little app to track my progress and take
a few notes along the way. And now, we’re going to try and
save it all in Google Photos. So the first thing
we’ll do with the API is go ahead and create an album. Simply send a POST request– excuse me– and you can
specify the title of the album. And in the response,
you’ll get back an ID, which we’ll
use in just a second, and a product URL,
which is, again, a deep link so you can show
the content in Google Photos. As you can see, the
album’s empty for now. So let’s fix that. Second step is to
upload the actual image bytes to Google servers. Because we support a wide
variety of media formats, you typically won’t need
to worry about transcoding between formats. You just upload the bytes, and
the service will give you back an upload token. And we’ll use it here in
the Create Media Item call, as you can see. We’ll also fill in the album
ID that we just created. And at this point, you’re
probably thinking, OK, thanks, Ben. We’ve talked a lot,
done a lot of plumbing, moved some IDs around. How about you show me
something interesting? Well, a big benefit of Google
Photos is being able to search by what you remember. Some of this happens
automatically, thanks to AI. So I could search
for the Eiffel Tower and find pictures I’ve
taken of the Eiffel Tower. But there are some things that
only you or your app knows. In this case, I’m taking
a photo of an area my friends told me to go and
look at called “the pound.” And it’s colloquial. It’s not on the maps,
so only I remember it. But I can put it
in the description with a few extra
details, and you’ll see that this, then, becomes
searchable in Google Photos. And all that context is
stored right with the image, so I can see it again
in years to come. This is a great tool you
can use in your apps. Maybe you’re building
a game, and you want players to be able to
save their favorite moments and include the player’s
name or the level that their character was at. Or maybe you’re just delivering
marathon finish line photos and want to be able to remind
the user of the great time they got. But let’s be honest,
individual photos aren’t all that
interesting on their own. So through the power of this
clicker, we’ll add a few more. Now, we mentioned
earlier that you can create rich
albums that contain text headings, locations, maps,
and even control the album layout. And in Google
Photos, we call these features album enrichments. Because there’s not a whole
lot else to do in the desert, each day I took a few
notes about the track. And now, using
album enrichments, I can group together my photos
to tell a meaningful story. To do this, we’ll go ahead
and call Add Enrichment and create a text enrichment. And finally, specifying the
position for the enrichment to be displayed in the album,
maybe after the last photo I took each day. And as you can see, this
creates headings and dividers, and our album’s really
starting to come together now. There are other kinds of
enrichments you can add, too. I could add individual
locations or even maps that describe the journey. And this can be useful for
all sorts of applications. Maybe you’re an
amusement park, and you want to be able to deliver
photos from the rides that someone’s just
taken in the park. Or maybe you’re just
trying to document the progress of a project,
like building your first home or seeing your child’s
art project progress throughout the school term. But for my hiking app,
the finished product looks a little bit like this. You can see it’s nicely laid
out with all the context safely stored in my Google
Photos library. And hey, now I can
easily share it with all of my friends and
colleagues back in the office. RUCHI BEZOLES: Yeah, I
think we’ve seen too many of Ben’s hiking photos. But Ben did cue up
the perfect segue to talk about the
sharing capabilities that the Library API offers. Did any of you go to the
I/O block party last night? So you may recall–
yes, I see some of you. You may recall seeing
this arcade photo booth. Now, the photo booth
was really fun. But what Ben and I
were most excited about was that it was integrated
with our Library API. So what this meant is you
didn’t have to fight over who gets that one physical
copy of that token from your photo booth. Instead, an attendee would
get a one-time shared album where they can gather
all their photos and videos that have
just been captured right on the device they’re using. And something to
note is the attendee doesn’t have to have the photo
booth app or Google Photos to get this link. You can also bring the
sharing functionality and make it work inside
of your mobile apps. So let’s take a look. Here’s an app for a soccer club. Players are discussing where
to meet up, sharing scores from the latest games. But to share photos today,
they have to close this app, open another app,
and then invite the same group to share photos. This doesn’t seem
that efficient. And so with the Library
API, these players will be able to
upload photos as they happen in the same conversation,
so everything can finally happen in your app. Let’s take a look at this code. With shared albums,
you’ll have the option to turn on or off comments or
allow users to collaborate, so add their own
photos into the album. In the response, they’ll
get a sharable URL. These are deep links that
open to Google Photos and work across any platform. And you get back a share token. This is what you
will use to invite people to join the shared
album from within your app. It’s what I showed you
in the soccer example. Now, you need to show your
album in your app, right? And to do this, you
send a GET request. And you’ll get back a
list of shared albums that a user has joined. So I’ll pause on this screen. The list you also get will
show attributes, like the title and cover photo, which you will
be able to build into your UI. This sharing
functionality really comes together in
this final use case I’d like to walk you through. But first, let me tell
you a little story. A few years ago, I decided
to gift my grandfather a digital photo frame. I felt like a really good
granddaughter, and he loved it. Grabbed a USB drive, put in
10 photos, and there it was. But years later, as many
of you have experienced, my poor grandpa is watching
the same 10 photos on rotation over and over and over again. Now, Nixplay, they create
beautiful digital photo frames, and they’re integrating
with the Library API to hopefully help
me fix this issue. Let me show you how it would
work in this context now. To get started, we would
create a shared album in the Nixplay app. And they can do this,
because they’re integrated with the Library API. You can also pick
an existing album you have in Google Photos, but
let’s go through a new album creation flow. So you can invite people,
your friends and family, to collaborate in this album. And then you can
select that album to be the main one you want
displayed on the frame. After that, it’s simple. As friends and family
add photos to that album, they’ll automatically be
displayed on the Nixplay frame you have in your home. So I did put this
to the test, and I connected my grandfather’s
Nixplay app to my shared album that I already had with him. I put photos in there
when I’m thinking of him or missing him as
I travel the world or just sitting at
home in San Francisco. So once we connected it up,
as I decided to share photos with my grandfather,
he has a front row seat to these moments in his
home all the way in Delhi without having to
do any extra work. BEN GREENWOOD: Thanks, Ruchi. So we’ve shown you
today three great ways that you can use the Library
API to bring the magic of Google Photos into your apps. We showed how you can create
seamless connections that allow users to bring their
content into your product and service and use smart
filters to help them find just what they’re looking for. We’ve talked about how you can
use the upload functionality to create more mobile-friendly
ways of delivering content to your users. And finally, we
talked about how you can build deep, engaging
sharing experiences right into your apps. We’ve also aimed to
take a lot of the hassle out of building a
seamless UX with features like automatic
thumbnailing and deep links that allow you to focus on
what makes your app unique. And all the APIs services
we’ve talked about today are available right now in
Developer Preview, which means you can activate the API
and start building right away. You can head over to
developers.google.com/photos, which has full documentation. And we have samples on GitHub
to help you get started. Now, we’ve been working with
some amazing early partners, and the demos we’ve shown are
a preview of things to come. But if you’re inspired
by what you’ve seen, you can also find a
link on our website to express interest
in joining the partner program for your own product. Now, I do want to
point out that we care very deeply
about the experience that Google Photos users have. And that’s why every
company we work with is expected to meet certain
app quality guidelines, and we also have strict
privacy provisions that govern how you can use the
data that users agree to share. We take those rules
very seriously. And so as your
app grows, you can expect a few chickens
from time to time with the Google Photos team. Finally, we really value your
suggestions as developers. And so as you start
to explore the API, please get in touch through
our developer channels and give us your feedback. You can also come along to the
Photos Office Hours here at I/O and meet with the team directly
to talk about your app. But that’s it for
this morning session. We’re really excited
about the road ahead and look forward
to working with you to develop great products
that work with Google Photos. We’ve been Ben and Ruchi. You’ve all been a
wonderful audience. Thanks very much. Have a great I/O. [APPLAUSE] [MUSIC PLAYING]

Only registered users can comment.

  1. What is the pricing model ? Are there some crop hints ? Is it possible to add more complex JSON metadata, like person metadata ? Or do we are forced to put a complex AI generated description to let the photo to be correctly searchable (ex: "blonde tall woman with green eyes near a blue car") ? Do you handle facets, in some standardized way, or in a custom way free to the developer (ex: "hair.color: ginger" key/value, added by some custom AI model in Google AutoML Vision) ? That product has the potential to compete ElasticSearch and Algolia for PWA in a common usage (ie that implies photos).

  2. I am normally a fan of Google's APIs, but I must say Google Photos has made it progressively more difficult to work with photo data over the past couple of years. This API makes it essentially impossible to, for example, confirm that a photo on my drive or on my device is uploaded to Photos. I can't get an MD5 signature to compare, I can't get the vast majority of metadata associated with the photo. We used to be able to find photos via Drive using the Photos type (now deprecated), we used to be able to use the Google Photos directory in Drive to find media in Photos, but the new Backup and Sync kills the ability to programatically confirm uploads or resolve between photos in Photos and photos in random folders in Drive. There is no API into Backup and Sync to tell what's updated and what's not, and the new API doesn't seem to support notification on changes. This API seems to be seriously lacking useful capabilities – the simplest hash, download, metadata features seem to be missing. Google: If I'm somehow 'not getting it', please point me to resources that explain how I can do something useful other than bringing up snapshots as in your demo.

  3. How about offline access? Presumably, if someone is hiking without a data connection and they take a photo, that photo will not be available through the API (even on the same device) before upload. However, it really should be.

  4. I have 2 questions:

    1) Is the Google ARCHIVE unlimited storage? I so wish I could archive a Google Album and the photos would remain in an album! Instead the archive is unorganized!

    2) Sometimes after I archive some photos they remain in the Photos section. And if I try to archive them again, they just turn gray. If I try to open a gray box/photo, it just spins and won't open! Do you, or any subscribers know what's going on here? I didn't see anything this specific in "Help". 😟

  5. You're funny! 😊 If my photos were taken from my phone, they are automatically copied into my Google Photos. From there I have been downloading my Google Albums to my EXTERNAL hard drive that is plugged into my laptop.

    If I take photos from my Canon camera, I reverse the process. I move the photos from my camera to my external hard drive and THEN upload them to Google Photos!!

    All the folders on my external hard drive are named "ORIGINALS + description". This is so I know that the photos in that folder are also stored Google Photos!

    In my albums on Google Photos I lable them "On Hard Drive + description". This way I know I already have the original stored on my external hard drive!

    For purposes of backing up my beloved photos, this had been a great process. Where it gets tough for me is doing all the edits BEFORE uploading or downloading! As you know, this is very time consuming! I've been editing them on my phone app called Pixlar or Photo Director… I wish I was proficient in Photoshop!

    Thanks for the great video and sharing this valuable information!

  6. Thanks guys, it's an awesome tool! But I did just use the POP edit and while it DID create a more vivid photo, it did NOT blur the background! What can I do?

Leave a Reply

Your email address will not be published. Required fields are marked *