Images and Inspiration With AI: Pinterest’s Jeremy King

Jeremy King leads a team of 1,400 passionate engineers working on the continuous improvement of Pinterest’s image-driven platform. With a background that includes heading up a translation team at eBay and overseeing the technology behind Walmart’s U.S. retail stores and e-commerce business, Jeremy is now responsible for technology operations at Pinterest. To support the company’s mission to inspire people to “create a life that they love,” he and his team rely on advanced AI, machine learning, and a graph database to index and build a network of images so that users can find inspiration — particularly when they aren’t completely sure what they’re looking for.

On this episode of the Me, Myself, and AI podcast, Jeremy joins hosts Sam Ransbotham and Shervin Khodabandeh to talk about some recent advances Pinterest has made in the image-recognition space and shares his views on how generative AI will transform image-based content like Pinterest’s.

Subscribe to Me, Myself, and AI on Apple Podcasts, Spotify, or Google Podcasts.

Transcript

Shervin Khodabandeh: How does an AI-based platform use generative AI to engage users? Find out on today’s episode.

Jeremy King: I’m Jeremy King from Pinterest, and you’re listening to Me, Myself, and AI.

Sam Ransbotham: Welcome to Me, Myself, and AI, a podcast on artificial intelligence in business. Each episode, we introduce you to someone innovating with AI. I’m Sam Ransbotham, professor of analytics at Boston College. I’m also the AI and business strategy guest editor at MIT Sloan Management Review.

Shervin Khodabandeh: And I’m Shervin Khodabandeh, senior partner with BCG and one of the leaders of our AI business. Together, MIT SMR and BCG have been researching and publishing on AI since 2017, interviewing hundreds of practitioners and surveying thousands of companies on what it takes to build and to deploy and scale AI capabilities and really transform the way organizations operate.

Sam Ransbotham: Welcome. Our guest today is Jeremy King, head of engineering at Pinterest. Jeremy, thanks for taking the time to talk with us.

Jeremy King: It’s great to be here. Thank you very much.

Sam Ransbotham: It’s always good to start with an overview; maybe tell us a bit about what Pinterest does and what you do at Pinterest.

Jeremy King: Pinterest is, we like to say, the destination on the internet for inspiration. We have hundreds of millions of people that come to us every day to figure out what they want to wear, what they want to do this afternoon, or what they want to make for dinner, or how to decorate their kid’s cake or remodel their kitchen. And so we want to build a platform that allows anyone to create a life that they love.

It’s a great place to work with that kind of mission, and I have a lot of people who, when I tell them where I work, get pretty elated about their own Pinterest boards and what’s going on in the world.

Sam Ransbotham: Great. Tell us more about working at Pinterest.

Jeremy King: I’ve been at Pinterest now, the head of engineering, for just over four years. And just like any leader, I’m trying to figure out how to unblock my team and hire great people. I tell people all the time that when I came to Pinterest, I was just so impressed with the level of talent. It’s a relatively small team. We have 4,500 employees in the world and about 1,400 engineers. And as a result, competing against some of the biggest companies in the world, this team has to be extremely high caliber. We have some of the best graph database people in the world; we have the best computer vision people in the world. These people are really fun to work with.

Sam Ransbotham: None of that, so far, mentioned artificial intelligence, so, what role does artificial intelligence play with those 1,400 engineers?

Jeremy King: Yeah, that’s an excellent point. One of the things that is true at Pinterest and now at many other companies is Pinterest was actually born with machine learning and AI from the base. It doesn’t exist without it. So talking about AI and ML separately from Pinterest is almost like a misnomer; it really doesn’t happen without it. Nearly everything that we do touches our machine learning systems. We do millions of inferences every second. Every single request comes all the way back to the graph and publishes back out a very specific use case to every single person.

It’s not cached. … Every person has a specific result. I mentioned that we have about 1,400 engineers. We have about 350 machine learning engineers. As a percentage, when I tell other CTOs … they’re like, “That’s a really high percentage of machine learning people compared to my company.” That’s true also on the data platform side; we have a wonderful leader, Dave Burgess, who runs our data platform. And you can imagine our data platform is about as important as any other part of our capabilities. And so I mentioned the graph, but it’s also, you know, the sort of normal SQL databases and the real-time systems that keep this thing running.

Shervin Khodabandeh: Jeremy, you said “graphs” a couple of times. What’s a graph?

Jeremy King: Essentially what a graph is, is if you take an entity, an object, and you say, “What’s related to that object?” — so, in this case, we’re using a nearest neighbor graph, which we’ve written lots of documents on, which is saying, “Hey, I have an image, in this case, and I have another image that’s related to this image.” So that allows me to build this sort of network of images. And the reason the graph works at Pinterest is because every pin is essentially added to a board, which essentially makes it a node in a graph, and so it continues to increase the graph.

So it allows you to … unlike a SQL database or a relational database, where you have to sort of tag indexes and indices, the graph can be indexed. Essentially any image could be indexed to any other image. It allows you to do searches on things that are related incredibly efficiently. That’s the power of Pinterest.

Shervin Khodabandeh: Jeremy, you mentioned AI has been part and parcel of Pinterest’s journey and has been ingrained in Pinterest’s tech stack from its very early inception, so it must be that generative AI is all over the map for you guys as well, and I’m sure it’s on everybody’s mind these days. So maybe tell us a bit about what you are doing there.

Jeremy King: We have a state-of-the-art machine learning environment that interconnects these data sets across all the surfaces. We call it the “home feed,” the search system, the related things that drive personalization and recommendations and engagement across Pinterest. [And] we’re growing really fast from monthly active users. … Gen Z is growing even the fastest. We added 13 million monthly active users, so we’re really focused on how to increase engagement.

And not surprisingly, a lot of the improvements in machine learning, even in the last two years, have been driving that engagement. We have, of course, built new features [that] allow Pinterest to become more shoppable. But as we’re seeing the capabilities of the machine learning models getting effectively 10 or a hundred times bigger, which is relatively common today, we’re seeing that give an outsize increase in our results as the result gets more specific.

And frankly, Pinterest is also about looking at [vastly] different things. So it’s not [that] every single item is about “What do you want to cook for dinner tonight?” It’s also “I know you were looking at birthday cakes last week” or “New Year’s is coming up,” and we’re going to interplay some of that. So we’ve been taking advantage of some of the advanced AI capabilities that have only come to life over the last couple years. And that includes GPU [graphics processing unit] work and all the things that make it cost-effective to do these things.

So tying back to your question of generative AI, not surprisingly, given that we’re an image platform, generative AI is very interesting to us, and I typically break it up into three buckets. When we’re thinking about large language models — LLMs — we talk about, No. 1, how do I make my team more productive? We’ve got a couple of pilots running. We haven’t decided exactly which way to go yet, but it’s looking really promising, and, like lots of other CTOs, we’re really excited about that.

Shervin Khodabandeh: That’s interesting. Tell us how generative AI increases efficiency.

Jeremy King: I was talking to a number of CTOs, and one thing I thought was really interesting is, this one particular CTO was saying that, in general, it’s increased productivity by 10% to 15%, but there is a small set of users where it’s increased their productivity by 50%.

Shervin Khodabandeh: That’s fantastic. I was just going to say, it also feels like, if you go back a decade ago, there were lots of images and unstructured text and that kind of stuff that, for AI to train on that data and make sense of it, human intervention was needed, both to tag the content but also to make sure that the outputs made sense. And it feels like now, with gen AI, that a lot of those humanlike judgment calls are going to be made more and more with gen AI, particularly with lots of unstructured text and images and video and that kind of content. Is that right, or am I going too far?

Jeremy King: I think it’s right. I mean, Pinterest’s system is built on embedding, so you take an image or a piece of text and you essentially tag it — essentially, to your point, this is what Pinterest has really been great at. You take our computer vision technology and effectively build these embeddings to detect people or content or couches or birthday cakes.

We’ve been really good at that and have really led the industry for a long time on this. And you’re not wrong that what happens is, it just gets better; it gets more accurate, more specific. In the old days, I could say, “Hey, I know this is a lamp, and I know it’s got pendants, and it’s made of crystal, and it’s gold,” and that sort of thing. But now I can say, “I know exactly how many tines it has on it; I know what kind of light bulbs they are; I know probably who manufactured it,” and these sorts of things. I can get way more specific.

And at Pinterest, we really think about these as two different things. People come to Pinterest because they don’t know exactly what they want. And this is where Pinterest thrives, and I think why we have a long-term differentiation in the market. Because if you know what you want, you can always go to Amazon or Home Depot or Wayfair to go buy it, but if you don’t know what you want, you start with Pinterest or do a million different searches because [you] don’t know how to describe, you know, “classic barnyard kitchen” — [you] don’t know those words.

So even putting it in something like ChatGPT, I don’t know how to say that — you know what I mean? I’m sure that’ll get better over time, but a lot of it will be image-based, too, as well.

Shervin Khodabandeh: Are you finding that the foundational models that are available do it for you, or you’re building your own domain-specific models?

Jeremy King: Both, I guess, is the answer. What we’re finding — and it’s still relatively early — in sort of the generative imagery, what we’re finding is that the smaller models actually are much more specific, and so I think that’s what’s going to happen. And my CTO friends are saying the same thing, which is, each one of these models is going to be very specific to the use case, and that helps it not only be more accurate but also makes it much, much cheaper to implement as well.

Sam Ransbotham: I think it’s interesting that you mentioned cakes several times, because decorating cakes is, I think, practically my only use of Pinterest. And it ties to Shervin’s comment about generative, because I deeply suspect that those cakes that I see there, no human could create. I certainly can’t create them. So to what degree are … people putting [up] images that don’t really exist or products that don’t exist? What’s Pinterest’s perspective on that?

Jeremy King: Excellent point. I think, yes, and we see this a lot [in] one of our biggest categories we call “art.” Art includes a whole bunch of things, including things that you would traditionally put on walls, and paintings, and that sort of thing. But it also includes things like body art, and we have tattoos and these sorts of things. And again, lots of those are great cases where people need a little bit of inspiration, and it can be generative, right? It may not exist in the world, but you want to see what it looks like on a human.

You can see this coming, where you’ll be able to print out something that’s generative. And this is already happening, right?

Sam Ransbotham: Yeah. That really aligns with the idea of inspiration.

Jeremy King: Yeah, absolutely. And things like cakes and that sort of thing are very interesting. One of the harder parts is, I’ve seen lots of work for home and home improvement, but what it’s doing is it’s generating a whole bunch of ideas that can’t be realized, where I can say, “Hey, people have yet to really break out the camera and take a picture of their room.” People do that, but we call it a 1% feature. Like, how do you get people to actually engage the phone or the phone camera?

But if you can upload an image — I’ve seen some beautiful renderings of, like, “Here’s what my room is. Rearrange the furniture in my room, the furniture I already have,” and things like that become really interesting. Or you say, “I want to replace this couch with this couch from Wayfair. I want to replace this lamp [with] some other lamp,” and here’s 20 different combinations, and you can kind of click through it.

Those are like you’re taking a virtual experience, almost AR-like [augmented reality-like], but you’re putting real products in there versus them being generated products. But it allows you to build user experiences that are much more enhancing.

Last year, we built AR Try On, which allows you to try on makeup, and what we’re seeing, again, while it doesn’t get a ton of usage, the people that do use it are 60% more likely to buy something. It’s amazingly engaging when people actually get there.

Sam Ransbotham: What’s tough at Pinterest? You mentioned a lot of good use cases, where you’re doing lots of things with these technologies. Certainly, [it] can’t all be wonderful. What’s hard?

Jeremy King: We spend a lot of time on inclusive search and results. In 2021, we launched hair-pattern search. It was kind of this first of technology: When you’re searching for hairstyles, how do you identify hair that looks like yours? This is hugely important in order to refine what you’re looking for.

We had these same kinds of problems with skin tone, where originally we were trying to use the early models for skin tone detection. And what we found is, it was more on face detection. And so our team had done some great work doing skin detection — it could be a side shot or a back shot or ear shot or a hand shot. You have no idea what kind of skin. So how do I detect what skin is? And then you can detect what kind of tone you’re looking for.

Same thing had to be applied to hair. And we’ve got all kinds of hair: We have shaved/bald, we have straight, we have wavy, we have curly, we have coily, we have protective, and these kinds of things make the results dramatically better.

You can save your hair pattern and your skin tone, and then we’ll tailor your results to that. And as you can imagine, this is a complex problem. It’s not just a USA thing; every different country has different types of hair patterns and skin tones and different kinds of fashion, and so these kinds of things are really hard, but they’re the great projects to work on.

And we’ve had a wonderful team. The advanced technology group here, in particular — we’ve been working on inclusive technology and all these advanced models, and they’re definitely hard, but they’re wonderful when you get them right.

Shervin Khodabandeh: We have a segment here: five rapid questions. I’ll just ask you a bunch of rapid-fire questions. Just tell me the first thing that comes to your mind. What’s your proudest AI moment?

Jeremy King: The inclusive product feature, and actually, hair pattern was my proudest one, because not only was I involved [with] the product from the beginning, but it’s also first in the industry. And the amount of response that we got from our pinners was incredible. So, yeah, it was a really good one.

Shervin Khodabandeh: What worries you about AI, aside from bias and some of the ethical issues?

Jeremy King: It’s funny. It’s on every single forum. It says, “Hey, are you worried about bias?” Let’s see. What worries me about AI? Hmm. Yeah, I’m not worried. I don’t worry about too many things, but that’s a CTO …

Shervin Khodabandeh: That’s good. That’s a good thing though, right?

Jeremy King: That’s a longtime CTO’s thing. If you worry about too much, you can’t survive in this job.

Shervin Khodabandeh: What’s your favorite activity that does not involve technology?

Jeremy King: Mountain biking.

Shervin Khodabandeh: Very cool.

Jeremy King: I try and go at least two times a week. I go with a crew of Silicon Valley tech [colleagues] on Friday mornings and then usually on the weekends with my brother and a few other people.

Shervin Khodabandeh: The first career you wanted: What did you want to be when you grew up?

Jeremy King: I think my 5-year-old baby book says fireman, but both my grandfather and my father were engineers, so I actually thought I would build houses, because I loved working with my hands outside. So that’s where I thought I was going to go — architecture or house-building.

Shervin Khodabandeh: What do you wish AI could do that it currently can’t?

Jeremy King: Teleportation: Get me through the security line faster! Maybe that’s coming. I’ve always been super excited about translation. I took over the translation team at eBay way back when, and there were some wonderful people that were working on this. And I’ve always thought about a universal translator as something that would just be so amazing. And it’s getting so close, but it still seems like so far away before there’ll be a consumer version of this device.

Shervin Khodabandeh: But why do you think that is? Because we’ve got LLMs now, and we’ve had, for a long time, audio to text. What’s the limiting factor here?

Jeremy King: I think it’s form factor; like, how do I get out my phone, hit “translate from Spanish to English” — you know, that kind of thing. I don’t know; if I’m walking up to a vendor in the street in San Francisco and I don’t speak the same language, and I want something. … It’s got to be close, but it’s a form factor thing. But I think it’ll come shortly.

Sam Ransbotham: Seems solvable.

Jeremy King: Yeah, solvable.

Sam Ransbotham: Jeremy, we really appreciate you taking the time to talk with us. It’s been very interesting to learn, sort of behind the scenes, what’s happening between all those images that we see but maybe didn’t appreciate all that goes into making something that looks as easy as Pinterest work. And it’s been pretty fascinating to see how much artificial intelligence and machine learning is behind that. Thanks for joining us.

Jeremy King: Of course. It’s been wonderful.

Shervin Khodabandeh: Thanks for listening today. On our next episode, Sam and I meet with Damini Satija and Matt Mahmoudi from Amnesty International. Please join us.

Allison Ryder: Thanks for listening to Me, Myself, and AI. We believe, like you, that the conversation about AI implementation doesn’t start and stop with this podcast. That’s why we’ve created a group on LinkedIn specifically for listeners like you. It’s called AI for Leaders, and if you join us, you can chat with show creators and hosts, ask your own questions, share your insights, and gain access to valuable resources about AI implementation from MIT SMR and BCG. You can access it by visiting mitsmr.com/AIforLeaders. We’ll put that link in the show notes, and we hope to see you there.