0:15

Pushkin. Artificial

0:20

intelligence is this weird, big

0:23

phrase that suddenly seems to be everywhere,

0:25

and it can be hard to know exactly what it means.

0:28

But when businesses say they're using artificial

0:30

intelligence, they usually mean one particular

0:33

thing. They mean automated

0:35

systems that can take in lots and lots

0:37

of data and use the data to make

0:40

predictions. This is called machine

0:42

learning, and it's spreading everywhere.

0:44

Drug companies use it to predict which molecules

0:47

are likely to work as medicines. Hedge

0:49

funds use it to predict which stocks are going

0:51

to go up or down. Instagram uses it

0:53

to predict which adds I'm most likely to click on

0:56

For the record, the machine has learned that I will

0:58

often click on ads for overpriced

1:00

workout clothes. Anyway, if

1:02

you want to understand what's happening with business

1:04

and technology today, you really

1:07

have to understand machine learning.

1:13

I'm Jacob Goldstein. This is What's Your Problem,

1:15

the show where entrepreneurs and engineers talk about

1:17

how they're going to change the world once they've solved

1:19

a few problems. My

1:23

guest today is Luis Vonon,

1:25

the founder and CEO of Duolingo.

1:28

Duolingo is both a wildly popular language

1:30

app and also a hardcore tech company

1:33

built on machine learning. Luis

1:35

used to be a professor of computer science at

1:37

Carnegie Mellon, and in our conversation

1:40

he was really candid about the technical

1:42

limits of what Duolingo can do today.

1:45

The app is good at teaching people to read and

1:47

to understand, he said, but Duolingo

1:50

is not as good at teaching people to speak

1:52

a new language. And solving that problem

1:55

turns out to be part of this great, big,

1:57

interesting frontier problem that

1:59

is relevant not just for Duolingo, but

2:02

for the whole field of artificial intelligence.

2:05

We started out talking about the origins

2:08

of Duolingo, which go back to a different problem,

2:10

one that Louis discovered before he'd ever heard of

2:12

machine learning. It was a problem he saw all

2:14

around him when he was growing up in Guatemala.

2:18

I was fortunate that my mother basically

2:20

spent essentially her entire net worth

2:23

on my education, and so I

2:25

was fortunately that I got a good education. But then I could

2:27

see the people who would get public

2:29

education barely learn how to read and write.

2:32

This is just what would happen, And you cannot expect that these

2:34

people are going to become, you know, the

2:36

CEO of a public company or anything like that,

2:39

because they kind of wont. Often people

2:41

talk about education as an

2:43

engine for reducing inequality,

2:46

but what you're describing is the exact opposite,

2:48

and it's education, when you have

2:50

to pay for it, is a mechanism for perpetuating

2:53

inequality. And I really believe that, and I believe

2:55

that's true in most countries in the world. There may be some

2:57

countries, you know, like the Scandinavian countries, where

2:59

pretty much everybody gets the same education everything,

3:01

right, Yeah, there may be some countries

3:03

like that, But in the vast majority of countries, if

3:06

you have money, you can get a much better

3:08

education. So I wanted to do something that would give

3:10

equal access to education to everybody, and

3:13

so we started with that. But then we started thinking,

3:15

Okay, if education is pretty

3:17

general, let's start

3:19

by teaching one thing. Eventually

3:21

we settled on teaching languages for

3:24

a number of reasons, the biggest one of which is

3:26

that learning English in particular

3:29

can completely change people's

3:31

lives. If you know English, you can double

3:34

your incompotential in most countries. Wow,

3:36

it's just as simple as that. And so it's it's why

3:39

why is that. Basically it opens

3:41

up for almost any job. You

3:43

can get a better version of that job. For example,

3:45

you could be a waiter, or you could

3:47

be a waiter at the five star hotel. You

3:50

could be an executive assistant, or

3:52

you could be an executive assistant for a multinational

3:55

ceoyeah yeah, okay, So

3:57

really teaching English the core, the core

4:00

sort of reducing inequality

4:03

dream of a language is

4:05

really teaching English to people in largely

4:07

in poorer countries. Yes,

4:10

so what we wanted to do was teach teach English.

4:12

But you know, if you're going to teach English, we may as well teach other languages,

4:15

So teach those and do

4:17

so for free. So

4:19

that's what Luis did and it worked. Today,

4:22

tens of millions of people use Dual

4:24

Lingo every month to learn English and

4:27

dozens of other languages for free. The

4:29

company makes money by selling ads and premium

4:31

subscriptions. It went public in twenty

4:34

twenty one and is currently worth billions

4:36

of dollars. And the company really is built

4:38

on machine learning. Luis gave me a few key

4:40

examples of the way the company uses the technology.

4:43

So let me tell you a few of the things that we do. One

4:46

of the things that we do we like

4:49

very much is we have data

4:52

on whenever people use dual

4:54

Lingo. We record every exercise

4:57

that they do and whether they

4:59

got it right or wrong, and if they got it wrong, why

5:01

they got it wrong. With all of this data, we're

5:04

able to do certain things with artificial intelligence.

5:06

For examples, for every exercise that we're

5:08

about to give you able to predict what is the probability

5:10

that you're going to get this exercise right or wrong. So,

5:13

in a sense, that is a thing that a teacher

5:16

in a classroom could do fairly

5:19

easily, right a teacher with twenty

5:21

students, But you're

5:23

able to do it with whatever how

5:26

many people use your app actively forty two million

5:28

per month, so the machine can

5:30

do that for all forty two million people at

5:32

the same time. More or less, yes,

5:35

and very accurately. Part of the secret source

5:37

of Duelingo is that we realized if

5:39

we were to only give you things

5:41

that you're not very good at, we'd basically

5:43

be giving you lessons from hell every time. So

5:46

we can't do that because that frustrates users. So

5:49

what we do is, whenever you start a listening and doing

5:51

we're actually trying to optimize for two things at the same

5:53

time. We're trying to teach you things you

5:55

know that you don't not very good at, but also

5:58

we're trying to keep you motivated and

6:00

engaged. Yea. And the way we do

6:02

that is we try to give you exercises

6:05

for which we know you have about an

6:07

eighty percent chance of getting them right. Huh.

6:10

And have you found that to be the sweet spot? I mean, have

6:12

you done like experiments and sort of turn

6:14

the dial. We've done that, and we're you know, we're not the

6:16

first to figure this out. I mean, there's a lot of literature and psychology,

6:19

etc. Just and the number is not exactly eighty

6:21

percent. It's a little higher than that. It's like eighty three percent or

6:23

something. But there's there's a number, and it

6:25

really is the case that if that number is higher, that

6:27

means these things are a little easier for you. Then

6:30

you get a little bored. You feel like you're not learning

6:32

right. If I'm getting ninety five percent right, I'm

6:34

like, what, I'm just wasting my time? And you feel bored because

6:36

it's like it's like a game that you always win. I mean, that's

6:38

that's nice. At the very beginning, but then you're just

6:40

not going to play it, and then

6:43

if it's lower than that, that means that things

6:45

are too hard for you. You get very frustrated and

6:47

you go away. And there's a lot of tricks

6:49

that you know, certainly app developers

6:51

play, and you know we play as well. So I'll

6:54

tell you another kind of similar trick. You know, we end

6:56

up applying it to language. But the easiest way to

6:58

understand this trick is with a slot

7:00

machine. When

7:02

you get two out of three, it's you almost

7:05

got it, you gotta do one more, you gotta do one

7:07

more. You just gotta do one more because you got

7:09

it. So there's this, there's this you're so close to psychological

7:12

trick that we played. It's like, oh, there's two out of three, almost

7:14

got it, But you knew I was going to get two out

7:16

of three. Yes, sure, you gave me two easy ones

7:19

and when there was super hard that's exactly

7:21

right. So so we we played this type of trick

7:23

where just people are like, almost

7:25

got it, and that gets them to do another one. So

7:27

you know, in our case, we just we basically spend a

7:29

lot of time training computers

7:32

to figure out what it is that makes people

7:34

use Duolingo for longer, and also that

7:36

we teach them more so

7:39

that that's a major use for artificial intelligence.

7:41

Main use is just in teaching better. After

7:46

the break a big problem, Louise and dual Lingo

7:48

are still trying to solve a problem

7:50

that turns out to be a big frontier problem

7:53

for all of artificial intelligence.

8:01

That's the end of the ads. Now we're going back

8:03

to the show. So let's talk now

8:05

about problems you haven't solved

8:07

yet. You know, like, what are you what are you trying

8:10

to figure out? What are you working on that that

8:12

isn't quite working yet. So dual linguals is

8:14

very effective at teaching you all kinds of things. But if

8:17

you go look under the hood or you know, what is it

8:19

that you're learning. You're learning reading really well.

8:21

You're learning writing pretty well,

8:23

but not as well as reading. You're learning

8:25

listening pretty well, but

8:28

you're not learning spontaneous speaking

8:30

very well. This, by the way, is also you're

8:32

not something you're not learning very well in university semesters,

8:35

Like you're basically not learning that well either in due lingo

8:37

or in university semesters. Okay, it's

8:39

just harder to teach in a sort of classroom.

8:43

What you need to do

8:45

to teach that is basically, have you really

8:48

interact with wealth?

8:50

For now another human and just

8:52

you just practice that a lot. Now,

8:54

here's the here's the thing about that. I

8:57

know how to get you to interact with another human. Just

8:59

put another human there. The problem is

9:02

about eighty percent of our

9:04

users just does not want to talk to

9:06

a stranger in a language that they're not

9:09

very good. So the problem that

9:11

we're trying to solve here is how

9:13

do we practice kind of spontaneous

9:16

conversation but without

9:18

having a human on the other side. And

9:21

we've been working on that, and you know we're

9:23

not there yet. Can I just interrupt? Because

9:25

I mean we were talking about artificial intelligence?

9:27

Right? The most famous test that

9:30

I know of, the most famous idea I know of

9:32

of artificial intelligence in a computer is

9:34

can a computer hold one end of a conversation?

9:37

Right? Like? That's the classic touring

9:40

test is like you're going to have

9:42

this like chat conversation and can

9:44

you tell if the person

9:46

on the other end is a person or a machine? Like?

9:48

That's the og artificial intelligence

9:51

idea, right, I mean, are you telling me, that's what you're trying

9:53

to solve. Not quite. I mean, it

9:56

would be awesome if we would solve that. I mean, but that's

9:58

the dream, right solution to that,

10:00

that is the dream. But notice in our case, we don't actually

10:02

care if the human can tell that there's a computer

10:05

on the other side. Okay, it's okay. As

10:07

long as it practices thing and as

10:09

long as you're able to carry on a conversation

10:11

in a way that seems a little natural or something, it's

10:13

okay. If if it, you know,

10:17

goes off the rails every now and then, So

10:19

tell me, what is it that you're trying to build.

10:21

This is exciting, Like what are you trying to do?

10:23

We're starting with text, by the way, so either

10:26

just basically a texting conversation. So think

10:28

of it as like a chat bot in uh,

10:31

you know, in Spanish, where it just you're

10:33

just having a real a little conversation.

10:35

You've had lots of people have had experience with chat

10:37

bots. Yes, they're right, Like you go to whatever,

10:40

cancel your cable and

10:43

they want you to text, and then

10:45

you realize you're texting with the machine. So like that's

10:47

the that's step one.

10:49

So that's the idea. That's

10:51

step one. We of course, I mean a

10:53

lot of those experiences with chatbots are are very

10:56

um, they're just very

10:58

geared at whatever it is you're trying to do.

11:00

So, for example, that chapel maybe very good at at

11:02

canceling your cable, but only that in

11:05

my experience, they're not even good at that. No,

11:07

they're not that great. So

11:09

we're trying to do that, and you know, we're not there yet.

11:11

I don't think so you're like out

11:13

on the frontier. We are. We are like you're trying

11:15

to, yes, and we're not there yet. I mean,

11:17

this is something that's going to take us, not just us,

11:20

I mean the whole academic

11:22

community and technology just a

11:24

few more years. But so let me ask

11:26

you this. Can we talk about that

11:28

in a way that would be like, can we

11:30

try and just go one level into

11:33

sort of what you're trying to do and like what

11:35

works and what doesn't work, and like why it's

11:37

hard? Yeah, I mean, you know, the

11:39

first, by the ways, the first way you think of

11:41

if you're trying to make a chap the first

11:43

thing you think of is, okay,

11:45

I'm just going to program the computer. Forget

11:48

about artificial intelligence. I'm just going to program

11:50

the computer to respond

11:52

to specific questions, and

11:55

how many possible questions could there

11:57

be? You start thinking, okay, well, when the

11:59

person says high, we're

12:01

going to program a think to say hi back. When

12:04

the person says, how are you doing, We're going to program

12:06

I think to say I'm doing pretty well? How about

12:08

you? Yeah? V zero of a chatbot.

12:10

And this, you know, this comes from you know, fifty years ago.

12:12

This is what you start doing. The problem is there's billions

12:15

of things that people can say, and so

12:17

we may have programmed the thing of what to say, but

12:19

you know how you're doing, and we can respond.

12:21

But if instead of asking that, they may ask

12:24

like, hey, did you watch

12:26

the game last night? And we just have

12:28

no idea how to respond to that. About

12:30

a decade ago, Louis says Ai, researchers

12:32

started trying a really different approach.

12:35

Rather than trying to teach computers every

12:37

rule, they started throwing massive

12:40

amounts of documents and texts at

12:42

computers and essentially telling the computers

12:45

figure out the patterns in all these documents.

12:47

So when somebody writes something like did

12:50

you watch the game last night? The computers

12:52

should be able to predict what kinds of

12:54

answers might follow. This strategy

12:56

clearly has not entirely worked yet. That's

12:58

why it's still a problem solving. It will

13:01

take both more text and more clever

13:03

algorithms to help computers make

13:05

sense of that text. But Louis

13:07

says, you can see progress every

13:10

time you open your Gmail or a Google

13:12

Doc. And I don't know if you've used, for example, you use

13:14

Google Docs lately or Gmail like

13:17

it finishes off your sentences now. And

13:19

basically the way this works is, you

13:21

know, this system has looked at

13:24

a ton ton of text that has been

13:26

written by a lot of people. In the case of Google Docs, I actually

13:28

don't know what they look at, but I wouldn't be surprised if

13:30

they look at everything that has ever been written in Google

13:32

Docs. I'm going to tell you one that happened

13:34

to me in Google Docs today when I

13:37

was typing notes for this interview, I

13:39

typed zone of pr

13:42

and then you know what, you know how it completed

13:44

it proximal development. Yes,

13:47

it knew I was going to write zone of

13:49

proximal development. Yep. No,

13:52

this is amazing, And they just see that if you'd

13:54

write the zone of per there's

13:56

like a ninety five percent chance that

13:58

it ends in proximal development. What is

14:00

the zone of proximal development? You

14:03

know? In teaching, you know, there's this concept

14:06

of just keeping you at this zone of proximal

14:08

development, which is always kind of challenging

14:10

you, giving you things that you don't know. But but

14:12

there are all things that are fair to give you. Proximal

14:15

means like close to or next two. Right.

14:17

So it's the idea is like you know a thing, yes,

14:19

then like what you want to teach the person is the very

14:21

next thing, right, It's like that's

14:23

right, It's like the frontier of your knowledge. I like it because

14:26

it applies to like the way you teach, but also

14:28

to your work, right, And like it's just

14:30

a nice life idea, right, It's like the

14:32

next thing you want, the next thing. And I

14:34

feel like the chat bot is maybe a version of

14:36

that at the level of your company. Yeah,

14:39

yeah, it really is. It really is a nice idea, and it

14:41

is I mean, and if you think about it, this

14:43

is what a great teacher does. You know. I've said

14:45

this inside the company at due

14:47

Lingo. UM. All we need to do is

14:50

first figure out what you know, by

14:52

the way, not that easy to figure out what you know. But let's first

14:54

figure out what you know and then just take you to that

14:57

zone of proximal development. Because now we know what you know,

14:59

just take you to the frontier

15:02

and then just keep expanding it as fast as possible.

15:04

That's all we need to do. Of course, this is easily said,

15:06

hard to do. Yeah.

15:08

Um, And is there a limit to what you can do

15:11

with a computer? Is there anything a teacher can

15:13

do? Is that a computer will never be

15:15

able to do? You know? Of course I do

15:17

lingual We love teachers. If they are a good

15:20

teacher and also have the time, they

15:22

are much more able to adapt to their students than a

15:24

computer is. Um. But

15:26

I don't believe that will always

15:28

be the case. I mean, I think at some point

15:31

it's not just teachers. I mean teachers, this is one thing.

15:33

I mean, at some point my belief

15:35

and this is of course just my belief.

15:37

People, not everybody agrees. They believe

15:40

that computers will be able to do every single thing that humans

15:42

can. Now you may start asking

15:44

really tough questions like can they love?

15:46

Yeah, I don't know, I don't know what they can love or not

15:48

but from the outside it will look just

15:51

as if they love so who

15:53

knows who knows what's going on inside? Who knows that they

15:55

that's like a big yeah, we're big philosophical

15:57

questions that I'm not here

15:59

today, and that's right, nor am I. But I do think from

16:01

input output behavior, I don't see why. I

16:04

don't see any reason why computers won't be

16:06

able to do everything that humans can. So they

16:08

can teach, but they can also write

16:10

a computer code. They can also run companies,

16:12

they can also make podcasts, they can do everything.

16:15

Should be able to do that. I think they should be able

16:17

to do that. I don't know when that'll happen, but

16:20

they should be able to do that. In

16:24

a minute, the lightning round, well, hear what

16:26

job Luise would love to do but thinks he wouldn't

16:28

be very good at. And the real reason

16:30

treasure Chess keep showing up in duo lingo.

16:39

And now back to the show. We're going to finish

16:41

with a lightning round, not counting duo Lingo.

16:44

What's your favorite app on your phone? Spotify?

16:47

What have you been listening to on Spotify? I'm

16:49

always a huge fan of the band called

16:51

Churches with a v

16:53

to Virchase. So

16:56

that's what I was listening to this morning on my work at

16:58

work. If you have a ten minute break in the middle of the

17:00

day, what do you do to relax? Played

17:03

this game called Class Royale. We

17:06

are a lot of the gaming mechanics that

17:08

we use for duel and will come from gaming

17:10

companies, like the treasure

17:12

chests, exactly right, the

17:14

treasure chests. If you ever played Class Royale, they

17:16

have the treasure chests. If somebody's

17:19

going to go to visit Guatemala

17:21

for the first time, what's one thing they should definitely

17:23

do? Oh? Um,

17:26

Decal is the Mayan ruins. Um.

17:28

You know, if I feel very strong, I've been to southern

17:31

Mexico where they have chi Chenitsa. It's

17:33

a joke compared to the Mayan ruins in Guada.

17:36

There's there's one pyramid

17:38

in Chichenitsa. There are four hundred in Guatemala

17:41

in Decal, So yeah,

17:43

they should like I like that. Not only

17:45

are you recommending Tikal, you're also taking

17:50

as I have no trouble with Chichenitsa.

17:52

It's just they are very good at marketing. Amazing.

17:56

What would you do if you couldn't do

17:59

the job you do. Now, well

18:01

there's what what would I actually do? On?

18:03

What would I'd like to do? I would love to be a writer. I don't think

18:05

i'd be a very good one. Um

18:07

So if I if I wasn't doing

18:09

the job that I'm doing right now, you know, I'd

18:11

probably be back to being a professor. How

18:14

will you know when it's time to retire? I'm

18:16

never retiring, That's what everybody

18:19

says. Well,

18:21

maybe I will, but I mean right now, I don't. I don't

18:24

want to do that. Luis

18:27

Vaughan is the founder and CEO of

18:29

Duelingo. Today's show

18:31

was produced by Edith Russelo. It was edited

18:34

by Kate Parkinson Morgan and Robert Smith,

18:36

and it was engineered by Amanda kay Wong. Theme

18:38

music by Louis Kara. Our development team

18:41

is Lee, Tom Mulad and Justine Lang. A

18:43

huge team of people makes What's Your Problem

18:45

possible. That team includes, but is

18:47

not limited to, Jacob Weisberg,

18:49

Mia Lobel, Heather Fain, John Schnars, Kerry

18:51

Brodie, Carli mcgleory, Christina Sullivan, Jason

18:54

Gambrell, Brand Hayes, Eric

18:56

Sandler, Maggie Taylor, Morgan Ratner, Nicolemrano,

18:58

Mary Beth Smith, Royston Deserve, Maya Kanig,

19:01

Daniello, Lakhan, Kazia Tan and David Clever.

19:03

What's Your Problem is a co production of Pushkin

19:06

Industries and iHeartMedia. To find

19:08

more Pushkin podcast Us, listen on the iHeartRadio

19:10

app, Apple Podcasts, or

19:12

wherever. I'm Jacob Goldstein and

19:14

I'll be back next week with another episode

19:17

of What's Your Problem.