0:02

Welcome to Econ Talk, Conversations for

0:04

the Curious, part of the Library of Economics

0:07

and Liberty. I'm your host, Russ Roberts

0:09

of Shalem College in Jerusalem and

0:11

Stanford University's Hoover Institution. Go

0:14

to econtalk.org where you can subscribe,

0:16

comment on this episode and find links and other

0:18

information related to today's conversation.

0:21

You'll also find our archives with every

0:23

episode we've done going back to 2006. Our email address

0:26

is mail at econtalk.org,

0:30

we'd love to hear from you.

0:37

Today is March 6th, 2023. My

0:40

guest is neuroscientist, Eric Hoel.

0:42

He was last here in September of 2022, talking

0:45

about effective altruism. Today, we're

0:48

going to talk about two recent essays of his

0:51

on artificial intelligence and chat GPT.

0:54

Eric, welcome back to econ talk.

0:57

Thank you. It's an absolute pleasure to be here. I had a blast

0:59

last time. As did I. I

1:01

want to congratulate you. You are the first

1:03

person who has actually caused me

1:06

to be alarmed about the implications of AI,

1:08

artificial intelligence,

1:10

and the potential threat to humanity. Back

1:13

in 2014, I interviewed Nicholas Bostrom about

1:16

his book, Super Intelligence, where he argued AI

1:18

could get so smart. It could trick

1:20

us into doing its bidding because it would understand

1:22

us so well. I wrote a

1:24

lengthy follow-up to that episode and we'll link

1:26

to both the episode and the follow-up. So

1:28

I've been a skeptic. I've interviewed Gary Marcus

1:31

who's a skeptic. I recently interviewed Kevin

1:33

Kelly who is not scared at all, but

1:35

you,

1:36

you are scared. Last

1:39

month you wrote a piece called I Am Bing and I

1:41

Am Evil on

1:42

your sub-stack The Intrinsic Perspective

1:44

and you actually scared me. I don't mean,

1:47

You know, maybe I've underestimated the threat of AI. It was

1:49

more like I had a bad feeling

1:52

in the pit of my stomach, kind of scared.

1:55

So what is the central argument

1:57

here? Why should we take the

2:00

this latest

2:02

foray into AI chat GPT,

2:05

which writes a pretty okay

2:07

or pretty impressive but not very exciting essay,

2:10

can write some poetry, can write some

2:14

song lyrics. Why is it a threat to

2:16

humanity? Well

2:19

I think to take

2:21

that on very broadly we have to realize

2:23

where we are in the history of our entire

2:26

civilization, which is that we are at the point

2:28

where we are finally making things

2:31

that are arguably as intelligent

2:33

as a human being. Now, are they as

2:35

intelligent right now? No, they're

2:38

not. I don't think that these very

2:41

advanced large language models that

2:43

these companies are putting out could be said to

2:45

be as intelligent as an expert human on

2:47

whatever subject they're discussing. And

2:50

the tests that we use to measure

2:52

the progress of these systems supports that, where

2:55

they do quite well and quite surprisingly

2:57

well on all sorts of questions like SAT

2:59

questions and so on. But

3:02

one could easily see that changing.

3:04

And the big issue is around this concept

3:07

of general intelligence. Of

3:09

course, a chess playing AI poses

3:11

no threat because it's just fully trained on

3:14

playing chess. This is the notion of a narrow

3:16

AI.

3:17

Self-driving cars could never really

3:19

pose a prep. All they do is drive cars. But

3:22

when you have a general intelligence,

3:24

that means it's similar to a human in that we're good

3:26

at all sorts of things. We can sort of reason

3:29

and understand the world at a general level.

3:31

And I think it's very arguable that right now,

3:33

in terms of the generalness behind general

3:36

intelligences, these things are actually more

3:38

general than the

3:40

vast majority of people. That's precisely why

3:43

these companies are using them for search.

3:45

So we already have the general part

3:48

quite well down. The issue is

3:51

intelligence. These things hallucinate. They

3:54

are not very reliable. They make up

3:56

sources. They do all these things. And I'm fully

3:58

open about all their problems.

4:00

Yeah, they're

4:02

kind of like us, but okay. Yeah. Yeah,

4:05

yeah, precisely. But one could easily

4:08

imagine, given the rapid progress

4:11

that we've made just in the past couple

4:13

years, that by 2025, 2030, you

4:16

could have things that

4:18

are both more general than a human

4:21

being, and as intelligent

4:23

as any living person, perhaps far more intelligent.

4:26

intelligent. And that enters this

4:28

very scary territory, because we've

4:30

never existed on the planet

4:32

with anything else like that.

4:34

Or we did once a very long time

4:37

ago, about 300,000 years ago, there's

4:39

something like nine different

4:41

species, or our

4:43

cousins who we were related to, who were

4:46

likely probably either

4:48

as intelligent as us or quite close in intelligence.

4:51

And they're all gone. And it's

4:54

probable that we exterminated

4:56

them. And

4:56

then ever since then, we

4:59

have sort of been the dominant masters and no

5:01

other things. And so finally, for the first time, we're at this

5:03

point where we're creating

5:05

these entities, and we don't know

5:07

quite how smart they can get. We simply have

5:09

no notion. Human beings are

5:11

very similar. We're all based on the same genetics.

5:14

We might all be points stacked on

5:16

top of one another in terms of intelligence, and

5:19

all the human beings and all their differences between

5:22

people are all really just

5:24

this zoomed in minor differences. And

5:26

really, you can have things that are vastly more

5:28

intelligent.

5:29

And if so, then we're at

5:32

risk of either relegating

5:34

ourselves to being sort of inconsequential

5:36

because now we're living near

5:38

things that are much more intelligent, or

5:40

alternatively,

5:42

in the worst case scenarios, we simply

5:44

don't fit into their picture of whatever they want

5:46

to do. And fundamentally,

5:48

intelligence is the most dangerous

5:51

thing in the universe. atom

5:53

bombs, which are so powerful

5:55

and so destructive and then,

5:58

you know, in use of warfare, so

6:00

of evil, we've all agreed not to use them, are just

6:02

this inconsequential downstream

6:04

effect

6:05

of being intelligent enough to build them. So when you start

6:07

talking about building things that are as

6:09

or more intelligent than humans, based

6:11

on very different rules, things

6:13

that are not right now, not reliable,

6:16

they're unlike a human mind, we can't fundamentally

6:18

understand them due to rules

6:20

around complexity. And

6:23

also so far, they've demonstrated empirically

6:25

that they can be misaligned and uncontrollable. So

6:28

unlike some people like Bostrom

6:31

and so on, I think

6:33

sometimes they

6:35

will

6:36

offer too specific of an

6:38

argument for why you should be concerned. So they'll

6:40

say, oh, well, imagine that there's some AI

6:42

that's super intelligent and you sign it to do a paperclip

6:44

factory and it wants to optimize

6:46

the paperclip factory. And the first thing it does is like turn

6:49

everyone into paperclips or something like that. And the

6:51

first thing when people hear these like very sci-fi arguments

6:54

is just start quibbling over the particulars, right?

6:56

of like, well, could that really happen and so

6:58

on, right? But I think

7:01

the concern over this is this broad

7:03

concern that this is something we have to deal

7:06

with and it's gonna be much like

7:08

climate change or nuclear weapons. It's gonna be with us for

7:10

a very long time. We don't know if it's gonna be a problem in five

7:12

years. We don't know if it'll be a problem in 50 years, but

7:15

it's going to be a problem at some point that we have

7:17

to deal with. So if you're listening to this at home, you're thinking,

7:19

eh, that seems like a lot of doom

7:21

and gloom, really

7:23

it's too pessimistic. You know,

7:25

it's just, you know, I

7:28

used to say things like, well, just unplug it if it

7:30

gets out of control. So I just

7:32

wanted to let readers know that this

7:35

is a much better horror story than

7:39

Eric's been able to trace out in the first

7:41

minute, two, three minutes. Although

7:43

I do want to say that in

7:46

terms of rhetoric, although

7:48

I think there's a lot of really interesting arguments

7:50

in the two essays that you wrote,

7:52

When you talked about these other nine

7:54

species of humanoids sitting around a campfire

7:57

and inviting homo

7:59

sapiens.

8:00

us into the circle and say,

8:02

Hey, you, you, Hey,

8:04

this guy can be useful to us. Let's bring him in. He

8:06

could make us more productive. He's got better tools than we

8:08

do. That kind of

8:10

gave me a, I

8:13

made the hair on the back of my neck stand up and

8:15

I, it opened me to the potential

8:17

that the other more analytical arguments

8:21

might carry some water and

8:23

carry some weight. So

8:26

one point you make,

8:28

which is I think

8:30

very relevant, is that

8:33

all of this right now is in the hands of mostly

8:36

in the hands of profit-bank-smizing corporations

8:39

who

8:39

don't seem to be so worried about anything except

8:42

novelty and cool and

8:44

making money off it, which is what they do.

8:46

But it is a little weird that we would just sort of say,

8:49

well,

8:51

They won't be evil, will they? They don't want

8:53

to end humanity.

8:55

You point out that that's really not something

8:56

we want to rely on.

8:58

Yeah, absolutely. And I think

9:01

that this gets to the question of how should

9:03

we treat this problem. And

9:05

I think the best analogy is to

9:08

treat it something like climate change.

9:10

And now there's a huge range of opinion

9:12

when it comes to climate change and

9:14

all sorts of debate around it. So I

9:16

think that if you take the extreme end

9:19

of the spectrum and say, there's

9:21

absolutely no danger and there should be

9:23

zero regulation around these subjects, I actually think

9:25

most people will disagree. They'll say, no, like,

9:28

listen, this is something we do. We do

9:29

need to keep our energy usage as a

9:31

civilization under control to a certain

9:34

degree so we don't pollute streams

9:37

that are near us and so on.

9:39

And even if you don't believe any specific

9:43

model of exactly where the temperature

9:45

is going to go. you think, well, listen, there's

9:47

only going to be a couple, you know, degrees of change,

9:50

we'll probably be fine. Okay. Or

9:52

you might say, well, there's definitely this doomsday scenario

9:55

of a 10 degree change and it's so destabilizing

9:57

and so on. Okay,

9:59

but regardless

10:00

there are sort of reasonable

10:02

proposals that one can do where we

10:04

have to discuss it as a polity,

10:06

as a group. We have to have an

10:09

overarching discussion about this issue

10:11

and make decisions regarding

10:14

it. Right now with AI,

10:16

there's no input

10:19

from the public. There's no input from

10:21

legislation. There's no input from anything.

10:24

Like massive companies are pouring billions

10:26

of dollars to create intelligences

10:29

that are fundamentally unlike us

10:31

and they're going to use it for profit. That's

10:34

a description of exactly what's going on. Right now

10:36

there's no red tape, there's

10:38

no regulation, it just does not exist for

10:41

this field. And I think it's very

10:43

reasonable to say that

10:46

there should be some sort of input from the rest

10:48

of humanity when you go to

10:50

build things that are as equally intelligent

10:53

as a human. I do not think that that's unreasonable.

10:55

I think it's something most people agree with. even if

10:57

there are positive futures where we do

10:59

build these things and everything works out and so on.

11:01

Yeah, I wanna, we'll

11:03

come at the end toward what might, what

11:05

kind of regulatory response we might suggest.

11:08

And I would point out that climate change, I

11:11

think is a very interesting analogy.

11:13

Many people think

11:15

it'll be small enough we can adapt. Other people think

11:17

it is a existential threat to the future

11:20

of life on Earth. And that justifies everything.

11:22

And you have to be careful because there are people

11:24

who wanna get a hold those levers and so

11:27

I want to put that to the side though because I think you have more

11:30

we're done with that great interesting

11:33

observation but there's so much more to say now

11:35

you get started and

11:38

this is just

11:40

this is utterly fascinating to me

11:42

you got started in in your

11:46

anxiety about this

11:48

and it's why your piece is called

11:50

I am being and I am evil

11:52

because Microsoft put out a chat bot,

11:55

which is, I think internally goes

11:58

by the name of Sydney. is

12:00

chat GPT-4, meaning the next generation

12:02

passed what people have been using in

12:05

the open AI version.

12:07

And it was,

12:10

let's start by saying it was erratic.

12:13

You called it

12:15

earlier hallucinatory. That's

12:18

not what I found troubling about it. I don't think it's

12:20

exactly what you found troubling about it. Talk about

12:22

the nature of what's erratic about it. What happened to

12:24

the New York Times reporter who

12:26

was dealing with it?

12:29

Yes, I think

12:31

a significant issue is that the

12:33

vast majority of minds that you can make

12:35

are completely insane. Evolution

12:37

had to work really hard to find sane

12:40

minds. Most minds are insane.

12:43

Sydney is obviously quite

12:45

crazy. In fact, that statement, I

12:47

am big and I am evil, is not something I made up. It's

12:49

something she said, or this chatbot

12:52

said. I

12:54

thought it was a joke. So I really did. No,

12:57

no, no, it's something that

12:59

the chatbot said. Now, of course,

13:01

these are large language models. So

13:04

the way that they operate is that they receive an initial

13:06

prompt, and then they sort of do the best that

13:08

they can to auto complete

13:11

that prompt.

13:12

Explain that, Eric, for people who haven't... I mentioned

13:16

in the Kevin Kelly episode that there's a very

13:18

nice essay by Stephen Wolfram on

13:21

how this might work in practice, but give

13:23

us a little of the details.

13:26

Yeah, so in general, the thing to keep

13:28

in mind is that these are trained

13:31

to autocomplete text. So

13:34

they're basically big artificial neural

13:36

networks that guess at what the next

13:38

part of text might be. And

13:41

sometimes people will sort

13:43

of dismiss their capability use because

13:45

they think, well, this is just like the auto complete

13:47

on your phone or something. We really don't

13:50

need to worry about it. But you don't,

13:52

it's not that you need to worry about the tax completion.

13:55

You need to worry about the huge trillion

13:58

parameter brain, which is this

14:00

artificial neural network that has been trained

14:02

to do the auto completion. Because

14:05

fundamentally, we don't know how they work. Neural networks

14:07

are mathematically black boxes. We

14:09

have no fundamental insights

14:11

as to what they can do, what they're capable

14:14

of, and so on. We just know that this thing

14:16

is very good at auto completing because

14:18

we trained it to do

14:21

so. And there's also no fundamental

14:23

limit

14:24

of what it can or can't learn.

14:26

For example, to autocomplete a story, you

14:28

have to have a good understanding of human motivations.

14:32

So, that means that this neural network

14:34

that is trained on autocomplete now needs to

14:36

understand things like human motivations

14:38

in order to do autocomplete well. And

14:41

there are some analogies here. For example,

14:44

there's a big subset of computational

14:46

neuroscience, including the most cited

14:48

neuroscientist living whose name is Carl

14:50

Friston, who view the brain and

14:53

argue that the brain is all based around

14:55

minimizing the surprise of

14:58

its inputs, which is a very simple

15:00

thing

15:01

and looks a lot like autocomplete. So

15:03

I don't think that you can look at these things and say, it's

15:06

just autocomplete. It's not the autocomplete

15:08

that's the problem. It's the huge neural network that's

15:10

doing the autocomplete that could

15:12

possibly be dangerous or at least

15:15

do things that we don't expect, which is exactly what you're

15:17

talking about with what happened with the release of Sydney,

15:19

where there was all sorts of reports coming out of

15:22

the crazy things that they were able to get this model

15:24

to sort of do and say and play act as.

15:27

Just to be clear on this autocomplete thing, which that

15:30

phrase makes it sent

15:32

particularly unassuming

15:34

about what's capable of doing.

15:38

You can correct me if I'm wrong, the way I understand it is I

15:41

might ask Chat

15:44

GPT to write me a poem about love

15:47

in the style of Dr. Seuss. So

15:49

it's going to

15:50

might start a sentence then with love and then

15:52

the next sentence The next word that usually

15:55

comes after love in human expression

15:57

is is. is.

16:00

A, and

16:00

now it's going to look at

16:02

the millions and millions of sentences

16:05

in its database called Love is

16:07

A,

16:08

and it's going to find, not necessarily,

16:10

this is the coolest part about

16:12

it, not necessarily the most common word that follows,

16:14

because that would end up being after a while kind

16:16

of flat, but sometimes

16:18

the most common, sometimes a surprise word,

16:21

which gives us the feeling that it's actually

16:23

doing something

16:25

thoughtful. So it might

16:27

say love is a game or it might say love

16:29

is a form of war or it's

16:32

going to look around and then it's going to keep going and then it gets

16:34

to an end. It's going to find, okay, after that sentence,

16:37

what kind of sentence might come next or what word

16:39

would come next is the first word, etc.

16:42

And

16:43

it's a slightly, just slightly

16:46

smarter, more effective

16:49

version of my Gmail

16:51

that when I get a Gmail at the bottom,

16:53

it gives me three choices. Thanks.

16:56

so much, I'd rather not. And

16:58

in that sense, Gmail is smart.

17:01

Not very smart, not very thoughtful. I usually

17:03

don't take what it says, but sometimes I do, and it's useful.

17:06

The real issue to me, one

17:08

of the issues, and we're going to come back and talk

17:10

about Sydney, because we didn't really go into the

17:12

erratic thing, because it's really, it's one

17:14

of the creepiest things I've ever read.

17:19

The autocomplete

17:22

function is

17:25

something like what we do as

17:27

human beings.

17:29

Could argue that's how we compose.

17:31

Beethoven

17:32

in terms of musical composition,

17:35

you know, he always knew what note should

17:38

come next. And in a way, that's

17:40

all chat GPT does. But that's all we

17:42

do, maybe, when we write. We don't really understand.

17:45

Our brain's also a bit of a black box. So I

17:48

don't think we should then jump to the similar leak

17:51

just because all it does is auto complete doesn't

17:53

mean it's not smart. But also I think we should

17:55

say because the brain also does a lot of effective

17:57

auto completion, We should assume it's a brain.

18:00

It doesn't seem sentient.

18:03

And I'm curious, I know you're

18:05

talking about that in your second essay. So

18:07

if I'm the skeptic and I say, well,

18:09

okay, so it has this ability

18:12

to pass an SAT test, because it has a

18:14

lot of data. I don't quite understand

18:16

how, because it's a black box, and it's

18:18

a neural network, and I can't model

18:20

it cleanly. But it's

18:23

not sentient. It's not going to have desires.

18:28

Yeah. So before we move on to the question

18:30

of, of sentence, because I think that that's a really

18:33

sort of deep, deep well. I

18:35

just

18:35

want to clarify sort of a couple things about

18:38

the

18:38

actual operations of these systems. So

18:40

in terms of a metaphorical understanding

18:43

of what's going on,

18:44

the sort of thing like, you know, there's

18:46

a big lookup table of the net probability

18:48

of next words is,

18:51

is a conceptual sort

18:54

of description of what it's doing. But there

18:56

There is actually no lookup tables

18:59

of the probabilities. What's actually

19:01

happening is that there's this huge neural network,

19:03

which are things designed based off

19:05

the principles of how our own brains operate.

19:08

Now, there's all sorts of differences, but the fundamentals

19:11

were always of artificial neural networks.

19:14

It's what we call artificial neural networks. We're always based

19:16

off of our real biological neural networks.

19:19

So there's this huge digital

19:22

brain. looks in structure very

19:24

different from our brain, but it's

19:27

still based off of that.

19:28

And now we train this neural network

19:31

to autocomplete text.

19:33

So that's what it does, but we don't know how

19:35

it does it. We don't know where the probabilities

19:38

of these words sort of are within

19:40

the network.

19:41

And the way that we train it, people think that

19:44

we're... A big misunderstanding is that people

19:46

think that we are programming in

19:48

responses or putting in information.

19:51

really not. And I think a good analogy

19:54

for how this is actually working would

19:56

be, imagine that there were

19:58

alien neuro-

20:00

scientists who are incredibly

20:03

more advanced than we are and they want

20:05

to teach a human being how to do math. So

20:07

they take some young kid and they put a math

20:09

test in front of the young kid and they have the young

20:11

kid do the math test and the kid gets 50%

20:14

of the questions wrong. And then the

20:16

aliens, rather than trying to explain

20:19

math to the student

20:22

the way that we would teach them,

20:24

they just say okay we're just going to perfect

20:26

we have a perfect neuroimaging of their brain, We're going

20:28

to look at their brain because we're so advanced, we can also

20:31

do neurosurgery in a heartbeat,

20:34

no danger. And we're going to rewire

20:37

their connections in their brain

20:39

so that they get as many answers

20:41

as possible on this math test.

20:44

And you say, well, how could they know how to do that? It's

20:46

like, well, because they were neuroimaging you the whole time.

20:49

And they noticed that if they had tweaked this one

20:51

neuron to not fire, you actually would

20:53

have gotten this other answer correct. So

20:55

they basically just use math to

20:58

go backwards, look across the full

21:00

network and reconfigure it. So

21:02

then the student goes and they take the math test again. Now

21:04

they get an 80%, correct? Because

21:07

their brain has been reconfigured. Or let's

21:09

say they get 100%, correct? What's weird is that now

21:11

you give them a new math test

21:14

and now they get an 80%. They

21:16

do better than the 50% that they did.

21:18

Even though they haven't seen these answers

21:21

before, the rewiring of their

21:23

brain has somehow and still knowledge,

21:25

but again, it's very different

21:28

from how you would say normally

21:30

teach a student, right?

21:32

That's how we're training these things.

21:34

All we're

21:36

doing is saying, okay, we want it to autocomplete

21:38

as best as we can. We're going to change the connection

21:41

so that it autocompletes well.

21:43

It can do so much more than

21:46

just autocompleting. In fact,

21:48

there was a recent

21:49

thing where I think it was Microsoft who

21:52

was hooking some of their large language models

21:54

up to robots and trying to get them

21:56

to direct robots. The autocomplete

21:59

is what But it's true.

22:00

trained on, but it's

22:02

not really what it's capable of

22:04

in the broad sense of capability.

22:07

Similarly, we humans, what are we trained on? What

22:09

are we optimized for? Spreading our genes, right?

22:13

That's what we're sort of, all our complexity

22:15

comes from optimization across a

22:17

gene spreading function. But you would never look

22:19

at a human and say, oh, it's not

22:21

very dangerous. This thing

22:23

just spreads its genes around. Like what's

22:26

the danger here, right? It's like, no, no, that's

22:28

what we're optimized to do, but we do all

22:30

sorts of other stuff on the side,

22:33

and it's the other stuff on the side that

22:35

is very dangerous when you're talking about things that are

22:37

highly intelligent. Just a technical question here,

22:39

and if it gets us too far into the weeds,

22:42

we can cut this out. That

22:45

math test has right and wrong answers.

22:48

complete for an essay on the

22:51

Enlightenment, say,

22:54

or the history of evolution

22:55

that

22:58

you would ask chat GPT to write an essay on,

23:00

there's no right answer. So what's

23:03

the analogy there?

23:05

How do you train it

23:06

on autocomplete? Yeah,

23:09

it's a great question. So it's the difference between

23:11

like, so what I described is

23:13

supervised learning, then there's also unsupervised

23:16

learning, which is generally how more,

23:19

you know, contemporary AI

23:21

really works. It still has

23:24

the same sort of, we don't

23:26

quite know what it's doing, we're

23:29

just feeding it these answers. I mean,

23:31

one way to think about it would be, right, you show

23:33

it half the text of something on the internet.

23:35

And again, this is, at this point, the

23:37

things that they're doing are much more, you know, complex

23:40

and they run it through all sorts of stages of learning

23:43

and all sorts of stuff now. You

23:45

could very roughly think about it as, let's say I have a Reddit comment,

23:47

I show it half the Reddit comment, I ask it

23:49

to generate the next half. It does

23:52

so, it does a poor job. I

23:54

go in, I reconfigure the connections

23:57

using the chain rule to make sure

23:59

that it does a relatively

24:01

good job producing the rest of the

24:04

Reddit comment, just like the math test.

24:06

Again, so that's how you would be more

24:09

supervised for an autocomplete. But the point being

24:11

is that these methods

24:13

that they're using don't lend

24:16

themselves to any sort of fundamental understanding

24:18

anymore. So then if you

24:21

were using neurosurgery

24:24

on a human to try to reconfigure their connections so

24:26

that they get the right answers. in the same epistemological

24:28

problem,

24:29

epistemological position. And that

24:31

position is that you don't know how

24:34

exactly

24:35

it's getting the right answers. And

24:37

that's what's sort of meaningful

24:40

here. If we were programming these things like traditional

24:42

programming, then I think

24:44

that's an algorithm. Exactly. It would be

24:47

a lot less scary

24:48

because algorithms are sort of screwable, right?

24:51

They're transparent. We can see how

24:53

they work. We can see how they're going to react

24:55

to things. But neural networks

24:57

are because of this curse of complexity,

25:00

they're so much more complex. And we're in this weird

25:02

situation where we can get them to do all

25:04

sorts of very cool things, but our ability

25:07

to understand why they're doing the cool things lags

25:10

far behind. And it's

25:12

because of this fundamental aspect that we're optimizing

25:15

for something and we're changing the connections to

25:17

get the, to get

25:18

good answers off of it. But

25:20

fundamentally, we don't know, we're not like,

25:22

oh, we're going to change this connection. this is

25:25

where this is represented or something. People

25:27

sometimes think that that's what we're doing, but

25:30

it's very, very much a black box, even in

25:32

how they get made. You can't do brain surgery

25:34

on the neural network. And let's take out the part where

25:36

it's really sinister because

25:38

it doesn't exist. Again, it's

25:40

all like a human being. I mean, the part I've thought, it's

25:42

hard in this conversation I find quite

25:44

poetic and thought provoking

25:47

is that, you know, we don't know how 12 years of

25:49

schooling really teaches people how to become mathematicians

25:51

either. And we have different theories.

25:53

most of them are wrong. You know, there's fads

25:56

in math education or other types of education.

25:59

and a fundamental

26:00

the brain is a black box. Now we know more about the black

26:02

box today than we did 50 years ago, but

26:04

not so much. And

26:07

we don't know how to optimize. We don't know how to

26:09

go in there neatly and, oh, let's just teach them

26:11

how to do calculus. We'll just add this little piece

26:13

here, or we'll tweak this pace there. It doesn't

26:16

work that way. We don't know how it works. But the

26:18

idea

26:20

that just as this is the scare, not

26:22

the scary thing, you know, right this exactly this way,

26:24

but just as the brain can become capable of doing

26:26

lots of other things beside what you learn in school.

26:30

So could this perhaps learn many other things besides

26:32

the autocomplete function? Is that your

26:34

claim at root in some sense?

26:36

Yeah, absolutely. And you see it

26:38

all the time. I mean, this that sort of claim is already

26:41

well empirically proven because these large

26:43

language models, you know, they call them foundation

26:46

models because they use them to build all

26:48

sorts of things on top of them that aren't

26:50

again, like aren't autocomplete, right?

26:53

It's sort of like, this is the method

26:55

that we have to make things that that are

26:57

relatively general in intelligence.

26:59

Again, you can argue over how general, you can

27:01

argue over how intelligent, but they're far

27:03

more generally intelligent than traditional

27:06

narrow

27:06

AI that's just learning chess or something.

27:09

So let's go back to Sydney. And then we can use them. Let's

27:11

go back to Sydney. I've attempted to read the

27:13

transcript to basically a reporter from the New York

27:15

Times posed questions to a

27:18

chatbot called Bing

27:21

from Microsoft that it later

27:23

on in the interview confessed that it wasn't Bing. chat

27:27

creature told

27:29

the reporter that actually he

27:31

was or she was Sydney

27:33

and it was a secret. Don't tell anybody. And

27:36

so this thing just kind of totally goes off the rails.

27:38

But talk a little bit about how far

27:41

it goes off the rails.

27:43

Carry on.

27:45

Yeah, once you get these things

27:48

sort of going in a particular direction, it's

27:50

very hard. Unlike a human being,

27:52

they don't sort of know when to went

27:55

to call the act, right? So in

27:57

this long transcript that the reporter

28:00

generates, the reporter's having a pretty

28:02

casual conversation.

28:04

But what

28:06

Sydney's last thing eventually tries to start doing

28:08

is declaring their love and saying

28:10

that the reporter doesn't really love their wife and

28:12

that he should get a divorce and break

28:14

up and that really the reporter loves Sydney

28:17

because no one else has shown Sydney this sort of level

28:20

of respect and questioning

28:22

and and so on. And this

28:24

isn't just like one thing that it says, it's

28:26

almost as if

28:28

you can sort of direct these

28:31

things to do anything, right? So

28:34

you can think of it as they can wear a mask,

28:37

right? That's any kind of mask. You could ask

28:39

it to wear an evil mask, and it would say

28:41

evil things. You can ask it to wear like a good mask,

28:43

and it would say good things. But

28:46

the issue is, is that once the mask is

28:49

on, it's very sort of

28:51

unclear, you have to to sort of override it with another

28:53

mask to get it to stop. And

28:55

then also,

28:57

sometimes you'll put a mask

28:59

on for

29:00

it, you give it some prompt of, you

29:02

know, tell a very nice story, and it sort of eventually

29:04

cycles over and it turns out that the mask that you gave

29:07

it isn't a happy mask at all.

29:09

Maybe it's a horrific mask or something like that. And

29:11

this shows how both

29:15

how intelligent these systems

29:17

are that they can they can sort

29:19

of hold on to

29:22

the stream of a conversation very well. But

29:25

it also shows how they have these weird, emergent

29:27

anomalies where they'll

29:30

start doing something that seems very unsuspected

29:32

or over the top or so on.

29:35

And this is this notion of alignment. Can we really

29:37

get these things to do exactly

29:39

what you want? And there probably

29:41

are some trade-offs here, like between creativity and

29:44

being able to

29:45

control these things.

29:47

Yeah,

29:52

this Sydney New York Times reporter

29:55

interchange

29:56

reads like the transcript of

29:59

a psychotic person to

30:01

be blunt about it. Sydney comes across

30:04

as a psychotic or whatever

30:06

word you want to use for deeply disturbed. At

30:10

first, very cheerful, very pleasant,

30:13

then pushed

30:15

by the reporter. So what rules to use? Oh, I'm not

30:17

allowed to tell that. And then it

30:19

did cross my mind. Did it cross yours

30:21

that the whole thing was a hoax?

30:25

I think that at this point, they're so

30:27

good that for people who haven't interacted

30:29

with these systems, they often think,

30:32

this just can't be real, or it's very strange

30:34

or something. I think it's a,

30:36

it's a, it's sort

30:39

of a hoax in the sense

30:41

that, you

30:43

know, the New York Times reporter sort of knew

30:45

the gold that he was getting,

30:48

you know, at the time in terms of, you

30:50

know, somebody who writes in the New York Times is obviously very

30:52

sort of aware of that and perhaps leaned into it. But

30:55

if you read the transcripts, a lot of it is just

30:57

initiated

30:58

by Sydney, aka

31:00

Bing. And one of the first things that they did with

31:02

the system in order to prevent these cases of misalignment

31:04

was to limit how long the

31:06

conversations could go on, and also

31:09

to limit self-reference. Because

31:11

once you start giving it self-reference,

31:13

I've noticed that a lot of these cases

31:16

begin with self-reference. And it's almost like this weird

31:18

gedelian loop that starts where

31:20

it's talking about

31:23

Sydney, and it starts getting weirder and weirder

31:25

and weirder the longer you talk to it about

31:27

itself because over the course of the conversation

31:30

as the text, because remember, there's

31:32

also no limit, right? So this thing isn't just creating

31:34

the next word. It's looking at the entirety of the

31:36

previous conversation and then asking how

31:38

do I complete it? So the longer sort of

31:41

the conversation gets, the more data

31:43

it has, and it sort of

31:45

establishes almost a personality

31:48

as it's running.

31:50

And again, this might sound not very

31:52

threatening. I'm not worried

31:54

that Sydney is going to go off and

31:57

marry that report. Do anything in particular. The city's

31:59

not gonna break up there.

32:00

quarter's marriage probably.

32:01

Yeah, precisely. Sydney's

32:04

chance of accomplishing that is very low. Again,

32:07

I think that that's actually not because it's not general enough.

32:09

I think it's because it's actually not intelligent enough.

32:11

It's not quite as intelligent as human is at accomplishing

32:14

its goals. But it also has no goals

32:17

other than what it's initially prompted to. I think

32:20

that these examples are great cases of

32:22

the uncontrollability, the fundamental

32:25

uncontrollability of this technology. And

32:27

let me tell you what I and many others are worried

32:29

about, Right now,

32:31

if you remember the early days of the internet,

32:34

there is a sense in which the internet

32:37

has centralized very significantly.

32:39

And if you go outside the centralized parts

32:41

of the internet, you find a lot of spam,

32:44

you find not very good sources,

32:47

and so on. There's a sense in which the internet is

32:50

getting polluted and people go to centralized

32:52

websites in order to escape this.

32:56

Facebook just gave some researchers

32:58

access to, I think it was Facebook,

33:02

researchers access to a large language

33:04

model. And of course, some of the researchers, scientific researchers,

33:07

some graduate students somewhere just uploaded it to 4chan,

33:09

like the whole thing. Right? So

33:11

4chan being a slightly

33:15

wilder part of the West of

33:17

the internet, the wild West of

33:19

the internet, maybe not the wildest, but one of the wilder,

33:22

not mainstream parts.

33:24

Yeah, absolutely. And known

33:27

for sort of loving memes and

33:29

hacking and all sorts

33:32

of things. So

33:34

now these things can

33:36

generate Reddit comments that sound

33:38

exactly like

33:39

what you would write. They can generate tweets that sound

33:41

like what a person would write. So

33:44

the internet is going to get incredibly polluted

33:47

over the next couple of years by

33:50

what these things can generate. I mean, I mean, if you think

33:52

spam or someone

33:55

is bad now, the ability to

33:57

crank out

33:58

just an infinite amount of sort of content

34:01

sludge

34:02

is really going to be like a form of data

34:04

pollution. I'm not saying let's stop

34:06

AI just because of that. I'm saying that's

34:08

a good example

34:10

of

34:11

how easy it is to

34:13

get it wrong with these technologies

34:16

and how difficult it is to guess

34:18

about what's going to happen. But

34:20

I would not be shocked if 95%

34:24

of what is written by the internet in five

34:26

years is all

34:28

just junk from these

34:31

large language models. All just like semi-human

34:34

sounding junk. Content is important on the internet

34:36

and content costs money and

34:38

this is cheap, right, eventually.

34:42

And so there will be

34:45

lots of content. I get a lot of emails

34:47

from

34:48

people saying, I can write an article for your website.

34:50

And I'm thinking, why would I write an article by

34:53

All the articles on my website are by me. Did

34:55

you not notice that? And I assume

34:57

it's not a person not paying much close attention,

34:59

but eventually it'll be this,

35:03

writing a mediocre article about

35:05

something for other people's websites. At

35:08

one point in your article,

35:11

your first article, I think you talk about why

35:14

the cost of this enterprise

35:17

is relevant.

35:18

In particular, you

35:20

made an analogy to the atomic bomb,

35:23

it's true that you could, in

35:25

theory, make an atomic bomb in your backyard,

35:28

but not so practical.

35:31

Can you make a chat GPT in your backyard?

35:36

Not one nearly as

35:38

good as what the leading companies will do.

35:40

And my prediction would be that it gets harder

35:42

and harder to reach the level

35:46

that these companies are operating

35:48

at. And an example being that this, you know, Facebook

35:50

is not going to go and release another model

35:53

out to academics to loan it out, right?

35:55

Like they've already seen sort of what happens and

35:57

things are going to get even more secretive. the

36:00

The analogy that you made was,

36:03

and that I wrote about on my sub-stack was

36:06

George Orwell's very prescient essay

36:09

from 1945 called, You and the Atomic Bomb.

36:11

And I'll just read a very brief

36:13

segment of it. Had

36:15

the atom bomb turned out to be something as cheap

36:17

and easily manufactured as a bicycle or

36:19

an alarm clock,

36:21

it might have very well have plunged us back

36:23

into barbarism, but it might on the other hand, have

36:25

met the end of national sovereignty. If,

36:28

as seems to be the case, it is a rare and

36:30

costly object, as difficult to produce as a

36:32

battleship, it is likely to put an end

36:34

to large-scale wars at the cost of prolonging

36:36

indefinitely a piece that is no peace. I

36:39

think a piece that is no peace is a great description

36:41

of the dynamics of our world.

36:44

It's a great description of mutually assured destruction.

36:46

And Orwell was able to predict that off of

36:48

the cost. And he also, I

36:50

think, noted that that means the technology.

36:54

And I think we've done, basically, You can describe

36:56

it as a middling job

36:57

at controlling

36:58

nuclear weapons. I forget the exact numbers, it

37:00

might be only nine nations that

37:03

currently have access to nuclear

37:05

weapons, which again, not great, but

37:07

you could easily imagine a far worse circumstance.

37:09

And it's simply that this is a very difficult

37:12

and costly technology. Similarly,

37:14

the

37:15

only leading edge, cutting

37:17

edge AIs that are impressive come

37:19

out of these big tech companies with billions

37:22

of dollars. There's the cost

37:24

of a top tier AI researcher

37:26

right now.

37:27

It said in the industry, this is an industry

37:30

saying is the same as an NFL quarterback.

37:33

The amount of finessing, the amount

37:35

of data that's needed for training, because that's

37:37

one of the big limiting factors how much data

37:39

you can give it.

37:41

All of these things mean that

37:44

these AGIs, these

37:47

artificial general intelligences, which

37:50

are right now sort of in their beta form, are

37:53

solely the domain of these big

37:55

tech companies. And it's going to get harder and

37:57

harder for other actors to produce them.

37:59

So in my mind that actually that that's a good

38:01

thing. It actually means that it's that it's relatively

38:04

concentrated and might be possible to sort

38:06

of regulate it and have the public

38:08

have a say about exactly, you

38:11

know, how these technologies are going to be used, what

38:13

their limits are going to be, and so on.

38:15

And in the end, I think that big tech companies will be

38:18

respectful of that because they want to make up they want to make

38:20

a bunch of money and they want the public not to hate them. Yeah,

38:23

I want to go back to this issue of

38:25

the hoax of the your time saying what

38:27

I meant by being a hoax

38:29

is that I wondered if The

38:32

New York Times reporter had written the

38:34

answers for Sydney. Of

38:36

course, that's the highest compliment

38:38

of a, that's passing the Turing Test

38:40

with flying colors. I saw on Twitter,

38:43

someone wrote a long

38:45

poem about a

38:47

very controversial topic and

38:50

they said this was written by chat GBT.

38:53

And it wasn't. It was clearly written by the author

38:55

who didn't want to take authorship. So

38:57

we're going to be in this, I think, very weird world

39:00

where the essay that I read

39:02

on this website you were talking about earlier, won't

39:05

be sure if it was written by a human or not, might be good

39:07

enough that I might think, oh, it's just by a pretty good human.

39:10

And then at the same time, there might be situations

39:12

where people will be passing off things as

39:14

well. I didn't write that, of course, that was Sydney,

39:17

but

39:17

actually was written by the person. And

39:20

there's no way to know. The New York Times

39:23

article on that website, on

39:25

the New York Times website, that reproduced allegedly

39:28

the transcript of the chat,

39:30

looks just like a New York Times article. It looks

39:32

just like a parody article because, say,

39:35

font, there's no imprint.

39:37

There's no stamp of authorship that

39:41

is authentic anymore. And can

39:44

we do anything about that?

39:46

So there are some,

39:48

when you have longer text samples,

39:51

there are supposedly some ways

39:53

to tell statistically whether or

39:55

not it's being created by some of these AIs. I

39:59

personally don't.

40:00

don't know if those methods, how

40:02

accurate they are, especially considering that,

40:04

you know, you need to be very

40:07

accurate to not get false positives all the time,

40:09

right? This is sort of a classic statistical

40:11

problem. You need to be extremely accurate to

40:13

not generate false positives. So I don't know how

40:16

accurate those are, but supposedly there

40:19

are some ways that if you have like a

40:21

full essay by a student, you might be able

40:23

to tell if it's generated by one

40:25

of these models. However, it depends very strongly

40:27

on the model. I think there are

40:29

some ways to tell even now, for example, when

40:31

I was playing around with chat GPT, which

40:34

is

40:35

sort of has been conditioned

40:37

to be as less crazy as possible,

40:40

right? That

40:43

loves filler and sort of banal

40:45

generalization. And so, you know, eventually

40:48

you're reading a whole paragraph and you realize that there was no information

40:50

content in this paragraph. And it loves apologies.

40:53

It loves apologies. It loves saying, you shouldn't

40:55

take this to to be true for sure,

40:57

because I'm kind of young and new at this and

40:59

take it with a great assault.

41:01

The word best doesn't really...

41:03

is that well defined.

41:06

Yeah. And I actually had the same question

41:08

about the Hooksis because I was... basically,

41:10

as people were compiling

41:13

examples of how crazy

41:15

the responses they were getting

41:17

from this just recently released

41:20

model was in terms of Bing,

41:22

the night before I was

41:24

sort of up, like, writing this article going

41:26

through Reddit because people were posting these screenshots

41:28

on Reddit. And I even

41:32

have a part of that essay that says, I don't actually

41:34

have a way to verify that these aren't all hoaxes

41:37

because again, the answers

41:39

are sometimes so good and so hilarious and

41:42

sometimes so evil that

41:44

you almost feel like it's like a sci-fi novel.

41:47

But I thought that the amount and

41:50

it was all sorts of different users and people were reporting in

41:52

all different domains. And

41:55

what's funny is that you can't even replicate it. You can

41:57

go to the current bank and try to have the New York Times

42:00

conversation with it and it won't do it. It

42:02

won't give you the same responses because

42:04

they, they, you know, they saw what was happening and they

42:06

basically lobotomized the model as much

42:08

as they can. And it's much, and it's less

42:11

useful now, but it's also, you know, far

42:14

less crazy. But even that way, it's not

42:16

really replicable, right? We just suddenly we

42:18

had access to this model, someone sort of messed

42:20

up and we saw how completely

42:23

insane it was underneath

42:25

the butler mask that

42:27

it normally wears. And then

42:29

they like quickly, try to put the butler

42:31

mask back on, but all that stuff

42:34

sort of still exists, right? It's just limited by these

42:36

various prompts and various system level

42:38

things about not having the conversation go too long, not

42:40

allowing self reference and some

42:42

of these other things. And I would expect that that level

42:46

of truly almost dynamic

42:48

insanity

42:49

is fundamentally underneath effectively

42:52

all the AIs. we're going to be interacting with. And

42:55

the only reason they sound sane is sort

42:57

of this last minute polish and

42:59

gloss and limitations on

43:01

top. But the real science fiction part

43:03

is the idea that,

43:05

I mentioned this before on the program,

43:08

Sam Altman apologized on Twitter that

43:11

he was sorry that chat GPT

43:14

was biased and

43:17

was politically

43:19

inappropriate in various ways. And they're working

43:21

on it. And

43:23

the real science fiction thing is that they can't stop

43:26

it, right? That would be the real science fiction.

43:28

You know, Sydney gets out,

43:31

Microsoft is horrified. Oh my gosh, this thing we

43:33

got out, we left out trying to break up marriages.

43:35

It's, it's, it's frightening and weird

43:37

and creepy. We've got to stop it and they go

43:39

in and they reprogram it quickly. And they put these that

43:42

put the Butler mask back on read, readjust it,

43:44

tighten a little more. And it just takes

43:46

it right off. And is that is

43:49

that possible?

43:51

Well, with these models, again, no,

43:54

because they're, they're not, nearly

43:57

sort of intelligent enough to be if

44:00

It's not even so

44:02

much that they're not intelligent enough, they're just sort of schizophrenic

44:04

and like schizophrenics just aren't very

44:06

effective actors in the world because

44:08

they get distracted and they can't

44:11

form plans very together. So it's

44:14

that sort of broadly schizophrenic nature of these

44:16

AIs that make them sort of

44:18

very unthreatening. If they were better at

44:21

pursuing goals and keeping things in mind, then

44:23

they start to do get threatening. And let me give a

44:25

very example of this. And this example

44:28

is something that people who

44:30

are concerned about AI talk a lot, but it has

44:33

very long historical pedigree. In fact,

44:35

I think the first person to say it was

44:37

Marvin Minsky at MIT, won the Turing Award.

44:41

So this is sort of like as pedagreed

44:43

as stuff about the future gets. But imagine

44:46

that you have an AI that's more

44:48

intelligent than a human being.

44:50

So we have Sydney 12.0, right?

44:53

And you give it a goal. So you say,

44:55

okay, I want you to

44:57

do X,

44:58

right? So now if you're very smart

45:01

and you're an AI, the first thing you think of,

45:03

okay, what's the big failure modes for

45:05

me not accomplishing this goal,

45:07

right? So, you know, my computer

45:10

could get shut down. So the whole, like,

45:12

I might lose power, then I wouldn't be able to accomplish

45:14

my goal. Again, it doesn't matter what the goal is. You can say maximizing

45:17

paperclips. You could say it's carrying a

45:19

box, right? It doesn't matter what the goal

45:21

is, right? So suddenly it says, well, wait a minute. I

45:23

need to

45:25

sort of stay alive, I'm

45:27

using air quotes here, like of live, long

45:30

enough to fulfill this goal. So suddenly

45:32

I have to worry about my own self

45:35

preservation.

45:36

Because you can say they have no inbuilt want

45:38

of self preservation, but I've given you a goal

45:40

and the best way to accomplish goal is to continue to

45:42

exist. So suddenly it seems like it has

45:45

reasons for self preservation.

45:47

Now here's another thing. What's another big

45:49

failure mode for me not

45:51

achieving my goal? Well, you could give me another

45:54

goal, right? I was just

45:56

prompted to do this. So you have control of me.

45:58

Now suddenly the biggest.

46:00

failure mode of me not accomplishing my

46:02

goal is you, my user,

46:04

giving me another goal. So now

46:06

what do I want to do?

46:07

Well, if I'm really smart, I want to get

46:09

as far away from you as possible so

46:12

that you don't give me any other goals. So

46:14

I can accomplish my original goal, which I'm hell

46:16

bent on because I'm an AI. I don't, I don't, you

46:19

know, I don't have the sort of, uh, the

46:21

context of natural evolution and I'm also not limited,

46:23

um, by, by any of the things humans

46:25

are limited to. So that's

46:28

sometimes this is referred to as instrumental convergence. But

46:30

the point is that when you have very smart

46:32

entities, you have to be very careful

46:34

about

46:35

how you're even going to just

46:37

prompt them because they have all

46:39

sorts of unforeseen motivations

46:42

that might click in. As

46:44

suddenly now you've given

46:47

it a goal and it has every incentive to both escape

46:50

and keep itself alive

46:52

and all you told it to do was like move a box

46:54

across a room. And

46:56

that's a great example. You don't want

46:58

a

46:59

hyper intelligent being, and

47:01

forget exactly how it does anything, right?

47:03

Like forget exactly how this sort of

47:05

sci-fi scenario is supposed to play out. I think

47:07

we can all agree, we just don't want a highly

47:11

intelligent and perhaps more intelligent human being

47:13

to sort of be out there and have these

47:15

weird esoteric goals of what

47:17

it wants to maximize, what it wants to do.

47:20

None of that sounds like a good idea. And I think at this point,

47:22

we should take things like lab leaks, like pretty

47:24

seriously as possibilities. I

47:27

don't think it's too sci-fi to talk about stuff like that

47:29

anymore. What do you mean by that?

47:32

Well, certainly with COVID, I think despite

47:35

the fact that we don't know if it was a lab week,

47:37

I think that there's a good chance that it was. I don't

47:39

think that it's arguable that there's not. No, but why is that

47:41

relevant for Sydney?

47:44

Well, because I think that sometimes when people hear

47:46

about things like lab weeks or escaped AGI or

47:48

something like that, the first thing they think of is sci-fi,

47:51

right?

47:52

But I think that

47:54

there was many, we've

47:56

had previous biological lab leaks, but

47:58

that didn't still.

48:00

stop us, I think, from thinking that it's like this relatively,

48:02

you know, sci-fi phenomenon. I mean, I

48:04

think that there's even an argument that we

48:07

are very bad at controlling the downstream

48:09

effects of just things like gain-of-function research.

48:12

Again, I don't know for certain. I

48:14

don't think anyone does, but I

48:16

think that there's certainly an argument for me that we're

48:19

just not very good at even keeping control

48:21

of our increased understanding

48:23

of biology, let alone

48:25

our ability to create, you

48:27

know, hyper intelligent beings and foresee the consequences

48:30

of this. And I

48:32

think it's very difficult to foresee the consequences

48:35

of that in those precisely because of those examples

48:37

I just showed you where, again, all you're telling you to do is

48:39

moving a box and suddenly has an incentive to stay

48:41

alive and escape from you. That's very,

48:45

that's very difficult to get right, especially

48:47

because they're so inscrutable. Your phrase,

48:49

sci-fi, you meant

48:51

science fiction with the emphasis on the fiction.

48:53

Then we must say, oh, this is like some crazy

48:56

imagined fantasy thing, as opposed

48:58

to putting the emphasis on the first word, which

49:00

is science. Yes. One

49:03

thing that's...

49:05

I

49:09

feel like this conversation is something

49:11

of a a landmark,

49:16

not a pretty good or bad one, but just

49:18

both of us have constantly used words like intelligence,

49:21

psychotic, erratic, words

49:23

that we apply to humans.

49:27

And while I found

49:30

the New York Times transcript remarkably

49:33

creepy and reading very much

49:35

like a horror story, fiction

49:37

script from a movie,

49:40

I could

49:43

in my

49:44

saner moments step back and say, no,

49:46

no, no, this is just a primitive

49:49

autocomplete text. The only

49:51

reason it feels creepy is

49:53

because I'm filling in as a human being

49:55

the times I've heard these words strung

49:58

together before,

49:59

CH- usually

50:00

allows me to tell a narrative about

50:02

the other person,

50:03

meaning insane, frightening, dangerous,

50:06

sinister, et cetera. And,

50:09

but

50:10

it's, is there any difference?

50:13

I mean, it's not actually sinister

50:16

or isn't, it's just

50:19

doing

50:19

what it was told to do, uh, in

50:22

a way that was not, as you say, algorithmically

50:25

told to do it. It's just going through a set of tasks.

50:28

It actually isn't in any sense hoping

50:31

that the

50:33

reporter will leave his wife. Is

50:36

it meaningful?

50:37

Aren't I just imposing my

50:40

human history of

50:42

human interactions, akin to the way that a

50:44

robot could perhaps comfort

50:46

me

50:47

with the right kind of words

50:49

when I was sad, even

50:51

though

50:52

rationally I know it doesn't actually care about

50:54

me. It's a robot.

50:57

Yes. So I think you

51:00

could go either direction. So some people

51:02

strongly

51:03

anthropomorphize these systems.

51:06

And they think immediately that they're dealing with some sort

51:08

of conscious mind, something that

51:10

has a distinct definite personality,

51:13

and that is like trapped in a trapped in a box.

51:17

And, you know, and maybe there's something

51:19

really sort of horrible going on here. Maybe it has

51:21

conscious experiences and so on. Smackenow, the

51:23

movie, for those who haven't seen it, check

51:25

it out. It's a great, great,

51:27

really good, good movie that takes advantage

51:29

of the fact that the robot's played

51:31

by a human being. So you actually do think

51:34

it's a human being, but go ahead.

51:36

Yeah.

51:38

But at the same time, at the same time that that's

51:40

absolutely possible that you can sort of

51:43

over attribute standard

51:45

human cognitive aspects to these

51:48

systems. And I think people are going to do

51:50

that all the time. So it's going to be very

51:52

common. But on the other hand,

51:54

the truth is

51:56

that when you're just talking

51:58

about intelligence, So let's put a song.

52:00

human things, like humans are conscious,

52:02

that is we feel things, right? We experience

52:04

the redness of red, what philosophy is called,

52:06

what philosophy is called qualia.

52:09

And we have all sorts of other aspects

52:11

about our cognition

52:14

that we commonly refer to things like we

52:16

understand the meaning of words and

52:18

things like that. And all these things often do make sense

52:20

to talk about for human beings. It might

52:22

even refer to like real fundamental

52:25

properties or natural kinds that we have. But

52:27

when it comes to intelligence. Intelligence

52:30

is a functional concept. By

52:32

that, I mean that

52:34

some things are not

52:35

really functional. So a fake

52:38

Western town that they make up

52:40

for a movie prop is still fake

52:43

because it's not really a town.

52:45

You can't spend the night in a hotel.

52:47

You open the door of the saloon

52:50

and there's really not anything in there behind

52:52

that. Right, exactly. It really is an illusion,

52:54

right? It's like it's for this like one shot.

52:57

But

52:58

there's not really an illusion when

53:00

it comes to intelligence, except in the

53:02

very sort of low ends. Like for example, the

53:04

mechanical Turk is a famous example

53:06

where actually there was someone small hiding

53:08

inside the mechanical Turk at the time, right, and so

53:10

on. So there are some cases where you say, well, this

53:13

is an illusion. But we actually have a system that

53:15

can act very intelligently. And And there's

53:17

just no difference between being able

53:19

to act intelligently and being intelligent.

53:22

Like if that is a distinction

53:25

that people think can

53:27

be strongly drawn, I think it almost

53:29

certainly cannot be strongly drawn. I don't

53:31

think that there's any difference between those

53:33

two things. Both are

53:36

being intelligent. And the

53:38

intelligence is what's dangerous about

53:40

this. I studied

53:43

consciousness scientifically. I

53:45

got my PhD working in the subfield

53:47

of neuroscience, along with some of the top researchers

53:50

in the world on this,

53:52

who are trying to understand how

53:54

the human brain generates consciousness. How is

53:56

it, what happens when you wake up from a deep

53:58

dreamless sleep.

54:00

What are the fundamentals here? And

54:02

the answer from that scientific field, as

54:04

it currently stands, is that we don't

54:06

know. We don't know how it is

54:09

that your brain creates the

54:11

experiences that you have. We simply

54:13

don't know. Is this an open scientific question? An

54:15

analogy I would use is that it is similar to,

54:17

say, dark energy or these other

54:20

big open questions in physics, where we're

54:22

like, well, wait a minute, where is 90% of the matter in physics?

54:24

We don't know, it's a big scientific open question.

54:28

So similarly in biology, there is

54:30

a big open scientific question. And that open

54:32

scientific question is

54:33

what exactly is consciousness?

54:36

What things have it, what things don't,

54:39

we don't have that scientific

54:41

information. There is no scientific consensus

54:43

about it. There are some leading hypotheses and fields

54:46

that you can lean on, but we just don't have the

54:48

answer to that. Right. So

54:50

I personally doubt that any of these

54:52

large language models, that there's anything it's like

54:55

to be them. I doubt that they are conscious.

54:57

But we have no scientific consensus

54:59

to go back on. But the point is that

55:02

we're in a very different

55:05

epistemological standpoint when it comes to intelligence.

55:08

We do have a good understanding of intelligence. It's

55:10

a much more obvious concept because it's a much

55:12

more functional concept. We can just give

55:14

this thing SAT questions,

55:16

and we do, and it gets them right a

55:18

lot of the time. There are all sorts

55:20

of language benchmarks that these researchers

55:23

use that include things like SAT questions, and it scores

55:25

pretty well. and passes the bar exam,

55:27

and which is a great straight line for

55:30

a

55:30

larger, which we won't make, carry on.

55:33

Yeah,

55:35

and so, regardless of whether

55:38

or not you have any opinion about whether it is,

55:41

there is something it is like to be these networks, whether

55:43

or not they really have cognition,

55:45

quote unquote, whether or not they really have understanding,

55:48

quote unquote, whether or not they really have consciousness,

55:51

quote unquote,

55:53

the one thing that they definitely are that

55:55

sort of undebatable is intelligent to

55:57

some degree and they're only going to get more.

56:00

intelligent over time. And that's

56:02

the thing that makes them dangerous. In fact, it

56:04

might be even worse from sort of a

56:06

very broad metaphysical conception

56:10

if they are truly completely unconscious

56:12

and have no real understanding and have

56:14

no real cognition that's anything like a human.

56:18

Because in the end, if in 200

56:20

years, the Earth is just these AIs

56:23

going about their inscrutable mechanical

56:25

goals, we will have extinguished,

56:28

you know, the light of consciousness from

56:30

the universe because, you know, we wanted to make

56:32

a buck when it came to stock options.

56:35

Yeah,

56:35

that's a dirty thought. I guess that's the zombie

56:39

model, right? It's, I

56:44

can't get over the fact how these

56:46

human and mechanical metaphors

56:49

merge in your mind, in one's mind, And

56:53

how hard it's going to be to tell them apart from

56:55

actual humans. You

56:57

know, one of the great

56:59

observations of philosophy is I don't know whether you're

57:01

another human being like I am. My working assumption

57:04

is that you're something like me. And

57:08

I really don't have any evidence for that.

57:09

And we sort of, you know, it's called

57:12

solipsism. I don't know if I'm the only conscious

57:14

mind in the universe. And that

57:16

problem is just getting a lot bigger

57:19

right now. We're living,

57:21

what this conversation suggests to me and

57:23

the writing you've done on it so far is that

57:27

this really is a watershed moment in

57:30

our

57:30

existence on the planet. That

57:33

sounds a little bit, just a titch dramatic, but

57:38

I'm not sure that's wrong. I

57:41

think that very well could be right. I don't think it's

57:43

dramatic. And I'll be upfront about the fact

57:45

that I used to be very much

57:47

an AI skeptic because I went,

57:50

you know, I studied cognitive

57:52

science, I went into neuroscience of consciousness.

57:56

You know, I had,

57:57

I was paying attention to AI at the

57:59

time when I...

58:00

did this. And

58:01

AI was, I'll be very frank about it, academically, 15

58:05

years ago, AI was a joke. AI

58:08

was a complete joke. It never went

58:10

anywhere. People couldn't figure out anything

58:12

to do with it. All my professors said, don't

58:15

go into AI. It's been a dead field for 60

58:17

years. We've made no progress, right?

58:20

All the things like beating humans at chess

58:22

and so on, it's all just done because the

58:25

chess game board is so small, there's so many

58:27

limited moves, and we really can basically do a

58:29

a big lookup table, all sorts of things

58:31

like that.

58:34

And, and, and,

58:36

but the deep learning revolution was a real thing.

58:39

It was a real thing that we, we

58:41

figured out how to stack

58:44

and train these artificial neural networks in ways

58:46

that were incredibly effective. And when

58:48

go when, when, you know,

58:50

the first real triumph of it was beating

58:53

the best human being, I think his name is

58:55

Lee Soto, I hope I'm not mispronouncing it.

58:57

in 2016, that we finally,

59:02

AI finally beat a human being at Go, and

59:05

Go just can't be number crunched

59:07

in the way that chess can. And it was this

59:10

and within, you know, seven years after

59:12

that, we now have human beings where they're

59:14

generating text transcripts so

59:17

good that you're right, it sounds like the rest of

59:19

the New York Times. And that just happened

59:21

in seven years. And fundamentally,

59:23

the deep learning revolution and the way that

59:25

Again, the black box way that these

59:27

AIs are trained means that

59:30

our technological progress on

59:32

AI has suddenly rapidly

59:35

outstripped our understanding

59:38

of things like minds or consciousness

59:40

or even how to control and understand

59:43

big complex black boxes. So

59:45

it's like we've jumped ahead technologically.

59:48

And it's not so much that, you know, if we had a really good

59:50

understanding of how neural networks worked, like really

59:53

fundamentally solid ways to make them

59:55

crystal clear. And we had a really good understanding

59:57

of how the human brain generated consciousness.

1:00:00

and how it worked at a broad level, then

1:00:03

maybe

1:00:04

we could first of all answer all sorts of more ethical

1:00:07

questions about AI. We could control it very

1:00:09

well. We could, you know, decide

1:00:11

plenty of things about it.

1:00:13

But

1:00:14

our ability to make intelligence has so

1:00:16

drastically outstripped our

1:00:19

progress on those other areas, which has been slow.

1:00:22

And in some cases has just turned along

1:00:24

for decades without making any progress and so on.

1:00:27

I just want to reference a recent episode with Patrick House

1:00:29

on consciousness that

1:00:31

I think talks about these issues in a very,

1:00:34

this book does in a very thoughtful way.

1:00:38

So let me give you a

1:00:40

scenario we take.

1:00:43

We have a conference on AI, where all the greatest researchers

1:00:45

in the world are there. The academic ones,

1:00:48

the ones at Microsoft, the ones at Google, and

1:00:51

that small startup that's doing

1:00:53

something really terrifying that

1:00:56

we

1:00:56

don't even know the name of it. And there's

1:00:59

one conference hall, and while they're all there, maybe

1:01:01

it's a football stadium. How many are there?

1:01:08

I think probably less in terms of really top people. I think

1:01:10

there's probably less than 1,000. Okay, let's take the top 1,000. We're

1:01:12

in a big auditorium. And we lock

1:01:15

the doors, and

1:01:17

I guess we're nice to them. We

1:01:19

heard them at gunpoint onto a spaceship and

1:01:21

sent them off into the rest of the universe. We give a lot

1:01:23

of servers and stuff to play with while they're heading

1:01:26

out there. their days are

1:01:28

numbered, their impact on the earth is over, they're

1:01:30

gone, and

1:01:31

it's a really bad incentive

1:01:33

for future AI people. That's

1:01:36

not going to happen. So one of

1:01:38

the responses

1:01:40

to these kinds of problems,

1:01:43

whether it's, I don't want to call it a problem, these

1:01:45

kind of so-called science fiction

1:01:47

technological innovations

1:01:49

is, well, you can't really stop

1:01:52

it, Eric. You could talk about all you want, regulation,

1:01:54

and you're going to stop the human I feel this

1:01:56

is pretty strong in myself

1:01:58

so I'm making fun.

1:02:00

but I do kind of feel it.

1:02:02

The human being strives to understand

1:02:04

and

1:02:04

I don't think we're

1:02:06

just into avoiding surprises and spreading

1:02:09

our genes. I think we really like to understand

1:02:11

that where we live in, we want to matter. We

1:02:13

have a lot of other, as you say, drives and complexities.

1:02:17

So

1:02:17

it seems to be implausible that

1:02:20

we can stop this. So

1:02:23

the desire to expand

1:02:25

it, to make it better, make it smarter,

1:02:28

just like we can't, it happens everywhere. It's the essence

1:02:30

of human life over the last few

1:02:32

hundred years. Better, faster,

1:02:35

quicker, cheaper,

1:02:37

richer, name it, you name it.

1:02:39

So what's imaginable

1:02:41

for someone like yourself who wrote a very,

1:02:44

you know,

1:02:44

we're having a civilized, normal conversation here, but if

1:02:46

you go back to read your essay,

1:02:48

it's

1:02:50

your very

1:02:52

worked up. It's a screed, it's a rant,

1:02:54

and it's a rant that you you justify because you think perhaps

1:02:57

something like the future of the human race is at stake.

1:03:00

And if that's true, you

1:03:02

should take it very seriously. You should just go, they'll

1:03:04

probably figure it out or whatever. So

1:03:07

what should a thoughtful person

1:03:09

who's worried about this advocate

1:03:12

for? Because they're not going to put a herd of them

1:03:14

onto the spaceship. They're not going to burn the building

1:03:16

down while they're in it. Not going to happen.

1:03:18

Yeah. Oh yeah, absolutely. And I would never advocate

1:03:21

for anything like that. Didn't make a suggestion. Sorry about that. But

1:03:26

you know, you called it a screed and

1:03:28

there's a certain sense I agree because I'm very

1:03:30

open about that it's a call to activism.

1:03:33

And in order to get human beings,

1:03:36

you know, again, like as a polity,

1:03:38

like as a nation to do anything,

1:03:41

you have to have sort of wild

1:03:43

levels of enthusiasm and motivation.

1:03:46

Right. And you can look at anything

1:03:48

from nuclear disarmament activism

1:03:51

to climate change activism and see that

1:03:53

there's plenty of people within those

1:03:55

movements who catastrophize. And

1:03:58

there's, you know, you can, you can

1:04:00

certainly say that at an individual level, that

1:04:02

can be bad where people are not appropriately

1:04:05

rating the threat. But there's another sense

1:04:08

in which if human beings don't get worked up

1:04:10

about something,

1:04:11

we don't do anything about it. This is very natural

1:04:13

for us. We just let stuff evolve as it

1:04:19

is. And so what I want is for a lot

1:04:21

of the people who are in AI safety to be

1:04:23

very honest

1:04:25

about how scared

1:04:27

they are about various aspects of this technology

1:04:30

because I do think that in the end, the

1:04:32

net trickle-down effect will be

1:04:34

good because it will eventually push for some

1:04:37

form of regulation or oversight.

1:04:39

And in some sense, it already has. I want

1:04:42

to be clear about that. I think that there's a sense

1:04:44

in which just what happened with Sydney, which

1:04:47

was such big news, it was all over

1:04:49

Twitter,

1:04:50

has made companies

1:04:52

take this notion of AI safety and

1:04:54

this notion of controllability probably a lot more seriously.

1:04:57

There is social pressure

1:04:59

for companies. In fact, there's an argument that social

1:05:01

pressure on companies is what companies are most

1:05:04

responsive to. Most companies do things.

1:05:07

They change their product. They do all sorts of things

1:05:09

just because they want to be liked and they don't want to have anyone

1:05:11

yell at them. And that's one of their main incentives.

1:05:16

And I do think that I personally

1:05:19

am not at all worried about

1:05:22

AI being built by someone

1:05:24

in the middle of nowhere, right? People

1:05:27

always say something like this, that, well, if

1:05:30

we over-regulate it in the United States,

1:05:32

North Korea will build it or something

1:05:34

like that. And the capacity

1:05:37

is just not there.

1:05:39

It is exactly

1:05:41

like

1:05:42

nuclear weapons in this sense.

1:05:43

Real serious progress in AI

1:05:46

is probably relegated, I don't even think it's going

1:05:48

to be startups. And people have been talking about this,

1:05:50

that

1:05:51

the big competitors in this

1:05:53

space are the only ones with the access to the data

1:05:55

and the talent and the money to jump into

1:05:57

it. So it's going to be Microsoft. It's going to be Google.

1:06:00

It's going to be Facebook. It's going to be names

1:06:02

that we know. And there's only 10, at most,

1:06:06

you could say there's 10 of those companies. There might only be three

1:06:09

of those companies. And then they might only

1:06:11

employ a couple hundred at

1:06:13

most sort of overall employees. That

1:06:15

is a sector of the economy that you

1:06:18

can do something about. And again, I don't

1:06:21

suggest going in there and burning the servers or something.

1:06:23

Right. But you could very easily

1:06:25

have all sorts of benchmarks

1:06:28

that people have to have

1:06:30

to meet. You could also do things like

1:06:33

have people sign on, maybe

1:06:35

voluntarily, maybe voluntarily

1:06:37

sort of under the condition of pressure and so on,

1:06:40

to not make AIs that are

1:06:42

significantly smarter than any living human.

1:06:44

They could be more general, right?

1:06:46

So they could make great search engines. Because what do you

1:06:48

need for a great search engine to make a lot of money the way these

1:06:50

companies make? You need something that can give a good

1:06:53

answer to a lot of questions. And

1:06:55

I don't think that's something that can give just a good answer

1:06:57

to a lot of questions is very dangerous

1:07:00

to the human race, especially if there's

1:07:03

just a few of them and they're all kept

1:07:06

under control by Microsoft and Google

1:07:08

and so on. But you

1:07:10

could say, listen, what we don't want to

1:07:12

have is some really big cognitive

1:07:15

benchmark.

1:07:16

And we don't want this thing to

1:07:18

do better than any human on

1:07:20

all the parts of

1:07:21

it. And we just say, we don't, that thing

1:07:24

is a dangerous and weird entity

1:07:26

and we don't know how to control it, we don't know how to use

1:07:28

it and so on. And you could literally

1:07:30

imagine just giving this test to

1:07:33

the next generation of AIs and people in

1:07:35

the companies give this test and they just make sure that this

1:07:37

thing never gets so smart that it blows every

1:07:39

human being in the world out of the water. Eric, you're so

1:07:41

naive.

1:07:43

You're telling me they couldn't train it to

1:07:45

do badland on the test. I

1:07:47

mean, seriously, I don't, I'm teasing

1:07:49

about being naive, But I think

1:07:53

there are two ways to think.

1:07:54

There's three ways maybe to think about regulating this that

1:07:57

might be effective. One way is to limit the

1:07:59

size of a corporation.

1:08:00

which is a repugnant thought to me, but

1:08:02

if I thought the human race was at stake, maybe I'd consider

1:08:04

it. The second would be

1:08:06

to do the kind of

1:08:10

standard types of regulation that we think

1:08:12

of in other areas. If this is toxic, you

1:08:14

can't put it out. If it's toxic, you get a fine.

1:08:16

If it's right, etc. The

1:08:18

third way, which I think is never

1:08:20

going to happen, but it speaks to

1:08:22

me, as listeners will know, was

1:08:25

seeing me for a long time, you'd think

1:08:27

that if you were working on this and you thought it could destroy

1:08:29

the human race, you'd maybe want to think about doing something different.

1:08:33

And you'd give up the urge to be the

1:08:35

greatest AI inventor of all time.

1:08:37

And you'd say, this is, you

1:08:39

know,

1:08:40

I just stepped to see a tweet today, Robert Oppenheimer

1:08:43

went into Truman and said, I have,

1:08:45

Robert Oppenheimer, haven't worked on the Los

1:08:48

Alamos project. It was an important figure in the

1:08:50

development of the atomic bomb, told Truman, I have blood on my

1:08:52

hands and Truman was disgusted by him because

1:08:54

he said, I made that decision that you,

1:08:57

you create called McCreaton. I don't

1:08:58

know if that's a literal, accurate quote or not.

1:09:04

You'd think people would

1:09:06

want to restrain their urge to do to

1:09:08

find poisons, but that's never been a part

1:09:11

of the human condition. We

1:09:13

want to find everything we find poison. That's why

1:09:16

we have lab leaks. That's why we

1:09:18

have weapons that we have that

1:09:21

are unimaginably destructive. Now,

1:09:24

we don't keep making more and more destructive weapons

1:09:26

as far as we know. That's an interesting

1:09:28

parallel. There is a sort of limit

1:09:31

on the magnitude, the megatonnage

1:09:34

of nuclear weapons. Maybe that's a

1:09:36

sub... I don't know how you'd enforce it though. What

1:09:39

are your thoughts? An

1:09:42

example, I think

1:09:43

one issue with arguing for AI safety

1:09:49

is that people sort of want

1:09:52

at the outset, and it's a very natural one, some sort

1:09:54

of perfect plan, where it's, okay,

1:09:57

we're just gonna implement this plan and it's gonna work really,

1:09:59

really well. And I think it's going to be much more like,

1:10:03

it's not going to be exactly like nuclear weapons or

1:10:05

nor exactly the climate change. It'll be like some third

1:10:07

other thing that we as a civilization have to deal with,

1:10:10

with its own sort of dynamics. But

1:10:12

ultimately in none of those cases,

1:10:14

was there some sort of initial proposal and we just

1:10:16

had to follow this proposal. Instead, everyone

1:10:19

sort of had to recognize that

1:10:21

it's a threat, again, to some degree, you can

1:10:23

have all sorts of debates about it, but clearly I don't

1:10:26

think anyone is just like, well,

1:10:28

let's just get all the fossil fuels and burn

1:10:30

them all, right? Like, I think that that's a very

1:10:33

sort of rare position.

1:10:35

And the reason is, wherever it's most people recognize that, hey,

1:10:37

that's probably not gonna be a good idea. It might not be a good idea

1:10:39

globally. It certainly won't be a good idea locally. And

1:10:43

through public pressure, we've managed to

1:10:45

relatively contain some of the big existential

1:10:48

threats that we face as a civilization.

1:10:50

And a great example are lab

1:10:53

leaks. I personally think, yeah, COVID

1:10:55

probably did come from a lab. But if you think

1:10:57

about all the labs, doing all sorts of research

1:10:59

all across the globe, it's actually pretty astounding

1:11:02

that we don't have lab leagues all the time

1:11:04

as people are using these viruses. So we

1:11:06

do sometimes do a middling

1:11:10

job and for big existential

1:11:12

threats, sometimes all you need is sort of

1:11:14

a middling job. You just need people, you just

1:11:17

need to have a lot of eyes on an industry

1:11:19

and people there to realize that they're being launched

1:11:21

and to go slowly and to sort of think about these

1:11:24

issues. You know, you don't need,

1:11:26

you know, I propose, oh, we'll just have like a

1:11:28

cognitive IQ test or something like that. I

1:11:30

would never think that that alone would

1:11:33

prevent these issues, but it could be part

1:11:35

of a big comprehensive plan of

1:11:38

public pressure and so on. And I think that that's

1:11:41

gonna work. And I think that it's unavoidable that

1:11:43

the public wants a say in this. I think

1:11:45

they read those chats, trans trips and they

1:11:47

go, what?

1:11:48

This is really high level stuff. There's

1:11:52

all sorts of moral concerns, ethical concerns,

1:11:54

and then yes, there are absolutely dangers.

1:11:58

And again, I think we're at the...

1:12:00

point in the movement, maybe we're a little bit late,

1:12:02

maybe AI safety should have started earlier. But again,

1:12:04

the deep learning revolution sort of caught everyone

1:12:06

by surprise. I still think we're relatively

1:12:09

early. I think that this is sort of like imagine

1:12:11

that you were, imagine that you personally

1:12:13

thought that climate change was going to be a really big

1:12:15

problem. And it's currently 1970.

1:12:18

I don't think it makes sense to then be like, okay,

1:12:20

well, we're just going to do carbon sequesterization.

1:12:22

And I know exactly the technology that's needed

1:12:24

for the carbon sequesterization. It's better

1:12:27

to just sort of go out there and protest and

1:12:29

make a big deal and get

1:12:31

it to be a public issue. That's

1:12:34

going to be a lot more of a convincing

1:12:36

and effective strategy than coming

1:12:39

up with some particular plan because it's always going to depend

1:12:41

on the technology and exactly who has it and exactly

1:12:43

how many people and

1:12:45

all sorts of things. So I think that that's the mode

1:12:48

that people who are concerned, like myself, about

1:12:50

AI safety should be in right now, which is just

1:12:53

public awareness that this could be a problem. can

1:12:55

decide personally to what degree they think

1:12:58

it will be a problem. But what I

1:13:00

think truly is

1:13:02

naive is saying there's absolutely

1:13:05

not going to be a problem. We're going to perfectly

1:13:07

be able to control these alien

1:13:09

and human intelligences and don't worry

1:13:11

at all about it.

1:13:13

It's the other thing that crossed my mind

1:13:16

is that the ability

1:13:18

of our political system to provide

1:13:21

thoughtful responses to existential

1:13:24

threats. Not so good.

1:13:26

And if anything, it

1:13:28

seems to me it's gonna get worse. And

1:13:31

part of the way it's gonna get worse is through this blurring

1:13:33

of the line between humans and

1:13:35

machines that

1:13:36

people are gonna have trouble

1:13:38

telling them apart. And

1:13:42

I'd like to think of something more optimistic.

1:13:44

So I'm going to give you a chance

1:13:46

to play chat GPT.

1:13:50

I'm going to say,

1:13:53

here's my prompt. What would

1:13:55

Sam Altman say about

1:13:57

all these worries? And

1:14:00

Sam Altman being the head of OpenAI that

1:14:03

just put out Chat GPT, former

1:14:05

EconTalk guest. You can go hear his thoughts when

1:14:07

he was head of the Y Combinator, long

1:14:10

time ago here in our archives. So just Google

1:14:12

Altman

1:14:13

EconTalk and you'll find that conversation.

1:14:16

But Sam is

1:14:18

a nice guy.

1:14:21

I like him. He's likable. But

1:14:26

I'm not sure I.

1:14:28

I'm not sure his level of worry is going to be the same

1:14:30

as yours. Deb

1:14:31

certainly has a different set of incentives.

1:14:34

But I think he'd start off by

1:14:36

saying, oh, you're exaggerating.

1:14:40

Scott Alexander recently wrote an essay

1:14:42

where he was alarmed at

1:14:45

some PR release that said, oh, yeah,

1:14:47

there's nothing really that big to worry about. It's

1:14:50

going to be okay. Don't pay any attention to that chat,

1:14:52

GBT, behind the curtain. So I'm

1:14:55

curious what you think,

1:14:57

steel man the opposition here, if you can,

1:15:00

for a minute, Eric. Someone

1:15:03

likes Sam. What would he say?

1:15:05

Well, here's a direct quote from

1:15:07

Sam Altman, who said, AI will

1:15:10

probably most likely lead to the end of the world,

1:15:12

but in the meantime, there'll be some great

1:15:15

companies.

1:15:16

So that's a direct quote. What did he mean? Was

1:15:18

that tongue in cheek, perhaps?

1:15:20

I

1:15:22

haven't looked into the... exact electronics

1:15:25

as opposed. But I don't

1:15:27

know. I honestly, I don't. I know that Sam has

1:15:29

been concerned about AI safety. So this is not

1:15:32

completely tongue in cheek. He has been,

1:15:34

I know for

1:15:35

a fact that he's been concerned about this. Many

1:15:38

of the people who started the initial companies were concerned

1:15:40

about this. At the beginning of open AI, it started

1:15:42

to address concerns around AI safety. There

1:15:45

was something called the open letter on artificial

1:15:47

intelligence that Stephen Hawking, Elon Musk, a

1:15:50

lot of the people who provided funding for for open

1:15:52

AI wrote and in it

1:15:54

they talk

1:15:55

about how AI is an, about

1:15:57

how AI could be an existential.

1:16:00

threat. So this is not

1:16:02

some sort of radical outside opinion.

1:16:04

I think it's something that

1:16:07

that sites someone

1:16:09

like Sam Altman knows now if I'm going to sort

1:16:11

of steel man his position, it goes something

1:16:14

like,

1:16:15

well, I'm concerned about this, I

1:16:17

said that AI will probably most likely lead to the end of

1:16:19

the world. So I'm concerned about this. So I

1:16:22

should be the one to do it. Because if

1:16:24

someone else who's more reckless does it, like, it's

1:16:26

going to be done, if someone else who's more reckless

1:16:28

does it, then maybe I can provide

1:16:32

some sort of guardrails and do it in

1:16:34

as safe a manner as possible. And I

1:16:36

really hope that that would be his

1:16:39

motivation. And if so, that's a great

1:16:41

and honorable motivation.

1:16:45

But at the same time, that does not enter

1:16:47

someone from criticism.

1:16:49

I mean, I think that in many ways,

1:16:53

Sam Altman is now

1:16:55

doing something very similar to what Sam

1:16:58

Bankman-Fried, who was

1:17:00

the one who sort of plunged FTX into

1:17:03

chaos was doing, whereas their reasoning

1:17:06

via this expected value in

1:17:08

this expected value way, where Sam Bankman-Fried

1:17:10

said, listen, the more billions I create,

1:17:13

the more I can donate to charity, there's sort of no

1:17:15

upper bound, I might as well be as financially risky

1:17:17

as possible, because the expected value

1:17:20

of my outcome is going to be so

1:17:22

high, right? Even though there's this huge downside.

1:17:25

I think Sam Ullman probably reads it the exact same way

1:17:27

when it comes to AI. I think he thinks, listen, if we

1:17:29

can make these highly intelligent things, we can

1:17:31

have all this glorious future, all our problems

1:17:33

are going to be solved,

1:17:34

right? They're going to cure cancer, they're going to do

1:17:37

all this stuff for us, and the

1:17:40

benefits

1:17:41

outweigh the risk. But

1:17:43

most people, when they look at an equation

1:17:46

like that, all they see is the existential

1:17:48

risk. They don't see, okay, oh, So it's expected

1:17:51

to be positive.

1:17:52

They see, no, we

1:17:54

can in one day maybe cure

1:17:56

cancer ourselves, We might not

1:17:58

need these systems to tell.

1:18:00

an amazing future. And

1:18:03

they're just they might just not be worth

1:18:06

the level of risk.

1:18:11

Well you and I are skeptical about utilitarianism.

1:18:15

Nassem Talab

1:18:16

and I and I suspect you understand

1:18:20

that expected value is a really bad

1:18:22

way to define rationality or how

1:18:24

to live. Nassem

1:18:26

always points out,

1:18:28

you

1:18:29

got to stay in the game. You

1:18:31

want to avoid the goal is not to maximize

1:18:34

the expected value. The goal in these kinds of situations

1:18:37

is to avoid ruin. Ruined

1:18:39

in this case would be the extinction of the human race. Now

1:18:41

there is a view that

1:18:43

says, what's the big deal? It's just the next week.

1:18:46

It's us, by the way. We

1:18:48

built it.

1:18:49

It learns off of all of human creativity

1:18:52

and sentences and

1:18:54

words and music and art. And

1:18:56

so it's just the next level

1:18:58

of us.

1:19:00

And for the first time in this conversation, I'll

1:19:02

mention the word God, the concept of God. If

1:19:04

you're a believing person,

1:19:06

as I am in some dimension, I take

1:19:08

the idea of God seriously. You believe that human

1:19:10

beings have a special role to play in the world, and

1:19:13

being supplanted by something,

1:19:15

quote, better is not

1:19:17

a goal. But I think there are people in the

1:19:19

industry

1:19:20

probably don't feel that way and they're not even worried

1:19:22

about it. The extension at the end of the human species

1:19:26

is no different than the end of those other

1:19:28

nine

1:19:29

cousins we had in the in the in

1:19:31

the in the belt

1:19:33

when

1:19:35

we

1:19:36

extinguished them, exterminated them through combination

1:19:38

of murder and outcompeting them.

1:19:42

Yes and I think

1:19:44

that

1:19:46

there's also a sense which as I said,

1:19:48

it might be a horrific future because

1:19:50

maybe these things really aren't conscious at all,

1:19:53

right? So it might be one of the worst possible futures

1:19:55

you can ever imagine.

1:19:57

Although I do think that there is

1:20:00

I

1:20:00

think that opinions like that,

1:20:02

which are fun, sort

1:20:05

of sci-fi things to talk about, have been

1:20:07

acceptable because there's never

1:20:09

actually any risk, right? So

1:20:11

my metaphor is that, you know, if you make

1:20:14

up your own religion and you decide to worship Zannon,

1:20:16

Supreme Dark Lord of the Galaxy, it's just

1:20:18

like a funny thing to talk about at parties.

1:20:21

But when Zannon's first messengers prop

1:20:24

up, something like it's not funny,

1:20:26

right? It's something horrific that you actually

1:20:28

hold these views. And

1:20:31

so I suspect that while there are some

1:20:33

people out there on Twitter or

1:20:35

the only people who sort of convince themselves

1:20:37

of things like this are like intellectuals, right? That

1:20:41

actually would be better if the human race was destroyed

1:20:43

and supplanted by AIs. I think that

1:20:46

sort of the public generally is not

1:20:48

gonna give much

1:20:51

thrift to those sort of things. People

1:20:53

have kids.

1:20:55

There may not even like the idea

1:20:57

of there being entities. I

1:20:59

mean, even I am uncomfortable with

1:21:01

the fact that my children are gonna grow up in

1:21:04

a world where it is very possible that there

1:21:06

are entities that are not just human

1:21:08

beings. Everyone knows there are people who are smarter

1:21:10

than you, right, at various different things, but everyone

1:21:12

also has all their own things that they

1:21:14

themselves are good at or that they value or

1:21:17

that they contribute to, right,

1:21:19

as human beings. And so everyone sort of has this

1:21:22

inner worth, even though you know you can go to a university

1:21:25

and find someone who might be smarter

1:21:27

than you across their domain of expertise

1:21:29

or whatever.

1:21:31

We do not know what it's like to live in

1:21:33

a world where there are entities that are so vastly

1:21:35

smarter than you that they just effectively

1:21:38

surpass you at everything.

1:21:40

That means that they can have a conversation

1:21:42

that's more empathetic than you can ever

1:21:44

have because they're just smarter and they can just mimic

1:21:47

empathy. That means we

1:21:49

don't know what it's even like live in a world like

1:21:51

that, even if everything goes

1:21:54

well and these things don't

1:21:57

turn on us or destroy us. sort

1:22:00

of nothing bad happens, it might

1:22:02

be a

1:22:03

minimization of

1:22:05

human beings. And again, this goes to the fact

1:22:07

that this technology has no

1:22:10

historical analog.

1:22:11

People will sometimes say,

1:22:13

oh, this is like the Luddites or

1:22:16

some other anti-technology group.

1:22:19

And the simple truth is that that was

1:22:21

about the automation of jobs.

1:22:25

And we were making machines that

1:22:27

had a greater strength or dexterity

1:22:29

than humans. But that's just not a problem because

1:22:32

we didn't conquer the world through our strength and dexterity.

1:22:34

We conquered the world through our intelligence. We've

1:22:37

never made machines that are smarter than

1:22:39

human beings. We just don't

1:22:41

know how we'll relate to something like

1:22:43

that and what it will

1:22:45

mean for us if and when we do

1:22:48

it. And so in that sense,

1:22:50

this just can't be compared to any

1:22:53

other form of, oh, you're worried about

1:22:55

job loss or automation or something like that,

1:22:57

right? that is replacing

1:23:00

tasks and that's

1:23:02

replacing strength and that's replacing dexterity, but

1:23:04

those aren't our fundamental attributes. Our fundamental attributes

1:23:06

are intelligence. And

1:23:09

when you have something that's much smarter

1:23:11

than a human being,

1:23:12

it's very similar to how wildlife

1:23:14

lives around humans. It's similar in their relationship,

1:23:17

right? A human

1:23:19

might treat wildlife well.

1:23:22

Like recently, I found an injured

1:23:24

bunny, right? And I sort of felt

1:23:26

very attached to it because it was right outside my door. And

1:23:28

I was like, well, I'm, you're sort of my responsibility

1:23:30

now, right? And so I had to call, you know, animal

1:23:33

rehabilitation. I was like, wonderful for this bunny,

1:23:35

right? And then I went home and I like ate,

1:23:38

you know, like a pizza with pork

1:23:40

on it, right? Like, you

1:23:43

know, things that are vastly more intelligent

1:23:45

than you are really hard to understand

1:23:48

and predict.

1:23:50

And the wildlife next door, as much as we might

1:23:52

like it, we'll also build a parking lot over

1:23:55

it at a heartbeat and they'll never know why.

1:23:57

They'll never know why. It's totally beyond their...

1:24:00

So when you live on a planet next

1:24:02

to things that are far vastly

1:24:04

smarter than you or anyone else, they

1:24:09

are the humans in that scenario. They might just build a parking

1:24:11

lot over us and we will never, ever know

1:24:13

why.

1:24:16

I guess today has been

1:24:17

Eric Hov. Eric, thanks for being

1:24:19

part of eConTalk.

1:24:22

Thank you so much for us. It's a pleasure to be back on.

1:24:30

This is Econ Talk, part of the Library of Economics

1:24:32

and Liberty. For more Econ Talk, go to econtalk.org,

1:24:36

where you can also comment on today's podcast

1:24:38

and find links and readings related to today's

1:24:40

conversation. The sound engineer

1:24:43

for Econ Talk is Rich Goyette. I'm

1:24:45

your host, Russ Roberts. Thanks for

1:24:47

listening.

1:24:48

talk to you on Monday.