0:25

David, the U. S. State Department recently

0:26

commissioned a report about how A.

0:32

I. employees feel about the safety of their

0:32

work and the incentives driving it.

0:38

And let's just say the report

0:38

was not particularly positive.

0:41

It turns out that a lot of people

0:41

working in the field of AI have a

0:47

lot of concerns about its safety.

0:50

Yeah, and the fact that it's coming

0:50

from people who work in this industry, I

0:54

think is, particularly telling indicator

0:54

of the fact that the general population,

0:59

ourselves included, don't really know

0:59

the full extent of the risks that are

1:05

associated with these new technologies. It's similar to, all the people who work

1:07

for places like Facebook that don't allow

1:12

their children on social media because

1:12

they know just how addictive it is.

1:17

So the fact that the people with

1:17

knowledge of the inside are the

1:20

ones that are, raising the red flag

1:20

should be a sign that we should pay

1:24

attention to this a little bit more.

1:26

Yeah, the authors of the

1:26

report that I mentioned spoke with

1:28

over 200 experts for it, including

1:28

employees at OpenAI, Google,

1:34

DeepMind, Meta, and Anthropic.

1:36

And these are AI labs that are

1:36

working towards what's known as

1:39

artificial general intelligence. Now the term AI is a very odd one,

1:41

like artificial intelligence, but

1:47

the basic idea is that techies are

1:47

working to create a simulation of the

1:54

intelligence that humans recognize

1:54

as such, which is like mostly

1:57

cognition, but also imagination, right?

1:59

So the development of Dall-E was not just

1:59

to stiff artists and have AI become like

2:07

the new creator of images, but actually,

2:07

I heard the head of frontiers research

2:12

at OpenAI, Mark Chen speak about in 2022.

2:17

He said that the idea was that In

2:17

order to have an artificial general

2:20

intelligence, you also have to

2:20

have image making capabilities.

2:23

So basically, even if artificial

2:23

intelligence is a misnomer, the

2:26

idea behind it, or the idea behind

2:26

artificial general intelligence, is

2:30

that we're trying to simulate the

2:30

full range of what humans recognize

2:36

to be intelligence, and in simulating

2:36

it, find ways to make it even better.

2:43

And so the reason I hesitate in talking

2:43

about this a little bit is because what

2:46

humans consider to be intelligence,

2:46

I think, is a pretty open question.

2:50

The fact that it's only recently

2:50

that those in the AI community have

2:53

considered image generation to be

2:53

a big part of it is pretty telling.

2:56

A lot of our intelligence notions have

2:56

to do with cognition specifically,

3:01

but also this, idea behind, what's

3:01

the point of it, I think, is like

3:06

something I have no good answers for,

3:06

but I have in the back of my mind.

3:11

Yeah, and I think one of the questions that we should be asking about a lot of this work

3:12

in artificial intelligence is what

3:17

definition of intelligence are they

3:17

working with to begin with that

3:20

they're trying to replicate, right? Because often a lot of the people in this

3:21

area are computer scientists, engineers.

3:28

People who define intelligence largely

3:28

just as information processing.

3:32

And when you adopt a broader

3:32

interpretation of intelligence that

3:35

includes, let's say, social intelligence,

3:35

emotional intelligence, moral thinking,

3:41

it starts getting more difficult to

3:41

know whether that's something that

3:44

you can replicate in a machine.

3:47

Also, one thing that I want to

3:47

mention in this context is that

3:50

it's not just that they're trying

3:50

to replicate human cognitive

3:52

capacities, as you mentioned, Ellie. Is that especially when you're thinking

3:54

about people who are closer to the

3:57

transhumanist community, they believe

3:57

that they are going to replicate

4:02

human intelligence in a machine. And once that intelligence is

4:05

created, we will merge with it

4:10

through computer-human interfaces such

4:10

that a new super intelligence will

4:15

emerge that is neither exclusively

4:15

human nor exclusively machinic.

4:20

And this idea of a possible

4:20

future where humans and machines fuse

4:25

is both for some a doomsday scenario

4:25

and for others a utopian scenario.

4:32

The idea is that machines won't be working

4:32

for us or, us working for machines.

4:39

But it's not even that we'll be

4:39

working together, it's rather

4:42

that like we will be one, right?

4:47

Which I think is attractive in certain

4:47

respects because I think a lot of

4:50

people's concerns around AI safety

4:50

have to do with this idea that in the

4:54

future we're going to have AI overlords.

4:57

But I think, for others risks losing

4:57

what makes humans or like what makes the

5:06

organic and especially once you bring

5:06

in the realities of the late capitalist

5:11

world that we live under even if you

5:11

think that in principle humans becoming

5:16

machinic and machines becoming human

5:16

like such that we ultimately merge

5:22

could be ideal or could be a good thing

5:22

I think many of us are concerned that

5:27

we'll end up getting co opted by a

5:27

profoundly exploitative economic order.

5:32

Henry

5:33

No, and I love how the same

5:33

scenario is literally utopia for one

5:37

group and dystopia for another group.

5:39

And it's interesting to think that

5:39

distinction between utopia and dystopia

5:43

might map onto the distinction between

5:43

employee and boss in contemporary

5:49

technology, because you mentioned,

5:49

that a lot of the employees who work

5:52

in tech today have a lot of worries

5:52

about the safety and the future of AI.

5:57

But that's very different than the

5:57

attitude that we find in what I sometimes

6:01

call the golden boys of contemporary tech,

6:01

like the Ray Kurzweil of the world, the

6:07

Elon Musk, so on and so forth, right?

6:09

For them, the bosses, it

6:09

really represents a utopia.

6:14

And I think there is no better

6:14

example of this than precisely Ray

6:17

Kurzweil's book, The Singularity is

6:17

Near, which came out a couple of years

6:21

ago, and it's a behemoth of a book. It's about 650 pages of him talking

6:23

about this utopian moment that is going

6:30

to happen in the near future that he

6:30

calls the singularity, which is the

6:35

moment when artificial intelligence will

6:35

not only surpass human intelligence,

6:39

but will also merge with it.

6:42

And he argues that at that point, we

6:42

will become a kind of super organism

6:46

that is nothing but disembodied

6:46

information and knowledge in like

6:50

a purified, crystallized form.

6:55

Kurzweil's position has been

6:55

attractive for a lot of people, but also

7:00

easy to parody because he does read as so

7:00

utopian about the potential of technology.

7:07

There are, a lot of people who have been

7:07

voicing significant concerns recently

7:13

about artificial general intelligence

7:13

specifically, and I think that is,

7:18

where we want to begin thinking about

7:18

the topic for this episode, which is

7:23

ultimately going to be less about like

7:23

actually what recommendations we have for

7:26

AI safety, not like you and me, we, but

7:26

even like the actual community, right?

7:31

Our guest today is an expert on

7:31

the character of the AI safety

7:37

communities rather than on if we do

7:37

X, Y, Z, we will be safe from AI.

7:42

But I think a few of the concerns that

7:42

are worth pointing out come from the

7:46

philosopher Nick Bostrom, who wrote

7:46

a really influential 2014 book called

7:50

Superintelligence that ended up having

7:50

a big influence on Elon Musk and a

7:55

bunch of other people in the tech world. And in this book, Bostrom suggests

7:57

that the rise of artificial general

8:02

intelligence can cause some significant

8:02

issues, including displacement of the

8:07

workforce by artificial intelligence,

8:07

political and military manipulation,

8:13

and even possibly human extinction.

8:16

And I think this threat of human

8:16

extinction that Bostrom articulates in

8:20

the book has been one of the main drivers

8:20

of recent concerns about AI safety, in

8:26

addition to the fact that our AI has

8:26

just improved significantly since Bostrom

8:30

published this book ten years ago.

8:32

Yeah, and Bostrom has this

8:32

really fun kind of thought experiment

8:35

to highlight some of these high end,

8:35

even if low probability, doom scenarios.

8:40

A fun thought experiment

8:40

about human extinction.

8:43

about human extinction.

8:44

So philosopher of him.

8:46

Yeah, just like turning everything

8:46

into an interesting theoretical point.

8:50

No, but it's called the

8:50

paperclip maximizer.

8:54

So he says, imagine that we create

8:54

an intelligent machine that has one

8:58

specific goal, and that goal can be

8:58

very benign, like making paperclips.

9:03

The problem is that we humans have

9:03

no way of guaranteeing that we

9:07

can predict exactly how that ends. algorithm or that machine will

9:09

go about achieving that end.

9:13

So it could be that the machine

9:13

decides, using its general intelligence

9:19

that it's been equipped with by its

9:19

human creators, that it needs to

9:23

use any possible resource in order

9:23

to create more and more paperclips.

9:28

And then it starts literally

9:28

killing human life in order to

9:32

produce more and more paperclips. And so here, it's not as if human

9:34

extinction would come about from an

9:40

algorithm that turns evil and wants

9:40

to dominate us, it would actually

9:44

just be something much more mechanical

9:44

and mundane that we can't control.

9:50

And so he's trying to make some

9:50

of the risks of AI more palpable

9:54

to a general audience by making

9:54

them seem more realistic.

9:58

By talking about

9:58

death by paperclips.

10:00

Yeah, it's not that they become diabolical. It's just that they are machines

10:02

and they are indifferent to us.

10:05

That's the point.

10:06

I feel like this thought

10:06

experiment is what you would get if

10:08

you asked Kafka to write office space. Let's talk with our expert guest

10:13

today who has far more intelligent

10:17

things to say about AI safety than

10:17

either of us could do justice to.

10:24

Today, we are

10:24

talking about AI safety.

10:27

How have philosophies stemming

10:27

from utilitarianism become so dominant in

10:31

discussions of AI safety in today's world?

10:34

What are the most salient

10:34

risks associated with AI today?

10:38

And how has the rise of

10:38

funding and interest in AI safety

10:42

come to shape today's understanding

10:42

of what the risks posed are?

10:48

Shazeda Ahmed is Chancellor's

10:48

Postdoctoral Fellow at UCLA's

10:52

Institute of American Cultures. A specialist in AI safety, AI

10:54

ethics, and technology in China, Dr.

10:59

Ahmed has held positions at

10:59

Princeton University, Citizen

11:02

Lab, and the AI Now Institute. In addition to her peer reviewed work, Dr.

11:07

Ahmed's research has been featured

11:07

in outlets including Wired,

11:10

Financial Times, The Verge, and CNBC.

11:14

Dr. Ahmed, welcome to Overthink.

11:16

We are so excited to have you.

11:18

Oh?

11:19

And there is so much

11:19

demand for us to have an expert

11:22

on AI and ethics and safety.

11:25

So there are a lot of people out

11:25

there who are our fans who are just

11:28

dying to hear this conversation. So thank you for your making the time.

11:31

Thank you so much for the invitation.

11:33

Let's jump in by thinking about

11:33

the recent explosion of interest in

11:37

AI, which has happened especially

11:37

in the wake of the recent successes

11:41

of LLMs, large language models.

11:43

I'm here thinking about especially

11:43

ChatGPT, but also other models.

11:48

And it makes sense that a lot of

11:48

people are posing questions about

11:52

the safety of contemporary AI.

11:56

But I think that when the average person

11:56

thinks about AI and safety together, they

12:01

often conjure up images of a dystopic

12:01

techno future, drawn from maybe their

12:07

favorite sci fi movie, where an algorithm

12:07

or a machine sort of malfunctions, runs

12:13

amok, and then, subjects us to its will.

12:18

And I think that most contemporary

12:18

discussions of people who work in this

12:23

field don't really take that form, right?

12:25

It's not about those scenarios. So can you tell us just by way of

12:27

beginning what people working on AI

12:32

and safety today are actually concerned

12:32

about when it comes to modern AI?

12:38

There are multiple ways of

12:38

approaching questions around AI and

12:41

safety outside of what the community

12:41

that calls itself AI safety works on.

12:45

So for many years before such a

12:45

community existed and used that

12:49

name for itself, you had people

12:49

working on issues like bias, right?

12:52

If you use a hiring algorithm in

12:52

your software that's screening job

12:57

candidates and it's trained on CVS that

12:57

are mostly white men who went to Ivy

13:02

League schools, what will that mean for

13:02

candidates who don't fit that description?

13:05

Bias, discrimination, false negatives

13:05

and positives and associating

13:09

certain characteristics with people.

13:12

The kind of classic real world examples

13:12

are face recognition systems that

13:16

are used for policing that recognize

13:16

the wrong person and can lead to

13:20

the wrongful arrest of that person. So those are the issues that had

13:22

been percolating and people have

13:25

been talking about for many years. But when it comes to these kinds

13:27

of more dystopic examples that you

13:31

brought up, that is very specifically

13:31

related to a community of people who

13:35

have come together around a series

13:35

of other interlinked movements.

13:38

So effective altruism, long

13:38

termism, existential risk, this

13:42

kind of group of people had been

13:42

thinking about how to promote human

13:46

flourishing centuries from today.

13:49

And they try to think on that

13:49

kind of long term horizon.

13:52

And thinking about that, they also had

13:52

to contemplate the possibilities of risks

13:56

to that flourishing not being possible.

13:59

And they've created a lot of hypothetical

13:59

scenarios about the possibility

14:02

that we could end up with artificial

14:02

intelligence that is smarter than humans.

14:06

And again, they would call that

14:06

artificial general intelligence or AGI.

14:10

And there's competing definitions

14:10

about what that entails.

14:14

when it comes from a company like

14:14

OpenAI, Sam Altman will repeatedly

14:17

say AGI is artificial intelligence

14:17

systems that outperform humans at

14:22

every economically valuable task.

14:24

A lot of the recent hype, a lot

14:24

of the recent headlines people are

14:28

seeing and parroting back about the

14:28

possibility that AI could kill us

14:31

all, it comes out of that community

14:31

and a lot of these speculative fears.

14:36

In the moment ChatGPT blew up and people

14:36

were really astounded by its performance

14:40

and the text outputs being in some cases

14:40

indistinguishable from something a human

14:45

being might write, you had a lot of

14:45

people who were on the fringes of those

14:49

communities and were skeptical suddenly

14:49

becoming very concerned about AI's risk.

14:54

But that is the main thing that happened, right? It's the ChatGPT was widely

14:56

released to the public and people

14:59

got to see its performance. In my opinion, that is a

15:01

marketing stunt, right?

15:03

What better way to get people to want to

15:03

use your product than to let them use it.

15:07

And also to get tons of free

15:07

feedback on the ways they're going

15:10

to try to break it and misuse it. So a lot of what my research, when I

15:12

was at Princeton working with four of

15:17

my colleagues, looking at how these

15:17

ideas came, to spread as quickly as they

15:22

did, what are their epistemic claims?

15:24

what does it mean to produce

15:24

knowledge on something so speculative?

15:28

And some of the issues that we saw

15:28

coming out of the communities of people

15:32

interested in this, they talk about

15:32

a thing called the alignment problem.

15:35

So they say we can live in a

15:35

utopic future with AGI if we

15:40

can align AGI with human values.

15:43

And that is a whole process. It's eliciting what those human values

15:44

are, figuring out how to encode them

15:48

into AI systems, testing them to

15:48

ensure that they would be aligned

15:52

under a variety of circumstances. And so the fear of the kind of scenarios

15:54

of AI posing a threat to the future of

16:00

human existence or an existential or X

16:00

risk comes from misalignment and there

16:05

are tons of people working on alignment. We did a whole study with people asking

16:07

what it even means to work on alignment.

16:11

if you really think this is the way, what

16:11

are you doing in your work every day and

16:14

how do you know you're getting there? And this is, so I hope this

16:16

unpacks a little bit of like,

16:19

why is it that policymakers are

16:19

suddenly thinking about this?

16:22

Why is it that this is in the news so

16:22

often, our research showed that this

16:27

community around these ideas, it's not.

16:30

It wasn't that big when we started

16:30

doing this research in 2022, but it's

16:33

very well funded through a series

16:33

of philanthropies and also tech

16:37

billionaires who are individual donors. There's a lot of voluntary

16:39

labor that goes into this.

16:42

There are people producing research

16:42

that they'll put up in web forums.

16:46

It's not peer reviewed, but there's

16:46

such an urgency around these ideas

16:50

within the community that people are

16:50

riffing off of each other and building

16:53

off of each other's work very quickly.

16:55

Yeah, and it's so interesting to

16:55

hear you talk about it because it's so

16:59

obvious that there's an intersection here

16:59

between technology and human values at

17:05

which philosophy is at the center and

17:05

it's funny, because on the one hand,

17:13

I feel like a lot of people in today's

17:13

world think that philosophy is this

17:16

sort of ivory tower pursuit or this like

17:16

useless, symptom of the decline of the

17:22

humanities, philosophy maybe used to

17:22

have its place, but it doesn't anymore.

17:26

And it's clear when you look into AI

17:26

safety and AI ethics that a lot of the

17:29

people who are developing some of the big

17:29

picture ideas around this are philosophers

17:34

and also some of the ideas that non

17:34

philosophers are building on, who are,

17:39

people who are important in this community

17:39

are coming from philosophy, especially

17:43

from utilitarianism in the 19th century.

17:46

And so I'm curious what you think

17:46

about this, because even though as a

17:50

philosopher, it's exciting to me to

17:50

think about there being an important

17:55

role that philosophy is playing in

17:55

the cutting edge of human technology.

17:59

It's also, I think, clear from

17:59

your research that this, is

18:03

being done in pretty problematic

18:03

ways in certain respects.

18:07

And so I think this is also a

18:07

question about who is in the

18:11

AI safety community, right?

18:13

They're coming from all these

18:13

different walks of life,

18:16

thinking about the intersection

18:16

between AGI ethics and policy.

18:21

Who are the stakeholders here and how

18:21

do you see philosophy playing into this?

18:26

Sure. I think I would break the stakeholders

18:27

down along disciplinary lines,

18:30

institutional lines, and then who

18:30

gets recruited into the community.

18:33

So along disciplinary lines, when we were

18:33

looking at who does AI safety work, right?

18:38

We started by looking at Sam

18:38

Bankman Fried's FTX future fund.

18:42

So Sam Bankman Fried is the now

18:42

disgraced former crypto billionaire.

18:45

He was a big effective altruist who

18:45

really seemed to believe in kind of

18:50

the values of the community and to back

18:50

up and explain what that is, right?

18:54

Effective altruism was a kind of

18:54

movement founded by graduate students

18:58

in philosophy from Oxford and Cambridge.

19:01

They were really interested in

19:01

applying utilitarianism and thinking

19:03

about maximizing human welfare,

19:03

not only in the current day, but

19:07

really in that far term future. And they came up with ideas

19:09

like earning to give, right?

19:11

So getting the highest paid job you can

19:11

and donating 10 percent and gradually

19:15

more of your income to charities that

19:15

they thought were related to cause areas.

19:20

So produce that vision of a future utopia.

19:23

And as I was mentioning, that gets tied

19:23

into them thinking about that long term

19:27

future thinking about threats to it

19:27

around the same time you had another

19:32

Oxford philosopher Nick Bostrom writing

19:32

books like super intelligence and kind

19:36

of Thinking through thought experiments

19:36

about that future that would involve,

19:42

either having quote unquote, super

19:42

intelligent, artificial intelligence

19:45

or artificial general intelligence. He has eugenicist thought experiments

19:46

in there about, what it would look

19:51

like if we selected out certain

19:51

fetuses in human reproduction and

19:56

try to reproduce for specific traits.

19:58

I bring this up to save the disciplinary

19:58

background of people who become

20:01

attracted to those things we noticed

20:01

were people who had studied computer

20:05

science and engineering, statistics,

20:05

math, some philosophy and physics, right?

20:10

There were like a few disciplines

20:10

that quite a few people came from.

20:14

They would start reading some

20:14

of the blogs in this space that

20:16

engage with some of these issues. They'd maybe read super intelligence

20:18

or read MacAskill's well, What We Owe

20:22

The Future is his newer book, but he

20:22

had older books on effective altruism.

20:25

They read Toby Ord's The Precipice

20:25

and this became the set of like books.

20:29

The canon for this kind

20:29

of growing community.

20:32

And then in terms of institutional

20:32

backgrounds, you have a lot of

20:35

people working in tech companies. I did graduate school at Berkeley in

20:37

the Bay Area, and I had been brushing

20:40

shoulders with these people for years,

20:40

but they had always been on the edges of

20:44

some of the conversations in academia.

20:47

You have certainly academic computer

20:47

scientists, but it's not mainstream.

20:51

Even now, there are, there's a whole

20:51

infrastructure in this, space that's been

20:56

making me think a lot about how one of

20:56

the things I really like about working

21:00

on tech from an interdisciplinary and

21:00

like justice oriented perspective is that

21:05

there was this paper a few years ago that

21:05

talked about social roles for computing.

21:09

And one of them was computing as

21:09

synecdoche or the idea that looking

21:13

at computational systems that have

21:13

social effects can make you reflect

21:17

on what is structurally, unacceptable

21:17

about those institutions, right?

21:22

So if you think about like

21:22

Virginia Eubanks book, Automating

21:25

Inequality, and she comes up with

21:25

this idea of the digital poorhouse.

21:28

She's talking about how algorithmic

21:28

systems in welfare distribution, in

21:34

determining, whether or not to allocate

21:34

housing to an unhoused person, that

21:38

these create a kind of digital poorhouse. And the point of a book like that

21:39

is to show that there are all these

21:42

social structural forces that get

21:42

baked into algorithms that create

21:45

a digital poorhouse effect, right? Not just that the technology

21:47

itself is doing it.

21:50

So computing as synecdoche has

21:50

been really interesting to apply

21:53

to looking at the institutions

21:53

popping up around AI safety because

21:56

they have their own philanthropies. They are creating their own non

21:58

profits, their own think tanks, their

22:01

own polling organizations that create

22:01

surveys and thus statistics around

22:06

what percent of the population believes

22:06

that AI is going to kill us all.

22:09

And then, pumping those things out to

22:09

the media and to policymakers in the same

22:14

way that many of these institutions that

22:14

have already existed but did not have

22:17

this worldview have been doing forever. And so a lot of the times people will

22:19

ask me, do you feel like this is a

22:23

distortion of those institutions? And I'm like, no, it's like

22:24

computing a synecdoche.

22:26

They're using all of those institutions.

22:29

The way they have been designed to be

22:29

used, there's a deep irony in AI history.

22:35

there's a recent book called How Data

22:35

Happened by Matt Jones and Chris Wiggins

22:38

where they have only one chapter on AI,

22:38

which I appreciate because AI is not

22:42

really the whole history of technology. And they talk about how AI was actually

22:44

basically like a marketing term that

22:48

the mathematician John McCarthy credits

22:48

himself as coming up with when he wanted

22:52

funding from the Rockefeller Foundation.

22:55

So they say that from its inception,

22:55

AI was a hype term to create a field

23:00

and distinguish it from other fields,

23:00

but it was really very much steeped

23:04

in, other things that were happening,

23:04

like symbolic approaches to creating

23:08

what we're now calling artificial

23:08

intelligence, and I find that history

23:12

so relevant when you look at how

23:12

that kind of Effective Altruism, AI

23:16

safety intersection, it wouldn't have

23:16

catapulted into public attention if

23:20

there weren't hundreds of millions

23:20

of dollars in these philanthropies

23:23

that this space has created. They have spun up National Science

23:24

Foundation grant money, like 20 million

23:28

grant pools to fund this work and to pump

23:28

it through more traditional institutions

23:34

that are slower and not really equipped

23:34

for what happens when something that many

23:39

researchers contest as pseudo scientific

23:39

and fear mongering kind of work takes up

23:43

residence and in terms of students asking

23:43

for courses on things like AI alignment

23:48

and forming campus groups, tech companies

23:48

potentially building teams around this,

23:52

tech companies reorienting themselves so

23:52

that suddenly everyone's working on large

23:56

language models when they could have been

23:56

working on all sorts of other things.

24:00

As I've been doing this work, I've been

24:00

paying attention to what does it say about

24:03

how fragile all of these institutions were

24:03

that they so quickly fell for this hype.

24:07

Yeah. And the infusion of cash into

24:08

this discussion is so real.

24:11

just in the last several years, I've

24:11

noticed as a philosopher and as somebody

24:15

who works in a humanities space.

24:17

And as a San Francisco resident...

24:19

yeah. Yeah. In San Francisco, I that

24:19

all over the place, right?

24:23

this venture capital interest in AI, and

24:23

also the kind of braggadocious attitude

24:29

that you get from people who are not

24:29

experts in philosophy to just appropriate

24:33

philosophical ideas and throw them

24:33

because it gives them a certain kind

24:36

of cultural and technological capital

24:36

that they are very happy to cash in on

24:41

the moment they get the opportunity. But the point here is that, yeah,

24:43

there's a lot of money rushing in into

24:46

this intersection of ethics and AI.

24:49

And a lot of this seems to involve,

24:49

in particular, the effective

24:54

altruism group, which is one

24:54

of the kind of sub communities.

24:59

that belong to this larger umbrella

24:59

of the AI safety community.

25:04

And so I want to talk about them for

25:04

a little bit because the effective

25:08

altruism group, of course, a lot of

25:08

their thinking is based on William

25:13

McCaskill's writings and publications.

25:16

And he himself was deeply influenced

25:16

by Peter Singer's utilitarianism.utop

25:20

And I want to get your

25:20

take on this community.

25:23

So tell us just in a few words

25:23

what effective altruism is and

25:27

then how you interpret the work

25:27

that it's doing in this space.

25:32

I would say there are two

25:32

things of late that have influenced

25:35

how I think about effective altruism. One is an article Amia Srinivasan wrote

25:37

in 2015 called Stop the Robot Apocalypse.

25:42

And, she was questioning, what

25:42

is the value of utilitarianism to

25:46

practical things like policymaking? What is effective altruism?

25:50

As of that date, what was it really doing?

25:52

And I think the conclusion she settled

25:52

on was that it made people feel

25:55

like they were contributing to some

25:55

kind of radical change when really

25:58

they were maintaining a status quo. And I would argue, yeah, that

26:01

was written eight years ago.

26:05

And what has really changed other than

26:05

in the last year or two when so much

26:08

money has been poured into this, it's

26:08

really changed conversations and some

26:12

of the things people are working on. But when you work on something like

26:13

AI safety and it's super loosely

26:18

defined from that community. It's very hard to point to specific

26:20

wins and, benchmarks or measurable

26:25

signs that you've made systems safer. And that is some of the critique that

26:27

I see coming from, really critical

26:31

colleagues in computer science is saying,

26:31

these are such narrow definitions.

26:34

We're not rather, these are such kind of.

26:36

vague terms and narrow definitions that

26:36

something like systems engineering,

26:43

the kinds of things you would do to

26:43

make sure that like a plane is safe.

26:45

You can do that for certain types of

26:45

AI systems, if you look at them in

26:49

their context, but a lot of this is

26:49

so decontextualized and so abstract.

26:53

So that's one thing. The other, when I think about effective

26:54

altruism, there's this really wonderful

26:58

graduate student in, Europe, Mollie

26:58

Gleiberman, who's been writing about the

27:02

difference between public EA and core EA.

27:06

And she's arguing that a lot of

27:06

the areas more related to maybe

27:10

Peter Singer's work, like issue

27:10

areas like animal welfare, right?

27:13

Effective altruists care

27:13

about pandemic prevention.

27:16

They've talked about preventing nuclear

27:16

destruction, that some of these ones

27:20

that seem reasonable to an outside

27:20

observer, more palatable, that is public

27:25

facing EA, but core EA, which is these

27:25

ideas that are more contestable, right?

27:31

The possibility that AI could kill us all. Some of these kind of more transhumanist

27:33

problematic beliefs that is at this

27:38

core and is kind of Trojan horsed

27:38

in by this other set of ideas.

27:42

I really recommend her work. I'm still digging into it.

27:44

And she makes comparisons to

27:44

the structure of this community

27:47

and the federalist society. that there is something so inherently

27:49

conservative about this while

27:53

presenting itself as radical. And I would argue that is of a piece

27:54

with how ideology becomes material

27:58

in Silicon Valley in general. this one has a twist, right?

28:01

Because there's this sort of Bay

28:01

Area meets Oxbridge tie, like those

28:06

are the two kind of core places

28:06

that are married in this movement.

28:10

But, a book I really love actually,

28:10

and I'd read in 20 20 or 2021.

28:17

And I thought, gosh, I wish

28:17

somebody would write this about, E.

28:19

A. And A. I. But there's probably not enough happening.

28:21

And then lo and behold, it's a few years

28:21

later, and I'm doing it is Adrian Daub.

28:26

What Tech Calls Thinking is such

28:26

a great little book looking at

28:30

just a few major figures, right?

28:32

Mark Zuckerberg of Meta, Peter Thiel of,

28:32

you name it, PayPal, Palantir, various

28:38

investments in other companies, and

28:38

trying to understand like, yeah, what

28:42

are the ideological underpinnings of

28:42

the things they have produced, right?

28:45

With Zuckerberg, he went to college

28:45

for a little bit at Stanford.

28:48

What, from the courses that he

28:48

was very likely to have taken

28:51

there because they were required,

28:51

are features we see in Facebook?

28:56

With Thiel, who's a big Ayn Rand

28:56

head and makes it known even now,

29:01

Oh, I didn't know that!

29:03

Yeah. Yeah. Ellie, I'll get you this book.

29:07

Thanks.

29:08

yeah, I think that idea of it's

29:08

really just maintains the status quo

29:13

and there's a public facing in a core

29:13

like that has been fueling a lot of how

29:17

I think about this at the same time. One of my questions about this is to

29:19

what extent when people enroll in these

29:24

communities, do they realize that? That there is that distinction.

29:27

To what extent do they

29:27

believe in the causes?

29:29

Because most of the people I've

29:29

interviewed are very earnest, and not

29:32

all of them are, identify as effective

29:32

altruists or believe in long termism.

29:36

There's different reasons and motivations

29:36

as this community spreads that people

29:40

who are not part of the kind of earlier

29:40

waves would come to it for, But something

29:45

that's really interesting that I noticed

29:45

is even as I learned about effective

29:48

altruism in 2019 from people I knew in

29:48

the Bay Area who were in China when I was

29:52

doing field work there, and they would

29:52

even then say while wearing EA swag, and

29:58

this actually comes up in like a recent

29:58

New Yorker article, people will wear

30:01

EA swag while saying, Oh, but I don't

30:01

really identify as an effective altruist.

30:05

I just like the community. And so this has stuck in the back

30:06

of my mind while I do the research

30:10

where it seems like some of the

30:10

biggest things people are looking

30:12

for are communities where they can

30:12

talk about these things without being

30:15

stigmatized and being taken seriously.

30:17

At the very beginning of doing this work,

30:17

we looked at something like 80, 000 hours,

30:22

which is a big EA career hub website.

30:26

It has. It's really thorough job boards that

30:26

you can scroll through based on what

30:30

EA cause area you're working on. So they have tons of jobs in

30:32

AI that they say these can

30:34

contribute to preventing AI X risk.

30:37

Go work for, OpenAI and thropic, but

30:37

even Microsoft and the UK government,

30:42

like suddenly the range of where you

30:42

can do that has expanded and they

30:45

have a framework of importantness,

30:45

tractability and neglectedness.

30:49

And so they were saying that

30:49

at the beginning, AI safety

30:52

is a marginalized issue. Everybody's working on ethics,

30:54

but nobody's working on

30:57

safety from existential risk. It's so important and it's

30:59

tractable, but it's so neglected.

31:03

And so I now point out that there's been

31:03

a flip where it's not neglected anymore.

31:06

And there's so much money going into this. What is this community actually

31:08

going to do with its power?

31:12

And I want to dive into the

31:12

ideas behind this a little bit more as

31:16

well, because as terrifying as it is to

31:16

hear that Peter Thiel is all into Ayn

31:22

Rand, who I think most people within

31:22

philosophical communities, if we consider

31:27

her a philosopher at all, consider her a

31:27

pretty straightforwardly bad philosopher.

31:31

It's just basically capitalist

31:31

ideology cloaked in,

31:35

philosophical sounding language. So if there's that side that's

31:37

pseudo philosophy or just like pretty

31:41

obviously bad philosophy, there's

31:41

also a lot of effective altruism.

31:44

This is probably the dominant movement,

31:44

as we alluded to before, that is grounded

31:48

in a much more, let's say, academically

31:48

respectable tradition, which is the

31:54

philosophical tradition of utilitarianism.

31:57

And one of the examples that comes to mind

31:57

for me here is that effective altruism

32:03

argued that it's better to provide

32:03

deworming medication in Kenya than it is

32:08

to provide educational funding because,

32:08

you can only help better educational

32:14

outcomes in Kenya if people have access

32:14

to deworming medication because, dying

32:20

due to parasites is a huge problem there.

32:23

So you have that, which is like a

32:23

utilitarian calculus that seems benign

32:27

to, like pretty straightforwardly helpful,

32:27

I think, from various vantage points.

32:32

But then more recently with this

32:32

rise of interest in AI, you get

32:35

I think far more controversial

32:35

utilitarian calcul calculacies?

32:40

Calculi? Anybody? Either of you? Or my more friends?

32:44

Calculations.

32:45

Okay, thank you. Thank you. Calculations. which is better than providing funding for

32:47

deworming in Kenya, let alone education.

32:53

People should be providing funding

32:53

for AI because AI is really what

32:58

poses an existential risk to humanity

32:58

altogether, not just to people in Kenya.

33:02

And so what you get, you mentioned

33:02

long termism before, what you get

33:05

is this idea that actually so many

33:05

human resources and so many people.

33:10

so much funding should be going into AI

33:10

at the expense of real world struggles

33:15

that people are dealing with now. And so I'm really curious

33:17

what you think about that.

33:19

Are there benefits to this or is it

33:19

just like a toxic ideology in your view?

33:24

Are like, should we really be

33:24

worried about the existential risks?

33:27

And even if it's in a different way from

33:27

the way that they're thinking of it, or

33:30

are they like getting something right? Yeah.

33:32

What do you think?

33:34

I had a really thoughtful friend

33:34

who has moved between the worlds of

33:38

people who are into this and skeptical

33:38

of it say, I do think things can get

33:42

really weird really soon, and that I

33:42

don't want to discard everything this

33:47

community is working on and I agree

33:47

because what this community has done is

33:53

they're not really challenging capitalism

33:53

or the marketing that leads people

33:57

to adopt AI systems at scale, right? They're accepting that's going to keep

33:59

happening because companies are very

34:01

good at that and people are bought in. And then they're imagining a future

34:04

where that just keeps happening.

34:06

But they've taken a bunch

34:06

of solutions off the table.

34:09

what are things we don't

34:09

have to use AI systems for?

34:12

What are you know, just in regulating

34:12

these technologies, which again, the

34:16

community is very divided on, and

34:16

they're never thinking about regulations.

34:20

People have been asking for years

34:20

around hate speech or copyright or,

34:25

these kinds of like bread and butter

34:25

issues they would see as too focused

34:28

on the present day and not deserving of

34:28

the urgency and the cash infusion and

34:32

the lobbying efforts as these kind of

34:32

more long term speculative ones are.

34:37

But I think even early in this project,

34:37

my colleagues and I were noticing

34:41

that there's short term technical

34:41

fixes coming out of some of the

34:46

work that this community is doing. But I think that they've

34:49

had to create a kind of self

34:51

aggrandizing hero narrative, right? It's, we've been able to watermark

34:53

GPT outputs so that it, somebody

34:58

can see that something was written

34:58

with ChatGPT and not by a human.

35:02

That's valuable.

35:04

Okay, this is good to know

35:04

for me since I'm on sabbatical,

35:06

not grading papers right now, but I

35:06

will be back in the classroom soon.

35:10

Yeah, I, don't use GPT

35:10

for a variety of reasons, and I am

35:15

curious about if they figured out the

35:15

watermarking or not, but I, something

35:20

like that is a bounded technical

35:20

fix that makes a lot of sense.

35:23

Will it or will it not prevent some

35:23

of the worst case scenarios that

35:27

this community is coming up with? I'm not sure, but I wonder if doing

35:28

that kind of really bounded ordinary

35:33

work and it's for a product by a company

35:33

needs to come with this narrative.

35:38

there's really a lot of questioning of

35:38

how will history remember us and the

35:41

idea that history will remember any of

35:41

the specific people working on this.

35:45

So I just, I've recently really

35:45

thought about how much of this

35:48

is both overcompensated labor

35:48

for what it actually produces and

35:54

how much it helps other people. And not very compensated and voluntary

35:56

sometimes labor that is part of this

36:01

narrative of like you're, going to

36:01

be the reason that we get utopia.

36:06

And I don't know for how long

36:06

people are going to be willing

36:09

to suspend that kind of belief.

36:13

So that's one piece of it that

36:13

I've been thinking about, in

36:16

terms of is this valuable or not. And I asked people who have access to it.

36:20

Some of the bigger names in this field

36:20

and to interview them publicly, ask them

36:24

what we stand to lose if they're wrong. They never have to talk about that.

36:28

They never have to talk about

36:28

maybe this was a huge waste of

36:30

money, resources, time, careers.

36:33

some journalism that was partially

36:33

inspired by our work coming out

36:37

of the Washington Post was looking

36:37

at how the rise of AI safety on

36:40

campus is basically how our college

36:40

students creating groups around this.

36:44

Kind of tapping themselves into this

36:44

network I described of philanthropies,

36:48

fellowships, boot camps, ways of being

36:48

in the network of people working on this,

36:52

and then getting to work in the companies

36:52

and telling yourself you're working

36:55

on these issues and, maybe making 400,

36:55

000 working at Anthropic on alignment.

37:00

And,

37:01

All right, guys, this

37:01

is my last day at Overthink.

37:04

I'm going to do that

37:04

next, I'm just kidding.

37:08

Now you understand how to do the

37:08

culture thing, if you read our research.

37:13

but if, there are some people whose

37:13

timelines of either we get it wrong and

37:18

everything's a disaster, we get it right.

37:20

And we live in utopia for people

37:20

who have shorter timelines,

37:22

like three to five years. I'm really curious about what they're

37:24

going to be saying in three to five years.

37:27

Will there be some new excuse for

37:27

why this didn't all shake out?

37:30

I also find it something that

37:30

this all makes me reflect on is

37:33

this community of people is not. They're just not as interdisciplinary

37:35

as I think they've been made out to be.

37:38

They're very limited ways of knowing and

37:38

selective cherry picking of things from

37:43

philosophy, from geopolitics, right?

37:45

Like for me in particular, most of my

37:45

work was based in China, and a lot of

37:49

my work is centered around what I'm

37:49

calling technological jingoism, or the

37:52

over securitization of issues related to

37:52

technology and society and the economy.

37:58

In the moment we're in right now with the

37:58

TikTok ban and having seen this coming

38:02

down the line for a very long time,

38:02

a lot of my research looks at how the

38:06

US government in particular is just so

38:06

afraid of the United States losing its

38:10

economic and geopolitical dominance over

38:10

the world that casting China or any other

38:16

adversary out to be this existential

38:16

threat to us and that supremacy becomes

38:22

an excuse for all kinds of things like

38:22

banning apps and making it out to seem

38:27

like all technology coming out of China is

38:27

surveilling you and part of a grand scheme

38:31

plan that the Chinese government has. Many of, these are also speculative

38:34

and not provable, and you're

38:37

supposed to just believe that it's

38:37

in the best interest of everyone.

38:41

I think with the TikTok ban, creators

38:41

are really clapping back and watching

38:45

like congressional hearings and coming

38:45

up with fairly like sharp critiques.

38:50

But it takes me back to AI safety

38:50

when I think, they're fully

38:54

bought into the jingoism as well. they will say things like, if we get

38:56

AGI and it leads to bad outcomes, at

39:00

least it would be better if it came out

39:00

of the United States than China, which

39:03

would just turn all of us authoritarian. And they've completely ignored

39:05

the authoritarian tendencies

39:08

within democracies ours.

39:10

I think this speaks to one of

39:10

the central dreams of the people in

39:15

the AI community, which is the dream

39:15

of taking their engineering technical

39:20

skills onto a social playing field

39:20

where they're no longer playing with

39:24

the engineering of systems and machines,

39:24

but they're actually tinkering with

39:28

human bonds and with time itself, right?

39:31

They're engineering. Yeah. Yeah. the future. So there is clearly a social mission here

39:33

that is essential for them to maintain

39:38

the goodness of their motivations and

39:38

their intentions in entering this terrain.

39:43

And so I want to ask you a question here

39:43

that picks up on this discussion that you

39:47

just started about the relationship to the

39:47

state and to government, because the AI

39:52

safety community, I would say as a whole,

39:52

and correct me here if I'm mistaken,

39:57

sees itself primarily as a social

39:57

watchdog, as this canary in the mine

40:02

that's keeping society from destroying

40:02

itself by blowing the whistle on this

40:06

doom that is coming down the hatch.

40:08

And by adopting this role, their hope

40:08

is to influence law and policy, right?

40:14

They want to determine the content of law.

40:17

And I want to know what some of

40:17

the more concrete legal proposals

40:23

and recommendations that grow

40:23

out of this community are.

40:27

What are they proposing for managing risk?

40:31

What role do they think that governments

40:31

should play in the regulation of private

40:36

industry, for example, in order to

40:36

ensure an AI safe future, or is it

40:42

that the people in this community are

40:42

just advocating self regulation on

40:46

the part of industry left and right?

40:48

So it's a bit of a mixed bag. You have people who are very afraid

40:49

that there will be malicious actors

40:53

who will use AI to build bioweapons.

40:56

And as my mentor here at UCLA, Safiya

40:56

Noble, and computer scientist Arvind

41:02

Narayanan and Sayash Kapoor who

41:02

write all about AI snake oil have all

41:05

commented the instructions for how

41:05

to build a bioweapon have been on the

41:08

internet since there was an internet. But what you'd really need is

41:10

like the resources and, you

41:14

need a lab, you'd need reagents. You'd actually have to build it.

41:16

You can't just tell a ChatGPT to

41:16

do it and it'll do it for you.

41:19

And so what is AI safety advocates

41:19

of basically preventing the creation

41:27

of bioweapons using AI, talking to

41:27

policymakers and saying things like,

41:30

we're afraid that there will be individual

41:30

people who will buy up enormous amounts

41:34

of compute resources that could be

41:34

used to build something like that.

41:37

There was a recent AI executive order

41:37

talking about using the defense production

41:42

act as like a way of regulating this and

41:42

making sure it doesn't happen, treating it

41:45

like you would the creation or acquisition

41:45

of other kinds of weapons that has taken

41:51

a lot of energy away from other kind of

41:51

more practical prosaic ways of regulating

41:56

AI in everyday life, because, as we

41:56

know, policymakers, time is limited.

41:59

The amount of work it takes to

41:59

get something passed into whether

42:02

an executive order or just any

42:02

other policy document is finite.

42:07

You have a subset of people

42:07

in the AI safety world.

42:11

Sometimes externally, people

42:11

would call them doomers, right?

42:14

The ones who are certain that this

42:14

will only lead to horrible outcomes

42:18

and are saying, let's just stop it

42:18

all until we have proper safeguards.

42:21

That, the thing I like to point

42:21

out about that community is it's

42:25

not particularly well funded. And as a critic like Timnit Gebru

42:26

of the Distributed AI Research

42:30

Institute, would say, They're also

42:30

fomenting hype in saying these systems

42:36

are so powerful, we must stop them.

42:38

Then you're contributing to what

42:38

computer scientist Deb Raji would

42:42

call the fallacy of functionality. And Deb has this amazing paper

42:43

with a bunch of colleagues around

42:46

typologizing the different ways and

42:46

points at which there's an assumed

42:51

functionality that is not always there.

42:53

It can come in, the design and the

42:53

marketing and the deployment, interacting

42:57

with the system that you're told does

42:57

something it doesn't actually do.

42:59

And that's just not something

42:59

this community engages with.

43:03

And that, honestly, I think a lot

43:03

of policymakers don't have the

43:06

time and resources to work on.

43:09

I've talked to civil society advocates

43:09

who are deeply skeptical of AI

43:11

safety, and they feel like AI safety

43:11

proponents have a monopoly on the

43:16

future and how people think about the

43:16

future because they did such a good

43:19

job of selling an idea of utopia.

43:21

That anybody sounding alarms

43:21

for caution, they haven't done a

43:25

particular job of painting a future. Anyone would want to live in because

43:26

they're so busy talking about the

43:29

things that are going wrong now. And then also self regulation

43:31

is something the companies

43:34

are absolutely advocating for. They really, they'll say, let's just

43:36

do things like red teaming or paying.

43:41

Often again, data workers, but

43:41

sometimes people within tech

43:44

companies to come up with, for

43:44

example, malicious prompts for GPT.

43:48

And then let's figure out how to

43:48

design an automated system that

43:52

would take in human feedback and

43:52

then be able to stop those prompts

43:55

from being addressed in the future. So some of the classic examples

43:56

are, Hey GPT, how do I burn my

43:59

house down and get, arson money?

44:01

And it's instead of telling you exactly

44:01

how to do that, it would automatically

44:05

know from feedback, I shouldn't tell you

44:05

how to do that and you shouldn't do that.

44:08

Okay. But of course, at the same time,

44:09

there are open source language models

44:13

that have not taken any of that into

44:13

account and are down and dirty, happy

44:17

to tell you about how to, conduct

44:17

ethnic cleansing, commit crimes.

44:24

So there's this suddenly like wide

44:24

landscape of options where like OpenAI

44:27

is the relatively safe one and something

44:27

produced by like Mistral is not.

44:32

And as we approach the end of our

44:32

time, I'm really curious where you stand

44:36

on all of this, because obviously your

44:36

research is on AI safety, which means

44:41

that you're studying the communities who

44:41

look at AI safety rather than, trying

44:46

yourself to figure out what AI is.

44:48

the future of AI safety looks

44:48

like, but of course, I imagine

44:52

that's given you some thoughts too.

44:55

Are you a doomer when it comes to AI?

44:57

Are you whatever the

44:57

opposite of a doomer is?

44:59

Are you somewhere in between? And what do you think is, what do you

45:01

think is the most promising feature of

45:05

what we may or may not correctly call AI? And what do you think is, the

45:08

most threatening possibility?

45:11

I don't think any of the pre

45:11

existing labels around any of this fit

45:15

where I put myself because something like

45:15

the concept of AI ethics being treated

45:20

as if it's in opposition to AI safety has

45:20

never made sense to me because people who

45:24

work on a wide range of like social and

45:24

labor and political issues involving AI

45:29

don't necessarily characterize themselves

45:29

as working on AI ethics, myself included.

45:33

I'm really interested in following

45:33

in the tradition of anthropologists,

45:36

like Diana Forsythe, who were studying

45:36

AI expert systems of the 1980s and

45:41

nineties and how, what did people

45:41

think they were doing when they did AI?

45:45

that was her question. And that's mine around this work.

45:49

What do I think is most promising? I think right now I still come back

45:50

to computing as synecdoche and making

45:54

just any of these technologies telling

45:54

us something else about the human

45:58

experience and what in our institutions

45:58

can be changed outside of technology.

46:03

Some of my favorite organizations

46:03

and individuals I've worked with

46:06

have often talked about how the

46:06

technology piece of what we're

46:08

working on is such a small part of it. And it's all of these other social

46:10

factors that are much more important

46:14

that we foreground in our work. And what would I say is

46:16

perhaps the most dangerous?

46:20

I think that we've been through so many

46:20

phases of saying, if you just taught

46:23

computer scientists ethics, they'd

46:23

somehow be empathetic to these causes.

46:28

I don't know that's enough. And that what that assumes about

46:30

a computer scientist as a person

46:33

and where they're coming from

46:33

seems to be misguided as well.

46:38

one of the critiques coming from within

46:38

this community or from outside of

46:41

this community about this community

46:41

from people like to Timnit Gebru a

46:44

brew and Emile Torres is that AGI

46:44

is like a eugenicist project, right?

46:47

And there's so much good writing

46:47

about technoableism or the idea

46:50

that technology should move us all

46:50

away from having bodies that are not

46:54

normatively abled and fit these like

46:54

ideals of an able bodied person that is

47:00

constantly improving themselves and that

47:00

disability has no place in the world.

47:04

Like this work is very

47:04

techno ableist, right?

47:07

Even just the concept of intelligence

47:07

and not interrogating it further and

47:12

you don't even have to grasp that

47:12

far to find the eugenicist arguments

47:16

in superintelligence and that nobody

47:16

in that community is really talking

47:21

about that because it's uncomfortable.

47:24

In arguing that AGI is like a,

47:24

eugenicist project, it's interesting

47:29

to look at then the building of

47:29

the field that I have studied.

47:32

parallels some of how race science

47:32

happened that you had to create journals

47:36

and venues to legitimize it as a science

47:36

when so many people contested it.

47:40

I would love to see AI safety

47:40

properly engaged with that.

47:44

I would love to see some of these ideas

47:44

around intelligence crumble and like what

47:48

alternatives could we have instead, but

47:48

that's not where money and time is going.

47:53

And of course, some of the

47:53

biggest proponents of this in

47:55

the mainstream our edgelords

47:55

like Elon Musk and Mark Andresen.

48:01

And so that really moves us away from

48:01

like any of the questions I just raised.

48:07

Dr. Ahmed, this has been a

48:08

wonderful discussion.

48:10

You have given us a lot to

48:10

chew on and to think about.

48:14

And we thank you for your time and for

48:14

this important work that you're doing.

48:18

Thanks so much.

48:19

Thanks

48:19

much .

48:25

Ellie: Enjoying Overthink? Please consider supporting the

48:26

podcast by joining our Patreon.

48:29

We are an independent,

48:29

self supporting show.

48:32

As a subscriber, you can help us

48:32

cover our key production costs, gain

48:36

access to extended episodes and other

48:36

bonus content, as well as joining our

48:40

community of listeners on Discord. For more, check out Overthink on Patreon.

48:44

com.

48:46

Ellie, that was such

48:46

a great discussion with Dr.

48:48

Ahmed, and I cannot mention that

48:48

this is the first time we do

48:53

an episode where a community of

48:53

experts is the subject matter.

48:58

and I found that just really illuminating,

48:58

but I wanted to mention that after

49:01

we finished the interview, Dr. Ahmed said, Ellie and David, above

49:03

all, what I really want is for your

49:07

listeners to walk away with a sense

49:07

of calm about the subject matter.

49:11

So what are your thoughts

49:11

about this Ellie?

49:13

Yeah. I was really surprised to hear that.

49:15

And to the point that we were

49:15

both like, Oh, we got to mention

49:18

that because it happened right

49:18

after we finished the recording.

49:21

Because I don't know, I feel like as

49:21

we were hearing all about the AI safety

49:27

community, as she was describing it,

49:27

I was coming away with the sense that

49:33

there must be major risks because

49:33

everybody is talking about them, even

49:37

though one of her key points was this

49:37

might not be worth funding in the

49:42

way that we're funding it, right? So I do think that's just something

49:44

to rest on, this idea that, maybe

49:49

the AI risks are not as extreme

49:49

as we think, and there is at least

49:53

something overblown about it. Not that we don't take these questions

49:54

seriously, but just One of the things

49:59

that stuck with me, and I'm not going

49:59

to remember the exact term right now

50:02

because we just finished recording, it

50:02

was something like functional fallacy,

50:06

or she, quoted somebody who worked in

50:06

this field who had said quite a number

50:11

of years ago, I think not with respect

50:11

to AI specifically, but with respect

50:14

to technology, that we tend to have

50:14

this fallacy in our thinking where

50:19

we assume that there's a danger of a

50:19

certain technology that technology has

50:24

not yet proven itself to have, right?

50:26

And so whether it's, oh, AI is going

50:26

to put all of the writers out of work

50:33

because now you could have AI write

50:33

a Shakespeare play, partakes of this

50:37

fallacy because AI has not at all proven

50:37

itself to be able to creatively write.

50:42

Yes, it has proven itself to be capable

50:42

of doing certain writing tasks such

50:47

as write copy for an Instagram post,

50:47

but I would say that writing copy for

50:52

an Instagram post doesn't require the

50:52

same kind of creative capabilities

50:56

that writing a Shakespeare play does. And so even if it's not out of the

50:57

realm of possibility that AI could be

51:00

genuinely creative in that sense, it is.

51:03

Not evident yet that it can be,

51:03

or that it's even tending that

51:08

way, given what we know so far. I'm reminded a little bit about

51:10

our AI in the arts episode, or

51:13

other AI episode, on this point.

51:15

And so I think that is useful to

51:15

keep in mind, this idea that AI might

51:19

lead to the extinction of the human

51:19

race, because it can end up, choosing

51:24

its own way, it can have freedom.

51:27

requires having an inkling that AI

51:27

could be capable of having freedom,

51:33

which we just like, it doesn't

51:33

have that functionality right now.

51:36

Yeah, no, and I actually would

51:36

go even further than that, because

51:39

what you're alluding to is the case in

51:39

which we see risks that are not there.

51:44

We just collectively hallucinate

51:44

these dangers and then make

51:48

decisions about policies and about

51:48

funding on the basis of that.

51:52

But I think the other side to that

51:52

coin is that because we're so concerned

51:56

with those long term, unrealistic

51:56

scenarios, we start missing dangers

52:02

that are already materializing, and

52:02

we don't even consider them as risks

52:07

because there's a section in one of Dr.

52:09

Ahmed's papers where she talks about

52:09

the importance of really being concrete

52:14

about what the risks actually are. And I decided to follow the reference,

52:16

and it's to a paper that was published in

52:20

2020 by Inyoluwa Debra Raji and Roel Doby.

52:25

And they talk about how the

52:25

AI safety community overall.

52:30

When they talk about risks,

52:30

they never talk about the risks

52:33

that actually matter to people. So for example, think about the

52:35

dangers of automating taxis.

52:40

That's not really front and

52:40

center in these discussions.

52:43

Usually you talk about more doomsday

52:43

scenarios, or even the dangers

52:48

associated with the production

52:48

of algorithms and machines.

52:52

So one of the examples that they

52:52

mentioned, and this is the only

52:55

one that I'll, list here, but

52:55

there are other ones in that piece.

52:59

They say often in order to train an

52:59

algorithm, you have to hire a lot of

53:03

humans to do the work of literally

53:03

training it on the data, right?

53:07

Teaching it how to classify things

53:07

so that it then learns a rule.

53:11

And right now, that's something

53:11

that we do by outsourcing that very

53:14

mechanical labor onto exploited

53:14

workers, often in the third world.

53:20

So I actually heard Shazeda

53:20

present on a panel at UCLA this

53:23

spring on AI safety and ethics.

53:26

And there was another really prominent

53:26

researcher there whose name I'm

53:30

unfortunately forgetting now, who

53:30

talked about how A lot of the content

53:34

moderation and other sort of really

53:34

unpleasant tasks that form the dark

53:39

side of internet labor have, not

53:39

only been outsourced to the Global

53:45

South, but actually map on perfectly

53:45

to former patterns of colonialism.

53:51

So who is content moderating

53:51

material in France?

53:55

People in North Africa. Who is content moderating

53:57

material in America?

54:00

The Philippines. Who is content moderating

54:02

material in Britain? India, right?

54:05

And I found that to be really interesting

54:05

here because I think it also pertains

54:10

to both a dark side and a potential

54:10

upside of AI, which is that if these

54:15

unpleasant tasks are being outsourced

54:15

to formerly colonized people, such

54:19

that we're still living in this, techno

54:19

colonial society, Could those tasks

54:25

further be outsourced to AI, right?

54:27

And that could be like a good thing. But of course, the downside of that

54:29

is then, okay, what is coming to fill

54:33

the labor vacuum for these people who

54:33

are currently employed by those means?

54:38

I don't have a lot of faith that the

54:38

neo colonialist capitalist regime that

54:43

we live under is suddenly going to

54:43

have like much more meaningful work

54:48

for the populations in those areas.

54:52

so I have a friend in Paris who

54:52

works in the space of content moderation,

54:56

and he told me that one of the problems

54:56

with the incorporation of AI is that even

55:01

though AI can catch a lot of the content

55:01

that needs to be filtered out, because

55:06

companies don't want to risk even one

55:06

sort ofpost or one photo passing their

55:12

safety filter, they still end up having

55:12

to use human, moderators to catch them.

55:19

And here we're talking about, some of

55:19

the nastiest kind of darkest corners

55:23

of the internet that, people from

55:23

these formerly colonized spaces are

55:28

having to deal with on a daily basis.

55:30

And so just the psychological impact

55:30

of having to watch, child porn,

55:35

murders, like brutal beatings on a day

55:35

to day basis for eight hours a day.

55:40

That's why I say it's so unpleasant.

55:42

Yeah, I know, and I

55:42

guess maybe, I'm, I want a

55:46

stronger term for that than that.

55:49

But the point being is that right

55:49

now it's not as if you can just

55:53

automate that completely, so you

55:53

still need that human component.

55:57

And as usual in this episode,

55:57

we're not coming away with a clear

56:00

sense of, you must do X, Y, Z, and

56:00

then we will avoid the threat of human

56:06

extinction on the one hand or further

56:06

exploitation on the other, which I think

56:11

is an important point because that idea

56:11

that we could have that clear X, Y, Z

56:16

is part of the dream of the effective

56:16

altruism movement and of other members

56:22

of the AI safety community that Dr.

56:24

Ahmed has given us really good

56:24

reason to be suspicious of.

56:30

We hope you enjoyed today's episode. Please rate and review us on Apple

56:31

Podcasts, Spotify, or wherever

56:36

you listen to your podcasts. Consider supporting us on Patreon

56:37

for exclusive access to bonus

56:41

content, live Q& As, and more. And thanks to those of you who already do.

56:45

To reach out to us and find episode

56:45

info, go to overthinkpodcast.

56:49

com and connect with us on Twitter and

56:49

Instagram at overthink underscore pod.

56:54

We'd like to thank our audio editor,

56:54

Aaron Morgan, our production assistant,

56:58

Emilio Esquivel Marquez, and Samuel P.

57:00

K. Smith for the original music. And to our listeners, thanks so

57:02

much for overthinking with us.