Episode Transcript
Transcripts are displayed as originally observed. Some content, including advertisements may have changed.
Use Ctrl + F to search
0:10
Welcome to Geopolitics Decanted. I'm
0:12
Dimitri Alperovitch, Chairman of Silverado Policy
0:14
Accelerator, a geopolitics think
0:16
tank in Washington, D.C. My guest today
0:18
is Brian Vordrin. Brian is the Assistant
0:21
Director of the Cyber Division of the FBI. Essentially,
0:24
all of the FBI's cyber investigations
0:26
on national security and criminal side fall
0:29
into Brian's responsibility. And
0:31
today we want to talk about a couple of topics
0:33
with Brian, namely the intelligence collection
0:35
program called 702, which has
0:37
been discussed a lot in D.C. these days as its
0:40
authorization expires at the end of the year. We're
0:42
going to be talking about its importance and some of its controversy
0:44
with Brian. So, Brian, welcome to the show.
0:47
Thanks Dimitri. I appreciate the opportunity to be here
0:49
with you today. Great. So
0:51
let's talk about 702. So first of all, for our
0:53
listeners that don't know it and aren't
0:55
tracking it closely, explain what
0:58
this program is. It started out
1:00
after 9-11, right, when
1:02
we realized that our collection capabilities
1:05
on foreign terrorists and other foreign
1:07
threat actors were not as good as they should be.
1:10
And we had the benefit of the fact that
1:13
so much of the internet infrastructure was based
1:15
in the United States with all of our telecommunications
1:18
companies, cloud companies, etc., that
1:21
a lot of the foreign threat actors were using. And
1:23
people realized that there was an opportunity to use that
1:26
for intelligence as well as criminal purposes,
1:29
right? Yep.
1:29
Yeah, all of that is accurate. And I'll go in a
1:32
little bit deeper here. You know, FISA
1:34
Section 702, broadly referred
1:36
to as 702, really does benefit the entire
1:38
intelligence community. It's not an FBI
1:41
only equity. It's an equity for
1:43
the entire intelligence community. And
1:45
essentially what it does is it provides the
1:48
intelligence community the ability to collect communications
1:50
of foreign persons located outside
1:53
of the United States. It's
1:55
important to know that it's only used against
1:57
foreign targets of intelligence severance.
2:00
surveillance again who are located outside
2:02
the United States. The one caveat
2:04
I would throw in there though for your audience is
2:07
if there is a foreign actor whether nation-state
2:10
based or criminal based that
2:12
we know is outside the US but that is using
2:15
US infrastructure that
2:17
person would still be viable for
2:19
collection. So
2:20
just to give you know the audience an
2:22
example so this could be sort of foreign
2:24
intelligence officer right let's say the Russian
2:27
SVR officer that may be
2:29
using US infrastructure maybe using
2:32
Gmail account or some other sort
2:34
of infrastructure where their communications are flowing
2:37
through the United States or it could be
2:39
a member of Al Qaeda doing the same thing
2:41
and the thinking here is that they're
2:44
foreigners they're foreign threat actors they
2:46
do not have the protections of the US
2:48
Constitution so you do not need
2:50
a warrant as you would for US
2:52
citizens to tap their communication and
2:54
that is what this program basically allows you to do
2:56
is that correct? That's very accurate to
2:58
me true yes if I can go a little bit
3:01
further and just give you some numbers and some data
3:03
right and I think it's important to understand the approval
3:05
process as well but in terms
3:08
of value 65% of all FBI raw technical
3:11
reporting came from 702 so
3:14
far in this fiscal year for us. So
3:16
what you're saying is that FBI
3:19
with all of its resources
3:21
you know legal wiretaps and using
3:23
criminal investigations and like and everything else when
3:26
you look at all of that
3:27
together 65%
3:28
of all
3:31
of your raw intelligence actually
3:33
comes from this one program.
3:35
It does and you know just to broaden out
3:37
what you shared certainly title 3
3:40
and FISA within the United States fall
3:42
into that but also does our human intelligence
3:44
program which is very very robust
3:47
but to your point 65% of all of
3:49
our raw intelligence reporting did come from 702
3:52
in the first half of fiscal year 23. 97% of raw technical
3:54
reporting
3:54
on civil
3:59
cyber activity came from 702 during
4:02
the same period. So as we look
4:05
at the cyber adversaries and the
4:07
cyber vectors and the threats, they're obviously
4:09
almost entirely based outside
4:12
of the United States. So the human
4:14
intelligence piece, the domestic collection via
4:16
FISA or via Title III is obviously limited.
4:19
That's why that number is so high. And 92%
4:23
of all technical reporting on emerging
4:25
and disruptive techs such as AI and ML
4:27
also came from 702. It's
4:29
a very, very important program. As
4:32
I think we're discussing here, the primary
4:34
national security threats that we collectively
4:37
face today now reside outside
4:39
of our borders. So we must collect our word
4:41
to protect inward. And 702
4:44
is the secret ingredient to that collection.
4:47
If it's okay, Demetre, I would like to walk through
4:49
the approval process for 702 collection.
4:52
Well, before we do that,
4:55
you were sharing with me before we got on some of
4:57
the other statistics that this is just the FBI
4:59
side of it.
4:59
But I think when you look at
5:02
even the overall intelligence that has been
5:04
collected by the entire community,
5:06
the NSA, CIA, and so forth, that
5:09
even they're quite relying on this. Then if
5:11
you look at the PDB, the presidential
5:14
briefing document, quite a bit of that
5:16
information comes from 702 as well. Isn't that
5:18
right?
5:19
That is accurate. 60% of
5:22
all PDB articles include 702-derived
5:25
intelligence. And that's an astronomically
5:27
high number. When you look at the breadth
5:30
of the intelligence community in the United States
5:32
and our partners and how we collect intelligence,
5:34
whether that's through technical or human means, that's
5:37
a very, very high number. So that is accurate.
5:40
60%, I mean, that is just staggering, right? When you think about
5:43
how the US intelligence community collects
5:46
the FBI, CIA, with their human assets,
5:49
got satellites, we've got imagery,
5:51
we've got NSA, wiretapping
5:53
things overseas and breaking into foreign networks,
5:56
and yet 60%
5:58
of the most important intelligence documents commence
6:00
that we produce that goes to the to
6:02
the present every day is coming from this
6:05
one program
6:06
it is and i mean i would even broad now
6:08
what you shared everything you shared is absolutely accurate
6:10
but the global partnerships we maintain
6:13
that seed the overall intelligent
6:15
psycho for the pt is even more
6:17
broad and so but their numbers very
6:19
high a very significant and
6:21
i guess that really speaks to the
6:24
importance of the u s tech sector right
6:26
because the tech sector has become
6:28
so dominant and so indispensable
6:30
to the world that foreign
6:32
threat actors have no choice but
6:34
to rely on some
6:36
ways on us infrastructure and that
6:38
gives us an enormous advantage that
6:40
is absolutely correct and that's an advantage
6:43
that we would like to keep and that's why
6:45
are partnerships and open and transparent
6:47
dialogue with all of those technology providers
6:50
as we navigate legal concerns
6:52
national concert is concerned national security
6:54
concern is our side but also
6:57
transparency and privacy concerns
6:59
on the business side or so
7:01
so important and something that's paramount
7:03
as we look at our relationships and no spaces
7:06
okay said looks like it's really really important
7:08
and if it does not yet renewed the
7:11
nation could be in deep trouble perhaps
7:13
on the counterterrorism side we could be back
7:15
in the pre nine eleven world where
7:17
we are much more blinds of to foreign
7:20
adversaries they're trying to kill americans
7:23
we can imagine the impact and foreign intelligence
7:26
services and back on cyber but
7:28
the program has received some controversy
7:30
and particularly lately with
7:32
the f b i there's been this report
7:34
they came out i understand it's been self
7:36
reported by the f b i but once
7:39
this information gets collected from us
7:41
tech companies under this program
7:43
it goes into government databases and
7:46
f b i and other agencies are allowed
7:48
to query it and the f b i said
7:50
that on almost three hundred thousand
7:53
occasions right there have been ways
7:55
that should not have been done against that data
7:58
to talk a little bit about the
8:00
reasons for that and how you're
8:03
thinking about mitigating that going forward.
8:05
Sure, absolutely. I have
8:08
a lot of information to respond to
8:10
that specific question. So what's
8:12
broadly being referenced here is US
8:14
person or what we refer to as
8:17
USPER query standards. That
8:20
should be broadly interpreted as any
8:22
US-based entity, not just a person. So
8:25
certainly organizations, email
8:27
addresses, people in the United States, basically
8:30
those who have constitutional protections. So
8:34
when we talk about notifications of victims or
8:36
warning targeted entities, this
8:38
is all based on a high threshold
8:40
set for USPER queries. And so one thing
8:42
that's important to us on the cyber side is
8:45
when we receive cyber threat intelligence that
8:48
indicates that a US person or US
8:50
organization may be in target, we
8:52
have a responsibility to engage with
8:54
them. So let me just jump in. So
8:57
basically, again, an example for our listeners
8:59
is you're using 702 to watch
9:02
a foreign threat actor, let's say the Russian
9:05
GRU intelligence service. And
9:08
that obviously you're watching their communications
9:11
as they're traversing US infrastructure. And
9:14
you're seeing that there is a victim
9:16
in the United States, could be an individual, could be an organization
9:19
that they're targeting. And what you
9:21
want to know as the FBI, because you
9:23
do a lot of these victim notifications and
9:25
I've been doing them for many years, is go to that
9:28
person or entity and tell them, hey,
9:30
you're being targeted by a foreign threat actor, you should
9:32
really look into your networks to kick them
9:35
out and shore it up. Is that right? That's
9:37
absolutely right. Yeah.
9:38
And so this USPER query standard
9:41
is quite complicated. And so I'll walk
9:43
through it. And every USPER
9:45
query that we conduct
9:48
has to meet these three. This
9:51
three-pronged analysis, or it's
9:53
considered essentially
9:55
a noncompliant query of our own holdings.
9:58
So let's walk through these. First is to have
10:00
an authorized purpose. So the person
10:03
conducting the query must have
10:05
the purpose of retrieving foreign intelligence
10:07
information or evidence of a crime from
10:09
raw FISA collection. The
10:11
second is it must be reasonably designed.
10:15
The query term must be reasonably tailored
10:17
to retrieve foreign intelligence information
10:19
or evidence of a crime without
10:22
unnecessarily retrieving other information
10:24
from raw FISA collection. And
10:27
finally it must be justified. So
10:29
the person conducting the query must have specific
10:31
factual basis to believe that the query
10:34
is reasonably likely to retrieve foreign intelligence
10:36
or evidence of a crime from FISA. So
10:39
let's just walk through a basic example in
10:41
the cyberspace. So let's say
10:43
the FBI is investigating foreign threats
10:45
potentially targeting a US person and
10:47
for the case of this example let's say it's a Chinese
10:49
nation state adversary targeting
10:52
a US person or a US organization
10:54
via cyber. It doesn't really matter.
10:57
We query that US person or US
10:59
organization's identifiers whether that's
11:02
a name, an email address, an IP
11:04
address against FISA data to
11:06
find whatever foreign intelligence information
11:08
regarding the threat we have in our holdings. So
11:11
we have met the threshold for having an authorized
11:14
purpose which is the first prong. Next
11:17
we have specific facts about the threat
11:19
and relevant foreign actors that make
11:21
it reasonably likely the query will return
11:24
results of raw FISA. So we've
11:26
met the second prong of the analysis
11:28
that the search is justified. However
11:31
we are looking for information about threats
11:33
coming from a particular threat country.
11:36
But we don't limit our query to run
11:39
just against Chinese cyber case
11:41
classifications in FISA holdings. Therefore
11:44
the search is not sufficiently reasonably
11:47
designed. To
11:49
be reasonably designed we would have restricted
11:52
our search to only Chinese cyber
11:54
case classifications to make sure it's properly
11:56
focused down. And Dimitri if
11:58
you're okay with it maybe I can't. run through a few other
12:01
historic examples where 702
12:04
has proven tremendously valuable. But
12:06
let me pause there first.
12:08
Yeah, well, actually, I would be curious,
12:10
you know, we hear about the
12:12
value of it in aggregate 60% of the
12:15
presidential intelligence briefing
12:17
and the like. But if you can share specific
12:19
information about the cases
12:22
where it has helped,
12:23
I think our listeners would really appreciate that. Yep,
12:26
absolutely. So here's just four examples
12:29
I pulled in preparation for today's conversation.
12:32
So FISA 702 played a very important
12:34
role in the US government's response to the
12:36
cyber attack on colonial pipeline. Obviously
12:40
using FISA 702. So
12:43
again, for our listeners, you may not recall,
12:45
this was a few years ago, when colonial 2011,
12:50
2021, sorry, when colonial pipeline,
12:52
one of the nation's largest
12:53
pipeline operators was attacked by
12:56
a Russian criminal group ransomware
12:58
group and had to shut down the pipeline,
13:01
which affected deliveries of fuel
13:03
to the East Coast was a huge news story
13:06
back a few years ago.
13:07
Yep, absolutely correctly. I think it was the
13:10
last weekend in April 2021, or the first
13:12
weekend in May of 2021, but
13:15
right around that time period. And when
13:17
that happened, our FBI Atlanta field
13:19
office had a tremendous relationship with colonial
13:22
and colonial pipeline was a very transparent
13:24
partner during those initial hours and
13:26
days of the incident they were facing. But
13:29
based on intelligence that colonial
13:31
shared with us, we were able
13:33
to then share information with the broader intelligence
13:35
community. And using 702 collection,
13:39
we acquired information very quickly that
13:41
verify the identity of the hacker,
13:44
as well as direct intelligence that allowed
13:46
us to recover the majority of the ransom paid
13:48
by colonial to the actor. So
13:51
use 702 to
13:53
find the identity of the
13:55
group and individual involved here. And then
13:58
there was a US government operation. to actually
14:00
seize the funds from the crypto
14:02
wallet that was used to receive
14:04
the ransom from colonial and the government
14:07
was able to retrieve the money from that
14:09
criminal actor. So 702 contributes
14:11
all of that.
14:12
Correct. Correct. I
14:15
think the next example I have is even a little bit
14:17
more personal, right? We hear a lot about
14:19
transnational repression here
14:21
in the United States and those activities are
14:23
obviously very, very
14:26
concerning to so many of us who
14:28
believe in the civil liberties that each
14:30
of us enjoy as part of this country, whether born
14:32
here or whether they have status
14:34
here. But here's a really good example.
14:37
The FBI used US person queries
14:39
against 702 holdings acquired
14:42
information to identify the
14:44
extent of a foreign government's transnational
14:47
repression activities here in the United States,
14:50
which actually included, if you can believe it, kidnapping
14:53
and assassination plots that were being
14:55
planned to happen here in the United States.
14:58
So foreign government, presumably their intelligence
15:00
services were planning to kidnap
15:03
or assassinate
15:05
US individuals
15:07
and you were able to discover that through 702.
15:10
Yep. And that intelligence led to
15:12
our understanding of what was going on
15:15
and then directly contributed to our ability
15:17
to disrupt those plots. Sometimes
15:20
I feel, Dimitri, that when we talk about these
15:22
examples, they're all Fortune
15:26
500 examples of major corporations
15:28
who we've helped and those are valuable.
15:30
But I think this transnational repression example
15:32
brings it to a very human and personal
15:35
level that shows the span
15:37
of impact that 702 does have. You
15:40
know, Brian,
15:40
one of the things that a lot of people
15:42
may not appreciate is the degree
15:44
to which the FBI and the
15:47
intelligence community gets involved in
15:49
cases of
15:51
kidnappings, particularly overseas,
15:53
of US citizens. So when
15:55
terrorist organizations or others are able
15:57
to snatch Americans, the intelligence
15:59
community really rallies and tries
16:02
to identify whether they're being held
16:04
and help to rescue them as has
16:07
happened on a number of occasions. And
16:10
I imagine that that's another way you could
16:12
use 702 is, you know,
16:14
you're capturing all these foreign communications, you
16:17
know, that an individual an American citizen
16:19
has been kidnapped. And I'm
16:22
sure you would go into that
16:23
database to do a query to figure out if
16:26
you have any intelligence on where they might be held. Is that
16:28
right?
16:28
That's all accurate. Yeah, that is a primary
16:31
mission space for us both domestically
16:33
and globally, or as we obviously refer
16:35
in the IC CONUS for continental
16:38
United States or OCONUS outside the continental
16:40
United States, but everything you said is
16:43
exactly accurate. And just again,
16:45
so that listeners are fully
16:47
understanding this. So the program 702
16:50
is targeted foreigners. So you start with,
16:53
I want to collect on this
16:55
nation state that is a threat to us, I want to collect
16:57
on this terrorist group.
16:58
But obviously, they may
17:01
be communicating with Americans
17:03
or they may be discussing
17:05
Americans in their communications.
17:08
And that's what those US persons
17:10
queries are targeted at,
17:13
right, is to try to understand either,
17:15
you know, a threat to an
17:17
American that these individuals may be discussing
17:20
or targeting, or maybe
17:22
it's a collaborator, right? It could be someone, you know,
17:25
in the terrorism case, a US citizen
17:27
that is involved in ISIS or Al
17:29
Qaeda. And obviously, you'd really
17:32
want to know that right? That's all accurate.
17:34
Yep. Alright, so at
17:36
this point, our listeners are probably thinking this
17:39
sounds incredibly valuable for
17:41
numerous reasons. But what's the catch? Right.
17:44
And the catch is that there's
17:46
there's been some compliance issues. And I'll
17:48
just quote, the Washington Post here,
17:51
the FBI has misused this powerful
17:53
digital surveillance tool more than 278,000 times,
17:55
including against crime victims January.
18:00
six right suspects, people arrested
18:02
at protests after the police killing of
18:04
George Floyd in 2020. In one
18:07
case, 19,000 donors to congressional
18:09
candidate, according to a newly unsealed
18:11
court document, which came from the FISA
18:14
court that the FBI reported these 278,000 cases to. So
18:19
that does not sound like, you know, tracking
18:22
cyber hackers and terrorists
18:24
and the likes. So how does that happen?
18:27
Yep. So
18:29
I'm going to go through this in great detail, because
18:31
I think your listeners deserve to know. And
18:34
so the 278,000 figure, there are certainly some useful
18:36
things I can
18:40
say, and I'll dig into these. So that
18:43
number comes from the FISC, which
18:45
is the FISA courts, April 2022 annual
18:49
section 702 opinion that
18:51
the FBI and DOJ directly contributed
18:54
to. So I think it's important
18:56
to note that that figure is accurate.
18:59
Much of that figure and the underlying
19:02
data behind it was provided by
19:04
the Department of Justice and by the FBI.
19:07
That is our obvious obligation to
19:09
the American public about transparency
19:13
and honesty when we don't
19:15
do what we're supposed to do. So
19:17
those were non-compliant queries against
19:20
raw FISA, not necessarily
19:22
just 702. And they
19:24
span the timeframe of April
19:26
1st to 2020 to March 31st to 2021.
19:29
So the
19:29
report- So basically a one
19:32
year period.
19:32
Exactly. A one year period. And the report
19:35
was essentially released one
19:37
year after that period ended. Over 99.7
19:42
of those errors would have been prevented
19:44
by reforms that were put in place by
19:46
the FBI in the last year and a half. And
19:48
we'll talk about some more of those here, but
19:51
they weren't in place at the time. So
19:53
specifically what would have stopped them
19:56
is the pre-approval requirement for batch
19:58
searches, results, and data. in more than 100
20:01
queries at a time since over 99.7%
20:04
of the queries were large batch
20:06
jobs. So DOJ-
20:09
So batch jobs meaning you weren't just looking
20:11
for one name
20:13
of organization or person,
20:15
you put in a whole slew
20:17
of them, right? Absolutely. Absolutely.
20:20
And the easiest way for me to talk about it because
20:22
many of the batch queries come
20:24
from cyber. Queries are
20:26
a broad range of IP addresses
20:29
that we want to look at at the same time. So it's
20:31
just a really good example of your listeners. Rather
20:34
than running one IP address at
20:36
a time, we would say, listen,
20:39
these 400 are relevant to the threats
20:41
that we're seeing. We're going to run them all at the same
20:43
time. So DOJ determined
20:46
that all the non-compliance, and this is in
20:48
this report, was
20:51
non-compliance, but was also unintentional.
20:54
That is that users were acting in good faith. So
20:56
this was not someone that was looking
20:59
at their previous girlfriend or
21:01
spying on their friends or like. Yeah. And I had two
21:04
things I would note there, right? I think
21:06
Director Ray actually spoke
21:08
to this in his recent testimony and hash
21:10
judiciary and received a very similar question. And
21:13
I would want to reference his testimony specifically,
21:16
and I don't have it in front of me. But I think
21:18
what he said is years ago, there was
21:20
an FBI employee terminated
21:23
for doing intentional things
21:25
in terms of queries of 702. But
21:28
the post-ANI review and the audit
21:30
of all this determined it's
21:33
important to know because I don't want to make excuses. They
21:35
were for an authorized purpose, but they didn't
21:38
follow the query standard, and therefore they
21:40
were non-compliant. But they
21:42
weren't intentionally misusing
21:44
the data for personal
21:45
reasons. And I
21:47
assume that if an FBI
21:50
employee or an employee of another intelligence
21:52
community agency is using it for the purpose
21:54
that I described, looking at their former
21:57
love interest or the like, that this is not a
21:59
case.
21:59
This is not just grounds for dismissal, but hopefully
22:02
there are criminal penalties that would apply here.
22:04
Yeah, I want to be careful with my response to
22:06
that, Demetri, because we have an entire,
22:09
and so does the Department of Justice, and so does
22:11
every other IC department and component
22:14
agency, an entire professional
22:16
responsibility referral
22:18
process to include the Office
22:21
of the Inspector General for every department
22:23
in the US government. And there's very
22:25
well-established protocols and processes
22:28
where if I became aware
22:30
of something like that, I would have a mandatory
22:33
referral process to both our internal
22:35
Office of Professional Responsibility and then
22:38
it essentially works to the department IG.
22:41
So it's very hard for me to say, would there, would there
22:43
not be criminal charges brought? But
22:46
I think the most important thing is that's
22:48
a mandatory referral from
22:50
whoever sees that activity to
22:54
the FBI's internal, most of
22:56
the inspection division essentially, and then
22:58
that gets referred to DOJ-OIG. So,
23:00
and then obviously the process runs from there.
23:03
Got it. So, is it fair to
23:05
say that the majority of
23:07
these non-compliance searches, the 278,000, were
23:12
not intentionally malicious
23:15
and that someone
23:16
just went beyond the established procedure
23:19
when they were running the queries? Is that a fair characterization?
23:22
Yeah, I would go a little bit further. I think that's accurate.
23:24
I would go a little bit further and I would say
23:26
as those queries were conducted, they did
23:29
not meet the intent of
23:31
the three-pronged analysis for the USPRA
23:33
query. They fell short in one area
23:35
or the other or in the batch query
23:37
component that we just discussed. So,
23:40
I think it's important
23:42
for me to say as somebody
23:45
who is responsible for a large
23:47
portion of the organization that
23:50
while unintentional, they were still
23:52
non-compliant, right? And
23:55
those two things should be
23:57
the same. They should be intentional and they
23:59
should be intentional. should be compliant. And
24:01
so, you know, I don't want to ever take
24:04
in kind of an easy button
24:06
out of the problem. The FBI has not done
24:08
that. We've been incredibly transparent with
24:11
the FISC and with oversight
24:13
from Congress and we will continue
24:16
to do those things because we think that
24:19
our errors need to be known, even
24:21
if unintentional, to the American public.
24:24
And that is going to make us stronger and
24:26
Congress conducting the right type of appropriate
24:29
oversight
24:29
and reform will make us all stronger
24:32
in the future. So again, very
24:34
important for me to say while unintentional,
24:36
they were still non-compliant. So
24:39
Brian, you mentioned that these queries
24:41
were conducted from April 2020 through March 2021 and today
24:47
they would not have been allowed because
24:49
you've changed some of these procedures and you've tightened
24:51
them up. But
24:53
you know, I think our listeners would have a question
24:55
of
24:55
why did it take so long, right? This program has been around
24:58
for well over a decade back in
25:00
the 2000s is when it was first created. So
25:02
why only, you know, 2021, 2022 did the procedures get changed?
25:05
Yeah, I actually
25:09
don't have a great answer
25:11
to that, Demetri. I think that as we
25:13
started conducting internal audits,
25:16
which is something that was somewhat new
25:19
to the organization in the late 2018-2019 era,
25:21
this became an area of focus for us in terms
25:25
of risk and compliance, right? And so when
25:28
you look at the stand-up of the Office
25:30
of Internal Audit and the FBI and
25:32
the new capacity to really review programmatic
25:36
risk, that's why this came
25:38
to light at this time frame.
25:40
And if you're okay with it, maybe I can walk through
25:42
the compliance and
25:44
the reforms that we put in place. Is that okay?
25:47
Yeah, let's talk about how can you ensure
25:49
this doesn't happen again. So I think it's
25:51
important to know we hear this term warrantless
25:54
searches used quite routinely
25:56
and I want to address that first and then I'll get
25:58
into the example.
25:59
compliance and reforms that we've put
26:02
in place. So, you
26:04
know, many courts in the United
26:07
States at the federal level have ruled
26:09
that
26:10
information and intelligence that
26:12
are in our holdings and
26:15
our ability to make connections with that data are
26:18
not searches, they are queries
26:20
and therefore they do not require warrants
26:22
that are obviously detailed under
26:25
the Fourth Amendment of reasonable search and
26:27
seizure. So in the case
26:29
of cyber, we're almost always looking
26:31
at net flow, packet capture and other
26:34
cyber indicators. But again
26:36
that Fourth Amendment becomes really really
26:38
important as I said because courts have
26:40
found that querying our already collected
26:42
information is not a violation of
26:44
the Fourth Amendment. And I think
26:46
words matter and when the American public
26:48
hears warrantless searches, it
26:51
puts a thought in their mind about what
26:54
is or is not happening. And I just
26:56
want to really take the opportunity to level set here
26:59
that many if not all US courts
27:01
have already ruled on this and said it's
27:03
not a search requirement under the Fourth Amendment,
27:06
it's a query standard for
27:08
us and we've already talked about that extensively.
27:10
So let me get into the state. So the analogy here would be
27:13
that in the physical world perhaps
27:16
you have at the FBI a
27:18
huge database of fingerprints
27:21
and DNA that you've collected from different
27:24
investigations, right, dusting up crime
27:27
scenes and the like. And if you have
27:29
a new case where you've collected
27:31
a fingerprint off a murder weapon, let's say,
27:34
you can go back into that database
27:36
and search for it because it's already
27:39
in government possession and that is not
27:41
a search right under under
27:43
US Constitution that requires a warrant because
27:46
you already have it. So here you have
27:48
collected this data
27:50
under these 702 authorities by
27:52
targeting foreign communications and
27:55
it's already in your possession. It's no
27:57
longer at the tech companies, right? It's brought into
28:00
the government
28:01
and now you can run queries against
28:03
it
28:04
under law authorized to be under certain conditions
28:07
So you can't just go into that database
28:09
and and fish for anything you like, but that
28:11
is not Constitutionally a Fourth
28:14
Amendment search, right? That's what you're referring to. That's
28:16
exactly correct so
28:18
as a result of our internal findings,
28:20
right we've implemented a series of measures
28:24
in 2021 and 2022 to
28:26
address the root causes of the non-compliance
28:29
and I'll touch on those here in a second but I think it's important
28:32
we view this as obviously and
28:34
appropriately a responsibility
28:36
to do on a continuous basis until
28:39
we're close to perfect. So
28:41
the first specific change
28:43
is opt-in standards where a user has
28:46
to specifically opt-in to
28:48
the three-prong analysis in the US
28:51
Person Query Standard thus affirmatively
28:54
Acknowledging the fact that they've met each
28:57
of those three prongs The
28:59
second is a batch query approval,
29:01
which we've touched on here now We
29:05
can we run batches of up to a
29:07
hundred, right? Which allows
29:09
us to control the volume and really
29:11
tied to the opt-in standards on each batch
29:14
And lastly sensitive
29:16
query approval you had noted earlier
29:18
in our conversation and I can't remember the exact
29:21
number I apologize about congressional
29:23
donors Obviously when they're sensitive
29:26
queries of congressional donors We
29:29
have to meet certain additional thresholds
29:31
and reviews to conduct those queries
29:34
Because of the sensitivity of who
29:36
whether that's a person or an organization
29:39
They've been conducted against or they will
29:41
be conducted against So
29:44
those are the the measures that have already
29:46
been put in place and as the as
29:48
the Fisk noted in its Post-reform
29:51
report, which was published this April They
29:55
found that the FBI has moved to over
29:57
a 98% compliance rate within
30:00
the query standard. So that 278,000 number
30:02
has fallen way, way, way down. But
30:07
I want to overemphasize this. We've
30:09
made improvements, but the number is 100% compliance,
30:13
not 98-plus percent compliance.
30:15
And there are still incidents in
30:17
noncompliance, and FBI leadership
30:20
has determined further remediations
30:23
are necessary. So in June of this year,
30:26
the FBI director announced these further
30:28
measures, and they're really directed
30:31
at accountability procedures and evaluating
30:33
executives responsible for divisions
30:35
at both headquarters and our field offices. So
30:38
now I have leadership
30:41
accountability, responsibility for division
30:43
compliance within the FBI cyber program
30:46
here at headquarters. And so we
30:48
continue to evaluate on a recursive
30:50
basis
30:51
what else we can do to
30:53
put the organization in the right place
30:56
from a leadership accountability perspective
30:58
from an execution perspective, but
31:00
also with a transparency to the American
31:03
public risk perspective. So let
31:05
me ask you this though, because this Washington
31:07
Post story refers to not
31:10
really foreign threat actors, right? But
31:12
January 6th riot suspects, people
31:14
arrested at protests after
31:16
George Floyd. And how does that
31:19
happen? Because 702 is a program targeted for
31:21
foreign
31:21
threat actors, nation
31:25
states, terrorist groups and the like. Yes, obviously
31:27
it ensnares communications that
31:30
refer to or even include
31:32
American citizens, but
31:34
why would that program be used to
31:36
look at sort of domestic criminal
31:39
threats? And I think this is where Americans
31:42
may be uneasy is I understand
31:45
the
31:46
way this program would help
31:48
me if I'm kidnapped overseas
31:51
and the FBI would look into this
31:53
data to try to identify and
31:55
save me. I understand how if
31:57
a cyber threat actor from the Chinese ministry.
31:59
of state security is targeting my company.
32:03
This data is very, very useful. But
32:05
if I am
32:06
engaged in something that,
32:09
you know, may violate the law just domestically
32:11
without any foreign nexus, could
32:14
this information be used to target
32:16
me and then prosecute me for that
32:18
crime
32:19
without requiring warrants
32:22
to tap my communication?
32:23
Sure. If it's okay to mean,
32:26
maybe I take it out of that context and use
32:28
just a basic bank robbery example,
32:30
right? Because I think we can all relate
32:32
to that. So the answer
32:35
is for a bank robbery subject, should
32:38
we or could we query
32:40
FISA and or 702 holdings under
32:43
the USPRA query standard? The
32:45
answer is almost always no. Right
32:48
now, there are some caveats, but let's just
32:50
walk through this. In
32:52
order to query a bank robbery
32:54
subject against FISA or 702 holdings,
32:58
we'd have to have some reason the specific
33:00
factual basis that I referenced to believe
33:02
there's evidence about the bank robbery
33:05
or foreign intelligence information related
33:08
to it in raw FISA or 702. We
33:11
in our world never say these things are never
33:13
possible, but in this example,
33:16
the chances of that being possible are
33:18
extremely low. So
33:20
what factual basis could
33:22
there be to believe the bank robber
33:24
is talking about the robbery with a FISA target
33:26
is a relevant question. And again, even
33:29
if there were- You can imagine, for example, if this is
33:31
a bank robbery that you
33:33
have evidence to believe is tied
33:35
to
33:36
Hezbollah doing fundraising for terror
33:38
organization in the Middle East, that would have a
33:40
foreign nexus. But this is a smash
33:43
and grab of a local community bank that
33:45
you don't have any prior information on a
33:48
foreign threat actor nexus,
33:51
that's off limits, right?
33:52
You are, that is exactly correct. And you
33:54
basically stole the words out of my mouth. We would
33:56
have to show that that person
33:59
was acting on behalf-
33:59
half of a foreign power to
34:02
do those queries against Pfizer 702 that's
34:05
absolutely accurate.
34:06
Got it. So basically
34:08
it's fair to say that this
34:11
always requires for you to go into this database
34:14
and search for a US person which includes both
34:17
organizations and individuals. You
34:19
need to show that foreign access
34:22
to a criminal or
34:25
terrorist or nation state threat
34:27
overseas, right?
34:28
That's absolutely correct. Yep. Got it.
34:31
So
34:33
clearly you can't comment on you
34:35
know ongoing legislation. I understand that
34:38
but it sounds like the FBI appreciates
34:40
that it has not been as compliant
34:43
as it should be in the past. It
34:45
is changing procedures and it sounds
34:48
like you're more welcoming to
34:50
more compliance going forward with regards
34:52
to this program to make sure it's not abused. Is
34:54
that fair to say?
34:55
That's very fair to say yes. And
34:58
tell us one more thing because that might be
35:00
a natural question that people listening
35:02
to us might have right now is why
35:04
would it be a bad idea even
35:07
though you've collected all this data under 702
35:09
it's in your databases. Why would
35:12
it be a bad idea to ask for a
35:14
search warrant before you
35:16
can do that query? Why would that be a problem?
35:19
Well it becomes a resource and
35:21
a time issue and it's a different
35:23
standard. So let's take
35:26
the standard portion of my answer
35:28
first. You know producing probable
35:31
cause that a crime is
35:34
or has been committed is a much higher
35:36
standard than a query standard
35:38
that we just have. And because
35:41
of that the conversation
35:43
that we've already had right about information
35:45
intelligence evidence in our holdings
35:48
should be available
35:49
to us as long as we follow
35:51
the compliance piece of it that's been
35:53
the part of this conversation versus
35:56
establishing a new threshold in
35:58
the legal world which is probable.
35:59
cause that's reason one. Reason
36:02
number two and three are really tied together which
36:04
is efficiency and speed. You
36:07
know for us to and let's take the example
36:09
of the provision
36:12
of decryption keys based on a
36:14
victimized organization sharing intelligence
36:17
with us. The process to query 702
36:20
based on the intelligence that was shared and meeting
36:22
the three prongs of the usper query standard
36:25
is fairly efficient and
36:27
allows us to work on behalf of in that
36:29
case the US based organization
36:32
to provide them relief against a nation-state
36:34
actor. Moving that into
36:36
a criminal search warrant process leads
36:38
to much more work and inefficiencies
36:42
and doesn't allow us to provide relief back
36:44
to those organizations as quickly. So that's
36:47
probably the best answer I can give you that question Demetri.
36:49
Well let's switch topics. It certainly sounds like 702
36:52
is incredibly
36:54
valuable to the national security of this country.
36:56
Hopefully it gets reauthorized
36:58
by Congress before it expires
36:59
at the end of the year. You know maybe there needs
37:02
to be more work on oversight to make sure
37:04
that these past compliance violations
37:06
do not recur but it sounds like things
37:08
are
37:09
moving forward and the court has acknowledged
37:11
that you guys are doing much much better. But let
37:14
me ask you
37:15
about another threat to
37:18
the nation and that is something
37:20
that the FBI and the Justice Department more broadly
37:23
talks about a lot which is
37:25
infiltration of this country by foreign
37:28
intelligence officers, recruitment of
37:31
individuals by foreign intelligence officers,
37:33
primarily China that has been very very
37:35
aggressive in this area
37:37
over the last decade and and you often and
37:39
I've been part of briefings that you've given
37:41
to companies your agents have on this
37:44
issue talk about how companies need to
37:46
start focusing on inside a threat in critical
37:48
areas be it defense contractors where
37:52
Chinese intelligence services may be recruiting
37:54
individuals to work for them
37:56
and pass sensitive information or even
37:58
now the
37:59
artificial intelligence companies, companies
38:02
like OpenAI that produces
38:04
chat GPT and others that
38:07
have these incredibly valuable
38:09
models with parameters in
38:11
them that you could literally walk out of
38:14
carrying on a USB stick, right, and
38:16
sent to China and you have an entire
38:18
policy, of course, that the US government,
38:21
the White House, has announced
38:23
to try to slow
38:25
down Chinese advances in artificial intelligence
38:28
given its importance to national security, weapons
38:30
systems design, and the like. We're trying
38:32
to limit export of advanced chips
38:35
to China to make sure that they can't build those models
38:37
and this seems like a big vulnerability
38:40
where the Chinese intelligence services can
38:42
recruit someone
38:43
inside these companies and just walk out with a model
38:46
without having to train it themselves because
38:48
of lack of chips.
38:50
What can FBI do
38:52
to better educate companies on this
38:54
issue, to better help them understand
38:57
the problem, and how to better deal with it?
39:00
Sure, yeah. I mean, we
39:02
put organizationally an enormous
39:05
amount of attention on this topic,
39:07
whether that is intelligence
39:09
that the FBI has, whether that's intelligence
39:12
that the intelligence community has, to share
39:14
with US-based organizations whether
39:16
their sector is being targeted by
39:19
the human vector or whether they
39:21
specifically as an organization have been
39:24
targeted by a human vector.
39:27
But there's an enormous amount of work that goes
39:29
into this at the FBI headquarters level but then
39:31
also through our counterintelligence program
39:34
in each one of our field offices. There's
39:36
a great 30-minute YouTube video
39:38
that was produced by the FBI probably
39:41
within the last 18 months called Made in Beijing
39:44
that talks about exactly what you're addressing,
39:46
which is the theft of intellectual property
39:49
from US-based organizations. And obviously
39:52
if we know about it, we will
39:54
advise and notify the victimized
39:56
organization or we will work it as
39:58
a criminal
39:59
conduct matter and try to get a
40:02
prosecution out of it. But again, that's
40:04
only a point about- And by the way, sometimes you've had
40:06
cases where you've highlighted this. Sometimes
40:08
there's a joint human and cyber nexus
40:11
to this, where I think it
40:13
was one of the aerospace companies where
40:15
they had recruited an individual to
40:17
implant malware on the network to
40:20
actually allow them to exaltrade data, right?
40:23
Yep, and we're seeing that more and more. It's
40:25
a blended or a hybrid threat between traditional
40:27
cyber vectors and now human vectors. Very,
40:30
very challenging to discover, even
40:33
for the most adept of us, but very challenging.
40:36
So when it comes to the insider
40:38
threat, we're large from
40:40
an organization's perspective, we
40:42
really encourage them to do their best
40:45
to identify, to engage and to protect.
40:47
And by identify, we mean, what are
40:49
your most valuable assets that your competitors
40:51
want to include China, then develop
40:54
partnerships with security, local FBI
40:56
field offices and establish a dialogue
40:58
to report threats, suspicious behavior
41:01
and vulnerabilities. And we'll talk about all these a little
41:03
bit more. Then protect, right?
41:05
Make your asset hard to steal, strengthen
41:07
your cybersecurity, be aware of foreign
41:10
travel, foreign ties, keep an eye on people
41:12
who visit your facilities. You
41:14
know, we often say, be careful
41:17
about the last person added to the
41:19
delegation visit, if a foreign
41:21
adversary is coming to visit you or
41:23
a hostile foreign nation has come to visit with
41:25
you. So if the Chinese are coming
41:27
to visit a defense industrial based company,
41:30
in two days before they add two people, that
41:33
is a clue to look at those two people. But
41:36
you know, Brian, and maybe this is
41:38
where the partnership comes in with local
41:41
field offices, the government has
41:43
much better resources
41:45
than private companies to look at this, right? So
41:48
if you are someone that is applying to work
41:50
in the US government or even receiving
41:52
a clearance, you're gonna get through
41:54
very extensive background checks, maybe
41:57
even polygraph, you're gonna be looked
41:59
at for all.
41:59
of your family relationships and everything else,
42:02
private companies can do that. In many
42:04
cases, legally are prevented from
42:06
doing that. So how
42:09
can the private sector
42:11
and the government and particularly the FBI work
42:15
more in partnership together on solving this
42:17
issue?
42:17
Sure. And it's a great question
42:20
and one that I receive more
42:22
often than not, which is can the
42:24
FBI help company A or
42:26
company B vet foreign nationals
42:28
that they're going to employ? And
42:31
the answer to that, as I would expect that all
42:33
of us would expect, is that we
42:35
can't vet. But let me walk
42:37
you through how we look at the problem
42:40
because I think it'll help your listeners
42:42
understand how and perhaps even
42:44
more importantly when to engage with the
42:46
FBI. But on that note,
42:48
we really do encourage, build those relationships
42:51
early so that when it's needed, you have
42:53
it and so you're not starting to screen on ground one.
42:56
So if a private
42:58
company or it doesn't
43:00
have to be a private company, it can be obviously an
43:03
NGO, but an organization based
43:05
in the United States, non-governmental organization
43:08
provides sufficient information to us
43:10
regarding an individual to
43:13
show a connection to
43:15
an unauthorized national
43:17
security purpose. So let's like dissect
43:19
that. Now we have a person
43:21
that is working for a corporation in
43:23
the United States and that corporation
43:26
through a little bit of due diligence realizes
43:28
their communication back to someone
43:31
or an entity in China that is perhaps
43:33
national security state sponsor affiliated.
43:36
The FBI can move forward with certain
43:38
authorized activities such as records
43:40
checks, information searches, other government
43:43
agency records, information searches, clarifying
43:46
interviews, these types of things. And by
43:48
the way, all of this is memorialized and
43:50
open source as part of the FBI's domestic
43:52
investigations and operations guide so
43:54
they can read it right in there. But that's
43:57
how almost all of our meaningful
43:59
counter. intelligence and insider threat investigations
44:02
begin by an organization
44:04
saying, hey, something doesn't look quite
44:07
right here. And because of Reason
44:09
ABC, we are going to engage
44:11
the FBI and we're going to start to have a more robust
44:14
conversation. So that is a way
44:16
forward when an organization identifies
44:18
something that is a little bit concerning to
44:20
them. But what we can't do,
44:23
and again, I think and hope that the
44:25
listeners would understand this is aligned with our
44:27
constitutional priorities,
44:30
is we can't just take a name and identifying
44:33
information of everyone who works for
44:35
a company and run them in our holdings to
44:38
see if there's anything there, right? We have
44:40
to have an authorized purpose. I guess
44:42
the root of this is that companies,
44:45
individuals to the extent that they have these
44:48
national security threats
44:49
should be reaching out to their local field office
44:53
and building those relationships. And I know in
44:55
particular larger offices, there is
44:57
often events that the FBI
44:59
puts on, annual events and the
45:01
like where they
45:03
do briefings for organizations, but also
45:06
is an opportunity to network and build those
45:08
close relationships. Right. I know you
45:10
kept mentioning Atlanta. I know that Atlanta
45:12
field office very well. I used to live there back
45:15
a number of years ago and they do a lot of those
45:17
types of activities.
45:18
All of that's accurate. But if
45:20
I can speak a little
45:22
bit about the second question
45:25
I get is, OK, Brian, we understand
45:27
that that's what we need to provide you. How do we build
45:30
our organizations so that we can uncover,
45:32
identify these things to provide them to
45:34
you? So let me just walk through
45:36
that because I think it's really important. So
45:40
number one, we do recommend that companies
45:43
build a dedicated insider threat
45:45
program and a comprehensive
45:47
one. And I'll go through what that means here
45:49
in just a second. But as part
45:51
of that, they should probably have a review board
45:55
not only to understand ethical
45:57
requirements within the organization about what
46:00
They are and are not going to track, but
46:02
almost a board to say, hey, we do or
46:04
don't think we have a problem with what has
46:07
been identified. And so that
46:09
insider threat program has to be super
46:11
robust. So we already talked
46:13
about collaboration with trusted government partners
46:15
identifying the crown jewels. But
46:18
then it's important to analyze the threats and vulnerabilities
46:21
to personnel, both physical and information
46:23
security. And there has to be
46:25
trend analysis done
46:28
against those variables that you're going to choose
46:30
to measure on a recursive, continuous
46:33
basis. The system alerts and triggers
46:35
have to identify anomalous behavior.
46:38
Again, physical and information
46:40
security. Physical could be anything
46:42
from financial problems
46:45
that are trying to beginning to show with somebody. The
46:47
anomalous
46:48
behavior could be
46:50
system administrator access that someone
46:52
doesn't have, or going to websites,
46:54
or using covert
46:57
email accounts or covert apps to
46:59
communicate with unidentified people. And obviously,
47:01
the risk proposition gets higher
47:04
for those in the U.S. doing classified
47:06
work. But even if I was a significant
47:09
company, you had mentioned one with
47:11
a artificial intelligence model, it's
47:13
worth a lot of money. And protecting that
47:15
IP is not only important to that company, but
47:17
it's important to our economic viability
47:19
as a country moving forward. You know,
47:22
perhaps not a term of art, but the
47:24
minimum people standards that
47:26
should be measured and evaluated. Here's
47:28
just a short list. Someone who disregards
47:31
rules and authority, unreported
47:34
foreign contacts or travel, sudden
47:36
or uncharacteristic behavioral changes,
47:38
inconsistencies in administrative records,
47:41
deception, sudden unexplained changes. And
47:45
then there's a big one, a change in financial status, problematic workplace attitudes,
47:48
family stress, and a big one
47:50
is remote access, noncompliance, privileged
47:52
user noncompliance. But
47:54
again, having that executive level buy-in
47:56
and business ability, the insider threat
47:59
risk board that I talked about, about critical
48:01
assessment reviews, detection and reporting,
48:04
data aggregation, analytics support, etc.
48:06
These things all become important from
48:09
a strategic to a tactical level
48:11
to make sure that we're bubbling
48:13
to the top those people that
48:16
may be a risk to the organization and
48:19
then it allows effective engagement
48:21
with the FBI about hey here's
48:23
what we're seeing here's why this person
48:25
we believe is a problem what can you
48:27
FBI do to potentially help us corroborate
48:29
that this to put us in a safe position. This
48:32
is a great list of risk factors
48:34
to look for you know it strikes me when
48:36
you look at publicly disclosed
48:39
cases of foreign
48:41
agents that have been caught either inside
48:44
US government or inside US companies
48:46
most of them have not been these brilliant
48:49
spies that you know went through
48:51
all these covert ways to hide their affiliations
48:54
many of them have had you know gambling problems
48:57
lots of foreign travel including
48:59
to two countries of concern like
49:01
China
49:02
and if only you just look at
49:05
it deeply enough it doesn't require
49:07
enormous resources to become quite
49:09
suspicious very very quickly right. Yeah
49:12
that's true and maybe I can just you
49:14
know foot stomp two things here right
49:17
we do view insider threats as
49:19
a human human problem that
49:21
requires a human solution right
49:24
we've seen organizations over
49:27
index on a tool heavy solution
49:30
versus really trying to identify
49:32
human behavior through human observation
49:36
to feed into the insider threat
49:38
program and that's why that program at
49:40
a strategic level is so important and I already
49:42
mentioned this but I really want to
49:45
reinforce it that program has
49:47
to have C-suite level buy-in and visibility
49:50
both buy-in and visibility so
49:53
that there's clarity and transparency
49:56
about how it's being administered how it's
49:58
being run and whether
49:59
when there are problems identified, that visibility
50:02
becomes important. Well, and I would expand
50:04
that it should have buy-in at the employee level
50:07
too, because at the end of the day, this
50:09
is a problem not just for one part of your company,
50:11
it's a problem for the entire company. And
50:14
you really wanna make sure that all employees are on
50:16
the lookout for threats, in the lookout
50:18
for changes in behavior of their colleagues that
50:20
could trigger suspicion. And that they
50:22
also don't feel like, you know, they're being surveilled
50:25
by the company in a, well,
50:27
certainly not unlawful way, but also
50:29
in a way that makes them uncomfortable, right? So,
50:33
you know, you need to build that shared purpose
50:35
and mission for everyone to say, look
50:37
guys, we are
50:39
under significant threat from a foreign
50:41
nation state, stealing our very sensitive intellectual
50:44
property that could potentially have even
50:46
a national security impact to the country,
50:48
and this is why we're doing X, Y, and Z,
50:50
right?
50:51
Yeah, I agree with all that, Dimitri. That just
50:53
is aligned with being a good organization
50:56
in terms of leadership and values to your employees,
50:58
yep.
50:58
Well, Brian, thank you so much for
51:01
coming on. This was really, really fascinating
51:03
and valuable, I think, to learn
51:06
about how the FBI uses this really
51:08
important program. Full disclosure to
51:10
our listeners, I've known you for quite some time.
51:12
We worked together on something called the Cyber
51:14
Safety Review Board, CSRP, that
51:17
is this joint public-private board
51:19
that was created to sort of mimic a little bit
51:21
the NTSB, the National
51:23
Transportation Safety Board in cyberspace
51:26
to investigate critical incidents. And
51:28
I
51:28
know you to be not just an incredibly
51:31
smart and dedicated professional,
51:33
but a true patriot, and the country
51:36
and the FBI is very lucky to have you. Thanks,
51:38
Dimitri. I really appreciate the opportunity
51:40
to be here with you today to talk about such
51:42
an important topic, and just
51:45
really appreciate everything you do to try to improve
51:47
our national security and transparency
51:49
with the American public, and you've been a tremendous
51:52
partner, and look forward to the continued work with you. Thanks
51:54
so much. Thank you.
51:56
Yeah. Way
51:59
down in the. Oh,
52:02
we're dialing home.
Podchaser is the ultimate destination for podcast data, search, and discovery. Learn More