0:01

Welcome to the Emerging Litigation Podcast

0:04

. This is a group project driven

0:06

by HB Litigation , now part of

0:08

Critical Legal Content and

0:10

VLEX Company's Fast Case and Law

0:12

Street Media . I'm your host , tom

0:14

Hagee , longtime litigation news

0:16

editor and publisher and current litigation

0:19

enthusiast . If you wish to reach

0:21

me , please check the appropriate links in the

0:23

show notes . This podcast is also

0:25

a companion to the Journal of Emerging Issues

0:27

and Litigation , for which I serve as

0:29

editor-in-chief , published by Fastcase

0:32

Full Court Press . And now

0:34

here's today's episode . If you like what

0:36

you hear , please give us a rating . So

0:41

welcome back to another episode . We're going to get practical

0:44

on this one , as we sometimes do Try

0:46

to break or mix things up a little

0:48

bit for you , and we're also

0:50

going to have our guest host , sarah Lordback . Sarah

0:53

, she's been doing a series for

0:55

us on legal tech

0:57

and you're lucky to have her , and

1:00

so am I . So here we go . This

1:02

one's going to be about eDiscovery . We have talked

1:04

about this one before . It's a huge

1:06

part of any litigation practice , as anybody

1:09

in litigation knows . It's

1:11

been transformed quite a bit by Technology

1:13

. Assisted Review Tools , or TAR

1:16

is the acronym . There . Ediscovery

1:18

is its own specialty . We've got

1:20

eDiscovery experts on staff , often

1:23

who know all there is to know about the

1:25

technology , the standards , the processes

1:27

or processes and practices

1:29

or practices . Every

1:32

litigator needs to understand how e-discovery

1:34

tools work . You've got to be able

1:36

to answer questions around the approach being

1:39

used , why a specific

1:41

approach was chosen . It's good

1:43

to understand the reliability of the assisted

1:45

review , a human oversight , implementation

1:49

too . You know it's funny , we're talking about humans

1:51

. That's been happening more and more in my conversations

1:54

recently . As we talk about these things , we

1:56

say , well , how are the humans doing ? That

1:58

was never a thing as far as I

2:00

know . I don't know maybe if I worked at NASA

2:03

or something . It's a lot of jargon here

2:05

. It's a lot of acronyms , specialized terminology

2:07

. That's what jargon is and it's always

2:09

changing because technology is changing . Also

2:12

changing are the legal standards and

2:14

ethics ethics

2:17

, competency , issues around technology . Because

2:20

so much of our work is done by a technology

2:22

vendor that has specialized tools . It

2:25

can feel like your review is based

2:27

on blind faith and that

2:29

finding the pieces to support your case requires

2:31

you to rely on dumb luck . Can

2:33

you do more than pray to the document

2:36

? Gods , yeah , for

2:38

those who celebrate , but I'll tell you what . Listen

2:41

as Sarah Lord interviews Lenora Gray . But I'll tell you what . Listen as

2:43

Sarah Lord interviews Lenora Gray . Lenora is an e-discovery expert and

2:45

data scientist , skilled in auditing

2:47

and evaluating e-discovery systems . In

2:50

her role at Redgrave Data , she designs

2:52

and analyzes structured and unstructured

2:54

data sets , builds predictive

2:56

models for use in TAR workflows , implements

2:59

automation solutions and develops custom

3:01

software . Before Redgrave , she spent 12

3:03

years as a paralegal role

3:11

in which she managed discovery teams . She's currently pursuing her MS in data science from Johns Hopkins

3:14

University . She earned her BS in computer

3:17

science from Florida Atlantic University

3:19

and , like I said , I welcome back

3:22

Sarah Lord to guest host

3:24

. Sarah is a former practicing

3:27

attorney with a decade of experience in data

3:29

analytics . She applies her experience

3:31

in law firms and businesses to explore

3:34

and address the cultural and practical barriers

3:36

to diversity in law , supporting

3:38

value creation through legal operations and

3:41

client-first , business-oriented

3:43

practices . In her recent work as managing

3:46

director of legal metrics , she led a team

3:48

of experts focused on providing

3:50

the tools to support data-driven

3:52

decision-making in legal operations

3:54

and closer collaboration between law

3:56

firms and their clients through automation

3:58

and standardization of

4:01

legal metrics . Sarah

4:03

earned her JD from New York University

4:05

School of Law . Also

4:09

, you want to check out our bonus content

4:11

. We're going to have that on YouTube and at

4:13

litigationconferencecom and

4:16

there you will see that Sarah makes it pretty clear

4:18

that she doesn't love that

4:20

initial review part of discovery

4:23

. I don't know . I think that comes

4:25

through . But check

4:27

out the YouTube video on

4:30

YouTube and at litigationconferencescom

4:32

for some bonus content . And

4:35

now here is Sarah Lord

4:37

interviewing Lenora Gray , a

4:40

data scientist at Redgrave

4:42

Data . I hope you enjoy it .

4:46

Okay , a data scientist at Redgrave Data . I hope you enjoy it . Okay , welcome to all you

4:48

podcast people in podcast

4:50

land . We are here to

4:52

talk to Lenora Gray . I'm very

4:55

excited about this discussion . Thank you for joining

4:57

me . Thank you for having me . We

4:59

are going to talk about tar

5:02

and e-discovery , which I know

5:04

everyone loves this topic

5:06

, so it's going to be very exciting . So

5:09

let's just start with the basics . How

5:12

would you describe technology-assisted

5:14

review for those who are new to litigation

5:16

or don't regularly work with

5:18

the review process ?

5:20

Well , technology-assisted review is a way for

5:22

us to use computers and machine

5:25

learning algorithms to assist to

5:27

make document reviews more efficient . So

5:29

we use a form of machine learning

5:31

that's called supervised learning . We

5:34

call it supervised because it learns by example

5:36

. So basically , let's

5:38

say you have a large collection of documents

5:41

that might be important for you to produce

5:43

to the other side , collection of documents that might be

5:45

important for you to , you know , produce to the other side

5:47

and you don't want to

5:49

review them all one by one because it would cost you a gazillion dollars and take you so much

5:51

time to do so . So you can have one or maybe a few attorneys , take

5:54

a small set of those documents , code

5:56

them for whatever issue you have relevance versus

5:59

non-relevance , privilege versus non-privilege

6:01

or any you know categorical issue

6:03

that you can , you know , devise privileged versus non-privileged

6:05

or any categorical issue that you can devise and then the supervised

6:07

learning algorithm uses that training data that your attorney's

6:09

coded and

6:17

it produces a statistical model that we call a classifier . The classifier can

6:19

then be used to predict the coding of the rest of that document

6:21

population . So it'll make a statistically valid

6:24

guess on what the coding would be if the attorneys

6:26

had looked at it and then the classifier

6:28

can also give you a confidence level on

6:31

its prediction . So and

6:33

then we can rank those documents by that confidence

6:35

level . Let's say , if it has a confidence of

6:37

one , I'm 100% sure that this document

6:39

is relevant as opposed to non-relevant

6:41

. We can rank them by the confidence

6:43

level and we can bring those documents to the attention

6:46

of the attorney so they can review the

6:48

relevant ones first . Or

6:50

, if we are very , very confident in

6:52

the accuracy

6:54

of our classifier , we

6:57

can just produce them as they are without the attorneys

6:59

having to look at them at all . So it basically saves

7:02

you a lot of time and effort when it comes

7:04

to conducting a document review . Okay

7:22

, so let's say I'm a litigator beginning

7:24

a discovery process . What are the primary TAR

7:26

options available to me ? And the first phase

7:28

, the active learning , which is the

7:30

supervised learning , iteratively

7:33

sorry , that's a hard word for me selects

7:35

documents for training

7:37

purposes , and reviewers code those documents

7:39

and the resulting coded

7:41

set is used to train and produce

7:43

the classifier , like we said before . And

7:53

the second phase once we are sure of the accuracy and we like what the classifier is doing , it's

7:55

used to classify the rest of the documents , and then the rest of the documents

7:57

might get reviewed by attorneys or , like lower

7:59

cost reviewers , contract reviewers instead

8:01

of senior attorneys , or just

8:03

a privilege screen , or they might be produced without

8:06

further review from anybody . There's

8:09

also TAR 2.0 , which omits

8:11

the second phase . So what it does is it

8:13

in each new iteration

8:15

we produce a new classifier and as

8:18

documents are coded in the collection , the

8:20

coded documents are used to produce a new classifier

8:22

, and this keeps going and keeps going until

8:25

an agreed upon number of

8:27

relevant documents have been found in the collection . It's

8:29

usually around 80% for legal cases

8:31

and , as I said before , there are some

8:33

hybrid options . But these are

8:36

the two basics and ultimately

8:38

, the choice between choosing TAR 1.0

8:40

and TAR 2.0 should be

8:42

guided by specific objections

8:45

, constraints and characteristics of your project

8:47

, your document collection , what it is the

8:49

issues are in your case . So consulting

8:52

with e-discovery experts at this juncture

8:54

kind of like what we do at Record Data they

8:57

can help you determine the most appropriate approach

8:59

, as , like we would know what to look for

9:01

at this stage that would have downstream effects

9:03

in the process going forward .

9:05

These tools incorporate generative

9:08

AI .

9:11

Some of the major players in the eDiscovery

9:13

platform are currently incorporating

9:15

Gen AI into their software , not

9:18

just for TAR review but for like deposition

9:20

prep , summarizing some of those

9:22

documents . But as far as

9:24

incorporating Gen AI at BreadGrid Data

9:26

, we decided to kind of take a step back and

9:29

make sure that generative AI , when

9:31

evaluated against the tools we're currently using

9:34

in eDiscovery , that it matches

9:36

up and is a viable option

9:39

for our clients . So what we're doing

9:41

now is we're currently doing research to

9:43

compare generative AI against our current

9:46

baseline machine learning algorithms like logistic

9:48

regression and support vector machines , which are

9:50

big statistical words , but it's basically

9:52

the computer algorithms that we currently use

9:55

and we're comparing them to GenAI . And

9:57

if GenAI proves to be at

9:59

least as effective as human review

10:02

or like a TAR 1.0 , while

10:04

it's still comparable in time and cost , then

10:06

it could be used in place of the current algorithms we

10:08

have . Yeah , in the resources I provided

10:10

, there's an article that my colleagues

10:12

just wrote Jeremy Pickens and Tara and

10:15

Will Zett that they published in the

10:17

Sedona Journal and it's outlining

10:19

the difference between using traditional machine learning

10:21

and Gen AI in TAR 1.0 workflows

10:24

. Also , gen

10:26

AI has potentials beyond the

10:28

you know regular traditional machine learning

10:30

because it can be used

10:32

for privilege review and summarization of

10:34

groups of documents and it has

10:36

a huge capability to point out

10:38

relevant passages and also provide

10:41

explanations of why it

10:43

classified a document a certain way . So it has

10:45

a leg up if we can prove it's as valid

10:47

and effective as what we're currently using . Also

10:50

, genai is being used behind the scenes

10:53

to improve language processing tools for

10:55

like name and entity extraction , redactions

10:58

and related tasks like that . So it

11:00

kind of has a lot of potential , but we want

11:02

to be sure that it's as effective as what we have now

11:04

.

11:05

You co-authored a

11:07

white paper called Beyond

11:10

the Bar Generative AI as

11:12

a Transformative Component in

11:14

Legal Document Review . Is

11:16

that the article you were just referring to

11:18

, that your colleagues

11:21

also co-authored , or is this a different one ?

11:23

Oh no , this is a different one . We're constantly doing

11:25

research and actual scientific

11:27

experimentation on these tools at RedRigData

11:30

. So our paper Beyond the

11:32

Bar was actually submitted to a

11:34

scientific conference recently . We

11:36

beefed up all of the mathematics to

11:38

make sure it was a scientifically

11:40

strong paper so that we could submit

11:42

it . So it would be beyond just a

11:45

white paper with our opinions on what was going

11:47

on . So for this paper Beyond

11:50

the Bar , which we did a collaborative research with

11:52

Relativity , which is one of the big e-discovery

11:54

platform players , we designed

11:56

a head-to-head comparison of

11:58

first-level human review and generative

12:00

AI . And I say first-level review because

12:03

when you do a fully human review it's usually

12:05

done in two stages . In

12:07

the first stage , you'll have a set

12:09

of contract attorneys , review and label

12:11

, you know , a set of documents , and

12:13

then in the second stage , more senior

12:16

attorneys will come in and check what they

12:18

have coded to make sure that they have done

12:20

the right thing , and there might be , you know , case

12:22

attorneys that you know guide them on

12:24

that path . In our experiment

12:26

we provided a review protocol to

12:28

both the contract attorneys and to the

12:30

generative AI algorithm as like a prompt

12:33

, like you do in ChatGPT , and

12:36

both of those , both of their coding

12:39

results were compared against

12:41

the gold standard of what the senior attorneys

12:43

would have coded the documents and

12:45

in the results we saw that the generative

12:48

AI system had much

12:50

higher recall than a human review , that

12:54

the generative AI system had much higher recall than a human review . It found 96%

12:56

of the relevant documents as opposed to 54% that were found

12:58

by the humans . E-discovery standards usually require around

13:00

80% recall , so this is very , very good

13:02

. However , the precision

13:05

measure for generative AI was lower

13:07

than the human reviewers . We had 60%

13:09

precision as opposed to 91%

13:12

precision from the humans . So

13:14

precision is a

13:17

measure of the documents that

13:19

you found that you coded as relevant

13:22

. How many of them were actually gold

13:24

standard relevant ? So , like , how

13:26

precise is your classification

13:29

? Precise is your classification

13:31

? So , while we were able to get a

13:33

much , much higher recall , it's

13:40

incurring extra costs in producing documents that are not relevant along with the relevant

13:42

ones , although it's finding most of the relevant ones . So there's more work that needs

13:44

to be done . But we have numerous avenues in improving

13:46

the performance . Like we

13:48

didn't do any fine tuning

13:50

of the prompt that was specific to the matter . We

13:53

, you know , did the one prompt and

13:55

so there's much more to come in experiments around

13:57

Gen AI . For document review For

13:59

future view .

14:01

here it seems like we

14:03

may be on a path

14:05

to a quicker review

14:07

process , a more affordable

14:09

review process and , for those of

14:11

us who do not enjoy first level

14:13

review , a less emotionally

14:16

painful process as

14:18

we enter into the future

14:20

of discovery .

14:22

And also a more robust review , because now

14:24

the general AI will be able to tell you why it made a

14:26

decision and also point out the passage of the document that it you know was

14:28

most relevant to its decision . So it kind of like gives you these you know was

14:30

most relevant to its decision . So it kind of

14:32

like gives you these you know breadcrumbs

14:35

in the actual documents to tell you why

14:37

I made this determination . So it's actually

14:39

. You can pinpoint

14:41

where it might go wrong and where it's

14:43

going very right easily without

14:45

you know with actual

14:47

text . So it helps you a lot . So let's

14:49

say that there's some passage , some stock

14:52

passage or some boilerplate

14:54

language in the documents that you haven't seen so far

14:56

, and it's picking it up and you know , making

14:58

it relevant when it shouldn't be . You can

15:00

, you know , flag that and say just

15:03

because it has this doesn't mean it's relevant . So that's not

15:05

going to be , you know , considered in the statistical

15:07

model . Go back and tell me

15:09

if this is really relevant , based on the rest of the text

15:11

.

15:11

That to me sounds very exciting . So

15:14

when will TAR

15:17

be replaced with fully

15:19

robotic review , which I am

15:21

going to call fur , because

15:23

why not ?

15:25

Is this one you made up , because I've never heard

15:27

of

15:29

it .

15:31

Yes , I wanted to catch on , so

15:33

let's all just start calling the next

15:36

phase fully robotic review

15:38

, aka we'll have tar and

15:40

fur . Now , when the fur gets in the tar it might

15:42

get a little messy , but I don't know .

15:43

Oh , my gosh . No , this concept

15:45

was so far off our radar it didn't

15:47

even have a name

15:50

.

15:50

I'm the first . I did it .

15:54

I think it's important to consider that , while TAR is about technology

15:56

, the success of a TAR effort depends

15:59

heavily on the larger e-discovery

16:02

process around it , and that larger e-discovery

16:04

process requires attorneys and

16:06

specialists and technologists

16:08

and people that know how to do the work

16:10

. The machine

16:13

isn't going to do everything , so the

16:15

negotiations around , for instance

16:17

, how the review collection is defined , are

16:20

supremely important , like negotiating

16:22

custodians , date ranges

16:24

, file types , keyword filters . This

16:26

can affect both the volume of the data that has to

16:28

be reviewed and it can also determine

16:30

how much of the data can be reviewed , be reviewed , and it can also determine how much of the data can be reviewed

16:33

through TAR , because there are certain file types that TAR

16:35

is just not , you know , fitted for . Certain

16:38

decisions made on the request for production

16:40

can affect the difficulty of the classification

16:42

task , so it can make it more difficult to TAR to

16:44

determine whether it's , you know , relevant

16:47

, non-relevant , privileged , non-privileged and

16:49

agreed upon deadlines strongly affect

16:51

which TAR workflow you'll be able to use and

16:53

which one is most practical . Some

16:56

of them take , you know , longer to get the classifier

16:58

right than others , so you know that

17:00

also depends . So humans will be needed for

17:02

each of those steps in the process , so

17:04

the time we will have a like a set it

17:07

and forget it review process , I think is a long

17:09

way out .

17:10

Okay , so TAR is still our best option

17:12

and , recognizing

17:14

that the technology is always evolving

17:16

, are there key terms that litigators

17:19

need to understand and key questions

17:21

they should ask to select the right

17:24

tool ?

17:25

Well , sometimes the TAR process are called

17:27

by names other than 1.0 or 2.0

17:29

. So practitioners would want to be aware

17:31

that there are other names for it . I would

17:33

actually encourage them to you know , figure

17:37

out what exactly they are doing and how it compares

17:39

to the standards that we have for TAR 1.0 or

17:41

2.0 , because there might be some hybrid process that

17:43

your provider is using that you don't know . The

17:47

TAR evaluation metrics are important and

17:49

knowing how those work and what they signify . You know

17:51

the TAR evaluation metrics are important and knowing how those work and what they signify

17:53

, the recall and precision numbers . Recall is the percentage of relevant

17:55

documents that you were able to find . Precision

17:57

is how accurate your classifier

18:00

is of the documents that you found

18:02

that you said were relevant . How many of them were really relevant

18:04

, based on gold standard . Those will

18:06

help you set objectives for your TAR process

18:08

and do the negotiation on the front end . You

18:11

know what level of recall are we looking for here . You

18:13

would want to know what a confidence interval is and

18:15

how they work , which is basically a range of values

18:18

that the true value is estimated to

18:20

be in at a certain confidence level . So

18:22

we'll say we found 96%

18:25

of the relevant documents and we're 95%

18:27

confident in that number . It's between , you know , 95.5

18:30

and 96.5 . It's somewhere in there . We

18:32

think it's 96 . And we're 95%

18:35

confident about that . Terms

18:37

that have to do with defining the review

18:39

collection , such as culling , which

18:42

is when you reduce the document population based

18:44

on the date range or file

18:47

type or keyword searches . A lot of that happens

18:49

, you know , happens in the processing

18:51

stages and things . Deduplication

18:53

is brought up a lot , which is where we replace

18:55

multiple identical copies of a document

18:58

in a document collection by just one representative

19:00

copy . The nesting

19:02

, which is where you remove a bunch

19:04

of system files . When

19:07

you collect documents from a computer system

19:09

or files , it comes with all of the accompanying

19:11

little junk files that you don't need to review . So

19:14

some of those are important to know when

19:17

you're negotiating TAR protocols

19:20

and I also have provided some

19:22

resources to get anyone starting and learning

19:24

about TAR and the glossary for TAR and

19:27

for a deeper understanding of TAR and

19:29

AI and e-discovery practice . We also

19:31

have training programs specialized

19:34

that are ideal for legal and IT professionals

19:36

and Red Grave Data

19:38

provides them through our education and training

19:41

program . So if you want to do a deep dive

19:43

on the essentials , the best practices and ethical

19:45

considerations . Consider going

19:48

to our website and looking it up , because we've got some

19:50

really good stuff there Fantastic

19:52

.

19:53

I've heard about system review

19:55

audits and the

19:57

value around those , but

20:00

I don't really know when they should

20:02

be considered . Can you tell

20:04

us a little bit about , from a best practice

20:07

standpoint , when we should be considering

20:09

review system audits ?

20:11

Well , tar processes and systems have

20:13

to be evaluated at a couple stages To

20:16

ensure that the classifier is bringing back the correct

20:18

documents . You will , you

20:20

know , evaluate the classifier when

20:22

it's created . So when you have in like a TAR

20:24

1 workflow , you have those attorneys

20:27

that go and code the small set of documents

20:29

, you use it to build a classifier

20:31

and then you check if the classifier

20:34

is doing its job correctly , if it's pulling back the

20:36

correct relevant documents , and

20:38

then you would also evaluate it at the stage where

20:41

you have to certify to the other parties

20:43

that the result of your coding has

20:45

met the established objectives . We recovered

20:48

at least 80% of the relevant documents

20:50

from this collection . We're certifying

20:54

it using this , and usually this is done by

20:56

random sampling . So

21:12

we'll select a subset of the documents from the entire set at use our

21:14

statistical equations to estimate

21:16

whether or not the TAR process was effective

21:18

, whether or not it's bringing back a requisite

21:22

amount of the relevant documents . However

21:24

, random sampling is just one of many methods

21:26

we can use to do this and

21:29

it's how both and it

21:31

helps us understand how well both the technical

21:33

and manual parts of the TAR process are performing

21:35

. But beyond the actual

21:37

mechanisms of evaluating a TAR system

21:40

. There's also the issue of visibility

21:43

, or the ability to monitor and understand

21:45

and predict outcomes of the TAR process

21:47

. So this goes beyond the TAR

21:49

system and encompasses like the broader

21:51

capacity to oversee

21:54

the review's progress , the efficiency

21:56

, the effectiveness . Our

21:58

approach at Rigor Data is usually

22:00

through our advanced software capabilities where we

22:02

like prepare dashboarding and predictive analytics

22:05

to provide

22:07

users like with an at-a-glance view of what's

22:09

happening in your review right now and

22:11

analyzing trends in your data so

22:13

we can aid in resource

22:16

allocations . Are there certain sets of documents

22:18

that you'll need to increase your contract attorneys

22:20

for ? If you increase the number of contract

22:22

attorneys , can you finish the review faster

22:25

? Will it diminish effectiveness

22:27

? We also use them for

22:29

forecasting for costs , for

22:32

better budgets and for cost efficiency

22:34

and to optimize the review

22:37

process Because sometimes

22:39

you can like switch modes . So

22:41

let's say at one point you're

22:44

reviewing like full

22:46

families . What if I

22:48

switch to only reviewing parent

22:50

documents ? Or what if I switch to

22:53

reviewing this set person , this one ? How long would

22:55

it take me to finish the TAR process ? So

22:57

we can help you detour

22:59

to a different lane if

23:02

it will make the process more optimized and

23:04

we use our software

23:07

and dashboarding for identifying

23:09

potential risks in the e-discovery process

23:11

that might cause issues downstream .

23:14

Okay . So when I'm running my

23:16

discovery process and I'm receiving

23:19

from my vendor the periodic

23:21

updates that tell me how

23:24

the review is progressing , from a recall

23:27

, from those perspectives , that

23:33

is considered a

23:36

system , a review system , audit

23:38

, just that , that

23:40

process .

23:42

Well , it depends on if you're auditing the

23:44

technology , which would be through

23:46

the random sampling and

23:48

, you know , evaluating the classifier itself , or

23:51

if you're evaluating the

23:53

effectiveness of the process , like is

23:55

the process most optimal ? Are

23:58

the costs optimized adequately

24:00

? Are your resources

24:03

being allocated adequately ? So there's

24:05

a whole ecosystem around

24:07

what the classifier itself is doing . That

24:09

also needs to be evaluated for the

24:11

most effective TAR process as

24:13

a whole and not just the part where

24:15

the documents are being coded , because there's this whole

24:18

ecosystem around it that affects it greatly

24:21

that usually attorneys

24:25

don't have much visibility into . But it's

24:27

our passion at Red

24:29

Grave Data to help people make more informed decisions

24:31

about their review processes through

24:34

showing them , through predictive

24:36

analytics , custom software on

24:38

top of your review platforms and

24:40

your search platforms , and

24:43

showing you what is happening

24:45

in the data , what is happening in your review

24:47

, if not real time , near

24:49

time , so that you can make decisions on

24:53

the ball about whether you need

24:55

to switch processes or switch lanes or detour

24:57

to another process to help you out in

25:00

optimizing your review process itself .

25:02

Gotcha .

25:03

Okay , and optimizing your review process

25:05

itself Gotcha , okay . So

25:11

the kind of data that I am used to seeing in a review process is just

25:13

one set of relevant information that could be referred to when people

25:16

use the phrase review system audits

25:18

. But there is this whole other

25:20

set of data that also can be encompassed , whole other

25:22

set of data that also can be encompassed , and

25:24

so it sounds like when it's it's

25:31

another like term of art where , when people talk to you about review system audits , you

25:33

need to clarify and make sure you know what they're including in that

25:35

and that you're getting all of the information

25:38

required to make informed decisions

25:40

Exactly and it's shown

25:42

to you in a way that you can actually

25:45

make decisions about it .

25:47

Let's say , for instance , in

25:50

a regular review , the collection itself is

25:52

growing all the time . Sometimes you're adding

25:54

more custodians , sometimes

25:56

date ranges change or search

26:01

terms change and things like that . How is that

26:03

going to affect the collection ? Search terms change and things

26:05

like that . How is that going to affect

26:07

the collection ? Like

26:10

, in some cases , you don't actually have the hard data as to how that's affecting

26:12

the collection or how long the review will go . But with these systems

26:14

that we place on top of you

26:16

know , let's say , a relativity instance where

26:19

we're pulling numbers and reporting

26:21

from your relativity instance and placing them

26:23

in the dashboard where you can see what's happening

26:25

with the data , I think it

26:27

makes it a lot more of a data

26:31

intensive process is your

26:33

decision-making becomes .

26:35

Okay , so that makes a lot of sense and it sounds

26:37

like it's something you really want

26:39

to incorporate into any

26:41

review of significant

26:44

size . Material size .

26:46

Absolutely , because you know like one misstep can

26:48

cost you so much as far as time

26:50

and effort and attorney review time

26:53

and taking your senior attorneys

26:55

away from strategy and you know back into

26:58

. You know doing samples and

27:00

I think our

27:03

approach here at Red Grave Data is one

27:05

that has great promise . You

27:08

know doing that last 20 percent that

27:10

the review and the e-discovery platforms

27:12

don't do , and also presenting it to

27:14

our customers clients so that they have

27:16

the specific information they need

27:27

in each matter to make the most

27:29

data-driven decision possible

27:31

.

27:32

Thank you so much . I

27:34

do want to encourage the listeners

27:36

. If you haven't read Beyond

27:38

the Bar Generative AI as a

27:41

transformative component in the legal document

27:43

review , I would encourage you to

27:45

do that . I found it really interesting

27:47

and informative and , even if you don't understand

27:50

everything in there , getting

27:53

that high level understanding from

27:55

the article on where the industry

27:57

might go and

27:59

really seeing

28:01

what kind of options

28:03

are being explored , I think is a

28:06

fantastic use of your time

28:08

. So if you're looking to inform

28:10

yourself a little more , I encourage

28:13

you to read this article .

28:15

And if you have any questions , feel free to reach out . I'm

28:17

happy to answer any questions about the paper

28:20

or any of the processes we talked about today

28:22

.

28:23

I do realize calling it an article instead of

28:25

a paper is definitely a misnomer . It is

28:27

definitely a paper . It is beyond article

28:29

length for the

28:31

kinds of articles you're used to getting , but

28:35

I really did enjoy it , so it's fantastic

28:37

. Thank you so much for sharing that .

28:39

No problem , thank you .

28:41

We are also going to be

28:43

making available

28:45

a TAR reference that

28:47

Lenora was kind enough to pull

28:50

together for us , so that

28:52

will be available on

28:54

Emerging Litigation Podcast website

28:56

and I want to thank

28:58

Tom Hagee and the Emerging Litigation

29:01

Podcast for letting me guest host

29:03

once again and the Emerging Litigation Podcast for letting me guest

29:05

host once again .

29:06

That concludes this episode of the Emerging

29:08

Litigation Podcast , a co-production

29:10

of HB Litigation , critical Legal

29:13

Content , vlex Fast

29:15

Case and our friends at Losty Media

29:17

. I'm Tom Hagee , your host

29:20

, which would explain why I'm talking . Please

29:22

feel free to reach out to me if you have ideas for

29:24

a future episode and

29:27

don't hesitate to share this with clients , colleagues , friends , animals

29:30

you may have left at home , teenagers you've irresponsibly

29:33

left unsupervised , and

29:35

certain classifications of fruits and vegetables

29:37

. And if you feel so moved , please give

29:39

us a rating . Those always help . Thank

29:41

you for listening .